npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.54 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (475) hide show

package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-01-foundations.md CHANGED Viewed

@@ -1,735 +1,735 @@
-# Module 1: Foundations
-Learn when to use S3 vs SFTP, understand the data source architecture, and master configuration patterns.
-## Table of Contents
-- [S3 vs SFTP: When to Use Each](#s3-vs-sftp-when-to-use-each)
-- [Decision Guide](#decision-guide)
-- [Architecture Overview](#04-REFERENCE/architecture-overview)
-- [Configuration Fundamentals](#configuration-fundamentals)
-- [Universal Interface Design](#universal-interface-design)
-- [Real-World Scenarios](#real-world-scenarios)
----
-## S3 vs SFTP: When to Use Each
-### Quick Comparison
-| Aspect               | S3DataSource                              | SftpDataSource                                     |
-| -------------------- | ----------------------------------------- | -------------------------------------------------- |
-| **Use Case**         | Cloud-native storage, scalable data lakes | Vendor integrations, enterprise systems                |
-| **Transport**        | HTTPS (presigned URLs)                    | SSH (SFTP protocol)                                |
-| **Authentication**   | AWS IAM credentials                       | Password or SSH private key                        |
-| **Scalability**      | Unlimited (AWS managed)                   | Limited by vendor server capacity                  |
-| **Cost Model**       | Pay-per-GB storage + requests             | Fixed vendor agreement                             |
-| **Latency**          | Low (global CDN)                          | Variable (depends on vendor infrastructure)        |
-| **Setup Complexity** | Low (AWS console or IaC)                  | Medium (SSH keys, firewall rules, IP whitelisting) |
-| **Platform Support** | Node.js, Deno, Versori                    | Node.js, Deno, Versori                             |
-| **Best For**         | Internal integrations, modern cloud apps  | External vendor/partner integrations               |
----
-## Decision Guide
-### Use S3 When:
-#### ✅ Building Cloud-Native Integrations
-**Scenario**: You're building a modern integration where you control both ends of the data pipeline.
-```typescript
-// S3 for cloud-native inventory ingestion
-import { S3DataSource, CSVParserService } from '@fluentcommerce/fc-connect-sdk';
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-bucket',
-    },
-  },
-  logger
-);
-// Read from your own S3 bucket
-const csvContent = await s3Source.downloadFile('inventory.csv');
-```
-**Why S3?**
-- You control the infrastructure
-- Scalable to any data volume
-- No vendor coordination needed
-- Easy to monitor and debug
-#### ✅ Need Scalable, Durable Storage
-**Scenario**: Processing large datasets (GB to TB scale) with high durability requirements.
-```
-S3 Advantages:
-┌────────────────────────────────────────────┐
-│ Scalability: Unlimited storage             │
-│ Durability: 99.999999999% (11 nines)       │
-│ Availability: 99.99% SLA                   │
-│ Versioning: Track file changes             │
-│ Lifecycle: Auto-archive to Glacier         │
-└────────────────────────────────────────────┘
-```
-**Example**: Multi-TB inventory extraction
-```typescript
-// Extract millions of inventory records to S3
-const s3Target = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-      region: 'us-west-2',
-      bucket: 'extraction-bucket',
-    },
-  },
-  logger
-);
-// S3 handles large files effortlessly
-await s3Target.writeFile(
-  's3://extraction-bucket/inventory-full.parquet',
-  parquetData // 10+ GB file
-);
-```
-#### ✅ Deploying to Serverless/Versori Platforms
-**Scenario**: Running in AWS Lambda, Versori, or other serverless environments.
-**Why S3?**
-- No persistent connections needed (stateless HTTP)
-- Presigned URLs work everywhere
-- No network complexity (firewall rules, IP whitelisting)
-- Native AWS integration
-```typescript
-// Versori connector using S3 (zero connection config)
-import { createClient, S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-export const inventorySync = webhook('s3-inventory', async (ctx) => {
-  const client = await createClient(ctx); // Auto-detects Versori context
-  const s3Source = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      s3Config: {
-        accessKeyId: activation.env.AWS_ACCESS_KEY_ID,
-        secretAccessKey: activation.env.AWS_SECRET_ACCESS_KEY,
-        region: 'us-east-1',
-        bucket: activation.env.S3_BUCKET,
-      },
-    },
-    log
-  );
-  // Works instantly - no connection pooling or SSH setup
-  const inventory = await s3Source.downloadFile('file.csv');
-});
-```
-#### ✅ Need Global Distribution
-**Scenario**: Multi-region deployments requiring low-latency access worldwide.
-```
-S3 + CloudFront Architecture:
-┌─────────────────┐
-│ S3 us-east-1    │◄─── Upload inventory files
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│ CloudFront CDN  │◄─── Global edge caching
-└────────┬────────┘
-         │
-    ┌────┴────┬────────┬────────┐
-    ▼         ▼        ▼        ▼
-  EU     Asia-Pacific  US     LATAM
-(low latency everywhere)
-```
----
-### Use SFTP When:
-#### ✅ Integrating with Vendor/Partner Systems
-**Scenario**: Your vendor provides inventory files via SFTP and won't change their process.
-```typescript
-// Vendor requires SFTP pickup
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-const vendorSftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-abc',
-    settings: {
-      host: 'sftp.vendor-abc.com',
-      port: 22,
-      username: 'integration_user',
-      privateKey: process.env.VENDOR_SFTP_KEY,
-      remotePath: '/outbound/inventory',
-      filePattern: 'inventory_*.csv',
-    },
-  },
-  logger
-);
-// Read vendor's latest file
-const files = await vendorSftp.listFiles();
-const latestFile = files[0];
-const inventory = await vendorSftp.downloadFile(latestFile.path);
-```
-**Why SFTP?**
-- Vendor dictates the integration method
-- No cloud infrastructure setup required on vendor side
-- Works with any vendor (no AWS account needed)
-- Standard protocol (RFC 4253) - universal compatibility
-#### ✅ Working with Enterprise Systems
-**Scenario**: Enterprise ERP system that only supports SFTP for data exchange.
-```
-Enterprise ERP Integration Pattern:
-┌─────────────────┐
-│ SAP / Oracle ERP│
-│                 │
-│ SFTP Module     │◄─── Only supported export method
-└────────┬────────┘
-         │ (SFTP pickup required)
-         ▼
-┌─────────────────┐
-│ SftpDataSource  │
-│ (SDK)           │
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│ Fluent Commerce │
-└─────────────────┘
-```
-**Example**: SAP inventory export
-```typescript
-const sapSftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'sap-prod',
-    settings: {
-      host: 'sap-server.enterprise.internal',
-      port: 22,
-      username: 'fluent_integration',
-      password: process.env.SAP_SFTP_PASSWORD,
-      remotePath: '/exports/inventory',
-      filePattern: 'INV_*.csv',
-    },
-  },
-  logger
-);
-```
-#### ✅ Required by Compliance/Security Policies
-**Scenario**: Company security policy mandates SFTP for PII or financial data.
-**SFTP Security Advantages**:
-- Direct encrypted channel (SSH)
-- No data stored in cloud
-- Audit logs on company servers
-- Fine-grained access control (SSH keys per user)
-- No external cloud dependencies
-```typescript
-// HIPAA-compliant patient data transfer
-const secureSftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'hipaa-sftp',
-    settings: {
-      host: 'secure-sftp.healthcare.internal',
-      port: 2222, // Non-standard port for security
-      username: 'phi_integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY,
-      passphrase: process.env.SFTP_KEY_PASSPHRASE,
-      remotePath: '/phi/exports',
-      strictHostKeyChecking: true,
-      knownHosts: process.env.SFTP_KNOWN_HOSTS,
-    },
-  },
-  logger
-);
-```
-#### ✅ Vendor Provides SFTP-Only Access
-**Scenario**: Third-party logistics (3PL) provider only offers SFTP for inventory updates.
-**Example Decision Matrix**:
-| Vendor Type       | Typical Protocol | SDK Choice       |
-| ----------------- | ---------------- | ---------------- |
-| 3PL / Warehouse   | SFTP             | `SftpDataSource` |
-| Cloud Marketplace | REST API + S3    | `S3DataSource`   |
-| Enterprise Supplier   | SFTP / FTP       | `SftpDataSource` |
-| Modern SaaS       | Webhook + S3     | `S3DataSource`   |
-| Enterprise ERP    | SFTP / EDI       | `SftpDataSource` |
----
-## Architecture Overview
-### S3DataSource Architecture
-```
-┌──────────────────────────────────────────────────────────┐
-│                     S3DataSource                          │
-│  (Presigned URL Architecture - Universal Compatibility)   │
-└──────────────────┬───────────────────────────────────────┘
-                   │
-         ┌─────────┴─────────┐
-         │                   │
-         ▼                   ▼
-┌─────────────────┐   ┌─────────────────┐
-│ S3PresignService│   │ Native fetch()  │
-│ (SigV4 signing) │   │ (HTTP client)   │
-└────────┬────────┘   └────────┬────────┘
-         │                     │
-         ▼                     ▼
-    Generate URL          Execute Request
-         │                     │
-         └──────────┬──────────┘
-                    ▼
-         ┌─────────────────────┐
-         │  AWS S3 Bucket      │
-         │  (HTTPS endpoint)   │
-         └─────────────────────┘
-Key Benefits:
-✅ No AWS SDK bundling (smaller package size)
-✅ Works in Node.js, Deno, Versori
-✅ Simple fetch() API (standard Web API)
-✅ Automatic signature generation
-```
-**Core Components**:
-- `S3DataSource`: High-level file operations API
-- `S3PresignService`: Generates signed URLs (AWS SigV4)
-- Native `fetch()`: Executes HTTP requests to S3
-**Why Presigned URLs?**
-- Universal compatibility (no platform-specific code)
-- Smaller bundle size (no AWS SDK)
-- Simpler debugging (plain HTTP requests)
-- Works with corporate proxies and firewalls
-### SftpDataSource Architecture
-```
-┌──────────────────────────────────────────────────────────┐
-│                   SftpDataSource                          │
-│         (Direct SSH - Connection Pooling)                 │
-└──────────────────┬───────────────────────────────────────┘
-                   │
-         ┌─────────┴─────────┐
-         │                   │
-         ▼                   ▼
-┌─────────────────┐   ┌─────────────────┐
-│ ssh2-sftp-client│   │ Connection Pool │
-│ (SFTP protocol) │   │ (max 5 default) │
-└────────┬────────┘   └────────┬────────┘
-         │                     │
-         ▼                     ▼
-    SSH Handshake         Reuse Connections
-         │                     │
-         └──────────┬──────────┘
-                    ▼
-         ┌─────────────────────┐
-         │  SFTP Server        │
-         │  (SSH port 22)      │
-         └─────────────────────┘
-Key Benefits:
-✅ Connection pooling (reduced overhead)
-✅ Automatic retry with exponential backoff
-✅ Password OR private key authentication
-✅ Direct SSH encryption (no intermediary)
-```
-**Core Components**:
-- `SftpDataSource`: High-level file operations API
-- `ssh2-sftp-client`: SSH/SFTP protocol implementation
-- Connection pool: Reuses SSH connections efficiently
-**Why Direct SSH?**
-- Industry standard for B2B file transfer
-- No cloud infrastructure required
-- Works with any vendor
-- Strong encryption (SSH)
----
-## Configuration Fundamentals
-### S3 Configuration Pattern
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-// Minimal S3 configuration
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      // Optional
-      sessionToken: process.env.AWS_SESSION_TOKEN,
-      endpoint: process.env.AWS_ENDPOINT, // For LocalStack testing
-    },
-  },
-  logger
-);
-// Usage
-const content = await s3Source.downloadFile('path/to/file.csv');
-```
-**Configuration Properties**:
-| Property                   | Required | Description        | Default     |
-| -------------------------- | -------- | ------------------ | ----------- |
-| `type`                     | ✅       | Data source type   | -           |
-| `s3Config.accessKeyId`     | ✅       | AWS access key     | -           |
-| `s3Config.secretAccessKey` | ✅       | AWS secret key     | -           |
-| `s3Config.region`          | ✅       | AWS region         | -           |
-| `s3Config.sessionToken`    | ❌       | STS session token  | -           |
-| `s3Config.endpoint`        | ❌       | Custom S3 endpoint | AWS default |
-### SFTP Configuration Pattern
-```typescript
-import {
-  SftpDataSource,
-  createConsoleLogger,
-  toStructuredLogger
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info'
-});
-// Minimal SFTP configuration (password auth)
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
-    connectionId: 'vendor-sftp',
-    name: 'Vendor SFTP Connection',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'integration_user',
-      password: process.env.SFTP_PASSWORD,
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-    },
-  },
-  logger
-);
-// Usage
-const files = await sftpSource.listFiles();
-const content = await sftpSource.downloadFile(files[0].path);
-```
-**Configuration Properties**:
-| Property                     | Required | Description                       | Default |
-| ---------------------------- | -------- | --------------------------------- | ------- |
-| `type`                       | ✅       | Data source type                  | -       |
-| `connectionId`               | ✅       | Unique connection ID              | -       |
-| `settings.host`              | ✅       | SFTP server hostname              | -       |
-| `settings.port`              | ❌       | SSH port                          | 22      |
-| `settings.username`          | ✅       | SSH username                      | -       |
-| `settings.password`          | ✅\*     | Password (or use privateKey)      | -       |
-| `settings.privateKey`        | ✅\*     | SSH private key (or use password) | -       |
-| `settings.passphrase`        | ❌       | Key passphrase (if encrypted)     | -       |
-| `settings.remotePath`        | ❌       | Remote directory path             | `/`     |
-| `settings.filePattern`       | ❌       | File glob pattern                 | `*`     |
-| `settings.maxConnections`    | ❌       | Connection pool size              | 5       |
-| `settings.connectionTimeout` | ❌       | Connection timeout (ms)           | 30000   |
-\* Either `password` or `privateKey` required
----
-## Universal Interface Design
-Both `S3DataSource` and `SftpDataSource` implement the same core interface, making it easy to swap implementations:
-```typescript
-// Note: The two data sources expose similar high-level operations,
-// but method names differ slightly.
-// S3DataSource: listFiles, downloadFile, uploadFile, deleteFile, copyFile, moveFile, validateConnection
-// SftpDataSource: listFiles, downloadFile, writeFile, uploadContent, deleteFile, copyFile, moveFile, createDirectory, fileExists, directoryExists, validateConnection
-```
-### Switching Between S3 and SFTP
-**Example**: Switch from S3 to SFTP without changing business logic
-```typescript
-// Option 1: S3
-const dataSource = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      /* ... */
-    },
-  },
-  logger
-);
-// Option 2: SFTP (same interface!)
-const dataSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor',
-    settings: {
-      /* ... */
-    },
-  },
-  logger
-);
-// Business logic stays identical
-const files = await dataSource.listFiles({ remotePath: 'inventory/' });
-for (const file of files) {
-  const content = await dataSource.downloadFile(file.path);
-  const records = parseCSV(content as string);
-  await processRecords(records);
-}
-```
-**Benefits**:
-- Write business logic once
-- Easy to test (swap for mock data source)
-- Flexible deployment (change config, not code)
-- Vendor migration (S3 → SFTP or vice versa)
-Note: On SFTP, `parseParquetContent` is not implemented and throws. Use `ParquetParserService` after downloading the buffer.
----
-## Real-World Scenarios
-### Scenario 1: Modern E-Commerce Integration
-**Requirements**:
-- Ingest inventory from internal warehouse system
-- High volume (100K+ SKUs daily)
-- Need to scale globally
-- Team controls infrastructure
-**Decision**: **S3DataSource**
-```typescript
-// Why S3:
-// ✅ Internal system (we control both ends)
-// ✅ High volume (S3 scales infinitely)
-// ✅ Global deployment (CloudFront distribution)
-// ✅ No vendor coordination needed
-const inventorySource = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-    },
-  },
-  logger
-);
-// Warehouse uploads to S3, we process
-const inventory = await inventorySource.downloadFile('daily-inventory.csv');
-```
----
-### Scenario 2: Third-Party Logistics (3PL) Integration
-**Requirements**:
-- 3PL provides inventory via SFTP only
-- Medium volume (10K SKUs, hourly updates)
-- 3PL won't change their process
-- Security: SSH key authentication required
-**Decision**: **SftpDataSource**
-```typescript
-// Why SFTP:
-// ✅ Vendor requirement (SFTP only)
-// ✅ No control over vendor infrastructure
-// ✅ Standard B2B integration protocol
-// ✅ SSH key authentication supported
-const threePlSftp = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: '3pl-vendor',
-    settings: {
-      host: 'sftp.3plvendor.com',
-      port: 22,
-      username: 'fluent_integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY,
-      remotePath: '/outbound/inventory',
-      filePattern: 'inventory_*.csv',
-    },
-  },
-  logger
-);
-// Poll for new files
-const files = await threePlSftp.listFiles();
-```
----
-### Scenario 3: Hybrid Architecture
-**Requirements**:
-- Ingest from multiple vendors (S3 + SFTP)
-- Extract to data lake (S3)
-- Unified processing pipeline
-**Decision**: **Both S3DataSource and SftpDataSource**
-```typescript
-// Vendor A: Provides S3 bucket
-const vendorA = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.VENDOR_A_KEY!,
-      secretAccessKey: process.env.VENDOR_A_SECRET!,
-      region: 'us-west-2',
-    },
-  },
-  logger
-);
-// Vendor B: Provides SFTP server
-const vendorB = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-b',
-    settings: {
-      host: 'sftp.vendorb.com',
-      username: 'integration',
-      privateKey: process.env.VENDOR_B_KEY,
-    },
-  },
-  logger
-);
-// Unified processing function (works with both!)
-async function ingestInventory(dataSource: DataSource, prefix: string) {
-  const files = await dataSource.listFiles(prefix);
-  for (const file of files) {
-    const content = await dataSource.downloadFile(file.path);
-    await processInventory(content);
-  }
-}
-// Process both vendors with same code
-await ingestInventory(vendorA, 'inventory/');
-await ingestInventory(vendorB, 'inventory/');
-```
----
-## Summary
-### Key Takeaways
-1. **S3 for cloud-native**: When you control infrastructure and need scalability
-2. **SFTP for vendor integrations**: When working with external partners or enterprise systems
-3. **Universal interface**: Same API for both S3 and SFTP
-4. **Presigned URLs (S3)**: Universal compatibility without AWS SDK bundling
-5. **Direct SSH (SFTP)**: Standard protocol for B2B file exchange
-### Decision Flowchart
-```
-Do you control the infrastructure?
-         │
-    ┌────┴────┐
-   YES       NO
-    │         │
-    │         └──► Use SFTP (vendor requirement)
-    │
-Is this a cloud-native integration?
-         │
-    ┌────┴────┐
-   YES       NO
-    │         │
-    └──► S3   └──► Is volume high (>10K records/hour)?
-                         │
-                    ┌────┴────┐
-                   YES       NO
-                    │         │
-                    └──► S3   └──► Either (your preference)
-```
----
-## Next Steps
-**Ready for S3?** → [Module 2: S3 Operations](./data-sources-02-s3-operations.md)
-**Ready for SFTP?** → [Module 3: SFTP Operations](./data-sources-03-sftp-operations.md)
-**Need both?** → Read both modules, then [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
----
-[← Back to Guide](../data-sources-readme.md) | [Next: S3 Operations →](./data-sources-02-s3-operations.md)
+# Module 1: Foundations
+Learn when to use S3 vs SFTP, understand the data source architecture, and master configuration patterns.
+## Table of Contents
+- [S3 vs SFTP: When to Use Each](#s3-vs-sftp-when-to-use-each)
+- [Decision Guide](#decision-guide)
+- [Architecture Overview](#04-REFERENCE/architecture-overview)
+- [Configuration Fundamentals](#configuration-fundamentals)
+- [Universal Interface Design](#universal-interface-design)
+- [Real-World Scenarios](#real-world-scenarios)
+---
+## S3 vs SFTP: When to Use Each
+### Quick Comparison
+| Aspect               | S3DataSource                              | SftpDataSource                                     |
+| -------------------- | ----------------------------------------- | -------------------------------------------------- |
+| **Use Case**         | Cloud-native storage, scalable data lakes | Vendor integrations, enterprise systems                |
+| **Transport**        | HTTPS (presigned URLs)                    | SSH (SFTP protocol)                                |
+| **Authentication**   | AWS IAM credentials                       | Password or SSH private key                        |
+| **Scalability**      | Unlimited (AWS managed)                   | Limited by vendor server capacity                  |
+| **Cost Model**       | Pay-per-GB storage + requests             | Fixed vendor agreement                             |
+| **Latency**          | Low (global CDN)                          | Variable (depends on vendor infrastructure)        |
+| **Setup Complexity** | Low (AWS console or IaC)                  | Medium (SSH keys, firewall rules, IP whitelisting) |
+| **Platform Support** | Node.js, Deno, Versori                    | Node.js, Deno, Versori                             |
+| **Best For**         | Internal integrations, modern cloud apps  | External vendor/partner integrations               |
+---
+## Decision Guide
+### Use S3 When:
+#### ✅ Building Cloud-Native Integrations
+**Scenario**: You're building a modern integration where you control both ends of the data pipeline.
+```typescript
+// S3 for cloud-native inventory ingestion
+import { S3DataSource, CSVParserService } from '@fluentcommerce/fc-connect-sdk';
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-bucket',
+    },
+  },
+  logger
+);
+// Read from your own S3 bucket
+const csvContent = await s3Source.downloadFile('inventory.csv');
+```
+**Why S3?**
+- You control the infrastructure
+- Scalable to any data volume
+- No vendor coordination needed
+- Easy to monitor and debug
+#### ✅ Need Scalable, Durable Storage
+**Scenario**: Processing large datasets (GB to TB scale) with high durability requirements.
+```
+S3 Advantages:
+┌────────────────────────────────────────────┐
+│ Scalability: Unlimited storage             │
+│ Durability: 99.999999999% (11 nines)       │
+│ Availability: 99.99% SLA                   │
+│ Versioning: Track file changes             │
+│ Lifecycle: Auto-archive to Glacier         │
+└────────────────────────────────────────────┘
+```
+**Example**: Multi-TB inventory extraction
+```typescript
+// Extract millions of inventory records to S3
+const s3Target = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+      region: 'us-west-2',
+      bucket: 'extraction-bucket',
+    },
+  },
+  logger
+);
+// S3 handles large files effortlessly
+await s3Target.writeFile(
+  's3://extraction-bucket/inventory-full.parquet',
+  parquetData // 10+ GB file
+);
+```
+#### ✅ Deploying to Serverless/Versori Platforms
+**Scenario**: Running in AWS Lambda, Versori, or other serverless environments.
+**Why S3?**
+- No persistent connections needed (stateless HTTP)
+- Presigned URLs work everywhere
+- No network complexity (firewall rules, IP whitelisting)
+- Native AWS integration
+```typescript
+// Versori connector using S3 (zero connection config)
+import { createClient, S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+export const inventorySync = webhook('s3-inventory', async (ctx) => {
+  const client = await createClient(ctx); // Auto-detects Versori context
+  const s3Source = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      s3Config: {
+        accessKeyId: activation.env.AWS_ACCESS_KEY_ID,
+        secretAccessKey: activation.env.AWS_SECRET_ACCESS_KEY,
+        region: 'us-east-1',
+        bucket: activation.env.S3_BUCKET,
+      },
+    },
+    log
+  );
+  // Works instantly - no connection pooling or SSH setup
+  const inventory = await s3Source.downloadFile('file.csv');
+});
+```
+#### ✅ Need Global Distribution
+**Scenario**: Multi-region deployments requiring low-latency access worldwide.
+```
+S3 + CloudFront Architecture:
+┌─────────────────┐
+│ S3 us-east-1    │◄─── Upload inventory files
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ CloudFront CDN  │◄─── Global edge caching
+└────────┬────────┘
+         │
+    ┌────┴────┬────────┬────────┐
+    ▼         ▼        ▼        ▼
+  EU     Asia-Pacific  US     LATAM
+(low latency everywhere)
+```
+---
+### Use SFTP When:
+#### ✅ Integrating with Vendor/Partner Systems
+**Scenario**: Your vendor provides inventory files via SFTP and won't change their process.
+```typescript
+// Vendor requires SFTP pickup
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+const vendorSftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-abc',
+    settings: {
+      host: 'sftp.vendor-abc.com',
+      port: 22,
+      username: 'integration_user',
+      privateKey: process.env.VENDOR_SFTP_KEY,
+      remotePath: '/outbound/inventory',
+      filePattern: 'inventory_*.csv',
+    },
+  },
+  logger
+);
+// Read vendor's latest file
+const files = await vendorSftp.listFiles();
+const latestFile = files[0];
+const inventory = await vendorSftp.downloadFile(latestFile.path);
+```
+**Why SFTP?**
+- Vendor dictates the integration method
+- No cloud infrastructure setup required on vendor side
+- Works with any vendor (no AWS account needed)
+- Standard protocol (RFC 4253) - universal compatibility
+#### ✅ Working with Enterprise Systems
+**Scenario**: Enterprise ERP system that only supports SFTP for data exchange.
+```
+Enterprise ERP Integration Pattern:
+┌─────────────────┐
+│ SAP / Oracle ERP│
+│                 │
+│ SFTP Module     │◄─── Only supported export method
+└────────┬────────┘
+         │ (SFTP pickup required)
+         ▼
+┌─────────────────┐
+│ SftpDataSource  │
+│ (SDK)           │
+└────────┬────────┘
+         │
+         ▼
+┌─────────────────┐
+│ Fluent Commerce │
+└─────────────────┘
+```
+**Example**: SAP inventory export
+```typescript
+const sapSftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'sap-prod',
+    settings: {
+      host: 'sap-server.enterprise.internal',
+      port: 22,
+      username: 'fluent_integration',
+      password: process.env.SAP_SFTP_PASSWORD,
+      remotePath: '/exports/inventory',
+      filePattern: 'INV_*.csv',
+    },
+  },
+  logger
+);
+```
+#### ✅ Required by Compliance/Security Policies
+**Scenario**: Company security policy mandates SFTP for PII or financial data.
+**SFTP Security Advantages**:
+- Direct encrypted channel (SSH)
+- No data stored in cloud
+- Audit logs on company servers
+- Fine-grained access control (SSH keys per user)
+- No external cloud dependencies
+```typescript
+// HIPAA-compliant patient data transfer
+const secureSftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'hipaa-sftp',
+    settings: {
+      host: 'secure-sftp.healthcare.internal',
+      port: 2222, // Non-standard port for security
+      username: 'phi_integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY,
+      passphrase: process.env.SFTP_KEY_PASSPHRASE,
+      remotePath: '/phi/exports',
+      strictHostKeyChecking: true,
+      knownHosts: process.env.SFTP_KNOWN_HOSTS,
+    },
+  },
+  logger
+);
+```
+#### ✅ Vendor Provides SFTP-Only Access
+**Scenario**: Third-party logistics (3PL) provider only offers SFTP for inventory updates.
+**Example Decision Matrix**:
+| Vendor Type       | Typical Protocol | SDK Choice       |
+| ----------------- | ---------------- | ---------------- |
+| 3PL / Warehouse   | SFTP             | `SftpDataSource` |
+| Cloud Marketplace | REST API + S3    | `S3DataSource`   |
+| Enterprise Supplier   | SFTP / FTP       | `SftpDataSource` |
+| Modern SaaS       | Webhook + S3     | `S3DataSource`   |
+| Enterprise ERP    | SFTP / EDI       | `SftpDataSource` |
+---
+## Architecture Overview
+### S3DataSource Architecture
+```
+┌──────────────────────────────────────────────────────────┐
+│                     S3DataSource                          │
+│  (Presigned URL Architecture - Universal Compatibility)   │
+└──────────────────┬───────────────────────────────────────┘
+                   │
+         ┌─────────┴─────────┐
+         │                   │
+         ▼                   ▼
+┌─────────────────┐   ┌─────────────────┐
+│ S3PresignService│   │ Native fetch()  │
+│ (SigV4 signing) │   │ (HTTP client)   │
+└────────┬────────┘   └────────┬────────┘
+         │                     │
+         ▼                     ▼
+    Generate URL          Execute Request
+         │                     │
+         └──────────┬──────────┘
+                    ▼
+         ┌─────────────────────┐
+         │  AWS S3 Bucket      │
+         │  (HTTPS endpoint)   │
+         └─────────────────────┘
+Key Benefits:
+✅ No AWS SDK bundling (smaller package size)
+✅ Works in Node.js, Deno, Versori
+✅ Simple fetch() API (standard Web API)
+✅ Automatic signature generation
+```
+**Core Components**:
+- `S3DataSource`: High-level file operations API
+- `S3PresignService`: Generates signed URLs (AWS SigV4)
+- Native `fetch()`: Executes HTTP requests to S3
+**Why Presigned URLs?**
+- Universal compatibility (no platform-specific code)
+- Smaller bundle size (no AWS SDK)
+- Simpler debugging (plain HTTP requests)
+- Works with corporate proxies and firewalls
+### SftpDataSource Architecture
+```
+┌──────────────────────────────────────────────────────────┐
+│                   SftpDataSource                          │
+│         (Direct SSH - Connection Pooling)                 │
+└──────────────────┬───────────────────────────────────────┘
+                   │
+         ┌─────────┴─────────┐
+         │                   │
+         ▼                   ▼
+┌─────────────────┐   ┌─────────────────┐
+│ ssh2-sftp-client│   │ Connection Pool │
+│ (SFTP protocol) │   │ (max 5 default) │
+└────────┬────────┘   └────────┬────────┘
+         │                     │
+         ▼                     ▼
+    SSH Handshake         Reuse Connections
+         │                     │
+         └──────────┬──────────┘
+                    ▼
+         ┌─────────────────────┐
+         │  SFTP Server        │
+         │  (SSH port 22)      │
+         └─────────────────────┘
+Key Benefits:
+✅ Connection pooling (reduced overhead)
+✅ Automatic retry with exponential backoff
+✅ Password OR private key authentication
+✅ Direct SSH encryption (no intermediary)
+```
+**Core Components**:
+- `SftpDataSource`: High-level file operations API
+- `ssh2-sftp-client`: SSH/SFTP protocol implementation
+- Connection pool: Reuses SSH connections efficiently
+**Why Direct SSH?**
+- Industry standard for B2B file transfer
+- No cloud infrastructure required
+- Works with any vendor
+- Strong encryption (SSH)
+---
+## Configuration Fundamentals
+### S3 Configuration Pattern
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+// Minimal S3 configuration
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      // Optional
+      sessionToken: process.env.AWS_SESSION_TOKEN,
+      endpoint: process.env.AWS_ENDPOINT, // For LocalStack testing
+    },
+  },
+  logger
+);
+// Usage
+const content = await s3Source.downloadFile('path/to/file.csv');
+```
+**Configuration Properties**:
+| Property                   | Required | Description        | Default     |
+| -------------------------- | -------- | ------------------ | ----------- |
+| `type`                     | ✅       | Data source type   | -           |
+| `s3Config.accessKeyId`     | ✅       | AWS access key     | -           |
+| `s3Config.secretAccessKey` | ✅       | AWS secret key     | -           |
+| `s3Config.region`          | ✅       | AWS region         | -           |
+| `s3Config.sessionToken`    | ❌       | STS session token  | -           |
+| `s3Config.endpoint`        | ❌       | Custom S3 endpoint | AWS default |
+### SFTP Configuration Pattern
+```typescript
+import {
+  SftpDataSource,
+  createConsoleLogger,
+  toStructuredLogger
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info'
+});
+// Minimal SFTP configuration (password auth)
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
+    connectionId: 'vendor-sftp',
+    name: 'Vendor SFTP Connection',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'integration_user',
+      password: process.env.SFTP_PASSWORD,
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+    },
+  },
+  logger
+);
+// Usage
+const files = await sftpSource.listFiles();
+const content = await sftpSource.downloadFile(files[0].path);
+```
+**Configuration Properties**:
+| Property                     | Required | Description                       | Default |
+| ---------------------------- | -------- | --------------------------------- | ------- |
+| `type`                       | ✅       | Data source type                  | -       |
+| `connectionId`               | ✅       | Unique connection ID              | -       |
+| `settings.host`              | ✅       | SFTP server hostname              | -       |
+| `settings.port`              | ❌       | SSH port                          | 22      |
+| `settings.username`          | ✅       | SSH username                      | -       |
+| `settings.password`          | ✅\*     | Password (or use privateKey)      | -       |
+| `settings.privateKey`        | ✅\*     | SSH private key (or use password) | -       |
+| `settings.passphrase`        | ❌       | Key passphrase (if encrypted)     | -       |
+| `settings.remotePath`        | ❌       | Remote directory path             | `/`     |
+| `settings.filePattern`       | ❌       | File glob pattern                 | `*`     |
+| `settings.maxConnections`    | ❌       | Connection pool size              | 5       |
+| `settings.connectionTimeout` | ❌       | Connection timeout (ms)           | 30000   |
+\* Either `password` or `privateKey` required
+---
+## Universal Interface Design
+Both `S3DataSource` and `SftpDataSource` implement the same core interface, making it easy to swap implementations:
+```typescript
+// Note: The two data sources expose similar high-level operations,
+// but method names differ slightly.
+// S3DataSource: listFiles, downloadFile, uploadFile, deleteFile, copyFile, moveFile, validateConnection
+// SftpDataSource: listFiles, downloadFile, writeFile, uploadContent, deleteFile, copyFile, moveFile, createDirectory, fileExists, directoryExists, validateConnection
+```
+### Switching Between S3 and SFTP
+**Example**: Switch from S3 to SFTP without changing business logic
+```typescript
+// Option 1: S3
+const dataSource = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      /* ... */
+    },
+  },
+  logger
+);
+// Option 2: SFTP (same interface!)
+const dataSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor',
+    settings: {
+      /* ... */
+    },
+  },
+  logger
+);
+// Business logic stays identical
+const files = await dataSource.listFiles({ remotePath: 'inventory/' });
+for (const file of files) {
+  const content = await dataSource.downloadFile(file.path);
+  const records = parseCSV(content as string);
+  await processRecords(records);
+}
+```
+**Benefits**:
+- Write business logic once
+- Easy to test (swap for mock data source)
+- Flexible deployment (change config, not code)
+- Vendor migration (S3 → SFTP or vice versa)
+Note: On SFTP, `parseParquetContent` is not implemented and throws. Use `ParquetParserService` after downloading the buffer.
+---
+## Real-World Scenarios
+### Scenario 1: Modern E-Commerce Integration
+**Requirements**:
+- Ingest inventory from internal warehouse system
+- High volume (100K+ SKUs daily)
+- Need to scale globally
+- Team controls infrastructure
+**Decision**: **S3DataSource**
+```typescript
+// Why S3:
+// ✅ Internal system (we control both ends)
+// ✅ High volume (S3 scales infinitely)
+// ✅ Global deployment (CloudFront distribution)
+// ✅ No vendor coordination needed
+const inventorySource = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+    },
+  },
+  logger
+);
+// Warehouse uploads to S3, we process
+const inventory = await inventorySource.downloadFile('daily-inventory.csv');
+```
+---
+### Scenario 2: Third-Party Logistics (3PL) Integration
+**Requirements**:
+- 3PL provides inventory via SFTP only
+- Medium volume (10K SKUs, hourly updates)
+- 3PL won't change their process
+- Security: SSH key authentication required
+**Decision**: **SftpDataSource**
+```typescript
+// Why SFTP:
+// ✅ Vendor requirement (SFTP only)
+// ✅ No control over vendor infrastructure
+// ✅ Standard B2B integration protocol
+// ✅ SSH key authentication supported
+const threePlSftp = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: '3pl-vendor',
+    settings: {
+      host: 'sftp.3plvendor.com',
+      port: 22,
+      username: 'fluent_integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY,
+      remotePath: '/outbound/inventory',
+      filePattern: 'inventory_*.csv',
+    },
+  },
+  logger
+);
+// Poll for new files
+const files = await threePlSftp.listFiles();
+```
+---
+### Scenario 3: Hybrid Architecture
+**Requirements**:
+- Ingest from multiple vendors (S3 + SFTP)
+- Extract to data lake (S3)
+- Unified processing pipeline
+**Decision**: **Both S3DataSource and SftpDataSource**
+```typescript
+// Vendor A: Provides S3 bucket
+const vendorA = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.VENDOR_A_KEY!,
+      secretAccessKey: process.env.VENDOR_A_SECRET!,
+      region: 'us-west-2',
+    },
+  },
+  logger
+);
+// Vendor B: Provides SFTP server
+const vendorB = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-b',
+    settings: {
+      host: 'sftp.vendorb.com',
+      username: 'integration',
+      privateKey: process.env.VENDOR_B_KEY,
+    },
+  },
+  logger
+);
+// Unified processing function (works with both!)
+async function ingestInventory(dataSource: DataSource, prefix: string) {
+  const files = await dataSource.listFiles(prefix);
+  for (const file of files) {
+    const content = await dataSource.downloadFile(file.path);
+    await processInventory(content);
+  }
+}
+// Process both vendors with same code
+await ingestInventory(vendorA, 'inventory/');
+await ingestInventory(vendorB, 'inventory/');
+```
+---
+## Summary
+### Key Takeaways
+1. **S3 for cloud-native**: When you control infrastructure and need scalability
+2. **SFTP for vendor integrations**: When working with external partners or enterprise systems
+3. **Universal interface**: Same API for both S3 and SFTP
+4. **Presigned URLs (S3)**: Universal compatibility without AWS SDK bundling
+5. **Direct SSH (SFTP)**: Standard protocol for B2B file exchange
+### Decision Flowchart
+```
+Do you control the infrastructure?
+         │
+    ┌────┴────┐
+   YES       NO
+    │         │
+    │         └──► Use SFTP (vendor requirement)
+    │
+Is this a cloud-native integration?
+         │
+    ┌────┴────┐
+   YES       NO
+    │         │
+    └──► S3   └──► Is volume high (>10K records/hour)?
+                         │
+                    ┌────┴────┐
+                   YES       NO
+                    │         │
+                    └──► S3   └──► Either (your preference)
+```
+---
+## Next Steps
+**Ready for S3?** → [Module 2: S3 Operations](./data-sources-02-s3-operations.md)
+**Ready for SFTP?** → [Module 3: SFTP Operations](./data-sources-03-sftp-operations.md)
+**Need both?** → Read both modules, then [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
+---
+[← Back to Guide](../data-sources-readme.md) | [Next: S3 Operations →](./data-sources-02-s3-operations.md)