npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.54 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (475) hide show

package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-03-sftp-operations.md CHANGED Viewed

@@ -1,1379 +1,1379 @@
-# Module 3: SFTP Operations
-Complete guide to SftpDataSource - SSH authentication, connection pooling, file operations, and production patterns.
-## Table of Contents
-- [SSH Architecture](#ssh-04-REFERENCE/architecture)
-- [Authentication Methods](#authentication-methods)
-- [Configuration](#configuration)
-- [File Operations](#file-operations)
-- [Connection Management](#connection-management)
-- [Multi-Format Support](#multi-format-support)
-- [Error Handling](#error-handling)
-- [Production Patterns](#production-patterns)
-- [API Reference](#api-reference)
----
-## SSH Architecture
-### How SftpDataSource Works
-`SftpDataSource` uses **direct SSH connections** via `ssh2-sftp-client` library:
-```
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ SftpDataSource (fc-connect-sdk)                           ‚
-”€€€€€€€€€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
-                   ‚
-                   ¼
-         Œ€€€€€€€€€€€€€€€€€€€€€
-         ‚ ssh2-sftp-client    ‚
-         ‚ (SFTP protocol)     ‚
-         ”€€€€€€€€¬€€€€€€€€€€€€˜
-                  ‚
-                  ¼
-    Establish SSH Connection (port 22)
-    Œ€€€€€€€€€€€€€€€€€€€€€€€€
-    ‚ SSH Handshake          ‚
-    ‚ - Protocol negotiation ‚
-    ‚ - Key exchange         ‚
-    ‚ - Authentication       ‚
-    ”€€€€€€€€¬€€€€€€€€€€€€€€€˜
-             ‚
-             ¼
-    Œ€€€€€€€€€€€€€€€€€€€€€€€€
-    ‚ Connection Pool        ‚
-    ‚ (Reuse connections)    ‚
-    ‚ Max: 5 (default)       ‚
-    ”€€€€€€€€¬€€€€€€€€€€€€€€€˜
-             ‚
-             ¼
-    Œ€€€€€€€€€€€€€€€€€€€€€€€€
-    ‚ SFTP Server            ‚
-    ‚ (Remote filesystem)    ‚
-    ”€€€€€€€€€€€€€€€€€€€€€€€€˜
-```
-**Key Benefits**:
-1. **Connection Pooling** - Reuses SSH connections for efficiency
-2. **Direct SSH** - No intermediary, full encryption
-3. **Universal Protocol** - Works with any SFTP server (RFC 4253)
-4. **Automatic Retry** - Handles connection failures with exponential backoff
-5. **No Platform Dependencies** - Works in Node.js, Deno, Versori
-### Connection Pool Behavior
-```typescript
-// Connection pool lifecycle:
-// 1. First operation creates connection
-// 2. Connection added to pool (max: 5 default)
-// 3. Subsequent operations reuse existing connections
-// 4. Idle connections auto-close after timeout
-// 5. Failed connections automatically recreated
-// Example: 10 file operations
-const files = ['file1.csv', 'file2.csv', ...]; // 10 files
-for (const file of files) {
-  // First 5 files: Create new SSH connections (pool: 0 → 5)
-  // Remaining files: Reuse pool connections (pool: stable at 5)
-  const content = await sftpSource.downloadFile(file);
-}
-```
-### Connection Pool Configuration
-Connection pooling supports configurable pool size and wait timeout.
-```typescript
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'high-throughput-sftp',
-    name: 'High Throughput SFTP',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY,
-      remotePath: '/data/inventory',
-      // Connection pool settings (NEW)
-      maxConnections: 5, // Max concurrent connections (default: 5)
-      // Note: Connection wait timeout is fixed at 30000ms (30 seconds)
-    },
-  },
-  logger
-);
-```
-### Connection Pool Wait Queue
-When the connection pool reaches maximum capacity, **new requests automatically wait in a queue** instead of failing.
-**How the Wait Queue Works:**
-```typescript
-// Scenario: High concurrency (10 simultaneous operations)
-const operations = Array.from({ length: 10 }, (_, i) => `file${i}.csv`);
-// All 10 operations start simultaneously
-const promises = operations.map(async file => {
-  // First 5 requests: Get connections immediately (pool: 0 → 5)
-  // Requests 6-10: Wait in queue until a connection is released
-  return await sftpSource.downloadFile(file);
-});
-await Promise.all(promises); // All complete successfully!
-```
-**Queue Behavior:**
-1. **Pool Full Detection**
-   - Pool has 5 active connections (default max)
-   - All connections currently in use
-   - New request arrives → added to wait queue
-2. **Automatic Waiting**
-   - Request waits up to 30 seconds (default timeout)
-   - Uses FIFO (First-In-First-Out) order
-   - No busy-waiting - efficient Promise-based waiting
-3. **Connection Released**
-   - Operation completes, connection returned to pool
-   - First waiter in queue gets the connection
-   - Process repeats until queue empty
-4. **Timeout Protection**
-   - If no connection available after 30 seconds:
-   - Throws `DataSourceError` with context:
-     - Pool size
-     - Queue length
-     - Timeout duration
-**Race Condition Protection:**
-```typescript
-// Internal tracking prevents race conditions:
-// - pendingConnections: Connections being created but not ready
-// - connectionCount: Active connections in pool
-// - Total = connectionCount + pendingConnections ≤ maxConnections
-// This prevents:
-//  Creating 7 connections when max is 5
-//  Two requests creating connections simultaneously
-// ✅ Safe concurrent connection creation
-```
-**Configuration Options:**
-```typescript
-// Connection pool configuration:
-// - maxConnections: Configurable (default: 5)
-// - connectionWaitTimeout: Fixed at 30000ms (30 seconds), not configurable
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'high-concurrency-sftp',
-    name: 'High Concurrency SFTP',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'integration_user',
-      password: process.env.SFTP_PASSWORD,
-      remotePath: '/data/inventory',
-    },
-  },
-  logger
-);
-// Pool automatically handles up to 5 concurrent operations
-// Additional operations wait in queue (up to 30s timeout)
-```
-**Wait Queue Logging:**
-```typescript
-// Enable debug logging to see queue behavior
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'debug',
-});
-const sftpSource = new SftpDataSource(config, logger);
-// When pool is full, you'll see:
-// "Connection pool full" {
-//   poolSize: 5,
-//   pendingConnections: 0,
-//   maxSize: 5,
-//   queueLength: 3  // 3 operations waiting
-// }
-// When connection is released:
-// "Connection returned to pool" {
-//   connectionId: 'vendor-sftp',
-//   poolSize: 5,
-//   totalConnections: 5,
-//   queueLength: 2  // One waiter got connection
-// }
-```
-**Error Handling:**
-```typescript
-import { Buffer } from 'node:buffer';
-import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
-try {
-  // If wait exceeds 30 seconds
-  await sftpSource.downloadFile('file.csv');
-} catch (error) {
-  if (error instanceof DataSourceError) {
-    console.error('SFTP Operation Failed:', {
-      message: error.message,
-      // "Connection pool exhausted - timeout waiting for available connection after 30000ms"
-      poolSize: error.context?.poolSize, // 5
-      maxSize: error.context?.maxSize, // 5
-      queueLength: error.context?.queueLength, // How many were waiting
-      timeoutMs: error.context?.timeoutMs, // 30000
-    });
-    // Possible solutions:
-    // 1. Reduce concurrency in your application
-    // 2. Increase operation timeout if operations are slow
-    // 3. Batch operations instead of parallel execution
-  }
-}
-```
-**Production Patterns:**
-**Pattern 1: Controlled Concurrency**
-```typescript
-// Use p-limit to control concurrency below pool limit
-import pLimit from 'p-limit';
-const limit = pLimit(3); // Lower than pool max (5)
-const files = ['file1.csv', 'file2.csv', ...]; // 20 files
-const promises = files.map(file =>
-  limit(() => sftpSource.downloadFile(file))
-);
-await Promise.all(promises);
-// Ensures max 3 concurrent operations
-// Never hits wait queue
-```
-**Pattern 2: Batch Processing**
-```typescript
-// Process in batches to avoid queue exhaustion
-async function processBatch(files: string[], batchSize = 4) {
-  for (let i = 0; i < files.length; i += batchSize) {
-    const batch = files.slice(i, i + batchSize);
-    await Promise.all(batch.map(file => sftpSource.downloadFile(file)));
-    console.log(`Batch ${i / batchSize + 1} complete`);
-  }
-}
-await processBatch(allFiles, 4); // 4 concurrent operations per batch
-```
-**Pattern 3: Graceful Degradation**
-```typescript
-// Retry with exponential backoff if pool exhausted
-async function downloadWithFallback(sftpSource: SftpDataSource, file: string, maxRetries = 3) {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      return await sftpSource.downloadFile(file);
-    } catch (error) {
-      if (error instanceof DataSourceError && error.message.includes('Connection pool exhausted')) {
-        if (attempt < maxRetries) {
-          const delay = Math.pow(2, attempt) * 1000;
-          console.log(`Pool exhausted, retry ${attempt} after ${delay}ms`);
-          await new Promise(resolve => setTimeout(resolve, delay));
-          continue;
-        }
-      }
-      throw error;
-    }
-  }
-}
-```
-**When Wait Queue Helps:**
-✅ **Good Use Cases:**
-- Burst traffic (10 operations arrive simultaneously)
-- Scheduled jobs with variable timing
-- Webhook-triggered SFTP operations
-- Prevents "connection pool exhausted" errors
-- Smooth handling of temporary overload
- **Not Suitable For:**
-- Sustained high concurrency (> 5 operations/second)
-- Long-running operations (> 30 seconds each)
-- Real-time critical operations with <1s SLA
-**Performance Impact:**
-```typescript
-// Scenario: 10 concurrent operations
-// Without wait queue: 5 succeed, 5 fail immediately
-// With wait queue: All 10 succeed (some wait)
-// Example timing:
-// Operations 1-5: Start immediately (0ms wait)
-// Operations 6-10: Wait for connection release
-//   - Op 6: Waits ~2s (when Op 1 completes)
-//   - Op 7: Waits ~3s (when Op 2 completes)
-//   - Op 8: Waits ~4s (when Op 3 completes)
-//   - Op 9: Waits ~5s (when Op 4 completes)
-//   - Op 10: Waits ~6s (when Op 5 completes)
-// Total throughput: Same (limited by pool size)
-// Reliability: Much better (no failures)
-```
----
-## Authentication Methods
-### Versori Platform - Secure Credential Management
-**✅ BEST PRACTICE: Use Versori Connection for Credential Storage**
-```typescript
-import { schedule, http } from '@versori/run';
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-export const sftpIngestion = schedule('sftp-sync', '0 * * * *')
-  .then(http('process', async (ctx) => {
-    const { log, activation } = ctx;
-    // Get SFTP credentials from Versori connection (Basic Auth)
-    const { connections } = ctx;
-    const conn = connections.versori_ftp_server;
-    const rawAccessToken = conn.accessToken;
-    const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
-    const [username, password] = rawBasicAuth.split(':');
-    // Host and port from activation variables
-    const sftpSource = new SftpDataSource({
-      type: 'SFTP_CSV',
-      connectionId: 'versori-sftp',
-      name: 'Versori SFTP',
-      settings: {
-        host: activation.getVariable('sftpHost'),
-        port: parseInt(activation.getVariable('sftpPort') || '22'),
-        username,  // From Versori connection (secure)
-        password,  // From Versori connection (secure)
-        remotePath: '/data/inventory'
-      }
-    }, log);  // Use native log from context
-    const files = await sftpSource.listFiles();
-    // Process files...
-  }));
-```
-**Benefits:**
-- ✅ Credentials stored securely in Versori vault
-- ✅ No sensitive data in activation variables
-- ✅ Connection can be reused across workflows
-- ✅ Easier credential rotation
-- ✅ Audit trail of credential usage
-**Connection Setup:**
-1. In Versori platform, create connection named `versori_ftp_server`
-2. Set Authentication Type: Basic Auth
-3. Enter SFTP username and password
-4. Reference in code: `ctx.connections.versori_ftp_server`
-**Credential Access Methods:**
-The example above shows one method (METHOD 3). Here are all three supported methods:
-```typescript
-import { Buffer } from 'node:buffer';  // Required for Deno/Versori
-// METHOD 1: activation.connections (direct access)
-const sftpConn = activation.connections.find(c => c.name === 'versori_ftp_server');
-const { username, password } = sftpConn.credentials[0].credential.data.basicAuth;
-// METHOD 2: credentials().get() with base64 decoding
-const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-const [username, password] = Buffer.from(sftpCred.accessToken, 'base64')
-  .toString('utf-8')
-  .split(':');
-// METHOD 3: ctx.connections with base64 decoding (shown above)
-const conn = ctx.connections.versori_ftp_server;
-const [username, password] = Buffer.from(conn.accessToken, 'base64')
-  .toString('utf-8')
-  .split(':');
-```
-**See:** Complete credential access guide at [`sftp-credential-access-security.md`](../data-sources-sftp-credential-access-security.md) for detailed examples, troubleshooting, and security best practices.
-### Standalone - Password Authentication
-```typescript
-import {
-  SftpDataSource,
-  createConsoleLogger,
-  toStructuredLogger,
-} from '@fluentcommerce/fc-connect-sdk';
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info',
-});
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp',
-    name: 'Vendor SFTP Connection',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'integration_user',
-      password: process.env.SFTP_PASSWORD, // Password auth
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-    },
-  },
-  logger
-);
-```
-**When to Use Password Auth**:
-- ✅ Standalone scripts (Node.js/Deno)
-- ✅ Simple setup (no key management)
-- ✅ Vendor provides password credentials
--  Less secure than SSH keys
--  Passwords can be intercepted if environment is compromised
-### SSH Private Key Authentication
-```typescript
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp-secure',
-    name: 'Secure Vendor SFTP',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'secure_integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY, // PEM format private key
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-    },
-  },
-  logger
-);
-```
-**Private Key Format (PEM)**:
-```bash
-# .env file - key content as string (NOT file path)
-SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
-MIIEpAIBAAKCAQEA1a2b3c4d5e6f7g8h9i0j...
-...
------END RSA PRIVATE KEY-----"
-```
-**When to Use SSH Key Auth**:
-- ✅ More secure (no password transmission)
-- ✅ Industry best practice for B2B integrations
-- ✅ Key rotation without password changes
-- ✅ Supports non-interactive authentication
-### Encrypted Private Key (with Passphrase)
-```typescript
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp-encrypted',
-    name: 'Encrypted Key SFTP',
-    settings: {
-      host: 'sftp.vendor.com',
-      port: 22,
-      username: 'secure_integration',
-      privateKey: process.env.SFTP_PRIVATE_KEY, // Encrypted PEM key
-      passphrase: process.env.SFTP_KEY_PASSPHRASE, // Key passphrase
-      remotePath: '/data/inventory',
-      filePattern: '*.csv',
-    },
-  },
-  logger
-);
-```
-**Environment Variables**:
-```bash
-SFTP_PRIVATE_KEY="-----BEGIN ENCRYPTED PRIVATE KEY-----
-...encrypted key content...
------END ENCRYPTED PRIVATE KEY-----"
-SFTP_KEY_PASSPHRASE=your-secure-passphrase
-```
----
-## Configuration
-### Basic Configuration
-```typescript
-import { SftpDataSource, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
-const config: SftpDataSourceConfig = {
-  type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
-  connectionId: 'vendor-sftp',
-  name: 'Vendor SFTP Connection',
-  settings: {
-    // Connection
-    host: 'sftp.vendor.com',
-    port: 22,
-    username: 'integration',
-    password: process.env.SFTP_PASSWORD, // or privateKey
-    // Paths
-    remotePath: '/data/inventory',
-    filePattern: '*.csv',
-    // Connection pool
-    connectionTimeout: 30000, // 30 seconds
-    keepaliveInterval: 5000, // 5 seconds keepalive
-    maxConnections: 5, // Pool size
-    // Write settings
-    writeCreateDirectories: true,
-    writeOverwrite: false,
-    writePermissions: '0644',
-  },
-};
-const sftpSource = new SftpDataSource(config, logger);
-```
-### Configuration Options
-| Property                          | Required | Description                           | Default   | Example             |
-| --------------------------------- | -------- | ------------------------------------- | --------- | ------------------- |
-| `type`                            | ✅       | Data source type                      | -         | `'SFTP_CSV'`        |
-| `connectionId`                    | ✅       | Unique connection ID                  | -         | `'vendor-sftp'`     |
-| `name`                            |        | Connection display name               | -         | `'Vendor SFTP'`     |
-| `settings.host`                   | ✅       | SFTP server hostname                  | -         | `'sftp.vendor.com'` |
-| `settings.port`                   |        | SSH port                              | 22        | `2222`              |
-| `settings.username`               | ✅       | SSH username                          | -         | `'integration'`     |
-| `settings.password`               | ✅\*     | Password auth                         | -         | `'secret'`          |
-| `settings.privateKey`             | ✅\*     | SSH private key (PEM)                 | -         | `'-----BEGIN...'`   |
-| `settings.passphrase`             |        | Key passphrase (if encrypted)         | -         | `'key-pass'`        |
-| `settings.remotePath`             |        | Default remote directory              | `/`       | `'/data/inventory'` |
-| `settings.filePattern`            |        | File glob pattern                     | `*`       | `'*.csv'`           |
-| `settings.maxConnections`         |        | Connection pool size       | 5         | `3`                 |
-| `settings.connectionTimeout`      |        | Connection timeout (ms)               | 30000     | `60000`             |
-| `settings.keepaliveInterval`      |        | Keepalive interval (ms)               | 5000      | `10000`             |
-| `settings.retry`                  |        | Retry configuration object | See below | See below           |
-| `settings.writeCreateDirectories` |        | Auto-create dirs on write             | `true`    | `false`             |
-| `settings.writeOverwrite`         |        | Overwrite existing files              | `false`   | `true`              |
-| `settings.writePermissions`       |        | File permissions (octal)              | `'0644'`  | `'0600'`            |
-\* Either `password` or `privateKey` required
-### Enhanced Retry Configuration
-Fine-grained retry control with exponential backoff, jitter, and custom predicates.
-```typescript
-settings: {
-  host: 'sftp.vendor.com',
-  username: 'integration',
-  privateKey: process.env.SFTP_PRIVATE_KEY,
-  // Enhanced retry configuration (optional)
-  retry: {
-    maxAttempts: 3,           // Max retry attempts (default: 3)
-    baseDelayMs: 1000,         // Initial delay in ms (default: 1000)
-    backoffFactor: 2,          // Exponential factor (default: 2)
-    maxDelayMs: 8000,          // Max delay cap (default: 8000)
-    jitter: 'full',            // Jitter strategy: 'none' | 'full' (default: 'full')
-    reconnectOnFailure: true,  // Reconnect between retries (default: true)
-    // Optional: Custom retry predicate
-    isRetryable: (error: Error) => {
-      // Return true to retry, false to fail immediately
-      return error.message.includes('ECONNRESET');
-    }
-  }
-}
-```
-**Retry Configuration Options:**
-| Option               | Type       | Default  | Description                                             |
-| -------------------- | ---------- | -------- | ------------------------------------------------------- |
-| `maxAttempts`        | `number`   | `3`      | Maximum retry attempts (including initial attempt)      |
-| `baseDelayMs`        | `number`   | `1000`   | Base delay in milliseconds                              |
-| `backoffFactor`      | `number`   | `2`      | Exponential backoff multiplier                          |
-| `maxDelayMs`         | `number`   | `8000`   | Maximum delay cap in milliseconds                       |
-| `jitter`             | `string`   | `'full'` | Jitter strategy: `'none'` or `'full'` (randomization)   |
-| `reconnectOnFailure` | `boolean`  | `true`   | Reconnect to SFTP between retry attempts                |
-| `isRetryable`        | `function` | Built-in | Custom function to determine if error should be retried |
-**Default Retry Behavior:**
-The SDK automatically retries these error types:
-- **Network errors**: `ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `EHOSTUNREACH`, `ECONNREFUSED`, `EPIPE`
-- **Transient messages**: `connection lost`, `timeout`, `socket hang up`, `network error`
-The SDK **never retries** these errors (require manual intervention):
-- **Authentication failures**: `permission denied`, `authentication`
-- **File not found**: `no such file`, `file not found`
-- **Configuration errors**: `algorithm`, `cipher`, `key exchange`
-### Environment Variables Pattern
-```bash
-# .env file
-SFTP_HOST=sftp.vendor.com
-SFTP_PORT=22
-SFTP_USERNAME=integration_user
-SFTP_PASSWORD=secret-password
-# Or SSH key (recommended)
-SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
-...key content...
------END RSA PRIVATE KEY-----"
-SFTP_PASSPHRASE=optional-passphrase
-# Paths
-SFTP_REMOTE_PATH=/data/inventory
-SFTP_FILE_PATTERN=*.csv
-```
-```typescript
-// Load from environment
-const sftpSource = new SftpDataSource(
-  {
-    type: 'SFTP_CSV',
-    connectionId: 'vendor-sftp',
-    settings: {
-      host: process.env.SFTP_HOST!,
-      port: parseInt(process.env.SFTP_PORT || '22'),
-      username: process.env.SFTP_USERNAME!,
-      password: process.env.SFTP_PASSWORD, // or
-      privateKey: process.env.SFTP_PRIVATE_KEY, // privateKey
-      passphrase: process.env.SFTP_PASSPHRASE,
-      remotePath: process.env.SFTP_REMOTE_PATH || '/',
-      filePattern: process.env.SFTP_FILE_PATTERN || '*',
-      maxConnections: parseInt(process.env.SFTP_MAX_CONN || '5'),
-    },
-  },
-  logger
-);
-```
----
-## File Operations
-### Listing Files
-```typescript
-// List all files in default remotePath
-const allFiles = await sftpSource.listFiles();
-// List with filters
-const recentFiles = await sftpSource.listFiles({
-  remotePath: '/data/current',
-  filePattern: '*.csv',
-  lastProcessedTimestamp: '2025-01-01T00:00:00Z', // Only files modified after
-});
-// List by file name substring
-const inventoryFiles = await sftpSource.listFiles({
-  fileName: 'inventory', // Contains 'inventory'
-});
-// Process file metadata
-for (const file of recentFiles) {
-  console.log({
-    name: file.name, // 'inventory-2025-01-15.csv'
-    path: file.path, // '/data/current/inventory-2025-01-15.csv'
-    size: file.size, // 1024000 (bytes)
-    lastModified: file.lastModified, // Date object
-    type: file.type, // 'file' or 'directory'
-    permissions: file.permissions, // '-rw-r--r--'
-  });
-}
-```
-**List Options**:
-| Option                   | Type     | Description                                 |
-| ------------------------ | -------- | ------------------------------------------- |
-| `remotePath`             | `string` | Remote directory to list (overrides config) |
-| `filePattern`            | `string` | Glob pattern filter (e.g., `'*.csv'`)       |
-| `fileName`               | `string` | Substring match filter                      |
-| `lastProcessedTimestamp` | `string` | ISO timestamp - only files modified after   |
-### Reading Files
-```typescript
-// Read text file (CSV, JSON, etc.)
-const csvContent = await sftpSource.downloadFile('/data/inventory.csv');
-// Returns: string
-// Read binary file (Parquet, images, etc.)
-const parquetData = await sftpSource.downloadFile('data.parquet');
-// Returns: Buffer
-// Read with full path
-const content = await sftpSource.downloadFile('/remote/path/to/file.csv');
-// Read relative to remotePath
-const content2 = await sftpSource.downloadFile('subdir/file.csv');
-// Resolves to: /data/inventory/subdir/file.csv (if remotePath=/data/inventory)
-```
-### Writing Files
-```typescript
-// Upload file content (string)
-await sftpSource.uploadFile('/exports/inventory.csv', csvData, {
-  createDirectories: true,
-  overwrite: false,
-  permissions: '0644',
-});
-// Upload binary content (Buffer)
-await sftpSource.uploadFile('/exports/data.parquet', parquetBuffer, {
-  createDirectories: true,
-  permissions: '0600', // Read/write owner only
-});
-// Upload with auto-create directories
-await sftpSource.uploadFile(
-  '/exports/2025/01/15/inventory.csv', // Creates /exports/2025/01/15/ if missing
-  csvData,
-  { createDirectories: true }
-);
-```
-**Write Options**:
-| Option              | Type                 | Description                | Default  |
-| ------------------- | -------------------- | -------------------------- | -------- |
-| `createDirectories` | `boolean`            | Create missing parent dirs | `true`   |
-| `overwrite`         | `boolean`            | Overwrite existing file    | `false`  |
-| `permissions`       | `string`             | File permissions (octal)   | `'0644'` |
-| `encoding`          | `'utf8' \| 'binary'` | Content encoding           | `'utf8'` |
-### Deleting Files
-```typescript
-// Delete single file
-await sftpSource.deleteFile('/temp/old-file.csv');
-// Delete with error handling
-try {
-  await sftpSource.deleteFile('/important-file.csv');
-  console.log('File deleted successfully');
-} catch (error: any) {
-  if (error.message.includes('No such file')) {
-    console.log('File already deleted or does not exist');
-  } else {
-    console.error('Delete failed:', error.message);
-  }
-}
-```
-### Moving Files (Atomic Rename)
-```typescript
-// Move file (atomic SFTP rename operation)
-await sftpSource.moveFile(
-  '/incoming/data.csv',
-  '/processed/data.csv',
-  false // Don't overwrite if target exists
-);
-// Move to archive with date-based organization
-const today = new Date().toISOString().split('T')[0];
-await sftpSource.moveFile('/incoming/inventory.csv', `/archive/${today}/inventory.csv`);
-```
-**IMPORTANT**: Unlike S3's copy+delete, SFTP `moveFile()` uses the **native SFTP rename command**, which is **atomic**. The file is renamed in a single operation with no risk of existing in both locations.
-### Copying Files
-```typescript
-// Copy file (download + upload)
-await sftpSource.copyFile(
-  '/source/inventory.csv',
-  '/backup/inventory_backup.csv',
-  true // Overwrite if exists
-);
-// Copy with different permissions
-await sftpSource.copyFile(
-  '/source/data.csv',
-  '/archive/data.csv',
-  false,
-  '0600' // Restrictive permissions for archive
-);
-```
-**Note**: `copyFile()` is implemented as download + upload (not atomic). For atomic operations, use `moveFile()`.
-### Directory Operations
-```typescript
-// Create directory
-await sftpSource.createDirectory('/exports/daily', true, '0755');
-// Create nested directories
-await sftpSource.createDirectory('/exports/2025/01/15', true);
-// Creates all missing parents: /exports → /exports/2025 → /exports/2025/01 → /exports/2025/01/15
-// Check if directory exists
-const dirExists = await sftpSource.directoryExists('/data/exports');
-console.log(`Directory exists: ${dirExists}`);
-// Check if file exists
-const fileExists = await sftpSource.fileExists('/data/inventory.csv');
-console.log(`File exists: ${fileExists}`);
-```
----
-## Connection Management
-### Connection Pool Information
-```typescript
-// Get connection details
-const connectionInfo = sftpSource.getConnectionInfo();
-console.log('Connection Info:', {
-  type: connectionInfo.type, // 'SFTP_CSV'
-  host: connectionInfo.host, // 'sftp.vendor.com'
-  port: connectionInfo.port, // 22
-  username: connectionInfo.username, // 'integration'
-  remotePath: connectionInfo.remotePath, // '/data/inventory'
-  filePattern: connectionInfo.filePattern, // '*.csv'
-  currentConnections: connectionInfo.connectionPoolSize, // 3
-  maxConnections: connectionInfo.maxConnections, // 5
-});
-```
-### Connection Health
-The SFTP data source automatically manages connection health:
-```
-Connection Lifecycle:
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ 1. Connection Created           ‚
-‚    - SSH handshake              ‚
-‚    - Authentication             ‚
-‚    - Add to pool                ‚
-”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
-           ‚
-           ¼
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ 2. Connection Active            ‚
-‚    - Reused for operations      ‚
-‚    - Keepalive pings (5s)       ‚
-‚    - Health monitoring          ‚
-”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
-           ‚
-           ¼
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ 3. Connection Idle              ‚
-‚    - No operations for timeout  ‚
-‚    - Auto-close after TTL       ‚
-‚    - Removed from pool          ‚
-”€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
-Failure Handling:
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ Operation Fails                 ‚
-‚ - Network error                 ‚
-‚ - Connection timeout            ‚
-‚ - Authentication failure        ‚
-”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
-           ‚
-           ¼
-Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
-‚ Automatic Retry                 ‚
-‚ - Exponential backoff           ‚
-‚ - New connection created        ‚
-‚ - Operation retried             ‚
-”€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
-```
-### Validating Connection
-```typescript
-// Test SFTP connectivity
-const isValid = await sftpSource.validateConnection();
-if (isValid) {
-  console.log('SFTP connection successful');
-} else {
-  console.error('Failed to connect to SFTP server');
-}
-// Use in startup health check
-async function startupHealthCheck() {
-  try {
-    const valid = await sftpSource.validateConnection();
-    if (!valid) {
-      throw new Error('SFTP connection validation failed');
-    }
-    console.log('❌œ“ SFTP connection validated');
-  } catch (error) {
-    console.error('❌œ— SFTP connection failed:', error);
-    process.exit(1);
-  }
-}
-```
----
-## Multi-Format Support
-### CSV Format
-```typescript
-// CSV-specific configuration
-const csvConfig: SftpDataSourceConfig = {
-  type: 'SFTP_CSV',
-  connectionId: 'vendor-csv',
-  settings: {
-    host: 'sftp.vendor.com',
-    username: 'integration',
-    privateKey: process.env.SFTP_KEY,
-    // CSV parsing options
-    csvDelimiter: ',',
-    csvHeaders: ['sku', 'quantity', 'location', 'status'],
-    requiredCsvHeaders: ['sku', 'quantity'], // Validation
-    csvSkipEmptyLines: true,
-    csvTrimValues: true,
-    csvQuote: '"',
-    csvEscape: '\\',
-  },
-};
-// Parse CSV content
-const csvContent = await sftpSource.downloadFile('inventory.csv');
-const records = sftpSource.parseCsvContent(csvContent, fileMetadata);
-// Uses csvDelimiter, csvSkipEmptyLines, etc. from config
-// Generate CSV content
-const csvOutput = sftpSource.writeCsvContent(records, {
-  delimiter: ',',
-  headers: ['sku', 'qty', 'location'],
-  includeHeader: true,
-});
-await sftpSource.writeFile('/exports/inventory.csv', csvOutput);
-```
-### JSON Format
-```typescript
-// JSON-specific configuration
-const jsonConfig: SftpDataSourceConfig = {
-  type: 'SFTP_JSON',
-  connectionId: 'vendor-json',
-  settings: {
-    host: 'sftp.vendor.com',
-    username: 'integration',
-    privateKey: process.env.SFTP_KEY,
-    // JSON options
-    jsonFormat: 'json', // or 'jsonl'
-    jsonValidate: true,
-    jsonMaxDepth: 10,
-    jsonIndent: 2,
-    jsonSortKeys: true,
-  },
-};
-// Parse JSON content
-const jsonContent = await sftpSource.downloadFile('data.json');
-const records = sftpSource.parseJsonContent(jsonContent, fileMetadata, {
-  format: 'json',
-  validate: true,
-  maxDepth: 10,
-});
-// Generate JSON content
-const jsonOutput = sftpSource.writeJsonContent(records, {
-  format: 'json',
-  indent: 2,
-  sortKeys: true,
-});
-await sftpSource.writeFile('/exports/data.json', jsonOutput);
-```
-### JSON Lines (JSONL) Format
-```typescript
-// Parse JSONL (one JSON object per line)
-const jsonlContent = await sftpSource.downloadFile('stream-data.jsonl');
-const records = sftpSource.parseJsonContent(jsonlContent, fileMetadata, {
-  format: 'jsonl',
-});
-// Generate JSONL content
-const jsonlOutput = sftpSource.writeJsonContent(records, {
-  format: 'jsonl',
-});
-// Output:
-// {"id":1,"name":"Product A"}
-// {"id":2,"name":"Product B"}
-await sftpSource.writeFile('/exports/stream.jsonl', jsonlOutput);
-```
-### Parquet Format
-```typescript
-// Parquet-specific configuration
-const parquetConfig: SftpDataSourceConfig = {
-  type: 'SFTP_PARQUET',
-  connectionId: 'vendor-parquet',
-  settings: {
-    host: 'sftp.vendor.com',
-    username: 'integration',
-    privateKey: process.env.SFTP_KEY,
-    // Parquet options
-    parquetCompression: 'snappy', // 'none', 'gzip', 'snappy', 'lz4'
-    parquetSchema: {
-      productRef: 'string',
-      quantity: 'int32',
-      price: 'double',
-    },
-    parquetRowGroupSize: 1000,
-  },
-};
-// Parse Parquet content (async)
-const parquetBuffer = await sftpSource.downloadFile('data.parquet');
-// Note: parseParquetContent currently throws (not implemented). Use ParquetParserService.
-// const records = await sftpSource.parseParquetContent(parquetBuffer as Buffer, fileMetadata);
-// Parquet generation would use ParquetParserService separately
-```
-### Processing Mixed File Types
-```typescript
-async function processMultipleFormats() {
-  const files = await sftpSource.listFiles({
-    filePattern: '*.{csv,json,parquet}',
-  });
-  for (const file of files) {
-    const extension = file.name.split('.').pop()?.toLowerCase();
-    let records: any[] = [];
-    switch (extension) {
-      case 'csv':
-        const csvContent = await sftpSource.downloadFile(file.path);
-        records = sftpSource.parseCsvContent(csvContent, file);
-        break;
-      case 'json':
-        const jsonContent = await sftpSource.downloadFile(file.path);
-        records = sftpSource.parseJsonContent(jsonContent, file, {
-          format: 'json',
-        });
-        break;
-      case 'parquet':
-        const parquetBuffer = await sftpSource.downloadFile(file.path);
-        // Use ParquetParserService here instead of parseParquetContent (not implemented)
-        // records = await sftpParser.parseParquetContent(parquetBuffer as Buffer, file);
-        break;
-      default:
-        logger.warn(`Unsupported format: ${extension}`);
-        continue;
-    }
-    await processRecords(records, file.name);
-  }
-}
-```
----
-## Error Handling
-### Common Error Patterns
-```typescript
-import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
-try {
-  await sftpSource.writeFile(remotePath, content);
-} catch (error) {
-  if (error instanceof DataSourceError) {
-    switch (error.operation) {
-      case 'uploadFile':
-        if (error.message.includes('Permission denied')) {
-          logger.error('SFTP permission error', {
-            remotePath,
-            suggestion: 'Check file permissions and directory access',
-          });
-        }
-        break;
-      case 'listFiles':
-        if (error.message.includes('No such file')) {
-          logger.warn('Remote directory not found', {
-            remotePath: error.context?.remotePath,
-            suggestion: 'Verify remote path exists',
-          });
-        }
-        break;
-      case 'downloadFile':
-        if (error.message.includes('Connection timeout')) {
-          logger.error('SFTP connection timeout', {
-            suggestion: 'Check network connectivity and server status',
-          });
-        }
-        break;
-    }
-  } else {
-    logger.error('Unexpected error', error);
-  }
-}
-```
-### Retry with Exponential Backoff
-```typescript
-async function uploadWithRetry(
-  content: string,
-  remotePath: string,
-  maxRetries: number = 3
-): Promise<void> {
-  let lastError: Error;
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      await sftpSource.writeFile(remotePath, content);
-      logger.info(`Upload successful on attempt ${attempt}`);
-      return;
-    } catch (error) {
-      lastError = error as Error;
-      logger.warn(`Upload attempt ${attempt} failed`, {
-        error: (error as Error).message,
-        retriesLeft: maxRetries - attempt,
-      });
-      if (attempt < maxRetries) {
-        const delay = Math.pow(2, attempt) * 1000; // 2s, 4s, 8s
-        logger.info(`Retrying in ${delay}ms...`);
-        await new Promise(resolve => setTimeout(resolve, delay));
-      }
-    }
-  }
-  throw new Error(`Upload failed after ${maxRetries} attempts: ${lastError!.message}`);
-}
-```
----
-## Production Patterns
-### Pattern 1: Vendor Inventory Ingestion
-```typescript
-import {
-  SftpDataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-} from '@fluentcommerce/fc-connect-sdk';
-async function ingestVendorInventory() {
-  const sftpSource = new SftpDataSource(
-    {
-      /* config */
-    },
-    logger
-  );
-  const csvParser = new CSVParserService();
-  const mapper = new UniversalMapper(mappingConfig);
-  const stateService = new StateService(kvAdapter);
-  // List new files
-  const files = await sftpSource.listFiles({
-    remotePath: '/outbound/inventory',
-    filePattern: 'inventory_*.csv',
-  });
-  for (const file of files) {
-    // Check if already processed
-    const stateKey = `processed:${file.name}:${file.lastModified?.toString()}`;
-    const processed = await stateService.isFileProcessed(kvAdapter as any, stateKey);
-    if (processed) {
-      logger.info('File already processed', { file: file.name });
-      continue;
-    }
-    try {
-      // Download and parse CSV
-      const content = await sftpSource.downloadFile(file.path);
-      const records = await csvParser.parse(content as string);
-      // Map fields
-      const result = await mapper.map(records);
-      if (!result.success) {
-        logger.error('Mapping failed', undefined, { errors: result.errors });
-        continue;
-      }
-      // Send to Fluent
-      await sendToFluent(result.data);
-      // Mark as processed
-      await stateService.updateSyncState(
-        kvAdapter as any,
-        [
-          {
-            fileName: file.name,
-            recordCount: records.length,
-            processedAt: new Date().toISOString(),
-          },
-        ],
-        'sftp-ingestion'
-      );
-      // Move to processed folder
-      await sftpSource.moveFile(file.path, file.path.replace('/outbound/', '/processed/'));
-      logger.info('File processed successfully', { file: file.name });
-    } catch (error) {
-      logger.error('Failed to process file', error as Error, { file: file.path });
-    }
-  }
-}
-```
-### Pattern 2: State Management with SFTP
-```typescript
-async function processFilesWithState() {
-  const files = await sftpSource.listFiles();
-  const stateService = new StateService(logger);
-  for (const file of files) {
-    // Use file name + modification time as unique key
-    const stateKey = `processed:${file.name}:${file.lastModified.toISOString()}`;
-    if (await stateService.isFileProcessed(kvAdapter as any, stateKey)) {
-      logger.info('File already processed', { fileName: file.name });
-      continue;
-    }
-    try {
-      const content = await sftpSource.downloadFile(file.path);
-      await processFile(content, file.name);
-      // Mark as processed
-      await stateService.updateSyncState(
-        kvAdapter as any,
-        [
-          {
-            fileName: file.name,
-            size: file.size,
-            processedAt: new Date().toISOString(),
-          },
-        ],
-        'sftp-processing'
-      );
-    } catch (error) {
-      logger.error('Failed to process file', error as Error, { file: file.path });
-    }
-  }
-}
-```
----
-## API Reference
-### SftpDataSource Constructor
-```typescript
-new SftpDataSource(config: SftpDataSourceConfig, logger?: StructuredLogger)
-```
-**Parameters**:
-- `config`: SFTP data source configuration
-- `logger`: Optional logging service
-### Methods
-| Method                  | Parameters                                         | Returns                   | Description                   |
-| ----------------------- | -------------------------------------------------- | ------------------------- | ----------------------------- | --------------------- |
-| `listFiles()`           | `options?`                                         | `Promise<FileMetadata[]>` | List files in remote path     |
-| `downloadFile()`        | `path`                                             | `Promise<string           | Buffer>`                      | Download file content |
-| `downloadFile()`        | `path`                                             | `Promise<Buffer>`         | Download file as buffer       |
-| `writeFile()`           | `path, content, options?`                          | `Promise<void>`           | Write file to SFTP server     |
-| `deleteFile()`          | `path`                                             | `Promise<void>`           | Delete file from server       |
-| `copyFile()`            | `sourcePath, targetPath, overwrite?, permissions?` | `Promise<void>`           | Copy file (download + upload) |
-| `moveFile()`            | `sourcePath, targetPath, overwrite?`               | `Promise<void>`           | Move file (atomic rename)     |
-| `fileExists()`          | `path`                                             | `Promise<boolean>`        | Check if file exists          |
-| `directoryExists()`     | `path`                                             | `Promise<boolean>`        | Check if directory exists     |
-| `createDirectory()`     | `path, recursive?, permissions?`                   | `Promise<void>`           | Create directory              |
-| `validateConnection()`  | -                                                  | `Promise<boolean>`        | Test SFTP connectivity        |
-| `getConnectionInfo()`   | -                                                  | `ConnectionInfo`          | Get connection details        |
-| `parseCsvContent()`     | `content, metadata`                                | `any[]`                   | Parse CSV content             |
-| `parseJsonContent()`    | `content, metadata, options?`                      | `any[]`                   | Parse JSON/JSONL content      |
-| `parseParquetContent()` | `buffer, metadata`                                 | `Promise<any[]>`          | Not implemented (throws)      |
-| `writeCsvContent()`     | `records, options?`                                | `string`                  | Generate CSV content          |
-| `writeJsonContent()`    | `records, options?`                                | `string`                  | Generate JSON/JSONL content   |
----
-## Next Steps
-**Learn file processing patterns** → [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
-**Explore advanced topics** → [Module 5: Advanced Topics](./data-sources-05-advanced-topics.md)
----
-[→ Back to S3 Operations](./data-sources-02-s3-operations.md) | [Next: File Processing Patterns →](./data-sources-04-file-patterns.md) | [→‘ Back to Guide](../data-sources-readme.md)
+# Module 3: SFTP Operations
+Complete guide to SftpDataSource - SSH authentication, connection pooling, file operations, and production patterns.
+## Table of Contents
+- [SSH Architecture](#ssh-04-REFERENCE/architecture)
+- [Authentication Methods](#authentication-methods)
+- [Configuration](#configuration)
+- [File Operations](#file-operations)
+- [Connection Management](#connection-management)
+- [Multi-Format Support](#multi-format-support)
+- [Error Handling](#error-handling)
+- [Production Patterns](#production-patterns)
+- [API Reference](#api-reference)
+---
+## SSH Architecture
+### How SftpDataSource Works
+`SftpDataSource` uses **direct SSH connections** via `ssh2-sftp-client` library:
+```
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ SftpDataSource (fc-connect-sdk)                           ‚
+”€€€€€€€€€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
+                   ‚
+                   ¼
+         Œ€€€€€€€€€€€€€€€€€€€€€
+         ‚ ssh2-sftp-client    ‚
+         ‚ (SFTP protocol)     ‚
+         ”€€€€€€€€¬€€€€€€€€€€€€˜
+                  ‚
+                  ¼
+    Establish SSH Connection (port 22)
+    Œ€€€€€€€€€€€€€€€€€€€€€€€€
+    ‚ SSH Handshake          ‚
+    ‚ - Protocol negotiation ‚
+    ‚ - Key exchange         ‚
+    ‚ - Authentication       ‚
+    ”€€€€€€€€¬€€€€€€€€€€€€€€€˜
+             ‚
+             ¼
+    Œ€€€€€€€€€€€€€€€€€€€€€€€€
+    ‚ Connection Pool        ‚
+    ‚ (Reuse connections)    ‚
+    ‚ Max: 5 (default)       ‚
+    ”€€€€€€€€¬€€€€€€€€€€€€€€€˜
+             ‚
+             ¼
+    Œ€€€€€€€€€€€€€€€€€€€€€€€€
+    ‚ SFTP Server            ‚
+    ‚ (Remote filesystem)    ‚
+    ”€€€€€€€€€€€€€€€€€€€€€€€€˜
+```
+**Key Benefits**:
+1. **Connection Pooling** - Reuses SSH connections for efficiency
+2. **Direct SSH** - No intermediary, full encryption
+3. **Universal Protocol** - Works with any SFTP server (RFC 4253)
+4. **Automatic Retry** - Handles connection failures with exponential backoff
+5. **No Platform Dependencies** - Works in Node.js, Deno, Versori
+### Connection Pool Behavior
+```typescript
+// Connection pool lifecycle:
+// 1. First operation creates connection
+// 2. Connection added to pool (max: 5 default)
+// 3. Subsequent operations reuse existing connections
+// 4. Idle connections auto-close after timeout
+// 5. Failed connections automatically recreated
+// Example: 10 file operations
+const files = ['file1.csv', 'file2.csv', ...]; // 10 files
+for (const file of files) {
+  // First 5 files: Create new SSH connections (pool: 0 → 5)
+  // Remaining files: Reuse pool connections (pool: stable at 5)
+  const content = await sftpSource.downloadFile(file);
+}
+```
+### Connection Pool Configuration
+Connection pooling supports configurable pool size and wait timeout.
+```typescript
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'high-throughput-sftp',
+    name: 'High Throughput SFTP',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY,
+      remotePath: '/data/inventory',
+      // Connection pool settings (NEW)
+      maxConnections: 5, // Max concurrent connections (default: 5)
+      // Note: Connection wait timeout is fixed at 30000ms (30 seconds)
+    },
+  },
+  logger
+);
+```
+### Connection Pool Wait Queue
+When the connection pool reaches maximum capacity, **new requests automatically wait in a queue** instead of failing.
+**How the Wait Queue Works:**
+```typescript
+// Scenario: High concurrency (10 simultaneous operations)
+const operations = Array.from({ length: 10 }, (_, i) => `file${i}.csv`);
+// All 10 operations start simultaneously
+const promises = operations.map(async file => {
+  // First 5 requests: Get connections immediately (pool: 0 → 5)
+  // Requests 6-10: Wait in queue until a connection is released
+  return await sftpSource.downloadFile(file);
+});
+await Promise.all(promises); // All complete successfully!
+```
+**Queue Behavior:**
+1. **Pool Full Detection**
+   - Pool has 5 active connections (default max)
+   - All connections currently in use
+   - New request arrives → added to wait queue
+2. **Automatic Waiting**
+   - Request waits up to 30 seconds (default timeout)
+   - Uses FIFO (First-In-First-Out) order
+   - No busy-waiting - efficient Promise-based waiting
+3. **Connection Released**
+   - Operation completes, connection returned to pool
+   - First waiter in queue gets the connection
+   - Process repeats until queue empty
+4. **Timeout Protection**
+   - If no connection available after 30 seconds:
+   - Throws `DataSourceError` with context:
+     - Pool size
+     - Queue length
+     - Timeout duration
+**Race Condition Protection:**
+```typescript
+// Internal tracking prevents race conditions:
+// - pendingConnections: Connections being created but not ready
+// - connectionCount: Active connections in pool
+// - Total = connectionCount + pendingConnections ≤ maxConnections
+// This prevents:
+//  Creating 7 connections when max is 5
+//  Two requests creating connections simultaneously
+// ✅ Safe concurrent connection creation
+```
+**Configuration Options:**
+```typescript
+// Connection pool configuration:
+// - maxConnections: Configurable (default: 5)
+// - connectionWaitTimeout: Fixed at 30000ms (30 seconds), not configurable
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'high-concurrency-sftp',
+    name: 'High Concurrency SFTP',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'integration_user',
+      password: process.env.SFTP_PASSWORD,
+      remotePath: '/data/inventory',
+    },
+  },
+  logger
+);
+// Pool automatically handles up to 5 concurrent operations
+// Additional operations wait in queue (up to 30s timeout)
+```
+**Wait Queue Logging:**
+```typescript
+// Enable debug logging to see queue behavior
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'debug',
+});
+const sftpSource = new SftpDataSource(config, logger);
+// When pool is full, you'll see:
+// "Connection pool full" {
+//   poolSize: 5,
+//   pendingConnections: 0,
+//   maxSize: 5,
+//   queueLength: 3  // 3 operations waiting
+// }
+// When connection is released:
+// "Connection returned to pool" {
+//   connectionId: 'vendor-sftp',
+//   poolSize: 5,
+//   totalConnections: 5,
+//   queueLength: 2  // One waiter got connection
+// }
+```
+**Error Handling:**
+```typescript
+import { Buffer } from 'node:buffer';
+import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
+try {
+  // If wait exceeds 30 seconds
+  await sftpSource.downloadFile('file.csv');
+} catch (error) {
+  if (error instanceof DataSourceError) {
+    console.error('SFTP Operation Failed:', {
+      message: error.message,
+      // "Connection pool exhausted - timeout waiting for available connection after 30000ms"
+      poolSize: error.context?.poolSize, // 5
+      maxSize: error.context?.maxSize, // 5
+      queueLength: error.context?.queueLength, // How many were waiting
+      timeoutMs: error.context?.timeoutMs, // 30000
+    });
+    // Possible solutions:
+    // 1. Reduce concurrency in your application
+    // 2. Increase operation timeout if operations are slow
+    // 3. Batch operations instead of parallel execution
+  }
+}
+```
+**Production Patterns:**
+**Pattern 1: Controlled Concurrency**
+```typescript
+// Use p-limit to control concurrency below pool limit
+import pLimit from 'p-limit';
+const limit = pLimit(3); // Lower than pool max (5)
+const files = ['file1.csv', 'file2.csv', ...]; // 20 files
+const promises = files.map(file =>
+  limit(() => sftpSource.downloadFile(file))
+);
+await Promise.all(promises);
+// Ensures max 3 concurrent operations
+// Never hits wait queue
+```
+**Pattern 2: Batch Processing**
+```typescript
+// Process in batches to avoid queue exhaustion
+async function processBatch(files: string[], batchSize = 4) {
+  for (let i = 0; i < files.length; i += batchSize) {
+    const batch = files.slice(i, i + batchSize);
+    await Promise.all(batch.map(file => sftpSource.downloadFile(file)));
+    console.log(`Batch ${i / batchSize + 1} complete`);
+  }
+}
+await processBatch(allFiles, 4); // 4 concurrent operations per batch
+```
+**Pattern 3: Graceful Degradation**
+```typescript
+// Retry with exponential backoff if pool exhausted
+async function downloadWithFallback(sftpSource: SftpDataSource, file: string, maxRetries = 3) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      return await sftpSource.downloadFile(file);
+    } catch (error) {
+      if (error instanceof DataSourceError && error.message.includes('Connection pool exhausted')) {
+        if (attempt < maxRetries) {
+          const delay = Math.pow(2, attempt) * 1000;
+          console.log(`Pool exhausted, retry ${attempt} after ${delay}ms`);
+          await new Promise(resolve => setTimeout(resolve, delay));
+          continue;
+        }
+      }
+      throw error;
+    }
+  }
+}
+```
+**When Wait Queue Helps:**
+✅ **Good Use Cases:**
+- Burst traffic (10 operations arrive simultaneously)
+- Scheduled jobs with variable timing
+- Webhook-triggered SFTP operations
+- Prevents "connection pool exhausted" errors
+- Smooth handling of temporary overload
+ **Not Suitable For:**
+- Sustained high concurrency (> 5 operations/second)
+- Long-running operations (> 30 seconds each)
+- Real-time critical operations with <1s SLA
+**Performance Impact:**
+```typescript
+// Scenario: 10 concurrent operations
+// Without wait queue: 5 succeed, 5 fail immediately
+// With wait queue: All 10 succeed (some wait)
+// Example timing:
+// Operations 1-5: Start immediately (0ms wait)
+// Operations 6-10: Wait for connection release
+//   - Op 6: Waits ~2s (when Op 1 completes)
+//   - Op 7: Waits ~3s (when Op 2 completes)
+//   - Op 8: Waits ~4s (when Op 3 completes)
+//   - Op 9: Waits ~5s (when Op 4 completes)
+//   - Op 10: Waits ~6s (when Op 5 completes)
+// Total throughput: Same (limited by pool size)
+// Reliability: Much better (no failures)
+```
+---
+## Authentication Methods
+### Versori Platform - Secure Credential Management
+**✅ BEST PRACTICE: Use Versori Connection for Credential Storage**
+```typescript
+import { schedule, http } from '@versori/run';
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+export const sftpIngestion = schedule('sftp-sync', '0 * * * *')
+  .then(http('process', async (ctx) => {
+    const { log, activation } = ctx;
+    // Get SFTP credentials from Versori connection (Basic Auth)
+    const { connections } = ctx;
+    const conn = connections.versori_ftp_server;
+    const rawAccessToken = conn.accessToken;
+    const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
+    const [username, password] = rawBasicAuth.split(':');
+    // Host and port from activation variables
+    const sftpSource = new SftpDataSource({
+      type: 'SFTP_CSV',
+      connectionId: 'versori-sftp',
+      name: 'Versori SFTP',
+      settings: {
+        host: activation.getVariable('sftpHost'),
+        port: parseInt(activation.getVariable('sftpPort') || '22'),
+        username,  // From Versori connection (secure)
+        password,  // From Versori connection (secure)
+        remotePath: '/data/inventory'
+      }
+    }, log);  // Use native log from context
+    const files = await sftpSource.listFiles();
+    // Process files...
+  }));
+```
+**Benefits:**
+- ✅ Credentials stored securely in Versori vault
+- ✅ No sensitive data in activation variables
+- ✅ Connection can be reused across workflows
+- ✅ Easier credential rotation
+- ✅ Audit trail of credential usage
+**Connection Setup:**
+1. In Versori platform, create connection named `versori_ftp_server`
+2. Set Authentication Type: Basic Auth
+3. Enter SFTP username and password
+4. Reference in code: `ctx.connections.versori_ftp_server`
+**Credential Access Methods:**
+The example above shows one method (METHOD 3). Here are all three supported methods:
+```typescript
+import { Buffer } from 'node:buffer';  // Required for Deno/Versori
+// METHOD 1: activation.connections (direct access)
+const sftpConn = activation.connections.find(c => c.name === 'versori_ftp_server');
+const { username, password } = sftpConn.credentials[0].credential.data.basicAuth;
+// METHOD 2: credentials().get() with base64 decoding
+const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+const [username, password] = Buffer.from(sftpCred.accessToken, 'base64')
+  .toString('utf-8')
+  .split(':');
+// METHOD 3: ctx.connections with base64 decoding (shown above)
+const conn = ctx.connections.versori_ftp_server;
+const [username, password] = Buffer.from(conn.accessToken, 'base64')
+  .toString('utf-8')
+  .split(':');
+```
+**See:** Complete credential access guide at [`sftp-credential-access-security.md`](../data-sources-sftp-credential-access-security.md) for detailed examples, troubleshooting, and security best practices.
+### Standalone - Password Authentication
+```typescript
+import {
+  SftpDataSource,
+  createConsoleLogger,
+  toStructuredLogger,
+} from '@fluentcommerce/fc-connect-sdk';
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info',
+});
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp',
+    name: 'Vendor SFTP Connection',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'integration_user',
+      password: process.env.SFTP_PASSWORD, // Password auth
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+    },
+  },
+  logger
+);
+```
+**When to Use Password Auth**:
+- ✅ Standalone scripts (Node.js/Deno)
+- ✅ Simple setup (no key management)
+- ✅ Vendor provides password credentials
+-  Less secure than SSH keys
+-  Passwords can be intercepted if environment is compromised
+### SSH Private Key Authentication
+```typescript
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp-secure',
+    name: 'Secure Vendor SFTP',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'secure_integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY, // PEM format private key
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+    },
+  },
+  logger
+);
+```
+**Private Key Format (PEM)**:
+```bash
+# .env file - key content as string (NOT file path)
+SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
+MIIEpAIBAAKCAQEA1a2b3c4d5e6f7g8h9i0j...
+...
+-----END RSA PRIVATE KEY-----"
+```
+**When to Use SSH Key Auth**:
+- ✅ More secure (no password transmission)
+- ✅ Industry best practice for B2B integrations
+- ✅ Key rotation without password changes
+- ✅ Supports non-interactive authentication
+### Encrypted Private Key (with Passphrase)
+```typescript
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp-encrypted',
+    name: 'Encrypted Key SFTP',
+    settings: {
+      host: 'sftp.vendor.com',
+      port: 22,
+      username: 'secure_integration',
+      privateKey: process.env.SFTP_PRIVATE_KEY, // Encrypted PEM key
+      passphrase: process.env.SFTP_KEY_PASSPHRASE, // Key passphrase
+      remotePath: '/data/inventory',
+      filePattern: '*.csv',
+    },
+  },
+  logger
+);
+```
+**Environment Variables**:
+```bash
+SFTP_PRIVATE_KEY="-----BEGIN ENCRYPTED PRIVATE KEY-----
+...encrypted key content...
+-----END ENCRYPTED PRIVATE KEY-----"
+SFTP_KEY_PASSPHRASE=your-secure-passphrase
+```
+---
+## Configuration
+### Basic Configuration
+```typescript
+import { SftpDataSource, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
+const config: SftpDataSourceConfig = {
+  type: 'SFTP_CSV', // SFTP_JSON, SFTP_JSONL, SFTP_PARQUET
+  connectionId: 'vendor-sftp',
+  name: 'Vendor SFTP Connection',
+  settings: {
+    // Connection
+    host: 'sftp.vendor.com',
+    port: 22,
+    username: 'integration',
+    password: process.env.SFTP_PASSWORD, // or privateKey
+    // Paths
+    remotePath: '/data/inventory',
+    filePattern: '*.csv',
+    // Connection pool
+    connectionTimeout: 30000, // 30 seconds
+    keepaliveInterval: 5000, // 5 seconds keepalive
+    maxConnections: 5, // Pool size
+    // Write settings
+    writeCreateDirectories: true,
+    writeOverwrite: false,
+    writePermissions: '0644',
+  },
+};
+const sftpSource = new SftpDataSource(config, logger);
+```
+### Configuration Options
+| Property                          | Required | Description                           | Default   | Example             |
+| --------------------------------- | -------- | ------------------------------------- | --------- | ------------------- |
+| `type`                            | ✅       | Data source type                      | -         | `'SFTP_CSV'`        |
+| `connectionId`                    | ✅       | Unique connection ID                  | -         | `'vendor-sftp'`     |
+| `name`                            |        | Connection display name               | -         | `'Vendor SFTP'`     |
+| `settings.host`                   | ✅       | SFTP server hostname                  | -         | `'sftp.vendor.com'` |
+| `settings.port`                   |        | SSH port                              | 22        | `2222`              |
+| `settings.username`               | ✅       | SSH username                          | -         | `'integration'`     |
+| `settings.password`               | ✅\*     | Password auth                         | -         | `'secret'`          |
+| `settings.privateKey`             | ✅\*     | SSH private key (PEM)                 | -         | `'-----BEGIN...'`   |
+| `settings.passphrase`             |        | Key passphrase (if encrypted)         | -         | `'key-pass'`        |
+| `settings.remotePath`             |        | Default remote directory              | `/`       | `'/data/inventory'` |
+| `settings.filePattern`            |        | File glob pattern                     | `*`       | `'*.csv'`           |
+| `settings.maxConnections`         |        | Connection pool size       | 5         | `3`                 |
+| `settings.connectionTimeout`      |        | Connection timeout (ms)               | 30000     | `60000`             |
+| `settings.keepaliveInterval`      |        | Keepalive interval (ms)               | 5000      | `10000`             |
+| `settings.retry`                  |        | Retry configuration object | See below | See below           |
+| `settings.writeCreateDirectories` |        | Auto-create dirs on write             | `true`    | `false`             |
+| `settings.writeOverwrite`         |        | Overwrite existing files              | `false`   | `true`              |
+| `settings.writePermissions`       |        | File permissions (octal)              | `'0644'`  | `'0600'`            |
+\* Either `password` or `privateKey` required
+### Enhanced Retry Configuration
+Fine-grained retry control with exponential backoff, jitter, and custom predicates.
+```typescript
+settings: {
+  host: 'sftp.vendor.com',
+  username: 'integration',
+  privateKey: process.env.SFTP_PRIVATE_KEY,
+  // Enhanced retry configuration (optional)
+  retry: {
+    maxAttempts: 3,           // Max retry attempts (default: 3)
+    baseDelayMs: 1000,         // Initial delay in ms (default: 1000)
+    backoffFactor: 2,          // Exponential factor (default: 2)
+    maxDelayMs: 8000,          // Max delay cap (default: 8000)
+    jitter: 'full',            // Jitter strategy: 'none' | 'full' (default: 'full')
+    reconnectOnFailure: true,  // Reconnect between retries (default: true)
+    // Optional: Custom retry predicate
+    isRetryable: (error: Error) => {
+      // Return true to retry, false to fail immediately
+      return error.message.includes('ECONNRESET');
+    }
+  }
+}
+```
+**Retry Configuration Options:**
+| Option               | Type       | Default  | Description                                             |
+| -------------------- | ---------- | -------- | ------------------------------------------------------- |
+| `maxAttempts`        | `number`   | `3`      | Maximum retry attempts (including initial attempt)      |
+| `baseDelayMs`        | `number`   | `1000`   | Base delay in milliseconds                              |
+| `backoffFactor`      | `number`   | `2`      | Exponential backoff multiplier                          |
+| `maxDelayMs`         | `number`   | `8000`   | Maximum delay cap in milliseconds                       |
+| `jitter`             | `string`   | `'full'` | Jitter strategy: `'none'` or `'full'` (randomization)   |
+| `reconnectOnFailure` | `boolean`  | `true`   | Reconnect to SFTP between retry attempts                |
+| `isRetryable`        | `function` | Built-in | Custom function to determine if error should be retried |
+**Default Retry Behavior:**
+The SDK automatically retries these error types:
+- **Network errors**: `ECONNRESET`, `ETIMEDOUT`, `ENOTFOUND`, `EHOSTUNREACH`, `ECONNREFUSED`, `EPIPE`
+- **Transient messages**: `connection lost`, `timeout`, `socket hang up`, `network error`
+The SDK **never retries** these errors (require manual intervention):
+- **Authentication failures**: `permission denied`, `authentication`
+- **File not found**: `no such file`, `file not found`
+- **Configuration errors**: `algorithm`, `cipher`, `key exchange`
+### Environment Variables Pattern
+```bash
+# .env file
+SFTP_HOST=sftp.vendor.com
+SFTP_PORT=22
+SFTP_USERNAME=integration_user
+SFTP_PASSWORD=secret-password
+# Or SSH key (recommended)
+SFTP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
+...key content...
+-----END RSA PRIVATE KEY-----"
+SFTP_PASSPHRASE=optional-passphrase
+# Paths
+SFTP_REMOTE_PATH=/data/inventory
+SFTP_FILE_PATTERN=*.csv
+```
+```typescript
+// Load from environment
+const sftpSource = new SftpDataSource(
+  {
+    type: 'SFTP_CSV',
+    connectionId: 'vendor-sftp',
+    settings: {
+      host: process.env.SFTP_HOST!,
+      port: parseInt(process.env.SFTP_PORT || '22'),
+      username: process.env.SFTP_USERNAME!,
+      password: process.env.SFTP_PASSWORD, // or
+      privateKey: process.env.SFTP_PRIVATE_KEY, // privateKey
+      passphrase: process.env.SFTP_PASSPHRASE,
+      remotePath: process.env.SFTP_REMOTE_PATH || '/',
+      filePattern: process.env.SFTP_FILE_PATTERN || '*',
+      maxConnections: parseInt(process.env.SFTP_MAX_CONN || '5'),
+    },
+  },
+  logger
+);
+```
+---
+## File Operations
+### Listing Files
+```typescript
+// List all files in default remotePath
+const allFiles = await sftpSource.listFiles();
+// List with filters
+const recentFiles = await sftpSource.listFiles({
+  remotePath: '/data/current',
+  filePattern: '*.csv',
+  lastProcessedTimestamp: '2025-01-01T00:00:00Z', // Only files modified after
+});
+// List by file name substring
+const inventoryFiles = await sftpSource.listFiles({
+  fileName: 'inventory', // Contains 'inventory'
+});
+// Process file metadata
+for (const file of recentFiles) {
+  console.log({
+    name: file.name, // 'inventory-2025-01-15.csv'
+    path: file.path, // '/data/current/inventory-2025-01-15.csv'
+    size: file.size, // 1024000 (bytes)
+    lastModified: file.lastModified, // Date object
+    type: file.type, // 'file' or 'directory'
+    permissions: file.permissions, // '-rw-r--r--'
+  });
+}
+```
+**List Options**:
+| Option                   | Type     | Description                                 |
+| ------------------------ | -------- | ------------------------------------------- |
+| `remotePath`             | `string` | Remote directory to list (overrides config) |
+| `filePattern`            | `string` | Glob pattern filter (e.g., `'*.csv'`)       |
+| `fileName`               | `string` | Substring match filter                      |
+| `lastProcessedTimestamp` | `string` | ISO timestamp - only files modified after   |
+### Reading Files
+```typescript
+// Read text file (CSV, JSON, etc.)
+const csvContent = await sftpSource.downloadFile('/data/inventory.csv');
+// Returns: string
+// Read binary file (Parquet, images, etc.)
+const parquetData = await sftpSource.downloadFile('data.parquet');
+// Returns: Buffer
+// Read with full path
+const content = await sftpSource.downloadFile('/remote/path/to/file.csv');
+// Read relative to remotePath
+const content2 = await sftpSource.downloadFile('subdir/file.csv');
+// Resolves to: /data/inventory/subdir/file.csv (if remotePath=/data/inventory)
+```
+### Writing Files
+```typescript
+// Upload file content (string)
+await sftpSource.uploadFile('/exports/inventory.csv', csvData, {
+  createDirectories: true,
+  overwrite: false,
+  permissions: '0644',
+});
+// Upload binary content (Buffer)
+await sftpSource.uploadFile('/exports/data.parquet', parquetBuffer, {
+  createDirectories: true,
+  permissions: '0600', // Read/write owner only
+});
+// Upload with auto-create directories
+await sftpSource.uploadFile(
+  '/exports/2025/01/15/inventory.csv', // Creates /exports/2025/01/15/ if missing
+  csvData,
+  { createDirectories: true }
+);
+```
+**Write Options**:
+| Option              | Type                 | Description                | Default  |
+| ------------------- | -------------------- | -------------------------- | -------- |
+| `createDirectories` | `boolean`            | Create missing parent dirs | `true`   |
+| `overwrite`         | `boolean`            | Overwrite existing file    | `false`  |
+| `permissions`       | `string`             | File permissions (octal)   | `'0644'` |
+| `encoding`          | `'utf8' \| 'binary'` | Content encoding           | `'utf8'` |
+### Deleting Files
+```typescript
+// Delete single file
+await sftpSource.deleteFile('/temp/old-file.csv');
+// Delete with error handling
+try {
+  await sftpSource.deleteFile('/important-file.csv');
+  console.log('File deleted successfully');
+} catch (error: any) {
+  if (error.message.includes('No such file')) {
+    console.log('File already deleted or does not exist');
+  } else {
+    console.error('Delete failed:', error.message);
+  }
+}
+```
+### Moving Files (Atomic Rename)
+```typescript
+// Move file (atomic SFTP rename operation)
+await sftpSource.moveFile(
+  '/incoming/data.csv',
+  '/processed/data.csv',
+  false // Don't overwrite if target exists
+);
+// Move to archive with date-based organization
+const today = new Date().toISOString().split('T')[0];
+await sftpSource.moveFile('/incoming/inventory.csv', `/archive/${today}/inventory.csv`);
+```
+**IMPORTANT**: Unlike S3's copy+delete, SFTP `moveFile()` uses the **native SFTP rename command**, which is **atomic**. The file is renamed in a single operation with no risk of existing in both locations.
+### Copying Files
+```typescript
+// Copy file (download + upload)
+await sftpSource.copyFile(
+  '/source/inventory.csv',
+  '/backup/inventory_backup.csv',
+  true // Overwrite if exists
+);
+// Copy with different permissions
+await sftpSource.copyFile(
+  '/source/data.csv',
+  '/archive/data.csv',
+  false,
+  '0600' // Restrictive permissions for archive
+);
+```
+**Note**: `copyFile()` is implemented as download + upload (not atomic). For atomic operations, use `moveFile()`.
+### Directory Operations
+```typescript
+// Create directory
+await sftpSource.createDirectory('/exports/daily', true, '0755');
+// Create nested directories
+await sftpSource.createDirectory('/exports/2025/01/15', true);
+// Creates all missing parents: /exports → /exports/2025 → /exports/2025/01 → /exports/2025/01/15
+// Check if directory exists
+const dirExists = await sftpSource.directoryExists('/data/exports');
+console.log(`Directory exists: ${dirExists}`);
+// Check if file exists
+const fileExists = await sftpSource.fileExists('/data/inventory.csv');
+console.log(`File exists: ${fileExists}`);
+```
+---
+## Connection Management
+### Connection Pool Information
+```typescript
+// Get connection details
+const connectionInfo = sftpSource.getConnectionInfo();
+console.log('Connection Info:', {
+  type: connectionInfo.type, // 'SFTP_CSV'
+  host: connectionInfo.host, // 'sftp.vendor.com'
+  port: connectionInfo.port, // 22
+  username: connectionInfo.username, // 'integration'
+  remotePath: connectionInfo.remotePath, // '/data/inventory'
+  filePattern: connectionInfo.filePattern, // '*.csv'
+  currentConnections: connectionInfo.connectionPoolSize, // 3
+  maxConnections: connectionInfo.maxConnections, // 5
+});
+```
+### Connection Health
+The SFTP data source automatically manages connection health:
+```
+Connection Lifecycle:
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ 1. Connection Created           ‚
+‚    - SSH handshake              ‚
+‚    - Authentication             ‚
+‚    - Add to pool                ‚
+”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
+           ‚
+           ¼
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ 2. Connection Active            ‚
+‚    - Reused for operations      ‚
+‚    - Keepalive pings (5s)       ‚
+‚    - Health monitoring          ‚
+”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
+           ‚
+           ¼
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ 3. Connection Idle              ‚
+‚    - No operations for timeout  ‚
+‚    - Auto-close after TTL       ‚
+‚    - Removed from pool          ‚
+”€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
+Failure Handling:
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ Operation Fails                 ‚
+‚ - Network error                 ‚
+‚ - Connection timeout            ‚
+‚ - Authentication failure        ‚
+”€€€€€€€€€€¬€€€€€€€€€€€€€€€€€€€€€€˜
+           ‚
+           ¼
+Œ€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€
+‚ Automatic Retry                 ‚
+‚ - Exponential backoff           ‚
+‚ - New connection created        ‚
+‚ - Operation retried             ‚
+”€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€€˜
+```
+### Validating Connection
+```typescript
+// Test SFTP connectivity
+const isValid = await sftpSource.validateConnection();
+if (isValid) {
+  console.log('SFTP connection successful');
+} else {
+  console.error('Failed to connect to SFTP server');
+}
+// Use in startup health check
+async function startupHealthCheck() {
+  try {
+    const valid = await sftpSource.validateConnection();
+    if (!valid) {
+      throw new Error('SFTP connection validation failed');
+    }
+    console.log('❌œ“ SFTP connection validated');
+  } catch (error) {
+    console.error('❌œ— SFTP connection failed:', error);
+    process.exit(1);
+  }
+}
+```
+---
+## Multi-Format Support
+### CSV Format
+```typescript
+// CSV-specific configuration
+const csvConfig: SftpDataSourceConfig = {
+  type: 'SFTP_CSV',
+  connectionId: 'vendor-csv',
+  settings: {
+    host: 'sftp.vendor.com',
+    username: 'integration',
+    privateKey: process.env.SFTP_KEY,
+    // CSV parsing options
+    csvDelimiter: ',',
+    csvHeaders: ['sku', 'quantity', 'location', 'status'],
+    requiredCsvHeaders: ['sku', 'quantity'], // Validation
+    csvSkipEmptyLines: true,
+    csvTrimValues: true,
+    csvQuote: '"',
+    csvEscape: '\\',
+  },
+};
+// Parse CSV content
+const csvContent = await sftpSource.downloadFile('inventory.csv');
+const records = sftpSource.parseCsvContent(csvContent, fileMetadata);
+// Uses csvDelimiter, csvSkipEmptyLines, etc. from config
+// Generate CSV content
+const csvOutput = sftpSource.writeCsvContent(records, {
+  delimiter: ',',
+  headers: ['sku', 'qty', 'location'],
+  includeHeader: true,
+});
+await sftpSource.writeFile('/exports/inventory.csv', csvOutput);
+```
+### JSON Format
+```typescript
+// JSON-specific configuration
+const jsonConfig: SftpDataSourceConfig = {
+  type: 'SFTP_JSON',
+  connectionId: 'vendor-json',
+  settings: {
+    host: 'sftp.vendor.com',
+    username: 'integration',
+    privateKey: process.env.SFTP_KEY,
+    // JSON options
+    jsonFormat: 'json', // or 'jsonl'
+    jsonValidate: true,
+    jsonMaxDepth: 10,
+    jsonIndent: 2,
+    jsonSortKeys: true,
+  },
+};
+// Parse JSON content
+const jsonContent = await sftpSource.downloadFile('data.json');
+const records = sftpSource.parseJsonContent(jsonContent, fileMetadata, {
+  format: 'json',
+  validate: true,
+  maxDepth: 10,
+});
+// Generate JSON content
+const jsonOutput = sftpSource.writeJsonContent(records, {
+  format: 'json',
+  indent: 2,
+  sortKeys: true,
+});
+await sftpSource.writeFile('/exports/data.json', jsonOutput);
+```
+### JSON Lines (JSONL) Format
+```typescript
+// Parse JSONL (one JSON object per line)
+const jsonlContent = await sftpSource.downloadFile('stream-data.jsonl');
+const records = sftpSource.parseJsonContent(jsonlContent, fileMetadata, {
+  format: 'jsonl',
+});
+// Generate JSONL content
+const jsonlOutput = sftpSource.writeJsonContent(records, {
+  format: 'jsonl',
+});
+// Output:
+// {"id":1,"name":"Product A"}
+// {"id":2,"name":"Product B"}
+await sftpSource.writeFile('/exports/stream.jsonl', jsonlOutput);
+```
+### Parquet Format
+```typescript
+// Parquet-specific configuration
+const parquetConfig: SftpDataSourceConfig = {
+  type: 'SFTP_PARQUET',
+  connectionId: 'vendor-parquet',
+  settings: {
+    host: 'sftp.vendor.com',
+    username: 'integration',
+    privateKey: process.env.SFTP_KEY,
+    // Parquet options
+    parquetCompression: 'snappy', // 'none', 'gzip', 'snappy', 'lz4'
+    parquetSchema: {
+      productRef: 'string',
+      quantity: 'int32',
+      price: 'double',
+    },
+    parquetRowGroupSize: 1000,
+  },
+};
+// Parse Parquet content (async)
+const parquetBuffer = await sftpSource.downloadFile('data.parquet');
+// Note: parseParquetContent currently throws (not implemented). Use ParquetParserService.
+// const records = await sftpSource.parseParquetContent(parquetBuffer as Buffer, fileMetadata);
+// Parquet generation would use ParquetParserService separately
+```
+### Processing Mixed File Types
+```typescript
+async function processMultipleFormats() {
+  const files = await sftpSource.listFiles({
+    filePattern: '*.{csv,json,parquet}',
+  });
+  for (const file of files) {
+    const extension = file.name.split('.').pop()?.toLowerCase();
+    let records: any[] = [];
+    switch (extension) {
+      case 'csv':
+        const csvContent = await sftpSource.downloadFile(file.path);
+        records = sftpSource.parseCsvContent(csvContent, file);
+        break;
+      case 'json':
+        const jsonContent = await sftpSource.downloadFile(file.path);
+        records = sftpSource.parseJsonContent(jsonContent, file, {
+          format: 'json',
+        });
+        break;
+      case 'parquet':
+        const parquetBuffer = await sftpSource.downloadFile(file.path);
+        // Use ParquetParserService here instead of parseParquetContent (not implemented)
+        // records = await sftpParser.parseParquetContent(parquetBuffer as Buffer, file);
+        break;
+      default:
+        logger.warn(`Unsupported format: ${extension}`);
+        continue;
+    }
+    await processRecords(records, file.name);
+  }
+}
+```
+---
+## Error Handling
+### Common Error Patterns
+```typescript
+import { DataSourceError } from '@fluentcommerce/fc-connect-sdk';
+try {
+  await sftpSource.writeFile(remotePath, content);
+} catch (error) {
+  if (error instanceof DataSourceError) {
+    switch (error.operation) {
+      case 'uploadFile':
+        if (error.message.includes('Permission denied')) {
+          logger.error('SFTP permission error', {
+            remotePath,
+            suggestion: 'Check file permissions and directory access',
+          });
+        }
+        break;
+      case 'listFiles':
+        if (error.message.includes('No such file')) {
+          logger.warn('Remote directory not found', {
+            remotePath: error.context?.remotePath,
+            suggestion: 'Verify remote path exists',
+          });
+        }
+        break;
+      case 'downloadFile':
+        if (error.message.includes('Connection timeout')) {
+          logger.error('SFTP connection timeout', {
+            suggestion: 'Check network connectivity and server status',
+          });
+        }
+        break;
+    }
+  } else {
+    logger.error('Unexpected error', error);
+  }
+}
+```
+### Retry with Exponential Backoff
+```typescript
+async function uploadWithRetry(
+  content: string,
+  remotePath: string,
+  maxRetries: number = 3
+): Promise<void> {
+  let lastError: Error;
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    try {
+      await sftpSource.writeFile(remotePath, content);
+      logger.info(`Upload successful on attempt ${attempt}`);
+      return;
+    } catch (error) {
+      lastError = error as Error;
+      logger.warn(`Upload attempt ${attempt} failed`, {
+        error: (error as Error).message,
+        retriesLeft: maxRetries - attempt,
+      });
+      if (attempt < maxRetries) {
+        const delay = Math.pow(2, attempt) * 1000; // 2s, 4s, 8s
+        logger.info(`Retrying in ${delay}ms...`);
+        await new Promise(resolve => setTimeout(resolve, delay));
+      }
+    }
+  }
+  throw new Error(`Upload failed after ${maxRetries} attempts: ${lastError!.message}`);
+}
+```
+---
+## Production Patterns
+### Pattern 1: Vendor Inventory Ingestion
+```typescript
+import {
+  SftpDataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+} from '@fluentcommerce/fc-connect-sdk';
+async function ingestVendorInventory() {
+  const sftpSource = new SftpDataSource(
+    {
+      /* config */
+    },
+    logger
+  );
+  const csvParser = new CSVParserService();
+  const mapper = new UniversalMapper(mappingConfig);
+  const stateService = new StateService(kvAdapter);
+  // List new files
+  const files = await sftpSource.listFiles({
+    remotePath: '/outbound/inventory',
+    filePattern: 'inventory_*.csv',
+  });
+  for (const file of files) {
+    // Check if already processed
+    const stateKey = `processed:${file.name}:${file.lastModified?.toString()}`;
+    const processed = await stateService.isFileProcessed(kvAdapter as any, stateKey);
+    if (processed) {
+      logger.info('File already processed', { file: file.name });
+      continue;
+    }
+    try {
+      // Download and parse CSV
+      const content = await sftpSource.downloadFile(file.path);
+      const records = await csvParser.parse(content as string);
+      // Map fields
+      const result = await mapper.map(records);
+      if (!result.success) {
+        logger.error('Mapping failed', undefined, { errors: result.errors });
+        continue;
+      }
+      // Send to Fluent
+      await sendToFluent(result.data);
+      // Mark as processed
+      await stateService.updateSyncState(
+        kvAdapter as any,
+        [
+          {
+            fileName: file.name,
+            recordCount: records.length,
+            processedAt: new Date().toISOString(),
+          },
+        ],
+        'sftp-ingestion'
+      );
+      // Move to processed folder
+      await sftpSource.moveFile(file.path, file.path.replace('/outbound/', '/processed/'));
+      logger.info('File processed successfully', { file: file.name });
+    } catch (error) {
+      logger.error('Failed to process file', error as Error, { file: file.path });
+    }
+  }
+}
+```
+### Pattern 2: State Management with SFTP
+```typescript
+async function processFilesWithState() {
+  const files = await sftpSource.listFiles();
+  const stateService = new StateService(logger);
+  for (const file of files) {
+    // Use file name + modification time as unique key
+    const stateKey = `processed:${file.name}:${file.lastModified.toISOString()}`;
+    if (await stateService.isFileProcessed(kvAdapter as any, stateKey)) {
+      logger.info('File already processed', { fileName: file.name });
+      continue;
+    }
+    try {
+      const content = await sftpSource.downloadFile(file.path);
+      await processFile(content, file.name);
+      // Mark as processed
+      await stateService.updateSyncState(
+        kvAdapter as any,
+        [
+          {
+            fileName: file.name,
+            size: file.size,
+            processedAt: new Date().toISOString(),
+          },
+        ],
+        'sftp-processing'
+      );
+    } catch (error) {
+      logger.error('Failed to process file', error as Error, { file: file.path });
+    }
+  }
+}
+```
+---
+## API Reference
+### SftpDataSource Constructor
+```typescript
+new SftpDataSource(config: SftpDataSourceConfig, logger?: StructuredLogger)
+```
+**Parameters**:
+- `config`: SFTP data source configuration
+- `logger`: Optional logging service
+### Methods
+| Method                  | Parameters                                         | Returns                   | Description                   |
+| ----------------------- | -------------------------------------------------- | ------------------------- | ----------------------------- | --------------------- |
+| `listFiles()`           | `options?`                                         | `Promise<FileMetadata[]>` | List files in remote path     |
+| `downloadFile()`        | `path`                                             | `Promise<string           | Buffer>`                      | Download file content |
+| `downloadFile()`        | `path`                                             | `Promise<Buffer>`         | Download file as buffer       |
+| `writeFile()`           | `path, content, options?`                          | `Promise<void>`           | Write file to SFTP server     |
+| `deleteFile()`          | `path`                                             | `Promise<void>`           | Delete file from server       |
+| `copyFile()`            | `sourcePath, targetPath, overwrite?, permissions?` | `Promise<void>`           | Copy file (download + upload) |
+| `moveFile()`            | `sourcePath, targetPath, overwrite?`               | `Promise<void>`           | Move file (atomic rename)     |
+| `fileExists()`          | `path`                                             | `Promise<boolean>`        | Check if file exists          |
+| `directoryExists()`     | `path`                                             | `Promise<boolean>`        | Check if directory exists     |
+| `createDirectory()`     | `path, recursive?, permissions?`                   | `Promise<void>`           | Create directory              |
+| `validateConnection()`  | -                                                  | `Promise<boolean>`        | Test SFTP connectivity        |
+| `getConnectionInfo()`   | -                                                  | `ConnectionInfo`          | Get connection details        |
+| `parseCsvContent()`     | `content, metadata`                                | `any[]`                   | Parse CSV content             |
+| `parseJsonContent()`    | `content, metadata, options?`                      | `any[]`                   | Parse JSON/JSONL content      |
+| `parseParquetContent()` | `buffer, metadata`                                 | `Promise<any[]>`          | Not implemented (throws)      |
+| `writeCsvContent()`     | `records, options?`                                | `string`                  | Generate CSV content          |
+| `writeJsonContent()`    | `records, options?`                                | `string`                  | Generate JSON/JSONL content   |
+---
+## Next Steps
+**Learn file processing patterns** → [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
+**Explore advanced topics** → [Module 5: Advanced Topics](./data-sources-05-advanced-topics.md)
+---
+[→ Back to S3 Operations](./data-sources-02-s3-operations.md) | [Next: File Processing Patterns →](./data-sources-04-file-patterns.md) | [→‘ Back to Guide](../data-sources-readme.md)