@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md

@@ -1,1083 +1,1083 @@
-# Module 06: Data Sources
-
-**Level:** Intermediate
-**Category:** Data Integration
-
-## Overview
-
-Data sources provide unified interfaces for reading and writing data from different storage systems (S3, SFTP, etc.). They handle connection management, file operations, format conversions, and error handling, allowing you to work with multiple data sources through a consistent API.
-
-## Architecture
-
-All data sources extend `AbstractDataSource`, which provides:
-
-**Required Methods (must implement):**
-- `listFiles()` - List files with filtering
-- `downloadFile()` - Download file content
-- `uploadFile()` - Upload file content (⚠️ **mandatory**)
-- `deleteFile()` - Delete a file
-- `copyFileInternal()` - Internal copy implementation
-- `validateConnection()` - Test connectivity
-
-**Template Methods (inherited):**
-- `moveFile()` - Move file using copyFileInternal() + deleteFile()
-- `archiveFile()` - Archive file to timestamped location using moveFile()
-
-Both S3DataSource and SftpDataSource implement all required methods and inherit the template methods. For advanced use cases (e.g., cross-bucket S3 operations), use `copyFile()` directly.
-
-## Table of Contents
-
-- [S3 Data Source](#s3-data-source)
-  - [Constructor](#constructor)
-  - [File Listing](#file-listing)
-  - [File Download](#file-download)
-  - [File Upload](#file-upload)
-  - [File Deletion](#file-deletion)
-  - [File Copy](#file-copy)
-  - [File Move](#file-move)
-  - [Connection Validation](#connection-validation)
-  - [Writing Parquet Files](#writing-parquet-files)
-- [SFTP Data Source](#sftp-data-source)
-  - [Constructor](#sftp-constructor)
-  - [Configuration Types](#configuration-types)
-  - [File Listing](#sftp-file-listing)
-  - [File Download](#sftp-file-download)
-  - [File Existence Checks](#file-existence-checks)
-  - [File Upload](#sftp-file-upload)
-  - [Directory Operations](#directory-operations)
-  - [File Deletion](#sftp-file-deletion)
-  - [File Move](#file-move-atomic)
-  - [File Copy](#sftp-file-copy)
-  - [Multi-Format Support](#multi-format-support)
-  - [Error Handling](#error-handling)
-  - [Connection Management](#connection-management)
-- [Complete Methods Reference](#complete-methods-reference)
-- [See Also](#see-also)
-
----
-
-## S3 Data Source
-
-Read and write files from Amazon S3 with presigned URLs.
-
-### Constructor
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const s3Config: S3DataSourceConfig = {
-  type: 'S3_CSV',
-  connectionId: 's3-api-ref',
-  name: 'S3 API Reference',
-  s3Config: {
-    bucket: 'my-bucket',
-    region: 'us-east-1',
-    accessKeyId: 'your-access-key',
-    secretAccessKey: 'your-secret-key',
-    sessionToken: 'optional-session-token', // Optional for temporary credentials
-  },
-};
-
-const s3 = new S3DataSource(s3Config, logger);
-```
-
-**Configuration Notes:**
-
-`S3DataSourceConfig` extends `DataSourceConfig`, which includes:
-
-- `type: string` - Data source type ('S3_CSV')
-- `connectionId?: string` - Optional connection identifier
-- `name?: string` - Optional descriptive name
-- `connection?: any` - Optional Versori platform connection object (for Versori integration)
-
-For **standalone use** (without Versori), only provide `type` and `s3Config`.
-For **Versori platform integration**, also provide `connection`, `connectionId`, and `name`.
-
-### File Listing
-
-**listFiles(params?)** - List objects in S3 bucket with filtering options
-
-```typescript
-// List all files in bucket
-const files = await s3.listFiles();
-
-// List files with prefix (folder)
-const files = await s3.listFiles({
-  prefix: 'inventory/',
-  maxKeys: 100,
-});
-
-// List files with delimiter (folder separator)
-const files = await s3.listFiles({
-  prefix: 'inventory/',
-  delimiter: '/',
-  maxKeys: 1000,
-});
-
-// Returns: FileMetadata[]
-// [
-//   {
-//     path: 'inventory/data.csv',
-//     name: 'data.csv',
-//     size: 2048,
-//     lastModified: '2025-01-15T10:30:00Z',
-//     contentType: 'text/csv',
-//     etag: '"abc123"',
-//     bucket: 'my-bucket',
-//     region: 'us-east-1',
-//     source: 'S3'
-//   }
-// ]
-```
-
-**Parameters:**
-
-- `prefix?: string` - Filter by object key prefix (e.g., 'folder/')
-- `delimiter?: string` - Delimiter for folder structure (typically '/')
-- `maxKeys?: number` - Maximum number of keys to return (default: 1000)
-- `continuationToken?: string` - Token for paginated results
-
-**Note:** S3 does NOT support wildcard pattern matching. Use prefix + manual filtering:
-
-```typescript
-// Get all CSV files in folder
-const files = await s3.listFiles({ prefix: 'inventory/' });
-
-// IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
-// The asterisk is a glob pattern character, not part of the file extension
-const csvFiles = files.filter(f => f.name.endsWith('.csv'));  // ✓ Correct
-
-// Common mistake to avoid:
-// const csvFiles = files.filter(f => f.name.endsWith('*.csv')); // ✗ WRONG - Never matches!
-
-// Get files matching pattern
-const productFiles = files.filter(f => f.name.startsWith('products_'));
-```
-
-### File Download
-
-**downloadFile(key, options?)** - Download file content from S3
-
-```typescript
-// Download as text (default UTF-8)
-const csvContent = await s3.downloadFile('inventory/data.csv');
-// Returns: string
-
-// Download as binary (for images, Parquet, etc.)
-const parquetBuffer = await s3.downloadFile('exports/data.parquet', {
-  encoding: 'binary',
-});
-// Returns: Buffer
-
-// Download with custom response headers
-const content = await s3.downloadFile('data/file.csv', {
-  responseHeaders: {
-    'Cache-Control': 'no-cache',
-  },
-});
-```
-
-**Parameters:**
-
-- `key: string` - S3 object key (path)
-- `options.encoding?: 'utf8' | 'binary'` - Encoding format (default: 'utf8')
-- `options.responseHeaders?: Record<string, string>` - Custom response headers
-
-**Returns:**
-
-- `Promise<string | Buffer>` - File content as string or Buffer
-
-### File Upload
-
-**uploadFile(key, content, options?)** - Upload file content to S3
-
-```typescript
-// Upload text content
-await s3.uploadFile('output/results.csv', csvContent, {
-  contentType: 'text/csv',
-});
-
-// Upload binary content (Buffer)
-const parquetBuffer = await s3.writeParquetContent(records);
-await s3.uploadFile('exports/data.parquet', parquetBuffer, {
-  contentType: 'application/octet-stream',
-});
-
-// Upload with metadata
-await s3.uploadFile('data/inventory.json', jsonContent, {
-  contentType: 'application/json',
-  metadata: {
-    'upload-date': new Date().toISOString(),
-    source: 'fluent-commerce',
-    'record-count': '1000',
-  },
-});
-```
-
-**Parameters:**
-
-- `key: string` - Destination S3 object key
-- `content: string | Buffer` - File content to upload
-- `options.contentType?: string` - MIME type (default: 'application/octet-stream')
-- `options.metadata?: Record<string, string>` - Custom metadata headers
-
-### File Deletion
-
-**deleteFile(key)** - Delete object from S3
-
-```typescript
-// Delete single file
-await s3.deleteFile('old-data/obsolete.csv');
-
-// Delete with error handling
-try {
-  await s3.deleteFile('temp/file.csv');
-  console.log('File deleted successfully');
-} catch (error) {
-  if (error.code === 'NoSuchKey') {
-    console.log('File does not exist');
-  }
-}
-```
-
-**Parameters:**
-
-- `key: string` - S3 object key to delete
-
-**Note:** Delete succeeds even if file doesn't exist (idempotent)
-
-### File Copy
-
-**copyFile(sourceKey, destKey, options?)** - Copy object within or across buckets
-
-```typescript
-// Copy within same bucket
-await s3.copyFile('data/file.csv', 'archive/file.csv');
-
-// Copy from different bucket
-await s3.copyFile('original.csv', 'backup/copy.csv', {
-  sourceBucket: 'source-bucket',
-});
-
-// Copy with metadata update
-await s3.copyFile('data/file.csv', 'processed/file.csv', {
-  metadata: {
-    'processed-date': new Date().toISOString(),
-    status: 'completed',
-  },
-});
-```
-
-**Parameters:**
-
-- `sourceKey: string` - Source object key
-- `destKey: string` - Destination object key
-- `options.sourceBucket?: string` - Source bucket (defaults to current bucket)
-- `options.metadata?: Record<string, string>` - Metadata for destination object
-
-### File Move
-
-**moveFile(sourcePath, destPath, deleteSource?)** - Move object within same bucket (template method from AbstractDataSource)
-
-```typescript
-// Move within same bucket
-await s3.moveFile('temp/processing.csv', 'archive/completed.csv');
-
-// Copy without deleting source
-await s3.moveFile('data/file.csv', 'backup/file.csv', false);
-```
-
-**Parameters:**
-
-- `sourcePath: string` - Source object key
-- `destPath: string` - Destination object key
-- `deleteSource?: boolean` - Delete source after copy (default: true)
-
-**Returns:** `Promise<void>`
-
-**Important Notes:**
-- S3 moveFile() is NOT atomic - it performs copy then delete. If delete fails, you'll have duplicate files.
-- **Same-bucket only:** This method only works within the same bucket.
-- **Cross-bucket moves:** For moving files between different S3 buckets, use `copyFile()` and `deleteFile()` directly.
-
-**Cross-Bucket Move Example:**
-
-```typescript
-// For cross-bucket operations, use copyFile() + deleteFile() directly
-await s3.copyFile('source.csv', 'archive/source.csv', {
-  sourceBucket: 'source-bucket',
-});
-await s3.deleteFile('source.csv'); // Deletes from current bucket
-```
-
-### File Archive
-
-**archiveFile(sourcePath, archivePrefix?)** - Archive file with timestamp (template method from AbstractDataSource)
-
-```typescript
-// Archive to default 'archive/' prefix
-const archivePath = await s3.archiveFile('data/inventory.csv');
-// Returns: 'archive/2025-10-29T12-30-45-123Z_inventory.csv'
-
-// Archive to custom prefix
-const archivePath = await s3.archiveFile('data/inventory.csv', 'backup/2025/');
-// Returns: 'backup/2025/2025-10-29T12-30-45-123Z_inventory.csv'
-```
-
-**Parameters:**
-
-- `sourcePath: string` - Source object key
-- `archivePrefix?: string` - Archive directory prefix (default: 'archive/')
-
-**Returns:** `Promise<string>` - The archive path where file was moved
-
-**How It Works:**
-1. Generates timestamped filename: `{archivePrefix}{timestamp}_{originalFilename}`
-2. Moves file to archive location using moveFile()
-3. Returns the full archive path
-
-### Connection Validation
-
-**validateConnection()** - Test S3 bucket access
-
-```typescript
-const isValid = await s3.validateConnection();
-if (!isValid) {
-  console.error('Cannot access S3 bucket');
-}
-```
-
-### Writing Parquet Files
-
-Both S3 and SFTP data sources support writing data records as Parquet format with automatic schema inference.
-
-```typescript
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const s3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    connectionId: 's3-parquet-write',
-    name: 'S3 Parquet Write',
-    s3Config: config.s3Config,
-  },
-  logger
-);
-
-// Prepare data records
-const records = [
-  {
-    id: 1,
-    name: 'Product A',
-    qty: 100,
-    price: 19.99,
-    active: true,
-    updatedAt: new Date(),
-  },
-  {
-    id: 2,
-    name: 'Product B',
-    qty: 50,
-    price: 29.99,
-    active: true,
-    updatedAt: new Date(),
-  },
-];
-
-// Write Parquet content with GZIP compression
-const parquetBuffer = await s3.writeParquetContent(records, {
-  compression: 'GZIP',
-  rowGroupSize: 50000,
-});
-
-// Upload to S3
-await s3.uploadFile('output/inventory.parquet', parquetBuffer, {
-  contentType: 'application/octet-stream',
-});
-```
-
-**ParquetWriteOptions:**
-
-```typescript
-interface ParquetWriteOptions {
-  compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
-  schema?: Record<string, any>; // Custom schema, auto-infers if not provided
-  rowGroupSize?: number; // Default: 50000
-}
-```
-
-**Automatic Schema Inference:**
-
-The SDK automatically infers Parquet schema from your data types:
-
-| JavaScript Type    | Parquet Type              |
-| ------------------ | ------------------------- |
-| `string`           | UTF8                      |
-| `number` (integer) | INT64                     |
-| `number` (float)   | DOUBLE                    |
-| `boolean`          | BOOLEAN                   |
-| `Date`             | TIMESTAMP_MILLIS          |
-| `null/undefined`   | UTF8 (optional)           |
-| `object`           | UTF8 (serialized as JSON) |
-
-**Custom Schema Example:**
-
-```typescript
-// Define custom schema for precise control
-const customSchema = {
-  productRef: { type: 'UTF8' },
-  qty: { type: 'INT32' },
-  price: { type: 'DECIMAL', precision: 10, scale: 2 },
-  locationRef: { type: 'UTF8' },
-  updatedAt: { type: 'TIMESTAMP_MILLIS' },
-  active: { type: 'BOOLEAN', optional: true },
-};
-
-const buffer = await s3.writeParquetContent(records, {
-  schema: customSchema,
-  compression: 'SNAPPY',
-});
-```
-
-**Compression Options:**
-
-- **UNCOMPRESSED**: No compression, fastest write, largest file
-- **GZIP**: Good compression ratio, slower, widely compatible
-- **SNAPPY**: Fast compression, good for large files
-- **LZO**: Fast compression (requires native library)
-- **BROTLI**: Best compression ratio, slower
-
-**Complete Workflow Example:**
-
-```typescript
-import {
-  S3DataSource,
-  generateFileName,
-  generateFilePath,
-  generateDateSubdirectories,
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Fetch data from Fluent Commerce
-const inventory = await client.graphql({
-  query: `query { inventoryPositions(first: 1000) { ... } }`,
-});
-
-// Transform to records
-const records = inventory.data.inventoryPositions.edges.map(e => ({
-  productRef: e.node.productRef,
-  locationRef: e.node.locationRef,
-  qty: e.node.qty,
-  updatedAt: new Date(),
-}));
-
-// Write as Parquet
-const buffer = await s3.writeParquetContent(records, {
-  compression: 'GZIP',
-});
-
-// Generate file name with timestamp
-const fileName = generateFileName('inventory', {
-  timestamp: 'yyyymmdd-hhmmss',
-  extension: 'parquet',
-});
-
-// Generate date-based path
-const subdirs = generateDateSubdirectories();
-const fullPath = generateFilePath('exports', fileName, subdirs);
-// Result: 'exports/2025/10/06/inventory-20251006-123045.parquet'
-
-// Upload to S3
-await s3.uploadFile(fullPath, buffer);
-
-console.log(`Exported ${records.length} records to ${fullPath}`);
-```
-
-## SFTP Data Source
-
-Direct SFTP file operations using `ssh2-sftp-client`. Works standalone without Versori dependencies.
-
-### SFTP Constructor
-
-```typescript
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const sftpConfig: SftpDataSourceConfig = {
-  type: 'SFTP_CSV', // or 'SFTP_JSON', 'SFTP_JSONL', 'SFTP_PARQUET'
-  connectionId: 'sftp-prod',
-  name: 'Production SFTP',
-  settings: {
-    // Direct connection credentials (no platform required)
-    host: 'sftp.example.com',
-    port: 22,
-    username: 'user',
-    password: 'password', // or use privateKey
-    privateKey: 'PEM string or Buffer', // Optional SSH key
-    passphrase: 'key passphrase', // Optional for encrypted keys
-    remotePath: '/data',
-    filePattern: '*.csv',
-
-    // Connection settings
-    connectionTimeout: 30000,
-    keepaliveInterval: 5000,
-    maxConnections: 5,
-
-    // Write operation settings
-    writeCreateDirectories: true,
-    writeOverwrite: false,
-    writePermissions: '0644',
-  },
-};
-
-const sftp = new SftpDataSource(sftpConfig, logger);
-```
-
-**Configuration Notes:**
-
-`SftpDataSourceConfig` extends `DataSourceConfig`, which includes:
-
-- `type: string` - Data source type ('SFTP_CSV', 'SFTP_JSON', 'SFTP_JSONL', or 'SFTP_PARQUET')
-- `connectionId?: string` - Optional connection identifier
-- `name?: string` - Optional descriptive name
-- `connection?: SftpConnection` - **Optional** - Only required for Versori platform integration
-
-**Authentication Options:**
-
-- For **standalone use** (no Versori), provide `username` and either `password` or `privateKey` in settings
-- For **Versori platform integration**, provide `connection` object (credentials managed by platform)
-
-### Configuration Types
-
-The SFTP data source uses ssh2-sftp-client internally:
-
-```typescript
-interface SftpDataSourceConfig extends DataSourceConfig {
-  type: 'SFTP_CSV' | 'SFTP_JSON' | 'SFTP_JSONL' | 'SFTP_PARQUET';
-  connection?: SftpConnection; // Optional - only for Versori
-  settings: {
-    // Direct SFTP connection (no connection object needed)
-    host: string;
-    port?: number;
-    username?: string;
-    password?: string;
-    privateKey?: string | Buffer;
-    passphrase?: string;
-    remotePath: string;
-    filePattern: string;
-
-    // CSV settings
-    csvDelimiter?: string;
-    csvHeaders?: string[];
-    csvSkipEmptyLines?: boolean;
-    csvTrimValues?: boolean;
-    csvQuote?: string;
-    csvEscape?: string;
-
-    // JSON settings
-    jsonFormat?: 'json' | 'jsonl';
-    jsonValidate?: boolean;
-    jsonMaxDepth?: number;
-    jsonIndent?: number;
-    jsonSortKeys?: boolean;
-
-    // Parquet settings
-    parquetCompression?: 'none' | 'snappy' | 'gzip' | 'lz4';
-    parquetSchema?: Record<string, string>;
-    parquetRowGroupSize?: number;
-
-    // Write operation settings
-    writeCreateDirectories?: boolean;
-    writeOverwrite?: boolean;
-    writePermissions?: string;
-
-    // Connection settings
-    encoding?: string;
-    connectionTimeout?: number;
-    keepAliveInterval?: number;
-    maxConnections?: number;
-  };
-}
-```
-
-### SFTP File Listing
-
-**listFiles(params?)** - List files on SFTP server with pattern matching and timestamp filtering
-
-```typescript
-// List all files with default pattern
-const files = await sftp.listFiles();
-
-// List files with custom pattern (wildcard supported)
-const files = await sftp.listFiles({
-  filePattern: '*.csv',
-  remotePath: '/data/inventory',
-});
-
-// List files with wildcard patterns
-const files = await sftp.listFiles({
-  filePattern: 'products_*.csv', // Match: products_2025.csv, products_test.csv
-  remotePath: '/data',
-});
-
-// Incremental sync - files modified after timestamp
-const files = await sftp.listFiles({
-  lastProcessedTimestamp: '2025-01-01T00:00:00Z',
-  filePattern: '*.csv',
-});
-
-// Filter by file name substring
-const files = await sftp.listFiles({
-  fileName: 'inventory', // Matches files containing 'inventory'
-  remotePath: '/data',
-});
-
-// Returns: FileMetadata[]
-// [
-//   {
-//     path: '/data/inventory/products.csv',
-//     name: 'products.csv',
-//     size: 1024,
-//     lastModified: '2025-01-15T10:30:00Z',
-//     contentType: 'text/csv',
-//     source: 'SFTP'
-//   }
-// ]
-```
-
-**Parameters:**
-
-- `remotePath?: string` - Remote directory path (defaults to config remotePath)
-- `filePattern?: string` - Wildcard pattern for filtering (e.g., '\_.csv', 'products\_\_.json')
-- `lastProcessedTimestamp?: string` - ISO timestamp for incremental sync
-- `fileName?: string` - Filter by file name substring
-
-**Pattern Matching:**
-
-- `*.csv` - All CSV files
-- `products_*.csv` - CSV files starting with 'products\_'
-- `*_2025_*.json` - JSON files containing '_2025_'
-- Case-insensitive matching
-
-**Note:** Unlike S3, SFTP DOES support wildcard pattern matching natively.
-
-### SFTP File Download
-
-**downloadFile(path, options?)** - Download file content from SFTP server
-
-```typescript
-// Download as text (default UTF-8)
-const csvContent = await sftp.downloadFile('/data/inventory.csv');
-// Returns: string
-
-// Download as buffer (for binary files)
-const buffer = await sftp.downloadFile('/data/file.parquet', {
-  asBuffer: true,
-});
-// Returns: Buffer
-
-// Download with custom encoding
-const content = await sftp.downloadFile('/data/file.txt', {
-  encoding: 'latin1',
-});
-// Returns: string
-
-// Download with size limit
-const content = await sftp.downloadFile('/data/large-file.csv', {
-  maxSize: 10 * 1024 * 1024, // 10 MB limit
-});
-```
-
-**Parameters:**
-
-- `path: string` - Remote file path
-- `options.encoding?: string` - Text encoding (default: 'utf8')
-- `options.asBuffer?: boolean` - Return Buffer instead of string
-- `options.maxSize?: number` - Maximum file size in bytes
-
-**Returns:**
-
-- `Promise<string | Buffer>` - File content
-
-### File Existence Checks
-
-```typescript
-// Check if file exists
-const fileExists = await sftp.fileExists('/data/inventory.csv');
-
-// Check if directory exists
-const dirExists = await sftp.directoryExists('/data/exports');
-```
-
-### SFTP File Upload
-
-**uploadContent(content, remotePath, options?)** - Upload content directly to SFTP server
-
-```typescript
-// Upload text content
-const csvContent = 'sku,qty\nPROD-001,100\nPROD-002,50';
-await sftp.uploadFile('/data/inventory.csv', {
-  overwrite: true,
-  createDirectories: true,
-}, csvContent);
-
-// Upload binary content (Buffer)
-const parquetBuffer = await sftp.writeParquetContent(records);
-await sftp.uploadFile('/exports/data.parquet', {
-  overwrite: true,
-  encoding: 'binary',
-}, parquetBuffer);
-
-// Upload JSON with auto-directory creation
-const jsonData = [
-  { productRef: 'PROD-001', qty: 100 },
-  { productRef: 'PROD-002', qty: 50 },
-];
-await sftp.uploadFile(null, 2, JSON.stringify(jsonData), '/exports/2025/01/15/data.json', {
-  createDirectories: true, // Creates /exports/2025/01/15/ if needed
-  permissions: '0644',
-});
-```
-
-**Parameters:**
-
-- `content: string | Buffer` - Content to upload
-- `remotePath: string` - Destination file path
-- `options.overwrite?: boolean` - Overwrite if file exists (default: false)
-- `options.createDirectories?: boolean` - Create parent directories (default: false)
-- `options.permissions?: string` - File permissions (default: '0644')
-- `options.encoding?: string` - Text encoding (default: 'utf8')
-
-**uploadFile(localPath, remotePath, options?)** - Upload local file to SFTP server
-
-```typescript
-// Upload local file
-await sftp.uploadFile('/local/data/inventory.csv', '/remote/data/inventory.csv', {
-  overwrite: true,
-  createDirectories: true,
-  permissions: '0644',
-});
-```
-
-**Parameters:**
-
-- `localPath: string` - Local file path
-- `remotePath: string` - Remote destination path
-- `options` - Same as uploadContent
-
-### Directory Operations
-
-**createDirectory(remotePath, recursive, permissions?)** - Create directory on SFTP server
-
-```typescript
-// Create single directory
-await sftp.createDirectory('/data/exports', false);
-
-// Create directory tree
-await sftp.createDirectory('/data/exports/2025/01/15', true, '0755');
-
-// Check before creating
-if (!(await sftp.directoryExists('/data/archive'))) {
-  await sftp.createDirectory('/data/archive', true);
-}
-```
-
-**Parameters:**
-
-- `remotePath: string` - Directory path to create
-- `recursive: boolean` - Create parent directories (default: false)
-- `permissions?: string` - Directory permissions (default: '0755')
-
-### SFTP File Deletion
-
-**deleteFile(remotePath)** - Delete file from SFTP server
-
-```typescript
-// Delete single file
-await sftp.deleteFile('/temp/processing.csv');
-
-// Delete with existence check
-if (await sftp.fileExists('/temp/old-file.csv')) {
-  await sftp.deleteFile('/temp/old-file.csv');
-}
-
-// Bulk delete
-const tempFiles = await sftp.listFiles({ remotePath: '/temp', filePattern: '*.tmp' });
-for (const file of tempFiles) {
-  await sftp.deleteFile(file.path);
-}
-```
-
-**Parameters:**
-
-- `remotePath: string` - File path to delete
-
-**Throws:** Error if file does not exist
-
-### File Move (Atomic)
-
-**moveFile(oldPath, newPath, overwrite?)** - Move/rename file on SFTP server
-
-```typescript
-// Move file between directories
-await sftp.moveFile('/data/processing.csv', '/data/archive/processed.csv', false);
-
-// Rename file in same directory
-await sftp.moveFile(
-  '/data/temp_file.csv',
-  '/data/inventory_2025.csv',
-  true // overwrite if exists
-);
-
-// Archive processed files
-const files = await sftp.listFiles({ remotePath: '/data/inbox' });
-for (const file of files) {
-  const archivePath = `/data/archive/${file.name}`;
-  await sftp.moveFile(file.path, archivePath, false);
-}
-```
-
-**Parameters:**
-
-- `oldPath: string` - Current file path
-- `newPath: string` - New file path
-- `overwrite?: boolean` - Overwrite destination (default: false)
-
-**Important:** SFTP moveFile() is ATOMIC - single server-side operation (unlike S3)
-
-### SFTP File Copy
-
-**copyFile(sourcePath, destPath, overwrite?)** - Copy file on SFTP server
-
-```typescript
-// Create backup
-await sftp.copyFile('/data/inventory.csv', '/backup/inventory_backup.csv', true);
-
-// Copy to multiple locations
-const sourceFile = '/data/master.csv';
-const destinations = ['/backup/master.csv', '/archive/2025/master.csv', '/staging/master.csv'];
-
-for (const dest of destinations) {
-  await sftp.copyFile(sourceFile, dest, true);
-}
-```
-
-**Parameters:**
-
-- `sourcePath: string` - Source file path
-- `destPath: string` - Destination file path
-- `overwrite?: boolean` - Overwrite destination (default: false)
-
-**Note:** Implemented as download + upload (SFTP protocol has no native copy command)
-
-### SFTP File Archive
-
-**archiveFile(sourcePath, archivePrefix?)** - Archive file with timestamp (template method from AbstractDataSource)
-
-```typescript
-// Archive to default 'archive/' prefix
-const archivePath = await sftp.archiveFile('/data/inventory.csv');
-// Returns: 'archive/2025-10-29T12-30-45-123Z_inventory.csv'
-
-// Archive to custom prefix
-const archivePath = await sftp.archiveFile('/data/inventory.csv', '/backup/2025/');
-// Returns: '/backup/2025/2025-10-29T12-30-45-123Z_inventory.csv'
-
-// Archive processed files after ingestion
-const files = await sftp.listFiles({ remotePath: '/data/inbox' });
-for (const file of files) {
-  await processFile(file);
-  const archivePath = await sftp.archiveFile(file.path, '/data/archive/');
-  console.log(`Archived ${file.name} to ${archivePath}`);
-}
-```
-
-**Parameters:**
-
-- `sourcePath: string` - Source file path
-- `archivePrefix?: string` - Archive directory prefix (default: 'archive/')
-
-**Returns:** `Promise<string>` - The archive path where file was moved
-
-**How It Works:**
-1. Generates timestamped filename: `{archivePrefix}{timestamp}_{originalFilename}`
-2. Moves file to archive location using moveFile()
-3. Returns the full archive path
-
-**Important:** SFTP archiveFile() is atomic since it uses SFTP's atomic moveFile() operation.
-
-### Multi-Format Support
-
-#### CSV Operations
-
-```typescript
-// Write CSV content with custom options
-const csvRecords = [
-  { sku: 'PROD-001', qty: 100, location: 'DC-001' },
-  { sku: 'PROD-002', qty: 50, location: 'DC-002' },
-];
-
-const csvContent = sftp.writeCsvContent(csvRecords, {
-  delimiter: ',',
-  headers: ['sku', 'qty', 'location'],
-  includeHeader: true,
-  quote: '"',
-  escape: '\\',
-  encoding: 'utf8',
-});
-
-// Upload CSV content
-await sftp.uploadFile('/data/inventory.csv', csvContent);
-```
-
-#### JSON Operations
-
-```typescript
-// Write JSON content
-const jsonRecords = [
-  { productRef: 'PROD-001', quantity: 100 },
-  { productRef: 'PROD-002', quantity: 50 },
-];
-
-const jsonContent = sftp.writeJsonContent(jsonRecords, {
-  format: 'json', // or 'jsonl' for JSON Lines
-  indent: 2,
-  sortKeys: true,
-  replacer: (key, value) => (key === 'quantity' ? Number(value) : value),
-});
-
-// Upload JSON content
-await sftp.uploadFile('/data/products.json', jsonContent);
-
-// Parse JSON content from downloaded file
-const downloadedJson = await sftp.downloadFile('products.json');
-const parsedRecords = sftp.parseJsonContent(downloadedJson, {
-  format: 'json',
-  validate: true,
-  maxDepth: 10,
-});
-```
-
-#### JSON Lines Support
-
-```typescript
-// Write JSON Lines (JSONL) format
-const jsonlContent = sftp.writeJsonContent(records, {
-  format: 'jsonl',
-});
-// Produces:
-// {"id":1,"name":"Product A"}
-// {"id":2,"name":"Product B"}
-
-// Parse JSONL content
-const jsonlRecords = sftp.parseJsonContent(jsonlContent, {
-  format: 'jsonl',
-});
-```
-
-**Parquet Operations (SFTP)**
-
-- Read/parse Parquet: Not implemented yet in SFTP data source. Attempting to call `parseParquetContent` will throw until hyparquet integration is added for SFTP. Use S3→download→parsers for Parquet reads.
-- Write Parquet: Implemented via `writeParquetContent(records, options)` producing a Buffer that you can upload with `uploadContent`.
-
-```typescript
-// Write Parquet content (supported)
-const parquetBuffer = await sftp.writeParquetContent(records, {
-  compression: 'gzip',
-  parquetSchema: {
-    productRef: 'string',
-    quantity: 'int32',
-    price: 'double',
-  },
-  parquetRowGroupSize: 1000,
-});
-await sftp.uploadFile('/exports/data.parquet', { overwrite: true }, parquetBuffer);
-
-// Read Parquet content (not yet supported on SFTP data source)
-// const records = await sftp.parseParquetContent(buffer, fileMetadata); // ❌ throws
-```
-
-### Error Handling
-
-The SFTP data source provides complete error handling:
-
-```typescript
-try {
-  await sftp.uploadFile(localPath, remotePath);
-} catch (error) {
-  if (error instanceof DataSourceError) {
-    console.error(`SFTP Error [${error.code}]:`, error.message);
-    console.error('Context:', error.context);
-    console.error('Operation:', error.operation);
-  }
-}
-```
-
-### Connection Management
-
-The SFTP data source includes sophisticated connection management:
-
-```typescript
-// Get connection information
-const connectionInfo = sftp.getConnectionInfo();
-console.log('Pool size:', connectionInfo.connectionPoolSize);
-console.log('Max pool size:', connectionInfo.maxPoolSize);
-
-// Connection pooling is handled automatically
-// Multiple operations reuse connections efficiently
-```
-
-## Complete Methods Reference
-
-### S3 Data Source Methods
-
-**File Operations:**
-
-- `listFiles(params?): Promise<FileMetadata[]>` - List objects with filtering
-- `downloadFile(key, options?): Promise<string | Buffer>` - Download object content
-- `uploadFile(key, content, options?): Promise<void>` - Upload object (required abstract method)
-- `deleteFile(key): Promise<void>` - Delete object
-- `copyFile(sourceKey, destKey, options?): Promise<void>` - Copy object (supports cross-bucket)
-- `moveFile(sourcePath, destPath, deleteSource?): Promise<void>` - Move object within same bucket (template method)
-- `archiveFile(sourcePath, archivePrefix?): Promise<string>` - Archive file with timestamp (template method)
-- `validateConnection(): Promise<boolean>` - Test bucket access
-- `writeParquetContent(records, options?): Promise<Buffer>` - Write Parquet format
-
-### SFTP Data Source Methods
-
-**File Operations:**
-
-- `listFiles(params?): Promise<FileMetadata[]>` - List files with wildcard pattern matching
-- `downloadFile(path, options?): Promise<string | Buffer>` - Download file content
-- `uploadFile(remotePath, content, options?): Promise<void>` - Upload content (required abstract method)
-- `deleteFile(remotePath): Promise<void>` - Delete file
-- `moveFile(oldPath, newPath, overwrite?): Promise<void>` - Move/rename file (atomic, template method)
-- `copyFile(sourcePath, destPath, overwrite?): Promise<void>` - Copy file
-- `archiveFile(sourcePath, archivePrefix?): Promise<string>` - Archive file with timestamp (template method)
-- `fileExists(remotePath): Promise<boolean>` - Check if file exists
-- `directoryExists(remotePath): Promise<boolean>` - Check if directory exists
-- `createDirectory(remotePath, recursive, permissions?): Promise<void>` - Create directory
-- `validateConnection(): Promise<boolean>` - Test SFTP connection
-- `dispose(): Promise<void>` - Clean up connections
-
-**Format Operations:**
-
-- `parseCsvContent(csvContent, fileMetadata): CsvRecord[]` - Parse CSV string
-- `writeCsvContent(records, options?): string` - Generate CSV string
-- `parseJsonContent(jsonContent, fileMetadata, options?): Record<string, unknown>[]` - Parse JSON/JSONL
-- `writeJsonContent(records, options?): string` - Generate JSON/JSONL string
-- `parseParquetContent(buffer, fileMetadata): Promise<ParquetDataRecord[]>` - Not supported; currently throws
-- `writeParquetContent(records, options?): Promise<Buffer>` - Generate Parquet buffer
-
-**Utility Methods:**
-
-- `getConnectionInfo(): ConnectionInfo` - Get connection pool status
-
-## See Also
-
-- [Module 07: Parsers](./api-reference-07-parsers.md) - Format-specific parsing
-- [Module 05: Extraction](../../extraction/extraction-readme.md) - Using data sources for extraction
-- [Module 10: Utilities](./api-reference-10-utilities.md) - Helper functions
-- [Getting Started Guide](../../../00-START-HERE/getting-started.md) - Basic setup
-
----
-
-**[← Previous: Services](./api-reference-05-services.md)** | **[API Reference Home](../api-reference-readme.md)** | **[Next: Parsers →](./api-reference-07-parsers.md)**
+# Module 06: Data Sources
+
+**Level:** Intermediate
+**Category:** Data Integration
+
+## Overview
+
+Data sources provide unified interfaces for reading and writing data from different storage systems (S3, SFTP, etc.). They handle connection management, file operations, format conversions, and error handling, allowing you to work with multiple data sources through a consistent API.
+
+## Architecture
+
+All data sources extend `AbstractDataSource`, which provides:
+
+**Required Methods (must implement):**
+- `listFiles()` - List files with filtering
+- `downloadFile()` - Download file content
+- `uploadFile()` - Upload file content (⚠️ **mandatory**)
+- `deleteFile()` - Delete a file
+- `copyFileInternal()` - Internal copy implementation
+- `validateConnection()` - Test connectivity
+
+**Template Methods (inherited):**
+- `moveFile()` - Move file using copyFileInternal() + deleteFile()
+- `archiveFile()` - Archive file to timestamped location using moveFile()
+
+Both S3DataSource and SftpDataSource implement all required methods and inherit the template methods. For advanced use cases (e.g., cross-bucket S3 operations), use `copyFile()` directly.
+
+## Table of Contents
+
+- [S3 Data Source](#s3-data-source)
+  - [Constructor](#constructor)
+  - [File Listing](#file-listing)
+  - [File Download](#file-download)
+  - [File Upload](#file-upload)
+  - [File Deletion](#file-deletion)
+  - [File Copy](#file-copy)
+  - [File Move](#file-move)
+  - [Connection Validation](#connection-validation)
+  - [Writing Parquet Files](#writing-parquet-files)
+- [SFTP Data Source](#sftp-data-source)
+  - [Constructor](#sftp-constructor)
+  - [Configuration Types](#configuration-types)
+  - [File Listing](#sftp-file-listing)
+  - [File Download](#sftp-file-download)
+  - [File Existence Checks](#file-existence-checks)
+  - [File Upload](#sftp-file-upload)
+  - [Directory Operations](#directory-operations)
+  - [File Deletion](#sftp-file-deletion)
+  - [File Move](#file-move-atomic)
+  - [File Copy](#sftp-file-copy)
+  - [Multi-Format Support](#multi-format-support)
+  - [Error Handling](#error-handling)
+  - [Connection Management](#connection-management)
+- [Complete Methods Reference](#complete-methods-reference)
+- [See Also](#see-also)
+
+---
+
+## S3 Data Source
+
+Read and write files from Amazon S3 with presigned URLs.
+
+### Constructor
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const s3Config: S3DataSourceConfig = {
+  type: 'S3_CSV',
+  connectionId: 's3-api-ref',
+  name: 'S3 API Reference',
+  s3Config: {
+    bucket: 'my-bucket',
+    region: 'us-east-1',
+    accessKeyId: 'your-access-key',
+    secretAccessKey: 'your-secret-key',
+    sessionToken: 'optional-session-token', // Optional for temporary credentials
+  },
+};
+
+const s3 = new S3DataSource(s3Config, logger);
+```
+
+**Configuration Notes:**
+
+`S3DataSourceConfig` extends `DataSourceConfig`, which includes:
+
+- `type: string` - Data source type ('S3_CSV')
+- `connectionId?: string` - Optional connection identifier
+- `name?: string` - Optional descriptive name
+- `connection?: any` - Optional Versori platform connection object (for Versori integration)
+
+For **standalone use** (without Versori), only provide `type` and `s3Config`.
+For **Versori platform integration**, also provide `connection`, `connectionId`, and `name`.
+
+### File Listing
+
+**listFiles(params?)** - List objects in S3 bucket with filtering options
+
+```typescript
+// List all files in bucket
+const files = await s3.listFiles();
+
+// List files with prefix (folder)
+const files = await s3.listFiles({
+  prefix: 'inventory/',
+  maxKeys: 100,
+});
+
+// List files with delimiter (folder separator)
+const files = await s3.listFiles({
+  prefix: 'inventory/',
+  delimiter: '/',
+  maxKeys: 1000,
+});
+
+// Returns: FileMetadata[]
+// [
+//   {
+//     path: 'inventory/data.csv',
+//     name: 'data.csv',
+//     size: 2048,
+//     lastModified: '2025-01-15T10:30:00Z',
+//     contentType: 'text/csv',
+//     etag: '"abc123"',
+//     bucket: 'my-bucket',
+//     region: 'us-east-1',
+//     source: 'S3'
+//   }
+// ]
+```
+
+**Parameters:**
+
+- `prefix?: string` - Filter by object key prefix (e.g., 'folder/')
+- `delimiter?: string` - Delimiter for folder structure (typically '/')
+- `maxKeys?: number` - Maximum number of keys to return (default: 1000)
+- `continuationToken?: string` - Token for paginated results
+
+**Note:** S3 does NOT support wildcard pattern matching. Use prefix + manual filtering:
+
+```typescript
+// Get all CSV files in folder
+const files = await s3.listFiles({ prefix: 'inventory/' });
+
+// IMPORTANT: Use .endsWith('.csv') NOT .endsWith('*.csv')
+// The asterisk is a glob pattern character, not part of the file extension
+const csvFiles = files.filter(f => f.name.endsWith('.csv'));  // ✓ Correct
+
+// Common mistake to avoid:
+// const csvFiles = files.filter(f => f.name.endsWith('*.csv')); // ✗ WRONG - Never matches!
+
+// Get files matching pattern
+const productFiles = files.filter(f => f.name.startsWith('products_'));
+```
+
+### File Download
+
+**downloadFile(key, options?)** - Download file content from S3
+
+```typescript
+// Download as text (default UTF-8)
+const csvContent = await s3.downloadFile('inventory/data.csv');
+// Returns: string
+
+// Download as binary (for images, Parquet, etc.)
+const parquetBuffer = await s3.downloadFile('exports/data.parquet', {
+  encoding: 'binary',
+});
+// Returns: Buffer
+
+// Download with custom response headers
+const content = await s3.downloadFile('data/file.csv', {
+  responseHeaders: {
+    'Cache-Control': 'no-cache',
+  },
+});
+```
+
+**Parameters:**
+
+- `key: string` - S3 object key (path)
+- `options.encoding?: 'utf8' | 'binary'` - Encoding format (default: 'utf8')
+- `options.responseHeaders?: Record<string, string>` - Custom response headers
+
+**Returns:**
+
+- `Promise<string | Buffer>` - File content as string or Buffer
+
+### File Upload
+
+**uploadFile(key, content, options?)** - Upload file content to S3
+
+```typescript
+// Upload text content
+await s3.uploadFile('output/results.csv', csvContent, {
+  contentType: 'text/csv',
+});
+
+// Upload binary content (Buffer)
+const parquetBuffer = await s3.writeParquetContent(records);
+await s3.uploadFile('exports/data.parquet', parquetBuffer, {
+  contentType: 'application/octet-stream',
+});
+
+// Upload with metadata
+await s3.uploadFile('data/inventory.json', jsonContent, {
+  contentType: 'application/json',
+  metadata: {
+    'upload-date': new Date().toISOString(),
+    source: 'fluent-commerce',
+    'record-count': '1000',
+  },
+});
+```
+
+**Parameters:**
+
+- `key: string` - Destination S3 object key
+- `content: string | Buffer` - File content to upload
+- `options.contentType?: string` - MIME type (default: 'application/octet-stream')
+- `options.metadata?: Record<string, string>` - Custom metadata headers
+
+### File Deletion
+
+**deleteFile(key)** - Delete object from S3
+
+```typescript
+// Delete single file
+await s3.deleteFile('old-data/obsolete.csv');
+
+// Delete with error handling
+try {
+  await s3.deleteFile('temp/file.csv');
+  console.log('File deleted successfully');
+} catch (error) {
+  if (error.code === 'NoSuchKey') {
+    console.log('File does not exist');
+  }
+}
+```
+
+**Parameters:**
+
+- `key: string` - S3 object key to delete
+
+**Note:** Delete succeeds even if file doesn't exist (idempotent)
+
+### File Copy
+
+**copyFile(sourceKey, destKey, options?)** - Copy object within or across buckets
+
+```typescript
+// Copy within same bucket
+await s3.copyFile('data/file.csv', 'archive/file.csv');
+
+// Copy from different bucket
+await s3.copyFile('original.csv', 'backup/copy.csv', {
+  sourceBucket: 'source-bucket',
+});
+
+// Copy with metadata update
+await s3.copyFile('data/file.csv', 'processed/file.csv', {
+  metadata: {
+    'processed-date': new Date().toISOString(),
+    status: 'completed',
+  },
+});
+```
+
+**Parameters:**
+
+- `sourceKey: string` - Source object key
+- `destKey: string` - Destination object key
+- `options.sourceBucket?: string` - Source bucket (defaults to current bucket)
+- `options.metadata?: Record<string, string>` - Metadata for destination object
+
+### File Move
+
+**moveFile(sourcePath, destPath, deleteSource?)** - Move object within same bucket (template method from AbstractDataSource)
+
+```typescript
+// Move within same bucket
+await s3.moveFile('temp/processing.csv', 'archive/completed.csv');
+
+// Copy without deleting source
+await s3.moveFile('data/file.csv', 'backup/file.csv', false);
+```
+
+**Parameters:**
+
+- `sourcePath: string` - Source object key
+- `destPath: string` - Destination object key
+- `deleteSource?: boolean` - Delete source after copy (default: true)
+
+**Returns:** `Promise<void>`
+
+**Important Notes:**
+- S3 moveFile() is NOT atomic - it performs copy then delete. If delete fails, you'll have duplicate files.
+- **Same-bucket only:** This method only works within the same bucket.
+- **Cross-bucket moves:** For moving files between different S3 buckets, use `copyFile()` and `deleteFile()` directly.
+
+**Cross-Bucket Move Example:**
+
+```typescript
+// For cross-bucket operations, use copyFile() + deleteFile() directly
+await s3.copyFile('source.csv', 'archive/source.csv', {
+  sourceBucket: 'source-bucket',
+});
+await s3.deleteFile('source.csv'); // Deletes from current bucket
+```
+
+### File Archive
+
+**archiveFile(sourcePath, archivePrefix?)** - Archive file with timestamp (template method from AbstractDataSource)
+
+```typescript
+// Archive to default 'archive/' prefix
+const archivePath = await s3.archiveFile('data/inventory.csv');
+// Returns: 'archive/2025-10-29T12-30-45-123Z_inventory.csv'
+
+// Archive to custom prefix
+const archivePath = await s3.archiveFile('data/inventory.csv', 'backup/2025/');
+// Returns: 'backup/2025/2025-10-29T12-30-45-123Z_inventory.csv'
+```
+
+**Parameters:**
+
+- `sourcePath: string` - Source object key
+- `archivePrefix?: string` - Archive directory prefix (default: 'archive/')
+
+**Returns:** `Promise<string>` - The archive path where file was moved
+
+**How It Works:**
+1. Generates timestamped filename: `{archivePrefix}{timestamp}_{originalFilename}`
+2. Moves file to archive location using moveFile()
+3. Returns the full archive path
+
+### Connection Validation
+
+**validateConnection()** - Test S3 bucket access
+
+```typescript
+const isValid = await s3.validateConnection();
+if (!isValid) {
+  console.error('Cannot access S3 bucket');
+}
+```
+
+### Writing Parquet Files
+
+Both S3 and SFTP data sources support writing data records as Parquet format with automatic schema inference.
+
+```typescript
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const s3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    connectionId: 's3-parquet-write',
+    name: 'S3 Parquet Write',
+    s3Config: config.s3Config,
+  },
+  logger
+);
+
+// Prepare data records
+const records = [
+  {
+    id: 1,
+    name: 'Product A',
+    qty: 100,
+    price: 19.99,
+    active: true,
+    updatedAt: new Date(),
+  },
+  {
+    id: 2,
+    name: 'Product B',
+    qty: 50,
+    price: 29.99,
+    active: true,
+    updatedAt: new Date(),
+  },
+];
+
+// Write Parquet content with GZIP compression
+const parquetBuffer = await s3.writeParquetContent(records, {
+  compression: 'GZIP',
+  rowGroupSize: 50000,
+});
+
+// Upload to S3
+await s3.uploadFile('output/inventory.parquet', parquetBuffer, {
+  contentType: 'application/octet-stream',
+});
+```
+
+**ParquetWriteOptions:**
+
+```typescript
+interface ParquetWriteOptions {
+  compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
+  schema?: Record<string, any>; // Custom schema, auto-infers if not provided
+  rowGroupSize?: number; // Default: 50000
+}
+```
+
+**Automatic Schema Inference:**
+
+The SDK automatically infers Parquet schema from your data types:
+
+| JavaScript Type    | Parquet Type              |
+| ------------------ | ------------------------- |
+| `string`           | UTF8                      |
+| `number` (integer) | INT64                     |
+| `number` (float)   | DOUBLE                    |
+| `boolean`          | BOOLEAN                   |
+| `Date`             | TIMESTAMP_MILLIS          |
+| `null/undefined`   | UTF8 (optional)           |
+| `object`           | UTF8 (serialized as JSON) |
+
+**Custom Schema Example:**
+
+```typescript
+// Define custom schema for precise control
+const customSchema = {
+  productRef: { type: 'UTF8' },
+  qty: { type: 'INT32' },
+  price: { type: 'DECIMAL', precision: 10, scale: 2 },
+  locationRef: { type: 'UTF8' },
+  updatedAt: { type: 'TIMESTAMP_MILLIS' },
+  active: { type: 'BOOLEAN', optional: true },
+};
+
+const buffer = await s3.writeParquetContent(records, {
+  schema: customSchema,
+  compression: 'SNAPPY',
+});
+```
+
+**Compression Options:**
+
+- **UNCOMPRESSED**: No compression, fastest write, largest file
+- **GZIP**: Good compression ratio, slower, widely compatible
+- **SNAPPY**: Fast compression, good for large files
+- **LZO**: Fast compression (requires native library)
+- **BROTLI**: Best compression ratio, slower
+
+**Complete Workflow Example:**
+
+```typescript
+import {
+  S3DataSource,
+  generateFileName,
+  generateFilePath,
+  generateDateSubdirectories,
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Fetch data from Fluent Commerce
+const inventory = await client.graphql({
+  query: `query { inventoryPositions(first: 1000) { ... } }`,
+});
+
+// Transform to records
+const records = inventory.data.inventoryPositions.edges.map(e => ({
+  productRef: e.node.productRef,
+  locationRef: e.node.locationRef,
+  qty: e.node.qty,
+  updatedAt: new Date(),
+}));
+
+// Write as Parquet
+const buffer = await s3.writeParquetContent(records, {
+  compression: 'GZIP',
+});
+
+// Generate file name with timestamp
+const fileName = generateFileName('inventory', {
+  timestamp: 'yyyymmdd-hhmmss',
+  extension: 'parquet',
+});
+
+// Generate date-based path
+const subdirs = generateDateSubdirectories();
+const fullPath = generateFilePath('exports', fileName, subdirs);
+// Result: 'exports/2025/10/06/inventory-20251006-123045.parquet'
+
+// Upload to S3
+await s3.uploadFile(fullPath, buffer);
+
+console.log(`Exported ${records.length} records to ${fullPath}`);
+```
+
+## SFTP Data Source
+
+Direct SFTP file operations using `ssh2-sftp-client`. Works standalone without Versori dependencies.
+
+### SFTP Constructor
+
+```typescript
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const sftpConfig: SftpDataSourceConfig = {
+  type: 'SFTP_CSV', // or 'SFTP_JSON', 'SFTP_JSONL', 'SFTP_PARQUET'
+  connectionId: 'sftp-prod',
+  name: 'Production SFTP',
+  settings: {
+    // Direct connection credentials (no platform required)
+    host: 'sftp.example.com',
+    port: 22,
+    username: 'user',
+    password: 'password', // or use privateKey
+    privateKey: 'PEM string or Buffer', // Optional SSH key
+    passphrase: 'key passphrase', // Optional for encrypted keys
+    remotePath: '/data',
+    filePattern: '*.csv',
+
+    // Connection settings
+    connectionTimeout: 30000,
+    keepaliveInterval: 5000,
+    maxConnections: 5,
+
+    // Write operation settings
+    writeCreateDirectories: true,
+    writeOverwrite: false,
+    writePermissions: '0644',
+  },
+};
+
+const sftp = new SftpDataSource(sftpConfig, logger);
+```
+
+**Configuration Notes:**
+
+`SftpDataSourceConfig` extends `DataSourceConfig`, which includes:
+
+- `type: string` - Data source type ('SFTP_CSV', 'SFTP_JSON', 'SFTP_JSONL', or 'SFTP_PARQUET')
+- `connectionId?: string` - Optional connection identifier
+- `name?: string` - Optional descriptive name
+- `connection?: SftpConnection` - **Optional** - Only required for Versori platform integration
+
+**Authentication Options:**
+
+- For **standalone use** (no Versori), provide `username` and either `password` or `privateKey` in settings
+- For **Versori platform integration**, provide `connection` object (credentials managed by platform)
+
+### Configuration Types
+
+The SFTP data source uses ssh2-sftp-client internally:
+
+```typescript
+interface SftpDataSourceConfig extends DataSourceConfig {
+  type: 'SFTP_CSV' | 'SFTP_JSON' | 'SFTP_JSONL' | 'SFTP_PARQUET';
+  connection?: SftpConnection; // Optional - only for Versori
+  settings: {
+    // Direct SFTP connection (no connection object needed)
+    host: string;
+    port?: number;
+    username?: string;
+    password?: string;
+    privateKey?: string | Buffer;
+    passphrase?: string;
+    remotePath: string;
+    filePattern: string;
+
+    // CSV settings
+    csvDelimiter?: string;
+    csvHeaders?: string[];
+    csvSkipEmptyLines?: boolean;
+    csvTrimValues?: boolean;
+    csvQuote?: string;
+    csvEscape?: string;
+
+    // JSON settings
+    jsonFormat?: 'json' | 'jsonl';
+    jsonValidate?: boolean;
+    jsonMaxDepth?: number;
+    jsonIndent?: number;
+    jsonSortKeys?: boolean;
+
+    // Parquet settings
+    parquetCompression?: 'none' | 'snappy' | 'gzip' | 'lz4';
+    parquetSchema?: Record<string, string>;
+    parquetRowGroupSize?: number;
+
+    // Write operation settings
+    writeCreateDirectories?: boolean;
+    writeOverwrite?: boolean;
+    writePermissions?: string;
+
+    // Connection settings
+    encoding?: string;
+    connectionTimeout?: number;
+    keepAliveInterval?: number;
+    maxConnections?: number;
+  };
+}
+```
+
+### SFTP File Listing
+
+**listFiles(params?)** - List files on SFTP server with pattern matching and timestamp filtering
+
+```typescript
+// List all files with default pattern
+const files = await sftp.listFiles();
+
+// List files with custom pattern (wildcard supported)
+const files = await sftp.listFiles({
+  filePattern: '*.csv',
+  remotePath: '/data/inventory',
+});
+
+// List files with wildcard patterns
+const files = await sftp.listFiles({
+  filePattern: 'products_*.csv', // Match: products_2025.csv, products_test.csv
+  remotePath: '/data',
+});
+
+// Incremental sync - files modified after timestamp
+const files = await sftp.listFiles({
+  lastProcessedTimestamp: '2025-01-01T00:00:00Z',
+  filePattern: '*.csv',
+});
+
+// Filter by file name substring
+const files = await sftp.listFiles({
+  fileName: 'inventory', // Matches files containing 'inventory'
+  remotePath: '/data',
+});
+
+// Returns: FileMetadata[]
+// [
+//   {
+//     path: '/data/inventory/products.csv',
+//     name: 'products.csv',
+//     size: 1024,
+//     lastModified: '2025-01-15T10:30:00Z',
+//     contentType: 'text/csv',
+//     source: 'SFTP'
+//   }
+// ]
+```
+
+**Parameters:**
+
+- `remotePath?: string` - Remote directory path (defaults to config remotePath)
+- `filePattern?: string` - Wildcard pattern for filtering (e.g., '\_.csv', 'products\_\_.json')
+- `lastProcessedTimestamp?: string` - ISO timestamp for incremental sync
+- `fileName?: string` - Filter by file name substring
+
+**Pattern Matching:**
+
+- `*.csv` - All CSV files
+- `products_*.csv` - CSV files starting with 'products\_'
+- `*_2025_*.json` - JSON files containing '_2025_'
+- Case-insensitive matching
+
+**Note:** Unlike S3, SFTP DOES support wildcard pattern matching natively.
+
+### SFTP File Download
+
+**downloadFile(path, options?)** - Download file content from SFTP server
+
+```typescript
+// Download as text (default UTF-8)
+const csvContent = await sftp.downloadFile('/data/inventory.csv');
+// Returns: string
+
+// Download as buffer (for binary files)
+const buffer = await sftp.downloadFile('/data/file.parquet', {
+  asBuffer: true,
+});
+// Returns: Buffer
+
+// Download with custom encoding
+const content = await sftp.downloadFile('/data/file.txt', {
+  encoding: 'latin1',
+});
+// Returns: string
+
+// Download with size limit
+const content = await sftp.downloadFile('/data/large-file.csv', {
+  maxSize: 10 * 1024 * 1024, // 10 MB limit
+});
+```
+
+**Parameters:**
+
+- `path: string` - Remote file path
+- `options.encoding?: string` - Text encoding (default: 'utf8')
+- `options.asBuffer?: boolean` - Return Buffer instead of string
+- `options.maxSize?: number` - Maximum file size in bytes
+
+**Returns:**
+
+- `Promise<string | Buffer>` - File content
+
+### File Existence Checks
+
+```typescript
+// Check if file exists
+const fileExists = await sftp.fileExists('/data/inventory.csv');
+
+// Check if directory exists
+const dirExists = await sftp.directoryExists('/data/exports');
+```
+
+### SFTP File Upload
+
+**uploadContent(content, remotePath, options?)** - Upload content directly to SFTP server
+
+```typescript
+// Upload text content
+const csvContent = 'sku,qty\nPROD-001,100\nPROD-002,50';
+await sftp.uploadFile('/data/inventory.csv', {
+  overwrite: true,
+  createDirectories: true,
+}, csvContent);
+
+// Upload binary content (Buffer)
+const parquetBuffer = await sftp.writeParquetContent(records);
+await sftp.uploadFile('/exports/data.parquet', {
+  overwrite: true,
+  encoding: 'binary',
+}, parquetBuffer);
+
+// Upload JSON with auto-directory creation
+const jsonData = [
+  { productRef: 'PROD-001', qty: 100 },
+  { productRef: 'PROD-002', qty: 50 },
+];
+await sftp.uploadFile(null, 2, JSON.stringify(jsonData), '/exports/2025/01/15/data.json', {
+  createDirectories: true, // Creates /exports/2025/01/15/ if needed
+  permissions: '0644',
+});
+```
+
+**Parameters:**
+
+- `content: string | Buffer` - Content to upload
+- `remotePath: string` - Destination file path
+- `options.overwrite?: boolean` - Overwrite if file exists (default: false)
+- `options.createDirectories?: boolean` - Create parent directories (default: false)
+- `options.permissions?: string` - File permissions (default: '0644')
+- `options.encoding?: string` - Text encoding (default: 'utf8')
+
+**uploadFile(localPath, remotePath, options?)** - Upload local file to SFTP server
+
+```typescript
+// Upload local file
+await sftp.uploadFile('/local/data/inventory.csv', '/remote/data/inventory.csv', {
+  overwrite: true,
+  createDirectories: true,
+  permissions: '0644',
+});
+```
+
+**Parameters:**
+
+- `localPath: string` - Local file path
+- `remotePath: string` - Remote destination path
+- `options` - Same as uploadContent
+
+### Directory Operations
+
+**createDirectory(remotePath, recursive, permissions?)** - Create directory on SFTP server
+
+```typescript
+// Create single directory
+await sftp.createDirectory('/data/exports', false);
+
+// Create directory tree
+await sftp.createDirectory('/data/exports/2025/01/15', true, '0755');
+
+// Check before creating
+if (!(await sftp.directoryExists('/data/archive'))) {
+  await sftp.createDirectory('/data/archive', true);
+}
+```
+
+**Parameters:**
+
+- `remotePath: string` - Directory path to create
+- `recursive: boolean` - Create parent directories (default: false)
+- `permissions?: string` - Directory permissions (default: '0755')
+
+### SFTP File Deletion
+
+**deleteFile(remotePath)** - Delete file from SFTP server
+
+```typescript
+// Delete single file
+await sftp.deleteFile('/temp/processing.csv');
+
+// Delete with existence check
+if (await sftp.fileExists('/temp/old-file.csv')) {
+  await sftp.deleteFile('/temp/old-file.csv');
+}
+
+// Bulk delete
+const tempFiles = await sftp.listFiles({ remotePath: '/temp', filePattern: '*.tmp' });
+for (const file of tempFiles) {
+  await sftp.deleteFile(file.path);
+}
+```
+
+**Parameters:**
+
+- `remotePath: string` - File path to delete
+
+**Throws:** Error if file does not exist
+
+### File Move (Atomic)
+
+**moveFile(oldPath, newPath, overwrite?)** - Move/rename file on SFTP server
+
+```typescript
+// Move file between directories
+await sftp.moveFile('/data/processing.csv', '/data/archive/processed.csv', false);
+
+// Rename file in same directory
+await sftp.moveFile(
+  '/data/temp_file.csv',
+  '/data/inventory_2025.csv',
+  true // overwrite if exists
+);
+
+// Archive processed files
+const files = await sftp.listFiles({ remotePath: '/data/inbox' });
+for (const file of files) {
+  const archivePath = `/data/archive/${file.name}`;
+  await sftp.moveFile(file.path, archivePath, false);
+}
+```
+
+**Parameters:**
+
+- `oldPath: string` - Current file path
+- `newPath: string` - New file path
+- `overwrite?: boolean` - Overwrite destination (default: false)
+
+**Important:** SFTP moveFile() is ATOMIC - single server-side operation (unlike S3)
+
+### SFTP File Copy
+
+**copyFile(sourcePath, destPath, overwrite?)** - Copy file on SFTP server
+
+```typescript
+// Create backup
+await sftp.copyFile('/data/inventory.csv', '/backup/inventory_backup.csv', true);
+
+// Copy to multiple locations
+const sourceFile = '/data/master.csv';
+const destinations = ['/backup/master.csv', '/archive/2025/master.csv', '/staging/master.csv'];
+
+for (const dest of destinations) {
+  await sftp.copyFile(sourceFile, dest, true);
+}
+```
+
+**Parameters:**
+
+- `sourcePath: string` - Source file path
+- `destPath: string` - Destination file path
+- `overwrite?: boolean` - Overwrite destination (default: false)
+
+**Note:** Implemented as download + upload (SFTP protocol has no native copy command)
+
+### SFTP File Archive
+
+**archiveFile(sourcePath, archivePrefix?)** - Archive file with timestamp (template method from AbstractDataSource)
+
+```typescript
+// Archive to default 'archive/' prefix
+const archivePath = await sftp.archiveFile('/data/inventory.csv');
+// Returns: 'archive/2025-10-29T12-30-45-123Z_inventory.csv'
+
+// Archive to custom prefix
+const archivePath = await sftp.archiveFile('/data/inventory.csv', '/backup/2025/');
+// Returns: '/backup/2025/2025-10-29T12-30-45-123Z_inventory.csv'
+
+// Archive processed files after ingestion
+const files = await sftp.listFiles({ remotePath: '/data/inbox' });
+for (const file of files) {
+  await processFile(file);
+  const archivePath = await sftp.archiveFile(file.path, '/data/archive/');
+  console.log(`Archived ${file.name} to ${archivePath}`);
+}
+```
+
+**Parameters:**
+
+- `sourcePath: string` - Source file path
+- `archivePrefix?: string` - Archive directory prefix (default: 'archive/')
+
+**Returns:** `Promise<string>` - The archive path where file was moved
+
+**How It Works:**
+1. Generates timestamped filename: `{archivePrefix}{timestamp}_{originalFilename}`
+2. Moves file to archive location using moveFile()
+3. Returns the full archive path
+
+**Important:** SFTP archiveFile() is atomic since it uses SFTP's atomic moveFile() operation.
+
+### Multi-Format Support
+
+#### CSV Operations
+
+```typescript
+// Write CSV content with custom options
+const csvRecords = [
+  { sku: 'PROD-001', qty: 100, location: 'DC-001' },
+  { sku: 'PROD-002', qty: 50, location: 'DC-002' },
+];
+
+const csvContent = sftp.writeCsvContent(csvRecords, {
+  delimiter: ',',
+  headers: ['sku', 'qty', 'location'],
+  includeHeader: true,
+  quote: '"',
+  escape: '\\',
+  encoding: 'utf8',
+});
+
+// Upload CSV content
+await sftp.uploadFile('/data/inventory.csv', csvContent);
+```
+
+#### JSON Operations
+
+```typescript
+// Write JSON content
+const jsonRecords = [
+  { productRef: 'PROD-001', quantity: 100 },
+  { productRef: 'PROD-002', quantity: 50 },
+];
+
+const jsonContent = sftp.writeJsonContent(jsonRecords, {
+  format: 'json', // or 'jsonl' for JSON Lines
+  indent: 2,
+  sortKeys: true,
+  replacer: (key, value) => (key === 'quantity' ? Number(value) : value),
+});
+
+// Upload JSON content
+await sftp.uploadFile('/data/products.json', jsonContent);
+
+// Parse JSON content from downloaded file
+const downloadedJson = await sftp.downloadFile('products.json');
+const parsedRecords = sftp.parseJsonContent(downloadedJson, {
+  format: 'json',
+  validate: true,
+  maxDepth: 10,
+});
+```
+
+#### JSON Lines Support
+
+```typescript
+// Write JSON Lines (JSONL) format
+const jsonlContent = sftp.writeJsonContent(records, {
+  format: 'jsonl',
+});
+// Produces:
+// {"id":1,"name":"Product A"}
+// {"id":2,"name":"Product B"}
+
+// Parse JSONL content
+const jsonlRecords = sftp.parseJsonContent(jsonlContent, {
+  format: 'jsonl',
+});
+```
+
+**Parquet Operations (SFTP)**
+
+- Read/parse Parquet: Not implemented yet in SFTP data source. Attempting to call `parseParquetContent` will throw until hyparquet integration is added for SFTP. Use S3→download→parsers for Parquet reads.
+- Write Parquet: Implemented via `writeParquetContent(records, options)` producing a Buffer that you can upload with `uploadContent`.
+
+```typescript
+// Write Parquet content (supported)
+const parquetBuffer = await sftp.writeParquetContent(records, {
+  compression: 'gzip',
+  parquetSchema: {
+    productRef: 'string',
+    quantity: 'int32',
+    price: 'double',
+  },
+  parquetRowGroupSize: 1000,
+});
+await sftp.uploadFile('/exports/data.parquet', { overwrite: true }, parquetBuffer);
+
+// Read Parquet content (not yet supported on SFTP data source)
+// const records = await sftp.parseParquetContent(buffer, fileMetadata); // ❌ throws
+```
+
+### Error Handling
+
+The SFTP data source provides complete error handling:
+
+```typescript
+try {
+  await sftp.uploadFile(localPath, remotePath);
+} catch (error) {
+  if (error instanceof DataSourceError) {
+    console.error(`SFTP Error [${error.code}]:`, error.message);
+    console.error('Context:', error.context);
+    console.error('Operation:', error.operation);
+  }
+}
+```
+
+### Connection Management
+
+The SFTP data source includes sophisticated connection management:
+
+```typescript
+// Get connection information
+const connectionInfo = sftp.getConnectionInfo();
+console.log('Pool size:', connectionInfo.connectionPoolSize);
+console.log('Max pool size:', connectionInfo.maxPoolSize);
+
+// Connection pooling is handled automatically
+// Multiple operations reuse connections efficiently
+```
+
+## Complete Methods Reference
+
+### S3 Data Source Methods
+
+**File Operations:**
+
+- `listFiles(params?): Promise<FileMetadata[]>` - List objects with filtering
+- `downloadFile(key, options?): Promise<string | Buffer>` - Download object content
+- `uploadFile(key, content, options?): Promise<void>` - Upload object (required abstract method)
+- `deleteFile(key): Promise<void>` - Delete object
+- `copyFile(sourceKey, destKey, options?): Promise<void>` - Copy object (supports cross-bucket)
+- `moveFile(sourcePath, destPath, deleteSource?): Promise<void>` - Move object within same bucket (template method)
+- `archiveFile(sourcePath, archivePrefix?): Promise<string>` - Archive file with timestamp (template method)
+- `validateConnection(): Promise<boolean>` - Test bucket access
+- `writeParquetContent(records, options?): Promise<Buffer>` - Write Parquet format
+
+### SFTP Data Source Methods
+
+**File Operations:**
+
+- `listFiles(params?): Promise<FileMetadata[]>` - List files with wildcard pattern matching
+- `downloadFile(path, options?): Promise<string | Buffer>` - Download file content
+- `uploadFile(remotePath, content, options?): Promise<void>` - Upload content (required abstract method)
+- `deleteFile(remotePath): Promise<void>` - Delete file
+- `moveFile(oldPath, newPath, overwrite?): Promise<void>` - Move/rename file (atomic, template method)
+- `copyFile(sourcePath, destPath, overwrite?): Promise<void>` - Copy file
+- `archiveFile(sourcePath, archivePrefix?): Promise<string>` - Archive file with timestamp (template method)
+- `fileExists(remotePath): Promise<boolean>` - Check if file exists
+- `directoryExists(remotePath): Promise<boolean>` - Check if directory exists
+- `createDirectory(remotePath, recursive, permissions?): Promise<void>` - Create directory
+- `validateConnection(): Promise<boolean>` - Test SFTP connection
+- `dispose(): Promise<void>` - Clean up connections
+
+**Format Operations:**
+
+- `parseCsvContent(csvContent, fileMetadata): CsvRecord[]` - Parse CSV string
+- `writeCsvContent(records, options?): string` - Generate CSV string
+- `parseJsonContent(jsonContent, fileMetadata, options?): Record<string, unknown>[]` - Parse JSON/JSONL
+- `writeJsonContent(records, options?): string` - Generate JSON/JSONL string
+- `parseParquetContent(buffer, fileMetadata): Promise<ParquetDataRecord[]>` - Not supported; currently throws
+- `writeParquetContent(records, options?): Promise<Buffer>` - Generate Parquet buffer
+
+**Utility Methods:**
+
+- `getConnectionInfo(): ConnectionInfo` - Get connection pool status
+
+## See Also
+
+- [Module 07: Parsers](./api-reference-07-parsers.md) - Format-specific parsing
+- [Module 05: Extraction](../../extraction/extraction-readme.md) - Using data sources for extraction
+- [Module 10: Utilities](./api-reference-10-utilities.md) - Helper functions
+- [Getting Started Guide](../../../00-START-HERE/getting-started.md) - Basic setup
+
+---
+
+**[← Previous: Services](./api-reference-05-services.md)** | **[API Reference Home](../api-reference-readme.md)** | **[Next: Parsers →](./api-reference-07-parsers.md)**