@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-02-s3-operations.md

@@ -1,1302 +1,1302 @@
-# Module 2: S3 Operations
-
-Complete guide to S3DataSource - presigned URL architecture, file operations, streaming, and production patterns.
-
-## Table of Contents
-
-- [Installation](#installation)
-- [Presigned URL Architecture](#presigned-url-architecture)
-- [Configuration](#configuration)
-- [File Operations](#file-operations)
-- [Parquet Generation](#parquet-generation)
-- [Streaming Patterns](#streaming-patterns)
-- [Platform Integration](#platform-integration)
-- [Production Patterns](#production-patterns)
-- [TypeScript Types](#typescript-types)
-- [Testing](#testing)
-- [Troubleshooting](#troubleshooting)
-- [Best Practices](#best-practices)
-- [Limitations](#limitations)
-- [API Reference](#api-reference)
-
----
-
-## Installation
-
-Install the SDK which includes S3DataSource and all dependencies:
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk
-```
-
-**Dependencies**: The SDK uses AWS SDK v3 (^3.879.0) internally for presigned URL signature generation via `S3PresignService`, but `S3DataSource` itself uses native `fetch` for all S3 operations, making it compatible with Node.js, Deno, and Versori.
-
----
-
-## Presigned URL Architecture
-
-### How S3DataSource Works
-
-`S3DataSource` uses a **presigned-URL-only architecture** for universal platform compatibility:
-
-```
-┌──────────────────────────────────────────────────────────┐
-│ S3DataSource (fc-connect-sdk)                             │
-└──────────────────┬───────────────────────────────────────┘
-                   │
-                   ▼
-         ┌─────────────────────┐
-         │ S3PresignService    │
-         │ (AWS SigV4 signing) │
-         └────────┬────────────┘
-                  │
-                  ▼
-    Generate Presigned URL (1-hour expiry)
-    https://bucket.s3.region.amazonaws.com/key?
-       X-Amz-Algorithm=AWS4-HMAC-SHA256
-       &X-Amz-Credential=...
-       &X-Amz-Signature=...
-                  │
-                  ▼
-         ┌─────────────────────┐
-         │ Native fetch()      │
-         │ (HTTP GET/PUT/...)  │
-         └────────┬────────────┘
-                  │
-                  ▼
-         ┌─────────────────────┐
-         │ AWS S3 Bucket       │
-         │ (validates sig)     │
-         └─────────────────────┘
-```
-
-**Key Benefits**:
-
-1. **No AWS SDK bundling** - Smaller package size, faster cold starts
-2. **Universal compatibility** - Works in Node.js, Deno, Versori without changes
-3. **Simple HTTP** - Standard `fetch()` API, easy to debug
-4. **Stateless** - No connection pooling or state management
-5. **Proxy-friendly** - Works with corporate proxies and firewalls
-
-### Signature Generation Process
-
-```typescript
-// How presigned URLs are generated internally:
-// 1. Create canonical request (method, path, query, headers)
-// 2. Sign with AWS SigV4 using access key + secret key
-// 3. Add signature to query string
-// 4. URL valid for 1 hour (default)
-
-// Example presigned GET URL:
-https://my-bucket.s3.us-east-1.amazonaws.com/inventory/file.csv?
-  X-Amz-Algorithm=AWS4-HMAC-SHA256&
-  X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20250115%2Fus-east-1%2Fs3%2Faws4_request&
-  X-Amz-Date=20250115T120000Z&
-  X-Amz-Expires=3600&
-  X-Amz-SignedHeaders=host&
-  X-Amz-Signature=abc123...
-```
-
-**Security Notes**:
-
-- Presigned URLs expire after 1 hour (3600 seconds)
-- Anyone with the URL can access the file during expiry window
-- Signatures are cryptographically secure (SHA256 HMAC)
-- No credentials exposed in URL (only signature)
-
----
-
-## Configuration
-
-### Basic Configuration
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger,
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info',
-});
-
-// Minimal S3 configuration
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
-    connectionId: 's3-minimal',
-    name: 'S3 Minimal',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-bucket', // Required
-    },
-  },
-  logger
-);
-```
-
-### Configuration Options
-
-| Property                   | Required | Description                         | Default     | Example                   |
-| -------------------------- | -------- | ----------------------------------- | ----------- | ------------------------- |
-| `type`                     | ✅       | Data source type                    | -           | `'S3_CSV'`                |
-| `s3Config.accessKeyId`     | ✅       | AWS access key ID                   | -           | `'AKIAIOSFODNN7...'`      |
-| `s3Config.secretAccessKey` | ✅       | AWS secret access key               | -           | `'wJalrXUtnFEMI/...'`     |
-| `s3Config.region`          | ✅       | AWS region                          | -           | `'us-east-1'`             |
-| `s3Config.bucket`          | ❌       | Default bucket name                 | -           | `'inventory-data'`        |
-| `s3Config.sessionToken`    | ❌       | STS session token (temporary creds) | -           | `'FwoGZXIv...'`           |
-| `s3Config.endpoint`        | ❌       | Custom S3 endpoint (for LocalStack) | AWS default | `'http://localhost:4566'` |
-| `s3Config.maxAttempts`     | ❌       | Max retry attempts       | `3`         | `5`                       |
-
-### Enhanced Retry Logic
-
-S3DataSource includes automatic retry with exponential backoff for all HTTP operations.
-
-```typescript
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-retry',
-    name: 'S3 Retry',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-bucket',
-      maxAttempts: 5, // Default: 3, increase for unreliable networks
-    },
-  },
-  logger
-);
-
-// All operations automatically retry on failure:
-await s3Source.downloadFile('data.csv'); // Retries on 429, 500, 502, 503, 504, network errors
-await s3Source.uploadFile('output.json', data); // Retries automatically
-await s3Source.listFiles({ prefix: 'inventory/' }); // Retries automatically
-```
-
-**Retry Behavior:**
-
-| Attempt              | Delay  | Total Wait | Retryable Conditions                         |
-| -------------------- | ------ | ---------- | -------------------------------------------- |
-| 1                    | 0ms    | 0ms        | HTTP 429, 500, 502, 503, 504, network errors |
-| 2                    | 500ms  | 500ms      | Same conditions                              |
-| 3 (default max)      | 1000ms | 1.5s       | Same conditions                              |
-| 4 (if maxAttempts=4) | 2000ms | 3.5s       | Same conditions                              |
-| 5 (if maxAttempts=5) | 4000ms | 7.5s       | Same conditions                              |
-
-**Backoff Formula:**
-
-- Base delay: 500ms
-- Factor: 2 (exponential)
-- Max delay per attempt: 8000ms (8 seconds)
-- Formula: `min(8000ms, 500ms * 2^(attempt-1))`
-
-**What Gets Retried:**
-
-- ✅ Rate limiting (429 Too Many Requests)
-- ✅ Server errors (500, 502, 503, 504)
-- ✅ Network failures (connection timeout, DNS errors)
-- ✅ Transient S3 errors
-
-**What Does NOT Get Retried:**
-
-- ❌ Client errors (400, 403, 404) - permanent failures
-- ❌ Authentication errors (401) - credential issues
-- ❌ Successful responses (2xx)
-
-**Logging:**
-
-```typescript
-// Retry attempts are automatically logged at WARN level
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'debug',
-});
-const s3Source = new S3DataSource(config, logger);
-
-await s3Source.downloadFile('data.csv');
-// Logs on retry:
-// - "S3 HTTP retry due to status" (for 429, 500, etc.)
-// - "S3 HTTP retry due to network error" (for connection failures)
-// - Includes: operationName, status, attempt, delay
-```
-
-**Production Recommendation:**
-
-```typescript
-// For production, increase maxAttempts for better resilience
-const productionConfig = {
-  type: 'S3_CSV',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1',
-    bucket: 'production-bucket',
-    maxAttempts: 5, // 5 attempts = up to 7.5s of automatic retries
-  },
-};
-```
-
-### Environment Variables Pattern
-
-```bash
-# .env file
-AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
-AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-AWS_REGION=us-east-1
-AWS_SESSION_TOKEN=FwoGZXIv...  # Optional
-SOURCE_BUCKET=inventory-source
-```
-
-```typescript
-// Load from environment
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-env',
-    name: 'S3 Environment',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION || 'us-east-1',
-      bucket: process.env.SOURCE_BUCKET,
-      sessionToken: process.env.AWS_SESSION_TOKEN, // Optional
-    },
-  },
-  logger
-);
-```
-
-### Dual AWS Credentials (Source + Target)
-
-Use separate AWS accounts for reading (source) and writing (target):
-
-```typescript
-// Source S3 (read inventory from vendor's AWS account)
-const sourceS3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-source-vendor',
-    name: 'S3 Source Vendor',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: process.env.AWS_REGION || 'us-east-1',
-      bucket: process.env.SOURCE_BUCKET,
-    },
-  },
-  logger
-);
-
-// Target S3 (write extraction results to our data lake)
-const targetS3 = new S3DataSource(
-  {
-    type: 'S3_PARQUET',
-    connectionId: 's3-target-own',
-    name: 'S3 Target Own',
-    s3Config: {
-      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
-      region: process.env.TARGET_AWS_REGION || 'us-west-2',
-      bucket: process.env.TARGET_BUCKET,
-    },
-  },
-  logger
-);
-
-// Usage: Read from vendor, write to our data lake
-const inventory = await sourceS3.downloadFile('s3://vendor-bucket/inventory.csv');
-const processed = await processInventory(inventory);
-await targetS3.uploadFile('s3://our-datalake/processed.parquet', processed);
-```
-
----
-
-## File Operations
-
-### Listing Files
-
-```typescript
-// List all files in bucket
-const allFiles = await s3Source.listFiles();
-
-// List files with prefix (folder-like structure)
-const inventoryFiles = await s3Source.listFiles({
-  prefix: 'inventory/',
-  maxKeys: 100,
-});
-
-// List with delimiter (folders)
-const topLevel = await s3Source.listFiles({
-  prefix: 'data/',
-  delimiter: '/',
-  maxKeys: 1000,
-});
-
-// Process file metadata
-for (const file of inventoryFiles) {
-  console.log({
-    path: file.path,
-    name: file.name,
-    size: file.size,
-    lastModified: file.lastModified,
-    contentType: file.contentType,
-    etag: file.etag,
-    bucket: file.bucket,
-    region: file.region,
-  });
-}
-```
-
-**List Options**:
-
-| Option              | Type     | Description                                        |
-| ------------------- | -------- | -------------------------------------------------- |
-| `prefix`            | `string` | Filter files by prefix (e.g., `'inventory/'`)      |
-| `delimiter`         | `string` | Delimiter for folder-like structure (e.g., `'/'`)  |
-| `maxKeys`           | `number` | Maximum files to return (default: 1000, max: 1000) |
-| `continuationToken` | `string` | Token for pagination (from previous response)      |
-
-**Example: List files modified in last 24 hours**
-
-```typescript
-const files = await s3Source.listFiles({ prefix: 'incoming/' });
-
-const yesterday = new Date(Date.now() - 24 * 60 * 60 * 1000);
-const recentFiles = files.filter(file => file.lastModified > yesterday);
-
-console.log(`${recentFiles.length} files modified in last 24 hours`);
-```
-
-### Reading Files
-
-```typescript
-// Read text file (CSV, JSON, etc.)
-const csvContent = await s3Source.downloadFile('inventory.csv');
-// Returns: string
-
-// Read binary file (Parquet, images, etc.)
-const parquetData = await s3Source.downloadFile('data.parquet', { encoding: 'binary' });
-// Returns: Buffer
-
-// Read JSON with parsing
-const jsonContent = await s3Source.downloadFile('config.json');
-const config = JSON.parse(jsonContent as string);
-
-// For existence checks, list by prefix and filter, or handle 404 on download
-if (await s3Source.listFiles({ prefix: 'file.csv' }).then(files => files.length > 0)) {
-  const content = await s3Source.downloadFile('file.csv');
-}
-```
-
-**Read Options**:
-
-| Option            | Type                     | Description             | Default  |
-| ----------------- | ------------------------ | ----------------------- | -------- |
-| `encoding`        | `'utf8' \| 'binary'`     | Text or binary encoding | `'utf8'` |
-| `responseHeaders` | `Record<string, string>` | Custom HTTP headers     | `{}`     |
-
-**Example: Read with error handling**
-
-```typescript
-try {
-  const content = await s3Source.downloadFile('file.csv');
-  console.log('File read successfully');
-} catch (error: any) {
-  if (error.context?.statusCode === 404) {
-    console.log('File not found');
-  } else if (error.context?.statusCode === 403) {
-    console.log('Access denied - check IAM permissions');
-  } else {
-    console.error('S3 error:', error.message);
-  }
-}
-```
-
-### Writing Files
-
-```typescript
-// Write CSV file
-await s3Source.uploadFile('output/results.csv', csvData, { contentType: 'text/csv' });
-
-// Write JSON with metadata
-await s3Source.uploadFile('output/results.json', JSON.stringify(data, null, 2), {
-  contentType: 'application/json',
-  metadata: {
-    'processed-date': new Date().toISOString(),
-    version: '1.0',
-    processor: 'fc-connect-sdk',
-    'record-count': data.length.toString(),
-  },
-});
-
-// Write binary file (Parquet, images, etc.)
-await s3Source.uploadFile('output/data.parquet', parquetBuffer, {
-  contentType: 'application/octet-stream',
-});
-
-// Automatic content type detection from extension
-await s3Source.uploadFile('file.csv', csvData);
-// Automatically sets contentType: 'text/csv'
-
-await s3Source.uploadFile('file.json', jsonData);
-// Automatically sets contentType: 'application/json'
-```
-
-**Write Options**:
-
-| Option            | Type                     | Description                                              |
-| ----------------- | ------------------------ | -------------------------------------------------------- |
-| `contentType`     | `string`                 | MIME type (auto-detected from extension if not provided) |
-| `metadata`        | `Record<string, string>` | Custom S3 object metadata (key-value pairs)              |
-| `cacheControl`    | `string`                 | HTTP Cache-Control header                                |
-| `contentEncoding` | `string`                 | HTTP Content-Encoding header (e.g., `'gzip'`)            |
-
-### Deleting Files
-
-```typescript
-// Delete single file
-await s3Source.deleteFile('temp/old-file.csv');
-
-// Delete with error handling
-try {
-  await s3Source.deleteFile('important-file.json');
-  console.log('File deleted successfully');
-} catch (error: any) {
-  if (error.context?.statusCode === 404) {
-    console.log('File already deleted or does not exist');
-  } else {
-    console.error('Delete failed:', error.message);
-  }
-}
-
-// Delete multiple files (batch)
-const filesToDelete = [
-  's3://bucket/temp/file1.csv',
-  's3://bucket/temp/file2.csv',
-  's3://bucket/temp/file3.csv',
-];
-
-for (const file of filesToDelete) {
-  try {
-    await s3Source.deleteFile(file);
-  } catch (error) {
-    console.error(`Failed to delete ${file}:`, error);
-  }
-}
-```
-
-### Copying Files
-
-```typescript
-// Copy within same bucket
-await s3Source.copyFile('source/data.csv', 'archive/data.csv');
-
-// Copy from different bucket
-await s3Source.copyFile('file.json', 'backup/file.json', { sourceBucket: 'source-bucket' });
-
-// Copy with metadata
-await s3Source.copyFile('source.csv', 'archive/source.csv', {
-  metadata: {
-    'copied-from': 'source.csv',
-    'copy-date': new Date().toISOString(),
-    'archived-by': 'fc-connect-sdk',
-  },
-});
-```
-
-### Moving Files (Copy + Delete)
-
-```typescript
-// Move file within bucket
-await s3Source.moveFile('incoming/data.csv', 'processed/data.csv');
-
-// Move to archive with date-based organization
-const today = new Date().toISOString().split('T')[0];
-await s3Source.moveFile('incoming/inventory.csv', `archive/${today}/inventory.csv`);
-```
-
-**IMPORTANT**: `moveFile()` is implemented as copy + delete, which is **NOT atomic**. If delete fails, the file exists in both locations. For critical operations:
-
-```typescript
-// Manual atomic-like move with verification
-try {
-  // Copy
-  await s3Source.copyFile(sourcePath, targetPath);
-
-  // Verify copy succeeded
-  const exists = await s3Source.listFiles({ prefix: targetPath }).then(files => files.length > 0);
-  if (!exists) {
-    throw new Error('Copy verification failed');
-  }
-
-  // Delete original
-  await s3Source.deleteFile(sourcePath);
-} catch (error) {
-  console.error('Move failed:', error);
-  // Cleanup: delete target if it exists
-  try {
-    await s3Source.deleteFile(targetPath);
-  } catch {}
-}
-```
-
-### Checking File Existence
-
-```typescript
-// Use listFiles + filter or handle 404 on download to determine existence
-if (await s3Source.listFiles({ prefix: 'config.json' }).then(files => files.length > 0)) {
-  const config = await s3Source.downloadFile('config.json');
-} else {
-  console.log('Config file not found, using defaults');
-  const config = defaultConfig;
-}
-```
-
----
-
-## Parquet Generation
-
-`S3DataSource` includes built-in Parquet file generation for efficient data storage and analytics:
-
-### Basic Parquet Generation
-
-```typescript
-import { S3DataSource, ParquetDataRecord } from '@fluentcommerce/fc-connect-sdk';
-
-// Prepare data records
-const records: ParquetDataRecord[] = [
-  { id: 1, sku: 'PROD-001', quantity: 100, price: 29.99, available: true },
-  { id: 2, sku: 'PROD-002', quantity: 200, price: 39.99, available: false },
-  { id: 3, sku: 'PROD-003', quantity: 150, price: 49.99, available: true },
-];
-
-// Generate Parquet buffer
-const parquetBuffer = await s3Source.writeParquetContent(records, {
-  compression: 'GZIP', // Options: UNCOMPRESSED, GZIP, SNAPPY, LZO, BROTLI
-  rowGroupSize: 50000, // Rows per group (default: 50000)
-});
-
-// Upload to S3
-await s3Source.uploadFile('s3://bucket/output/inventory.parquet', parquetBuffer, {
-  contentType: 'application/octet-stream',
-});
-```
-
-### Automatic Schema Inference
-
-Parquet schema is automatically inferred from record structure:
-
-```typescript
-// JavaScript type → Parquet type mapping
-const records = [
-  {
-    id: 1, // number (int) → INT64
-    sku: 'PROD-001', // string → UTF8
-    quantity: 100, // number (int) → INT64
-    price: 29.99, // number (float) → DOUBLE
-    available: true, // boolean → BOOLEAN
-    lastUpdated: new Date(), // Date → TIMESTAMP_MILLIS
-    metadata: { tags: ['new', 'hot'] }, // object → UTF8 (JSON serialized)
-  },
-];
-
-const buffer = await s3Source.writeParquetContent(records);
-```
-
-**Type Mapping**:
-
-- `string` → UTF8
-- `number` (integer) → INT64
-- `number` (float/decimal) → DOUBLE
-- `boolean` → BOOLEAN
-- `Date` → TIMESTAMP_MILLIS
-- `object` → UTF8 (JSON serialized)
-- `null` / `undefined` → NULL
-
-### Compression Options
-
-```typescript
-// No compression (fastest, largest file)
-const uncompressed = await s3Source.writeParquetContent(records, {
-  compression: 'UNCOMPRESSED',
-});
-
-// GZIP compression (good balance, widely supported)
-const gzipped = await s3Source.writeParquetContent(records, {
-  compression: 'GZIP',
-});
-
-// SNAPPY compression (fastest compression, good for analytics)
-const snappy = await s3Source.writeParquetContent(records, {
-  compression: 'SNAPPY',
-});
-
-// BROTLI compression (best compression ratio)
-const brotli = await s3Source.writeParquetContent(records, {
-  compression: 'BROTLI',
-});
-```
-
-**Compression Comparison**:
-
-| Algorithm    | Speed     | Compression Ratio | Use Case                      |
-| ------------ | --------- | ----------------- | ----------------------------- |
-| UNCOMPRESSED | Fastest   | 1:1               | Small files, temp data        |
-| SNAPPY       | Very fast | 2:1 to 3:1        | Analytics, real-time queries  |
-| GZIP         | Medium    | 4:1 to 6:1        | General purpose, wide support |
-| BROTLI       | Slowest   | 5:1 to 8:1        | Long-term storage, archival   |
-
-### Complex Object Handling
-
-```typescript
-// Complex nested objects are automatically serialized to JSON
-const records = [
-  {
-    id: 1,
-    productDetails: {
-      name: 'Product A',
-      category: 'Electronics',
-      attributes: { color: 'black', size: 'M' },
-    },
-    tags: ['new', 'featured', 'sale'],
-    timestamp: new Date(),
-  },
-];
-
-const buffer = await s3Source.writeParquetContent(records);
-// productDetails and tags are stored as JSON strings in Parquet
-```
-
-### Empty Record Handling
-
-```typescript
-// Empty arrays return minimal valid Parquet file
-const emptyBuffer = await s3Source.writeParquetContent([], {
-  compression: 'GZIP',
-});
-
-// Still a valid Parquet file (can be opened by analytics tools)
-await s3Source.uploadFile('s3://bucket/empty.parquet', emptyBuffer);
-```
-
----
-
-## Streaming Patterns
-
-### Processing Large Files
-
-```typescript
-// Download large file in chunks to minimize memory usage
-async function processLargeFile(dataSource: S3DataSource, key: string) {
-  // Download file (S3DataSource buffers in memory)
-  const content = await dataSource.downloadFile(key, { encoding: 'binary' });
-
-  // Process in chunks
-  const chunkSize = 1024 * 1024; // 1MB chunks
-  let offset = 0;
-
-  while (offset < content.length) {
-    const chunk = (content as Buffer).slice(offset, offset + chunkSize);
-    await processChunk(chunk);
-    offset += chunkSize;
-  }
-}
-
-async function processChunk(chunk: Buffer) {
-  console.log(`Processing ${chunk.length} bytes`);
-  // Your processing logic here
-}
-```
-
-**Note**: For very large files (>100MB), consider using S3 Select or reading directly with AWS SDK if memory is constrained.
-
----
-
-## Platform Integration
-
-### Node.js
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger,
-} from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info',
-});
-
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-      region: 'us-east-1',
-      bucket: 'my-bucket',
-    },
-  },
-  logger
-);
-```
-
-### Deno
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger,
-} from 'npm:@fluentcommerce/fc-connect-sdk@latest';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'info',
-});
-
-const s3Source = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    s3Config: {
-      accessKeyId: Deno.env.get('AWS_ACCESS_KEY_ID')!,
-      secretAccessKey: Deno.env.get('AWS_SECRET_ACCESS_KEY')!,
-      region: 'us-east-1',
-      bucket: 'my-bucket',
-    },
-  },
-  logger
-);
-```
-
-### Versori
-
-```typescript
-import {
-  S3DataSource,
-  createConsoleLogger,
-  toStructuredLogger,
-} from '@fluentcommerce/fc-connect-sdk';
-import { http } from '@versori/run';
-
-export const processFiles = http('s3-process', async ({ activation, log }) => {
-  const logger = toStructuredLogger(createConsoleLogger(), {
-    logLevel: 'info',
-  });
-
-  const s3Source = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      s3Config: {
-        accessKeyId: activation.env.AWS_ACCESS_KEY_ID,
-        secretAccessKey: activation.env.AWS_SECRET_ACCESS_KEY,
-        region: activation.env.AWS_REGION,
-        bucket: activation.env.S3_BUCKET,
-      },
-    },
-    logger
-  );
-
-  const files = await s3Source.listFiles({ prefix: 'inventory/' });
-  // Process files...
-});
-```
-
----
-
-## Production Patterns
-
-### Pattern 1: Inventory Ingestion from S3
-
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function ingestInventoryFromS3() {
-  const s3Source = new S3DataSource(
-    {
-      /* config */
-    },
-    logger
-  );
-  const csvParser = new CSVParserService();
-  const mapper = new UniversalMapper(mappingConfig);
-  const stateService = new StateService(kvAdapter);
-
-  // List inventory files
-  const files = await s3Source.listFiles({ prefix: 'inventory/' });
-
-  for (const file of files) {
-    // Check if already processed
-    const processed = await stateService.isFileProcessed(file.etag);
-    if (processed) {
-      console.log(`Skipping already processed file: ${file.name}`);
-      continue;
-    }
-
-    try {
-      // Download and parse CSV
-      const content = await s3Source.downloadFile(file.path);
-      const records = await csvParser.parse(content as string);
-
-      // Map fields
-      const result = await mapper.map(records);
-      if (!result.success) {
-        logger.error('Mapping failed', undefined, { errors: result.errors });
-        continue;
-      }
-
-      // Send to Fluent (using Batch API)
-      await sendToFluent(result.data);
-
-      // Mark as processed
-      await stateService.markFileProcessed(file.etag, {
-        fileName: file.name,
-        recordCount: records.length,
-        processedAt: new Date().toISOString(),
-      });
-
-      // Archive file
-      await s3Source.moveFile(file.path, file.path.replace('inventory/', 'processed/'));
-
-      logger.info('File processed successfully', { file: file.name });
-    } catch (error) {
-      logger.error('Failed to process file', error as Error, { file: file.path });
-    }
-  }
-}
-```
-
-### Pattern 2: Data Extraction to S3
-
-```typescript
-async function extractInventoryToS3(client: FluentClient, targetS3: S3DataSource) {
-  // Query Fluent inventory
-  const inventory = await client.graphql({
-    query: `query GetInventory {
-      virtualPositions(first: 1000) {
-        edges {
-          node {
-            ref
-            productRef
-            groupRef
-            quantity
-            status
-            updatedOn
-          }
-        }
-      }
-    }`,
-  });
-
-  const records = inventory.virtualPositions.edges.map(e => e.node);
-
-  // Generate Parquet
-  const parquetBuffer = await targetS3.writeParquetContent(records, {
-    compression: 'SNAPPY',
-    rowGroupSize: 50000,
-  });
-
-  // Upload with timestamp
-  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-  const outputPath = `s3://extraction-bucket/inventory/${timestamp}/full-export.parquet`;
-
-  await targetS3.uploadFile(outputPath, parquetBuffer, {
-    contentType: 'application/octet-stream',
-    metadata: {
-      'extraction-timestamp': timestamp,
-      'record-count': records.length.toString(),
-      compression: 'SNAPPY',
-    },
-  });
-
-  logger.info('Extraction completed', { path: outputPath, records: records.length });
-}
-```
-
----
-
-## TypeScript Types
-
-```typescript
-// S3 Configuration
-interface S3Config {
-  accessKeyId: string;
-  secretAccessKey: string;
-  region: string;
-  sessionToken?: string;
-  bucket?: string;
-  endpoint?: string;
-  forcePathStyle?: boolean;
-  maxAttempts?: number;
-}
-
-// S3DataSource Configuration
-interface S3DataSourceConfig {
-  type: 'S3_CSV' | 'S3_JSON' | 'S3_XML' | 'S3_PARQUET';
-  connectionId: string;
-  name: string;
-  s3Config: S3Config;
-}
-
-// File Metadata from S3DataSource
-interface FileMetadata {
-  path: string;
-  name: string;
-  size: number;
-  lastModified: Date;
-  contentType?: string;
-  etag?: string;
-  bucket: string;
-  region: string;
-}
-
-// List Options
-interface ListOptions {
-  prefix?: string;
-  delimiter?: string;
-  maxKeys?: number;
-  continuationToken?: string;
-}
-
-// Download Options
-interface DownloadOptions {
-  encoding?: 'utf8' | 'binary';
-  responseHeaders?: Record<string, string>;
-}
-
-// Upload Options
-interface UploadOptions {
-  contentType?: string;
-  metadata?: Record<string, string>;
-  cacheControl?: string;
-  contentEncoding?: string;
-}
-
-// Copy Options
-interface CopyOptions {
-  sourceBucket?: string;
-  metadata?: Record<string, string>;
-  metadataDirective?: 'COPY' | 'REPLACE';
-}
-
-// Parquet Options
-interface ParquetOptions {
-  compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
-  rowGroupSize?: number;
-  schema?: Record<string, string>;
-}
-
-// Parquet Data Record
-interface ParquetDataRecord {
-  [key: string]: string | number | boolean | Date | object | null;
-}
-```
-
----
-
-## Testing
-
-The SDK includes test utilities for validating S3 operations:
-
-```typescript
-import {
-  S3SDKTester,
-  S3PresignedTester,
-  S3ComparisonTester
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Test SDK implementation (if using S3Service)
-const sdkTester = new S3SDKTester(config);
-const sdkResults = await sdkTester.testAll('test-bucket');
-
-// Test Presigned implementation
-const presignedTester = new S3PresignedTester(config, fetch);
-const presignedResults = await presignedTester.testAll('test-bucket');
-
-// Compare both implementations
-const comparer = new S3ComparisonTester();
-const comparison = await comparer.compareImplementations(
-  config, 
-  'test-bucket'
-);
-console.log('Match rate:', comparison.summary.matchRate + '%');
-```
-
-**Note**: For `S3DataSource`, you typically don't need these testers since it only uses presigned URLs. Use these if you're testing `S3Service` with dual mode support.
-
-### Basic Connection Test
-
-```typescript
-// Simple connectivity test
-const isValid = await s3Source.validateConnection();
-if (!isValid) {
-  throw new Error('S3 connection failed - check credentials and region');
-}
-```
-
----
-
-## Troubleshooting
-
-### Common Issues and Solutions
-
-#### "S3DataSource requires explicit bucket configuration"
-
-```typescript
-// ❌ Wrong - missing bucket
-const dataSource = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-test',
-  name: 'S3 Test',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1'
-    // Missing: bucket
-  }
-}, logger);
-
-// ✅ Correct - includes bucket
-const dataSource = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-test',
-  name: 'S3 Test',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-east-1',
-    bucket: 'my-bucket' // Required
-  }
-}, logger);
-```
-
-#### Region Mismatch Errors
-
-```typescript
-// Ensure your client region matches the bucket region
-const dataSource = new S3DataSource({
-  type: 'S3_CSV',
-  connectionId: 's3-test',
-  name: 'S3 Test',
-  s3Config: {
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    region: 'us-west-2', // Must match bucket region
-    bucket: 'my-bucket'
-  }
-}, logger);
-```
-
-#### Need Multipart Upload Support
-
-```typescript
-// S3DataSource does not support multipart upload
-// For files > 100MB requiring multipart, use S3Service instead:
-import { S3Service } from '@fluentcommerce/fc-connect-sdk';
-
-const s3 = new S3Service({
-  accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-  region: 'us-east-1'
-});
-
-// Now multipart works
-if (s3.hasAdvancedFeatures()) {
-  await s3.uploadLargeFile('bucket', 'large-file.bin', buffer, {
-    partSize: 10 * 1024 * 1024
-  });
-}
-```
-
-#### Content Type Detection
-
-The S3DataSource automatically detects content types based on file extensions:
-
-```typescript
-// Automatic content type detection
-const contentTypes = {
-  '.csv': 'text/csv',
-  '.json': 'application/json',
-  '.xml': 'application/xml',
-  '.txt': 'text/plain',
-  '.parquet': 'application/octet-stream',
-  '.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
-  '.xls': 'application/vnd.ms-excel',
-  '.pdf': 'application/pdf',
-  '.zip': 'application/zip',
-  '.gz': 'application/gzip'
-};
-
-// Files are automatically assigned correct content type
-await s3DataSource.uploadFile('data.csv', csvContent);  // text/csv
-await s3DataSource.uploadFile('config.json', jsonContent);  // application/json
-```
-
-### When to Use S3DataSource vs S3Service
-
-| Use Case | Recommended Class | Reason |
-|----------|------------------|---------|
-| Standard file operations (read, write, delete) | `S3DataSource` | Simpler API, universal compatibility |
-| Node.js, Deno, or Versori connectors | `S3DataSource` | Works everywhere with presigned URLs |
-| Files < 100MB | `S3DataSource` | Efficient for standard file sizes |
-| Need metadata and content type support | `S3DataSource` | Full support via presigned URL headers |
-| Files > 100MB requiring multipart upload | `S3Service` | Has `uploadLargeFile()` method |
-| Need S3 Select (query CSV/JSON) | `S3Service` | Has `selectObjectContent()` method |
-| Need advanced AWS SDK features | `S3Service` | Direct SDK access |
-| Testing both implementations | `S3Service` | Has dual mode support |
-
-### Architecture Comparison
-
-**S3DataSource** (Recommended for most use cases):
-- ✅ Universal: Works in Node.js, Deno, Versori
-- ✅ Simple: Presigned URLs only, no mode switching
-- ✅ Lightweight: No AWS SDK bundled in application
-- ✅ Consistent: Same behavior everywhere
-- ❌ No multipart upload
-- ❌ No S3 Select
-
-**S3Service** (For advanced features):
-- ✅ Multipart upload for large files
-- ✅ S3 Select query support
-- ✅ Direct AWS SDK access
-- ✅ Dual mode (SDK or presigned)
-- ❌ More complex API
-- ❌ Larger bundle size (includes AWS SDK)
-
----
-
-## Best Practices
-
-### 1. Choose the Right Class
-
-- Use `S3DataSource` for standard file operations (most common)
-- Use `S3Service` only when you need multipart upload or S3 Select
-
-### 2. Handle Large Files Properly
-
-- `S3DataSource` works well for files up to 50MB
-- For files > 50MB, process in chunks to avoid memory issues
-- For files > 100MB requiring multipart, switch to `S3Service`
-
-### 3. Implement Proper Error Handling
-
-- Always catch and handle `DataSourceError`
-- Check `error.context.statusCode` for specific S3 errors (404, 403, etc.)
-- Built-in retry logic handles transient failures automatically
-- Log errors with context for debugging
-
-### 4. Optimize List Operations
-
-- Use `prefix` parameter to limit results to specific paths
-- Use `delimiter` for folder-like organization
-- Set appropriate `maxKeys` to control page size
-- Implement pagination with `continuationToken` for large buckets
-
-### 5. Security Best Practices
-
-- Never hardcode credentials in source code
-- Use environment variables for AWS credentials
-- Implement least-privilege IAM policies
-- Use separate credentials for source and target operations
-- Leverage session tokens for temporary credentials
-
-### 6. Performance Optimization
-
-- Implement parallel uploads/downloads with concurrency control (5-10 concurrent)
-- Use `encoding: 'binary'` for non-text files
-- Process files in chunks for very large datasets
-- Batch operations to reduce HTTP overhead
-
-### 7. Data Organization
-
-- Use prefixes to organize files (e.g., `inventory/2024/01/15/file.csv`)
-- Set appropriate content types - `S3DataSource` auto-detects from extensions
-- Add metadata to track processing status and timestamps
-- Clean up temporary files after processing
-
-### 8. Platform Integration
-
-- `S3DataSource` works identically in Node.js, Deno, and Versori
-- Optionally pass custom `httpClient` to constructor for connection pooling
-- Use structured logging to track operations across platforms
-
----
-
-## Limitations
-
-### S3DataSource Limitations
-
-**What S3DataSource Does NOT Support**:
-- ❌ **Multipart Upload**: Cannot upload files > 5GB (S3 limit for single PUT)
-- ❌ **S3 Select**: Cannot query CSV/JSON files without downloading
-- ❌ **IAM Role Authentication**: Requires explicit credentials for presigned URLs
-- ❌ **Advanced AWS SDK Features**: No direct SDK access
-
-**Design Trade-offs**:
-- **Presigned URL Expiry**: URLs expire after 1 hour (default) - sufficient for most operations
-- **XML Parsing**: List operations parse S3 XML responses (minimal overhead)
-- **Memory Usage**: Loads entire files into memory - process in chunks for files > 50MB
-- **URL Length Limits**: Presigned URLs limited to ~8KB (rarely an issue)
-
-**When These Limitations Matter**:
-- Files > 100MB → Use `S3Service` with multipart upload
-- Need to query CSV/Parquet → Use `S3Service` with S3 Select
-- Running on AWS EC2/Lambda with IAM roles → Use `S3Service` to leverage IAM roles
-
-### General Limitations
-
-- All operations load files into memory (Buffer or string)
-- For very large files (> 100MB), implement chunked processing
-- Presigned URLs generated at request time (small latency overhead)
-- Built-in retry logic handles transient failures (no manual retry needed)
-
----
-
-## API Reference
-
-### S3DataSource Constructor
-
-```typescript
-new S3DataSource(config: S3DataSourceConfig, logger?: StructuredLogger, httpClient?: typeof fetch)
-```
-
-**Parameters**:
-
-- `config`: S3 data source configuration
-- `logger`: Optional logging service
-- `httpClient`: Optional custom fetch implementation (defaults to global `fetch`)
-
-### Methods
-
-| Method                  | Parameters                         | Returns                     | Description                      |
-| ----------------------- | ---------------------------------- | --------------------------- | -------------------------------- |
-| `listFiles()`           | `options?`                         | `Promise<FileMetadata[]>`   | List files in bucket             |
-| `downloadFile()`        | `path, options?`                   | `Promise<string \| Buffer>` | Read file content                |
-| `uploadFile()`          | `path, content, options?`          | `Promise<void>`             | Write file to S3                 |
-| `deleteFile()`          | `path`                             | `Promise<void>`             | Delete file from S3              |
-| `copyFile()`            | `sourcePath, targetPath, options?` | `Promise<void>`             | Copy file within/between buckets |
-| `moveFile()`            | `sourcePath, targetPath`           | `Promise<void>`             | Move file (copy + delete)        |
-| `validateConnection()`  | -                                  | `Promise<boolean>`          | Test S3 connectivity             |
-| `writeParquetContent()` | `records, options?`                | `Promise<Buffer>`           | Generate Parquet buffer          |
-
----
-
-## Next Steps
-
-**Master SFTP operations** → [Module 3: SFTP Operations](./data-sources-03-sftp-operations.md)
-
-**Learn file processing patterns** → [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
-
-**Explore advanced topics** → [Module 5: Advanced Topics](./data-sources-05-advanced-topics.md)
-
----
-
-[← Back to Foundations](./data-sources-01-foundations.md) | [Next: SFTP Operations →](./data-sources-03-sftp-operations.md) | [↑ Back to Guide](../data-sources-readme.md)
+# Module 2: S3 Operations
+
+Complete guide to S3DataSource - presigned URL architecture, file operations, streaming, and production patterns.
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Presigned URL Architecture](#presigned-url-architecture)
+- [Configuration](#configuration)
+- [File Operations](#file-operations)
+- [Parquet Generation](#parquet-generation)
+- [Streaming Patterns](#streaming-patterns)
+- [Platform Integration](#platform-integration)
+- [Production Patterns](#production-patterns)
+- [TypeScript Types](#typescript-types)
+- [Testing](#testing)
+- [Troubleshooting](#troubleshooting)
+- [Best Practices](#best-practices)
+- [Limitations](#limitations)
+- [API Reference](#api-reference)
+
+---
+
+## Installation
+
+Install the SDK which includes S3DataSource and all dependencies:
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk
+```
+
+**Dependencies**: The SDK uses AWS SDK v3 (^3.879.0) internally for presigned URL signature generation via `S3PresignService`, but `S3DataSource` itself uses native `fetch` for all S3 operations, making it compatible with Node.js, Deno, and Versori.
+
+---
+
+## Presigned URL Architecture
+
+### How S3DataSource Works
+
+`S3DataSource` uses a **presigned-URL-only architecture** for universal platform compatibility:
+
+```
+┌──────────────────────────────────────────────────────────┐
+│ S3DataSource (fc-connect-sdk)                             │
+└──────────────────┬───────────────────────────────────────┘
+                   │
+                   ▼
+         ┌─────────────────────┐
+         │ S3PresignService    │
+         │ (AWS SigV4 signing) │
+         └────────┬────────────┘
+                  │
+                  ▼
+    Generate Presigned URL (1-hour expiry)
+    https://bucket.s3.region.amazonaws.com/key?
+       X-Amz-Algorithm=AWS4-HMAC-SHA256
+       &X-Amz-Credential=...
+       &X-Amz-Signature=...
+                  │
+                  ▼
+         ┌─────────────────────┐
+         │ Native fetch()      │
+         │ (HTTP GET/PUT/...)  │
+         └────────┬────────────┘
+                  │
+                  ▼
+         ┌─────────────────────┐
+         │ AWS S3 Bucket       │
+         │ (validates sig)     │
+         └─────────────────────┘
+```
+
+**Key Benefits**:
+
+1. **No AWS SDK bundling** - Smaller package size, faster cold starts
+2. **Universal compatibility** - Works in Node.js, Deno, Versori without changes
+3. **Simple HTTP** - Standard `fetch()` API, easy to debug
+4. **Stateless** - No connection pooling or state management
+5. **Proxy-friendly** - Works with corporate proxies and firewalls
+
+### Signature Generation Process
+
+```typescript
+// How presigned URLs are generated internally:
+// 1. Create canonical request (method, path, query, headers)
+// 2. Sign with AWS SigV4 using access key + secret key
+// 3. Add signature to query string
+// 4. URL valid for 1 hour (default)
+
+// Example presigned GET URL:
+https://my-bucket.s3.us-east-1.amazonaws.com/inventory/file.csv?
+  X-Amz-Algorithm=AWS4-HMAC-SHA256&
+  X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20250115%2Fus-east-1%2Fs3%2Faws4_request&
+  X-Amz-Date=20250115T120000Z&
+  X-Amz-Expires=3600&
+  X-Amz-SignedHeaders=host&
+  X-Amz-Signature=abc123...
+```
+
+**Security Notes**:
+
+- Presigned URLs expire after 1 hour (3600 seconds)
+- Anyone with the URL can access the file during expiry window
+- Signatures are cryptographically secure (SHA256 HMAC)
+- No credentials exposed in URL (only signature)
+
+---
+
+## Configuration
+
+### Basic Configuration
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger,
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info',
+});
+
+// Minimal S3 configuration
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV', // S3_JSON, S3_XML, S3_PARQUET
+    connectionId: 's3-minimal',
+    name: 'S3 Minimal',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-bucket', // Required
+    },
+  },
+  logger
+);
+```
+
+### Configuration Options
+
+| Property                   | Required | Description                         | Default     | Example                   |
+| -------------------------- | -------- | ----------------------------------- | ----------- | ------------------------- |
+| `type`                     | ✅       | Data source type                    | -           | `'S3_CSV'`                |
+| `s3Config.accessKeyId`     | ✅       | AWS access key ID                   | -           | `'AKIAIOSFODNN7...'`      |
+| `s3Config.secretAccessKey` | ✅       | AWS secret access key               | -           | `'wJalrXUtnFEMI/...'`     |
+| `s3Config.region`          | ✅       | AWS region                          | -           | `'us-east-1'`             |
+| `s3Config.bucket`          | ❌       | Default bucket name                 | -           | `'inventory-data'`        |
+| `s3Config.sessionToken`    | ❌       | STS session token (temporary creds) | -           | `'FwoGZXIv...'`           |
+| `s3Config.endpoint`        | ❌       | Custom S3 endpoint (for LocalStack) | AWS default | `'http://localhost:4566'` |
+| `s3Config.maxAttempts`     | ❌       | Max retry attempts       | `3`         | `5`                       |
+
+### Enhanced Retry Logic
+
+S3DataSource includes automatic retry with exponential backoff for all HTTP operations.
+
+```typescript
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-retry',
+    name: 'S3 Retry',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-bucket',
+      maxAttempts: 5, // Default: 3, increase for unreliable networks
+    },
+  },
+  logger
+);
+
+// All operations automatically retry on failure:
+await s3Source.downloadFile('data.csv'); // Retries on 429, 500, 502, 503, 504, network errors
+await s3Source.uploadFile('output.json', data); // Retries automatically
+await s3Source.listFiles({ prefix: 'inventory/' }); // Retries automatically
+```
+
+**Retry Behavior:**
+
+| Attempt              | Delay  | Total Wait | Retryable Conditions                         |
+| -------------------- | ------ | ---------- | -------------------------------------------- |
+| 1                    | 0ms    | 0ms        | HTTP 429, 500, 502, 503, 504, network errors |
+| 2                    | 500ms  | 500ms      | Same conditions                              |
+| 3 (default max)      | 1000ms | 1.5s       | Same conditions                              |
+| 4 (if maxAttempts=4) | 2000ms | 3.5s       | Same conditions                              |
+| 5 (if maxAttempts=5) | 4000ms | 7.5s       | Same conditions                              |
+
+**Backoff Formula:**
+
+- Base delay: 500ms
+- Factor: 2 (exponential)
+- Max delay per attempt: 8000ms (8 seconds)
+- Formula: `min(8000ms, 500ms * 2^(attempt-1))`
+
+**What Gets Retried:**
+
+- ✅ Rate limiting (429 Too Many Requests)
+- ✅ Server errors (500, 502, 503, 504)
+- ✅ Network failures (connection timeout, DNS errors)
+- ✅ Transient S3 errors
+
+**What Does NOT Get Retried:**
+
+- ❌ Client errors (400, 403, 404) - permanent failures
+- ❌ Authentication errors (401) - credential issues
+- ❌ Successful responses (2xx)
+
+**Logging:**
+
+```typescript
+// Retry attempts are automatically logged at WARN level
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'debug',
+});
+const s3Source = new S3DataSource(config, logger);
+
+await s3Source.downloadFile('data.csv');
+// Logs on retry:
+// - "S3 HTTP retry due to status" (for 429, 500, etc.)
+// - "S3 HTTP retry due to network error" (for connection failures)
+// - Includes: operationName, status, attempt, delay
+```
+
+**Production Recommendation:**
+
+```typescript
+// For production, increase maxAttempts for better resilience
+const productionConfig = {
+  type: 'S3_CSV',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1',
+    bucket: 'production-bucket',
+    maxAttempts: 5, // 5 attempts = up to 7.5s of automatic retries
+  },
+};
+```
+
+### Environment Variables Pattern
+
+```bash
+# .env file
+AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
+AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+AWS_REGION=us-east-1
+AWS_SESSION_TOKEN=FwoGZXIv...  # Optional
+SOURCE_BUCKET=inventory-source
+```
+
+```typescript
+// Load from environment
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-env',
+    name: 'S3 Environment',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION || 'us-east-1',
+      bucket: process.env.SOURCE_BUCKET,
+      sessionToken: process.env.AWS_SESSION_TOKEN, // Optional
+    },
+  },
+  logger
+);
+```
+
+### Dual AWS Credentials (Source + Target)
+
+Use separate AWS accounts for reading (source) and writing (target):
+
+```typescript
+// Source S3 (read inventory from vendor's AWS account)
+const sourceS3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-source-vendor',
+    name: 'S3 Source Vendor',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: process.env.AWS_REGION || 'us-east-1',
+      bucket: process.env.SOURCE_BUCKET,
+    },
+  },
+  logger
+);
+
+// Target S3 (write extraction results to our data lake)
+const targetS3 = new S3DataSource(
+  {
+    type: 'S3_PARQUET',
+    connectionId: 's3-target-own',
+    name: 'S3 Target Own',
+    s3Config: {
+      accessKeyId: process.env.TARGET_AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.TARGET_AWS_SECRET_ACCESS_KEY!,
+      region: process.env.TARGET_AWS_REGION || 'us-west-2',
+      bucket: process.env.TARGET_BUCKET,
+    },
+  },
+  logger
+);
+
+// Usage: Read from vendor, write to our data lake
+const inventory = await sourceS3.downloadFile('s3://vendor-bucket/inventory.csv');
+const processed = await processInventory(inventory);
+await targetS3.uploadFile('s3://our-datalake/processed.parquet', processed);
+```
+
+---
+
+## File Operations
+
+### Listing Files
+
+```typescript
+// List all files in bucket
+const allFiles = await s3Source.listFiles();
+
+// List files with prefix (folder-like structure)
+const inventoryFiles = await s3Source.listFiles({
+  prefix: 'inventory/',
+  maxKeys: 100,
+});
+
+// List with delimiter (folders)
+const topLevel = await s3Source.listFiles({
+  prefix: 'data/',
+  delimiter: '/',
+  maxKeys: 1000,
+});
+
+// Process file metadata
+for (const file of inventoryFiles) {
+  console.log({
+    path: file.path,
+    name: file.name,
+    size: file.size,
+    lastModified: file.lastModified,
+    contentType: file.contentType,
+    etag: file.etag,
+    bucket: file.bucket,
+    region: file.region,
+  });
+}
+```
+
+**List Options**:
+
+| Option              | Type     | Description                                        |
+| ------------------- | -------- | -------------------------------------------------- |
+| `prefix`            | `string` | Filter files by prefix (e.g., `'inventory/'`)      |
+| `delimiter`         | `string` | Delimiter for folder-like structure (e.g., `'/'`)  |
+| `maxKeys`           | `number` | Maximum files to return (default: 1000, max: 1000) |
+| `continuationToken` | `string` | Token for pagination (from previous response)      |
+
+**Example: List files modified in last 24 hours**
+
+```typescript
+const files = await s3Source.listFiles({ prefix: 'incoming/' });
+
+const yesterday = new Date(Date.now() - 24 * 60 * 60 * 1000);
+const recentFiles = files.filter(file => file.lastModified > yesterday);
+
+console.log(`${recentFiles.length} files modified in last 24 hours`);
+```
+
+### Reading Files
+
+```typescript
+// Read text file (CSV, JSON, etc.)
+const csvContent = await s3Source.downloadFile('inventory.csv');
+// Returns: string
+
+// Read binary file (Parquet, images, etc.)
+const parquetData = await s3Source.downloadFile('data.parquet', { encoding: 'binary' });
+// Returns: Buffer
+
+// Read JSON with parsing
+const jsonContent = await s3Source.downloadFile('config.json');
+const config = JSON.parse(jsonContent as string);
+
+// For existence checks, list by prefix and filter, or handle 404 on download
+if (await s3Source.listFiles({ prefix: 'file.csv' }).then(files => files.length > 0)) {
+  const content = await s3Source.downloadFile('file.csv');
+}
+```
+
+**Read Options**:
+
+| Option            | Type                     | Description             | Default  |
+| ----------------- | ------------------------ | ----------------------- | -------- |
+| `encoding`        | `'utf8' \| 'binary'`     | Text or binary encoding | `'utf8'` |
+| `responseHeaders` | `Record<string, string>` | Custom HTTP headers     | `{}`     |
+
+**Example: Read with error handling**
+
+```typescript
+try {
+  const content = await s3Source.downloadFile('file.csv');
+  console.log('File read successfully');
+} catch (error: any) {
+  if (error.context?.statusCode === 404) {
+    console.log('File not found');
+  } else if (error.context?.statusCode === 403) {
+    console.log('Access denied - check IAM permissions');
+  } else {
+    console.error('S3 error:', error.message);
+  }
+}
+```
+
+### Writing Files
+
+```typescript
+// Write CSV file
+await s3Source.uploadFile('output/results.csv', csvData, { contentType: 'text/csv' });
+
+// Write JSON with metadata
+await s3Source.uploadFile('output/results.json', JSON.stringify(data, null, 2), {
+  contentType: 'application/json',
+  metadata: {
+    'processed-date': new Date().toISOString(),
+    version: '1.0',
+    processor: 'fc-connect-sdk',
+    'record-count': data.length.toString(),
+  },
+});
+
+// Write binary file (Parquet, images, etc.)
+await s3Source.uploadFile('output/data.parquet', parquetBuffer, {
+  contentType: 'application/octet-stream',
+});
+
+// Automatic content type detection from extension
+await s3Source.uploadFile('file.csv', csvData);
+// Automatically sets contentType: 'text/csv'
+
+await s3Source.uploadFile('file.json', jsonData);
+// Automatically sets contentType: 'application/json'
+```
+
+**Write Options**:
+
+| Option            | Type                     | Description                                              |
+| ----------------- | ------------------------ | -------------------------------------------------------- |
+| `contentType`     | `string`                 | MIME type (auto-detected from extension if not provided) |
+| `metadata`        | `Record<string, string>` | Custom S3 object metadata (key-value pairs)              |
+| `cacheControl`    | `string`                 | HTTP Cache-Control header                                |
+| `contentEncoding` | `string`                 | HTTP Content-Encoding header (e.g., `'gzip'`)            |
+
+### Deleting Files
+
+```typescript
+// Delete single file
+await s3Source.deleteFile('temp/old-file.csv');
+
+// Delete with error handling
+try {
+  await s3Source.deleteFile('important-file.json');
+  console.log('File deleted successfully');
+} catch (error: any) {
+  if (error.context?.statusCode === 404) {
+    console.log('File already deleted or does not exist');
+  } else {
+    console.error('Delete failed:', error.message);
+  }
+}
+
+// Delete multiple files (batch)
+const filesToDelete = [
+  's3://bucket/temp/file1.csv',
+  's3://bucket/temp/file2.csv',
+  's3://bucket/temp/file3.csv',
+];
+
+for (const file of filesToDelete) {
+  try {
+    await s3Source.deleteFile(file);
+  } catch (error) {
+    console.error(`Failed to delete ${file}:`, error);
+  }
+}
+```
+
+### Copying Files
+
+```typescript
+// Copy within same bucket
+await s3Source.copyFile('source/data.csv', 'archive/data.csv');
+
+// Copy from different bucket
+await s3Source.copyFile('file.json', 'backup/file.json', { sourceBucket: 'source-bucket' });
+
+// Copy with metadata
+await s3Source.copyFile('source.csv', 'archive/source.csv', {
+  metadata: {
+    'copied-from': 'source.csv',
+    'copy-date': new Date().toISOString(),
+    'archived-by': 'fc-connect-sdk',
+  },
+});
+```
+
+### Moving Files (Copy + Delete)
+
+```typescript
+// Move file within bucket
+await s3Source.moveFile('incoming/data.csv', 'processed/data.csv');
+
+// Move to archive with date-based organization
+const today = new Date().toISOString().split('T')[0];
+await s3Source.moveFile('incoming/inventory.csv', `archive/${today}/inventory.csv`);
+```
+
+**IMPORTANT**: `moveFile()` is implemented as copy + delete, which is **NOT atomic**. If delete fails, the file exists in both locations. For critical operations:
+
+```typescript
+// Manual atomic-like move with verification
+try {
+  // Copy
+  await s3Source.copyFile(sourcePath, targetPath);
+
+  // Verify copy succeeded
+  const exists = await s3Source.listFiles({ prefix: targetPath }).then(files => files.length > 0);
+  if (!exists) {
+    throw new Error('Copy verification failed');
+  }
+
+  // Delete original
+  await s3Source.deleteFile(sourcePath);
+} catch (error) {
+  console.error('Move failed:', error);
+  // Cleanup: delete target if it exists
+  try {
+    await s3Source.deleteFile(targetPath);
+  } catch {}
+}
+```
+
+### Checking File Existence
+
+```typescript
+// Use listFiles + filter or handle 404 on download to determine existence
+if (await s3Source.listFiles({ prefix: 'config.json' }).then(files => files.length > 0)) {
+  const config = await s3Source.downloadFile('config.json');
+} else {
+  console.log('Config file not found, using defaults');
+  const config = defaultConfig;
+}
+```
+
+---
+
+## Parquet Generation
+
+`S3DataSource` includes built-in Parquet file generation for efficient data storage and analytics:
+
+### Basic Parquet Generation
+
+```typescript
+import { S3DataSource, ParquetDataRecord } from '@fluentcommerce/fc-connect-sdk';
+
+// Prepare data records
+const records: ParquetDataRecord[] = [
+  { id: 1, sku: 'PROD-001', quantity: 100, price: 29.99, available: true },
+  { id: 2, sku: 'PROD-002', quantity: 200, price: 39.99, available: false },
+  { id: 3, sku: 'PROD-003', quantity: 150, price: 49.99, available: true },
+];
+
+// Generate Parquet buffer
+const parquetBuffer = await s3Source.writeParquetContent(records, {
+  compression: 'GZIP', // Options: UNCOMPRESSED, GZIP, SNAPPY, LZO, BROTLI
+  rowGroupSize: 50000, // Rows per group (default: 50000)
+});
+
+// Upload to S3
+await s3Source.uploadFile('s3://bucket/output/inventory.parquet', parquetBuffer, {
+  contentType: 'application/octet-stream',
+});
+```
+
+### Automatic Schema Inference
+
+Parquet schema is automatically inferred from record structure:
+
+```typescript
+// JavaScript type → Parquet type mapping
+const records = [
+  {
+    id: 1, // number (int) → INT64
+    sku: 'PROD-001', // string → UTF8
+    quantity: 100, // number (int) → INT64
+    price: 29.99, // number (float) → DOUBLE
+    available: true, // boolean → BOOLEAN
+    lastUpdated: new Date(), // Date → TIMESTAMP_MILLIS
+    metadata: { tags: ['new', 'hot'] }, // object → UTF8 (JSON serialized)
+  },
+];
+
+const buffer = await s3Source.writeParquetContent(records);
+```
+
+**Type Mapping**:
+
+- `string` → UTF8
+- `number` (integer) → INT64
+- `number` (float/decimal) → DOUBLE
+- `boolean` → BOOLEAN
+- `Date` → TIMESTAMP_MILLIS
+- `object` → UTF8 (JSON serialized)
+- `null` / `undefined` → NULL
+
+### Compression Options
+
+```typescript
+// No compression (fastest, largest file)
+const uncompressed = await s3Source.writeParquetContent(records, {
+  compression: 'UNCOMPRESSED',
+});
+
+// GZIP compression (good balance, widely supported)
+const gzipped = await s3Source.writeParquetContent(records, {
+  compression: 'GZIP',
+});
+
+// SNAPPY compression (fastest compression, good for analytics)
+const snappy = await s3Source.writeParquetContent(records, {
+  compression: 'SNAPPY',
+});
+
+// BROTLI compression (best compression ratio)
+const brotli = await s3Source.writeParquetContent(records, {
+  compression: 'BROTLI',
+});
+```
+
+**Compression Comparison**:
+
+| Algorithm    | Speed     | Compression Ratio | Use Case                      |
+| ------------ | --------- | ----------------- | ----------------------------- |
+| UNCOMPRESSED | Fastest   | 1:1               | Small files, temp data        |
+| SNAPPY       | Very fast | 2:1 to 3:1        | Analytics, real-time queries  |
+| GZIP         | Medium    | 4:1 to 6:1        | General purpose, wide support |
+| BROTLI       | Slowest   | 5:1 to 8:1        | Long-term storage, archival   |
+
+### Complex Object Handling
+
+```typescript
+// Complex nested objects are automatically serialized to JSON
+const records = [
+  {
+    id: 1,
+    productDetails: {
+      name: 'Product A',
+      category: 'Electronics',
+      attributes: { color: 'black', size: 'M' },
+    },
+    tags: ['new', 'featured', 'sale'],
+    timestamp: new Date(),
+  },
+];
+
+const buffer = await s3Source.writeParquetContent(records);
+// productDetails and tags are stored as JSON strings in Parquet
+```
+
+### Empty Record Handling
+
+```typescript
+// Empty arrays return minimal valid Parquet file
+const emptyBuffer = await s3Source.writeParquetContent([], {
+  compression: 'GZIP',
+});
+
+// Still a valid Parquet file (can be opened by analytics tools)
+await s3Source.uploadFile('s3://bucket/empty.parquet', emptyBuffer);
+```
+
+---
+
+## Streaming Patterns
+
+### Processing Large Files
+
+```typescript
+// Download large file in chunks to minimize memory usage
+async function processLargeFile(dataSource: S3DataSource, key: string) {
+  // Download file (S3DataSource buffers in memory)
+  const content = await dataSource.downloadFile(key, { encoding: 'binary' });
+
+  // Process in chunks
+  const chunkSize = 1024 * 1024; // 1MB chunks
+  let offset = 0;
+
+  while (offset < content.length) {
+    const chunk = (content as Buffer).slice(offset, offset + chunkSize);
+    await processChunk(chunk);
+    offset += chunkSize;
+  }
+}
+
+async function processChunk(chunk: Buffer) {
+  console.log(`Processing ${chunk.length} bytes`);
+  // Your processing logic here
+}
+```
+
+**Note**: For very large files (>100MB), consider using S3 Select or reading directly with AWS SDK if memory is constrained.
+
+---
+
+## Platform Integration
+
+### Node.js
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger,
+} from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info',
+});
+
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+      region: 'us-east-1',
+      bucket: 'my-bucket',
+    },
+  },
+  logger
+);
+```
+
+### Deno
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger,
+} from 'npm:@fluentcommerce/fc-connect-sdk@latest';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'info',
+});
+
+const s3Source = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    s3Config: {
+      accessKeyId: Deno.env.get('AWS_ACCESS_KEY_ID')!,
+      secretAccessKey: Deno.env.get('AWS_SECRET_ACCESS_KEY')!,
+      region: 'us-east-1',
+      bucket: 'my-bucket',
+    },
+  },
+  logger
+);
+```
+
+### Versori
+
+```typescript
+import {
+  S3DataSource,
+  createConsoleLogger,
+  toStructuredLogger,
+} from '@fluentcommerce/fc-connect-sdk';
+import { http } from '@versori/run';
+
+export const processFiles = http('s3-process', async ({ activation, log }) => {
+  const logger = toStructuredLogger(createConsoleLogger(), {
+    logLevel: 'info',
+  });
+
+  const s3Source = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      s3Config: {
+        accessKeyId: activation.env.AWS_ACCESS_KEY_ID,
+        secretAccessKey: activation.env.AWS_SECRET_ACCESS_KEY,
+        region: activation.env.AWS_REGION,
+        bucket: activation.env.S3_BUCKET,
+      },
+    },
+    logger
+  );
+
+  const files = await s3Source.listFiles({ prefix: 'inventory/' });
+  // Process files...
+});
+```
+
+---
+
+## Production Patterns
+
+### Pattern 1: Inventory Ingestion from S3
+
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function ingestInventoryFromS3() {
+  const s3Source = new S3DataSource(
+    {
+      /* config */
+    },
+    logger
+  );
+  const csvParser = new CSVParserService();
+  const mapper = new UniversalMapper(mappingConfig);
+  const stateService = new StateService(kvAdapter);
+
+  // List inventory files
+  const files = await s3Source.listFiles({ prefix: 'inventory/' });
+
+  for (const file of files) {
+    // Check if already processed
+    const processed = await stateService.isFileProcessed(file.etag);
+    if (processed) {
+      console.log(`Skipping already processed file: ${file.name}`);
+      continue;
+    }
+
+    try {
+      // Download and parse CSV
+      const content = await s3Source.downloadFile(file.path);
+      const records = await csvParser.parse(content as string);
+
+      // Map fields
+      const result = await mapper.map(records);
+      if (!result.success) {
+        logger.error('Mapping failed', undefined, { errors: result.errors });
+        continue;
+      }
+
+      // Send to Fluent (using Batch API)
+      await sendToFluent(result.data);
+
+      // Mark as processed
+      await stateService.markFileProcessed(file.etag, {
+        fileName: file.name,
+        recordCount: records.length,
+        processedAt: new Date().toISOString(),
+      });
+
+      // Archive file
+      await s3Source.moveFile(file.path, file.path.replace('inventory/', 'processed/'));
+
+      logger.info('File processed successfully', { file: file.name });
+    } catch (error) {
+      logger.error('Failed to process file', error as Error, { file: file.path });
+    }
+  }
+}
+```
+
+### Pattern 2: Data Extraction to S3
+
+```typescript
+async function extractInventoryToS3(client: FluentClient, targetS3: S3DataSource) {
+  // Query Fluent inventory
+  const inventory = await client.graphql({
+    query: `query GetInventory {
+      virtualPositions(first: 1000) {
+        edges {
+          node {
+            ref
+            productRef
+            groupRef
+            quantity
+            status
+            updatedOn
+          }
+        }
+      }
+    }`,
+  });
+
+  const records = inventory.virtualPositions.edges.map(e => e.node);
+
+  // Generate Parquet
+  const parquetBuffer = await targetS3.writeParquetContent(records, {
+    compression: 'SNAPPY',
+    rowGroupSize: 50000,
+  });
+
+  // Upload with timestamp
+  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+  const outputPath = `s3://extraction-bucket/inventory/${timestamp}/full-export.parquet`;
+
+  await targetS3.uploadFile(outputPath, parquetBuffer, {
+    contentType: 'application/octet-stream',
+    metadata: {
+      'extraction-timestamp': timestamp,
+      'record-count': records.length.toString(),
+      compression: 'SNAPPY',
+    },
+  });
+
+  logger.info('Extraction completed', { path: outputPath, records: records.length });
+}
+```
+
+---
+
+## TypeScript Types
+
+```typescript
+// S3 Configuration
+interface S3Config {
+  accessKeyId: string;
+  secretAccessKey: string;
+  region: string;
+  sessionToken?: string;
+  bucket?: string;
+  endpoint?: string;
+  forcePathStyle?: boolean;
+  maxAttempts?: number;
+}
+
+// S3DataSource Configuration
+interface S3DataSourceConfig {
+  type: 'S3_CSV' | 'S3_JSON' | 'S3_XML' | 'S3_PARQUET';
+  connectionId: string;
+  name: string;
+  s3Config: S3Config;
+}
+
+// File Metadata from S3DataSource
+interface FileMetadata {
+  path: string;
+  name: string;
+  size: number;
+  lastModified: Date;
+  contentType?: string;
+  etag?: string;
+  bucket: string;
+  region: string;
+}
+
+// List Options
+interface ListOptions {
+  prefix?: string;
+  delimiter?: string;
+  maxKeys?: number;
+  continuationToken?: string;
+}
+
+// Download Options
+interface DownloadOptions {
+  encoding?: 'utf8' | 'binary';
+  responseHeaders?: Record<string, string>;
+}
+
+// Upload Options
+interface UploadOptions {
+  contentType?: string;
+  metadata?: Record<string, string>;
+  cacheControl?: string;
+  contentEncoding?: string;
+}
+
+// Copy Options
+interface CopyOptions {
+  sourceBucket?: string;
+  metadata?: Record<string, string>;
+  metadataDirective?: 'COPY' | 'REPLACE';
+}
+
+// Parquet Options
+interface ParquetOptions {
+  compression?: 'UNCOMPRESSED' | 'GZIP' | 'SNAPPY' | 'LZO' | 'BROTLI';
+  rowGroupSize?: number;
+  schema?: Record<string, string>;
+}
+
+// Parquet Data Record
+interface ParquetDataRecord {
+  [key: string]: string | number | boolean | Date | object | null;
+}
+```
+
+---
+
+## Testing
+
+The SDK includes test utilities for validating S3 operations:
+
+```typescript
+import {
+  S3SDKTester,
+  S3PresignedTester,
+  S3ComparisonTester
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Test SDK implementation (if using S3Service)
+const sdkTester = new S3SDKTester(config);
+const sdkResults = await sdkTester.testAll('test-bucket');
+
+// Test Presigned implementation
+const presignedTester = new S3PresignedTester(config, fetch);
+const presignedResults = await presignedTester.testAll('test-bucket');
+
+// Compare both implementations
+const comparer = new S3ComparisonTester();
+const comparison = await comparer.compareImplementations(
+  config, 
+  'test-bucket'
+);
+console.log('Match rate:', comparison.summary.matchRate + '%');
+```
+
+**Note**: For `S3DataSource`, you typically don't need these testers since it only uses presigned URLs. Use these if you're testing `S3Service` with dual mode support.
+
+### Basic Connection Test
+
+```typescript
+// Simple connectivity test
+const isValid = await s3Source.validateConnection();
+if (!isValid) {
+  throw new Error('S3 connection failed - check credentials and region');
+}
+```
+
+---
+
+## Troubleshooting
+
+### Common Issues and Solutions
+
+#### "S3DataSource requires explicit bucket configuration"
+
+```typescript
+// ❌ Wrong - missing bucket
+const dataSource = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-test',
+  name: 'S3 Test',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1'
+    // Missing: bucket
+  }
+}, logger);
+
+// ✅ Correct - includes bucket
+const dataSource = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-test',
+  name: 'S3 Test',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-east-1',
+    bucket: 'my-bucket' // Required
+  }
+}, logger);
+```
+
+#### Region Mismatch Errors
+
+```typescript
+// Ensure your client region matches the bucket region
+const dataSource = new S3DataSource({
+  type: 'S3_CSV',
+  connectionId: 's3-test',
+  name: 'S3 Test',
+  s3Config: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    region: 'us-west-2', // Must match bucket region
+    bucket: 'my-bucket'
+  }
+}, logger);
+```
+
+#### Need Multipart Upload Support
+
+```typescript
+// S3DataSource does not support multipart upload
+// For files > 100MB requiring multipart, use S3Service instead:
+import { S3Service } from '@fluentcommerce/fc-connect-sdk';
+
+const s3 = new S3Service({
+  accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+  region: 'us-east-1'
+});
+
+// Now multipart works
+if (s3.hasAdvancedFeatures()) {
+  await s3.uploadLargeFile('bucket', 'large-file.bin', buffer, {
+    partSize: 10 * 1024 * 1024
+  });
+}
+```
+
+#### Content Type Detection
+
+The S3DataSource automatically detects content types based on file extensions:
+
+```typescript
+// Automatic content type detection
+const contentTypes = {
+  '.csv': 'text/csv',
+  '.json': 'application/json',
+  '.xml': 'application/xml',
+  '.txt': 'text/plain',
+  '.parquet': 'application/octet-stream',
+  '.xlsx': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
+  '.xls': 'application/vnd.ms-excel',
+  '.pdf': 'application/pdf',
+  '.zip': 'application/zip',
+  '.gz': 'application/gzip'
+};
+
+// Files are automatically assigned correct content type
+await s3DataSource.uploadFile('data.csv', csvContent);  // text/csv
+await s3DataSource.uploadFile('config.json', jsonContent);  // application/json
+```
+
+### When to Use S3DataSource vs S3Service
+
+| Use Case | Recommended Class | Reason |
+|----------|------------------|---------|
+| Standard file operations (read, write, delete) | `S3DataSource` | Simpler API, universal compatibility |
+| Node.js, Deno, or Versori connectors | `S3DataSource` | Works everywhere with presigned URLs |
+| Files < 100MB | `S3DataSource` | Efficient for standard file sizes |
+| Need metadata and content type support | `S3DataSource` | Full support via presigned URL headers |
+| Files > 100MB requiring multipart upload | `S3Service` | Has `uploadLargeFile()` method |
+| Need S3 Select (query CSV/JSON) | `S3Service` | Has `selectObjectContent()` method |
+| Need advanced AWS SDK features | `S3Service` | Direct SDK access |
+| Testing both implementations | `S3Service` | Has dual mode support |
+
+### Architecture Comparison
+
+**S3DataSource** (Recommended for most use cases):
+- ✅ Universal: Works in Node.js, Deno, Versori
+- ✅ Simple: Presigned URLs only, no mode switching
+- ✅ Lightweight: No AWS SDK bundled in application
+- ✅ Consistent: Same behavior everywhere
+- ❌ No multipart upload
+- ❌ No S3 Select
+
+**S3Service** (For advanced features):
+- ✅ Multipart upload for large files
+- ✅ S3 Select query support
+- ✅ Direct AWS SDK access
+- ✅ Dual mode (SDK or presigned)
+- ❌ More complex API
+- ❌ Larger bundle size (includes AWS SDK)
+
+---
+
+## Best Practices
+
+### 1. Choose the Right Class
+
+- Use `S3DataSource` for standard file operations (most common)
+- Use `S3Service` only when you need multipart upload or S3 Select
+
+### 2. Handle Large Files Properly
+
+- `S3DataSource` works well for files up to 50MB
+- For files > 50MB, process in chunks to avoid memory issues
+- For files > 100MB requiring multipart, switch to `S3Service`
+
+### 3. Implement Proper Error Handling
+
+- Always catch and handle `DataSourceError`
+- Check `error.context.statusCode` for specific S3 errors (404, 403, etc.)
+- Built-in retry logic handles transient failures automatically
+- Log errors with context for debugging
+
+### 4. Optimize List Operations
+
+- Use `prefix` parameter to limit results to specific paths
+- Use `delimiter` for folder-like organization
+- Set appropriate `maxKeys` to control page size
+- Implement pagination with `continuationToken` for large buckets
+
+### 5. Security Best Practices
+
+- Never hardcode credentials in source code
+- Use environment variables for AWS credentials
+- Implement least-privilege IAM policies
+- Use separate credentials for source and target operations
+- Leverage session tokens for temporary credentials
+
+### 6. Performance Optimization
+
+- Implement parallel uploads/downloads with concurrency control (5-10 concurrent)
+- Use `encoding: 'binary'` for non-text files
+- Process files in chunks for very large datasets
+- Batch operations to reduce HTTP overhead
+
+### 7. Data Organization
+
+- Use prefixes to organize files (e.g., `inventory/2024/01/15/file.csv`)
+- Set appropriate content types - `S3DataSource` auto-detects from extensions
+- Add metadata to track processing status and timestamps
+- Clean up temporary files after processing
+
+### 8. Platform Integration
+
+- `S3DataSource` works identically in Node.js, Deno, and Versori
+- Optionally pass custom `httpClient` to constructor for connection pooling
+- Use structured logging to track operations across platforms
+
+---
+
+## Limitations
+
+### S3DataSource Limitations
+
+**What S3DataSource Does NOT Support**:
+- ❌ **Multipart Upload**: Cannot upload files > 5GB (S3 limit for single PUT)
+- ❌ **S3 Select**: Cannot query CSV/JSON files without downloading
+- ❌ **IAM Role Authentication**: Requires explicit credentials for presigned URLs
+- ❌ **Advanced AWS SDK Features**: No direct SDK access
+
+**Design Trade-offs**:
+- **Presigned URL Expiry**: URLs expire after 1 hour (default) - sufficient for most operations
+- **XML Parsing**: List operations parse S3 XML responses (minimal overhead)
+- **Memory Usage**: Loads entire files into memory - process in chunks for files > 50MB
+- **URL Length Limits**: Presigned URLs limited to ~8KB (rarely an issue)
+
+**When These Limitations Matter**:
+- Files > 100MB → Use `S3Service` with multipart upload
+- Need to query CSV/Parquet → Use `S3Service` with S3 Select
+- Running on AWS EC2/Lambda with IAM roles → Use `S3Service` to leverage IAM roles
+
+### General Limitations
+
+- All operations load files into memory (Buffer or string)
+- For very large files (> 100MB), implement chunked processing
+- Presigned URLs generated at request time (small latency overhead)
+- Built-in retry logic handles transient failures (no manual retry needed)
+
+---
+
+## API Reference
+
+### S3DataSource Constructor
+
+```typescript
+new S3DataSource(config: S3DataSourceConfig, logger?: StructuredLogger, httpClient?: typeof fetch)
+```
+
+**Parameters**:
+
+- `config`: S3 data source configuration
+- `logger`: Optional logging service
+- `httpClient`: Optional custom fetch implementation (defaults to global `fetch`)
+
+### Methods
+
+| Method                  | Parameters                         | Returns                     | Description                      |
+| ----------------------- | ---------------------------------- | --------------------------- | -------------------------------- |
+| `listFiles()`           | `options?`                         | `Promise<FileMetadata[]>`   | List files in bucket             |
+| `downloadFile()`        | `path, options?`                   | `Promise<string \| Buffer>` | Read file content                |
+| `uploadFile()`          | `path, content, options?`          | `Promise<void>`             | Write file to S3                 |
+| `deleteFile()`          | `path`                             | `Promise<void>`             | Delete file from S3              |
+| `copyFile()`            | `sourcePath, targetPath, options?` | `Promise<void>`             | Copy file within/between buckets |
+| `moveFile()`            | `sourcePath, targetPath`           | `Promise<void>`             | Move file (copy + delete)        |
+| `validateConnection()`  | -                                  | `Promise<boolean>`          | Test S3 connectivity             |
+| `writeParquetContent()` | `records, options?`                | `Promise<Buffer>`           | Generate Parquet buffer          |
+
+---
+
+## Next Steps
+
+**Master SFTP operations** → [Module 3: SFTP Operations](./data-sources-03-sftp-operations.md)
+
+**Learn file processing patterns** → [Module 4: File Processing Patterns](./data-sources-04-file-patterns.md)
+
+**Explore advanced topics** → [Module 5: Advanced Topics](./data-sources-05-advanced-topics.md)
+
+---
+
+[← Back to Foundations](./data-sources-01-foundations.md) | [Next: SFTP Operations →](./data-sources-03-sftp-operations.md) | [↑ Back to Guide](../data-sources-readme.md)