@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

package/docs/02-CORE-GUIDES/data-sources/modules/data-sources-07-troubleshooting.md

@@ -1,387 +1,387 @@
-# Module 7: Troubleshooting
-
-**Level:** Intermediate
-**Category:** Debugging & Support
-
-## Overview
-
-Common data source issues and their solutions for S3 and SFTP integrations.
-
-## Table of Contents
-
-- [Connection Failures](#connection-failures)
-- [Permission Errors](#permission-errors)
-- [Memory Leaks](#memory-leaks)
-- [Performance Bottlenecks](#performance-bottlenecks)
-- [File Not Found Errors](#file-not-found-errors)
-- [Timeout Issues](#timeout-issues)
-- [Debug Mode](#debug-mode)
-- [Common Error Messages](#common-error-messages)
-
----
-
-## Connection Failures
-
-### S3 Connection Issues
-
-**Symptom:** `NetworkingError: getaddrinfo ENOTFOUND`
-
-**Causes:**
-- Invalid AWS region
-- Network connectivity issues
-- DNS resolution failures
-
-**Solutions:**
-```typescript
-// Verify region
-const s3 = new S3DataSource({
-  type: 'S3_CSV',
-  settings: {
-    region: 'us-east-1',  // Verify correct region
-    bucket: 'my-bucket'
-  }
-}, log);
-
-// Test connection
-try {
-  await s3.listFiles({ keyPrefix: 'test/' });
-  console.log('S3 connection successful');
-} catch (error) {
-  console.error('S3 connection failed:', error);
-}
-```
-
-### SFTP Connection Issues
-
-**Symptom:** `Error: All configured authentication methods failed`
-
-**Causes:**
-- Wrong username/password
-- Invalid private key
-- Host key verification failure
-
-**Solutions:**
-```typescript
-// Check credentials
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  settings: {
-    host: 'sftp.example.com',
-    port: 22,
-    username: process.env.SFTP_USERNAME,
-    password: process.env.SFTP_PASSWORD
-  }
-}, log);
-
-// Test connection
-try {
-  await sftp.listFiles({ remotePath: '/' });
-  console.log('SFTP connection successful');
-} catch (error) {
-  console.error('SFTP connection failed:', error.message);
-}
-
-await sftp.dispose();
-```
-
----
-
-## Permission Errors
-
-### S3 Access Denied
-
-**Symptom:** `AccessDenied: Access Denied`
-
-**Causes:**
-- Missing IAM permissions
-- Bucket policy restrictions
-- Wrong AWS credentials
-
-**Solutions:**
-
-**Required IAM Permissions:**
-```json
-{
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Effect": "Allow",
-      "Action": [
-        "s3:ListBucket",
-        "s3:GetObject",
-        "s3:PutObject",
-        "s3:DeleteObject"
-      ],
-      "Resource": [
-        "arn:aws:s3:::my-bucket",
-        "arn:aws:s3:::my-bucket/*"
-      ]
-    }
-  ]
-}
-```
-
-**Verify Credentials:**
-```typescript
-console.log('Using AWS credentials:', {
-  accessKeyId: process.env.AWS_ACCESS_KEY_ID?.substring(0, 4) + '...',
-  region: process.env.AWS_REGION
-});
-```
-
-### SFTP Permission Denied
-
-**Symptom:** `Error: Permission denied`
-
-**Causes:**
-- Insufficient file permissions
-- Wrong directory path
-- User lacks access to path
-
-**Solutions:**
-```typescript
-// Verify path exists
-const files = await sftp.listFiles({ remotePath: '/' });
-console.log('Root directory accessible:', files.length > 0);
-
-// Check file permissions
-await sftp.readFile({
-  remotePath: '/path',
-  filename: 'test.txt'
-});
-```
-
----
-
-## Memory Leaks
-
-### Undisposed Connections
-
-**Symptom:** Memory usage grows over time
-
-**Cause:** Not disposing data sources
-
-**Solution:**
-```typescript
-// ✅ CORRECT: Always dispose
-const s3 = new S3DataSource(config, log);
-try {
-  await s3.listFiles({ keyPrefix: 'data/' });
-} finally {
-  await s3.dispose();  // Always dispose!
-}
-
-// ✅ CORRECT: Use try-finally for SFTP too
-const sftp = new SftpDataSource(config, log);
-try {
-  await sftp.listFiles({ remotePath: '/data' });
-} finally {
-  await sftp.dispose();
-}
-```
-
-### Large File Accumulation
-
-**Symptom:** Out of memory errors when processing large files
-
-**Cause:** Loading entire files into memory
-
-**Solution:** Use streaming
-```typescript
-// ❌ WRONG: Load entire file
-const content = await s3.readFile('large-file.csv');
-
-// ✅ CORRECT: Stream large files
-const stream = await s3.readFileStream('large-file.csv');
-for await (const chunk of stream) {
-  await processChunk(chunk);
-}
-```
-
----
-
-## Performance Bottlenecks
-
-### Slow File Listing
-
-**Symptom:** `listFiles()` takes minutes
-
-**Cause:** Too many files in bucket/directory
-
-**Solution:** Use prefix filtering
-```typescript
-// ❌ SLOW: List entire bucket
-const allFiles = await s3.listFiles({});
-
-// ✅ FAST: Filter by prefix
-const todayFiles = await s3.listFiles({
-  keyPrefix: `data/date=${new Date().toISOString().split('T')[0]}/`
-});
-```
-
-### Sequential File Processing
-
-**Symptom:** Processing many files takes hours
-
-**Cause:** Processing files one at a time
-
-**Solution:** Parallel processing
-```typescript
-// ✅ Process files in parallel
-const files = await s3.listFiles({ keyPrefix: 'data/' });
-
-const CONCURRENCY = 5;
-for (let i = 0; i < files.length; i += CONCURRENCY) {
-  const chunk = files.slice(i, i + CONCURRENCY);
-  await Promise.all(chunk.map(file => processFile(file)));
-}
-```
-
----
-
-## File Not Found Errors
-
-### S3 NoSuchKey
-
-**Symptom:** `NoSuchKey: The specified key does not exist`
-
-**Cause:** File doesn't exist or wrong key
-
-**Solution:**
-```typescript
-// Check if file exists
-const files = await s3.listFiles({
-  keyPrefix: 'data/',
-  filePattern: 'inventory.csv'
-});
-
-if (files.length === 0) {
-  console.error('File not found');
-} else {
-  const content = await s3.readFile(files[0].key);
-}
-```
-
-### SFTP File Not Found
-
-**Symptom:** `Error: No such file`
-
-**Cause:** Wrong path or filename
-
-**Solution:**
-```typescript
-// List directory contents
-const files = await sftp.listFiles({ remotePath: '/data' });
-console.log('Available files:', files.map(f => f.filename));
-
-// Use exact filename from listing
-if (files.length > 0) {
-  const content = await sftp.readFile({
-    remotePath: '/data',
-    filename: files[0].filename
-  });
-}
-```
-
----
-
-## Timeout Issues
-
-### S3 Request Timeout
-
-**Symptom:** `RequestTimeout: Request did not complete within timeout`
-
-**Cause:** Network issues or large file
-
-**Solution:** Increase timeout
-```typescript
-const s3 = new S3DataSource({
-  type: 'S3_CSV',
-  settings: {
-    bucket: 'my-bucket',
-    region: 'us-east-1',
-    requestTimeout: 60000  // 60 seconds
-  }
-}, log);
-```
-
-### SFTP Connection Timeout
-
-**Symptom:** `Error: Timeout while connecting to server`
-
-**Cause:** Slow network or server issues
-
-**Solution:**
-```typescript
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  settings: {
-    host: 'sftp.example.com',
-    port: 22,
-    connectionTimeout: 30000  // 30 seconds
-  }
-}, log);
-```
-
----
-
-## Debug Mode
-
-Enable verbose logging to debug issues:
-
-```typescript
-import { createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
-
-const logger = toStructuredLogger(createConsoleLogger(), {
-  logLevel: 'debug'  // Enable debug logs
-});
-
-const s3 = new S3DataSource(config, logger);
-
-// Now all operations log detailed info
-await s3.listFiles({ keyPrefix: 'data/' });
-// DEBUG: Listing objects in bucket my-bucket with prefix data/
-// DEBUG: Found 5 objects matching pattern
-```
-
----
-
-## Common Error Messages
-
-### S3 Errors
-
-| Error | Cause | Solution |
-|-------|-------|----------|
-| `NoSuchBucket` | Bucket doesn't exist | Verify bucket name |
-| `AccessDenied` | Missing IAM permissions | Add required permissions |
-| `InvalidBucketName` | Invalid bucket name | Use valid S3 bucket name |
-| `SignatureDoesNotMatch` | Wrong AWS secret key | Verify credentials |
-
-### SFTP Errors
-
-| Error | Cause | Solution |
-|-------|-------|----------|
-| `All configured authentication methods failed` | Wrong credentials | Verify username/password |
-| `Permission denied` | Insufficient permissions | Check file/directory permissions |
-| `No such file` | File doesn't exist | Verify path and filename |
-| `Connection timeout` | Network issues | Increase timeout, check network |
-
----
-
-## See Also
-
-**Related Modules:**
-- [Module 01: Foundations](./data-sources-01-foundations.md) - Basic setup
-- [Module 02: S3 Operations](./data-sources-02-s3-operations.md) - S3 API
-- [Module 03: SFTP Operations](./data-sources-03-sftp-operations.md) - SFTP API
-- [Module 05: Advanced Topics](./data-sources-05-advanced-topics.md) - Performance tuning
-
-**Troubleshooting Guides:**
-- [Quick Reference](../../../00-START-HERE/troubleshooting-quick-reference.md) - Common fixes
-
----
-
-**Navigation:**
-- [Back to Module List](./readme.md)
-- [Previous: Module 06 - Integration Patterns](./data-sources-06-integration-patterns.md)
-- [Next: Module 08 - API Reference](./data-sources-08-api-reference.md)
+# Module 7: Troubleshooting
+
+**Level:** Intermediate
+**Category:** Debugging & Support
+
+## Overview
+
+Common data source issues and their solutions for S3 and SFTP integrations.
+
+## Table of Contents
+
+- [Connection Failures](#connection-failures)
+- [Permission Errors](#permission-errors)
+- [Memory Leaks](#memory-leaks)
+- [Performance Bottlenecks](#performance-bottlenecks)
+- [File Not Found Errors](#file-not-found-errors)
+- [Timeout Issues](#timeout-issues)
+- [Debug Mode](#debug-mode)
+- [Common Error Messages](#common-error-messages)
+
+---
+
+## Connection Failures
+
+### S3 Connection Issues
+
+**Symptom:** `NetworkingError: getaddrinfo ENOTFOUND`
+
+**Causes:**
+- Invalid AWS region
+- Network connectivity issues
+- DNS resolution failures
+
+**Solutions:**
+```typescript
+// Verify region
+const s3 = new S3DataSource({
+  type: 'S3_CSV',
+  settings: {
+    region: 'us-east-1',  // Verify correct region
+    bucket: 'my-bucket'
+  }
+}, log);
+
+// Test connection
+try {
+  await s3.listFiles({ keyPrefix: 'test/' });
+  console.log('S3 connection successful');
+} catch (error) {
+  console.error('S3 connection failed:', error);
+}
+```
+
+### SFTP Connection Issues
+
+**Symptom:** `Error: All configured authentication methods failed`
+
+**Causes:**
+- Wrong username/password
+- Invalid private key
+- Host key verification failure
+
+**Solutions:**
+```typescript
+// Check credentials
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  settings: {
+    host: 'sftp.example.com',
+    port: 22,
+    username: process.env.SFTP_USERNAME,
+    password: process.env.SFTP_PASSWORD
+  }
+}, log);
+
+// Test connection
+try {
+  await sftp.listFiles({ remotePath: '/' });
+  console.log('SFTP connection successful');
+} catch (error) {
+  console.error('SFTP connection failed:', error.message);
+}
+
+await sftp.dispose();
+```
+
+---
+
+## Permission Errors
+
+### S3 Access Denied
+
+**Symptom:** `AccessDenied: Access Denied`
+
+**Causes:**
+- Missing IAM permissions
+- Bucket policy restrictions
+- Wrong AWS credentials
+
+**Solutions:**
+
+**Required IAM Permissions:**
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:ListBucket",
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:DeleteObject"
+      ],
+      "Resource": [
+        "arn:aws:s3:::my-bucket",
+        "arn:aws:s3:::my-bucket/*"
+      ]
+    }
+  ]
+}
+```
+
+**Verify Credentials:**
+```typescript
+console.log('Using AWS credentials:', {
+  accessKeyId: process.env.AWS_ACCESS_KEY_ID?.substring(0, 4) + '...',
+  region: process.env.AWS_REGION
+});
+```
+
+### SFTP Permission Denied
+
+**Symptom:** `Error: Permission denied`
+
+**Causes:**
+- Insufficient file permissions
+- Wrong directory path
+- User lacks access to path
+
+**Solutions:**
+```typescript
+// Verify path exists
+const files = await sftp.listFiles({ remotePath: '/' });
+console.log('Root directory accessible:', files.length > 0);
+
+// Check file permissions
+await sftp.readFile({
+  remotePath: '/path',
+  filename: 'test.txt'
+});
+```
+
+---
+
+## Memory Leaks
+
+### Undisposed Connections
+
+**Symptom:** Memory usage grows over time
+
+**Cause:** Not disposing data sources
+
+**Solution:**
+```typescript
+// ✅ CORRECT: Always dispose
+const s3 = new S3DataSource(config, log);
+try {
+  await s3.listFiles({ keyPrefix: 'data/' });
+} finally {
+  await s3.dispose();  // Always dispose!
+}
+
+// ✅ CORRECT: Use try-finally for SFTP too
+const sftp = new SftpDataSource(config, log);
+try {
+  await sftp.listFiles({ remotePath: '/data' });
+} finally {
+  await sftp.dispose();
+}
+```
+
+### Large File Accumulation
+
+**Symptom:** Out of memory errors when processing large files
+
+**Cause:** Loading entire files into memory
+
+**Solution:** Use streaming
+```typescript
+// ❌ WRONG: Load entire file
+const content = await s3.readFile('large-file.csv');
+
+// ✅ CORRECT: Stream large files
+const stream = await s3.readFileStream('large-file.csv');
+for await (const chunk of stream) {
+  await processChunk(chunk);
+}
+```
+
+---
+
+## Performance Bottlenecks
+
+### Slow File Listing
+
+**Symptom:** `listFiles()` takes minutes
+
+**Cause:** Too many files in bucket/directory
+
+**Solution:** Use prefix filtering
+```typescript
+// ❌ SLOW: List entire bucket
+const allFiles = await s3.listFiles({});
+
+// ✅ FAST: Filter by prefix
+const todayFiles = await s3.listFiles({
+  keyPrefix: `data/date=${new Date().toISOString().split('T')[0]}/`
+});
+```
+
+### Sequential File Processing
+
+**Symptom:** Processing many files takes hours
+
+**Cause:** Processing files one at a time
+
+**Solution:** Parallel processing
+```typescript
+// ✅ Process files in parallel
+const files = await s3.listFiles({ keyPrefix: 'data/' });
+
+const CONCURRENCY = 5;
+for (let i = 0; i < files.length; i += CONCURRENCY) {
+  const chunk = files.slice(i, i + CONCURRENCY);
+  await Promise.all(chunk.map(file => processFile(file)));
+}
+```
+
+---
+
+## File Not Found Errors
+
+### S3 NoSuchKey
+
+**Symptom:** `NoSuchKey: The specified key does not exist`
+
+**Cause:** File doesn't exist or wrong key
+
+**Solution:**
+```typescript
+// Check if file exists
+const files = await s3.listFiles({
+  keyPrefix: 'data/',
+  filePattern: 'inventory.csv'
+});
+
+if (files.length === 0) {
+  console.error('File not found');
+} else {
+  const content = await s3.readFile(files[0].key);
+}
+```
+
+### SFTP File Not Found
+
+**Symptom:** `Error: No such file`
+
+**Cause:** Wrong path or filename
+
+**Solution:**
+```typescript
+// List directory contents
+const files = await sftp.listFiles({ remotePath: '/data' });
+console.log('Available files:', files.map(f => f.filename));
+
+// Use exact filename from listing
+if (files.length > 0) {
+  const content = await sftp.readFile({
+    remotePath: '/data',
+    filename: files[0].filename
+  });
+}
+```
+
+---
+
+## Timeout Issues
+
+### S3 Request Timeout
+
+**Symptom:** `RequestTimeout: Request did not complete within timeout`
+
+**Cause:** Network issues or large file
+
+**Solution:** Increase timeout
+```typescript
+const s3 = new S3DataSource({
+  type: 'S3_CSV',
+  settings: {
+    bucket: 'my-bucket',
+    region: 'us-east-1',
+    requestTimeout: 60000  // 60 seconds
+  }
+}, log);
+```
+
+### SFTP Connection Timeout
+
+**Symptom:** `Error: Timeout while connecting to server`
+
+**Cause:** Slow network or server issues
+
+**Solution:**
+```typescript
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  settings: {
+    host: 'sftp.example.com',
+    port: 22,
+    connectionTimeout: 30000  // 30 seconds
+  }
+}, log);
+```
+
+---
+
+## Debug Mode
+
+Enable verbose logging to debug issues:
+
+```typescript
+import { createConsoleLogger, toStructuredLogger } from '@fluentcommerce/fc-connect-sdk';
+
+const logger = toStructuredLogger(createConsoleLogger(), {
+  logLevel: 'debug'  // Enable debug logs
+});
+
+const s3 = new S3DataSource(config, logger);
+
+// Now all operations log detailed info
+await s3.listFiles({ keyPrefix: 'data/' });
+// DEBUG: Listing objects in bucket my-bucket with prefix data/
+// DEBUG: Found 5 objects matching pattern
+```
+
+---
+
+## Common Error Messages
+
+### S3 Errors
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `NoSuchBucket` | Bucket doesn't exist | Verify bucket name |
+| `AccessDenied` | Missing IAM permissions | Add required permissions |
+| `InvalidBucketName` | Invalid bucket name | Use valid S3 bucket name |
+| `SignatureDoesNotMatch` | Wrong AWS secret key | Verify credentials |
+
+### SFTP Errors
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| `All configured authentication methods failed` | Wrong credentials | Verify username/password |
+| `Permission denied` | Insufficient permissions | Check file/directory permissions |
+| `No such file` | File doesn't exist | Verify path and filename |
+| `Connection timeout` | Network issues | Increase timeout, check network |
+
+---
+
+## See Also
+
+**Related Modules:**
+- [Module 01: Foundations](./data-sources-01-foundations.md) - Basic setup
+- [Module 02: S3 Operations](./data-sources-02-s3-operations.md) - S3 API
+- [Module 03: SFTP Operations](./data-sources-03-sftp-operations.md) - SFTP API
+- [Module 05: Advanced Topics](./data-sources-05-advanced-topics.md) - Performance tuning
+
+**Troubleshooting Guides:**
+- [Quick Reference](../../../00-START-HERE/troubleshooting-quick-reference.md) - Common fixes
+
+---
+
+**Navigation:**
+- [Back to Module List](./readme.md)
+- [Previous: Module 06 - Integration Patterns](./data-sources-06-integration-patterns.md)
+- [Next: Module 08 - API Reference](./data-sources-08-api-reference.md)