@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/04-REFERENCE/architecture/architecture-adding-new-data-sources.md

@@ -1,686 +1,686 @@
-# Adding New Data Sources to fc-connect-sdk
-
-This guide explains how to add new data sources (GCP Storage, Azure Blob Storage, etc.) to the SDK following the established factory pattern.
-
-> **⚠️ IMPORTANT:**
->
-> `uploadFile()` is a **required** abstract method. All data source implementations
-> MUST implement this method or TypeScript will throw compilation errors.
-
-## Architecture Overview
-
-The SDK uses a **factory pattern** with abstract base classes for data source extensibility:
-
-```
-AbstractDataSource (base class)
-  ├── REQUIRED Contract (abstract methods - MUST implement):
-  │   ├── listFiles() - List available files
-  │   ├── downloadFile() - Download file content
-  │   ├── uploadFile() - Upload file ⚠️ **REQUIRED**
-  │   ├── deleteFile() - Delete file
-  │   ├── copyFileInternal() - Copy file
-  │   └── validateConnection() - Test connection
-  │
-  ├── ⚠️ IMPORTANT:
-  │   uploadFile() is REQUIRED. Implementations without it will fail type checking.
-  │
-  ├── TEMPLATE Methods (inherited - don't override):
-  │   ├── moveFile() - Uses copyFileInternal() + deleteFile()
-  │   └── archiveFile() - Uses moveFile() with timestamp
-  ├── Built-in: Retry logic, connection caching, file filtering
-  └── Implementations extend this
-
-DataSourceFactory (interface)
-  ├── create(config, logger): AbstractDataSource
-  ├── supports(type): boolean
-  └── getDefaultConfig(type): Partial<DataSourceConfig>
-
-DataSourceRegistry (singleton)
-  ├── Stores factories by type (S3_CSV, SFTP_CSV, etc.)
-  ├── Creates instances via registered factories
-  └── Validates type support
-```
-
-### Current Implementations
-
-| Data Source | File | Size | Types Supported |
-|------------|------|------|----------------|
-| **S3DataSource** | `s3-data-source.ts` | 33KB | S3_CSV, S3_JSON, S3_XML, S3_PARQUET |
-| **SftpDataSource** | `sftp-data-source.ts` | 99KB | SFTP_CSV, SFTP_JSON, SFTP_XML, SFTP_PARQUET |
-
-## Step-by-Step: Adding GCP Storage Data Source
-
-### Step 1: Create Data Source Class
-
-Create `src/data-sources/gcp-storage-data-source.ts`:
-
-```typescript
-import { AbstractDataSource, DataSourceFactory } from './abstract-data-source';
-import { DataSourceConfig, FileMetadata, StructuredLogger } from '../types';
-import { Storage } from '@google-cloud/storage';
-
-/**
- * GCP Storage configuration
- */
-export interface GcpStorageConfig {
-  projectId: string;
-  keyFilename?: string;  // Path to service account JSON
-  credentials?: object;  // Or inline credentials
-  bucket: string;
-  prefix?: string;
-}
-
-/**
- * GCP Storage Data Source Configuration
- */
-export interface GcpStorageDataSourceConfig extends DataSourceConfig {
-  type: 'GCP_CSV' | 'GCP_JSON' | 'GCP_XML' | 'GCP_PARQUET';
-  gcpConfig: GcpStorageConfig;
-}
-
-/**
- * GCP Storage Data Source Implementation
- */
-export class GcpStorageDataSource extends AbstractDataSource {
-  type = 'GCP_STORAGE';
-  private storage: Storage;
-  private bucket: string;
-
-  constructor(config: GcpStorageDataSourceConfig, logger: StructuredLogger) {
-    super(config, logger);
-
-    // Initialize GCP Storage client
-    this.storage = new Storage({
-      projectId: config.gcpConfig.projectId,
-      keyFilename: config.gcpConfig.keyFilename,
-      credentials: config.gcpConfig.credentials,
-    });
-
-    this.bucket = config.gcpConfig.bucket;
-  }
-
-  /**
-   * List files in GCP bucket
-   */
-  async listFiles(params?: Record<string, unknown>): Promise<FileMetadata[]> {
-    try {
-      const prefix = params?.prefix as string | undefined;
-      const [files] = await this.storage.bucket(this.bucket).getFiles({ prefix });
-
-      return files.map(file => ({
-        name: file.name,
-        path: `gs://${this.bucket}/${file.name}`,
-        size: parseInt(file.metadata.size as string, 10),
-        lastModified: file.metadata.updated,
-        etag: file.metadata.etag,
-      }));
-    } catch (error) {
-      this.handleError('listFiles', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Download file from GCP bucket
-   */
-  async downloadFile(path: string, options?: Record<string, unknown>): Promise<string | Buffer> {
-    try {
-      const file = this.storage.bucket(this.bucket).file(path);
-      const [content] = await file.download();
-
-      if (options?.encoding === 'utf8') {
-        return content.toString('utf8');
-      }
-
-      return content;
-    } catch (error) {
-      this.handleError('downloadFile', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Upload file to GCP bucket
-   *
-   * REQUIRED by AbstractDataSource contract
-   *
-   * @param path Remote path in bucket
-   * @param content Content to upload (string or Buffer)
-   * @param options Optional upload parameters
-   */
-  async uploadFile(
-    path: string,
-    content: Buffer | string,
-    options?: Record<string, unknown>
-  ): Promise<void> {
-    try {
-      const file = this.storage.bucket(this.bucket).file(path);
-      await file.save(content, {
-        contentType: options?.contentType as string | undefined,
-        metadata: options?.metadata as Record<string, string> | undefined,
-      });
-
-      this.logger.info('File uploaded to GCP', {
-        bucket: this.bucket,
-        path,
-        size: typeof content === 'string' ? content.length : content.length,
-      });
-    } catch (error) {
-      this.handleError('uploadFile', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Delete file from GCP bucket
-   *
-   * REQUIRED by AbstractDataSource contract
-   */
-  async deleteFile(path: string): Promise<void> {
-    try {
-      await this.storage.bucket(this.bucket).file(path).delete();
-      this.logger.info('File deleted from GCP', { bucket: this.bucket, path });
-    } catch (error) {
-      this.handleError('deleteFile', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Copy file within GCP bucket (implements abstract method)
-   *
-   * REQUIRED by AbstractDataSource contract - used by moveFile() template method
-   *
-   * @param sourcePath Source file path
-   * @param destPath Destination file path
-   */
-  async copyFileInternal(sourcePath: string, destPath: string): Promise<void> {
-    try {
-      const sourceFile = this.storage.bucket(this.bucket).file(sourcePath);
-      const destFile = this.storage.bucket(this.bucket).file(destPath);
-
-      await sourceFile.copy(destFile);
-
-      this.logger.info('File copied in GCP', {
-        bucket: this.bucket,
-        sourcePath,
-        destPath,
-      });
-    } catch (error) {
-      this.handleError('copyFileInternal', error);
-      throw error;
-    }
-  }
-
-  /**
-   * Validate GCP connection
-   */
-  async validateConnection(): Promise<boolean> {
-    try {
-      const [exists] = await this.storage.bucket(this.bucket).exists();
-      return exists;
-    } catch (error) {
-      this.logger.error('GCP connection validation failed', error as Error);
-      return false;
-    }
-  }
-
-  /**
-   * Validate GCP configuration
-   */
-  protected validateConfig(config: DataSourceConfig): void {
-    super.validateConfig(config);
-
-    const gcpConfig = (config as GcpStorageDataSourceConfig).gcpConfig;
-
-    if (!gcpConfig.projectId) {
-      throw new Error('GCP project ID is required');
-    }
-
-    if (!gcpConfig.bucket) {
-      throw new Error('GCP bucket name is required');
-    }
-
-    if (!gcpConfig.keyFilename && !gcpConfig.credentials) {
-      throw new Error('Either keyFilename or credentials must be provided');
-    }
-  }
-
-  /**
-   * Cleanup resources
-   */
-  async dispose(): Promise<void> {
-    await super.dispose();
-    // GCP Storage client doesn't require explicit disposal
-  }
-}
-
-/**
- * GCP Storage Data Source Factory
- */
-export class GcpStorageDataSourceFactory implements DataSourceFactory {
-  supports(type: string): boolean {
-    const supportedTypes = ['GCP_CSV', 'GCP_JSON', 'GCP_XML', 'GCP_PARQUET'];
-    return supportedTypes.includes(type.toUpperCase());
-  }
-
-  create(config: DataSourceConfig, logger: StructuredLogger): GcpStorageDataSource {
-    if (!this.supports(config.type)) {
-      throw new Error(`Factory does not support type: ${config.type}`);
-    }
-
-    return new GcpStorageDataSource(config as GcpStorageDataSourceConfig, logger);
-  }
-
-  getDefaultConfig(_type: string): Partial<GcpStorageDataSourceConfig> {
-    return {
-      type: 'GCP_CSV',
-      gcpConfig: {
-        projectId: '',
-        bucket: '',
-        keyFilename: '',
-      },
-    };
-  }
-}
-```
-
-### Step 2: Register Factory in Index
-
-Update `src/data-sources/index.ts`:
-
-```typescript
-// Add exports
-export { GcpStorageDataSource, GcpStorageDataSourceFactory } from './gcp-storage-data-source';
-export type { GcpStorageDataSourceConfig } from './gcp-storage-data-source';
-
-// Update initializeDataSources()
-export function initializeDataSources(): void {
-  const registry = DataSourceRegistry.getInstance();
-
-  // Existing registrations...
-  registry.register('S3_CSV', new S3DataSourceFactory());
-  registry.register('SFTP_CSV', new SftpDataSourceFactory());
-
-  // NEW: Register GCP Storage
-  const gcpFactory = new GcpStorageDataSourceFactory();
-  registry.register('GCP_CSV', gcpFactory);
-  registry.register('GCP_JSON', gcpFactory);
-  registry.register('GCP_XML', gcpFactory);
-  registry.register('GCP_PARQUET', gcpFactory);
-}
-```
-
-### Step 3: Add Types to SDK Types
-
-Update `src/types/index.ts`:
-
-```typescript
-export type DataSourceType =
-  | 'S3_CSV'
-  | 'S3_JSON'
-  | 'S3_XML'
-  | 'S3_PARQUET'
-  | 'SFTP_CSV'
-  | 'SFTP_JSON'
-  | 'SFTP_XML'
-  | 'SFTP_PARQUET'
-  | 'GCP_CSV'    // NEW
-  | 'GCP_JSON'   // NEW
-  | 'GCP_XML'    // NEW
-  | 'GCP_PARQUET'; // NEW
-```
-
-### Step 4: Install Dependencies
-
-```bash
-npm install @google-cloud/storage
-```
-
-Update `package.json`:
-
-```json
-{
-  "dependencies": {
-    "@google-cloud/storage": "^7.0.0"
-  }
-}
-```
-
-### Step 5: Add Unit Tests
-
-Create `tests/unit/data-sources/gcp-storage-data-source.test.ts`:
-
-```typescript
-import { GcpStorageDataSource, GcpStorageDataSourceFactory } from '../../../src/data-sources/gcp-storage-data-source';
-import { createConsoleLogger } from '../../../src/utils/logging';
-
-describe('GcpStorageDataSource', () => {
-  const mockConfig = {
-    type: 'GCP_CSV' as const,
-    connectionId: 'test-gcp',
-    name: 'Test GCP Storage',
-    gcpConfig: {
-      projectId: 'test-project',
-      bucket: 'test-bucket',
-      credentials: { /* mock credentials */ },
-    },
-  };
-
-  const logger = createConsoleLogger();
-
-  describe('Factory', () => {
-    it('should support GCP types', () => {
-      const factory = new GcpStorageDataSourceFactory();
-      expect(factory.supports('GCP_CSV')).toBe(true);
-      expect(factory.supports('GCP_JSON')).toBe(true);
-      expect(factory.supports('S3_CSV')).toBe(false);
-    });
-
-    it('should create GCP data source instance', () => {
-      const factory = new GcpStorageDataSourceFactory();
-      const dataSource = factory.create(mockConfig, logger);
-      expect(dataSource).toBeInstanceOf(GcpStorageDataSource);
-    });
-  });
-
-  describe('Configuration Validation', () => {
-    it('should throw error if projectId missing', () => {
-      const invalidConfig = { ...mockConfig, gcpConfig: { ...mockConfig.gcpConfig, projectId: '' } };
-      expect(() => new GcpStorageDataSource(invalidConfig, logger)).toThrow('GCP project ID is required');
-    });
-
-    it('should throw error if bucket missing', () => {
-      const invalidConfig = { ...mockConfig, gcpConfig: { ...mockConfig.gcpConfig, bucket: '' } };
-      expect(() => new GcpStorageDataSource(invalidConfig, logger)).toThrow('GCP bucket name is required');
-    });
-  });
-});
-```
-
-### Step 6: Update Documentation
-
-Create `docs/02-CORE-GUIDES/data-sources/gcp-storage.md`:
-
-```markdown
-# GCP Storage Data Source
-
-Upload and download files from Google Cloud Platform Storage buckets.
-
-## Installation
-
-```bash
-npm install @google-cloud/storage
-```
-
-## Configuration
-
-```typescript
-import { GcpStorageDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const gcpDataSource = new GcpStorageDataSource({
-  type: 'GCP_CSV',
-  connectionId: 'gcp-prod',
-  name: 'GCP Production Storage',
-  gcpConfig: {
-    projectId: 'my-project-id',
-    keyFilename: '/path/to/service-account.json',
-    bucket: 'my-bucket',
-  },
-}, logger);
-
-// List files
-const files = await gcpDataSource.listFiles({ prefix: 'inventory/' });
-
-// Download file
-const content = await gcpDataSource.downloadFile('inventory/positions.csv');
-
-// Upload file (REQUIRED method)
-await gcpDataSource.uploadFile('output/file.csv', Buffer.from('data'), {
-  contentType: 'text/csv',
-});
-```
-
-## Authentication Options
-
-### Service Account Key File
-
-```typescript
-gcpConfig: {
-  projectId: 'my-project',
-  keyFilename: './service-account.json',
-  bucket: 'my-bucket',
-}
-```
-
-### Inline Credentials
-
-```typescript
-gcpConfig: {
-  projectId: 'my-project',
-  credentials: {
-    client_email: 'service@project.iam.gserviceaccount.com',
-    private_key: '-----BEGIN PRIVATE KEY-----\n...',
-  },
-  bucket: 'my-bucket',
-}
-```
-
-### Application Default Credentials (ADC)
-
-When running in GCP (Cloud Run, Cloud Functions), credentials are automatic:
-
-```typescript
-gcpConfig: {
-  projectId: 'my-project',
-  bucket: 'my-bucket',
-  // keyFilename and credentials omitted - uses ADC
-}
-```
-```
-
-## Step 7: Test Integration
-
-```bash
-# Run unit tests
-npm test -- gcp-storage-data-source.test.ts
-
-# Run integration tests (requires real GCP credentials)
-npm run test:integration:gcp
-```
-
----
-
-## Adding Azure Blob Storage (Similar Pattern)
-
-Follow the same steps for Azure:
-
-1. Create `azure-blob-data-source.ts`
-2. Implement `AzureBlobDataSource extends AbstractDataSource`
-3. Implement `AzureBlobDataSourceFactory implements DataSourceFactory`
-4. Register factory with types: `AZURE_CSV`, `AZURE_JSON`, etc.
-5. Add `@azure/storage-blob` dependency
-6. Add tests and documentation
-
-**Key differences:**
-- Use `BlobServiceClient` from `@azure/storage-blob`
-- Authentication: connection strings, SAS tokens, or DefaultAzureCredential
-- Storage structure: containers instead of buckets
-
----
-
-## Factory Pattern Benefits
-
-### ✅ Extensibility
-Add new data sources without modifying existing code
-
-### ✅ Type Safety
-Each data source has strongly-typed config interfaces
-
-### ✅ Consistency
-All data sources share common operations via `AbstractDataSource`
-
-### ✅ Testability
-Easy to mock factories for unit tests
-
-### ✅ Registry Management
-Centralized registration and discovery of data sources
-
----
-
-## Best Practices
-
-### 1. Retry Logic
-Use built-in retry utilities from `AbstractDataSource`:
-
-```typescript
-async downloadFile(path: string): Promise<Buffer> {
-  return await this.getFileWithRetry(path, 3, 1000); // 3 retries, 1s delay
-}
-```
-
-### 2. Connection Caching
-Cache clients to avoid repeated initialization:
-
-```typescript
-private getClient(): StorageClient {
-  const cached = this.getCachedConnection<StorageClient>('client');
-  if (cached) return cached;
-
-  const client = new StorageClient(this.config);
-  this.setCachedConnection('client', client);
-  return client;
-}
-```
-
-### 3. Cleanup Resources
-Always implement `dispose()` for connection cleanup:
-
-```typescript
-async dispose(): Promise<void> {
-  await super.dispose();
-  await this.client?.close();
-}
-```
-
-### 4. Error Handling
-Use consistent error handling from base class:
-
-```typescript
-try {
-  return await operation();
-} catch (error) {
-  this.handleError('operationName', error);
-  throw error;
-}
-```
-
-### 5. Configuration Validation
-Validate config in constructor:
-
-```typescript
-protected validateConfig(config: DataSourceConfig): void {
-  super.validateConfig(config); // Base validation
-
-  if (!config.customField) {
-    throw new DataSourceError('Custom field is required');
-  }
-}
-```
-
----
-
-## Testing Your Data Source
-
-### Unit Tests (Mock Client)
-
-```typescript
-jest.mock('@google-cloud/storage');
-
-describe('GcpStorageDataSource', () => {
-  it('should list files', async () => {
-    const mockStorage = {
-      bucket: jest.fn().mockReturnValue({
-        getFiles: jest.fn().mockResolvedValue([mockFiles]),
-      }),
-    };
-
-    const dataSource = new GcpStorageDataSource(config, logger);
-    const files = await dataSource.listFiles();
-    expect(files).toHaveLength(3);
-  });
-});
-```
-
-### Integration Tests (Real Credentials)
-
-```typescript
-describe('GcpStorageDataSource Integration', () => {
-  const realConfig = {
-    type: 'GCP_CSV' as const,
-    connectionId: 'test-gcp',
-    name: 'Test GCP',
-    gcpConfig: {
-      projectId: process.env.GCP_PROJECT_ID!,
-      keyFilename: process.env.GCP_KEY_FILE!,
-      bucket: process.env.GCP_TEST_BUCKET!,
-    },
-  };
-
-  it('should connect to real GCP bucket', async () => {
-    const dataSource = new GcpStorageDataSource(realConfig, logger);
-    const isValid = await dataSource.validateConnection();
-    expect(isValid).toBe(true);
-  });
-});
-```
-
----
-
-## Summary
-
-Adding a new data source requires:
-
-1. ✅ Extend `AbstractDataSource`
-2. ✅ Implement **4 required methods**:
-   - `listFiles()` - List available files
-   - `downloadFile()` - Download file content
-   - `uploadFile()` - **Upload file content (REQUIRED)**
-   - `validateConnection()` - Test connection
-3. ✅ Implement `DataSourceFactory`
-4. ✅ Register factory with type aliases
-5. ✅ Add type definitions
-6. ✅ Install dependencies
-7. ✅ Write unit tests
-8. ✅ Add documentation
-9. ✅ Test integration
-
-**Time to add new data source:** ~2-4 hours for experienced developers
-
-**Maintenance cost:** Low - inherit retry, caching, filtering from base class
-
----
-
-## ⚠️ IMPORTANT: uploadFile() is Mandatory
-
-**`uploadFile()` is a required abstract method** in `AbstractDataSource`.
-
-**Why?**
-- All 13 extraction templates depend on upload functionality
-- Extraction workflows must export data to S3/SFTP/GCP/Azure
-- Ensures consistency across all data source implementations
-
-**Signature:**
-```typescript
-abstract uploadFile(
-  path: string,
-  content: string | Buffer,
-  options?: Record<string, unknown>
-): Promise<void>;
-```
-
-**Implementation Examples:**
-- **S3**: Uses presigned PUT URLs
-- **SFTP**: Uses ssh2-sftp-client with retry logic
-- **GCP**: Uses `@google-cloud/storage` file.save()
-- **Azure**: Uses `@azure/storage-blob` uploadData()
+# Adding New Data Sources to fc-connect-sdk
+
+This guide explains how to add new data sources (GCP Storage, Azure Blob Storage, etc.) to the SDK following the established factory pattern.
+
+> **⚠️ IMPORTANT:**
+>
+> `uploadFile()` is a **required** abstract method. All data source implementations
+> MUST implement this method or TypeScript will throw compilation errors.
+
+## Architecture Overview
+
+The SDK uses a **factory pattern** with abstract base classes for data source extensibility:
+
+```
+AbstractDataSource (base class)
+  ├── REQUIRED Contract (abstract methods - MUST implement):
+  │   ├── listFiles() - List available files
+  │   ├── downloadFile() - Download file content
+  │   ├── uploadFile() - Upload file ⚠️ **REQUIRED**
+  │   ├── deleteFile() - Delete file
+  │   ├── copyFileInternal() - Copy file
+  │   └── validateConnection() - Test connection
+  │
+  ├── ⚠️ IMPORTANT:
+  │   uploadFile() is REQUIRED. Implementations without it will fail type checking.
+  │
+  ├── TEMPLATE Methods (inherited - don't override):
+  │   ├── moveFile() - Uses copyFileInternal() + deleteFile()
+  │   └── archiveFile() - Uses moveFile() with timestamp
+  ├── Built-in: Retry logic, connection caching, file filtering
+  └── Implementations extend this
+
+DataSourceFactory (interface)
+  ├── create(config, logger): AbstractDataSource
+  ├── supports(type): boolean
+  └── getDefaultConfig(type): Partial<DataSourceConfig>
+
+DataSourceRegistry (singleton)
+  ├── Stores factories by type (S3_CSV, SFTP_CSV, etc.)
+  ├── Creates instances via registered factories
+  └── Validates type support
+```
+
+### Current Implementations
+
+| Data Source | File | Size | Types Supported |
+|------------|------|------|----------------|
+| **S3DataSource** | `s3-data-source.ts` | 33KB | S3_CSV, S3_JSON, S3_XML, S3_PARQUET |
+| **SftpDataSource** | `sftp-data-source.ts` | 99KB | SFTP_CSV, SFTP_JSON, SFTP_XML, SFTP_PARQUET |
+
+## Step-by-Step: Adding GCP Storage Data Source
+
+### Step 1: Create Data Source Class
+
+Create `src/data-sources/gcp-storage-data-source.ts`:
+
+```typescript
+import { AbstractDataSource, DataSourceFactory } from './abstract-data-source';
+import { DataSourceConfig, FileMetadata, StructuredLogger } from '../types';
+import { Storage } from '@google-cloud/storage';
+
+/**
+ * GCP Storage configuration
+ */
+export interface GcpStorageConfig {
+  projectId: string;
+  keyFilename?: string;  // Path to service account JSON
+  credentials?: object;  // Or inline credentials
+  bucket: string;
+  prefix?: string;
+}
+
+/**
+ * GCP Storage Data Source Configuration
+ */
+export interface GcpStorageDataSourceConfig extends DataSourceConfig {
+  type: 'GCP_CSV' | 'GCP_JSON' | 'GCP_XML' | 'GCP_PARQUET';
+  gcpConfig: GcpStorageConfig;
+}
+
+/**
+ * GCP Storage Data Source Implementation
+ */
+export class GcpStorageDataSource extends AbstractDataSource {
+  type = 'GCP_STORAGE';
+  private storage: Storage;
+  private bucket: string;
+
+  constructor(config: GcpStorageDataSourceConfig, logger: StructuredLogger) {
+    super(config, logger);
+
+    // Initialize GCP Storage client
+    this.storage = new Storage({
+      projectId: config.gcpConfig.projectId,
+      keyFilename: config.gcpConfig.keyFilename,
+      credentials: config.gcpConfig.credentials,
+    });
+
+    this.bucket = config.gcpConfig.bucket;
+  }
+
+  /**
+   * List files in GCP bucket
+   */
+  async listFiles(params?: Record<string, unknown>): Promise<FileMetadata[]> {
+    try {
+      const prefix = params?.prefix as string | undefined;
+      const [files] = await this.storage.bucket(this.bucket).getFiles({ prefix });
+
+      return files.map(file => ({
+        name: file.name,
+        path: `gs://${this.bucket}/${file.name}`,
+        size: parseInt(file.metadata.size as string, 10),
+        lastModified: file.metadata.updated,
+        etag: file.metadata.etag,
+      }));
+    } catch (error) {
+      this.handleError('listFiles', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Download file from GCP bucket
+   */
+  async downloadFile(path: string, options?: Record<string, unknown>): Promise<string | Buffer> {
+    try {
+      const file = this.storage.bucket(this.bucket).file(path);
+      const [content] = await file.download();
+
+      if (options?.encoding === 'utf8') {
+        return content.toString('utf8');
+      }
+
+      return content;
+    } catch (error) {
+      this.handleError('downloadFile', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Upload file to GCP bucket
+   *
+   * REQUIRED by AbstractDataSource contract
+   *
+   * @param path Remote path in bucket
+   * @param content Content to upload (string or Buffer)
+   * @param options Optional upload parameters
+   */
+  async uploadFile(
+    path: string,
+    content: Buffer | string,
+    options?: Record<string, unknown>
+  ): Promise<void> {
+    try {
+      const file = this.storage.bucket(this.bucket).file(path);
+      await file.save(content, {
+        contentType: options?.contentType as string | undefined,
+        metadata: options?.metadata as Record<string, string> | undefined,
+      });
+
+      this.logger.info('File uploaded to GCP', {
+        bucket: this.bucket,
+        path,
+        size: typeof content === 'string' ? content.length : content.length,
+      });
+    } catch (error) {
+      this.handleError('uploadFile', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Delete file from GCP bucket
+   *
+   * REQUIRED by AbstractDataSource contract
+   */
+  async deleteFile(path: string): Promise<void> {
+    try {
+      await this.storage.bucket(this.bucket).file(path).delete();
+      this.logger.info('File deleted from GCP', { bucket: this.bucket, path });
+    } catch (error) {
+      this.handleError('deleteFile', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Copy file within GCP bucket (implements abstract method)
+   *
+   * REQUIRED by AbstractDataSource contract - used by moveFile() template method
+   *
+   * @param sourcePath Source file path
+   * @param destPath Destination file path
+   */
+  async copyFileInternal(sourcePath: string, destPath: string): Promise<void> {
+    try {
+      const sourceFile = this.storage.bucket(this.bucket).file(sourcePath);
+      const destFile = this.storage.bucket(this.bucket).file(destPath);
+
+      await sourceFile.copy(destFile);
+
+      this.logger.info('File copied in GCP', {
+        bucket: this.bucket,
+        sourcePath,
+        destPath,
+      });
+    } catch (error) {
+      this.handleError('copyFileInternal', error);
+      throw error;
+    }
+  }
+
+  /**
+   * Validate GCP connection
+   */
+  async validateConnection(): Promise<boolean> {
+    try {
+      const [exists] = await this.storage.bucket(this.bucket).exists();
+      return exists;
+    } catch (error) {
+      this.logger.error('GCP connection validation failed', error as Error);
+      return false;
+    }
+  }
+
+  /**
+   * Validate GCP configuration
+   */
+  protected validateConfig(config: DataSourceConfig): void {
+    super.validateConfig(config);
+
+    const gcpConfig = (config as GcpStorageDataSourceConfig).gcpConfig;
+
+    if (!gcpConfig.projectId) {
+      throw new Error('GCP project ID is required');
+    }
+
+    if (!gcpConfig.bucket) {
+      throw new Error('GCP bucket name is required');
+    }
+
+    if (!gcpConfig.keyFilename && !gcpConfig.credentials) {
+      throw new Error('Either keyFilename or credentials must be provided');
+    }
+  }
+
+  /**
+   * Cleanup resources
+   */
+  async dispose(): Promise<void> {
+    await super.dispose();
+    // GCP Storage client doesn't require explicit disposal
+  }
+}
+
+/**
+ * GCP Storage Data Source Factory
+ */
+export class GcpStorageDataSourceFactory implements DataSourceFactory {
+  supports(type: string): boolean {
+    const supportedTypes = ['GCP_CSV', 'GCP_JSON', 'GCP_XML', 'GCP_PARQUET'];
+    return supportedTypes.includes(type.toUpperCase());
+  }
+
+  create(config: DataSourceConfig, logger: StructuredLogger): GcpStorageDataSource {
+    if (!this.supports(config.type)) {
+      throw new Error(`Factory does not support type: ${config.type}`);
+    }
+
+    return new GcpStorageDataSource(config as GcpStorageDataSourceConfig, logger);
+  }
+
+  getDefaultConfig(_type: string): Partial<GcpStorageDataSourceConfig> {
+    return {
+      type: 'GCP_CSV',
+      gcpConfig: {
+        projectId: '',
+        bucket: '',
+        keyFilename: '',
+      },
+    };
+  }
+}
+```
+
+### Step 2: Register Factory in Index
+
+Update `src/data-sources/index.ts`:
+
+```typescript
+// Add exports
+export { GcpStorageDataSource, GcpStorageDataSourceFactory } from './gcp-storage-data-source';
+export type { GcpStorageDataSourceConfig } from './gcp-storage-data-source';
+
+// Update initializeDataSources()
+export function initializeDataSources(): void {
+  const registry = DataSourceRegistry.getInstance();
+
+  // Existing registrations...
+  registry.register('S3_CSV', new S3DataSourceFactory());
+  registry.register('SFTP_CSV', new SftpDataSourceFactory());
+
+  // NEW: Register GCP Storage
+  const gcpFactory = new GcpStorageDataSourceFactory();
+  registry.register('GCP_CSV', gcpFactory);
+  registry.register('GCP_JSON', gcpFactory);
+  registry.register('GCP_XML', gcpFactory);
+  registry.register('GCP_PARQUET', gcpFactory);
+}
+```
+
+### Step 3: Add Types to SDK Types
+
+Update `src/types/index.ts`:
+
+```typescript
+export type DataSourceType =
+  | 'S3_CSV'
+  | 'S3_JSON'
+  | 'S3_XML'
+  | 'S3_PARQUET'
+  | 'SFTP_CSV'
+  | 'SFTP_JSON'
+  | 'SFTP_XML'
+  | 'SFTP_PARQUET'
+  | 'GCP_CSV'    // NEW
+  | 'GCP_JSON'   // NEW
+  | 'GCP_XML'    // NEW
+  | 'GCP_PARQUET'; // NEW
+```
+
+### Step 4: Install Dependencies
+
+```bash
+npm install @google-cloud/storage
+```
+
+Update `package.json`:
+
+```json
+{
+  "dependencies": {
+    "@google-cloud/storage": "^7.0.0"
+  }
+}
+```
+
+### Step 5: Add Unit Tests
+
+Create `tests/unit/data-sources/gcp-storage-data-source.test.ts`:
+
+```typescript
+import { GcpStorageDataSource, GcpStorageDataSourceFactory } from '../../../src/data-sources/gcp-storage-data-source';
+import { createConsoleLogger } from '../../../src/utils/logging';
+
+describe('GcpStorageDataSource', () => {
+  const mockConfig = {
+    type: 'GCP_CSV' as const,
+    connectionId: 'test-gcp',
+    name: 'Test GCP Storage',
+    gcpConfig: {
+      projectId: 'test-project',
+      bucket: 'test-bucket',
+      credentials: { /* mock credentials */ },
+    },
+  };
+
+  const logger = createConsoleLogger();
+
+  describe('Factory', () => {
+    it('should support GCP types', () => {
+      const factory = new GcpStorageDataSourceFactory();
+      expect(factory.supports('GCP_CSV')).toBe(true);
+      expect(factory.supports('GCP_JSON')).toBe(true);
+      expect(factory.supports('S3_CSV')).toBe(false);
+    });
+
+    it('should create GCP data source instance', () => {
+      const factory = new GcpStorageDataSourceFactory();
+      const dataSource = factory.create(mockConfig, logger);
+      expect(dataSource).toBeInstanceOf(GcpStorageDataSource);
+    });
+  });
+
+  describe('Configuration Validation', () => {
+    it('should throw error if projectId missing', () => {
+      const invalidConfig = { ...mockConfig, gcpConfig: { ...mockConfig.gcpConfig, projectId: '' } };
+      expect(() => new GcpStorageDataSource(invalidConfig, logger)).toThrow('GCP project ID is required');
+    });
+
+    it('should throw error if bucket missing', () => {
+      const invalidConfig = { ...mockConfig, gcpConfig: { ...mockConfig.gcpConfig, bucket: '' } };
+      expect(() => new GcpStorageDataSource(invalidConfig, logger)).toThrow('GCP bucket name is required');
+    });
+  });
+});
+```
+
+### Step 6: Update Documentation
+
+Create `docs/02-CORE-GUIDES/data-sources/gcp-storage.md`:
+
+```markdown
+# GCP Storage Data Source
+
+Upload and download files from Google Cloud Platform Storage buckets.
+
+## Installation
+
+```bash
+npm install @google-cloud/storage
+```
+
+## Configuration
+
+```typescript
+import { GcpStorageDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const gcpDataSource = new GcpStorageDataSource({
+  type: 'GCP_CSV',
+  connectionId: 'gcp-prod',
+  name: 'GCP Production Storage',
+  gcpConfig: {
+    projectId: 'my-project-id',
+    keyFilename: '/path/to/service-account.json',
+    bucket: 'my-bucket',
+  },
+}, logger);
+
+// List files
+const files = await gcpDataSource.listFiles({ prefix: 'inventory/' });
+
+// Download file
+const content = await gcpDataSource.downloadFile('inventory/positions.csv');
+
+// Upload file (REQUIRED method)
+await gcpDataSource.uploadFile('output/file.csv', Buffer.from('data'), {
+  contentType: 'text/csv',
+});
+```
+
+## Authentication Options
+
+### Service Account Key File
+
+```typescript
+gcpConfig: {
+  projectId: 'my-project',
+  keyFilename: './service-account.json',
+  bucket: 'my-bucket',
+}
+```
+
+### Inline Credentials
+
+```typescript
+gcpConfig: {
+  projectId: 'my-project',
+  credentials: {
+    client_email: 'service@project.iam.gserviceaccount.com',
+    private_key: '-----BEGIN PRIVATE KEY-----\n...',
+  },
+  bucket: 'my-bucket',
+}
+```
+
+### Application Default Credentials (ADC)
+
+When running in GCP (Cloud Run, Cloud Functions), credentials are automatic:
+
+```typescript
+gcpConfig: {
+  projectId: 'my-project',
+  bucket: 'my-bucket',
+  // keyFilename and credentials omitted - uses ADC
+}
+```
+```
+
+## Step 7: Test Integration
+
+```bash
+# Run unit tests
+npm test -- gcp-storage-data-source.test.ts
+
+# Run integration tests (requires real GCP credentials)
+npm run test:integration:gcp
+```
+
+---
+
+## Adding Azure Blob Storage (Similar Pattern)
+
+Follow the same steps for Azure:
+
+1. Create `azure-blob-data-source.ts`
+2. Implement `AzureBlobDataSource extends AbstractDataSource`
+3. Implement `AzureBlobDataSourceFactory implements DataSourceFactory`
+4. Register factory with types: `AZURE_CSV`, `AZURE_JSON`, etc.
+5. Add `@azure/storage-blob` dependency
+6. Add tests and documentation
+
+**Key differences:**
+- Use `BlobServiceClient` from `@azure/storage-blob`
+- Authentication: connection strings, SAS tokens, or DefaultAzureCredential
+- Storage structure: containers instead of buckets
+
+---
+
+## Factory Pattern Benefits
+
+### ✅ Extensibility
+Add new data sources without modifying existing code
+
+### ✅ Type Safety
+Each data source has strongly-typed config interfaces
+
+### ✅ Consistency
+All data sources share common operations via `AbstractDataSource`
+
+### ✅ Testability
+Easy to mock factories for unit tests
+
+### ✅ Registry Management
+Centralized registration and discovery of data sources
+
+---
+
+## Best Practices
+
+### 1. Retry Logic
+Use built-in retry utilities from `AbstractDataSource`:
+
+```typescript
+async downloadFile(path: string): Promise<Buffer> {
+  return await this.getFileWithRetry(path, 3, 1000); // 3 retries, 1s delay
+}
+```
+
+### 2. Connection Caching
+Cache clients to avoid repeated initialization:
+
+```typescript
+private getClient(): StorageClient {
+  const cached = this.getCachedConnection<StorageClient>('client');
+  if (cached) return cached;
+
+  const client = new StorageClient(this.config);
+  this.setCachedConnection('client', client);
+  return client;
+}
+```
+
+### 3. Cleanup Resources
+Always implement `dispose()` for connection cleanup:
+
+```typescript
+async dispose(): Promise<void> {
+  await super.dispose();
+  await this.client?.close();
+}
+```
+
+### 4. Error Handling
+Use consistent error handling from base class:
+
+```typescript
+try {
+  return await operation();
+} catch (error) {
+  this.handleError('operationName', error);
+  throw error;
+}
+```
+
+### 5. Configuration Validation
+Validate config in constructor:
+
+```typescript
+protected validateConfig(config: DataSourceConfig): void {
+  super.validateConfig(config); // Base validation
+
+  if (!config.customField) {
+    throw new DataSourceError('Custom field is required');
+  }
+}
+```
+
+---
+
+## Testing Your Data Source
+
+### Unit Tests (Mock Client)
+
+```typescript
+jest.mock('@google-cloud/storage');
+
+describe('GcpStorageDataSource', () => {
+  it('should list files', async () => {
+    const mockStorage = {
+      bucket: jest.fn().mockReturnValue({
+        getFiles: jest.fn().mockResolvedValue([mockFiles]),
+      }),
+    };
+
+    const dataSource = new GcpStorageDataSource(config, logger);
+    const files = await dataSource.listFiles();
+    expect(files).toHaveLength(3);
+  });
+});
+```
+
+### Integration Tests (Real Credentials)
+
+```typescript
+describe('GcpStorageDataSource Integration', () => {
+  const realConfig = {
+    type: 'GCP_CSV' as const,
+    connectionId: 'test-gcp',
+    name: 'Test GCP',
+    gcpConfig: {
+      projectId: process.env.GCP_PROJECT_ID!,
+      keyFilename: process.env.GCP_KEY_FILE!,
+      bucket: process.env.GCP_TEST_BUCKET!,
+    },
+  };
+
+  it('should connect to real GCP bucket', async () => {
+    const dataSource = new GcpStorageDataSource(realConfig, logger);
+    const isValid = await dataSource.validateConnection();
+    expect(isValid).toBe(true);
+  });
+});
+```
+
+---
+
+## Summary
+
+Adding a new data source requires:
+
+1. ✅ Extend `AbstractDataSource`
+2. ✅ Implement **4 required methods**:
+   - `listFiles()` - List available files
+   - `downloadFile()` - Download file content
+   - `uploadFile()` - **Upload file content (REQUIRED)**
+   - `validateConnection()` - Test connection
+3. ✅ Implement `DataSourceFactory`
+4. ✅ Register factory with type aliases
+5. ✅ Add type definitions
+6. ✅ Install dependencies
+7. ✅ Write unit tests
+8. ✅ Add documentation
+9. ✅ Test integration
+
+**Time to add new data source:** ~2-4 hours for experienced developers
+
+**Maintenance cost:** Low - inherit retry, caching, filtering from base class
+
+---
+
+## ⚠️ IMPORTANT: uploadFile() is Mandatory
+
+**`uploadFile()` is a required abstract method** in `AbstractDataSource`.
+
+**Why?**
+- All 13 extraction templates depend on upload functionality
+- Extraction workflows must export data to S3/SFTP/GCP/Azure
+- Ensures consistency across all data source implementations
+
+**Signature:**
+```typescript
+abstract uploadFile(
+  path: string,
+  content: string | Buffer,
+  options?: Record<string, unknown>
+): Promise<void>;
+```
+
+**Implementation Examples:**
+- **S3**: Uses presigned PUT URLs
+- **SFTP**: Uses ssh2-sftp-client with retry logic
+- **GCP**: Uses `@google-cloud/storage` file.save()
+- **Azure**: Uses `@azure/storage-blob` uploadData()