@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/patterns/patterns-master-data-etl.md

@@ -1,2399 +1,2399 @@
-# Pattern: Master Data ETL - Generic Framework for Loading Any Entity
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
-
-**Status**: Production Ready
-
-**Complexity**: Intermediate
-
-**Est. Time**: 30-60 minutes
-
-**Use Cases**: Locations, Products, Controls, Carriers, Customers, Pricing, Categories, etc.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [What You'll Build](#what-youll-build)
-- [SDK Methods Used](#sdk-methods-used)
-- [The Generic Pattern](#the-generic-pattern)
-- [Example 1: Location Master Data](#example-1-location-master-data)
-- [Example 2: Product Catalog](#example-2-product-catalog)
-- [Example 3: Control/Config Data](#example-3-controlconfig-data)
-- [Source Strategies](#source-strategies)
-- [Load Strategies](#load-strategies)
-- [Configuration Schema](#configuration-schema)
-- [Extending to Other Entities](#extending-to-other-entities)
-- [Testing](#testing)
-- [Common Issues](#common-issues)
-- [Related Guides](#related-guides)
-
----
-
-## Overview
-
-Master data ETL is the process of extracting reference data from external systems and loading it into Fluent Commerce. Unlike transactional data (orders, inventory updates), master data changes infrequently and defines the core entities of your commerce platform.
-
-**Common Master Data Entities:**
-
-- **Locations**: Stores, warehouses, distribution centers
-- **Products**: SKUs, product catalogs, variants
-- **Controls**: Business rules, configuration parameters
-- **Carriers**: Shipping carriers, service levels
-- **Customers**: Customer profiles, segments
-- **Categories**: Product taxonomies, merchandising hierarchies
-- **Pricing**: Price lists, promotional rules
-
-**Why This Pattern?**
-
-This guide provides a **generic, configuration-driven framework** that works for ANY master data entity. Instead of writing custom code for each entity type, you configure JSON mappings and reuse the same pipeline.
-
-**Key Benefits:**
-
-- ✅ **One Pattern for All Entities**: Same code works for locations, products, controls, etc.
-- ✅ **Configuration-Driven**: No code changes needed for new entity types
-- ✅ **Multiple Source Formats**: CSV, JSON, XML support out-of-the-box
-- ✅ **Multiple Load Strategies**: GraphQL mutations or Event API
-- ✅ **Automatic Deduplication**: State management prevents duplicate loads
-- ✅ **Production-Ready Error Handling**: Comprehensive logging and retry logic
-
----
-
-## What You'll Build
-
-A **reusable ETL framework** with these capabilities:
-
-1. **Extract**: Read master data from S3/SFTP in CSV, JSON, or XML format
-2. **Parse**: Convert file format to JavaScript objects
-3. **Transform**: Map source fields to Fluent schema using field mappings
-4. **Load**: Submit to Fluent via GraphQL mutations or Event API
-5. **Track**: Prevent duplicate processing using state management
-
-**Pipeline Flow:**
-
-```
-Source File (S3/SFTP)
-  ↓
-Extract (DataSource)
-  ↓
-Parse (CSVParserService/JSONParser/XMLParser)
-  ↓
-Transform (UniversalMapper)
-  ↓
-Load (GraphQL Mutation or Event API)
-  ↓
-Track (StateService)
-```
-
-**Configuration-Driven Design:**
-
-Instead of hardcoded logic:
-
-```typescript
-// ❌ WRONG: Hardcoded for each entity type
-if (entityType === 'location') {
- // Custom location loading logic
-} else if (entityType === 'product') {
- // Custom product loading logic
-}
-```
-
-You use JSON configuration:
-
-```typescript
-// ✅ CORRECT: Generic pipeline with config
-const config = loadConfig('location-mapping.json');
-await etlPipeline.run(sourceFile, config);
-```
-
-> Tips:
->
-> - For identifiers that may look numeric (SKU/GTIN/UPC), add `resolver: "sdk.toString"` in mappings to force string output.
-> - When parsing XML sources where leading zeros matter, configure the XML parser with `parseNumbers: false` to prevent numeric coercion.
-
----
-
-## SDK Methods Used
-
-| Method                           | Purpose                | Pattern     |
-| ---------------------------------------------------------- | -------------------------------------- | --------------- |
-| `S3DataSource.downloadFile()`               | Download master data file from S3   | Source     |
-| `SftpDataSource.downloadFile()`              | Download master data file from SFTP  | Source     |
-| `CSVParserService.parse()`                 | Parse CSV master data         | Parse      |
-| `JSONParserService.parse()`                | Parse JSON master data         | Parse      |
-| `XMLParserService.parse()`                 | Parse XML master data         | Parse      |
-| `UniversalMapper.map()`                  | Transform source data to Fluent schema | Transform    |
-| `GraphQLMutationMapper.map()`               | Generate GraphQL mutations from data  | Load (Option 1) |
-| `FluentClient.graphql()`                  | Execute GraphQL mutations       | Load (Option 1) |
-| `FluentClient.sendEvent()`                 | Send events to Event API        | Load (Option 2) |
-| `StateService.markFileProcessed(kv, fileName, workflowId)` | Prevent duplicate processing      | Track      |
-
----
-
-## The Generic Pattern
-
-### Architecture Overview
-
-The master data ETL pattern follows a **four-phase pipeline** that adapts to any entity type through configuration:
-
-```
-┌──────────────────────────────────────────────────────────────┐
-│ PHASE 1: EXTRACT                       │
-│ - List files from S3/SFTP                  │
-│ - Filter by pattern (*.csv, locations_*.json, etc.)     │
-│ - Download file content                   │
-│ - Skip already-processed files (state check)        │
-└──────────────────────────────────────────────────────────────┘
-              ↓
-┌──────────────────────────────────────────────────────────────┐
-│ PHASE 2: PARSE                        │
-│ - Auto-detect format (CSV, JSON, XML)            │
-│ - Parse content to JavaScript objects            │
-│ - Validate structure (required fields, types)        │
-│ - Handle parsing errors gracefully              │
-└──────────────────────────────────────────────────────────────┘
-              ↓
-┌──────────────────────────────────────────────────────────────┐
-│ PHASE 3: TRANSFORM                      │
-│ - Apply field mappings (source to Fluent schema)      │
-│ - Execute resolvers (transformations, calculations)     │
-│ - Validate required fields                  │
-│ - Enrich with defaults/constants              │
-└──────────────────────────────────────────────────────────────┘
-              ↓
-┌──────────────────────────────────────────────────────────────┐
-│ PHASE 4: LOAD                        │
-│ - Choose strategy (GraphQL Mutation vs Event API)      │
-│ - Batch if needed (large datasets)             │
-│ - Execute load operation                   │
-│ - Handle errors and retries                 │
-│ - Mark file as processed (state update)           │
-└──────────────────────────────────────────────────────────────┘
-```
-
-### Configuration-Driven Design
-
-**Core Principle**: All entity-specific logic lives in JSON configuration files, not in code.
-
-**Generic Configuration Structure:**
-
-```json
-{
- "entityType": "location", // What you're loading
- "sourceConfig": {
-  // Where to get data
-  "type": "S3_CSV",
-  "bucket": "master-data",
-  "prefix": "locations/",
-  "filePattern": "*.csv"
- },
- "parseConfig": {
-  // How to parse data
-  "format": "csv",
-  "delimiter": ",",
-  "headers": true
- },
- "mappingConfig": {
-  // How to transform data
-  "version": "1.0",
-  "fields": {
-   "ref": { "source": "location_id", "required": true },
-   "name": { "source": "location_name" },
-   "status": { "value": "ACTIVE" }
-  }
- },
- "loadConfig": {
-  // How to load into Fluent
-  "strategy": "graphql",
-  "mutation": "createLocation",
-  "batchSize": 100
- }
-}
-```
-
-**Reusability**: Change `entityType` to "product", update field mappings → same code loads products!
-
-### Works for ANY Entity Type
-
-This pattern is entity-agnostic because:
-
-1. **Generic Source Reading**: S3DataSource/SftpDataSource work with any file
-2. **Format-Agnostic Parsing**: Parsers handle CSV/JSON/XML regardless of entity type
-3. **Flexible Mapping**: UniversalMapper adapts to any source→target schema
-4. **Mutation Generation**: GraphQLMutationMapper works with any mutation
-5. **State Management**: StateService tracks processing for any entity
-
-**Example Entity Types:**
-
-| Entity   | Source Format | Mutation     | Complexity     |
-| ---------- | ------------- | ---------------- | ------------------- |
-| Locations | CSV      | `createLocation` | Simple       |
-| Products  | JSON     | `createProduct` | Medium (variants)  |
-| Controls  | JSON     | `createControl` | Simple       |
-| Carriers  | XML      | `createCarrier` | Simple       |
-| Categories | JSON     | `createCategory` | Medium (hierarchy) |
-| Customers | CSV      | `createCustomer` | Medium (attributes) |
-
-All use the **same pipeline** with different **configurations**.
-
----
-
-## Example 1: Location Master Data
-
-### Overview
-
-Load store/warehouse locations from CSV files into Fluent Commerce.
-
-**Business Context:**
-
-- Retail locations change infrequently (openings, closings, updates)
-- Source: Retail management system exports CSV daily
-- Destination: Fluent Location entities
-- Frequency: Daily batch, event-driven on new file
-
-### Source Data Formats
-
-**CSV Format:**
-
-```csv
-location_id,location_name,type,address_line1,city,state,zip,country,latitude,longitude,status
-LOC001,Downtown Store,STORE,123 Main St,New York,NY,10001,US,40.7128,-74.0060,ACTIVE
-LOC002,Warehouse East,WAREHOUSE,456 Industrial Rd,Newark,NJ,07102,US,40.7357,-74.1724,ACTIVE
-LOC003,Pop-Up Shop,STORE,789 Fashion Ave,New York,NY,10018,US,40.7549,-73.9840,INACTIVE
-```
-
-**JSON Format:**
-
-```json
-{
- "locations": [
-  {
-   "locationId": "LOC001",
-   "name": "Downtown Store",
-   "type": "STORE",
-   "address": {
-    "street": "123 Main St",
-    "city": "New York",
-    "state": "NY",
-    "zip": "10001",
-    "country": "US"
-   },
-   "coordinates": {
-    "lat": 40.7128,
-    "lng": -74.006
-   },
-   "status": "ACTIVE"
-  }
- ]
-}
-```
-
-**XML Format:**
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<LocationFeed>
- <Location>
-  <ID>LOC001</ID>
-  <Name>Downtown Store</Name>
-  <Type>STORE</Type>
-  <Address>
-   <Line1>123 Main St</Line1>
-   <City>New York</City>
-   <State>NY</State>
-   <Zip>10001</Zip>
-   <Country>US</Country>
-  </Address>
-  <Latitude>40.7128</Latitude>
-  <Longitude>-74.0060</Longitude>
-  <Status>ACTIVE</Status>
- </Location>
-</LocationFeed>
-```
-
-### Field Mapping Configuration
-
-**`config/location-mapping.json`:**
-
-```json
-{
- "version": "1.0",
- "description": "Map external location data to Fluent Location schema",
- "fields": {
-  "ref": {
-   "source": "location_id",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "type": {
-   "source": "type",
-   "required": true,
-   "resolver": "sdk.uppercase"
-  },
-  "name": {
-   "source": "location_name",
-   "required": true
-  },
-  "status": {
-   "source": "status",
-   "defaultValue": "ACTIVE",
-   "resolver": "sdk.uppercase"
-  },
-  "primaryAddress": {
-   "fields": {
-    "street": { "source": "address_line1" },
-    "city": { "source": "city" },
-    "state": { "source": "state" },
-    "postcode": { "source": "zip" },
-    "country": { "source": "country" }
-   }
-  },
-  "coordinates": {
-   "fields": {
-    "latitude": { "source": "latitude", "resolver": "sdk.parseFloat" },
-    "longitude": { "source": "longitude", "resolver": "sdk.parseFloat" }
-   }
-  },
-  "retailerId": {
-   "value": "${RETAILER_ID}",
-   "required": true
-  }
- }
-}
-```
-
-**Key Features:**
-
-- ✅ Required field validation (`ref`, `type`, `name`)
-- ✅ Default values (`status` defaults to "ACTIVE")
-- ✅ Built-in resolvers (`sdk.trim`, `sdk.uppercase`, `sdk.parseFloat`)
-- ✅ Nested object mapping (`primaryAddress`, `coordinates`)
-- ✅ Environment variable support (`${RETAILER_ID}`)
-
-### GraphQL Mutation Approach
-
-**Target Mutation:**
-
-```graphql
-mutation CreateLocation($input: CreateLocationInput!) {
- createLocation(input: $input) {
-  id
-  ref
-  name
-  status
- }
-}
-```
-
-**Mutation Variables (after transformation):**
-
-```json
-{
- "input": {
-  "ref": "LOC001",
-  "type": "STORE",
-  "name": "Downtown Store",
-  "status": "ACTIVE",
-  "primaryAddress": {
-   "street": "123 Main St",
-   "city": "New York",
-   "state": "NY",
-   "postcode": "10001",
-   "country": "US"
-  },
-  "coordinates": {
-   "latitude": 40.7128,
-   "longitude": -74.006
-  },
-  "retailerId": "my-retailer"
- }
-}
-```
-
-### Complete Working Code
-
-**`location-etl.ts`:**
-
-```typescript
-// FC Connect SDK+
-// Install: npm install @fluentcommerce/fc-connect-sdk@latest
-// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
-// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
-
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
-import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-// Access openKv from context: const { openKv } = ctx;
-import * as fs from 'fs';
-
-// Initialize state service (prevents duplicate processing)
-// ✅ CORRECT: Access openKv from Versori context
-export async function masterDataETL(ctx: any, configPath: string) {
- // Load configuration
- const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
-
- // Initialize SDK components
- const logger = console; // Replace with proper logger
- const { openKv } = ctx;
- const fluentClient = await createClient({
-  config: {
-   baseUrl: process.env.FLUENT_BASE_URL!,
-   clientId: process.env.FLUENT_CLIENT_ID!,
-   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-   retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  logger,
- });
-
- // Initialize data source (S3 in this example)
- const s3DataSource = new S3DataSource(
-  {
-   type: 'S3_CSV',
-   s3Config: {
-    bucket: config.sourceConfig.bucket,
-    region: process.env.AWS_REGION!,
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-   },
-  },
-  logger
- );
-
- // Initialize state service (prevents duplicate processing)
- const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
- const stateService = new StateService(logger);
-
- // Initialize parser based on format
- const parser = config.parseConfig.format === 'csv' ? new CSVParserService() : null; // Add JSON/XML parsers as needed
-
- // Initialize mapper
- const mapper = new UniversalMapper(config.mappingConfig, { logger, fluentClient });
-
- logger.info(`Starting ${config.entityType} ETL process`);
-
- try {
-  // PHASE 1: EXTRACT - List files from source
-  const files = await s3DataSource.listFiles({
-   prefix: config.sourceConfig.prefix,
-  });
-
-  logger.info(`Found ${files.length} files to process`);
-
-  // Process each file
-  for (const file of files) {
-   const fileKey = `${config.entityType}:${file.name}`;
-
-   // Check if already processed
-   const alreadyProcessed = await stateService.isFileProcessed(kvAdapter, fileKey);
-   if (alreadyProcessed) {
-    logger.info(`Skipping already processed file: ${file.name}`);
-    continue;
-   }
-
-   logger.info(`Processing file: ${file.name}`);
-
-   try {
-    // PHASE 2: PARSE - Download and parse file
-    const fileContent = await s3DataSource.downloadFile(file.path);
-    const records = await parser!.parse(fileContent as string);
-
-    logger.info(`Parsed ${records.length} records from ${file.name}`);
-
-    // PHASE 3: TRANSFORM - Map each record
-    const transformedRecords = [];
-    for (const record of records) {
-     const result = await mapper.map(record);
-     if (result.success) {
-      transformedRecords.push(result.data);
-     } else {
-      logger.error(`Mapping failed for record:`, {
-       record,
-       errors: result.errors,
-      });
-     }
-    }
-
-    logger.info(`Transformed ${transformedRecords.length} records successfully`);
-
-    // PHASE 4: LOAD - Submit to Fluent Commerce
-    if (config.loadConfig.strategy === 'graphql') {
-     await loadViaGraphQL(fluentClient, transformedRecords, config.loadConfig, logger);
-    } else if (config.loadConfig.strategy === 'event') {
-     await loadViaEventAPI(fluentClient, transformedRecords, config.loadConfig, logger);
-    }
-
-    // Update sync state with processed file metadata
-    await stateService.updateSyncState(
-     kvAdapter,
-     [
-      {
-       fileName: file.name,
-       lastModified: new Date().toISOString(),
-       recordCount: transformedRecords.length,
-      },
-     ],
-     'master-data-etl'
-    );
-
-    logger.info(`Successfully processed file: ${file.name}`);
-   } catch (error) {
-    logger.error(`Failed to process file: ${file.name}`, error);
-    // Continue to next file instead of failing entire batch
-    continue;
-   }
-  }
-
-  logger.info(`${config.entityType} ETL process completed`);
- } catch (error) {
-  logger.error(`${config.entityType} ETL process failed`, error);
-  throw error;
- }
-}
-
-/**
- * Load data via GraphQL mutations
- */
-async function loadViaGraphQL(client: any, records: any[], loadConfig: any, logger: any) {
- const batchSize = loadConfig.batchSize || 100;
- const mutation = loadConfig.mutation;
-
- logger.info(`Loading ${records.length} records via GraphQL mutation: ${mutation}`);
-
- // Process in batches
- for (let i = 0; i < records.length; i += batchSize) {
-  const batch = records.slice(i, i + batchSize);
-  logger.info(`Processing batch ${i / batchSize + 1} (${batch.length} records)`);
-
-  // Execute mutations for batch
-  for (const record of batch) {
-   const query = `
-    mutation ${mutation}($input: ${capitalize(mutation)}Input!) {
-     ${mutation}(input: $input) {
-      id
-      ref
-     }
-    }
-   `;
-
-   try {
-    const result = await client.graphql({
-     query,
-     variables: { input: record },
-    });
-
-    logger.debug(`Created ${mutation}:`, result.data[mutation]);
-   } catch (error) {
-    logger.error(`Failed to create ${mutation}:`, { record, error });
-    // Continue to next record instead of failing entire batch
-   }
-  }
- }
-
- logger.info(`GraphQL load completed`);
-}
-
-/**
- * Load data via Event API
- */
-async function loadViaEventAPI(client: any, records: any[], loadConfig: any, logger: any) {
- const eventName = loadConfig.eventName;
-
- logger.info(`Loading ${records.length} records via Event API: ${eventName}`);
-
- for (const record of records) {
-  try {
-   await client.sendEvent({
-    name: eventName,
-    entityRef: record.ref,
-    entityType: loadConfig.entityType,
-    retailerId: record.retailerId,
-    attributes: record,
-   });
-
-   logger.debug(`Sent event ${eventName} for ${record.ref}`);
-  } catch (error) {
-   logger.error(`Failed to send event for ${record.ref}:`, error);
-  }
- }
-
- logger.info(`Event API load completed`);
-}
-
-// Helper function
-function capitalize(str: string): string {
- return str.charAt(0).toUpperCase() + str.slice(1);
-}
-
-// Example usage
-if (require.main === module) {
- masterDataETL('config/location-etl-config.json')
-  .then(() => console.log('ETL completed'))
-  .catch(err => {
-   console.error('ETL failed:', err);
-   process.exit(1);
-  });
-}
-```
-
-**Configuration File (`config/location-etl-config.json`):**
-
-```json
-{
- "entityType": "location",
- "sourceConfig": {
-  "type": "S3_CSV",
-  "bucket": "master-data",
-  "prefix": "locations/",
-  "filePattern": "*.csv"
- },
- "parseConfig": {
-  "format": "csv",
-  "delimiter": ",",
-  "headers": true
- },
- "mappingConfig": {
-  "version": "1.0",
-  "fields": {
-   "ref": { "source": "location_id", "required": true },
-   "name": { "source": "location_name", "required": true },
-   "type": { "source": "type", "required": true },
-   "status": { "source": "status", "defaultValue": "ACTIVE" },
-   "primaryAddress": {
-    "fields": {
-     "street": { "source": "address_line1" },
-     "city": { "source": "city" },
-     "state": { "source": "state" },
-     "postcode": { "source": "zip" },
-     "country": { "source": "country" }
-    }
-   },
-   "retailerId": { "value": "${RETAILER_ID}" }
-  }
- },
- "loadConfig": {
-  "strategy": "graphql",
-  "mutation": "createLocation",
-  "batchSize": 100
- }
-}
-```
-
-**Key Design Decisions:**
-
-1. **Configuration-Driven**: All entity logic in JSON config, not code
-2. **Error Handling**: Continue processing on individual record failures
-3. **State Management**: Prevent duplicate processing of files
-4. **Batching**: Process large datasets in manageable chunks
-5. **Logging**: Comprehensive logging for debugging and monitoring
-
----
-
-## Example 2: Product Catalog
-
-### Overview
-
-Load product catalog with variants (parent-child relationships) from JSON files.
-
-**Business Context:**
-
-- Product data includes parent products and child variants (size, color, etc.)
-- Source: Product Information Management (PIM) system exports JSON
-- Destination: Fluent Product entities
-- Complexity: Parent-child relationships require nested mapping
-
-### Source Data Formats
-
-**JSON Format (with variants):**
-
-```json
-{
- "products": [
-  {
-   "productId": "PROD001",
-   "name": "Classic T-Shirt",
-   "description": "Premium cotton t-shirt",
-   "brand": "MyBrand",
-   "category": "Apparel",
-   "variants": [
-    {
-     "sku": "PROD001-S-RED",
-     "size": "S",
-     "color": "Red",
-     "price": 29.99,
-     "barcode": "123456789001"
-    },
-    {
-     "sku": "PROD001-M-RED",
-     "size": "M",
-     "color": "Red",
-     "price": 29.99,
-     "barcode": "123456789002"
-    }
-   ]
-  }
- ]
-}
-```
-
-**CSV Format (flattened variants):**
-
-```csv
-product_id,product_name,sku,size,color,price,barcode
-PROD001,Classic T-Shirt,PROD001-S-RED,S,Red,29.99,123456789001
-PROD001,Classic T-Shirt,PROD001-M-RED,M,Red,29.99,123456789002
-```
-
-### Field Mapping Configuration
-
-**`config/product-mapping.json`:**
-
-```json
-{
- "version": "1.0",
- "description": "Map PIM product data to Fluent Product schema",
- "fields": {
-  "ref": {
-   "source": "productId",
-   "required": true
-  },
-  "type": {
-   "value": "STANDARD",
-   "required": true
-  },
-  "name": {
-   "source": "name",
-   "required": true
-  },
-  "summary": {
-   "source": "description"
-  },
-  "gtin": {
-   "source": "barcode"
-  },
-  "status": {
-   "value": "ACTIVE"
-  },
-  "retailerId": {
-   "value": "${RETAILER_ID}"
-  },
-  "attributes": {
-   "fields": {
-    "brand": { "source": "brand" },
-    "category": { "source": "category" }
-   }
-  },
-  "variants": {
-   "source": "variants",
-   "isArray": true,
-   "fields": {
-    "ref": { "source": "$.sku", "required": true },
-    "attributes": {
-     "fields": {
-      "size": { "source": "$.size" },
-      "color": { "source": "$.color" }
-     }
-    },
-    "prices": {
-     "fields": {
-      "currency": { "value": "USD" },
-      "value": { "source": "$.price", "resolver": "sdk.parseFloat" }
-     }
-    },
-    "gtin": { "source": "$.barcode" }
-   }
-  }
- }
-}
-```
-
-**Key Features:**
-
-- ✅ Parent-child relationships (`variants[]` array mapping)
-- ✅ Relative paths within arrays (`$.sku` references current variant)
-- ✅ Nested objects (`attributes`, `prices`)
-- ✅ Static values (`type: "STANDARD"`, `currency: "USD"`)
-
-### Complete Working Code
-
-**`product-etl.ts`:**
-
-```typescript
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-import { S3DataSource, JSONParserService } from '@fluentcommerce/fc-connect-sdk';
-import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-// Access openKv from context: const { openKv } = ctx;
-import * as fs from 'fs';
-
-/**
- * Product Catalog ETL with Parent-Child Relationships
- */
-export async function productCatalogETL(ctx: any) {
- const logger = console;
- const { openKv } = ctx;
- const fluentClient = await createClient({
-  config: {
-   baseUrl: process.env.FLUENT_BASE_URL!,
-   clientId: process.env.FLUENT_CLIENT_ID!,
-   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-   retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  logger,
- });
-
- // Initialize components
- const s3DataSource = new S3DataSource(
-  {
-   type: 'S3_CSV',
-   s3Config: {
-    bucket: 'product-catalog',
-    region: process.env.AWS_REGION!,
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-   },
-  },
-  logger
- );
-
- const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
- const stateService = new StateService(logger);
-
- const jsonParser = new JSONParserService();
-
- // Load mapping configuration
- const mappingConfig = JSON.parse(fs.readFileSync('config/product-mapping.json', 'utf-8'));
-
- const mapper = new UniversalMapper(mappingConfig, { logger, fluentClient });
-
- logger.info('Starting product catalog ETL');
-
- try {
-  // List JSON files
-  const files = await s3DataSource.listFiles({ prefix: 'products/' });
-
-  for (const file of files) {
-   if (!file.name.endsWith('.json')) continue;
-
-   const fileKey = `product:${file.name}`;
-
-   // Check if processed
-   if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
-    logger.info(`Skipping processed file: ${file.name}`);
-    continue;
-   }
-
-   logger.info(`Processing file: ${file.name}`);
-
-   try {
-    // Download and parse JSON
-    const fileContent = await s3DataSource.downloadFile(file.path);
-    const data = await jsonParser.parse(fileContent as string, {
-     dataPath: 'products', // Extract products array from root
-    });
-
-    const products = Array.isArray(data) ? data : [data];
-    logger.info(`Parsed ${products.length} products from ${file.name}`);
-
-    // Transform and load each product
-    for (const product of products) {
-     const result = await mapper.map(product);
-
-     if (result.success) {
-      // Create parent product
-      await createProduct(fluentClient, result.data, logger);
-
-      // Create variants if present
-      if (result.data.variants && result.data.variants.length > 0) {
-       for (const variant of result.data.variants) {
-        await createVariant(fluentClient, result.data.ref, variant, logger);
-       }
-      }
-     } else {
-      logger.error('Product mapping failed:', {
-       product,
-       errors: result.errors,
-      });
-     }
-    }
-
-    // Mark as processed
-    await stateService.updateSyncState(
-     kvAdapter,
-     [
-      {
-       fileName: file.name,
-       lastModified: new Date().toISOString(),
-       recordCount: products.length,
-      },
-     ],
-     'product-catalog-etl'
-    );
-
-    logger.info(`Successfully processed: ${file.name}`);
-   } catch (error) {
-    logger.error(`Failed to process file: ${file.name}`, error);
-    continue;
-   }
-  }
-
-  logger.info('Product catalog ETL completed');
- } catch (error) {
-  logger.error('Product catalog ETL failed', error);
-  throw error;
- }
-}
-
-/**
- * Create product via GraphQL
- */
-async function createProduct(client: any, productData: any, logger: any) {
- const mutation = `
-  mutation CreateProduct($input: CreateProductInput!) {
-   createProduct(input: $input) {
-    id
-    ref
-    name
-    status
-   }
-  }
- `;
-
- try {
-  const result = await client.graphql({
-   query: mutation,
-   variables: { input: productData },
-  });
-
-  logger.info(`Created product: ${productData.ref}`, result.data.createProduct);
- } catch (error) {
-  logger.error(`Failed to create product: ${productData.ref}`, error);
-  throw error;
- }
-}
-
-/**
- * Create variant via GraphQL
- */
-async function createVariant(client: any, parentRef: string, variantData: any, logger: any) {
- const mutation = `
-  mutation CreateVariant($input: CreateVariantInput!) {
-   createVariant(input: $input) {
-    id
-    ref
-    attributes
-   }
-  }
- `;
-
- try {
-  const result = await client.graphql({
-   query: mutation,
-   variables: {
-    input: {
-     ...variantData,
-     parentRef,
-    },
-   },
-  });
-
-  logger.info(`Created variant: ${variantData.ref}`, result.data.createVariant);
- } catch (error) {
-  logger.error(`Failed to create variant: ${variantData.ref}`, error);
-  // Don't throw - continue with other variants
- }
-}
-
-// Example usage
-if (require.main === module) {
- productCatalogETL()
-  .then(() => console.log('ETL completed'))
-  .catch(err => {
-   console.error('ETL failed:', err);
-   process.exit(1);
-  });
-}
-```
-
-**Key Design Decisions:**
-
-1. **Parent-Child Processing**: Create parent product first, then variants
-2. **Array Mapping**: `variants[]` notation handles nested arrays
-3. **Relative Paths**: `$.sku` resolves relative to current variant item
-4. **Error Isolation**: Variant creation failures don't stop parent processing
-
----
-
-## Example 3: Control/Config Data
-
-### Overview
-
-Load business rules and configuration parameters from JSON files.
-
-**Business Context:**
-
-- Controls define business rules, thresholds, flags
-- Source: Configuration management system exports JSON
-- Destination: Fluent Control entities
-- Characteristics: Simple structure, validation logic important
-
-### Source Data Format
-
-**JSON Format:**
-
-```json
-{
- "controls": [
-  {
-   "name": "ORDER_TIMEOUT_MINUTES",
-   "value": "30",
-   "type": "INTEGER",
-   "context": "ORDER_MANAGEMENT",
-   "description": "Order allocation timeout in minutes"
-  },
-  {
-   "name": "ENABLE_AUTO_ALLOCATION",
-   "value": "true",
-   "type": "BOOLEAN",
-   "context": "ORDER_MANAGEMENT",
-   "description": "Enable automatic order allocation"
-  },
-  {
-   "name": "DEFAULT_CARRIER",
-   "value": "FEDEX",
-   "type": "STRING",
-   "context": "FULFILLMENT",
-   "description": "Default shipping carrier"
-  }
- ]
-}
-```
-
-### Field Mapping Configuration
-
-**`config/control-mapping.json`:**
-
-```json
-{
- "version": "1.0",
- "description": "Map configuration controls to Fluent Control schema",
- "fields": {
-  "name": {
-   "source": "name",
-   "required": true,
-   "resolver": "sdk.uppercase"
-  },
-  "value": {
-   "source": "value",
-   "required": true,
-   "resolver": "custom.validateControlValue"
-  },
-  "type": {
-   "source": "type",
-   "required": true,
-   "resolver": "sdk.uppercase"
-  },
-  "context": {
-   "source": "context",
-   "defaultValue": "GLOBAL"
-  },
-  "description": {
-   "source": "description"
-  },
-  "status": {
-   "value": "ACTIVE"
-  },
-  "retailerId": {
-   "value": "${RETAILER_ID}"
-  }
- }
-}
-```
-
-### Validation Logic
-
-**Custom resolver for control value validation:**
-
-```typescript
-import { FieldResolverFunction } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Validate control value based on type
- */
-export const validateControlValue: FieldResolverFunction = (
- value: any,
- sourceData: any,
- config: any,
- helpers: any
-) => {
- const type = sourceData.type;
-
- switch (type) {
-  case 'INTEGER':
-   const intValue = helpers.parseIntSafe(value, null);
-   if (intValue === null) {
-    throw new Error(`Invalid INTEGER value: ${value}`);
-   }
-   return intValue.toString();
-
-  case 'FLOAT':
-   const floatValue = helpers.parseFloatSafe(value, null);
-   if (floatValue === null) {
-    throw new Error(`Invalid FLOAT value: ${value}`);
-   }
-   return floatValue.toString();
-
-  case 'BOOLEAN':
-   if (!['true', 'false'].includes(String(value).toLowerCase())) {
-    throw new Error(`Invalid BOOLEAN value: ${value}`);
-   }
-   return String(value).toLowerCase();
-
-  case 'STRING':
-   return String(value);
-
-  case 'JSON':
-   try {
-    JSON.parse(value);
-    return value;
-   } catch (error) {
-    throw new Error(`Invalid JSON value: ${value}`);
-   }
-
-  default:
-   helpers.log.warn(`Unknown control type: ${type}, treating as STRING`);
-   return String(value);
- }
-};
-```
-
-### Complete Working Code
-
-**`control-etl.ts`:**
-
-```typescript
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-import { S3DataSource, JSONParserService } from '@fluentcommerce/fc-connect-sdk';
-import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-// Access openKv from context: const { openKv } = ctx;
-import * as fs from 'fs';
-import { validateControlValue } from './resolvers/control-validators';
-
-/**
- * Control/Config Data ETL
- */
-export async function controlDataETL(ctx: any) {
- const logger = console;
- const { openKv } = ctx;
- const fluentClient = await createClient({
-  config: {
-   baseUrl: process.env.FLUENT_BASE_URL!,
-   clientId: process.env.FLUENT_CLIENT_ID!,
-   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-   retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
-  logger,
- });
-
- // Initialize components
- const s3DataSource = new S3DataSource(
-  {
-   type: 'S3_CSV',
-   s3Config: {
-    bucket: 'config-data',
-    region: process.env.AWS_REGION!,
-    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-   },
-  },
-  logger
- );
-
- const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
- const stateService = new StateService(logger);
-
- const jsonParser = new JSONParserService();
-
- // Load mapping configuration
- const mappingConfig = JSON.parse(fs.readFileSync('config/control-mapping.json', 'utf-8'));
-
- // Initialize mapper with custom resolvers
- const mapper = new UniversalMapper(mappingConfig, {
-  logger,
-  fluentClient,
-  customResolvers: {
-   'custom.validateControlValue': validateControlValue,
-  },
- });
-
- logger.info('Starting control data ETL');
-
- try {
-  // List JSON files
-  const files = await s3DataSource.listFiles({ prefix: 'controls/' });
-
-  for (const file of files) {
-   if (!file.name.endsWith('.json')) continue;
-
-   const fileKey = `control:${file.name}`;
-
-   // Check if processed
-   if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
-    logger.info(`Skipping processed file: ${file.name}`);
-    continue;
-   }
-
-   logger.info(`Processing file: ${file.name}`);
-
-   try {
-    // Download and parse JSON
-    const fileContent = await s3DataSource.downloadFile(file.path);
-    const data = await jsonParser.parse(fileContent as string, {
-     dataPath: 'controls',
-    });
-
-    const controls = Array.isArray(data) ? data : [data];
-    logger.info(`Parsed ${controls.length} controls from ${file.name}`);
-
-    // Transform and load each control
-    let successCount = 0;
-    let failureCount = 0;
-
-    for (const control of controls) {
-     try {
-      const result = await mapper.map(control);
-
-      if (result.success) {
-       await createControl(fluentClient, result.data, logger);
-       successCount++;
-      } else {
-       logger.error('Control mapping failed:', {
-        control,
-        errors: result.errors,
-       });
-       failureCount++;
-      }
-     } catch (error) {
-      logger.error(`Failed to process control: ${control.name}`, error);
-      failureCount++;
-     }
-    }
-
-    logger.info(`Control processing summary:`, {
-     total: controls.length,
-     success: successCount,
-     failures: failureCount,
-    });
-
-    // Mark as processed
-    await stateService.updateSyncState(
-     kvAdapter,
-     [
-      {
-       fileName: file.name,
-       lastModified: new Date().toISOString(),
-       recordCount: controls.length,
-      },
-     ],
-     'control-data-etl'
-    );
-
-    logger.info(`Successfully processed: ${file.name}`);
-   } catch (error) {
-    logger.error(`Failed to process file: ${file.name}`, error);
-    continue;
-   }
-  }
-
-  logger.info('Control data ETL completed');
- } catch (error) {
-  logger.error('Control data ETL failed', error);
-  throw error;
- }
-}
-
-/**
- * Create control via GraphQL
- */
-async function createControl(client: any, controlData: any, logger: any) {
- const mutation = `
-  mutation CreateControl($input: CreateControlInput!) {
-   createControl(input: $input) {
-    id
-    name
-    value
-    type
-    context
-   }
-  }
- `;
-
- try {
-  const result = await client.graphql({
-   query: mutation,
-   variables: { input: controlData },
-  });
-
-  logger.info(`Created control: ${controlData.name}`, {
-   value: controlData.value,
-   type: controlData.type,
-  });
- } catch (error) {
-  logger.error(`Failed to create control: ${controlData.name}`, error);
-  throw error;
- }
-}
-
-// Example usage
-if (require.main === module) {
- controlDataETL()
-  .then(() => console.log('ETL completed'))
-  .catch(err => {
-   console.error('ETL failed:', err);
-   process.exit(1);
-  });
-}
-```
-
-**Key Design Decisions:**
-
-1. **Type Validation**: Custom resolver validates value based on type field
-2. **Error Tracking**: Count successes/failures for summary reporting
-3. **Atomic Processing**: Each control processed independently
-4. **Detailed Logging**: Track validation failures with context
-
----
-
-## Source Strategies
-
-### S3 with Event Notifications
-
-**Use Case**: Process files as soon as they're uploaded to S3
-
-**Setup:**
-
-```typescript
-import { webhook } from '@versori/run/webhooks';
-import { masterDataETL } from './location-etl';
-
-export const s3LocationETL = webhook('s3-location-upload', {
- response: { mode: 'sync' },
-})
- .then(async ({ data }) => {
-  // Parse S3 event notification
-  const s3Event = data.Records[0].s3;
-  const bucket = s3Event.bucket.name;
-  const key = decodeURIComponent(s3Event.object.key.replace(/\+/g, ' '));
-
-  console.log(`S3 event received: ${bucket}/${key}`);
-
-  // Run ETL for this specific file
-  await masterDataETL('config/location-etl-config.json');
-
-  return { success: true, message: 'Location ETL completed' };
- })
- .catch(({ error }) => {
-  console.error('S3 location ETL failed:', error);
-  return { success: false, error: error.message };
- });
-```
-
-**S3 Bucket Configuration:**
-
-```json
-{
- "LambdaFunctionConfigurations": [
-  {
-   "LambdaFunctionArn": "arn:aws:lambda:...:function:versori-webhook",
-   "Events": ["s3:ObjectCreated:*"],
-   "Filter": {
-    "Key": {
-     "FilterRules": [
-      {
-       "Name": "prefix",
-       "Value": "locations/"
-      },
-      {
-       "Name": "suffix",
-       "Value": ".csv"
-      }
-     ]
-    }
-   }
-  }
- ]
-}
-```
-
-### SFTP with Polling
-
-**Use Case**: Poll SFTP server periodically for new files
-
-#### SFTP Credential Access
-
-**Versori Platform** has three methods for accessing SFTP credentials:
-
-1. **Connection Variables (Recommended)** - Direct access to connection config:
-  ```typescript
-  const { host, port, username, password, privateKey } = ctx.activation.connections.sftp_server;
-  ```
-
-2. **Credentials API** - For base64-encoded credentials:
-  ```typescript
-  const creds = await ctx.credentials().getAccessToken('sftp_server');
-  ```
-
-3. **Connection String Parsing** - Decode `connectionVariables.connectionString`:
-  ```typescript
-  const connStr = ctx.activation.connections.sftp_server.connectionString;
-  // Parse: sftp://username:password@host:port
-  ```
-
-**Standalone Node.js/Deno**: Use environment variables directly:
-```typescript
-const config = {
- host: process.env.SFTP_HOST!,
- port: parseInt(process.env.SFTP_PORT || '22'),
- username: process.env.SFTP_USERNAME!,
- password: process.env.SFTP_PASSWORD,
- privateKey: process.env.SFTP_PRIVATE_KEY,
-};
-```
-
-**Security Best Practices:**
-- Always prefer SSH keys over passwords
-- Never log credential values
-- Use scoped credentials (read-only when possible)
-- Rotate credentials regularly
-
-**See:** [SFTP Credential Access Security Guide](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for complete details.
-
-#### Setup Example
-
-```typescript
-import { schedule } from '@versori/run/schedule';
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-import { Buffer } from 'node:buffer'; // Required for Deno/Versori
-
-export const sftpPolling = schedule('location-sftp-poll', {
- cron: '0 */6 * * *', // Every 6 hours
- retry: { attempts: 3 },
-}).then(async (ctx) => {
- const { log, activation } = ctx;
-
- // Access SFTP credentials (Versori)
- const { host, port, username, password, privateKey } = activation.connections.sftp_server;
-
- // Initialize SFTP data source
- const sftpSource = new SftpDataSource(
-  {
-   type: 'SFTP_CSV',
-   settings: {
-    host,
-    port: port || 22,
-    username,
-    password,
-    privateKey,
-    remotePath: '/data/locations',
-    filePattern: '*.csv',
-   },
-  },
-  log
- );
-
- // List new files
- const files = await sftpSource.listFiles();
- log.info(`Found ${files.length} files on SFTP`);
-
- for (const file of files) {
-  try {
-   // Download file
-   const content = await sftpSource.downloadFile(file.path);
-
-   // Process file (same ETL logic as S3)
-   // ... (extract, parse, transform, load)
-
-   // Move to processed folder
-   await sftpSource.moveFile(file.path, `/data/locations/processed/${file.name}`);
-  } catch (error) {
-   log.error(`Failed to process ${file.name}:`, error);
-  }
- }
-
- return { success: true, filesProcessed: files.length };
-});
-```
-
----
-
-## Load Strategies
-
-### GraphQL Mutation Approach
-
-**When to Use:**
-
-- Simple entity creation/updates
-- Direct control over mutations
-- Schema validation needed
-- Small to medium datasets (<10K records)
-
-**Advantages:**
-
-- ✅ Type-safe with GraphQL schema
-- ✅ Immediate validation feedback
-- ✅ Fine-grained control over mutations
-- ✅ Can return created IDs
-
-**Example:**
-
-```typescript
-async function loadViaGraphQL(records: any[], mutation: string, logger: any) {
- const client = await createClient({
-  config: {
-   baseUrl: process.env.FLUENT_BASE_URL!,
-   clientId: process.env.FLUENT_CLIENT_ID!,
-   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-   retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
- });
-
- for (const record of records) {
-  const query = `
-   mutation ${mutation}($input: ${capitalize(mutation)}Input!) {
-    ${mutation}(input: $input) {
-     id
-     ref
-    }
-   }
-  `;
-
-  try {
-   const result = await client.graphql({
-    query,
-    variables: { input: record },
-   });
-
-   logger.info(`Created ${mutation}:`, result.data[mutation]);
-  } catch (error: any) {
-   logger.error(`Failed ${mutation}:`, {
-    record,
-    error: error.message,
-    details: error.response?.errors,
-   });
-  }
- }
-}
-```
-
-### Event API Approach
-
-**When to Use:**
-
-- Asynchronous processing acceptable
-- Triggering workflows/rules needed
-- Large datasets (>10K records)
-- Need event-driven architecture
-
-**Advantages:**
-
-- ✅ Asynchronous (better for large datasets)
-- ✅ Triggers workflows and rules
-- ✅ Decoupled from mutations
-- ✅ Better performance for bulk loads
-
-**Example:**
-
-```typescript
-async function loadViaEventAPI(records: any[], eventName: string, logger: any) {
- const client = await createClient({
-  config: {
-   baseUrl: process.env.FLUENT_BASE_URL!,
-   clientId: process.env.FLUENT_CLIENT_ID!,
-   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-   retailerId: process.env.FLUENT_RETAILER_ID!,
-  },
- });
-
- for (const record of records) {
-  try {
-   await client.sendEvent({
-    name: eventName,
-    entityRef: record.ref,
-    entityType: 'LOCATION',
-    retailerId: record.retailerId,
-    attributes: record,
-   });
-
-   logger.info(`Sent event ${eventName} for ${record.ref}`);
-  } catch (error: any) {
-   logger.error(`Failed to send event for ${record.ref}:`, error);
-  }
- }
-}
-```
-
-**Event-Driven Workflow:**
-
-```
-ETL Process          Fluent Commerce
-  ↓
-Send Event (LOCATION_CREATED)
-  ↓               ↓
-              Workflow Triggers
-                  ↓
-              Process Event
-                  ↓
-              Create Location
-                  ↓
-              Apply Business Rules
-                  ↓
-              Send Notifications
-```
-
----
-
-## Configuration Schema
-
-### Generic Configuration Template
-
-Use this template for ANY entity type:
-
-```json
-{
- "entityType": "<entity-name>",
- "description": "<what this ETL does>",
- "sourceConfig": {
-  "type": "S3_CSV | SFTP_CSV | S3_JSON | SFTP_JSON",
-  "bucket": "<s3-bucket-name>",
-  "prefix": "<folder-prefix>",
-  "filePattern": "*.csv | *.json | *.xml",
-  "sftp": {
-   "host": "${SFTP_HOST}",
-   "port": 22,
-   "username": "${SFTP_USERNAME}",
-   "password": "${SFTP_PASSWORD}",
-   "privateKey": "${SFTP_PRIVATE_KEY}", // Recommended over password
-   "remotePath": "/data/<entity>",
-   "filePattern": "*.<format>"
-  }
- },
- "parseConfig": {
-  "format": "csv | json | xml",
-  "delimiter": "," | "|" | "\t",
-  "headers": true | false,
-  "encoding": "utf8 | utf16",
-  "json": {
-   "dataPath": "root.path.to.array",
-   "jsonLines": false
-  },
-  "xml": {
-   "itemPath": "//Item",
-   "includeAttributes": true
-  }
- },
- "mappingConfig": {
-  "version": "1.0",
-  "description": "Field mappings",
-  "fields": {
-   "targetField": {
-    "source": "sourceField",
-    "required": true | false,
-    "defaultValue": "default",
-    "resolver": "sdk.* | custom.*"
-   }
-  }
- },
- "loadConfig": {
-  "strategy": "graphql | event",
-  "mutation": "createEntity | updateEntity",
-  "eventName": "ENTITY_CREATED",
-  "batchSize": 100,
-  "retryAttempts": 3
- },
- "scheduleConfig": {
-  "enabled": true,
-  "cron": "0 0 * * *",
-  "timezone": "UTC"
- }
-}
-```
-
-### How to Adapt for New Entity Types
-
-**Step-by-Step Guide:**
-
-1. **Copy Template**: Start with generic configuration template
-2. **Set Entity Type**: Change `entityType` to your entity name
-3. **Configure Source**: Set bucket/path for your data source
-4. **Configure Parser**: Set format and parsing options
-5. **Define Field Mappings**: Map source fields to Fluent schema
-6. **Configure Load Strategy**: Choose GraphQL or Event API
-7. **Test**: Run ETL with sample data
-8. **Deploy**: Schedule or trigger via webhook
-
-**Example Adaptation (Customer Entity):**
-
-```json
-{
- "entityType": "customer",
- "description": "Load customer data from CRM system",
- "sourceConfig": {
-  "type": "S3_CSV",
-  "bucket": "crm-exports",
-  "prefix": "customers/",
-  "filePattern": "customers_*.csv"
- },
- "parseConfig": {
-  "format": "csv",
-  "delimiter": ",",
-  "headers": true
- },
- "mappingConfig": {
-  "version": "1.0",
-  "fields": {
-   "ref": {
-    "source": "customer_id",
-    "required": true
-   },
-   "firstName": {
-    "source": "first_name",
-    "required": true
-   },
-   "lastName": {
-    "source": "last_name",
-    "required": true
-   },
-   "email": {
-    "source": "email_address",
-    "required": true,
-    "resolver": "sdk.lowercase"
-   },
-   "primaryPhone": {
-    "source": "phone_number",
-    "resolver": "custom.formatPhoneNumber"
-   },
-   "retailerId": {
-    "value": "${RETAILER_ID}"
-   }
-  }
- },
- "loadConfig": {
-  "strategy": "graphql",
-  "mutation": "createCustomer",
-  "batchSize": 50
- }
-}
-```
-
----
-
-## Extending to Other Entities
-
-### Step-by-Step Guide
-
-**1. Identify Entity Schema**
-
-Understand the Fluent schema for your entity:
-
-```graphql
-# Example: Carrier schema
-type Carrier {
- id: ID!
- ref: String!
- name: String!
- type: String!
- status: String
- services: [CarrierService]
- retailerId: ID!
-}
-
-input CreateCarrierInput {
- ref: String!
- name: String!
- type: String!
- status: String
- services: [CarrierServiceInput]
- retailerId: ID!
-}
-```
-
-**2. Create Field Mapping Configuration**
-
-Map source data to schema:
-
-```json
-{
- "version": "1.0",
- "fields": {
-  "ref": {
-   "source": "carrier_code",
-   "required": true,
-   "resolver": "sdk.uppercase"
-  },
-  "name": {
-   "source": "carrier_name",
-   "required": true
-  },
-  "type": {
-   "source": "carrier_type",
-   "required": true
-  },
-  "status": {
-   "value": "ACTIVE"
-  },
-  "services": {
-   "source": "services",
-   "isArray": true,
-   "fields": {
-    "name": { "source": "$.service_name" },
-    "code": { "source": "$.service_code" },
-    "deliveryDays": { "source": "$.delivery_days", "resolver": "sdk.parseInt" }
-   }
-  },
-  "retailerId": {
-   "value": "${RETAILER_ID}"
-  }
- }
-}
-```
-
-**3. Set Up Data Source**
-
-Configure where data comes from:
-
-```typescript
-const carrierSource = new S3DataSource(
- {
-  type: 'S3_CSV',
-  s3Config: {
-   bucket: 'carrier-data',
-   region: process.env.AWS_REGION!,
-   accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-   secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-  },
- },
- logger
-);
-```
-
-**4. Run Generic ETL Pipeline**
-
-Use the same ETL code:
-
-```typescript
-await masterDataETL('config/carrier-etl-config.json');
-```
-
-### Customer Example
-
-**Source CSV:**
-
-```csv
-customer_id,first_name,last_name,email,phone,segment,status
-CUST001,John,Doe,john.doe@email.com,555-1234,VIP,ACTIVE
-CUST002,Jane,Smith,jane.smith@email.com,555-5678,STANDARD,ACTIVE
-```
-
-**Mapping Configuration:**
-
-```json
-{
- "version": "1.0",
- "fields": {
-  "ref": { "source": "customer_id", "required": true },
-  "firstName": { "source": "first_name", "required": true },
-  "lastName": { "source": "last_name", "required": true },
-  "email": { "source": "email", "required": true, "resolver": "sdk.lowercase" },
-  "primaryPhone": { "source": "phone" },
-  "status": { "source": "status", "defaultValue": "ACTIVE" },
-  "attributes": {
-   "fields": {
-    "segment": { "source": "segment" }
-   }
-  },
-  "retailerId": { "value": "${RETAILER_ID}" }
- }
-}
-```
-
-### Carrier Example
-
-**Source JSON:**
-
-```json
-{
- "carriers": [
-  {
-   "code": "FEDEX",
-   "name": "FedEx",
-   "type": "PARCEL",
-   "services": [
-    { "service_code": "GROUND", "service_name": "FedEx Ground", "delivery_days": 3 },
-    { "service_code": "2DAY", "service_name": "FedEx 2 Day", "delivery_days": 2 }
-   ]
-  }
- ]
-}
-```
-
-**Mapping Configuration:**
-
-```json
-{
- "version": "1.0",
- "fields": {
-  "ref": { "source": "code", "required": true },
-  "name": { "source": "name", "required": true },
-  "type": { "source": "type", "required": true },
-  "status": { "value": "ACTIVE" },
-  "services": {
-   "source": "services",
-   "isArray": true,
-   "fields": {
-    "code": { "source": "$.service_code" },
-    "name": { "source": "$.service_name" },
-    "transitDays": { "source": "$.delivery_days", "resolver": "sdk.parseInt" }
-   }
-  },
-  "retailerId": { "value": "${RETAILER_ID}" }
- }
-}
-```
-
-### Pricing Example
-
-**Source CSV:**
-
-```csv
-sku,price_list,currency,base_price,sale_price,start_date,end_date
-PROD001-S-RED,US_RETAIL,USD,29.99,24.99,2024-01-01,2024-12-31
-PROD001-M-RED,US_RETAIL,USD,29.99,24.99,2024-01-01,2024-12-31
-```
-
-**Mapping Configuration:**
-
-```json
-{
- "version": "1.0",
- "fields": {
-  "ref": {
-   "resolver": "custom.generatePriceRef",
-   "required": true
-  },
-  "sku": {
-   "source": "sku",
-   "required": true
-  },
-  "priceList": {
-   "source": "price_list",
-   "required": true
-  },
-  "currency": {
-   "source": "currency",
-   "required": true,
-   "resolver": "sdk.uppercase"
-  },
-  "value": {
-   "source": "base_price",
-   "required": true,
-   "resolver": "sdk.parseFloat"
-  },
-  "salePrice": {
-   "source": "sale_price",
-   "resolver": "sdk.parseFloat"
-  },
-  "validFrom": {
-   "source": "start_date",
-   "resolver": "sdk.formatDate"
-  },
-  "validTo": {
-   "source": "end_date",
-   "resolver": "sdk.formatDate"
-  },
-  "retailerId": {
-   "value": "${RETAILER_ID}"
-  }
- }
-}
-```
-
-**Custom Resolver:**
-
-```typescript
-export const generatePriceRef: FieldResolverFunction = (
- value: any,
- sourceData: any,
- config: any,
- helpers: any
-) => {
- // Generate unique ref: SKU-PRICELIST
- return `${sourceData.sku}-${sourceData.price_list}`;
-};
-```
-
----
-
-## Testing
-
-### Unit Testing Field Mappings
-
-```typescript
-import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-import * as fs from 'fs';
-
-describe('Location Mapping', () => {
- let mapper: UniversalMapper;
-
- beforeEach(() => {
-  const config = JSON.parse(fs.readFileSync('config/location-mapping.json', 'utf-8'));
-  mapper = new UniversalMapper(config);
- });
-
- it('should map location data correctly', async () => {
-  const sourceData = {
-   location_id: 'LOC001',
-   location_name: 'Downtown Store',
-   type: 'store',
-   status: 'active',
-   address_line1: '123 Main St',
-   city: 'New York',
-   state: 'NY',
-   zip: '10001',
-   country: 'US',
-   latitude: '40.7128',
-   longitude: '-74.0060',
-  };
-
-  const result = await mapper.map(sourceData);
-
-  expect(result.success).toBe(true);
-  expect(result.data).toMatchObject({
-   ref: 'LOC001',
-   name: 'Downtown Store',
-   type: 'STORE',
-   status: 'ACTIVE',
-   primaryAddress: {
-    street: '123 Main St',
-    city: 'New York',
-    state: 'NY',
-    postcode: '10001',
-    country: 'US',
-   },
-   coordinates: {
-    latitude: 40.7128,
-    longitude: -74.006,
-   },
-  });
- });
-
- it('should handle required field validation', async () => {
-  const sourceData = {
-   location_name: 'Store Without ID',
-   // Missing location_id (required)
-  };
-
-  const result = await mapper.map(sourceData);
-
-  expect(result.success).toBe(false);
-  expect(result.errors).toContain("Required field 'ref' is missing or empty");
- });
-
- it('should apply default values', async () => {
-  const sourceData = {
-   location_id: 'LOC001',
-   location_name: 'Store',
-   type: 'STORE',
-   // Missing status - should default to "ACTIVE"
-  };
-
-  const result = await mapper.map(sourceData);
-
-  expect(result.success).toBe(true);
-  expect(result.data.status).toBe('ACTIVE');
- });
-});
-```
-
-### Integration Testing ETL Pipeline
-
-```typescript
-import { masterDataETL } from './location-etl';
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-import * as fs from 'fs';
-
-describe('Location ETL Integration', () => {
- let s3DataSource: S3DataSource;
-
- beforeEach(() => {
-  s3DataSource = new S3DataSource(
-   {
-    type: 'S3_CSV',
-    s3Config: {
-     bucket: 'test-bucket',
-     region: 'us-east-1',
-     accessKeyId: process.env.TEST_AWS_KEY!,
-     secretAccessKey: process.env.TEST_AWS_SECRET!,
-    },
-   },
-   console
-  );
- });
-
- it('should process location CSV file end-to-end', async () => {
-  // Upload test file to S3
-  const testData = `location_id,location_name,type,status
-LOC001,Test Store,STORE,ACTIVE
-LOC002,Test Warehouse,WAREHOUSE,ACTIVE`;
-
-  await s3DataSource.uploadFile('locations/test.csv', testData, { contentType: 'text/csv' });
-
-  // Run ETL
-  await masterDataETL('config/location-etl-config.json');
-
-  // Verify locations were created (query Fluent API)
-  const result = await fluentClient.graphql({
-   query: `
-    query {
-     locations(first: 10, filter: { ref: { in: ["LOC001", "LOC002"] } }) {
-      edges {
-       node {
-        ref
-        name
-        type
-       }
-      }
-     }
-    }
-   `,
-  });
-
-  expect(result.data.locations.edges).toHaveLength(2);
-  expect(result.data.locations.edges[0].node.ref).toBe('LOC001');
- }, 30000); // 30s timeout for integration test
-});
-```
-
-### End-to-End Testing
-
-```typescript
-import { masterDataETL } from './location-etl';
-
-describe('Location ETL E2E', () => {
- it('should handle full ETL lifecycle', async () => {
-  // 1. Upload test data to S3
-  // 2. Trigger ETL process
-  // 3. Verify data in Fluent
-  // 4. Verify state tracking (file marked as processed)
-  // 5. Verify idempotency (re-running doesn't duplicate)
-  // TODO: Implement full E2E test scenario
- });
-});
-```
-
----
-
-## Common Issues
-
-### Issue: Duplicate Records Created
-
-**Symptom**: Same entity created multiple times
-
-**Root Cause**: State management not working or file processed multiple times
-
-**Solution:**
-
-```typescript
-// Ensure state service is initialized
-const kvAdapter = new VersoriKVAdapter(openKv());
-const stateService = new StateService(logger);
-
-// Check BEFORE processing
-const fileKey = `${entityType}:${file.name}`;
-if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
- logger.info(`Skipping already processed file: ${file.name}`);
- continue;
-}
-
-// Mark AFTER successful processing
-await stateService.updateSyncState(
- kvAdapter,
- [
-  {
-   fileName: file.name,
-   lastModified: new Date().toISOString(),
-   recordCount: records.length,
-  },
- ],
- 'master-data-etl'
-);
-```
-
-### Issue: Field Mapping Errors
-
-**Symptom**: "Required field missing" or "Mapping failed"
-
-**Root Cause**: Source field name doesn't match configuration
-
-**Solution:**
-
-```typescript
-// Debug source data structure
-logger.debug('Source data:', JSON.stringify(sourceData, null, 2));
-
-// Check field names match exactly (case-sensitive)
-{
- "ref": {
-  "source": "location_id", // Must match CSV header exactly
-  "required": true
- }
-}
-
-// Use custom resolver for flexible field access
-{
- "ref": {
-  "resolver": "custom.extractRef",
-  "required": true
- }
-}
-```
-
-### Issue: Large File Memory Issues
-
-**Symptom**: Out of memory errors with large CSV/JSON files
-
-**Root Cause**: Loading entire file into memory
-
-**Solution:**
-
-```typescript
-// Use streaming parsers for large files
-const csvParser = new CSVParserService();
-
-// Parse with streaming (yields records one-by-one)
-for await (const record of csvParser.parseStreaming(fileContent)) {
- const result = await mapper.map(record);
- if (result.success) {
-  await loadRecord(result.data);
- }
-}
-
-// Or batch process
-for await (const batch of csvParser.parseStreaming(fileContent, {}, 100)) {
- await loadBatch(batch);
-}
-```
-
-### Issue: GraphQL Mutation Timeouts
-
-**Symptom**: Mutations timing out for large datasets
-
-**Root Cause**: Synchronous processing of many records
-
-**Solution:**
-
-```typescript
-// Use batching and concurrency limits
-import pLimit from 'p-limit';
-
-const limit = pLimit(5); // Max 5 concurrent mutations
-
-const promises = records.map(record => limit(() => createViaGraphQL(record)));
-
-await Promise.all(promises);
-
-// Or use Event API for async processing
-await loadViaEventAPI(records, 'LOCATION_CREATED', logger);
-```
-
-### Issue: Type Coercion Errors
-
-**Symptom**: "Expected number, got string" in GraphQL mutations
-
-**Root Cause**: CSV parsers return all values as strings
-
-**Solution:**
-
-```typescript
-// Use SDK resolvers for type coercion
-{
- "latitude": {
-  "source": "latitude",
-  "resolver": "sdk.parseFloat" // String → number
- },
- "active": {
-  "source": "is_active",
-  "resolver": "sdk.boolean" // "true" → true
- },
- "quantity": {
-  "source": "qty",
-  "resolver": "sdk.parseInt" // "10" → 10
- }
-}
-```
-
----
-
-## Related Guides
-
-### SDK Documentation
-
-- [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Complete field mapping reference
-- [S3 Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - S3 integration details
-- [SFTP Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - SFTP integration details
-- [SFTP Credential Access Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) - Secure credential handling for SFTP
-- [CSV Parser](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - CSV parsing options
-- [JSON Parser](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - JSON parsing options
-- [State Management](../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md) - Preventing duplicates
-
-### Use Case Patterns
-
-- [Inventory Ingestion](../../02-CORE-GUIDES/ingestion/ingestion-readme.md) - Similar pattern for inventory data
-- [Order Integration](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md) - Transaction data vs master data
-- [Catalog Sync](../../01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md) - Product catalog patterns
-
-### Platform Integration
-
-- [Versori Connector Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Deploy ETL as connector
-- [Webhook Triggers](../../04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md#webhook-functions-receiving-external-requests) - Event-driven ETL
-- [Scheduled Jobs](../../04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md#scheduled-functions-time-based-recurring-tasks) - Periodic ETL execution
-
----
-
-## Summary
-
-This **Master Data ETL Pattern** provides a **generic, configuration-driven framework** for loading ANY entity type into Fluent Commerce.
-
-**Key Takeaways:**
-
-1. ✅ **One Pattern for All Entities**: Same code works for locations, products, controls, carriers, etc.
-2. ✅ **Configuration-Driven**: No code changes needed for new entity types
-3. ✅ **Four-Phase Pipeline**: Extract → Parse → Transform → Load
-4. ✅ **Multiple Source Formats**: CSV, JSON, XML support
-5. ✅ **Multiple Load Strategies**: GraphQL mutations or Event API
-6. ✅ **Production-Ready**: State management, error handling, logging
-
-**Getting Started:**
-
-1. Copy the generic ETL code
-2. Create field mapping configuration for your entity
-3. Configure data source (S3/SFTP)
-4. Run ETL pipeline
-5. Monitor logs and verify data in Fluent
-
-**Next Steps:**
-
-- Review the [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for advanced mapping patterns
-- Explore [S3 Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for source configuration options
-- Check [Versori Connector Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) to deploy as a production connector
-
----
-
-**Need Help?**
-
-- 📖 Documentation: `fc-connect-sdk/docs/`
-- 💬 Support: Fluent Commerce support team
-- 🐛 Issues: GitHub repository issues
+# Pattern: Master Data ETL - Generic Framework for Loading Any Entity
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: Use latest - `npm install @fluentcommerce/fc-connect-sdk@latest`
+
+**Status**: Production Ready
+
+**Complexity**: Intermediate
+
+**Est. Time**: 30-60 minutes
+
+**Use Cases**: Locations, Products, Controls, Carriers, Customers, Pricing, Categories, etc.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [What You'll Build](#what-youll-build)
+- [SDK Methods Used](#sdk-methods-used)
+- [The Generic Pattern](#the-generic-pattern)
+- [Example 1: Location Master Data](#example-1-location-master-data)
+- [Example 2: Product Catalog](#example-2-product-catalog)
+- [Example 3: Control/Config Data](#example-3-controlconfig-data)
+- [Source Strategies](#source-strategies)
+- [Load Strategies](#load-strategies)
+- [Configuration Schema](#configuration-schema)
+- [Extending to Other Entities](#extending-to-other-entities)
+- [Testing](#testing)
+- [Common Issues](#common-issues)
+- [Related Guides](#related-guides)
+
+---
+
+## Overview
+
+Master data ETL is the process of extracting reference data from external systems and loading it into Fluent Commerce. Unlike transactional data (orders, inventory updates), master data changes infrequently and defines the core entities of your commerce platform.
+
+**Common Master Data Entities:**
+
+- **Locations**: Stores, warehouses, distribution centers
+- **Products**: SKUs, product catalogs, variants
+- **Controls**: Business rules, configuration parameters
+- **Carriers**: Shipping carriers, service levels
+- **Customers**: Customer profiles, segments
+- **Categories**: Product taxonomies, merchandising hierarchies
+- **Pricing**: Price lists, promotional rules
+
+**Why This Pattern?**
+
+This guide provides a **generic, configuration-driven framework** that works for ANY master data entity. Instead of writing custom code for each entity type, you configure JSON mappings and reuse the same pipeline.
+
+**Key Benefits:**
+
+- ✅ **One Pattern for All Entities**: Same code works for locations, products, controls, etc.
+- ✅ **Configuration-Driven**: No code changes needed for new entity types
+- ✅ **Multiple Source Formats**: CSV, JSON, XML support out-of-the-box
+- ✅ **Multiple Load Strategies**: GraphQL mutations or Event API
+- ✅ **Automatic Deduplication**: State management prevents duplicate loads
+- ✅ **Production-Ready Error Handling**: Comprehensive logging and retry logic
+
+---
+
+## What You'll Build
+
+A **reusable ETL framework** with these capabilities:
+
+1. **Extract**: Read master data from S3/SFTP in CSV, JSON, or XML format
+2. **Parse**: Convert file format to JavaScript objects
+3. **Transform**: Map source fields to Fluent schema using field mappings
+4. **Load**: Submit to Fluent via GraphQL mutations or Event API
+5. **Track**: Prevent duplicate processing using state management
+
+**Pipeline Flow:**
+
+```
+Source File (S3/SFTP)
+  ↓
+Extract (DataSource)
+  ↓
+Parse (CSVParserService/JSONParser/XMLParser)
+  ↓
+Transform (UniversalMapper)
+  ↓
+Load (GraphQL Mutation or Event API)
+  ↓
+Track (StateService)
+```
+
+**Configuration-Driven Design:**
+
+Instead of hardcoded logic:
+
+```typescript
+// ❌ WRONG: Hardcoded for each entity type
+if (entityType === 'location') {
+ // Custom location loading logic
+} else if (entityType === 'product') {
+ // Custom product loading logic
+}
+```
+
+You use JSON configuration:
+
+```typescript
+// ✅ CORRECT: Generic pipeline with config
+const config = loadConfig('location-mapping.json');
+await etlPipeline.run(sourceFile, config);
+```
+
+> Tips:
+>
+> - For identifiers that may look numeric (SKU/GTIN/UPC), add `resolver: "sdk.toString"` in mappings to force string output.
+> - When parsing XML sources where leading zeros matter, configure the XML parser with `parseNumbers: false` to prevent numeric coercion.
+
+---
+
+## SDK Methods Used
+
+| Method                           | Purpose                | Pattern     |
+| ---------------------------------------------------------- | -------------------------------------- | --------------- |
+| `S3DataSource.downloadFile()`               | Download master data file from S3   | Source     |
+| `SftpDataSource.downloadFile()`              | Download master data file from SFTP  | Source     |
+| `CSVParserService.parse()`                 | Parse CSV master data         | Parse      |
+| `JSONParserService.parse()`                | Parse JSON master data         | Parse      |
+| `XMLParserService.parse()`                 | Parse XML master data         | Parse      |
+| `UniversalMapper.map()`                  | Transform source data to Fluent schema | Transform    |
+| `GraphQLMutationMapper.map()`               | Generate GraphQL mutations from data  | Load (Option 1) |
+| `FluentClient.graphql()`                  | Execute GraphQL mutations       | Load (Option 1) |
+| `FluentClient.sendEvent()`                 | Send events to Event API        | Load (Option 2) |
+| `StateService.markFileProcessed(kv, fileName, workflowId)` | Prevent duplicate processing      | Track      |
+
+---
+
+## The Generic Pattern
+
+### Architecture Overview
+
+The master data ETL pattern follows a **four-phase pipeline** that adapts to any entity type through configuration:
+
+```
+┌──────────────────────────────────────────────────────────────┐
+│ PHASE 1: EXTRACT                       │
+│ - List files from S3/SFTP                  │
+│ - Filter by pattern (*.csv, locations_*.json, etc.)     │
+│ - Download file content                   │
+│ - Skip already-processed files (state check)        │
+└──────────────────────────────────────────────────────────────┘
+              ↓
+┌──────────────────────────────────────────────────────────────┐
+│ PHASE 2: PARSE                        │
+│ - Auto-detect format (CSV, JSON, XML)            │
+│ - Parse content to JavaScript objects            │
+│ - Validate structure (required fields, types)        │
+│ - Handle parsing errors gracefully              │
+└──────────────────────────────────────────────────────────────┘
+              ↓
+┌──────────────────────────────────────────────────────────────┐
+│ PHASE 3: TRANSFORM                      │
+│ - Apply field mappings (source to Fluent schema)      │
+│ - Execute resolvers (transformations, calculations)     │
+│ - Validate required fields                  │
+│ - Enrich with defaults/constants              │
+└──────────────────────────────────────────────────────────────┘
+              ↓
+┌──────────────────────────────────────────────────────────────┐
+│ PHASE 4: LOAD                        │
+│ - Choose strategy (GraphQL Mutation vs Event API)      │
+│ - Batch if needed (large datasets)             │
+│ - Execute load operation                   │
+│ - Handle errors and retries                 │
+│ - Mark file as processed (state update)           │
+└──────────────────────────────────────────────────────────────┘
+```
+
+### Configuration-Driven Design
+
+**Core Principle**: All entity-specific logic lives in JSON configuration files, not in code.
+
+**Generic Configuration Structure:**
+
+```json
+{
+ "entityType": "location", // What you're loading
+ "sourceConfig": {
+  // Where to get data
+  "type": "S3_CSV",
+  "bucket": "master-data",
+  "prefix": "locations/",
+  "filePattern": "*.csv"
+ },
+ "parseConfig": {
+  // How to parse data
+  "format": "csv",
+  "delimiter": ",",
+  "headers": true
+ },
+ "mappingConfig": {
+  // How to transform data
+  "version": "1.0",
+  "fields": {
+   "ref": { "source": "location_id", "required": true },
+   "name": { "source": "location_name" },
+   "status": { "value": "ACTIVE" }
+  }
+ },
+ "loadConfig": {
+  // How to load into Fluent
+  "strategy": "graphql",
+  "mutation": "createLocation",
+  "batchSize": 100
+ }
+}
+```
+
+**Reusability**: Change `entityType` to "product", update field mappings → same code loads products!
+
+### Works for ANY Entity Type
+
+This pattern is entity-agnostic because:
+
+1. **Generic Source Reading**: S3DataSource/SftpDataSource work with any file
+2. **Format-Agnostic Parsing**: Parsers handle CSV/JSON/XML regardless of entity type
+3. **Flexible Mapping**: UniversalMapper adapts to any source→target schema
+4. **Mutation Generation**: GraphQLMutationMapper works with any mutation
+5. **State Management**: StateService tracks processing for any entity
+
+**Example Entity Types:**
+
+| Entity   | Source Format | Mutation     | Complexity     |
+| ---------- | ------------- | ---------------- | ------------------- |
+| Locations | CSV      | `createLocation` | Simple       |
+| Products  | JSON     | `createProduct` | Medium (variants)  |
+| Controls  | JSON     | `createControl` | Simple       |
+| Carriers  | XML      | `createCarrier` | Simple       |
+| Categories | JSON     | `createCategory` | Medium (hierarchy) |
+| Customers | CSV      | `createCustomer` | Medium (attributes) |
+
+All use the **same pipeline** with different **configurations**.
+
+---
+
+## Example 1: Location Master Data
+
+### Overview
+
+Load store/warehouse locations from CSV files into Fluent Commerce.
+
+**Business Context:**
+
+- Retail locations change infrequently (openings, closings, updates)
+- Source: Retail management system exports CSV daily
+- Destination: Fluent Location entities
+- Frequency: Daily batch, event-driven on new file
+
+### Source Data Formats
+
+**CSV Format:**
+
+```csv
+location_id,location_name,type,address_line1,city,state,zip,country,latitude,longitude,status
+LOC001,Downtown Store,STORE,123 Main St,New York,NY,10001,US,40.7128,-74.0060,ACTIVE
+LOC002,Warehouse East,WAREHOUSE,456 Industrial Rd,Newark,NJ,07102,US,40.7357,-74.1724,ACTIVE
+LOC003,Pop-Up Shop,STORE,789 Fashion Ave,New York,NY,10018,US,40.7549,-73.9840,INACTIVE
+```
+
+**JSON Format:**
+
+```json
+{
+ "locations": [
+  {
+   "locationId": "LOC001",
+   "name": "Downtown Store",
+   "type": "STORE",
+   "address": {
+    "street": "123 Main St",
+    "city": "New York",
+    "state": "NY",
+    "zip": "10001",
+    "country": "US"
+   },
+   "coordinates": {
+    "lat": 40.7128,
+    "lng": -74.006
+   },
+   "status": "ACTIVE"
+  }
+ ]
+}
+```
+
+**XML Format:**
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<LocationFeed>
+ <Location>
+  <ID>LOC001</ID>
+  <Name>Downtown Store</Name>
+  <Type>STORE</Type>
+  <Address>
+   <Line1>123 Main St</Line1>
+   <City>New York</City>
+   <State>NY</State>
+   <Zip>10001</Zip>
+   <Country>US</Country>
+  </Address>
+  <Latitude>40.7128</Latitude>
+  <Longitude>-74.0060</Longitude>
+  <Status>ACTIVE</Status>
+ </Location>
+</LocationFeed>
+```
+
+### Field Mapping Configuration
+
+**`config/location-mapping.json`:**
+
+```json
+{
+ "version": "1.0",
+ "description": "Map external location data to Fluent Location schema",
+ "fields": {
+  "ref": {
+   "source": "location_id",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "type": {
+   "source": "type",
+   "required": true,
+   "resolver": "sdk.uppercase"
+  },
+  "name": {
+   "source": "location_name",
+   "required": true
+  },
+  "status": {
+   "source": "status",
+   "defaultValue": "ACTIVE",
+   "resolver": "sdk.uppercase"
+  },
+  "primaryAddress": {
+   "fields": {
+    "street": { "source": "address_line1" },
+    "city": { "source": "city" },
+    "state": { "source": "state" },
+    "postcode": { "source": "zip" },
+    "country": { "source": "country" }
+   }
+  },
+  "coordinates": {
+   "fields": {
+    "latitude": { "source": "latitude", "resolver": "sdk.parseFloat" },
+    "longitude": { "source": "longitude", "resolver": "sdk.parseFloat" }
+   }
+  },
+  "retailerId": {
+   "value": "${RETAILER_ID}",
+   "required": true
+  }
+ }
+}
+```
+
+**Key Features:**
+
+- ✅ Required field validation (`ref`, `type`, `name`)
+- ✅ Default values (`status` defaults to "ACTIVE")
+- ✅ Built-in resolvers (`sdk.trim`, `sdk.uppercase`, `sdk.parseFloat`)
+- ✅ Nested object mapping (`primaryAddress`, `coordinates`)
+- ✅ Environment variable support (`${RETAILER_ID}`)
+
+### GraphQL Mutation Approach
+
+**Target Mutation:**
+
+```graphql
+mutation CreateLocation($input: CreateLocationInput!) {
+ createLocation(input: $input) {
+  id
+  ref
+  name
+  status
+ }
+}
+```
+
+**Mutation Variables (after transformation):**
+
+```json
+{
+ "input": {
+  "ref": "LOC001",
+  "type": "STORE",
+  "name": "Downtown Store",
+  "status": "ACTIVE",
+  "primaryAddress": {
+   "street": "123 Main St",
+   "city": "New York",
+   "state": "NY",
+   "postcode": "10001",
+   "country": "US"
+  },
+  "coordinates": {
+   "latitude": 40.7128,
+   "longitude": -74.006
+  },
+  "retailerId": "my-retailer"
+ }
+}
+```
+
+### Complete Working Code
+
+**`location-etl.ts`:**
+
+```typescript
+// FC Connect SDK+
+// Install: npm install @fluentcommerce/fc-connect-sdk@latest
+// Docs: https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk
+// GitHub: https://github.com/fluentcommerce/fc-connect-sdk
+
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import { CSVParserService } from '@fluentcommerce/fc-connect-sdk';
+import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+// Access openKv from context: const { openKv } = ctx;
+import * as fs from 'fs';
+
+// Initialize state service (prevents duplicate processing)
+// ✅ CORRECT: Access openKv from Versori context
+export async function masterDataETL(ctx: any, configPath: string) {
+ // Load configuration
+ const config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
+
+ // Initialize SDK components
+ const logger = console; // Replace with proper logger
+ const { openKv } = ctx;
+ const fluentClient = await createClient({
+  config: {
+   baseUrl: process.env.FLUENT_BASE_URL!,
+   clientId: process.env.FLUENT_CLIENT_ID!,
+   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+   retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  logger,
+ });
+
+ // Initialize data source (S3 in this example)
+ const s3DataSource = new S3DataSource(
+  {
+   type: 'S3_CSV',
+   s3Config: {
+    bucket: config.sourceConfig.bucket,
+    region: process.env.AWS_REGION!,
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+   },
+  },
+  logger
+ );
+
+ // Initialize state service (prevents duplicate processing)
+ const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
+ const stateService = new StateService(logger);
+
+ // Initialize parser based on format
+ const parser = config.parseConfig.format === 'csv' ? new CSVParserService() : null; // Add JSON/XML parsers as needed
+
+ // Initialize mapper
+ const mapper = new UniversalMapper(config.mappingConfig, { logger, fluentClient });
+
+ logger.info(`Starting ${config.entityType} ETL process`);
+
+ try {
+  // PHASE 1: EXTRACT - List files from source
+  const files = await s3DataSource.listFiles({
+   prefix: config.sourceConfig.prefix,
+  });
+
+  logger.info(`Found ${files.length} files to process`);
+
+  // Process each file
+  for (const file of files) {
+   const fileKey = `${config.entityType}:${file.name}`;
+
+   // Check if already processed
+   const alreadyProcessed = await stateService.isFileProcessed(kvAdapter, fileKey);
+   if (alreadyProcessed) {
+    logger.info(`Skipping already processed file: ${file.name}`);
+    continue;
+   }
+
+   logger.info(`Processing file: ${file.name}`);
+
+   try {
+    // PHASE 2: PARSE - Download and parse file
+    const fileContent = await s3DataSource.downloadFile(file.path);
+    const records = await parser!.parse(fileContent as string);
+
+    logger.info(`Parsed ${records.length} records from ${file.name}`);
+
+    // PHASE 3: TRANSFORM - Map each record
+    const transformedRecords = [];
+    for (const record of records) {
+     const result = await mapper.map(record);
+     if (result.success) {
+      transformedRecords.push(result.data);
+     } else {
+      logger.error(`Mapping failed for record:`, {
+       record,
+       errors: result.errors,
+      });
+     }
+    }
+
+    logger.info(`Transformed ${transformedRecords.length} records successfully`);
+
+    // PHASE 4: LOAD - Submit to Fluent Commerce
+    if (config.loadConfig.strategy === 'graphql') {
+     await loadViaGraphQL(fluentClient, transformedRecords, config.loadConfig, logger);
+    } else if (config.loadConfig.strategy === 'event') {
+     await loadViaEventAPI(fluentClient, transformedRecords, config.loadConfig, logger);
+    }
+
+    // Update sync state with processed file metadata
+    await stateService.updateSyncState(
+     kvAdapter,
+     [
+      {
+       fileName: file.name,
+       lastModified: new Date().toISOString(),
+       recordCount: transformedRecords.length,
+      },
+     ],
+     'master-data-etl'
+    );
+
+    logger.info(`Successfully processed file: ${file.name}`);
+   } catch (error) {
+    logger.error(`Failed to process file: ${file.name}`, error);
+    // Continue to next file instead of failing entire batch
+    continue;
+   }
+  }
+
+  logger.info(`${config.entityType} ETL process completed`);
+ } catch (error) {
+  logger.error(`${config.entityType} ETL process failed`, error);
+  throw error;
+ }
+}
+
+/**
+ * Load data via GraphQL mutations
+ */
+async function loadViaGraphQL(client: any, records: any[], loadConfig: any, logger: any) {
+ const batchSize = loadConfig.batchSize || 100;
+ const mutation = loadConfig.mutation;
+
+ logger.info(`Loading ${records.length} records via GraphQL mutation: ${mutation}`);
+
+ // Process in batches
+ for (let i = 0; i < records.length; i += batchSize) {
+  const batch = records.slice(i, i + batchSize);
+  logger.info(`Processing batch ${i / batchSize + 1} (${batch.length} records)`);
+
+  // Execute mutations for batch
+  for (const record of batch) {
+   const query = `
+    mutation ${mutation}($input: ${capitalize(mutation)}Input!) {
+     ${mutation}(input: $input) {
+      id
+      ref
+     }
+    }
+   `;
+
+   try {
+    const result = await client.graphql({
+     query,
+     variables: { input: record },
+    });
+
+    logger.debug(`Created ${mutation}:`, result.data[mutation]);
+   } catch (error) {
+    logger.error(`Failed to create ${mutation}:`, { record, error });
+    // Continue to next record instead of failing entire batch
+   }
+  }
+ }
+
+ logger.info(`GraphQL load completed`);
+}
+
+/**
+ * Load data via Event API
+ */
+async function loadViaEventAPI(client: any, records: any[], loadConfig: any, logger: any) {
+ const eventName = loadConfig.eventName;
+
+ logger.info(`Loading ${records.length} records via Event API: ${eventName}`);
+
+ for (const record of records) {
+  try {
+   await client.sendEvent({
+    name: eventName,
+    entityRef: record.ref,
+    entityType: loadConfig.entityType,
+    retailerId: record.retailerId,
+    attributes: record,
+   });
+
+   logger.debug(`Sent event ${eventName} for ${record.ref}`);
+  } catch (error) {
+   logger.error(`Failed to send event for ${record.ref}:`, error);
+  }
+ }
+
+ logger.info(`Event API load completed`);
+}
+
+// Helper function
+function capitalize(str: string): string {
+ return str.charAt(0).toUpperCase() + str.slice(1);
+}
+
+// Example usage
+if (require.main === module) {
+ masterDataETL('config/location-etl-config.json')
+  .then(() => console.log('ETL completed'))
+  .catch(err => {
+   console.error('ETL failed:', err);
+   process.exit(1);
+  });
+}
+```
+
+**Configuration File (`config/location-etl-config.json`):**
+
+```json
+{
+ "entityType": "location",
+ "sourceConfig": {
+  "type": "S3_CSV",
+  "bucket": "master-data",
+  "prefix": "locations/",
+  "filePattern": "*.csv"
+ },
+ "parseConfig": {
+  "format": "csv",
+  "delimiter": ",",
+  "headers": true
+ },
+ "mappingConfig": {
+  "version": "1.0",
+  "fields": {
+   "ref": { "source": "location_id", "required": true },
+   "name": { "source": "location_name", "required": true },
+   "type": { "source": "type", "required": true },
+   "status": { "source": "status", "defaultValue": "ACTIVE" },
+   "primaryAddress": {
+    "fields": {
+     "street": { "source": "address_line1" },
+     "city": { "source": "city" },
+     "state": { "source": "state" },
+     "postcode": { "source": "zip" },
+     "country": { "source": "country" }
+    }
+   },
+   "retailerId": { "value": "${RETAILER_ID}" }
+  }
+ },
+ "loadConfig": {
+  "strategy": "graphql",
+  "mutation": "createLocation",
+  "batchSize": 100
+ }
+}
+```
+
+**Key Design Decisions:**
+
+1. **Configuration-Driven**: All entity logic in JSON config, not code
+2. **Error Handling**: Continue processing on individual record failures
+3. **State Management**: Prevent duplicate processing of files
+4. **Batching**: Process large datasets in manageable chunks
+5. **Logging**: Comprehensive logging for debugging and monitoring
+
+---
+
+## Example 2: Product Catalog
+
+### Overview
+
+Load product catalog with variants (parent-child relationships) from JSON files.
+
+**Business Context:**
+
+- Product data includes parent products and child variants (size, color, etc.)
+- Source: Product Information Management (PIM) system exports JSON
+- Destination: Fluent Product entities
+- Complexity: Parent-child relationships require nested mapping
+
+### Source Data Formats
+
+**JSON Format (with variants):**
+
+```json
+{
+ "products": [
+  {
+   "productId": "PROD001",
+   "name": "Classic T-Shirt",
+   "description": "Premium cotton t-shirt",
+   "brand": "MyBrand",
+   "category": "Apparel",
+   "variants": [
+    {
+     "sku": "PROD001-S-RED",
+     "size": "S",
+     "color": "Red",
+     "price": 29.99,
+     "barcode": "123456789001"
+    },
+    {
+     "sku": "PROD001-M-RED",
+     "size": "M",
+     "color": "Red",
+     "price": 29.99,
+     "barcode": "123456789002"
+    }
+   ]
+  }
+ ]
+}
+```
+
+**CSV Format (flattened variants):**
+
+```csv
+product_id,product_name,sku,size,color,price,barcode
+PROD001,Classic T-Shirt,PROD001-S-RED,S,Red,29.99,123456789001
+PROD001,Classic T-Shirt,PROD001-M-RED,M,Red,29.99,123456789002
+```
+
+### Field Mapping Configuration
+
+**`config/product-mapping.json`:**
+
+```json
+{
+ "version": "1.0",
+ "description": "Map PIM product data to Fluent Product schema",
+ "fields": {
+  "ref": {
+   "source": "productId",
+   "required": true
+  },
+  "type": {
+   "value": "STANDARD",
+   "required": true
+  },
+  "name": {
+   "source": "name",
+   "required": true
+  },
+  "summary": {
+   "source": "description"
+  },
+  "gtin": {
+   "source": "barcode"
+  },
+  "status": {
+   "value": "ACTIVE"
+  },
+  "retailerId": {
+   "value": "${RETAILER_ID}"
+  },
+  "attributes": {
+   "fields": {
+    "brand": { "source": "brand" },
+    "category": { "source": "category" }
+   }
+  },
+  "variants": {
+   "source": "variants",
+   "isArray": true,
+   "fields": {
+    "ref": { "source": "$.sku", "required": true },
+    "attributes": {
+     "fields": {
+      "size": { "source": "$.size" },
+      "color": { "source": "$.color" }
+     }
+    },
+    "prices": {
+     "fields": {
+      "currency": { "value": "USD" },
+      "value": { "source": "$.price", "resolver": "sdk.parseFloat" }
+     }
+    },
+    "gtin": { "source": "$.barcode" }
+   }
+  }
+ }
+}
+```
+
+**Key Features:**
+
+- ✅ Parent-child relationships (`variants[]` array mapping)
+- ✅ Relative paths within arrays (`$.sku` references current variant)
+- ✅ Nested objects (`attributes`, `prices`)
+- ✅ Static values (`type: "STANDARD"`, `currency: "USD"`)
+
+### Complete Working Code
+
+**`product-etl.ts`:**
+
+```typescript
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+import { S3DataSource, JSONParserService } from '@fluentcommerce/fc-connect-sdk';
+import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+// Access openKv from context: const { openKv } = ctx;
+import * as fs from 'fs';
+
+/**
+ * Product Catalog ETL with Parent-Child Relationships
+ */
+export async function productCatalogETL(ctx: any) {
+ const logger = console;
+ const { openKv } = ctx;
+ const fluentClient = await createClient({
+  config: {
+   baseUrl: process.env.FLUENT_BASE_URL!,
+   clientId: process.env.FLUENT_CLIENT_ID!,
+   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+   retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  logger,
+ });
+
+ // Initialize components
+ const s3DataSource = new S3DataSource(
+  {
+   type: 'S3_CSV',
+   s3Config: {
+    bucket: 'product-catalog',
+    region: process.env.AWS_REGION!,
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+   },
+  },
+  logger
+ );
+
+ const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
+ const stateService = new StateService(logger);
+
+ const jsonParser = new JSONParserService();
+
+ // Load mapping configuration
+ const mappingConfig = JSON.parse(fs.readFileSync('config/product-mapping.json', 'utf-8'));
+
+ const mapper = new UniversalMapper(mappingConfig, { logger, fluentClient });
+
+ logger.info('Starting product catalog ETL');
+
+ try {
+  // List JSON files
+  const files = await s3DataSource.listFiles({ prefix: 'products/' });
+
+  for (const file of files) {
+   if (!file.name.endsWith('.json')) continue;
+
+   const fileKey = `product:${file.name}`;
+
+   // Check if processed
+   if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
+    logger.info(`Skipping processed file: ${file.name}`);
+    continue;
+   }
+
+   logger.info(`Processing file: ${file.name}`);
+
+   try {
+    // Download and parse JSON
+    const fileContent = await s3DataSource.downloadFile(file.path);
+    const data = await jsonParser.parse(fileContent as string, {
+     dataPath: 'products', // Extract products array from root
+    });
+
+    const products = Array.isArray(data) ? data : [data];
+    logger.info(`Parsed ${products.length} products from ${file.name}`);
+
+    // Transform and load each product
+    for (const product of products) {
+     const result = await mapper.map(product);
+
+     if (result.success) {
+      // Create parent product
+      await createProduct(fluentClient, result.data, logger);
+
+      // Create variants if present
+      if (result.data.variants && result.data.variants.length > 0) {
+       for (const variant of result.data.variants) {
+        await createVariant(fluentClient, result.data.ref, variant, logger);
+       }
+      }
+     } else {
+      logger.error('Product mapping failed:', {
+       product,
+       errors: result.errors,
+      });
+     }
+    }
+
+    // Mark as processed
+    await stateService.updateSyncState(
+     kvAdapter,
+     [
+      {
+       fileName: file.name,
+       lastModified: new Date().toISOString(),
+       recordCount: products.length,
+      },
+     ],
+     'product-catalog-etl'
+    );
+
+    logger.info(`Successfully processed: ${file.name}`);
+   } catch (error) {
+    logger.error(`Failed to process file: ${file.name}`, error);
+    continue;
+   }
+  }
+
+  logger.info('Product catalog ETL completed');
+ } catch (error) {
+  logger.error('Product catalog ETL failed', error);
+  throw error;
+ }
+}
+
+/**
+ * Create product via GraphQL
+ */
+async function createProduct(client: any, productData: any, logger: any) {
+ const mutation = `
+  mutation CreateProduct($input: CreateProductInput!) {
+   createProduct(input: $input) {
+    id
+    ref
+    name
+    status
+   }
+  }
+ `;
+
+ try {
+  const result = await client.graphql({
+   query: mutation,
+   variables: { input: productData },
+  });
+
+  logger.info(`Created product: ${productData.ref}`, result.data.createProduct);
+ } catch (error) {
+  logger.error(`Failed to create product: ${productData.ref}`, error);
+  throw error;
+ }
+}
+
+/**
+ * Create variant via GraphQL
+ */
+async function createVariant(client: any, parentRef: string, variantData: any, logger: any) {
+ const mutation = `
+  mutation CreateVariant($input: CreateVariantInput!) {
+   createVariant(input: $input) {
+    id
+    ref
+    attributes
+   }
+  }
+ `;
+
+ try {
+  const result = await client.graphql({
+   query: mutation,
+   variables: {
+    input: {
+     ...variantData,
+     parentRef,
+    },
+   },
+  });
+
+  logger.info(`Created variant: ${variantData.ref}`, result.data.createVariant);
+ } catch (error) {
+  logger.error(`Failed to create variant: ${variantData.ref}`, error);
+  // Don't throw - continue with other variants
+ }
+}
+
+// Example usage
+if (require.main === module) {
+ productCatalogETL()
+  .then(() => console.log('ETL completed'))
+  .catch(err => {
+   console.error('ETL failed:', err);
+   process.exit(1);
+  });
+}
+```
+
+**Key Design Decisions:**
+
+1. **Parent-Child Processing**: Create parent product first, then variants
+2. **Array Mapping**: `variants[]` notation handles nested arrays
+3. **Relative Paths**: `$.sku` resolves relative to current variant item
+4. **Error Isolation**: Variant creation failures don't stop parent processing
+
+---
+
+## Example 3: Control/Config Data
+
+### Overview
+
+Load business rules and configuration parameters from JSON files.
+
+**Business Context:**
+
+- Controls define business rules, thresholds, flags
+- Source: Configuration management system exports JSON
+- Destination: Fluent Control entities
+- Characteristics: Simple structure, validation logic important
+
+### Source Data Format
+
+**JSON Format:**
+
+```json
+{
+ "controls": [
+  {
+   "name": "ORDER_TIMEOUT_MINUTES",
+   "value": "30",
+   "type": "INTEGER",
+   "context": "ORDER_MANAGEMENT",
+   "description": "Order allocation timeout in minutes"
+  },
+  {
+   "name": "ENABLE_AUTO_ALLOCATION",
+   "value": "true",
+   "type": "BOOLEAN",
+   "context": "ORDER_MANAGEMENT",
+   "description": "Enable automatic order allocation"
+  },
+  {
+   "name": "DEFAULT_CARRIER",
+   "value": "FEDEX",
+   "type": "STRING",
+   "context": "FULFILLMENT",
+   "description": "Default shipping carrier"
+  }
+ ]
+}
+```
+
+### Field Mapping Configuration
+
+**`config/control-mapping.json`:**
+
+```json
+{
+ "version": "1.0",
+ "description": "Map configuration controls to Fluent Control schema",
+ "fields": {
+  "name": {
+   "source": "name",
+   "required": true,
+   "resolver": "sdk.uppercase"
+  },
+  "value": {
+   "source": "value",
+   "required": true,
+   "resolver": "custom.validateControlValue"
+  },
+  "type": {
+   "source": "type",
+   "required": true,
+   "resolver": "sdk.uppercase"
+  },
+  "context": {
+   "source": "context",
+   "defaultValue": "GLOBAL"
+  },
+  "description": {
+   "source": "description"
+  },
+  "status": {
+   "value": "ACTIVE"
+  },
+  "retailerId": {
+   "value": "${RETAILER_ID}"
+  }
+ }
+}
+```
+
+### Validation Logic
+
+**Custom resolver for control value validation:**
+
+```typescript
+import { FieldResolverFunction } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Validate control value based on type
+ */
+export const validateControlValue: FieldResolverFunction = (
+ value: any,
+ sourceData: any,
+ config: any,
+ helpers: any
+) => {
+ const type = sourceData.type;
+
+ switch (type) {
+  case 'INTEGER':
+   const intValue = helpers.parseIntSafe(value, null);
+   if (intValue === null) {
+    throw new Error(`Invalid INTEGER value: ${value}`);
+   }
+   return intValue.toString();
+
+  case 'FLOAT':
+   const floatValue = helpers.parseFloatSafe(value, null);
+   if (floatValue === null) {
+    throw new Error(`Invalid FLOAT value: ${value}`);
+   }
+   return floatValue.toString();
+
+  case 'BOOLEAN':
+   if (!['true', 'false'].includes(String(value).toLowerCase())) {
+    throw new Error(`Invalid BOOLEAN value: ${value}`);
+   }
+   return String(value).toLowerCase();
+
+  case 'STRING':
+   return String(value);
+
+  case 'JSON':
+   try {
+    JSON.parse(value);
+    return value;
+   } catch (error) {
+    throw new Error(`Invalid JSON value: ${value}`);
+   }
+
+  default:
+   helpers.log.warn(`Unknown control type: ${type}, treating as STRING`);
+   return String(value);
+ }
+};
+```
+
+### Complete Working Code
+
+**`control-etl.ts`:**
+
+```typescript
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+import { S3DataSource, JSONParserService } from '@fluentcommerce/fc-connect-sdk';
+import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+// Access openKv from context: const { openKv } = ctx;
+import * as fs from 'fs';
+import { validateControlValue } from './resolvers/control-validators';
+
+/**
+ * Control/Config Data ETL
+ */
+export async function controlDataETL(ctx: any) {
+ const logger = console;
+ const { openKv } = ctx;
+ const fluentClient = await createClient({
+  config: {
+   baseUrl: process.env.FLUENT_BASE_URL!,
+   clientId: process.env.FLUENT_CLIENT_ID!,
+   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+   retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+  logger,
+ });
+
+ // Initialize components
+ const s3DataSource = new S3DataSource(
+  {
+   type: 'S3_CSV',
+   s3Config: {
+    bucket: 'config-data',
+    region: process.env.AWS_REGION!,
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+   },
+  },
+  logger
+ );
+
+ const kvAdapter = new VersoriKVAdapter(openKv(':project:'));
+ const stateService = new StateService(logger);
+
+ const jsonParser = new JSONParserService();
+
+ // Load mapping configuration
+ const mappingConfig = JSON.parse(fs.readFileSync('config/control-mapping.json', 'utf-8'));
+
+ // Initialize mapper with custom resolvers
+ const mapper = new UniversalMapper(mappingConfig, {
+  logger,
+  fluentClient,
+  customResolvers: {
+   'custom.validateControlValue': validateControlValue,
+  },
+ });
+
+ logger.info('Starting control data ETL');
+
+ try {
+  // List JSON files
+  const files = await s3DataSource.listFiles({ prefix: 'controls/' });
+
+  for (const file of files) {
+   if (!file.name.endsWith('.json')) continue;
+
+   const fileKey = `control:${file.name}`;
+
+   // Check if processed
+   if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
+    logger.info(`Skipping processed file: ${file.name}`);
+    continue;
+   }
+
+   logger.info(`Processing file: ${file.name}`);
+
+   try {
+    // Download and parse JSON
+    const fileContent = await s3DataSource.downloadFile(file.path);
+    const data = await jsonParser.parse(fileContent as string, {
+     dataPath: 'controls',
+    });
+
+    const controls = Array.isArray(data) ? data : [data];
+    logger.info(`Parsed ${controls.length} controls from ${file.name}`);
+
+    // Transform and load each control
+    let successCount = 0;
+    let failureCount = 0;
+
+    for (const control of controls) {
+     try {
+      const result = await mapper.map(control);
+
+      if (result.success) {
+       await createControl(fluentClient, result.data, logger);
+       successCount++;
+      } else {
+       logger.error('Control mapping failed:', {
+        control,
+        errors: result.errors,
+       });
+       failureCount++;
+      }
+     } catch (error) {
+      logger.error(`Failed to process control: ${control.name}`, error);
+      failureCount++;
+     }
+    }
+
+    logger.info(`Control processing summary:`, {
+     total: controls.length,
+     success: successCount,
+     failures: failureCount,
+    });
+
+    // Mark as processed
+    await stateService.updateSyncState(
+     kvAdapter,
+     [
+      {
+       fileName: file.name,
+       lastModified: new Date().toISOString(),
+       recordCount: controls.length,
+      },
+     ],
+     'control-data-etl'
+    );
+
+    logger.info(`Successfully processed: ${file.name}`);
+   } catch (error) {
+    logger.error(`Failed to process file: ${file.name}`, error);
+    continue;
+   }
+  }
+
+  logger.info('Control data ETL completed');
+ } catch (error) {
+  logger.error('Control data ETL failed', error);
+  throw error;
+ }
+}
+
+/**
+ * Create control via GraphQL
+ */
+async function createControl(client: any, controlData: any, logger: any) {
+ const mutation = `
+  mutation CreateControl($input: CreateControlInput!) {
+   createControl(input: $input) {
+    id
+    name
+    value
+    type
+    context
+   }
+  }
+ `;
+
+ try {
+  const result = await client.graphql({
+   query: mutation,
+   variables: { input: controlData },
+  });
+
+  logger.info(`Created control: ${controlData.name}`, {
+   value: controlData.value,
+   type: controlData.type,
+  });
+ } catch (error) {
+  logger.error(`Failed to create control: ${controlData.name}`, error);
+  throw error;
+ }
+}
+
+// Example usage
+if (require.main === module) {
+ controlDataETL()
+  .then(() => console.log('ETL completed'))
+  .catch(err => {
+   console.error('ETL failed:', err);
+   process.exit(1);
+  });
+}
+```
+
+**Key Design Decisions:**
+
+1. **Type Validation**: Custom resolver validates value based on type field
+2. **Error Tracking**: Count successes/failures for summary reporting
+3. **Atomic Processing**: Each control processed independently
+4. **Detailed Logging**: Track validation failures with context
+
+---
+
+## Source Strategies
+
+### S3 with Event Notifications
+
+**Use Case**: Process files as soon as they're uploaded to S3
+
+**Setup:**
+
+```typescript
+import { webhook } from '@versori/run/webhooks';
+import { masterDataETL } from './location-etl';
+
+export const s3LocationETL = webhook('s3-location-upload', {
+ response: { mode: 'sync' },
+})
+ .then(async ({ data }) => {
+  // Parse S3 event notification
+  const s3Event = data.Records[0].s3;
+  const bucket = s3Event.bucket.name;
+  const key = decodeURIComponent(s3Event.object.key.replace(/\+/g, ' '));
+
+  console.log(`S3 event received: ${bucket}/${key}`);
+
+  // Run ETL for this specific file
+  await masterDataETL('config/location-etl-config.json');
+
+  return { success: true, message: 'Location ETL completed' };
+ })
+ .catch(({ error }) => {
+  console.error('S3 location ETL failed:', error);
+  return { success: false, error: error.message };
+ });
+```
+
+**S3 Bucket Configuration:**
+
+```json
+{
+ "LambdaFunctionConfigurations": [
+  {
+   "LambdaFunctionArn": "arn:aws:lambda:...:function:versori-webhook",
+   "Events": ["s3:ObjectCreated:*"],
+   "Filter": {
+    "Key": {
+     "FilterRules": [
+      {
+       "Name": "prefix",
+       "Value": "locations/"
+      },
+      {
+       "Name": "suffix",
+       "Value": ".csv"
+      }
+     ]
+    }
+   }
+  }
+ ]
+}
+```
+
+### SFTP with Polling
+
+**Use Case**: Poll SFTP server periodically for new files
+
+#### SFTP Credential Access
+
+**Versori Platform** has three methods for accessing SFTP credentials:
+
+1. **Connection Variables (Recommended)** - Direct access to connection config:
+  ```typescript
+  const { host, port, username, password, privateKey } = ctx.activation.connections.sftp_server;
+  ```
+
+2. **Credentials API** - For base64-encoded credentials:
+  ```typescript
+  const creds = await ctx.credentials().getAccessToken('sftp_server');
+  ```
+
+3. **Connection String Parsing** - Decode `connectionVariables.connectionString`:
+  ```typescript
+  const connStr = ctx.activation.connections.sftp_server.connectionString;
+  // Parse: sftp://username:password@host:port
+  ```
+
+**Standalone Node.js/Deno**: Use environment variables directly:
+```typescript
+const config = {
+ host: process.env.SFTP_HOST!,
+ port: parseInt(process.env.SFTP_PORT || '22'),
+ username: process.env.SFTP_USERNAME!,
+ password: process.env.SFTP_PASSWORD,
+ privateKey: process.env.SFTP_PRIVATE_KEY,
+};
+```
+
+**Security Best Practices:**
+- Always prefer SSH keys over passwords
+- Never log credential values
+- Use scoped credentials (read-only when possible)
+- Rotate credentials regularly
+
+**See:** [SFTP Credential Access Security Guide](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for complete details.
+
+#### Setup Example
+
+```typescript
+import { schedule } from '@versori/run/schedule';
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+import { Buffer } from 'node:buffer'; // Required for Deno/Versori
+
+export const sftpPolling = schedule('location-sftp-poll', {
+ cron: '0 */6 * * *', // Every 6 hours
+ retry: { attempts: 3 },
+}).then(async (ctx) => {
+ const { log, activation } = ctx;
+
+ // Access SFTP credentials (Versori)
+ const { host, port, username, password, privateKey } = activation.connections.sftp_server;
+
+ // Initialize SFTP data source
+ const sftpSource = new SftpDataSource(
+  {
+   type: 'SFTP_CSV',
+   settings: {
+    host,
+    port: port || 22,
+    username,
+    password,
+    privateKey,
+    remotePath: '/data/locations',
+    filePattern: '*.csv',
+   },
+  },
+  log
+ );
+
+ // List new files
+ const files = await sftpSource.listFiles();
+ log.info(`Found ${files.length} files on SFTP`);
+
+ for (const file of files) {
+  try {
+   // Download file
+   const content = await sftpSource.downloadFile(file.path);
+
+   // Process file (same ETL logic as S3)
+   // ... (extract, parse, transform, load)
+
+   // Move to processed folder
+   await sftpSource.moveFile(file.path, `/data/locations/processed/${file.name}`);
+  } catch (error) {
+   log.error(`Failed to process ${file.name}:`, error);
+  }
+ }
+
+ return { success: true, filesProcessed: files.length };
+});
+```
+
+---
+
+## Load Strategies
+
+### GraphQL Mutation Approach
+
+**When to Use:**
+
+- Simple entity creation/updates
+- Direct control over mutations
+- Schema validation needed
+- Small to medium datasets (<10K records)
+
+**Advantages:**
+
+- ✅ Type-safe with GraphQL schema
+- ✅ Immediate validation feedback
+- ✅ Fine-grained control over mutations
+- ✅ Can return created IDs
+
+**Example:**
+
+```typescript
+async function loadViaGraphQL(records: any[], mutation: string, logger: any) {
+ const client = await createClient({
+  config: {
+   baseUrl: process.env.FLUENT_BASE_URL!,
+   clientId: process.env.FLUENT_CLIENT_ID!,
+   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+   retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+ });
+
+ for (const record of records) {
+  const query = `
+   mutation ${mutation}($input: ${capitalize(mutation)}Input!) {
+    ${mutation}(input: $input) {
+     id
+     ref
+    }
+   }
+  `;
+
+  try {
+   const result = await client.graphql({
+    query,
+    variables: { input: record },
+   });
+
+   logger.info(`Created ${mutation}:`, result.data[mutation]);
+  } catch (error: any) {
+   logger.error(`Failed ${mutation}:`, {
+    record,
+    error: error.message,
+    details: error.response?.errors,
+   });
+  }
+ }
+}
+```
+
+### Event API Approach
+
+**When to Use:**
+
+- Asynchronous processing acceptable
+- Triggering workflows/rules needed
+- Large datasets (>10K records)
+- Need event-driven architecture
+
+**Advantages:**
+
+- ✅ Asynchronous (better for large datasets)
+- ✅ Triggers workflows and rules
+- ✅ Decoupled from mutations
+- ✅ Better performance for bulk loads
+
+**Example:**
+
+```typescript
+async function loadViaEventAPI(records: any[], eventName: string, logger: any) {
+ const client = await createClient({
+  config: {
+   baseUrl: process.env.FLUENT_BASE_URL!,
+   clientId: process.env.FLUENT_CLIENT_ID!,
+   clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+   retailerId: process.env.FLUENT_RETAILER_ID!,
+  },
+ });
+
+ for (const record of records) {
+  try {
+   await client.sendEvent({
+    name: eventName,
+    entityRef: record.ref,
+    entityType: 'LOCATION',
+    retailerId: record.retailerId,
+    attributes: record,
+   });
+
+   logger.info(`Sent event ${eventName} for ${record.ref}`);
+  } catch (error: any) {
+   logger.error(`Failed to send event for ${record.ref}:`, error);
+  }
+ }
+}
+```
+
+**Event-Driven Workflow:**
+
+```
+ETL Process          Fluent Commerce
+  ↓
+Send Event (LOCATION_CREATED)
+  ↓               ↓
+              Workflow Triggers
+                  ↓
+              Process Event
+                  ↓
+              Create Location
+                  ↓
+              Apply Business Rules
+                  ↓
+              Send Notifications
+```
+
+---
+
+## Configuration Schema
+
+### Generic Configuration Template
+
+Use this template for ANY entity type:
+
+```json
+{
+ "entityType": "<entity-name>",
+ "description": "<what this ETL does>",
+ "sourceConfig": {
+  "type": "S3_CSV | SFTP_CSV | S3_JSON | SFTP_JSON",
+  "bucket": "<s3-bucket-name>",
+  "prefix": "<folder-prefix>",
+  "filePattern": "*.csv | *.json | *.xml",
+  "sftp": {
+   "host": "${SFTP_HOST}",
+   "port": 22,
+   "username": "${SFTP_USERNAME}",
+   "password": "${SFTP_PASSWORD}",
+   "privateKey": "${SFTP_PRIVATE_KEY}", // Recommended over password
+   "remotePath": "/data/<entity>",
+   "filePattern": "*.<format>"
+  }
+ },
+ "parseConfig": {
+  "format": "csv | json | xml",
+  "delimiter": "," | "|" | "\t",
+  "headers": true | false,
+  "encoding": "utf8 | utf16",
+  "json": {
+   "dataPath": "root.path.to.array",
+   "jsonLines": false
+  },
+  "xml": {
+   "itemPath": "//Item",
+   "includeAttributes": true
+  }
+ },
+ "mappingConfig": {
+  "version": "1.0",
+  "description": "Field mappings",
+  "fields": {
+   "targetField": {
+    "source": "sourceField",
+    "required": true | false,
+    "defaultValue": "default",
+    "resolver": "sdk.* | custom.*"
+   }
+  }
+ },
+ "loadConfig": {
+  "strategy": "graphql | event",
+  "mutation": "createEntity | updateEntity",
+  "eventName": "ENTITY_CREATED",
+  "batchSize": 100,
+  "retryAttempts": 3
+ },
+ "scheduleConfig": {
+  "enabled": true,
+  "cron": "0 0 * * *",
+  "timezone": "UTC"
+ }
+}
+```
+
+### How to Adapt for New Entity Types
+
+**Step-by-Step Guide:**
+
+1. **Copy Template**: Start with generic configuration template
+2. **Set Entity Type**: Change `entityType` to your entity name
+3. **Configure Source**: Set bucket/path for your data source
+4. **Configure Parser**: Set format and parsing options
+5. **Define Field Mappings**: Map source fields to Fluent schema
+6. **Configure Load Strategy**: Choose GraphQL or Event API
+7. **Test**: Run ETL with sample data
+8. **Deploy**: Schedule or trigger via webhook
+
+**Example Adaptation (Customer Entity):**
+
+```json
+{
+ "entityType": "customer",
+ "description": "Load customer data from CRM system",
+ "sourceConfig": {
+  "type": "S3_CSV",
+  "bucket": "crm-exports",
+  "prefix": "customers/",
+  "filePattern": "customers_*.csv"
+ },
+ "parseConfig": {
+  "format": "csv",
+  "delimiter": ",",
+  "headers": true
+ },
+ "mappingConfig": {
+  "version": "1.0",
+  "fields": {
+   "ref": {
+    "source": "customer_id",
+    "required": true
+   },
+   "firstName": {
+    "source": "first_name",
+    "required": true
+   },
+   "lastName": {
+    "source": "last_name",
+    "required": true
+   },
+   "email": {
+    "source": "email_address",
+    "required": true,
+    "resolver": "sdk.lowercase"
+   },
+   "primaryPhone": {
+    "source": "phone_number",
+    "resolver": "custom.formatPhoneNumber"
+   },
+   "retailerId": {
+    "value": "${RETAILER_ID}"
+   }
+  }
+ },
+ "loadConfig": {
+  "strategy": "graphql",
+  "mutation": "createCustomer",
+  "batchSize": 50
+ }
+}
+```
+
+---
+
+## Extending to Other Entities
+
+### Step-by-Step Guide
+
+**1. Identify Entity Schema**
+
+Understand the Fluent schema for your entity:
+
+```graphql
+# Example: Carrier schema
+type Carrier {
+ id: ID!
+ ref: String!
+ name: String!
+ type: String!
+ status: String
+ services: [CarrierService]
+ retailerId: ID!
+}
+
+input CreateCarrierInput {
+ ref: String!
+ name: String!
+ type: String!
+ status: String
+ services: [CarrierServiceInput]
+ retailerId: ID!
+}
+```
+
+**2. Create Field Mapping Configuration**
+
+Map source data to schema:
+
+```json
+{
+ "version": "1.0",
+ "fields": {
+  "ref": {
+   "source": "carrier_code",
+   "required": true,
+   "resolver": "sdk.uppercase"
+  },
+  "name": {
+   "source": "carrier_name",
+   "required": true
+  },
+  "type": {
+   "source": "carrier_type",
+   "required": true
+  },
+  "status": {
+   "value": "ACTIVE"
+  },
+  "services": {
+   "source": "services",
+   "isArray": true,
+   "fields": {
+    "name": { "source": "$.service_name" },
+    "code": { "source": "$.service_code" },
+    "deliveryDays": { "source": "$.delivery_days", "resolver": "sdk.parseInt" }
+   }
+  },
+  "retailerId": {
+   "value": "${RETAILER_ID}"
+  }
+ }
+}
+```
+
+**3. Set Up Data Source**
+
+Configure where data comes from:
+
+```typescript
+const carrierSource = new S3DataSource(
+ {
+  type: 'S3_CSV',
+  s3Config: {
+   bucket: 'carrier-data',
+   region: process.env.AWS_REGION!,
+   accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+   secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+  },
+ },
+ logger
+);
+```
+
+**4. Run Generic ETL Pipeline**
+
+Use the same ETL code:
+
+```typescript
+await masterDataETL('config/carrier-etl-config.json');
+```
+
+### Customer Example
+
+**Source CSV:**
+
+```csv
+customer_id,first_name,last_name,email,phone,segment,status
+CUST001,John,Doe,john.doe@email.com,555-1234,VIP,ACTIVE
+CUST002,Jane,Smith,jane.smith@email.com,555-5678,STANDARD,ACTIVE
+```
+
+**Mapping Configuration:**
+
+```json
+{
+ "version": "1.0",
+ "fields": {
+  "ref": { "source": "customer_id", "required": true },
+  "firstName": { "source": "first_name", "required": true },
+  "lastName": { "source": "last_name", "required": true },
+  "email": { "source": "email", "required": true, "resolver": "sdk.lowercase" },
+  "primaryPhone": { "source": "phone" },
+  "status": { "source": "status", "defaultValue": "ACTIVE" },
+  "attributes": {
+   "fields": {
+    "segment": { "source": "segment" }
+   }
+  },
+  "retailerId": { "value": "${RETAILER_ID}" }
+ }
+}
+```
+
+### Carrier Example
+
+**Source JSON:**
+
+```json
+{
+ "carriers": [
+  {
+   "code": "FEDEX",
+   "name": "FedEx",
+   "type": "PARCEL",
+   "services": [
+    { "service_code": "GROUND", "service_name": "FedEx Ground", "delivery_days": 3 },
+    { "service_code": "2DAY", "service_name": "FedEx 2 Day", "delivery_days": 2 }
+   ]
+  }
+ ]
+}
+```
+
+**Mapping Configuration:**
+
+```json
+{
+ "version": "1.0",
+ "fields": {
+  "ref": { "source": "code", "required": true },
+  "name": { "source": "name", "required": true },
+  "type": { "source": "type", "required": true },
+  "status": { "value": "ACTIVE" },
+  "services": {
+   "source": "services",
+   "isArray": true,
+   "fields": {
+    "code": { "source": "$.service_code" },
+    "name": { "source": "$.service_name" },
+    "transitDays": { "source": "$.delivery_days", "resolver": "sdk.parseInt" }
+   }
+  },
+  "retailerId": { "value": "${RETAILER_ID}" }
+ }
+}
+```
+
+### Pricing Example
+
+**Source CSV:**
+
+```csv
+sku,price_list,currency,base_price,sale_price,start_date,end_date
+PROD001-S-RED,US_RETAIL,USD,29.99,24.99,2024-01-01,2024-12-31
+PROD001-M-RED,US_RETAIL,USD,29.99,24.99,2024-01-01,2024-12-31
+```
+
+**Mapping Configuration:**
+
+```json
+{
+ "version": "1.0",
+ "fields": {
+  "ref": {
+   "resolver": "custom.generatePriceRef",
+   "required": true
+  },
+  "sku": {
+   "source": "sku",
+   "required": true
+  },
+  "priceList": {
+   "source": "price_list",
+   "required": true
+  },
+  "currency": {
+   "source": "currency",
+   "required": true,
+   "resolver": "sdk.uppercase"
+  },
+  "value": {
+   "source": "base_price",
+   "required": true,
+   "resolver": "sdk.parseFloat"
+  },
+  "salePrice": {
+   "source": "sale_price",
+   "resolver": "sdk.parseFloat"
+  },
+  "validFrom": {
+   "source": "start_date",
+   "resolver": "sdk.formatDate"
+  },
+  "validTo": {
+   "source": "end_date",
+   "resolver": "sdk.formatDate"
+  },
+  "retailerId": {
+   "value": "${RETAILER_ID}"
+  }
+ }
+}
+```
+
+**Custom Resolver:**
+
+```typescript
+export const generatePriceRef: FieldResolverFunction = (
+ value: any,
+ sourceData: any,
+ config: any,
+ helpers: any
+) => {
+ // Generate unique ref: SKU-PRICELIST
+ return `${sourceData.sku}-${sourceData.price_list}`;
+};
+```
+
+---
+
+## Testing
+
+### Unit Testing Field Mappings
+
+```typescript
+import { UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+import * as fs from 'fs';
+
+describe('Location Mapping', () => {
+ let mapper: UniversalMapper;
+
+ beforeEach(() => {
+  const config = JSON.parse(fs.readFileSync('config/location-mapping.json', 'utf-8'));
+  mapper = new UniversalMapper(config);
+ });
+
+ it('should map location data correctly', async () => {
+  const sourceData = {
+   location_id: 'LOC001',
+   location_name: 'Downtown Store',
+   type: 'store',
+   status: 'active',
+   address_line1: '123 Main St',
+   city: 'New York',
+   state: 'NY',
+   zip: '10001',
+   country: 'US',
+   latitude: '40.7128',
+   longitude: '-74.0060',
+  };
+
+  const result = await mapper.map(sourceData);
+
+  expect(result.success).toBe(true);
+  expect(result.data).toMatchObject({
+   ref: 'LOC001',
+   name: 'Downtown Store',
+   type: 'STORE',
+   status: 'ACTIVE',
+   primaryAddress: {
+    street: '123 Main St',
+    city: 'New York',
+    state: 'NY',
+    postcode: '10001',
+    country: 'US',
+   },
+   coordinates: {
+    latitude: 40.7128,
+    longitude: -74.006,
+   },
+  });
+ });
+
+ it('should handle required field validation', async () => {
+  const sourceData = {
+   location_name: 'Store Without ID',
+   // Missing location_id (required)
+  };
+
+  const result = await mapper.map(sourceData);
+
+  expect(result.success).toBe(false);
+  expect(result.errors).toContain("Required field 'ref' is missing or empty");
+ });
+
+ it('should apply default values', async () => {
+  const sourceData = {
+   location_id: 'LOC001',
+   location_name: 'Store',
+   type: 'STORE',
+   // Missing status - should default to "ACTIVE"
+  };
+
+  const result = await mapper.map(sourceData);
+
+  expect(result.success).toBe(true);
+  expect(result.data.status).toBe('ACTIVE');
+ });
+});
+```
+
+### Integration Testing ETL Pipeline
+
+```typescript
+import { masterDataETL } from './location-etl';
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+import * as fs from 'fs';
+
+describe('Location ETL Integration', () => {
+ let s3DataSource: S3DataSource;
+
+ beforeEach(() => {
+  s3DataSource = new S3DataSource(
+   {
+    type: 'S3_CSV',
+    s3Config: {
+     bucket: 'test-bucket',
+     region: 'us-east-1',
+     accessKeyId: process.env.TEST_AWS_KEY!,
+     secretAccessKey: process.env.TEST_AWS_SECRET!,
+    },
+   },
+   console
+  );
+ });
+
+ it('should process location CSV file end-to-end', async () => {
+  // Upload test file to S3
+  const testData = `location_id,location_name,type,status
+LOC001,Test Store,STORE,ACTIVE
+LOC002,Test Warehouse,WAREHOUSE,ACTIVE`;
+
+  await s3DataSource.uploadFile('locations/test.csv', testData, { contentType: 'text/csv' });
+
+  // Run ETL
+  await masterDataETL('config/location-etl-config.json');
+
+  // Verify locations were created (query Fluent API)
+  const result = await fluentClient.graphql({
+   query: `
+    query {
+     locations(first: 10, filter: { ref: { in: ["LOC001", "LOC002"] } }) {
+      edges {
+       node {
+        ref
+        name
+        type
+       }
+      }
+     }
+    }
+   `,
+  });
+
+  expect(result.data.locations.edges).toHaveLength(2);
+  expect(result.data.locations.edges[0].node.ref).toBe('LOC001');
+ }, 30000); // 30s timeout for integration test
+});
+```
+
+### End-to-End Testing
+
+```typescript
+import { masterDataETL } from './location-etl';
+
+describe('Location ETL E2E', () => {
+ it('should handle full ETL lifecycle', async () => {
+  // 1. Upload test data to S3
+  // 2. Trigger ETL process
+  // 3. Verify data in Fluent
+  // 4. Verify state tracking (file marked as processed)
+  // 5. Verify idempotency (re-running doesn't duplicate)
+  // TODO: Implement full E2E test scenario
+ });
+});
+```
+
+---
+
+## Common Issues
+
+### Issue: Duplicate Records Created
+
+**Symptom**: Same entity created multiple times
+
+**Root Cause**: State management not working or file processed multiple times
+
+**Solution:**
+
+```typescript
+// Ensure state service is initialized
+const kvAdapter = new VersoriKVAdapter(openKv());
+const stateService = new StateService(logger);
+
+// Check BEFORE processing
+const fileKey = `${entityType}:${file.name}`;
+if (await stateService.isFileProcessed(kvAdapter, fileKey)) {
+ logger.info(`Skipping already processed file: ${file.name}`);
+ continue;
+}
+
+// Mark AFTER successful processing
+await stateService.updateSyncState(
+ kvAdapter,
+ [
+  {
+   fileName: file.name,
+   lastModified: new Date().toISOString(),
+   recordCount: records.length,
+  },
+ ],
+ 'master-data-etl'
+);
+```
+
+### Issue: Field Mapping Errors
+
+**Symptom**: "Required field missing" or "Mapping failed"
+
+**Root Cause**: Source field name doesn't match configuration
+
+**Solution:**
+
+```typescript
+// Debug source data structure
+logger.debug('Source data:', JSON.stringify(sourceData, null, 2));
+
+// Check field names match exactly (case-sensitive)
+{
+ "ref": {
+  "source": "location_id", // Must match CSV header exactly
+  "required": true
+ }
+}
+
+// Use custom resolver for flexible field access
+{
+ "ref": {
+  "resolver": "custom.extractRef",
+  "required": true
+ }
+}
+```
+
+### Issue: Large File Memory Issues
+
+**Symptom**: Out of memory errors with large CSV/JSON files
+
+**Root Cause**: Loading entire file into memory
+
+**Solution:**
+
+```typescript
+// Use streaming parsers for large files
+const csvParser = new CSVParserService();
+
+// Parse with streaming (yields records one-by-one)
+for await (const record of csvParser.parseStreaming(fileContent)) {
+ const result = await mapper.map(record);
+ if (result.success) {
+  await loadRecord(result.data);
+ }
+}
+
+// Or batch process
+for await (const batch of csvParser.parseStreaming(fileContent, {}, 100)) {
+ await loadBatch(batch);
+}
+```
+
+### Issue: GraphQL Mutation Timeouts
+
+**Symptom**: Mutations timing out for large datasets
+
+**Root Cause**: Synchronous processing of many records
+
+**Solution:**
+
+```typescript
+// Use batching and concurrency limits
+import pLimit from 'p-limit';
+
+const limit = pLimit(5); // Max 5 concurrent mutations
+
+const promises = records.map(record => limit(() => createViaGraphQL(record)));
+
+await Promise.all(promises);
+
+// Or use Event API for async processing
+await loadViaEventAPI(records, 'LOCATION_CREATED', logger);
+```
+
+### Issue: Type Coercion Errors
+
+**Symptom**: "Expected number, got string" in GraphQL mutations
+
+**Root Cause**: CSV parsers return all values as strings
+
+**Solution:**
+
+```typescript
+// Use SDK resolvers for type coercion
+{
+ "latitude": {
+  "source": "latitude",
+  "resolver": "sdk.parseFloat" // String → number
+ },
+ "active": {
+  "source": "is_active",
+  "resolver": "sdk.boolean" // "true" → true
+ },
+ "quantity": {
+  "source": "qty",
+  "resolver": "sdk.parseInt" // "10" → 10
+ }
+}
+```
+
+---
+
+## Related Guides
+
+### SDK Documentation
+
+- [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Complete field mapping reference
+- [S3 Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - S3 integration details
+- [SFTP Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - SFTP integration details
+- [SFTP Credential Access Security](../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) - Secure credential handling for SFTP
+- [CSV Parser](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - CSV parsing options
+- [JSON Parser](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - JSON parsing options
+- [State Management](../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md) - Preventing duplicates
+
+### Use Case Patterns
+
+- [Inventory Ingestion](../../02-CORE-GUIDES/ingestion/ingestion-readme.md) - Similar pattern for inventory data
+- [Order Integration](../../03-PATTERN-GUIDES/connector-scenarios/connector-scenarios-readme.md) - Transaction data vs master data
+- [Catalog Sync](../../01-TEMPLATES/versori/workflows/ingestion/graphql-mutations/template-ingestion-s3-csv-location-graphql.md) - Product catalog patterns
+
+### Platform Integration
+
+- [Versori Connector Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Deploy ETL as connector
+- [Webhook Triggers](../../04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md#webhook-functions-receiving-external-requests) - Event-driven ETL
+- [Scheduled Jobs](../../04-REFERENCE/platforms/versori/modules/platforms-versori-04-workflows.md#scheduled-functions-time-based-recurring-tasks) - Periodic ETL execution
+
+---
+
+## Summary
+
+This **Master Data ETL Pattern** provides a **generic, configuration-driven framework** for loading ANY entity type into Fluent Commerce.
+
+**Key Takeaways:**
+
+1. ✅ **One Pattern for All Entities**: Same code works for locations, products, controls, carriers, etc.
+2. ✅ **Configuration-Driven**: No code changes needed for new entity types
+3. ✅ **Four-Phase Pipeline**: Extract → Parse → Transform → Load
+4. ✅ **Multiple Source Formats**: CSV, JSON, XML support
+5. ✅ **Multiple Load Strategies**: GraphQL mutations or Event API
+6. ✅ **Production-Ready**: State management, error handling, logging
+
+**Getting Started:**
+
+1. Copy the generic ETL code
+2. Create field mapping configuration for your entity
+3. Configure data source (S3/SFTP)
+4. Run ETL pipeline
+5. Monitor logs and verify data in Fluent
+
+**Next Steps:**
+
+- Review the [Universal Mapping Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for advanced mapping patterns
+- Explore [S3 Data Source](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for source configuration options
+- Check [Versori Connector Guide](../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) to deploy as a production connector
+
+---
+
+**Need Help?**
+
+- 📖 Documentation: `fc-connect-sdk/docs/`
+- 💬 Support: Fluent Commerce support team
+- 🐛 Issues: GitHub repository issues