@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-01-introduction.md

@@ -1,626 +1,626 @@
-# Module 1: Introduction to Data Ingestion
-
-[← Back to Ingestion Guide](../ingestion-readme.md)
-
-**Module 1 of 9** | **Level**: Beginner | **Time**: 10 minutes
-
----
-
-## Overview
-
-This module introduces data ingestion concepts, explains when to ingest data into Fluent Commerce, and covers the key features of the FC Connect SDK's ingestion system.
-
-## Learning Objectives
-
-By the end of this module, you will:
-- ✅ Understand what data ingestion is and why it's important
-- ✅ Know when to use the Batch API vs GraphQL mutations
-- ✅ Understand the complete ingestion workflow
-- ✅ Recognize the business use cases for inventory ingestion
-- ✅ Understand the critical Batch API limitations
-
----
-
-## What is Data Ingestion?
-
-**Data ingestion** is the process of importing data from external systems into Fluent Commerce. This typically involves:
-
-1. **Reading** data from a source (S3, SFTP, API, database)
-2. **Parsing** the data format (CSV, Parquet, XML, JSON)
-3. **Transforming** the data to match Fluent's schema
-4. **Validating** the transformed data
-5. **Sending** the data to Fluent Commerce via Batch API or GraphQL mutations
-6. **Tracking** the ingestion state to prevent duplicates
-
-### Why is Ingestion Important?
-
-Fluent Commerce acts as the central orchestration layer for order management and inventory. To maintain accurate data, it needs regular updates from:
-
-- **Warehouse Management Systems (WMS)** - Real-time inventory levels
-- **E-commerce Platforms** - Product catalogs, pricing, availability
-- **ERP Systems** - Master data, locations, suppliers
-- **3PL Partners** - External inventory positions
-- **Data Lakes** - Historical data, analytics exports
-
-Without reliable ingestion, Fluent Commerce cannot:
-- Route orders to correct locations
-- Provide accurate inventory availability
-- Optimize fulfillment decisions
-- Track inventory across the network
-
----
-
-## When to Use the Batch API
-
-The Fluent Commerce **Batch API** is optimized for **bulk inventory updates** only.
-
-### ✅ Use Batch API For:
-
-| Use Case | Why Batch API? | Example |
-|----------|----------------|---------|
-| **Daily Inventory Sync** | Bulk updates, high volume | WMS → Fluent daily full sync |
-| **Stock Adjustments** | Multiple positions at once | Cycle count results from warehouse |
-| **Location Transfers** | Update source and destination | Move inventory between warehouses |
-| **3PL Inventory Updates** | External partner data feeds | SFTP CSV from fulfillment partner |
-
-### ❌ Do NOT Use Batch API For:
-
-| Entity Type | Use Instead | Notes |
-|-------------|-------------|-------|
-| **Orders** | `createOrder` mutation | Batch API does NOT support orders |
-| **Products** | `createProduct`, `updateProduct` | Batch API does NOT support products |
-| **Locations** | Location-specific mutations | Batch API does NOT support locations |
-| **Customers** | `createCustomer` mutation | Batch API does NOT support customers |
-
-### Critical Batch API Limitations
-
-The Batch API has **strict limitations**:
-
-```typescript
-// ✅ CORRECT - Batch API supports ONLY this combination
-{
-  "entityType": "INVENTORY",  // ✅ Only INVENTORY
-  "action": "UPSERT",                  // ✅ Only UPSERT action
-  "entities": [...]                    // ✅ Array of InventoryPositionInput
-}
-
-// ❌ WRONG - These are NOT supported
-{
-  "entityType": "ORDER",               // ❌ Orders not supported
-  "action": "CREATE"                   // ❌ CREATE action not supported
-}
-```
-
-**Schema Reference:**
-```json
-// From docs/schema/fluent-commerce-schema.json
-{
-  "enums": {
-    "EntityType": {
-      "values": ["INVENTORY"],
-      "note": "Batch API ONLY supports INVENTORY"
-    },
-    "BatchAction": {
-      "values": ["UPSERT"],
-      "note": "Batch API ONLY supports UPSERT action"
-    }
-  }
-}
-```
-
----
-
-## Complete Ingestion Workflow
-
-The SDK provides a **complete ingestion framework** that handles all stages:
-
-```mermaid
-graph TD
-    A[Data Source] --> B{File Already Processed?}
-    B -->|Yes| Z[Skip File]
-    B -->|No| C[Acquire Lock]
-    C --> D[Parse Data]
-    D --> E[Transform with UniversalMapper]
-    E --> F[Validate Schema]
-    F --> G[Create/Reuse Job]
-    G --> H[Send Batches]
-    H --> I[Monitor Status]
-    I --> J[Mark as Processed]
-    J --> K[Release Lock]
-    K --> L[Archive/Delete Source]
-```
-
-### Workflow Components
-
-1. **Data Sources** - Where data comes from
-   - S3 buckets (most common)
-   - SFTP servers
-   - Local files (development)
-   - HTTP APIs
-
-2. **Parsers** - Convert file formats to structured data
-   - CSV → Array of objects
-   - Parquet → Column-oriented records
-   - XML → Parsed object tree
-   - JSON → Native objects
-
-3. **Field Mapper** - Transform source → Fluent schema
-   - `UniversalMapper` - Declarative field mapping
-   - SDK Resolvers - Built-in transformations (`sdk.parseInt`, `sdk.uppercase`, etc.)
-   - Custom Resolvers - Business logic transformations
-
-4. **Validator** - Ensure data meets requirements
-   - Required fields present
-   - Correct data types
-   - Business rules (quantity >= 0, etc.)
-
-5. **Job Manager** - Coordinate batch operations
-   - Job strategies (DAILY, PER_FILE, BATCHES_PER_JOB)
-   - Batch size optimization
-   - Retry logic for failed batches
-
-6. **State Manager** - Prevent duplicate processing
-   - Track processed files
-   - Distributed locking
-   - Idempotent operations
-
----
-
-## Business Use Cases
-
-### Use Case 1: Daily Inventory Sync
-
-**Scenario**: WMS exports full inventory snapshot to S3 daily at 2 AM.
-
-**Requirements:**
-- ✅ Process only new files (state management)
-- ✅ Handle 50,000+ inventory positions
-- ✅ Complete within 30 minutes
-- ✅ Prevent duplicate processing
-
-**Solution**: Building blocks composition with DAILY job reuse
-```typescript
-import {
-  createClient,
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-  VersoriKVAdapter
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize services
-// In Versori: use ctx.log directly
-// In standalone: use createConsoleLogger() or custom logger
-const logger = toStructuredLogger(createConsoleLogger(), {
-  service: 'inventory-ingestion',
-  correlationId: generateCorrelationId()
-});
-
-const stateService = new StateService(logger);
-const kvAdapter = new VersoriKVAdapter(openKv());
-
-const client = await createClient({
-  config: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    username: process.env.FLUENT_USERNAME!,  // Required for password grant
-    password: process.env.FLUENT_PASSWORD!,  // Required for password grant
-    retailerId: process.env.FLUENT_RETAILER_ID!
-  }
-});
-
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-inventory',
-    name: 'S3 Inventory Source',
-    s3Config: {
-      bucket: 'inventory-bucket',
-      region: 'us-east-1',
-      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-    },
-  },
-  logger
-);
-
-const parser = new CSVParserService(logger);
-const mapper = new UniversalMapper({ fields: inventoryMapping.fields });
-
-// Create daily job (reuse for all batches)
-const job = await client.createJob({
-  name: 'Daily Inventory Sync',
-  retailerId: process.env.FLUENT_RETAILER_ID!
-});
-
-// Process files with state management
-const files = await s3.listFiles({ prefix: 'data/' });
-
-for (const file of files) {
-  // Check if already processed
-  const fileKey = ['file', 'default', file.path];
-  const alreadyProcessed = await kvAdapter.get(fileKey);
-
-  if (alreadyProcessed) continue;
-
-  const content = await s3.downloadFile(file.path);
-  const records = await parser.parse(content as string);
-  const result = await mapper.map(records);
-
-  // Send in batches
-  for (let i = 0; i < result.data.length; i += 1000) {
-    const batch = result.data.slice(i, i + 1000);
-    await client.sendBatch(job.id, {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      source: 'S3_CSV_IMPORT',
-      event: 'INVENTORY_UPDATE',
-      entities: batch
-    });
-  }
-
-  await stateService.updateSyncState(kvAdapter, [{
-    fileName: file.path,
-    lastModified: new Date().toISOString(),
-    recordCount: result.data.length
-  }]);
-}
-```
-
-### Use Case 2: Real-Time Stock Adjustments
-
-**Scenario**: Warehouse performs cycle counts and sends SFTP files every hour.
-
-**Requirements:**
-- ✅ Process multiple files per day
-- ✅ Each file is independent
-- ✅ Track which files processed
-- ✅ Handle network interruptions
-
-**Solution**: Building blocks with PER_FILE job creation and state tracking
-```typescript
-import {
-  createClient,
-  SftpDataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-  VersoriKVAdapter
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize services
-const logger = toStructuredLogger(createConsoleLogger(), {
-  service: 'inventory-ingestion',
-  correlationId: generateCorrelationId()
-});
-
-const stateService = new StateService(logger);
-const kvAdapter = new VersoriKVAdapter(openKv());
-
-const client = await createClient({
-  config: {
-    baseUrl: process.env.FLUENT_BASE_URL!,
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    username: process.env.FLUENT_USERNAME!,  // Required for password grant
-    password: process.env.FLUENT_PASSWORD!,  // Required for password grant
-    retailerId: process.env.FLUENT_RETAILER_ID!
-  }
-});
-
-const sftp = new SftpDataSource({
-  type: 'SFTP_CSV',
-  connectionId: 'sftp-warehouse',
-  name: 'Warehouse SFTP',
-  settings: {
-    host: 'sftp.warehouse.com',
-    port: 22,
-    username: process.env.SFTP_USERNAME!,
-    privateKey: process.env.SFTP_PRIVATE_KEY!,
-    remotePath: '/inventory/hourly',
-    filePattern: 'cycle_count_*.csv'
-  }
-}, logger);
-
-const parser = new CSVParserService(logger);
-const mapper = new UniversalMapper({ fields: adjustmentMapping.fields });
-
-// Process files - create new job for each file
-const files = await sftp.listFiles({ filePattern: 'cycle_count_*.csv' });
-
-for (const file of files) {
-  // Check if already processed
-  const fileKey = ['file', 'default', file.path];
-  const alreadyProcessed = await kvAdapter.get(fileKey);
-
-  if (alreadyProcessed) continue;
-
-  try {
-    const content = await sftp.downloadFile(file.path);
-    const records = await parser.parse(content as string);
-    const result = await mapper.map(records);
-
-    // Create new job for this file (PER_FILE strategy)
-    const job = await client.createJob({
-      name: `Cycle Count - ${file.path}`,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    });
-
-    // Send in batches of 500
-    for (let i = 0; i < result.data.length; i += 500) {
-      const batch = result.data.slice(i, i + 500);
-      await client.sendBatch(job.id, {
-        action: 'UPSERT',
-        entityType: 'INVENTORY',
-        source: 'SFTP_CSV_IMPORT',
-        event: 'CYCLE_COUNT_UPDATE',
-        entities: batch
-      });
-    }
-
-    await stateService.updateSyncState(kvAdapter, [{
-      fileName: file.path,
-      lastModified: new Date().toISOString(),
-      recordCount: result.data.length
-    }]);
-  } catch (error) {
-    console.error(`Failed to process ${file.path}`, error);
-  }
-}
-```
-
-### Use Case 3: 3PL Partner Inventory
-
-**Scenario**: Third-party logistics partner sends inventory feeds via SFTP twice daily.
-
-**Requirements:**
-- ✅ Validate partner data format
-- ✅ Enrich with location mapping
-- ✅ Handle missing/invalid records
-- ✅ Audit trail for compliance
-
-**Solution**: Custom resolvers with validation
-```typescript
-const mapper = new UniversalMapper({
-  fields: {
-    ref: { source: 'partner_sku', required: true },
-    productRef: { source: 'partner_sku', required: true },
-    locationRef: {
-      source: 'partner_location_code',
-      resolver: 'custom.mapPartnerLocation',  // ✅ Custom business logic
-      required: true
-    },
-    qty: {
-      source: 'available_quantity',
-      resolver: 'sdk.parseInt',
-      required: true,
-      validation: (value) => value >= 0  // ✅ Business rule
-    }
-  }
-}, {
-  customResolvers: {
-    'custom.mapPartnerLocation': (value, sourceData, config, helpers) => {
-      const mapping = {
-        'P1': 'WH-PARTNER-EAST',
-        'P2': 'WH-PARTNER-WEST'
-      };
-      const mapped = mapping[value];
-      if (!mapped) {
-        throw new Error(`Unknown partner location: ${value}`);
-      }
-      return mapped;
-    }
-  }
-});
-```
-
----
-
-## Key Ingestion Features
-
-### 1. Format Flexibility
-
-The SDK supports **multiple input formats** with the same workflow:
-
-| Format | Parser | Use Case |
-|--------|--------|----------|
-| CSV | `CSVParserService` | Most common - simple, human-readable |
-| Parquet | `ParquetParserService` | Large datasets - columnar, compressed |
-| XML | `XMLParserService` | Enterprise systems - SFCC, SAP, Magento |
-| JSON | `JSONParserService` | APIs, webhooks, modern systems |
-
-### 2. State Management
-
-**Problem**: Without state management, reprocessing files creates duplicates.
-
-**Solution**: `StateService` tracks processed files with distributed locking:
-
-```typescript
-const logger = toStructuredLogger(createConsoleLogger(), {
-  service: 'inventory-ingestion',
-  correlationId: generateCorrelationId()
-});
-
-const stateService = new StateService(logger);
-const kvAdapter = new VersoriKVAdapter(openKv());
-
-// Check if file already processed
-if (await stateService.isFileProcessed(kvAdapter, 'file.csv')) {
-  logger.info('File already processed, skipping');
-  return;
-}
-
-// Acquire lock for concurrent safety
-const lockAcquired = await stateService.acquireLock('ingestion-lock', kvAdapter, 15);
-if (!lockAcquired) {
-  logger.info('File locked by another process');
-  return;
-}
-
-try {
-  // Process file
-  await processFile('file.csv');
-
-  // Mark as processed
-  await stateService.updateSyncState(kvAdapter, [{
-    fileName: 'file.csv',
-    lastModified: new Date().toISOString(),
-    recordCount: recordCount
-  }]);
-} finally {
-  await stateService.releaseLock('ingestion-lock', kvAdapter);
-}
-```
-
-### 3. Performance Optimization
-
-**Batch Size Optimization**: Automatically calculate optimal batch size based on data:
-
-```typescript
-function calculateOptimalBatchSize(
-  totalRecords: number,
-  avgRecordSize: number
-): number {
-  const MAX_BATCH_SIZE = 10000;
-  const MAX_PAYLOAD_SIZE = 10 * 1024 * 1024; // 10MB
-
-  const recordsPerPayload = Math.floor(MAX_PAYLOAD_SIZE / avgRecordSize);
-  return Math.min(recordsPerPayload, MAX_BATCH_SIZE);
-}
-
-const batchSize = calculateOptimalBatchSize(50000, 200);
-// Result: Optimal size based on record size and payload limits
-```
-
-**Parallel Processing**: Process multiple files/batches with configurable strategies:
-
-```typescript
-import { createClient, S3DataSource, CSVParserService, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize services
-const client = await createClient({ config });
-const s3 = new S3DataSource(
-  {
-    type: 'S3_CSV',
-    connectionId: 's3-inventory',
-    name: 'S3 Inventory',
-    s3Config: s3Config,
-  },
-  logger
-);
-const parser = new CSVParserService();
-const mapper = new UniversalMapper(mappingConfig);
-
-// Create daily job (reuse for all batches)
-const job = await client.createJob({
-  name: 'Daily Inventory Sync',
-  retailerId: process.env.FLUENT_RETAILER_ID!
-});
-
-// Process multiple files
-const files = await s3.listFiles({ prefix: 'data/', maxKeys: 10 });
-
-for (const file of files) {
-  const content = await s3.downloadFile(file.path);
-  const records = await parser.parse(content as string);
-  const result = await mapper.map(records);
-
-  // Send in batches of 1000
-  for (let i = 0; i < result.data.length; i += 1000) {
-    const batch = result.data.slice(i, i + 1000);
-    await client.sendBatch(job.id, {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: batch
-    });
-  }
-}
-```
-
-### 4. Error Handling
-
-**Fail-Fast with Recovery**: Stop on critical errors, retry transient failures:
-
-```typescript
-class RetryableIngestion {
-  async executeWithRetry<T>(
-    operation: () => Promise<T>,
-    context: string
-  ): Promise<T> {
-    let lastError: Error;
-    let delay = 1000;
-
-    for (let attempt = 1; attempt <= 3; attempt++) {
-      try {
-        return await operation();
-      } catch (error) {
-        lastError = error;
-
-        // Retry network errors, fail on validation errors
-        if (!this.isRetryable(error)) {
-          throw error;  // ✅ Fail fast on validation errors
-        }
-
-        console.log(`Retry ${attempt}/3 for ${context}`);
-        await this.sleep(delay);
-        delay *= 2;  // Exponential backoff
-      }
-    }
-
-    throw new Error(`Failed after 3 retries: ${lastError.message}`);
-  }
-
-  private isRetryable(error: any): boolean {
-    return error.code === 'NETWORK_ERROR' ||
-           error.code === 'TIMEOUT_ERROR' ||
-           error.code === 'RATE_LIMIT_ERROR';
-  }
-}
-```
-
----
-
-## Ingestion vs Extraction
-
-The SDK provides **two complementary workflows**:
-
-| Aspect | Ingestion (this guide) | Extraction |
-|--------|------------------------|-----------|
-| **Direction** | External → Fluent Commerce | Fluent Commerce → External |
-| **Primary API** | Batch API (UPSERT) | GraphQL Queries |
-| **Use Case** | Update inventory positions | Export data for analytics/BI |
-| **Data Source** | S3, SFTP, files | Fluent Commerce GraphQL API |
-| **Data Destination** | Fluent Commerce | S3, SFTP, files, databases |
-| **Typical Frequency** | Hourly, daily | Daily, weekly, on-demand |
-| **State Management** | Prevent duplicate ingestion | Track extraction checkpoints |
-
-**See Also**: [Data Extraction Guide](../../extraction/) for the reverse workflow.
-
----
-
-## Key Takeaways
-
-- 🎯 **Batch API is for inventory only** - Use GraphQL mutations for orders, products, locations
-- 🎯 **State management prevents duplicates** - Essential for production workflows
-- 🎯 **UniversalMapper transforms data** - Declarative field mapping with validation
-- 🎯 **Job strategies optimize performance** - DAILY for frequent updates, PER_FILE for large datasets
-- 🎯 **Error handling is critical** - Implement retry logic and fail-fast validation
-
----
-
-## Next Steps
-
-Continue to [Module 2: Quick Start](../../auto-pagination/modules/auto-pagination-02-quick-start.md) to build your first ingestion workflow with a complete working example.
-
----
-
-[← Previous: Main Guide](../ingestion-readme.md) | [Back to Guide](../ingestion-readme.md) | [Next: Quick Start →](../../auto-pagination/modules/auto-pagination-02-quick-start.md)
-
-## Related Documentation
-
-- [Batch API Reference](../../api-reference/modules/api-reference-01-client-api.md#job-batch-operations) - Complete API documentation
-- [Universal Mapping Guide](../../mapping/mapping-readme.md) - Field transformation patterns
-- [State Management](./02-core-guides-ingestion-07-state-management.md) - Duplicate prevention in detail
-- [Orchestrators Guide](../../extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md) - High-level workflow coordination
+# Module 1: Introduction to Data Ingestion
+
+[← Back to Ingestion Guide](../ingestion-readme.md)
+
+**Module 1 of 9** | **Level**: Beginner | **Time**: 10 minutes
+
+---
+
+## Overview
+
+This module introduces data ingestion concepts, explains when to ingest data into Fluent Commerce, and covers the key features of the FC Connect SDK's ingestion system.
+
+## Learning Objectives
+
+By the end of this module, you will:
+- ✅ Understand what data ingestion is and why it's important
+- ✅ Know when to use the Batch API vs GraphQL mutations
+- ✅ Understand the complete ingestion workflow
+- ✅ Recognize the business use cases for inventory ingestion
+- ✅ Understand the critical Batch API limitations
+
+---
+
+## What is Data Ingestion?
+
+**Data ingestion** is the process of importing data from external systems into Fluent Commerce. This typically involves:
+
+1. **Reading** data from a source (S3, SFTP, API, database)
+2. **Parsing** the data format (CSV, Parquet, XML, JSON)
+3. **Transforming** the data to match Fluent's schema
+4. **Validating** the transformed data
+5. **Sending** the data to Fluent Commerce via Batch API or GraphQL mutations
+6. **Tracking** the ingestion state to prevent duplicates
+
+### Why is Ingestion Important?
+
+Fluent Commerce acts as the central orchestration layer for order management and inventory. To maintain accurate data, it needs regular updates from:
+
+- **Warehouse Management Systems (WMS)** - Real-time inventory levels
+- **E-commerce Platforms** - Product catalogs, pricing, availability
+- **ERP Systems** - Master data, locations, suppliers
+- **3PL Partners** - External inventory positions
+- **Data Lakes** - Historical data, analytics exports
+
+Without reliable ingestion, Fluent Commerce cannot:
+- Route orders to correct locations
+- Provide accurate inventory availability
+- Optimize fulfillment decisions
+- Track inventory across the network
+
+---
+
+## When to Use the Batch API
+
+The Fluent Commerce **Batch API** is optimized for **bulk inventory updates** only.
+
+### ✅ Use Batch API For:
+
+| Use Case | Why Batch API? | Example |
+|----------|----------------|---------|
+| **Daily Inventory Sync** | Bulk updates, high volume | WMS → Fluent daily full sync |
+| **Stock Adjustments** | Multiple positions at once | Cycle count results from warehouse |
+| **Location Transfers** | Update source and destination | Move inventory between warehouses |
+| **3PL Inventory Updates** | External partner data feeds | SFTP CSV from fulfillment partner |
+
+### ❌ Do NOT Use Batch API For:
+
+| Entity Type | Use Instead | Notes |
+|-------------|-------------|-------|
+| **Orders** | `createOrder` mutation | Batch API does NOT support orders |
+| **Products** | `createProduct`, `updateProduct` | Batch API does NOT support products |
+| **Locations** | Location-specific mutations | Batch API does NOT support locations |
+| **Customers** | `createCustomer` mutation | Batch API does NOT support customers |
+
+### Critical Batch API Limitations
+
+The Batch API has **strict limitations**:
+
+```typescript
+// ✅ CORRECT - Batch API supports ONLY this combination
+{
+  "entityType": "INVENTORY",  // ✅ Only INVENTORY
+  "action": "UPSERT",                  // ✅ Only UPSERT action
+  "entities": [...]                    // ✅ Array of InventoryPositionInput
+}
+
+// ❌ WRONG - These are NOT supported
+{
+  "entityType": "ORDER",               // ❌ Orders not supported
+  "action": "CREATE"                   // ❌ CREATE action not supported
+}
+```
+
+**Schema Reference:**
+```json
+// From docs/schema/fluent-commerce-schema.json
+{
+  "enums": {
+    "EntityType": {
+      "values": ["INVENTORY"],
+      "note": "Batch API ONLY supports INVENTORY"
+    },
+    "BatchAction": {
+      "values": ["UPSERT"],
+      "note": "Batch API ONLY supports UPSERT action"
+    }
+  }
+}
+```
+
+---
+
+## Complete Ingestion Workflow
+
+The SDK provides a **complete ingestion framework** that handles all stages:
+
+```mermaid
+graph TD
+    A[Data Source] --> B{File Already Processed?}
+    B -->|Yes| Z[Skip File]
+    B -->|No| C[Acquire Lock]
+    C --> D[Parse Data]
+    D --> E[Transform with UniversalMapper]
+    E --> F[Validate Schema]
+    F --> G[Create/Reuse Job]
+    G --> H[Send Batches]
+    H --> I[Monitor Status]
+    I --> J[Mark as Processed]
+    J --> K[Release Lock]
+    K --> L[Archive/Delete Source]
+```
+
+### Workflow Components
+
+1. **Data Sources** - Where data comes from
+   - S3 buckets (most common)
+   - SFTP servers
+   - Local files (development)
+   - HTTP APIs
+
+2. **Parsers** - Convert file formats to structured data
+   - CSV → Array of objects
+   - Parquet → Column-oriented records
+   - XML → Parsed object tree
+   - JSON → Native objects
+
+3. **Field Mapper** - Transform source → Fluent schema
+   - `UniversalMapper` - Declarative field mapping
+   - SDK Resolvers - Built-in transformations (`sdk.parseInt`, `sdk.uppercase`, etc.)
+   - Custom Resolvers - Business logic transformations
+
+4. **Validator** - Ensure data meets requirements
+   - Required fields present
+   - Correct data types
+   - Business rules (quantity >= 0, etc.)
+
+5. **Job Manager** - Coordinate batch operations
+   - Job strategies (DAILY, PER_FILE, BATCHES_PER_JOB)
+   - Batch size optimization
+   - Retry logic for failed batches
+
+6. **State Manager** - Prevent duplicate processing
+   - Track processed files
+   - Distributed locking
+   - Idempotent operations
+
+---
+
+## Business Use Cases
+
+### Use Case 1: Daily Inventory Sync
+
+**Scenario**: WMS exports full inventory snapshot to S3 daily at 2 AM.
+
+**Requirements:**
+- ✅ Process only new files (state management)
+- ✅ Handle 50,000+ inventory positions
+- ✅ Complete within 30 minutes
+- ✅ Prevent duplicate processing
+
+**Solution**: Building blocks composition with DAILY job reuse
+```typescript
+import {
+  createClient,
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+  VersoriKVAdapter
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize services
+// In Versori: use ctx.log directly
+// In standalone: use createConsoleLogger() or custom logger
+const logger = toStructuredLogger(createConsoleLogger(), {
+  service: 'inventory-ingestion',
+  correlationId: generateCorrelationId()
+});
+
+const stateService = new StateService(logger);
+const kvAdapter = new VersoriKVAdapter(openKv());
+
+const client = await createClient({
+  config: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    username: process.env.FLUENT_USERNAME!,  // Required for password grant
+    password: process.env.FLUENT_PASSWORD!,  // Required for password grant
+    retailerId: process.env.FLUENT_RETAILER_ID!
+  }
+});
+
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-inventory',
+    name: 'S3 Inventory Source',
+    s3Config: {
+      bucket: 'inventory-bucket',
+      region: 'us-east-1',
+      accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+    },
+  },
+  logger
+);
+
+const parser = new CSVParserService(logger);
+const mapper = new UniversalMapper({ fields: inventoryMapping.fields });
+
+// Create daily job (reuse for all batches)
+const job = await client.createJob({
+  name: 'Daily Inventory Sync',
+  retailerId: process.env.FLUENT_RETAILER_ID!
+});
+
+// Process files with state management
+const files = await s3.listFiles({ prefix: 'data/' });
+
+for (const file of files) {
+  // Check if already processed
+  const fileKey = ['file', 'default', file.path];
+  const alreadyProcessed = await kvAdapter.get(fileKey);
+
+  if (alreadyProcessed) continue;
+
+  const content = await s3.downloadFile(file.path);
+  const records = await parser.parse(content as string);
+  const result = await mapper.map(records);
+
+  // Send in batches
+  for (let i = 0; i < result.data.length; i += 1000) {
+    const batch = result.data.slice(i, i + 1000);
+    await client.sendBatch(job.id, {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      source: 'S3_CSV_IMPORT',
+      event: 'INVENTORY_UPDATE',
+      entities: batch
+    });
+  }
+
+  await stateService.updateSyncState(kvAdapter, [{
+    fileName: file.path,
+    lastModified: new Date().toISOString(),
+    recordCount: result.data.length
+  }]);
+}
+```
+
+### Use Case 2: Real-Time Stock Adjustments
+
+**Scenario**: Warehouse performs cycle counts and sends SFTP files every hour.
+
+**Requirements:**
+- ✅ Process multiple files per day
+- ✅ Each file is independent
+- ✅ Track which files processed
+- ✅ Handle network interruptions
+
+**Solution**: Building blocks with PER_FILE job creation and state tracking
+```typescript
+import {
+  createClient,
+  SftpDataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+  VersoriKVAdapter
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize services
+const logger = toStructuredLogger(createConsoleLogger(), {
+  service: 'inventory-ingestion',
+  correlationId: generateCorrelationId()
+});
+
+const stateService = new StateService(logger);
+const kvAdapter = new VersoriKVAdapter(openKv());
+
+const client = await createClient({
+  config: {
+    baseUrl: process.env.FLUENT_BASE_URL!,
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    username: process.env.FLUENT_USERNAME!,  // Required for password grant
+    password: process.env.FLUENT_PASSWORD!,  // Required for password grant
+    retailerId: process.env.FLUENT_RETAILER_ID!
+  }
+});
+
+const sftp = new SftpDataSource({
+  type: 'SFTP_CSV',
+  connectionId: 'sftp-warehouse',
+  name: 'Warehouse SFTP',
+  settings: {
+    host: 'sftp.warehouse.com',
+    port: 22,
+    username: process.env.SFTP_USERNAME!,
+    privateKey: process.env.SFTP_PRIVATE_KEY!,
+    remotePath: '/inventory/hourly',
+    filePattern: 'cycle_count_*.csv'
+  }
+}, logger);
+
+const parser = new CSVParserService(logger);
+const mapper = new UniversalMapper({ fields: adjustmentMapping.fields });
+
+// Process files - create new job for each file
+const files = await sftp.listFiles({ filePattern: 'cycle_count_*.csv' });
+
+for (const file of files) {
+  // Check if already processed
+  const fileKey = ['file', 'default', file.path];
+  const alreadyProcessed = await kvAdapter.get(fileKey);
+
+  if (alreadyProcessed) continue;
+
+  try {
+    const content = await sftp.downloadFile(file.path);
+    const records = await parser.parse(content as string);
+    const result = await mapper.map(records);
+
+    // Create new job for this file (PER_FILE strategy)
+    const job = await client.createJob({
+      name: `Cycle Count - ${file.path}`,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    });
+
+    // Send in batches of 500
+    for (let i = 0; i < result.data.length; i += 500) {
+      const batch = result.data.slice(i, i + 500);
+      await client.sendBatch(job.id, {
+        action: 'UPSERT',
+        entityType: 'INVENTORY',
+        source: 'SFTP_CSV_IMPORT',
+        event: 'CYCLE_COUNT_UPDATE',
+        entities: batch
+      });
+    }
+
+    await stateService.updateSyncState(kvAdapter, [{
+      fileName: file.path,
+      lastModified: new Date().toISOString(),
+      recordCount: result.data.length
+    }]);
+  } catch (error) {
+    console.error(`Failed to process ${file.path}`, error);
+  }
+}
+```
+
+### Use Case 3: 3PL Partner Inventory
+
+**Scenario**: Third-party logistics partner sends inventory feeds via SFTP twice daily.
+
+**Requirements:**
+- ✅ Validate partner data format
+- ✅ Enrich with location mapping
+- ✅ Handle missing/invalid records
+- ✅ Audit trail for compliance
+
+**Solution**: Custom resolvers with validation
+```typescript
+const mapper = new UniversalMapper({
+  fields: {
+    ref: { source: 'partner_sku', required: true },
+    productRef: { source: 'partner_sku', required: true },
+    locationRef: {
+      source: 'partner_location_code',
+      resolver: 'custom.mapPartnerLocation',  // ✅ Custom business logic
+      required: true
+    },
+    qty: {
+      source: 'available_quantity',
+      resolver: 'sdk.parseInt',
+      required: true,
+      validation: (value) => value >= 0  // ✅ Business rule
+    }
+  }
+}, {
+  customResolvers: {
+    'custom.mapPartnerLocation': (value, sourceData, config, helpers) => {
+      const mapping = {
+        'P1': 'WH-PARTNER-EAST',
+        'P2': 'WH-PARTNER-WEST'
+      };
+      const mapped = mapping[value];
+      if (!mapped) {
+        throw new Error(`Unknown partner location: ${value}`);
+      }
+      return mapped;
+    }
+  }
+});
+```
+
+---
+
+## Key Ingestion Features
+
+### 1. Format Flexibility
+
+The SDK supports **multiple input formats** with the same workflow:
+
+| Format | Parser | Use Case |
+|--------|--------|----------|
+| CSV | `CSVParserService` | Most common - simple, human-readable |
+| Parquet | `ParquetParserService` | Large datasets - columnar, compressed |
+| XML | `XMLParserService` | Enterprise systems - SFCC, SAP, Magento |
+| JSON | `JSONParserService` | APIs, webhooks, modern systems |
+
+### 2. State Management
+
+**Problem**: Without state management, reprocessing files creates duplicates.
+
+**Solution**: `StateService` tracks processed files with distributed locking:
+
+```typescript
+const logger = toStructuredLogger(createConsoleLogger(), {
+  service: 'inventory-ingestion',
+  correlationId: generateCorrelationId()
+});
+
+const stateService = new StateService(logger);
+const kvAdapter = new VersoriKVAdapter(openKv());
+
+// Check if file already processed
+if (await stateService.isFileProcessed(kvAdapter, 'file.csv')) {
+  logger.info('File already processed, skipping');
+  return;
+}
+
+// Acquire lock for concurrent safety
+const lockAcquired = await stateService.acquireLock('ingestion-lock', kvAdapter, 15);
+if (!lockAcquired) {
+  logger.info('File locked by another process');
+  return;
+}
+
+try {
+  // Process file
+  await processFile('file.csv');
+
+  // Mark as processed
+  await stateService.updateSyncState(kvAdapter, [{
+    fileName: 'file.csv',
+    lastModified: new Date().toISOString(),
+    recordCount: recordCount
+  }]);
+} finally {
+  await stateService.releaseLock('ingestion-lock', kvAdapter);
+}
+```
+
+### 3. Performance Optimization
+
+**Batch Size Optimization**: Automatically calculate optimal batch size based on data:
+
+```typescript
+function calculateOptimalBatchSize(
+  totalRecords: number,
+  avgRecordSize: number
+): number {
+  const MAX_BATCH_SIZE = 10000;
+  const MAX_PAYLOAD_SIZE = 10 * 1024 * 1024; // 10MB
+
+  const recordsPerPayload = Math.floor(MAX_PAYLOAD_SIZE / avgRecordSize);
+  return Math.min(recordsPerPayload, MAX_BATCH_SIZE);
+}
+
+const batchSize = calculateOptimalBatchSize(50000, 200);
+// Result: Optimal size based on record size and payload limits
+```
+
+**Parallel Processing**: Process multiple files/batches with configurable strategies:
+
+```typescript
+import { createClient, S3DataSource, CSVParserService, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize services
+const client = await createClient({ config });
+const s3 = new S3DataSource(
+  {
+    type: 'S3_CSV',
+    connectionId: 's3-inventory',
+    name: 'S3 Inventory',
+    s3Config: s3Config,
+  },
+  logger
+);
+const parser = new CSVParserService();
+const mapper = new UniversalMapper(mappingConfig);
+
+// Create daily job (reuse for all batches)
+const job = await client.createJob({
+  name: 'Daily Inventory Sync',
+  retailerId: process.env.FLUENT_RETAILER_ID!
+});
+
+// Process multiple files
+const files = await s3.listFiles({ prefix: 'data/', maxKeys: 10 });
+
+for (const file of files) {
+  const content = await s3.downloadFile(file.path);
+  const records = await parser.parse(content as string);
+  const result = await mapper.map(records);
+
+  // Send in batches of 1000
+  for (let i = 0; i < result.data.length; i += 1000) {
+    const batch = result.data.slice(i, i + 1000);
+    await client.sendBatch(job.id, {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: batch
+    });
+  }
+}
+```
+
+### 4. Error Handling
+
+**Fail-Fast with Recovery**: Stop on critical errors, retry transient failures:
+
+```typescript
+class RetryableIngestion {
+  async executeWithRetry<T>(
+    operation: () => Promise<T>,
+    context: string
+  ): Promise<T> {
+    let lastError: Error;
+    let delay = 1000;
+
+    for (let attempt = 1; attempt <= 3; attempt++) {
+      try {
+        return await operation();
+      } catch (error) {
+        lastError = error;
+
+        // Retry network errors, fail on validation errors
+        if (!this.isRetryable(error)) {
+          throw error;  // ✅ Fail fast on validation errors
+        }
+
+        console.log(`Retry ${attempt}/3 for ${context}`);
+        await this.sleep(delay);
+        delay *= 2;  // Exponential backoff
+      }
+    }
+
+    throw new Error(`Failed after 3 retries: ${lastError.message}`);
+  }
+
+  private isRetryable(error: any): boolean {
+    return error.code === 'NETWORK_ERROR' ||
+           error.code === 'TIMEOUT_ERROR' ||
+           error.code === 'RATE_LIMIT_ERROR';
+  }
+}
+```
+
+---
+
+## Ingestion vs Extraction
+
+The SDK provides **two complementary workflows**:
+
+| Aspect | Ingestion (this guide) | Extraction |
+|--------|------------------------|-----------|
+| **Direction** | External → Fluent Commerce | Fluent Commerce → External |
+| **Primary API** | Batch API (UPSERT) | GraphQL Queries |
+| **Use Case** | Update inventory positions | Export data for analytics/BI |
+| **Data Source** | S3, SFTP, files | Fluent Commerce GraphQL API |
+| **Data Destination** | Fluent Commerce | S3, SFTP, files, databases |
+| **Typical Frequency** | Hourly, daily | Daily, weekly, on-demand |
+| **State Management** | Prevent duplicate ingestion | Track extraction checkpoints |
+
+**See Also**: [Data Extraction Guide](../../extraction/) for the reverse workflow.
+
+---
+
+## Key Takeaways
+
+- 🎯 **Batch API is for inventory only** - Use GraphQL mutations for orders, products, locations
+- 🎯 **State management prevents duplicates** - Essential for production workflows
+- 🎯 **UniversalMapper transforms data** - Declarative field mapping with validation
+- 🎯 **Job strategies optimize performance** - DAILY for frequent updates, PER_FILE for large datasets
+- 🎯 **Error handling is critical** - Implement retry logic and fail-fast validation
+
+---
+
+## Next Steps
+
+Continue to [Module 2: Quick Start](../../auto-pagination/modules/auto-pagination-02-quick-start.md) to build your first ingestion workflow with a complete working example.
+
+---
+
+[← Previous: Main Guide](../ingestion-readme.md) | [Back to Guide](../ingestion-readme.md) | [Next: Quick Start →](../../auto-pagination/modules/auto-pagination-02-quick-start.md)
+
+## Related Documentation
+
+- [Batch API Reference](../../api-reference/modules/api-reference-01-client-api.md#job-batch-operations) - Complete API documentation
+- [Universal Mapping Guide](../../mapping/mapping-readme.md) - Field transformation patterns
+- [State Management](./02-core-guides-ingestion-07-state-management.md) - Duplicate prevention in detail
+- [Orchestrators Guide](../../extraction/modules/02-core-guides-extraction-08-extraction-orchestrator.md) - High-level workflow coordination