npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.53 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (495) hide show

package/docs/04-REFERENCE/architecture/architecture-03-data-flow.md CHANGED Viewed

@@ -1,690 +1,690 @@
-# Data Flow Architecture
-This document explains the three primary data flow patterns in the FC Connect SDK: Ingestion, Extraction, and Transformation.
-## Overview
-The SDK supports three main data flow patterns:
-```mermaid
-graph TB
-    subgraph "1. Ingestion Flow"
-        I1[External Files<br/>CSV, Parquet] --> I2[Parse & Map] --> I3[Batch API] --> I4[Fluent Commerce]
-    end
-    subgraph "2. Extraction Flow"
-        E1[Fluent Commerce] --> E2[GraphQL Query] --> E3[Transform & Map] --> E4[External Storage<br/>S3, SFTP]
-    end
-    subgraph "3. Transformation Flow"
-        T1[External Data<br/>XML, JSON] --> T2[Parse & Map] --> T3[GraphQL Mutation] --> T4[Fluent Commerce]
-    end
-    style I1 fill:#ffebee
-    style I2 fill:#fff3e0
-    style I3 fill:#e3f2fd
-    style I4 fill:#e8f5e9
-    style E1 fill:#e8f5e9
-    style E2 fill:#e3f2fd
-    style E3 fill:#f3e5f5
-    style E4 fill:#ffebee
-    style T1 fill:#ffebee
-    style T2 fill:#fff3e0
-    style T3 fill:#e3f2fd
-    style T4 fill:#e8f5e9
-```
-## 1. Ingestion Flow
-Ingestion processes external files (CSV, Parquet) and sends data to Fluent Commerce via the Batch API.
-### High-Level Flow
-```mermaid
-flowchart TD
-    START[Start Ingestion] --> DISCOVER[Discover Files<br/>S3/SFTP]
-    DISCOVER --> CHECK{File Already<br/>Processed?}
-    CHECK -->|Yes| SKIP[Skip File]
-    CHECK -->|No| DOWNLOAD[Download File]
-    DOWNLOAD --> PARSE[Parse File<br/>CSV/Parquet Parser]
-    PARSE --> MAP[Map Fields<br/>Universal Mapper]
-    MAP --> BATCH[Create Batches<br/>Group rows]
-    BATCH --> JOB{Job Exists?}
-    JOB -->|No| CREATE_JOB[Create New Job]
-    JOB -->|Yes| USE_JOB[Use Existing Job]
-    CREATE_JOB --> SEND
-    USE_JOB --> SEND[Send Batches<br/>to Fluent API]
-    SEND --> VERIFY{All Batches<br/>Successful?}
-    VERIFY -->|Yes| ARCHIVE[Archive File]
-    VERIFY -->|No| ERROR[Handle Errors]
-    ARCHIVE --> STATE[Update State<br/>Mark Processed]
-    STATE --> NEXT{More Files?}
-    NEXT -->|Yes| DISCOVER
-    NEXT -->|No| END[Complete]
-    ERROR --> NEXT
-    style START fill:#e3f2fd
-    style PARSE fill:#fff3e0
-    style MAP fill:#f3e5f5
-    style SEND fill:#e8f5e9
-    style END fill:#e3f2fd
-```
-### Detailed Ingestion Components
-```mermaid
-sequenceDiagram
-    participant DS as Data Source<br/>(S3/SFTP)
-    participant Parser
-    participant Mapper
-    participant State
-    participant BatchAPI
-    participant Fluent
-    DS->>Parser: Stream file data
-    Parser->>Parser: Validate format
-    Parser-->>Mapper: Row by row
-    loop For Each Row
-        Mapper->>Mapper: Apply field mappings
-        Mapper->>Mapper: Run resolvers
-        Mapper-->>BatchAPI: Mapped entity
-    end
-    BatchAPI->>State: Check job status
-    State-->>BatchAPI: Job ID or create new
-    BatchAPI->>Fluent: Create job (if needed)
-    Fluent-->>BatchAPI: Job ID
-    loop For Each Batch
-        BatchAPI->>Fluent: Send batch
-        Fluent-->>BatchAPI: Batch ID
-    end
-    BatchAPI->>State: Mark file processed
-    BatchAPI->>DS: Archive file
-```
-### Job Strategies
-The SDK supports three job creation strategies:
-```mermaid
-graph TB
-    subgraph "DAILY Strategy"
-        D1[Create Job Once] --> D2[Reuse All Day]
-        D2 --> D3[Send All Batches<br/>to Same Job]
-    end
-    subgraph "SINGLE Strategy"
-        S1[Create One Job] --> S2[Send All Batches]
-        S2 --> S3[Complete]
-    end
-    subgraph "PER_FILE Strategy"
-        P1[Process File] --> P2[Create New Job]
-        P2 --> P3[Send All Batches]
-        P3 --> P4[Next File]
-        P4 --> P1
-    end
-    style D1 fill:#e8f5e9
-    style D2 fill:#e8f5e9
-    style D3 fill:#e8f5e9
-    style S1 fill:#fff3e0
-    style S2 fill:#fff3e0
-    style S3 fill:#fff3e0
-    style P1 fill:#f3e5f5
-    style P2 fill:#f3e5f5
-    style P3 fill:#f3e5f5
-    style P4 fill:#f3e5f5
-```
-**When to Use Each:**
-- **DAILY**: High-volume, continuous ingestion (recommended) - Reuses same job for all files processed in a day
-- **SINGLE**: One-time ingestion - Creates one job for entire workflow run
-- **PER_FILE**: Low volume, need file-level tracking - Creates new job per file
-### File Archival
-```mermaid
-flowchart LR
-    SOURCE[Source Bucket] --> PROCESS{Processing<br/>Success?}
-    PROCESS -->|Success| STRATEGY{Archival<br/>Strategy}
-    PROCESS -->|Failure| ERROR[Error Bucket]
-    STRATEGY -->|Move| MOVE[Copy to Archive<br/>Delete from Source]
-    STRATEGY -->|Copy| COPY[Copy to Archive<br/>Keep Source]
-    STRATEGY -->|Copy-Delete| CD[Copy to Archive<br/>Explicit Delete]
-    MOVE --> ARCHIVE[Archive Bucket]
-    COPY --> ARCHIVE
-    CD --> ARCHIVE
-    style SOURCE fill:#e3f2fd
-    style ARCHIVE fill:#e8f5e9
-    style ERROR fill:#ffebee
-```
-## 2. Extraction Flow
-Extraction queries data from Fluent Commerce and saves it to external storage.
-> **Note:** This section shows the optional ExtractionOrchestrator. For the recommended
-> building-block approach, see [Building-Block Extraction Pattern](#building-block-extraction) below.
-### High-Level Flow
-```mermaid
-flowchart TD
-    START[Start Extraction] --> QUERY[Execute GraphQL Query]
-    QUERY --> PAGE{More Pages?}
-    PAGE -->|Yes| FETCH[Fetch Next Page<br/>Cursor-based]
-    PAGE -->|No| TRANSFORM[Transform Data]
-    FETCH --> ACCUMULATE[Accumulate Results]
-    ACCUMULATE --> PAGE
-    TRANSFORM --> MAP[Apply Field Mappings]
-    MAP --> FORMAT{Output Format?}
-    FORMAT -->|JSON| JSON[Convert to JSON]
-    FORMAT -->|Parquet| PARQUET[Convert to Parquet]
-    FORMAT -->|CSV| CSV[Convert to CSV]
-    JSON --> UPLOAD[Upload to S3/SFTP]
-    PARQUET --> UPLOAD
-    CSV --> UPLOAD
-    UPLOAD --> STATE[Update State]
-    STATE --> END[Complete]
-    style START fill:#e3f2fd
-    style QUERY fill:#fff3e0
-    style MAP fill:#f3e5f5
-    style UPLOAD fill:#e8f5e9
-    style END fill:#e3f2fd
-```
-### Detailed Extraction Components
-```mermaid
-sequenceDiagram
-    participant Client
-    participant Fluent
-    participant Mapper
-    participant Storage
-    Client->>Fluent: Execute GraphQL query (page 1)
-    Fluent-->>Client: Results + cursor
-    loop Pagination
-        Client->>Fluent: Fetch next page
-        Fluent-->>Client: Results + cursor
-    end
-    Client->>Mapper: Apply extraction mappings
-    Mapper->>Mapper: Transform fields
-    Mapper-->>Client: Mapped data
-    Client->>Storage: Upload results
-    Storage-->>Client: Upload confirmation
-    Client-->>Client: Extraction complete
-```
-### Pagination Strategy
-The SDK uses cursor-based pagination following Relay specification:
-⚠️ **SDK AUTO-PAGINATION BEHAVIOR:**
-The SDK's auto-pagination feature automatically handles cursor extraction from edges.
-**Users should rely on `client.graphql()` with auto-pagination enabled** instead of implementing manual pagination.
-#### Recommended Approach (Let SDK Handle It)
-```typescript
-// ✅ CORRECT - Let SDK handle pagination automatically
-const result = await client.graphql({
-  query: VIRTUAL_POSITIONS_QUERY,
-  variables: { first: 100 },
-  // Auto-pagination enabled by default for queries with pagination variables
-});
-// SDK internally extracts cursors from edges - you don't need to do this manually
-```
-#### Manual Cursor Extraction (Advanced - Not Recommended)
-If you need manual control (rare), the SDK extracts cursors like this internally:
-```typescript
-// ❌ WRONG - Don't implement this yourself, SDK does it for you
-const cursor = edges[edges.length - 1]?.cursor;
-// ✅ CORRECT - Use SDK's auto-pagination instead
-const result = await client.graphql({
-  query: '...',
-  variables: { first: 100 }
-  // SDK handles cursor extraction automatically
-});
-```
-**Why This Matters:** Fluent Commerce returns cursors in edges, not pageInfo.endCursor.
-The SDK's auto-pagination handles this automatically. Manual implementation is error-prone and unnecessary.
-**See:** [Auto-Pagination Guide](../../02-CORE-GUIDES/auto-pagination/) for complete documentation.
-```mermaid
-flowchart LR
-    Q1[Query Page 1<br/>first: 100] --> R1[Results + edges array]
-    R1 --> EXTRACT1[Extract: edges-last-.cursor]
-    EXTRACT1 --> Q2[Query Page 2<br/>after: lastEdge.cursor]
-    Q2 --> R2[Results + edges array]
-    R2 --> EXTRACT2[Extract: edges-last-.cursor]
-    EXTRACT2 --> Q3[Query Page 3<br/>after: lastEdge.cursor]
-    Q3 --> R3[Results + hasNextPage: false]
-    R3 --> DONE[All Data Fetched]
-    style Q1 fill:#e3f2fd
-    style Q2 fill:#e3f2fd
-    style Q3 fill:#e3f2fd
-    style EXTRACT1 fill:#fff3e0
-    style EXTRACT2 fill:#fff3e0
-    style DONE fill:#e8f5e9
-```
-**Pagination Features:**
-- Automatic cursor management (extracts from edge cursors)
-- Handles Fluent-specific cursor extraction
-- Configurable page size
-- Rate limiting between pages
-- Memory-efficient streaming
-**Correct Pagination Implementation:**
-```typescript
-let hasNextPage = true;
-let cursor: string | undefined = undefined;
-while (hasNextPage) {
-  const result = await client.graphql({
-    query: `
-      query GetData($first: Int, $after: String) {
-        virtualPositions(first: $first, after: $after) {
-          edges {
-            node { id ref quantity }
-            cursor  # ✅ Must include cursor in query
-          }
-          pageInfo {
-            hasNextPage  # ✅ Fluent returns this
-            # ❌ NO endCursor field in Fluent API
-          }
-        }
-      }
-    `,
-    variables: { first: 100, after: cursor },
-  });
-  const edges = result.data.virtualPositions.edges;
-  // Extract cursor from last edge (Fluent-specific)
-  cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
-  hasNextPage = result.data.virtualPositions.pageInfo.hasNextPage;
-  // Process data
-  for (const edge of edges) {
-    processRecord(edge.node);
-  }
-}
-```
-### Building-Block Extraction (Recommended) {#building-block-extraction}
-For maximum flexibility, compose SDK services directly instead of using the orchestrator:
-```typescript
-// ✅ RECOMMENDED - Direct client.graphql() usage with auto-pagination
-const result = await client.graphql({
-  query: VIRTUAL_POSITIONS_QUERY,
-  variables: { first: 100 },
-  // Auto-pagination enabled by default
-});
-// Extract and transform data
-const items = result.data?.virtualPositions?.edges?.map(e => e.node) || [];
-// Use UniversalMapper if needed
-const mapper = new UniversalMapper(mappingConfig);
-const transformed = await Promise.all(items.map(item => mapper.map(item)));
-// Upload to S3/SFTP
-await s3.uploadFile('output.json', JSON.stringify(transformed));
-```
-**Extraction Flow Diagram (Building Blocks):**
-```mermaid
-flowchart TD
-    START[Start Extraction] --> CLIENT[client.graphql with auto-pagination]
-    CLIENT --> EXTRACT[Extract nodes from edges]
-    EXTRACT --> TRANSFORM[Transform with UniversalMapper]
-    TRANSFORM --> UPLOAD[Upload to S3/SFTP]
-    UPLOAD --> END[Complete]
-    style START fill:#e3f2fd
-    style CLIENT fill:#fff3e0
-    style TRANSFORM fill:#f3e5f5
-    style UPLOAD fill:#e8f5e9
-    style END fill:#e3f2fd
-```
-**Recommendation:** Use building blocks for production. Use orchestrator for prototyping.
-## 3. Transformation Flow
-Transformation converts external data formats (XML, JSON) to Fluent Commerce GraphQL mutations.
-### High-Level Flow
-```mermaid
-flowchart TD
-    START[Receive Data<br/>XML/JSON] --> PARSE[Parse Input<br/>XMLParser/JSONParser]
-    PARSE --> MAP[Apply Mutation Mapping<br/>GraphQLMutationMapper]
-    MAP --> VALIDATE[Validate Mutation]
-    VALIDATE --> EXECUTE[Execute GraphQL Mutation]
-    EXECUTE --> CHECK{Mutation<br/>Successful?}
-    CHECK -->|Yes| STATE[Update State]
-    CHECK -->|No| ERROR[Handle Error]
-    STATE --> RESPONSE[Return Success]
-    ERROR --> RETRY{Retry?}
-    RETRY -->|Yes| EXECUTE
-    RETRY -->|No| FAIL[Return Failure]
-    RESPONSE --> END[Complete]
-    FAIL --> END
-    style START fill:#e3f2fd
-    style PARSE fill:#fff3e0
-    style MAP fill:#f3e5f5
-    style EXECUTE fill:#e8f5e9
-    style END fill:#e3f2fd
-    style ERROR fill:#ffebee
-```
-### SFCC Order Example
-Real-world example: Converting SFCC XML orders to Fluent Commerce mutations.
-```mermaid
-sequenceDiagram
-    participant SFCC
-    participant Webhook
-    participant XMLParser
-    participant MutationMapper
-    participant Fluent
-    SFCC->>Webhook: POST /order (XML payload)
-    Webhook->>XMLParser: Parse XML
-    XMLParser->>XMLParser: Extract fields<br/>(order-no, customer, items)
-    XMLParser-->>MutationMapper: Parsed data
-    MutationMapper->>MutationMapper: Load mapping config
-    MutationMapper->>MutationMapper: Build GraphQL mutation
-    MutationMapper->>MutationMapper: Map nested arrays<br/>(line items, addresses)
-    MutationMapper->>Fluent: Execute createOrder mutation
-    Fluent-->>MutationMapper: Order created
-    MutationMapper-->>Webhook: Success response
-    Webhook-->>SFCC: 200 OK
-```
-### Mutation Mapping Process
-```mermaid
-flowchart TB
-    XML[XML/JSON Input] --> PARSE[Parse to Object]
-    PARSE --> CONFIG[Load Mapping Config]
-    CONFIG --> ROOT[Map Root Fields]
-    ROOT --> NESTED{Has Nested<br/>Arrays?}
-    NESTED -->|Yes| ARRAY[Map Array Items]
-    NESTED -->|No| BUILD
-    ARRAY --> CONTEXT[Switch Context<br/>to Array Item]
-    CONTEXT --> MAP_ITEM[Map Item Fields]
-    MAP_ITEM --> NEXT{More Items?}
-    NEXT -->|Yes| ARRAY
-    NEXT -->|No| BUILD[Build GraphQL Mutation]
-    BUILD --> VARIABLES[Extract Variables]
-    VARIABLES --> MUTATION[Complete Mutation Object]
-    MUTATION --> EXECUTE[Execute Against API]
-    style XML fill:#e3f2fd
-    style PARSE fill:#fff3e0
-    style CONFIG fill:#f3e5f5
-    style EXECUTE fill:#e8f5e9
-```
-## Universal Mapping Engine
-All three flows use the same Universal Mapper with consistent field resolution:
-```mermaid
-flowchart TD
-    START[Field Config] --> PRIORITY{Resolution<br/>Priority}
-    PRIORITY -->|1| STATIC{Has Static<br/>Value?}
-    PRIORITY -->|2| SOURCE{Has Source<br/>Path?}
-    PRIORITY -->|3| RESOLVER{Has Resolver?}
-    PRIORITY -->|4| DEFAULT{Has Default?}
-    STATIC -->|Yes| RETURN1[Return Static Value]
-    STATIC -->|No| SOURCE
-    SOURCE -->|Yes| EXTRACT[Extract from Source Data]
-    SOURCE -->|No| RESOLVER
-    EXTRACT --> RESOLVER
-    RESOLVER -->|Yes| TRANSFORM[Apply Resolver<br/>sdk.* or custom.*]
-    RESOLVER -->|No| DEFAULT
-    TRANSFORM --> VALIDATE
-    DEFAULT -->|Yes| RETURN2[Return Default]
-    DEFAULT -->|No| REQUIRED{Field<br/>Required?}
-    REQUIRED -->|Yes| ERROR[Throw Error]
-    REQUIRED -->|No| RETURN3[Return undefined]
-    RETURN1 --> VALIDATE[Validate Result]
-    RETURN2 --> VALIDATE
-    RETURN3 --> VALIDATE
-    VALIDATE --> END[Return Mapped Value]
-    style START fill:#e3f2fd
-    style TRANSFORM fill:#fff3e0
-    style VALIDATE fill:#e8f5e9
-    style ERROR fill:#ffebee
-    style END fill:#e3f2fd
-```
-## State Management Flow
-All flows integrate with state management to prevent duplicate processing:
-```mermaid
-sequenceDiagram
-    participant Flow
-    participant StateService
-    participant KVStore
-    Flow->>StateService: isFileProcessed(kv, fileName, workflowId)
-    StateService->>KVStore: Get file key
-    KVStore-->>StateService: File status
-    alt File Already Processed
-        StateService-->>Flow: true (skip)
-    else File Not Processed
-        StateService-->>Flow: false (process)
-        Flow->>Flow: Process file
-        Flow->>StateService: updateSyncState(kv, processedFiles, workflowId)
-        StateService->>StateService: Add to processedFiles array
-        StateService->>KVStore: Update state
-        KVStore-->>StateService: Confirmation
-        StateService-->>Flow: Marked
-    end
-```
-**State Object Structure:**
-```typescript
-{
-  processedFiles: [
-    { key: 'file1.csv', timestamp: '2025-01-15T10:00:00Z' },
-    { key: 'file2.csv', timestamp: '2025-01-15T11:00:00Z' }
-  ],
-  dailyJobId: 'job-123',
-  currentJobBatchCount: 5,
-  lastProcessedTimestamp: '2025-01-15T11:00:00Z'
-}
-```
-## Error Handling Across Flows
-```mermaid
-flowchart TD
-    ERROR[Error Occurs] --> TYPE{Error Type}
-    TYPE -->|4xx Client Error| CLIENT[Client Error Handler]
-    TYPE -->|5xx Server Error| SERVER[Server Error Handler]
-    TYPE -->|Network Error| NETWORK[Network Error Handler]
-    TYPE -->|Validation Error| VALIDATION[Validation Error Handler]
-    CLIENT --> LOG1[Log Error]
-    LOG1 --> FAIL[Fail Fast<br/>No Retry]
-    SERVER --> LOG2[Log Error]
-    LOG2 --> RETRY{Retry Count<br/>< Max?}
-    RETRY -->|Yes| BACKOFF[Exponential Backoff<br/>1s, 2s, 4s]
-    RETRY -->|No| FAIL
-    BACKOFF --> REATTEMPT[Reattempt Operation]
-    REATTEMPT --> SUCCESS{Successful?}
-    SUCCESS -->|Yes| CONTINUE[Continue Processing]
-    SUCCESS -->|No| RETRY
-    NETWORK --> LOG3[Log Error]
-    LOG3 --> RETRY
-    VALIDATION --> LOG4[Log Error]
-    LOG4 --> STRATEGY{Error Strategy}
-    STRATEGY -->|fail-fast| FAIL
-    STRATEGY -->|continue-on-error| SKIP[Skip Item<br/>Continue]
-    SKIP --> CONTINUE
-    style ERROR fill:#ffebee
-    style FAIL fill:#ff5252
-    style CONTINUE fill:#e8f5e9
-    style BACKOFF fill:#fff3e0
-```
-## Performance Optimizations
-### Streaming Pipeline
-```mermaid
-flowchart LR
-    FILE[Large File<br/>1GB] --> STREAM1[Stream Reader]
-    STREAM1 --> CHUNK1[Chunk 1]
-    STREAM1 --> CHUNK2[Chunk 2]
-    STREAM1 --> CHUNK3[Chunk N]
-    CHUNK1 --> PARSE1[Parse]
-    CHUNK2 --> PARSE2[Parse]
-    CHUNK3 --> PARSE3[Parse]
-    PARSE1 --> MAP1[Map]
-    PARSE2 --> MAP2[Map]
-    PARSE3 --> MAP3[Map]
-    MAP1 --> BATCH1[Batch 1]
-    MAP2 --> BATCH2[Batch 2]
-    MAP3 --> BATCH3[Batch N]
-    BATCH1 --> API[Fluent API]
-    BATCH2 --> API
-    BATCH3 --> API
-    style FILE fill:#ffebee
-    style STREAM1 fill:#e3f2fd
-    style API fill:#e8f5e9
-```
-**Benefits:**
-- Constant memory usage regardless of file size
-- Parallel processing of chunks
-- Early error detection
-- Faster overall throughput
-### Parallel Operations
-```mermaid
-graph TB
-    START[Multiple Files] --> SPLIT[Split into Batches]
-    SPLIT --> P1[Process File 1]
-    SPLIT --> P2[Process File 2]
-    SPLIT --> P3[Process File 3]
-    SPLIT --> PN[Process File N]
-    P1 --> WAIT[Wait All Complete]
-    P2 --> WAIT
-    P3 --> WAIT
-    PN --> WAIT
-    WAIT --> AGGREGATE[Aggregate Results]
-    AGGREGATE --> END[Complete]
-    style START fill:#e3f2fd
-    style SPLIT fill:#fff3e0
-    style WAIT fill:#f3e5f5
-    style END fill:#e8f5e9
-```
-## Next Steps
-- [Service Layer Architecture](./architecture-04-service-layer.md) - Explore service organization
-- [Integration Architecture](./architecture-05-integration-architecture.md) - External system integrations
-- [State Management](./architecture-06-state-management.md) - Distributed state patterns
-- [Universal Mapping Guide](../../02-CORE-GUIDES/mapping/mapping-readme.md) - Field mapping details
+# Data Flow Architecture
+This document explains the three primary data flow patterns in the FC Connect SDK: Ingestion, Extraction, and Transformation.
+## Overview
+The SDK supports three main data flow patterns:
+```mermaid
+graph TB
+    subgraph "1. Ingestion Flow"
+        I1[External Files<br/>CSV, Parquet] --> I2[Parse & Map] --> I3[Batch API] --> I4[Fluent Commerce]
+    end
+    subgraph "2. Extraction Flow"
+        E1[Fluent Commerce] --> E2[GraphQL Query] --> E3[Transform & Map] --> E4[External Storage<br/>S3, SFTP]
+    end
+    subgraph "3. Transformation Flow"
+        T1[External Data<br/>XML, JSON] --> T2[Parse & Map] --> T3[GraphQL Mutation] --> T4[Fluent Commerce]
+    end
+    style I1 fill:#ffebee
+    style I2 fill:#fff3e0
+    style I3 fill:#e3f2fd
+    style I4 fill:#e8f5e9
+    style E1 fill:#e8f5e9
+    style E2 fill:#e3f2fd
+    style E3 fill:#f3e5f5
+    style E4 fill:#ffebee
+    style T1 fill:#ffebee
+    style T2 fill:#fff3e0
+    style T3 fill:#e3f2fd
+    style T4 fill:#e8f5e9
+```
+## 1. Ingestion Flow
+Ingestion processes external files (CSV, Parquet) and sends data to Fluent Commerce via the Batch API.
+### High-Level Flow
+```mermaid
+flowchart TD
+    START[Start Ingestion] --> DISCOVER[Discover Files<br/>S3/SFTP]
+    DISCOVER --> CHECK{File Already<br/>Processed?}
+    CHECK -->|Yes| SKIP[Skip File]
+    CHECK -->|No| DOWNLOAD[Download File]
+    DOWNLOAD --> PARSE[Parse File<br/>CSV/Parquet Parser]
+    PARSE --> MAP[Map Fields<br/>Universal Mapper]
+    MAP --> BATCH[Create Batches<br/>Group rows]
+    BATCH --> JOB{Job Exists?}
+    JOB -->|No| CREATE_JOB[Create New Job]
+    JOB -->|Yes| USE_JOB[Use Existing Job]
+    CREATE_JOB --> SEND
+    USE_JOB --> SEND[Send Batches<br/>to Fluent API]
+    SEND --> VERIFY{All Batches<br/>Successful?}
+    VERIFY -->|Yes| ARCHIVE[Archive File]
+    VERIFY -->|No| ERROR[Handle Errors]
+    ARCHIVE --> STATE[Update State<br/>Mark Processed]
+    STATE --> NEXT{More Files?}
+    NEXT -->|Yes| DISCOVER
+    NEXT -->|No| END[Complete]
+    ERROR --> NEXT
+    style START fill:#e3f2fd
+    style PARSE fill:#fff3e0
+    style MAP fill:#f3e5f5
+    style SEND fill:#e8f5e9
+    style END fill:#e3f2fd
+```
+### Detailed Ingestion Components
+```mermaid
+sequenceDiagram
+    participant DS as Data Source<br/>(S3/SFTP)
+    participant Parser
+    participant Mapper
+    participant State
+    participant BatchAPI
+    participant Fluent
+    DS->>Parser: Stream file data
+    Parser->>Parser: Validate format
+    Parser-->>Mapper: Row by row
+    loop For Each Row
+        Mapper->>Mapper: Apply field mappings
+        Mapper->>Mapper: Run resolvers
+        Mapper-->>BatchAPI: Mapped entity
+    end
+    BatchAPI->>State: Check job status
+    State-->>BatchAPI: Job ID or create new
+    BatchAPI->>Fluent: Create job (if needed)
+    Fluent-->>BatchAPI: Job ID
+    loop For Each Batch
+        BatchAPI->>Fluent: Send batch
+        Fluent-->>BatchAPI: Batch ID
+    end
+    BatchAPI->>State: Mark file processed
+    BatchAPI->>DS: Archive file
+```
+### Job Strategies
+The SDK supports three job creation strategies:
+```mermaid
+graph TB
+    subgraph "DAILY Strategy"
+        D1[Create Job Once] --> D2[Reuse All Day]
+        D2 --> D3[Send All Batches<br/>to Same Job]
+    end
+    subgraph "SINGLE Strategy"
+        S1[Create One Job] --> S2[Send All Batches]
+        S2 --> S3[Complete]
+    end
+    subgraph "PER_FILE Strategy"
+        P1[Process File] --> P2[Create New Job]
+        P2 --> P3[Send All Batches]
+        P3 --> P4[Next File]
+        P4 --> P1
+    end
+    style D1 fill:#e8f5e9
+    style D2 fill:#e8f5e9
+    style D3 fill:#e8f5e9
+    style S1 fill:#fff3e0
+    style S2 fill:#fff3e0
+    style S3 fill:#fff3e0
+    style P1 fill:#f3e5f5
+    style P2 fill:#f3e5f5
+    style P3 fill:#f3e5f5
+    style P4 fill:#f3e5f5
+```
+**When to Use Each:**
+- **DAILY**: High-volume, continuous ingestion (recommended) - Reuses same job for all files processed in a day
+- **SINGLE**: One-time ingestion - Creates one job for entire workflow run
+- **PER_FILE**: Low volume, need file-level tracking - Creates new job per file
+### File Archival
+```mermaid
+flowchart LR
+    SOURCE[Source Bucket] --> PROCESS{Processing<br/>Success?}
+    PROCESS -->|Success| STRATEGY{Archival<br/>Strategy}
+    PROCESS -->|Failure| ERROR[Error Bucket]
+    STRATEGY -->|Move| MOVE[Copy to Archive<br/>Delete from Source]
+    STRATEGY -->|Copy| COPY[Copy to Archive<br/>Keep Source]
+    STRATEGY -->|Copy-Delete| CD[Copy to Archive<br/>Explicit Delete]
+    MOVE --> ARCHIVE[Archive Bucket]
+    COPY --> ARCHIVE
+    CD --> ARCHIVE
+    style SOURCE fill:#e3f2fd
+    style ARCHIVE fill:#e8f5e9
+    style ERROR fill:#ffebee
+```
+## 2. Extraction Flow
+Extraction queries data from Fluent Commerce and saves it to external storage.
+> **Note:** This section shows the optional ExtractionOrchestrator. For the recommended
+> building-block approach, see [Building-Block Extraction Pattern](#building-block-extraction) below.
+### High-Level Flow
+```mermaid
+flowchart TD
+    START[Start Extraction] --> QUERY[Execute GraphQL Query]
+    QUERY --> PAGE{More Pages?}
+    PAGE -->|Yes| FETCH[Fetch Next Page<br/>Cursor-based]
+    PAGE -->|No| TRANSFORM[Transform Data]
+    FETCH --> ACCUMULATE[Accumulate Results]
+    ACCUMULATE --> PAGE
+    TRANSFORM --> MAP[Apply Field Mappings]
+    MAP --> FORMAT{Output Format?}
+    FORMAT -->|JSON| JSON[Convert to JSON]
+    FORMAT -->|Parquet| PARQUET[Convert to Parquet]
+    FORMAT -->|CSV| CSV[Convert to CSV]
+    JSON --> UPLOAD[Upload to S3/SFTP]
+    PARQUET --> UPLOAD
+    CSV --> UPLOAD
+    UPLOAD --> STATE[Update State]
+    STATE --> END[Complete]
+    style START fill:#e3f2fd
+    style QUERY fill:#fff3e0
+    style MAP fill:#f3e5f5
+    style UPLOAD fill:#e8f5e9
+    style END fill:#e3f2fd
+```
+### Detailed Extraction Components
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Fluent
+    participant Mapper
+    participant Storage
+    Client->>Fluent: Execute GraphQL query (page 1)
+    Fluent-->>Client: Results + cursor
+    loop Pagination
+        Client->>Fluent: Fetch next page
+        Fluent-->>Client: Results + cursor
+    end
+    Client->>Mapper: Apply extraction mappings
+    Mapper->>Mapper: Transform fields
+    Mapper-->>Client: Mapped data
+    Client->>Storage: Upload results
+    Storage-->>Client: Upload confirmation
+    Client-->>Client: Extraction complete
+```
+### Pagination Strategy
+The SDK uses cursor-based pagination following Relay specification:
+⚠️ **SDK AUTO-PAGINATION BEHAVIOR:**
+The SDK's auto-pagination feature automatically handles cursor extraction from edges.
+**Users should rely on `client.graphql()` with auto-pagination enabled** instead of implementing manual pagination.
+#### Recommended Approach (Let SDK Handle It)
+```typescript
+// ✅ CORRECT - Let SDK handle pagination automatically
+const result = await client.graphql({
+  query: VIRTUAL_POSITIONS_QUERY,
+  variables: { first: 100 },
+  // Auto-pagination enabled by default for queries with pagination variables
+});
+// SDK internally extracts cursors from edges - you don't need to do this manually
+```
+#### Manual Cursor Extraction (Advanced - Not Recommended)
+If you need manual control (rare), the SDK extracts cursors like this internally:
+```typescript
+// ❌ WRONG - Don't implement this yourself, SDK does it for you
+const cursor = edges[edges.length - 1]?.cursor;
+// ✅ CORRECT - Use SDK's auto-pagination instead
+const result = await client.graphql({
+  query: '...',
+  variables: { first: 100 }
+  // SDK handles cursor extraction automatically
+});
+```
+**Why This Matters:** Fluent Commerce returns cursors in edges, not pageInfo.endCursor.
+The SDK's auto-pagination handles this automatically. Manual implementation is error-prone and unnecessary.
+**See:** [Auto-Pagination Guide](../../02-CORE-GUIDES/auto-pagination/) for complete documentation.
+```mermaid
+flowchart LR
+    Q1[Query Page 1<br/>first: 100] --> R1[Results + edges array]
+    R1 --> EXTRACT1[Extract: edges-last-.cursor]
+    EXTRACT1 --> Q2[Query Page 2<br/>after: lastEdge.cursor]
+    Q2 --> R2[Results + edges array]
+    R2 --> EXTRACT2[Extract: edges-last-.cursor]
+    EXTRACT2 --> Q3[Query Page 3<br/>after: lastEdge.cursor]
+    Q3 --> R3[Results + hasNextPage: false]
+    R3 --> DONE[All Data Fetched]
+    style Q1 fill:#e3f2fd
+    style Q2 fill:#e3f2fd
+    style Q3 fill:#e3f2fd
+    style EXTRACT1 fill:#fff3e0
+    style EXTRACT2 fill:#fff3e0
+    style DONE fill:#e8f5e9
+```
+**Pagination Features:**
+- Automatic cursor management (extracts from edge cursors)
+- Handles Fluent-specific cursor extraction
+- Configurable page size
+- Rate limiting between pages
+- Memory-efficient streaming
+**Correct Pagination Implementation:**
+```typescript
+let hasNextPage = true;
+let cursor: string | undefined = undefined;
+while (hasNextPage) {
+  const result = await client.graphql({
+    query: `
+      query GetData($first: Int, $after: String) {
+        virtualPositions(first: $first, after: $after) {
+          edges {
+            node { id ref quantity }
+            cursor  # ✅ Must include cursor in query
+          }
+          pageInfo {
+            hasNextPage  # ✅ Fluent returns this
+            # ❌ NO endCursor field in Fluent API
+          }
+        }
+      }
+    `,
+    variables: { first: 100, after: cursor },
+  });
+  const edges = result.data.virtualPositions.edges;
+  // Extract cursor from last edge (Fluent-specific)
+  cursor = edges.length > 0 ? edges[edges.length - 1].cursor : null;
+  hasNextPage = result.data.virtualPositions.pageInfo.hasNextPage;
+  // Process data
+  for (const edge of edges) {
+    processRecord(edge.node);
+  }
+}
+```
+### Building-Block Extraction (Recommended) {#building-block-extraction}
+For maximum flexibility, compose SDK services directly instead of using the orchestrator:
+```typescript
+// ✅ RECOMMENDED - Direct client.graphql() usage with auto-pagination
+const result = await client.graphql({
+  query: VIRTUAL_POSITIONS_QUERY,
+  variables: { first: 100 },
+  // Auto-pagination enabled by default
+});
+// Extract and transform data
+const items = result.data?.virtualPositions?.edges?.map(e => e.node) || [];
+// Use UniversalMapper if needed
+const mapper = new UniversalMapper(mappingConfig);
+const transformed = await Promise.all(items.map(item => mapper.map(item)));
+// Upload to S3/SFTP
+await s3.uploadFile('output.json', JSON.stringify(transformed));
+```
+**Extraction Flow Diagram (Building Blocks):**
+```mermaid
+flowchart TD
+    START[Start Extraction] --> CLIENT[client.graphql with auto-pagination]
+    CLIENT --> EXTRACT[Extract nodes from edges]
+    EXTRACT --> TRANSFORM[Transform with UniversalMapper]
+    TRANSFORM --> UPLOAD[Upload to S3/SFTP]
+    UPLOAD --> END[Complete]
+    style START fill:#e3f2fd
+    style CLIENT fill:#fff3e0
+    style TRANSFORM fill:#f3e5f5
+    style UPLOAD fill:#e8f5e9
+    style END fill:#e3f2fd
+```
+**Recommendation:** Use building blocks for production. Use orchestrator for prototyping.
+## 3. Transformation Flow
+Transformation converts external data formats (XML, JSON) to Fluent Commerce GraphQL mutations.
+### High-Level Flow
+```mermaid
+flowchart TD
+    START[Receive Data<br/>XML/JSON] --> PARSE[Parse Input<br/>XMLParser/JSONParser]
+    PARSE --> MAP[Apply Mutation Mapping<br/>GraphQLMutationMapper]
+    MAP --> VALIDATE[Validate Mutation]
+    VALIDATE --> EXECUTE[Execute GraphQL Mutation]
+    EXECUTE --> CHECK{Mutation<br/>Successful?}
+    CHECK -->|Yes| STATE[Update State]
+    CHECK -->|No| ERROR[Handle Error]
+    STATE --> RESPONSE[Return Success]
+    ERROR --> RETRY{Retry?}
+    RETRY -->|Yes| EXECUTE
+    RETRY -->|No| FAIL[Return Failure]
+    RESPONSE --> END[Complete]
+    FAIL --> END
+    style START fill:#e3f2fd
+    style PARSE fill:#fff3e0
+    style MAP fill:#f3e5f5
+    style EXECUTE fill:#e8f5e9
+    style END fill:#e3f2fd
+    style ERROR fill:#ffebee
+```
+### SFCC Order Example
+Real-world example: Converting SFCC XML orders to Fluent Commerce mutations.
+```mermaid
+sequenceDiagram
+    participant SFCC
+    participant Webhook
+    participant XMLParser
+    participant MutationMapper
+    participant Fluent
+    SFCC->>Webhook: POST /order (XML payload)
+    Webhook->>XMLParser: Parse XML
+    XMLParser->>XMLParser: Extract fields<br/>(order-no, customer, items)
+    XMLParser-->>MutationMapper: Parsed data
+    MutationMapper->>MutationMapper: Load mapping config
+    MutationMapper->>MutationMapper: Build GraphQL mutation
+    MutationMapper->>MutationMapper: Map nested arrays<br/>(line items, addresses)
+    MutationMapper->>Fluent: Execute createOrder mutation
+    Fluent-->>MutationMapper: Order created
+    MutationMapper-->>Webhook: Success response
+    Webhook-->>SFCC: 200 OK
+```
+### Mutation Mapping Process
+```mermaid
+flowchart TB
+    XML[XML/JSON Input] --> PARSE[Parse to Object]
+    PARSE --> CONFIG[Load Mapping Config]
+    CONFIG --> ROOT[Map Root Fields]
+    ROOT --> NESTED{Has Nested<br/>Arrays?}
+    NESTED -->|Yes| ARRAY[Map Array Items]
+    NESTED -->|No| BUILD
+    ARRAY --> CONTEXT[Switch Context<br/>to Array Item]
+    CONTEXT --> MAP_ITEM[Map Item Fields]
+    MAP_ITEM --> NEXT{More Items?}
+    NEXT -->|Yes| ARRAY
+    NEXT -->|No| BUILD[Build GraphQL Mutation]
+    BUILD --> VARIABLES[Extract Variables]
+    VARIABLES --> MUTATION[Complete Mutation Object]
+    MUTATION --> EXECUTE[Execute Against API]
+    style XML fill:#e3f2fd
+    style PARSE fill:#fff3e0
+    style CONFIG fill:#f3e5f5
+    style EXECUTE fill:#e8f5e9
+```
+## Universal Mapping Engine
+All three flows use the same Universal Mapper with consistent field resolution:
+```mermaid
+flowchart TD
+    START[Field Config] --> PRIORITY{Resolution<br/>Priority}
+    PRIORITY -->|1| STATIC{Has Static<br/>Value?}
+    PRIORITY -->|2| SOURCE{Has Source<br/>Path?}
+    PRIORITY -->|3| RESOLVER{Has Resolver?}
+    PRIORITY -->|4| DEFAULT{Has Default?}
+    STATIC -->|Yes| RETURN1[Return Static Value]
+    STATIC -->|No| SOURCE
+    SOURCE -->|Yes| EXTRACT[Extract from Source Data]
+    SOURCE -->|No| RESOLVER
+    EXTRACT --> RESOLVER
+    RESOLVER -->|Yes| TRANSFORM[Apply Resolver<br/>sdk.* or custom.*]
+    RESOLVER -->|No| DEFAULT
+    TRANSFORM --> VALIDATE
+    DEFAULT -->|Yes| RETURN2[Return Default]
+    DEFAULT -->|No| REQUIRED{Field<br/>Required?}
+    REQUIRED -->|Yes| ERROR[Throw Error]
+    REQUIRED -->|No| RETURN3[Return undefined]
+    RETURN1 --> VALIDATE[Validate Result]
+    RETURN2 --> VALIDATE
+    RETURN3 --> VALIDATE
+    VALIDATE --> END[Return Mapped Value]
+    style START fill:#e3f2fd
+    style TRANSFORM fill:#fff3e0
+    style VALIDATE fill:#e8f5e9
+    style ERROR fill:#ffebee
+    style END fill:#e3f2fd
+```
+## State Management Flow
+All flows integrate with state management to prevent duplicate processing:
+```mermaid
+sequenceDiagram
+    participant Flow
+    participant StateService
+    participant KVStore
+    Flow->>StateService: isFileProcessed(kv, fileName, workflowId)
+    StateService->>KVStore: Get file key
+    KVStore-->>StateService: File status
+    alt File Already Processed
+        StateService-->>Flow: true (skip)
+    else File Not Processed
+        StateService-->>Flow: false (process)
+        Flow->>Flow: Process file
+        Flow->>StateService: updateSyncState(kv, processedFiles, workflowId)
+        StateService->>StateService: Add to processedFiles array
+        StateService->>KVStore: Update state
+        KVStore-->>StateService: Confirmation
+        StateService-->>Flow: Marked
+    end
+```
+**State Object Structure:**
+```typescript
+{
+  processedFiles: [
+    { key: 'file1.csv', timestamp: '2025-01-15T10:00:00Z' },
+    { key: 'file2.csv', timestamp: '2025-01-15T11:00:00Z' }
+  ],
+  dailyJobId: 'job-123',
+  currentJobBatchCount: 5,
+  lastProcessedTimestamp: '2025-01-15T11:00:00Z'
+}
+```
+## Error Handling Across Flows
+```mermaid
+flowchart TD
+    ERROR[Error Occurs] --> TYPE{Error Type}
+    TYPE -->|4xx Client Error| CLIENT[Client Error Handler]
+    TYPE -->|5xx Server Error| SERVER[Server Error Handler]
+    TYPE -->|Network Error| NETWORK[Network Error Handler]
+    TYPE -->|Validation Error| VALIDATION[Validation Error Handler]
+    CLIENT --> LOG1[Log Error]
+    LOG1 --> FAIL[Fail Fast<br/>No Retry]
+    SERVER --> LOG2[Log Error]
+    LOG2 --> RETRY{Retry Count<br/>< Max?}
+    RETRY -->|Yes| BACKOFF[Exponential Backoff<br/>1s, 2s, 4s]
+    RETRY -->|No| FAIL
+    BACKOFF --> REATTEMPT[Reattempt Operation]
+    REATTEMPT --> SUCCESS{Successful?}
+    SUCCESS -->|Yes| CONTINUE[Continue Processing]
+    SUCCESS -->|No| RETRY
+    NETWORK --> LOG3[Log Error]
+    LOG3 --> RETRY
+    VALIDATION --> LOG4[Log Error]
+    LOG4 --> STRATEGY{Error Strategy}
+    STRATEGY -->|fail-fast| FAIL
+    STRATEGY -->|continue-on-error| SKIP[Skip Item<br/>Continue]
+    SKIP --> CONTINUE
+    style ERROR fill:#ffebee
+    style FAIL fill:#ff5252
+    style CONTINUE fill:#e8f5e9
+    style BACKOFF fill:#fff3e0
+```
+## Performance Optimizations
+### Streaming Pipeline
+```mermaid
+flowchart LR
+    FILE[Large File<br/>1GB] --> STREAM1[Stream Reader]
+    STREAM1 --> CHUNK1[Chunk 1]
+    STREAM1 --> CHUNK2[Chunk 2]
+    STREAM1 --> CHUNK3[Chunk N]
+    CHUNK1 --> PARSE1[Parse]
+    CHUNK2 --> PARSE2[Parse]
+    CHUNK3 --> PARSE3[Parse]
+    PARSE1 --> MAP1[Map]
+    PARSE2 --> MAP2[Map]
+    PARSE3 --> MAP3[Map]
+    MAP1 --> BATCH1[Batch 1]
+    MAP2 --> BATCH2[Batch 2]
+    MAP3 --> BATCH3[Batch N]
+    BATCH1 --> API[Fluent API]
+    BATCH2 --> API
+    BATCH3 --> API
+    style FILE fill:#ffebee
+    style STREAM1 fill:#e3f2fd
+    style API fill:#e8f5e9
+```
+**Benefits:**
+- Constant memory usage regardless of file size
+- Parallel processing of chunks
+- Early error detection
+- Faster overall throughput
+### Parallel Operations
+```mermaid
+graph TB
+    START[Multiple Files] --> SPLIT[Split into Batches]
+    SPLIT --> P1[Process File 1]
+    SPLIT --> P2[Process File 2]
+    SPLIT --> P3[Process File 3]
+    SPLIT --> PN[Process File N]
+    P1 --> WAIT[Wait All Complete]
+    P2 --> WAIT
+    P3 --> WAIT
+    PN --> WAIT
+    WAIT --> AGGREGATE[Aggregate Results]
+    AGGREGATE --> END[Complete]
+    style START fill:#e3f2fd
+    style SPLIT fill:#fff3e0
+    style WAIT fill:#f3e5f5
+    style END fill:#e8f5e9
+```
+## Next Steps
+- [Service Layer Architecture](./architecture-04-service-layer.md) - Explore service organization
+- [Integration Architecture](./architecture-05-integration-architecture.md) - External system integrations
+- [State Management](./architecture-06-state-management.md) - Distributed state patterns
+- [Universal Mapping Guide](../../02-CORE-GUIDES/mapping/mapping-readme.md) - Field mapping details