@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/00-START-HERE/sdk-philosophy.md

@@ -1,794 +1,794 @@
-# SDK Philosophy & Architecture
-
-## Table of Contents
-
-1. [Core Principle](#core-principle)
-2. [What the SDK Provides](#what-the-sdk-provides)
-3. [What YOU Build](#what-you-build)
-4. [Platform Support](#platform-support)
-5. [Common Patterns](#common-patterns)
-6. [Special Case: Inventory Ingestion](#special-case-inventory-ingestion)
-7. [SDK Architecture Diagram](#sdk-architecture-diagram)
-8. [Guidelines](#guidelines)
-9. [Decision Guide](#decision-guide)
-10. [Documentation](#documentation)
-
----
-
-## Core Principle
-
-> **The SDK provides services with methods. YOU build the workflow.**
-
-The SDK is **NOT** an opinionated framework. It's a **toolkit of composable services** that you combine however you want.
-
-**Key Philosophy:**
-
-- ✅ Provide building blocks (services, parsers, mappers)
-- ✅ Let developers compose them into custom workflows
-- ✅ Support multiple runtimes (Node.js, Deno, Versori)
-- ✅ No imposed patterns or opinions
-
----
-
-## What the SDK Provides
-
-### 1. Core Services (Building Blocks)
-
-```typescript
-// ============================================
-// CLIENT & AUTHENTICATION
-// ============================================
-
-createClient(); // Universal client factory (auto-detects context)
-FluentClient; // Standalone Node.js/Deno client (OAuth2)
-FluentVersoriClient; // Versori platform client (connection-based)
-
-// ============================================
-// DATA SOURCES
-// ============================================
-
-S3DataSource; // Read/write S3 (streaming support)
-SftpDataSource; // SFTP file operations
-InventoryDataSource; // Base class for custom sources
-
-// ============================================
-// PARSERS
-// ============================================
-
-CSVParserService; // CSV with validation
-JSONParserService; // JSON with streaming
-XMLParserService; // XML with XPath-like selectors
-ParquetParserService; // Parquet columnar format
-
-// ============================================
-// TRANSFORMATION & MAPPING
-// ============================================
-
-UniversalMapper; // Field mapping for ALL transformations
-GraphQLMutationMapper; // XML/JSON → GraphQL mutations (schema-driven)
-
-// ============================================
-// ORCHESTRATORS (Optional)
-// ============================================
-
-ExtractionOrchestrator; // High-level extraction with auto-pagination
-
-// ============================================
-// BATCH & JOB MANAGEMENT
-// ============================================
-
-FluentBatchManager; // Batch operation orchestration
-PartialBatchRecovery; // Handle partial batch failures gracefully
-JobTracker; // Job lifecycle and state tracking
-
-// ============================================
-// VALIDATION SERVICES
-// ============================================
-
-PreflightValidator; // Pre-execution validation for connectors
-SchemaValidationService; // Dynamic schema validation
-WebhookValidationService; // Webhook signature validation
-
-// ============================================
-// STATE & UTILITIES
-// ============================================
-
-StateService; // Track processed files, prevent duplicates
-createConsoleLogger; // Simple console logger
-toStructuredLogger; // Add context to logger
-createWorkflowLogger; // Pre-configured workflow logger
-createServiceLogger; // Pre-configured service logger
-VersoriKVAdapter; // Versori key-value storage adapter
-S3PresignService; // S3 presigned URL generation
-```
-
-### 2. Key Methods on Each Service
-
-```typescript
-// Client operations
-client.graphql({ query, variables }); // ONE method for BOTH queries and mutations
-client.sendEvent(event, mode);
-client.createJob(jobConfig);
-client.sendBatch(jobId, batchData);
-client.getBatchStatus(jobId, batchId);
-client.validateWebhook(payload, options); // Webhook signature validation
-
-// Data sources
-dataSource.listFiles({ prefix });
-dataSource.downloadFile(key);
-dataSource.uploadFile(key);
-dataSource.streamFile(key);
-
-// Parsers
-parser.parse(content);
-parser.parseStream(stream, options);
-
-// Mapping
-mapper.map(sourceData); // Returns { success, data, errors }
-
-// State management
-stateService.isFileProcessed(key);
-stateService.markFileProcessed(key, metadata);
-stateService.getCheckpoint(key);
-stateService.setCheckpoint(key, value);
-```
-
-**That's it. The SDK provides services with methods. YOU decide how to use them.**
-
----
-
-## What YOU Build
-
-### Your Application/Connector Logic
-
-You control:
-
-- ✅ **Which services to use** (S3 or SFTP? CSV or Parquet?)
-- ✅ **Execution order** (parse first or download first?)
-- ✅ **API choice** (GraphQL mutations, Batch API, or Events?)
-- ✅ **Error handling** (retry? skip? log?)
-- ✅ **Business rules** (filter records? validate data?)
-- ✅ **File management** (archive? delete? rename?)
-
-### Example: Custom Workflow
-
-```typescript
-import {
-  createClient,
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  StateService,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function myCustomWorkflow() {
-  // 1. Initialize SDK services
-  const client = await createClient({
-    config: {
-      baseUrl: 'https://api.fluentcommerce.com',
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: '1',
-    },
-  });
-
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_CSV',
-      s3Config: {
-        bucket: 'my-inventory-bucket',
-        region: 'us-east-1',
-        accessKeyId,
-        secretAccessKey,
-      },
-    },
-    logger
-  );
-
-  const parser = new CSVParserService();
-
-  const mapper = new UniversalMapper({
-    fields: {
-      ref: { source: 'product_id', required: true },
-      type: { source: 'product_type', resolver: 'sdk.uppercase' },
-      qty: { source: 'quantity', resolver: 'sdk.parseInt' },
-    },
-  });
-
-  // 2. YOUR CUSTOM LOGIC: List and filter files
-  const files = await s3.listFiles('inbound/');
-  const csvFiles = files.filter(f => f.name.endsWith('.csv'));
-
-  // 3. YOUR CUSTOM LOGIC: Process each file
-  for (const file of csvFiles) {
-    const content = await s3.downloadFile(file.name);
-    const records = await parser.parse(content);
-
-    // 4. YOUR CUSTOM LOGIC: Transform and validate
-    const validRecords = [];
-    for (const record of records) {
-      const result = await mapper.map(record);
-
-      if (result.success) {
-        validRecords.push(result.data);
-      } else {
-        logger.warn(`Skipping invalid record: ${result.errors}`);
-      }
-    }
-
-    // 5. YOUR CUSTOM LOGIC: Send to Fluent Commerce
-    for (const record of validRecords) {
-      await client.graphql({
-        query: `
-          mutation CreateProduct($input: CreateProductInput!) {
-            createProduct(input: $input) {
-              id
-              ref
-            }
-          }
-        `,
-        variables: { input: record },
-      });
-    }
-
-    // 6. YOUR CUSTOM LOGIC: Archive processed file
-    await s3.moveFile(file.name, `archive/${file.name}`);
-  }
-}
-```
-
----
-
-## Platform Support
-
-### Versori Platform (Primary)
-
-The SDK is **optimized for Versori** - a serverless integration platform.
-
-```typescript
-// Versori webhook (automatic context detection)
-import { webhook } from '@versori/run';
-import { createClient, S3DataSource, CSVParserService } from '@fluentcommerce/fc-connect-sdk';
-
-export const inventoryIngestion = webhook(
-  's3-event',
-  async (ctx) => {
-    const { log } = ctx;
-    // SDK automatically detects Versori context
-    const client = await createClient(ctx);
-
-    const s3 = new S3DataSource({ bucket: 'inventory' }, log);
-    const parser = new CSVParserService();
-
-    // Your workflow logic here...
-
-    return { success: true };
-  }
-);
-```
-
-**Versori Benefits:**
-
-- ✅ No server management
-- ✅ Built-in OAuth2 connection management
-- ✅ Key-value storage for state
-- ✅ Automatic scaling
-- ✅ Webhook triggers
-
-### Node.js / Deno (Standalone)
-
-The SDK also works standalone in Node.js or Deno:
-
-```typescript
-// Node.js or Deno
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-
-const client = await createClient({
-  config: {
-    baseUrl: 'https://api.fluentcommerce.com',
-    clientId: process.env.FLUENT_CLIENT_ID!,
-    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-    retailerId: process.env.FLUENT_RETAILER_ID,
-  },
-});
-```
-
-**Cross-Runtime Support:**
-
-- ✅ Node.js ≥18.0.0
-- ✅ Deno ≥1.30.0
-- ✅ Versori platform
-
----
-
-## Common Patterns
-
-### Pattern 1: CSV to GraphQL Mutations
-
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  createClient,
-} from '@fluentcommerce/fc-connect-sdk';
-
-// Read CSV from S3
-const s3 = new S3DataSource({ bucket: 'products' }, logger);
-const parser = new CSVParserService();
-
-const content = await s3.downloadFile('products.csv');
-const records = await parser.parse(content);
-
-// Map CSV fields to GraphQL input
-const mapper = new UniversalMapper({
-  fields: {
-    ref: { source: 'sku' },
-    name: { source: 'product_name' },
-    price: { source: 'price', resolver: 'sdk.parseFloat' },
-  },
-});
-
-// Send to Fluent Commerce
-const client = await createClient({ config });
-
-for (const record of records) {
-  const result = await mapper.map(record);
-  if (result.success) {
-    await client.graphql({
-      query: `
-        mutation CreateProduct($input: CreateProductInput!) {
-          createProduct(input: $input) { id }
-        }
-      `,
-      variables: { input: result.data },
-    });
-  }
-}
-```
-
-### Pattern 2: XML to GraphQL (SFCC Orders)
-
-```typescript
-import {
-  XMLParserService,
-  GraphQLMutationMapper,
-  createClient,
-} from '@fluentcommerce/fc-connect-sdk';
-
-const xmlParser = new XMLParserService();
-const orderXml = await fetchSFCCOrder();
-
-// Parse XML
-const orderData = await xmlParser.parse(orderXml);
-
-// Map XML → GraphQL using schema-driven mapping
-const mapper = new GraphQLMutationMapper(
-  {
-    mutation: {
-      name: 'createOrder',
-      operation: `mutation CreateOrder($input: CreateOrderInput!) { ... }`,
-    },
-    fields: {
-      'input.ref': { source: 'order.@order-no' },
-      'input.type': { source: 'order.@type' },
-      'input.customer.firstName': { source: 'order.customer.customer-name' },
-    },
-  },
-  client,
-  logger
-);
-
-const result = await mapper.map(orderData);
-// Automatically executes GraphQL mutation
-```
-
-### Pattern 3: State Management (Prevent Duplicates)
-
-```typescript
-import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
-
-// In Versori
-const kvAdapter = new VersoriKVAdapter(openKv());
-const stateService = new StateService(kvAdapter);
-
-// Check if file already processed
-const fileKey = 'inventory-2024-01-15.csv';
-const processed = await stateService.isFileProcessed(fileKey);
-
-if (!processed) {
-  // Process file...
-  await processFile(fileKey);
-
-  // Mark as processed
-  await stateService.markFileProcessed(fileKey, {
-    processedAt: new Date().toISOString(),
-    recordCount: 1000,
-  });
-}
-```
-
-### Pattern 4: Parquet to Batch API (High-Volume Inventory)
-
-```typescript
-import { S3DataSource, ParquetParserService, createClient } from '@fluentcommerce/fc-connect-sdk';
-
-const s3 = new S3DataSource({ bucket: 'inventory' }, logger);
-const parser = new ParquetParserService(logger);
-const client = await createClient({ config });
-
-// Read Parquet (compressed columnar format)
-const buffer = await s3.downloadFile('inventory.parquet');
-const records = await parser.parse(buffer);
-
-// Create Batch API job (INVENTORY only)
-const job = await client.createJob({
-  name: 'Inventory Update',
-  retailerId: '1',
-});
-
-// Send in batches of 1000
-const batches = chunkArray(records, 1000);
-for (const batch of batches) {
-  await client.sendBatch(job.id, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    entities: batch,
-  });
-}
-```
-
----
-
-## Special Case: Inventory Ingestion
-
-### Why Inventory is Different
-
-**Fluent Commerce has a specialized Batch API for inventory:**
-
-- ⚡ High-volume optimization (millions of records)
-- 📦 Job + Batches pattern
-- ⚠️ **ONLY works for INVENTORY entity**
-- 🚫 NOT for products, orders, locations, etc.
-
-### Use Building Blocks (Full Control)
-
-```typescript
-import {
-  S3DataSource,
-  CSVParserService,
-  UniversalMapper,
-  createClient,
-} from '@fluentcommerce/fc-connect-sdk';
-
-const s3 = new S3DataSource(config, logger);
-const parser = new CSVParserService();
-const mapper = new UniversalMapper(mappingConfig);
-const client = await createClient({ config: clientConfig });
-
-const files = await s3.listFiles('inventory/');
-
-for (const file of files) {
-  const content = await s3.downloadFile(file.name);
-  const records = await parser.parse(content);
-
-  // Map records
-  const inventory = [];
-  for (const record of records) {
-    const result = await mapper.map(record);
-    if (result.success) inventory.push(result.data);
-  }
-
-  // Create job (INVENTORY only)
-  const job = await client.createJob({
-    name: `Inventory - ${file.name}`,
-    retailerId: '1',
-    entity: 'INVENTORY',
-  });
-
-  // Send batches
-  const batches = chunkArray(inventory, 1000);
-  for (const batch of batches) {
-    await client.sendBatch(job.id, {
-      action: 'UPSERT',
-      entityType: 'INVENTORY',
-      entities: batch,
-    });
-  }
-
-  await s3.moveFile(file.name, `archive/${file.name}`);
-}
-```
-
-### For ALL Entities
-
-**All entities (Products, Locations, Orders, Customers, Fulfillments, Inventory, etc.) can use:**
-
-**GraphQL Mutations** (recommended for flexibility):
-```typescript
-for (const entity of entities) {
-  await client.graphql({
-    query: `
-      mutation CreateEntity($input: CreateEntityInput!) {
-        createEntity(input: $input) {
-          id
-          ref
-        }
-      }
-    `,
-    variables: { input: entity },
-  });
-}
-```
-
-**Batch API** (for high-volume inventory only):
-```typescript
-// Create job
-const job = await client.createJob({
-  name: 'Inventory Update',
-  retailerId: '1',
-});
-
-// Send in batches
-const batches = chunkArray(records, 1000);
-for (const batch of batches) {
-  await client.sendBatch(job.id, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    entities: batch,
-  });
-}
-```
-
----
-
-## SDK Architecture Diagram
-
-### High-Level Architecture
-
-```mermaid
-graph TB
-    subgraph app["YOUR APPLICATION<br/>(Custom Workflow Logic)"]
-        workflow["When to run: webhook/cron/manual<br/>What to process: files/entities<br/>How to execute: GraphQL/Batch/Events<br/>Error handling: retry/skip/alert<br/>Business rules: validation/filtering"]
-    end
-
-    subgraph sdk["SDK BUILDING BLOCKS<br/>(Services with Methods)"]
-        direction TB
-
-        subgraph data["Data Layer"]
-            sources["Data Sources<br/>• S3DataSource<br/>• SftpDataSource<br/>• InventoryDataSource"]
-            parsers["Parsers<br/>• CSVParserService<br/>• JSONParserService<br/>• XMLParserService<br/>• ParquetParserService"]
-        end
-
-        subgraph transform["Transformation Layer"]
-            mapper["UniversalMapper<br/>(Field Mapping)"]
-            gqlmapper["GraphQLMutationMapper<br/>(XML/JSON → GraphQL)"]
-        end
-
-        subgraph client["API Client Layer"]
-            api["FluentClient<br/>• graphql()<br/>• sendEvent()<br/>• createJob()<br/>• sendBatch()"]
-        end
-
-        subgraph support["Support Services"]
-            state["StateService<br/>(Prevent duplicates)"]
-            logging["createConsoleLogger<br/>toStructuredLogger<br/>createWorkflowLogger<br/>(Function-based logging)"]
-        end
-
-        subgraph orch["OPTIONAL: Orchestrators"]
-            exorch["ExtractionOrchestrator<br/>(Extraction with auto-pagination)"]
-        end
-    end
-
-    subgraph platforms["PLATFORM SUPPORT"]
-        versori["Versori<br/>(Primary)"]
-        nodejs["Node.js<br/>≥18.0.0"]
-        deno["Deno<br/>≥1.30.0"]
-    end
-
-    workflow --> sdk
-    sdk --> platforms
-
-    style app fill:#e1f5ff
-    style sdk fill:#fff4e6
-    style platforms fill:#f0f0f0
-    style orch fill:#ffe6e6
-```
-
-### Data Flow Patterns
-
-```mermaid
-graph LR
-    subgraph pattern1["Pattern 1: CSV to GraphQL"]
-        s3_1["S3 CSV File"]
-        parse_1["CSVParserService"]
-        map_1["UniversalMapper"]
-        gql_1["client.graphql()"]
-        fluent_1["Fluent Commerce"]
-
-        s3_1 --> parse_1
-        parse_1 --> map_1
-        map_1 --> gql_1
-        gql_1 --> fluent_1
-    end
-
-    subgraph pattern2["Pattern 2: XML to GraphQL (SFCC)"]
-        xml_2["SFCC XML"]
-        parse_2["XMLParserService"]
-        map_2["GraphQLMutationMapper"]
-        auto_2["Auto-executes<br/>mutation"]
-        fluent_2["Fluent Commerce"]
-
-        xml_2 --> parse_2
-        parse_2 --> map_2
-        map_2 --> auto_2
-        auto_2 --> fluent_2
-    end
-
-    subgraph pattern3["Pattern 3: Inventory (Batch API)"]
-        parquet_3["Parquet File"]
-        parse_3["ParquetParser"]
-        job_3["createJob()"]
-        batch_3["sendBatch()"]
-        fluent_3["Fluent Commerce<br/>(INVENTORY only)"]
-
-        parquet_3 --> parse_3
-        parse_3 --> job_3
-        job_3 --> batch_3
-        batch_3 --> fluent_3
-    end
-
-    style pattern1 fill:#e8f5e9
-    style pattern2 fill:#e3f2fd
-    style pattern3 fill:#fff3e0
-```
-
----
-
-## Guidelines
-
-### ✅ SDK Provides
-
-- **Services**: Classes with methods (composable building blocks)
-- **Methods**: Async functions returning Promises
-- **Types**: Complete TypeScript type definitions
-- **Utilities**: Helper functions and adapters
-- **Platform adapters**: Versori KV, S3 presign, etc.
-- **Cross-runtime support**: Node.js, Deno, Versori
-
-### ❌ SDK Does NOT Provide
-
-- **Workflow orchestration** (except optional inventory orchestrators)
-- **Opinionated patterns** (you decide the flow)
-- **Business logic** (you implement validation, filtering, etc.)
-- **Execution schedules** (use Versori webhooks, cron, etc.)
-- **UI/Dashboard** (use Versori platform or build your own)
-
----
-
-## Decision Guide
-
-```mermaid
-graph TD
-    start["What are you building?"]
-    approach["Choose SDK Approach"]
-
-    start --> approach
-
-    approach --> building_blocks["✅ Use Building Blocks<br/>(Recommended)"]
-
-    building_blocks --> bb_benefits["Benefits:<br/>• Full control<br/>• Better testability<br/>• Easier debugging<br/>• Complex workflows<br/>• Multi-system integration<br/>• Works for ALL entities"]
-
-    style building_blocks fill:#c8e6c9,stroke:#4caf50,stroke-width:3px
-```
-
-### Use Building Blocks (Recommended):
-
-- ✅ Works for **ALL** entities (Inventory, Products, Orders, Locations, etc.)
-- ✅ Custom workflow logic
-- ✅ Multi-step processes (validate → transform → enrich → send)
-- ✅ Integration with multiple systems
-- ✅ Production connectors (recommended for maintainability)
-- ✅ Full control over error handling and retries
-- ✅ Complex business rules or data validation
-- ✅ Better testability and debugging
-
-### Recommendation:
-
-**Always use building blocks for production connectors.** They provide:
-
-- Better testability
-- Easier debugging
-- Full control over execution
-- Better error handling
-- Easier to maintain and extend
-- Works consistently across all entity types
-
----
-
-## Documentation
-
-The SDK documentation follows the same philosophy - **modular building blocks you can navigate however you want**.
-
-### Modular Documentation Structure
-
-```
-docs/
-├── 00-START-HERE/         # Essential docs - read first
-├── 01-TEMPLATES/          # Complete working examples
-├── 02-CORE-GUIDES/        # Progressive learning paths
-│   ├── api-reference/     # Complete API documentation (13 modules)
-│   ├── ingestion/         # Data into Fluent
-│   ├── extraction/        # Data from Fluent
-│   ├── auto-pagination/   # Cursor pagination guide
-│   ├── data-sources/      # S3, SFTP guides
-│   ├── parsers/           # CSV, JSON, XML, Parquet
-│   ├── webhook-validation/# Webhook security
-│   └── mapping/           # Field transformation
-├── 03-PATTERN-GUIDES/     # Reusable patterns
-└── 04-REFERENCE/          # Deep dives & platform docs
-    ├── architecture/      # System design
-    ├── testing/           # Test utilities
-    ├── platforms/versori/ # Versori integration
-    └── resolver-signature/# Resolver details
-```
-
-### Navigate by Your Needs
-
-**Learning Path** (Beginner → Advanced):
-
-1. [Client API](../02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md) - Create clients, basic operations
-2. [Parsers](../02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md) - CSV, JSON, XML, Parquet
-3. [Data Sources](../02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md) - S3, SFTP integration
-4. [Services](../02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md) - UniversalMapper, StateService
-5. [Extraction](../02-CORE-GUIDES/extraction/extraction-readme.md) - ExtractionOrchestrator for data export
-
-**Quick Reference**: [One-page API cheat sheet](../02-CORE-GUIDES/api-reference/api-reference-quick-reference.md)
-
-**By Use Case**: See [API Reference](../02-CORE-GUIDES/api-reference/api-reference-readme.md) or [Getting Started](./getting-started.md)
-
-**Just like the SDK provides services you compose, the documentation provides modules you navigate however you need!**
-
----
-
-## Summary
-
-**The SDK Philosophy in 3 Points:**
-
-1. **Provide Building Blocks** - Services with methods you can compose
-2. **You Control the Workflow** - Decide how to use them
-3. **Platform Flexibility** - Works on Versori, Node.js, and Deno
-
-**Example:**
-
-```typescript
-// SDK gives you these tools:
-const client = await createClient({ config });
-const s3 = new S3DataSource({ bucket: 'data' }, logger);
-const parser = new CSVParserService();
-const mapper = new UniversalMapper(mappingConfig);
-
-// YOU build the workflow:
-async function myWorkflow() {
-  const files = await s3.listFiles();           // SDK method
-  const content = await s3.downloadFile(file);  // SDK method
-  const records = await parser.parse(content);  // SDK method
-
-  // Map records
-  const mapped = [];
-  for (const record of records) {
-    const result = await mapper.map(record);    // SDK method
-    if (result.success) mapped.push(result.data);
-  }
-
-  // YOUR CUSTOM LOGIC HERE
-  for (const record of mapped) {
-    await client.graphql({ query: /* ... */, variables: /* ... */ });  // SDK method
-    // or Batch API, or Events, or external API, or Database, or Whatever!
-  }
-
-  await s3.moveFile(file, `archive/${file}`);  // SDK method
-}
-```
-
-**As long as the SDK provides the services with methods, developers can build anything!**
+# SDK Philosophy & Architecture
+
+## Table of Contents
+
+1. [Core Principle](#core-principle)
+2. [What the SDK Provides](#what-the-sdk-provides)
+3. [What YOU Build](#what-you-build)
+4. [Platform Support](#platform-support)
+5. [Common Patterns](#common-patterns)
+6. [Special Case: Inventory Ingestion](#special-case-inventory-ingestion)
+7. [SDK Architecture Diagram](#sdk-architecture-diagram)
+8. [Guidelines](#guidelines)
+9. [Decision Guide](#decision-guide)
+10. [Documentation](#documentation)
+
+---
+
+## Core Principle
+
+> **The SDK provides services with methods. YOU build the workflow.**
+
+The SDK is **NOT** an opinionated framework. It's a **toolkit of composable services** that you combine however you want.
+
+**Key Philosophy:**
+
+- ✅ Provide building blocks (services, parsers, mappers)
+- ✅ Let developers compose them into custom workflows
+- ✅ Support multiple runtimes (Node.js, Deno, Versori)
+- ✅ No imposed patterns or opinions
+
+---
+
+## What the SDK Provides
+
+### 1. Core Services (Building Blocks)
+
+```typescript
+// ============================================
+// CLIENT & AUTHENTICATION
+// ============================================
+
+createClient(); // Universal client factory (auto-detects context)
+FluentClient; // Standalone Node.js/Deno client (OAuth2)
+FluentVersoriClient; // Versori platform client (connection-based)
+
+// ============================================
+// DATA SOURCES
+// ============================================
+
+S3DataSource; // Read/write S3 (streaming support)
+SftpDataSource; // SFTP file operations
+InventoryDataSource; // Base class for custom sources
+
+// ============================================
+// PARSERS
+// ============================================
+
+CSVParserService; // CSV with validation
+JSONParserService; // JSON with streaming
+XMLParserService; // XML with XPath-like selectors
+ParquetParserService; // Parquet columnar format
+
+// ============================================
+// TRANSFORMATION & MAPPING
+// ============================================
+
+UniversalMapper; // Field mapping for ALL transformations
+GraphQLMutationMapper; // XML/JSON → GraphQL mutations (schema-driven)
+
+// ============================================
+// ORCHESTRATORS (Optional)
+// ============================================
+
+ExtractionOrchestrator; // High-level extraction with auto-pagination
+
+// ============================================
+// BATCH & JOB MANAGEMENT
+// ============================================
+
+FluentBatchManager; // Batch operation orchestration
+PartialBatchRecovery; // Handle partial batch failures gracefully
+JobTracker; // Job lifecycle and state tracking
+
+// ============================================
+// VALIDATION SERVICES
+// ============================================
+
+PreflightValidator; // Pre-execution validation for connectors
+SchemaValidationService; // Dynamic schema validation
+WebhookValidationService; // Webhook signature validation
+
+// ============================================
+// STATE & UTILITIES
+// ============================================
+
+StateService; // Track processed files, prevent duplicates
+createConsoleLogger; // Simple console logger
+toStructuredLogger; // Add context to logger
+createWorkflowLogger; // Pre-configured workflow logger
+createServiceLogger; // Pre-configured service logger
+VersoriKVAdapter; // Versori key-value storage adapter
+S3PresignService; // S3 presigned URL generation
+```
+
+### 2. Key Methods on Each Service
+
+```typescript
+// Client operations
+client.graphql({ query, variables }); // ONE method for BOTH queries and mutations
+client.sendEvent(event, mode);
+client.createJob(jobConfig);
+client.sendBatch(jobId, batchData);
+client.getBatchStatus(jobId, batchId);
+client.validateWebhook(payload, options); // Webhook signature validation
+
+// Data sources
+dataSource.listFiles({ prefix });
+dataSource.downloadFile(key);
+dataSource.uploadFile(key);
+dataSource.streamFile(key);
+
+// Parsers
+parser.parse(content);
+parser.parseStream(stream, options);
+
+// Mapping
+mapper.map(sourceData); // Returns { success, data, errors }
+
+// State management
+stateService.isFileProcessed(key);
+stateService.markFileProcessed(key, metadata);
+stateService.getCheckpoint(key);
+stateService.setCheckpoint(key, value);
+```
+
+**That's it. The SDK provides services with methods. YOU decide how to use them.**
+
+---
+
+## What YOU Build
+
+### Your Application/Connector Logic
+
+You control:
+
+- ✅ **Which services to use** (S3 or SFTP? CSV or Parquet?)
+- ✅ **Execution order** (parse first or download first?)
+- ✅ **API choice** (GraphQL mutations, Batch API, or Events?)
+- ✅ **Error handling** (retry? skip? log?)
+- ✅ **Business rules** (filter records? validate data?)
+- ✅ **File management** (archive? delete? rename?)
+
+### Example: Custom Workflow
+
+```typescript
+import {
+  createClient,
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  StateService,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function myCustomWorkflow() {
+  // 1. Initialize SDK services
+  const client = await createClient({
+    config: {
+      baseUrl: 'https://api.fluentcommerce.com',
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: '1',
+    },
+  });
+
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_CSV',
+      s3Config: {
+        bucket: 'my-inventory-bucket',
+        region: 'us-east-1',
+        accessKeyId,
+        secretAccessKey,
+      },
+    },
+    logger
+  );
+
+  const parser = new CSVParserService();
+
+  const mapper = new UniversalMapper({
+    fields: {
+      ref: { source: 'product_id', required: true },
+      type: { source: 'product_type', resolver: 'sdk.uppercase' },
+      qty: { source: 'quantity', resolver: 'sdk.parseInt' },
+    },
+  });
+
+  // 2. YOUR CUSTOM LOGIC: List and filter files
+  const files = await s3.listFiles('inbound/');
+  const csvFiles = files.filter(f => f.name.endsWith('.csv'));
+
+  // 3. YOUR CUSTOM LOGIC: Process each file
+  for (const file of csvFiles) {
+    const content = await s3.downloadFile(file.name);
+    const records = await parser.parse(content);
+
+    // 4. YOUR CUSTOM LOGIC: Transform and validate
+    const validRecords = [];
+    for (const record of records) {
+      const result = await mapper.map(record);
+
+      if (result.success) {
+        validRecords.push(result.data);
+      } else {
+        logger.warn(`Skipping invalid record: ${result.errors}`);
+      }
+    }
+
+    // 5. YOUR CUSTOM LOGIC: Send to Fluent Commerce
+    for (const record of validRecords) {
+      await client.graphql({
+        query: `
+          mutation CreateProduct($input: CreateProductInput!) {
+            createProduct(input: $input) {
+              id
+              ref
+            }
+          }
+        `,
+        variables: { input: record },
+      });
+    }
+
+    // 6. YOUR CUSTOM LOGIC: Archive processed file
+    await s3.moveFile(file.name, `archive/${file.name}`);
+  }
+}
+```
+
+---
+
+## Platform Support
+
+### Versori Platform (Primary)
+
+The SDK is **optimized for Versori** - a serverless integration platform.
+
+```typescript
+// Versori webhook (automatic context detection)
+import { webhook } from '@versori/run';
+import { createClient, S3DataSource, CSVParserService } from '@fluentcommerce/fc-connect-sdk';
+
+export const inventoryIngestion = webhook(
+  's3-event',
+  async (ctx) => {
+    const { log } = ctx;
+    // SDK automatically detects Versori context
+    const client = await createClient(ctx);
+
+    const s3 = new S3DataSource({ bucket: 'inventory' }, log);
+    const parser = new CSVParserService();
+
+    // Your workflow logic here...
+
+    return { success: true };
+  }
+);
+```
+
+**Versori Benefits:**
+
+- ✅ No server management
+- ✅ Built-in OAuth2 connection management
+- ✅ Key-value storage for state
+- ✅ Automatic scaling
+- ✅ Webhook triggers
+
+### Node.js / Deno (Standalone)
+
+The SDK also works standalone in Node.js or Deno:
+
+```typescript
+// Node.js or Deno
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+
+const client = await createClient({
+  config: {
+    baseUrl: 'https://api.fluentcommerce.com',
+    clientId: process.env.FLUENT_CLIENT_ID!,
+    clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+    retailerId: process.env.FLUENT_RETAILER_ID,
+  },
+});
+```
+
+**Cross-Runtime Support:**
+
+- ✅ Node.js ≥18.0.0
+- ✅ Deno ≥1.30.0
+- ✅ Versori platform
+
+---
+
+## Common Patterns
+
+### Pattern 1: CSV to GraphQL Mutations
+
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  createClient,
+} from '@fluentcommerce/fc-connect-sdk';
+
+// Read CSV from S3
+const s3 = new S3DataSource({ bucket: 'products' }, logger);
+const parser = new CSVParserService();
+
+const content = await s3.downloadFile('products.csv');
+const records = await parser.parse(content);
+
+// Map CSV fields to GraphQL input
+const mapper = new UniversalMapper({
+  fields: {
+    ref: { source: 'sku' },
+    name: { source: 'product_name' },
+    price: { source: 'price', resolver: 'sdk.parseFloat' },
+  },
+});
+
+// Send to Fluent Commerce
+const client = await createClient({ config });
+
+for (const record of records) {
+  const result = await mapper.map(record);
+  if (result.success) {
+    await client.graphql({
+      query: `
+        mutation CreateProduct($input: CreateProductInput!) {
+          createProduct(input: $input) { id }
+        }
+      `,
+      variables: { input: result.data },
+    });
+  }
+}
+```
+
+### Pattern 2: XML to GraphQL (SFCC Orders)
+
+```typescript
+import {
+  XMLParserService,
+  GraphQLMutationMapper,
+  createClient,
+} from '@fluentcommerce/fc-connect-sdk';
+
+const xmlParser = new XMLParserService();
+const orderXml = await fetchSFCCOrder();
+
+// Parse XML
+const orderData = await xmlParser.parse(orderXml);
+
+// Map XML → GraphQL using schema-driven mapping
+const mapper = new GraphQLMutationMapper(
+  {
+    mutation: {
+      name: 'createOrder',
+      operation: `mutation CreateOrder($input: CreateOrderInput!) { ... }`,
+    },
+    fields: {
+      'input.ref': { source: 'order.@order-no' },
+      'input.type': { source: 'order.@type' },
+      'input.customer.firstName': { source: 'order.customer.customer-name' },
+    },
+  },
+  client,
+  logger
+);
+
+const result = await mapper.map(orderData);
+// Automatically executes GraphQL mutation
+```
+
+### Pattern 3: State Management (Prevent Duplicates)
+
+```typescript
+import { StateService, VersoriKVAdapter } from '@fluentcommerce/fc-connect-sdk';
+
+// In Versori
+const kvAdapter = new VersoriKVAdapter(openKv());
+const stateService = new StateService(kvAdapter);
+
+// Check if file already processed
+const fileKey = 'inventory-2024-01-15.csv';
+const processed = await stateService.isFileProcessed(fileKey);
+
+if (!processed) {
+  // Process file...
+  await processFile(fileKey);
+
+  // Mark as processed
+  await stateService.markFileProcessed(fileKey, {
+    processedAt: new Date().toISOString(),
+    recordCount: 1000,
+  });
+}
+```
+
+### Pattern 4: Parquet to Batch API (High-Volume Inventory)
+
+```typescript
+import { S3DataSource, ParquetParserService, createClient } from '@fluentcommerce/fc-connect-sdk';
+
+const s3 = new S3DataSource({ bucket: 'inventory' }, logger);
+const parser = new ParquetParserService(logger);
+const client = await createClient({ config });
+
+// Read Parquet (compressed columnar format)
+const buffer = await s3.downloadFile('inventory.parquet');
+const records = await parser.parse(buffer);
+
+// Create Batch API job (INVENTORY only)
+const job = await client.createJob({
+  name: 'Inventory Update',
+  retailerId: '1',
+});
+
+// Send in batches of 1000
+const batches = chunkArray(records, 1000);
+for (const batch of batches) {
+  await client.sendBatch(job.id, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    entities: batch,
+  });
+}
+```
+
+---
+
+## Special Case: Inventory Ingestion
+
+### Why Inventory is Different
+
+**Fluent Commerce has a specialized Batch API for inventory:**
+
+- ⚡ High-volume optimization (millions of records)
+- 📦 Job + Batches pattern
+- ⚠️ **ONLY works for INVENTORY entity**
+- 🚫 NOT for products, orders, locations, etc.
+
+### Use Building Blocks (Full Control)
+
+```typescript
+import {
+  S3DataSource,
+  CSVParserService,
+  UniversalMapper,
+  createClient,
+} from '@fluentcommerce/fc-connect-sdk';
+
+const s3 = new S3DataSource(config, logger);
+const parser = new CSVParserService();
+const mapper = new UniversalMapper(mappingConfig);
+const client = await createClient({ config: clientConfig });
+
+const files = await s3.listFiles('inventory/');
+
+for (const file of files) {
+  const content = await s3.downloadFile(file.name);
+  const records = await parser.parse(content);
+
+  // Map records
+  const inventory = [];
+  for (const record of records) {
+    const result = await mapper.map(record);
+    if (result.success) inventory.push(result.data);
+  }
+
+  // Create job (INVENTORY only)
+  const job = await client.createJob({
+    name: `Inventory - ${file.name}`,
+    retailerId: '1',
+    entity: 'INVENTORY',
+  });
+
+  // Send batches
+  const batches = chunkArray(inventory, 1000);
+  for (const batch of batches) {
+    await client.sendBatch(job.id, {
+      action: 'UPSERT',
+      entityType: 'INVENTORY',
+      entities: batch,
+    });
+  }
+
+  await s3.moveFile(file.name, `archive/${file.name}`);
+}
+```
+
+### For ALL Entities
+
+**All entities (Products, Locations, Orders, Customers, Fulfillments, Inventory, etc.) can use:**
+
+**GraphQL Mutations** (recommended for flexibility):
+```typescript
+for (const entity of entities) {
+  await client.graphql({
+    query: `
+      mutation CreateEntity($input: CreateEntityInput!) {
+        createEntity(input: $input) {
+          id
+          ref
+        }
+      }
+    `,
+    variables: { input: entity },
+  });
+}
+```
+
+**Batch API** (for high-volume inventory only):
+```typescript
+// Create job
+const job = await client.createJob({
+  name: 'Inventory Update',
+  retailerId: '1',
+});
+
+// Send in batches
+const batches = chunkArray(records, 1000);
+for (const batch of batches) {
+  await client.sendBatch(job.id, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    entities: batch,
+  });
+}
+```
+
+---
+
+## SDK Architecture Diagram
+
+### High-Level Architecture
+
+```mermaid
+graph TB
+    subgraph app["YOUR APPLICATION<br/>(Custom Workflow Logic)"]
+        workflow["When to run: webhook/cron/manual<br/>What to process: files/entities<br/>How to execute: GraphQL/Batch/Events<br/>Error handling: retry/skip/alert<br/>Business rules: validation/filtering"]
+    end
+
+    subgraph sdk["SDK BUILDING BLOCKS<br/>(Services with Methods)"]
+        direction TB
+
+        subgraph data["Data Layer"]
+            sources["Data Sources<br/>• S3DataSource<br/>• SftpDataSource<br/>• InventoryDataSource"]
+            parsers["Parsers<br/>• CSVParserService<br/>• JSONParserService<br/>• XMLParserService<br/>• ParquetParserService"]
+        end
+
+        subgraph transform["Transformation Layer"]
+            mapper["UniversalMapper<br/>(Field Mapping)"]
+            gqlmapper["GraphQLMutationMapper<br/>(XML/JSON → GraphQL)"]
+        end
+
+        subgraph client["API Client Layer"]
+            api["FluentClient<br/>• graphql()<br/>• sendEvent()<br/>• createJob()<br/>• sendBatch()"]
+        end
+
+        subgraph support["Support Services"]
+            state["StateService<br/>(Prevent duplicates)"]
+            logging["createConsoleLogger<br/>toStructuredLogger<br/>createWorkflowLogger<br/>(Function-based logging)"]
+        end
+
+        subgraph orch["OPTIONAL: Orchestrators"]
+            exorch["ExtractionOrchestrator<br/>(Extraction with auto-pagination)"]
+        end
+    end
+
+    subgraph platforms["PLATFORM SUPPORT"]
+        versori["Versori<br/>(Primary)"]
+        nodejs["Node.js<br/>≥18.0.0"]
+        deno["Deno<br/>≥1.30.0"]
+    end
+
+    workflow --> sdk
+    sdk --> platforms
+
+    style app fill:#e1f5ff
+    style sdk fill:#fff4e6
+    style platforms fill:#f0f0f0
+    style orch fill:#ffe6e6
+```
+
+### Data Flow Patterns
+
+```mermaid
+graph LR
+    subgraph pattern1["Pattern 1: CSV to GraphQL"]
+        s3_1["S3 CSV File"]
+        parse_1["CSVParserService"]
+        map_1["UniversalMapper"]
+        gql_1["client.graphql()"]
+        fluent_1["Fluent Commerce"]
+
+        s3_1 --> parse_1
+        parse_1 --> map_1
+        map_1 --> gql_1
+        gql_1 --> fluent_1
+    end
+
+    subgraph pattern2["Pattern 2: XML to GraphQL (SFCC)"]
+        xml_2["SFCC XML"]
+        parse_2["XMLParserService"]
+        map_2["GraphQLMutationMapper"]
+        auto_2["Auto-executes<br/>mutation"]
+        fluent_2["Fluent Commerce"]
+
+        xml_2 --> parse_2
+        parse_2 --> map_2
+        map_2 --> auto_2
+        auto_2 --> fluent_2
+    end
+
+    subgraph pattern3["Pattern 3: Inventory (Batch API)"]
+        parquet_3["Parquet File"]
+        parse_3["ParquetParser"]
+        job_3["createJob()"]
+        batch_3["sendBatch()"]
+        fluent_3["Fluent Commerce<br/>(INVENTORY only)"]
+
+        parquet_3 --> parse_3
+        parse_3 --> job_3
+        job_3 --> batch_3
+        batch_3 --> fluent_3
+    end
+
+    style pattern1 fill:#e8f5e9
+    style pattern2 fill:#e3f2fd
+    style pattern3 fill:#fff3e0
+```
+
+---
+
+## Guidelines
+
+### ✅ SDK Provides
+
+- **Services**: Classes with methods (composable building blocks)
+- **Methods**: Async functions returning Promises
+- **Types**: Complete TypeScript type definitions
+- **Utilities**: Helper functions and adapters
+- **Platform adapters**: Versori KV, S3 presign, etc.
+- **Cross-runtime support**: Node.js, Deno, Versori
+
+### ❌ SDK Does NOT Provide
+
+- **Workflow orchestration** (except optional inventory orchestrators)
+- **Opinionated patterns** (you decide the flow)
+- **Business logic** (you implement validation, filtering, etc.)
+- **Execution schedules** (use Versori webhooks, cron, etc.)
+- **UI/Dashboard** (use Versori platform or build your own)
+
+---
+
+## Decision Guide
+
+```mermaid
+graph TD
+    start["What are you building?"]
+    approach["Choose SDK Approach"]
+
+    start --> approach
+
+    approach --> building_blocks["✅ Use Building Blocks<br/>(Recommended)"]
+
+    building_blocks --> bb_benefits["Benefits:<br/>• Full control<br/>• Better testability<br/>• Easier debugging<br/>• Complex workflows<br/>• Multi-system integration<br/>• Works for ALL entities"]
+
+    style building_blocks fill:#c8e6c9,stroke:#4caf50,stroke-width:3px
+```
+
+### Use Building Blocks (Recommended):
+
+- ✅ Works for **ALL** entities (Inventory, Products, Orders, Locations, etc.)
+- ✅ Custom workflow logic
+- ✅ Multi-step processes (validate → transform → enrich → send)
+- ✅ Integration with multiple systems
+- ✅ Production connectors (recommended for maintainability)
+- ✅ Full control over error handling and retries
+- ✅ Complex business rules or data validation
+- ✅ Better testability and debugging
+
+### Recommendation:
+
+**Always use building blocks for production connectors.** They provide:
+
+- Better testability
+- Easier debugging
+- Full control over execution
+- Better error handling
+- Easier to maintain and extend
+- Works consistently across all entity types
+
+---
+
+## Documentation
+
+The SDK documentation follows the same philosophy - **modular building blocks you can navigate however you want**.
+
+### Modular Documentation Structure
+
+```
+docs/
+├── 00-START-HERE/         # Essential docs - read first
+├── 01-TEMPLATES/          # Complete working examples
+├── 02-CORE-GUIDES/        # Progressive learning paths
+│   ├── api-reference/     # Complete API documentation (13 modules)
+│   ├── ingestion/         # Data into Fluent
+│   ├── extraction/        # Data from Fluent
+│   ├── auto-pagination/   # Cursor pagination guide
+│   ├── data-sources/      # S3, SFTP guides
+│   ├── parsers/           # CSV, JSON, XML, Parquet
+│   ├── webhook-validation/# Webhook security
+│   └── mapping/           # Field transformation
+├── 03-PATTERN-GUIDES/     # Reusable patterns
+└── 04-REFERENCE/          # Deep dives & platform docs
+    ├── architecture/      # System design
+    ├── testing/           # Test utilities
+    ├── platforms/versori/ # Versori integration
+    └── resolver-signature/# Resolver details
+```
+
+### Navigate by Your Needs
+
+**Learning Path** (Beginner → Advanced):
+
+1. [Client API](../02-CORE-GUIDES/api-reference/modules/api-reference-01-client-api.md) - Create clients, basic operations
+2. [Parsers](../02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md) - CSV, JSON, XML, Parquet
+3. [Data Sources](../02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md) - S3, SFTP integration
+4. [Services](../02-CORE-GUIDES/api-reference/modules/api-reference-05-services.md) - UniversalMapper, StateService
+5. [Extraction](../02-CORE-GUIDES/extraction/extraction-readme.md) - ExtractionOrchestrator for data export
+
+**Quick Reference**: [One-page API cheat sheet](../02-CORE-GUIDES/api-reference/api-reference-quick-reference.md)
+
+**By Use Case**: See [API Reference](../02-CORE-GUIDES/api-reference/api-reference-readme.md) or [Getting Started](./getting-started.md)
+
+**Just like the SDK provides services you compose, the documentation provides modules you navigate however you need!**
+
+---
+
+## Summary
+
+**The SDK Philosophy in 3 Points:**
+
+1. **Provide Building Blocks** - Services with methods you can compose
+2. **You Control the Workflow** - Decide how to use them
+3. **Platform Flexibility** - Works on Versori, Node.js, and Deno
+
+**Example:**
+
+```typescript
+// SDK gives you these tools:
+const client = await createClient({ config });
+const s3 = new S3DataSource({ bucket: 'data' }, logger);
+const parser = new CSVParserService();
+const mapper = new UniversalMapper(mappingConfig);
+
+// YOU build the workflow:
+async function myWorkflow() {
+  const files = await s3.listFiles();           // SDK method
+  const content = await s3.downloadFile(file);  // SDK method
+  const records = await parser.parse(content);  // SDK method
+
+  // Map records
+  const mapped = [];
+  for (const record of records) {
+    const result = await mapper.map(record);    // SDK method
+    if (result.success) mapped.push(result.data);
+  }
+
+  // YOUR CUSTOM LOGIC HERE
+  for (const record of mapped) {
+    await client.graphql({ query: /* ... */, variables: /* ... */ });  // SDK method
+    // or Batch API, or Events, or external API, or Database, or Whatever!
+  }
+
+  await s3.moveFile(file, `archive/${file}`);  // SDK method
+}
+```
+
+**As long as the SDK provides the services with methods, developers can build anything!**