@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/02-CORE-GUIDES/auto-pagination/modules/auto-pagination-05-sdk-integration.md

@@ -1,847 +1,847 @@
-# Module 5: SDK Ecosystem Integration
-
-> **Learning Objective:** Integrate auto-pagination with other SDK services like UniversalMapper, ExtractionOrchestrator, S3DataSource, and Versori platform.
->
-> **Level:** Advanced
-
-## Table of Contents
-
-1. `Pagination + UniversalMapper (Transform Data)`
-2. `Pagination + ExtractionOrchestrator (Export to S3)`
-3. `Pagination + Batch Ingestion (Reverse Flow)`
-4. [Versori Platform Integration](#versori-platform-integration)
-5. [Complete ETL Pipelines](#complete-etl-pipelines)
-6. [Performance Optimization Patterns](#performance-optimization-patterns)
-
----
-
-## Pagination + UniversalMapper
-
-Transform paginated data using SDK's Universal Mapper.
-
-### Pattern: Fetch → Transform → Process
-
-```typescript
-import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-async function fetchAndTransformInventory() {
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-
-  // 1. Fetch data with auto-pagination
-  const result = await client.graphql({
-    query: `
-      query GetVirtualPositions($first: Int!, $after: String) {
-        virtualPositions(first: $first, after: $after) {
-          edges {
-            node {
-              id
-              productRef
-              quantity
-              groupRef
-              status
-              updatedOn
-            }
-            cursor
-          }
-          pageInfo { hasNextPage }
-        }
-      }
-    `,
-    variables: { first: 150 },
-    pagination: {
-      maxRecords: 10000,
-      onProgress: (page, records) => {
-        console.log(`📄 Fetched ${records} positions...`);
-      }
-    }
-  });
-
-  // 2. Define mapping configuration
-  const mappingConfig = {
-    fields: {
-      sku: { source: 'node.productRef', required: true },
-      qty: { source: 'node.quantity', resolver: 'sdk.parseInt' },
-      location: { source: 'node.groupRef', required: true },
-      status: { source: 'node.status', resolver: 'sdk.lowercase' },
-      lastUpdated: { source: 'node.updatedOn', resolver: 'sdk.formatDate' }
-    }
-  };
-
-  // 3. Transform data with UniversalMapper
-  const mapper = new UniversalMapper(mappingConfig);
-  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
-
-  if (!transformed.success) {
-    console.error('Mapping errors:', transformed.errors);
-    throw new Error('Data transformation failed');
-  }
-
-  // 4. Process transformed data
-  console.log(`✅ Transformed ${transformed.data.length} records`);
-
-  return transformed.data;
-  // [{ sku: 'SKU-001', qty: 150, location: 'WH-EAST', status: 'active', lastUpdated: '2024-01-15' }, ...]
-}
-```
-
-### Custom Resolvers with Pagination
-
-```typescript
-// Define custom transformation logic
-const mapper = new UniversalMapper(mappingConfig, {
-  customResolvers: {
-    'custom.calculateValue': (value, sourceData, helpers) => {
-      const qty = helpers.parseIntSafe(sourceData.node.quantity, 0);
-      const price = helpers.parseFloatSafe(sourceData.node.price, 0);
-      return qty * price;
-    },
-    'custom.determineAvailability': (value, sourceData) => {
-      return sourceData.node.quantity > 0 && sourceData.node.status === 'ACTIVE'
-        ? 'IN_STOCK'
-        : 'OUT_OF_STOCK';
-    }
-  }
-});
-
-const paginatedResult = await client.graphql({ query, variables, pagination });
-const transformed = await mapper.mapArray(paginatedResult.data.virtualPositions.edges);
-```
-
----
-
-## When to Use FluentClient.graphql() vs ExtractionOrchestrator
-
-**Decision Guide:**
-
-| Scenario | Use | Why |
-|----------|-----|-----|
-| **Query + Transform Only** | `FluentClient.graphql()` with `pagination` | Direct access, minimal overhead, full control |
-| **Query + Export to S3/SFTP** | `ExtractionOrchestrator` | Built-in export, file generation, retry logic |
-| **Custom Processing Logic** | `FluentClient.graphql()` with `pagination` | Maximum flexibility for your workflow |
-| **Standard ETL Pipelines** | `ExtractionOrchestrator` | Pre-built patterns, less code |
-| **Multiple Export Formats** | `ExtractionOrchestrator` | Supports JSON, XML, CSV, Parquet |
-| **Need Progress Tracking** | Both support via callbacks | Use whichever fits your workflow |
-
-### FluentClient.graphql() - Direct Approach
-
-**Best for:** Custom workflows, transformation-only, direct data access
-
-```typescript
-import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-const client = await createClient({ config: { /* ... */ } });
-
-// 1. Fetch with auto-pagination
-const result = await client.graphql({
-  query: `query GetVirtualPositions($first: Int!, $after: String) { ... }`,
-  variables: { first: 150 },
-  pagination: { maxRecords: 10000 }
-});
-
-// 2. Transform with UniversalMapper
-const mapper = new UniversalMapper({ fields: { /* ... */ } });
-const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
-
-// 3. Do whatever you want with the data
-await yourCustomLogic(transformed.data);
-```
-
-### ExtractionOrchestrator - High-Level Approach
-
-**Best for:** Standard exports to S3/SFTP, pre-built patterns
-
-```typescript
-import { ExtractionOrchestrator, S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-const orchestrator = new ExtractionOrchestrator(client, logger);
-
-// All-in-one: Query + Transform + Export
-const result = await orchestrator.extract({
-  query: `query GetVirtualPositions($first: Int!, $after: String) { ... }`,
-  extractionPath: 'virtualPositions.edges',
-  destination: s3DataSource,
-  format: 'json',
-  filename: 'inventory-export.json',
-  pagination: { maxRecords: 10000 }
-});
-
-// File is already on S3!
-```
-
-**Key Difference:** `ExtractionOrchestrator` = `FluentClient.graphql()` + file export + retry logic + pre-built patterns.
-
----
-
-## Pagination + ExtractionOrchestrator
-
-Export paginated data to S3 using ExtractionOrchestrator.
-
-### Pattern: Fetch → Transform → Export
-
-```typescript
-import {
-  createClient,
-  ExtractionOrchestrator,
-  S3DataSource,
-  UniversalMapper,
-  createConsoleLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function exportInventoryToS3() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-
-  // 1. Setup S3 destination
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_JSON',
-      connectionId: 's3-export',
-      name: 'S3 Export',
-      s3Config: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-        bucket: 'inventory-exports'
-      }
-    },
-    logger
-  );
-
-  // 2. Paginate inventory data
-  const result = await client.graphql({
-    query: `
-      query GetVirtualPositions($first: Int!, $after: String) {
-        virtualPositions(first: $first, after: $after) {
-          edges {
-            node {
-              id
-              productRef
-              quantity
-              groupRef
-              status
-              createdOn
-              updatedOn
-            }
-            cursor
-          }
-          pageInfo { hasNextPage }
-        }
-      }
-    `,
-    variables: { first: 150 },
-    pagination: {
-      maxRecords: 15000,
-      timeoutMs: 300000
-    }
-  });
-
-  console.log(`✅ Fetched ${result.extensions.autoPagination.totalRecords} positions`);
-
-  // 3. Transform data
-  const mapper = new UniversalMapper({
-    fields: {
-      sku: { source: 'node.productRef' },
-      quantity: { source: 'node.quantity', resolver: 'sdk.parseInt' },
-      warehouse: { source: 'node.groupRef' },
-      status: { source: 'node.status' },
-      lastModified: { source: 'node.updatedOn', resolver: 'sdk.formatDate' }
-    }
-  });
-
-  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
-
-  if (!transformed.success) {
-    throw new Error(`Transformation failed: ${transformed.errors.length} errors`);
-  }
-
-  // 4. Export to S3
-  const timestamp = new Date().toISOString().split('T')[0];
-  const filename = `inventory-export-${timestamp}.json`;
-
-  await s3.uploadFile(filename, JSON.stringify(transformed.data, null, 2));
-
-  console.log(`✅ Exported to s3://inventory-exports/${filename}`);
-
-  return {
-    filename,
-    recordCount: transformed.data.length,
-    stats: result.extensions.autoPagination
-  };
-}
-```
-
-### Chunked S3 Export (Large Datasets)
-
-For very large datasets, export in chunks to avoid memory issues:
-
-```typescript
-import { createClient, S3DataSource, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function exportInventoryInChunks() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_JSON',
-      connectionId: 's3-chunked-export',
-      name: 'S3 Chunked Export',
-      s3Config: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-        bucket: 'inventory-exports'
-      }
-    },
-    logger
-  );
-
-  let after: string | null = null;
-  let chunkNumber = 0;
-  const CHUNK_SIZE = 5000;
-
-  while (true) {
-    // Fetch chunk with pagination
-    const result = await client.graphql({
-      query: virtualPositionsQuery,
-      variables: { first: 150, after },
-      pagination: {
-        maxRecords: CHUNK_SIZE,
-        maxPages: 35  // ~35 pages * 150 = 5250 records
-      }
-    });
-
-    const edges = result.data.virtualPositions.edges;
-    if (edges.length === 0) break;
-
-    // Transform chunk
-    const mapper = new UniversalMapper(mappingConfig);
-    const transformed = await mapper.mapArray(edges);
-
-    // Export chunk to S3
-    const filename = `inventory-chunk-${chunkNumber.toString().padStart(4, '0')}.json`;
-    await s3.uploadFile(filename, JSON.stringify(transformed.data, null, 2));
-
-    console.log(`✅ Exported chunk ${chunkNumber}: ${edges.length} records`);
-
-    // Check if we got fewer records than max (means we're done)
-    if (result.extensions.autoPagination.truncated &&
-        result.extensions.autoPagination.truncationReason === 'maxRecords') {
-      after = edges[edges.length - 1].cursor;
-      chunkNumber++;
-    } else {
-      break;  // No more data
-    }
-  }
-
-  console.log(`✅ Export complete: ${chunkNumber + 1} chunks`);
-}
-```
-
----
-
-## Pagination + Batch Ingestion
-
-Reverse flow: Fetch from S3 → Paginate → Transform → Batch Ingest to Fluent.
-
-### Pattern: S3 → Transform → Batch API
-
-```typescript
-import {
-  createClient,
-  S3DataSource,
-  UniversalMapper,
-  createConsoleLogger
-} from '@fluentcommerce/fc-connect-sdk';
-
-async function ingestInventoryFromS3() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_JSON',
-      connectionId: 's3-ingestion',
-      name: 'S3 Ingestion',
-      s3Config: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-        bucket: 'inventory-updates'
-      }
-    },
-    logger
-  );
-
-  // 1. Read data from S3
-  const fileContent = await s3.downloadFile('inventory-updates.json');
-  const inventoryData = JSON.parse(fileContent);
-
-  console.log(`📥 Loaded ${inventoryData.length} records from S3`);
-
-  // 2. Transform to Fluent format
-  const mapper = new UniversalMapper({
-    fields: {
-      ref: { source: 'sku', required: true },
-      quantity: { source: 'qty', resolver: 'sdk.parseInt' },
-      groupRef: { source: 'warehouse', required: true },
-      catalogue: {
-        resolver: (value, sourceData) => ({
-          ref: process.env.FLUENT_CATALOGUE_REF!
-        })
-      }
-    }
-  });
-
-  const transformed = await mapper.mapArray(inventoryData);
-
-  if (!transformed.success) {
-    throw new Error(`Transformation failed: ${transformed.errors.length} errors`);
-  }
-
-  // 3. Create batch job
-  const job = await client.createJob({
-    name: `inventory-import-${Date.now()}`,
-    retailerId: process.env.FLUENT_RETAILER_ID!
-  });
-
-  console.log(`📦 Created job: ${job.id}`);
-
-  // 4. Send batch (use client methods directly)
-  const batchResult = await client.sendBatch(job.id, {
-    action: 'UPSERT',
-    entityType: 'INVENTORY',
-    source: 'S3_IMPORT',
-    event: 'INVENTORY_UPDATE',
-    entities: transformed.data
-  });
-
-  console.log(`✅ Batch sent: ${batchResult.id}`);
-
-  // 5. Poll for completion (manual polling)
-  let status = await client.getBatchStatus(job.id, batchResult.id);
-  while (status.status === 'PROCESSING') {
-    await new Promise(resolve => setTimeout(resolve, 2000));
-    status = await client.getBatchStatus(job.id, batchResult.id);
-  }
-
-  console.log(`✅ Batch complete: ${status.processedCount} processed`);
-
-  return status;
-}
-```
-
----
-
-## Versori Platform Integration
-
-Use pagination within Versori workflows.
-
-### Pattern: Versori Webhook → Paginate → Process
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
-
-export const inventorySync = webhook('inventory-sync', {
-  response: { mode: 'sync' }
-})
-  .then(fn('sync-inventory', async (ctx) => {
-    // Auto-detects Versori context!
-    const client = await createClient(ctx); // Auto-detects Versori context
-
-    // Paginate inventory positions
-    const result = await client.graphql({
-      query: `
-        query GetVirtualPositions($first: Int!, $after: String) {
-          virtualPositions(first: $first, after: $after) {
-            edges {
-              node {
-                id
-                productRef
-                quantity
-                groupRef
-                status
-              }
-              cursor
-            }
-            pageInfo { hasNextPage }
-          }
-        }
-      `,
-      variables: { first: 150 },
-      pagination: {
-        maxRecords: 10000,
-        timeoutMs: 60000,  // Versori has request timeouts
-        onProgress: (page, records) => {
-          ctx.log.info(`Fetched ${records} positions (page ${page})`);
-        }
-      }
-    });
-
-    const positions = result.data.virtualPositions.edges;
-
-    // Transform for external system
-    const mapper = new UniversalMapper({
-      fields: {
-        sku: { source: 'node.productRef' },
-        qty: { source: 'node.quantity' },
-        location: { source: 'node.groupRef' }
-      }
-    });
-
-    const transformed = await mapper.mapArray(positions);
-
-    // Send to external API, S3, or return
-    log.info(`✅ Processed ${transformed.data.length} positions`);
-
-    return {
-      status: 'success',
-      recordCount: transformed.data.length,
-      data: transformed.data
-    };
-  }));
-```
-
-### Scheduled Versori Job with Pagination
-
-```typescript
-import { schedule, fn } from '@versori/run';
-import { createClient } from '@fluentcommerce/fc-connect-sdk';
-
-export const dailyInventoryExport = schedule('daily-export', {
-  cron: '0 2 * * *'  // 2 AM daily
-})
-  .then(fn('export-inventory', async ({ connections, log, openKv, activation }) => {
-    const client = await createClient(ctx); // Auto-detects Versori context
-    const kv = openKv();
-
-    // Get last export timestamp
-    const lastExport = await kv.get('last-export-timestamp');
-    const fromDate = lastExport?.value || new Date(Date.now() - 86400000).toISOString();
-
-    log.info(`Exporting inventory changes since ${fromDate}`);
-
-    // Paginate only changed records
-    const result = await client.graphql({
-      query: `
-        query GetUpdatedPositions($first: Int!, $after: String, $from: DateTime!) {
-          virtualPositions(
-            first: $first
-            after: $after
-            updatedOn: { from: $from }
-          ) {
-            edges {
-              node {
-                id
-                productRef
-                quantity
-                groupRef
-                updatedOn
-              }
-              cursor
-            }
-            pageInfo { hasNextPage }
-          }
-        }
-      `,
-      variables: { first: 150, from: fromDate },
-      pagination: { maxRecords: 50000 }
-    });
-
-    // Store current timestamp
-    await kv.set('last-export-timestamp', new Date().toISOString());
-
-    log.info(`✅ Exported ${result.extensions.autoPagination.totalRecords} positions`);
-
-    return {
-      status: 'success',
-      recordCount: result.extensions.autoPagination.totalRecords,
-      fromDate,
-      toDate: new Date().toISOString()
-    };
-  }));
-```
-
----
-
-## Complete ETL Pipelines
-
-### ETL Example 1: Fluent → Transform → Database
-
-```typescript
-import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-import { Pool } from 'pg';  // PostgreSQL client
-
-async function syncInventoryToDatabase() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-  const db = new Pool(/* db config */);
-
-  // 1. Paginate from Fluent
-  const result = await client.graphql({
-    query: virtualPositionsQuery,
-    variables: { first: 150 },
-    pagination: { maxRecords: 20000 }
-  });
-
-  // 2. Transform
-  const mapper = new UniversalMapper({
-    fields: {
-      sku: { source: 'node.productRef' },
-      quantity: { source: 'node.quantity', resolver: 'sdk.parseInt' },
-      warehouse: { source: 'node.groupRef' },
-      last_synced: { resolver: () => new Date().toISOString() }
-    }
-  });
-
-  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
-
-  // 3. Batch insert to database
-  const dbClient = await db.connect();
-  try {
-    await dbClient.query('BEGIN');
-
-    for (const record of transformed.data) {
-      await dbClient.query(`
-        INSERT INTO inventory (sku, quantity, warehouse, last_synced)
-        VALUES ($1, $2, $3, $4)
-        ON CONFLICT (sku, warehouse)
-        DO UPDATE SET quantity = $2, last_synced = $4
-      `, [record.sku, record.quantity, record.warehouse, record.last_synced]);
-    }
-
-    await dbClient.query('COMMIT');
-    console.log(`✅ Synced ${transformed.data.length} records to database`);
-  } catch (error) {
-    await dbClient.query('ROLLBACK');
-    throw error;
-  } finally {
-    dbClient.release();
-  }
-}
-```
-
-### ETL Example 2: S3 → Transform → Paginate → Fluent
-
-```typescript
-import { createClient, S3DataSource, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function importOrdersFromS3() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-  const s3 = new S3DataSource(
-    {
-      type: 'S3_JSON',
-      connectionId: 's3-orders',
-      name: 'S3 Orders',
-      s3Config: {
-        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-        region: process.env.AWS_REGION!,
-        bucket: 'orders-bucket'
-      }
-    },
-    logger
-  );
-
-  // 1. Read from S3
-  const orders = JSON.parse(await s3.downloadFile('orders.json') as string);
-
-  // 2. Transform
-  const mapper = new UniversalMapper({
-    fields: {
-      ref: { source: 'order_id', required: true },
-      type: { resolver: () => 'STANDARD' },
-      totalPrice: { source: 'total', resolver: 'sdk.parseFloat' },
-      items: {
-        resolver: (value, sourceData) => {
-          return sourceData.line_items.map(item => ({
-            productRef: item.sku,
-            quantity: item.qty,
-            price: item.price
-          }));
-        }
-      }
-    }
-  });
-
-  const transformed = await mapper.mapArray(orders);
-
-  // 3. Create orders in Fluent (batched)
-  const job = await client.createJob({
-    name: `order-import-${Date.now()}`,
-    retailerId: process.env.FLUENT_RETAILER_ID!
-  });
-
-  const batch = await client.sendBatch(job.id, {
-    entities: transformed.data,
-    entityType: 'ORDER'
-  });
-
-  // Poll for batch completion
-  let status = await client.getBatchStatus(job.id, batch.id);
-  while (status.status === 'PROCESSING') {
-    await new Promise(resolve => setTimeout(resolve, 2000));
-    status = await client.getBatchStatus(job.id, batch.id);
-  }
-
-  console.log(`✅ Imported ${transformed.data.length} orders`);
-}
-```
-
----
-
-## Performance Optimization Patterns
-
-### Pattern 1: Parallel Pagination (Multiple Entity Types)
-
-```typescript
-import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function fetchAllEntitiesInParallel() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-
-  // Fetch multiple entity types in parallel
-  const [positions, locations, products] = await Promise.all([
-    client.graphql({
-      query: virtualPositionsQuery,
-      variables: { first: 150 },
-      pagination: { maxRecords: 10000 }
-    }),
-    client.graphql({
-      query: locationsQuery,
-      variables: { first: 100 },
-      pagination: { maxRecords: 5000 }
-    }),
-    client.graphql({
-      query: productsQuery,
-      variables: { first: 75 },
-      pagination: { maxRecords: 5000 }
-    })
-  ]);
-
-  return {
-    positions: positions.data.virtualPositions.edges,
-    locations: locations.data.locations.edges,
-    products: products.data.products.edges
-  };
-}
-```
-
-### Pattern 2: Streaming Transformation
-
-Transform data as it's paginated (memory efficient):
-
-```typescript
-import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
-
-async function streamingTransform() {
-  const logger = createConsoleLogger();
-  
-  const client = await createClient({
-    config: {
-      baseUrl: process.env.FLUENT_BASE_URL!,
-      clientId: process.env.FLUENT_CLIENT_ID!,
-      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
-      retailerId: process.env.FLUENT_RETAILER_ID!
-    }
-  });
-  const mapper = new UniversalMapper(/* config */);
-  let processedCount = 0;
-
-  const result = await client.graphql({
-    query: virtualPositionsQuery,
-    variables: { first: 150 },
-    pagination: {
-      maxRecords: 50000,
-      onProgress: async (page, totalRecords) => {
-        // Note: This runs after EACH page, but we can't transform mid-pagination
-        // For true streaming, use manual pagination instead
-        console.log(`Progress: ${totalRecords} records`);
-      }
-    }
-  });
-
-  // Transform after all pages fetched
-  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
-  return transformed.data;
-}
-```
-
----
-
-## Key Takeaways
-
-✅ **UniversalMapper**: Transform paginated data with field mapping and resolvers
-✅ **ExtractionOrchestrator**: Export paginated data to S3 in JSON/Parquet format
-✅ **Batch Ingestion**: Reverse flow from S3/external → Fluent
-✅ **Versori Integration**: Auto-detects context, works in webhooks/schedules
-✅ **ETL Pipelines**: Chain pagination with transformation and destination writes
-✅ **Performance**: Use parallel fetching for multiple entity types
-
----
-
-## Next Steps
-
-Learn how to troubleshoot and optimize pagination:
-
-**Continue to:** [Module 6: Troubleshooting →](./auto-pagination-06-troubleshooting.md)
-
----
-
-[← Back to Index](../../advanced-services/advanced-services-readme.md) | [← Previous: Advanced Patterns](./auto-pagination-04-advanced-patterns.md) | [Next: Troubleshooting →](./auto-pagination-06-troubleshooting.md)
+# Module 5: SDK Ecosystem Integration
+
+> **Learning Objective:** Integrate auto-pagination with other SDK services like UniversalMapper, ExtractionOrchestrator, S3DataSource, and Versori platform.
+>
+> **Level:** Advanced
+
+## Table of Contents
+
+1. `Pagination + UniversalMapper (Transform Data)`
+2. `Pagination + ExtractionOrchestrator (Export to S3)`
+3. `Pagination + Batch Ingestion (Reverse Flow)`
+4. [Versori Platform Integration](#versori-platform-integration)
+5. [Complete ETL Pipelines](#complete-etl-pipelines)
+6. [Performance Optimization Patterns](#performance-optimization-patterns)
+
+---
+
+## Pagination + UniversalMapper
+
+Transform paginated data using SDK's Universal Mapper.
+
+### Pattern: Fetch → Transform → Process
+
+```typescript
+import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+async function fetchAndTransformInventory() {
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+
+  // 1. Fetch data with auto-pagination
+  const result = await client.graphql({
+    query: `
+      query GetVirtualPositions($first: Int!, $after: String) {
+        virtualPositions(first: $first, after: $after) {
+          edges {
+            node {
+              id
+              productRef
+              quantity
+              groupRef
+              status
+              updatedOn
+            }
+            cursor
+          }
+          pageInfo { hasNextPage }
+        }
+      }
+    `,
+    variables: { first: 150 },
+    pagination: {
+      maxRecords: 10000,
+      onProgress: (page, records) => {
+        console.log(`📄 Fetched ${records} positions...`);
+      }
+    }
+  });
+
+  // 2. Define mapping configuration
+  const mappingConfig = {
+    fields: {
+      sku: { source: 'node.productRef', required: true },
+      qty: { source: 'node.quantity', resolver: 'sdk.parseInt' },
+      location: { source: 'node.groupRef', required: true },
+      status: { source: 'node.status', resolver: 'sdk.lowercase' },
+      lastUpdated: { source: 'node.updatedOn', resolver: 'sdk.formatDate' }
+    }
+  };
+
+  // 3. Transform data with UniversalMapper
+  const mapper = new UniversalMapper(mappingConfig);
+  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
+
+  if (!transformed.success) {
+    console.error('Mapping errors:', transformed.errors);
+    throw new Error('Data transformation failed');
+  }
+
+  // 4. Process transformed data
+  console.log(`✅ Transformed ${transformed.data.length} records`);
+
+  return transformed.data;
+  // [{ sku: 'SKU-001', qty: 150, location: 'WH-EAST', status: 'active', lastUpdated: '2024-01-15' }, ...]
+}
+```
+
+### Custom Resolvers with Pagination
+
+```typescript
+// Define custom transformation logic
+const mapper = new UniversalMapper(mappingConfig, {
+  customResolvers: {
+    'custom.calculateValue': (value, sourceData, helpers) => {
+      const qty = helpers.parseIntSafe(sourceData.node.quantity, 0);
+      const price = helpers.parseFloatSafe(sourceData.node.price, 0);
+      return qty * price;
+    },
+    'custom.determineAvailability': (value, sourceData) => {
+      return sourceData.node.quantity > 0 && sourceData.node.status === 'ACTIVE'
+        ? 'IN_STOCK'
+        : 'OUT_OF_STOCK';
+    }
+  }
+});
+
+const paginatedResult = await client.graphql({ query, variables, pagination });
+const transformed = await mapper.mapArray(paginatedResult.data.virtualPositions.edges);
+```
+
+---
+
+## When to Use FluentClient.graphql() vs ExtractionOrchestrator
+
+**Decision Guide:**
+
+| Scenario | Use | Why |
+|----------|-----|-----|
+| **Query + Transform Only** | `FluentClient.graphql()` with `pagination` | Direct access, minimal overhead, full control |
+| **Query + Export to S3/SFTP** | `ExtractionOrchestrator` | Built-in export, file generation, retry logic |
+| **Custom Processing Logic** | `FluentClient.graphql()` with `pagination` | Maximum flexibility for your workflow |
+| **Standard ETL Pipelines** | `ExtractionOrchestrator` | Pre-built patterns, less code |
+| **Multiple Export Formats** | `ExtractionOrchestrator` | Supports JSON, XML, CSV, Parquet |
+| **Need Progress Tracking** | Both support via callbacks | Use whichever fits your workflow |
+
+### FluentClient.graphql() - Direct Approach
+
+**Best for:** Custom workflows, transformation-only, direct data access
+
+```typescript
+import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+const client = await createClient({ config: { /* ... */ } });
+
+// 1. Fetch with auto-pagination
+const result = await client.graphql({
+  query: `query GetVirtualPositions($first: Int!, $after: String) { ... }`,
+  variables: { first: 150 },
+  pagination: { maxRecords: 10000 }
+});
+
+// 2. Transform with UniversalMapper
+const mapper = new UniversalMapper({ fields: { /* ... */ } });
+const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
+
+// 3. Do whatever you want with the data
+await yourCustomLogic(transformed.data);
+```
+
+### ExtractionOrchestrator - High-Level Approach
+
+**Best for:** Standard exports to S3/SFTP, pre-built patterns
+
+```typescript
+import { ExtractionOrchestrator, S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+const orchestrator = new ExtractionOrchestrator(client, logger);
+
+// All-in-one: Query + Transform + Export
+const result = await orchestrator.extract({
+  query: `query GetVirtualPositions($first: Int!, $after: String) { ... }`,
+  extractionPath: 'virtualPositions.edges',
+  destination: s3DataSource,
+  format: 'json',
+  filename: 'inventory-export.json',
+  pagination: { maxRecords: 10000 }
+});
+
+// File is already on S3!
+```
+
+**Key Difference:** `ExtractionOrchestrator` = `FluentClient.graphql()` + file export + retry logic + pre-built patterns.
+
+---
+
+## Pagination + ExtractionOrchestrator
+
+Export paginated data to S3 using ExtractionOrchestrator.
+
+### Pattern: Fetch → Transform → Export
+
+```typescript
+import {
+  createClient,
+  ExtractionOrchestrator,
+  S3DataSource,
+  UniversalMapper,
+  createConsoleLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function exportInventoryToS3() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+
+  // 1. Setup S3 destination
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_JSON',
+      connectionId: 's3-export',
+      name: 'S3 Export',
+      s3Config: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+        bucket: 'inventory-exports'
+      }
+    },
+    logger
+  );
+
+  // 2. Paginate inventory data
+  const result = await client.graphql({
+    query: `
+      query GetVirtualPositions($first: Int!, $after: String) {
+        virtualPositions(first: $first, after: $after) {
+          edges {
+            node {
+              id
+              productRef
+              quantity
+              groupRef
+              status
+              createdOn
+              updatedOn
+            }
+            cursor
+          }
+          pageInfo { hasNextPage }
+        }
+      }
+    `,
+    variables: { first: 150 },
+    pagination: {
+      maxRecords: 15000,
+      timeoutMs: 300000
+    }
+  });
+
+  console.log(`✅ Fetched ${result.extensions.autoPagination.totalRecords} positions`);
+
+  // 3. Transform data
+  const mapper = new UniversalMapper({
+    fields: {
+      sku: { source: 'node.productRef' },
+      quantity: { source: 'node.quantity', resolver: 'sdk.parseInt' },
+      warehouse: { source: 'node.groupRef' },
+      status: { source: 'node.status' },
+      lastModified: { source: 'node.updatedOn', resolver: 'sdk.formatDate' }
+    }
+  });
+
+  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
+
+  if (!transformed.success) {
+    throw new Error(`Transformation failed: ${transformed.errors.length} errors`);
+  }
+
+  // 4. Export to S3
+  const timestamp = new Date().toISOString().split('T')[0];
+  const filename = `inventory-export-${timestamp}.json`;
+
+  await s3.uploadFile(filename, JSON.stringify(transformed.data, null, 2));
+
+  console.log(`✅ Exported to s3://inventory-exports/${filename}`);
+
+  return {
+    filename,
+    recordCount: transformed.data.length,
+    stats: result.extensions.autoPagination
+  };
+}
+```
+
+### Chunked S3 Export (Large Datasets)
+
+For very large datasets, export in chunks to avoid memory issues:
+
+```typescript
+import { createClient, S3DataSource, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function exportInventoryInChunks() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_JSON',
+      connectionId: 's3-chunked-export',
+      name: 'S3 Chunked Export',
+      s3Config: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+        bucket: 'inventory-exports'
+      }
+    },
+    logger
+  );
+
+  let after: string | null = null;
+  let chunkNumber = 0;
+  const CHUNK_SIZE = 5000;
+
+  while (true) {
+    // Fetch chunk with pagination
+    const result = await client.graphql({
+      query: virtualPositionsQuery,
+      variables: { first: 150, after },
+      pagination: {
+        maxRecords: CHUNK_SIZE,
+        maxPages: 35  // ~35 pages * 150 = 5250 records
+      }
+    });
+
+    const edges = result.data.virtualPositions.edges;
+    if (edges.length === 0) break;
+
+    // Transform chunk
+    const mapper = new UniversalMapper(mappingConfig);
+    const transformed = await mapper.mapArray(edges);
+
+    // Export chunk to S3
+    const filename = `inventory-chunk-${chunkNumber.toString().padStart(4, '0')}.json`;
+    await s3.uploadFile(filename, JSON.stringify(transformed.data, null, 2));
+
+    console.log(`✅ Exported chunk ${chunkNumber}: ${edges.length} records`);
+
+    // Check if we got fewer records than max (means we're done)
+    if (result.extensions.autoPagination.truncated &&
+        result.extensions.autoPagination.truncationReason === 'maxRecords') {
+      after = edges[edges.length - 1].cursor;
+      chunkNumber++;
+    } else {
+      break;  // No more data
+    }
+  }
+
+  console.log(`✅ Export complete: ${chunkNumber + 1} chunks`);
+}
+```
+
+---
+
+## Pagination + Batch Ingestion
+
+Reverse flow: Fetch from S3 → Paginate → Transform → Batch Ingest to Fluent.
+
+### Pattern: S3 → Transform → Batch API
+
+```typescript
+import {
+  createClient,
+  S3DataSource,
+  UniversalMapper,
+  createConsoleLogger
+} from '@fluentcommerce/fc-connect-sdk';
+
+async function ingestInventoryFromS3() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_JSON',
+      connectionId: 's3-ingestion',
+      name: 'S3 Ingestion',
+      s3Config: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+        bucket: 'inventory-updates'
+      }
+    },
+    logger
+  );
+
+  // 1. Read data from S3
+  const fileContent = await s3.downloadFile('inventory-updates.json');
+  const inventoryData = JSON.parse(fileContent);
+
+  console.log(`📥 Loaded ${inventoryData.length} records from S3`);
+
+  // 2. Transform to Fluent format
+  const mapper = new UniversalMapper({
+    fields: {
+      ref: { source: 'sku', required: true },
+      quantity: { source: 'qty', resolver: 'sdk.parseInt' },
+      groupRef: { source: 'warehouse', required: true },
+      catalogue: {
+        resolver: (value, sourceData) => ({
+          ref: process.env.FLUENT_CATALOGUE_REF!
+        })
+      }
+    }
+  });
+
+  const transformed = await mapper.mapArray(inventoryData);
+
+  if (!transformed.success) {
+    throw new Error(`Transformation failed: ${transformed.errors.length} errors`);
+  }
+
+  // 3. Create batch job
+  const job = await client.createJob({
+    name: `inventory-import-${Date.now()}`,
+    retailerId: process.env.FLUENT_RETAILER_ID!
+  });
+
+  console.log(`📦 Created job: ${job.id}`);
+
+  // 4. Send batch (use client methods directly)
+  const batchResult = await client.sendBatch(job.id, {
+    action: 'UPSERT',
+    entityType: 'INVENTORY',
+    source: 'S3_IMPORT',
+    event: 'INVENTORY_UPDATE',
+    entities: transformed.data
+  });
+
+  console.log(`✅ Batch sent: ${batchResult.id}`);
+
+  // 5. Poll for completion (manual polling)
+  let status = await client.getBatchStatus(job.id, batchResult.id);
+  while (status.status === 'PROCESSING') {
+    await new Promise(resolve => setTimeout(resolve, 2000));
+    status = await client.getBatchStatus(job.id, batchResult.id);
+  }
+
+  console.log(`✅ Batch complete: ${status.processedCount} processed`);
+
+  return status;
+}
+```
+
+---
+
+## Versori Platform Integration
+
+Use pagination within Versori workflows.
+
+### Pattern: Versori Webhook → Paginate → Process
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { createClient, UniversalMapper } from '@fluentcommerce/fc-connect-sdk';
+
+export const inventorySync = webhook('inventory-sync', {
+  response: { mode: 'sync' }
+})
+  .then(fn('sync-inventory', async (ctx) => {
+    // Auto-detects Versori context!
+    const client = await createClient(ctx); // Auto-detects Versori context
+
+    // Paginate inventory positions
+    const result = await client.graphql({
+      query: `
+        query GetVirtualPositions($first: Int!, $after: String) {
+          virtualPositions(first: $first, after: $after) {
+            edges {
+              node {
+                id
+                productRef
+                quantity
+                groupRef
+                status
+              }
+              cursor
+            }
+            pageInfo { hasNextPage }
+          }
+        }
+      `,
+      variables: { first: 150 },
+      pagination: {
+        maxRecords: 10000,
+        timeoutMs: 60000,  // Versori has request timeouts
+        onProgress: (page, records) => {
+          ctx.log.info(`Fetched ${records} positions (page ${page})`);
+        }
+      }
+    });
+
+    const positions = result.data.virtualPositions.edges;
+
+    // Transform for external system
+    const mapper = new UniversalMapper({
+      fields: {
+        sku: { source: 'node.productRef' },
+        qty: { source: 'node.quantity' },
+        location: { source: 'node.groupRef' }
+      }
+    });
+
+    const transformed = await mapper.mapArray(positions);
+
+    // Send to external API, S3, or return
+    log.info(`✅ Processed ${transformed.data.length} positions`);
+
+    return {
+      status: 'success',
+      recordCount: transformed.data.length,
+      data: transformed.data
+    };
+  }));
+```
+
+### Scheduled Versori Job with Pagination
+
+```typescript
+import { schedule, fn } from '@versori/run';
+import { createClient } from '@fluentcommerce/fc-connect-sdk';
+
+export const dailyInventoryExport = schedule('daily-export', {
+  cron: '0 2 * * *'  // 2 AM daily
+})
+  .then(fn('export-inventory', async ({ connections, log, openKv, activation }) => {
+    const client = await createClient(ctx); // Auto-detects Versori context
+    const kv = openKv();
+
+    // Get last export timestamp
+    const lastExport = await kv.get('last-export-timestamp');
+    const fromDate = lastExport?.value || new Date(Date.now() - 86400000).toISOString();
+
+    log.info(`Exporting inventory changes since ${fromDate}`);
+
+    // Paginate only changed records
+    const result = await client.graphql({
+      query: `
+        query GetUpdatedPositions($first: Int!, $after: String, $from: DateTime!) {
+          virtualPositions(
+            first: $first
+            after: $after
+            updatedOn: { from: $from }
+          ) {
+            edges {
+              node {
+                id
+                productRef
+                quantity
+                groupRef
+                updatedOn
+              }
+              cursor
+            }
+            pageInfo { hasNextPage }
+          }
+        }
+      `,
+      variables: { first: 150, from: fromDate },
+      pagination: { maxRecords: 50000 }
+    });
+
+    // Store current timestamp
+    await kv.set('last-export-timestamp', new Date().toISOString());
+
+    log.info(`✅ Exported ${result.extensions.autoPagination.totalRecords} positions`);
+
+    return {
+      status: 'success',
+      recordCount: result.extensions.autoPagination.totalRecords,
+      fromDate,
+      toDate: new Date().toISOString()
+    };
+  }));
+```
+
+---
+
+## Complete ETL Pipelines
+
+### ETL Example 1: Fluent → Transform → Database
+
+```typescript
+import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+import { Pool } from 'pg';  // PostgreSQL client
+
+async function syncInventoryToDatabase() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+  const db = new Pool(/* db config */);
+
+  // 1. Paginate from Fluent
+  const result = await client.graphql({
+    query: virtualPositionsQuery,
+    variables: { first: 150 },
+    pagination: { maxRecords: 20000 }
+  });
+
+  // 2. Transform
+  const mapper = new UniversalMapper({
+    fields: {
+      sku: { source: 'node.productRef' },
+      quantity: { source: 'node.quantity', resolver: 'sdk.parseInt' },
+      warehouse: { source: 'node.groupRef' },
+      last_synced: { resolver: () => new Date().toISOString() }
+    }
+  });
+
+  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
+
+  // 3. Batch insert to database
+  const dbClient = await db.connect();
+  try {
+    await dbClient.query('BEGIN');
+
+    for (const record of transformed.data) {
+      await dbClient.query(`
+        INSERT INTO inventory (sku, quantity, warehouse, last_synced)
+        VALUES ($1, $2, $3, $4)
+        ON CONFLICT (sku, warehouse)
+        DO UPDATE SET quantity = $2, last_synced = $4
+      `, [record.sku, record.quantity, record.warehouse, record.last_synced]);
+    }
+
+    await dbClient.query('COMMIT');
+    console.log(`✅ Synced ${transformed.data.length} records to database`);
+  } catch (error) {
+    await dbClient.query('ROLLBACK');
+    throw error;
+  } finally {
+    dbClient.release();
+  }
+}
+```
+
+### ETL Example 2: S3 → Transform → Paginate → Fluent
+
+```typescript
+import { createClient, S3DataSource, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function importOrdersFromS3() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+  const s3 = new S3DataSource(
+    {
+      type: 'S3_JSON',
+      connectionId: 's3-orders',
+      name: 'S3 Orders',
+      s3Config: {
+        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+        region: process.env.AWS_REGION!,
+        bucket: 'orders-bucket'
+      }
+    },
+    logger
+  );
+
+  // 1. Read from S3
+  const orders = JSON.parse(await s3.downloadFile('orders.json') as string);
+
+  // 2. Transform
+  const mapper = new UniversalMapper({
+    fields: {
+      ref: { source: 'order_id', required: true },
+      type: { resolver: () => 'STANDARD' },
+      totalPrice: { source: 'total', resolver: 'sdk.parseFloat' },
+      items: {
+        resolver: (value, sourceData) => {
+          return sourceData.line_items.map(item => ({
+            productRef: item.sku,
+            quantity: item.qty,
+            price: item.price
+          }));
+        }
+      }
+    }
+  });
+
+  const transformed = await mapper.mapArray(orders);
+
+  // 3. Create orders in Fluent (batched)
+  const job = await client.createJob({
+    name: `order-import-${Date.now()}`,
+    retailerId: process.env.FLUENT_RETAILER_ID!
+  });
+
+  const batch = await client.sendBatch(job.id, {
+    entities: transformed.data,
+    entityType: 'ORDER'
+  });
+
+  // Poll for batch completion
+  let status = await client.getBatchStatus(job.id, batch.id);
+  while (status.status === 'PROCESSING') {
+    await new Promise(resolve => setTimeout(resolve, 2000));
+    status = await client.getBatchStatus(job.id, batch.id);
+  }
+
+  console.log(`✅ Imported ${transformed.data.length} orders`);
+}
+```
+
+---
+
+## Performance Optimization Patterns
+
+### Pattern 1: Parallel Pagination (Multiple Entity Types)
+
+```typescript
+import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function fetchAllEntitiesInParallel() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+
+  // Fetch multiple entity types in parallel
+  const [positions, locations, products] = await Promise.all([
+    client.graphql({
+      query: virtualPositionsQuery,
+      variables: { first: 150 },
+      pagination: { maxRecords: 10000 }
+    }),
+    client.graphql({
+      query: locationsQuery,
+      variables: { first: 100 },
+      pagination: { maxRecords: 5000 }
+    }),
+    client.graphql({
+      query: productsQuery,
+      variables: { first: 75 },
+      pagination: { maxRecords: 5000 }
+    })
+  ]);
+
+  return {
+    positions: positions.data.virtualPositions.edges,
+    locations: locations.data.locations.edges,
+    products: products.data.products.edges
+  };
+}
+```
+
+### Pattern 2: Streaming Transformation
+
+Transform data as it's paginated (memory efficient):
+
+```typescript
+import { createClient, UniversalMapper, createConsoleLogger } from '@fluentcommerce/fc-connect-sdk';
+
+async function streamingTransform() {
+  const logger = createConsoleLogger();
+  
+  const client = await createClient({
+    config: {
+      baseUrl: process.env.FLUENT_BASE_URL!,
+      clientId: process.env.FLUENT_CLIENT_ID!,
+      clientSecret: process.env.FLUENT_CLIENT_SECRET!,
+      retailerId: process.env.FLUENT_RETAILER_ID!
+    }
+  });
+  const mapper = new UniversalMapper(/* config */);
+  let processedCount = 0;
+
+  const result = await client.graphql({
+    query: virtualPositionsQuery,
+    variables: { first: 150 },
+    pagination: {
+      maxRecords: 50000,
+      onProgress: async (page, totalRecords) => {
+        // Note: This runs after EACH page, but we can't transform mid-pagination
+        // For true streaming, use manual pagination instead
+        console.log(`Progress: ${totalRecords} records`);
+      }
+    }
+  });
+
+  // Transform after all pages fetched
+  const transformed = await mapper.mapArray(result.data.virtualPositions.edges);
+  return transformed.data;
+}
+```
+
+---
+
+## Key Takeaways
+
+✅ **UniversalMapper**: Transform paginated data with field mapping and resolvers
+✅ **ExtractionOrchestrator**: Export paginated data to S3 in JSON/Parquet format
+✅ **Batch Ingestion**: Reverse flow from S3/external → Fluent
+✅ **Versori Integration**: Auto-detects context, works in webhooks/schedules
+✅ **ETL Pipelines**: Chain pagination with transformation and destination writes
+✅ **Performance**: Use parallel fetching for multiple entity types
+
+---
+
+## Next Steps
+
+Learn how to troubleshoot and optimize pagination:
+
+**Continue to:** [Module 6: Troubleshooting →](./auto-pagination-06-troubleshooting.md)
+
+---
+
+[← Back to Index](../../advanced-services/advanced-services-readme.md) | [← Previous: Advanced Patterns](./auto-pagination-04-advanced-patterns.md) | [Next: Troubleshooting →](./auto-pagination-06-troubleshooting.md)