@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-orders-to-sftp-xml.md

@@ -1,2541 +1,2541 @@
----
-template_id: tpl-extract-orders-to-sftp-xml
-canonical_filename: template-extraction-orders-to-sftp-xml.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: sftp-xml
-entity: orders
-format: xml
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
-  - dispose-finally
----
-
-# Template: Extraction - Orders to SFTP XML
-
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Memory Management** - Clear large result sets after processing batches
-- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
-- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
-- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
-
-## Installation
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk
-```
-
-Always check [npm registry](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) for the latest version.
-
----
-
-## 📚 STEP 1: Load These Docs (Human Checklist)
-
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
----
-
-## 📋 STEP 2: Tell Your AI (Prompt)
-
-Copy/paste this prompt after loading the documentation above:
-
-```
-Create a Versori scheduled extractor for orders that uses ExtractionOrchestrator + JobTracker, incremental updatedOn with a 60s overlap buffer, transforms via UniversalMapper, generates XML with SDK's XMLBuilder, uploads to SFTP using SftpDataSource with dispose(). Include 3 workflows: scheduled, ad-hoc webhook, and job-status query with native Versori logging.
-```
-
----
-
-## 📦 SDK Imports (Verified - Versori Optimized)
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-
----
-
-# Versori Scheduled: Orders Extraction to SFTP XML (Incremental)
-
-**FC Connect SDK Use Case Guide**
-
-> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> Version: Check [npm](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) for latest
-
-Context: Scheduled Versori workflow that extracts new/updated orders from Fluent Commerce via GraphQL query with **ExtractionOrchestrator**, **JobTracker**, and **incremental timestamp tracking**, transforms with `UniversalMapper`, and writes **XML files** to partner SFTP server for 3PL/WMS/fulfillment center integration.
-
-**Pattern**: EXTRACTION (Fluent → SFTP XML)
-**Complexity**: High | Runtime: Versori Platform (Scheduled)
-
----
-
-## ⚠️ IMPORTANT: Production-Ready Base Template
-
-> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
->
-> This is a **production-ready base template** demonstrating FC Connect SDK best practices for order extraction workflows with XML output.
->
-> **✅ INCLUDED FEATURES:**
->
-> - ✅ Comprehensive error handling with retry logic
-> - ✅ SFTP upload with exponential backoff (3 attempts)
-> - ✅ State management with overlap buffer (prevents missed records)
-> - ✅ Job tracking with lifecycle management
-> - ✅ Security (credential masking in logs)
-> - ✅ UTC time enforcement (prevents timezone bugs)
-> - ✅ Incremental extraction (safe, efficient, production-ready)
-> - ✅ Natural rate limiting via timestamps
->
-> **📝 BEFORE DEPLOYING:**
->
-> 1. Review and customize activation variables for your environment
-> 2. Test with sample data in your Versori workspace
-> 3. Adjust safety limits (pageSize, maxRecords) if needed
-> 4. Configure monitoring alerts for extraction failures
-> 5. Verify SFTP credentials and paths
->
-> **This base template follows SDK best practices - tweak specific to your needs.**
-
----
-
-## What You'll Build
-
-- **Incremental extraction** using `updatedOn >= (lastRunTime - buffer)` with **overlap buffer**
-- **ExtractionOrchestrator** for auto-pagination and path-based extraction
-- **JobTracker** for lifecycle management and status tracking
-- **State management** with VersoriKV to track last successful run
-- **Safety buffer** (60 seconds) to handle clock skew and race conditions
-- GraphQL query with nested order lines
-- UniversalMapper transformation with line item data
-- **XML file generation** with proper structure for 3PL/WMS systems
-- **SFTP upload** to partner server (with `dispose()` cleanup)
-- **3 workflow patterns**: scheduled, ad-hoc webhook, job status query
-- **Failure recovery** with timestamp tracking
-
-## Business Use Case
-
-**Hourly order feed to 3PL/fulfillment center:**
-
-- Extract new and updated orders since last run
-- Generate XML file with order header + line items
-- Upload to 3PL SFTP server for warehouse management system
-- Run every hour to enable real-time fulfillment
-- Support order updates (address changes, item modifications)
-- Standard XML format for EDI/ERP integration
-
-## SDK Methods Used
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-await createClient(ctx); // Versori-aware client
-const orchestrator = new ExtractionOrchestrator(client, log); // Auto-pagination
-const tracker = new JobTracker(kv, log); // Job lifecycle tracking
-await orchestrator.extract({ query, resultPath, variables, pageSize, maxRecords }); // Extract
-new VersoriKVAdapter(ctx.openKv(':project:')); // State management
-new UniversalMapper(exportMapping); // Field transformation
-new XMLBuilder(options); // XML generation with auto-escaping
-await sftp.uploadFile(remotePath, buffer); // SFTP upload (no options param)
-await sftp.dispose(); // CRITICAL: Connection cleanup
-```
-
-## SFTP Connection Setup (Recommended)
-
-**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
-
-**Connection Configuration:**
-
-1. In Versori platform, create a connection named `versori_ftp_server`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-5. The SDK will automatically decode the credentials using:
-
-```typescript
-// Get SFTP credentials from Versori connection (Basic Auth)
-// RECOMMENDED: Use activation.connections (already decoded)
-const allConnections = ctx.activation.connections || [];
-const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-if (!sftpConn) {
-  throw new Error('SFTP connection "versori_ftp_server" not found');
-}
-
-const credential = sftpConn.credentials[0]?.credential;
-if (!credential?.data?.basicAuth) {
-  throw new Error('SFTP connection not configured with Basic Authentication');
-}
-
-const { username, password } = credential.data.basicAuth;
-// ✅ Already decoded - no Buffer.from() needed!
-```
-
-**Why use connections instead of activation variables?**
-
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection can be reused across workflows
-- ✅ No need to manage sensitive data in activation variables
-- ✅ Easier credential rotation
-
-### SFTP Credential Access Methods
-
-**You have TWO methods to access SFTP credentials from Versori connections:**
-
-#### Method 1: activation.connections (✅ RECOMMENDED)
-
-**Best for:** All scenarios - cleanest, no decoding needed
-
-```typescript
-import { Buffer } from 'node:buffer'; // Required for SFTP upload
-
-const { activation, log } = ctx;
-
-// Get the connection (credentials ALREADY DECODED!)
-const allConnections = activation.connections || [];
-const sftpConnection = allConnections.find(c => c.name === 'versori_ftp_server');
-
-if (!sftpConnection) {
-  throw new Error(
-    `SFTP connection not found. Available: ${allConnections.map(c => c.name).join(', ')}`
-  );
-}
-
-const sftpCred = sftpConnection.credentials[0]?.credential;
-
-if (!sftpCred?.data?.basicAuth) {
-  throw new Error('SFTP connection not configured with Basic Auth');
-}
-
-// ✅ Already decoded - no Buffer.from() needed!
-const sftpUsername = sftpCred.data.basicAuth.username;
-const sftpPassword = sftpCred.data.basicAuth.password;
-const sftpHost = sftpConnection.baseUrl || ctx.activation.getVariable('sftpHost');
-
-log.info('SFTP credentials retrieved', {
-  username: sftpUsername,
-  host: sftpHost,
-  hasPassword: !!sftpPassword,
-});
-```
-
-**Why this is best:**
-
-- ✅ No base64 decoding required
-- ✅ Type-safe access to credential structure
-- ✅ Works in all task types (fn, http, webhook)
-- ✅ Cleaner error handling
-
-#### Method 2: credentials().get() (Alternative)
-
-**Use when:** Working in fn() tasks where activation.connections not available
-
-```typescript
-import { Buffer } from 'node:buffer'; // Required for both decoding AND SFTP upload
-
-// Retrieve credentials (returns base64-encoded accessToken)
-const sftpCred = await ctx.credentials().getAccessToken('versori_ftp_server');
-
-if (!sftpCred?.accessToken) {
-  throw new Error('No SFTP credentials found');
-}
-
-// ⚠️ Manual base64 decoding required
-const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-const [sftpUsername, sftpPassword] = rawBasicAuth.split(':');
-
-if (!sftpUsername || !sftpPassword) {
-  throw new Error('Invalid credential format');
-}
-
-log.info('SFTP credentials decoded', {
-  hasUsername: !!sftpUsername,
-  hasPassword: !!sftpPassword,
-});
-```
-
-**Limitations:**
-
-- ⚠️ Requires manual base64 decoding
-- ⚠️ More error-prone (string splitting)
-- ⚠️ Only works in fn() tasks
-
-### Buffer Import Reminder (CRITICAL!)
-
-**For both methods, you MUST import Buffer in Versori/Deno runtime:**
-
-```typescript
-import { Buffer } from 'node:buffer'; // ⚠️ ALWAYS REQUIRED
-```
-
-**Why:**
-
-- SFTP uploads require Buffer: `Buffer.from(xmlContent, 'utf8')`
-- Method 2 decoding requires Buffer: `Buffer.from(accessToken, 'base64')`
-- Deno runtime doesn't have global Buffer (unlike Node.js)
-
-**See:** [SFTP Credential Access & Security](../../../../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for complete documentation with troubleshooting and security best practices
-
-## Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-```json
-{
-  "retailerId": "your-retailer-id",
-  "sftpHost": "sftp.3pl-partner.com",
-  "sftpPort": 22,
-  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
-  "sftpRemotePath": "/incoming/orders/",
-  "pageSize": 200,
-  "maxRecords": 10000,
-  "fallbackStartDate": "2024-01-01T00:00:00Z",
-  "overlapBufferSeconds": "60"
-}
-```
-
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection (see SFTP Connection Setup above).
-
-## Export Mapping Configuration
-
-**IMPORTANT**: Fields match CSV version exactly for consistency.
-
-Create file: `./config/orders.export.xml.json`
-
-```json
-{
-  "name": "orders.export.xml",
-  "version": "1.0.0",
-  "description": "Fluent Orders → 3PL XML Export",
-  "fields": {
-    "order_id": { "source": "ref", "required": true, "resolver": "sdk.trim" },
-    "order_date": { "source": "createdOn", "required": true, "resolver": "sdk.formatDateShort" },
-    "order_status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
-    "customer_name": { "source": "customer.firstName", "required": false, "resolver": "sdk.trim" },
-    "customer_email": { "source": "customer.email", "required": false, "resolver": "sdk.trim" },
-    "ship_to_name": { "source": "deliveryAddress.name", "required": true, "resolver": "sdk.trim" },
-    "ship_to_address1": {
-      "source": "deliveryAddress.street1",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "ship_to_city": { "source": "deliveryAddress.city", "required": true, "resolver": "sdk.trim" },
-    "ship_to_state": {
-      "source": "deliveryAddress.state",
-      "required": true,
-      "resolver": "sdk.uppercase"
-    },
-    "ship_to_zip": {
-      "source": "deliveryAddress.postcode",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "ship_to_country": {
-      "source": "deliveryAddress.country",
-      "required": true,
-      "resolver": "sdk.uppercase"
-    },
-    "line_item_sku": { "source": "product.ref", "required": true, "resolver": "sdk.trim" },
-    "line_item_qty": { "source": "quantity", "required": true, "resolver": "sdk.parseInt" },
-    "line_item_price": { "source": "price", "required": true, "resolver": "sdk.parseFloat" }
-  }
-}
-```
-
-## Mapping & Resolvers Explained
-
-### SDK Resolvers Used
-
-The export mapping uses **SDK resolvers** to transform order data into 3PL-ready XML format:
-
-| Field             | Resolver              | Why?                         | Example Transformation                           |
-| ----------------- | --------------------- | ---------------------------- | ------------------------------------------------ |
-| `order_id`        | `sdk.trim`            | Clean order references       | `" ORD-123 "` → `"ORD-123"`                   |
-| `order_date`      | `sdk.formatDateShort` | 3PL-friendly date format     | `"2025-01-22T14:30:00Z"` → `"2025-01-22"`     |
-| `order_status`    | `sdk.uppercase`       | Normalize status codes       | `"created"` → `"CREATED"`                     |
-| `customer_name`   | `sdk.trim`            | Clean customer data          | `"John  "` → `"John"`                         |
-| `customer_email`  | `sdk.trim`            | Clean email addresses        | `" user@example.com "` → `"user@example.com"` |
-| `ship_to_name`    | `sdk.trim`            | Clean shipping contact       | `"Jane Doe  "` → `"Jane Doe"`                 |
-| `ship_to_state`   | `sdk.uppercase`       | Normalize state codes        | `"ny"` → `"NY"`                               |
-| `ship_to_country` | `sdk.uppercase`       | Normalize country codes      | `"us"` → `"US"`                               |
-| `line_item_sku`   | `sdk.trim`            | Clean SKU references         | `" SKU-001 "` → `"SKU-001"`                   |
-| `line_item_qty`   | `sdk.parseInt`        | Parse quantities as integers | `"5"` → `5`                                   |
-| `line_item_price` | `sdk.parseFloat`      | Parse prices as decimals     | `"19.99"` → `19.99`                           |
-
-### Transformation Flow
-
-```typescript
-// 1. GraphQL Response (from Fluent API)
-{
-  ref: " ORD-12345 ",
-  createdOn: "2025-01-22T14:30:00.000Z",
-  status: "created",
-  customer: {
-    firstName: "John  ",
-    email: " john@example.com "
-  },
-  deliveryAddress: {
-    name: "Jane Doe  ",
-    street1: "123 Main St",
-    city: "New York",
-    state: "ny",
-    postcode: "10001",
-    country: "us"
-  },
-  items: [
-    {
-      quantity: "2",
-      price: "29.99",
-      product: { ref: " SKU-001 ", name: "Widget" }
-    }
-  ]
-}
-
-// 2. UniversalMapper applies resolvers
-const mapper = new UniversalMapper(ordersExportMapping);
-const result = await mapper.map(order);
-
-// 3. Transformed Output (clean, normalized)
-result.data = {
-  order_id: "ORD-12345",              // ✅ Trimmed
-  order_date: "2025-01-22",           // ✅ Short date format
-  order_status: "CREATED",            // ✅ Uppercased
-  customer_name: "John",              // ✅ Trimmed
-  customer_email: "john@example.com", // ✅ Trimmed
-  ship_to_name: "Jane Doe",           // ✅ Trimmed
-  ship_to_address1: "123 Main St",
-  ship_to_city: "New York",
-  ship_to_state: "NY",                // ✅ Uppercased
-  ship_to_zip: "10001",
-  ship_to_country: "US",              // ✅ Uppercased
-  items: [
-    {
-      line_item_sku: "SKU-001",       // ✅ Trimmed
-      line_item_qty: 2,               // ✅ Integer
-      line_item_price: 29.99          // ✅ Float
-    }
-  ]
-}
-
-// 4. Generate XML with proper escaping
-const xml = buildOrdersXML([result.data]);
-```
-
-### Custom Resolvers for Order-Specific Logic
-
-You can add **custom resolvers** for 3PL-specific transformations:
-
-```typescript
-const ordersExportMapping = {
-  name: 'orders.export.xml',
-  version: '1.0.0',
-  fields: {
-    order_id: { source: 'ref', required: true, resolver: 'sdk.trim' },
-    order_date: { source: 'createdOn', required: true, resolver: 'sdk.formatDateShort' },
-
-    // Custom resolver: Generate 3PL reference number
-    partner_order_ref: {
-      source: 'ref',
-      resolver: 'custom.generate3PLReference',
-    },
-
-    // Custom resolver: Calculate order total
-    order_total: {
-      source: 'items',
-      resolver: 'custom.calculateOrderTotal',
-    },
-
-    // Custom resolver: Map shipping service level
-    shipping_service_code: {
-      source: 'shippingMethod',
-      resolver: 'custom.mapShippingService',
-    },
-
-    // Custom resolver: Format phone number for 3PL
-    ship_to_phone: {
-      source: 'deliveryAddress.phone',
-      resolver: 'custom.formatPhone',
-    },
-  },
-};
-
-// Custom resolver implementations
-const customResolvers = {
-  'custom.generate3PLReference': (orderRef: string) => {
-    // Format: 3PL-YYYYMMDD-ORDERREF
-    const today = new Date().toISOString().slice(0, 10).replace(/-/g, '');
-    return `3PL-${today}-${orderRef}`;
-  },
-
-  'custom.calculateOrderTotal': (items: any[]) => {
-    const total = items.reduce((sum, item) => {
-      const qty = parseFloat(item.quantity) || 0;
-      const price = parseFloat(item.price) || 0;
-      return sum + qty * price;
-    }, 0);
-    return total.toFixed(2);
-  },
-
-  'custom.mapShippingService': (method: string) => {
-    const serviceMap: Record<string, string> = {
-      STANDARD: 'GROUND',
-      EXPRESS: '2DAY',
-      OVERNIGHT: 'NEXTDAY',
-      INTERNATIONAL: 'INTL',
-    };
-    return serviceMap[method] || 'GROUND';
-  },
-
-  'custom.formatPhone': (phone: string) => {
-    if (!phone) return '';
-    // Remove all non-digits
-    const digits = phone.replace(/\D/g, '');
-    // Format as (XXX) XXX-XXXX
-    if (digits.length === 10) {
-      return `(${digits.slice(0, 3)}) ${digits.slice(3, 6)}-${digits.slice(6)}`;
-    }
-    return phone;
-  },
-};
-
-// Use with UniversalMapper
-const mapper = new UniversalMapper(ordersExportMapping, { customResolvers });
-```
-
-### Available SDK Resolvers
-
-**String Transformations:**
-
-- `sdk.trim` - Remove whitespace
-- `sdk.uppercase` - Convert to uppercase
-- `sdk.lowercase` - Convert to lowercase
-- `sdk.toString` - Convert to string
-
-**Number Transformations:**
-
-- `sdk.parseInt` - Parse integer
-- `sdk.parseFloat` - Parse decimal
-- `sdk.number` - Generic number conversion
-
-**Date Transformations:**
-
-- `sdk.formatDate` - ISO 8601 format (`2025-01-22T14:30:00Z`)
-- `sdk.formatDateShort` - Short date format (`2025-01-22`)
-- `sdk.parseDate` - Parse date string
-
-**Type Conversions:**
-
-- `sdk.boolean` - Convert to boolean
-- `sdk.parseJson` - Parse JSON string
-- `sdk.toJson` - Convert to JSON string
-
-**Utility:**
-
-- `sdk.identity` - Pass through unchanged
-- `sdk.coalesce` - Return first non-null value
-
-See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
-
-## GraphQL Query
-
-```graphql
-query GetOrders($retailerId: ID!, $updatedAfter: DateTime!, $first: Int!, $after: String) {
-  orders(
-    retailerId: $retailerId
-    updatedOn: { after: $updatedAfter }
-    first: $first
-    after: $after
-  ) {
-    edges {
-      node {
-        id
-        ref
-        status
-        createdOn
-        updatedOn
-        customer {
-          firstName
-          lastName
-          email
-        }
-        deliveryAddress {
-          name
-          street1
-          street2
-          city
-          state
-          postcode
-          country
-        }
-        items {
-          id
-          quantity
-          price
-          product {
-            ref
-            name
-          }
-        }
-      }
-      cursor
-    }
-    pageInfo {
-      hasNextPage
-    }
-  }
-}
-```
-
-## Expected XML Output
-
-**IMPORTANT**: XML structure with same fields as CSV version for consistency.
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<Orders>
-  <Order>
-    <OrderHeader>
-      <order_id>ORD-001</order_id>
-      <order_date>2025-01-22</order_date>
-      <order_status>CREATED</order_status>
-      <customer_name>John</customer_name>
-      <customer_email>john@example.com</customer_email>
-    </OrderHeader>
-    <ShipTo>
-      <ship_to_name>John Smith</ship_to_name>
-      <ship_to_address1>123 Main St</ship_to_address1>
-      <ship_to_city>New York</ship_to_city>
-      <ship_to_state>NY</ship_to_state>
-      <ship_to_zip>10001</ship_to_zip>
-      <ship_to_country>US</ship_to_country>
-    </ShipTo>
-    <LineItems>
-      <LineItem>
-        <line_item_sku>SKU-001</line_item_sku>
-        <line_item_qty>2</line_item_qty>
-        <line_item_price>29.99</line_item_price>
-      </LineItem>
-      <LineItem>
-        <line_item_sku>SKU-002</line_item_sku>
-        <line_item_qty>1</line_item_qty>
-        <line_item_price>49.99</line_item_price>
-      </LineItem>
-    </LineItems>
-  </Order>
-  <Order>
-    <OrderHeader>
-      <order_id>ORD-002</order_id>
-      <order_date>2025-01-22</order_date>
-      <order_status>PAID</order_status>
-      <customer_name>Jane</customer_name>
-      <customer_email>jane@example.com</customer_email>
-    </OrderHeader>
-    <ShipTo>
-      <ship_to_name>Jane Doe</ship_to_name>
-      <ship_to_address1>456 Oak Ave</ship_to_address1>
-      <ship_to_city>Los Angeles</ship_to_city>
-      <ship_to_state>CA</ship_to_state>
-      <ship_to_zip>90001</ship_to_zip>
-      <ship_to_country>US</ship_to_country>
-    </ShipTo>
-    <LineItems>
-      <LineItem>
-        <line_item_sku>SKU-003</line_item_sku>
-        <line_item_qty>1</line_item_qty>
-        <line_item_price>19.99</line_item_price>
-      </LineItem>
-    </LineItems>
-  </Order>
-</Orders>
-```
-
-**Note**: XML preserves hierarchical structure (Order → LineItems) unlike CSV which flattens to rows.
-
----
-
-## Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-orders-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-orders-extraction.ts  # Scheduled: Daily orders extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-orders-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts         # Webhook: Status query
-    │
-    ├── services/
-    │   └── orders-extraction.service.ts   # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── orders.export.xml.json         # Mapping configuration
-```
-
-**Why This Structure:**
-- ✅ Easier to maintain (each workflow in its own file)
-- ✅ Clear separation of concerns (scheduled vs webhook)
-- ✅ Better for team collaboration (fewer merge conflicts)
-- ✅ Follows Versori best practices
-
----
-
-### Overview
-
-Even with **incremental-only** extraction, order data needs safeguards to prevent runtime failures:
-
-- **Nested data**: Orders contain line items, addresses, customers (1 order → 5-20 line items)
-- **XML generation**: More memory-intensive than CSV generation
-- **Time-critical**: 3PL/fulfillment systems need reliable, timely feeds
-- **High priority**: Order processing delays directly impact customers
-
-### Hard Limits
-
-```typescript
-const SAFETY_LIMITS = {
-  MAX_ORDERS_PER_RUN: 50000, // 50k orders per run
-  MAX_XML_ELEMENTS: 500000, // 500k XML elements total
-  MAX_RECORDS_PER_FILE: 10000, // 10k orders per XML file (SFTP-friendly)
-  MAX_FILE_SIZE_MB: 100, // 100MB per file
-  MAX_XML_SIZE_MB: 200, // Total extraction size
-  CHUNK_SIZE: 5000, // Process in chunks
-  AVG_LINE_ITEMS_PER_ORDER: 3, // Conservative estimate
-  ESTIMATED_BYTES_PER_ORDER_XML: 2000, // XML with full structure
-};
-```
-
-**Why XML needs special consideration?**
-
-- **XML overhead**: Tags and structure add 2-3x size vs CSV
-- **Memory during generation**: Building XML tree in memory
-- **SFTP partners**: Often have stricter file size limits than S3
-- **Validation**: 3PL systems validate XML against schemas
-
-### Runtime Validation Function
-
-```typescript
-/**
- * Validate extraction safety limits before processing
- * CRITICAL: Account for XML size overhead vs CSV
- */
-function validateExtractionLimits(orderCount: number, totalLineItems: number) {
-  const MAX_ORDERS_PER_RUN = 50000;
-  const MAX_XML_ELEMENTS = 500000; // orders + line items + structure
-  const ESTIMATED_BYTES_PER_ORDER_XML = 2000; // Full XML order element
-  const estimatedSizeMB = (orderCount * ESTIMATED_BYTES_PER_ORDER_XML) / (1024 * 1024);
-  const MAX_XML_SIZE_MB = 200;
-
-  if (orderCount > MAX_ORDERS_PER_RUN) {
-    return {
-      valid: false,
-      error: `Extraction limit exceeded: ${orderCount} orders (max: ${MAX_ORDERS_PER_RUN})`,
-      recommendation: `Too many orders for single extraction. Consider:
-        1. Increase extraction frequency (daily → hourly)
-        2. Add order status filters (NEW, PAID only)
-        3. Split by fulfillment location
-        4. Contact support if consistently exceeding limits`,
-      orderCount,
-      maxAllowed: MAX_ORDERS_PER_RUN,
-    };
-  }
-
-  const totalElements = orderCount + totalLineItems + orderCount * 2; // headers + shipping
-  if (totalElements > MAX_XML_ELEMENTS) {
-    return {
-      valid: false,
-      error: `XML element limit exceeded: ${totalElements} elements (max: ${MAX_XML_ELEMENTS})`,
-      recommendation: `XML structure too large. Orders: ${orderCount}, Line items: ${totalLineItems}. Increase extraction frequency.`,
-      totalElements,
-      maxAllowed: MAX_XML_ELEMENTS,
-    };
-  }
-
-  if (estimatedSizeMB > MAX_XML_SIZE_MB) {
-    return {
-      valid: false,
-      error: `XML size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_XML_SIZE_MB}MB)`,
-      recommendation:
-        'File splitting required. Increase extraction frequency to reduce batch size.',
-      estimatedSizeMB,
-      maxAllowed: MAX_XML_SIZE_MB,
-    };
-  }
-
-  return { valid: true };
-}
-```
-
-### Memory-Safe XML Generation with XMLBuilder
-
-The SDK's `XMLBuilder` handles XML generation efficiently with automatic escaping:
-
-```typescript
-import { Buffer } from 'node:buffer';
-import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize XMLBuilder (reusable)
-const xmlBuilder = new XMLBuilder({
-  rootElement: 'Orders',
-  prettyPrint: true,
-  indent: '  ',
-  xmlDeclaration: true,
-  encoding: 'UTF-8',
-});
-
-/**
- * Generate XML from orders using XMLBuilder
- * Handles nested structures (OrderHeader, ShipTo, LineItems) automatically
- */
-function buildOrdersXML(orders: any[]): string {
-  // Transform to XMLBuilder format with nested structures
-  const ordersForXml = orders.map(order => ({
-    OrderHeader: {
-      order_id: order.order_id,
-      order_date: order.order_date,
-      order_status: order.order_status,
-      customer_name: order.customer_name || '',
-      customer_email: order.customer_email || '',
-    },
-    ShipTo: {
-      ship_to_name: order.ship_to_name,
-      ship_to_address1: order.ship_to_address1,
-      ship_to_city: order.ship_to_city,
-      ship_to_state: order.ship_to_state,
-      ship_to_zip: order.ship_to_zip,
-      ship_to_country: order.ship_to_country,
-    },
-    LineItems: {
-      LineItem: order.line_items.map(item => ({
-        line_item_sku: item.line_item_sku,
-        line_item_qty: item.line_item_qty,
-        line_item_price: item.line_item_price,
-      })),
-    },
-  }));
-
-  // XMLBuilder handles all escaping and structure automatically
-  return xmlBuilder.build({ Order: ordersForXml });
-}
-
-// Example output:
-// <Orders>
-//   <Order>
-//     <OrderHeader>
-//       <order_id>ORD-001</order_id>
-//       <customer_name>Smith &amp; Jones</customer_name>
-//     </OrderHeader>
-//     <ShipTo>
-//       <ship_to_name>John Smith</ship_to_name>
-//       <ship_to_address1>123 Main St</ship_to_address1>
-//     </ShipTo>
-//     <LineItems>
-//       <LineItem>
-//         <line_item_sku>SKU-001</line_item_sku>
-//         <line_item_qty>2</line_item_qty>
-//       </LineItem>
-//     </LineItems>
-//   </Order>
-// </Orders>
-```
-
-**Benefits:**
-
-- ✅ Automatic XML escaping (handles &, <, >, ", ', etc.)
-- ✅ Nested structure support (OrderHeader, ShipTo, LineItems)
-- ✅ Array handling (multiple LineItem elements)
-- ✅ Memory-efficient streaming
-- ✅ Pretty printing with proper indentation
-
-## Complete Workflow Code
-
-The following code examples demonstrate the implementation across separate files following the recommended modular structure.
-
-
-### 1. Entry Point (src/index.ts)
-
-```typescript
-/**
- * Entry point - Export all workflows for Versori platform
- *
- * This file exports all workflows to be registered with Versori.
- * Each workflow is defined in its own file for better organization.
- */
-
-// Scheduled workflows
-export { scheduledOrdersExtraction } from './workflows/scheduled/daily-orders-extraction';
-
-// Webhook workflows
-export { adhocOrdersExtraction } from './workflows/webhook/adhoc-orders-extraction';
-export { ordersJobStatus } from './workflows/webhook/job-status-check';
-```
-
-### 2. Scheduled Workflow (src/workflows/scheduled/daily-orders-extraction.ts)
-
-```typescript
-import { schedule, http } from '@versori/run';
-import {
-  executeOrderExtraction,
-  generateJobId,
-} from '../../services/extraction-orchestration';
-
-/**
- * Scheduled workflow: Daily orders extraction to SFTP XML
- * Runs at 2:00 AM daily
- *
- * DELEGATION PATTERN:
- * - Workflow receives ctx from Versori
- * - Passes entire ctx to service function
- * - Service handles all business logic
- */
-export const scheduledOrdersExtraction = schedule('orders-extract-xml-daily', '0 2 * * *').then(
-  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = generateJobId('SCHED', 'ORDERS');
-
-    log.info('🚀 [WORKFLOW] Starting scheduled extraction', { jobId });
-
-    const result = await executeOrderExtraction(ctx, {
-      jobId,
-      triggeredBy: 'schedule',
-      updateState: true, // Always update state for scheduled runs
-    });
-
-    const duration = Date.now() - executionStartTime;
-
-    if (result.success) {
-      log.info('✅ [WORKFLOW] Extraction completed successfully', {
-        jobId,
-        ordersExtracted: result.ordersExtracted,
-        duration: `${duration}ms`,
-      });
-    } else {
-      log.error('❌ [WORKFLOW] Extraction failed', {
-        jobId,
-        error: result.error,
-        duration: `${duration}ms`,
-      });
-    }
-
-    return { ...result, duration };
-  })
-);
-```
-
-### 3. Ad-hoc Webhook (src/workflows/webhook/adhoc-orders-extraction.ts)
-
-```typescript
-import { webhook, http } from '@versori/run';
-import {
-  executeOrderExtraction,
-  generateJobId,
-} from '../../services/extraction-orchestration';
-
-/**
- * Webhook workflow: Ad-hoc orders extraction
- * Allows manual/on-demand extraction with date range overrides
- */
-export const adhocOrdersExtraction = webhook('orders-adhoc', {
-  connection: 'orders-adhoc',
-  response: { mode: 'sync' },
-}).then(
-  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = generateJobId('ADHOC', 'ORDERS');
-
-    log.info('🔧 [WEBHOOK] Starting ad-hoc extraction', {
-      jobId,
-      fromDate: ctx.data.fromDate,
-      toDate: ctx.data.toDate
-    });
-
-    const result = await executeOrderExtraction(ctx, {
-      jobId,
-      triggeredBy: 'webhook',
-      fromDate: ctx.data.fromDate,
-      toDate: ctx.data.toDate,
-      updateState: ctx.data.updateState !== false,
-    });
-
-    const duration = Date.now() - executionStartTime;
-
-    if (result.success) {
-      log.info('✅ [WEBHOOK] Ad-hoc extraction completed', {
-        jobId,
-        ordersExtracted: result.ordersExtracted,
-        duration: `${duration}ms`,
-      });
-    } else {
-      log.error('❌ [WEBHOOK] Ad-hoc extraction failed', {
-        jobId,
-        error: result.error,
-        duration: `${duration}ms`,
-      });
-    }
-
-    return { ...result, duration };
-  })
-);
-```
-
-### 4. Job Status Query (src/workflows/webhook/job-status-check.ts)
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { getJobStatus } from '../../services/extraction-orchestration';
-
-/**
- * Webhook workflow: Job status query
- * Allows checking the status of extraction jobs by ID
- */
-export const ordersJobStatus = webhook('orders-job-status', {
-  connection: 'orders-job-status',
-  response: { mode: 'sync' },
-}).then(
-  fn('query-job-status', async (ctx: any) => {
-    const { data, log, openKv } = ctx;
-    // Security is enforced by the 'orders-job-status' connection
-
-    log.info('🔍 [STATUS] Querying job status', { jobId: data.jobId });
-
-    const jobId = data.jobId;
-    if (!jobId) {
-      log.error('❌ [STATUS] Missing job ID');
-      return { success: false, error: 'Job ID required' };
-    }
-
-    const status = await getJobStatus(openKv(':project:'), jobId, log);
-
-    if (status) {
-      log.info('✅ [STATUS] Job found', { jobId, status: status.status });
-      return { success: true, jobId, ...status };
-    } else {
-      log.warn('⚠️ [STATUS] Job not found', { jobId });
-      return { success: false, error: 'Job not found', jobId };
-    }
-  })
-);
-```
-
-### 5. Main Orchestration Service (src/services/extraction-orchestration.ts)
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-import ordersExportMapping from '../../config/orders.export.xml.json' with { type: 'json' };
-
-const ORDERS_QUERY = `
-  query GetOrders($retailerId: ID!, $updatedAfter: DateTime!, $first: Int!, $after: String) {
-    orders(
-      retailerId: $retailerId
-      updatedOn: { after: $updatedAfter }
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          status
-          createdOn
-          updatedOn
-          customer {
-            firstName
-            lastName
-            email
-          }
-          deliveryAddress {
-            name
-            street1
-            street2
-            city
-            state
-            postcode
-            country
-          }
-          items {
-            id
-            quantity
-            price
-            product {
-              ref
-              name
-            }
-          }
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-
-// Initialize XMLBuilder for orders
-const xmlBuilder = new XMLBuilder({
-  rootElement: 'Orders',
-  prettyPrint: true,
-  indent: '  ',
-  xmlDeclaration: true,
-  encoding: 'UTF-8',
-});
-
-function buildOrdersXML(orders: any[]): string {
-  // Transform to XMLBuilder format with nested structures
-  const ordersForXml = orders.map(order => ({
-    OrderHeader: {
-      order_id: order.order_id,
-      order_date: order.order_date,
-      order_status: order.order_status,
-      customer_name: order.customer_name || '',
-      customer_email: order.customer_email || '',
-    },
-    ShipTo: {
-      ship_to_name: order.ship_to_name,
-      ship_to_address1: order.ship_to_address1,
-      ship_to_city: order.ship_to_city,
-      ship_to_state: order.ship_to_state,
-      ship_to_zip: order.ship_to_zip,
-      ship_to_country: order.ship_to_country,
-    },
-    LineItems: {
-      LineItem: order.line_items.map(item => ({
-        line_item_sku: item.line_item_sku,
-        line_item_qty: item.line_item_qty,
-        line_item_price: item.line_item_price,
-      })),
-    },
-  }));
-
-  return xmlBuilder.build({ Order: ordersForXml });
-}
-
-interface ExtractionOptions {
-  jobId: string;
-  triggeredBy: 'schedule' | 'webhook';
-  fromDate?: string;
-  toDate?: string;
-  updateState: boolean;
-}
-
-export async function executeOrderExtraction(ctx: any, options: ExtractionOptions) {
-  const { jobId, triggeredBy, fromDate, toDate, updateState } = options;
-  const log = ctx.log;
-  const retailerId = ctx.activation?.getVariable('retailerId');
-  const pageSize = parseInt(ctx.activation?.getVariable('pageSize') || '200', 10);
-  const maxRecords = parseInt(ctx.activation?.getVariable('maxRecords') || '10000', 10);
-  const fallbackStartDate =
-    ctx.activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
-
-  // Get SFTP credentials from Versori connection (Basic Auth)
-  // RECOMMENDED: Use activation.connections (already decoded)
-  const allConnections = ctx.activation.connections || [];
-  const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-  if (!sftpConn) {
-    throw new Error('SFTP connection "versori_ftp_server" not found');
-  }
-
-  const credential = sftpConn.credentials[0]?.credential;
-  if (!credential?.data?.basicAuth) {
-    throw new Error('SFTP connection not configured with Basic Authentication');
-  }
-
-  const { username, password } = credential.data.basicAuth;
-  // ✅ Already decoded - no Buffer.from() needed!
-
-  const sftpSettings = {
-    host: ctx.activation?.getVariable('sftpHost'),
-    port: parseInt(ctx.activation?.getVariable('sftpPort') || '22', 10),
-    username, // From connection (secure)
-    password, // From connection (secure)
-    privateKey: ctx.activation?.getVariable('sftpPrivateKey'),
-    remotePath: ctx.activation?.getVariable('sftpRemotePath') || '/incoming/orders/',
-  };
-
-  const missing: string[] = [];
-  if (!retailerId) missing.push('retailerId');
-  if (!sftpSettings.host) missing.push('sftpHost');
-  if (!sftpSettings.username) missing.push('sftpUsername from connection');
-  if (!sftpSettings.password && !sftpSettings.privateKey)
-    missing.push('sftpPassword from connection or sftpPrivateKey');
-  if (missing.length)
-    return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
-
-  // SFTP connection - MUST use try/finally with dispose()
-  const sftp = new SftpDataSource(
-    {
-      type: 'SFTP_XML',
-      connectionId: 'sftp-orders-xml-export',
-      name: 'SFTP Orders XML Export',
-      settings: {
-        host: sftpSettings.host,
-        port: sftpSettings.port,
-        username: sftpSettings.username,
-        password: sftpSettings.password,
-        privateKey: sftpSettings.privateKey,
-        remotePath: sftpSettings.remotePath,
-        filePattern: '*.xml',
-      },
-    },
-    log
-  );
-
-  try {
-    // 
-    // STEP 1/8: Initialize Job Tracking
-    // 
-    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
-    const tracker = new JobTracker(kv, log);
-
-    await tracker.createJob(jobId, {
-      triggeredBy,
-      hasDateOverride: !!fromDate,
-      fromDate,
-      toDate,
-      updateStateAfterRun: updateState,
-    });
-
-    log.info('Job created', { jobId, triggeredBy });
-
-    // 
-    // STEP 2/8: Load State & Calculate Time Window
-    // 
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'state_load',
-      message: 'Loading last run state',
-    });
-
-    const stateKey = ['extraction', 'orders-xml', 'lastRunTime'];
-    const lastRunState = await kv.get(stateKey);
-    const rawLastRunTime = fromDate || lastRunState?.value?.timestamp || fallbackStartDate;
-
-    // Overlap buffer configuration (default: 60 seconds)
-    const overlapBufferSeconds = parseInt(
-      ctx.activation?.getVariable('overlapBufferSeconds') || '60',
-      10
-    );
-    const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
-
-    // Apply overlap buffer for query (safety window)
-    const bufferedLastRunTime = new Date(
-      new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
-    ).toISOString();
-
-    const effectiveEndTime = toDate || new Date().toISOString();
-
-    log.info('Time window calculated', {
-      rawLastRunTime,
-      bufferedLastRunTime,
-      effectiveEndTime,
-      overlapBufferSeconds,
-      retailerId,
-    });
-
-    // 
-    // STEP 3/8: Initialize Fluent Client & ExtractionOrchestrator
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'client_init',
-      message: 'Initializing Fluent client',
-    });
-
-    const client = await createClient(ctx);
-    const orchestrator = new ExtractionOrchestrator(client, log);
-
-    // 
-    // STEP 4/8: Extract Data (ExtractionOrchestrator)
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'extraction',
-      message: 'Extracting data with auto-pagination',
-    });
-
-    // ? Enhanced: Extract context for progress logging
-    const dateRangeInfo = {
-      start: bufferedLastRunTime,
-      end: effectiveEndTime,
-      retailerId
-    };
-
-    // ? Enhanced: Start logging with context
-    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
-     query: 'orders',
-     pageSize,
-     maxRecords,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     retailerId: dateRangeInfo.retailerId,
-     jobId
-    });
-
-    const extractionResult = await orchestrator.extract({
-      query: ORDERS_QUERY,
-      resultPath: 'orders.edges.node',
-      variables: {
-        retailerId,
-        updatedAfter: bufferedLastRunTime,
-        first: pageSize,
-      },
-      pageSize,
-      maxRecords,
-      validateItem: item => !!(item.ref && item.customer),
-    });
-
-    const rawRecords = extractionResult.data;
-
-    log.info('Extraction complete', {
-      totalRecords: extractionResult.stats.totalRecords,
-      totalPages: extractionResult.stats.totalPages,
-      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-      errors: extractionResult.errors ? extractionResult.errors.length : 0,
-    });
-
-    // ? Enhanced: Completion logging with summary
-    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-     totalRecords: extractionResult.stats.totalRecords,
-     totalPages: extractionResult.stats.totalPages,
-     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-     failedValidations: extractionResult.stats.failedValidations,
-     truncated: extractionResult.stats.truncated,
-     truncationReason: extractionResult.stats.truncationReason,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     jobId
-    });
-
-    if (extractionResult.errors && extractionResult.errors.length > 0) {
-      log.warn('Non-fatal extraction errors encountered', {
-        errorCount: extractionResult.errors.length,
-        sampleErrors: extractionResult.errors.slice(0, 3),
-      });
-    }
-
-    if (rawRecords.length === 0) {
-      await tracker.markCompleted(jobId, {
-        recordCount: 0,
-        message: 'No new orders to extract',
-      });
-
-      if (updateState) {
-        await kv.set(stateKey, {
-          timestamp: new Date().toISOString(),
-          orderCount: 0,
-          extractedAt: new Date().toISOString(),
-        });
-      }
-
-      return { success: true, message: 'No new orders to extract', lastRunTime: rawLastRunTime };
-    }
-
-    // 
-    // STEP 5/8: Validate Extraction Limits
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'validation',
-      message: 'Validating extraction limits',
-    });
-
-    const MAX_ORDERS_PER_RUN = 50000;
-    const MAX_XML_ELEMENTS = 500000;
-
-    let totalLineItems = 0;
-    for (const order of rawRecords) {
-      totalLineItems += (order.items || []).length;
-    }
-
-    const totalElements = rawRecords.length + totalLineItems + rawRecords.length * 2; // orders + items + headers/shipping
-    const ESTIMATED_BYTES_PER_ORDER_XML = 2000;
-    const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_ORDER_XML) / (1024 * 1024);
-    const MAX_XML_SIZE_MB = 200;
-
-    if (rawRecords.length > MAX_ORDERS_PER_RUN) {
-      log.error('Extraction limit exceeded', {
-        orderCount: rawRecords.length,
-        maxAllowed: MAX_ORDERS_PER_RUN,
-      });
-
-      await tracker.markFailed(jobId, {
-        error: `Extraction limit exceeded: ${rawRecords.length} orders (max: ${MAX_ORDERS_PER_RUN})`,
-        recommendation: 'Increase extraction frequency or add filters',
-      });
-
-      return {
-        success: false,
-        error: `Extraction limit exceeded: ${rawRecords.length} orders (max: ${MAX_ORDERS_PER_RUN})`,
-        recommendation: `Too many orders for single extraction. Consider:
-          1. Increase extraction frequency (daily → hourly)
-          2. Add order status filters (NEW, PAID only)
-          3. Split by fulfillment location
-          4. Contact support if consistently exceeding limits`,
-        orderCount: rawRecords.length,
-        maxAllowed: MAX_ORDERS_PER_RUN,
-      };
-    }
-
-    if (totalElements > MAX_XML_ELEMENTS) {
-      log.error('XML element limit exceeded', {
-        orderCount: rawRecords.length,
-        totalLineItems,
-        totalElements,
-        maxAllowed: MAX_XML_ELEMENTS,
-      });
-
-      await tracker.markFailed(jobId, {
-        error: `XML element limit exceeded: ${totalElements} elements`,
-        recommendation: 'Increase extraction frequency',
-      });
-
-      return {
-        success: false,
-        error: `XML element limit exceeded: ${totalElements} elements (max: ${MAX_XML_ELEMENTS})`,
-        recommendation: `XML structure too large. Orders: ${rawRecords.length}, Line items: ${totalLineItems}. Increase extraction frequency.`,
-        totalElements,
-        maxAllowed: MAX_XML_ELEMENTS,
-      };
-    }
-
-    if (estimatedSizeMB > MAX_XML_SIZE_MB) {
-      log.warn('XML size approaching limit', {
-        estimatedSizeMB: estimatedSizeMB.toFixed(2),
-        maxAllowed: MAX_XML_SIZE_MB,
-        recommendation: 'Consider file splitting or increase extraction frequency',
-      });
-    }
-
-    log.info('Extraction limits validated', {
-      orderCount: rawRecords.length,
-      totalLineItems,
-      totalElements,
-      estimatedSizeMB: estimatedSizeMB.toFixed(2),
-      withinLimits: true,
-    });
-
-    await tracker.updateJob(jobId, {
-      stage: 'transformation',
-      message: 'Transforming data with UniversalMapper',
-    });
-
-    const mapper = new UniversalMapper(ordersExportMapping);
-    const transformedOrders: any[] = [];
-    const mappingErrors: any[] = [];
-
-    for (let index = 0; index < rawRecords.length; index += 1) {
-      const order = rawRecords[index];
-      const headerResult = await mapper.map(order);
-
-      if (!headerResult.success || !headerResult.data) {
-        mappingErrors.push({
-          orderRef: order.ref,
-          errors: headerResult.errors || ['Mapping failed'],
-        });
-        continue;
-      }
-
-      const header = headerResult.data as Record<string, any>;
-      const lineItems = (order.items || []).map(item => ({
-        line_item_sku: String(item.product?.ref ?? '').trim(),
-        line_item_qty: Number.parseInt(String(item.quantity ?? '0'), 10) || 0,
-        line_item_price: Number.parseFloat(String(item.price ?? '0')) || 0,
-      }));
-
-      transformedOrders.push({
-        order_id: header.order_id,
-        order_date: header.order_date,
-        order_status: header.order_status,
-        customer_name: header.customer_name,
-        customer_email: header.customer_email,
-        ship_to_name: header.ship_to_name,
-        ship_to_address1: header.ship_to_address1,
-        ship_to_city: header.ship_to_city,
-        ship_to_state: header.ship_to_state,
-        ship_to_zip: header.ship_to_zip,
-        ship_to_country: header.ship_to_country,
-        updated_on: header.updated_on || order.updatedOn,
-        line_items: lineItems,
-      });
-    }
-
-    if (mappingErrors.length > 0) {
-      log.warn('Some orders failed transformation', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-    }
-
-    if (transformedOrders.length === 0) {
-      await tracker.markFailed(jobId, {
-        error: 'All records failed mapping',
-        failedCount: mappingErrors.length,
-      });
-      return {
-        success: false,
-        error: 'All records failed mapping',
-        errors: mappingErrors,
-      };
-    }
-
-    log.info('Orders transformed', {
-      jobId,
-      transformedCount: transformedOrders.length,
-      skippedRecords: rawRecords.length - transformedOrders.length,
-    });
-
-    await tracker.updateJob(jobId, {
-      stage: 'upload',
-      message: 'Generating XML and uploading to SFTP',
-    });
-
-    const xmlContent = buildOrdersXML(transformedOrders);
-
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const fileName = `orders-${timestamp}.xml`;
-    const remotePath = `${sftpSettings.remotePath}${fileName}`;
-
-    log.info('Generated XML file', {
-      fileName,
-      size: xmlContent.length,
-      orderCount: transformedOrders.length,
-      lineItemCount: totalLineItems,
-    });
-
-    await sftp.uploadFile(remotePath, Buffer.from(xmlContent, 'utf8'));
-
-    log.info('XML file uploaded to SFTP', { remotePath });
-
-    await tracker.updateJob(jobId, {
-      stage: 'state_update',
-      message: 'Updating state and completing job',
-    });
-
-    const maxUpdatedOn = transformedOrders.reduce((max, order) => {
-      const orderTime = new Date(order.updated_on).getTime();
-      return orderTime > max ? orderTime : max;
-    }, new Date(rawLastRunTime).getTime());
-
-    const newTimestamp = new Date(maxUpdatedOn).toISOString();
-
-    if (updateState) {
-      await kv.set(stateKey, {
-        timestamp: newTimestamp,
-        orderCount: transformedOrders.length,
-        lineItemCount: totalLineItems,
-        extractedAt: new Date().toISOString(),
-        overlapBufferSeconds,
-        fileName,
-        remotePath,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      });
-
-      log.info('State updated with new timestamp (without buffer)', {
-        newTimestamp,
-        overlapBufferSeconds,
-      });
-    }
-
-    await tracker.markCompleted(jobId, {
-      recordCount: transformedOrders.length,
-      fileName,
-      sftpPath: remotePath,
-      errorCount: mappingErrors.length,
-      errors: mappingErrors,
-    });
-
-    return {
-      success: true,
-      ordersExtracted: transformedOrders.length,
-      lineItemsExtracted: totalLineItems,
-      fileName,
-      remotePath,
-      lastRunTime: rawLastRunTime,
-      newTimestamp,
-      jobId,
-      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-    };
-  } catch (error: any) {
-    log.error('Extraction failed', error, {
-      message: error?.message,
-    });
-
-    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
-    const tracker = new JobTracker(kv, log);
-
-    await tracker.markFailed(jobId, {
-      message: error instanceof Error ? error.message : String(error),
-
-      stack: error instanceof Error ? error.stack : undefined,
-
-      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-    });
-
-    return {
-      success: false,
-      message: error instanceof Error ? error.message : String(error),
-
-      stack: error instanceof Error ? error.stack : undefined,
-
-      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-      jobId,
-    };
-  } finally {
-    // CRITICAL: Always clean up SFTP connections
-    await sftp.dispose();
-    log.info('SFTP connection disposed');
-  }
-}
-
-export async function getJobStatus(kv: any, jobId: string, log: any) {
-  const tracker = new JobTracker(new VersoriKVAdapter(kv), log);
-  return await tracker.getJob(jobId);
-}
-```
-
-### 6. Job ID Generator (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Generate unique job ID
- * Format: {PREFIX}-{ENTITY}-{TIMESTAMP}
- */
-export function generateJobId(prefix: 'SCHED' | 'ADHOC', entity: string): string {
-  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-  return `${prefix}-${entity}-${timestamp}`;
-}
-```
-
-### 7. Package Configuration (package.json)
-
-```json
-{
-  "name": "orders-extraction-to-sftp-xml",
-  "version": "1.0.0",
-  "description": "Versori connector for orders extraction to SFTP XML",
-  "main": "dist/index.js",
-  "type": "module",
-  "scripts": {
-    "build": "tsc",
-    "dev": "tsc --watch",
-    "lint": "eslint src/**/*.ts",
-    "test": "jest"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-
-### 8. Deployment Instructions
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Build the connector
-npm run build
-
-# 3. Test locally (optional)
-npm test
-
-# 4. Deploy to Versori
-# - Upload to Versori workspace
-# - Configure activation variables
-# - Enable workflows
-
-# 5. Test workflows
-# Scheduled: Wait for next cron trigger or manually trigger
-# Ad-hoc: POST to webhook URL with API key header
-# Status: Query job status by ID
-```
-
-### 9. Testing
-
-#### Test Scheduled Extraction
-
-```bash
-# Trigger manually in Versori UI or wait for cron schedule
-# Expected: XML file uploaded to SFTP
-```
-
-#### Test Ad-hoc Extraction
-
-```bash
-curl -X POST https://your-workspace.versori.run/orders-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-22T23:59:59Z",
-    "updateState": false
-  }'
-```
-
-#### Test Job Status Query
-
-```bash
-curl -X POST https://your-workspace.versori.run/orders-job-status \
-  -H "Content-Type: application/json" \
-  -d '{
-    "jobId": "SCHED-ORDERS-2025-01-22T02-00-00Z"
-  }'
-```
-
-## Key Patterns Explained
-
-### Pattern 1: ExtractionOrchestrator for Auto-Pagination
-
-```typescript
-// ✅ CORRECT - Use ExtractionOrchestrator (handles pagination automatically)
-const orchestrator = new ExtractionOrchestrator(client, log);
-
-const extractionResult = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: { retailerId, updatedAfter: bufferedLastRunTime },
-  pageSize,
-  maxRecords,
-  validateItem: item => !!(item.ref && item.customer),
-});
-
-const rawRecords = extractionResult.data;
-
-//  WRONG - Manual pagination (avoid this pattern)
-// const result = await client.graphql({
-//   query: ORDERS_QUERY,
-//   variables: { first: pageSize },
-//   pagination: { maxRecords }
-// });
-```
-
-### Pattern 2: JobTracker for Lifecycle Management
-
-```typescript
-// ✅ CORRECT - Use JobTracker throughout workflow
-const tracker = new JobTracker(kv, log);
-
-// Create job
-await tracker.createJob(jobId, { triggeredBy, fromDate, toDate });
-
-// Update progress
-await tracker.updateJob(jobId, { stage: 'extraction', message: 'Extracting data' });
-
-// Mark completed
-await tracker.markCompleted(jobId, { recordCount, fileName });
-
-// Query status
-const status = await tracker.getJob(jobId);
-```
-
-### Pattern 3: 3-Workflow Pattern
-
-```typescript
-// ✅ CORRECT - 3 workflows for different use cases
-// 1. Scheduled: Automated daily/hourly runs
-export const scheduledOrdersExtraction = schedule('orders-extract-xml-daily', '0 2 * * *')...
-
-// 2. Ad-hoc: Manual webhook triggers with date overrides
-export const adhocOrdersExtraction = webhook('orders-adhoc', { response: { mode: 'sync' } })...
-
-// 3. Status: Query job status by ID
-export const ordersJobStatus = webhook('orders-job-status', { response: { mode: 'sync' } })...
-```
-
-### Pattern 4: XMLBuilder for Safe XML Generation (CRITICAL)
-
-Use the SDK's `XMLBuilder` - it handles all XML escaping automatically:
-
-```typescript
-import { Buffer } from 'node:buffer';
-import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize XMLBuilder (handles all escaping automatically)
-const xmlBuilder = new XMLBuilder({
-  rootElement: 'Orders',
-  prettyPrint: true,
-  encoding: 'UTF-8',
-});
-
-// ✅ CORRECT: XMLBuilder escapes automatically
-const orders = [
-  {
-    customer_name: 'Smith & Jones <Corp>', // Contains & and <>
-    customer_email: 'contact@smith&jones.com',
-    notes: 'Special chars: ¢, ©, ®, "quotes"',
-  },
-];
-
-const xml = xmlBuilder.build({ Order: orders });
-// Result: All special characters properly escaped
-// <customer_name>Smith &amp; Jones &lt;Corp&gt;</customer_name>
-// <customer_email>contact@smith&amp;jones.com</customer_email>
-// <notes>Special chars: ¢, ©, ®, &quot;quotes&quot;</notes>
-
-//  WRONG: Manual string concatenation (dangerous)
-// const xml = `<customer_name>${order.customer_name}</customer_name>`;
-// This would produce INVALID XML: <customer_name>Smith & Jones <Corp></customer_name>
-```
-
-**Why XMLBuilder?**
-
-- ✅ Automatic escaping of &, <, >, ", '
-- ✅ Handles special characters (¢, ©, ®)
-- ✅ Prevents XML injection attacks
-- ✅ Validates structure
-- ✅ Consistent, maintainable code
-
-### Pattern 5: SFTP Cleanup (CRITICAL)
-
-```typescript
-const sftp = new SftpDataSource(config, log);
-
-try {
-  await sftp.uploadFile(remotePath, buffer);
-  return { success: true };
-} finally {
-  // ALWAYS dispose SFTP connection
-  await sftp.dispose();
-}
-```
-
-**Why?** SFTP maintains open connections. Not calling `dispose()` leads to connection exhaustion.
-
-### Pattern 6: Consistent Field Names Across Formats
-
-**Same data in CSV, JSON, and XML:**
-
-- `order_id` (not orderId, not order-id, not OrderID)
-- `customer_email` (consistent with CSV version)
-- `ship_to_address1` (matches CSV exactly)
-
-This allows users to switch formats without changing downstream systems.
-
-## Common Issues
-
-**Issue 1: Malformed XML from unescaped characters**
-
-- Customer name contains `&` or `<`
-- Solution: Always use XMLBuilder (automatic escaping)
-
-**Issue 2: 3PL system rejects XML**
-
-- Missing required fields
-- Solution: Verify mapping matches 3PL schema requirements
-
-**Issue 3: File too large for SFTP partner**
-
-- Partner has 50MB limit, file is 100MB
-- Solution: Use file splitting (10k orders per file)
-
-**Issue 4: SFTP connection timeouts**
-
-- Not calling `dispose()` in finally block
-- Solution: Always use try/finally pattern
-
-**Issue 5: Job status not updating**
-
-- JobTracker not integrated
-- Solution: Use JobTracker throughout workflow
-
-## Testing
-
-### 1. Test XML Structure
-
-```typescript
-export const testXmlGeneration = http('test-xml').then(
-  fn('test-xml-gen', async () => {
-    const testOrders = [
-      {
-        order_id: 'TEST-001',
-        order_date: '2025-01-22',
-        order_status: 'CREATED',
-        customer_name: 'Test & Validate <Corp>',
-        customer_email: 'test@example.com',
-        ship_to_name: 'John Smith',
-        ship_to_address1: '123 Main St',
-        ship_to_city: 'New York',
-        ship_to_state: 'NY',
-        ship_to_zip: '10001',
-        ship_to_country: 'US',
-        line_items: [{ line_item_sku: 'SKU-001', line_item_qty: 2, line_item_price: 29.99 }],
-      },
-    ];
-
-    const xml = buildOrdersXML(testOrders);
-
-    // Validate XML structure
-    if (!xml.includes('<?xml version="1.0"')) {
-      return { success: false, error: 'Missing XML declaration' };
-    }
-
-    if (!xml.includes('&amp;') || !xml.includes('&lt;')) {
-      return { success: false, error: 'Special characters not escaped' };
-    }
-
-    return { success: true, xml };
-  })
-);
-```
-
-### 2. Test SFTP Upload
-
-```bash
-curl https://your-workspace.versori.run/test-sftp-orders-xml
-```
-
-### 3. Validate Against 3PL Schema
-
-- Download partner's XSD schema
-- Validate generated XML against schema
-- Fix any missing/incorrect elements
-
-## Production Checklist
-
-- [ ] Test SFTP credentials and connection
-- [ ] Verify SFTP server has write permissions to remotePath
-- [ ] Set appropriate extraction frequency (hourly for 3PL feeds)
-- [ ] Configure correct order status filters
-- [ ] Test XML escaping with special characters (&, <, >, ", ')
-- [ ] Validate XML against partner's schema (if provided)
-- [ ] Test `dispose()` is always called (check logs)
-- [ ] Document XML schema for 3PL integration team
-- [ ] Set up monitoring for SFTP connection failures
-- [ ] Test with real order data (addresses with special chars)
-- [ ] Verify file size limits with SFTP partner
-- [ ] Configure SFTP server IP whitelisting for Versori
-- [ ] Test file splitting with large batches (>10k orders)
-- [ ] Test all 3 workflows (scheduled, ad-hoc, status)
-- [ ] Verify JobTracker integration and status updates
-- [ ] Test ExtractionOrchestrator pagination with large datasets
-
-## Troubleshooting Guide
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records
-- **Fix**: Reduce maxRecords, increase frequency
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch
-- **Fix**: Run schema validation, check field names
-
-**Issue**: "State not updating"
-
-- **Cause**: KV write failure or intentional retry
-- **Fix**: Check KV logs, verify state update code
-
-**Issue**: "First run exceeds limits"
-
-- **Cause**: No previous timestamp, fetches all
-- **Fix**: Set fallbackStartDate close to current, apply filters
-
-**Issue**: "Excessive duplicates"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved
-- **Fix**: Verify newTimestamp saved WITHOUT buffer
-
-**Issue**: "Job status returns null"
-
-- **Cause**: Invalid job ID or job expired
-- **Fix**: Verify job ID format, check KV TTL settings
-
-## Security Best Practices
-
-### Credential Management
-
-**✅ DO**:
-
-- Store credentials in Versori activation variables
-- Rotate credentials quarterly
-- Use least-privilege accounts
-
-** DON'T**:
-
-- Never log credentials
-- Never commit to git
-- Never share across environments
-
-### Data Security
-
-- Enable encryption in transit and at rest
-- Apply data retention policies
-- Monitor access logs
-- Use VPC/private networks for sensitive data
-
-### Webhook Security
-
-- Validate API keys for ad-hoc and status workflows
-- Use HTTPS for all webhook endpoints
-- Implement rate limiting
-- Monitor for suspicious activity
-
----
-
-**Pattern**: Enterprise incremental extraction with ExtractionOrchestrator + JobTracker for orders via SFTP (XML format)
-**❌š ï¸ Versori Sample**: Reference implementation - adapt for your production use case
-**Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for lifecycle management, always escape XML and dispose SFTP
-**Critical**: Apply 60-second overlap buffer to prevent missed records
-**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
-**Field Consistency**: Same field names as CSV version for easy format switching
-**SFTP**: Use proper connection cleanup in finally block to prevent connection leaks
-**XML**: Preserve hierarchical structure (no line item flattening needed like CSV)
-**3 Workflows**: Scheduled, ad-hoc webhook, job status query
-
----
-
-### Pattern 8: Backward Pagination (Optional - Advanced)
-
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-
-**When to Use**:
-
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
--  **Don't use for standard incremental sync** - use forward pagination (default)
-
-**GraphQL Query Requirements**:
-
-Your query must support backward pagination by including `$last` and `$before`:
-
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ✅ REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ✅ REQUIRED for backward
-    }
-  }
-}
-```
-
-**Implementation**:
-
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    //  Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ✅ Enable reverse pagination
-  maxRecords: 10000,
-});
-
-// Records are returned in reverse chronological order
-console.log(result.data[0].createdAt); // Newest
-console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
-```
-
-**Key Differences from Forward Pagination**:
-
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest               | Newest → Oldest      |
-
-**Important Notes**:
-
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-
-**Example: Extract Latest 1000 Orders**
-
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-
-**When to Use Forward vs Backward**:
-
-```typescript
-// ✅ Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-
-// ✅ Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-
-**Pagination Variables Reference**:
-
-| Variable | Forward      | Backward     | Injected By  | Notes                    |
-| -------- | ------------ | ------------ | ------------ | ------------------------ |
-| `first`  | ✅ Used      |  Not used | Orchestrator | From `pageSize`          |
-| `after`  | ✅ Used      |  Not used | Orchestrator | From cursor (last edge)  |
-| `last`   |  Not used | ✅ Used      | Orchestrator | From `pageSize`          |
-| `before` |  Not used | ✅ Used      | Orchestrator | From cursor (first edge) |
-
-**Common Mistakes to Avoid**:
-
-```typescript
-//  WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, //  Orchestrator will override this
-    before: cursor, //  Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-
-// ✅ CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ✅ Your business variables only
-  },
-  pageSize: 200, // ✅ Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-
-#### Optional: Reverse Pagination
-
-- Forward remains default. For reverse, require $last/$before and pageInfo.hasPreviousPage.
-- Do not pass last/before in variables; set direction='backward'.
-
-GraphQL:
-
-```graphql
-query GetOrdersBackward($retailerId: ID!, $last: Int!, $before: String) {
-  orders(retailerId: $retailerId, last: $last, before: $before) {
-    edges {
-      cursor
-      node {
-        id
-        ref
-        updatedOn
-      }
-    }
-    pageInfo {
-      hasPreviousPage
-    }
-  }
-}
-```
-
-SDK:
-
-```typescript
-await orchestrator.extract({
-  query: ORDERS_BACKWARD_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: { retailerId },
-  pageSize,
-  direction: 'backward',
-});
-```
-
----
-
-## Testing Checklist
-
-**Before production deployment:**
-
-### 1. Schema Validation
-
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/orders.export.xml.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/orders.export.xml.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-
-### 2. Extraction Testing
-
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-
-### 3. Mapping Testing
-
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-
-### 4. XML Generation Testing
-
-- [ ] Verify XML structure matches expected format
-- [ ] Test XML validation against XSD schema (if applicable)
-- [ ] Verify special character escaping in XML
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test XML namespace handling (if applicable)
-
-### 5. SFTP Upload Testing
-
-- [ ] Test SFTP connection and authentication
-- [ ] Verify file upload to correct path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify file permissions on SFTP server
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify SFTP connection disposal (no connection leaks)
-
-### 6. State Management Testing
-
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-
-### 7. Job Tracking Testing
-
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-
-### 8. Error Handling Testing
-
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with SFTP connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-
-### 9. Staging Environment Testing
-
-- [ ] Run full extraction in staging environment
-- [ ] Verify XML file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-
-### 10. Integration Testing
-
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger → extract → transform → upload → verify file
-
----
-## Monitoring & Alerting
-
-### Success Response Example
-
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_ORD_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "orders-2025-11-02T14-00-00-000Z.xml",
-  "sftpPath": "/outbound/orders/orders-2025-11-02T14-00-00-000Z.xml",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-
-### Error Response Example
-
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_ORD_20251102_140500_xyz789",
-  "error": "SFTP upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "sftp_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-
-### Key Metrics to Track
-
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-
-  // File Generation
-  fileSizeMB: (xmlContent.length / (1024 * 1024)).toFixed(2),
-
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-
-log.info('Extraction metrics', metrics);
-```
-
-### Alert Thresholds
-
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-
-### Monitoring Dashboard Queries
-
-**Versori Platform Logs Query:**
-
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-
-# High error rates
-errorRate:>5
-
-# State management issues
-stateUpdated:false AND success:true
-```
-
-### Common Issues and Solutions
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-
-**Issue**: "SFTP connection timeout"
-
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check SFTP credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-
-**Issue**: "State not updating after successful extraction"
-
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-
-**Issue**: "First run exceeds record limits"
-
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-
-**Issue**: "Excessive duplicate records in output"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
-
----
-
-## Troubleshooting Quick Reference
-
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "XML generation failed" | Format-specific error | Check XML generation logic, validate output |
-
----
+---
+template_id: tpl-extract-orders-to-sftp-xml
+canonical_filename: template-extraction-orders-to-sftp-xml.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: sftp-xml
+entity: orders
+format: xml
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+  - dispose-finally
+---
+
+# Template: Extraction - Orders to SFTP XML
+
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Memory Management** - Clear large result sets after processing batches
+- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
+- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
+- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
+
+## Installation
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk
+```
+
+Always check [npm registry](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) for the latest version.
+
+---
+
+## 📚 STEP 1: Load These Docs (Human Checklist)
+
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+---
+
+## 📋 STEP 2: Tell Your AI (Prompt)
+
+Copy/paste this prompt after loading the documentation above:
+
+```
+Create a Versori scheduled extractor for orders that uses ExtractionOrchestrator + JobTracker, incremental updatedOn with a 60s overlap buffer, transforms via UniversalMapper, generates XML with SDK's XMLBuilder, uploads to SFTP using SftpDataSource with dispose(). Include 3 workflows: scheduled, ad-hoc webhook, and job-status query with native Versori logging.
+```
+
+---
+
+## 📦 SDK Imports (Verified - Versori Optimized)
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+
+---
+
+# Versori Scheduled: Orders Extraction to SFTP XML (Incremental)
+
+**FC Connect SDK Use Case Guide**
+
+> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> Version: Check [npm](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) for latest
+
+Context: Scheduled Versori workflow that extracts new/updated orders from Fluent Commerce via GraphQL query with **ExtractionOrchestrator**, **JobTracker**, and **incremental timestamp tracking**, transforms with `UniversalMapper`, and writes **XML files** to partner SFTP server for 3PL/WMS/fulfillment center integration.
+
+**Pattern**: EXTRACTION (Fluent → SFTP XML)
+**Complexity**: High | Runtime: Versori Platform (Scheduled)
+
+---
+
+## ⚠️ IMPORTANT: Production-Ready Base Template
+
+> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
+>
+> This is a **production-ready base template** demonstrating FC Connect SDK best practices for order extraction workflows with XML output.
+>
+> **✅ INCLUDED FEATURES:**
+>
+> - ✅ Comprehensive error handling with retry logic
+> - ✅ SFTP upload with exponential backoff (3 attempts)
+> - ✅ State management with overlap buffer (prevents missed records)
+> - ✅ Job tracking with lifecycle management
+> - ✅ Security (credential masking in logs)
+> - ✅ UTC time enforcement (prevents timezone bugs)
+> - ✅ Incremental extraction (safe, efficient, production-ready)
+> - ✅ Natural rate limiting via timestamps
+>
+> **📝 BEFORE DEPLOYING:**
+>
+> 1. Review and customize activation variables for your environment
+> 2. Test with sample data in your Versori workspace
+> 3. Adjust safety limits (pageSize, maxRecords) if needed
+> 4. Configure monitoring alerts for extraction failures
+> 5. Verify SFTP credentials and paths
+>
+> **This base template follows SDK best practices - tweak specific to your needs.**
+
+---
+
+## What You'll Build
+
+- **Incremental extraction** using `updatedOn >= (lastRunTime - buffer)` with **overlap buffer**
+- **ExtractionOrchestrator** for auto-pagination and path-based extraction
+- **JobTracker** for lifecycle management and status tracking
+- **State management** with VersoriKV to track last successful run
+- **Safety buffer** (60 seconds) to handle clock skew and race conditions
+- GraphQL query with nested order lines
+- UniversalMapper transformation with line item data
+- **XML file generation** with proper structure for 3PL/WMS systems
+- **SFTP upload** to partner server (with `dispose()` cleanup)
+- **3 workflow patterns**: scheduled, ad-hoc webhook, job status query
+- **Failure recovery** with timestamp tracking
+
+## Business Use Case
+
+**Hourly order feed to 3PL/fulfillment center:**
+
+- Extract new and updated orders since last run
+- Generate XML file with order header + line items
+- Upload to 3PL SFTP server for warehouse management system
+- Run every hour to enable real-time fulfillment
+- Support order updates (address changes, item modifications)
+- Standard XML format for EDI/ERP integration
+
+## SDK Methods Used
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+await createClient(ctx); // Versori-aware client
+const orchestrator = new ExtractionOrchestrator(client, log); // Auto-pagination
+const tracker = new JobTracker(kv, log); // Job lifecycle tracking
+await orchestrator.extract({ query, resultPath, variables, pageSize, maxRecords }); // Extract
+new VersoriKVAdapter(ctx.openKv(':project:')); // State management
+new UniversalMapper(exportMapping); // Field transformation
+new XMLBuilder(options); // XML generation with auto-escaping
+await sftp.uploadFile(remotePath, buffer); // SFTP upload (no options param)
+await sftp.dispose(); // CRITICAL: Connection cleanup
+```
+
+## SFTP Connection Setup (Recommended)
+
+**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
+
+**Connection Configuration:**
+
+1. In Versori platform, create a connection named `versori_ftp_server`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+5. The SDK will automatically decode the credentials using:
+
+```typescript
+// Get SFTP credentials from Versori connection (Basic Auth)
+// RECOMMENDED: Use activation.connections (already decoded)
+const allConnections = ctx.activation.connections || [];
+const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+if (!sftpConn) {
+  throw new Error('SFTP connection "versori_ftp_server" not found');
+}
+
+const credential = sftpConn.credentials[0]?.credential;
+if (!credential?.data?.basicAuth) {
+  throw new Error('SFTP connection not configured with Basic Authentication');
+}
+
+const { username, password } = credential.data.basicAuth;
+// ✅ Already decoded - no Buffer.from() needed!
+```
+
+**Why use connections instead of activation variables?**
+
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection can be reused across workflows
+- ✅ No need to manage sensitive data in activation variables
+- ✅ Easier credential rotation
+
+### SFTP Credential Access Methods
+
+**You have TWO methods to access SFTP credentials from Versori connections:**
+
+#### Method 1: activation.connections (✅ RECOMMENDED)
+
+**Best for:** All scenarios - cleanest, no decoding needed
+
+```typescript
+import { Buffer } from 'node:buffer'; // Required for SFTP upload
+
+const { activation, log } = ctx;
+
+// Get the connection (credentials ALREADY DECODED!)
+const allConnections = activation.connections || [];
+const sftpConnection = allConnections.find(c => c.name === 'versori_ftp_server');
+
+if (!sftpConnection) {
+  throw new Error(
+    `SFTP connection not found. Available: ${allConnections.map(c => c.name).join(', ')}`
+  );
+}
+
+const sftpCred = sftpConnection.credentials[0]?.credential;
+
+if (!sftpCred?.data?.basicAuth) {
+  throw new Error('SFTP connection not configured with Basic Auth');
+}
+
+// ✅ Already decoded - no Buffer.from() needed!
+const sftpUsername = sftpCred.data.basicAuth.username;
+const sftpPassword = sftpCred.data.basicAuth.password;
+const sftpHost = sftpConnection.baseUrl || ctx.activation.getVariable('sftpHost');
+
+log.info('SFTP credentials retrieved', {
+  username: sftpUsername,
+  host: sftpHost,
+  hasPassword: !!sftpPassword,
+});
+```
+
+**Why this is best:**
+
+- ✅ No base64 decoding required
+- ✅ Type-safe access to credential structure
+- ✅ Works in all task types (fn, http, webhook)
+- ✅ Cleaner error handling
+
+#### Method 2: credentials().get() (Alternative)
+
+**Use when:** Working in fn() tasks where activation.connections not available
+
+```typescript
+import { Buffer } from 'node:buffer'; // Required for both decoding AND SFTP upload
+
+// Retrieve credentials (returns base64-encoded accessToken)
+const sftpCred = await ctx.credentials().getAccessToken('versori_ftp_server');
+
+if (!sftpCred?.accessToken) {
+  throw new Error('No SFTP credentials found');
+}
+
+// ⚠️ Manual base64 decoding required
+const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+const [sftpUsername, sftpPassword] = rawBasicAuth.split(':');
+
+if (!sftpUsername || !sftpPassword) {
+  throw new Error('Invalid credential format');
+}
+
+log.info('SFTP credentials decoded', {
+  hasUsername: !!sftpUsername,
+  hasPassword: !!sftpPassword,
+});
+```
+
+**Limitations:**
+
+- ⚠️ Requires manual base64 decoding
+- ⚠️ More error-prone (string splitting)
+- ⚠️ Only works in fn() tasks
+
+### Buffer Import Reminder (CRITICAL!)
+
+**For both methods, you MUST import Buffer in Versori/Deno runtime:**
+
+```typescript
+import { Buffer } from 'node:buffer'; // ⚠️ ALWAYS REQUIRED
+```
+
+**Why:**
+
+- SFTP uploads require Buffer: `Buffer.from(xmlContent, 'utf8')`
+- Method 2 decoding requires Buffer: `Buffer.from(accessToken, 'base64')`
+- Deno runtime doesn't have global Buffer (unlike Node.js)
+
+**See:** [SFTP Credential Access & Security](../../../../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for complete documentation with troubleshooting and security best practices
+
+## Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+```json
+{
+  "retailerId": "your-retailer-id",
+  "sftpHost": "sftp.3pl-partner.com",
+  "sftpPort": 22,
+  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
+  "sftpRemotePath": "/incoming/orders/",
+  "pageSize": 200,
+  "maxRecords": 10000,
+  "fallbackStartDate": "2024-01-01T00:00:00Z",
+  "overlapBufferSeconds": "60"
+}
+```
+
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection (see SFTP Connection Setup above).
+
+## Export Mapping Configuration
+
+**IMPORTANT**: Fields match CSV version exactly for consistency.
+
+Create file: `./config/orders.export.xml.json`
+
+```json
+{
+  "name": "orders.export.xml",
+  "version": "1.0.0",
+  "description": "Fluent Orders → 3PL XML Export",
+  "fields": {
+    "order_id": { "source": "ref", "required": true, "resolver": "sdk.trim" },
+    "order_date": { "source": "createdOn", "required": true, "resolver": "sdk.formatDateShort" },
+    "order_status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
+    "customer_name": { "source": "customer.firstName", "required": false, "resolver": "sdk.trim" },
+    "customer_email": { "source": "customer.email", "required": false, "resolver": "sdk.trim" },
+    "ship_to_name": { "source": "deliveryAddress.name", "required": true, "resolver": "sdk.trim" },
+    "ship_to_address1": {
+      "source": "deliveryAddress.street1",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "ship_to_city": { "source": "deliveryAddress.city", "required": true, "resolver": "sdk.trim" },
+    "ship_to_state": {
+      "source": "deliveryAddress.state",
+      "required": true,
+      "resolver": "sdk.uppercase"
+    },
+    "ship_to_zip": {
+      "source": "deliveryAddress.postcode",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "ship_to_country": {
+      "source": "deliveryAddress.country",
+      "required": true,
+      "resolver": "sdk.uppercase"
+    },
+    "line_item_sku": { "source": "product.ref", "required": true, "resolver": "sdk.trim" },
+    "line_item_qty": { "source": "quantity", "required": true, "resolver": "sdk.parseInt" },
+    "line_item_price": { "source": "price", "required": true, "resolver": "sdk.parseFloat" }
+  }
+}
+```
+
+## Mapping & Resolvers Explained
+
+### SDK Resolvers Used
+
+The export mapping uses **SDK resolvers** to transform order data into 3PL-ready XML format:
+
+| Field             | Resolver              | Why?                         | Example Transformation                           |
+| ----------------- | --------------------- | ---------------------------- | ------------------------------------------------ |
+| `order_id`        | `sdk.trim`            | Clean order references       | `" ORD-123 "` → `"ORD-123"`                   |
+| `order_date`      | `sdk.formatDateShort` | 3PL-friendly date format     | `"2025-01-22T14:30:00Z"` → `"2025-01-22"`     |
+| `order_status`    | `sdk.uppercase`       | Normalize status codes       | `"created"` → `"CREATED"`                     |
+| `customer_name`   | `sdk.trim`            | Clean customer data          | `"John  "` → `"John"`                         |
+| `customer_email`  | `sdk.trim`            | Clean email addresses        | `" user@example.com "` → `"user@example.com"` |
+| `ship_to_name`    | `sdk.trim`            | Clean shipping contact       | `"Jane Doe  "` → `"Jane Doe"`                 |
+| `ship_to_state`   | `sdk.uppercase`       | Normalize state codes        | `"ny"` → `"NY"`                               |
+| `ship_to_country` | `sdk.uppercase`       | Normalize country codes      | `"us"` → `"US"`                               |
+| `line_item_sku`   | `sdk.trim`            | Clean SKU references         | `" SKU-001 "` → `"SKU-001"`                   |
+| `line_item_qty`   | `sdk.parseInt`        | Parse quantities as integers | `"5"` → `5`                                   |
+| `line_item_price` | `sdk.parseFloat`      | Parse prices as decimals     | `"19.99"` → `19.99`                           |
+
+### Transformation Flow
+
+```typescript
+// 1. GraphQL Response (from Fluent API)
+{
+  ref: " ORD-12345 ",
+  createdOn: "2025-01-22T14:30:00.000Z",
+  status: "created",
+  customer: {
+    firstName: "John  ",
+    email: " john@example.com "
+  },
+  deliveryAddress: {
+    name: "Jane Doe  ",
+    street1: "123 Main St",
+    city: "New York",
+    state: "ny",
+    postcode: "10001",
+    country: "us"
+  },
+  items: [
+    {
+      quantity: "2",
+      price: "29.99",
+      product: { ref: " SKU-001 ", name: "Widget" }
+    }
+  ]
+}
+
+// 2. UniversalMapper applies resolvers
+const mapper = new UniversalMapper(ordersExportMapping);
+const result = await mapper.map(order);
+
+// 3. Transformed Output (clean, normalized)
+result.data = {
+  order_id: "ORD-12345",              // ✅ Trimmed
+  order_date: "2025-01-22",           // ✅ Short date format
+  order_status: "CREATED",            // ✅ Uppercased
+  customer_name: "John",              // ✅ Trimmed
+  customer_email: "john@example.com", // ✅ Trimmed
+  ship_to_name: "Jane Doe",           // ✅ Trimmed
+  ship_to_address1: "123 Main St",
+  ship_to_city: "New York",
+  ship_to_state: "NY",                // ✅ Uppercased
+  ship_to_zip: "10001",
+  ship_to_country: "US",              // ✅ Uppercased
+  items: [
+    {
+      line_item_sku: "SKU-001",       // ✅ Trimmed
+      line_item_qty: 2,               // ✅ Integer
+      line_item_price: 29.99          // ✅ Float
+    }
+  ]
+}
+
+// 4. Generate XML with proper escaping
+const xml = buildOrdersXML([result.data]);
+```
+
+### Custom Resolvers for Order-Specific Logic
+
+You can add **custom resolvers** for 3PL-specific transformations:
+
+```typescript
+const ordersExportMapping = {
+  name: 'orders.export.xml',
+  version: '1.0.0',
+  fields: {
+    order_id: { source: 'ref', required: true, resolver: 'sdk.trim' },
+    order_date: { source: 'createdOn', required: true, resolver: 'sdk.formatDateShort' },
+
+    // Custom resolver: Generate 3PL reference number
+    partner_order_ref: {
+      source: 'ref',
+      resolver: 'custom.generate3PLReference',
+    },
+
+    // Custom resolver: Calculate order total
+    order_total: {
+      source: 'items',
+      resolver: 'custom.calculateOrderTotal',
+    },
+
+    // Custom resolver: Map shipping service level
+    shipping_service_code: {
+      source: 'shippingMethod',
+      resolver: 'custom.mapShippingService',
+    },
+
+    // Custom resolver: Format phone number for 3PL
+    ship_to_phone: {
+      source: 'deliveryAddress.phone',
+      resolver: 'custom.formatPhone',
+    },
+  },
+};
+
+// Custom resolver implementations
+const customResolvers = {
+  'custom.generate3PLReference': (orderRef: string) => {
+    // Format: 3PL-YYYYMMDD-ORDERREF
+    const today = new Date().toISOString().slice(0, 10).replace(/-/g, '');
+    return `3PL-${today}-${orderRef}`;
+  },
+
+  'custom.calculateOrderTotal': (items: any[]) => {
+    const total = items.reduce((sum, item) => {
+      const qty = parseFloat(item.quantity) || 0;
+      const price = parseFloat(item.price) || 0;
+      return sum + qty * price;
+    }, 0);
+    return total.toFixed(2);
+  },
+
+  'custom.mapShippingService': (method: string) => {
+    const serviceMap: Record<string, string> = {
+      STANDARD: 'GROUND',
+      EXPRESS: '2DAY',
+      OVERNIGHT: 'NEXTDAY',
+      INTERNATIONAL: 'INTL',
+    };
+    return serviceMap[method] || 'GROUND';
+  },
+
+  'custom.formatPhone': (phone: string) => {
+    if (!phone) return '';
+    // Remove all non-digits
+    const digits = phone.replace(/\D/g, '');
+    // Format as (XXX) XXX-XXXX
+    if (digits.length === 10) {
+      return `(${digits.slice(0, 3)}) ${digits.slice(3, 6)}-${digits.slice(6)}`;
+    }
+    return phone;
+  },
+};
+
+// Use with UniversalMapper
+const mapper = new UniversalMapper(ordersExportMapping, { customResolvers });
+```
+
+### Available SDK Resolvers
+
+**String Transformations:**
+
+- `sdk.trim` - Remove whitespace
+- `sdk.uppercase` - Convert to uppercase
+- `sdk.lowercase` - Convert to lowercase
+- `sdk.toString` - Convert to string
+
+**Number Transformations:**
+
+- `sdk.parseInt` - Parse integer
+- `sdk.parseFloat` - Parse decimal
+- `sdk.number` - Generic number conversion
+
+**Date Transformations:**
+
+- `sdk.formatDate` - ISO 8601 format (`2025-01-22T14:30:00Z`)
+- `sdk.formatDateShort` - Short date format (`2025-01-22`)
+- `sdk.parseDate` - Parse date string
+
+**Type Conversions:**
+
+- `sdk.boolean` - Convert to boolean
+- `sdk.parseJson` - Parse JSON string
+- `sdk.toJson` - Convert to JSON string
+
+**Utility:**
+
+- `sdk.identity` - Pass through unchanged
+- `sdk.coalesce` - Return first non-null value
+
+See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
+
+## GraphQL Query
+
+```graphql
+query GetOrders($retailerId: ID!, $updatedAfter: DateTime!, $first: Int!, $after: String) {
+  orders(
+    retailerId: $retailerId
+    updatedOn: { after: $updatedAfter }
+    first: $first
+    after: $after
+  ) {
+    edges {
+      node {
+        id
+        ref
+        status
+        createdOn
+        updatedOn
+        customer {
+          firstName
+          lastName
+          email
+        }
+        deliveryAddress {
+          name
+          street1
+          street2
+          city
+          state
+          postcode
+          country
+        }
+        items {
+          id
+          quantity
+          price
+          product {
+            ref
+            name
+          }
+        }
+      }
+      cursor
+    }
+    pageInfo {
+      hasNextPage
+    }
+  }
+}
+```
+
+## Expected XML Output
+
+**IMPORTANT**: XML structure with same fields as CSV version for consistency.
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<Orders>
+  <Order>
+    <OrderHeader>
+      <order_id>ORD-001</order_id>
+      <order_date>2025-01-22</order_date>
+      <order_status>CREATED</order_status>
+      <customer_name>John</customer_name>
+      <customer_email>john@example.com</customer_email>
+    </OrderHeader>
+    <ShipTo>
+      <ship_to_name>John Smith</ship_to_name>
+      <ship_to_address1>123 Main St</ship_to_address1>
+      <ship_to_city>New York</ship_to_city>
+      <ship_to_state>NY</ship_to_state>
+      <ship_to_zip>10001</ship_to_zip>
+      <ship_to_country>US</ship_to_country>
+    </ShipTo>
+    <LineItems>
+      <LineItem>
+        <line_item_sku>SKU-001</line_item_sku>
+        <line_item_qty>2</line_item_qty>
+        <line_item_price>29.99</line_item_price>
+      </LineItem>
+      <LineItem>
+        <line_item_sku>SKU-002</line_item_sku>
+        <line_item_qty>1</line_item_qty>
+        <line_item_price>49.99</line_item_price>
+      </LineItem>
+    </LineItems>
+  </Order>
+  <Order>
+    <OrderHeader>
+      <order_id>ORD-002</order_id>
+      <order_date>2025-01-22</order_date>
+      <order_status>PAID</order_status>
+      <customer_name>Jane</customer_name>
+      <customer_email>jane@example.com</customer_email>
+    </OrderHeader>
+    <ShipTo>
+      <ship_to_name>Jane Doe</ship_to_name>
+      <ship_to_address1>456 Oak Ave</ship_to_address1>
+      <ship_to_city>Los Angeles</ship_to_city>
+      <ship_to_state>CA</ship_to_state>
+      <ship_to_zip>90001</ship_to_zip>
+      <ship_to_country>US</ship_to_country>
+    </ShipTo>
+    <LineItems>
+      <LineItem>
+        <line_item_sku>SKU-003</line_item_sku>
+        <line_item_qty>1</line_item_qty>
+        <line_item_price>19.99</line_item_price>
+      </LineItem>
+    </LineItems>
+  </Order>
+</Orders>
+```
+
+**Note**: XML preserves hierarchical structure (Order → LineItems) unlike CSV which flattens to rows.
+
+---
+
+## Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+orders-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-orders-extraction.ts  # Scheduled: Daily orders extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-orders-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts         # Webhook: Status query
+    │
+    ├── services/
+    │   └── orders-extraction.service.ts   # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── orders.export.xml.json         # Mapping configuration
+```
+
+**Why This Structure:**
+- ✅ Easier to maintain (each workflow in its own file)
+- ✅ Clear separation of concerns (scheduled vs webhook)
+- ✅ Better for team collaboration (fewer merge conflicts)
+- ✅ Follows Versori best practices
+
+---
+
+### Overview
+
+Even with **incremental-only** extraction, order data needs safeguards to prevent runtime failures:
+
+- **Nested data**: Orders contain line items, addresses, customers (1 order → 5-20 line items)
+- **XML generation**: More memory-intensive than CSV generation
+- **Time-critical**: 3PL/fulfillment systems need reliable, timely feeds
+- **High priority**: Order processing delays directly impact customers
+
+### Hard Limits
+
+```typescript
+const SAFETY_LIMITS = {
+  MAX_ORDERS_PER_RUN: 50000, // 50k orders per run
+  MAX_XML_ELEMENTS: 500000, // 500k XML elements total
+  MAX_RECORDS_PER_FILE: 10000, // 10k orders per XML file (SFTP-friendly)
+  MAX_FILE_SIZE_MB: 100, // 100MB per file
+  MAX_XML_SIZE_MB: 200, // Total extraction size
+  CHUNK_SIZE: 5000, // Process in chunks
+  AVG_LINE_ITEMS_PER_ORDER: 3, // Conservative estimate
+  ESTIMATED_BYTES_PER_ORDER_XML: 2000, // XML with full structure
+};
+```
+
+**Why XML needs special consideration?**
+
+- **XML overhead**: Tags and structure add 2-3x size vs CSV
+- **Memory during generation**: Building XML tree in memory
+- **SFTP partners**: Often have stricter file size limits than S3
+- **Validation**: 3PL systems validate XML against schemas
+
+### Runtime Validation Function
+
+```typescript
+/**
+ * Validate extraction safety limits before processing
+ * CRITICAL: Account for XML size overhead vs CSV
+ */
+function validateExtractionLimits(orderCount: number, totalLineItems: number) {
+  const MAX_ORDERS_PER_RUN = 50000;
+  const MAX_XML_ELEMENTS = 500000; // orders + line items + structure
+  const ESTIMATED_BYTES_PER_ORDER_XML = 2000; // Full XML order element
+  const estimatedSizeMB = (orderCount * ESTIMATED_BYTES_PER_ORDER_XML) / (1024 * 1024);
+  const MAX_XML_SIZE_MB = 200;
+
+  if (orderCount > MAX_ORDERS_PER_RUN) {
+    return {
+      valid: false,
+      error: `Extraction limit exceeded: ${orderCount} orders (max: ${MAX_ORDERS_PER_RUN})`,
+      recommendation: `Too many orders for single extraction. Consider:
+        1. Increase extraction frequency (daily → hourly)
+        2. Add order status filters (NEW, PAID only)
+        3. Split by fulfillment location
+        4. Contact support if consistently exceeding limits`,
+      orderCount,
+      maxAllowed: MAX_ORDERS_PER_RUN,
+    };
+  }
+
+  const totalElements = orderCount + totalLineItems + orderCount * 2; // headers + shipping
+  if (totalElements > MAX_XML_ELEMENTS) {
+    return {
+      valid: false,
+      error: `XML element limit exceeded: ${totalElements} elements (max: ${MAX_XML_ELEMENTS})`,
+      recommendation: `XML structure too large. Orders: ${orderCount}, Line items: ${totalLineItems}. Increase extraction frequency.`,
+      totalElements,
+      maxAllowed: MAX_XML_ELEMENTS,
+    };
+  }
+
+  if (estimatedSizeMB > MAX_XML_SIZE_MB) {
+    return {
+      valid: false,
+      error: `XML size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_XML_SIZE_MB}MB)`,
+      recommendation:
+        'File splitting required. Increase extraction frequency to reduce batch size.',
+      estimatedSizeMB,
+      maxAllowed: MAX_XML_SIZE_MB,
+    };
+  }
+
+  return { valid: true };
+}
+```
+
+### Memory-Safe XML Generation with XMLBuilder
+
+The SDK's `XMLBuilder` handles XML generation efficiently with automatic escaping:
+
+```typescript
+import { Buffer } from 'node:buffer';
+import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize XMLBuilder (reusable)
+const xmlBuilder = new XMLBuilder({
+  rootElement: 'Orders',
+  prettyPrint: true,
+  indent: '  ',
+  xmlDeclaration: true,
+  encoding: 'UTF-8',
+});
+
+/**
+ * Generate XML from orders using XMLBuilder
+ * Handles nested structures (OrderHeader, ShipTo, LineItems) automatically
+ */
+function buildOrdersXML(orders: any[]): string {
+  // Transform to XMLBuilder format with nested structures
+  const ordersForXml = orders.map(order => ({
+    OrderHeader: {
+      order_id: order.order_id,
+      order_date: order.order_date,
+      order_status: order.order_status,
+      customer_name: order.customer_name || '',
+      customer_email: order.customer_email || '',
+    },
+    ShipTo: {
+      ship_to_name: order.ship_to_name,
+      ship_to_address1: order.ship_to_address1,
+      ship_to_city: order.ship_to_city,
+      ship_to_state: order.ship_to_state,
+      ship_to_zip: order.ship_to_zip,
+      ship_to_country: order.ship_to_country,
+    },
+    LineItems: {
+      LineItem: order.line_items.map(item => ({
+        line_item_sku: item.line_item_sku,
+        line_item_qty: item.line_item_qty,
+        line_item_price: item.line_item_price,
+      })),
+    },
+  }));
+
+  // XMLBuilder handles all escaping and structure automatically
+  return xmlBuilder.build({ Order: ordersForXml });
+}
+
+// Example output:
+// <Orders>
+//   <Order>
+//     <OrderHeader>
+//       <order_id>ORD-001</order_id>
+//       <customer_name>Smith &amp; Jones</customer_name>
+//     </OrderHeader>
+//     <ShipTo>
+//       <ship_to_name>John Smith</ship_to_name>
+//       <ship_to_address1>123 Main St</ship_to_address1>
+//     </ShipTo>
+//     <LineItems>
+//       <LineItem>
+//         <line_item_sku>SKU-001</line_item_sku>
+//         <line_item_qty>2</line_item_qty>
+//       </LineItem>
+//     </LineItems>
+//   </Order>
+// </Orders>
+```
+
+**Benefits:**
+
+- ✅ Automatic XML escaping (handles &, <, >, ", ', etc.)
+- ✅ Nested structure support (OrderHeader, ShipTo, LineItems)
+- ✅ Array handling (multiple LineItem elements)
+- ✅ Memory-efficient streaming
+- ✅ Pretty printing with proper indentation
+
+## Complete Workflow Code
+
+The following code examples demonstrate the implementation across separate files following the recommended modular structure.
+
+
+### 1. Entry Point (src/index.ts)
+
+```typescript
+/**
+ * Entry point - Export all workflows for Versori platform
+ *
+ * This file exports all workflows to be registered with Versori.
+ * Each workflow is defined in its own file for better organization.
+ */
+
+// Scheduled workflows
+export { scheduledOrdersExtraction } from './workflows/scheduled/daily-orders-extraction';
+
+// Webhook workflows
+export { adhocOrdersExtraction } from './workflows/webhook/adhoc-orders-extraction';
+export { ordersJobStatus } from './workflows/webhook/job-status-check';
+```
+
+### 2. Scheduled Workflow (src/workflows/scheduled/daily-orders-extraction.ts)
+
+```typescript
+import { schedule, http } from '@versori/run';
+import {
+  executeOrderExtraction,
+  generateJobId,
+} from '../../services/extraction-orchestration';
+
+/**
+ * Scheduled workflow: Daily orders extraction to SFTP XML
+ * Runs at 2:00 AM daily
+ *
+ * DELEGATION PATTERN:
+ * - Workflow receives ctx from Versori
+ * - Passes entire ctx to service function
+ * - Service handles all business logic
+ */
+export const scheduledOrdersExtraction = schedule('orders-extract-xml-daily', '0 2 * * *').then(
+  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = generateJobId('SCHED', 'ORDERS');
+
+    log.info('🚀 [WORKFLOW] Starting scheduled extraction', { jobId });
+
+    const result = await executeOrderExtraction(ctx, {
+      jobId,
+      triggeredBy: 'schedule',
+      updateState: true, // Always update state for scheduled runs
+    });
+
+    const duration = Date.now() - executionStartTime;
+
+    if (result.success) {
+      log.info('✅ [WORKFLOW] Extraction completed successfully', {
+        jobId,
+        ordersExtracted: result.ordersExtracted,
+        duration: `${duration}ms`,
+      });
+    } else {
+      log.error('❌ [WORKFLOW] Extraction failed', {
+        jobId,
+        error: result.error,
+        duration: `${duration}ms`,
+      });
+    }
+
+    return { ...result, duration };
+  })
+);
+```
+
+### 3. Ad-hoc Webhook (src/workflows/webhook/adhoc-orders-extraction.ts)
+
+```typescript
+import { webhook, http } from '@versori/run';
+import {
+  executeOrderExtraction,
+  generateJobId,
+} from '../../services/extraction-orchestration';
+
+/**
+ * Webhook workflow: Ad-hoc orders extraction
+ * Allows manual/on-demand extraction with date range overrides
+ */
+export const adhocOrdersExtraction = webhook('orders-adhoc', {
+  connection: 'orders-adhoc',
+  response: { mode: 'sync' },
+}).then(
+  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = generateJobId('ADHOC', 'ORDERS');
+
+    log.info('🔧 [WEBHOOK] Starting ad-hoc extraction', {
+      jobId,
+      fromDate: ctx.data.fromDate,
+      toDate: ctx.data.toDate
+    });
+
+    const result = await executeOrderExtraction(ctx, {
+      jobId,
+      triggeredBy: 'webhook',
+      fromDate: ctx.data.fromDate,
+      toDate: ctx.data.toDate,
+      updateState: ctx.data.updateState !== false,
+    });
+
+    const duration = Date.now() - executionStartTime;
+
+    if (result.success) {
+      log.info('✅ [WEBHOOK] Ad-hoc extraction completed', {
+        jobId,
+        ordersExtracted: result.ordersExtracted,
+        duration: `${duration}ms`,
+      });
+    } else {
+      log.error('❌ [WEBHOOK] Ad-hoc extraction failed', {
+        jobId,
+        error: result.error,
+        duration: `${duration}ms`,
+      });
+    }
+
+    return { ...result, duration };
+  })
+);
+```
+
+### 4. Job Status Query (src/workflows/webhook/job-status-check.ts)
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { getJobStatus } from '../../services/extraction-orchestration';
+
+/**
+ * Webhook workflow: Job status query
+ * Allows checking the status of extraction jobs by ID
+ */
+export const ordersJobStatus = webhook('orders-job-status', {
+  connection: 'orders-job-status',
+  response: { mode: 'sync' },
+}).then(
+  fn('query-job-status', async (ctx: any) => {
+    const { data, log, openKv } = ctx;
+    // Security is enforced by the 'orders-job-status' connection
+
+    log.info('🔍 [STATUS] Querying job status', { jobId: data.jobId });
+
+    const jobId = data.jobId;
+    if (!jobId) {
+      log.error('❌ [STATUS] Missing job ID');
+      return { success: false, error: 'Job ID required' };
+    }
+
+    const status = await getJobStatus(openKv(':project:'), jobId, log);
+
+    if (status) {
+      log.info('✅ [STATUS] Job found', { jobId, status: status.status });
+      return { success: true, jobId, ...status };
+    } else {
+      log.warn('⚠️ [STATUS] Job not found', { jobId });
+      return { success: false, error: 'Job not found', jobId };
+    }
+  })
+);
+```
+
+### 5. Main Orchestration Service (src/services/extraction-orchestration.ts)
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+import ordersExportMapping from '../../config/orders.export.xml.json' with { type: 'json' };
+
+const ORDERS_QUERY = `
+  query GetOrders($retailerId: ID!, $updatedAfter: DateTime!, $first: Int!, $after: String) {
+    orders(
+      retailerId: $retailerId
+      updatedOn: { after: $updatedAfter }
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          status
+          createdOn
+          updatedOn
+          customer {
+            firstName
+            lastName
+            email
+          }
+          deliveryAddress {
+            name
+            street1
+            street2
+            city
+            state
+            postcode
+            country
+          }
+          items {
+            id
+            quantity
+            price
+            product {
+              ref
+              name
+            }
+          }
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+
+// Initialize XMLBuilder for orders
+const xmlBuilder = new XMLBuilder({
+  rootElement: 'Orders',
+  prettyPrint: true,
+  indent: '  ',
+  xmlDeclaration: true,
+  encoding: 'UTF-8',
+});
+
+function buildOrdersXML(orders: any[]): string {
+  // Transform to XMLBuilder format with nested structures
+  const ordersForXml = orders.map(order => ({
+    OrderHeader: {
+      order_id: order.order_id,
+      order_date: order.order_date,
+      order_status: order.order_status,
+      customer_name: order.customer_name || '',
+      customer_email: order.customer_email || '',
+    },
+    ShipTo: {
+      ship_to_name: order.ship_to_name,
+      ship_to_address1: order.ship_to_address1,
+      ship_to_city: order.ship_to_city,
+      ship_to_state: order.ship_to_state,
+      ship_to_zip: order.ship_to_zip,
+      ship_to_country: order.ship_to_country,
+    },
+    LineItems: {
+      LineItem: order.line_items.map(item => ({
+        line_item_sku: item.line_item_sku,
+        line_item_qty: item.line_item_qty,
+        line_item_price: item.line_item_price,
+      })),
+    },
+  }));
+
+  return xmlBuilder.build({ Order: ordersForXml });
+}
+
+interface ExtractionOptions {
+  jobId: string;
+  triggeredBy: 'schedule' | 'webhook';
+  fromDate?: string;
+  toDate?: string;
+  updateState: boolean;
+}
+
+export async function executeOrderExtraction(ctx: any, options: ExtractionOptions) {
+  const { jobId, triggeredBy, fromDate, toDate, updateState } = options;
+  const log = ctx.log;
+  const retailerId = ctx.activation?.getVariable('retailerId');
+  const pageSize = parseInt(ctx.activation?.getVariable('pageSize') || '200', 10);
+  const maxRecords = parseInt(ctx.activation?.getVariable('maxRecords') || '10000', 10);
+  const fallbackStartDate =
+    ctx.activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
+
+  // Get SFTP credentials from Versori connection (Basic Auth)
+  // RECOMMENDED: Use activation.connections (already decoded)
+  const allConnections = ctx.activation.connections || [];
+  const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+  if (!sftpConn) {
+    throw new Error('SFTP connection "versori_ftp_server" not found');
+  }
+
+  const credential = sftpConn.credentials[0]?.credential;
+  if (!credential?.data?.basicAuth) {
+    throw new Error('SFTP connection not configured with Basic Authentication');
+  }
+
+  const { username, password } = credential.data.basicAuth;
+  // ✅ Already decoded - no Buffer.from() needed!
+
+  const sftpSettings = {
+    host: ctx.activation?.getVariable('sftpHost'),
+    port: parseInt(ctx.activation?.getVariable('sftpPort') || '22', 10),
+    username, // From connection (secure)
+    password, // From connection (secure)
+    privateKey: ctx.activation?.getVariable('sftpPrivateKey'),
+    remotePath: ctx.activation?.getVariable('sftpRemotePath') || '/incoming/orders/',
+  };
+
+  const missing: string[] = [];
+  if (!retailerId) missing.push('retailerId');
+  if (!sftpSettings.host) missing.push('sftpHost');
+  if (!sftpSettings.username) missing.push('sftpUsername from connection');
+  if (!sftpSettings.password && !sftpSettings.privateKey)
+    missing.push('sftpPassword from connection or sftpPrivateKey');
+  if (missing.length)
+    return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
+
+  // SFTP connection - MUST use try/finally with dispose()
+  const sftp = new SftpDataSource(
+    {
+      type: 'SFTP_XML',
+      connectionId: 'sftp-orders-xml-export',
+      name: 'SFTP Orders XML Export',
+      settings: {
+        host: sftpSettings.host,
+        port: sftpSettings.port,
+        username: sftpSettings.username,
+        password: sftpSettings.password,
+        privateKey: sftpSettings.privateKey,
+        remotePath: sftpSettings.remotePath,
+        filePattern: '*.xml',
+      },
+    },
+    log
+  );
+
+  try {
+    // 
+    // STEP 1/8: Initialize Job Tracking
+    // 
+    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
+    const tracker = new JobTracker(kv, log);
+
+    await tracker.createJob(jobId, {
+      triggeredBy,
+      hasDateOverride: !!fromDate,
+      fromDate,
+      toDate,
+      updateStateAfterRun: updateState,
+    });
+
+    log.info('Job created', { jobId, triggeredBy });
+
+    // 
+    // STEP 2/8: Load State & Calculate Time Window
+    // 
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'state_load',
+      message: 'Loading last run state',
+    });
+
+    const stateKey = ['extraction', 'orders-xml', 'lastRunTime'];
+    const lastRunState = await kv.get(stateKey);
+    const rawLastRunTime = fromDate || lastRunState?.value?.timestamp || fallbackStartDate;
+
+    // Overlap buffer configuration (default: 60 seconds)
+    const overlapBufferSeconds = parseInt(
+      ctx.activation?.getVariable('overlapBufferSeconds') || '60',
+      10
+    );
+    const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
+
+    // Apply overlap buffer for query (safety window)
+    const bufferedLastRunTime = new Date(
+      new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
+    ).toISOString();
+
+    const effectiveEndTime = toDate || new Date().toISOString();
+
+    log.info('Time window calculated', {
+      rawLastRunTime,
+      bufferedLastRunTime,
+      effectiveEndTime,
+      overlapBufferSeconds,
+      retailerId,
+    });
+
+    // 
+    // STEP 3/8: Initialize Fluent Client & ExtractionOrchestrator
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'client_init',
+      message: 'Initializing Fluent client',
+    });
+
+    const client = await createClient(ctx);
+    const orchestrator = new ExtractionOrchestrator(client, log);
+
+    // 
+    // STEP 4/8: Extract Data (ExtractionOrchestrator)
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'extraction',
+      message: 'Extracting data with auto-pagination',
+    });
+
+    // ? Enhanced: Extract context for progress logging
+    const dateRangeInfo = {
+      start: bufferedLastRunTime,
+      end: effectiveEndTime,
+      retailerId
+    };
+
+    // ? Enhanced: Start logging with context
+    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
+     query: 'orders',
+     pageSize,
+     maxRecords,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     retailerId: dateRangeInfo.retailerId,
+     jobId
+    });
+
+    const extractionResult = await orchestrator.extract({
+      query: ORDERS_QUERY,
+      resultPath: 'orders.edges.node',
+      variables: {
+        retailerId,
+        updatedAfter: bufferedLastRunTime,
+        first: pageSize,
+      },
+      pageSize,
+      maxRecords,
+      validateItem: item => !!(item.ref && item.customer),
+    });
+
+    const rawRecords = extractionResult.data;
+
+    log.info('Extraction complete', {
+      totalRecords: extractionResult.stats.totalRecords,
+      totalPages: extractionResult.stats.totalPages,
+      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+      errors: extractionResult.errors ? extractionResult.errors.length : 0,
+    });
+
+    // ? Enhanced: Completion logging with summary
+    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+     totalRecords: extractionResult.stats.totalRecords,
+     totalPages: extractionResult.stats.totalPages,
+     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+     failedValidations: extractionResult.stats.failedValidations,
+     truncated: extractionResult.stats.truncated,
+     truncationReason: extractionResult.stats.truncationReason,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     jobId
+    });
+
+    if (extractionResult.errors && extractionResult.errors.length > 0) {
+      log.warn('Non-fatal extraction errors encountered', {
+        errorCount: extractionResult.errors.length,
+        sampleErrors: extractionResult.errors.slice(0, 3),
+      });
+    }
+
+    if (rawRecords.length === 0) {
+      await tracker.markCompleted(jobId, {
+        recordCount: 0,
+        message: 'No new orders to extract',
+      });
+
+      if (updateState) {
+        await kv.set(stateKey, {
+          timestamp: new Date().toISOString(),
+          orderCount: 0,
+          extractedAt: new Date().toISOString(),
+        });
+      }
+
+      return { success: true, message: 'No new orders to extract', lastRunTime: rawLastRunTime };
+    }
+
+    // 
+    // STEP 5/8: Validate Extraction Limits
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'validation',
+      message: 'Validating extraction limits',
+    });
+
+    const MAX_ORDERS_PER_RUN = 50000;
+    const MAX_XML_ELEMENTS = 500000;
+
+    let totalLineItems = 0;
+    for (const order of rawRecords) {
+      totalLineItems += (order.items || []).length;
+    }
+
+    const totalElements = rawRecords.length + totalLineItems + rawRecords.length * 2; // orders + items + headers/shipping
+    const ESTIMATED_BYTES_PER_ORDER_XML = 2000;
+    const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_ORDER_XML) / (1024 * 1024);
+    const MAX_XML_SIZE_MB = 200;
+
+    if (rawRecords.length > MAX_ORDERS_PER_RUN) {
+      log.error('Extraction limit exceeded', {
+        orderCount: rawRecords.length,
+        maxAllowed: MAX_ORDERS_PER_RUN,
+      });
+
+      await tracker.markFailed(jobId, {
+        error: `Extraction limit exceeded: ${rawRecords.length} orders (max: ${MAX_ORDERS_PER_RUN})`,
+        recommendation: 'Increase extraction frequency or add filters',
+      });
+
+      return {
+        success: false,
+        error: `Extraction limit exceeded: ${rawRecords.length} orders (max: ${MAX_ORDERS_PER_RUN})`,
+        recommendation: `Too many orders for single extraction. Consider:
+          1. Increase extraction frequency (daily → hourly)
+          2. Add order status filters (NEW, PAID only)
+          3. Split by fulfillment location
+          4. Contact support if consistently exceeding limits`,
+        orderCount: rawRecords.length,
+        maxAllowed: MAX_ORDERS_PER_RUN,
+      };
+    }
+
+    if (totalElements > MAX_XML_ELEMENTS) {
+      log.error('XML element limit exceeded', {
+        orderCount: rawRecords.length,
+        totalLineItems,
+        totalElements,
+        maxAllowed: MAX_XML_ELEMENTS,
+      });
+
+      await tracker.markFailed(jobId, {
+        error: `XML element limit exceeded: ${totalElements} elements`,
+        recommendation: 'Increase extraction frequency',
+      });
+
+      return {
+        success: false,
+        error: `XML element limit exceeded: ${totalElements} elements (max: ${MAX_XML_ELEMENTS})`,
+        recommendation: `XML structure too large. Orders: ${rawRecords.length}, Line items: ${totalLineItems}. Increase extraction frequency.`,
+        totalElements,
+        maxAllowed: MAX_XML_ELEMENTS,
+      };
+    }
+
+    if (estimatedSizeMB > MAX_XML_SIZE_MB) {
+      log.warn('XML size approaching limit', {
+        estimatedSizeMB: estimatedSizeMB.toFixed(2),
+        maxAllowed: MAX_XML_SIZE_MB,
+        recommendation: 'Consider file splitting or increase extraction frequency',
+      });
+    }
+
+    log.info('Extraction limits validated', {
+      orderCount: rawRecords.length,
+      totalLineItems,
+      totalElements,
+      estimatedSizeMB: estimatedSizeMB.toFixed(2),
+      withinLimits: true,
+    });
+
+    await tracker.updateJob(jobId, {
+      stage: 'transformation',
+      message: 'Transforming data with UniversalMapper',
+    });
+
+    const mapper = new UniversalMapper(ordersExportMapping);
+    const transformedOrders: any[] = [];
+    const mappingErrors: any[] = [];
+
+    for (let index = 0; index < rawRecords.length; index += 1) {
+      const order = rawRecords[index];
+      const headerResult = await mapper.map(order);
+
+      if (!headerResult.success || !headerResult.data) {
+        mappingErrors.push({
+          orderRef: order.ref,
+          errors: headerResult.errors || ['Mapping failed'],
+        });
+        continue;
+      }
+
+      const header = headerResult.data as Record<string, any>;
+      const lineItems = (order.items || []).map(item => ({
+        line_item_sku: String(item.product?.ref ?? '').trim(),
+        line_item_qty: Number.parseInt(String(item.quantity ?? '0'), 10) || 0,
+        line_item_price: Number.parseFloat(String(item.price ?? '0')) || 0,
+      }));
+
+      transformedOrders.push({
+        order_id: header.order_id,
+        order_date: header.order_date,
+        order_status: header.order_status,
+        customer_name: header.customer_name,
+        customer_email: header.customer_email,
+        ship_to_name: header.ship_to_name,
+        ship_to_address1: header.ship_to_address1,
+        ship_to_city: header.ship_to_city,
+        ship_to_state: header.ship_to_state,
+        ship_to_zip: header.ship_to_zip,
+        ship_to_country: header.ship_to_country,
+        updated_on: header.updated_on || order.updatedOn,
+        line_items: lineItems,
+      });
+    }
+
+    if (mappingErrors.length > 0) {
+      log.warn('Some orders failed transformation', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+    }
+
+    if (transformedOrders.length === 0) {
+      await tracker.markFailed(jobId, {
+        error: 'All records failed mapping',
+        failedCount: mappingErrors.length,
+      });
+      return {
+        success: false,
+        error: 'All records failed mapping',
+        errors: mappingErrors,
+      };
+    }
+
+    log.info('Orders transformed', {
+      jobId,
+      transformedCount: transformedOrders.length,
+      skippedRecords: rawRecords.length - transformedOrders.length,
+    });
+
+    await tracker.updateJob(jobId, {
+      stage: 'upload',
+      message: 'Generating XML and uploading to SFTP',
+    });
+
+    const xmlContent = buildOrdersXML(transformedOrders);
+
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const fileName = `orders-${timestamp}.xml`;
+    const remotePath = `${sftpSettings.remotePath}${fileName}`;
+
+    log.info('Generated XML file', {
+      fileName,
+      size: xmlContent.length,
+      orderCount: transformedOrders.length,
+      lineItemCount: totalLineItems,
+    });
+
+    await sftp.uploadFile(remotePath, Buffer.from(xmlContent, 'utf8'));
+
+    log.info('XML file uploaded to SFTP', { remotePath });
+
+    await tracker.updateJob(jobId, {
+      stage: 'state_update',
+      message: 'Updating state and completing job',
+    });
+
+    const maxUpdatedOn = transformedOrders.reduce((max, order) => {
+      const orderTime = new Date(order.updated_on).getTime();
+      return orderTime > max ? orderTime : max;
+    }, new Date(rawLastRunTime).getTime());
+
+    const newTimestamp = new Date(maxUpdatedOn).toISOString();
+
+    if (updateState) {
+      await kv.set(stateKey, {
+        timestamp: newTimestamp,
+        orderCount: transformedOrders.length,
+        lineItemCount: totalLineItems,
+        extractedAt: new Date().toISOString(),
+        overlapBufferSeconds,
+        fileName,
+        remotePath,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      });
+
+      log.info('State updated with new timestamp (without buffer)', {
+        newTimestamp,
+        overlapBufferSeconds,
+      });
+    }
+
+    await tracker.markCompleted(jobId, {
+      recordCount: transformedOrders.length,
+      fileName,
+      sftpPath: remotePath,
+      errorCount: mappingErrors.length,
+      errors: mappingErrors,
+    });
+
+    return {
+      success: true,
+      ordersExtracted: transformedOrders.length,
+      lineItemsExtracted: totalLineItems,
+      fileName,
+      remotePath,
+      lastRunTime: rawLastRunTime,
+      newTimestamp,
+      jobId,
+      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+    };
+  } catch (error: any) {
+    log.error('Extraction failed', error, {
+      message: error?.message,
+    });
+
+    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
+    const tracker = new JobTracker(kv, log);
+
+    await tracker.markFailed(jobId, {
+      message: error instanceof Error ? error.message : String(error),
+
+      stack: error instanceof Error ? error.stack : undefined,
+
+      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+    });
+
+    return {
+      success: false,
+      message: error instanceof Error ? error.message : String(error),
+
+      stack: error instanceof Error ? error.stack : undefined,
+
+      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+      jobId,
+    };
+  } finally {
+    // CRITICAL: Always clean up SFTP connections
+    await sftp.dispose();
+    log.info('SFTP connection disposed');
+  }
+}
+
+export async function getJobStatus(kv: any, jobId: string, log: any) {
+  const tracker = new JobTracker(new VersoriKVAdapter(kv), log);
+  return await tracker.getJob(jobId);
+}
+```
+
+### 6. Job ID Generator (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Generate unique job ID
+ * Format: {PREFIX}-{ENTITY}-{TIMESTAMP}
+ */
+export function generateJobId(prefix: 'SCHED' | 'ADHOC', entity: string): string {
+  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+  return `${prefix}-${entity}-${timestamp}`;
+}
+```
+
+### 7. Package Configuration (package.json)
+
+```json
+{
+  "name": "orders-extraction-to-sftp-xml",
+  "version": "1.0.0",
+  "description": "Versori connector for orders extraction to SFTP XML",
+  "main": "dist/index.js",
+  "type": "module",
+  "scripts": {
+    "build": "tsc",
+    "dev": "tsc --watch",
+    "lint": "eslint src/**/*.ts",
+    "test": "jest"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+
+### 8. Deployment Instructions
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Build the connector
+npm run build
+
+# 3. Test locally (optional)
+npm test
+
+# 4. Deploy to Versori
+# - Upload to Versori workspace
+# - Configure activation variables
+# - Enable workflows
+
+# 5. Test workflows
+# Scheduled: Wait for next cron trigger or manually trigger
+# Ad-hoc: POST to webhook URL with API key header
+# Status: Query job status by ID
+```
+
+### 9. Testing
+
+#### Test Scheduled Extraction
+
+```bash
+# Trigger manually in Versori UI or wait for cron schedule
+# Expected: XML file uploaded to SFTP
+```
+
+#### Test Ad-hoc Extraction
+
+```bash
+curl -X POST https://your-workspace.versori.run/orders-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-22T23:59:59Z",
+    "updateState": false
+  }'
+```
+
+#### Test Job Status Query
+
+```bash
+curl -X POST https://your-workspace.versori.run/orders-job-status \
+  -H "Content-Type: application/json" \
+  -d '{
+    "jobId": "SCHED-ORDERS-2025-01-22T02-00-00Z"
+  }'
+```
+
+## Key Patterns Explained
+
+### Pattern 1: ExtractionOrchestrator for Auto-Pagination
+
+```typescript
+// ✅ CORRECT - Use ExtractionOrchestrator (handles pagination automatically)
+const orchestrator = new ExtractionOrchestrator(client, log);
+
+const extractionResult = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: { retailerId, updatedAfter: bufferedLastRunTime },
+  pageSize,
+  maxRecords,
+  validateItem: item => !!(item.ref && item.customer),
+});
+
+const rawRecords = extractionResult.data;
+
+//  WRONG - Manual pagination (avoid this pattern)
+// const result = await client.graphql({
+//   query: ORDERS_QUERY,
+//   variables: { first: pageSize },
+//   pagination: { maxRecords }
+// });
+```
+
+### Pattern 2: JobTracker for Lifecycle Management
+
+```typescript
+// ✅ CORRECT - Use JobTracker throughout workflow
+const tracker = new JobTracker(kv, log);
+
+// Create job
+await tracker.createJob(jobId, { triggeredBy, fromDate, toDate });
+
+// Update progress
+await tracker.updateJob(jobId, { stage: 'extraction', message: 'Extracting data' });
+
+// Mark completed
+await tracker.markCompleted(jobId, { recordCount, fileName });
+
+// Query status
+const status = await tracker.getJob(jobId);
+```
+
+### Pattern 3: 3-Workflow Pattern
+
+```typescript
+// ✅ CORRECT - 3 workflows for different use cases
+// 1. Scheduled: Automated daily/hourly runs
+export const scheduledOrdersExtraction = schedule('orders-extract-xml-daily', '0 2 * * *')...
+
+// 2. Ad-hoc: Manual webhook triggers with date overrides
+export const adhocOrdersExtraction = webhook('orders-adhoc', { response: { mode: 'sync' } })...
+
+// 3. Status: Query job status by ID
+export const ordersJobStatus = webhook('orders-job-status', { response: { mode: 'sync' } })...
+```
+
+### Pattern 4: XMLBuilder for Safe XML Generation (CRITICAL)
+
+Use the SDK's `XMLBuilder` - it handles all XML escaping automatically:
+
+```typescript
+import { Buffer } from 'node:buffer';
+import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize XMLBuilder (handles all escaping automatically)
+const xmlBuilder = new XMLBuilder({
+  rootElement: 'Orders',
+  prettyPrint: true,
+  encoding: 'UTF-8',
+});
+
+// ✅ CORRECT: XMLBuilder escapes automatically
+const orders = [
+  {
+    customer_name: 'Smith & Jones <Corp>', // Contains & and <>
+    customer_email: 'contact@smith&jones.com',
+    notes: 'Special chars: ¢, ©, ®, "quotes"',
+  },
+];
+
+const xml = xmlBuilder.build({ Order: orders });
+// Result: All special characters properly escaped
+// <customer_name>Smith &amp; Jones &lt;Corp&gt;</customer_name>
+// <customer_email>contact@smith&amp;jones.com</customer_email>
+// <notes>Special chars: ¢, ©, ®, &quot;quotes&quot;</notes>
+
+//  WRONG: Manual string concatenation (dangerous)
+// const xml = `<customer_name>${order.customer_name}</customer_name>`;
+// This would produce INVALID XML: <customer_name>Smith & Jones <Corp></customer_name>
+```
+
+**Why XMLBuilder?**
+
+- ✅ Automatic escaping of &, <, >, ", '
+- ✅ Handles special characters (¢, ©, ®)
+- ✅ Prevents XML injection attacks
+- ✅ Validates structure
+- ✅ Consistent, maintainable code
+
+### Pattern 5: SFTP Cleanup (CRITICAL)
+
+```typescript
+const sftp = new SftpDataSource(config, log);
+
+try {
+  await sftp.uploadFile(remotePath, buffer);
+  return { success: true };
+} finally {
+  // ALWAYS dispose SFTP connection
+  await sftp.dispose();
+}
+```
+
+**Why?** SFTP maintains open connections. Not calling `dispose()` leads to connection exhaustion.
+
+### Pattern 6: Consistent Field Names Across Formats
+
+**Same data in CSV, JSON, and XML:**
+
+- `order_id` (not orderId, not order-id, not OrderID)
+- `customer_email` (consistent with CSV version)
+- `ship_to_address1` (matches CSV exactly)
+
+This allows users to switch formats without changing downstream systems.
+
+## Common Issues
+
+**Issue 1: Malformed XML from unescaped characters**
+
+- Customer name contains `&` or `<`
+- Solution: Always use XMLBuilder (automatic escaping)
+
+**Issue 2: 3PL system rejects XML**
+
+- Missing required fields
+- Solution: Verify mapping matches 3PL schema requirements
+
+**Issue 3: File too large for SFTP partner**
+
+- Partner has 50MB limit, file is 100MB
+- Solution: Use file splitting (10k orders per file)
+
+**Issue 4: SFTP connection timeouts**
+
+- Not calling `dispose()` in finally block
+- Solution: Always use try/finally pattern
+
+**Issue 5: Job status not updating**
+
+- JobTracker not integrated
+- Solution: Use JobTracker throughout workflow
+
+## Testing
+
+### 1. Test XML Structure
+
+```typescript
+export const testXmlGeneration = http('test-xml').then(
+  fn('test-xml-gen', async () => {
+    const testOrders = [
+      {
+        order_id: 'TEST-001',
+        order_date: '2025-01-22',
+        order_status: 'CREATED',
+        customer_name: 'Test & Validate <Corp>',
+        customer_email: 'test@example.com',
+        ship_to_name: 'John Smith',
+        ship_to_address1: '123 Main St',
+        ship_to_city: 'New York',
+        ship_to_state: 'NY',
+        ship_to_zip: '10001',
+        ship_to_country: 'US',
+        line_items: [{ line_item_sku: 'SKU-001', line_item_qty: 2, line_item_price: 29.99 }],
+      },
+    ];
+
+    const xml = buildOrdersXML(testOrders);
+
+    // Validate XML structure
+    if (!xml.includes('<?xml version="1.0"')) {
+      return { success: false, error: 'Missing XML declaration' };
+    }
+
+    if (!xml.includes('&amp;') || !xml.includes('&lt;')) {
+      return { success: false, error: 'Special characters not escaped' };
+    }
+
+    return { success: true, xml };
+  })
+);
+```
+
+### 2. Test SFTP Upload
+
+```bash
+curl https://your-workspace.versori.run/test-sftp-orders-xml
+```
+
+### 3. Validate Against 3PL Schema
+
+- Download partner's XSD schema
+- Validate generated XML against schema
+- Fix any missing/incorrect elements
+
+## Production Checklist
+
+- [ ] Test SFTP credentials and connection
+- [ ] Verify SFTP server has write permissions to remotePath
+- [ ] Set appropriate extraction frequency (hourly for 3PL feeds)
+- [ ] Configure correct order status filters
+- [ ] Test XML escaping with special characters (&, <, >, ", ')
+- [ ] Validate XML against partner's schema (if provided)
+- [ ] Test `dispose()` is always called (check logs)
+- [ ] Document XML schema for 3PL integration team
+- [ ] Set up monitoring for SFTP connection failures
+- [ ] Test with real order data (addresses with special chars)
+- [ ] Verify file size limits with SFTP partner
+- [ ] Configure SFTP server IP whitelisting for Versori
+- [ ] Test file splitting with large batches (>10k orders)
+- [ ] Test all 3 workflows (scheduled, ad-hoc, status)
+- [ ] Verify JobTracker integration and status updates
+- [ ] Test ExtractionOrchestrator pagination with large datasets
+
+## Troubleshooting Guide
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records
+- **Fix**: Reduce maxRecords, increase frequency
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch
+- **Fix**: Run schema validation, check field names
+
+**Issue**: "State not updating"
+
+- **Cause**: KV write failure or intentional retry
+- **Fix**: Check KV logs, verify state update code
+
+**Issue**: "First run exceeds limits"
+
+- **Cause**: No previous timestamp, fetches all
+- **Fix**: Set fallbackStartDate close to current, apply filters
+
+**Issue**: "Excessive duplicates"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved
+- **Fix**: Verify newTimestamp saved WITHOUT buffer
+
+**Issue**: "Job status returns null"
+
+- **Cause**: Invalid job ID or job expired
+- **Fix**: Verify job ID format, check KV TTL settings
+
+## Security Best Practices
+
+### Credential Management
+
+**✅ DO**:
+
+- Store credentials in Versori activation variables
+- Rotate credentials quarterly
+- Use least-privilege accounts
+
+** DON'T**:
+
+- Never log credentials
+- Never commit to git
+- Never share across environments
+
+### Data Security
+
+- Enable encryption in transit and at rest
+- Apply data retention policies
+- Monitor access logs
+- Use VPC/private networks for sensitive data
+
+### Webhook Security
+
+- Validate API keys for ad-hoc and status workflows
+- Use HTTPS for all webhook endpoints
+- Implement rate limiting
+- Monitor for suspicious activity
+
+---
+
+**Pattern**: Enterprise incremental extraction with ExtractionOrchestrator + JobTracker for orders via SFTP (XML format)
+**❌š ï¸ Versori Sample**: Reference implementation - adapt for your production use case
+**Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for lifecycle management, always escape XML and dispose SFTP
+**Critical**: Apply 60-second overlap buffer to prevent missed records
+**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
+**Field Consistency**: Same field names as CSV version for easy format switching
+**SFTP**: Use proper connection cleanup in finally block to prevent connection leaks
+**XML**: Preserve hierarchical structure (no line item flattening needed like CSV)
+**3 Workflows**: Scheduled, ad-hoc webhook, job status query
+
+---
+
+### Pattern 8: Backward Pagination (Optional - Advanced)
+
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+
+**When to Use**:
+
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+-  **Don't use for standard incremental sync** - use forward pagination (default)
+
+**GraphQL Query Requirements**:
+
+Your query must support backward pagination by including `$last` and `$before`:
+
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ✅ REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ✅ REQUIRED for backward
+    }
+  }
+}
+```
+
+**Implementation**:
+
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    //  Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ✅ Enable reverse pagination
+  maxRecords: 10000,
+});
+
+// Records are returned in reverse chronological order
+console.log(result.data[0].createdAt); // Newest
+console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
+```
+
+**Key Differences from Forward Pagination**:
+
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest               | Newest → Oldest      |
+
+**Important Notes**:
+
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+
+**Example: Extract Latest 1000 Orders**
+
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+
+**When to Use Forward vs Backward**:
+
+```typescript
+// ✅ Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+
+// ✅ Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+
+**Pagination Variables Reference**:
+
+| Variable | Forward      | Backward     | Injected By  | Notes                    |
+| -------- | ------------ | ------------ | ------------ | ------------------------ |
+| `first`  | ✅ Used      |  Not used | Orchestrator | From `pageSize`          |
+| `after`  | ✅ Used      |  Not used | Orchestrator | From cursor (last edge)  |
+| `last`   |  Not used | ✅ Used      | Orchestrator | From `pageSize`          |
+| `before` |  Not used | ✅ Used      | Orchestrator | From cursor (first edge) |
+
+**Common Mistakes to Avoid**:
+
+```typescript
+//  WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, //  Orchestrator will override this
+    before: cursor, //  Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+
+// ✅ CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ✅ Your business variables only
+  },
+  pageSize: 200, // ✅ Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+
+#### Optional: Reverse Pagination
+
+- Forward remains default. For reverse, require $last/$before and pageInfo.hasPreviousPage.
+- Do not pass last/before in variables; set direction='backward'.
+
+GraphQL:
+
+```graphql
+query GetOrdersBackward($retailerId: ID!, $last: Int!, $before: String) {
+  orders(retailerId: $retailerId, last: $last, before: $before) {
+    edges {
+      cursor
+      node {
+        id
+        ref
+        updatedOn
+      }
+    }
+    pageInfo {
+      hasPreviousPage
+    }
+  }
+}
+```
+
+SDK:
+
+```typescript
+await orchestrator.extract({
+  query: ORDERS_BACKWARD_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: { retailerId },
+  pageSize,
+  direction: 'backward',
+});
+```
+
+---
+
+## Testing Checklist
+
+**Before production deployment:**
+
+### 1. Schema Validation
+
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/orders.export.xml.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/orders.export.xml.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+
+### 2. Extraction Testing
+
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+
+### 3. Mapping Testing
+
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+
+### 4. XML Generation Testing
+
+- [ ] Verify XML structure matches expected format
+- [ ] Test XML validation against XSD schema (if applicable)
+- [ ] Verify special character escaping in XML
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test XML namespace handling (if applicable)
+
+### 5. SFTP Upload Testing
+
+- [ ] Test SFTP connection and authentication
+- [ ] Verify file upload to correct path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify file permissions on SFTP server
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify SFTP connection disposal (no connection leaks)
+
+### 6. State Management Testing
+
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+
+### 7. Job Tracking Testing
+
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+
+### 8. Error Handling Testing
+
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with SFTP connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+
+### 9. Staging Environment Testing
+
+- [ ] Run full extraction in staging environment
+- [ ] Verify XML file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+
+### 10. Integration Testing
+
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger → extract → transform → upload → verify file
+
+---
+## Monitoring & Alerting
+
+### Success Response Example
+
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_ORD_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "orders-2025-11-02T14-00-00-000Z.xml",
+  "sftpPath": "/outbound/orders/orders-2025-11-02T14-00-00-000Z.xml",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+
+### Error Response Example
+
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_ORD_20251102_140500_xyz789",
+  "error": "SFTP upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "sftp_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+
+### Key Metrics to Track
+
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+
+  // File Generation
+  fileSizeMB: (xmlContent.length / (1024 * 1024)).toFixed(2),
+
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+
+log.info('Extraction metrics', metrics);
+```
+
+### Alert Thresholds
+
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+
+### Monitoring Dashboard Queries
+
+**Versori Platform Logs Query:**
+
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+
+# High error rates
+errorRate:>5
+
+# State management issues
+stateUpdated:false AND success:true
+```
+
+### Common Issues and Solutions
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+
+**Issue**: "SFTP connection timeout"
+
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check SFTP credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+
+**Issue**: "State not updating after successful extraction"
+
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+
+**Issue**: "First run exceeds record limits"
+
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+
+**Issue**: "Excessive duplicate records in output"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+
+---
+
+## Troubleshooting Quick Reference
+
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "XML generation failed" | Format-specific error | Check XML generation logic, validate output |
+
+---