@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-sftp-xml.md

@@ -1,2445 +1,2445 @@
----
-template_id: tpl-extract-products-to-sftp-xml
-canonical_filename: template-extraction-products-to-sftp-xml.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: sftp-xml
-entity: products
-format: xml
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
-  - dispose-finally
----
-
-# Template: Extraction - Products to SFTP XML
-
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Memory Management** - Clear large result sets after processing batches
-- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
-- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
-- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
-
-## Installation
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk@latest
-```
-
-Use the latest SDK version to ensure compatibility with this template.
-
----
-
-## 📚 STEP 1: Load These Docs (Human Checklist)
-
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
----
-
-## 📋 Implementation Prompt
-
-```
-Create a Versori scheduled extractor for products that uses ExtractionOrchestrator + JobTracker, incremental updatedOn with a 60s overlap buffer, transforms via UniversalMapper, generates XML with XMLBuilder, uploads to SFTP using SftpDataSource with dispose(). Include 3 workflows: scheduled, ad-hoc webhook, and job-status query with native Versori logging.
-```
-
----
-
-## 📦 SDK Imports (Verified - Versori Optimized)
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-
----
-
-# Versori Scheduled: Products Extraction to SFTP XML (Incremental)
-
-**FC Connect SDK Use Case Guide**
-
-> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> Version: Use ^0.1.39 - `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
-
-Context: Scheduled Versori workflow that extracts product catalog from Fluent Commerce via GraphQL query with **ExtractionOrchestrator**, **JobTracker**, and **incremental timestamp tracking**, transforms with `UniversalMapper`, and writes **XML files** to partner SFTP server for marketplace/partner integrations (Amazon, eBay, distributors).
-
-**Pattern**: EXTRACTION (Fluent → SFTP XML)
-**Complexity**: High | Runtime: Versori Platform (Scheduled)
-
----
-
-## ⚠️ IMPORTANT: Production-Ready Base Template
-
-> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
->
-> This is a **production-ready base template** demonstrating FC Connect SDK best practices for product extraction workflows with XML output.
->
-> **✅ INCLUDED FEATURES:**
->
-> - ✅ Comprehensive error handling with retry logic
-> - ✅ SFTP upload with exponential backoff (3 attempts)
-> - ✅ State management with overlap buffer (prevents missed records)
-> - ✅ Job tracking with lifecycle management
-> - ✅ Security (credential masking in logs)
-> - ✅ UTC time enforcement (prevents timezone bugs)
-> - ✅ Incremental extraction (safe, efficient, production-ready)
-> - ✅ Natural rate limiting via timestamps
->
-> **📝 BEFORE DEPLOYING:**
->
-> 1. Review and customize activation variables for your environment
-> 2. Test with sample data in your Versori workspace
-> 3. Adjust safety limits (pageSize, maxRecords) if needed
-> 4. Configure monitoring alerts for extraction failures
-> 5. Verify SFTP credentials and paths
->
-> **This base template follows SDK best practices - tweak specific to your needs.**
-
----
-
-## What You'll Build
-
-- **Incremental extraction** using `updatedOn >= (lastRunTime - buffer)` with **overlap buffer**
-- **ExtractionOrchestrator** for auto-pagination and path-based extraction
-- **JobTracker** for lifecycle management and status tracking
-- **State management** with VersoriKV to track last successful run
-- **Safety buffer** (60 seconds) to handle clock skew and race conditions
-- GraphQL query for product catalog (SKU, title, description, pricing)
-- `UniversalMapper` transformation for partner schema
-- XML file generation with product catalog data
-- **SFTP upload** to partner server (with `dispose()` cleanup)
-- **3 workflow patterns**: scheduled, ad-hoc webhook, job status query
-- **Failure recovery** with timestamp tracking
-
-## Business Use Case
-
-**Daily product catalog feed to marketplace/partner:**
-
-- Extract new and updated products since last run
-- Generate XML file with product data for partner consumption
-- Upload to partner SFTP server for marketplace integration
-- Run daily to keep product catalog synchronized
-- Support product updates (price changes, inventory status)
-- Standard XML format for EDI/ERP integration
-
-## SDK Methods Used
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-
-await createClient(ctx); // Versori-aware client
-const orchestrator = new ExtractionOrchestrator(client, log); // Auto-pagination
-const tracker = new JobTracker(kv, log); // Job lifecycle tracking
-await orchestrator.extract({ query, resultPath, variables, pageSize, maxRecords }); // Extract
-new VersoriKVAdapter(ctx.openKv(':project:')); // State management
-new UniversalMapper(exportMapping); // Field transformation
-new XMLBuilder(options); // XML generation with auto-escaping
-await sftp.uploadFile(remotePath, buffer); // SFTP upload
-await sftp.dispose(); // CRITICAL: Connection cleanup
-```
-
-## SFTP Connection Setup & Credential Access
-
-### Method 1: Versori Connections (Recommended)
-
-**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
-
-**Connection Configuration:**
-
-1. In Versori platform, create a connection named `versori_ftp_server`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-
-**Access Method: `activation.connections` (Recommended)**
-
-```typescript
-import { Buffer } from 'node:buffer'; // ⚠️ CRITICAL: Required for Deno/Versori runtime
-
-// Get SFTP credentials from Versori connection (Basic Auth)
-// RECOMMENDED: Use activation.connections (already decoded)
-const allConnections = ctx.activation.connections || [];
-const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-if (!sftpConn) {
-  throw new Error('SFTP connection "versori_ftp_server" not found');
-}
-
-const credential = sftpConn.credentials[0]?.credential;
-if (!credential?.data?.basicAuth) {
-  throw new Error('SFTP connection not configured with Basic Authentication');
-}
-
-const { username, password } = credential.data.basicAuth;
-// ✅ Already decoded - no Buffer.from() needed!
-```
-
-**Alternative: `credentials().getAccessToken()` (Explicit)**
-
-```typescript
-import { Buffer } from 'node:buffer'; // ⚠️ CRITICAL: Required for Deno/Versori runtime
-
-const sftpCred = await ctx.credentials().getAccessToken('versori_ftp_server');
-const rawAccessToken = sftpCred.accessToken;
-const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-```
-
-**Why use connections instead of activation variables?**
-
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection can be reused across workflows
-- ✅ No need to manage sensitive data in activation variables
-- ✅ Easier credential rotation
-- ✅ Centralized credential management across projects
-
-**🔍– Complete Guide:** See `docs/02-CORE-GUIDES/data-sources/sftp-credential-access-security.md` for comprehensive security patterns and credential management best practices.
-
-### Method 2: Activation Variables (Alternative)
-
-Store credentials directly in activation variables (less secure):
-
-```typescript
-const sftpUsername = ctx.activation?.getVariable('sftpUsername');
-const sftpPassword = ctx.activation?.getVariable('sftpPassword');
-```
-
-**When to use activation variables:**
-
-- Quick prototyping or testing
-- Non-production environments
-- Single-use credentials
-
-**⚠️ Security Warning:** Activation variables are less secure than Versori connections. Always prefer connection-based credential storage for production.
-
-### Buffer Import for Deno/Versori (CRITICAL)
-
-**⚠️ ALWAYS import Buffer** when using SFTP operations in Versori/Deno runtime:
-
-```typescript
-import { Buffer } from 'node:buffer';
-```
-
-**Why?** Unlike Node.js where `Buffer` is global, Deno requires explicit imports from Node.js built-ins using the `node:` prefix.
-
-**Common use cases:**
-
-- SFTP uploads: `Buffer.from(content, 'utf8')`
-- Base64 decoding: `Buffer.from(str, 'base64').toString('utf-8')`
-- Binary data: `Buffer.from(data)`
-
-**Error:** `Buffer is not defined` →' Add `import { Buffer } from 'node:buffer';`
-
-## Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-```json
-{
-  "retailerId": "your-retailer-id",
-  "sftpHost": "sftp.partner.com",
-  "sftpPort": 22,
-  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
-  "sftpRemotePath": "/incoming/products/",
-  "pageSize": 200,
-  "maxRecords": 50000,
-  "fallbackStartDate": "2024-01-01T00:00:00Z",
-  "overlapBufferSeconds": "60",
-  "productStatus": "ACTIVE"
-}
-```
-
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection (see SFTP Connection Setup above).
-
-## Export Mapping Configuration
-
-**IMPORTANT**: Fields match CSV version exactly for consistency.
-
-Create file: `./config/products.export.xml.json`
-
-```json
-{
-  "name": "products.export.xml",
-  "version": "1.0.0",
-  "description": "Fluent Products → Partner SFTP XML Export",
-  "fields": {
-    "sku": { "source": "ref", "required": true, "resolver": "sdk.trim" },
-    "title": { "source": "name", "required": true, "resolver": "sdk.trim" },
-    "description": { "source": "summary", "required": false, "resolver": "sdk.trim" },
-    "gtin": { "source": "gtin", "required": false, "resolver": "sdk.trim" },
-    "type": { "source": "type", "required": false, "resolver": "sdk.uppercase" },
-    "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
-    "price": { "source": "price", "required": false, "resolver": "sdk.parseFloat" },
-    "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
-    "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
-    "created_on": { "source": "createdOn", "required": true, "resolver": "sdk.toString" },
-    "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
-  }
-}
-```
-
-## Mapping & Resolvers Explained
-
-### SDK Resolvers Used
-
-The export mapping uses **SDK resolvers** to transform GraphQL data into the target XML format:
-
-| Field           | Resolver         | Why?                                         | Example Transformation                  |
-| --------------- | ---------------- | -------------------------------------------- | --------------------------------------- |
-| `sku`           | `sdk.trim`       | Remove leading/trailing whitespace from SKUs | `" ABC-123 "` → `"ABC-123"`          |
-| `title`         | `sdk.trim`       | Clean product names                          | `"Widget  "` → `"Widget"`            |
-| `description`   | `sdk.trim`       | Clean descriptions                           | `"  Description"` → `"Description"`  |
-| `gtin`          | `sdk.trim`       | Clean barcode numbers                        | `" 012345678901"` → `"012345678901"` |
-| `type`          | `sdk.uppercase`  | Normalize product type codes                 | `"standard"` → `"STANDARD"`          |
-| `status`        | `sdk.uppercase`  | Normalize status values                      | `"active"` → `"ACTIVE"`              |
-| `price`         | `sdk.parseFloat` | Parse price as decimal                       | `"29.99"` → `29.99`                  |
-| `catalogue_ref` | `sdk.trim`       | Clean catalogue references                   | `" CAT-001 "` → `"CAT-001"`          |
-| `created_on`    | `sdk.toString`   | Ensure timestamp is string                   | `Date` → `"2025-01-21T10:30:00Z"`    |
-| `updated_on`    | `sdk.toString`   | Ensure timestamp is string                   | `Date` → `"2025-01-21T10:30:00Z"`    |
-
-### Transformation Flow
-
-```typescript
-// 1. GraphQL Response (from Fluent API)
-{
-  ref: " SKU-001 ",           // → Has whitespace
-  name: "Premium Widget  ",   // → Has trailing space
-  type: "standard",           // → Lowercase
-  status: "active",           // → Lowercase
-  price: "29.99",             // → String
-  catalogue: {
-    ref: " CAT-001 ",
-    name: " Default Catalogue "
-  },
-  updatedOn: "2025-01-21T10:30:00Z"
-}
-
-// 2. UniversalMapper applies resolvers
-const mapper = new UniversalMapper(productsExportMapping);
-const result = await mapper.map(node);
-
-// 3. Transformed Output (clean, normalized)
-result.data = {
-  sku: "SKU-001",             // ✅ Trimmed
-  title: "Premium Widget",    // ✅ Trimmed
-  type: "STANDARD",           // ✅ Uppercased
-  status: "ACTIVE",           // ✅ Uppercased
-  price: 29.99,               // ✅ Float
-  catalogue_ref: "CAT-001",   // ✅ Trimmed from nested object
-  catalogue_name: "Default Catalogue", // ✅ Trimmed from nested object
-  updated_on: "2025-01-21T10:30:00Z"
-}
-```
-
-### Custom Resolvers for Product-Specific Logic
-
-You can add **custom resolvers** for business-specific transformations:
-
-```typescript
-const productsExportMapping = {
-  name: 'products.export.xml',
-  version: '1.0.0',
-  fields: {
-    sku: { source: 'ref', required: true, resolver: 'sdk.trim' },
-    title: { source: 'name', required: true, resolver: 'sdk.trim' },
-
-    // Custom resolver: Calculate display price with tax
-    display_price: {
-      source: 'price',
-      resolver: 'custom.calculateDisplayPrice',
-    },
-
-    // Custom resolver: Map internal categories to partner categories
-    partner_category: {
-      source: 'category',
-      resolver: 'custom.mapCategory',
-    },
-
-    // Custom resolver: Generate SEO-friendly URL slug
-    url_slug: {
-      source: 'name',
-      resolver: 'custom.generateSlug',
-    },
-  },
-};
-
-// Custom resolver implementations
-const customResolvers = {
-  'custom.calculateDisplayPrice': (price: number) => {
-    const TAX_RATE = 0.1;
-    return (price * (1 + TAX_RATE)).toFixed(2);
-  },
-
-  'custom.mapCategory': (category: string) => {
-    const categoryMap: Record<string, string> = {
-      ELECTRONICS: 'Electronics & Gadgets',
-      APPAREL: 'Clothing & Fashion',
-      HOME: 'Home & Living',
-    };
-    return categoryMap[category] || 'General';
-  },
-
-  'custom.generateSlug': (name: string) => {
-    return name
-      .toLowerCase()
-      .replace(/[^\w\s-]/g, '')
-      .replace(/\s+/g, '-');
-  },
-};
-
-// Use with UniversalMapper
-const mapper = new UniversalMapper(productsExportMapping, { customResolvers });
-```
-
-### Available SDK Resolvers
-
-**String Transformations:**
-
-- `sdk.trim` - Remove whitespace
-- `sdk.uppercase` - Convert to uppercase
-- `sdk.lowercase` - Convert to lowercase
-- `sdk.toString` - Convert to string
-
-**Number Transformations:**
-
-- `sdk.parseInt` - Parse integer
-- `sdk.parseFloat` - Parse decimal
-- `sdk.number` - Generic number conversion
-
-**Date Transformations:**
-
-- `sdk.formatDate` - ISO 8601 format (`2025-01-22T14:30:00Z`)
-- `sdk.formatDateShort` - Short date format (`2025-01-22`)
-- `sdk.parseDate` - Parse date string
-
-**Type Conversions:**
-
-- `sdk.boolean` - Convert to boolean
-- `sdk.parseJson` - Parse JSON string
-- `sdk.toJson` - Convert to JSON string
-
-**Utility:**
-
-- `sdk.identity` - Pass through unchanged
-- `sdk.coalesce` - Return first non-null value
-
-See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
-
-## GraphQL Query
-
-```graphql
-query GetProducts(
-  $catalogues: [ProductCatalogueKey]
-  $dateRangeFilter: DateRange
-  $first: Int!
-  $after: String
-) {
-  products(catalogueRef: $catalogues, updatedOn: $dateRangeFilter, first: $first, after: $after) {
-    edges {
-      node {
-        id
-        ref
-        name
-        type
-        status
-        gtin
-        price
-        attributes
-        catalogue {
-          ref
-          name
-        }
-        createdOn
-        updatedOn
-      }
-      cursor
-    }
-    pageInfo {
-      hasNextPage
-    }
-  }
-}
-```
-
-## Expected XML Output
-
-**IMPORTANT**: XML structure with same fields as CSV version for consistency.
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<Products>
-  <Product>
-    <sku>SKU-001</sku>
-    <title>Premium Widget</title>
-    <description>High-quality widget for all purposes</description>
-    <gtin>012345678901</gtin>
-    <type>STANDARD</type>
-    <status>ACTIVE</status>
-    <price>29.99</price>
-    <catalogue_ref>CAT-001</catalogue_ref>
-    <catalogue_name>Default Catalogue</catalogue_name>
-    <created_on>2025-01-21T10:30:00Z</created_on>
-    <updated_on>2025-01-21T10:30:00Z</updated_on>
-  </Product>
-  <Product>
-    <sku>SKU-002</sku>
-    <title>Deluxe Gadget</title>
-    <description>Advanced gadget with premium features</description>
-    <gtin>012345678902</gtin>
-    <type>STANDARD</type>
-    <status>ACTIVE</status>
-    <price>49.99</price>
-    <catalogue_ref>CAT-001</catalogue_ref>
-    <catalogue_name>Default Catalogue</catalogue_name>
-    <created_on>2025-01-21T14:15:00Z</created_on>
-    <updated_on>2025-01-21T14:15:00Z</updated_on>
-  </Product>
-</Products>
-```
-
-**Note**: XML preserves hierarchical structure unlike CSV which flattens to rows.
-
-## Production Safety & Guardrails
-
-### Overview
-
-Product catalogs require strict guardrails even with incremental extraction:
-
-- **Large initial extractions**: First run can include entire catalog (100k+ products)
-- **Bulk updates**: Marketing campaigns can update thousands of products at once
-- **XML overhead**: 2-3x larger than JSON/CSV for same data
-- **SFTP limits**: Partner servers may reject large files
-- **Memory pressure**: Product records are larger (descriptions, attributes)
-
-### Hard Limits
-
-```typescript
-const SAFETY_LIMITS = {
-  MAX_RECORDS_PER_RUN: 50000, // 50k products per run (XML overhead)
-  MAX_RECORDS_PER_FILE: 10000, // 10k per XML file (SFTP-friendly)
-  MAX_FILE_SIZE_MB: 150, // 150MB per file
-  MAX_XML_SIZE_MB: 300, // Total extraction size
-  CHUNK_SIZE: 5000, // Process in chunks
-  ESTIMATED_BYTES_PER_PRODUCT_XML: 3000, // 3KB per product in XML (conservative)
-};
-```
-
-**Why different from JSON?**
-
-- XML has 2-3x size overhead (tags, attributes, whitespace)
-- Products have rich text content (descriptions, attributes)
-- Partner SFTP servers often have smaller file size limits than S3
-- 10k products per file = ~30MB (manageable for most SFTP systems)
-
-### Runtime Validation Function
-
-```typescript
-/**
- * Validate extraction safety limits before processing
- * CRITICAL: Account for XML size overhead vs CSV
- */
-function validateExtractionLimits(productCount: number) {
-  const MAX_PRODUCTS_PER_RUN = 50000;
-  const ESTIMATED_BYTES_PER_PRODUCT_XML = 3000; // Full XML product element
-  const estimatedSizeMB = (productCount * ESTIMATED_BYTES_PER_PRODUCT_XML) / (1024 * 1024);
-  const MAX_XML_SIZE_MB = 300;
-
-  if (productCount > MAX_PRODUCTS_PER_RUN) {
-    return {
-      valid: false,
-      error: `Extraction limit exceeded: ${productCount} products (max: ${MAX_PRODUCTS_PER_RUN})`,
-      recommendation: `Too many products for single extraction. Consider:
-        1. Increase extraction frequency (daily → hourly)
-        2. Add product status filters (ACTIVE only)
-        3. Split by catalogue
-        4. Contact support if consistently exceeding limits`,
-      productCount,
-      maxAllowed: MAX_PRODUCTS_PER_RUN,
-    };
-  }
-
-  if (estimatedSizeMB > MAX_XML_SIZE_MB) {
-    return {
-      valid: false,
-      error: `XML size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_XML_SIZE_MB}MB)`,
-      recommendation:
-        'File splitting required. Increase extraction frequency to reduce batch size.',
-      estimatedSizeMB,
-      maxAllowed: MAX_XML_SIZE_MB,
-    };
-  }
-
-  return { valid: true };
-}
-```
-
----
-
-## Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-products-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-products-extraction.ts  # Scheduled: Daily products extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-products-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts          # Webhook: Status query
-    │
-    ├── services/
-    │   └── products-extraction.service.ts   # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── products.export.xml.json         # Mapping configuration
-```
-
----
-
-## Complete Workflow Code
-
-The code below demonstrates the implementation of each component in the modular structure.
-
-### 1. Entry Point (`index.ts`)
-
-```typescript
-/**
- * Entry point - Export all workflows for Versori platform
- *
- * This file exports all workflows to be registered with Versori.
- * Each workflow is defined in its own file for better organization.
- */
-
-// Scheduled workflows
-export { dailyProductsExtraction } from './src/workflows/scheduled/daily-products-extraction';
-
-// Webhook workflows
-export { adhocProductsExtraction } from './src/workflows/webhook/adhoc-products-extraction';
-export { productsExtractionJobStatus } from './src/workflows/webhook/job-status-check';
-```
-
-### 2. Workflows (src/workflows/products-extraction.ts)
-
-```typescript
-// ⚠️ IMPORTANT: Do NOT import openKv - access it from context!
-import { schedule, webhook, http, fn } from '@versori/run';
-import {
-  executeProductExtraction,
-  getJobStatus,
-  generateJobId,
-} from '../services/products-extraction.service';
-
-// 
-// WORKFLOW 1: Scheduled Extraction (Daily at 2 AM)
-// 
-
-export const scheduledProductsExtraction = schedule('products-extract-xml-daily', '0 2 * * *').then(
-  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async ctx => {
-    const jobId = generateJobId('SCHED', 'PRODUCTS');
-
-    const result = await executeProductExtraction(ctx, {
-      jobId,
-      triggeredBy: 'schedule',
-      updateState: true, // Always update state for scheduled runs
-    });
-
-    return result;
-  })
-);
-
-// 
-// WORKFLOW 2: Ad-hoc Webhook Extraction
-// 
-
-export const adhocProductsExtraction = webhook('products-adhoc', {
-  connection: 'products-adhoc',
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-}).then(
-  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async ctx => {
-    // Security is enforced by the 'products-adhoc' connection
-    const { log } = ctx;
-    const jobId = generateJobId('ADHOC', 'PRODUCTS');
-
-    log.info('🚀 [WEBHOOK] Adhoc products extraction triggered', {
-      jobId,
-      fromDate: ctx.data.fromDate,
-      toDate: ctx.data.toDate,
-      updateState: ctx.data.updateState,
-    });
-
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductExtraction(ctx, {
-      jobId,
-      triggeredBy: 'webhook',
-      fromDate: ctx.data.fromDate,
-      toDate: ctx.data.toDate,
-      updateState: ctx.data.updateState === true,
-    })
-      .then((result) => {
-        log.info('✅ [BACKGROUND] Products extraction completed successfully', {
-          jobId,
-          recordCount: result.recordCount,
-          fileName: result.fileName,
-        });
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        log.error('❌ [BACKGROUND] Products extraction failed', {
-          jobId,
-          error: errorMessage,
-          stack: error instanceof Error ? error.stack : undefined,
-        });
-      });
-
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Products extraction started in background',
-      statusEndpoint: `https://{workspace}.versori.run/products-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-
-// 
-// WORKFLOW 3: Job Status Query
-// 
-
-export const productsJobStatus = webhook('products-job-status', {
-  connection: 'products-job-status',
-  response: { mode: 'sync' },
-}).then(
-  fn('query-job-status', async ctx => {
-    const { data, log, openKv } = ctx;
-    // Security is enforced by the 'products-job-status' connection
-
-    const jobId = data.jobId;
-    if (!jobId) {
-      return { success: false, error: 'Job ID required' };
-    }
-
-    const status = await getJobStatus(openKv(':project:'), jobId, log);
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
-
-### 3. Main Orchestration Service (`src/services/products-extraction.service.ts`)
-
-**Note:** This service file should be renamed from `extraction-orchestration.ts` to `products-extraction.service.ts` to match the new workflow structure.
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  XMLBuilder,
-  SftpDataSource,
-  VersoriKVAdapter,
-} from '@fluentcommerce/fc-connect-sdk';
-import productsExportMapping from '../../config/products.export.xml.json' with { type: 'json' };
-
-const PRODUCTS_EXTRACTION_QUERY = `
-  query GetProducts(
-    $catalogues: [ProductCatalogueKey]
-    $dateRangeFilter: DateRange
-    $first: Int!
-    $after: String
-  ) {
-    products(
-      catalogueRef: $catalogues
-      updatedOn: $dateRangeFilter
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          name
-          type
-          status
-          gtin
-          price
-          attributes
-          catalogue {
-            ref
-            name
-          }
-          createdOn
-          updatedOn
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-
-// Initialize XMLBuilder for products
-const xmlBuilder = new XMLBuilder({
-  rootElement: 'Products',
-  prettyPrint: true,
-  indent: '  ',
-  xmlDeclaration: true,
-  encoding: 'UTF-8',
-});
-
-function buildProductsXML(products: any[]): string {
-  // Transform to XMLBuilder format
-  const productsForXml = products.map(p => ({
-    sku: p.sku,
-    title: p.title,
-    description: p.description || '',
-    gtin: p.gtin || '',
-    type: p.type || '',
-    status: p.status,
-    price: String(p.price || ''),
-    catalogue_ref: p.catalogue_ref || '',
-    catalogue_name: p.catalogue_name || '',
-    created_on: p.created_on,
-    updated_on: p.updated_on,
-  }));
-
-  return xmlBuilder.build({ Product: productsForXml });
-}
-
-interface ProductExtractionParams {
-  jobId: string;
-  triggeredBy: 'schedule' | 'webhook';
-  fromDate?: string;
-  toDate?: string;
-  updateState: boolean;
-}
-
-export async function executeProductExtraction(ctx: any, options: ProductExtractionParams) {
-  const { jobId, triggeredBy, fromDate, toDate, updateState } = options;
-  const log = ctx.log;
-  const retailerId = ctx.activation?.getVariable('retailerId');
-  const pageSize = parseInt(ctx.activation?.getVariable('pageSize') || '200', 10);
-  const maxRecords = parseInt(ctx.activation?.getVariable('maxRecords') || '50000', 10);
-  const fallbackStartDate =
-    ctx.activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
-  const productStatus = ctx.activation?.getVariable('productStatus') || 'ACTIVE';
-
-  // Get SFTP credentials from Versori connection (Basic Auth)
-  // RECOMMENDED: Use activation.connections (already decoded)
-  const allConnections = ctx.activation.connections || [];
-  const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-  if (!sftpConn) {
-    throw new Error('SFTP connection "versori_ftp_server" not found');
-  }
-
-  const credential = sftpConn.credentials[0]?.credential;
-  if (!credential?.data?.basicAuth) {
-    throw new Error('SFTP connection not configured with Basic Authentication');
-  }
-
-  const { username, password } = credential.data.basicAuth;
-  // ✅ Already decoded - no Buffer.from() needed!
-
-  const sftpSettings = {
-    host: ctx.activation?.getVariable('sftpHost'),
-    port: parseInt(ctx.activation?.getVariable('sftpPort') || '22', 10),
-    username, // From connection (secure)
-    password, // From connection (secure)
-    privateKey: ctx.activation?.getVariable('sftpPrivateKey'),
-    remotePath: ctx.activation?.getVariable('sftpRemotePath') || '/incoming/products/',
-  };
-
-  const missing: string[] = [];
-  if (!retailerId) missing.push('retailerId');
-  if (!sftpSettings.host) missing.push('sftpHost');
-  if (missing.length)
-    return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
-
-  // SFTP connection - MUST use try/finally with dispose()
-  const sftp = new SftpDataSource(
-    {
-      type: 'SFTP_XML',
-      connectionId: 'sftp-products-xml-export',
-      name: 'SFTP Products XML Export',
-      settings: {
-        host: sftpSettings.host,
-        port: sftpSettings.port,
-        username: sftpSettings.username,
-        password: sftpSettings.password,
-        privateKey: sftpSettings.privateKey,
-        remotePath: sftpSettings.remotePath,
-        filePattern: '*.xml',
-      },
-    },
-    log
-  );
-
-  try {
-    // 
-    // STEP 1/8: Initialize Job Tracking
-    // 
-    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
-    const tracker = new JobTracker(kv, log);
-
-    await tracker.createJob(jobId, {
-      triggeredBy,
-      hasDateOverride: !!fromDate,
-      fromDate,
-      toDate,
-      updateStateAfterRun: updateState,
-    });
-
-    log.info('Job created', { jobId, triggeredBy });
-
-    // 
-    // STEP 2/8: Load State & Calculate Time Window
-    // 
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'state_load',
-      message: 'Loading last run state',
-    });
-
-    const stateKey = ['extraction', 'products-xml', 'lastProductSync'];
-    const lastRunState = await kv.get(stateKey);
-    const rawLastRunTime = fromDate || lastRunState?.value?.timestamp || fallbackStartDate;
-
-    // Overlap buffer configuration (default: 60 seconds)
-    const overlapBufferSeconds = parseInt(
-      ctx.activation?.getVariable('overlapBufferSeconds') || '60',
-      10
-    );
-    const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
-
-    // Apply overlap buffer for query (safety window)
-    const bufferedLastRunTime = new Date(
-      new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
-    ).toISOString();
-
-    const effectiveEndTime = toDate || new Date().toISOString();
-
-    log.info('🔍 Time window calculated', {
-      rawLastRunTime,
-      bufferedLastRunTime,
-      effectiveEndTime,
-      overlapBufferSeconds,
-      retailerId,
-      productStatus,
-    });
-
-    // 
-    // STEP 3/8: Initialize Fluent Client & ExtractionOrchestrator
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'client_init',
-      message: 'Initializing Fluent client',
-    });
-
-    const client = await createClient(ctx);
-    const orchestrator = new ExtractionOrchestrator(client, log);
-
-    // 
-    // STEP 4/8: Extract Data (ExtractionOrchestrator)
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'extraction',
-      message: 'Extracting data with auto-pagination',
-    });
-
-    // ? Enhanced: Extract context for progress logging
-    const dateRangeInfo = {
-      start: bufferedLastRunTime || 'N/A',
-      end: effectiveEndTime || 'N/A',
-      catalogues: 'all'
-    };
-
-    // ? Enhanced: Start logging with context
-    log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
-     query: 'products',
-     pageSize,
-     maxRecords,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     catalogues: dateRangeInfo.catalogues,
-     jobId
-    });
-
-    const extractionResult = await orchestrator.extract({
-      query: PRODUCTS_EXTRACTION_QUERY,
-      resultPath: 'products.edges.node',
-      variables: {
-        catalogues: null, // All catalogues
-        dateRangeFilter: {
-          after: bufferedLastRunTime,
-          before: effectiveEndTime, // End of extraction window
-        },
-        first: pageSize,
-      },
-      pageSize,
-      maxRecords,
-      validateItem: item => !!(item.ref && item.name),
-    });
-
-    const rawRecords = extractionResult.data;
-
-    log.info('Extraction complete', {
-      totalRecords: extractionResult.stats.totalRecords,
-      totalPages: extractionResult.stats.totalPages,
-      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-      errors: extractionResult.errors ? extractionResult.errors.length : 0,
-    });
-
-    // ? Enhanced: Completion logging with summary
-    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-     totalRecords: extractionResult.stats.totalRecords,
-     totalPages: extractionResult.stats.totalPages,
-     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-     failedValidations: extractionResult.stats.failedValidations,
-     truncated: extractionResult.stats.truncated,
-     truncationReason: extractionResult.stats.truncationReason,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     jobId
-    });
-
-    if (extractionResult.errors && extractionResult.errors.length > 0) {
-      log.warn('Non-fatal extraction errors encountered', {
-        errorCount: extractionResult.errors.length,
-        sampleErrors: extractionResult.errors.slice(0, 3),
-      });
-    }
-
-    if (rawRecords.length === 0) {
-      await tracker.markCompleted(jobId, {
-        recordCount: 0,
-        message: 'No new products to extract',
-      });
-
-      if (updateState) {
-        await kv.set(stateKey, {
-          timestamp: new Date().toISOString(),
-          productCount: 0,
-          extractedAt: new Date().toISOString(),
-        });
-      }
-
-      return { success: true, message: 'No new products to extract', lastRunTime: rawLastRunTime };
-    }
-
-    // 
-    // STEP 5/8: Validate Extraction Limits
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'validation',
-      message: 'Validating extraction limits',
-    });
-
-    const MAX_PRODUCTS_PER_RUN = 50000;
-    const ESTIMATED_BYTES_PER_PRODUCT_XML = 3000;
-    const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_PRODUCT_XML) / (1024 * 1024);
-    const MAX_XML_SIZE_MB = 300;
-
-    if (rawRecords.length > MAX_PRODUCTS_PER_RUN) {
-      log.error('Extraction limit exceeded', {
-        productCount: rawRecords.length,
-        maxAllowed: MAX_PRODUCTS_PER_RUN,
-      });
-
-      await tracker.markFailed(jobId, {
-        error: `Extraction limit exceeded: ${rawRecords.length} products (max: ${MAX_PRODUCTS_PER_RUN})`,
-        recommendation: 'Increase extraction frequency or add filters',
-      });
-
-      return {
-        success: false,
-        error: `Extraction limit exceeded: ${rawRecords.length} products (max: ${MAX_PRODUCTS_PER_RUN})`,
-        recommendation: `Too many products for single extraction. Consider:
-          1. Increase extraction frequency (daily → hourly)
-          2. Add product status filters (ACTIVE only)
-          3. Split by catalogue
-          4. Contact support if consistently exceeding limits`,
-        productCount: rawRecords.length,
-        maxAllowed: MAX_PRODUCTS_PER_RUN,
-      };
-    }
-
-    if (estimatedSizeMB > MAX_XML_SIZE_MB) {
-      log.warn('XML size approaching limit', {
-        estimatedSizeMB: estimatedSizeMB.toFixed(2),
-        maxAllowed: MAX_XML_SIZE_MB,
-        recommendation: 'Consider file splitting or increase extraction frequency',
-      });
-    }
-
-    log.info('Extraction limits validated', {
-      productCount: rawRecords.length,
-      estimatedSizeMB: estimatedSizeMB.toFixed(2),
-      withinLimits: true,
-    });
-
-    // 
-    // STEP 6/8: Transform Data (UniversalMapper)
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'transformation',
-      message: 'Transforming data with UniversalMapper',
-    });
-
-    const mapper = new UniversalMapper(productsExportMapping);
-    const mappingResult = await mapper.map(rawRecords);
-
-    if (!mappingResult.success) {
-      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-      await tracker.markFailed(jobId, {
-        error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
-        failedCount: mappingErrors.length,
-      });
-      return {
-        success: false,
-        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-        errors: mappingErrors,
-      };
-    }
-
-    const transformedProducts = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-    const mappingErrors = mappingResult.errors || [];
-
-    if (mappingErrors.length > 0) {
-      log.warn('Some products failed transformation', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-    }
-
-    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-        jobId,
-        skippedFields: mappingResult.skippedFields,
-        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-      });
-    }
-
-    if (transformedProducts.length === 0) {
-      await tracker.markFailed(jobId, {
-        error: 'All records failed mapping',
-        failedCount: mappingErrors.length,
-      });
-      return {
-        success: false,
-        error: 'All records failed mapping',
-        errors: mappingErrors,
-      };
-    }
-
-    log.info('Products transformed', {
-      jobId,
-      transformedCount: transformedProducts.length,
-      skippedRecords: rawRecords.length - transformedProducts.length,
-    });
-
-    // 
-    // STEP 7/8: Generate XML & Upload to SFTP
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'upload',
-      message: 'Generating XML and uploading to SFTP',
-    });
-
-    const xmlContent = buildProductsXML(transformedProducts);
-
-    // Generate timestamped filename
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const fileName = `products-${timestamp}.xml`;
-    const remotePath = `${sftpSettings.remotePath}${fileName}`;
-
-    log.info('Generated XML file', {
-      fileName,
-      size: xmlContent.length,
-      productCount: transformedProducts.length,
-    });
-
-    // Upload to SFTP
-    await sftp.uploadFile(remotePath, Buffer.from(xmlContent, 'utf8'));
-
-    log.info('XML file uploaded to SFTP', { remotePath });
-
-    // 
-    // STEP 8/8: Update State & Complete Job
-    // 
-    await tracker.updateJob(jobId, {
-      stage: 'state_update',
-      message: 'Updating state and completing job',
-    });
-
-    // Calculate max updatedOn from extracted products
-    const maxUpdatedOn = transformedProducts.reduce((max, product) => {
-      const productTime = new Date(product.updated_on).getTime();
-      return productTime > max ? productTime : max;
-    }, new Date(rawLastRunTime).getTime());
-
-    const newTimestamp = new Date(maxUpdatedOn).toISOString();
-
-    // Update state with new timestamp (WITHOUT buffer)
-    if (updateState) {
-      await kv.set(stateKey, {
-        timestamp: newTimestamp, // ← NO buffer applied
-        productCount: transformedProducts.length,
-        extractedAt: new Date().toISOString(),
-        overlapBufferSeconds,
-        fileName,
-        remotePath,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      });
-
-      log.info('State updated with new timestamp (without buffer)', {
-        newTimestamp,
-        overlapBufferSeconds,
-      });
-    }
-
-    await tracker.markCompleted(jobId, {
-      recordCount: transformedProducts.length,
-      fileName,
-      sftpPath: remotePath,
-      errorCount: mappingErrors.length,
-      errors: mappingErrors,
-    });
-
-    return {
-      success: true,
-      productsExtracted: transformedProducts.length,
-      fileName,
-      remotePath,
-      lastRunTime: rawLastRunTime,
-      newTimestamp,
-      jobId,
-      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-    };
-  } catch (error: any) {
-    log.error('Extraction failed', error, {
-      message: error?.message,
-    });
-
-    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
-    const tracker = new JobTracker(kv, log);
-
-    await tracker.markFailed(jobId, {
-      message: error instanceof Error ? error.message : String(error),
-
-      stack: error instanceof Error ? error.stack : undefined,
-
-      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-    });
-
-    return {
-      success: false,
-      message: error instanceof Error ? error.message : String(error),
-
-      stack: error instanceof Error ? error.stack : undefined,
-
-      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-      jobId,
-    };
-  } finally {
-    // CRITICAL: Always clean up SFTP connections
-    await sftp.dispose();
-    log.info('SFTP connection disposed');
-  }
-}
-
-export async function getJobStatus(kv: any, jobId: string, log: any) {
-  const tracker = new JobTracker(new VersoriKVAdapter(kv), log);
-  return await tracker.getJob(jobId);
-}
-```
-
-### 4. Job ID Generator (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Generate unique job ID
- * Format: {PREFIX}-{ENTITY}-{TIMESTAMP}
- */
-export function generateJobId(prefix: 'SCHED' | 'ADHOC', entity: string): string {
-  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-  return `${prefix}-${entity}-${timestamp}`;
-}
-```
-
-### 5. Package Configuration (package.json)
-
-```json
-{
-  "name": "products-extraction-to-sftp-xml",
-  "version": "1.0.0",
-  "description": "Versori connector for products extraction to SFTP XML",
-  "main": "dist/index.js",
-  "type": "module",
-  "scripts": {
-    "build": "tsc",
-    "dev": "tsc --watch",
-    "lint": "eslint src/**/*.ts",
-    "test": "jest"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-
-### 6. Deployment Instructions
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Build the connector
-npm run build
-
-# 3. Test locally (optional)
-npm test
-
-# 4. Deploy to Versori
-# - Upload to Versori workspace
-# - Configure activation variables
-# - Enable workflows
-
-# 5. Test workflows
-# Scheduled: Wait for next cron trigger or manually trigger
-# Ad-hoc: POST to webhook URL with API key header
-# Status: Query job status by ID
-```
-
-### 7. Testing
-
-#### Test Scheduled Extraction
-
-```bash
-# Trigger manually in Versori UI or wait for cron schedule
-# Expected: XML file uploaded to SFTP
-```
-
-#### Test Ad-hoc Extraction
-
-```bash
-curl -X POST https://your-workspace.versori.run/products-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-22T23:59:59Z",
-    "updateState": false
-  }'
-```
-
-#### Test Job Status Query
-
-```bash
-curl -X POST https://your-workspace.versori.run/products-job-status \
-  -H "Content-Type: application/json" \
-  -d '{
-    "jobId": "SCHED-PRODUCTS-2025-01-22T02-00-00Z"
-  }'
-```
-
-## Key Patterns Explained
-
-### Pattern 1: ExtractionOrchestrator for Auto-Pagination
-
-```typescript
-// ✅ CORRECT - Use ExtractionOrchestrator (handles pagination automatically)
-const orchestrator = new ExtractionOrchestrator(client, log);
-
-const extractionResult = await orchestrator.extract({
-  query: PRODUCTS_EXTRACTION_QUERY,
-  resultPath: 'products.edges.node',
-  variables: { dateRangeFilter: { after: bufferedLastRunTime } },
-  pageSize,
-  maxRecords,
-  validateItem: item => !!(item.ref && item.name),
-});
-
-const records = extractionResult.data;
-
-//  WRONG - Manual pagination (avoid this pattern)
-// const result = await client.graphql({
-//   query: PRODUCTS_QUERY,
-//   variables: { first: pageSize },
-//   pagination: { maxRecords }
-// });
-```
-
-### Pattern 2: JobTracker for Lifecycle Management
-
-```typescript
-// ✅ CORRECT - Use JobTracker throughout workflow
-const tracker = new JobTracker(kv, log);
-
-// Create job
-await tracker.createJob(jobId, { triggeredBy, fromDate, toDate });
-
-// Update progress
-await tracker.updateJob(jobId, { stage: 'extraction', message: 'Extracting data' });
-
-// Mark completed
-await tracker.markCompleted(jobId, { recordCount, fileName });
-
-// Query status
-const status = await tracker.getJob(jobId);
-```
-
-### Pattern 3: 3-Workflow Pattern
-
-```typescript
-// ✅ CORRECT - 3 workflows for different use cases
-// 1. Scheduled: Automated daily/hourly runs
-export const scheduledProductsExtraction = schedule('products-extract-xml-daily', '0 2 * * *').then(...)
-
-// 2. Ad-hoc: Manual webhook triggers with date overrides
-export const adhocProductsExtraction = webhook('products-adhoc', {
-  connection: 'products-adhoc',
-  response: { mode: 'sync' },
-}).then(...)
-
-// 3. Status: Query job status by ID
-export const productsJobStatus = webhook('products-job-status', {
-  connection: 'products-job-status',
-  response: { mode: 'sync' },
-}).then(...)
-```
-
-### Pattern 4: XMLBuilder for Safe XML Generation (CRITICAL)
-
-Use the SDK's `XMLBuilder` - it handles all XML escaping automatically:
-
-```typescript
-import { Buffer } from 'node:buffer';
-import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
-
-// Initialize XMLBuilder (handles all escaping automatically)
-const xmlBuilder = new XMLBuilder({
-  rootElement: 'Products',
-  prettyPrint: true,
-  encoding: 'UTF-8',
-});
-
-// ✅ CORRECT: XMLBuilder escapes automatically
-const products = [
-  {
-    title: 'Smith & Jones <Corp>', // Contains & and <>
-    description: 'Special chars: ¢, ©, ®, "quotes"',
-  },
-];
-
-const xml = xmlBuilder.build({ Product: products });
-// Result: All special characters properly escaped
-// <title>Smith &amp; Jones &lt;Corp&gt;</title>
-// <description>Special chars: ¢, ©, ®, &quot;quotes&quot;</description>
-
-//  WRONG: Manual string concatenation (dangerous)
-// const xml = `<title>${product.title}</title>`;
-// This would produce INVALID XML: <title>Smith & Jones <Corp></title>
-```
-
-**Why XMLBuilder?**
-
-- ✅ Automatic escaping of &, <, >, ", '
-- ✅ Handles special characters (¢, ©, ®)
-- ✅ Prevents XML injection attacks
-- ✅ Validates structure
-- ✅ Consistent, maintainable code
-
-### Pattern 5: SFTP Cleanup (CRITICAL)
-
-```typescript
-const sftp = new SftpDataSource(config, log);
-
-try {
-  await sftp.uploadFile(remotePath, buffer);
-  return { success: true };
-} finally {
-  // ALWAYS dispose SFTP connection
-  await sftp.dispose();
-}
-```
-
-**Why?** SFTP maintains open connections. Not calling `dispose()` leads to connection exhaustion.
-
-### Pattern 6: Consistent Field Names Across Formats
-
-**Same data in CSV, JSON, and XML:**
-
-- `sku` (not productId, not sku_ref, not SKU)
-- `title` (consistent with CSV version)
-- `catalogue_ref` (matches CSV exactly)
-
-This allows users to switch formats without changing downstream systems.
-
----
-
-### Pattern 7: State Management & Date Overrides
-
-**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
-
-**How it works**:
-
-VersoriKV stores the last successful extraction timestamp to enable incremental sync:
-
-```typescript
-interface ExtractionState {
-  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
-  recordCount: number; // Number of records extracted
-  extractedAt: string; // When extraction completed
-  fileName?: string; // Generated filename
-  remotePath?: string; // SFTP upload path
-  overlapBufferSeconds?: number; // Buffer configuration
-}
-```
-
-**State Priority Chain** (highest to lowest):
-
-1. **`fromDate` override** (manual date in webhook payload) - Highest priority
-2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
-3. **`fallbackStartDate`** (activation variable) - First run fallback
-
-**Three Scenarios**:
-
-#### Scenario 1: Normal Scheduled Runs (Incremental)
-
-```typescript
-// Payload: {} (empty - no overrides)
-
-// Behavior:
-// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
-// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
-// 3. Extract records updated since buffered time
-// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
-// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
-// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
-```
-
-**Test**:
-
-```bash
-# Trigger scheduled run (no payload needed)
-# State advances automatically
-curl -X POST https://workspace.versori.run/products-extract-daily
-```
-
-#### Scenario 2: Ad-hoc Extraction WITH State Update
-
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
-
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
-// 3. Extract all records since 2025-01-01
-// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
-// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
-// 6. Next scheduled run starts from this new timestamp
-```
-
-**Use Case**: One-time catch-up extraction that advances the state pointer.
-
-**Test**:
-
-```bash
-curl -X POST https://workspace.versori.run/products-extract-webhook \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "updateState": true
-  }'
-```
-
-#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
-
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
-
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z"
-// 3. Extract all records since 2025-01-01
-// 4. DO NOT update state
-// 5. Next scheduled run uses previous timestamp (unaffected)
-```
-
-**Use Case**: Historical backfill or testing without affecting incremental sync.
-
-**Test**:
-
-```bash
-curl -X POST https://workspace.versori.run/products-extract-webhook \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-31T23:59:59Z",
-    "updateState": false
-  }'
-```
-
-**Why this matters**:
-
-- **Incremental sync** relies on state continuity
-- **Manual overrides** allow catch-up without breaking incremental flow
-- **Overlap buffer** prevents missed records at time boundaries
-- **State isolation** lets you test/backfill without affecting production sync
-
----
-
-### Pattern 8: Optional GraphQL Query Logging
-
-**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
-
-**When to use**:
-
-- ✅ Debugging pagination issues
-- ✅ Verifying query variables (dates, filters, limits)
-- ✅ Development and testing
--  Production (verbose logs, potential secrets in variables)
-
-**How to enable**:
-
-Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
-
-**Implementation**:
-
-```typescript
-// In your extraction workflow
-const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
-
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Query Debug', {
-    query: PRODUCTS_QUERY,
-    variables: {
-      catalogues,
-      dateRangeFilter: bufferedLastRunTime,
-      first: pageSize,
-      after: null, // First page
-    },
-    pagination: {
-      pageSize,
-      maxRecords,
-      currentPage: 1,
-    },
-  });
-}
-
-const extractionResult = await orchestrator.extract({
-  query: PRODUCTS_QUERY,
-  resultPath: 'products.edges.node',
-  variables: {
-    catalogues,
-    dateRangeFilter: bufferedLastRunTime,
-  },
-  pageSize,
-  maxRecords,
-});
-
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Response Debug', {
-    totalRecords: extractionResult.stats.totalRecords,
-    totalPages: extractionResult.stats.totalPages,
-    validRecords: extractionResult.stats.validRecords ?? extractionResult.data.length,
-    firstRecordId: extractionResult.data[0]?.id,
-    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
-  });
-}
-```
-
-**What gets logged**:
-
-```json
-{
-  "level": "info",
-  "message": "GraphQL Query Debug",
-  "query": "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)",
-  "variables": {
-    "catalogues": [{ "ref": "DEFAULT_CATALOGUE" }],
-    "dateRangeFilter": "2025-01-22T09:59:00Z",
-    "first": 200,
-    "after": null
-  },
-  "pagination": {
-    "pageSize": 200,
-    "maxRecords": 50000,
-    "currentPage": 1
-  }
-}
-```
-
-**Versori Environment Variables**:
-
-Add to activation settings:
-
-```json
-{
-  "DEBUG_GRAPHQL": "true"
-}
-```
-
-**Testing**:
-
-```bash
-# Enable debug logging
-curl -X POST https://workspace.versori.run/products-extract-daily
-
-# Check Versori logs for "GraphQL Query Debug" entries
-# Verify query structure and variables are correct
-```
-
-**Sample Debug Output**:
-
-```
-[INFO] GraphQL Query Debug
-  query: "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)"
-  variables: { catalogues: [{ ref: "DEFAULT_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
-  pagination: { pageSize: 200, maxRecords: 50000, currentPage: 1 }
-
-[INFO] Extraction complete
-  totalRecords: 1250
-  totalPages: 7
-  validRecords: 1250
-  failedValidations: 0
-
-[INFO] GraphQL Response Debug
-  totalRecords: 1250
-  totalPages: 7
-  validRecords: 1250
-  firstRecordId: "product_abc"
-  lastRecordId: "product_xyz"
-```
-
-**Key Benefits**:
-
-- Quickly identify pagination configuration issues
-- Verify date filters are applied correctly
-- Debug "no records found" scenarios
-- Validate ExtractionOrchestrator variable injection
-
-**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
-
----
-
-## Common Issues
-
-**Issue 1: Malformed XML from unescaped characters**
-
-- Customer name contains `&` or `<`
-- Solution: Always use XMLBuilder (automatic escaping)
-
-**Issue 2: Partner system rejects XML**
-
-- Missing required fields
-- Solution: Verify mapping matches partner schema requirements
-
-**Issue 3: File too large for SFTP partner**
-
-- Partner has 50MB limit, file is 100MB
-- Solution: Use file splitting (10k products per file)
-
-**Issue 4: SFTP connection timeouts**
-
-- Not calling `dispose()` in finally block
-- Solution: Always use try/finally pattern
-
-**Issue 5: Job status not updating**
-
-- JobTracker not integrated
-- Solution: Use JobTracker throughout workflow
-
-## Testing
-
-### 1. Test XML Structure
-
-```typescript
-export const testXmlGeneration = http('test-xml').then(
-  fn('test-xml-gen', async () => {
-    const testProducts = [
-      {
-        sku: 'TEST-001',
-        title: 'Test & Validate <Product>',
-        status: 'ACTIVE',
-        price: 29.99,
-        created_on: '2025-01-22T10:00:00Z',
-        updated_on: '2025-01-22T10:00:00Z',
-      },
-    ];
-
-    const xml = buildProductsXML(testProducts);
-
-    // Validate XML structure
-    if (!xml.includes('<?xml version="1.0"')) {
-      return { success: false, error: 'Missing XML declaration' };
-    }
-
-    if (!xml.includes('&amp;') || !xml.includes('&lt;')) {
-      return { success: false, error: 'Special characters not escaped' };
-    }
-
-    return { success: true, xml };
-  })
-);
-```
-
-### 2. Test SFTP Upload
-
-```bash
-curl https://your-workspace.versori.run/test-sftp-products-xml
-```
-
-### 3. Validate Against Partner Schema
-
-- Download partner's XSD schema
-- Validate generated XML against schema
-- Fix any missing/incorrect elements
-
-## Production Checklist
-
-- [ ] Test SFTP credentials and connection
-- [ ] Verify SFTP server has write permissions to remotePath
-- [ ] Set appropriate extraction frequency (daily for product feeds)
-- [ ] Configure correct product status filters
-- [ ] Test XML escaping with special characters (&, <, >, ", ')
-- [ ] Validate XML against partner's schema (if provided)
-- [ ] Test `dispose()` is always called (check logs)
-- [ ] Document XML schema for partner integration team
-- [ ] Set up monitoring for SFTP connection failures
-- [ ] Test with real product data (names with special chars)
-- [ ] Verify file size limits with SFTP partner
-- [ ] Configure SFTP server IP whitelisting for Versori
-- [ ] Test file splitting with large batches (>10k products)
-- [ ] Test all 3 workflows (scheduled, ad-hoc, status)
-- [ ] Verify JobTracker integration and status updates
-- [ ] Test ExtractionOrchestrator pagination with large datasets
-
-## Troubleshooting Guide
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records
-- **Fix**: Reduce maxRecords, increase frequency
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch
-- **Fix**: Run schema validation, check field names
-
-**Issue**: "State not updating"
-
-- **Cause**: KV write failure or intentional retry
-- **Fix**: Check KV logs, verify state update code
-
-**Issue**: "First run exceeds limits"
-
-- **Cause**: No previous timestamp, fetches all
-- **Fix**: Set fallbackStartDate close to current, apply filters
-
-**Issue**: "Excessive duplicates"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved
-- **Fix**: Verify newTimestamp saved WITHOUT buffer
-
-**Issue**: "Job status returns null"
-
-- **Cause**: Invalid job ID or job expired
-- **Fix**: Verify job ID format, check KV TTL settings
-
-## Security Best Practices
-
-### Credential Management
-
-**✅ DO**:
-
-- Store credentials in Versori activation variables
-- Rotate credentials quarterly
-- Use least-privilege accounts
-
-** DON'T**:
-
-- Never log credentials
-- Never commit to git
-- Never share across environments
-
-### Data Security
-
-- Enable encryption in transit and at rest
-- Apply data retention policies
-- Monitor access logs
-- Use VPC/private networks for sensitive data
-
-### Webhook Security
-
-- Validate API keys for ad-hoc and status workflows
-- Use HTTPS for all webhook endpoints
-- Implement rate limiting
-- Monitor for suspicious activity
-
----
-
-**Pattern**: Enterprise incremental extraction with ExtractionOrchestrator + JobTracker for products via SFTP (XML format)
-**❌š ï¸ Versori Sample**: Reference implementation - adapt for your production use case
-**Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for lifecycle management, always escape XML and dispose SFTP
-**Critical**: Apply 60-second overlap buffer to prevent missed records
-**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
-**Field Consistency**: Same field names as CSV version for easy format switching
-**SFTP**: Use proper connection cleanup in finally block to prevent connection leaks
-**XML**: Preserve hierarchical structure (no flattening needed like CSV)
-**3 Workflows**: Scheduled, ad-hoc webhook, job status query
-
----
-
-### Pattern 8: Backward Pagination (Optional - Advanced)
-
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-
-**When to Use**:
-
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
--  **Don't use for standard incremental sync** - use forward pagination (default)
-
-**GraphQL Query Requirements**:
-
-Your query must support backward pagination by including `$last` and `$before`:
-
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ✅ REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ✅ REQUIRED for backward
-    }
-  }
-}
-```
-
-**Implementation**:
-
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    //  Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ✅ Enable reverse pagination
-  maxRecords: 10000,
-});
-
-// Records are returned in reverse chronological order
-console.log(result.data[0].createdAt); // Newest
-console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
-```
-
-**Key Differences from Forward Pagination**:
-
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest               | Newest → Oldest      |
-
-**Important Notes**:
-
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-
-**Example: Extract Latest 1000 Orders**
-
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-
-**When to Use Forward vs Backward**:
-
-```typescript
-// ✅ Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-
-// ✅ Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-
-**Pagination Variables Reference**:
-
-| Variable | Forward      | Backward     | Injected By  | Notes                    |
-| -------- | ------------ | ------------ | ------------ | ------------------------ |
-| `first`  | ✅ Used      |  Not used | Orchestrator | From `pageSize`          |
-| `after`  | ✅ Used      |  Not used | Orchestrator | From cursor (last edge)  |
-| `last`   |  Not used | ✅ Used      | Orchestrator | From `pageSize`          |
-| `before` |  Not used | ✅ Used      | Orchestrator | From cursor (first edge) |
-
-**Common Mistakes to Avoid**:
-
-```typescript
-//  WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, //  Orchestrator will override this
-    before: cursor, //  Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-
-// ✅ CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ✅ Your business variables only
-  },
-  pageSize: 200, // ✅ Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-
-#### Optional: Reverse Pagination
-
-- For reverse ordering, add $last/$before and pageInfo.hasPreviousPage to your query and set direction='backward'.
-
-GraphQL:
-
-```graphql
-query GetProductsBackward($last: Int!, $before: String) {
-  products(last: $last, before: $before) {
-    edges {
-      cursor
-      node {
-        id
-        ref
-        updatedOn
-      }
-    }
-    pageInfo {
-      hasPreviousPage
-    }
-  }
-}
-```
-
-SDK:
-
-```typescript
-await orchestrator.extract({
-  query: PRODUCTS_BACKWARD_QUERY,
-  resultPath: 'products.edges.node',
-  variables: {},
-  pageSize,
-  direction: 'backward',
-});
-```
-
----
-
-## Testing Checklist
-
-**Before production deployment:**
-
-### 1. Schema Validation
-
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/products.export.xml.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/products.export.xml.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-
-### 2. Extraction Testing
-
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-
-### 3. Mapping Testing
-
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-
-### 4. XML Generation Testing
-
-- [ ] Verify XML structure matches expected format
-- [ ] Test XML validation against XSD schema (if applicable)
-- [ ] Verify special character escaping in XML
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test XML namespace handling (if applicable)
-
-### 5. SFTP Upload Testing
-
-- [ ] Test SFTP connection and authentication
-- [ ] Verify file upload to correct path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify file permissions on SFTP server
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify SFTP connection disposal (no connection leaks)
-
-### 6. State Management Testing
-
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-
-### 7. Job Tracking Testing
-
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-
-### 8. Error Handling Testing
-
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with SFTP connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-
-### 9. Staging Environment Testing
-
-- [ ] Run full extraction in staging environment
-- [ ] Verify XML file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-
-### 10. Integration Testing
-
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger → extract → transform → upload → verify file
-
----
-## Monitoring & Alerting
-
-### Success Response Example
-
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_PRD_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "products-2025-11-02T14-00-00-000Z.xml",
-  "sftpPath": "/outbound/products/products-2025-11-02T14-00-00-000Z.xml",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-
-### Error Response Example
-
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_PRD_20251102_140500_xyz789",
-  "error": "SFTP upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "sftp_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-
-### Key Metrics to Track
-
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-
-  // File Generation
-  fileSizeMB: (xmlContent.length / (1024 * 1024)).toFixed(2),
-
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-
-log.info('Extraction metrics', metrics);
-```
-
-### Alert Thresholds
-
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-
-### Monitoring Dashboard Queries
-
-**Versori Platform Logs Query:**
-
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-
-# High error rates
-errorRate:>5
-
-# State management issues
-stateUpdated:false AND success:true
-```
-
-### Common Issues and Solutions
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-
-**Issue**: "SFTP connection timeout"
-
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check SFTP credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-
-**Issue**: "State not updating after successful extraction"
-
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-
-**Issue**: "First run exceeds record limits"
-
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-
-**Issue**: "Excessive duplicate records in output"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
-
----
-
-## Troubleshooting Quick Reference
-
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "XML generation failed" | Format-specific error | Check XML generation logic, validate output |
-
----
+---
+template_id: tpl-extract-products-to-sftp-xml
+canonical_filename: template-extraction-products-to-sftp-xml.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: sftp-xml
+entity: products
+format: xml
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+  - dispose-finally
+---
+
+# Template: Extraction - Products to SFTP XML
+
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Memory Management** - Clear large result sets after processing batches
+- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
+- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
+- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
+
+## Installation
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk@latest
+```
+
+Use the latest SDK version to ensure compatibility with this template.
+
+---
+
+## 📚 STEP 1: Load These Docs (Human Checklist)
+
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+---
+
+## 📋 Implementation Prompt
+
+```
+Create a Versori scheduled extractor for products that uses ExtractionOrchestrator + JobTracker, incremental updatedOn with a 60s overlap buffer, transforms via UniversalMapper, generates XML with XMLBuilder, uploads to SFTP using SftpDataSource with dispose(). Include 3 workflows: scheduled, ad-hoc webhook, and job-status query with native Versori logging.
+```
+
+---
+
+## 📦 SDK Imports (Verified - Versori Optimized)
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+
+---
+
+# Versori Scheduled: Products Extraction to SFTP XML (Incremental)
+
+**FC Connect SDK Use Case Guide**
+
+> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> Version: Use ^0.1.39 - `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
+
+Context: Scheduled Versori workflow that extracts product catalog from Fluent Commerce via GraphQL query with **ExtractionOrchestrator**, **JobTracker**, and **incremental timestamp tracking**, transforms with `UniversalMapper`, and writes **XML files** to partner SFTP server for marketplace/partner integrations (Amazon, eBay, distributors).
+
+**Pattern**: EXTRACTION (Fluent → SFTP XML)
+**Complexity**: High | Runtime: Versori Platform (Scheduled)
+
+---
+
+## ⚠️ IMPORTANT: Production-Ready Base Template
+
+> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
+>
+> This is a **production-ready base template** demonstrating FC Connect SDK best practices for product extraction workflows with XML output.
+>
+> **✅ INCLUDED FEATURES:**
+>
+> - ✅ Comprehensive error handling with retry logic
+> - ✅ SFTP upload with exponential backoff (3 attempts)
+> - ✅ State management with overlap buffer (prevents missed records)
+> - ✅ Job tracking with lifecycle management
+> - ✅ Security (credential masking in logs)
+> - ✅ UTC time enforcement (prevents timezone bugs)
+> - ✅ Incremental extraction (safe, efficient, production-ready)
+> - ✅ Natural rate limiting via timestamps
+>
+> **📝 BEFORE DEPLOYING:**
+>
+> 1. Review and customize activation variables for your environment
+> 2. Test with sample data in your Versori workspace
+> 3. Adjust safety limits (pageSize, maxRecords) if needed
+> 4. Configure monitoring alerts for extraction failures
+> 5. Verify SFTP credentials and paths
+>
+> **This base template follows SDK best practices - tweak specific to your needs.**
+
+---
+
+## What You'll Build
+
+- **Incremental extraction** using `updatedOn >= (lastRunTime - buffer)` with **overlap buffer**
+- **ExtractionOrchestrator** for auto-pagination and path-based extraction
+- **JobTracker** for lifecycle management and status tracking
+- **State management** with VersoriKV to track last successful run
+- **Safety buffer** (60 seconds) to handle clock skew and race conditions
+- GraphQL query for product catalog (SKU, title, description, pricing)
+- `UniversalMapper` transformation for partner schema
+- XML file generation with product catalog data
+- **SFTP upload** to partner server (with `dispose()` cleanup)
+- **3 workflow patterns**: scheduled, ad-hoc webhook, job status query
+- **Failure recovery** with timestamp tracking
+
+## Business Use Case
+
+**Daily product catalog feed to marketplace/partner:**
+
+- Extract new and updated products since last run
+- Generate XML file with product data for partner consumption
+- Upload to partner SFTP server for marketplace integration
+- Run daily to keep product catalog synchronized
+- Support product updates (price changes, inventory status)
+- Standard XML format for EDI/ERP integration
+
+## SDK Methods Used
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+
+await createClient(ctx); // Versori-aware client
+const orchestrator = new ExtractionOrchestrator(client, log); // Auto-pagination
+const tracker = new JobTracker(kv, log); // Job lifecycle tracking
+await orchestrator.extract({ query, resultPath, variables, pageSize, maxRecords }); // Extract
+new VersoriKVAdapter(ctx.openKv(':project:')); // State management
+new UniversalMapper(exportMapping); // Field transformation
+new XMLBuilder(options); // XML generation with auto-escaping
+await sftp.uploadFile(remotePath, buffer); // SFTP upload
+await sftp.dispose(); // CRITICAL: Connection cleanup
+```
+
+## SFTP Connection Setup & Credential Access
+
+### Method 1: Versori Connections (Recommended)
+
+**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
+
+**Connection Configuration:**
+
+1. In Versori platform, create a connection named `versori_ftp_server`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+
+**Access Method: `activation.connections` (Recommended)**
+
+```typescript
+import { Buffer } from 'node:buffer'; // ⚠️ CRITICAL: Required for Deno/Versori runtime
+
+// Get SFTP credentials from Versori connection (Basic Auth)
+// RECOMMENDED: Use activation.connections (already decoded)
+const allConnections = ctx.activation.connections || [];
+const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+if (!sftpConn) {
+  throw new Error('SFTP connection "versori_ftp_server" not found');
+}
+
+const credential = sftpConn.credentials[0]?.credential;
+if (!credential?.data?.basicAuth) {
+  throw new Error('SFTP connection not configured with Basic Authentication');
+}
+
+const { username, password } = credential.data.basicAuth;
+// ✅ Already decoded - no Buffer.from() needed!
+```
+
+**Alternative: `credentials().getAccessToken()` (Explicit)**
+
+```typescript
+import { Buffer } from 'node:buffer'; // ⚠️ CRITICAL: Required for Deno/Versori runtime
+
+const sftpCred = await ctx.credentials().getAccessToken('versori_ftp_server');
+const rawAccessToken = sftpCred.accessToken;
+const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+```
+
+**Why use connections instead of activation variables?**
+
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection can be reused across workflows
+- ✅ No need to manage sensitive data in activation variables
+- ✅ Easier credential rotation
+- ✅ Centralized credential management across projects
+
+**🔍– Complete Guide:** See `docs/02-CORE-GUIDES/data-sources/sftp-credential-access-security.md` for comprehensive security patterns and credential management best practices.
+
+### Method 2: Activation Variables (Alternative)
+
+Store credentials directly in activation variables (less secure):
+
+```typescript
+const sftpUsername = ctx.activation?.getVariable('sftpUsername');
+const sftpPassword = ctx.activation?.getVariable('sftpPassword');
+```
+
+**When to use activation variables:**
+
+- Quick prototyping or testing
+- Non-production environments
+- Single-use credentials
+
+**⚠️ Security Warning:** Activation variables are less secure than Versori connections. Always prefer connection-based credential storage for production.
+
+### Buffer Import for Deno/Versori (CRITICAL)
+
+**⚠️ ALWAYS import Buffer** when using SFTP operations in Versori/Deno runtime:
+
+```typescript
+import { Buffer } from 'node:buffer';
+```
+
+**Why?** Unlike Node.js where `Buffer` is global, Deno requires explicit imports from Node.js built-ins using the `node:` prefix.
+
+**Common use cases:**
+
+- SFTP uploads: `Buffer.from(content, 'utf8')`
+- Base64 decoding: `Buffer.from(str, 'base64').toString('utf-8')`
+- Binary data: `Buffer.from(data)`
+
+**Error:** `Buffer is not defined` →' Add `import { Buffer } from 'node:buffer';`
+
+## Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+```json
+{
+  "retailerId": "your-retailer-id",
+  "sftpHost": "sftp.partner.com",
+  "sftpPort": 22,
+  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
+  "sftpRemotePath": "/incoming/products/",
+  "pageSize": 200,
+  "maxRecords": 50000,
+  "fallbackStartDate": "2024-01-01T00:00:00Z",
+  "overlapBufferSeconds": "60",
+  "productStatus": "ACTIVE"
+}
+```
+
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection (see SFTP Connection Setup above).
+
+## Export Mapping Configuration
+
+**IMPORTANT**: Fields match CSV version exactly for consistency.
+
+Create file: `./config/products.export.xml.json`
+
+```json
+{
+  "name": "products.export.xml",
+  "version": "1.0.0",
+  "description": "Fluent Products → Partner SFTP XML Export",
+  "fields": {
+    "sku": { "source": "ref", "required": true, "resolver": "sdk.trim" },
+    "title": { "source": "name", "required": true, "resolver": "sdk.trim" },
+    "description": { "source": "summary", "required": false, "resolver": "sdk.trim" },
+    "gtin": { "source": "gtin", "required": false, "resolver": "sdk.trim" },
+    "type": { "source": "type", "required": false, "resolver": "sdk.uppercase" },
+    "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
+    "price": { "source": "price", "required": false, "resolver": "sdk.parseFloat" },
+    "catalogue_ref": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
+    "catalogue_name": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
+    "created_on": { "source": "createdOn", "required": true, "resolver": "sdk.toString" },
+    "updated_on": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
+  }
+}
+```
+
+## Mapping & Resolvers Explained
+
+### SDK Resolvers Used
+
+The export mapping uses **SDK resolvers** to transform GraphQL data into the target XML format:
+
+| Field           | Resolver         | Why?                                         | Example Transformation                  |
+| --------------- | ---------------- | -------------------------------------------- | --------------------------------------- |
+| `sku`           | `sdk.trim`       | Remove leading/trailing whitespace from SKUs | `" ABC-123 "` → `"ABC-123"`          |
+| `title`         | `sdk.trim`       | Clean product names                          | `"Widget  "` → `"Widget"`            |
+| `description`   | `sdk.trim`       | Clean descriptions                           | `"  Description"` → `"Description"`  |
+| `gtin`          | `sdk.trim`       | Clean barcode numbers                        | `" 012345678901"` → `"012345678901"` |
+| `type`          | `sdk.uppercase`  | Normalize product type codes                 | `"standard"` → `"STANDARD"`          |
+| `status`        | `sdk.uppercase`  | Normalize status values                      | `"active"` → `"ACTIVE"`              |
+| `price`         | `sdk.parseFloat` | Parse price as decimal                       | `"29.99"` → `29.99`                  |
+| `catalogue_ref` | `sdk.trim`       | Clean catalogue references                   | `" CAT-001 "` → `"CAT-001"`          |
+| `created_on`    | `sdk.toString`   | Ensure timestamp is string                   | `Date` → `"2025-01-21T10:30:00Z"`    |
+| `updated_on`    | `sdk.toString`   | Ensure timestamp is string                   | `Date` → `"2025-01-21T10:30:00Z"`    |
+
+### Transformation Flow
+
+```typescript
+// 1. GraphQL Response (from Fluent API)
+{
+  ref: " SKU-001 ",           // → Has whitespace
+  name: "Premium Widget  ",   // → Has trailing space
+  type: "standard",           // → Lowercase
+  status: "active",           // → Lowercase
+  price: "29.99",             // → String
+  catalogue: {
+    ref: " CAT-001 ",
+    name: " Default Catalogue "
+  },
+  updatedOn: "2025-01-21T10:30:00Z"
+}
+
+// 2. UniversalMapper applies resolvers
+const mapper = new UniversalMapper(productsExportMapping);
+const result = await mapper.map(node);
+
+// 3. Transformed Output (clean, normalized)
+result.data = {
+  sku: "SKU-001",             // ✅ Trimmed
+  title: "Premium Widget",    // ✅ Trimmed
+  type: "STANDARD",           // ✅ Uppercased
+  status: "ACTIVE",           // ✅ Uppercased
+  price: 29.99,               // ✅ Float
+  catalogue_ref: "CAT-001",   // ✅ Trimmed from nested object
+  catalogue_name: "Default Catalogue", // ✅ Trimmed from nested object
+  updated_on: "2025-01-21T10:30:00Z"
+}
+```
+
+### Custom Resolvers for Product-Specific Logic
+
+You can add **custom resolvers** for business-specific transformations:
+
+```typescript
+const productsExportMapping = {
+  name: 'products.export.xml',
+  version: '1.0.0',
+  fields: {
+    sku: { source: 'ref', required: true, resolver: 'sdk.trim' },
+    title: { source: 'name', required: true, resolver: 'sdk.trim' },
+
+    // Custom resolver: Calculate display price with tax
+    display_price: {
+      source: 'price',
+      resolver: 'custom.calculateDisplayPrice',
+    },
+
+    // Custom resolver: Map internal categories to partner categories
+    partner_category: {
+      source: 'category',
+      resolver: 'custom.mapCategory',
+    },
+
+    // Custom resolver: Generate SEO-friendly URL slug
+    url_slug: {
+      source: 'name',
+      resolver: 'custom.generateSlug',
+    },
+  },
+};
+
+// Custom resolver implementations
+const customResolvers = {
+  'custom.calculateDisplayPrice': (price: number) => {
+    const TAX_RATE = 0.1;
+    return (price * (1 + TAX_RATE)).toFixed(2);
+  },
+
+  'custom.mapCategory': (category: string) => {
+    const categoryMap: Record<string, string> = {
+      ELECTRONICS: 'Electronics & Gadgets',
+      APPAREL: 'Clothing & Fashion',
+      HOME: 'Home & Living',
+    };
+    return categoryMap[category] || 'General';
+  },
+
+  'custom.generateSlug': (name: string) => {
+    return name
+      .toLowerCase()
+      .replace(/[^\w\s-]/g, '')
+      .replace(/\s+/g, '-');
+  },
+};
+
+// Use with UniversalMapper
+const mapper = new UniversalMapper(productsExportMapping, { customResolvers });
+```
+
+### Available SDK Resolvers
+
+**String Transformations:**
+
+- `sdk.trim` - Remove whitespace
+- `sdk.uppercase` - Convert to uppercase
+- `sdk.lowercase` - Convert to lowercase
+- `sdk.toString` - Convert to string
+
+**Number Transformations:**
+
+- `sdk.parseInt` - Parse integer
+- `sdk.parseFloat` - Parse decimal
+- `sdk.number` - Generic number conversion
+
+**Date Transformations:**
+
+- `sdk.formatDate` - ISO 8601 format (`2025-01-22T14:30:00Z`)
+- `sdk.formatDateShort` - Short date format (`2025-01-22`)
+- `sdk.parseDate` - Parse date string
+
+**Type Conversions:**
+
+- `sdk.boolean` - Convert to boolean
+- `sdk.parseJson` - Parse JSON string
+- `sdk.toJson` - Convert to JSON string
+
+**Utility:**
+
+- `sdk.identity` - Pass through unchanged
+- `sdk.coalesce` - Return first non-null value
+
+See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
+
+## GraphQL Query
+
+```graphql
+query GetProducts(
+  $catalogues: [ProductCatalogueKey]
+  $dateRangeFilter: DateRange
+  $first: Int!
+  $after: String
+) {
+  products(catalogueRef: $catalogues, updatedOn: $dateRangeFilter, first: $first, after: $after) {
+    edges {
+      node {
+        id
+        ref
+        name
+        type
+        status
+        gtin
+        price
+        attributes
+        catalogue {
+          ref
+          name
+        }
+        createdOn
+        updatedOn
+      }
+      cursor
+    }
+    pageInfo {
+      hasNextPage
+    }
+  }
+}
+```
+
+## Expected XML Output
+
+**IMPORTANT**: XML structure with same fields as CSV version for consistency.
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<Products>
+  <Product>
+    <sku>SKU-001</sku>
+    <title>Premium Widget</title>
+    <description>High-quality widget for all purposes</description>
+    <gtin>012345678901</gtin>
+    <type>STANDARD</type>
+    <status>ACTIVE</status>
+    <price>29.99</price>
+    <catalogue_ref>CAT-001</catalogue_ref>
+    <catalogue_name>Default Catalogue</catalogue_name>
+    <created_on>2025-01-21T10:30:00Z</created_on>
+    <updated_on>2025-01-21T10:30:00Z</updated_on>
+  </Product>
+  <Product>
+    <sku>SKU-002</sku>
+    <title>Deluxe Gadget</title>
+    <description>Advanced gadget with premium features</description>
+    <gtin>012345678902</gtin>
+    <type>STANDARD</type>
+    <status>ACTIVE</status>
+    <price>49.99</price>
+    <catalogue_ref>CAT-001</catalogue_ref>
+    <catalogue_name>Default Catalogue</catalogue_name>
+    <created_on>2025-01-21T14:15:00Z</created_on>
+    <updated_on>2025-01-21T14:15:00Z</updated_on>
+  </Product>
+</Products>
+```
+
+**Note**: XML preserves hierarchical structure unlike CSV which flattens to rows.
+
+## Production Safety & Guardrails
+
+### Overview
+
+Product catalogs require strict guardrails even with incremental extraction:
+
+- **Large initial extractions**: First run can include entire catalog (100k+ products)
+- **Bulk updates**: Marketing campaigns can update thousands of products at once
+- **XML overhead**: 2-3x larger than JSON/CSV for same data
+- **SFTP limits**: Partner servers may reject large files
+- **Memory pressure**: Product records are larger (descriptions, attributes)
+
+### Hard Limits
+
+```typescript
+const SAFETY_LIMITS = {
+  MAX_RECORDS_PER_RUN: 50000, // 50k products per run (XML overhead)
+  MAX_RECORDS_PER_FILE: 10000, // 10k per XML file (SFTP-friendly)
+  MAX_FILE_SIZE_MB: 150, // 150MB per file
+  MAX_XML_SIZE_MB: 300, // Total extraction size
+  CHUNK_SIZE: 5000, // Process in chunks
+  ESTIMATED_BYTES_PER_PRODUCT_XML: 3000, // 3KB per product in XML (conservative)
+};
+```
+
+**Why different from JSON?**
+
+- XML has 2-3x size overhead (tags, attributes, whitespace)
+- Products have rich text content (descriptions, attributes)
+- Partner SFTP servers often have smaller file size limits than S3
+- 10k products per file = ~30MB (manageable for most SFTP systems)
+
+### Runtime Validation Function
+
+```typescript
+/**
+ * Validate extraction safety limits before processing
+ * CRITICAL: Account for XML size overhead vs CSV
+ */
+function validateExtractionLimits(productCount: number) {
+  const MAX_PRODUCTS_PER_RUN = 50000;
+  const ESTIMATED_BYTES_PER_PRODUCT_XML = 3000; // Full XML product element
+  const estimatedSizeMB = (productCount * ESTIMATED_BYTES_PER_PRODUCT_XML) / (1024 * 1024);
+  const MAX_XML_SIZE_MB = 300;
+
+  if (productCount > MAX_PRODUCTS_PER_RUN) {
+    return {
+      valid: false,
+      error: `Extraction limit exceeded: ${productCount} products (max: ${MAX_PRODUCTS_PER_RUN})`,
+      recommendation: `Too many products for single extraction. Consider:
+        1. Increase extraction frequency (daily → hourly)
+        2. Add product status filters (ACTIVE only)
+        3. Split by catalogue
+        4. Contact support if consistently exceeding limits`,
+      productCount,
+      maxAllowed: MAX_PRODUCTS_PER_RUN,
+    };
+  }
+
+  if (estimatedSizeMB > MAX_XML_SIZE_MB) {
+    return {
+      valid: false,
+      error: `XML size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_XML_SIZE_MB}MB)`,
+      recommendation:
+        'File splitting required. Increase extraction frequency to reduce batch size.',
+      estimatedSizeMB,
+      maxAllowed: MAX_XML_SIZE_MB,
+    };
+  }
+
+  return { valid: true };
+}
+```
+
+---
+
+## Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+products-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-products-extraction.ts  # Scheduled: Daily products extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-products-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts          # Webhook: Status query
+    │
+    ├── services/
+    │   └── products-extraction.service.ts   # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── products.export.xml.json         # Mapping configuration
+```
+
+---
+
+## Complete Workflow Code
+
+The code below demonstrates the implementation of each component in the modular structure.
+
+### 1. Entry Point (`index.ts`)
+
+```typescript
+/**
+ * Entry point - Export all workflows for Versori platform
+ *
+ * This file exports all workflows to be registered with Versori.
+ * Each workflow is defined in its own file for better organization.
+ */
+
+// Scheduled workflows
+export { dailyProductsExtraction } from './src/workflows/scheduled/daily-products-extraction';
+
+// Webhook workflows
+export { adhocProductsExtraction } from './src/workflows/webhook/adhoc-products-extraction';
+export { productsExtractionJobStatus } from './src/workflows/webhook/job-status-check';
+```
+
+### 2. Workflows (src/workflows/products-extraction.ts)
+
+```typescript
+// ⚠️ IMPORTANT: Do NOT import openKv - access it from context!
+import { schedule, webhook, http, fn } from '@versori/run';
+import {
+  executeProductExtraction,
+  getJobStatus,
+  generateJobId,
+} from '../services/products-extraction.service';
+
+// 
+// WORKFLOW 1: Scheduled Extraction (Daily at 2 AM)
+// 
+
+export const scheduledProductsExtraction = schedule('products-extract-xml-daily', '0 2 * * *').then(
+  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async ctx => {
+    const jobId = generateJobId('SCHED', 'PRODUCTS');
+
+    const result = await executeProductExtraction(ctx, {
+      jobId,
+      triggeredBy: 'schedule',
+      updateState: true, // Always update state for scheduled runs
+    });
+
+    return result;
+  })
+);
+
+// 
+// WORKFLOW 2: Ad-hoc Webhook Extraction
+// 
+
+export const adhocProductsExtraction = webhook('products-adhoc', {
+  connection: 'products-adhoc',
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+}).then(
+  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async ctx => {
+    // Security is enforced by the 'products-adhoc' connection
+    const { log } = ctx;
+    const jobId = generateJobId('ADHOC', 'PRODUCTS');
+
+    log.info('🚀 [WEBHOOK] Adhoc products extraction triggered', {
+      jobId,
+      fromDate: ctx.data.fromDate,
+      toDate: ctx.data.toDate,
+      updateState: ctx.data.updateState,
+    });
+
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductExtraction(ctx, {
+      jobId,
+      triggeredBy: 'webhook',
+      fromDate: ctx.data.fromDate,
+      toDate: ctx.data.toDate,
+      updateState: ctx.data.updateState === true,
+    })
+      .then((result) => {
+        log.info('✅ [BACKGROUND] Products extraction completed successfully', {
+          jobId,
+          recordCount: result.recordCount,
+          fileName: result.fileName,
+        });
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        log.error('❌ [BACKGROUND] Products extraction failed', {
+          jobId,
+          error: errorMessage,
+          stack: error instanceof Error ? error.stack : undefined,
+        });
+      });
+
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Products extraction started in background',
+      statusEndpoint: `https://{workspace}.versori.run/products-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+
+// 
+// WORKFLOW 3: Job Status Query
+// 
+
+export const productsJobStatus = webhook('products-job-status', {
+  connection: 'products-job-status',
+  response: { mode: 'sync' },
+}).then(
+  fn('query-job-status', async ctx => {
+    const { data, log, openKv } = ctx;
+    // Security is enforced by the 'products-job-status' connection
+
+    const jobId = data.jobId;
+    if (!jobId) {
+      return { success: false, error: 'Job ID required' };
+    }
+
+    const status = await getJobStatus(openKv(':project:'), jobId, log);
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+
+### 3. Main Orchestration Service (`src/services/products-extraction.service.ts`)
+
+**Note:** This service file should be renamed from `extraction-orchestration.ts` to `products-extraction.service.ts` to match the new workflow structure.
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  XMLBuilder,
+  SftpDataSource,
+  VersoriKVAdapter,
+} from '@fluentcommerce/fc-connect-sdk';
+import productsExportMapping from '../../config/products.export.xml.json' with { type: 'json' };
+
+const PRODUCTS_EXTRACTION_QUERY = `
+  query GetProducts(
+    $catalogues: [ProductCatalogueKey]
+    $dateRangeFilter: DateRange
+    $first: Int!
+    $after: String
+  ) {
+    products(
+      catalogueRef: $catalogues
+      updatedOn: $dateRangeFilter
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          name
+          type
+          status
+          gtin
+          price
+          attributes
+          catalogue {
+            ref
+            name
+          }
+          createdOn
+          updatedOn
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+
+// Initialize XMLBuilder for products
+const xmlBuilder = new XMLBuilder({
+  rootElement: 'Products',
+  prettyPrint: true,
+  indent: '  ',
+  xmlDeclaration: true,
+  encoding: 'UTF-8',
+});
+
+function buildProductsXML(products: any[]): string {
+  // Transform to XMLBuilder format
+  const productsForXml = products.map(p => ({
+    sku: p.sku,
+    title: p.title,
+    description: p.description || '',
+    gtin: p.gtin || '',
+    type: p.type || '',
+    status: p.status,
+    price: String(p.price || ''),
+    catalogue_ref: p.catalogue_ref || '',
+    catalogue_name: p.catalogue_name || '',
+    created_on: p.created_on,
+    updated_on: p.updated_on,
+  }));
+
+  return xmlBuilder.build({ Product: productsForXml });
+}
+
+interface ProductExtractionParams {
+  jobId: string;
+  triggeredBy: 'schedule' | 'webhook';
+  fromDate?: string;
+  toDate?: string;
+  updateState: boolean;
+}
+
+export async function executeProductExtraction(ctx: any, options: ProductExtractionParams) {
+  const { jobId, triggeredBy, fromDate, toDate, updateState } = options;
+  const log = ctx.log;
+  const retailerId = ctx.activation?.getVariable('retailerId');
+  const pageSize = parseInt(ctx.activation?.getVariable('pageSize') || '200', 10);
+  const maxRecords = parseInt(ctx.activation?.getVariable('maxRecords') || '50000', 10);
+  const fallbackStartDate =
+    ctx.activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
+  const productStatus = ctx.activation?.getVariable('productStatus') || 'ACTIVE';
+
+  // Get SFTP credentials from Versori connection (Basic Auth)
+  // RECOMMENDED: Use activation.connections (already decoded)
+  const allConnections = ctx.activation.connections || [];
+  const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+  if (!sftpConn) {
+    throw new Error('SFTP connection "versori_ftp_server" not found');
+  }
+
+  const credential = sftpConn.credentials[0]?.credential;
+  if (!credential?.data?.basicAuth) {
+    throw new Error('SFTP connection not configured with Basic Authentication');
+  }
+
+  const { username, password } = credential.data.basicAuth;
+  // ✅ Already decoded - no Buffer.from() needed!
+
+  const sftpSettings = {
+    host: ctx.activation?.getVariable('sftpHost'),
+    port: parseInt(ctx.activation?.getVariable('sftpPort') || '22', 10),
+    username, // From connection (secure)
+    password, // From connection (secure)
+    privateKey: ctx.activation?.getVariable('sftpPrivateKey'),
+    remotePath: ctx.activation?.getVariable('sftpRemotePath') || '/incoming/products/',
+  };
+
+  const missing: string[] = [];
+  if (!retailerId) missing.push('retailerId');
+  if (!sftpSettings.host) missing.push('sftpHost');
+  if (missing.length)
+    return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
+
+  // SFTP connection - MUST use try/finally with dispose()
+  const sftp = new SftpDataSource(
+    {
+      type: 'SFTP_XML',
+      connectionId: 'sftp-products-xml-export',
+      name: 'SFTP Products XML Export',
+      settings: {
+        host: sftpSettings.host,
+        port: sftpSettings.port,
+        username: sftpSettings.username,
+        password: sftpSettings.password,
+        privateKey: sftpSettings.privateKey,
+        remotePath: sftpSettings.remotePath,
+        filePattern: '*.xml',
+      },
+    },
+    log
+  );
+
+  try {
+    // 
+    // STEP 1/8: Initialize Job Tracking
+    // 
+    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
+    const tracker = new JobTracker(kv, log);
+
+    await tracker.createJob(jobId, {
+      triggeredBy,
+      hasDateOverride: !!fromDate,
+      fromDate,
+      toDate,
+      updateStateAfterRun: updateState,
+    });
+
+    log.info('Job created', { jobId, triggeredBy });
+
+    // 
+    // STEP 2/8: Load State & Calculate Time Window
+    // 
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'state_load',
+      message: 'Loading last run state',
+    });
+
+    const stateKey = ['extraction', 'products-xml', 'lastProductSync'];
+    const lastRunState = await kv.get(stateKey);
+    const rawLastRunTime = fromDate || lastRunState?.value?.timestamp || fallbackStartDate;
+
+    // Overlap buffer configuration (default: 60 seconds)
+    const overlapBufferSeconds = parseInt(
+      ctx.activation?.getVariable('overlapBufferSeconds') || '60',
+      10
+    );
+    const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
+
+    // Apply overlap buffer for query (safety window)
+    const bufferedLastRunTime = new Date(
+      new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
+    ).toISOString();
+
+    const effectiveEndTime = toDate || new Date().toISOString();
+
+    log.info('🔍 Time window calculated', {
+      rawLastRunTime,
+      bufferedLastRunTime,
+      effectiveEndTime,
+      overlapBufferSeconds,
+      retailerId,
+      productStatus,
+    });
+
+    // 
+    // STEP 3/8: Initialize Fluent Client & ExtractionOrchestrator
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'client_init',
+      message: 'Initializing Fluent client',
+    });
+
+    const client = await createClient(ctx);
+    const orchestrator = new ExtractionOrchestrator(client, log);
+
+    // 
+    // STEP 4/8: Extract Data (ExtractionOrchestrator)
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'extraction',
+      message: 'Extracting data with auto-pagination',
+    });
+
+    // ? Enhanced: Extract context for progress logging
+    const dateRangeInfo = {
+      start: bufferedLastRunTime || 'N/A',
+      end: effectiveEndTime || 'N/A',
+      catalogues: 'all'
+    };
+
+    // ? Enhanced: Start logging with context
+    log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
+     query: 'products',
+     pageSize,
+     maxRecords,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     catalogues: dateRangeInfo.catalogues,
+     jobId
+    });
+
+    const extractionResult = await orchestrator.extract({
+      query: PRODUCTS_EXTRACTION_QUERY,
+      resultPath: 'products.edges.node',
+      variables: {
+        catalogues: null, // All catalogues
+        dateRangeFilter: {
+          after: bufferedLastRunTime,
+          before: effectiveEndTime, // End of extraction window
+        },
+        first: pageSize,
+      },
+      pageSize,
+      maxRecords,
+      validateItem: item => !!(item.ref && item.name),
+    });
+
+    const rawRecords = extractionResult.data;
+
+    log.info('Extraction complete', {
+      totalRecords: extractionResult.stats.totalRecords,
+      totalPages: extractionResult.stats.totalPages,
+      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+      errors: extractionResult.errors ? extractionResult.errors.length : 0,
+    });
+
+    // ? Enhanced: Completion logging with summary
+    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+     totalRecords: extractionResult.stats.totalRecords,
+     totalPages: extractionResult.stats.totalPages,
+     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+     failedValidations: extractionResult.stats.failedValidations,
+     truncated: extractionResult.stats.truncated,
+     truncationReason: extractionResult.stats.truncationReason,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     jobId
+    });
+
+    if (extractionResult.errors && extractionResult.errors.length > 0) {
+      log.warn('Non-fatal extraction errors encountered', {
+        errorCount: extractionResult.errors.length,
+        sampleErrors: extractionResult.errors.slice(0, 3),
+      });
+    }
+
+    if (rawRecords.length === 0) {
+      await tracker.markCompleted(jobId, {
+        recordCount: 0,
+        message: 'No new products to extract',
+      });
+
+      if (updateState) {
+        await kv.set(stateKey, {
+          timestamp: new Date().toISOString(),
+          productCount: 0,
+          extractedAt: new Date().toISOString(),
+        });
+      }
+
+      return { success: true, message: 'No new products to extract', lastRunTime: rawLastRunTime };
+    }
+
+    // 
+    // STEP 5/8: Validate Extraction Limits
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'validation',
+      message: 'Validating extraction limits',
+    });
+
+    const MAX_PRODUCTS_PER_RUN = 50000;
+    const ESTIMATED_BYTES_PER_PRODUCT_XML = 3000;
+    const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_PRODUCT_XML) / (1024 * 1024);
+    const MAX_XML_SIZE_MB = 300;
+
+    if (rawRecords.length > MAX_PRODUCTS_PER_RUN) {
+      log.error('Extraction limit exceeded', {
+        productCount: rawRecords.length,
+        maxAllowed: MAX_PRODUCTS_PER_RUN,
+      });
+
+      await tracker.markFailed(jobId, {
+        error: `Extraction limit exceeded: ${rawRecords.length} products (max: ${MAX_PRODUCTS_PER_RUN})`,
+        recommendation: 'Increase extraction frequency or add filters',
+      });
+
+      return {
+        success: false,
+        error: `Extraction limit exceeded: ${rawRecords.length} products (max: ${MAX_PRODUCTS_PER_RUN})`,
+        recommendation: `Too many products for single extraction. Consider:
+          1. Increase extraction frequency (daily → hourly)
+          2. Add product status filters (ACTIVE only)
+          3. Split by catalogue
+          4. Contact support if consistently exceeding limits`,
+        productCount: rawRecords.length,
+        maxAllowed: MAX_PRODUCTS_PER_RUN,
+      };
+    }
+
+    if (estimatedSizeMB > MAX_XML_SIZE_MB) {
+      log.warn('XML size approaching limit', {
+        estimatedSizeMB: estimatedSizeMB.toFixed(2),
+        maxAllowed: MAX_XML_SIZE_MB,
+        recommendation: 'Consider file splitting or increase extraction frequency',
+      });
+    }
+
+    log.info('Extraction limits validated', {
+      productCount: rawRecords.length,
+      estimatedSizeMB: estimatedSizeMB.toFixed(2),
+      withinLimits: true,
+    });
+
+    // 
+    // STEP 6/8: Transform Data (UniversalMapper)
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'transformation',
+      message: 'Transforming data with UniversalMapper',
+    });
+
+    const mapper = new UniversalMapper(productsExportMapping);
+    const mappingResult = await mapper.map(rawRecords);
+
+    if (!mappingResult.success) {
+      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+      await tracker.markFailed(jobId, {
+        error: mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
+        failedCount: mappingErrors.length,
+      });
+      return {
+        success: false,
+        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+        errors: mappingErrors,
+      };
+    }
+
+    const transformedProducts = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+    const mappingErrors = mappingResult.errors || [];
+
+    if (mappingErrors.length > 0) {
+      log.warn('Some products failed transformation', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+    }
+
+    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+        jobId,
+        skippedFields: mappingResult.skippedFields,
+        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+      });
+    }
+
+    if (transformedProducts.length === 0) {
+      await tracker.markFailed(jobId, {
+        error: 'All records failed mapping',
+        failedCount: mappingErrors.length,
+      });
+      return {
+        success: false,
+        error: 'All records failed mapping',
+        errors: mappingErrors,
+      };
+    }
+
+    log.info('Products transformed', {
+      jobId,
+      transformedCount: transformedProducts.length,
+      skippedRecords: rawRecords.length - transformedProducts.length,
+    });
+
+    // 
+    // STEP 7/8: Generate XML & Upload to SFTP
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'upload',
+      message: 'Generating XML and uploading to SFTP',
+    });
+
+    const xmlContent = buildProductsXML(transformedProducts);
+
+    // Generate timestamped filename
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const fileName = `products-${timestamp}.xml`;
+    const remotePath = `${sftpSettings.remotePath}${fileName}`;
+
+    log.info('Generated XML file', {
+      fileName,
+      size: xmlContent.length,
+      productCount: transformedProducts.length,
+    });
+
+    // Upload to SFTP
+    await sftp.uploadFile(remotePath, Buffer.from(xmlContent, 'utf8'));
+
+    log.info('XML file uploaded to SFTP', { remotePath });
+
+    // 
+    // STEP 8/8: Update State & Complete Job
+    // 
+    await tracker.updateJob(jobId, {
+      stage: 'state_update',
+      message: 'Updating state and completing job',
+    });
+
+    // Calculate max updatedOn from extracted products
+    const maxUpdatedOn = transformedProducts.reduce((max, product) => {
+      const productTime = new Date(product.updated_on).getTime();
+      return productTime > max ? productTime : max;
+    }, new Date(rawLastRunTime).getTime());
+
+    const newTimestamp = new Date(maxUpdatedOn).toISOString();
+
+    // Update state with new timestamp (WITHOUT buffer)
+    if (updateState) {
+      await kv.set(stateKey, {
+        timestamp: newTimestamp, // ← NO buffer applied
+        productCount: transformedProducts.length,
+        extractedAt: new Date().toISOString(),
+        overlapBufferSeconds,
+        fileName,
+        remotePath,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      });
+
+      log.info('State updated with new timestamp (without buffer)', {
+        newTimestamp,
+        overlapBufferSeconds,
+      });
+    }
+
+    await tracker.markCompleted(jobId, {
+      recordCount: transformedProducts.length,
+      fileName,
+      sftpPath: remotePath,
+      errorCount: mappingErrors.length,
+      errors: mappingErrors,
+    });
+
+    return {
+      success: true,
+      productsExtracted: transformedProducts.length,
+      fileName,
+      remotePath,
+      lastRunTime: rawLastRunTime,
+      newTimestamp,
+      jobId,
+      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+    };
+  } catch (error: any) {
+    log.error('Extraction failed', error, {
+      message: error?.message,
+    });
+
+    const kv = new VersoriKVAdapter(ctx.openKv(':project:'));
+    const tracker = new JobTracker(kv, log);
+
+    await tracker.markFailed(jobId, {
+      message: error instanceof Error ? error.message : String(error),
+
+      stack: error instanceof Error ? error.stack : undefined,
+
+      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+    });
+
+    return {
+      success: false,
+      message: error instanceof Error ? error.message : String(error),
+
+      stack: error instanceof Error ? error.stack : undefined,
+
+      errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+      jobId,
+    };
+  } finally {
+    // CRITICAL: Always clean up SFTP connections
+    await sftp.dispose();
+    log.info('SFTP connection disposed');
+  }
+}
+
+export async function getJobStatus(kv: any, jobId: string, log: any) {
+  const tracker = new JobTracker(new VersoriKVAdapter(kv), log);
+  return await tracker.getJob(jobId);
+}
+```
+
+### 4. Job ID Generator (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Generate unique job ID
+ * Format: {PREFIX}-{ENTITY}-{TIMESTAMP}
+ */
+export function generateJobId(prefix: 'SCHED' | 'ADHOC', entity: string): string {
+  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+  return `${prefix}-${entity}-${timestamp}`;
+}
+```
+
+### 5. Package Configuration (package.json)
+
+```json
+{
+  "name": "products-extraction-to-sftp-xml",
+  "version": "1.0.0",
+  "description": "Versori connector for products extraction to SFTP XML",
+  "main": "dist/index.js",
+  "type": "module",
+  "scripts": {
+    "build": "tsc",
+    "dev": "tsc --watch",
+    "lint": "eslint src/**/*.ts",
+    "test": "jest"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+
+### 6. Deployment Instructions
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Build the connector
+npm run build
+
+# 3. Test locally (optional)
+npm test
+
+# 4. Deploy to Versori
+# - Upload to Versori workspace
+# - Configure activation variables
+# - Enable workflows
+
+# 5. Test workflows
+# Scheduled: Wait for next cron trigger or manually trigger
+# Ad-hoc: POST to webhook URL with API key header
+# Status: Query job status by ID
+```
+
+### 7. Testing
+
+#### Test Scheduled Extraction
+
+```bash
+# Trigger manually in Versori UI or wait for cron schedule
+# Expected: XML file uploaded to SFTP
+```
+
+#### Test Ad-hoc Extraction
+
+```bash
+curl -X POST https://your-workspace.versori.run/products-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-22T23:59:59Z",
+    "updateState": false
+  }'
+```
+
+#### Test Job Status Query
+
+```bash
+curl -X POST https://your-workspace.versori.run/products-job-status \
+  -H "Content-Type: application/json" \
+  -d '{
+    "jobId": "SCHED-PRODUCTS-2025-01-22T02-00-00Z"
+  }'
+```
+
+## Key Patterns Explained
+
+### Pattern 1: ExtractionOrchestrator for Auto-Pagination
+
+```typescript
+// ✅ CORRECT - Use ExtractionOrchestrator (handles pagination automatically)
+const orchestrator = new ExtractionOrchestrator(client, log);
+
+const extractionResult = await orchestrator.extract({
+  query: PRODUCTS_EXTRACTION_QUERY,
+  resultPath: 'products.edges.node',
+  variables: { dateRangeFilter: { after: bufferedLastRunTime } },
+  pageSize,
+  maxRecords,
+  validateItem: item => !!(item.ref && item.name),
+});
+
+const records = extractionResult.data;
+
+//  WRONG - Manual pagination (avoid this pattern)
+// const result = await client.graphql({
+//   query: PRODUCTS_QUERY,
+//   variables: { first: pageSize },
+//   pagination: { maxRecords }
+// });
+```
+
+### Pattern 2: JobTracker for Lifecycle Management
+
+```typescript
+// ✅ CORRECT - Use JobTracker throughout workflow
+const tracker = new JobTracker(kv, log);
+
+// Create job
+await tracker.createJob(jobId, { triggeredBy, fromDate, toDate });
+
+// Update progress
+await tracker.updateJob(jobId, { stage: 'extraction', message: 'Extracting data' });
+
+// Mark completed
+await tracker.markCompleted(jobId, { recordCount, fileName });
+
+// Query status
+const status = await tracker.getJob(jobId);
+```
+
+### Pattern 3: 3-Workflow Pattern
+
+```typescript
+// ✅ CORRECT - 3 workflows for different use cases
+// 1. Scheduled: Automated daily/hourly runs
+export const scheduledProductsExtraction = schedule('products-extract-xml-daily', '0 2 * * *').then(...)
+
+// 2. Ad-hoc: Manual webhook triggers with date overrides
+export const adhocProductsExtraction = webhook('products-adhoc', {
+  connection: 'products-adhoc',
+  response: { mode: 'sync' },
+}).then(...)
+
+// 3. Status: Query job status by ID
+export const productsJobStatus = webhook('products-job-status', {
+  connection: 'products-job-status',
+  response: { mode: 'sync' },
+}).then(...)
+```
+
+### Pattern 4: XMLBuilder for Safe XML Generation (CRITICAL)
+
+Use the SDK's `XMLBuilder` - it handles all XML escaping automatically:
+
+```typescript
+import { Buffer } from 'node:buffer';
+import { XMLBuilder } from '@fluentcommerce/fc-connect-sdk';
+
+// Initialize XMLBuilder (handles all escaping automatically)
+const xmlBuilder = new XMLBuilder({
+  rootElement: 'Products',
+  prettyPrint: true,
+  encoding: 'UTF-8',
+});
+
+// ✅ CORRECT: XMLBuilder escapes automatically
+const products = [
+  {
+    title: 'Smith & Jones <Corp>', // Contains & and <>
+    description: 'Special chars: ¢, ©, ®, "quotes"',
+  },
+];
+
+const xml = xmlBuilder.build({ Product: products });
+// Result: All special characters properly escaped
+// <title>Smith &amp; Jones &lt;Corp&gt;</title>
+// <description>Special chars: ¢, ©, ®, &quot;quotes&quot;</description>
+
+//  WRONG: Manual string concatenation (dangerous)
+// const xml = `<title>${product.title}</title>`;
+// This would produce INVALID XML: <title>Smith & Jones <Corp></title>
+```
+
+**Why XMLBuilder?**
+
+- ✅ Automatic escaping of &, <, >, ", '
+- ✅ Handles special characters (¢, ©, ®)
+- ✅ Prevents XML injection attacks
+- ✅ Validates structure
+- ✅ Consistent, maintainable code
+
+### Pattern 5: SFTP Cleanup (CRITICAL)
+
+```typescript
+const sftp = new SftpDataSource(config, log);
+
+try {
+  await sftp.uploadFile(remotePath, buffer);
+  return { success: true };
+} finally {
+  // ALWAYS dispose SFTP connection
+  await sftp.dispose();
+}
+```
+
+**Why?** SFTP maintains open connections. Not calling `dispose()` leads to connection exhaustion.
+
+### Pattern 6: Consistent Field Names Across Formats
+
+**Same data in CSV, JSON, and XML:**
+
+- `sku` (not productId, not sku_ref, not SKU)
+- `title` (consistent with CSV version)
+- `catalogue_ref` (matches CSV exactly)
+
+This allows users to switch formats without changing downstream systems.
+
+---
+
+### Pattern 7: State Management & Date Overrides
+
+**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
+
+**How it works**:
+
+VersoriKV stores the last successful extraction timestamp to enable incremental sync:
+
+```typescript
+interface ExtractionState {
+  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
+  recordCount: number; // Number of records extracted
+  extractedAt: string; // When extraction completed
+  fileName?: string; // Generated filename
+  remotePath?: string; // SFTP upload path
+  overlapBufferSeconds?: number; // Buffer configuration
+}
+```
+
+**State Priority Chain** (highest to lowest):
+
+1. **`fromDate` override** (manual date in webhook payload) - Highest priority
+2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
+3. **`fallbackStartDate`** (activation variable) - First run fallback
+
+**Three Scenarios**:
+
+#### Scenario 1: Normal Scheduled Runs (Incremental)
+
+```typescript
+// Payload: {} (empty - no overrides)
+
+// Behavior:
+// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
+// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
+// 3. Extract records updated since buffered time
+// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
+// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
+// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
+```
+
+**Test**:
+
+```bash
+# Trigger scheduled run (no payload needed)
+# State advances automatically
+curl -X POST https://workspace.versori.run/products-extract-daily
+```
+
+#### Scenario 2: Ad-hoc Extraction WITH State Update
+
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
+
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
+// 3. Extract all records since 2025-01-01
+// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
+// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
+// 6. Next scheduled run starts from this new timestamp
+```
+
+**Use Case**: One-time catch-up extraction that advances the state pointer.
+
+**Test**:
+
+```bash
+curl -X POST https://workspace.versori.run/products-extract-webhook \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "updateState": true
+  }'
+```
+
+#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
+
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
+
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z"
+// 3. Extract all records since 2025-01-01
+// 4. DO NOT update state
+// 5. Next scheduled run uses previous timestamp (unaffected)
+```
+
+**Use Case**: Historical backfill or testing without affecting incremental sync.
+
+**Test**:
+
+```bash
+curl -X POST https://workspace.versori.run/products-extract-webhook \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-31T23:59:59Z",
+    "updateState": false
+  }'
+```
+
+**Why this matters**:
+
+- **Incremental sync** relies on state continuity
+- **Manual overrides** allow catch-up without breaking incremental flow
+- **Overlap buffer** prevents missed records at time boundaries
+- **State isolation** lets you test/backfill without affecting production sync
+
+---
+
+### Pattern 8: Optional GraphQL Query Logging
+
+**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
+
+**When to use**:
+
+- ✅ Debugging pagination issues
+- ✅ Verifying query variables (dates, filters, limits)
+- ✅ Development and testing
+-  Production (verbose logs, potential secrets in variables)
+
+**How to enable**:
+
+Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
+
+**Implementation**:
+
+```typescript
+// In your extraction workflow
+const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
+
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Query Debug', {
+    query: PRODUCTS_QUERY,
+    variables: {
+      catalogues,
+      dateRangeFilter: bufferedLastRunTime,
+      first: pageSize,
+      after: null, // First page
+    },
+    pagination: {
+      pageSize,
+      maxRecords,
+      currentPage: 1,
+    },
+  });
+}
+
+const extractionResult = await orchestrator.extract({
+  query: PRODUCTS_QUERY,
+  resultPath: 'products.edges.node',
+  variables: {
+    catalogues,
+    dateRangeFilter: bufferedLastRunTime,
+  },
+  pageSize,
+  maxRecords,
+});
+
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Response Debug', {
+    totalRecords: extractionResult.stats.totalRecords,
+    totalPages: extractionResult.stats.totalPages,
+    validRecords: extractionResult.stats.validRecords ?? extractionResult.data.length,
+    firstRecordId: extractionResult.data[0]?.id,
+    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
+  });
+}
+```
+
+**What gets logged**:
+
+```json
+{
+  "level": "info",
+  "message": "GraphQL Query Debug",
+  "query": "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)",
+  "variables": {
+    "catalogues": [{ "ref": "DEFAULT_CATALOGUE" }],
+    "dateRangeFilter": "2025-01-22T09:59:00Z",
+    "first": 200,
+    "after": null
+  },
+  "pagination": {
+    "pageSize": 200,
+    "maxRecords": 50000,
+    "currentPage": 1
+  }
+}
+```
+
+**Versori Environment Variables**:
+
+Add to activation settings:
+
+```json
+{
+  "DEBUG_GRAPHQL": "true"
+}
+```
+
+**Testing**:
+
+```bash
+# Enable debug logging
+curl -X POST https://workspace.versori.run/products-extract-daily
+
+# Check Versori logs for "GraphQL Query Debug" entries
+# Verify query structure and variables are correct
+```
+
+**Sample Debug Output**:
+
+```
+[INFO] GraphQL Query Debug
+  query: "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)"
+  variables: { catalogues: [{ ref: "DEFAULT_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
+  pagination: { pageSize: 200, maxRecords: 50000, currentPage: 1 }
+
+[INFO] Extraction complete
+  totalRecords: 1250
+  totalPages: 7
+  validRecords: 1250
+  failedValidations: 0
+
+[INFO] GraphQL Response Debug
+  totalRecords: 1250
+  totalPages: 7
+  validRecords: 1250
+  firstRecordId: "product_abc"
+  lastRecordId: "product_xyz"
+```
+
+**Key Benefits**:
+
+- Quickly identify pagination configuration issues
+- Verify date filters are applied correctly
+- Debug "no records found" scenarios
+- Validate ExtractionOrchestrator variable injection
+
+**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
+
+---
+
+## Common Issues
+
+**Issue 1: Malformed XML from unescaped characters**
+
+- Customer name contains `&` or `<`
+- Solution: Always use XMLBuilder (automatic escaping)
+
+**Issue 2: Partner system rejects XML**
+
+- Missing required fields
+- Solution: Verify mapping matches partner schema requirements
+
+**Issue 3: File too large for SFTP partner**
+
+- Partner has 50MB limit, file is 100MB
+- Solution: Use file splitting (10k products per file)
+
+**Issue 4: SFTP connection timeouts**
+
+- Not calling `dispose()` in finally block
+- Solution: Always use try/finally pattern
+
+**Issue 5: Job status not updating**
+
+- JobTracker not integrated
+- Solution: Use JobTracker throughout workflow
+
+## Testing
+
+### 1. Test XML Structure
+
+```typescript
+export const testXmlGeneration = http('test-xml').then(
+  fn('test-xml-gen', async () => {
+    const testProducts = [
+      {
+        sku: 'TEST-001',
+        title: 'Test & Validate <Product>',
+        status: 'ACTIVE',
+        price: 29.99,
+        created_on: '2025-01-22T10:00:00Z',
+        updated_on: '2025-01-22T10:00:00Z',
+      },
+    ];
+
+    const xml = buildProductsXML(testProducts);
+
+    // Validate XML structure
+    if (!xml.includes('<?xml version="1.0"')) {
+      return { success: false, error: 'Missing XML declaration' };
+    }
+
+    if (!xml.includes('&amp;') || !xml.includes('&lt;')) {
+      return { success: false, error: 'Special characters not escaped' };
+    }
+
+    return { success: true, xml };
+  })
+);
+```
+
+### 2. Test SFTP Upload
+
+```bash
+curl https://your-workspace.versori.run/test-sftp-products-xml
+```
+
+### 3. Validate Against Partner Schema
+
+- Download partner's XSD schema
+- Validate generated XML against schema
+- Fix any missing/incorrect elements
+
+## Production Checklist
+
+- [ ] Test SFTP credentials and connection
+- [ ] Verify SFTP server has write permissions to remotePath
+- [ ] Set appropriate extraction frequency (daily for product feeds)
+- [ ] Configure correct product status filters
+- [ ] Test XML escaping with special characters (&, <, >, ", ')
+- [ ] Validate XML against partner's schema (if provided)
+- [ ] Test `dispose()` is always called (check logs)
+- [ ] Document XML schema for partner integration team
+- [ ] Set up monitoring for SFTP connection failures
+- [ ] Test with real product data (names with special chars)
+- [ ] Verify file size limits with SFTP partner
+- [ ] Configure SFTP server IP whitelisting for Versori
+- [ ] Test file splitting with large batches (>10k products)
+- [ ] Test all 3 workflows (scheduled, ad-hoc, status)
+- [ ] Verify JobTracker integration and status updates
+- [ ] Test ExtractionOrchestrator pagination with large datasets
+
+## Troubleshooting Guide
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records
+- **Fix**: Reduce maxRecords, increase frequency
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch
+- **Fix**: Run schema validation, check field names
+
+**Issue**: "State not updating"
+
+- **Cause**: KV write failure or intentional retry
+- **Fix**: Check KV logs, verify state update code
+
+**Issue**: "First run exceeds limits"
+
+- **Cause**: No previous timestamp, fetches all
+- **Fix**: Set fallbackStartDate close to current, apply filters
+
+**Issue**: "Excessive duplicates"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved
+- **Fix**: Verify newTimestamp saved WITHOUT buffer
+
+**Issue**: "Job status returns null"
+
+- **Cause**: Invalid job ID or job expired
+- **Fix**: Verify job ID format, check KV TTL settings
+
+## Security Best Practices
+
+### Credential Management
+
+**✅ DO**:
+
+- Store credentials in Versori activation variables
+- Rotate credentials quarterly
+- Use least-privilege accounts
+
+** DON'T**:
+
+- Never log credentials
+- Never commit to git
+- Never share across environments
+
+### Data Security
+
+- Enable encryption in transit and at rest
+- Apply data retention policies
+- Monitor access logs
+- Use VPC/private networks for sensitive data
+
+### Webhook Security
+
+- Validate API keys for ad-hoc and status workflows
+- Use HTTPS for all webhook endpoints
+- Implement rate limiting
+- Monitor for suspicious activity
+
+---
+
+**Pattern**: Enterprise incremental extraction with ExtractionOrchestrator + JobTracker for products via SFTP (XML format)
+**❌š ï¸ Versori Sample**: Reference implementation - adapt for your production use case
+**Key Learning**: Use ExtractionOrchestrator for auto-pagination, JobTracker for lifecycle management, always escape XML and dispose SFTP
+**Critical**: Apply 60-second overlap buffer to prevent missed records
+**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
+**Field Consistency**: Same field names as CSV version for easy format switching
+**SFTP**: Use proper connection cleanup in finally block to prevent connection leaks
+**XML**: Preserve hierarchical structure (no flattening needed like CSV)
+**3 Workflows**: Scheduled, ad-hoc webhook, job status query
+
+---
+
+### Pattern 8: Backward Pagination (Optional - Advanced)
+
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+
+**When to Use**:
+
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+-  **Don't use for standard incremental sync** - use forward pagination (default)
+
+**GraphQL Query Requirements**:
+
+Your query must support backward pagination by including `$last` and `$before`:
+
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ✅ REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ✅ REQUIRED for backward
+    }
+  }
+}
+```
+
+**Implementation**:
+
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    //  Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ✅ Enable reverse pagination
+  maxRecords: 10000,
+});
+
+// Records are returned in reverse chronological order
+console.log(result.data[0].createdAt); // Newest
+console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
+```
+
+**Key Differences from Forward Pagination**:
+
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest               | Newest → Oldest      |
+
+**Important Notes**:
+
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+
+**Example: Extract Latest 1000 Orders**
+
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+
+**When to Use Forward vs Backward**:
+
+```typescript
+// ✅ Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+
+// ✅ Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+
+**Pagination Variables Reference**:
+
+| Variable | Forward      | Backward     | Injected By  | Notes                    |
+| -------- | ------------ | ------------ | ------------ | ------------------------ |
+| `first`  | ✅ Used      |  Not used | Orchestrator | From `pageSize`          |
+| `after`  | ✅ Used      |  Not used | Orchestrator | From cursor (last edge)  |
+| `last`   |  Not used | ✅ Used      | Orchestrator | From `pageSize`          |
+| `before` |  Not used | ✅ Used      | Orchestrator | From cursor (first edge) |
+
+**Common Mistakes to Avoid**:
+
+```typescript
+//  WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, //  Orchestrator will override this
+    before: cursor, //  Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+
+// ✅ CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ✅ Your business variables only
+  },
+  pageSize: 200, // ✅ Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+
+#### Optional: Reverse Pagination
+
+- For reverse ordering, add $last/$before and pageInfo.hasPreviousPage to your query and set direction='backward'.
+
+GraphQL:
+
+```graphql
+query GetProductsBackward($last: Int!, $before: String) {
+  products(last: $last, before: $before) {
+    edges {
+      cursor
+      node {
+        id
+        ref
+        updatedOn
+      }
+    }
+    pageInfo {
+      hasPreviousPage
+    }
+  }
+}
+```
+
+SDK:
+
+```typescript
+await orchestrator.extract({
+  query: PRODUCTS_BACKWARD_QUERY,
+  resultPath: 'products.edges.node',
+  variables: {},
+  pageSize,
+  direction: 'backward',
+});
+```
+
+---
+
+## Testing Checklist
+
+**Before production deployment:**
+
+### 1. Schema Validation
+
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/products.export.xml.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/products.export.xml.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+
+### 2. Extraction Testing
+
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+
+### 3. Mapping Testing
+
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+
+### 4. XML Generation Testing
+
+- [ ] Verify XML structure matches expected format
+- [ ] Test XML validation against XSD schema (if applicable)
+- [ ] Verify special character escaping in XML
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test XML namespace handling (if applicable)
+
+### 5. SFTP Upload Testing
+
+- [ ] Test SFTP connection and authentication
+- [ ] Verify file upload to correct path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify file permissions on SFTP server
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify SFTP connection disposal (no connection leaks)
+
+### 6. State Management Testing
+
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+
+### 7. Job Tracking Testing
+
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+
+### 8. Error Handling Testing
+
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with SFTP connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+
+### 9. Staging Environment Testing
+
+- [ ] Run full extraction in staging environment
+- [ ] Verify XML file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+
+### 10. Integration Testing
+
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger → extract → transform → upload → verify file
+
+---
+## Monitoring & Alerting
+
+### Success Response Example
+
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_PRD_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "products-2025-11-02T14-00-00-000Z.xml",
+  "sftpPath": "/outbound/products/products-2025-11-02T14-00-00-000Z.xml",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+
+### Error Response Example
+
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_PRD_20251102_140500_xyz789",
+  "error": "SFTP upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "sftp_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+
+### Key Metrics to Track
+
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+
+  // File Generation
+  fileSizeMB: (xmlContent.length / (1024 * 1024)).toFixed(2),
+
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+
+log.info('Extraction metrics', metrics);
+```
+
+### Alert Thresholds
+
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+
+### Monitoring Dashboard Queries
+
+**Versori Platform Logs Query:**
+
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+
+# High error rates
+errorRate:>5
+
+# State management issues
+stateUpdated:false AND success:true
+```
+
+### Common Issues and Solutions
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+
+**Issue**: "SFTP connection timeout"
+
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check SFTP credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+
+**Issue**: "State not updating after successful extraction"
+
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+
+**Issue**: "First run exceeds record limits"
+
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+
+**Issue**: "Excessive duplicate records in output"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+
+---
+
+## Troubleshooting Quick Reference
+
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "XML generation failed" | Format-specific error | Check XML generation logic, validate output |
+
+---