@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-products-to-s3-json.md

@@ -1,2384 +1,2384 @@
----
-template_id: tpl-extract-products-graphql-to-s3-json
-canonical_filename: template-extraction-products-to-s3-json.md
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: s3-json
-entity: products
-format: json
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
----
-
-# Template: Extraction - Products GraphQL to S3 JSON
-
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
----
-
-## 📋 Implementation Prompt
-
-Copy/paste the standardized prompt from `docs/template-loading-matrix.md#prompts`.
-
----
-
-## 💻 STEP 3: Implementation (Verified Imports)
-
-```ts
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-  VersoriKVAdapter,
-  ExtractionOrchestrator,
-  JobTracker,
-} from '@fluentcommerce/fc-connect-sdk';
-```
-
-These are the only SDK imports required for this template. Keep type-only imports out of code samples. Prefer the orchestrator-based flow when you need built-in pagination and stats.
-
----
-
-# Versori Workflows: Product Catalog Extraction to S3 JSON
-
-**FC Connect SDK Use Case Guide**
-
-> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> Version: ^0.1.39
-
-Context: Three Versori workflows for extracting product catalog from Fluent Commerce via GraphQL query with **incremental timestamp tracking**, **ad hoc extraction**, and **job status monitoring**. Transforms with `UniversalMapper`, writes JSON files to S3 for marketplace integration (Amazon, eBay, etc).
-
-**Pattern**: EXTRACTION (Fluent → S3 JSON)
-**Complexity**: Medium | Runtime: Versori Platform
-**Format**: JSON with metadata wrapper
-
----
-
-## ⚠️ IMPORTANT: Production-Ready Base Template
-
-> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
->
-> This is a **production-ready base template** demonstrating FC Connect SDK best practices for product extraction workflows with JSON output to S3.
->
-> **✅ INCLUDED FEATURES:**
->
-> - ✅ Comprehensive error handling with retry logic
-> - ✅ S3 upload with proper error handling
-> - ✅ State management with overlap buffer (prevents missed records)
-> - ✅ Job tracking with lifecycle management
-> - ✅ Security (credential masking in logs)
-> - ✅ UTC time enforcement (prevents timezone bugs)
-> - ✅ Incremental extraction (safe, efficient, production-ready)
-> - ✅ Natural rate limiting via timestamps
->
-> **📝 BEFORE DEPLOYING:**
->
-> 1. Review and customize activation variables for your environment
-> 2. Test with sample data in your Versori workspace
-> 3. Adjust safety limits (pageSize, maxRecords) if needed
-> 4. Configure monitoring alerts for extraction failures
-> 5. Verify S3 bucket credentials and paths
->
-> **This base template follows SDK best practices - tweak specific to your needs.**
-
----
-
-## What You'll Build
-
-**Three Versori Workflows:**
-
-1. **Scheduled Extraction** - Daily/hourly incremental product updates
-2. **Ad Hoc Extraction** - On-demand HTTP webhook for manual triggers
-3. **Job Status Checker** - Monitor extraction job status via JobTracker
-
-**Features:**
-
-- **Incremental extraction** using `updatedOn > lastRunTime` filter
-- **State management** with VersoriKVAdapter to track last successful run
-- **Job tracking** with JobTracker for status monitoring
-- GraphQL query with auto-pagination
-- UniversalMapper transformation for marketplace schema
-- JSON file generation with product catalog
-- S3 upload to marketplace integration bucket
-- **Failure recovery** with timestamp tracking
-- **Pretty print option** for human-readable JSON
-
-## Business Use Case
-
-**Daily product catalog sync to marketplaces:**
-
-- Extract only changed products since last run
-- Export as JSON for marketplace API consumption
-- Run daily at midnight for overnight processing
-- Include SKU, title, description, pricing, attributes
-- Enable Amazon/eBay listing updates
-- Support ad hoc manual refresh on demand
-- Monitor job status for workflow integration
-
-## SDK Methods Used
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-  VersoriKVAdapter,
-  JobTracker,
-} from '@fluentcommerce/fc-connect-sdk';
-
-await createClient(ctx); // Versori-aware client
-await client.graphql({ query, variables, pagination }); // GraphQL with auto-pagination
-new VersoriKVAdapter(ctx.openKv(':project:')); // State management
-new JobTracker(ctx.openKv(':project:'), ctx.log); // Job tracking
-new UniversalMapper(exportMapping); // Field transformation
-const jsonBuilder = new JSONBuilder({ prettyPrint: true, indent: 2 });
-const jsonContent = jsonBuilder.build(dataObject); // JSON generation
-await s3.upload(key, Buffer.from(jsonContent, 'utf8'), options); // S3 upload
-```
-
-## Activation Variables
-
-```json
-{
-  "retailerId": "your-retailer-id",
-  "s3BucketName": "marketplace-catalog-exports",
-  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
-  "awsSecretAccessKey": "********",
-  "awsRegion": "us-east-1",
-  "s3Prefix": "products/daily/",
-  "pageSize": 200,
-  "maxRecords": 50000,
-  "fallbackStartDate": "2024-01-01T00:00:00Z",
-  "overlapBufferSeconds": "60",
-  "prettyPrint": "false",
-  "validateConnectionOnStart": "false"
-}
-```
-
-**New Variables (v2.1.0):**
-- `validateConnectionOnStart`: Optional fail-fast connection validation. When `"true"`, validates connection before extraction. Default: `"false"` (validation on first API call)
-
-**Configuration Notes:**
-- All variables are required except `validateConnectionOnStart`, `overlapBufferSeconds`, and `prettyPrint`
-- Credentials should be stored securely in Versori activation variables
-- `pageSize` and `maxRecords` can be adjusted based on data volume and performance requirements
-
-## Export Mapping Configuration
-
-Create file: `./config/products.export.json`
-
-```json
-{
-  "name": "products.export",
-  "version": "1.0.0",
-  "description": "Fluent Products → Marketplace JSON Export",
-  "fields": {
-    "sku": { "source": "ref", "required": true, "resolver": "sdk.trim" },
-    "title": { "source": "name", "required": true, "resolver": "sdk.trim" },
-    "description": { "source": "summary", "required": false, "resolver": "sdk.trim" },
-    "gtin": { "source": "gtin", "required": false, "resolver": "sdk.trim" },
-    "type": { "source": "type", "required": false, "resolver": "sdk.uppercase" },
-    "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
-    "createdOn": { "source": "createdOn", "required": false, "resolver": "sdk.toString" },
-    "lastUpdated": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
-  }
-}
-```
-
-**Note**: Actual GraphQL query structure should be verified against your Fluent schema using introspection. The fields above are examples - adjust based on your actual schema.
-
-## GraphQL Query
-
-**Note**: This query is an **example**. Verify against your Fluent Commerce schema using introspection.
-
-```graphql
-query GetProducts($retailerId: ID!, $updatedAfter: DateTime, $first: Int!, $after: String) {
-  products(
-    retailerId: $retailerId
-    updatedOn: { after: $updatedAfter }
-    first: $first
-    after: $after
-  ) {
-    edges {
-      node {
-        id
-        ref
-        name
-        summary
-        gtin
-        type
-        status
-        createdOn
-        updatedOn
-      }
-      cursor
-    }
-    pageInfo {
-      hasNextPage
-    }
-  }
-}
-```
-
----
-
-## Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced) - Multi-step processes with state persistence
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Durable Workflows (Advanced Pattern)
-
-**⚠️ Note:** Durable workflows are an advanced pattern not yet used in this template. They're documented here for completeness.
-
-**Use Case:** Multi-step processes requiring state persistence, long-running operations (hours/days), human approval workflows, or complex retry/error handling across steps.
-
-**Example Pattern:**
-```typescript
-import { workflow } from '@versori/run';
-
-workflow('order-fulfillment', { connection: 'fluent_commerce' }, async (ctx, step) => {
-  // Step 1: Validate order
-  const order = await step.run('validate-order', async () => {
-    return await validateOrder(ctx);
-  });
-
-  // Step 2: Allocate inventory (with retry)
-  const allocation = await step.run('allocate-inventory', {
-    retry: { max: 3, delay: '1s' }
-  }, async () => {
-    return await allocateInventory(order);
-  });
-
-  // Step 3: Wait for approval (human-in-loop)
-  await step.sleep('wait-for-approval', '1h');
-
-  // Step 4: Create fulfillment
-  return await step.run('create-fulfillment', async () => {
-    return await createFulfillment(allocation);
-  });
-});
-```
-
-**When to use workflow():**
-- ✅ Multi-step processes requiring state persistence
-- ✅ Long-running operations (hours/days)
-- ✅ Human approval workflows
-- ✅ Complex retry/error handling across steps
-- ❌ Simple CRUD operations (use schedule/webhook + http)
-- ❌ Fire-and-forget patterns (use schedule + http)
-
-### Recommended Project Structure
-
-```
-products-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-products-extraction.ts  # Scheduled: Daily products extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-products-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts          # Webhook: Status query
-    │
-    ├── services/
-    │   └── products-extraction.service.ts   # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── products.export.json.json       # Mapping configuration
-```
-
----
-
-```json
-{
-  "metadata": {
-    "extractedAt": "2025-01-22T00:00:00.000Z",
-    "productCount": 2,
-    "incrementalFrom": "2025-01-21T00:00:00.000Z",
-    "incrementalTo": "2025-01-22T00:00:00.000Z"
-  },
-  "products": [
-    {
-      "sku": "SKU-001",
-      "title": "Premium Widget",
-      "description": "High-quality widget for all purposes",
-      "gtin": "012345678901",
-      "type": "STANDARD",
-      "status": "ACTIVE",
-      "createdOn": "2025-01-15T10:00:00Z",
-      "lastUpdated": "2025-01-21T10:30:00Z"
-    },
-    {
-      "sku": "SKU-002",
-      "title": "Deluxe Gadget",
-      "description": "Advanced gadget with premium features",
-      "gtin": "012345678902",
-      "type": "STANDARD",
-      "status": "ACTIVE",
-      "createdOn": "2025-01-16T11:00:00Z",
-      "lastUpdated": "2025-01-21T14:15:00Z"
-    }
-  ]
-}
-```
-
-## Mapping & Resolvers Explained
-
-### SDK Resolvers Used
-
-| Field         | Resolver        | Why?                    | Example                       |
-| ------------- | --------------- | ----------------------- | ----------------------------- |
-| `sku`         | `sdk.trim`      | Clean SKU               | " SKU-001 " → "SKU-001"       |
-| `title`       | `sdk.trim`      | Clean names             | " Widget " → "Widget"         |
-| `description` | `sdk.trim`      | Remove extra whitespace | " desc " → "desc"             |
-| `gtin`        | `sdk.trim`      | Normalize GTIN/UPC      | " 012345 " → "012345"         |
-| `type`        | `sdk.uppercase` | Normalize type          | "standard" → "STANDARD"       |
-| `status`      | `sdk.uppercase` | Normalize status        | "active" → "ACTIVE"           |
-| `createdOn`   | `sdk.toString`  | Ensure ISO string       | Date → "2025-01-15T10:00:00Z" |
-| `lastUpdated` | `sdk.toString`  | Ensure ISO string       | Date → "2025-01-22T14:30:00Z" |
-
-### Transformation Flow
-
-```typescript
-// 1) GraphQL node (example)
-{
-  ref: " SKU-001 ",
-  name: " Widget ",
-  summary: "  High quality  ",
-  gtin: " 012345678901 ",
-  type: "standard",
-  status: "active",
-  createdOn: "2025-01-15T10:00:00.000Z",
-  updatedOn: "2025-01-21T10:30:00.000Z"
-}
-
-// 2) Map with UniversalMapper
-const mapper = new UniversalMapper(productsExportMapping);
-const result = await mapper.map(node);
-
-// 3) Output
-result.data = {
-  sku: "SKU-001",
-  title: "Widget",
-  description: "High quality",
-  gtin: "012345678901",
-  type: "STANDARD",
-  status: "ACTIVE",
-  createdOn: "2025-01-15T10:00:00.000Z",
-  lastUpdated: "2025-01-21T10:30:00.000Z"
-};
-```
-
-### Custom Resolvers for Product-Specific Logic
-
-While the mapping above uses built-in SDK resolvers, you can extend with custom business logic for marketplace integration:
-
-```typescript
-const customResolvers = {
-  /**
-   * Extract category hierarchy for marketplace navigation
-   */
-  'custom.formatCategoryPath': (product: any) => {
-    const categories = product.categories || [];
-    return categories
-      .map((c: any) => c.name?.trim())
-      .filter(Boolean)
-      .join(' > ');
-    // Example: "Electronics > Computers > Laptops"
-  },
-
-  /**
-   * Generate marketplace-compatible SKU with prefix
-   */
-  'custom.generateMarketplaceSKU': (ref: string, retailerId: string) => {
-    return `${retailerId}-${ref.trim()}`.toUpperCase();
-    // Example: "RETAILER123-SKU-001"
-  },
-
-  /**
-   * Format price for marketplace API (convert cents to dollars)
-   */
-  'custom.formatPrice': (priceInCents: number) => {
-    const dollars = (priceInCents || 0) / 100;
-    return dollars.toFixed(2);
-    // Example: 2499 → "24.99"
-  },
-
-  /**
-   * Extract primary image URL from attributes
-   */
-  'custom.getPrimaryImage': (product: any) => {
-    const images = product.attributes?.images || [];
-    return images.length > 0 ? images[0]?.url : null;
-  },
-
-  /**
-   * Generate marketplace title with brand and product name
-   */
-  'custom.generateMarketplaceTitle': (product: any) => {
-    const brand = product.attributes?.brand?.trim() || '';
-    const name = product.name?.trim() || '';
-    const maxLength = 200; // Amazon/eBay limit
-
-    const title = brand ? `${brand} - ${name}` : name;
-    return title.length > maxLength ? title.substring(0, maxLength - 3) + '...' : title;
-  },
-
-  /**
-   * Extract product attributes for marketplace listing
-   */
-  'custom.extractAttributes': (product: any) => {
-    const attrs = product.attributes || {};
-    return {
-      brand: attrs.brand?.trim() || 'Unbranded',
-      color: attrs.color?.trim() || null,
-      size: attrs.size?.trim() || null,
-      weight: attrs.weight ? `${attrs.weight} ${attrs.weightUnit || 'lb'}` : null,
-      dimensions: attrs.dimensions || null,
-      material: attrs.material?.trim() || null,
-    };
-  },
-
-  /**
-   * Determine product eligibility for marketplaces
-   */
-  'custom.checkMarketplaceEligibility': (product: any) => {
-    const status = (product.status || '').toUpperCase();
-    const hasGTIN = !!product.gtin;
-    const hasImages = (product.attributes?.images || []).length > 0;
-    const hasPrice = (product.prices || []).length > 0;
-
-    return {
-      isActive: status === 'ACTIVE',
-      hasRequiredFields: hasGTIN && hasImages && hasPrice,
-      isEligibleForAmazon: status === 'ACTIVE' && hasGTIN && hasImages && hasPrice,
-      isEligibleForEbay: status === 'ACTIVE' && hasImages && hasPrice,
-      missingFields: [!hasGTIN && 'GTIN', !hasImages && 'Images', !hasPrice && 'Price'].filter(
-        Boolean
-      ),
-    };
-  },
-
-  /**
-   * Format for marketplace API submission
-   */
-  'custom.formatForMarketplace': (product: any) => {
-    return {
-      sku: product.ref?.trim(),
-      title: product.name?.trim(),
-      description: product.summary?.trim() || product.name?.trim(),
-      category: product.categories?.[0]?.name || 'Uncategorized',
-      brand: product.attributes?.brand?.trim() || 'Generic',
-      gtin: product.gtin?.trim() || null,
-      price: (product.prices?.[0]?.value || 0) / 100, // cents to dollars
-      currency: product.prices?.[0]?.currency || 'USD',
-      images: (product.attributes?.images || []).map((img: any) => img.url),
-      status: product.status?.toUpperCase(),
-      lastUpdated: product.updatedOn,
-    };
-  },
-};
-
-// Use custom resolvers with UniversalMapper
-const mapper = new UniversalMapper(productsExportMapping, {
-  customResolvers,
-});
-```
-
-### Available SDK Resolvers
-
-The SDK provides these built-in resolvers (no custom code needed):
-
-**String Transformations:**
-
-- `sdk.trim` - Remove leading/trailing whitespace
-- `sdk.uppercase` - Convert to uppercase
-- `sdk.lowercase` - Convert to lowercase
-- `sdk.toString` - Convert to string
-
-**Number Parsing:**
-
-- `sdk.parseInt` - Parse as integer
-- `sdk.parseFloat` - Parse as decimal (ideal for prices)
-- `sdk.number` - Parse as number (auto-detect int/float)
-
-**Date Formatting:**
-
-- `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
-- `sdk.formatDateShort` - Short date format
-- `sdk.parseDate` - Parse various date formats
-
-**Type Conversions:**
-
-- `sdk.boolean` - Convert to boolean
-- `sdk.parseJson` - Parse JSON strings (useful for attributes)
-- `sdk.toJson` - Convert to JSON string
-
-**Utilities:**
-
-- `sdk.identity` - Return value unchanged
-- `sdk.coalesce` - Return first non-null value
-
-See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
-
-## Production Safety & Guardrails
-
-### Overview
-
-Even with **incremental-only** extraction, product catalogs need safeguards to prevent runtime failures:
-
-- **Memory limits**: Product records can be large (descriptions, attributes, images)
-- **S3 upload limits**: Single files > 5GB require multipart upload
-- **Processing time**: Large catalogs can timeout
-- **JSON parsing**: Massive JSON files stress memory and downstream parsers
-
-### Hard Limits
-
-```typescript
-const SAFETY_LIMITS = {
-  MAX_RECORDS_PER_RUN: 100000, // 100k products per run
-  MAX_RECORDS_PER_FILE: 25000, // 25k per JSON file
-  MAX_FILE_SIZE_MB: 250, // 250MB per file
-  MAX_JSON_SIZE_MB: 500, // Total extraction size
-  CHUNK_SIZE: 10000, // Process in chunks
-  ESTIMATED_BYTES_PER_PRODUCT: 2048, // Conservative estimate (2KB per product)
-};
-```
-
-**Why these limits?**
-
-- Products have more fields than orders/fulfillments (descriptions, attributes, variants)
-- JSON is less compact than CSV
-- Marketplace APIs often have file size limits (Amazon: 100MB compressed)
-
-### Runtime Validation Function
-
-```typescript
-/**
- * Validate extraction safety limits before processing
- */
-function validateExtractionLimits(recordCount: number, estimatedSizeMB: number) {
-  const MAX_RECORDS_PER_RUN = 100000;
-  const MAX_JSON_SIZE_MB = 500;
-
-  if (recordCount > MAX_RECORDS_PER_RUN) {
-    return {
-      valid: false,
-      error: `Extraction limit exceeded: ${recordCount} records (max: ${MAX_RECORDS_PER_RUN})`,
-      recommendation: `Catalog too large for single extraction. Consider:
-        1. Increase extraction frequency to reduce batch size
-        2. Filter by product status (ACTIVE only)
-        3. Split by product type or category
-        4. Use multiple extractions with different filters`,
-      recordCount,
-      maxAllowed: MAX_RECORDS_PER_RUN,
-    };
-  }
-
-  if (estimatedSizeMB > MAX_JSON_SIZE_MB) {
-    return {
-      valid: false,
-      error: `JSON size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_JSON_SIZE_MB}MB)`,
-      recommendation: 'File splitting required. Consider filtering or splitting by category.',
-      estimatedSizeMB,
-      maxAllowed: MAX_JSON_SIZE_MB,
-    };
-  }
-
-  return { valid: true };
-}
-```
-
-## Complete Workflows Implementation
-
-The code examples below demonstrate what goes in each file according to the modular structure shown in the "Recommended Project Structure" section above.
-
-### Workflow 1: Scheduled Incremental Extraction
-**File:** `src/workflows/scheduled/daily-products-extraction.ts`
-
-```typescript
-import { schedule, http } from '@versori/run';
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-  VersoriKVAdapter,
-  JobTracker,
-} from '@fluentcommerce/fc-connect-sdk';
-import productsExportMapping from './config/products.export.json' with { type: 'json' };
-
-const PRODUCTS_QUERY = `
-  query GetProducts(
-    $retailerId: ID!
-    $updatedAfter: DateTime
-    $first: Int!
-    $after: String
-  ) {
-    products(
-      retailerId: $retailerId
-      updatedOn: { after: $updatedAfter }
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          name
-          summary
-          gtin
-          type
-          status
-          createdOn
-          updatedOn
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-
-/**
- * WORKFLOW 1/3: Scheduled Daily Product Extraction (Incremental)
- *
- * Runs daily at midnight to extract changed products
- */
-export const scheduledProductsExtraction = schedule('scheduled-products-extraction', '0 0 * * *').then(
-  http('extract-products', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, activation } = ctx;
-    const executionStartTime = Date.now();
-
-    log.info('=== EXECUTION START ===', { timestamp: new Date().toISOString() });
-
-    // STEP 1/8: Parse activation variables
-    const retailerId = activation?.getVariable('retailerId');
-    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
-    const maxRecords = parseInt(activation?.getVariable('maxRecords') || '50000', 10);
-    const fallbackStartDate =
-      activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
-    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
-
-    const s3Config = {
-      bucket: activation?.getVariable('s3BucketName'),
-      region: activation?.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation?.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
-    };
-    const s3Prefix = activation?.getVariable('s3Prefix') || 'products/daily/';
-
-    // Validate required variables
-    const missing: string[] = [];
-    if (!retailerId) missing.push('retailerId');
-    if (!s3Config.bucket) missing.push('s3BucketName');
-    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
-    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
-    if (missing.length) {
-      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
-    }
-
-    try {
-      // STEP 2/8: Initialize JobTracker and start tracking
-      const tracker = new JobTracker(openKv(':project:'), log);
-      const jobId = `products-extraction-${Date.now()}`;
-
-      await tracker.createJob(jobId, {
-        type: 'extraction',
-        entity: 'products',
-        mode: 'scheduled',
-        retailerId,
-        startTime: executionStartTime,
-      });
-
-      // STEP 3/8: Load last successful extraction timestamp with overlap buffer
-      const kv = new VersoriKVAdapter(openKv(':project:'));
-      const stateKey = ['extraction', 'products', 'lastRunTime'];
-      const lastRunState = await kv.get(stateKey);
-      const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
-
-      // Overlap buffer configuration (default: 60 seconds)
-      const overlapBufferSeconds = parseInt(
-        activation?.getVariable('overlapBufferSeconds') || '60',
-        10
-      );
-      const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
-
-      // Apply overlap buffer for query (safety window)
-      const bufferedLastRunTime = new Date(
-        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
-      ).toISOString();
-
-      const toDate = undefined; // No manual override for scheduled extraction
-
-      log.info('🔍 Starting incremental products extraction with overlap buffer', {
-        jobId,
-        rawLastRunTime,
-        bufferedLastRunTime,
-        effectiveEndTime: toDate || new Date().toISOString(),
-        overlapBufferSeconds,
-        retailerId,
-        maxRecords,
-      });
-
-      // STEP 4/8: Initialize Fluent client + ExtractionOrchestrator
-      // ✅ Optional: Validate connection immediately (fail-fast mode)
-      // Set activation variable 'validateConnectionOnStart' = 'true' to enable
-      // When enabled: Executes query { me { ref } } to verify authentication
-      // When disabled: Fast creation, validation happens on first API call (default)
-      const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
-      const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
-      
-      if (validateConnection) {
-        log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
-      }
-      
-      const orchestrator = new ExtractionOrchestrator(client, log);
-
-      // STEP 5/8: Execute extraction with auto-pagination (WITH overlap buffer)
-      const effectiveEndTime = toDate || new Date().toISOString();
-      
-      // ? Enhanced: Extract context for progress logging
-      const dateRangeInfo = {
-        start: bufferedLastRunTime,
-        end: effectiveEndTime,
-        retailerId
-      };
-
-      // ? Enhanced: Start logging with context
-      log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
-       query: 'products',
-       pageSize,
-       maxRecords,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       retailerId: dateRangeInfo.retailerId,
-       jobId
-      });
-
-      const extractionResult = await orchestrator.extract({
-        query: PRODUCTS_QUERY,
-        resultPath: 'products.edges.node',
-        variables: {
-          retailerId,
-          dateRangeFilter: {
-            after: bufferedLastRunTime,
-            before: effectiveEndTime, // End of extraction window
-          },
-        },
-        pageSize,
-        maxRecords,
-        validateItem: item => !!item.ref,
-      });
-
-      const rawRecords = extractionResult.data;
-
-      log.info('Products extraction completed', {
-        jobId,
-        totalRecords: extractionResult.stats.totalRecords,
-        totalPages: extractionResult.stats.totalPages,
-        validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-        errors: extractionResult.errors ? extractionResult.errors.length : 0,
-      });
-
-      // ? Enhanced: Completion logging with summary
-      log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-       totalRecords: extractionResult.stats.totalRecords,
-       totalPages: extractionResult.stats.totalPages,
-       validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-       failedValidations: extractionResult.stats.failedValidations,
-       truncated: extractionResult.stats.truncated,
-       truncationReason: extractionResult.stats.truncationReason,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       jobId
-      });
-
-      if (extractionResult.errors && extractionResult.errors.length > 0) {
-        log.warn('Non-fatal extraction errors encountered', {
-          jobId,
-          errorCount: extractionResult.errors.length,
-          sampleErrors: extractionResult.errors.slice(0, 3),
-        });
-      }
-
-      if (rawRecords.length === 0) {
-        log.info('No new products to extract');
-        await kv.set(stateKey, {
-          timestamp: new Date().toISOString(),
-          productCount: 0,
-          extractedAt: new Date().toISOString(),
-        });
-        await tracker.markCompleted(jobId, {
-          recordsProcessed: 0,
-          message: 'No new products to extract',
-        });
-        return {
-          success: true,
-          message: 'No new products to extract',
-          jobId,
-          lastRunTime: rawLastRunTime,
-        };
-      }
-
-      log.info('Products retrieved', { jobId, count: rawRecords.length });
-
-      // STEP 6/8: Validate extraction limits
-      const MAX_RECORDS_PER_RUN = 100000;
-      const ESTIMATED_BYTES_PER_PRODUCT = 2048;
-      const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_PRODUCT) / (1024 * 1024);
-      const MAX_JSON_SIZE_MB = 500;
-
-      if (rawRecords.length > MAX_RECORDS_PER_RUN) {
-        await tracker.markFailed(jobId, {
-          error: 'Extraction limit exceeded',
-          recordCount: rawRecords.length,
-          maxAllowed: MAX_RECORDS_PER_RUN,
-        });
-        log.error('Extraction limit exceeded', {
-          recordCount: rawRecords.length,
-          maxAllowed: MAX_RECORDS_PER_RUN,
-        });
-        return {
-          success: false,
-          error: `Extraction limit exceeded: ${rawRecords.length} records (max: ${MAX_RECORDS_PER_RUN})`,
-          recommendation: `Catalog too large for single extraction. Consider:
-            1. Increase extraction frequency to reduce batch size
-            2. Filter by product status (ACTIVE only)
-            3. Split by product type or category
-            4. Use multiple extractions with different filters`,
-          recordCount: rawRecords.length,
-          maxAllowed: MAX_RECORDS_PER_RUN,
-        };
-      }
-
-      if (estimatedSizeMB > MAX_JSON_SIZE_MB) {
-        log.warn('JSON size approaching limit', {
-          estimatedSizeMB: estimatedSizeMB.toFixed(2),
-          maxAllowed: MAX_JSON_SIZE_MB,
-          recommendation: 'Consider file splitting or category-based filtering',
-        });
-      }
-
-      await tracker.updateJobProgress(jobId, {
-        stage: 'validation_complete',
-        recordCount: rawRecords.length,
-        estimatedSizeMB: estimatedSizeMB.toFixed(2),
-      });
-
-      // STEP 7/8: Transform with UniversalMapper
-      const mapper = new UniversalMapper(productsExportMapping);
-      const mappingResult = await mapper.map(rawRecords);
-
-      if (!mappingResult.success) {
-        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-        await tracker.markFailed(
-          jobId,
-          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
-          {
-            failedCount: mappingErrors.length,
-            errors: mappingErrors,
-          }
-        );
-        return {
-          success: false,
-          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-          errors: mappingErrors,
-        };
-      }
-
-      const transformedProducts = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-      const mappingErrors = mappingResult.errors || [];
-
-      if (mappingErrors.length > 0) {
-        log.warn('Some products failed transformation', {
-          jobId,
-          errorCount: mappingErrors.length,
-          sampleErrors: mappingErrors.slice(0, 3),
-        });
-      }
-
-      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-          jobId,
-          skippedFields: mappingResult.skippedFields,
-          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-        });
-      }
-
-      if (transformedProducts.length === 0) {
-        await tracker.markFailed(jobId, 'All records failed mapping', {
-          failedCount: mappingErrors.length,
-          errors: mappingErrors,
-        });
-        return {
-          success: false,
-          error: 'All records failed mapping',
-          errors: mappingErrors,
-        };
-      }
-
-      log.info('Records transformed', {
-        jobId,
-        successful: transformedProducts.length,
-        skippedRecords: rawRecords.length - transformedProducts.length,
-      });
-
-      await tracker.updateJobProgress(jobId, {
-        stage: 'transformation_complete',
-        transformedCount: transformedProducts.length,
-        failedCount: mappingErrors.length,
-      });
-
-      // Calculate max updatedOn for next run (WITHOUT buffer)
-      const maxUpdatedOn = transformedProducts.reduce((max, product) => {
-        const productTime = new Date(product.lastUpdated).getTime();
-        return productTime > max ? productTime : max;
-      }, new Date(rawLastRunTime).getTime());
-
-      const newTimestamp = new Date(maxUpdatedOn).toISOString();
-
-      // Build JSON with metadata
-      const jsonOutput = {
-        metadata: {
-          extractedAt: new Date().toISOString(),
-          productCount: transformedProducts.length,
-          incrementalFrom: rawLastRunTime,
-          incrementalTo: newTimestamp,
-        },
-        products: transformedProducts,
-      };
-
-      // Use JSONBuilder for consistent JSON generation
-      const jsonBuilder = new JSONBuilder({
-        prettyPrint,
-        indent: 2,
-      });
-      const jsonContent = jsonBuilder.build(jsonOutput);
-
-      // Generate timestamped filename
-      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-      const fileName = `products-${timestamp}.json`;
-      const s3Key = `${s3Prefix}${fileName}`;
-
-      log.info('Generated JSON file', {
-        jobId,
-        fileName,
-        size: jsonContent.length,
-        productCount: transformedProducts.length,
-      });
-
-      // STEP 8/8: Upload to S3
-      const s3 = new S3DataSource(
-        {
-          type: 'S3_JSON',
-          connectionId: 's3-products-export',
-          name: 'S3 Products Export',
-          s3Config,
-        },
-        log
-      );
-
-      await s3.upload(s3Key, Buffer.from(jsonContent, 'utf8'), {
-        contentType: 'application/json',
-        metadata: {
-          productCount: String(transformedProducts.length),
-          extractedAt: new Date().toISOString(),
-          incrementalFrom: rawLastRunTime,
-          incrementalTo: newTimestamp,
-        },
-      });
-
-      log.info('JSON file uploaded to S3', { jobId, s3Key });
-
-      // Update state with new timestamp (WITHOUT buffer)
-      await kv.set(stateKey, {
-        timestamp: newTimestamp, // ← NO buffer applied
-        productCount: transformedProducts.length,
-        extractedAt: new Date().toISOString(),
-        overlapBufferSeconds,
-        fileName,
-        s3Key,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      });
-
-      log.info('State updated with new timestamp (without buffer)', {
-        jobId,
-        newTimestamp,
-        overlapBufferSeconds,
-      });
-
-      // Complete job tracking
-      const executionDurationMs = Date.now() - executionStartTime;
-      await tracker.markCompleted(jobId, {
-        recordsProcessed: transformedProducts.length,
-        recordsFailed: mappingErrors.length,
-        fileName,
-        s3Key,
-        newTimestamp,
-        executionDurationMs,
-        errors: mappingErrors,
-      });
-
-      log.info('=== EXECUTION END ===', {
-        timestamp: new Date().toISOString(),
-        durationMs: executionDurationMs,
-        success: true,
-      });
-
-      return {
-        success: true,
-        jobId,
-        productsExtracted: transformedProducts.length,
-        recordsFailed: mappingErrors.length,
-        fileName,
-        s3Key,
-        lastRunTime: rawLastRunTime,
-        newTimestamp,
-        executionDurationMs,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      };
-    } catch (error: any) {
-      const executionDurationMs = Date.now() - executionStartTime;
-      log.error('Extraction failed', error, {
-        message: error?.message,
-        executionDurationMs,
-      });
-
-      log.info('=== EXECUTION END ===', {
-        timestamp: new Date().toISOString(),
-        durationMs: executionDurationMs,
-        success: false,
-      });
-
-      return {
-        success: false,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-        executionDurationMs,
-      };
-    }
-  })
-);
-```
-
-### Workflow 2: Ad Hoc HTTP Extraction
-**File:** `src/workflows/webhook/adhoc-products-extraction.ts`
-
-```typescript
-/**
- * WORKFLOW 2/3: Ad Hoc Product Extraction (HTTP Webhook)
- *
- * Manual trigger for on-demand product catalog extraction
- *
- * Usage (payload examples):
- * 1) Incremental (use stored state):
- *    {}
- *
- * 2) Date range override (preferred keys):
- *    {
- *      "fromDate": "2025-01-01T00:00:00Z",
- *      "toDate": "2025-01-22T00:00:00Z",
- *      "updateState": false
- *    }
- *
- * 3) Backward-compatible keys (startDate/endDate) are also accepted and mapped:
- *    {
- *      "startDate": "2025-01-01T00:00:00Z",
- *      "endDate": "2025-01-22T00:00:00Z"
- *    }
- */
-export const adHocProductsExtraction = webhook('extract-products-adhoc', {
-  connection: 'products-adhoc',
-}).then(
-  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, activation, data } = ctx;
-    const executionStartTime = Date.now();
-
-    log.info('=== EXECUTION START ===', { timestamp: new Date().toISOString() });
-
-    // Parse request body and support both new (fromDate/toDate) and alternative (startDate/endDate) keys
-    const requestData = typeof data === 'string' ? JSON.parse(data) : data;
-    const fromDate = requestData?.fromDate || requestData?.startDate;
-    const toDate = requestData?.toDate || requestData?.endDate;
-    const updateState = requestData?.updateState === true; // default: false
-    const maxRecordsOverride = requestData?.maxRecords;
-
-    const retailerId = activation?.getVariable('retailerId');
-    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
-    const maxRecords =
-      maxRecordsOverride || parseInt(activation?.getVariable('maxRecords') || '50000', 10);
-    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
-
-    const s3Config = {
-      bucket: activation?.getVariable('s3BucketName'),
-      region: activation?.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation?.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
-    };
-    const s3Prefix = activation?.getVariable('s3Prefix') || 'products/adhoc/';
-
-    // Validate required variables
-    const missing: string[] = [];
-    if (!retailerId) missing.push('retailerId');
-    if (!s3Config.bucket) missing.push('s3BucketName');
-    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
-    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
-    if (missing.length) {
-      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
-    }
-
-    try {
-      // Initialize JobTracker
-      const tracker = new JobTracker(openKv(':project:'), log);
-      const jobId = `products-adhoc-${Date.now()}`;
-
-      await tracker.createJob(jobId, {
-        type: 'extraction',
-        entity: 'products',
-        mode: 'adhoc',
-        retailerId,
-        fromDate,
-        toDate,
-        updateState,
-        startTime: executionStartTime,
-      });
-
-      log.info('Starting ad hoc products extraction', {
-        jobId,
-        retailerId,
-        fromDate: fromDate || 'not specified',
-        toDate: toDate || 'not specified',
-        maxRecords,
-        updateState,
-      });
-
-      // Initialize Fluent client + Orchestrator
-      // ✅ Optional: Validate connection immediately (fail-fast mode)
-      // Set activation variable 'validateConnectionOnStart' = 'true' to enable
-      // When enabled: Executes query { me { ref } } to verify authentication
-      // When disabled: Fast creation, validation happens on first API call (default)
-      const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
-      const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
-      
-      if (validateConnection) {
-        log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
-      }
-      
-      const orchestrator = new ExtractionOrchestrator(client, log);
-
-      // Execute extraction with auto-pagination
-      // ? Enhanced: Extract context for progress logging
-      const dateRangeInfo = {
-        start: fromDate || 'all',
-        end: toDate || 'now',
-        retailerId
-      };
-
-      // ? Enhanced: Start logging with context
-      log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
-       query: 'products',
-       pageSize,
-       maxRecords,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       retailerId: dateRangeInfo.retailerId,
-       jobId
-      });
-
-      const extraction = await orchestrator.extract({
-        query: PRODUCTS_QUERY,
-        resultPath: 'products.edges.node',
-        variables: {
-          retailerId,
-          ...(fromDate ? { updatedAfter: fromDate } : {}),
-        },
-        pageSize,
-        maxRecords,
-        validateItem: (item: any) => !!item.ref,
-      });
-
-      const nodes = extraction.data || [];
-
-      // ? Enhanced: Completion logging with summary
-      log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-       totalRecords: extraction.stats.totalRecords,
-       totalPages: extraction.stats.totalPages,
-       validRecords: extraction.stats.validRecords ?? nodes.length,
-       failedValidations: extraction.stats.failedValidations,
-       truncated: extraction.stats.truncated,
-       truncationReason: extraction.stats.truncationReason,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       jobId
-      });
-
-      if (nodes.length === 0) {
-        await tracker.markCompleted(jobId, {
-          recordsProcessed: 0,
-          message: 'No products found',
-        });
-        return {
-          success: true,
-          message: 'No products found',
-          jobId,
-          fromDate,
-          toDate,
-          stateUpdated: false,
-        };
-      }
-
-      // Transform with UniversalMapper (bulk mapping)
-      const mapper = new UniversalMapper(productsExportMapping);
-      const mappingResult = await mapper.map(nodes);
-
-      if (!mappingResult.success) {
-        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-        log.error('Mapping failed - terminating job', {
-          jobId,
-          errorCount: mappingErrors.length,
-          sampleErrors: mappingErrors.slice(0, 3),
-        });
-
-        await tracker.markFailed(
-          jobId,
-          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
-          {
-            errors: mappingErrors,
-            failedCount: mappingErrors.length,
-          }
-        );
-        return {
-          success: false,
-          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-          jobId,
-          errors: mappingErrors,
-        };
-      }
-
-      const transformedProducts = mappingResult.data || [];
-      const mappingErrors = mappingResult.errors || [];
-
-      if (mappingErrors.length > 0) {
-        log.warn('Some records failed transformation', {
-          jobId,
-          errorCount: mappingErrors.length,
-          sampleErrors: mappingErrors.slice(0, 3),
-        });
-      }
-
-      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-          jobId,
-          skippedFields: mappingResult.skippedFields,
-          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-        });
-      }
-
-      if (transformedProducts.length === 0) {
-        await tracker.markFailed(jobId, 'All records failed mapping', {
-          failedCount: mappingErrors.length,
-          errors: mappingErrors,
-        });
-        return {
-          success: false,
-          error: 'All records failed mapping',
-          jobId,
-          errors: mappingErrors,
-        };
-      }
-
-      // Compute newTimestamp from transformed records (WITHOUT buffer)
-      const newTimestamp = new Date(
-        transformedProducts.reduce(
-          (max, product) => {
-            const t = new Date(product.lastUpdated).getTime();
-            return t > max ? t : max;
-          },
-          fromDate ? new Date(fromDate).getTime() : 0
-        )
-      ).toISOString();
-
-      // Build JSON output
-      const jsonOutput = {
-        metadata: {
-          extractedAt: new Date().toISOString(),
-          productCount: transformedProducts.length,
-          mode: 'adhoc',
-          ...(fromDate && { fromDate }),
-          ...(toDate && { toDate }),
-          stateUpdated: updateState,
-        },
-        products: transformedProducts,
-      };
-
-      // Use JSONBuilder for consistent JSON generation
-      const jsonBuilder = new JSONBuilder({
-        prettyPrint,
-        indent: 2,
-      });
-      const jsonContent = jsonBuilder.build(jsonOutput);
-
-      // Upload to S3
-      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-      const fileName = `products-adhoc-${timestamp}.json`;
-      const s3Key = `${s3Prefix}${fileName}`;
-
-      const s3 = new S3DataSource(
-        {
-          type: 'S3_JSON',
-          connectionId: 's3-products-export',
-          name: 'S3 Products Export',
-          s3Config,
-        },
-        log
-      );
-
-      await s3.upload(s3Key, Buffer.from(jsonContent, 'utf8'), {
-        contentType: 'application/json',
-        metadata: {
-          productCount: String(transformedProducts.length),
-          extractedAt: new Date().toISOString(),
-          mode: 'adhoc',
-        },
-      });
-
-      // Optionally update state (adhoc): respects updateState flag
-      let stateUpdated = false;
-      if (updateState) {
-        const kv = new VersoriKVAdapter(openKv(':project:'));
-        const stateKey = ['extraction', 'products', 'lastRunTime'];
-        await kv.set(stateKey, {
-          timestamp: newTimestamp, // WITHOUT buffer
-          productCount: transformedProducts.length,
-          extractedAt: new Date().toISOString(),
-          fileName,
-          s3Key,
-          errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-        });
-        stateUpdated = true;
-      }
-
-      // Complete job tracking
-      const executionDurationMs = Date.now() - executionStartTime;
-      await tracker.markCompleted(jobId, {
-        recordsProcessed: transformedProducts.length,
-        recordsFailed: mappingErrors.length,
-        fileName,
-        s3Key,
-        newTimestamp,
-        stateUpdated,
-        executionDurationMs,
-        errors: mappingErrors,
-      });
-
-      log.info('=== EXECUTION END ===', {
-        timestamp: new Date().toISOString(),
-        durationMs: executionDurationMs,
-        success: true,
-      });
-
-      return {
-        success: true,
-        jobId,
-        productsExtracted: transformedProducts.length,
-        recordsFailed: mappingErrors.length,
-        fileName,
-        s3Key,
-        newTimestamp,
-        stateUpdated,
-        executionDurationMs,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      };
-    } catch (error: any) {
-      const executionDurationMs = Date.now() - executionStartTime;
-      log.error('Ad hoc extraction failed', error, { executionDurationMs });
-
-      log.info('=== EXECUTION END ===', {
-        timestamp: new Date().toISOString(),
-        durationMs: executionDurationMs,
-        success: false,
-      });
-
-      return {
-        success: false,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-        executionDurationMs,
-      };
-    }
-  })
-);
-```
-
-### Workflow 3: Job Status Checker
-**File:** `src/workflows/webhook/job-status-check.ts`
-
-```typescript
-/**
- * WORKFLOW 3/3: Job Status Checker
- *
- * Check status of extraction job
- *
- * Usage:
- * POST /job-status
- * {
- *   "jobId": "products-extraction-1234567890"
- * }
- */
-export const checkJobStatus = webhook('products-job-status', {
-  connection: 'products-job-status',
-}).then(
-  http('query-job-status', {}, async ctx => {
-    const { log, openKv, data } = ctx;
-
-    const requestData = typeof data === 'string' ? JSON.parse(data) : data;
-    const jobId = requestData?.jobId;
-
-    if (!jobId) {
-      return {
-        success: false,
-        error: 'Missing required field: jobId',
-      };
-    }
-
-    try {
-      const tracker = new JobTracker(openKv(':project:'), log);
-      const job = await tracker.getJob(jobId);
-
-      if (!job) {
-        return {
-          success: false,
-          error: `Job not found: ${jobId}`,
-        };
-      }
-
-      return {
-        success: true,
-        job,
-      };
-    } catch (error: any) {
-      log.error('Failed to get job status', error);
-      return {
-        success: false,
-        message: error instanceof Error ? error.message : String(error),
-
-        stack: error instanceof Error ? error.stack : undefined,
-
-        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-      };
-    }
-  })
-);
-```
-
-### Entry Point (Workflow Registration)
-**File:** `index.ts`
-
-```typescript
-/**
- * Entry point - Export all workflows for Versori platform
- *
- * This file exports all workflows to be registered with Versori.
- * Each workflow is defined in its own file for better organization.
- */
-
-// Scheduled workflows
-export { scheduledProductsExtraction } from './src/workflows/scheduled/daily-products-extraction';
-
-// Webhook workflows
-export { adHocProductsExtraction } from './src/workflows/webhook/adhoc-products-extraction';
-export { checkJobStatus } from './src/workflows/webhook/job-status-check';
-```
-
----
-
-## Important: Schema Verification
-
-**Before using this template**, verify the GraphQL query structure against your actual Fluent Commerce schema:
-
-```bash
-# Introspect your schema
-cd fc-connect-sdk
-npx fc-connect introspect-schema --url https://your-fluent-api.com/graphql
-
-# Check available fields on Product type
-npx fc-connect introspect-schema --url <url> --output schema.json
-# Then search for "Product" type in schema.json
-```
-
-The query structure shown above is an **example**. Adjust field names to match your actual schema.
-
-## Use Cases
-
-**1. Amazon Seller Central Integration:**
-
-- Daily product feed for listing updates
-- Include GTIN/UPC for catalog matching
-- Export pricing/inventory separately
-
-**2. eBay Marketplace Sync:**
-
-- Catalog updates for active listings
-- Include product descriptions and attributes
-- Handle variation products
-
-**3. Internal Product Master:**
-
-- Central product catalog for all systems
-- Include extended attributes
-- Version control for catalog changes
-
-**4. Ad Hoc Refresh:**
-
-- Manual catalog refresh for urgent updates
-- Integration testing with sample data
-- Troubleshooting specific product changes
-
-## Production Checklist
-
-- [ ] **Verify GraphQL query against actual schema** (use introspection)
-- [ ] Set appropriate extraction frequency (daily for catalog)
-- [ ] Configure correct product status filter (ACTIVE only?)
-- [ ] Test with real product data changes
-- [ ] Verify S3 bucket permissions
-- [ ] Set up monitoring/alerts for failed extractions
-- [ ] Document JSON schema for downstream consumers
-- [ ] Configure S3 lifecycle policy for old files
-- [ ] Test incremental extraction (only changed products)
-- [ ] Test ad hoc extraction workflow
-- [ ] Test job status monitoring
-- [ ] Verify overlap buffer prevents missed records
-
----
-
-### Pattern 7: State Management & Date Overrides
-
-**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
-
-**How it works**:
-
-VersoriKV stores the last successful extraction timestamp to enable incremental sync:
-
-```typescript
-interface ExtractionState {
-  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
-  recordCount: number; // Number of records extracted
-  extractedAt: string; // When extraction completed
-  fileName?: string; // Generated filename
-  s3Key?: string; // S3 upload path
-  overlapBufferSeconds?: number; // Buffer configuration
-}
-```
-
-**State Priority Chain** (highest to lowest):
-
-1. **`fromDate` override** (manual date in webhook payload) - Highest priority
-2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
-3. **`fallbackStartDate`** (activation variable) - First run fallback
-
-**Three Scenarios**:
-
-#### Scenario 1: Normal Scheduled Runs (Incremental)
-
-```typescript
-// Payload: {} (empty - no overrides)
-
-// Behavior:
-// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
-// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
-// 3. Extract records updated since buffered time
-// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
-// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
-// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
-```
-
-**Test**:
-
-```bash
-# Trigger scheduled run (no payload needed)
-# State advances automatically
-curl -X POST https://workspace.versori.run/products-extract-daily
-```
-
-#### Scenario 2: Ad-hoc Extraction WITH State Update
-
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
-
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
-// 3. Extract all records since 2025-01-01
-// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
-// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
-// 6. Next scheduled run starts from this new timestamp
-```
-
-**Use Case**: One-time catch-up extraction that advances the state pointer.
-
-**Test**:
-
-```bash
-curl -X POST https://workspace.versori.run/products-extract-webhook \
-  -H "x-api-key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "updateState": true
-  }'
-```
-
-#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
-
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
-
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z"
-// 3. Extract all records since 2025-01-01
-// 4. DO NOT update state
-// 5. Next scheduled run uses previous timestamp (unaffected)
-```
-
-**Use Case**: Historical backfill or testing without affecting incremental sync.
-
-**Test**:
-
-```bash
-curl -X POST https://workspace.versori.run/products-extract-webhook \
-  -H "x-api-key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-31T23:59:59Z",
-    "updateState": false
-  }'
-```
-
-**Why this matters**:
-
-- **Incremental sync** relies on state continuity
-- **Manual overrides** allow catch-up without breaking incremental flow
-- **Overlap buffer** prevents missed records at time boundaries
-- **State isolation** lets you test/backfill without affecting production sync
-
----
-
-### Pattern 8: Optional GraphQL Query Logging
-
-**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
-
-**When to use**:
-
-- ✅ Debugging pagination issues
-- ✅ Verifying query variables (dates, filters, limits)
-- ✅ Development and testing
-- ❌ Production (verbose logs, potential secrets in variables)
-
-**How to enable**:
-
-Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
-
-**Implementation**:
-
-```typescript
-// In your extraction workflow
-const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
-
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Query Debug', {
-    query: PRODUCTS_QUERY,
-    variables: {
-      catalogues,
-      dateRangeFilter,
-      first: pageSize,
-      after: null, // First page
-    },
-    pagination: {
-      pageSize,
-      maxRecords,
-      currentPage: 1,
-    },
-  });
-}
-
-const extractionResult = await orchestrator.extract({
-  query: PRODUCTS_QUERY,
-  resultPath: 'products.edges.node',
-  variables: {
-    catalogues,
-    dateRangeFilter,
-  },
-  pageSize,
-  maxRecords,
-});
-
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Response Debug', {
-    totalRecords: extractionResult.stats.totalRecords,
-    totalPages: extractionResult.stats.totalPages,
-    truncated: extractionResult.stats.truncated,
-    truncationReason: extractionResult.stats.truncationReason,
-    validRecords: extractionResult.stats.validRecords ?? extractionResult.data.length,
-    firstRecordId: extractionResult.data[0]?.id,
-    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
-  });
-}
-```
-
-**What gets logged**:
-
-```json
-{
-  "level": "info",
-  "message": "GraphQL Query Debug",
-  "query": "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)",
-  "variables": {
-    "catalogues": [{ "ref": "DEFAULT_CATALOGUE" }],
-    "dateRangeFilter": "2025-01-22T09:59:00Z",
-    "first": 200,
-    "after": null
-  },
-  "pagination": {
-    "pageSize": 200,
-    "maxRecords": 50000,
-    "currentPage": 1
-  }
-}
-```
-
-**Versori Environment Variables**:
-
-Add to activation settings:
-
-```json
-{
-  "DEBUG_GRAPHQL": "true"
-}
-```
-
-**Testing**:
-
-```bash
-# Enable debug logging
-curl -X POST https://workspace.versori.run/products-extract-daily
-
-# Check Versori logs for "GraphQL Query Debug" entries
-# Verify query structure and variables are correct
-```
-
-**Sample Debug Output**:
-
-```
-[INFO] GraphQL Query Debug
-  query: "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)"
-  variables: { catalogues: [{ ref: "DEFAULT_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
-  pagination: { pageSize: 200, maxRecords: 50000, currentPage: 1 }
-
-[INFO] Extraction complete
-  totalRecords: 1250
-  totalPages: 7
-  truncated: false
-  validRecords: 1250
-
-[INFO] GraphQL Response Debug
-  totalRecords: 1250
-  totalPages: 7
-  truncated: false
-  truncationReason: null
-  validRecords: 1250
-  firstRecordId: "product_abc"
-  lastRecordId: "product_xyz"
-```
-
-**Key Benefits**:
-
-- Quickly identify pagination configuration issues
-- Verify date filters are applied correctly
-- Debug "no records found" scenarios
-- Validate ExtractionOrchestrator variable injection
-
-**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
-
----
-
-## Troubleshooting Guide
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records
-- **Fix**: Reduce maxRecords, increase frequency
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch
-- **Fix**: Run schema validation, check field names
-
-**Issue**: "State not updating"
-
-- **Cause**: KV write failure or intentional retry
-- **Fix**: Check KV logs, verify state update code
-
-**Issue**: "First run exceeds limits"
-
-- **Cause**: No previous timestamp, fetches all
-- **Fix**: Set fallbackStartDate close to current, apply filters
-
-**Issue**: "Excessive duplicates"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved
-- **Fix**: Verify newTimestamp saved WITHOUT buffer
-
-**Issue**: "Job status not found"
-
-- **Cause**: JobTracker not initialized or wrong jobId
-- **Fix**: Verify JobTracker initialization and jobId format
-
----
-
-**Pattern**: Enterprise incremental extraction with overlap buffer for product catalog - Three-workflow approach
-**⚠️ Versori Sample**: Reference implementation - adapt for your production use case
-**Key Learning**: ALWAYS verify GraphQL schema before using queries
-**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
-**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
-**Schema**: Use schema introspection - don't guess field names
-**Three Workflows**: Scheduled incremental, Ad hoc HTTP trigger, Job status monitoring
-
----
-
-## See Also
-
-**Validation & Best Practices:**
-
-- [CLI Validation Workflow](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md) - Validate GraphQL queries and mappings before deployment
-- [Extraction Modes Guide](../extraction-modes-guide.md) - Comparison of incremental vs dateRange vs historical modes
-
-**SDK Documentation:**
-
-- [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Complete field mapping documentation
-- [SDK CLI Tools](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Command-line tools reference
-
-**Related Extraction Patterns:**
-
-- [Products to SFTP XML](./template-extraction-products-to-sftp-xml.md) - Same entity, different format/destination
-- [Virtual Positions to S3 JSON](./template-extraction-virtual-positions-to-s3-json.md) - Different entity, JSON format
-
----
-
-### Pattern 9: Backward Pagination (Optional - Advanced)
-
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-
-**When to Use**:
-
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
-- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
-
-**GraphQL Query Requirements**:
-
-Your query must support backward pagination by including `$last` and `$before`:
-
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ✅ REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ✅ REQUIRED for backward
-    }
-  }
-}
-```
-
-**Implementation**:
-
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    // ❌ Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ✅ Enable reverse pagination
-  maxRecords: 10000,
-});
-
-// Records are returned in reverse chronological order
-log.info('Extraction order verification', {
-  newest: result.data[0].createdAt,
-  oldest: result.data[result.data.length - 1].createdAt
-});
-```
-
-**Key Differences from Forward Pagination**:
-
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
-
-**Important Notes**:
-
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-
-**Example: Extract Latest 1000 Orders**
-
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-
-**When to Use Forward vs Backward**:
-
-```typescript
-// ✅ Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-
-// ✅ Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-
-**Pagination Variables Reference**:
-
-| Variable | Forward     | Backward    | Injected By  | Notes                    |
-| -------- | ----------- | ----------- | ------------ | ------------------------ |
-| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
-| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
-| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
-| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
-
-**Common Mistakes to Avoid**:
-
-```typescript
-// ❌ WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, // ❌ Orchestrator will override this
-    before: cursor, // ❌ Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-
-// ✅ CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ✅ Your business variables only
-  },
-  pageSize: 200, // ✅ Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-
-#### Optional: Reverse Pagination
-
-- Default: forward ($first/$after) + pageInfo.hasNextPage.
-- Reverse: query must use $last/$before; response must include pageInfo.hasPreviousPage; set direction='backward'.
-
-GraphQL:
-
-```graphql
-query GetProductsBackward(
-  $catalogues: [CatalogueKey]
-  $dateRangeFilter: DateRange
-  $last: Int!
-  $before: String
-) {
-  products(catalogues: $catalogues, updatedOn: $dateRangeFilter, last: $last, before: $before) {
-    edges {
-      cursor
-      node {
-        id
-        ref
-        updatedOn
-      }
-    }
-    pageInfo {
-      hasPreviousPage
-    }
-  }
-}
-```
-
-SDK:
-
-```typescript
-await orchestrator.extract({
-  query: PRODUCTS_BACKWARD_QUERY,
-  resultPath: 'products.edges.node',
-  variables: { catalogues, dateRangeFilter },
-  pageSize,
-  direction: 'backward',
-});
-```
-
----
-
-## Testing Checklist
-
-**Before production deployment:**
-
-### 1. Schema Validation
-
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/products.export.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/products.export.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-
-### 2. Extraction Testing
-
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-
-### 3. Mapping Testing
-
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-
-### 4. JSON Generation Testing
-
-- [ ] Verify JSON structure matches expected format
-- [ ] Test JSON validation against schema (if applicable)
-- [ ] Verify proper nesting and structure
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test special character escaping
-
-### 5. S3 Upload Testing
-
-- [ ] Test S3 connection and authentication
-- [ ] Verify file upload to correct bucket and path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify S3 object metadata
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify file permissions and ACLs
-
-### 6. State Management Testing
-
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-
-### 7. Job Tracking Testing
-
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-
-### 8. Error Handling Testing
-
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with S3 connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-
-### 9. Staging Environment Testing
-
-- [ ] Run full extraction in staging environment
-- [ ] Verify JSON file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-
-### 10. Integration Testing
-
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger → extract → transform → upload → verify file
-
----
-## Monitoring & Alerting
-
-### Success Response Example
-
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_PRD_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "products-2025-11-02T14-00-00-000Z.json",
-  "s3Path": "s3://bucket/products/products-2025-11-02T14-00-00-000Z.json",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-
-### Error Response Example
-
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_PRD_20251102_140500_xyz789",
-  "error": "S3 upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "s3_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-
-### Key Metrics to Track
-
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-
-  // File Generation
-  fileSizeMB: (jsonContent.length / (1024 * 1024)).toFixed(2),
-
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-
-log.info('Extraction metrics', metrics);
-```
-
-### Alert Thresholds
-
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-
-### Monitoring Dashboard Queries
-
-**Versori Platform Logs Query:**
-
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-
-# High error rates
-errorRate:>5
-
-# State management issues
-stateUpdated:false AND success:true
-```
-
-### Common Issues and Solutions
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-
-**Issue**: "S3 connection timeout"
-
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check S3 credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-
-**Issue**: "State not updating after successful extraction"
-
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-
-**Issue**: "First run exceeds record limits"
-
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-
-**Issue**: "Excessive duplicate records in output"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
-
----
-
-## Troubleshooting Quick Reference
-
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "JSON generation failed" | Format-specific error | Check JSON generation logic, validate output |
-
----
+---
+template_id: tpl-extract-products-graphql-to-s3-json
+canonical_filename: template-extraction-products-to-s3-json.md
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: s3-json
+entity: products
+format: json
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+---
+
+# Template: Extraction - Products GraphQL to S3 JSON
+
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+---
+
+## 📋 Implementation Prompt
+
+Copy/paste the standardized prompt from `docs/template-loading-matrix.md#prompts`.
+
+---
+
+## 💻 STEP 3: Implementation (Verified Imports)
+
+```ts
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+  VersoriKVAdapter,
+  ExtractionOrchestrator,
+  JobTracker,
+} from '@fluentcommerce/fc-connect-sdk';
+```
+
+These are the only SDK imports required for this template. Keep type-only imports out of code samples. Prefer the orchestrator-based flow when you need built-in pagination and stats.
+
+---
+
+# Versori Workflows: Product Catalog Extraction to S3 JSON
+
+**FC Connect SDK Use Case Guide**
+
+> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> Version: ^0.1.39
+
+Context: Three Versori workflows for extracting product catalog from Fluent Commerce via GraphQL query with **incremental timestamp tracking**, **ad hoc extraction**, and **job status monitoring**. Transforms with `UniversalMapper`, writes JSON files to S3 for marketplace integration (Amazon, eBay, etc).
+
+**Pattern**: EXTRACTION (Fluent → S3 JSON)
+**Complexity**: Medium | Runtime: Versori Platform
+**Format**: JSON with metadata wrapper
+
+---
+
+## ⚠️ IMPORTANT: Production-Ready Base Template
+
+> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
+>
+> This is a **production-ready base template** demonstrating FC Connect SDK best practices for product extraction workflows with JSON output to S3.
+>
+> **✅ INCLUDED FEATURES:**
+>
+> - ✅ Comprehensive error handling with retry logic
+> - ✅ S3 upload with proper error handling
+> - ✅ State management with overlap buffer (prevents missed records)
+> - ✅ Job tracking with lifecycle management
+> - ✅ Security (credential masking in logs)
+> - ✅ UTC time enforcement (prevents timezone bugs)
+> - ✅ Incremental extraction (safe, efficient, production-ready)
+> - ✅ Natural rate limiting via timestamps
+>
+> **📝 BEFORE DEPLOYING:**
+>
+> 1. Review and customize activation variables for your environment
+> 2. Test with sample data in your Versori workspace
+> 3. Adjust safety limits (pageSize, maxRecords) if needed
+> 4. Configure monitoring alerts for extraction failures
+> 5. Verify S3 bucket credentials and paths
+>
+> **This base template follows SDK best practices - tweak specific to your needs.**
+
+---
+
+## What You'll Build
+
+**Three Versori Workflows:**
+
+1. **Scheduled Extraction** - Daily/hourly incremental product updates
+2. **Ad Hoc Extraction** - On-demand HTTP webhook for manual triggers
+3. **Job Status Checker** - Monitor extraction job status via JobTracker
+
+**Features:**
+
+- **Incremental extraction** using `updatedOn > lastRunTime` filter
+- **State management** with VersoriKVAdapter to track last successful run
+- **Job tracking** with JobTracker for status monitoring
+- GraphQL query with auto-pagination
+- UniversalMapper transformation for marketplace schema
+- JSON file generation with product catalog
+- S3 upload to marketplace integration bucket
+- **Failure recovery** with timestamp tracking
+- **Pretty print option** for human-readable JSON
+
+## Business Use Case
+
+**Daily product catalog sync to marketplaces:**
+
+- Extract only changed products since last run
+- Export as JSON for marketplace API consumption
+- Run daily at midnight for overnight processing
+- Include SKU, title, description, pricing, attributes
+- Enable Amazon/eBay listing updates
+- Support ad hoc manual refresh on demand
+- Monitor job status for workflow integration
+
+## SDK Methods Used
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+  VersoriKVAdapter,
+  JobTracker,
+} from '@fluentcommerce/fc-connect-sdk';
+
+await createClient(ctx); // Versori-aware client
+await client.graphql({ query, variables, pagination }); // GraphQL with auto-pagination
+new VersoriKVAdapter(ctx.openKv(':project:')); // State management
+new JobTracker(ctx.openKv(':project:'), ctx.log); // Job tracking
+new UniversalMapper(exportMapping); // Field transformation
+const jsonBuilder = new JSONBuilder({ prettyPrint: true, indent: 2 });
+const jsonContent = jsonBuilder.build(dataObject); // JSON generation
+await s3.upload(key, Buffer.from(jsonContent, 'utf8'), options); // S3 upload
+```
+
+## Activation Variables
+
+```json
+{
+  "retailerId": "your-retailer-id",
+  "s3BucketName": "marketplace-catalog-exports",
+  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
+  "awsSecretAccessKey": "********",
+  "awsRegion": "us-east-1",
+  "s3Prefix": "products/daily/",
+  "pageSize": 200,
+  "maxRecords": 50000,
+  "fallbackStartDate": "2024-01-01T00:00:00Z",
+  "overlapBufferSeconds": "60",
+  "prettyPrint": "false",
+  "validateConnectionOnStart": "false"
+}
+```
+
+**New Variables (v2.1.0):**
+- `validateConnectionOnStart`: Optional fail-fast connection validation. When `"true"`, validates connection before extraction. Default: `"false"` (validation on first API call)
+
+**Configuration Notes:**
+- All variables are required except `validateConnectionOnStart`, `overlapBufferSeconds`, and `prettyPrint`
+- Credentials should be stored securely in Versori activation variables
+- `pageSize` and `maxRecords` can be adjusted based on data volume and performance requirements
+
+## Export Mapping Configuration
+
+Create file: `./config/products.export.json`
+
+```json
+{
+  "name": "products.export",
+  "version": "1.0.0",
+  "description": "Fluent Products → Marketplace JSON Export",
+  "fields": {
+    "sku": { "source": "ref", "required": true, "resolver": "sdk.trim" },
+    "title": { "source": "name", "required": true, "resolver": "sdk.trim" },
+    "description": { "source": "summary", "required": false, "resolver": "sdk.trim" },
+    "gtin": { "source": "gtin", "required": false, "resolver": "sdk.trim" },
+    "type": { "source": "type", "required": false, "resolver": "sdk.uppercase" },
+    "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
+    "createdOn": { "source": "createdOn", "required": false, "resolver": "sdk.toString" },
+    "lastUpdated": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
+  }
+}
+```
+
+**Note**: Actual GraphQL query structure should be verified against your Fluent schema using introspection. The fields above are examples - adjust based on your actual schema.
+
+## GraphQL Query
+
+**Note**: This query is an **example**. Verify against your Fluent Commerce schema using introspection.
+
+```graphql
+query GetProducts($retailerId: ID!, $updatedAfter: DateTime, $first: Int!, $after: String) {
+  products(
+    retailerId: $retailerId
+    updatedOn: { after: $updatedAfter }
+    first: $first
+    after: $after
+  ) {
+    edges {
+      node {
+        id
+        ref
+        name
+        summary
+        gtin
+        type
+        status
+        createdOn
+        updatedOn
+      }
+      cursor
+    }
+    pageInfo {
+      hasNextPage
+    }
+  }
+}
+```
+
+---
+
+## Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced) - Multi-step processes with state persistence
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Durable Workflows (Advanced Pattern)
+
+**⚠️ Note:** Durable workflows are an advanced pattern not yet used in this template. They're documented here for completeness.
+
+**Use Case:** Multi-step processes requiring state persistence, long-running operations (hours/days), human approval workflows, or complex retry/error handling across steps.
+
+**Example Pattern:**
+```typescript
+import { workflow } from '@versori/run';
+
+workflow('order-fulfillment', { connection: 'fluent_commerce' }, async (ctx, step) => {
+  // Step 1: Validate order
+  const order = await step.run('validate-order', async () => {
+    return await validateOrder(ctx);
+  });
+
+  // Step 2: Allocate inventory (with retry)
+  const allocation = await step.run('allocate-inventory', {
+    retry: { max: 3, delay: '1s' }
+  }, async () => {
+    return await allocateInventory(order);
+  });
+
+  // Step 3: Wait for approval (human-in-loop)
+  await step.sleep('wait-for-approval', '1h');
+
+  // Step 4: Create fulfillment
+  return await step.run('create-fulfillment', async () => {
+    return await createFulfillment(allocation);
+  });
+});
+```
+
+**When to use workflow():**
+- ✅ Multi-step processes requiring state persistence
+- ✅ Long-running operations (hours/days)
+- ✅ Human approval workflows
+- ✅ Complex retry/error handling across steps
+- ❌ Simple CRUD operations (use schedule/webhook + http)
+- ❌ Fire-and-forget patterns (use schedule + http)
+
+### Recommended Project Structure
+
+```
+products-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-products-extraction.ts  # Scheduled: Daily products extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-products-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts          # Webhook: Status query
+    │
+    ├── services/
+    │   └── products-extraction.service.ts   # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── products.export.json.json       # Mapping configuration
+```
+
+---
+
+```json
+{
+  "metadata": {
+    "extractedAt": "2025-01-22T00:00:00.000Z",
+    "productCount": 2,
+    "incrementalFrom": "2025-01-21T00:00:00.000Z",
+    "incrementalTo": "2025-01-22T00:00:00.000Z"
+  },
+  "products": [
+    {
+      "sku": "SKU-001",
+      "title": "Premium Widget",
+      "description": "High-quality widget for all purposes",
+      "gtin": "012345678901",
+      "type": "STANDARD",
+      "status": "ACTIVE",
+      "createdOn": "2025-01-15T10:00:00Z",
+      "lastUpdated": "2025-01-21T10:30:00Z"
+    },
+    {
+      "sku": "SKU-002",
+      "title": "Deluxe Gadget",
+      "description": "Advanced gadget with premium features",
+      "gtin": "012345678902",
+      "type": "STANDARD",
+      "status": "ACTIVE",
+      "createdOn": "2025-01-16T11:00:00Z",
+      "lastUpdated": "2025-01-21T14:15:00Z"
+    }
+  ]
+}
+```
+
+## Mapping & Resolvers Explained
+
+### SDK Resolvers Used
+
+| Field         | Resolver        | Why?                    | Example                       |
+| ------------- | --------------- | ----------------------- | ----------------------------- |
+| `sku`         | `sdk.trim`      | Clean SKU               | " SKU-001 " → "SKU-001"       |
+| `title`       | `sdk.trim`      | Clean names             | " Widget " → "Widget"         |
+| `description` | `sdk.trim`      | Remove extra whitespace | " desc " → "desc"             |
+| `gtin`        | `sdk.trim`      | Normalize GTIN/UPC      | " 012345 " → "012345"         |
+| `type`        | `sdk.uppercase` | Normalize type          | "standard" → "STANDARD"       |
+| `status`      | `sdk.uppercase` | Normalize status        | "active" → "ACTIVE"           |
+| `createdOn`   | `sdk.toString`  | Ensure ISO string       | Date → "2025-01-15T10:00:00Z" |
+| `lastUpdated` | `sdk.toString`  | Ensure ISO string       | Date → "2025-01-22T14:30:00Z" |
+
+### Transformation Flow
+
+```typescript
+// 1) GraphQL node (example)
+{
+  ref: " SKU-001 ",
+  name: " Widget ",
+  summary: "  High quality  ",
+  gtin: " 012345678901 ",
+  type: "standard",
+  status: "active",
+  createdOn: "2025-01-15T10:00:00.000Z",
+  updatedOn: "2025-01-21T10:30:00.000Z"
+}
+
+// 2) Map with UniversalMapper
+const mapper = new UniversalMapper(productsExportMapping);
+const result = await mapper.map(node);
+
+// 3) Output
+result.data = {
+  sku: "SKU-001",
+  title: "Widget",
+  description: "High quality",
+  gtin: "012345678901",
+  type: "STANDARD",
+  status: "ACTIVE",
+  createdOn: "2025-01-15T10:00:00.000Z",
+  lastUpdated: "2025-01-21T10:30:00.000Z"
+};
+```
+
+### Custom Resolvers for Product-Specific Logic
+
+While the mapping above uses built-in SDK resolvers, you can extend with custom business logic for marketplace integration:
+
+```typescript
+const customResolvers = {
+  /**
+   * Extract category hierarchy for marketplace navigation
+   */
+  'custom.formatCategoryPath': (product: any) => {
+    const categories = product.categories || [];
+    return categories
+      .map((c: any) => c.name?.trim())
+      .filter(Boolean)
+      .join(' > ');
+    // Example: "Electronics > Computers > Laptops"
+  },
+
+  /**
+   * Generate marketplace-compatible SKU with prefix
+   */
+  'custom.generateMarketplaceSKU': (ref: string, retailerId: string) => {
+    return `${retailerId}-${ref.trim()}`.toUpperCase();
+    // Example: "RETAILER123-SKU-001"
+  },
+
+  /**
+   * Format price for marketplace API (convert cents to dollars)
+   */
+  'custom.formatPrice': (priceInCents: number) => {
+    const dollars = (priceInCents || 0) / 100;
+    return dollars.toFixed(2);
+    // Example: 2499 → "24.99"
+  },
+
+  /**
+   * Extract primary image URL from attributes
+   */
+  'custom.getPrimaryImage': (product: any) => {
+    const images = product.attributes?.images || [];
+    return images.length > 0 ? images[0]?.url : null;
+  },
+
+  /**
+   * Generate marketplace title with brand and product name
+   */
+  'custom.generateMarketplaceTitle': (product: any) => {
+    const brand = product.attributes?.brand?.trim() || '';
+    const name = product.name?.trim() || '';
+    const maxLength = 200; // Amazon/eBay limit
+
+    const title = brand ? `${brand} - ${name}` : name;
+    return title.length > maxLength ? title.substring(0, maxLength - 3) + '...' : title;
+  },
+
+  /**
+   * Extract product attributes for marketplace listing
+   */
+  'custom.extractAttributes': (product: any) => {
+    const attrs = product.attributes || {};
+    return {
+      brand: attrs.brand?.trim() || 'Unbranded',
+      color: attrs.color?.trim() || null,
+      size: attrs.size?.trim() || null,
+      weight: attrs.weight ? `${attrs.weight} ${attrs.weightUnit || 'lb'}` : null,
+      dimensions: attrs.dimensions || null,
+      material: attrs.material?.trim() || null,
+    };
+  },
+
+  /**
+   * Determine product eligibility for marketplaces
+   */
+  'custom.checkMarketplaceEligibility': (product: any) => {
+    const status = (product.status || '').toUpperCase();
+    const hasGTIN = !!product.gtin;
+    const hasImages = (product.attributes?.images || []).length > 0;
+    const hasPrice = (product.prices || []).length > 0;
+
+    return {
+      isActive: status === 'ACTIVE',
+      hasRequiredFields: hasGTIN && hasImages && hasPrice,
+      isEligibleForAmazon: status === 'ACTIVE' && hasGTIN && hasImages && hasPrice,
+      isEligibleForEbay: status === 'ACTIVE' && hasImages && hasPrice,
+      missingFields: [!hasGTIN && 'GTIN', !hasImages && 'Images', !hasPrice && 'Price'].filter(
+        Boolean
+      ),
+    };
+  },
+
+  /**
+   * Format for marketplace API submission
+   */
+  'custom.formatForMarketplace': (product: any) => {
+    return {
+      sku: product.ref?.trim(),
+      title: product.name?.trim(),
+      description: product.summary?.trim() || product.name?.trim(),
+      category: product.categories?.[0]?.name || 'Uncategorized',
+      brand: product.attributes?.brand?.trim() || 'Generic',
+      gtin: product.gtin?.trim() || null,
+      price: (product.prices?.[0]?.value || 0) / 100, // cents to dollars
+      currency: product.prices?.[0]?.currency || 'USD',
+      images: (product.attributes?.images || []).map((img: any) => img.url),
+      status: product.status?.toUpperCase(),
+      lastUpdated: product.updatedOn,
+    };
+  },
+};
+
+// Use custom resolvers with UniversalMapper
+const mapper = new UniversalMapper(productsExportMapping, {
+  customResolvers,
+});
+```
+
+### Available SDK Resolvers
+
+The SDK provides these built-in resolvers (no custom code needed):
+
+**String Transformations:**
+
+- `sdk.trim` - Remove leading/trailing whitespace
+- `sdk.uppercase` - Convert to uppercase
+- `sdk.lowercase` - Convert to lowercase
+- `sdk.toString` - Convert to string
+
+**Number Parsing:**
+
+- `sdk.parseInt` - Parse as integer
+- `sdk.parseFloat` - Parse as decimal (ideal for prices)
+- `sdk.number` - Parse as number (auto-detect int/float)
+
+**Date Formatting:**
+
+- `sdk.formatDate` - ISO 8601 date only (YYYY-MM-DD)
+- `sdk.formatDateShort` - Short date format
+- `sdk.parseDate` - Parse various date formats
+
+**Type Conversions:**
+
+- `sdk.boolean` - Convert to boolean
+- `sdk.parseJson` - Parse JSON strings (useful for attributes)
+- `sdk.toJson` - Convert to JSON string
+
+**Utilities:**
+
+- `sdk.identity` - Return value unchanged
+- `sdk.coalesce` - Return first non-null value
+
+See [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) for complete resolver documentation.
+
+## Production Safety & Guardrails
+
+### Overview
+
+Even with **incremental-only** extraction, product catalogs need safeguards to prevent runtime failures:
+
+- **Memory limits**: Product records can be large (descriptions, attributes, images)
+- **S3 upload limits**: Single files > 5GB require multipart upload
+- **Processing time**: Large catalogs can timeout
+- **JSON parsing**: Massive JSON files stress memory and downstream parsers
+
+### Hard Limits
+
+```typescript
+const SAFETY_LIMITS = {
+  MAX_RECORDS_PER_RUN: 100000, // 100k products per run
+  MAX_RECORDS_PER_FILE: 25000, // 25k per JSON file
+  MAX_FILE_SIZE_MB: 250, // 250MB per file
+  MAX_JSON_SIZE_MB: 500, // Total extraction size
+  CHUNK_SIZE: 10000, // Process in chunks
+  ESTIMATED_BYTES_PER_PRODUCT: 2048, // Conservative estimate (2KB per product)
+};
+```
+
+**Why these limits?**
+
+- Products have more fields than orders/fulfillments (descriptions, attributes, variants)
+- JSON is less compact than CSV
+- Marketplace APIs often have file size limits (Amazon: 100MB compressed)
+
+### Runtime Validation Function
+
+```typescript
+/**
+ * Validate extraction safety limits before processing
+ */
+function validateExtractionLimits(recordCount: number, estimatedSizeMB: number) {
+  const MAX_RECORDS_PER_RUN = 100000;
+  const MAX_JSON_SIZE_MB = 500;
+
+  if (recordCount > MAX_RECORDS_PER_RUN) {
+    return {
+      valid: false,
+      error: `Extraction limit exceeded: ${recordCount} records (max: ${MAX_RECORDS_PER_RUN})`,
+      recommendation: `Catalog too large for single extraction. Consider:
+        1. Increase extraction frequency to reduce batch size
+        2. Filter by product status (ACTIVE only)
+        3. Split by product type or category
+        4. Use multiple extractions with different filters`,
+      recordCount,
+      maxAllowed: MAX_RECORDS_PER_RUN,
+    };
+  }
+
+  if (estimatedSizeMB > MAX_JSON_SIZE_MB) {
+    return {
+      valid: false,
+      error: `JSON size limit exceeded: ${estimatedSizeMB}MB (max: ${MAX_JSON_SIZE_MB}MB)`,
+      recommendation: 'File splitting required. Consider filtering or splitting by category.',
+      estimatedSizeMB,
+      maxAllowed: MAX_JSON_SIZE_MB,
+    };
+  }
+
+  return { valid: true };
+}
+```
+
+## Complete Workflows Implementation
+
+The code examples below demonstrate what goes in each file according to the modular structure shown in the "Recommended Project Structure" section above.
+
+### Workflow 1: Scheduled Incremental Extraction
+**File:** `src/workflows/scheduled/daily-products-extraction.ts`
+
+```typescript
+import { schedule, http } from '@versori/run';
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+  VersoriKVAdapter,
+  JobTracker,
+} from '@fluentcommerce/fc-connect-sdk';
+import productsExportMapping from './config/products.export.json' with { type: 'json' };
+
+const PRODUCTS_QUERY = `
+  query GetProducts(
+    $retailerId: ID!
+    $updatedAfter: DateTime
+    $first: Int!
+    $after: String
+  ) {
+    products(
+      retailerId: $retailerId
+      updatedOn: { after: $updatedAfter }
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          name
+          summary
+          gtin
+          type
+          status
+          createdOn
+          updatedOn
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+
+/**
+ * WORKFLOW 1/3: Scheduled Daily Product Extraction (Incremental)
+ *
+ * Runs daily at midnight to extract changed products
+ */
+export const scheduledProductsExtraction = schedule('scheduled-products-extraction', '0 0 * * *').then(
+  http('extract-products', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, activation } = ctx;
+    const executionStartTime = Date.now();
+
+    log.info('=== EXECUTION START ===', { timestamp: new Date().toISOString() });
+
+    // STEP 1/8: Parse activation variables
+    const retailerId = activation?.getVariable('retailerId');
+    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
+    const maxRecords = parseInt(activation?.getVariable('maxRecords') || '50000', 10);
+    const fallbackStartDate =
+      activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
+    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
+
+    const s3Config = {
+      bucket: activation?.getVariable('s3BucketName'),
+      region: activation?.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation?.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
+    };
+    const s3Prefix = activation?.getVariable('s3Prefix') || 'products/daily/';
+
+    // Validate required variables
+    const missing: string[] = [];
+    if (!retailerId) missing.push('retailerId');
+    if (!s3Config.bucket) missing.push('s3BucketName');
+    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
+    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
+    if (missing.length) {
+      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
+    }
+
+    try {
+      // STEP 2/8: Initialize JobTracker and start tracking
+      const tracker = new JobTracker(openKv(':project:'), log);
+      const jobId = `products-extraction-${Date.now()}`;
+
+      await tracker.createJob(jobId, {
+        type: 'extraction',
+        entity: 'products',
+        mode: 'scheduled',
+        retailerId,
+        startTime: executionStartTime,
+      });
+
+      // STEP 3/8: Load last successful extraction timestamp with overlap buffer
+      const kv = new VersoriKVAdapter(openKv(':project:'));
+      const stateKey = ['extraction', 'products', 'lastRunTime'];
+      const lastRunState = await kv.get(stateKey);
+      const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
+
+      // Overlap buffer configuration (default: 60 seconds)
+      const overlapBufferSeconds = parseInt(
+        activation?.getVariable('overlapBufferSeconds') || '60',
+        10
+      );
+      const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
+
+      // Apply overlap buffer for query (safety window)
+      const bufferedLastRunTime = new Date(
+        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
+      ).toISOString();
+
+      const toDate = undefined; // No manual override for scheduled extraction
+
+      log.info('🔍 Starting incremental products extraction with overlap buffer', {
+        jobId,
+        rawLastRunTime,
+        bufferedLastRunTime,
+        effectiveEndTime: toDate || new Date().toISOString(),
+        overlapBufferSeconds,
+        retailerId,
+        maxRecords,
+      });
+
+      // STEP 4/8: Initialize Fluent client + ExtractionOrchestrator
+      // ✅ Optional: Validate connection immediately (fail-fast mode)
+      // Set activation variable 'validateConnectionOnStart' = 'true' to enable
+      // When enabled: Executes query { me { ref } } to verify authentication
+      // When disabled: Fast creation, validation happens on first API call (default)
+      const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
+      const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
+      
+      if (validateConnection) {
+        log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
+      }
+      
+      const orchestrator = new ExtractionOrchestrator(client, log);
+
+      // STEP 5/8: Execute extraction with auto-pagination (WITH overlap buffer)
+      const effectiveEndTime = toDate || new Date().toISOString();
+      
+      // ? Enhanced: Extract context for progress logging
+      const dateRangeInfo = {
+        start: bufferedLastRunTime,
+        end: effectiveEndTime,
+        retailerId
+      };
+
+      // ? Enhanced: Start logging with context
+      log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
+       query: 'products',
+       pageSize,
+       maxRecords,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       retailerId: dateRangeInfo.retailerId,
+       jobId
+      });
+
+      const extractionResult = await orchestrator.extract({
+        query: PRODUCTS_QUERY,
+        resultPath: 'products.edges.node',
+        variables: {
+          retailerId,
+          dateRangeFilter: {
+            after: bufferedLastRunTime,
+            before: effectiveEndTime, // End of extraction window
+          },
+        },
+        pageSize,
+        maxRecords,
+        validateItem: item => !!item.ref,
+      });
+
+      const rawRecords = extractionResult.data;
+
+      log.info('Products extraction completed', {
+        jobId,
+        totalRecords: extractionResult.stats.totalRecords,
+        totalPages: extractionResult.stats.totalPages,
+        validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+        errors: extractionResult.errors ? extractionResult.errors.length : 0,
+      });
+
+      // ? Enhanced: Completion logging with summary
+      log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+       totalRecords: extractionResult.stats.totalRecords,
+       totalPages: extractionResult.stats.totalPages,
+       validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+       failedValidations: extractionResult.stats.failedValidations,
+       truncated: extractionResult.stats.truncated,
+       truncationReason: extractionResult.stats.truncationReason,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       jobId
+      });
+
+      if (extractionResult.errors && extractionResult.errors.length > 0) {
+        log.warn('Non-fatal extraction errors encountered', {
+          jobId,
+          errorCount: extractionResult.errors.length,
+          sampleErrors: extractionResult.errors.slice(0, 3),
+        });
+      }
+
+      if (rawRecords.length === 0) {
+        log.info('No new products to extract');
+        await kv.set(stateKey, {
+          timestamp: new Date().toISOString(),
+          productCount: 0,
+          extractedAt: new Date().toISOString(),
+        });
+        await tracker.markCompleted(jobId, {
+          recordsProcessed: 0,
+          message: 'No new products to extract',
+        });
+        return {
+          success: true,
+          message: 'No new products to extract',
+          jobId,
+          lastRunTime: rawLastRunTime,
+        };
+      }
+
+      log.info('Products retrieved', { jobId, count: rawRecords.length });
+
+      // STEP 6/8: Validate extraction limits
+      const MAX_RECORDS_PER_RUN = 100000;
+      const ESTIMATED_BYTES_PER_PRODUCT = 2048;
+      const estimatedSizeMB = (rawRecords.length * ESTIMATED_BYTES_PER_PRODUCT) / (1024 * 1024);
+      const MAX_JSON_SIZE_MB = 500;
+
+      if (rawRecords.length > MAX_RECORDS_PER_RUN) {
+        await tracker.markFailed(jobId, {
+          error: 'Extraction limit exceeded',
+          recordCount: rawRecords.length,
+          maxAllowed: MAX_RECORDS_PER_RUN,
+        });
+        log.error('Extraction limit exceeded', {
+          recordCount: rawRecords.length,
+          maxAllowed: MAX_RECORDS_PER_RUN,
+        });
+        return {
+          success: false,
+          error: `Extraction limit exceeded: ${rawRecords.length} records (max: ${MAX_RECORDS_PER_RUN})`,
+          recommendation: `Catalog too large for single extraction. Consider:
+            1. Increase extraction frequency to reduce batch size
+            2. Filter by product status (ACTIVE only)
+            3. Split by product type or category
+            4. Use multiple extractions with different filters`,
+          recordCount: rawRecords.length,
+          maxAllowed: MAX_RECORDS_PER_RUN,
+        };
+      }
+
+      if (estimatedSizeMB > MAX_JSON_SIZE_MB) {
+        log.warn('JSON size approaching limit', {
+          estimatedSizeMB: estimatedSizeMB.toFixed(2),
+          maxAllowed: MAX_JSON_SIZE_MB,
+          recommendation: 'Consider file splitting or category-based filtering',
+        });
+      }
+
+      await tracker.updateJobProgress(jobId, {
+        stage: 'validation_complete',
+        recordCount: rawRecords.length,
+        estimatedSizeMB: estimatedSizeMB.toFixed(2),
+      });
+
+      // STEP 7/8: Transform with UniversalMapper
+      const mapper = new UniversalMapper(productsExportMapping);
+      const mappingResult = await mapper.map(rawRecords);
+
+      if (!mappingResult.success) {
+        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+        await tracker.markFailed(
+          jobId,
+          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
+          {
+            failedCount: mappingErrors.length,
+            errors: mappingErrors,
+          }
+        );
+        return {
+          success: false,
+          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+          errors: mappingErrors,
+        };
+      }
+
+      const transformedProducts = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+      const mappingErrors = mappingResult.errors || [];
+
+      if (mappingErrors.length > 0) {
+        log.warn('Some products failed transformation', {
+          jobId,
+          errorCount: mappingErrors.length,
+          sampleErrors: mappingErrors.slice(0, 3),
+        });
+      }
+
+      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+          jobId,
+          skippedFields: mappingResult.skippedFields,
+          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+        });
+      }
+
+      if (transformedProducts.length === 0) {
+        await tracker.markFailed(jobId, 'All records failed mapping', {
+          failedCount: mappingErrors.length,
+          errors: mappingErrors,
+        });
+        return {
+          success: false,
+          error: 'All records failed mapping',
+          errors: mappingErrors,
+        };
+      }
+
+      log.info('Records transformed', {
+        jobId,
+        successful: transformedProducts.length,
+        skippedRecords: rawRecords.length - transformedProducts.length,
+      });
+
+      await tracker.updateJobProgress(jobId, {
+        stage: 'transformation_complete',
+        transformedCount: transformedProducts.length,
+        failedCount: mappingErrors.length,
+      });
+
+      // Calculate max updatedOn for next run (WITHOUT buffer)
+      const maxUpdatedOn = transformedProducts.reduce((max, product) => {
+        const productTime = new Date(product.lastUpdated).getTime();
+        return productTime > max ? productTime : max;
+      }, new Date(rawLastRunTime).getTime());
+
+      const newTimestamp = new Date(maxUpdatedOn).toISOString();
+
+      // Build JSON with metadata
+      const jsonOutput = {
+        metadata: {
+          extractedAt: new Date().toISOString(),
+          productCount: transformedProducts.length,
+          incrementalFrom: rawLastRunTime,
+          incrementalTo: newTimestamp,
+        },
+        products: transformedProducts,
+      };
+
+      // Use JSONBuilder for consistent JSON generation
+      const jsonBuilder = new JSONBuilder({
+        prettyPrint,
+        indent: 2,
+      });
+      const jsonContent = jsonBuilder.build(jsonOutput);
+
+      // Generate timestamped filename
+      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+      const fileName = `products-${timestamp}.json`;
+      const s3Key = `${s3Prefix}${fileName}`;
+
+      log.info('Generated JSON file', {
+        jobId,
+        fileName,
+        size: jsonContent.length,
+        productCount: transformedProducts.length,
+      });
+
+      // STEP 8/8: Upload to S3
+      const s3 = new S3DataSource(
+        {
+          type: 'S3_JSON',
+          connectionId: 's3-products-export',
+          name: 'S3 Products Export',
+          s3Config,
+        },
+        log
+      );
+
+      await s3.upload(s3Key, Buffer.from(jsonContent, 'utf8'), {
+        contentType: 'application/json',
+        metadata: {
+          productCount: String(transformedProducts.length),
+          extractedAt: new Date().toISOString(),
+          incrementalFrom: rawLastRunTime,
+          incrementalTo: newTimestamp,
+        },
+      });
+
+      log.info('JSON file uploaded to S3', { jobId, s3Key });
+
+      // Update state with new timestamp (WITHOUT buffer)
+      await kv.set(stateKey, {
+        timestamp: newTimestamp, // ← NO buffer applied
+        productCount: transformedProducts.length,
+        extractedAt: new Date().toISOString(),
+        overlapBufferSeconds,
+        fileName,
+        s3Key,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      });
+
+      log.info('State updated with new timestamp (without buffer)', {
+        jobId,
+        newTimestamp,
+        overlapBufferSeconds,
+      });
+
+      // Complete job tracking
+      const executionDurationMs = Date.now() - executionStartTime;
+      await tracker.markCompleted(jobId, {
+        recordsProcessed: transformedProducts.length,
+        recordsFailed: mappingErrors.length,
+        fileName,
+        s3Key,
+        newTimestamp,
+        executionDurationMs,
+        errors: mappingErrors,
+      });
+
+      log.info('=== EXECUTION END ===', {
+        timestamp: new Date().toISOString(),
+        durationMs: executionDurationMs,
+        success: true,
+      });
+
+      return {
+        success: true,
+        jobId,
+        productsExtracted: transformedProducts.length,
+        recordsFailed: mappingErrors.length,
+        fileName,
+        s3Key,
+        lastRunTime: rawLastRunTime,
+        newTimestamp,
+        executionDurationMs,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      };
+    } catch (error: any) {
+      const executionDurationMs = Date.now() - executionStartTime;
+      log.error('Extraction failed', error, {
+        message: error?.message,
+        executionDurationMs,
+      });
+
+      log.info('=== EXECUTION END ===', {
+        timestamp: new Date().toISOString(),
+        durationMs: executionDurationMs,
+        success: false,
+      });
+
+      return {
+        success: false,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+        executionDurationMs,
+      };
+    }
+  })
+);
+```
+
+### Workflow 2: Ad Hoc HTTP Extraction
+**File:** `src/workflows/webhook/adhoc-products-extraction.ts`
+
+```typescript
+/**
+ * WORKFLOW 2/3: Ad Hoc Product Extraction (HTTP Webhook)
+ *
+ * Manual trigger for on-demand product catalog extraction
+ *
+ * Usage (payload examples):
+ * 1) Incremental (use stored state):
+ *    {}
+ *
+ * 2) Date range override (preferred keys):
+ *    {
+ *      "fromDate": "2025-01-01T00:00:00Z",
+ *      "toDate": "2025-01-22T00:00:00Z",
+ *      "updateState": false
+ *    }
+ *
+ * 3) Backward-compatible keys (startDate/endDate) are also accepted and mapped:
+ *    {
+ *      "startDate": "2025-01-01T00:00:00Z",
+ *      "endDate": "2025-01-22T00:00:00Z"
+ *    }
+ */
+export const adHocProductsExtraction = webhook('extract-products-adhoc', {
+  connection: 'products-adhoc',
+}).then(
+  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, activation, data } = ctx;
+    const executionStartTime = Date.now();
+
+    log.info('=== EXECUTION START ===', { timestamp: new Date().toISOString() });
+
+    // Parse request body and support both new (fromDate/toDate) and alternative (startDate/endDate) keys
+    const requestData = typeof data === 'string' ? JSON.parse(data) : data;
+    const fromDate = requestData?.fromDate || requestData?.startDate;
+    const toDate = requestData?.toDate || requestData?.endDate;
+    const updateState = requestData?.updateState === true; // default: false
+    const maxRecordsOverride = requestData?.maxRecords;
+
+    const retailerId = activation?.getVariable('retailerId');
+    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
+    const maxRecords =
+      maxRecordsOverride || parseInt(activation?.getVariable('maxRecords') || '50000', 10);
+    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
+
+    const s3Config = {
+      bucket: activation?.getVariable('s3BucketName'),
+      region: activation?.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation?.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
+    };
+    const s3Prefix = activation?.getVariable('s3Prefix') || 'products/adhoc/';
+
+    // Validate required variables
+    const missing: string[] = [];
+    if (!retailerId) missing.push('retailerId');
+    if (!s3Config.bucket) missing.push('s3BucketName');
+    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
+    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
+    if (missing.length) {
+      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
+    }
+
+    try {
+      // Initialize JobTracker
+      const tracker = new JobTracker(openKv(':project:'), log);
+      const jobId = `products-adhoc-${Date.now()}`;
+
+      await tracker.createJob(jobId, {
+        type: 'extraction',
+        entity: 'products',
+        mode: 'adhoc',
+        retailerId,
+        fromDate,
+        toDate,
+        updateState,
+        startTime: executionStartTime,
+      });
+
+      log.info('Starting ad hoc products extraction', {
+        jobId,
+        retailerId,
+        fromDate: fromDate || 'not specified',
+        toDate: toDate || 'not specified',
+        maxRecords,
+        updateState,
+      });
+
+      // Initialize Fluent client + Orchestrator
+      // ✅ Optional: Validate connection immediately (fail-fast mode)
+      // Set activation variable 'validateConnectionOnStart' = 'true' to enable
+      // When enabled: Executes query { me { ref } } to verify authentication
+      // When disabled: Fast creation, validation happens on first API call (default)
+      const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
+      const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
+      
+      if (validateConnection) {
+        log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
+      }
+      
+      const orchestrator = new ExtractionOrchestrator(client, log);
+
+      // Execute extraction with auto-pagination
+      // ? Enhanced: Extract context for progress logging
+      const dateRangeInfo = {
+        start: fromDate || 'all',
+        end: toDate || 'now',
+        retailerId
+      };
+
+      // ? Enhanced: Start logging with context
+      log.info(`📊 [ExtractionOrchestrator] Starting extraction`, {
+       query: 'products',
+       pageSize,
+       maxRecords,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       retailerId: dateRangeInfo.retailerId,
+       jobId
+      });
+
+      const extraction = await orchestrator.extract({
+        query: PRODUCTS_QUERY,
+        resultPath: 'products.edges.node',
+        variables: {
+          retailerId,
+          ...(fromDate ? { updatedAfter: fromDate } : {}),
+        },
+        pageSize,
+        maxRecords,
+        validateItem: (item: any) => !!item.ref,
+      });
+
+      const nodes = extraction.data || [];
+
+      // ? Enhanced: Completion logging with summary
+      log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+       totalRecords: extraction.stats.totalRecords,
+       totalPages: extraction.stats.totalPages,
+       validRecords: extraction.stats.validRecords ?? nodes.length,
+       failedValidations: extraction.stats.failedValidations,
+       truncated: extraction.stats.truncated,
+       truncationReason: extraction.stats.truncationReason,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       jobId
+      });
+
+      if (nodes.length === 0) {
+        await tracker.markCompleted(jobId, {
+          recordsProcessed: 0,
+          message: 'No products found',
+        });
+        return {
+          success: true,
+          message: 'No products found',
+          jobId,
+          fromDate,
+          toDate,
+          stateUpdated: false,
+        };
+      }
+
+      // Transform with UniversalMapper (bulk mapping)
+      const mapper = new UniversalMapper(productsExportMapping);
+      const mappingResult = await mapper.map(nodes);
+
+      if (!mappingResult.success) {
+        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+        log.error('Mapping failed - terminating job', {
+          jobId,
+          errorCount: mappingErrors.length,
+          sampleErrors: mappingErrors.slice(0, 3),
+        });
+
+        await tracker.markFailed(
+          jobId,
+          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
+          {
+            errors: mappingErrors,
+            failedCount: mappingErrors.length,
+          }
+        );
+        return {
+          success: false,
+          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+          jobId,
+          errors: mappingErrors,
+        };
+      }
+
+      const transformedProducts = mappingResult.data || [];
+      const mappingErrors = mappingResult.errors || [];
+
+      if (mappingErrors.length > 0) {
+        log.warn('Some records failed transformation', {
+          jobId,
+          errorCount: mappingErrors.length,
+          sampleErrors: mappingErrors.slice(0, 3),
+        });
+      }
+
+      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+          jobId,
+          skippedFields: mappingResult.skippedFields,
+          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+        });
+      }
+
+      if (transformedProducts.length === 0) {
+        await tracker.markFailed(jobId, 'All records failed mapping', {
+          failedCount: mappingErrors.length,
+          errors: mappingErrors,
+        });
+        return {
+          success: false,
+          error: 'All records failed mapping',
+          jobId,
+          errors: mappingErrors,
+        };
+      }
+
+      // Compute newTimestamp from transformed records (WITHOUT buffer)
+      const newTimestamp = new Date(
+        transformedProducts.reduce(
+          (max, product) => {
+            const t = new Date(product.lastUpdated).getTime();
+            return t > max ? t : max;
+          },
+          fromDate ? new Date(fromDate).getTime() : 0
+        )
+      ).toISOString();
+
+      // Build JSON output
+      const jsonOutput = {
+        metadata: {
+          extractedAt: new Date().toISOString(),
+          productCount: transformedProducts.length,
+          mode: 'adhoc',
+          ...(fromDate && { fromDate }),
+          ...(toDate && { toDate }),
+          stateUpdated: updateState,
+        },
+        products: transformedProducts,
+      };
+
+      // Use JSONBuilder for consistent JSON generation
+      const jsonBuilder = new JSONBuilder({
+        prettyPrint,
+        indent: 2,
+      });
+      const jsonContent = jsonBuilder.build(jsonOutput);
+
+      // Upload to S3
+      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+      const fileName = `products-adhoc-${timestamp}.json`;
+      const s3Key = `${s3Prefix}${fileName}`;
+
+      const s3 = new S3DataSource(
+        {
+          type: 'S3_JSON',
+          connectionId: 's3-products-export',
+          name: 'S3 Products Export',
+          s3Config,
+        },
+        log
+      );
+
+      await s3.upload(s3Key, Buffer.from(jsonContent, 'utf8'), {
+        contentType: 'application/json',
+        metadata: {
+          productCount: String(transformedProducts.length),
+          extractedAt: new Date().toISOString(),
+          mode: 'adhoc',
+        },
+      });
+
+      // Optionally update state (adhoc): respects updateState flag
+      let stateUpdated = false;
+      if (updateState) {
+        const kv = new VersoriKVAdapter(openKv(':project:'));
+        const stateKey = ['extraction', 'products', 'lastRunTime'];
+        await kv.set(stateKey, {
+          timestamp: newTimestamp, // WITHOUT buffer
+          productCount: transformedProducts.length,
+          extractedAt: new Date().toISOString(),
+          fileName,
+          s3Key,
+          errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+        });
+        stateUpdated = true;
+      }
+
+      // Complete job tracking
+      const executionDurationMs = Date.now() - executionStartTime;
+      await tracker.markCompleted(jobId, {
+        recordsProcessed: transformedProducts.length,
+        recordsFailed: mappingErrors.length,
+        fileName,
+        s3Key,
+        newTimestamp,
+        stateUpdated,
+        executionDurationMs,
+        errors: mappingErrors,
+      });
+
+      log.info('=== EXECUTION END ===', {
+        timestamp: new Date().toISOString(),
+        durationMs: executionDurationMs,
+        success: true,
+      });
+
+      return {
+        success: true,
+        jobId,
+        productsExtracted: transformedProducts.length,
+        recordsFailed: mappingErrors.length,
+        fileName,
+        s3Key,
+        newTimestamp,
+        stateUpdated,
+        executionDurationMs,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      };
+    } catch (error: any) {
+      const executionDurationMs = Date.now() - executionStartTime;
+      log.error('Ad hoc extraction failed', error, { executionDurationMs });
+
+      log.info('=== EXECUTION END ===', {
+        timestamp: new Date().toISOString(),
+        durationMs: executionDurationMs,
+        success: false,
+      });
+
+      return {
+        success: false,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+        executionDurationMs,
+      };
+    }
+  })
+);
+```
+
+### Workflow 3: Job Status Checker
+**File:** `src/workflows/webhook/job-status-check.ts`
+
+```typescript
+/**
+ * WORKFLOW 3/3: Job Status Checker
+ *
+ * Check status of extraction job
+ *
+ * Usage:
+ * POST /job-status
+ * {
+ *   "jobId": "products-extraction-1234567890"
+ * }
+ */
+export const checkJobStatus = webhook('products-job-status', {
+  connection: 'products-job-status',
+}).then(
+  http('query-job-status', {}, async ctx => {
+    const { log, openKv, data } = ctx;
+
+    const requestData = typeof data === 'string' ? JSON.parse(data) : data;
+    const jobId = requestData?.jobId;
+
+    if (!jobId) {
+      return {
+        success: false,
+        error: 'Missing required field: jobId',
+      };
+    }
+
+    try {
+      const tracker = new JobTracker(openKv(':project:'), log);
+      const job = await tracker.getJob(jobId);
+
+      if (!job) {
+        return {
+          success: false,
+          error: `Job not found: ${jobId}`,
+        };
+      }
+
+      return {
+        success: true,
+        job,
+      };
+    } catch (error: any) {
+      log.error('Failed to get job status', error);
+      return {
+        success: false,
+        message: error instanceof Error ? error.message : String(error),
+
+        stack: error instanceof Error ? error.stack : undefined,
+
+        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+      };
+    }
+  })
+);
+```
+
+### Entry Point (Workflow Registration)
+**File:** `index.ts`
+
+```typescript
+/**
+ * Entry point - Export all workflows for Versori platform
+ *
+ * This file exports all workflows to be registered with Versori.
+ * Each workflow is defined in its own file for better organization.
+ */
+
+// Scheduled workflows
+export { scheduledProductsExtraction } from './src/workflows/scheduled/daily-products-extraction';
+
+// Webhook workflows
+export { adHocProductsExtraction } from './src/workflows/webhook/adhoc-products-extraction';
+export { checkJobStatus } from './src/workflows/webhook/job-status-check';
+```
+
+---
+
+## Important: Schema Verification
+
+**Before using this template**, verify the GraphQL query structure against your actual Fluent Commerce schema:
+
+```bash
+# Introspect your schema
+cd fc-connect-sdk
+npx fc-connect introspect-schema --url https://your-fluent-api.com/graphql
+
+# Check available fields on Product type
+npx fc-connect introspect-schema --url <url> --output schema.json
+# Then search for "Product" type in schema.json
+```
+
+The query structure shown above is an **example**. Adjust field names to match your actual schema.
+
+## Use Cases
+
+**1. Amazon Seller Central Integration:**
+
+- Daily product feed for listing updates
+- Include GTIN/UPC for catalog matching
+- Export pricing/inventory separately
+
+**2. eBay Marketplace Sync:**
+
+- Catalog updates for active listings
+- Include product descriptions and attributes
+- Handle variation products
+
+**3. Internal Product Master:**
+
+- Central product catalog for all systems
+- Include extended attributes
+- Version control for catalog changes
+
+**4. Ad Hoc Refresh:**
+
+- Manual catalog refresh for urgent updates
+- Integration testing with sample data
+- Troubleshooting specific product changes
+
+## Production Checklist
+
+- [ ] **Verify GraphQL query against actual schema** (use introspection)
+- [ ] Set appropriate extraction frequency (daily for catalog)
+- [ ] Configure correct product status filter (ACTIVE only?)
+- [ ] Test with real product data changes
+- [ ] Verify S3 bucket permissions
+- [ ] Set up monitoring/alerts for failed extractions
+- [ ] Document JSON schema for downstream consumers
+- [ ] Configure S3 lifecycle policy for old files
+- [ ] Test incremental extraction (only changed products)
+- [ ] Test ad hoc extraction workflow
+- [ ] Test job status monitoring
+- [ ] Verify overlap buffer prevents missed records
+
+---
+
+### Pattern 7: State Management & Date Overrides
+
+**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
+
+**How it works**:
+
+VersoriKV stores the last successful extraction timestamp to enable incremental sync:
+
+```typescript
+interface ExtractionState {
+  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
+  recordCount: number; // Number of records extracted
+  extractedAt: string; // When extraction completed
+  fileName?: string; // Generated filename
+  s3Key?: string; // S3 upload path
+  overlapBufferSeconds?: number; // Buffer configuration
+}
+```
+
+**State Priority Chain** (highest to lowest):
+
+1. **`fromDate` override** (manual date in webhook payload) - Highest priority
+2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
+3. **`fallbackStartDate`** (activation variable) - First run fallback
+
+**Three Scenarios**:
+
+#### Scenario 1: Normal Scheduled Runs (Incremental)
+
+```typescript
+// Payload: {} (empty - no overrides)
+
+// Behavior:
+// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
+// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
+// 3. Extract records updated since buffered time
+// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
+// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
+// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
+```
+
+**Test**:
+
+```bash
+# Trigger scheduled run (no payload needed)
+# State advances automatically
+curl -X POST https://workspace.versori.run/products-extract-daily
+```
+
+#### Scenario 2: Ad-hoc Extraction WITH State Update
+
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
+
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
+// 3. Extract all records since 2025-01-01
+// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
+// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
+// 6. Next scheduled run starts from this new timestamp
+```
+
+**Use Case**: One-time catch-up extraction that advances the state pointer.
+
+**Test**:
+
+```bash
+curl -X POST https://workspace.versori.run/products-extract-webhook \
+  -H "x-api-key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "updateState": true
+  }'
+```
+
+#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
+
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
+
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z"
+// 3. Extract all records since 2025-01-01
+// 4. DO NOT update state
+// 5. Next scheduled run uses previous timestamp (unaffected)
+```
+
+**Use Case**: Historical backfill or testing without affecting incremental sync.
+
+**Test**:
+
+```bash
+curl -X POST https://workspace.versori.run/products-extract-webhook \
+  -H "x-api-key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-31T23:59:59Z",
+    "updateState": false
+  }'
+```
+
+**Why this matters**:
+
+- **Incremental sync** relies on state continuity
+- **Manual overrides** allow catch-up without breaking incremental flow
+- **Overlap buffer** prevents missed records at time boundaries
+- **State isolation** lets you test/backfill without affecting production sync
+
+---
+
+### Pattern 8: Optional GraphQL Query Logging
+
+**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
+
+**When to use**:
+
+- ✅ Debugging pagination issues
+- ✅ Verifying query variables (dates, filters, limits)
+- ✅ Development and testing
+- ❌ Production (verbose logs, potential secrets in variables)
+
+**How to enable**:
+
+Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
+
+**Implementation**:
+
+```typescript
+// In your extraction workflow
+const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
+
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Query Debug', {
+    query: PRODUCTS_QUERY,
+    variables: {
+      catalogues,
+      dateRangeFilter,
+      first: pageSize,
+      after: null, // First page
+    },
+    pagination: {
+      pageSize,
+      maxRecords,
+      currentPage: 1,
+    },
+  });
+}
+
+const extractionResult = await orchestrator.extract({
+  query: PRODUCTS_QUERY,
+  resultPath: 'products.edges.node',
+  variables: {
+    catalogues,
+    dateRangeFilter,
+  },
+  pageSize,
+  maxRecords,
+});
+
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Response Debug', {
+    totalRecords: extractionResult.stats.totalRecords,
+    totalPages: extractionResult.stats.totalPages,
+    truncated: extractionResult.stats.truncated,
+    truncationReason: extractionResult.stats.truncationReason,
+    validRecords: extractionResult.stats.validRecords ?? extractionResult.data.length,
+    firstRecordId: extractionResult.data[0]?.id,
+    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
+  });
+}
+```
+
+**What gets logged**:
+
+```json
+{
+  "level": "info",
+  "message": "GraphQL Query Debug",
+  "query": "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)",
+  "variables": {
+    "catalogues": [{ "ref": "DEFAULT_CATALOGUE" }],
+    "dateRangeFilter": "2025-01-22T09:59:00Z",
+    "first": 200,
+    "after": null
+  },
+  "pagination": {
+    "pageSize": 200,
+    "maxRecords": 50000,
+    "currentPage": 1
+  }
+}
+```
+
+**Versori Environment Variables**:
+
+Add to activation settings:
+
+```json
+{
+  "DEBUG_GRAPHQL": "true"
+}
+```
+
+**Testing**:
+
+```bash
+# Enable debug logging
+curl -X POST https://workspace.versori.run/products-extract-daily
+
+# Check Versori logs for "GraphQL Query Debug" entries
+# Verify query structure and variables are correct
+```
+
+**Sample Debug Output**:
+
+```
+[INFO] GraphQL Query Debug
+  query: "query GetProducts($catalogues: [ProductCatalogueKey], $dateRangeFilter: DateRange, ...)"
+  variables: { catalogues: [{ ref: "DEFAULT_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
+  pagination: { pageSize: 200, maxRecords: 50000, currentPage: 1 }
+
+[INFO] Extraction complete
+  totalRecords: 1250
+  totalPages: 7
+  truncated: false
+  validRecords: 1250
+
+[INFO] GraphQL Response Debug
+  totalRecords: 1250
+  totalPages: 7
+  truncated: false
+  truncationReason: null
+  validRecords: 1250
+  firstRecordId: "product_abc"
+  lastRecordId: "product_xyz"
+```
+
+**Key Benefits**:
+
+- Quickly identify pagination configuration issues
+- Verify date filters are applied correctly
+- Debug "no records found" scenarios
+- Validate ExtractionOrchestrator variable injection
+
+**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
+
+---
+
+## Troubleshooting Guide
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records
+- **Fix**: Reduce maxRecords, increase frequency
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch
+- **Fix**: Run schema validation, check field names
+
+**Issue**: "State not updating"
+
+- **Cause**: KV write failure or intentional retry
+- **Fix**: Check KV logs, verify state update code
+
+**Issue**: "First run exceeds limits"
+
+- **Cause**: No previous timestamp, fetches all
+- **Fix**: Set fallbackStartDate close to current, apply filters
+
+**Issue**: "Excessive duplicates"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved
+- **Fix**: Verify newTimestamp saved WITHOUT buffer
+
+**Issue**: "Job status not found"
+
+- **Cause**: JobTracker not initialized or wrong jobId
+- **Fix**: Verify JobTracker initialization and jobId format
+
+---
+
+**Pattern**: Enterprise incremental extraction with overlap buffer for product catalog - Three-workflow approach
+**⚠️ Versori Sample**: Reference implementation - adapt for your production use case
+**Key Learning**: ALWAYS verify GraphQL schema before using queries
+**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
+**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
+**Schema**: Use schema introspection - don't guess field names
+**Three Workflows**: Scheduled incremental, Ad hoc HTTP trigger, Job status monitoring
+
+---
+
+## See Also
+
+**Validation & Best Practices:**
+
+- [CLI Validation Workflow](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-11-cli-tools.md) - Validate GraphQL queries and mappings before deployment
+- [Extraction Modes Guide](../extraction-modes-guide.md) - Comparison of incremental vs dateRange vs historical modes
+
+**SDK Documentation:**
+
+- [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Complete field mapping documentation
+- [SDK CLI Tools](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) - Command-line tools reference
+
+**Related Extraction Patterns:**
+
+- [Products to SFTP XML](./template-extraction-products-to-sftp-xml.md) - Same entity, different format/destination
+- [Virtual Positions to S3 JSON](./template-extraction-virtual-positions-to-s3-json.md) - Different entity, JSON format
+
+---
+
+### Pattern 9: Backward Pagination (Optional - Advanced)
+
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+
+**When to Use**:
+
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
+
+**GraphQL Query Requirements**:
+
+Your query must support backward pagination by including `$last` and `$before`:
+
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ✅ REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ✅ REQUIRED for backward
+    }
+  }
+}
+```
+
+**Implementation**:
+
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    // ❌ Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ✅ Enable reverse pagination
+  maxRecords: 10000,
+});
+
+// Records are returned in reverse chronological order
+log.info('Extraction order verification', {
+  newest: result.data[0].createdAt,
+  oldest: result.data[result.data.length - 1].createdAt
+});
+```
+
+**Key Differences from Forward Pagination**:
+
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
+
+**Important Notes**:
+
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+
+**Example: Extract Latest 1000 Orders**
+
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+
+**When to Use Forward vs Backward**:
+
+```typescript
+// ✅ Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+
+// ✅ Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+
+**Pagination Variables Reference**:
+
+| Variable | Forward     | Backward    | Injected By  | Notes                    |
+| -------- | ----------- | ----------- | ------------ | ------------------------ |
+| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
+| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
+| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
+| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
+
+**Common Mistakes to Avoid**:
+
+```typescript
+// ❌ WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, // ❌ Orchestrator will override this
+    before: cursor, // ❌ Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+
+// ✅ CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ✅ Your business variables only
+  },
+  pageSize: 200, // ✅ Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+
+#### Optional: Reverse Pagination
+
+- Default: forward ($first/$after) + pageInfo.hasNextPage.
+- Reverse: query must use $last/$before; response must include pageInfo.hasPreviousPage; set direction='backward'.
+
+GraphQL:
+
+```graphql
+query GetProductsBackward(
+  $catalogues: [CatalogueKey]
+  $dateRangeFilter: DateRange
+  $last: Int!
+  $before: String
+) {
+  products(catalogues: $catalogues, updatedOn: $dateRangeFilter, last: $last, before: $before) {
+    edges {
+      cursor
+      node {
+        id
+        ref
+        updatedOn
+      }
+    }
+    pageInfo {
+      hasPreviousPage
+    }
+  }
+}
+```
+
+SDK:
+
+```typescript
+await orchestrator.extract({
+  query: PRODUCTS_BACKWARD_QUERY,
+  resultPath: 'products.edges.node',
+  variables: { catalogues, dateRangeFilter },
+  pageSize,
+  direction: 'backward',
+});
+```
+
+---
+
+## Testing Checklist
+
+**Before production deployment:**
+
+### 1. Schema Validation
+
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/products.export.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/products.export.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+
+### 2. Extraction Testing
+
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+
+### 3. Mapping Testing
+
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+
+### 4. JSON Generation Testing
+
+- [ ] Verify JSON structure matches expected format
+- [ ] Test JSON validation against schema (if applicable)
+- [ ] Verify proper nesting and structure
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test special character escaping
+
+### 5. S3 Upload Testing
+
+- [ ] Test S3 connection and authentication
+- [ ] Verify file upload to correct bucket and path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify S3 object metadata
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify file permissions and ACLs
+
+### 6. State Management Testing
+
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+
+### 7. Job Tracking Testing
+
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+
+### 8. Error Handling Testing
+
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with S3 connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+
+### 9. Staging Environment Testing
+
+- [ ] Run full extraction in staging environment
+- [ ] Verify JSON file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+
+### 10. Integration Testing
+
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger → extract → transform → upload → verify file
+
+---
+## Monitoring & Alerting
+
+### Success Response Example
+
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_PRD_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "products-2025-11-02T14-00-00-000Z.json",
+  "s3Path": "s3://bucket/products/products-2025-11-02T14-00-00-000Z.json",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+
+### Error Response Example
+
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_PRD_20251102_140500_xyz789",
+  "error": "S3 upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "s3_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+
+### Key Metrics to Track
+
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+
+  // File Generation
+  fileSizeMB: (jsonContent.length / (1024 * 1024)).toFixed(2),
+
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+
+log.info('Extraction metrics', metrics);
+```
+
+### Alert Thresholds
+
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+
+### Monitoring Dashboard Queries
+
+**Versori Platform Logs Query:**
+
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+
+# High error rates
+errorRate:>5
+
+# State management issues
+stateUpdated:false AND success:true
+```
+
+### Common Issues and Solutions
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+
+**Issue**: "S3 connection timeout"
+
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check S3 credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+
+**Issue**: "State not updating after successful extraction"
+
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+
+**Issue**: "First run exceeds record limits"
+
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+
+**Issue**: "Excessive duplicate records in output"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+
+---
+
+## Troubleshooting Quick Reference
+
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "JSON generation failed" | Format-specific error | Check JSON generation logic, validate output |
+
+---