npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.53 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (495) hide show

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-json.md CHANGED Viewed

@@ -1,2042 +1,2042 @@
----
-template_id: tpl-extract-virtual-positions-graphql-to-s3-json
-canonical_filename: template-extraction-virtual-positions-to-s3-json.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: s3-json
-entity: virtual-positions
-format: json
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
----
-# Template: Extraction - Virtual Positions GraphQL to S3 JSON
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Memory Management** - Clear large result sets after processing batches
-- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
-- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
----
-## 📚 STEP 1: Load These Docs (Human Checklist)
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
----
-## 📋 Implementation Prompt
-```
-Create a Versori scheduled extractor for virtualPositions using ExtractionOrchestrator with JobTracker for job lifecycle management, supports incremental runs with overlap buffer, transforms via UniversalMapper, and uploads JSON to S3 using S3DataSource. Use native Versori logging and KV state. Include three workflows: scheduled incremental, ad hoc manual trigger, and job status query.
-```
----
-## 📦 SDK Imports (Verified - Versori Optimized)
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-} from '@fluentcommerce/fc-connect-sdk';
-import { schedule, webhook, http } from '@versori/run';
-```
----
-# Versori Scheduled: Virtual Positions Extraction to S3 JSON (ATP/Allocation)
-**FC Connect SDK Use Case Guide**
-> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> Version: Use ^0.1.39 - `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
-Context: Scheduled Versori workflow that extracts virtual positions (ATP/allocated inventory) from Fluent Commerce via GraphQL query using **ExtractionOrchestrator with JobTracker**, transforms with `UniversalMapper`, and writes JSON files to S3 for API consumption and real-time order promising.
-**Pattern**: EXTRACTION (Fluent → S3 JSON)
-**Complexity**: Medium | Runtime: Versori Platform (Scheduled)
----
-## ⚠️ IMPORTANT: Production-Ready Base Template
-> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
->
-> This is a **production-ready base template** demonstrating FC Connect SDK best practices for virtual position extraction workflows with JSON output to S3.
->
-> **✅ INCLUDED FEATURES:**
->
-> - ✅ Comprehensive error handling with retry logic
-> - ✅ S3 upload with proper error handling
-> - ✅ State management with overlap buffer (prevents missed records)
-> - ✅ Job tracking with lifecycle management
-> - ✅ Security (credential masking in logs)
-> - ✅ UTC time enforcement (prevents timezone bugs)
-> - ✅ Incremental extraction (safe, efficient, production-ready)
-> - ✅ Natural rate limiting via timestamps
->
-> **📝 BEFORE DEPLOYING:**
->
-> 1. Review and customize activation variables for your environment
-> 2. Test with sample data in your Versori workspace
-> 3. Adjust safety limits (pageSize, maxRecords) if needed
-> 4. Configure monitoring alerts for extraction failures
-> 5. Verify S3 bucket credentials and paths
->
-> **This base template follows SDK best practices - tweak specific to your needs.**
----
-## What You'll Build
-- **Three Versori workflows**: Scheduled incremental, ad hoc manual trigger, job status query
-- **ExtractionOrchestrator** for high-level extraction orchestration
-- **JobTracker** for job lifecycle and state management
-- **Incremental extraction** using `updatedOn > lastRunTime` filter with overlap buffer
-- **State management** with VersoriKV to track last successful run
-- GraphQL query with auto-pagination
-- UniversalMapper transformation with nested locationLink data
-- JSON file generation with metadata wrapper
-- S3 upload for API consumption
-- **Pretty print option** for human-readable JSON
-- **Failure recovery** with timestamp tracking
-## Business Use Case
-**Real-time ATP API for external systems:**
-- Extract ATP calculations every 15 minutes
-- Export as JSON for REST API consumption
-- Support webhooks/polling by downstream systems
-- Lightweight incremental updates for order promising
-- Enable real-time inventory allocation decisions
-- Feed to order management and ecommerce platforms
-## Virtual Positions Explained
-**VirtualPosition** = ATP (Available To Promise) calculation
-- Represents virtual/allocated inventory for promising
-- Calculated quantity available for new orders
-- Grouped by location or category
-- Used for: Order promising, allocation tracking, ATP feeds
-**vs InventoryPosition** = Physical on-hand calculation
-- Actual stock in warehouse
-- Used for: Stock reporting
-**vs InventoryQuantity** = Detailed quantity records
-- Individual quantity records by type
-- Used for: Audit trails
-## SDK Methods Used
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-} from '@fluentcommerce/fc-connect-sdk';
-await createClient(ctx); // Versori-aware client
-new ExtractionOrchestrator(client, log); // High-level orchestration
-new JobTracker(ctx.openKv(':project:'), log); // Job tracking
-await orchestrator.extractToS3Json({ query, mapping, s3Config, ... }); // Auto-pagination + mapping + upload
-new UniversalMapper(exportMapping); // Field transformation
-const jsonBuilder = new JSONBuilder({ prettyPrint: true, indent: 2 });
-const jsonContent = jsonBuilder.build(dataObject); // JSON generation
-await s3.uploadFile(key, Buffer.from(jsonContent, 'utf8'), options); // S3 upload
-```
-## Activation Variables
-```json
-{
-  "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE",
-  "s3BucketName": "atp-api-exports",
-  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
-  "awsSecretAccessKey": "********",
-  "awsRegion": "us-east-1",
-  "s3Prefix": "virtual-positions/api/",
-  "pageSize": 200,
-  "maxRecords": 100000,
-  "fallbackStartDate": "2024-01-01T00:00:00Z",
-  "overlapBufferSeconds": "60",
-  "positionTypes": "",
-  "groupRefs": "",
-  "productRefs": "",
-  "prettyPrint": "true",
-  "DEBUG_GRAPHQL": "false"
-}
-```
-## Package Configuration
-Create file: `package.json`
-```json
-{
-  "name": "virtual-positions-extraction-json",
-  "version": "1.0.0",
-  "type": "module",
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  }
-}
-```
-## Export Mapping Configuration
-Create file: `./config/virtual-positions.export.json`
-```json
-{
-  "name": "virtual-positions.export",
-  "version": "1.0.0",
-  "description": "Fluent Virtual Positions → JSON API Export (with nested locationLink)",
-  "fields": {
-    "positionRef": { "source": "ref", "required": true, "resolver": "sdk.trim" },
-    "productRef": { "source": "productRef", "required": true, "resolver": "sdk.trim" },
-    "locationRef": { "source": "locationLink.ref", "required": false, "resolver": "sdk.trim" },
-    "locationName": { "source": "locationLink.name", "required": false, "resolver": "sdk.trim" },
-    "quantity": { "source": "quantity", "required": true, "resolver": "sdk.parseInt" },
-    "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
-    "groupRef": { "source": "groupRef", "required": false, "resolver": "sdk.trim" },
-    "status": { "source": "status", "required": false, "resolver": "sdk.uppercase" },
-    "catalogueRef": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
-    "catalogueName": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
-    "createdOn": { "source": "createdOn", "required": true, "resolver": "sdk.toString" },
-    "updatedOn": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
-  }
-}
-```
-## GraphQL Query
-**Note**: This query is verified against Fluent Commerce schema introspection.
-```graphql
-query GetVirtualPositions(
-  $catalogues: [VirtualCatalogueKey]
-  $dateRangeFilter: DateRange
-  $productRefs: [String!]
-  $types: [String!]
-  $groupRefs: [String]
-  $first: Int!
-  $after: String
-) {
-  virtualPositions(
-    catalogues: $catalogues
-    updatedOn: $dateRangeFilter
-    productRef: $productRefs
-    type: $types
-    groupRef: $groupRefs
-    first: $first
-    after: $after
-  ) {
-    edges {
-      node {
-        id
-        ref
-        productRef
-        quantity
-        type
-        groupRef
-        status
-        catalogue {
-          ref
-          name
-        }
-        createdOn
-        updatedOn
-        locationLink {
-          ref
-          name
-          status
-        }
-      }
-      cursor
-    }
-    pageInfo {
-      hasNextPage
-    }
-  }
-}
-```
----
-## Versori Workflows Structure
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-### Recommended Project Structure
-```
-virtual-positions-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-virtual-positions-extraction.ts  # Scheduled: Daily extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-virtual-positions-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts                   # Webhook: Status query
-    │
-    ├── services/
-    │   └── virtual-positions-extraction.service.ts    # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── virtual-positions.export.json             # Mapping configuration
-```
----
-```json
-{
-  "metadata": {
-    "extractedAt": "2025-01-22T14:30:00.000Z",
-    "recordCount": 3,
-    "incrementalFrom": "2025-01-22T10:00:00.000Z",
-    "incrementalTo": "2025-01-22T14:30:00.000Z",
-    "jobId": "job_abc123",
-    "extractionMode": "incremental"
-  },
-  "data": [
-    {
-      "positionRef": "VPOS-001",
-      "productRef": "PROD-SKU-001",
-      "locationRef": "LOC-001",
-      "locationName": "NYC Warehouse",
-      "quantity": 75,
-      "type": "DEFAULT",
-      "groupRef": "GROUP-EAST",
-      "status": "ACTIVE",
-      "catalogueRef": "DEFAULT_VIRTUAL_CATALOGUE",
-      "catalogueName": "Default Virtual",
-      "createdOn": "2025-01-15T10:00:00Z",
-      "updatedOn": "2025-01-22T08:30:00Z"
-    },
-    {
-      "positionRef": "VPOS-002",
-      "productRef": "PROD-SKU-002",
-      "locationRef": "LOC-002",
-      "locationName": "LA Warehouse",
-      "quantity": 50,
-      "type": "DEFAULT",
-      "groupRef": "GROUP-WEST",
-      "status": "ACTIVE",
-      "catalogueRef": "DEFAULT_VIRTUAL_CATALOGUE",
-      "catalogueName": "Default Virtual",
-      "createdOn": "2025-01-16T11:00:00Z",
-      "updatedOn": "2025-01-22T09:15:00Z"
-    },
-    {
-      "positionRef": "VPOS-003",
-      "productRef": "PROD-SKU-003",
-      "locationRef": "LOC-001",
-      "locationName": "NYC Warehouse",
-      "quantity": 100,
-      "type": "SEASONAL",
-      "groupRef": "GROUP-EAST",
-      "status": "ACTIVE",
-      "catalogueRef": "SEASONAL_VIRTUAL_CATALOGUE",
-      "catalogueName": "Seasonal Virtual",
-      "createdOn": "2025-01-17T12:00:00Z",
-      "updatedOn": "2025-01-22T10:00:00Z"
-    }
-  ]
-}
-```
-## Complete Workflow Implementation
-The following sections demonstrate the three workflows. Each workflow should be in its own file following the modular project structure shown above.
-### Entry Point: index.ts
-**File:** `index.ts`
-```typescript
-/**
- * Entry point - Export all workflows for Versori platform
- *
- * This file exports all workflows to be registered with Versori.
- * Each workflow is defined in its own file for better organization.
- */
-// Scheduled workflows
-export { virtualPositionsExtractionJson } from './src/workflows/scheduled/daily-virtual-positions-extraction';
-// Webhook workflows
-export { virtualPositionsManualExtraction } from './src/workflows/webhook/adhoc-virtual-positions-extraction';
-export { virtualPositionsJobStatus } from './src/workflows/webhook/job-status-check';
-```
-### Workflow 1: Scheduled Incremental Extraction
-**File:** `src/workflows/scheduled/daily-virtual-positions-extraction.ts`
-```typescript
-import { schedule, http } from '@versori/run';
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-} from '@fluentcommerce/fc-connect-sdk';
-import virtualPositionsExportMapping from './config/virtual-positions.export.json' with { type: 'json' };
-// GraphQL query
-const VIRTUAL_POSITIONS_QUERY = `
-  query GetVirtualPositions(
-    $catalogues: [VirtualCatalogueKey]
-    $dateRangeFilter: DateRange
-    $productRefs: [String!]
-    $types: [String!]
-    $groupRefs: [String]
-    $first: Int!
-    $after: String
-  ) {
-    virtualPositions(
-      catalogues: $catalogues
-      updatedOn: $dateRangeFilter
-      productRef: $productRefs
-      type: $types
-      groupRef: $groupRefs
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          productRef
-          quantity
-          type
-          groupRef
-          status
-          catalogue {
-            ref
-            name
-          }
-          createdOn
-          updatedOn
-          locationLink {
-            ref
-            name
-            status
-          }
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-export const virtualPositionsExtractionJson = schedule(
-  'virtual-positions-extract-json-15min',
-  '*/15 * * * *'
-).then(
-  http('extract-virtual-positions-json', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, activation } = ctx;
-    const executionStartTime = Date.now();
-    log.info('🚀 [EXTRACTION] Starting virtual positions extraction to S3 JSON', {
-      timestamp: new Date().toISOString(),
-    });
-    // STEP 1/8: Parse activation variables
-    const catalogueRefsStr =
-      activation?.getVariable('catalogueRefs') || 'DEFAULT_VIRTUAL_CATALOGUE';
-    const catalogueRefsList = catalogueRefsStr
-      .split(',')
-      .map(ref => ref.trim())
-      .filter(Boolean);
-    const catalogues = catalogueRefsList.map(ref => ({ ref }));
-    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
-    const maxRecords = parseInt(activation?.getVariable('maxRecords') || '100000', 10);
-    const fallbackStartDate =
-      activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
-    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
-    const overlapBufferSeconds = parseInt(
-      activation?.getVariable('overlapBufferSeconds') || '60',
-      10
-    );
-    const s3Config = {
-      bucket: activation?.getVariable('s3BucketName'),
-      region: activation?.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation?.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
-    };
-    const s3Prefix = activation?.getVariable('s3Prefix') || 'virtual-positions/api/';
-    // Optional filtering
-    const positionTypesStr = activation?.getVariable('positionTypes') || '';
-    const groupRefsStr = activation?.getVariable('groupRefs') || '';
-    const productRefsStr = activation?.getVariable('productRefs') || '';
-    const positionTypes = positionTypesStr
-      ? positionTypesStr
-          .split(',')
-          .map(t => t.trim())
-          .filter(Boolean)
-      : undefined;
-    const groupRefs = groupRefsStr
-      ? groupRefsStr
-          .split(',')
-          .map(g => g.trim())
-          .filter(Boolean)
-      : undefined;
-    const productRefs = productRefsStr
-      ? productRefsStr
-          .split(',')
-          .map(p => p.trim())
-          .filter(Boolean)
-      : undefined;
-    // Validate required variables
-    const missing: string[] = [];
-    if (catalogues.length === 0) missing.push('catalogueRefs');
-    if (!s3Config.bucket) missing.push('s3BucketName');
-    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
-    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
-    if (missing.length)
-      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
-    try {
-      // STEP 2/8: Initialize SDK services
-      log.info('📦 [INITIALIZATION] Creating Fluent Commerce client');
-      const client = await createClient(ctx);
-      const orchestrator = new ExtractionOrchestrator(client, log);
-      const jobTracker = new JobTracker(openKv(':project:'), log);
-      log.info('✅ [INITIALIZATION] SDK services initialized successfully');
-      // STEP 3/8: Determine incremental date range with overlap buffer
-      log.info('📅 [STATE] Loading extraction state');
-      const stateKey = ['extraction', 'virtual-positions-json', 'lastRunTime'];
-      const kv = openKv(':project:');
-      const lastRunState = await kv.get(stateKey);
-      const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
-      // Apply overlap buffer for query (safety window)
-      const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
-      const bufferedLastRunTime = new Date(
-        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
-      ).toISOString();
-      const effectiveEndTime = new Date().toISOString();
-      const dateRangeFilter = {
-        from: bufferedLastRunTime,
-        to: effectiveEndTime, // End of extraction window
-      };
-      log.info('✅ [STATE] Incremental extraction mode with overlap buffer', {
-        rawLastRunTime,
-        bufferedLastRunTime,
-        effectiveEndTime,
-        overlapBufferSeconds,
-      });
-      // STEP 4/8: Create extraction job
-      log.info('📝 [JOB] Creating extraction job');
-      const jobId = `virtual-positions-json-${Date.now()}`;
-      await jobTracker.createJob(jobId, {
-        type: 'extraction',
-        entity: 'virtualPositions',
-        mode: 'incremental',
-        dateRangeFilter,
-        startTime: executionStartTime,
-      });
-      log.info('✅ [JOB] Extraction job created', {
-        jobId,
-        catalogues,
-        dateRangeFilter,
-        maxRecords,
-        positionTypes,
-        groupRefs,
-        productRefs,
-      });
-      // ? Enhanced: Extract context for progress logging
-      const dateRangeInfo = {
-        start: dateRangeFilter?.from || 'N/A',
-        end: dateRangeFilter?.to || 'N/A',
-        catalogues: catalogues.map((c: any) => c.ref || c).join(', ') || 'all',
-        types: positionTypes?.join(', ') || 'all',
-        groups: groupRefs?.join(', ') || 'none',
-        products: productRefs?.join(', ') || 'none'
-      };
-      // ? Enhanced: Start logging with context
-      log.info(`🔍 [EXTRACTION] Starting GraphQL extraction`, {
-       query: 'virtualPositions',
-       pageSize,
-       maxRecords,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       catalogues: dateRangeInfo.catalogues,
-       positionTypes: dateRangeInfo.types,
-       groupRefs: dateRangeInfo.groups,
-       productRefs: dateRangeInfo.products,
-       jobId
-      });
-      // STEP 5/8: Execute extraction with auto-pagination (WITH overlap buffer)
-      const extractionStartTime = Date.now();
-      const extractionResult = await orchestrator.extract({
-        query: VIRTUAL_POSITIONS_QUERY,
-        resultPath: 'virtualPositions.edges.node',
-        variables: {
-          catalogues,
-          dateRangeFilter,
-          types: positionTypes,
-          groupRefs,
-          productRefs,
-        },
-        pageSize,
-        maxRecords,
-        validateItem: item => !!(item.ref && item.productRef),
-      });
-      const rawRecords = extractionResult.data;
-      const extractionDuration = Date.now() - extractionStartTime;
-      log.info('✅ [EXTRACTION] Virtual position extraction completed', {
-        jobId,
-        totalRecords: extractionResult.stats.totalRecords,
-        totalPages: extractionResult.stats.totalPages,
-        validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-        errors: extractionResult.errors ? extractionResult.errors.length : 0,
-        duration: `${extractionDuration}ms`,
-      });
-      // ? Enhanced: Completion logging with summary
-      log.info(`✅ [STATS] Extraction statistics`, {
-       totalRecords: extractionResult.stats.totalRecords,
-       totalPages: extractionResult.stats.totalPages,
-       validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-       failedValidations: extractionResult.stats.failedValidations,
-       truncated: extractionResult.stats.truncated,
-       truncationReason: extractionResult.stats.truncationReason,
-       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-       duration: `${extractionDuration}ms`,
-       jobId
-      });
-      if (extractionResult.errors && extractionResult.errors.length > 0) {
-        log.warn('Non-fatal extraction errors encountered', {
-          jobId,
-          errorCount: extractionResult.errors.length,
-          sampleErrors: extractionResult.errors.slice(0, 3),
-        });
-      }
-      if (rawRecords.length === 0) {
-        log.info('ℹ️ [EXTRACTION] No virtual position records to extract');
-        await jobTracker.markCompleted(jobId, { recordCount: 0 });
-        const totalDuration = Date.now() - executionStartTime;
-        log.info('✅ [COMPLETE] Extraction workflow completed', {
-          duration: `${totalDuration}ms`,
-          recordCount: 0,
-        });
-        return {
-          success: true,
-          message: 'No records to extract',
-          jobId,
-          dateRangeFilter,
-          duration: totalDuration,
-        };
-      }
-      log.info('📦 [DATA] Virtual position records retrieved', { count: rawRecords.length, jobId });
-      // STEP 6/8: Transform with UniversalMapper
-      log.info('🔄 [MAPPING] Starting field transformation');
-      const mapper = new UniversalMapper(virtualPositionsExportMapping);
-      const mappingResult = await mapper.map(rawRecords);
-      if (!mappingResult.success) {
-        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-        log.error('❌ [MAPPING] Transformation failed', {
-          errorCount: mappingErrors.length,
-          sampleErrors: mappingErrors.slice(0, 3),
-        });
-        await jobTracker.markFailed(
-          jobId,
-          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
-          {
-            errors: mappingErrors,
-          }
-        );
-        return {
-          success: false,
-          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-          jobId,
-          errors: mappingErrors,
-          recommendation: 'Check mapping configuration in config/virtual-positions.export.json. Verify all source paths exist in GraphQL response.',
-        };
-      }
-      const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-      const mappingErrors = mappingResult.errors || [];
-      if (mappingErrors.length > 0) {
-        log.warn('Some records failed transformation', {
-          jobId,
-          errorCount: mappingErrors.length,
-          sampleErrors: mappingErrors.slice(0, 3),
-        });
-      }
-      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-          jobId,
-          skippedFields: mappingResult.skippedFields,
-          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-        });
-      }
-      if (transformedRecords.length === 0) {
-        log.error('❌ [MAPPING] All records failed mapping', {
-          totalRecords: rawRecords.length,
-          errorCount: mappingErrors.length,
-        });
-        await jobTracker.markFailed(jobId, 'All records failed mapping', { errors: mappingErrors });
-        return {
-          success: false,
-          error: 'All records failed mapping',
-          jobId,
-          errors: mappingErrors,
-          recommendation: 'Review mapping errors above. Common issues: missing required fields, invalid resolver names, incorrect source paths.',
-        };
-      }
-      log.info('✅ [MAPPING] Records transformed successfully', {
-        successful: transformedRecords.length,
-        skippedRecords: rawRecords.length - transformedRecords.length,
-        errorRate: ((mappingErrors.length / rawRecords.length) * 100).toFixed(2) + '%',
-        jobId,
-      });
-      // STEP 7/8: Calculate max updatedOn for next incremental run (WITHOUT buffer)
-      const maxUpdatedOn = transformedRecords.reduce((max, record) => {
-        const recordTime = new Date(record.updatedOn).getTime();
-        return recordTime > max ? recordTime : max;
-      }, new Date(rawLastRunTime).getTime());
-      const newTimestamp = new Date(maxUpdatedOn).toISOString();
-      // Build JSON with metadata
-      const jsonOutput = {
-        metadata: {
-          extractedAt: new Date().toISOString(),
-          recordCount: transformedRecords.length,
-          incrementalFrom: rawLastRunTime,
-          incrementalTo: newTimestamp,
-          jobId,
-          extractionMode: 'incremental',
-        },
-        data: transformedRecords,
-      };
-      // Use JSONBuilder for consistent JSON generation
-      const jsonBuilder = new JSONBuilder({
-        prettyPrint,
-        indent: 2,
-      });
-      const jsonContent = jsonBuilder.build(jsonOutput);
-      // Generate timestamped filename
-      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-      const fileName = `virtual-positions-${timestamp}.json`;
-      const s3Key = `${s3Prefix}${fileName}`;
-      log.info('📄 [JSON] Generated JSON file', {
-        fileName,
-        size: `${(jsonContent.length / 1024).toFixed(2)} KB`,
-        recordCount: transformedRecords.length,
-        jobId,
-      });
-      // STEP 8/8: Upload to S3
-      log.info('☁️ [S3] Starting S3 upload');
-      const s3 = new S3DataSource(
-        {
-          type: 'S3_JSON',
-          connectionId: 's3-virtual-positions-json-export',
-          name: 'S3 Virtual Positions JSON Export',
-          s3Config,
-        },
-        log
-      );
-      await s3.uploadFile(s3Key, Buffer.from(jsonContent, 'utf8'), {
-        contentType: 'application/json',
-        metadata: {
-          recordCount: String(transformedRecords.length),
-          extractedAt: new Date().toISOString(),
-          jobId,
-          incrementalFrom: rawLastRunTime,
-          incrementalTo: newTimestamp,
-        },
-      });
-      log.info('✅ [S3] JSON file uploaded successfully', { s3Key, jobId });
-      // Update state with new timestamp (WITHOUT buffer - critical!)
-      await kv.set(stateKey, {
-        timestamp: newTimestamp, // ← NO buffer applied
-        recordCount: transformedRecords.length,
-        extractedAt: new Date().toISOString(),
-        overlapBufferSeconds, // Track buffer config
-        fileName,
-        s3Key,
-        jobId,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      });
-      log.info('💾 [STATE] State updated with new timestamp (without buffer)', {
-        newTimestamp,
-        overlapBufferSeconds,
-        jobId,
-      });
-      await jobTracker.markCompleted(jobId, {
-        recordCount: transformedRecords.length,
-        fileName,
-        s3Key,
-        errors: mappingErrors,
-      });
-      const totalDuration = Date.now() - executionStartTime;
-      log.info('✅ [COMPLETE] Extraction workflow completed successfully', {
-        duration: `${totalDuration}ms`,
-        extractionDuration: `${extractionDuration}ms`,
-        recordsExtracted: transformedRecords.length,
-        recordsFailed: mappingErrors.length,
-        s3Key,
-        jobId,
-      });
-      return {
-        success: true,
-        recordsExtracted: transformedRecords.length,
-        recordsFailed: mappingErrors.length,
-        fileName,
-        s3Key,
-        jobId,
-        newTimestamp,
-        duration: totalDuration,
-        extractionDuration,
-        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-      };
-    } catch (error: any) {
-      const totalDuration = Date.now() - executionStartTime;
-      log.error('❌ [FAILED] Extraction workflow failed', {
-        message: error?.message,
-        duration: `${totalDuration}ms`,
-        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-      });
-      return {
-        success: false,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
-        duration: totalDuration,
-        recommendation: 'Check error message above. Common issues: connection failures, invalid GraphQL query, S3 credentials, or mapping configuration errors.',
-      };
-    }
-  })
-);
-```
-### Workflow 2: Ad Hoc Manual Trigger
-**File:** `src/workflows/webhook/adhoc-virtual-positions-extraction.ts`
-```typescript
-import { webhook, http } from '@versori/run';
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  S3DataSource,
-  JSONBuilder,
-} from '@fluentcommerce/fc-connect-sdk';
-import virtualPositionsExportMapping from './config/virtual-positions.export.json' with { type: 'json' };
-export const virtualPositionsManualExtraction = webhook('manual-extract-virtual-positions', {
-  connection: 'virtual-positions-adhoc',
-  response: {
-    mode: 'sync',
-    onSuccess: ctx =>
-      new Response(JSON.stringify(ctx.data), {
-        status: 200,
-        headers: { 'Content-Type': 'application/json' },
-      }),
-    onError: ctx =>
-      new Response(JSON.stringify({ error: ctx.data }), {
-        status: 500,
-        headers: { 'Content-Type': 'application/json' },
-      }),
-  },
-}).then(
-  http('trigger-extraction', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, activation } = ctx;
-    const executionStartTime = Date.now();
-    log.info('🚀 [EXTRACTION] Starting ad hoc virtual positions extraction', {
-      timestamp: new Date().toISOString(),
-    });
-    // STEP 1/8: Parse request body for custom date range
-    const body = ctx.request?.body || {};
-    const startDate = body.startDate || new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); // Last 24 hours
-    const endDate = body.endDate || new Date().toISOString();
-    const catalogueRefs = body.catalogueRefs || 'DEFAULT_VIRTUAL_CATALOGUE';
-    const catalogues = catalogueRefs.split(',').map((ref: string) => ({ ref: ref.trim() }));
-    const s3Config = {
-      bucket: activation?.getVariable('s3BucketName'),
-      region: activation?.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation?.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
-    };
-    const s3Prefix = activation?.getVariable('s3Prefix') || 'virtual-positions/adhoc/';
-    log.info('📋 [CONFIG] Manual extraction triggered', { startDate, endDate, catalogues });
-    // STEP 2/8: Initialize SDK services
-    log.info('📦 [INITIALIZATION] Creating Fluent Commerce client');
-    const client = await createClient(ctx);
-    const orchestrator = new ExtractionOrchestrator(client, log);
-    const jobTracker = new JobTracker(openKv(':project:'), log);
-    log.info('✅ [INITIALIZATION] SDK services initialized successfully');
-    // STEP 3/8: Create extraction job
-    log.info('📝 [JOB] Creating ad hoc extraction job');
-    const jobId = `virtual-positions-adhoc-${Date.now()}`;
-    await jobTracker.createJob(jobId, {
-      type: 'extraction',
-      entity: 'virtualPositions',
-      mode: 'adhoc',
-      dateRange: { from: startDate, to: endDate },
-      startTime: executionStartTime,
-    });
-    // ? Enhanced: Extract context for progress logging
-    const dateRangeInfo = {
-      start: startDate || 'N/A',
-      end: endDate || 'N/A',
-      catalogues: catalogues?.map((c: any) => c.ref || c).join(', ') || 'all'
-    };
-    // ? Enhanced: Start logging with context
-    log.info(`🔍 [EXTRACTION] Starting GraphQL extraction`, {
-     query: 'virtualPositions',
-     pageSize: 200,
-     maxRecords: 100000,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     catalogues: dateRangeInfo.catalogues,
-     jobId
-    });
-    // STEP 4/8: Execute extraction with orchestrator
-    const extractionStartTime = Date.now();
-    const extractionResult = await orchestrator.extract({
-      query: VIRTUAL_POSITIONS_QUERY,
-      resultPath: 'virtualPositions.edges.node',
-      variables: {
-        catalogues,
-        dateRangeFilter: { from: startDate, to: endDate },
-      },
-      pageSize: 200,
-      maxRecords: 100000,
-      validateItem: (item: any) => !!(item.ref && item.productRef),
-    });
-    const rawRecords = extractionResult.data;
-    const extractionDuration = Date.now() - extractionStartTime;
-    log.info('✅ [EXTRACTION] Manual extraction completed', {
-      jobId,
-      totalRecords: extractionResult.stats.totalRecords,
-      totalPages: extractionResult.stats.totalPages,
-      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-      errors: extractionResult.errors ? extractionResult.errors.length : 0,
-      duration: `${extractionDuration}ms`,
-    });
-    // ? Enhanced: Completion logging with summary
-    log.info(`✅ [STATS] Extraction statistics`, {
-     totalRecords: extractionResult.stats.totalRecords,
-     totalPages: extractionResult.stats.totalPages,
-     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-     failedValidations: extractionResult.stats.failedValidations,
-     truncated: extractionResult.stats.truncated,
-     truncationReason: extractionResult.stats.truncationReason,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     duration: `${extractionDuration}ms`,
-     jobId
-    });
-    if (extractionResult.errors && extractionResult.errors.length > 0) {
-      log.warn('Non-fatal extraction errors encountered', {
-        jobId,
-        errorCount: extractionResult.errors.length,
-        sampleErrors: extractionResult.errors.slice(0, 3),
-      });
-    }
-    if (rawRecords.length === 0) {
-      log.info('ℹ️ [EXTRACTION] No records found for specified date range');
-      await jobTracker.markCompleted(jobId, {
-        recordCount: 0,
-        message: 'No records to extract',
-      });
-      const totalDuration = Date.now() - executionStartTime;
-      log.info('✅ [COMPLETE] Extraction workflow completed', {
-        duration: `${totalDuration}ms`,
-        recordCount: 0,
-      });
-      return {
-        success: true,
-        jobId,
-        recordCount: 0,
-        fileName: undefined,
-        s3Key: undefined,
-        duration: totalDuration,
-      };
-    }
-    // STEP 5/8: Transform records
-    log.info('🔄 [MAPPING] Starting field transformation');
-    const mapper = new UniversalMapper(virtualPositionsExportMapping);
-    const mappingResult = await mapper.map(rawRecords);
-    if (!mappingResult.success) {
-      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-      log.error('❌ [MAPPING] Transformation failed', {
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-      await jobTracker.markFailed(
-        jobId,
-        mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
-        {
-          errors: mappingErrors,
-        }
-      );
-      return {
-        success: false,
-        jobId,
-        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-        errors: mappingErrors,
-        recommendation: 'Check mapping configuration in config/virtual-positions.export.json. Verify all source paths exist in GraphQL response.',
-      };
-    }
-    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-    const mappingErrors = mappingResult.errors || [];
-    if (mappingErrors.length > 0) {
-      log.warn('Manual extraction mapping warnings', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-    }
-    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-        jobId,
-        skippedFields: mappingResult.skippedFields,
-        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-      });
-    }
-    if (transformedRecords.length === 0) {
-      log.error('❌ [MAPPING] All records failed mapping', {
-        totalRecords: rawRecords.length,
-        errorCount: mappingErrors.length,
-      });
-      await jobTracker.markFailed(jobId, 'All records failed mapping', { errors: mappingErrors });
-      return {
-        success: false,
-        jobId,
-        error: 'All records failed mapping',
-        errors: mappingErrors,
-        recommendation: 'Review mapping errors above. Common issues: missing required fields, invalid resolver names, incorrect source paths.',
-      };
-    }
-    log.info('✅ [MAPPING] Records transformed successfully', {
-      successful: transformedRecords.length,
-      skippedRecords: rawRecords.length - transformedRecords.length,
-      errorRate: ((mappingErrors.length / rawRecords.length) * 100).toFixed(2) + '%',
-      jobId,
-    });
-    // STEP 6/8: Build JSON output
-    log.info('📄 [JSON] Building JSON output');
-    const jsonOutput = {
-      metadata: {
-        extractedAt: new Date().toISOString(),
-        recordCount: transformedRecords.length,
-        dateRangeFrom: startDate,
-        dateRangeTo: endDate,
-        jobId,
-        extractionMode: 'adhoc',
-      },
-      data: transformedRecords,
-    };
-    // Use JSONBuilder for consistent JSON generation
-    const jsonBuilder = new JSONBuilder({
-      prettyPrint: true,
-      indent: 2,
-    });
-    const jsonContent = jsonBuilder.build(jsonOutput);
-    // STEP 7/8: Upload to S3
-    log.info('☁️ [S3] Starting S3 upload');
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const fileName = `virtual-positions-adhoc-${timestamp}.json`;
-    const s3Key = `${s3Prefix}${fileName}`;
-    const s3 = new S3DataSource(
-      {
-        type: 'S3_JSON',
-        connectionId: 's3-virtual-positions-json-export',
-        name: 'S3 Virtual Positions JSON Export',
-        s3Config,
-      },
-      log
-    );
-    await s3.uploadFile(s3Key, Buffer.from(jsonContent, 'utf8'), {
-      contentType: 'application/json',
-      metadata: {
-        recordCount: String(transformedRecords.length),
-        extractedAt: new Date().toISOString(),
-        jobId,
-      },
-    });
-    log.info('✅ [S3] JSON file uploaded successfully', { s3Key, jobId });
-    // STEP 8/8: Complete job
-    await jobTracker.markCompleted(jobId, {
-      recordCount: transformedRecords.length,
-      fileName,
-      s3Key,
-      errors: mappingErrors,
-    });
-    const totalDuration = Date.now() - executionStartTime;
-    log.info('✅ [COMPLETE] Ad hoc extraction workflow completed successfully', {
-      duration: `${totalDuration}ms`,
-      extractionDuration: `${extractionDuration}ms`,
-      jobId,
-      recordCount: transformedRecords.length,
-      s3Key,
-    });
-    return {
-      success: true,
-      jobId,
-      recordCount: transformedRecords.length,
-      fileName,
-      s3Key,
-      duration: totalDuration,
-      extractionDuration,
-      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-    };
-  })
-);
-```
-### Workflow 3: Job Status Query
-**File:** `src/workflows/webhook/job-status-check.ts`
-```typescript
-import { webhook, http } from '@versori/run';
-import { Buffer } from 'node:buffer';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-export const virtualPositionsJobStatus = webhook('job-status', {
-  connection: 'virtual-positions-job-status',
-  response: {
-    mode: 'sync',
-    onSuccess: ctx =>
-      new Response(JSON.stringify(ctx.data), {
-        status: 200,
-        headers: { 'Content-Type': 'application/json' },
-      }),
-    onError: ctx =>
-      new Response(JSON.stringify({ error: ctx.data }), {
-        status: 404,
-        headers: { 'Content-Type': 'application/json' },
-      }),
-  },
-}).then(
-  http('query-job-status', async ctx => {
-    const { log, openKv, activation } = ctx;
-    const jobId = ctx.request?.query?.jobId;
-    if (!jobId) {
-      return { error: 'Missing jobId query parameter' };
-    }
-    const jobTracker = new JobTracker(openKv(':project:'), log);
-    const jobState = await jobTracker.getJob(jobId);
-    if (!jobState) {
-      return { error: `Job not found: ${jobId}` };
-    }
-    return {
-      success: true,
-      job: jobState,
-    };
-  })
-);
-```
-## Key Differences from Manual Implementation
-1. **ExtractionOrchestrator**: High-level orchestration vs manual steps
-2. **JobTracker**: Automatic job lifecycle tracking
-3. **Three Workflows**: Scheduled, ad hoc, and status query patterns
-4. **STEP X/8 Comments**: Clear orchestration progression
-5. **Overlap Buffer Pattern**: Query WITH buffer, save WITHOUT buffer
-## Production Checklist
-- [ ] Set `catalogueRefs` to correct virtual catalogue
-- [ ] Configure extraction schedule (15min for real-time, hourly for standard)
-- [ ] Set `maxRecords` based on ATP volume (100k+ for large retailers)
-- [ ] Set `pageSize` to balance throughput and memory (200-500 recommended)
-- [ ] Enable/disable `prettyPrint` based on use case (false for production)
-- [ ] Verify S3 bucket permissions and CORS configuration
-- [ ] Set up S3 lifecycle policy to archive old JSON files
-- [ ] Test with incremental mode first (validates state management)
-- [ ] Document JSON schema for API consumers
-- [ ] Test failure recovery (state rollback on error)
-- [ ] Set up alerts for extraction failures
-- [ ] Test with real-time incremental changes (15min schedule)
-- [ ] Test ad hoc manual trigger workflow
-- [ ] Test job status query workflow
----
----
-### Pattern 7: State Management & Date Overrides
-**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
-**How it works**:
-VersoriKV stores the last successful extraction timestamp to enable incremental sync:
-```typescript
-interface ExtractionState {
-  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
-  recordCount: number; // Number of records extracted
-  extractedAt: string; // When extraction completed
-  fileName?: string; // Generated filename
-  s3Key?: string; // S3 upload path
-  overlapBufferSeconds?: number; // Buffer configuration
-}
-```
-**State Priority Chain** (highest to lowest):
-1. **`fromDate` override** (manual date in webhook payload) - Highest priority
-2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
-3. **`fallbackStartDate`** (activation variable) - First run fallback
-**Three Scenarios**:
-#### Scenario 1: Normal Scheduled Runs (Incremental)
-```typescript
-// Payload: {} (empty - no overrides)
-// Behavior:
-// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
-// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
-// 3. Extract records updated since buffered time
-// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
-// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
-// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
-```
-**Test**:
-```bash
-# Trigger scheduled run (no payload needed)
-# State advances automatically
-curl -X POST https://workspace.versori.run/virtual-positions-extract-json-15min
-```
-#### Scenario 2: Ad-hoc Extraction WITH State Update
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
-// 3. Extract all records since 2025-01-01
-// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
-// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
-// 6. Next scheduled run starts from this new timestamp
-```
-**Use Case**: One-time catch-up extraction that advances the state pointer.
-**Test**:
-```bash
-curl -X POST https://workspace.versori.run/manual-extract-virtual-positions \
-  -H "Content-Type: application/json" \
-  -d '{
-    "startDate": "2025-01-01T00:00:00Z",
-    "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE"
-  }'
-```
-#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z"
-// 3. Extract all records since 2025-01-01
-// 4. DO NOT update state
-// 5. Next scheduled run uses previous timestamp (unaffected)
-```
-**Use Case**: Historical backfill or testing without affecting incremental sync.
-**Test**:
-```bash
-curl -X POST https://workspace.versori.run/manual-extract-virtual-positions \
-  -H "Content-Type: application/json" \
-  -d '{
-    "startDate": "2025-01-01T00:00:00Z",
-    "endDate": "2025-01-31T23:59:59Z",
-    "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE"
-  }'
-```
-**Why this matters**:
-- **Incremental sync** relies on state continuity
-- **Manual overrides** allow catch-up without breaking incremental flow
-- **Overlap buffer** prevents missed records at time boundaries
-- **State isolation** lets you test/backfill without affecting production sync
----
-### Pattern 8: Optional GraphQL Query Logging
-**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
-**When to use**:
-- ✅ Debugging pagination issues
-- ✅ Verifying query variables (dates, filters, limits)
-- ✅ Development and testing
-- ❌ Production (verbose logs, potential secrets in variables)
-**How to enable**:
-Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
-**Implementation**:
-```typescript
-// In your extraction workflow
-const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Query Debug', {
-    query: VIRTUAL_POSITIONS_QUERY,
-    variables: {
-      catalogues,
-      dateRangeFilter,
-      first: pageSize,
-      after: null, // First page
-    },
-    pagination: {
-      pageSize,
-      maxRecords,
-      currentPage: 1,
-    },
-  });
-}
-const extractionResult = await orchestrator.extract({
-  query: VIRTUAL_POSITIONS_QUERY,
-  resultPath: 'virtualPositions.edges.node',
-  variables: {
-    catalogues,
-    dateRangeFilter,
-  },
-  pageSize,
-  maxRecords,
-});
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Response Debug', {
-    totalRecords: extractionResult.stats.totalRecords,
-    totalPages: extractionResult.stats.totalPages,
-    truncated: extractionResult.stats.truncated,
-    truncationReason: extractionResult.stats.truncationReason,
-    firstRecordId: extractionResult.data[0]?.id,
-    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
-  });
-}
-```
-**What gets logged**:
-```json
-{
-  "level": "info",
-  "message": "GraphQL Query Debug",
-  "query": "query GetVirtualPositions($catalogues: [VirtualCatalogueKey], $dateRangeFilter: DateRange, ...)",
-  "variables": {
-    "catalogues": [{ "ref": "DEFAULT_VIRTUAL_CATALOGUE" }],
-    "dateRangeFilter": "2025-01-22T09:59:00Z",
-    "first": 200,
-    "after": null
-  },
-  "pagination": {
-    "pageSize": 200,
-    "maxRecords": 100000,
-    "currentPage": 1
-  }
-}
-```
-**Versori Environment Variables**:
-Add to activation settings:
-```json
-{
-  "DEBUG_GRAPHQL": "true"
-}
-```
-**Testing**:
-```bash
-# Enable debug logging
-curl -X POST https://workspace.versori.run/virtual-positions-extract-json-15min
-# Check Versori logs for "GraphQL Query Debug" entries
-# Verify query structure and variables are correct
-```
-**Sample Debug Output**:
-```
-[INFO] GraphQL Query Debug
-  query: "query GetVirtualPositions($catalogues: [VirtualCatalogueKey], $dateRangeFilter: DateRange, ...)"
-  variables: { catalogues: [{ ref: "DEFAULT_VIRTUAL_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
-  pagination: { pageSize: 200, maxRecords: 100000, currentPage: 1 }
-[INFO] Extraction complete
-  totalRecords: 850
-  totalPages: 5
-  truncated: false
-[INFO] GraphQL Response Debug
-  totalRecords: 850
-  totalPages: 5
-  truncated: false
-  truncationReason: undefined
-  firstRecordId: "vp_001"
-  lastRecordId: "vp_850"
-```
-**Key Benefits**:
-- Quickly identify pagination configuration issues
-- Verify date filters are applied correctly
-- Debug "no records found" scenarios
-- Validate ExtractionOrchestrator variable injection
-**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
----
-**Pattern**: Enterprise incremental extraction with overlap buffer for virtual positions (ATP) - JSON format with ExtractionOrchestrator
-**Key Learning**: Use `catalogues` (plural, array) input for the Fluent API
-**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
-**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
-**Orchestration**: ExtractionOrchestrator + JobTracker for production-grade extraction workflows
-**Schema**: virtualPositions accepts `catalogues: [VirtualCatalogueKey]` where VirtualCatalogueKey = `{ ref: String! }`
-**Use Case**: Real-time ATP updates for order management systems; use camelCase for JSON fields
----
-### Pattern 8: Backward Pagination (Optional - Advanced)
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-**When to Use**:
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
-- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
-**GraphQL Query Requirements**:
-Your query must support backward pagination by including `$last` and `$before`:
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ✅ REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ✅ REQUIRED for backward
-    }
-  }
-}
-```
-**Implementation**:
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    // ❌ Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ✅ Enable reverse pagination
-  maxRecords: 10000,
-});
-// Records are returned in reverse chronological order
-log.info('Record order', {
-  newestRecord: result.data[0].createdAt,
-  oldestRecord: result.data[result.data.length - 1].createdAt
-});
-```
-**Key Differences from Forward Pagination**:
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
-**Important Notes**:
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-**Example: Extract Latest 1000 Orders**
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-**When to Use Forward vs Backward**:
-```typescript
-// ✅ Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-// ✅ Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-**Pagination Variables Reference**:
-| Variable | Forward     | Backward    | Injected By  | Notes                    |
-| -------- | ----------- | ----------- | ------------ | ------------------------ |
-| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
-| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
-| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
-| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
-**Common Mistakes to Avoid**:
-```typescript
-// ❌ WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, // ❌ Orchestrator will override this
-    before: cursor, // ❌ Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-// ✅ CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ✅ Your business variables only
-  },
-  pageSize: 200, // ✅ Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-#### Optional: Reverse Pagination
-- Forward default; reverse optional with $last/$before + pageInfo.hasPreviousPage.
-GraphQL:
-```graphql
-query GetVirtualPositionsBackward($last: Int!, $before: String) {
-  virtualPositions(last: $last, before: $before) {
-    edges {
-      cursor
-      node {
-        id
-        ref
-        updatedOn
-      }
-    }
-    pageInfo {
-      hasPreviousPage
-    }
-  }
-}
-```
-SDK:
-```typescript
-await orchestrator.extract({
-  query: VIRTUAL_POSITIONS_BACKWARD_QUERY,
-  resultPath: 'virtualPositions.edges.node',
-  variables: {},
-  pageSize,
-  direction: 'backward',
-});
-```
----
-## Testing Checklist
-**Before production deployment:**
-### 1. Schema Validation
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/virtual-positions.export.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/virtual-positions.export.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-### 2. Extraction Testing
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-### 3. Mapping Testing
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-### 4. JSON Generation Testing
-- [ ] Verify JSON structure matches expected format
-- [ ] Test JSON validation against schema (if applicable)
-- [ ] Verify proper nesting and structure
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test special character escaping
-### 5. S3 Upload Testing
-- [ ] Test S3 connection and authentication
-- [ ] Verify file upload to correct bucket and path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify S3 object metadata
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify file permissions and ACLs
-### 6. State Management Testing
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-### 7. Job Tracking Testing
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-### 8. Error Handling Testing
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with S3 connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-### 9. Staging Environment Testing
-- [ ] Run full extraction in staging environment
-- [ ] Verify JSON file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-### 10. Integration Testing
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger → extract → transform → upload → verify file
----
-## Monitoring & Alerting
-### Success Response Example
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_VP_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "virtual-positions-2025-11-02T14-00-00-000Z.json",
-  "s3Path": "s3://bucket/virtual-positions/virtual-positions-2025-11-02T14-00-00-000Z.json",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-### Error Response Example
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_VP_20251102_140500_xyz789",
-  "error": "S3 upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "s3_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-### Key Metrics to Track
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-  // File Generation
-  fileSizeMB: (jsonContent.length / (1024 * 1024)).toFixed(2),
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-log.info('Extraction metrics', metrics);
-```
-### Alert Thresholds
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-### Monitoring Dashboard Queries
-**Versori Platform Logs Query:**
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-# High error rates
-errorRate:>5
-# State management issues
-stateUpdated:false AND success:true
-```
-### Common Issues and Solutions
-**Issue**: "Extraction timeout after 10 minutes"
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-**Issue**: "Mapping errors for 50% of records"
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-**Issue**: "S3 connection timeout"
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check S3 credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-**Issue**: "State not updating after successful extraction"
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-**Issue**: "First run exceeds record limits"
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-**Issue**: "Excessive duplicate records in output"
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
----
-## Troubleshooting Quick Reference
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "JSON generation failed" | Format-specific error | Check JSON generation logic, validate output |
----
+---
+template_id: tpl-extract-virtual-positions-graphql-to-s3-json
+canonical_filename: template-extraction-virtual-positions-to-s3-json.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: s3-json
+entity: virtual-positions
+format: json
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+---
+# Template: Extraction - Virtual Positions GraphQL to S3 JSON
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Memory Management** - Clear large result sets after processing batches
+- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
+- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
+---
+## 📚 STEP 1: Load These Docs (Human Checklist)
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+---
+## 📋 Implementation Prompt
+```
+Create a Versori scheduled extractor for virtualPositions using ExtractionOrchestrator with JobTracker for job lifecycle management, supports incremental runs with overlap buffer, transforms via UniversalMapper, and uploads JSON to S3 using S3DataSource. Use native Versori logging and KV state. Include three workflows: scheduled incremental, ad hoc manual trigger, and job status query.
+```
+---
+## 📦 SDK Imports (Verified - Versori Optimized)
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+} from '@fluentcommerce/fc-connect-sdk';
+import { schedule, webhook, http } from '@versori/run';
+```
+---
+# Versori Scheduled: Virtual Positions Extraction to S3 JSON (ATP/Allocation)
+**FC Connect SDK Use Case Guide**
+> SDK: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> Version: Use ^0.1.39 - `npm install @fluentcommerce/fc-connect-sdk@^0.1.39`
+Context: Scheduled Versori workflow that extracts virtual positions (ATP/allocated inventory) from Fluent Commerce via GraphQL query using **ExtractionOrchestrator with JobTracker**, transforms with `UniversalMapper`, and writes JSON files to S3 for API consumption and real-time order promising.
+**Pattern**: EXTRACTION (Fluent → S3 JSON)
+**Complexity**: Medium | Runtime: Versori Platform (Scheduled)
+---
+## ⚠️ IMPORTANT: Production-Ready Base Template
+> **📋 BASE TEMPLATE - Ready for Production (Customize for Your Needs)**
+>
+> This is a **production-ready base template** demonstrating FC Connect SDK best practices for virtual position extraction workflows with JSON output to S3.
+>
+> **✅ INCLUDED FEATURES:**
+>
+> - ✅ Comprehensive error handling with retry logic
+> - ✅ S3 upload with proper error handling
+> - ✅ State management with overlap buffer (prevents missed records)
+> - ✅ Job tracking with lifecycle management
+> - ✅ Security (credential masking in logs)
+> - ✅ UTC time enforcement (prevents timezone bugs)
+> - ✅ Incremental extraction (safe, efficient, production-ready)
+> - ✅ Natural rate limiting via timestamps
+>
+> **📝 BEFORE DEPLOYING:**
+>
+> 1. Review and customize activation variables for your environment
+> 2. Test with sample data in your Versori workspace
+> 3. Adjust safety limits (pageSize, maxRecords) if needed
+> 4. Configure monitoring alerts for extraction failures
+> 5. Verify S3 bucket credentials and paths
+>
+> **This base template follows SDK best practices - tweak specific to your needs.**
+---
+## What You'll Build
+- **Three Versori workflows**: Scheduled incremental, ad hoc manual trigger, job status query
+- **ExtractionOrchestrator** for high-level extraction orchestration
+- **JobTracker** for job lifecycle and state management
+- **Incremental extraction** using `updatedOn > lastRunTime` filter with overlap buffer
+- **State management** with VersoriKV to track last successful run
+- GraphQL query with auto-pagination
+- UniversalMapper transformation with nested locationLink data
+- JSON file generation with metadata wrapper
+- S3 upload for API consumption
+- **Pretty print option** for human-readable JSON
+- **Failure recovery** with timestamp tracking
+## Business Use Case
+**Real-time ATP API for external systems:**
+- Extract ATP calculations every 15 minutes
+- Export as JSON for REST API consumption
+- Support webhooks/polling by downstream systems
+- Lightweight incremental updates for order promising
+- Enable real-time inventory allocation decisions
+- Feed to order management and ecommerce platforms
+## Virtual Positions Explained
+**VirtualPosition** = ATP (Available To Promise) calculation
+- Represents virtual/allocated inventory for promising
+- Calculated quantity available for new orders
+- Grouped by location or category
+- Used for: Order promising, allocation tracking, ATP feeds
+**vs InventoryPosition** = Physical on-hand calculation
+- Actual stock in warehouse
+- Used for: Stock reporting
+**vs InventoryQuantity** = Detailed quantity records
+- Individual quantity records by type
+- Used for: Audit trails
+## SDK Methods Used
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+} from '@fluentcommerce/fc-connect-sdk';
+await createClient(ctx); // Versori-aware client
+new ExtractionOrchestrator(client, log); // High-level orchestration
+new JobTracker(ctx.openKv(':project:'), log); // Job tracking
+await orchestrator.extractToS3Json({ query, mapping, s3Config, ... }); // Auto-pagination + mapping + upload
+new UniversalMapper(exportMapping); // Field transformation
+const jsonBuilder = new JSONBuilder({ prettyPrint: true, indent: 2 });
+const jsonContent = jsonBuilder.build(dataObject); // JSON generation
+await s3.uploadFile(key, Buffer.from(jsonContent, 'utf8'), options); // S3 upload
+```
+## Activation Variables
+```json
+{
+  "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE",
+  "s3BucketName": "atp-api-exports",
+  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
+  "awsSecretAccessKey": "********",
+  "awsRegion": "us-east-1",
+  "s3Prefix": "virtual-positions/api/",
+  "pageSize": 200,
+  "maxRecords": 100000,
+  "fallbackStartDate": "2024-01-01T00:00:00Z",
+  "overlapBufferSeconds": "60",
+  "positionTypes": "",
+  "groupRefs": "",
+  "productRefs": "",
+  "prettyPrint": "true",
+  "DEBUG_GRAPHQL": "false"
+}
+```
+## Package Configuration
+Create file: `package.json`
+```json
+{
+  "name": "virtual-positions-extraction-json",
+  "version": "1.0.0",
+  "type": "module",
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  }
+}
+```
+## Export Mapping Configuration
+Create file: `./config/virtual-positions.export.json`
+```json
+{
+  "name": "virtual-positions.export",
+  "version": "1.0.0",
+  "description": "Fluent Virtual Positions → JSON API Export (with nested locationLink)",
+  "fields": {
+    "positionRef": { "source": "ref", "required": true, "resolver": "sdk.trim" },
+    "productRef": { "source": "productRef", "required": true, "resolver": "sdk.trim" },
+    "locationRef": { "source": "locationLink.ref", "required": false, "resolver": "sdk.trim" },
+    "locationName": { "source": "locationLink.name", "required": false, "resolver": "sdk.trim" },
+    "quantity": { "source": "quantity", "required": true, "resolver": "sdk.parseInt" },
+    "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
+    "groupRef": { "source": "groupRef", "required": false, "resolver": "sdk.trim" },
+    "status": { "source": "status", "required": false, "resolver": "sdk.uppercase" },
+    "catalogueRef": { "source": "catalogue.ref", "required": false, "resolver": "sdk.trim" },
+    "catalogueName": { "source": "catalogue.name", "required": false, "resolver": "sdk.trim" },
+    "createdOn": { "source": "createdOn", "required": true, "resolver": "sdk.toString" },
+    "updatedOn": { "source": "updatedOn", "required": true, "resolver": "sdk.toString" }
+  }
+}
+```
+## GraphQL Query
+**Note**: This query is verified against Fluent Commerce schema introspection.
+```graphql
+query GetVirtualPositions(
+  $catalogues: [VirtualCatalogueKey]
+  $dateRangeFilter: DateRange
+  $productRefs: [String!]
+  $types: [String!]
+  $groupRefs: [String]
+  $first: Int!
+  $after: String
+) {
+  virtualPositions(
+    catalogues: $catalogues
+    updatedOn: $dateRangeFilter
+    productRef: $productRefs
+    type: $types
+    groupRef: $groupRefs
+    first: $first
+    after: $after
+  ) {
+    edges {
+      node {
+        id
+        ref
+        productRef
+        quantity
+        type
+        groupRef
+        status
+        catalogue {
+          ref
+          name
+        }
+        createdOn
+        updatedOn
+        locationLink {
+          ref
+          name
+          status
+        }
+      }
+      cursor
+    }
+    pageInfo {
+      hasNextPage
+    }
+  }
+}
+```
+---
+## Versori Workflows Structure
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+### Recommended Project Structure
+```
+virtual-positions-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-virtual-positions-extraction.ts  # Scheduled: Daily extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-virtual-positions-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts                   # Webhook: Status query
+    │
+    ├── services/
+    │   └── virtual-positions-extraction.service.ts    # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── virtual-positions.export.json             # Mapping configuration
+```
+---
+```json
+{
+  "metadata": {
+    "extractedAt": "2025-01-22T14:30:00.000Z",
+    "recordCount": 3,
+    "incrementalFrom": "2025-01-22T10:00:00.000Z",
+    "incrementalTo": "2025-01-22T14:30:00.000Z",
+    "jobId": "job_abc123",
+    "extractionMode": "incremental"
+  },
+  "data": [
+    {
+      "positionRef": "VPOS-001",
+      "productRef": "PROD-SKU-001",
+      "locationRef": "LOC-001",
+      "locationName": "NYC Warehouse",
+      "quantity": 75,
+      "type": "DEFAULT",
+      "groupRef": "GROUP-EAST",
+      "status": "ACTIVE",
+      "catalogueRef": "DEFAULT_VIRTUAL_CATALOGUE",
+      "catalogueName": "Default Virtual",
+      "createdOn": "2025-01-15T10:00:00Z",
+      "updatedOn": "2025-01-22T08:30:00Z"
+    },
+    {
+      "positionRef": "VPOS-002",
+      "productRef": "PROD-SKU-002",
+      "locationRef": "LOC-002",
+      "locationName": "LA Warehouse",
+      "quantity": 50,
+      "type": "DEFAULT",
+      "groupRef": "GROUP-WEST",
+      "status": "ACTIVE",
+      "catalogueRef": "DEFAULT_VIRTUAL_CATALOGUE",
+      "catalogueName": "Default Virtual",
+      "createdOn": "2025-01-16T11:00:00Z",
+      "updatedOn": "2025-01-22T09:15:00Z"
+    },
+    {
+      "positionRef": "VPOS-003",
+      "productRef": "PROD-SKU-003",
+      "locationRef": "LOC-001",
+      "locationName": "NYC Warehouse",
+      "quantity": 100,
+      "type": "SEASONAL",
+      "groupRef": "GROUP-EAST",
+      "status": "ACTIVE",
+      "catalogueRef": "SEASONAL_VIRTUAL_CATALOGUE",
+      "catalogueName": "Seasonal Virtual",
+      "createdOn": "2025-01-17T12:00:00Z",
+      "updatedOn": "2025-01-22T10:00:00Z"
+    }
+  ]
+}
+```
+## Complete Workflow Implementation
+The following sections demonstrate the three workflows. Each workflow should be in its own file following the modular project structure shown above.
+### Entry Point: index.ts
+**File:** `index.ts`
+```typescript
+/**
+ * Entry point - Export all workflows for Versori platform
+ *
+ * This file exports all workflows to be registered with Versori.
+ * Each workflow is defined in its own file for better organization.
+ */
+// Scheduled workflows
+export { virtualPositionsExtractionJson } from './src/workflows/scheduled/daily-virtual-positions-extraction';
+// Webhook workflows
+export { virtualPositionsManualExtraction } from './src/workflows/webhook/adhoc-virtual-positions-extraction';
+export { virtualPositionsJobStatus } from './src/workflows/webhook/job-status-check';
+```
+### Workflow 1: Scheduled Incremental Extraction
+**File:** `src/workflows/scheduled/daily-virtual-positions-extraction.ts`
+```typescript
+import { schedule, http } from '@versori/run';
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+} from '@fluentcommerce/fc-connect-sdk';
+import virtualPositionsExportMapping from './config/virtual-positions.export.json' with { type: 'json' };
+// GraphQL query
+const VIRTUAL_POSITIONS_QUERY = `
+  query GetVirtualPositions(
+    $catalogues: [VirtualCatalogueKey]
+    $dateRangeFilter: DateRange
+    $productRefs: [String!]
+    $types: [String!]
+    $groupRefs: [String]
+    $first: Int!
+    $after: String
+  ) {
+    virtualPositions(
+      catalogues: $catalogues
+      updatedOn: $dateRangeFilter
+      productRef: $productRefs
+      type: $types
+      groupRef: $groupRefs
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          productRef
+          quantity
+          type
+          groupRef
+          status
+          catalogue {
+            ref
+            name
+          }
+          createdOn
+          updatedOn
+          locationLink {
+            ref
+            name
+            status
+          }
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+export const virtualPositionsExtractionJson = schedule(
+  'virtual-positions-extract-json-15min',
+  '*/15 * * * *'
+).then(
+  http('extract-virtual-positions-json', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, activation } = ctx;
+    const executionStartTime = Date.now();
+    log.info('🚀 [EXTRACTION] Starting virtual positions extraction to S3 JSON', {
+      timestamp: new Date().toISOString(),
+    });
+    // STEP 1/8: Parse activation variables
+    const catalogueRefsStr =
+      activation?.getVariable('catalogueRefs') || 'DEFAULT_VIRTUAL_CATALOGUE';
+    const catalogueRefsList = catalogueRefsStr
+      .split(',')
+      .map(ref => ref.trim())
+      .filter(Boolean);
+    const catalogues = catalogueRefsList.map(ref => ({ ref }));
+    const pageSize = parseInt(activation?.getVariable('pageSize') || '200', 10);
+    const maxRecords = parseInt(activation?.getVariable('maxRecords') || '100000', 10);
+    const fallbackStartDate =
+      activation?.getVariable('fallbackStartDate') || '2024-01-01T00:00:00Z';
+    const prettyPrint = activation?.getVariable('prettyPrint') === 'true';
+    const overlapBufferSeconds = parseInt(
+      activation?.getVariable('overlapBufferSeconds') || '60',
+      10
+    );
+    const s3Config = {
+      bucket: activation?.getVariable('s3BucketName'),
+      region: activation?.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation?.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
+    };
+    const s3Prefix = activation?.getVariable('s3Prefix') || 'virtual-positions/api/';
+    // Optional filtering
+    const positionTypesStr = activation?.getVariable('positionTypes') || '';
+    const groupRefsStr = activation?.getVariable('groupRefs') || '';
+    const productRefsStr = activation?.getVariable('productRefs') || '';
+    const positionTypes = positionTypesStr
+      ? positionTypesStr
+          .split(',')
+          .map(t => t.trim())
+          .filter(Boolean)
+      : undefined;
+    const groupRefs = groupRefsStr
+      ? groupRefsStr
+          .split(',')
+          .map(g => g.trim())
+          .filter(Boolean)
+      : undefined;
+    const productRefs = productRefsStr
+      ? productRefsStr
+          .split(',')
+          .map(p => p.trim())
+          .filter(Boolean)
+      : undefined;
+    // Validate required variables
+    const missing: string[] = [];
+    if (catalogues.length === 0) missing.push('catalogueRefs');
+    if (!s3Config.bucket) missing.push('s3BucketName');
+    if (!s3Config.accessKeyId) missing.push('awsAccessKeyId');
+    if (!s3Config.secretAccessKey) missing.push('awsSecretAccessKey');
+    if (missing.length)
+      return { success: false, error: `Missing required variables: ${missing.join(', ')}` };
+    try {
+      // STEP 2/8: Initialize SDK services
+      log.info('📦 [INITIALIZATION] Creating Fluent Commerce client');
+      const client = await createClient(ctx);
+      const orchestrator = new ExtractionOrchestrator(client, log);
+      const jobTracker = new JobTracker(openKv(':project:'), log);
+      log.info('✅ [INITIALIZATION] SDK services initialized successfully');
+      // STEP 3/8: Determine incremental date range with overlap buffer
+      log.info('📅 [STATE] Loading extraction state');
+      const stateKey = ['extraction', 'virtual-positions-json', 'lastRunTime'];
+      const kv = openKv(':project:');
+      const lastRunState = await kv.get(stateKey);
+      const rawLastRunTime = lastRunState?.value?.timestamp || fallbackStartDate;
+      // Apply overlap buffer for query (safety window)
+      const OVERLAP_BUFFER_MS = overlapBufferSeconds * 1000;
+      const bufferedLastRunTime = new Date(
+        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_MS
+      ).toISOString();
+      const effectiveEndTime = new Date().toISOString();
+      const dateRangeFilter = {
+        from: bufferedLastRunTime,
+        to: effectiveEndTime, // End of extraction window
+      };
+      log.info('✅ [STATE] Incremental extraction mode with overlap buffer', {
+        rawLastRunTime,
+        bufferedLastRunTime,
+        effectiveEndTime,
+        overlapBufferSeconds,
+      });
+      // STEP 4/8: Create extraction job
+      log.info('📝 [JOB] Creating extraction job');
+      const jobId = `virtual-positions-json-${Date.now()}`;
+      await jobTracker.createJob(jobId, {
+        type: 'extraction',
+        entity: 'virtualPositions',
+        mode: 'incremental',
+        dateRangeFilter,
+        startTime: executionStartTime,
+      });
+      log.info('✅ [JOB] Extraction job created', {
+        jobId,
+        catalogues,
+        dateRangeFilter,
+        maxRecords,
+        positionTypes,
+        groupRefs,
+        productRefs,
+      });
+      // ? Enhanced: Extract context for progress logging
+      const dateRangeInfo = {
+        start: dateRangeFilter?.from || 'N/A',
+        end: dateRangeFilter?.to || 'N/A',
+        catalogues: catalogues.map((c: any) => c.ref || c).join(', ') || 'all',
+        types: positionTypes?.join(', ') || 'all',
+        groups: groupRefs?.join(', ') || 'none',
+        products: productRefs?.join(', ') || 'none'
+      };
+      // ? Enhanced: Start logging with context
+      log.info(`🔍 [EXTRACTION] Starting GraphQL extraction`, {
+       query: 'virtualPositions',
+       pageSize,
+       maxRecords,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       catalogues: dateRangeInfo.catalogues,
+       positionTypes: dateRangeInfo.types,
+       groupRefs: dateRangeInfo.groups,
+       productRefs: dateRangeInfo.products,
+       jobId
+      });
+      // STEP 5/8: Execute extraction with auto-pagination (WITH overlap buffer)
+      const extractionStartTime = Date.now();
+      const extractionResult = await orchestrator.extract({
+        query: VIRTUAL_POSITIONS_QUERY,
+        resultPath: 'virtualPositions.edges.node',
+        variables: {
+          catalogues,
+          dateRangeFilter,
+          types: positionTypes,
+          groupRefs,
+          productRefs,
+        },
+        pageSize,
+        maxRecords,
+        validateItem: item => !!(item.ref && item.productRef),
+      });
+      const rawRecords = extractionResult.data;
+      const extractionDuration = Date.now() - extractionStartTime;
+      log.info('✅ [EXTRACTION] Virtual position extraction completed', {
+        jobId,
+        totalRecords: extractionResult.stats.totalRecords,
+        totalPages: extractionResult.stats.totalPages,
+        validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+        errors: extractionResult.errors ? extractionResult.errors.length : 0,
+        duration: `${extractionDuration}ms`,
+      });
+      // ? Enhanced: Completion logging with summary
+      log.info(`✅ [STATS] Extraction statistics`, {
+       totalRecords: extractionResult.stats.totalRecords,
+       totalPages: extractionResult.stats.totalPages,
+       validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+       failedValidations: extractionResult.stats.failedValidations,
+       truncated: extractionResult.stats.truncated,
+       truncationReason: extractionResult.stats.truncationReason,
+       dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+       duration: `${extractionDuration}ms`,
+       jobId
+      });
+      if (extractionResult.errors && extractionResult.errors.length > 0) {
+        log.warn('Non-fatal extraction errors encountered', {
+          jobId,
+          errorCount: extractionResult.errors.length,
+          sampleErrors: extractionResult.errors.slice(0, 3),
+        });
+      }
+      if (rawRecords.length === 0) {
+        log.info('ℹ️ [EXTRACTION] No virtual position records to extract');
+        await jobTracker.markCompleted(jobId, { recordCount: 0 });
+        const totalDuration = Date.now() - executionStartTime;
+        log.info('✅ [COMPLETE] Extraction workflow completed', {
+          duration: `${totalDuration}ms`,
+          recordCount: 0,
+        });
+        return {
+          success: true,
+          message: 'No records to extract',
+          jobId,
+          dateRangeFilter,
+          duration: totalDuration,
+        };
+      }
+      log.info('📦 [DATA] Virtual position records retrieved', { count: rawRecords.length, jobId });
+      // STEP 6/8: Transform with UniversalMapper
+      log.info('🔄 [MAPPING] Starting field transformation');
+      const mapper = new UniversalMapper(virtualPositionsExportMapping);
+      const mappingResult = await mapper.map(rawRecords);
+      if (!mappingResult.success) {
+        const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+        log.error('❌ [MAPPING] Transformation failed', {
+          errorCount: mappingErrors.length,
+          sampleErrors: mappingErrors.slice(0, 3),
+        });
+        await jobTracker.markFailed(
+          jobId,
+          mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
+          {
+            errors: mappingErrors,
+          }
+        );
+        return {
+          success: false,
+          error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+          jobId,
+          errors: mappingErrors,
+          recommendation: 'Check mapping configuration in config/virtual-positions.export.json. Verify all source paths exist in GraphQL response.',
+        };
+      }
+      const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+      const mappingErrors = mappingResult.errors || [];
+      if (mappingErrors.length > 0) {
+        log.warn('Some records failed transformation', {
+          jobId,
+          errorCount: mappingErrors.length,
+          sampleErrors: mappingErrors.slice(0, 3),
+        });
+      }
+      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+        log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+          jobId,
+          skippedFields: mappingResult.skippedFields,
+          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+        });
+      }
+      if (transformedRecords.length === 0) {
+        log.error('❌ [MAPPING] All records failed mapping', {
+          totalRecords: rawRecords.length,
+          errorCount: mappingErrors.length,
+        });
+        await jobTracker.markFailed(jobId, 'All records failed mapping', { errors: mappingErrors });
+        return {
+          success: false,
+          error: 'All records failed mapping',
+          jobId,
+          errors: mappingErrors,
+          recommendation: 'Review mapping errors above. Common issues: missing required fields, invalid resolver names, incorrect source paths.',
+        };
+      }
+      log.info('✅ [MAPPING] Records transformed successfully', {
+        successful: transformedRecords.length,
+        skippedRecords: rawRecords.length - transformedRecords.length,
+        errorRate: ((mappingErrors.length / rawRecords.length) * 100).toFixed(2) + '%',
+        jobId,
+      });
+      // STEP 7/8: Calculate max updatedOn for next incremental run (WITHOUT buffer)
+      const maxUpdatedOn = transformedRecords.reduce((max, record) => {
+        const recordTime = new Date(record.updatedOn).getTime();
+        return recordTime > max ? recordTime : max;
+      }, new Date(rawLastRunTime).getTime());
+      const newTimestamp = new Date(maxUpdatedOn).toISOString();
+      // Build JSON with metadata
+      const jsonOutput = {
+        metadata: {
+          extractedAt: new Date().toISOString(),
+          recordCount: transformedRecords.length,
+          incrementalFrom: rawLastRunTime,
+          incrementalTo: newTimestamp,
+          jobId,
+          extractionMode: 'incremental',
+        },
+        data: transformedRecords,
+      };
+      // Use JSONBuilder for consistent JSON generation
+      const jsonBuilder = new JSONBuilder({
+        prettyPrint,
+        indent: 2,
+      });
+      const jsonContent = jsonBuilder.build(jsonOutput);
+      // Generate timestamped filename
+      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+      const fileName = `virtual-positions-${timestamp}.json`;
+      const s3Key = `${s3Prefix}${fileName}`;
+      log.info('📄 [JSON] Generated JSON file', {
+        fileName,
+        size: `${(jsonContent.length / 1024).toFixed(2)} KB`,
+        recordCount: transformedRecords.length,
+        jobId,
+      });
+      // STEP 8/8: Upload to S3
+      log.info('☁️ [S3] Starting S3 upload');
+      const s3 = new S3DataSource(
+        {
+          type: 'S3_JSON',
+          connectionId: 's3-virtual-positions-json-export',
+          name: 'S3 Virtual Positions JSON Export',
+          s3Config,
+        },
+        log
+      );
+      await s3.uploadFile(s3Key, Buffer.from(jsonContent, 'utf8'), {
+        contentType: 'application/json',
+        metadata: {
+          recordCount: String(transformedRecords.length),
+          extractedAt: new Date().toISOString(),
+          jobId,
+          incrementalFrom: rawLastRunTime,
+          incrementalTo: newTimestamp,
+        },
+      });
+      log.info('✅ [S3] JSON file uploaded successfully', { s3Key, jobId });
+      // Update state with new timestamp (WITHOUT buffer - critical!)
+      await kv.set(stateKey, {
+        timestamp: newTimestamp, // ← NO buffer applied
+        recordCount: transformedRecords.length,
+        extractedAt: new Date().toISOString(),
+        overlapBufferSeconds, // Track buffer config
+        fileName,
+        s3Key,
+        jobId,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      });
+      log.info('💾 [STATE] State updated with new timestamp (without buffer)', {
+        newTimestamp,
+        overlapBufferSeconds,
+        jobId,
+      });
+      await jobTracker.markCompleted(jobId, {
+        recordCount: transformedRecords.length,
+        fileName,
+        s3Key,
+        errors: mappingErrors,
+      });
+      const totalDuration = Date.now() - executionStartTime;
+      log.info('✅ [COMPLETE] Extraction workflow completed successfully', {
+        duration: `${totalDuration}ms`,
+        extractionDuration: `${extractionDuration}ms`,
+        recordsExtracted: transformedRecords.length,
+        recordsFailed: mappingErrors.length,
+        s3Key,
+        jobId,
+      });
+      return {
+        success: true,
+        recordsExtracted: transformedRecords.length,
+        recordsFailed: mappingErrors.length,
+        fileName,
+        s3Key,
+        jobId,
+        newTimestamp,
+        duration: totalDuration,
+        extractionDuration,
+        errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+      };
+    } catch (error: any) {
+      const totalDuration = Date.now() - executionStartTime;
+      log.error('❌ [FAILED] Extraction workflow failed', {
+        message: error?.message,
+        duration: `${totalDuration}ms`,
+        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+      });
+      return {
+        success: false,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'UnknownError',
+        duration: totalDuration,
+        recommendation: 'Check error message above. Common issues: connection failures, invalid GraphQL query, S3 credentials, or mapping configuration errors.',
+      };
+    }
+  })
+);
+```
+### Workflow 2: Ad Hoc Manual Trigger
+**File:** `src/workflows/webhook/adhoc-virtual-positions-extraction.ts`
+```typescript
+import { webhook, http } from '@versori/run';
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  S3DataSource,
+  JSONBuilder,
+} from '@fluentcommerce/fc-connect-sdk';
+import virtualPositionsExportMapping from './config/virtual-positions.export.json' with { type: 'json' };
+export const virtualPositionsManualExtraction = webhook('manual-extract-virtual-positions', {
+  connection: 'virtual-positions-adhoc',
+  response: {
+    mode: 'sync',
+    onSuccess: ctx =>
+      new Response(JSON.stringify(ctx.data), {
+        status: 200,
+        headers: { 'Content-Type': 'application/json' },
+      }),
+    onError: ctx =>
+      new Response(JSON.stringify({ error: ctx.data }), {
+        status: 500,
+        headers: { 'Content-Type': 'application/json' },
+      }),
+  },
+}).then(
+  http('trigger-extraction', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, activation } = ctx;
+    const executionStartTime = Date.now();
+    log.info('🚀 [EXTRACTION] Starting ad hoc virtual positions extraction', {
+      timestamp: new Date().toISOString(),
+    });
+    // STEP 1/8: Parse request body for custom date range
+    const body = ctx.request?.body || {};
+    const startDate = body.startDate || new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(); // Last 24 hours
+    const endDate = body.endDate || new Date().toISOString();
+    const catalogueRefs = body.catalogueRefs || 'DEFAULT_VIRTUAL_CATALOGUE';
+    const catalogues = catalogueRefs.split(',').map((ref: string) => ({ ref: ref.trim() }));
+    const s3Config = {
+      bucket: activation?.getVariable('s3BucketName'),
+      region: activation?.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation?.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation?.getVariable('awsSecretAccessKey'),
+    };
+    const s3Prefix = activation?.getVariable('s3Prefix') || 'virtual-positions/adhoc/';
+    log.info('📋 [CONFIG] Manual extraction triggered', { startDate, endDate, catalogues });
+    // STEP 2/8: Initialize SDK services
+    log.info('📦 [INITIALIZATION] Creating Fluent Commerce client');
+    const client = await createClient(ctx);
+    const orchestrator = new ExtractionOrchestrator(client, log);
+    const jobTracker = new JobTracker(openKv(':project:'), log);
+    log.info('✅ [INITIALIZATION] SDK services initialized successfully');
+    // STEP 3/8: Create extraction job
+    log.info('📝 [JOB] Creating ad hoc extraction job');
+    const jobId = `virtual-positions-adhoc-${Date.now()}`;
+    await jobTracker.createJob(jobId, {
+      type: 'extraction',
+      entity: 'virtualPositions',
+      mode: 'adhoc',
+      dateRange: { from: startDate, to: endDate },
+      startTime: executionStartTime,
+    });
+    // ? Enhanced: Extract context for progress logging
+    const dateRangeInfo = {
+      start: startDate || 'N/A',
+      end: endDate || 'N/A',
+      catalogues: catalogues?.map((c: any) => c.ref || c).join(', ') || 'all'
+    };
+    // ? Enhanced: Start logging with context
+    log.info(`🔍 [EXTRACTION] Starting GraphQL extraction`, {
+     query: 'virtualPositions',
+     pageSize: 200,
+     maxRecords: 100000,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     catalogues: dateRangeInfo.catalogues,
+     jobId
+    });
+    // STEP 4/8: Execute extraction with orchestrator
+    const extractionStartTime = Date.now();
+    const extractionResult = await orchestrator.extract({
+      query: VIRTUAL_POSITIONS_QUERY,
+      resultPath: 'virtualPositions.edges.node',
+      variables: {
+        catalogues,
+        dateRangeFilter: { from: startDate, to: endDate },
+      },
+      pageSize: 200,
+      maxRecords: 100000,
+      validateItem: (item: any) => !!(item.ref && item.productRef),
+    });
+    const rawRecords = extractionResult.data;
+    const extractionDuration = Date.now() - extractionStartTime;
+    log.info('✅ [EXTRACTION] Manual extraction completed', {
+      jobId,
+      totalRecords: extractionResult.stats.totalRecords,
+      totalPages: extractionResult.stats.totalPages,
+      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+      errors: extractionResult.errors ? extractionResult.errors.length : 0,
+      duration: `${extractionDuration}ms`,
+    });
+    // ? Enhanced: Completion logging with summary
+    log.info(`✅ [STATS] Extraction statistics`, {
+     totalRecords: extractionResult.stats.totalRecords,
+     totalPages: extractionResult.stats.totalPages,
+     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+     failedValidations: extractionResult.stats.failedValidations,
+     truncated: extractionResult.stats.truncated,
+     truncationReason: extractionResult.stats.truncationReason,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     duration: `${extractionDuration}ms`,
+     jobId
+    });
+    if (extractionResult.errors && extractionResult.errors.length > 0) {
+      log.warn('Non-fatal extraction errors encountered', {
+        jobId,
+        errorCount: extractionResult.errors.length,
+        sampleErrors: extractionResult.errors.slice(0, 3),
+      });
+    }
+    if (rawRecords.length === 0) {
+      log.info('ℹ️ [EXTRACTION] No records found for specified date range');
+      await jobTracker.markCompleted(jobId, {
+        recordCount: 0,
+        message: 'No records to extract',
+      });
+      const totalDuration = Date.now() - executionStartTime;
+      log.info('✅ [COMPLETE] Extraction workflow completed', {
+        duration: `${totalDuration}ms`,
+        recordCount: 0,
+      });
+      return {
+        success: true,
+        jobId,
+        recordCount: 0,
+        fileName: undefined,
+        s3Key: undefined,
+        duration: totalDuration,
+      };
+    }
+    // STEP 5/8: Transform records
+    log.info('🔄 [MAPPING] Starting field transformation');
+    const mapper = new UniversalMapper(virtualPositionsExportMapping);
+    const mappingResult = await mapper.map(rawRecords);
+    if (!mappingResult.success) {
+      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+      log.error('❌ [MAPPING] Transformation failed', {
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+      await jobTracker.markFailed(
+        jobId,
+        mappingErrors[0] || 'UniversalMapper returned unsuccessful result',
+        {
+          errors: mappingErrors,
+        }
+      );
+      return {
+        success: false,
+        jobId,
+        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+        errors: mappingErrors,
+        recommendation: 'Check mapping configuration in config/virtual-positions.export.json. Verify all source paths exist in GraphQL response.',
+      };
+    }
+    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+    const mappingErrors = mappingResult.errors || [];
+    if (mappingErrors.length > 0) {
+      log.warn('Manual extraction mapping warnings', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+    }
+    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+        jobId,
+        skippedFields: mappingResult.skippedFields,
+        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+      });
+    }
+    if (transformedRecords.length === 0) {
+      log.error('❌ [MAPPING] All records failed mapping', {
+        totalRecords: rawRecords.length,
+        errorCount: mappingErrors.length,
+      });
+      await jobTracker.markFailed(jobId, 'All records failed mapping', { errors: mappingErrors });
+      return {
+        success: false,
+        jobId,
+        error: 'All records failed mapping',
+        errors: mappingErrors,
+        recommendation: 'Review mapping errors above. Common issues: missing required fields, invalid resolver names, incorrect source paths.',
+      };
+    }
+    log.info('✅ [MAPPING] Records transformed successfully', {
+      successful: transformedRecords.length,
+      skippedRecords: rawRecords.length - transformedRecords.length,
+      errorRate: ((mappingErrors.length / rawRecords.length) * 100).toFixed(2) + '%',
+      jobId,
+    });
+    // STEP 6/8: Build JSON output
+    log.info('📄 [JSON] Building JSON output');
+    const jsonOutput = {
+      metadata: {
+        extractedAt: new Date().toISOString(),
+        recordCount: transformedRecords.length,
+        dateRangeFrom: startDate,
+        dateRangeTo: endDate,
+        jobId,
+        extractionMode: 'adhoc',
+      },
+      data: transformedRecords,
+    };
+    // Use JSONBuilder for consistent JSON generation
+    const jsonBuilder = new JSONBuilder({
+      prettyPrint: true,
+      indent: 2,
+    });
+    const jsonContent = jsonBuilder.build(jsonOutput);
+    // STEP 7/8: Upload to S3
+    log.info('☁️ [S3] Starting S3 upload');
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const fileName = `virtual-positions-adhoc-${timestamp}.json`;
+    const s3Key = `${s3Prefix}${fileName}`;
+    const s3 = new S3DataSource(
+      {
+        type: 'S3_JSON',
+        connectionId: 's3-virtual-positions-json-export',
+        name: 'S3 Virtual Positions JSON Export',
+        s3Config,
+      },
+      log
+    );
+    await s3.uploadFile(s3Key, Buffer.from(jsonContent, 'utf8'), {
+      contentType: 'application/json',
+      metadata: {
+        recordCount: String(transformedRecords.length),
+        extractedAt: new Date().toISOString(),
+        jobId,
+      },
+    });
+    log.info('✅ [S3] JSON file uploaded successfully', { s3Key, jobId });
+    // STEP 8/8: Complete job
+    await jobTracker.markCompleted(jobId, {
+      recordCount: transformedRecords.length,
+      fileName,
+      s3Key,
+      errors: mappingErrors,
+    });
+    const totalDuration = Date.now() - executionStartTime;
+    log.info('✅ [COMPLETE] Ad hoc extraction workflow completed successfully', {
+      duration: `${totalDuration}ms`,
+      extractionDuration: `${extractionDuration}ms`,
+      jobId,
+      recordCount: transformedRecords.length,
+      s3Key,
+    });
+    return {
+      success: true,
+      jobId,
+      recordCount: transformedRecords.length,
+      fileName,
+      s3Key,
+      duration: totalDuration,
+      extractionDuration,
+      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+    };
+  })
+);
+```
+### Workflow 3: Job Status Query
+**File:** `src/workflows/webhook/job-status-check.ts`
+```typescript
+import { webhook, http } from '@versori/run';
+import { Buffer } from 'node:buffer';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+export const virtualPositionsJobStatus = webhook('job-status', {
+  connection: 'virtual-positions-job-status',
+  response: {
+    mode: 'sync',
+    onSuccess: ctx =>
+      new Response(JSON.stringify(ctx.data), {
+        status: 200,
+        headers: { 'Content-Type': 'application/json' },
+      }),
+    onError: ctx =>
+      new Response(JSON.stringify({ error: ctx.data }), {
+        status: 404,
+        headers: { 'Content-Type': 'application/json' },
+      }),
+  },
+}).then(
+  http('query-job-status', async ctx => {
+    const { log, openKv, activation } = ctx;
+    const jobId = ctx.request?.query?.jobId;
+    if (!jobId) {
+      return { error: 'Missing jobId query parameter' };
+    }
+    const jobTracker = new JobTracker(openKv(':project:'), log);
+    const jobState = await jobTracker.getJob(jobId);
+    if (!jobState) {
+      return { error: `Job not found: ${jobId}` };
+    }
+    return {
+      success: true,
+      job: jobState,
+    };
+  })
+);
+```
+## Key Differences from Manual Implementation
+1. **ExtractionOrchestrator**: High-level orchestration vs manual steps
+2. **JobTracker**: Automatic job lifecycle tracking
+3. **Three Workflows**: Scheduled, ad hoc, and status query patterns
+4. **STEP X/8 Comments**: Clear orchestration progression
+5. **Overlap Buffer Pattern**: Query WITH buffer, save WITHOUT buffer
+## Production Checklist
+- [ ] Set `catalogueRefs` to correct virtual catalogue
+- [ ] Configure extraction schedule (15min for real-time, hourly for standard)
+- [ ] Set `maxRecords` based on ATP volume (100k+ for large retailers)
+- [ ] Set `pageSize` to balance throughput and memory (200-500 recommended)
+- [ ] Enable/disable `prettyPrint` based on use case (false for production)
+- [ ] Verify S3 bucket permissions and CORS configuration
+- [ ] Set up S3 lifecycle policy to archive old JSON files
+- [ ] Test with incremental mode first (validates state management)
+- [ ] Document JSON schema for API consumers
+- [ ] Test failure recovery (state rollback on error)
+- [ ] Set up alerts for extraction failures
+- [ ] Test with real-time incremental changes (15min schedule)
+- [ ] Test ad hoc manual trigger workflow
+- [ ] Test job status query workflow
+---
+---
+### Pattern 7: State Management & Date Overrides
+**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
+**How it works**:
+VersoriKV stores the last successful extraction timestamp to enable incremental sync:
+```typescript
+interface ExtractionState {
+  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
+  recordCount: number; // Number of records extracted
+  extractedAt: string; // When extraction completed
+  fileName?: string; // Generated filename
+  s3Key?: string; // S3 upload path
+  overlapBufferSeconds?: number; // Buffer configuration
+}
+```
+**State Priority Chain** (highest to lowest):
+1. **`fromDate` override** (manual date in webhook payload) - Highest priority
+2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
+3. **`fallbackStartDate`** (activation variable) - First run fallback
+**Three Scenarios**:
+#### Scenario 1: Normal Scheduled Runs (Incremental)
+```typescript
+// Payload: {} (empty - no overrides)
+// Behavior:
+// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
+// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
+// 3. Extract records updated since buffered time
+// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
+// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
+// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
+```
+**Test**:
+```bash
+# Trigger scheduled run (no payload needed)
+# State advances automatically
+curl -X POST https://workspace.versori.run/virtual-positions-extract-json-15min
+```
+#### Scenario 2: Ad-hoc Extraction WITH State Update
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
+// 3. Extract all records since 2025-01-01
+// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
+// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
+// 6. Next scheduled run starts from this new timestamp
+```
+**Use Case**: One-time catch-up extraction that advances the state pointer.
+**Test**:
+```bash
+curl -X POST https://workspace.versori.run/manual-extract-virtual-positions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "startDate": "2025-01-01T00:00:00Z",
+    "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE"
+  }'
+```
+#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z"
+// 3. Extract all records since 2025-01-01
+// 4. DO NOT update state
+// 5. Next scheduled run uses previous timestamp (unaffected)
+```
+**Use Case**: Historical backfill or testing without affecting incremental sync.
+**Test**:
+```bash
+curl -X POST https://workspace.versori.run/manual-extract-virtual-positions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "startDate": "2025-01-01T00:00:00Z",
+    "endDate": "2025-01-31T23:59:59Z",
+    "catalogueRefs": "DEFAULT_VIRTUAL_CATALOGUE"
+  }'
+```
+**Why this matters**:
+- **Incremental sync** relies on state continuity
+- **Manual overrides** allow catch-up without breaking incremental flow
+- **Overlap buffer** prevents missed records at time boundaries
+- **State isolation** lets you test/backfill without affecting production sync
+---
+### Pattern 8: Optional GraphQL Query Logging
+**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
+**When to use**:
+- ✅ Debugging pagination issues
+- ✅ Verifying query variables (dates, filters, limits)
+- ✅ Development and testing
+- ❌ Production (verbose logs, potential secrets in variables)
+**How to enable**:
+Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
+**Implementation**:
+```typescript
+// In your extraction workflow
+const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Query Debug', {
+    query: VIRTUAL_POSITIONS_QUERY,
+    variables: {
+      catalogues,
+      dateRangeFilter,
+      first: pageSize,
+      after: null, // First page
+    },
+    pagination: {
+      pageSize,
+      maxRecords,
+      currentPage: 1,
+    },
+  });
+}
+const extractionResult = await orchestrator.extract({
+  query: VIRTUAL_POSITIONS_QUERY,
+  resultPath: 'virtualPositions.edges.node',
+  variables: {
+    catalogues,
+    dateRangeFilter,
+  },
+  pageSize,
+  maxRecords,
+});
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Response Debug', {
+    totalRecords: extractionResult.stats.totalRecords,
+    totalPages: extractionResult.stats.totalPages,
+    truncated: extractionResult.stats.truncated,
+    truncationReason: extractionResult.stats.truncationReason,
+    firstRecordId: extractionResult.data[0]?.id,
+    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
+  });
+}
+```
+**What gets logged**:
+```json
+{
+  "level": "info",
+  "message": "GraphQL Query Debug",
+  "query": "query GetVirtualPositions($catalogues: [VirtualCatalogueKey], $dateRangeFilter: DateRange, ...)",
+  "variables": {
+    "catalogues": [{ "ref": "DEFAULT_VIRTUAL_CATALOGUE" }],
+    "dateRangeFilter": "2025-01-22T09:59:00Z",
+    "first": 200,
+    "after": null
+  },
+  "pagination": {
+    "pageSize": 200,
+    "maxRecords": 100000,
+    "currentPage": 1
+  }
+}
+```
+**Versori Environment Variables**:
+Add to activation settings:
+```json
+{
+  "DEBUG_GRAPHQL": "true"
+}
+```
+**Testing**:
+```bash
+# Enable debug logging
+curl -X POST https://workspace.versori.run/virtual-positions-extract-json-15min
+# Check Versori logs for "GraphQL Query Debug" entries
+# Verify query structure and variables are correct
+```
+**Sample Debug Output**:
+```
+[INFO] GraphQL Query Debug
+  query: "query GetVirtualPositions($catalogues: [VirtualCatalogueKey], $dateRangeFilter: DateRange, ...)"
+  variables: { catalogues: [{ ref: "DEFAULT_VIRTUAL_CATALOGUE" }], dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
+  pagination: { pageSize: 200, maxRecords: 100000, currentPage: 1 }
+[INFO] Extraction complete
+  totalRecords: 850
+  totalPages: 5
+  truncated: false
+[INFO] GraphQL Response Debug
+  totalRecords: 850
+  totalPages: 5
+  truncated: false
+  truncationReason: undefined
+  firstRecordId: "vp_001"
+  lastRecordId: "vp_850"
+```
+**Key Benefits**:
+- Quickly identify pagination configuration issues
+- Verify date filters are applied correctly
+- Debug "no records found" scenarios
+- Validate ExtractionOrchestrator variable injection
+**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
+---
+**Pattern**: Enterprise incremental extraction with overlap buffer for virtual positions (ATP) - JSON format with ExtractionOrchestrator
+**Key Learning**: Use `catalogues` (plural, array) input for the Fluent API
+**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
+**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
+**Orchestration**: ExtractionOrchestrator + JobTracker for production-grade extraction workflows
+**Schema**: virtualPositions accepts `catalogues: [VirtualCatalogueKey]` where VirtualCatalogueKey = `{ ref: String! }`
+**Use Case**: Real-time ATP updates for order management systems; use camelCase for JSON fields
+---
+### Pattern 8: Backward Pagination (Optional - Advanced)
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+**When to Use**:
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
+**GraphQL Query Requirements**:
+Your query must support backward pagination by including `$last` and `$before`:
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ✅ REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ✅ REQUIRED for backward
+    }
+  }
+}
+```
+**Implementation**:
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    // ❌ Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ✅ Enable reverse pagination
+  maxRecords: 10000,
+});
+// Records are returned in reverse chronological order
+log.info('Record order', {
+  newestRecord: result.data[0].createdAt,
+  oldestRecord: result.data[result.data.length - 1].createdAt
+});
+```
+**Key Differences from Forward Pagination**:
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
+**Important Notes**:
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+**Example: Extract Latest 1000 Orders**
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+**When to Use Forward vs Backward**:
+```typescript
+// ✅ Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+// ✅ Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+**Pagination Variables Reference**:
+| Variable | Forward     | Backward    | Injected By  | Notes                    |
+| -------- | ----------- | ----------- | ------------ | ------------------------ |
+| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
+| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
+| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
+| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
+**Common Mistakes to Avoid**:
+```typescript
+// ❌ WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, // ❌ Orchestrator will override this
+    before: cursor, // ❌ Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+// ✅ CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ✅ Your business variables only
+  },
+  pageSize: 200, // ✅ Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+#### Optional: Reverse Pagination
+- Forward default; reverse optional with $last/$before + pageInfo.hasPreviousPage.
+GraphQL:
+```graphql
+query GetVirtualPositionsBackward($last: Int!, $before: String) {
+  virtualPositions(last: $last, before: $before) {
+    edges {
+      cursor
+      node {
+        id
+        ref
+        updatedOn
+      }
+    }
+    pageInfo {
+      hasPreviousPage
+    }
+  }
+}
+```
+SDK:
+```typescript
+await orchestrator.extract({
+  query: VIRTUAL_POSITIONS_BACKWARD_QUERY,
+  resultPath: 'virtualPositions.edges.node',
+  variables: {},
+  pageSize,
+  direction: 'backward',
+});
+```
+---
+## Testing Checklist
+**Before production deployment:**
+### 1. Schema Validation
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/virtual-positions.export.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/virtual-positions.export.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+### 2. Extraction Testing
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+### 3. Mapping Testing
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+### 4. JSON Generation Testing
+- [ ] Verify JSON structure matches expected format
+- [ ] Test JSON validation against schema (if applicable)
+- [ ] Verify proper nesting and structure
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test special character escaping
+### 5. S3 Upload Testing
+- [ ] Test S3 connection and authentication
+- [ ] Verify file upload to correct bucket and path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify S3 object metadata
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify file permissions and ACLs
+### 6. State Management Testing
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+### 7. Job Tracking Testing
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+### 8. Error Handling Testing
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with S3 connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+### 9. Staging Environment Testing
+- [ ] Run full extraction in staging environment
+- [ ] Verify JSON file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+### 10. Integration Testing
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger → extract → transform → upload → verify file
+---
+## Monitoring & Alerting
+### Success Response Example
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_VP_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "virtual-positions-2025-11-02T14-00-00-000Z.json",
+  "s3Path": "s3://bucket/virtual-positions/virtual-positions-2025-11-02T14-00-00-000Z.json",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+### Error Response Example
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_VP_20251102_140500_xyz789",
+  "error": "S3 upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "s3_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+### Key Metrics to Track
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+  // File Generation
+  fileSizeMB: (jsonContent.length / (1024 * 1024)).toFixed(2),
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+log.info('Extraction metrics', metrics);
+```
+### Alert Thresholds
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+### Monitoring Dashboard Queries
+**Versori Platform Logs Query:**
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+# High error rates
+errorRate:>5
+# State management issues
+stateUpdated:false AND success:true
+```
+### Common Issues and Solutions
+**Issue**: "Extraction timeout after 10 minutes"
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+**Issue**: "Mapping errors for 50% of records"
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+**Issue**: "S3 connection timeout"
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check S3 credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+**Issue**: "State not updating after successful extraction"
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+**Issue**: "First run exceeds record limits"
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+**Issue**: "Excessive duplicate records in output"
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+---
+## Troubleshooting Quick Reference
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "JSON generation failed" | Format-specific error | Check JSON generation logic, validate output |
+---