npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.53 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (495) hide show

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-virtual-positions-to-s3-csv.md CHANGED Viewed

@@ -1,2355 +1,2355 @@
----
-template_id: tpl-extract-virtual-positions-to-s3-csv
-canonical_filename: template-extraction-virtual-positions-to-s3-csv.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: s3-csv
-entity: virtual-positions
-format: csv
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
----
-# Template: Extraction - Virtual Positions to S3 CSV
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Memory Management** - Clear large result sets after processing batches
-- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
-- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
----
-## 📚 STEP 1: Load These Docs (Human Checklist)
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
----
-## 📋 Implementation Prompt
-```
-I need a Versori scheduled extractor that:
-1) Queries Fluent Commerce GraphQL for Virtual Positions (ATP) with auto-pagination
-2) Supports incremental runs via KV state (with an overlap buffer)
-3) Transforms results using UniversalMapper per mapping JSON
-4) Generates CSV and uploads to S3
-5) Tracks progress with JobTracker and exposes a job-status webhook
-6) Uses native Versori log (LoggingService removed - use native log)
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
----
-## 📋 Template Overview
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, page size/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, GraphQL selection set/filters, validators/resolvers) are adjusted in code as needed. It extracts virtual positions (available-to-promise inventory) from Fluent Commerce via GraphQL, transforms the data into CSV, and uploads the result to S3.
-### What This Template Does
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    EXTRACTION WORKFLOW                           │
-└─────────────────────────────────────────────────────────────────┘
-1. TRIGGER
-   ├─ Scheduled (Cron): Runs automatically every hour
-   ├─ Ad hoc (Webhook): Manual trigger with optional date override
-   └─ Status Query (Webhook): Check job progress
-2. EXTRACT (ExtractionOrchestrator)
-   ├─ Query Fluent GraphQL API for virtual positions
-   ├─ Auto-pagination (handles large datasets)
-   ├─ Apply date filters (incremental or manual range)
-   └─ Validate each record (optional)
-3. TRANSFORM (UniversalMapper)
-   ├─ Map GraphQL fields to CSV schema
-   ├─ Apply SDK resolvers (trim, uppercase, parseInt, etc.)
-   ├─ Extract nested data (product, location, group)
-   └─ Handle transformation errors
-4. GENERATE CSV (CSVParserService)
-   ├─ Convert transformed records to CSV
-   ├─ Include headers
-   ├─ Handle special characters
-   └─ Generate timestamped filename
-5. UPLOAD (S3DataSource)
-   ├─ Connect to S3
-   ├─ Upload CSV file with retry logic
-   ├─ Set content type and metadata
-   └─ Verify upload success
-6. TRACK JOB (JobTracker)
-   ├─ Create job with unique ID
-   ├─ Update status at each step
-   ├─ Store job result in KV
-   └─ Enable status queries via webhook
-7. UPDATE STATE (VersoriKVAdapter)
-   ├─ Calculate max updatedOn from records
-   ├─ Store timestamp for next incremental run
-   ├─ Apply overlap buffer (prevent missed records)
-   └─ Skip update if manual date override
-```
-### Key Features
-- Job tracking with status queries
-- Execution modes: scheduled, ad hoc, status query
-- Uses ExtractionOrchestrator, UniversalMapper, JobTracker, CSVParserService
-- Error handling, retry logic
-- Reusable services suitable for similar use cases
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-### 📦 Package Information
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
-```bash
-npm install @fluentcommerce/fc-connect-sdk@^0.1.39
-```
----
-**Templates are designed for direct deployment; customize via activation variables.**
----
-## 📦 SDK Imports (Verified - Versori Optimized)
-```typescript
-// ✅ VERIFIED IMPORTS - These match actual SDK exports
-import { Buffer } from 'node:buffer';
-import {
-  createClient, // Universal client factory (auto-detects Versori/Node/Deno)
-  ExtractionOrchestrator, // High-level extraction with auto-pagination
-  JobTracker, // Job status tracking in KV store
-  UniversalMapper, // Field mapping with SDK resolvers
-  CSVParserService, // CSV generation (NOT CSVBuilder!)
-  S3DataSource, // S3 operations with retry logic
-} from '@fluentcommerce/fc-connect-sdk';
-// Versori platform imports
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-**✅ VERSORI PLATFORM - Use Native Logs:**
-- Use `log` from context: `const { log } = ctx;`
-- Don't import or use LoggingService for Versori connectors
-- Native Versori logs are simpler and automatically integrated with platform monitoring
----
-## ⚙️ Activation Variables
-**Configuration is driven by activation variables - modify these instead of code:**
-```json
-{
-  "groupRef": "DEFAULT_GROUP",
-  "s3BucketName": "inventory-analytics-exports",
-  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
-  "awsSecretAccessKey": "********",
-  "awsRegion": "us-east-1",
-  "s3Prefix": "virtual-positions/daily/",
-  "fileNamePrefix": "virtualpositions",
-  "pageSize": 200,
-  "maxRecords": 100000,
-  "overlapBufferSeconds": 60,
-  "fallbackStartDate": "2024-01-01T00:00:00Z"
-}
-```
-Note: Webhook security is enforced via Versori connections. Use `connection` on webhook handlers; configure auth on the connection.
-### Variable Explanations
-| Variable               | Purpose                   | Default                    | Added in Template | Customization Hints              |
-| ---------------------- | ------------------------- | -------------------------- | ----------------- | -------------------------------- |
-| `groupRef`             | Filter by inventory group | `DEFAULT_GROUP`            | v1.0.0            | Change to filter by group        |
-| `s3BucketName`         | Target S3 bucket          | -                          | v1.0.0            | Required - your analytics bucket |
-| `awsAccessKeyId`       | AWS access key            | -                          | v1.0.0            | Required - AWS credential        |
-| `awsSecretAccessKey`   | AWS secret key            | -                          | v1.0.0            | Required - AWS credential        |
-| `awsRegion`            | AWS region                | `us-east-1`                | v1.0.0            | Match bucket region              |
-| `s3Prefix`             | S3 key prefix             | `virtual-positions/daily/` | v1.0.0            | Customize folder structure       |
-| `fileNamePrefix`       | CSV filename prefix       | `virtualpositions`         | v1.0.0            | Customize naming convention      |
-| `pageSize`             | Records per GraphQL page  | `200`                      | v1.0.0            | Increase for fewer API calls     |
-| `maxRecords`           | Total extraction limit    | `100000`                   | v1.0.0            | Safety limit - adjust for volume |
-| `overlapBufferSeconds` | Incremental safety window | `60`                       | v1.0.0            | Prevents missed records          |
-| `fallbackStartDate`    | First run start date      | `2024-01-01T00:00:00Z`     | **v1.1.0**        | **NEW:** Used when no state exists |
-**🆕 New in v1.1.0:** Added `fallbackStartDate` for first run configuration
----
-### 🔄 State Management & Incremental Sync
-**How incremental sync works:**
-1. **First Run:** Uses `DEFAULT_FALLBACK` date (2024-01-01T00:00:00Z)
-2. **Subsequent Runs:** Uses `lastVirtualPositionSync` timestamp from KV store
-3. **Overlap Buffer:** Subtracts 60 seconds to catch late-arriving records
-4. **State Update:** After successful upload, stores max `updatedOn` for next run
-**Incremental vs Manual Modes:**
-| Mode                        | When to Use          | State Update | Payload Example                                                |
-| --------------------------- | -------------------- | ------------ | -------------------------------------------------------------- |
-| **Incremental**             | Daily scheduled sync | ✅ Yes       | `{}` (empty - uses last sync)                                  |
-| **Manual Range**            | Historical backfill  | ❌ No        | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": false }` |
-| **Manual Range with State** | One-time catch-up    | ✅ Yes       | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": true }`  |
-**Why overlap buffer?**
-Records updated near the sync time might not appear in the query due to:
-- Clock drift between systems
-- Transaction timing in the database
-- GraphQL query execution timing
-The 60-second buffer ensures these edge-case records are captured in the next run, preventing data loss.
-**When to skip state update (`updateState: false`):**
-- Historical backfills (don't affect ongoing incremental sync)
-- Testing/debugging specific date ranges
-- Reprocessing old data without changing the sync pointer
----
-### 📁 S3 Path Configuration
-**Note:** Folder paths and filename patterns are configured separately.
-- **Folder Path:** Configured via `s3Prefix` activation variable (e.g., `virtual-positions/daily/`)
-- **Filename Pattern:** Configured via `fileNamePrefix` activation variable (e.g., `virtualpositions`)
-**The workflow generates files like:** `{s3Prefix}{fileNamePrefix}-{timestamp}.csv`
-**Example:** With `s3Prefix="virtual-positions/daily/"` and `fileNamePrefix="virtualpositions"`, a generated file will look like:
-```
-virtual-positions/daily/virtualpositions-2025-10-27T18-30-45Z.csv
-```
-Format may vary slightly (colons replaced; includes Z).
-**Note:** To change the upload folder, modify the `s3Prefix` activation variable (not in code).
----
-### Auto-pagination and limits (ExtractionOrchestrator)
-**What:** ExtractionOrchestrator handles GraphQL Relay cursor-based pagination automatically.
-**Why:** Prevents manual pagination loop code, handles large datasets efficiently.
-**How:** You configure `pageSize` and `maxRecords`; the orchestrator injects `$first` and `$after` variables automatically and loops until `pageInfo.hasNextPage === false` or `maxRecords` is reached.
-**Critical:** Your query MUST include `edges { cursor }` and `pageInfo { hasNextPage }` fields, or pagination will fail.
-#### Configuration Parameters
-| Parameter        | Purpose                        | Example                         | Effect                     |
-| ---------------- | ------------------------------ | ------------------------------- | -------------------------- |
-| `pageSize`       | Records per GraphQL request    | `200`                           | Controls `first` variable  |
-| `maxRecords`     | Total extraction limit         | `100000`                        | Hard stop across all pages |
-| `resultPath`     | Where records live in response | `"virtualPositions.edges.node"` | Flattening path            |
-| `validateItem()` | Optional record filter         | `(item) => !!item.ref`          | Skips invalid records      |
-#### GraphQL Query Requirements
-**Your query MUST include these pagination fields:**
-```graphql
-query GetVirtualPositions(
-  $groupRef: String
-  $dateRangeFilter: DateRange
-  $first: Int! # ← Orchestrator injects this
-  $after: String # ← Orchestrator injects this
-) {
-  virtualPositions(
-    groupRef: $groupRef
-    updatedOn: $dateRangeFilter
-    first: $first # ← Pagination page size
-    after: $after # ← Pagination cursor
-  ) {
-    edges {
-      # ← REQUIRED: Relay connection structure
-      node {
-        # ← REQUIRED: Actual records here
-        id
-        ref
-        # ... your fields
-      }
-      cursor # ← REQUIRED: Orchestrator uses this for next page
-    }
-    pageInfo {
-      # ← REQUIRED: Orchestrator checks this
-      hasNextPage # ← REQUIRED: Tells orchestrator to continue
-    }
-  }
-}
-```
----
-## 📄 Mapping Configuration
-**File:** `config/virtual-positions.export.csv.json`
-```json
-{
-  "name": "virtual-positions.export.csv",
-  "version": "1.0.0",
-  "description": "Virtual Positions → CSV Export Mapping",
-  "fields": {
-    "position_ref": {
-      "source": "ref",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "product_ref": {
-      "source": "productRef",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "group_ref": {
-      "source": "groupRef",
-      "required": false,
-      "resolver": "sdk.trim"
-    },
-    "quantity": {
-      "source": "quantity",
-      "required": true,
-      "resolver": "sdk.parseInt"
-    },
-    "position_type": {
-      "source": "type",
-      "required": true,
-      "resolver": "sdk.uppercase"
-    },
-    "status": {
-      "source": "status",
-      "required": false,
-      "resolver": "sdk.uppercase"
-    },
-    "location_ref": {
-      "source": "locationRef",
-      "required": false,
-      "resolver": "sdk.trim"
-    },
-    "created_on": {
-      "source": "createdOn",
-      "required": true,
-      "resolver": "sdk.toString"
-    },
-    "updated_on": {
-      "source": "updatedOn",
-      "required": true,
-      "resolver": "sdk.toString"
-    }
-  }
-}
-```
-**AI Customization Hints:**
-- Add fields: Copy existing field config, change `source` path
-- Remove fields: Delete field from config
-- Change resolvers: Replace `sdk.trim` with `sdk.uppercase`, etc.
-- Nested fields: Use dot notation (if available in schema)
-Note: Customize mapping by editing the JSON above; prefer built-in resolvers. See SDK Universal Mapping guide for advanced usage.
-**CRITICAL FIELD DIFFERENCES:**
-- **VirtualPosition** uses `quantity` field (NOT `onHand`)
-- **VirtualPosition** has `groupRef` (NOT `catalogue`)
-- **InventoryPosition** uses `onHand` field and has `catalogue`
----
-## Versori Workflows Structure
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-### Recommended Project Structure
-```
-virtual-positions-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-virtual-positions-extraction.ts  # Scheduled: Daily extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-virtual-positions-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts                   # Webhook: Status query
-    │
-    ├── services/
-    │   └── virtual-positions-extraction.service.ts    # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── virtual-positions.export.csv.json         # Mapping configuration
-```
-**Benefits of This Structure:**
-- ✅ Clear separation of concerns (scheduled vs webhook)
-- ✅ Easier testing and maintenance
-- ✅ Better code organization
-- ✅ Consistent naming conventions
----
-## 🔧 Complete Production Code
-The code examples below demonstrate the components of the modular structure. Organize your production code following the "Recommended Project Structure" section above with workflows separated into `scheduled/` and `webhook/` folders.
----
-### 1. Entry Point (index.ts)
-```typescript
-/**
- * Entry Point - Export all workflows for Versori platform
- *
- * This file exports all workflows to be registered with Versori.
- * Each workflow is defined in its own file for better organization.
- *
- * AI CUSTOMIZATION:
- * - Add new workflows by importing and exporting them
- * - Remove workflows by commenting out exports
- * - Change workflow names in import statements
- */
-// Scheduled workflows
-export { scheduledVirtualPositionsExtraction } from './workflows/scheduled/daily-virtual-positions-extraction';
-// Webhook workflows
-export { adhocVirtualPositionsExtraction } from './workflows/webhook/adhoc-virtual-positions-extraction';
-export { virtualPositionsJobStatus } from './workflows/webhook/job-status-check';
-```
----
-### 2. Scheduled Workflow (src/workflows/scheduled/daily-virtual-positions-extraction.ts)
-```typescript
-/**
- * Scheduled workflow: Daily virtual positions extraction to S3 CSV
- * Runs every hour at minute 0
- *
- * DELEGATION PATTERN:
- * - Workflow receives ctx from Versori
- * - Passes entire ctx to service function
- * - Service handles all business logic
- *
- * AI CUSTOMIZATION:
- * - Change schedule: Modify cron expression in schedule()
- *   Examples: '*/30 * * * *' (every 30 min), '0 2 * * *' (daily at 2 AM)
- * - Add filtering: Pass additional params to service function
- */
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeVirtualPositionExtraction } from '../../services/extraction-orchestration';
-/**
- * 🚀 WORKFLOW ENTRY POINT - Scheduled Extraction
- */
-export const scheduledVirtualPositionsExtraction = schedule(
-  'virtual-positions-scheduled',
-  '0 * * * *'  // ← CUSTOMIZE: Cron expression (every hour)
-).then(
-  http('execute-scheduled-extraction', { connection: 'fluent_commerce', validateConnection: true }, async (ctx: any) => {
-    const { log, openKv } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `VP_SCHEDULED_${Date.now()}`;
-    // 📊 Initialize job tracking
-    const tracker = new JobTracker(openKv(':project:'), log);
-    log.info('🚀 [WORKFLOW] Starting scheduled extraction', { jobId });
-    await tracker.createJob(jobId, {
-      triggeredBy: 'schedule',
-      stage: 'initialization',
-      startTime: executionStartTime,
-    });
-    await tracker.updateJob(jobId, { status: 'processing' });
-    try {
-      // DELEGATION: Pass entire ctx to service function
-      const result = await executeVirtualPositionExtraction(ctx, {
-        jobId,
-        triggeredBy: 'schedule',
-        updateState: true,  // Always update state for scheduled runs
-      });
-      const duration = Date.now() - executionStartTime;
-      if (result.success) {
-        await tracker.markCompleted(jobId, { ...result, duration });
-        log.info('✅ [WORKFLOW] Extraction completed successfully', {
-          jobId,
-          recordsExtracted: result.recordsExtracted,
-          fileName: result.fileName,
-          duration: `${duration}ms`,
-        });
-      } else {
-        await tracker.markFailed(jobId, result.error || 'Unknown error');
-        log.error('❌ [WORKFLOW] Extraction failed', { jobId, error: result.error });
-      }
-      return { success: true, jobId, duration, ...result };
-    } catch (e: any) {
-      const duration = Date.now() - executionStartTime;
-      const errorMessage = e instanceof Error ? e.message : String(e);
-      await tracker.markFailed(jobId, errorMessage);
-      log.error('❌ [WORKFLOW] Extraction failed with exception', {
-        jobId,
-        error: errorMessage,
-        stack: e instanceof Error ? e.stack : undefined,
-        duration: `${duration}ms`,
-        recommendation: 'Check connection config, credentials, and S3 permissions',
-      });
-      return { success: false, jobId, duration, error: errorMessage };
-    }
-  })
-);
----
-### 3. Ad hoc Webhook (src/workflows/webhook/adhoc-virtual-positions-extraction.ts)
-```typescript
-/**
- * Webhook workflow: Ad hoc virtual positions extraction with optional date override
- *
- * DELEGATION PATTERN:
- * - Workflow receives ctx and payload from Versori
- * - Passes entire ctx to service function
- * - Service handles all business logic
- *
- * WEBHOOK PAYLOAD EXAMPLES:
- *
- * 1. Incremental (use last sync timestamp):
- *    {}
- *
- * 2. Date range (manual override):
- *    {
- *      "fromDate": "2025-01-01T00:00:00Z",
- *      "toDate": "2025-01-31T23:59:59Z",
- *      "updateState": false
- *    }
- *
- * AI CUSTOMIZATION:
- * - Add request validation
- * - Add custom filters from payload
- */
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeVirtualPositionExtraction } from '../../services/extraction-orchestration';
-/**
- * 🚀 WORKFLOW ENTRY POINT - Ad hoc Extraction
- */
-export const adhocVirtualPositionsExtraction = webhook(
-  'virtual-positions-adhoc',
-  { connection: 'virtual-positions-adhoc', response: { mode: 'sync' } }
-).then(
-  http('execute-adhoc-extraction', { connection: 'fluent_commerce', validateConnection: true }, async (ctx: any) => {
-    const { data, log, openKv } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `VP_ADHOC_${Date.now()}`;
-    // 📊 Initialize job tracking
-    const tracker = new JobTracker(openKv(':project:'), log);
-    // Extract optional date override from webhook payload
-    const fromDate = data.fromDate as string | undefined;
-    const toDate = data.toDate as string | undefined;
-    const updateState = data.updateState === true;  // Default false
-    log.info('🚀 [WORKFLOW] Ad hoc extraction triggered via webhook', {
-      jobId,
-      hasDateOverride: !!fromDate,
-      fromDate: fromDate || 'not specified',
-      toDate: toDate || 'not specified',
-      updateState,
-    });
-    await tracker.createJob(jobId, {
-      triggeredBy: 'webhook',
-      hasDateOverride: !!fromDate,
-      fromDate,
-      toDate,
-      updateStateAfterRun: updateState,
-      stage: 'initialization',
-      startTime: executionStartTime,
-    });
-    await tracker.updateJob(jobId, { status: 'processing' });
-    try {
-      // DELEGATION: Pass entire ctx to service function
-      const result = await executeVirtualPositionExtraction(ctx, {
-        jobId,
-        triggeredBy: 'webhook',
-        fromDate,
-        toDate,
-        updateState,
-      });
-      const duration = Date.now() - executionStartTime;
-      if (result.success) {
-        await tracker.markCompleted(jobId, { ...result, duration });
-        log.info('✅ [WORKFLOW] Ad hoc extraction completed', {
-          jobId,
-          recordsExtracted: result.recordsExtracted,
-          fileName: result.fileName,
-          isManualOverride: !!fromDate,
-          stateUpdated: result.stateUpdated,
-          duration: `${duration}ms`,
-        });
-        return {
-          success: true,
-          jobId,
-          duration,
-          recordsExtracted: result.recordsExtracted,
-          fileName: result.fileName,
-          s3Path: result.s3Path,
-          statusUrl: `/webhooks/virtual-positions-job-status?jobId=${jobId}`,
-          dateRange: fromDate ? { from: fromDate, to: toDate || 'not specified', updateState } : undefined,
-        };
-      } else {
-        await tracker.markFailed(jobId, result.error || 'Unknown error');
-        log.error('❌ [WORKFLOW] Ad hoc extraction failed', { jobId, error: result.error });
-        return {
-          success: false,
-          jobId,
-          duration,
-          error: result.error,
-        };
-      }
-    } catch (e: any) {
-      const duration = Date.now() - executionStartTime;
-      const errorMessage = e instanceof Error ? e.message : String(e);
-      await tracker.markFailed(jobId, errorMessage);
-      log.error('❌ [WORKFLOW] Ad hoc extraction failed with exception', {
-        jobId,
-        error: errorMessage,
-        stack: e instanceof Error ? e.stack : undefined,
-        duration: `${duration}ms`,
-        recommendation: 'Check webhook payload format, date range validity, and S3 permissions',
-      });
-      return { success: false, jobId, duration, error: errorMessage };
-    }
-  })
-);
----
-### 4. Job Status Webhook (src/workflows/webhook/job-status-check.ts)
-```typescript
-/**
- * Webhook workflow: Query job status and progress
- *
- * QUERY EXAMPLES:
- *
- * 1. HTTP GET:
- *    GET /webhooks/virtual-positions-job-status?jobId=VP_SCHEDULED_1234567890
- *
- * 2. HTTP POST:
- *    POST /webhooks/virtual-positions-job-status
- *    { "jobId": "VP_ADHOC_1234567890" }
- *
- * AI CUSTOMIZATION:
- * - Add request validation
- * - Customize response format
- */
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-/**
- * 🚀 WORKFLOW ENTRY POINT - Job Status Query
- */
-export const virtualPositionsJobStatus = webhook(
-  'virtual-positions-job-status',
-  { connection: 'virtual-positions-job-status', response: { mode: 'sync' } }
-).then(
-  fn('query-job-status', async (ctx: any) => {
-    const { data, log, openKv } = ctx;
-    // Get jobId from query param or POST body
-    const jobId = data.jobId as string;
-    if (!jobId) {
-      log.error('❌ Job ID not provided in request');
-      return {
-        success: false,
-        error: 'Job ID is required. Provide jobId in query param or request body.',
-        recommendation: 'Include jobId in query string or POST body',
-      };
-    }
-    log.info('📊 [WORKFLOW] Querying job status', { jobId });
-    try {
-      const tracker = new JobTracker(openKv(':project:'), log);
-      const status = await tracker.getJob(jobId);
-      if (!status) {
-        log.info('❌ Job not found', { jobId });
-        return {
-          success: false,
-          error: 'Job not found',
-          jobId,
-          recommendation: 'Verify jobId is correct and job exists',
-        };
-      }
-      log.info('✅ [WORKFLOW] Job status retrieved', { jobId, status: status.status });
-      return {
-        success: true,
-        jobId,
-        ...status,
-      };
-    } catch (e: any) {
-      const errorMessage = e instanceof Error ? e.message : String(e);
-      log.error('❌ [WORKFLOW] Failed to query job status', {
-        jobId,
-        error: errorMessage,
-        stack: e instanceof Error ? e.stack : undefined,
-      });
-      return {
-        success: false,
-        jobId,
-        error: errorMessage,
-        recommendation: 'Check KV store connectivity and permissions',
-      };
-    }
-  })
-);
-```
----
-## 5. Main Orchestration Service (src/services/extraction-orchestration.ts)
-```typescript
-/**
- * MAIN ORCHESTRATION SERVICE
- *
- * This is the heart of the extraction workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Determine date range (incremental vs manual)
- * 3. Extract data using ExtractionOrchestrator
- * 4. Transform using UniversalMapper
- * 5. Generate CSV using CSVParserService
- * 6. Upload to S3
- * 7. Track job progress with JobTracker
- * 8. Update state for next run
- *
- * NAMING PATTERN (consistent across all use cases):
- * - Interface: {Entity}ExtractionParams (e.g., VirtualPositionExtractionParams)
- * - Result: {Entity}ExtractionResult (e.g., VirtualPositionExtractionResult)
- * - Main function: execute{Entity}Extraction (e.g., executeVirtualPositionExtraction)
- *
- * AI CUSTOMIZATION HINTS:
- * - Change entity: Replace "VirtualPosition" with "Order", "Product", etc.
- * - Change output: Replace CSVParserService with XMLBuilder
- * - Change destination: Replace S3DataSource with SftpDataSource
- * - Add steps: Insert new service calls between existing steps
- */
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  CSVParserService,
-  S3DataSource,
-} from '@fluentcommerce/fc-connect-sdk';
-import mappingConfig from '../../config/virtual-positions.export.csv.json' with { type: 'json' };
-/**
- * Parameters for extraction workflow
- *
- * NAMING: {Entity}ExtractionParams
- */
-export interface VirtualPositionExtractionParams {
-  jobId: string;
-  triggeredBy: 'schedule' | 'webhook';
-  fromDate?: string; // Optional: manual date override
-  toDate?: string; // Optional: manual date override
-  updateState: boolean; // Whether to update lastSync timestamp
-  // AI CUSTOMIZATION: Add filters specific to entity
-  positionTypes?: string[]; // e.g., ['AVAILABLE_TO_PROMISE']
-  groupRef?: string; // e.g., 'DEFAULT_GROUP'
-}
-/**
- * Result from extraction workflow
- *
- * NAMING: {Entity}ExtractionResult
- */
-export interface VirtualPositionExtractionResult {
-  success: boolean;
-  jobId: string;
-  recordsExtracted: number;
-  fileName?: string;
-  s3Path?: string;
-  error?: string;
-  errors?: any[];
-  isManualOverride?: boolean;
-  stateUpdated?: boolean;
-  newTimestamp?: string;
-}
-/**
- * GraphQL Query for Virtual Positions
- *
- * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
- */
-const VIRTUAL_POSITIONS_EXTRACTION_QUERY = `
-  query GetVirtualPositions(
-    $groupRef: String
-    $dateRangeFilter: DateRange
-    $productRefs: [String!]
-    $types: [String!]
-    $first: Int!
-    $after: String
-  ) {
-    virtualPositions(
-      groupRef: $groupRef
-      updatedOn: $dateRangeFilter
-      productRef: $productRefs
-      type: $types
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          productRef
-          groupRef
-          locationRef
-          quantity
-          type
-          status
-          createdOn
-          updatedOn
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-/**
- * Query job status from KV store
- *
- * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
- */
-export async function getJobStatus(
-  kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
-  jobId: string,
-  log: any // ✅ Native Versori log from context
-): Promise<any | undefined> {
-  try {
-    const tracker = new JobTracker(kv, log);
-    return await tracker.getJob(jobId);
-  } catch (error: any) {
-    log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
- stack: error instanceof Error ? error.stack : undefined,
- errorType: error instanceof Error ? error.constructor.name : 'Error', });
-    return undefined;
-  }
-}
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * NAMING: execute{Entity}Extraction (e.g., executeVirtualPositionExtraction)
- *
- * This function implements the complete workflow in steps.
- * Each step is clearly commented for AI understanding.
- */
-export async function executeVirtualPositionExtraction(
-  ctx: any,
-  params: VirtualPositionExtractionParams
-): Promise<VirtualPositionExtractionResult> {
-  // ✅ VERSORI PLATFORM: Extract native log from context
-  const { log, openKv, activation } = ctx;
-  const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
-  // Open KV store for state management and job tracking
-  // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
-  // ✅ Pass native log to JobTracker
-  const kv = openKv(':project:');
-  const tracker = new JobTracker(kv, log);
-  try {
-    // ═══════════════════════════════════════════════════════════
-    // STEP 1/8: Initialize Job Tracking
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 1/8] Initializing job tracking', { jobId });
-    await tracker.createJob(jobId, {
-      triggeredBy,
-      hasDateOverride: !!fromDate,
-      fromDate,
-      toDate,
-      updateStateAfterRun: updateState,
-    });
-    // ═══════════════════════════════════════════════════════════
-    // STEP 2/8: Initialize Fluent Client
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 2/8] Initializing Fluent Commerce client', { jobId });
-    const client = await createClient(ctx);
-    if (!client) {
-      throw new Error('Failed to create Fluent Commerce client');
-    }
-    // ═══════════════════════════════════════════════════════════
-    // STEP 3/8: Determine Date Range
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 3/8] Determining date range for extraction', { jobId });
-    // State key for incremental sync tracking
-    // NAMING: last{Entity}Sync (e.g., lastVirtualPositionSync)
-    const STATE_KEY = 'lastVirtualPositionSync';
-    const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
-    const OVERLAP_BUFFER_SECONDS = parseInt(
-      activation.getVariable('overlapBufferSeconds') || '60',
-      10
-    );
-    let dateRangeFilter: { from?: string; to?: string } | null = null;
-    const isManualOverride = !!fromDate;
-    if (isManualOverride) {
-      // Manual date override from webhook
-      dateRangeFilter = { from: fromDate, to: toDate };
-      log.info('Using manual date override', { fromDate, toDate });
-    } else {
-      // Incremental sync - get last sync timestamp
-      const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
-      // Apply overlap buffer (prevents missed records)
-      const bufferedLastRunTime = new Date(
-        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
-      ).toISOString();
-      const effectiveEndTime = toDate || new Date().toISOString();
-      dateRangeFilter = {
-        from: bufferedLastRunTime,
-        to: effectiveEndTime, // End of extraction window
-      };
-      log.info('Using incremental sync with overlap buffer', {
-        rawLastRunTime,
-        bufferedLastRunTime,
-        effectiveEndTime,
-        overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
-      });
-    }
-    // ═══════════════════════════════════════════════════════════
-    // STEP 4/8: Extract Data (ExtractionOrchestrator)
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 4/8] Extracting data from Fluent Commerce', { jobId });
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'extraction',
-      message: 'Extracting data with auto-pagination',
-    });
-    // Get groupRef from params or activation
-    const groupRef = params.groupRef || activation.getVariable('groupRef');
-    // Configure extraction
-    const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
-    const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
-    // Initialize ExtractionOrchestrator
-    const orchestrator = new ExtractionOrchestrator(client, log);
-    // ? Enhanced: Extract context for progress logging
-    const dateRangeInfo = {
-      start: dateRangeFilter?.from || 'N/A',
-      end: dateRangeFilter?.to || 'N/A',
-      groupRef: groupRef || 'none',
-      types: params.positionTypes?.join(', ') || 'all'
-    };
-    // ? Enhanced: Start logging with context
-    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
-     query: 'virtualPositions',
-     pageSize,
-     maxRecords,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     groupRef: dateRangeInfo.groupRef,
-     positionTypes: dateRangeInfo.types,
-     jobId
-    });
-    // Execute extraction with auto-pagination
-    const extractionResult = await orchestrator.extract({
-      query: VIRTUAL_POSITIONS_EXTRACTION_QUERY,
-      resultPath: 'virtualPositions.edges.node',
-      variables: {
-        groupRef,
-        dateRangeFilter,
-        types: params.positionTypes,
-        // Note: Don't include 'first' or 'after' here; orchestrator injects them
-      },
-      pageSize,
-      maxRecords,
-      // Optional: validate each record
-      validateItem: (item: any) => {
-        return !!(item.ref && item.productRef && item.quantity !== undefined);
-      },
-    });
-    const rawRecords = extractionResult.data;
-    log.info('[STEP 4/8] Extraction completed', {
-      jobId,
-      totalRecords: extractionResult.stats.totalRecords,
-      totalPages: extractionResult.stats.totalPages,
-      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-      errors: extractionResult.errors ? extractionResult.errors.length : 0,
-    });
-    // ? Enhanced: Completion logging with summary
-    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-     totalRecords: extractionResult.stats.totalRecords,
-     totalPages: extractionResult.stats.totalPages,
-     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-     failedValidations: extractionResult.stats.failedValidations,
-     truncated: extractionResult.stats.truncated,
-     truncationReason: extractionResult.stats.truncationReason,
-     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
-     jobId
-    });
-    if (extractionResult.errors && extractionResult.errors.length > 0) {
-      log.warn('Non-fatal extraction errors encountered', {
-        jobId,
-        errorCount: extractionResult.errors.length,
-        sampleErrors: extractionResult.errors.slice(0, 3),
-      });
-    }
-    // Handle empty result
-    if (rawRecords.length === 0) {
-      log.info('No records to process');
-      // Update state even with no records (prevents re-querying empty window)
-      if (updateState && !isManualOverride) {
-        await kv.set(STATE_KEY, new Date().toISOString());
-      }
-      await tracker.markCompleted(jobId, {
-        recordCount: 0,
-        message: 'No records to extract',
-      });
-      return {
-        success: true,
-        jobId,
-        recordsExtracted: 0,
-      };
-    }
-    // ═══════════════════════════════════════════════════════════
-    // STEP 5/8: Transform Data (UniversalMapper)
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 5/8] Transforming data with UniversalMapper', {
-      jobId,
-      recordCount: rawRecords.length,
-    });
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'transformation',
-      message: `Transforming ${rawRecords.length} records`,
-    });
-    const mapper = new UniversalMapper(mappingConfig);
-    const mappingResult = await mapper.map(rawRecords);
-    if (!mappingResult.success) {
-      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-      log.error('[STEP 5/8] Mapping failed - terminating job', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-      await tracker.markFailed(
-        jobId,
-        new Error(mappingErrors[0] || 'UniversalMapper returned unsuccessful result')
-      );
-      return {
-        success: false,
-        jobId,
-        recordsExtracted: 0,
-        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-      };
-    }
-    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-    const mappingErrors = mappingResult.errors || [];
-    if (mappingErrors.length > 0) {
-      log.warn('[STEP 5/8] Some records failed transformation', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-    }
-    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-        jobId,
-        skippedFields: mappingResult.skippedFields,
-        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-      });
-    }
-    if (transformedRecords.length === 0) {
-      throw new Error('All records failed transformation');
-    }
-    log.info('Transformation complete', {
-      jobId,
-      successful: transformedRecords.length,
-      skippedRecords: rawRecords.length - transformedRecords.length,
-    });
-    // ═══════════════════════════════════════════════════════════
-    // STEP 6/8: Generate CSV (CSVParserService)
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 6/8] Generating CSV file', { jobId });
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'csv_generation',
-      message: `Generating CSV for ${transformedRecords.length} records`,
-    });
-    // Initialize CSVParserService
-    const csvParser = new CSVParserService();
-    // Generate CSV content (synchronous, not async)
-    const csvContent = csvParser.stringify(transformedRecords, { headers: true });
-    // Generate filename
-    const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'virtualpositions';
-    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-    const fileName = `${fileNamePrefix}-${timestamp}.csv`;
-    log.info('CSV file generated', {
-      fileName,
-      sizeBytes: csvContent.length,
-      recordCount: transformedRecords.length,
-    });
-    // ═══════════════════════════════════════════════════════════
-    // STEP 7/8: Upload to S3 (S3DataSource)
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 7/8] Uploading to S3', { jobId, fileName });
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 's3_upload',
-      message: `Uploading ${fileName} to S3`,
-    });
-    // Get S3 configuration from activation variables
-    const s3Config = {
-      bucket: activation.getVariable('s3BucketName'),
-      region: activation.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation.getVariable('awsSecretAccessKey'),
-    };
-    const s3Prefix = activation.getVariable('s3Prefix') || 'virtual-positions/daily/';
-    // Validate S3 config
-    if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
-      throw new Error(
-        'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
-      );
-    }
-    // Initialize S3 data source
-    // ✅ VERSORI PLATFORM: Pass native log from context
-    const s3 = new S3DataSource(
-      {
-        type: 'S3_CSV',
-        connectionId: 'virtual-positions-s3',
-        name: 'Virtual Positions S3 Upload',
-        s3Config,
-      },
-      log
-    );
-    // Construct S3 key
-    const s3Key = `${s3Prefix}${fileName}`;
-    // Upload with retry logic (built into S3DataSource)
-    await s3.uploadFile(
-      s3Key,
-      Buffer.from(csvContent, 'utf-8'),
-      'text/csv',
-      {
-        recordCount: String(transformedRecords.length),
-        extractedAt: new Date().toISOString(),
-        jobId,
-      }
-    );
-    log.info('S3 upload successful', { fileName, s3Key });
-    // ═══════════════════════════════════════════════════════════
-    // STEP 8/8: Update State & Complete Job
-    // ═══════════════════════════════════════════════════════════
-    log.info('[STEP 8/8] Updating state and completing job', { jobId });
-    // Calculate new timestamp for next incremental run
-    let newTimestamp: string | undefined;
-    if (updateState && !isManualOverride) {
-      // Find max updatedOn from extracted records
-      const maxUpdatedOn = rawRecords.reduce(
-        (max, record) => {
-          const recordTime = new Date(record.updatedOn).getTime();
-          return recordTime > max ? recordTime : max;
-        },
-        new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
-      );
-      newTimestamp = new Date(maxUpdatedOn).toISOString();
-      // Store new timestamp (WITHOUT buffer - buffer only applied on read)
-      await kv.set(STATE_KEY, newTimestamp);
-      log.info('State updated', {
-        oldTimestamp: dateRangeFilter?.from,
-        newTimestamp,
-      });
-    }
-    // Mark job as completed
-    await tracker.markCompleted(jobId, {
-      recordCount: transformedRecords.length,
-      fileName,
-      s3Key,
-      errorCount: mappingErrors.length,
-      isManualOverride,
-      stateUpdated: updateState && !isManualOverride,
-      newTimestamp,
-    });
-    return {
-      success: true,
-      jobId,
-      recordsExtracted: transformedRecords.length,
-      fileName,
-      s3Path: s3Key,
-      isManualOverride,
-      stateUpdated: updateState && !isManualOverride,
-      newTimestamp,
-      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-    };
-  } catch (error: any) {
-    log.error('Extraction workflow failed', {
-      jobId,
-      message: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      errorType: error instanceof Error ? error.constructor.name : 'Error',
-    });
-    // Mark job as failed
-    await tracker.markFailed(jobId, error);
-    return {
-      success: false,
-      jobId,
-      recordsExtracted: 0,
-      message: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      errorType: error instanceof Error ? error.constructor.name : 'Error',
-    };
-  }
-}
-```
----
-## 6. Utility Functions (src/utils/job-id-generator.ts)
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking extraction workflows
- *
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
- * Example: SCHEDULED_VP_20251027_183045_a1b2c3
- */
-/**
- * Generate unique job ID
- *
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
- * @param entity - Entity abbreviation (VP=Virtual Positions, IP, ORD, PRD)
- * @returns Unique job ID string
- */
-export function generateJobId(type: string, entity: string): string {
-  const now = new Date();
-  // Format: YYYYMMDD
-  const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
-  // Format: HHMMSS
-  const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
-  // Random suffix (6 chars)
-  const randomStr = Math.random().toString(36).substring(2, 8);
-  return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
-}
-/**
- * Parse job ID components
- */
-export function parseJobId(jobId: string): {
-  type: string;
-  entity: string;
-  date: string;
-  time: string;
-  random: string;
-} | null {
-  const parts = jobId.split('_');
-  if (parts.length !== 5) {
-    return null;
-  }
-  return {
-    type: parts[0],
-    entity: parts[1],
-    date: parts[2],
-    time: parts[3],
-    random: parts[4],
-  };
-}
-```
----
-## 7. Package Configuration
-### package.json
-```json
-{
-  "name": "virtual-positions-to-s3-csv",
-  "version": "1.0.0",
-  "description": "Extract virtual positions (ATP) from Fluent Commerce and export to S3 as CSV",
-  "type": "module",
-  "main": "src/index.ts",
-  "scripts": {
-    "dev": "versori dev",
-    "build": "versori build",
-    "deploy": "versori deploy"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-### tsconfig.json
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
----
-## 8. Deployment Instructions
-### Deploy to Versori
-```bash
-# 1. Install dependencies
-npm install
-# 2. Test locally (if using Versori CLI)
-npm run dev
-# 3. Deploy to Versori platform
-npm run deploy
-```
-### Configure Activation Variables
-In Versori platform settings, configure:
-```json
-{
-  "groupRef": "DEFAULT_GROUP",
-  "s3BucketName": "inventory-analytics-exports",
-  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
-  "awsSecretAccessKey": "********",
-  "awsRegion": "us-east-1",
-  "s3Prefix": "virtual-positions/daily/",
-  "fileNamePrefix": "virtualpositions",
-  "pageSize": 200,
-  "maxRecords": 100000,
-  "overlapBufferSeconds": 60
-}
-```
----
-## 9. Testing
-### Test Scheduled Extraction
-The scheduled workflow runs automatically based on cron schedule.
-**Check logs:**
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client
-[STEP 3/8] Determining date range for extraction
-[STEP 4/8] Extracting data from Fluent Commerce
-[STEP 5/8] Transforming data with UniversalMapper
-[STEP 6/8] Generating CSV file
-[STEP 7/8] Uploading to S3
-[STEP 8/8] Updating state and completing job
-```
-### Test Ad hoc Extraction
-```bash
-# Incremental (uses last sync timestamp)
-curl -X POST https://api.versori.com/webhooks/virtual-positions-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{}'
-# Date range override
-curl -X POST https://api.versori.com/webhooks/virtual-positions-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-31T23:59:59Z",
-    "updateState": false
-  }'
-```
-### Test Job Status Query
-```bash
-curl -X POST https://api.versori.com/webhooks/virtual-positions-job-status \
-  -H "Content-Type: application/json" \
-  -d '{
-    "jobId": "ADHOC_VP_20251027_183045_abc123"
-  }'
-```
-**Response:**
-```json
-{
-  "success": true,
-  "jobId": "ADHOC_VP_20251027_183045_abc123",
-  "status": "processing",
-  "stage": "transformation",
-  "message": "Transforming 15000 records",
-  "createdAt": "2025-10-27T18:30:45.000Z",
-  "startedAt": "2025-10-27T18:30:46.000Z"
-}
-```
----
----
-### Pattern 7: State Management & Date Overrides
-**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
-**How it works**:
-VersoriKV stores the last successful extraction timestamp to enable incremental sync:
-```typescript
-interface ExtractionState {
-  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
-  recordCount: number; // Number of records extracted
-  extractedAt: string; // When extraction completed
-  fileName?: string; // Generated filename
-  s3Key?: string; // S3 upload path
-  overlapBufferSeconds?: number; // Buffer configuration
-}
-```
-**State Priority Chain** (highest to lowest):
-1. **`fromDate` override** (manual date in webhook payload) - Highest priority
-2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
-3. **`fallbackStartDate`** (activation variable) - First run fallback
-**Three Scenarios**:
-#### Scenario 1: Normal Scheduled Runs (Incremental)
-```typescript
-// Payload: {} (empty - no overrides)
-// Behavior:
-// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
-// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
-// 3. Extract records updated since buffered time
-// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
-// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
-// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
-```
-**Test**:
-```bash
-# Trigger scheduled run (no payload needed)
-# State advances automatically
-curl -X POST https://workspace.versori.run/virtual-positions-scheduled
-```
-#### Scenario 2: Ad-hoc Extraction WITH State Update
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
-// 3. Extract all records since 2025-01-01
-// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
-// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
-// 6. Next scheduled run starts from this new timestamp
-```
-**Use Case**: One-time catch-up extraction that advances the state pointer.
-**Test**:
-```bash
-curl -X POST https://workspace.versori.run/virtual-positions-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "updateState": true
-  }'
-```
-#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
-```typescript
-// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
-// Behavior:
-// 1. Ignore stored state
-// 2. Use fromDate: "2025-01-01T00:00:00Z"
-// 3. Extract all records since 2025-01-01
-// 4. DO NOT update state
-// 5. Next scheduled run uses previous timestamp (unaffected)
-```
-**Use Case**: Historical backfill or testing without affecting incremental sync.
-**Test**:
-```bash
-curl -X POST https://workspace.versori.run/virtual-positions-adhoc \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-31T23:59:59Z",
-    "updateState": false
-  }'
-```
-**Why this matters**:
-- **Incremental sync** relies on state continuity
-- **Manual overrides** allow catch-up without breaking incremental flow
-- **Overlap buffer** prevents missed records at time boundaries
-- **State isolation** lets you test/backfill without affecting production sync
----
-### Pattern 8: Optional GraphQL Query Logging
-**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
-**When to use**:
-- ✅ Debugging pagination issues
-- ✅ Verifying query variables (dates, filters, limits)
-- ✅ Development and testing
-- ❌ Production (verbose logs, potential secrets in variables)
-**How to enable**:
-Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
-**Implementation**:
-```typescript
-// In your extraction workflow
-const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Query Debug', {
-    query: VIRTUAL_POSITIONS_QUERY,
-    variables: {
-      groupRef,
-      dateRangeFilter,
-      first: pageSize,
-      after: null, // First page
-    },
-    pagination: {
-      pageSize,
-      maxRecords,
-      currentPage: 1,
-    },
-  });
-}
-const extractionResult = await orchestrator.extract({
-  query: VIRTUAL_POSITIONS_QUERY,
-  resultPath: 'virtualPositions.edges.node',
-  variables: {
-    groupRef,
-    dateRangeFilter,
-  },
-  pageSize,
-  maxRecords,
-});
-if (DEBUG_GRAPHQL) {
-  log.info('GraphQL Response Debug', {
-    totalRecords: extractionResult.stats.totalRecords,
-    totalPages: extractionResult.stats.totalPages,
-    truncated: extractionResult.stats.truncated,
-    truncationReason: extractionResult.stats.truncationReason,
-    firstRecordId: extractionResult.data[0]?.id,
-    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
-  });
-}
-```
-**What gets logged**:
-```json
-{
-  "level": "info",
-  "message": "GraphQL Query Debug",
-  "query": "query GetVirtualPositions($groupRef: String, $dateRangeFilter: DateRange, ...)",
-  "variables": {
-    "groupRef": "DEFAULT_GROUP",
-    "dateRangeFilter": "2025-01-22T09:59:00Z",
-    "first": 200,
-    "after": null
-  },
-  "pagination": {
-    "pageSize": 200,
-    "maxRecords": 100000,
-    "currentPage": 1
-  }
-}
-```
-**Versori Environment Variables**:
-Add to activation settings:
-```json
-{
-  "DEBUG_GRAPHQL": "true"
-}
-```
-**Testing**:
-```bash
-# Enable debug logging
-curl -X POST https://workspace.versori.run/virtual-positions-scheduled
-# Check Versori logs for "GraphQL Query Debug" entries
-# Verify query structure and variables are correct
-```
-**Sample Debug Output**:
-```
-[INFO] GraphQL Query Debug
-  query: "query GetVirtualPositions($groupRef: String, $dateRangeFilter: DateRange, ...)"
-  variables: { groupRef: "DEFAULT_GROUP", dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
-  pagination: { pageSize: 200, maxRecords: 100000, currentPage: 1 }
-[INFO] Extraction complete
-  totalRecords: 850
-  totalPages: 5
-  truncated: false
-[INFO] GraphQL Response Debug
-  totalRecords: 850
-  totalPages: 5
-  truncated: false
-  truncationReason: undefined
-  firstRecordId: "vp_001"
-  lastRecordId: "vp_850"
-```
-**Key Benefits**:
-- Quickly identify pagination configuration issues
-- Verify date filters are applied correctly
-- Debug "no records found" scenarios
-- Validate ExtractionOrchestrator variable injection
-**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
----
-## 10. Troubleshooting
-**Issue**: "No records extracted"
-- Check dateRange (manual override vs incremental)
-- Check groupRef filter
-**Issue**: "S3 upload failed"
-- Job fails; state not advanced
-- Next run retries same window
-- Check S3 credentials and bucket permissions
-**Issue**: "GraphQL pagination error"
-- Ensure edges.cursor and pageInfo.hasNextPage are in query
-**Issue**: "Memory pressure"
-- Lower pageSize or maxRecords
-- Consider file splitting for large extractions
----
-## 11. Replication Checklist
-**To replicate this template for other entities/formats:**
-1. **File Naming:** Replace `virtual-positions`, `VP`, `VirtualPosition` with your entity name
-2. **GraphQL Query:** Update query constant and field selection to match your entity schema
-3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
-4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
-5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
-6. **State Key:** Update KV key (e.g., `lastOrderSync`)
-7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
-8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
-9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
----
-**Pattern**: Enterprise incremental extraction with overlap buffer for virtual positions (ATP)
-**Key Learning**: Use `groupRef` (string, singular) input for filtering virtual positions
-**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
-**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
-**SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
-**Field Difference**: VirtualPosition uses `quantity` field (NOT `onHand` like InventoryPosition)
----
-### Pattern 8: Backward Pagination (Optional - Advanced)
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-**When to Use**:
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
-- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
-**GraphQL Query Requirements**:
-Your query must support backward pagination by including `$last` and `$before`:
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ✅ REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ✅ REQUIRED for backward
-    }
-  }
-}
-```
-**Implementation**:
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    // ❌ Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ✅ Enable reverse pagination
-  maxRecords: 10000,
-});
-// Records are returned in reverse chronological order
-console.log(result.data[0].createdAt); // Newest
-console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
-```
-**Key Differences from Forward Pagination**:
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
-**Important Notes**:
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-**Example: Extract Latest 1000 Orders**
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-**When to Use Forward vs Backward**:
-```typescript
-// ✅ Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-// ✅ Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-**Pagination Variables Reference**:
-| Variable | Forward     | Backward    | Injected By  | Notes                    |
-| -------- | ----------- | ----------- | ------------ | ------------------------ |
-| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
-| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
-| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
-| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
-**Common Mistakes to Avoid**:
-```typescript
-// ❌ WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, // ❌ Orchestrator will override this
-    before: cursor, // ❌ Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-// ✅ CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ✅ Your business variables only
-  },
-  pageSize: 200, // ✅ Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-#### Optional: Reverse Pagination
-- Default: forward ($first/$after) + pageInfo.hasNextPage.
-- Reverse: declare $last/$before; ensure pageInfo.hasPreviousPage; set direction='backward'.
-GraphQL:
-```graphql
-query GetVirtualPositionsBackward($groupRef: String, $last: Int!, $before: String) {
-  virtualPositions(groupRef: $groupRef, last: $last, before: $before) {
-    edges {
-      cursor
-      node {
-        id
-        ref
-        updatedOn
-      }
-    }
-    pageInfo {
-      hasPreviousPage
-    }
-  }
-}
-```
-SDK:
-```typescript
-await orchestrator.extract({
-  query: VIRTUAL_POSITIONS_BACKWARD_QUERY,
-  resultPath: 'virtualPositions.edges.node',
-  variables: { groupRef },
-  pageSize,
-  direction: 'backward',
-});
-```
----
-## Testing Checklist
-**Before production deployment:**
-### 1. Schema Validation
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/virtual-positions.export.csv.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/virtual-positions.export.csv.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-### 2. Extraction Testing
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-### 3. Mapping Testing
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-### 4. CSV Generation Testing
-- [ ] Verify CSV structure matches expected format
-- [ ] Test CSV validation against schema (if applicable)
-- [ ] Verify header row is present and correct
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test special character handling (commas, quotes, newlines)
-### 5. S3 Upload Testing
-- [ ] Test S3 connection and authentication
-- [ ] Verify file upload to correct bucket and path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify S3 object metadata
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify file permissions and ACLs
-### 6. State Management Testing
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-### 7. Job Tracking Testing
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-### 8. Error Handling Testing
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with S3 connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-### 9. Staging Environment Testing
-- [ ] Run full extraction in staging environment
-- [ ] Verify CSV file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-### 10. Integration Testing
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger → extract → transform → upload → verify file
----
-## Monitoring & Alerting
-### Success Response Example
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_VP_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "virtual-positions-2025-11-02T14-00-00-000Z.csv",
-  "s3Path": "s3://bucket/virtual-positions/virtual-positions-2025-11-02T14-00-00-000Z.csv",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-### Error Response Example
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_VP_20251102_140500_xyz789",
-  "error": "S3 upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "s3_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-### Key Metrics to Track
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-  // File Generation
-  fileSizeMB: (csvContent.length / (1024 * 1024)).toFixed(2),
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-log.info('Extraction metrics', metrics);
-```
-### Alert Thresholds
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-### Monitoring Dashboard Queries
-**Versori Platform Logs Query:**
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-# High error rates
-errorRate:>5
-# State management issues
-stateUpdated:false AND success:true
-```
-### Common Issues and Solutions
-**Issue**: "Extraction timeout after 10 minutes"
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-**Issue**: "Mapping errors for 50% of records"
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-**Issue**: "S3 connection timeout"
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check S3 credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-**Issue**: "State not updating after successful extraction"
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-**Issue**: "First run exceeds record limits"
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-**Issue**: "Excessive duplicate records in output"
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
----
-## Troubleshooting Quick Reference
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "CSV generation failed" | Format-specific error | Check CSV generation logic, validate output |
----
+---
+template_id: tpl-extract-virtual-positions-to-s3-csv
+canonical_filename: template-extraction-virtual-positions-to-s3-csv.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: s3-csv
+entity: virtual-positions
+format: csv
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+---
+# Template: Extraction - Virtual Positions to S3 CSV
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Memory Management** - Clear large result sets after processing batches
+- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
+- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
+---
+## 📚 STEP 1: Load These Docs (Human Checklist)
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+---
+## 📋 Implementation Prompt
+```
+I need a Versori scheduled extractor that:
+1) Queries Fluent Commerce GraphQL for Virtual Positions (ATP) with auto-pagination
+2) Supports incremental runs via KV state (with an overlap buffer)
+3) Transforms results using UniversalMapper per mapping JSON
+4) Generates CSV and uploads to S3
+5) Tracks progress with JobTracker and exposes a job-status webhook
+6) Uses native Versori log (LoggingService removed - use native log)
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+---
+## 📋 Template Overview
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, page size/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, GraphQL selection set/filters, validators/resolvers) are adjusted in code as needed. It extracts virtual positions (available-to-promise inventory) from Fluent Commerce via GraphQL, transforms the data into CSV, and uploads the result to S3.
+### What This Template Does
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    EXTRACTION WORKFLOW                           │
+└─────────────────────────────────────────────────────────────────┘
+1. TRIGGER
+   ├─ Scheduled (Cron): Runs automatically every hour
+   ├─ Ad hoc (Webhook): Manual trigger with optional date override
+   └─ Status Query (Webhook): Check job progress
+2. EXTRACT (ExtractionOrchestrator)
+   ├─ Query Fluent GraphQL API for virtual positions
+   ├─ Auto-pagination (handles large datasets)
+   ├─ Apply date filters (incremental or manual range)
+   └─ Validate each record (optional)
+3. TRANSFORM (UniversalMapper)
+   ├─ Map GraphQL fields to CSV schema
+   ├─ Apply SDK resolvers (trim, uppercase, parseInt, etc.)
+   ├─ Extract nested data (product, location, group)
+   └─ Handle transformation errors
+4. GENERATE CSV (CSVParserService)
+   ├─ Convert transformed records to CSV
+   ├─ Include headers
+   ├─ Handle special characters
+   └─ Generate timestamped filename
+5. UPLOAD (S3DataSource)
+   ├─ Connect to S3
+   ├─ Upload CSV file with retry logic
+   ├─ Set content type and metadata
+   └─ Verify upload success
+6. TRACK JOB (JobTracker)
+   ├─ Create job with unique ID
+   ├─ Update status at each step
+   ├─ Store job result in KV
+   └─ Enable status queries via webhook
+7. UPDATE STATE (VersoriKVAdapter)
+   ├─ Calculate max updatedOn from records
+   ├─ Store timestamp for next incremental run
+   ├─ Apply overlap buffer (prevent missed records)
+   └─ Skip update if manual date override
+```
+### Key Features
+- Job tracking with status queries
+- Execution modes: scheduled, ad hoc, status query
+- Uses ExtractionOrchestrator, UniversalMapper, JobTracker, CSVParserService
+- Error handling, retry logic
+- Reusable services suitable for similar use cases
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+### 📦 Package Information
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
+```bash
+npm install @fluentcommerce/fc-connect-sdk@^0.1.39
+```
+---
+**Templates are designed for direct deployment; customize via activation variables.**
+---
+## 📦 SDK Imports (Verified - Versori Optimized)
+```typescript
+// ✅ VERIFIED IMPORTS - These match actual SDK exports
+import { Buffer } from 'node:buffer';
+import {
+  createClient, // Universal client factory (auto-detects Versori/Node/Deno)
+  ExtractionOrchestrator, // High-level extraction with auto-pagination
+  JobTracker, // Job status tracking in KV store
+  UniversalMapper, // Field mapping with SDK resolvers
+  CSVParserService, // CSV generation (NOT CSVBuilder!)
+  S3DataSource, // S3 operations with retry logic
+} from '@fluentcommerce/fc-connect-sdk';
+// Versori platform imports
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+**✅ VERSORI PLATFORM - Use Native Logs:**
+- Use `log` from context: `const { log } = ctx;`
+- Don't import or use LoggingService for Versori connectors
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+---
+## ⚙️ Activation Variables
+**Configuration is driven by activation variables - modify these instead of code:**
+```json
+{
+  "groupRef": "DEFAULT_GROUP",
+  "s3BucketName": "inventory-analytics-exports",
+  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
+  "awsSecretAccessKey": "********",
+  "awsRegion": "us-east-1",
+  "s3Prefix": "virtual-positions/daily/",
+  "fileNamePrefix": "virtualpositions",
+  "pageSize": 200,
+  "maxRecords": 100000,
+  "overlapBufferSeconds": 60,
+  "fallbackStartDate": "2024-01-01T00:00:00Z"
+}
+```
+Note: Webhook security is enforced via Versori connections. Use `connection` on webhook handlers; configure auth on the connection.
+### Variable Explanations
+| Variable               | Purpose                   | Default                    | Added in Template | Customization Hints              |
+| ---------------------- | ------------------------- | -------------------------- | ----------------- | -------------------------------- |
+| `groupRef`             | Filter by inventory group | `DEFAULT_GROUP`            | v1.0.0            | Change to filter by group        |
+| `s3BucketName`         | Target S3 bucket          | -                          | v1.0.0            | Required - your analytics bucket |
+| `awsAccessKeyId`       | AWS access key            | -                          | v1.0.0            | Required - AWS credential        |
+| `awsSecretAccessKey`   | AWS secret key            | -                          | v1.0.0            | Required - AWS credential        |
+| `awsRegion`            | AWS region                | `us-east-1`                | v1.0.0            | Match bucket region              |
+| `s3Prefix`             | S3 key prefix             | `virtual-positions/daily/` | v1.0.0            | Customize folder structure       |
+| `fileNamePrefix`       | CSV filename prefix       | `virtualpositions`         | v1.0.0            | Customize naming convention      |
+| `pageSize`             | Records per GraphQL page  | `200`                      | v1.0.0            | Increase for fewer API calls     |
+| `maxRecords`           | Total extraction limit    | `100000`                   | v1.0.0            | Safety limit - adjust for volume |
+| `overlapBufferSeconds` | Incremental safety window | `60`                       | v1.0.0            | Prevents missed records          |
+| `fallbackStartDate`    | First run start date      | `2024-01-01T00:00:00Z`     | **v1.1.0**        | **NEW:** Used when no state exists |
+**🆕 New in v1.1.0:** Added `fallbackStartDate` for first run configuration
+---
+### 🔄 State Management & Incremental Sync
+**How incremental sync works:**
+1. **First Run:** Uses `DEFAULT_FALLBACK` date (2024-01-01T00:00:00Z)
+2. **Subsequent Runs:** Uses `lastVirtualPositionSync` timestamp from KV store
+3. **Overlap Buffer:** Subtracts 60 seconds to catch late-arriving records
+4. **State Update:** After successful upload, stores max `updatedOn` for next run
+**Incremental vs Manual Modes:**
+| Mode                        | When to Use          | State Update | Payload Example                                                |
+| --------------------------- | -------------------- | ------------ | -------------------------------------------------------------- |
+| **Incremental**             | Daily scheduled sync | ✅ Yes       | `{}` (empty - uses last sync)                                  |
+| **Manual Range**            | Historical backfill  | ❌ No        | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": false }` |
+| **Manual Range with State** | One-time catch-up    | ✅ Yes       | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": true }`  |
+**Why overlap buffer?**
+Records updated near the sync time might not appear in the query due to:
+- Clock drift between systems
+- Transaction timing in the database
+- GraphQL query execution timing
+The 60-second buffer ensures these edge-case records are captured in the next run, preventing data loss.
+**When to skip state update (`updateState: false`):**
+- Historical backfills (don't affect ongoing incremental sync)
+- Testing/debugging specific date ranges
+- Reprocessing old data without changing the sync pointer
+---
+### 📁 S3 Path Configuration
+**Note:** Folder paths and filename patterns are configured separately.
+- **Folder Path:** Configured via `s3Prefix` activation variable (e.g., `virtual-positions/daily/`)
+- **Filename Pattern:** Configured via `fileNamePrefix` activation variable (e.g., `virtualpositions`)
+**The workflow generates files like:** `{s3Prefix}{fileNamePrefix}-{timestamp}.csv`
+**Example:** With `s3Prefix="virtual-positions/daily/"` and `fileNamePrefix="virtualpositions"`, a generated file will look like:
+```
+virtual-positions/daily/virtualpositions-2025-10-27T18-30-45Z.csv
+```
+Format may vary slightly (colons replaced; includes Z).
+**Note:** To change the upload folder, modify the `s3Prefix` activation variable (not in code).
+---
+### Auto-pagination and limits (ExtractionOrchestrator)
+**What:** ExtractionOrchestrator handles GraphQL Relay cursor-based pagination automatically.
+**Why:** Prevents manual pagination loop code, handles large datasets efficiently.
+**How:** You configure `pageSize` and `maxRecords`; the orchestrator injects `$first` and `$after` variables automatically and loops until `pageInfo.hasNextPage === false` or `maxRecords` is reached.
+**Critical:** Your query MUST include `edges { cursor }` and `pageInfo { hasNextPage }` fields, or pagination will fail.
+#### Configuration Parameters
+| Parameter        | Purpose                        | Example                         | Effect                     |
+| ---------------- | ------------------------------ | ------------------------------- | -------------------------- |
+| `pageSize`       | Records per GraphQL request    | `200`                           | Controls `first` variable  |
+| `maxRecords`     | Total extraction limit         | `100000`                        | Hard stop across all pages |
+| `resultPath`     | Where records live in response | `"virtualPositions.edges.node"` | Flattening path            |
+| `validateItem()` | Optional record filter         | `(item) => !!item.ref`          | Skips invalid records      |
+#### GraphQL Query Requirements
+**Your query MUST include these pagination fields:**
+```graphql
+query GetVirtualPositions(
+  $groupRef: String
+  $dateRangeFilter: DateRange
+  $first: Int! # ← Orchestrator injects this
+  $after: String # ← Orchestrator injects this
+) {
+  virtualPositions(
+    groupRef: $groupRef
+    updatedOn: $dateRangeFilter
+    first: $first # ← Pagination page size
+    after: $after # ← Pagination cursor
+  ) {
+    edges {
+      # ← REQUIRED: Relay connection structure
+      node {
+        # ← REQUIRED: Actual records here
+        id
+        ref
+        # ... your fields
+      }
+      cursor # ← REQUIRED: Orchestrator uses this for next page
+    }
+    pageInfo {
+      # ← REQUIRED: Orchestrator checks this
+      hasNextPage # ← REQUIRED: Tells orchestrator to continue
+    }
+  }
+}
+```
+---
+## 📄 Mapping Configuration
+**File:** `config/virtual-positions.export.csv.json`
+```json
+{
+  "name": "virtual-positions.export.csv",
+  "version": "1.0.0",
+  "description": "Virtual Positions → CSV Export Mapping",
+  "fields": {
+    "position_ref": {
+      "source": "ref",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "product_ref": {
+      "source": "productRef",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "group_ref": {
+      "source": "groupRef",
+      "required": false,
+      "resolver": "sdk.trim"
+    },
+    "quantity": {
+      "source": "quantity",
+      "required": true,
+      "resolver": "sdk.parseInt"
+    },
+    "position_type": {
+      "source": "type",
+      "required": true,
+      "resolver": "sdk.uppercase"
+    },
+    "status": {
+      "source": "status",
+      "required": false,
+      "resolver": "sdk.uppercase"
+    },
+    "location_ref": {
+      "source": "locationRef",
+      "required": false,
+      "resolver": "sdk.trim"
+    },
+    "created_on": {
+      "source": "createdOn",
+      "required": true,
+      "resolver": "sdk.toString"
+    },
+    "updated_on": {
+      "source": "updatedOn",
+      "required": true,
+      "resolver": "sdk.toString"
+    }
+  }
+}
+```
+**AI Customization Hints:**
+- Add fields: Copy existing field config, change `source` path
+- Remove fields: Delete field from config
+- Change resolvers: Replace `sdk.trim` with `sdk.uppercase`, etc.
+- Nested fields: Use dot notation (if available in schema)
+Note: Customize mapping by editing the JSON above; prefer built-in resolvers. See SDK Universal Mapping guide for advanced usage.
+**CRITICAL FIELD DIFFERENCES:**
+- **VirtualPosition** uses `quantity` field (NOT `onHand`)
+- **VirtualPosition** has `groupRef` (NOT `catalogue`)
+- **InventoryPosition** uses `onHand` field and has `catalogue`
+---
+## Versori Workflows Structure
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+### Recommended Project Structure
+```
+virtual-positions-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-virtual-positions-extraction.ts  # Scheduled: Daily extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-virtual-positions-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts                   # Webhook: Status query
+    │
+    ├── services/
+    │   └── virtual-positions-extraction.service.ts    # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── virtual-positions.export.csv.json         # Mapping configuration
+```
+**Benefits of This Structure:**
+- ✅ Clear separation of concerns (scheduled vs webhook)
+- ✅ Easier testing and maintenance
+- ✅ Better code organization
+- ✅ Consistent naming conventions
+---
+## 🔧 Complete Production Code
+The code examples below demonstrate the components of the modular structure. Organize your production code following the "Recommended Project Structure" section above with workflows separated into `scheduled/` and `webhook/` folders.
+---
+### 1. Entry Point (index.ts)
+```typescript
+/**
+ * Entry Point - Export all workflows for Versori platform
+ *
+ * This file exports all workflows to be registered with Versori.
+ * Each workflow is defined in its own file for better organization.
+ *
+ * AI CUSTOMIZATION:
+ * - Add new workflows by importing and exporting them
+ * - Remove workflows by commenting out exports
+ * - Change workflow names in import statements
+ */
+// Scheduled workflows
+export { scheduledVirtualPositionsExtraction } from './workflows/scheduled/daily-virtual-positions-extraction';
+// Webhook workflows
+export { adhocVirtualPositionsExtraction } from './workflows/webhook/adhoc-virtual-positions-extraction';
+export { virtualPositionsJobStatus } from './workflows/webhook/job-status-check';
+```
+---
+### 2. Scheduled Workflow (src/workflows/scheduled/daily-virtual-positions-extraction.ts)
+```typescript
+/**
+ * Scheduled workflow: Daily virtual positions extraction to S3 CSV
+ * Runs every hour at minute 0
+ *
+ * DELEGATION PATTERN:
+ * - Workflow receives ctx from Versori
+ * - Passes entire ctx to service function
+ * - Service handles all business logic
+ *
+ * AI CUSTOMIZATION:
+ * - Change schedule: Modify cron expression in schedule()
+ *   Examples: '*/30 * * * *' (every 30 min), '0 2 * * *' (daily at 2 AM)
+ * - Add filtering: Pass additional params to service function
+ */
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeVirtualPositionExtraction } from '../../services/extraction-orchestration';
+/**
+ * 🚀 WORKFLOW ENTRY POINT - Scheduled Extraction
+ */
+export const scheduledVirtualPositionsExtraction = schedule(
+  'virtual-positions-scheduled',
+  '0 * * * *'  // ← CUSTOMIZE: Cron expression (every hour)
+).then(
+  http('execute-scheduled-extraction', { connection: 'fluent_commerce', validateConnection: true }, async (ctx: any) => {
+    const { log, openKv } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `VP_SCHEDULED_${Date.now()}`;
+    // 📊 Initialize job tracking
+    const tracker = new JobTracker(openKv(':project:'), log);
+    log.info('🚀 [WORKFLOW] Starting scheduled extraction', { jobId });
+    await tracker.createJob(jobId, {
+      triggeredBy: 'schedule',
+      stage: 'initialization',
+      startTime: executionStartTime,
+    });
+    await tracker.updateJob(jobId, { status: 'processing' });
+    try {
+      // DELEGATION: Pass entire ctx to service function
+      const result = await executeVirtualPositionExtraction(ctx, {
+        jobId,
+        triggeredBy: 'schedule',
+        updateState: true,  // Always update state for scheduled runs
+      });
+      const duration = Date.now() - executionStartTime;
+      if (result.success) {
+        await tracker.markCompleted(jobId, { ...result, duration });
+        log.info('✅ [WORKFLOW] Extraction completed successfully', {
+          jobId,
+          recordsExtracted: result.recordsExtracted,
+          fileName: result.fileName,
+          duration: `${duration}ms`,
+        });
+      } else {
+        await tracker.markFailed(jobId, result.error || 'Unknown error');
+        log.error('❌ [WORKFLOW] Extraction failed', { jobId, error: result.error });
+      }
+      return { success: true, jobId, duration, ...result };
+    } catch (e: any) {
+      const duration = Date.now() - executionStartTime;
+      const errorMessage = e instanceof Error ? e.message : String(e);
+      await tracker.markFailed(jobId, errorMessage);
+      log.error('❌ [WORKFLOW] Extraction failed with exception', {
+        jobId,
+        error: errorMessage,
+        stack: e instanceof Error ? e.stack : undefined,
+        duration: `${duration}ms`,
+        recommendation: 'Check connection config, credentials, and S3 permissions',
+      });
+      return { success: false, jobId, duration, error: errorMessage };
+    }
+  })
+);
+---
+### 3. Ad hoc Webhook (src/workflows/webhook/adhoc-virtual-positions-extraction.ts)
+```typescript
+/**
+ * Webhook workflow: Ad hoc virtual positions extraction with optional date override
+ *
+ * DELEGATION PATTERN:
+ * - Workflow receives ctx and payload from Versori
+ * - Passes entire ctx to service function
+ * - Service handles all business logic
+ *
+ * WEBHOOK PAYLOAD EXAMPLES:
+ *
+ * 1. Incremental (use last sync timestamp):
+ *    {}
+ *
+ * 2. Date range (manual override):
+ *    {
+ *      "fromDate": "2025-01-01T00:00:00Z",
+ *      "toDate": "2025-01-31T23:59:59Z",
+ *      "updateState": false
+ *    }
+ *
+ * AI CUSTOMIZATION:
+ * - Add request validation
+ * - Add custom filters from payload
+ */
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeVirtualPositionExtraction } from '../../services/extraction-orchestration';
+/**
+ * 🚀 WORKFLOW ENTRY POINT - Ad hoc Extraction
+ */
+export const adhocVirtualPositionsExtraction = webhook(
+  'virtual-positions-adhoc',
+  { connection: 'virtual-positions-adhoc', response: { mode: 'sync' } }
+).then(
+  http('execute-adhoc-extraction', { connection: 'fluent_commerce', validateConnection: true }, async (ctx: any) => {
+    const { data, log, openKv } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `VP_ADHOC_${Date.now()}`;
+    // 📊 Initialize job tracking
+    const tracker = new JobTracker(openKv(':project:'), log);
+    // Extract optional date override from webhook payload
+    const fromDate = data.fromDate as string | undefined;
+    const toDate = data.toDate as string | undefined;
+    const updateState = data.updateState === true;  // Default false
+    log.info('🚀 [WORKFLOW] Ad hoc extraction triggered via webhook', {
+      jobId,
+      hasDateOverride: !!fromDate,
+      fromDate: fromDate || 'not specified',
+      toDate: toDate || 'not specified',
+      updateState,
+    });
+    await tracker.createJob(jobId, {
+      triggeredBy: 'webhook',
+      hasDateOverride: !!fromDate,
+      fromDate,
+      toDate,
+      updateStateAfterRun: updateState,
+      stage: 'initialization',
+      startTime: executionStartTime,
+    });
+    await tracker.updateJob(jobId, { status: 'processing' });
+    try {
+      // DELEGATION: Pass entire ctx to service function
+      const result = await executeVirtualPositionExtraction(ctx, {
+        jobId,
+        triggeredBy: 'webhook',
+        fromDate,
+        toDate,
+        updateState,
+      });
+      const duration = Date.now() - executionStartTime;
+      if (result.success) {
+        await tracker.markCompleted(jobId, { ...result, duration });
+        log.info('✅ [WORKFLOW] Ad hoc extraction completed', {
+          jobId,
+          recordsExtracted: result.recordsExtracted,
+          fileName: result.fileName,
+          isManualOverride: !!fromDate,
+          stateUpdated: result.stateUpdated,
+          duration: `${duration}ms`,
+        });
+        return {
+          success: true,
+          jobId,
+          duration,
+          recordsExtracted: result.recordsExtracted,
+          fileName: result.fileName,
+          s3Path: result.s3Path,
+          statusUrl: `/webhooks/virtual-positions-job-status?jobId=${jobId}`,
+          dateRange: fromDate ? { from: fromDate, to: toDate || 'not specified', updateState } : undefined,
+        };
+      } else {
+        await tracker.markFailed(jobId, result.error || 'Unknown error');
+        log.error('❌ [WORKFLOW] Ad hoc extraction failed', { jobId, error: result.error });
+        return {
+          success: false,
+          jobId,
+          duration,
+          error: result.error,
+        };
+      }
+    } catch (e: any) {
+      const duration = Date.now() - executionStartTime;
+      const errorMessage = e instanceof Error ? e.message : String(e);
+      await tracker.markFailed(jobId, errorMessage);
+      log.error('❌ [WORKFLOW] Ad hoc extraction failed with exception', {
+        jobId,
+        error: errorMessage,
+        stack: e instanceof Error ? e.stack : undefined,
+        duration: `${duration}ms`,
+        recommendation: 'Check webhook payload format, date range validity, and S3 permissions',
+      });
+      return { success: false, jobId, duration, error: errorMessage };
+    }
+  })
+);
+---
+### 4. Job Status Webhook (src/workflows/webhook/job-status-check.ts)
+```typescript
+/**
+ * Webhook workflow: Query job status and progress
+ *
+ * QUERY EXAMPLES:
+ *
+ * 1. HTTP GET:
+ *    GET /webhooks/virtual-positions-job-status?jobId=VP_SCHEDULED_1234567890
+ *
+ * 2. HTTP POST:
+ *    POST /webhooks/virtual-positions-job-status
+ *    { "jobId": "VP_ADHOC_1234567890" }
+ *
+ * AI CUSTOMIZATION:
+ * - Add request validation
+ * - Customize response format
+ */
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * 🚀 WORKFLOW ENTRY POINT - Job Status Query
+ */
+export const virtualPositionsJobStatus = webhook(
+  'virtual-positions-job-status',
+  { connection: 'virtual-positions-job-status', response: { mode: 'sync' } }
+).then(
+  fn('query-job-status', async (ctx: any) => {
+    const { data, log, openKv } = ctx;
+    // Get jobId from query param or POST body
+    const jobId = data.jobId as string;
+    if (!jobId) {
+      log.error('❌ Job ID not provided in request');
+      return {
+        success: false,
+        error: 'Job ID is required. Provide jobId in query param or request body.',
+        recommendation: 'Include jobId in query string or POST body',
+      };
+    }
+    log.info('📊 [WORKFLOW] Querying job status', { jobId });
+    try {
+      const tracker = new JobTracker(openKv(':project:'), log);
+      const status = await tracker.getJob(jobId);
+      if (!status) {
+        log.info('❌ Job not found', { jobId });
+        return {
+          success: false,
+          error: 'Job not found',
+          jobId,
+          recommendation: 'Verify jobId is correct and job exists',
+        };
+      }
+      log.info('✅ [WORKFLOW] Job status retrieved', { jobId, status: status.status });
+      return {
+        success: true,
+        jobId,
+        ...status,
+      };
+    } catch (e: any) {
+      const errorMessage = e instanceof Error ? e.message : String(e);
+      log.error('❌ [WORKFLOW] Failed to query job status', {
+        jobId,
+        error: errorMessage,
+        stack: e instanceof Error ? e.stack : undefined,
+      });
+      return {
+        success: false,
+        jobId,
+        error: errorMessage,
+        recommendation: 'Check KV store connectivity and permissions',
+      };
+    }
+  })
+);
+```
+---
+## 5. Main Orchestration Service (src/services/extraction-orchestration.ts)
+```typescript
+/**
+ * MAIN ORCHESTRATION SERVICE
+ *
+ * This is the heart of the extraction workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Determine date range (incremental vs manual)
+ * 3. Extract data using ExtractionOrchestrator
+ * 4. Transform using UniversalMapper
+ * 5. Generate CSV using CSVParserService
+ * 6. Upload to S3
+ * 7. Track job progress with JobTracker
+ * 8. Update state for next run
+ *
+ * NAMING PATTERN (consistent across all use cases):
+ * - Interface: {Entity}ExtractionParams (e.g., VirtualPositionExtractionParams)
+ * - Result: {Entity}ExtractionResult (e.g., VirtualPositionExtractionResult)
+ * - Main function: execute{Entity}Extraction (e.g., executeVirtualPositionExtraction)
+ *
+ * AI CUSTOMIZATION HINTS:
+ * - Change entity: Replace "VirtualPosition" with "Order", "Product", etc.
+ * - Change output: Replace CSVParserService with XMLBuilder
+ * - Change destination: Replace S3DataSource with SftpDataSource
+ * - Add steps: Insert new service calls between existing steps
+ */
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  CSVParserService,
+  S3DataSource,
+} from '@fluentcommerce/fc-connect-sdk';
+import mappingConfig from '../../config/virtual-positions.export.csv.json' with { type: 'json' };
+/**
+ * Parameters for extraction workflow
+ *
+ * NAMING: {Entity}ExtractionParams
+ */
+export interface VirtualPositionExtractionParams {
+  jobId: string;
+  triggeredBy: 'schedule' | 'webhook';
+  fromDate?: string; // Optional: manual date override
+  toDate?: string; // Optional: manual date override
+  updateState: boolean; // Whether to update lastSync timestamp
+  // AI CUSTOMIZATION: Add filters specific to entity
+  positionTypes?: string[]; // e.g., ['AVAILABLE_TO_PROMISE']
+  groupRef?: string; // e.g., 'DEFAULT_GROUP'
+}
+/**
+ * Result from extraction workflow
+ *
+ * NAMING: {Entity}ExtractionResult
+ */
+export interface VirtualPositionExtractionResult {
+  success: boolean;
+  jobId: string;
+  recordsExtracted: number;
+  fileName?: string;
+  s3Path?: string;
+  error?: string;
+  errors?: any[];
+  isManualOverride?: boolean;
+  stateUpdated?: boolean;
+  newTimestamp?: string;
+}
+/**
+ * GraphQL Query for Virtual Positions
+ *
+ * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
+ */
+const VIRTUAL_POSITIONS_EXTRACTION_QUERY = `
+  query GetVirtualPositions(
+    $groupRef: String
+    $dateRangeFilter: DateRange
+    $productRefs: [String!]
+    $types: [String!]
+    $first: Int!
+    $after: String
+  ) {
+    virtualPositions(
+      groupRef: $groupRef
+      updatedOn: $dateRangeFilter
+      productRef: $productRefs
+      type: $types
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          productRef
+          groupRef
+          locationRef
+          quantity
+          type
+          status
+          createdOn
+          updatedOn
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+/**
+ * Query job status from KV store
+ *
+ * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
+ */
+export async function getJobStatus(
+  kv: any, // ✅ Versori KV (compatible with JobTracker's KVAdapter interface)
+  jobId: string,
+  log: any // ✅ Native Versori log from context
+): Promise<any | undefined> {
+  try {
+    const tracker = new JobTracker(kv, log);
+    return await tracker.getJob(jobId);
+  } catch (error: any) {
+    log.error('Failed to get job status', { jobId, message: error instanceof Error ? error.message : String(error),
+ stack: error instanceof Error ? error.stack : undefined,
+ errorType: error instanceof Error ? error.constructor.name : 'Error', });
+    return undefined;
+  }
+}
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * NAMING: execute{Entity}Extraction (e.g., executeVirtualPositionExtraction)
+ *
+ * This function implements the complete workflow in steps.
+ * Each step is clearly commented for AI understanding.
+ */
+export async function executeVirtualPositionExtraction(
+  ctx: any,
+  params: VirtualPositionExtractionParams
+): Promise<VirtualPositionExtractionResult> {
+  // ✅ VERSORI PLATFORM: Extract native log from context
+  const { log, openKv, activation } = ctx;
+  const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
+  // Open KV store for state management and job tracking
+  // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
+  // ✅ Pass native log to JobTracker
+  const kv = openKv(':project:');
+  const tracker = new JobTracker(kv, log);
+  try {
+    // ═══════════════════════════════════════════════════════════
+    // STEP 1/8: Initialize Job Tracking
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 1/8] Initializing job tracking', { jobId });
+    await tracker.createJob(jobId, {
+      triggeredBy,
+      hasDateOverride: !!fromDate,
+      fromDate,
+      toDate,
+      updateStateAfterRun: updateState,
+    });
+    // ═══════════════════════════════════════════════════════════
+    // STEP 2/8: Initialize Fluent Client
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 2/8] Initializing Fluent Commerce client', { jobId });
+    const client = await createClient(ctx);
+    if (!client) {
+      throw new Error('Failed to create Fluent Commerce client');
+    }
+    // ═══════════════════════════════════════════════════════════
+    // STEP 3/8: Determine Date Range
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 3/8] Determining date range for extraction', { jobId });
+    // State key for incremental sync tracking
+    // NAMING: last{Entity}Sync (e.g., lastVirtualPositionSync)
+    const STATE_KEY = 'lastVirtualPositionSync';
+    const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
+    const OVERLAP_BUFFER_SECONDS = parseInt(
+      activation.getVariable('overlapBufferSeconds') || '60',
+      10
+    );
+    let dateRangeFilter: { from?: string; to?: string } | null = null;
+    const isManualOverride = !!fromDate;
+    if (isManualOverride) {
+      // Manual date override from webhook
+      dateRangeFilter = { from: fromDate, to: toDate };
+      log.info('Using manual date override', { fromDate, toDate });
+    } else {
+      // Incremental sync - get last sync timestamp
+      const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
+      // Apply overlap buffer (prevents missed records)
+      const bufferedLastRunTime = new Date(
+        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
+      ).toISOString();
+      const effectiveEndTime = toDate || new Date().toISOString();
+      dateRangeFilter = {
+        from: bufferedLastRunTime,
+        to: effectiveEndTime, // End of extraction window
+      };
+      log.info('Using incremental sync with overlap buffer', {
+        rawLastRunTime,
+        bufferedLastRunTime,
+        effectiveEndTime,
+        overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
+      });
+    }
+    // ═══════════════════════════════════════════════════════════
+    // STEP 4/8: Extract Data (ExtractionOrchestrator)
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 4/8] Extracting data from Fluent Commerce', { jobId });
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'extraction',
+      message: 'Extracting data with auto-pagination',
+    });
+    // Get groupRef from params or activation
+    const groupRef = params.groupRef || activation.getVariable('groupRef');
+    // Configure extraction
+    const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
+    const maxRecords = parseInt(activation.getVariable('maxRecords') || '100000', 10);
+    // Initialize ExtractionOrchestrator
+    const orchestrator = new ExtractionOrchestrator(client, log);
+    // ? Enhanced: Extract context for progress logging
+    const dateRangeInfo = {
+      start: dateRangeFilter?.from || 'N/A',
+      end: dateRangeFilter?.to || 'N/A',
+      groupRef: groupRef || 'none',
+      types: params.positionTypes?.join(', ') || 'all'
+    };
+    // ? Enhanced: Start logging with context
+    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
+     query: 'virtualPositions',
+     pageSize,
+     maxRecords,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     groupRef: dateRangeInfo.groupRef,
+     positionTypes: dateRangeInfo.types,
+     jobId
+    });
+    // Execute extraction with auto-pagination
+    const extractionResult = await orchestrator.extract({
+      query: VIRTUAL_POSITIONS_EXTRACTION_QUERY,
+      resultPath: 'virtualPositions.edges.node',
+      variables: {
+        groupRef,
+        dateRangeFilter,
+        types: params.positionTypes,
+        // Note: Don't include 'first' or 'after' here; orchestrator injects them
+      },
+      pageSize,
+      maxRecords,
+      // Optional: validate each record
+      validateItem: (item: any) => {
+        return !!(item.ref && item.productRef && item.quantity !== undefined);
+      },
+    });
+    const rawRecords = extractionResult.data;
+    log.info('[STEP 4/8] Extraction completed', {
+      jobId,
+      totalRecords: extractionResult.stats.totalRecords,
+      totalPages: extractionResult.stats.totalPages,
+      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+      errors: extractionResult.errors ? extractionResult.errors.length : 0,
+    });
+    // ? Enhanced: Completion logging with summary
+    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+     totalRecords: extractionResult.stats.totalRecords,
+     totalPages: extractionResult.stats.totalPages,
+     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+     failedValidations: extractionResult.stats.failedValidations,
+     truncated: extractionResult.stats.truncated,
+     truncationReason: extractionResult.stats.truncationReason,
+     dateRange: `${dateRangeInfo.start} to ${dateRangeInfo.end}`,
+     jobId
+    });
+    if (extractionResult.errors && extractionResult.errors.length > 0) {
+      log.warn('Non-fatal extraction errors encountered', {
+        jobId,
+        errorCount: extractionResult.errors.length,
+        sampleErrors: extractionResult.errors.slice(0, 3),
+      });
+    }
+    // Handle empty result
+    if (rawRecords.length === 0) {
+      log.info('No records to process');
+      // Update state even with no records (prevents re-querying empty window)
+      if (updateState && !isManualOverride) {
+        await kv.set(STATE_KEY, new Date().toISOString());
+      }
+      await tracker.markCompleted(jobId, {
+        recordCount: 0,
+        message: 'No records to extract',
+      });
+      return {
+        success: true,
+        jobId,
+        recordsExtracted: 0,
+      };
+    }
+    // ═══════════════════════════════════════════════════════════
+    // STEP 5/8: Transform Data (UniversalMapper)
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 5/8] Transforming data with UniversalMapper', {
+      jobId,
+      recordCount: rawRecords.length,
+    });
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'transformation',
+      message: `Transforming ${rawRecords.length} records`,
+    });
+    const mapper = new UniversalMapper(mappingConfig);
+    const mappingResult = await mapper.map(rawRecords);
+    if (!mappingResult.success) {
+      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+      log.error('[STEP 5/8] Mapping failed - terminating job', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+      await tracker.markFailed(
+        jobId,
+        new Error(mappingErrors[0] || 'UniversalMapper returned unsuccessful result')
+      );
+      return {
+        success: false,
+        jobId,
+        recordsExtracted: 0,
+        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+      };
+    }
+    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+    const mappingErrors = mappingResult.errors || [];
+    if (mappingErrors.length > 0) {
+      log.warn('[STEP 5/8] Some records failed transformation', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+    }
+    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+        jobId,
+        skippedFields: mappingResult.skippedFields,
+        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+      });
+    }
+    if (transformedRecords.length === 0) {
+      throw new Error('All records failed transformation');
+    }
+    log.info('Transformation complete', {
+      jobId,
+      successful: transformedRecords.length,
+      skippedRecords: rawRecords.length - transformedRecords.length,
+    });
+    // ═══════════════════════════════════════════════════════════
+    // STEP 6/8: Generate CSV (CSVParserService)
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 6/8] Generating CSV file', { jobId });
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'csv_generation',
+      message: `Generating CSV for ${transformedRecords.length} records`,
+    });
+    // Initialize CSVParserService
+    const csvParser = new CSVParserService();
+    // Generate CSV content (synchronous, not async)
+    const csvContent = csvParser.stringify(transformedRecords, { headers: true });
+    // Generate filename
+    const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'virtualpositions';
+    const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+    const fileName = `${fileNamePrefix}-${timestamp}.csv`;
+    log.info('CSV file generated', {
+      fileName,
+      sizeBytes: csvContent.length,
+      recordCount: transformedRecords.length,
+    });
+    // ═══════════════════════════════════════════════════════════
+    // STEP 7/8: Upload to S3 (S3DataSource)
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 7/8] Uploading to S3', { jobId, fileName });
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 's3_upload',
+      message: `Uploading ${fileName} to S3`,
+    });
+    // Get S3 configuration from activation variables
+    const s3Config = {
+      bucket: activation.getVariable('s3BucketName'),
+      region: activation.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation.getVariable('awsSecretAccessKey'),
+    };
+    const s3Prefix = activation.getVariable('s3Prefix') || 'virtual-positions/daily/';
+    // Validate S3 config
+    if (!s3Config.bucket || !s3Config.accessKeyId || !s3Config.secretAccessKey) {
+      throw new Error(
+        'S3 configuration incomplete: missing bucket, accessKeyId, or secretAccessKey'
+      );
+    }
+    // Initialize S3 data source
+    // ✅ VERSORI PLATFORM: Pass native log from context
+    const s3 = new S3DataSource(
+      {
+        type: 'S3_CSV',
+        connectionId: 'virtual-positions-s3',
+        name: 'Virtual Positions S3 Upload',
+        s3Config,
+      },
+      log
+    );
+    // Construct S3 key
+    const s3Key = `${s3Prefix}${fileName}`;
+    // Upload with retry logic (built into S3DataSource)
+    await s3.uploadFile(
+      s3Key,
+      Buffer.from(csvContent, 'utf-8'),
+      'text/csv',
+      {
+        recordCount: String(transformedRecords.length),
+        extractedAt: new Date().toISOString(),
+        jobId,
+      }
+    );
+    log.info('S3 upload successful', { fileName, s3Key });
+    // ═══════════════════════════════════════════════════════════
+    // STEP 8/8: Update State & Complete Job
+    // ═══════════════════════════════════════════════════════════
+    log.info('[STEP 8/8] Updating state and completing job', { jobId });
+    // Calculate new timestamp for next incremental run
+    let newTimestamp: string | undefined;
+    if (updateState && !isManualOverride) {
+      // Find max updatedOn from extracted records
+      const maxUpdatedOn = rawRecords.reduce(
+        (max, record) => {
+          const recordTime = new Date(record.updatedOn).getTime();
+          return recordTime > max ? recordTime : max;
+        },
+        new Date(dateRangeFilter?.from || DEFAULT_FALLBACK).getTime()
+      );
+      newTimestamp = new Date(maxUpdatedOn).toISOString();
+      // Store new timestamp (WITHOUT buffer - buffer only applied on read)
+      await kv.set(STATE_KEY, newTimestamp);
+      log.info('State updated', {
+        oldTimestamp: dateRangeFilter?.from,
+        newTimestamp,
+      });
+    }
+    // Mark job as completed
+    await tracker.markCompleted(jobId, {
+      recordCount: transformedRecords.length,
+      fileName,
+      s3Key,
+      errorCount: mappingErrors.length,
+      isManualOverride,
+      stateUpdated: updateState && !isManualOverride,
+      newTimestamp,
+    });
+    return {
+      success: true,
+      jobId,
+      recordsExtracted: transformedRecords.length,
+      fileName,
+      s3Path: s3Key,
+      isManualOverride,
+      stateUpdated: updateState && !isManualOverride,
+      newTimestamp,
+      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+    };
+  } catch (error: any) {
+    log.error('Extraction workflow failed', {
+      jobId,
+      message: error instanceof Error ? error.message : String(error),
+      stack: error instanceof Error ? error.stack : undefined,
+      errorType: error instanceof Error ? error.constructor.name : 'Error',
+    });
+    // Mark job as failed
+    await tracker.markFailed(jobId, error);
+    return {
+      success: false,
+      jobId,
+      recordsExtracted: 0,
+      message: error instanceof Error ? error.message : String(error),
+      stack: error instanceof Error ? error.stack : undefined,
+      errorType: error instanceof Error ? error.constructor.name : 'Error',
+    };
+  }
+}
+```
+---
+## 6. Utility Functions (src/utils/job-id-generator.ts)
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking extraction workflows
+ *
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
+ * Example: SCHEDULED_VP_20251027_183045_a1b2c3
+ */
+/**
+ * Generate unique job ID
+ *
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
+ * @param entity - Entity abbreviation (VP=Virtual Positions, IP, ORD, PRD)
+ * @returns Unique job ID string
+ */
+export function generateJobId(type: string, entity: string): string {
+  const now = new Date();
+  // Format: YYYYMMDD
+  const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
+  // Format: HHMMSS
+  const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
+  // Random suffix (6 chars)
+  const randomStr = Math.random().toString(36).substring(2, 8);
+  return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
+}
+/**
+ * Parse job ID components
+ */
+export function parseJobId(jobId: string): {
+  type: string;
+  entity: string;
+  date: string;
+  time: string;
+  random: string;
+} | null {
+  const parts = jobId.split('_');
+  if (parts.length !== 5) {
+    return null;
+  }
+  return {
+    type: parts[0],
+    entity: parts[1],
+    date: parts[2],
+    time: parts[3],
+    random: parts[4],
+  };
+}
+```
+---
+## 7. Package Configuration
+### package.json
+```json
+{
+  "name": "virtual-positions-to-s3-csv",
+  "version": "1.0.0",
+  "description": "Extract virtual positions (ATP) from Fluent Commerce and export to S3 as CSV",
+  "type": "module",
+  "main": "src/index.ts",
+  "scripts": {
+    "dev": "versori dev",
+    "build": "versori build",
+    "deploy": "versori deploy"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+### tsconfig.json
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+---
+## 8. Deployment Instructions
+### Deploy to Versori
+```bash
+# 1. Install dependencies
+npm install
+# 2. Test locally (if using Versori CLI)
+npm run dev
+# 3. Deploy to Versori platform
+npm run deploy
+```
+### Configure Activation Variables
+In Versori platform settings, configure:
+```json
+{
+  "groupRef": "DEFAULT_GROUP",
+  "s3BucketName": "inventory-analytics-exports",
+  "awsAccessKeyId": "AKIAXXXXXXXXXXXX",
+  "awsSecretAccessKey": "********",
+  "awsRegion": "us-east-1",
+  "s3Prefix": "virtual-positions/daily/",
+  "fileNamePrefix": "virtualpositions",
+  "pageSize": 200,
+  "maxRecords": 100000,
+  "overlapBufferSeconds": 60
+}
+```
+---
+## 9. Testing
+### Test Scheduled Extraction
+The scheduled workflow runs automatically based on cron schedule.
+**Check logs:**
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client
+[STEP 3/8] Determining date range for extraction
+[STEP 4/8] Extracting data from Fluent Commerce
+[STEP 5/8] Transforming data with UniversalMapper
+[STEP 6/8] Generating CSV file
+[STEP 7/8] Uploading to S3
+[STEP 8/8] Updating state and completing job
+```
+### Test Ad hoc Extraction
+```bash
+# Incremental (uses last sync timestamp)
+curl -X POST https://api.versori.com/webhooks/virtual-positions-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{}'
+# Date range override
+curl -X POST https://api.versori.com/webhooks/virtual-positions-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-31T23:59:59Z",
+    "updateState": false
+  }'
+```
+### Test Job Status Query
+```bash
+curl -X POST https://api.versori.com/webhooks/virtual-positions-job-status \
+  -H "Content-Type: application/json" \
+  -d '{
+    "jobId": "ADHOC_VP_20251027_183045_abc123"
+  }'
+```
+**Response:**
+```json
+{
+  "success": true,
+  "jobId": "ADHOC_VP_20251027_183045_abc123",
+  "status": "processing",
+  "stage": "transformation",
+  "message": "Transforming 15000 records",
+  "createdAt": "2025-10-27T18:30:45.000Z",
+  "startedAt": "2025-10-27T18:30:46.000Z"
+}
+```
+---
+---
+### Pattern 7: State Management & Date Overrides
+**Use Case**: Understand how state management works with scheduled and ad-hoc extractions.
+**How it works**:
+VersoriKV stores the last successful extraction timestamp to enable incremental sync:
+```typescript
+interface ExtractionState {
+  timestamp: string; // Last run timestamp (WITHOUT overlap buffer)
+  recordCount: number; // Number of records extracted
+  extractedAt: string; // When extraction completed
+  fileName?: string; // Generated filename
+  s3Key?: string; // S3 upload path
+  overlapBufferSeconds?: number; // Buffer configuration
+}
+```
+**State Priority Chain** (highest to lowest):
+1. **`fromDate` override** (manual date in webhook payload) - Highest priority
+2. **Stored state** (`await kv.get(stateKey)`) - Normal incremental mode
+3. **`fallbackStartDate`** (activation variable) - First run fallback
+**Three Scenarios**:
+#### Scenario 1: Normal Scheduled Runs (Incremental)
+```typescript
+// Payload: {} (empty - no overrides)
+// Behavior:
+// 1. Load last timestamp from KV: "2025-01-22T10:00:00Z"
+// 2. Apply overlap buffer: "2025-01-22T09:59:00Z" (query WITH buffer)
+// 3. Extract records updated since buffered time
+// 4. Calculate MAX(updatedOn) from results: "2025-01-22T14:30:00Z"
+// 5. Save new timestamp WITHOUT buffer: "2025-01-22T14:30:00Z"
+// 6. Next run starts from "2025-01-22T14:29:00Z" (with buffer)
+```
+**Test**:
+```bash
+# Trigger scheduled run (no payload needed)
+# State advances automatically
+curl -X POST https://workspace.versori.run/virtual-positions-scheduled
+```
+#### Scenario 2: Ad-hoc Extraction WITH State Update
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": true }
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z" (no buffer applied to manual dates)
+// 3. Extract all records since 2025-01-01
+// 4. Calculate MAX(updatedOn): "2025-01-22T14:30:00Z"
+// 5. Save new timestamp: "2025-01-22T14:30:00Z" (updates state!)
+// 6. Next scheduled run starts from this new timestamp
+```
+**Use Case**: One-time catch-up extraction that advances the state pointer.
+**Test**:
+```bash
+curl -X POST https://workspace.versori.run/virtual-positions-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "updateState": true
+  }'
+```
+#### Scenario 3: Ad-hoc Extraction WITHOUT State Update
+```typescript
+// Payload: { "fromDate": "2025-01-01T00:00:00Z", "updateState": false }
+// Behavior:
+// 1. Ignore stored state
+// 2. Use fromDate: "2025-01-01T00:00:00Z"
+// 3. Extract all records since 2025-01-01
+// 4. DO NOT update state
+// 5. Next scheduled run uses previous timestamp (unaffected)
+```
+**Use Case**: Historical backfill or testing without affecting incremental sync.
+**Test**:
+```bash
+curl -X POST https://workspace.versori.run/virtual-positions-adhoc \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-31T23:59:59Z",
+    "updateState": false
+  }'
+```
+**Why this matters**:
+- **Incremental sync** relies on state continuity
+- **Manual overrides** allow catch-up without breaking incremental flow
+- **Overlap buffer** prevents missed records at time boundaries
+- **State isolation** lets you test/backfill without affecting production sync
+---
+### Pattern 8: Optional GraphQL Query Logging
+**Use Case**: Debug extraction issues by logging the exact GraphQL query sent to Fluent Commerce API.
+**When to use**:
+- ✅ Debugging pagination issues
+- ✅ Verifying query variables (dates, filters, limits)
+- ✅ Development and testing
+- ❌ Production (verbose logs, potential secrets in variables)
+**How to enable**:
+Set `DEBUG_GRAPHQL=true` environment variable in Versori activation settings.
+**Implementation**:
+```typescript
+// In your extraction workflow
+const DEBUG_GRAPHQL = activation?.getVariable('DEBUG_GRAPHQL') === 'true';
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Query Debug', {
+    query: VIRTUAL_POSITIONS_QUERY,
+    variables: {
+      groupRef,
+      dateRangeFilter,
+      first: pageSize,
+      after: null, // First page
+    },
+    pagination: {
+      pageSize,
+      maxRecords,
+      currentPage: 1,
+    },
+  });
+}
+const extractionResult = await orchestrator.extract({
+  query: VIRTUAL_POSITIONS_QUERY,
+  resultPath: 'virtualPositions.edges.node',
+  variables: {
+    groupRef,
+    dateRangeFilter,
+  },
+  pageSize,
+  maxRecords,
+});
+if (DEBUG_GRAPHQL) {
+  log.info('GraphQL Response Debug', {
+    totalRecords: extractionResult.stats.totalRecords,
+    totalPages: extractionResult.stats.totalPages,
+    truncated: extractionResult.stats.truncated,
+    truncationReason: extractionResult.stats.truncationReason,
+    firstRecordId: extractionResult.data[0]?.id,
+    lastRecordId: extractionResult.data[extractionResult.data.length - 1]?.id,
+  });
+}
+```
+**What gets logged**:
+```json
+{
+  "level": "info",
+  "message": "GraphQL Query Debug",
+  "query": "query GetVirtualPositions($groupRef: String, $dateRangeFilter: DateRange, ...)",
+  "variables": {
+    "groupRef": "DEFAULT_GROUP",
+    "dateRangeFilter": "2025-01-22T09:59:00Z",
+    "first": 200,
+    "after": null
+  },
+  "pagination": {
+    "pageSize": 200,
+    "maxRecords": 100000,
+    "currentPage": 1
+  }
+}
+```
+**Versori Environment Variables**:
+Add to activation settings:
+```json
+{
+  "DEBUG_GRAPHQL": "true"
+}
+```
+**Testing**:
+```bash
+# Enable debug logging
+curl -X POST https://workspace.versori.run/virtual-positions-scheduled
+# Check Versori logs for "GraphQL Query Debug" entries
+# Verify query structure and variables are correct
+```
+**Sample Debug Output**:
+```
+[INFO] GraphQL Query Debug
+  query: "query GetVirtualPositions($groupRef: String, $dateRangeFilter: DateRange, ...)"
+  variables: { groupRef: "DEFAULT_GROUP", dateRangeFilter: "2025-01-22T09:59:00Z", first: 200, after: null }
+  pagination: { pageSize: 200, maxRecords: 100000, currentPage: 1 }
+[INFO] Extraction complete
+  totalRecords: 850
+  totalPages: 5
+  truncated: false
+[INFO] GraphQL Response Debug
+  totalRecords: 850
+  totalPages: 5
+  truncated: false
+  truncationReason: undefined
+  firstRecordId: "vp_001"
+  lastRecordId: "vp_850"
+```
+**Key Benefits**:
+- Quickly identify pagination configuration issues
+- Verify date filters are applied correctly
+- Debug "no records found" scenarios
+- Validate ExtractionOrchestrator variable injection
+**Production Best Practice**: Disable `DEBUG_GRAPHQL` in production to reduce log volume and avoid logging sensitive data.
+---
+## 10. Troubleshooting
+**Issue**: "No records extracted"
+- Check dateRange (manual override vs incremental)
+- Check groupRef filter
+**Issue**: "S3 upload failed"
+- Job fails; state not advanced
+- Next run retries same window
+- Check S3 credentials and bucket permissions
+**Issue**: "GraphQL pagination error"
+- Ensure edges.cursor and pageInfo.hasNextPage are in query
+**Issue**: "Memory pressure"
+- Lower pageSize or maxRecords
+- Consider file splitting for large extractions
+---
+## 11. Replication Checklist
+**To replicate this template for other entities/formats:**
+1. **File Naming:** Replace `virtual-positions`, `VP`, `VirtualPosition` with your entity name
+2. **GraphQL Query:** Update query constant and field selection to match your entity schema
+3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
+4. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledOrdersExtraction`)
+5. **Service Function:** Rename main function (e.g., `executeOrderExtraction`)
+6. **State Key:** Update KV key (e.g., `lastOrderSync`)
+7. **Output Format:** For XML use `XMLBuilder`, for JSON use `JSON.stringify()`, for CSV use `CSVParserService`
+8. **Upload Destination:** For SFTP replace `S3DataSource` with `SftpDataSource` (and add `dispose()` in finally block)
+9. **Job ID Entity Code:** Update entity abbreviation in generateJobId() (e.g., 'ORD' for orders)
+---
+**Pattern**: Enterprise incremental extraction with overlap buffer for virtual positions (ATP)
+**Key Learning**: Use `groupRef` (string, singular) input for filtering virtual positions
+**Critical**: Apply 60-second overlap buffer to prevent missed records due to clock skew
+**Buffer Pattern**: Query WITH buffer (`updatedOn >= lastRunTime - 60s`), save WITHOUT buffer (`MAX(updatedOn)`)
+**SDK Services**: ExtractionOrchestrator, UniversalMapper, CSVParserService, S3DataSource, JobTracker
+**Field Difference**: VirtualPosition uses `quantity` field (NOT `onHand` like InventoryPosition)
+---
+### Pattern 8: Backward Pagination (Optional - Advanced)
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+**When to Use**:
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
+**GraphQL Query Requirements**:
+Your query must support backward pagination by including `$last` and `$before`:
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ✅ REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ✅ REQUIRED for backward
+    }
+  }
+}
+```
+**Implementation**:
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    // ❌ Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ✅ Enable reverse pagination
+  maxRecords: 10000,
+});
+// Records are returned in reverse chronological order
+console.log(result.data[0].createdAt); // Newest
+console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
+```
+**Key Differences from Forward Pagination**:
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest                  | Newest → Oldest         |
+**Important Notes**:
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+**Example: Extract Latest 1000 Orders**
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+**When to Use Forward vs Backward**:
+```typescript
+// ✅ Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+// ✅ Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+**Pagination Variables Reference**:
+| Variable | Forward     | Backward    | Injected By  | Notes                    |
+| -------- | ----------- | ----------- | ------------ | ------------------------ |
+| `first`  | ✅ Used     | ❌ Not used | Orchestrator | From `pageSize`          |
+| `after`  | ✅ Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
+| `last`   | ❌ Not used | ✅ Used     | Orchestrator | From `pageSize`          |
+| `before` | ❌ Not used | ✅ Used     | Orchestrator | From cursor (first edge) |
+**Common Mistakes to Avoid**:
+```typescript
+// ❌ WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, // ❌ Orchestrator will override this
+    before: cursor, // ❌ Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+// ✅ CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ✅ Your business variables only
+  },
+  pageSize: 200, // ✅ Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+#### Optional: Reverse Pagination
+- Default: forward ($first/$after) + pageInfo.hasNextPage.
+- Reverse: declare $last/$before; ensure pageInfo.hasPreviousPage; set direction='backward'.
+GraphQL:
+```graphql
+query GetVirtualPositionsBackward($groupRef: String, $last: Int!, $before: String) {
+  virtualPositions(groupRef: $groupRef, last: $last, before: $before) {
+    edges {
+      cursor
+      node {
+        id
+        ref
+        updatedOn
+      }
+    }
+    pageInfo {
+      hasPreviousPage
+    }
+  }
+}
+```
+SDK:
+```typescript
+await orchestrator.extract({
+  query: VIRTUAL_POSITIONS_BACKWARD_QUERY,
+  resultPath: 'virtualPositions.edges.node',
+  variables: { groupRef },
+  pageSize,
+  direction: 'backward',
+});
+```
+---
+## Testing Checklist
+**Before production deployment:**
+### 1. Schema Validation
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/virtual-positions.export.csv.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/virtual-positions.export.csv.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+### 2. Extraction Testing
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+### 3. Mapping Testing
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+### 4. CSV Generation Testing
+- [ ] Verify CSV structure matches expected format
+- [ ] Test CSV validation against schema (if applicable)
+- [ ] Verify header row is present and correct
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test special character handling (commas, quotes, newlines)
+### 5. S3 Upload Testing
+- [ ] Test S3 connection and authentication
+- [ ] Verify file upload to correct bucket and path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify S3 object metadata
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify file permissions and ACLs
+### 6. State Management Testing
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+### 7. Job Tracking Testing
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+### 8. Error Handling Testing
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with S3 connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+### 9. Staging Environment Testing
+- [ ] Run full extraction in staging environment
+- [ ] Verify CSV file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+### 10. Integration Testing
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger → extract → transform → upload → verify file
+---
+## Monitoring & Alerting
+### Success Response Example
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_VP_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "virtual-positions-2025-11-02T14-00-00-000Z.csv",
+  "s3Path": "s3://bucket/virtual-positions/virtual-positions-2025-11-02T14-00-00-000Z.csv",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+### Error Response Example
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_VP_20251102_140500_xyz789",
+  "error": "S3 upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "s3_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+### Key Metrics to Track
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+  // File Generation
+  fileSizeMB: (csvContent.length / (1024 * 1024)).toFixed(2),
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+log.info('Extraction metrics', metrics);
+```
+### Alert Thresholds
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+### Monitoring Dashboard Queries
+**Versori Platform Logs Query:**
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+# High error rates
+errorRate:>5
+# State management issues
+stateUpdated:false AND success:true
+```
+### Common Issues and Solutions
+**Issue**: "Extraction timeout after 10 minutes"
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+**Issue**: "Mapping errors for 50% of records"
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+**Issue**: "S3 connection timeout"
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check S3 credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+**Issue**: "State not updating after successful extraction"
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+**Issue**: "First run exceeds record limits"
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+**Issue**: "Excessive duplicate records in output"
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+---
+## Troubleshooting Quick Reference
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "S3 authentication failed" | Invalid credentials | Verify S3 credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "CSV generation failed" | Format-specific error | Check CSV generation logic, validate output |
+---