@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/extraction/graphql-queries/template-extraction-fulfillments-to-sftp-csv.md

@@ -1,2062 +1,2062 @@
----
-template_id: tpl-extract-fulfillments-to-sftp-csv
-canonical_filename: template-extraction-fulfillments-to-sftp-csv.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: extraction
-source: fluent-graphql
-destination: sftp-csv
-entity: fulfillments
-format: csv
-logging: versori
-status: stable
-features:
-  - memory-management
-  - enhanced-logging
-  - pagination-progress
-  - dispose-finally
----
-
-# Template: Extraction - Fulfillments to SFTP CSV
-
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Memory Management** - Clear large result sets after processing batches
-- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
-- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
-- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
-
----
-
-## 📚 STEP 1: Load These Docs (Human Checklist)
-
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
----
-
-## 📋 STEP 2: Tell Your AI (Prompt)
-
-Copy/paste this prompt into your AI tool after loading the documentation above:
-
-```
-I need a Versori scheduled extractor that:
-
-1) Queries Fluent Commerce GraphQL for Fulfillments with auto-pagination
-2) Supports incremental runs via KV state (with an overlap buffer)
-3) Transforms results using UniversalMapper per mapping JSON
-4) Generates CSV and uploads to SFTP
-5) Tracks progress with JobTracker and exposes a job-status webhook
-6) Uses native Versori log (LoggingService removed - use native log)
-
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
-
----
-
-## 📋 Template Overview
-
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP credentials, schedule, page size/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, GraphQL selection set/filters, validators/resolvers) are adjusted in code as needed. It extracts fulfillments from Fluent Commerce via GraphQL, transforms the data into CSV, and uploads the result to SFTP.
-
-### What This Template Does
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    EXTRACTION WORKFLOW                           │
-└─────────────────────────────────────────────────────────────────┘
-
-1. TRIGGER
-   ├─ Scheduled (Cron): Runs automatically every 4 hours
-   ├─ Ad hoc (Webhook): Manual trigger with optional date override
-   └─ Status Query (Webhook): Check job progress
-
-2. EXTRACT (ExtractionOrchestrator)
-   ├─ Query Fluent GraphQL API for fulfillments
-   ├─ Auto-pagination (handles large datasets)
-   ├─ Apply date filters (incremental or manual range)
-   └─ Validate each record (optional)
-
-3. TRANSFORM (UniversalMapper)
-   ├─ Map GraphQL fields to CSV schema
-   ├─ Apply SDK resolvers (trim, uppercase, formatDateShort, etc.)
-   ├─ Extract nested data (order.customer.email)
-   └─ Handle transformation errors
-
-4. GENERATE CSV (CSVParserService)
-   ├─ Convert transformed records to CSV
-   ├─ Include headers
-   ├─ Handle special characters
-   └─ Generate timestamped filename
-
-5. UPLOAD (SftpDataSource)
-   ├─ Connect to SFTP server
-   ├─ Upload CSV file with retry logic
-   ├─ Verify upload success
-   └─ CRITICAL: Call dispose() in finally block
-
-6. TRACK JOB (JobTracker)
-   ├─ Create job with unique ID
-   ├─ Update status at each step
-   ├─ Store job result in KV
-   └─ Enable status queries via webhook
-
-7. UPDATE STATE (VersoriKVAdapter)
-   ├─ Calculate max updatedOn from records
-   ├─ Store timestamp for next incremental run
-   ├─ Apply overlap buffer (prevent missed records)
-   └─ Skip update if manual date override
-```
-
-### Key Features
-
-- Job tracking with status queries
-- Execution modes: scheduled, ad hoc, status query
-- Uses ExtractionOrchestrator, UniversalMapper, JobTracker, CSVParserService
-- Error handling, retry logic
-- Reusable services suitable for similar use cases
-
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-
-### 📦 Package Information
-
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk@^0.1.39
-```
-
----
-
-**Templates are designed for direct deployment; customize via activation variables.**
-
----
-
-## 📦 SDK Imports (Verified - Versori Optimized)
-
-```typescript
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  CSVParserService,
-  SftpDataSource,
-} from '@fluentcommerce/fc-connect-sdk';
-
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-
-**? VERSORI PLATFORM - Use Native Logs:**
-
-- Use `log` from context: `const { log } = ctx;`
-- Don't import or use LoggingService for Versori connectors
-- Native Versori logs are simpler and automatically integrated with platform monitoring
-
----
-
-## ⚙️ Configuration
-
-### SFTP Connection Setup (Recommended)
-
-**? BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
-
-**Connection Configuration:**
-
-1. In Versori platform, create a connection named `versori_ftp_server`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-5. The SDK will automatically decode the credentials using:
-
-```typescript
-// Get SFTP credentials from Versori connection (Basic Auth)
-// RECOMMENDED: Use activation.connections (already decoded)
-const allConnections = ctx.activation.connections || [];
-const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-if (!sftpConn) {
-  throw new Error('SFTP connection "versori_ftp_server" not found');
-}
-
-const credential = sftpConn.credentials[0]?.credential;
-if (!credential?.data?.basicAuth) {
-  throw new Error('SFTP connection not configured with Basic Authentication');
-}
-
-const { username, password } = credential.data.basicAuth;
-// ? Already decoded - no Buffer.from() needed!
-```
-
-**Why use connections instead of activation variables?**
-
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection can be reused across workflows
-- ✅ No need to manage sensitive data in activation variables
-- ✅ Easier credential rotation
-- ✅ Better separation of concerns (config vs secrets)
-- ✅ Follows industry security best practices
-
-### Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-```json
-{
-  "retailerId": "your-retailer-id",
-  "sftpHost": "sftp.partner.com",
-  "sftpPort": 22,
-  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
-  "sftpRemotePath": "/incoming/fulfillments/",
-  "fileNamePrefix": "fulfillments",
-  "pageSize": 200,
-  "maxRecords": 10000,
-  "overlapBufferSeconds": 60,
-  "fulfillmentStatuses": "SHIPPED,DELIVERED",
-  "validateConnectionOnStart": "false"
-}
-```
-
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` connection (see above).
-
-Note: Webhook security is enforced via Versori connections. Configure auth on the connection and reference it in `webhook({ connection: '...' })`.
-
-### Variable Explanations
-
-| Variable                     | Purpose                     | Default                   | Customization Hints              |
-| ---------------------------- | --------------------------- | ------------------------- | -------------------------------- |
-| `retailerId`                 | Fluent retailer ID          | -                         | Required - your retailer ID      |
-| **SFTP Credentials**         | _From Connection_           |                           | _See connection setup above_     |
-| `sftpHost`                   | SFTP server hostname        | -                         | Required - partner SFTP server   |
-| `sftpPort`                   | SFTP server port            | `22`                      | Standard SFTP port               |
-| `sftpPrivateKey`             | SFTP private key (optional) | -                         | Alternative to password auth     |
-| `sftpRemotePath`             | SFTP upload directory       | `/incoming/fulfillments/` | Customize folder structure       |
-| `fileNamePrefix`             | CSV filename prefix         | `fulfillments`            | Customize naming convention      |
-| `pageSize`                   | Records per GraphQL page    | `200`                     | Increase for fewer API calls     |
-| `maxRecords`                 | Total extraction limit      | `50000`                   | Safety limit - adjust for volume |
-| `overlapBufferSeconds`       | Incremental safety window   | `60`                      | Prevents missed records          |
-| `fulfillmentStatuses`        | Status filter (comma-sep)   | `SHIPPED,DELIVERED`       | Filter by fulfillment status     |
-| `validateConnectionOnStart`  | Validate auth on startup    | `false`                   | `true` = fail-fast, `false` = fast startup |
-
----
-
-### 📋 State Management & Incremental Sync
-
-**How incremental sync works:**
-
-1. **First Run:** Uses `DEFAULT_FALLBACK` date (2024-01-01T00:00:00Z)
-2. **Subsequent Runs:** Uses `lastFulfillmentSync` timestamp from KV store
-3. **Overlap Buffer:** Subtracts 60 seconds to catch late-arriving records
-4. **State Update:** After successful upload, stores max `updatedOn` for next run
-
-**Incremental vs Manual Modes:**
-
-| Mode                        | When to Use          | State Update | Payload Example                                                |
-| --------------------------- | -------------------- | ------------ | -------------------------------------------------------------- |
-| **Incremental**             | Daily scheduled sync | ? Yes       | `{}` (empty - uses last sync)                                  |
-| **Manual Range**            | Historical backfill  | ❌ No        | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": false }` |
-| **Manual Range with State** | One-time catch-up    | ? Yes       | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": true }`  |
-
-**Why overlap buffer?**
-
-Records updated near the sync time might not appear in the query due to:
-
-- Clock drift between systems
-- Transaction timing in the database
-- GraphQL query execution timing
-
-The 60-second buffer ensures these edge-case records are captured in the next run, preventing data loss.
-
-**When to skip state update (`updateState: false`):**
-
-- Historical backfills (don't affect ongoing incremental sync)
-- Testing/debugging specific date ranges
-- Reprocessing old data without changing the sync pointer
-
----
-
-### 📁 SFTP Configuration
-
-**Note:** SFTP credentials and paths are configured separately.
-
-- **SFTP Host:** Configured via `sftpHost` activation variable
-- **SFTP Path:** Configured via `sftpRemotePath` activation variable (e.g., `/incoming/fulfillments/`)
-- **Filename Pattern:** Configured via `fileNamePrefix` activation variable (e.g., `fulfillments`)
-
-**The workflow generates files like:** `{sftpRemotePath}{fileNamePrefix}-{timestamp}.csv`
-
-**Example:** With `sftpRemotePath="/incoming/fulfillments/"` and `fileNamePrefix="fulfillments"`, a generated file will look like:
-
-```
-/incoming/fulfillments/fulfillments-2025-10-27T18-30-45Z.csv
-```
-
-**Note:** To change the upload folder, modify the `sftpRemotePath` activation variable (not in code).
-
-**CRITICAL:** Always call `sftp.dispose()` in a finally block to close connections properly.
-
----
-
-### Auto-pagination and limits (ExtractionOrchestrator)
-
-**What:** ExtractionOrchestrator handles GraphQL Relay cursor-based pagination automatically.
-
-**Why:** Prevents manual pagination loop code, handles large datasets efficiently.
-
-**How:** You configure `pageSize` and `maxRecords`; the orchestrator injects `$first` and `$after` variables automatically and loops until `pageInfo.hasNextPage === false` or `maxRecords` is reached.
-
-**Critical:** Your query MUST include `edges { cursor }` and `pageInfo { hasNextPage }` fields, or pagination will fail.
-
-#### Configuration Parameters
-
-| Parameter        | Purpose                        | Example                     | Effect                     |
-| ---------------- | ------------------------------ | --------------------------- | -------------------------- |
-| `pageSize`       | Records per GraphQL request    | `200`                       | Controls `first` variable  |
-| `maxRecords`     | Total extraction limit         | `50000`                     | Hard stop across all pages |
-| `resultPath`     | Where records live in response | `"fulfillments.edges.node"` | Flattening path            |
-| `validateItem()` | Optional record filter         | `(item) => !!item.ref`      | Skips invalid records      |
-
-#### GraphQL Query Requirements
-
-**Your query MUST include these pagination fields:**
-
-```graphql
-query GetFulfillments(
-  $retailerId: ID!
-  $dateRangeFilter: DateRange
-  $statuses: [String!]
-  $first: Int! # ← Orchestrator injects this
-  $after: String # ← Orchestrator injects this
-) {
-  fulfillments(
-    retailerId: $retailerId
-    updatedOn: $dateRangeFilter
-    status: $statuses
-    first: $first # ← Pagination page size
-    after: $after # ← Pagination cursor
-  ) {
-    edges {
-      # ← REQUIRED: Relay connection structure
-      node {
-        # ← REQUIRED: Actual records here
-        id
-        ref
-        # ... your fields
-      }
-      cursor # ← REQUIRED: Orchestrator uses this for next page
-    }
-    pageInfo {
-      # ← REQUIRED: Orchestrator checks this
-      hasNextPage # ← REQUIRED: Tells orchestrator to continue
-    }
-  }
-}
-```
-
----
-
-### Pattern: Backward Pagination (Optional - Advanced)
-
-**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
-
-**When to Use**:
-
-- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
-- ✅ Time-bounded reverse traversal for auditing
-- ✅ Display newest-first in UI/reports
-- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
-
-**GraphQL Query Requirements**:
-
-Your query must support backward pagination by including `$last` and `$before`:
-
-```graphql
-query GetData(
-  $retailerId: ID!
-  $first: Int # For forward pagination
-  $after: String # For forward pagination
-  $last: Int # For backward pagination
-  $before: String # For backward pagination
-) {
-  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
-    edges {
-      cursor # ? REQUIRED
-      node {
-        id
-        createdAt
-        # ... other fields
-      }
-    }
-    pageInfo {
-      hasNextPage # For forward
-      hasPreviousPage # ? REQUIRED for backward
-    }
-  }
-}
-```
-
-**Implementation**:
-
-```typescript
-// Backward pagination - newest records first
-const result = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    retailerId,
-    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
-    // ❌ Don't include last/before - orchestrator injects them
-  },
-  pageSize: 200,
-  direction: 'backward', // ? Enable reverse pagination
-  maxRecords: 10000,
-});
-
-// Records are returned in reverse chronological order
-console.log(result.data[0].createdAt); // Newest
-console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
-```
-
-**Key Differences from Forward Pagination**:
-
-| Aspect                 | Forward (Default)                | Backward                |
-| ---------------------- | -------------------------------- | ----------------------- |
-| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
-| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
-| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
-| **Cursor Source**      | Last edge of page                | First edge of page      |
-| **Record Order**       | Oldest → Newest                | Newest → Oldest       |
-
-**Important Notes**:
-
-1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
-
-2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
-
-3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
-
-4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
-
-**Example: Extract Latest 1000 Orders**
-
-```typescript
-const latestOrders = await orchestrator.extract({
-  query: ORDERS_QUERY,
-  resultPath: 'orders.edges.node',
-  variables: {
-    retailerId,
-    statuses: ['BOOKED', 'ALLOCATED'],
-  },
-  direction: 'backward', // Start from newest
-  maxRecords: 1000, // Stop after 1000 records
-  pageSize: 100, // 100 per page = 10 pages
-});
-
-// latestOrders.data[0] is the newest order
-// latestOrders.data[999] is the 1000th newest order
-```
-
-**When to Use Forward vs Backward**:
-
-```typescript
-// ? Forward (default) - For incremental sync
-const incrementalData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  variables: {
-    dateRangeFilter: { from: lastSyncTime, to: now },
-  },
-  // direction defaults to 'forward'
-  // Processes oldest → newest for proper sequencing
-});
-
-// ? Backward - For "latest N records" use cases
-const latestData = await orchestrator.extract({
-  query: YOUR_QUERY,
-  resultPath: 'data.edges.node',
-  direction: 'backward',
-  maxRecords: 100, // Just get latest 100
-  // Gets newest → oldest
-});
-```
-
-**Pagination Variables Reference**:
-
-| Variable | Forward     | Backward    | Injected By  | Notes                    |
-| -------- | ----------- | ----------- | ------------ | ------------------------ |
-| `first`  | ? Used     | ❌ Not used | Orchestrator | From `pageSize`          |
-| `after`  | ? Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
-| `last`   | ❌ Not used | ? Used     | Orchestrator | From `pageSize`          |
-| `before` | ❌ Not used | ? Used     | Orchestrator | From cursor (first edge) |
-
-**Common Mistakes to Avoid**:
-
-```typescript
-// ❌ WRONG - Don't pass pagination variables
-const result = await orchestrator.extract({
-  variables: {
-    last: 200, // ❌ Orchestrator will override this
-    before: cursor, // ❌ Orchestrator manages cursor
-  },
-  direction: 'backward',
-});
-
-// ? CORRECT - Let orchestrator inject pagination
-const result = await orchestrator.extract({
-  variables: {
-    retailerId, // ? Your business variables only
-  },
-  pageSize: 200, // ? Orchestrator uses this for last/before
-  direction: 'backward',
-});
-```
-
----
-
-## 📄 Mapping Configuration
-
-**File:** `config/fulfillments.export.csv.json`
-
-```json
-{
-  "name": "fulfillments.export.csv",
-  "version": "1.0.0",
-  "description": "Fulfillments → CSV Export Mapping",
-  "fields": {
-    "order_number": {
-      "source": "orderRef",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "fulfillment_id": {
-      "source": "ref",
-      "required": true,
-      "resolver": "sdk.trim"
-    },
-    "tracking_number": {
-      "source": "trackingNumber",
-      "required": false,
-      "resolver": "sdk.trim"
-    },
-    "carrier": {
-      "source": "carrier",
-      "required": false,
-      "resolver": "sdk.uppercase"
-    },
-    "service_level": {
-      "source": "serviceLevel",
-      "required": false,
-      "resolver": "sdk.trim"
-    },
-    "status": {
-      "source": "status",
-      "required": true,
-      "resolver": "sdk.uppercase"
-    },
-    "ship_date": {
-      "source": "shippedOn",
-      "required": false,
-      "resolver": "sdk.formatDateShort"
-    },
-    "delivery_date": {
-      "source": "deliveredOn",
-      "required": false,
-      "resolver": "sdk.formatDateShort"
-    },
-    "customer_email": {
-      "source": "order.customer.email",
-      "required": false,
-      "resolver": "sdk.trim"
-    },
-    "last_updated": {
-      "source": "updatedOn",
-      "required": true,
-      "resolver": "sdk.toString"
-    }
-  }
-}
-```
-
-**AI Customization Hints:**
-
-- Add fields: Copy existing field config, change `source` path
-- Remove fields: Delete field from config
-- Change resolvers: Replace `sdk.trim` with `sdk.uppercase`, etc.
-- Nested fields: Use dot notation like `order.customer.email`
-
-Note: Customize mapping by editing the JSON above; prefer built-in resolvers. See SDK Universal Mapping guide for advanced usage.
-
----
-
-## Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-fulfillments-extraction/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-fulfillments-extraction.ts  # Scheduled: Daily extraction
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-fulfillments-extraction.ts  # Webhook: Manual trigger
-    │       └── job-status-check.ts               # Webhook: Status query
-    │
-    ├── services/
-    │   └── fulfillments-extraction.service.ts    # Shared orchestration logic (reusable)
-    │
-    └── config/
-        └── fulfillments.export.csv.json         # Mapping configuration
-```
-
-
-### 1. Entry Point (`index.ts`)
-
-**Pattern:** MemoryInterpreter (export all workflows from single entry point)
-
-**Benefits:**
-- ✅ Versori platform discovers all workflows automatically
-- ✅ Clean separation of concerns (entry point vs workflow logic)
-- ✅ Easy to add/remove workflows (just update exports)
-- ✅ Single source of truth for workflow registration
-
-```typescript
-// ═══════════════════════════════════════════════════════════
-// 🚀 VERSORI FULFILLMENTS EXTRACTION
-// ═══════════════════════════════════════════════════════════
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * This file is the entry point for the Versori deployment.
- * It registers three workflows:
- * 1. Scheduled extraction (runs automatically)
- * 2. Ad hoc webhook (manual trigger)
- * 3. Job status webhook (query progress)
- *
- * AI CUSTOMIZATION:
- * - Add new workflows by importing and registering them
- * - Remove workflows by commenting out registration
- * - Change workflow names in import statements
- */
-// ═══════════════════════════════════════════════════════════
-
-import {
-  scheduledFulfillmentsExtraction,
-  adhocFulfillmentsExtraction,
-  fulfillmentsJobStatus,
-} from './workflows/fulfillments-extraction';
-
-// Register workflows with Versori platform
-// The platform will expose these as executable endpoints
-
-export {
-  scheduledFulfillmentsExtraction, // Cron-based auto-run
-  adhocFulfillmentsExtraction, // Manual webhook trigger
-  fulfillmentsJobStatus, // Job status query
-};
-```
-
----
-
-### 2. Workflows (src/workflows/fulfillments-extraction.ts)
-
-```typescript
-/**
- * Workflows - Defines 3 execution patterns for fulfillments extraction
- *
- * WORKFLOW 1: Scheduled (Cron) - Runs automatically every 4 hours
- * WORKFLOW 2: Ad hoc (Webhook) - Manual trigger with optional date override
- * WORKFLOW 3: Job Status (Webhook) - Query job progress
- *
- * AI CUSTOMIZATION HINTS:
- * - Change schedule: Modify cron expression in schedule()
- * - Add filtering: Pass additional params to executeFulfillmentExtraction()
- * - Change response format: Modify return object structure
- */
-
-import { schedule, webhook, http, fn } from '@versori/run';
-import { executeFulfillmentExtraction, getJobStatus } from '../services/extraction-orchestration';
-import { generateJobId } from '../utils/job-id-generator';
-
-/**
- * WORKFLOW 1: Scheduled Extraction
- *
- * Purpose: Automated extraction every 4 hours for incremental sync
- * Trigger: Cron schedule (every 4 hours at minute 0)
- * State Update: Always updates lastSync timestamp
- *
- * AI CUSTOMIZATION:
- * - Change schedule: Modify the cron expression string
- *   Examples:
- *   - Every hour: '0 * * * *'
- *   - Every 30 min: '*/30 * * * *'
- *   - Daily at 2 AM: '0 2 * * *'
- */
-export const scheduledFulfillmentsExtraction = schedule(
-  'fulfillments-scheduled',
-  '0 */4 * * *'  // ← CUSTOMIZE: Cron expression
-)
-.then(
-  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async (ctx) => {
-    const { log } = ctx;
-
-    // Generate unique job ID for tracking
-    // Format: SCHEDULED_FULFILLMENTS_YYYYMMDD_HHMMSS_random
-    const jobId = generateJobId('SCHEDULED', 'FULFILLMENTS');
-
-    log.info('Scheduled extraction triggered', { jobId });
-
-    try {
-      // Execute main workflow (extraction → transform → upload)
-      const result = await executeFulfillmentExtraction(ctx, {
-        jobId,
-        triggeredBy: 'schedule',
-        updateState: true,        // Always update state for scheduled runs
-      });
-
-      log.info('Scheduled extraction completed', {
-        jobId,
-        recordCount: result.recordsExtracted,
-        fileName: result.fileName
-      });
-
-      return result;
-
-    } catch (error: any) {
-      log.error('Scheduled extraction failed', {
-        jobId,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'Error',
-      });
-      throw error;
-    }
-  })
-);
-
-/**
- * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
- *
- * Purpose: Manual extraction with optional date range override
- * Trigger: Webhook POST to /webhooks/fulfillments-adhoc
- * State Update: Optional (controlled by request payload)
- *
- * WEBHOOK PAYLOAD EXAMPLES:
- *
- * 1. Incremental (use last sync timestamp):
- *    {}
- *
- * 2. Date range (manual override):
- *    {
- *      "fromDate": "2025-01-01T00:00:00Z",
- *      "toDate": "2025-01-31T23:59:59Z",
- *      "updateState": false
- *    }
- *
- * AI CUSTOMIZATION:
- * - Add request validation
- * - Add authentication check
- * - Add custom filters from payload
- */
-export const adhocFulfillmentsExtraction = webhook(
-  'fulfillments-adhoc',
-  { connection: 'fulfillments-adhoc', response: { mode: 'sync' } }
-)
-.then(
-  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async (ctx) => {
-    const { data, log } = ctx;
-
-    // Generate unique job ID
-    const jobId = generateJobId('ADHOC', 'FULFILLMENTS');
-
-    // SECURITY: Authentication is enforced by Versori connection configuration
-    // Configure auth on the connection and reference it in webhook({ connection: '...' })
-
-    // Extract optional date override from webhook payload
-    const fromDate = data.fromDate as string | undefined;
-    const toDate = data.toDate as string | undefined;
-    const updateState = data.updateState === true;  // Default false; advance state only if explicitly true
-
-    log.info('Ad hoc extraction triggered via webhook', {
-      jobId,
-      hasDateOverride: !!fromDate,
-      fromDate: fromDate || 'not specified',
-      toDate: toDate || 'not specified',
-      updateState
-    });
-
-    try {
-      // Execute main workflow with optional overrides
-      const result = await executeFulfillmentExtraction(ctx, {
-        jobId,
-        triggeredBy: 'webhook',
-        fromDate,              // Optional: override start date
-        toDate,                // Optional: override end date
-        updateState,           // Optional: skip state update for historical queries
-      });
-
-      log.info('Ad hoc extraction completed', {
-        jobId,
-        recordCount: result.recordsExtracted,
-        fileName: result.fileName,
-        isManualOverride: !!fromDate,
-        stateUpdated: result.stateUpdated
-      });
-
-      // Return success with job details
-      return {
-        success: true,
-        jobId,
-        recordsExtracted: result.recordsExtracted,
-        fileName: result.fileName,
-        sftpPath: result.sftpPath,
-        statusUrl: `/webhooks/fulfillments-job-status?jobId=${jobId}`,
-        dateRange: fromDate ? {
-          from: fromDate,
-          to: toDate || 'not specified',
-          updateState
-        } : undefined
-      };
-
-    } catch (error: any) {
-      log.error('Ad hoc extraction failed', {
-        jobId,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'Error',
-      });
-
-      return {
-        success: false,
-        jobId,
-        error: error instanceof Error ? error.message : String(error)
-      };
-    }
-  })
-);
-
-/**
- * WORKFLOW 3: Job Status Query
- *
- * Purpose: Check job progress and status
- * Trigger: Webhook GET/POST to /webhooks/fulfillments-job-status?jobId=xxx
- * Returns: Current job status, stage, progress
- *
- * QUERY EXAMPLES:
- *
- * 1. HTTP GET:
- *    GET /webhooks/fulfillments-job-status?jobId=ADHOC_FULFILLMENTS_20251027_183045_abc123
- *
- * 2. HTTP POST:
- *    POST /webhooks/fulfillments-job-status
- *    { "jobId": "ADHOC_FULFILLMENTS_20251027_183045_abc123" }
- */
-export const fulfillmentsJobStatus = webhook(
-  'fulfillments-job-status',
-  { connection: 'fulfillments-job-status', response: { mode: 'sync' } }
-)
-.then(
-  fn('query-job-status', async (ctx) => {
-    const { data, log, openKv, activation } = ctx;
-    const req = ctx.request();
-
-    // SECURITY: Authentication is enforced by Versori connection configuration
-    // Configure auth on the connection and reference it in webhook({ connection: '...' })
-
-    // Get jobId from query param or POST body
-    const jobId = data.jobId as string;
-
-    if (!jobId) {
-      log.error('Job ID not provided in request');
-      return {
-        success: false,
-        error: 'Job ID is required. Provide jobId in query param or request body.'
-      };
-    }
-
-    log.info('Querying job status', { jobId });
-
-    try {
-      // Query job status from KV store
-      const status = await getJobStatus(openKv(':project:'), jobId, log);
-
-      if (!status) {
-        log.info('Job not found', { jobId });
-        return {
-          success: false,
-          error: 'Job not found',
-          jobId
-        };
-      }
-
-      log.info('Job status retrieved', { jobId, status: status.status });
-
-      return {
-        success: true,
-        jobId,
-        ...status
-      };
-
-    } catch (error: any) {
-      log.error('Failed to query job status', {
-        jobId,
-        message: error instanceof Error ? error.message : String(error),
-        stack: error instanceof Error ? error.stack : undefined,
-        errorType: error instanceof Error ? error.constructor.name : 'Error',
-      });
-
-      return {
-        success: false,
-        jobId,
-        error: error instanceof Error ? error.message : String(error)
-      };
-    }
-  })
-);
-```
-
----
-
-## 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
-
-```typescript
-/**
- * MAIN ORCHESTRATION SERVICE
- *
- * This is the heart of the extraction workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Determine date range (incremental vs manual)
- * 3. Extract data using ExtractionOrchestrator
- * 4. Transform using UniversalMapper
- * 5. Generate CSV using CSVParserService
- * 6. Upload to SFTP
- * 7. Track job progress with JobTracker
- * 8. Update state for next run
- *
- * NAMING PATTERN (consistent across all use cases):
- * - Interface: {Entity}ExtractionParams (e.g., FulfillmentExtractionParams)
- * - Result: {Entity}ExtractionResult (e.g., FulfillmentExtractionResult)
- * - Main function: execute{Entity}Extraction (e.g., executeFulfillmentExtraction)
- *
- * AI CUSTOMIZATION HINTS:
- * - Change entity: Replace "Fulfillment" with "Order", "Product", etc.
- * - Change output: Replace CSVParserService with XMLBuilder
- * - Change destination: Replace SftpDataSource with S3DataSource
- * - Add steps: Insert new service calls between existing steps
- */
-
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  ExtractionOrchestrator,
-  JobTracker,
-  UniversalMapper,
-  CSVParserService,
-  SftpDataSource,
-} from '@fluentcommerce/fc-connect-sdk';
-
-import mappingConfig from '../../config/fulfillments.export.csv.json' with { type: 'json' };
-
-// ? VERSORI PLATFORM: Use native log from context, not LoggingService
-
-/**
- * Parameters for extraction workflow
- *
- * NAMING: {Entity}ExtractionParams
- */
-export interface FulfillmentExtractionParams {
-  jobId: string;
-  triggeredBy: 'schedule' | 'webhook';
-  fromDate?: string; // Optional: manual date override
-  toDate?: string; // Optional: manual date override
-  updateState: boolean; // Whether to update rawLastRunTime timestamp
-
-  // AI CUSTOMIZATION: Add filters specific to entity
-  fulfillmentStatuses?: string[]; // e.g., ['SHIPPED', 'DELIVERED']
-  retailerId?: string; // e.g., 'YOUR_RETAILER_ID'
-}
-
-/**
- * Result from extraction workflow
- *
- * NAMING: {Entity}ExtractionResult
- */
-export interface FulfillmentExtractionResult {
-  success: boolean;
-  jobId: string;
-  recordsExtracted: number;
-  fileName?: string;
-  sftpPath?: string;
-  error?: string;
-  errors?: any[];
-  isManualOverride?: boolean;
-  stateUpdated?: boolean;
-  newTimestamp?: string;
-}
-
-/**
- * GraphQL Query for Fulfillments
- *
- * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
- */
-const FULFILLMENTS_EXTRACTION_QUERY = `
-  query GetFulfillments(
-    $updatedAfter: DateTime!
-    $statuses: [String!]
-    $first: Int!
-    $after: String
-  ) {
-    fulfillments(
-      updatedOn: { after: $updatedAfter }
-      status: $statuses
-      first: $first
-      after: $after
-    ) {
-      edges {
-        node {
-          id
-          ref
-          orderRef
-          status
-          trackingNumber
-          carrier
-          serviceLevel
-          shippedOn
-          deliveredOn
-          updatedOn
-          order {
-            customer {
-              email
-            }
-          }
-        }
-        cursor
-      }
-      pageInfo {
-        hasNextPage
-      }
-    }
-  }
-`;
-
-/**
- * Query job status from KV store
- *
- * ? VERSORI PLATFORM: Uses native log from context (passed as parameter)
- */
-export async function getJobStatus(
-  kv: any, // ? Versori KV (compatible with JobTracker's KVAdapter interface)
-  jobId: string,
-  log: any // ? Native Versori log from context
-): Promise<any | undefined> {
-  try {
-    const tracker = new JobTracker(kv, log);
-    return await tracker.getJob(jobId);
-  } catch (error: any) {
-    log.error('Failed to get job status', {
-      jobId,
-      message: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      errorType: error instanceof Error ? error.constructor.name : 'Error',
-    });
-    return undefined;
-  }
-}
-
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * NAMING: execute{Entity}Extraction (e.g., executeFulfillmentExtraction)
- *
- * This function implements the complete workflow in steps.
- * Each step is clearly commented for AI understanding.
- */
-export async function executeFulfillmentExtraction(
-  ctx: any,
-  params: FulfillmentExtractionParams
-): Promise<FulfillmentExtractionResult> {
-  // ? VERSORI PLATFORM: Extract native log from context
-  const { log, openKv, activation } = ctx;
-  const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
-
-  // Open KV store for state management and job tracking
-  // ? Pass raw Versori KV directly - it matches KVAdapter interface
-  // ? Pass native log to JobTracker
-  const kv = openKv(':project:');
-  const tracker = new JobTracker(kv, log);
-
-  try {
-    // ═══════════════════════════════════════════════════════════
-    // STEP 1/8: Initialize Job Tracking
-    // ═══════════════════════════════════════════════════════════
-    log.info('🚀 [STEP 1/8] Initializing job tracking', { jobId });
-
-    await tracker.createJob(jobId, {
-      triggeredBy,
-      hasDateOverride: !!fromDate,
-      fromDate,
-      toDate,
-      updateStateAfterRun: updateState,
-    });
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 2/8: Initialize Fluent Client
-    // ═══════════════════════════════════════════════════════════
-    log.info('🚀 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
-
-    // ✅ Optional: Validate connection immediately (fail-fast mode)
-    // Set activation variable 'validateConnectionOnStart' = 'true' to enable
-    // When enabled: Executes query { me { ref } } to verify authentication
-    // When disabled: Fast creation, validation happens on first API call (default)
-    const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
-    const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
-
-    if (!client) {
-      throw new Error('Failed to create Fluent Commerce client');
-    }
-
-    if (validateConnection) {
-      log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
-    }
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 3/8: Determine Date Range
-    // ═══════════════════════════════════════════════════════════
-    log.info('🔍 [STEP 3/8] Determining date range for extraction', { jobId });
-
-    // State key for incremental sync tracking
-    // NAMING: last{Entity}Sync (e.g., lastFulfillmentSync)
-    const STATE_KEY = 'lastFulfillmentSync';
-    const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
-    const OVERLAP_BUFFER_SECONDS = parseInt(
-      activation.getVariable('overlapBufferSeconds') || '60',
-      10
-    );
-
-    let bufferedLastRunTime: string;
-    const isManualOverride = !!fromDate;
-
-    if (isManualOverride) {
-      // Manual date override from webhook
-      bufferedLastRunTime = fromDate!;
-      log.info('Using manual date override', { fromDate, toDate });
-    } else {
-      // Incremental sync - get last sync timestamp
-      const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
-
-      // Apply overlap buffer (prevents missed records)
-      bufferedLastRunTime = new Date(
-        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
-      ).toISOString();
-
-      log.info('Using incremental sync with overlap buffer', {
-        rawLastRunTime,
-        bufferedLastRunTime,
-        overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
-      });
-    }
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 4/8: Extract Data (ExtractionOrchestrator)
-    // ═══════════════════════════════════════════════════════════
-    log.info('📥 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
-
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'extraction',
-      message: 'Extracting data with auto-pagination',
-    });
-
-    // Configure extraction
-    const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
-    const maxRecords = parseInt(activation.getVariable('maxRecords') || '10000', 10);
-
-    // Parse status filter
-    const fulfillmentStatuses =
-      params.fulfillmentStatuses ||
-      (activation.getVariable('fulfillmentStatuses') || 'SHIPPED,DELIVERED').split(',');
-
-    // Initialize ExtractionOrchestrator
-    const orchestrator = new ExtractionOrchestrator(client, log);
-
-    // ? Enhanced: Extract context for progress logging
-    const dateRangeInfo = {
-      start: bufferedLastRunTime || 'N/A',
-      end: new Date().toISOString(),
-      statuses: fulfillmentStatuses.join(', ') || 'all'
-    };
-
-    // Duration tracking
-    const extractionStartTime = Date.now();
-
-    // ? Enhanced: Start logging with context
-    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
-     query: 'fulfillments',
-     pageSize,
-     maxRecords,
-     dateRange: `from ${dateRangeInfo.start}`,
-     statuses: dateRangeInfo.statuses,
-     jobId
-    });
-
-    // Execute extraction with auto-pagination
-    const extractionResult = await orchestrator.extract({
-      query: FULFILLMENTS_EXTRACTION_QUERY,
-      resultPath: 'fulfillments.edges.node',
-      variables: {
-        updatedAfter: bufferedLastRunTime,
-        statuses: fulfillmentStatuses,
-        // Note: Don't include 'first' or 'after' here; orchestrator injects them
-      },
-      pageSize,
-      maxRecords,
-      // Optional: validate each record
-      validateItem: (item: any) => {
-        return !!(item.ref && item.orderRef);
-      },
-    });
-
-    const rawRecords = extractionResult.data;
-
-    // Calculate extraction duration
-    const extractionDuration = Date.now() - extractionStartTime;
-
-    log.info('✅ [STEP 4/8] Extraction completed', {
-      jobId,
-      totalRecords: extractionResult.stats.totalRecords,
-      totalPages: extractionResult.stats.totalPages,
-      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-      errors: extractionResult.errors ? extractionResult.errors.length : 0,
-      duration: `${extractionDuration}ms`,
-    });
-
-    // ? Enhanced: Completion logging with summary
-    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
-     totalRecords: extractionResult.stats.totalRecords,
-     totalPages: extractionResult.stats.totalPages,
-     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
-     failedValidations: extractionResult.stats.failedValidations,
-     truncated: extractionResult.stats.truncated,
-     truncationReason: extractionResult.stats.truncationReason,
-     dateRange: `from ${dateRangeInfo.start}`,
-     duration: `${extractionDuration}ms`,
-     jobId
-    });
-
-    if (extractionResult.errors && extractionResult.errors.length > 0) {
-      log.warn('Non-fatal extraction errors encountered', {
-        jobId,
-        errorCount: extractionResult.errors.length,
-        sampleErrors: extractionResult.errors.slice(0, 3),
-      });
-    }
-
-    // Handle empty result
-    if (rawRecords.length === 0) {
-      log.info('No records to process');
-
-      // Update state even with no records (prevents re-querying empty window)
-      if (updateState) {
-        await kv.set(STATE_KEY, new Date().toISOString());
-      }
-
-      await tracker.markCompleted(jobId, {
-        recordCount: 0,
-        message: 'No records to extract',
-      });
-
-      return {
-        success: true,
-        jobId,
-        recordsExtracted: 0,
-      };
-    }
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 5/8: Transform Data (UniversalMapper)
-    // ═══════════════════════════════════════════════════════════
-    log.info('🔄 [STEP 5/8] Transforming data with UniversalMapper', {
-      jobId,
-      recordCount: rawRecords.length,
-    });
-
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'transformation',
-      message: `Transforming ${rawRecords.length} records`,
-    });
-
-    const mapper = new UniversalMapper(mappingConfig);
-    const mappingResult = await mapper.map(rawRecords);
-
-    if (!mappingResult.success) {
-      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
-      log.error('[STEP 5/8] Mapping failed - terminating job', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-
-      await tracker.markFailed(
-        jobId,
-        new Error(mappingErrors[0] || 'UniversalMapper returned unsuccessful result')
-      );
-
-      return {
-        success: false,
-        jobId,
-        recordsExtracted: 0,
-        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
-      };
-    }
-
-    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
-    const mappingErrors = mappingResult.errors || [];
-
-    if (mappingErrors.length > 0) {
-      log.warn('[STEP 5/8] Some records failed transformation', {
-        jobId,
-        errorCount: mappingErrors.length,
-        sampleErrors: mappingErrors.slice(0, 3),
-      });
-    }
-
-    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-        jobId,
-        skippedFields: mappingResult.skippedFields,
-        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-      });
-    }
-
-    if (transformedRecords.length === 0) {
-      throw new Error('All records failed transformation');
-    }
-
-    log.info('Transformation complete', {
-      jobId,
-      successful: transformedRecords.length,
-      skippedRecords: rawRecords.length - transformedRecords.length,
-    });
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 6/8: Generate CSV (CSVParserService)
-    // ═══════════════════════════════════════════════════════════
-    log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
-
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'csv_generation',
-      message: `Generating CSV for ${transformedRecords.length} records`,
-    });
-
-    // Initialize CSVParserService
-    const csvParser = new CSVParserService();
-
-    // Generate CSV content
-    const csvContent = csvParser.stringify(transformedRecords, { headers: true });
-
-    // Generate filename using helper function
-    const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'fulfillments';
-    const extractFileName = (prefix: string) => {
-      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
-      return `${prefix}-${timestamp}.csv`;
-    };
-    const fileName = extractFileName(fileNamePrefix);
-
-    log.info('✅ CSV file generated', {
-      fileName,
-      sizeBytes: csvContent.length,
-      recordCount: transformedRecords.length,
-    });
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 7/8: Upload to SFTP (SftpDataSource)
-    // ═══════════════════════════════════════════════════════════
-    log.info('📤 [STEP 7/8] Uploading to SFTP', { jobId, fileName });
-
-    await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'sftp_upload',
-      message: `Uploading ${fileName} to SFTP`,
-    });
-
-    // Get SFTP credentials from Versori connection (Basic Auth)
-    // RECOMMENDED: Use activation.connections (already decoded)
-    const allConnections = ctx.activation.connections || [];
-    const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
-
-    if (!sftpConn) {
-      throw new Error('SFTP connection "versori_ftp_server" not found');
-    }
-
-    const credential = sftpConn.credentials[0]?.credential;
-    if (!credential?.data?.basicAuth) {
-      throw new Error('SFTP connection not configured with Basic Authentication');
-    }
-
-    const { username, password } = credential.data.basicAuth;
-    // ? Already decoded - no Buffer.from() needed!
-
-    // Get other SFTP config from activation variables
-    const sftpConfig = {
-      host: activation.getVariable('sftpHost'),
-      port: parseInt(activation.getVariable('sftpPort') || '22', 10),
-      username, // From connection
-      password, // From connection
-      privateKey: activation.getVariable('sftpPrivateKey'), // Optional: if using key-based auth
-    };
-    const sftpRemotePath = activation.getVariable('sftpRemotePath') || '/incoming/fulfillments/';
-
-    // Validate SFTP config
-    if (!sftpConfig.host || !sftpConfig.username) {
-      throw new Error('SFTP configuration incomplete: missing host or username from connection');
-    }
-
-    if (!sftpConfig.password && !sftpConfig.privateKey) {
-      throw new Error(
-        'SFTP configuration incomplete: missing password from connection or privateKey from activation'
-      );
-    }
-
-    // Initialize SFTP data source
-    // ? VERSORI PLATFORM: Pass native log from context
-    // ? Declare as let for proper disposal in finally block
-    let sftp: SftpDataSource | undefined;
-
-    // Construct SFTP path
-    const sftpPath = `${sftpRemotePath}${fileName}`;
-
-    try {
-      sftp = new SftpDataSource(
-        {
-          type: 'SFTP_CSV',
-          connectionId: 'fulfillments-sftp',
-          name: 'Fulfillments SFTP Upload',
-          settings: {
-            host: sftpConfig.host!,
-            port: sftpConfig.port,
-            username: sftpConfig.username!,
-            password: sftpConfig.password,
-            privateKey: sftpConfig.privateKey,
-            remotePath: sftpRemotePath,
-            filePattern: '*.csv',
-          },
-        },
-        log
-      );
-
-      // Create remote directory if needed
-      await sftp.createDirectory(sftpRemotePath, true);
-
-      // Upload with retry logic (built into SftpDataSource)
-      await sftp.uploadFile(sftpPath, Buffer.from(csvContent, 'utf-8'), {
-        createDirectories: true,
-      });
-
-      log.info('✅ SFTP upload successful', { fileName, remotePath: sftpPath });
-    } finally {
-      // ?? CRITICAL: Always dispose SFTP connection
-      if (sftp) {
-        await sftp.dispose();
-        log.info('🔌 SFTP connection disposed');
-      }
-    }
-
-    // ═══════════════════════════════════════════════════════════
-    // STEP 8/8: Update State & Complete Job
-    // ═══════════════════════════════════════════════════════════
-    log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
-
-    // Calculate new timestamp for next incremental run
-    let newTimestamp: string | undefined;
-
-    if (updateState) {
-      // Find max updatedOn from extracted records
-      const maxUpdatedOn = rawRecords.reduce((max, record) => {
-        const recordTime = new Date(record.updatedOn).getTime();
-        return recordTime > max ? recordTime : max;
-      }, new Date(bufferedLastRunTime).getTime());
-
-      newTimestamp = new Date(maxUpdatedOn).toISOString();
-
-      // Store new timestamp (WITHOUT buffer - buffer only applied on read)
-      await kv.set(STATE_KEY, newTimestamp);
-
-      log.info('✅ State updated', {
-        oldTimestamp: bufferedLastRunTime,
-        newTimestamp,
-      });
-    }
-
-    // Mark job as completed
-    await tracker.markCompleted(jobId, {
-      recordCount: transformedRecords.length,
-      fileName,
-      sftpPath,
-      errorCount: mappingErrors.length,
-      isManualOverride,
-      stateUpdated: updateState,
-      newTimestamp,
-    });
-
-    log.info('✅ [COMPLETE] Extraction workflow completed successfully', { jobId });
-
-    return {
-      success: true,
-      jobId,
-      recordsExtracted: transformedRecords.length,
-      fileName,
-      sftpPath,
-      isManualOverride,
-      stateUpdated: updateState,
-      newTimestamp,
-      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
-    };
-  } catch (error: any) {
-    log.error('❌ [FATAL] Extraction workflow failed', {
-      jobId,
-      message: error instanceof Error ? error.message : String(error),
-      stack: error instanceof Error ? error.stack : undefined,
-      errorType: error instanceof Error ? error.constructor.name : 'Error',
-      recommendation: 'Check logs for detailed error information. Verify activation variables, connection credentials, and SFTP configuration.',
-    });
-
-    // Mark job as failed
-    await tracker.markFailed(jobId, error);
-
-    return {
-      success: false,
-      jobId,
-      recordsExtracted: 0,
-      error: error instanceof Error ? error.message : String(error),
-    };
-  }
-}
-```
-
----
-
-## 4. Utility Functions (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking extraction workflows
- *
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
- * Example: SCHEDULED_FULFILLMENTS_20251027_183045_a1b2c3
- */
-
-/**
- * Generate unique job ID
- *
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
- * @param entity - Entity abbreviation (FULFILLMENTS, VP, ORD, PRD)
- * @returns Unique job ID string
- */
-export function generateJobId(type: string, entity: string): string {
-  const now = new Date();
-
-  // Format: YYYYMMDD
-  const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
-
-  // Format: HHMMSS
-  const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
-
-  // Random suffix (6 chars)
-  const randomStr = Math.random().toString(36).substring(2, 8);
-
-  return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
-}
-
-/**
- * Parse job ID components
- */
-export function parseJobId(jobId: string): {
-  type: string;
-  entity: string;
-  date: string;
-  time: string;
-  random: string;
-} | null {
-  const parts = jobId.split('_');
-
-  if (parts.length !== 5) {
-    return null;
-  }
-
-  return {
-    type: parts[0],
-    entity: parts[1],
-    date: parts[2],
-    time: parts[3],
-    random: parts[4],
-  };
-}
-```
-
----
-
-## 5. Package Configuration
-
-### package.json
-
-```json
-{
-  "name": "fulfillments-to-sftp-csv",
-  "version": "1.0.0",
-  "description": "Extract fulfillments from Fluent Commerce and export to SFTP as CSV",
-  "type": "module",
-  "main": "src/index.ts",
-  "scripts": {
-    "dev": "versori dev",
-    "build": "versori build",
-    "deploy": "versori deploy"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-
-### tsconfig.json
-
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
-
----
-
-## 6. Deployment Instructions
-
-### Deploy to Versori
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Test locally (if using Versori CLI)
-npm run dev
-
-# 3. Deploy to Versori platform
-npm run deploy
-```
-
-### Configure SFTP Connection
-
-1. Create a Versori connection named `versori_ftp_server`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter your SFTP **Username** and **Password**
-
-### Configure Activation Variables
-
-In Versori platform settings, configure:
-
-```json
-{
-  "retailerId": "your-retailer-id",
-  "sftpHost": "sftp.partner.com",
-  "sftpPort": 22,
-  "sftpRemotePath": "/incoming/fulfillments/",
-  "fileNamePrefix": "fulfillments",
-  "pageSize": 200,
-  "maxRecords": 50000,
-  "overlapBufferSeconds": 60,
-  "fulfillmentStatuses": "SHIPPED,DELIVERED",
-  "webhookApiKey": "your-secure-api-key-here"
-}
-```
-
-> **Note:** `sftpUsername` and `sftpPassword` are now stored in the `versori_ftp_server` connection.
-
----
-
-## 7. Testing
-
-### Test Scheduled Extraction
-
-The scheduled workflow runs automatically based on cron schedule.
-
-**Check logs:**
-
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client
-[STEP 3/8] Determining date range for extraction
-[STEP 4/8] Extracting data from Fluent Commerce
-[STEP 5/8] Transforming data with UniversalMapper
-[STEP 6/8] Generating CSV file
-[STEP 7/8] Uploading to SFTP
-[STEP 8/8] Updating state and completing job
-```
-
-### Test Ad hoc Extraction
-
-```bash
-# Incremental (uses last sync timestamp)
-curl -X POST https://api.versori.com/webhooks/fulfillments-adhoc \
-  -H "X-API-Key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{}'
-
-# Date range override
-curl -X POST https://api.versori.com/webhooks/fulfillments-adhoc \
-  -H "X-API-Key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "fromDate": "2025-01-01T00:00:00Z",
-    "toDate": "2025-01-31T23:59:59Z",
-    "updateState": false
-  }'
-```
-
-### Test Job Status Query
-
-```bash
-curl -X POST https://api.versori.com/webhooks/fulfillments-job-status \
-  -H "X-API-Key: your-api-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "jobId": "ADHOC_FULFILLMENTS_20251027_183045_abc123"
-  }'
-```
-
----
-
-## Testing Checklist
-
-**Before production deployment:**
-
-### 1. Schema Validation
-
-- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
-- [ ] Run `npx fc-connect validate-schema --mapping ./config/fulfillments.export.csv.json --schema ./fluent-schema.json`
-- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/fulfillments.export.csv.json --schema ./fluent-schema.json`
-- [ ] Verify all `source` paths in mapping exist in GraphQL schema
-- [ ] Verify query structure matches schema (fields, types, filters)
-
-### 2. Extraction Testing
-
-- [ ] Test with small dataset first (maxRecords=10)
-- [ ] Verify ExtractionOrchestrator pagination works correctly
-- [ ] Test with multiple pages of data (verify cursor handling)
-- [ ] Verify date range filtering (updatedOn filter)
-- [ ] Test empty result handling (no records in date range)
-- [ ] Verify extraction stops at maxRecords limit
-
-### 3. Mapping Testing
-
-- [ ] Verify required fields are populated
-- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
-- [ ] Test custom resolvers with edge cases (if any)
-- [ ] Verify nested field extraction
-- [ ] Test with null/missing fields
-- [ ] Verify mapping error collection works
-
-### 4. CSV Generation Testing
-
-- [ ] Verify CSV structure matches expected format
-- [ ] Test CSV validation against schema (if applicable)
-- [ ] Verify header row is present and correct
-- [ ] Test with large datasets (>1000 records)
-- [ ] Verify UTF-8 encoding
-- [ ] Test special character handling (commas, quotes, newlines)
-
-### 5. SFTP Upload Testing
-
-- [ ] Test SFTP connection and authentication
-- [ ] Verify file upload to correct path
-- [ ] Test file naming convention (timestamp format)
-- [ ] Verify file permissions on SFTP server
-- [ ] Test upload retry logic (simulate network failure)
-- [ ] Verify SFTP connection disposal (no connection leaks)
-
-### 6. State Management Testing
-
-- [ ] Verify overlap buffer prevents missed records (60-second default)
-- [ ] Test state recovery after extraction failure
-- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
-- [ ] Test first run with no previous state (uses fallbackStartDate)
-- [ ] Verify state update only happens on successful upload
-- [ ] Test manual date override (doesn't update state)
-
-### 7. Job Tracking Testing
-
-- [ ] Test job creation with JobTracker
-- [ ] Verify job status updates at each stage
-- [ ] Test job completion with metadata
-- [ ] Test job failure handling
-- [ ] Query job status via webhook endpoint
-- [ ] Verify job status persists in KV store
-
-### 8. Error Handling Testing
-
-- [ ] Test with invalid GraphQL query
-- [ ] Test with mapping errors (invalid field paths)
-- [ ] Test with SFTP connection failures
-- [ ] Test with authentication failures
-- [ ] Test with network timeouts
-- [ ] Verify error logging includes context (jobId, stage, error details)
-- [ ] Test error threshold logic (if applicable)
-
-### 9. Staging Environment Testing
-
-- [ ] Run full extraction in staging environment
-- [ ] Verify CSV file format with downstream system
-- [ ] Monitor extraction duration and resource usage
-- [ ] Test with production-like data volumes
-- [ ] Verify no performance degradation over time
-
-### 10. Integration Testing
-
-- [ ] Test scheduled workflow (cron trigger)
-- [ ] Test ad hoc webhook trigger
-- [ ] Test job status query webhook
-- [ ] Verify activation variables are read correctly
-- [ ] Test with different extraction modes (incremental, date range)
-- [ ] End-to-end test: trigger ? extract ? transform ? upload ? verify file
-
----
-## Monitoring & Alerting
-
-### Success Response Example
-
-```json
-{
-  "success": true,
-  "jobId": "SCHEDULED_FUL_20251102_140000_abc123",
-  "recordsExtracted": 1523,
-  "fileName": "fulfillments-2025-11-02T14-00-00-000Z.csv",
-  "sftpPath": "/outbound/fulfillments/fulfillments-2025-11-02T14-00-00-000Z.csv",
-  "metrics": {
-    "extractionDurationMs": 12543,
-    "totalPages": 8,
-    "pageSize": 200,
-    "mappingErrors": 0,
-    "fileSizeBytes": 524288,
-    "uploadDurationMs": 1234
-  },
-  "timestamps": {
-    "extractionStart": "2025-11-02T14:00:00.000Z",
-    "extractionEnd": "2025-11-02T14:00:12.543Z",
-    "uploadComplete": "2025-11-02T14:00:13.777Z"
-  },
-  "state": {
-    "previousTimestamp": "2025-11-02T13:00:00.000Z",
-    "newTimestamp": "2025-11-02T13:59:58.123Z",
-    "stateUpdated": true,
-    "overlapBufferSeconds": 60
-  }
-}
-```
-
-### Error Response Example
-
-```json
-{
-  "success": false,
-  "jobId": "ADHOC_FUL_20251102_140500_xyz789",
-  "error": "SFTP upload failed: Connection timeout",
-  "errorCategory": "NETWORK",
-  "recordsExtracted": 0,
-  "stage": "sftp_upload",
-  "details": {
-    "message": "Failed to upload file after 3 retry attempts",
-    "retryAttempts": 3,
-    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
-  },
-  "state": {
-    "stateUpdated": false,
-    "willRetryNextRun": true,
-    "note": "State not advanced - next extraction will retry same time window"
-  }
-}
-```
-
-### Key Metrics to Track
-
-```typescript
-const METRICS = {
-  // Extraction Performance
-  extractionDurationMs: Date.now() - extractionStart,
-  recordCount: records.length,
-  pageCount: extractionResult.stats.totalPages,
-  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
-
-  // Transformation Performance
-  transformedCount: transformedRecords.length,
-  failedCount: mappingErrors.length,
-  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
-
-  // File Generation
-  fileSizeMB: (csvContent.length / (1024 * 1024)).toFixed(2),
-
-  // Upload Performance
-  uploadDurationMs: uploadEnd - uploadStart,
-  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
-
-  // State Management
-  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
-  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
-};
-
-log.info('Extraction metrics', metrics);
-```
-
-### Alert Thresholds
-
-```typescript
-const ALERT_THRESHOLDS = {
-  // Duration Alerts
-  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
-  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
-  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
-
-  // Error Rate Alerts
-  MAX_ERROR_RATE: 0.05, // 5% mapping errors
-  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
-
-  // Volume Alerts
-  MAX_RECORDS_PER_RUN: 100000,
-  MIN_RECORDS_WARNING: 0, // Alert if no records found
-  MAX_FILE_SIZE_MB: 150, // 150MB
-
-  // State Alerts
-  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
-  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
-};
-
-// Check thresholds
-if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
-  log.warn('Extraction duration exceeded threshold', {
-    duration: metrics.extractionDurationMs,
-    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
-    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
-  });
-}
-```
-
-### Monitoring Dashboard Queries
-
-**Versori Platform Logs Query:**
-
-```
-# Successful extractions
-log_level:info AND message:"Extraction complete" AND jobId:*
-
-# Failed extractions
-log_level:error AND message:"Extraction workflow failed" AND jobId:*
-
-# Performance issues
-extractionDurationMs:>300000 OR uploadDurationMs:>120000
-
-# High error rates
-errorRate:>5
-
-# State management issues
-stateUpdated:false AND success:true
-```
-
-### Common Issues and Solutions
-
-**Issue**: "Extraction timeout after 10 minutes"
-
-- **Cause**: Too many records in single extraction
-- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
-- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
-
-**Issue**: "Mapping errors for 50% of records"
-
-- **Cause**: Schema mismatch between GraphQL response and mapping config
-- **Fix**: Run schema validation, update mapping config paths
-- **Prevention**: Use `npx fc-connect validate-schema` before deployment
-
-**Issue**: "SFTP connection timeout"
-
-- **Cause**: Network issues, firewall, or connection pool exhaustion
-- **Fix**: Check SFTP credentials, verify network connectivity
-- **Prevention**: Implement connection health checks, monitor connection status
-
-**Issue**: "State not updating after successful extraction"
-
-- **Cause**: KV write failure or intentional retry logic
-- **Fix**: Check KV logs, verify state update code executed
-- **Prevention**: Add KV write verification, log state updates explicitly
-
-**Issue**: "First run exceeds record limits"
-
-- **Cause**: No previous timestamp, fetches all historical records
-- **Fix**: Set fallbackStartDate close to current date, apply additional filters
-- **Prevention**: Use appropriate fallbackStartDate for initial runs
-
-**Issue**: "Excessive duplicate records in output"
-
-- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
-- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
-- **Prevention**: Monitor duplicate rates, verify state update logic
-
----
-
-## Troubleshooting Quick Reference
-
-| Error Message | Likely Cause | Solution |
-|--------------|--------------|----------|
-| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
-| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
-| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
-| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
-| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
-| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
-| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
-| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
-| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
-| "CSV generation failed" | Format-specific error | Check CSV generation logic, validate output |
-
----
+---
+template_id: tpl-extract-fulfillments-to-sftp-csv
+canonical_filename: template-extraction-fulfillments-to-sftp-csv.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: extraction
+source: fluent-graphql
+destination: sftp-csv
+entity: fulfillments
+format: csv
+logging: versori
+status: stable
+features:
+  - memory-management
+  - enhanced-logging
+  - pagination-progress
+  - dispose-finally
+---
+
+# Template: Extraction - Fulfillments to SFTP CSV
+
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Memory Management** - Clear large result sets after processing batches
+- ✅ **Enhanced Logging** - Pagination progress tracking with emoji indicators (📊, 📥, ✅)
+- ✅ **Pagination Progress** - Real-time page-by-page progress logging with metrics
+- ✅ **Resource Cleanup** - SFTP dispose in finally blocks prevents connection leaks
+
+---
+
+## 📚 STEP 1: Load These Docs (Human Checklist)
+
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/extraction/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+---
+
+## 📋 STEP 2: Tell Your AI (Prompt)
+
+Copy/paste this prompt into your AI tool after loading the documentation above:
+
+```
+I need a Versori scheduled extractor that:
+
+1) Queries Fluent Commerce GraphQL for Fulfillments with auto-pagination
+2) Supports incremental runs via KV state (with an overlap buffer)
+3) Transforms results using UniversalMapper per mapping JSON
+4) Generates CSV and uploads to SFTP
+5) Tracks progress with JobTracker and exposes a job-status webhook
+6) Uses native Versori log (LoggingService removed - use native log)
+
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+
+---
+
+## 📋 Template Overview
+
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP credentials, schedule, page size/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, GraphQL selection set/filters, validators/resolvers) are adjusted in code as needed. It extracts fulfillments from Fluent Commerce via GraphQL, transforms the data into CSV, and uploads the result to SFTP.
+
+### What This Template Does
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    EXTRACTION WORKFLOW                           │
+└─────────────────────────────────────────────────────────────────┘
+
+1. TRIGGER
+   ├─ Scheduled (Cron): Runs automatically every 4 hours
+   ├─ Ad hoc (Webhook): Manual trigger with optional date override
+   └─ Status Query (Webhook): Check job progress
+
+2. EXTRACT (ExtractionOrchestrator)
+   ├─ Query Fluent GraphQL API for fulfillments
+   ├─ Auto-pagination (handles large datasets)
+   ├─ Apply date filters (incremental or manual range)
+   └─ Validate each record (optional)
+
+3. TRANSFORM (UniversalMapper)
+   ├─ Map GraphQL fields to CSV schema
+   ├─ Apply SDK resolvers (trim, uppercase, formatDateShort, etc.)
+   ├─ Extract nested data (order.customer.email)
+   └─ Handle transformation errors
+
+4. GENERATE CSV (CSVParserService)
+   ├─ Convert transformed records to CSV
+   ├─ Include headers
+   ├─ Handle special characters
+   └─ Generate timestamped filename
+
+5. UPLOAD (SftpDataSource)
+   ├─ Connect to SFTP server
+   ├─ Upload CSV file with retry logic
+   ├─ Verify upload success
+   └─ CRITICAL: Call dispose() in finally block
+
+6. TRACK JOB (JobTracker)
+   ├─ Create job with unique ID
+   ├─ Update status at each step
+   ├─ Store job result in KV
+   └─ Enable status queries via webhook
+
+7. UPDATE STATE (VersoriKVAdapter)
+   ├─ Calculate max updatedOn from records
+   ├─ Store timestamp for next incremental run
+   ├─ Apply overlap buffer (prevent missed records)
+   └─ Skip update if manual date override
+```
+
+### Key Features
+
+- Job tracking with status queries
+- Execution modes: scheduled, ad hoc, status query
+- Uses ExtractionOrchestrator, UniversalMapper, JobTracker, CSVParserService
+- Error handling, retry logic
+- Reusable services suitable for similar use cases
+
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+
+### 📦 Package Information
+
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk@^0.1.39
+```
+
+---
+
+**Templates are designed for direct deployment; customize via activation variables.**
+
+---
+
+## 📦 SDK Imports (Verified - Versori Optimized)
+
+```typescript
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  CSVParserService,
+  SftpDataSource,
+} from '@fluentcommerce/fc-connect-sdk';
+
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+
+**? VERSORI PLATFORM - Use Native Logs:**
+
+- Use `log` from context: `const { log } = ctx;`
+- Don't import or use LoggingService for Versori connectors
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+
+---
+
+## ⚙️ Configuration
+
+### SFTP Connection Setup (Recommended)
+
+**? BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
+
+**Connection Configuration:**
+
+1. In Versori platform, create a connection named `versori_ftp_server`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+5. The SDK will automatically decode the credentials using:
+
+```typescript
+// Get SFTP credentials from Versori connection (Basic Auth)
+// RECOMMENDED: Use activation.connections (already decoded)
+const allConnections = ctx.activation.connections || [];
+const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+if (!sftpConn) {
+  throw new Error('SFTP connection "versori_ftp_server" not found');
+}
+
+const credential = sftpConn.credentials[0]?.credential;
+if (!credential?.data?.basicAuth) {
+  throw new Error('SFTP connection not configured with Basic Authentication');
+}
+
+const { username, password } = credential.data.basicAuth;
+// ? Already decoded - no Buffer.from() needed!
+```
+
+**Why use connections instead of activation variables?**
+
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection can be reused across workflows
+- ✅ No need to manage sensitive data in activation variables
+- ✅ Easier credential rotation
+- ✅ Better separation of concerns (config vs secrets)
+- ✅ Follows industry security best practices
+
+### Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+```json
+{
+  "retailerId": "your-retailer-id",
+  "sftpHost": "sftp.partner.com",
+  "sftpPort": 22,
+  "sftpPrivateKey": "-----BEGIN PRIVATE KEY-----...-----END PRIVATE KEY-----",
+  "sftpRemotePath": "/incoming/fulfillments/",
+  "fileNamePrefix": "fulfillments",
+  "pageSize": 200,
+  "maxRecords": 10000,
+  "overlapBufferSeconds": 60,
+  "fulfillmentStatuses": "SHIPPED,DELIVERED",
+  "validateConnectionOnStart": "false"
+}
+```
+
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` connection (see above).
+
+Note: Webhook security is enforced via Versori connections. Configure auth on the connection and reference it in `webhook({ connection: '...' })`.
+
+### Variable Explanations
+
+| Variable                     | Purpose                     | Default                   | Customization Hints              |
+| ---------------------------- | --------------------------- | ------------------------- | -------------------------------- |
+| `retailerId`                 | Fluent retailer ID          | -                         | Required - your retailer ID      |
+| **SFTP Credentials**         | _From Connection_           |                           | _See connection setup above_     |
+| `sftpHost`                   | SFTP server hostname        | -                         | Required - partner SFTP server   |
+| `sftpPort`                   | SFTP server port            | `22`                      | Standard SFTP port               |
+| `sftpPrivateKey`             | SFTP private key (optional) | -                         | Alternative to password auth     |
+| `sftpRemotePath`             | SFTP upload directory       | `/incoming/fulfillments/` | Customize folder structure       |
+| `fileNamePrefix`             | CSV filename prefix         | `fulfillments`            | Customize naming convention      |
+| `pageSize`                   | Records per GraphQL page    | `200`                     | Increase for fewer API calls     |
+| `maxRecords`                 | Total extraction limit      | `50000`                   | Safety limit - adjust for volume |
+| `overlapBufferSeconds`       | Incremental safety window   | `60`                      | Prevents missed records          |
+| `fulfillmentStatuses`        | Status filter (comma-sep)   | `SHIPPED,DELIVERED`       | Filter by fulfillment status     |
+| `validateConnectionOnStart`  | Validate auth on startup    | `false`                   | `true` = fail-fast, `false` = fast startup |
+
+---
+
+### 📋 State Management & Incremental Sync
+
+**How incremental sync works:**
+
+1. **First Run:** Uses `DEFAULT_FALLBACK` date (2024-01-01T00:00:00Z)
+2. **Subsequent Runs:** Uses `lastFulfillmentSync` timestamp from KV store
+3. **Overlap Buffer:** Subtracts 60 seconds to catch late-arriving records
+4. **State Update:** After successful upload, stores max `updatedOn` for next run
+
+**Incremental vs Manual Modes:**
+
+| Mode                        | When to Use          | State Update | Payload Example                                                |
+| --------------------------- | -------------------- | ------------ | -------------------------------------------------------------- |
+| **Incremental**             | Daily scheduled sync | ? Yes       | `{}` (empty - uses last sync)                                  |
+| **Manual Range**            | Historical backfill  | ❌ No        | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": false }` |
+| **Manual Range with State** | One-time catch-up    | ? Yes       | `{ "fromDate": "2024-01-01T00:00:00Z", "updateState": true }`  |
+
+**Why overlap buffer?**
+
+Records updated near the sync time might not appear in the query due to:
+
+- Clock drift between systems
+- Transaction timing in the database
+- GraphQL query execution timing
+
+The 60-second buffer ensures these edge-case records are captured in the next run, preventing data loss.
+
+**When to skip state update (`updateState: false`):**
+
+- Historical backfills (don't affect ongoing incremental sync)
+- Testing/debugging specific date ranges
+- Reprocessing old data without changing the sync pointer
+
+---
+
+### 📁 SFTP Configuration
+
+**Note:** SFTP credentials and paths are configured separately.
+
+- **SFTP Host:** Configured via `sftpHost` activation variable
+- **SFTP Path:** Configured via `sftpRemotePath` activation variable (e.g., `/incoming/fulfillments/`)
+- **Filename Pattern:** Configured via `fileNamePrefix` activation variable (e.g., `fulfillments`)
+
+**The workflow generates files like:** `{sftpRemotePath}{fileNamePrefix}-{timestamp}.csv`
+
+**Example:** With `sftpRemotePath="/incoming/fulfillments/"` and `fileNamePrefix="fulfillments"`, a generated file will look like:
+
+```
+/incoming/fulfillments/fulfillments-2025-10-27T18-30-45Z.csv
+```
+
+**Note:** To change the upload folder, modify the `sftpRemotePath` activation variable (not in code).
+
+**CRITICAL:** Always call `sftp.dispose()` in a finally block to close connections properly.
+
+---
+
+### Auto-pagination and limits (ExtractionOrchestrator)
+
+**What:** ExtractionOrchestrator handles GraphQL Relay cursor-based pagination automatically.
+
+**Why:** Prevents manual pagination loop code, handles large datasets efficiently.
+
+**How:** You configure `pageSize` and `maxRecords`; the orchestrator injects `$first` and `$after` variables automatically and loops until `pageInfo.hasNextPage === false` or `maxRecords` is reached.
+
+**Critical:** Your query MUST include `edges { cursor }` and `pageInfo { hasNextPage }` fields, or pagination will fail.
+
+#### Configuration Parameters
+
+| Parameter        | Purpose                        | Example                     | Effect                     |
+| ---------------- | ------------------------------ | --------------------------- | -------------------------- |
+| `pageSize`       | Records per GraphQL request    | `200`                       | Controls `first` variable  |
+| `maxRecords`     | Total extraction limit         | `50000`                     | Hard stop across all pages |
+| `resultPath`     | Where records live in response | `"fulfillments.edges.node"` | Flattening path            |
+| `validateItem()` | Optional record filter         | `(item) => !!item.ref`      | Skips invalid records      |
+
+#### GraphQL Query Requirements
+
+**Your query MUST include these pagination fields:**
+
+```graphql
+query GetFulfillments(
+  $retailerId: ID!
+  $dateRangeFilter: DateRange
+  $statuses: [String!]
+  $first: Int! # ← Orchestrator injects this
+  $after: String # ← Orchestrator injects this
+) {
+  fulfillments(
+    retailerId: $retailerId
+    updatedOn: $dateRangeFilter
+    status: $statuses
+    first: $first # ← Pagination page size
+    after: $after # ← Pagination cursor
+  ) {
+    edges {
+      # ← REQUIRED: Relay connection structure
+      node {
+        # ← REQUIRED: Actual records here
+        id
+        ref
+        # ... your fields
+      }
+      cursor # ← REQUIRED: Orchestrator uses this for next page
+    }
+    pageInfo {
+      # ← REQUIRED: Orchestrator checks this
+      hasNextPage # ← REQUIRED: Tells orchestrator to continue
+    }
+  }
+}
+```
+
+---
+
+### Pattern: Backward Pagination (Optional - Advanced)
+
+**Use Case**: Extract data in reverse chronological order (newest to oldest) instead of oldest to newest.
+
+**When to Use**:
+
+- ✅ Need most recent records first (e.g., latest orders, recent inventory updates)
+- ✅ Time-bounded reverse traversal for auditing
+- ✅ Display newest-first in UI/reports
+- ❌ **Don't use for standard incremental sync** - use forward pagination (default)
+
+**GraphQL Query Requirements**:
+
+Your query must support backward pagination by including `$last` and `$before`:
+
+```graphql
+query GetData(
+  $retailerId: ID!
+  $first: Int # For forward pagination
+  $after: String # For forward pagination
+  $last: Int # For backward pagination
+  $before: String # For backward pagination
+) {
+  data(retailerId: $retailerId, first: $first, after: $after, last: $last, before: $before) {
+    edges {
+      cursor # ? REQUIRED
+      node {
+        id
+        createdAt
+        # ... other fields
+      }
+    }
+    pageInfo {
+      hasNextPage # For forward
+      hasPreviousPage # ? REQUIRED for backward
+    }
+  }
+}
+```
+
+**Implementation**:
+
+```typescript
+// Backward pagination - newest records first
+const result = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    retailerId,
+    dateRangeFilter: { from: bufferedLastRunTime, to: effectiveEndTime },
+    // ❌ Don't include last/before - orchestrator injects them
+  },
+  pageSize: 200,
+  direction: 'backward', // ? Enable reverse pagination
+  maxRecords: 10000,
+});
+
+// Records are returned in reverse chronological order
+console.log(result.data[0].createdAt); // Newest
+console.log(result.data[result.data.length - 1].createdAt); // Oldest (within range)
+```
+
+**Key Differences from Forward Pagination**:
+
+| Aspect                 | Forward (Default)                | Backward                |
+| ---------------------- | -------------------------------- | ----------------------- |
+| **Direction**          | `direction: 'forward'` (default) | `direction: 'backward'` |
+| **Variables Injected** | `first`, `after`                 | `last`, `before`        |
+| **PageInfo Field**     | `hasNextPage`                    | `hasPreviousPage`       |
+| **Cursor Source**      | Last edge of page                | First edge of page      |
+| **Record Order**       | Oldest → Newest                | Newest → Oldest       |
+
+**Important Notes**:
+
+1. **Orchestrator injects variables**: Don't pass `last` or `before` in your variables object - the orchestrator injects them based on `pageSize` and cursor tracking.
+
+2. **Query signature**: Your GraphQL query must declare `$last` and `$before` parameters even if you don't pass them explicitly.
+
+3. **PageInfo requirement**: Response must include `pageInfo.hasPreviousPage` or the orchestrator will throw an error.
+
+4. **Cursor requirement**: Each edge must include `cursor` field for pagination to work.
+
+**Example: Extract Latest 1000 Orders**
+
+```typescript
+const latestOrders = await orchestrator.extract({
+  query: ORDERS_QUERY,
+  resultPath: 'orders.edges.node',
+  variables: {
+    retailerId,
+    statuses: ['BOOKED', 'ALLOCATED'],
+  },
+  direction: 'backward', // Start from newest
+  maxRecords: 1000, // Stop after 1000 records
+  pageSize: 100, // 100 per page = 10 pages
+});
+
+// latestOrders.data[0] is the newest order
+// latestOrders.data[999] is the 1000th newest order
+```
+
+**When to Use Forward vs Backward**:
+
+```typescript
+// ? Forward (default) - For incremental sync
+const incrementalData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  variables: {
+    dateRangeFilter: { from: lastSyncTime, to: now },
+  },
+  // direction defaults to 'forward'
+  // Processes oldest → newest for proper sequencing
+});
+
+// ? Backward - For "latest N records" use cases
+const latestData = await orchestrator.extract({
+  query: YOUR_QUERY,
+  resultPath: 'data.edges.node',
+  direction: 'backward',
+  maxRecords: 100, // Just get latest 100
+  // Gets newest → oldest
+});
+```
+
+**Pagination Variables Reference**:
+
+| Variable | Forward     | Backward    | Injected By  | Notes                    |
+| -------- | ----------- | ----------- | ------------ | ------------------------ |
+| `first`  | ? Used     | ❌ Not used | Orchestrator | From `pageSize`          |
+| `after`  | ? Used     | ❌ Not used | Orchestrator | From cursor (last edge)  |
+| `last`   | ❌ Not used | ? Used     | Orchestrator | From `pageSize`          |
+| `before` | ❌ Not used | ? Used     | Orchestrator | From cursor (first edge) |
+
+**Common Mistakes to Avoid**:
+
+```typescript
+// ❌ WRONG - Don't pass pagination variables
+const result = await orchestrator.extract({
+  variables: {
+    last: 200, // ❌ Orchestrator will override this
+    before: cursor, // ❌ Orchestrator manages cursor
+  },
+  direction: 'backward',
+});
+
+// ? CORRECT - Let orchestrator inject pagination
+const result = await orchestrator.extract({
+  variables: {
+    retailerId, // ? Your business variables only
+  },
+  pageSize: 200, // ? Orchestrator uses this for last/before
+  direction: 'backward',
+});
+```
+
+---
+
+## 📄 Mapping Configuration
+
+**File:** `config/fulfillments.export.csv.json`
+
+```json
+{
+  "name": "fulfillments.export.csv",
+  "version": "1.0.0",
+  "description": "Fulfillments → CSV Export Mapping",
+  "fields": {
+    "order_number": {
+      "source": "orderRef",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "fulfillment_id": {
+      "source": "ref",
+      "required": true,
+      "resolver": "sdk.trim"
+    },
+    "tracking_number": {
+      "source": "trackingNumber",
+      "required": false,
+      "resolver": "sdk.trim"
+    },
+    "carrier": {
+      "source": "carrier",
+      "required": false,
+      "resolver": "sdk.uppercase"
+    },
+    "service_level": {
+      "source": "serviceLevel",
+      "required": false,
+      "resolver": "sdk.trim"
+    },
+    "status": {
+      "source": "status",
+      "required": true,
+      "resolver": "sdk.uppercase"
+    },
+    "ship_date": {
+      "source": "shippedOn",
+      "required": false,
+      "resolver": "sdk.formatDateShort"
+    },
+    "delivery_date": {
+      "source": "deliveredOn",
+      "required": false,
+      "resolver": "sdk.formatDateShort"
+    },
+    "customer_email": {
+      "source": "order.customer.email",
+      "required": false,
+      "resolver": "sdk.trim"
+    },
+    "last_updated": {
+      "source": "updatedOn",
+      "required": true,
+      "resolver": "sdk.toString"
+    }
+  }
+}
+```
+
+**AI Customization Hints:**
+
+- Add fields: Copy existing field config, change `source` path
+- Remove fields: Delete field from config
+- Change resolvers: Replace `sdk.trim` with `sdk.uppercase`, etc.
+- Nested fields: Use dot notation like `order.customer.email`
+
+Note: Customize mapping by editing the JSON above; prefer built-in resolvers. See SDK Universal Mapping guide for advanced usage.
+
+---
+
+## Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+fulfillments-extraction/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-fulfillments-extraction.ts  # Scheduled: Daily extraction
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-fulfillments-extraction.ts  # Webhook: Manual trigger
+    │       └── job-status-check.ts               # Webhook: Status query
+    │
+    ├── services/
+    │   └── fulfillments-extraction.service.ts    # Shared orchestration logic (reusable)
+    │
+    └── config/
+        └── fulfillments.export.csv.json         # Mapping configuration
+```
+
+
+### 1. Entry Point (`index.ts`)
+
+**Pattern:** MemoryInterpreter (export all workflows from single entry point)
+
+**Benefits:**
+- ✅ Versori platform discovers all workflows automatically
+- ✅ Clean separation of concerns (entry point vs workflow logic)
+- ✅ Easy to add/remove workflows (just update exports)
+- ✅ Single source of truth for workflow registration
+
+```typescript
+// ═══════════════════════════════════════════════════════════
+// 🚀 VERSORI FULFILLMENTS EXTRACTION
+// ═══════════════════════════════════════════════════════════
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * This file is the entry point for the Versori deployment.
+ * It registers three workflows:
+ * 1. Scheduled extraction (runs automatically)
+ * 2. Ad hoc webhook (manual trigger)
+ * 3. Job status webhook (query progress)
+ *
+ * AI CUSTOMIZATION:
+ * - Add new workflows by importing and registering them
+ * - Remove workflows by commenting out registration
+ * - Change workflow names in import statements
+ */
+// ═══════════════════════════════════════════════════════════
+
+import {
+  scheduledFulfillmentsExtraction,
+  adhocFulfillmentsExtraction,
+  fulfillmentsJobStatus,
+} from './workflows/fulfillments-extraction';
+
+// Register workflows with Versori platform
+// The platform will expose these as executable endpoints
+
+export {
+  scheduledFulfillmentsExtraction, // Cron-based auto-run
+  adhocFulfillmentsExtraction, // Manual webhook trigger
+  fulfillmentsJobStatus, // Job status query
+};
+```
+
+---
+
+### 2. Workflows (src/workflows/fulfillments-extraction.ts)
+
+```typescript
+/**
+ * Workflows - Defines 3 execution patterns for fulfillments extraction
+ *
+ * WORKFLOW 1: Scheduled (Cron) - Runs automatically every 4 hours
+ * WORKFLOW 2: Ad hoc (Webhook) - Manual trigger with optional date override
+ * WORKFLOW 3: Job Status (Webhook) - Query job progress
+ *
+ * AI CUSTOMIZATION HINTS:
+ * - Change schedule: Modify cron expression in schedule()
+ * - Add filtering: Pass additional params to executeFulfillmentExtraction()
+ * - Change response format: Modify return object structure
+ */
+
+import { schedule, webhook, http, fn } from '@versori/run';
+import { executeFulfillmentExtraction, getJobStatus } from '../services/extraction-orchestration';
+import { generateJobId } from '../utils/job-id-generator';
+
+/**
+ * WORKFLOW 1: Scheduled Extraction
+ *
+ * Purpose: Automated extraction every 4 hours for incremental sync
+ * Trigger: Cron schedule (every 4 hours at minute 0)
+ * State Update: Always updates lastSync timestamp
+ *
+ * AI CUSTOMIZATION:
+ * - Change schedule: Modify the cron expression string
+ *   Examples:
+ *   - Every hour: '0 * * * *'
+ *   - Every 30 min: '*/30 * * * *'
+ *   - Daily at 2 AM: '0 2 * * *'
+ */
+export const scheduledFulfillmentsExtraction = schedule(
+  'fulfillments-scheduled',
+  '0 */4 * * *'  // ← CUSTOMIZE: Cron expression
+)
+.then(
+  http('execute-scheduled-extraction', { connection: 'fluent_commerce' }, async (ctx) => {
+    const { log } = ctx;
+
+    // Generate unique job ID for tracking
+    // Format: SCHEDULED_FULFILLMENTS_YYYYMMDD_HHMMSS_random
+    const jobId = generateJobId('SCHEDULED', 'FULFILLMENTS');
+
+    log.info('Scheduled extraction triggered', { jobId });
+
+    try {
+      // Execute main workflow (extraction → transform → upload)
+      const result = await executeFulfillmentExtraction(ctx, {
+        jobId,
+        triggeredBy: 'schedule',
+        updateState: true,        // Always update state for scheduled runs
+      });
+
+      log.info('Scheduled extraction completed', {
+        jobId,
+        recordCount: result.recordsExtracted,
+        fileName: result.fileName
+      });
+
+      return result;
+
+    } catch (error: any) {
+      log.error('Scheduled extraction failed', {
+        jobId,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'Error',
+      });
+      throw error;
+    }
+  })
+);
+
+/**
+ * WORKFLOW 2: Ad hoc Extraction (Manual Trigger)
+ *
+ * Purpose: Manual extraction with optional date range override
+ * Trigger: Webhook POST to /webhooks/fulfillments-adhoc
+ * State Update: Optional (controlled by request payload)
+ *
+ * WEBHOOK PAYLOAD EXAMPLES:
+ *
+ * 1. Incremental (use last sync timestamp):
+ *    {}
+ *
+ * 2. Date range (manual override):
+ *    {
+ *      "fromDate": "2025-01-01T00:00:00Z",
+ *      "toDate": "2025-01-31T23:59:59Z",
+ *      "updateState": false
+ *    }
+ *
+ * AI CUSTOMIZATION:
+ * - Add request validation
+ * - Add authentication check
+ * - Add custom filters from payload
+ */
+export const adhocFulfillmentsExtraction = webhook(
+  'fulfillments-adhoc',
+  { connection: 'fulfillments-adhoc', response: { mode: 'sync' } }
+)
+.then(
+  http('execute-adhoc-extraction', { connection: 'fluent_commerce' }, async (ctx) => {
+    const { data, log } = ctx;
+
+    // Generate unique job ID
+    const jobId = generateJobId('ADHOC', 'FULFILLMENTS');
+
+    // SECURITY: Authentication is enforced by Versori connection configuration
+    // Configure auth on the connection and reference it in webhook({ connection: '...' })
+
+    // Extract optional date override from webhook payload
+    const fromDate = data.fromDate as string | undefined;
+    const toDate = data.toDate as string | undefined;
+    const updateState = data.updateState === true;  // Default false; advance state only if explicitly true
+
+    log.info('Ad hoc extraction triggered via webhook', {
+      jobId,
+      hasDateOverride: !!fromDate,
+      fromDate: fromDate || 'not specified',
+      toDate: toDate || 'not specified',
+      updateState
+    });
+
+    try {
+      // Execute main workflow with optional overrides
+      const result = await executeFulfillmentExtraction(ctx, {
+        jobId,
+        triggeredBy: 'webhook',
+        fromDate,              // Optional: override start date
+        toDate,                // Optional: override end date
+        updateState,           // Optional: skip state update for historical queries
+      });
+
+      log.info('Ad hoc extraction completed', {
+        jobId,
+        recordCount: result.recordsExtracted,
+        fileName: result.fileName,
+        isManualOverride: !!fromDate,
+        stateUpdated: result.stateUpdated
+      });
+
+      // Return success with job details
+      return {
+        success: true,
+        jobId,
+        recordsExtracted: result.recordsExtracted,
+        fileName: result.fileName,
+        sftpPath: result.sftpPath,
+        statusUrl: `/webhooks/fulfillments-job-status?jobId=${jobId}`,
+        dateRange: fromDate ? {
+          from: fromDate,
+          to: toDate || 'not specified',
+          updateState
+        } : undefined
+      };
+
+    } catch (error: any) {
+      log.error('Ad hoc extraction failed', {
+        jobId,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'Error',
+      });
+
+      return {
+        success: false,
+        jobId,
+        error: error instanceof Error ? error.message : String(error)
+      };
+    }
+  })
+);
+
+/**
+ * WORKFLOW 3: Job Status Query
+ *
+ * Purpose: Check job progress and status
+ * Trigger: Webhook GET/POST to /webhooks/fulfillments-job-status?jobId=xxx
+ * Returns: Current job status, stage, progress
+ *
+ * QUERY EXAMPLES:
+ *
+ * 1. HTTP GET:
+ *    GET /webhooks/fulfillments-job-status?jobId=ADHOC_FULFILLMENTS_20251027_183045_abc123
+ *
+ * 2. HTTP POST:
+ *    POST /webhooks/fulfillments-job-status
+ *    { "jobId": "ADHOC_FULFILLMENTS_20251027_183045_abc123" }
+ */
+export const fulfillmentsJobStatus = webhook(
+  'fulfillments-job-status',
+  { connection: 'fulfillments-job-status', response: { mode: 'sync' } }
+)
+.then(
+  fn('query-job-status', async (ctx) => {
+    const { data, log, openKv, activation } = ctx;
+    const req = ctx.request();
+
+    // SECURITY: Authentication is enforced by Versori connection configuration
+    // Configure auth on the connection and reference it in webhook({ connection: '...' })
+
+    // Get jobId from query param or POST body
+    const jobId = data.jobId as string;
+
+    if (!jobId) {
+      log.error('Job ID not provided in request');
+      return {
+        success: false,
+        error: 'Job ID is required. Provide jobId in query param or request body.'
+      };
+    }
+
+    log.info('Querying job status', { jobId });
+
+    try {
+      // Query job status from KV store
+      const status = await getJobStatus(openKv(':project:'), jobId, log);
+
+      if (!status) {
+        log.info('Job not found', { jobId });
+        return {
+          success: false,
+          error: 'Job not found',
+          jobId
+        };
+      }
+
+      log.info('Job status retrieved', { jobId, status: status.status });
+
+      return {
+        success: true,
+        jobId,
+        ...status
+      };
+
+    } catch (error: any) {
+      log.error('Failed to query job status', {
+        jobId,
+        message: error instanceof Error ? error.message : String(error),
+        stack: error instanceof Error ? error.stack : undefined,
+        errorType: error instanceof Error ? error.constructor.name : 'Error',
+      });
+
+      return {
+        success: false,
+        jobId,
+        error: error instanceof Error ? error.message : String(error)
+      };
+    }
+  })
+);
+```
+
+---
+
+## 3. Main Orchestration Service (src/services/extraction-orchestration.ts)
+
+```typescript
+/**
+ * MAIN ORCHESTRATION SERVICE
+ *
+ * This is the heart of the extraction workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Determine date range (incremental vs manual)
+ * 3. Extract data using ExtractionOrchestrator
+ * 4. Transform using UniversalMapper
+ * 5. Generate CSV using CSVParserService
+ * 6. Upload to SFTP
+ * 7. Track job progress with JobTracker
+ * 8. Update state for next run
+ *
+ * NAMING PATTERN (consistent across all use cases):
+ * - Interface: {Entity}ExtractionParams (e.g., FulfillmentExtractionParams)
+ * - Result: {Entity}ExtractionResult (e.g., FulfillmentExtractionResult)
+ * - Main function: execute{Entity}Extraction (e.g., executeFulfillmentExtraction)
+ *
+ * AI CUSTOMIZATION HINTS:
+ * - Change entity: Replace "Fulfillment" with "Order", "Product", etc.
+ * - Change output: Replace CSVParserService with XMLBuilder
+ * - Change destination: Replace SftpDataSource with S3DataSource
+ * - Add steps: Insert new service calls between existing steps
+ */
+
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  ExtractionOrchestrator,
+  JobTracker,
+  UniversalMapper,
+  CSVParserService,
+  SftpDataSource,
+} from '@fluentcommerce/fc-connect-sdk';
+
+import mappingConfig from '../../config/fulfillments.export.csv.json' with { type: 'json' };
+
+// ? VERSORI PLATFORM: Use native log from context, not LoggingService
+
+/**
+ * Parameters for extraction workflow
+ *
+ * NAMING: {Entity}ExtractionParams
+ */
+export interface FulfillmentExtractionParams {
+  jobId: string;
+  triggeredBy: 'schedule' | 'webhook';
+  fromDate?: string; // Optional: manual date override
+  toDate?: string; // Optional: manual date override
+  updateState: boolean; // Whether to update rawLastRunTime timestamp
+
+  // AI CUSTOMIZATION: Add filters specific to entity
+  fulfillmentStatuses?: string[]; // e.g., ['SHIPPED', 'DELIVERED']
+  retailerId?: string; // e.g., 'YOUR_RETAILER_ID'
+}
+
+/**
+ * Result from extraction workflow
+ *
+ * NAMING: {Entity}ExtractionResult
+ */
+export interface FulfillmentExtractionResult {
+  success: boolean;
+  jobId: string;
+  recordsExtracted: number;
+  fileName?: string;
+  sftpPath?: string;
+  error?: string;
+  errors?: any[];
+  isManualOverride?: boolean;
+  stateUpdated?: boolean;
+  newTimestamp?: string;
+}
+
+/**
+ * GraphQL Query for Fulfillments
+ *
+ * NAMING: {ENTITY}_EXTRACTION_QUERY (uppercase constant)
+ */
+const FULFILLMENTS_EXTRACTION_QUERY = `
+  query GetFulfillments(
+    $updatedAfter: DateTime!
+    $statuses: [String!]
+    $first: Int!
+    $after: String
+  ) {
+    fulfillments(
+      updatedOn: { after: $updatedAfter }
+      status: $statuses
+      first: $first
+      after: $after
+    ) {
+      edges {
+        node {
+          id
+          ref
+          orderRef
+          status
+          trackingNumber
+          carrier
+          serviceLevel
+          shippedOn
+          deliveredOn
+          updatedOn
+          order {
+            customer {
+              email
+            }
+          }
+        }
+        cursor
+      }
+      pageInfo {
+        hasNextPage
+      }
+    }
+  }
+`;
+
+/**
+ * Query job status from KV store
+ *
+ * ? VERSORI PLATFORM: Uses native log from context (passed as parameter)
+ */
+export async function getJobStatus(
+  kv: any, // ? Versori KV (compatible with JobTracker's KVAdapter interface)
+  jobId: string,
+  log: any // ? Native Versori log from context
+): Promise<any | undefined> {
+  try {
+    const tracker = new JobTracker(kv, log);
+    return await tracker.getJob(jobId);
+  } catch (error: any) {
+    log.error('Failed to get job status', {
+      jobId,
+      message: error instanceof Error ? error.message : String(error),
+      stack: error instanceof Error ? error.stack : undefined,
+      errorType: error instanceof Error ? error.constructor.name : 'Error',
+    });
+    return undefined;
+  }
+}
+
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * NAMING: execute{Entity}Extraction (e.g., executeFulfillmentExtraction)
+ *
+ * This function implements the complete workflow in steps.
+ * Each step is clearly commented for AI understanding.
+ */
+export async function executeFulfillmentExtraction(
+  ctx: any,
+  params: FulfillmentExtractionParams
+): Promise<FulfillmentExtractionResult> {
+  // ? VERSORI PLATFORM: Extract native log from context
+  const { log, openKv, activation } = ctx;
+  const { jobId, triggeredBy, fromDate, toDate, updateState } = params;
+
+  // Open KV store for state management and job tracking
+  // ? Pass raw Versori KV directly - it matches KVAdapter interface
+  // ? Pass native log to JobTracker
+  const kv = openKv(':project:');
+  const tracker = new JobTracker(kv, log);
+
+  try {
+    // ═══════════════════════════════════════════════════════════
+    // STEP 1/8: Initialize Job Tracking
+    // ═══════════════════════════════════════════════════════════
+    log.info('🚀 [STEP 1/8] Initializing job tracking', { jobId });
+
+    await tracker.createJob(jobId, {
+      triggeredBy,
+      hasDateOverride: !!fromDate,
+      fromDate,
+      toDate,
+      updateStateAfterRun: updateState,
+    });
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 2/8: Initialize Fluent Client
+    // ═══════════════════════════════════════════════════════════
+    log.info('🚀 [STEP 2/8] Initializing Fluent Commerce client', { jobId });
+
+    // ✅ Optional: Validate connection immediately (fail-fast mode)
+    // Set activation variable 'validateConnectionOnStart' = 'true' to enable
+    // When enabled: Executes query { me { ref } } to verify authentication
+    // When disabled: Fast creation, validation happens on first API call (default)
+    const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
+    const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
+
+    if (!client) {
+      throw new Error('Failed to create Fluent Commerce client');
+    }
+
+    if (validateConnection) {
+      log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
+    }
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 3/8: Determine Date Range
+    // ═══════════════════════════════════════════════════════════
+    log.info('🔍 [STEP 3/8] Determining date range for extraction', { jobId });
+
+    // State key for incremental sync tracking
+    // NAMING: last{Entity}Sync (e.g., lastFulfillmentSync)
+    const STATE_KEY = 'lastFulfillmentSync';
+    const DEFAULT_FALLBACK = '2024-01-01T00:00:00Z';
+    const OVERLAP_BUFFER_SECONDS = parseInt(
+      activation.getVariable('overlapBufferSeconds') || '60',
+      10
+    );
+
+    let bufferedLastRunTime: string;
+    const isManualOverride = !!fromDate;
+
+    if (isManualOverride) {
+      // Manual date override from webhook
+      bufferedLastRunTime = fromDate!;
+      log.info('Using manual date override', { fromDate, toDate });
+    } else {
+      // Incremental sync - get last sync timestamp
+      const rawLastRunTime = (await kv.get<string>(STATE_KEY)) || DEFAULT_FALLBACK;
+
+      // Apply overlap buffer (prevents missed records)
+      bufferedLastRunTime = new Date(
+        new Date(rawLastRunTime).getTime() - OVERLAP_BUFFER_SECONDS * 1000
+      ).toISOString();
+
+      log.info('Using incremental sync with overlap buffer', {
+        rawLastRunTime,
+        bufferedLastRunTime,
+        overlapBufferSeconds: OVERLAP_BUFFER_SECONDS,
+      });
+    }
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 4/8: Extract Data (ExtractionOrchestrator)
+    // ═══════════════════════════════════════════════════════════
+    log.info('📥 [STEP 4/8] Extracting data from Fluent Commerce', { jobId });
+
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'extraction',
+      message: 'Extracting data with auto-pagination',
+    });
+
+    // Configure extraction
+    const pageSize = parseInt(activation.getVariable('pageSize') || '200', 10);
+    const maxRecords = parseInt(activation.getVariable('maxRecords') || '10000', 10);
+
+    // Parse status filter
+    const fulfillmentStatuses =
+      params.fulfillmentStatuses ||
+      (activation.getVariable('fulfillmentStatuses') || 'SHIPPED,DELIVERED').split(',');
+
+    // Initialize ExtractionOrchestrator
+    const orchestrator = new ExtractionOrchestrator(client, log);
+
+    // ? Enhanced: Extract context for progress logging
+    const dateRangeInfo = {
+      start: bufferedLastRunTime || 'N/A',
+      end: new Date().toISOString(),
+      statuses: fulfillmentStatuses.join(', ') || 'all'
+    };
+
+    // Duration tracking
+    const extractionStartTime = Date.now();
+
+    // ? Enhanced: Start logging with context
+    log.info(`🔍 [ExtractionOrchestrator] Starting extraction`, {
+     query: 'fulfillments',
+     pageSize,
+     maxRecords,
+     dateRange: `from ${dateRangeInfo.start}`,
+     statuses: dateRangeInfo.statuses,
+     jobId
+    });
+
+    // Execute extraction with auto-pagination
+    const extractionResult = await orchestrator.extract({
+      query: FULFILLMENTS_EXTRACTION_QUERY,
+      resultPath: 'fulfillments.edges.node',
+      variables: {
+        updatedAfter: bufferedLastRunTime,
+        statuses: fulfillmentStatuses,
+        // Note: Don't include 'first' or 'after' here; orchestrator injects them
+      },
+      pageSize,
+      maxRecords,
+      // Optional: validate each record
+      validateItem: (item: any) => {
+        return !!(item.ref && item.orderRef);
+      },
+    });
+
+    const rawRecords = extractionResult.data;
+
+    // Calculate extraction duration
+    const extractionDuration = Date.now() - extractionStartTime;
+
+    log.info('✅ [STEP 4/8] Extraction completed', {
+      jobId,
+      totalRecords: extractionResult.stats.totalRecords,
+      totalPages: extractionResult.stats.totalPages,
+      validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+      errors: extractionResult.errors ? extractionResult.errors.length : 0,
+      duration: `${extractionDuration}ms`,
+    });
+
+    // ? Enhanced: Completion logging with summary
+    log.info(`✅ [ExtractionOrchestrator] Extraction completed`, {
+     totalRecords: extractionResult.stats.totalRecords,
+     totalPages: extractionResult.stats.totalPages,
+     validRecords: extractionResult.stats.validRecords ?? rawRecords.length,
+     failedValidations: extractionResult.stats.failedValidations,
+     truncated: extractionResult.stats.truncated,
+     truncationReason: extractionResult.stats.truncationReason,
+     dateRange: `from ${dateRangeInfo.start}`,
+     duration: `${extractionDuration}ms`,
+     jobId
+    });
+
+    if (extractionResult.errors && extractionResult.errors.length > 0) {
+      log.warn('Non-fatal extraction errors encountered', {
+        jobId,
+        errorCount: extractionResult.errors.length,
+        sampleErrors: extractionResult.errors.slice(0, 3),
+      });
+    }
+
+    // Handle empty result
+    if (rawRecords.length === 0) {
+      log.info('No records to process');
+
+      // Update state even with no records (prevents re-querying empty window)
+      if (updateState) {
+        await kv.set(STATE_KEY, new Date().toISOString());
+      }
+
+      await tracker.markCompleted(jobId, {
+        recordCount: 0,
+        message: 'No records to extract',
+      });
+
+      return {
+        success: true,
+        jobId,
+        recordsExtracted: 0,
+      };
+    }
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 5/8: Transform Data (UniversalMapper)
+    // ═══════════════════════════════════════════════════════════
+    log.info('🔄 [STEP 5/8] Transforming data with UniversalMapper', {
+      jobId,
+      recordCount: rawRecords.length,
+    });
+
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'transformation',
+      message: `Transforming ${rawRecords.length} records`,
+    });
+
+    const mapper = new UniversalMapper(mappingConfig);
+    const mappingResult = await mapper.map(rawRecords);
+
+    if (!mappingResult.success) {
+      const mappingErrors = mappingResult.errors || ['Unknown mapping failure'];
+      log.error('[STEP 5/8] Mapping failed - terminating job', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+
+      await tracker.markFailed(
+        jobId,
+        new Error(mappingErrors[0] || 'UniversalMapper returned unsuccessful result')
+      );
+
+      return {
+        success: false,
+        jobId,
+        recordsExtracted: 0,
+        error: `Transformation failed: ${mappingErrors[0] || 'Unknown error'}`,
+      };
+    }
+
+    const transformedRecords = Array.isArray(mappingResult.data) ? mappingResult.data : [];
+    const mappingErrors = mappingResult.errors || [];
+
+    if (mappingErrors.length > 0) {
+      log.warn('[STEP 5/8] Some records failed transformation', {
+        jobId,
+        errorCount: mappingErrors.length,
+        sampleErrors: mappingErrors.slice(0, 3),
+      });
+    }
+
+    if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+      log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+        jobId,
+        skippedFields: mappingResult.skippedFields,
+        note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+      });
+    }
+
+    if (transformedRecords.length === 0) {
+      throw new Error('All records failed transformation');
+    }
+
+    log.info('Transformation complete', {
+      jobId,
+      successful: transformedRecords.length,
+      skippedRecords: rawRecords.length - transformedRecords.length,
+    });
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 6/8: Generate CSV (CSVParserService)
+    // ═══════════════════════════════════════════════════════════
+    log.info('📄 [STEP 6/8] Generating CSV file', { jobId });
+
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'csv_generation',
+      message: `Generating CSV for ${transformedRecords.length} records`,
+    });
+
+    // Initialize CSVParserService
+    const csvParser = new CSVParserService();
+
+    // Generate CSV content
+    const csvContent = csvParser.stringify(transformedRecords, { headers: true });
+
+    // Generate filename using helper function
+    const fileNamePrefix = activation.getVariable('fileNamePrefix') || 'fulfillments';
+    const extractFileName = (prefix: string) => {
+      const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
+      return `${prefix}-${timestamp}.csv`;
+    };
+    const fileName = extractFileName(fileNamePrefix);
+
+    log.info('✅ CSV file generated', {
+      fileName,
+      sizeBytes: csvContent.length,
+      recordCount: transformedRecords.length,
+    });
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 7/8: Upload to SFTP (SftpDataSource)
+    // ═══════════════════════════════════════════════════════════
+    log.info('📤 [STEP 7/8] Uploading to SFTP', { jobId, fileName });
+
+    await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'sftp_upload',
+      message: `Uploading ${fileName} to SFTP`,
+    });
+
+    // Get SFTP credentials from Versori connection (Basic Auth)
+    // RECOMMENDED: Use activation.connections (already decoded)
+    const allConnections = ctx.activation.connections || [];
+    const sftpConn = allConnections.find(c => c.name === 'versori_ftp_server');
+
+    if (!sftpConn) {
+      throw new Error('SFTP connection "versori_ftp_server" not found');
+    }
+
+    const credential = sftpConn.credentials[0]?.credential;
+    if (!credential?.data?.basicAuth) {
+      throw new Error('SFTP connection not configured with Basic Authentication');
+    }
+
+    const { username, password } = credential.data.basicAuth;
+    // ? Already decoded - no Buffer.from() needed!
+
+    // Get other SFTP config from activation variables
+    const sftpConfig = {
+      host: activation.getVariable('sftpHost'),
+      port: parseInt(activation.getVariable('sftpPort') || '22', 10),
+      username, // From connection
+      password, // From connection
+      privateKey: activation.getVariable('sftpPrivateKey'), // Optional: if using key-based auth
+    };
+    const sftpRemotePath = activation.getVariable('sftpRemotePath') || '/incoming/fulfillments/';
+
+    // Validate SFTP config
+    if (!sftpConfig.host || !sftpConfig.username) {
+      throw new Error('SFTP configuration incomplete: missing host or username from connection');
+    }
+
+    if (!sftpConfig.password && !sftpConfig.privateKey) {
+      throw new Error(
+        'SFTP configuration incomplete: missing password from connection or privateKey from activation'
+      );
+    }
+
+    // Initialize SFTP data source
+    // ? VERSORI PLATFORM: Pass native log from context
+    // ? Declare as let for proper disposal in finally block
+    let sftp: SftpDataSource | undefined;
+
+    // Construct SFTP path
+    const sftpPath = `${sftpRemotePath}${fileName}`;
+
+    try {
+      sftp = new SftpDataSource(
+        {
+          type: 'SFTP_CSV',
+          connectionId: 'fulfillments-sftp',
+          name: 'Fulfillments SFTP Upload',
+          settings: {
+            host: sftpConfig.host!,
+            port: sftpConfig.port,
+            username: sftpConfig.username!,
+            password: sftpConfig.password,
+            privateKey: sftpConfig.privateKey,
+            remotePath: sftpRemotePath,
+            filePattern: '*.csv',
+          },
+        },
+        log
+      );
+
+      // Create remote directory if needed
+      await sftp.createDirectory(sftpRemotePath, true);
+
+      // Upload with retry logic (built into SftpDataSource)
+      await sftp.uploadFile(sftpPath, Buffer.from(csvContent, 'utf-8'), {
+        createDirectories: true,
+      });
+
+      log.info('✅ SFTP upload successful', { fileName, remotePath: sftpPath });
+    } finally {
+      // ?? CRITICAL: Always dispose SFTP connection
+      if (sftp) {
+        await sftp.dispose();
+        log.info('🔌 SFTP connection disposed');
+      }
+    }
+
+    // ═══════════════════════════════════════════════════════════
+    // STEP 8/8: Update State & Complete Job
+    // ═══════════════════════════════════════════════════════════
+    log.info('💾 [STEP 8/8] Updating state and completing job', { jobId });
+
+    // Calculate new timestamp for next incremental run
+    let newTimestamp: string | undefined;
+
+    if (updateState) {
+      // Find max updatedOn from extracted records
+      const maxUpdatedOn = rawRecords.reduce((max, record) => {
+        const recordTime = new Date(record.updatedOn).getTime();
+        return recordTime > max ? recordTime : max;
+      }, new Date(bufferedLastRunTime).getTime());
+
+      newTimestamp = new Date(maxUpdatedOn).toISOString();
+
+      // Store new timestamp (WITHOUT buffer - buffer only applied on read)
+      await kv.set(STATE_KEY, newTimestamp);
+
+      log.info('✅ State updated', {
+        oldTimestamp: bufferedLastRunTime,
+        newTimestamp,
+      });
+    }
+
+    // Mark job as completed
+    await tracker.markCompleted(jobId, {
+      recordCount: transformedRecords.length,
+      fileName,
+      sftpPath,
+      errorCount: mappingErrors.length,
+      isManualOverride,
+      stateUpdated: updateState,
+      newTimestamp,
+    });
+
+    log.info('✅ [COMPLETE] Extraction workflow completed successfully', { jobId });
+
+    return {
+      success: true,
+      jobId,
+      recordsExtracted: transformedRecords.length,
+      fileName,
+      sftpPath,
+      isManualOverride,
+      stateUpdated: updateState,
+      newTimestamp,
+      errors: mappingErrors.length > 0 ? mappingErrors : undefined,
+    };
+  } catch (error: any) {
+    log.error('❌ [FATAL] Extraction workflow failed', {
+      jobId,
+      message: error instanceof Error ? error.message : String(error),
+      stack: error instanceof Error ? error.stack : undefined,
+      errorType: error instanceof Error ? error.constructor.name : 'Error',
+      recommendation: 'Check logs for detailed error information. Verify activation variables, connection credentials, and SFTP configuration.',
+    });
+
+    // Mark job as failed
+    await tracker.markFailed(jobId, error);
+
+    return {
+      success: false,
+      jobId,
+      recordsExtracted: 0,
+      error: error instanceof Error ? error.message : String(error),
+    };
+  }
+}
+```
+
+---
+
+## 4. Utility Functions (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking extraction workflows
+ *
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
+ * Example: SCHEDULED_FULFILLMENTS_20251027_183045_a1b2c3
+ */
+
+/**
+ * Generate unique job ID
+ *
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
+ * @param entity - Entity abbreviation (FULFILLMENTS, VP, ORD, PRD)
+ * @returns Unique job ID string
+ */
+export function generateJobId(type: string, entity: string): string {
+  const now = new Date();
+
+  // Format: YYYYMMDD
+  const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
+
+  // Format: HHMMSS
+  const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
+
+  // Random suffix (6 chars)
+  const randomStr = Math.random().toString(36).substring(2, 8);
+
+  return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
+}
+
+/**
+ * Parse job ID components
+ */
+export function parseJobId(jobId: string): {
+  type: string;
+  entity: string;
+  date: string;
+  time: string;
+  random: string;
+} | null {
+  const parts = jobId.split('_');
+
+  if (parts.length !== 5) {
+    return null;
+  }
+
+  return {
+    type: parts[0],
+    entity: parts[1],
+    date: parts[2],
+    time: parts[3],
+    random: parts[4],
+  };
+}
+```
+
+---
+
+## 5. Package Configuration
+
+### package.json
+
+```json
+{
+  "name": "fulfillments-to-sftp-csv",
+  "version": "1.0.0",
+  "description": "Extract fulfillments from Fluent Commerce and export to SFTP as CSV",
+  "type": "module",
+  "main": "src/index.ts",
+  "scripts": {
+    "dev": "versori dev",
+    "build": "versori build",
+    "deploy": "versori deploy"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+
+### tsconfig.json
+
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+
+---
+
+## 6. Deployment Instructions
+
+### Deploy to Versori
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Test locally (if using Versori CLI)
+npm run dev
+
+# 3. Deploy to Versori platform
+npm run deploy
+```
+
+### Configure SFTP Connection
+
+1. Create a Versori connection named `versori_ftp_server`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter your SFTP **Username** and **Password**
+
+### Configure Activation Variables
+
+In Versori platform settings, configure:
+
+```json
+{
+  "retailerId": "your-retailer-id",
+  "sftpHost": "sftp.partner.com",
+  "sftpPort": 22,
+  "sftpRemotePath": "/incoming/fulfillments/",
+  "fileNamePrefix": "fulfillments",
+  "pageSize": 200,
+  "maxRecords": 50000,
+  "overlapBufferSeconds": 60,
+  "fulfillmentStatuses": "SHIPPED,DELIVERED",
+  "webhookApiKey": "your-secure-api-key-here"
+}
+```
+
+> **Note:** `sftpUsername` and `sftpPassword` are now stored in the `versori_ftp_server` connection.
+
+---
+
+## 7. Testing
+
+### Test Scheduled Extraction
+
+The scheduled workflow runs automatically based on cron schedule.
+
+**Check logs:**
+
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client
+[STEP 3/8] Determining date range for extraction
+[STEP 4/8] Extracting data from Fluent Commerce
+[STEP 5/8] Transforming data with UniversalMapper
+[STEP 6/8] Generating CSV file
+[STEP 7/8] Uploading to SFTP
+[STEP 8/8] Updating state and completing job
+```
+
+### Test Ad hoc Extraction
+
+```bash
+# Incremental (uses last sync timestamp)
+curl -X POST https://api.versori.com/webhooks/fulfillments-adhoc \
+  -H "X-API-Key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{}'
+
+# Date range override
+curl -X POST https://api.versori.com/webhooks/fulfillments-adhoc \
+  -H "X-API-Key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "fromDate": "2025-01-01T00:00:00Z",
+    "toDate": "2025-01-31T23:59:59Z",
+    "updateState": false
+  }'
+```
+
+### Test Job Status Query
+
+```bash
+curl -X POST https://api.versori.com/webhooks/fulfillments-job-status \
+  -H "X-API-Key: your-api-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "jobId": "ADHOC_FULFILLMENTS_20251027_183045_abc123"
+  }'
+```
+
+---
+
+## Testing Checklist
+
+**Before production deployment:**
+
+### 1. Schema Validation
+
+- [ ] Run `npx fc-connect introspect-schema --url <your-graphql-url>`
+- [ ] Run `npx fc-connect validate-schema --mapping ./config/fulfillments.export.csv.json --schema ./fluent-schema.json`
+- [ ] Run `npx fc-connect analyze-coverage --mapping ./config/fulfillments.export.csv.json --schema ./fluent-schema.json`
+- [ ] Verify all `source` paths in mapping exist in GraphQL schema
+- [ ] Verify query structure matches schema (fields, types, filters)
+
+### 2. Extraction Testing
+
+- [ ] Test with small dataset first (maxRecords=10)
+- [ ] Verify ExtractionOrchestrator pagination works correctly
+- [ ] Test with multiple pages of data (verify cursor handling)
+- [ ] Verify date range filtering (updatedOn filter)
+- [ ] Test empty result handling (no records in date range)
+- [ ] Verify extraction stops at maxRecords limit
+
+### 3. Mapping Testing
+
+- [ ] Verify required fields are populated
+- [ ] Verify SDK resolvers work correctly (sdk.trim, sdk.parseInt, sdk.formatDate, etc.)
+- [ ] Test custom resolvers with edge cases (if any)
+- [ ] Verify nested field extraction
+- [ ] Test with null/missing fields
+- [ ] Verify mapping error collection works
+
+### 4. CSV Generation Testing
+
+- [ ] Verify CSV structure matches expected format
+- [ ] Test CSV validation against schema (if applicable)
+- [ ] Verify header row is present and correct
+- [ ] Test with large datasets (>1000 records)
+- [ ] Verify UTF-8 encoding
+- [ ] Test special character handling (commas, quotes, newlines)
+
+### 5. SFTP Upload Testing
+
+- [ ] Test SFTP connection and authentication
+- [ ] Verify file upload to correct path
+- [ ] Test file naming convention (timestamp format)
+- [ ] Verify file permissions on SFTP server
+- [ ] Test upload retry logic (simulate network failure)
+- [ ] Verify SFTP connection disposal (no connection leaks)
+
+### 6. State Management Testing
+
+- [ ] Verify overlap buffer prevents missed records (60-second default)
+- [ ] Test state recovery after extraction failure
+- [ ] Verify timestamp saved WITHOUT buffer (MAX(updatedOn))
+- [ ] Test first run with no previous state (uses fallbackStartDate)
+- [ ] Verify state update only happens on successful upload
+- [ ] Test manual date override (doesn't update state)
+
+### 7. Job Tracking Testing
+
+- [ ] Test job creation with JobTracker
+- [ ] Verify job status updates at each stage
+- [ ] Test job completion with metadata
+- [ ] Test job failure handling
+- [ ] Query job status via webhook endpoint
+- [ ] Verify job status persists in KV store
+
+### 8. Error Handling Testing
+
+- [ ] Test with invalid GraphQL query
+- [ ] Test with mapping errors (invalid field paths)
+- [ ] Test with SFTP connection failures
+- [ ] Test with authentication failures
+- [ ] Test with network timeouts
+- [ ] Verify error logging includes context (jobId, stage, error details)
+- [ ] Test error threshold logic (if applicable)
+
+### 9. Staging Environment Testing
+
+- [ ] Run full extraction in staging environment
+- [ ] Verify CSV file format with downstream system
+- [ ] Monitor extraction duration and resource usage
+- [ ] Test with production-like data volumes
+- [ ] Verify no performance degradation over time
+
+### 10. Integration Testing
+
+- [ ] Test scheduled workflow (cron trigger)
+- [ ] Test ad hoc webhook trigger
+- [ ] Test job status query webhook
+- [ ] Verify activation variables are read correctly
+- [ ] Test with different extraction modes (incremental, date range)
+- [ ] End-to-end test: trigger ? extract ? transform ? upload ? verify file
+
+---
+## Monitoring & Alerting
+
+### Success Response Example
+
+```json
+{
+  "success": true,
+  "jobId": "SCHEDULED_FUL_20251102_140000_abc123",
+  "recordsExtracted": 1523,
+  "fileName": "fulfillments-2025-11-02T14-00-00-000Z.csv",
+  "sftpPath": "/outbound/fulfillments/fulfillments-2025-11-02T14-00-00-000Z.csv",
+  "metrics": {
+    "extractionDurationMs": 12543,
+    "totalPages": 8,
+    "pageSize": 200,
+    "mappingErrors": 0,
+    "fileSizeBytes": 524288,
+    "uploadDurationMs": 1234
+  },
+  "timestamps": {
+    "extractionStart": "2025-11-02T14:00:00.000Z",
+    "extractionEnd": "2025-11-02T14:00:12.543Z",
+    "uploadComplete": "2025-11-02T14:00:13.777Z"
+  },
+  "state": {
+    "previousTimestamp": "2025-11-02T13:00:00.000Z",
+    "newTimestamp": "2025-11-02T13:59:58.123Z",
+    "stateUpdated": true,
+    "overlapBufferSeconds": 60
+  }
+}
+```
+
+### Error Response Example
+
+```json
+{
+  "success": false,
+  "jobId": "ADHOC_FUL_20251102_140500_xyz789",
+  "error": "SFTP upload failed: Connection timeout",
+  "errorCategory": "NETWORK",
+  "recordsExtracted": 0,
+  "stage": "sftp_upload",
+  "details": {
+    "message": "Failed to upload file after 3 retry attempts",
+    "retryAttempts": 3,
+    "lastError": "ETIMEDOUT: Connection timed out after 30000ms"
+  },
+  "state": {
+    "stateUpdated": false,
+    "willRetryNextRun": true,
+    "note": "State not advanced - next extraction will retry same time window"
+  }
+}
+```
+
+### Key Metrics to Track
+
+```typescript
+const METRICS = {
+  // Extraction Performance
+  extractionDurationMs: Date.now() - extractionStart,
+  recordCount: records.length,
+  pageCount: extractionResult.stats.totalPages,
+  avgRecordsPerPage: records.length / extractionResult.stats.totalPages,
+
+  // Transformation Performance
+  transformedCount: transformedRecords.length,
+  failedCount: mappingErrors.length,
+  errorRate: ((mappingErrors.length / records.length) * 100).toFixed(2) + '%',
+
+  // File Generation
+  fileSizeMB: (csvContent.length / (1024 * 1024)).toFixed(2),
+
+  // Upload Performance
+  uploadDurationMs: uploadEnd - uploadStart,
+  uploadSpeedMBps: (fileSizeMB / (uploadDurationMs / 1000)).toFixed(2),
+
+  // State Management
+  timeSinceLastRun: Date.now() - new Date(lastTimestamp).getTime(),
+  recordsPerMinute: (records.length / (extractionDurationMs / 60000)).toFixed(0),
+};
+
+log.info('Extraction metrics', metrics);
+```
+
+### Alert Thresholds
+
+```typescript
+const ALERT_THRESHOLDS = {
+  // Duration Alerts
+  EXTRACTION_DURATION_MS: 5 * 60 * 1000, // 5 minutes
+  UPLOAD_DURATION_MS: 2 * 60 * 1000,      // 2 minutes
+  TOTAL_DURATION_MS: 10 * 60 * 1000,      // 10 minutes
+
+  // Error Rate Alerts
+  MAX_ERROR_RATE: 0.05, // 5% mapping errors
+  MAX_VALIDATION_FAILURES: 0.02, // 2% validation failures
+
+  // Volume Alerts
+  MAX_RECORDS_PER_RUN: 100000,
+  MIN_RECORDS_WARNING: 0, // Alert if no records found
+  MAX_FILE_SIZE_MB: 150, // 150MB
+
+  // State Alerts
+  MAX_TIME_SINCE_LAST_RUN_HOURS: 25, // Alert if >25 hours (should run hourly)
+  MAX_OVERLAP_BUFFER_SECONDS: 300,   // Alert if buffer >5 minutes
+};
+
+// Check thresholds
+if (metrics.extractionDurationMs > ALERT_THRESHOLDS.EXTRACTION_DURATION_MS) {
+  log.warn('Extraction duration exceeded threshold', {
+    duration: metrics.extractionDurationMs,
+    threshold: ALERT_THRESHOLDS.EXTRACTION_DURATION_MS,
+    recommendation: 'Consider reducing maxRecords or increasing extraction frequency'
+  });
+}
+```
+
+### Monitoring Dashboard Queries
+
+**Versori Platform Logs Query:**
+
+```
+# Successful extractions
+log_level:info AND message:"Extraction complete" AND jobId:*
+
+# Failed extractions
+log_level:error AND message:"Extraction workflow failed" AND jobId:*
+
+# Performance issues
+extractionDurationMs:>300000 OR uploadDurationMs:>120000
+
+# High error rates
+errorRate:>5
+
+# State management issues
+stateUpdated:false AND success:true
+```
+
+### Common Issues and Solutions
+
+**Issue**: "Extraction timeout after 10 minutes"
+
+- **Cause**: Too many records in single extraction
+- **Fix**: Reduce maxRecords, increase extraction frequency, or optimize query filters
+- **Prevention**: Monitor recordCount trends, set appropriate maxRecords
+
+**Issue**: "Mapping errors for 50% of records"
+
+- **Cause**: Schema mismatch between GraphQL response and mapping config
+- **Fix**: Run schema validation, update mapping config paths
+- **Prevention**: Use `npx fc-connect validate-schema` before deployment
+
+**Issue**: "SFTP connection timeout"
+
+- **Cause**: Network issues, firewall, or connection pool exhaustion
+- **Fix**: Check SFTP credentials, verify network connectivity
+- **Prevention**: Implement connection health checks, monitor connection status
+
+**Issue**: "State not updating after successful extraction"
+
+- **Cause**: KV write failure or intentional retry logic
+- **Fix**: Check KV logs, verify state update code executed
+- **Prevention**: Add KV write verification, log state updates explicitly
+
+**Issue**: "First run exceeds record limits"
+
+- **Cause**: No previous timestamp, fetches all historical records
+- **Fix**: Set fallbackStartDate close to current date, apply additional filters
+- **Prevention**: Use appropriate fallbackStartDate for initial runs
+
+**Issue**: "Excessive duplicate records in output"
+
+- **Cause**: Overlap buffer (expected) or timestamp not saved correctly
+- **Fix**: Verify newTimestamp saved WITHOUT buffer, check state persistence
+- **Prevention**: Monitor duplicate rates, verify state update logic
+
+---
+
+## Troubleshooting Quick Reference
+
+| Error Message | Likely Cause | Solution |
+|--------------|--------------|----------|
+| "Failed to create Fluent Commerce client" | Authentication failure | Check OAuth2 credentials, verify connection config |
+| "GraphQL query validation error" | Invalid query syntax | Validate query against schema with introspection tool |
+| "Pagination cursor invalid" | Stale cursor or query change | Reset extraction, verify cursor handling in query |
+| "Mapping failed: field not found" | Schema mismatch | Run schema validation, update mapping paths |
+| "SFTP authentication failed" | Invalid credentials | Verify SFTP credentials in activation variables |
+| "Connection pool exhausted" | Too many concurrent requests | Reduce concurrency, increase pool size |
+| "KV operation failed" | Versori KV issue | Check Versori platform status, retry operation |
+| "Job status not found" | Invalid jobId or expired | Verify jobId format, check KV retention policy |
+| "Memory limit exceeded" | Dataset too large | Reduce maxRecords, enable streaming mode |
+| "CSV generation failed" | Format-specific error | Check CSV generation logic, validate output |
+
+---