npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.54 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (475) hide show

package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-json-product-event.md CHANGED Viewed

@@ -1,2602 +1,2602 @@
----
-template_id: tpl-ingest-sftp-json-to-product-event
-canonical_filename: template-ingestion-sftp-json-product-event.md
-sdk_version: ^0.1.39
-runtime: versori
-direction: ingestion
-source: sftp-json
-destination: fluent-event-api
-entity: product
-format: json
-logging: versori
-status: stable
-features:
-  - batched-events
-  - attribute-transformation
-  - memory-management
-  - json-serialization-handling
-  - enhanced-logging
----
-# Template: Ingestion - SFTP JSON to Product Event
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
----
-## 📋 Implementation Prompt
-```
-I need a Versori scheduled ingestion that:
-1) Discovers JSON files on SFTP with file tracking to skip duplicates
-2) Downloads and parses JSON (standard or JSON Lines format) - NO array normalization
-3) Transforms records with UniversalMapper per mapping JSON
-4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
-5) Archives files to processed/ or errors/ and writes optional rejection report
-6) Tracks progress with JobTracker and exposes a job-status webhook
-7) Uses native Versori log from context
-8) **Refactored with 3 service functions**: processFile(), sendEvents(), writeEventLog()
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
----
-## 📋 Template Overview
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, JSON structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP JSON, transforms it, and sends events to the Fluent Commerce Event API.
-### What This Template Does
-```
-┌────────────────────────────────────────────────────────────────┐
-│          INGESTION WORKFLOW              │
-└────────────────────────────────────────────────────────────────┘
-1. TRIGGER
-  ├─ Scheduled (Cron): Runs automatically every hour
-  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
-  └─ Status Query (Webhook): Check job progress
-2. DISCOVER FILES (SftpDataSource)
-  ├─ List files from SFTP directory
-  ├─ Filter by pattern (products_*.json)
-  ├─ Check file tracking (skip processed)
-  └─ Sort by oldest first
-3. DOWNLOAD & PARSE (JSONParserService)
-  ├─ Download file from SFTP
-  ├─ Parse JSON to JavaScript object
-  ├─ Auto-detect format (JSON vs JSON Lines)
-  └─ Normalize to array structure
-4. TRANSFORM (UniversalMapper)
-  ├─ Map JSON fields to Fluent schema
-  ├─ Apply SDK resolvers (trim, uppercase, etc.)
-  ├─ Direct passthrough for nested objects
-  ├─ Direct passthrough for arrays
-  └─ Collect transformation errors
-5. SEND EVENTS (Event API)
-  ├─ Loop through transformed products
-  ├─ Send UPSERT_PRODUCT event (async)
-  ├─ Track success/failure count
-  └─ Continue on individual failures
-6. ARCHIVE (SftpDataSource)
-  ├─ Move file to processed/ folder
-  ├─ Or move to errors/ if validation failed
-  ├─ Generate timestamped archive name
-  └─ Verify archive success
-7. TRACK STATE (VersoriFileTracker)
-  ├─ Mark file as processed
-  ├─ Store processing metadata
-  ├─ Prevent duplicate processing
-  └─ Track record counts
-8. TRACK JOB (JobTracker)
-  ├─ Update job status at each step
-  ├─ Store final result in KV
-  ├─ Enable status queries via webhook
-  └─ Handle errors gracefully
-```
-### Key Features
-- **Refactored Architecture**: 3 service functions for clean separation of concerns
- - `processFile()`: SftpDataSource + JSONParserService + UniversalMapper (NO array normalization)
- - `sendEvents()`: Loop records, `client.sendEvent()` with per-record error handling
- - `writeEventLog()`: Write logs to SFTP with Buffer
-- **Per-file Processing**: Download → Parse → Map → Send → Archive (one file at a time)
-- **Job tracking with status queries**: JobTracker persists to Versori KV
-- **Execution modes**: scheduled, ad hoc, status query
-- **Uses SftpDataSource, JSONParserService, UniversalMapper, JobTracker**
-- **Error handling, retry logic, and SFTP cleanup**
-- **SFTP Deduplication:** VersoriFileTracker (KV-based state tracking)
-- **Key Difference from S3:** KV state tracking vs physical file movement
-- **File tracking**: VersoriFileTracker prevents duplicates; `forceReprocess` bypasses
-- **Event API**: Per-record failures don't block other records; rejection reports written to `errors/`
-- **SFTP cleanup in finally block**
-- **Auto-detection of JSON vs JSON Lines format**
-- **Buffer import for Deno/Versori compatibility**
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-### 📦 Package Information
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
-```bash
-npm install @fluentcommerce/fc-connect-sdk
-```
----
-**Templates are designed for direct deployment; customize via activation variables.**
----
-## 📦 SDK Imports (Verified - Versori Optimized)
-```typescript
-// ✅ VERIFIED IMPORTS - These match actual SDK exports
-import { Buffer } from 'node:buffer';
-import {
- createClient, // Universal client factory
- SftpDataSource, // SFTP operations with connection pooling
- JSONParserService, // JSON parsing with format auto-detection
- UniversalMapper, // Field mapping with SDK resolvers
- VersoriFileTracker, // File processing state tracker
- JobTracker, // Job status tracking
-} from '@fluentcommerce/fc-connect-sdk';
-import type { FluentClient, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
-// Versori platform imports
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-**✅ VERSORI PLATFORM - Use Native Logs:**
-- Use `log` from context: `const { log } = ctx;`
-- Native Versori logs are simpler and automatically integrated with platform monitoring
----
-## 🔐 SFTP Connection Setup (Recommended)
-**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
-**Connection Configuration:**
-1. In Versori platform, create a connection named `SFTP`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-### Code Implementation
-The workflow retrieves credentials programmatically from the Versori connection:
-```typescript
-import { Buffer } from 'node:buffer';
-// Retrieve SFTP credentials from connection configuration
-log.info('Retrieving SFTP credentials from connection configuration');
-let sftpUsername: string;
-let sftpPassword: string;
-try {
-  // Retrieve credentials from the 'SFTP' connection
-  const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-  if (!sftpCred?.accessToken) {
-    throw new Error('No SFTP credentials found in connection configuration');
-  }
-  // Decode base64 accessToken to get "username:password"
-  const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-  // Split on ':' to extract username and password
-  const parts = rawBasicAuth.split(':');
-  if (parts.length !== 2) {
-    throw new Error('Invalid SFTP credential format - expected username:password');
-  }
-  sftpUsername = parts[0];
-  sftpPassword = parts[1];
-  log.info('SFTP credentials retrieved successfully', {
-    hasUsername: !!sftpUsername,
-    hasPassword: !!sftpPassword,
-    usernameLength: sftpUsername.length,
-    passwordLength: sftpPassword.length,
-  });
-} catch (error: any) {
-  log.error('Failed to retrieve SFTP credentials', {
-    message: error instanceof Error ? error.message : String(error),
-    stack: error instanceof Error ? error.stack : undefined,
-    errorType: error instanceof Error ? error.constructor.name : 'Error'
-  });
-  return {
-    success: false,
-    error: 'Failed to retrieve SFTP credentials from connection configuration',
-    details: error?.message,
-    recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
-  };
-}
-// Use credentials in SftpDataSource
-const sftp = new SftpDataSource({
-  type: 'SFTP_JSON',
-  connectionId: 'sftp-product-sync',
-  name: 'Product Sync SFTP',
-  settings: {
-    host: activation.getVariable('sftpHost'),
-    port: parseInt(activation.getVariable('sftpPort') || '22', 10),
-    username: sftpUsername, // From connection
-    password: sftpPassword, // From connection
-    remotePath: activation.getVariable('sftpRemotePath'),
-    filePattern: activation.getVariable('filePattern'),
-    encoding: 'utf8',
-  },
-}, log);
-```
-### Accessing SFTP Credentials (Alternative Methods)
-Versori provides multiple methods to access connection credentials. The template uses `ctx.credentials().getAccessToken('SFTP')` which matches batch-api templates. Alternative methods include:
-#### Method 1: activation.connections (Simpler Syntax)
-**Alternative approach** - Direct access via `activation.connections`:
-```typescript
-const { activation } = ctx;
-// Access connection object
-const conn = activation.connections.SFTP;
-// Decode Basic Auth credentials
-const rawBasicAuth = Buffer.from(conn.accessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-log.info('SFTP credentials loaded', { username, host: activation.getVariable('sftpHost') });
-```
-**When to use:** Simple synchronous access, single connection scenarios
-#### Method 2: credentials().get() (Dynamic Selection)
-**Alternative approach** - Use `credentials().get()` for dynamic credential selection:
-```typescript
-const { credentials } = ctx;
-// Get specific connection by name
-const sftpCred = await credentials().get('SFTP');
-// Decode Basic Auth credentials
-const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-log.info('SFTP credentials loaded via credentials()', { username });
-```
-**When to use:**
-- Multiple SFTP connections (dev/prod, multiple partners)
-- Dynamic credential selection based on runtime logic
-- Need to list all available connections
----
-## ⚙️ Activation Variables
-**Configuration is driven by activation variables - modify these instead of code:**
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
-```bash
-# SFTP Configuration
-SFTP_HOST=sftp.partner.com
-SFTP_PORT=22
-# SFTP Paths
-SFTP_REMOTE_PATH=/products/incoming
-SFTP_ARCHIVE_PATH=/products/processed
-SFTP_ERROR_PATH=/products/errors
-# File Processing
-FILE_PATTERN=products_*.json
-JSON_FORMAT=json                             # json or jsonlines
-MAX_FILES_PER_RUN=10
-# Product Configuration
-CATALOGUE_REF=PC:MASTER:2
-CATALOGUE_TYPE=MASTER
-# Event API Configuration
-EVENT_NAME=UPSERT_PRODUCT
-EVENT_MODE=async
-EVENT_CONCURRENCY=1
-# Fluent Configuration (via Versori connection)
-# Connection: fluent_commerce (OAuth2)
-FLUENT_RETAILER_ID=1
-# Feature Toggles
-REQUIRE_ABSOLUTE_PATHS=true                    # "true" for AWS Transfer Family, "false" for standard OpenSSH
-ENABLE_FILE_TRACKING=true                      # Enable/disable file tracking (default: true)
-# Event API Performance
-USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
-MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
-MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
-```
-**Security:**
-- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#-sftp-connection-setup-recommended) section below.
-- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
-### Variable Explanations
-| Variable        | Purpose             | Default          | Customization Hints         |
-| ----------------------- | -------------------------------- | ------------------------- | ----------------------------------- |
-| `SFTP_HOST`       | SFTP server hostname       | -             | Required - SFTP server address   |
-| `SFTP_PORT`       | SFTP server port         | `22`           | Standard SFTP port         |
-| `SFTP_REMOTE_PATH`   | Incoming files directory     | `/products/incoming`   | Where new files arrive       |
-| `SFTP_ARCHIVE_PATH`   | Processed files archive     | `/products/processed`   | Where files move after success   |
-| `SFTP_ERROR_PATH`     | Failed files directory      | `/products/errors`    | Where files move after errors    |
-| `FILE_PATTERN`      | File name filter         | `products_*.json`     | Glob pattern for matching files   |
-| `JSON_FORMAT`      | JSON format type         | `json`          | `json` or `jsonlines`        |
-| `CATALOGUE_REF`     | Product catalogue reference   | `PC:MASTER:2`       | Target catalogue in Fluent     |
-| `CATALOGUE_TYPE`     | Catalogue type          | `MASTER`         | Usually MASTER or STANDARD     |
-| `EVENT_NAME`       | Event to send          | `UPSERT_PRODUCT`     | Event name from Rubix        |
-| `EVENT_MODE`       | Event processing mode      | `async`          | async (recommended) or sync     |
-| `MAX_FILES_PER_RUN`    | Max files per execution     | `10`           | Prevent timeout on large batches  |
-| `FLUENT_RETAILER_ID`   | Retailer ID for Event API    | -             | Required - Fluent retailer ID   |
-| `EVENT_CONCURRENCY`   | Event sending concurrency    | `1`            | `1` = sequential, `>1` = parallel (3-10 recommended) |
-| `REQUIRE_ABSOLUTE_PATHS` | Require absolute SFTP paths   | `true`          | Recommended for clarity       |
-| `ENABLE_FILE_TRACKING`  | Enable KV-based file deduplication | `true`        | Set to `false` to disable file tracking |
-| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
-| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100`          | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
-| `MEMORY_BATCH_SIZE`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
----
-## 🔒 SDK Automatic Behaviors (v0.1.40+)
-**The SDK automatically validates and retries for improved reliability:**
-### retailerId Validation
-- **SDK validates** `retailerId` before calling `sendEvent()`
-- **Checks:** `event.retailerId || client.retailerId`
-- **If missing:** Throws `"retailerId is required for Event API..."`
-- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
-### 401 Auth Retry
-- **Automatic retry** for platform auth failures (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s)
-- **Applies to:** All `sendEvent()` calls (async and sync modes)
-- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
-### 5xx Server Retry
-- **Automatic retry** for transient server errors (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
-- **Protects:** Against Fluent API transient failures
-### No Code Changes Required
-- All templates remain compatible
-- Retry logic is automatic and transparent
-- Better error messages guide configuration
-**See:** [Event API Guide](./event-api-guide.md) for complete details
----
-### 📁 SFTP Deduplication Pattern (VersoriFileTracker)
-**CRITICAL: SFTP uses VersoriFileTracker (KV state), NOT physical file movement**
-VersoriFileTracker is **SFTP-specific** and tracks processed files in Versori KV storage. This prevents reprocessing even if files remain in the incoming directory.
-#### How SFTP Deduplication Works
-**4-Step Process:**
-1. **Discovery:** List files from `sftpIncomingPath` matching `filePattern`
-2. **Filter:** Check VersoriFileTracker - skip files already processed
-3. **Process:** Download, parse, transform, and send events
-4. **Track State:** Mark file as processed in KV (prevents future reprocessing)
-#### File Lifecycle Example
-```
-INITIAL STATE:
-sftp://host/products/incoming/products_20250124_001.json
-↓ (workflow discovers file)
-CHECKING STATE:
-- List files from "/products/incoming/"
-- VersoriFileTracker.isProcessed("products_20250124_001.json") → false
-↓ (not processed before)
-PROCESSING:
-- Download: products_20250124_001.json
-- Parse JSON → Map to events → Send to Event API
-↓ (success)
-STATE TRACKING:
-- VersoriFileTracker.markProcessed("products_20250124_001.json")
-- Move file to "/products/processed/products_20250124_001-20250124T183045.json"
-↓ (next run)
-DEDUPLICATION:
-- List files from "/products/incoming/" (file may or may not still be there)
-- VersoriFileTracker.isProcessed("products_20250124_001.json") → true
-- Skip file automatically
-```
-#### Key Differences: SFTP vs S3
-| Aspect          | SFTP (This Template)       | S3 (Other Templates)        |
-| ------------------------ | -------------------------------- | ----------------------------------- |
-| **Deduplication Method** | VersoriFileTracker (KV state)  | Physical file movement       |
-| **State Storage**    | KV store tracks processed files | None needed             |
-| **File Tracking**    | ✅ VersoriFileTracker required  | ❌ NO VersoriFileTracker      |
-| **Reprocessing**     | Clear KV state or forceReprocess | Move files back to incoming/    |
-| **Complexity**      | More complex (state management) | Simpler (no state management)    |
-| **Files After Process** | May remain in incoming/     | Physically moved to processed/   |
-#### Why SFTP Needs VersoriFileTracker
-**SFTP file operations are less atomic than S3:**
-- Files may remain in incoming directory even after processing
-- Partner systems may not clean up files immediately
-- Multiple workflows may access the same SFTP directory
-- Network issues can leave files in inconsistent state
-- **KV state tracking** provides reliable deduplication regardless of file state
-#### Reprocessing Files (If Needed)
-To reprocess a file that was already tracked:
-**Option 1: Clear KV State (Manual)**
-```typescript
-// In Versori console or workflow
-const kv = openKv(':project:');
-const tracker = new VersoriFileTracker(kv, 'product-ingestion');
-await tracker.clearFile('products_20250124_001.json');
-```
-**Option 2: Webhook with forceReprocess**
-```bash
-# Set forceReprocess: true in webhook payload
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "forceReprocess": true,
-  "filePattern": "products_20250124_001.json"
- }'
-```
-**Use cases for reprocessing:**
-- Fixing bad data that was previously ingested
-- Reingesting after mapping config changes
-- Testing with production files
-- Recovering from partial failures
-#### Summary
-✅ **DO:** Use VersoriFileTracker for SFTP deduplication
-✅ **DO:** Move files to processed/ or errors/ directories for housekeeping
-✅ **DO:** Import and use VersoriFileTracker in SFTP templates
-✅ **DO:** Use KV state tracking for reliable deduplication
-❌ **DON'T:** Rely on physical file movement alone (files may remain)
-❌ **DON'T:** Use VersoriFileTracker for S3 templates (different use case)
----
-### 📁 SFTP Path vs File Pattern
-**Critical:** Directory paths and file patterns are configured separately.
-- **Directory Paths:** Configured via activation variables:
- - `sftpIncomingPath` - Where new files arrive (e.g., `/products/incoming`)
- - `sftpArchivePath` - Where successful files are archived (e.g., `/products/processed`)
- - `sftpErrorPath` - Where failed files are moved (e.g., `/products/errors`)
-- **File Pattern:** Configured via `filePattern` activation variable
- - Matches file names (not full paths)
- - Supports wildcards: `*.json`, `products_*.json`, `urgent_*.json`
-**How it works:**
-```typescript
-// The workflow lists files like this:
-sftp.listFiles({
- path: '/products/incoming', // ← Path from activation variable
- pattern: 'products_*.json', // ← Pattern matches file names
-});
-```
-**Example:** With these settings:
-```json
-{
- "sftpIncomingPath": "/products/incoming",
- "filePattern": "products_*.json"
-}
-```
-The workflow will find files like:
-- ✅ `/products/incoming/products_20250124.json`
-- ✅ `/products/incoming/products_urgent.json`
-- ❌ `/products/incoming/orders_20250124.json` (doesn't match pattern)
-- ❌ `/orders/incoming/products_20250124.json` (wrong path)
-**Note:** To process files from different paths, change `sftpIncomingPath` (not `filePattern`).
----
-## 📄 Mapping Configuration
-**File:** `config/products.import.json`
-```json
-{
- "name": "products.import.json",
- "version": "1.0.0",
- "description": "JSON → Product Event Mapping",
- "fields": {
-  "ref": {
-   "source": "ref",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "type": {
-   "source": "type",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "STANDARD"
-  },
-  "status": {
-   "source": "status",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "ACTIVE"
-  },
-  "gtin": {
-   "source": "gtin",
-   "resolver": "sdk.trim"
-  },
-  "name": {
-   "source": "name",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "summary": {
-   "source": "summary"
-  },
-  "categoryRefs": {
-   "source": "categoryRefs"
-  },
-  "price": {
-   "source": "price"
-  },
-  "taxType": {
-   "source": "taxType"
-  },
-  "attributes": {
-   "source": "attributes",
-   "defaultValue": []
-  }
- }
-}
-```
-**Note on JSON Mapping:**
-Unlike CSV (flat structure), JSON often has nested objects and arrays that map directly to Fluent schema:
-```json
-{
- "ref": "G_PROD_001",
- "categoryRefs": ["CAT1", "CAT2"],
- "price": [
-  { "type": "DEFAULT", "currency": "USD", "value": 52.0 },
-  { "type": "SPECIAL", "currency": "USD", "value": 9.0 }
- ],
- "taxType": {
-  "country": "AU",
-  "group": "Tax Group",
-  "tariff": "Tax Tariff"
- }
-}
-```
-These fields can use **direct passthrough** (no custom resolvers needed):
-```json
-{
- "categoryRefs": { "source": "categoryRefs" },
- "price": { "source": "price" },
- "taxType": { "source": "taxType" }
-}
-```
-For advanced patterns, see SDK Universal Mapping guide.
----
-## Expected JSON Format
-### Standard JSON (Array of Objects)
-```json
-[
- {
-  "ref": "G_PROD_WITH_NO_STANDARD",
-  "type": "VARIANT",
-  "status": "ACTIVE",
-  "gtin": "MH01-XS-Orange",
-  "name": "Chaz Kangeroo Hoodie-XS-Orange main",
-  "summary": "<p>test short description</p>",
-  "categoryRefs": ["STANDARD_CATEGORY"],
-  "price": [
-   { "type": "DEFAULT", "currency": "USD", "value": 52.0 },
-   { "type": "SPECIAL", "currency": "USD", "value": 9.0 }
-  ],
-  "taxType": {
-   "country": "AU",
-   "group": "Tax Group",
-   "tariff": "Tax Tariff"
-  }
- },
- {
-  "ref": "G_PROD_002",
-  "type": "VARIANT",
-  "status": "ACTIVE",
-  "gtin": "MH02-M-Blue",
-  "name": "Kangeroo Hoodie-M-Blue",
-  "summary": "<p>Blue variant</p>",
-  "categoryRefs": ["STANDARD_CATEGORY", "SEASONAL"],
-  "price": [
-   { "type": "DEFAULT", "currency": "USD", "value": 45.0 },
-   { "type": "SPECIAL", "currency": "USD", "value": 35.0 }
-  ],
-  "taxType": {
-   "country": "AU",
-   "group": "Tax Group",
-   "tariff": "Tax Tariff"
-  }
- }
-]
-```
-### JSON Lines Format (One JSON Object Per Line)
-```json
-{"ref":"G_PROD_WITH_NO_STANDARD","type":"VARIANT","status":"ACTIVE","gtin":"MH01-XS-Orange","name":"Chaz Kangeroo Hoodie-XS-Orange main","summary":"<p>test short description</p>","categoryRefs":["STANDARD_CATEGORY"],"price":[{"type":"DEFAULT","currency":"USD","value":52.0},{"type":"SPECIAL","currency":"USD","value":9.0}],"taxType":{"country":"AU","group":"Tax Group","tariff":"Tax Tariff"}}
-{"ref":"G_PROD_002","type":"VARIANT","status":"ACTIVE","gtin":"MH02-M-Blue","name":"Kangeroo Hoodie-M-Blue","summary":"<p>Blue variant</p>","categoryRefs":["STANDARD_CATEGORY","SEASONAL"],"price":[{"type":"DEFAULT","currency":"USD","value":45.0},{"type":"SPECIAL","currency":"USD","value":35.0}],"taxType":{"country":"AU","group":"Tax Group","tariff":"Tax Tariff"}}
-```
-**JSONParserService auto-detects format** - NO array normalization, returns parsed data as-is.
----
-## Event Sending Configuration
-**Simple Configuration:** One variable controls everything
-```json
-{
- "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
-}
-```
-**How it works:**
-- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
-- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
-- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
-- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
-**Configuration Guidelines:**
-- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
-- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
-- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
-- **Note:** Fluent API supports concurrent requests - adjust based on your needs
-**Why Single Variable?**
-- **Simpler:** One variable instead of two (`mode` + `concurrency`)
-- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
-- **Less config:** Fewer activation variables to manage
-- **Flexible:** Easy to tune performance (just change the number)
----
-## 🔧 Production Reference Implementation
-### Processing Modes
-| Mode    | Default | What happens                                            | When to use                         |
-| ---------- | ------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
-| `per-file` | ✅   | Process one file end-to-end (discover → parse → Event API → archive) before moving on        | Recommended for production SFTP feeds and very large files  |
-| `chunked` | ➖   | Processes `fileChunkSize` files at a time, completing each chunk before the next          | High-volume feeds where per-file would take too long     |
-| `batch`  | ➖   | Loads every matched file into memory, produces a single Event API job, and archives when complete  | Tiny datasets or smoke tests only              |
-Set the mode via activation variables:
-```json
-{
- "processingMode": "per-file",
- "fileChunkSize": "5",
- "maxFilesPerRun": "25"
-}
-```
-### Versori Workflows Structure
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-### Recommended Project Structure
-```
-product-event-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── product.types.ts            # Shared type definitions
-```
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
----
-## Workflow Files
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-All time-based triggers that run automatically on cron schedules.
-#### `src/workflows/scheduled/daily-product-sync.ts`
-**Purpose**: Automatic Daily product sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-/**
- * Scheduled Workflow: Daily Product Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: product-sync.service.ts
- */
-export const dailyProductSync = schedule(
-  'product-sync-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv } = ctx;
-    const jobId = `product-sync-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-    log.info('🚀 [WORKFLOW START] Daily product sync workflow triggered', { jobId });
-    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
-    await tracker.updateJob(jobId, { status: 'processing' });
-    try {
-      // Reuse shared orchestration logic
-      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
-      await tracker.markCompleted(jobId, result);
-      log.info('✅ [WORKFLOW END] Daily product sync completed successfully', {
-        jobId,
-        duration: result.duration,
-        filesProcessed: result.filesProcessed
-      });
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      log.error('❌ [WORKFLOW ERROR] Daily product sync failed', {
-        jobId,
-        message: e?.message,
-        stack: e?.stack
-      });
-      await tracker.markFailed(jobId, e);
-      return { success: false, jobId, error: e?.message };
-    }
-  })
-);
-```
----
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-All HTTP-based triggers that create webhook endpoints.
-#### `src/workflows/webhook/adhoc-product-sync.ts`
-**Purpose**: Manual product sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-/**
- * Webhook: Manual Product Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
- * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5 }
- *
- * Pattern: webhook().then(http()) - needs Fluent API access
- * Uses shared service: product-sync.service.ts
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const adhocProductSync = webhook('product-sync-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'product-sync-adhoc', // Versori validates API key
-}).then(
-  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, data } = ctx;
-    const jobId = `product-sync-adhoc-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const filePattern = data?.filePattern as string;
-    const maxFiles = data?.maxFiles as number;
-    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-    });
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'manual',
-      stage: 'initialization',
-      status: 'queued',
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
-      .then((result) => {
-        log.info('✅ [BACKGROUND] Product sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          filesFailed: result.filesFailed,
-          recordsProcessed: result.recordsProcessed,
-          duration: result.duration,
-        });
-        return tracker.markCompleted(jobId, result);
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-        log.error('❌ [BACKGROUND] Product sync failed', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-        });
-        return tracker.markFailed(jobId, errorMessage);
-      });
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Product sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-#### `src/workflows/webhook/job-status-check.ts`
-**Purpose**: Query job status
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
-**Request body**: `{ "jobId": "product-sync-1234567890" }`
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
- * Request body: { "jobId": "product-sync-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const productSyncJobStatus = webhook('product-sync-job-status', {
-  response: { mode: 'sync' },
-  connection: 'product-sync-job-status',
-}).then(
-  fn('status', async ctx => {
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-    if (!jobId) {
-      return { success: false, error: 'jobId required' };
-    }
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
----
-### 3. Entry Point (`index.ts`)
-**Purpose**: Register all workflows with Versori platform
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * Versori automatically discovers and registers exported workflows
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- * - src/services/           → Shared service logic (reusable across workflows)
- */
-// Import scheduled workflows
-import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
-// Import webhook workflows
-import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
-import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
-// Register all workflows with Versori platform
-export {
-  // Scheduled (time-based triggers)
-  dailyProductSync,
-  // Webhooks (HTTP-based triggers)
-  adhocProductSync,
-  productSyncJobStatus,
-};
-```
-**What Gets Exposed:**
-- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
-- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
-- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
----
-### Adding New Workflows
-**To add a scheduled workflow:**
-1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-**To add a webhook workflow:**
-1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-**Example - Adding hourly delta sync:**
-```typescript
-// src/workflows/scheduled/hourly-delta-sync.ts
-export const hourlyDeltaSync = schedule(
-  'product-delta-hourly',
-  '0 * * * *' // Every hour
-).then(
-  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
-    // Delta sync logic (skip BPP)
-    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
-    return result;
-  })
-);
-// index.ts (add to imports and exports)
-import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
-export { daily_product_sync, hourlyDeltaSync, ... };
-```
----
-## 3. Type Definitions (src/types/product-ingestion.types.ts)
-```typescript
-/**
- * Type Definitions for Product Ingestion
- *
- * Centralized type definitions for product ingestion workflow
- */
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Product interface - represents transformed product data
- */
-export interface Product {
- ref: string;
- type?: string;
- status?: string;
- name: string;
- summary?: string;
- gtin?: string;
- catalogue?: { ref?: string };
- attributes?: Record<string, unknown>;
-}
-/**
- * Event configuration
- */
-export interface EventConfig {
- eventName: string;
- catalogueRef: string;
- catalogueType: string;
- eventMode: 'async' | 'sync';
-}
-/**
- * Event result - tracks success/failure counts
- */
-export interface EventResult {
- eventsSent: number;
- eventsFailed: number;
- errors: Array<{ productRef: string; error: string }>;
-}
-/**
- * Process file result
- */
-export interface ProcessFileResult {
- success: boolean;
- products: Product[];
- error?: string;
-}
-/**
- * Versori Context Interface
- * Represents the Versori runtime context passed to workflow functions
- */
-export interface VersoriContext {
- log: {
-  info: (message: string, data?: Record<string, unknown>) => void;
-  warn: (message: string, data?: Record<string, unknown>) => void;
-  error: (message: string, data?: Record<string, unknown>) => void;
-  debug?: (message: string, data?: Record<string, unknown>) => void;
- };
- openKv: (namespace: string) => {
-  get: (key: string) => Promise<unknown>;
-  set: (key: string, value: unknown) => Promise<void>;
-  delete: (key: string) => Promise<void>;
- };
- activation: {
-  getVariable: (name: string) => string | undefined;
-  connections?: Record<string, unknown>;
- };
- connections?: Record<string, unknown>;
- data?: unknown;
- fetch?: typeof fetch;
-}
-/**
- * Parameters for ingestion workflow
- */
-export interface ProductIngestionParams {
- jobId: string;
- triggeredBy: 'schedule' | 'webhook';
- filePattern?: string;
- maxFiles?: number;
- forceReprocess?: boolean;
- catalogueRef?: string;
- priority?: 'normal' | 'high';
-}
-/**
- * Result from ingestion workflow
- */
-export interface ProductIngestionResult {
- success: boolean;
- jobId: string;
- filesProcessed: number;
- filesFailed: number;
- filesSkipped?: number;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- fileResults: FileProcessingResult[];
- error?: string;
-}
-/**
- * Per-file processing result
- */
-export interface FileProcessingResult {
- fileName: string;
- success: boolean;
- skipped?: boolean;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- duration: number;
- error?: string;
-}
-```
----
-## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
-```typescript
-/**
- * Product File Processor Service
- *
- * Downloads JSON files from SFTP, parses, and transforms with UniversalMapper.
- * JSON-specific: Handles both array format and object with products array.
- */
-import {
- SftpDataSource,
- JSONParserService,
- UniversalMapper,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
-/**
- * Service for processing JSON product files
- */
-export class ProductFileProcessorService {
- constructor(
-  private sftp: SftpDataSource,
-  private jsonParser: JSONParserService,
-  private mapper: UniversalMapper,
-  private catalogueRef: string
- ) {}
- /**
-  * Download JSON file from SFTP, parse, and transform with UniversalMapper
-  *
-  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
-  * - Explicit null assignments after each step
-  * - Finally block guarantees cleanup
-  * - Prevents OOM errors on large JSON files
-  */
- async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
-  // ✅ CRITICAL: Variables for cleanup tracking
-  let jsonContent: string | null = null;
-  let parsed: unknown | null = null;
-  try {
-   // STEP 1: Download JSON content
-   jsonContent = (await this.sftp.downloadFile(remoteFilePath, {
-    encoding: 'utf8',
-   })) as string;
-   // STEP 2: Parse JSON with type safety
-   try {
-    parsed = await this.jsonParser.parse(jsonContent);
-   } catch (parseError: unknown) {
-    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
-    return {
-     success: false,
-     products: [],
-     error: `JSON parse error: ${errorMessage}`,
-    };
-   }
-   // ✅ Clear jsonContent from memory - no longer needed after parsing
-   jsonContent = null;
-   // STEP 3: JSON-SPECIFIC - Normalize based on structure (array vs object with products array)
-   const rawProducts = Array.isArray(parsed)
-    ? parsed
-    : (parsed as any)?.products
-     ? Array.isArray((parsed as any).products)
-      ? (parsed as any).products
-      : [(parsed as any).products]
-     : [];
-   if (rawProducts.length === 0) {
-    return {
-     success: false,
-     products: [],
-     error: 'No products found in JSON',
-    };
-   }
-   // ✅ Clear parsed data from memory - only need rawProducts now
-   parsed = null;
-   // STEP 4: Transform with UniversalMapper
-   // JSON-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
-   const sourceDataWithContext = rawProducts.map(item => ({
-    ...item,
-    $context: { catalogueRef: this.catalogueRef },
-   }));
-   const mappingResult = await this.mapper.map(sourceDataWithContext);
-   if (!mappingResult.success) {
-    return {
-     success: false,
-     products: [],
-     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
-    };
-   }
-   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-     skippedFields: mappingResult.skippedFields,
-     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-    });
-   }
-   return {
-    success: true,
-    products: mappingResult.data as Product[],
-   };
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   return {
-    success: false,
-    products: [],
-    error: errorMessage,
-   };
-  } finally {
-   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
-   jsonContent = null;
-   parsed = null;
-  }
- }
-}
-```
----
-## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
-```typescript
-/**
- * Event Sender Service
- *
- * Sends product events to Fluent Commerce Event API with per-record error handling.
- * Continues processing on individual failures (Event API best practice).
- * Supports configurable concurrency (sequential or parallel).
- */
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
-/**
- * Service for sending events to Fluent Commerce Event API
- *
- * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
- */
-export class EventSenderService {
- constructor(
-  private client: FluentClient,
-  private log?: any  // ✅ Optional logger for progress tracking
- ) {}
- /**
-  * Send events to Fluent Commerce Event API with configurable concurrency
-  *
-  * **Performance Characteristics:**
-  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
-  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
-  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
-  *
-  * **Implementation Strategy:**
-  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
-  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
-  * - Both modes: Per-record error tracking (failures don't block other events)
-  *
-  * @param products - Array of products to send as events
-  * @param eventConfig - Event configuration (name, catalogue ref, mode)
-  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
-  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
-  */
- async sendEvents(
-  products: Product[],
-  eventConfig: EventConfig,
-  concurrency: number = 1
- ): Promise<EventResult> {
-  // Validate concurrency (guard against invalid values)
-  const safeConc = Math.max(1, Math.floor(concurrency));
-  // Result accumulators
-  let eventsSent = 0;
-  let eventsFailed = 0;
-  const errors: Array<{ productRef: string; error: string }> = [];
-  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
-  if (this.log) {
-   this.log.info('📤 Starting event sending', {
-    totalProducts: products.length,
-    concurrency: safeConc,
-    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
-    eventName: eventConfig.eventName,
-    catalogueRef: eventConfig.catalogueRef
-   });
-  }
-  // Helper: Build event payload (DRY - reused in both modes)
-  const buildPayload = (product: Product) => ({
-   name: eventConfig.eventName,
-   entityRef: eventConfig.catalogueRef,
-   entityType: 'PRODUCT_CATALOGUE' as const,
-   entitySubtype: eventConfig.catalogueType,
-   rootEntityRef: eventConfig.catalogueRef,
-   rootEntityType: 'PRODUCT_CATALOGUE' as const,
-   attributes: product,
-  });
-  // ============================================================================
-  // SEQUENTIAL MODE (concurrency === 1)
-  // ============================================================================
-  if (safeConc === 1) {
-   for (let i = 0; i < products.length; i++) {
-    const product = products[i];
-    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
-    if (this.log && i % 10 === 0) {
-     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
-      productRef: product.ref,
-      progress: `${((i / products.length) * 100).toFixed(1)}%`
-     });
-    }
-    try {
-     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
-     eventsSent++;
-    } catch (err: unknown) {
-     eventsFailed++;
-     const errorMsg = err instanceof Error ? err.message : String(err);
-     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
-     // Continue processing (failure doesn't block other products)
-    }
-   }
-   // ✅ PRODUCTION ENHANCEMENT: Log completion
-   if (this.log) {
-    this.log.info('✅ Sequential event sending completed', {
-     totalProducts: products.length,
-     eventsSent,
-     eventsFailed
-    });
-   }
-   return { eventsSent, eventsFailed, errors };
-  }
-  // ============================================================================
-  // PARALLEL MODE (concurrency > 1)
-  // ============================================================================
-  const totalChunks = Math.ceil(products.length / safeConc);
-  for (let i = 0; i < products.length; i += safeConc) {
-   const chunk = products.slice(i, i + safeConc);
-   const chunkNumber = Math.floor(i / safeConc) + 1;
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
-   if (this.log) {
-    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
-     chunkNumber,
-     totalChunks,
-     productsInChunk: chunk.length,
-     productRange: `${i + 1}-${i + chunk.length}`,
-     totalProducts: products.length,
-     progress: `${((i / products.length) * 100).toFixed(1)}%`
-    });
-   }
-   // Fire all requests in chunk concurrently
-   const results = await Promise.allSettled(
-    chunk.map(product =>
-     this.client
-      .sendEvent(buildPayload(product), eventConfig.eventMode)
-      .then(() => ({ success: true as const, product }))
-      .catch(error => ({ success: false as const, product, error }))
-    )
-   );
-   // Aggregate chunk results into totals
-   let chunkSuccess = 0;
-   let chunkFailed = 0;
-   for (const result of results) {
-    if (result.status === 'fulfilled' && result.value.success) {
-     eventsSent++;
-     chunkSuccess++;
-    } else {
-     eventsFailed++;
-     chunkFailed++;
-     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
-     const product = result.status === 'fulfilled' ? result.value.product : null;
-     errors.push({
-      productRef: product?.ref || 'unknown',
-      error: error?.message || String(error) || 'unknown error',
-     });
-    }
-   }
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
-   if (this.log) {
-    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
-     chunkNumber,
-     totalChunks,
-     chunkSuccess,
-     chunkFailed,
-     totalSentSoFar: eventsSent,
-     totalFailedSoFar: eventsFailed,
-     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
-    });
-   }
-  }
-  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
-  if (this.log) {
-   this.log.info('✅ Parallel event sending completed', {
-    totalProducts: products.length,
-    totalChunks,
-    concurrency: safeConc,
-    eventsSent,
-    eventsFailed
-   });
-  }
-  return { eventsSent, eventsFailed, errors };
- }
-}
-```
----
-## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
-```typescript
-/**
- * Event Logger Service
- *
- * Writes event processing logs to SFTP for audit/debugging.
- * Optional - can be used to create rejection reports.
- */
-import { Buffer } from 'node:buffer';
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Service for writing event logs to SFTP
- */
-export class EventLoggerService {
- constructor(private sftp: SftpDataSource) {}
- /**
-  * Write event log to SFTP
-  *
-  * @param remotePath - SFTP path for log file
-  * @param logData - Log data to write (JSON format)
-  */
- async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
-  try {
-   const logContent = JSON.stringify(logData, null, 2);
-   await this.sftp.uploadFile(remotePath, logContent);
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   throw new Error(`Failed to write event log: ${errorMessage}`);
-  }
- }
-}
-```
----
-## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
-```typescript
-/**
- * MAIN INGESTION ORCHESTRATION SERVICE
- *
- * This is the heart of the ingestion workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Discover files on SFTP
- * 3. Download and parse JSON files
- * 4. Transform data with UniversalMapper
- * 5. Send events to Fluent Commerce
- * 6. Archive processed files
- * 7. Track file processing state
- * 8. Track job progress with JobTracker
- */
-import { Buffer } from 'node:buffer';
-import {
- createClient,
- SftpDataSource,
- JSONParserService,
- UniversalMapper,
- VersoriFileTracker,
- JobTracker,
- type FluentClient,
- type JobStatus,
-} from '@fluentcommerce/fc-connect-sdk';
-import type {
- ProductIngestionParams,
- ProductIngestionResult,
- FileProcessingResult,
- EventConfig,
-} from '../types/product-ingestion.types';
-import { ProductFileProcessorService } from './product-file-processor.service';
-import { EventSenderService } from './event-sender.service';
-import { EventLoggerService } from './event-logger.service';
-import { joinSftpPath, timestampedName, extractFileName } from '../utils/sftp-path.utils';
-import mappingConfig from '../../config/products.import.json' with { type: 'json' };
-// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
-/**
- * Query job status from KV store
- */
-export async function getJobStatus(
- kv: ReturnType<VersoriContext['openKv']>,
- jobId: string,
- log: VersoriContext['log']
-): Promise<JobStatus | undefined> {
- try {
-  const tracker = new JobTracker(kv, log);
-  return await tracker.getJob(jobId);
- } catch (error: unknown) {
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  log.error('Failed to get job status', {
-   jobId,
-   message: errorMessage,
-   stack: error instanceof Error ? error.stack : undefined,
-   errorType: error instanceof Error ? error.constructor.name : 'Error'
-  });
-  return undefined;
- }
-}
-/**
- * MAIN ORCHESTRATION FUNCTION
- */
-export async function executeProductIngestion(
- ctx: VersoriContext,
- params: ProductIngestionParams
-): Promise<ProductIngestionResult> {
- const { log, openKv, activation } = ctx;
- const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
- const kv = openKv(':project:');
- const tracker = new JobTracker(kv, log);
- const startTime = Date.now();
- const fileResults: FileProcessingResult[] = [];
- // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
- let sftp: SftpDataSource | undefined;
- try {
-  // ═══════════════════════════════════════════════════════════
-  // STEP 1/8: Initialize Job Tracking
-  // ═══════════════════════════════════════════════════════════
-  log.info('🔧 [STEP 1/8] Initializing job tracking', { jobId });
-  await tracker.createJob(jobId, {
-   triggeredBy,
-   filePattern: filePattern || 'default',
-   maxFiles: maxFiles || 'unlimited',
-   forceReprocess: !!forceReprocess
-  });
-  // ═══════════════════════════════════════════════════════════
-  // STEP 2/8: Initialize Fluent Client & SFTP Connection
-  // ═══════════════════════════════════════════════════════════
-  log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and SFTP', { jobId });
-  const client = await createClient(ctx);
-  if (!client) {
-   const errorMsg = 'Failed to create Fluent Commerce client';
-   log.error('❌ ' + errorMsg, {
-    recommendation: 'Verify fluent_commerce connection is configured with valid credentials'
-   });
-   throw new Error(errorMsg);
-  }
-  // ✅ CRITICAL: Set retailerId for Event API calls
-  const fluentRetailerId = activation.getVariable('fluentRetailerId');
-  if (!fluentRetailerId) {
-   const errorMsg = 'fluentRetailerId is required for Event API calls';
-   log.error('❌ ' + errorMsg, {
-    recommendation: 'Add fluentRetailerId to activation variables (e.g., "1")'
-   });
-   throw new Error(errorMsg);
-  }
-  client.setRetailerId(fluentRetailerId);
-  log.info('✅ RetailerId set for Event API', { retailerId: fluentRetailerId });
-  // Get SFTP credentials from connection (Basic Auth)
-  log.info('🔐 Retrieving SFTP credentials from connection');
-  let sftpUsername: string;
-  let sftpPassword: string;
-  try {
-   // Retrieve credentials from the 'SFTP' connection
-   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-   if (!sftpCred?.accessToken) {
-    throw new Error('No SFTP credentials found in connection configuration');
-   }
-   // Decode base64 accessToken to get "username:password"
-   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-   const parts = rawBasicAuth.split(':');
-   if (parts.length !== 2) {
-    throw new Error('Invalid SFTP credential format - expected username:password');
-   }
-   sftpUsername = parts[0];
-   sftpPassword = parts[1];
-   log.info('✅ SFTP credentials retrieved successfully', {
-    hasUsername: !!sftpUsername,
-    hasPassword: !!sftpPassword,
-    usernameLength: sftpUsername.length,
-    passwordLength: sftpPassword.length,
-   });
-  } catch (credError: any) {
-   log.error('❌ Failed to retrieve SFTP credentials', {
-    message: credError instanceof Error ? credError.message : String(credError),
-    stack: credError instanceof Error ? credError.stack : undefined,
-    errorType: credError instanceof Error ? credError.constructor.name : 'Error',
-    recommendation: 'Ensure SFTP connection is configured with Basic Authentication'
-   });
-   throw credError;
-  }
-  // Get SFTP configuration from activation variables
-  const sftpConfig = {
-   host: activation.getVariable('sftpHost'),
-   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
-   username: sftpUsername,
-   password: sftpPassword,
-   privateKey: activation.getVariable('sftpPrivateKey'),
-   incomingPath: activation.getVariable('sftpIncomingPath') || '/products/incoming',
-   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
-   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
-   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.json',
-   enableFileTracking: activation.getVariable('enableFileTracking') !== 'false' // Default: true
-  };
-  // Validate SFTP config
-  if (!sftpConfig.host) {
-   const errorMsg = 'SFTP configuration incomplete: missing sftpHost';
-   log.error('❌ ' + errorMsg, {
-    recommendation: 'Add sftpHost to activation variables (e.g., "sftp.partner.com")'
-   });
-   throw new Error(errorMsg);
-  }
-  if (!sftpConfig.password && !sftpConfig.privateKey) {
-   const errorMsg = 'SFTP configuration incomplete: missing password or privateKey';
-   log.error('❌ ' + errorMsg, {
-    recommendation: 'Ensure SFTP connection has valid credentials OR set sftpPrivateKey'
-   });
-   throw new Error(errorMsg);
-  }
-  // Initialize SFTP data source
-  log.info('🔧 Initializing SFTP data source', {
-   host: sftpConfig.host,
-   port: sftpConfig.port,
-   remotePath: sftpConfig.incomingPath,
-   filePattern: sftpConfig.filePattern,
-   enableFileTracking: sftpConfig.enableFileTracking
-  });
-  sftp = new SftpDataSource({
-   type: 'SFTP_JSON',
-   connectionId: 'sftp-product-ingestion',
-   name: 'Product Ingestion SFTP',
-   settings: {
-    host: sftpConfig.host,
-    port: sftpConfig.port,
-    username: sftpConfig.username,
-    password: sftpConfig.password,
-    privateKey: sftpConfig.privateKey,
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern,
-    encoding: 'utf8'
-   }
-  }, log);
-  try {
-   // Validate SFTP connection
-   await sftp.validateConnection();
-   log.info('✅ SFTP connection validated successfully');
-   // Ensure archive directories exist (recursive)
-   await sftp.createDirectory(sftpConfig.archivePath, true);
-   await sftp.createDirectory(sftpConfig.errorPath, true);
-   // ═══════════════════════════════════════════════════════════
-   // STEP 3/8: Discover Files on SFTP
-   // ═══════════════════════════════════════════════════════════
-   log.info('📂 [STEP 3/8] Discovering files on SFTP', {
-    jobId,
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern,
-    enableFileTracking: sftpConfig.enableFileTracking
-   });
-   await tracker.updateJob(jobId, {
-    status: 'processing',
-    stage: 'file_discovery',
-    message: 'Discovering files on SFTP'
-   });
-   // List files from SFTP
-   const allFiles = await sftp.listFiles({
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern
-   });
-   log.info(`📋 Discovered ${allFiles.length} files matching pattern`, {
-    pattern: sftpConfig.filePattern,
-    count: allFiles.length
-   });
-   // Apply maxFiles limit if specified
-   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
-   if (filesToProcess.length === 0) {
-    log.info('ℹ️ No files to process', {
-     remotePath: sftpConfig.incomingPath,
-     filePattern: sftpConfig.filePattern
-    });
-    await tracker.markCompleted(jobId, {
-     filesProcessed: 0,
-     message: 'No files found to process'
-    });
-    return {
-     success: true,
-     jobId,
-     filesProcessed: 0,
-     filesFailed: 0,
-     recordsProcessed: 0,
-     eventsSent: 0,
-     eventsFailed: 0,
-     fileResults: []
-    };
-   }
-   log.info(`🔄 Processing ${filesToProcess.length} files`, {
-    totalDiscovered: allFiles.length,
-    toProcess: filesToProcess.length,
-    maxFilesLimit: maxFiles || 'unlimited'
-   });
-   // Initialize services
-   const jsonParser = new JSONParserService();
-   const mapper = new UniversalMapper(mappingConfig);
-   // Get event configuration
-   const eventConfig: EventConfig = {
-    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
-    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
-    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
-    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
-   };
-   // Get event sending configuration
-   const eventConcurrency = Math.max(
-    1,
-    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
-   );
-   log.info(`Event concurrency: ${eventConcurrency}`, {
-    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
-    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
-   });
-   // Initialize class-based services
-   const fileProcessor = new ProductFileProcessorService(
-    sftp,
-    jsonParser,
-    mapper,
-    eventConfig.catalogueRef
-   );
-   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
-   const eventSender = new EventSenderService(client, log);
-   const eventLogger = new EventLoggerService(sftp);
-   // File tracker for deduplication
-   const fileTracker = new VersoriFileTracker(kv, 'product-ingestion', log);
-   // Use direct KV with dot-separated keys (NATS KV safe)
-   const processedKeyBase = 'fc_sdk.processed_files';
-   // Get SFTP path configuration
-   const requireAbsolutePaths = activation.getVariable('requireAbsolutePaths') === 'true';
-   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
-    const file = filesToProcess[fileIndex];
-    const fileStartTime = Date.now();
-    // Extract file name using SDK utility
-    const remoteFilePath = (file as any).path || file.name;
-    const baseName = extractFileName(remoteFilePath);
-    log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`, {
-     remotePath: remoteFilePath,
-     fileSize: (file as any).size || 'unknown'
-    });
-    try {
-     // Check if file already processed (unless force reprocess)
-     if (!forceReprocess && sftpConfig.enableFileTracking) {
-      const isProcessed = await fileTracker.isProcessed(baseName);
-      if (isProcessed) {
-       log.info(`⏭️ Skipping already processed file: ${baseName}`, {
-        reason: 'File tracking enabled and file previously processed'
-       });
-       fileResults.push({
-        fileName: baseName,
-        success: true,
-        skipped: true,
-        recordsProcessed: 0,
-        eventsSent: 0,
-        eventsFailed: 0,
-        duration: 0,
-       });
-       continue;
-      }
-     }
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'downloading',
-      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
-     });
-     // ═══════════════════════════════════════════════════════════
-     // STEP 4/8: Process File (Download + Parse + Map)
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📥 [STEP 4/8] Processing file: ${baseName}`);
-     const fileResult = await fileProcessor.downloadParseAndTransform(remoteFilePath);
-     log.info(`✅ File processing completed: ${baseName}`, {
-      success: fileResult.success,
-      recordCount: fileResult.products.length,
-      error: fileResult.error
-     });
-     if (!fileResult.success || fileResult.products.length === 0) {
-      const archivedName = timestampedName(baseName);
-      await sftp.moveFile(
-       remoteFilePath,
-       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
-      );
-      fileResults.push({
-       fileName: baseName,
-       success: false,
-       recordsProcessed: fileResult.products.length,
-       eventsSent: 0,
-       eventsFailed: 0,
-       duration: Date.now() - fileStartTime,
-       error: fileResult.error || 'Processing failed'
-      });
-      continue;
-     }
-     // ═══════════════════════════════════════════════════════════
-     // STEP 5/8: Send Events
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
-     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
-     log.info(`🚀 [EventSender] Starting event submission for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventName: eventConfig.eventName,
-      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
-      sampleProductRefs: sampleProductRefs.join(', '),
-      eventMode: eventConfig.eventMode
-     });
-     const eventResult = await eventSender.sendEvents(
-      fileResult.products,
-      eventConfig,
-      eventConcurrency
-     );
-     const eventsSent = eventResult.eventsSent;
-     const eventsFailed = eventResult.eventsFailed;
-     const eventDuration = Date.now() - fileStartTime;
-     log.info(`✅ [EventSender] Event submission completed for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
-      duration: eventDuration,
-      avgTimePerEvent: eventsSent > 0 ? Math.round(eventDuration / eventsSent) + 'ms' : 'N/A'
-     });
-     // ═══════════════════════════════════════════════════════════
-     // STEP 6/8: Archive File & Track State
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'archiving',
-      message: `Archiving ${baseName}`
-     });
-     const archivedName = timestampedName(baseName);
-     const targetDir = eventsFailed > 0 ? sftpConfig.errorPath : sftpConfig.archivePath;
-     log.info(`📁 Moving file to ${eventsFailed > 0 ? 'error' : 'processed'} directory`, {
-      sourcePath: remoteFilePath,
-      targetPath: joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
-     });
-     await sftp.moveFile(
-      remoteFilePath,
-      joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
-     );
-     // Mark as processed in file tracker (if enabled)
-     if (sftpConfig.enableFileTracking) {
-      await fileTracker.markProcessed(baseName, {
-       recordsProcessed: fileResult.products.length,
-       eventsSent,
-       eventsFailed,
-      });
-      log.info(`✅ File tracking state updated for: ${baseName}`);
-     }
-     log.info(`✅ File archived successfully: ${baseName}`, {
-      archivedName,
-      targetDir,
-      duration: Date.now() - fileStartTime
-     });
-     // Optional: Write event log for audit/debugging
-     if (eventsFailed > 0) {
-      const logPath = joinSftpPath(
-       requireAbsolutePaths,
-       sftpConfig.errorPath,
-       `${baseName.replace(/\.json$/i, '')}-event-log.json`
-      );
-      await eventLogger.writeEventLog(logPath, {
-       fileName: baseName,
-       totalRecords: fileResult.products.length,
-       eventsSent,
-       eventsFailed,
-       errors: eventResult.errors,
-       processedAt: new Date().toISOString()
-      });
-     }
-     fileResults.push({
-      fileName: baseName,
-      success: true,
-      recordsProcessed: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      duration: Date.now() - fileStartTime
-     });
-    } catch (error: unknown) {
-     // ✅ Enhanced error logging: Extract all error details for visibility + recommendations
-     const errorMessage = error instanceof Error ? error.message : String(error);
-     const errorDetails = {
-      message: errorMessage,
-      stack: error instanceof Error ? error.stack : undefined,
-      fileName: (error as any)?.fileName,
-      lineNumber: (error as any)?.lineNumber,
-      originalError: (error as any)?.context?.originalError?.message,
-      errorType: error instanceof Error ? error.name : 'Error',
-     };
-     // Provide actionable recommendations based on error type
-     let recommendation = 'Check error details above for specific issue';
-     if (errorMessage.includes('SFTP')) {
-      recommendation = 'Verify SFTP connection: check host, credentials, and network access';
-     } else if (errorMessage.includes('parse') || errorMessage.includes('JSON')) {
-      recommendation = 'Verify JSON file format: check for valid JSON syntax and encoding';
-     } else if (errorMessage.includes('mapping') || errorMessage.includes('validation')) {
-      recommendation = 'Review mapping configuration: ensure all required fields are present';
-     } else if (errorMessage.includes('retailerId') || errorMessage.includes('Event API')) {
-      recommendation = 'Verify fluentRetailerId activation variable and Event API connection';
-     }
-     log.error(`❌ Error processing file ${baseName}:`, {
-      ...errorDetails,
-      recommendation
-     });
-     try {
-      const archivedName = timestampedName(baseName);
-      await sftp.moveFile(
-       remoteFilePath,
-       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
-      );
-     } catch (moveError: unknown) {
-      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
-      log.error('Could not archive failed file', {
-       file: baseName,
-       message: moveErrorMessage,
-       stack: moveError instanceof Error ? moveError.stack : undefined,
-       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
-      });
-     }
-     fileResults.push({
-      fileName: baseName,
-      success: false,
-      recordsProcessed: 0,
-      eventsSent: 0,
-      eventsFailed: 0,
-      duration: Date.now() - fileStartTime,
-      error: errorMessage
-     });
-    }
-   }
-   // ═══════════════════════════════════════════════════════════
-   // STEP 8/8: Complete Job & Calculate Totals
-   // ═══════════════════════════════════════════════════════════
-   const totalDuration = Date.now() - startTime;
-   log.info('🏁 [STEP 8/8] Completing job and calculating totals', { jobId });
-   const filesProcessed = fileResults.filter(r => r.success && !r.skipped).length;
-   const filesFailed = fileResults.filter(r => !r.success).length;
-   const filesSkipped = fileResults.filter(r => r.skipped).length;
-   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
-   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
-   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
-   await tracker.markCompleted(jobId, {
-    filesProcessed,
-    filesFailed,
-    filesSkipped,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: totalDuration
-   });
-   log.info('✅ Ingestion completed successfully', {
-    jobId,
-    filesProcessed,
-    filesFailed,
-    filesSkipped,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: totalDuration,
-    avgFileTime: filesProcessed > 0 ? Math.round(totalDuration / filesProcessed) + 'ms' : 'N/A'
-   });
-   return {
-    success: true,
-    jobId,
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    fileResults
-   };
-  } finally {
-   // ⚠️ CRITICAL: Always dispose SFTP connection (safe null check)
-   if (sftp) {
-    await sftp.dispose();
-    log.info('SFTP connection disposed');
-   }
-  }
- } catch (error: unknown) {
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  const errorStack = error instanceof Error ? error.stack : undefined;
-  log.error('Ingestion workflow failed', {
-   jobId,
-   message: errorMessage,
-   stack: errorStack,
-   errorType: error instanceof Error ? error.constructor.name : 'Error'
-  });
-  try {
-   await tracker.markFailed(jobId, error);
-  } catch (trackingError: unknown) {
-   const trackingErrorMessage =
-    trackingError instanceof Error ? trackingError.message : String(trackingError);
-   log.warn('Failed to mark job as failed in tracker', {
-    jobId,
-    trackingError: trackingErrorMessage,
-   });
-  }
-  return {
-   success: false,
-   jobId,
-   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
-   filesFailed: fileResults.filter(r => !r.success).length,
-   filesSkipped: fileResults.filter(r => r.skipped).length,
-   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
-   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
-   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
-   fileResults,
-   error: errorMessage,
-  };
- } finally {
-  // ⚠️ CRITICAL: Ensure SFTP is disposed even if outer error occurs
-  if (sftp) {
-   try {
-    await sftp.dispose();
-    log.info('SFTP connection disposed (outer finally)');
-   } catch (disposeError: unknown) {
-    const disposeErrorMessage =
-     disposeError instanceof Error ? disposeError.message : String(disposeError);
-    log.warn('Error disposing SFTP in outer finally', { error: disposeErrorMessage });
-   }
-  }
- }
-}
-```
----
-## 8. Utility Functions (src/utils/)
-### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
-```typescript
-/**
- * SFTP Path Utilities
- *
- * Helper functions for SFTP path operations with AWS Transfer Family support.
- * Handles both absolute paths (AWS Transfer Family) and relative paths (standard OpenSSH).
- */
-/**
- * Join SFTP path segments safely
- *
- * Handles different SFTP server path requirements:
- * - AWS Transfer Family: Requires absolute paths (leading /)
- * - Standard OpenSSH: Supports relative paths
- *
- * @param requireAbsolutePath - true for AWS Transfer Family, false for standard OpenSSH
- * @param parts - Path segments to join
- * @returns Properly formatted SFTP path
- */
-export function joinSftpPath(requireAbsolutePath: boolean, ...parts: string[]): string {
- const cleaned = parts
-  .filter(Boolean)
-  .map(p => String(p).replace(/^\/+|\/+$/g, ''))
-  .join('/');
- return requireAbsolutePath && !cleaned.startsWith('/') ? `/${cleaned}` : cleaned;
-}
-/**
- * Generate timestamped filename for archival
- *
- * Adds ISO timestamp to filename before extension for unique archival.
- * Handles JSON files specifically but can be adapted for other formats.
- *
- * @param name - Original filename
- * @returns Filename with timestamp appended
- */
-export function timestampedName(name: string): string {
- const ts = new Date().toISOString().replace(/[:.]/g, '-');
- const base = name.replace(/\.json$/i, '');
- return `${base}-${ts}.json`;
-}
-/**
- * Extract base filename from full SFTP path
- *
- * @param filePath - Full SFTP path (e.g., '/products/incoming/file.json')
- * @returns Base filename only (e.g., 'file.json')
- */
-export function extractFileName(filePath: string): string {
- return filePath.split('/').pop() || filePath;
-}
-// Alias for backward compatibility
-export const getBaseName = extractFileName;
-```
----
-### Job ID Generator (src/utils/job-id-generator.ts)
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking ingestion workflows
- *
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
- * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
- */
-export function generateJobId(type: string, entity: string): string {
- const now = new Date();
- const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
- const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
- const randomStr = Math.random().toString(36).substring(2, 8);
- return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
-}
-/**
- * Parse job ID components
- */
-export function parseJobId(jobId: string): {
- type: string;
- entity: string;
- date: string;
- time: string;
- random: string;
-} | null {
- const parts = jobId.split('_');
- if (parts.length !== 5) {
-  return null;
- }
- return {
-  type: parts[0],
-  entity: parts[1],
-  date: parts[2],
-  time: parts[3],
-  random: parts[4],
- };
-}
-```
----
-## 9. Package Configuration
-#### `package.json`
-```json
-{
- "name": "sftp-json-product-event",
- "version": "1.2.0",
- "type": "module",
- "main": "src/index.ts",
- "scripts": {
-  "dev": "versori dev",
-  "build": "versori build",
-  "deploy": "versori deploy"
- },
- "dependencies": {
-  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-  "@versori/run": "latest"
- },
- "devDependencies": {
-  "@types/node": "^20.0.0",
-  "typescript": "^5.0.0"
- }
-}
-```
-#### `tsconfig.json`
-```json
-{
-  "compilerOptions": {
-    "lib": ["ES2022", "DOM"],
-    "module": "ES2022",
-    "target": "ES2022",
-    "moduleResolution": "bundler",
-    "strict": true,
-    "esModuleInterop": true,
-    "skipLibCheck": true,
-    "resolveJsonModule": true,
-    "isolatedModules": true
-  }
-}
-```
-**Note:** This hybrid configuration works for both local development and Versori deployment. It provides proper TypeScript checking while maintaining compatibility with Versori's Deno runtime.
----
-## 10. Deployment Instructions
-### Deploy to Versori
-```bash
-# 1. Install dependencies
-npm install
-# 2. Test locally (if using Versori CLI)
-npm run dev
-# 3. Deploy to Versori platform
-npm run deploy
-```
-### Configure Activation Variables
-In Versori platform settings, configure all variables listed in the Activation Variables section above.
----
-## 11. Testing
-### Test Scheduled Ingestion
-Upload a test JSON file to SFTP incoming path and wait for the scheduled run.
-**Check logs:**
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and SFTP
-[STEP 3/8] Discovering files on SFTP
-[FILE 1/1] Processing file: products_20250124.json
-[STEP 4/8] Downloading and parsing: products_20250124.json
-[STEP 5/8] Transforming 5 products from products_20250124.json
-[STEP 6/8] Sending 5 events to Fluent Commerce
-[STEP 7/8] Archiving file: products_20250124.json
-[STEP 8/8] Completing job and calculating totals
-```
-### Test Ad hoc Ingestion
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{}'
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "filePattern": "urgent_*.json", }'
-# Force reprocess
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "forceReprocess": true,
-  "filePattern": "products_20250124.json"
- }'
-```
-### Test Job Status Query
-```bash
-curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "jobId": "ADHOC_PROD_20251024_183045_abc123"
- }'
-```
----
-## Monitoring
-### Success Response
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 50,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.json",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 50,
-   "eventsFailed": 0
-  }
- ],
- "duration": 12345
-}
-```
-### Partial Success Response
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 45,
- "eventsFailed": 5,
- "results": [
-  {
-   "file": "products_2025-01-22.json",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 45,
-   "eventsFailed": 5,
-   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
-  }
- ],
- "duration": 12345
-}
-```
-### Error Response
-```json
-{
- "success": false,
- "filesProcessed": 0,
- "filesFailed": 1,
- "totalRecords": 0,
- "eventsSent": 0,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.json",
-   "success": false,
-   "error": "JSON parse error: Invalid structure"
-  }
- ],
- "duration": 876
-}
-```
-### Monitoring Metrics
-Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
-- **Files Processed** - Total files successfully processed
-- **Events Sent** - Total events sent to Fluent Commerce
-- **Events Failed** - Events that failed (check rejection reports)
-- **Processing Duration** - Time taken for complete workflow
-- **Rate Limiting** - Watch for 429 errors indicating throttling
-Use the status webhook for dashboards and automated monitoring.
----
-- **No files found:** Check `sftpIncomingPath` and `filePattern` (file names only)
-- **JSON parse failed:** Move to errors/; validate JSON structure and encoding
-- **Files skipped (already processed):** Check VersoriFileTracker KV state; use `forceReprocess: true` if needed
-- **High event failures:** Inspect logs; consider Batch API for very high volumes
-- **429 throttling:** Add small backoff/delay or use Batch API
-- **SFTP connection errors:** Verify credentials (connection object), host, port, and permissions
-- **Connection pool exhausted:** Check connection cleanup in finally blocks
----
-## 13. Key Takeaways
-- ✅ **Refactored Architecture (v2.0.0)** - 3 service functions for clean separation:
- - `processFile()`: Download + Parse + Map (NO array normalization)
- - `sendEvents()`: Loop records, send events with per-record error handling
- - `writeEventLog()`: Write rejection logs to SFTP with Buffer
-- ✅ **Per-file Processing** - One file at a time for memory efficiency
-- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
-- ✅ **Buffer import** - Always import Buffer: `import { Buffer } from 'node:buffer';`
-- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
-- ✅ **JobTracker** - Track job lifecycle with KV persistence
-- ✅ **VersoriFileTracker** - KV-based deduplication for SFTP (required)
-- ✅ **SFTP deduplication** - KV state tracking prevents reprocessing
-- ✅ **SFTP dispose()** - Always cleanup in finally block
-- ✅ **SFTP method names** - listFiles, downloadFile, moveFile (not \*Object)
-- ✅ **Connection credentials** - Use `activation.connections` for Basic Auth
-- ✅ **Per-record error handling** - Continue on individual failures
-- ✅ **Externalized mapping** - Use JSON config file for field mappings
-- ✅ **File-level error handling** - Don't stop on single file failure
-- ✅ **JSON auto-detection** - Supports standard JSON and JSON Lines (NO normalization)
-- ✅ **SFTP vs S3** - Different deduplication strategies (KV state vs physical movement)
----
-[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)
+---
+template_id: tpl-ingest-sftp-json-to-product-event
+canonical_filename: template-ingestion-sftp-json-product-event.md
+sdk_version: ^0.1.39
+runtime: versori
+direction: ingestion
+source: sftp-json
+destination: fluent-event-api
+entity: product
+format: json
+logging: versori
+status: stable
+features:
+  - batched-events
+  - attribute-transformation
+  - memory-management
+  - json-serialization-handling
+  - enhanced-logging
+---
+# Template: Ingestion - SFTP JSON to Product Event
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+---
+## 📋 Implementation Prompt
+```
+I need a Versori scheduled ingestion that:
+1) Discovers JSON files on SFTP with file tracking to skip duplicates
+2) Downloads and parses JSON (standard or JSON Lines format) - NO array normalization
+3) Transforms records with UniversalMapper per mapping JSON
+4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
+5) Archives files to processed/ or errors/ and writes optional rejection report
+6) Tracks progress with JobTracker and exposes a job-status webhook
+7) Uses native Versori log from context
+8) **Refactored with 3 service functions**: processFile(), sendEvents(), writeEventLog()
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+---
+## 📋 Template Overview
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, JSON structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP JSON, transforms it, and sends events to the Fluent Commerce Event API.
+### What This Template Does
+```
+┌────────────────────────────────────────────────────────────────┐
+│          INGESTION WORKFLOW              │
+└────────────────────────────────────────────────────────────────┘
+1. TRIGGER
+  ├─ Scheduled (Cron): Runs automatically every hour
+  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
+  └─ Status Query (Webhook): Check job progress
+2. DISCOVER FILES (SftpDataSource)
+  ├─ List files from SFTP directory
+  ├─ Filter by pattern (products_*.json)
+  ├─ Check file tracking (skip processed)
+  └─ Sort by oldest first
+3. DOWNLOAD & PARSE (JSONParserService)
+  ├─ Download file from SFTP
+  ├─ Parse JSON to JavaScript object
+  ├─ Auto-detect format (JSON vs JSON Lines)
+  └─ Normalize to array structure
+4. TRANSFORM (UniversalMapper)
+  ├─ Map JSON fields to Fluent schema
+  ├─ Apply SDK resolvers (trim, uppercase, etc.)
+  ├─ Direct passthrough for nested objects
+  ├─ Direct passthrough for arrays
+  └─ Collect transformation errors
+5. SEND EVENTS (Event API)
+  ├─ Loop through transformed products
+  ├─ Send UPSERT_PRODUCT event (async)
+  ├─ Track success/failure count
+  └─ Continue on individual failures
+6. ARCHIVE (SftpDataSource)
+  ├─ Move file to processed/ folder
+  ├─ Or move to errors/ if validation failed
+  ├─ Generate timestamped archive name
+  └─ Verify archive success
+7. TRACK STATE (VersoriFileTracker)
+  ├─ Mark file as processed
+  ├─ Store processing metadata
+  ├─ Prevent duplicate processing
+  └─ Track record counts
+8. TRACK JOB (JobTracker)
+  ├─ Update job status at each step
+  ├─ Store final result in KV
+  ├─ Enable status queries via webhook
+  └─ Handle errors gracefully
+```
+### Key Features
+- **Refactored Architecture**: 3 service functions for clean separation of concerns
+ - `processFile()`: SftpDataSource + JSONParserService + UniversalMapper (NO array normalization)
+ - `sendEvents()`: Loop records, `client.sendEvent()` with per-record error handling
+ - `writeEventLog()`: Write logs to SFTP with Buffer
+- **Per-file Processing**: Download → Parse → Map → Send → Archive (one file at a time)
+- **Job tracking with status queries**: JobTracker persists to Versori KV
+- **Execution modes**: scheduled, ad hoc, status query
+- **Uses SftpDataSource, JSONParserService, UniversalMapper, JobTracker**
+- **Error handling, retry logic, and SFTP cleanup**
+- **SFTP Deduplication:** VersoriFileTracker (KV-based state tracking)
+- **Key Difference from S3:** KV state tracking vs physical file movement
+- **File tracking**: VersoriFileTracker prevents duplicates; `forceReprocess` bypasses
+- **Event API**: Per-record failures don't block other records; rejection reports written to `errors/`
+- **SFTP cleanup in finally block**
+- **Auto-detection of JSON vs JSON Lines format**
+- **Buffer import for Deno/Versori compatibility**
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+### 📦 Package Information
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
+```bash
+npm install @fluentcommerce/fc-connect-sdk
+```
+---
+**Templates are designed for direct deployment; customize via activation variables.**
+---
+## 📦 SDK Imports (Verified - Versori Optimized)
+```typescript
+// ✅ VERIFIED IMPORTS - These match actual SDK exports
+import { Buffer } from 'node:buffer';
+import {
+ createClient, // Universal client factory
+ SftpDataSource, // SFTP operations with connection pooling
+ JSONParserService, // JSON parsing with format auto-detection
+ UniversalMapper, // Field mapping with SDK resolvers
+ VersoriFileTracker, // File processing state tracker
+ JobTracker, // Job status tracking
+} from '@fluentcommerce/fc-connect-sdk';
+import type { FluentClient, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
+// Versori platform imports
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+**✅ VERSORI PLATFORM - Use Native Logs:**
+- Use `log` from context: `const { log } = ctx;`
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+---
+## 🔐 SFTP Connection Setup (Recommended)
+**✅ BEST PRACTICE:** Store SFTP credentials in a Versori connection object with Basic Auth:
+**Connection Configuration:**
+1. In Versori platform, create a connection named `SFTP`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+### Code Implementation
+The workflow retrieves credentials programmatically from the Versori connection:
+```typescript
+import { Buffer } from 'node:buffer';
+// Retrieve SFTP credentials from connection configuration
+log.info('Retrieving SFTP credentials from connection configuration');
+let sftpUsername: string;
+let sftpPassword: string;
+try {
+  // Retrieve credentials from the 'SFTP' connection
+  const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+  if (!sftpCred?.accessToken) {
+    throw new Error('No SFTP credentials found in connection configuration');
+  }
+  // Decode base64 accessToken to get "username:password"
+  const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+  // Split on ':' to extract username and password
+  const parts = rawBasicAuth.split(':');
+  if (parts.length !== 2) {
+    throw new Error('Invalid SFTP credential format - expected username:password');
+  }
+  sftpUsername = parts[0];
+  sftpPassword = parts[1];
+  log.info('SFTP credentials retrieved successfully', {
+    hasUsername: !!sftpUsername,
+    hasPassword: !!sftpPassword,
+    usernameLength: sftpUsername.length,
+    passwordLength: sftpPassword.length,
+  });
+} catch (error: any) {
+  log.error('Failed to retrieve SFTP credentials', {
+    message: error instanceof Error ? error.message : String(error),
+    stack: error instanceof Error ? error.stack : undefined,
+    errorType: error instanceof Error ? error.constructor.name : 'Error'
+  });
+  return {
+    success: false,
+    error: 'Failed to retrieve SFTP credentials from connection configuration',
+    details: error?.message,
+    recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
+  };
+}
+// Use credentials in SftpDataSource
+const sftp = new SftpDataSource({
+  type: 'SFTP_JSON',
+  connectionId: 'sftp-product-sync',
+  name: 'Product Sync SFTP',
+  settings: {
+    host: activation.getVariable('sftpHost'),
+    port: parseInt(activation.getVariable('sftpPort') || '22', 10),
+    username: sftpUsername, // From connection
+    password: sftpPassword, // From connection
+    remotePath: activation.getVariable('sftpRemotePath'),
+    filePattern: activation.getVariable('filePattern'),
+    encoding: 'utf8',
+  },
+}, log);
+```
+### Accessing SFTP Credentials (Alternative Methods)
+Versori provides multiple methods to access connection credentials. The template uses `ctx.credentials().getAccessToken('SFTP')` which matches batch-api templates. Alternative methods include:
+#### Method 1: activation.connections (Simpler Syntax)
+**Alternative approach** - Direct access via `activation.connections`:
+```typescript
+const { activation } = ctx;
+// Access connection object
+const conn = activation.connections.SFTP;
+// Decode Basic Auth credentials
+const rawBasicAuth = Buffer.from(conn.accessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+log.info('SFTP credentials loaded', { username, host: activation.getVariable('sftpHost') });
+```
+**When to use:** Simple synchronous access, single connection scenarios
+#### Method 2: credentials().get() (Dynamic Selection)
+**Alternative approach** - Use `credentials().get()` for dynamic credential selection:
+```typescript
+const { credentials } = ctx;
+// Get specific connection by name
+const sftpCred = await credentials().get('SFTP');
+// Decode Basic Auth credentials
+const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+log.info('SFTP credentials loaded via credentials()', { username });
+```
+**When to use:**
+- Multiple SFTP connections (dev/prod, multiple partners)
+- Dynamic credential selection based on runtime logic
+- Need to list all available connections
+---
+## ⚙️ Activation Variables
+**Configuration is driven by activation variables - modify these instead of code:**
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
+```bash
+# SFTP Configuration
+SFTP_HOST=sftp.partner.com
+SFTP_PORT=22
+# SFTP Paths
+SFTP_REMOTE_PATH=/products/incoming
+SFTP_ARCHIVE_PATH=/products/processed
+SFTP_ERROR_PATH=/products/errors
+# File Processing
+FILE_PATTERN=products_*.json
+JSON_FORMAT=json                             # json or jsonlines
+MAX_FILES_PER_RUN=10
+# Product Configuration
+CATALOGUE_REF=PC:MASTER:2
+CATALOGUE_TYPE=MASTER
+# Event API Configuration
+EVENT_NAME=UPSERT_PRODUCT
+EVENT_MODE=async
+EVENT_CONCURRENCY=1
+# Fluent Configuration (via Versori connection)
+# Connection: fluent_commerce (OAuth2)
+FLUENT_RETAILER_ID=1
+# Feature Toggles
+REQUIRE_ABSOLUTE_PATHS=true                    # "true" for AWS Transfer Family, "false" for standard OpenSSH
+ENABLE_FILE_TRACKING=true                      # Enable/disable file tracking (default: true)
+# Event API Performance
+USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
+MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
+MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
+```
+**Security:**
+- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#-sftp-connection-setup-recommended) section below.
+- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
+### Variable Explanations
+| Variable        | Purpose             | Default          | Customization Hints         |
+| ----------------------- | -------------------------------- | ------------------------- | ----------------------------------- |
+| `SFTP_HOST`       | SFTP server hostname       | -             | Required - SFTP server address   |
+| `SFTP_PORT`       | SFTP server port         | `22`           | Standard SFTP port         |
+| `SFTP_REMOTE_PATH`   | Incoming files directory     | `/products/incoming`   | Where new files arrive       |
+| `SFTP_ARCHIVE_PATH`   | Processed files archive     | `/products/processed`   | Where files move after success   |
+| `SFTP_ERROR_PATH`     | Failed files directory      | `/products/errors`    | Where files move after errors    |
+| `FILE_PATTERN`      | File name filter         | `products_*.json`     | Glob pattern for matching files   |
+| `JSON_FORMAT`      | JSON format type         | `json`          | `json` or `jsonlines`        |
+| `CATALOGUE_REF`     | Product catalogue reference   | `PC:MASTER:2`       | Target catalogue in Fluent     |
+| `CATALOGUE_TYPE`     | Catalogue type          | `MASTER`         | Usually MASTER or STANDARD     |
+| `EVENT_NAME`       | Event to send          | `UPSERT_PRODUCT`     | Event name from Rubix        |
+| `EVENT_MODE`       | Event processing mode      | `async`          | async (recommended) or sync     |
+| `MAX_FILES_PER_RUN`    | Max files per execution     | `10`           | Prevent timeout on large batches  |
+| `FLUENT_RETAILER_ID`   | Retailer ID for Event API    | -             | Required - Fluent retailer ID   |
+| `EVENT_CONCURRENCY`   | Event sending concurrency    | `1`            | `1` = sequential, `>1` = parallel (3-10 recommended) |
+| `REQUIRE_ABSOLUTE_PATHS` | Require absolute SFTP paths   | `true`          | Recommended for clarity       |
+| `ENABLE_FILE_TRACKING`  | Enable KV-based file deduplication | `true`        | Set to `false` to disable file tracking |
+| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
+| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100`          | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
+| `MEMORY_BATCH_SIZE`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
+---
+## 🔒 SDK Automatic Behaviors (v0.1.40+)
+**The SDK automatically validates and retries for improved reliability:**
+### retailerId Validation
+- **SDK validates** `retailerId` before calling `sendEvent()`
+- **Checks:** `event.retailerId || client.retailerId`
+- **If missing:** Throws `"retailerId is required for Event API..."`
+- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
+### 401 Auth Retry
+- **Automatic retry** for platform auth failures (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s)
+- **Applies to:** All `sendEvent()` calls (async and sync modes)
+- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
+### 5xx Server Retry
+- **Automatic retry** for transient server errors (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
+- **Protects:** Against Fluent API transient failures
+### No Code Changes Required
+- All templates remain compatible
+- Retry logic is automatic and transparent
+- Better error messages guide configuration
+**See:** [Event API Guide](./event-api-guide.md) for complete details
+---
+### 📁 SFTP Deduplication Pattern (VersoriFileTracker)
+**CRITICAL: SFTP uses VersoriFileTracker (KV state), NOT physical file movement**
+VersoriFileTracker is **SFTP-specific** and tracks processed files in Versori KV storage. This prevents reprocessing even if files remain in the incoming directory.
+#### How SFTP Deduplication Works
+**4-Step Process:**
+1. **Discovery:** List files from `sftpIncomingPath` matching `filePattern`
+2. **Filter:** Check VersoriFileTracker - skip files already processed
+3. **Process:** Download, parse, transform, and send events
+4. **Track State:** Mark file as processed in KV (prevents future reprocessing)
+#### File Lifecycle Example
+```
+INITIAL STATE:
+sftp://host/products/incoming/products_20250124_001.json
+↓ (workflow discovers file)
+CHECKING STATE:
+- List files from "/products/incoming/"
+- VersoriFileTracker.isProcessed("products_20250124_001.json") → false
+↓ (not processed before)
+PROCESSING:
+- Download: products_20250124_001.json
+- Parse JSON → Map to events → Send to Event API
+↓ (success)
+STATE TRACKING:
+- VersoriFileTracker.markProcessed("products_20250124_001.json")
+- Move file to "/products/processed/products_20250124_001-20250124T183045.json"
+↓ (next run)
+DEDUPLICATION:
+- List files from "/products/incoming/" (file may or may not still be there)
+- VersoriFileTracker.isProcessed("products_20250124_001.json") → true
+- Skip file automatically
+```
+#### Key Differences: SFTP vs S3
+| Aspect          | SFTP (This Template)       | S3 (Other Templates)        |
+| ------------------------ | -------------------------------- | ----------------------------------- |
+| **Deduplication Method** | VersoriFileTracker (KV state)  | Physical file movement       |
+| **State Storage**    | KV store tracks processed files | None needed             |
+| **File Tracking**    | ✅ VersoriFileTracker required  | ❌ NO VersoriFileTracker      |
+| **Reprocessing**     | Clear KV state or forceReprocess | Move files back to incoming/    |
+| **Complexity**      | More complex (state management) | Simpler (no state management)    |
+| **Files After Process** | May remain in incoming/     | Physically moved to processed/   |
+#### Why SFTP Needs VersoriFileTracker
+**SFTP file operations are less atomic than S3:**
+- Files may remain in incoming directory even after processing
+- Partner systems may not clean up files immediately
+- Multiple workflows may access the same SFTP directory
+- Network issues can leave files in inconsistent state
+- **KV state tracking** provides reliable deduplication regardless of file state
+#### Reprocessing Files (If Needed)
+To reprocess a file that was already tracked:
+**Option 1: Clear KV State (Manual)**
+```typescript
+// In Versori console or workflow
+const kv = openKv(':project:');
+const tracker = new VersoriFileTracker(kv, 'product-ingestion');
+await tracker.clearFile('products_20250124_001.json');
+```
+**Option 2: Webhook with forceReprocess**
+```bash
+# Set forceReprocess: true in webhook payload
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "forceReprocess": true,
+  "filePattern": "products_20250124_001.json"
+ }'
+```
+**Use cases for reprocessing:**
+- Fixing bad data that was previously ingested
+- Reingesting after mapping config changes
+- Testing with production files
+- Recovering from partial failures
+#### Summary
+✅ **DO:** Use VersoriFileTracker for SFTP deduplication
+✅ **DO:** Move files to processed/ or errors/ directories for housekeeping
+✅ **DO:** Import and use VersoriFileTracker in SFTP templates
+✅ **DO:** Use KV state tracking for reliable deduplication
+❌ **DON'T:** Rely on physical file movement alone (files may remain)
+❌ **DON'T:** Use VersoriFileTracker for S3 templates (different use case)
+---
+### 📁 SFTP Path vs File Pattern
+**Critical:** Directory paths and file patterns are configured separately.
+- **Directory Paths:** Configured via activation variables:
+ - `sftpIncomingPath` - Where new files arrive (e.g., `/products/incoming`)
+ - `sftpArchivePath` - Where successful files are archived (e.g., `/products/processed`)
+ - `sftpErrorPath` - Where failed files are moved (e.g., `/products/errors`)
+- **File Pattern:** Configured via `filePattern` activation variable
+ - Matches file names (not full paths)
+ - Supports wildcards: `*.json`, `products_*.json`, `urgent_*.json`
+**How it works:**
+```typescript
+// The workflow lists files like this:
+sftp.listFiles({
+ path: '/products/incoming', // ← Path from activation variable
+ pattern: 'products_*.json', // ← Pattern matches file names
+});
+```
+**Example:** With these settings:
+```json
+{
+ "sftpIncomingPath": "/products/incoming",
+ "filePattern": "products_*.json"
+}
+```
+The workflow will find files like:
+- ✅ `/products/incoming/products_20250124.json`
+- ✅ `/products/incoming/products_urgent.json`
+- ❌ `/products/incoming/orders_20250124.json` (doesn't match pattern)
+- ❌ `/orders/incoming/products_20250124.json` (wrong path)
+**Note:** To process files from different paths, change `sftpIncomingPath` (not `filePattern`).
+---
+## 📄 Mapping Configuration
+**File:** `config/products.import.json`
+```json
+{
+ "name": "products.import.json",
+ "version": "1.0.0",
+ "description": "JSON → Product Event Mapping",
+ "fields": {
+  "ref": {
+   "source": "ref",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "type": {
+   "source": "type",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "STANDARD"
+  },
+  "status": {
+   "source": "status",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "ACTIVE"
+  },
+  "gtin": {
+   "source": "gtin",
+   "resolver": "sdk.trim"
+  },
+  "name": {
+   "source": "name",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "summary": {
+   "source": "summary"
+  },
+  "categoryRefs": {
+   "source": "categoryRefs"
+  },
+  "price": {
+   "source": "price"
+  },
+  "taxType": {
+   "source": "taxType"
+  },
+  "attributes": {
+   "source": "attributes",
+   "defaultValue": []
+  }
+ }
+}
+```
+**Note on JSON Mapping:**
+Unlike CSV (flat structure), JSON often has nested objects and arrays that map directly to Fluent schema:
+```json
+{
+ "ref": "G_PROD_001",
+ "categoryRefs": ["CAT1", "CAT2"],
+ "price": [
+  { "type": "DEFAULT", "currency": "USD", "value": 52.0 },
+  { "type": "SPECIAL", "currency": "USD", "value": 9.0 }
+ ],
+ "taxType": {
+  "country": "AU",
+  "group": "Tax Group",
+  "tariff": "Tax Tariff"
+ }
+}
+```
+These fields can use **direct passthrough** (no custom resolvers needed):
+```json
+{
+ "categoryRefs": { "source": "categoryRefs" },
+ "price": { "source": "price" },
+ "taxType": { "source": "taxType" }
+}
+```
+For advanced patterns, see SDK Universal Mapping guide.
+---
+## Expected JSON Format
+### Standard JSON (Array of Objects)
+```json
+[
+ {
+  "ref": "G_PROD_WITH_NO_STANDARD",
+  "type": "VARIANT",
+  "status": "ACTIVE",
+  "gtin": "MH01-XS-Orange",
+  "name": "Chaz Kangeroo Hoodie-XS-Orange main",
+  "summary": "<p>test short description</p>",
+  "categoryRefs": ["STANDARD_CATEGORY"],
+  "price": [
+   { "type": "DEFAULT", "currency": "USD", "value": 52.0 },
+   { "type": "SPECIAL", "currency": "USD", "value": 9.0 }
+  ],
+  "taxType": {
+   "country": "AU",
+   "group": "Tax Group",
+   "tariff": "Tax Tariff"
+  }
+ },
+ {
+  "ref": "G_PROD_002",
+  "type": "VARIANT",
+  "status": "ACTIVE",
+  "gtin": "MH02-M-Blue",
+  "name": "Kangeroo Hoodie-M-Blue",
+  "summary": "<p>Blue variant</p>",
+  "categoryRefs": ["STANDARD_CATEGORY", "SEASONAL"],
+  "price": [
+   { "type": "DEFAULT", "currency": "USD", "value": 45.0 },
+   { "type": "SPECIAL", "currency": "USD", "value": 35.0 }
+  ],
+  "taxType": {
+   "country": "AU",
+   "group": "Tax Group",
+   "tariff": "Tax Tariff"
+  }
+ }
+]
+```
+### JSON Lines Format (One JSON Object Per Line)
+```json
+{"ref":"G_PROD_WITH_NO_STANDARD","type":"VARIANT","status":"ACTIVE","gtin":"MH01-XS-Orange","name":"Chaz Kangeroo Hoodie-XS-Orange main","summary":"<p>test short description</p>","categoryRefs":["STANDARD_CATEGORY"],"price":[{"type":"DEFAULT","currency":"USD","value":52.0},{"type":"SPECIAL","currency":"USD","value":9.0}],"taxType":{"country":"AU","group":"Tax Group","tariff":"Tax Tariff"}}
+{"ref":"G_PROD_002","type":"VARIANT","status":"ACTIVE","gtin":"MH02-M-Blue","name":"Kangeroo Hoodie-M-Blue","summary":"<p>Blue variant</p>","categoryRefs":["STANDARD_CATEGORY","SEASONAL"],"price":[{"type":"DEFAULT","currency":"USD","value":45.0},{"type":"SPECIAL","currency":"USD","value":35.0}],"taxType":{"country":"AU","group":"Tax Group","tariff":"Tax Tariff"}}
+```
+**JSONParserService auto-detects format** - NO array normalization, returns parsed data as-is.
+---
+## Event Sending Configuration
+**Simple Configuration:** One variable controls everything
+```json
+{
+ "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
+}
+```
+**How it works:**
+- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
+- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
+- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
+- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
+**Configuration Guidelines:**
+- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
+- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
+- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
+- **Note:** Fluent API supports concurrent requests - adjust based on your needs
+**Why Single Variable?**
+- **Simpler:** One variable instead of two (`mode` + `concurrency`)
+- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
+- **Less config:** Fewer activation variables to manage
+- **Flexible:** Easy to tune performance (just change the number)
+---
+## 🔧 Production Reference Implementation
+### Processing Modes
+| Mode    | Default | What happens                                            | When to use                         |
+| ---------- | ------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
+| `per-file` | ✅   | Process one file end-to-end (discover → parse → Event API → archive) before moving on        | Recommended for production SFTP feeds and very large files  |
+| `chunked` | ➖   | Processes `fileChunkSize` files at a time, completing each chunk before the next          | High-volume feeds where per-file would take too long     |
+| `batch`  | ➖   | Loads every matched file into memory, produces a single Event API job, and archives when complete  | Tiny datasets or smoke tests only              |
+Set the mode via activation variables:
+```json
+{
+ "processingMode": "per-file",
+ "fileChunkSize": "5",
+ "maxFilesPerRun": "25"
+}
+```
+### Versori Workflows Structure
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+### Recommended Project Structure
+```
+product-event-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── product.types.ts            # Shared type definitions
+```
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+---
+## Workflow Files
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+All time-based triggers that run automatically on cron schedules.
+#### `src/workflows/scheduled/daily-product-sync.ts`
+**Purpose**: Automatic Daily product sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+/**
+ * Scheduled Workflow: Daily Product Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: product-sync.service.ts
+ */
+export const dailyProductSync = schedule(
+  'product-sync-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv } = ctx;
+    const jobId = `product-sync-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+    log.info('🚀 [WORKFLOW START] Daily product sync workflow triggered', { jobId });
+    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
+    await tracker.updateJob(jobId, { status: 'processing' });
+    try {
+      // Reuse shared orchestration logic
+      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
+      await tracker.markCompleted(jobId, result);
+      log.info('✅ [WORKFLOW END] Daily product sync completed successfully', {
+        jobId,
+        duration: result.duration,
+        filesProcessed: result.filesProcessed
+      });
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      log.error('❌ [WORKFLOW ERROR] Daily product sync failed', {
+        jobId,
+        message: e?.message,
+        stack: e?.stack
+      });
+      await tracker.markFailed(jobId, e);
+      return { success: false, jobId, error: e?.message };
+    }
+  })
+);
+```
+---
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+All HTTP-based triggers that create webhook endpoints.
+#### `src/workflows/webhook/adhoc-product-sync.ts`
+**Purpose**: Manual product sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+/**
+ * Webhook: Manual Product Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
+ * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5 }
+ *
+ * Pattern: webhook().then(http()) - needs Fluent API access
+ * Uses shared service: product-sync.service.ts
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const adhocProductSync = webhook('product-sync-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'product-sync-adhoc', // Versori validates API key
+}).then(
+  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, data } = ctx;
+    const jobId = `product-sync-adhoc-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const filePattern = data?.filePattern as string;
+    const maxFiles = data?.maxFiles as number;
+    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+    });
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'manual',
+      stage: 'initialization',
+      status: 'queued',
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
+      .then((result) => {
+        log.info('✅ [BACKGROUND] Product sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          filesFailed: result.filesFailed,
+          recordsProcessed: result.recordsProcessed,
+          duration: result.duration,
+        });
+        return tracker.markCompleted(jobId, result);
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+        log.error('❌ [BACKGROUND] Product sync failed', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+        });
+        return tracker.markFailed(jobId, errorMessage);
+      });
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Product sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+#### `src/workflows/webhook/job-status-check.ts`
+**Purpose**: Query job status
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
+**Request body**: `{ "jobId": "product-sync-1234567890" }`
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
+ * Request body: { "jobId": "product-sync-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const productSyncJobStatus = webhook('product-sync-job-status', {
+  response: { mode: 'sync' },
+  connection: 'product-sync-job-status',
+}).then(
+  fn('status', async ctx => {
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+    if (!jobId) {
+      return { success: false, error: 'jobId required' };
+    }
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+---
+### 3. Entry Point (`index.ts`)
+**Purpose**: Register all workflows with Versori platform
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * Versori automatically discovers and registers exported workflows
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ * - src/services/           → Shared service logic (reusable across workflows)
+ */
+// Import scheduled workflows
+import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
+// Import webhook workflows
+import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
+import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
+// Register all workflows with Versori platform
+export {
+  // Scheduled (time-based triggers)
+  dailyProductSync,
+  // Webhooks (HTTP-based triggers)
+  adhocProductSync,
+  productSyncJobStatus,
+};
+```
+**What Gets Exposed:**
+- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
+- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
+- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
+---
+### Adding New Workflows
+**To add a scheduled workflow:**
+1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+**To add a webhook workflow:**
+1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+**Example - Adding hourly delta sync:**
+```typescript
+// src/workflows/scheduled/hourly-delta-sync.ts
+export const hourlyDeltaSync = schedule(
+  'product-delta-hourly',
+  '0 * * * *' // Every hour
+).then(
+  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
+    // Delta sync logic (skip BPP)
+    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
+    return result;
+  })
+);
+// index.ts (add to imports and exports)
+import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
+export { daily_product_sync, hourlyDeltaSync, ... };
+```
+---
+## 3. Type Definitions (src/types/product-ingestion.types.ts)
+```typescript
+/**
+ * Type Definitions for Product Ingestion
+ *
+ * Centralized type definitions for product ingestion workflow
+ */
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Product interface - represents transformed product data
+ */
+export interface Product {
+ ref: string;
+ type?: string;
+ status?: string;
+ name: string;
+ summary?: string;
+ gtin?: string;
+ catalogue?: { ref?: string };
+ attributes?: Record<string, unknown>;
+}
+/**
+ * Event configuration
+ */
+export interface EventConfig {
+ eventName: string;
+ catalogueRef: string;
+ catalogueType: string;
+ eventMode: 'async' | 'sync';
+}
+/**
+ * Event result - tracks success/failure counts
+ */
+export interface EventResult {
+ eventsSent: number;
+ eventsFailed: number;
+ errors: Array<{ productRef: string; error: string }>;
+}
+/**
+ * Process file result
+ */
+export interface ProcessFileResult {
+ success: boolean;
+ products: Product[];
+ error?: string;
+}
+/**
+ * Versori Context Interface
+ * Represents the Versori runtime context passed to workflow functions
+ */
+export interface VersoriContext {
+ log: {
+  info: (message: string, data?: Record<string, unknown>) => void;
+  warn: (message: string, data?: Record<string, unknown>) => void;
+  error: (message: string, data?: Record<string, unknown>) => void;
+  debug?: (message: string, data?: Record<string, unknown>) => void;
+ };
+ openKv: (namespace: string) => {
+  get: (key: string) => Promise<unknown>;
+  set: (key: string, value: unknown) => Promise<void>;
+  delete: (key: string) => Promise<void>;
+ };
+ activation: {
+  getVariable: (name: string) => string | undefined;
+  connections?: Record<string, unknown>;
+ };
+ connections?: Record<string, unknown>;
+ data?: unknown;
+ fetch?: typeof fetch;
+}
+/**
+ * Parameters for ingestion workflow
+ */
+export interface ProductIngestionParams {
+ jobId: string;
+ triggeredBy: 'schedule' | 'webhook';
+ filePattern?: string;
+ maxFiles?: number;
+ forceReprocess?: boolean;
+ catalogueRef?: string;
+ priority?: 'normal' | 'high';
+}
+/**
+ * Result from ingestion workflow
+ */
+export interface ProductIngestionResult {
+ success: boolean;
+ jobId: string;
+ filesProcessed: number;
+ filesFailed: number;
+ filesSkipped?: number;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ fileResults: FileProcessingResult[];
+ error?: string;
+}
+/**
+ * Per-file processing result
+ */
+export interface FileProcessingResult {
+ fileName: string;
+ success: boolean;
+ skipped?: boolean;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ duration: number;
+ error?: string;
+}
+```
+---
+## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
+```typescript
+/**
+ * Product File Processor Service
+ *
+ * Downloads JSON files from SFTP, parses, and transforms with UniversalMapper.
+ * JSON-specific: Handles both array format and object with products array.
+ */
+import {
+ SftpDataSource,
+ JSONParserService,
+ UniversalMapper,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
+/**
+ * Service for processing JSON product files
+ */
+export class ProductFileProcessorService {
+ constructor(
+  private sftp: SftpDataSource,
+  private jsonParser: JSONParserService,
+  private mapper: UniversalMapper,
+  private catalogueRef: string
+ ) {}
+ /**
+  * Download JSON file from SFTP, parse, and transform with UniversalMapper
+  *
+  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
+  * - Explicit null assignments after each step
+  * - Finally block guarantees cleanup
+  * - Prevents OOM errors on large JSON files
+  */
+ async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
+  // ✅ CRITICAL: Variables for cleanup tracking
+  let jsonContent: string | null = null;
+  let parsed: unknown | null = null;
+  try {
+   // STEP 1: Download JSON content
+   jsonContent = (await this.sftp.downloadFile(remoteFilePath, {
+    encoding: 'utf8',
+   })) as string;
+   // STEP 2: Parse JSON with type safety
+   try {
+    parsed = await this.jsonParser.parse(jsonContent);
+   } catch (parseError: unknown) {
+    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
+    return {
+     success: false,
+     products: [],
+     error: `JSON parse error: ${errorMessage}`,
+    };
+   }
+   // ✅ Clear jsonContent from memory - no longer needed after parsing
+   jsonContent = null;
+   // STEP 3: JSON-SPECIFIC - Normalize based on structure (array vs object with products array)
+   const rawProducts = Array.isArray(parsed)
+    ? parsed
+    : (parsed as any)?.products
+     ? Array.isArray((parsed as any).products)
+      ? (parsed as any).products
+      : [(parsed as any).products]
+     : [];
+   if (rawProducts.length === 0) {
+    return {
+     success: false,
+     products: [],
+     error: 'No products found in JSON',
+    };
+   }
+   // ✅ Clear parsed data from memory - only need rawProducts now
+   parsed = null;
+   // STEP 4: Transform with UniversalMapper
+   // JSON-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
+   const sourceDataWithContext = rawProducts.map(item => ({
+    ...item,
+    $context: { catalogueRef: this.catalogueRef },
+   }));
+   const mappingResult = await this.mapper.map(sourceDataWithContext);
+   if (!mappingResult.success) {
+    return {
+     success: false,
+     products: [],
+     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
+    };
+   }
+   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+     skippedFields: mappingResult.skippedFields,
+     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+    });
+   }
+   return {
+    success: true,
+    products: mappingResult.data as Product[],
+   };
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   return {
+    success: false,
+    products: [],
+    error: errorMessage,
+   };
+  } finally {
+   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
+   jsonContent = null;
+   parsed = null;
+  }
+ }
+}
+```
+---
+## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
+```typescript
+/**
+ * Event Sender Service
+ *
+ * Sends product events to Fluent Commerce Event API with per-record error handling.
+ * Continues processing on individual failures (Event API best practice).
+ * Supports configurable concurrency (sequential or parallel).
+ */
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
+/**
+ * Service for sending events to Fluent Commerce Event API
+ *
+ * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
+ */
+export class EventSenderService {
+ constructor(
+  private client: FluentClient,
+  private log?: any  // ✅ Optional logger for progress tracking
+ ) {}
+ /**
+  * Send events to Fluent Commerce Event API with configurable concurrency
+  *
+  * **Performance Characteristics:**
+  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
+  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
+  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
+  *
+  * **Implementation Strategy:**
+  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
+  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
+  * - Both modes: Per-record error tracking (failures don't block other events)
+  *
+  * @param products - Array of products to send as events
+  * @param eventConfig - Event configuration (name, catalogue ref, mode)
+  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
+  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
+  */
+ async sendEvents(
+  products: Product[],
+  eventConfig: EventConfig,
+  concurrency: number = 1
+ ): Promise<EventResult> {
+  // Validate concurrency (guard against invalid values)
+  const safeConc = Math.max(1, Math.floor(concurrency));
+  // Result accumulators
+  let eventsSent = 0;
+  let eventsFailed = 0;
+  const errors: Array<{ productRef: string; error: string }> = [];
+  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
+  if (this.log) {
+   this.log.info('📤 Starting event sending', {
+    totalProducts: products.length,
+    concurrency: safeConc,
+    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
+    eventName: eventConfig.eventName,
+    catalogueRef: eventConfig.catalogueRef
+   });
+  }
+  // Helper: Build event payload (DRY - reused in both modes)
+  const buildPayload = (product: Product) => ({
+   name: eventConfig.eventName,
+   entityRef: eventConfig.catalogueRef,
+   entityType: 'PRODUCT_CATALOGUE' as const,
+   entitySubtype: eventConfig.catalogueType,
+   rootEntityRef: eventConfig.catalogueRef,
+   rootEntityType: 'PRODUCT_CATALOGUE' as const,
+   attributes: product,
+  });
+  // ============================================================================
+  // SEQUENTIAL MODE (concurrency === 1)
+  // ============================================================================
+  if (safeConc === 1) {
+   for (let i = 0; i < products.length; i++) {
+    const product = products[i];
+    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
+    if (this.log && i % 10 === 0) {
+     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
+      productRef: product.ref,
+      progress: `${((i / products.length) * 100).toFixed(1)}%`
+     });
+    }
+    try {
+     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
+     eventsSent++;
+    } catch (err: unknown) {
+     eventsFailed++;
+     const errorMsg = err instanceof Error ? err.message : String(err);
+     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
+     // Continue processing (failure doesn't block other products)
+    }
+   }
+   // ✅ PRODUCTION ENHANCEMENT: Log completion
+   if (this.log) {
+    this.log.info('✅ Sequential event sending completed', {
+     totalProducts: products.length,
+     eventsSent,
+     eventsFailed
+    });
+   }
+   return { eventsSent, eventsFailed, errors };
+  }
+  // ============================================================================
+  // PARALLEL MODE (concurrency > 1)
+  // ============================================================================
+  const totalChunks = Math.ceil(products.length / safeConc);
+  for (let i = 0; i < products.length; i += safeConc) {
+   const chunk = products.slice(i, i + safeConc);
+   const chunkNumber = Math.floor(i / safeConc) + 1;
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
+   if (this.log) {
+    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
+     chunkNumber,
+     totalChunks,
+     productsInChunk: chunk.length,
+     productRange: `${i + 1}-${i + chunk.length}`,
+     totalProducts: products.length,
+     progress: `${((i / products.length) * 100).toFixed(1)}%`
+    });
+   }
+   // Fire all requests in chunk concurrently
+   const results = await Promise.allSettled(
+    chunk.map(product =>
+     this.client
+      .sendEvent(buildPayload(product), eventConfig.eventMode)
+      .then(() => ({ success: true as const, product }))
+      .catch(error => ({ success: false as const, product, error }))
+    )
+   );
+   // Aggregate chunk results into totals
+   let chunkSuccess = 0;
+   let chunkFailed = 0;
+   for (const result of results) {
+    if (result.status === 'fulfilled' && result.value.success) {
+     eventsSent++;
+     chunkSuccess++;
+    } else {
+     eventsFailed++;
+     chunkFailed++;
+     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
+     const product = result.status === 'fulfilled' ? result.value.product : null;
+     errors.push({
+      productRef: product?.ref || 'unknown',
+      error: error?.message || String(error) || 'unknown error',
+     });
+    }
+   }
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
+   if (this.log) {
+    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
+     chunkNumber,
+     totalChunks,
+     chunkSuccess,
+     chunkFailed,
+     totalSentSoFar: eventsSent,
+     totalFailedSoFar: eventsFailed,
+     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
+    });
+   }
+  }
+  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
+  if (this.log) {
+   this.log.info('✅ Parallel event sending completed', {
+    totalProducts: products.length,
+    totalChunks,
+    concurrency: safeConc,
+    eventsSent,
+    eventsFailed
+   });
+  }
+  return { eventsSent, eventsFailed, errors };
+ }
+}
+```
+---
+## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
+```typescript
+/**
+ * Event Logger Service
+ *
+ * Writes event processing logs to SFTP for audit/debugging.
+ * Optional - can be used to create rejection reports.
+ */
+import { Buffer } from 'node:buffer';
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Service for writing event logs to SFTP
+ */
+export class EventLoggerService {
+ constructor(private sftp: SftpDataSource) {}
+ /**
+  * Write event log to SFTP
+  *
+  * @param remotePath - SFTP path for log file
+  * @param logData - Log data to write (JSON format)
+  */
+ async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
+  try {
+   const logContent = JSON.stringify(logData, null, 2);
+   await this.sftp.uploadFile(remotePath, logContent);
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   throw new Error(`Failed to write event log: ${errorMessage}`);
+  }
+ }
+}
+```
+---
+## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
+```typescript
+/**
+ * MAIN INGESTION ORCHESTRATION SERVICE
+ *
+ * This is the heart of the ingestion workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Discover files on SFTP
+ * 3. Download and parse JSON files
+ * 4. Transform data with UniversalMapper
+ * 5. Send events to Fluent Commerce
+ * 6. Archive processed files
+ * 7. Track file processing state
+ * 8. Track job progress with JobTracker
+ */
+import { Buffer } from 'node:buffer';
+import {
+ createClient,
+ SftpDataSource,
+ JSONParserService,
+ UniversalMapper,
+ VersoriFileTracker,
+ JobTracker,
+ type FluentClient,
+ type JobStatus,
+} from '@fluentcommerce/fc-connect-sdk';
+import type {
+ ProductIngestionParams,
+ ProductIngestionResult,
+ FileProcessingResult,
+ EventConfig,
+} from '../types/product-ingestion.types';
+import { ProductFileProcessorService } from './product-file-processor.service';
+import { EventSenderService } from './event-sender.service';
+import { EventLoggerService } from './event-logger.service';
+import { joinSftpPath, timestampedName, extractFileName } from '../utils/sftp-path.utils';
+import mappingConfig from '../../config/products.import.json' with { type: 'json' };
+// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
+/**
+ * Query job status from KV store
+ */
+export async function getJobStatus(
+ kv: ReturnType<VersoriContext['openKv']>,
+ jobId: string,
+ log: VersoriContext['log']
+): Promise<JobStatus | undefined> {
+ try {
+  const tracker = new JobTracker(kv, log);
+  return await tracker.getJob(jobId);
+ } catch (error: unknown) {
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  log.error('Failed to get job status', {
+   jobId,
+   message: errorMessage,
+   stack: error instanceof Error ? error.stack : undefined,
+   errorType: error instanceof Error ? error.constructor.name : 'Error'
+  });
+  return undefined;
+ }
+}
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ */
+export async function executeProductIngestion(
+ ctx: VersoriContext,
+ params: ProductIngestionParams
+): Promise<ProductIngestionResult> {
+ const { log, openKv, activation } = ctx;
+ const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
+ const kv = openKv(':project:');
+ const tracker = new JobTracker(kv, log);
+ const startTime = Date.now();
+ const fileResults: FileProcessingResult[] = [];
+ // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
+ let sftp: SftpDataSource | undefined;
+ try {
+  // ═══════════════════════════════════════════════════════════
+  // STEP 1/8: Initialize Job Tracking
+  // ═══════════════════════════════════════════════════════════
+  log.info('🔧 [STEP 1/8] Initializing job tracking', { jobId });
+  await tracker.createJob(jobId, {
+   triggeredBy,
+   filePattern: filePattern || 'default',
+   maxFiles: maxFiles || 'unlimited',
+   forceReprocess: !!forceReprocess
+  });
+  // ═══════════════════════════════════════════════════════════
+  // STEP 2/8: Initialize Fluent Client & SFTP Connection
+  // ═══════════════════════════════════════════════════════════
+  log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and SFTP', { jobId });
+  const client = await createClient(ctx);
+  if (!client) {
+   const errorMsg = 'Failed to create Fluent Commerce client';
+   log.error('❌ ' + errorMsg, {
+    recommendation: 'Verify fluent_commerce connection is configured with valid credentials'
+   });
+   throw new Error(errorMsg);
+  }
+  // ✅ CRITICAL: Set retailerId for Event API calls
+  const fluentRetailerId = activation.getVariable('fluentRetailerId');
+  if (!fluentRetailerId) {
+   const errorMsg = 'fluentRetailerId is required for Event API calls';
+   log.error('❌ ' + errorMsg, {
+    recommendation: 'Add fluentRetailerId to activation variables (e.g., "1")'
+   });
+   throw new Error(errorMsg);
+  }
+  client.setRetailerId(fluentRetailerId);
+  log.info('✅ RetailerId set for Event API', { retailerId: fluentRetailerId });
+  // Get SFTP credentials from connection (Basic Auth)
+  log.info('🔐 Retrieving SFTP credentials from connection');
+  let sftpUsername: string;
+  let sftpPassword: string;
+  try {
+   // Retrieve credentials from the 'SFTP' connection
+   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+   if (!sftpCred?.accessToken) {
+    throw new Error('No SFTP credentials found in connection configuration');
+   }
+   // Decode base64 accessToken to get "username:password"
+   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+   const parts = rawBasicAuth.split(':');
+   if (parts.length !== 2) {
+    throw new Error('Invalid SFTP credential format - expected username:password');
+   }
+   sftpUsername = parts[0];
+   sftpPassword = parts[1];
+   log.info('✅ SFTP credentials retrieved successfully', {
+    hasUsername: !!sftpUsername,
+    hasPassword: !!sftpPassword,
+    usernameLength: sftpUsername.length,
+    passwordLength: sftpPassword.length,
+   });
+  } catch (credError: any) {
+   log.error('❌ Failed to retrieve SFTP credentials', {
+    message: credError instanceof Error ? credError.message : String(credError),
+    stack: credError instanceof Error ? credError.stack : undefined,
+    errorType: credError instanceof Error ? credError.constructor.name : 'Error',
+    recommendation: 'Ensure SFTP connection is configured with Basic Authentication'
+   });
+   throw credError;
+  }
+  // Get SFTP configuration from activation variables
+  const sftpConfig = {
+   host: activation.getVariable('sftpHost'),
+   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
+   username: sftpUsername,
+   password: sftpPassword,
+   privateKey: activation.getVariable('sftpPrivateKey'),
+   incomingPath: activation.getVariable('sftpIncomingPath') || '/products/incoming',
+   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
+   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
+   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.json',
+   enableFileTracking: activation.getVariable('enableFileTracking') !== 'false' // Default: true
+  };
+  // Validate SFTP config
+  if (!sftpConfig.host) {
+   const errorMsg = 'SFTP configuration incomplete: missing sftpHost';
+   log.error('❌ ' + errorMsg, {
+    recommendation: 'Add sftpHost to activation variables (e.g., "sftp.partner.com")'
+   });
+   throw new Error(errorMsg);
+  }
+  if (!sftpConfig.password && !sftpConfig.privateKey) {
+   const errorMsg = 'SFTP configuration incomplete: missing password or privateKey';
+   log.error('❌ ' + errorMsg, {
+    recommendation: 'Ensure SFTP connection has valid credentials OR set sftpPrivateKey'
+   });
+   throw new Error(errorMsg);
+  }
+  // Initialize SFTP data source
+  log.info('🔧 Initializing SFTP data source', {
+   host: sftpConfig.host,
+   port: sftpConfig.port,
+   remotePath: sftpConfig.incomingPath,
+   filePattern: sftpConfig.filePattern,
+   enableFileTracking: sftpConfig.enableFileTracking
+  });
+  sftp = new SftpDataSource({
+   type: 'SFTP_JSON',
+   connectionId: 'sftp-product-ingestion',
+   name: 'Product Ingestion SFTP',
+   settings: {
+    host: sftpConfig.host,
+    port: sftpConfig.port,
+    username: sftpConfig.username,
+    password: sftpConfig.password,
+    privateKey: sftpConfig.privateKey,
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern,
+    encoding: 'utf8'
+   }
+  }, log);
+  try {
+   // Validate SFTP connection
+   await sftp.validateConnection();
+   log.info('✅ SFTP connection validated successfully');
+   // Ensure archive directories exist (recursive)
+   await sftp.createDirectory(sftpConfig.archivePath, true);
+   await sftp.createDirectory(sftpConfig.errorPath, true);
+   // ═══════════════════════════════════════════════════════════
+   // STEP 3/8: Discover Files on SFTP
+   // ═══════════════════════════════════════════════════════════
+   log.info('📂 [STEP 3/8] Discovering files on SFTP', {
+    jobId,
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern,
+    enableFileTracking: sftpConfig.enableFileTracking
+   });
+   await tracker.updateJob(jobId, {
+    status: 'processing',
+    stage: 'file_discovery',
+    message: 'Discovering files on SFTP'
+   });
+   // List files from SFTP
+   const allFiles = await sftp.listFiles({
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern
+   });
+   log.info(`📋 Discovered ${allFiles.length} files matching pattern`, {
+    pattern: sftpConfig.filePattern,
+    count: allFiles.length
+   });
+   // Apply maxFiles limit if specified
+   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
+   if (filesToProcess.length === 0) {
+    log.info('ℹ️ No files to process', {
+     remotePath: sftpConfig.incomingPath,
+     filePattern: sftpConfig.filePattern
+    });
+    await tracker.markCompleted(jobId, {
+     filesProcessed: 0,
+     message: 'No files found to process'
+    });
+    return {
+     success: true,
+     jobId,
+     filesProcessed: 0,
+     filesFailed: 0,
+     recordsProcessed: 0,
+     eventsSent: 0,
+     eventsFailed: 0,
+     fileResults: []
+    };
+   }
+   log.info(`🔄 Processing ${filesToProcess.length} files`, {
+    totalDiscovered: allFiles.length,
+    toProcess: filesToProcess.length,
+    maxFilesLimit: maxFiles || 'unlimited'
+   });
+   // Initialize services
+   const jsonParser = new JSONParserService();
+   const mapper = new UniversalMapper(mappingConfig);
+   // Get event configuration
+   const eventConfig: EventConfig = {
+    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
+    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
+    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
+    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
+   };
+   // Get event sending configuration
+   const eventConcurrency = Math.max(
+    1,
+    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
+   );
+   log.info(`Event concurrency: ${eventConcurrency}`, {
+    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
+    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
+   });
+   // Initialize class-based services
+   const fileProcessor = new ProductFileProcessorService(
+    sftp,
+    jsonParser,
+    mapper,
+    eventConfig.catalogueRef
+   );
+   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
+   const eventSender = new EventSenderService(client, log);
+   const eventLogger = new EventLoggerService(sftp);
+   // File tracker for deduplication
+   const fileTracker = new VersoriFileTracker(kv, 'product-ingestion', log);
+   // Use direct KV with dot-separated keys (NATS KV safe)
+   const processedKeyBase = 'fc_sdk.processed_files';
+   // Get SFTP path configuration
+   const requireAbsolutePaths = activation.getVariable('requireAbsolutePaths') === 'true';
+   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
+    const file = filesToProcess[fileIndex];
+    const fileStartTime = Date.now();
+    // Extract file name using SDK utility
+    const remoteFilePath = (file as any).path || file.name;
+    const baseName = extractFileName(remoteFilePath);
+    log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`, {
+     remotePath: remoteFilePath,
+     fileSize: (file as any).size || 'unknown'
+    });
+    try {
+     // Check if file already processed (unless force reprocess)
+     if (!forceReprocess && sftpConfig.enableFileTracking) {
+      const isProcessed = await fileTracker.isProcessed(baseName);
+      if (isProcessed) {
+       log.info(`⏭️ Skipping already processed file: ${baseName}`, {
+        reason: 'File tracking enabled and file previously processed'
+       });
+       fileResults.push({
+        fileName: baseName,
+        success: true,
+        skipped: true,
+        recordsProcessed: 0,
+        eventsSent: 0,
+        eventsFailed: 0,
+        duration: 0,
+       });
+       continue;
+      }
+     }
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'downloading',
+      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
+     });
+     // ═══════════════════════════════════════════════════════════
+     // STEP 4/8: Process File (Download + Parse + Map)
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📥 [STEP 4/8] Processing file: ${baseName}`);
+     const fileResult = await fileProcessor.downloadParseAndTransform(remoteFilePath);
+     log.info(`✅ File processing completed: ${baseName}`, {
+      success: fileResult.success,
+      recordCount: fileResult.products.length,
+      error: fileResult.error
+     });
+     if (!fileResult.success || fileResult.products.length === 0) {
+      const archivedName = timestampedName(baseName);
+      await sftp.moveFile(
+       remoteFilePath,
+       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
+      );
+      fileResults.push({
+       fileName: baseName,
+       success: false,
+       recordsProcessed: fileResult.products.length,
+       eventsSent: 0,
+       eventsFailed: 0,
+       duration: Date.now() - fileStartTime,
+       error: fileResult.error || 'Processing failed'
+      });
+      continue;
+     }
+     // ═══════════════════════════════════════════════════════════
+     // STEP 5/8: Send Events
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
+     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
+     log.info(`🚀 [EventSender] Starting event submission for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventName: eventConfig.eventName,
+      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
+      sampleProductRefs: sampleProductRefs.join(', '),
+      eventMode: eventConfig.eventMode
+     });
+     const eventResult = await eventSender.sendEvents(
+      fileResult.products,
+      eventConfig,
+      eventConcurrency
+     );
+     const eventsSent = eventResult.eventsSent;
+     const eventsFailed = eventResult.eventsFailed;
+     const eventDuration = Date.now() - fileStartTime;
+     log.info(`✅ [EventSender] Event submission completed for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
+      duration: eventDuration,
+      avgTimePerEvent: eventsSent > 0 ? Math.round(eventDuration / eventsSent) + 'ms' : 'N/A'
+     });
+     // ═══════════════════════════════════════════════════════════
+     // STEP 6/8: Archive File & Track State
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'archiving',
+      message: `Archiving ${baseName}`
+     });
+     const archivedName = timestampedName(baseName);
+     const targetDir = eventsFailed > 0 ? sftpConfig.errorPath : sftpConfig.archivePath;
+     log.info(`📁 Moving file to ${eventsFailed > 0 ? 'error' : 'processed'} directory`, {
+      sourcePath: remoteFilePath,
+      targetPath: joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
+     });
+     await sftp.moveFile(
+      remoteFilePath,
+      joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
+     );
+     // Mark as processed in file tracker (if enabled)
+     if (sftpConfig.enableFileTracking) {
+      await fileTracker.markProcessed(baseName, {
+       recordsProcessed: fileResult.products.length,
+       eventsSent,
+       eventsFailed,
+      });
+      log.info(`✅ File tracking state updated for: ${baseName}`);
+     }
+     log.info(`✅ File archived successfully: ${baseName}`, {
+      archivedName,
+      targetDir,
+      duration: Date.now() - fileStartTime
+     });
+     // Optional: Write event log for audit/debugging
+     if (eventsFailed > 0) {
+      const logPath = joinSftpPath(
+       requireAbsolutePaths,
+       sftpConfig.errorPath,
+       `${baseName.replace(/\.json$/i, '')}-event-log.json`
+      );
+      await eventLogger.writeEventLog(logPath, {
+       fileName: baseName,
+       totalRecords: fileResult.products.length,
+       eventsSent,
+       eventsFailed,
+       errors: eventResult.errors,
+       processedAt: new Date().toISOString()
+      });
+     }
+     fileResults.push({
+      fileName: baseName,
+      success: true,
+      recordsProcessed: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      duration: Date.now() - fileStartTime
+     });
+    } catch (error: unknown) {
+     // ✅ Enhanced error logging: Extract all error details for visibility + recommendations
+     const errorMessage = error instanceof Error ? error.message : String(error);
+     const errorDetails = {
+      message: errorMessage,
+      stack: error instanceof Error ? error.stack : undefined,
+      fileName: (error as any)?.fileName,
+      lineNumber: (error as any)?.lineNumber,
+      originalError: (error as any)?.context?.originalError?.message,
+      errorType: error instanceof Error ? error.name : 'Error',
+     };
+     // Provide actionable recommendations based on error type
+     let recommendation = 'Check error details above for specific issue';
+     if (errorMessage.includes('SFTP')) {
+      recommendation = 'Verify SFTP connection: check host, credentials, and network access';
+     } else if (errorMessage.includes('parse') || errorMessage.includes('JSON')) {
+      recommendation = 'Verify JSON file format: check for valid JSON syntax and encoding';
+     } else if (errorMessage.includes('mapping') || errorMessage.includes('validation')) {
+      recommendation = 'Review mapping configuration: ensure all required fields are present';
+     } else if (errorMessage.includes('retailerId') || errorMessage.includes('Event API')) {
+      recommendation = 'Verify fluentRetailerId activation variable and Event API connection';
+     }
+     log.error(`❌ Error processing file ${baseName}:`, {
+      ...errorDetails,
+      recommendation
+     });
+     try {
+      const archivedName = timestampedName(baseName);
+      await sftp.moveFile(
+       remoteFilePath,
+       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
+      );
+     } catch (moveError: unknown) {
+      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
+      log.error('Could not archive failed file', {
+       file: baseName,
+       message: moveErrorMessage,
+       stack: moveError instanceof Error ? moveError.stack : undefined,
+       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
+      });
+     }
+     fileResults.push({
+      fileName: baseName,
+      success: false,
+      recordsProcessed: 0,
+      eventsSent: 0,
+      eventsFailed: 0,
+      duration: Date.now() - fileStartTime,
+      error: errorMessage
+     });
+    }
+   }
+   // ═══════════════════════════════════════════════════════════
+   // STEP 8/8: Complete Job & Calculate Totals
+   // ═══════════════════════════════════════════════════════════
+   const totalDuration = Date.now() - startTime;
+   log.info('🏁 [STEP 8/8] Completing job and calculating totals', { jobId });
+   const filesProcessed = fileResults.filter(r => r.success && !r.skipped).length;
+   const filesFailed = fileResults.filter(r => !r.success).length;
+   const filesSkipped = fileResults.filter(r => r.skipped).length;
+   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
+   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
+   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
+   await tracker.markCompleted(jobId, {
+    filesProcessed,
+    filesFailed,
+    filesSkipped,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: totalDuration
+   });
+   log.info('✅ Ingestion completed successfully', {
+    jobId,
+    filesProcessed,
+    filesFailed,
+    filesSkipped,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: totalDuration,
+    avgFileTime: filesProcessed > 0 ? Math.round(totalDuration / filesProcessed) + 'ms' : 'N/A'
+   });
+   return {
+    success: true,
+    jobId,
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    fileResults
+   };
+  } finally {
+   // ⚠️ CRITICAL: Always dispose SFTP connection (safe null check)
+   if (sftp) {
+    await sftp.dispose();
+    log.info('SFTP connection disposed');
+   }
+  }
+ } catch (error: unknown) {
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  const errorStack = error instanceof Error ? error.stack : undefined;
+  log.error('Ingestion workflow failed', {
+   jobId,
+   message: errorMessage,
+   stack: errorStack,
+   errorType: error instanceof Error ? error.constructor.name : 'Error'
+  });
+  try {
+   await tracker.markFailed(jobId, error);
+  } catch (trackingError: unknown) {
+   const trackingErrorMessage =
+    trackingError instanceof Error ? trackingError.message : String(trackingError);
+   log.warn('Failed to mark job as failed in tracker', {
+    jobId,
+    trackingError: trackingErrorMessage,
+   });
+  }
+  return {
+   success: false,
+   jobId,
+   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
+   filesFailed: fileResults.filter(r => !r.success).length,
+   filesSkipped: fileResults.filter(r => r.skipped).length,
+   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
+   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
+   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
+   fileResults,
+   error: errorMessage,
+  };
+ } finally {
+  // ⚠️ CRITICAL: Ensure SFTP is disposed even if outer error occurs
+  if (sftp) {
+   try {
+    await sftp.dispose();
+    log.info('SFTP connection disposed (outer finally)');
+   } catch (disposeError: unknown) {
+    const disposeErrorMessage =
+     disposeError instanceof Error ? disposeError.message : String(disposeError);
+    log.warn('Error disposing SFTP in outer finally', { error: disposeErrorMessage });
+   }
+  }
+ }
+}
+```
+---
+## 8. Utility Functions (src/utils/)
+### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
+```typescript
+/**
+ * SFTP Path Utilities
+ *
+ * Helper functions for SFTP path operations with AWS Transfer Family support.
+ * Handles both absolute paths (AWS Transfer Family) and relative paths (standard OpenSSH).
+ */
+/**
+ * Join SFTP path segments safely
+ *
+ * Handles different SFTP server path requirements:
+ * - AWS Transfer Family: Requires absolute paths (leading /)
+ * - Standard OpenSSH: Supports relative paths
+ *
+ * @param requireAbsolutePath - true for AWS Transfer Family, false for standard OpenSSH
+ * @param parts - Path segments to join
+ * @returns Properly formatted SFTP path
+ */
+export function joinSftpPath(requireAbsolutePath: boolean, ...parts: string[]): string {
+ const cleaned = parts
+  .filter(Boolean)
+  .map(p => String(p).replace(/^\/+|\/+$/g, ''))
+  .join('/');
+ return requireAbsolutePath && !cleaned.startsWith('/') ? `/${cleaned}` : cleaned;
+}
+/**
+ * Generate timestamped filename for archival
+ *
+ * Adds ISO timestamp to filename before extension for unique archival.
+ * Handles JSON files specifically but can be adapted for other formats.
+ *
+ * @param name - Original filename
+ * @returns Filename with timestamp appended
+ */
+export function timestampedName(name: string): string {
+ const ts = new Date().toISOString().replace(/[:.]/g, '-');
+ const base = name.replace(/\.json$/i, '');
+ return `${base}-${ts}.json`;
+}
+/**
+ * Extract base filename from full SFTP path
+ *
+ * @param filePath - Full SFTP path (e.g., '/products/incoming/file.json')
+ * @returns Base filename only (e.g., 'file.json')
+ */
+export function extractFileName(filePath: string): string {
+ return filePath.split('/').pop() || filePath;
+}
+// Alias for backward compatibility
+export const getBaseName = extractFileName;
+```
+---
+### Job ID Generator (src/utils/job-id-generator.ts)
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking ingestion workflows
+ *
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
+ * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
+ */
+export function generateJobId(type: string, entity: string): string {
+ const now = new Date();
+ const dateStr = now.toISOString().slice(0, 10).replace(/-/g, '');
+ const timeStr = now.toISOString().slice(11, 19).replace(/:/g, '');
+ const randomStr = Math.random().toString(36).substring(2, 8);
+ return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
+}
+/**
+ * Parse job ID components
+ */
+export function parseJobId(jobId: string): {
+ type: string;
+ entity: string;
+ date: string;
+ time: string;
+ random: string;
+} | null {
+ const parts = jobId.split('_');
+ if (parts.length !== 5) {
+  return null;
+ }
+ return {
+  type: parts[0],
+  entity: parts[1],
+  date: parts[2],
+  time: parts[3],
+  random: parts[4],
+ };
+}
+```
+---
+## 9. Package Configuration
+#### `package.json`
+```json
+{
+ "name": "sftp-json-product-event",
+ "version": "1.2.0",
+ "type": "module",
+ "main": "src/index.ts",
+ "scripts": {
+  "dev": "versori dev",
+  "build": "versori build",
+  "deploy": "versori deploy"
+ },
+ "dependencies": {
+  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+  "@versori/run": "latest"
+ },
+ "devDependencies": {
+  "@types/node": "^20.0.0",
+  "typescript": "^5.0.0"
+ }
+}
+```
+#### `tsconfig.json`
+```json
+{
+  "compilerOptions": {
+    "lib": ["ES2022", "DOM"],
+    "module": "ES2022",
+    "target": "ES2022",
+    "moduleResolution": "bundler",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "resolveJsonModule": true,
+    "isolatedModules": true
+  }
+}
+```
+**Note:** This hybrid configuration works for both local development and Versori deployment. It provides proper TypeScript checking while maintaining compatibility with Versori's Deno runtime.
+---
+## 10. Deployment Instructions
+### Deploy to Versori
+```bash
+# 1. Install dependencies
+npm install
+# 2. Test locally (if using Versori CLI)
+npm run dev
+# 3. Deploy to Versori platform
+npm run deploy
+```
+### Configure Activation Variables
+In Versori platform settings, configure all variables listed in the Activation Variables section above.
+---
+## 11. Testing
+### Test Scheduled Ingestion
+Upload a test JSON file to SFTP incoming path and wait for the scheduled run.
+**Check logs:**
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and SFTP
+[STEP 3/8] Discovering files on SFTP
+[FILE 1/1] Processing file: products_20250124.json
+[STEP 4/8] Downloading and parsing: products_20250124.json
+[STEP 5/8] Transforming 5 products from products_20250124.json
+[STEP 6/8] Sending 5 events to Fluent Commerce
+[STEP 7/8] Archiving file: products_20250124.json
+[STEP 8/8] Completing job and calculating totals
+```
+### Test Ad hoc Ingestion
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{}'
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "filePattern": "urgent_*.json", }'
+# Force reprocess
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "forceReprocess": true,
+  "filePattern": "products_20250124.json"
+ }'
+```
+### Test Job Status Query
+```bash
+curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "jobId": "ADHOC_PROD_20251024_183045_abc123"
+ }'
+```
+---
+## Monitoring
+### Success Response
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 50,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.json",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 50,
+   "eventsFailed": 0
+  }
+ ],
+ "duration": 12345
+}
+```
+### Partial Success Response
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 45,
+ "eventsFailed": 5,
+ "results": [
+  {
+   "file": "products_2025-01-22.json",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 45,
+   "eventsFailed": 5,
+   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
+  }
+ ],
+ "duration": 12345
+}
+```
+### Error Response
+```json
+{
+ "success": false,
+ "filesProcessed": 0,
+ "filesFailed": 1,
+ "totalRecords": 0,
+ "eventsSent": 0,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.json",
+   "success": false,
+   "error": "JSON parse error: Invalid structure"
+  }
+ ],
+ "duration": 876
+}
+```
+### Monitoring Metrics
+Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
+- **Files Processed** - Total files successfully processed
+- **Events Sent** - Total events sent to Fluent Commerce
+- **Events Failed** - Events that failed (check rejection reports)
+- **Processing Duration** - Time taken for complete workflow
+- **Rate Limiting** - Watch for 429 errors indicating throttling
+Use the status webhook for dashboards and automated monitoring.
+---
+- **No files found:** Check `sftpIncomingPath` and `filePattern` (file names only)
+- **JSON parse failed:** Move to errors/; validate JSON structure and encoding
+- **Files skipped (already processed):** Check VersoriFileTracker KV state; use `forceReprocess: true` if needed
+- **High event failures:** Inspect logs; consider Batch API for very high volumes
+- **429 throttling:** Add small backoff/delay or use Batch API
+- **SFTP connection errors:** Verify credentials (connection object), host, port, and permissions
+- **Connection pool exhausted:** Check connection cleanup in finally blocks
+---
+## 13. Key Takeaways
+- ✅ **Refactored Architecture (v2.0.0)** - 3 service functions for clean separation:
+ - `processFile()`: Download + Parse + Map (NO array normalization)
+ - `sendEvents()`: Loop records, send events with per-record error handling
+ - `writeEventLog()`: Write rejection logs to SFTP with Buffer
+- ✅ **Per-file Processing** - One file at a time for memory efficiency
+- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
+- ✅ **Buffer import** - Always import Buffer: `import { Buffer } from 'node:buffer';`
+- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
+- ✅ **JobTracker** - Track job lifecycle with KV persistence
+- ✅ **VersoriFileTracker** - KV-based deduplication for SFTP (required)
+- ✅ **SFTP deduplication** - KV state tracking prevents reprocessing
+- ✅ **SFTP dispose()** - Always cleanup in finally block
+- ✅ **SFTP method names** - listFiles, downloadFile, moveFile (not \*Object)
+- ✅ **Connection credentials** - Use `activation.connections` for Basic Auth
+- ✅ **Per-record error handling** - Continue on individual failures
+- ✅ **Externalized mapping** - Use JSON config file for field mappings
+- ✅ **File-level error handling** - Don't stop on single file failure
+- ✅ **JSON auto-detection** - Supports standard JSON and JSON Lines (NO normalization)
+- ✅ **SFTP vs S3** - Different deduplication strategies (KV state vs physical movement)
+---
+[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)