npm - @fluentcommerce/fc-connect-sdk - Versions diffs - 0.1.53 → 0.1.55 - Mend

@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (495) hide show

package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-csv-product-event.md CHANGED Viewed

@@ -1,2295 +1,2295 @@
----
-template_id: tpl-ingest-sftp-csv-to-product-event
-canonical_filename: template-ingestion-sftp-csv-product-event.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: ingestion
-source: sftp-csv
-destination: fluent-event-api
-entity: product
-format: csv
-logging: versori
-status: stable
-features:
-  - batched-events
-  - attribute-transformation
-  - memory-management
-  - json-serialization-handling
-  - enhanced-logging
----
-# Template: Ingestion - SFTP CSV to Product Event
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
-- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
-- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large CSV files)
-- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
-- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
----
-## 📚 STEP 1: Load These Docs Into Your AI (Human Checklist)
-1. REQUIRED (load all)
-  - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-  - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
-  - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
----
-## 📋 Implementation Prompt
-```
-I need a Versori scheduled ingestion that:
-1) Discovers CSV files on SFTP with file tracking to skip duplicates
-2) Downloads and parses CSV (returns array directly - no normalization needed)
-3) Transforms records with UniversalMapper per mapping JSON
-4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
-5) Archives files to processed/ or errors/ and writes optional rejection report
-6) Tracks progress with JobTracker and exposes a job-status webhook
-7) Uses native Versori log from context
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
----
-## 📋 Template Overview
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP CSV, transforms it, and sends events to the Fluent Commerce Event API.
-### What This Template Does
-```
-�������������������������������������������������������������������
-�          INGESTION WORKFLOW              �
-�������������������������������������������������������������������
-1. TRIGGER
-  �� Scheduled (Cron): Runs automatically every hour
-  �� Ad hoc (Webhook): Manual trigger for immediate processing
-  �� Status Query (Webhook): Check job progress
-2. DISCOVER FILES (SftpDataSource)
-  �� List files from SFTP directory
-  �� Filter by pattern (products_*.csv)
-  �� Check file tracking (skip processed)
-  "� Sort by oldest first
-3. FOR EACH FILE (Per-File Processing)
-  �� Process one file completely before moving to next
-  "� Steps 4-6 happen for each file
-4. PROCESS FILE (Service Function)
-  �� Download CSV from SFTP
-  �� Parse CSV to array (CSVParserService)
-  �� Transform with UniversalMapper
-  "� Return transformed products
-5. SEND EVENTS (Service Function)
-  �� Loop through transformed products
-  �� Send UPSERT_PRODUCT event (async)
-  �� Track success/failure count
-  "� Continue on individual failures
-6. ARCHIVE & TRACK (SftpDataSource + KV)
-  �� Move file to processed/ or errors/
-  �� Generate timestamped archive name
-  �� Mark as processed in KV
-  �� Optional: Write event log
-  "� Store file result
-7. REPEAT for next file (loop back to step 3)
-8. COMPLETE JOB (JobTracker)
-  �� Calculate totals (all files)
-  �� Store final result in KV
-  �� Enable status queries via webhook
-  "� Return complete results
-```
-### Key Features
-- **Service Functions**: Modular design with 3 service functions:
- 1. `processFile()` - Downloads, parses, transforms (SftpDataSource + CSVParserService + UniversalMapper)
- 2. `sendEvents()` - Loops through products, sends events with per-record error handling
- 3. `writeEventLog()` - Writes event logs to SFTP using Buffer (for audit/debugging)
-- **Per-File Processing**: Each file is processed completely (download ? parse ? map ? send ? archive) before moving to next
-- **Job Tracking**: JobTracker persists stage/status to Versori KV for visibility and status queries
-- **Execution Modes**: Scheduled (cron), ad hoc (webhook), status query (webhook)
-- **File Tracking**: KV-based file tracking prevents duplicates; `forceReprocess` bypasses
-- **Event API**: Per-record failures don't block other records; optional rejection reports
-- **SFTP Cleanup**: Connection disposal in finally block ensures cleanup
-Note: JobTracker recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-### 📦 Package Information
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-**Installation:**
-```bash
-npm install @fluentcommerce/fc-connect-sdk@latest
-```
-**Version:** Use latest SDK version from npm
----
-**Templates are designed for direct deployment; customize via activation variables.**
----
-## 📦 SDK Imports (Verified - Versori Optimized)
-```typescript
-// ? VERIFIED IMPORTS - These match actual SDK exports
-import { Buffer } from 'node:buffer';
-import {
- createClient, // Universal client factory
- SftpDataSource, // SFTP operations with connection pooling
- CSVParserService, // CSV parsing
- UniversalMapper, // Field mapping with SDK resolvers
- VersoriFileTracker, // File processing state tracker
- JobTracker, // Job status tracking
-} from "@fluentcommerce/fc-connect-sdk";
-import type {
- FluentClient,
- SftpDataSourceConfig,
-} from "@fluentcommerce/fc-connect-sdk";
-// Versori platform imports
-import { schedule, webhook, http, fn, credentials } from "@versori/run";
-```
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-**? VERSORI PLATFORM - Use Native Logs:**
-- Use `log` from context: `const { log } = ctx;`
-- Native Versori logs are simpler and automatically integrated with platform monitoring
----
-## SFTP Connection Setup
-**Two methods available for accessing SFTP credentials:**
-### Method 1: activation.connections (Recommended)
-**? BEST PRACTICE - Most Secure & Flexible**
-Store SFTP credentials in a Versori connection and access via `activation.connections`:
-**Connection Configuration:**
-1. In Versori platform, create a connection named `SFTP`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-**Access in code:**
-```typescript
-const { activation } = ctx;
-const conn = activation.connections.SFTP;
-const rawAccessToken = conn.accessToken;
-const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-```
-**Benefits:**
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection reusable across workflows
-- ✅ Access all connections dynamically
-- ✅ No activation variables needed for secrets
-- ✅ Easier credential rotation
-- ✅ Built-in connection health monitoring
-**Security:** Connections are encrypted at rest and in transit by Versori platform.
-### Method 2: credentials().get() (Alternative)
-**Alternative approach using credentials API:**
-```typescript
-const { credentials } = ctx;
-const conn = credentials().get('SFTP');
-const rawAccessToken = conn.accessToken;
-const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
-const [username, password] = rawBasicAuth.split(':');
-```
-**When to use:**
-- Single-connection workflows
-- Workflows using `credentials()` API
-- Simpler syntax for single credential access
-**Comparison:**
-| Feature | `activation.connections` | `credentials().get()` |
-|---------|-------------------------|----------------------|
-| **Security** | Same (Versori vault) | Same (Versori vault) |
-| **Access Pattern** | Object property | Method call |
-| **Multiple Connections** | Enumerate all via `Object.keys()` | Need connection names |
-| **Dynamic Discovery** | Yes | Limited |
-| **Future-proof** | Yes (recommended API) | Yes (maintained) |
-**📚 Complete guide:** See [SFTP Credential Access Security Guide](../../../../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for:
-- Detailed security comparison
-- Error handling patterns
-- Connection validation
-- Multi-connection scenarios
-- Best practices
-**Template uses:** `activation.connections` (Method 1) throughout - most flexible for production use.
-## ⚙️ Activation Variables
-**Configuration is driven by activation variables - modify these instead of code:**
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
-```bash
-# SFTP Configuration
-SFTP_HOST=sftp.partner.com
-SFTP_PORT=22
-# SFTP Paths
-SFTP_REMOTE_PATH=/incoming/products/
-SFTP_ARCHIVE_PATH=/archive/products/
-SFTP_ERROR_PATH=/errors/products/
-# File Processing
-FILE_PATTERN=.csv
-MAX_FILES_PER_RUN=10
-# Product Configuration
-CATALOGUE_REF=PC:MASTER:2
-CATALOGUE_TYPE=MASTER
-# Event API Configuration
-EVENT_NAME=UPSERT_PRODUCT
-EVENT_MODE=async
-# Fluent Configuration (via Versori connection)
-# Connection: fluent_commerce (OAuth2)
-FLUENT_RETAILER_ID=1
-# Feature Toggles
-ENABLE_FILE_TRACKING=true                      # Enable/disable file tracking (default: true)
-# Event API Performance
-USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
-MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
-MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
-```
-**Security:**
-- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup) section below.
-- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
-### Variable Explanations
-| Variable      | Purpose           | Default        | Customization Hints       |
-| ------------------- | --------------------------- | --------------------- | -------------------------------- |
-| `FLUENT_RETAILER_ID` | Retailer ID for Event API  | -           | Required - Fluent retailer ID  |
-| **SFTP Credentials**  | *From Connection*      |            | *See connection setup above*                   |
-| `SFTP_HOST`     | SFTP server hostname    | -           | Required - your SFTP server   |
-| `SFTP_PORT`     | SFTP server port      | `22`         | Usually 22, sometimes 2222    |
-| `SFTP_REMOTE_PATH` | Incoming files directory  | `/incoming/products/` | Where new files arrive      |
-| `SFTP_ARCHIVE_PATH` | Processed files archive   | `/archive/products/` | Where files move after success  |
-| `SFTP_ERROR_PATH`   | Failed files directory   | `/errors/products/`  | Where files move after errors  |
-| `FILE_PATTERN`    | File name filter      | `.csv`   | Extension or glob pattern for matching files |
-| `CATALOGUE_REF`   | Product catalogue reference | `PC:MASTER:2`     | Target catalogue in Fluent    |
-| `CATALOGUE_TYPE`   | Catalogue type       | `MASTER`       | Usually MASTER or STANDARD    |
-| `EVENT_NAME`     | Event to send        | `UPSERT_PRODUCT`   | Event name from Rubix      |
-| `EVENT_MODE`     | Event processing mode    | `async`        | async (recommended) or sync   |
-| `MAX_FILES_PER_RUN`  | Max files per execution   | `10`         | Prevent timeout on large batches (0 = unlimited) |
-| `ENABLE_FILE_TRACKING` | Enable file deduplication | `"true"`      | `"true"` = skip already processed files, `"false"` = process all files |
-| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false` | `true` = batched events (10x faster for 100+ products), `false` = individual events |
-| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100` | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
-| `MEMORY_BATCH_SIZE` | Products per mapping batch | `250` | Processes products in batches during mapping (prevents OOM on large files) |
----
-## 🔒 SDK Automatic Behaviors (v0.1.40+)
-**The SDK automatically validates and retries for improved reliability:**
-### retailerId Validation
-- **SDK validates** `retailerId` before calling `sendEvent()`
-- **Checks:** `event.retailerId || client.retailerId`
-- **If missing:** Throws `"retailerId is required for Event API..."`
-- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
-### 401 Auth Retry
-- **Automatic retry** for platform auth failures (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s)
-- **Applies to:** All `sendEvent()` calls (async and sync modes)
-- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
-### 5xx Server Retry
-- **Automatic retry** for transient server errors (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
-- **Protects:** Against Fluent API transient failures
-### No Code Changes Required
-- All templates remain compatible
-- Retry logic is automatic and transparent
-- Better error messages guide configuration
-**See:** [Event API Guide](./event-api-guide.md) for complete details
----
-### 📁 File Tracking & Deduplication
-**How file tracking prevents duplicates:**
-1. **Discovery:** List all files matching `filePattern` from `sftpIncomingPath`
-2. **Filter:** Query VersoriFileTracker to check which files were already processed
-3. **Process:** Only process untracked files (skips duplicates automatically)
-4. **Mark:** Mark file as processed after successful Event API ingestion
-5. **Archive:** Move file to `sftpProcessedPath` (or `sftpErrorPath` on failure)
-**File lifecycle:**
-```
-/incoming/products_20250124_001.csv
- ?� (discovered)
-VersoriFileTracker.wasFileProcessed() ? false
- ?� (process file)
-Parse CSV ? Map to events ? Send to Event API
- ?� (success)
-VersoriFileTracker.markFileProcessed()
- ?� (archive)
-/processed/products_20250124_001-20250124T183045.csv
-```
-**Force reprocess:**
-Set `forceReprocess: true` in webhook payload to bypass file tracker and reprocess files. Useful for:
-- Fixing bad data that was previously ingested
-- Reingesting after mapping config changes
-- Testing with production files
-**Note:** Force reprocess doesn't prevent marking as processed - files are still tracked after reprocessing.
----
-### 📁 SFTP Path vs File Pattern
-**Critical:** Folder paths and file patterns are configured separately.
-- **Folder Paths:** Configured via activation variables:
- - `sftpIncomingPath` - Where new files arrive (e.g., `/products/incoming`)
- - `sftpProcessedPath` - Where successful files are archived (e.g., `/products/processed`)
- - `sftpErrorPath` - Where failed files are moved (e.g., `/products/errors`)
-- **File Pattern:** Configured via `filePattern` activation variable
- - Matches filenames ONLY (not folder paths)
- - Supports wildcards: `*.csv`, `products_*.csv`, `urgent_*.csv`
-**How it works:**
-```typescript
-// The workflow lists files like this:
-sftp.listFiles({
- remotePath: "/products/incoming", // ?� Folder from activation variable
- filePattern: "products_*.csv", // ?� Pattern matches filenames only
-});
-```
-**Example:** With these settings:
-```json
-{
- "sftpIncomingPath": "/products/incoming",
- "filePattern": "products_*.csv"
-}
-```
-The workflow will find files like:
-- ✅ `/products/incoming/products_20250124.csv`
-- ✅ `/products/incoming/products_urgent.csv`
-- ❌ `/products/incoming/orders_20250124.csv` (doesn't match pattern)
-- ❌ `/orders/incoming/products_20250124.csv` (wrong folder)
-**Note:** To process files from different folders, change `sftpIncomingPath` (not `filePattern`).
----
-## 📄 Mapping Configuration
-**File:** `config/products.import.csv.json`
-```json
-{
- "name": "products.import.csv",
- "version": "1.0.0",
- "description": "CSV ? Product Event Mapping",
- "fields": {
-  "ref": {
-   "source": "ref",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "type": {
-   "source": "type",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "STANDARD"
-  },
-  "status": {
-   "source": "status",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "ACTIVE"
-  },
-  "gtin": {
-   "source": "gtin",
-   "resolver": "sdk.trim"
-  },
-  "name": {
-   "source": "name",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "summary": {
-   "source": "summary"
-  },
-  "categoryRefs": {
-   "source": "category_refs",
-   "resolver": "custom.splitByPipe"
-  },
-  "price": {
-   "resolver": "custom.buildPriceArray"
-  },
-  "taxType": {
-   "resolver": "custom.buildTaxType"
-  },
-  "attributes": {
-   "defaultValue": []
-  }
- }
-}
-```
-**Custom Resolvers:**
-```typescript
-const customResolvers = {
- /**
-  * Split pipe-separated string into array
-  * Example: "CAT1|CAT2|CAT3" ? ["CAT1", "CAT2", "CAT3"]
-  */
- 'custom.splitByPipe': (value: string) => {
-  if (!value || value.trim() === '') return [];
-  return value.split('|').map(v => v.trim()).filter(v => v !== '');
- },
- /**
-  * Build price array from flat CSV columns
-  * Expects: price_default_currency, price_default_value, price_special_currency, price_special_value
-  */
- 'custom.buildPriceArray': (value: any, sourceData: any) => {
-  const prices = [];
-  // Add default price
-  if (sourceData.price_default_value) {
-   prices.push({
-    type: 'DEFAULT',
-    currency: sourceData.price_default_currency || 'USD',
-    value: sourceData.price_default_value,
-   });
-  }
-  // Add special price if exists
-  if (sourceData.price_special_value) {
-   prices.push({
-    type: 'SPECIAL',
-    currency: sourceData.price_special_currency || 'USD',
-    value: sourceData.price_special_value,
-   });
-  }
-  return prices;
- },
- /**
-  * Build tax type object from flat CSV columns
-  * Expects: tax_country, tax_group, tax_tariff
-  */
- 'custom.buildTaxType': (value: any, sourceData: any) => {
-  if (!sourceData.tax_country) return undefined;
-  return {
-   country: sourceData.tax_country,
-   group: sourceData.tax_group,
-   tariff: sourceData.tax_tariff,
-  };
- },
-};
-```
-Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
-### Mapping customization and custom resolvers (how-to)
-(Condensed) Provide custom resolvers via UniversalMapper options if required.
-### Parse + map
-Handled inside orchestration (download ? parse ? map). No extra helpers needed.
----
-### Error handling & rejection reports
-When moving failed files to errors/, optionally write a JSON rejection report with totals and error list for audit purposes.
----
-## Expected CSV Format
-```csv
-ref,type,status,gtin,name,summary,category_refs,price_default_currency,price_default_value,price_special_currency,price_special_value,tax_country,tax_group,tax_tariff,catalogue_ref,catalogue_type
-G_PROD_WITH_NO_STANDARD,VARIANT,ACTIVE,MH01-XS-Orange,Chaz Kangeroo Hoodie-XS-Orange main,<p>test short description</p>,STANDARD_CATEGORY,USD,52.000000,USD,9.000000,AU,Tax Group,Tax Tariff,PC:MASTER:2,MASTER
-G_PROD_002,VARIANT,ACTIVE,MH02-M-Blue,Kangeroo Hoodie-M-Blue,<p>Blue variant</p>,STANDARD_CATEGORY|SEASONAL,USD,45.000000,USD,35.000000,AU,Tax Group,Tax Tariff,PC:MASTER:2,MASTER
-```
----
-## Event Sending Configuration
-**Simple Configuration:** One variable controls everything
-```json
-{
- "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
-}
-```
-**How it works:**
-- `eventConcurrency: 1` ? Sequential (sends events one at a time, safe default)
-- `eventConcurrency: 3` ? Parallel (sends 3 events concurrently)
-- `eventConcurrency: 5` ? Parallel (sends 5 events concurrently)
-- `eventConcurrency: 10` ? Parallel (sends 10 events concurrently)
-**Configuration Guidelines:**
-- **Conservative:** `1` ? Sequential (safe, predictable, ~1 req/sec)
-- **Balanced:** `3-5` ? Parallel (most common, ~3-5 req/sec)
-- **Aggressive:** `10` ? Parallel (high-volume, ~10 req/sec)
-- **Note:** Fluent API supports concurrent requests - adjust based on your needs
-**Why Single Variable?**
-- **Simpler:** One variable instead of two (`mode` + `concurrency`)
-- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
-- **Less config:** Fewer activation variables to manage
-- **Flexible:** Easy to tune performance (just change the number)
----
-## 🔧 Production Implementation
-### Processing Modes
-| Mode    | Default | What happens                                           | When to use                       |
-| ---------- | ------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
-| `per-file` | ?   | Process one file end-to-end (discover ? parse ? Event API ? archive) before moving on       | Recommended for production SFTP feeds and very large files |
-| `chunked` | ?   | Processes `fileChunkSize` files at a time, completing each chunk before the next         | High-volume feeds where per-file would take too long   |
-| `batch`  | ?   | Loads every matched file into memory, produces a single Event API job, and archives when complete | Tiny datasets or smoke tests only            |
-Set the mode via activation variables:
-```json
-{
- "processingMode": "per-file",
- "fileChunkSize": "5",
- "maxFilesPerRun": "25"
-}
-```
----
-## 🔧 Complete Production Code
-### Versori Workflows Structure
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-### Recommended Project Structure
-```
-product-event-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── product.types.ts            # Shared type definitions
-```
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
----
-## Workflow Files
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-All time-based triggers that run automatically on cron schedules.
-#### `src/workflows/scheduled/daily-product-sync.ts`
-**Purpose**: Automatic Daily product sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-/**
- * Scheduled Workflow: Daily Product Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: product-sync.service.ts
- */
-export const dailyProductSync = schedule(
-  'product-sync-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-product-sync', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log, openKv } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `product-sync-${executionStartTime}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-    log.info('🚀 [WORKFLOW] Starting scheduled product sync', { jobId });
-    await tracker.createJob(jobId, {
-      triggeredBy: 'schedule',
-      stage: 'initialization',
-      startTime: executionStartTime
-    });
-    await tracker.updateJob(jobId, { status: 'processing' });
-    try {
-      // Reuse shared orchestration logic
-      const result = await executeProductIngestion(ctx, jobId, tracker);
-      if (result.success) {
-        await tracker.markCompleted(jobId, result);
-        log.info('✅ [WORKFLOW] Product sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          duration: result.duration
-        });
-      } else {
-        await tracker.markFailed(jobId, result.error || 'Unknown error');
-        log.error('❌ [WORKFLOW] Product sync failed', { jobId, error: result.error });
-      }
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      const errorMessage = e instanceof Error ? e.message : String(e);
-      await tracker.markFailed(jobId, errorMessage);
-      log.error('❌ [WORKFLOW] Product sync failed with exception', {
-        jobId,
-        error: errorMessage,
-        stack: e instanceof Error ? e.stack : undefined
-      });
-      return { success: false, jobId, error: errorMessage };
-    }
-  })
-);
-```
----
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-All HTTP-based triggers that create webhook endpoints.
-#### `src/workflows/webhook/adhoc-product-sync.ts`
-**Purpose**: Manual product sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-/**
- * Webhook: Manual Product Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
- * Request body (optional): { filePattern: "urgent_*.csv", maxFiles: 5 }
- *
- * Pattern: webhook().then(http()) - needs Fluent API access
- * Uses shared service: product-sync.service.ts
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const adhocProductSync = webhook('product-sync-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'product-sync-adhoc', // Versori validates API key
-}).then(
-  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log, openKv, data } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `product-sync-adhoc-${executionStartTime}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const filePattern = data?.filePattern as string;
-    const maxFiles = data?.maxFiles as number;
-    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-    });
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'webhook',
-      stage: 'initialization',
-      status: 'queued',
-      startTime: executionStartTime,
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductIngestion(ctx, jobId, tracker)
-      .then((result) => {
-        if (result.success) {
-          log.info('✅ [BACKGROUND] Product sync completed successfully', {
-            jobId,
-            filesProcessed: result.filesProcessed,
-            filesFailed: result.filesFailed,
-            recordsProcessed: result.recordsProcessed,
-            duration: result.duration,
-          });
-          return tracker.markCompleted(jobId, result);
-        } else {
-          log.error('❌ [BACKGROUND] Product sync failed', {
-            jobId,
-            error: result.error,
-          });
-          return tracker.markFailed(jobId, result.error || 'Unknown error');
-        }
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-        log.error('❌ [BACKGROUND] Product sync failed with exception', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-        });
-        return tracker.markFailed(jobId, errorMessage);
-      });
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Product sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-#### `src/workflows/webhook/job-status-check.ts`
-**Purpose**: Query job status
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
-**Request body**: `{ "jobId": "product-sync-1234567890" }`
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
- * Request body: { "jobId": "product-sync-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const productSyncJobStatus = webhook('product-sync-job-status', {
-  response: { mode: 'sync' },
-  connection: 'product-sync-job-status',
-}).then(
-  fn('status', async ctx => {
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-    if (!jobId) {
-      return { success: false, error: 'jobId required' };
-    }
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
----
-### 3. Entry Point (`index.ts`)
-**Purpose**: Register all workflows with Versori platform
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * Versori automatically discovers and registers exported workflows
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- */
-// Import scheduled workflows
-import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
-// Import webhook workflows
-import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
-import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
-// Register all workflows
-export {
-  // Scheduled (time-based triggers)
-  dailyProductSync,
-  // Webhooks (HTTP-based triggers)
-  adhocProductSync,
-  productSyncJobStatus,
-};
-```
-**What Gets Exposed:**
-- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
-- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
-- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
----
-### Adding New Workflows
-**To add a scheduled workflow:**
-1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-**To add a webhook workflow:**
-1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-**Example - Adding hourly delta sync:**
-```typescript
-// src/workflows/scheduled/hourly-delta-sync.ts
-export const hourlyDeltaSync = schedule(
-  'product-delta-hourly',
-  '0 * * * *' // Every hour
-).then(
-  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
-    // Delta sync logic (skip BPP)
-    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
-    return result;
-  })
-);
-// index.ts (add to imports and exports)
-import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
-export { dailyProductSync, hourlyDeltaSync, ... };
-```
----
-## 3. Type Definitions (src/types/product-ingestion.types.ts)
-```typescript
-/**
- * Type Definitions for Product Ingestion
- *
- * Centralized type definitions for product ingestion workflow
- */
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Product interface - represents transformed product data
- */
-export interface Product {
- ref: string;
- type?: string;
- status?: string;
- name: string;
- summary?: string;
- gtin?: string;
- catalogue?: { ref?: string };
- attributes?: Record<string, unknown>;
-}
-/**
- * Event configuration
- */
-export interface EventConfig {
- eventName: string;
- catalogueRef: string;
- catalogueType: string;
- eventMode: 'async' | 'sync';
-}
-/**
- * Event result - tracks success/failure counts
- */
-export interface EventResult {
- eventsSent: number;
- eventsFailed: number;
- errors: Array<{ productRef: string; error: string }>;
-}
-/**
- * Process file result
- */
-export interface ProcessFileResult {
- success: boolean;
- products: Product[];
- error?: string;
-}
-/**
- * Versori Context Interface
- * Represents the Versori runtime context passed to workflow functions
- */
-export interface VersoriContext {
- log: {
-  info: (message: string, data?: Record<string, unknown>) => void;
-  warn: (message: string, data?: Record<string, unknown>) => void;
-  error: (message: string, data?: Record<string, unknown>) => void;
-  debug?: (message: string, data?: Record<string, unknown>) => void;
- };
- openKv: (namespace: string) => {
-  get: (key: string) => Promise<unknown>;
-  set: (key: string, value: unknown) => Promise<void>;
-  delete: (key: string) => Promise<void>;
- };
- activation: {
-  getVariable: (name: string) => string | undefined;
-  connections?: Record<string, unknown>;
- };
- connections?: Record<string, unknown>;
- data?: unknown;
- fetch?: typeof fetch;
-}
-/**
- * Parameters for ingestion workflow
- */
-export interface ProductIngestionParams {
- jobId: string;
- triggeredBy: 'schedule' | 'webhook';
- filePattern?: string;
- maxFiles?: number;
- forceReprocess?: boolean;
- catalogueRef?: string;
- priority?: 'normal' | 'high';
-}
-/**
- * Result from ingestion workflow
- */
-export interface ProductIngestionResult {
- success: boolean;
- jobId: string;
- filesProcessed: number;
- filesFailed: number;
- filesSkipped?: number;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- fileResults: FileProcessingResult[];
- error?: string;
-}
-/**
- * Per-file processing result
- */
-export interface FileProcessingResult {
- fileName: string;
- success: boolean;
- skipped?: boolean;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- duration: number;
- error?: string;
-}
-```
----
-## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
-```typescript
-/**
- * Product File Processor Service
- *
- * Downloads CSV files from SFTP, parses, and transforms with UniversalMapper.
- * CSV-specific: Returns arrays directly (no normalization needed).
- */
-import {
- SftpDataSource,
- CSVParserService,
- UniversalMapper,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
-/**
- * Service for processing CSV product files
- */
-export class ProductFileProcessorService {
- constructor(
-  private sftp: SftpDataSource,
-  private csvParser: CSVParserService,
-  private mapper: UniversalMapper,
-  private catalogueRef: string
- ) {}
- /**
-  * Download CSV file from SFTP, parse, and transform with UniversalMapper
-  *
-  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
-  * - Explicit null assignments after each step
-  * - Finally block guarantees cleanup
-  * - Prevents OOM errors on large CSV files
-  */
- async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
-  // ✅ CRITICAL: Variables for cleanup tracking
-  let csvContent: string | null = null;
-  let parsed: unknown | null = null;
-  try {
-   // STEP 1: Download CSV content
-   csvContent = (await this.sftp.downloadFile(remoteFilePath, {
-    encoding: 'utf8',
-   })) as string;
-   // STEP 2: Parse CSV with type safety
-   try {
-    parsed = await this.csvParser.parse(csvContent);
-   } catch (parseError: unknown) {
-    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
-    return {
-     success: false,
-     products: [],
-     error: `CSV parse error: ${errorMessage}`,
-    };
-   }
-   // ✅ Clear csvContent from memory - no longer needed after parsing
-   csvContent = null;
-   // STEP 3: CSV-SPECIFIC - Returns array directly, no normalization needed
-   const rawProducts = Array.isArray(parsed) ? parsed : [];
-   if (rawProducts.length === 0) {
-    return {
-     success: false,
-     products: [],
-     error: 'No products found in CSV',
-    };
-   }
-   // ✅ Clear parsed data from memory - only need rawProducts now
-   parsed = null;
-   // STEP 4: Transform with UniversalMapper
-   // CSV-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
-   const sourceDataWithContext = rawProducts.map(item => ({
-    ...item,
-    $context: { catalogueRef: this.catalogueRef },
-   }));
-   const mappingResult = await this.mapper.map(sourceDataWithContext);
-   if (!mappingResult.success) {
-    return {
-     success: false,
-     products: [],
-     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
-    };
-   }
-   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-     skippedFields: mappingResult.skippedFields,
-     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-    });
-   }
-   return {
-    success: true,
-    products: mappingResult.data as Product[],
-   };
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   return {
-    success: false,
-    products: [],
-    error: errorMessage,
-   };
-  } finally {
-   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
-   csvContent = null;
-   parsed = null;
-  }
- }
-}
-```
----
-## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
-```typescript
-/**
- * Event Sender Service
- *
- * Sends product events to Fluent Commerce Event API with per-record error handling.
- * Continues processing on individual failures (Event API best practice).
- * Supports configurable concurrency (sequential or parallel).
- */
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
-/**
- * Service for sending events to Fluent Commerce Event API
- *
- * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
- */
-export class EventSenderService {
- constructor(
-  private client: FluentClient,
-  private log?: any  // ✅ Optional logger for progress tracking
- ) {}
- /**
-  * Send events to Fluent Commerce Event API with configurable concurrency
-  *
-  * **Performance Characteristics:**
-  * - `concurrency: 1` ? Sequential processing (safe default, ~1 event/sec)
-  * - `concurrency: 3-5` ? Balanced throughput (~3-5 events/sec, good for most cases)
-  * - `concurrency: 10` ? High-volume processing (~10 events/sec, 100+ products)
-  *
-  * **Implementation Strategy:**
-  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
-  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
-  * - Both modes: Per-record error tracking (failures don't block other events)
-  *
-  * @param products - Array of products to send as events
-  * @param eventConfig - Event configuration (name, catalogue ref, mode)
-  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
-  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
-  */
- async sendEvents(
-  products: Product[],
-  eventConfig: EventConfig,
-  concurrency: number = 1
- ): Promise<EventResult> {
-  // Validate concurrency (guard against invalid values)
-  const safeConc = Math.max(1, Math.floor(concurrency));
-  // Result accumulators
-  let eventsSent = 0;
-  let eventsFailed = 0;
-  const errors: Array<{ productRef: string; error: string }> = [];
-  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
-  if (this.log) {
-   this.log.info('📤 Starting event sending', {
-    totalProducts: products.length,
-    concurrency: safeConc,
-    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
-    eventName: eventConfig.eventName,
-    catalogueRef: eventConfig.catalogueRef
-   });
-  }
-  // Helper: Build event payload (DRY - reused in both modes)
-  const buildPayload = (product: Product) => ({
-   name: eventConfig.eventName,
-   entityRef: eventConfig.catalogueRef,
-   entityType: 'PRODUCT_CATALOGUE' as const,
-   entitySubtype: eventConfig.catalogueType,
-   rootEntityRef: eventConfig.catalogueRef,
-   rootEntityType: 'PRODUCT_CATALOGUE' as const,
-   attributes: product,
-  });
-  // ============================================================================
-  // SEQUENTIAL MODE (concurrency === 1)
-  // ============================================================================
-  if (safeConc === 1) {
-   for (let i = 0; i < products.length; i++) {
-    const product = products[i];
-    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
-    if (this.log && i % 10 === 0) {
-     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
-      productRef: product.ref,
-      progress: `${((i / products.length) * 100).toFixed(1)}%`
-     });
-    }
-    try {
-     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
-     eventsSent++;
-    } catch (err: unknown) {
-     eventsFailed++;
-     const errorMsg = err instanceof Error ? err.message : String(err);
-     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
-     // Continue processing (failure doesn't block other products)
-    }
-   }
-   // ✅ PRODUCTION ENHANCEMENT: Log completion
-   if (this.log) {
-    this.log.info('✅ Sequential event sending completed', {
-     totalProducts: products.length,
-     eventsSent,
-     eventsFailed
-    });
-   }
-   return { eventsSent, eventsFailed, errors };
-  }
-  // ============================================================================
-  // PARALLEL MODE (concurrency > 1)
-  // ============================================================================
-  const totalChunks = Math.ceil(products.length / safeConc);
-  for (let i = 0; i < products.length; i += safeConc) {
-   const chunk = products.slice(i, i + safeConc);
-   const chunkNumber = Math.floor(i / safeConc) + 1;
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
-   if (this.log) {
-    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
-     chunkNumber,
-     totalChunks,
-     productsInChunk: chunk.length,
-     productRange: `${i + 1}-${i + chunk.length}`,
-     totalProducts: products.length,
-     progress: `${((i / products.length) * 100).toFixed(1)}%`
-    });
-   }
-   // Fire all requests in chunk concurrently
-   const results = await Promise.allSettled(
-    chunk.map(product =>
-     this.client
-      .sendEvent(buildPayload(product), eventConfig.eventMode)
-      .then(() => ({ success: true as const, product }))
-      .catch(error => ({ success: false as const, product, error }))
-    )
-   );
-   // Aggregate chunk results into totals
-   let chunkSuccess = 0;
-   let chunkFailed = 0;
-   for (const result of results) {
-    if (result.status === 'fulfilled' && result.value.success) {
-     eventsSent++;
-     chunkSuccess++;
-    } else {
-     eventsFailed++;
-     chunkFailed++;
-     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
-     const product = result.status === 'fulfilled' ? result.value.product : null;
-     errors.push({
-      productRef: product?.ref || 'unknown',
-      error: error?.message || String(error) || 'unknown error',
-     });
-    }
-   }
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
-   if (this.log) {
-    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
-     chunkNumber,
-     totalChunks,
-     chunkSuccess,
-     chunkFailed,
-     totalSentSoFar: eventsSent,
-     totalFailedSoFar: eventsFailed,
-     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
-    });
-   }
-  }
-  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
-  if (this.log) {
-   this.log.info('✅ Parallel event sending completed', {
-    totalProducts: products.length,
-    totalChunks,
-    concurrency: safeConc,
-    eventsSent,
-    eventsFailed
-   });
-  }
-  return { eventsSent, eventsFailed, errors };
- }
-}
-```
----
-## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
-```typescript
-/**
- * Event Logger Service
- *
- * Writes event processing logs to SFTP for audit/debugging.
- * Optional - can be used to create rejection reports.
- */
-import { Buffer } from 'node:buffer';
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-/**
- * Service for writing event logs to SFTP
- */
-export class EventLoggerService {
- constructor(private sftp: SftpDataSource) {}
- /**
-  * Write event log to SFTP
-  *
-  * @param remotePath - SFTP path for log file
-  * @param logData - Log data to write (JSON format)
-  */
- async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
-  try {
-   const logContent = JSON.stringify(logData, null, 2);
-   await this.sftp.uploadFile(remotePath, logContent);
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   throw new Error(`Failed to write event log: ${errorMessage}`);
-  }
- }
-}
-```
----
-## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
-```typescript
-/**
- * MAIN INGESTION ORCHESTRATION SERVICE
- *
- * This is the heart of the ingestion workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Discover files on SFTP
- * 3. Download and parse CSV files
- * 4. Transform data with UniversalMapper
- * 5. Send events to Fluent Commerce
- * 6. Archive processed files
- * 7. Track file processing state
- * 8. Track job progress with JobTracker
- *
- * NAMING PATTERN (consistent across all use cases):
- * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
- * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
- * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- *
- * - Change entity: Replace "Product" with "Inventory", "Location", etc.
- * - Change format: Replace CSVParserService with XMLParserService/JSONParserService
- * - Change source: Replace SftpDataSource with S3DataSource
- * - Change destination: Replace sendEvent with Batch API or GraphQL mutations
- */
-import { Buffer } from 'node:buffer';
-import {
- createClient,
- SftpDataSource,
- CSVParserService,
- UniversalMapper,
- VersoriFileTracker,
- JobTracker,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { FileMetadata } from '@fluentcommerce/fc-connect-sdk';
-import { ProductFileProcessorService } from './product-file-processor.service';
-import { EventSenderService } from './event-sender.service';
-import { EventLoggerService } from './event-logger.service';
-import { extractFileName } from '../utils/sftp-path.utils';
-import mappingConfig from '../../config/products.import.csv.json' with { type: 'json' };
-// ? VERSORI PLATFORM: Use native log from context, not LoggingService
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- * This function implements the complete ingestion workflow in 8 steps.
- * Each step is clearly commented for AI understanding.
- *
- * @param ctx - Versori context object containing fetch, connections, log, activation, openKv
- * @param jobId - Job ID for tracking
- * @param tracker - JobTracker instance for job lifecycle management
- */
-export async function executeProductIngestion(
- ctx: any,
- jobId: string,
- tracker: JobTracker
-) {
- // ? VERSORI PLATFORM: Extract native log from context
- const { log, activation, openKv } = ctx;
- const startTime = Date.now();
- log.info('🚀 [EXECUTION BOUNDARY] Starting product ingestion', { jobId });
- // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
- let sftp: SftpDataSource | undefined;
- try {
-  // �����������������������������������������������������������
-  // STEP 1: Initialize Fluent Client
-  // �����������������������������������������������������������
-  log.info('📡 [STEP 1/7] Initializing Fluent Commerce client', { jobId });
-  const client = await createClient(ctx);
-  log.info('✅ [DEBUG] Client created successfully');
-  /**
-   * setRetailerId() - REQUIRED for Event/Batch API operations
-   *
-   * retailerId must be set ONCE on the client after creation.
-   * This is required for Event API (sendEvent) operations.
-   */
-  const fluentRetailerId = activation.getVariable('fluentRetailerId');
-  if (!fluentRetailerId) {
-   log.error('❌ Missing required fluentRetailerId activation variable');
-   return {
-    success: false,
-    error: 'fluentRetailerId activation variable is required for Event API operations',
-    recommendation: 'Please configure fluentRetailerId in the Activation Variables section',
-   };
-  }
-  client.setRetailerId(fluentRetailerId);
-  log.info('✅ Client initialized with retailerId', { retailerId: fluentRetailerId });
-  // ========================================
-  // STEP 2: SFTP CREDENTIAL RETRIEVAL
-  // ========================================
-  log.info('🔐 [STEP 2/7] Retrieving SFTP credentials from connection');
-  let sftpUsername: string;
-  let sftpPassword: string;
-  try {
-   // Retrieve credentials from the 'SFTP' connection
-   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-   if (!sftpCred?.accessToken) {
-    throw new Error('No SFTP credentials found in connection configuration');
-   }
-   // Decode base64 accessToken to get "username:password"
-   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-   // Split on ':' to extract username and password
-   const parts = rawBasicAuth.split(':');
-   if (parts.length !== 2) {
-    throw new Error('Invalid SFTP credential format - expected username:password');
-   }
-   sftpUsername = parts[0];
-   sftpPassword = parts[1];
-   log.info('✅ SFTP credentials retrieved successfully', {
-    hasUsername: !!sftpUsername,
-    hasPassword: !!sftpPassword,
-   });
-  } catch (error: any) {
-   log.error('❌ Failed to retrieve SFTP credentials', {
-    message: error instanceof Error ? error.message : String(error),
-    stack: error instanceof Error ? error.stack : undefined,
-   });
-   return {
-    success: false,
-    error: 'Failed to retrieve SFTP credentials from connection configuration',
-    details: error?.message,
-    recommendation: 'Please ensure the SFTP connection is configured with Basic Authentication',
-   };
-  }
-  // ========================================
-  // STEP 3: CONFIGURATION & INITIALIZATION
-  // ========================================
-  log.info('⚙️ [STEP 3/7] Configuring SFTP and services', { jobId });
-  // Configure SFTP
-  const sftpConfig = {
-   host: activation.getVariable('sftpHost'),
-   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
-   username: sftpUsername,
-   password: sftpPassword,
-   remotePath: activation.getVariable('sftpRemotePath') || '/incoming/products/',
-   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
-   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
-   filePattern: activation.getVariable('filePattern') || '.csv',
-   enableFileTracking: activation.getVariable('enableFileTracking') !== 'false', // default: true
-  };
-  // Validate SFTP config
-  if (!sftpConfig.host) {
-   throw new Error('SFTP configuration incomplete: missing host');
-  }
-  // Initialize SFTP data source
-  sftp = new SftpDataSource(
-   {
-    type: 'SFTP_CSV',
-    connectionId: 'sftp-product-ingestion',
-    name: 'product-sync',
-    settings: {
-     host: sftpConfig.host,
-     port: sftpConfig.port,
-     username: sftpUsername,
-     password: sftpPassword,
-     remotePath: sftpConfig.remotePath,
-     archivePath: sftpConfig.archivePath,
-     errorPath: sftpConfig.errorPath,
-     filePattern: sftpConfig.filePattern,
-     encoding: 'utf8',
-     requireAbsolutePaths: true,
-    },
-   },
-   log
-  );
-  await sftp.validateConnection();
-  log.info('✅ SFTP connection validated successfully');
-  // Initialize services
-  const csvParser = new CSVParserService();
-  const mapper = new UniversalMapper(mappingConfig);
-  const kv = openKv(':project:');
-  const fileTracker = new VersoriFileTracker(kv, 'sftp-csv-product-sync');
-  // Initialize class-based services
-  const fileProcessor = new ProductFileProcessorService(
-    sftp,
-    csvParser,
-    mapper,
-    activation.getVariable('catalogueRef') || 'PC:MASTER:2'
-  );
-  // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
-  const eventSender = new EventSenderService(client, log);
-  const eventLogger = new EventLoggerService(sftp);
-  // �����������������������������������������������������������
-  // STEP 4: DISCOVER FILES ON SFTP
-  // �����������������������������������������������������������
-  log.info('📂 [STEP 4/7] Discovering files on SFTP', {
-   jobId,
-   remotePath: sftpConfig.remotePath,
-   filePattern: sftpConfig.filePattern,
-  });
-  await tracker.updateJob(jobId, {
-   status: 'processing',
-   stage: 'file_discovery',
-   message: 'Discovering files on SFTP',
-  });
-  // List files from SFTP
-  const files = await sftp.listFiles({
-   remotePath: sftpConfig.remotePath,
-   filePattern: sftpConfig.filePattern,
-  });
-  const maxFilesPerRun = parseInt(activation.getVariable('maxFilesPerRun') || '10', 10);
-  const csvFiles = maxFilesPerRun > 0 ? files.slice(0, maxFilesPerRun) : files;
-  log.info(`✅ Found ${csvFiles.length} CSV files matching pattern`);
-  if (csvFiles.length === 0) {
-   log.info('ℹ️ No files to process');
-   return {
-    success: true,
-    message: 'No files to process',
-    filesProcessed: 0,
-    filesFailed: 0,
-    recordsProcessed: 0,
-    eventsSent: 0,
-    eventsFailed: 0,
-    results: [],
-    duration: Date.now() - startTime,
-   };
-  }
-  log.info(`📋 Processing ${csvFiles.length} files`);
-  // ========================================
-  // STEP 5: PROCESS FILES
-  // ========================================
-  const results: any[] = [];
-  const eventConfig = {
-   catalogueRef: activation.getVariable('catalogueRef') || 'PC:MASTER:2',
-   catalogueType: activation.getVariable('catalogueType') || 'MASTER',
-   eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
-   eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
-  };
-  // Get event sending configuration
-  const eventConcurrency = Math.max(
-    1,
-    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
-  );
-  // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
-  // concurrency: 1 = sequential, concurrency > 1 = parallel
-  log.info(`Event concurrency: ${eventConcurrency}`, {
-    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
-    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
-  });
-  for (const file of csvFiles) {
-   const fileStartTime = Date.now();
-   const fileName = extractFileName(file.name);
-   try {
-    // Check if already processed (file tracking)
-    if (sftpConfig.enableFileTracking && (await fileTracker.wasFileProcessed(fileName))) {
-     log.info(`⏭️ Skipping already processed: ${fileName}`);
-     results.push({ file: fileName, success: true, skipped: true });
-     continue;
-    }
-    log.info(`🔄 Processing file: ${fileName}`);
-    // Download and parse
-    const csvContent = (await sftp.downloadFile(file.path, { encoding: 'utf8' })) as string;
-    const records = await csvParser.parse(csvContent);
-    const products: any[] = Array.isArray(records) ? records : records ? [records] : [];
-    log.info(`📊 Parsed ${products.length} products from ${fileName}`);
-    if (products.length === 0) {
-     await sftp.moveFile(file.path, `${sftpConfig.archivePath}${fileName}`, true);
-     results.push({ file: fileName, success: true, recordCount: 0 });
-     continue;
-    }
-    // Transform with UniversalMapper
-    const mapped = await mapper.map(products);
-    if (!mapped.success) {
-     throw new Error(`Mapping failed: ${mapped.errors?.join(', ')}`);
-    }
-    log.info(`✅ Transformed ${mapped.data.length} products for ${fileName}`);
-    // ═══════════════════════════════════════════════════════════
-    // STEP 6: Send Events
-    // ═══════════════════════════════════════════════════════════
-    log.info(`📤 [STEP 6/7] Sending events for ${fileName}`);
-    // ✅ Enhanced: Extract context for progress logging
-    const sampleProductRefs = mapped.data.slice(0, 5).map((p: any) => p.ref || p.skuRef);
-    const eventType = eventConfig.eventName || 'UPSERT_PRODUCT';
-    // ✅ Enhanced: Start logging with context
-    log.info(`[EventSender] Sending events for file "${fileName}"`, {
-      totalProducts: mapped.data.length,
-      eventType,
-      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
-      sampleProductRefs: sampleProductRefs.join(', '),
-      eventMode: eventConfig.eventMode || 'async'
-    });
-    const eventResult = await eventSender.sendEvents(
-      mapped.data,
-      eventConfig,
-      eventConcurrency
-    );
-    const eventsSent = eventResult.eventsSent;
-    const eventsFailed = eventResult.eventsFailed;
-    log.info(`✅ Events sent for ${fileName}`, {
-      successful: eventsSent,
-      failed: eventsFailed
-    });
-    // ✅ Enhanced: Completion logging with summary
-    log.info(`[EventSender] Event submission completed for file "${fileName}"`, {
-      totalProducts: mapped.data.length,
-      eventsSent,
-      eventsFailed,
-      successRate: mapped.data.length > 0 ? `${Math.round((eventsSent / mapped.data.length) * 100)}%` : '0%',
-      eventType
-    });
-    // Archive file and mark as processed
-    log.info(`📦 [STEP 7/7] Archiving file: ${fileName}`);
-    await sftp.moveFile(file.path, `${sftpConfig.archivePath}${fileName}`, true);
-    if (sftpConfig.enableFileTracking) {
-     await fileTracker.markFileProcessed(fileName, {
-      recordCount: products.length,
-      eventsSent,
-      eventsFailed,
-     });
-    }
-    results.push({
-     file: fileName,
-     success: true,
-     recordCount: products.length,
-     eventsSent,
-     eventsFailed,
-     duration: Date.now() - fileStartTime,
-    });
-    log.info(`✅ Successfully processed ${fileName}`, {
-     recordCount: products.length,
-     eventsSent,
-     eventsFailed,
-     duration: Date.now() - fileStartTime,
-    });
-   } catch (error: any) {
-    log.error(`❌ Error processing ${fileName}:`, {
-     message: error?.message || 'Unknown error',
-     stack: error?.stack,
-    });
-    // Move to error directory
-    try {
-     await sftp.moveFile(file.path, `${sftpConfig.errorPath}${fileName}`, true);
-    } catch (moveError) {
-     log.error(`❌ Failed to move error file: ${fileName}`);
-    }
-    results.push({ file: fileName, success: false, error: error.message });
-   }
-  }
-  // ========================================
-  // STEP 6: CLEANUP
-  // ========================================
-  try {
-   await sftp.dispose();
-   log.info('✅ [EXECUTION BOUNDARY] SFTP connection disposed successfully');
-  } catch (disposeError: any) {
-   log.error('⚠️ Failed to dispose SFTP connection', {
-    message: disposeError instanceof Error ? disposeError.message : String(disposeError),
-   });
-  }
-  const duration = Date.now() - startTime;
-  const filesProcessed = results.filter(r => r.success && !r.skipped).length;
-  const filesFailed = results.filter(r => !r.success).length;
-  const filesSkipped = results.filter(r => r.skipped).length;
-  log.info('🎉 [EXECUTION BOUNDARY] Product ingestion completed', {
-   jobId,
-   filesProcessed,
-   filesFailed,
-   filesSkipped,
-   duration,
-  });
-  return {
-   success: true,
-   filesProcessed,
-   filesSkipped,
-   filesFailed,
-   results,
-   duration,
-  };
- } catch (error: any) {
-  log.error('💥 [EXECUTION BOUNDARY] Product ingestion failed', {
-   jobId,
-   message: error instanceof Error ? error.message : String(error),
-   stack: error instanceof Error ? error.stack : undefined,
-  });
-  return {
-   success: false,
-   error: error.message,
-   duration: Date.now() - startTime,
-  };
- }
-}
-```
----
-## 8. Utility Functions (src/utils/)
-### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
-```typescript
-/**
- * SFTP Path Utilities
- *
- * Helper functions for SFTP path operations.
- */
-/**
- * Extract base filename from full SFTP path
- *
- * @param filePath - Full SFTP path or filename
- * @returns Base filename only
- *
- * @example
- * ```typescript
- * extractFileName('/products/incoming/products_20250101.csv')
- * // Returns: 'products_20250101.csv'
- *
- * extractFileName('products_20250101.csv')
- * // Returns: 'products_20250101.csv'
- * ```
- */
-export function extractFileName(filePath: string): string {
- return filePath.split('/').pop() || filePath;
-}
-```
----
-### Job ID Generator (src/utils/job-id-generator.ts)
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking ingestion workflows
- *
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
- * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
- *
- * NAMING: generate{Entity}JobId or generateJobId (generic)
- */
-/**
- * Generate unique job ID
- *
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
- * @param entity - Entity abbreviation (VP, IP, ORD, PRD)
- * @returns Unique job ID string
- */
-export function generateJobId(type: string, entity: string): string {
- const now = new Date();
- // Format: YYYYMMDD
- const dateStr = now.toISOString().slice(0, 10).replace(/-/g, "");
- // Format: HHMMSS
- const timeStr = now.toISOString().slice(11, 19).replace(/:/g, "");
- // Random suffix (6 chars)
- const randomStr = Math.random().toString(36).substring(2, 8);
- return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
-}
-/**
- * Parse job ID components
- *
- * @param jobId - Job ID to parse
- * @returns Parsed components or null if invalid
- */
-export function parseJobId(
- jobId: string
-): {
- type: string;
- entity: string;
- date: string;
- time: string;
- random: string;
-} | null {
- const parts = jobId.split("_");
- if (parts.length !== 5) {
-  return null;
- }
- return {
-  type: parts[0],
-  entity: parts[1],
-  date: parts[2],
-  time: parts[3],
-  random: parts[4],
- };
-}
-```
----
-## 5. Package Configuration
-### package.json
-```json
-{
- "name": "sftp-csv-product-event",
- "version": "1.2.0",
- "description": "Ingest product CSV files from SFTP and send to Fluent Commerce Event API",
- "type": "module",
- "main": "src/index.ts",
- "scripts": {
-  "dev": "versori dev",
-  "build": "versori build",
-  "deploy": "versori deploy"
- },
- "dependencies": {
-  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-  "@versori/run": "latest"
- },
- "devDependencies": {
-  "@types/node": "^20.0.0",
-  "typescript": "^5.0.0"
- }
-}
-```
-### tsconfig.json
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
----
-## 6. Deployment Instructions
-### Deploy to Versori
-```bash
-# 1. Install dependencies
-npm install
-# 2. Test locally (if using Versori CLI)
-npm run dev
-# 3. Deploy to Versori platform
-npm run deploy
-```
-### Configure Activation Variables
-In Versori platform settings, configure:
-```json
-{
- "sftpHost": "sftp.partner.com",
- "sftpPort": 22,
- "sftpPrivateKey": "",
- "sftpIncomingPath": "/products/incoming",
- "sftpProcessedPath": "/products/processed",
- "sftpErrorPath": "/products/errors",
- "filePattern": "products_*.csv",
- "catalogueRef": "PC:MASTER:2",
- "catalogueType": "MASTER",
- "eventName": "UPSERT_PRODUCT",
- "eventMode": "async",
- "maxFilesPerRun": 10,
- "requireAbsolutePaths": "true"
-}
-```
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection.
----
-## 7. Performance Notes
-- **Event sending:** Sequential processing (<100 records) is simple and reliable; consider parallel batches (Promise.allSettled) for 100-1000 records
-- **Batch API threshold:** For high volumes (>1000 records), use Batch API instead of Event API for better throughput
-- **File concurrency:** Files process sequentially by default; for very high volumes, consider processing 3-5 files in parallel if SFTP limits allow
-- **Rate limiting:** If 429 throttling is observed, add small backoff/delay or switch to Batch API
----
-## 8. Testing
-### Test Scheduled Ingestion
-Upload a test CSV file to SFTP incoming directory and wait for the scheduled run.
-**Check logs:**
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and SFTP
-[STEP 3/8] Discovering files on SFTP
-[FILE 1/1] Processing file: products_20250124.csv
-[STEP 4/8] Downloading and parsing: products_20250124.csv
-[STEP 5/8] Transforming 5 products from products_20250124.csv
-[STEP 6/8] Sending 5 events to Fluent Commerce
-[STEP 7/8] Archiving file: products_20250124.csv
-[STEP 8/8] Completing job and calculating totals
-```
-### Test Ad hoc Ingestion
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "Content-Type: application/json" \
- -d '{}'
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "Content-Type: application/json" \
- -d '{
-  "filePattern": "urgent_*.csv", }'
-```
-### Test Job Status Query
-```bash
-curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
- -H "Content-Type: application/json" \
- -d '{
-  "jobId": "ADHOC_PROD_20251024_183045_abc123"
- }'
-```
----
-## Monitoring
-### Success Response
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 50,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 50,
-   "eventsFailed": 0
-  }
- ],
- "duration": 12345
-}
-```
-### Partial Success Response
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 45,
- "eventsFailed": 5,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 45,
-   "eventsFailed": 5,
-   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
-  }
- ],
- "duration": 12345
-}
-```
-### Error Response
-```json
-{
- "success": false,
- "filesProcessed": 0,
- "filesFailed": 1,
- "totalRecords": 0,
- "eventsSent": 0,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": false,
-   "error": "CSV parse error: Invalid structure"
-  }
- ],
- "duration": 876
-}
-```
-### Monitoring Metrics
-Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
-- **Files Processed** - Total files successfully processed
-- **Events Sent** - Total events sent to Fluent Commerce
-- **Events Failed** - Events that failed (check rejection reports)
-- **Processing Duration** - Time taken for complete workflow
-- **Rate Limiting** - Watch for 429 errors indicating throttling
-Use the status webhook for dashboards and automated monitoring.
----
-**To replicate this template for other entities/formats:**
-1. **File Naming:** Replace `product`, `PRD` with your entity name across all files
-2. **CSV Structure:** Update expected CSV format and parsing logic
-3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
-4. **Event Name:** Update event name (e.g., `UPSERT_INVENTORY`, `CREATE_LOCATION`)
-5. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledInventoryIngestion`)
-6. **Service Function:** Rename main function (e.g., `executeInventoryIngestion`)
-7. **File Pattern:** Update SFTP file pattern (e.g., `inventory_*.csv`)
-8. **Data Source:** For S3 use `S3DataSource`, for XML use `XMLParserService`
----
-## 10. Template summary
-This template provides:
-- SFTP ? Fluent Commerce ingestion workflow
-- File discovery and tracking (VersoriFileTracker)
-- CSV parsing and transformation (CSVParserService + UniversalMapper)
-- Event API integration with per-record error handling
-- File archival (success/error paths)
-- Job tracking (scheduled, ad hoc, status query modes)
-- SFTP connection cleanup and retry logic
----
-## 🔧 Troubleshooting (quick)
-- No files found: Check `sftpIncomingPath` and `filePattern` (names only).
-- CSV parse failed: Move to errors/; validate CSV structure and encoding.
-- High event failures: Inspect rejection report; consider Batch API for very high volumes.
-- 429 throttling: Add small backoff/delay or use Batch API.
-## 🗺️ Mapping tips (compact)
-- Required fields: mark in mapping JSON; failed items won't be sent.
-- Arrays: use custom resolver `custom.splitByPipe` for pipe-separated values.
-- Nested objects: use custom resolvers `custom.buildPriceArray` and `custom.buildTaxType`.
-- Resolvers: prefer SDK built-ins (trim, uppercase, parseFloat).
----
+---
+template_id: tpl-ingest-sftp-csv-to-product-event
+canonical_filename: template-ingestion-sftp-csv-product-event.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: ingestion
+source: sftp-csv
+destination: fluent-event-api
+entity: product
+format: csv
+logging: versori
+status: stable
+features:
+  - batched-events
+  - attribute-transformation
+  - memory-management
+  - json-serialization-handling
+  - enhanced-logging
+---
+# Template: Ingestion - SFTP CSV to Product Event
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
+- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
+- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large CSV files)
+- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
+- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
+---
+## 📚 STEP 1: Load These Docs Into Your AI (Human Checklist)
+1. REQUIRED (load all)
+  - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+  - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+  - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
+  - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+---
+## 📋 Implementation Prompt
+```
+I need a Versori scheduled ingestion that:
+1) Discovers CSV files on SFTP with file tracking to skip duplicates
+2) Downloads and parses CSV (returns array directly - no normalization needed)
+3) Transforms records with UniversalMapper per mapping JSON
+4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
+5) Archives files to processed/ or errors/ and writes optional rejection report
+6) Tracks progress with JobTracker and exposes a job-status webhook
+7) Uses native Versori log from context
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+---
+## 📋 Template Overview
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP CSV, transforms it, and sends events to the Fluent Commerce Event API.
+### What This Template Does
+```
+�������������������������������������������������������������������
+�          INGESTION WORKFLOW              �
+�������������������������������������������������������������������
+1. TRIGGER
+  �� Scheduled (Cron): Runs automatically every hour
+  �� Ad hoc (Webhook): Manual trigger for immediate processing
+  �� Status Query (Webhook): Check job progress
+2. DISCOVER FILES (SftpDataSource)
+  �� List files from SFTP directory
+  �� Filter by pattern (products_*.csv)
+  �� Check file tracking (skip processed)
+  "� Sort by oldest first
+3. FOR EACH FILE (Per-File Processing)
+  �� Process one file completely before moving to next
+  "� Steps 4-6 happen for each file
+4. PROCESS FILE (Service Function)
+  �� Download CSV from SFTP
+  �� Parse CSV to array (CSVParserService)
+  �� Transform with UniversalMapper
+  "� Return transformed products
+5. SEND EVENTS (Service Function)
+  �� Loop through transformed products
+  �� Send UPSERT_PRODUCT event (async)
+  �� Track success/failure count
+  "� Continue on individual failures
+6. ARCHIVE & TRACK (SftpDataSource + KV)
+  �� Move file to processed/ or errors/
+  �� Generate timestamped archive name
+  �� Mark as processed in KV
+  �� Optional: Write event log
+  "� Store file result
+7. REPEAT for next file (loop back to step 3)
+8. COMPLETE JOB (JobTracker)
+  �� Calculate totals (all files)
+  �� Store final result in KV
+  �� Enable status queries via webhook
+  "� Return complete results
+```
+### Key Features
+- **Service Functions**: Modular design with 3 service functions:
+ 1. `processFile()` - Downloads, parses, transforms (SftpDataSource + CSVParserService + UniversalMapper)
+ 2. `sendEvents()` - Loops through products, sends events with per-record error handling
+ 3. `writeEventLog()` - Writes event logs to SFTP using Buffer (for audit/debugging)
+- **Per-File Processing**: Each file is processed completely (download ? parse ? map ? send ? archive) before moving to next
+- **Job Tracking**: JobTracker persists stage/status to Versori KV for visibility and status queries
+- **Execution Modes**: Scheduled (cron), ad hoc (webhook), status query (webhook)
+- **File Tracking**: KV-based file tracking prevents duplicates; `forceReprocess` bypasses
+- **Event API**: Per-record failures don't block other records; optional rejection reports
+- **SFTP Cleanup**: Connection disposal in finally block ensures cleanup
+Note: JobTracker recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+### 📦 Package Information
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+**Installation:**
+```bash
+npm install @fluentcommerce/fc-connect-sdk@latest
+```
+**Version:** Use latest SDK version from npm
+---
+**Templates are designed for direct deployment; customize via activation variables.**
+---
+## 📦 SDK Imports (Verified - Versori Optimized)
+```typescript
+// ? VERIFIED IMPORTS - These match actual SDK exports
+import { Buffer } from 'node:buffer';
+import {
+ createClient, // Universal client factory
+ SftpDataSource, // SFTP operations with connection pooling
+ CSVParserService, // CSV parsing
+ UniversalMapper, // Field mapping with SDK resolvers
+ VersoriFileTracker, // File processing state tracker
+ JobTracker, // Job status tracking
+} from "@fluentcommerce/fc-connect-sdk";
+import type {
+ FluentClient,
+ SftpDataSourceConfig,
+} from "@fluentcommerce/fc-connect-sdk";
+// Versori platform imports
+import { schedule, webhook, http, fn, credentials } from "@versori/run";
+```
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+**? VERSORI PLATFORM - Use Native Logs:**
+- Use `log` from context: `const { log } = ctx;`
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+---
+## SFTP Connection Setup
+**Two methods available for accessing SFTP credentials:**
+### Method 1: activation.connections (Recommended)
+**? BEST PRACTICE - Most Secure & Flexible**
+Store SFTP credentials in a Versori connection and access via `activation.connections`:
+**Connection Configuration:**
+1. In Versori platform, create a connection named `SFTP`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+**Access in code:**
+```typescript
+const { activation } = ctx;
+const conn = activation.connections.SFTP;
+const rawAccessToken = conn.accessToken;
+const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+```
+**Benefits:**
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection reusable across workflows
+- ✅ Access all connections dynamically
+- ✅ No activation variables needed for secrets
+- ✅ Easier credential rotation
+- ✅ Built-in connection health monitoring
+**Security:** Connections are encrypted at rest and in transit by Versori platform.
+### Method 2: credentials().get() (Alternative)
+**Alternative approach using credentials API:**
+```typescript
+const { credentials } = ctx;
+const conn = credentials().get('SFTP');
+const rawAccessToken = conn.accessToken;
+const rawBasicAuth = Buffer.from(rawAccessToken, 'base64').toString('utf-8');
+const [username, password] = rawBasicAuth.split(':');
+```
+**When to use:**
+- Single-connection workflows
+- Workflows using `credentials()` API
+- Simpler syntax for single credential access
+**Comparison:**
+| Feature | `activation.connections` | `credentials().get()` |
+|---------|-------------------------|----------------------|
+| **Security** | Same (Versori vault) | Same (Versori vault) |
+| **Access Pattern** | Object property | Method call |
+| **Multiple Connections** | Enumerate all via `Object.keys()` | Need connection names |
+| **Dynamic Discovery** | Yes | Limited |
+| **Future-proof** | Yes (recommended API) | Yes (maintained) |
+**📚 Complete guide:** See [SFTP Credential Access Security Guide](../../../../../02-CORE-GUIDES/data-sources/data-sources-sftp-credential-access-security.md) for:
+- Detailed security comparison
+- Error handling patterns
+- Connection validation
+- Multi-connection scenarios
+- Best practices
+**Template uses:** `activation.connections` (Method 1) throughout - most flexible for production use.
+## ⚙️ Activation Variables
+**Configuration is driven by activation variables - modify these instead of code:**
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
+```bash
+# SFTP Configuration
+SFTP_HOST=sftp.partner.com
+SFTP_PORT=22
+# SFTP Paths
+SFTP_REMOTE_PATH=/incoming/products/
+SFTP_ARCHIVE_PATH=/archive/products/
+SFTP_ERROR_PATH=/errors/products/
+# File Processing
+FILE_PATTERN=.csv
+MAX_FILES_PER_RUN=10
+# Product Configuration
+CATALOGUE_REF=PC:MASTER:2
+CATALOGUE_TYPE=MASTER
+# Event API Configuration
+EVENT_NAME=UPSERT_PRODUCT
+EVENT_MODE=async
+# Fluent Configuration (via Versori connection)
+# Connection: fluent_commerce (OAuth2)
+FLUENT_RETAILER_ID=1
+# Feature Toggles
+ENABLE_FILE_TRACKING=true                      # Enable/disable file tracking (default: true)
+# Event API Performance
+USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
+MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
+MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
+```
+**Security:**
+- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup) section below.
+- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
+### Variable Explanations
+| Variable      | Purpose           | Default        | Customization Hints       |
+| ------------------- | --------------------------- | --------------------- | -------------------------------- |
+| `FLUENT_RETAILER_ID` | Retailer ID for Event API  | -           | Required - Fluent retailer ID  |
+| **SFTP Credentials**  | *From Connection*      |            | *See connection setup above*                   |
+| `SFTP_HOST`     | SFTP server hostname    | -           | Required - your SFTP server   |
+| `SFTP_PORT`     | SFTP server port      | `22`         | Usually 22, sometimes 2222    |
+| `SFTP_REMOTE_PATH` | Incoming files directory  | `/incoming/products/` | Where new files arrive      |
+| `SFTP_ARCHIVE_PATH` | Processed files archive   | `/archive/products/` | Where files move after success  |
+| `SFTP_ERROR_PATH`   | Failed files directory   | `/errors/products/`  | Where files move after errors  |
+| `FILE_PATTERN`    | File name filter      | `.csv`   | Extension or glob pattern for matching files |
+| `CATALOGUE_REF`   | Product catalogue reference | `PC:MASTER:2`     | Target catalogue in Fluent    |
+| `CATALOGUE_TYPE`   | Catalogue type       | `MASTER`       | Usually MASTER or STANDARD    |
+| `EVENT_NAME`     | Event to send        | `UPSERT_PRODUCT`   | Event name from Rubix      |
+| `EVENT_MODE`     | Event processing mode    | `async`        | async (recommended) or sync   |
+| `MAX_FILES_PER_RUN`  | Max files per execution   | `10`         | Prevent timeout on large batches (0 = unlimited) |
+| `ENABLE_FILE_TRACKING` | Enable file deduplication | `"true"`      | `"true"` = skip already processed files, `"false"` = process all files |
+| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false` | `true` = batched events (10x faster for 100+ products), `false` = individual events |
+| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100` | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
+| `MEMORY_BATCH_SIZE` | Products per mapping batch | `250` | Processes products in batches during mapping (prevents OOM on large files) |
+---
+## 🔒 SDK Automatic Behaviors (v0.1.40+)
+**The SDK automatically validates and retries for improved reliability:**
+### retailerId Validation
+- **SDK validates** `retailerId` before calling `sendEvent()`
+- **Checks:** `event.retailerId || client.retailerId`
+- **If missing:** Throws `"retailerId is required for Event API..."`
+- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
+### 401 Auth Retry
+- **Automatic retry** for platform auth failures (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s)
+- **Applies to:** All `sendEvent()` calls (async and sync modes)
+- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
+### 5xx Server Retry
+- **Automatic retry** for transient server errors (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
+- **Protects:** Against Fluent API transient failures
+### No Code Changes Required
+- All templates remain compatible
+- Retry logic is automatic and transparent
+- Better error messages guide configuration
+**See:** [Event API Guide](./event-api-guide.md) for complete details
+---
+### 📁 File Tracking & Deduplication
+**How file tracking prevents duplicates:**
+1. **Discovery:** List all files matching `filePattern` from `sftpIncomingPath`
+2. **Filter:** Query VersoriFileTracker to check which files were already processed
+3. **Process:** Only process untracked files (skips duplicates automatically)
+4. **Mark:** Mark file as processed after successful Event API ingestion
+5. **Archive:** Move file to `sftpProcessedPath` (or `sftpErrorPath` on failure)
+**File lifecycle:**
+```
+/incoming/products_20250124_001.csv
+ ?� (discovered)
+VersoriFileTracker.wasFileProcessed() ? false
+ ?� (process file)
+Parse CSV ? Map to events ? Send to Event API
+ ?� (success)
+VersoriFileTracker.markFileProcessed()
+ ?� (archive)
+/processed/products_20250124_001-20250124T183045.csv
+```
+**Force reprocess:**
+Set `forceReprocess: true` in webhook payload to bypass file tracker and reprocess files. Useful for:
+- Fixing bad data that was previously ingested
+- Reingesting after mapping config changes
+- Testing with production files
+**Note:** Force reprocess doesn't prevent marking as processed - files are still tracked after reprocessing.
+---
+### 📁 SFTP Path vs File Pattern
+**Critical:** Folder paths and file patterns are configured separately.
+- **Folder Paths:** Configured via activation variables:
+ - `sftpIncomingPath` - Where new files arrive (e.g., `/products/incoming`)
+ - `sftpProcessedPath` - Where successful files are archived (e.g., `/products/processed`)
+ - `sftpErrorPath` - Where failed files are moved (e.g., `/products/errors`)
+- **File Pattern:** Configured via `filePattern` activation variable
+ - Matches filenames ONLY (not folder paths)
+ - Supports wildcards: `*.csv`, `products_*.csv`, `urgent_*.csv`
+**How it works:**
+```typescript
+// The workflow lists files like this:
+sftp.listFiles({
+ remotePath: "/products/incoming", // ?� Folder from activation variable
+ filePattern: "products_*.csv", // ?� Pattern matches filenames only
+});
+```
+**Example:** With these settings:
+```json
+{
+ "sftpIncomingPath": "/products/incoming",
+ "filePattern": "products_*.csv"
+}
+```
+The workflow will find files like:
+- ✅ `/products/incoming/products_20250124.csv`
+- ✅ `/products/incoming/products_urgent.csv`
+- ❌ `/products/incoming/orders_20250124.csv` (doesn't match pattern)
+- ❌ `/orders/incoming/products_20250124.csv` (wrong folder)
+**Note:** To process files from different folders, change `sftpIncomingPath` (not `filePattern`).
+---
+## 📄 Mapping Configuration
+**File:** `config/products.import.csv.json`
+```json
+{
+ "name": "products.import.csv",
+ "version": "1.0.0",
+ "description": "CSV ? Product Event Mapping",
+ "fields": {
+  "ref": {
+   "source": "ref",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "type": {
+   "source": "type",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "STANDARD"
+  },
+  "status": {
+   "source": "status",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "ACTIVE"
+  },
+  "gtin": {
+   "source": "gtin",
+   "resolver": "sdk.trim"
+  },
+  "name": {
+   "source": "name",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "summary": {
+   "source": "summary"
+  },
+  "categoryRefs": {
+   "source": "category_refs",
+   "resolver": "custom.splitByPipe"
+  },
+  "price": {
+   "resolver": "custom.buildPriceArray"
+  },
+  "taxType": {
+   "resolver": "custom.buildTaxType"
+  },
+  "attributes": {
+   "defaultValue": []
+  }
+ }
+}
+```
+**Custom Resolvers:**
+```typescript
+const customResolvers = {
+ /**
+  * Split pipe-separated string into array
+  * Example: "CAT1|CAT2|CAT3" ? ["CAT1", "CAT2", "CAT3"]
+  */
+ 'custom.splitByPipe': (value: string) => {
+  if (!value || value.trim() === '') return [];
+  return value.split('|').map(v => v.trim()).filter(v => v !== '');
+ },
+ /**
+  * Build price array from flat CSV columns
+  * Expects: price_default_currency, price_default_value, price_special_currency, price_special_value
+  */
+ 'custom.buildPriceArray': (value: any, sourceData: any) => {
+  const prices = [];
+  // Add default price
+  if (sourceData.price_default_value) {
+   prices.push({
+    type: 'DEFAULT',
+    currency: sourceData.price_default_currency || 'USD',
+    value: sourceData.price_default_value,
+   });
+  }
+  // Add special price if exists
+  if (sourceData.price_special_value) {
+   prices.push({
+    type: 'SPECIAL',
+    currency: sourceData.price_special_currency || 'USD',
+    value: sourceData.price_special_value,
+   });
+  }
+  return prices;
+ },
+ /**
+  * Build tax type object from flat CSV columns
+  * Expects: tax_country, tax_group, tax_tariff
+  */
+ 'custom.buildTaxType': (value: any, sourceData: any) => {
+  if (!sourceData.tax_country) return undefined;
+  return {
+   country: sourceData.tax_country,
+   group: sourceData.tax_group,
+   tariff: sourceData.tax_tariff,
+  };
+ },
+};
+```
+Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
+### Mapping customization and custom resolvers (how-to)
+(Condensed) Provide custom resolvers via UniversalMapper options if required.
+### Parse + map
+Handled inside orchestration (download ? parse ? map). No extra helpers needed.
+---
+### Error handling & rejection reports
+When moving failed files to errors/, optionally write a JSON rejection report with totals and error list for audit purposes.
+---
+## Expected CSV Format
+```csv
+ref,type,status,gtin,name,summary,category_refs,price_default_currency,price_default_value,price_special_currency,price_special_value,tax_country,tax_group,tax_tariff,catalogue_ref,catalogue_type
+G_PROD_WITH_NO_STANDARD,VARIANT,ACTIVE,MH01-XS-Orange,Chaz Kangeroo Hoodie-XS-Orange main,<p>test short description</p>,STANDARD_CATEGORY,USD,52.000000,USD,9.000000,AU,Tax Group,Tax Tariff,PC:MASTER:2,MASTER
+G_PROD_002,VARIANT,ACTIVE,MH02-M-Blue,Kangeroo Hoodie-M-Blue,<p>Blue variant</p>,STANDARD_CATEGORY|SEASONAL,USD,45.000000,USD,35.000000,AU,Tax Group,Tax Tariff,PC:MASTER:2,MASTER
+```
+---
+## Event Sending Configuration
+**Simple Configuration:** One variable controls everything
+```json
+{
+ "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
+}
+```
+**How it works:**
+- `eventConcurrency: 1` ? Sequential (sends events one at a time, safe default)
+- `eventConcurrency: 3` ? Parallel (sends 3 events concurrently)
+- `eventConcurrency: 5` ? Parallel (sends 5 events concurrently)
+- `eventConcurrency: 10` ? Parallel (sends 10 events concurrently)
+**Configuration Guidelines:**
+- **Conservative:** `1` ? Sequential (safe, predictable, ~1 req/sec)
+- **Balanced:** `3-5` ? Parallel (most common, ~3-5 req/sec)
+- **Aggressive:** `10` ? Parallel (high-volume, ~10 req/sec)
+- **Note:** Fluent API supports concurrent requests - adjust based on your needs
+**Why Single Variable?**
+- **Simpler:** One variable instead of two (`mode` + `concurrency`)
+- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
+- **Less config:** Fewer activation variables to manage
+- **Flexible:** Easy to tune performance (just change the number)
+---
+## 🔧 Production Implementation
+### Processing Modes
+| Mode    | Default | What happens                                           | When to use                       |
+| ---------- | ------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
+| `per-file` | ?   | Process one file end-to-end (discover ? parse ? Event API ? archive) before moving on       | Recommended for production SFTP feeds and very large files |
+| `chunked` | ?   | Processes `fileChunkSize` files at a time, completing each chunk before the next         | High-volume feeds where per-file would take too long   |
+| `batch`  | ?   | Loads every matched file into memory, produces a single Event API job, and archives when complete | Tiny datasets or smoke tests only            |
+Set the mode via activation variables:
+```json
+{
+ "processingMode": "per-file",
+ "fileChunkSize": "5",
+ "maxFilesPerRun": "25"
+}
+```
+---
+## 🔧 Complete Production Code
+### Versori Workflows Structure
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+### Recommended Project Structure
+```
+product-event-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── product.types.ts            # Shared type definitions
+```
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+---
+## Workflow Files
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+All time-based triggers that run automatically on cron schedules.
+#### `src/workflows/scheduled/daily-product-sync.ts`
+**Purpose**: Automatic Daily product sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+/**
+ * Scheduled Workflow: Daily Product Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: product-sync.service.ts
+ */
+export const dailyProductSync = schedule(
+  'product-sync-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-product-sync', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log, openKv } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `product-sync-${executionStartTime}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+    log.info('🚀 [WORKFLOW] Starting scheduled product sync', { jobId });
+    await tracker.createJob(jobId, {
+      triggeredBy: 'schedule',
+      stage: 'initialization',
+      startTime: executionStartTime
+    });
+    await tracker.updateJob(jobId, { status: 'processing' });
+    try {
+      // Reuse shared orchestration logic
+      const result = await executeProductIngestion(ctx, jobId, tracker);
+      if (result.success) {
+        await tracker.markCompleted(jobId, result);
+        log.info('✅ [WORKFLOW] Product sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          duration: result.duration
+        });
+      } else {
+        await tracker.markFailed(jobId, result.error || 'Unknown error');
+        log.error('❌ [WORKFLOW] Product sync failed', { jobId, error: result.error });
+      }
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      const errorMessage = e instanceof Error ? e.message : String(e);
+      await tracker.markFailed(jobId, errorMessage);
+      log.error('❌ [WORKFLOW] Product sync failed with exception', {
+        jobId,
+        error: errorMessage,
+        stack: e instanceof Error ? e.stack : undefined
+      });
+      return { success: false, jobId, error: errorMessage };
+    }
+  })
+);
+```
+---
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+All HTTP-based triggers that create webhook endpoints.
+#### `src/workflows/webhook/adhoc-product-sync.ts`
+**Purpose**: Manual product sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+/**
+ * Webhook: Manual Product Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
+ * Request body (optional): { filePattern: "urgent_*.csv", maxFiles: 5 }
+ *
+ * Pattern: webhook().then(http()) - needs Fluent API access
+ * Uses shared service: product-sync.service.ts
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const adhocProductSync = webhook('product-sync-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'product-sync-adhoc', // Versori validates API key
+}).then(
+  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log, openKv, data } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `product-sync-adhoc-${executionStartTime}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const filePattern = data?.filePattern as string;
+    const maxFiles = data?.maxFiles as number;
+    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+    });
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'webhook',
+      stage: 'initialization',
+      status: 'queued',
+      startTime: executionStartTime,
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductIngestion(ctx, jobId, tracker)
+      .then((result) => {
+        if (result.success) {
+          log.info('✅ [BACKGROUND] Product sync completed successfully', {
+            jobId,
+            filesProcessed: result.filesProcessed,
+            filesFailed: result.filesFailed,
+            recordsProcessed: result.recordsProcessed,
+            duration: result.duration,
+          });
+          return tracker.markCompleted(jobId, result);
+        } else {
+          log.error('❌ [BACKGROUND] Product sync failed', {
+            jobId,
+            error: result.error,
+          });
+          return tracker.markFailed(jobId, result.error || 'Unknown error');
+        }
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+        log.error('❌ [BACKGROUND] Product sync failed with exception', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+        });
+        return tracker.markFailed(jobId, errorMessage);
+      });
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Product sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+#### `src/workflows/webhook/job-status-check.ts`
+**Purpose**: Query job status
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
+**Request body**: `{ "jobId": "product-sync-1234567890" }`
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
+ * Request body: { "jobId": "product-sync-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const productSyncJobStatus = webhook('product-sync-job-status', {
+  response: { mode: 'sync' },
+  connection: 'product-sync-job-status',
+}).then(
+  fn('status', async ctx => {
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+    if (!jobId) {
+      return { success: false, error: 'jobId required' };
+    }
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+---
+### 3. Entry Point (`index.ts`)
+**Purpose**: Register all workflows with Versori platform
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * Versori automatically discovers and registers exported workflows
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ */
+// Import scheduled workflows
+import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
+// Import webhook workflows
+import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
+import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
+// Register all workflows
+export {
+  // Scheduled (time-based triggers)
+  dailyProductSync,
+  // Webhooks (HTTP-based triggers)
+  adhocProductSync,
+  productSyncJobStatus,
+};
+```
+**What Gets Exposed:**
+- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
+- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
+- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
+---
+### Adding New Workflows
+**To add a scheduled workflow:**
+1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+**To add a webhook workflow:**
+1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+**Example - Adding hourly delta sync:**
+```typescript
+// src/workflows/scheduled/hourly-delta-sync.ts
+export const hourlyDeltaSync = schedule(
+  'product-delta-hourly',
+  '0 * * * *' // Every hour
+).then(
+  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
+    // Delta sync logic (skip BPP)
+    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
+    return result;
+  })
+);
+// index.ts (add to imports and exports)
+import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
+export { dailyProductSync, hourlyDeltaSync, ... };
+```
+---
+## 3. Type Definitions (src/types/product-ingestion.types.ts)
+```typescript
+/**
+ * Type Definitions for Product Ingestion
+ *
+ * Centralized type definitions for product ingestion workflow
+ */
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Product interface - represents transformed product data
+ */
+export interface Product {
+ ref: string;
+ type?: string;
+ status?: string;
+ name: string;
+ summary?: string;
+ gtin?: string;
+ catalogue?: { ref?: string };
+ attributes?: Record<string, unknown>;
+}
+/**
+ * Event configuration
+ */
+export interface EventConfig {
+ eventName: string;
+ catalogueRef: string;
+ catalogueType: string;
+ eventMode: 'async' | 'sync';
+}
+/**
+ * Event result - tracks success/failure counts
+ */
+export interface EventResult {
+ eventsSent: number;
+ eventsFailed: number;
+ errors: Array<{ productRef: string; error: string }>;
+}
+/**
+ * Process file result
+ */
+export interface ProcessFileResult {
+ success: boolean;
+ products: Product[];
+ error?: string;
+}
+/**
+ * Versori Context Interface
+ * Represents the Versori runtime context passed to workflow functions
+ */
+export interface VersoriContext {
+ log: {
+  info: (message: string, data?: Record<string, unknown>) => void;
+  warn: (message: string, data?: Record<string, unknown>) => void;
+  error: (message: string, data?: Record<string, unknown>) => void;
+  debug?: (message: string, data?: Record<string, unknown>) => void;
+ };
+ openKv: (namespace: string) => {
+  get: (key: string) => Promise<unknown>;
+  set: (key: string, value: unknown) => Promise<void>;
+  delete: (key: string) => Promise<void>;
+ };
+ activation: {
+  getVariable: (name: string) => string | undefined;
+  connections?: Record<string, unknown>;
+ };
+ connections?: Record<string, unknown>;
+ data?: unknown;
+ fetch?: typeof fetch;
+}
+/**
+ * Parameters for ingestion workflow
+ */
+export interface ProductIngestionParams {
+ jobId: string;
+ triggeredBy: 'schedule' | 'webhook';
+ filePattern?: string;
+ maxFiles?: number;
+ forceReprocess?: boolean;
+ catalogueRef?: string;
+ priority?: 'normal' | 'high';
+}
+/**
+ * Result from ingestion workflow
+ */
+export interface ProductIngestionResult {
+ success: boolean;
+ jobId: string;
+ filesProcessed: number;
+ filesFailed: number;
+ filesSkipped?: number;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ fileResults: FileProcessingResult[];
+ error?: string;
+}
+/**
+ * Per-file processing result
+ */
+export interface FileProcessingResult {
+ fileName: string;
+ success: boolean;
+ skipped?: boolean;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ duration: number;
+ error?: string;
+}
+```
+---
+## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
+```typescript
+/**
+ * Product File Processor Service
+ *
+ * Downloads CSV files from SFTP, parses, and transforms with UniversalMapper.
+ * CSV-specific: Returns arrays directly (no normalization needed).
+ */
+import {
+ SftpDataSource,
+ CSVParserService,
+ UniversalMapper,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
+/**
+ * Service for processing CSV product files
+ */
+export class ProductFileProcessorService {
+ constructor(
+  private sftp: SftpDataSource,
+  private csvParser: CSVParserService,
+  private mapper: UniversalMapper,
+  private catalogueRef: string
+ ) {}
+ /**
+  * Download CSV file from SFTP, parse, and transform with UniversalMapper
+  *
+  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
+  * - Explicit null assignments after each step
+  * - Finally block guarantees cleanup
+  * - Prevents OOM errors on large CSV files
+  */
+ async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
+  // ✅ CRITICAL: Variables for cleanup tracking
+  let csvContent: string | null = null;
+  let parsed: unknown | null = null;
+  try {
+   // STEP 1: Download CSV content
+   csvContent = (await this.sftp.downloadFile(remoteFilePath, {
+    encoding: 'utf8',
+   })) as string;
+   // STEP 2: Parse CSV with type safety
+   try {
+    parsed = await this.csvParser.parse(csvContent);
+   } catch (parseError: unknown) {
+    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
+    return {
+     success: false,
+     products: [],
+     error: `CSV parse error: ${errorMessage}`,
+    };
+   }
+   // ✅ Clear csvContent from memory - no longer needed after parsing
+   csvContent = null;
+   // STEP 3: CSV-SPECIFIC - Returns array directly, no normalization needed
+   const rawProducts = Array.isArray(parsed) ? parsed : [];
+   if (rawProducts.length === 0) {
+    return {
+     success: false,
+     products: [],
+     error: 'No products found in CSV',
+    };
+   }
+   // ✅ Clear parsed data from memory - only need rawProducts now
+   parsed = null;
+   // STEP 4: Transform with UniversalMapper
+   // CSV-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
+   const sourceDataWithContext = rawProducts.map(item => ({
+    ...item,
+    $context: { catalogueRef: this.catalogueRef },
+   }));
+   const mappingResult = await this.mapper.map(sourceDataWithContext);
+   if (!mappingResult.success) {
+    return {
+     success: false,
+     products: [],
+     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
+    };
+   }
+   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+     skippedFields: mappingResult.skippedFields,
+     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+    });
+   }
+   return {
+    success: true,
+    products: mappingResult.data as Product[],
+   };
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   return {
+    success: false,
+    products: [],
+    error: errorMessage,
+   };
+  } finally {
+   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
+   csvContent = null;
+   parsed = null;
+  }
+ }
+}
+```
+---
+## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
+```typescript
+/**
+ * Event Sender Service
+ *
+ * Sends product events to Fluent Commerce Event API with per-record error handling.
+ * Continues processing on individual failures (Event API best practice).
+ * Supports configurable concurrency (sequential or parallel).
+ */
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
+/**
+ * Service for sending events to Fluent Commerce Event API
+ *
+ * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
+ */
+export class EventSenderService {
+ constructor(
+  private client: FluentClient,
+  private log?: any  // ✅ Optional logger for progress tracking
+ ) {}
+ /**
+  * Send events to Fluent Commerce Event API with configurable concurrency
+  *
+  * **Performance Characteristics:**
+  * - `concurrency: 1` ? Sequential processing (safe default, ~1 event/sec)
+  * - `concurrency: 3-5` ? Balanced throughput (~3-5 events/sec, good for most cases)
+  * - `concurrency: 10` ? High-volume processing (~10 events/sec, 100+ products)
+  *
+  * **Implementation Strategy:**
+  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
+  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
+  * - Both modes: Per-record error tracking (failures don't block other events)
+  *
+  * @param products - Array of products to send as events
+  * @param eventConfig - Event configuration (name, catalogue ref, mode)
+  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
+  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
+  */
+ async sendEvents(
+  products: Product[],
+  eventConfig: EventConfig,
+  concurrency: number = 1
+ ): Promise<EventResult> {
+  // Validate concurrency (guard against invalid values)
+  const safeConc = Math.max(1, Math.floor(concurrency));
+  // Result accumulators
+  let eventsSent = 0;
+  let eventsFailed = 0;
+  const errors: Array<{ productRef: string; error: string }> = [];
+  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
+  if (this.log) {
+   this.log.info('📤 Starting event sending', {
+    totalProducts: products.length,
+    concurrency: safeConc,
+    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
+    eventName: eventConfig.eventName,
+    catalogueRef: eventConfig.catalogueRef
+   });
+  }
+  // Helper: Build event payload (DRY - reused in both modes)
+  const buildPayload = (product: Product) => ({
+   name: eventConfig.eventName,
+   entityRef: eventConfig.catalogueRef,
+   entityType: 'PRODUCT_CATALOGUE' as const,
+   entitySubtype: eventConfig.catalogueType,
+   rootEntityRef: eventConfig.catalogueRef,
+   rootEntityType: 'PRODUCT_CATALOGUE' as const,
+   attributes: product,
+  });
+  // ============================================================================
+  // SEQUENTIAL MODE (concurrency === 1)
+  // ============================================================================
+  if (safeConc === 1) {
+   for (let i = 0; i < products.length; i++) {
+    const product = products[i];
+    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
+    if (this.log && i % 10 === 0) {
+     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
+      productRef: product.ref,
+      progress: `${((i / products.length) * 100).toFixed(1)}%`
+     });
+    }
+    try {
+     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
+     eventsSent++;
+    } catch (err: unknown) {
+     eventsFailed++;
+     const errorMsg = err instanceof Error ? err.message : String(err);
+     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
+     // Continue processing (failure doesn't block other products)
+    }
+   }
+   // ✅ PRODUCTION ENHANCEMENT: Log completion
+   if (this.log) {
+    this.log.info('✅ Sequential event sending completed', {
+     totalProducts: products.length,
+     eventsSent,
+     eventsFailed
+    });
+   }
+   return { eventsSent, eventsFailed, errors };
+  }
+  // ============================================================================
+  // PARALLEL MODE (concurrency > 1)
+  // ============================================================================
+  const totalChunks = Math.ceil(products.length / safeConc);
+  for (let i = 0; i < products.length; i += safeConc) {
+   const chunk = products.slice(i, i + safeConc);
+   const chunkNumber = Math.floor(i / safeConc) + 1;
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
+   if (this.log) {
+    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
+     chunkNumber,
+     totalChunks,
+     productsInChunk: chunk.length,
+     productRange: `${i + 1}-${i + chunk.length}`,
+     totalProducts: products.length,
+     progress: `${((i / products.length) * 100).toFixed(1)}%`
+    });
+   }
+   // Fire all requests in chunk concurrently
+   const results = await Promise.allSettled(
+    chunk.map(product =>
+     this.client
+      .sendEvent(buildPayload(product), eventConfig.eventMode)
+      .then(() => ({ success: true as const, product }))
+      .catch(error => ({ success: false as const, product, error }))
+    )
+   );
+   // Aggregate chunk results into totals
+   let chunkSuccess = 0;
+   let chunkFailed = 0;
+   for (const result of results) {
+    if (result.status === 'fulfilled' && result.value.success) {
+     eventsSent++;
+     chunkSuccess++;
+    } else {
+     eventsFailed++;
+     chunkFailed++;
+     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
+     const product = result.status === 'fulfilled' ? result.value.product : null;
+     errors.push({
+      productRef: product?.ref || 'unknown',
+      error: error?.message || String(error) || 'unknown error',
+     });
+    }
+   }
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
+   if (this.log) {
+    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
+     chunkNumber,
+     totalChunks,
+     chunkSuccess,
+     chunkFailed,
+     totalSentSoFar: eventsSent,
+     totalFailedSoFar: eventsFailed,
+     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
+    });
+   }
+  }
+  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
+  if (this.log) {
+   this.log.info('✅ Parallel event sending completed', {
+    totalProducts: products.length,
+    totalChunks,
+    concurrency: safeConc,
+    eventsSent,
+    eventsFailed
+   });
+  }
+  return { eventsSent, eventsFailed, errors };
+ }
+}
+```
+---
+## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
+```typescript
+/**
+ * Event Logger Service
+ *
+ * Writes event processing logs to SFTP for audit/debugging.
+ * Optional - can be used to create rejection reports.
+ */
+import { Buffer } from 'node:buffer';
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+/**
+ * Service for writing event logs to SFTP
+ */
+export class EventLoggerService {
+ constructor(private sftp: SftpDataSource) {}
+ /**
+  * Write event log to SFTP
+  *
+  * @param remotePath - SFTP path for log file
+  * @param logData - Log data to write (JSON format)
+  */
+ async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
+  try {
+   const logContent = JSON.stringify(logData, null, 2);
+   await this.sftp.uploadFile(remotePath, logContent);
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   throw new Error(`Failed to write event log: ${errorMessage}`);
+  }
+ }
+}
+```
+---
+## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
+```typescript
+/**
+ * MAIN INGESTION ORCHESTRATION SERVICE
+ *
+ * This is the heart of the ingestion workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Discover files on SFTP
+ * 3. Download and parse CSV files
+ * 4. Transform data with UniversalMapper
+ * 5. Send events to Fluent Commerce
+ * 6. Archive processed files
+ * 7. Track file processing state
+ * 8. Track job progress with JobTracker
+ *
+ * NAMING PATTERN (consistent across all use cases):
+ * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
+ * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
+ * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ *
+ * - Change entity: Replace "Product" with "Inventory", "Location", etc.
+ * - Change format: Replace CSVParserService with XMLParserService/JSONParserService
+ * - Change source: Replace SftpDataSource with S3DataSource
+ * - Change destination: Replace sendEvent with Batch API or GraphQL mutations
+ */
+import { Buffer } from 'node:buffer';
+import {
+ createClient,
+ SftpDataSource,
+ CSVParserService,
+ UniversalMapper,
+ VersoriFileTracker,
+ JobTracker,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { FileMetadata } from '@fluentcommerce/fc-connect-sdk';
+import { ProductFileProcessorService } from './product-file-processor.service';
+import { EventSenderService } from './event-sender.service';
+import { EventLoggerService } from './event-logger.service';
+import { extractFileName } from '../utils/sftp-path.utils';
+import mappingConfig from '../../config/products.import.csv.json' with { type: 'json' };
+// ? VERSORI PLATFORM: Use native log from context, not LoggingService
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ * This function implements the complete ingestion workflow in 8 steps.
+ * Each step is clearly commented for AI understanding.
+ *
+ * @param ctx - Versori context object containing fetch, connections, log, activation, openKv
+ * @param jobId - Job ID for tracking
+ * @param tracker - JobTracker instance for job lifecycle management
+ */
+export async function executeProductIngestion(
+ ctx: any,
+ jobId: string,
+ tracker: JobTracker
+) {
+ // ? VERSORI PLATFORM: Extract native log from context
+ const { log, activation, openKv } = ctx;
+ const startTime = Date.now();
+ log.info('🚀 [EXECUTION BOUNDARY] Starting product ingestion', { jobId });
+ // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
+ let sftp: SftpDataSource | undefined;
+ try {
+  // �����������������������������������������������������������
+  // STEP 1: Initialize Fluent Client
+  // �����������������������������������������������������������
+  log.info('📡 [STEP 1/7] Initializing Fluent Commerce client', { jobId });
+  const client = await createClient(ctx);
+  log.info('✅ [DEBUG] Client created successfully');
+  /**
+   * setRetailerId() - REQUIRED for Event/Batch API operations
+   *
+   * retailerId must be set ONCE on the client after creation.
+   * This is required for Event API (sendEvent) operations.
+   */
+  const fluentRetailerId = activation.getVariable('fluentRetailerId');
+  if (!fluentRetailerId) {
+   log.error('❌ Missing required fluentRetailerId activation variable');
+   return {
+    success: false,
+    error: 'fluentRetailerId activation variable is required for Event API operations',
+    recommendation: 'Please configure fluentRetailerId in the Activation Variables section',
+   };
+  }
+  client.setRetailerId(fluentRetailerId);
+  log.info('✅ Client initialized with retailerId', { retailerId: fluentRetailerId });
+  // ========================================
+  // STEP 2: SFTP CREDENTIAL RETRIEVAL
+  // ========================================
+  log.info('🔐 [STEP 2/7] Retrieving SFTP credentials from connection');
+  let sftpUsername: string;
+  let sftpPassword: string;
+  try {
+   // Retrieve credentials from the 'SFTP' connection
+   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+   if (!sftpCred?.accessToken) {
+    throw new Error('No SFTP credentials found in connection configuration');
+   }
+   // Decode base64 accessToken to get "username:password"
+   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+   // Split on ':' to extract username and password
+   const parts = rawBasicAuth.split(':');
+   if (parts.length !== 2) {
+    throw new Error('Invalid SFTP credential format - expected username:password');
+   }
+   sftpUsername = parts[0];
+   sftpPassword = parts[1];
+   log.info('✅ SFTP credentials retrieved successfully', {
+    hasUsername: !!sftpUsername,
+    hasPassword: !!sftpPassword,
+   });
+  } catch (error: any) {
+   log.error('❌ Failed to retrieve SFTP credentials', {
+    message: error instanceof Error ? error.message : String(error),
+    stack: error instanceof Error ? error.stack : undefined,
+   });
+   return {
+    success: false,
+    error: 'Failed to retrieve SFTP credentials from connection configuration',
+    details: error?.message,
+    recommendation: 'Please ensure the SFTP connection is configured with Basic Authentication',
+   };
+  }
+  // ========================================
+  // STEP 3: CONFIGURATION & INITIALIZATION
+  // ========================================
+  log.info('⚙️ [STEP 3/7] Configuring SFTP and services', { jobId });
+  // Configure SFTP
+  const sftpConfig = {
+   host: activation.getVariable('sftpHost'),
+   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
+   username: sftpUsername,
+   password: sftpPassword,
+   remotePath: activation.getVariable('sftpRemotePath') || '/incoming/products/',
+   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
+   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
+   filePattern: activation.getVariable('filePattern') || '.csv',
+   enableFileTracking: activation.getVariable('enableFileTracking') !== 'false', // default: true
+  };
+  // Validate SFTP config
+  if (!sftpConfig.host) {
+   throw new Error('SFTP configuration incomplete: missing host');
+  }
+  // Initialize SFTP data source
+  sftp = new SftpDataSource(
+   {
+    type: 'SFTP_CSV',
+    connectionId: 'sftp-product-ingestion',
+    name: 'product-sync',
+    settings: {
+     host: sftpConfig.host,
+     port: sftpConfig.port,
+     username: sftpUsername,
+     password: sftpPassword,
+     remotePath: sftpConfig.remotePath,
+     archivePath: sftpConfig.archivePath,
+     errorPath: sftpConfig.errorPath,
+     filePattern: sftpConfig.filePattern,
+     encoding: 'utf8',
+     requireAbsolutePaths: true,
+    },
+   },
+   log
+  );
+  await sftp.validateConnection();
+  log.info('✅ SFTP connection validated successfully');
+  // Initialize services
+  const csvParser = new CSVParserService();
+  const mapper = new UniversalMapper(mappingConfig);
+  const kv = openKv(':project:');
+  const fileTracker = new VersoriFileTracker(kv, 'sftp-csv-product-sync');
+  // Initialize class-based services
+  const fileProcessor = new ProductFileProcessorService(
+    sftp,
+    csvParser,
+    mapper,
+    activation.getVariable('catalogueRef') || 'PC:MASTER:2'
+  );
+  // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
+  const eventSender = new EventSenderService(client, log);
+  const eventLogger = new EventLoggerService(sftp);
+  // �����������������������������������������������������������
+  // STEP 4: DISCOVER FILES ON SFTP
+  // �����������������������������������������������������������
+  log.info('📂 [STEP 4/7] Discovering files on SFTP', {
+   jobId,
+   remotePath: sftpConfig.remotePath,
+   filePattern: sftpConfig.filePattern,
+  });
+  await tracker.updateJob(jobId, {
+   status: 'processing',
+   stage: 'file_discovery',
+   message: 'Discovering files on SFTP',
+  });
+  // List files from SFTP
+  const files = await sftp.listFiles({
+   remotePath: sftpConfig.remotePath,
+   filePattern: sftpConfig.filePattern,
+  });
+  const maxFilesPerRun = parseInt(activation.getVariable('maxFilesPerRun') || '10', 10);
+  const csvFiles = maxFilesPerRun > 0 ? files.slice(0, maxFilesPerRun) : files;
+  log.info(`✅ Found ${csvFiles.length} CSV files matching pattern`);
+  if (csvFiles.length === 0) {
+   log.info('ℹ️ No files to process');
+   return {
+    success: true,
+    message: 'No files to process',
+    filesProcessed: 0,
+    filesFailed: 0,
+    recordsProcessed: 0,
+    eventsSent: 0,
+    eventsFailed: 0,
+    results: [],
+    duration: Date.now() - startTime,
+   };
+  }
+  log.info(`📋 Processing ${csvFiles.length} files`);
+  // ========================================
+  // STEP 5: PROCESS FILES
+  // ========================================
+  const results: any[] = [];
+  const eventConfig = {
+   catalogueRef: activation.getVariable('catalogueRef') || 'PC:MASTER:2',
+   catalogueType: activation.getVariable('catalogueType') || 'MASTER',
+   eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
+   eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
+  };
+  // Get event sending configuration
+  const eventConcurrency = Math.max(
+    1,
+    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
+  );
+  // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
+  // concurrency: 1 = sequential, concurrency > 1 = parallel
+  log.info(`Event concurrency: ${eventConcurrency}`, {
+    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
+    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
+  });
+  for (const file of csvFiles) {
+   const fileStartTime = Date.now();
+   const fileName = extractFileName(file.name);
+   try {
+    // Check if already processed (file tracking)
+    if (sftpConfig.enableFileTracking && (await fileTracker.wasFileProcessed(fileName))) {
+     log.info(`⏭️ Skipping already processed: ${fileName}`);
+     results.push({ file: fileName, success: true, skipped: true });
+     continue;
+    }
+    log.info(`🔄 Processing file: ${fileName}`);
+    // Download and parse
+    const csvContent = (await sftp.downloadFile(file.path, { encoding: 'utf8' })) as string;
+    const records = await csvParser.parse(csvContent);
+    const products: any[] = Array.isArray(records) ? records : records ? [records] : [];
+    log.info(`📊 Parsed ${products.length} products from ${fileName}`);
+    if (products.length === 0) {
+     await sftp.moveFile(file.path, `${sftpConfig.archivePath}${fileName}`, true);
+     results.push({ file: fileName, success: true, recordCount: 0 });
+     continue;
+    }
+    // Transform with UniversalMapper
+    const mapped = await mapper.map(products);
+    if (!mapped.success) {
+     throw new Error(`Mapping failed: ${mapped.errors?.join(', ')}`);
+    }
+    log.info(`✅ Transformed ${mapped.data.length} products for ${fileName}`);
+    // ═══════════════════════════════════════════════════════════
+    // STEP 6: Send Events
+    // ═══════════════════════════════════════════════════════════
+    log.info(`📤 [STEP 6/7] Sending events for ${fileName}`);
+    // ✅ Enhanced: Extract context for progress logging
+    const sampleProductRefs = mapped.data.slice(0, 5).map((p: any) => p.ref || p.skuRef);
+    const eventType = eventConfig.eventName || 'UPSERT_PRODUCT';
+    // ✅ Enhanced: Start logging with context
+    log.info(`[EventSender] Sending events for file "${fileName}"`, {
+      totalProducts: mapped.data.length,
+      eventType,
+      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
+      sampleProductRefs: sampleProductRefs.join(', '),
+      eventMode: eventConfig.eventMode || 'async'
+    });
+    const eventResult = await eventSender.sendEvents(
+      mapped.data,
+      eventConfig,
+      eventConcurrency
+    );
+    const eventsSent = eventResult.eventsSent;
+    const eventsFailed = eventResult.eventsFailed;
+    log.info(`✅ Events sent for ${fileName}`, {
+      successful: eventsSent,
+      failed: eventsFailed
+    });
+    // ✅ Enhanced: Completion logging with summary
+    log.info(`[EventSender] Event submission completed for file "${fileName}"`, {
+      totalProducts: mapped.data.length,
+      eventsSent,
+      eventsFailed,
+      successRate: mapped.data.length > 0 ? `${Math.round((eventsSent / mapped.data.length) * 100)}%` : '0%',
+      eventType
+    });
+    // Archive file and mark as processed
+    log.info(`📦 [STEP 7/7] Archiving file: ${fileName}`);
+    await sftp.moveFile(file.path, `${sftpConfig.archivePath}${fileName}`, true);
+    if (sftpConfig.enableFileTracking) {
+     await fileTracker.markFileProcessed(fileName, {
+      recordCount: products.length,
+      eventsSent,
+      eventsFailed,
+     });
+    }
+    results.push({
+     file: fileName,
+     success: true,
+     recordCount: products.length,
+     eventsSent,
+     eventsFailed,
+     duration: Date.now() - fileStartTime,
+    });
+    log.info(`✅ Successfully processed ${fileName}`, {
+     recordCount: products.length,
+     eventsSent,
+     eventsFailed,
+     duration: Date.now() - fileStartTime,
+    });
+   } catch (error: any) {
+    log.error(`❌ Error processing ${fileName}:`, {
+     message: error?.message || 'Unknown error',
+     stack: error?.stack,
+    });
+    // Move to error directory
+    try {
+     await sftp.moveFile(file.path, `${sftpConfig.errorPath}${fileName}`, true);
+    } catch (moveError) {
+     log.error(`❌ Failed to move error file: ${fileName}`);
+    }
+    results.push({ file: fileName, success: false, error: error.message });
+   }
+  }
+  // ========================================
+  // STEP 6: CLEANUP
+  // ========================================
+  try {
+   await sftp.dispose();
+   log.info('✅ [EXECUTION BOUNDARY] SFTP connection disposed successfully');
+  } catch (disposeError: any) {
+   log.error('⚠️ Failed to dispose SFTP connection', {
+    message: disposeError instanceof Error ? disposeError.message : String(disposeError),
+   });
+  }
+  const duration = Date.now() - startTime;
+  const filesProcessed = results.filter(r => r.success && !r.skipped).length;
+  const filesFailed = results.filter(r => !r.success).length;
+  const filesSkipped = results.filter(r => r.skipped).length;
+  log.info('🎉 [EXECUTION BOUNDARY] Product ingestion completed', {
+   jobId,
+   filesProcessed,
+   filesFailed,
+   filesSkipped,
+   duration,
+  });
+  return {
+   success: true,
+   filesProcessed,
+   filesSkipped,
+   filesFailed,
+   results,
+   duration,
+  };
+ } catch (error: any) {
+  log.error('💥 [EXECUTION BOUNDARY] Product ingestion failed', {
+   jobId,
+   message: error instanceof Error ? error.message : String(error),
+   stack: error instanceof Error ? error.stack : undefined,
+  });
+  return {
+   success: false,
+   error: error.message,
+   duration: Date.now() - startTime,
+  };
+ }
+}
+```
+---
+## 8. Utility Functions (src/utils/)
+### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
+```typescript
+/**
+ * SFTP Path Utilities
+ *
+ * Helper functions for SFTP path operations.
+ */
+/**
+ * Extract base filename from full SFTP path
+ *
+ * @param filePath - Full SFTP path or filename
+ * @returns Base filename only
+ *
+ * @example
+ * ```typescript
+ * extractFileName('/products/incoming/products_20250101.csv')
+ * // Returns: 'products_20250101.csv'
+ *
+ * extractFileName('products_20250101.csv')
+ * // Returns: 'products_20250101.csv'
+ * ```
+ */
+export function extractFileName(filePath: string): string {
+ return filePath.split('/').pop() || filePath;
+}
+```
+---
+### Job ID Generator (src/utils/job-id-generator.ts)
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking ingestion workflows
+ *
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
+ * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
+ *
+ * NAMING: generate{Entity}JobId or generateJobId (generic)
+ */
+/**
+ * Generate unique job ID
+ *
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
+ * @param entity - Entity abbreviation (VP, IP, ORD, PRD)
+ * @returns Unique job ID string
+ */
+export function generateJobId(type: string, entity: string): string {
+ const now = new Date();
+ // Format: YYYYMMDD
+ const dateStr = now.toISOString().slice(0, 10).replace(/-/g, "");
+ // Format: HHMMSS
+ const timeStr = now.toISOString().slice(11, 19).replace(/:/g, "");
+ // Random suffix (6 chars)
+ const randomStr = Math.random().toString(36).substring(2, 8);
+ return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
+}
+/**
+ * Parse job ID components
+ *
+ * @param jobId - Job ID to parse
+ * @returns Parsed components or null if invalid
+ */
+export function parseJobId(
+ jobId: string
+): {
+ type: string;
+ entity: string;
+ date: string;
+ time: string;
+ random: string;
+} | null {
+ const parts = jobId.split("_");
+ if (parts.length !== 5) {
+  return null;
+ }
+ return {
+  type: parts[0],
+  entity: parts[1],
+  date: parts[2],
+  time: parts[3],
+  random: parts[4],
+ };
+}
+```
+---
+## 5. Package Configuration
+### package.json
+```json
+{
+ "name": "sftp-csv-product-event",
+ "version": "1.2.0",
+ "description": "Ingest product CSV files from SFTP and send to Fluent Commerce Event API",
+ "type": "module",
+ "main": "src/index.ts",
+ "scripts": {
+  "dev": "versori dev",
+  "build": "versori build",
+  "deploy": "versori deploy"
+ },
+ "dependencies": {
+  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+  "@versori/run": "latest"
+ },
+ "devDependencies": {
+  "@types/node": "^20.0.0",
+  "typescript": "^5.0.0"
+ }
+}
+```
+### tsconfig.json
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+---
+## 6. Deployment Instructions
+### Deploy to Versori
+```bash
+# 1. Install dependencies
+npm install
+# 2. Test locally (if using Versori CLI)
+npm run dev
+# 3. Deploy to Versori platform
+npm run deploy
+```
+### Configure Activation Variables
+In Versori platform settings, configure:
+```json
+{
+ "sftpHost": "sftp.partner.com",
+ "sftpPort": 22,
+ "sftpPrivateKey": "",
+ "sftpIncomingPath": "/products/incoming",
+ "sftpProcessedPath": "/products/processed",
+ "sftpErrorPath": "/products/errors",
+ "filePattern": "products_*.csv",
+ "catalogueRef": "PC:MASTER:2",
+ "catalogueType": "MASTER",
+ "eventName": "UPSERT_PRODUCT",
+ "eventMode": "async",
+ "maxFilesPerRun": 10,
+ "requireAbsolutePaths": "true"
+}
+```
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `versori_ftp_server` Basic Auth connection.
+---
+## 7. Performance Notes
+- **Event sending:** Sequential processing (<100 records) is simple and reliable; consider parallel batches (Promise.allSettled) for 100-1000 records
+- **Batch API threshold:** For high volumes (>1000 records), use Batch API instead of Event API for better throughput
+- **File concurrency:** Files process sequentially by default; for very high volumes, consider processing 3-5 files in parallel if SFTP limits allow
+- **Rate limiting:** If 429 throttling is observed, add small backoff/delay or switch to Batch API
+---
+## 8. Testing
+### Test Scheduled Ingestion
+Upload a test CSV file to SFTP incoming directory and wait for the scheduled run.
+**Check logs:**
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and SFTP
+[STEP 3/8] Discovering files on SFTP
+[FILE 1/1] Processing file: products_20250124.csv
+[STEP 4/8] Downloading and parsing: products_20250124.csv
+[STEP 5/8] Transforming 5 products from products_20250124.csv
+[STEP 6/8] Sending 5 events to Fluent Commerce
+[STEP 7/8] Archiving file: products_20250124.csv
+[STEP 8/8] Completing job and calculating totals
+```
+### Test Ad hoc Ingestion
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{}'
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{
+  "filePattern": "urgent_*.csv", }'
+```
+### Test Job Status Query
+```bash
+curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
+ -H "Content-Type: application/json" \
+ -d '{
+  "jobId": "ADHOC_PROD_20251024_183045_abc123"
+ }'
+```
+---
+## Monitoring
+### Success Response
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 50,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 50,
+   "eventsFailed": 0
+  }
+ ],
+ "duration": 12345
+}
+```
+### Partial Success Response
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 45,
+ "eventsFailed": 5,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 45,
+   "eventsFailed": 5,
+   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
+  }
+ ],
+ "duration": 12345
+}
+```
+### Error Response
+```json
+{
+ "success": false,
+ "filesProcessed": 0,
+ "filesFailed": 1,
+ "totalRecords": 0,
+ "eventsSent": 0,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": false,
+   "error": "CSV parse error: Invalid structure"
+  }
+ ],
+ "duration": 876
+}
+```
+### Monitoring Metrics
+Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
+- **Files Processed** - Total files successfully processed
+- **Events Sent** - Total events sent to Fluent Commerce
+- **Events Failed** - Events that failed (check rejection reports)
+- **Processing Duration** - Time taken for complete workflow
+- **Rate Limiting** - Watch for 429 errors indicating throttling
+Use the status webhook for dashboards and automated monitoring.
+---
+**To replicate this template for other entities/formats:**
+1. **File Naming:** Replace `product`, `PRD` with your entity name across all files
+2. **CSV Structure:** Update expected CSV format and parsing logic
+3. **Mapping Config:** Create new mapping file in `config/` with correct field paths
+4. **Event Name:** Update event name (e.g., `UPSERT_INVENTORY`, `CREATE_LOCATION`)
+5. **Workflows:** Rename workflow exports to match entity (e.g., `scheduledInventoryIngestion`)
+6. **Service Function:** Rename main function (e.g., `executeInventoryIngestion`)
+7. **File Pattern:** Update SFTP file pattern (e.g., `inventory_*.csv`)
+8. **Data Source:** For S3 use `S3DataSource`, for XML use `XMLParserService`
+---
+## 10. Template summary
+This template provides:
+- SFTP ? Fluent Commerce ingestion workflow
+- File discovery and tracking (VersoriFileTracker)
+- CSV parsing and transformation (CSVParserService + UniversalMapper)
+- Event API integration with per-record error handling
+- File archival (success/error paths)
+- Job tracking (scheduled, ad hoc, status query modes)
+- SFTP connection cleanup and retry logic
+---
+## 🔧 Troubleshooting (quick)
+- No files found: Check `sftpIncomingPath` and `filePattern` (names only).
+- CSV parse failed: Move to errors/; validate CSV structure and encoding.
+- High event failures: Inspect rejection report; consider Batch API for very high volumes.
+- 429 throttling: Add small backoff/delay or use Batch API.
+## 🗺️ Mapping tips (compact)
+- Required fields: mark in mapping JSON; failed items won't be sent.
+- Arrays: use custom resolver `custom.splitByPipe` for pipe-separated values.
+- Nested objects: use custom resolvers `custom.buildPriceArray` and `custom.buildTaxType`.
+- Resolvers: prefer SDK built-ins (trim, uppercase, parseFloat).
+---