@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-sftp-parquet-product-event.md

@@ -1,2589 +1,2589 @@
----
-template_id: tpl-ingest-sftp-parquet-to-product-event
-canonical_filename: template-ingestion-sftp-parquet-product-event.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: ingestion
-source: sftp-parquet
-destination: fluent-event-api
-entity: product
-format: parquet
-logging: versori
-status: stable
-features:
-  - batched-events
-  - attribute-transformation
-  - memory-management
-  - json-serialization-handling
-  - enhanced-logging
----
-
-# Template: Ingestion - SFTP Parquet to Product Event
-
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
-- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
-- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large Parquet files)
-- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
-- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
-
----
-
-## 📋 Implementation Prompt
-
-```
-I need a Versori scheduled ingestion that:
-
-1) Discovers Parquet files on SFTP with file tracking to skip duplicates
-2) Downloads binary Parquet files as Buffer and parses with ParquetParserService
-3) Transforms records with UniversalMapper per mapping JSON (no array normalization needed)
-4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
-5) Archives files to processed/ or errors/ and writes optional rejection report
-6) Tracks progress with JobTracker and exposes a job-status webhook
-7) Uses native Versori log from context
-
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
-
----
-
-## 📋 Template Overview
-
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, Parquet schema, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP Parquet files, transforms it, and sends events to the Fluent Commerce Event API.
-
-### What This Template Does
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│          INGESTION WORKFLOW              │
-└─────────────────────────────────────────────────────────────────┘
-
-1. TRIGGER
-  ├─ Scheduled (Cron): Runs automatically every hour
-  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
-  └─ Status Query (Webhook): Check job progress
-
-2. DISCOVER FILES (SftpDataSource)
-  ├─ List files from SFTP directory
-  ├─ Filter by pattern (products_*.parquet)
-  ├─ Check file tracking (skip processed)
-  └─ Sort by oldest first
-
-3. DOWNLOAD & PARSE (ParquetParserService)
-  ├─ Download file as Buffer (binary format)
-  ├─ Parse Parquet columnar data
-  ├─ Returns array directly (no normalization needed)
-  └─ Validate Parquet structure
-
-4. TRANSFORM (UniversalMapper)
-  ├─ Map Parquet fields to Fluent schema
-  ├─ Apply SDK resolvers (trim, uppercase, etc.)
-  ├─ Handle nested objects (price, taxType)
-  ├─ Handle arrays (categoryRefs)
-  └─ Collect transformation errors
-
-5. SEND EVENTS (Event API)
-  ├─ Loop through transformed products
-  ├─ Send UPSERT_PRODUCT event (async)
-  ├─ Track success/failure count
-  └─ Continue on individual failures
-
-6. ARCHIVE (SftpDataSource)
-  ├─ Move file to processed/ folder
-  ├─ Or move to errors/ if validation failed
-  ├─ Generate timestamped archive name
-  └─ Verify archive success
-
-7. TRACK STATE (VersoriFileTracker)
-  ├─ Mark file as processed
-  ├─ Store processing metadata
-  ├─ Prevent duplicate processing
-  └─ Track record counts
-
-8. TRACK JOB (JobTracker)
-  ├─ Update job status at each step
-  ├─ Store final result in KV
-  ├─ Enable status queries via webhook
-  └─ Handle errors gracefully
-```
-
-### Key Features
-
-- Job tracking with status queries
-- Execution modes: scheduled, ad hoc, status query
-- Uses SftpDataSource, ParquetParserService, UniversalMapper, VersoriFileTracker, JobTracker
-- Error handling, retry logic, and SFTP cleanup
-- File tracking: VersoriFileTracker prevents duplicates; `forceReprocess` bypasses
-- Event API: Per-record failures don't block other records; rejection reports written to `errors/`
-- SFTP dispose() in finally block
-- Binary format handling (Buffer download, not string)
-
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-
-### 📦 Package Information
-
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk@latest
-```
-
-**Note:** Always use the latest SDK version for bug fixes and new features.
-
----
-
-**Templates are designed for direct deployment; customize via activation variables.**
-
----
-
-## 📦 SDK Imports (Verified - Versori Optimized)
-
-```typescript
-// ✅ VERIFIED IMPORTS - These match actual SDK exports
-import { Buffer } from 'node:buffer';
-import {
- createClient, // Universal client factory
- SftpDataSource, // SFTP operations with connection pooling
- ParquetParserService, // Parquet binary parsing
- UniversalMapper, // Field mapping with SDK resolvers
- VersoriFileTracker, // File processing state tracker
- JobTracker, // Job status tracking
-} from '@fluentcommerce/fc-connect-sdk';
-
-import type { FluentClient, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
-
-// Versori platform imports
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-
-**✅ VERSORI PLATFORM - Use Native Logs:**
-
-- Use `log` from context: `const { log } = ctx;`
-- Don't import LoggingService, StructuredLogger for Versori connectors
-- Native Versori logs are simpler and automatically integrated with platform monitoring
-
----
-
-## ⚙️ Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
-
-```bash
-# SFTP Configuration
-SFTP_HOST=sftp.partner.com
-SFTP_PORT=22
-
-# SFTP Paths
-SFTP_REMOTE_PATH=/products/incoming
-SFTP_ARCHIVE_PATH=/products/processed
-SFTP_ERROR_PATH=/products/errors
-
-# File Processing
-FILE_PATTERN=products_*.parquet
-MAX_FILES_PER_RUN=10
-
-# Product Configuration
-CATALOGUE_REF=PC:MASTER:2
-CATALOGUE_TYPE=MASTER
-
-# Event API Configuration
-EVENT_NAME=UPSERT_PRODUCT
-EVENT_MODE=async
-EVENT_CONCURRENCY=1
-
-# Fluent Configuration (via Versori connection)
-# Connection: fluent_commerce (OAuth2)
-FLUENT_RETAILER_ID=1
-
-# Event API Performance
-USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
-MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
-MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
-
-# Feature Toggles
-VALIDATE_CONNECTION_ON_START=false            # Validate Fluent connection early (default: false)
-ENABLE_FILE_TRACKING=true                     # Enable/disable file tracking (default: true)
-REQUIRE_ABSOLUTE_PATHS=true                   # "true" for AWS Transfer Family, "false" for standard OpenSSH
-```
-
-**Security:**
-- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup) section below.
-- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
-
-### Variable Explanations
-
-| Variable        | Purpose             | Default          | Customization Hints         |
-| ----------------------- | -------------------------------- | ------------------------- | ----------------------------------- |
-| `FLUENT_RETAILER_ID`   | Retailer ID for Event API    | -             | Required - Fluent retailer ID   |
-| `EVENT_CONCURRENCY`   | Event sending concurrency    | `1`            | `1` = sequential, `>1` = parallel (3-10 recommended) |
-| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
-| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100`          | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
-| `MEMORY_BATCH_SIZE`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
-| `VALIDATE_CONNECTION_ON_START` | Validate Fluent connection early | `false` | `true` = fail-fast mode, `false` = validate on first call |
-| `ENABLE_FILE_TRACKING` | Skip already-processed files | `true` | `true` = use KV tracking, `false` = process all files |
-| `SFTP_HOST`       | SFTP server hostname       | -             | Required - SFTP server address   |
-| `SFTP_PORT`       | SFTP server port         | `22`           | Standard SFTP port         |
-| `SFTP_REMOTE_PATH`   | Incoming files directory     | `/products/incoming`   | Where new files arrive       |
-| `SFTP_ARCHIVE_PATH`   | Processed files archive     | `/products/processed`   | Where files move after success   |
-| `SFTP_ERROR_PATH`     | Failed files directory      | `/products/errors`    | Where files move after errors    |
-| `FILE_PATTERN`      | File name filter         | `products_*.parquet`   | Glob pattern for matching files   |
-| `CATALOGUE_REF`     | Product catalogue reference   | `PC:MASTER:2`       | Target catalogue in Fluent     |
-| `CATALOGUE_TYPE`     | Catalogue type          | `MASTER`         | Usually MASTER or STANDARD     |
-| `EVENT_NAME`       | Event to send          | `UPSERT_PRODUCT`     | Event name from Rubix        |
-| `EVENT_MODE`       | Event processing mode      | `async`          | async (recommended) or sync     |
-| `MAX_FILES_PER_RUN`    | Max files per execution     | `10`           | Prevent timeout on large batches  |
-| `REQUIRE_ABSOLUTE_PATHS` | Require absolute SFTP paths   | `true`          | Recommended for clarity       |
-
----
-
-## 🔒 SDK Automatic Behaviors (v0.1.40+)
-
-**The SDK automatically validates and retries for improved reliability:**
-
-### retailerId Validation
-- **SDK validates** `retailerId` before calling `sendEvent()`
-- **Checks:** `event.retailerId || client.retailerId`
-- **If missing:** Throws `"retailerId is required for Event API..."`
-- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
-
-### 401 Auth Retry
-- **Automatic retry** for platform auth failures (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s)
-- **Applies to:** All `sendEvent()` calls (async and sync modes)
-- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
-
-### 5xx Server Retry
-- **Automatic retry** for transient server errors (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
-- **Protects:** Against Fluent API transient failures
-
-### No Code Changes Required
-- All templates remain compatible
-- Retry logic is automatic and transparent
-- Better error messages guide configuration
-
-**See:** [Event API Guide](./event-api-guide.md) for complete details
-
----
-
-## SFTP Connection Setup
-
-**CRITICAL: Buffer Import for Deno/Versori Runtime**
-
-```typescript
-import { Buffer } from 'node:buffer'; // ← Required for credential decoding!
-```
-
-**Why:** Deno/Versori runtime doesn't have `Buffer` as a global. You'll get "Buffer is not defined" error without this import.
-
----
-
-### Two Methods for SFTP Credential Access
-
-Versori provides **two methods** to access SFTP credentials securely:
-
-#### Method 1: Basic Auth Connection (Recommended)
-
-**Best Practice:** Store SFTP credentials in a Versori connection object with Basic Auth.
-
-**Setup:**
-1. In Versori platform, create a connection named `SFTP`
-2. Set **Authentication Type**: `Basic Auth`
-3. Enter **Username**: Your SFTP username
-4. Enter **Password**: Your SFTP password
-
-**Access in Code:**
-```typescript
-import { Buffer } from 'node:buffer'; // Required!
-
-// Retrieve credentials from the 'SFTP' connection
-const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-const [sftpUsername, sftpPassword] = rawBasicAuth.split(':');
-```
-
-**Why use this method?**
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection can be reused across workflows
-- ✅ No sensitive data in activation variables
-- ✅ Easier credential rotation
-
-#### Method 2: Connection Variables (Alternative)
-
-**Alternative:** Use Versori connection variables API.
-
-**Setup:**
-1. Create a connection with **any** authentication type
-2. Add custom variables: `sftp_username`, `sftp_password`
-
-**Access in Code:**
-```typescript
-const { connections } = ctx;
-const sftpUsername = connections.getVariable('sftp_username');
-const sftpPassword = connections.getVariable('sftp_password');
-```
-
-**Why use this method?**
-- ✅ Credentials stored securely in Versori vault
-- ✅ Connection can be reused across workflows
-- ✅ No sensitive data in activation variables
-- ✅ Easier credential rotation
-
----
-
-## 🔧 Production Implementation
-
-### Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-product-event-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── product.types.ts            # Shared type definitions
-```
-
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
-
----
-
-## Workflow Files
-
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-
-All time-based triggers that run automatically on cron schedules.
-
-#### `src/workflows/scheduled/daily-product-sync.ts`
-
-**Purpose**: Automatic Daily product sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Scheduled Workflow: Daily Product Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: product-sync.service.ts
- */
-export const dailyProductSync = schedule(
-  'product-sync-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv } = ctx;
-    const jobId = `product-sync-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
-    await tracker.updateJob(jobId, { status: 'processing' });
-
-    try {
-      // Reuse shared orchestration logic
-      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
-      await tracker.markCompleted(jobId, result);
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      await tracker.markFailed(jobId, e);
-      return { success: false, jobId, error: e?.message };
-    }
-  })
-);
-```
-
----
-
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-
-All HTTP-based triggers that create webhook endpoints.
-
-#### `src/workflows/webhook/adhoc-product-sync.ts`
-
-**Purpose**: Manual product sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Webhook: Manual Product Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
- * Request body (optional): { filePattern: "urgent_*.parquet", maxFiles: 5 }
- *
- * Pattern: webhook().then(http()) - needs Fluent API access
- * Uses shared service: product-sync.service.ts
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const adhocProductSync = webhook('product-sync-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'product-sync-adhoc', // Versori validates API key
-}).then(
-  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, data } = ctx;
-    const jobId = `product-sync-adhoc-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    const filePattern = data?.filePattern as string;
-    const maxFiles = data?.maxFiles as number;
-
-    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-    });
-
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'manual',
-      stage: 'initialization',
-      status: 'queued',
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
-      .then((result) => {
-        log.info('✅ [BACKGROUND] Product sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          filesFailed: result.filesFailed,
-          recordsProcessed: result.recordsProcessed,
-        });
-        return tracker.markCompleted(jobId, result);
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-
-        log.error('❌ [BACKGROUND] Product sync failed', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-        });
-
-        return tracker.markFailed(jobId, errorMessage);
-      });
-
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Product sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-
-#### `src/workflows/webhook/job-status-check.ts`
-
-**Purpose**: Query job status
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
-**Request body**: `{ "jobId": "product-sync-1234567890" }`
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
- * Request body: { "jobId": "product-sync-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const productSyncJobStatus = webhook('product-sync-job-status', {
-  response: { mode: 'sync' },
-  connection: 'product-sync-job-status',
-}).then(
-  fn('status', async ctx => {
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-
-    if (!jobId) {
-      return { success: false, error: 'jobId required' };
-    }
-
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
-
----
-
-### 3. Entry Point (`index.ts`)
-
-**Purpose**: Register all workflows with Versori platform
-
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * Versori automatically discovers and registers exported workflows
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- */
-
-// Import scheduled workflows
-import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
-
-// Import webhook workflows
-import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
-import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
-
-// Register all workflows
-export {
-  // Scheduled (time-based triggers)
-  dailyProductSync,
-
-  // Webhooks (HTTP-based triggers)
-  adhocProductSync,
-  productSyncJobStatus,
-};
-```
-
-**What Gets Exposed:**
-- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
-- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
-- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
-
----
-
-### Adding New Workflows
-
-**To add a scheduled workflow:**
-1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**To add a webhook workflow:**
-1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**Example - Adding hourly delta sync:**
-
-```typescript
-// src/workflows/scheduled/hourly-delta-sync.ts
-export const hourlyDeltaSync = schedule(
-  'product-delta-hourly',
-  '0 * * * *' // Every hour
-).then(
-  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
-    // Delta sync logic (skip BPP)
-    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
-    return result;
-  })
-);
-
-// index.ts (add to imports and exports)
-import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
-export { daily_product_sync, hourlyDeltaSync, ... };
-```
-
----
-## 🔍 Complete Production Code
-
-### 1. Entry Point (index.ts)
-
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * This file registers three workflows:
- * 1. Scheduled ingestion (runs automatically)
- * 2. Ad hoc webhook (manual trigger)
- * 3. Job status webhook (query progress)
- */
-
-import {
- scheduledProductIngestion,
- adhocProductIngestion,
- productIngestionJobStatus,
-} from "./src/workflows/product-ingestion";
-
-// Register workflows with Versori platform
-export {
- scheduledProductIngestion, // Cron-based auto-run
- adhocProductIngestion, // Manual webhook trigger
- productIngestionJobStatus, // Job status query
-};
-```
-
----
-
-### 2. Workflows (src/workflows/product-ingestion.ts)
-
-```typescript
-/**
- * Workflows - Defines 3 execution patterns for product ingestion
- *
- * WORKFLOW 1: Scheduled (Cron) - Runs automatically every hour
- * WORKFLOW 2: Ad hoc (Webhook) - Manual trigger for immediate processing
- * WORKFLOW 3: Job Status (Webhook) - Query job progress
- */
-
-import { schedule, webhook, http, fn } from '@versori/run';
-import { executeProductIngestion, getJobStatus } from '../services/product-ingestion.service';
-import { generateJobId } from '../utils/job-id-generator';
-
-/**
- * WORKFLOW 1: Scheduled Ingestion
- *
- * Purpose: Automated hourly product file processing
- * Trigger: Cron schedule (every hour at minute 0)
- * File Tracking: Uses VersoriFileTracker to skip processed files
- */
-export const scheduledProductIngestion = schedule(
- 'product-ingestion-scheduled',
- '0 * * * *' // → CUSTOMIZE: Cron expression
-)
-.then(
- http('execute-scheduled-ingestion', { connection: 'fluent_commerce' }, async (ctx) => {
-  const { log } = ctx;
-  const startTime = Date.now();
-
-  // Generate unique job ID for tracking
-  const jobId = generateJobId('SCHEDULED', 'PROD');
-
-  log.info('⏰ Scheduled ingestion triggered', { jobId });
-
-  try {
-   const result = await executeProductIngestion(ctx, {
-    jobId,
-    triggeredBy: 'schedule',
-   });
-
-   const duration = Date.now() - startTime;
-   log.info('✅ Scheduled ingestion completed', {
-    jobId,
-    filesProcessed: result.filesProcessed,
-    recordsProcessed: result.recordsProcessed,
-    duration: `${duration}ms`
-   });
-
-   return result;
-
-  } catch (error: unknown) {
-   const duration = Date.now() - startTime;
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   const errorStack = error instanceof Error ? error.stack : undefined;
-   log.error('❌ Scheduled ingestion failed', {
-    jobId,
-    message: errorMessage,
-    stack: errorStack,
-    errorType: error instanceof Error ? error.constructor.name : 'Error',
-    duration: `${duration}ms`,
-    recommendation: 'Check SFTP connection settings and credentials'
-   });
-   throw error;
-  }
- })
-);
-
-/**
- * WORKFLOW 2: Ad hoc Ingestion (Manual Trigger)
- *
- * Purpose: Manual file processing with optional filters
- * Trigger: Webhook POST to /webhooks/product-ingestion-adhoc
- */
-export const adhocProductIngestion = webhook('product-ingestion-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-})
-.then(
- http('execute-adhoc-ingestion', { connection: 'fluent_commerce' }, async (ctx) => {
-  const { data, log } = ctx;
-  const startTime = Date.now();
-
-  const jobId = generateJobId('ADHOC', 'PROD');
-
-  const filePattern = data.filePattern as string | undefined;
-  const maxFiles = data.maxFiles as number | undefined;
-  const forceReprocess = data.forceReprocess as boolean | undefined;
-
-  log.info('🚀 [WEBHOOK] Adhoc product ingestion triggered', {
-   jobId,
-   filePattern: filePattern || 'default',
-   maxFiles: maxFiles || 'unlimited',
-   forceReprocess: !!forceReprocess
-  });
-
-  // ✅ Fire-and-forget: Start background processing WITHOUT await
-  // The promise continues execution after we return the response
-  executeProductIngestion(ctx, {
-    jobId,
-    triggeredBy: 'webhook',
-    filePattern,
-    maxFiles,
-    forceReprocess,
-  })
-    .then((result) => {
-      const duration = Date.now() - startTime;
-      log.info('✅ [BACKGROUND] Product ingestion completed successfully', {
-        jobId,
-        filesProcessed: result.filesProcessed,
-        recordsProcessed: result.recordsProcessed,
-        filesFailed: result.filesFailed,
-        duration: `${duration}ms`
-      });
-    })
-    .catch((error: unknown) => {
-      const duration = Date.now() - startTime;
-      const errorMessage = error instanceof Error ? error.message : String(error);
-      const errorStack = error instanceof Error ? error.stack : undefined;
-      log.error('❌ [BACKGROUND] Product ingestion failed', {
-        jobId,
-        message: errorMessage,
-        stack: errorStack,
-        errorType: error instanceof Error ? error.constructor.name : 'Error',
-        duration: `${duration}ms`,
-        recommendation: 'Verify webhook payload and SFTP access'
-      });
-    });
-
-  // Return immediately with jobId (response sent with this return value)
-  return {
-    success: true,
-    jobId,
-    message: 'Product ingestion started in background',
-    statusEndpoint: `https://{workspace}.versori.run/product-ingestion-job-status`,
-    note: 'Poll the status endpoint with jobId to check progress',
-  };
- })
-);
-
-/**
- * WORKFLOW 3: Job Status Query
- *
- * Purpose: Check job progress and status
- * Trigger: Webhook GET/POST to /webhooks/product-ingestion-job-status?jobId=xxx
- */
-export const productIngestionJobStatus = webhook(
- 'product-ingestion-job-status'
-)
-.then(
- fn('query-job-status', async (ctx) => {
-  const { data, log, openKv } = ctx;
-
-  const jobId = data.jobId as string;
-
-  if (!jobId) {
-   log.error('❌ Job ID not provided in request');
-   return {
-    success: false,
-    error: 'Job ID is required. Provide jobId in query param or request body.',
-    recommendation: 'Include jobId in request body: { "jobId": "SCHEDULED_PROD_..." }'
-   };
-  }
-
-  log.info('🔍 Querying job status', { jobId });
-
-  try {
-   const status = await getJobStatus(openKv(':project:'), jobId, log);
-
-   if (!status) {
-    log.info('⚠️ Job not found', { jobId });
-    return {
-     success: false,
-     error: 'Job not found',
-     jobId,
-     recommendation: 'Verify jobId is correct and job exists'
-    };
-   }
-
-   log.info('✅ Job status retrieved', { jobId, status: status.status });
-
-   return {
-    success: true,
-    jobId,
-    ...status
-   };
-
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   log.error('❌ Failed to query job status', {
-    jobId,
-    message: errorMessage,
-    stack: error instanceof Error ? error.stack : undefined,
-    errorType: error instanceof Error ? error.constructor.name : 'Error',
-    recommendation: 'Check KV store connectivity'
-   });
-
-   return {
-    success: false,
-    jobId,
-    error: errorMessage
-   };
-  }
- })
-);
-```
-
----
-
-## 3. Type Definitions (src/types/product-ingestion.types.ts)
-
-```typescript
-/**
- * Type Definitions for Product Ingestion
- *
- * Centralized type definitions for product ingestion workflow
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Product interface - represents transformed product data
- */
-export interface Product {
- ref: string;
- type?: string;
- status?: string;
- name: string;
- summary?: string;
- gtin?: string;
- catalogue?: { ref?: string };
- attributes?: Record<string, unknown>;
-}
-
-/**
- * Event configuration
- */
-export interface EventConfig {
- eventName: string;
- catalogueRef: string;
- catalogueType: string;
- eventMode: 'async' | 'sync';
-}
-
-/**
- * Event result - tracks success/failure counts
- */
-export interface EventResult {
- eventsSent: number;
- eventsFailed: number;
- errors: Array<{ productRef: string; error: string }>;
-}
-
-/**
- * Process file result
- */
-export interface ProcessFileResult {
- success: boolean;
- products: Product[];
- error?: string;
-}
-
-/**
- * Versori Context Interface
- * Represents the Versori runtime context passed to workflow functions
- */
-export interface VersoriContext {
- log: {
-  info: (message: string, data?: Record<string, unknown>) => void;
-  warn: (message: string, data?: Record<string, unknown>) => void;
-  error: (message: string, data?: Record<string, unknown>) => void;
-  debug?: (message: string, data?: Record<string, unknown>) => void;
- };
- openKv: (namespace: string) => {
-  get: (key: string) => Promise<unknown>;
-  set: (key: string, value: unknown) => Promise<void>;
-  delete: (key: string) => Promise<void>;
- };
- activation: {
-  getVariable: (name: string) => string | undefined;
-  connections?: Record<string, unknown>;
- };
- connections?: Record<string, unknown>;
- data?: unknown;
- fetch?: typeof fetch;
-}
-
-/**
- * Parameters for ingestion workflow
- */
-export interface ProductIngestionParams {
- jobId: string;
- triggeredBy: 'schedule' | 'webhook';
- filePattern?: string;
- maxFiles?: number;
- forceReprocess?: boolean;
- catalogueRef?: string;
- priority?: 'normal' | 'high';
-}
-
-/**
- * Result from ingestion workflow
- */
-export interface ProductIngestionResult {
- success: boolean;
- jobId: string;
- filesProcessed: number;
- filesFailed: number;
- filesSkipped?: number;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- fileResults: FileProcessingResult[];
- error?: string;
-}
-
-/**
- * Per-file processing result
- */
-export interface FileProcessingResult {
- fileName: string;
- success: boolean;
- skipped?: boolean;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- duration: number;
- error?: string;
-}
-```
-
----
-
-## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
-
-```typescript
-/**
- * Product File Processor Service
- *
- * Downloads Parquet files from SFTP, parses, and transforms with UniversalMapper.
- * Parquet-specific: Downloads as Buffer (binary format), uses parseSimple() method.
- */
-
-import { Buffer } from 'node:buffer';
-import {
- SftpDataSource,
- ParquetParserService,
- UniversalMapper,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
-
-/**
- * Service for processing Parquet product files
- */
-export class ProductFileProcessorService {
- constructor(
-  private sftp: SftpDataSource,
-  private parquetParser: ParquetParserService,
-  private mapper: UniversalMapper,
-  private catalogueRef: string
- ) {}
-
- /**
-  * Download Parquet file from SFTP, parse, and transform with UniversalMapper
-  *
-  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
-  * - Explicit null assignments after each step
-  * - Finally block guarantees cleanup
-  * - Prevents OOM errors on large Parquet files
-  */
- async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
-  // ✅ CRITICAL: Variables for cleanup tracking
-  let buffer: Buffer | null = null;
-  let parsed: unknown | null = null;
-
-  try {
-   // STEP 1: Download Parquet file as Buffer (binary format)
-   buffer = (await this.sftp.downloadFile(remoteFilePath)) as Buffer;
-
-   // STEP 2: Parse Parquet with type safety
-   try {
-    // ✅ PARQUET: Use parseSimple() method (returns array directly)
-    parsed = await this.parquetParser.parseSimple(buffer, remoteFilePath);
-   } catch (parseError: unknown) {
-    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
-    return {
-     success: false,
-     products: [],
-     error: `Parquet parse error: ${errorMessage}`,
-    };
-   }
-
-   // ✅ Clear buffer from memory - no longer needed after parsing
-   buffer = null;
-
-   // STEP 3: PARQUET-SPECIFIC - Returns array directly, no normalization needed
-   const rawProducts = Array.isArray(parsed) ? parsed : [];
-
-   if (rawProducts.length === 0) {
-    return {
-     success: false,
-     products: [],
-     error: 'No products found in Parquet file',
-    };
-   }
-
-   // ✅ Clear parsed data from memory - only need rawProducts now
-   parsed = null;
-
-   // STEP 4: Transform with UniversalMapper
-   // PARQUET-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
-   const sourceDataWithContext = rawProducts.map(item => ({
-    ...item,
-    $context: { catalogueRef: this.catalogueRef },
-   }));
-
-   const mappingResult = await this.mapper.map(sourceDataWithContext);
-
-   if (!mappingResult.success) {
-    return {
-     success: false,
-     products: [],
-     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
-    };
-   }
-
-   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-     skippedFields: mappingResult.skippedFields,
-     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-    });
-   }
-
-   return {
-    success: true,
-    products: mappingResult.data as Product[],
-   };
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   return {
-    success: false,
-    products: [],
-    error: errorMessage,
-   };
-  } finally {
-   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
-   buffer = null;
-   parsed = null;
-  }
- }
-}
-```
-
----
-
-## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
-
-```typescript
-/**
- * Event Sender Service
- *
- * Sends product events to Fluent Commerce Event API with per-record error handling.
- * Continues processing on individual failures (Event API best practice).
- * Supports configurable concurrency (sequential or parallel).
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
-
-/**
- * Service for sending events to Fluent Commerce Event API
- *
- * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
- */
-export class EventSenderService {
- constructor(
-  private client: FluentClient,
-  private log?: any  // ✅ Optional logger for progress tracking
- ) {}
-
- /**
-  * Send events to Fluent Commerce Event API with configurable concurrency
-  *
-  * **Performance Characteristics:**
-  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
-  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
-  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
-  *
-  * **Implementation Strategy:**
-  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
-  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
-  * - Both modes: Per-record error tracking (failures don't block other events)
-  *
-  * @param products - Array of products to send as events
-  * @param eventConfig - Event configuration (name, catalogue ref, mode)
-  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
-  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
-  */
- async sendEvents(
-  products: Product[],
-  eventConfig: EventConfig,
-  concurrency: number = 1
- ): Promise<EventResult> {
-  // Validate concurrency (guard against invalid values)
-  const safeConc = Math.max(1, Math.floor(concurrency));
-
-  // Result accumulators
-  let eventsSent = 0;
-  let eventsFailed = 0;
-  const errors: Array<{ productRef: string; error: string }> = [];
-
-  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
-  if (this.log) {
-   this.log.info('📤 Starting event sending', {
-    totalProducts: products.length,
-    concurrency: safeConc,
-    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
-    eventName: eventConfig.eventName,
-    catalogueRef: eventConfig.catalogueRef
-   });
-  }
-
-  // Helper: Build event payload (DRY - reused in both modes)
-  const buildPayload = (product: Product) => ({
-   name: eventConfig.eventName,
-   entityRef: eventConfig.catalogueRef,
-   entityType: 'PRODUCT_CATALOGUE' as const,
-   entitySubtype: eventConfig.catalogueType,
-   rootEntityRef: eventConfig.catalogueRef,
-   rootEntityType: 'PRODUCT_CATALOGUE' as const,
-   attributes: product,
-  });
-
-  // ============================================================================
-  // SEQUENTIAL MODE (concurrency === 1)
-  // ============================================================================
-  if (safeConc === 1) {
-   for (let i = 0; i < products.length; i++) {
-    const product = products[i];
-
-    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
-    if (this.log && i % 10 === 0) {
-     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
-      productRef: product.ref,
-      progress: `${((i / products.length) * 100).toFixed(1)}%`
-     });
-    }
-
-    try {
-     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
-     eventsSent++;
-    } catch (err: unknown) {
-     eventsFailed++;
-     const errorMsg = err instanceof Error ? err.message : String(err);
-     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
-     // Continue processing (failure doesn't block other products)
-    }
-   }
-
-   // ✅ PRODUCTION ENHANCEMENT: Log completion
-   if (this.log) {
-    this.log.info('✅ Sequential event sending completed', {
-     totalProducts: products.length,
-     eventsSent,
-     eventsFailed
-    });
-   }
-
-   return { eventsSent, eventsFailed, errors };
-  }
-
-  // ============================================================================
-  // PARALLEL MODE (concurrency > 1)
-  // ============================================================================
-  const totalChunks = Math.ceil(products.length / safeConc);
-
-  for (let i = 0; i < products.length; i += safeConc) {
-   const chunk = products.slice(i, i + safeConc);
-   const chunkNumber = Math.floor(i / safeConc) + 1;
-
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
-   if (this.log) {
-    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
-     chunkNumber,
-     totalChunks,
-     productsInChunk: chunk.length,
-     productRange: `${i + 1}-${i + chunk.length}`,
-     totalProducts: products.length,
-     progress: `${((i / products.length) * 100).toFixed(1)}%`
-    });
-   }
-
-   // Fire all requests in chunk concurrently
-   const results = await Promise.allSettled(
-    chunk.map(product =>
-     this.client
-      .sendEvent(buildPayload(product), eventConfig.eventMode)
-      .then(() => ({ success: true as const, product }))
-      .catch(error => ({ success: false as const, product, error }))
-    )
-   );
-
-   // Aggregate chunk results into totals
-   let chunkSuccess = 0;
-   let chunkFailed = 0;
-
-   for (const result of results) {
-    if (result.status === 'fulfilled' && result.value.success) {
-     eventsSent++;
-     chunkSuccess++;
-    } else {
-     eventsFailed++;
-     chunkFailed++;
-     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
-     const product = result.status === 'fulfilled' ? result.value.product : null;
-     errors.push({
-      productRef: product?.ref || 'unknown',
-      error: error?.message || String(error) || 'unknown error',
-     });
-    }
-   }
-
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
-   if (this.log) {
-    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
-     chunkNumber,
-     totalChunks,
-     chunkSuccess,
-     chunkFailed,
-     totalSentSoFar: eventsSent,
-     totalFailedSoFar: eventsFailed,
-     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
-    });
-   }
-  }
-
-  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
-  if (this.log) {
-   this.log.info('✅ Parallel event sending completed', {
-    totalProducts: products.length,
-    totalChunks,
-    concurrency: safeConc,
-    eventsSent,
-    eventsFailed
-   });
-  }
-
-  return { eventsSent, eventsFailed, errors };
- }
-}
-```
-
----
-
-## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
-
-```typescript
-/**
- * Event Logger Service
- *
- * Writes event processing logs to SFTP for audit/debugging.
- * Optional - can be used to create rejection reports.
- */
-
-import { Buffer } from 'node:buffer';
-import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Service for writing event logs to SFTP
- */
-export class EventLoggerService {
- constructor(private sftp: SftpDataSource) {}
-
- /**
-  * Write event log to SFTP
-  *
-  * @param remotePath - SFTP path for log file
-  * @param logData - Log data to write (JSON format)
-  */
- async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
-  try {
-   const logContent = JSON.stringify(logData, null, 2);
-   await this.sftp.uploadFile(remotePath, logContent);
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   throw new Error(`Failed to write event log: ${errorMessage}`);
-  }
- }
-}
-```
-
----
-
-## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
-
-```typescript
-/**
- * MAIN INGESTION ORCHESTRATION SERVICE
- *
- * This is the heart of the ingestion workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Discover files on SFTP
- * 3. Download and parse Parquet files
- * 4. Transform data with UniversalMapper
- * 5. Send events to Fluent Commerce
- * 6. Archive processed files
- * 7. Track file processing state
- * 8. Track job progress with JobTracker
- *
- * NAMING PATTERN (consistent across all use cases):
- * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
- * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
- * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- *
- * - Change entity: Replace "Product" with "Inventory", "Location", etc.
- * - Change format: Replace ParquetParserService with XMLParserService/JSONParserService
- * - Change source: Replace SftpDataSource with S3DataSource
- * - Change destination: Replace sendEvent with Batch API or GraphQL mutations
- */
-
-import { Buffer } from 'node:buffer';
-import {
- createClient,
- SftpDataSource,
- ParquetParserService,
- UniversalMapper,
- VersoriFileTracker,
- JobTracker,
- type FluentClient,
- type JobStatus,
-} from '@fluentcommerce/fc-connect-sdk';
-import type {
- ProductIngestionParams,
- ProductIngestionResult,
- FileProcessingResult,
- EventConfig,
-} from '../types/product-ingestion.types';
-import { ProductFileProcessorService } from './product-file-processor.service';
-import { EventSenderService } from './event-sender.service';
-import { EventLoggerService } from './event-logger.service';
-import { joinSftpPath, timestampedName, getBaseName } from '../utils/sftp-path.utils';
-
-import mappingConfig from '../../config/products.import.parquet.json' with { type: 'json' };
-
-// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
-
-/**
- * Query job status from KV store
- *
- * NAMING: get{Entity}JobStatus or just getJobStatus (generic)
- *
- * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
- */
-export async function getJobStatus(
- kv: ReturnType<VersoriContext['openKv']>,
- jobId: string,
- log: VersoriContext['log']
-): Promise<JobStatus | undefined> {
- try {
-  const tracker = new JobTracker(kv, log);
-  return await tracker.getJob(jobId); // ✅ Use getJob() not getJobStatus()
- } catch (error: unknown) {
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  log.error('Failed to get job status', {
-   jobId,
-   message: errorMessage,
-   stack: error instanceof Error ? error.stack : undefined,
-   errorType: error instanceof Error ? error.constructor.name : 'Error'
-  });
-  return undefined;
- }
-}
-
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- * This function implements the complete ingestion workflow in 8 steps.
- * Each step is clearly commented for AI understanding.
- */
-export async function executeProductIngestion(
- ctx: VersoriContext,
- params: ProductIngestionParams
-): Promise<ProductIngestionResult> {
-
-  // ✅ VERSORI PLATFORM: Extract native log from context (LoggingService was removed - use native log)
- const { log, openKv, activation } = ctx;
- const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
-
- // Open KV store for state management and job tracking
- // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
- // ✅ Pass native log to JobTracker
- const kv = openKv(':project:');
- const tracker = new JobTracker(kv, log);
-
- const startTime = Date.now();
- const fileResults: FileProcessingResult[] = [];
-
- // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
- let sftp: SftpDataSource | undefined;
-
- try {
-  // 
-  // STEP 1: Initialize Job Tracking
-  // 
-  log.info('📊 [STEP 1/8] Initializing job tracking', { jobId });
-
-  await tracker.createJob(jobId, {
-   triggeredBy,
-   filePattern: filePattern || 'default',
-   maxFiles: maxFiles || 'unlimited',
-   forceReprocess: !!forceReprocess
-  });
-
-  // 
-  // STEP 2: Initialize Fluent Client & SFTP Connection
-  // 
-  log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and SFTP', { jobId });
-
-  // ✅ Optional: Validate connection immediately (fail-fast mode)
-  // Set activation variable 'validateConnectionOnStart' = 'true' to enable
-  // When enabled: Executes query { me { ref } } to verify authentication
-  // When disabled: Fast creation, validation happens on first API call (default)
-  const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
-  const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
-
-  if (!client) {
-   throw new Error('Failed to create Fluent Commerce client');
-  }
-
-  if (validateConnection) {
-   log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
-  }
-
-  // ✅ CRITICAL: Set retailerId for Event API calls
-  // Event API requires retailerId - fail fast if not configured
-  const fluentRetailerId = activation.getVariable('fluentRetailerId');
-  if (!fluentRetailerId) {
-   throw new Error('fluentRetailerId is required for Event API calls');
-  }
-  client.setRetailerId(fluentRetailerId);
-  log.info('RetailerId set for Event API', { retailerId: fluentRetailerId });
-
-  // ========================================
-  // SFTP CREDENTIAL RETRIEVAL
-  // ========================================
-
-  /**
-   * Retrieve SFTP credentials from connection configuration
-   *
-   * This approach retrieves credentials stored in the Versori connection settings.
-   * The connection must be configured in the UI with Basic Authentication.
-   *
-   * Steps:
-   * 1. Call ctx.credentials().getAccessToken('SFTP') to get base64-encoded credentials
-   * 2. Decode the accessToken from base64 to get "username:password" string
-   * 3. Split on ':' to extract username and password
-   *
-   * Connection Name: 'SFTP' (must match the connection name in Versori UI)
-   * Auth Type: Basic Authentication (username + password)
-   *
-   * This method provides:
-   * - Centralized credential management through Versori UI
-   * - Better security (credentials not stored in integration variables)
-   * - Easier credential rotation and updates
-   */
-  log.info('Retrieving SFTP credentials from connection configuration');
-
-  let sftpUsername: string;
-  let sftpPassword: string;
-
-  try {
-   // Retrieve credentials from the 'SFTP' connection
-   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
-
-   if (!sftpCred?.accessToken) {
-    throw new Error('No SFTP credentials found in connection configuration');
-   }
-
-   // Decode base64 accessToken to get "username:password"
-   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
-
-   // Split on ':' to extract username and password
-   const parts = rawBasicAuth.split(':');
-
-   if (parts.length !== 2) {
-    throw new Error('Invalid SFTP credential format - expected username:password');
-   }
-
-   sftpUsername = parts[0];
-   sftpPassword = parts[1];
-
-   log.info('SFTP credentials retrieved successfully', {
-    hasUsername: !!sftpUsername,
-    hasPassword: !!sftpPassword,
-    usernameLength: sftpUsername.length,
-    passwordLength: sftpPassword.length,
-   });
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   const errorStack = error instanceof Error ? error.stack : undefined;
-   log.error('Failed to retrieve SFTP credentials', {
-    message: errorMessage,
-    stack: errorStack,
-    errorType: error instanceof Error ? error.constructor.name : 'Error'
-   });
-
-   return {
-    success: false,
-    error: 'Failed to retrieve SFTP credentials from connection configuration',
-    details: errorMessage,
-    recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
-   } as ProductIngestionResult;
-  }
-
-  // Get SFTP configuration from activation variables
-  const sftpConfig = {
-   host: activation.getVariable('sftpHost'),
-   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
-   sftpUsername, // From connection (secure)
-   sftpPassword, // From connection (secure)
-   privateKey: activation.getVariable('sftpPrivateKey'),
-   incomingPath: activation.getVariable('sftpIncomingPath') || '/products/incoming',
-   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
-   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
-   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.parquet'
-  };
-
-  // Validate SFTP config
-  if (!sftpConfig.host) {
-   throw new Error('SFTP configuration incomplete: missing host');
-  }
-
-  if (!sftpConfig.sftpPassword && !sftpConfig.privateKey) {
-   throw new Error('SFTP configuration incomplete: missing password or privateKey');
-  }
-
-  // Initialize SFTP data source
-  // ✅ VERSORI PLATFORM: Pass native log from context
-  // ✅ PARQUET: No encoding needed (binary format)
-  sftp = new SftpDataSource({
-   type: 'SFTP_PARQUET',
-   connectionId: 'sftp-product-ingestion',
-   name: 'Product Ingestion SFTP',
-   settings: {
-    host: sftpConfig.host,
-    port: sftpConfig.port,
-    username: sftpConfig.sftpUsername,
-    password: sftpConfig.sftpPassword,
-    privateKey: sftpConfig.privateKey,
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern,
-   }
-  }, log);
-
-  try {
-   // Validate SFTP connection
-   await sftp.validateConnection();
-   log.info('SFTP connection validated');
-
-   // Ensure archive directories exist (recursive)
-   await sftp.createDirectory(sftpConfig.archivePath, true);
-   await sftp.createDirectory(sftpConfig.errorPath, true);
-
-   // 
-   // STEP 3: Discover Files on SFTP
-   // 
-   log.info('🔍 [STEP 3/8] Discovering files on SFTP', {
-    jobId,
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern
-   });
-
-   await tracker.updateJob(jobId, {
-    status: 'processing',
-    stage: 'file_discovery',
-    message: 'Discovering files on SFTP'
-   });
-
-   // List files from SFTP
-   const allFiles = await sftp.listFiles({
-    remotePath: sftpConfig.incomingPath,
-    filePattern: sftpConfig.filePattern
-   });
-
-   log.info(`Discovered ${allFiles.length} files matching pattern`);
-
-   // Apply maxFiles limit if specified
-   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
-
-   if (filesToProcess.length === 0) {
-    log.info('No files to process');
-
-    await tracker.markCompleted(jobId, {
-     filesProcessed: 0,
-     message: 'No files found to process'
-    });
-
-  return {
-     success: true,
-     jobId,
-    filesProcessed: 0,
-    filesFailed: 0,
-    recordsProcessed: 0,
-    eventsSent: 0,
-    eventsFailed: 0,
-     fileResults: []
-    };
-   }
-
-   log.info(`Processing ${filesToProcess.length} files`);
-
-   // Initialize services
-   // ✅ PARQUET: Use ParquetParserService instead of CSVParserService
-   const parquetParser = new ParquetParserService(log);
-   const mapper = new UniversalMapper(mappingConfig);
-   
-   // Get event configuration
-   const eventConfig: EventConfig = {
-    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
-    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
-    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
-    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
-   };
-
-   // Get event sending configuration
-   const eventConcurrency = Math.max(
-    1,
-    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
-   );
-   // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
-   // concurrency: 1 = sequential, concurrency > 1 = parallel
-
-   log.info(`Event concurrency: ${eventConcurrency}`, {
-    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
-    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
-   });
-
-   // Initialize class-based services
-   const fileProcessor = new ProductFileProcessorService(
-    sftp,
-    parquetParser,
-    mapper,
-    eventConfig.catalogueRef
-   );
-   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
-   const eventSender = new EventSenderService(client, log);
-   const eventLogger = new EventLoggerService(sftp);
-
-   // Use direct KV with dot-separated keys (NATS KV safe)
-   const processedKeyBase = 'fc_sdk.processed_files';
-
-   // 
-   // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
-   // 
-
-   // Get SFTP path configuration
-   const requireAbsolutePaths = activation.getVariable('requireAbsolutePaths') === 'true';
-
-   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
-    const file = filesToProcess[fileIndex];
-    const fileStartTime = Date.now();
-
-    // Use full remote path from SFTP listing and derive basename + KV-safe key
-    const remoteFilePath = (file as any).path || file.name;
-    const baseName = getBaseName(remoteFilePath);
-    const safeKey = baseName.replace(/[^A-Za-z0-9._-]/g, '_');
-
-    log.info(`[FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`);
-
-    try {
-     // ✅ File tracking: Skip already processed files (configurable)
-     // Set enableFileTracking=true (default) to skip duplicates
-     // Set forceReprocess=true to bypass tracking and reprocess all files
-     const enableFileTracking = activation.getVariable('enableFileTracking') !== 'false';
-
-     if (enableFileTracking && !forceReprocess) {
-      const processedEntry = await kv.get(`${processedKeyBase}.${safeKey}`);
-      if (processedEntry) {
-       log.info(`⏭️ Skipping already processed file: ${baseName}`);
-       fileResults.push({
-        fileName: baseName,
-        success: true,
-        skipped: true,
-        recordsProcessed: 0,
-        eventsSent: 0,
-        eventsFailed: 0,
-        duration: 0,
-       });
-       continue;
-      }
-     }
-
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'downloading',
-      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
-     });
-
-     // ═══════════════════════════════════════════════════════════
-     // STEP 4: Process File (Download + Parse + Map)
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📦 [STEP 4/8] Processing file: ${baseName}`);
-
-     // Process file: download → parse → transform
-     const fileResult = await fileProcessor.downloadParseAndTransform(remoteFilePath);
-
-     if (!fileResult.success || fileResult.products.length === 0) {
-      // Move failed file to errors and record result
-      const archivedName = timestampedName(baseName);
-      await sftp.moveFile(
-     remoteFilePath,
-       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
-      );
-
-      fileResults.push({
-       fileName: baseName,
-       success: false,
-       recordsProcessed: fileResult.products.length,
-       eventsSent: 0,
-       eventsFailed: 0,
-       duration: Date.now() - fileStartTime,
-       error: fileResult.error || 'Processing failed'
-      });
-
-      continue;
-     }
-
-     // ═══════════════════════════════════════════════════════════
-     // STEP 5: Send Events
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
-
-     // ? Enhanced: Extract context for progress logging
-     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
-     const eventType = eventConfig.eventType || 'UPSERT_PRODUCT';
-
-     // ? Enhanced: Start logging with context
-     log.info(`[EventSender] Sending events for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventType,
-      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
-      sampleProductRefs: sampleProductRefs.join(', '),
-      eventMode: eventConfig.eventMode || 'async'
-     });
-
-     const eventResult = await eventSender.sendEvents(
-      fileResult.products,
-      eventConfig,
-      eventConcurrency
-     );
-
-     const eventsSent = eventResult.eventsSent;
-     const eventsFailed = eventResult.eventsFailed;
-
-     log.info(`Events sent for ${baseName}`, {
-      successful: eventsSent,
-      failed: eventsFailed
-     });
-
-     // ? Enhanced: Completion logging with summary
-     log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
-      eventType
-     });
-
-     // ═══════════════════════════════════════════════════════════
-     // STEP 6: Archive File & Track State
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📁 [STEP 6/8] Archiving file: ${baseName}`);
-
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'archiving',
-      message: `Archiving ${baseName}`
-     });
-
-     // Conditionally archive: errors → errorPath, else → archivePath (timestamped)
-     const archivedName = timestampedName(baseName);
-     const targetDir = eventsFailed > 0 ? sftpConfig.errorPath : sftpConfig.archivePath;
-     await sftp.moveFile(
-      remoteFilePath,
-      joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
-     );
-
-     // Mark as processed in KV (dot-separated key; NATS KV safe)
-     await kv.set(`${processedKeyBase}.${safeKey}`, {
-      recordCount: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      processedAt: new Date().toISOString()
-     });
-
-     log.info(`File archived successfully: ${baseName}`);
-
-     // Optional: Write event log for audit/debugging
-     if (eventsFailed > 0) {
-      const logPath = joinSftpPath(
-       requireAbsolutePaths,
-       sftpConfig.errorPath,
-       `${baseName.replace(/\.parquet$/i, '')}-event-log.json`
-      );
-      await eventLogger.writeEventLog(logPath, {
-       fileName: baseName,
-       totalRecords: fileResult.products.length,
-       eventsSent,
-       eventsFailed,
-       errors: eventResult.errors,
-       processedAt: new Date().toISOString()
-      });
-     }
-
-     // Store file result
-     fileResults.push({
-      fileName: baseName,
-     success: true,
-      recordsProcessed: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      duration: Date.now() - fileStartTime
-     });
-
-    } catch (error: unknown) {
-     // ✅ Enhanced error logging: Extract all error details for visibility
-     const errorMessage = error instanceof Error ? error.message : String(error);
-     const errorDetails = {
-      message: errorMessage,
-      stack: error instanceof Error ? error.stack : undefined,
-      fileName: (error as any)?.fileName,
-      lineNumber: (error as any)?.lineNumber,
-      originalError: (error as any)?.context?.originalError?.message,
-      errorType: error instanceof Error ? error.name : 'Error',
-     };
-     log.error(`Error processing file ${baseName}:`, errorDetails);
-
-     // Best-effort: move to error archive on unexpected failure
-     try {
-      const archivedName = timestampedName(baseName);
-      await sftp.moveFile(
-       remoteFilePath,
-       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
-      );
-     } catch (moveError: unknown) {
-      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
-      log.error('Could not archive failed file', {
-       file: baseName,
-       message: moveErrorMessage,
-       stack: moveError instanceof Error ? moveError.stack : undefined,
-       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
-      });
-     }
-
-     fileResults.push({
-      fileName: baseName,
-      success: false,
-      recordsProcessed: 0,
-      eventsSent: 0,
-      eventsFailed: 0,
-      duration: Date.now() - fileStartTime,
-      error: errorMessage
-     });
-    }
-   }
-
-   // 
-   // STEP 8: Complete Job & Calculate Totals
-   // 
-   log.info('✅ [STEP 8/8] Completing job and calculating totals', { jobId });
-
-   const filesProcessed = fileResults.filter(r => r.success).length;
-   const filesFailed = fileResults.filter(r => !r.success).length;
-   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
-   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
-   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
-
-
-   await tracker.markCompleted(jobId, {
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: Date.now() - startTime
-   });
-
-   const duration = Date.now() - startTime;
-   log.info('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
-   log.info('✅ INGESTION WORKFLOW COMPLETED SUCCESSFULLY');
-   log.info('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
-   log.info('📊 Final Summary:', {
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: `${duration}ms`
-   });
-
-   return {
-    success: true,
-    jobId,
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    fileResults
-   };
-
-  } finally {
-   // ❌š ï¸ CRITICAL: Always dispose SFTP connection
-   if (sftp) {
-    await sftp.dispose();
-    log.info('SFTP connection disposed');
-   }
-  }
-
- } catch (error: unknown) {
-  const duration = Date.now() - startTime;
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  const errorStack = error instanceof Error ? error.stack : undefined;
-
-  log.error('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
-  log.error('❌ INGESTION WORKFLOW FAILED');
-  log.error('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
-  log.error('💥 Error Details:', {
-   jobId,
-   message: errorMessage,
-   stack: errorStack,
-   errorType: error instanceof Error ? error.constructor.name : 'Error',
-   duration: `${duration}ms`
-  });
-
-  // Provide actionable recommendations based on error type
-  if (errorMessage.includes('SFTP') || errorMessage.includes('connection')) {
-   log.error('💡 Recommendation: Check SFTP credentials and network connectivity');
-  } else if (errorMessage.includes('parse') || errorMessage.includes('Parquet')) {
-   log.error('💡 Recommendation: Verify Parquet file format and schema compatibility');
-  } else if (errorMessage.includes('Event API') || errorMessage.includes('401')) {
-   log.error('💡 Recommendation: Verify Fluent Commerce credentials and retailerId');
-  } else if (errorMessage.includes('KV') || errorMessage.includes('storage')) {
-   log.error('💡 Recommendation: Check Versori KV store connectivity');
-  } else {
-   log.error('💡 Recommendation: Review error stack trace and activation variables');
-  }
-
-  // Try to mark job as failed, but don't let tracking errors mask workflow error
-  try {
-   await tracker.markFailed(jobId, error);
-  } catch (trackingError: unknown) {
-   const trackingErrorMessage =
-    trackingError instanceof Error ? trackingError.message : String(trackingError);
-   log.warn('Failed to mark job as failed in tracker', {
-    jobId,
-    trackingError: trackingErrorMessage,
-   });
-  }
-
-  // Determine recommendation based on error type
-  let recommendation = 'Review error details and check activation variables';
-  if (errorMessage.includes('SFTP') || errorMessage.includes('connection')) {
-   recommendation = 'Verify SFTP credentials, host, port, and network connectivity. Check if SFTP server is accessible.';
-  } else if (errorMessage.includes('parse') || errorMessage.includes('Parquet')) {
-   recommendation = 'Verify Parquet file format, schema compatibility, and file integrity. Check if file is corrupted.';
-  } else if (errorMessage.includes('Event API') || errorMessage.includes('401')) {
-   recommendation = 'Verify Fluent Commerce credentials (clientId, clientSecret) and retailerId configuration.';
-  } else if (errorMessage.includes('KV') || errorMessage.includes('storage')) {
-   recommendation = 'Check Versori KV store connectivity and permissions. Verify namespace configuration.';
-  }
-
-  return {
-   success: false,
-   jobId,
-   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
-   filesFailed: fileResults.filter(r => !r.success).length,
-   filesSkipped: fileResults.filter(r => r.skipped).length,
-   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
-   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
-   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
-   fileResults,
-   error: errorMessage,
-   recommendation,
-   duration: `${duration}ms`
-  };
- } finally {
-  // ⚠️ CRITICAL: Ensure SFTP is disposed even if outer error occurs
-  // This outer finally ensures disposal if error happens after SFTP creation
-  // but before inner try block (e.g., during connection validation)
-  if (sftp) {
-   try {
-    await sftp.dispose();
-    log.info('SFTP connection disposed (outer finally)');
-   } catch (disposeError: unknown) {
-    const disposeErrorMessage =
-     disposeError instanceof Error ? disposeError.message : String(disposeError);
-    log.warn('Error disposing SFTP in outer finally', { error: disposeErrorMessage });
-   }
-  }
- }
-}
-```
-
----
-
-## 8. Utility Functions (src/utils/)
-
-### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
-
-```typescript
-/**
- * SFTP Path Utilities
- *
- * Helper functions for SFTP path operations with AWS Transfer Family support.
- * Handles both absolute paths (AWS Transfer Family) and relative paths (standard OpenSSH).
- */
-
-/**
- * Join SFTP path segments safely
- *
- * Handles different SFTP server path requirements:
- * - AWS Transfer Family: Requires absolute paths (leading /)
- * - Standard OpenSSH: Supports relative paths
- *
- * @param requireAbsolutePath - true for AWS Transfer Family, false for standard OpenSSH
- * @param parts - Path segments to join
- * @returns Properly formatted SFTP path
- *
- * @example
- * ```typescript
- * // AWS Transfer Family (absolute paths)
- * joinSftpPath(true, 'products', 'processed', 'file.parquet')
- * // Returns: '/products/processed/file.parquet'
- *
- * // Standard OpenSSH (relative paths)
- * joinSftpPath(false, 'products', 'processed', 'file.parquet')
- * // Returns: 'products/processed/file.parquet'
- * ```
- */
-export function joinSftpPath(requireAbsolutePath: boolean, ...parts: string[]): string {
- // Clean each segment (remove leading/trailing slashes)
- const cleaned = parts
-  .filter(Boolean)
-  .map(p => String(p).replace(/^\/+|\/+$/g, ''))
-  .join('/');
-
- // Add leading slash if required (AWS Transfer Family)
- return requireAbsolutePath && !cleaned.startsWith('/') ? `/${cleaned}` : cleaned;
-}
-
-/**
- * Generate timestamped filename for archival
- *
- * Adds ISO timestamp to filename before extension for unique archival.
- * Handles Parquet files specifically but can be adapted for other formats.
- *
- * @param name - Original filename
- * @returns Filename with timestamp appended
- *
- * @example
- * ```typescript
- * timestampedName('products.parquet')
- * // Returns: 'products-2025-11-01T18-30-45-123Z.parquet'
- * ```
- */
-export function timestampedName(name: string): string {
- const ts = new Date().toISOString().replace(/[:.]/g, '-');
- const base = name.replace(/\.parquet$/i, '');
- return `${base}-${ts}.parquet`;
-}
-
-/**
- * Extract base filename from full SFTP path
- *
- * @param filePath - Full SFTP path (e.g., '/products/incoming/file.parquet')
- * @returns Base filename only (e.g., 'file.parquet')
- *
- * @example
- * ```typescript
- * getBaseName('/products/incoming/products_20250101.parquet')
- * // Returns: 'products_20250101.parquet'
- * ```
- */
-export function getBaseName(filePath: string): string {
- return filePath.split('/').pop() || filePath;
-}
-```
-
----
-
-### Job ID Generator (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking ingestion workflows
- *
- * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
- * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
- *
- * NAMING: generate{Entity}JobId or generateJobId (generic)
- */
-
-/**
- * Generate unique job ID
- *
- * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
- * @param entity - Entity abbreviation (VP, IP, ORD, PRD)
- * @returns Unique job ID string
- */
-export function generateJobId(type: string, entity: string): string {
- const now = new Date();
-
- // Format: YYYYMMDD
- const dateStr = now.toISOString().slice(0, 10).replace(/-/g, "");
-
- // Format: HHMMSS
- const timeStr = now.toISOString().slice(11, 19).replace(/:/g, "");
-
- // Random suffix (6 chars)
- const randomStr = Math.random().toString(36).substring(2, 8);
-
- return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
-}
-
-/**
- * Parse job ID components
- *
- * @param jobId - Job ID to parse
- * @returns Parsed components or null if invalid
- */
-export function parseJobId(
- jobId: string
-): {
- type: string;
- entity: string;
- date: string;
- time: string;
- random: string;
-} | null {
- const parts = jobId.split("_");
-
- if (parts.length !== 5) {
-  return null;
- }
-
- return {
-  type: parts[0],
-  entity: parts[1],
-  date: parts[2],
-  time: parts[3],
-  random: parts[4],
- };
-}
-```
-
----
-
-### 9. Mapping (`config/products.import.parquet.json`)
-
-```json
-{
- "name": "products.import.parquet",
- "version": "1.2.0",
- "description": "Parquet → Product Event Mapping",
- "fields": {
-  "ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
-  "type": { "source": "type", "resolver": "sdk.uppercase", "defaultValue": "STANDARD" },
-  "status": { "source": "status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" },
-  "gtin": { "source": "gtin" },
-  "name": { "source": "name", "required": true, "resolver": "sdk.trim" },
-  "categoryRefs": { "source": "category_refs", "resolver": "custom.splitByPipe" },
-  "price": { "resolver": "custom.buildPriceArray" },
-  "taxType": { "resolver": "custom.buildTaxType" },
-  "attributes": { "defaultValue": [] }
- }
-}
-```
-
----
-
-### 10. Package Configuration
-
-#### `package.json`
-
-```json
-{
- "name": "sftp-parquet-product-event",
- "version": "1.2.0",
- "type": "module",
- "main": "src/index.ts",
- "scripts": {
-  "dev": "versori dev",
-  "build": "versori build",
-  "deploy": "versori deploy"
- },
- "dependencies": {
-  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-  "@versori/run": "latest"
- },
- "devDependencies": {
-  "@types/node": "^20.0.0",
-  "typescript": "^5.0.0"
- }
-}
-```
-
-#### `tsconfig.json`
-
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
-
----
-
-## Event Sending Configuration
-
-**Simple Configuration:** One variable controls everything
-
-```json
-{
- "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
-}
-```
-
-**How it works:**
-
-- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
-- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
-- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
-- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
-
-**Configuration Guidelines:**
-
-- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
-- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
-- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
-- **Note:** Fluent API supports concurrent requests - adjust based on your needs
-
-**Why Single Variable?**
-
-- **Simpler:** One variable instead of two (`mode` + `concurrency`)
-- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
-- **Less config:** Fewer activation variables to manage
-- **Flexible:** Easy to tune performance (just change the number)
-
----
-
-## Production Implementation
-
-### Project Structure
-
-**Production-Ready Modular Structure:**
-
-```
-sftp-parquet-product-event/
-├── package.json
-├── tsconfig.json
-├── index.ts                 # Workflow entry point (exports workflows)
-└── src/
-  ├── workflows/
-  │  └── product-ingestion.ts       # Main orchestrator (coordinates services)
-  ├── services/
-  │  ├── product-file-processor.service.ts # Download, parse, transform Parquet
-  │  ├── event-sender.service.ts      # Send events to Fluent API
-  │  ├── event-logger.service.ts      # Write event logs to SFTP
-  │  └── product-ingestion.service.ts   # Main orchestration logic
-  ├── types/
-  │  └── product-ingestion.types.ts    # TypeScript interfaces
-  ├── utils/
-  │  ├── sftp-path.utils.ts        # SFTP path helpers
-  │  └── job-id-generator.ts        # Job ID utilities
-  └── config/
-    └── products.import.parquet.json   # Mapping configuration
-```
-
-**Benefits of This Modular Structure:**
-
-- ✅ **Separation of Concerns**: Each service has one clear responsibility
-- ✅ **Reusability**: Services can be imported and used in other workflows
-- ✅ **Testability**: Easy to write unit tests for individual services
-- ✅ **Maintainability**: Changes isolated to specific service files
-- ✅ **Type Safety**: Centralized type definitions prevent duplication
-- ✅ **Scalability**: Easy to add new services without modifying existing code
-- ✅ **Gold Standard Compliant**: 100% compliance with Event API checklist
-
-**Key Services:**
-
-1. **ProductFileProcessorService**: Downloads Parquet from SFTP, parses with `ParquetParserService`, transforms with `UniversalMapper`
-2. **EventSenderService**: Sends events to Fluent API with configurable concurrency (sequential or parallel)
-3. **EventLoggerService**: Writes event processing logs to SFTP for audit/debugging
-4. **ProductIngestionService**: Main orchestration - coordinates all services through 8-step workflow
-
----
-
-## 6. Deployment Instructions
-
-### Deploy to Versori
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Test locally (if using Versori CLI)
-npm run dev
-
-# 3. Deploy to Versori platform
-npm run deploy
-```
-
-### Configure Activation Variables
-
-In Versori platform settings, configure all variables listed in the Activation Variables section above.
-
----
-
-## 7. Testing
-
-### Test Scheduled Ingestion
-
-Upload a test Parquet file to SFTP incoming directory and wait for the scheduled run.
-
-**Check logs:**
-
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and SFTP
-[STEP 3/8] Discovering files on SFTP
-[FILE 1/1] Processing file: products_20250124.parquet
-[STEP 4/8] Downloading and parsing: products_20250124.parquet
-[STEP 5/8] Transforming 5 products from products_20250124.parquet
-[STEP 6/8] Sending 5 events to Fluent Commerce
-[STEP 7/8] Archiving file: products_20250124.parquet
-[STEP 8/8] Completing job and calculating totals
-```
-
-### Test Ad hoc Ingestion
-
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "Content-Type: application/json" \
- -d '{}'
-
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "Content-Type: application/json" \
- -d '{
-  "filePattern": "urgent_*.parquet", }'
-```
-
-### Test Job Status Query
-
-```bash
-curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
- -H "Content-Type: application/json" \
- -d '{
-  "jobId": "ADHOC_PROD_20251024_183045_abc123"
- }'
-```
-
----
-
-## Monitoring
-
-### Success Response
-
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 50,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.parquet",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 50,
-   "eventsFailed": 0
-  }
- ],
- "duration": 12345
-}
-```
-
-### Partial Success Response
-
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 45,
- "eventsFailed": 5,
- "results": [
-  {
-   "file": "products_2025-01-22.parquet",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 45,
-   "eventsFailed": 5,
-   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
-  }
- ],
- "duration": 12345
-}
-```
-
-### Error Response
-
-```json
-{
- "success": false,
- "filesProcessed": 0,
- "filesFailed": 1,
- "totalRecords": 0,
- "eventsSent": 0,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.parquet",
-   "success": false,
-   "error": "Parquet parse error: Invalid structure"
-  }
- ],
- "duration": 876
-}
-```
-
-### Monitoring Metrics
-
-Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
-
-- **Files Processed** - Total files successfully processed
-- **Events Sent** - Total events sent to Fluent Commerce
-- **Events Failed** - Events that failed (check rejection reports)
-- **Processing Duration** - Time taken for complete workflow
-- **Rate Limiting** - Watch for 429 errors indicating throttling
-
-Use the status webhook for dashboards and automated monitoring.
-
----
-
-- **No files found:** Check `sftpIncomingPath` and `filePattern` (names only)
-- **Parquet parse failed:** Verify Parquet schema compatibility and file corruption
-- **Buffer download issues:** Ensure SFTP doesn't apply text transformations to binary files
-- **High event failures:** Inspect rejection report; consider Batch API for very high volumes
-- **429 throttling:** Add small backoff/delay or use Batch API
-- **SFTP connection errors:** Verify credentials, host, port, and `requireAbsolutePaths` setting
-- **Memory issues:** Large Parquet files may need chunked processing (contact support)
-
----
-
-## 9. Key Takeaways
-
-### Core Features
-- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
-- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
-- ✅ **Processing modes** - per-file (default), chunked, batch
-- ✅ **Unified file processor** - Three service functions: `processFile()`, `sendEvents()`, `writeEventLog()`
-- ✅ **Per-file workflow** - Download → Parse → Map → Send Events → Write Log → Archive
-- ✅ **JobTracker** - Track job lifecycle with KV persistence
-- ✅ **VersoriFileTracker** - Prevent duplicate file processing
-- ✅ **SFTP dispose()** - Always cleanup in finally block
-- ✅ **Buffer import required** - `import { Buffer } from 'node:buffer'` for Deno/Versori
-- ✅ **Binary download** - Parquet requires Buffer format, not string
-- ✅ **No normalization** - Parser returns array directly
-- ✅ **Safe path join** - Handle AWS Transfer Family vs standard OpenSSH
-- ✅ **Per-record error handling** - Continue on individual failures
-- ✅ **Event logs to SFTP** - Write rejection reports for failed events using `Buffer.from()`
-- ✅ **Externalized mapping** - Use JSON config file for field mappings
-- ✅ **File-level error handling** - Don't stop on single file failure
-
-### Production Code Improvements (Applied)
-1. ✅ **Fixed index.ts path** - Uses `./src/workflows/product-ingestion` (correct relative path)
-2. ✅ **Emoji logging** - All workflow steps use emojis (⏰, 🔧, 🔍, 📊, 🔌, 📦, 📤, 📁, ✅, ❌)
-3. ✅ **Execution boundaries** - Clear start/end markers with decorative lines
-4. ✅ **validateConnection: true** - Added optional `validateConnectionOnStart` variable (fail-fast mode)
-5. ✅ **enableFileTracking toggle** - Configurable file tracking via activation variable
-6. ✅ **extractFileName usage** - Uses `getBaseName()` utility to extract filename from path
-7. ✅ **SFTP variable declaration** - Proper `let sftpUsername: string; let sftpPassword: string;` declarations
-8. ✅ **Duration tracking** - All workflows track and log execution duration in milliseconds
-9. ✅ **Error recommendations** - Context-aware error recommendations based on error type
-10. ✅ **Documented new variables** - Added `validateConnectionOnStart` and `enableFileTracking` to activation variables table
-
----
-
-[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)
+---
+template_id: tpl-ingest-sftp-parquet-to-product-event
+canonical_filename: template-ingestion-sftp-parquet-product-event.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: ingestion
+source: sftp-parquet
+destination: fluent-event-api
+entity: product
+format: parquet
+logging: versori
+status: stable
+features:
+  - batched-events
+  - attribute-transformation
+  - memory-management
+  - json-serialization-handling
+  - enhanced-logging
+---
+
+# Template: Ingestion - SFTP Parquet to Product Event
+
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
+- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
+- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large Parquet files)
+- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
+- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
+
+---
+
+## 📋 Implementation Prompt
+
+```
+I need a Versori scheduled ingestion that:
+
+1) Discovers Parquet files on SFTP with file tracking to skip duplicates
+2) Downloads binary Parquet files as Buffer and parses with ParquetParserService
+3) Transforms records with UniversalMapper per mapping JSON (no array normalization needed)
+4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
+5) Archives files to processed/ or errors/ and writes optional rejection report
+6) Tracks progress with JobTracker and exposes a job-status webhook
+7) Uses native Versori log from context
+
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+
+---
+
+## 📋 Template Overview
+
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, SFTP connection, schedule, file patterns/limits) are configured via activation variables. Data shape and logic (mapping JSON, Parquet schema, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from SFTP Parquet files, transforms it, and sends events to the Fluent Commerce Event API.
+
+### What This Template Does
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│          INGESTION WORKFLOW              │
+└─────────────────────────────────────────────────────────────────┘
+
+1. TRIGGER
+  ├─ Scheduled (Cron): Runs automatically every hour
+  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
+  └─ Status Query (Webhook): Check job progress
+
+2. DISCOVER FILES (SftpDataSource)
+  ├─ List files from SFTP directory
+  ├─ Filter by pattern (products_*.parquet)
+  ├─ Check file tracking (skip processed)
+  └─ Sort by oldest first
+
+3. DOWNLOAD & PARSE (ParquetParserService)
+  ├─ Download file as Buffer (binary format)
+  ├─ Parse Parquet columnar data
+  ├─ Returns array directly (no normalization needed)
+  └─ Validate Parquet structure
+
+4. TRANSFORM (UniversalMapper)
+  ├─ Map Parquet fields to Fluent schema
+  ├─ Apply SDK resolvers (trim, uppercase, etc.)
+  ├─ Handle nested objects (price, taxType)
+  ├─ Handle arrays (categoryRefs)
+  └─ Collect transformation errors
+
+5. SEND EVENTS (Event API)
+  ├─ Loop through transformed products
+  ├─ Send UPSERT_PRODUCT event (async)
+  ├─ Track success/failure count
+  └─ Continue on individual failures
+
+6. ARCHIVE (SftpDataSource)
+  ├─ Move file to processed/ folder
+  ├─ Or move to errors/ if validation failed
+  ├─ Generate timestamped archive name
+  └─ Verify archive success
+
+7. TRACK STATE (VersoriFileTracker)
+  ├─ Mark file as processed
+  ├─ Store processing metadata
+  ├─ Prevent duplicate processing
+  └─ Track record counts
+
+8. TRACK JOB (JobTracker)
+  ├─ Update job status at each step
+  ├─ Store final result in KV
+  ├─ Enable status queries via webhook
+  └─ Handle errors gracefully
+```
+
+### Key Features
+
+- Job tracking with status queries
+- Execution modes: scheduled, ad hoc, status query
+- Uses SftpDataSource, ParquetParserService, UniversalMapper, VersoriFileTracker, JobTracker
+- Error handling, retry logic, and SFTP cleanup
+- File tracking: VersoriFileTracker prevents duplicates; `forceReprocess` bypasses
+- Event API: Per-record failures don't block other records; rejection reports written to `errors/`
+- SFTP dispose() in finally block
+- Binary format handling (Buffer download, not string)
+
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+
+### 📦 Package Information
+
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk@latest
+```
+
+**Note:** Always use the latest SDK version for bug fixes and new features.
+
+---
+
+**Templates are designed for direct deployment; customize via activation variables.**
+
+---
+
+## 📦 SDK Imports (Verified - Versori Optimized)
+
+```typescript
+// ✅ VERIFIED IMPORTS - These match actual SDK exports
+import { Buffer } from 'node:buffer';
+import {
+ createClient, // Universal client factory
+ SftpDataSource, // SFTP operations with connection pooling
+ ParquetParserService, // Parquet binary parsing
+ UniversalMapper, // Field mapping with SDK resolvers
+ VersoriFileTracker, // File processing state tracker
+ JobTracker, // Job status tracking
+} from '@fluentcommerce/fc-connect-sdk';
+
+import type { FluentClient, SftpDataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
+
+// Versori platform imports
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+
+**✅ VERSORI PLATFORM - Use Native Logs:**
+
+- Use `log` from context: `const { log } = ctx;`
+- Don't import LoggingService, StructuredLogger for Versori connectors
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+
+---
+
+## ⚙️ Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+> **Note:** `sftpUsername` and `sftpPassword` are fetched from the `SFTP` Basic Auth connection (see SFTP Connection Setup above).
+
+```bash
+# SFTP Configuration
+SFTP_HOST=sftp.partner.com
+SFTP_PORT=22
+
+# SFTP Paths
+SFTP_REMOTE_PATH=/products/incoming
+SFTP_ARCHIVE_PATH=/products/processed
+SFTP_ERROR_PATH=/products/errors
+
+# File Processing
+FILE_PATTERN=products_*.parquet
+MAX_FILES_PER_RUN=10
+
+# Product Configuration
+CATALOGUE_REF=PC:MASTER:2
+CATALOGUE_TYPE=MASTER
+
+# Event API Configuration
+EVENT_NAME=UPSERT_PRODUCT
+EVENT_MODE=async
+EVENT_CONCURRENCY=1
+
+# Fluent Configuration (via Versori connection)
+# Connection: fluent_commerce (OAuth2)
+FLUENT_RETAILER_ID=1
+
+# Event API Performance
+USE_BATCHED_EVENTS=false                       # Use batched UPSERT_PRODUCTS (10x faster for 100+ products)
+MAX_PRODUCTS_UNDER_BATCHED_EVENT=100           # Products per batch (only used when USE_BATCHED_EVENTS=true)
+MEMORY_BATCH_SIZE=250                          # Products per mapping batch (prevents OOM on large files)
+
+# Feature Toggles
+VALIDATE_CONNECTION_ON_START=false            # Validate Fluent connection early (default: false)
+ENABLE_FILE_TRACKING=true                     # Enable/disable file tracking (default: true)
+REQUIRE_ABSOLUTE_PATHS=true                   # "true" for AWS Transfer Family, "false" for standard OpenSSH
+```
+
+**Security:**
+- **SFTP Authentication:** Credentials stored in Versori connection vault (not in activation variables). See [SFTP Connection Setup](#sftp-connection-setup) section below.
+- **Webhook Authentication:** Enforced by Versori connection configuration. Configure your webhook connection with API key authentication in the Versori Dashboard, then reference it in `webhook({ connection: 'webhook-auth' })`.
+
+### Variable Explanations
+
+| Variable        | Purpose             | Default          | Customization Hints         |
+| ----------------------- | -------------------------------- | ------------------------- | ----------------------------------- |
+| `FLUENT_RETAILER_ID`   | Retailer ID for Event API    | -             | Required - Fluent retailer ID   |
+| `EVENT_CONCURRENCY`   | Event sending concurrency    | `1`            | `1` = sequential, `>1` = parallel (3-10 recommended) |
+| `USE_BATCHED_EVENTS` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
+| `MAX_PRODUCTS_UNDER_BATCHED_EVENT` | Products per batch | `100`          | Only used when `USE_BATCHED_EVENTS=true` (default: 100) |
+| `MEMORY_BATCH_SIZE`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
+| `VALIDATE_CONNECTION_ON_START` | Validate Fluent connection early | `false` | `true` = fail-fast mode, `false` = validate on first call |
+| `ENABLE_FILE_TRACKING` | Skip already-processed files | `true` | `true` = use KV tracking, `false` = process all files |
+| `SFTP_HOST`       | SFTP server hostname       | -             | Required - SFTP server address   |
+| `SFTP_PORT`       | SFTP server port         | `22`           | Standard SFTP port         |
+| `SFTP_REMOTE_PATH`   | Incoming files directory     | `/products/incoming`   | Where new files arrive       |
+| `SFTP_ARCHIVE_PATH`   | Processed files archive     | `/products/processed`   | Where files move after success   |
+| `SFTP_ERROR_PATH`     | Failed files directory      | `/products/errors`    | Where files move after errors    |
+| `FILE_PATTERN`      | File name filter         | `products_*.parquet`   | Glob pattern for matching files   |
+| `CATALOGUE_REF`     | Product catalogue reference   | `PC:MASTER:2`       | Target catalogue in Fluent     |
+| `CATALOGUE_TYPE`     | Catalogue type          | `MASTER`         | Usually MASTER or STANDARD     |
+| `EVENT_NAME`       | Event to send          | `UPSERT_PRODUCT`     | Event name from Rubix        |
+| `EVENT_MODE`       | Event processing mode      | `async`          | async (recommended) or sync     |
+| `MAX_FILES_PER_RUN`    | Max files per execution     | `10`           | Prevent timeout on large batches  |
+| `REQUIRE_ABSOLUTE_PATHS` | Require absolute SFTP paths   | `true`          | Recommended for clarity       |
+
+---
+
+## 🔒 SDK Automatic Behaviors (v0.1.40+)
+
+**The SDK automatically validates and retries for improved reliability:**
+
+### retailerId Validation
+- **SDK validates** `retailerId` before calling `sendEvent()`
+- **Checks:** `event.retailerId || client.retailerId`
+- **If missing:** Throws `"retailerId is required for Event API..."`
+- **Configuration:** Set via `FLUENT_RETAILER_ID` activation variable (recommended)
+
+### 401 Auth Retry
+- **Automatic retry** for platform auth failures (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s)
+- **Applies to:** All `sendEvent()` calls (async and sync modes)
+- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
+
+### 5xx Server Retry
+- **Automatic retry** for transient server errors (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
+- **Protects:** Against Fluent API transient failures
+
+### No Code Changes Required
+- All templates remain compatible
+- Retry logic is automatic and transparent
+- Better error messages guide configuration
+
+**See:** [Event API Guide](./event-api-guide.md) for complete details
+
+---
+
+## SFTP Connection Setup
+
+**CRITICAL: Buffer Import for Deno/Versori Runtime**
+
+```typescript
+import { Buffer } from 'node:buffer'; // ← Required for credential decoding!
+```
+
+**Why:** Deno/Versori runtime doesn't have `Buffer` as a global. You'll get "Buffer is not defined" error without this import.
+
+---
+
+### Two Methods for SFTP Credential Access
+
+Versori provides **two methods** to access SFTP credentials securely:
+
+#### Method 1: Basic Auth Connection (Recommended)
+
+**Best Practice:** Store SFTP credentials in a Versori connection object with Basic Auth.
+
+**Setup:**
+1. In Versori platform, create a connection named `SFTP`
+2. Set **Authentication Type**: `Basic Auth`
+3. Enter **Username**: Your SFTP username
+4. Enter **Password**: Your SFTP password
+
+**Access in Code:**
+```typescript
+import { Buffer } from 'node:buffer'; // Required!
+
+// Retrieve credentials from the 'SFTP' connection
+const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+const [sftpUsername, sftpPassword] = rawBasicAuth.split(':');
+```
+
+**Why use this method?**
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection can be reused across workflows
+- ✅ No sensitive data in activation variables
+- ✅ Easier credential rotation
+
+#### Method 2: Connection Variables (Alternative)
+
+**Alternative:** Use Versori connection variables API.
+
+**Setup:**
+1. Create a connection with **any** authentication type
+2. Add custom variables: `sftp_username`, `sftp_password`
+
+**Access in Code:**
+```typescript
+const { connections } = ctx;
+const sftpUsername = connections.getVariable('sftp_username');
+const sftpPassword = connections.getVariable('sftp_password');
+```
+
+**Why use this method?**
+- ✅ Credentials stored securely in Versori vault
+- ✅ Connection can be reused across workflows
+- ✅ No sensitive data in activation variables
+- ✅ Easier credential rotation
+
+---
+
+## 🔧 Production Implementation
+
+### Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+product-event-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── product.types.ts            # Shared type definitions
+```
+
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+
+---
+
+## Workflow Files
+
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+
+All time-based triggers that run automatically on cron schedules.
+
+#### `src/workflows/scheduled/daily-product-sync.ts`
+
+**Purpose**: Automatic Daily product sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Scheduled Workflow: Daily Product Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: product-sync.service.ts
+ */
+export const dailyProductSync = schedule(
+  'product-sync-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv } = ctx;
+    const jobId = `product-sync-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
+    await tracker.updateJob(jobId, { status: 'processing' });
+
+    try {
+      // Reuse shared orchestration logic
+      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
+      await tracker.markCompleted(jobId, result);
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      await tracker.markFailed(jobId, e);
+      return { success: false, jobId, error: e?.message };
+    }
+  })
+);
+```
+
+---
+
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+
+All HTTP-based triggers that create webhook endpoints.
+
+#### `src/workflows/webhook/adhoc-product-sync.ts`
+
+**Purpose**: Manual product sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Webhook: Manual Product Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
+ * Request body (optional): { filePattern: "urgent_*.parquet", maxFiles: 5 }
+ *
+ * Pattern: webhook().then(http()) - needs Fluent API access
+ * Uses shared service: product-sync.service.ts
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const adhocProductSync = webhook('product-sync-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'product-sync-adhoc', // Versori validates API key
+}).then(
+  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, data } = ctx;
+    const jobId = `product-sync-adhoc-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    const filePattern = data?.filePattern as string;
+    const maxFiles = data?.maxFiles as number;
+
+    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+    });
+
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'manual',
+      stage: 'initialization',
+      status: 'queued',
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
+      .then((result) => {
+        log.info('✅ [BACKGROUND] Product sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          filesFailed: result.filesFailed,
+          recordsProcessed: result.recordsProcessed,
+        });
+        return tracker.markCompleted(jobId, result);
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+
+        log.error('❌ [BACKGROUND] Product sync failed', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+        });
+
+        return tracker.markFailed(jobId, errorMessage);
+      });
+
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Product sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+
+#### `src/workflows/webhook/job-status-check.ts`
+
+**Purpose**: Query job status
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
+**Request body**: `{ "jobId": "product-sync-1234567890" }`
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
+ * Request body: { "jobId": "product-sync-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const productSyncJobStatus = webhook('product-sync-job-status', {
+  response: { mode: 'sync' },
+  connection: 'product-sync-job-status',
+}).then(
+  fn('status', async ctx => {
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+
+    if (!jobId) {
+      return { success: false, error: 'jobId required' };
+    }
+
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+
+---
+
+### 3. Entry Point (`index.ts`)
+
+**Purpose**: Register all workflows with Versori platform
+
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * Versori automatically discovers and registers exported workflows
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ */
+
+// Import scheduled workflows
+import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
+
+// Import webhook workflows
+import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
+import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
+
+// Register all workflows
+export {
+  // Scheduled (time-based triggers)
+  dailyProductSync,
+
+  // Webhooks (HTTP-based triggers)
+  adhocProductSync,
+  productSyncJobStatus,
+};
+```
+
+**What Gets Exposed:**
+- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
+- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
+- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
+
+---
+
+### Adding New Workflows
+
+**To add a scheduled workflow:**
+1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**To add a webhook workflow:**
+1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**Example - Adding hourly delta sync:**
+
+```typescript
+// src/workflows/scheduled/hourly-delta-sync.ts
+export const hourlyDeltaSync = schedule(
+  'product-delta-hourly',
+  '0 * * * *' // Every hour
+).then(
+  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
+    // Delta sync logic (skip BPP)
+    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
+    return result;
+  })
+);
+
+// index.ts (add to imports and exports)
+import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
+export { daily_product_sync, hourlyDeltaSync, ... };
+```
+
+---
+## 🔍 Complete Production Code
+
+### 1. Entry Point (index.ts)
+
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * This file registers three workflows:
+ * 1. Scheduled ingestion (runs automatically)
+ * 2. Ad hoc webhook (manual trigger)
+ * 3. Job status webhook (query progress)
+ */
+
+import {
+ scheduledProductIngestion,
+ adhocProductIngestion,
+ productIngestionJobStatus,
+} from "./src/workflows/product-ingestion";
+
+// Register workflows with Versori platform
+export {
+ scheduledProductIngestion, // Cron-based auto-run
+ adhocProductIngestion, // Manual webhook trigger
+ productIngestionJobStatus, // Job status query
+};
+```
+
+---
+
+### 2. Workflows (src/workflows/product-ingestion.ts)
+
+```typescript
+/**
+ * Workflows - Defines 3 execution patterns for product ingestion
+ *
+ * WORKFLOW 1: Scheduled (Cron) - Runs automatically every hour
+ * WORKFLOW 2: Ad hoc (Webhook) - Manual trigger for immediate processing
+ * WORKFLOW 3: Job Status (Webhook) - Query job progress
+ */
+
+import { schedule, webhook, http, fn } from '@versori/run';
+import { executeProductIngestion, getJobStatus } from '../services/product-ingestion.service';
+import { generateJobId } from '../utils/job-id-generator';
+
+/**
+ * WORKFLOW 1: Scheduled Ingestion
+ *
+ * Purpose: Automated hourly product file processing
+ * Trigger: Cron schedule (every hour at minute 0)
+ * File Tracking: Uses VersoriFileTracker to skip processed files
+ */
+export const scheduledProductIngestion = schedule(
+ 'product-ingestion-scheduled',
+ '0 * * * *' // → CUSTOMIZE: Cron expression
+)
+.then(
+ http('execute-scheduled-ingestion', { connection: 'fluent_commerce' }, async (ctx) => {
+  const { log } = ctx;
+  const startTime = Date.now();
+
+  // Generate unique job ID for tracking
+  const jobId = generateJobId('SCHEDULED', 'PROD');
+
+  log.info('⏰ Scheduled ingestion triggered', { jobId });
+
+  try {
+   const result = await executeProductIngestion(ctx, {
+    jobId,
+    triggeredBy: 'schedule',
+   });
+
+   const duration = Date.now() - startTime;
+   log.info('✅ Scheduled ingestion completed', {
+    jobId,
+    filesProcessed: result.filesProcessed,
+    recordsProcessed: result.recordsProcessed,
+    duration: `${duration}ms`
+   });
+
+   return result;
+
+  } catch (error: unknown) {
+   const duration = Date.now() - startTime;
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   const errorStack = error instanceof Error ? error.stack : undefined;
+   log.error('❌ Scheduled ingestion failed', {
+    jobId,
+    message: errorMessage,
+    stack: errorStack,
+    errorType: error instanceof Error ? error.constructor.name : 'Error',
+    duration: `${duration}ms`,
+    recommendation: 'Check SFTP connection settings and credentials'
+   });
+   throw error;
+  }
+ })
+);
+
+/**
+ * WORKFLOW 2: Ad hoc Ingestion (Manual Trigger)
+ *
+ * Purpose: Manual file processing with optional filters
+ * Trigger: Webhook POST to /webhooks/product-ingestion-adhoc
+ */
+export const adhocProductIngestion = webhook('product-ingestion-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+})
+.then(
+ http('execute-adhoc-ingestion', { connection: 'fluent_commerce' }, async (ctx) => {
+  const { data, log } = ctx;
+  const startTime = Date.now();
+
+  const jobId = generateJobId('ADHOC', 'PROD');
+
+  const filePattern = data.filePattern as string | undefined;
+  const maxFiles = data.maxFiles as number | undefined;
+  const forceReprocess = data.forceReprocess as boolean | undefined;
+
+  log.info('🚀 [WEBHOOK] Adhoc product ingestion triggered', {
+   jobId,
+   filePattern: filePattern || 'default',
+   maxFiles: maxFiles || 'unlimited',
+   forceReprocess: !!forceReprocess
+  });
+
+  // ✅ Fire-and-forget: Start background processing WITHOUT await
+  // The promise continues execution after we return the response
+  executeProductIngestion(ctx, {
+    jobId,
+    triggeredBy: 'webhook',
+    filePattern,
+    maxFiles,
+    forceReprocess,
+  })
+    .then((result) => {
+      const duration = Date.now() - startTime;
+      log.info('✅ [BACKGROUND] Product ingestion completed successfully', {
+        jobId,
+        filesProcessed: result.filesProcessed,
+        recordsProcessed: result.recordsProcessed,
+        filesFailed: result.filesFailed,
+        duration: `${duration}ms`
+      });
+    })
+    .catch((error: unknown) => {
+      const duration = Date.now() - startTime;
+      const errorMessage = error instanceof Error ? error.message : String(error);
+      const errorStack = error instanceof Error ? error.stack : undefined;
+      log.error('❌ [BACKGROUND] Product ingestion failed', {
+        jobId,
+        message: errorMessage,
+        stack: errorStack,
+        errorType: error instanceof Error ? error.constructor.name : 'Error',
+        duration: `${duration}ms`,
+        recommendation: 'Verify webhook payload and SFTP access'
+      });
+    });
+
+  // Return immediately with jobId (response sent with this return value)
+  return {
+    success: true,
+    jobId,
+    message: 'Product ingestion started in background',
+    statusEndpoint: `https://{workspace}.versori.run/product-ingestion-job-status`,
+    note: 'Poll the status endpoint with jobId to check progress',
+  };
+ })
+);
+
+/**
+ * WORKFLOW 3: Job Status Query
+ *
+ * Purpose: Check job progress and status
+ * Trigger: Webhook GET/POST to /webhooks/product-ingestion-job-status?jobId=xxx
+ */
+export const productIngestionJobStatus = webhook(
+ 'product-ingestion-job-status'
+)
+.then(
+ fn('query-job-status', async (ctx) => {
+  const { data, log, openKv } = ctx;
+
+  const jobId = data.jobId as string;
+
+  if (!jobId) {
+   log.error('❌ Job ID not provided in request');
+   return {
+    success: false,
+    error: 'Job ID is required. Provide jobId in query param or request body.',
+    recommendation: 'Include jobId in request body: { "jobId": "SCHEDULED_PROD_..." }'
+   };
+  }
+
+  log.info('🔍 Querying job status', { jobId });
+
+  try {
+   const status = await getJobStatus(openKv(':project:'), jobId, log);
+
+   if (!status) {
+    log.info('⚠️ Job not found', { jobId });
+    return {
+     success: false,
+     error: 'Job not found',
+     jobId,
+     recommendation: 'Verify jobId is correct and job exists'
+    };
+   }
+
+   log.info('✅ Job status retrieved', { jobId, status: status.status });
+
+   return {
+    success: true,
+    jobId,
+    ...status
+   };
+
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   log.error('❌ Failed to query job status', {
+    jobId,
+    message: errorMessage,
+    stack: error instanceof Error ? error.stack : undefined,
+    errorType: error instanceof Error ? error.constructor.name : 'Error',
+    recommendation: 'Check KV store connectivity'
+   });
+
+   return {
+    success: false,
+    jobId,
+    error: errorMessage
+   };
+  }
+ })
+);
+```
+
+---
+
+## 3. Type Definitions (src/types/product-ingestion.types.ts)
+
+```typescript
+/**
+ * Type Definitions for Product Ingestion
+ *
+ * Centralized type definitions for product ingestion workflow
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Product interface - represents transformed product data
+ */
+export interface Product {
+ ref: string;
+ type?: string;
+ status?: string;
+ name: string;
+ summary?: string;
+ gtin?: string;
+ catalogue?: { ref?: string };
+ attributes?: Record<string, unknown>;
+}
+
+/**
+ * Event configuration
+ */
+export interface EventConfig {
+ eventName: string;
+ catalogueRef: string;
+ catalogueType: string;
+ eventMode: 'async' | 'sync';
+}
+
+/**
+ * Event result - tracks success/failure counts
+ */
+export interface EventResult {
+ eventsSent: number;
+ eventsFailed: number;
+ errors: Array<{ productRef: string; error: string }>;
+}
+
+/**
+ * Process file result
+ */
+export interface ProcessFileResult {
+ success: boolean;
+ products: Product[];
+ error?: string;
+}
+
+/**
+ * Versori Context Interface
+ * Represents the Versori runtime context passed to workflow functions
+ */
+export interface VersoriContext {
+ log: {
+  info: (message: string, data?: Record<string, unknown>) => void;
+  warn: (message: string, data?: Record<string, unknown>) => void;
+  error: (message: string, data?: Record<string, unknown>) => void;
+  debug?: (message: string, data?: Record<string, unknown>) => void;
+ };
+ openKv: (namespace: string) => {
+  get: (key: string) => Promise<unknown>;
+  set: (key: string, value: unknown) => Promise<void>;
+  delete: (key: string) => Promise<void>;
+ };
+ activation: {
+  getVariable: (name: string) => string | undefined;
+  connections?: Record<string, unknown>;
+ };
+ connections?: Record<string, unknown>;
+ data?: unknown;
+ fetch?: typeof fetch;
+}
+
+/**
+ * Parameters for ingestion workflow
+ */
+export interface ProductIngestionParams {
+ jobId: string;
+ triggeredBy: 'schedule' | 'webhook';
+ filePattern?: string;
+ maxFiles?: number;
+ forceReprocess?: boolean;
+ catalogueRef?: string;
+ priority?: 'normal' | 'high';
+}
+
+/**
+ * Result from ingestion workflow
+ */
+export interface ProductIngestionResult {
+ success: boolean;
+ jobId: string;
+ filesProcessed: number;
+ filesFailed: number;
+ filesSkipped?: number;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ fileResults: FileProcessingResult[];
+ error?: string;
+}
+
+/**
+ * Per-file processing result
+ */
+export interface FileProcessingResult {
+ fileName: string;
+ success: boolean;
+ skipped?: boolean;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ duration: number;
+ error?: string;
+}
+```
+
+---
+
+## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
+
+```typescript
+/**
+ * Product File Processor Service
+ *
+ * Downloads Parquet files from SFTP, parses, and transforms with UniversalMapper.
+ * Parquet-specific: Downloads as Buffer (binary format), uses parseSimple() method.
+ */
+
+import { Buffer } from 'node:buffer';
+import {
+ SftpDataSource,
+ ParquetParserService,
+ UniversalMapper,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
+
+/**
+ * Service for processing Parquet product files
+ */
+export class ProductFileProcessorService {
+ constructor(
+  private sftp: SftpDataSource,
+  private parquetParser: ParquetParserService,
+  private mapper: UniversalMapper,
+  private catalogueRef: string
+ ) {}
+
+ /**
+  * Download Parquet file from SFTP, parse, and transform with UniversalMapper
+  *
+  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
+  * - Explicit null assignments after each step
+  * - Finally block guarantees cleanup
+  * - Prevents OOM errors on large Parquet files
+  */
+ async downloadParseAndTransform(remoteFilePath: string): Promise<ProcessFileResult> {
+  // ✅ CRITICAL: Variables for cleanup tracking
+  let buffer: Buffer | null = null;
+  let parsed: unknown | null = null;
+
+  try {
+   // STEP 1: Download Parquet file as Buffer (binary format)
+   buffer = (await this.sftp.downloadFile(remoteFilePath)) as Buffer;
+
+   // STEP 2: Parse Parquet with type safety
+   try {
+    // ✅ PARQUET: Use parseSimple() method (returns array directly)
+    parsed = await this.parquetParser.parseSimple(buffer, remoteFilePath);
+   } catch (parseError: unknown) {
+    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
+    return {
+     success: false,
+     products: [],
+     error: `Parquet parse error: ${errorMessage}`,
+    };
+   }
+
+   // ✅ Clear buffer from memory - no longer needed after parsing
+   buffer = null;
+
+   // STEP 3: PARQUET-SPECIFIC - Returns array directly, no normalization needed
+   const rawProducts = Array.isArray(parsed) ? parsed : [];
+
+   if (rawProducts.length === 0) {
+    return {
+     success: false,
+     products: [],
+     error: 'No products found in Parquet file',
+    };
+   }
+
+   // ✅ Clear parsed data from memory - only need rawProducts now
+   parsed = null;
+
+   // STEP 4: Transform with UniversalMapper
+   // PARQUET-SPECIFIC: Merge context using spread pattern (not { inventory: ... })
+   const sourceDataWithContext = rawProducts.map(item => ({
+    ...item,
+    $context: { catalogueRef: this.catalogueRef },
+   }));
+
+   const mappingResult = await this.mapper.map(sourceDataWithContext);
+
+   if (!mappingResult.success) {
+    return {
+     success: false,
+     products: [],
+     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
+    };
+   }
+
+   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+     skippedFields: mappingResult.skippedFields,
+     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+    });
+   }
+
+   return {
+    success: true,
+    products: mappingResult.data as Product[],
+   };
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   return {
+    success: false,
+    products: [],
+    error: errorMessage,
+   };
+  } finally {
+   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
+   buffer = null;
+   parsed = null;
+  }
+ }
+}
+```
+
+---
+
+## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
+
+```typescript
+/**
+ * Event Sender Service
+ *
+ * Sends product events to Fluent Commerce Event API with per-record error handling.
+ * Continues processing on individual failures (Event API best practice).
+ * Supports configurable concurrency (sequential or parallel).
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
+
+/**
+ * Service for sending events to Fluent Commerce Event API
+ *
+ * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
+ */
+export class EventSenderService {
+ constructor(
+  private client: FluentClient,
+  private log?: any  // ✅ Optional logger for progress tracking
+ ) {}
+
+ /**
+  * Send events to Fluent Commerce Event API with configurable concurrency
+  *
+  * **Performance Characteristics:**
+  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
+  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
+  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
+  *
+  * **Implementation Strategy:**
+  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
+  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
+  * - Both modes: Per-record error tracking (failures don't block other events)
+  *
+  * @param products - Array of products to send as events
+  * @param eventConfig - Event configuration (name, catalogue ref, mode)
+  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
+  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
+  */
+ async sendEvents(
+  products: Product[],
+  eventConfig: EventConfig,
+  concurrency: number = 1
+ ): Promise<EventResult> {
+  // Validate concurrency (guard against invalid values)
+  const safeConc = Math.max(1, Math.floor(concurrency));
+
+  // Result accumulators
+  let eventsSent = 0;
+  let eventsFailed = 0;
+  const errors: Array<{ productRef: string; error: string }> = [];
+
+  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
+  if (this.log) {
+   this.log.info('📤 Starting event sending', {
+    totalProducts: products.length,
+    concurrency: safeConc,
+    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
+    eventName: eventConfig.eventName,
+    catalogueRef: eventConfig.catalogueRef
+   });
+  }
+
+  // Helper: Build event payload (DRY - reused in both modes)
+  const buildPayload = (product: Product) => ({
+   name: eventConfig.eventName,
+   entityRef: eventConfig.catalogueRef,
+   entityType: 'PRODUCT_CATALOGUE' as const,
+   entitySubtype: eventConfig.catalogueType,
+   rootEntityRef: eventConfig.catalogueRef,
+   rootEntityType: 'PRODUCT_CATALOGUE' as const,
+   attributes: product,
+  });
+
+  // ============================================================================
+  // SEQUENTIAL MODE (concurrency === 1)
+  // ============================================================================
+  if (safeConc === 1) {
+   for (let i = 0; i < products.length; i++) {
+    const product = products[i];
+
+    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
+    if (this.log && i % 10 === 0) {
+     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
+      productRef: product.ref,
+      progress: `${((i / products.length) * 100).toFixed(1)}%`
+     });
+    }
+
+    try {
+     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
+     eventsSent++;
+    } catch (err: unknown) {
+     eventsFailed++;
+     const errorMsg = err instanceof Error ? err.message : String(err);
+     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
+     // Continue processing (failure doesn't block other products)
+    }
+   }
+
+   // ✅ PRODUCTION ENHANCEMENT: Log completion
+   if (this.log) {
+    this.log.info('✅ Sequential event sending completed', {
+     totalProducts: products.length,
+     eventsSent,
+     eventsFailed
+    });
+   }
+
+   return { eventsSent, eventsFailed, errors };
+  }
+
+  // ============================================================================
+  // PARALLEL MODE (concurrency > 1)
+  // ============================================================================
+  const totalChunks = Math.ceil(products.length / safeConc);
+
+  for (let i = 0; i < products.length; i += safeConc) {
+   const chunk = products.slice(i, i + safeConc);
+   const chunkNumber = Math.floor(i / safeConc) + 1;
+
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
+   if (this.log) {
+    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
+     chunkNumber,
+     totalChunks,
+     productsInChunk: chunk.length,
+     productRange: `${i + 1}-${i + chunk.length}`,
+     totalProducts: products.length,
+     progress: `${((i / products.length) * 100).toFixed(1)}%`
+    });
+   }
+
+   // Fire all requests in chunk concurrently
+   const results = await Promise.allSettled(
+    chunk.map(product =>
+     this.client
+      .sendEvent(buildPayload(product), eventConfig.eventMode)
+      .then(() => ({ success: true as const, product }))
+      .catch(error => ({ success: false as const, product, error }))
+    )
+   );
+
+   // Aggregate chunk results into totals
+   let chunkSuccess = 0;
+   let chunkFailed = 0;
+
+   for (const result of results) {
+    if (result.status === 'fulfilled' && result.value.success) {
+     eventsSent++;
+     chunkSuccess++;
+    } else {
+     eventsFailed++;
+     chunkFailed++;
+     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
+     const product = result.status === 'fulfilled' ? result.value.product : null;
+     errors.push({
+      productRef: product?.ref || 'unknown',
+      error: error?.message || String(error) || 'unknown error',
+     });
+    }
+   }
+
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
+   if (this.log) {
+    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
+     chunkNumber,
+     totalChunks,
+     chunkSuccess,
+     chunkFailed,
+     totalSentSoFar: eventsSent,
+     totalFailedSoFar: eventsFailed,
+     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
+    });
+   }
+  }
+
+  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
+  if (this.log) {
+   this.log.info('✅ Parallel event sending completed', {
+    totalProducts: products.length,
+    totalChunks,
+    concurrency: safeConc,
+    eventsSent,
+    eventsFailed
+   });
+  }
+
+  return { eventsSent, eventsFailed, errors };
+ }
+}
+```
+
+---
+
+## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
+
+```typescript
+/**
+ * Event Logger Service
+ *
+ * Writes event processing logs to SFTP for audit/debugging.
+ * Optional - can be used to create rejection reports.
+ */
+
+import { Buffer } from 'node:buffer';
+import { SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Service for writing event logs to SFTP
+ */
+export class EventLoggerService {
+ constructor(private sftp: SftpDataSource) {}
+
+ /**
+  * Write event log to SFTP
+  *
+  * @param remotePath - SFTP path for log file
+  * @param logData - Log data to write (JSON format)
+  */
+ async writeEventLog(remotePath: string, logData: unknown): Promise<void> {
+  try {
+   const logContent = JSON.stringify(logData, null, 2);
+   await this.sftp.uploadFile(remotePath, logContent);
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   throw new Error(`Failed to write event log: ${errorMessage}`);
+  }
+ }
+}
+```
+
+---
+
+## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
+
+```typescript
+/**
+ * MAIN INGESTION ORCHESTRATION SERVICE
+ *
+ * This is the heart of the ingestion workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Discover files on SFTP
+ * 3. Download and parse Parquet files
+ * 4. Transform data with UniversalMapper
+ * 5. Send events to Fluent Commerce
+ * 6. Archive processed files
+ * 7. Track file processing state
+ * 8. Track job progress with JobTracker
+ *
+ * NAMING PATTERN (consistent across all use cases):
+ * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
+ * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
+ * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ *
+ * - Change entity: Replace "Product" with "Inventory", "Location", etc.
+ * - Change format: Replace ParquetParserService with XMLParserService/JSONParserService
+ * - Change source: Replace SftpDataSource with S3DataSource
+ * - Change destination: Replace sendEvent with Batch API or GraphQL mutations
+ */
+
+import { Buffer } from 'node:buffer';
+import {
+ createClient,
+ SftpDataSource,
+ ParquetParserService,
+ UniversalMapper,
+ VersoriFileTracker,
+ JobTracker,
+ type FluentClient,
+ type JobStatus,
+} from '@fluentcommerce/fc-connect-sdk';
+import type {
+ ProductIngestionParams,
+ ProductIngestionResult,
+ FileProcessingResult,
+ EventConfig,
+} from '../types/product-ingestion.types';
+import { ProductFileProcessorService } from './product-file-processor.service';
+import { EventSenderService } from './event-sender.service';
+import { EventLoggerService } from './event-logger.service';
+import { joinSftpPath, timestampedName, getBaseName } from '../utils/sftp-path.utils';
+
+import mappingConfig from '../../config/products.import.parquet.json' with { type: 'json' };
+
+// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
+
+/**
+ * Query job status from KV store
+ *
+ * NAMING: get{Entity}JobStatus or just getJobStatus (generic)
+ *
+ * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
+ */
+export async function getJobStatus(
+ kv: ReturnType<VersoriContext['openKv']>,
+ jobId: string,
+ log: VersoriContext['log']
+): Promise<JobStatus | undefined> {
+ try {
+  const tracker = new JobTracker(kv, log);
+  return await tracker.getJob(jobId); // ✅ Use getJob() not getJobStatus()
+ } catch (error: unknown) {
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  log.error('Failed to get job status', {
+   jobId,
+   message: errorMessage,
+   stack: error instanceof Error ? error.stack : undefined,
+   errorType: error instanceof Error ? error.constructor.name : 'Error'
+  });
+  return undefined;
+ }
+}
+
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ * This function implements the complete ingestion workflow in 8 steps.
+ * Each step is clearly commented for AI understanding.
+ */
+export async function executeProductIngestion(
+ ctx: VersoriContext,
+ params: ProductIngestionParams
+): Promise<ProductIngestionResult> {
+
+  // ✅ VERSORI PLATFORM: Extract native log from context (LoggingService was removed - use native log)
+ const { log, openKv, activation } = ctx;
+ const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
+
+ // Open KV store for state management and job tracking
+ // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
+ // ✅ Pass native log to JobTracker
+ const kv = openKv(':project:');
+ const tracker = new JobTracker(kv, log);
+
+ const startTime = Date.now();
+ const fileResults: FileProcessingResult[] = [];
+
+ // ⚠️ CRITICAL: Declare SFTP outside try block for double-finally pattern
+ let sftp: SftpDataSource | undefined;
+
+ try {
+  // 
+  // STEP 1: Initialize Job Tracking
+  // 
+  log.info('📊 [STEP 1/8] Initializing job tracking', { jobId });
+
+  await tracker.createJob(jobId, {
+   triggeredBy,
+   filePattern: filePattern || 'default',
+   maxFiles: maxFiles || 'unlimited',
+   forceReprocess: !!forceReprocess
+  });
+
+  // 
+  // STEP 2: Initialize Fluent Client & SFTP Connection
+  // 
+  log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and SFTP', { jobId });
+
+  // ✅ Optional: Validate connection immediately (fail-fast mode)
+  // Set activation variable 'validateConnectionOnStart' = 'true' to enable
+  // When enabled: Executes query { me { ref } } to verify authentication
+  // When disabled: Fast creation, validation happens on first API call (default)
+  const validateConnection = activation.getVariable('validateConnectionOnStart') === 'true';
+  const client = await createClient(ctx, validateConnection ? { validateConnection: true } : undefined);
+
+  if (!client) {
+   throw new Error('Failed to create Fluent Commerce client');
+  }
+
+  if (validateConnection) {
+   log.info('✅ Connection validated successfully - authentication confirmed', { jobId });
+  }
+
+  // ✅ CRITICAL: Set retailerId for Event API calls
+  // Event API requires retailerId - fail fast if not configured
+  const fluentRetailerId = activation.getVariable('fluentRetailerId');
+  if (!fluentRetailerId) {
+   throw new Error('fluentRetailerId is required for Event API calls');
+  }
+  client.setRetailerId(fluentRetailerId);
+  log.info('RetailerId set for Event API', { retailerId: fluentRetailerId });
+
+  // ========================================
+  // SFTP CREDENTIAL RETRIEVAL
+  // ========================================
+
+  /**
+   * Retrieve SFTP credentials from connection configuration
+   *
+   * This approach retrieves credentials stored in the Versori connection settings.
+   * The connection must be configured in the UI with Basic Authentication.
+   *
+   * Steps:
+   * 1. Call ctx.credentials().getAccessToken('SFTP') to get base64-encoded credentials
+   * 2. Decode the accessToken from base64 to get "username:password" string
+   * 3. Split on ':' to extract username and password
+   *
+   * Connection Name: 'SFTP' (must match the connection name in Versori UI)
+   * Auth Type: Basic Authentication (username + password)
+   *
+   * This method provides:
+   * - Centralized credential management through Versori UI
+   * - Better security (credentials not stored in integration variables)
+   * - Easier credential rotation and updates
+   */
+  log.info('Retrieving SFTP credentials from connection configuration');
+
+  let sftpUsername: string;
+  let sftpPassword: string;
+
+  try {
+   // Retrieve credentials from the 'SFTP' connection
+   const sftpCred = await ctx.credentials().getAccessToken('SFTP');
+
+   if (!sftpCred?.accessToken) {
+    throw new Error('No SFTP credentials found in connection configuration');
+   }
+
+   // Decode base64 accessToken to get "username:password"
+   const rawBasicAuth = Buffer.from(sftpCred.accessToken, 'base64').toString('utf-8');
+
+   // Split on ':' to extract username and password
+   const parts = rawBasicAuth.split(':');
+
+   if (parts.length !== 2) {
+    throw new Error('Invalid SFTP credential format - expected username:password');
+   }
+
+   sftpUsername = parts[0];
+   sftpPassword = parts[1];
+
+   log.info('SFTP credentials retrieved successfully', {
+    hasUsername: !!sftpUsername,
+    hasPassword: !!sftpPassword,
+    usernameLength: sftpUsername.length,
+    passwordLength: sftpPassword.length,
+   });
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   const errorStack = error instanceof Error ? error.stack : undefined;
+   log.error('Failed to retrieve SFTP credentials', {
+    message: errorMessage,
+    stack: errorStack,
+    errorType: error instanceof Error ? error.constructor.name : 'Error'
+   });
+
+   return {
+    success: false,
+    error: 'Failed to retrieve SFTP credentials from connection configuration',
+    details: errorMessage,
+    recommendation: 'Please ensure the SFTP connection is configured in the Connections section with Basic Authentication (username and password)',
+   } as ProductIngestionResult;
+  }
+
+  // Get SFTP configuration from activation variables
+  const sftpConfig = {
+   host: activation.getVariable('sftpHost'),
+   port: parseInt(activation.getVariable('sftpPort') || '22', 10),
+   sftpUsername, // From connection (secure)
+   sftpPassword, // From connection (secure)
+   privateKey: activation.getVariable('sftpPrivateKey'),
+   incomingPath: activation.getVariable('sftpIncomingPath') || '/products/incoming',
+   archivePath: activation.getVariable('sftpArchivePath') || '/archive/products/',
+   errorPath: activation.getVariable('sftpErrorPath') || '/errors/products/',
+   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.parquet'
+  };
+
+  // Validate SFTP config
+  if (!sftpConfig.host) {
+   throw new Error('SFTP configuration incomplete: missing host');
+  }
+
+  if (!sftpConfig.sftpPassword && !sftpConfig.privateKey) {
+   throw new Error('SFTP configuration incomplete: missing password or privateKey');
+  }
+
+  // Initialize SFTP data source
+  // ✅ VERSORI PLATFORM: Pass native log from context
+  // ✅ PARQUET: No encoding needed (binary format)
+  sftp = new SftpDataSource({
+   type: 'SFTP_PARQUET',
+   connectionId: 'sftp-product-ingestion',
+   name: 'Product Ingestion SFTP',
+   settings: {
+    host: sftpConfig.host,
+    port: sftpConfig.port,
+    username: sftpConfig.sftpUsername,
+    password: sftpConfig.sftpPassword,
+    privateKey: sftpConfig.privateKey,
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern,
+   }
+  }, log);
+
+  try {
+   // Validate SFTP connection
+   await sftp.validateConnection();
+   log.info('SFTP connection validated');
+
+   // Ensure archive directories exist (recursive)
+   await sftp.createDirectory(sftpConfig.archivePath, true);
+   await sftp.createDirectory(sftpConfig.errorPath, true);
+
+   // 
+   // STEP 3: Discover Files on SFTP
+   // 
+   log.info('🔍 [STEP 3/8] Discovering files on SFTP', {
+    jobId,
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern
+   });
+
+   await tracker.updateJob(jobId, {
+    status: 'processing',
+    stage: 'file_discovery',
+    message: 'Discovering files on SFTP'
+   });
+
+   // List files from SFTP
+   const allFiles = await sftp.listFiles({
+    remotePath: sftpConfig.incomingPath,
+    filePattern: sftpConfig.filePattern
+   });
+
+   log.info(`Discovered ${allFiles.length} files matching pattern`);
+
+   // Apply maxFiles limit if specified
+   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
+
+   if (filesToProcess.length === 0) {
+    log.info('No files to process');
+
+    await tracker.markCompleted(jobId, {
+     filesProcessed: 0,
+     message: 'No files found to process'
+    });
+
+  return {
+     success: true,
+     jobId,
+    filesProcessed: 0,
+    filesFailed: 0,
+    recordsProcessed: 0,
+    eventsSent: 0,
+    eventsFailed: 0,
+     fileResults: []
+    };
+   }
+
+   log.info(`Processing ${filesToProcess.length} files`);
+
+   // Initialize services
+   // ✅ PARQUET: Use ParquetParserService instead of CSVParserService
+   const parquetParser = new ParquetParserService(log);
+   const mapper = new UniversalMapper(mappingConfig);
+   
+   // Get event configuration
+   const eventConfig: EventConfig = {
+    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
+    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
+    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
+    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
+   };
+
+   // Get event sending configuration
+   const eventConcurrency = Math.max(
+    1,
+    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
+   );
+   // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
+   // concurrency: 1 = sequential, concurrency > 1 = parallel
+
+   log.info(`Event concurrency: ${eventConcurrency}`, {
+    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
+    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
+   });
+
+   // Initialize class-based services
+   const fileProcessor = new ProductFileProcessorService(
+    sftp,
+    parquetParser,
+    mapper,
+    eventConfig.catalogueRef
+   );
+   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
+   const eventSender = new EventSenderService(client, log);
+   const eventLogger = new EventLoggerService(sftp);
+
+   // Use direct KV with dot-separated keys (NATS KV safe)
+   const processedKeyBase = 'fc_sdk.processed_files';
+
+   // 
+   // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
+   // 
+
+   // Get SFTP path configuration
+   const requireAbsolutePaths = activation.getVariable('requireAbsolutePaths') === 'true';
+
+   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
+    const file = filesToProcess[fileIndex];
+    const fileStartTime = Date.now();
+
+    // Use full remote path from SFTP listing and derive basename + KV-safe key
+    const remoteFilePath = (file as any).path || file.name;
+    const baseName = getBaseName(remoteFilePath);
+    const safeKey = baseName.replace(/[^A-Za-z0-9._-]/g, '_');
+
+    log.info(`[FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`);
+
+    try {
+     // ✅ File tracking: Skip already processed files (configurable)
+     // Set enableFileTracking=true (default) to skip duplicates
+     // Set forceReprocess=true to bypass tracking and reprocess all files
+     const enableFileTracking = activation.getVariable('enableFileTracking') !== 'false';
+
+     if (enableFileTracking && !forceReprocess) {
+      const processedEntry = await kv.get(`${processedKeyBase}.${safeKey}`);
+      if (processedEntry) {
+       log.info(`⏭️ Skipping already processed file: ${baseName}`);
+       fileResults.push({
+        fileName: baseName,
+        success: true,
+        skipped: true,
+        recordsProcessed: 0,
+        eventsSent: 0,
+        eventsFailed: 0,
+        duration: 0,
+       });
+       continue;
+      }
+     }
+
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'downloading',
+      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
+     });
+
+     // ═══════════════════════════════════════════════════════════
+     // STEP 4: Process File (Download + Parse + Map)
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📦 [STEP 4/8] Processing file: ${baseName}`);
+
+     // Process file: download → parse → transform
+     const fileResult = await fileProcessor.downloadParseAndTransform(remoteFilePath);
+
+     if (!fileResult.success || fileResult.products.length === 0) {
+      // Move failed file to errors and record result
+      const archivedName = timestampedName(baseName);
+      await sftp.moveFile(
+     remoteFilePath,
+       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
+      );
+
+      fileResults.push({
+       fileName: baseName,
+       success: false,
+       recordsProcessed: fileResult.products.length,
+       eventsSent: 0,
+       eventsFailed: 0,
+       duration: Date.now() - fileStartTime,
+       error: fileResult.error || 'Processing failed'
+      });
+
+      continue;
+     }
+
+     // ═══════════════════════════════════════════════════════════
+     // STEP 5: Send Events
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
+
+     // ? Enhanced: Extract context for progress logging
+     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
+     const eventType = eventConfig.eventType || 'UPSERT_PRODUCT';
+
+     // ? Enhanced: Start logging with context
+     log.info(`[EventSender] Sending events for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventType,
+      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
+      sampleProductRefs: sampleProductRefs.join(', '),
+      eventMode: eventConfig.eventMode || 'async'
+     });
+
+     const eventResult = await eventSender.sendEvents(
+      fileResult.products,
+      eventConfig,
+      eventConcurrency
+     );
+
+     const eventsSent = eventResult.eventsSent;
+     const eventsFailed = eventResult.eventsFailed;
+
+     log.info(`Events sent for ${baseName}`, {
+      successful: eventsSent,
+      failed: eventsFailed
+     });
+
+     // ? Enhanced: Completion logging with summary
+     log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
+      eventType
+     });
+
+     // ═══════════════════════════════════════════════════════════
+     // STEP 6: Archive File & Track State
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📁 [STEP 6/8] Archiving file: ${baseName}`);
+
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'archiving',
+      message: `Archiving ${baseName}`
+     });
+
+     // Conditionally archive: errors → errorPath, else → archivePath (timestamped)
+     const archivedName = timestampedName(baseName);
+     const targetDir = eventsFailed > 0 ? sftpConfig.errorPath : sftpConfig.archivePath;
+     await sftp.moveFile(
+      remoteFilePath,
+      joinSftpPath(requireAbsolutePaths, targetDir, archivedName)
+     );
+
+     // Mark as processed in KV (dot-separated key; NATS KV safe)
+     await kv.set(`${processedKeyBase}.${safeKey}`, {
+      recordCount: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      processedAt: new Date().toISOString()
+     });
+
+     log.info(`File archived successfully: ${baseName}`);
+
+     // Optional: Write event log for audit/debugging
+     if (eventsFailed > 0) {
+      const logPath = joinSftpPath(
+       requireAbsolutePaths,
+       sftpConfig.errorPath,
+       `${baseName.replace(/\.parquet$/i, '')}-event-log.json`
+      );
+      await eventLogger.writeEventLog(logPath, {
+       fileName: baseName,
+       totalRecords: fileResult.products.length,
+       eventsSent,
+       eventsFailed,
+       errors: eventResult.errors,
+       processedAt: new Date().toISOString()
+      });
+     }
+
+     // Store file result
+     fileResults.push({
+      fileName: baseName,
+     success: true,
+      recordsProcessed: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      duration: Date.now() - fileStartTime
+     });
+
+    } catch (error: unknown) {
+     // ✅ Enhanced error logging: Extract all error details for visibility
+     const errorMessage = error instanceof Error ? error.message : String(error);
+     const errorDetails = {
+      message: errorMessage,
+      stack: error instanceof Error ? error.stack : undefined,
+      fileName: (error as any)?.fileName,
+      lineNumber: (error as any)?.lineNumber,
+      originalError: (error as any)?.context?.originalError?.message,
+      errorType: error instanceof Error ? error.name : 'Error',
+     };
+     log.error(`Error processing file ${baseName}:`, errorDetails);
+
+     // Best-effort: move to error archive on unexpected failure
+     try {
+      const archivedName = timestampedName(baseName);
+      await sftp.moveFile(
+       remoteFilePath,
+       joinSftpPath(requireAbsolutePaths, sftpConfig.errorPath, archivedName)
+      );
+     } catch (moveError: unknown) {
+      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
+      log.error('Could not archive failed file', {
+       file: baseName,
+       message: moveErrorMessage,
+       stack: moveError instanceof Error ? moveError.stack : undefined,
+       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
+      });
+     }
+
+     fileResults.push({
+      fileName: baseName,
+      success: false,
+      recordsProcessed: 0,
+      eventsSent: 0,
+      eventsFailed: 0,
+      duration: Date.now() - fileStartTime,
+      error: errorMessage
+     });
+    }
+   }
+
+   // 
+   // STEP 8: Complete Job & Calculate Totals
+   // 
+   log.info('✅ [STEP 8/8] Completing job and calculating totals', { jobId });
+
+   const filesProcessed = fileResults.filter(r => r.success).length;
+   const filesFailed = fileResults.filter(r => !r.success).length;
+   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
+   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
+   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
+
+
+   await tracker.markCompleted(jobId, {
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: Date.now() - startTime
+   });
+
+   const duration = Date.now() - startTime;
+   log.info('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+   log.info('✅ INGESTION WORKFLOW COMPLETED SUCCESSFULLY');
+   log.info('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+   log.info('📊 Final Summary:', {
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: `${duration}ms`
+   });
+
+   return {
+    success: true,
+    jobId,
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    fileResults
+   };
+
+  } finally {
+   // ❌š ï¸ CRITICAL: Always dispose SFTP connection
+   if (sftp) {
+    await sftp.dispose();
+    log.info('SFTP connection disposed');
+   }
+  }
+
+ } catch (error: unknown) {
+  const duration = Date.now() - startTime;
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  const errorStack = error instanceof Error ? error.stack : undefined;
+
+  log.error('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  log.error('❌ INGESTION WORKFLOW FAILED');
+  log.error('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  log.error('💥 Error Details:', {
+   jobId,
+   message: errorMessage,
+   stack: errorStack,
+   errorType: error instanceof Error ? error.constructor.name : 'Error',
+   duration: `${duration}ms`
+  });
+
+  // Provide actionable recommendations based on error type
+  if (errorMessage.includes('SFTP') || errorMessage.includes('connection')) {
+   log.error('💡 Recommendation: Check SFTP credentials and network connectivity');
+  } else if (errorMessage.includes('parse') || errorMessage.includes('Parquet')) {
+   log.error('💡 Recommendation: Verify Parquet file format and schema compatibility');
+  } else if (errorMessage.includes('Event API') || errorMessage.includes('401')) {
+   log.error('💡 Recommendation: Verify Fluent Commerce credentials and retailerId');
+  } else if (errorMessage.includes('KV') || errorMessage.includes('storage')) {
+   log.error('💡 Recommendation: Check Versori KV store connectivity');
+  } else {
+   log.error('💡 Recommendation: Review error stack trace and activation variables');
+  }
+
+  // Try to mark job as failed, but don't let tracking errors mask workflow error
+  try {
+   await tracker.markFailed(jobId, error);
+  } catch (trackingError: unknown) {
+   const trackingErrorMessage =
+    trackingError instanceof Error ? trackingError.message : String(trackingError);
+   log.warn('Failed to mark job as failed in tracker', {
+    jobId,
+    trackingError: trackingErrorMessage,
+   });
+  }
+
+  // Determine recommendation based on error type
+  let recommendation = 'Review error details and check activation variables';
+  if (errorMessage.includes('SFTP') || errorMessage.includes('connection')) {
+   recommendation = 'Verify SFTP credentials, host, port, and network connectivity. Check if SFTP server is accessible.';
+  } else if (errorMessage.includes('parse') || errorMessage.includes('Parquet')) {
+   recommendation = 'Verify Parquet file format, schema compatibility, and file integrity. Check if file is corrupted.';
+  } else if (errorMessage.includes('Event API') || errorMessage.includes('401')) {
+   recommendation = 'Verify Fluent Commerce credentials (clientId, clientSecret) and retailerId configuration.';
+  } else if (errorMessage.includes('KV') || errorMessage.includes('storage')) {
+   recommendation = 'Check Versori KV store connectivity and permissions. Verify namespace configuration.';
+  }
+
+  return {
+   success: false,
+   jobId,
+   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
+   filesFailed: fileResults.filter(r => !r.success).length,
+   filesSkipped: fileResults.filter(r => r.skipped).length,
+   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
+   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
+   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
+   fileResults,
+   error: errorMessage,
+   recommendation,
+   duration: `${duration}ms`
+  };
+ } finally {
+  // ⚠️ CRITICAL: Ensure SFTP is disposed even if outer error occurs
+  // This outer finally ensures disposal if error happens after SFTP creation
+  // but before inner try block (e.g., during connection validation)
+  if (sftp) {
+   try {
+    await sftp.dispose();
+    log.info('SFTP connection disposed (outer finally)');
+   } catch (disposeError: unknown) {
+    const disposeErrorMessage =
+     disposeError instanceof Error ? disposeError.message : String(disposeError);
+    log.warn('Error disposing SFTP in outer finally', { error: disposeErrorMessage });
+   }
+  }
+ }
+}
+```
+
+---
+
+## 8. Utility Functions (src/utils/)
+
+### SFTP Path Helpers (src/utils/sftp-path.utils.ts)
+
+```typescript
+/**
+ * SFTP Path Utilities
+ *
+ * Helper functions for SFTP path operations with AWS Transfer Family support.
+ * Handles both absolute paths (AWS Transfer Family) and relative paths (standard OpenSSH).
+ */
+
+/**
+ * Join SFTP path segments safely
+ *
+ * Handles different SFTP server path requirements:
+ * - AWS Transfer Family: Requires absolute paths (leading /)
+ * - Standard OpenSSH: Supports relative paths
+ *
+ * @param requireAbsolutePath - true for AWS Transfer Family, false for standard OpenSSH
+ * @param parts - Path segments to join
+ * @returns Properly formatted SFTP path
+ *
+ * @example
+ * ```typescript
+ * // AWS Transfer Family (absolute paths)
+ * joinSftpPath(true, 'products', 'processed', 'file.parquet')
+ * // Returns: '/products/processed/file.parquet'
+ *
+ * // Standard OpenSSH (relative paths)
+ * joinSftpPath(false, 'products', 'processed', 'file.parquet')
+ * // Returns: 'products/processed/file.parquet'
+ * ```
+ */
+export function joinSftpPath(requireAbsolutePath: boolean, ...parts: string[]): string {
+ // Clean each segment (remove leading/trailing slashes)
+ const cleaned = parts
+  .filter(Boolean)
+  .map(p => String(p).replace(/^\/+|\/+$/g, ''))
+  .join('/');
+
+ // Add leading slash if required (AWS Transfer Family)
+ return requireAbsolutePath && !cleaned.startsWith('/') ? `/${cleaned}` : cleaned;
+}
+
+/**
+ * Generate timestamped filename for archival
+ *
+ * Adds ISO timestamp to filename before extension for unique archival.
+ * Handles Parquet files specifically but can be adapted for other formats.
+ *
+ * @param name - Original filename
+ * @returns Filename with timestamp appended
+ *
+ * @example
+ * ```typescript
+ * timestampedName('products.parquet')
+ * // Returns: 'products-2025-11-01T18-30-45-123Z.parquet'
+ * ```
+ */
+export function timestampedName(name: string): string {
+ const ts = new Date().toISOString().replace(/[:.]/g, '-');
+ const base = name.replace(/\.parquet$/i, '');
+ return `${base}-${ts}.parquet`;
+}
+
+/**
+ * Extract base filename from full SFTP path
+ *
+ * @param filePath - Full SFTP path (e.g., '/products/incoming/file.parquet')
+ * @returns Base filename only (e.g., 'file.parquet')
+ *
+ * @example
+ * ```typescript
+ * getBaseName('/products/incoming/products_20250101.parquet')
+ * // Returns: 'products_20250101.parquet'
+ * ```
+ */
+export function getBaseName(filePath: string): string {
+ return filePath.split('/').pop() || filePath;
+}
+```
+
+---
+
+### Job ID Generator (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking ingestion workflows
+ *
+ * FORMAT: {TYPE}_{ENTITY}_{YYYYMMDD}_{HHMMSS}_{RANDOM}
+ * Example: SCHEDULED_PRD_20251024_183045_a1b2c3
+ *
+ * NAMING: generate{Entity}JobId or generateJobId (generic)
+ */
+
+/**
+ * Generate unique job ID
+ *
+ * @param type - Job trigger type (SCHEDULED, ADHOC, MANUAL)
+ * @param entity - Entity abbreviation (VP, IP, ORD, PRD)
+ * @returns Unique job ID string
+ */
+export function generateJobId(type: string, entity: string): string {
+ const now = new Date();
+
+ // Format: YYYYMMDD
+ const dateStr = now.toISOString().slice(0, 10).replace(/-/g, "");
+
+ // Format: HHMMSS
+ const timeStr = now.toISOString().slice(11, 19).replace(/:/g, "");
+
+ // Random suffix (6 chars)
+ const randomStr = Math.random().toString(36).substring(2, 8);
+
+ return `${type}_${entity}_${dateStr}_${timeStr}_${randomStr}`;
+}
+
+/**
+ * Parse job ID components
+ *
+ * @param jobId - Job ID to parse
+ * @returns Parsed components or null if invalid
+ */
+export function parseJobId(
+ jobId: string
+): {
+ type: string;
+ entity: string;
+ date: string;
+ time: string;
+ random: string;
+} | null {
+ const parts = jobId.split("_");
+
+ if (parts.length !== 5) {
+  return null;
+ }
+
+ return {
+  type: parts[0],
+  entity: parts[1],
+  date: parts[2],
+  time: parts[3],
+  random: parts[4],
+ };
+}
+```
+
+---
+
+### 9. Mapping (`config/products.import.parquet.json`)
+
+```json
+{
+ "name": "products.import.parquet",
+ "version": "1.2.0",
+ "description": "Parquet → Product Event Mapping",
+ "fields": {
+  "ref": { "source": "ref", "required": true, "resolver": "sdk.trim" },
+  "type": { "source": "type", "resolver": "sdk.uppercase", "defaultValue": "STANDARD" },
+  "status": { "source": "status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" },
+  "gtin": { "source": "gtin" },
+  "name": { "source": "name", "required": true, "resolver": "sdk.trim" },
+  "categoryRefs": { "source": "category_refs", "resolver": "custom.splitByPipe" },
+  "price": { "resolver": "custom.buildPriceArray" },
+  "taxType": { "resolver": "custom.buildTaxType" },
+  "attributes": { "defaultValue": [] }
+ }
+}
+```
+
+---
+
+### 10. Package Configuration
+
+#### `package.json`
+
+```json
+{
+ "name": "sftp-parquet-product-event",
+ "version": "1.2.0",
+ "type": "module",
+ "main": "src/index.ts",
+ "scripts": {
+  "dev": "versori dev",
+  "build": "versori build",
+  "deploy": "versori deploy"
+ },
+ "dependencies": {
+  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+  "@versori/run": "latest"
+ },
+ "devDependencies": {
+  "@types/node": "^20.0.0",
+  "typescript": "^5.0.0"
+ }
+}
+```
+
+#### `tsconfig.json`
+
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+
+---
+
+## Event Sending Configuration
+
+**Simple Configuration:** One variable controls everything
+
+```json
+{
+ "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
+}
+```
+
+**How it works:**
+
+- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
+- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
+- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
+- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
+
+**Configuration Guidelines:**
+
+- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
+- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
+- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
+- **Note:** Fluent API supports concurrent requests - adjust based on your needs
+
+**Why Single Variable?**
+
+- **Simpler:** One variable instead of two (`mode` + `concurrency`)
+- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
+- **Less config:** Fewer activation variables to manage
+- **Flexible:** Easy to tune performance (just change the number)
+
+---
+
+## Production Implementation
+
+### Project Structure
+
+**Production-Ready Modular Structure:**
+
+```
+sftp-parquet-product-event/
+├── package.json
+├── tsconfig.json
+├── index.ts                 # Workflow entry point (exports workflows)
+└── src/
+  ├── workflows/
+  │  └── product-ingestion.ts       # Main orchestrator (coordinates services)
+  ├── services/
+  │  ├── product-file-processor.service.ts # Download, parse, transform Parquet
+  │  ├── event-sender.service.ts      # Send events to Fluent API
+  │  ├── event-logger.service.ts      # Write event logs to SFTP
+  │  └── product-ingestion.service.ts   # Main orchestration logic
+  ├── types/
+  │  └── product-ingestion.types.ts    # TypeScript interfaces
+  ├── utils/
+  │  ├── sftp-path.utils.ts        # SFTP path helpers
+  │  └── job-id-generator.ts        # Job ID utilities
+  └── config/
+    └── products.import.parquet.json   # Mapping configuration
+```
+
+**Benefits of This Modular Structure:**
+
+- ✅ **Separation of Concerns**: Each service has one clear responsibility
+- ✅ **Reusability**: Services can be imported and used in other workflows
+- ✅ **Testability**: Easy to write unit tests for individual services
+- ✅ **Maintainability**: Changes isolated to specific service files
+- ✅ **Type Safety**: Centralized type definitions prevent duplication
+- ✅ **Scalability**: Easy to add new services without modifying existing code
+- ✅ **Gold Standard Compliant**: 100% compliance with Event API checklist
+
+**Key Services:**
+
+1. **ProductFileProcessorService**: Downloads Parquet from SFTP, parses with `ParquetParserService`, transforms with `UniversalMapper`
+2. **EventSenderService**: Sends events to Fluent API with configurable concurrency (sequential or parallel)
+3. **EventLoggerService**: Writes event processing logs to SFTP for audit/debugging
+4. **ProductIngestionService**: Main orchestration - coordinates all services through 8-step workflow
+
+---
+
+## 6. Deployment Instructions
+
+### Deploy to Versori
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Test locally (if using Versori CLI)
+npm run dev
+
+# 3. Deploy to Versori platform
+npm run deploy
+```
+
+### Configure Activation Variables
+
+In Versori platform settings, configure all variables listed in the Activation Variables section above.
+
+---
+
+## 7. Testing
+
+### Test Scheduled Ingestion
+
+Upload a test Parquet file to SFTP incoming directory and wait for the scheduled run.
+
+**Check logs:**
+
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and SFTP
+[STEP 3/8] Discovering files on SFTP
+[FILE 1/1] Processing file: products_20250124.parquet
+[STEP 4/8] Downloading and parsing: products_20250124.parquet
+[STEP 5/8] Transforming 5 products from products_20250124.parquet
+[STEP 6/8] Sending 5 events to Fluent Commerce
+[STEP 7/8] Archiving file: products_20250124.parquet
+[STEP 8/8] Completing job and calculating totals
+```
+
+### Test Ad hoc Ingestion
+
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{}'
+
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{
+  "filePattern": "urgent_*.parquet", }'
+```
+
+### Test Job Status Query
+
+```bash
+curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
+ -H "Content-Type: application/json" \
+ -d '{
+  "jobId": "ADHOC_PROD_20251024_183045_abc123"
+ }'
+```
+
+---
+
+## Monitoring
+
+### Success Response
+
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 50,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.parquet",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 50,
+   "eventsFailed": 0
+  }
+ ],
+ "duration": 12345
+}
+```
+
+### Partial Success Response
+
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 45,
+ "eventsFailed": 5,
+ "results": [
+  {
+   "file": "products_2025-01-22.parquet",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 45,
+   "eventsFailed": 5,
+   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
+  }
+ ],
+ "duration": 12345
+}
+```
+
+### Error Response
+
+```json
+{
+ "success": false,
+ "filesProcessed": 0,
+ "filesFailed": 1,
+ "totalRecords": 0,
+ "eventsSent": 0,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.parquet",
+   "success": false,
+   "error": "Parquet parse error: Invalid structure"
+  }
+ ],
+ "duration": 876
+}
+```
+
+### Monitoring Metrics
+
+Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
+
+- **Files Processed** - Total files successfully processed
+- **Events Sent** - Total events sent to Fluent Commerce
+- **Events Failed** - Events that failed (check rejection reports)
+- **Processing Duration** - Time taken for complete workflow
+- **Rate Limiting** - Watch for 429 errors indicating throttling
+
+Use the status webhook for dashboards and automated monitoring.
+
+---
+
+- **No files found:** Check `sftpIncomingPath` and `filePattern` (names only)
+- **Parquet parse failed:** Verify Parquet schema compatibility and file corruption
+- **Buffer download issues:** Ensure SFTP doesn't apply text transformations to binary files
+- **High event failures:** Inspect rejection report; consider Batch API for very high volumes
+- **429 throttling:** Add small backoff/delay or use Batch API
+- **SFTP connection errors:** Verify credentials, host, port, and `requireAbsolutePaths` setting
+- **Memory issues:** Large Parquet files may need chunked processing (contact support)
+
+---
+
+## 9. Key Takeaways
+
+### Core Features
+- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
+- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
+- ✅ **Processing modes** - per-file (default), chunked, batch
+- ✅ **Unified file processor** - Three service functions: `processFile()`, `sendEvents()`, `writeEventLog()`
+- ✅ **Per-file workflow** - Download → Parse → Map → Send Events → Write Log → Archive
+- ✅ **JobTracker** - Track job lifecycle with KV persistence
+- ✅ **VersoriFileTracker** - Prevent duplicate file processing
+- ✅ **SFTP dispose()** - Always cleanup in finally block
+- ✅ **Buffer import required** - `import { Buffer } from 'node:buffer'` for Deno/Versori
+- ✅ **Binary download** - Parquet requires Buffer format, not string
+- ✅ **No normalization** - Parser returns array directly
+- ✅ **Safe path join** - Handle AWS Transfer Family vs standard OpenSSH
+- ✅ **Per-record error handling** - Continue on individual failures
+- ✅ **Event logs to SFTP** - Write rejection reports for failed events using `Buffer.from()`
+- ✅ **Externalized mapping** - Use JSON config file for field mappings
+- ✅ **File-level error handling** - Don't stop on single file failure
+
+### Production Code Improvements (Applied)
+1. ✅ **Fixed index.ts path** - Uses `./src/workflows/product-ingestion` (correct relative path)
+2. ✅ **Emoji logging** - All workflow steps use emojis (⏰, 🔧, 🔍, 📊, 🔌, 📦, 📤, 📁, ✅, ❌)
+3. ✅ **Execution boundaries** - Clear start/end markers with decorative lines
+4. ✅ **validateConnection: true** - Added optional `validateConnectionOnStart` variable (fail-fast mode)
+5. ✅ **enableFileTracking toggle** - Configurable file tracking via activation variable
+6. ✅ **extractFileName usage** - Uses `getBaseName()` utility to extract filename from path
+7. ✅ **SFTP variable declaration** - Proper `let sftpUsername: string; let sftpPassword: string;` declarations
+8. ✅ **Duration tracking** - All workflows track and log execution duration in milliseconds
+9. ✅ **Error recommendations** - Context-aware error recommendations based on error type
+10. ✅ **Documented new variables** - Added `validateConnectionOnStart` and `enableFileTracking` to activation variables table
+
+---
+
+[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)