@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-csv-product-event.md

@@ -1,2312 +1,2312 @@
----
-template_id: tpl-ingest-s3-csv-to-product-event
-canonical_filename: template-ingestion-s3-csv-product-event.md
-sdk_version: ^0.1.39
-runtime: versori
-versori_run_version: ^0.4.4
-direction: ingestion
-source: s3-csv
-destination: fluent-event-api
-entity: product
-format: csv
-logging: versori
-status: stable
-features:
-  - batched-events
-  - attribute-transformation
-  - memory-management
-  - json-serialization-handling
-  - enhanced-logging
----
-
-# Template: Ingestion - S3 CSV to Product Event
-
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
----
-
-## 📋 Implementation Prompt
-
-```
-I need a Versori scheduled ingestion that:
-
-1) Lists CSV files on S3 (deduplication via moveObject - NO VersoriFileTracker)
-2) Downloads and parses CSV with field validation (using modular processFile() function)
-3) Transforms records with UniversalMapper per mapping JSON
-4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling (using sendEvents() function)
-5) Writes event logs to S3 using writeEventLog() function (non-blocking)
-6) Moves files to processed/ or errors/ prefixes for natural deduplication
-7) Tracks progress with JobTracker and exposes a job-status webhook
-8) Uses native Versori log from context
-9) Uses modular service functions (NO IngestionOrchestrator - removed from SDK)
-10) Imports Buffer from 'node:buffer' for Deno/Versori compatibility
-
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
-
----
-
-## 📋 Template Overview
-
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, file patterns) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from S3 CSV, transforms it, and sends events to the Fluent Commerce Event API.
-
-### What This Template Does
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│          INGESTION WORKFLOW              │
-└─────────────────────────────────────────────────────────────────┘
-
-1. TRIGGER
-  ├─ Scheduled (Cron): Runs automatically every hour
-  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
-  └─ Status Query (Webhook): Check job progress
-
-2. DISCOVER FILES (S3DataSource)
-  ├─ List objects from S3 bucket
-  ├─ Filter by prefix and pattern (products/*.csv)
-  ├─ NO VersoriFileTracker (S3 uses moveObject for deduplication)
-  └─ Sort by last modified
-
-3. DOWNLOAD & PARSE (CSVParserService)
-  ├─ Download object from S3
-  ├─ Parse CSV to JavaScript array
-  ├─ Extract headers from first row
-  └─ Validate CSV structure
-
-4. TRANSFORM (UniversalMapper)
-  ├─ Map CSV fields to Fluent schema
-  ├─ Apply SDK resolvers (trim, uppercase, etc.)
-  ├─ Handle nested objects (price, taxType)
-  ├─ Handle arrays (categoryRefs)
-  └─ Collect transformation errors
-
-5. SEND EVENTS (Event API)
-  ├─ Loop through transformed products
-  ├─ Send UPSERT_PRODUCT event (async)
-  ├─ Track success/failure count
-  └─ Continue on individual failures
-
-6. ARCHIVE (S3DataSource - Natural Deduplication)
-  ├─ Move object to processed/ prefix (success)
-  ├─ Or move to errors/ prefix (failures)
-  ├─ Generate timestamped archive name
-  └─ Files in incoming/ are removed naturally
-
-7. TRACK JOB (JobTracker)
-  ├─ Update job status at each step
-  ├─ Store final result in KV
-  ├─ Enable status queries via webhook
-  └─ Handle errors gracefully
-```
-
-### Key Features
-
-- **Modular service functions** - processFile(), sendEvents(), writeEventLog() for composability
-- Job tracking with status queries via JobTracker
-- Execution modes: scheduled, ad hoc, status query
-- Uses S3DataSource, CSVParserService, UniversalMapper, JobTracker (NO IngestionOrchestrator)
-- Error handling, retry logic, and S3 cleanup
-- **S3 Deduplication:** moveObject to processed/errors prefixes (NO VersoriFileTracker)
-- **Key Difference from SFTP:** Physical file movement vs KV state tracking
-- Event API: Per-record failures don't block other records
-- S3 dispose() in finally block
-- Buffer import for Deno/Versori compatibility
-
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-
-### 📦 Package Information
-
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
-**Versori Runtime:** [@versori/run](https://www.npmjs.com/package/@versori/run) `^0.4.4`
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk@latest
-```
-
----
-
-**Templates are designed for direct deployment; customize via activation variables.**
-
----
-
-## 📦 SDK Imports (Verified - Versori Optimized)
-
-```typescript
-// ✅ VERIFIED IMPORTS - These match actual SDK exports
-import {
- createClient, // Universal client factory
- S3DataSource, // S3 operations with retry logic
- CSVParserService, // CSV parsing
- UniversalMapper, // Field mapping with SDK resolvers
- JobTracker, // Job status tracking
-} from '@fluentcommerce/fc-connect-sdk';
-
-import type { FluentClient, S3DataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
-
-// Versori platform imports
-import { schedule, webhook, http, fn } from '@versori/run';
-```
-
-**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
-
-**✅ VERSORI PLATFORM - Use Native Logs:**
-
-- Use `log` from context: `const { log } = ctx;`
-- Native Versori logs are simpler and automatically integrated with platform monitoring
-
----
-
-## ⚙️ Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-```json
-{
- "fluentRetailerId": "1",
- "eventConcurrency": 1,
- "s3Bucket": "my-product-bucket",
- "s3Region": "us-east-1",
- "s3AccessKeyId": "AKIA...",
- "s3SecretAccessKey": "********",
- "s3IncomingPrefix": "products/incoming/",
- "s3ProcessedPrefix": "products/processed/",
- "s3ErrorPrefix": "products/errors/",
- "s3LogsPrefix": "products/logs/",
- "filePattern": "products_*.csv",
- "catalogueRef": "PC:MASTER:2",
- "catalogueType": "MASTER",
- "eventName": "UPSERT_PRODUCT",
- "eventMode": "async",
- "maxFilesPerRun": 10,
- "enableFileTracking": false,
- "useBatchedEvents": false,
- "maxProductsUnderBatchedEvent": 100,
- "memoryBatchSize": 250
-}
-```
-
-Note: Webhook security is handled by Versori's native connection authentication. The `connection` parameter in webhook definitions ensures only authenticated requests are processed.
-
-### Variable Explanations
-
-| Variable      | Purpose           | Default        | Customization Hints         |
-| ------------------- | ---------------------------- | --------------------- | ----------------------------------- |
-| `fluentRetailerId` | Retailer ID for Event API  | -           | Required - Fluent retailer ID   |
-| `eventConcurrency` | Event sending concurrency  | `1`          | `1` = sequential, `>1` = parallel (3-10 recommended) |
-| `s3Bucket`     | S3 bucket name        | -           | Required - your S3 bucket      |
-| `s3Region`     | S3 region          | `us-east-1`      | AWS region (e.g., us-west-2)    |
-| `s3AccessKeyId`   | AWS access key ID      | -           | Required - AWS credentials     |
-| `s3SecretAccessKey` | AWS secret access key    | -           | Required - AWS credentials     |
-| `s3IncomingPrefix` | Incoming files prefix    | `products/incoming/` | Where new files arrive       |
-| `s3ProcessedPrefix` | Processed files archive   | `products/processed/` | Where files move after success   |
-| `s3ErrorPrefix`   | Failed files directory    | `products/errors/`  | Where files move after errors    |
-| `s3LogsPrefix`   | Event logs directory     | `products/logs/`   | Where event logs are written    |
-| `filePattern`    | File name filter       | `products_*.csv`   | Glob pattern for matching files   |
-| `catalogueRef`   | Product catalogue reference | `PC:MASTER:2`     | Target catalogue in Fluent     |
-| `catalogueType`   | Catalogue type        | `MASTER`       | Usually MASTER or STANDARD     |
-| `eventName`     | Event to send        | `UPSERT_PRODUCT`   | Event name from Rubix        |
-| `eventMode`     | Event processing mode    | `async`        | async (recommended) or sync     |
-| `maxFilesPerRun`  | Max files per execution   | `10`         | Prevent timeout on large batches  |
-| `enableFileTracking` | Enable KV-based file tracking | `false`        | `false` = moveObject only (default), `true` = KV + moveObject |
-| `webhookApiKey`   | API key for webhook security | -           | Required for webhook authentication |
-| `useBatchedEvents` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
-| `maxProductsUnderBatchedEvent` | Products per batch | `100`          | Only used when `useBatchedEvents=true` (default: 100) |
-| `memoryBatchSize`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
-
----
-
-## 🔒 SDK Automatic Behaviors (v0.1.40+)
-
-**The SDK automatically validates and retries for improved reliability:**
-
-###retailerId Validation
-- **SDK validates** `retailerId` before calling `sendEvent()`
-- **Checks:** `event.retailerId || client.retailerId`
-- **If missing:** Throws `"retailerId is required for Event API..."`
-- **Configuration:** Set via `fluentRetailerId` activation variable (recommended)
-
-### 401 Auth Retry
-- **Automatic retry** for platform auth failures (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s)
-- **Applies to:** All `sendEvent()` calls (async and sync modes)
-- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
-
-### 5xx Server Retry
-- **Automatic retry** for transient server errors (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
-- **Protects:** Against Fluent API transient failures
-
-### No Code Changes Required
-- All templates remain compatible
-- Retry logic is automatic and transparent
-- Better error messages guide configuration
-
-**See:** [Event API Guide](./event-api-guide.md) for complete details
-
----
-
-### 📁 S3 Deduplication Pattern (NO VersoriFileTracker)
-
-**CRITICAL: S3 uses physical file movement, NOT VersoriFileTracker**
-
-VersoriFileTracker is **SFTP-specific** and should **NOT** be used with S3. S3 achieves deduplication through physical file movement.
-
-#### How S3 Deduplication Works
-
-**3-Step Process:**
-
-1. **Discovery:** List objects from `s3IncomingPrefix` matching `filePattern`
-2. **Process:** Download, parse, transform, and send events
-3. **Archive:** Move object to `s3ProcessedPrefix` or `s3ErrorPrefix`
-4. **Natural Deduplication:** Once moved, files are no longer in incoming/
-
-#### File Lifecycle Example
-
-```
-INITIAL STATE:
-s3://bucket/products/incoming/products_20250124_001.csv
-
-↓ (workflow discovers file)
-
-PROCESSING:
-- List objects from "products/incoming/"
-- Download: products_20250124_001.csv
-- Parse CSV → Map to events → Send to Event API
-
-↓ (success)
-
-ARCHIVE (moveObject):
-s3://bucket/products/processed/products_20250124_001-20250124T183045.csv
-
-↓ (result)
-
-INCOMING FOLDER NOW EMPTY:
-s3://bucket/products/incoming/ (file is gone - no duplicates possible)
-```
-
-#### Key Differences: S3 vs SFTP
-
-| Aspect          | S3 (This Template)      | SFTP (Other Templates)      |
-| ------------------------ | ----------------------------- | -------------------------------- |
-| **Deduplication Method** | Physical file movement    | VersoriFileTracker (KV state)  |
-| **State Storage**    | None needed          | KV store tracks processed files |
-| **File Tracking**    | ❌ NO VersoriFileTracker   | ✅ VersoriFileTracker required  |
-| **Reprocessing**     | Move files back to incoming/ | Clear KV state or forceReprocess |
-| **Complexity**      | Simpler (no state management) | More complex (state management) |
-
-#### Why S3 Doesn't Need VersoriFileTracker
-
-**S3 moveObject provides natural deduplication:**
-
-- Files are **physically moved** from `incoming/` to `processed/` or `errors/`
-- Once moved, the file **no longer exists** in the incoming prefix
-- Next discovery lists `incoming/` → file is not found → no duplicate processing
-- **No KV state tracking needed** - the S3 bucket structure IS the state
-
-#### Reprocessing Files (If Needed)
-
-To reprocess a file that was already archived:
-
-**Option 1: Manual S3 Move**
-
-```bash
-# Move file back from processed/ to incoming/
-aws s3 mv \
- s3://bucket/products/processed/products_20250124_001-timestamp.csv \
- s3://bucket/products/incoming/products_20250124_001.csv
-```
-
-**Option 2: Webhook with forceReprocess**
-
-```bash
-# Set forceReprocess: true in webhook payload
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "forceReprocess": true,
-  "filePattern": "products_20250124_001.csv"
- }'
-```
-
-**Use cases for reprocessing:**
-
-- Fixing bad data that was previously ingested
-- Reingesting after mapping config changes
-- Testing with production files
-
-#### Summary
-
-✅ **DO:** Use S3 moveObject for deduplication
-✅ **DO:** Move files to processed/ or errors/ prefixes
-❌ **DON'T:** Import VersoriFileTracker in S3 templates
-❌ **DON'T:** Use KV state tracking for S3 files
-✅ **DO:** Use VersoriFileTracker for SFTP templates (different use case)
-
----
-
-### 📁 S3 Prefix vs File Pattern
-
-**Critical:** Folder prefixes and file patterns are configured separately.
-
-- **Folder Prefixes:** Configured via activation variables:
- - `s3IncomingPrefix` - Where new files arrive (e.g., `products/incoming/`)
- - `s3ProcessedPrefix` - Where successful files are archived (e.g., `products/processed/`)
- - `s3ErrorPrefix` - Where failed files are moved (e.g., `products/errors/`)
-
-- **File Pattern:** Configured via `filePattern` activation variable
- - Matches object keys (filenames)
- - Supports wildcards: `*.csv`, `products_*.csv`, `urgent_*.csv`
-
-**How it works:**
-
-```typescript
-// The workflow lists objects like this:
-s3.listObjects({
- prefix: 'products/incoming/', // ← Prefix from activation variable
- pattern: 'products_*.csv', // ← Pattern matches object keys
-});
-```
-
-**Example:** With these settings:
-
-```json
-{
- "s3IncomingPrefix": "products/incoming/",
- "filePattern": "products_*.csv"
-}
-```
-
-The workflow will find objects like:
-
-- ✅ `products/incoming/products_20250124.csv`
-- ✅ `products/incoming/products_urgent.csv`
-- ❌ `products/incoming/orders_20250124.csv` (doesn't match pattern)
-- ❌ `orders/incoming/products_20250124.csv` (wrong prefix)
-
-**Note:** To process files from different prefixes, change `s3IncomingPrefix` (not `filePattern`).
-
----
-
-## 📄 Mapping Configuration
-
-**File:** `config/products.import.csv.json`
-
-```json
-{
- "name": "products.import.csv",
- "version": "1.0.0",
- "description": "CSV → Product Event Mapping",
- "fields": {
-  "ref": {
-   "source": "ref",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "type": {
-   "source": "type",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "STANDARD"
-  },
-  "status": {
-   "source": "status",
-   "resolver": "sdk.uppercase",
-   "defaultValue": "ACTIVE"
-  },
-  "gtin": {
-   "source": "gtin",
-   "resolver": "sdk.trim"
-  },
-  "name": {
-   "source": "name",
-   "required": true,
-   "resolver": "sdk.trim"
-  },
-  "summary": {
-   "source": "summary"
-  },
-  "categoryRefs": {
-   "source": "category_refs",
-   "resolver": "custom.splitByPipe"
-  },
-  "price": {
-   "resolver": "custom.buildPriceArray"
-  },
-  "taxType": {
-   "resolver": "custom.buildTaxType"
-  },
-  "attributes": {
-   "defaultValue": []
-  }
- }
-}
-```
-
-**Custom Resolvers:**
-
-```typescript
-const customResolvers = {
- /**
-  * Split pipe-separated string into array
-  * Example: "CAT1|CAT2|CAT3" → ["CAT1", "CAT2", "CAT3"]
-  */
- 'custom.splitByPipe': (value: string) => {
-  if (!value || value.trim() === '') return [];
-  return value
-   .split('|')
-   .map(v => v.trim())
-   .filter(v => v !== '');
- },
-
- /**
-  * Build price array from flat CSV columns
-  */
- 'custom.buildPriceArray': (value: any, sourceData: any) => {
-  const prices = [];
-  if (sourceData.price_default_value) {
-   prices.push({
-    type: 'DEFAULT',
-    currency: sourceData.price_default_currency || 'USD',
-    value: sourceData.price_default_value,
-   });
-  }
-  if (sourceData.price_special_value) {
-   prices.push({
-    type: 'SPECIAL',
-    currency: sourceData.price_special_currency || 'USD',
-    value: sourceData.price_special_value,
-   });
-  }
-  return prices;
- },
-
- /**
-  * Build tax type object from flat CSV columns
-  */
- 'custom.buildTaxType': (value: any, sourceData: any) => {
-  if (!sourceData.tax_country) return undefined;
-  return {
-   country: sourceData.tax_country,
-   group: sourceData.tax_group,
-   tariff: sourceData.tax_tariff,
-  };
- },
-};
-```
-
-Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
-
----
-
-## Expected CSV Format
-
-```csv
-ref,type,status,gtin,name,summary,category_refs,price_default_currency,price_default_value,price_special_currency,price_special_value,tax_country,tax_group,tax_tariff
-G_PROD_WITH_NO_STANDARD,VARIANT,ACTIVE,MH01-XS-Orange,Chaz Kangeroo Hoodie-XS-Orange main,<p>test short description</p>,STANDARD_CATEGORY,USD,52.000000,USD,9.000000,AU,Tax Group,Tax Tariff
-G_PROD_002,VARIANT,ACTIVE,MH02-M-Blue,Kangeroo Hoodie-M-Blue,<p>Blue variant</p>,STANDARD_CATEGORY|SEASONAL,USD,45.000000,USD,35.000000,AU,Tax Group,Tax Tariff
-G_PROD_003,STANDARD,ACTIVE,MH03-BASE,Base Hoodie Product,<p>Base product</p>,STANDARD_CATEGORY,USD,50.000000,,,AU,Tax Group,Tax Tariff
-```
-
----
-
-## Event Sending Configuration
-
-**Simple Configuration:** One variable controls everything
-
-```json
-{
- "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
-}
-```
-
-**How it works:**
-
-- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
-- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
-- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
-- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
-
-**Configuration Guidelines:**
-
-- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
-- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
-- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
-- **Note:** Fluent API supports concurrent requests - adjust based on your needs
-
-**Why Single Variable?**
-
-- **Simpler:** One variable instead of two (`mode` + `concurrency`)
-- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
-- **Less config:** Fewer activation variables to manage
-- **Flexible:** Easy to tune performance (just change the number)
-
----
-
-## 🔧 Production Reference Implementation
-
-### Processing Approach
-
-This template uses **per-file sequential processing**:
-- Discovers files from S3 incoming prefix
-- Processes each file completely (download → parse → map → send events → archive) before moving to the next
-- Files are processed sequentially to prevent timeout issues and ensure proper error handling
-- Use `maxFilesPerRun` activation variable to limit files per execution (default: 10)
-
-**Why per-file processing?**
-- ✅ Better error isolation - one bad file doesn't affect others
-- ✅ Progressive archiving - files move to processed/errors immediately
-- ✅ Lower memory footprint - one file in memory at a time
-- ✅ Simpler code - no complex batching logic needed
-
-### Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-product-event-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── product.types.ts            # Shared type definitions
-```
-
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
-
----
-
-## Workflow Files
-
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-
-All time-based triggers that run automatically on cron schedules.
-
-#### `src/workflows/scheduled/daily-product-sync.ts`
-
-**Purpose**: Automatic Daily product sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Scheduled Workflow: Daily Product Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: product-sync.service.ts
- */
-export const dailyProductSync = schedule(
-  'product-sync-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv } = ctx;
-    const jobId = `product-sync-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
-    await tracker.updateJob(jobId, { status: 'processing' });
-
-    try {
-      // Reuse shared orchestration logic
-      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
-      await tracker.markCompleted(jobId, result);
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      await tracker.markFailed(jobId, e);
-      return { success: false, jobId, error: e?.message };
-    }
-  })
-);
-```
-
----
-
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-
-All HTTP-based triggers that create webhook endpoints.
-
-#### `src/workflows/webhook/adhoc-product-sync.ts`
-
-**Purpose**: Manual product sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Webhook: Manual Product Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
- * Request body (optional): { filePattern: "urgent_*.csv", maxFiles: 5 }
- *
- * Pattern: webhook().then(http()) - needs Fluent API access
- * Uses shared service: product-sync.service.ts
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const adhocProductSync = webhook('product-sync-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'product-sync-adhoc', // Versori validates API key
-}).then(
-  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, data } = ctx;
-    const jobId = `product-sync-adhoc-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    const filePattern = data?.filePattern as string;
-    const maxFiles = data?.maxFiles as number;
-
-    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-    });
-
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'manual',
-      stage: 'initialization',
-      status: 'queued',
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
-      .then((result) => {
-        log.info('✅ [BACKGROUND] Product sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          filesFailed: result.filesFailed,
-          recordsProcessed: result.recordsProcessed,
-        });
-        return tracker.markCompleted(jobId, result);
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-
-        log.error('❌ [BACKGROUND] Product sync failed', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-        });
-
-        return tracker.markFailed(jobId, errorMessage);
-      });
-
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Product sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-
-#### `src/workflows/webhook/job-status-check.ts`
-
-**Purpose**: Query job status
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
-**Request body**: `{ "jobId": "product-sync-1234567890" }`
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
- * Request body: { "jobId": "product-sync-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const productSyncJobStatus = webhook('product-sync-job-status', {
-  response: { mode: 'sync' },
-  connection: 'product-sync-job-status',
-}).then(
-  fn('status', async ctx => {
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-
-    if (!jobId) {
-      return { success: false, error: 'jobId required' };
-    }
-
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
-
----
-
-### 3. Entry Point (`index.ts`)
-
-**Purpose**: Register all workflows with Versori platform (MemoryInterpreter pattern)
-
-```typescript
-/**
- * ═══════════════════════════════════════════════════════════════
- * 🚀 VERSORI PRODUCT SYNC - ENTRY POINT
- * ═══════════════════════════════════════════════════════════════
- *
- * Entry Point - Registers all workflows with Versori platform
- *
- * PATTERN: MemoryInterpreter
- * - Export all workflows from a single file
- * - Versori automatically discovers and registers exported workflows
- * - Clean separation: scheduled vs webhook triggers
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- * - src/services/            → Shared business logic
- *
- * ═══════════════════════════════════════════════════════════════
- */
-
-// Import scheduled workflows
-import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
-
-// Import webhook workflows
-import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
-import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
-
-// ═══════════════════════════════════════════════════════════════
-// Register all workflows with Versori
-// ═══════════════════════════════════════════════════════════════
-export {
-  // Scheduled (time-based triggers)
-  dailyProductSync,
-
-  // Webhooks (HTTP-based triggers)
-  adhocProductSync,
-  productSyncJobStatus,
-};
-```
-
-**What Gets Exposed:**
-- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
-- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
-- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
-
----
-
-### Adding New Workflows
-
-**To add a scheduled workflow:**
-1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**To add a webhook workflow:**
-1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**Example - Adding hourly delta sync:**
-
-```typescript
-// src/workflows/scheduled/hourly-delta-sync.ts
-export const hourlyDeltaSync = schedule(
-  'product-delta-hourly',
-  '0 * * * *' // Every hour
-).then(
-  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
-    // Delta sync logic (skip BPP)
-    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
-    return result;
-  })
-);
-
-// index.ts (add to imports and exports)
-import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
-export { daily_product_sync, hourlyDeltaSync, ... };
-```
-
----
-## 3. Type Definitions (src/types/product-ingestion.types.ts)
-
-```typescript
-/**
- * Type Definitions for Product Ingestion
- *
- * Centralized type definitions for product ingestion workflow
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Product interface - represents transformed product data
- */
-export interface Product {
- ref: string;
- type?: string;
- status?: string;
- name: string;
- summary?: string;
- gtin?: string;
- catalogue?: { ref?: string };
- attributes?: Record<string, unknown>;
-}
-
-/**
- * Event configuration
- */
-export interface EventConfig {
- eventName: string;
- catalogueRef: string;
- catalogueType: string;
- eventMode: 'async' | 'sync';
-}
-
-/**
- * Event result - tracks success/failure counts
- */
-export interface EventResult {
- eventsSent: number;
- eventsFailed: number;
- errors: Array<{ productRef: string; error: string }>;
-}
-
-/**
- * Process file result
- */
-export interface ProcessFileResult {
- success: boolean;
- products: Product[];
- error?: string;
-}
-
-/**
- * Versori Context Interface
- * Represents the Versori runtime context passed to workflow functions
- */
-export interface VersoriContext {
- log: {
-  info: (message: string, data?: Record<string, unknown>) => void;
-  warn: (message: string, data?: Record<string, unknown>) => void;
-  error: (message: string, data?: Record<string, unknown>) => void;
-  debug?: (message: string, data?: Record<string, unknown>) => void;
- };
- openKv: (namespace: string) => {
-  get: (key: string) => Promise<unknown>;
-  set: (key: string, value: unknown) => Promise<void>;
-  delete: (key: string) => Promise<void>;
- };
- activation: {
-  getVariable: (name: string) => string | undefined;
-  connections?: Record<string, unknown>;
- };
- connections?: Record<string, unknown>;
- data?: unknown;
- fetch?: typeof fetch;
-}
-
-/**
- * Parameters for ingestion workflow
- */
-export interface ProductIngestionParams {
- jobId: string;
- triggeredBy: 'schedule' | 'webhook';
- filePattern?: string;
- maxFiles?: number;
- forceReprocess?: boolean;
- catalogueRef?: string;
- priority?: 'normal' | 'high';
-}
-
-/**
- * Result from ingestion workflow
- */
-export interface ProductIngestionResult {
- success: boolean;
- jobId: string;
- filesProcessed: number;
- filesFailed: number;
- filesSkipped?: number;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- fileResults: FileProcessingResult[];
- error?: string;
-}
-
-/**
- * Per-file processing result
- */
-export interface FileProcessingResult {
- fileName: string;
- success: boolean;
- skipped?: boolean;
- recordsProcessed: number;
- eventsSent: number;
- eventsFailed: number;
- duration: number;
- error?: string;
-}
-```
-
-## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
-
-```typescript
-/**
- * Product File Processor Service
- *
- * Downloads CSV files from S3, parses, and transforms with UniversalMapper.
- * CSV-specific: Returns array directly - no normalization needed.
- */
-
-import {
- S3DataSource,
- CSVParserService,
- UniversalMapper,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
-
-/**
- * Service for processing CSV product files from S3
- */
-export class ProductFileProcessorService {
- constructor(
-  private s3: S3DataSource,
-  private csvParser: CSVParserService,
-  private mapper: UniversalMapper,
-  private catalogueRef: string
- ) {}
-
- /**
-  * Download CSV file from S3, parse, and transform with UniversalMapper
-  *
-  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
-  * - Explicit null assignments after each step
-  * - Finally block guarantees cleanup
-  * - Prevents OOM errors on large CSV files
-  */
- async downloadParseAndTransform(objectKey: string): Promise<ProcessFileResult> {
-  // ✅ CRITICAL: Variables for cleanup tracking
-  let csvContent: string | null = null;
-  let parsed: unknown | null = null;
-
-  try {
-   // STEP 1: Download CSV content from S3
-   csvContent = (await this.s3.downloadObject(objectKey, {
-    encoding: 'utf8',
-   })) as string;
-
-   // STEP 2: Parse CSV with type safety
-   try {
-    parsed = await this.csvParser.parse(csvContent);
-   } catch (parseError: unknown) {
-    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
-    return {
-     success: false,
-     products: [],
-     error: `CSV parse error: ${errorMessage}`,
-    };
-   }
-
-   // ✅ Clear CSV content from memory - no longer needed after parsing
-   csvContent = null;
-
-   // STEP 3: CSV-SPECIFIC - Returns array directly, no normalization needed
-   const rawProducts = Array.isArray(parsed) ? parsed : [];
-
-   if (rawProducts.length === 0) {
-    return {
-     success: false,
-     products: [],
-     error: 'No products found in CSV',
-    };
-   }
-
-   // ✅ Clear parsed data from memory - only need rawProducts now
-   parsed = null;
-
-   // STEP 4: Transform with UniversalMapper
-   // ✅ S3 CSV: Merge context using spread pattern (same as SFTP CSV)
-   const sourceDataWithContext = rawProducts.map(item => ({
-    ...item,
-    $context: { catalogueRef: this.catalogueRef },
-   }));
-
-   const mappingResult = await this.mapper.map(sourceDataWithContext);
-
-   if (!mappingResult.success) {
-    return {
-     success: false,
-     products: [],
-     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
-    };
-   }
-
-   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-     skippedFields: mappingResult.skippedFields,
-     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-    });
-   }
-
-   return {
-    success: true,
-    products: mappingResult.data as Product[],
-   };
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   return {
-    success: false,
-    products: [],
-    error: errorMessage,
-   };
-  } finally {
-   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
-   csvContent = null;
-   parsed = null;
-  }
- }
-}
-```
-
----
-
-## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
-
-```typescript
-/**
- * Event Sender Service
- *
- * Sends product events to Fluent Commerce Event API with per-record error handling.
- * Continues processing on individual failures (Event API best practice).
- * Supports configurable concurrency (sequential or parallel).
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
-
-/**
- * Service for sending events to Fluent Commerce Event API
- *
- * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
- */
-export class EventSenderService {
- constructor(
-  private client: FluentClient,
-  private log?: any  // ✅ Optional logger for progress tracking
- ) {}
-
- /**
-  * Send events to Fluent Commerce Event API with configurable concurrency
-  *
-  * **Performance Characteristics:**
-  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
-  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
-  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
-  *
-  * **Implementation Strategy:**
-  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
-  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
-  * - Both modes: Per-record error tracking (failures don't block other events)
-  *
-  * @param products - Array of products to send as events
-  * @param eventConfig - Event configuration (name, catalogue ref, mode)
-  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
-  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
-  */
- async sendEvents(
-  products: Product[],
-  eventConfig: EventConfig,
-  concurrency: number = 1
- ): Promise<EventResult> {
-  // Validate concurrency (guard against invalid values)
-  const safeConc = Math.max(1, Math.floor(concurrency));
-
-  // Result accumulators
-  let eventsSent = 0;
-  let eventsFailed = 0;
-  const errors: Array<{ productRef: string; error: string }> = [];
-
-  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
-  if (this.log) {
-   this.log.info('📤 Starting event sending', {
-    totalProducts: products.length,
-    concurrency: safeConc,
-    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
-    eventName: eventConfig.eventName,
-    catalogueRef: eventConfig.catalogueRef
-   });
-  }
-
-  // Helper: Build event payload (DRY - reused in both modes)
-  const buildPayload = (product: Product) => ({
-   name: eventConfig.eventName,
-   entityRef: eventConfig.catalogueRef,
-   entityType: 'PRODUCT_CATALOGUE' as const,
-   entitySubtype: eventConfig.catalogueType,
-   rootEntityRef: eventConfig.catalogueRef,
-   rootEntityType: 'PRODUCT_CATALOGUE' as const,
-   attributes: product,
-  });
-
-  // ============================================================================
-  // SEQUENTIAL MODE (concurrency === 1)
-  // ============================================================================
-  if (safeConc === 1) {
-   for (let i = 0; i < products.length; i++) {
-    const product = products[i];
-
-    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
-    if (this.log && i % 10 === 0) {
-     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
-      productRef: product.ref,
-      progress: `${((i / products.length) * 100).toFixed(1)}%`
-     });
-    }
-
-    try {
-     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
-     eventsSent++;
-    } catch (err: unknown) {
-     eventsFailed++;
-     const errorMsg = err instanceof Error ? err.message : String(err);
-     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
-     // Continue processing (failure doesn't block other products)
-    }
-   }
-
-   // ✅ PRODUCTION ENHANCEMENT: Log completion
-   if (this.log) {
-    this.log.info('✅ Sequential event sending completed', {
-     totalProducts: products.length,
-     eventsSent,
-     eventsFailed
-    });
-   }
-
-   return { eventsSent, eventsFailed, errors };
-  }
-
-  // ============================================================================
-  // PARALLEL MODE (concurrency > 1)
-  // ============================================================================
-  const totalChunks = Math.ceil(products.length / safeConc);
-
-  for (let i = 0; i < products.length; i += safeConc) {
-   const chunk = products.slice(i, i + safeConc);
-   const chunkNumber = Math.floor(i / safeConc) + 1;
-
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
-   if (this.log) {
-    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
-     chunkNumber,
-     totalChunks,
-     productsInChunk: chunk.length,
-     productRange: `${i + 1}-${i + chunk.length}`,
-     totalProducts: products.length,
-     progress: `${((i / products.length) * 100).toFixed(1)}%`
-    });
-   }
-
-   // Fire all requests in chunk concurrently
-   const results = await Promise.allSettled(
-    chunk.map(product =>
-     this.client
-      .sendEvent(buildPayload(product), eventConfig.eventMode)
-      .then(() => ({ success: true as const, product }))
-      .catch(error => ({ success: false as const, product, error }))
-    )
-   );
-
-   // Aggregate chunk results into totals
-   let chunkSuccess = 0;
-   let chunkFailed = 0;
-
-   for (const result of results) {
-    if (result.status === 'fulfilled' && result.value.success) {
-     eventsSent++;
-     chunkSuccess++;
-    } else {
-     eventsFailed++;
-     chunkFailed++;
-     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
-     const product = result.status === 'fulfilled' ? result.value.product : null;
-     errors.push({
-      productRef: product?.ref || 'unknown',
-      error: error?.message || String(error) || 'unknown error',
-     });
-    }
-   }
-
-   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
-   if (this.log) {
-    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
-     chunkNumber,
-     totalChunks,
-     chunkSuccess,
-     chunkFailed,
-     totalSentSoFar: eventsSent,
-     totalFailedSoFar: eventsFailed,
-     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
-    });
-   }
-  }
-
-  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
-  if (this.log) {
-   this.log.info('✅ Parallel event sending completed', {
-    totalProducts: products.length,
-    totalChunks,
-    concurrency: safeConc,
-    eventsSent,
-    eventsFailed
-   });
-  }
-
-  return { eventsSent, eventsFailed, errors };
- }
-}
-```
-
----
-
-## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
-
-```typescript
-/**
- * Event Logger Service
- *
- * Writes event processing logs to S3 for audit/debugging.
- * Optional - can be used to create rejection reports.
- */
-
-import { Buffer } from 'node:buffer';
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Service for writing event logs to S3
- */
-export class EventLoggerService {
- constructor(private s3: S3DataSource) {}
-
- /**
-  * Write event log to S3
-  *
-  * @param objectKey - S3 object key for log file
-  * @param logData - Log data to write (JSON format)
-  */
- async writeEventLog(objectKey: string, logData: unknown): Promise<void> {
-  try {
-   const logContent = JSON.stringify(logData, null, 2);
-   await this.s3.uploadFile(objectKey, logContent);
-  } catch (error: unknown) {
-   const errorMessage = error instanceof Error ? error.message : String(error);
-   throw new Error(`Failed to write event log: ${errorMessage}`);
-  }
- }
-}
-```
-
-## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
-
-```typescript
-/**
- * ═══════════════════════════════════════════════════════════════
- * 📦 MAIN INGESTION ORCHESTRATION SERVICE
- * ═══════════════════════════════════════════════════════════════
- *
- * This is the heart of the ingestion workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Discover files on S3
- * 3. Download and parse CSV files
- * 4. Transform data with UniversalMapper
- * 5. Send events to Fluent Commerce
- * 6. Archive processed files
- * 7. Track file processing state (S3 uses moveObject for deduplication)
- * 8. Track job progress with JobTracker
- *
- * KEY DIFFERENCES FOR S3 CSV:
- * - S3DataSource instead of SftpDataSource
- * - CSVParserService instead of XMLParserService
- * - No VersoriFileTracker (S3 uses moveObject for deduplication)
- * - S3 listObjects instead of SFTP listFiles
- * - S3 moveObject instead of SFTP moveFile
- * - CSV returns array directly (no normalization needed)
- * - CSV spread pattern for context injection
- *
- * ═══════════════════════════════════════════════════════════════
- */
-
-import { Buffer } from 'node:buffer';
-import {
- createClient,
- S3DataSource,
- CSVParserService,
- UniversalMapper,
- JobTracker,
- type FluentClient,
- type JobStatus,
-} from '@fluentcommerce/fc-connect-sdk';
-import type {
- ProductIngestionParams,
- ProductIngestionResult,
- FileProcessingResult,
- EventConfig,
-} from '../types/product-ingestion.types';
-import { ProductFileProcessorService } from './product-file-processor.service';
-import { EventSenderService } from './event-sender.service';
-import { EventLoggerService } from './event-logger.service';
-import { timestampedName } from '../utils/s3-path.utils';
-
-import mappingConfig from '../../config/products.import.csv.json' with { type: 'json' };
-
-// Custom resolvers for CSV mapping (prices, categories, taxType)
-const customResolvers = {
- 'custom.splitByPipe': (value: string) => {
-  if (!value || value.trim() === '') return [];
-  return value
-   .split('|')
-   .map(v => v.trim())
-   .filter(v => v !== '');
- },
- 'custom.buildPriceArray': (value: any, sourceData: any) => {
-  const prices = [];
-  if (sourceData.price_default_value) {
-   prices.push({
-    type: 'DEFAULT',
-    currency: sourceData.price_default_currency || 'USD',
-    value: sourceData.price_default_value,
-   });
-  }
-  if (sourceData.price_special_value) {
-   prices.push({
-    type: 'SPECIAL',
-    currency: sourceData.price_special_currency || 'USD',
-    value: sourceData.price_special_value,
-   });
-  }
-  return prices;
- },
- 'custom.buildTaxType': (value: any, sourceData: any) => {
-  if (!sourceData.tax_country) return undefined;
-  return {
-   country: sourceData.tax_country,
-   group: sourceData.tax_group,
-   tariff: sourceData.tax_tariff,
-  };
- },
-};
-
-/**
- * Query job status from KV store
- */
-export async function getJobStatus(
- kv: ReturnType<VersoriContext['openKv']>,
- jobId: string,
- log: VersoriContext['log']
-): Promise<JobStatus | undefined> {
- try {
-  const tracker = new JobTracker(kv, log);
-  return await tracker.getJob(jobId);
- } catch (error: unknown) {
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  log.error('Failed to get job status', {
-   jobId,
-   message: errorMessage,
-   stack: error instanceof Error ? error.stack : undefined,
-   errorType: error instanceof Error ? error.constructor.name : 'Error',
-  });
-  return undefined;
- }
-}
-
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * This function implements the complete ingestion workflow in 8 steps.
- */
-export async function executeProductIngestion(
- ctx: VersoriContext,
- params: ProductIngestionParams
-): Promise<ProductIngestionResult> {
-
- const { log, openKv, activation } = ctx;
- const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
-
- const kv = openKv(':project:');
- const tracker = new JobTracker(kv, log);
-
- const startTime = Date.now();
- const fileResults: FileProcessingResult[] = [];
-
- // ⚠️ CRITICAL: Declare S3 outside try block for disposal pattern
- let s3: S3DataSource | undefined;
-
- try {
-  // ═══════════════════════════════════════════════════════════
-  // 🚀 STEP 1/8: Initialize Job Tracking
-  // ═══════════════════════════════════════════════════════════
-  log.info('🚀 [STEP 1/8] Initializing job tracking', { jobId });
-
-  await tracker.createJob(jobId, {
-   triggeredBy,
-   filePattern: filePattern || 'default',
-   maxFiles: maxFiles || 'unlimited',
-   forceReprocess: !!forceReprocess,
-   startTime,
-  });
-
-  // ═══════════════════════════════════════════════════════════
-  // 🔧 STEP 2/8: Initialize Fluent Client & S3 Connection
-  // ═══════════════════════════════════════════════════════════
-  log.info('🔧 [STEP 2/8] Initializing Fluent Commerce client and S3', { jobId });
-
-  const client = await createClient(ctx);
-
-  if (!client) {
-   const error = 'Failed to create Fluent Commerce client';
-   log.error(`❌ ${error}`);
-   throw new Error(error);
-  }
-
-  log.info('✅ [Client] Fluent Commerce client created successfully');
-
-  // ✅ CRITICAL: Set retailerId for Event API calls
-  const fluentRetailerId = activation.getVariable('fluentRetailerId');
-  if (!fluentRetailerId) {
-   const error = 'fluentRetailerId is required for Event API calls';
-   log.error(`❌ ${error}`, {
-    recommendation: 'Set fluentRetailerId in activation variables',
-    variable: 'fluentRetailerId',
-   });
-   throw new Error(error);
-  }
-  client.setRetailerId(fluentRetailerId);
-  log.info('✅ [Client] RetailerId set for Event API', { retailerId: fluentRetailerId });
-
-  // Get S3 configuration from activation variables
-  const s3Config = {
-   bucket: activation.getVariable('s3Bucket'),
-   region: activation.getVariable('s3Region') || 'us-east-1',
-   accessKeyId: activation.getVariable('s3AccessKeyId'),
-   secretAccessKey: activation.getVariable('s3SecretAccessKey'),
-   incomingPrefix: activation.getVariable('s3IncomingPrefix') || 'products/incoming/',
-   processedPrefix: activation.getVariable('s3ProcessedPrefix') || 'products/processed/',
-   errorPrefix: activation.getVariable('s3ErrorPrefix') || 'products/errors/',
-   logsPrefix: activation.getVariable('s3LogsPrefix') || 'products/logs/',
-   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.csv'
-  };
-
-  // Validate S3 config
-  if (!s3Config.bucket) {
-   const error = 'S3 configuration incomplete: missing bucket';
-   log.error(`❌ ${error}`, {
-    recommendation: 'Set s3Bucket in activation variables',
-    variable: 's3Bucket',
-   });
-   throw new Error(error);
-  }
-
-  if (!s3Config.accessKeyId || !s3Config.secretAccessKey) {
-   const error = 'S3 configuration incomplete: missing accessKeyId or secretAccessKey';
-   log.error(`❌ ${error}`, {
-    recommendation: 'Set s3AccessKeyId and s3SecretAccessKey in activation variables',
-    variables: ['s3AccessKeyId', 's3SecretAccessKey'],
-   });
-   throw new Error(error);
-  }
-
-  // Initialize S3 data source
-  s3 = new S3DataSource(
-   {
-    type: 'S3_CSV',
-    connectionId: 's3-product-ingestion',
-    name: 'Product Ingestion S3',
-    s3Config: {
-     bucket: s3Config.bucket,
-     region: s3Config.region,
-     accessKeyId: s3Config.accessKeyId,
-     secretAccessKey: s3Config.secretAccessKey,
-    },
-    validateConnection: true, // ✅ Enable connection validation
-   },
-   log
-  );
-
-  try {
-   // Validate S3 connection
-   log.info('🔍 [S3] Validating S3 connection...');
-   await s3.validateConnection();
-   log.info('✅ [S3] S3 connection validated successfully');
-
-   // ═══════════════════════════════════════════════════════════
-   // 🔍 STEP 3/8: Discover Files on S3
-   // ═══════════════════════════════════════════════════════════
-   log.info('🔍 [STEP 3/8] Discovering files on S3', {
-    jobId,
-    prefix: s3Config.incomingPrefix,
-    filePattern: s3Config.filePattern
-   });
-
-   await tracker.updateJob(jobId, {
-    status: 'processing',
-    stage: 'file_discovery',
-    message: 'Discovering files on S3'
-   });
-
-   // List objects from S3
-   const allFiles = await s3.listObjects({
-    prefix: s3Config.incomingPrefix,
-    pattern: s3Config.filePattern
-   });
-
-   log.info(`✅ [Discovery] Found ${allFiles.length} files matching pattern`);
-
-   // Apply maxFiles limit if specified
-   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
-
-   if (filesToProcess.length === 0) {
-    log.info('⚠️ [Discovery] No files to process');
-
-    await tracker.markCompleted(jobId, {
-     filesProcessed: 0,
-     message: 'No files found to process',
-     duration: Date.now() - startTime,
-    });
-
-    return {
-     success: true,
-     jobId,
-     filesProcessed: 0,
-     filesFailed: 0,
-     recordsProcessed: 0,
-     eventsSent: 0,
-     eventsFailed: 0,
-     fileResults: [],
-     duration: Date.now() - startTime,
-    };
-   }
-
-   log.info(`📄 [Discovery] Processing ${filesToProcess.length} files`);
-
-   // Initialize services
-   const csvParser = new CSVParserService();
-   const mapper = new UniversalMapper(mappingConfig, { customResolvers });
-   
-   // Get event configuration
-   const eventConfig: EventConfig = {
-    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
-    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
-    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
-    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
-   };
-
-   // Get event sending configuration
-   const eventConcurrency = Math.max(
-    1,
-    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
-   );
-
-   log.info(`Event concurrency: ${eventConcurrency}`, {
-    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
-    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
-   });
-
-   // Initialize class-based services
-   const fileProcessor = new ProductFileProcessorService(
-    s3,
-    csvParser,
-    mapper,
-    eventConfig.catalogueRef
-   );
-   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
-   const eventSender = new EventSenderService(client, log);
-   const eventLogger = new EventLoggerService(s3);
-
-   // ═══════════════════════════════════════════════════════════
-   // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
-   // ═══════════════════════════════════════════════════════════
-
-   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
-    const file = filesToProcess[fileIndex];
-    const fileStartTime = Date.now();
-
-    // Use object key from S3 listing
-    const objectKey = file.key || (file as any).name;
-    const baseName = objectKey.split('/').pop() || objectKey;
-
-    // ═══════════════════════════════════════════════════════════
-    log.info(`\n${'═'.repeat(60)}`);
-    log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing: ${baseName}`);
-    log.info(`${'═'.repeat(60)}`);
-    // ═══════════════════════════════════════════════════════════
-
-    try {
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'downloading',
-      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
-     });
-
-     // ═══════════════════════════════════════════════════════════
-     // 📥 STEP 4: Process File (Download + Parse + Map)
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📥 [STEP 4/8] Processing file: ${baseName}`);
-
-     // Process file: download → parse → transform
-     const fileResult = await fileProcessor.downloadParseAndTransform(objectKey);
-
-     if (!fileResult.success || fileResult.products.length === 0) {
-      // Move failed file to errors and record result
-      const archivedName = timestampedName(baseName);
-      const errorKey = `${s3Config.errorPrefix}${archivedName}`;
-      await s3.moveObject(objectKey, errorKey);
-
-      fileResults.push({
-       fileName: baseName,
-       success: false,
-       recordsProcessed: fileResult.products.length,
-       eventsSent: 0,
-       eventsFailed: 0,
-       duration: Date.now() - fileStartTime,
-       error: fileResult.error || 'Processing failed'
-      });
-
-      continue;
-     }
-
-     // ═══════════════════════════════════════════════════════════
-     // 🚀 STEP 5: Send Events
-     // ═══════════════════════════════════════════════════════════
-     log.info(`🚀 [STEP 5/8] Sending events for ${baseName}`);
-
-     // Extract context for progress logging
-     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
-     const eventType = eventConfig.eventName || 'UPSERT_PRODUCT';
-
-     log.info(`[EventSender] Sending events for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventType,
-      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
-      sampleProductRefs: sampleProductRefs.join(', '),
-      eventMode: eventConfig.eventMode || 'async'
-     });
-
-     const eventResult = await eventSender.sendEvents(
-       fileResult.products,
-       eventConfig,
-       eventConcurrency
-     );
-
-     const eventsSent = eventResult.eventsSent;
-     const eventsFailed = eventResult.eventsFailed;
-
-     log.info(`✅ [Events] Sent ${eventsSent} events for ${baseName}`, {
-       successful: eventsSent,
-       failed: eventsFailed
-     });
-
-     // Completion logging with summary
-     log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
-      totalProducts: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
-      eventType,
-      duration: Date.now() - fileStartTime,
-     });
-
-     // ═══════════════════════════════════════════════════════════
-     // 📦 STEP 6: Archive File
-     // ═══════════════════════════════════════════════════════════
-     log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
-
-     await tracker.updateJob(jobId, {
-      status: 'processing',
-      stage: 'archiving',
-      message: `Archiving ${baseName}`
-     });
-
-     // Conditionally archive: errors → errorPrefix, else → processedPrefix (timestamped)
-     // ✅ S3: Uses moveObject for deduplication (no VersoriFileTracker needed)
-     const archivedName = timestampedName(baseName);
-     const targetPrefix = eventsFailed > 0 ? s3Config.errorPrefix : s3Config.processedPrefix;
-     const targetKey = `${targetPrefix}${archivedName}`;
-     await s3.moveObject(objectKey, targetKey);
-
-     log.info(`✅ [Archive] File archived successfully: ${baseName}`, { targetKey });
-
-     // Optional: Write event log for audit/debugging
-     if (eventsFailed > 0) {
-      const logKey = `${s3Config.logsPrefix}${baseName.replace(/\.csv$/i, '')}-event-log.json`;
-      await eventLogger.writeEventLog(logKey, {
-       fileName: baseName,
-       totalRecords: fileResult.products.length,
-       eventsSent,
-       eventsFailed,
-       errors: eventResult.errors,
-       processedAt: new Date().toISOString()
-      });
-     }
-
-     // Store file result
-     fileResults.push({
-      fileName: baseName,
-      success: true,
-      recordsProcessed: fileResult.products.length,
-      eventsSent,
-      eventsFailed,
-      duration: Date.now() - fileStartTime
-     });
-
-    } catch (error: unknown) {
-     // ❌ Enhanced error logging with recommendations
-     const errorMessage = error instanceof Error ? error.message : String(error);
-     const errorDetails = {
-      message: errorMessage,
-      stack: error instanceof Error ? error.stack : undefined,
-      fileName: (error as any)?.fileName,
-      lineNumber: (error as any)?.lineNumber,
-      originalError: (error as any)?.context?.originalError?.message,
-      errorType: error instanceof Error ? error.name : 'Error',
-      recommendation: 'Check file format, S3 permissions, and mapping configuration',
-     };
-     log.error(`❌ [Error] Failed to process file ${baseName}:`, errorDetails);
-
-     // Best-effort: move to error archive on unexpected failure
-     try {
-      const archivedName = timestampedName(baseName);
-      const errorKey = `${s3Config.errorPrefix}${archivedName}`;
-      await s3.moveObject(objectKey, errorKey);
-     } catch (moveError: unknown) {
-      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
-      log.error('Could not archive failed file', {
-       file: baseName,
-       message: moveErrorMessage,
-       stack: moveError instanceof Error ? moveError.stack : undefined,
-       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
-      });
-     }
-
-     fileResults.push({
-      fileName: baseName,
-      success: false,
-      recordsProcessed: 0,
-      eventsSent: 0,
-      eventsFailed: 0,
-      duration: Date.now() - fileStartTime,
-      error: errorMessage
-     });
-    }
-   }
-
-   // ═══════════════════════════════════════════════════════════
-   // ✅ STEP 8: Complete Job & Calculate Totals
-   // ═══════════════════════════════════════════════════════════
-   log.info('✅ [STEP 8/8] Completing job and calculating totals', { jobId });
-
-   const filesProcessed = fileResults.filter(r => r.success).length;
-   const filesFailed = fileResults.filter(r => !r.success).length;
-   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
-   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
-   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
-   const totalDuration = Date.now() - startTime;
-
-   await tracker.markCompleted(jobId, {
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: totalDuration
-   });
-
-   // ═══════════════════════════════════════════════════════════
-   log.info(`\n${'═'.repeat(60)}`);
-   log.info('✅ [COMPLETE] Ingestion completed successfully');
-   log.info(`${'═'.repeat(60)}`);
-   // ═══════════════════════════════════════════════════════════
-
-   log.info('[Summary]', {
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    duration: `${totalDuration}ms`,
-   });
-
-   return {
-    success: true,
-    jobId,
-    filesProcessed,
-    filesFailed,
-    recordsProcessed: totalRecordsProcessed,
-    eventsSent: totalEventsSent,
-    eventsFailed: totalEventsFailed,
-    fileResults
-   };
-
-  } finally {
-   // ⚠️ CRITICAL: Always dispose S3 connection
-   if (s3) {
-    await s3.dispose();
-    log.info('✅ [Cleanup] S3 connection disposed');
-   }
-  }
-
- } catch (error: unknown) {
-  const errorMessage = error instanceof Error ? error.message : String(error);
-  const errorStack = error instanceof Error ? error.stack : undefined;
-  log.error('❌ [FATAL] Ingestion workflow failed', {
-   jobId,
-   error: errorMessage,
-   stack: errorStack,
-   recommendation: 'Check logs for detailed error information and verify all activation variables',
-  });
-
-  // Try to mark job as failed, but don't let tracking errors mask workflow error
-  try {
-   await tracker.markFailed(jobId, error);
-  } catch (trackingError: unknown) {
-   const trackingErrorMessage =
-    trackingError instanceof Error ? trackingError.message : String(trackingError);
-   log.warn('Failed to mark job as failed in tracker', {
-    jobId,
-    trackingError: trackingErrorMessage,
-   });
-  }
-
-  return {
-   success: false,
-   jobId,
-   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
-   filesFailed: fileResults.filter(r => !r.success).length,
-   filesSkipped: fileResults.filter(r => r.skipped).length,
-   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
-   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
-   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
-   fileResults,
-   error: errorMessage,
-  };
- } finally {
-  // ⚠️ CRITICAL: Ensure S3 is disposed even if outer error occurs
-  if (s3) {
-   try {
-    await s3.dispose();
-    log.info('✅ [Cleanup] S3 connection disposed (outer finally)');
-   } catch (disposeError: unknown) {
-    const disposeErrorMessage =
-     disposeError instanceof Error ? disposeError.message : String(disposeError);
-    log.warn('⚠️ [Warning] Error disposing S3 in outer finally', { error: disposeErrorMessage });
-   }
-  }
- }
-}
-```
-
----
-
-## 8. Utility Functions (src/utils/)
-
-### S3 Path Helpers (src/utils/s3-path.utils.ts)
-
-```typescript
-/**
- * S3 Path Utilities
- *
- * Helper functions for S3 path operations.
- */
-
-/**
- * Generate timestamped filename for archival
- *
- * Adds ISO timestamp to filename before extension for unique archival.
- * Handles CSV files specifically but can be adapted for other formats.
- *
- * @param name - Original filename
- * @returns Filename with timestamp appended
- *
- * @example
- * ```typescript
- * timestampedName('products.csv')
- * // Returns: 'products-2025-11-01T18-30-45-123Z.csv'
- * ```
- */
-export function timestampedName(name: string): string {
- const ts = new Date().toISOString().replace(/[:.]/g, '-');
- const base = name.replace(/\.csv$/i, '');
- return `${base}-${ts}.csv`;
-}
-```
-
-### Job ID Generator (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking ingestion runs.
- * Format: {PREFIX}_{ENTITY}_{DATE}_{TIME}_{RANDOM}
- * Example: SCHEDULED_PROD_20250124_183045_abc123
- */
-
-/**
- * Generate unique job ID
- *
- * @param prefix - Job prefix (SCHEDULED, ADHOC, etc.)
- * @param entity - Entity type (PROD, INV, etc.)
- * @returns Unique job ID string
- */
-export function generateJobId(prefix: string, entity: string): string {
- const now = new Date();
- const date = now.toISOString().slice(0, 10).replace(/-/g, '');
- const time = now.toISOString().slice(11, 19).replace(/:/g, '');
- const random = Math.random().toString(36).slice(2, 8).toUpperCase();
- return `${prefix}_${entity}_${date}_${time}_${random}`;
-}
-```
-
----
-
-## 📄 Mapping Configuration
-
-**File:** `config/products.import.csv.json`
-
-```json
-{
- "name": "products.import.csv",
- "version": "1.2.0",
- "description": "CSV → Product Event Mapping",
- "fields": {
-  "ref": {
-   "source": "ref",
-   "required": true,
-   "resolver": "sdk.trim",
-   "comment": "Product reference from CSV"
-  },
-  "type": {
-   "source": "type",
-   "required": true,
-   "resolver": "sdk.uppercase",
-   "comment": "Product type (STANDARD, VARIANT)"
-  },
-  "status": {
-   "source": "status",
-   "required": true,
-   "resolver": "sdk.uppercase",
-   "comment": "Product status (ACTIVE, INACTIVE)"
-  },
-  "name": {
-   "source": "name",
-   "required": true,
-   "resolver": "sdk.trim",
-   "comment": "Product name/title"
-  },
-  "summary": {
-   "source": "summary",
-   "required": false,
-   "comment": "Product short description"
-  },
-  "gtin": {
-   "source": "gtin",
-   "required": false,
-   "resolver": "sdk.trim",
-   "comment": "Global Trade Item Number (barcode)"
-  },
-  "catalogue.ref": {
-   "source": "$context.catalogueRef",
-   "required": false,
-   "comment": "Catalogue reference (injected from context, NOT in CSV)"
-  },
-  "metadata.source": {
-   "value": "S3_CSV",
-   "required": false,
-   "comment": "Static value - identifies data source (NOT in CSV)"
-  },
-  "categoryRefs": {
-   "source": "category_refs",
-   "resolver": "custom.splitByPipe",
-   "comment": "Pipe-separated categories (custom resolver)"
-  },
-  "price": {
-   "resolver": "custom.buildPriceArray",
-   "comment": "Build price array from default/special fields (custom resolver)"
-  },
-  "taxType": {
-   "resolver": "custom.buildTaxType",
-   "comment": "Build taxType object from country/group/tariff (custom resolver)"
-  }
- }
-}
-```
-
-**Note:** Custom resolvers (`custom.splitByPipe`, `custom.buildPriceArray`, `custom.buildTaxType`) are defined in the Main Orchestration Service.
-
----
-
-## 9. Package Configuration
-
-### `package.json`
-
-```json
-{
- "name": "s3-csv-product-event-ingestion",
- "version": "1.2.0",
- "description": "S3 CSV Product Event Ingestion Connector",
- "type": "module",
- "main": "src/index.ts",
- "dependencies": {
-  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-  "@versori/run": "^1.0.0"
- },
- "devDependencies": {
-  "@types/node": "^20.0.0",
-  "typescript": "^5.0.0"
- }
-}
-```
-
----
-
-## 6. Deployment Instructions
-
-### Deploy to Versori
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Test locally (if using Versori CLI)
-npm run dev
-
-# 3. Deploy to Versori platform
-npm run deploy
-```
-
-### Configure Activation Variables
-
-In Versori platform settings, configure all variables listed in the Activation Variables section above.
-
----
-
-## 7. Testing
-
-### Test Scheduled Ingestion
-
-Upload a test CSV file to S3 incoming prefix and wait for the scheduled run.
-
-**Check logs:**
-
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and S3
-[STEP 3/8] Discovering files on S3
-[FILE 1/1] Processing file: products_20250124.csv
-[STEP 4/8] Downloading and parsing: products_20250124.csv
-[STEP 5/8] Transforming 5 products from products_20250124.csv
-[STEP 6/8] Sending 5 events to Fluent Commerce
-[STEP 7/8] Archiving file: products_20250124.csv
-[STEP 8/8] Completing job and calculating totals
-```
-
-### Test Ad hoc Ingestion
-
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{}'
-
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "filePattern": "urgent_*.csv", }'
-```
-
-### Test Job Status Query
-
-```bash
-curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
- -H "X-API-Key: your-secret-key" \
- -H "Content-Type: application/json" \
- -d '{
-  "jobId": "ADHOC_PROD_20251024_183045_abc123"
- }'
-```
-
----
-
-## Monitoring
-
-### Success Response
-
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 50,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 50,
-   "eventsFailed": 0
-  }
- ],
- "duration": 12345
-}
-```
-
-### Partial Success Response
-
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "totalRecords": 50,
- "eventsSent": 45,
- "eventsFailed": 5,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": true,
-   "recordsProcessed": 50,
-   "eventsSent": 45,
-   "eventsFailed": 5,
-   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
-  }
- ],
- "duration": 12345
-}
-```
-
-### Error Response
-
-```json
-{
- "success": false,
- "filesProcessed": 0,
- "filesFailed": 1,
- "totalRecords": 0,
- "eventsSent": 0,
- "eventsFailed": 0,
- "results": [
-  {
-   "file": "products_2025-01-22.csv",
-   "success": false,
-   "error": "CSV parse error: Invalid structure"
-  }
- ],
- "duration": 876
-}
-```
-
-### Monitoring Metrics
-
-Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
-
-- **Files Processed** - Total files successfully processed
-- **Events Sent** - Total events sent to Fluent Commerce
-- **Events Failed** - Events that failed (check rejection reports)
-- **Processing Duration** - Time taken for complete workflow
-- **Rate Limiting** - Watch for 429 errors indicating throttling
-
-Use the status webhook for dashboards and automated monitoring.
-
----
-
-- **No files found:** Check `s3IncomingPrefix` and `filePattern` (object keys only)
-- **CSV parse failed:** Move to errors/; validate CSV structure and encoding
-- **High event failures:** Inspect logs; consider Batch API for very high volumes
-- **429 throttling:** Add small backoff/delay or use Batch API
-- **S3 connection errors:** Verify credentials, bucket, region, and permissions
-
----
-
-## 9. Key Takeaways
-
-- ✅ **Modular service functions** - processFile(), sendEvents(), writeEventLog() keep code composable
-- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
-- ✅ **Buffer import required** - `import { Buffer } from 'node:buffer';` for Deno/Versori compatibility
-- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
-- ✅ **JobTracker** - Track job lifecycle with KV persistence
-- ❌ **NO VersoriFileTracker** - S3 uses moveObject for deduplication (SFTP-specific only)
-- ❌ **NO IngestionOrchestrator** - Removed from SDK - use inline modular services instead
-- ✅ **S3 deduplication** - Physical file movement to processed/errors prefixes
-- ✅ **S3 dispose()** - Always cleanup in finally block
-- ✅ **S3 method names** - listObjects, downloadObject, moveObject (not \*File)
-- ✅ **Per-record error handling** - Continue on individual failures
-- ✅ **Externalized mapping** - Use JSON config file for field mappings
-- ✅ **File-level error handling** - Don't stop on single file failure
-- ✅ **S3 vs SFTP** - Different deduplication strategies (moveObject vs KV state)
-- ✅ **uploadFile signature** - Uses Buffer.from(content, 'utf8') for text content
-
----
-
-[← Back to Versori Event API Templates](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) | [Versori Platform Guide →](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+---
+template_id: tpl-ingest-s3-csv-to-product-event
+canonical_filename: template-ingestion-s3-csv-product-event.md
+sdk_version: ^0.1.39
+runtime: versori
+versori_run_version: ^0.4.4
+direction: ingestion
+source: s3-csv
+destination: fluent-event-api
+entity: product
+format: csv
+logging: versori
+status: stable
+features:
+  - batched-events
+  - attribute-transformation
+  - memory-management
+  - json-serialization-handling
+  - enhanced-logging
+---
+
+# Template: Ingestion - S3 CSV to Product Event
+
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+---
+
+## 📋 Implementation Prompt
+
+```
+I need a Versori scheduled ingestion that:
+
+1) Lists CSV files on S3 (deduplication via moveObject - NO VersoriFileTracker)
+2) Downloads and parses CSV with field validation (using modular processFile() function)
+3) Transforms records with UniversalMapper per mapping JSON
+4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling (using sendEvents() function)
+5) Writes event logs to S3 using writeEventLog() function (non-blocking)
+6) Moves files to processed/ or errors/ prefixes for natural deduplication
+7) Tracks progress with JobTracker and exposes a job-status webhook
+8) Uses native Versori log from context
+9) Uses modular service functions (NO IngestionOrchestrator - removed from SDK)
+10) Imports Buffer from 'node:buffer' for Deno/Versori compatibility
+
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+
+---
+
+## 📋 Template Overview
+
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, file patterns) are configured via activation variables. Data shape and logic (mapping JSON, CSV structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from S3 CSV, transforms it, and sends events to the Fluent Commerce Event API.
+
+### What This Template Does
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│          INGESTION WORKFLOW              │
+└─────────────────────────────────────────────────────────────────┘
+
+1. TRIGGER
+  ├─ Scheduled (Cron): Runs automatically every hour
+  ├─ Ad hoc (Webhook): Manual trigger for immediate processing
+  └─ Status Query (Webhook): Check job progress
+
+2. DISCOVER FILES (S3DataSource)
+  ├─ List objects from S3 bucket
+  ├─ Filter by prefix and pattern (products/*.csv)
+  ├─ NO VersoriFileTracker (S3 uses moveObject for deduplication)
+  └─ Sort by last modified
+
+3. DOWNLOAD & PARSE (CSVParserService)
+  ├─ Download object from S3
+  ├─ Parse CSV to JavaScript array
+  ├─ Extract headers from first row
+  └─ Validate CSV structure
+
+4. TRANSFORM (UniversalMapper)
+  ├─ Map CSV fields to Fluent schema
+  ├─ Apply SDK resolvers (trim, uppercase, etc.)
+  ├─ Handle nested objects (price, taxType)
+  ├─ Handle arrays (categoryRefs)
+  └─ Collect transformation errors
+
+5. SEND EVENTS (Event API)
+  ├─ Loop through transformed products
+  ├─ Send UPSERT_PRODUCT event (async)
+  ├─ Track success/failure count
+  └─ Continue on individual failures
+
+6. ARCHIVE (S3DataSource - Natural Deduplication)
+  ├─ Move object to processed/ prefix (success)
+  ├─ Or move to errors/ prefix (failures)
+  ├─ Generate timestamped archive name
+  └─ Files in incoming/ are removed naturally
+
+7. TRACK JOB (JobTracker)
+  ├─ Update job status at each step
+  ├─ Store final result in KV
+  ├─ Enable status queries via webhook
+  └─ Handle errors gracefully
+```
+
+### Key Features
+
+- **Modular service functions** - processFile(), sendEvents(), writeEventLog() for composability
+- Job tracking with status queries via JobTracker
+- Execution modes: scheduled, ad hoc, status query
+- Uses S3DataSource, CSVParserService, UniversalMapper, JobTracker (NO IngestionOrchestrator)
+- Error handling, retry logic, and S3 cleanup
+- **S3 Deduplication:** moveObject to processed/errors prefixes (NO VersoriFileTracker)
+- **Key Difference from SFTP:** Physical file movement vs KV state tracking
+- Event API: Per-record failures don't block other records
+- S3 dispose() in finally block
+- Buffer import for Deno/Versori compatibility
+
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+
+### 📦 Package Information
+
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk) `^0.1.30`
+**Versori Runtime:** [@versori/run](https://www.npmjs.com/package/@versori/run) `^0.4.4`
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk@latest
+```
+
+---
+
+**Templates are designed for direct deployment; customize via activation variables.**
+
+---
+
+## 📦 SDK Imports (Verified - Versori Optimized)
+
+```typescript
+// ✅ VERIFIED IMPORTS - These match actual SDK exports
+import {
+ createClient, // Universal client factory
+ S3DataSource, // S3 operations with retry logic
+ CSVParserService, // CSV parsing
+ UniversalMapper, // Field mapping with SDK resolvers
+ JobTracker, // Job status tracking
+} from '@fluentcommerce/fc-connect-sdk';
+
+import type { FluentClient, S3DataSourceConfig } from '@fluentcommerce/fc-connect-sdk';
+
+// Versori platform imports
+import { schedule, webhook, http, fn } from '@versori/run';
+```
+
+**Note:** All imports are from actual SDK exports - this code compiles and runs as-is.
+
+**✅ VERSORI PLATFORM - Use Native Logs:**
+
+- Use `log` from context: `const { log } = ctx;`
+- Native Versori logs are simpler and automatically integrated with platform monitoring
+
+---
+
+## ⚙️ Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+```json
+{
+ "fluentRetailerId": "1",
+ "eventConcurrency": 1,
+ "s3Bucket": "my-product-bucket",
+ "s3Region": "us-east-1",
+ "s3AccessKeyId": "AKIA...",
+ "s3SecretAccessKey": "********",
+ "s3IncomingPrefix": "products/incoming/",
+ "s3ProcessedPrefix": "products/processed/",
+ "s3ErrorPrefix": "products/errors/",
+ "s3LogsPrefix": "products/logs/",
+ "filePattern": "products_*.csv",
+ "catalogueRef": "PC:MASTER:2",
+ "catalogueType": "MASTER",
+ "eventName": "UPSERT_PRODUCT",
+ "eventMode": "async",
+ "maxFilesPerRun": 10,
+ "enableFileTracking": false,
+ "useBatchedEvents": false,
+ "maxProductsUnderBatchedEvent": 100,
+ "memoryBatchSize": 250
+}
+```
+
+Note: Webhook security is handled by Versori's native connection authentication. The `connection` parameter in webhook definitions ensures only authenticated requests are processed.
+
+### Variable Explanations
+
+| Variable      | Purpose           | Default        | Customization Hints         |
+| ------------------- | ---------------------------- | --------------------- | ----------------------------------- |
+| `fluentRetailerId` | Retailer ID for Event API  | -           | Required - Fluent retailer ID   |
+| `eventConcurrency` | Event sending concurrency  | `1`          | `1` = sequential, `>1` = parallel (3-10 recommended) |
+| `s3Bucket`     | S3 bucket name        | -           | Required - your S3 bucket      |
+| `s3Region`     | S3 region          | `us-east-1`      | AWS region (e.g., us-west-2)    |
+| `s3AccessKeyId`   | AWS access key ID      | -           | Required - AWS credentials     |
+| `s3SecretAccessKey` | AWS secret access key    | -           | Required - AWS credentials     |
+| `s3IncomingPrefix` | Incoming files prefix    | `products/incoming/` | Where new files arrive       |
+| `s3ProcessedPrefix` | Processed files archive   | `products/processed/` | Where files move after success   |
+| `s3ErrorPrefix`   | Failed files directory    | `products/errors/`  | Where files move after errors    |
+| `s3LogsPrefix`   | Event logs directory     | `products/logs/`   | Where event logs are written    |
+| `filePattern`    | File name filter       | `products_*.csv`   | Glob pattern for matching files   |
+| `catalogueRef`   | Product catalogue reference | `PC:MASTER:2`     | Target catalogue in Fluent     |
+| `catalogueType`   | Catalogue type        | `MASTER`       | Usually MASTER or STANDARD     |
+| `eventName`     | Event to send        | `UPSERT_PRODUCT`   | Event name from Rubix        |
+| `eventMode`     | Event processing mode    | `async`        | async (recommended) or sync     |
+| `maxFilesPerRun`  | Max files per execution   | `10`         | Prevent timeout on large batches  |
+| `enableFileTracking` | Enable KV-based file tracking | `false`        | `false` = moveObject only (default), `true` = KV + moveObject |
+| `webhookApiKey`   | API key for webhook security | -           | Required for webhook authentication |
+| `useBatchedEvents` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
+| `maxProductsUnderBatchedEvent` | Products per batch | `100`          | Only used when `useBatchedEvents=true` (default: 100) |
+| `memoryBatchSize`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
+
+---
+
+## 🔒 SDK Automatic Behaviors (v0.1.40+)
+
+**The SDK automatically validates and retries for improved reliability:**
+
+###retailerId Validation
+- **SDK validates** `retailerId` before calling `sendEvent()`
+- **Checks:** `event.retailerId || client.retailerId`
+- **If missing:** Throws `"retailerId is required for Event API..."`
+- **Configuration:** Set via `fluentRetailerId` activation variable (recommended)
+
+### 401 Auth Retry
+- **Automatic retry** for platform auth failures (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s)
+- **Applies to:** All `sendEvent()` calls (async and sync modes)
+- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
+
+### 5xx Server Retry
+- **Automatic retry** for transient server errors (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
+- **Protects:** Against Fluent API transient failures
+
+### No Code Changes Required
+- All templates remain compatible
+- Retry logic is automatic and transparent
+- Better error messages guide configuration
+
+**See:** [Event API Guide](./event-api-guide.md) for complete details
+
+---
+
+### 📁 S3 Deduplication Pattern (NO VersoriFileTracker)
+
+**CRITICAL: S3 uses physical file movement, NOT VersoriFileTracker**
+
+VersoriFileTracker is **SFTP-specific** and should **NOT** be used with S3. S3 achieves deduplication through physical file movement.
+
+#### How S3 Deduplication Works
+
+**3-Step Process:**
+
+1. **Discovery:** List objects from `s3IncomingPrefix` matching `filePattern`
+2. **Process:** Download, parse, transform, and send events
+3. **Archive:** Move object to `s3ProcessedPrefix` or `s3ErrorPrefix`
+4. **Natural Deduplication:** Once moved, files are no longer in incoming/
+
+#### File Lifecycle Example
+
+```
+INITIAL STATE:
+s3://bucket/products/incoming/products_20250124_001.csv
+
+↓ (workflow discovers file)
+
+PROCESSING:
+- List objects from "products/incoming/"
+- Download: products_20250124_001.csv
+- Parse CSV → Map to events → Send to Event API
+
+↓ (success)
+
+ARCHIVE (moveObject):
+s3://bucket/products/processed/products_20250124_001-20250124T183045.csv
+
+↓ (result)
+
+INCOMING FOLDER NOW EMPTY:
+s3://bucket/products/incoming/ (file is gone - no duplicates possible)
+```
+
+#### Key Differences: S3 vs SFTP
+
+| Aspect          | S3 (This Template)      | SFTP (Other Templates)      |
+| ------------------------ | ----------------------------- | -------------------------------- |
+| **Deduplication Method** | Physical file movement    | VersoriFileTracker (KV state)  |
+| **State Storage**    | None needed          | KV store tracks processed files |
+| **File Tracking**    | ❌ NO VersoriFileTracker   | ✅ VersoriFileTracker required  |
+| **Reprocessing**     | Move files back to incoming/ | Clear KV state or forceReprocess |
+| **Complexity**      | Simpler (no state management) | More complex (state management) |
+
+#### Why S3 Doesn't Need VersoriFileTracker
+
+**S3 moveObject provides natural deduplication:**
+
+- Files are **physically moved** from `incoming/` to `processed/` or `errors/`
+- Once moved, the file **no longer exists** in the incoming prefix
+- Next discovery lists `incoming/` → file is not found → no duplicate processing
+- **No KV state tracking needed** - the S3 bucket structure IS the state
+
+#### Reprocessing Files (If Needed)
+
+To reprocess a file that was already archived:
+
+**Option 1: Manual S3 Move**
+
+```bash
+# Move file back from processed/ to incoming/
+aws s3 mv \
+ s3://bucket/products/processed/products_20250124_001-timestamp.csv \
+ s3://bucket/products/incoming/products_20250124_001.csv
+```
+
+**Option 2: Webhook with forceReprocess**
+
+```bash
+# Set forceReprocess: true in webhook payload
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "forceReprocess": true,
+  "filePattern": "products_20250124_001.csv"
+ }'
+```
+
+**Use cases for reprocessing:**
+
+- Fixing bad data that was previously ingested
+- Reingesting after mapping config changes
+- Testing with production files
+
+#### Summary
+
+✅ **DO:** Use S3 moveObject for deduplication
+✅ **DO:** Move files to processed/ or errors/ prefixes
+❌ **DON'T:** Import VersoriFileTracker in S3 templates
+❌ **DON'T:** Use KV state tracking for S3 files
+✅ **DO:** Use VersoriFileTracker for SFTP templates (different use case)
+
+---
+
+### 📁 S3 Prefix vs File Pattern
+
+**Critical:** Folder prefixes and file patterns are configured separately.
+
+- **Folder Prefixes:** Configured via activation variables:
+ - `s3IncomingPrefix` - Where new files arrive (e.g., `products/incoming/`)
+ - `s3ProcessedPrefix` - Where successful files are archived (e.g., `products/processed/`)
+ - `s3ErrorPrefix` - Where failed files are moved (e.g., `products/errors/`)
+
+- **File Pattern:** Configured via `filePattern` activation variable
+ - Matches object keys (filenames)
+ - Supports wildcards: `*.csv`, `products_*.csv`, `urgent_*.csv`
+
+**How it works:**
+
+```typescript
+// The workflow lists objects like this:
+s3.listObjects({
+ prefix: 'products/incoming/', // ← Prefix from activation variable
+ pattern: 'products_*.csv', // ← Pattern matches object keys
+});
+```
+
+**Example:** With these settings:
+
+```json
+{
+ "s3IncomingPrefix": "products/incoming/",
+ "filePattern": "products_*.csv"
+}
+```
+
+The workflow will find objects like:
+
+- ✅ `products/incoming/products_20250124.csv`
+- ✅ `products/incoming/products_urgent.csv`
+- ❌ `products/incoming/orders_20250124.csv` (doesn't match pattern)
+- ❌ `orders/incoming/products_20250124.csv` (wrong prefix)
+
+**Note:** To process files from different prefixes, change `s3IncomingPrefix` (not `filePattern`).
+
+---
+
+## 📄 Mapping Configuration
+
+**File:** `config/products.import.csv.json`
+
+```json
+{
+ "name": "products.import.csv",
+ "version": "1.0.0",
+ "description": "CSV → Product Event Mapping",
+ "fields": {
+  "ref": {
+   "source": "ref",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "type": {
+   "source": "type",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "STANDARD"
+  },
+  "status": {
+   "source": "status",
+   "resolver": "sdk.uppercase",
+   "defaultValue": "ACTIVE"
+  },
+  "gtin": {
+   "source": "gtin",
+   "resolver": "sdk.trim"
+  },
+  "name": {
+   "source": "name",
+   "required": true,
+   "resolver": "sdk.trim"
+  },
+  "summary": {
+   "source": "summary"
+  },
+  "categoryRefs": {
+   "source": "category_refs",
+   "resolver": "custom.splitByPipe"
+  },
+  "price": {
+   "resolver": "custom.buildPriceArray"
+  },
+  "taxType": {
+   "resolver": "custom.buildTaxType"
+  },
+  "attributes": {
+   "defaultValue": []
+  }
+ }
+}
+```
+
+**Custom Resolvers:**
+
+```typescript
+const customResolvers = {
+ /**
+  * Split pipe-separated string into array
+  * Example: "CAT1|CAT2|CAT3" → ["CAT1", "CAT2", "CAT3"]
+  */
+ 'custom.splitByPipe': (value: string) => {
+  if (!value || value.trim() === '') return [];
+  return value
+   .split('|')
+   .map(v => v.trim())
+   .filter(v => v !== '');
+ },
+
+ /**
+  * Build price array from flat CSV columns
+  */
+ 'custom.buildPriceArray': (value: any, sourceData: any) => {
+  const prices = [];
+  if (sourceData.price_default_value) {
+   prices.push({
+    type: 'DEFAULT',
+    currency: sourceData.price_default_currency || 'USD',
+    value: sourceData.price_default_value,
+   });
+  }
+  if (sourceData.price_special_value) {
+   prices.push({
+    type: 'SPECIAL',
+    currency: sourceData.price_special_currency || 'USD',
+    value: sourceData.price_special_value,
+   });
+  }
+  return prices;
+ },
+
+ /**
+  * Build tax type object from flat CSV columns
+  */
+ 'custom.buildTaxType': (value: any, sourceData: any) => {
+  if (!sourceData.tax_country) return undefined;
+  return {
+   country: sourceData.tax_country,
+   group: sourceData.tax_group,
+   tariff: sourceData.tax_tariff,
+  };
+ },
+};
+```
+
+Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
+
+---
+
+## Expected CSV Format
+
+```csv
+ref,type,status,gtin,name,summary,category_refs,price_default_currency,price_default_value,price_special_currency,price_special_value,tax_country,tax_group,tax_tariff
+G_PROD_WITH_NO_STANDARD,VARIANT,ACTIVE,MH01-XS-Orange,Chaz Kangeroo Hoodie-XS-Orange main,<p>test short description</p>,STANDARD_CATEGORY,USD,52.000000,USD,9.000000,AU,Tax Group,Tax Tariff
+G_PROD_002,VARIANT,ACTIVE,MH02-M-Blue,Kangeroo Hoodie-M-Blue,<p>Blue variant</p>,STANDARD_CATEGORY|SEASONAL,USD,45.000000,USD,35.000000,AU,Tax Group,Tax Tariff
+G_PROD_003,STANDARD,ACTIVE,MH03-BASE,Base Hoodie Product,<p>Base product</p>,STANDARD_CATEGORY,USD,50.000000,,,AU,Tax Group,Tax Tariff
+```
+
+---
+
+## Event Sending Configuration
+
+**Simple Configuration:** One variable controls everything
+
+```json
+{
+ "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
+}
+```
+
+**How it works:**
+
+- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
+- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
+- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
+- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
+
+**Configuration Guidelines:**
+
+- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
+- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
+- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
+- **Note:** Fluent API supports concurrent requests - adjust based on your needs
+
+**Why Single Variable?**
+
+- **Simpler:** One variable instead of two (`mode` + `concurrency`)
+- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
+- **Less config:** Fewer activation variables to manage
+- **Flexible:** Easy to tune performance (just change the number)
+
+---
+
+## 🔧 Production Reference Implementation
+
+### Processing Approach
+
+This template uses **per-file sequential processing**:
+- Discovers files from S3 incoming prefix
+- Processes each file completely (download → parse → map → send events → archive) before moving to the next
+- Files are processed sequentially to prevent timeout issues and ensure proper error handling
+- Use `maxFilesPerRun` activation variable to limit files per execution (default: 10)
+
+**Why per-file processing?**
+- ✅ Better error isolation - one bad file doesn't affect others
+- ✅ Progressive archiving - files move to processed/errors immediately
+- ✅ Lower memory footprint - one file in memory at a time
+- ✅ Simpler code - no complex batching logic needed
+
+### Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+product-event-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── product.types.ts            # Shared type definitions
+```
+
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+
+---
+
+## Workflow Files
+
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+
+All time-based triggers that run automatically on cron schedules.
+
+#### `src/workflows/scheduled/daily-product-sync.ts`
+
+**Purpose**: Automatic Daily product sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Scheduled Workflow: Daily Product Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: product-sync.service.ts
+ */
+export const dailyProductSync = schedule(
+  'product-sync-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv } = ctx;
+    const jobId = `product-sync-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
+    await tracker.updateJob(jobId, { status: 'processing' });
+
+    try {
+      // Reuse shared orchestration logic
+      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker);
+      await tracker.markCompleted(jobId, result);
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      await tracker.markFailed(jobId, e);
+      return { success: false, jobId, error: e?.message };
+    }
+  })
+);
+```
+
+---
+
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+
+All HTTP-based triggers that create webhook endpoints.
+
+#### `src/workflows/webhook/adhoc-product-sync.ts`
+
+**Purpose**: Manual product sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Webhook: Manual Product Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
+ * Request body (optional): { filePattern: "urgent_*.csv", maxFiles: 5 }
+ *
+ * Pattern: webhook().then(http()) - needs Fluent API access
+ * Uses shared service: product-sync.service.ts
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const adhocProductSync = webhook('product-sync-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'product-sync-adhoc', // Versori validates API key
+}).then(
+  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, data } = ctx;
+    const jobId = `product-sync-adhoc-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    const filePattern = data?.filePattern as string;
+    const maxFiles = data?.maxFiles as number;
+
+    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+    });
+
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'manual',
+      stage: 'initialization',
+      status: 'queued',
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
+      .then((result) => {
+        log.info('✅ [BACKGROUND] Product sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          filesFailed: result.filesFailed,
+          recordsProcessed: result.recordsProcessed,
+        });
+        return tracker.markCompleted(jobId, result);
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+
+        log.error('❌ [BACKGROUND] Product sync failed', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+        });
+
+        return tracker.markFailed(jobId, errorMessage);
+      });
+
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Product sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+
+#### `src/workflows/webhook/job-status-check.ts`
+
+**Purpose**: Query job status
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
+**Request body**: `{ "jobId": "product-sync-1234567890" }`
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
+ * Request body: { "jobId": "product-sync-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const productSyncJobStatus = webhook('product-sync-job-status', {
+  response: { mode: 'sync' },
+  connection: 'product-sync-job-status',
+}).then(
+  fn('status', async ctx => {
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+
+    if (!jobId) {
+      return { success: false, error: 'jobId required' };
+    }
+
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+
+---
+
+### 3. Entry Point (`index.ts`)
+
+**Purpose**: Register all workflows with Versori platform (MemoryInterpreter pattern)
+
+```typescript
+/**
+ * ═══════════════════════════════════════════════════════════════
+ * 🚀 VERSORI PRODUCT SYNC - ENTRY POINT
+ * ═══════════════════════════════════════════════════════════════
+ *
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * PATTERN: MemoryInterpreter
+ * - Export all workflows from a single file
+ * - Versori automatically discovers and registers exported workflows
+ * - Clean separation: scheduled vs webhook triggers
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ * - src/services/            → Shared business logic
+ *
+ * ═══════════════════════════════════════════════════════════════
+ */
+
+// Import scheduled workflows
+import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
+
+// Import webhook workflows
+import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
+import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
+
+// ═══════════════════════════════════════════════════════════════
+// Register all workflows with Versori
+// ═══════════════════════════════════════════════════════════════
+export {
+  // Scheduled (time-based triggers)
+  dailyProductSync,
+
+  // Webhooks (HTTP-based triggers)
+  adhocProductSync,
+  productSyncJobStatus,
+};
+```
+
+**What Gets Exposed:**
+- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
+- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
+- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
+
+---
+
+### Adding New Workflows
+
+**To add a scheduled workflow:**
+1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**To add a webhook workflow:**
+1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**Example - Adding hourly delta sync:**
+
+```typescript
+// src/workflows/scheduled/hourly-delta-sync.ts
+export const hourlyDeltaSync = schedule(
+  'product-delta-hourly',
+  '0 * * * *' // Every hour
+).then(
+  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
+    // Delta sync logic (skip BPP)
+    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
+    return result;
+  })
+);
+
+// index.ts (add to imports and exports)
+import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
+export { daily_product_sync, hourlyDeltaSync, ... };
+```
+
+---
+## 3. Type Definitions (src/types/product-ingestion.types.ts)
+
+```typescript
+/**
+ * Type Definitions for Product Ingestion
+ *
+ * Centralized type definitions for product ingestion workflow
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Product interface - represents transformed product data
+ */
+export interface Product {
+ ref: string;
+ type?: string;
+ status?: string;
+ name: string;
+ summary?: string;
+ gtin?: string;
+ catalogue?: { ref?: string };
+ attributes?: Record<string, unknown>;
+}
+
+/**
+ * Event configuration
+ */
+export interface EventConfig {
+ eventName: string;
+ catalogueRef: string;
+ catalogueType: string;
+ eventMode: 'async' | 'sync';
+}
+
+/**
+ * Event result - tracks success/failure counts
+ */
+export interface EventResult {
+ eventsSent: number;
+ eventsFailed: number;
+ errors: Array<{ productRef: string; error: string }>;
+}
+
+/**
+ * Process file result
+ */
+export interface ProcessFileResult {
+ success: boolean;
+ products: Product[];
+ error?: string;
+}
+
+/**
+ * Versori Context Interface
+ * Represents the Versori runtime context passed to workflow functions
+ */
+export interface VersoriContext {
+ log: {
+  info: (message: string, data?: Record<string, unknown>) => void;
+  warn: (message: string, data?: Record<string, unknown>) => void;
+  error: (message: string, data?: Record<string, unknown>) => void;
+  debug?: (message: string, data?: Record<string, unknown>) => void;
+ };
+ openKv: (namespace: string) => {
+  get: (key: string) => Promise<unknown>;
+  set: (key: string, value: unknown) => Promise<void>;
+  delete: (key: string) => Promise<void>;
+ };
+ activation: {
+  getVariable: (name: string) => string | undefined;
+  connections?: Record<string, unknown>;
+ };
+ connections?: Record<string, unknown>;
+ data?: unknown;
+ fetch?: typeof fetch;
+}
+
+/**
+ * Parameters for ingestion workflow
+ */
+export interface ProductIngestionParams {
+ jobId: string;
+ triggeredBy: 'schedule' | 'webhook';
+ filePattern?: string;
+ maxFiles?: number;
+ forceReprocess?: boolean;
+ catalogueRef?: string;
+ priority?: 'normal' | 'high';
+}
+
+/**
+ * Result from ingestion workflow
+ */
+export interface ProductIngestionResult {
+ success: boolean;
+ jobId: string;
+ filesProcessed: number;
+ filesFailed: number;
+ filesSkipped?: number;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ fileResults: FileProcessingResult[];
+ error?: string;
+}
+
+/**
+ * Per-file processing result
+ */
+export interface FileProcessingResult {
+ fileName: string;
+ success: boolean;
+ skipped?: boolean;
+ recordsProcessed: number;
+ eventsSent: number;
+ eventsFailed: number;
+ duration: number;
+ error?: string;
+}
+```
+
+## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
+
+```typescript
+/**
+ * Product File Processor Service
+ *
+ * Downloads CSV files from S3, parses, and transforms with UniversalMapper.
+ * CSV-specific: Returns array directly - no normalization needed.
+ */
+
+import {
+ S3DataSource,
+ CSVParserService,
+ UniversalMapper,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { ProcessFileResult, Product } from '../types/product-ingestion.types';
+
+/**
+ * Service for processing CSV product files from S3
+ */
+export class ProductFileProcessorService {
+ constructor(
+  private s3: S3DataSource,
+  private csvParser: CSVParserService,
+  private mapper: UniversalMapper,
+  private catalogueRef: string
+ ) {}
+
+ /**
+  * Download CSV file from S3, parse, and transform with UniversalMapper
+  *
+  * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
+  * - Explicit null assignments after each step
+  * - Finally block guarantees cleanup
+  * - Prevents OOM errors on large CSV files
+  */
+ async downloadParseAndTransform(objectKey: string): Promise<ProcessFileResult> {
+  // ✅ CRITICAL: Variables for cleanup tracking
+  let csvContent: string | null = null;
+  let parsed: unknown | null = null;
+
+  try {
+   // STEP 1: Download CSV content from S3
+   csvContent = (await this.s3.downloadObject(objectKey, {
+    encoding: 'utf8',
+   })) as string;
+
+   // STEP 2: Parse CSV with type safety
+   try {
+    parsed = await this.csvParser.parse(csvContent);
+   } catch (parseError: unknown) {
+    const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
+    return {
+     success: false,
+     products: [],
+     error: `CSV parse error: ${errorMessage}`,
+    };
+   }
+
+   // ✅ Clear CSV content from memory - no longer needed after parsing
+   csvContent = null;
+
+   // STEP 3: CSV-SPECIFIC - Returns array directly, no normalization needed
+   const rawProducts = Array.isArray(parsed) ? parsed : [];
+
+   if (rawProducts.length === 0) {
+    return {
+     success: false,
+     products: [],
+     error: 'No products found in CSV',
+    };
+   }
+
+   // ✅ Clear parsed data from memory - only need rawProducts now
+   parsed = null;
+
+   // STEP 4: Transform with UniversalMapper
+   // ✅ S3 CSV: Merge context using spread pattern (same as SFTP CSV)
+   const sourceDataWithContext = rawProducts.map(item => ({
+    ...item,
+    $context: { catalogueRef: this.catalogueRef },
+   }));
+
+   const mappingResult = await this.mapper.map(sourceDataWithContext);
+
+   if (!mappingResult.success) {
+    return {
+     success: false,
+     products: [],
+     error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
+    };
+   }
+
+   if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+    this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+     skippedFields: mappingResult.skippedFields,
+     note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+    });
+   }
+
+   return {
+    success: true,
+    products: mappingResult.data as Product[],
+   };
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   return {
+    success: false,
+    products: [],
+    error: errorMessage,
+   };
+  } finally {
+   // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
+   csvContent = null;
+   parsed = null;
+  }
+ }
+}
+```
+
+---
+
+## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
+
+```typescript
+/**
+ * Event Sender Service
+ *
+ * Sends product events to Fluent Commerce Event API with per-record error handling.
+ * Continues processing on individual failures (Event API best practice).
+ * Supports configurable concurrency (sequential or parallel).
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
+
+/**
+ * Service for sending events to Fluent Commerce Event API
+ *
+ * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
+ */
+export class EventSenderService {
+ constructor(
+  private client: FluentClient,
+  private log?: any  // ✅ Optional logger for progress tracking
+ ) {}
+
+ /**
+  * Send events to Fluent Commerce Event API with configurable concurrency
+  *
+  * **Performance Characteristics:**
+  * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
+  * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
+  * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
+  *
+  * **Implementation Strategy:**
+  * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
+  * - Concurrency > 1: Chunked parallel processing with bounded concurrency
+  * - Both modes: Per-record error tracking (failures don't block other events)
+  *
+  * @param products - Array of products to send as events
+  * @param eventConfig - Event configuration (name, catalogue ref, mode)
+  * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
+  * @returns EventResult with counts (eventsSent/eventsFailed) and error details
+  */
+ async sendEvents(
+  products: Product[],
+  eventConfig: EventConfig,
+  concurrency: number = 1
+ ): Promise<EventResult> {
+  // Validate concurrency (guard against invalid values)
+  const safeConc = Math.max(1, Math.floor(concurrency));
+
+  // Result accumulators
+  let eventsSent = 0;
+  let eventsFailed = 0;
+  const errors: Array<{ productRef: string; error: string }> = [];
+
+  // ✅ PRODUCTION ENHANCEMENT: Log event sending start
+  if (this.log) {
+   this.log.info('📤 Starting event sending', {
+    totalProducts: products.length,
+    concurrency: safeConc,
+    processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
+    eventName: eventConfig.eventName,
+    catalogueRef: eventConfig.catalogueRef
+   });
+  }
+
+  // Helper: Build event payload (DRY - reused in both modes)
+  const buildPayload = (product: Product) => ({
+   name: eventConfig.eventName,
+   entityRef: eventConfig.catalogueRef,
+   entityType: 'PRODUCT_CATALOGUE' as const,
+   entitySubtype: eventConfig.catalogueType,
+   rootEntityRef: eventConfig.catalogueRef,
+   rootEntityType: 'PRODUCT_CATALOGUE' as const,
+   attributes: product,
+  });
+
+  // ============================================================================
+  // SEQUENTIAL MODE (concurrency === 1)
+  // ============================================================================
+  if (safeConc === 1) {
+   for (let i = 0; i < products.length; i++) {
+    const product = products[i];
+
+    // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
+    if (this.log && i % 10 === 0) {
+     this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
+      productRef: product.ref,
+      progress: `${((i / products.length) * 100).toFixed(1)}%`
+     });
+    }
+
+    try {
+     await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
+     eventsSent++;
+    } catch (err: unknown) {
+     eventsFailed++;
+     const errorMsg = err instanceof Error ? err.message : String(err);
+     errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
+     // Continue processing (failure doesn't block other products)
+    }
+   }
+
+   // ✅ PRODUCTION ENHANCEMENT: Log completion
+   if (this.log) {
+    this.log.info('✅ Sequential event sending completed', {
+     totalProducts: products.length,
+     eventsSent,
+     eventsFailed
+    });
+   }
+
+   return { eventsSent, eventsFailed, errors };
+  }
+
+  // ============================================================================
+  // PARALLEL MODE (concurrency > 1)
+  // ============================================================================
+  const totalChunks = Math.ceil(products.length / safeConc);
+
+  for (let i = 0; i < products.length; i += safeConc) {
+   const chunk = products.slice(i, i + safeConc);
+   const chunkNumber = Math.floor(i / safeConc) + 1;
+
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
+   if (this.log) {
+    this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
+     chunkNumber,
+     totalChunks,
+     productsInChunk: chunk.length,
+     productRange: `${i + 1}-${i + chunk.length}`,
+     totalProducts: products.length,
+     progress: `${((i / products.length) * 100).toFixed(1)}%`
+    });
+   }
+
+   // Fire all requests in chunk concurrently
+   const results = await Promise.allSettled(
+    chunk.map(product =>
+     this.client
+      .sendEvent(buildPayload(product), eventConfig.eventMode)
+      .then(() => ({ success: true as const, product }))
+      .catch(error => ({ success: false as const, product, error }))
+    )
+   );
+
+   // Aggregate chunk results into totals
+   let chunkSuccess = 0;
+   let chunkFailed = 0;
+
+   for (const result of results) {
+    if (result.status === 'fulfilled' && result.value.success) {
+     eventsSent++;
+     chunkSuccess++;
+    } else {
+     eventsFailed++;
+     chunkFailed++;
+     const error = result.status === 'fulfilled' ? result.value.error : result.reason;
+     const product = result.status === 'fulfilled' ? result.value.product : null;
+     errors.push({
+      productRef: product?.ref || 'unknown',
+      error: error?.message || String(error) || 'unknown error',
+     });
+    }
+   }
+
+   // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
+   if (this.log) {
+    this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
+     chunkNumber,
+     totalChunks,
+     chunkSuccess,
+     chunkFailed,
+     totalSentSoFar: eventsSent,
+     totalFailedSoFar: eventsFailed,
+     progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
+    });
+   }
+  }
+
+  // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
+  if (this.log) {
+   this.log.info('✅ Parallel event sending completed', {
+    totalProducts: products.length,
+    totalChunks,
+    concurrency: safeConc,
+    eventsSent,
+    eventsFailed
+   });
+  }
+
+  return { eventsSent, eventsFailed, errors };
+ }
+}
+```
+
+---
+
+## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
+
+```typescript
+/**
+ * Event Logger Service
+ *
+ * Writes event processing logs to S3 for audit/debugging.
+ * Optional - can be used to create rejection reports.
+ */
+
+import { Buffer } from 'node:buffer';
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Service for writing event logs to S3
+ */
+export class EventLoggerService {
+ constructor(private s3: S3DataSource) {}
+
+ /**
+  * Write event log to S3
+  *
+  * @param objectKey - S3 object key for log file
+  * @param logData - Log data to write (JSON format)
+  */
+ async writeEventLog(objectKey: string, logData: unknown): Promise<void> {
+  try {
+   const logContent = JSON.stringify(logData, null, 2);
+   await this.s3.uploadFile(objectKey, logContent);
+  } catch (error: unknown) {
+   const errorMessage = error instanceof Error ? error.message : String(error);
+   throw new Error(`Failed to write event log: ${errorMessage}`);
+  }
+ }
+}
+```
+
+## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
+
+```typescript
+/**
+ * ═══════════════════════════════════════════════════════════════
+ * 📦 MAIN INGESTION ORCHESTRATION SERVICE
+ * ═══════════════════════════════════════════════════════════════
+ *
+ * This is the heart of the ingestion workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Discover files on S3
+ * 3. Download and parse CSV files
+ * 4. Transform data with UniversalMapper
+ * 5. Send events to Fluent Commerce
+ * 6. Archive processed files
+ * 7. Track file processing state (S3 uses moveObject for deduplication)
+ * 8. Track job progress with JobTracker
+ *
+ * KEY DIFFERENCES FOR S3 CSV:
+ * - S3DataSource instead of SftpDataSource
+ * - CSVParserService instead of XMLParserService
+ * - No VersoriFileTracker (S3 uses moveObject for deduplication)
+ * - S3 listObjects instead of SFTP listFiles
+ * - S3 moveObject instead of SFTP moveFile
+ * - CSV returns array directly (no normalization needed)
+ * - CSV spread pattern for context injection
+ *
+ * ═══════════════════════════════════════════════════════════════
+ */
+
+import { Buffer } from 'node:buffer';
+import {
+ createClient,
+ S3DataSource,
+ CSVParserService,
+ UniversalMapper,
+ JobTracker,
+ type FluentClient,
+ type JobStatus,
+} from '@fluentcommerce/fc-connect-sdk';
+import type {
+ ProductIngestionParams,
+ ProductIngestionResult,
+ FileProcessingResult,
+ EventConfig,
+} from '../types/product-ingestion.types';
+import { ProductFileProcessorService } from './product-file-processor.service';
+import { EventSenderService } from './event-sender.service';
+import { EventLoggerService } from './event-logger.service';
+import { timestampedName } from '../utils/s3-path.utils';
+
+import mappingConfig from '../../config/products.import.csv.json' with { type: 'json' };
+
+// Custom resolvers for CSV mapping (prices, categories, taxType)
+const customResolvers = {
+ 'custom.splitByPipe': (value: string) => {
+  if (!value || value.trim() === '') return [];
+  return value
+   .split('|')
+   .map(v => v.trim())
+   .filter(v => v !== '');
+ },
+ 'custom.buildPriceArray': (value: any, sourceData: any) => {
+  const prices = [];
+  if (sourceData.price_default_value) {
+   prices.push({
+    type: 'DEFAULT',
+    currency: sourceData.price_default_currency || 'USD',
+    value: sourceData.price_default_value,
+   });
+  }
+  if (sourceData.price_special_value) {
+   prices.push({
+    type: 'SPECIAL',
+    currency: sourceData.price_special_currency || 'USD',
+    value: sourceData.price_special_value,
+   });
+  }
+  return prices;
+ },
+ 'custom.buildTaxType': (value: any, sourceData: any) => {
+  if (!sourceData.tax_country) return undefined;
+  return {
+   country: sourceData.tax_country,
+   group: sourceData.tax_group,
+   tariff: sourceData.tax_tariff,
+  };
+ },
+};
+
+/**
+ * Query job status from KV store
+ */
+export async function getJobStatus(
+ kv: ReturnType<VersoriContext['openKv']>,
+ jobId: string,
+ log: VersoriContext['log']
+): Promise<JobStatus | undefined> {
+ try {
+  const tracker = new JobTracker(kv, log);
+  return await tracker.getJob(jobId);
+ } catch (error: unknown) {
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  log.error('Failed to get job status', {
+   jobId,
+   message: errorMessage,
+   stack: error instanceof Error ? error.stack : undefined,
+   errorType: error instanceof Error ? error.constructor.name : 'Error',
+  });
+  return undefined;
+ }
+}
+
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * This function implements the complete ingestion workflow in 8 steps.
+ */
+export async function executeProductIngestion(
+ ctx: VersoriContext,
+ params: ProductIngestionParams
+): Promise<ProductIngestionResult> {
+
+ const { log, openKv, activation } = ctx;
+ const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
+
+ const kv = openKv(':project:');
+ const tracker = new JobTracker(kv, log);
+
+ const startTime = Date.now();
+ const fileResults: FileProcessingResult[] = [];
+
+ // ⚠️ CRITICAL: Declare S3 outside try block for disposal pattern
+ let s3: S3DataSource | undefined;
+
+ try {
+  // ═══════════════════════════════════════════════════════════
+  // 🚀 STEP 1/8: Initialize Job Tracking
+  // ═══════════════════════════════════════════════════════════
+  log.info('🚀 [STEP 1/8] Initializing job tracking', { jobId });
+
+  await tracker.createJob(jobId, {
+   triggeredBy,
+   filePattern: filePattern || 'default',
+   maxFiles: maxFiles || 'unlimited',
+   forceReprocess: !!forceReprocess,
+   startTime,
+  });
+
+  // ═══════════════════════════════════════════════════════════
+  // 🔧 STEP 2/8: Initialize Fluent Client & S3 Connection
+  // ═══════════════════════════════════════════════════════════
+  log.info('🔧 [STEP 2/8] Initializing Fluent Commerce client and S3', { jobId });
+
+  const client = await createClient(ctx);
+
+  if (!client) {
+   const error = 'Failed to create Fluent Commerce client';
+   log.error(`❌ ${error}`);
+   throw new Error(error);
+  }
+
+  log.info('✅ [Client] Fluent Commerce client created successfully');
+
+  // ✅ CRITICAL: Set retailerId for Event API calls
+  const fluentRetailerId = activation.getVariable('fluentRetailerId');
+  if (!fluentRetailerId) {
+   const error = 'fluentRetailerId is required for Event API calls';
+   log.error(`❌ ${error}`, {
+    recommendation: 'Set fluentRetailerId in activation variables',
+    variable: 'fluentRetailerId',
+   });
+   throw new Error(error);
+  }
+  client.setRetailerId(fluentRetailerId);
+  log.info('✅ [Client] RetailerId set for Event API', { retailerId: fluentRetailerId });
+
+  // Get S3 configuration from activation variables
+  const s3Config = {
+   bucket: activation.getVariable('s3Bucket'),
+   region: activation.getVariable('s3Region') || 'us-east-1',
+   accessKeyId: activation.getVariable('s3AccessKeyId'),
+   secretAccessKey: activation.getVariable('s3SecretAccessKey'),
+   incomingPrefix: activation.getVariable('s3IncomingPrefix') || 'products/incoming/',
+   processedPrefix: activation.getVariable('s3ProcessedPrefix') || 'products/processed/',
+   errorPrefix: activation.getVariable('s3ErrorPrefix') || 'products/errors/',
+   logsPrefix: activation.getVariable('s3LogsPrefix') || 'products/logs/',
+   filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.csv'
+  };
+
+  // Validate S3 config
+  if (!s3Config.bucket) {
+   const error = 'S3 configuration incomplete: missing bucket';
+   log.error(`❌ ${error}`, {
+    recommendation: 'Set s3Bucket in activation variables',
+    variable: 's3Bucket',
+   });
+   throw new Error(error);
+  }
+
+  if (!s3Config.accessKeyId || !s3Config.secretAccessKey) {
+   const error = 'S3 configuration incomplete: missing accessKeyId or secretAccessKey';
+   log.error(`❌ ${error}`, {
+    recommendation: 'Set s3AccessKeyId and s3SecretAccessKey in activation variables',
+    variables: ['s3AccessKeyId', 's3SecretAccessKey'],
+   });
+   throw new Error(error);
+  }
+
+  // Initialize S3 data source
+  s3 = new S3DataSource(
+   {
+    type: 'S3_CSV',
+    connectionId: 's3-product-ingestion',
+    name: 'Product Ingestion S3',
+    s3Config: {
+     bucket: s3Config.bucket,
+     region: s3Config.region,
+     accessKeyId: s3Config.accessKeyId,
+     secretAccessKey: s3Config.secretAccessKey,
+    },
+    validateConnection: true, // ✅ Enable connection validation
+   },
+   log
+  );
+
+  try {
+   // Validate S3 connection
+   log.info('🔍 [S3] Validating S3 connection...');
+   await s3.validateConnection();
+   log.info('✅ [S3] S3 connection validated successfully');
+
+   // ═══════════════════════════════════════════════════════════
+   // 🔍 STEP 3/8: Discover Files on S3
+   // ═══════════════════════════════════════════════════════════
+   log.info('🔍 [STEP 3/8] Discovering files on S3', {
+    jobId,
+    prefix: s3Config.incomingPrefix,
+    filePattern: s3Config.filePattern
+   });
+
+   await tracker.updateJob(jobId, {
+    status: 'processing',
+    stage: 'file_discovery',
+    message: 'Discovering files on S3'
+   });
+
+   // List objects from S3
+   const allFiles = await s3.listObjects({
+    prefix: s3Config.incomingPrefix,
+    pattern: s3Config.filePattern
+   });
+
+   log.info(`✅ [Discovery] Found ${allFiles.length} files matching pattern`);
+
+   // Apply maxFiles limit if specified
+   const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
+
+   if (filesToProcess.length === 0) {
+    log.info('⚠️ [Discovery] No files to process');
+
+    await tracker.markCompleted(jobId, {
+     filesProcessed: 0,
+     message: 'No files found to process',
+     duration: Date.now() - startTime,
+    });
+
+    return {
+     success: true,
+     jobId,
+     filesProcessed: 0,
+     filesFailed: 0,
+     recordsProcessed: 0,
+     eventsSent: 0,
+     eventsFailed: 0,
+     fileResults: [],
+     duration: Date.now() - startTime,
+    };
+   }
+
+   log.info(`📄 [Discovery] Processing ${filesToProcess.length} files`);
+
+   // Initialize services
+   const csvParser = new CSVParserService();
+   const mapper = new UniversalMapper(mappingConfig, { customResolvers });
+   
+   // Get event configuration
+   const eventConfig: EventConfig = {
+    catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
+    catalogueType: activation.getVariable('catalogueType') || 'MASTER',
+    eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
+    eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
+   };
+
+   // Get event sending configuration
+   const eventConcurrency = Math.max(
+    1,
+    parseInt(activation.getVariable('eventConcurrency') || '1', 10)
+   );
+
+   log.info(`Event concurrency: ${eventConcurrency}`, {
+    mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
+    concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
+   });
+
+   // Initialize class-based services
+   const fileProcessor = new ProductFileProcessorService(
+    s3,
+    csvParser,
+    mapper,
+    eventConfig.catalogueRef
+   );
+   // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
+   const eventSender = new EventSenderService(client, log);
+   const eventLogger = new EventLoggerService(s3);
+
+   // ═══════════════════════════════════════════════════════════
+   // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
+   // ═══════════════════════════════════════════════════════════
+
+   for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
+    const file = filesToProcess[fileIndex];
+    const fileStartTime = Date.now();
+
+    // Use object key from S3 listing
+    const objectKey = file.key || (file as any).name;
+    const baseName = objectKey.split('/').pop() || objectKey;
+
+    // ═══════════════════════════════════════════════════════════
+    log.info(`\n${'═'.repeat(60)}`);
+    log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing: ${baseName}`);
+    log.info(`${'═'.repeat(60)}`);
+    // ═══════════════════════════════════════════════════════════
+
+    try {
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'downloading',
+      message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
+     });
+
+     // ═══════════════════════════════════════════════════════════
+     // 📥 STEP 4: Process File (Download + Parse + Map)
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📥 [STEP 4/8] Processing file: ${baseName}`);
+
+     // Process file: download → parse → transform
+     const fileResult = await fileProcessor.downloadParseAndTransform(objectKey);
+
+     if (!fileResult.success || fileResult.products.length === 0) {
+      // Move failed file to errors and record result
+      const archivedName = timestampedName(baseName);
+      const errorKey = `${s3Config.errorPrefix}${archivedName}`;
+      await s3.moveObject(objectKey, errorKey);
+
+      fileResults.push({
+       fileName: baseName,
+       success: false,
+       recordsProcessed: fileResult.products.length,
+       eventsSent: 0,
+       eventsFailed: 0,
+       duration: Date.now() - fileStartTime,
+       error: fileResult.error || 'Processing failed'
+      });
+
+      continue;
+     }
+
+     // ═══════════════════════════════════════════════════════════
+     // 🚀 STEP 5: Send Events
+     // ═══════════════════════════════════════════════════════════
+     log.info(`🚀 [STEP 5/8] Sending events for ${baseName}`);
+
+     // Extract context for progress logging
+     const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
+     const eventType = eventConfig.eventName || 'UPSERT_PRODUCT';
+
+     log.info(`[EventSender] Sending events for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventType,
+      concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
+      sampleProductRefs: sampleProductRefs.join(', '),
+      eventMode: eventConfig.eventMode || 'async'
+     });
+
+     const eventResult = await eventSender.sendEvents(
+       fileResult.products,
+       eventConfig,
+       eventConcurrency
+     );
+
+     const eventsSent = eventResult.eventsSent;
+     const eventsFailed = eventResult.eventsFailed;
+
+     log.info(`✅ [Events] Sent ${eventsSent} events for ${baseName}`, {
+       successful: eventsSent,
+       failed: eventsFailed
+     });
+
+     // Completion logging with summary
+     log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
+      totalProducts: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
+      eventType,
+      duration: Date.now() - fileStartTime,
+     });
+
+     // ═══════════════════════════════════════════════════════════
+     // 📦 STEP 6: Archive File
+     // ═══════════════════════════════════════════════════════════
+     log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
+
+     await tracker.updateJob(jobId, {
+      status: 'processing',
+      stage: 'archiving',
+      message: `Archiving ${baseName}`
+     });
+
+     // Conditionally archive: errors → errorPrefix, else → processedPrefix (timestamped)
+     // ✅ S3: Uses moveObject for deduplication (no VersoriFileTracker needed)
+     const archivedName = timestampedName(baseName);
+     const targetPrefix = eventsFailed > 0 ? s3Config.errorPrefix : s3Config.processedPrefix;
+     const targetKey = `${targetPrefix}${archivedName}`;
+     await s3.moveObject(objectKey, targetKey);
+
+     log.info(`✅ [Archive] File archived successfully: ${baseName}`, { targetKey });
+
+     // Optional: Write event log for audit/debugging
+     if (eventsFailed > 0) {
+      const logKey = `${s3Config.logsPrefix}${baseName.replace(/\.csv$/i, '')}-event-log.json`;
+      await eventLogger.writeEventLog(logKey, {
+       fileName: baseName,
+       totalRecords: fileResult.products.length,
+       eventsSent,
+       eventsFailed,
+       errors: eventResult.errors,
+       processedAt: new Date().toISOString()
+      });
+     }
+
+     // Store file result
+     fileResults.push({
+      fileName: baseName,
+      success: true,
+      recordsProcessed: fileResult.products.length,
+      eventsSent,
+      eventsFailed,
+      duration: Date.now() - fileStartTime
+     });
+
+    } catch (error: unknown) {
+     // ❌ Enhanced error logging with recommendations
+     const errorMessage = error instanceof Error ? error.message : String(error);
+     const errorDetails = {
+      message: errorMessage,
+      stack: error instanceof Error ? error.stack : undefined,
+      fileName: (error as any)?.fileName,
+      lineNumber: (error as any)?.lineNumber,
+      originalError: (error as any)?.context?.originalError?.message,
+      errorType: error instanceof Error ? error.name : 'Error',
+      recommendation: 'Check file format, S3 permissions, and mapping configuration',
+     };
+     log.error(`❌ [Error] Failed to process file ${baseName}:`, errorDetails);
+
+     // Best-effort: move to error archive on unexpected failure
+     try {
+      const archivedName = timestampedName(baseName);
+      const errorKey = `${s3Config.errorPrefix}${archivedName}`;
+      await s3.moveObject(objectKey, errorKey);
+     } catch (moveError: unknown) {
+      const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
+      log.error('Could not archive failed file', {
+       file: baseName,
+       message: moveErrorMessage,
+       stack: moveError instanceof Error ? moveError.stack : undefined,
+       errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
+      });
+     }
+
+     fileResults.push({
+      fileName: baseName,
+      success: false,
+      recordsProcessed: 0,
+      eventsSent: 0,
+      eventsFailed: 0,
+      duration: Date.now() - fileStartTime,
+      error: errorMessage
+     });
+    }
+   }
+
+   // ═══════════════════════════════════════════════════════════
+   // ✅ STEP 8: Complete Job & Calculate Totals
+   // ═══════════════════════════════════════════════════════════
+   log.info('✅ [STEP 8/8] Completing job and calculating totals', { jobId });
+
+   const filesProcessed = fileResults.filter(r => r.success).length;
+   const filesFailed = fileResults.filter(r => !r.success).length;
+   const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
+   const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
+   const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
+   const totalDuration = Date.now() - startTime;
+
+   await tracker.markCompleted(jobId, {
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: totalDuration
+   });
+
+   // ═══════════════════════════════════════════════════════════
+   log.info(`\n${'═'.repeat(60)}`);
+   log.info('✅ [COMPLETE] Ingestion completed successfully');
+   log.info(`${'═'.repeat(60)}`);
+   // ═══════════════════════════════════════════════════════════
+
+   log.info('[Summary]', {
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    duration: `${totalDuration}ms`,
+   });
+
+   return {
+    success: true,
+    jobId,
+    filesProcessed,
+    filesFailed,
+    recordsProcessed: totalRecordsProcessed,
+    eventsSent: totalEventsSent,
+    eventsFailed: totalEventsFailed,
+    fileResults
+   };
+
+  } finally {
+   // ⚠️ CRITICAL: Always dispose S3 connection
+   if (s3) {
+    await s3.dispose();
+    log.info('✅ [Cleanup] S3 connection disposed');
+   }
+  }
+
+ } catch (error: unknown) {
+  const errorMessage = error instanceof Error ? error.message : String(error);
+  const errorStack = error instanceof Error ? error.stack : undefined;
+  log.error('❌ [FATAL] Ingestion workflow failed', {
+   jobId,
+   error: errorMessage,
+   stack: errorStack,
+   recommendation: 'Check logs for detailed error information and verify all activation variables',
+  });
+
+  // Try to mark job as failed, but don't let tracking errors mask workflow error
+  try {
+   await tracker.markFailed(jobId, error);
+  } catch (trackingError: unknown) {
+   const trackingErrorMessage =
+    trackingError instanceof Error ? trackingError.message : String(trackingError);
+   log.warn('Failed to mark job as failed in tracker', {
+    jobId,
+    trackingError: trackingErrorMessage,
+   });
+  }
+
+  return {
+   success: false,
+   jobId,
+   filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
+   filesFailed: fileResults.filter(r => !r.success).length,
+   filesSkipped: fileResults.filter(r => r.skipped).length,
+   recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
+   eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
+   eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
+   fileResults,
+   error: errorMessage,
+  };
+ } finally {
+  // ⚠️ CRITICAL: Ensure S3 is disposed even if outer error occurs
+  if (s3) {
+   try {
+    await s3.dispose();
+    log.info('✅ [Cleanup] S3 connection disposed (outer finally)');
+   } catch (disposeError: unknown) {
+    const disposeErrorMessage =
+     disposeError instanceof Error ? disposeError.message : String(disposeError);
+    log.warn('⚠️ [Warning] Error disposing S3 in outer finally', { error: disposeErrorMessage });
+   }
+  }
+ }
+}
+```
+
+---
+
+## 8. Utility Functions (src/utils/)
+
+### S3 Path Helpers (src/utils/s3-path.utils.ts)
+
+```typescript
+/**
+ * S3 Path Utilities
+ *
+ * Helper functions for S3 path operations.
+ */
+
+/**
+ * Generate timestamped filename for archival
+ *
+ * Adds ISO timestamp to filename before extension for unique archival.
+ * Handles CSV files specifically but can be adapted for other formats.
+ *
+ * @param name - Original filename
+ * @returns Filename with timestamp appended
+ *
+ * @example
+ * ```typescript
+ * timestampedName('products.csv')
+ * // Returns: 'products-2025-11-01T18-30-45-123Z.csv'
+ * ```
+ */
+export function timestampedName(name: string): string {
+ const ts = new Date().toISOString().replace(/[:.]/g, '-');
+ const base = name.replace(/\.csv$/i, '');
+ return `${base}-${ts}.csv`;
+}
+```
+
+### Job ID Generator (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking ingestion runs.
+ * Format: {PREFIX}_{ENTITY}_{DATE}_{TIME}_{RANDOM}
+ * Example: SCHEDULED_PROD_20250124_183045_abc123
+ */
+
+/**
+ * Generate unique job ID
+ *
+ * @param prefix - Job prefix (SCHEDULED, ADHOC, etc.)
+ * @param entity - Entity type (PROD, INV, etc.)
+ * @returns Unique job ID string
+ */
+export function generateJobId(prefix: string, entity: string): string {
+ const now = new Date();
+ const date = now.toISOString().slice(0, 10).replace(/-/g, '');
+ const time = now.toISOString().slice(11, 19).replace(/:/g, '');
+ const random = Math.random().toString(36).slice(2, 8).toUpperCase();
+ return `${prefix}_${entity}_${date}_${time}_${random}`;
+}
+```
+
+---
+
+## 📄 Mapping Configuration
+
+**File:** `config/products.import.csv.json`
+
+```json
+{
+ "name": "products.import.csv",
+ "version": "1.2.0",
+ "description": "CSV → Product Event Mapping",
+ "fields": {
+  "ref": {
+   "source": "ref",
+   "required": true,
+   "resolver": "sdk.trim",
+   "comment": "Product reference from CSV"
+  },
+  "type": {
+   "source": "type",
+   "required": true,
+   "resolver": "sdk.uppercase",
+   "comment": "Product type (STANDARD, VARIANT)"
+  },
+  "status": {
+   "source": "status",
+   "required": true,
+   "resolver": "sdk.uppercase",
+   "comment": "Product status (ACTIVE, INACTIVE)"
+  },
+  "name": {
+   "source": "name",
+   "required": true,
+   "resolver": "sdk.trim",
+   "comment": "Product name/title"
+  },
+  "summary": {
+   "source": "summary",
+   "required": false,
+   "comment": "Product short description"
+  },
+  "gtin": {
+   "source": "gtin",
+   "required": false,
+   "resolver": "sdk.trim",
+   "comment": "Global Trade Item Number (barcode)"
+  },
+  "catalogue.ref": {
+   "source": "$context.catalogueRef",
+   "required": false,
+   "comment": "Catalogue reference (injected from context, NOT in CSV)"
+  },
+  "metadata.source": {
+   "value": "S3_CSV",
+   "required": false,
+   "comment": "Static value - identifies data source (NOT in CSV)"
+  },
+  "categoryRefs": {
+   "source": "category_refs",
+   "resolver": "custom.splitByPipe",
+   "comment": "Pipe-separated categories (custom resolver)"
+  },
+  "price": {
+   "resolver": "custom.buildPriceArray",
+   "comment": "Build price array from default/special fields (custom resolver)"
+  },
+  "taxType": {
+   "resolver": "custom.buildTaxType",
+   "comment": "Build taxType object from country/group/tariff (custom resolver)"
+  }
+ }
+}
+```
+
+**Note:** Custom resolvers (`custom.splitByPipe`, `custom.buildPriceArray`, `custom.buildTaxType`) are defined in the Main Orchestration Service.
+
+---
+
+## 9. Package Configuration
+
+### `package.json`
+
+```json
+{
+ "name": "s3-csv-product-event-ingestion",
+ "version": "1.2.0",
+ "description": "S3 CSV Product Event Ingestion Connector",
+ "type": "module",
+ "main": "src/index.ts",
+ "dependencies": {
+  "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+  "@versori/run": "^1.0.0"
+ },
+ "devDependencies": {
+  "@types/node": "^20.0.0",
+  "typescript": "^5.0.0"
+ }
+}
+```
+
+---
+
+## 6. Deployment Instructions
+
+### Deploy to Versori
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Test locally (if using Versori CLI)
+npm run dev
+
+# 3. Deploy to Versori platform
+npm run deploy
+```
+
+### Configure Activation Variables
+
+In Versori platform settings, configure all variables listed in the Activation Variables section above.
+
+---
+
+## 7. Testing
+
+### Test Scheduled Ingestion
+
+Upload a test CSV file to S3 incoming prefix and wait for the scheduled run.
+
+**Check logs:**
+
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and S3
+[STEP 3/8] Discovering files on S3
+[FILE 1/1] Processing file: products_20250124.csv
+[STEP 4/8] Downloading and parsing: products_20250124.csv
+[STEP 5/8] Transforming 5 products from products_20250124.csv
+[STEP 6/8] Sending 5 events to Fluent Commerce
+[STEP 7/8] Archiving file: products_20250124.csv
+[STEP 8/8] Completing job and calculating totals
+```
+
+### Test Ad hoc Ingestion
+
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{}'
+
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "filePattern": "urgent_*.csv", }'
+```
+
+### Test Job Status Query
+
+```bash
+curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
+ -H "X-API-Key: your-secret-key" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "jobId": "ADHOC_PROD_20251024_183045_abc123"
+ }'
+```
+
+---
+
+## Monitoring
+
+### Success Response
+
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 50,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 50,
+   "eventsFailed": 0
+  }
+ ],
+ "duration": 12345
+}
+```
+
+### Partial Success Response
+
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "totalRecords": 50,
+ "eventsSent": 45,
+ "eventsFailed": 5,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": true,
+   "recordsProcessed": 50,
+   "eventsSent": 45,
+   "eventsFailed": 5,
+   "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
+  }
+ ],
+ "duration": 12345
+}
+```
+
+### Error Response
+
+```json
+{
+ "success": false,
+ "filesProcessed": 0,
+ "filesFailed": 1,
+ "totalRecords": 0,
+ "eventsSent": 0,
+ "eventsFailed": 0,
+ "results": [
+  {
+   "file": "products_2025-01-22.csv",
+   "success": false,
+   "error": "CSV parse error: Invalid structure"
+  }
+ ],
+ "duration": 876
+}
+```
+
+### Monitoring Metrics
+
+Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
+
+- **Files Processed** - Total files successfully processed
+- **Events Sent** - Total events sent to Fluent Commerce
+- **Events Failed** - Events that failed (check rejection reports)
+- **Processing Duration** - Time taken for complete workflow
+- **Rate Limiting** - Watch for 429 errors indicating throttling
+
+Use the status webhook for dashboards and automated monitoring.
+
+---
+
+- **No files found:** Check `s3IncomingPrefix` and `filePattern` (object keys only)
+- **CSV parse failed:** Move to errors/; validate CSV structure and encoding
+- **High event failures:** Inspect logs; consider Batch API for very high volumes
+- **429 throttling:** Add small backoff/delay or use Batch API
+- **S3 connection errors:** Verify credentials, bucket, region, and permissions
+
+---
+
+## 9. Key Takeaways
+
+- ✅ **Modular service functions** - processFile(), sendEvents(), writeEventLog() keep code composable
+- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
+- ✅ **Buffer import required** - `import { Buffer } from 'node:buffer';` for Deno/Versori compatibility
+- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
+- ✅ **JobTracker** - Track job lifecycle with KV persistence
+- ❌ **NO VersoriFileTracker** - S3 uses moveObject for deduplication (SFTP-specific only)
+- ❌ **NO IngestionOrchestrator** - Removed from SDK - use inline modular services instead
+- ✅ **S3 deduplication** - Physical file movement to processed/errors prefixes
+- ✅ **S3 dispose()** - Always cleanup in finally block
+- ✅ **S3 method names** - listObjects, downloadObject, moveObject (not \*File)
+- ✅ **Per-record error handling** - Continue on individual failures
+- ✅ **Externalized mapping** - Use JSON config file for field mappings
+- ✅ **File-level error handling** - Don't stop on single file failure
+- ✅ **S3 vs SFTP** - Different deduplication strategies (moveObject vs KV state)
+- ✅ **uploadFile signature** - Uses Buffer.from(content, 'utf8') for text content
+
+---
+
+[← Back to Versori Event API Templates](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) | [Versori Platform Guide →](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)