@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/ingestion/event-api/template-ingestion-s3-xml-product-event.md

@@ -1,2395 +1,2395 @@
----
-template_id: tpl-ingest-s3-xml-to-product-event
-canonical_filename: template-ingestion-s3-xml-product-event.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: ingestion
-source: s3-xml
-destination: fluent-event-api
-entity: product
-format: xml
-logging: versori
-status: stable
-features:
-  - batched-events
-  - attribute-transformation
-  - memory-management
-  - json-serialization-handling
-  - enhanced-logging
----
-
-# Template: Ingestion - S3 XML to Product Event
-
-**Template Version:** 2.0.0
-**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
-**Last Updated:** 2025-01-24
-**Deployment Target:** Versori Platform
-
-**🆕 Version 2.0.0 Enhancements:**
-- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
-- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
-- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large XML files)
-- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
-- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
-
----
-
-## 📚 STEP 1: Load These Docs Into Your AI (Human Checklist)
-
-1. REQUIRED (load all)
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
-   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
-Copy-paste list (open these):
-fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
-fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
-fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
-fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
-
----
-
-## 📋 Implementation Prompt
-
-```
-I need a Versori scheduled ingestion that:
-
-1) Lists XML files on S3 (deduplication via moveObject - NO VersoriFileTracker)
-2) Downloads and parses XML with array normalization (CRITICAL: single object vs array)
-3) Transforms records with UniversalMapper per mapping JSON
-4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
-5) Moves files to processed/ or errors/ prefixes for natural deduplication
-6) Tracks progress with JobTracker and exposes a job-status webhook
-7) Uses native Versori log from context
-
-Use the loaded docs to fill in SDK specifics and best practices.
-Keep the structure identical to the template; only adapt where needed.
-```
-
----
-
-## 📋 Template Overview
-
-This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, file patterns) are configured via activation variables. Data shape and logic (mapping JSON, XML structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from S3 XML, transforms it, and sends events to the Fluent Commerce Event API.
-
-### What This Template Does
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    INGESTION WORKFLOW                            │
-└─────────────────────────────────────────────────────────────────┘
-
-1. TRIGGER
-   ├─ Scheduled (Cron): Runs automatically every hour
-   ├─ Ad hoc (Webhook): Manual trigger for immediate processing
-   └─ Status Query (Webhook): Check job progress
-
-2. DISCOVER FILES (S3DataSource)
-   ├─ List objects from S3 bucket
-   ├─ Filter by prefix and pattern (products/*.xml)
-   ├─ NO VersoriFileTracker (S3 uses moveObject for deduplication)
-   └─ Sort by last modified
-
-3. DOWNLOAD & PARSE (XMLParserService)
-   ├─ Download object from S3
-   ├─ Parse XML to JavaScript object
-   ├─ ⚠️ CRITICAL: Normalize single object vs array
-   └─ Validate XML structure
-
-4. TRANSFORM (UniversalMapper)
-   ├─ Map XML fields to Fluent schema
-   ├─ Apply SDK resolvers (trim, uppercase, etc.)
-   ├─ Handle nested objects (price, taxType)
-   ├─ Handle arrays (categoryRefs)
-   └─ Collect transformation errors
-
-5. SEND EVENTS (Event API)
-   ├─ Loop through transformed products
-   ├─ Send UPSERT_PRODUCT event (async)
-   ├─ Track success/failure count
-   └─ Continue on individual failures
-
-6. ARCHIVE (S3DataSource - Natural Deduplication)
-   ├─ Move object to processed/ prefix (success)
-   ├─ Or move to errors/ prefix (failures)
-   ├─ Generate timestamped archive name
-   └─ Files in incoming/ are removed naturally
-
-7. TRACK JOB (JobTracker)
-   ├─ Update job status at each step
-   ├─ Store final result in KV
-   ├─ Enable status queries via webhook
-   └─ Handle errors gracefully
-```
-
-### Key Features
-
-- Job tracking with status queries
-- Execution modes: scheduled, ad hoc, status query
-- Uses S3DataSource, XMLParserService, UniversalMapper, JobTracker
-- Error handling, retry logic, and S3 cleanup
-- **S3 Deduplication:** moveObject to processed/errors prefixes (NO VersoriFileTracker)
-- **Key Difference from SFTP:** Physical file movement vs KV state tracking
-- **XML Array Normalization:** CRITICAL handling of single object vs array
-- Event API: Per-record failures don't block other records
-- S3 dispose() in finally block
-
-Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
-
-### 📦 Package Information
-
-**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-
-Use the latest SDK version:
-
-```bash
-npm install @fluentcommerce/fc-connect-sdk@^0.1.39
-```
-
----
-
-**Templates are designed for direct deployment; customize via activation variables.**
-
----
-
-## ⚙️ Activation Variables
-
-**Configuration is driven by activation variables - modify these instead of code:**
-
-```json
-{
-  "fluentRetailerId": "1",
-  "eventConcurrency": 1,
-  "s3Bucket": "my-product-bucket",
-  "s3Region": "us-east-1",
-  "s3AccessKeyId": "AKIA...",
-  "s3SecretAccessKey": "********",
-  "s3IncomingPrefix": "products/incoming/",
-  "s3ProcessedPrefix": "products/processed/",
-  "s3ErrorPrefix": "products/errors/",
-  "s3LogsPrefix": "products/logs/",
-  "filePattern": "products_*.xml",
-  "catalogueRef": "PC:MASTER:2",
-  "catalogueType": "MASTER",
-  "eventName": "UPSERT_PRODUCT",
-  "eventMode": "async",
-  "maxFilesPerRun": 10,
-  "processingMode": "per-file",
-  "fileChunkSize": 5,
-  "validateConnection": true,
-  "enableFileTracking": true,
-  "trackDuration": true,
-  "useBatchedEvents": false,
-  "maxProductsUnderBatchedEvent": 100,
-  "memoryBatchSize": 250
-}
-```
-
-Note: Webhook security is handled by Versori's native connection authentication. The `connection` parameter in webhook definitions ensures only authenticated requests are processed.
-
-### Variable Explanations
-
-| Variable            | Purpose                      | Default               | Customization Hints                 |
-| ------------------- | ---------------------------- | --------------------- | ----------------------------------- |
-| `fluentRetailerId`  | Retailer ID for Event API    | -                     | Required - Fluent retailer ID      |
-| `eventConcurrency`  | Event sending concurrency    | `1`                   | `1` = sequential, `>1` = parallel (3-10 recommended) |
-| `s3Bucket`          | S3 bucket name               | -                     | Required - your S3 bucket           |
-| `s3Region`          | S3 region                    | `us-east-1`           | AWS region (e.g., us-west-2)        |
-| `s3AccessKeyId`     | AWS access key ID            | -                     | Required - AWS credentials          |
-| `s3SecretAccessKey` | AWS secret access key        | -                     | Required - AWS credentials          |
-| `s3IncomingPrefix`  | Incoming files prefix        | `products/incoming/`  | Where new files arrive              |
-| `s3ProcessedPrefix` | Processed files archive      | `products/processed/` | Where files move after success      |
-| `s3ErrorPrefix`     | Failed files directory       | `products/errors/`    | Where files move after errors       |
-| `s3LogsPrefix`      | Event processing logs        | `products/logs/`      | Where event logs are written        |
-| `filePattern`       | File name filter             | `products_*.xml`      | Glob pattern for matching files     |
-| `catalogueRef`      | Product catalogue reference  | `PC:MASTER:2`         | Target catalogue in Fluent          |
-| `catalogueType`     | Catalogue type               | `MASTER`              | Usually MASTER or STANDARD          |
-| `eventName`         | Event to send                | `UPSERT_PRODUCT`      | Event name from Rubix               |
-| `eventMode`         | Event processing mode        | `async`               | async (recommended) or sync         |
-| `maxFilesPerRun`    | Max files per execution      | `10`                  | Prevent timeout on large batches    |
-| `processingMode`    | Processing mode              | `per-file`            | per-file, chunked, or batch         |
-| `fileChunkSize`     | Files per chunk (chunked)    | `5`                   | Used when processingMode=chunked    |
-| `validateConnection` | Validate S3 on initialization | `true`               | Fail-fast if S3 credentials invalid |
-| `enableFileTracking` | Enable file deduplication    | `true`               | Track processed files in KV store   |
-| `trackDuration`     | Track processing duration     | `true`               | Log timing metrics for performance  |
-| `useBatchedEvents` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
-| `maxProductsUnderBatchedEvent` | Products per batch | `100`          | Only used when `useBatchedEvents=true` (default: 100) |
-| `memoryBatchSize`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
-
----
-
-## 🔒 SDK Automatic Behaviors (v0.1.40+)
-
-**The SDK automatically validates and retries for improved reliability:**
-
-### retailerId Validation
-- **SDK validates** `retailerId` before calling `sendEvent()`
-- **Checks:** `event.retailerId || client.retailerId`
-- **If missing:** Throws `"retailerId is required for Event API..."`
-- **Configuration:** Set via `fluentRetailerId` activation variable (recommended)
-
-### 401 Auth Retry
-- **Automatic retry** for platform auth failures (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s)
-- **Applies to:** All `sendEvent()` calls (async and sync modes)
-- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
-
-### 5xx Server Retry
-- **Automatic retry** for transient server errors (3 attempts)
-- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
-- **Protects:** Against Fluent API transient failures
-
-### No Code Changes Required
-- All templates remain compatible
-- Retry logic is automatic and transparent
-- Better error messages guide configuration
-
-**See:** [Event API Guide](./event-api-guide.md) for complete details
-
----
-
-### 📁 S3 Deduplication Pattern (NO VersoriFileTracker)
-
-**CRITICAL: S3 uses physical file movement, NOT VersoriFileTracker**
-
-VersoriFileTracker is **SFTP-specific** and should **NOT** be used with S3. S3 achieves deduplication through physical file movement.
-
-#### How S3 Deduplication Works
-
-**3-Step Process:**
-
-1. **Discovery:** List objects from `s3IncomingPrefix` matching `filePattern`
-2. **Process:** Download, parse, transform, and send events
-3. **Archive:** Move object to `s3ProcessedPrefix` or `s3ErrorPrefix`
-4. **Natural Deduplication:** Once moved, files are no longer in incoming/
-
-#### File Lifecycle Example
-
-```
-INITIAL STATE:
-s3://bucket/products/incoming/products_20250124_001.xml
-
-↓ (workflow discovers file)
-
-PROCESSING:
-- List objects from "products/incoming/"
-- Download: products_20250124_001.xml
-- Parse XML → Normalize array → Map to events → Send to Event API
-
-↓ (success)
-
-ARCHIVE (moveObject):
-s3://bucket/products/processed/products_20250124_001-20250124T183045.xml
-
-↓ (result)
-
-INCOMING FOLDER NOW EMPTY:
-s3://bucket/products/incoming/ (file is gone - no duplicates possible)
-```
-
-#### Key Differences: S3 vs SFTP
-
-| Aspect                   | S3 (This Template)            | SFTP (Other Templates)           |
-| ------------------------ | ----------------------------- | -------------------------------- |
-| **Deduplication Method** | Physical file movement        | VersoriFileTracker (KV state)    |
-| **State Storage**        | None needed                   | KV store tracks processed files  |
-| **File Tracking**        | ❌ NO VersoriFileTracker      | ✅ VersoriFileTracker required   |
-| **Reprocessing**         | Move files back to incoming/  | Clear KV state or forceReprocess |
-| **Complexity**           | Simpler (no state management) | More complex (state management)  |
-
-#### Summary
-
-✅ **DO:** Use S3 moveObject for deduplication
-✅ **DO:** Move files to processed/ or errors/ prefixes
-❌ **DON'T:** Import VersoriFileTracker in S3 templates
-❌ **DON'T:** Use KV state tracking for S3 files
-✅ **DO:** Use VersoriFileTracker for SFTP templates (different use case)
-
----
-
-## When to Use Event API vs Batch API
-
-### ✅ Use Event API (`sendEvent`) For:
-
-| Entity Type         | Use Case                               | Why Event API                                |
-| ------------------- | -------------------------------------- | -------------------------------------------- |
-| **Products**        | Product catalog sync, variant updates  | Triggers workflows, validates business rules |
-| **Locations**       | Store/warehouse setup                  | Requires workflow orchestration              |
-| **Customers**       | Customer registration, profile updates | Needs workflow for downstream systems        |
-| **Orders**          | Single order creation                  | Event-driven fulfillment workflows           |
-| **Custom Entities** | Any entity needing workflow triggers   | Full Rubix workflow support                  |
-
-### ❌ Use Batch API Instead For:
-
-| Entity Type        | Use Case                           | Why Batch API                                   |
-| ------------------ | ---------------------------------- | ----------------------------------------------- |
-| **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
-
-### 🔄 Use GraphQL Mutations For:
-
-| Scenario              | Why GraphQL                            |
-| --------------------- | -------------------------------------- |
-| **Single operations** | Create one order, update one product   |
-| **Complex queries**   | Fetch data with relationships          |
-| **Testing/debugging** | Direct API control, immediate feedback |
-
----
-
-## XML File Format
-
-### Sample: products.xml
-
-Based on your `UPSERT_PRODUCT` event payload:
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<products>
-  <product>
-    <ref>G_PROD_WITH_NO_STANDARD</ref>
-    <type>VARIANT</type>
-    <status>ACTIVE</status>
-    <gtin>MH01-XS-Orange</gtin>
-    <name>Chaz Kangeroo Hoodie-XS-Orange main</name>
-    <summary>&lt;p&gt;test short description&lt;/p&gt;</summary>
-    <categoryRefs>
-      <ref>STANDARD_CATEGORY</ref>
-    </categoryRefs>
-    <price>
-      <item>
-        <type>DEFAULT</type>
-        <currency>USD</currency>
-        <value>52.000000</value>
-      </item>
-      <item>
-        <type>SPECIAL</type>
-        <currency>USD</currency>
-        <value>9.000000</value>
-      </item>
-    </price>
-    <taxType>
-      <country>AU</country>
-      <group>Tax Group</group>
-      <tariff>Tax Tariff</tariff>
-    </taxType>
-    <catalogue>
-      <ref>PC:MASTER:2</ref>
-      <type>MASTER</type>
-    </catalogue>
-  </product>
-  <product>
-    <ref>G_PROD_002</ref>
-    <type>VARIANT</type>
-    <status>ACTIVE</status>
-    <gtin>MH02-M-Blue</gtin>
-    <name>Kangeroo Hoodie-M-Blue</name>
-    <summary>&lt;p&gt;Blue variant&lt;/p&gt;</summary>
-    <categoryRefs>
-      <ref>STANDARD_CATEGORY</ref>
-      <ref>SEASONAL</ref>
-    </categoryRefs>
-    <price>
-      <item>
-        <type>DEFAULT</type>
-        <currency>USD</currency>
-        <value>45.000000</value>
-      </item>
-      <item>
-        <type>SPECIAL</type>
-        <currency>USD</currency>
-        <value>35.000000</value>
-      </item>
-    </price>
-    <taxType>
-      <country>AU</country>
-      <group>Tax Group</group>
-      <tariff>Tax Tariff</tariff>
-    </taxType>
-    <catalogue>
-      <ref>PC:MASTER:2</ref>
-      <type>MASTER</type>
-    </catalogue>
-  </product>
-</products>
-```
-
-**XML Field Mapping:**
-
-- Nested structure for objects (e.g., `<taxType><country>AU</country></taxType>`)
-- Arrays with repeated elements (e.g., `<categoryRefs><ref>CAT1</ref><ref>CAT2</ref></categoryRefs>`)
-- HTML content in CDATA or escaped (e.g., `&lt;p&gt;description&lt;/p&gt;`)
-
----
-
-## Event Sending Configuration
-
-**Simple Configuration:** One variable controls everything
-
-```json
-{
-  "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
-}
-```
-
-**How it works:**
-
-- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
-- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
-- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
-- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
-
-**Configuration Guidelines:**
-
-- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
-- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
-- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
-- **Note:** Fluent API supports concurrent requests - adjust based on your needs
-
-**Why Single Variable?**
-
-- **Simpler:** One variable instead of two (`mode` + `concurrency`)
-- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
-- **Less config:** Fewer activation variables to manage
-- **Flexible:** Easy to tune performance (just change the number)
-
----
-
-## 🔧 Production Reference Implementation
-
-### Processing Modes
-
-| Mode       | Default | What happens                                                                                      | When to use                                              |
-| ---------- | ------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
-| `per-file` | ✅      | Process one file end-to-end (discover → parse → Event API → archive) before moving on             | Recommended for production S3 feeds and very large files |
-| `chunked`  | ➖      | Processes `fileChunkSize` files at a time, completing each chunk before the next                  | High-volume feeds where per-file would take too long     |
-| `batch`    | ➖      | Loads every matched file into memory, produces a single Event API job, and archives when complete | Tiny datasets or smoke tests only                        |
-
-Set the mode via activation variables:
-
-```json
-{
-  "processingMode": "per-file",
-  "fileChunkSize": "5",
-  "maxFilesPerRun": "25"
-}
-```
-
-### Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-product-event-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── product.types.ts            # Shared type definitions
-```
-
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
-
----
-
-## Workflow Files
-
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-
-All time-based triggers that run automatically on cron schedules.
-
-#### `src/workflows/scheduled/daily-product-sync.ts`
-
-**Purpose**: Automatic Daily product sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Scheduled Workflow: Daily Product Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: product-sync.service.ts
- */
-export const dailyProductSync = schedule(
-  'product-sync-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv } = ctx;
-    const jobId = `product-sync-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    log.info('🚀 Starting scheduled product sync', { jobId });
-
-    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
-    await tracker.updateJob(jobId, { status: 'processing' });
-
-    try {
-      // Reuse shared orchestration logic
-      log.info('⚙️ Executing product ingestion workflow', { jobId });
-      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'schedule' }, tracker);
-      await tracker.markCompleted(jobId, result);
-      log.info('✅ Scheduled sync completed successfully', { jobId, ...result });
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      log.error('❌ Scheduled sync failed', { jobId, error: e?.message });
-      await tracker.markFailed(jobId, e);
-      return { success: false, jobId, error: e?.message };
-    }
-  })
-);
-```
-
----
-
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-
-All HTTP-based triggers that create webhook endpoints.
-
-#### `src/workflows/webhook/adhoc-product-sync.ts`
-
-**Purpose**: Manual product sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { executeProductIngestion } from '../../services/product-sync.service';
-
-/**
- * Webhook: Manual Product Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
- * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
- *
- * Pattern: webhook().then(http()) - needs Fluent API access
- * Uses shared service: product-sync.service.ts
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const adhocProductSync = webhook('product-sync-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'product-sync-adhoc', // Versori validates API key
-}).then(
-  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
-    const { log, openKv, data } = ctx;
-    const jobId = `product-sync-adhoc-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    const filePattern = data?.filePattern as string;
-    const maxFiles = data?.maxFiles as number;
-
-    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-    });
-
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'manual',
-      stage: 'initialization',
-      status: 'queued',
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
-      .then((result) => {
-        log.info('✅ [BACKGROUND] Product sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          filesFailed: result.filesFailed,
-          recordsProcessed: result.recordsProcessed,
-        });
-        return tracker.markCompleted(jobId, result);
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-
-        log.error('❌ [BACKGROUND] Product sync failed', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-        });
-
-        return tracker.markFailed(jobId, errorMessage);
-      });
-
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Product sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-
-#### `src/workflows/webhook/job-status-check.ts`
-
-**Purpose**: Query job status
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
-**Request body**: `{ "jobId": "product-sync-1234567890" }`
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
- * Request body: { "jobId": "product-sync-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- *
- * SECURITY: Authentication handled via connection parameter
- * No manual API key validation needed - Versori manages this via connection auth
- */
-export const productSyncJobStatus = webhook('product-sync-job-status', {
-  response: { mode: 'sync' },
-  connection: 'product-sync-job-status',
-}).then(
-  fn('status', async ctx => {
-    // ═══════════════════════════════════════════════════════════
-    // EXECUTION BOUNDARY: Job Status Query Start
-    // ═══════════════════════════════════════════════════════════
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-
-    log.info('🔍 Job status query received', { jobId });
-
-    if (!jobId) {
-      log.warn('⚠️ Missing jobId in request');
-      return { success: false, error: 'jobId required' };
-    }
-
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-
-    if (status) {
-      log.info('✅ Job status found', { jobId, status: status.status });
-      return { success: true, jobId, ...status };
-    } else {
-      log.warn('⚠️ Job not found', { jobId });
-      return { success: false, error: 'Job not found', jobId };
-    }
-    // ═══════════════════════════════════════════════════════════
-    // EXECUTION BOUNDARY: Job Status Query End
-    // ═══════════════════════════════════════════════════════════
-  })
-);
-```
-
----
-
-### 3. Entry Point (`index.ts`)
-
-**Purpose**: Register all workflows with Versori platform using MemoryInterpreter pattern
-
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * PATTERN: MemoryInterpreter for workflow registration
- * - Workflow definitions stored in memory
- * - Versori runtime discovers and executes via exports
- * - No file I/O during workflow registration
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- */
-
-// Import scheduled workflows
-import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
-
-// Import webhook workflows
-import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
-import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
-
-// MemoryInterpreter Pattern: Export workflows for Versori runtime discovery
-// Workflows are held in memory and executed when triggered by platform
-export {
-  // Scheduled (time-based triggers)
-  dailyProductSync,
-
-  // Webhooks (HTTP-based triggers)
-  adhocProductSync,
-  productSyncJobStatus,
-};
-```
-
-**What Gets Exposed:**
-- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
-- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
-- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
-
----
-
-### Adding New Workflows
-
-**To add a scheduled workflow:**
-1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**To add a webhook workflow:**
-1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
-2. Export the workflow from the file
-3. Import and re-export in `index.ts`
-
-**Example - Adding hourly delta sync:**
-
-```typescript
-// src/workflows/scheduled/hourly-delta-sync.ts
-export const hourlyDeltaSync = schedule(
-  'product-delta-hourly',
-  '0 * * * *' // Every hour
-).then(
-  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
-    // Delta sync logic (skip BPP)
-    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
-    return result;
-  })
-);
-
-// index.ts (add to imports and exports)
-import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
-export { daily_product_sync, hourlyDeltaSync, ... };
-```
-
----
-## 3. Type Definitions (src/types/product-ingestion.types.ts)
-
-```typescript
-/**
- * Type Definitions for Product Ingestion
- *
- * Centralized type definitions for product ingestion workflow
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Product interface - represents transformed product data
- */
-export interface Product {
-  ref: string;
-  type?: string;
-  status?: string;
-  name: string;
-  summary?: string;
-  gtin?: string;
-  catalogue?: { ref?: string };
-  attributes?: Record<string, unknown>;
-}
-
-/**
- * Event configuration
- */
-export interface EventConfig {
-  eventName: string;
-  catalogueRef: string;
-  catalogueType: string;
-  eventMode: 'async' | 'sync';
-}
-
-/**
- * Event result - tracks success/failure counts
- */
-export interface EventResult {
-  eventsSent: number;
-  eventsFailed: number;
-  errors: Array<{ productRef: string; error: string }>;
-}
-
-/**
- * Process file result
- */
-export interface ProcessFileResult {
-  success: boolean;
-  products: Product[];
-  error?: string;
-}
-
-/**
- * Versori Context Interface
- * Represents the Versori runtime context passed to workflow functions
- */
-export interface VersoriContext {
-  log: {
-    info: (message: string, data?: Record<string, unknown>) => void;
-    warn: (message: string, data?: Record<string, unknown>) => void;
-    error: (message: string, data?: Record<string, unknown>) => void;
-    debug?: (message: string, data?: Record<string, unknown>) => void;
-  };
-  openKv: (namespace: string) => {
-    get: (key: string) => Promise<unknown>;
-    set: (key: string, value: unknown) => Promise<void>;
-    delete: (key: string) => Promise<void>;
-  };
-  activation: {
-    getVariable: (name: string) => string | undefined;
-    connections?: Record<string, unknown>;
-  };
-  connections?: Record<string, unknown>;
-  data?: unknown;
-  fetch?: typeof fetch;
-}
-
-/**
- * Parameters for ingestion workflow
- */
-export interface ProductIngestionParams {
-  jobId: string;
-  triggeredBy: 'schedule' | 'webhook';
-  filePattern?: string;
-  maxFiles?: number;
-  forceReprocess?: boolean;
-  catalogueRef?: string;
-  priority?: 'normal' | 'high';
-}
-
-/**
- * Result from ingestion workflow
- */
-export interface ProductIngestionResult {
-  success: boolean;
-  jobId: string;
-  filesProcessed: number;
-  filesFailed: number;
-  filesSkipped?: number;
-  recordsProcessed: number;
-  eventsSent: number;
-  eventsFailed: number;
-  fileResults: FileProcessingResult[];
-  error?: string;
-}
-
-/**
- * Per-file processing result
- */
-export interface FileProcessingResult {
-  fileName: string;
-  success: boolean;
-  skipped?: boolean;
-  recordsProcessed: number;
-  eventsSent: number;
-  eventsFailed: number;
-  duration: number;
-  error?: string;
-}
-
-/**
- * Parsed XML document structure
- */
-export interface ParsedProductsDocument {
-  products?: {
-    product?: Product | Product[];
-  };
-}
-```
-
----
-
-## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
-
-```typescript
-/**
- * Product File Processor Service
- *
- * Downloads XML files from S3, parses, and transforms with UniversalMapper.
- * XML-specific: Handles array normalization (single vs multiple products).
- */
-
-import {
-  S3DataSource,
-  XMLParserService,
-  UniversalMapper,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { ProcessFileResult, Product, ParsedProductsDocument } from '../types/product-ingestion.types';
-
-/**
- * Service for processing XML product files from S3
- */
-export class ProductFileProcessorService {
-  constructor(
-    private s3: S3DataSource,
-    private xmlParser: XMLParserService,
-    private mapper: UniversalMapper,
-    private catalogueRef: string
-  ) {}
-
-  /**
-   * Download XML file from S3, parse, and transform with UniversalMapper
-   *
-   * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
-   * - Explicit null assignments after each step
-   * - Finally block guarantees cleanup
-   * - Prevents OOM errors on large XML files
-   */
-  async downloadParseAndTransform(objectKey: string): Promise<ProcessFileResult> {
-    // ✅ CRITICAL: Variables for cleanup tracking
-    let xmlContent: string | null = null;
-    let parsed: unknown | null = null;
-    let rawProducts: any[] | null = null;
-
-    try {
-      // STEP 1: Download XML content from S3
-      xmlContent = (await this.s3.downloadObject(objectKey, {
-        encoding: 'utf8',
-      })) as string;
-
-      // STEP 2: Parse XML with type safety
-      try {
-        parsed = await this.xmlParser.parse(xmlContent);
-      } catch (parseError: unknown) {
-        const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
-        return {
-          success: false,
-          products: [],
-          error: `XML parse error: ${errorMessage}`,
-        };
-      }
-
-      // ✅ Clear xmlContent from memory - no longer needed after parsing
-      xmlContent = null;
-
-      // Type guard: Validate parsed structure
-      if (!this.isParsedProductsDocument(parsed)) {
-        return {
-          success: false,
-          products: [],
-          error: 'Invalid XML structure: missing products element',
-        };
-      }
-
-      // STEP 3: Normalize products array (single element → object, multiple → array)
-      rawProducts = Array.isArray(parsed.products?.product)
-        ? parsed.products.product
-        : parsed.products?.product
-          ? [parsed.products.product]
-          : [];
-
-      if (rawProducts.length === 0) {
-        return {
-          success: false,
-          products: [],
-          error: 'No products found in XML',
-        };
-      }
-
-      // ✅ Clear parsed data from memory - only need rawProducts now
-      parsed = null;
-
-      // STEP 4: Transform with UniversalMapper
-      // ✅ S3 XML: Merge context using spread pattern (same as SFTP XML)
-      const sourceDataWithContext = rawProducts.map(item => ({
-        ...item,
-        $context: { catalogueRef: this.catalogueRef },
-      }));
-
-      const mappingResult = await this.mapper.map(sourceDataWithContext);
-
-      if (!mappingResult.success) {
-        return {
-          success: false,
-          products: [],
-          error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
-        };
-      }
-
-      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-        this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-          skippedFields: mappingResult.skippedFields,
-          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-        });
-      }
-
-      return {
-        success: true,
-        products: mappingResult.data as Product[],
-      };
-    } catch (error: unknown) {
-      const errorMessage = error instanceof Error ? error.message : String(error);
-      return {
-        success: false,
-        products: [],
-        error: errorMessage,
-      };
-    } finally {
-      // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
-      xmlContent = null;
-      parsed = null;
-      rawProducts = null;
-    }
-  }
-
-  /**
-   * Type guard: Check if parsed value is a valid ParsedProductsDocument
-   */
-  private isParsedProductsDocument(value: unknown): value is ParsedProductsDocument {
-    if (typeof value !== 'object' || value === null) {
-      return false;
-    }
-    const doc = value as Record<string, unknown>;
-    return 'products' in doc;
-  }
-}
-```
-
----
-
-## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
-
-```typescript
-/**
- * Event Sender Service
- *
- * Sends product events to Fluent Commerce Event API with per-record error handling.
- * Continues processing on individual failures (Event API best practice).
- * Supports configurable concurrency (sequential or parallel).
- */
-
-import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
-import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
-
-/**
- * Service for sending events to Fluent Commerce Event API
- *
- * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
- */
-export class EventSenderService {
-  constructor(
-    private client: FluentClient,
-    private log?: any  // ✅ Optional logger for progress tracking
-  ) {}
-
-  /**
-   * Send events to Fluent Commerce Event API with configurable concurrency
-   *
-   * **Performance Characteristics:**
-   * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
-   * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
-   * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
-   *
-   * **Implementation Strategy:**
-   * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
-   * - Concurrency > 1: Chunked parallel processing with bounded concurrency
-   * - Both modes: Per-record error tracking (failures don't block other events)
-   *
-   * @param products - Array of products to send as events
-   * @param eventConfig - Event configuration (name, catalogue ref, mode)
-   * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
-   * @returns EventResult with counts (eventsSent/eventsFailed) and error details
-   */
-  async sendEvents(
-    products: Product[],
-    eventConfig: EventConfig,
-    concurrency: number = 1
-  ): Promise<EventResult> {
-    // Validate concurrency (guard against invalid values)
-    const safeConc = Math.max(1, Math.floor(concurrency));
-
-    // Result accumulators
-    let eventsSent = 0;
-    let eventsFailed = 0;
-    const errors: Array<{ productRef: string; error: string }> = [];
-
-    // ✅ PRODUCTION ENHANCEMENT: Log event sending start
-    if (this.log) {
-      this.log.info('📤 Starting event sending', {
-        totalProducts: products.length,
-        concurrency: safeConc,
-        processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
-        eventName: eventConfig.eventName,
-        catalogueRef: eventConfig.catalogueRef
-      });
-    }
-
-    // Helper: Build event payload (DRY - reused in both modes)
-    const buildPayload = (product: Product) => ({
-      name: eventConfig.eventName,
-      entityRef: eventConfig.catalogueRef,
-      entityType: 'PRODUCT_CATALOGUE' as const,
-      entitySubtype: eventConfig.catalogueType,
-      rootEntityRef: eventConfig.catalogueRef,
-      rootEntityType: 'PRODUCT_CATALOGUE' as const,
-      attributes: product,
-    });
-
-    // ============================================================================
-    // SEQUENTIAL MODE (concurrency === 1)
-    // ============================================================================
-    if (safeConc === 1) {
-      for (let i = 0; i < products.length; i++) {
-        const product = products[i];
-
-        // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
-        if (this.log && i % 10 === 0) {
-          this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
-            productRef: product.ref,
-            progress: `${((i / products.length) * 100).toFixed(1)}%`
-          });
-        }
-
-        try {
-          await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
-          eventsSent++;
-        } catch (err: unknown) {
-          eventsFailed++;
-          const errorMsg = err instanceof Error ? err.message : String(err);
-          errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
-          // Continue processing (failure doesn't block other products)
-        }
-      }
-
-      // ✅ PRODUCTION ENHANCEMENT: Log completion
-      if (this.log) {
-        this.log.info('✅ Sequential event sending completed', {
-          totalProducts: products.length,
-          eventsSent,
-          eventsFailed
-        });
-      }
-
-      return { eventsSent, eventsFailed, errors };
-    }
-
-    // ============================================================================
-    // PARALLEL MODE (concurrency > 1)
-    // ============================================================================
-    const totalChunks = Math.ceil(products.length / safeConc);
-
-    for (let i = 0; i < products.length; i += safeConc) {
-      const chunk = products.slice(i, i + safeConc);
-      const chunkNumber = Math.floor(i / safeConc) + 1;
-
-      // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
-      if (this.log) {
-        this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
-          chunkNumber,
-          totalChunks,
-          productsInChunk: chunk.length,
-          productRange: `${i + 1}-${i + chunk.length}`,
-          totalProducts: products.length,
-          progress: `${((i / products.length) * 100).toFixed(1)}%`
-        });
-      }
-
-      // Fire all requests in chunk concurrently
-      const results = await Promise.allSettled(
-        chunk.map(product =>
-          this.client
-            .sendEvent(buildPayload(product), eventConfig.eventMode)
-            .then(() => ({ success: true as const, product }))
-            .catch(error => ({ success: false as const, product, error }))
-        )
-      );
-
-      // Aggregate chunk results into totals
-      let chunkSuccess = 0;
-      let chunkFailed = 0;
-
-      for (const result of results) {
-        if (result.status === 'fulfilled' && result.value.success) {
-          eventsSent++;
-          chunkSuccess++;
-        } else {
-          eventsFailed++;
-          chunkFailed++;
-          const error = result.status === 'fulfilled' ? result.value.error : result.reason;
-          const product = result.status === 'fulfilled' ? result.value.product : null;
-          errors.push({
-            productRef: product?.ref || 'unknown',
-            error: error?.message || String(error) || 'unknown error',
-          });
-        }
-      }
-
-      // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
-      if (this.log) {
-        this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
-          chunkNumber,
-          totalChunks,
-          chunkSuccess,
-          chunkFailed,
-          totalSentSoFar: eventsSent,
-          totalFailedSoFar: eventsFailed,
-          progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
-        });
-      }
-    }
-
-    // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
-    if (this.log) {
-      this.log.info('✅ Parallel event sending completed', {
-        totalProducts: products.length,
-        totalChunks,
-        concurrency: safeConc,
-        eventsSent,
-        eventsFailed
-      });
-    }
-
-    return { eventsSent, eventsFailed, errors };
-  }
-}
-```
-
----
-
-## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
-
-```typescript
-/**
- * Event Logger Service
- *
- * Writes event processing logs to S3 for audit/debugging.
- * Optional - can be used to create rejection reports.
- */
-
-import { Buffer } from 'node:buffer';
-import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Service for writing event logs to S3
- */
-export class EventLoggerService {
-  constructor(private s3: S3DataSource) {}
-
-  /**
-   * Write event log to S3
-   *
-   * @param objectKey - S3 object key for log file
-   * @param logData - Log data to write (JSON format)
-   */
-  async writeEventLog(objectKey: string, logData: unknown): Promise<void> {
-    try {
-      const logContent = JSON.stringify(logData, null, 2);
-      await this.s3.uploadFile(objectKey, logContent);
-    } catch (error: unknown) {
-      const errorMessage = error instanceof Error ? error.message : String(error);
-      throw new Error(`Failed to write event log: ${errorMessage}`);
-    }
-  }
-}
-```
-
-## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
-
-```typescript
-/**
- * MAIN INGESTION ORCHESTRATION SERVICE
- *
- * This is the heart of the ingestion workflow. It coordinates all steps:
- * 1. Initialize clients and services
- * 2. Discover files on S3
- * 3. Download and parse XML files
- * 4. Transform data with UniversalMapper
- * 5. Send events to Fluent Commerce
- * 6. Archive processed files
- * 7. Track file processing state (S3 uses moveObject for deduplication)
- * 8. Track job progress with JobTracker
- *
- * NAMING PATTERN (consistent across all use cases):
- * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
- * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
- * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- * KEY DIFFERENCES FOR S3:
- * - S3DataSource instead of SftpDataSource
- * - No VersoriFileTracker (S3 uses moveObject for deduplication)
- * - S3 listObjects instead of SFTP listFiles
- * - S3 moveObject instead of SFTP moveFile
- * - S3 dispose() for cleanup (S3DataSource does need dispose)
- */
-
-import { Buffer } from 'node:buffer';
-import {
-  createClient,
-  S3DataSource,
-  XMLParserService,
-  UniversalMapper,
-  JobTracker,
-  type FluentClient,
-  type JobStatus,
-} from '@fluentcommerce/fc-connect-sdk';
-import type {
-  ProductIngestionParams,
-  ProductIngestionResult,
-  FileProcessingResult,
-  EventConfig,
-} from '../types/product-ingestion.types';
-import { ProductFileProcessorService } from './product-file-processor.service';
-import { EventSenderService } from './event-sender.service';
-import { EventLoggerService } from './event-logger.service';
-import { timestampedName } from '../utils/s3-path.utils';
-
-import mappingConfig from '../../config/products.import.xml.json' with { type: 'json' };
-
-// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
-
-/**
- * Query job status from KV store
- *
- * NAMING: get{Entity}JobStatus or just getJobStatus (generic)
- *
- * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
- */
-export async function getJobStatus(
-  kv: ReturnType<VersoriContext['openKv']>,
-  jobId: string,
-  log: VersoriContext['log']
-): Promise<JobStatus | undefined> {
-  try {
-    const tracker = new JobTracker(kv, log);
-    return await tracker.getJob(jobId);  // ✅ Use getJob() not getJobStatus()
-  } catch (error: unknown) {
-    const errorMessage = error instanceof Error ? error.message : String(error);
-    log.error('Failed to get job status', {
-      jobId,
-      message: errorMessage,
-      stack: error instanceof Error ? error.stack : undefined,
-      errorType: error instanceof Error ? error.constructor.name : 'Error',
-    });
-    return undefined;
-  }
-}
-
-/**
- * MAIN ORCHESTRATION FUNCTION
- *
- * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
- *
- * This function implements the complete ingestion workflow in 8 steps.
- * Each step is clearly commented for AI understanding.
- */
-export async function executeProductIngestion(
-  ctx,
-  params: ProductIngestionParams
-): Promise<ProductIngestionResult> {
-
-  // ✅ VERSORI PLATFORM: Extract native log from context (LoggingService was removed - use native log)
-  const { log, openKv, activation } = ctx;
-  const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
-
-  // Open KV store for state management and job tracking
-  // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
-  // ✅ Pass native log to JobTracker
-  const kv = openKv(':project:');
-  const tracker = new JobTracker(kv, log);
-
-  const startTime = Date.now();
-  const fileResults: FileProcessingResult[] = [];
-
-  // ⚠️ CRITICAL: Declare S3 outside try block for disposal pattern
-  let s3: S3DataSource | undefined;
-
-  try {
-    // 
-    // STEP 1: Initialize Job Tracking
-    // 
-    log.info('📋 [STEP 1/8] Initializing job tracking', { jobId });
-
-    await tracker.createJob(jobId, {
-      triggeredBy,
-      filePattern: filePattern || 'default',
-      maxFiles: maxFiles || 'unlimited',
-      forceReprocess: !!forceReprocess
-    });
-
-    // 
-    // STEP 2: Initialize Fluent Client & S3 Connection
-    // 
-    log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and S3', { jobId });
-
-    const client = await createClient(ctx);
-
-    if (!client) {
-      throw new Error('Failed to create Fluent Commerce client');
-    }
-
-    // ✅ CRITICAL: Set retailerId for Event API calls
-    // Event API requires retailerId - fail fast if not configured
-    const fluentRetailerId = activation.getVariable('fluentRetailerId');
-    if (!fluentRetailerId) {
-      throw new Error('fluentRetailerId is required for Event API calls');
-    }
-    client.setRetailerId(fluentRetailerId);
-    log.info('✅ RetailerId set for Event API', { retailerId: fluentRetailerId });
-
-    // Get S3 configuration from activation variables
-    const s3Config = {
-      bucket: activation.getVariable('s3Bucket'),
-      region: activation.getVariable('s3Region') || 'us-east-1',
-      accessKeyId: activation.getVariable('s3AccessKeyId'),
-      secretAccessKey: activation.getVariable('s3SecretAccessKey'),
-      incomingPrefix: activation.getVariable('s3IncomingPrefix') || 'products/incoming/',
-      processedPrefix: activation.getVariable('s3ProcessedPrefix') || 'products/processed/',
-      errorPrefix: activation.getVariable('s3ErrorPrefix') || 'products/errors/',
-      logsPrefix: activation.getVariable('s3LogsPrefix') || 'products/logs/',
-      filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.xml'
-    };
-
-    // Validate S3 config
-    if (!s3Config.bucket) {
-      throw new Error('S3 configuration incomplete: missing bucket');
-    }
-
-    if (!s3Config.accessKeyId || !s3Config.secretAccessKey) {
-      throw new Error('S3 configuration incomplete: missing accessKeyId or secretAccessKey');
-    }
-
-    // Initialize S3 data source
-    // ✅ VERSORI PLATFORM: Pass native log from context
-    s3 = new S3DataSource(
-      {
-        type: 'S3_XML',
-        connectionId: 's3-product-ingestion',
-        name: 'Product Ingestion S3',
-        s3Config: {
-          bucket: s3Config.bucket,
-          region: s3Config.region,
-          accessKeyId: s3Config.accessKeyId,
-          secretAccessKey: s3Config.secretAccessKey,
-        },
-        validateConnection: true, // ✅ Enable connection validation on init
-      },
-      log
-    );
-
-    try {
-      // Connection already validated during initialization
-      // await s3.validateConnection(); // No longer needed - done in constructor
-      log.info('✅ S3 connection validated');
-
-      // 
-      // STEP 3: Discover Files on S3
-      // 
-      log.info('🔍 [STEP 3/8] Discovering files on S3', {
-        jobId,
-        prefix: s3Config.incomingPrefix,
-        filePattern: s3Config.filePattern
-      });
-
-      await tracker.updateJob(jobId, {
-        status: 'processing',
-        stage: 'file_discovery',
-        message: 'Discovering files on S3'
-      });
-
-      // List objects from S3
-      const allFiles = await s3.listObjects({
-        prefix: s3Config.incomingPrefix,
-        pattern: s3Config.filePattern
-      });
-
-      log.info(`📁 Discovered ${allFiles.length} files matching pattern`);
-
-      // Apply maxFiles limit if specified
-      const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
-
-      if (filesToProcess.length === 0) {
-        log.info('No files to process');
-
-        await tracker.markCompleted(jobId, {
-          filesProcessed: 0,
-          message: 'No files found to process'
-        });
-
-        return {
-          success: true,
-          jobId,
-          filesProcessed: 0,
-          filesFailed: 0,
-          recordsProcessed: 0,
-          eventsSent: 0,
-          eventsFailed: 0,
-          fileResults: []
-        };
-      }
-
-      log.info(`⚙️ Processing ${filesToProcess.length} files`);
-
-      // Initialize services
-      const xmlParser = new XMLParserService();
-      const mapper = new UniversalMapper(mappingConfig);
-      
-      // Get event configuration
-      const eventConfig: EventConfig = {
-        catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
-        catalogueType: activation.getVariable('catalogueType') || 'MASTER',
-        eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
-        eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
-      };
-
-      // Get event sending configuration
-      const eventConcurrency = Math.max(
-        1,
-        parseInt(activation.getVariable('eventConcurrency') || '1', 10)
-      );
-      // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
-      // concurrency: 1 = sequential, concurrency > 1 = parallel
-
-      log.info(`Event concurrency: ${eventConcurrency}`, {
-        mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
-        concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
-      });
-
-      // Initialize class-based services
-      const fileProcessor = new ProductFileProcessorService(
-        s3,
-        xmlParser,
-        mapper,
-        eventConfig.catalogueRef
-      );
-      // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
-      const eventSender = new EventSenderService(client, log);
-      const eventLogger = new EventLoggerService(s3);
-
-      // 
-      // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
-      // 
-
-      for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
-        const file = filesToProcess[fileIndex];
-        const fileStartTime = Date.now();
-
-        // Use object key from S3 listing
-        const objectKey = file.key || (file as any).name;
-        const baseName = objectKey.split('/').pop() || objectKey;
-
-        log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`);
-
-        try {
-          await tracker.updateJob(jobId, {
-            status: 'processing',
-            stage: 'downloading',
-            message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
-          });
-
-          // ═══════════════════════════════════════════════════════════
-          // STEP 4: Process File (Download + Parse + Map)
-          // ═══════════════════════════════════════════════════════════
-          log.info(`⚙️ [STEP 4/8] Processing file: ${baseName}`);
-
-          // Process file: download → parse → transform
-          const fileResult = await fileProcessor.downloadParseAndTransform(objectKey);
-
-          if (!fileResult.success || fileResult.products.length === 0) {
-            // Move failed file to errors and record result
-            const archivedName = timestampedName(baseName);
-            const errorKey = `${s3Config.errorPrefix}${archivedName}`;
-            await s3.moveObject(objectKey, errorKey);
-
-            fileResults.push({
-              fileName: baseName,
-              success: false,
-              recordsProcessed: fileResult.products.length,
-              eventsSent: 0,
-              eventsFailed: 0,
-              duration: Date.now() - fileStartTime,
-              error: fileResult.error || 'Processing failed'
-            });
-
-            continue;
-          }
-
-          // ═══════════════════════════════════════════════════════════
-          // STEP 5: Send Events
-          // ═══════════════════════════════════════════════════════════
-          log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
-
-          // ? Enhanced: Extract context for progress logging
-          const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
-          const eventType = eventConfig.eventType || 'UPSERT_PRODUCT';
-
-          // ? Enhanced: Start logging with context
-          log.info(`[EventSender] Sending events for file "${baseName}"`, {
-           totalProducts: fileResult.products.length,
-           eventType,
-           concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
-           sampleProductRefs: sampleProductRefs.join(', '),
-           eventMode: eventConfig.eventMode || 'async'
-          });
-
-          const eventResult = await eventSender.sendEvents(
-            fileResult.products,
-            eventConfig,
-            eventConcurrency
-          );
-
-          const eventsSent = eventResult.eventsSent;
-          const eventsFailed = eventResult.eventsFailed;
-
-          log.info(`✅ Events sent for ${baseName}`, {
-            successful: eventsSent,
-            failed: eventsFailed
-          });
-
-          // ? Enhanced: Completion logging with summary
-          log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
-           totalProducts: fileResult.products.length,
-           eventsSent,
-           eventsFailed,
-           successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
-           eventType
-          });
-
-          // ═══════════════════════════════════════════════════════════
-          // STEP 6: Archive File
-          // ═══════════════════════════════════════════════════════════
-          log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
-
-          await tracker.updateJob(jobId, {
-            status: 'processing',
-            stage: 'archiving',
-            message: `Archiving ${baseName}`
-          });
-
-          // Conditionally archive: errors → errorPrefix, else → processedPrefix (timestamped)
-          // ✅ S3: Uses moveObject for deduplication (no VersoriFileTracker needed)
-          const archivedName = timestampedName(baseName);
-          const targetPrefix = eventsFailed > 0 ? s3Config.errorPrefix : s3Config.processedPrefix;
-          const targetKey = `${targetPrefix}${archivedName}`;
-          await s3.moveObject(objectKey, targetKey);
-
-          log.info(`✅ File archived successfully: ${baseName}`, { targetKey });
-
-          // Optional: Write event log for audit/debugging
-          if (eventsFailed > 0) {
-            const logKey = `${s3Config.logsPrefix}${baseName.replace(/\.xml$/i, '')}-event-log.json`;
-            await eventLogger.writeEventLog(logKey, {
-              fileName: baseName,
-              totalRecords: fileResult.products.length,
-              eventsSent,
-              eventsFailed,
-              errors: eventResult.errors,
-              processedAt: new Date().toISOString()
-            });
-          }
-
-          // Store file result
-          fileResults.push({
-            fileName: baseName,
-            success: true,
-            recordsProcessed: fileResult.products.length,
-            eventsSent,
-            eventsFailed,
-            duration: Date.now() - fileStartTime
-          });
-
-        } catch (error: unknown) {
-          // ✅ Enhanced error logging: Extract all error details for visibility
-          const errorMessage = error instanceof Error ? error.message : String(error);
-          const errorDetails = {
-            message: errorMessage,
-            stack: error instanceof Error ? error.stack : undefined,
-            fileName: (error as any)?.fileName,
-            lineNumber: (error as any)?.lineNumber,
-            originalError: (error as any)?.context?.originalError?.message,
-            errorType: error instanceof Error ? error.name : 'Error',
-          };
-          log.error(`Error processing file ${baseName}:`, errorDetails);
-
-          // Best-effort: move to error archive on unexpected failure
-          try {
-            const archivedName = timestampedName(baseName);
-            const errorKey = `${s3Config.errorPrefix}${archivedName}`;
-            await s3.moveObject(objectKey, errorKey);
-          } catch (moveError: unknown) {
-            const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
-            log.error('Could not archive failed file', {
-              file: baseName,
-              message: moveErrorMessage,
-              stack: moveError instanceof Error ? moveError.stack : undefined,
-              errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
-            });
-          }
-
-          fileResults.push({
-            fileName: baseName,
-            success: false,
-            recordsProcessed: 0,
-            eventsSent: 0,
-            eventsFailed: 0,
-            duration: Date.now() - fileStartTime,
-            error: errorMessage
-          });
-        }
-      }
-
-      // 
-      // STEP 8: Complete Job & Calculate Totals
-      // 
-      log.info('🏁 [STEP 8/8] Completing job and calculating totals', { jobId });
-
-      const filesProcessed = fileResults.filter(r => r.success).length;
-      const filesFailed = fileResults.filter(r => !r.success).length;
-      const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
-      const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
-      const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
-
-
-      await tracker.markCompleted(jobId, {
-        filesProcessed,
-        filesFailed,
-        recordsProcessed: totalRecordsProcessed,
-        eventsSent: totalEventsSent,
-        eventsFailed: totalEventsFailed,
-        duration: Date.now() - startTime
-      });
-
-      log.info('✅ Ingestion completed successfully', {
-        filesProcessed,
-        filesFailed,
-        recordsProcessed: totalRecordsProcessed,
-        eventsSent: totalEventsSent,
-        eventsFailed: totalEventsFailed,
-        duration: Date.now() - startTime
-      });
-
-      return {
-        success: true,
-        jobId,
-        filesProcessed,
-        filesFailed,
-        recordsProcessed: totalRecordsProcessed,
-        eventsSent: totalEventsSent,
-        eventsFailed: totalEventsFailed,
-        fileResults
-      };
-
-    } finally {
-      // ❌š ï¸ CRITICAL: Always dispose S3 connection
-      if (s3) {
-        await s3.dispose();
-        log.info('S3 connection disposed');
-      }
-    }
-
-  } catch (error: unknown) {
-    const errorMessage = error instanceof Error ? error.message : String(error);
-    const errorStack = error instanceof Error ? error.stack : undefined;
-
-    // Generate contextual recommendations based on error type
-    const recommendations = [];
-    if (errorMessage.includes('S3') || errorMessage.includes('bucket')) {
-      recommendations.push('Verify S3 credentials (accessKeyId, secretAccessKey)');
-      recommendations.push('Check S3 bucket name and region configuration');
-      recommendations.push('Ensure bucket permissions allow ListObjects and GetObject');
-    }
-    if (errorMessage.includes('XML') || errorMessage.includes('parse')) {
-      recommendations.push('Validate XML file structure against expected schema');
-      recommendations.push('Check for special characters or encoding issues');
-      recommendations.push('Review XML parser configuration');
-    }
-    if (errorMessage.includes('event') || errorMessage.includes('API')) {
-      recommendations.push('Verify Fluent API credentials and retailerId');
-      recommendations.push('Check network connectivity to Fluent API');
-      recommendations.push('Review event payload structure');
-    }
-    if (errorMessage.includes('mapping')) {
-      recommendations.push('Review mapping configuration in config/*.json');
-      recommendations.push('Check for missing required fields');
-      recommendations.push('Validate resolver functions');
-    }
-
-    log.error('❌ Ingestion workflow failed', {
-      jobId,
-      error: errorMessage,
-      stack: errorStack,
-      recommendations: recommendations.length > 0 ? recommendations : ['Review logs for detailed error information'],
-    });
-
-    // Try to mark job as failed, but don't let tracking errors mask workflow error
-    try {
-      await tracker.markFailed(jobId, error);
-    } catch (trackingError: unknown) {
-      const trackingErrorMessage =
-        trackingError instanceof Error ? trackingError.message : String(trackingError);
-      log.warn('Failed to mark job as failed in tracker', {
-        jobId,
-        trackingError: trackingErrorMessage,
-      });
-    }
-
-    return {
-      success: false,
-      jobId,
-      filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
-      filesFailed: fileResults.filter(r => !r.success).length,
-      filesSkipped: fileResults.filter(r => r.skipped).length,
-      recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
-      eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
-      eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
-      fileResults,
-      error: errorMessage,
-    };
-  } finally {
-    // ⚠️ CRITICAL: Ensure S3 is disposed even if outer error occurs
-    // This outer finally ensures disposal if error happens after S3 creation
-    // but before inner try block (e.g., during connection validation)
-    if (s3) {
-      try {
-        await s3.dispose();
-        log.info('🔌 S3 connection disposed (outer finally)');
-      } catch (disposeError: unknown) {
-        const disposeErrorMessage =
-          disposeError instanceof Error ? disposeError.message : String(disposeError);
-        log.warn('⚠️ Error disposing S3 in outer finally', { error: disposeErrorMessage });
-      }
-    }
-  }
-  // ═══════════════════════════════════════════════════════════
-  // EXECUTION BOUNDARY: Product Ingestion End
-  // ═══════════════════════════════════════════════════════════
-}
-```
-
-## 8. Utility Functions (src/utils/)
-
-### S3 Path Helpers (src/utils/s3-path.utils.ts)
-
-```typescript
-/**
- * S3 Path Utilities
- *
- * Helper functions for S3 path operations.
- */
-
-/**
- * Generate timestamped filename for archival
- *
- * Adds ISO timestamp to filename before extension for unique archival.
- * Handles XML files specifically but can be adapted for other formats.
- *
- * @param name - Original filename
- * @returns Filename with timestamp appended
- *
- * @example
- * ```typescript
- * timestampedName('products.xml')
- * // Returns: 'products-2025-11-01T18-30-45-123Z.xml'
- * ```
- */
-export function timestampedName(name: string): string {
-  const ts = new Date().toISOString().replace(/[:.]/g, '-');
-  const base = name.replace(/\.xml$/i, '');
-  return `${base}-${ts}.xml`;
-}
-```
-
-### Job ID Generator (src/utils/job-id-generator.ts)
-
-```typescript
-/**
- * Job ID Generator
- *
- * Generates unique job IDs for tracking ingestion runs.
- * Format: {PREFIX}_{ENTITY}_{DATE}_{TIME}_{RANDOM}
- * Example: SCHEDULED_PROD_20250124_183045_abc123
- */
-
-/**
- * Generate unique job ID
- *
- * @param prefix - Job prefix (SCHEDULED, ADHOC, etc.)
- * @param entity - Entity type (PROD, INV, etc.)
- * @returns Unique job ID string
- */
-export function generateJobId(prefix: string, entity: string): string {
-  const now = new Date();
-  const date = now.toISOString().slice(0, 10).replace(/-/g, '');
-  const time = now.toISOString().slice(11, 19).replace(/:/g, '');
-  const random = Math.random().toString(36).slice(2, 8).toUpperCase();
-  return `${prefix}_${entity}_${date}_${time}_${random}`;
-}
-```
-
-## 📄 Mapping Configuration
-
-**File:** `config/products.import.xml.json`
-
-```json
-{
-  "name": "products.import.xml",
-  "version": "1.2.0",
-  "description": "XML → Product Event Mapping",
-  "fields": {
-    "ref": {
-      "source": "ref",
-      "required": true,
-      "resolver": "sdk.trim",
-      "comment": "Product reference from XML element"
-    },
-    "type": {
-      "source": "type",
-      "required": true,
-      "resolver": "sdk.uppercase",
-      "comment": "Product type (STANDARD, VARIANT)"
-    },
-    "status": {
-      "source": "status",
-      "required": true,
-      "resolver": "sdk.uppercase",
-      "comment": "Product status (ACTIVE, INACTIVE)"
-    },
-    "name": {
-      "source": "name",
-      "required": true,
-      "resolver": "sdk.trim",
-      "comment": "Product name/title"
-    },
-    "summary": {
-      "source": "summary",
-      "required": false,
-      "comment": "Product short description"
-    },
-    "gtin": {
-      "source": "gtin",
-      "required": false,
-      "resolver": "sdk.trim",
-      "comment": "Global Trade Item Number (barcode)"
-    },
-    "catalogue.ref": {
-      "source": "$context.catalogueRef",
-      "required": false,
-      "comment": "Catalogue reference (injected from context, NOT in XML)"
-    },
-    "metadata.source": {
-      "value": "S3_XML",
-      "required": false,
-      "comment": "Static value - identifies data source (NOT in XML)"
-    }
-  }
-}
-```
-
----
-
-## 9. Package Configuration
-
-### `package.json`
-
-```json
-{
-  "name": "s3-xml-product-event-ingestion",
-  "version": "1.2.0",
-  "description": "S3 XML Product Event Ingestion Connector",
-  "type": "module",
-  "main": "src/index.ts",
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "^1.0.0"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-
----
-
-**File:** `config/products.import.xml.json`
-
-```json
-{
-  "name": "products.import.xml",
-  "version": "1.1.0",
-  "description": "XML → Product Event Mapping",
-  "fields": {
-    "ref": { "source": "product.ref", "required": true, "resolver": "sdk.trim" },
-    "type": { "source": "product.type", "resolver": "sdk.uppercase", "defaultValue": "STANDARD" },
-    "status": { "source": "product.status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" },
-    "gtin": { "source": "product.gtin" },
-    "name": { "source": "product.name", "required": true, "resolver": "sdk.trim" },
-    "summary": { "source": "product.summary" },
-    "categoryRefs": { "source": "product.categoryRefs.ref", "isArray": true },
-    "price": {
-      "source": "product.price.item",
-      "isArray": true,
-      "fields": {
-        "type": { "source": "type" },
-        "currency": { "source": "currency" },
-        "value": { "source": "value" }
-      }
-    },
-    "taxType": {
-      "source": "product.taxType",
-      "fields": {
-        "country": { "source": "country" },
-        "group": { "source": "group" },
-        "tariff": { "source": "tariff" }
-      }
-    },
-    "attributes": { "defaultValue": [] }
-  }
-}
-```
-
-Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
-
----
-
-## Expected XML Format
-
-**Sample:** `products_20250124.xml`
-
-```xml
-<?xml version="1.0" encoding="UTF-8"?>
-<products>
-  <product>
-    <ref>G_PROD_WITH_NO_STANDARD</ref>
-    <type>VARIANT</type>
-    <status>ACTIVE</status>
-    <gtin>MH01-XS-Orange</gtin>
-    <name>Chaz Kangeroo Hoodie-XS-Orange main</name>
-    <summary>&lt;p&gt;test short description&lt;/p&gt;</summary>
-    <categoryRefs>
-      <ref>STANDARD_CATEGORY</ref>
-    </categoryRefs>
-    <price>
-      <item>
-        <type>DEFAULT</type>
-        <currency>USD</currency>
-        <value>52.000000</value>
-      </item>
-      <item>
-        <type>SPECIAL</type>
-        <currency>USD</currency>
-        <value>9.000000</value>
-      </item>
-    </price>
-    <taxType>
-      <country>AU</country>
-      <group>Tax Group</group>
-      <tariff>Tax Tariff</tariff>
-    </taxType>
-    <catalogue>
-      <ref>PC:MASTER:2</ref>
-      <type>MASTER</type>
-    </catalogue>
-  </product>
-  <product>
-    <ref>G_PROD_002</ref>
-    <type>VARIANT</type>
-    <status>ACTIVE</status>
-    <gtin>MH02-M-Blue</gtin>
-    <name>Kangeroo Hoodie-M-Blue</name>
-    <summary>&lt;p&gt;Blue variant medium size&lt;/p&gt;</summary>
-    <categoryRefs>
-      <ref>STANDARD_CATEGORY</ref>
-      <ref>SEASONAL</ref>
-    </categoryRefs>
-    <price>
-      <item>
-        <type>DEFAULT</type>
-        <currency>USD</currency>
-        <value>45.000000</value>
-      </item>
-      <item>
-        <type>SPECIAL</type>
-        <currency>USD</currency>
-        <value>35.000000</value>
-      </item>
-    </price>
-    <taxType>
-      <country>AU</country>
-      <group>Tax Group</group>
-      <tariff>Tax Tariff</tariff>
-    </taxType>
-    <catalogue>
-      <ref>PC:MASTER:2</ref>
-      <type>MASTER</type>
-    </catalogue>
-  </product>
-</products>
-```
-
-**XML Field Mapping:**
-
-- Nested structure for objects (e.g., `<taxType><country>AU</country></taxType>`)
-- Arrays with repeated elements (e.g., `<categoryRefs><ref>CAT1</ref><ref>CAT2</ref></categoryRefs>`)
-- HTML content in CDATA or escaped (e.g., `&lt;p&gt;description&lt;/p&gt;`)
-
----
-
-## 9. Package Configuration
-
-### `package.json`
-
-```json
-{
-  "name": "s3-xml-product-event",
-  "version": "1.1.0",
-  "type": "module",
-  "main": "src/index.ts",
-  "scripts": {
-    "dev": "versori dev",
-    "build": "versori build",
-    "deploy": "versori deploy"
-  },
-  "dependencies": {
-    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
-    "@versori/run": "latest"
-  },
-  "devDependencies": {
-    "@types/node": "^20.0.0",
-    "typescript": "^5.0.0"
-  }
-}
-```
-
-### `tsconfig.json`
-
-```json
-{
-  "compilerOptions": {
-    "module": "ES2022",
-    "target": "ES2024",
-    "moduleResolution": "node"
-  }
-}
-```
-
----
-
-## 10. Deployment Instructions
-
-### Deploy to Versori
-
-```bash
-# 1. Install dependencies
-npm install
-
-# 2. Test locally (if using Versori CLI)
-npm run dev
-
-# 3. Deploy to Versori platform
-npm run deploy
-```
-
-### Configure Activation Variables
-
-In Versori platform settings, configure all variables listed in the Activation Variables section above.
-
----
-
-## 11. Testing
-
-### Test Scheduled Ingestion
-
-Upload a test XML file to S3 incoming prefix and wait for the scheduled run.
-
-**Check logs:**
-
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and S3
-[STEP 3/8] Discovering files on S3
-[FILE 1/1] Processing file: products_20250124.xml
-[STEP 4/8] Downloading and parsing: products_20250124.xml
-[STEP 5/8] Transforming 5 products from products_20250124.xml
-[STEP 6/8] Sending 5 events to Fluent Commerce
-[STEP 7/8] Archiving file: products_20250124.xml
-[STEP 8/8] Completing job and calculating totals
-```
-
-### Test Ad hoc Ingestion
-
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
-  -H "X-API-Key: your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{}'
-
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
-  -H "X-API-Key: your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "filePattern": "urgent_*.xml",  }'
-```
-
-### Test Job Status Query
-
-```bash
-curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
-  -H "X-API-Key: your-secret-key" \
-  -H "Content-Type: application/json" \
-  -d '{
-    "jobId": "ADHOC_PROD_20251024_183045_abc123"
-  }'
-```
-
----
-
-## Monitoring
-
-### Success Response
-
-```json
-{
-  "success": true,
-  "filesProcessed": 1,
-  "filesSkipped": 0,
-  "filesFailed": 0,
-  "totalRecords": 50,
-  "eventsSent": 50,
-  "eventsFailed": 0,
-  "results": [
-    {
-      "file": "products_2025-01-22.xml",
-      "success": true,
-      "recordsProcessed": 50,
-      "eventsSent": 50,
-      "eventsFailed": 0
-    }
-  ],
-  "duration": 12345
-}
-```
-
-### Partial Success Response
-
-```json
-{
-  "success": true,
-  "filesProcessed": 1,
-  "filesSkipped": 0,
-  "filesFailed": 0,
-  "totalRecords": 50,
-  "eventsSent": 45,
-  "eventsFailed": 5,
-  "results": [
-    {
-      "file": "products_2025-01-22.xml",
-      "success": true,
-      "recordsProcessed": 50,
-      "eventsSent": 45,
-      "eventsFailed": 5,
-      "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
-    }
-  ],
-  "duration": 12345
-}
-```
-
-### Error Response
-
-```json
-{
-  "success": false,
-  "filesProcessed": 0,
-  "filesFailed": 1,
-  "totalRecords": 0,
-  "eventsSent": 0,
-  "eventsFailed": 0,
-  "results": [
-    {
-      "file": "products_2025-01-22.xml",
-      "success": false,
-      "error": "XML parse error: Invalid structure"
-    }
-  ],
-  "duration": 876
-}
-```
-
-### Monitoring Metrics
-
-Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
-
-- **Files Processed** - Total files successfully processed
-- **Events Sent** - Total events sent to Fluent Commerce
-- **Events Failed** - Events that failed (check rejection reports)
-- **Processing Duration** - Time taken for complete workflow
-- **Rate Limiting** - Watch for 429 errors indicating throttling
-
-Use the status webhook for dashboards and automated monitoring.
-
----
-
-- **No files found:** Check `s3IncomingPrefix` and `filePattern` (object keys only)
-- **XML parse failed:** Move to errors/; validate XML structure and encoding
-- **XML array normalization:** Check if single object vs array - ALWAYS normalize
-- **High event failures:** Inspect logs; consider Batch API for very high volumes
-- **429 throttling:** Add small backoff/delay or use Batch API
-- **S3 connection errors:** Verify credentials, bucket, region, and permissions
-
----
-
-## 13. Key Takeaways
-
-- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
-- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
-- ✅ **JobTracker** - Track job lifecycle with KV persistence
-- ❌ **NO VersoriFileTracker** - S3 uses moveObject for deduplication (SFTP-specific only)
-- ✅ **S3 deduplication** - Physical file movement to processed/errors prefixes
-- ✅ **S3 dispose()** - Always cleanup in finally block
-- ✅ **S3 method names** - listObjects, downloadObject, moveObject (not \*File)
-- ✅ **XML array normalization** - CRITICAL: single object vs array handling
-- ✅ **Per-record error handling** - Continue on individual failures
-- ✅ **Externalized mapping** - Use JSON config file for field mappings
-- ✅ **File-level error handling** - Don't stop on single file failure
-- ✅ **S3 vs SFTP** - Different deduplication strategies (moveObject vs KV state)
-- ✅ **Processing modes** - per-file (default), chunked, batch
-- ✅ **Modular services** - orchestrator, processor, dispatcher, archive
-
----
-
-[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)
+---
+template_id: tpl-ingest-s3-xml-to-product-event
+canonical_filename: template-ingestion-s3-xml-product-event.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: ingestion
+source: s3-xml
+destination: fluent-event-api
+entity: product
+format: xml
+logging: versori
+status: stable
+features:
+  - batched-events
+  - attribute-transformation
+  - memory-management
+  - json-serialization-handling
+  - enhanced-logging
+---
+
+# Template: Ingestion - S3 XML to Product Event
+
+**Template Version:** 2.0.0
+**SDK Version:** @fluentcommerce/fc-connect-sdk@^0.1.39
+**Last Updated:** 2025-01-24
+**Deployment Target:** Versori Platform
+
+**🆕 Version 2.0.0 Enhancements:**
+- ✅ **Batched Events Support** - UPSERT_PRODUCTS with configurable batch sizes (10x faster for 100+ products)
+- ✅ **Attribute Transformation** - Automatic conversion of flat attributes to Fluent Commerce array format
+- ✅ **Memory Management** - Configurable batch processing with explicit cleanup (prevents OOM on large XML files)
+- ✅ **JSON Serialization Handling** - Prevents 503 errors from non-serializable webhook responses
+- ✅ **Enhanced Event Logging** - Status-prefixed logs (SUCCESS_/ERROR_) with comprehensive metrics and progress tracking
+
+---
+
+## 📚 STEP 1: Load These Docs Into Your AI (Human Checklist)
+
+1. REQUIRED (load all)
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+   - [ ] fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+   - [ ] fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
+   - [ ] fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+Copy-paste list (open these):
+fc-connect-sdk/docs/02-CORE-GUIDES/api-reference/
+fc-connect-sdk/docs/02-CORE-GUIDES/mapping/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/data-sources/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/parsers/
+fc-connect-sdk/docs/03-PATTERN-GUIDES/ingestion/
+fc-connect-sdk/docs/04-REFERENCE/platforms/versori/
+
+---
+
+## 📋 Implementation Prompt
+
+```
+I need a Versori scheduled ingestion that:
+
+1) Lists XML files on S3 (deduplication via moveObject - NO VersoriFileTracker)
+2) Downloads and parses XML with array normalization (CRITICAL: single object vs array)
+3) Transforms records with UniversalMapper per mapping JSON
+4) Sends UPSERT_PRODUCT events (async) to Fluent Commerce with per-record error handling
+5) Moves files to processed/ or errors/ prefixes for natural deduplication
+6) Tracks progress with JobTracker and exposes a job-status webhook
+7) Uses native Versori log from context
+
+Use the loaded docs to fill in SDK specifics and best practices.
+Keep the structure identical to the template; only adapt where needed.
+```
+
+---
+
+## 📋 Template Overview
+
+This connector runs on the Versori platform. Most operational settings (Fluent account/connection, S3 credentials, schedule, file patterns) are configured via activation variables. Data shape and logic (mapping JSON, XML structure, parsing rules, per-record handling) are adjusted in code as needed. It reads product data from S3 XML, transforms it, and sends events to the Fluent Commerce Event API.
+
+### What This Template Does
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    INGESTION WORKFLOW                            │
+└─────────────────────────────────────────────────────────────────┘
+
+1. TRIGGER
+   ├─ Scheduled (Cron): Runs automatically every hour
+   ├─ Ad hoc (Webhook): Manual trigger for immediate processing
+   └─ Status Query (Webhook): Check job progress
+
+2. DISCOVER FILES (S3DataSource)
+   ├─ List objects from S3 bucket
+   ├─ Filter by prefix and pattern (products/*.xml)
+   ├─ NO VersoriFileTracker (S3 uses moveObject for deduplication)
+   └─ Sort by last modified
+
+3. DOWNLOAD & PARSE (XMLParserService)
+   ├─ Download object from S3
+   ├─ Parse XML to JavaScript object
+   ├─ ⚠️ CRITICAL: Normalize single object vs array
+   └─ Validate XML structure
+
+4. TRANSFORM (UniversalMapper)
+   ├─ Map XML fields to Fluent schema
+   ├─ Apply SDK resolvers (trim, uppercase, etc.)
+   ├─ Handle nested objects (price, taxType)
+   ├─ Handle arrays (categoryRefs)
+   └─ Collect transformation errors
+
+5. SEND EVENTS (Event API)
+   ├─ Loop through transformed products
+   ├─ Send UPSERT_PRODUCT event (async)
+   ├─ Track success/failure count
+   └─ Continue on individual failures
+
+6. ARCHIVE (S3DataSource - Natural Deduplication)
+   ├─ Move object to processed/ prefix (success)
+   ├─ Or move to errors/ prefix (failures)
+   ├─ Generate timestamped archive name
+   └─ Files in incoming/ are removed naturally
+
+7. TRACK JOB (JobTracker)
+   ├─ Update job status at each step
+   ├─ Store final result in KV
+   ├─ Enable status queries via webhook
+   └─ Handle errors gracefully
+```
+
+### Key Features
+
+- Job tracking with status queries
+- Execution modes: scheduled, ad hoc, status query
+- Uses S3DataSource, XMLParserService, UniversalMapper, JobTracker
+- Error handling, retry logic, and S3 cleanup
+- **S3 Deduplication:** moveObject to processed/errors prefixes (NO VersoriFileTracker)
+- **Key Difference from SFTP:** Physical file movement vs KV state tracking
+- **XML Array Normalization:** CRITICAL handling of single object vs array
+- Event API: Per-record failures don't block other records
+- S3 dispose() in finally block
+
+Note: JobTracker persists stage/status to Versori KV for visibility, job-status webhooks, and auditing. Recommended for production multi-step flows; can be skipped for trivial single-step utilities.
+
+### 📦 Package Information
+
+**SDK:** [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+
+Use the latest SDK version:
+
+```bash
+npm install @fluentcommerce/fc-connect-sdk@^0.1.39
+```
+
+---
+
+**Templates are designed for direct deployment; customize via activation variables.**
+
+---
+
+## ⚙️ Activation Variables
+
+**Configuration is driven by activation variables - modify these instead of code:**
+
+```json
+{
+  "fluentRetailerId": "1",
+  "eventConcurrency": 1,
+  "s3Bucket": "my-product-bucket",
+  "s3Region": "us-east-1",
+  "s3AccessKeyId": "AKIA...",
+  "s3SecretAccessKey": "********",
+  "s3IncomingPrefix": "products/incoming/",
+  "s3ProcessedPrefix": "products/processed/",
+  "s3ErrorPrefix": "products/errors/",
+  "s3LogsPrefix": "products/logs/",
+  "filePattern": "products_*.xml",
+  "catalogueRef": "PC:MASTER:2",
+  "catalogueType": "MASTER",
+  "eventName": "UPSERT_PRODUCT",
+  "eventMode": "async",
+  "maxFilesPerRun": 10,
+  "processingMode": "per-file",
+  "fileChunkSize": 5,
+  "validateConnection": true,
+  "enableFileTracking": true,
+  "trackDuration": true,
+  "useBatchedEvents": false,
+  "maxProductsUnderBatchedEvent": 100,
+  "memoryBatchSize": 250
+}
+```
+
+Note: Webhook security is handled by Versori's native connection authentication. The `connection` parameter in webhook definitions ensures only authenticated requests are processed.
+
+### Variable Explanations
+
+| Variable            | Purpose                      | Default               | Customization Hints                 |
+| ------------------- | ---------------------------- | --------------------- | ----------------------------------- |
+| `fluentRetailerId`  | Retailer ID for Event API    | -                     | Required - Fluent retailer ID      |
+| `eventConcurrency`  | Event sending concurrency    | `1`                   | `1` = sequential, `>1` = parallel (3-10 recommended) |
+| `s3Bucket`          | S3 bucket name               | -                     | Required - your S3 bucket           |
+| `s3Region`          | S3 region                    | `us-east-1`           | AWS region (e.g., us-west-2)        |
+| `s3AccessKeyId`     | AWS access key ID            | -                     | Required - AWS credentials          |
+| `s3SecretAccessKey` | AWS secret access key        | -                     | Required - AWS credentials          |
+| `s3IncomingPrefix`  | Incoming files prefix        | `products/incoming/`  | Where new files arrive              |
+| `s3ProcessedPrefix` | Processed files archive      | `products/processed/` | Where files move after success      |
+| `s3ErrorPrefix`     | Failed files directory       | `products/errors/`    | Where files move after errors       |
+| `s3LogsPrefix`      | Event processing logs        | `products/logs/`      | Where event logs are written        |
+| `filePattern`       | File name filter             | `products_*.xml`      | Glob pattern for matching files     |
+| `catalogueRef`      | Product catalogue reference  | `PC:MASTER:2`         | Target catalogue in Fluent          |
+| `catalogueType`     | Catalogue type               | `MASTER`              | Usually MASTER or STANDARD          |
+| `eventName`         | Event to send                | `UPSERT_PRODUCT`      | Event name from Rubix               |
+| `eventMode`         | Event processing mode        | `async`               | async (recommended) or sync         |
+| `maxFilesPerRun`    | Max files per execution      | `10`                  | Prevent timeout on large batches    |
+| `processingMode`    | Processing mode              | `per-file`            | per-file, chunked, or batch         |
+| `fileChunkSize`     | Files per chunk (chunked)    | `5`                   | Used when processingMode=chunked    |
+| `validateConnection` | Validate S3 on initialization | `true`               | Fail-fast if S3 credentials invalid |
+| `enableFileTracking` | Enable file deduplication    | `true`               | Track processed files in KV store   |
+| `trackDuration`     | Track processing duration     | `true`               | Log timing metrics for performance  |
+| `useBatchedEvents` | Use batched UPSERT_PRODUCTS | `false`        | `true` = batched events (10x faster for 100+ products), `false` = individual events |
+| `maxProductsUnderBatchedEvent` | Products per batch | `100`          | Only used when `useBatchedEvents=true` (default: 100) |
+| `memoryBatchSize`  | Products per mapping batch  | `250`          | Processes products in batches during mapping (prevents OOM on large files) |
+
+---
+
+## 🔒 SDK Automatic Behaviors (v0.1.40+)
+
+**The SDK automatically validates and retries for improved reliability:**
+
+### retailerId Validation
+- **SDK validates** `retailerId` before calling `sendEvent()`
+- **Checks:** `event.retailerId || client.retailerId`
+- **If missing:** Throws `"retailerId is required for Event API..."`
+- **Configuration:** Set via `fluentRetailerId` activation variable (recommended)
+
+### 401 Auth Retry
+- **Automatic retry** for platform auth failures (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s)
+- **Applies to:** All `sendEvent()` calls (async and sync modes)
+- **Log:** `"[fc-connect-sdk:auth] Platform auth failure (401), retrying..."`
+
+### 5xx Server Retry
+- **Automatic retry** for transient server errors (3 attempts)
+- **Delay:** Exponential backoff (1s → 2s → 4s, capped at 10s)
+- **Protects:** Against Fluent API transient failures
+
+### No Code Changes Required
+- All templates remain compatible
+- Retry logic is automatic and transparent
+- Better error messages guide configuration
+
+**See:** [Event API Guide](./event-api-guide.md) for complete details
+
+---
+
+### 📁 S3 Deduplication Pattern (NO VersoriFileTracker)
+
+**CRITICAL: S3 uses physical file movement, NOT VersoriFileTracker**
+
+VersoriFileTracker is **SFTP-specific** and should **NOT** be used with S3. S3 achieves deduplication through physical file movement.
+
+#### How S3 Deduplication Works
+
+**3-Step Process:**
+
+1. **Discovery:** List objects from `s3IncomingPrefix` matching `filePattern`
+2. **Process:** Download, parse, transform, and send events
+3. **Archive:** Move object to `s3ProcessedPrefix` or `s3ErrorPrefix`
+4. **Natural Deduplication:** Once moved, files are no longer in incoming/
+
+#### File Lifecycle Example
+
+```
+INITIAL STATE:
+s3://bucket/products/incoming/products_20250124_001.xml
+
+↓ (workflow discovers file)
+
+PROCESSING:
+- List objects from "products/incoming/"
+- Download: products_20250124_001.xml
+- Parse XML → Normalize array → Map to events → Send to Event API
+
+↓ (success)
+
+ARCHIVE (moveObject):
+s3://bucket/products/processed/products_20250124_001-20250124T183045.xml
+
+↓ (result)
+
+INCOMING FOLDER NOW EMPTY:
+s3://bucket/products/incoming/ (file is gone - no duplicates possible)
+```
+
+#### Key Differences: S3 vs SFTP
+
+| Aspect                   | S3 (This Template)            | SFTP (Other Templates)           |
+| ------------------------ | ----------------------------- | -------------------------------- |
+| **Deduplication Method** | Physical file movement        | VersoriFileTracker (KV state)    |
+| **State Storage**        | None needed                   | KV store tracks processed files  |
+| **File Tracking**        | ❌ NO VersoriFileTracker      | ✅ VersoriFileTracker required   |
+| **Reprocessing**         | Move files back to incoming/  | Clear KV state or forceReprocess |
+| **Complexity**           | Simpler (no state management) | More complex (state management)  |
+
+#### Summary
+
+✅ **DO:** Use S3 moveObject for deduplication
+✅ **DO:** Move files to processed/ or errors/ prefixes
+❌ **DON'T:** Import VersoriFileTracker in S3 templates
+❌ **DON'T:** Use KV state tracking for S3 files
+✅ **DO:** Use VersoriFileTracker for SFTP templates (different use case)
+
+---
+
+## When to Use Event API vs Batch API
+
+### ✅ Use Event API (`sendEvent`) For:
+
+| Entity Type         | Use Case                               | Why Event API                                |
+| ------------------- | -------------------------------------- | -------------------------------------------- |
+| **Products**        | Product catalog sync, variant updates  | Triggers workflows, validates business rules |
+| **Locations**       | Store/warehouse setup                  | Requires workflow orchestration              |
+| **Customers**       | Customer registration, profile updates | Needs workflow for downstream systems        |
+| **Orders**          | Single order creation                  | Event-driven fulfillment workflows           |
+| **Custom Entities** | Any entity needing workflow triggers   | Full Rubix workflow support                  |
+
+### ❌ Use Batch API Instead For:
+
+| Entity Type        | Use Case                           | Why Batch API                                   |
+| ------------------ | ---------------------------------- | ----------------------------------------------- |
+| **Inventory ONLY** | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
+
+### 🔄 Use GraphQL Mutations For:
+
+| Scenario              | Why GraphQL                            |
+| --------------------- | -------------------------------------- |
+| **Single operations** | Create one order, update one product   |
+| **Complex queries**   | Fetch data with relationships          |
+| **Testing/debugging** | Direct API control, immediate feedback |
+
+---
+
+## XML File Format
+
+### Sample: products.xml
+
+Based on your `UPSERT_PRODUCT` event payload:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<products>
+  <product>
+    <ref>G_PROD_WITH_NO_STANDARD</ref>
+    <type>VARIANT</type>
+    <status>ACTIVE</status>
+    <gtin>MH01-XS-Orange</gtin>
+    <name>Chaz Kangeroo Hoodie-XS-Orange main</name>
+    <summary>&lt;p&gt;test short description&lt;/p&gt;</summary>
+    <categoryRefs>
+      <ref>STANDARD_CATEGORY</ref>
+    </categoryRefs>
+    <price>
+      <item>
+        <type>DEFAULT</type>
+        <currency>USD</currency>
+        <value>52.000000</value>
+      </item>
+      <item>
+        <type>SPECIAL</type>
+        <currency>USD</currency>
+        <value>9.000000</value>
+      </item>
+    </price>
+    <taxType>
+      <country>AU</country>
+      <group>Tax Group</group>
+      <tariff>Tax Tariff</tariff>
+    </taxType>
+    <catalogue>
+      <ref>PC:MASTER:2</ref>
+      <type>MASTER</type>
+    </catalogue>
+  </product>
+  <product>
+    <ref>G_PROD_002</ref>
+    <type>VARIANT</type>
+    <status>ACTIVE</status>
+    <gtin>MH02-M-Blue</gtin>
+    <name>Kangeroo Hoodie-M-Blue</name>
+    <summary>&lt;p&gt;Blue variant&lt;/p&gt;</summary>
+    <categoryRefs>
+      <ref>STANDARD_CATEGORY</ref>
+      <ref>SEASONAL</ref>
+    </categoryRefs>
+    <price>
+      <item>
+        <type>DEFAULT</type>
+        <currency>USD</currency>
+        <value>45.000000</value>
+      </item>
+      <item>
+        <type>SPECIAL</type>
+        <currency>USD</currency>
+        <value>35.000000</value>
+      </item>
+    </price>
+    <taxType>
+      <country>AU</country>
+      <group>Tax Group</group>
+      <tariff>Tax Tariff</tariff>
+    </taxType>
+    <catalogue>
+      <ref>PC:MASTER:2</ref>
+      <type>MASTER</type>
+    </catalogue>
+  </product>
+</products>
+```
+
+**XML Field Mapping:**
+
+- Nested structure for objects (e.g., `<taxType><country>AU</country></taxType>`)
+- Arrays with repeated elements (e.g., `<categoryRefs><ref>CAT1</ref><ref>CAT2</ref></categoryRefs>`)
+- HTML content in CDATA or escaped (e.g., `&lt;p&gt;description&lt;/p&gt;`)
+
+---
+
+## Event Sending Configuration
+
+**Simple Configuration:** One variable controls everything
+
+```json
+{
+  "eventConcurrency": 1 // 1 = sequential, 3-10 = parallel
+}
+```
+
+**How it works:**
+
+- `eventConcurrency: 1` → Sequential (sends events one at a time, safe default)
+- `eventConcurrency: 3` → Parallel (sends 3 events concurrently)
+- `eventConcurrency: 5` → Parallel (sends 5 events concurrently)
+- `eventConcurrency: 10` → Parallel (sends 10 events concurrently)
+
+**Configuration Guidelines:**
+
+- **Conservative:** `1` → Sequential (safe, predictable, ~1 req/sec)
+- **Balanced:** `3-5` → Parallel (most common, ~3-5 req/sec)
+- **Aggressive:** `10` → Parallel (high-volume, ~10 req/sec)
+- **Note:** Fluent API supports concurrent requests - adjust based on your needs
+
+**Why Single Variable?**
+
+- **Simpler:** One variable instead of two (`mode` + `concurrency`)
+- **Clearer:** `concurrency: 1` = sequential, `concurrency > 1` = parallel
+- **Less config:** Fewer activation variables to manage
+- **Flexible:** Easy to tune performance (just change the number)
+
+---
+
+## 🔧 Production Reference Implementation
+
+### Processing Modes
+
+| Mode       | Default | What happens                                                                                      | When to use                                              |
+| ---------- | ------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
+| `per-file` | ✅      | Process one file end-to-end (discover → parse → Event API → archive) before moving on             | Recommended for production S3 feeds and very large files |
+| `chunked`  | ➖      | Processes `fileChunkSize` files at a time, completing each chunk before the next                  | High-volume feeds where per-file would take too long     |
+| `batch`    | ➖      | Loads every matched file into memory, produces a single Event API job, and archives when complete | Tiny datasets or smoke tests only                        |
+
+Set the mode via activation variables:
+
+```json
+{
+  "processingMode": "per-file",
+  "fileChunkSize": "5",
+  "maxFilesPerRun": "25"
+}
+```
+
+### Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+product-event-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-product-sync.ts   # Scheduled: Daily product sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-product-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── product-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── product.types.ts            # Shared type definitions
+```
+
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+
+---
+
+## Workflow Files
+
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+
+All time-based triggers that run automatically on cron schedules.
+
+#### `src/workflows/scheduled/daily-product-sync.ts`
+
+**Purpose**: Automatic Daily product sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Scheduled Workflow: Daily Product Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: product-sync.service.ts
+ */
+export const dailyProductSync = schedule(
+  'product-sync-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-product-sync', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv } = ctx;
+    const jobId = `product-sync-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    log.info('🚀 Starting scheduled product sync', { jobId });
+
+    await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' });
+    await tracker.updateJob(jobId, { status: 'processing' });
+
+    try {
+      // Reuse shared orchestration logic
+      log.info('⚙️ Executing product ingestion workflow', { jobId });
+      const result = await executeProductIngestion(ctx, { jobId, triggeredBy: 'schedule' }, tracker);
+      await tracker.markCompleted(jobId, result);
+      log.info('✅ Scheduled sync completed successfully', { jobId, ...result });
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      log.error('❌ Scheduled sync failed', { jobId, error: e?.message });
+      await tracker.markFailed(jobId, e);
+      return { success: false, jobId, error: e?.message };
+    }
+  })
+);
+```
+
+---
+
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+
+All HTTP-based triggers that create webhook endpoints.
+
+#### `src/workflows/webhook/adhoc-product-sync.ts`
+
+**Purpose**: Manual product sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { executeProductIngestion } from '../../services/product-sync.service';
+
+/**
+ * Webhook: Manual Product Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-adhoc
+ * Request body (optional): { filePattern: "urgent_*.xml", maxFiles: 5 }
+ *
+ * Pattern: webhook().then(http()) - needs Fluent API access
+ * Uses shared service: product-sync.service.ts
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const adhocProductSync = webhook('product-sync-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'product-sync-adhoc', // Versori validates API key
+}).then(
+  http('run-product-sync-adhoc', { connection: 'fluent_commerce' }, async ctx => {
+    const { log, openKv, data } = ctx;
+    const jobId = `product-sync-adhoc-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    const filePattern = data?.filePattern as string;
+    const maxFiles = data?.maxFiles as number;
+
+    log.info('🚀 [WEBHOOK] Adhoc product sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+    });
+
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'manual',
+      stage: 'initialization',
+      status: 'queued',
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    executeProductIngestion(ctx, { jobId, triggeredBy: 'manual' }, tracker)
+      .then((result) => {
+        log.info('✅ [BACKGROUND] Product sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          filesFailed: result.filesFailed,
+          recordsProcessed: result.recordsProcessed,
+        });
+        return tracker.markCompleted(jobId, result);
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+
+        log.error('❌ [BACKGROUND] Product sync failed', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+        });
+
+        return tracker.markFailed(jobId, errorMessage);
+      });
+
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Product sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/product-sync-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+
+#### `src/workflows/webhook/job-status-check.ts`
+
+**Purpose**: Query job status
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/product-sync-job-status`
+**Request body**: `{ "jobId": "product-sync-1234567890" }`
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/product-sync-job-status
+ * Request body: { "jobId": "product-sync-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ *
+ * SECURITY: Authentication handled via connection parameter
+ * No manual API key validation needed - Versori manages this via connection auth
+ */
+export const productSyncJobStatus = webhook('product-sync-job-status', {
+  response: { mode: 'sync' },
+  connection: 'product-sync-job-status',
+}).then(
+  fn('status', async ctx => {
+    // ═══════════════════════════════════════════════════════════
+    // EXECUTION BOUNDARY: Job Status Query Start
+    // ═══════════════════════════════════════════════════════════
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+
+    log.info('🔍 Job status query received', { jobId });
+
+    if (!jobId) {
+      log.warn('⚠️ Missing jobId in request');
+      return { success: false, error: 'jobId required' };
+    }
+
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+
+    if (status) {
+      log.info('✅ Job status found', { jobId, status: status.status });
+      return { success: true, jobId, ...status };
+    } else {
+      log.warn('⚠️ Job not found', { jobId });
+      return { success: false, error: 'Job not found', jobId };
+    }
+    // ═══════════════════════════════════════════════════════════
+    // EXECUTION BOUNDARY: Job Status Query End
+    // ═══════════════════════════════════════════════════════════
+  })
+);
+```
+
+---
+
+### 3. Entry Point (`index.ts`)
+
+**Purpose**: Register all workflows with Versori platform using MemoryInterpreter pattern
+
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * PATTERN: MemoryInterpreter for workflow registration
+ * - Workflow definitions stored in memory
+ * - Versori runtime discovers and executes via exports
+ * - No file I/O during workflow registration
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ */
+
+// Import scheduled workflows
+import { dailyProductSync } from './src/workflows/scheduled/daily-product-sync';
+
+// Import webhook workflows
+import { adhocProductSync } from './src/workflows/webhook/adhoc-product-sync';
+import { productSyncJobStatus } from './src/workflows/webhook/job-status-check';
+
+// MemoryInterpreter Pattern: Export workflows for Versori runtime discovery
+// Workflows are held in memory and executed when triggered by platform
+export {
+  // Scheduled (time-based triggers)
+  dailyProductSync,
+
+  // Webhooks (HTTP-based triggers)
+  adhocProductSync,
+  productSyncJobStatus,
+};
+```
+
+**What Gets Exposed:**
+- ✅ `adhocProductSync` → `https://{workspace}.versori.run/product-sync-adhoc`
+- ✅ `productSyncJobStatus` → `https://{workspace}.versori.run/product-sync-job-status`
+- ❌ `dailyProductSync` → NOT exposed (runs automatically on cron)
+
+---
+
+### Adding New Workflows
+
+**To add a scheduled workflow:**
+1. Create file in `src/workflows/scheduled/` with descriptive name (e.g., `hourly-delta-sync.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**To add a webhook workflow:**
+1. Create file in `src/workflows/webhook/` with descriptive name (e.g., `error-notification.ts`)
+2. Export the workflow from the file
+3. Import and re-export in `index.ts`
+
+**Example - Adding hourly delta sync:**
+
+```typescript
+// src/workflows/scheduled/hourly-delta-sync.ts
+export const hourlyDeltaSync = schedule(
+  'product-delta-hourly',
+  '0 * * * *' // Every hour
+).then(
+  http('run-delta-sync', { connection: 'fluent_commerce' }, async ctx => {
+    // Delta sync logic (skip BPP)
+    const result = await runIngestion(ctx, jobId, tracker, { bppEnabled: false });
+    return result;
+  })
+);
+
+// index.ts (add to imports and exports)
+import { hourlyDeltaSync } from './src/workflows/scheduled/hourly-delta-sync';
+export { daily_product_sync, hourlyDeltaSync, ... };
+```
+
+---
+## 3. Type Definitions (src/types/product-ingestion.types.ts)
+
+```typescript
+/**
+ * Type Definitions for Product Ingestion
+ *
+ * Centralized type definitions for product ingestion workflow
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Product interface - represents transformed product data
+ */
+export interface Product {
+  ref: string;
+  type?: string;
+  status?: string;
+  name: string;
+  summary?: string;
+  gtin?: string;
+  catalogue?: { ref?: string };
+  attributes?: Record<string, unknown>;
+}
+
+/**
+ * Event configuration
+ */
+export interface EventConfig {
+  eventName: string;
+  catalogueRef: string;
+  catalogueType: string;
+  eventMode: 'async' | 'sync';
+}
+
+/**
+ * Event result - tracks success/failure counts
+ */
+export interface EventResult {
+  eventsSent: number;
+  eventsFailed: number;
+  errors: Array<{ productRef: string; error: string }>;
+}
+
+/**
+ * Process file result
+ */
+export interface ProcessFileResult {
+  success: boolean;
+  products: Product[];
+  error?: string;
+}
+
+/**
+ * Versori Context Interface
+ * Represents the Versori runtime context passed to workflow functions
+ */
+export interface VersoriContext {
+  log: {
+    info: (message: string, data?: Record<string, unknown>) => void;
+    warn: (message: string, data?: Record<string, unknown>) => void;
+    error: (message: string, data?: Record<string, unknown>) => void;
+    debug?: (message: string, data?: Record<string, unknown>) => void;
+  };
+  openKv: (namespace: string) => {
+    get: (key: string) => Promise<unknown>;
+    set: (key: string, value: unknown) => Promise<void>;
+    delete: (key: string) => Promise<void>;
+  };
+  activation: {
+    getVariable: (name: string) => string | undefined;
+    connections?: Record<string, unknown>;
+  };
+  connections?: Record<string, unknown>;
+  data?: unknown;
+  fetch?: typeof fetch;
+}
+
+/**
+ * Parameters for ingestion workflow
+ */
+export interface ProductIngestionParams {
+  jobId: string;
+  triggeredBy: 'schedule' | 'webhook';
+  filePattern?: string;
+  maxFiles?: number;
+  forceReprocess?: boolean;
+  catalogueRef?: string;
+  priority?: 'normal' | 'high';
+}
+
+/**
+ * Result from ingestion workflow
+ */
+export interface ProductIngestionResult {
+  success: boolean;
+  jobId: string;
+  filesProcessed: number;
+  filesFailed: number;
+  filesSkipped?: number;
+  recordsProcessed: number;
+  eventsSent: number;
+  eventsFailed: number;
+  fileResults: FileProcessingResult[];
+  error?: string;
+}
+
+/**
+ * Per-file processing result
+ */
+export interface FileProcessingResult {
+  fileName: string;
+  success: boolean;
+  skipped?: boolean;
+  recordsProcessed: number;
+  eventsSent: number;
+  eventsFailed: number;
+  duration: number;
+  error?: string;
+}
+
+/**
+ * Parsed XML document structure
+ */
+export interface ParsedProductsDocument {
+  products?: {
+    product?: Product | Product[];
+  };
+}
+```
+
+---
+
+## 4. Service: Product File Processor (`src/services/product-file-processor.service.ts`)
+
+```typescript
+/**
+ * Product File Processor Service
+ *
+ * Downloads XML files from S3, parses, and transforms with UniversalMapper.
+ * XML-specific: Handles array normalization (single vs multiple products).
+ */
+
+import {
+  S3DataSource,
+  XMLParserService,
+  UniversalMapper,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { ProcessFileResult, Product, ParsedProductsDocument } from '../types/product-ingestion.types';
+
+/**
+ * Service for processing XML product files from S3
+ */
+export class ProductFileProcessorService {
+  constructor(
+    private s3: S3DataSource,
+    private xmlParser: XMLParserService,
+    private mapper: UniversalMapper,
+    private catalogueRef: string
+  ) {}
+
+  /**
+   * Download XML file from S3, parse, and transform with UniversalMapper
+   *
+   * ✅ PRODUCTION ENHANCEMENT: Memory cleanup pattern
+   * - Explicit null assignments after each step
+   * - Finally block guarantees cleanup
+   * - Prevents OOM errors on large XML files
+   */
+  async downloadParseAndTransform(objectKey: string): Promise<ProcessFileResult> {
+    // ✅ CRITICAL: Variables for cleanup tracking
+    let xmlContent: string | null = null;
+    let parsed: unknown | null = null;
+    let rawProducts: any[] | null = null;
+
+    try {
+      // STEP 1: Download XML content from S3
+      xmlContent = (await this.s3.downloadObject(objectKey, {
+        encoding: 'utf8',
+      })) as string;
+
+      // STEP 2: Parse XML with type safety
+      try {
+        parsed = await this.xmlParser.parse(xmlContent);
+      } catch (parseError: unknown) {
+        const errorMessage = parseError instanceof Error ? parseError.message : String(parseError);
+        return {
+          success: false,
+          products: [],
+          error: `XML parse error: ${errorMessage}`,
+        };
+      }
+
+      // ✅ Clear xmlContent from memory - no longer needed after parsing
+      xmlContent = null;
+
+      // Type guard: Validate parsed structure
+      if (!this.isParsedProductsDocument(parsed)) {
+        return {
+          success: false,
+          products: [],
+          error: 'Invalid XML structure: missing products element',
+        };
+      }
+
+      // STEP 3: Normalize products array (single element → object, multiple → array)
+      rawProducts = Array.isArray(parsed.products?.product)
+        ? parsed.products.product
+        : parsed.products?.product
+          ? [parsed.products.product]
+          : [];
+
+      if (rawProducts.length === 0) {
+        return {
+          success: false,
+          products: [],
+          error: 'No products found in XML',
+        };
+      }
+
+      // ✅ Clear parsed data from memory - only need rawProducts now
+      parsed = null;
+
+      // STEP 4: Transform with UniversalMapper
+      // ✅ S3 XML: Merge context using spread pattern (same as SFTP XML)
+      const sourceDataWithContext = rawProducts.map(item => ({
+        ...item,
+        $context: { catalogueRef: this.catalogueRef },
+      }));
+
+      const mappingResult = await this.mapper.map(sourceDataWithContext);
+
+      if (!mappingResult.success) {
+        return {
+          success: false,
+          products: [],
+          error: `Mapping validation failed: ${mappingResult.errors?.join(', ')}`,
+        };
+      }
+
+      if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+        this.log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+          skippedFields: mappingResult.skippedFields,
+          note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+        });
+      }
+
+      return {
+        success: true,
+        products: mappingResult.data as Product[],
+      };
+    } catch (error: unknown) {
+      const errorMessage = error instanceof Error ? error.message : String(error);
+      return {
+        success: false,
+        products: [],
+        error: errorMessage,
+      };
+    } finally {
+      // ✅ CRITICAL: Ensure all large objects are cleared even if error occurs
+      xmlContent = null;
+      parsed = null;
+      rawProducts = null;
+    }
+  }
+
+  /**
+   * Type guard: Check if parsed value is a valid ParsedProductsDocument
+   */
+  private isParsedProductsDocument(value: unknown): value is ParsedProductsDocument {
+    if (typeof value !== 'object' || value === null) {
+      return false;
+    }
+    const doc = value as Record<string, unknown>;
+    return 'products' in doc;
+  }
+}
+```
+
+---
+
+## 5. Service: Event Sender (`src/services/event-sender.service.ts`)
+
+```typescript
+/**
+ * Event Sender Service
+ *
+ * Sends product events to Fluent Commerce Event API with per-record error handling.
+ * Continues processing on individual failures (Event API best practice).
+ * Supports configurable concurrency (sequential or parallel).
+ */
+
+import type { FluentClient } from '@fluentcommerce/fc-connect-sdk';
+import type { EventResult, EventConfig, Product } from '../types/product-ingestion.types';
+
+/**
+ * Service for sending events to Fluent Commerce Event API
+ *
+ * ✅ PRODUCTION ENHANCEMENT: Optional logger for detailed progress tracking
+ */
+export class EventSenderService {
+  constructor(
+    private client: FluentClient,
+    private log?: any  // ✅ Optional logger for progress tracking
+  ) {}
+
+  /**
+   * Send events to Fluent Commerce Event API with configurable concurrency
+   *
+   * **Performance Characteristics:**
+   * - `concurrency: 1` → Sequential processing (safe default, ~1 event/sec)
+   * - `concurrency: 3-5` → Balanced throughput (~3-5 events/sec, good for most cases)
+   * - `concurrency: 10` → High-volume processing (~10 events/sec, 100+ products)
+   *
+   * **Implementation Strategy:**
+   * - Concurrency = 1: Optimized sequential loop (no Promise.allSettled overhead)
+   * - Concurrency > 1: Chunked parallel processing with bounded concurrency
+   * - Both modes: Per-record error tracking (failures don't block other events)
+   *
+   * @param products - Array of products to send as events
+   * @param eventConfig - Event configuration (name, catalogue ref, mode)
+   * @param concurrency - Number of concurrent event requests (default: 1, min: 1)
+   * @returns EventResult with counts (eventsSent/eventsFailed) and error details
+   */
+  async sendEvents(
+    products: Product[],
+    eventConfig: EventConfig,
+    concurrency: number = 1
+  ): Promise<EventResult> {
+    // Validate concurrency (guard against invalid values)
+    const safeConc = Math.max(1, Math.floor(concurrency));
+
+    // Result accumulators
+    let eventsSent = 0;
+    let eventsFailed = 0;
+    const errors: Array<{ productRef: string; error: string }> = [];
+
+    // ✅ PRODUCTION ENHANCEMENT: Log event sending start
+    if (this.log) {
+      this.log.info('📤 Starting event sending', {
+        totalProducts: products.length,
+        concurrency: safeConc,
+        processingMode: safeConc === 1 ? 'sequential (one at a time)' : `parallel (${safeConc} concurrently)`,
+        eventName: eventConfig.eventName,
+        catalogueRef: eventConfig.catalogueRef
+      });
+    }
+
+    // Helper: Build event payload (DRY - reused in both modes)
+    const buildPayload = (product: Product) => ({
+      name: eventConfig.eventName,
+      entityRef: eventConfig.catalogueRef,
+      entityType: 'PRODUCT_CATALOGUE' as const,
+      entitySubtype: eventConfig.catalogueType,
+      rootEntityRef: eventConfig.catalogueRef,
+      rootEntityType: 'PRODUCT_CATALOGUE' as const,
+      attributes: product,
+    });
+
+    // ============================================================================
+    // SEQUENTIAL MODE (concurrency === 1)
+    // ============================================================================
+    if (safeConc === 1) {
+      for (let i = 0; i < products.length; i++) {
+        const product = products[i];
+
+        // ✅ PRODUCTION ENHANCEMENT: Log progress every 10 products
+        if (this.log && i % 10 === 0) {
+          this.log.info(`📤 Sending product ${i + 1}/${products.length}`, {
+            productRef: product.ref,
+            progress: `${((i / products.length) * 100).toFixed(1)}%`
+          });
+        }
+
+        try {
+          await this.client.sendEvent(buildPayload(product), eventConfig.eventMode);
+          eventsSent++;
+        } catch (err: unknown) {
+          eventsFailed++;
+          const errorMsg = err instanceof Error ? err.message : String(err);
+          errors.push({ productRef: product?.ref || 'unknown', error: errorMsg });
+          // Continue processing (failure doesn't block other products)
+        }
+      }
+
+      // ✅ PRODUCTION ENHANCEMENT: Log completion
+      if (this.log) {
+        this.log.info('✅ Sequential event sending completed', {
+          totalProducts: products.length,
+          eventsSent,
+          eventsFailed
+        });
+      }
+
+      return { eventsSent, eventsFailed, errors };
+    }
+
+    // ============================================================================
+    // PARALLEL MODE (concurrency > 1)
+    // ============================================================================
+    const totalChunks = Math.ceil(products.length / safeConc);
+
+    for (let i = 0; i < products.length; i += safeConc) {
+      const chunk = products.slice(i, i + safeConc);
+      const chunkNumber = Math.floor(i / safeConc) + 1;
+
+      // ✅ PRODUCTION ENHANCEMENT: Log chunk progress
+      if (this.log) {
+        this.log.info(`📦 Processing chunk ${chunkNumber}/${totalChunks}`, {
+          chunkNumber,
+          totalChunks,
+          productsInChunk: chunk.length,
+          productRange: `${i + 1}-${i + chunk.length}`,
+          totalProducts: products.length,
+          progress: `${((i / products.length) * 100).toFixed(1)}%`
+        });
+      }
+
+      // Fire all requests in chunk concurrently
+      const results = await Promise.allSettled(
+        chunk.map(product =>
+          this.client
+            .sendEvent(buildPayload(product), eventConfig.eventMode)
+            .then(() => ({ success: true as const, product }))
+            .catch(error => ({ success: false as const, product, error }))
+        )
+      );
+
+      // Aggregate chunk results into totals
+      let chunkSuccess = 0;
+      let chunkFailed = 0;
+
+      for (const result of results) {
+        if (result.status === 'fulfilled' && result.value.success) {
+          eventsSent++;
+          chunkSuccess++;
+        } else {
+          eventsFailed++;
+          chunkFailed++;
+          const error = result.status === 'fulfilled' ? result.value.error : result.reason;
+          const product = result.status === 'fulfilled' ? result.value.product : null;
+          errors.push({
+            productRef: product?.ref || 'unknown',
+            error: error?.message || String(error) || 'unknown error',
+          });
+        }
+      }
+
+      // ✅ PRODUCTION ENHANCEMENT: Log chunk completion
+      if (this.log) {
+        this.log.info(`✅ Chunk ${chunkNumber}/${totalChunks} completed`, {
+          chunkNumber,
+          totalChunks,
+          chunkSuccess,
+          chunkFailed,
+          totalSentSoFar: eventsSent,
+          totalFailedSoFar: eventsFailed,
+          progress: `${(((i + chunk.length) / products.length) * 100).toFixed(1)}%`
+        });
+      }
+    }
+
+    // ✅ PRODUCTION ENHANCEMENT: Log parallel completion
+    if (this.log) {
+      this.log.info('✅ Parallel event sending completed', {
+        totalProducts: products.length,
+        totalChunks,
+        concurrency: safeConc,
+        eventsSent,
+        eventsFailed
+      });
+    }
+
+    return { eventsSent, eventsFailed, errors };
+  }
+}
+```
+
+---
+
+## 6. Service: Event Logger (`src/services/event-logger.service.ts`)
+
+```typescript
+/**
+ * Event Logger Service
+ *
+ * Writes event processing logs to S3 for audit/debugging.
+ * Optional - can be used to create rejection reports.
+ */
+
+import { Buffer } from 'node:buffer';
+import { S3DataSource } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Service for writing event logs to S3
+ */
+export class EventLoggerService {
+  constructor(private s3: S3DataSource) {}
+
+  /**
+   * Write event log to S3
+   *
+   * @param objectKey - S3 object key for log file
+   * @param logData - Log data to write (JSON format)
+   */
+  async writeEventLog(objectKey: string, logData: unknown): Promise<void> {
+    try {
+      const logContent = JSON.stringify(logData, null, 2);
+      await this.s3.uploadFile(objectKey, logContent);
+    } catch (error: unknown) {
+      const errorMessage = error instanceof Error ? error.message : String(error);
+      throw new Error(`Failed to write event log: ${errorMessage}`);
+    }
+  }
+}
+```
+
+## 7. Main Orchestration Service (src/services/product-ingestion.service.ts)
+
+```typescript
+/**
+ * MAIN INGESTION ORCHESTRATION SERVICE
+ *
+ * This is the heart of the ingestion workflow. It coordinates all steps:
+ * 1. Initialize clients and services
+ * 2. Discover files on S3
+ * 3. Download and parse XML files
+ * 4. Transform data with UniversalMapper
+ * 5. Send events to Fluent Commerce
+ * 6. Archive processed files
+ * 7. Track file processing state (S3 uses moveObject for deduplication)
+ * 8. Track job progress with JobTracker
+ *
+ * NAMING PATTERN (consistent across all use cases):
+ * - Interface: {Entity}IngestionParams (e.g., ProductIngestionParams)
+ * - Result: {Entity}IngestionResult (e.g., ProductIngestionResult)
+ * - Main function: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ * KEY DIFFERENCES FOR S3:
+ * - S3DataSource instead of SftpDataSource
+ * - No VersoriFileTracker (S3 uses moveObject for deduplication)
+ * - S3 listObjects instead of SFTP listFiles
+ * - S3 moveObject instead of SFTP moveFile
+ * - S3 dispose() for cleanup (S3DataSource does need dispose)
+ */
+
+import { Buffer } from 'node:buffer';
+import {
+  createClient,
+  S3DataSource,
+  XMLParserService,
+  UniversalMapper,
+  JobTracker,
+  type FluentClient,
+  type JobStatus,
+} from '@fluentcommerce/fc-connect-sdk';
+import type {
+  ProductIngestionParams,
+  ProductIngestionResult,
+  FileProcessingResult,
+  EventConfig,
+} from '../types/product-ingestion.types';
+import { ProductFileProcessorService } from './product-file-processor.service';
+import { EventSenderService } from './event-sender.service';
+import { EventLoggerService } from './event-logger.service';
+import { timestampedName } from '../utils/s3-path.utils';
+
+import mappingConfig from '../../config/products.import.xml.json' with { type: 'json' };
+
+// ✅ VERSORI PLATFORM: Use native log from context, not LoggingService
+
+/**
+ * Query job status from KV store
+ *
+ * NAMING: get{Entity}JobStatus or just getJobStatus (generic)
+ *
+ * ✅ VERSORI PLATFORM: Uses native log from context (passed as parameter)
+ */
+export async function getJobStatus(
+  kv: ReturnType<VersoriContext['openKv']>,
+  jobId: string,
+  log: VersoriContext['log']
+): Promise<JobStatus | undefined> {
+  try {
+    const tracker = new JobTracker(kv, log);
+    return await tracker.getJob(jobId);  // ✅ Use getJob() not getJobStatus()
+  } catch (error: unknown) {
+    const errorMessage = error instanceof Error ? error.message : String(error);
+    log.error('Failed to get job status', {
+      jobId,
+      message: errorMessage,
+      stack: error instanceof Error ? error.stack : undefined,
+      errorType: error instanceof Error ? error.constructor.name : 'Error',
+    });
+    return undefined;
+  }
+}
+
+/**
+ * MAIN ORCHESTRATION FUNCTION
+ *
+ * NAMING: execute{Entity}Ingestion (e.g., executeProductIngestion)
+ *
+ * This function implements the complete ingestion workflow in 8 steps.
+ * Each step is clearly commented for AI understanding.
+ */
+export async function executeProductIngestion(
+  ctx,
+  params: ProductIngestionParams
+): Promise<ProductIngestionResult> {
+
+  // ✅ VERSORI PLATFORM: Extract native log from context (LoggingService was removed - use native log)
+  const { log, openKv, activation } = ctx;
+  const { jobId, triggeredBy, filePattern, maxFiles, forceReprocess } = params;
+
+  // Open KV store for state management and job tracking
+  // ✅ Pass raw Versori KV directly - it matches KVAdapter interface
+  // ✅ Pass native log to JobTracker
+  const kv = openKv(':project:');
+  const tracker = new JobTracker(kv, log);
+
+  const startTime = Date.now();
+  const fileResults: FileProcessingResult[] = [];
+
+  // ⚠️ CRITICAL: Declare S3 outside try block for disposal pattern
+  let s3: S3DataSource | undefined;
+
+  try {
+    // 
+    // STEP 1: Initialize Job Tracking
+    // 
+    log.info('📋 [STEP 1/8] Initializing job tracking', { jobId });
+
+    await tracker.createJob(jobId, {
+      triggeredBy,
+      filePattern: filePattern || 'default',
+      maxFiles: maxFiles || 'unlimited',
+      forceReprocess: !!forceReprocess
+    });
+
+    // 
+    // STEP 2: Initialize Fluent Client & S3 Connection
+    // 
+    log.info('🔌 [STEP 2/8] Initializing Fluent Commerce client and S3', { jobId });
+
+    const client = await createClient(ctx);
+
+    if (!client) {
+      throw new Error('Failed to create Fluent Commerce client');
+    }
+
+    // ✅ CRITICAL: Set retailerId for Event API calls
+    // Event API requires retailerId - fail fast if not configured
+    const fluentRetailerId = activation.getVariable('fluentRetailerId');
+    if (!fluentRetailerId) {
+      throw new Error('fluentRetailerId is required for Event API calls');
+    }
+    client.setRetailerId(fluentRetailerId);
+    log.info('✅ RetailerId set for Event API', { retailerId: fluentRetailerId });
+
+    // Get S3 configuration from activation variables
+    const s3Config = {
+      bucket: activation.getVariable('s3Bucket'),
+      region: activation.getVariable('s3Region') || 'us-east-1',
+      accessKeyId: activation.getVariable('s3AccessKeyId'),
+      secretAccessKey: activation.getVariable('s3SecretAccessKey'),
+      incomingPrefix: activation.getVariable('s3IncomingPrefix') || 'products/incoming/',
+      processedPrefix: activation.getVariable('s3ProcessedPrefix') || 'products/processed/',
+      errorPrefix: activation.getVariable('s3ErrorPrefix') || 'products/errors/',
+      logsPrefix: activation.getVariable('s3LogsPrefix') || 'products/logs/',
+      filePattern: filePattern || activation.getVariable('filePattern') || 'products_*.xml'
+    };
+
+    // Validate S3 config
+    if (!s3Config.bucket) {
+      throw new Error('S3 configuration incomplete: missing bucket');
+    }
+
+    if (!s3Config.accessKeyId || !s3Config.secretAccessKey) {
+      throw new Error('S3 configuration incomplete: missing accessKeyId or secretAccessKey');
+    }
+
+    // Initialize S3 data source
+    // ✅ VERSORI PLATFORM: Pass native log from context
+    s3 = new S3DataSource(
+      {
+        type: 'S3_XML',
+        connectionId: 's3-product-ingestion',
+        name: 'Product Ingestion S3',
+        s3Config: {
+          bucket: s3Config.bucket,
+          region: s3Config.region,
+          accessKeyId: s3Config.accessKeyId,
+          secretAccessKey: s3Config.secretAccessKey,
+        },
+        validateConnection: true, // ✅ Enable connection validation on init
+      },
+      log
+    );
+
+    try {
+      // Connection already validated during initialization
+      // await s3.validateConnection(); // No longer needed - done in constructor
+      log.info('✅ S3 connection validated');
+
+      // 
+      // STEP 3: Discover Files on S3
+      // 
+      log.info('🔍 [STEP 3/8] Discovering files on S3', {
+        jobId,
+        prefix: s3Config.incomingPrefix,
+        filePattern: s3Config.filePattern
+      });
+
+      await tracker.updateJob(jobId, {
+        status: 'processing',
+        stage: 'file_discovery',
+        message: 'Discovering files on S3'
+      });
+
+      // List objects from S3
+      const allFiles = await s3.listObjects({
+        prefix: s3Config.incomingPrefix,
+        pattern: s3Config.filePattern
+      });
+
+      log.info(`📁 Discovered ${allFiles.length} files matching pattern`);
+
+      // Apply maxFiles limit if specified
+      const filesToProcess = maxFiles ? allFiles.slice(0, maxFiles) : allFiles;
+
+      if (filesToProcess.length === 0) {
+        log.info('No files to process');
+
+        await tracker.markCompleted(jobId, {
+          filesProcessed: 0,
+          message: 'No files found to process'
+        });
+
+        return {
+          success: true,
+          jobId,
+          filesProcessed: 0,
+          filesFailed: 0,
+          recordsProcessed: 0,
+          eventsSent: 0,
+          eventsFailed: 0,
+          fileResults: []
+        };
+      }
+
+      log.info(`⚙️ Processing ${filesToProcess.length} files`);
+
+      // Initialize services
+      const xmlParser = new XMLParserService();
+      const mapper = new UniversalMapper(mappingConfig);
+      
+      // Get event configuration
+      const eventConfig: EventConfig = {
+        catalogueRef: params.catalogueRef || activation.getVariable('catalogueRef') || 'PC:MASTER:2',
+        catalogueType: activation.getVariable('catalogueType') || 'MASTER',
+        eventName: activation.getVariable('eventName') || 'UPSERT_PRODUCT',
+        eventMode: (activation.getVariable('eventMode') || 'async') as 'async' | 'sync',
+      };
+
+      // Get event sending configuration
+      const eventConcurrency = Math.max(
+        1,
+        parseInt(activation.getVariable('eventConcurrency') || '1', 10)
+      );
+      // Validate: Ensure concurrency is at least 1 (sequential) or higher (parallel)
+      // concurrency: 1 = sequential, concurrency > 1 = parallel
+
+      log.info(`Event concurrency: ${eventConcurrency}`, {
+        mode: eventConcurrency === 1 ? 'sequential' : 'parallel',
+        concurrentRequests: eventConcurrency === 1 ? 'N/A' : eventConcurrency,
+      });
+
+      // Initialize class-based services
+      const fileProcessor = new ProductFileProcessorService(
+        s3,
+        xmlParser,
+        mapper,
+        eventConfig.catalogueRef
+      );
+      // ✅ PRODUCTION ENHANCEMENT: Pass log to EventSenderService for detailed progress tracking
+      const eventSender = new EventSenderService(client, log);
+      const eventLogger = new EventLoggerService(s3);
+
+      // 
+      // STEP 4-7: Process Each File (Download → Parse → Transform → Send → Archive)
+      // 
+
+      for (let fileIndex = 0; fileIndex < filesToProcess.length; fileIndex++) {
+        const file = filesToProcess[fileIndex];
+        const fileStartTime = Date.now();
+
+        // Use object key from S3 listing
+        const objectKey = file.key || (file as any).name;
+        const baseName = objectKey.split('/').pop() || objectKey;
+
+        log.info(`📄 [FILE ${fileIndex + 1}/${filesToProcess.length}] Processing file: ${baseName}`);
+
+        try {
+          await tracker.updateJob(jobId, {
+            status: 'processing',
+            stage: 'downloading',
+            message: `Processing file ${fileIndex + 1} of ${filesToProcess.length}: ${baseName}`
+          });
+
+          // ═══════════════════════════════════════════════════════════
+          // STEP 4: Process File (Download + Parse + Map)
+          // ═══════════════════════════════════════════════════════════
+          log.info(`⚙️ [STEP 4/8] Processing file: ${baseName}`);
+
+          // Process file: download → parse → transform
+          const fileResult = await fileProcessor.downloadParseAndTransform(objectKey);
+
+          if (!fileResult.success || fileResult.products.length === 0) {
+            // Move failed file to errors and record result
+            const archivedName = timestampedName(baseName);
+            const errorKey = `${s3Config.errorPrefix}${archivedName}`;
+            await s3.moveObject(objectKey, errorKey);
+
+            fileResults.push({
+              fileName: baseName,
+              success: false,
+              recordsProcessed: fileResult.products.length,
+              eventsSent: 0,
+              eventsFailed: 0,
+              duration: Date.now() - fileStartTime,
+              error: fileResult.error || 'Processing failed'
+            });
+
+            continue;
+          }
+
+          // ═══════════════════════════════════════════════════════════
+          // STEP 5: Send Events
+          // ═══════════════════════════════════════════════════════════
+          log.info(`📤 [STEP 5/8] Sending events for ${baseName}`);
+
+          // ? Enhanced: Extract context for progress logging
+          const sampleProductRefs = fileResult.products.slice(0, 5).map((p: any) => p.ref || p.skuRef);
+          const eventType = eventConfig.eventType || 'UPSERT_PRODUCT';
+
+          // ? Enhanced: Start logging with context
+          log.info(`[EventSender] Sending events for file "${baseName}"`, {
+           totalProducts: fileResult.products.length,
+           eventType,
+           concurrency: eventConcurrency === 1 ? 'sequential' : `parallel (${eventConcurrency})`,
+           sampleProductRefs: sampleProductRefs.join(', '),
+           eventMode: eventConfig.eventMode || 'async'
+          });
+
+          const eventResult = await eventSender.sendEvents(
+            fileResult.products,
+            eventConfig,
+            eventConcurrency
+          );
+
+          const eventsSent = eventResult.eventsSent;
+          const eventsFailed = eventResult.eventsFailed;
+
+          log.info(`✅ Events sent for ${baseName}`, {
+            successful: eventsSent,
+            failed: eventsFailed
+          });
+
+          // ? Enhanced: Completion logging with summary
+          log.info(`[EventSender] Event submission completed for file "${baseName}"`, {
+           totalProducts: fileResult.products.length,
+           eventsSent,
+           eventsFailed,
+           successRate: fileResult.products.length > 0 ? `${Math.round((eventsSent / fileResult.products.length) * 100)}%` : '0%',
+           eventType
+          });
+
+          // ═══════════════════════════════════════════════════════════
+          // STEP 6: Archive File
+          // ═══════════════════════════════════════════════════════════
+          log.info(`📦 [STEP 6/8] Archiving file: ${baseName}`);
+
+          await tracker.updateJob(jobId, {
+            status: 'processing',
+            stage: 'archiving',
+            message: `Archiving ${baseName}`
+          });
+
+          // Conditionally archive: errors → errorPrefix, else → processedPrefix (timestamped)
+          // ✅ S3: Uses moveObject for deduplication (no VersoriFileTracker needed)
+          const archivedName = timestampedName(baseName);
+          const targetPrefix = eventsFailed > 0 ? s3Config.errorPrefix : s3Config.processedPrefix;
+          const targetKey = `${targetPrefix}${archivedName}`;
+          await s3.moveObject(objectKey, targetKey);
+
+          log.info(`✅ File archived successfully: ${baseName}`, { targetKey });
+
+          // Optional: Write event log for audit/debugging
+          if (eventsFailed > 0) {
+            const logKey = `${s3Config.logsPrefix}${baseName.replace(/\.xml$/i, '')}-event-log.json`;
+            await eventLogger.writeEventLog(logKey, {
+              fileName: baseName,
+              totalRecords: fileResult.products.length,
+              eventsSent,
+              eventsFailed,
+              errors: eventResult.errors,
+              processedAt: new Date().toISOString()
+            });
+          }
+
+          // Store file result
+          fileResults.push({
+            fileName: baseName,
+            success: true,
+            recordsProcessed: fileResult.products.length,
+            eventsSent,
+            eventsFailed,
+            duration: Date.now() - fileStartTime
+          });
+
+        } catch (error: unknown) {
+          // ✅ Enhanced error logging: Extract all error details for visibility
+          const errorMessage = error instanceof Error ? error.message : String(error);
+          const errorDetails = {
+            message: errorMessage,
+            stack: error instanceof Error ? error.stack : undefined,
+            fileName: (error as any)?.fileName,
+            lineNumber: (error as any)?.lineNumber,
+            originalError: (error as any)?.context?.originalError?.message,
+            errorType: error instanceof Error ? error.name : 'Error',
+          };
+          log.error(`Error processing file ${baseName}:`, errorDetails);
+
+          // Best-effort: move to error archive on unexpected failure
+          try {
+            const archivedName = timestampedName(baseName);
+            const errorKey = `${s3Config.errorPrefix}${archivedName}`;
+            await s3.moveObject(objectKey, errorKey);
+          } catch (moveError: unknown) {
+            const moveErrorMessage = moveError instanceof Error ? moveError.message : String(moveError);
+            log.error('Could not archive failed file', {
+              file: baseName,
+              message: moveErrorMessage,
+              stack: moveError instanceof Error ? moveError.stack : undefined,
+              errorType: moveError instanceof Error ? moveError.constructor.name : 'Error'
+            });
+          }
+
+          fileResults.push({
+            fileName: baseName,
+            success: false,
+            recordsProcessed: 0,
+            eventsSent: 0,
+            eventsFailed: 0,
+            duration: Date.now() - fileStartTime,
+            error: errorMessage
+          });
+        }
+      }
+
+      // 
+      // STEP 8: Complete Job & Calculate Totals
+      // 
+      log.info('🏁 [STEP 8/8] Completing job and calculating totals', { jobId });
+
+      const filesProcessed = fileResults.filter(r => r.success).length;
+      const filesFailed = fileResults.filter(r => !r.success).length;
+      const totalRecordsProcessed = fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0);
+      const totalEventsSent = fileResults.reduce((sum, r) => sum + r.eventsSent, 0);
+      const totalEventsFailed = fileResults.reduce((sum, r) => sum + r.eventsFailed, 0);
+
+
+      await tracker.markCompleted(jobId, {
+        filesProcessed,
+        filesFailed,
+        recordsProcessed: totalRecordsProcessed,
+        eventsSent: totalEventsSent,
+        eventsFailed: totalEventsFailed,
+        duration: Date.now() - startTime
+      });
+
+      log.info('✅ Ingestion completed successfully', {
+        filesProcessed,
+        filesFailed,
+        recordsProcessed: totalRecordsProcessed,
+        eventsSent: totalEventsSent,
+        eventsFailed: totalEventsFailed,
+        duration: Date.now() - startTime
+      });
+
+      return {
+        success: true,
+        jobId,
+        filesProcessed,
+        filesFailed,
+        recordsProcessed: totalRecordsProcessed,
+        eventsSent: totalEventsSent,
+        eventsFailed: totalEventsFailed,
+        fileResults
+      };
+
+    } finally {
+      // ❌š ï¸ CRITICAL: Always dispose S3 connection
+      if (s3) {
+        await s3.dispose();
+        log.info('S3 connection disposed');
+      }
+    }
+
+  } catch (error: unknown) {
+    const errorMessage = error instanceof Error ? error.message : String(error);
+    const errorStack = error instanceof Error ? error.stack : undefined;
+
+    // Generate contextual recommendations based on error type
+    const recommendations = [];
+    if (errorMessage.includes('S3') || errorMessage.includes('bucket')) {
+      recommendations.push('Verify S3 credentials (accessKeyId, secretAccessKey)');
+      recommendations.push('Check S3 bucket name and region configuration');
+      recommendations.push('Ensure bucket permissions allow ListObjects and GetObject');
+    }
+    if (errorMessage.includes('XML') || errorMessage.includes('parse')) {
+      recommendations.push('Validate XML file structure against expected schema');
+      recommendations.push('Check for special characters or encoding issues');
+      recommendations.push('Review XML parser configuration');
+    }
+    if (errorMessage.includes('event') || errorMessage.includes('API')) {
+      recommendations.push('Verify Fluent API credentials and retailerId');
+      recommendations.push('Check network connectivity to Fluent API');
+      recommendations.push('Review event payload structure');
+    }
+    if (errorMessage.includes('mapping')) {
+      recommendations.push('Review mapping configuration in config/*.json');
+      recommendations.push('Check for missing required fields');
+      recommendations.push('Validate resolver functions');
+    }
+
+    log.error('❌ Ingestion workflow failed', {
+      jobId,
+      error: errorMessage,
+      stack: errorStack,
+      recommendations: recommendations.length > 0 ? recommendations : ['Review logs for detailed error information'],
+    });
+
+    // Try to mark job as failed, but don't let tracking errors mask workflow error
+    try {
+      await tracker.markFailed(jobId, error);
+    } catch (trackingError: unknown) {
+      const trackingErrorMessage =
+        trackingError instanceof Error ? trackingError.message : String(trackingError);
+      log.warn('Failed to mark job as failed in tracker', {
+        jobId,
+        trackingError: trackingErrorMessage,
+      });
+    }
+
+    return {
+      success: false,
+      jobId,
+      filesProcessed: fileResults.filter(r => r.success && !r.skipped).length,
+      filesFailed: fileResults.filter(r => !r.success).length,
+      filesSkipped: fileResults.filter(r => r.skipped).length,
+      recordsProcessed: fileResults.reduce((sum, r) => sum + r.recordsProcessed, 0),
+      eventsSent: fileResults.reduce((sum, r) => sum + r.eventsSent, 0),
+      eventsFailed: fileResults.reduce((sum, r) => sum + r.eventsFailed, 0),
+      fileResults,
+      error: errorMessage,
+    };
+  } finally {
+    // ⚠️ CRITICAL: Ensure S3 is disposed even if outer error occurs
+    // This outer finally ensures disposal if error happens after S3 creation
+    // but before inner try block (e.g., during connection validation)
+    if (s3) {
+      try {
+        await s3.dispose();
+        log.info('🔌 S3 connection disposed (outer finally)');
+      } catch (disposeError: unknown) {
+        const disposeErrorMessage =
+          disposeError instanceof Error ? disposeError.message : String(disposeError);
+        log.warn('⚠️ Error disposing S3 in outer finally', { error: disposeErrorMessage });
+      }
+    }
+  }
+  // ═══════════════════════════════════════════════════════════
+  // EXECUTION BOUNDARY: Product Ingestion End
+  // ═══════════════════════════════════════════════════════════
+}
+```
+
+## 8. Utility Functions (src/utils/)
+
+### S3 Path Helpers (src/utils/s3-path.utils.ts)
+
+```typescript
+/**
+ * S3 Path Utilities
+ *
+ * Helper functions for S3 path operations.
+ */
+
+/**
+ * Generate timestamped filename for archival
+ *
+ * Adds ISO timestamp to filename before extension for unique archival.
+ * Handles XML files specifically but can be adapted for other formats.
+ *
+ * @param name - Original filename
+ * @returns Filename with timestamp appended
+ *
+ * @example
+ * ```typescript
+ * timestampedName('products.xml')
+ * // Returns: 'products-2025-11-01T18-30-45-123Z.xml'
+ * ```
+ */
+export function timestampedName(name: string): string {
+  const ts = new Date().toISOString().replace(/[:.]/g, '-');
+  const base = name.replace(/\.xml$/i, '');
+  return `${base}-${ts}.xml`;
+}
+```
+
+### Job ID Generator (src/utils/job-id-generator.ts)
+
+```typescript
+/**
+ * Job ID Generator
+ *
+ * Generates unique job IDs for tracking ingestion runs.
+ * Format: {PREFIX}_{ENTITY}_{DATE}_{TIME}_{RANDOM}
+ * Example: SCHEDULED_PROD_20250124_183045_abc123
+ */
+
+/**
+ * Generate unique job ID
+ *
+ * @param prefix - Job prefix (SCHEDULED, ADHOC, etc.)
+ * @param entity - Entity type (PROD, INV, etc.)
+ * @returns Unique job ID string
+ */
+export function generateJobId(prefix: string, entity: string): string {
+  const now = new Date();
+  const date = now.toISOString().slice(0, 10).replace(/-/g, '');
+  const time = now.toISOString().slice(11, 19).replace(/:/g, '');
+  const random = Math.random().toString(36).slice(2, 8).toUpperCase();
+  return `${prefix}_${entity}_${date}_${time}_${random}`;
+}
+```
+
+## 📄 Mapping Configuration
+
+**File:** `config/products.import.xml.json`
+
+```json
+{
+  "name": "products.import.xml",
+  "version": "1.2.0",
+  "description": "XML → Product Event Mapping",
+  "fields": {
+    "ref": {
+      "source": "ref",
+      "required": true,
+      "resolver": "sdk.trim",
+      "comment": "Product reference from XML element"
+    },
+    "type": {
+      "source": "type",
+      "required": true,
+      "resolver": "sdk.uppercase",
+      "comment": "Product type (STANDARD, VARIANT)"
+    },
+    "status": {
+      "source": "status",
+      "required": true,
+      "resolver": "sdk.uppercase",
+      "comment": "Product status (ACTIVE, INACTIVE)"
+    },
+    "name": {
+      "source": "name",
+      "required": true,
+      "resolver": "sdk.trim",
+      "comment": "Product name/title"
+    },
+    "summary": {
+      "source": "summary",
+      "required": false,
+      "comment": "Product short description"
+    },
+    "gtin": {
+      "source": "gtin",
+      "required": false,
+      "resolver": "sdk.trim",
+      "comment": "Global Trade Item Number (barcode)"
+    },
+    "catalogue.ref": {
+      "source": "$context.catalogueRef",
+      "required": false,
+      "comment": "Catalogue reference (injected from context, NOT in XML)"
+    },
+    "metadata.source": {
+      "value": "S3_XML",
+      "required": false,
+      "comment": "Static value - identifies data source (NOT in XML)"
+    }
+  }
+}
+```
+
+---
+
+## 9. Package Configuration
+
+### `package.json`
+
+```json
+{
+  "name": "s3-xml-product-event-ingestion",
+  "version": "1.2.0",
+  "description": "S3 XML Product Event Ingestion Connector",
+  "type": "module",
+  "main": "src/index.ts",
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "^1.0.0"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+
+---
+
+**File:** `config/products.import.xml.json`
+
+```json
+{
+  "name": "products.import.xml",
+  "version": "1.1.0",
+  "description": "XML → Product Event Mapping",
+  "fields": {
+    "ref": { "source": "product.ref", "required": true, "resolver": "sdk.trim" },
+    "type": { "source": "product.type", "resolver": "sdk.uppercase", "defaultValue": "STANDARD" },
+    "status": { "source": "product.status", "resolver": "sdk.uppercase", "defaultValue": "ACTIVE" },
+    "gtin": { "source": "product.gtin" },
+    "name": { "source": "product.name", "required": true, "resolver": "sdk.trim" },
+    "summary": { "source": "product.summary" },
+    "categoryRefs": { "source": "product.categoryRefs.ref", "isArray": true },
+    "price": {
+      "source": "product.price.item",
+      "isArray": true,
+      "fields": {
+        "type": { "source": "type" },
+        "currency": { "source": "currency" },
+        "value": { "source": "value" }
+      }
+    },
+    "taxType": {
+      "source": "product.taxType",
+      "fields": {
+        "country": { "source": "country" },
+        "group": { "source": "group" },
+        "tariff": { "source": "tariff" }
+      }
+    },
+    "attributes": { "defaultValue": [] }
+  }
+}
+```
+
+Note: Adjust fields in the JSON above as needed; prefer built-in resolvers. For advanced patterns, see SDK Universal Mapping guide.
+
+---
+
+## Expected XML Format
+
+**Sample:** `products_20250124.xml`
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<products>
+  <product>
+    <ref>G_PROD_WITH_NO_STANDARD</ref>
+    <type>VARIANT</type>
+    <status>ACTIVE</status>
+    <gtin>MH01-XS-Orange</gtin>
+    <name>Chaz Kangeroo Hoodie-XS-Orange main</name>
+    <summary>&lt;p&gt;test short description&lt;/p&gt;</summary>
+    <categoryRefs>
+      <ref>STANDARD_CATEGORY</ref>
+    </categoryRefs>
+    <price>
+      <item>
+        <type>DEFAULT</type>
+        <currency>USD</currency>
+        <value>52.000000</value>
+      </item>
+      <item>
+        <type>SPECIAL</type>
+        <currency>USD</currency>
+        <value>9.000000</value>
+      </item>
+    </price>
+    <taxType>
+      <country>AU</country>
+      <group>Tax Group</group>
+      <tariff>Tax Tariff</tariff>
+    </taxType>
+    <catalogue>
+      <ref>PC:MASTER:2</ref>
+      <type>MASTER</type>
+    </catalogue>
+  </product>
+  <product>
+    <ref>G_PROD_002</ref>
+    <type>VARIANT</type>
+    <status>ACTIVE</status>
+    <gtin>MH02-M-Blue</gtin>
+    <name>Kangeroo Hoodie-M-Blue</name>
+    <summary>&lt;p&gt;Blue variant medium size&lt;/p&gt;</summary>
+    <categoryRefs>
+      <ref>STANDARD_CATEGORY</ref>
+      <ref>SEASONAL</ref>
+    </categoryRefs>
+    <price>
+      <item>
+        <type>DEFAULT</type>
+        <currency>USD</currency>
+        <value>45.000000</value>
+      </item>
+      <item>
+        <type>SPECIAL</type>
+        <currency>USD</currency>
+        <value>35.000000</value>
+      </item>
+    </price>
+    <taxType>
+      <country>AU</country>
+      <group>Tax Group</group>
+      <tariff>Tax Tariff</tariff>
+    </taxType>
+    <catalogue>
+      <ref>PC:MASTER:2</ref>
+      <type>MASTER</type>
+    </catalogue>
+  </product>
+</products>
+```
+
+**XML Field Mapping:**
+
+- Nested structure for objects (e.g., `<taxType><country>AU</country></taxType>`)
+- Arrays with repeated elements (e.g., `<categoryRefs><ref>CAT1</ref><ref>CAT2</ref></categoryRefs>`)
+- HTML content in CDATA or escaped (e.g., `&lt;p&gt;description&lt;/p&gt;`)
+
+---
+
+## 9. Package Configuration
+
+### `package.json`
+
+```json
+{
+  "name": "s3-xml-product-event",
+  "version": "1.1.0",
+  "type": "module",
+  "main": "src/index.ts",
+  "scripts": {
+    "dev": "versori dev",
+    "build": "versori build",
+    "deploy": "versori deploy"
+  },
+  "dependencies": {
+    "@fluentcommerce/fc-connect-sdk": "^0.1.39",
+    "@versori/run": "latest"
+  },
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}
+```
+
+### `tsconfig.json`
+
+```json
+{
+  "compilerOptions": {
+    "module": "ES2022",
+    "target": "ES2024",
+    "moduleResolution": "node"
+  }
+}
+```
+
+---
+
+## 10. Deployment Instructions
+
+### Deploy to Versori
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Test locally (if using Versori CLI)
+npm run dev
+
+# 3. Deploy to Versori platform
+npm run deploy
+```
+
+### Configure Activation Variables
+
+In Versori platform settings, configure all variables listed in the Activation Variables section above.
+
+---
+
+## 11. Testing
+
+### Test Scheduled Ingestion
+
+Upload a test XML file to S3 incoming prefix and wait for the scheduled run.
+
+**Check logs:**
+
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and S3
+[STEP 3/8] Discovering files on S3
+[FILE 1/1] Processing file: products_20250124.xml
+[STEP 4/8] Downloading and parsing: products_20250124.xml
+[STEP 5/8] Transforming 5 products from products_20250124.xml
+[STEP 6/8] Sending 5 events to Fluent Commerce
+[STEP 7/8] Archiving file: products_20250124.xml
+[STEP 8/8] Completing job and calculating totals
+```
+
+### Test Ad hoc Ingestion
+
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+  -H "X-API-Key: your-secret-key" \
+  -H "Content-Type: application/json" \
+  -d '{}'
+
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/product-ingestion-adhoc \
+  -H "X-API-Key: your-secret-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "filePattern": "urgent_*.xml",  }'
+```
+
+### Test Job Status Query
+
+```bash
+curl -X POST https://api.versori.com/webhooks/product-ingestion-job-status \
+  -H "X-API-Key: your-secret-key" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "jobId": "ADHOC_PROD_20251024_183045_abc123"
+  }'
+```
+
+---
+
+## Monitoring
+
+### Success Response
+
+```json
+{
+  "success": true,
+  "filesProcessed": 1,
+  "filesSkipped": 0,
+  "filesFailed": 0,
+  "totalRecords": 50,
+  "eventsSent": 50,
+  "eventsFailed": 0,
+  "results": [
+    {
+      "file": "products_2025-01-22.xml",
+      "success": true,
+      "recordsProcessed": 50,
+      "eventsSent": 50,
+      "eventsFailed": 0
+    }
+  ],
+  "duration": 12345
+}
+```
+
+### Partial Success Response
+
+```json
+{
+  "success": true,
+  "filesProcessed": 1,
+  "filesSkipped": 0,
+  "filesFailed": 0,
+  "totalRecords": 50,
+  "eventsSent": 45,
+  "eventsFailed": 5,
+  "results": [
+    {
+      "file": "products_2025-01-22.xml",
+      "success": true,
+      "recordsProcessed": 50,
+      "eventsSent": 45,
+      "eventsFailed": 5,
+      "errors": ["PRD-001: Invalid SKU format", "PRD-002: Missing required field"]
+    }
+  ],
+  "duration": 12345
+}
+```
+
+### Error Response
+
+```json
+{
+  "success": false,
+  "filesProcessed": 0,
+  "filesFailed": 1,
+  "totalRecords": 0,
+  "eventsSent": 0,
+  "eventsFailed": 0,
+  "results": [
+    {
+      "file": "products_2025-01-22.xml",
+      "success": false,
+      "error": "XML parse error: Invalid structure"
+    }
+  ],
+  "duration": 876
+}
+```
+
+### Monitoring Metrics
+
+Monitor these metrics via Versori logs filtered by `jobId` or `stage`:
+
+- **Files Processed** - Total files successfully processed
+- **Events Sent** - Total events sent to Fluent Commerce
+- **Events Failed** - Events that failed (check rejection reports)
+- **Processing Duration** - Time taken for complete workflow
+- **Rate Limiting** - Watch for 429 errors indicating throttling
+
+Use the status webhook for dashboards and automated monitoring.
+
+---
+
+- **No files found:** Check `s3IncomingPrefix` and `filePattern` (object keys only)
+- **XML parse failed:** Move to errors/; validate XML structure and encoding
+- **XML array normalization:** Check if single object vs array - ALWAYS normalize
+- **High event failures:** Inspect logs; consider Batch API for very high volumes
+- **429 throttling:** Add small backoff/delay or use Batch API
+- **S3 connection errors:** Verify credentials, bucket, region, and permissions
+
+---
+
+## 13. Key Takeaways
+
+- ✅ **Native Versori logs** - Use `log` from context (LoggingService removed - use native log)
+- ✅ **3 workflows** - Scheduled, ad hoc webhook, job status webhook
+- ✅ **JobTracker** - Track job lifecycle with KV persistence
+- ❌ **NO VersoriFileTracker** - S3 uses moveObject for deduplication (SFTP-specific only)
+- ✅ **S3 deduplication** - Physical file movement to processed/errors prefixes
+- ✅ **S3 dispose()** - Always cleanup in finally block
+- ✅ **S3 method names** - listObjects, downloadObject, moveObject (not \*File)
+- ✅ **XML array normalization** - CRITICAL: single object vs array handling
+- ✅ **Per-record error handling** - Continue on individual failures
+- ✅ **Externalized mapping** - Use JSON config file for field mappings
+- ✅ **File-level error handling** - Don't stop on single file failure
+- ✅ **S3 vs SFTP** - Different deduplication strategies (moveObject vs KV state)
+- ✅ **Processing modes** - per-file (default), chunked, batch
+- ✅ **Modular services** - orchestrator, processor, dispatcher, archive
+
+---
+
+[← Back to Versori Event API Templates](../../readme.md) | [Versori Platform Guide →](../../../../../04-REFERENCE/platforms/versori/platforms-versori-readme.md)