@fluentcommerce/fc-connect-sdk 0.1.53 → 0.1.55

package/docs/01-TEMPLATES/versori/workflows/ingestion/batch-api/template-ingestion-s3-json-inventory-batch.md

@@ -1,1952 +1,1952 @@
----
-template_id: tpl-ingest-s3-json-inventory-batch
-canonical_filename: template-ingestion-s3-json-inventory-batch.md
-version: 2.0.0
-sdk_version: ^0.1.39
-runtime: versori
-direction: ingestion
-source: s3-json
-destination: fluent-batch-api
-entity: inventory
-format: json
-logging: versori
-status: stable
-features:
-  - batch-api-integration
-  - memory-management
-  - enhanced-logging
-  - attribute-transformation
----
-
-# Template: Ingestion - S3 JSON Inventory to Batch API
-
-**FC Connect SDK Use Case Guide**
-
-> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
-> **Version**: @fluentcommerce/fc-connect-sdk@^0.1.39
-
-**🆕 Production Code Enhancements (Applied):**
-1. ✅ Batch API with retry logic and BPP change detection
-2. ✅ Memory management (clearing large arrays after batch processing)
-3. ✅ Enhanced logging with emoji progress tracking (📦 batch creation, 📤 batch sending, ✅ completion)
-4. ✅ Attribute transformation with nested field support
-
-**Template Version:** 2.0.0
-**Last Updated:** 2025-01-24
-
-**Context**: Versori scheduled workflow that reads inventory JSON files from S3, transforms data with UniversalMapper, and sends bulk inventory updates to Fluent Commerce Batch API with BPP change detection
-
-**Complexity**: Medium
-
-**Runtime**: Versori Platform
-
-**Estimated Lines**: ~520 lines
-
----
-
-## STEP 1: Understand This Template
-
-**What This Template Does:**
-
-- Scheduled Versori workflow for bulk inventory ingestion from S3 JSON files
-- Reads JSON files from S3 with retry logic and streaming support
-- Parses JSON with format detection (standard JSON vs JSON Lines)
-- Transforms data using UniversalMapper with direct field access (no special notation)
-- Sends bulk updates to Fluent Commerce Batch API with chunking
-- Uses BPP (Batch Pre-Processing) for change detection
-- Tracks processing state with VersoriFileTracker to prevent duplicates
-- Archives processed files and handles errors gracefully
-
-**Key SDK Components:**
-
-- `createClient()` - Universal client factory (auto-detects Versori context)
-- `S3DataSource` - S3 operations with streaming (NEW: enhanced retry logic)
-- `JSONParserService` - JSON parsing with format auto-detection (JSON vs JSON Lines)
-- `UniversalMapper` - Field transformation with SDK resolvers
-- `VersoriFileTracker` - State management (prevent duplicate processing)
-- `JobTracker` - Job lifecycle tracking
-- Native Versori `log` - Use `log` from context
-
-**Entity Type:**
-
-- **InventoryQuantity** - Fluent entity for inventory positions and quantities
-- **EntityType: 'INVENTORY'** - Used in Batch API `sendBatch()` call
-- **Batch API Method** - Uses `createJob()` and `sendBatch()` (not Event API)
-
-**Critical Patterns:**
-
-- **JSON Format Support**: Standard JSON objects and JSON Lines (one record per line)
-- **Direct Field Access**: Use simple paths like `"locationRef"` (no `@` prefix needed for JSON)
-- **Safe S3 Paths**: Use absolute paths in S3DataSource config
-- **Always Dispose**: Call `s3.dispose()` in `finally` block to release connections
-- **BPP Configuration**: Enabled by default for full snapshots, skip for delta feeds
-- **Progress Logging**: Enhanced logging with context (sample SKUs, locations)
-- **Detailed Logging Toggle**: Optional detailed payload logging for debugging (default: disabled)
-- **File Tracking Toggle**: Optional file tracking to prevent duplicates (default: enabled)
-- **Retry Logic**: Enhanced S3 operations with exponential backoff
-
-**When to Use This Template:**
-
-- ✅ Daily/hourly full inventory snapshots from S3 JSON files
-- ✅ Bulk inventory updates with BPP change detection
-- ✅ JSON or JSON Lines format data
-- ✅ Need state management to prevent duplicate processing
-- ✅ Files should be archived after processing
-
-**When NOT to Use:**
-
-- ❌ Single inventory updates (use GraphQL mutation instead)
-- ❌ Products, Locations, Customers (use Event API templates)
-- ❌ Real-time inventory events (use Event API)
-- ❌ XML/CSV/Parquet formats (use appropriate format template)
-
----
-
-## STEP 2: AI Prompt
-
-**Copy this prompt to generate the complete implementation:**
-
-```
-Create a Versori scheduled workflow for S3 JSON inventory ingestion to Fluent Commerce Batch API.
-
-REQUIREMENTS:
-1. Runtime: Versori Platform (scheduled workflow)
-2. Source: S3 JSON files from s3://bucket/inventory/
-3. Destination: Fluent Commerce Batch API (InventoryQuantity entity)
-4. Format: JSON (standard JSON objects and JSON Lines support)
-5. Entity: InventoryQuantity (EntityType: 'INVENTORY')
-
-KEY FEATURES:
-- S3 JSON file discovery with VersoriFileTracker for state management
-- JSON parsing with format auto-detection (JSON vs JSON Lines)
-- Direct field mapping using UniversalMapper (no special notation needed)
-- Batch API with chunking (1000 records per batch) and BPP change detection
-- Safe S3 paths with absolute path requirements
-- S3 dispose() in finally block
-- Job lifecycle tracking with JobTracker
-- File archival on S3 after successful processing
-- Error handling with exponential backoff retry
-
-CRITICAL REQUIREMENTS:
-1. JSON Parser: JSONParserService with format auto-detection
-2. Array Extraction: Handle both standard JSON arrays and JSON Lines
-3. Mapping Config: Use direct paths like "locationRef" (no @ prefix)
-4. Entity Type: 'INVENTORY' (for InventoryQuantity)
-5. BPP: Enabled by default (full snapshots), skip for delta feeds
-6. S3 Config: Include proper AWS credentials configuration
-7. S3 Dispose: Always call s3.dispose() in finally block
-8. Native Logging: Use log from context
-9. Retry Logic: Enhanced S3 operations with exponential backoff
-
-SDK METHODS TO USE:
-- createClient(ctx) - Pass entire Versori context, auto-detects platform
-- client.setRetailerId(retailerId) - REQUIRED after createClient for Batch API operations
-- new S3DataSource(config, log) - Config includes AWS credentials
-- await s3.listFiles({ prefix, maxKeys })
-- await s3.downloadFile(key, { encoding: 'utf8' })
-- await s3.moveFile(sourceKey, destKey)
-- await s3.dispose() - MUST call in finally block
-- new JSONParserService()
-- await parser.parse(jsonContent, { format: 'json' | 'jsonl' })
-- new UniversalMapper(mappingConfig)
-- await mapper.map(records)
-- new VersoriFileTracker(ctx.openKv(':project:'), 'prefix')
-- await fileTracker.wasFileProcessed(fileName)
-- await fileTracker.markFileProcessed(fileName, metadata)
-- new JobTracker(ctx.openKv(':project:'), log)
-- await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' })
-- await tracker.updateJob(jobId, { status: 'processing' })
-- await tracker.markCompleted(jobId, details)
-- await tracker.markFailed(jobId, error)
-- await client.createJob({ name, meta: { preprocessing: 'skip' } })
-- await client.sendBatch(jobId, { action: 'UPSERT', entityType: 'INVENTORY', entities })
-- Fire-and-forget batch submission (no polling)
-
-FORBIDDEN PATTERNS:
-- ❌ LoggingService (removed - use native log on Versori)
-- ❌ Don't use @ prefix for JSON fields (that's for XML attributes only)
-- ❌ Don't forget to wrap records in proper structure for mapping
-- ❌ Don't use Event API (use Batch API)
-- ❌ Don't forget to call s3.dispose() in finally block
-- ❌ Don't use relative S3 paths (use absolute paths)
-
-MAPPING CONFIGURATION FILE: config/inventory.batch.json
-Structure:
-{
- "name": "inventory.batch.json",
- "version": "1.0.0",
- "description": "JSON inventory to Fluent Commerce Batch API mapping",
- "fields": {
-  "locationRef": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
-  "skuRef": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
-  "qty": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
-  "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
-  "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
-  "expectedOn": { "source": "expectedOn", "required": false, "resolver": "sdk.formatDate" },
-  "attributes.expiryDate": { "source": "attributes.expiryDate", "required": false, "resolver": "sdk.formatDate" },
-  "attributes.batchNumber": { "source": "attributes.batchNumber", "required": false },
-  "attributes.condition": { "source": "attributes.condition", "required": false, "defaultValue": "NEW", "resolver": "sdk.uppercase" },
-  "attributes.storageZone": { "source": "attributes.storageZone", "required": false }
- }
-}
-
-GENERATE:
-1. package.json with dependencies
-2. index.ts (workflow entry point with scheduled trigger)
-3. src/workflows/scheduled/daily-inventory-sync.ts (scheduled workflow)
-4. src/workflows/webhook/adhoc-inventory-sync.ts (webhook workflow)
-5. src/workflows/webhook/job-status-check.ts (status webhook)
-6. src/services/inventory-sync.service.ts (shared orchestration logic)
-7. src/types/inventory.types.ts (TypeScript type definitions)
-8. config/inventory.batch.json (mapping configuration - external JSON file)
-9. .env.example (environment variables)
-
-NOTE: Use external JSON files for mapping configuration (not TypeScript .config files)
-
-Ensure all code is production-ready with proper error handling, S3 dispose() in finally block. Polling is intentionally omitted (fire-and-forget).
-```
-
----
-
-## What You'll Build
-
-### Project Structure
-
-```
-s3-json-inventory-batch-sync/
-├── package.json
-├── index.ts                 # Workflow entry point
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-inventory-sync.ts   # Scheduled: Daily inventory sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-inventory-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── inventory-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    ├── config/
-    │   └── inventory.batch.json       # Mapping configuration (external JSON)
-    │
-    └── types/
-        └── inventory.types.ts        # TypeScript interfaces
-```
-
-### Features
-
-- ✅ Scheduled Versori workflow (daily/hourly inventory sync)
-- ✅ S3 file download with enhanced retry logic
-- ✅ JSON parsing with format auto-detection (standard JSON vs JSON Lines)
-- ✅ Direct field mapping (no special notation needed for JSON)
-- ✅ Field mapping with UniversalMapper and external JSON config
-- ✅ Batch API job creation and chunked processing (1000 records/batch)
-- ✅ BPP (Batch Pre-Processing) change detection
-- ✅ Fire-and-forget batch submission with audit logging
-- ✅ State management (VersoriFileTracker + JobTracker prevent duplicates)
-- ✅ File archival on S3 (success → processed/, failure → errors/)
-- ✅ Comprehensive error handling with exponential backoff retry
-- ✅ Modular architecture (reusable services, easy to test)
-
----
-
-## Versori Workflows Structure
-
-**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
-
-**Trigger Types:**
-- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
-- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
-- **`workflow()`** → Durable workflows (advanced, rarely used)
-
-**Execution Steps (chained to triggers):**
-- **`http()`** → External API calls (chained from schedule/webhook)
-- **`fn()`** → Internal processing (chained from schedule/webhook)
-
-### Recommended Project Structure
-
-```
-s3-json-inventory-batch-sync/
-├── index.ts                              # Entry point - exports all workflows
-└── src/
-    ├── workflows/
-    │   ├── scheduled/
-    │   │   └── daily-inventory-sync.ts   # Scheduled: Daily inventory sync
-    │   │
-    │   └── webhook/
-    │       ├── adhoc-inventory-sync.ts   # Webhook: Manual trigger
-    │       └── job-status-check.ts       # Webhook: Status query
-    │
-    ├── services/
-    │   └── inventory-sync.service.ts     # Shared orchestration logic (reusable)
-    │
-    └── types/
-        └── inventory.types.ts            # Shared type definitions
-```
-
-**Benefits:**
-- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
-- ✅ Descriptive file names (easy to browse and understand)
-- ✅ Scalable (add new workflows without cluttering)
-- ✅ Reusable code in `services/` (DRY principle)
-- ✅ Easy to modify individual workflows without affecting others
-
----
-
-## Workflow Files
-
-### 1. Scheduled Workflows (`src/workflows/scheduled/`)
-
-All time-based triggers that run automatically on cron schedules.
-
-#### `src/workflows/scheduled/daily-inventory-sync.ts`
-
-**Purpose**: Automatic daily inventory sync
-**Trigger**: Cron schedule (`0 2 * * *`)
-**Exposed as Endpoint**: ❌ NO - Runs automatically
-
-```typescript
-import { schedule, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { runIngestion } from '../../services/inventory-sync.service';
-
-/**
- * Scheduled Workflow: Daily Inventory Sync
- *
- * Runs automatically daily at 2 AM UTC
- * NOT exposed as HTTP endpoint - Versori executes on schedule
- *
- * Uses shared service: inventory-sync.service.ts
- */
-export const dailyInventorySync = schedule(
-  'inventory-batch-scheduled',
-  '0 2 * * *' // Daily at 2 AM UTC
-).then(
-  http('run-inventory-batch', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log, openKv } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `inventory-batch-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    log.info('═══════════════════════════════════════════════════════════════');
-    log.info('🚀 [WORKFLOW] Starting scheduled inventory sync', { jobId });
-    log.info('═══════════════════════════════════════════════════════════════');
-
-    await tracker.createJob(jobId, {
-      triggeredBy: 'schedule',
-      stage: 'initialization',
-      startTime: executionStartTime,
-    });
-
-    await tracker.updateJob(jobId, { status: 'processing' });
-
-    try {
-      // Reuse shared orchestration logic
-      const result = await runIngestion(ctx, jobId, tracker);
-
-      if (result.success) {
-        const duration = Date.now() - executionStartTime;
-        await tracker.markCompleted(jobId, { ...result, duration });
-        log.info('✅ [WORKFLOW] Inventory sync completed successfully', {
-          jobId,
-          filesProcessed: result.filesProcessed,
-          duration: `${duration}ms (${(duration / 1000).toFixed(2)}s)`,
-        });
-      } else {
-        await tracker.markFailed(jobId, result.error || 'Unknown error');
-        log.error('❌ [WORKFLOW] Inventory sync failed', {
-          jobId,
-          error: result.error,
-        });
-      }
-
-      log.info('═══════════════════════════════════════════════════════════════');
-      log.info('🏁 [WORKFLOW] Execution complete', {
-        jobId,
-        success: result.success,
-        duration: `${Date.now() - executionStartTime}ms`,
-      });
-      log.info('═══════════════════════════════════════════════════════════════');
-
-      return { success: true, jobId, ...result };
-    } catch (e: any) {
-      const errorMessage = e instanceof Error ? e.message : String(e);
-      await tracker.markFailed(jobId, errorMessage);
-      log.error('❌ [WORKFLOW] Inventory sync failed with exception', {
-        jobId,
-        error: errorMessage,
-        stack: e instanceof Error ? e.stack : undefined,
-        recommendation: 'Check SFTP credentials, network connectivity, and file permissions',
-      });
-      return { success: false, jobId, error: errorMessage };
-    }
-  })
-);
-```
-
----
-
-### 2. Webhook Workflows (`src/workflows/webhook/`)
-
-All HTTP-based triggers that create webhook endpoints.
-
-#### `src/workflows/webhook/adhoc-inventory-sync.ts`
-
-**Purpose**: Manual inventory sync trigger (on-demand)
-**Trigger**: HTTP POST
-**Endpoint**: `POST https://{workspace}.versori.run/inventory-batch-adhoc`
-**Use Cases**: Testing, priority processing, ad-hoc runs
-
-```typescript
-import { webhook, http } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-import { runIngestion } from '../../services/inventory-sync.service';
-
-/**
- * Webhook: Manual Inventory Sync Trigger
- *
- * Endpoint: POST https://{workspace}.versori.run/inventory-batch-adhoc
- * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5 }
- *
- * Pattern: Sync mode + fire-and-forget
- * - Returns jobId immediately
- * - Background processing continues without blocking response
- * - ✅ Works because Versori keeps execution context alive for unawaited promises
- *
- * Uses shared service: inventory-sync.service.ts
- */
-export const adhocInventorySync = webhook('inventory-batch-adhoc', {
-  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
-  connection: 'inventory-batch-adhoc', // Versori validates API key
-}).then(
-  http('run-inventory-batch-adhoc', { connection: 'fluent_commerce' }, async (ctx: any) => {
-    const { log, openKv, data } = ctx;
-    const executionStartTime = Date.now();
-    const jobId = `inventory-batch-adhoc-${Date.now()}`;
-    const tracker = new JobTracker(openKv(':project:'), log);
-
-    const filePattern = data?.filePattern as string || '*.json';
-    const maxFiles = data?.maxFiles as number;
-
-    log.info('🚀 [WEBHOOK] Adhoc inventory sync triggered', {
-      jobId,
-      filePattern,
-      maxFiles,
-      requestData: data,
-    });
-
-    // Create job entry FIRST (awaited to ensure job exists in KV)
-    await tracker.createJob(jobId, {
-      triggeredBy: 'manual',
-      stage: 'initialization',
-      status: 'queued',
-      startTime: executionStartTime,
-      options: { filePattern, maxFiles },
-      createdAt: new Date().toISOString(),
-    });
-
-    // ✅ Fire-and-forget: Start background processing WITHOUT await
-    // The promise continues execution after we return the response
-    runIngestion(ctx, jobId, tracker)
-      .then((result) => {
-        const duration = Date.now() - executionStartTime;
-        if (result.success) {
-          log.info('✅ [BACKGROUND] Inventory sync completed successfully', {
-            jobId,
-            filesProcessed: result.filesProcessed,
-            filesFailed: result.filesFailed,
-            recordsProcessed: result.recordsProcessed,
-            duration: `${duration}ms (${(duration / 1000).toFixed(2)}s)`,
-          });
-          return tracker.markCompleted(jobId, { ...result, duration });
-        } else {
-          log.error('❌ [BACKGROUND] Inventory sync failed', {
-            jobId,
-            error: result.error,
-          });
-          return tracker.markFailed(jobId, result.error || 'Unknown error');
-        }
-      })
-      .catch((error: unknown) => {
-        const errorMessage = error instanceof Error ? error.message : String(error);
-        const errorStack = error instanceof Error ? error.stack : undefined;
-
-        log.error('❌ [BACKGROUND] Inventory sync failed with exception', {
-          jobId,
-          error: errorMessage,
-          stack: errorStack,
-          errorType: error instanceof Error ? error.constructor.name : typeof error,
-          recommendation: 'Check S3 credentials, bucket permissions, and network connectivity',
-        });
-
-        return tracker.markFailed(jobId, errorMessage);
-      });
-
-    // Return immediately with jobId (response sent with this return value)
-    return {
-      success: true,
-      jobId,
-      message: 'Inventory sync started in background',
-      statusEndpoint: `https://{workspace}.versori.run/inventory-batch-job-status`,
-      note: 'Poll the status endpoint with jobId to check progress',
-    };
-  })
-);
-```
-
----
-
-#### `src/workflows/webhook/job-status-check.ts`
-
-**Purpose**: Query job status and progress
-**Trigger**: HTTP POST/GET
-**Endpoint**: `POST https://{workspace}.versori.run/inventory-batch-job-status`
-**Request Body**: `{ "jobId": "inventory-batch-1234567890" }`
-
-```typescript
-import { webhook, fn } from '@versori/run';
-import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
-
-/**
- * Webhook: Job Status Check
- *
- * Endpoint: POST https://{workspace}.versori.run/inventory-batch-job-status
- * Request body: { "jobId": "inventory-batch-1234567890" }
- *
- * Pattern: webhook().then(fn()) - no external API needed, only KV storage
- * Lightweight: Only queries KV store, no Fluent API calls
- */
-export const jobStatusCheck = webhook('inventory-batch-job-status', {
-  response: { mode: 'sync' },
-  connection: 'inventory-batch-job-status',
-}).then(
-  fn('status', async ctx => {
-    const { data, log, openKv } = ctx;
-    const jobId = data?.jobId as string;
-
-    if (!jobId) {
-      return { success: false, error: 'jobId required' };
-    }
-
-    const tracker = new JobTracker(openKv(':project:'), log);
-    const status = await tracker.getJob(jobId);
-
-    return status
-      ? { success: true, jobId, ...status }
-      : { success: false, error: 'Job not found', jobId };
-  })
-);
-```
-
----
-
-### 3. Entry Point (`index.ts`)
-
-**Purpose**: Register all workflows with Versori platform
-
-```typescript
-/**
- * Entry Point - Registers all workflows with Versori platform
- *
- * Versori automatically discovers and registers exported workflows
- *
- * File Structure:
- * - src/workflows/scheduled/ → Time-based triggers (cron)
- * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
- */
-
-// Scheduled workflows
-export { dailyInventorySync } from './src/workflows/scheduled/daily-inventory-sync';
-
-// Webhook workflows
-export { adhocInventorySync } from './src/workflows/webhook/adhoc-inventory-sync';
-export { jobStatusCheck } from './src/workflows/webhook/job-status-check';
-```
-
-**What Gets Exposed:**
-- ✅ `adhocInventorySync` → `https://{workspace}.versori.run/inventory-batch-adhoc`
-- ✅ `jobStatusCheck` → `https://{workspace}.versori.run/inventory-batch-job-status`
-- ❌ `dailyInventorySync` → NOT exposed (runs automatically on cron)
-
-## When to Use Batch API vs Event API
-
-### ✅ Use Batch API (`createJob`/`sendBatch`) For:
-
-| Entity Type      | Use Case              | Why Batch API                  |
-| --------------------- | ---------------------------------- | ----------------------------------------------- |
-| **Inventory**     | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
-| **InventoryQuantity** | Inventory positions and quantities | Native Batch API entity type          |
-
-### ❌ Use Event API Instead For:
-
-| Entity Type     | Use Case               | Why Event API / GraphQL            |
-| ------------------- | ------------------------------------- | --------------------------------------------- |
-| **Products**    | Product catalog sync, variant updates | Triggers workflows, validates business rules |
-| **Locations**    | Store/warehouse setup         | Requires workflow orchestration        |
-| **Customers**    | Customer registration/profile updates | Prefer GraphQL mutations (no Rubix support)  |
-| **Orders**     | Updates/events (not creation)     | Events fine for updates; creation via GraphQL |
-| **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support          |
-
-### 🔍 Use GraphQL Mutations For:
-
-| Scenario       | Why GraphQL              |
-| --------------------- | -------------------------------------- |
-| **Single operations** | Create one record, update one record  |
-| **Complex queries**  | Fetch data with relationships     |
-| **Testing/debugging** | Direct API control, immediate feedback |
-
-Note:
-
-- Orders: Use `createOrder` GraphQL mutation for order creation (this triggers the Order CREATED event in Rubix). Order updates/events are fine via Event API.
-- Customers: Use GraphQL mutations (no Rubix workflow support for customers).
-
----
-
-## Understanding BPP (Batch Pre-Processing)
-
-**BPP** is Fluent's change detection system that filters out unchanged records before workflow processing.
-
-Note on defaults and control:
-
-- BPP is a Fluent platform feature. The default on/off behavior is controlled at your Fluent account level, not by the SDK.
-- If you omit `meta.preprocessing` in `createJob()`, the account-level default applies.
-- To force behavior per job, set `meta.preprocessing` explicitly (`'skip'` to disable for delta feeds).
-
-### When to Use BPP (Default - Enabled)
-
-**✅ Full Inventory Snapshots:**
-
-- Daily complete inventory dumps
-- Entire warehouse stock files
-- Records may be identical to previous run
-
-**Example:**
-
-```typescript
-// BPP enabled by default - filters unchanged records
-const job = await client.createJob({
- name: 'Daily Full Inventory',
- retailerId: 'my-retailer',
- // BPP automatically enabled - no meta needed
-});
-```
-
-**What BPP does:**
-
-- Compares incoming records with existing records in Fluent
-- Filters out records with no changes
-- Only passes changed/new records to workflows
-- Significantly reduces workflow processing load
-
-### When to Skip BPP
-
-**✅ Delta Feeds (Pre-Filtered Data):**
-
-- Hourly change files (only updates since last run)
-- Event-driven inventory changes
-- Pre-filtered data from source system
-- All records are guaranteed to be changes
-
-**Example:**
-
-```typescript
-// Skip BPP for delta feeds - all records are changes
-const job = await client.createJob({
- name: 'Hourly Delta Inventory',
- retailerId: 'my-retailer',
- meta: {
-  preprocessing: 'skip', // All records already filtered
- },
-});
-```
-
-**Performance Impact:**
-
-- **Full snapshot + BPP**: 10,000 records → 500 changes → Fast
-- **Full snapshot, no BPP**: 10,000 records → 10,000 processed → Slow
-- **Delta feed + BPP**: 500 records → 500 changes → Fast (but BPP overhead wasted)
-- **Delta feed, no BPP**: 500 records → 500 processed → Fastest
-
-### Decision Guide
-
-| Source Data Type    | BPP Setting       | Reason                     |
-| ---------------------- | ----------------------- | ----------------------------------------------- |
-| **Daily full dump**  | Enabled (default)    | Most records unchanged, BPP filters efficiently |
-| **Hourly delta feed** | `preprocessing: 'skip'` | All records are changes, BPP overhead wasted  |
-| **Initial load**    | Enabled (default)    | No previous data, but good practice       |
-| **Manual corrections** | Enabled (default)    | Unknown change ratio, let BPP filter      |
-| **Real-time events**  | `preprocessing: 'skip'` | Each record is a change event          |
-
----
-
-## Processing Modes
-
-**⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
-
-The SDK supports three processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
-
-### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
-
-**When to use:**
-
-- Multiple large files that shouldn't be in memory together
-- Need file-level consistency (file 3 fails → files 1-2 already archived)
-- Memory constraints
-- Fault isolation (one file failure doesn't affect others)
-
-**Flow:**
-
-```
-FOR EACH FILE:
- 1. Download file
- 2. Parse JSON
- 3. Map records
- 4. Create Batch API job
- 5. Send batches
- 6. Write log
- 7. Archive file
- 8. Mark file processed
-
-IF FILE FAILS:
- - Move to /errors/
- - Continue to next file
- - Other files unaffected
-```
-
-**Configuration:**
-
-```json
-{
- "processingMode": "per-file",
- "maxFilesPerRun": 10
-}
-```
-
-**Benefits:**
-
-- ✅ File-level atomicity (each file processed independently)
-- ✅ Low memory footprint (1 file at a time)
-- ✅ Clear error isolation (failed file doesn't block others)
-- ✅ Incremental progress (files archived as completed)
-
-**Example implementation (see code section below)**
-
-### Mode 2: Batch Processing (All Files at Once) ⚠️ OPTIONAL - Choose if needed
-
-**When to use:**
-
-- Small files (all fit in memory comfortably)
-- Need one Batch API job for all files
-- Atomic processing (all files or none)
-
-**Flow:**
-
-```
-1. Download ALL files
-2. Parse ALL files
-3. Map ALL records
-4. Create ONE Batch API job
-5. Send ALL batches
-6. Write logs
-7. Archive ALL files
-8. Mark ALL processed
-```
-
-**Configuration:**
-
-```json
-{
- "processingMode": "batch",
- "maxFilesPerRun": 10
-}
-```
-
-**Benefits:**
-
-- ✅ Single job for all files (simplifies tracking)
-- ✅ Faster for many small files (parallel parsing possible)
-
-**Drawbacks:**
-
-- ❌ High memory usage (all files in memory)
-- ❌ All-or-nothing (one failure affects all)
-- ❌ No incremental progress
-
-### Mode 3: Chunked Per-File Processing (Balanced) ⚠️ OPTIONAL - Choose if needed
-
-**When to use:**
-
-- Many files (e.g., 100+ files)
-- Process N files at a time
-- Balance between memory and parallelism
-
-**Flow:**
-
-```
-CHUNK files into groups of N (e.g., 5 files per chunk):
-
-FOR EACH CHUNK:
- FOR EACH FILE in chunk:
-  1. Download file
-  2. Parse JSON
-  3. Map records
-  4. Create Batch API job
-  5. Send batches
-  6. Archive file
-
- Wait for chunk to complete before next chunk
-```
-
-**Configuration:**
-
-```json
-{
- "processingMode": "chunked",
- "fileChunkSize": 5,
- "maxFilesPerRun": 100
-}
-```
-
-**Benefits:**
-
-- ✅ Bounded memory usage (N files at a time)
-- ✅ Better throughput than per-file (parallel processing within chunk)
-- ✅ Incremental progress (per-chunk)
-
-**Use cases:**
-
-- High-volume file ingestion (100+ files per run)
-- Rate limiting (control API load)
-- Resource-constrained environments
-
-### Comparison Matrix
-
-| Aspect       | Per-File              | Batch             | Chunked                 |
-| ------------------- | ---------------------------------- | ----------------------------- | --------------------------------------- |
-| **Memory Usage**  | Low (1 file at a time)       | High (all files)       | Medium (N files)            |
-| **Consistency**   | Per-file atomic          | All-or-nothing        | Per-chunk atomic            |
-| **Fault Isolation** | ✅ Best (1 file fails → others OK) | ❌ Worst (1 fails → all fail) | ✅ Good (chunk fails → other chunks OK) |
-| **Performance**   | Slower (sequential)        | Fastest (parallel parsing)  | Balanced                |
-| **Batch API Jobs** | N jobs (1 per file)        | 1 job (all files)       | N jobs (1 per file)           |
-| **Use Case**    | Large files, strict consistency  | Small files, fast processing | Many files, balanced approach      |
-| **Recommended For** | Production (safest)        | Testing, small datasets    | High-volume production         |
-
-> **📋 This Template Implementation:**
-> 
-> **✅ This template implements ALL THREE modes** and selects based on `PROCESSING_MODE` variable.
-> 
-> **Choose ONE mode per connector:**
-> - **Default:** `PROCESSING_MODE=per-file` (recommended for production)
-> - **Alternative:** `PROCESSING_MODE=batch` (for small files, atomic processing)
-> - **Alternative:** `PROCESSING_MODE=chunked` (for high-volume scenarios)
-> 
-> **Important:** Do NOT use multiple modes in the same connector. Each connector should use ONE consistent pattern.
-
-### Decision Guide
-
-| Source Data  | File Count | File Size  | Recommended Mode | Reason                 |
-| -------------- | ---------- | ----------- | ---------------- | --------------------------------------- |
-| Daily snapshot | 1-10    | >10MB each | **Per-File**   | Memory efficient, fault isolation    |
-| Hourly deltas | 10-50   | <1MB each  | **Batch**    | Fast processing, small memory footprint |
-| Real-time feed | 100+    | <100KB each | **Chunked**   | Balanced throughput + memory      |
-| Initial load  | 1     | >100MB   | **Per-File**   | Memory safety              |
-| Testing    | 1-5    | Any     | **Batch**    | Simplicity               |
-
----
-
-## JSON File Format
-
-### Sample: inventory.json (Standard JSON)
-
-```json
-{
- "inventory": [
-  {
-   "locationRef": "LOC001",
-   "skuRef": "SKU-12345",
-   "qty": 100,
-   "type": "LAST_ON_HAND",
-   "status": "ACTIVE",
-   "expectedOn": "2025-01-25",
-   "attributes": {
-    "expiryDate": "2026-12-31",
-    "batchNumber": "BATCH-A001",
-    "condition": "NEW",
-    "storageZone": "ZONE-A"
-   }
-  },
-  {
-   "locationRef": "LOC001",
-   "skuRef": "SKU-67890",
-   "qty": 50,
-   "type": "LAST_ON_HAND",
-   "status": "ACTIVE",
-   "expectedOn": "2025-01-25",
-   "attributes": {
-    "expiryDate": "2026-06-30",
-    "batchNumber": "BATCH-A002",
-    "condition": "NEW",
-    "storageZone": "ZONE-B"
-   }
-  }
- ]
-}
-```
-
-### Sample: inventory.jsonl (JSON Lines)
-
-```jsonl
-{"locationRef":"LOC001","skuRef":"SKU-12345","qty":100,"type":"LAST_ON_HAND","status":"ACTIVE","expectedOn":"2025-01-25","attributes":{"expiryDate":"2026-12-31","batchNumber":"BATCH-A001","condition":"NEW","storageZone":"ZONE-A"}}
-{"locationRef":"LOC001","skuRef":"SKU-67890","qty":50,"type":"LAST_ON_HAND","status":"ACTIVE","expectedOn":"2025-01-25","attributes":{"expiryDate":"2026-06-30","batchNumber":"BATCH-A002","condition":"NEW","storageZone":"ZONE-B"}}
-```
-
-**JSON Structure:**
-
-- Standard JSON: Root object with nested arrays (`{ "inventory": [...] }`)
-- JSON Lines: One JSON object per line (streaming-friendly for large files)
-- JSONParserService auto-detects format based on content
-- Both formats support nested objects with dot notation
-
-**Field Descriptions:**
-
-- `locationRef`: Fluent location reference
-- `skuRef`: Fluent SKU/product reference
-- `qty`: Inventory quantity (integer)
-- `type`: Inventory type (LAST_ON_HAND for full snapshots, DELTA for incremental changes)
-- `status`: Record status (ACTIVE, INACTIVE)
-- `expectedOn`: Expected date (ISO 8601 or parseable format)
-- `attributes.expiryDate`: Product expiry date (optional)
-- `attributes.batchNumber`: Manufacturing batch/lot number (optional)
-- `attributes.condition`: Product condition (NEW, USED, DAMAGED)
-- `attributes.storageZone`: Warehouse zone identifier (optional)
-
-### JSON Field Mapping
-
-**IMPORTANT**: JSON uses direct field access (no special prefix needed):
-
-```json
-{
- "fields": {
-  "locationRef": { "source": "locationRef", "required": true },
-  "skuRef": { "source": "skuRef", "required": true }
- }
-}
-```
-
-**Why**: Unlike XML attributes (which need `@` prefix), JSON fields are accessed directly by name.
-
-### Array Handling (Standard JSON vs JSON Lines)
-
-**JSONParserService behavior:**
-
-- **Standard JSON**: Returns object or array based on structure
-- **JSON Lines**: Returns array of objects (one per line)
-
-**Solution**: Always normalize to array after parsing:
-
-```typescript
-const parsed = await parser.parse(jsonContent, { format: 'json' | 'jsonl' });
-
-// Extract inventory array - handle different JSON structures
-let records: any[] = [];
-if (format === 'jsonl') {
- records = Array.isArray(parsed) ? parsed : [parsed];
-} else {
- if (Array.isArray(parsed)) {
-  records = parsed; // Root level array
- } else if (parsed?.inventory && Array.isArray(parsed.inventory)) {
-  records = parsed.inventory; // { "inventory": [...] }
- } else if (parsed?.records && Array.isArray(parsed.records)) {
-  records = parsed.records; // { "records": [...] }
- } else {
-  records = [parsed]; // Single object
- }
-}
-```
-
----
-
-## Service Implementation
-
-### File: `src/services/inventory-sync.service.ts`
-
-**Complete implementation showing all SDK patterns:**
-
-```typescript
-import { Buffer } from 'node:buffer';  // ✅ Required for Versori/Deno runtime
-import {
-  createClient,
-  S3DataSource,
-  JSONParserService,
-  UniversalMapper,
-  VersoriFileTracker,
-  JobTracker,
-  extractFileName,
-} from '@fluentcommerce/fc-connect-sdk';
-import type { FileMetadata } from '@fluentcommerce/fc-connect-sdk';
-import inventoryMapping from '../config/inventory.batch.json' with { type: 'json' };  // ✅ External JSON import
-
-/**
- * Shared inventory ingestion orchestration logic
- * Used by both scheduled and webhook workflows
- *
- * @param ctx - Versori HTTP context
- * @param jobId - Unique job identifier
- * @param tracker - JobTracker instance
- */
-export async function runIngestion(
-  ctx: any,
-  jobId: string,
-  tracker: JobTracker
-) {
-  const { log, activation, openKv } = ctx;
-  const startTime = Date.now();
-  let s3: S3DataSource | null = null;
-
-  // Optional feature toggles
-  const enableFileTracking = activation.getVariable('enableFileTracking') !== 'false';
-  const detailedLogging = activation.getVariable('detailedLogging') === 'true';
-
-  try {
-    // ========================================
-    // CLIENT INITIALIZATION (with retailerId and validation)
-    // ========================================
-    log.info('🔧 [InventorySync] Initializing Fluent Commerce client...');
-
-    const client = await createClient({
-      ...ctx,
-      validateConnection: true,  // ✅ Validate connection on creation
-    });
-
-    // ✅ CRITICAL: Set retailerId for Batch API
-    const retailerId = activation.getVariable('fluentRetailerId');
-    if (!retailerId) {
-      log.error('❌ [InventorySync] Missing required fluentRetailerId activation variable');
-      throw new Error('fluentRetailerId activation variable is required for Batch API');
-    }
-    client.setRetailerId(retailerId);
-
-    log.info('✅ [InventorySync] Client initialized and validated', { retailerId });
-
-    // ========================================
-    // S3 DATA SOURCE INITIALIZATION
-    // ========================================
-    log.info('🔧 [InventorySync] Initializing S3 data source...');
-
-    const s3Config = {
-      bucket: activation.getVariable('s3BucketName'),
-      region: activation.getVariable('awsRegion') || 'us-east-1',
-      accessKeyId: activation.getVariable('awsAccessKeyId'),
-      secretAccessKey: activation.getVariable('awsSecretAccessKey'),
-      prefix: activation.getVariable('s3Prefix') || 'inventory/',
-    };
-
-    s3 = new S3DataSource(
-      {
-        type: 'S3_JSON',
-        connectionId: 's3-inventory-sync',
-        name: 'inventory-sync',
-        s3Config,
-      },
-      log
-    );
-
-    log.info('✅ [InventorySync] S3 data source initialized', {
-      bucket: s3Config.bucket,
-      region: s3Config.region,
-      prefix: s3Config.prefix,
-    });
-
-    // ========================================
-    // SERVICE INITIALIZATION
-    // ========================================
-    const parser = new JSONParserService();
-    const mapper = new UniversalMapper(inventoryMapping);
-    const kv = openKv(':project:');
-    const fileTracker = enableFileTracking
-      ? new VersoriFileTracker(kv, 's3-json-inventory-sync')
-      : null;
-    const bppEnabled = activation.getVariable('bppEnabled') !== 'false';
-
-    if (detailedLogging) {
-      log.info('🔍 [InventorySync] Configuration loaded', {
-        bppEnabled,
-        enableFileTracking,
-        detailedLogging,
-      });
-    }
-
-    // ========================================
-    // FILE DISCOVERY
-    // ========================================
-    log.info('🔍 [InventorySync] Discovering files on S3...');
-
-    const files = await s3.listFiles({ prefix: s3Config.prefix });
-    log.info('📄 [InventorySync] File discovery complete', { count: files.length });
-
-    if (files.length === 0) {
-      log.info('⚠️ [InventorySync] No files found to process');
-      return { success: true, message: 'No files to process', filesProcessed: 0 };
-    }
-
-    // ========================================
-    // FILE PROCESSING
-    // ========================================
-    const results = [];
-    let fileIndex = 0;
-
-    for (const file of files) {
-      fileIndex++;
-      const fileStartTime = Date.now();
-      const fileName = extractFileName(file.path);
-
-      log.info(`📄 [FILE ${fileIndex}/${files.length}] Processing: ${fileName}`);
-
-      try {
-        // Check if already processed
-        if (fileTracker && (await fileTracker.wasFileProcessed(file.name))) {
-          log.info(`⏭️ [FILE ${fileIndex}/${files.length}] Skipping already processed: ${fileName}`);
-          results.push({ fileName, success: true, skipped: true });
-          continue;
-        }
-
-        // Download and parse
-        if (detailedLogging) {
-          log.info(`🔍 [FILE ${fileIndex}/${files.length}] Downloading: ${fileName}`);
-        }
-        const jsonContent = (await s3.downloadFile(file.path, { encoding: 'utf8' })) as string;
-        const parsed = await parser.parse(jsonContent);
-
-        // Extract inventory array
-        const records = Array.isArray(parsed) ? parsed :
-                       parsed?.inventory ? parsed.inventory :
-                       [parsed];
-
-        if (records.length === 0) {
-          log.warn(`⚠️ [FILE ${fileIndex}/${files.length}] No records found in: ${fileName}`);
-          continue;
-        }
-
-        log.info(`🔄 [FILE ${fileIndex}/${files.length}] Transforming ${records.length} records`);
-
-        // Transform records
-        const mappedRecords = [];
-        const aggregatedSkippedFields: string[] = [];
-        for (const rec of records) {
-          const sourceDataWithContext = { ...rec, $context: { retailerId } };
-          const mappingResult = await mapper.map(sourceDataWithContext);
-          if (mappingResult.success) {
-            mappedRecords.push(mappingResult.data);
-          }
-          // Aggregate skipped fields
-          if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
-            for (const fieldName of mappingResult.skippedFields) {
-              if (!aggregatedSkippedFields.includes(fieldName)) {
-                aggregatedSkippedFields.push(fieldName);
-              }
-            }
-          }
-        }
-
-        if (aggregatedSkippedFields.length > 0) {
-          log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
-            file: fileName,
-            skippedFields: aggregatedSkippedFields,
-            note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
-          });
-        }
-
-        if (mappedRecords.length === 0) {
-          log.warn(`⚠️ [FILE ${fileIndex}/${files.length}] No valid records after mapping: ${fileName}`, {
-            recommendation: 'Check mapping configuration and required fields',
-          });
-          continue;
-        }
-
-        // Create job and send batches
-        log.info(`📦 [FILE ${fileIndex}/${files.length}] Creating Batch API job for ${mappedRecords.length} records`);
-
-        const job = await client.createJob({
-          name: `s3-json-${fileName}-${Date.now()}`,
-          meta: bppEnabled ? undefined : { preprocessing: 'skip' },
-        });
-
-        await tracker.updateJob(jobId, {
-          status: 'processing',
-          details: { fileName, recordCount: mappedRecords.length },
-        });
-
-        // ? Enhanced: Extract context for progress logging
-        const uniqueLocations = [...new Set(mappedRecords.map((r: any) => r.locationRef))];
-        const sampleSKUs = mappedRecords.slice(0, 5).map((r: any) => r.skuRef);
-
-        // ? Enhanced: Start logging with context
-        log.info(`📤 [BatchProcessor] Sending batch for file "${fileName}"`, {
-          totalRecords: mappedRecords.length,
-          locations: uniqueLocations.join(', '),
-          sampleSKUs: sampleSKUs.join(', '),
-          jobId: job.id
-        });
-
-        // Send batches (fire-and-forget)
-        await client.sendBatch(job.id, {
-          action: 'UPSERT',
-          entityType: 'INVENTORY',
-          entities: mappedRecords,
-        });
-
-        // ? Enhanced: Completion logging
-        log.info(`✅ [BatchProcessor] Batch sent successfully for file "${fileName}"`, {
-          totalRecords: mappedRecords.length,
-          jobId: job.id,
-          duration: Date.now() - fileStartTime
-        });
-
-        // ? Enhanced: Clear large arrays after processing to free memory
-        mappedRecords.length = 0;
-
-        // Mark as processed
-        if (fileTracker) {
-          await fileTracker.markFileProcessed(file.name, {
-            recordCount: mappedRecords.length,
-          });
-        }
-
-        const fileDuration = Date.now() - fileStartTime;
-        results.push({
-          fileName,
-          success: true,
-          recordCount: mappedRecords.length,
-          duration: fileDuration,
-        });
-
-        log.info(`✅ [FILE ${fileIndex}/${files.length}] Successfully processed: ${fileName}`, {
-          records: mappedRecords.length,
-          jobId: job.id,
-          duration: `${fileDuration}ms (${(fileDuration / 1000).toFixed(2)}s)`,
-        });
-
-      } catch (error: any) {
-        const fileDuration = Date.now() - fileStartTime;
-        log.error(`❌ [FILE ${fileIndex}/${files.length}] Processing failed: ${fileName}`, {
-          error: error?.message,
-          stack: error?.stack,
-          duration: `${fileDuration}ms`,
-          recommendation: 'Check S3 permissions, file format, and mapping configuration',
-        });
-        results.push({ fileName, success: false, error: error?.message, duration: fileDuration });
-      }
-    }
-
-    const totalDuration = Date.now() - startTime;
-    const successCount = results.filter((r) => r.success && !r.skipped).length;
-
-    log.info('✅ [InventorySync] Batch processing complete', {
-      total: files.length,
-      processed: successCount,
-      skipped: results.filter((r) => r.skipped).length,
-      failed: results.filter((r) => !r.success).length,
-      duration: `${totalDuration}ms (${(totalDuration / 1000).toFixed(2)}s)`,
-    });
-
-    return {
-      success: true,
-      results,
-      filesProcessed: successCount,
-      duration: totalDuration,
-    };
-
-  } catch (error: any) {
-    const totalDuration = Date.now() - startTime;
-    log.error('❌ [InventorySync] Fatal error during ingestion', {
-      message: error?.message,
-      stack: error?.stack,
-      duration: `${totalDuration}ms (${(totalDuration / 1000).toFixed(2)}s)`,
-      recommendation: 'Check activation variables, S3 credentials, and Fluent API connectivity',
-    });
-    throw error;  // Re-throw for workflow error handling
-  } finally {
-    // ✅ CRITICAL: Always dispose S3 connection
-    if (s3) {
-      try {
-        await s3.dispose();
-        log.info('✅ [InventorySync] S3 connection disposed successfully');
-      } catch (disposeError: any) {
-        log.error('⚠️ [InventorySync] Failed to dispose S3 connection', {
-          error: disposeError?.message,
-        });
-      }
-    }
-  }
-}
-```
-
-**Key Patterns Demonstrated:**
-- ✅ Buffer import for Versori/Deno compatibility
-- ✅ External JSON mapping import with `{ type: 'json' }`
-- ✅ `validateConnection: true` for client initialization
-- ✅ `setRetailerId()` call after client creation (REQUIRED for Batch API)
-- ✅ Emoji-based logging (🚀 ✅ ❌ ⚠️ 🔍 📄) for visual log scanning
-- ✅ Execution boundaries with ═══ separators
-- ✅ `extractFileName()` usage for clean file names
-- ✅ Optional feature toggles (enableFileTracking, detailedLogging)
-- ✅ Duration tracking at file and workflow levels (ms and seconds)
-- ✅ Recommendations in error logs for actionable debugging
-- ✅ S3 `dispose()` in finally block with error handling
-- ✅ VersoriFileTracker for state management (optional via toggle)
-- ✅ JobTracker for job lifecycle tracking
-
----
-
-## Code Flow Explanation
-
-### Initialization Phase
-
-**Logger Setup:**
-
-```typescript
-// ✅ CORRECT: Use native Versori log from context
-const { log } = ctx;
-log.info('Starting workflow');
-
-// ❌ WRONG: LoggingService (removed - use native log on Versori)
-// import { LoggingService } from '@fluentcommerce/fc-connect-sdk';
-// const logging = new LoggingService();
-// const log = logging.createLogger({ logLevel: 'info' });
-```
-
-- Versori provides `log` in the context - use it directly
-
-**Client Creation:**
-
-```typescript
-// Pass entire Versori context object
-const client = await createClient(ctx);
-```
-
-- `createClient(ctx)` accepts entire Versori context (fetch, connections, log, activation)
-- Auto-detects Versori platform from context
-- Returns `FluentClient` configured with OAuth2 from connections.fluent_commerce
-- OAuth2 authentication handled automatically
-- This is the **CORRECT** pattern for Versori workflows
-
-**S3 Data Source:**
-
-```typescript
-const s3 = new S3DataSource(
- {
-  type: 'S3_JSON', // Type indicates JSON files (metadata only)
-  connectionId: 's3-inventory-sync', // Unique identifier (required)
-  name: 'inventory-sync', // Human-readable name (required)
-  s3Config: {
-   bucket: 'my-inventory-bucket',
-   region: 'us-east-1',
-   accessKeyId: 'AKIAXXXX',
-   secretAccessKey: 'xxxxxxxx',
-  },
- },
- log
-);
-```
-
-- **Constructor params:** `(config: S3DataSourceConfig, log)` - Versori native log, no explicit typing needed
-- `connectionId` and `name` are required in config
-- `type` must be `'S3_JSON'` for JSON files
-- Enhanced retry logic with exponential backoff
-
-### File Discovery Phase
-
-**List Files:**
-
-```typescript
-const files = await s3.listFiles({
- prefix: config.s3.prefix, // Override config (optional)
- maxKeys: 1000, // Maximum files to retrieve
-});
-```
-
-- Returns `FileMetadata[]` with: `name`, `lastModified`, `size`, `path`, `source`
-- Files sorted by modification time (newest first)
-- Directories excluded automatically
-
-**State Check:**
-
-```typescript
-const alreadyProcessed = await fileTracker.wasFileProcessed(file.name);
-if (alreadyProcessed) {
- continue; // Skip already processed files
-}
-```
-
-- Uses Versori KV store (`openKv()`) for distributed state
-- Prevents duplicate processing across workflow runs
-- State persists between executions
-
-### File Processing Phase
-
-**Download File:**
-
-```typescript
-const jsonContent = (await s3.downloadFile(file.path, {
- encoding: 'utf8',
-})) as string;
-```
-
-- **Important:** Cast to `string` when `encoding` is specified
-- Without encoding, returns `Buffer`
-- Supports streaming for large files
-
-**Parse JSON:**
-
-```typescript
-const jsonFormat = activation.getVariable('jsonFormat') || 'json';
-const parsed = await parser.parse(jsonContent, { format: jsonFormat });
-
-// Extract inventory array - handle different JSON structures
-let records: any[] = [];
-if (jsonFormat === 'jsonl') {
- records = Array.isArray(parsed) ? parsed : [parsed];
-} else {
- if (Array.isArray(parsed)) {
-  records = parsed; // Root level array
- } else if (parsed?.inventory && Array.isArray(parsed.inventory)) {
-  records = parsed.inventory; // { "inventory": [...] }
- } else {
-  records = [parsed]; // Single object
- }
-}
-```
-
-- Parses JSON into JavaScript object or array
-- **Format detection**: Auto-detects standard JSON vs JSON Lines
-- Always normalize to array for consistent processing
-- Throws `ParsingError` on invalid JSON
-
-**JSON Field Handling:**
-
-- Direct field access by name (no special prefix needed)
-- Example: `{ "locationRef": "LOC001" }` → `{ locationRef: 'LOC001' }`
-- Use simple paths like `"locationRef"` in mapping config
-
-**Transform Data:**
-
-```typescript
-const mapper = new UniversalMapper(inventoryMapping);
-
-// Map each record directly (no wrapper needed for JSON)
-const mappingResult = await mapper.map(rec);
-
-if (!mappingResult.success) {
- // Mapping validation failed for required fields
- log.warn('Mapping failed:', mappingResult.errors);
- continue; // Skip invalid records
-}
-
-mappedRecords.push(mappingResult.data);
-```
-
-- Direct mapping (no need to wrap in object like XML)
-- `MappingResult`: `{ success: boolean, data: any, errors?: string[] }`
-- `success: false` only if required fields fail
-- Optional field errors reported but don't fail mapping
-
-### Batch API Phase
-
-**Create Job:**
-
-```typescript
-// Full snapshot (BPP enabled by default)
-const job = await client.createJob({
- name: 'Daily Full Inventory',
- retailerId: 'my-retailer',
- // BPP automatically enabled - filters unchanged records
-});
-
-// Delta feed (skip BPP)
-const job = await client.createJob({
- name: 'Hourly Deltas',
- retailerId: 'my-retailer',
- meta: {
-  preprocessing: 'skip', // All records are changes
- },
-});
-```
-
-- Returns job object with `id` and metadata
-- `retailerId` required for tenant isolation
-- BPP enabled by default, skip with `meta: { preprocessing: 'skip' }`
-
-**Send Batches:**
-
-```typescript
-const batch = await client.sendBatch(jobId, {
- action: 'UPSERT', // Most common: create or update
- entityType: 'INVENTORY', // Entity type for inventory
- source: 'S3_JSON', // Data source identifier (optional)
- event: 'InventoryChanged', // Valid Rubix event (optional, defaults to InventoryChanged)
- entities: chunk, // Array of inventory records
-});
-```
-
-- `action`: `'UPSERT'` (currently the only action supported by Fluent Batch API)
-- `entityType`: `'INVENTORY'` for inventory records
-- `entities`: Array of transformed inventory objects
-- Returns batch object with `id` for tracking
-
-**Fire-and-Forget Pattern:**
-
-```typescript
-const batch = await client.sendBatch(jobId, { ... });
-log.info('Batch sent (fire-and-forget)', { batchId: batch.id });
-
-// Record batch details for audit trail
-result.batches.push({
- batchId: batch.id,
- recordCount: chunk.length,
- timestamp: new Date().toISOString(),
- status: 'SENT',
-});
-```
-
-- No polling - batches sent and tracked immediately
-- Batch details recorded for audit logging
-- Log files written to S3 for tracking (optional, configurable)
-
-### Cleanup Phase
-
-**Archive File:**
-
-```typescript
-await s3.moveFile(
- file.path,
- `${config.s3.archivePrefix}${file.name}`
-);
-```
-
-- Moves file on S3 (atomic operation)
-- Creates destination prefix if needed
-- Original file removed after successful move
-
-**Mark Processed:**
-
-```typescript
-await fileTracker.markFileProcessed(file.name, {
- recordCount: batchResults.totalSent,
- batchCount: batchResults.batchCount,
- jobId: job.id,
-});
-```
-
-- Stores metadata in Versori KV store
-- Metadata optional but useful for audit trails
-- Prevents reprocessing in future runs
-
-**Dispose Resources:**
-
-```typescript
-await s3.dispose();
-```
-
-- Releases S3 connection pool
-- Should be called at end of workflow in `finally` block
-- Ensures proper cleanup
-
----
-
-## Versori Activation Variables
-
-Configure in Versori platform settings:
-
-```bash
-# S3 Configuration
-S3_BUCKET_NAME=my-inventory-bucket
-AWS_REGION=us-east-1
-AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXXX
-AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-
-# S3 Paths
-S3_PREFIX=inventory/
-ARCHIVE_PREFIX=processed/
-ERROR_PREFIX=errors/
-LOG_PREFIX=logs/
-
-# File Processing
-FILE_PATTERN=.json
-JSON_FORMAT=json # or 'jsonl' for JSON Lines
-MAX_FILES=10
-
-# Processing Mode Configuration
-# ⚠️ IMPORTANT: Choose ONE mode per connector (these are alternatives, not used together)
-# Options: "per-file" (default, recommended), "batch", "chunked"
-PROCESSING_MODE=per-file
-# For chunked mode only: number of files to process per chunk (default: 5)
-FILE_CHUNK_SIZE=5
-
-# Batch Log Configuration
-LOG_ENABLED=true          # Enable/disable batch log writing (default: true)
-LOG_FORMAT=json           # Log format: json or text (default: json)
-
-# Fluent Configuration (via Versori connection)
-# Connection: fluent_commerce (OAuth2)
-FLUENT_RETAILER_ID=my-retailer
-
-# Batch Processing
-BATCH_SIZE=1000
-
-# BPP Configuration
-# true = Full snapshot (BPP filters unchanged records - DEFAULT)
-# false = Delta feed (skip BPP, all records are changes)
-BPP_ENABLED=true
-
-# Optional Feature Toggles (NEW)
-ENABLE_FILE_TRACKING=true    # Track processed files to prevent duplicates (default: true)
-DETAILED_LOGGING=false       # Enable verbose logging for debugging (default: false)
-```
-
----
-
-## Batch API Payload Example
-
-What gets sent to Fluent Commerce Batch API:
-
-```json
-{
- "action": "UPSERT",
- "entityType": "INVENTORY",
- "source": "S3_JSON",
- "event": "InventoryChanged",
- "entities": [
-  {
-   "retailerId": 1,
-   "locationRef": "LOC001",
-   "skuRef": "SKU-12345",
-   "qty": 100,
-   "type": "LAST_ON_HAND",
-   "status": "ACTIVE",
-   "expectedOn": "2025-01-25T00:00:00.000Z",
-   "attributes": {
-    "expiryDate": "2026-12-31T00:00:00.000Z",
-    "batchNumber": "BATCH-A001",
-    "condition": "NEW",
-    "storageZone": "ZONE-A"
-   }
-  },
-  {
-   "retailerId": 1,
-   "locationRef": "LOC001",
-   "skuRef": "SKU-67890",
-   "qty": 50,
-   "type": "LAST_ON_HAND",
-   "status": "ACTIVE",
-   "expectedOn": "2025-01-25T00:00:00.000Z",
-   "attributes": {
-    "expiryDate": "2026-06-30T00:00:00.000Z",
-    "batchNumber": "BATCH-A002",
-    "condition": "NEW",
-    "storageZone": "ZONE-B"
-   }
-  }
- ]
-}
-```
-
-> **⚠️ CRITICAL:** Each entity MUST include `retailerId` - it's required in the entity payload, not just in createJob()
-
----
-
-## Versori Deployment
-
-Use the Versori CLI for deploy and operations:
-
-```bash
-# Install dependencies
-npm install
-
-# Deploy to Versori
-versori deploy
-
-# View logs
-versori logs s3-json-inventory-batch-sync
-
-# Trigger manual run (if defined)
-versori run ingest-now
-```
-
----
-
-## Testing
-
-### Test Scheduled Batch
-
-Upload a test JSON file to S3 incoming prefix and wait for the scheduled run.
-
-**Check logs:**
-
-```
-[STEP 1/8] Initializing job tracking
-[STEP 2/8] Initializing Fluent Commerce client and S3
-[STEP 3/8] Discovering files on S3
-[FILE 1/1] Processing file: inventory_20250124.json
-[STEP 4/8] Downloading and parsing: inventory_20250124.json
-[STEP 5/8] Transforming 5000 inventory records from inventory_20250124.json
-[STEP 6/8] Creating batch job and sending 5 batches to Fluent Commerce
-[STEP 7/8] Archiving file: inventory_20250124.json
-[STEP 8/8] Completing job and calculating totals
-```
-
-### Test Ad hoc Batch
-
-```bash
-# Process all pending files
-curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
- -H "Content-Type: application/json" \
- -d '{}'
-
-# Process specific pattern
-curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
- -H "Content-Type: application/json" \
- -d '{
-  "filePattern": "urgent_*.json"
- }'
-
-# Force reprocess
-curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
- -H "Content-Type: application/json" \
- -d '{
-  "forceReprocess": true,
-  "filePattern": "inventory_20250124.json"
- }'
-```
-
-### Test Job Status Query
-
-```bash
-curl -X POST https://api.versori.com/webhooks/inventory-batch-job-status \
- -H "Content-Type: application/json" \
- -d '{
-  "jobId": "ADHOC_INV_20251024_183045_abc123"
- }'
-```
-
-### Verify Batch Job in Fluent
-
-After processing, check the Batch job status in Fluent Commerce:
-
-```bash
-# Upload test file to S3
-aws s3 cp inventory-test.json s3://my-bucket/inventory/
-
-# Query job status via GraphQL
-curl -X POST https://your-fluent-instance.com/graphql \
- -H "Authorization: Bearer YOUR_TOKEN" \
- -H "Content-Type: application/json" \
- -d '{
-  "query": "query { job(id: \"job-123456\") { id status recordCount processedCount } }"
- }'
-```
-
----
-
-## Monitoring
-
-### Success Response
-
-```json
-{
- "success": true,
- "filesProcessed": 1,
- "filesSkipped": 0,
- "filesFailed": 0,
- "results": [
-  {
-   "file": "inventory_2025-01-22.json",
-   "success": true,
-   "recordCount": 5000,
-   "batchCount": 5,
-   "jobId": "job-123456",
-   "duration": 12345
-  }
- ],
- "duration": 13456
-}
-```
-
-### Error Response
-
-```json
-{
- "success": false,
- "filesProcessed": 0,
- "filesFailed": 1,
- "results": [
-  {
-   "file": "inventory_2025-01-22.json",
-   "success": false,
-   "error": "No valid records after mapping"
-  }
- ],
- "duration": 876
-}
-```
-
----
-
-## Common Pitfalls and Solutions
-
-### 1. JSON Format Detection
-
-❌ **Wrong:**
-
-```typescript
-// Assuming format without checking
-const records = parsed.inventory;
-```
-
-✅ **Correct:**
-
-```typescript
-// Handle both standard JSON and JSON Lines
-let records: any[] = [];
-if (format === 'jsonl') {
- records = Array.isArray(parsed) ? parsed : [parsed];
-} else {
- if (Array.isArray(parsed)) records = parsed;
- else if (parsed?.inventory) records = parsed.inventory;
- else records = [parsed];
-}
-```
-
-**Why:** JSON files can have different structures; always normalize.
-
-### 2. JSON Field Mapping
-
-❌ **Wrong:**
-
-```json
-{
- "fields": {
-  "locationRef": { "source": "@locationRef" }
- }
-}
-```
-
-✅ **Correct:**
-
-```json
-{
- "fields": {
-  "locationRef": { "source": "locationRef" }
- }
-}
-```
-
-**Why:** JSON uses direct field access (no `@` prefix like XML attributes).
-
-### 3. Mapping Data Structure
-
-❌ **Wrong:**
-
-```typescript
-// Wrapping JSON record unnecessarily
-const mappingResult = await mapper.map({ inventory: rec });
-```
-
-✅ **Correct:**
-
-```typescript
-// Map JSON record directly
-const mappingResult = await mapper.map(rec);
-```
-
-**Why:** Unlike XML, JSON records don't need wrapper objects for mapping.
-
-### 4. BPP Configuration
-
-❌ **Wrong:**
-
-```typescript
-// Using BPP for delta feeds (wastes processing time)
-const job = await client.createJob({
- name: 'Hourly Deltas',
- retailerId: 'my-retailer',
- // BPP enabled by default - unnecessary for delta feeds
-});
-```
-
-✅ **Correct:**
-
-```typescript
-// Skip BPP for delta feeds (all records are changes)
-const job = await client.createJob({
- name: 'Hourly Deltas',
- retailerId: 'my-retailer',
- meta: {
-  preprocessing: 'skip', // Skip BPP - all records are changes
- },
-});
-```
-
-**Why:** Delta feeds are pre-filtered by source system, BPP overhead is wasted.
-
-### 5. Large JSON Files
-
-❌ **Wrong:**
-
-```typescript
-// Loading entire 500MB JSON into memory
-const parsed = await parser.parse(largeJsonContent);
-```
-
-✅ **Correct:**
-
-```typescript
-// Use JSON Lines format for streaming
-const format = 'jsonl';
-const parsed = await parser.parse(largeJsonContent, { format });
-```
-
-**Why:** JSON Lines processes one record at a time, preventing memory issues.
-
----
-
-## Key Takeaways
-
-- 🎯 **Use Batch API for inventory** - Not Event API (Event API is for products/orders/etc.)
-- 🎯 **Processing mode selection** - Per-file (default) for safety, batch for speed, chunked for scale
-- 🎯 **TRUE modular architecture** - Separate service files with clear responsibilities (see Modular Structure section)
-- 🎯 **BPP by default** - Enabled for full snapshots, skip for delta feeds
-- 🎯 **Format detection** - Auto-detects standard JSON vs JSON Lines
-- 🎯 **Direct field access** - Use simple paths like `"locationRef"` (no special notation)
-- 🎯 **Map records directly** - No need to wrap in object (unlike XML)
-- 🎯 **Chunk records** - Default 1000 per batch, tune based on record size
-- 🎯 **EntityType: INVENTORY** - Correct entity type for inventory records
-- 🎯 **State management** - VersoriFileTracker + JobTracker prevent duplicates
-- 🎯 **Enhanced retry logic** - S3 operations with exponential backoff
-- 🎯 **Always dispose** - Release S3 resources with `dispose()` in finally block
-- 🎯 **Error handling** - Move failed files to error folder, don't fail entire workflow
-- 🎯 **Native logging** - Use `log` from context on Versori platform
-- 🎯 **Streaming for large files** - Use JSON Lines format for files >100MB
-- 🎯 **Emoji logging** - Use 🚀 ✅ ❌ ⚠️ 🔍 📄 for visual log scanning
-- 🎯 **Execution boundaries** - Use ═══ separators for workflow start/end
-- 🎯 **validateConnection** - Enable connection validation on client creation
-- 🎯 **Optional toggles** - Support enableFileTracking and detailedLogging flags
-- 🎯 **Duration tracking** - Track execution time at file and workflow levels
-- 🎯 **Error recommendations** - Include actionable suggestions in error logs
-
----
-
-## Related Documentation
-
-- [Batch API vs Event API Decision Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
-- [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
-- [S3 Data Source](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md#s3-data-source)
-- [JSON Parser](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md#json-parser-service)
-- [State Management](../../../../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md)
-- [Job Tracker](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md)
-- [BPP Documentation](../../../../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md#batch-pre-processing-bpp-change-detection)
-
----
-
-[→ Back to Versori Scheduled Workflows](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) | [Versori Platform Guide →](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+---
+template_id: tpl-ingest-s3-json-inventory-batch
+canonical_filename: template-ingestion-s3-json-inventory-batch.md
+version: 2.0.0
+sdk_version: ^0.1.39
+runtime: versori
+direction: ingestion
+source: s3-json
+destination: fluent-batch-api
+entity: inventory
+format: json
+logging: versori
+status: stable
+features:
+  - batch-api-integration
+  - memory-management
+  - enhanced-logging
+  - attribute-transformation
+---
+
+# Template: Ingestion - S3 JSON Inventory to Batch API
+
+**FC Connect SDK Use Case Guide**
+
+> **SDK**: [@fluentcommerce/fc-connect-sdk](https://www.npmjs.com/package/@fluentcommerce/fc-connect-sdk)
+> **Version**: @fluentcommerce/fc-connect-sdk@^0.1.39
+
+**🆕 Production Code Enhancements (Applied):**
+1. ✅ Batch API with retry logic and BPP change detection
+2. ✅ Memory management (clearing large arrays after batch processing)
+3. ✅ Enhanced logging with emoji progress tracking (📦 batch creation, 📤 batch sending, ✅ completion)
+4. ✅ Attribute transformation with nested field support
+
+**Template Version:** 2.0.0
+**Last Updated:** 2025-01-24
+
+**Context**: Versori scheduled workflow that reads inventory JSON files from S3, transforms data with UniversalMapper, and sends bulk inventory updates to Fluent Commerce Batch API with BPP change detection
+
+**Complexity**: Medium
+
+**Runtime**: Versori Platform
+
+**Estimated Lines**: ~520 lines
+
+---
+
+## STEP 1: Understand This Template
+
+**What This Template Does:**
+
+- Scheduled Versori workflow for bulk inventory ingestion from S3 JSON files
+- Reads JSON files from S3 with retry logic and streaming support
+- Parses JSON with format detection (standard JSON vs JSON Lines)
+- Transforms data using UniversalMapper with direct field access (no special notation)
+- Sends bulk updates to Fluent Commerce Batch API with chunking
+- Uses BPP (Batch Pre-Processing) for change detection
+- Tracks processing state with VersoriFileTracker to prevent duplicates
+- Archives processed files and handles errors gracefully
+
+**Key SDK Components:**
+
+- `createClient()` - Universal client factory (auto-detects Versori context)
+- `S3DataSource` - S3 operations with streaming (NEW: enhanced retry logic)
+- `JSONParserService` - JSON parsing with format auto-detection (JSON vs JSON Lines)
+- `UniversalMapper` - Field transformation with SDK resolvers
+- `VersoriFileTracker` - State management (prevent duplicate processing)
+- `JobTracker` - Job lifecycle tracking
+- Native Versori `log` - Use `log` from context
+
+**Entity Type:**
+
+- **InventoryQuantity** - Fluent entity for inventory positions and quantities
+- **EntityType: 'INVENTORY'** - Used in Batch API `sendBatch()` call
+- **Batch API Method** - Uses `createJob()` and `sendBatch()` (not Event API)
+
+**Critical Patterns:**
+
+- **JSON Format Support**: Standard JSON objects and JSON Lines (one record per line)
+- **Direct Field Access**: Use simple paths like `"locationRef"` (no `@` prefix needed for JSON)
+- **Safe S3 Paths**: Use absolute paths in S3DataSource config
+- **Always Dispose**: Call `s3.dispose()` in `finally` block to release connections
+- **BPP Configuration**: Enabled by default for full snapshots, skip for delta feeds
+- **Progress Logging**: Enhanced logging with context (sample SKUs, locations)
+- **Detailed Logging Toggle**: Optional detailed payload logging for debugging (default: disabled)
+- **File Tracking Toggle**: Optional file tracking to prevent duplicates (default: enabled)
+- **Retry Logic**: Enhanced S3 operations with exponential backoff
+
+**When to Use This Template:**
+
+- ✅ Daily/hourly full inventory snapshots from S3 JSON files
+- ✅ Bulk inventory updates with BPP change detection
+- ✅ JSON or JSON Lines format data
+- ✅ Need state management to prevent duplicate processing
+- ✅ Files should be archived after processing
+
+**When NOT to Use:**
+
+- ❌ Single inventory updates (use GraphQL mutation instead)
+- ❌ Products, Locations, Customers (use Event API templates)
+- ❌ Real-time inventory events (use Event API)
+- ❌ XML/CSV/Parquet formats (use appropriate format template)
+
+---
+
+## STEP 2: AI Prompt
+
+**Copy this prompt to generate the complete implementation:**
+
+```
+Create a Versori scheduled workflow for S3 JSON inventory ingestion to Fluent Commerce Batch API.
+
+REQUIREMENTS:
+1. Runtime: Versori Platform (scheduled workflow)
+2. Source: S3 JSON files from s3://bucket/inventory/
+3. Destination: Fluent Commerce Batch API (InventoryQuantity entity)
+4. Format: JSON (standard JSON objects and JSON Lines support)
+5. Entity: InventoryQuantity (EntityType: 'INVENTORY')
+
+KEY FEATURES:
+- S3 JSON file discovery with VersoriFileTracker for state management
+- JSON parsing with format auto-detection (JSON vs JSON Lines)
+- Direct field mapping using UniversalMapper (no special notation needed)
+- Batch API with chunking (1000 records per batch) and BPP change detection
+- Safe S3 paths with absolute path requirements
+- S3 dispose() in finally block
+- Job lifecycle tracking with JobTracker
+- File archival on S3 after successful processing
+- Error handling with exponential backoff retry
+
+CRITICAL REQUIREMENTS:
+1. JSON Parser: JSONParserService with format auto-detection
+2. Array Extraction: Handle both standard JSON arrays and JSON Lines
+3. Mapping Config: Use direct paths like "locationRef" (no @ prefix)
+4. Entity Type: 'INVENTORY' (for InventoryQuantity)
+5. BPP: Enabled by default (full snapshots), skip for delta feeds
+6. S3 Config: Include proper AWS credentials configuration
+7. S3 Dispose: Always call s3.dispose() in finally block
+8. Native Logging: Use log from context
+9. Retry Logic: Enhanced S3 operations with exponential backoff
+
+SDK METHODS TO USE:
+- createClient(ctx) - Pass entire Versori context, auto-detects platform
+- client.setRetailerId(retailerId) - REQUIRED after createClient for Batch API operations
+- new S3DataSource(config, log) - Config includes AWS credentials
+- await s3.listFiles({ prefix, maxKeys })
+- await s3.downloadFile(key, { encoding: 'utf8' })
+- await s3.moveFile(sourceKey, destKey)
+- await s3.dispose() - MUST call in finally block
+- new JSONParserService()
+- await parser.parse(jsonContent, { format: 'json' | 'jsonl' })
+- new UniversalMapper(mappingConfig)
+- await mapper.map(records)
+- new VersoriFileTracker(ctx.openKv(':project:'), 'prefix')
+- await fileTracker.wasFileProcessed(fileName)
+- await fileTracker.markFileProcessed(fileName, metadata)
+- new JobTracker(ctx.openKv(':project:'), log)
+- await tracker.createJob(jobId, { triggeredBy: 'schedule', stage: 'initialization' })
+- await tracker.updateJob(jobId, { status: 'processing' })
+- await tracker.markCompleted(jobId, details)
+- await tracker.markFailed(jobId, error)
+- await client.createJob({ name, meta: { preprocessing: 'skip' } })
+- await client.sendBatch(jobId, { action: 'UPSERT', entityType: 'INVENTORY', entities })
+- Fire-and-forget batch submission (no polling)
+
+FORBIDDEN PATTERNS:
+- ❌ LoggingService (removed - use native log on Versori)
+- ❌ Don't use @ prefix for JSON fields (that's for XML attributes only)
+- ❌ Don't forget to wrap records in proper structure for mapping
+- ❌ Don't use Event API (use Batch API)
+- ❌ Don't forget to call s3.dispose() in finally block
+- ❌ Don't use relative S3 paths (use absolute paths)
+
+MAPPING CONFIGURATION FILE: config/inventory.batch.json
+Structure:
+{
+ "name": "inventory.batch.json",
+ "version": "1.0.0",
+ "description": "JSON inventory to Fluent Commerce Batch API mapping",
+ "fields": {
+  "locationRef": { "source": "locationRef", "required": true, "resolver": "sdk.trim" },
+  "skuRef": { "source": "skuRef", "required": true, "resolver": "sdk.trim" },
+  "qty": { "source": "qty", "required": true, "resolver": "sdk.parseInt" },
+  "type": { "source": "type", "required": true, "resolver": "sdk.uppercase" },
+  "status": { "source": "status", "required": true, "resolver": "sdk.uppercase" },
+  "expectedOn": { "source": "expectedOn", "required": false, "resolver": "sdk.formatDate" },
+  "attributes.expiryDate": { "source": "attributes.expiryDate", "required": false, "resolver": "sdk.formatDate" },
+  "attributes.batchNumber": { "source": "attributes.batchNumber", "required": false },
+  "attributes.condition": { "source": "attributes.condition", "required": false, "defaultValue": "NEW", "resolver": "sdk.uppercase" },
+  "attributes.storageZone": { "source": "attributes.storageZone", "required": false }
+ }
+}
+
+GENERATE:
+1. package.json with dependencies
+2. index.ts (workflow entry point with scheduled trigger)
+3. src/workflows/scheduled/daily-inventory-sync.ts (scheduled workflow)
+4. src/workflows/webhook/adhoc-inventory-sync.ts (webhook workflow)
+5. src/workflows/webhook/job-status-check.ts (status webhook)
+6. src/services/inventory-sync.service.ts (shared orchestration logic)
+7. src/types/inventory.types.ts (TypeScript type definitions)
+8. config/inventory.batch.json (mapping configuration - external JSON file)
+9. .env.example (environment variables)
+
+NOTE: Use external JSON files for mapping configuration (not TypeScript .config files)
+
+Ensure all code is production-ready with proper error handling, S3 dispose() in finally block. Polling is intentionally omitted (fire-and-forget).
+```
+
+---
+
+## What You'll Build
+
+### Project Structure
+
+```
+s3-json-inventory-batch-sync/
+├── package.json
+├── index.ts                 # Workflow entry point
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-inventory-sync.ts   # Scheduled: Daily inventory sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-inventory-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── inventory-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    ├── config/
+    │   └── inventory.batch.json       # Mapping configuration (external JSON)
+    │
+    └── types/
+        └── inventory.types.ts        # TypeScript interfaces
+```
+
+### Features
+
+- ✅ Scheduled Versori workflow (daily/hourly inventory sync)
+- ✅ S3 file download with enhanced retry logic
+- ✅ JSON parsing with format auto-detection (standard JSON vs JSON Lines)
+- ✅ Direct field mapping (no special notation needed for JSON)
+- ✅ Field mapping with UniversalMapper and external JSON config
+- ✅ Batch API job creation and chunked processing (1000 records/batch)
+- ✅ BPP (Batch Pre-Processing) change detection
+- ✅ Fire-and-forget batch submission with audit logging
+- ✅ State management (VersoriFileTracker + JobTracker prevent duplicates)
+- ✅ File archival on S3 (success → processed/, failure → errors/)
+- ✅ Comprehensive error handling with exponential backoff retry
+- ✅ Modular architecture (reusable services, easy to test)
+
+---
+
+## Versori Workflows Structure
+
+**Key Concept**: Versori workflows are organized by **trigger type** at the first level, then by **specific workflow** with descriptive file names.
+
+**Trigger Types:**
+- **`schedule()`** → Time-based triggers (cron expressions) - NOT exposed as HTTP endpoints
+- **`webhook()`** → HTTP-based triggers (event-driven) - Creates HTTP endpoints
+- **`workflow()`** → Durable workflows (advanced, rarely used)
+
+**Execution Steps (chained to triggers):**
+- **`http()`** → External API calls (chained from schedule/webhook)
+- **`fn()`** → Internal processing (chained from schedule/webhook)
+
+### Recommended Project Structure
+
+```
+s3-json-inventory-batch-sync/
+├── index.ts                              # Entry point - exports all workflows
+└── src/
+    ├── workflows/
+    │   ├── scheduled/
+    │   │   └── daily-inventory-sync.ts   # Scheduled: Daily inventory sync
+    │   │
+    │   └── webhook/
+    │       ├── adhoc-inventory-sync.ts   # Webhook: Manual trigger
+    │       └── job-status-check.ts       # Webhook: Status query
+    │
+    ├── services/
+    │   └── inventory-sync.service.ts     # Shared orchestration logic (reusable)
+    │
+    └── types/
+        └── inventory.types.ts            # Shared type definitions
+```
+
+**Benefits:**
+- ✅ Clear trigger separation (`scheduled/` vs `webhook/`)
+- ✅ Descriptive file names (easy to browse and understand)
+- ✅ Scalable (add new workflows without cluttering)
+- ✅ Reusable code in `services/` (DRY principle)
+- ✅ Easy to modify individual workflows without affecting others
+
+---
+
+## Workflow Files
+
+### 1. Scheduled Workflows (`src/workflows/scheduled/`)
+
+All time-based triggers that run automatically on cron schedules.
+
+#### `src/workflows/scheduled/daily-inventory-sync.ts`
+
+**Purpose**: Automatic daily inventory sync
+**Trigger**: Cron schedule (`0 2 * * *`)
+**Exposed as Endpoint**: ❌ NO - Runs automatically
+
+```typescript
+import { schedule, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { runIngestion } from '../../services/inventory-sync.service';
+
+/**
+ * Scheduled Workflow: Daily Inventory Sync
+ *
+ * Runs automatically daily at 2 AM UTC
+ * NOT exposed as HTTP endpoint - Versori executes on schedule
+ *
+ * Uses shared service: inventory-sync.service.ts
+ */
+export const dailyInventorySync = schedule(
+  'inventory-batch-scheduled',
+  '0 2 * * *' // Daily at 2 AM UTC
+).then(
+  http('run-inventory-batch', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log, openKv } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `inventory-batch-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    log.info('═══════════════════════════════════════════════════════════════');
+    log.info('🚀 [WORKFLOW] Starting scheduled inventory sync', { jobId });
+    log.info('═══════════════════════════════════════════════════════════════');
+
+    await tracker.createJob(jobId, {
+      triggeredBy: 'schedule',
+      stage: 'initialization',
+      startTime: executionStartTime,
+    });
+
+    await tracker.updateJob(jobId, { status: 'processing' });
+
+    try {
+      // Reuse shared orchestration logic
+      const result = await runIngestion(ctx, jobId, tracker);
+
+      if (result.success) {
+        const duration = Date.now() - executionStartTime;
+        await tracker.markCompleted(jobId, { ...result, duration });
+        log.info('✅ [WORKFLOW] Inventory sync completed successfully', {
+          jobId,
+          filesProcessed: result.filesProcessed,
+          duration: `${duration}ms (${(duration / 1000).toFixed(2)}s)`,
+        });
+      } else {
+        await tracker.markFailed(jobId, result.error || 'Unknown error');
+        log.error('❌ [WORKFLOW] Inventory sync failed', {
+          jobId,
+          error: result.error,
+        });
+      }
+
+      log.info('═══════════════════════════════════════════════════════════════');
+      log.info('🏁 [WORKFLOW] Execution complete', {
+        jobId,
+        success: result.success,
+        duration: `${Date.now() - executionStartTime}ms`,
+      });
+      log.info('═══════════════════════════════════════════════════════════════');
+
+      return { success: true, jobId, ...result };
+    } catch (e: any) {
+      const errorMessage = e instanceof Error ? e.message : String(e);
+      await tracker.markFailed(jobId, errorMessage);
+      log.error('❌ [WORKFLOW] Inventory sync failed with exception', {
+        jobId,
+        error: errorMessage,
+        stack: e instanceof Error ? e.stack : undefined,
+        recommendation: 'Check SFTP credentials, network connectivity, and file permissions',
+      });
+      return { success: false, jobId, error: errorMessage };
+    }
+  })
+);
+```
+
+---
+
+### 2. Webhook Workflows (`src/workflows/webhook/`)
+
+All HTTP-based triggers that create webhook endpoints.
+
+#### `src/workflows/webhook/adhoc-inventory-sync.ts`
+
+**Purpose**: Manual inventory sync trigger (on-demand)
+**Trigger**: HTTP POST
+**Endpoint**: `POST https://{workspace}.versori.run/inventory-batch-adhoc`
+**Use Cases**: Testing, priority processing, ad-hoc runs
+
+```typescript
+import { webhook, http } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+import { runIngestion } from '../../services/inventory-sync.service';
+
+/**
+ * Webhook: Manual Inventory Sync Trigger
+ *
+ * Endpoint: POST https://{workspace}.versori.run/inventory-batch-adhoc
+ * Request body (optional): { filePattern: "urgent_*.json", maxFiles: 5 }
+ *
+ * Pattern: Sync mode + fire-and-forget
+ * - Returns jobId immediately
+ * - Background processing continues without blocking response
+ * - ✅ Works because Versori keeps execution context alive for unawaited promises
+ *
+ * Uses shared service: inventory-sync.service.ts
+ */
+export const adhocInventorySync = webhook('inventory-batch-adhoc', {
+  response: { mode: 'sync' }, // ✅ Sync mode: response sent when handler returns
+  connection: 'inventory-batch-adhoc', // Versori validates API key
+}).then(
+  http('run-inventory-batch-adhoc', { connection: 'fluent_commerce' }, async (ctx: any) => {
+    const { log, openKv, data } = ctx;
+    const executionStartTime = Date.now();
+    const jobId = `inventory-batch-adhoc-${Date.now()}`;
+    const tracker = new JobTracker(openKv(':project:'), log);
+
+    const filePattern = data?.filePattern as string || '*.json';
+    const maxFiles = data?.maxFiles as number;
+
+    log.info('🚀 [WEBHOOK] Adhoc inventory sync triggered', {
+      jobId,
+      filePattern,
+      maxFiles,
+      requestData: data,
+    });
+
+    // Create job entry FIRST (awaited to ensure job exists in KV)
+    await tracker.createJob(jobId, {
+      triggeredBy: 'manual',
+      stage: 'initialization',
+      status: 'queued',
+      startTime: executionStartTime,
+      options: { filePattern, maxFiles },
+      createdAt: new Date().toISOString(),
+    });
+
+    // ✅ Fire-and-forget: Start background processing WITHOUT await
+    // The promise continues execution after we return the response
+    runIngestion(ctx, jobId, tracker)
+      .then((result) => {
+        const duration = Date.now() - executionStartTime;
+        if (result.success) {
+          log.info('✅ [BACKGROUND] Inventory sync completed successfully', {
+            jobId,
+            filesProcessed: result.filesProcessed,
+            filesFailed: result.filesFailed,
+            recordsProcessed: result.recordsProcessed,
+            duration: `${duration}ms (${(duration / 1000).toFixed(2)}s)`,
+          });
+          return tracker.markCompleted(jobId, { ...result, duration });
+        } else {
+          log.error('❌ [BACKGROUND] Inventory sync failed', {
+            jobId,
+            error: result.error,
+          });
+          return tracker.markFailed(jobId, result.error || 'Unknown error');
+        }
+      })
+      .catch((error: unknown) => {
+        const errorMessage = error instanceof Error ? error.message : String(error);
+        const errorStack = error instanceof Error ? error.stack : undefined;
+
+        log.error('❌ [BACKGROUND] Inventory sync failed with exception', {
+          jobId,
+          error: errorMessage,
+          stack: errorStack,
+          errorType: error instanceof Error ? error.constructor.name : typeof error,
+          recommendation: 'Check S3 credentials, bucket permissions, and network connectivity',
+        });
+
+        return tracker.markFailed(jobId, errorMessage);
+      });
+
+    // Return immediately with jobId (response sent with this return value)
+    return {
+      success: true,
+      jobId,
+      message: 'Inventory sync started in background',
+      statusEndpoint: `https://{workspace}.versori.run/inventory-batch-job-status`,
+      note: 'Poll the status endpoint with jobId to check progress',
+    };
+  })
+);
+```
+
+---
+
+#### `src/workflows/webhook/job-status-check.ts`
+
+**Purpose**: Query job status and progress
+**Trigger**: HTTP POST/GET
+**Endpoint**: `POST https://{workspace}.versori.run/inventory-batch-job-status`
+**Request Body**: `{ "jobId": "inventory-batch-1234567890" }`
+
+```typescript
+import { webhook, fn } from '@versori/run';
+import { JobTracker } from '@fluentcommerce/fc-connect-sdk';
+
+/**
+ * Webhook: Job Status Check
+ *
+ * Endpoint: POST https://{workspace}.versori.run/inventory-batch-job-status
+ * Request body: { "jobId": "inventory-batch-1234567890" }
+ *
+ * Pattern: webhook().then(fn()) - no external API needed, only KV storage
+ * Lightweight: Only queries KV store, no Fluent API calls
+ */
+export const jobStatusCheck = webhook('inventory-batch-job-status', {
+  response: { mode: 'sync' },
+  connection: 'inventory-batch-job-status',
+}).then(
+  fn('status', async ctx => {
+    const { data, log, openKv } = ctx;
+    const jobId = data?.jobId as string;
+
+    if (!jobId) {
+      return { success: false, error: 'jobId required' };
+    }
+
+    const tracker = new JobTracker(openKv(':project:'), log);
+    const status = await tracker.getJob(jobId);
+
+    return status
+      ? { success: true, jobId, ...status }
+      : { success: false, error: 'Job not found', jobId };
+  })
+);
+```
+
+---
+
+### 3. Entry Point (`index.ts`)
+
+**Purpose**: Register all workflows with Versori platform
+
+```typescript
+/**
+ * Entry Point - Registers all workflows with Versori platform
+ *
+ * Versori automatically discovers and registers exported workflows
+ *
+ * File Structure:
+ * - src/workflows/scheduled/ → Time-based triggers (cron)
+ * - src/workflows/webhook/   → HTTP-based triggers (webhooks)
+ */
+
+// Scheduled workflows
+export { dailyInventorySync } from './src/workflows/scheduled/daily-inventory-sync';
+
+// Webhook workflows
+export { adhocInventorySync } from './src/workflows/webhook/adhoc-inventory-sync';
+export { jobStatusCheck } from './src/workflows/webhook/job-status-check';
+```
+
+**What Gets Exposed:**
+- ✅ `adhocInventorySync` → `https://{workspace}.versori.run/inventory-batch-adhoc`
+- ✅ `jobStatusCheck` → `https://{workspace}.versori.run/inventory-batch-job-status`
+- ❌ `dailyInventorySync` → NOT exposed (runs automatically on cron)
+
+## When to Use Batch API vs Event API
+
+### ✅ Use Batch API (`createJob`/`sendBatch`) For:
+
+| Entity Type      | Use Case              | Why Batch API                  |
+| --------------------- | ---------------------------------- | ----------------------------------------------- |
+| **Inventory**     | Bulk inventory updates, daily sync | Optimized for high-volume, BPP change detection |
+| **InventoryQuantity** | Inventory positions and quantities | Native Batch API entity type          |
+
+### ❌ Use Event API Instead For:
+
+| Entity Type     | Use Case               | Why Event API / GraphQL            |
+| ------------------- | ------------------------------------- | --------------------------------------------- |
+| **Products**    | Product catalog sync, variant updates | Triggers workflows, validates business rules |
+| **Locations**    | Store/warehouse setup         | Requires workflow orchestration        |
+| **Customers**    | Customer registration/profile updates | Prefer GraphQL mutations (no Rubix support)  |
+| **Orders**     | Updates/events (not creation)     | Events fine for updates; creation via GraphQL |
+| **Custom Entities** | Any entity needing workflow triggers | Full Rubix workflow support          |
+
+### 🔍 Use GraphQL Mutations For:
+
+| Scenario       | Why GraphQL              |
+| --------------------- | -------------------------------------- |
+| **Single operations** | Create one record, update one record  |
+| **Complex queries**  | Fetch data with relationships     |
+| **Testing/debugging** | Direct API control, immediate feedback |
+
+Note:
+
+- Orders: Use `createOrder` GraphQL mutation for order creation (this triggers the Order CREATED event in Rubix). Order updates/events are fine via Event API.
+- Customers: Use GraphQL mutations (no Rubix workflow support for customers).
+
+---
+
+## Understanding BPP (Batch Pre-Processing)
+
+**BPP** is Fluent's change detection system that filters out unchanged records before workflow processing.
+
+Note on defaults and control:
+
+- BPP is a Fluent platform feature. The default on/off behavior is controlled at your Fluent account level, not by the SDK.
+- If you omit `meta.preprocessing` in `createJob()`, the account-level default applies.
+- To force behavior per job, set `meta.preprocessing` explicitly (`'skip'` to disable for delta feeds).
+
+### When to Use BPP (Default - Enabled)
+
+**✅ Full Inventory Snapshots:**
+
+- Daily complete inventory dumps
+- Entire warehouse stock files
+- Records may be identical to previous run
+
+**Example:**
+
+```typescript
+// BPP enabled by default - filters unchanged records
+const job = await client.createJob({
+ name: 'Daily Full Inventory',
+ retailerId: 'my-retailer',
+ // BPP automatically enabled - no meta needed
+});
+```
+
+**What BPP does:**
+
+- Compares incoming records with existing records in Fluent
+- Filters out records with no changes
+- Only passes changed/new records to workflows
+- Significantly reduces workflow processing load
+
+### When to Skip BPP
+
+**✅ Delta Feeds (Pre-Filtered Data):**
+
+- Hourly change files (only updates since last run)
+- Event-driven inventory changes
+- Pre-filtered data from source system
+- All records are guaranteed to be changes
+
+**Example:**
+
+```typescript
+// Skip BPP for delta feeds - all records are changes
+const job = await client.createJob({
+ name: 'Hourly Delta Inventory',
+ retailerId: 'my-retailer',
+ meta: {
+  preprocessing: 'skip', // All records already filtered
+ },
+});
+```
+
+**Performance Impact:**
+
+- **Full snapshot + BPP**: 10,000 records → 500 changes → Fast
+- **Full snapshot, no BPP**: 10,000 records → 10,000 processed → Slow
+- **Delta feed + BPP**: 500 records → 500 changes → Fast (but BPP overhead wasted)
+- **Delta feed, no BPP**: 500 records → 500 processed → Fastest
+
+### Decision Guide
+
+| Source Data Type    | BPP Setting       | Reason                     |
+| ---------------------- | ----------------------- | ----------------------------------------------- |
+| **Daily full dump**  | Enabled (default)    | Most records unchanged, BPP filters efficiently |
+| **Hourly delta feed** | `preprocessing: 'skip'` | All records are changes, BPP overhead wasted  |
+| **Initial load**    | Enabled (default)    | No previous data, but good practice       |
+| **Manual corrections** | Enabled (default)    | Unknown change ratio, let BPP filter      |
+| **Real-time events**  | `preprocessing: 'skip'` | Each record is a change event          |
+
+---
+
+## Processing Modes
+
+**⚠️ IMPORTANT:** Choose ONE processing mode per connector. These are alternative patterns, not features to use together.
+
+The SDK supports three processing modes for handling multiple files. **Select the mode that best fits your use case** and configure your connector accordingly:
+
+### Mode 1: Per-File Processing (Recommended Default) ✅ IMPLEMENTED
+
+**When to use:**
+
+- Multiple large files that shouldn't be in memory together
+- Need file-level consistency (file 3 fails → files 1-2 already archived)
+- Memory constraints
+- Fault isolation (one file failure doesn't affect others)
+
+**Flow:**
+
+```
+FOR EACH FILE:
+ 1. Download file
+ 2. Parse JSON
+ 3. Map records
+ 4. Create Batch API job
+ 5. Send batches
+ 6. Write log
+ 7. Archive file
+ 8. Mark file processed
+
+IF FILE FAILS:
+ - Move to /errors/
+ - Continue to next file
+ - Other files unaffected
+```
+
+**Configuration:**
+
+```json
+{
+ "processingMode": "per-file",
+ "maxFilesPerRun": 10
+}
+```
+
+**Benefits:**
+
+- ✅ File-level atomicity (each file processed independently)
+- ✅ Low memory footprint (1 file at a time)
+- ✅ Clear error isolation (failed file doesn't block others)
+- ✅ Incremental progress (files archived as completed)
+
+**Example implementation (see code section below)**
+
+### Mode 2: Batch Processing (All Files at Once) ⚠️ OPTIONAL - Choose if needed
+
+**When to use:**
+
+- Small files (all fit in memory comfortably)
+- Need one Batch API job for all files
+- Atomic processing (all files or none)
+
+**Flow:**
+
+```
+1. Download ALL files
+2. Parse ALL files
+3. Map ALL records
+4. Create ONE Batch API job
+5. Send ALL batches
+6. Write logs
+7. Archive ALL files
+8. Mark ALL processed
+```
+
+**Configuration:**
+
+```json
+{
+ "processingMode": "batch",
+ "maxFilesPerRun": 10
+}
+```
+
+**Benefits:**
+
+- ✅ Single job for all files (simplifies tracking)
+- ✅ Faster for many small files (parallel parsing possible)
+
+**Drawbacks:**
+
+- ❌ High memory usage (all files in memory)
+- ❌ All-or-nothing (one failure affects all)
+- ❌ No incremental progress
+
+### Mode 3: Chunked Per-File Processing (Balanced) ⚠️ OPTIONAL - Choose if needed
+
+**When to use:**
+
+- Many files (e.g., 100+ files)
+- Process N files at a time
+- Balance between memory and parallelism
+
+**Flow:**
+
+```
+CHUNK files into groups of N (e.g., 5 files per chunk):
+
+FOR EACH CHUNK:
+ FOR EACH FILE in chunk:
+  1. Download file
+  2. Parse JSON
+  3. Map records
+  4. Create Batch API job
+  5. Send batches
+  6. Archive file
+
+ Wait for chunk to complete before next chunk
+```
+
+**Configuration:**
+
+```json
+{
+ "processingMode": "chunked",
+ "fileChunkSize": 5,
+ "maxFilesPerRun": 100
+}
+```
+
+**Benefits:**
+
+- ✅ Bounded memory usage (N files at a time)
+- ✅ Better throughput than per-file (parallel processing within chunk)
+- ✅ Incremental progress (per-chunk)
+
+**Use cases:**
+
+- High-volume file ingestion (100+ files per run)
+- Rate limiting (control API load)
+- Resource-constrained environments
+
+### Comparison Matrix
+
+| Aspect       | Per-File              | Batch             | Chunked                 |
+| ------------------- | ---------------------------------- | ----------------------------- | --------------------------------------- |
+| **Memory Usage**  | Low (1 file at a time)       | High (all files)       | Medium (N files)            |
+| **Consistency**   | Per-file atomic          | All-or-nothing        | Per-chunk atomic            |
+| **Fault Isolation** | ✅ Best (1 file fails → others OK) | ❌ Worst (1 fails → all fail) | ✅ Good (chunk fails → other chunks OK) |
+| **Performance**   | Slower (sequential)        | Fastest (parallel parsing)  | Balanced                |
+| **Batch API Jobs** | N jobs (1 per file)        | 1 job (all files)       | N jobs (1 per file)           |
+| **Use Case**    | Large files, strict consistency  | Small files, fast processing | Many files, balanced approach      |
+| **Recommended For** | Production (safest)        | Testing, small datasets    | High-volume production         |
+
+> **📋 This Template Implementation:**
+> 
+> **✅ This template implements ALL THREE modes** and selects based on `PROCESSING_MODE` variable.
+> 
+> **Choose ONE mode per connector:**
+> - **Default:** `PROCESSING_MODE=per-file` (recommended for production)
+> - **Alternative:** `PROCESSING_MODE=batch` (for small files, atomic processing)
+> - **Alternative:** `PROCESSING_MODE=chunked` (for high-volume scenarios)
+> 
+> **Important:** Do NOT use multiple modes in the same connector. Each connector should use ONE consistent pattern.
+
+### Decision Guide
+
+| Source Data  | File Count | File Size  | Recommended Mode | Reason                 |
+| -------------- | ---------- | ----------- | ---------------- | --------------------------------------- |
+| Daily snapshot | 1-10    | >10MB each | **Per-File**   | Memory efficient, fault isolation    |
+| Hourly deltas | 10-50   | <1MB each  | **Batch**    | Fast processing, small memory footprint |
+| Real-time feed | 100+    | <100KB each | **Chunked**   | Balanced throughput + memory      |
+| Initial load  | 1     | >100MB   | **Per-File**   | Memory safety              |
+| Testing    | 1-5    | Any     | **Batch**    | Simplicity               |
+
+---
+
+## JSON File Format
+
+### Sample: inventory.json (Standard JSON)
+
+```json
+{
+ "inventory": [
+  {
+   "locationRef": "LOC001",
+   "skuRef": "SKU-12345",
+   "qty": 100,
+   "type": "LAST_ON_HAND",
+   "status": "ACTIVE",
+   "expectedOn": "2025-01-25",
+   "attributes": {
+    "expiryDate": "2026-12-31",
+    "batchNumber": "BATCH-A001",
+    "condition": "NEW",
+    "storageZone": "ZONE-A"
+   }
+  },
+  {
+   "locationRef": "LOC001",
+   "skuRef": "SKU-67890",
+   "qty": 50,
+   "type": "LAST_ON_HAND",
+   "status": "ACTIVE",
+   "expectedOn": "2025-01-25",
+   "attributes": {
+    "expiryDate": "2026-06-30",
+    "batchNumber": "BATCH-A002",
+    "condition": "NEW",
+    "storageZone": "ZONE-B"
+   }
+  }
+ ]
+}
+```
+
+### Sample: inventory.jsonl (JSON Lines)
+
+```jsonl
+{"locationRef":"LOC001","skuRef":"SKU-12345","qty":100,"type":"LAST_ON_HAND","status":"ACTIVE","expectedOn":"2025-01-25","attributes":{"expiryDate":"2026-12-31","batchNumber":"BATCH-A001","condition":"NEW","storageZone":"ZONE-A"}}
+{"locationRef":"LOC001","skuRef":"SKU-67890","qty":50,"type":"LAST_ON_HAND","status":"ACTIVE","expectedOn":"2025-01-25","attributes":{"expiryDate":"2026-06-30","batchNumber":"BATCH-A002","condition":"NEW","storageZone":"ZONE-B"}}
+```
+
+**JSON Structure:**
+
+- Standard JSON: Root object with nested arrays (`{ "inventory": [...] }`)
+- JSON Lines: One JSON object per line (streaming-friendly for large files)
+- JSONParserService auto-detects format based on content
+- Both formats support nested objects with dot notation
+
+**Field Descriptions:**
+
+- `locationRef`: Fluent location reference
+- `skuRef`: Fluent SKU/product reference
+- `qty`: Inventory quantity (integer)
+- `type`: Inventory type (LAST_ON_HAND for full snapshots, DELTA for incremental changes)
+- `status`: Record status (ACTIVE, INACTIVE)
+- `expectedOn`: Expected date (ISO 8601 or parseable format)
+- `attributes.expiryDate`: Product expiry date (optional)
+- `attributes.batchNumber`: Manufacturing batch/lot number (optional)
+- `attributes.condition`: Product condition (NEW, USED, DAMAGED)
+- `attributes.storageZone`: Warehouse zone identifier (optional)
+
+### JSON Field Mapping
+
+**IMPORTANT**: JSON uses direct field access (no special prefix needed):
+
+```json
+{
+ "fields": {
+  "locationRef": { "source": "locationRef", "required": true },
+  "skuRef": { "source": "skuRef", "required": true }
+ }
+}
+```
+
+**Why**: Unlike XML attributes (which need `@` prefix), JSON fields are accessed directly by name.
+
+### Array Handling (Standard JSON vs JSON Lines)
+
+**JSONParserService behavior:**
+
+- **Standard JSON**: Returns object or array based on structure
+- **JSON Lines**: Returns array of objects (one per line)
+
+**Solution**: Always normalize to array after parsing:
+
+```typescript
+const parsed = await parser.parse(jsonContent, { format: 'json' | 'jsonl' });
+
+// Extract inventory array - handle different JSON structures
+let records: any[] = [];
+if (format === 'jsonl') {
+ records = Array.isArray(parsed) ? parsed : [parsed];
+} else {
+ if (Array.isArray(parsed)) {
+  records = parsed; // Root level array
+ } else if (parsed?.inventory && Array.isArray(parsed.inventory)) {
+  records = parsed.inventory; // { "inventory": [...] }
+ } else if (parsed?.records && Array.isArray(parsed.records)) {
+  records = parsed.records; // { "records": [...] }
+ } else {
+  records = [parsed]; // Single object
+ }
+}
+```
+
+---
+
+## Service Implementation
+
+### File: `src/services/inventory-sync.service.ts`
+
+**Complete implementation showing all SDK patterns:**
+
+```typescript
+import { Buffer } from 'node:buffer';  // ✅ Required for Versori/Deno runtime
+import {
+  createClient,
+  S3DataSource,
+  JSONParserService,
+  UniversalMapper,
+  VersoriFileTracker,
+  JobTracker,
+  extractFileName,
+} from '@fluentcommerce/fc-connect-sdk';
+import type { FileMetadata } from '@fluentcommerce/fc-connect-sdk';
+import inventoryMapping from '../config/inventory.batch.json' with { type: 'json' };  // ✅ External JSON import
+
+/**
+ * Shared inventory ingestion orchestration logic
+ * Used by both scheduled and webhook workflows
+ *
+ * @param ctx - Versori HTTP context
+ * @param jobId - Unique job identifier
+ * @param tracker - JobTracker instance
+ */
+export async function runIngestion(
+  ctx: any,
+  jobId: string,
+  tracker: JobTracker
+) {
+  const { log, activation, openKv } = ctx;
+  const startTime = Date.now();
+  let s3: S3DataSource | null = null;
+
+  // Optional feature toggles
+  const enableFileTracking = activation.getVariable('enableFileTracking') !== 'false';
+  const detailedLogging = activation.getVariable('detailedLogging') === 'true';
+
+  try {
+    // ========================================
+    // CLIENT INITIALIZATION (with retailerId and validation)
+    // ========================================
+    log.info('🔧 [InventorySync] Initializing Fluent Commerce client...');
+
+    const client = await createClient({
+      ...ctx,
+      validateConnection: true,  // ✅ Validate connection on creation
+    });
+
+    // ✅ CRITICAL: Set retailerId for Batch API
+    const retailerId = activation.getVariable('fluentRetailerId');
+    if (!retailerId) {
+      log.error('❌ [InventorySync] Missing required fluentRetailerId activation variable');
+      throw new Error('fluentRetailerId activation variable is required for Batch API');
+    }
+    client.setRetailerId(retailerId);
+
+    log.info('✅ [InventorySync] Client initialized and validated', { retailerId });
+
+    // ========================================
+    // S3 DATA SOURCE INITIALIZATION
+    // ========================================
+    log.info('🔧 [InventorySync] Initializing S3 data source...');
+
+    const s3Config = {
+      bucket: activation.getVariable('s3BucketName'),
+      region: activation.getVariable('awsRegion') || 'us-east-1',
+      accessKeyId: activation.getVariable('awsAccessKeyId'),
+      secretAccessKey: activation.getVariable('awsSecretAccessKey'),
+      prefix: activation.getVariable('s3Prefix') || 'inventory/',
+    };
+
+    s3 = new S3DataSource(
+      {
+        type: 'S3_JSON',
+        connectionId: 's3-inventory-sync',
+        name: 'inventory-sync',
+        s3Config,
+      },
+      log
+    );
+
+    log.info('✅ [InventorySync] S3 data source initialized', {
+      bucket: s3Config.bucket,
+      region: s3Config.region,
+      prefix: s3Config.prefix,
+    });
+
+    // ========================================
+    // SERVICE INITIALIZATION
+    // ========================================
+    const parser = new JSONParserService();
+    const mapper = new UniversalMapper(inventoryMapping);
+    const kv = openKv(':project:');
+    const fileTracker = enableFileTracking
+      ? new VersoriFileTracker(kv, 's3-json-inventory-sync')
+      : null;
+    const bppEnabled = activation.getVariable('bppEnabled') !== 'false';
+
+    if (detailedLogging) {
+      log.info('🔍 [InventorySync] Configuration loaded', {
+        bppEnabled,
+        enableFileTracking,
+        detailedLogging,
+      });
+    }
+
+    // ========================================
+    // FILE DISCOVERY
+    // ========================================
+    log.info('🔍 [InventorySync] Discovering files on S3...');
+
+    const files = await s3.listFiles({ prefix: s3Config.prefix });
+    log.info('📄 [InventorySync] File discovery complete', { count: files.length });
+
+    if (files.length === 0) {
+      log.info('⚠️ [InventorySync] No files found to process');
+      return { success: true, message: 'No files to process', filesProcessed: 0 };
+    }
+
+    // ========================================
+    // FILE PROCESSING
+    // ========================================
+    const results = [];
+    let fileIndex = 0;
+
+    for (const file of files) {
+      fileIndex++;
+      const fileStartTime = Date.now();
+      const fileName = extractFileName(file.path);
+
+      log.info(`📄 [FILE ${fileIndex}/${files.length}] Processing: ${fileName}`);
+
+      try {
+        // Check if already processed
+        if (fileTracker && (await fileTracker.wasFileProcessed(file.name))) {
+          log.info(`⏭️ [FILE ${fileIndex}/${files.length}] Skipping already processed: ${fileName}`);
+          results.push({ fileName, success: true, skipped: true });
+          continue;
+        }
+
+        // Download and parse
+        if (detailedLogging) {
+          log.info(`🔍 [FILE ${fileIndex}/${files.length}] Downloading: ${fileName}`);
+        }
+        const jsonContent = (await s3.downloadFile(file.path, { encoding: 'utf8' })) as string;
+        const parsed = await parser.parse(jsonContent);
+
+        // Extract inventory array
+        const records = Array.isArray(parsed) ? parsed :
+                       parsed?.inventory ? parsed.inventory :
+                       [parsed];
+
+        if (records.length === 0) {
+          log.warn(`⚠️ [FILE ${fileIndex}/${files.length}] No records found in: ${fileName}`);
+          continue;
+        }
+
+        log.info(`🔄 [FILE ${fileIndex}/${files.length}] Transforming ${records.length} records`);
+
+        // Transform records
+        const mappedRecords = [];
+        const aggregatedSkippedFields: string[] = [];
+        for (const rec of records) {
+          const sourceDataWithContext = { ...rec, $context: { retailerId } };
+          const mappingResult = await mapper.map(sourceDataWithContext);
+          if (mappingResult.success) {
+            mappedRecords.push(mappingResult.data);
+          }
+          // Aggregate skipped fields
+          if (mappingResult.skippedFields && mappingResult.skippedFields.length > 0) {
+            for (const fieldName of mappingResult.skippedFields) {
+              if (!aggregatedSkippedFields.includes(fieldName)) {
+                aggregatedSkippedFields.push(fieldName);
+              }
+            }
+          }
+        }
+
+        if (aggregatedSkippedFields.length > 0) {
+          log.info('ℹ️ [MAPPING] Optional fields skipped (undefined values)', {
+            file: fileName,
+            skippedFields: aggregatedSkippedFields,
+            note: 'These fields were not present in source data. Add defaultValue to mapping config if they should always appear.',
+          });
+        }
+
+        if (mappedRecords.length === 0) {
+          log.warn(`⚠️ [FILE ${fileIndex}/${files.length}] No valid records after mapping: ${fileName}`, {
+            recommendation: 'Check mapping configuration and required fields',
+          });
+          continue;
+        }
+
+        // Create job and send batches
+        log.info(`📦 [FILE ${fileIndex}/${files.length}] Creating Batch API job for ${mappedRecords.length} records`);
+
+        const job = await client.createJob({
+          name: `s3-json-${fileName}-${Date.now()}`,
+          meta: bppEnabled ? undefined : { preprocessing: 'skip' },
+        });
+
+        await tracker.updateJob(jobId, {
+          status: 'processing',
+          details: { fileName, recordCount: mappedRecords.length },
+        });
+
+        // ? Enhanced: Extract context for progress logging
+        const uniqueLocations = [...new Set(mappedRecords.map((r: any) => r.locationRef))];
+        const sampleSKUs = mappedRecords.slice(0, 5).map((r: any) => r.skuRef);
+
+        // ? Enhanced: Start logging with context
+        log.info(`📤 [BatchProcessor] Sending batch for file "${fileName}"`, {
+          totalRecords: mappedRecords.length,
+          locations: uniqueLocations.join(', '),
+          sampleSKUs: sampleSKUs.join(', '),
+          jobId: job.id
+        });
+
+        // Send batches (fire-and-forget)
+        await client.sendBatch(job.id, {
+          action: 'UPSERT',
+          entityType: 'INVENTORY',
+          entities: mappedRecords,
+        });
+
+        // ? Enhanced: Completion logging
+        log.info(`✅ [BatchProcessor] Batch sent successfully for file "${fileName}"`, {
+          totalRecords: mappedRecords.length,
+          jobId: job.id,
+          duration: Date.now() - fileStartTime
+        });
+
+        // ? Enhanced: Clear large arrays after processing to free memory
+        mappedRecords.length = 0;
+
+        // Mark as processed
+        if (fileTracker) {
+          await fileTracker.markFileProcessed(file.name, {
+            recordCount: mappedRecords.length,
+          });
+        }
+
+        const fileDuration = Date.now() - fileStartTime;
+        results.push({
+          fileName,
+          success: true,
+          recordCount: mappedRecords.length,
+          duration: fileDuration,
+        });
+
+        log.info(`✅ [FILE ${fileIndex}/${files.length}] Successfully processed: ${fileName}`, {
+          records: mappedRecords.length,
+          jobId: job.id,
+          duration: `${fileDuration}ms (${(fileDuration / 1000).toFixed(2)}s)`,
+        });
+
+      } catch (error: any) {
+        const fileDuration = Date.now() - fileStartTime;
+        log.error(`❌ [FILE ${fileIndex}/${files.length}] Processing failed: ${fileName}`, {
+          error: error?.message,
+          stack: error?.stack,
+          duration: `${fileDuration}ms`,
+          recommendation: 'Check S3 permissions, file format, and mapping configuration',
+        });
+        results.push({ fileName, success: false, error: error?.message, duration: fileDuration });
+      }
+    }
+
+    const totalDuration = Date.now() - startTime;
+    const successCount = results.filter((r) => r.success && !r.skipped).length;
+
+    log.info('✅ [InventorySync] Batch processing complete', {
+      total: files.length,
+      processed: successCount,
+      skipped: results.filter((r) => r.skipped).length,
+      failed: results.filter((r) => !r.success).length,
+      duration: `${totalDuration}ms (${(totalDuration / 1000).toFixed(2)}s)`,
+    });
+
+    return {
+      success: true,
+      results,
+      filesProcessed: successCount,
+      duration: totalDuration,
+    };
+
+  } catch (error: any) {
+    const totalDuration = Date.now() - startTime;
+    log.error('❌ [InventorySync] Fatal error during ingestion', {
+      message: error?.message,
+      stack: error?.stack,
+      duration: `${totalDuration}ms (${(totalDuration / 1000).toFixed(2)}s)`,
+      recommendation: 'Check activation variables, S3 credentials, and Fluent API connectivity',
+    });
+    throw error;  // Re-throw for workflow error handling
+  } finally {
+    // ✅ CRITICAL: Always dispose S3 connection
+    if (s3) {
+      try {
+        await s3.dispose();
+        log.info('✅ [InventorySync] S3 connection disposed successfully');
+      } catch (disposeError: any) {
+        log.error('⚠️ [InventorySync] Failed to dispose S3 connection', {
+          error: disposeError?.message,
+        });
+      }
+    }
+  }
+}
+```
+
+**Key Patterns Demonstrated:**
+- ✅ Buffer import for Versori/Deno compatibility
+- ✅ External JSON mapping import with `{ type: 'json' }`
+- ✅ `validateConnection: true` for client initialization
+- ✅ `setRetailerId()` call after client creation (REQUIRED for Batch API)
+- ✅ Emoji-based logging (🚀 ✅ ❌ ⚠️ 🔍 📄) for visual log scanning
+- ✅ Execution boundaries with ═══ separators
+- ✅ `extractFileName()` usage for clean file names
+- ✅ Optional feature toggles (enableFileTracking, detailedLogging)
+- ✅ Duration tracking at file and workflow levels (ms and seconds)
+- ✅ Recommendations in error logs for actionable debugging
+- ✅ S3 `dispose()` in finally block with error handling
+- ✅ VersoriFileTracker for state management (optional via toggle)
+- ✅ JobTracker for job lifecycle tracking
+
+---
+
+## Code Flow Explanation
+
+### Initialization Phase
+
+**Logger Setup:**
+
+```typescript
+// ✅ CORRECT: Use native Versori log from context
+const { log } = ctx;
+log.info('Starting workflow');
+
+// ❌ WRONG: LoggingService (removed - use native log on Versori)
+// import { LoggingService } from '@fluentcommerce/fc-connect-sdk';
+// const logging = new LoggingService();
+// const log = logging.createLogger({ logLevel: 'info' });
+```
+
+- Versori provides `log` in the context - use it directly
+
+**Client Creation:**
+
+```typescript
+// Pass entire Versori context object
+const client = await createClient(ctx);
+```
+
+- `createClient(ctx)` accepts entire Versori context (fetch, connections, log, activation)
+- Auto-detects Versori platform from context
+- Returns `FluentClient` configured with OAuth2 from connections.fluent_commerce
+- OAuth2 authentication handled automatically
+- This is the **CORRECT** pattern for Versori workflows
+
+**S3 Data Source:**
+
+```typescript
+const s3 = new S3DataSource(
+ {
+  type: 'S3_JSON', // Type indicates JSON files (metadata only)
+  connectionId: 's3-inventory-sync', // Unique identifier (required)
+  name: 'inventory-sync', // Human-readable name (required)
+  s3Config: {
+   bucket: 'my-inventory-bucket',
+   region: 'us-east-1',
+   accessKeyId: 'AKIAXXXX',
+   secretAccessKey: 'xxxxxxxx',
+  },
+ },
+ log
+);
+```
+
+- **Constructor params:** `(config: S3DataSourceConfig, log)` - Versori native log, no explicit typing needed
+- `connectionId` and `name` are required in config
+- `type` must be `'S3_JSON'` for JSON files
+- Enhanced retry logic with exponential backoff
+
+### File Discovery Phase
+
+**List Files:**
+
+```typescript
+const files = await s3.listFiles({
+ prefix: config.s3.prefix, // Override config (optional)
+ maxKeys: 1000, // Maximum files to retrieve
+});
+```
+
+- Returns `FileMetadata[]` with: `name`, `lastModified`, `size`, `path`, `source`
+- Files sorted by modification time (newest first)
+- Directories excluded automatically
+
+**State Check:**
+
+```typescript
+const alreadyProcessed = await fileTracker.wasFileProcessed(file.name);
+if (alreadyProcessed) {
+ continue; // Skip already processed files
+}
+```
+
+- Uses Versori KV store (`openKv()`) for distributed state
+- Prevents duplicate processing across workflow runs
+- State persists between executions
+
+### File Processing Phase
+
+**Download File:**
+
+```typescript
+const jsonContent = (await s3.downloadFile(file.path, {
+ encoding: 'utf8',
+})) as string;
+```
+
+- **Important:** Cast to `string` when `encoding` is specified
+- Without encoding, returns `Buffer`
+- Supports streaming for large files
+
+**Parse JSON:**
+
+```typescript
+const jsonFormat = activation.getVariable('jsonFormat') || 'json';
+const parsed = await parser.parse(jsonContent, { format: jsonFormat });
+
+// Extract inventory array - handle different JSON structures
+let records: any[] = [];
+if (jsonFormat === 'jsonl') {
+ records = Array.isArray(parsed) ? parsed : [parsed];
+} else {
+ if (Array.isArray(parsed)) {
+  records = parsed; // Root level array
+ } else if (parsed?.inventory && Array.isArray(parsed.inventory)) {
+  records = parsed.inventory; // { "inventory": [...] }
+ } else {
+  records = [parsed]; // Single object
+ }
+}
+```
+
+- Parses JSON into JavaScript object or array
+- **Format detection**: Auto-detects standard JSON vs JSON Lines
+- Always normalize to array for consistent processing
+- Throws `ParsingError` on invalid JSON
+
+**JSON Field Handling:**
+
+- Direct field access by name (no special prefix needed)
+- Example: `{ "locationRef": "LOC001" }` → `{ locationRef: 'LOC001' }`
+- Use simple paths like `"locationRef"` in mapping config
+
+**Transform Data:**
+
+```typescript
+const mapper = new UniversalMapper(inventoryMapping);
+
+// Map each record directly (no wrapper needed for JSON)
+const mappingResult = await mapper.map(rec);
+
+if (!mappingResult.success) {
+ // Mapping validation failed for required fields
+ log.warn('Mapping failed:', mappingResult.errors);
+ continue; // Skip invalid records
+}
+
+mappedRecords.push(mappingResult.data);
+```
+
+- Direct mapping (no need to wrap in object like XML)
+- `MappingResult`: `{ success: boolean, data: any, errors?: string[] }`
+- `success: false` only if required fields fail
+- Optional field errors reported but don't fail mapping
+
+### Batch API Phase
+
+**Create Job:**
+
+```typescript
+// Full snapshot (BPP enabled by default)
+const job = await client.createJob({
+ name: 'Daily Full Inventory',
+ retailerId: 'my-retailer',
+ // BPP automatically enabled - filters unchanged records
+});
+
+// Delta feed (skip BPP)
+const job = await client.createJob({
+ name: 'Hourly Deltas',
+ retailerId: 'my-retailer',
+ meta: {
+  preprocessing: 'skip', // All records are changes
+ },
+});
+```
+
+- Returns job object with `id` and metadata
+- `retailerId` required for tenant isolation
+- BPP enabled by default, skip with `meta: { preprocessing: 'skip' }`
+
+**Send Batches:**
+
+```typescript
+const batch = await client.sendBatch(jobId, {
+ action: 'UPSERT', // Most common: create or update
+ entityType: 'INVENTORY', // Entity type for inventory
+ source: 'S3_JSON', // Data source identifier (optional)
+ event: 'InventoryChanged', // Valid Rubix event (optional, defaults to InventoryChanged)
+ entities: chunk, // Array of inventory records
+});
+```
+
+- `action`: `'UPSERT'` (currently the only action supported by Fluent Batch API)
+- `entityType`: `'INVENTORY'` for inventory records
+- `entities`: Array of transformed inventory objects
+- Returns batch object with `id` for tracking
+
+**Fire-and-Forget Pattern:**
+
+```typescript
+const batch = await client.sendBatch(jobId, { ... });
+log.info('Batch sent (fire-and-forget)', { batchId: batch.id });
+
+// Record batch details for audit trail
+result.batches.push({
+ batchId: batch.id,
+ recordCount: chunk.length,
+ timestamp: new Date().toISOString(),
+ status: 'SENT',
+});
+```
+
+- No polling - batches sent and tracked immediately
+- Batch details recorded for audit logging
+- Log files written to S3 for tracking (optional, configurable)
+
+### Cleanup Phase
+
+**Archive File:**
+
+```typescript
+await s3.moveFile(
+ file.path,
+ `${config.s3.archivePrefix}${file.name}`
+);
+```
+
+- Moves file on S3 (atomic operation)
+- Creates destination prefix if needed
+- Original file removed after successful move
+
+**Mark Processed:**
+
+```typescript
+await fileTracker.markFileProcessed(file.name, {
+ recordCount: batchResults.totalSent,
+ batchCount: batchResults.batchCount,
+ jobId: job.id,
+});
+```
+
+- Stores metadata in Versori KV store
+- Metadata optional but useful for audit trails
+- Prevents reprocessing in future runs
+
+**Dispose Resources:**
+
+```typescript
+await s3.dispose();
+```
+
+- Releases S3 connection pool
+- Should be called at end of workflow in `finally` block
+- Ensures proper cleanup
+
+---
+
+## Versori Activation Variables
+
+Configure in Versori platform settings:
+
+```bash
+# S3 Configuration
+S3_BUCKET_NAME=my-inventory-bucket
+AWS_REGION=us-east-1
+AWS_ACCESS_KEY_ID=AKIAXXXXXXXXXXXX
+AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# S3 Paths
+S3_PREFIX=inventory/
+ARCHIVE_PREFIX=processed/
+ERROR_PREFIX=errors/
+LOG_PREFIX=logs/
+
+# File Processing
+FILE_PATTERN=.json
+JSON_FORMAT=json # or 'jsonl' for JSON Lines
+MAX_FILES=10
+
+# Processing Mode Configuration
+# ⚠️ IMPORTANT: Choose ONE mode per connector (these are alternatives, not used together)
+# Options: "per-file" (default, recommended), "batch", "chunked"
+PROCESSING_MODE=per-file
+# For chunked mode only: number of files to process per chunk (default: 5)
+FILE_CHUNK_SIZE=5
+
+# Batch Log Configuration
+LOG_ENABLED=true          # Enable/disable batch log writing (default: true)
+LOG_FORMAT=json           # Log format: json or text (default: json)
+
+# Fluent Configuration (via Versori connection)
+# Connection: fluent_commerce (OAuth2)
+FLUENT_RETAILER_ID=my-retailer
+
+# Batch Processing
+BATCH_SIZE=1000
+
+# BPP Configuration
+# true = Full snapshot (BPP filters unchanged records - DEFAULT)
+# false = Delta feed (skip BPP, all records are changes)
+BPP_ENABLED=true
+
+# Optional Feature Toggles (NEW)
+ENABLE_FILE_TRACKING=true    # Track processed files to prevent duplicates (default: true)
+DETAILED_LOGGING=false       # Enable verbose logging for debugging (default: false)
+```
+
+---
+
+## Batch API Payload Example
+
+What gets sent to Fluent Commerce Batch API:
+
+```json
+{
+ "action": "UPSERT",
+ "entityType": "INVENTORY",
+ "source": "S3_JSON",
+ "event": "InventoryChanged",
+ "entities": [
+  {
+   "retailerId": 1,
+   "locationRef": "LOC001",
+   "skuRef": "SKU-12345",
+   "qty": 100,
+   "type": "LAST_ON_HAND",
+   "status": "ACTIVE",
+   "expectedOn": "2025-01-25T00:00:00.000Z",
+   "attributes": {
+    "expiryDate": "2026-12-31T00:00:00.000Z",
+    "batchNumber": "BATCH-A001",
+    "condition": "NEW",
+    "storageZone": "ZONE-A"
+   }
+  },
+  {
+   "retailerId": 1,
+   "locationRef": "LOC001",
+   "skuRef": "SKU-67890",
+   "qty": 50,
+   "type": "LAST_ON_HAND",
+   "status": "ACTIVE",
+   "expectedOn": "2025-01-25T00:00:00.000Z",
+   "attributes": {
+    "expiryDate": "2026-06-30T00:00:00.000Z",
+    "batchNumber": "BATCH-A002",
+    "condition": "NEW",
+    "storageZone": "ZONE-B"
+   }
+  }
+ ]
+}
+```
+
+> **⚠️ CRITICAL:** Each entity MUST include `retailerId` - it's required in the entity payload, not just in createJob()
+
+---
+
+## Versori Deployment
+
+Use the Versori CLI for deploy and operations:
+
+```bash
+# Install dependencies
+npm install
+
+# Deploy to Versori
+versori deploy
+
+# View logs
+versori logs s3-json-inventory-batch-sync
+
+# Trigger manual run (if defined)
+versori run ingest-now
+```
+
+---
+
+## Testing
+
+### Test Scheduled Batch
+
+Upload a test JSON file to S3 incoming prefix and wait for the scheduled run.
+
+**Check logs:**
+
+```
+[STEP 1/8] Initializing job tracking
+[STEP 2/8] Initializing Fluent Commerce client and S3
+[STEP 3/8] Discovering files on S3
+[FILE 1/1] Processing file: inventory_20250124.json
+[STEP 4/8] Downloading and parsing: inventory_20250124.json
+[STEP 5/8] Transforming 5000 inventory records from inventory_20250124.json
+[STEP 6/8] Creating batch job and sending 5 batches to Fluent Commerce
+[STEP 7/8] Archiving file: inventory_20250124.json
+[STEP 8/8] Completing job and calculating totals
+```
+
+### Test Ad hoc Batch
+
+```bash
+# Process all pending files
+curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{}'
+
+# Process specific pattern
+curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{
+  "filePattern": "urgent_*.json"
+ }'
+
+# Force reprocess
+curl -X POST https://api.versori.com/webhooks/inventory-batch-adhoc \
+ -H "Content-Type: application/json" \
+ -d '{
+  "forceReprocess": true,
+  "filePattern": "inventory_20250124.json"
+ }'
+```
+
+### Test Job Status Query
+
+```bash
+curl -X POST https://api.versori.com/webhooks/inventory-batch-job-status \
+ -H "Content-Type: application/json" \
+ -d '{
+  "jobId": "ADHOC_INV_20251024_183045_abc123"
+ }'
+```
+
+### Verify Batch Job in Fluent
+
+After processing, check the Batch job status in Fluent Commerce:
+
+```bash
+# Upload test file to S3
+aws s3 cp inventory-test.json s3://my-bucket/inventory/
+
+# Query job status via GraphQL
+curl -X POST https://your-fluent-instance.com/graphql \
+ -H "Authorization: Bearer YOUR_TOKEN" \
+ -H "Content-Type: application/json" \
+ -d '{
+  "query": "query { job(id: \"job-123456\") { id status recordCount processedCount } }"
+ }'
+```
+
+---
+
+## Monitoring
+
+### Success Response
+
+```json
+{
+ "success": true,
+ "filesProcessed": 1,
+ "filesSkipped": 0,
+ "filesFailed": 0,
+ "results": [
+  {
+   "file": "inventory_2025-01-22.json",
+   "success": true,
+   "recordCount": 5000,
+   "batchCount": 5,
+   "jobId": "job-123456",
+   "duration": 12345
+  }
+ ],
+ "duration": 13456
+}
+```
+
+### Error Response
+
+```json
+{
+ "success": false,
+ "filesProcessed": 0,
+ "filesFailed": 1,
+ "results": [
+  {
+   "file": "inventory_2025-01-22.json",
+   "success": false,
+   "error": "No valid records after mapping"
+  }
+ ],
+ "duration": 876
+}
+```
+
+---
+
+## Common Pitfalls and Solutions
+
+### 1. JSON Format Detection
+
+❌ **Wrong:**
+
+```typescript
+// Assuming format without checking
+const records = parsed.inventory;
+```
+
+✅ **Correct:**
+
+```typescript
+// Handle both standard JSON and JSON Lines
+let records: any[] = [];
+if (format === 'jsonl') {
+ records = Array.isArray(parsed) ? parsed : [parsed];
+} else {
+ if (Array.isArray(parsed)) records = parsed;
+ else if (parsed?.inventory) records = parsed.inventory;
+ else records = [parsed];
+}
+```
+
+**Why:** JSON files can have different structures; always normalize.
+
+### 2. JSON Field Mapping
+
+❌ **Wrong:**
+
+```json
+{
+ "fields": {
+  "locationRef": { "source": "@locationRef" }
+ }
+}
+```
+
+✅ **Correct:**
+
+```json
+{
+ "fields": {
+  "locationRef": { "source": "locationRef" }
+ }
+}
+```
+
+**Why:** JSON uses direct field access (no `@` prefix like XML attributes).
+
+### 3. Mapping Data Structure
+
+❌ **Wrong:**
+
+```typescript
+// Wrapping JSON record unnecessarily
+const mappingResult = await mapper.map({ inventory: rec });
+```
+
+✅ **Correct:**
+
+```typescript
+// Map JSON record directly
+const mappingResult = await mapper.map(rec);
+```
+
+**Why:** Unlike XML, JSON records don't need wrapper objects for mapping.
+
+### 4. BPP Configuration
+
+❌ **Wrong:**
+
+```typescript
+// Using BPP for delta feeds (wastes processing time)
+const job = await client.createJob({
+ name: 'Hourly Deltas',
+ retailerId: 'my-retailer',
+ // BPP enabled by default - unnecessary for delta feeds
+});
+```
+
+✅ **Correct:**
+
+```typescript
+// Skip BPP for delta feeds (all records are changes)
+const job = await client.createJob({
+ name: 'Hourly Deltas',
+ retailerId: 'my-retailer',
+ meta: {
+  preprocessing: 'skip', // Skip BPP - all records are changes
+ },
+});
+```
+
+**Why:** Delta feeds are pre-filtered by source system, BPP overhead is wasted.
+
+### 5. Large JSON Files
+
+❌ **Wrong:**
+
+```typescript
+// Loading entire 500MB JSON into memory
+const parsed = await parser.parse(largeJsonContent);
+```
+
+✅ **Correct:**
+
+```typescript
+// Use JSON Lines format for streaming
+const format = 'jsonl';
+const parsed = await parser.parse(largeJsonContent, { format });
+```
+
+**Why:** JSON Lines processes one record at a time, preventing memory issues.
+
+---
+
+## Key Takeaways
+
+- 🎯 **Use Batch API for inventory** - Not Event API (Event API is for products/orders/etc.)
+- 🎯 **Processing mode selection** - Per-file (default) for safety, batch for speed, chunked for scale
+- 🎯 **TRUE modular architecture** - Separate service files with clear responsibilities (see Modular Structure section)
+- 🎯 **BPP by default** - Enabled for full snapshots, skip for delta feeds
+- 🎯 **Format detection** - Auto-detects standard JSON vs JSON Lines
+- 🎯 **Direct field access** - Use simple paths like `"locationRef"` (no special notation)
+- 🎯 **Map records directly** - No need to wrap in object (unlike XML)
+- 🎯 **Chunk records** - Default 1000 per batch, tune based on record size
+- 🎯 **EntityType: INVENTORY** - Correct entity type for inventory records
+- 🎯 **State management** - VersoriFileTracker + JobTracker prevent duplicates
+- 🎯 **Enhanced retry logic** - S3 operations with exponential backoff
+- 🎯 **Always dispose** - Release S3 resources with `dispose()` in finally block
+- 🎯 **Error handling** - Move failed files to error folder, don't fail entire workflow
+- 🎯 **Native logging** - Use `log` from context on Versori platform
+- 🎯 **Streaming for large files** - Use JSON Lines format for files >100MB
+- 🎯 **Emoji logging** - Use 🚀 ✅ ❌ ⚠️ 🔍 📄 for visual log scanning
+- 🎯 **Execution boundaries** - Use ═══ separators for workflow start/end
+- 🎯 **validateConnection** - Enable connection validation on client creation
+- 🎯 **Optional toggles** - Support enableFileTracking and detailedLogging flags
+- 🎯 **Duration tracking** - Track execution time at file and workflow levels
+- 🎯 **Error recommendations** - Include actionable suggestions in error logs
+
+---
+
+## Related Documentation
+
+- [Batch API vs Event API Decision Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+- [Universal Mapping Guide](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)
+- [S3 Data Source](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-06-data-sources.md#s3-data-source)
+- [JSON Parser](../../../../../02-CORE-GUIDES/api-reference/modules/api-reference-07-parsers.md#json-parser-service)
+- [State Management](../../../../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-07-state-management.md)
+- [Job Tracker](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-job-tracker.md)
+- [BPP Documentation](../../../../../02-CORE-GUIDES/ingestion/modules/02-core-guides-ingestion-06-batch-api.md#batch-pre-processing-bpp-change-detection)
+
+---
+
+[→ Back to Versori Scheduled Workflows](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md) | [Versori Platform Guide →](../../../../../02-CORE-GUIDES/advanced-services/advanced-services-readme.md)