@fluentcommerce/fc-connect-sdk 0.1.54 → 0.1.56

package/docs/04-REFERENCE/platforms/versori/platforms-versori-workflow-task-types.md

@@ -1,708 +1,708 @@
-# Versori Workflow Task Types - Complete Reference
-
-**@versori/run Package Version**: v0.4.8 (verified from npm)
-**SDK Compatibility**: Use latest @fluentcommerce/fc-connect-sdk
-
----
-
-## Table of Contents
-
-1. [Overview](#overview)
-2. [Task Chaining Methods](#task-chaining-methods)
-3. [Parallel Processing with `.unpack()`](#parallel-processing-with-unpack)
-4. [Serial Processing](#serial-processing)
-5. [Working Examples](#working-examples)
-6. [Performance Characteristics](#performance-characteristics)
-7. [Platform Constraints](#platform-constraints)
-8. [Decision Matrix](#decision-matrix)
-
----
-
-## Overview
-
-Versori's `@versori/run` package provides a fluent API for building workflows with support for parallel and serial array processing. All Task types (`fn`, `http`, `schedule`) support the full chain of methods including `.unpack()` and `.parallel()`.
-
-### Core Task Methods
-
-| Method | Returns | Purpose | Code Location |
-|--------|---------|---------|---------------|
-| `.then()` | `Task` | Chain next task | `Task.d.ts:20` |
-| `.catch()` | `Task` | Handle errors | `Task.d.ts:29` |
-| `.background()` | `Task` | Run in background | `Task.d.ts:34` |
-| `.unpack()` | `ArrayTask` | Split array into elements | `Task.d.ts:35-36` |
-| `.clone()` | `Task` | Clone task | `Task.d.ts:40` |
-
-### ArrayTask Methods (after unpack())
-
-| Method | Returns | Purpose | Code Location |
-|--------|---------|---------|---------------|
-| `.parallel()` | `Task` | Process array elements in parallel | `Task.d.ts:59` |
-| `.serial()` | `Task` | Process array elements sequentially | `Task.d.ts:67` |
-
----
-
-## Task Chaining Methods
-
-### 1. `.then<NextOut>(task)` → `Task<In, NextOut>`
-
-Chain the next task after current task completes.
-
-**Implementation**: `Task.d.ts:20` in `@versori/run` package
-
-```typescript
-schedule('example', '0 * * * *')
-  .then(fn('step1', (ctx) => ({ result: 'data' })))
-  .then(fn('step2', (ctx) => {
-    console.log(ctx.data.result); // 'data'
-    return { next: 'value' };
-  }));
-```
-
-### 2. `.catch<NextOut>(task)` → `Task<In, NextOut>`
-
-Handle errors from previous task. Workflow continues after catch.
-
-**Implementation**: `Task.d.ts:29` in `@versori/run` package
-
-```typescript
-schedule('example', '0 * * * *')
-  .then(fn('may-fail', (ctx) => {
-    throw new Error('Something failed');
-  }))
-  .catch(fn('handle-error', (ctx) => {
-    console.error(ctx.data); // Error object
-    return { recovered: true };
-  }));
-```
-
-### 3. `.background()` → `Task<In, void>`
-
-Run task in background. Next task receives `void` as input.
-
-**Implementation**: `Task.d.ts:34` in `@versori/run` package
-
-```typescript
-schedule('example', '0 * * * *')
-  .then(fn('long-running', (ctx) => {
-    // Long operation
-  }))
-  .background() // Doesn't wait for completion
-  .then(fn('continues-immediately', (ctx) => {
-    // ctx.data is void
-  }));
-```
-
----
-
-## Parallel Processing with unpack()
-
-### `.unpack()` Method
-
-Splits an array output into individual elements, returning an `ArrayTask` that provides `.parallel()` and `.serial()` methods.
-
-**Signatures**:
-```typescript
-// Auto-unpack array
-unpack<NextOut = Out extends Array<infer Elem> ? Elem : Out>(): ArrayTask<In, NextOut>
-
-// Custom mapper
-unpack<NextOut>(mapper: (data: Out) => NextOut[]): ArrayTask<In, NextOut>
-```
-
-**Implementation Locations**:
-- Interface: `Task.d.ts:35-36`
-- FnTask: `FnTask.js:52-57`
-- HttpTask: `HttpTask.js:70-75`
-- UnpackTask class: `UnpackTask.js`
-- Compiler: `unpack.js:21-30`
-
-All files are in the `@versori/run` package under `node_modules/@versori/run/esm/src/`
-
-### `.parallel()` Method
-
-Process each array element concurrently using RxJS `mergeMap`.
-
-**Signature**:
-```typescript
-parallel<NextOut>(task: Taskable<OutElem, NextOut, number>): Task<In, NextOut[]>
-```
-
-**Implementation Locations** in `@versori/run` package:
-- Interface: `Task.d.ts:59`
-- UnpackTask: `UnpackTask.js:44-46`
-- ParallelTask class: `ParallelTask.js:25-47`
-- Compiler: `parallel.js:15-27`
-
-**Key Characteristics**:
-- ✅ **Maintains order**: Results are sorted by original index
-- ✅ **Type-safe**: Full TypeScript support with generics
-- ✅ **No concurrency limit**: Processes all elements simultaneously
-- ⚠️ **Memory usage**: All results held in memory
-- ⚠️ **Error handling**: One failure stops entire workflow
-- ⚠️ **NOT fault-tolerant**: If any task fails, **all results are lost** (no partial results)
-- ⚠️ **NOT ACID compliant**: No atomicity guarantees, no rollback mechanism
-- ⚠️ **Shared timeout**: All parallel tasks share the same workflow timeout (30s/5min)
-
-### ⚠️ CRITICAL: Fault Tolerance Limitations
-
-**Problem**: `.parallel()` uses RxJS `mergeMap` + `toArray()` which stops on first error. This means:
-
-1. **One failure stops everything**: If any parallel task throws an error, the entire workflow fails
-2. **No partial results**: You won't receive successful results even if most tasks succeeded
-3. **No error isolation**: Failed tasks prevent successful tasks from completing
-
-**Example of the problem**:
-
-```typescript
-schedule('process-files', '0 2 * * *')
-  .then(fn('fetch-files', (ctx) => ['file1.xml', 'file2.xml', 'file3.xml']))
-  .unpack()
-  .parallel(fn('process-file', async (ctx) => {
-    // ❌ If file2.xml fails here...
-    return await processFile(ctx.data);
-  }));
-
-// Result:
-// ✅ file1.xml might complete
-// ❌ file2.xml fails → ENTIRE WORKFLOW FAILS
-// ❌ file3.xml NEVER RUNS (or gets cancelled)
-// ❌ No partial results returned
-```
-
-**Why this happens**: The RxJS `toArray()` operator never completes if the stream errors before all items are processed.
-
-### ✅ Recommended: Use `Promise.allSettled()` for Fault Tolerance
-
-For **production workloads** where failures are expected, use `Promise.allSettled()` inside a `fn()` task:
-
-```typescript
-schedule('process-files', '0 2 * * *')
-  .then(fn('process-all-files', async (ctx) => {
-    const files = ['file1.xml', 'file2.xml', 'file3.xml'];
-    
-    // ✅ Fault-tolerant: Continues even if some fail
-    const results = await Promise.allSettled(
-      files.map(file => processFile(file))
-    );
-    
-    // Process results
-    const successes = results
-      .filter(r => r.status === 'fulfilled')
-      .map(r => r.value);
-    
-    const failures = results
-      .filter(r => r.status === 'rejected')
-      .map(r => ({ file: r.reason.file, error: r.reason.message }));
-    
-    ctx.log.info(`Processed ${successes.length} files, ${failures.length} failed`);
-    
-    return { successes, failures };
-  }));
-```
-
-**Comparison**:
-
-| Feature | `.unpack().parallel()` | `Promise.allSettled()` |
-|---------|------------------------|------------------------|
-| Fault tolerance | ❌ Fails on first error | ✅ Continues on errors |
-| Partial results | ❌ No results if any fail | ✅ Returns all results |
-| Error isolation | ❌ No isolation | ✅ Complete isolation |
-| Best for | All-or-nothing workflows | Real-world with failures |
-
-See [Module 4: Workflows](./modules/platforms-versori-04-workflows.md#-critical-parallel-processing-with-unpack-and-parallel) for detailed guidance.
-
-### Execution Engine
-
-The parallel compiler uses RxJS operators:
-
-```javascript
-// From parallel.js:15-27
-function compileParallel(ctx, task) {
-    const baseOperator = ctx.compiler.compileTask(ctx, task._base);
-    const eachOperator = ctx.compiler.compileTask(ctx, task._each);
-
-    return (src) => src.pipe(
-        baseOperator,
-        mergeMap((ctx) => {
-            return from(ctx.data).pipe(
-                map((data, idx) => ctx.withData(data).setIndex(idx)),
-                // Process items in parallel using mergeMap
-                mergeMap((item) => eachOperator(of(item)).pipe(
-                    map((result) => result.setIndex(item.idx))
-                )),
-                toArray(),
-                map((results) => {
-                    // Sort by original index to maintain order
-                    results.sort((a, b) => a.idx - b.idx);
-                    return ctx.withData(results.map((r) => r.data));
-                })
-            );
-        })
-    );
-}
-```
-
----
-
-## Serial Processing
-
-### `.serial()` Method
-
-Process each array element sequentially (one after another).
-
-**Signature**:
-```typescript
-serial<NextOut>(task: Taskable<OutElem, NextOut, number>): Task<In, NextOut[]>
-```
-
-**Implementation** in `@versori/run` package:
-- Interface: `Task.d.ts:67`
-- UnpackTask: `UnpackTask.js:48-50`
-- SerialTask class: `SerialTask.js`
-- Compiler: `serial.js`
-
-**Use When**:
-- Order of execution matters
-- Items depend on previous items
-- Rate limiting required
-- Precise progress tracking needed
-
----
-
-## Working Examples
-
-### Example 1: Parallel File Processing
-
-Process multiple SFTP XML files concurrently:
-
-```typescript
-import { schedule, fn, http } from '@versori/run';
-import { createClient, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
-
-export const parallelFileProcessing = schedule('parallel-files', '0 */6 * * *')
-  .then(fn('list-files', async (ctx) => {
-    const sftp = new SftpDataSource(sftpConfig, ctx.log);
-    await sftp.connect();
-
-    const files = await sftp.listFiles({
-      remotePath: '/incoming/inventory/',
-      filePattern: '.xml'
-    });
-
-    await sftp.disconnect();
-
-    return files; // Array of file metadata
-  }))
-  .unpack() // Split array into individual files
-  .parallel( // Process each file concurrently
-    http('process-file', { connection: 'fluent_commerce' }, async (ctx) => {
-      const file = ctx.data; // Single file metadata
-      const client = await createClient(ctx);
-
-      // Download and process file
-      const sftp = new SftpDataSource(sftpConfig, ctx.log);
-      await sftp.connect();
-      const content = await sftp.downloadFile(file.path);
-      await sftp.disconnect();
-
-      // Parse, map, and send to Fluent
-      const records = await parseAndMap(content);
-      const job = await client.createJob({ name: `Process ${file.name}` });
-      await client.sendBatch(job.id, {
-        action: 'UPSERT',
-        entityType: 'INVENTORY_POSITION',
-        entities: records
-      });
-
-      return { file: file.name, jobId: job.id, recordCount: records.length };
-    })
-  );
-```
-
-**Performance**:
-- 10 files × 30s each = 300s (sequential)
-- 10 files processed simultaneously = 30s (parallel)
-- **10x speedup**
-
-### Example 2: Custom Batching with Mapper
-
-Group items into batches, then process batches in parallel:
-
-```typescript
-export const batchedParallelProcessing = schedule('batched', '0 */6 * * *')
-  .then(fn('fetch-items', (ctx) => {
-    // Fetch 1000 inventory records
-    return fetchAllInventoryRecords(); // Returns array of 1000 items
-  }))
-  .unpack((items) => {
-    // Custom mapper: group into batches of 100
-    const batches = [];
-    for (let i = 0; i < items.length; i += 100) {
-      batches.push(items.slice(i, i + 100));
-    }
-    return batches; // Array of 10 batches
-  })
-  .parallel( // Process each batch concurrently
-    http('process-batch', { connection: 'fluent_commerce' }, async (ctx) => {
-      const batch = ctx.data; // Array of 100 items
-      const client = await createClient(ctx);
-
-      const job = await client.createJob({ name: 'Batch Process' });
-      await client.sendBatch(job.id, {
-        action: 'UPSERT',
-        entityType: 'INVENTORY_POSITION',
-        entities: batch
-      });
-
-      return { batchSize: batch.length, jobId: job.id };
-    })
-  );
-```
-
-### Example 3: Parallel GraphQL Queries
-
-Execute multiple independent GraphQL queries concurrently:
-
-```typescript
-export const parallelQueries = http(
-  'multi-query',
-  { connection: 'fluent_commerce' },
-  async (ctx) => {
-    return [
-      { entity: 'products', query: '...' },
-      { entity: 'locations', query: '...' },
-      { entity: 'inventory', query: '...' }
-    ];
-  }
-)
-  .unpack()
-  .parallel(
-    http('execute-query', { connection: 'fluent_commerce' }, async (ctx) => {
-      const { entity, query } = ctx.data;
-      const client = await createClient(ctx);
-
-      const result = await client.graphql({ query });
-      return { entity, data: result.data };
-    })
-  );
-```
-
-**Performance**:
-- Sequential: 60s + 30s + 30s = 120s
-- Parallel: max(60s, 30s, 30s) = 60s
-- **2x speedup**
-
-### Example 4: Serial Processing (Order Matters)
-
-Process items sequentially when order is critical:
-
-```typescript
-export const serialProcessing = schedule('serial', '0 * * * *')
-  .then(fn('prepare-updates', (ctx) => {
-    return [
-      { action: 'delete', id: '123' },
-      { action: 'create', id: '123', data: {...} },
-      { action: 'update', id: '123', field: 'status' }
-    ];
-  }))
-  .unpack()
-  .serial( // Process sequentially (delete before create)
-    http('execute-action', { connection: 'fluent_commerce' }, async (ctx) => {
-      const action = ctx.data;
-      const client = await createClient(ctx);
-
-      // Execute action based on type
-      return await executeAction(client, action);
-    })
-  );
-```
-
----
-
-## Performance Characteristics
-
-### Memory Usage
-
-| Array Size | Memory Impact | Recommendation |
-|------------|---------------|----------------|
-| 1-10 items | Negligible | ✅ Use parallel |
-| 10-50 items | Low (~1-5MB) | ✅ Use parallel |
-| 50-100 items | Moderate (~5-10MB) | ⚠️ Consider batching |
-| 100-500 items | High (~10-50MB) | ❌ Use SDK Promise.allSettled with concurrency limit |
-| 500+ items | Very High (>50MB) | ❌ Use sequential or chunked approach |
-
-**Note**: Versori runtime has ~512MB memory limit (platform constraint, not SDK limitation).
-
-### Execution Time
-
-```
-Parallel:   T = max(T1, T2, ..., Tn)  // Longest task
-Serial:     T = T1 + T2 + ... + Tn    // Sum of all tasks
-```
-
-**When Parallel is Faster**:
-- Independent tasks
-- Similar execution times
-- Network I/O bound operations
-
-**When Serial is Better**:
-- Dependent tasks
-- Rate-limited APIs
-- Memory constraints
-- Progress tracking required
-
----
-
-## Platform Constraints
-
-### Timeout Limits
-
-**⚠️ Important**: Timeout limits are enforced at the **Versori platform infrastructure level**, not in the `@versori/run` package code.
-
-| Trigger Type | Platform Timeout | Source |
-|--------------|------------------|--------|
-| `http()` | Not explicitly documented | Versori platform |
-| `webhook()` | Not explicitly documented | Versori platform |
-| `schedule()` | Not explicitly documented | Versori platform |
-
-**Recommended Safe Limits** (conservative estimates based on typical cloud function timeouts):
-- `http()`: Assume 30-60 seconds
-- `webhook()`: Assume 30-60 seconds
-- `schedule()`: Assume 5-15 minutes
-
-**How to Verify Your Limits**:
-1. Check Versori platform documentation at https://docs.versori.com
-2. Contact Versori support for your specific workspace limits
-3. Test with long-running workflows to identify actual timeout
-
-**Timeout Not Found in SDK Code**:
-- Searched `@versori/run` package (v0.4.8) source code
-- No hardcoded timeout constants found in:
-  - `http.js`, `webhook.js`, `schedule.js` compilers
-  - Interpreter implementations
-  - Task execution engines
-- Timeouts are platform-enforced, not SDK-defined
-
-### Concurrency Limits
-
-| Method | Concurrency Control | Max Concurrent |
-|--------|---------------------|----------------|
-| `.parallel()` | None (all at once) | Unlimited |
-| `.serial()` | 1 at a time | 1 |
-| SDK `Promise.allSettled` | Configurable | 1-100 |
-
-**Platform-Level Limits** (Versori infrastructure):
-- ~10-50 concurrent HTTP connections per workflow (estimated)
-- Memory: ~512MB per workflow execution (estimated)
-- CPU: Shared among concurrent tasks
-
----
-
-## Decision Matrix
-
-### When to Use `.unpack().parallel()`
-
-✅ **Use When**:
-- 3-50 independent items to process
-- Each item takes similar time (10-60s)
-- Total estimated time < 4 minutes (safe buffer for platform timeout)
-- Downstream APIs can handle concurrent requests
-- All-or-nothing outcome is acceptable
-- **Examples**: Multi-file SFTP ingestion, parallel GraphQL queries, multi-source data fetch
-
-❌ **Don't Use When**:
-- Need bounded concurrency (limit to 5-10 concurrent)
-- Items have variable processing time (1s to 5min)
-- Need partial success handling
-- Downstream has strict rate limits
-- Need custom retry logic per item
-- **Alternative**: Use SDK `Promise.allSettled` with bounded concurrency
-
-### When to Use `.unpack().serial()`
-
-✅ **Use When**:
-- Order of execution matters
-- Items depend on previous items
-- Rate-limited APIs (1 req/sec)
-- Precise progress tracking needed
-- **Examples**: Database migrations, sequential state updates, ordered batch operations
-
-❌ **Don't Use When**:
-- Items are independent
-- Need speed over order
-- Processing 100+ items (too slow)
-- **Alternative**: Use `.parallel()` or chunked approach
-
-### When to Use SDK Promise.allSettled
-
-✅ **Use When**:
-- Need bounded concurrency (e.g., 5-10 concurrent)
-- Need per-item error handling
-- Want partial success (commit what works)
-- Have custom retry logic
-- **Examples**: Batch API submissions, Event API calls (already in SDK templates)
-
-**Current SDK Usage**:
-- `BatchProcessorService`: Already uses `Promise.allSettled` with configurable concurrency (1-10)
-- `EventSenderService`: Already uses `Promise.allSettled` for parallel event submission
-- **No changes needed** - already optimal!
-
----
-
-## Recommended Parallel Processing Patterns
-
-### Pattern 1: Small, Reliable, All-or-Nothing (Use `.parallel()`)
-
-```typescript
-// ✅ Good use case: Fetching required entity types
-export const fetchRequiredData = http(
-  'fetch-all',
-  { connection: 'fluent_commerce' },
-  async (ctx) => {
-    return [
-      { entity: 'PRODUCT', query: '...' },
-      { entity: 'LOCATION', query: '...' },
-      { entity: 'CATEGORY', query: '...' }
-    ];
-  }
-)
-  .unpack()
-  .parallel(  // ✅ Okay here - all 3 must succeed
-    http('execute', { connection: 'fluent_commerce' }, async (ctx) => {
-      const client = await createClient(ctx);
-      return await client.graphql({ query: ctx.data.query });
-    })
-  );
-```
-
-**When this fails:** You probably want the workflow to fail anyway since all entities are required.
-
----
-
-### Pattern 2: Large, Unreliable, Partial Success (Use `Promise.allSettled`)
-
-```typescript
-// ✅ Better approach: Batch file processing
-export const batchFileIngestion = schedule('ingest', '0 */6 * * *')
-  .then(
-    http('process-all', { connection: 'fluent_commerce' }, async (ctx) => {
-      const sftp = new SftpDataSource(config, ctx.log);
-      const files = await sftp.listFiles('/incoming/');
-
-      // ✅ Use Promise.allSettled for fault tolerance
-      const results = await Promise.allSettled(
-        files.map(async (file) => {
-          try {
-            const content = await sftp.downloadFile(file.path);
-            const records = await parseXML(content);
-
-            const client = await createClient(ctx);
-            const job = await client.createJob({ name: `Process ${file.name}` });
-            await client.sendBatch(job.id, {
-              action: 'UPSERT',
-              entityType: 'INVENTORY_POSITION',
-              entities: records
-            });
-
-            return { success: true, file: file.name, count: records.length };
-          } catch (error) {
-            return { success: false, file: file.name, error: error.message };
-          }
-        })
-      );
-
-      const successes = results
-        .filter(r => r.status === 'fulfilled' && r.value.success)
-        .map(r => r.value);
-
-      const failures = results
-        .filter(r => r.status === 'rejected' || !r.value?.success);
-
-      // ✅ Partial success is valuable
-      ctx.log.info(`Processed ${successes.length}/${files.length} files`);
-
-      return { successes, failures };
-    })
-  );
-```
-
-**When this fails:** You still get partial results (processed 45/50 files successfully).
-
----
-
-### Pattern 3: `.parallel()` with Safe Error Handling
-
-```typescript
-// ✅ If you MUST use .parallel(), wrap errors
-export const safeParallel = schedule('process', '0 2 * * *')
-  .then(fn('fetch', () => ['file1.xml', 'file2.xml', 'file3.xml']))
-  .unpack()
-  .parallel(fn('process', async (ctx) => {
-    try {
-      const result = await processFile(ctx.data);
-      return { success: true, file: ctx.data, result };
-    } catch (error) {
-      // ✅ Return error object instead of throwing
-      return { success: false, file: ctx.data, error: error.message };
-    }
-  }))
-  .then(fn('handle-results', (ctx) => {
-    const successes = ctx.data.filter(r => r.success);
-    const failures = ctx.data.filter(r => !r.success);
-
-    if (failures.length > 0) {
-      ctx.log.error(`${failures.length} files failed`, failures);
-    }
-
-    return { successes, failures };
-  }));
-```
-
-**Note:** This pattern requires careful error handling in EVERY parallel task. `Promise.allSettled` is simpler and more reliable.
-
----
-
-## Compiler Registration
-
-All task types are registered in the ObservableCompiler:
-
-**Location**: `ObservableCompiler.js:40-46` in `@versori/run` package
-
-```javascript
-// Register all available compilers
-this.registerTask(parallelCompiler);
-this.registerTask(serialCompiler);
-this.registerTask(catchCompiler);
-this.registerTask(chainCompiler);
-this.registerTask(fnCompiler);
-this.registerTask(backgroundCompiler);
-this.registerTask(unpackCompiler);
-this.registerTask(httpCompiler);
-```
-
----
-
-## Summary
-
-- ✅ `.unpack()` and `.parallel()` **are fully supported** in @versori/run v0.4.0+
-- ✅ Verified from actual npm package source code
-- ✅ Implemented using RxJS `mergeMap` for true parallel execution
-- ✅ Maintains order in output array
-- ⚠️ No built-in concurrency limits - processes all items simultaneously
-- ⚠️ Platform timeout limits enforced by Versori infrastructure (not SDK)
-- ⚠️ Best for 3-50 independent items with similar execution times
-- ✅ For Batch/Event API: SDK's `Promise.allSettled` approach is already optimal
-
----
-
-## Related Documentation
-
-- [Module 4: Workflows](./modules/platforms-versori-04-workflows.md) - Core workflow patterns
-- Versori Run SDK TypeScript Definitions - Available in `@versori/run` package at `node_modules/@versori/run/esm/src/dsl/Task.d.ts`
-- SDK Batch Processing Service - See source at `src/services/orchestration/ingestion-service.ts`
-
----
-
-**Last Verified**: 2025-01-04
-**Package Version**: @versori/run@0.4.8
-**Source**: npm registry + local node_modules inspection
+# Versori Workflow Task Types - Complete Reference
+
+**@versori/run Package Version**: v0.4.8 (verified from npm)
+**SDK Compatibility**: Use latest @fluentcommerce/fc-connect-sdk
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Task Chaining Methods](#task-chaining-methods)
+3. [Parallel Processing with `.unpack()`](#parallel-processing-with-unpack)
+4. [Serial Processing](#serial-processing)
+5. [Working Examples](#working-examples)
+6. [Performance Characteristics](#performance-characteristics)
+7. [Platform Constraints](#platform-constraints)
+8. [Decision Matrix](#decision-matrix)
+
+---
+
+## Overview
+
+Versori's `@versori/run` package provides a fluent API for building workflows with support for parallel and serial array processing. All Task types (`fn`, `http`, `schedule`) support the full chain of methods including `.unpack()` and `.parallel()`.
+
+### Core Task Methods
+
+| Method | Returns | Purpose | Code Location |
+|--------|---------|---------|---------------|
+| `.then()` | `Task` | Chain next task | `Task.d.ts:20` |
+| `.catch()` | `Task` | Handle errors | `Task.d.ts:29` |
+| `.background()` | `Task` | Run in background | `Task.d.ts:34` |
+| `.unpack()` | `ArrayTask` | Split array into elements | `Task.d.ts:35-36` |
+| `.clone()` | `Task` | Clone task | `Task.d.ts:40` |
+
+### ArrayTask Methods (after unpack())
+
+| Method | Returns | Purpose | Code Location |
+|--------|---------|---------|---------------|
+| `.parallel()` | `Task` | Process array elements in parallel | `Task.d.ts:59` |
+| `.serial()` | `Task` | Process array elements sequentially | `Task.d.ts:67` |
+
+---
+
+## Task Chaining Methods
+
+### 1. `.then<NextOut>(task)` → `Task<In, NextOut>`
+
+Chain the next task after current task completes.
+
+**Implementation**: `Task.d.ts:20` in `@versori/run` package
+
+```typescript
+schedule('example', '0 * * * *')
+  .then(fn('step1', (ctx) => ({ result: 'data' })))
+  .then(fn('step2', (ctx) => {
+    console.log(ctx.data.result); // 'data'
+    return { next: 'value' };
+  }));
+```
+
+### 2. `.catch<NextOut>(task)` → `Task<In, NextOut>`
+
+Handle errors from previous task. Workflow continues after catch.
+
+**Implementation**: `Task.d.ts:29` in `@versori/run` package
+
+```typescript
+schedule('example', '0 * * * *')
+  .then(fn('may-fail', (ctx) => {
+    throw new Error('Something failed');
+  }))
+  .catch(fn('handle-error', (ctx) => {
+    console.error(ctx.data); // Error object
+    return { recovered: true };
+  }));
+```
+
+### 3. `.background()` → `Task<In, void>`
+
+Run task in background. Next task receives `void` as input.
+
+**Implementation**: `Task.d.ts:34` in `@versori/run` package
+
+```typescript
+schedule('example', '0 * * * *')
+  .then(fn('long-running', (ctx) => {
+    // Long operation
+  }))
+  .background() // Doesn't wait for completion
+  .then(fn('continues-immediately', (ctx) => {
+    // ctx.data is void
+  }));
+```
+
+---
+
+## Parallel Processing with unpack()
+
+### `.unpack()` Method
+
+Splits an array output into individual elements, returning an `ArrayTask` that provides `.parallel()` and `.serial()` methods.
+
+**Signatures**:
+```typescript
+// Auto-unpack array
+unpack<NextOut = Out extends Array<infer Elem> ? Elem : Out>(): ArrayTask<In, NextOut>
+
+// Custom mapper
+unpack<NextOut>(mapper: (data: Out) => NextOut[]): ArrayTask<In, NextOut>
+```
+
+**Implementation Locations**:
+- Interface: `Task.d.ts:35-36`
+- FnTask: `FnTask.js:52-57`
+- HttpTask: `HttpTask.js:70-75`
+- UnpackTask class: `UnpackTask.js`
+- Compiler: `unpack.js:21-30`
+
+All files are in the `@versori/run` package under `node_modules/@versori/run/esm/src/`
+
+### `.parallel()` Method
+
+Process each array element concurrently using RxJS `mergeMap`.
+
+**Signature**:
+```typescript
+parallel<NextOut>(task: Taskable<OutElem, NextOut, number>): Task<In, NextOut[]>
+```
+
+**Implementation Locations** in `@versori/run` package:
+- Interface: `Task.d.ts:59`
+- UnpackTask: `UnpackTask.js:44-46`
+- ParallelTask class: `ParallelTask.js:25-47`
+- Compiler: `parallel.js:15-27`
+
+**Key Characteristics**:
+- ✅ **Maintains order**: Results are sorted by original index
+- ✅ **Type-safe**: Full TypeScript support with generics
+- ✅ **No concurrency limit**: Processes all elements simultaneously
+- ⚠️ **Memory usage**: All results held in memory
+- ⚠️ **Error handling**: One failure stops entire workflow
+- ⚠️ **NOT fault-tolerant**: If any task fails, **all results are lost** (no partial results)
+- ⚠️ **NOT ACID compliant**: No atomicity guarantees, no rollback mechanism
+- ⚠️ **Shared timeout**: All parallel tasks share the same workflow timeout (30s/5min)
+
+### ⚠️ CRITICAL: Fault Tolerance Limitations
+
+**Problem**: `.parallel()` uses RxJS `mergeMap` + `toArray()` which stops on first error. This means:
+
+1. **One failure stops everything**: If any parallel task throws an error, the entire workflow fails
+2. **No partial results**: You won't receive successful results even if most tasks succeeded
+3. **No error isolation**: Failed tasks prevent successful tasks from completing
+
+**Example of the problem**:
+
+```typescript
+schedule('process-files', '0 2 * * *')
+  .then(fn('fetch-files', (ctx) => ['file1.xml', 'file2.xml', 'file3.xml']))
+  .unpack()
+  .parallel(fn('process-file', async (ctx) => {
+    // ❌ If file2.xml fails here...
+    return await processFile(ctx.data);
+  }));
+
+// Result:
+// ✅ file1.xml might complete
+// ❌ file2.xml fails → ENTIRE WORKFLOW FAILS
+// ❌ file3.xml NEVER RUNS (or gets cancelled)
+// ❌ No partial results returned
+```
+
+**Why this happens**: The RxJS `toArray()` operator never completes if the stream errors before all items are processed.
+
+### ✅ Recommended: Use `Promise.allSettled()` for Fault Tolerance
+
+For **production workloads** where failures are expected, use `Promise.allSettled()` inside a `fn()` task:
+
+```typescript
+schedule('process-files', '0 2 * * *')
+  .then(fn('process-all-files', async (ctx) => {
+    const files = ['file1.xml', 'file2.xml', 'file3.xml'];
+    
+    // ✅ Fault-tolerant: Continues even if some fail
+    const results = await Promise.allSettled(
+      files.map(file => processFile(file))
+    );
+    
+    // Process results
+    const successes = results
+      .filter(r => r.status === 'fulfilled')
+      .map(r => r.value);
+    
+    const failures = results
+      .filter(r => r.status === 'rejected')
+      .map(r => ({ file: r.reason.file, error: r.reason.message }));
+    
+    ctx.log.info(`Processed ${successes.length} files, ${failures.length} failed`);
+    
+    return { successes, failures };
+  }));
+```
+
+**Comparison**:
+
+| Feature | `.unpack().parallel()` | `Promise.allSettled()` |
+|---------|------------------------|------------------------|
+| Fault tolerance | ❌ Fails on first error | ✅ Continues on errors |
+| Partial results | ❌ No results if any fail | ✅ Returns all results |
+| Error isolation | ❌ No isolation | ✅ Complete isolation |
+| Best for | All-or-nothing workflows | Real-world with failures |
+
+See [Module 4: Workflows](./modules/platforms-versori-04-workflows.md#-critical-parallel-processing-with-unpack-and-parallel) for detailed guidance.
+
+### Execution Engine
+
+The parallel compiler uses RxJS operators:
+
+```javascript
+// From parallel.js:15-27
+function compileParallel(ctx, task) {
+    const baseOperator = ctx.compiler.compileTask(ctx, task._base);
+    const eachOperator = ctx.compiler.compileTask(ctx, task._each);
+
+    return (src) => src.pipe(
+        baseOperator,
+        mergeMap((ctx) => {
+            return from(ctx.data).pipe(
+                map((data, idx) => ctx.withData(data).setIndex(idx)),
+                // Process items in parallel using mergeMap
+                mergeMap((item) => eachOperator(of(item)).pipe(
+                    map((result) => result.setIndex(item.idx))
+                )),
+                toArray(),
+                map((results) => {
+                    // Sort by original index to maintain order
+                    results.sort((a, b) => a.idx - b.idx);
+                    return ctx.withData(results.map((r) => r.data));
+                })
+            );
+        })
+    );
+}
+```
+
+---
+
+## Serial Processing
+
+### `.serial()` Method
+
+Process each array element sequentially (one after another).
+
+**Signature**:
+```typescript
+serial<NextOut>(task: Taskable<OutElem, NextOut, number>): Task<In, NextOut[]>
+```
+
+**Implementation** in `@versori/run` package:
+- Interface: `Task.d.ts:67`
+- UnpackTask: `UnpackTask.js:48-50`
+- SerialTask class: `SerialTask.js`
+- Compiler: `serial.js`
+
+**Use When**:
+- Order of execution matters
+- Items depend on previous items
+- Rate limiting required
+- Precise progress tracking needed
+
+---
+
+## Working Examples
+
+### Example 1: Parallel File Processing
+
+Process multiple SFTP XML files concurrently:
+
+```typescript
+import { schedule, fn, http } from '@versori/run';
+import { createClient, SftpDataSource } from '@fluentcommerce/fc-connect-sdk';
+
+export const parallelFileProcessing = schedule('parallel-files', '0 */6 * * *')
+  .then(fn('list-files', async (ctx) => {
+    const sftp = new SftpDataSource(sftpConfig, ctx.log);
+    await sftp.connect();
+
+    const files = await sftp.listFiles({
+      remotePath: '/incoming/inventory/',
+      filePattern: '.xml'
+    });
+
+    await sftp.disconnect();
+
+    return files; // Array of file metadata
+  }))
+  .unpack() // Split array into individual files
+  .parallel( // Process each file concurrently
+    http('process-file', { connection: 'fluent_commerce' }, async (ctx) => {
+      const file = ctx.data; // Single file metadata
+      const client = await createClient(ctx);
+
+      // Download and process file
+      const sftp = new SftpDataSource(sftpConfig, ctx.log);
+      await sftp.connect();
+      const content = await sftp.downloadFile(file.path);
+      await sftp.disconnect();
+
+      // Parse, map, and send to Fluent
+      const records = await parseAndMap(content);
+      const job = await client.createJob({ name: `Process ${file.name}` });
+      await client.sendBatch(job.id, {
+        action: 'UPSERT',
+        entityType: 'INVENTORY_POSITION',
+        entities: records
+      });
+
+      return { file: file.name, jobId: job.id, recordCount: records.length };
+    })
+  );
+```
+
+**Performance**:
+- 10 files × 30s each = 300s (sequential)
+- 10 files processed simultaneously = 30s (parallel)
+- **10x speedup**
+
+### Example 2: Custom Batching with Mapper
+
+Group items into batches, then process batches in parallel:
+
+```typescript
+export const batchedParallelProcessing = schedule('batched', '0 */6 * * *')
+  .then(fn('fetch-items', (ctx) => {
+    // Fetch 1000 inventory records
+    return fetchAllInventoryRecords(); // Returns array of 1000 items
+  }))
+  .unpack((items) => {
+    // Custom mapper: group into batches of 100
+    const batches = [];
+    for (let i = 0; i < items.length; i += 100) {
+      batches.push(items.slice(i, i + 100));
+    }
+    return batches; // Array of 10 batches
+  })
+  .parallel( // Process each batch concurrently
+    http('process-batch', { connection: 'fluent_commerce' }, async (ctx) => {
+      const batch = ctx.data; // Array of 100 items
+      const client = await createClient(ctx);
+
+      const job = await client.createJob({ name: 'Batch Process' });
+      await client.sendBatch(job.id, {
+        action: 'UPSERT',
+        entityType: 'INVENTORY_POSITION',
+        entities: batch
+      });
+
+      return { batchSize: batch.length, jobId: job.id };
+    })
+  );
+```
+
+### Example 3: Parallel GraphQL Queries
+
+Execute multiple independent GraphQL queries concurrently:
+
+```typescript
+export const parallelQueries = http(
+  'multi-query',
+  { connection: 'fluent_commerce' },
+  async (ctx) => {
+    return [
+      { entity: 'products', query: '...' },
+      { entity: 'locations', query: '...' },
+      { entity: 'inventory', query: '...' }
+    ];
+  }
+)
+  .unpack()
+  .parallel(
+    http('execute-query', { connection: 'fluent_commerce' }, async (ctx) => {
+      const { entity, query } = ctx.data;
+      const client = await createClient(ctx);
+
+      const result = await client.graphql({ query });
+      return { entity, data: result.data };
+    })
+  );
+```
+
+**Performance**:
+- Sequential: 60s + 30s + 30s = 120s
+- Parallel: max(60s, 30s, 30s) = 60s
+- **2x speedup**
+
+### Example 4: Serial Processing (Order Matters)
+
+Process items sequentially when order is critical:
+
+```typescript
+export const serialProcessing = schedule('serial', '0 * * * *')
+  .then(fn('prepare-updates', (ctx) => {
+    return [
+      { action: 'delete', id: '123' },
+      { action: 'create', id: '123', data: {...} },
+      { action: 'update', id: '123', field: 'status' }
+    ];
+  }))
+  .unpack()
+  .serial( // Process sequentially (delete before create)
+    http('execute-action', { connection: 'fluent_commerce' }, async (ctx) => {
+      const action = ctx.data;
+      const client = await createClient(ctx);
+
+      // Execute action based on type
+      return await executeAction(client, action);
+    })
+  );
+```
+
+---
+
+## Performance Characteristics
+
+### Memory Usage
+
+| Array Size | Memory Impact | Recommendation |
+|------------|---------------|----------------|
+| 1-10 items | Negligible | ✅ Use parallel |
+| 10-50 items | Low (~1-5MB) | ✅ Use parallel |
+| 50-100 items | Moderate (~5-10MB) | ⚠️ Consider batching |
+| 100-500 items | High (~10-50MB) | ❌ Use SDK Promise.allSettled with concurrency limit |
+| 500+ items | Very High (>50MB) | ❌ Use sequential or chunked approach |
+
+**Note**: Versori runtime has ~512MB memory limit (platform constraint, not SDK limitation).
+
+### Execution Time
+
+```
+Parallel:   T = max(T1, T2, ..., Tn)  // Longest task
+Serial:     T = T1 + T2 + ... + Tn    // Sum of all tasks
+```
+
+**When Parallel is Faster**:
+- Independent tasks
+- Similar execution times
+- Network I/O bound operations
+
+**When Serial is Better**:
+- Dependent tasks
+- Rate-limited APIs
+- Memory constraints
+- Progress tracking required
+
+---
+
+## Platform Constraints
+
+### Timeout Limits
+
+**⚠️ Important**: Timeout limits are enforced at the **Versori platform infrastructure level**, not in the `@versori/run` package code.
+
+| Trigger Type | Platform Timeout | Source |
+|--------------|------------------|--------|
+| `http()` | Not explicitly documented | Versori platform |
+| `webhook()` | Not explicitly documented | Versori platform |
+| `schedule()` | Not explicitly documented | Versori platform |
+
+**Recommended Safe Limits** (conservative estimates based on typical cloud function timeouts):
+- `http()`: Assume 30-60 seconds
+- `webhook()`: Assume 30-60 seconds
+- `schedule()`: Assume 5-15 minutes
+
+**How to Verify Your Limits**:
+1. Check Versori platform documentation at https://docs.versori.com
+2. Contact Versori support for your specific workspace limits
+3. Test with long-running workflows to identify actual timeout
+
+**Timeout Not Found in SDK Code**:
+- Searched `@versori/run` package (v0.4.8) source code
+- No hardcoded timeout constants found in:
+  - `http.js`, `webhook.js`, `schedule.js` compilers
+  - Interpreter implementations
+  - Task execution engines
+- Timeouts are platform-enforced, not SDK-defined
+
+### Concurrency Limits
+
+| Method | Concurrency Control | Max Concurrent |
+|--------|---------------------|----------------|
+| `.parallel()` | None (all at once) | Unlimited |
+| `.serial()` | 1 at a time | 1 |
+| SDK `Promise.allSettled` | Configurable | 1-100 |
+
+**Platform-Level Limits** (Versori infrastructure):
+- ~10-50 concurrent HTTP connections per workflow (estimated)
+- Memory: ~512MB per workflow execution (estimated)
+- CPU: Shared among concurrent tasks
+
+---
+
+## Decision Matrix
+
+### When to Use `.unpack().parallel()`
+
+✅ **Use When**:
+- 3-50 independent items to process
+- Each item takes similar time (10-60s)
+- Total estimated time < 4 minutes (safe buffer for platform timeout)
+- Downstream APIs can handle concurrent requests
+- All-or-nothing outcome is acceptable
+- **Examples**: Multi-file SFTP ingestion, parallel GraphQL queries, multi-source data fetch
+
+❌ **Don't Use When**:
+- Need bounded concurrency (limit to 5-10 concurrent)
+- Items have variable processing time (1s to 5min)
+- Need partial success handling
+- Downstream has strict rate limits
+- Need custom retry logic per item
+- **Alternative**: Use SDK `Promise.allSettled` with bounded concurrency
+
+### When to Use `.unpack().serial()`
+
+✅ **Use When**:
+- Order of execution matters
+- Items depend on previous items
+- Rate-limited APIs (1 req/sec)
+- Precise progress tracking needed
+- **Examples**: Database migrations, sequential state updates, ordered batch operations
+
+❌ **Don't Use When**:
+- Items are independent
+- Need speed over order
+- Processing 100+ items (too slow)
+- **Alternative**: Use `.parallel()` or chunked approach
+
+### When to Use SDK Promise.allSettled
+
+✅ **Use When**:
+- Need bounded concurrency (e.g., 5-10 concurrent)
+- Need per-item error handling
+- Want partial success (commit what works)
+- Have custom retry logic
+- **Examples**: Batch API submissions, Event API calls (already in SDK templates)
+
+**Current SDK Usage**:
+- `BatchProcessorService`: Already uses `Promise.allSettled` with configurable concurrency (1-10)
+- `EventSenderService`: Already uses `Promise.allSettled` for parallel event submission
+- **No changes needed** - already optimal!
+
+---
+
+## Recommended Parallel Processing Patterns
+
+### Pattern 1: Small, Reliable, All-or-Nothing (Use `.parallel()`)
+
+```typescript
+// ✅ Good use case: Fetching required entity types
+export const fetchRequiredData = http(
+  'fetch-all',
+  { connection: 'fluent_commerce' },
+  async (ctx) => {
+    return [
+      { entity: 'PRODUCT', query: '...' },
+      { entity: 'LOCATION', query: '...' },
+      { entity: 'CATEGORY', query: '...' }
+    ];
+  }
+)
+  .unpack()
+  .parallel(  // ✅ Okay here - all 3 must succeed
+    http('execute', { connection: 'fluent_commerce' }, async (ctx) => {
+      const client = await createClient(ctx);
+      return await client.graphql({ query: ctx.data.query });
+    })
+  );
+```
+
+**When this fails:** You probably want the workflow to fail anyway since all entities are required.
+
+---
+
+### Pattern 2: Large, Unreliable, Partial Success (Use `Promise.allSettled`)
+
+```typescript
+// ✅ Better approach: Batch file processing
+export const batchFileIngestion = schedule('ingest', '0 */6 * * *')
+  .then(
+    http('process-all', { connection: 'fluent_commerce' }, async (ctx) => {
+      const sftp = new SftpDataSource(config, ctx.log);
+      const files = await sftp.listFiles('/incoming/');
+
+      // ✅ Use Promise.allSettled for fault tolerance
+      const results = await Promise.allSettled(
+        files.map(async (file) => {
+          try {
+            const content = await sftp.downloadFile(file.path);
+            const records = await parseXML(content);
+
+            const client = await createClient(ctx);
+            const job = await client.createJob({ name: `Process ${file.name}` });
+            await client.sendBatch(job.id, {
+              action: 'UPSERT',
+              entityType: 'INVENTORY_POSITION',
+              entities: records
+            });
+
+            return { success: true, file: file.name, count: records.length };
+          } catch (error) {
+            return { success: false, file: file.name, error: error.message };
+          }
+        })
+      );
+
+      const successes = results
+        .filter(r => r.status === 'fulfilled' && r.value.success)
+        .map(r => r.value);
+
+      const failures = results
+        .filter(r => r.status === 'rejected' || !r.value?.success);
+
+      // ✅ Partial success is valuable
+      ctx.log.info(`Processed ${successes.length}/${files.length} files`);
+
+      return { successes, failures };
+    })
+  );
+```
+
+**When this fails:** You still get partial results (processed 45/50 files successfully).
+
+---
+
+### Pattern 3: `.parallel()` with Safe Error Handling
+
+```typescript
+// ✅ If you MUST use .parallel(), wrap errors
+export const safeParallel = schedule('process', '0 2 * * *')
+  .then(fn('fetch', () => ['file1.xml', 'file2.xml', 'file3.xml']))
+  .unpack()
+  .parallel(fn('process', async (ctx) => {
+    try {
+      const result = await processFile(ctx.data);
+      return { success: true, file: ctx.data, result };
+    } catch (error) {
+      // ✅ Return error object instead of throwing
+      return { success: false, file: ctx.data, error: error.message };
+    }
+  }))
+  .then(fn('handle-results', (ctx) => {
+    const successes = ctx.data.filter(r => r.success);
+    const failures = ctx.data.filter(r => !r.success);
+
+    if (failures.length > 0) {
+      ctx.log.error(`${failures.length} files failed`, failures);
+    }
+
+    return { successes, failures };
+  }));
+```
+
+**Note:** This pattern requires careful error handling in EVERY parallel task. `Promise.allSettled` is simpler and more reliable.
+
+---
+
+## Compiler Registration
+
+All task types are registered in the ObservableCompiler:
+
+**Location**: `ObservableCompiler.js:40-46` in `@versori/run` package
+
+```javascript
+// Register all available compilers
+this.registerTask(parallelCompiler);
+this.registerTask(serialCompiler);
+this.registerTask(catchCompiler);
+this.registerTask(chainCompiler);
+this.registerTask(fnCompiler);
+this.registerTask(backgroundCompiler);
+this.registerTask(unpackCompiler);
+this.registerTask(httpCompiler);
+```
+
+---
+
+## Summary
+
+- ✅ `.unpack()` and `.parallel()` **are fully supported** in @versori/run v0.4.0+
+- ✅ Verified from actual npm package source code
+- ✅ Implemented using RxJS `mergeMap` for true parallel execution
+- ✅ Maintains order in output array
+- ⚠️ No built-in concurrency limits - processes all items simultaneously
+- ⚠️ Platform timeout limits enforced by Versori infrastructure (not SDK)
+- ⚠️ Best for 3-50 independent items with similar execution times
+- ✅ For Batch/Event API: SDK's `Promise.allSettled` approach is already optimal
+
+---
+
+## Related Documentation
+
+- [Module 4: Workflows](./modules/platforms-versori-04-workflows.md) - Core workflow patterns
+- Versori Run SDK TypeScript Definitions - Available in `@versori/run` package at `node_modules/@versori/run/esm/src/dsl/Task.d.ts`
+- SDK Batch Processing Service - See source at `src/services/orchestration/ingestion-service.ts`
+
+---
+
+**Last Verified**: 2025-01-04
+**Package Version**: @versori/run@0.4.8
+**Source**: npm registry + local node_modules inspection