yaml-flow 2.0.0 → 2.2.0

package/README.md

@@ -509,11 +509,322 @@ interface StepMachineStore {
 
 ---
 
+## Batch Processing
+
+yaml-flow includes a `batch()` utility for running multiple items through a flow concurrently. It works with both Step Machine and Event Graph — you provide the processor, it manages concurrency.
+
+### Quick Start
+
+```typescript
+import { batch } from 'yaml-flow/batch';
+import { createStepMachine } from 'yaml-flow/step-machine';
+
+const tickets = [
+  { id: 'T-001', message: 'Billing error' },
+  { id: 'T-002', message: 'App crashes on login' },
+  { id: 'T-003', message: 'Password reset help' },
+];
+
+const result = await batch(tickets, {
+  concurrency: 3,
+  processor: async (ticket) => {
+    const machine = createStepMachine(flow, handlers);
+    return machine.run({ message: ticket.message });
+  },
+  onProgress: (p) => console.log(`${p.percent}% done`),
+});
+
+console.log(`${result.completed} succeeded, ${result.failed} failed`);
+```
+
+### Options
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `concurrency` | `number` | `5` | Max parallel processors |
+| `processor` | `(item, index) => Promise<TResult>` | *required* | Async function to process each item |
+| `signal` | `AbortSignal` | — | Cancel remaining items |
+| `onItemComplete` | `(item, result, index) => void` | — | Called when an item succeeds |
+| `onItemError` | `(item, error, index) => void` | — | Called when an item fails |
+| `onProgress` | `(progress) => void` | — | Called after each item with `{ completed, failed, active, pending, total, percent, elapsedMs }` |
+
+### Result Shape
+
+```typescript
+{
+  items: BatchItemResult[];  // Per-item: { item, index, status, result?, error?, durationMs }
+  completed: number;         // Items that succeeded
+  failed: number;            // Items that threw
+  total: number;
+  durationMs: number;        // Wall-clock time for entire batch
+}
+```
+
+### Works with Event Graph too
+
+```typescript
+import { batch } from 'yaml-flow/batch';
+import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+
+const result = await batch(items, {
+  concurrency: 5,
+  processor: async (item) => {
+    let state = createInitialExecutionState(graph, `run-${item.id}`);
+    state = apply(state, { type: 'inject-tokens', tokens: [item.readyToken], timestamp: Date.now() }, graph);
+    // ... drive graph loop with next() + apply()
+    return state;
+  },
+});
+```
+
+---
+
+## Config Utilities
+
+Pure pre-processing transforms you apply before passing config to the engine. They never touch engine state — just config in, config out.
+
+### Variable Interpolation
+
+Replace `${KEY}` patterns in any config object. Works with both GraphConfig and StepFlowConfig.
+
+```typescript
+import { resolveVariables } from 'yaml-flow/config';
+
+const resolved = resolveVariables(graphConfig, {
+  ENTITY_ID: 'ticket-42',
+  TOOLS_DIR: '/opt/tools',
+  WORKDIR: '/data/workdata',
+});
+// Every ${ENTITY_ID} in task configs, cmd-args, etc. → replaced
+```
+
+### Config Templates
+
+DRY reusable config blocks. Tasks reference a named template via `config-template`; the function deep-merges template + task overrides and removes the reference.
+
+```typescript
+import { resolveConfigTemplates } from 'yaml-flow/config';
+
+const config = {
+  configTemplates: {                              // or 'config-templates' (kebab-case)
+    PYTHON_TOOL: { cmd: 'python', timeout: 30000, cwd: '/workdata' },
+    NODE_CMD:    { cmd: 'node',   timeout: 60000 },
+  },
+  tasks: {
+    analyze:  { provides: ['analysis'], config: { 'config-template': 'PYTHON_TOOL', 'cmd-args': 'analyze.py' } },
+    build:    { provides: ['build'],    config: { 'config-template': 'NODE_CMD',    script: 'build.js' } },
+  },
+};
+
+const resolved = resolveConfigTemplates(config);
+// analyze.config → { cmd: 'python', timeout: 30000, cwd: '/workdata', 'cmd-args': 'analyze.py' }
+// configTemplates key removed from output
+```
+
+### Composing Both
+
+Templates first (expands references), then variables (fills in `${...}` placeholders):
+
+```typescript
+import { resolveConfigTemplates, resolveVariables } from 'yaml-flow/config';
+
+const raw = loadYaml('pipeline.yaml');                  // has configTemplates + ${VAR} refs
+const resolved = resolveVariables(
+  resolveConfigTemplates(raw),
+  { ENTITY_ID: 'url-42', TOOLS_DIR: '/opt/tools' },
+);
+```
+
+---
+
+## Graph-of-Graphs Pattern
+
+Real-world pipelines are often **layered**: an outer orchestration graph where some tasks are themselves entire sub-workflows — each processing a batch of items through their own DAG or step flow. yaml-flow doesn't bake this into the engine (the pure scheduler stays simple), but the primitives compose cleanly.
+
+### The Shape
+
+```
+Outer graph (event-graph)
+├── prep-workdata          → plain task
+├── copy-input-files       → plain task
+├── evidence-gathering     → batch × inner event-graph (N items, 5 concurrent)
+├── grade-synthesis        → batch × inner step-machine (N items, 3 concurrent)
+├── analyze-mismatches     → plain task
+├── [health-check ∥ report]→ parallel plain tasks
+└── archive-results        → waits for both
+```
+
+The outer graph sequences coarse stages. Some stages fan out into batches where each item runs through its own sub-workflow. The sub can be either mode.
+
+### How to Wire It
+
+Each "sub-graph task" in the outer graph is just a handler that composes `resolveConfigTemplates` → `resolveVariables` → `batch` → engine:
+
+```typescript
+import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+import { createStepMachine } from 'yaml-flow/step-machine';
+import { batch } from 'yaml-flow/batch';
+import { resolveVariables, resolveConfigTemplates } from 'yaml-flow/config';
+
+// Outer graph handler for a sub-graph task (event-graph sub)
+async function runEvidenceBatch(items, rawSubConfig) {
+  return batch(items, {
+    concurrency: 5,
+    processor: async (item) => {
+      // Resolve config per-item (each item gets its own ENTITY_ID)
+      const config = resolveVariables(
+        resolveConfigTemplates(rawSubConfig),
+        { ENTITY_ID: item.id, TOOLS_DIR: '/opt/tools' },
+      );
+      // Drive the inner event-graph
+      let state = createInitialExecutionState(config, `run-${item.id}`);
+      while (true) {
+        const { eligibleTasks, isComplete } = next(config, state);
+        if (isComplete) break;
+        for (const task of eligibleTasks) {
+          state = apply(state, { type: 'task-started', taskName: task, timestamp: new Date().toISOString() }, config);
+          const result = await executeTask(task, config.tasks[task], item);
+          state = apply(state, { type: 'task-completed', taskName: task, result, timestamp: new Date().toISOString() }, config);
+        }
+      }
+      return state;
+    },
+  });
+}
+
+// Outer graph handler for a sub-graph task (step-machine sub)
+async function runGradeBatch(items, flowConfig, handlers) {
+  return batch(items, {
+    concurrency: 3,
+    processor: async (item) => {
+      const machine = createStepMachine(flowConfig, handlers);
+      return machine.run({ entityId: item.id, evidence: item.evidence });
+    },
+  });
+}
+```
+
+### Driving the Outer Graph
+
+The outer graph itself is an event-graph. Each handler maps to a task:
+
+```typescript
+const outerHandlers = {
+  'prep-workdata':     async () => { /* setup */ },
+  'copy-input-files':  async () => { /* parse CSV, return items */ },
+  'evidence-batch':    async (ctx) => runEvidenceBatch(ctx.items, evidenceConfig),
+  'grade-batch':       async (ctx) => runGradeBatch(ctx.items, gradeFlow, gradeHandlers),
+  'analyze-mismatches':async (ctx) => { /* compare grades */ },
+  'health-check':      async (ctx) => { /* validate */ },
+  'generate-report':   async (ctx) => { /* summarize */ },
+  'archive':           async (ctx) => { /* move outputs */ },
+};
+
+// Simple outer loop
+let state = createInitialExecutionState(outerGraph, 'pipeline-run-1');
+while (true) {
+  const { eligibleTasks, isComplete } = next(outerGraph, state);
+  if (isComplete) break;
+  await Promise.all(eligibleTasks.map(async (taskName) => {
+    state = apply(state, { type: 'task-started', taskName, timestamp: now() }, outerGraph);
+    try {
+      await outerHandlers[taskName](context);
+      state = apply(state, { type: 'task-completed', taskName, timestamp: now() }, outerGraph);
+    } catch (err) {
+      state = apply(state, { type: 'task-failed', taskName, error: err.message, timestamp: now() }, outerGraph);
+    }
+  }));
+}
+```
+
+### Why Not Bake It Into the Engine?
+
+- The pure scheduler (`next`/`apply`) stays a simple `f(state, event) → newState`.
+- Sub-graph execution involves file I/O, process spawning, HTTP calls — all driver concerns.
+- Every deployment customizes how sub-tasks execute: in-process, `execSync`, HTTP, serverless.
+- The primitives (`batch` + `resolveVariables` + `resolveConfigTemplates` + both engines) compose without coupling.
+
+See the [examples/graph-of-graphs/](./examples/graph-of-graphs/) directory for complete runnable examples.
+
+---
+
+## Execution Plan (Dry Run)
+
+Compute the full execution plan from a graph config without running anything — like `terraform plan` for workflows.
+
+```typescript
+import { planExecution } from 'yaml-flow/event-graph';
+
+const plan = planExecution(graph);
+
+plan.phases;          // [['prep'], ['copy'], ['evidence'], ['synthesis'], ['analyze'], ['health', 'report'], ['archive']]
+plan.depth;           // 7
+plan.maxParallelism;  // 2
+plan.entryPoints;     // ['prep']
+plan.leafTasks;       // ['archive']
+plan.conflicts;       // { 'output-token': ['task-a', 'task-b'] }  — multiple producers
+plan.unreachableTokens; // ['human-approval']  — required but no task produces it
+plan.blockedTasks;    // ['approve']  — blocked by unreachable tokens
+plan.dependencies;    // { 'copy': ['prep'], 'evidence': ['copy'], ... }
+```
+
+---
+
+## Mermaid Diagrams
+
+Generate Mermaid syntax from any config — useful for docs, debugging, and CI reports.
+
+```typescript
+import { graphToMermaid, flowToMermaid } from 'yaml-flow/event-graph';
+
+// Event graph → dependency diagram
+console.log(graphToMermaid(graph));
+// graph TD
+//   build([build])
+//   test[test]
+//   deploy[[deploy]]
+//   build -->|artifact| test
+//   test -->|tested| deploy
+
+// Step machine → flowchart
+console.log(flowToMermaid(flow));
+// graph TD
+//   START(( ))
+//   START --> classify
+//   classify -->|billing| handle
+//   handle -->|resolved| done
+//   done([done: resolved])
+```
+
+Options: `{ direction: 'LR' | 'TD', showTokens: boolean, title: string }`.  
+Entry points (no requires) get rounded shapes, leaf tasks get double-bracketed shapes, unreachable deps get warning markers.
+
+---
+
+## Loading & Exporting Graph Configs
+
+```typescript
+import { loadGraphConfig, exportGraphConfig, exportGraphConfigToFile } from 'yaml-flow/event-graph';
+
+// Load from file, URL, JSON string, or object (validates automatically)
+const graph = await loadGraphConfig('./pipeline.yaml');
+const graph2 = await loadGraphConfig('https://example.com/graph.json');
+
+// Export to string
+const json = exportGraphConfig(graph);                         // JSON (default)
+const yaml = exportGraphConfig(graph, { format: 'yaml' });     // YAML
+
+// Export to file (format auto-detected from extension)
+await exportGraphConfigToFile(graph, './output/pipeline.yaml');
+```
+
+---
+
 ## Package Exports
 
 ```typescript
-// Everything (both modes + stores)
-import { StepMachine, next, apply, MemoryStore } from 'yaml-flow';
+// Everything (both modes + stores + batch)
+import { StepMachine, next, apply, MemoryStore, batch } from 'yaml-flow';
 
 // Step Machine only
 import { StepMachine, createStepMachine, loadStepFlow } from 'yaml-flow/step-machine';
@@ -522,11 +833,20 @@ import { applyStepResult, checkCircuitBreaker, createInitialState } from 'yaml-f
 // Event Graph only
 import { next, apply, applyAll, getCandidateTasks } from 'yaml-flow/event-graph';
 import { createInitialExecutionState, isExecutionComplete, detectStuckState } from 'yaml-flow/event-graph';
+import { planExecution, graphToMermaid, flowToMermaid } from 'yaml-flow/event-graph';
+import { loadGraphConfig, validateGraphConfig, exportGraphConfig } from 'yaml-flow/event-graph';
 import { TASK_STATUS, COMPLETION_STRATEGIES, CONFLICT_STRATEGIES } from 'yaml-flow/event-graph';
 
 // Stores
 import { MemoryStore, LocalStorageStore, FileStore } from 'yaml-flow/stores';
 
+// Batch
+import { batch } from 'yaml-flow/batch';
+import type { BatchOptions, BatchResult, BatchItemResult, BatchProgress } from 'yaml-flow/batch';
+
+// Config utilities
+import { resolveVariables, resolveConfigTemplates } from 'yaml-flow/config';
+
 // Backward compatibility (v1 names → v2)
 import { FlowEngine, createEngine } from 'yaml-flow';  // aliases for StepMachine, createStepMachine
 ```
@@ -562,6 +882,13 @@ import { FlowEngine, createEngine } from 'yaml-flow';  // aliases for StepMachin
 | `isExecutionComplete(graph, state)` | Check completion against configured strategy |
 | `detectStuckState({graph, state, ...})` | Check if execution is stuck |
 | `addDynamicTask(graph, name, config)` | Immutably add a task to a graph config |
+| `planExecution(graph)` | Dry-run: compute phases, parallelism, conflicts, unreachable tokens |
+| `graphToMermaid(graph, options?)` | Generate Mermaid dependency diagram from an event-graph |
+| `flowToMermaid(flow, options?)` | Generate Mermaid flowchart from a step-machine |
+| `loadGraphConfig(source)` | Load + validate a YAML/JSON/URL graph config |
+| `validateGraphConfig(config)` | Validate a GraphConfig, returns error strings |
+| `exportGraphConfig(config, options?)` | Export a GraphConfig to JSON or YAML string |
+| `exportGraphConfigToFile(config, path)` | Export a GraphConfig to a file |
 
 ### Event Types (for `apply()`)
 
@@ -587,6 +914,9 @@ See the [examples/](./examples) directory:
 | [AI Conversation](./examples/node/ai-conversation.ts) | Step Machine | Retry, circuit breakers, component injection |
 | [Research Pipeline](./examples/event-graph/research-pipeline.ts) | Event Graph | Parallel tasks, goal-based completion |
 | [CI/CD Pipeline](./examples/event-graph/ci-cd-pipeline.ts) | Event Graph | External events, conditional routing, failure tokens |
+| [Batch Tickets](./examples/batch/batch-step-machine.ts) | Batch | Concurrent processing, progress tracking |
+| [URL Pipeline](./examples/graph-of-graphs/url-processing-pipeline.ts) | Graph-of-Graphs | Outer event-graph → batch × inner event-graph per item |
+| [Multi-Stage ETL](./examples/graph-of-graphs/multi-stage-etl.ts) | Graph-of-Graphs | Mixed modes: event-graph outer → step-machine + event-graph subs |
 | [Order Processing](./examples/flows/order-processing.yaml) | Step Machine | YAML flow definition |
 | [Browser Demo](./examples/browser/index.html) | Step Machine | In-browser usage |
 

package/dist/batch/index.cjs

@@ -0,0 +1,109 @@
+'use strict';
+
+// src/batch/runner.ts
+async function batch(items, options) {
+  const {
+    concurrency = 5,
+    processor,
+    onItemComplete,
+    onItemError,
+    onProgress,
+    signal
+  } = options;
+  const total = items.length;
+  const results = new Array(total);
+  const batchStart = Date.now();
+  let completed = 0;
+  let failed = 0;
+  let nextIndex = 0;
+  function makeProgress(active) {
+    const done = completed + failed;
+    return {
+      completed,
+      failed,
+      active,
+      pending: total - done - active,
+      total,
+      percent: total === 0 ? 100 : Math.round(done / total * 100),
+      elapsedMs: Date.now() - batchStart
+    };
+  }
+  if (total === 0) {
+    return { items: [], completed: 0, failed: 0, total: 0, durationMs: 0 };
+  }
+  return new Promise((resolve) => {
+    let active = 0;
+    function tryStartNext() {
+      while (active < concurrency && nextIndex < total) {
+        if (signal?.aborted) {
+          while (nextIndex < total) {
+            const idx2 = nextIndex++;
+            results[idx2] = {
+              item: items[idx2],
+              index: idx2,
+              status: "failed",
+              error: new Error("Batch aborted"),
+              durationMs: 0
+            };
+            failed++;
+          }
+          if (active === 0 && completed + failed === total) {
+            resolve({
+              items: results,
+              completed,
+              failed,
+              total,
+              durationMs: Date.now() - batchStart
+            });
+          }
+          break;
+        }
+        const idx = nextIndex++;
+        const item = items[idx];
+        active++;
+        const itemStart = Date.now();
+        processor(item, idx).then((result) => {
+          results[idx] = {
+            item,
+            index: idx,
+            status: "completed",
+            result,
+            durationMs: Date.now() - itemStart
+          };
+          completed++;
+          onItemComplete?.(item, result, idx);
+        }).catch((err) => {
+          const error = err instanceof Error ? err : new Error(String(err));
+          results[idx] = {
+            item,
+            index: idx,
+            status: "failed",
+            error,
+            durationMs: Date.now() - itemStart
+          };
+          failed++;
+          onItemError?.(item, error, idx);
+        }).finally(() => {
+          active--;
+          onProgress?.(makeProgress(active));
+          if (completed + failed === total) {
+            resolve({
+              items: results,
+              completed,
+              failed,
+              total,
+              durationMs: Date.now() - batchStart
+            });
+          } else {
+            tryStartNext();
+          }
+        });
+      }
+    }
+    tryStartNext();
+  });
+}
+
+exports.batch = batch;
+//# sourceMappingURL=index.cjs.map
+//# sourceMappingURL=index.cjs.map

package/dist/batch/index.cjs.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../../src/batch/runner.ts"],"names":["idx"],"mappings":";;;AAoDA,eAAsB,KAAA,CACpB,OACA,OAAA,EACsC;AACtC,EAAA,MAAM;AAAA,IACJ,WAAA,GAAc,CAAA;AAAA,IACd,SAAA;AAAA,IACA,cAAA;AAAA,IACA,WAAA;AAAA,IACA,UAAA;AAAA,IACA;AAAA,GACF,GAAI,OAAA;AAEJ,EAAA,MAAM,QAAQ,KAAA,CAAM,MAAA;AACpB,EAAA,MAAM,OAAA,GAA6C,IAAI,KAAA,CAAM,KAAK,CAAA;AAClE,EAAA,MAAM,UAAA,GAAa,KAAK,GAAA,EAAI;AAE5B,EAAA,IAAI,SAAA,GAAY,CAAA;AAChB,EAAA,IAAI,MAAA,GAAS,CAAA;AACb,EAAA,IAAI,SAAA,GAAY,CAAA;AAEhB,EAAA,SAAS,aAAa,MAAA,EAA+B;AACnD,IAAA,MAAM,OAAO,SAAA,GAAY,MAAA;AACzB,IAAA,OAAO;AAAA,MACL,SAAA;AAAA,MACA,MAAA;AAAA,MACA,MAAA;AAAA,MACA,OAAA,EAAS,QAAQ,IAAA,GAAO,MAAA;AAAA,MACxB,KAAA;AAAA,MACA,OAAA,EAAS,UAAU,CAAA,GAAI,GAAA,GAAM,KAAK,KAAA,CAAO,IAAA,GAAO,QAAS,GAAG,CAAA;AAAA,MAC5D,SAAA,EAAW,IAAA,CAAK,GAAA,EAAI,GAAI;AAAA,KAC1B;AAAA,EACF;AAGA,EAAA,IAAI,UAAU,CAAA,EAAG;AACf,IAAA,OAAO,EAAE,KAAA,EAAO,EAAC,EAAG,SAAA,EAAW,CAAA,EAAG,MAAA,EAAQ,CAAA,EAAG,KAAA,EAAO,CAAA,EAAG,UAAA,EAAY,CAAA,EAAE;AAAA,EACvE;AAEA,EAAA,OAAO,IAAI,OAAA,CAAqC,CAAC,OAAA,KAAY;AAC3D,IAAA,IAAI,MAAA,GAAS,CAAA;AAEb,IAAA,SAAS,YAAA,GAAe;AACtB,MAAA,OAAO,MAAA,GAAS,WAAA,IAAe,SAAA,GAAY,KAAA,EAAO;AAEhD,QAAA,IAAI,QAAQ,OAAA,EAAS;AAEnB,UAAA,OAAO,YAAY,KAAA,EAAO;AACxB,YAAA,MAAMA,IAAAA,GAAM,SAAA,EAAA;AACZ,YAAA,OAAA,CAAQA,IAAG,CAAA,GAAI;AAAA,cACb,IAAA,EAAM,MAAMA,IAAG,CAAA;AAAA,cACf,KAAA,EAAOA,IAAAA;AAAA,cACP,MAAA,EAAQ,QAAA;AAAA,cACR,KAAA,EAAO,IAAI,KAAA,CAAM,eAAe,CAAA;AAAA,cAChC,UAAA,EAAY;AAAA,aACd;AACA,YAAA,MAAA,EAAA;AAAA,UACF;AAEA,UAAA,IAAI,MAAA,KAAW,CAAA,IAAK,SAAA,GAAY,MAAA,KAAW,KAAA,EAAO;AAChD,YAAA,OAAA,CAAQ;AAAA,cACN,KAAA,EAAO,OAAA;AAAA,cACP,SAAA;AAAA,cACA,MAAA;AAAA,cACA,KAAA;AAAA,cACA,UAAA,EAAY,IAAA,CAAK,GAAA,EAAI,GAAI;AAAA,aAC1B,CAAA;AAAA,UACH;AACA,UAAA;AAAA,QACF;AAEA,QAAA,MAAM,GAAA,GAAM,SAAA,EAAA;AACZ,QAAA,MAAM,IAAA,GAAO,MAAM,GAAG,CAAA;AACtB,QAAA,MAAA,EAAA;AACA,QAAA,MAAM,SAAA,GAAY,KAAK,GAAA,EAAI;AAE3B,QAAA,SAAA,CAAU,IAAA,EAAM,GAAG,CAAA,CAChB,IAAA,CAAK,CAAC,MAAA,KAAW;AAChB,UAAA,OAAA,CAAQ,GAAG,CAAA,GAAI;AAAA,YACb,IAAA;AAAA,YACA,KAAA,EAAO,GAAA;AAAA,YACP,MAAA,EAAQ,WAAA;AAAA,YACR,MAAA;AAAA,YACA,UAAA,EAAY,IAAA,CAAK,GAAA,EAAI,GAAI;AAAA,WAC3B;AACA,UAAA,SAAA,EAAA;AACA,UAAA,cAAA,GAAiB,IAAA,EAAM,QAAQ,GAAG,CAAA;AAAA,QACpC,CAAC,CAAA,CACA,KAAA,CAAM,CAAC,GAAA,KAAQ;AACd,UAAA,MAAM,KAAA,GAAQ,eAAe,KAAA,GAAQ,GAAA,GAAM,IAAI,KAAA,CAAM,MAAA,CAAO,GAAG,CAAC,CAAA;AAChE,UAAA,OAAA,CAAQ,GAAG,CAAA,GAAI;AAAA,YACb,IAAA;AAAA,YACA,KAAA,EAAO,GAAA;AAAA,YACP,MAAA,EAAQ,QAAA;AAAA,YACR,KAAA;AAAA,YACA,UAAA,EAAY,IAAA,CAAK,GAAA,EAAI,GAAI;AAAA,WAC3B;AACA,UAAA,MAAA,EAAA;AACA,UAAA,WAAA,GAAc,IAAA,EAAM,OAAO,GAAG,CAAA;AAAA,QAChC,CAAC,CAAA,CACA,OAAA,CAAQ,MAAM;AACb,UAAA,MAAA,EAAA;AACA,UAAA,UAAA,GAAa,YAAA,CAAa,MAAM,CAAC,CAAA;AAEjC,UAAA,IAAI,SAAA,GAAY,WAAW,KAAA,EAAO;AAChC,YAAA,OAAA,CAAQ;AAAA,cACN,KAAA,EAAO,OAAA;AAAA,cACP,SAAA;AAAA,cACA,MAAA;AAAA,cACA,KAAA;AAAA,cACA,UAAA,EAAY,IAAA,CAAK,GAAA,EAAI,GAAI;AAAA,aAC1B,CAAA;AAAA,UACH,CAAA,MAAO;AACL,YAAA,YAAA,EAAa;AAAA,UACf;AAAA,QACF,CAAC,CAAA;AAAA,MACL;AAAA,IACF;AAEA,IAAA,YAAA,EAAa;AAAA,EACf,CAAC,CAAA;AACH","file":"index.cjs","sourcesContent":["/**\n * Batch Runner — Core\n *\n * Slot-based concurrent processor. Pure control flow — no I/O opinions.\n *\n * @example Step Machine batch\n * ```ts\n * import { batch } from 'yaml-flow/batch';\n * import { createStepMachine, loadStepFlow } from 'yaml-flow/step-machine';\n *\n * const flow = await loadStepFlow('./support-ticket.yaml');\n * const results = await batch(tickets, {\n *   concurrency: 5,\n *   processor: async (ticket) => {\n *     const machine = createStepMachine(flow, handlers);\n *     return machine.run(ticket);\n *   },\n * });\n * ```\n *\n * @example Event Graph batch\n * ```ts\n * import { batch } from 'yaml-flow/batch';\n * import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';\n *\n * const results = await batch(items, {\n *   concurrency: 3,\n *   processor: async (item, index) => {\n *     let state = createInitialExecutionState(graph, `exec-${index}`);\n *     state = apply(state, { type: 'inject-tokens', tokens: [item.token], timestamp: new Date().toISOString() }, graph);\n *     // ... drive the graph loop\n *     return state;\n *   },\n * });\n * ```\n */\n\nimport type {\n  BatchOptions,\n  BatchResult,\n  BatchItemResult,\n  BatchProgress,\n} from './types.js';\n\n/**\n * Run an array of items through an async processor with concurrency control.\n *\n * - Items are started in order, up to `concurrency` at a time.\n * - Results are returned in the original item order.\n * - If a processor throws, the item is marked as failed; other items continue.\n * - An AbortSignal prevents new items from starting (in-flight items are not cancelled).\n */\nexport async function batch<TItem, TResult>(\n  items: TItem[],\n  options: BatchOptions<TItem, TResult>\n): Promise<BatchResult<TItem, TResult>> {\n  const {\n    concurrency = 5,\n    processor,\n    onItemComplete,\n    onItemError,\n    onProgress,\n    signal,\n  } = options;\n\n  const total = items.length;\n  const results: BatchItemResult<TItem, TResult>[] = new Array(total);\n  const batchStart = Date.now();\n\n  let completed = 0;\n  let failed = 0;\n  let nextIndex = 0;\n\n  function makeProgress(active: number): BatchProgress {\n    const done = completed + failed;\n    return {\n      completed,\n      failed,\n      active,\n      pending: total - done - active,\n      total,\n      percent: total === 0 ? 100 : Math.round((done / total) * 100),\n      elapsedMs: Date.now() - batchStart,\n    };\n  }\n\n  // Empty input — short-circuit\n  if (total === 0) {\n    return { items: [], completed: 0, failed: 0, total: 0, durationMs: 0 };\n  }\n\n  return new Promise<BatchResult<TItem, TResult>>((resolve) => {\n    let active = 0;\n\n    function tryStartNext() {\n      while (active < concurrency && nextIndex < total) {\n        // Respect abort signal — don't start new items\n        if (signal?.aborted) {\n          // Mark remaining as failed with abort error\n          while (nextIndex < total) {\n            const idx = nextIndex++;\n            results[idx] = {\n              item: items[idx],\n              index: idx,\n              status: 'failed',\n              error: new Error('Batch aborted'),\n              durationMs: 0,\n            };\n            failed++;\n          }\n          // If nothing is in-flight, resolve immediately\n          if (active === 0 && completed + failed === total) {\n            resolve({\n              items: results,\n              completed,\n              failed,\n              total,\n              durationMs: Date.now() - batchStart,\n            });\n          }\n          break;\n        }\n\n        const idx = nextIndex++;\n        const item = items[idx];\n        active++;\n        const itemStart = Date.now();\n\n        processor(item, idx)\n          .then((result) => {\n            results[idx] = {\n              item,\n              index: idx,\n              status: 'completed',\n              result,\n              durationMs: Date.now() - itemStart,\n            };\n            completed++;\n            onItemComplete?.(item, result, idx);\n          })\n          .catch((err) => {\n            const error = err instanceof Error ? err : new Error(String(err));\n            results[idx] = {\n              item,\n              index: idx,\n              status: 'failed',\n              error,\n              durationMs: Date.now() - itemStart,\n            };\n            failed++;\n            onItemError?.(item, error, idx);\n          })\n          .finally(() => {\n            active--;\n            onProgress?.(makeProgress(active));\n\n            if (completed + failed === total) {\n              resolve({\n                items: results,\n                completed,\n                failed,\n                total,\n                durationMs: Date.now() - batchStart,\n              });\n            } else {\n              tryStartNext();\n            }\n          });\n      }\n    }\n\n    tryStartNext();\n  });\n}\n"]}

package/dist/batch/index.d.cts

@@ -0,0 +1,126 @@
+/**
+ * Batch Runner — Types
+ *
+ * Generic concurrent batch processor.
+ * Works with both step-machine and event-graph (or any async processor).
+ */
+interface BatchOptions<TItem, TResult> {
+    /**
+     * Max concurrent items in flight (slots).
+     * @default 5
+     */
+    concurrency?: number;
+    /**
+     * The async function that processes a single item.
+     * Receives the item and its 0-based index.
+     */
+    processor: (item: TItem, index: number) => Promise<TResult>;
+    /**
+     * Called when a single item completes successfully.
+     */
+    onItemComplete?: (item: TItem, result: TResult, index: number) => void;
+    /**
+     * Called when a single item fails (processor threw).
+     * If not provided, the error is captured in BatchItemResult.
+     */
+    onItemError?: (item: TItem, error: Error, index: number) => void;
+    /**
+     * Called after every item settles (success or failure).
+     * Receives a snapshot of progress.
+     */
+    onProgress?: (progress: BatchProgress) => void;
+    /**
+     * AbortSignal — if aborted, no new items are started.
+     * Items already in-flight are NOT cancelled (your processor should check its own signal).
+     */
+    signal?: AbortSignal;
+}
+interface BatchProgress {
+    /** Items completed successfully so far */
+    completed: number;
+    /** Items that threw an error */
+    failed: number;
+    /** Items currently in-flight */
+    active: number;
+    /** Items not yet started */
+    pending: number;
+    /** Total items */
+    total: number;
+    /** Percentage complete (0–100) */
+    percent: number;
+    /** Elapsed time in ms since batch started */
+    elapsedMs: number;
+}
+interface BatchResult<TItem, TResult> {
+    /** All item results in original order */
+    items: BatchItemResult<TItem, TResult>[];
+    /** Summary counts */
+    completed: number;
+    failed: number;
+    total: number;
+    /** Total wall-clock time in ms */
+    durationMs: number;
+}
+interface BatchItemResult<TItem, TResult> {
+    /** Original item */
+    item: TItem;
+    /** 0-based index in the input array */
+    index: number;
+    /** 'completed' or 'failed' */
+    status: 'completed' | 'failed';
+    /** Result if completed */
+    result?: TResult;
+    /** Error if failed */
+    error?: Error;
+    /** Per-item wall-clock time in ms */
+    durationMs: number;
+}
+
+/**
+ * Batch Runner — Core
+ *
+ * Slot-based concurrent processor. Pure control flow — no I/O opinions.
+ *
+ * @example Step Machine batch
+ * ```ts
+ * import { batch } from 'yaml-flow/batch';
+ * import { createStepMachine, loadStepFlow } from 'yaml-flow/step-machine';
+ *
+ * const flow = await loadStepFlow('./support-ticket.yaml');
+ * const results = await batch(tickets, {
+ *   concurrency: 5,
+ *   processor: async (ticket) => {
+ *     const machine = createStepMachine(flow, handlers);
+ *     return machine.run(ticket);
+ *   },
+ * });
+ * ```
+ *
+ * @example Event Graph batch
+ * ```ts
+ * import { batch } from 'yaml-flow/batch';
+ * import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+ *
+ * const results = await batch(items, {
+ *   concurrency: 3,
+ *   processor: async (item, index) => {
+ *     let state = createInitialExecutionState(graph, `exec-${index}`);
+ *     state = apply(state, { type: 'inject-tokens', tokens: [item.token], timestamp: new Date().toISOString() }, graph);
+ *     // ... drive the graph loop
+ *     return state;
+ *   },
+ * });
+ * ```
+ */
+
+/**
+ * Run an array of items through an async processor with concurrency control.
+ *
+ * - Items are started in order, up to `concurrency` at a time.
+ * - Results are returned in the original item order.
+ * - If a processor throws, the item is marked as failed; other items continue.
+ * - An AbortSignal prevents new items from starting (in-flight items are not cancelled).
+ */
+declare function batch<TItem, TResult>(items: TItem[], options: BatchOptions<TItem, TResult>): Promise<BatchResult<TItem, TResult>>;
+
+export { type BatchItemResult, type BatchOptions, type BatchProgress, type BatchResult, batch };