yaml-flow 1.0.0 → 2.1.0

package/README.md

@@ -1,225 +1,504 @@
 # yaml-flow
 
-A lightweight, isomorphic workflow engine with declarative YAML flows and pluggable persistence.
+Two workflow engines in one package. Pick the model that fits your problem.
 
 [![npm version](https://badge.fury.io/js/yaml-flow.svg)](https://www.npmjs.com/package/yaml-flow)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-## Features
+```
+npm install yaml-flow
+```
 
-- **Isomorphic** — Runs in both browser and Node.js
-- **Declarative Flows** — Define workflows in YAML with JSON Schema validation
-- **Pure Function Handlers** — Step handlers are simple `(input, context) => result` functions
-- **Pluggable Storage** — Bring your own persistence (memory, localStorage, file, Redis, etc.)
-- **Zero Core Dependencies** — Lightweight core, optional add-ons for YAML parsing
-- **Resumable** — Pause and resume flows from persisted state
-- **Circuit Breakers** — Prevent infinite loops with configurable limits
-- **Retry Logic** — Built-in exponential backoff for failed steps
-- **Event System** — Subscribe to flow events for UI updates
+## Which Mode Do I Need?
 
-## Installation
+yaml-flow ships two execution models. They solve fundamentally different problems.
 
-```bash
-npm install yaml-flow
-```
+| | **Step Machine** | **Event Graph** |
+|---|---|---|
+| **Mental model** | Flowchart — each step decides what runs next | Dependency graph — tasks self-select when their inputs are ready |
+| **Core function** | `applyStepResult(flow, state, step, result) → newState` | `next(graph, state) → eligibleTasks` + `apply(state, event, graph) → newState` |
+| **Config shape** | `steps:` with `transitions:` mapping | `tasks:` with `requires:` / `provides:` arrays |
+| **Who decides next?** | The sender (current step's result picks the transition) | The receivers (tasks become eligible when their required tokens appear) |
+| **Has a driver?** | Yes — `StepMachine` class runs the loop for you | No — you call `next()` and `apply()` yourself (or write a 10-line loop) |
+| **Import** | `import { StepMachine } from 'yaml-flow/step-machine'` | `import { next, apply } from 'yaml-flow/event-graph'` |
+
+### Use Step Machine when…
+
+- Your workflow is a **known sequence** of steps with conditional branching.
+- You need **pause/resume** built in.
+- The work is conversational: receive input → process → decide next step → repeat.
+- You want to hand the library a YAML file and some handler functions and call `.run()`.
+
+**Examples:** form wizards, approval chains, AI chat loops, order processing pipelines, ETL with linear stages.
+
+### Use Event Graph when…
+
+- Tasks have **complex dependencies** — diamonds, fan-out, fan-in, optional paths.
+- You don't know the execution order up front; it **emerges from the data**.
+- Multiple tasks can run **in parallel** once their inputs are satisfied.
+- You need **conflict resolution** when two tasks compete to produce the same output.
+- Your system is **event-driven**: external events inject tokens that unblock downstream tasks.
+- You want the engine to be a **pure function** you embed in your own scheduler, serverless function, or agent loop.
+
+**Examples:** CI/CD pipelines, agent tool orchestration, research workflows (fetch → analyse A | analyse B → merge), build systems, multi-model AI routing, eligibility engines.
 
-## Quick Start
+### When in doubt
 
-### 1. Define Your Flow (YAML)
+Start with **Step Machine** if your workflow diagram is a straight line with branches.  
+Start with **Event Graph** if your workflow diagram has diamonds or parallel lanes.
+
+Both are pure `f(state, input) → newState` at their core. You can always call the reducer directly without the driver class.
+
+---
+
+## Step Machine — Quick Start
+
+### 1. Define a flow
 
 ```yaml
-# my-flow.yaml
+# support-ticket.yaml
 settings:
-  start_step: greet
-  max_total_steps: 10
+  start_step: classify
+  max_total_steps: 20
 
 steps:
-  greet:
-    produces_data:
-      - message
+  classify:
+    produces_data: [category, priority]
+    transitions:
+      billing: handle_billing
+      technical: handle_technical
+      general: handle_general
+
+  handle_billing:
+    expects_data: [category]
+    produces_data: [resolution]
     transitions:
-      success: done
-      failure: error
+      resolved: close_ticket
+      escalate: escalate_ticket
+
+  handle_technical:
+    expects_data: [category]
+    produces_data: [resolution]
+    transitions:
+      resolved: close_ticket
+      escalate: escalate_ticket
+    retry:
+      max_attempts: 2
+      delay_ms: 1000
+
+  handle_general:
+    expects_data: [category]
+    produces_data: [resolution]
+    transitions:
+      resolved: close_ticket
+
+  escalate_ticket:
+    expects_data: [category, priority]
+    produces_data: [escalation_id]
+    transitions:
+      done: close_ticket
 
 terminal_states:
-  done:
-    return_intent: success
-    return_artifacts: message
-  
-  error:
-    return_intent: error
-    return_artifacts: false
+  close_ticket:
+    return_intent: resolved
+    return_artifacts: [resolution]
 ```
 
-### 2. Create Step Handlers
+### 2. Write handlers and run
 
 ```typescript
-import { createEngine, loadFlow, MemoryStore } from 'yaml-flow';
+import { createStepMachine, loadStepFlow } from 'yaml-flow/step-machine';
+import { MemoryStore } from 'yaml-flow/stores/memory';
+
+const flow = await loadStepFlow('./support-ticket.yaml');
 
 const handlers = {
-  greet: async (input, ctx) => {
-    return {
-      result: 'success',
-      data: { message: `Hello, ${input.name}!` }
-    };
-  }
+  classify: async (input) => {
+    const category = detectCategory(input.message);
+    return { result: category, data: { category, priority: 'high' } };
+  },
+  handle_billing: async (input, ctx) => {
+    const answer = await ctx.components.ai.ask(`Billing issue: ${input.category}`);
+    return { result: 'resolved', data: { resolution: answer } };
+  },
+  handle_technical: async (input, ctx) => {
+    const answer = await ctx.components.ai.ask(`Tech issue: ${input.category}`);
+    return answer.confidence > 0.7
+      ? { result: 'resolved', data: { resolution: answer.text } }
+      : { result: 'escalate' };
+  },
+  handle_general: async (input) => {
+    return { result: 'resolved', data: { resolution: 'See FAQ.' } };
+  },
+  escalate_ticket: async (input, ctx) => {
+    const id = await ctx.components.ticketSystem.escalate(input.category);
+    return { result: 'done', data: { escalation_id: id } };
+  },
 };
+
+const machine = createStepMachine(flow, handlers, {
+  store: new MemoryStore(),
+  components: { ai: myAIClient, ticketSystem: myTicketAPI },
+});
+
+const result = await machine.run({ message: 'I was double-charged' });
+// result.intent  → 'resolved'
+// result.data    → { resolution: '...' }
+// result.stepHistory → ['classify', 'handle_billing']
 ```
 
-### 3. Run the Flow
+### Step Machine features at a glance
+
+| Feature | Config |
+|---|---|
+| Transitions | `transitions: { success: next_step, failure: error_step }` |
+| Retry | `retry: { max_attempts: 3, delay_ms: 1000, backoff_multiplier: 2 }` |
+| Circuit breaker | `circuit_breaker: { max_iterations: 5, on_open: fallback }` |
+| Pause / resume | `await machine.pause(runId)` / `await machine.resume(runId)` |
+| Cancellation | Pass `signal: AbortController.signal` in options |
+| Events | `machine.on('step:complete', fn)` — also `flow:start`, `flow:complete`, `transition`, etc. |
+| Data flow | `expects_data` filters what a handler receives; `produces_data` documents what it returns |
+
+### Using the pure reducer directly (no driver)
 
 ```typescript
-const flow = await loadFlow('./my-flow.yaml');
-const engine = createEngine(flow, handlers);
+import { createInitialState, applyStepResult, checkCircuitBreaker, computeStepInput } from 'yaml-flow/step-machine';
+
+let state = createInitialState(flow, 'run-1');
 
-const result = await engine.run({ name: 'World' });
-console.log(result.data.message); // "Hello, World!"
+// Your own loop
+while (true) {
+  const cb = checkCircuitBreaker(flow, state, state.currentStep);
+  if (cb.broken) { state = cb.newState; continue; }
+  state = cb.newState;
+
+  const input = computeStepInput(flow, state.currentStep, allData);
+  const stepResult = await handlers[state.currentStep](input, context);
+  const { newState, isTerminal } = applyStepResult(flow, state, state.currentStep, stepResult);
+  state = newState;
+  if (isTerminal) break;
+}
 ```
 
-## Flow Configuration
+---
 
-### Settings
+## Event Graph — Quick Start
+
+The event graph has no driver class. You call two pure functions in a loop:
+
+- **`next(graph, state)`** → tells you which tasks are eligible right now
+- **`apply(state, event, graph)`** → applies an event and returns the new state
+
+You decide how to actually execute the tasks (call an API, run a function, ask an LLM, etc.).
+
+### 1. Define a graph
 
 ```yaml
+# research-pipeline.yaml
 settings:
-  start_step: my_step      # Required: First step to execute
-  max_total_steps: 100     # Optional: Circuit breaker (default: 100)
-  timeout_ms: 60000        # Optional: Flow timeout in ms
+  completion: goal-reached
+  goal: [final-report]
+  conflict_strategy: parallel-all
+
+tasks:
+  fetch_sources:
+    provides: [raw-sources]
+
+  analyse_sentiment:
+    requires: [raw-sources]
+    provides: [sentiment-result]
+
+  analyse_entities:
+    requires: [raw-sources]
+    provides: [entity-result]
+
+  merge_analysis:
+    requires: [sentiment-result, entity-result]
+    provides: [merged-analysis]
+
+  generate_report:
+    requires: [merged-analysis]
+    provides: [final-report]
+```
+
+`fetch_sources` runs first (no requires). Once it completes, both `analyse_sentiment` and `analyse_entities` become eligible in parallel. `merge_analysis` waits for both. `generate_report` waits for the merge. Done when `final-report` appears.
+
+### 2. Write a driver loop
+
+```typescript
+import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+import { parse } from 'yaml';
+import { readFileSync } from 'fs';
+
+const graph = parse(readFileSync('./research-pipeline.yaml', 'utf8'));
+let state = createInitialExecutionState(graph, 'exec-1');
+
+while (true) {
+  const schedule = next(graph, state);
+
+  if (schedule.isComplete) {
+    console.log('Done!', state.availableOutputs);
+    break;
+  }
+  if (schedule.stuckDetection.is_stuck) {
+    console.error('Stuck:', schedule.stuckDetection.stuck_description);
+    break;
+  }
+
+  // Execute all eligible tasks (in parallel)
+  const results = await Promise.all(
+    schedule.eligibleTasks.map(async (taskName) => {
+      state = apply(state, { type: 'task-started', taskName, timestamp: new Date().toISOString() }, graph);
+
+      try {
+        const output = await executeTask(taskName, state);
+        return { taskName, success: true, result: output };
+      } catch (err) {
+        return { taskName, success: false, error: err.message };
+      }
+    })
+  );
+
+  // Feed results back into the reducer
+  for (const r of results) {
+    if (r.success) {
+      state = apply(state, { type: 'task-completed', taskName: r.taskName, result: r.result, timestamp: new Date().toISOString() }, graph);
+    } else {
+      state = apply(state, { type: 'task-failed', taskName: r.taskName, error: r.error, timestamp: new Date().toISOString() }, graph);
+    }
+  }
+}
+```
+
+That's the entire integration. ~30 lines. The engine is pure; your loop owns the I/O.
+
+### Event Graph features at a glance
+
+| Feature | Config | What it does |
+|---|---|---|
+| Dependencies | `requires: [a, b]` / `provides: [c]` | Task runs when all required tokens are available |
+| Conditional routing | `on: { positive: [pos-result], negative: [neg-result] }` | Different outputs based on task result |
+| Failure tokens | `on_failure: [data-unavailable]` | Inject tokens on failure so downstream alternatives can activate |
+| Retry | `retry: { max_attempts: 3 }` | Auto-retry on failure (task resets to not-started) |
+| Repeatable tasks | `repeatable: true` or `repeatable: { max: 5 }` | Task can re-execute when its inputs refresh |
+| Circuit breaker | `circuit_breaker: { max_executions: 10, on_break: [stop-token] }` | Inject tokens after N executions |
+| External events | `apply(state, { type: 'inject-tokens', tokens: ['user-approved'] })` | Unblock tasks waiting on external input |
+| Dynamic tasks | `apply(state, { type: 'task-creation', taskName: 'new', taskConfig: {...} })` | Add tasks at runtime |
+| Completion strategies | `completion: all-tasks-done \| all-outputs-done \| goal-reached \| manual` | When is the graph "done"? |
+| Conflict resolution | `conflict_strategy: alphabetical \| priority-first \| parallel-all \| ...` | What happens when two tasks produce the same output? |
+| Stuck detection | Automatic via `next()` | Returns `is_stuck: true` with description when no progress is possible |
+
+### Completion strategies explained
+
+| Strategy | Meaning |
+|---|---|
+| `all-tasks-done` | Every task has completed (or failed/inactivated) |
+| `all-outputs-done` | Every `provides` token from every task is available |
+| `goal-reached` | Specific tokens listed in `settings.goal` are available |
+| `only-resolved` | All non-failed tasks have completed |
+| `manual` | Never auto-completes; you decide when to stop |
+
+### Conflict resolution strategies
+
+When multiple eligible tasks produce the same output token, only one should run (unless you want parallel-all). The `conflict_strategy` setting controls the selection:
+
+| Strategy | Behaviour |
+|---|---|
+| `alphabetical` | Pick the alphabetically first task name (default, deterministic) |
+| `priority-first` | Pick the task with the highest `priority` value |
+| `duration-first` | Pick the task with the lowest `estimatedDuration` |
+| `cost-optimized` | Pick the task with the lowest `estimatedCost` |
+| `resource-aware` | Pick the task with the lowest total resource requirements |
+| `round-robin` | Rotate among competing tasks across scheduler calls |
+| `random-select` | Pick one at random |
+| `parallel-all` | Run all competing tasks (no conflict resolution) |
+| `user-choice` | Return all candidates; let the caller decide |
+| `skip-conflicts` | Skip all tasks involved in a conflict |
+
+---
+
+## Practical Patterns
+
+### Pattern: AI Agent Tool Orchestration (Event Graph)
+
+An agent needs to gather evidence from multiple sources, then synthesize.
+
+```yaml
+settings:
+  completion: goal-reached
+  goal: [final-answer]
+  conflict_strategy: parallel-all
+
+tasks:
+  search_web:
+    provides: [web-results]
+  search_database:
+    provides: [db-results]
+  search_documents:
+    provides: [doc-results]
+
+  synthesize:
+    requires: [web-results, db-results, doc-results]
+    provides: [draft-answer]
+
+  verify:
+    requires: [draft-answer]
+    provides: [final-answer]
+    on:
+      verified: [final-answer]
+      rejected: [needs-revision]
+    on_failure: [verification-skipped]
+
+  revise:
+    requires: [needs-revision]
+    provides: [draft-answer]
+    repeatable: { max: 3 }
 ```
 
-### Steps
+The three searches run in parallel. `synthesize` waits for all three. `verify` can produce different token sets depending on its result. If rejected, `revise` picks up and feeds back into `verify` (up to 3 times). If verify itself fails, `verification-skipped` unblocks any downstream task waiting on it.
+
+### Pattern: Order Processing Pipeline (Step Machine)
 
 ```yaml
+settings:
+  start_step: validate
+  max_total_steps: 15
+
 steps:
-  my_step:
-    description: "What this step does"
-    expects_data:          # Input data keys
-      - input_key
-    produces_data:         # Output data keys
-      - output_key
-    transitions:           # Result -> next step mapping
-      success: next_step
-      failure: error_step
-    retry:                 # Optional retry config
+  validate:
+    produces_data: [validated_order]
+    transitions:
+      valid: charge_payment
+      invalid: reject
+
+  charge_payment:
+    expects_data: [validated_order]
+    produces_data: [payment_id]
+    transitions:
+      success: ship
+      declined: reject
+    retry:
       max_attempts: 3
-      delay_ms: 1000
+      delay_ms: 2000
       backoff_multiplier: 2
-    circuit_breaker:       # Optional loop protection
-      max_iterations: 5
-      on_open: fallback_step
-```
 
-### Terminal States
+  ship:
+    expects_data: [validated_order, payment_id]
+    produces_data: [tracking_number]
+    transitions:
+      shipped: confirm
+      failure: refund
+
+  refund:
+    expects_data: [payment_id]
+    produces_data: [refund_id]
+    transitions:
+      done: reject
+
+  confirm:
+    expects_data: [tracking_number]
+    transitions:
+      done: complete
 
-```yaml
 terminal_states:
-  success:
-    return_intent: "success"
-    return_artifacts:
-      - result_data
-      - metadata
-  
-  error:
-    return_intent: "error"
+  complete:
+    return_intent: success
+    return_artifacts: [tracking_number, payment_id]
+  reject:
+    return_intent: rejected
     return_artifacts: false
 ```
 
-## Step Handlers
+Linear with branches. The current step decides what's next. Retry on payment failures with exponential backoff.
 
-Step handlers are pure async functions:
+### Pattern: Inject External Events (Event Graph)
 
 ```typescript
-type StepHandler = (input: StepInput, context: StepContext) => Promise<StepResult>;
+// A task is waiting for human approval
+const graph = {
+  settings: { completion: 'goal-reached', goal: ['deployed'] },
+  tasks: {
+    build:    { provides: ['build-artifact'] },
+    test:     { requires: ['build-artifact'], provides: ['test-passed'] },
+    approve:  { requires: ['test-passed', 'human-approval'], provides: ['approved'] },
+    deploy:   { requires: ['approved'], provides: ['deployed'] },
+  },
+};
 
-interface StepInput {
-  [key: string]: unknown;  // Data from expects_data
-}
+let state = createInitialExecutionState(graph, 'deploy-1');
 
-interface StepContext {
-  runId: string;           // Current run ID
-  stepName: string;        // Current step name
-  components: object;      // Injected dependencies
-  store: FlowStore;        // Direct store access
-  signal?: AbortSignal;    // Cancellation signal
-  emit: (event, data) => void;  // Event emitter
-}
+// ... run build and test normally ...
 
-interface StepResult {
-  result: string;          // Transition key (e.g., 'success', 'failure')
-  data?: object;           // Output data (matches produces_data)
-}
+// Later, when a human clicks "Approve" in your UI:
+state = apply(state, {
+  type: 'inject-tokens',
+  tokens: ['human-approval'],
+  timestamp: new Date().toISOString(),
+}, graph);
+
+// Now next(graph, state) will return 'approve' as eligible
 ```
 
-### Example Handler
+### Pattern: Conditional Branching in Event Graph
 
-```typescript
-const handlers = {
-  async processOrder(input, ctx) {
-    const { order } = input;
-    
-    // Access injected database client
-    const db = ctx.components.db;
-    
-    try {
-      const savedOrder = await db.orders.save(order);
-      
-      return {
-        result: 'success',
-        data: { 
-          order_id: savedOrder.id,
-          status: 'pending'
-        }
-      };
-    } catch (error) {
-      return {
-        result: 'failure',
-        data: { error: error.message }
-      };
-    }
-  }
-};
+```yaml
+tasks:
+  classify_image:
+    provides: [classification]
+    on:
+      photo: [is-photo]
+      document: [is-document]
+      screenshot: [is-screenshot]
+
+  enhance_photo:
+    requires: [is-photo]
+    provides: [processed-image]
+
+  ocr_document:
+    requires: [is-document]
+    provides: [extracted-text]
+
+  crop_screenshot:
+    requires: [is-screenshot]
+    provides: [processed-image]
 ```
 
+Only one downstream path activates based on the classifier result. This is the event-graph equivalent of a switch statement.
+
+---
+
 ## Storage Adapters
 
-### Memory (Default)
+All three stores work with both modes. Step Machine uses them for run state persistence. Event Graph state is plain JSON — serialize it yourself or use a store.
 
-```typescript
-import { MemoryStore } from 'yaml-flow';
+### Memory (default, all environments)
 
-const engine = createEngine(flow, handlers, {
-  store: new MemoryStore()
-});
+```typescript
+import { MemoryStore } from 'yaml-flow/stores/memory';
 ```
 
-### LocalStorage (Browser)
+### LocalStorage (browser)
 
 ```typescript
 import { LocalStorageStore } from 'yaml-flow/stores/localStorage';
-
-const engine = createEngine(flow, handlers, {
-  store: new LocalStorageStore({ prefix: 'myapp' })
-});
+new LocalStorageStore({ prefix: 'myapp' });
 ```
 
 ### File System (Node.js)
 
 ```typescript
 import { FileStore } from 'yaml-flow/stores/file';
-
-const engine = createEngine(flow, handlers, {
-  store: new FileStore({ directory: './flow-data' })
-});
+new FileStore({ directory: './flow-data' });
 ```
 
 ### Custom Store
 
-Implement the `FlowStore` interface:
+Implement the `StepMachineStore` interface:
 
 ```typescript
-interface FlowStore {
-  saveRunState(runId: string, state: RunState): Promise<void>;
-  loadRunState(runId: string): Promise<RunState | null>;
+interface StepMachineStore {
+  saveRunState(runId: string, state: StepMachineState): Promise<void>;
+  loadRunState(runId: string): Promise<StepMachineState | null>;
   deleteRunState(runId: string): Promise<void>;
   setData(runId: string, key: string, value: unknown): Promise<void>;
   getData(runId: string, key: string): Promise<unknown>;
@@ -228,152 +507,353 @@ interface FlowStore {
 }
 ```
 
-## Component Injection
+---
 
-Inject external dependencies (databases, API clients, etc.):
+## Batch Processing
+
+yaml-flow includes a `batch()` utility for running multiple items through a flow concurrently. It works with both Step Machine and Event Graph — you provide the processor, it manages concurrency.
+
+### Quick Start
 
 ```typescript
-const engine = createEngine(flow, handlers, {
-  components: {
-    db: databaseClient,
-    api: httpClient,
-    cache: redisClient,
-    ai: openAIClient
-  }
+import { batch } from 'yaml-flow/batch';
+import { createStepMachine } from 'yaml-flow/step-machine';
+
+const tickets = [
+  { id: 'T-001', message: 'Billing error' },
+  { id: 'T-002', message: 'App crashes on login' },
+  { id: 'T-003', message: 'Password reset help' },
+];
+
+const result = await batch(tickets, {
+  concurrency: 3,
+  processor: async (ticket) => {
+    const machine = createStepMachine(flow, handlers);
+    return machine.run({ message: ticket.message });
+  },
+  onProgress: (p) => console.log(`${p.percent}% done`),
 });
 
-// Access in handlers
-const handlers = {
-  async fetchData(input, ctx) {
-    const result = await ctx.components.api.get('/data');
-    return { result: 'success', data: { fetched: result } };
-  }
-};
+console.log(`${result.completed} succeeded, ${result.failed} failed`);
 ```
 
-## Events
+### Options
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `concurrency` | `number` | `5` | Max parallel processors |
+| `processor` | `(item, index) => Promise<TResult>` | *required* | Async function to process each item |
+| `signal` | `AbortSignal` | — | Cancel remaining items |
+| `onItemComplete` | `(item, result, index) => void` | — | Called when an item succeeds |
+| `onItemError` | `(item, error, index) => void` | — | Called when an item fails |
+| `onProgress` | `(progress) => void` | — | Called after each item with `{ completed, failed, active, pending, total, percent, elapsedMs }` |
 
-Subscribe to flow events:
+### Result Shape
 
 ```typescript
-const engine = createEngine(flow, handlers);
+{
+  items: BatchItemResult[];  // Per-item: { item, index, status, result?, error?, durationMs }
+  completed: number;         // Items that succeeded
+  failed: number;            // Items that threw
+  total: number;
+  durationMs: number;        // Wall-clock time for entire batch
+}
+```
 
-// Subscribe to events
-const unsubscribe = engine.on('step:complete', (event) => {
-  console.log(`Step ${event.data.step} completed with ${event.data.result}`);
-});
+### Works with Event Graph too
 
-// Available events:
-// - flow:start, flow:complete, flow:error, flow:paused, flow:resumed
-// - step:start, step:complete, step:error
-// - transition
+```typescript
+import { batch } from 'yaml-flow/batch';
+import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+
+const result = await batch(items, {
+  concurrency: 5,
+  processor: async (item) => {
+    let state = createInitialExecutionState(graph, `run-${item.id}`);
+    state = apply(state, { type: 'inject-tokens', tokens: [item.readyToken], timestamp: Date.now() }, graph);
+    // ... drive graph loop with next() + apply()
+    return state;
+  },
+});
 ```
 
-## Pause & Resume
+---
 
-```typescript
-// Start flow
-const result = engine.run({ data: 'value' });
+## Config Utilities
 
-// Pause (from another context)
-await engine.pause(runId);
+Pure pre-processing transforms you apply before passing config to the engine. They never touch engine state — just config in, config out.
 
-// Later: resume
-const resumed = await engine.resume(runId);
-```
+### Variable Interpolation
 
-## Cancellation
+Replace `${KEY}` patterns in any config object. Works with both GraphConfig and StepFlowConfig.
 
 ```typescript
-const controller = new AbortController();
+import { resolveVariables } from 'yaml-flow/config';
 
-const engine = createEngine(flow, handlers, {
-  signal: controller.signal
+const resolved = resolveVariables(graphConfig, {
+  ENTITY_ID: 'ticket-42',
+  TOOLS_DIR: '/opt/tools',
+  WORKDIR: '/data/workdata',
 });
+// Every ${ENTITY_ID} in task configs, cmd-args, etc. → replaced
+```
 
-// Start flow
-const resultPromise = engine.run();
+### Config Templates
 
-// Cancel
-controller.abort();
+DRY reusable config blocks. Tasks reference a named template via `config-template`; the function deep-merges template + task overrides and removes the reference.
+
+```typescript
+import { resolveConfigTemplates } from 'yaml-flow/config';
+
+const config = {
+  configTemplates: {                              // or 'config-templates' (kebab-case)
+    PYTHON_TOOL: { cmd: 'python', timeout: 30000, cwd: '/workdata' },
+    NODE_CMD:    { cmd: 'node',   timeout: 60000 },
+  },
+  tasks: {
+    analyze:  { provides: ['analysis'], config: { 'config-template': 'PYTHON_TOOL', 'cmd-args': 'analyze.py' } },
+    build:    { provides: ['build'],    config: { 'config-template': 'NODE_CMD',    script: 'build.js' } },
+  },
+};
+
+const resolved = resolveConfigTemplates(config);
+// analyze.config → { cmd: 'python', timeout: 30000, cwd: '/workdata', 'cmd-args': 'analyze.py' }
+// configTemplates key removed from output
 ```
 
-## Browser Usage
+### Composing Both
 
-### With Bundler (Vite, webpack, etc.)
+Templates first (expands references), then variables (fills in `${...}` placeholders):
 
 ```typescript
-import { createEngine, MemoryStore } from 'yaml-flow';
+import { resolveConfigTemplates, resolveVariables } from 'yaml-flow/config';
 
-// Flow as JSON (pre-parsed at build time)
-const flow = {
-  settings: { start_step: 'start' },
-  steps: { start: { transitions: { success: 'done' } } },
-  terminal_states: { done: { return_intent: 'success' } }
-};
+const raw = loadYaml('pipeline.yaml');                  // has configTemplates + ${VAR} refs
+const resolved = resolveVariables(
+  resolveConfigTemplates(raw),
+  { ENTITY_ID: 'url-42', TOOLS_DIR: '/opt/tools' },
+);
+```
+
+---
+
+## Graph-of-Graphs Pattern
+
+Real-world pipelines are often **layered**: an outer orchestration graph where some tasks are themselves entire sub-workflows — each processing a batch of items through their own DAG or step flow. yaml-flow doesn't bake this into the engine (the pure scheduler stays simple), but the primitives compose cleanly.
+
+### The Shape
 
-const engine = createEngine(flow, handlers);
 ```
+Outer graph (event-graph)
+├── prep-workdata          → plain task
+├── copy-input-files       → plain task
+├── evidence-gathering     → batch × inner event-graph (N items, 5 concurrent)
+├── grade-synthesis        → batch × inner step-machine (N items, 3 concurrent)
+├── analyze-mismatches     → plain task
+├── [health-check ∥ report]→ parallel plain tasks
+└── archive-results        → waits for both
+```
+
+The outer graph sequences coarse stages. Some stages fan out into batches where each item runs through its own sub-workflow. The sub can be either mode.
+
+### How to Wire It
 
-### From URL
+Each "sub-graph task" in the outer graph is just a handler that composes `resolveConfigTemplates` → `resolveVariables` → `batch` → engine:
 
 ```typescript
-import { loadFlowFromUrl, createEngine } from 'yaml-flow';
+import { next, apply, createInitialExecutionState } from 'yaml-flow/event-graph';
+import { createStepMachine } from 'yaml-flow/step-machine';
+import { batch } from 'yaml-flow/batch';
+import { resolveVariables, resolveConfigTemplates } from 'yaml-flow/config';
+
+// Outer graph handler for a sub-graph task (event-graph sub)
+async function runEvidenceBatch(items, rawSubConfig) {
+  return batch(items, {
+    concurrency: 5,
+    processor: async (item) => {
+      // Resolve config per-item (each item gets its own ENTITY_ID)
+      const config = resolveVariables(
+        resolveConfigTemplates(rawSubConfig),
+        { ENTITY_ID: item.id, TOOLS_DIR: '/opt/tools' },
+      );
+      // Drive the inner event-graph
+      let state = createInitialExecutionState(config, `run-${item.id}`);
+      while (true) {
+        const { eligibleTasks, isComplete } = next(config, state);
+        if (isComplete) break;
+        for (const task of eligibleTasks) {
+          state = apply(state, { type: 'task-started', taskName: task, timestamp: new Date().toISOString() }, config);
+          const result = await executeTask(task, config.tasks[task], item);
+          state = apply(state, { type: 'task-completed', taskName: task, result, timestamp: new Date().toISOString() }, config);
+        }
+      }
+      return state;
+    },
+  });
+}
 
-const flow = await loadFlowFromUrl('/flows/my-flow.json');
-const engine = createEngine(flow, handlers);
+// Outer graph handler for a sub-graph task (step-machine sub)
+async function runGradeBatch(items, flowConfig, handlers) {
+  return batch(items, {
+    concurrency: 3,
+    processor: async (item) => {
+      const machine = createStepMachine(flowConfig, handlers);
+      return machine.run({ entityId: item.id, evidence: item.evidence });
+    },
+  });
+}
 ```
 
-## JSON Schema
+### Driving the Outer Graph
 
-Use the included JSON Schema for IDE autocomplete:
+The outer graph itself is an event-graph. Each handler maps to a task:
 
-```yaml
-# yaml-language-server: $schema=node_modules/yaml-flow/schema/flow.schema.json
+```typescript
+const outerHandlers = {
+  'prep-workdata':     async () => { /* setup */ },
+  'copy-input-files':  async () => { /* parse CSV, return items */ },
+  'evidence-batch':    async (ctx) => runEvidenceBatch(ctx.items, evidenceConfig),
+  'grade-batch':       async (ctx) => runGradeBatch(ctx.items, gradeFlow, gradeHandlers),
+  'analyze-mismatches':async (ctx) => { /* compare grades */ },
+  'health-check':      async (ctx) => { /* validate */ },
+  'generate-report':   async (ctx) => { /* summarize */ },
+  'archive':           async (ctx) => { /* move outputs */ },
+};
 
-settings:
-  start_step: my_step
-  # ... IDE autocomplete works here
+// Simple outer loop
+let state = createInitialExecutionState(outerGraph, 'pipeline-run-1');
+while (true) {
+  const { eligibleTasks, isComplete } = next(outerGraph, state);
+  if (isComplete) break;
+  await Promise.all(eligibleTasks.map(async (taskName) => {
+    state = apply(state, { type: 'task-started', taskName, timestamp: now() }, outerGraph);
+    try {
+      await outerHandlers[taskName](context);
+      state = apply(state, { type: 'task-completed', taskName, timestamp: now() }, outerGraph);
+    } catch (err) {
+      state = apply(state, { type: 'task-failed', taskName, error: err.message, timestamp: now() }, outerGraph);
+    }
+  }));
+}
 ```
 
-## API Reference
+### Why Not Bake It Into the Engine?
 
-### `createEngine(flow, handlers, options?)`
+- The pure scheduler (`next`/`apply`) stays a simple `f(state, event) → newState`.
+- Sub-graph execution involves file I/O, process spawning, HTTP calls — all driver concerns.
+- Every deployment customizes how sub-tasks execute: in-process, `execSync`, HTTP, serverless.
+- The primitives (`batch` + `resolveVariables` + `resolveConfigTemplates` + both engines) compose without coupling.
 
-Create a new flow engine instance.
+See the [examples/graph-of-graphs/](./examples/graph-of-graphs/) directory for complete runnable examples.
 
-### `loadFlow(source)`
+---
 
-Load and validate a flow from file path, URL, or object.
+## Package Exports
 
-### `FlowEngine.run(initialData?)`
+```typescript
+// Everything (both modes + stores + batch)
+import { StepMachine, next, apply, MemoryStore, batch } from 'yaml-flow';
 
-Execute the flow from start.
+// Step Machine only
+import { StepMachine, createStepMachine, loadStepFlow } from 'yaml-flow/step-machine';
+import { applyStepResult, checkCircuitBreaker, createInitialState } from 'yaml-flow/step-machine';
 
-### `FlowEngine.resume(runId)`
+// Event Graph only
+import { next, apply, applyAll, getCandidateTasks } from 'yaml-flow/event-graph';
+import { createInitialExecutionState, isExecutionComplete, detectStuckState } from 'yaml-flow/event-graph';
+import { TASK_STATUS, COMPLETION_STRATEGIES, CONFLICT_STRATEGIES } from 'yaml-flow/event-graph';
 
-Resume a paused flow.
+// Stores
+import { MemoryStore, LocalStorageStore, FileStore } from 'yaml-flow/stores';
 
-### `FlowEngine.pause(runId)`
+// Batch
+import { batch } from 'yaml-flow/batch';
+import type { BatchOptions, BatchResult, BatchItemResult, BatchProgress } from 'yaml-flow/batch';
 
-Pause a running flow.
+// Config utilities
+import { resolveVariables, resolveConfigTemplates } from 'yaml-flow/config';
 
-### `FlowEngine.on(event, listener)`
+// Backward compatibility (v1 names → v2)
+import { FlowEngine, createEngine } from 'yaml-flow';  // aliases for StepMachine, createStepMachine
+```
 
-Subscribe to flow events.
+---
 
-### `FlowEngine.getStore()`
+## API Reference
 
-Get the store instance.
+### Step Machine
+
+| Export | Description |
+|---|---|
+| `createStepMachine(flow, handlers, options?)` | Create and validate a StepMachine instance |
+| `StepMachine.run(initialData?)` | Execute flow from start, returns `StepMachineResult` |
+| `StepMachine.pause(runId)` | Pause a running flow |
+| `StepMachine.resume(runId)` | Resume a paused flow |
+| `StepMachine.on(event, listener)` | Subscribe to events (`step:start`, `step:complete`, `flow:complete`, `transition`, etc.) |
+| `loadStepFlow(path)` | Load + validate a YAML/JSON flow file |
+| `applyStepResult(flow, state, step, result)` | Pure reducer: apply a step result to state |
+| `checkCircuitBreaker(flow, state, step)` | Pure: check/increment circuit breaker |
+| `computeStepInput(flow, step, allData)` | Pure: filter data to what a step expects |
+| `createInitialState(flow, runId)` | Pure: create starting state |
+
+### Event Graph
+
+| Export | Description |
+|---|---|
+| `next(graph, state)` | Pure scheduler: returns `{ eligibleTasks, isComplete, stuckDetection, conflicts }` |
+| `apply(state, event, graph)` | Pure reducer: apply one event, returns new state |
+| `applyAll(state, events, graph)` | Apply multiple events sequentially |
+| `createInitialExecutionState(graph, executionId)` | Create starting state for a graph |
+| `getCandidateTasks(graph, state)` | Low-level: just the eligible task list |
+| `isExecutionComplete(graph, state)` | Check completion against configured strategy |
+| `detectStuckState({graph, state, ...})` | Check if execution is stuck |
+| `addDynamicTask(graph, name, config)` | Immutably add a task to a graph config |
+
+### Event Types (for `apply()`)
+
+| Event | Fields | Effect |
+|---|---|---|
+| `task-started` | `taskName` | Sets task status to `running` |
+| `task-completed` | `taskName`, `result?` | Marks completed, adds `provides` tokens (or `on[result]` tokens) |
+| `task-failed` | `taskName`, `error` | Retries or marks failed, injects `on_failure` tokens |
+| `task-progress` | `taskName`, `message?`, `progress?` | Updates progress/messages |
+| `inject-tokens` | `tokens[]` | Adds tokens to available outputs (unblocks waiting tasks) |
+| `agent-action` | `action: start\|stop\|pause\|resume` | Controls execution lifecycle |
+| `task-creation` | `taskName`, `taskConfig` | Adds a new task to execution state |
+
+---
 
 ## Examples
 
-See the [examples](./examples) directory:
+See the [examples/](./examples) directory:
+
+| Example | Mode | Demonstrates |
+|---|---|---|
+| [Simple Greeting](./examples/node/simple-greeting.ts) | Step Machine | Basic flow with file store |
+| [AI Conversation](./examples/node/ai-conversation.ts) | Step Machine | Retry, circuit breakers, component injection |
+| [Research Pipeline](./examples/event-graph/research-pipeline.ts) | Event Graph | Parallel tasks, goal-based completion |
+| [CI/CD Pipeline](./examples/event-graph/ci-cd-pipeline.ts) | Event Graph | External events, conditional routing, failure tokens |
+| [Batch Tickets](./examples/batch/batch-step-machine.ts) | Batch | Concurrent processing, progress tracking |
+| [URL Pipeline](./examples/graph-of-graphs/url-processing-pipeline.ts) | Graph-of-Graphs | Outer event-graph → batch × inner event-graph per item |
+| [Multi-Stage ETL](./examples/graph-of-graphs/multi-stage-etl.ts) | Graph-of-Graphs | Mixed modes: event-graph outer → step-machine + event-graph subs |
+| [Order Processing](./examples/flows/order-processing.yaml) | Step Machine | YAML flow definition |
+| [Browser Demo](./examples/browser/index.html) | Step Machine | In-browser usage |
+
+---
+
+## Migrating from v1
+
+v2 is backward compatible. The old names still work:
+
+```typescript
+// v1 (still works)
+import { FlowEngine, createEngine } from 'yaml-flow';
+
+// v2 (preferred)
+import { StepMachine, createStepMachine } from 'yaml-flow/step-machine';
+```
 
-- [Simple Greeting (Node.js)](./examples/node/simple-greeting.ts)
-- [AI Conversation (Node.js)](./examples/node/ai-conversation.ts)
-- [Browser Demo](./examples/browser/index.html)
-- [Order Processing (Flow)](./examples/flows/order-processing.yaml)
+The `FlowStore` interface is now `StepMachineStore` (same shape). `RunState` is now `StepMachineState` (same shape). Both old names resolve to the new types.
 
 ## License