@asaidimu/utils-pipeline 1.0.2 → 1.0.4

package/README.md

@@ -1,328 +1,169 @@
-## Pipeline Utility
+# @asaidimu/utils-pipeline
 
-A staged, concurrent pipeline orchestration framework for complex async workflows with parallel execution, cancellation, and single-flight deduplication.
+A production-grade, asynchronous, type-safe pipeline engine featuring conditional routing, checkpoint-based pause/resume mechanisms, concurrent execution, and atomic state transactions.
 
-### Why Use This?
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Version](https://img.shields.io/badge/version-1.0.0-green.svg)](package.json)
 
-Traditional async orchestration often leads to:
+## Features
 
-- Nested callbacks and manual Promise.all coordination
-- Race conditions in shared state
-- Difficult cancellation across multiple operations
-- Duplicate work from concurrent identical requests
+- **Type-safe State Management**: Unified store with atomic updates across steps and stages.
+- **Conditional Routing**: Dynamically jump between stages, terminate early, or suspend execution.
+- **Pause & Resume**: Suspend a pipeline at any stage, persist its state and artifacts, and resume later—even after a process restart.
+- **Concurrent Execution**: Parallelize steps within a stage or run multiple sub-pipelines simultaneously.
+- **Atomic Transactions**: State patches are committed atomically at stage boundaries, ensuring consistency.
+- **Deep Observability**: Lifecycle events for every pipeline, stage, and step with full ancestry paths.
+- **Abort Support**: Built-in support for `AbortSignal` to cancel long-running operations.
 
-This pipeline solves these by providing:
+## Installation
 
-- **Declarative stages** with automatic parallelization
-- **Isolated execution contexts** per run
-- **Built-in deduplication** for identical concurrent executions
-- **Scoped cancellation** via AbortController
-- **Comprehensive lifecycle events** for monitoring
-
-### Core Concepts
-
-#### StageCarry
-
-Data flows through the pipeline as a record mapping step keys to their outputs:
-
-```typescript
-type StageCarry = Record<string, any>;
-// Example: { fetch: { userId: 123 }, enrich: { ... }, save: { saved: true } }
-```
-
-#### PipelineStep
-
-A single unit of work with three properties:
-
-- `key` - Unique identifier (used as the output key in StageCarry)
-- `order` - Stage number (steps with same order run in parallel)
-- `action` - Async function receiving current carry and returning `Result<T>`
-
-```typescript
-interface PipelineStep<TIn extends StageCarry = StageCarry, TOut = any> {
-  key: string;
-  order: number;
-  action: (input: TIn, signal?: AbortSignal) => Promise<Result<TOut>>;
-}
+```bash
+bun add @asaidimu/utils-pipeline
+# or
+npm install @asaidimu/utils-pipeline
 ```
 
-#### Result Type
-
-All pipeline actions must return a `Result` to distinguish success from failure:
-
-```typescript
-type Result<T, E = SystemError> =
-  | { ok: true; value: T }
-  | { ok: false; error: E };
-```
+## Routing Sequential Pipeline (RSP)
 
-### Usage Examples
+The `Routing Sequential Pipeline` is the primary engine for complex, stateful workflows.
 
-#### Basic Sequential Pipeline
+### Basic Usage
 
 ```typescript
-const pipeline = new Pipeline([
-  { key: "parse", action: async ({ parse }) => Result.ok(JSON.parse(parse)) },
-  { key: "validate", action: async ({ parse }) => Result.ok(validate(parse)) },
-  {
-    key: "store",
-    action: async ({ validate }) => Result.ok(db.insert(validate)),
-  },
-]);
-
-const result = await pipeline.execute("parse", '{"id":1}');
-// result.value = { store: insertResult }
-```
+import {
+  PipelineFactory,
+  type RoutingPipelineDefinition,
+} from "@asaidimu/utils-pipeline";
 
-#### Parallel Processing
+interface MyState {
+  counter: number;
+}
 
-```typescript
-const pipeline = new Pipeline([
-  { key: "fetch", action: async () => Result.ok(await api.fetch()) },
-  [
+const definition: RoutingPipelineDefinition<MyState> = {
+  id: "my-pipeline",
+  label: "My Business Process",
+  stages: [
     {
-      key: "cache",
-      action: async ({ fetch }) => Result.ok(await redis.set(fetch)),
+      id: "init",
+      order: 1,
+      label: "Initialization",
+      steps: {
+        setup: {
+          id: "setup",
+          label: "Setup Data",
+          action: async (ctx) => ({ counter: 1 }),
+        },
+      },
     },
     {
-      key: "notify",
-      action: async ({ fetch }) => Result.ok(await webhook.send(fetch)),
+      id: "process",
+      order: 2,
+      label: "Main Processing",
+      steps: {
+        work: {
+          id: "work",
+          label: "Do Work",
+          action: async (ctx) => {
+            const counter = await ctx.use((c) => c.select((s) => s.counter));
+            return { counter: state.counter + 1 };
+          },
+        },
+      },
+      // Conditional routing after this stage completes
+      router: (state) => (state.counter > 5 ? "end" : "process"),
     },
     {
-      key: "log",
-      action: async ({ fetch }) => Result.ok(await logger.info(fetch)),
+      id: "end",
+      order: 3,
+      label: "Finalize",
+      steps: {
+        cleanup: {
+          id: "cleanup",
+          label: "Cleanup",
+          action: async () => ({}),
+        },
+      },
     },
   ],
-]);
-// cache, notify, and log execute simultaneously after fetch completes
-```
-
-#### Starting Mid-Pipeline
-
-```typescript
-// Skip initial stages, start from "process" step
-const result = await pipeline.execute("process", { data: "pre-processed" });
-// Only executes stages with order >= process.order
-```
-
-#### Deduplication (Single-Flight)
-
-```typescript
-// Three identical calls execute concurrently but only run the action once
-const [a, b, c] = await Promise.all([
-  pipeline.execute("expensive", "user-123"),
-  pipeline.execute("expensive", "user-123"),
-  pipeline.execute("expensive", "user-123"),
-]);
-// a, b, c receive the same result; action called exactly once
-```
-
-#### Cancellation & Abort Signals
-
-```typescript
-const controller = new AbortController();
-
-// Start execution with abort capability
-const executionPromise = pipeline.execute("long-running", params, {
-  signal: controller.signal,
-});
-
-// Cancel after 1 second
-setTimeout(() => controller.abort(), 1000);
-
-const result = await executionPromise;
-// result.ok === false with error.code === "CANCELLED"
-```
-
-#### Event Monitoring
-
-```typescript
-// Global events for monitoring
-pipeline.on("step:success", ({ stepKey, executionId, result }) => {
-  console.log(`[${executionId}] ${stepKey} completed`, result);
-});
-
-pipeline.on("failure", ({ stepKey, error }) => {
-  console.error(`Pipeline failed at ${stepKey}:`, error.message);
-});
-
-// Scoped events for a specific run
-const ctx = pipeline.prepare("step", params);
-ctx.on("step:success", (payload) => {
-  // Only triggered for this execution
-});
-await ctx.run();
-```
-
-#### Manual Context Management
+};
 
-```typescript
-// For fine-grained control over concurrent runs
-const context = pipeline.prepare(
-  "process",
-  { data: "input" },
-  { signal: externalAbortSignal }, // optional
-);
-
-// Add scoped listeners
-context.on("stage:success", ({ order, carry }) => {
-  console.log(`Stage ${order} completed`);
+const factory = new PipelineFactory(definition, {
+  storeFactory: async (runId) => createStore(runId), // Return a DataStore instance
+  initialStateFactory: () => ({ counter: 0 }),
 });
 
-// Execute with full control
+// Prepare and run
+const context = await factory.prepare();
 const result = await context.run();
 
-// Cancel individual run
-context.abort();
+if (result.ok && result.value.status === "succeeded") {
+  console.log("Pipeline finished:", result.value.finalState);
+}
 ```
 
-### Dynamic Step Registration
-
-```typescript
-const pipeline = new Pipeline();
-
-// Add steps at runtime
-pipeline
-  .step({ key: "a", order: 0, action: ok("first") })
-  .step({ key: "b", order: 1, action: ok("second") })
-  .step({ key: "c", order: 1, action: ok("parallel with b") });
-
-await pipeline.execute("a", null);
-```
+### Pause and Resume
 
-### Error Handling Patterns
+A router can signal a `pause`, which suspends the pipeline and writes a checkpoint.
 
 ```typescript
-// Returning a failed Result stops the pipeline
-const step = {
-  key: "validate",
-  action: async (carry) => {
-    if (!carry.data.valid) {
-      return Result.fail(
-        new SystemError({
-          code: "VALIDATION_ERROR",
-          message: "Invalid data",
-          operation: "validate",
-        }),
-      );
-    }
-    return Result.ok(carry.data);
-  },
+// Inside a stage definition:
+router: (state) => {
+  if (state.needsApproval) {
+    return { pause: "processing-stage", timeout: 86400000 }; // 24h timeout
+  }
+  return undefined; // natural advance
 };
 
-// Throwing also works (automatically wrapped in SystemError)
-const another = {
-  key: "risky",
-  action: async () => {
-    throw new Error("Something went wrong");
-    // Becomes SystemError with code INTERNAL_ERROR
-  },
-};
+// Later, to resume:
+const result = await factory.resume(runId);
+if (result.ok) {
+  const context = result.value;
+  await context.run();
+}
 ```
 
-### Lifecycle Events
-
-| Event           | Payload                            | When                                 |
-| --------------- | ---------------------------------- | ------------------------------------ |
-| `start`         | `{ stepKey, executionId }`         | Before any steps execute             |
-| `step:success`  | `{ stepKey, executionId, result }` | Each successful step                 |
-| `step:failure`  | `{ stepKey, executionId, error }`  | Each failed step (even in parallel)  |
-| `stage:success` | `{ order, executionId, carry }`    | After all steps in a stage complete  |
-| `terminate`     | `{ carry, executionId }`           | When pipeline completes successfully |
-| `failure`       | `{ stepKey, executionId, error }`  | On first pipeline failure            |
-
-### API Reference
+## Sequential Pipeline (Simple)
 
-#### `Pipeline` Class
-
-| Method                                          | Description                                               |
-| ----------------------------------------------- | --------------------------------------------------------- |
-| `constructor(definition?)`                      | Create pipeline with optional declarative definition      |
-| `step<TIn, TOut>(definition)`                   | Register or overwrite a step, returns `this` for chaining |
-| `execute<TIn>(stepKey, params, options?)`       | Execute pipeline starting from `stepKey`                  |
-| `prepare<TIn>(stepKey, params, options?, bus?)` | Create isolated execution context without running         |
-| `on<K>(event, handler)`                         | Subscribe to global pipeline events                       |
-| `destroy()`                                     | Clear all steps, listeners, and deduplication cache       |
-
-#### `PipelineExecutionContext` Class
-
-| Method                  | Description                                          |
-| ----------------------- | ---------------------------------------------------- |
-| `run()`                 | Execute the pipeline from this context's entry point |
-| `abort()`               | Cancel this specific execution                       |
-| `on<K>(event, handler)` | Subscribe to events scoped to this execution         |
-
-#### `Result` Utilities
+For simpler, linear workflows without conditional routing or persistence, use the standard `Pipeline` class.
 
 ```typescript
-// Success
-Result.ok(value); // -> { ok: true, value }
-
-// Failure
-Result.fail(error); // -> { ok: false, error }
+import { Pipeline, Result } from "@asaidimu/utils-pipeline";
 
-// From the @core/error package
-Errors.internalError(original, message); // Creates SystemError
-Errors.invalidCommand(message); // Creates INVALID_COMMAND error
-```
-
-### Pipeline Definition Format
-
-The definition uses array position to determine stage order:
+const pipeline = new Pipeline([
+  { key: "step1", order: 0, action: async () => Result.ok("First") },
+  { key: "step2", order: 1, action: async () => Result.ok("Second") },
+]);
 
-```typescript
-type PipelineDefinition = Array<
-  PipelineStepDefinition | PipelineStepDefinition[]
->;
-
-// Index 0 = stage order 0
-// Index 1 = stage order 1, etc.
-
-// Flat = single step per stage
-[{ key: "a" }, { key: "b" }, { key: "c" }][
-  // Nested = parallel steps at that stage
-  ({ key: "a" }, [{ key: "b" }, { key: "c" }], { key: "d" })
-];
-// Stage 0: a
-// Stage 1: b, c (parallel)
-// Stage 2: d
+const result = await pipeline.execute("step1", null);
 ```
 
-### Best Practices
-
-1. **Keep steps focused** - Each step should do one thing and do it well
-2. **Use descriptive keys** - Keys become property names in StageCarry
-3. **Handle errors gracefully** - Return `Result.fail()` with meaningful error codes
-4. **Leverage deduplication** - Identical concurrent executions are automatically deduplicated
-5. **Monitor with events** - Use lifecycle events for logging, metrics, and debugging
-6. **Abort signals for timeouts** - Pass AbortSignal to enable cancellation
-7. **Avoid side effects between stages** - Only pass data through StageCarry
+## API Reference
 
-### Testing
+### `PipelineFactory`
 
-The pipeline is designed for testability:
+- `prepare(entry?, runId?)`: Creates a new `RunContext`.
+- `resume(runId)`: Reconstructs a `RunContext` from a persisted checkpoint.
 
-```typescript
-import { Pipeline } from "@core/pipeline";
+### `RunContext`
 
-// Mock steps with controlled delays
-const mockAction = vi.fn().mockResolvedValue(Result.ok("mocked"));
+- `run()`: Starts or continues execution.
+- `abort()`: Signals the pipeline to terminate at the next stage boundary.
+- `on(event, handler)`: Subscribes to lifecycle events.
 
-const pipeline = new Pipeline([{ key: "test", action: mockAction }]);
+### Lifecycle Events
 
-await pipeline.execute("test", null);
-expect(mockAction).toHaveBeenCalled();
-```
+- `pipeline:start` / `pipeline:success` / `pipeline:failure` / `pipeline:paused`
+- `stage:start` / `stage:success` / `stage:failure` / `stage:paused`
+- `step:start` / `step:success` / `step:failure`
+- `router:evaluated`
 
-### See Also
+## Best Practices
 
-- [`@core/error`](./error) - SystemError and Result types
-- [`@core/events`](./events) - EventBus implementation
+1. **Atomic Steps**: Each step should perform a single, focused operation.
+2. **Pure Store Interaction**: Steps should ideally only interact with the world via the provided store and context.
+3. **Handle Cancellation**: Long-running steps should check `pcxt.signal` or pass it to underlying async calls.
+4. **Descriptive Labels**: Use human-readable labels for stages and steps as they appear in event paths.
 
-This utility is particularly valuable for:
+## License
 
-- **Data processing pipelines** (ETL, image processing, document workflows)
-- **API orchestration** (parallel requests with aggregation)
-- **Build systems** (task graphs with dependencies)
-- **Form validation** (parallel validation rules)
-- **Background job processing** (staged workflows with cancellation)
+MIT