@asaidimu/utils-pipeline 1.0.0

package/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Saidimu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,328 @@
+## Pipeline Utility
+
+A staged, concurrent pipeline orchestration framework for complex async workflows with parallel execution, cancellation, and single-flight deduplication.
+
+### Why Use This?
+
+Traditional async orchestration often leads to:
+
+- Nested callbacks and manual Promise.all coordination
+- Race conditions in shared state
+- Difficult cancellation across multiple operations
+- Duplicate work from concurrent identical requests
+
+This pipeline solves these by providing:
+
+- **Declarative stages** with automatic parallelization
+- **Isolated execution contexts** per run
+- **Built-in deduplication** for identical concurrent executions
+- **Scoped cancellation** via AbortController
+- **Comprehensive lifecycle events** for monitoring
+
+### Core Concepts
+
+#### StageCarry
+
+Data flows through the pipeline as a record mapping step keys to their outputs:
+
+```typescript
+type StageCarry = Record<string, any>;
+// Example: { fetch: { userId: 123 }, enrich: { ... }, save: { saved: true } }
+```
+
+#### PipelineStep
+
+A single unit of work with three properties:
+
+- `key` - Unique identifier (used as the output key in StageCarry)
+- `order` - Stage number (steps with same order run in parallel)
+- `action` - Async function receiving current carry and returning `Result<T>`
+
+```typescript
+interface PipelineStep<TIn extends StageCarry = StageCarry, TOut = any> {
+  key: string;
+  order: number;
+  action: (input: TIn, signal?: AbortSignal) => Promise<Result<TOut>>;
+}
+```
+
+#### Result Type
+
+All pipeline actions must return a `Result` to distinguish success from failure:
+
+```typescript
+type Result<T, E = SystemError> =
+  | { ok: true; value: T }
+  | { ok: false; error: E };
+```
+
+### Usage Examples
+
+#### Basic Sequential Pipeline
+
+```typescript
+const pipeline = new Pipeline([
+  { key: "parse", action: async ({ parse }) => Result.ok(JSON.parse(parse)) },
+  { key: "validate", action: async ({ parse }) => Result.ok(validate(parse)) },
+  {
+    key: "store",
+    action: async ({ validate }) => Result.ok(db.insert(validate)),
+  },
+]);
+
+const result = await pipeline.execute("parse", '{"id":1}');
+// result.value = { store: insertResult }
+```
+
+#### Parallel Processing
+
+```typescript
+const pipeline = new Pipeline([
+  { key: "fetch", action: async () => Result.ok(await api.fetch()) },
+  [
+    {
+      key: "cache",
+      action: async ({ fetch }) => Result.ok(await redis.set(fetch)),
+    },
+    {
+      key: "notify",
+      action: async ({ fetch }) => Result.ok(await webhook.send(fetch)),
+    },
+    {
+      key: "log",
+      action: async ({ fetch }) => Result.ok(await logger.info(fetch)),
+    },
+  ],
+]);
+// cache, notify, and log execute simultaneously after fetch completes
+```
+
+#### Starting Mid-Pipeline
+
+```typescript
+// Skip initial stages, start from "process" step
+const result = await pipeline.execute("process", { data: "pre-processed" });
+// Only executes stages with order >= process.order
+```
+
+#### Deduplication (Single-Flight)
+
+```typescript
+// Three identical calls execute concurrently but only run the action once
+const [a, b, c] = await Promise.all([
+  pipeline.execute("expensive", "user-123"),
+  pipeline.execute("expensive", "user-123"),
+  pipeline.execute("expensive", "user-123"),
+]);
+// a, b, c receive the same result; action called exactly once
+```
+
+#### Cancellation & Abort Signals
+
+```typescript
+const controller = new AbortController();
+
+// Start execution with abort capability
+const executionPromise = pipeline.execute("long-running", params, {
+  signal: controller.signal,
+});
+
+// Cancel after 1 second
+setTimeout(() => controller.abort(), 1000);
+
+const result = await executionPromise;
+// result.ok === false with error.code === "CANCELLED"
+```
+
+#### Event Monitoring
+
+```typescript
+// Global events for monitoring
+pipeline.on("step:success", ({ stepKey, executionId, result }) => {
+  console.log(`[${executionId}] ${stepKey} completed`, result);
+});
+
+pipeline.on("failure", ({ stepKey, error }) => {
+  console.error(`Pipeline failed at ${stepKey}:`, error.message);
+});
+
+// Scoped events for a specific run
+const ctx = pipeline.prepare("step", params);
+ctx.on("step:success", (payload) => {
+  // Only triggered for this execution
+});
+await ctx.run();
+```
+
+#### Manual Context Management
+
+```typescript
+// For fine-grained control over concurrent runs
+const context = pipeline.prepare(
+  "process",
+  { data: "input" },
+  { signal: externalAbortSignal }, // optional
+);
+
+// Add scoped listeners
+context.on("stage:success", ({ order, carry }) => {
+  console.log(`Stage ${order} completed`);
+});
+
+// Execute with full control
+const result = await context.run();
+
+// Cancel individual run
+context.abort();
+```
+
+### Dynamic Step Registration
+
+```typescript
+const pipeline = new Pipeline();
+
+// Add steps at runtime
+pipeline
+  .step({ key: "a", order: 0, action: ok("first") })
+  .step({ key: "b", order: 1, action: ok("second") })
+  .step({ key: "c", order: 1, action: ok("parallel with b") });
+
+await pipeline.execute("a", null);
+```
+
+### Error Handling Patterns
+
+```typescript
+// Returning a failed Result stops the pipeline
+const step = {
+  key: "validate",
+  action: async (carry) => {
+    if (!carry.data.valid) {
+      return Result.fail(
+        new SystemError({
+          code: "VALIDATION_ERROR",
+          message: "Invalid data",
+          operation: "validate",
+        }),
+      );
+    }
+    return Result.ok(carry.data);
+  },
+};
+
+// Throwing also works (automatically wrapped in SystemError)
+const another = {
+  key: "risky",
+  action: async () => {
+    throw new Error("Something went wrong");
+    // Becomes SystemError with code INTERNAL_ERROR
+  },
+};
+```
+
+### Lifecycle Events
+
+| Event           | Payload                            | When                                 |
+| --------------- | ---------------------------------- | ------------------------------------ |
+| `start`         | `{ stepKey, executionId }`         | Before any steps execute             |
+| `step:success`  | `{ stepKey, executionId, result }` | Each successful step                 |
+| `step:failure`  | `{ stepKey, executionId, error }`  | Each failed step (even in parallel)  |
+| `stage:success` | `{ order, executionId, carry }`    | After all steps in a stage complete  |
+| `terminate`     | `{ carry, executionId }`           | When pipeline completes successfully |
+| `failure`       | `{ stepKey, executionId, error }`  | On first pipeline failure            |
+
+### API Reference
+
+#### `Pipeline` Class
+
+| Method                                          | Description                                               |
+| ----------------------------------------------- | --------------------------------------------------------- |
+| `constructor(definition?)`                      | Create pipeline with optional declarative definition      |
+| `step<TIn, TOut>(definition)`                   | Register or overwrite a step, returns `this` for chaining |
+| `execute<TIn>(stepKey, params, options?)`       | Execute pipeline starting from `stepKey`                  |
+| `prepare<TIn>(stepKey, params, options?, bus?)` | Create isolated execution context without running         |
+| `on<K>(event, handler)`                         | Subscribe to global pipeline events                       |
+| `destroy()`                                     | Clear all steps, listeners, and deduplication cache       |
+
+#### `PipelineExecutionContext` Class
+
+| Method                  | Description                                          |
+| ----------------------- | ---------------------------------------------------- |
+| `run()`                 | Execute the pipeline from this context's entry point |
+| `abort()`               | Cancel this specific execution                       |
+| `on<K>(event, handler)` | Subscribe to events scoped to this execution         |
+
+#### `Result` Utilities
+
+```typescript
+// Success
+Result.ok(value); // -> { ok: true, value }
+
+// Failure
+Result.fail(error); // -> { ok: false, error }
+
+// From the @core/error package
+Errors.internalError(original, message); // Creates SystemError
+Errors.invalidCommand(message); // Creates INVALID_COMMAND error
+```
+
+### Pipeline Definition Format
+
+The definition uses array position to determine stage order:
+
+```typescript
+type PipelineDefinition = Array<
+  PipelineStepDefinition | PipelineStepDefinition[]
+>;
+
+// Index 0 = stage order 0
+// Index 1 = stage order 1, etc.
+
+// Flat = single step per stage
+[{ key: "a" }, { key: "b" }, { key: "c" }][
+  // Nested = parallel steps at that stage
+  ({ key: "a" }, [{ key: "b" }, { key: "c" }], { key: "d" })
+];
+// Stage 0: a
+// Stage 1: b, c (parallel)
+// Stage 2: d
+```
+
+### Best Practices
+
+1. **Keep steps focused** - Each step should do one thing and do it well
+2. **Use descriptive keys** - Keys become property names in StageCarry
+3. **Handle errors gracefully** - Return `Result.fail()` with meaningful error codes
+4. **Leverage deduplication** - Identical concurrent executions are automatically deduplicated
+5. **Monitor with events** - Use lifecycle events for logging, metrics, and debugging
+6. **Abort signals for timeouts** - Pass AbortSignal to enable cancellation
+7. **Avoid side effects between stages** - Only pass data through StageCarry
+
+### Testing
+
+The pipeline is designed for testability:
+
+```typescript
+import { Pipeline } from "@core/pipeline";
+
+// Mock steps with controlled delays
+const mockAction = vi.fn().mockResolvedValue(Result.ok("mocked"));
+
+const pipeline = new Pipeline([{ key: "test", action: mockAction }]);
+
+await pipeline.execute("test", null);
+expect(mockAction).toHaveBeenCalled();
+```
+
+### See Also
+
+- [`@core/error`](./error) - SystemError and Result types
+- [`@core/events`](./events) - EventBus implementation
+
+This utility is particularly valuable for:
+
+- **Data processing pipelines** (ETL, image processing, document workflows)
+- **API orchestration** (parallel requests with aggregation)
+- **Build systems** (task graphs with dependencies)
+- **Form validation** (parallel validation rules)
+- **Background job processing** (staged workflows with cancellation)