tubeless 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Chet Mancini
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,106 @@
+# Tubeless
+
+Typed, observable data pipelines you import from TypeScript or run from a Bun
+CLI. It is a library, not a hosted workflow engine or a Make/npm-scripts
+replacement.
+
+The first public version is `0.1.0`. The API is still being proven; expect
+change before 1.0.
+
+```sh
+npm install tubeless
+```
+
+Also: `pnpm add tubeless`, `yarn add tubeless`, `bun add tubeless`. Library
+imports are ESM-only on Node.js 22+. The `tubeless` CLI requires Bun 1.3.14+;
+`npx tubeless` is not supported. Linux and macOS are supported; Windows is
+untested. See [install and runtime](./docs/getting-started.md#install).
+
+## Quick start
+
+```ts
+import { createSteps, definePipeline, requireOutputs } from "tubeless";
+
+interface ImportOptions {
+  lines: readonly string[];
+}
+
+const step = createSteps<ImportOptions>();
+
+const load = step("load", {
+  run: (_inputs, context) => context.options.lines,
+});
+
+const normalize = step("normalize", {
+  dependsOn: [load],
+  run: ({ load: rows }) => rows.map((row) => row.trim().toLowerCase()).filter(Boolean),
+});
+
+export const ImportPipeline = definePipeline({
+  id: "import",
+  steps: [load, normalize],
+  targets: [normalize],
+  finalize: requireOutputs([normalize], ({ normalize }) => normalize),
+});
+
+const rows = await ImportPipeline.runOrThrow({ lines: [" Alpha ", "", "Beta"] });
+// ["alpha", "beta"]
+```
+
+`createSteps` types domain options only. Runs accept built-in controls beside
+them. Use `runOrThrow` when failure should throw, `run` for the structured
+report, `plan` when nothing should execute, and `toMermaid` for the static
+graph.
+
+## Inspect, plan, or run
+
+```sh
+bunx tubeless inspect ./pipelines/my-pipeline.ts
+bunx tubeless plan ./pipelines/my-pipeline.ts --target publish --explain
+bunx tubeless graph ./pipelines/my-pipeline.ts --markdown
+bunx tubeless run ./pipelines/my-command.ts -- --source input.json --target publish
+```
+
+The CLI loads TypeScript modules with Bun. `run` executes a
+`definePipelineCommand` export; application flags go after `--`. See
+[the CLI](./docs/cli.md) and [the local studio](./docs/studio.md).
+
+## Choose the right pattern
+
+| You need to…                              | Start with                                                                |
+| ----------------------------------------- | ------------------------------------------------------------------------- |
+| Run typed steps in dependency order       | [Sequential pipeline](./examples/typed-import.ts)                         |
+| Validate external boundary values         | [Validated boundaries](./examples/validated-boundaries.ts)                |
+| Preview writes safely                     | [Dry runs and write gates](./examples/publish-with-gates.ts)              |
+| Skip work intentionally at runtime        | [Conditional step](./examples/conditional-step.ts)                        |
+| Continue independent work after a failure | [Best-effort execution](./examples/best-effort.ts)                        |
+| Reuse a pipeline inside another           | [Child pipeline](./examples/child-pipeline.ts)                            |
+| Run one child pipeline for many items     | [Fan-out and progress](./examples/fan-out-progress.ts)                    |
+| Resume long API work safely               | [Retry, rate limit, and checkpoint](./examples/resumable-enrichment.ts)   |
+| Turn a pipeline into a typed script       | [Pipeline CLI](./examples/cli-job.ts)                                     |
+| Render plans and errors consistently      | [Human and JSON rendering](./examples/rendering.ts)                       |
+| Test deterministic execution              | [Cancellation and test injection](./examples/cancellation-and-testing.ts) |
+| Export lifecycle telemetry                | [Structured tracing](./examples/tracing.ts)                               |
+| Persist and inspect local runs            | [Local observability](./examples/local-observability.ts)                  |
+
+The [recipe index](./docs/recipes.md) explains when to use each pattern.
+
+## Next
+
+- [Getting started](./docs/getting-started.md)
+- [CLI](./docs/cli.md)
+- [Local studio](./docs/studio.md)
+- [Core concepts](./docs/concepts.md)
+- [Documentation map](./docs/README.md)
+- [Generated API inventory](./docs/api-reference.md)
+- [Agent guide](./docs/agent-guide.md)
+
+## Contributing
+
+Pull requests are accepted for now; the maintainer set stays small. See
+[CONTRIBUTING.md](./CONTRIBUTING.md). Report vulnerabilities privately
+through [SECURITY.md](./SECURITY.md).
+
+## License
+
+MIT

package/dist/abort.d.ts

@@ -0,0 +1,12 @@
+/**
+ * Shared `AbortSignal` handling for every module that accepts one (pipeline execution,
+ * retry/backoff, rate limiting, batching). Centralized so they all produce the same
+ * abort-error shape and the same cancellable-sleep behavior, differing only in the
+ * `label` used in error messages (e.g. "Retry", "Pipeline run").
+ */
+/** True only for standard AbortErrors or errors created from an observed abort signal. */
+export declare function isAbortError(cause: unknown): cause is Error;
+export declare function createAbortError(signal: AbortSignal, label: string): Error;
+export declare function throwIfAborted(signal: AbortSignal | undefined, label: string): void;
+export declare function abortableSleep(durationMs: number, signal: AbortSignal | undefined, label: string): Promise<void>;
+//# sourceMappingURL=abort.d.ts.map

package/dist/abort.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"abort.d.ts","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAIH,0FAA0F;AAC1F,wBAAgB,YAAY,CAAC,KAAK,EAAE,OAAO,GAAG,KAAK,IAAI,KAAK,CAE3D;AAED,wBAAgB,gBAAgB,CAAC,MAAM,EAAE,WAAW,EAAE,KAAK,EAAE,MAAM,GAAG,KAAK,CAQ1E;AAED,wBAAgB,cAAc,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,EAAE,KAAK,EAAE,MAAM,GAAG,IAAI,CAInF;AAED,wBAAgB,cAAc,CAC5B,UAAU,EAAE,MAAM,EAClB,MAAM,EAAE,WAAW,GAAG,SAAS,EAC/B,KAAK,EAAE,MAAM,GACZ,OAAO,CAAC,IAAI,CAAC,CA0Bf"}

package/dist/abort.js

@@ -0,0 +1,49 @@
+/**
+ * Shared `AbortSignal` handling for every module that accepts one (pipeline execution,
+ * retry/backoff, rate limiting, batching). Centralized so they all produce the same
+ * abort-error shape and the same cancellable-sleep behavior, differing only in the
+ * `label` used in error messages (e.g. "Retry", "Pipeline run").
+ */
+const abortErrors = new WeakSet();
+/** True only for standard AbortErrors or errors created from an observed abort signal. */
+export function isAbortError(cause) {
+    return cause instanceof Error && (cause.name === "AbortError" || abortErrors.has(cause));
+}
+export function createAbortError(signal, label) {
+    const { reason } = signal;
+    const error = reason instanceof Error
+        ? reason
+        : new Error(reason === undefined ? `${label} aborted` : `${label} aborted: ${reason}`);
+    abortErrors.add(error);
+    return error;
+}
+export function throwIfAborted(signal, label) {
+    if (signal?.aborted) {
+        throw createAbortError(signal, label);
+    }
+}
+export function abortableSleep(durationMs, signal, label) {
+    if (durationMs <= 0) {
+        return Promise.resolve();
+    }
+    if (!signal) {
+        return new Promise((resolve) => setTimeout(resolve, durationMs));
+    }
+    if (signal.aborted) {
+        return Promise.reject(createAbortError(signal, label));
+    }
+    return new Promise((resolve, reject) => {
+        const cleanup = () => signal.removeEventListener("abort", onAbort);
+        const timeout = setTimeout(() => {
+            cleanup();
+            resolve();
+        }, durationMs);
+        const onAbort = () => {
+            clearTimeout(timeout);
+            cleanup();
+            reject(createAbortError(signal, label));
+        };
+        signal.addEventListener("abort", onAbort, { once: true });
+    });
+}
+//# sourceMappingURL=abort.js.map

package/dist/abort.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"abort.js","sourceRoot":"","sources":["../src/abort.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,MAAM,WAAW,GAAG,IAAI,OAAO,EAAS,CAAC;AAEzC,0FAA0F;AAC1F,MAAM,UAAU,YAAY,CAAC,KAAc;IACzC,OAAO,KAAK,YAAY,KAAK,IAAI,CAAC,KAAK,CAAC,IAAI,KAAK,YAAY,IAAI,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC;AAC3F,CAAC;AAED,MAAM,UAAU,gBAAgB,CAAC,MAAmB,EAAE,KAAa;IACjE,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,CAAC;IAC1B,MAAM,KAAK,GACT,MAAM,YAAY,KAAK;QACrB,CAAC,CAAC,MAAM;QACR,CAAC,CAAC,IAAI,KAAK,CAAC,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,GAAG,KAAK,UAAU,CAAC,CAAC,CAAC,GAAG,KAAK,aAAa,MAAM,EAAE,CAAC,CAAC;IAC3F,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IACvB,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,UAAU,cAAc,CAAC,MAA+B,EAAE,KAAa;IAC3E,IAAI,MAAM,EAAE,OAAO,EAAE,CAAC;QACpB,MAAM,gBAAgB,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;IACxC,CAAC;AACH,CAAC;AAED,MAAM,UAAU,cAAc,CAC5B,UAAkB,EAClB,MAA+B,EAC/B,KAAa;IAEb,IAAI,UAAU,IAAI,CAAC,EAAE,CAAC;QACpB,OAAO,OAAO,CAAC,OAAO,EAAE,CAAC;IAC3B,CAAC;IAED,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,UAAU,CAAC,CAAC,CAAC;IACnE,CAAC;IAED,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;QACnB,OAAO,OAAO,CAAC,MAAM,CAAC,gBAAgB,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;IACzD,CAAC;IAED,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACrC,MAAM,OAAO,GAAG,GAAG,EAAE,CAAC,MAAM,CAAC,mBAAmB,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QACnE,MAAM,OAAO,GAAG,UAAU,CAAC,GAAG,EAAE;YAC9B,OAAO,EAAE,CAAC;YACV,OAAO,EAAE,CAAC;QACZ,CAAC,EAAE,UAAU,CAAC,CAAC;QACf,MAAM,OAAO,GAAG,GAAG,EAAE;YACnB,YAAY,CAAC,OAAO,CAAC,CAAC;YACtB,OAAO,EAAE,CAAC;YACV,MAAM,CAAC,gBAAgB,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC,CAAC;QAC1C,CAAC,CAAC;QACF,MAAM,CAAC,gBAAgB,CAAC,OAAO,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;IAC5D,CAAC,CAAC,CAAC;AACL,CAAC"}

package/dist/batch.d.ts

@@ -0,0 +1,22 @@
+export interface RunConcurrentOptions {
+    /** Maximum in-flight workers. Defaults to one for deterministic sequential work. */
+    concurrency?: number;
+    /** Stops scheduling new items once aborted; in-flight workers are allowed to settle. */
+    signal?: AbortSignal;
+}
+export type ConcurrentWorker<T, R> = (item: T, index: number, signal?: AbortSignal) => Promise<R>;
+export declare function chunk<T>(items: readonly T[], size: number): T[][];
+/**
+ * Run individual items with bounded, lazy scheduling and input-order results.
+ *
+ * The first worker failure (or abort) stops future scheduling. Workers already
+ * in flight are allowed to settle before that failure is rethrown, avoiding
+ * orphaned work while keeping active work bounded by `concurrency`.
+ */
+export declare function runConcurrent<T, R>(items: readonly T[], options: RunConcurrentOptions, worker: ConcurrentWorker<T, R>): Promise<R[]>;
+export declare function runBatched<T, R>(items: readonly T[], options: {
+    size: number;
+    concurrency?: number;
+    signal?: AbortSignal;
+}, worker: (batch: T[], batchIndex: number) => Promise<R>): Promise<R[]>;
+//# sourceMappingURL=batch.d.ts.map

package/dist/batch.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"batch.d.ts","sourceRoot":"","sources":["../src/batch.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,oBAAoB;IACnC,oFAAoF;IACpF,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,wFAAwF;IACxF,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,MAAM,gBAAgB,CAAC,CAAC,EAAE,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,WAAW,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC;AAElG,wBAAgB,KAAK,CAAC,CAAC,EAAE,KAAK,EAAE,SAAS,CAAC,EAAE,EAAE,IAAI,EAAE,MAAM,GAAG,CAAC,EAAE,EAAE,CASjE;AASD;;;;;;GAMG;AACH,wBAAsB,aAAa,CAAC,CAAC,EAAE,CAAC,EACtC,KAAK,EAAE,SAAS,CAAC,EAAE,EACnB,OAAO,EAAE,oBAAoB,EAC7B,MAAM,EAAE,gBAAgB,CAAC,CAAC,EAAE,CAAC,CAAC,GAC7B,OAAO,CAAC,CAAC,EAAE,CAAC,CAEd;AAoDD,wBAAsB,UAAU,CAAC,CAAC,EAAE,CAAC,EACnC,KAAK,EAAE,SAAS,CAAC,EAAE,EACnB,OAAO,EAAE;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,WAAW,CAAC,EAAE,MAAM,CAAC;IAAC,MAAM,CAAC,EAAE,WAAW,CAAA;CAAE,EACrE,MAAM,EAAE,CAAC,KAAK,EAAE,CAAC,EAAE,EAAE,UAAU,EAAE,MAAM,KAAK,OAAO,CAAC,CAAC,CAAC,GACrD,OAAO,CAAC,CAAC,EAAE,CAAC,CAQd"}

package/dist/batch.js

@@ -0,0 +1,73 @@
+import { throwIfAborted } from "./abort.js";
+export function chunk(items, size) {
+    if (!Number.isInteger(size) || size <= 0) {
+        throw new Error(`chunk size must be a positive integer, got ${size}`);
+    }
+    const out = [];
+    for (let index = 0; index < items.length; index += size) {
+        out.push(items.slice(index, index + size));
+    }
+    return out;
+}
+function resolveConcurrency(concurrency) {
+    if (concurrency !== undefined && (!Number.isInteger(concurrency) || concurrency <= 0)) {
+        throw new Error(`concurrency must be a positive finite integer, got ${concurrency}`);
+    }
+    return Math.max(1, concurrency ?? 1);
+}
+/**
+ * Run individual items with bounded, lazy scheduling and input-order results.
+ *
+ * The first worker failure (or abort) stops future scheduling. Workers already
+ * in flight are allowed to settle before that failure is rethrown, avoiding
+ * orphaned work while keeping active work bounded by `concurrency`.
+ */
+export async function runConcurrent(items, options, worker) {
+    return runConcurrentWithLabel(items, options, worker, "Concurrent run");
+}
+async function runConcurrentWithLabel(items, options, worker, label) {
+    const concurrency = resolveConcurrency(options.concurrency);
+    if (items.length === 0) {
+        return [];
+    }
+    const results = new Array(items.length);
+    let failed = false;
+    let failure;
+    let nextIndex = 0;
+    async function runNext() {
+        while (!failed) {
+            try {
+                throwIfAborted(options.signal, label);
+            }
+            catch (error) {
+                failed = true;
+                failure = error;
+                return;
+            }
+            const index = nextIndex++;
+            if (index >= items.length) {
+                return;
+            }
+            try {
+                results[index] = await worker(items[index], index, options.signal);
+            }
+            catch (error) {
+                if (!failed) {
+                    failed = true;
+                    failure = error;
+                }
+                return;
+            }
+        }
+    }
+    await Promise.all(Array.from({ length: Math.min(concurrency, items.length) }, () => runNext()));
+    if (failed) {
+        throw failure;
+    }
+    return results;
+}
+export async function runBatched(items, options, worker) {
+    const batches = chunk(items, options.size);
+    return runConcurrentWithLabel(batches, options, (batch, index) => worker(batch, index), "Batch run");
+}
+//# sourceMappingURL=batch.js.map

package/dist/batch.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"batch.js","sourceRoot":"","sources":["../src/batch.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAW5C,MAAM,UAAU,KAAK,CAAI,KAAmB,EAAE,IAAY;IACxD,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,EAAE,CAAC;QACzC,MAAM,IAAI,KAAK,CAAC,8CAA8C,IAAI,EAAE,CAAC,CAAC;IACxE,CAAC;IACD,MAAM,GAAG,GAAU,EAAE,CAAC;IACtB,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,KAAK,IAAI,IAAI,EAAE,CAAC;QACxD,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,KAAK,GAAG,IAAI,CAAC,CAAC,CAAC;IAC7C,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,kBAAkB,CAAC,WAA+B;IACzD,IAAI,WAAW,KAAK,SAAS,IAAI,CAAC,CAAC,MAAM,CAAC,SAAS,CAAC,WAAW,CAAC,IAAI,WAAW,IAAI,CAAC,CAAC,EAAE,CAAC;QACtF,MAAM,IAAI,KAAK,CAAC,sDAAsD,WAAW,EAAE,CAAC,CAAC;IACvF,CAAC;IACD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,WAAW,IAAI,CAAC,CAAC,CAAC;AACvC,CAAC;AAED;;;;;;GAMG;AACH,MAAM,CAAC,KAAK,UAAU,aAAa,CACjC,KAAmB,EACnB,OAA6B,EAC7B,MAA8B;IAE9B,OAAO,sBAAsB,CAAC,KAAK,EAAE,OAAO,EAAE,MAAM,EAAE,gBAAgB,CAAC,CAAC;AAC1E,CAAC;AAED,KAAK,UAAU,sBAAsB,CACnC,KAAmB,EACnB,OAA6B,EAC7B,MAA8B,EAC9B,KAAa;IAEb,MAAM,WAAW,GAAG,kBAAkB,CAAC,OAAO,CAAC,WAAW,CAAC,CAAC;IAC5D,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACvB,OAAO,EAAE,CAAC;IACZ,CAAC;IAED,MAAM,OAAO,GAAG,IAAI,KAAK,CAAI,KAAK,CAAC,MAAM,CAAC,CAAC;IAC3C,IAAI,MAAM,GAAG,KAAK,CAAC;IACnB,IAAI,OAAgB,CAAC;IACrB,IAAI,SAAS,GAAG,CAAC,CAAC;IAElB,KAAK,UAAU,OAAO;QACpB,OAAO,CAAC,MAAM,EAAE,CAAC;YACf,IAAI,CAAC;gBACH,cAAc,CAAC,OAAO,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;YACxC,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,MAAM,GAAG,IAAI,CAAC;gBACd,OAAO,GAAG,KAAK,CAAC;gBAChB,OAAO;YACT,CAAC;YAED,MAAM,KAAK,GAAG,SAAS,EAAE,CAAC;YAC1B,IAAI,KAAK,IAAI,KAAK,CAAC,MAAM,EAAE,CAAC;gBAC1B,OAAO;YACT,CAAC;YAED,IAAI,CAAC;gBACH,OAAO,CAAC,KAAK,CAAC,GAAG,MAAM,MAAM,CAAC,KAAK,CAAC,KAAK,CAAE,EAAE,KAAK,EAAE,OAAO,CAAC,MAAM,CAAC,CAAC;YACtE,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,MAAM,EAAE,CAAC;oBACZ,MAAM,GAAG,IAAI,CAAC;oBACd,OAAO,GAAG,KAAK,CAAC;gBAClB,CAAC;gBACD,OAAO;YACT,CAAC;QACH,CAAC;IACH,CAAC;IAED,MAAM,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC;IAChG,IAAI,MAAM,EAAE,CAAC;QACX,MAAM,OAAO,CAAC;IAChB,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,UAAU,CAC9B,KAAmB,EACnB,OAAqE,EACrE,MAAsD;IAEtD,MAAM,OAAO,GAAG,KAAK,CAAC,KAAK,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3C,OAAO,sBAAsB,CAC3B,OAAO,EACP,OAAO,EACP,CAAC,KAAK,EAAE,KAAK,EAAE,EAAE,CAAC,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,EACtC,WAAW,CACZ,CAAC;AACJ,CAAC"}

package/dist/checkpoint.d.ts

@@ -0,0 +1,33 @@
+export interface CheckpointStore<TMeta = unknown> {
+    has(key: string): boolean;
+    record(key: string, meta?: TMeta): void;
+    entries(): ReadonlyMap<string, TMeta | undefined>;
+    flush(): void;
+    /**
+     * Whether to call this is the caller's choice: clear-on-completion suits one-shot
+     * resumable jobs (a full successful run means "start fresh next time"), while
+     * never-clear suits pipelines with an incrementally-growing "done" set (e.g. entity
+     * enrichment) where auto-clearing would cause reprocessing and duplicate output.
+     */
+    clear(): void;
+}
+export interface OpenCheckpointOptions {
+    /**
+     * Called when the checkpoint file exists but fails to parse, right before falling
+     * back to an empty checkpoint. Starting fresh from a corrupt file is silent
+     * reprocessing of every previously-checkpointed item — the default logs a warning to
+     * `console` so that cost is visible instead of masked. Pass a no-op to suppress it.
+     */
+    onCorruptFile?: (cause: unknown) => void;
+}
+export declare function openCheckpoint<TMeta = unknown>(filePath: string, options?: OpenCheckpointOptions): CheckpointStore<TMeta>;
+/**
+ * Runs `persist()`, and only once it resolves without throwing, records every item in
+ * `batch` into `checkpoint` and flushes it. A `persist` failure (a write error, etc.)
+ * propagates and leaves the checkpoint untouched for this batch — the caller decides
+ * whether to skip the batch and retry it later or fail the whole run. This makes the
+ * checkpoint-outruns-persistence ordering bug impossible to write by construction:
+ * there is no way to call `checkpoint.record()` before `persist()` has already succeeded.
+ */
+export declare function withCheckpointedBatch<TItem>(checkpoint: CheckpointStore, batch: readonly TItem[], keyOf: (item: TItem) => string, persist: () => Promise<void> | void): Promise<void>;
+//# sourceMappingURL=checkpoint.d.ts.map

package/dist/checkpoint.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"checkpoint.d.ts","sourceRoot":"","sources":["../src/checkpoint.ts"],"names":[],"mappings":"AAGA,MAAM,WAAW,eAAe,CAAC,KAAK,GAAG,OAAO;IAC9C,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC;IAC1B,MAAM,CAAC,GAAG,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC;IACxC,OAAO,IAAI,WAAW,CAAC,MAAM,EAAE,KAAK,GAAG,SAAS,CAAC,CAAC;IAClD,KAAK,IAAI,IAAI,CAAC;IACd;;;;;OAKG;IACH,KAAK,IAAI,IAAI,CAAC;CACf;AAsBD,MAAM,WAAW,qBAAqB;IACpC;;;;;OAKG;IACH,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,CAAC;CAC1C;AAED,wBAAgB,cAAc,CAAC,KAAK,GAAG,OAAO,EAC5C,QAAQ,EAAE,MAAM,EAChB,OAAO,GAAE,qBAA0B,GAClC,eAAe,CAAC,KAAK,CAAC,CA2CxB;AAED;;;;;;;GAOG;AACH,wBAAsB,qBAAqB,CAAC,KAAK,EAC/C,UAAU,EAAE,eAAe,EAC3B,KAAK,EAAE,SAAS,KAAK,EAAE,EACvB,KAAK,EAAE,CAAC,IAAI,EAAE,KAAK,KAAK,MAAM,EAC9B,OAAO,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,GAClC,OAAO,CAAC,IAAI,CAAC,CAMf"}

package/dist/checkpoint.js

@@ -0,0 +1,71 @@
+import * as fs from "fs";
+import * as path from "path";
+function loadEntries(filePath, onCorruptFile) {
+    if (!fs.existsSync(filePath)) {
+        return new Map();
+    }
+    try {
+        // SAFETY: the file is expected to hold a JSON object whose values are
+        // TMeta entries, as written by this store's flush(). Object.entries only
+        // yields string keys, so the `as Record<string, TMeta>` documents the
+        // caller-supplied TMeta contract; any mismatch is caught by the try/catch.
+        const parsed = JSON.parse(fs.readFileSync(filePath, "utf8"));
+        return new Map(Object.entries(parsed));
+    }
+    catch (error) {
+        onCorruptFile(error);
+        return new Map();
+    }
+}
+export function openCheckpoint(filePath, options = {}) {
+    const onCorruptFile = options.onCorruptFile ??
+        ((cause) => {
+            console.warn(`Checkpoint file ${filePath} is corrupt; starting fresh (all previously checkpointed items will be reprocessed): ${cause instanceof Error ? cause.message : String(cause)}`);
+        });
+    const entries = loadEntries(filePath, onCorruptFile);
+    return {
+        has: (key) => entries.has(key),
+        record: (key, meta) => {
+            entries.set(key, meta);
+        },
+        entries: () => entries,
+        flush: () => {
+            fs.mkdirSync(path.dirname(filePath), { recursive: true });
+            // Records made without a `meta` argument store `undefined`, which JSON.stringify
+            // silently drops from object properties — without the replacer below, every
+            // meta-less key would vanish from the file on flush. Substituting `null` keeps
+            // the key present; callers that never pass `meta` only care about `has()`, not
+            // the exact value round-tripped through `entries()`.
+            const json = JSON.stringify(Object.fromEntries(entries), (_key, value) => (value === undefined ? null : value), 2);
+            // Write to a sibling temp file and rename over the destination rather than
+            // writeFileSync-ing the target directly: a crash or process kill mid-write would
+            // otherwise leave a truncated, unparseable checkpoint file, and the next run would
+            // silently treat it as empty and reprocess everything already done. Rename is
+            // atomic on the same filesystem, so the checkpoint file is always either the
+            // previous complete flush or the new complete one, never a partial write.
+            const tmpPath = `${filePath}.tmp-${process.pid}`;
+            fs.writeFileSync(tmpPath, `${json}\n`);
+            fs.renameSync(tmpPath, filePath);
+        },
+        clear: () => {
+            entries.clear();
+            fs.rmSync(filePath, { force: true });
+        },
+    };
+}
+/**
+ * Runs `persist()`, and only once it resolves without throwing, records every item in
+ * `batch` into `checkpoint` and flushes it. A `persist` failure (a write error, etc.)
+ * propagates and leaves the checkpoint untouched for this batch — the caller decides
+ * whether to skip the batch and retry it later or fail the whole run. This makes the
+ * checkpoint-outruns-persistence ordering bug impossible to write by construction:
+ * there is no way to call `checkpoint.record()` before `persist()` has already succeeded.
+ */
+export async function withCheckpointedBatch(checkpoint, batch, keyOf, persist) {
+    await persist();
+    for (const item of batch) {
+        checkpoint.record(keyOf(item));
+    }
+    checkpoint.flush();
+}
+//# sourceMappingURL=checkpoint.js.map

package/dist/checkpoint.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"checkpoint.js","sourceRoot":"","sources":["../src/checkpoint.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,IAAI,CAAC;AACzB,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AAgB7B,SAAS,WAAW,CAClB,QAAgB,EAChB,aAAuC;IAEvC,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;QAC7B,OAAO,IAAI,GAAG,EAAE,CAAC;IACnB,CAAC;IACD,IAAI,CAAC;QACH,sEAAsE;QACtE,yEAAyE;QACzE,sEAAsE;QACtE,2EAA2E;QAC3E,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,YAAY,CAAC,QAAQ,EAAE,MAAM,CAAC,CAA0B,CAAC;QACtF,OAAO,IAAI,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,CAAC;IACzC,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,aAAa,CAAC,KAAK,CAAC,CAAC;QACrB,OAAO,IAAI,GAAG,EAAE,CAAC;IACnB,CAAC;AACH,CAAC;AAYD,MAAM,UAAU,cAAc,CAC5B,QAAgB,EAChB,OAAO,GAA0B,EAAE;IAEnC,MAAM,aAAa,GACjB,OAAO,CAAC,aAAa;QACrB,CAAC,CAAC,KAAc,EAAE,EAAE;YAClB,OAAO,CAAC,IAAI,CACV,mBAAmB,QAAQ,wFAAwF,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAC5K,CAAC;QACJ,CAAC,CAAC,CAAC;IACL,MAAM,OAAO,GAAG,WAAW,CAAQ,QAAQ,EAAE,aAAa,CAAC,CAAC;IAE5D,OAAO;QACL,GAAG,EAAE,CAAC,GAAG,EAAE,EAAE,CAAC,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC;QAC9B,MAAM,EAAE,CAAC,GAAG,EAAE,IAAI,EAAE,EAAE;YACpB,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;QACzB,CAAC;QACD,OAAO,EAAE,GAAG,EAAE,CAAC,OAAO;QACtB,KAAK,EAAE,GAAG,EAAE;YACV,EAAE,CAAC,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;YAC1D,iFAAiF;YACjF,4EAA4E;YAC5E,+EAA+E;YAC/E,+EAA+E;YAC/E,qDAAqD;YACrD,MAAM,IAAI,GAAG,IAAI,CAAC,SAAS,CACzB,MAAM,CAAC,WAAW,CAAC,OAAO,CAAC,EAC3B,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,EACrD,CAAC,CACF,CAAC;YACF,2EAA2E;YAC3E,iFAAiF;YACjF,mFAAmF;YACnF,8EAA8E;YAC9E,6EAA6E;YAC7E,0EAA0E;YAC1E,MAAM,OAAO,GAAG,GAAG,QAAQ,QAAQ,OAAO,CAAC,GAAG,EAAE,CAAC;YACjD,EAAE,CAAC,aAAa,CAAC,OAAO,EAAE,GAAG,IAAI,IAAI,CAAC,CAAC;YACvC,EAAE,CAAC,UAAU,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;QACnC,CAAC;QACD,KAAK,EAAE,GAAG,EAAE;YACV,OAAO,CAAC,KAAK,EAAE,CAAC;YAChB,EAAE,CAAC,MAAM,CAAC,QAAQ,EAAE,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;QACvC,CAAC;KACF,CAAC;AACJ,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,qBAAqB,CACzC,UAA2B,EAC3B,KAAuB,EACvB,KAA8B,EAC9B,OAAmC;IAEnC,MAAM,OAAO,EAAE,CAAC;IAChB,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,UAAU,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC;IACjC,CAAC;IACD,UAAU,CAAC,KAAK,EAAE,CAAC;AACrB,CAAC"}

package/dist/child-execution.d.ts

@@ -0,0 +1,30 @@
+import { type ToMappedChildStepProgressOptions } from "./mapped-child-progress.js";
+import { type Pipeline, type PipelineExecutionContext, type PipelineRun, type PipelineRunOptions, type PipelineStepContext } from "./pipeline.js";
+export declare class PipelineChildError extends Error {
+    readonly cancelled: boolean;
+    constructor(message: string, cancelled?: boolean, cause?: unknown);
+}
+type ChildPipeline = Pipeline<object, unknown, string, string>;
+type ChildInputs = Record<string, unknown>;
+interface ChildExecutionDependencies<TParentOptions extends object> {
+    createExecutionError(result: PipelineRun<unknown>, message: string): Error;
+    isCancellation(cause: unknown, context: PipelineExecutionContext<TParentOptions>): boolean;
+}
+export interface SingleChildExecutionConfig<TParentOptions extends object> {
+    pipeline: ChildPipeline;
+    mapOptions(inputs: ChildInputs, context: PipelineExecutionContext<TParentOptions>): PipelineRunOptions;
+    mapResult?(value: unknown, result: PipelineRun<unknown>, context: PipelineStepContext<TParentOptions>): unknown;
+}
+export interface MappedChildExecutionConfig<TParentOptions extends object> {
+    pipeline: ChildPipeline;
+    items(inputs: ChildInputs, context: PipelineExecutionContext<TParentOptions>): readonly unknown[] | Promise<readonly unknown[]>;
+    key(item: unknown, index: number): string;
+    concurrency?: number | ((inputs: ChildInputs, context: PipelineExecutionContext<TParentOptions>) => number);
+    progress?: ToMappedChildStepProgressOptions;
+    mapOptions(item: unknown, index: number, inputs: ChildInputs, context: PipelineExecutionContext<TParentOptions>): PipelineRunOptions;
+    mapResult?(value: unknown, result: PipelineRun<unknown>, item: unknown, index: number, context: PipelineStepContext<TParentOptions>): unknown;
+}
+export declare function createSingleChildRunner<TParentOptions extends object>(config: SingleChildExecutionConfig<TParentOptions>, dependencies: ChildExecutionDependencies<TParentOptions>): (inputs: ChildInputs, context: PipelineStepContext<TParentOptions>) => Promise<unknown>;
+export declare function createMappedChildRunner<TParentOptions extends object>(config: MappedChildExecutionConfig<TParentOptions>, dependencies: ChildExecutionDependencies<TParentOptions>): (inputs: ChildInputs, context: PipelineStepContext<TParentOptions>) => Promise<unknown[]>;
+export {};
+//# sourceMappingURL=child-execution.d.ts.map

package/dist/child-execution.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"child-execution.d.ts","sourceRoot":"","sources":["../src/child-execution.ts"],"names":[],"mappings":"AAEA,OAAO,EAGL,KAAK,gCAAgC,EACtC,MAAM,4BAA4B,CAAC;AAEpC,OAAO,EAGL,KAAK,QAAQ,EAGb,KAAK,wBAAwB,EAG7B,KAAK,WAAW,EAEhB,KAAK,kBAAkB,EACvB,KAAK,mBAAmB,EACzB,MAAM,eAAe,CAAC;AAGvB,qBAAa,kBAAmB,SAAQ,KAAK;IAGzC,QAAQ,CAAC,SAAS;IAFpB,YACE,OAAO,EAAE,MAAM,EACN,SAAS,UAAQ,EAC1B,KAAK,CAAC,EAAE,OAAO,EAIhB;CACF;AAED,KAAK,aAAa,GAAG,QAAQ,CAAC,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;AAC/D,KAAK,WAAW,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;AAE3C,UAAU,0BAA0B,CAAC,cAAc,SAAS,MAAM;IAChE,oBAAoB,CAAC,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,EAAE,OAAO,EAAE,MAAM,GAAG,KAAK,CAAC;IAC3E,cAAc,CAAC,KAAK,EAAE,OAAO,EAAE,OAAO,EAAE,wBAAwB,CAAC,cAAc,CAAC,GAAG,OAAO,CAAC;CAC5F;AAED,MAAM,WAAW,0BAA0B,CAAC,cAAc,SAAS,MAAM;IACvE,QAAQ,EAAE,aAAa,CAAC;IACxB,UAAU,CACR,MAAM,EAAE,WAAW,EACnB,OAAO,EAAE,wBAAwB,CAAC,cAAc,CAAC,GAChD,kBAAkB,CAAC;IACtB,SAAS,CAAC,CACR,KAAK,EAAE,OAAO,EACd,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,EAC5B,OAAO,EAAE,mBAAmB,CAAC,cAAc,CAAC,GAC3C,OAAO,CAAC;CACZ;AAED,MAAM,WAAW,0BAA0B,CAAC,cAAc,SAAS,MAAM;IACvE,QAAQ,EAAE,aAAa,CAAC;IACxB,KAAK,CACH,MAAM,EAAE,WAAW,EACnB,OAAO,EAAE,wBAAwB,CAAC,cAAc,CAAC,GAChD,SAAS,OAAO,EAAE,GAAG,OAAO,CAAC,SAAS,OAAO,EAAE,CAAC,CAAC;IACpD,GAAG,CAAC,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAAC;IAC1C,WAAW,CAAC,EACR,MAAM,GACN,CAAC,CAAC,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,wBAAwB,CAAC,cAAc,CAAC,KAAK,MAAM,CAAC,CAAC;IACzF,QAAQ,CAAC,EAAE,gCAAgC,CAAC;IAC5C,UAAU,CACR,IAAI,EAAE,OAAO,EACb,KAAK,EAAE,MAAM,EACb,MAAM,EAAE,WAAW,EACnB,OAAO,EAAE,wBAAwB,CAAC,cAAc,CAAC,GAChD,kBAAkB,CAAC;IACtB,SAAS,CAAC,CACR,KAAK,EAAE,OAAO,EACd,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,EAC5B,IAAI,EAAE,OAAO,EACb,KAAK,EAAE,MAAM,EACb,OAAO,EAAE,mBAAmB,CAAC,cAAc,CAAC,GAC3C,OAAO,CAAC;CACZ;AA0GD,wBAAgB,uBAAuB,CAAC,cAAc,SAAS,MAAM,EACnE,MAAM,EAAE,0BAA0B,CAAC,cAAc,CAAC,EAClD,YAAY,EAAE,0BAA0B,CAAC,cAAc,CAAC,GACvD,CAAC,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,mBAAmB,CAAC,cAAc,CAAC,KAAK,OAAO,CAAC,OAAO,CAAC,CAkDzF;AAED,wBAAgB,uBAAuB,CAAC,cAAc,SAAS,MAAM,EACnE,MAAM,EAAE,0BAA0B,CAAC,cAAc,CAAC,EAClD,YAAY,EAAE,0BAA0B,CAAC,cAAc,CAAC,GACvD,CAAC,MAAM,EAAE,WAAW,EAAE,OAAO,EAAE,mBAAmB,CAAC,cAAc,CAAC,KAAK,OAAO,CAAC,OAAO,EAAE,CAAC,CAgL3F"}