automategraph 0.0.0-stage → 1.5.0

package/README.md

@@ -1,3 +1,419 @@
-# Temporary Holding Version
+# AutomateGraph
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+An isomorphic execution engine for graph definitions — typed conditional edges, deep-equality
+routing, explicit `Next`/`goto` routing, a function registry, and graph documents that load
+their own scripts. **Zero runtime dependencies**, one file for the core, identical API in
+Node.js and the browser.
+
+```bash
+npm install automategraph
+```
+
+```ts
+import { GraphExecutionEngine } from "automategraph";
+
+const engine = new GraphExecutionEngine({
+  id: "greeting",
+  version: "1.0.0",
+  entryNodes: ["classify"],
+  terminalNodes: ["respond"],
+  nodes: {
+    classify: {
+      id: "classify",
+      run: () => "priority",
+      next: [
+        { test: value => value === "priority", nextNode: "respond" },
+        { nextNode: "respond" }                       // else edge
+      ]
+    },
+    respond: { id: "respond", run: scope => "hello" }
+  }
+});
+
+const report = await engine.run();
+
+report.status;                    // "COMPLETED"
+report.results.map(r => r.nodeId); // ["classify", "respond"]
+```
+
+Three entry points:
+
+| Import | What |
+|---|---|
+| `automategraph` | the engine: `GraphExecutionEngine`, `Next`, `deepEquals`, `FunctionRegistry`, `lintGraph`, storage contract |
+| `automategraph/graph-json` | `parseGraphJSON()` / `createEngineFromJSON()` — graphs as data, with total path-carrying validation |
+| `automategraph/node/graph-scripts` | **Node-only** `loadGraphScripts()` — executes the scripts a document declares |
+
+`npm run example:edges`, `example:graphjson` and `example:scripts` in the repository show the
+three levels: typed edges, a JSON document with an injected registry, and a document that
+carries its own code.
+
+---
+
+## The test suite behind it
+
+This repository is also the engine's proof: **100 unit + 20 complex integration + 20
+async-callback + 19 upgrade + 11 conformance + 13 lint + 26 edges + 28 registry + 19 graphjson
++ 26 scripts = 282 tests, all passing**, with a self-contained web dashboard that shows every
+result, observation and finding.
+
+The engine is **isomorphic** (browser + Node, no Node APIs in the core — proven by a
+DOM-only TypeScript build) and ships a **storage adapter contract** with a conformance
+suite that any backend must pass.
+
+## Run it
+
+```bash
+npm install          # typescript + @types/node only
+npm test             # compile + run all 282 tests + regenerate the report
+npm run build:browser # prove the core is Node-free (DOM libs, no Node types)
+npm run serve        # serve report/index.html on http://0.0.0.0:4173
+npm run report       # test + serve in one shot
+```
+
+The runner prints a summary and writes:
+
+| Artifact | Purpose |
+|---|---|
+| `report/index.html` | self-contained dashboard (no external assets) |
+| `report/report.json` | machine-readable full results, logs and findings |
+
+While the server is running, `GET /run` recompiles and re-executes the whole suite;
+open pages auto-refresh when the results change.
+
+## Layout
+
+```
+src/automategraph.ts     the engine, kept verbatim
+tests/harness.ts         test registry, per-test timeout, log capture, runner
+tests/helpers.ts         fixtures: node/g/linear builders, SpyStorage, plugin factory, capture
+tests/unit.test.ts       100 unit tests (9 categories)
+tests/complex.test.ts    20 complex integration scenarios
+tests/async.test.ts      20 async node-callback cases
+tests/upgrade.test.ts    19 Wave-1 (v1.1) feature cases: policies, budgets, linting, isomorphism
+tests/conformance.test.ts 11 storage-adapter contract cases (checker with teeth + wire format)
+tests/lint.test.ts       13 cases for the v1.2 SUCCESS_DEAD_END rule
+tests/edges.test.ts      26 typed-edge cases: conditionals, else/parallel, Next/goto routing
+tests/registry.test.ts   28 function-registry cases: named run/test, lazy resolution, fingerprints
+tests/graphjson.test.ts  19 graph-JSON cases: parsing, total validation, engine from a data file
+tests/scripts.test.ts    26 script-loading cases: exports, self-registration, formats, the freeze
+tests/fixtures/scripts/  the script fixtures those cases load (cjs/mjs/js, global channel, collisions)
+src/graph-json.ts        the JSON loader: parseGraphJSON() + createEngineFromJSON()
+src/node/graph-scripts.ts  *** Node-only *** loadGraphScripts(): executes a document's scripts
+examples/                the a→b/c/d/e/f picture three ways (guards, router, typed edges)
+docs/diagrams/           C-19 resilience scenario: SVG, Mermaid (.mmd) and annotated notes
+src/storage-adapter.ts   the StorageAdapter contract + canonical JSON wire format
+src/storage-conformance.ts runStorageConformance(): ship it in your adapter's CI
+CHANGELOG.md             version history: v1.1 policies, v1.2 lint, v1.3 edges, v1.4 registry, v1.5 scripts
+UPGRADE-PROPOSAL.md      the roadmap this wave came from
+tests/report.ts          HTML/JSON report generator
+tests/runner.ts          entry point (npm test)
+tools/server.mjs         static report server with auto-refresh
+```
+
+## Coverage
+
+| Unit category | Tests |
+|---|---|
+| Graph validation (ids, versions, entries/terminals, refs, timeouts, retry bounds) | 12 |
+| Error classes & utilities (hierarchy, codes, ids, timestamps, throw coercion) | 19 |
+| EventBus (required vs non-required, FIFO, isolation, teardown) | 12 |
+| InMemoryStorage (atomicity, copy isolation, concurrency) | 8 |
+| Graph freezing (deep immutability of definitions) | 8 |
+| Engine lifecycle (state machine, single run, plugin init/shutdown) | 10 |
+| Execution & scheduler (paths, fan-out, contexts, events, checkpoints, settlement) | 20 |
+| Retry (RETRY looping, delayMs, attempt accounting) | 6 |
+| Cancellation (cooperative abort, events, terminal no-ops) | 5 |
+
+The 20 complex scenarios cover multi-branch pipelines, chaos graphs, FAIL_FAST,
+ONTHROW repair, diamond fan-in, cycles, 250-node concurrency barriers, 40-node
+checkpoint chains, plugin shutdown, mid-run checkpoint failure, timeouts,
+cancellation, event ordering and listener isolation. See the report for the
+full list with per-test findings.
+
+### Async node callbacks (`tests/async.test.ts`, 20 cases)
+
+`NodeDefinition.run` is typed `(context) => unknown | Promise<unknown>`, so `async`
+callbacks are first-class. This pack proves the async contract on every path:
+awaited resolution, thenable-returning non-async callbacks, `undefined` results,
+rejections (plain, `AutomateGraphError`-shaped), async retry + `onThrow` repair,
+async timeout preemption, cooperative abort (both the resolve-after-abort and
+reject-because-of-abort paths), checkpoints after async completion, awaited
+`context.emit` ordering, parallel async branches, mixed sync/async pipelines,
+abandoned-work isolation and a full plugin/storage integration run.
+
+These 20 cases are counted separately so your canonical **100 unit + 20 complex**
+suites stay intact.
+
+## Storage adapter contract (v1.1)
+
+Anything that can hold a JSON string can be a backend. Implement `StorageAdapter`
+(with required `saveAtomic`/`load` and optional `listExecutionIds`/`delete`/`close`,
+plus a `capabilities` self-description), then prove it:
+
+```ts
+import { runStorageConformance } from "../src/storage-conformance";
+
+const report = await runStorageConformance({
+  name: "my-sqlite-adapter",
+  create: () => createSqliteAdapter(path),
+  failNextSave: adapter => adapter.failNextSave() // optional: enables atomicity checks
+});
+
+if (!report.passed) {
+  console.error(report.violations); // ROUNDTRIP_FAILED, NO_ALIASING_ON_LOAD, …
+  process.exit(1);
+}
+```
+
+The checker is part of this repo and backs its own test suite, so the contract cannot
+drift from the tests that enforce it. It catches the classic adapter bugs: aliasing
+(storing live references), torn writes (committing before the failure is raised — it
+distinguishes *"rejects"* from *"rejects without corrupting state"*), broken listing,
+non-idempotent deletes and non-deterministic serialization. `serializeSnapshot()` /
+`deserializeSnapshot()` provide a versioned, byte-stable envelope for storage.
+
+## Guaranteed semantics (v1.1 → v1.5)
+
+Verified by named tests; the dashboard lists all 24 findings with the case that produced each.
+
+1. **Terminal completion is policy-driven** — `FIRST_TERMINAL` (default) latches on the first
+   terminal and drops the rest *visibly* via `activation.dropped`; `ALL_TERMINALS` waits for every
+   declared terminal; `DRAIN` executes all reachable work before settling (the opt-in fix for the
+   v1.0 silent-drop of ONTHROW recovery paths).
+2. **Cancellation is symmetric** — a callback that rejects because of an abort settles
+   `CANCELLED` with a handled `GRAPH_CANCELLED` record, exactly like one that resolves after it.
+3. **The drain has a deadline** — set `cancellationTimeoutMs` and ignored aborts cannot hang
+   `run()`: in-flight activations become `ABANDONED`, `drainTimedOut: true`, late outputs are
+   never adopted. Unset ⇒ v1.0 wait-forever.
+4. **`FAIL_FAST` is sticky at both levels** and is reported through `activation.dropped`.
+5. **`FALLBACK` actually routes** when `NodeDefinition.fallback` is declared (error handled);
+   without a target it degrades to `CONTINUE`.
+6. **Budgets hold** — `maxConcurrency` caps parallelism, `maxActivations` terminates runaway
+   cycles with `STEP_BUDGET_EXCEEDED` instead of spinning forever.
+7. **`NodeResult.error` carries the failure record** (including `NODE_TIMEOUT`).
+8. **Linting catches structural mistakes before they burn work** — unreachable nodes/terminals,
+   dead ends, duplicates, ignored fallback targets, and (v1.2) **success dead ends**
+   (`lint: "error"` rejects the run at pre-flight, before any node callback runs).
+9. **The scheduler is honest about drops** — every cleared activation is reported with a reason.
+10. **Fan-in has no join semantics** — a node with N incoming edges activates once per edge;
+    last writer wins (unchanged from v1.0; use `ALL_TERMINALS`/`DRAIN` or a map primitive to
+    model joins until Wave 4's join primitive lands).
+11. **Checkpoint failure is sticky**, keeps the last valid snapshot, and surfaces as
+    `CHECKPOINT_SAVE_FAILED` plus node-level `NODE_EXECUTION_ERROR`.
+12. **Work is notified properly** — `graph.finished` is the only emit whose rejection is caught;
+    a throwing required listener elsewhere escapes into the node error path (documented, test C-17).
+13. **The core is isomorphic** — no Node APIs, no Buffers; `setTimeout`/`AbortController` only.
+14. **A non-terminal node whose success path has nowhere to go is a lint error (v1.2)** —
+    `SUCCESS_DEAD_END` fires when a node has no `next` target but does declare error transitions
+    (`onThrow` / a live `fallback`). Its successful output is silently unreachable; the graph can
+    only "continue" when the node fails. `DEAD_END_NODE` remains for nodes with no transitions at
+    all. The two are mutually exclusive, and the engine never invents a successor.
+
+15. **`next` is an ordered edge list, not just a fan-out (v1.3)** — string entries stay
+    unconditional (parallel) as in v1.0, object entries `{ nextNode, result, test, parallel }`
+    are exclusive and evaluated in declaration order, first match wins. `result` matches by
+    deep equality, `result` + `test` together must both pass, and an entry with no matcher is
+    the `else` edge. No match ⇒ nothing is scheduled (silent).
+16. **A node can route itself (v1.3)** — `return new Next(target, value)` or
+    `context.goto(target)` followed by the normal `return value`. An explicit route replaces the
+    whole edge list for that activation, may target any node in the graph, and is validated at
+    runtime (`INVALID_GOTO_TARGET` when the target does not exist). `Next` wins when both are
+    used; each retry attempt starts with no route set.
+17. **Terminal detection happens before routing** — a node listed in `terminalNodes` latches
+    logical completion and never schedules successors, so its edges are decorative
+    (`TERMINAL_EDGE_IGNORED` warns about it).
+
+18. **A function may live in a registry (v1.4)** — `NodeDefinition.run` and `NextEdge.test`
+    accept the NAME of a registered function. The engine merges its own `{ functions }`
+    table with `GraphDefinition.functions` (graph entries win) and resolves names **lazily,
+    at evaluation time**, so `engine.registerFunction()` after construction — or mid-run —
+    still takes effect. `FnRegistry.register` validation and `FUNCTION_NAME_PATTERN` apply
+    to keys; the pattern allows namespacing (`vendor:retry`, `order.chargeCard`).
+19. **An unresolved name is `UNKNOWN_FUNCTION`, permanently (v1.4)** — the activation is
+    recorded FAILED with an unhandled error and NOTHING is scheduled: no retry, no
+    `onThrow`, no `fallback`. An unresolvable edge `test` records the error once per
+    node+name and simply never matches, so the rest of the chain still evaluates — the run
+    settles FAILED either way.
+20. **Identity is observable (v1.4)** — activations carry `fn` (results, `node.started`,
+    `node.succeeded`, `node.failed`), and the merged registry is reported as
+    `{ fingerprint, names }` on `graph.ready`, `ExecutionReport.registry` and every
+    checkpoint. The fingerprint is `fnr1:<count>:<hash>` over sorted names + function
+    source text, so it moves when an implementation changes and stays put when it does not.
+
+21. **A document may declare where its functions come from (v1.5)** — `scripts` entries carry
+    `{ namespace, path | source | url, format? }`. The isomorphic parser validates the
+    declarations (shape, exactly one source, unique namespaces, JSON paths) and the Node
+    loader (`src/node/graph-scripts.ts`) executes them. **Validation runs before any code
+    executes**: a document with a broken graph never loads a script. `createEngineFromJSON`
+    does not load scripts — the pipeline stays two-step.
+22. **A script provides functions in two ways (v1.5)** — it EXPORTS them (`module.exports = {…}`,
+    `export const …`, `export default {…}` — its functions are merged; non-function exports are
+    ignored and reported under `ignoredExports`), or it REGISTERS itself from inside its own
+    body via `globalThis.GraphEngine.registerEngineNamespace(namespace, functions)`. The loader
+    installs that global for the duration of the load only and restores the realm afterwards.
+    Names keep their exported spelling — **the namespace is a claim slot, not a name prefix**.
+23. **Namespaces are frozen, once per process (v1.5)** — a second registration of the same
+    namespace throws `NAMESPACE_ALREADY_REGISTERED`, from inside the script when it
+    self-registers. Claims are made at registration time: a script that registers and then
+    throws leaves its namespace claimed. `resetGraphScriptNamespaces()` clears the claims for
+    tests/dev; production should not need it. Two scripts providing the same NAME — in
+    different namespaces — is a separate gate: `FUNCTION_NAME_COLLISION` unless the load opts
+    into `{ collisions: "override" }`.
+24. **Formats are auto-detected with a safe fallback (v1.5)** — `.cjs` → `require`, `.mjs` →
+    dynamic `import`, `.js`/URL → source hint, then CJS, falling back to ESM (and the reverse on
+    a format error). An explicit `format` is never retried. Because the fallback REPLAYS the
+    script body, it is reserved for module-format failures (`ERR_REQUIRE_ESM`, ESM syntax under
+    `require`); a script that throws while running is executed exactly once.
+
+### Script loading (v1.5)
+
+```json
+{
+  "id": "order-flow", "version": "1.5.0",
+  "entryNodes": ["classify"], "terminalNodes": ["ship", "manual"],
+  "scripts": [
+    { "namespace": "order", "path": "./order.functions.cjs" },
+    { "namespace": "audit", "source": "globalThis.GraphEngine.registerEngineNamespace('audit', { auditDecision: () => 'ok' });" }
+  ],
+  "nodes": {
+    "classify": { "run": "classifyOrder", "next": [{ "test": "isPriority", "nextNode": "charge" }, { "nextNode": "manual" }] },
+    "charge": { "run": "chargeCard", "next": ["ship"] },
+    "ship": { "run": "shipOrder" },
+    "manual": { "run": "manualReview" }
+  }
+}
+```
+
+```ts
+import { loadGraphScripts } from "./src/node/graph-scripts";   // Node-only
+
+const { functions, namespaces, scripts, registry } = await loadGraphScripts(document);
+const engine = createEngineFromJSON(document, { functions, lint: "error" });
+
+registry.fingerprint;   // fnr1:7:… — the engine reports the same value
+```
+
+A script may equally not export anything and register itself:
+
+```js
+/* order.functions.cjs — either strategy, or both, in one file */
+module.exports = { classifyOrder: () => "priority", isPriority: v => v === "priority" };
+```
+
+**SECURITY.** Executing a script named by a document is arbitrary code execution with the full
+privileges of the process — there is no sandbox, deliberately (the same-realm strategy). Treat
+"who may supply a graph document" as a security boundary: load documents you trust, from
+directories you control. The loader is Node-only precisely so this capability never ships into
+a browser bundle by accident.
+
+### Function registry (v1.4)
+
+```ts
+const engine = new GraphExecutionEngine(graph, {
+  functions: {
+    ...nodeFunctions({ chargeCard: context => context.attempt }),   // (context: NodeContext)
+    ...edgeTests({ isPriority: value => value === "high" })          // (value: unknown)
+  }
+});
+
+engine.registerFunction("vendor:retry", context => "retried");       // any time
+
+// graph data may name them instead of inlining them:
+node("charge", "chargeCard", {
+  next: [{ test: "isPriority", nextNode: "ship" }, { nextNode: "manual" }]
+})
+```
+
+`nodeFunctions()` / `edgeTests()` exist because one table holds both signatures — a bare
+object literal cannot infer a parameter type from that union, these wrappers restore it.
+
+### Graph JSON loader (v1.4)
+
+`src/graph-json.ts` turns a document into a `GraphDefinition` — functions included, by name:
+
+```json
+{
+  "id": "order-flow", "version": "1.0.0",
+  "entryNodes": ["classify"], "terminalNodes": ["ship", "manual"],
+  "completionPolicy": "FIRST_TERMINAL",
+  "nodes": {
+    "classify": { "run": "classifyOrder",
+                  "next": [{ "test": "isPriority", "nextNode": "charge" },
+                           { "nextNode": "manual" }] },
+    "charge": { "run": "chargeCard", "next": ["ship"], "onThrow": "manual",
+                "errorPolicy": "RETRY", "retry": { "maxAttempts": 2 } },
+    "ship": { "run": "shipOrder" },
+    "manual": { "run": "manualReview" }
+  }
+}
+```
+
+```ts
+const engine = createEngineFromJSON(document, { functions, lint: "error" });
+```
+
+Parsing needs no registry (names resolve lazily), validation is total — every problem is
+collected with its JSON path (`nodes.a.next[0].test`) and thrown as one `GraphJSONError`
+carrying `issues` — node `id` defaults to its key, unknown properties are rejected at every
+level, and `functions` inside the document is an error that points at the engine options.
+Run `npm run example:graphjson` for the same document against four registries.
+
+### Typed edges and explicit routing (v1.3)
+
+```ts
+node("a", () => result, {
+  next: [
+    { result: 1, nextNode: "b" },              // deep equality
+    { test: r => r === 2, nextNode: "c" },     // predicate
+    { nextNode: "d" },                         // else edge
+    "audit"                                    // parallel: always fires
+  ],
+  onThrow: "e",                                // error transition (unchanged)
+  retry: { maxAttempts: 2 }, errorPolicy: "RETRY"
+})
+
+node("b", context => {                         // void node forcing a jump
+  context.goto("f");
+  return undefined;
+}, { next: ["f"] })                            // declared so lint sees f
+```
+
+Evaluation, after a node succeeds and its checkpoint is written:
+
+```
+if explicit route (Next | goto)  → schedule exactly that node, ignore every edge
+else for each edge in order:
+     parallel            → schedule (never blocked)
+     exclusive, taken?   → skip
+     exclusive, matches? → schedule, mark taken   // first match wins
+```
+
+New lint codes: `UNREACHABLE_EDGE` (an exclusive edge declared after an else edge),
+`PARALLEL_EDGE_MATCHER_IGNORED` (parallel edges are unconditional), `TERMINAL_EDGE_IGNORED`.
+
+See `examples/branching-edges.ts` for the `a → b/c/d/e/f` picture running as **one execution**
+— no router, no gate, no `DRAIN` policy — compared against the two earlier approaches in
+`examples/README.md`, and `npm run example:scripts` for a document whose functions arrive from
+a script file plus an inline self-registering script (including the freeze and the reset).
+
+### The success dead-end rule (v1.2)
+
+Discovered while explaining the C-19 resilience fixture, where `slow-vendor` declared
+`timeoutMs` + `onThrow` but **no `next`** — so it passed linting even though its success path
+(the 25% case where the vendor answers in time) led nowhere:
+
+```text
+SUCCESS_DEAD_END (error) · Node "slow-vendor" is not a terminal node and cannot continue
+after success (no next target); only its error transitions are declared.
+Add next targets or declare it as a terminal node
+```
+
+Fixes, in order of preference:
+
+1. **Handle the failure inside the node** so it always succeeds and has a single live edge:
+   `try { … } catch { return cachedPrice; }` then `next: ["decide"]`.
+2. **Declare `next`** for the success path — remember fan-in then produces one activation per
+   incoming edge (see `completionPolicy`).
+3. **Declare it a terminal node** if success really is the end — and pair it with
+   `completionPolicy: "ALL_TERMINALS"` / `"DRAIN"` so `FIRST_TERMINAL` cannot drop other branches.