fyflow-orchestration 0.7.0 → 0.8.0

package/README.md

@@ -14,12 +14,10 @@ worker returned, routing can **fan out** into parallel branches, and a **join**
                          +--> [review]        (low confidence routes away entirely)
 ```
 
-The scheduler runs tasks. This library decides **which task runs next, and why** — because
-`fyflow-scheduler` has no dependency graph of its own, so sequence, decision, fan-out and join are
-built here rather than wrapped from below. See [doc/concept.md](doc/concept.md) for the reasoning.
+The scheduler runs tasks; this library decides **which task runs next**. `fyflow-scheduler` has no
+dependency graph of its own, so sequence, decision, fan-out and join live here.
 
-**0.7.0 is the first release**, on npm as `fyflow-orchestration` and on JSR as `@fyflow/orchestration`.
-The manifests read `0.7.0`.
+Published on npm as `fyflow-orchestration` and on JSR as `@fyflow/orchestration`. The manifests read `0.8.0`.
 
 ## Install
 
@@ -28,33 +26,27 @@ npm install fyflow-orchestration fyflow-scheduler    # Node, and a browser bundl
 deno add jsr:@fyflow/orchestration                  # Deno
 ```
 
-npm carries an esbuild bundle per platform (`dist/node`, `dist/browser`) and the type declarations;
-JSR carries the TypeScript source. Both carry this README, [AGENTS.md](AGENTS.md) — the complete
-reference — and the `LICENSE` file.
-
-The scheduler is a **peer dependency**, so your installer resolves it rather than this library
-bundling it: `fyflow-scheduler` is pinned to **0.5.2** and installed from npm, and under Deno it comes
-from JSR at the same version. **Node 24 or a current Deno.** The library's own suites also run in the
-browser, but a flow does not: a worker there has to be a bundled `.js` asset, because nothing in a
-browser executes TypeScript.
+- **Runtimes:** Node 24 or a current Deno. A flow does not run in a browser: a worker there would have
+  to be a bundled `.js` asset.
+- **The scheduler is a peer dependency**, pinned to the exact version in this package's
+  `peerDependencies`. Deno takes it from JSR at the same version.
+- **What ships:** npm has bundles for Node and browser bundlers plus type declarations, and JSR has
+  the TypeScript source. Both include [AGENTS.md](AGENTS.md), the complete reference.
 
 ## Getting started
 
-**Two JSON documents, and one file of your own.** A **registry** says what exists — a worker module,
-and a node type that runs on it. A **flow** says what happens — nodes naming those types, and the
-edges between them. `createRuntime` builds everything the registry names, and each run hands back
-`{ driver, done, header }` — the driver, a promise for the run's end, and the run's checkpoint header
-(the example below needs only `done`, so that is all it names). The one thing you write as code is the
-worker — [examples/getting-started/worker.ts](examples/getting-started/worker.ts), a class with a
-`run` method, and nothing else.
+**Two JSON documents, and one file of your own.** A **registry** says what exists: worker modules,
+and the node types that run on them. A **flow** says what happens: nodes naming those types, and the
+edges between them. `createRuntime` builds what the registry names, and `start` runs a flow on it.
+The only code you write is the worker,
+[examples/getting-started/worker.ts](examples/getting-started/worker.ts): a class with a `run` method.
 
 ```ts
 import { createRuntime } from 'fyflow-orchestration/scheduler';
 import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
 
 // THE REGISTRY — what exists. One worker module, and one node type that runs on it. This is data:
-// in your program it is a JSON file you import. The worker url is relative, and `base` below says
-// to what; the file itself is `examples/getting-started/worker.ts` — a class with a `run` method.
+// in your program it is a JSON file you import. The worker url is relative to `base` below.
 const registry: RegistryDocument = {
   workers: {
     greeter: {
@@ -66,9 +58,7 @@ const registry: RegistryDocument = {
 };
 
 // THE FLOW — what happens. Two nodes of that type, each with its own `settings`, then an end. A flow
-// names registry KEYS and nothing else — no file, no url, no class — so it is portable across
-// runtimes that agree on the keys. The end node's `outcome` is a LABEL of yours that the driver does
-// not interpret; the run's own outcome is the fold's — `state.outcome` below.
+// names registry keys only (no files, urls or classes), so it is portable between runtimes.
 const flow: FlowDefinition = {
   flowId: 'hello', version: 1, entry: 'greet',
   nodes: [
@@ -78,29 +68,22 @@ const flow: FlowDefinition = {
   ],
 };
 
-// ONE call builds the pool, the scheduler and everything between, and refuses every defect in the
-// REGISTRY at construction, by name. It never sees the flow: a defect THERE is refused where the
-// flow arrives, at `start`, which is what builds the `Driver` — so a `settings` value the worker's
-// schema rejects names its node from the call three lines below, not from this one. `base` is what
-// relative worker urls resolve against — `import.meta.url` when the registry sits beside this file.
+// ONE call builds the pools and the scheduler, and refuses a defective registry here, by name. A
+// defect in the FLOW is refused by `start`, below.
 const runtime = createRuntime(registry, {}, { base: import.meta.url });
 
-// `start` hands back `{ driver, done }` — the driver already running, and `done` a promise that
-// resolves with the run's terminal `RunState` (ADR 0058). **It never rejects**: `failed`, `stuck`
-// and `stopped` are terminal OUTCOMES, so a run that goes wrong arrives as `state.outcome` and not
-// as a `catch`. This snippet wants only the end, so it names only `done`. *A run with a person in
-// it does not end by itself — section 7 is where that is taught, and section 8 is what `done` waits
-// for.*
+// `start` runs the flow. `done` resolves with the run's final state and never rejects: a run that
+// goes wrong arrives as `state.outcome` ('failed', 'stuck', 'halted'), not as a `catch`.
 const { done } = runtime.start(flow, { text: 'hello' });
 const state = await done;
 await runtime.shutdown();
 
-// What the run did, read off its state: `greet` turned `hello` into `hello world`, and `shout` was
-// handed THAT as its input — the previous node's result is the next node's `input`.
+// The previous node's result is the next node's input: `greet` made `hello world`, `shout` added `!`.
 const shout = Object.values(state.nodes).find((n) => n.nodeId === 'shout');
 console.log(state.outcome);          // 'completed'
 console.log(shout?.result);          // { text: 'hello world !' }
 
+// Verified by the docs gate:
 if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
 if (JSON.stringify(shout?.result) !== '{"text":"hello world !"}') throw new Error(`shout returned ${JSON.stringify(shout?.result)}`);
 ```
@@ -108,37 +91,32 @@ if (JSON.stringify(shout?.result) !== '{"text":"hello world !"}') throw new Erro
 
 ## A flow that decides
 
-**The same call, with the node kinds that make it a flow.** A **decision** is an edge with a `when`
-naming a predicate — code you supply, under a name the registry declares. A **fan-out** opens
-branches that run concurrently; a **join** waits for them and hands the next node every branch's
-result keyed by branch path. Two runs on one runtime, at high and low confidence, take different
-roads — and which nodes ran is read off each run's state, not asserted from a diagram. The worker is
-[examples/getting-started/tag.ts](examples/getting-started/tag.ts), five lines that tag what they see.
+A **decision** is an edge with a `when` naming a predicate: code you supply, under a name the
+registry declares. A **fan-out** opens branches that run concurrently; a **join** waits for them and
+hands the next node every branch's result, keyed by branch path. Below, two runs at high and low
+confidence take different roads. The worker is
+[examples/getting-started/tag.ts](examples/getting-started/tag.ts).
 
 ```ts
 import { createRuntime } from 'fyflow-orchestration/scheduler';
 import { type FlowDefinition, type Predicate, type RegistryDocument, type RunState } from 'fyflow-orchestration';
 
-// THE REGISTRY — one worker, one type, and the NAME of a predicate the flow may use. A predicate is
-// code (ADR 0002), supplied beside the document under that name; a flow using a name nobody supplied
-// is refused at construction, not at the decision. The worker tags its input with the node's label.
+// THE REGISTRY — one worker, one type, and the NAME of a predicate the flow may use. The predicate
+// itself is code, passed to `createRuntime` under that name. The worker tags its input with a label.
 const registry: RegistryDocument = {
   workers: { tag: { url: '../../examples/getting-started/tag.ts' } },   // in your program: './tag.ts'
   nodeTypes: { Tag: { worker: 'tag', inline: true } },
   predicates: ['isLowConfidence'],
 };
-// A predicate reads the EFFECTIVE outcome, and `result` is `null` for an optional task that
-// failed (`OR-R9`) — so it is guarded, as section 4's is. Unreachable here, because `ingest` is
-// not optional; guarded anyway, because this is the form a consumer copies.
+// `result` is `null` when an optional task failed, so a predicate guards it.
 const isLowConfidence: Predicate = (ctx) => {
   const r = ctx.outcome.result as { confidence?: number } | null;
   return typeof r?.confidence === 'number' && r.confidence < 0.5;
 };
 
-// THE FLOW — a DECISION, a FAN-OUT, a JOIN. `ingest`'s first edge is taken when its predicate says so,
-// otherwise the last, unconditional one. `split` opens two branches that run concurrently; `collect`
-// waits for both and hands `publish` every branch's result keyed by branch path (JP-R12). `review` is
-// the road the run takes when confidence is low — it never reaches the join, and the run completes.
+// THE FLOW — a DECISION, a FAN-OUT, a JOIN. `ingest` goes to `review` when the predicate holds,
+// otherwise to `split`. `split` runs two branches concurrently; `collect` waits for both and hands
+// `publish` every branch's result, keyed by branch path.
 const flow: FlowDefinition = {
   flowId: 'decide', version: 1, entry: 'ingest',
   nodes: [
@@ -154,44 +132,42 @@ const flow: FlowDefinition = {
   ],
 };
 
-// ONE runtime, TWO runs. Pools belong to the runtime and are shared across runs; each run is its own
-// `Driver` with its own fold, and the join policy defaults to `live-token` (ADR 0003).
+// ONE runtime, TWO runs. Pools belong to the runtime and are shared; each run has its own state.
 const runtime = createRuntime(registry, { predicates: { isLowConfidence } }, { base: import.meta.url });
-// Each run has its OWN `done`, so two runs are two promises — `start` is called twice and the whole
-// of the wait is `await`. Nothing here folds anything: the runtime keeps the fold (ADR 0058).
 const run = (payload: unknown) => runtime.start(flow, payload).done;
 const high = await run({ confidence: 0.9 });
 const low = await run({ confidence: 0.2 });
 await runtime.shutdown();
 
-// WHICH nodes ran, read off each run's own state — the decision took a different edge each time.
+// Which nodes ran: the decision took a different edge each time.
 const ran = (s: RunState) => Object.values(s.nodes).map((n) => n.nodeId).sort();
 const resultOf = (s: RunState, id: string) => Object.values(s.nodes).find((n) => n.nodeId === id)?.result;
 console.log(ran(high));                    // [ 'classify', 'enrich', 'ingest', 'publish' ]
 console.log(ran(low));                     // [ 'ingest', 'review' ]
 console.log(resultOf(high, 'publish'));    // { '0': { confidence: 0.9, ingest: true, enrich: true }, '1': { …, classify: true }, publish: true }
 
+// Verified by the docs gate:
 if (high.outcome !== 'completed' || low.outcome !== 'completed') throw new Error(`outcomes ${high.outcome} / ${low.outcome}`);
 if (ran(high).join() !== 'classify,enrich,ingest,publish') throw new Error(`high ran ${ran(high).join()}`);
 if (ran(low).join() !== 'ingest,review') throw new Error(`low ran ${ran(low).join()}`);
-// `publish` received BOTH branches' results, keyed by branch path (JP-R12), and each branch's value
-// carries the whole road it came by. What this string pins is the branch-to-KEY MAPPING: the ASCENDING
-// order that rule also fixes is not observable at depth 1, because "0" and "1" are integer-like keys
-// and the language orders those ascending whatever the driver did. A nested region would show it.
 const want = '{"0":{"confidence":0.9,"ingest":true,"enrich":true},"1":{"confidence":0.9,"ingest":true,"classify":true},"publish":true}';
 if (JSON.stringify(resultOf(high, 'publish')) !== want) throw new Error(`publish received ${JSON.stringify(resultOf(high, 'publish'))}`);
 ```
 
 ## Checking a flow before it runs
 
-**A schema is for the AUTHOR, and this is what it buys.** Every worker declares what it expects and
-what it returns; `checkWiring` gives every arc of a flow one of five verdicts from those declarations
-BEFORE anything runs, `pathsOf` says what there is to reach for, and `tryTransform` runs a candidate
-JSONata expression on the producer's own declared example and holds the result to the consumer's
-schema. Below: a flow with a real miswire, the transform that repairs it found in two tries, a
-TypeScript signature generated from the same schemas, and then the run. The two workers are
+Every worker declares JSON schemas for what it expects and what it returns, so a flow can be checked
+before anything runs:
+
+- `checkWiring` gives every arc of a flow a verdict from those declarations.
+- `pathsOf` lists what a producer's output offers.
+- `tryTransform` runs a candidate JSONata expression on the producer's declared example and checks
+  the result against the consumer's schema.
+- `signatureOf` generates the TypeScript signature a worker is written against.
+
+Below, a flow with a miswire is repaired in two tries, then run. The two workers are
 [examples/getting-started/reader.ts](examples/getting-started/reader.ts) and
-[summarise.ts](examples/getting-started/summarise.ts), whose schemas oppose on purpose.
+[summarise.ts](examples/getting-started/summarise.ts), whose schemas deliberately do not match.
 
 ```ts
 import { checkWiring, describePaths, pathsOf, signatureOf, tryTransform, type FlowDefinition } from 'fyflow-orchestration';
@@ -203,8 +179,7 @@ import { SESSION_BASE, SESSION_REGISTRY, SESSION_WORKERS } from '../../examples/
 // and the languages a transform may be written in. `createRuntime` takes the whole document below.
 const registry = { workers: SESSION_WORKERS, nodeTypes: SESSION_REGISTRY.nodeTypes, languages: { jsonata } };
 
-// THE FLOW, as an author first writes it: read a batch, summarise it. It is a VALID document —
-// `validateDefinition` has nothing to say about it, because nothing here is about the grammar.
+// THE FLOW, as an author first writes it: read a batch, summarise it. Its grammar is valid.
 const flow: FlowDefinition = {
   flowId: 'session', version: 1, entry: 'read',
   nodes: [
@@ -229,40 +204,39 @@ console.log(describePaths(pathsOf(SESSION_WORKERS.reader.outputSchema)));
 // ? $.lines[]                    string
 //   $.source                     string
 
-// STEP 3 — a first candidate. It is run on the producer's declared EXAMPLE, and the result is held
-// to the consumer's input schema, so the answer is about a value rather than about two declarations.
+// STEP 3 — a first candidate, run on the producer's declared example and checked against the
+// consumer's input schema.
 const attempt = await tryTransform(registry, flow, 'sum', { lang: 'jsonata', expr: '{ "text": lines }' });
 console.log(`${attempt[0]!.kind}: ${attempt[0]!.why}`);
 // transform-fails: the transform's result on the example fails the consumer's inputSchema — $.text expected string, got array
 if (attempt[0]!.kind !== 'transform-fails') throw new Error('an array is not a string, and the loop says which');
 
-// STEP 4 — ONE expression edit, and the same loop. `transform-passes` is never `fits`: it is a fact
-// about the sample the verdict names, which is why `sample.source` is on it (a declared example
-// here; where a schema declares none, a generated WITNESS, and the verdict says so).
+// STEP 4 — one edit, and the same loop. `transform-passes` is about the sample it names, which is
+// the declared example here, or a generated one where a schema declares none.
 const good = { lang: 'jsonata', expr: '{ "text": $join(lines, " ") }' };
 const passed = await tryTransform(registry, flow, 'sum', good);
 console.log(`${passed[0]!.kind} on the ${passed[0]!.sample!.source}: ${JSON.stringify(passed[0]!.sample!.result)}`);
 if (passed[0]!.kind !== 'transform-passes') throw new Error(`expected the sample to pass, got ${passed[0]!.kind}`);
 
-// STEP 5 — the signature the worker is written against, generated from the same three schemas.
-// `examples/getting-started/session-types.ts` is this text, checked in, and `summarise.ts` imports
-// it. A type holds the SHAPE: `words` is `number` here and its schema says `minimum: 0`.
+// STEP 5 — the signature the worker is written against, generated from the same schemas.
+// `examples/getting-started/session-types.ts` is this text, and `summarise.ts` imports it.
 const signature = signatureOf('Summarise', SESSION_WORKERS.summarise);
 console.log(signature.split('\n').filter((l) => l.startsWith('export')).join('\n'));
 if (!signature.includes('"text": string;')) throw new Error('the required key is required in the type too');
 
-// STEP 6 — and only now the runtime. Pools are built ONCE, so a flow whose transform was decided
-// during the session has to reach `createRuntime` as `flows`, or `start` throws `LanguageNotBuilt`
-// naming the type. Check, then build.
+// STEP 6 — only now the runtime. Check the wired flow against it, then start it; `start` refuses
+// from that same check.
 const wired: FlowDefinition = { ...flow, nodes: flow.nodes.map((n) => (n.id === 'sum' ? { ...n, transform: good } : n)) };
-const runtime = createRuntime(SESSION_REGISTRY, { languages: { jsonata } }, { base: SESSION_BASE, flows: [wired] });
+const runtime = createRuntime(SESSION_REGISTRY, { languages: { jsonata } }, { base: SESSION_BASE });
+const refused = runtime.check(wired);
+if (refused.length !== 0) throw new Error(`the runtime refuses the wired flow: ${JSON.stringify(refused)}`);
+const unknownLang: FlowDefinition = { ...flow, nodes: flow.nodes.map((n) => (n.id === 'sum' ? { ...n, transform: { lang: 'nope', expr: '$' } } : n)) };
+if (!runtime.check(unknownLang).some((f) => f.rule === 'FD-V35' && f.at === 'sum')) throw new Error('a language the registry does not name is FD-V35, at check');
 const state = await runtime.start(wired, { source: 'inbox' }).done;
 await runtime.shutdown();
 console.log(state.outcome, Object.values(state.nodes).find((n) => n.nodeId === 'sum')?.result);  // completed { words: 3 }
 
-// The expected values are the DOCUMENT's, not literals beside the run: the consumer's own declared
-// output example, and the producer's declared line count. A registry whose examples stop describing
-// its workers is red here and in `test/definition/assist.test.ts`, which runs the same chain.
+// Verified by the docs gate:
 if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
 const got = Object.values(state.nodes).find((n) => n.nodeId === 'sum')?.result;
 if (JSON.stringify(got) !== JSON.stringify(SESSION_WORKERS.summarise.outputSchema.examples[0])) {
@@ -277,21 +251,34 @@ if (after.counts.no !== 0) throw new Error('and the flow that ran has no unwired
 
 ## More examples
 
-The three above are the shape of it. [AGENTS.md](AGENTS.md) is the full reference — every section's
-snippet is extracted and executed on every `npm run gate`, so none of it can drift from the code:
+[AGENTS.md](AGENTS.md) is the full reference, and every snippet in it is tested:
 
 | | |
 |---|---|
-| Holding a **result** to the schema its worker declares | [AGENTS.md](AGENTS.md) section 16 |
+| A task **cancelled** by its id — the run ends `cancelled` and refuses a resume | [AGENTS.md](AGENTS.md) section 6 |
+| **Cancelling a run** — `driver.cancel`, the run `cancelling` until its last task has stopped | section 6 |
+| **Pausing** a run and unpausing it — every dispatch held, then handed on in order | section 8 |
+| A node that **waits for a person** by design — an `ask`, which keeps what reached it and holds the answer to a schema | section 7 |
+| A task's **messages and progress**, each with its run, node and attempt as fields — `onTaskMessage`, `onTaskProgress` | section 8 |
+| A **worker's own messages**, outside any task — `worker.message` on its pool, reached through `createScheduler` | section 8 |
+| Reaching what reached the **run** from any node or edge — `needs: { invoice: "$payload" }` | section 3 |
+| Reaching the item that opened a **map instance** from any node in it — `needs: { invoice: "$item" }` | section 3 |
+| **Stopping** a run to resume it on another runtime — `handle.stop` | section 18 |
+| Writing a run's **header first**, before its first event — `onHeader` | section 18 |
+| A resumed task **keeps its retry budget** — the attempts it already made, handed on as `attemptsSpent` | section 18 |
+| **Checking** a flow against a running runtime before starting it — `runtime.check` | section 12 |
+| **Trying** a transform on a value you give, the entry node included | section 15 |
+| Holding a **result** to the schema its worker declares | section 16 |
 | Fanning out **per item** — a region inside a `map` region | section 17 |
 | **Saving** a run and **resuming** it from the stored events | section 18 |
 | What a node may **remember** across branches — a `gate` | section 19 |
+| Routing with **no code** — the built-in `if` and `switch` gates | section 19 |
 | Wiring a gate **before it runs** | section 20 |
 
-Runnable programs live in [examples/](examples/) — including
-[examples/flows/triage.flow.json](examples/flows/triage.flow.json) with
-[examples/run-flow.ts](examples/run-flow.ts), which puts a custom resource group, a node-type class,
-a named pool instance, a decision, a fan-out, a join and a declared transform on real workers.
+Runnable programs are in [examples/](examples/). For a larger one, see
+[examples/run-flow.ts](examples/run-flow.ts) running
+[examples/flows/triage.flow.json](examples/flows/triage.flow.json): a custom resource group, a
+node-type class, a named pool, a decision, a fan-out, a join and a transform.
 
 ## Running the tests
 
@@ -299,22 +286,19 @@ a named pool instance, a decision, a fan-out, a join and a declared transform on
 npm install
 npm test          # the unit and routing suites
 npm run test:l6   # real runs on a real scheduler
-npm run gate      # everything above, in order
+npm run gate      # lint, typecheck, checks, all suites, Node and Deno runs
 ```
 
-`npm run gate` is the full ladder — lint, typecheck, the static checks, the suites in three runtimes,
-and the real-scheduler runs in Node and Deno.
-
 ## Documentation
 
 | | |
 |---|---|
-| [AGENTS.md](AGENTS.md) | the complete reference: every snippet in it is typechecked by the gate, each one tagged `run` is executed, and a snippet tagged `check` is typechecked only |
+| [AGENTS.md](AGENTS.md) | the complete API reference, with tested snippets |
 | [doc/concept.md](doc/concept.md) | what this is and why it is shaped this way |
-| [spec/](spec/) | the normative rules — the flow grammar, run state, join policies, task ids |
-| [doc/README.md](doc/README.md) | the development index: plans, work items, decisions, open questions |
-| [CLAUDE.md](CLAUDE.md) | guidance for an agent working ON this library |
+| [spec/](spec/) | the normative rules: flow grammar, run state, join policies, task ids |
+| [doc/README.md](doc/README.md) | for contributors: plans, work items, decisions, open questions |
+| [CLAUDE.md](CLAUDE.md) | for an agent working on this library |
 
 ## Licence
 
-MIT — the `LICENSE` file at the repository root, which both registries carry beside the code.
+MIT. See `LICENSE`.