fyflow-orchestration 0.0.0-stage → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +3757 -0
- package/LICENSE +21 -0
- package/README.md +319 -2
- package/dist/browser/chunks/chunk-YVOP3DIN.js +4456 -0
- package/dist/browser/index.js +435 -0
- package/dist/browser/jsonata/index.js +75 -0
- package/dist/browser/scheduler/index.js +666 -0
- package/dist/node/chunks/chunk-YVOP3DIN.js +4456 -0
- package/dist/node/index.js +435 -0
- package/dist/node/jsonata/index.js +75 -0
- package/dist/node/scheduler/index.js +666 -0
- package/dist/types/index.d.ts +17 -0
- package/dist/types/jsonata/index.d.ts +2 -0
- package/dist/types/scheduler/index.d.ts +1 -0
- package/dist/types/src/definition/assist.d.ts +221 -0
- package/dist/types/src/definition/regions.d.ts +359 -0
- package/dist/types/src/definition/registry.d.ts +185 -0
- package/dist/types/src/definition/schema.d.ts +91 -0
- package/dist/types/src/definition/types.d.ts +748 -0
- package/dist/types/src/definition/validate.d.ts +299 -0
- package/dist/types/src/definition/wiring.d.ts +186 -0
- package/dist/types/src/definition/witness.d.ts +39 -0
- package/dist/types/src/driver/driver.d.ts +1364 -0
- package/dist/types/src/driver/events.d.ts +87 -0
- package/dist/types/src/driver/ids.d.ts +49 -0
- package/dist/types/src/driver/listener.d.ts +41 -0
- package/dist/types/src/driver/log.d.ts +59 -0
- package/dist/types/src/driver/policy.d.ts +65 -0
- package/dist/types/src/driver/routing.d.ts +85 -0
- package/dist/types/src/fold/policies.d.ts +169 -0
- package/dist/types/src/fold/run-state.d.ts +307 -0
- package/dist/types/src/jsonata/index.d.ts +40 -0
- package/dist/types/src/scheduler/bootstrap.d.ts +97 -0
- package/dist/types/src/scheduler/envelope-worker.d.ts +199 -0
- package/dist/types/src/scheduler/index.d.ts +13 -0
- package/dist/types/src/scheduler/runtime.d.ts +321 -0
- package/package.json +137 -4
package/AGENTS.md
ADDED
|
@@ -0,0 +1,3757 @@
|
|
|
1
|
+
# fyflow-orchestration — Reference
|
|
2
|
+
|
|
3
|
+
Flow orchestration over [`fyflow-scheduler`](https://github.com/fyflow/fyflow-scheduler): sequence,
|
|
4
|
+
decision routing, fan-out and join.
|
|
5
|
+
|
|
6
|
+
> **Every ```ts snippet in this file is extracted and typechecked against the real `index.ts` by
|
|
7
|
+
> `npm run gate`, and most of them are executed.** `scripts/check-docs.mjs` pulls them out, rewrites
|
|
8
|
+
> the `fyflow-orchestration` import to this repository's own, and compiles them together. **A stale
|
|
9
|
+
> snippet is a failing gate rather than a lie** — which is why they are worth reading.
|
|
10
|
+
>
|
|
11
|
+
> `run` snippets execute and must assert something; `check` snippets are typechecked only, because
|
|
12
|
+
> they need a live scheduler or exist to show a type. **The prose between them is not checked**, so
|
|
13
|
+
> where a claim matters it is pushed into a snippet's assertion rather than left in a sentence.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## 1. What this library is, and what it is not
|
|
18
|
+
|
|
19
|
+
`fyflow-scheduler` runs tasks. **It has no dependency graph** — no `dependsOn`, no `parents`, no
|
|
20
|
+
ordering — and its own reference lists them under *"Things that do NOT exist"*.
|
|
21
|
+
|
|
22
|
+
**This library decides which task runs next, and why.** It owns four things the scheduler does not
|
|
23
|
+
have: **sequence**, **decision routing**, **fan-out** and **join**.
|
|
24
|
+
|
|
25
|
+
**And it tells you whether the flow you wrote FITS the workers you named, before you run it.** Where
|
|
26
|
+
a worker declares what it expects and what it returns — it need not, and `unchecked` is the verdict
|
|
27
|
+
when it does not — `checkWiring` gives every arc of a flow one of five verdicts from those
|
|
28
|
+
declarations, `tryTransform` runs a candidate expression on a producer's own
|
|
29
|
+
declared example and holds the result to the consumer's schema, and `signatureOf` turns the same
|
|
30
|
+
schemas into the TypeScript a worker's author writes against. **Section 15 is a session that does all
|
|
31
|
+
three on a real miswire and then runs the flow it repaired** — it is executed by `npm run gate` like
|
|
32
|
+
everything else here.
|
|
33
|
+
|
|
34
|
+
| | belongs to |
|
|
35
|
+
|---|---|
|
|
36
|
+
| worker pools, threads, retry, resource limits, task execution | `fyflow-scheduler` |
|
|
37
|
+
| what runs after what, and on what condition | **here** |
|
|
38
|
+
|
|
39
|
+
**This package depends on the scheduler, and says so where your installer reads it.** Since
|
|
40
|
+
[ADR 0048](doc/decisions/0048-the-scheduler-is-a-declared-dependency-reached-by-one-bare-specifier.md)
|
|
41
|
+
`fyflow-scheduler` is a `peerDependencies` entry at **exactly** the version section 12 pins — npm 7
|
|
42
|
+
and later install it for you — and the JSR package maps it to `@fyflow/scheduler` at the same
|
|
43
|
+
version. The driver is written against that version's event shapes, and the range is exact because
|
|
44
|
+
there is no upstream changelog to read a wider one from. **You still pass your own scheduler in**:
|
|
45
|
+
`SchedulerLike` is a two-method structural interface, a real `FyflowScheduler` satisfies it as it is,
|
|
46
|
+
and the one thing the library builds itself is a `FyflowTask` when you omit `createTask`. *(This
|
|
47
|
+
paragraph read "does not depend on the scheduler … install whichever version you like" from W9 to
|
|
48
|
+
W57, while section 12 said to pin it — two sentences a consumer reads first, contradicting each
|
|
49
|
+
other twenty-one sections apart.)*
|
|
50
|
+
|
|
51
|
+
```ts check
|
|
52
|
+
import type { SchedulerLike } from 'fyflow-orchestration';
|
|
53
|
+
|
|
54
|
+
// The entire surface this library needs from a scheduler. The listener takes `object` — what an
|
|
55
|
+
// `EventTarget` hands a listener — and the driver reads `detail` off it (section 12, Q71).
|
|
56
|
+
const shape: SchedulerLike = {
|
|
57
|
+
addTask(_task: unknown) { return undefined; },
|
|
58
|
+
addEventListener(_type: string, _listener: (e: object) => void) {},
|
|
59
|
+
};
|
|
60
|
+
void shape;
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## 2. Getting started — a flow and its runtime from two documents
|
|
66
|
+
|
|
67
|
+
**Two documents, and one file of your own.** A REGISTRY says what exists — a worker module, and a
|
|
68
|
+
node type that runs on it. A FLOW says what happens — nodes naming those types, and the edges between
|
|
69
|
+
them. `createRuntime` builds every pool, group and scheduler the registry names and hands back
|
|
70
|
+
**`{ driver, done, header }`** per run — the driver already running, and `done` a promise that resolves with
|
|
71
|
+
the run's terminal state ([ADR 0058](doc/decisions/0058-runtime-start-returns-a-completion-handle-the-runtime-keeps.md)). **The only code you write is the worker**:
|
|
72
|
+
`examples/getting-started/worker.ts`, a class with a `run` method and nothing else — no base class,
|
|
73
|
+
no import. Everything below is executed by the gate on every `npm run gate`, and the same snippet is
|
|
74
|
+
in `README.md`, held byte-equal to this one by `check-docs`'s `X8`
|
|
75
|
+
([W69](doc/work-items/w69-a-getting-started-example-a-consumer-can-copy.md)).
|
|
76
|
+
|
|
77
|
+
```ts run
|
|
78
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
79
|
+
import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
80
|
+
|
|
81
|
+
// THE REGISTRY — what exists. One worker module, and one node type that runs on it. This is data:
|
|
82
|
+
// in your program it is a JSON file you import. The worker url is relative, and `base` below says
|
|
83
|
+
// to what; the file itself is `examples/getting-started/worker.ts` — a class with a `run` method.
|
|
84
|
+
const registry: RegistryDocument = {
|
|
85
|
+
workers: {
|
|
86
|
+
greeter: {
|
|
87
|
+
url: '../../examples/getting-started/worker.ts', // in your program: './worker.ts'
|
|
88
|
+
settingsSchema: { type: 'object', required: ['say'], properties: { say: { type: 'string' } } },
|
|
89
|
+
},
|
|
90
|
+
},
|
|
91
|
+
nodeTypes: { Greeter: { worker: 'greeter', inline: true } },
|
|
92
|
+
};
|
|
93
|
+
|
|
94
|
+
// THE FLOW — what happens. Two nodes of that type, each with its own `settings`, then an end. A flow
|
|
95
|
+
// names registry KEYS and nothing else — no file, no url, no class — so it is portable across
|
|
96
|
+
// runtimes that agree on the keys. The end node's `outcome` is a LABEL of yours that the driver does
|
|
97
|
+
// not interpret; the run's own outcome is the fold's — `state.outcome` below.
|
|
98
|
+
const flow: FlowDefinition = {
|
|
99
|
+
flowId: 'hello', version: 1, entry: 'greet',
|
|
100
|
+
nodes: [
|
|
101
|
+
{ id: 'greet', kind: 'task', type: 'Greeter', settings: { say: 'world' }, edges: [{ to: 'shout' }] },
|
|
102
|
+
{ id: 'shout', kind: 'task', type: 'Greeter', settings: { say: '!' }, edges: [{ to: 'done' }] },
|
|
103
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
104
|
+
],
|
|
105
|
+
};
|
|
106
|
+
|
|
107
|
+
// ONE call builds the pool, the scheduler and everything between, and refuses every defect in the
|
|
108
|
+
// REGISTRY at construction, by name. It never sees the flow: a defect THERE is refused where the
|
|
109
|
+
// flow arrives, at `start`, which is what builds the `Driver` — so a `settings` value the worker's
|
|
110
|
+
// schema rejects names its node from the call three lines below, not from this one. `base` is what
|
|
111
|
+
// relative worker urls resolve against — `import.meta.url` when the registry sits beside this file.
|
|
112
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
113
|
+
|
|
114
|
+
// `start` hands back `{ driver, done }` — the driver already running, and `done` a promise that
|
|
115
|
+
// resolves with the run's terminal `RunState` (ADR 0058). **It never rejects**: `failed`, `stuck`
|
|
116
|
+
// and `stopped` are terminal OUTCOMES, so a run that goes wrong arrives as `state.outcome` and not
|
|
117
|
+
// as a `catch`. This snippet wants only the end, so it names only `done`. *A run with a person in
|
|
118
|
+
// it does not end by itself — section 7 is where that is taught, and section 8 is what `done` waits
|
|
119
|
+
// for.*
|
|
120
|
+
const { done } = runtime.start(flow, { text: 'hello' });
|
|
121
|
+
const state = await done;
|
|
122
|
+
await runtime.shutdown();
|
|
123
|
+
|
|
124
|
+
// What the run did, read off its state: `greet` turned `hello` into `hello world`, and `shout` was
|
|
125
|
+
// handed THAT as its input — the previous node's result is the next node's `input`.
|
|
126
|
+
const shout = Object.values(state.nodes).find((n) => n.nodeId === 'shout');
|
|
127
|
+
console.log(state.outcome); // 'completed'
|
|
128
|
+
console.log(shout?.result); // { text: 'hello world !' }
|
|
129
|
+
|
|
130
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
131
|
+
if (JSON.stringify(shout?.result) !== '{"text":"hello world !"}') throw new Error(`shout returned ${JSON.stringify(shout?.result)}`);
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
**What to read next.** Everything this snippet leaves out is `examples/flows/triage.flow.json` with
|
|
135
|
+
`examples/run-flow.ts` — a custom resource group, a node-type CLASS with its own derivation, a node
|
|
136
|
+
on a NAMED pool instance, a decision edge, a fan-out and its join, and a declared transform — which is
|
|
137
|
+
[P18](doc/plans/p18-a-flow-and-its-runtime-from-documents-alone.md)'s exit artifact and runs on real
|
|
138
|
+
workers in both L6 runtimes. Section 3 is the definition grammar; section 12 is the runtime in full,
|
|
139
|
+
including the driver with **no scheduler at all**; section 13 is node types; **section 15 is what
|
|
140
|
+
their schemas buy you before a run**.
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
## 3. The flow definition
|
|
145
|
+
|
|
146
|
+
**Six** node kinds. A definition is JSON: no functions, so it can be stored, versioned and sent over
|
|
147
|
+
a wire.
|
|
148
|
+
|
|
149
|
+
> **`gate` is the sixth, and since [W93](doc/work-items/w93-what-a-gate-does-in-a-run.md) a `Driver` RUNS
|
|
150
|
+
> one** — `spec/flow-definition.md` v0.23-v0.24,
|
|
151
|
+
> [ADR 0066](doc/decisions/0066-a-gate-is-a-decision-with-memory-that-names-a-registered-type-and-wires-every-route-exactly-once.md)
|
|
152
|
+
> and [ADR 0067](doc/decisions/0067-a-gate-runs-in-the-driver-its-memory-is-the-drivers-and-its-decision-is-recorded.md),
|
|
153
|
+
> [W92](doc/work-items/w92-what-a-gate-is.md). A gate is a decision WITH MEMORY: it routes the branch that
|
|
154
|
+
> reaches it by what the run's other branches have already shown it — `distinct`, the one built-in type,
|
|
155
|
+
> sends the FIRST branch per key down its default route `first` and every later one down `duplicate` — and it
|
|
156
|
+
> hands on exactly what reached it. The driver evaluates it where a branch reaches it — the gate's memory is
|
|
157
|
+
> ONE per gate per RUN, the driver's, rebuilt by replay on a resume — records each decision as a `gated` fold
|
|
158
|
+
> input (`RunState.gated`), and stops the run with an `UndeclaredRoute` if a type returns a route it does not
|
|
159
|
+
> declare (`FD-R5`). *(Until W93 `new Driver` refused a flow holding one, at construction.)* Every route its
|
|
160
|
+
> type declares must be wired by exactly one edge — a drop is an edge to an `end` — and an edge with no
|
|
161
|
+
> `route` carries the type's DEFAULT route, never every route.
|
|
162
|
+
|
|
163
|
+
> **`map` is the fifth, and since [W33](doc/work-items/w33-a-map-at-run-time-arity.md) a `Driver`
|
|
164
|
+
> RUNS one.** It is a valid definition since `spec/flow-definition.md` v0.8 —
|
|
165
|
+
> [ADR 0036](doc/decisions/0036-a-map-is-a-second-node-kind-and-a-branch-path-inside-one-is-a-property-of-the-dispatch.md),
|
|
166
|
+
> answering [Q12](doc/questions/q12.md): a `fanout` declares N branches known at authoring time, a
|
|
167
|
+
> `map` declares **one** and a run opens N instances of it, N coming from the value the map node
|
|
168
|
+
> receives. `over` is a **key name** on that value and must resolve to an array, or the run stops
|
|
169
|
+
> with an `UnresolvedMapArity` naming the node and the key; an empty array is a lawful map of zero
|
|
170
|
+
> items and makes the run `stuck`, because the join can never fire.
|
|
171
|
+
>
|
|
172
|
+
> **What each instance receives, and what the join hands on.** Each instance is dispatched with
|
|
173
|
+
> **its own item** — `over`'s array at that instance's index, as its own copy, so no two instances
|
|
174
|
+
> share an object — and the join carries **every item's result keyed by that item's branch path**. A
|
|
175
|
+
> map of three items at the top level dispatches three tasks and its successor receives
|
|
176
|
+
> `{ "0": …, "1": …, "2": … }`; **a map nested in a fan-out branch keys its collection with the whole
|
|
177
|
+
> path**, so a map inside branch 1 gives `{ "1.0": …, "1.1": … }`. *(This said "keyed by its item
|
|
178
|
+
> index", which is the depth-1 case stated as if it were general — W34's conformity reader.)*
|
|
179
|
+
> [ADR 0037](doc/decisions/0037-what-a-branch-receives-is-a-property-of-the-fan-out-and-what-a-join-carries-is-a-property-of-the-region.md),
|
|
180
|
+
> and see *What a join carries downstream* in section 5 for what happens when an item's branch does
|
|
181
|
+
> not arrive.
|
|
182
|
+
>
|
|
183
|
+
> *(This paragraph read **"every instance receives the whole value the map node received rather than
|
|
184
|
+
> its own item, and a satisfied join still carries `null`"** until W34, which is where
|
|
185
|
+
> [Q11](doc/questions/q11.md) and [Q36](doc/questions/q36.md) were deliberately held so the rule
|
|
186
|
+
> would be written from cases rather than from a mechanism.)*
|
|
187
|
+
>
|
|
188
|
+
> **This paragraph is the minimum that is not false and is not the feature write-up.** It read
|
|
189
|
+
> *"NOTHING DISPATCHES ONE YET"* until W33, which is [Q157](doc/questions/q157.md)'s second
|
|
190
|
+
> occurrence: no [P10](doc/plans/p10-is-fan-out-one-feature-or-two.md) item owns this file, W35 is
|
|
191
|
+
> the natural owner, and W33 took it as a declared late exception to keep the reference true.
|
|
192
|
+
|
|
193
|
+
```ts run
|
|
194
|
+
import { validateDefinition, type FlowDefinition } from 'fyflow-orchestration';
|
|
195
|
+
|
|
196
|
+
const shapes: FlowDefinition = {
|
|
197
|
+
flowId: 'shapes',
|
|
198
|
+
version: 1,
|
|
199
|
+
entry: 'score',
|
|
200
|
+
nodes: [
|
|
201
|
+
// A task runs a worker. `optional` decides what a failure MEANS — section 6.
|
|
202
|
+
// `retryPolicy` and `workerGroups` say what the node wants of its OWN task — below.
|
|
203
|
+
{ id: 'score', kind: 'task', type: 'ScoreWorker', optional: false,
|
|
204
|
+
retryPolicy: { maxRetries: 1, backoffMs: 250 }, // exactly the scheduler's shape
|
|
205
|
+
workerGroups: ['gpu'], // must be a group the registry declares
|
|
206
|
+
edges: [
|
|
207
|
+
{ on: ['failed'], to: 'quarantine' }, // eligible only when the task failed
|
|
208
|
+
{ when: { predicate: 'isLowConfidence' }, to: 'review' }, // guarded by a registered predicate
|
|
209
|
+
{ to: 'publish' }, // the default: no `on`, no `when`
|
|
210
|
+
] },
|
|
211
|
+
{ id: 'review', kind: 'task', type: 'ReviewWorker', edges: [{ to: 'publish' }] },
|
|
212
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
213
|
+
{ id: 'quarantine', kind: 'end', outcome: 'quarantined' },
|
|
214
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
215
|
+
],
|
|
216
|
+
};
|
|
217
|
+
|
|
218
|
+
// Not merely well-typed — VALID, and checked here rather than asserted. Note the
|
|
219
|
+
// `on: ['failed']` edge is declared FIRST: "the default comes last" is scoped per
|
|
220
|
+
// terminal state, not to the array. See section 4.
|
|
221
|
+
const bad = validateDefinition(shapes, {
|
|
222
|
+
predicates: ['isLowConfidence'],
|
|
223
|
+
workers: { flow: { url: 'worker.js' } }, nodeTypes: { ScoreWorker: { worker: 'flow' }, ReviewWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
224
|
+
groups: ['gpu'], // the third set — pass Object.keys(scheduler.groups)
|
|
225
|
+
});
|
|
226
|
+
if (bad.length) throw new Error(`section 3's own example is invalid: ${bad.map((f) => f.rule).join()}`);
|
|
227
|
+
|
|
228
|
+
// And `groups` is what makes `workerGroups` checkable at all: leave it out and the SAME
|
|
229
|
+
// definition is refused, because a registry that lists no groups declares none. The scheduler
|
|
230
|
+
// would have run the task with the unknown group silently ignored — no throw, no
|
|
231
|
+
// `resource.blocked` — so construction is the only place a typo in a group id is caught.
|
|
232
|
+
const noGroups = validateDefinition(shapes, {
|
|
233
|
+
predicates: ['isLowConfidence'], workers: { flow: { url: 'worker.js' } }, nodeTypes: { ScoreWorker: { worker: 'flow' }, ReviewWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
234
|
+
});
|
|
235
|
+
if (!noGroups.some((f) => f.rule === 'FD-V31' && f.at === 'score')) {
|
|
236
|
+
throw new Error(`expected FD-V31 at score, got ${noGroups.map((f) => `${f.rule}@${f.at}`).join() || 'nothing'}`);
|
|
237
|
+
}
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
| kind | | refused if |
|
|
241
|
+
|---|---|---|
|
|
242
|
+
| `task` | runs a worker — of its node **`type`**, REQUIRED since v0.19, on the type's default pool or a named **`pool`** instance of it, with per-node **`settings`** and optionally a **`transform`** applied to its input before the worker sees it (section 13). Has `edges`, and optionally **`needs`**, **`retryPolicy`** and **`workerGroups`** — what it says about its own task, below | it has no `edges`, or **no edge of it is eligible for `done`** (`FD-V7`); it has no `type` (`FD-V23`) — `workerType` is retired and is an undeclared key like any other (`FD-V17`); its `retryPolicy` is not exactly `{ maxRetries, backoffMs }` — an integer >= 0 and a finite number >= 0, no other key (`FD-V30`); its `workerGroups` is not a non-empty array of distinct ids the registry's `groups` declares (`FD-V31`); its `type` is not a registered node type whose worker the registry declares (`FD-V32`); its `settings` is not a plain object the type's worker `settingsSchema` accepts (`FD-V33`); its `transform` names a language the registry does not (`FD-V35`), an expression that language's compiler refuses (`FD-V36`), or is not `{ lang, expr }` (`FD-V37`); its `pool` is not a registered instance of its type (`FD-V38`) |
|
|
243
|
+
| `fanout` | `branches: string[]` — the node each branch starts at. Dispatches nothing itself | fewer than two branches, a repeated entry, any `edges` at all, or a branch entry that is not a `task` (`FD-V9`) |
|
|
244
|
+
| `map` | `over: '<key>'` and `branch: '<node id>'` — **one** branch, opened once per item at run time. Dispatches nothing itself; the RUN opens N instances of its one branch. Optionally **`needs`**, which is what `over` is then a key on | any `edges` at all, a `branch` that is not a `task`, or an `over` outside `^[A-Za-z0-9_-]{1,64}$` (`FD-V25`); a missing `over` or `branch` (`FD-V23`) |
|
|
245
|
+
| `join` | `of: '<fanout id>'`. Fires when its policy says so | it does not have **exactly one** edge, or that edge carries `when` or `on` (`FD-V7`) |
|
|
246
|
+
| `end` | terminal. `outcome` is **your label**, never interpreted | — |
|
|
247
|
+
| `gate` | `type: '<gate type>'`, optional `settings`, and `edges` of `{ to, route? }` — routes the branch that reaches it and hands on what reached it. `settings.key` is a dotted key path over that value (`series.id`; no index). **Not run by any driver yet** | its `type` is not a registered gate type (`FD-V39`); its `settings` are not a plain object its type's schema accepts, or its `key` is not a key path (`FD-V40`); a route is unwired, wired twice, or not its type's, or a gate edge carries `when`, `on` or `needs` (`FD-V41`) — and `route` on any other node's edge is `FD-V41` too |
|
|
248
|
+
|
|
249
|
+
**A `task` node must have an edge that can be taken when it succeeds.** A node whose only edge is
|
|
250
|
+
`{ when: { predicate: 'p' }, to: 'x' }` is refused at construction — it is the mistake the format most invites, and
|
|
251
|
+
`FD-V7` is what catches it.
|
|
252
|
+
|
|
253
|
+
**`end.outcome` is a string you choose.** The library never reads it. What the *run* is called is a
|
|
254
|
+
separate, closed vocabulary — section 8.
|
|
255
|
+
|
|
256
|
+
### `retryPolicy` and `workerGroups` — what a node says about its own task
|
|
257
|
+
|
|
258
|
+
`optional` was the one scheduler task field a node could set; since
|
|
259
|
+
`spec/flow-definition.md` v0.16 ([W58](doc/work-items/w58-what-a-node-says-about-its-own-task.md))
|
|
260
|
+
there are two more, and both are **passed through to the scheduler task exactly as `optional` is**.
|
|
261
|
+
They are declared keys rather than an opaque bag, so a typo is refused at construction instead of
|
|
262
|
+
reaching the scheduler, which would drop it in silence.
|
|
263
|
+
|
|
264
|
+
| key | | |
|
|
265
|
+
|---|---|---|
|
|
266
|
+
| `retryPolicy` | `{ maxRetries, backoffMs }` — **exactly** the scheduler's shape, an integer >= 0 and a finite number >= 0, no other key (`FD-V30`) | the scheduler retries a failed attempt **inside the same task id**, and the outcome record's `attempts` says how many retries were used. `{}` and a string count are refused because the scheduler would accept them and mean something else |
|
|
267
|
+
| `workerGroups` | a non-empty array of distinct group ids, every one in the registry's **`groups`** set (`FD-V31`) | the task acquires these **in addition to** its pool's `groups`. Pass `groups: Object.keys(scheduler.groups)` to the `Driver`, as you pass `nodeTypes` — **omitting it means the registry declares NO groups**, and any node naming one is refused, because a group id the scheduler does not know is silently ignored at run time |
|
|
268
|
+
|
|
269
|
+
**A driver may carry a `defaultRetryPolicy`**, applied to every task node that declares none — held
|
|
270
|
+
to `FD-V30` at construction like a node's own — and
|
|
271
|
+
**the node wins, whole** — a node with its own policy is dispatched with exactly that object, and the
|
|
272
|
+
default is consulted only when the node declares none. There is no `defaultWorkerGroups`: a pool's own `groups` already applies to
|
|
273
|
+
every task it runs, and the scheduler unions the two. A node that declares neither key is dispatched
|
|
274
|
+
with the exact object it was before, so nothing you have already written changes.
|
|
275
|
+
|
|
276
|
+
**Flow-level retry — re-dispatching a NODE under a new attempt, possibly routing elsewhere — is
|
|
277
|
+
still not built** (section 11). This is the scheduler's retry of one task, and it is the whole of
|
|
278
|
+
what `attempts` counts.
|
|
279
|
+
|
|
280
|
+
### `needs` — reaching an output the previous node did not produce
|
|
281
|
+
|
|
282
|
+
By default **a node receives the previous node's result**, deep-copied. That is still true of every
|
|
283
|
+
node that says nothing, and it is what the whole of this section above describes.
|
|
284
|
+
|
|
285
|
+
**`needs` is how a node asks for something else** —
|
|
286
|
+
[ADR 0040](doc/decisions/0040-what-a-node-can-reach-is-the-runs-own-outputs-and-a-node-id-names-them-relative-to-a-branch-path.md),
|
|
287
|
+
answering [Q11](doc/questions/q11.md). It is an optional object on a `task` or a `map`, mapping the
|
|
288
|
+
keys of the payload you want to **declared references**. *(Since v0.12 an **edge** may carry one too,
|
|
289
|
+
and it says what that edge's PREDICATE sees rather than what a worker receives — section 4.)* A node that declares one receives **exactly**
|
|
290
|
+
that object and nothing else — there is no wrapper and no reserved key.
|
|
291
|
+
|
|
292
|
+
| a reference is | and you get |
|
|
293
|
+
|---|---|
|
|
294
|
+
| `"."` | what the node would have received anyway — the previous node's result, and for a `map` instance **its own item** |
|
|
295
|
+
| `"someNode"` | that node's result, **on your own branch** |
|
|
296
|
+
| `["someNode"]` | that node's results as an **array**, in ascending branch-path order. Legal only when you are OUTSIDE a region that node is in — which is what makes it more than one |
|
|
297
|
+
| **a `?` on either of those** — `"someNode?"`, `["someNode?"]` | the same thing, except that **nothing to give leaves the key out** (or gives `[]` for the array form) instead of stopping the run. This is how you say *if it ran* |
|
|
298
|
+
|
|
299
|
+
**Which of the two aimed forms is legal is decided by the graph and refused at construction**
|
|
300
|
+
(`FD-V28`), so you are told which to write rather than discovering the shape at run time. The message
|
|
301
|
+
names the form to use instead.
|
|
302
|
+
|
|
303
|
+
**And the bare form is a CLAIM that the node ran.** If a run can reach you without it having run — a
|
|
304
|
+
decision routes around it, or it sits in a region whose branch may be lost — that is `FD-V29` at
|
|
305
|
+
construction, and the fix is the `?`. **It is not dominance**: a `fanout`'s branches are all entered,
|
|
306
|
+
so a branch entry is certain from after the join even though it is not on EVERY path there. See
|
|
307
|
+
[ADR 0042](doc/decisions/0042-a-reference-says-whether-it-is-conditional-and-the-bare-form-means-the-target-certainly-ran.md).
|
|
308
|
+
|
|
309
|
+
```ts run
|
|
310
|
+
import { Driver, validateDefinition, type FlowDefinition } from 'fyflow-orchestration';
|
|
311
|
+
|
|
312
|
+
// *Score each item against the run's config* — a map instance seeing its own item AND
|
|
313
|
+
// the context around it. Before ADR 0040 `config` was reachable from a `score` worker
|
|
314
|
+
// by no route at all.
|
|
315
|
+
const withContext: FlowDefinition = {
|
|
316
|
+
flowId: 'score-against-config', version: 1, entry: 'ingest',
|
|
317
|
+
nodes: [
|
|
318
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'split' }] },
|
|
319
|
+
{ id: 'split', kind: 'map', over: 'items', branch: 'score' },
|
|
320
|
+
{ id: 'score', kind: 'task', type: 'ScoreWorker',
|
|
321
|
+
needs: { item: '.', run: 'ingest' }, // its own item, and ingest's whole result
|
|
322
|
+
edges: [{ to: 'collect' }] },
|
|
323
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'publish' }] },
|
|
324
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
325
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
326
|
+
],
|
|
327
|
+
};
|
|
328
|
+
|
|
329
|
+
// *Score, then fan out over the scores* — a map whose `over` names a key composed from a
|
|
330
|
+
// whole region's output. No branch numbers appear anywhere, so inserting a branch cannot
|
|
331
|
+
// silently re-point it.
|
|
332
|
+
const overRegion: FlowDefinition = {
|
|
333
|
+
flowId: 'fan-out-over-the-scores', version: 1, entry: 'ingest',
|
|
334
|
+
nodes: [
|
|
335
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'split' }] },
|
|
336
|
+
{ id: 'split', kind: 'map', over: 'items', branch: 'score' },
|
|
337
|
+
{ id: 'score', kind: 'task', type: 'ScoreWorker', edges: [{ to: 'collect' }] },
|
|
338
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'rank' }] },
|
|
339
|
+
{ id: 'rank', kind: 'map', over: 'scores', branch: 'publish',
|
|
340
|
+
needs: { scores: ['score'] } }, // every score, in item order
|
|
341
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'gather' }] },
|
|
342
|
+
{ id: 'gather', kind: 'join', of: 'rank', edges: [{ to: 'done' }] },
|
|
343
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
344
|
+
],
|
|
345
|
+
};
|
|
346
|
+
|
|
347
|
+
const registry = {
|
|
348
|
+
predicates: [],
|
|
349
|
+
workers: { flow: { url: 'worker.js' } }, nodeTypes: { IngestWorker: { worker: 'flow' }, ScoreWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
350
|
+
};
|
|
351
|
+
for (const def of [withContext, overRegion]) {
|
|
352
|
+
const bad = validateDefinition(def, registry);
|
|
353
|
+
if (bad.length) throw new Error(`${def.flowId} is invalid: ${bad.map((f) => f.rule).join()}`);
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
// And the wrong FORM is refused at construction, naming the one to write instead.
|
|
357
|
+
const wrongForm: FlowDefinition = {
|
|
358
|
+
...overRegion,
|
|
359
|
+
nodes: overRegion.nodes.map((n) =>
|
|
360
|
+
n.id === 'rank' ? { ...n, needs: { scores: 'score' } } as typeof n : n),
|
|
361
|
+
};
|
|
362
|
+
const findings = validateDefinition(wrongForm, registry);
|
|
363
|
+
if (!findings.some((f) => f.rule === 'FD-V28' && f.detail.includes('write ["score"]'))) {
|
|
364
|
+
throw new Error('FD-V28 did not name the form to write instead');
|
|
365
|
+
}
|
|
366
|
+
|
|
367
|
+
// *Route to review if an earlier step flagged it, and publish either way* — a reference
|
|
368
|
+
// to a value only SOME paths produce. Written bare it is FD-V29 at construction; the `?`
|
|
369
|
+
// is how you say *if it ran*, and then `verdict` is simply missing on the other road.
|
|
370
|
+
const ifItRan = (ref: string): FlowDefinition => ({
|
|
371
|
+
flowId: 'publish-either-way', version: 1, entry: 'decide',
|
|
372
|
+
nodes: [
|
|
373
|
+
{ id: 'decide', kind: 'task', type: 'ScoreWorker', edges: [
|
|
374
|
+
{ when: { predicate: 'isLowConfidence' }, to: 'review' },
|
|
375
|
+
{ to: 'publish' },
|
|
376
|
+
] },
|
|
377
|
+
{ id: 'review', kind: 'task', type: 'ScoreWorker', edges: [{ to: 'publish' }] },
|
|
378
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker',
|
|
379
|
+
needs: { own: '.', verdict: ref },
|
|
380
|
+
edges: [{ to: 'done' }] },
|
|
381
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
382
|
+
],
|
|
383
|
+
});
|
|
384
|
+
const guarded = { predicates: ['isLowConfidence'], workers: registry.workers, nodeTypes: registry.nodeTypes };
|
|
385
|
+
const bare = validateDefinition(ifItRan('review'), guarded);
|
|
386
|
+
if (!bare.some((f) => f.rule === 'FD-V29' && f.detail.includes('"review?"'))) {
|
|
387
|
+
throw new Error('FD-V29 did not refuse the bare form, or did not name the fix');
|
|
388
|
+
}
|
|
389
|
+
if (validateDefinition(ifItRan('review?'), guarded).length) {
|
|
390
|
+
throw new Error('the conditional form should be accepted');
|
|
391
|
+
}
|
|
392
|
+
void Driver;
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
**What it costs.** An **unconditional** reference that names a node with nothing to give you stops the
|
|
396
|
+
run with an `UnresolvedReference` (`FD-R3`, section 9). Since v0.13 most of that is caught at
|
|
397
|
+
construction — `FD-V27` checks the node is *upstream of you* and `FD-V29` that it must have RUN — so
|
|
398
|
+
what is left at run time is **whether the join a policy fired had actually waited for it**. *(This
|
|
399
|
+
read *"the TWO things the graph cannot know"* and named the arity case first — two `map` regions of
|
|
400
|
+
different arity, where a reader inside one could see neither output of the other. **`spec/`
|
|
401
|
+
v0.15 removes that road**: [ADR 0044](doc/decisions/0044-a-reference-is-visible-when-two-paths-agree-on-the-regions-that-contain-both-of-them.md)
|
|
402
|
+
makes a reader see all of a sibling region's outputs, so the case is a defect that was fixed rather
|
|
403
|
+
than a limit of what a validator can see, and one road is left. `doc/workflow.md` section 1 rule 5's
|
|
404
|
+
third direction, and this sentence is the copy a consumer reads first — it said *the resolver follows
|
|
405
|
+
in W50* for exactly as long as that was true.)*
|
|
406
|
+
And a reference names a **node**, never a path inside its value: `needs: { run: 'ingest' }` gives you
|
|
407
|
+
the whole of `ingest`'s result and your worker reads `.config` off it.
|
|
408
|
+
|
|
409
|
+
**And inside a `map`, what it costs is what you point it at.** Each instance gets its OWN COPY of
|
|
410
|
+
each key, so a reference to a large result is copied once per item. Measured at a map arity of 1000
|
|
411
|
+
over a ~126 KiB payload, **one case per process**: no `needs` **5.5 ms**, `{item: '.'}` **7.7 ms**,
|
|
412
|
+
one more key naming a genuinely small node **8.7 ms**, and the same key naming the large one
|
|
413
|
+
**613.6 ms** — **71x**, and the `?` suffix costs nothing (**617.4 ms**). The mechanism makes one copy
|
|
414
|
+
per instance and the cost is linear in the size of each, so the product is what you pay. That is also
|
|
415
|
+
why a reference names a whole node and never a path inside it — a path language would hide this
|
|
416
|
+
choice. *(These five figures are compared to `spec/flow-definition.md` section 3's own table by
|
|
417
|
+
`check-docs`' `X6`, both ways — three documents carried them and nothing compared any two, which is
|
|
418
|
+
`doc/workflow.md` section 5 rule 8 obligation (e)'s third shape.)*
|
|
419
|
+
|
|
420
|
+
**Nothing warns about it, and that is a decision rather than a gap.** There is no threshold to warn
|
|
421
|
+
at; the static predicate that needs no threshold fires on **every** intended use of the feature,
|
|
422
|
+
including the flow it exists for, so it restates the definition rather than diagnosing anything; and
|
|
423
|
+
a report channel exists (`onUnmintedTask`'s shape) and is declined because *this reference is resolved
|
|
424
|
+
once per item* is a fact about your document rather than about your run. **Measuring is not the
|
|
425
|
+
obstacle** — a size measurement is about 0.03% of what it would warn about at arity 1000.
|
|
426
|
+
`spec/flow-definition.md` section 3 carries the whole verdict and the measurements.
|
|
427
|
+
|
|
428
|
+
### Validation happens at construction, not at routing time
|
|
429
|
+
|
|
430
|
+
`new Driver(...)` validates the whole definition before anything is queued. A typo cannot become a
|
|
431
|
+
routing failure discovered months later on the one workload that takes that branch.
|
|
432
|
+
|
|
433
|
+
```ts run
|
|
434
|
+
import { validateDefinition, type FlowDefinition } from 'fyflow-orchestration';
|
|
435
|
+
|
|
436
|
+
const broken = {
|
|
437
|
+
flowId: 'broken', version: 1, entry: 'a',
|
|
438
|
+
nodes: [{ id: 'a', kind: 'task', type: 'NoSuchType', edges: [{ to: 'ghost' }] }],
|
|
439
|
+
} as unknown as FlowDefinition;
|
|
440
|
+
|
|
441
|
+
const findings = validateDefinition(broken, { predicates: [], workers: { flow: { url: 'worker.js' } }, nodeTypes: { RealType: { worker: 'flow' } } });
|
|
442
|
+
console.log(findings.map((f) => `${f.rule} at ${f.at}`).sort());
|
|
443
|
+
// FD-V2 (edge to a missing node) and FD-V32 (a node type the registry does not hold)
|
|
444
|
+
const rules = findings.map((f) => f.rule).sort();
|
|
445
|
+
if (!rules.includes('FD-V2') || !rules.includes('FD-V32')) {
|
|
446
|
+
throw new Error(`expected FD-V2 and FD-V32, got ${rules.join(',')}`);
|
|
447
|
+
}
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
There are **42 refusals** over the id range `FD-V1` … `FD-V46` *(41 over `FD-V1` … `FD-V45` until W109 shipped the first BUILT-IN node type and added `FD-V46`, which refuses a node of a type that only transforms and declares no `transform`; 37 over `FD-V1` … `FD-V41` until W108 made a predicate DATA and added the four a `when` needs; 34 over `FD-V1` … `FD-V38` until W92 added the gate's three; 35 over `FD-V1` … `FD-V37` until W65 retired `workerType` and with it `FD-V5` and `FD-V34`, adding `FD-V38`; 32 over `FD-V1` … `FD-V34` until W61 added the three transform rules; 29 over `FD-V1` … `FD-V31` until W59 added the three node-type rules — and this sentence stood at 29 twelve lines below the table W59 rewrote, until its conformity reader read the two together)*, each with a failing example in
|
|
451
|
+
`spec/examples/invalid/`. Four ids in that range are **deliberately absent** and have no rule behind
|
|
452
|
+
them: `FD-V14`, which never existed; **`FD-V15`, retired when nested fan-out became
|
|
453
|
+
expressible**; and **`FD-V5` and `FD-V34`, retired with `workerType`** at v0.19 (section 13). The ids are never renumbered, so it is safe to assert on them. The full
|
|
454
|
+
list is `spec/flow-definition.md`.
|
|
455
|
+
|
|
456
|
+
**`FD-V14` is deliberately absent, and its id is held open so the gap is not read as a lost rule.**
|
|
457
|
+
It would have said *every path from a branch entry reaches the join* — which would make a decision
|
|
458
|
+
that routes a branch away from its join invalid input, and that case is the one this library exists
|
|
459
|
+
to get right (`ADR 0003`). Do not add it.
|
|
460
|
+
|
|
461
|
+
---
|
|
462
|
+
|
|
463
|
+
## 4. Decisions and predicates
|
|
464
|
+
|
|
465
|
+
**A `when` is an object, and it is either a registered NAME or a DECLARED comparison.** Since v0.27
|
|
466
|
+
([ADR 0074](doc/decisions/0074-a-predicate-is-a-registered-name-or-a-declared-comparison-over-the-context.md))
|
|
467
|
+
a flow document can say what a branch depends on without any code at all, which is what lets a UI or
|
|
468
|
+
an agent emit a complete flow. A name still works and is what a predicate that COMPUTES uses.
|
|
469
|
+
|
|
470
|
+
```ts run
|
|
471
|
+
import { validateDefinition, type FlowDefinition } from 'fyflow-orchestration';
|
|
472
|
+
|
|
473
|
+
// A `when` is EXACTLY ONE of three shapes, told apart by which keys it carries.
|
|
474
|
+
const flow: FlowDefinition = {
|
|
475
|
+
flowId: 'declared', version: 1, entry: 'score',
|
|
476
|
+
nodes: [
|
|
477
|
+
{ id: 'score', kind: 'task', type: 'ScoreWorker', edges: [
|
|
478
|
+
// 1. a COMPARISON — a key path over the context, an operator, a literal.
|
|
479
|
+
{ when: { path: 'outcome.result.confidence', op: 'lt', value: 0.5 }, to: 'review' },
|
|
480
|
+
// 2. a flat GROUP — `all` or `any`, whose members are comparisons.
|
|
481
|
+
{ when: { any: [
|
|
482
|
+
{ path: 'outcome.result.score', op: 'gte', value: 90 },
|
|
483
|
+
// `default` stands in where the path does not resolve. It is what `?? 0` is, and it
|
|
484
|
+
// is what makes `lte` the negation of `gt` — without it BOTH are false when absent.
|
|
485
|
+
{ path: 'outcome.result.retries', op: 'gt', value: 2, default: 0 },
|
|
486
|
+
] }, to: 'review' },
|
|
487
|
+
// 3. a registered NAME — for anything the subset cannot say.
|
|
488
|
+
{ when: { predicate: 'needsAHuman' }, to: 'review' },
|
|
489
|
+
{ to: 'publish' },
|
|
490
|
+
] },
|
|
491
|
+
{ id: 'review', kind: 'task', type: 'ReviewWorker', edges: [{ to: 'done' }] },
|
|
492
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
493
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
494
|
+
],
|
|
495
|
+
};
|
|
496
|
+
const registry = {
|
|
497
|
+
predicates: ['needsAHuman'],
|
|
498
|
+
workers: { flow: { url: 'worker.js' } },
|
|
499
|
+
nodeTypes: { ScoreWorker: { worker: 'flow' }, ReviewWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
500
|
+
};
|
|
501
|
+
if (validateDefinition(flow, registry).length !== 0) throw new Error('the three shapes are all valid');
|
|
502
|
+
|
|
503
|
+
// **Every refusal is at CONSTRUCTION, with the key it is about.** Four rules:
|
|
504
|
+
const refuse = (when: unknown) => validateDefinition(
|
|
505
|
+
{ ...flow, nodes: flow.nodes.map((n) => n.id === 'score' ? { ...n, edges: [{ when, to: 'review' }, { to: 'publish' }] } : n) } as FlowDefinition,
|
|
506
|
+
registry).map((f) => f.rule);
|
|
507
|
+
|
|
508
|
+
// FD-V42 — the bare STRING that was the form until v0.27, and the message names the migration.
|
|
509
|
+
if (refuse('needsAHuman')[0] !== 'FD-V42') throw new Error('a bare string is FD-V42');
|
|
510
|
+
// FD-V43 — a path whose first segment is neither `outcome` nor `needs` would resolve to
|
|
511
|
+
// nothing in SILENCE, so it is refused rather than answered.
|
|
512
|
+
if (refuse({ path: 'result.confidence', op: 'lt', value: 0.5 })[0] !== 'FD-V43') throw new Error('a rootless path is FD-V43');
|
|
513
|
+
// FD-V44 — the seven operators are a CLOSED set and one outside it is named.
|
|
514
|
+
if (refuse({ path: 'outcome.result.name', op: 'matches', value: '^a' })[0] !== 'FD-V44') throw new Error('an unknown operator is FD-V44');
|
|
515
|
+
// FD-V45 — a group is FLAT, and a member naming a registered predicate could be IMPURE.
|
|
516
|
+
if (refuse({ all: [{ any: [{ path: 'outcome.result.a', op: 'eq', value: 1 }] }] })[0] !== 'FD-V45') throw new Error('nesting is FD-V45');
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
| | |
|
|
520
|
+
|---|---|
|
|
521
|
+
| `path` | a key path over the context, **rooted at `outcome` or `needs`** — `outcome.result.x`, `outcome.taskId`, `needs.earlier.flag`. No array index, no wildcard, no expression |
|
|
522
|
+
| `op` | one of `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `startsWith`, and **nothing else** |
|
|
523
|
+
| `value` | a scalar: `null`, a boolean, a finite number or a string |
|
|
524
|
+
| `default` | optional — what is compared when the path does not resolve **or resolves to `null`**, which is exactly `??`. Without it an absent path is **no match**, and a resolved `null` is compared as the scalar it is |
|
|
525
|
+
| `length` | optional `true` — compare the LENGTH of an array or a string, after `default` |
|
|
526
|
+
|
|
527
|
+
**Three things to know before you write one.**
|
|
528
|
+
**A comparison between kinds is no match, never a coercion** — `{ op: 'lt', value: 50 }` is false
|
|
529
|
+
against the string `"9"` where JavaScript's `<` would say true.
|
|
530
|
+
**An absent path is no match for every operator, including `ne`** — so `ne` does not mean *the key
|
|
531
|
+
is missing or different*; add a `default` if that is what you want, and a `default` covers a key
|
|
532
|
+
holding `null` as well as a missing one.
|
|
533
|
+
**Write a negation as the complementary operator** — `gte` for *not* `lt` — and know its scope:
|
|
534
|
+
that holds over NUMBERS, and over an absent path where you declared a `default`. Over a value an
|
|
535
|
+
ordering operator cannot compare, a string or a boolean, **both it and its complement are false**.
|
|
536
|
+
`startsWith` has no complement here at all.
|
|
537
|
+
**A declared predicate CANNOT be impure.** There is no clock, no random source and no way to name
|
|
538
|
+
anything but the context, so re-evaluating it over the same log gives the same answer. A registered
|
|
539
|
+
name has no such guarantee — which is the trade, and why a regular expression, a date window or a
|
|
540
|
+
lookup stays a name.
|
|
541
|
+
|
|
542
|
+
### A registered NAME, which is what the subset cannot say
|
|
543
|
+
|
|
544
|
+
**An edge names a predicate; it does not carry one.** The definition stays data, and an unknown name
|
|
545
|
+
fails at construction rather than mid-run.
|
|
546
|
+
|
|
547
|
+
```ts run
|
|
548
|
+
import { eligibleEdges, selectEdge, type FlowNode, type Predicate } from 'fyflow-orchestration';
|
|
549
|
+
|
|
550
|
+
const node: FlowNode = {
|
|
551
|
+
id: 'score', kind: 'task', type: 'ScoreWorker',
|
|
552
|
+
edges: [{ when: { predicate: 'isLowConfidence' }, to: 'review' }, { to: 'publish' }],
|
|
553
|
+
};
|
|
554
|
+
|
|
555
|
+
const isLowConfidence: Predicate = (ctx) => {
|
|
556
|
+
const r = ctx.outcome.result as { confidence?: number } | null;
|
|
557
|
+
return typeof r?.confidence === 'number' && r.confidence < 0.5;
|
|
558
|
+
};
|
|
559
|
+
|
|
560
|
+
const outcome = {
|
|
561
|
+
kind: 'outcome' as const, taskId: 'r1:score:-:0', terminalState: 'done' as const,
|
|
562
|
+
result: { confidence: 0.2 }, error: null, attempts: 0,
|
|
563
|
+
schedulerTimestamp: 1, observedAt: 1, startTime: 1, endTime: 2, executionTime: 1,
|
|
564
|
+
};
|
|
565
|
+
|
|
566
|
+
// Eligibility is decided FIRST, by terminal state; `when` is evaluated only over
|
|
567
|
+
// the edges that are already eligible.
|
|
568
|
+
console.log(eligibleEdges(node, 'done').map((e) => e.to)); // [ 'review', 'publish' ]
|
|
569
|
+
|
|
570
|
+
// **A context BUILDER, not a context** - each edge declares its own `needs`, so each
|
|
571
|
+
// predicate sees a different `ctx.needs`. `selectEdge` calls this only for an edge
|
|
572
|
+
// that carries a `when`, so a default edge costs nothing.
|
|
573
|
+
const taken = selectEdge(node, 'r1', 'done', () => ({ outcome, needs: {} }),
|
|
574
|
+
new Map([['isLowConfidence', isLowConfidence]]));
|
|
575
|
+
console.log(taken.taken?.to); // 'review'
|
|
576
|
+
if (taken.taken?.to !== 'review') throw new Error('the guarded edge should win at 0.2');
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
**Edges are tried in declaration order** and the first whose `when` passes is taken. An edge with no
|
|
580
|
+
`when` is the default and matches immediately, so **it must come last among the edges eligible for
|
|
581
|
+
that terminal state** — not last in the array. `FD-V7` is scoped per terminal state, which is why
|
|
582
|
+
section 3's example can declare its `on: ['failed']` edge first: for `done` the eligible edges are
|
|
583
|
+
`review` then `publish`, and the default is last of those.
|
|
584
|
+
|
|
585
|
+
A predicate receives the node's **effective outcome** — the same shape whether the task settled or a
|
|
586
|
+
human resolved it. It should be pure: it is called during routing, and the library does not enforce
|
|
587
|
+
that.
|
|
588
|
+
|
|
589
|
+
### An edge can let its predicate see an EARLIER node
|
|
590
|
+
|
|
591
|
+
By default a predicate sees **only its own node's outcome**. *Route to review if an earlier step
|
|
592
|
+
flagged low confidence* was not expressible until v0.12 —
|
|
593
|
+
[ADR 0041](doc/decisions/0041-a-decision-sees-its-nodes-outcome-and-whatever-its-own-edge-declares.md),
|
|
594
|
+
answering [Q8](doc/questions/q8.md).
|
|
595
|
+
|
|
596
|
+
**An edge may carry `needs`, and its predicate sees the result under `ctx.needs`.** Same grammar as
|
|
597
|
+
a node's (section 3) — a node id, or a one-element array holding one — **minus `"."`**, which an edge
|
|
598
|
+
has no use for: *what a predicate would have received anyway* is `ctx.outcome`.
|
|
599
|
+
|
|
600
|
+
```ts run
|
|
601
|
+
import { validateDefinition, type FlowDefinition, type Predicate } from 'fyflow-orchestration';
|
|
602
|
+
|
|
603
|
+
const registry = {
|
|
604
|
+
predicates: ['flaggedEarlier'],
|
|
605
|
+
workers: { flow: { url: 'worker.js' } }, nodeTypes: { IngestWorker: { worker: 'flow' }, ScoreWorker: { worker: 'flow' }, ReviewWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
606
|
+
};
|
|
607
|
+
|
|
608
|
+
const flow: FlowDefinition = {
|
|
609
|
+
flowId: 'route-on-an-earlier-step', version: 1, entry: 'ingest',
|
|
610
|
+
nodes: [
|
|
611
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'score' }] },
|
|
612
|
+
{ id: 'score', kind: 'task', type: 'ScoreWorker', edges: [
|
|
613
|
+
// this edge's predicate can see `ingest`; the default edge below sees `{}`
|
|
614
|
+
{ when: { predicate: 'flaggedEarlier' }, to: 'review', needs: { earlier: 'ingest' } },
|
|
615
|
+
{ to: 'publish' },
|
|
616
|
+
] },
|
|
617
|
+
{ id: 'review', kind: 'task', type: 'ReviewWorker', edges: [{ to: 'publish' }] },
|
|
618
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
619
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
620
|
+
],
|
|
621
|
+
};
|
|
622
|
+
|
|
623
|
+
const flaggedEarlier: Predicate = (ctx) =>
|
|
624
|
+
(ctx.needs as { earlier?: { flagged?: boolean } }).earlier?.flagged === true;
|
|
625
|
+
|
|
626
|
+
const bad = validateDefinition(flow, registry);
|
|
627
|
+
if (bad.length) throw new Error(`invalid: ${bad.map((f) => f.rule).join()}`);
|
|
628
|
+
|
|
629
|
+
// And the predicate is CALLED, so the one thing this section exists to show is
|
|
630
|
+
// executed rather than only typechecked.
|
|
631
|
+
const asDriverWouldBuildIt = {
|
|
632
|
+
outcome: { result: { score: 0.9 } } as never,
|
|
633
|
+
needs: { earlier: { flagged: true } },
|
|
634
|
+
};
|
|
635
|
+
if (!flaggedEarlier(asDriverWouldBuildIt)) throw new Error('ctx.needs.earlier was not what the edge declared');
|
|
636
|
+
if (flaggedEarlier({ ...asDriverWouldBuildIt, needs: {} })) {
|
|
637
|
+
throw new Error('an edge that declares nothing must give {} and the predicate must say no');
|
|
638
|
+
}
|
|
639
|
+
|
|
640
|
+
// `"."` is refused ON AN EDGE, and the message names what you already have.
|
|
641
|
+
const withSelf = {
|
|
642
|
+
...flow,
|
|
643
|
+
nodes: flow.nodes.map((n) =>
|
|
644
|
+
n.id === 'score'
|
|
645
|
+
? { ...n, edges: [{ when: { predicate: 'flaggedEarlier' }, to: 'review', needs: { x: '.' } }, { to: 'publish' }] }
|
|
646
|
+
: n),
|
|
647
|
+
} as FlowDefinition;
|
|
648
|
+
const findings = validateDefinition(withSelf, registry);
|
|
649
|
+
if (!findings.some((f) => f.rule === 'FD-V26' && f.detail.includes('ctx.outcome'))) {
|
|
650
|
+
throw new Error('FD-V26 did not refuse `"."` on an edge');
|
|
651
|
+
}
|
|
652
|
+
```
|
|
653
|
+
|
|
654
|
+
**It is on the EDGE and not on the node, for two reasons worth knowing.** A `needs` on the node
|
|
655
|
+
changes what its **worker** receives, so you could not give a predicate reach without editing a
|
|
656
|
+
worker; and *what the node received* is not something the driver can reconstruct when the decision is
|
|
657
|
+
made — the **entry** node's payload is never written to the log at all, and reconstructing any other
|
|
658
|
+
node's would mean re-deriving the whole payload contract a second time. **A reference resolves from
|
|
659
|
+
what the run produced**, which is available either way.
|
|
660
|
+
|
|
661
|
+
**`ctx.needs` is `{}` when the edge declares nothing**, never `undefined`; it is **frozen**, so a
|
|
662
|
+
predicate that writes to it throws rather than editing what the next predicate reads; and a reference
|
|
663
|
+
the run did not produce stops the run with an `UnresolvedReference` (`FD-R3`, section 9) exactly as a
|
|
664
|
+
node's does. **An edge with no `when` may carry a `needs` and nothing will ever read it** — accepted
|
|
665
|
+
and inert, which `spec/flow-definition.md` section 3 records rather than refuses.
|
|
666
|
+
|
|
667
|
+
---
|
|
668
|
+
|
|
669
|
+
## 5. Fan-out, join, and which policy to pass
|
|
670
|
+
|
|
671
|
+
```ts run
|
|
672
|
+
import { validateDefinition, type FlowDefinition } from 'fyflow-orchestration';
|
|
673
|
+
|
|
674
|
+
const fanout: FlowDefinition = {
|
|
675
|
+
flowId: 'score-both', version: 1, entry: 'ingest',
|
|
676
|
+
nodes: [
|
|
677
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'split' }] },
|
|
678
|
+
{ id: 'split', kind: 'fanout', branches: ['scoreA', 'scoreB'] },
|
|
679
|
+
{ id: 'scoreA', kind: 'task', type: 'ScoreWorker', edges: [{ to: 'collect' }] },
|
|
680
|
+
{ id: 'scoreB', kind: 'task', type: 'ScoreWorker', edges: [{ to: 'collect' }] },
|
|
681
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'publish' }] },
|
|
682
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
683
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
684
|
+
],
|
|
685
|
+
};
|
|
686
|
+
|
|
687
|
+
const bad = validateDefinition(fanout, {
|
|
688
|
+
predicates: [],
|
|
689
|
+
workers: { flow: { url: 'worker.js' } }, nodeTypes: { IngestWorker: { worker: 'flow' }, ScoreWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
690
|
+
});
|
|
691
|
+
if (bad.length) throw new Error(`section 5's own example is invalid: ${bad.map((f) => f.rule).join()}`);
|
|
692
|
+
```
|
|
693
|
+
|
|
694
|
+
### Pass `liveToken`
|
|
695
|
+
|
|
696
|
+
Three policies are exported. **Use `liveToken`.**
|
|
697
|
+
|
|
698
|
+
```ts run
|
|
699
|
+
import { POLICIES, liveToken, arrivalCount, liveTokenRequiringArrival } from 'fyflow-orchestration';
|
|
700
|
+
|
|
701
|
+
const ids = Object.keys(POLICIES).sort();
|
|
702
|
+
console.log(ids);
|
|
703
|
+
if (ids.join() !== 'arrival-count,live-token,live-token-requiring-arrival') {
|
|
704
|
+
throw new Error(`this reference lists three policies and the library exports ${ids.join(', ')}`);
|
|
705
|
+
}
|
|
706
|
+
console.log(liveToken.id, arrivalCount.id, liveTokenRequiringArrival.id);
|
|
707
|
+
if (POLICIES['live-token'] !== liveToken) throw new Error('POLICIES must expose the same objects');
|
|
708
|
+
```
|
|
709
|
+
|
|
710
|
+
**That `throw` used to be a trailing comment**, and it is the one line of this
|
|
711
|
+
section a reader could rely on being current: `check-docs` executes the snippet
|
|
712
|
+
and asserts nothing about a comment, so a fourth policy would have left it
|
|
713
|
+
silently wrong with the whole gate green. `JP-R9` and `npm run check:policies`
|
|
714
|
+
are the other half — they compare `POLICIES` against `spec/join-policy.md`
|
|
715
|
+
section 3, which is where the names are actually fixed.
|
|
716
|
+
|
|
717
|
+
`arrivalCount` waits for **N** branches, where N is fixed when the fan-out opens. That is wrong
|
|
718
|
+
whenever *"how many will arrive"* differs from *"how many opened"* — and **every non-success outcome
|
|
719
|
+
makes it differ**. A branch routed away by a decision never arrives. A branch whose `optional` task
|
|
720
|
+
fails never arrives. The counter never completes, and the run waits forever with nothing queued and
|
|
721
|
+
nothing running. **Nothing throws** — but run state does report it: such a run folds to **`stuck`**,
|
|
722
|
+
the sixth outcome, and `driver.joinStatusOf()` names the join that can no longer fire
|
|
723
|
+
([ADR 0015](doc/decisions/0015-a-run-whose-join-can-no-longer-fire-is-stuck.md)). A hang you have to
|
|
724
|
+
notice yourself is still a hang.
|
|
725
|
+
|
|
726
|
+
`liveToken` counts *live branches* instead: opening one increments, and arriving, exiting or being
|
|
727
|
+
lost each decrement. The join fires when the count reaches zero **having been positive**, whichever
|
|
728
|
+
road each branch took.
|
|
729
|
+
|
|
730
|
+
`liveTokenRequiringArrival` is `liveToken` **with one clause added**: it fires only if at least one
|
|
731
|
+
branch actually arrived. On a flow where every branch is routed away, `liveToken` fires a join **no
|
|
732
|
+
branch reached** and carries **`{}`** onward — an empty collection, which is what a join with no
|
|
733
|
+
arrivals has to say; this one instead reports the join as unfireable — through
|
|
734
|
+
`onUnsatisfiableJoin`, with the policy's own state, and the run folds to **`stuck`** with
|
|
735
|
+
`regions[…][…].joined === false` whether or not you registered that callback — **two subscripts**,
|
|
736
|
+
because `RunState.regions` is keyed by a region's opener and then by that opener's own `branchPath`
|
|
737
|
+
([ADR 0062](doc/decisions/0062-a-region-instance-is-its-opener-and-the-openers-own-branch-path.md)).
|
|
738
|
+
A region whose opener is at top level has one entry, at `'-'`.
|
|
739
|
+
|
|
740
|
+
```ts run
|
|
741
|
+
import {
|
|
742
|
+
liveToken, liveTokenRequiringArrival,
|
|
743
|
+
type JoinPolicy, type Region, type RegionEvent,
|
|
744
|
+
} from 'fyflow-orchestration';
|
|
745
|
+
|
|
746
|
+
const region: Region = { fanoutId: 'split', joinId: 'collect', branches: ['a', 'b'], members: [] };
|
|
747
|
+
// `regionPath` is the OPENER'S own branchPath, so the pair (regionId, regionPath)
|
|
748
|
+
// names ONE region instance — `'-'` here because `split` opens at top level. A map
|
|
749
|
+
// enters one opener once per item, so its node id alone stopped naming one region.
|
|
750
|
+
const everyBranchExits: RegionEvent[] = [
|
|
751
|
+
{ type: 'branch.opened', regionId: 'split', regionPath: '-', branchPath: '0' },
|
|
752
|
+
{ type: 'branch.opened', regionId: 'split', regionPath: '-', branchPath: '1' },
|
|
753
|
+
{ type: 'branch.exited', regionId: 'split', regionPath: '-', branchPath: '0', at: 'stop' },
|
|
754
|
+
{ type: 'branch.exited', regionId: 'split', regionPath: '-', branchPath: '1', at: 'stop' },
|
|
755
|
+
];
|
|
756
|
+
|
|
757
|
+
function verdict<S>(policy: JoinPolicy<S>, events: RegionEvent[]) {
|
|
758
|
+
let state = policy.open(region);
|
|
759
|
+
for (const event of events) state = policy.step(state, event);
|
|
760
|
+
return policy.status(state);
|
|
761
|
+
}
|
|
762
|
+
|
|
763
|
+
console.log(verdict(liveToken, everyBranchExits), verdict(liveTokenRequiringArrival, everyBranchExits));
|
|
764
|
+
if (verdict(liveToken, everyBranchExits) !== 'satisfied') {
|
|
765
|
+
throw new Error('liveToken fires with zero arrivals');
|
|
766
|
+
}
|
|
767
|
+
if (verdict(liveTokenRequiringArrival, everyBranchExits) !== 'unsatisfiable') {
|
|
768
|
+
throw new Error('liveTokenRequiringArrival declines a join no branch reached');
|
|
769
|
+
}
|
|
770
|
+
```
|
|
771
|
+
|
|
772
|
+
**Which to pass is a judgement about your flow, and neither answer is safe by default.** If every
|
|
773
|
+
branch of a fan-out ending somewhere else is a legitimate way for your flow to finish, `liveToken`
|
|
774
|
+
is right and the run completes. If it means the work the join was going to combine never happened,
|
|
775
|
+
`liveTokenRequiringArrival` reports that join dead and does not run its successor on nothing. The
|
|
776
|
+
recommendation is still `liveToken`, because a run that stops is the more expensive mistake to make
|
|
777
|
+
by accident.
|
|
778
|
+
|
|
779
|
+
**"Stops the run" is exact at depth 1 and not at depth 2**, which is worth knowing before you nest a
|
|
780
|
+
fan-out. A dead join stops **its own** region; if that region sits inside a branch of another one,
|
|
781
|
+
the branch **ends** and the outer join gets to decide for itself — so the outer join may fire and its
|
|
782
|
+
successor may run, doing real work in a run that will still fold to `stuck` (`joined: false` on the
|
|
783
|
+
dead region is what `stuck` reads, and it reads every region). Section 11's nested-fan-out row, and
|
|
784
|
+
[ADR 0030](doc/decisions/0030-a-join-is-transitive-and-a-dead-join-ends-the-branch-it-sits-in.md).
|
|
785
|
+
|
|
786
|
+
**One refinement is deliberately still unbuilt.** `arrivalCount`'s **N is fixed when the fan-out
|
|
787
|
+
opens** and is *not* decremented when a branch is routed away. Decrementing it is
|
|
788
|
+
[`JP-R7`](spec/join-policy.md), and
|
|
789
|
+
[ADR 0025](doc/decisions/0025-the-zero-arrival-join-is-reportable-and-arrival-counting-is-not-repaired.md)
|
|
790
|
+
refused it: measured over the join matrix, every row where it differs from `liveToken` is a row where
|
|
791
|
+
it **waits forever** and `liveToken` proceeds, and it can never say why. It is `liveToken` weakened.
|
|
792
|
+
|
|
793
|
+
**`arrivalCount` is still exported deliberately** — it is the falsifiable form of the simpler
|
|
794
|
+
mechanism, and the comparison that chose between them is only meaningful while both can be run. It
|
|
795
|
+
is not a supported choice. See
|
|
796
|
+
[ADR 0003](doc/decisions/0003-a-join-is-satisfied-when-its-region-has-no-live-branches.md) and
|
|
797
|
+
[ADR 0025](doc/decisions/0025-the-zero-arrival-join-is-reportable-and-arrival-counting-is-not-repaired.md).
|
|
798
|
+
|
|
799
|
+
### What a join carries downstream
|
|
800
|
+
|
|
801
|
+
**The arrived branches' results, keyed by branch path.** A satisfied join dispatches its successor
|
|
802
|
+
with an object whose keys are the `branchPath`s of the branches that **reached** it, and whose values
|
|
803
|
+
are what each of those branches carried in — so a fan-out's results are not thrown away, and the
|
|
804
|
+
value is the same whatever order the branches finished in.
|
|
805
|
+
|
|
806
|
+
```ts run
|
|
807
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
808
|
+
import { type FlowDefinition, type FoldInput, type RegistryDocument } from 'fyflow-orchestration';
|
|
809
|
+
|
|
810
|
+
// Two workers of your own: one returns a score after a stated delay, the other tags whatever it was
|
|
811
|
+
// handed so the collection is legible in what it returns.
|
|
812
|
+
const registry: RegistryDocument = {
|
|
813
|
+
workers: {
|
|
814
|
+
scorer: { url: '../../examples/getting-started/scorer.ts' },
|
|
815
|
+
tag: { url: '../../examples/getting-started/tag.ts' },
|
|
816
|
+
},
|
|
817
|
+
nodeTypes: {
|
|
818
|
+
Scorer: { worker: 'scorer', inline: true },
|
|
819
|
+
Tag: { worker: 'tag', inline: true },
|
|
820
|
+
},
|
|
821
|
+
};
|
|
822
|
+
const flow: FlowDefinition = {
|
|
823
|
+
flowId: 'collect', version: 1, entry: 'split',
|
|
824
|
+
nodes: [
|
|
825
|
+
{ id: 'split', kind: 'fanout', branches: ['scoreA', 'scoreB'] },
|
|
826
|
+
// `scoreA` is branch path "0" and is made SLOW on purpose, so "1" settles first.
|
|
827
|
+
{ id: 'scoreA', kind: 'task', type: 'Scorer', settings: { score: 0.9, ms: 40 }, edges: [{ to: 'collect' }] },
|
|
828
|
+
{ id: 'scoreB', kind: 'task', type: 'Scorer', settings: { score: 0.4 }, edges: [{ to: 'collect' }] },
|
|
829
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'publish' }] },
|
|
830
|
+
{ id: 'publish', kind: 'task', type: 'Tag', settings: { label: 'publish' }, edges: [{ to: 'done' }] },
|
|
831
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
832
|
+
],
|
|
833
|
+
};
|
|
834
|
+
|
|
835
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
836
|
+
// **`onFoldInput` is still here and what it is for is the ORDER, not the END.** `await done` is the
|
|
837
|
+
// end (ADR 0058). `RunState.nodes` keys everything by task id and carries no settlement order — *which
|
|
838
|
+
// branch settled first* is a fact only the input SEQUENCE carries, and it is what the
|
|
839
|
+
// claim below is measured against. **This is the residue**: `done` takes the termination out of a
|
|
840
|
+
// snippet and leaves whatever else the callback was doing.
|
|
841
|
+
const inputs: FoldInput[] = [];
|
|
842
|
+
const state = await runtime.start(flow, {}, { onFoldInput: (input) => { inputs.push(input); } }).done;
|
|
843
|
+
await runtime.shutdown();
|
|
844
|
+
|
|
845
|
+
// **The order is READ, not assumed.** The delay above ASKS for branch "1" to settle first; this is
|
|
846
|
+
// what the run actually did, and the claim below is about the collection being independent of it.
|
|
847
|
+
const settledOrder = inputs.flatMap((i) =>
|
|
848
|
+
(i.kind === 'outcome' && /:score[AB]:/.test(i.record.taskId) ? [i.record.taskId.split(':')[1]] : []));
|
|
849
|
+
console.log(settledOrder.join(' then ')); // 'scoreB then scoreA'
|
|
850
|
+
|
|
851
|
+
// `publish` is the join's successor, so what it received IS the collection — and `tag` returns what
|
|
852
|
+
// it was handed with its own label added, so the collection is readable in the result.
|
|
853
|
+
const published = Object.values(state.nodes).find((n) => n.nodeId === 'publish')?.result;
|
|
854
|
+
console.log(JSON.stringify(published)); // {"0":{"score":0.9},"1":{"score":0.4},"publish":true}
|
|
855
|
+
|
|
856
|
+
const want = '{"0":{"score":0.9},"1":{"score":0.4},"publish":true}';
|
|
857
|
+
if (JSON.stringify(published) !== want) throw new Error(`the join carried ${JSON.stringify(published)}`);
|
|
858
|
+
// **B settled FIRST and is still under key "1".** The collection is keyed by BRANCH PATH, not by
|
|
859
|
+
// arrival order, which is the whole point — and the two assertions are a pair: the first says which
|
|
860
|
+
// order the run took, the second says the record did not follow it.
|
|
861
|
+
if (settledOrder.join() !== 'scoreB,scoreA') throw new Error(`settled ${settledOrder.join()} — the arrangement did not hold, so the claim below is untested`);
|
|
862
|
+
```
|
|
863
|
+
|
|
864
|
+
**Three things to know before you write code against it.**
|
|
865
|
+
|
|
866
|
+
1. **A branch that did not arrive has no key.** Routed away to an `end`, lost with no eligible edge,
|
|
867
|
+
or still suspended on a person — all three are absent rather than `null`. So `Object.keys(payload)`
|
|
868
|
+
is *which branches arrived*, and it is **not** the width of the fan-out. Code that assumes N
|
|
869
|
+
entries is wrong on every non-success outcome.
|
|
870
|
+
2. **`{}` is a real answer, not an error.** Under `liveToken` a join fires once its region has no live
|
|
871
|
+
branches, and every branch reaching an `end` satisfies that with nothing arrived. If that should be
|
|
872
|
+
a failure instead, pass `liveTokenRequiringArrival` — see above.
|
|
873
|
+
3. **The keys are strings, in ascending branch-path order**, compared segment by segment as numbers —
|
|
874
|
+
so `"0"`, `"1"`, `"2"`, `"10"` at the top level, and `"1.0"`, `"1.1"`, `"1.10"` inside branch 1.
|
|
875
|
+
A nested region's inner join carries its own object, which becomes the value of the outer branch it
|
|
876
|
+
sits in. **Every key of one collection has the same number of segments**, because they are the
|
|
877
|
+
branch paths of one region. *(An earlier version of this bullet illustrated the order as `"1"`,
|
|
878
|
+
`"1.0"`, `"2"`, `"10"` — a mix of depths that no single collection can hold, and one that
|
|
879
|
+
`Object.keys` would not even return in that order, because JavaScript re-orders integer-like keys
|
|
880
|
+
ahead of string ones. W34's conformity reader.)*
|
|
881
|
+
|
|
882
|
+
*(This section read **"`null`, currently … this is [Q36](doc/questions/q36.md) and it is open"**
|
|
883
|
+
until [ADR 0037](doc/decisions/0037-what-a-branch-receives-is-a-property-of-the-fan-out-and-what-a-join-carries-is-a-property-of-the-region.md).
|
|
884
|
+
The argument for the `null` was real and is unchanged — which branch finishes last is genuinely
|
|
885
|
+
non-deterministic — and what answered it is that carrying **all** the arrivals keyed by branch is not
|
|
886
|
+
picking one. `spec/join-policy.md` section 4, `JP-R12`.)*
|
|
887
|
+
|
|
888
|
+
### The same on the RUNTIME — a decision, a fan-out and a join on real workers
|
|
889
|
+
|
|
890
|
+
*Everything in this section runs on a REAL `FyflowScheduler`. The three snippets above this line
|
|
891
|
+
touch no scheduler at all — they validate a document, list the policies, and call a policy
|
|
892
|
+
directly — and the one that did build a fake, *What a join carries downstream*, now builds a
|
|
893
|
+
runtime like everything else ([W71](doc/work-items/w71-the-reference-stops-teaching-a-fake-scheduler.md)).
|
|
894
|
+
*(This sentence has been wrong twice in two days and in opposite directions. It read *everything
|
|
895
|
+
above this line runs on a fake scheduler* while section 2 above it built a real one — W70's
|
|
896
|
+
conformity reader — and then, scoped to this section, it read the same thing while ZERO snippets
|
|
897
|
+
here did, because W71 converted the only one that had. Both times an item rewrote the code inside
|
|
898
|
+
the fences and left every sentence around them untouched.)* This is the same three node kinds on
|
|
899
|
+
`createRuntime`, two runs on one runtime, every value read off the run — [W70](doc/work-items/w70-a-flow-that-decides-and-a-custom-group-on-the-runtime.md).
|
|
900
|
+
It is also `README.md`'s second snippet, held byte-equal to this one by `check-docs`'s `X8`.*
|
|
901
|
+
|
|
902
|
+
```ts run
|
|
903
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
904
|
+
import { type FlowDefinition, type Predicate, type RegistryDocument, type RunState } from 'fyflow-orchestration';
|
|
905
|
+
|
|
906
|
+
// THE REGISTRY — one worker, one type, and the NAME of a predicate the flow may use. A predicate is
|
|
907
|
+
// code (ADR 0002), supplied beside the document under that name; a flow using a name nobody supplied
|
|
908
|
+
// is refused at construction, not at the decision. The worker tags its input with the node's label.
|
|
909
|
+
const registry: RegistryDocument = {
|
|
910
|
+
workers: { tag: { url: '../../examples/getting-started/tag.ts' } }, // in your program: './tag.ts'
|
|
911
|
+
nodeTypes: { Tag: { worker: 'tag', inline: true } },
|
|
912
|
+
predicates: ['isLowConfidence'],
|
|
913
|
+
};
|
|
914
|
+
// A predicate reads the EFFECTIVE outcome, and `result` is `null` for an optional task that
|
|
915
|
+
// failed (`OR-R9`) — so it is guarded, as section 4's is. Unreachable here, because `ingest` is
|
|
916
|
+
// not optional; guarded anyway, because this is the form a consumer copies.
|
|
917
|
+
const isLowConfidence: Predicate = (ctx) => {
|
|
918
|
+
const r = ctx.outcome.result as { confidence?: number } | null;
|
|
919
|
+
return typeof r?.confidence === 'number' && r.confidence < 0.5;
|
|
920
|
+
};
|
|
921
|
+
|
|
922
|
+
// THE FLOW — a DECISION, a FAN-OUT, a JOIN. `ingest`'s first edge is taken when its predicate says so,
|
|
923
|
+
// otherwise the last, unconditional one. `split` opens two branches that run concurrently; `collect`
|
|
924
|
+
// waits for both and hands `publish` every branch's result keyed by branch path (JP-R12). `review` is
|
|
925
|
+
// the road the run takes when confidence is low — it never reaches the join, and the run completes.
|
|
926
|
+
const flow: FlowDefinition = {
|
|
927
|
+
flowId: 'decide', version: 1, entry: 'ingest',
|
|
928
|
+
nodes: [
|
|
929
|
+
{ id: 'ingest', kind: 'task', type: 'Tag', settings: { label: 'ingest' },
|
|
930
|
+
edges: [{ when: { predicate: 'isLowConfidence' }, to: 'review' }, { to: 'split' }] },
|
|
931
|
+
{ id: 'review', kind: 'task', type: 'Tag', settings: { label: 'review' }, edges: [{ to: 'done' }] },
|
|
932
|
+
{ id: 'split', kind: 'fanout', branches: ['enrich', 'classify'] },
|
|
933
|
+
{ id: 'enrich', kind: 'task', type: 'Tag', settings: { label: 'enrich' }, edges: [{ to: 'collect' }] },
|
|
934
|
+
{ id: 'classify', kind: 'task', type: 'Tag', settings: { label: 'classify' }, edges: [{ to: 'collect' }] },
|
|
935
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'publish' }] },
|
|
936
|
+
{ id: 'publish', kind: 'task', type: 'Tag', settings: { label: 'publish' }, edges: [{ to: 'done' }] },
|
|
937
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
938
|
+
],
|
|
939
|
+
};
|
|
940
|
+
|
|
941
|
+
// ONE runtime, TWO runs. Pools belong to the runtime and are shared across runs; each run is its own
|
|
942
|
+
// `Driver` with its own fold, and the join policy defaults to `live-token` (ADR 0003).
|
|
943
|
+
const runtime = createRuntime(registry, { predicates: { isLowConfidence } }, { base: import.meta.url });
|
|
944
|
+
// Each run has its OWN `done`, so two runs are two promises — `start` is called twice and the whole
|
|
945
|
+
// of the wait is `await`. Nothing here folds anything: the runtime keeps the fold (ADR 0058).
|
|
946
|
+
const run = (payload: unknown) => runtime.start(flow, payload).done;
|
|
947
|
+
const high = await run({ confidence: 0.9 });
|
|
948
|
+
const low = await run({ confidence: 0.2 });
|
|
949
|
+
await runtime.shutdown();
|
|
950
|
+
|
|
951
|
+
// WHICH nodes ran, read off each run's own state — the decision took a different edge each time.
|
|
952
|
+
const ran = (s: RunState) => Object.values(s.nodes).map((n) => n.nodeId).sort();
|
|
953
|
+
const resultOf = (s: RunState, id: string) => Object.values(s.nodes).find((n) => n.nodeId === id)?.result;
|
|
954
|
+
console.log(ran(high)); // [ 'classify', 'enrich', 'ingest', 'publish' ]
|
|
955
|
+
console.log(ran(low)); // [ 'ingest', 'review' ]
|
|
956
|
+
console.log(resultOf(high, 'publish')); // { '0': { confidence: 0.9, ingest: true, enrich: true }, '1': { …, classify: true }, publish: true }
|
|
957
|
+
|
|
958
|
+
if (high.outcome !== 'completed' || low.outcome !== 'completed') throw new Error(`outcomes ${high.outcome} / ${low.outcome}`);
|
|
959
|
+
if (ran(high).join() !== 'classify,enrich,ingest,publish') throw new Error(`high ran ${ran(high).join()}`);
|
|
960
|
+
if (ran(low).join() !== 'ingest,review') throw new Error(`low ran ${ran(low).join()}`);
|
|
961
|
+
// `publish` received BOTH branches' results, keyed by branch path (JP-R12), and each branch's value
|
|
962
|
+
// carries the whole road it came by. What this string pins is the branch-to-KEY MAPPING: the ASCENDING
|
|
963
|
+
// order that rule also fixes is not observable at depth 1, because "0" and "1" are integer-like keys
|
|
964
|
+
// and the language orders those ascending whatever the driver did. A nested region would show it.
|
|
965
|
+
const want = '{"0":{"confidence":0.9,"ingest":true,"enrich":true},"1":{"confidence":0.9,"ingest":true,"classify":true},"publish":true}';
|
|
966
|
+
if (JSON.stringify(resultOf(high, 'publish')) !== want) throw new Error(`publish received ${JSON.stringify(resultOf(high, 'publish'))}`);
|
|
967
|
+
```
|
|
968
|
+
|
|
969
|
+
---
|
|
970
|
+
|
|
971
|
+
## 6. When something fails
|
|
972
|
+
|
|
973
|
+
**A non-success outcome routes only where an edge opts in.** This is the rule most people guess
|
|
974
|
+
wrong.
|
|
975
|
+
|
|
976
|
+
```ts run
|
|
977
|
+
import { eligibleEdges, type FlowNode } from 'fyflow-orchestration';
|
|
978
|
+
|
|
979
|
+
const node: FlowNode = {
|
|
980
|
+
id: 'extract', kind: 'task', type: 'ExtractWorker', optional: true,
|
|
981
|
+
edges: [{ on: ['failed'], to: 'quarantine' }, { to: 'publish' }],
|
|
982
|
+
};
|
|
983
|
+
|
|
984
|
+
console.log(eligibleEdges(node, 'done').map((e) => e.to)); // [ 'publish' ]
|
|
985
|
+
console.log(eligibleEdges(node, 'failed').map((e) => e.to)); // [ 'quarantine' ]
|
|
986
|
+
if (eligibleEdges(node, 'failed').map((e) => e.to).join() !== 'quarantine') {
|
|
987
|
+
throw new Error('an edge with no `on` must not be eligible for a failure');
|
|
988
|
+
}
|
|
989
|
+
```
|
|
990
|
+
|
|
991
|
+
An edge with no `on` defaults to `['done']`. **If nothing opts into a failure, the path ends there** —
|
|
992
|
+
that is a legitimate way to write a flow, not an error, and it is declared by omission.
|
|
993
|
+
|
|
994
|
+
### `optional` decides what a failure *means*
|
|
995
|
+
|
|
996
|
+
| the task | the scheduler ends it | the branch |
|
|
997
|
+
|---|---|---|
|
|
998
|
+
| `optional: true` | `failed` | routes if an edge opts in; otherwise **lost** |
|
|
999
|
+
| `optional: false` (default) | **`user_action`** | **suspends** — section 7 |
|
|
1000
|
+
|
|
1001
|
+
**`optional: false` does not mean "must not fail".** It means *"a human decides"*. If you want the
|
|
1002
|
+
flow to handle failure without a person, set `optional: true` and give it an `on: ['failed']` edge.
|
|
1003
|
+
|
|
1004
|
+
**An `on: ['failed']` edge on a non-optional node is not dead code**, which is the natural reading of
|
|
1005
|
+
the table above and is wrong. `resolve(taskId, 'failed', …)` routes exactly as an `optional` node's
|
|
1006
|
+
failure would, and `RS-R3` notes it is **the only road by which a non-`optional` node reaches an
|
|
1007
|
+
`on: ['failed']` edge**. That is the half of the human-in-the-loop story section 7 completes.
|
|
1008
|
+
|
|
1009
|
+
---
|
|
1010
|
+
|
|
1011
|
+
## 7. The human in the loop
|
|
1012
|
+
|
|
1013
|
+
A non-optional task that runs out of retries ends in `user_action`. The branch **suspends**: the join
|
|
1014
|
+
keeps waiting, and the run is not stuck.
|
|
1015
|
+
|
|
1016
|
+
> **A `user_action` task fires TWO events, and the first one is called `task.failed`.** The scheduler
|
|
1017
|
+
> sets `state = 'user_action'` and then dispatches `task.failed` *and* `task.user_action` carrying
|
|
1018
|
+
> the same object. **Never key on the event name alone**: that first `task.failed` is for a task
|
|
1019
|
+
> that did not fail, and treating it as one both double-counts the settlement and routes a
|
|
1020
|
+
> human-intervention case down an error edge. The driver drops it and records on the second — it
|
|
1021
|
+
> matters if you subscribe to the scheduler yourself.
|
|
1022
|
+
> [ADR 0007](doc/decisions/0007-the-driver-normalizes-events-into-an-immutable-outcome-record.md)
|
|
1023
|
+
> rule 2; the scheduler's own reference does not document it for the ordinary path.
|
|
1024
|
+
|
|
1025
|
+
```ts run
|
|
1026
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1027
|
+
import { fold, type FlowDefinition, type FoldInput, type RegistryDocument } from 'fyflow-orchestration';
|
|
1028
|
+
|
|
1029
|
+
// A worker that FAILS — `examples/getting-started/flaky.ts`, a class with a `run` that throws. The
|
|
1030
|
+
// node is not `optional`, so when its retries are spent the scheduler parks it for a person.
|
|
1031
|
+
const registry: RegistryDocument = {
|
|
1032
|
+
workers: { flaky: { url: '../../examples/getting-started/flaky.ts' } },
|
|
1033
|
+
nodeTypes: { Flaky: { worker: 'flaky', inline: true } },
|
|
1034
|
+
};
|
|
1035
|
+
const flow: FlowDefinition = {
|
|
1036
|
+
flowId: 'review', version: 1, entry: 'score',
|
|
1037
|
+
nodes: [
|
|
1038
|
+
{ id: 'score', kind: 'task', type: 'Flaky', settings: {}, edges: [
|
|
1039
|
+
{ on: ['failed'], to: 'quarantine' },
|
|
1040
|
+
{ to: 'done' },
|
|
1041
|
+
] },
|
|
1042
|
+
{ id: 'quarantine', kind: 'end', outcome: 'quarantined' },
|
|
1043
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1044
|
+
],
|
|
1045
|
+
};
|
|
1046
|
+
|
|
1047
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
1048
|
+
const inputs: FoldInput[] = [];
|
|
1049
|
+
let parked: string | null = null;
|
|
1050
|
+
|
|
1051
|
+
// **The run does not end; it STOPS AND WAITS.** `awaiting_action` is the run's outcome and
|
|
1052
|
+
// `suspended` is the node's — and those are the words the FOLD uses. The scheduler's own event is
|
|
1053
|
+
// called `task.user_action`, which is not what you poll on, and is the mistake this snippet used to
|
|
1054
|
+
// invite by delivering that event to a hand-built stub.
|
|
1055
|
+
const { driver, done } = runtime.start(flow, {}, { onFoldInput: (input) => { inputs.push(input); } });
|
|
1056
|
+
|
|
1057
|
+
// **`done` IS STILL PENDING HERE, and that is the trap this section exists to teach.** Its
|
|
1058
|
+
// settlement rule is terminality and nothing else (ADR 0058 rule 3), and `isTerminal` is
|
|
1059
|
+
// `outcome !== 'running' && outcome !== 'awaiting_action'` — so a run parked on a person is not
|
|
1060
|
+
// terminal and `done` waits for the resolution. **A promise cannot tell you a run has PARKED**,
|
|
1061
|
+
// because parking is not the end; the poll below is for the state `done` is waiting THROUGH, and it
|
|
1062
|
+
// is the one thing here a completion handle does not give you.
|
|
1063
|
+
// **THE DEADLINE, ARMED FIRST — one line, because `done` IS a promise.** There is no library
|
|
1064
|
+
// default and no `StartOptions` entry for one (ADR 0058 rule 8): a default would be wrong for
|
|
1065
|
+
// exactly this run, where the honest wait is *as long as it takes a person*. What a hand-rolled
|
|
1066
|
+
// `new Promise` could not give whoever held it is a way to RACE it. **It is armed before the wait
|
|
1067
|
+
// and not after**, because what `done` is
|
|
1068
|
+
// waiting for is the decision below — a race started once the person has already decided is a race
|
|
1069
|
+
// whose other side has already lost, and it would demonstrate the shape while being unable to fire.
|
|
1070
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
1071
|
+
const nobodyCame = new Promise<never>((_, reject) => {
|
|
1072
|
+
timer = setTimeout(() => reject(new Error('nobody resolved the suspended task in time')), 10_000);
|
|
1073
|
+
});
|
|
1074
|
+
|
|
1075
|
+
// THE PERSON, deciding while `done` is pending. Started, not awaited: it polls the fold for the
|
|
1076
|
+
// park — the state `done` waits THROUGH — and resolves the task, which is what settles `done`.
|
|
1077
|
+
const person = new Promise<void>((resolve, reject) => {
|
|
1078
|
+
const poll = () => {
|
|
1079
|
+
const s = fold(inputs);
|
|
1080
|
+
if (s.outcome !== 'awaiting_action') { setTimeout(poll, 5); return; }
|
|
1081
|
+
parked = Object.keys(s.nodes).find((id) => s.nodes[id].state === 'suspended') ?? null;
|
|
1082
|
+
if (parked === null) { reject(new Error('awaiting_action with no suspended node')); return; }
|
|
1083
|
+
|
|
1084
|
+
// ONE pair of events, ONE record. The scheduler dispatches `task.failed` AND
|
|
1085
|
+
// `task.user_action` carrying the same object; the driver drops the first, whose detail
|
|
1086
|
+
// already reads `user_action`, and records on the second. Read off a REAL pair, not a
|
|
1087
|
+
// hand-delivered one.
|
|
1088
|
+
const records = driver.snapshot().filter((r) => r.taskId === parked);
|
|
1089
|
+
if (records.length !== 1) { reject(new Error(`the event pair produced ${records.length} records`)); return; }
|
|
1090
|
+
|
|
1091
|
+
// The only way anything outside this process changes a run — and it takes THREE arguments.
|
|
1092
|
+
// `actor` is opaque: never interpreted, never authenticated (Q29).
|
|
1093
|
+
const accepted = driver.resolve(parked, 'failed', { actor: 'a-person' });
|
|
1094
|
+
const again = driver.resolve(parked, 'failed', { actor: 'a-person' });
|
|
1095
|
+
if (!accepted || again) { reject(new Error('a second resolution must be refused, not applied twice')); return; }
|
|
1096
|
+
resolve();
|
|
1097
|
+
};
|
|
1098
|
+
poll();
|
|
1099
|
+
});
|
|
1100
|
+
|
|
1101
|
+
// **`done` AND the person, against the clock.** Neither the poll nor the deadline folds anything or
|
|
1102
|
+
// decides what "over" means: the first is the state the run PARKS in and the second is your patience.
|
|
1103
|
+
//
|
|
1104
|
+
// **`person` is inside the race and not awaited after it**, which is two defects in one line and
|
|
1105
|
+
// both were real. Awaited afterwards it is UNBOUNDED — a run that reaches a terminal outcome without
|
|
1106
|
+
// ever parking leaves the poll looping for ever, which is the very shape this section teaches the
|
|
1107
|
+
// counter-measure for; and its three `reject`s reached a promise nothing had attached to yet, so
|
|
1108
|
+
// Node killed the process with `ERR_UNHANDLED_REJECTION` and the snippet failed as a CRASH rather
|
|
1109
|
+
// than as a throw. *Both found by W78's conformity reader, one line past the deadline written to
|
|
1110
|
+
// prevent the first.*
|
|
1111
|
+
const state = await Promise.race([Promise.all([done, person]).then(([s]) => s), nobodyCame]);
|
|
1112
|
+
clearTimeout(timer);
|
|
1113
|
+
await runtime.shutdown();
|
|
1114
|
+
|
|
1115
|
+
// `resolvedAs: 'failed'` took the `on: ['failed']` edge — the only road there for a non-optional
|
|
1116
|
+
// node — so the run ends at `quarantine`. The END node's `outcome` is a label of yours; the RUN's
|
|
1117
|
+
// outcome is the fold's, and it is `completed` because a path reached an end.
|
|
1118
|
+
const ended = Object.values(state.nodes).find((n) => n.nodeId === 'score');
|
|
1119
|
+
console.log(state.outcome, ended?.state, state.pathEnds.map((p) => p.at).join());
|
|
1120
|
+
// 'completed' 'failed' 'quarantine'
|
|
1121
|
+
// The node's own state is `failed`, not `resolved`: `resolvedAs` is what a person DECIDED,
|
|
1122
|
+
// and the node records the terminal state that decision gave it.
|
|
1123
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
1124
|
+
if (parked !== 'r1:score:-:0') throw new Error(`parked at ${parked}`);
|
|
1125
|
+
if (ended?.state !== 'failed') throw new Error(`the node settled ${ended?.state}`);
|
|
1126
|
+
// **WHICH END the run reached is what `resolvedAs` decided**, and until W71's adequacy reader
|
|
1127
|
+
// read this snippet nothing observed it: both roads fold to `completed`, so resolving 'done'
|
|
1128
|
+
// instead of 'failed' changed the road and no assertion moved. `pathEnds` carries the end each
|
|
1129
|
+
// path reached, which is the whole of the paragraph below.
|
|
1130
|
+
if (state.pathEnds.map((p) => p.at).join() !== 'quarantine') {
|
|
1131
|
+
throw new Error(`the run ended at ${state.pathEnds.map((p) => p.at).join()}, not at quarantine`);
|
|
1132
|
+
}
|
|
1133
|
+
```
|
|
1134
|
+
|
|
1135
|
+
`resolve()` returns `false` if the task is not suspended — a rejected command, not a run failure, so
|
|
1136
|
+
you may retry. It does **one** thing: supply an outcome for a dispatch that already happened. It
|
|
1137
|
+
cannot re-run the node and it cannot send it somewhere else. It also returns `false` once the run has
|
|
1138
|
+
been **stopped** by a refusal (section 9), and there retrying can never succeed: check
|
|
1139
|
+
`driver.hasFailed()` to tell the two `false`s apart.
|
|
1140
|
+
|
|
1141
|
+
A resolved node is **indistinguishable** from one that settled that way: same node state, same
|
|
1142
|
+
`result`, same routing.
|
|
1143
|
+
|
|
1144
|
+
> **A suspended run never finishes on its own, and that is correct.** If you are waiting for a run to
|
|
1145
|
+
> reach a terminal outcome, `awaiting_action` is not one — see section 8.
|
|
1146
|
+
|
|
1147
|
+
---
|
|
1148
|
+
|
|
1149
|
+
## 8. Run state
|
|
1150
|
+
|
|
1151
|
+
**Run state is a pure fold of an ordered log.** The driver produces the sequence; `fold` turns it
|
|
1152
|
+
into state and touches nothing else.
|
|
1153
|
+
|
|
1154
|
+
```ts run
|
|
1155
|
+
import { fold, isTerminal, type FoldInput } from 'fyflow-orchestration';
|
|
1156
|
+
|
|
1157
|
+
const settled = (taskId: string, state: 'done' | 'failed' | 'user_action'): FoldInput => ({
|
|
1158
|
+
kind: 'outcome',
|
|
1159
|
+
record: { kind: 'outcome', taskId, terminalState: state, result: null, error: null,
|
|
1160
|
+
attempts: 0, schedulerTimestamp: 1, observedAt: 1, startTime: 1, endTime: 2, executionTime: 1 },
|
|
1161
|
+
});
|
|
1162
|
+
|
|
1163
|
+
const dispatched = (taskId: string): FoldInput => ({ kind: 'dispatched', taskId });
|
|
1164
|
+
const ended = (reason: 'end' | 'no-eligible-edge'): FoldInput =>
|
|
1165
|
+
({ kind: 'pathEnded', reason, at: 'x' });
|
|
1166
|
+
const openJoin: FoldInput =
|
|
1167
|
+
{ kind: 'joinSettled', regionId: 'split', regionPath: '-', joinId: 'collect', fired: false };
|
|
1168
|
+
|
|
1169
|
+
// Each row adds ONE input to the row below it and must outrank it. Read downwards:
|
|
1170
|
+
// this is the precedence table, executed.
|
|
1171
|
+
const rows: [string, FoldInput[]][] = [
|
|
1172
|
+
['completed', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end')]],
|
|
1173
|
+
['failed', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end'),
|
|
1174
|
+
ended('no-eligible-edge')]],
|
|
1175
|
+
['stuck', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end'),
|
|
1176
|
+
ended('no-eligible-edge'), openJoin]],
|
|
1177
|
+
['running', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end'),
|
|
1178
|
+
ended('no-eligible-edge'), openJoin, dispatched('r1:b:-:0')]],
|
|
1179
|
+
['awaiting_action', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end'),
|
|
1180
|
+
ended('no-eligible-edge'), openJoin, dispatched('r1:b:-:0'),
|
|
1181
|
+
dispatched('r1:c:-:0'), settled('r1:c:-:0', 'user_action')]],
|
|
1182
|
+
['stopped', [dispatched('r1:a:-:0'), settled('r1:a:-:0', 'done'), ended('end'),
|
|
1183
|
+
ended('no-eligible-edge'), openJoin, dispatched('r1:b:-:0'),
|
|
1184
|
+
dispatched('r1:c:-:0'), settled('r1:c:-:0', 'user_action'),
|
|
1185
|
+
{ kind: 'stopped', reason: 'DuplicateTaskId: x' }]],
|
|
1186
|
+
];
|
|
1187
|
+
for (const [expected, inputs] of rows) {
|
|
1188
|
+
const got = fold(inputs).outcome;
|
|
1189
|
+
console.log(expected, '->', got);
|
|
1190
|
+
if (got !== expected) throw new Error(`precedence: expected ${expected}, got ${got}`);
|
|
1191
|
+
}
|
|
1192
|
+
|
|
1193
|
+
// `isTerminal` is false for exactly `running` and `awaiting_action`.
|
|
1194
|
+
const terminal = Object.fromEntries(rows.map(([o, i]) => [o, isTerminal(fold(i))]));
|
|
1195
|
+
console.log(terminal); // completed/failed/stuck/stopped true, running/awaiting_action false
|
|
1196
|
+
if (!terminal.stuck) throw new Error('`stuck` is terminal — that is the point of it existing');
|
|
1197
|
+
if (terminal.running || terminal.awaiting_action) throw new Error('a live run is not terminal');
|
|
1198
|
+
if (!terminal.completed || !terminal.failed || !terminal.stopped) {
|
|
1199
|
+
throw new Error('completed, failed and stopped are all terminal');
|
|
1200
|
+
}
|
|
1201
|
+
|
|
1202
|
+
// `completed` means every live path reached an `end` — NOT "no node settled failed".
|
|
1203
|
+
// A failed node that routed on through an `on: ["failed"]` edge completes the run.
|
|
1204
|
+
const routedOn = fold([dispatched('r1:a:-:0'), settled('r1:a:-:0', 'failed'), ended('end')]);
|
|
1205
|
+
console.log(routedOn.outcome); // 'completed'
|
|
1206
|
+
if (routedOn.outcome !== 'completed') throw new Error('a failed node that routed on still completes');
|
|
1207
|
+
```
|
|
1208
|
+
|
|
1209
|
+
In your own program you fold what the driver gives you:
|
|
1210
|
+
|
|
1211
|
+
```ts check
|
|
1212
|
+
import { fold, isTerminal, type Driver } from 'fyflow-orchestration';
|
|
1213
|
+
|
|
1214
|
+
declare const driver: Driver;
|
|
1215
|
+
const state = fold(driver.foldInputs());
|
|
1216
|
+
if (isTerminal(state)) console.log('finished as', state.outcome);
|
|
1217
|
+
```
|
|
1218
|
+
|
|
1219
|
+
### The six outcomes, highest first
|
|
1220
|
+
|
|
1221
|
+
| | means |
|
|
1222
|
+
|---|---|
|
|
1223
|
+
| `stopped` | a refusal terminated the run — section 9 |
|
|
1224
|
+
| `awaiting_action` | a node is suspended and nobody has resolved it |
|
|
1225
|
+
| `running` | a dispatched node has not settled |
|
|
1226
|
+
| **`stuck`** | nothing is live and **nothing ever will be**: a join has not fired and cannot |
|
|
1227
|
+
| `failed` | a path ended with no eligible edge |
|
|
1228
|
+
| `completed` | every live path reached an `end` node |
|
|
1229
|
+
|
|
1230
|
+
**Read the outcome at a quiescent point** — after the driver's turn, not part-way through it. The
|
|
1231
|
+
fold is defined over any prefix, and a prefix taken mid-turn is not a resting state: a settlement is
|
|
1232
|
+
recorded before the routing it causes, so there are instants where nothing is running and the run is
|
|
1233
|
+
not over. If you fold `driver.foldInputs()` after the fact you are always at a quiescent point; if
|
|
1234
|
+
you fold incrementally from `onFoldInput`, read on a `queueMicrotask` — and that microtask is
|
|
1235
|
+
**load-bearing rather than decoration**, which is why `done` asks its own predicate at one
|
|
1236
|
+
([ADR 0058](doc/decisions/0058-runtime-start-returns-a-completion-handle-the-runtime-keeps.md)
|
|
1237
|
+
rule 6). *(That phrase lived in §2's copied block until [W78](doc/work-items/w78-the-run-a-consumer-holds-in-one-line.md)
|
|
1238
|
+
reduced it to `await done`; ADR 0058's Context cites it, so it moved here rather than going out with
|
|
1239
|
+
the block — W78's conformity reader, finding 3.)*
|
|
1240
|
+
|
|
1241
|
+
**`isTerminal(state)` is how you wait for a run without a clock.** It is false for `running` and
|
|
1242
|
+
`awaiting_action` and true for everything else — including `stuck`, which is the point: before
|
|
1243
|
+
`stuck` existed, a permanently hung run folded to `completed` and a waiter reported success.
|
|
1244
|
+
|
|
1245
|
+
**And `done` is that rule applied for you.** `runtime.start` returns `{ driver, done, header }` —
|
|
1246
|
+
`header` is the run's CHECKPOINT header since
|
|
1247
|
+
[ADR 0064](doc/decisions/0064-a-checkpoint-is-the-whole-event-sequence-and-a-header-and-it-is-a-copy.md),
|
|
1248
|
+
and `driver.checkpoint()` hands out the header with every event so far (`spec/run-state.md`
|
|
1249
|
+
section 8); `runtime.resume(flow, checkpoint)` resumes a run from one, returning the same handle
|
|
1250
|
+
([ADR 0065](doc/decisions/0065-a-resume-replays-the-stored-stimuli-refuses-what-it-cannot-regenerate-and-re-dispatches-what-was-in-flight.md), `spec/run-state.md` section 9 — section 18 teaches it, as an executed snippet) — and `done`
|
|
1251
|
+
resolves with the first `RunState` this predicate calls terminal
|
|
1252
|
+
([ADR 0058](doc/decisions/0058-runtime-start-returns-a-completion-handle-the-runtime-keeps.md)). It
|
|
1253
|
+
is the same two functions — the runtime keeps one `RunState`, steps it per input and asks
|
|
1254
|
+
`isTerminal` at a quiescent point — so there is one definition of *over* with one more caller, and
|
|
1255
|
+
nothing above about reading at rest stops being true. **Three things follow, and two of them catch
|
|
1256
|
+
people:**
|
|
1257
|
+
|
|
1258
|
+
- **`done` never rejects.** `failed`, `stuck` and `stopped` are terminal OUTCOMES, not errors, so a
|
|
1259
|
+
run that goes wrong arrives as `state.outcome`. A driver refusal resolves it with `stopped` and
|
|
1260
|
+
`stoppedBy` naming the refusal — section 9.
|
|
1261
|
+
- **`awaiting_action` is NOT terminal, so `done` waits through it.** A run parked on a person stays
|
|
1262
|
+
pending until somebody calls `resolve()`, which is correct and is the trap: if you are waiting for
|
|
1263
|
+
a run and a human is in it, you are waiting for the human. Section 7 is the worked case.
|
|
1264
|
+
- **There is no deadline and there is no option for one**, because once `done` is a promise
|
|
1265
|
+
`Promise.race([done, timeout])` is your one line — and a library default would be wrong for the
|
|
1266
|
+
bullet above.
|
|
1267
|
+
|
|
1268
|
+
---
|
|
1269
|
+
|
|
1270
|
+
## 9. Refusals
|
|
1271
|
+
|
|
1272
|
+
Two kinds, and **the distinction is the stack, not the refusal class** (`OR-R10`). The same class can
|
|
1273
|
+
do either: whether you are on the stack when it is raised is what decides. A refusal raised while
|
|
1274
|
+
*your* call is running is thrown; one raised inside a scheduler event listener is reported, because
|
|
1275
|
+
`EventTarget.dispatchEvent` **discards a listener throw** and there is nobody to catch it.
|
|
1276
|
+
|
|
1277
|
+
**Thrown**, because you are on the stack and can act:
|
|
1278
|
+
|
|
1279
|
+
```ts run
|
|
1280
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1281
|
+
import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
1282
|
+
|
|
1283
|
+
// **THROWN — raised while YOUR call is running.** A registry defect is refused by `createRuntime`,
|
|
1284
|
+
// on your stack, so a `try` catches it and names the entry.
|
|
1285
|
+
let thrown = '';
|
|
1286
|
+
try {
|
|
1287
|
+
createRuntime({
|
|
1288
|
+
workers: { scorer: { url: '../../examples/getting-started/scorer.ts' } },
|
|
1289
|
+
nodeTypes: { Scorer: { worker: 'typo' } }, // no such worker key
|
|
1290
|
+
} as RegistryDocument, {}, { base: import.meta.url });
|
|
1291
|
+
} catch (e) {
|
|
1292
|
+
thrown = (e as Error).constructor.name;
|
|
1293
|
+
}
|
|
1294
|
+
console.log(thrown); // 'WorkerMissing'
|
|
1295
|
+
if (thrown !== 'WorkerMissing') throw new Error(`construction refused with ${thrown}`);
|
|
1296
|
+
|
|
1297
|
+
// **REPORTED — raised inside a scheduler event listener.** Nobody is on the stack to catch it:
|
|
1298
|
+
// `EventTarget.dispatchEvent` DISCARDS a listener throw, so the driver reports it instead and the
|
|
1299
|
+
// run stops. This is the half a `try` around your own call can never see.
|
|
1300
|
+
const registry: RegistryDocument = {
|
|
1301
|
+
workers: { scorer: { url: '../../examples/getting-started/scorer.ts' } },
|
|
1302
|
+
nodeTypes: { Scorer: { worker: 'scorer', inline: true } },
|
|
1303
|
+
};
|
|
1304
|
+
const flow: FlowDefinition = {
|
|
1305
|
+
flowId: 'bad-arity', version: 1, entry: 'first',
|
|
1306
|
+
nodes: [
|
|
1307
|
+
{ id: 'first', kind: 'task', type: 'Scorer', settings: { score: 0.9 }, edges: [{ to: 'each' }] },
|
|
1308
|
+
// `over` names a key of what `first` returned — and `score` is a NUMBER, not a list.
|
|
1309
|
+
{ id: 'each', kind: 'map', over: 'score', branch: 'one' },
|
|
1310
|
+
{ id: 'one', kind: 'task', type: 'Scorer', settings: { score: 1 }, edges: [{ to: 'join' }] },
|
|
1311
|
+
{ id: 'join', kind: 'join', of: 'each', edges: [{ to: 'done' }] },
|
|
1312
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1313
|
+
],
|
|
1314
|
+
};
|
|
1315
|
+
|
|
1316
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
1317
|
+
const reported = await new Promise<{ kind: string; message: string }>((resolve) => {
|
|
1318
|
+
// A DEADLINE resolving to a SENTINEL, so that *nothing was reported* is this snippet's own
|
|
1319
|
+
// failure rather than a hang. W71's adequacy reader made the driver swallow the refusal and
|
|
1320
|
+
// the snippet died as `Detected unsettled top-level await` — a crash is not a detection.
|
|
1321
|
+
const deadline = setTimeout(() => resolve({ kind: 'NOTHING WAS REPORTED', message: '' }), 10_000);
|
|
1322
|
+
runtime.start(flow, {}, { onRunFailure: (f) => { clearTimeout(deadline); resolve({ kind: f.kind, message: f.message }); } });
|
|
1323
|
+
});
|
|
1324
|
+
await runtime.shutdown();
|
|
1325
|
+
|
|
1326
|
+
console.log(reported.kind); // 'UnresolvedMapArity'
|
|
1327
|
+
if (reported.kind !== 'UnresolvedMapArity') throw new Error(`the run reported ${reported.kind}`);
|
|
1328
|
+
// Same library, same kind of mistake, two different roads out — and which road it takes is decided
|
|
1329
|
+
// by whether you were on the stack, not by the class (`OR-R10`).
|
|
1330
|
+
// *(The OTHER way this leg can go wrong — the refusal ESCAPING the listener instead of being
|
|
1331
|
+
// reported — takes the whole process down through `EventTarget`'s own rethrow, so it cannot be
|
|
1332
|
+
// this snippet's assertion whatever the snippet does. The deadline above closes the half that
|
|
1333
|
+
// can be closed: a refusal SWALLOWED in silence.)*
|
|
1334
|
+
// **`MissingJoinPolicy` is NOT reachable on this road**, and it is in the error-class enumeration below: a fan-out
|
|
1335
|
+
// with no policy is refused when YOU build a `Driver`, and `createRuntime` always supplies one
|
|
1336
|
+
// (`live-token` by default), so the runtime cannot raise it. A refusal a road cannot reach is
|
|
1337
|
+
// still a refusal — it belongs to the road in section 12, which is where a caller passes its own
|
|
1338
|
+
// scheduler and its own policy.
|
|
1339
|
+
```
|
|
1340
|
+
|
|
1341
|
+
`InvalidDefinition` carries the violated rule ids on `.rules`, so you can act on them rather than
|
|
1342
|
+
parse a string. `InvalidRunId` is thrown by the **constructor** for a `runId` outside `[A-Za-z0-9_-]{1,64}`
|
|
1343
|
+
(`ID-R2`) — `:` is excluded because it is the task-id delimiter, and excluding it is what makes the
|
|
1344
|
+
task-id parse total.
|
|
1345
|
+
|
|
1346
|
+
**A defect in the REGISTRY's SHAPE is thrown at `new Driver` too, as a NAMED class extending
|
|
1347
|
+
`RegistryError`** — `spec/flow-definition.md` v0.19, section 13 here — each carrying `at`, the entry's
|
|
1348
|
+
key, and a message naming the entry. `new Driver` runs the shape half only, six classes:
|
|
1349
|
+
`WorkerMissing` (a type naming no worker entry, an entry with no `url`), `PoolTypeMissing`,
|
|
1350
|
+
`NameCollision` (a pool named as a type), `SchemaRefused` (a keyword outside the subset), and
|
|
1351
|
+
`UndeclaredKey` and `DerivationRefused` for a type's or a pool's field of the wrong shape. The other
|
|
1352
|
+
five — `ConfigRefused`, `TypeDefaultRefused`, `GroupTypeMissing`, `GroupConfigRefused`,
|
|
1353
|
+
`NameMissing` — need a derivation run or a group built, and are thrown by
|
|
1354
|
+
`resolveRegistry(document, code)` when a registry DOCUMENT is resolved against the code that
|
|
1355
|
+
supplies its predicates, languages, group types and node-type classes, by `createRuntime` (which
|
|
1356
|
+
calls that resolver first — section 12), and by the bootstrap helper before it builds a pool. The
|
|
1357
|
+
twelfth, **`GateTypeRefused`** (v0.23), is thrown by `checkRegistry` when a gate type it holds is not
|
|
1358
|
+
a `GateType` — the built-in `distinct` on every call, and any the registry adds in `gateTypes` — and a gate
|
|
1359
|
+
type named as a built-in is `NameCollision` — at `new Driver`, which since W93 takes yours as `gateTypes`, and
|
|
1360
|
+
at `createRuntime`, which takes them as `code.gateTypes` and hands them to every run (section 3). None is `InvalidDefinition`, on purpose: a registry defect is fixed in the registry and a flow defect in the
|
|
1361
|
+
flow, and the class says which document to open.
|
|
1362
|
+
|
|
1363
|
+
**And four the RUNTIME adds, all `RuntimeError` subclasses from `fyflow-orchestration/scheduler`**
|
|
1364
|
+
(W66): `GroupUnusable` — a group factory returned something the scheduler cannot use, naming the
|
|
1365
|
+
member and the group's key; `GroupNotBuildable` — a group type with no `create`; `LanguageNotBuilt` —
|
|
1366
|
+
`start` was given a flow that transforms on a type no pool was built for; and, since W89, `RunIdInUse`
|
|
1367
|
+
— a `start` or `resume` naming a run id that runtime has already built a driver for, a finished run's included (`RN-R25`, ADR 0091). A bare
|
|
1368
|
+
`RuntimeError` is the fifth road: a relative worker url with no `base`, and a predicate or adapter supplied that the
|
|
1369
|
+
document names nowhere. **They are not exported from the package ROOT**, so the enumeration below —
|
|
1370
|
+
which walks the root's own exports — does not see them, and `BootstrapError` has never been there
|
|
1371
|
+
either.
|
|
1372
|
+
|
|
1373
|
+
**And more that throw, because you called them.** Some are the same classes the next paragraph calls
|
|
1374
|
+
reported, which is why the rule is about the stack:
|
|
1375
|
+
|
|
1376
|
+
| | |
|
|
1377
|
+
|---|---|
|
|
1378
|
+
| `driver.start()` a **second time** | throws `MultipleRuns` — a driver serves one run and its id was fixed at construction (`ID-R3`) |
|
|
1379
|
+
| `driver.resolve()` with a `result` that is not structured-cloneable | throws the platform's own `DataCloneError`, **not** `UncloneableResult`, and the run is not stopped |
|
|
1380
|
+
| `driver.start()` with a payload that is not structured-cloneable | throws the platform's own `DataCloneError` with the driver UNSTARTED — a run whose entry payload cannot be copied into its checkpoint header cannot be checkpointed, so it is refused before it exists (`RN-R15`, [ADR 0064](doc/decisions/0064-a-checkpoint-is-the-whole-event-sequence-and-a-header-and-it-is-a-copy.md)). **New in W88, and BREAKING**: such a payload used to start a run on an inline pool |
|
|
1381
|
+
| `driver.checkpoint()` before `start()` | throws `NotStarted` — there is no run, and no entry payload to put in a header (`RN-R14`) |
|
|
1382
|
+
| `driver.resume(checkpoint)` or `runtime.resume(flow, checkpoint)` with an entry payload that is not structured-cloneable | throws the platform's `DataCloneError` before anything is replayed — the driver unstarted, and on a runtime no run id spent (`RN-R15`) |
|
|
1383
|
+
| `driver.checkpoint()` on a driver whose resume was refused | throws `NotStarted` — a refused resume leaves the driver SPENT and deaf to its scheduler, with no run to checkpoint (`RN-R23`) — **except** a resume refused `stopped` because its replay regenerated a stop in the lost tail: that driver keeps the STOPPED run, and its checkpoint is one a store may take. *(This row said `NotStarted` without the exception until W90 read it against `RN-R23`.)* |
|
|
1384
|
+
| `driver.resume(checkpoint)` or `runtime.resume(flow, checkpoint)` on a checkpoint it will not resume | throws `ResumeRefused`, whose `reason` is `format`, `run`, `flow`, `stopped` or `diverged` and whose `index` names the position for the last two — nothing is handed to a scheduler ([ADR 0065](doc/decisions/0065-a-resume-replays-the-stored-stimuli-refuses-what-it-cannot-regenerate-and-re-dispatches-what-was-in-flight.md)). `resume()` on a driver that already started or resumed throws `MultipleRuns` (`ID-R3`). **On a runtime, a refusal DURING the replay spends the run id there** — a driver was built for it, so that runtime answers the id with `RunIdInUse` from then on (ADR 0091 keeps the id after it releases the driver); one refused by its header spends nothing ([Q371](doc/questions/q371.md), section 18) |
|
|
1385
|
+
|
|
1386
|
+
`MultipleRuns` is thrown by a second `start()`: a driver's sequence and join state carry no run id,
|
|
1387
|
+
so a second run would silently merge into the first. **One driver, one run** — and the run id is a
|
|
1388
|
+
constructor option, so there is no second id to pass.
|
|
1389
|
+
|
|
1390
|
+
**A second driver on the same scheduler is the supported answer**, which is what `MultipleRuns`'
|
|
1391
|
+
message tells you to build — **under one condition.** Give each driver a **different** run id: two
|
|
1392
|
+
sharing one id mint the same task ids, so `dispatched.has()` is true in both, neither can see the
|
|
1393
|
+
collision, and one settlement dispatches the successor twice. Nothing enforces this and nothing can:
|
|
1394
|
+
a driver is handed two scheduler methods and neither asks who else is subscribed.
|
|
1395
|
+
|
|
1396
|
+
**The scheduler may carry work this library did not mint** — a nightly cleanup, another library, a
|
|
1397
|
+
worker's own `context.spawnTask()` children. A task id that does not parse was minted by no driver,
|
|
1398
|
+
so it is **not** any run's: the driver reports it through `onUnmintedTask` and drops it, and no run
|
|
1399
|
+
stops (`ID-R5`). *This used to be the second condition, and it stopped **every** driver on the
|
|
1400
|
+
scheduler; ADR 0027 is where it changed.*
|
|
1401
|
+
|
|
1402
|
+
```ts check
|
|
1403
|
+
import { Driver, type UnmintedTask } from 'fyflow-orchestration';
|
|
1404
|
+
|
|
1405
|
+
declare const opts: ConstructorParameters<typeof Driver>[1];
|
|
1406
|
+
const watched = {
|
|
1407
|
+
...opts,
|
|
1408
|
+
// Optional: without it the run still routes correctly, because a settlement that
|
|
1409
|
+
// parses as nothing was never yours. Register it if you do not control everything
|
|
1410
|
+
// on the scheduler — it is the only thing that would tell you the ids stopped
|
|
1411
|
+
// being the ones you dispatched.
|
|
1412
|
+
onUnmintedTask: (t: UnmintedTask) =>
|
|
1413
|
+
console.warn(`run ${t.runId}: ${t.event} for ${String(t.taskId)} — ${t.reason}`),
|
|
1414
|
+
};
|
|
1415
|
+
void watched;
|
|
1416
|
+
```
|
|
1417
|
+
|
|
1418
|
+
**Do not let this callback throw.** It runs inside the driver's scheduler listener, so a throw from it
|
|
1419
|
+
is caught at that boundary and **stops your run** — under your own error class, for a task that was
|
|
1420
|
+
never yours, which is the exact damage the rule exists to remove. Log and return.
|
|
1421
|
+
**A settlement reaches only the driver that dispatched that task id.** One
|
|
1422
|
+
for another run is ignored, and one carrying **this** driver's run id for a task it never
|
|
1423
|
+
dispatched stops the run with `UndispatchedTask` (`ID-R4`). The second half is a refusal rather than
|
|
1424
|
+
a filter for `ID-R1`'s reason — within one run the driver mints every id it dispatches, so a
|
|
1425
|
+
settlement it did not mint means its bookkeeping is wrong.
|
|
1426
|
+
|
|
1427
|
+
**Reported, not thrown**, when there is nobody on the stack: `DuplicateTaskId`, `UncloneableResult`,
|
|
1428
|
+
`NoMatchingEdge` and `UndispatchedTask` raised *inside a scheduler event listener*. Give the driver
|
|
1429
|
+
`onRunFailure` or you will not hear about them — the run stops and folds to `stopped`, silently.
|
|
1430
|
+
|
|
1431
|
+
**`InvalidTaskId` is the ninth class and it is the one that does NOT stop a run.** It is thrown by
|
|
1432
|
+
`buildTaskId` and `parseTaskId`, both exported, and it is how you check an id yourself — *did some
|
|
1433
|
+
driver mint this?* is a `try`/`catch` around `parseTaskId`. Inside the driver's listener it is caught
|
|
1434
|
+
rather than reported: the settlement goes to `onUnmintedTask` and the run carries on (`ID-R5`), so
|
|
1435
|
+
**it never reaches `onRunFailure` and is never a `RunFailure.kind`.**
|
|
1436
|
+
|
|
1437
|
+
```ts run
|
|
1438
|
+
import { InvalidTaskId, parseTaskId } from 'fyflow-orchestration';
|
|
1439
|
+
|
|
1440
|
+
let mine = false;
|
|
1441
|
+
let foreignFailedAsInvalidTaskId = false;
|
|
1442
|
+
try { parseTaskId('run-1:ingest:-:0'); mine = true; } catch { /* not ours */ }
|
|
1443
|
+
try { parseTaskId('nightly-cleanup'); } catch (e) { foreignFailedAsInvalidTaskId = e instanceof InvalidTaskId; }
|
|
1444
|
+
console.log(mine, foreignFailedAsInvalidTaskId);
|
|
1445
|
+
if (!mine) throw new Error('a four-segment id in the charset must parse');
|
|
1446
|
+
if (!foreignFailedAsInvalidTaskId) throw new Error('a foreign task id must fail, and as InvalidTaskId');
|
|
1447
|
+
```
|
|
1448
|
+
|
|
1449
|
+
*Written without a rethrow on purpose.* The obvious form of this — a `mintedByADriver(id)` helper
|
|
1450
|
+
whose `catch` rethrows anything that is not an `InvalidTaskId` — is the shape the driver itself uses,
|
|
1451
|
+
and **W18 shipped it and then measured that its assertions could all be deleted with the gate
|
|
1452
|
+
green**: `check-docs`' `X4a` asks whether a `run` snippet contains a `throw`, and the helper's own
|
|
1453
|
+
rethrow answered yes on the assertions' behalf. The general weakness is
|
|
1454
|
+
[Q104](doc/questions/q104.md); here the snippet simply does not give `X4a` a `throw` that is not
|
|
1455
|
+
an assertion.
|
|
1456
|
+
|
|
1457
|
+
```ts check
|
|
1458
|
+
import { Driver, type RunFailure } from 'fyflow-orchestration';
|
|
1459
|
+
|
|
1460
|
+
declare const opts: ConstructorParameters<typeof Driver>[1];
|
|
1461
|
+
const withReporting = {
|
|
1462
|
+
...opts,
|
|
1463
|
+
onRunFailure: (f: RunFailure) => console.error(`run ${f.runId} stopped: ${f.kind} — ${f.message}`),
|
|
1464
|
+
};
|
|
1465
|
+
void withReporting;
|
|
1466
|
+
```
|
|
1467
|
+
|
|
1468
|
+
`f.kind` is the **class name**, so these are the strings you switch on. Imported here rather than
|
|
1469
|
+
quoted in prose, so renaming one in `src/` fails this gate instead of silently falsifying the
|
|
1470
|
+
paragraphs above.
|
|
1471
|
+
|
|
1472
|
+
**Two of the classes belong to `map`, and they fail for different people.**
|
|
1473
|
+
|
|
1474
|
+
`UnresolvedBranchPath` is `ID-R6` — `branchPathOf` was asked for the path of a node inside a **`map`**
|
|
1475
|
+
region without the item index that region's branch is indexed by, and refused rather than guessing,
|
|
1476
|
+
because guessing mints one task id for every dispatch and the run then dies at `ID-R1` naming a
|
|
1477
|
+
duplicate. **A `Driver` supplies that index and never sees this**; it is what a caller composing
|
|
1478
|
+
`branchPathOf` or `branchIndexOf` themselves gets, and it names the map rather than the symptom.
|
|
1479
|
+
|
|
1480
|
+
`UnresolvedMapArity` is **W33's, and it is a data error rather than a definition error** — which is
|
|
1481
|
+
why it is not an `InvalidDefinition`. A `map` node's `over` is a key NAME (`FD-V25` checks the
|
|
1482
|
+
charset and can check nothing else, because no validator can see a value that does not exist yet),
|
|
1483
|
+
and at run time the value it names must be an **array**. An absent key, an object, a number, a `null`
|
|
1484
|
+
payload or a string payload each stop the run with this class naming the node and the key. **The
|
|
1485
|
+
definition is valid and the previous node did not produce what the flow says it does**, which is a
|
|
1486
|
+
different person's bug — a flow author reading `InvalidDefinition` would go and read their flow,
|
|
1487
|
+
where there is nothing to find. **An empty array is not this**: zero items is a lawful arity.
|
|
1488
|
+
|
|
1489
|
+
`UnresolvedReference` is **W41's**, and it is [ADR 0040](doc/decisions/0040-what-a-node-can-reach-is-the-runs-own-outputs-and-a-node-id-names-them-relative-to-a-branch-path.md)'s
|
|
1490
|
+
`FD-R3`. **A node OR AN EDGE** declared a `needs` naming another node — the edge site is
|
|
1491
|
+
[ADR 0041](doc/decisions/0041-a-decision-sees-its-nodes-outcome-and-whatever-its-own-edge-declares.md)'s
|
|
1492
|
+
and means **a predicate can now stop a run**, with the message saying `edge needs[…]` so you are sent
|
|
1493
|
+
to the edge — and **no output of that node is on this
|
|
1494
|
+
node's branch path** — a decision routed around it, or it settled `user_action` and nobody has
|
|
1495
|
+
resolved it. `FD-V27` has already checked at instantiation that the declaring node is *reachable
|
|
1496
|
+
from* the named one, which is everything the graph can say; whether it actually ran is a fact about
|
|
1497
|
+
this run. **It is a different class from `UnresolvedMapArity` deliberately**: that one says *the
|
|
1498
|
+
previous node did not return an array*, which is a **worker's** bug, and this says *the node you
|
|
1499
|
+
named did not run on your path*, which is the **flow's** shape.
|
|
1500
|
+
|
|
1501
|
+
`BoundaryRefused` is **W73's, and it is the one class here that STOPS NOTHING** — [P19](doc/plans/p19-what-a-schema-buys-an-author.md) decision 5's `FD-R4`. The two values no static check can reach are held to the workers' declared
|
|
1502
|
+
schemas ONCE each: the run's payload at `start()`, against the `inputSchema` of every task it reaches with no
|
|
1503
|
+
worker in between (the entry, or every branch task of an entry fan-out, or a map's branch once per item),
|
|
1504
|
+
and a person's `done` resolution at `resolve()`, against the parked node's worker `outputSchema`. Refused, it is
|
|
1505
|
+
**thrown at you as a REJECTED COMMAND**: nothing appended, nothing reported, the run unaffected — at `start`
|
|
1506
|
+
the driver is still UNSTARTED and a corrected payload starts it, at `resolve` the task is still suspended and
|
|
1507
|
+
a corrected value (the worker's declared example is one) resolves it. It carries `boundary`, `nodeId`,
|
|
1508
|
+
`taskId` and the schema's `findings`, and its message names the worker and every finding, so you read
|
|
1509
|
+
the VALUE and not the flow. A worker declaring no schema on that side is unchecked; `validateBoundaries: false`
|
|
1510
|
+
on `DriverOptions` or `RuntimeOptions` turns both off. **`FD-R4` is never per settlement**: it holds the run's PAYLOAD once and a person's RESOLUTION
|
|
1511
|
+
once, as rejected commands. What a WORKER returns is checked at authoring time by
|
|
1512
|
+
`npm run check:workers` and by the wiring check (section 13) — **and, since
|
|
1513
|
+
[ADR 0060](doc/decisions/0060-a-run-time-result-check-is-a-reading-and-never-a-refusal.md), by the
|
|
1514
|
+
RUN as well if you ask it to**: `OR-R13`, section 16, opt-in and a READING rather than a refusal.
|
|
1515
|
+
*(This read *"not by the run"*, which W80 falsified and which stood four sections above the section
|
|
1516
|
+
that teaches the opposite — W82's conformity reader, finding 2.)*
|
|
1517
|
+
|
|
1518
|
+
*(This paragraph read *"`UnresolvedBranchPath` is the tenth class, and it belongs to a feature nothing
|
|
1519
|
+
dispatches yet … no `Driver` opens one"* until W33, which is the item that made it run.
|
|
1520
|
+
[Q157](doc/questions/q157.md) is the gap behind both edits: no
|
|
1521
|
+
[P10](doc/plans/p10-is-fan-out-one-feature-or-two.md) item owns this file, and W35 is the natural
|
|
1522
|
+
owner of describing the feature rather than merely not lying about it.)*
|
|
1523
|
+
|
|
1524
|
+
**And the list is checked against the PACKAGE, not against itself** — [Q101](doc/questions/q101.md).
|
|
1525
|
+
Until W18 this snippet imported eight classes, mapped them to `.name`, and compared the result to a
|
|
1526
|
+
literal it also contained. That is a closed list agreeing with a copy of itself: it was executed by
|
|
1527
|
+
`npm run gate`, it **omitted `InvalidTaskId`** — exported from the package root the whole time — and
|
|
1528
|
+
it passed. *Rot is what the gate prevents; a list that never looks outside itself is a third thing the
|
|
1529
|
+
gate cannot see.* The second half below is what makes adding a refusal class in `src/` and not
|
|
1530
|
+
mentioning it here a **failure** rather than an omission nobody can observe:
|
|
1531
|
+
|
|
1532
|
+
```ts run
|
|
1533
|
+
import * as lib from 'fyflow-orchestration';
|
|
1534
|
+
import {
|
|
1535
|
+
BoundaryRefused,
|
|
1536
|
+
ConfigRefused, DerivationRefused, DuplicateTaskId, GateTypeRefused, GroupConfigRefused, GroupTypeMissing, InvalidDefinition,
|
|
1537
|
+
InvalidRunId, InvalidTaskId, MissingJoinPolicy, MultipleRuns, NameCollision, NameMissing, NoMatchingEdge, NotStarted,
|
|
1538
|
+
PoolTypeMissing, RegistryError, ResumeRefused, SchemaRefused, TypeDefaultRefused, UncloneableResult, UndeclaredKey,
|
|
1539
|
+
UndeclaredRoute, UndispatchedTask, UnresolvedBranchPath, UnresolvedMapArity, UnresolvedReference, WorkerMissing,
|
|
1540
|
+
} from 'fyflow-orchestration';
|
|
1541
|
+
|
|
1542
|
+
const documented = [
|
|
1543
|
+
DuplicateTaskId, UncloneableResult, NoMatchingEdge, UndispatchedTask, // reported inside a listener
|
|
1544
|
+
MultipleRuns, InvalidRunId, InvalidDefinition, MissingJoinPolicy, // thrown at you
|
|
1545
|
+
NotStarted, // thrown at you from checkpoint() before start() — there is no run yet (RN-R14)
|
|
1546
|
+
ResumeRefused, // thrown at you from resume() — a checkpoint it will not resume, `reason` says why (RN-R23)
|
|
1547
|
+
RegistryError, WorkerMissing, PoolTypeMissing, ConfigRefused, TypeDefaultRefused, GroupTypeMissing, // thrown at you — a REGISTRY defect,
|
|
1548
|
+
GroupConfigRefused, NameCollision, SchemaRefused, UndeclaredKey, NameMissing, DerivationRefused, // the base class and its twelve (section 13)
|
|
1549
|
+
GateTypeRefused, // a registered GATE type that is not a GateType — checkRegistry (v0.23)
|
|
1550
|
+
UnresolvedMapArity, UnresolvedReference, // reported in a listener; reported AND thrown from start()/resolve()
|
|
1551
|
+
UndeclaredRoute, // the same two roads — a GATE's type returned a route it does not declare (FD-R5, W93)
|
|
1552
|
+
BoundaryRefused, // thrown at you from start()/resolve() with the run UNTOUCHED — a rejected command (FD-R4)
|
|
1553
|
+
InvalidTaskId, UnresolvedBranchPath, // neither — see above
|
|
1554
|
+
].map((c) => c.name);
|
|
1555
|
+
|
|
1556
|
+
// Every Error subclass the package root exports, discovered rather than restated.
|
|
1557
|
+
const exported = Object.entries(lib)
|
|
1558
|
+
.filter(([, v]) => typeof v === 'function' && Error.isPrototypeOf(v as object))
|
|
1559
|
+
.map(([name]) => name);
|
|
1560
|
+
|
|
1561
|
+
const missing = exported.filter((n) => !documented.includes(n));
|
|
1562
|
+
const stale = documented.filter((n) => !exported.includes(n));
|
|
1563
|
+
console.log(documented.length, exported.length);
|
|
1564
|
+
if (missing.length) throw new Error(`section 9 does not mention: ${missing.join(' ')}`);
|
|
1565
|
+
if (stale.length) throw new Error(`section 9 names classes the package no longer exports: ${stale.join(' ')}`);
|
|
1566
|
+
```
|
|
1567
|
+
|
|
1568
|
+
After a refusal the run is **stopped**: nothing routes for it again, and its state folds to
|
|
1569
|
+
`stopped` whatever its nodes say.
|
|
1570
|
+
|
|
1571
|
+
---
|
|
1572
|
+
|
|
1573
|
+
## 10. Common mistakes
|
|
1574
|
+
|
|
1575
|
+
- **Expecting a failure to route by default.** It does not. Add `on: ['failed']`, or accept that the
|
|
1576
|
+
path ends — section 6.
|
|
1577
|
+
- **Using `optional: false` to mean "important".** It means *"a human decides"*, and it produces a
|
|
1578
|
+
suspended run rather than a failed one.
|
|
1579
|
+
- **Passing `arrivalCount`.** It hangs on an ordinary `optional` failure. Section 5.
|
|
1580
|
+
- **Waiting on `scheduler.completed`.** It fires **every time the scheduler drains**, which with a
|
|
1581
|
+
flow is between every step. It is not "the run finished". `await done` — the second half of what
|
|
1582
|
+
`runtime.start` returns (section 8).
|
|
1583
|
+
- **Expecting `await done` to reject when a run goes wrong.** It resolves, always: `failed`, `stuck`
|
|
1584
|
+
and `stopped` are outcomes you read off the state, and `stoppedBy` names a refusal. A `.catch` on
|
|
1585
|
+
`done` is a `.catch` that never fires.
|
|
1586
|
+
- **Waiting on `done` for a run with a person in it.** `awaiting_action` is not terminal, so `done`
|
|
1587
|
+
is still pending — by design. Poll the fold for the park, resolve the task, then await; section 7
|
|
1588
|
+
is that snippet, and `Promise.race([done, timeout])` is the deadline nothing ships for you.
|
|
1589
|
+
- **Reading run state mid-turn.** Section 8.
|
|
1590
|
+
- **Retaining a scheduler event's `detail`.** It is the live task object and it mutates — including
|
|
1591
|
+
its `timestamp`, so a retained detail reports the *most recent* event's time. `{...e.detail}` is
|
|
1592
|
+
**not** compliance: it still aliases `result` and `error`. The driver copies at the listener
|
|
1593
|
+
boundary, so the four values that carry the run — records from `snapshot()`, inputs from
|
|
1594
|
+
`foldInputs()` and `onFoldInput`, and events from `onRegionEvent` — are **frozen copies**, safe to
|
|
1595
|
+
hold. *Until [W76](doc/work-items/w76-what-start-accepts.md) the fourth of those was reachable
|
|
1596
|
+
only by building a `Driver` by hand, so this bullet named four values and the snippet below could
|
|
1597
|
+
assert three; `createRuntime` forwards `onRegionEvent` now, and all four are on one road.*
|
|
1598
|
+
**The other accessors return copies that are not frozen** (`dispatchedIds()`,
|
|
1599
|
+
`regionsForPolicies()`, `effectiveOutcomeOf()`, `terminalStateOf()`): writing to one changes
|
|
1600
|
+
nothing in the driver, but it is not a document to edit and hand back.
|
|
1601
|
+
- **Returning a CLASS INSTANCE from a worker.** It does not throw and it does not arrive intact.
|
|
1602
|
+
`structuredClone` copies the fields and drops the prototype, so the next node receives a plain
|
|
1603
|
+
object: `instanceof` is false, the methods are gone, and **nothing is reported** — `OR-R11` in
|
|
1604
|
+
`spec/outcome-record.md` section 4. **Return data, read data.** The loud half is a result that
|
|
1605
|
+
cannot be cloned at all (a function, a `Promise`, a `WeakMap`), which on an **inline** pool stops
|
|
1606
|
+
the run with `UncloneableResult`, and on a **thread** pool is refused first by the scheduler's
|
|
1607
|
+
own clone, so the value never reaches the driver, `UncloneableResult` never fires and
|
|
1608
|
+
`onRunFailure` is not called — the task ends as any failure does: a non-optional task out of
|
|
1609
|
+
retries ends `user_action` and parks the run `awaiting_action`, and an `optional` one ends
|
|
1610
|
+
`failed` and routes as a failure (`OR-R6`; the node type's `inline` decides which pool); the class
|
|
1611
|
+
instance is the quiet half, and it is the one people hit.
|
|
1612
|
+
- **Assuming `joinStatusOf().describe` is yours.** It is whatever **your** policy's `describe()`
|
|
1613
|
+
returned. The two shipped policies return a fresh object; a policy that returns its own state hands
|
|
1614
|
+
you that state, and writing to it changes what the join decides. Return a copy from `describe()`
|
|
1615
|
+
— [Q82](doc/questions/q82.md).
|
|
1616
|
+
|
|
1617
|
+
The first half of that is executed rather than asserted:
|
|
1618
|
+
|
|
1619
|
+
```ts run
|
|
1620
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1621
|
+
import { type FlowDefinition, type FoldInput, type RegistryDocument } from 'fyflow-orchestration';
|
|
1622
|
+
|
|
1623
|
+
const registry: RegistryDocument = {
|
|
1624
|
+
workers: { scorer: { url: '../../examples/getting-started/scorer.ts' } },
|
|
1625
|
+
nodeTypes: { Scorer: { worker: 'scorer', inline: true } },
|
|
1626
|
+
};
|
|
1627
|
+
const flow: FlowDefinition = {
|
|
1628
|
+
flowId: 'frozen', version: 1, entry: 'a',
|
|
1629
|
+
nodes: [
|
|
1630
|
+
{ id: 'a', kind: 'task', type: 'Scorer', settings: { score: 0.1 }, edges: [{ to: 'split' }] },
|
|
1631
|
+
{ id: 'split', kind: 'fanout', branches: ['b', 'c'] },
|
|
1632
|
+
{ id: 'b', kind: 'task', type: 'Scorer', settings: { score: 0.2 }, edges: [{ to: 'j' }] },
|
|
1633
|
+
{ id: 'c', kind: 'task', type: 'Scorer', settings: { score: 0.3 }, edges: [{ to: 'j' }] },
|
|
1634
|
+
{ id: 'j', kind: 'join', of: 'split', edges: [{ to: 'e' }] },
|
|
1635
|
+
{ id: 'e', kind: 'end', outcome: 'completed' },
|
|
1636
|
+
],
|
|
1637
|
+
};
|
|
1638
|
+
|
|
1639
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
1640
|
+
const inputs: FoldInput[] = [];
|
|
1641
|
+
const frozen: Record<string, boolean> = {};
|
|
1642
|
+
// **The handles this road HAS, named rather than counted** — a count is not a membership, and
|
|
1643
|
+
// re-sourcing one of these onto another left a count of three unmoved (W71's adequacy reader).
|
|
1644
|
+
const HANDLES = ['onFoldInput', 'onRegionEvent', 'snapshot', 'foldInputs'];
|
|
1645
|
+
|
|
1646
|
+
// **`onRegionEvent` is FORWARDED, so this road can watch a fan-out open and close** — W76. It was
|
|
1647
|
+
// not, until W76: `StartOptions` is a `Pick` of the driver's options, and of the driver's twenty
|
|
1648
|
+
// (eighteen when W76 wrote this; `onNonconformingResult` is ADR 0060's, W80's; `gateTypes` ADR 0067's, W93's)
|
|
1649
|
+
// this was the one no `createRuntime` consumer could reach at all, which is
|
|
1650
|
+
// [Q297](doc/questions/q297.md). **The previous version of this snippet asserted the opposite and
|
|
1651
|
+
// passed a CAST**, because that was the only way to try; there is no cast now, and the two counts
|
|
1652
|
+
// below are what changed. *The reason it went unnoticed for so long is that the snippet which
|
|
1653
|
+
// asserted it built its own `Driver`, and a reference that constructs its own driver can
|
|
1654
|
+
// demonstrate options its readers have no way to pass.*
|
|
1655
|
+
//
|
|
1656
|
+
// **What a consumer gets is the driver's own event, already frozen** (`OR-R7`,
|
|
1657
|
+
// [Q73](doc/questions/q73.md)) — nothing is copied or re-frozen at the runtime seam, which is why
|
|
1658
|
+
// the freeze is asserted here on the road a consumer actually uses rather than inferred from the
|
|
1659
|
+
// driver's own test.
|
|
1660
|
+
let regionEvents = 0;
|
|
1661
|
+
const regionKinds: string[] = [];
|
|
1662
|
+
const { driver, done } = runtime.start(flow, {}, {
|
|
1663
|
+
onFoldInput: (i: FoldInput) => { frozen.onFoldInput = Object.isFrozen(i); inputs.push(i); },
|
|
1664
|
+
onRegionEvent: (e) => {
|
|
1665
|
+
regionEvents += 1;
|
|
1666
|
+
regionKinds.push(e.type);
|
|
1667
|
+
frozen.onRegionEvent = (frozen.onRegionEvent ?? true) && Object.isFrozen(e);
|
|
1668
|
+
},
|
|
1669
|
+
});
|
|
1670
|
+
|
|
1671
|
+
// **The end, and its deadline, in two lines** — `await done` is the wait (ADR 0058) and
|
|
1672
|
+
// `Promise.race` is the deadline (rule 8). `onFoldInput` above is still here because THIS snippet
|
|
1673
|
+
// is about what the four handles hand out, and one of the four IS that callback.
|
|
1674
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
1675
|
+
const tooLong = new Promise<never>((_, reject) => {
|
|
1676
|
+
timer = setTimeout(() => reject(new Error('the run never reached a terminal state')), 10_000);
|
|
1677
|
+
});
|
|
1678
|
+
await Promise.race([done, tooLong]);
|
|
1679
|
+
clearTimeout(timer);
|
|
1680
|
+
await runtime.shutdown();
|
|
1681
|
+
|
|
1682
|
+
// The two PULL handles, read after the run — and each must have handed back something, because
|
|
1683
|
+
// `Object.isFrozen([])` is true for free and an empty answer would pass while asserting nothing.
|
|
1684
|
+
const snapshot = driver.snapshot();
|
|
1685
|
+
const foldInputs = driver.foldInputs();
|
|
1686
|
+
if (snapshot.length === 0 || foldInputs.length === 0) throw new Error('a pull handle returned nothing, so freezing it says nothing');
|
|
1687
|
+
frozen.snapshot = Object.isFrozen(snapshot) && snapshot.every(Object.isFrozen);
|
|
1688
|
+
frozen.foldInputs = Object.isFrozen(foldInputs) && foldInputs.every(Object.isFrozen);
|
|
1689
|
+
|
|
1690
|
+
console.log(frozen, regionEvents, [...new Set(regionKinds)].sort());
|
|
1691
|
+
// all four handles true, then 4, then [ 'branch.arrived', 'branch.opened' ]
|
|
1692
|
+
|
|
1693
|
+
// **Where each claim below is observed a SECOND time**, because every guard in an executed snippet
|
|
1694
|
+
// is bound by its own PRESENCE and by nothing else ([Q310](doc/questions/q310.md)) — W76's adequacy
|
|
1695
|
+
// reader made all six guards and all four `isFrozen` readings vacuous at once with `check:docs` at
|
|
1696
|
+
// exit 0. The four freeze claims are re-asserted per element in `test/driver/outbound.test.ts` and
|
|
1697
|
+
// `test/driver/fold-inputs.test.ts`; the region-event count, kinds and freeze ACROSS THE RUNTIME
|
|
1698
|
+
// SEAM — which is the part those files cannot see, because they build a `Driver` by hand — are in
|
|
1699
|
+
// `test/scheduler/runtime.test.ts`, where the forwarded stream is held equal to the driver's own
|
|
1700
|
+
// region stream element for element. *Nothing here is the only observer of anything.*
|
|
1701
|
+
for (const [name, isFrozen] of Object.entries(frozen)) {
|
|
1702
|
+
if (!isFrozen) throw new Error(`${name} handed out something you can write to`);
|
|
1703
|
+
}
|
|
1704
|
+
// **The guard that fires if the answer changes BACK, and it is FIRST on purpose.** Q297 was a
|
|
1705
|
+
// member of `StartOptions` going missing, which nothing noticed for two plans; this is what would
|
|
1706
|
+
// notice now, from a consumer's side of the seam rather than from the type. *It sat BELOW the
|
|
1707
|
+
// membership check in the first version and was unreachable: `frozen.onRegionEvent` is written in
|
|
1708
|
+
// the same callback as `regionEvents`, so a regression removes the key too and the membership
|
|
1709
|
+
// throws first — a regression caught, by a message about `HANDLES` that says nothing about Q297.
|
|
1710
|
+
// W76's conformity reader; the tree's own "right verdict by the wrong route".*
|
|
1711
|
+
if (regionEvents === 0) {
|
|
1712
|
+
throw new Error('onRegionEvent fired 0 times: the runtime has stopped forwarding it, and a consumer on this road cannot watch a fan-out again — Q297 has regressed');
|
|
1713
|
+
}
|
|
1714
|
+
if (Object.keys(frozen).sort().join() !== HANDLES.slice().sort().join()) {
|
|
1715
|
+
throw new Error(`expected exactly ${HANDLES.join()}, saw ${Object.keys(frozen).sort().join()}`);
|
|
1716
|
+
}
|
|
1717
|
+
// The KINDS are the membership and the count stands beside them: a two-branch fan-out opens twice
|
|
1718
|
+
// and arrives twice, and a snippet asserting only `> 0` would pass on one branch of one kind.
|
|
1719
|
+
if ([...new Set(regionKinds)].sort().join() !== 'branch.arrived,branch.opened') {
|
|
1720
|
+
throw new Error(`expected a fan-out's opens and arrivals, saw ${[...new Set(regionKinds)].sort().join() || '(nothing)'}`);
|
|
1721
|
+
}
|
|
1722
|
+
if (regionEvents !== 4) throw new Error(`expected 4 region events on a two-branch fan-out — two opened, two arrived — and saw ${regionEvents}`);
|
|
1723
|
+
```
|
|
1724
|
+
- **Subscribing to the scheduler yourself and counting `task.failed`.** A `user_action` task fires
|
|
1725
|
+
it *and* `task.user_action`. Key on the terminal state — section 7.
|
|
1726
|
+
- **Sharing one `Driver` between runs.** `start()` is called at most once and a second throws
|
|
1727
|
+
`MultipleRuns` — section 9. *Sharing one **scheduler** between `Driver`s is fine*, and used to be
|
|
1728
|
+
the other half of this bullet: a settlement now reaches only the driver that dispatched it.
|
|
1729
|
+
|
|
1730
|
+
---
|
|
1731
|
+
|
|
1732
|
+
**`OR-R11`, executed rather than asserted** — a worker returns a class with a method, and what
|
|
1733
|
+
the driver logs is a plain object with the field and no method:
|
|
1734
|
+
|
|
1735
|
+
```ts run
|
|
1736
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1737
|
+
import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
1738
|
+
|
|
1739
|
+
// `classy` returns `new Score(0.9)` — a class with a method, and it CHECKS that it does before
|
|
1740
|
+
// handing it over, so the thing being lost below is known to have existed. `inspect` is the NEXT
|
|
1741
|
+
// NODE, and reports what arrived. `uncloneable` returns a function, which is the loud half.
|
|
1742
|
+
const registry: RegistryDocument = {
|
|
1743
|
+
workers: {
|
|
1744
|
+
classy: { url: '../../examples/getting-started/classy.ts' },
|
|
1745
|
+
inspect: { url: '../../examples/getting-started/inspect.ts' },
|
|
1746
|
+
uncloneable: { url: '../../examples/getting-started/uncloneable.ts' },
|
|
1747
|
+
},
|
|
1748
|
+
nodeTypes: {
|
|
1749
|
+
Classy: { worker: 'classy', inline: true },
|
|
1750
|
+
Inspect: { worker: 'inspect', inline: true },
|
|
1751
|
+
Uncloneable: { worker: 'uncloneable', inline: true },
|
|
1752
|
+
},
|
|
1753
|
+
};
|
|
1754
|
+
const lossy: FlowDefinition = {
|
|
1755
|
+
flowId: 'or-r11', version: 1, entry: 'a',
|
|
1756
|
+
nodes: [
|
|
1757
|
+
{ id: 'a', kind: 'task', type: 'Classy', settings: { value: 0.9 }, edges: [{ to: 'next' }] },
|
|
1758
|
+
{ id: 'next', kind: 'task', type: 'Inspect', settings: {}, edges: [{ to: 'done' }] },
|
|
1759
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1760
|
+
],
|
|
1761
|
+
};
|
|
1762
|
+
const loud: FlowDefinition = {
|
|
1763
|
+
flowId: 'uncloneable', version: 1, entry: 'a',
|
|
1764
|
+
nodes: [
|
|
1765
|
+
{ id: 'a', kind: 'task', type: 'Uncloneable', settings: {}, edges: [{ to: 'done' }] },
|
|
1766
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1767
|
+
],
|
|
1768
|
+
};
|
|
1769
|
+
|
|
1770
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
1771
|
+
const failures: string[] = [];
|
|
1772
|
+
// `onRunFailure` is what the LOUD half below is read from; `done` is the end. They are two
|
|
1773
|
+
// different facts about the same run, which is why both are here.
|
|
1774
|
+
const run = (flow: FlowDefinition) =>
|
|
1775
|
+
runtime.start(flow, {}, { onRunFailure: (f) => { failures.push(f.kind); } }).done;
|
|
1776
|
+
|
|
1777
|
+
// **THE QUIET HALF.** The fields cross; the behaviour does not; nothing is reported.
|
|
1778
|
+
const quiet = await run(lossy);
|
|
1779
|
+
const arrived = Object.values(quiet.nodes).find((n) => n.nodeId === 'next')?.result as
|
|
1780
|
+
{ sawMethod?: string; prototypeIs?: string; keys?: string[] };
|
|
1781
|
+
console.log(arrived.sawMethod, arrived.prototypeIs, JSON.stringify(arrived.keys));
|
|
1782
|
+
// 'undefined' 'Object.prototype' ["value"]
|
|
1783
|
+
if (arrived.keys?.join() !== 'value') throw new Error(`the next node received ${JSON.stringify(arrived.keys)}`);
|
|
1784
|
+
if (arrived.sawMethod !== 'undefined') throw new Error('the BEHAVIOUR reached the next node, which OR-R11 says it cannot');
|
|
1785
|
+
if (arrived.prototypeIs !== 'Object.prototype') throw new Error('the PROTOTYPE survived the boundary');
|
|
1786
|
+
if (quiet.outcome !== 'completed') throw new Error(`the lossy run ${quiet.outcome}`);
|
|
1787
|
+
if (failures.length !== 0) throw new Error(`the quiet half reported ${failures.join()} — OR-R11 is silent by design`);
|
|
1788
|
+
|
|
1789
|
+
// **THE LOUD HALF, and it is what makes the line above mean anything.** A result that cannot be
|
|
1790
|
+
// cloned at ALL stops the run and IS reported — so `failures` has a case that fills it, and
|
|
1791
|
+
// `failures.length === 0` above is a measurement rather than a sentence about an empty array.
|
|
1792
|
+
//
|
|
1793
|
+
// **And `done` RESOLVES here rather than rejecting** (ADR 0058 rules 3 and 4): a refusal is a
|
|
1794
|
+
// terminal OUTCOME, not an error, so there is no `.catch` — the run arrives as `stopped` with
|
|
1795
|
+
// `stoppedBy` naming the refusal. One settlement rule for every way a run can end.
|
|
1796
|
+
const stoppedRun = await run(loud);
|
|
1797
|
+
await runtime.shutdown();
|
|
1798
|
+
console.log(failures.join(), stoppedRun.outcome, stoppedRun.stoppedBy?.split(':')[0]);
|
|
1799
|
+
// 'UncloneableResult' 'stopped' 'UncloneableResult'
|
|
1800
|
+
if (failures.join() !== 'UncloneableResult') throw new Error(`the loud half reported ${failures.join() || 'nothing'}`);
|
|
1801
|
+
if (stoppedRun.outcome !== 'stopped') throw new Error(`a refused run resolved \`done\` with ${stoppedRun.outcome}, not stopped`);
|
|
1802
|
+
if (!stoppedRun.stoppedBy?.startsWith('UncloneableResult')) throw new Error(`stoppedBy: ${stoppedRun.stoppedBy}`);
|
|
1803
|
+
```
|
|
1804
|
+
|
|
1805
|
+
## 11. What does NOT exist — do not write code using it
|
|
1806
|
+
|
|
1807
|
+
| not available | instead |
|
|
1808
|
+
|---|---|
|
|
1809
|
+
| flow-level **retry** — re-dispatching a NODE under a new attempt, possibly routing elsewhere | **task-level retry IS built** since [W58](doc/work-items/w58-what-a-node-says-about-its-own-task.md): declare `retryPolicy` on the node (section 3) and the scheduler retries inside the same task id, reported on the record's `attempts`. Re-dispatching the node is [Q16](doc/questions/q16.md)'s other half and the task id's fourth segment, and nothing mints one |
|
|
1810
|
+
| **cancellation** | nothing. The scheduler cannot cancel a queued task either |
|
|
1811
|
+
| **durability** — resuming a run after a restart | nothing. The fold is designed to make it additive; it is not built |
|
|
1812
|
+
| **nested fan-out** — a branch that fans out again | **built, and the join over it is settled.** `FD-V15` is retired; a task inside a nested region is dispatched at its whole dotted `branchPath` (`r1:scoreC:1.0:0`); and every policy in section 5 is **transitive** — each region answers exactly as it would if it were not nested ([ADR 0030](doc/decisions/0030-a-join-is-transitive-and-a-dead-join-ends-the-branch-it-sits-in.md)). One thing to know: if an inner join is reported through `onUnsatisfiableJoin`, the outer branch containing it **ends** rather than waiting, so the outer join may still fire and its successor may run — and once that work settles the run folds to `stuck`, because a region that never fired is what `stuck` reads. **Fold too early and you will see `running`.** The four matrix rows and the join seam are tested at **depth 2**; the dead-join cascade is asserted at depth 3, and nothing else about depth 3 is a tested claim |
|
|
1813
|
+
| ~~**map-style fan-out**~~ — **BUILT** since [W33](doc/work-items/w33-a-map-at-run-time-arity.md); see section 3 |
|
|
1814
|
+
| a **join timeout** | nothing. A stuck join is detected by the `stuck` outcome, with no clock |
|
|
1815
|
+
| **per-join policy choice** | one policy per driver |
|
|
1816
|
+
| a **declared event family** for `fyflow-react-components` | `onRegionEvent`, `onFoldInput` and `joinStatusOf()` — see below |
|
|
1817
|
+
|
|
1818
|
+
**On that last row.** Flow structure is invisible to the *scheduler*, so a UI wired only to a
|
|
1819
|
+
scheduler cannot show which node a run is at, which branch it took, or what a join is waiting on.
|
|
1820
|
+
That is [Q21](doc/questions/q21.md), and it is about the scheduler's stream, not about this
|
|
1821
|
+
library — **this library does expose all three.** `onFoldInput` is the push half of `foldInputs()`,
|
|
1822
|
+
so you can fold incrementally and see a run finish without polling; `onRegionEvent` reports branches
|
|
1823
|
+
opening, arriving, exiting, suspending and resuming; `joinStatusOf()` answers what a join is waiting
|
|
1824
|
+
on. What is missing is an agreed envelope shape between the two repositories, not the data.
|
|
1825
|
+
|
|
1826
|
+
---
|
|
1827
|
+
|
|
1828
|
+
## 12. Wiring it to a real scheduler
|
|
1829
|
+
|
|
1830
|
+
**The short way is a registry DOCUMENT and one call** — `createRuntime` in
|
|
1831
|
+
`fyflow-orchestration/scheduler`, [W66](doc/work-items/w66-the-runtime-entry.md). It resolves every
|
|
1832
|
+
name the document uses against the code you supply, refuses every defect at construction by a named
|
|
1833
|
+
error, builds one pool per type instance and one object per group, holds a `FyflowScheduler` over
|
|
1834
|
+
them, and hands you **`{ driver, done, header }` per run** — the driver already running, `done` a
|
|
1835
|
+
promise that resolves with the run's terminal `RunState`, and `header` the run's checkpoint header
|
|
1836
|
+
([ADR 0058](doc/decisions/0058-runtime-start-returns-a-completion-handle-the-runtime-keeps.md);
|
|
1837
|
+
section 8 is what it waits for). **What stays yours is exactly what is code**: predicates,
|
|
1838
|
+
language adapters, custom group types and custom node-type classes, in one `code` argument.
|
|
1839
|
+
|
|
1840
|
+
```ts run
|
|
1841
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1842
|
+
import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
1843
|
+
|
|
1844
|
+
// The registry, as data. In your program this is a JSON file you import; here it is a literal so the
|
|
1845
|
+
// snippet is self-contained. A worker url may be RELATIVE, and then `base` is what it resolves
|
|
1846
|
+
// against — pass `import.meta.url`, because inside a bundle the library cannot guess it.
|
|
1847
|
+
const registry: RegistryDocument = {
|
|
1848
|
+
workers: {
|
|
1849
|
+
ingest: { url: '../../examples/workers/ingest.ts' },
|
|
1850
|
+
publish: { url: '../../examples/workers/publish.ts' },
|
|
1851
|
+
},
|
|
1852
|
+
groups: { gate: { type: 'ConcurrentLimitGroup', config: { limit: 1 } } },
|
|
1853
|
+
nodeTypes: {
|
|
1854
|
+
Ingest: { worker: 'ingest', inline: true },
|
|
1855
|
+
Publish: { worker: 'publish', inline: true, groups: ['gate'] },
|
|
1856
|
+
},
|
|
1857
|
+
};
|
|
1858
|
+
|
|
1859
|
+
// The flow, as data. It names registry KEYS and nothing else — no file, no URL, no class.
|
|
1860
|
+
const flow: FlowDefinition = {
|
|
1861
|
+
flowId: 'wired', version: 1, entry: 'ingest',
|
|
1862
|
+
nodes: [
|
|
1863
|
+
{ id: 'ingest', kind: 'task', type: 'Ingest', edges: [{ to: 'publish' }] },
|
|
1864
|
+
{ id: 'publish', kind: 'task', type: 'Publish', edges: [{ to: 'done' }] },
|
|
1865
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1866
|
+
],
|
|
1867
|
+
};
|
|
1868
|
+
|
|
1869
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url });
|
|
1870
|
+
|
|
1871
|
+
// A run is over when its OWN fold says so — never `scheduler.completed`, which fires every time the
|
|
1872
|
+
// scheduler drains (section 8). `await done` IS that, in one line: the runtime keeps the fold and
|
|
1873
|
+
// resolves with the terminal state (ADR 0058). This is the one snippet here that wants BOTH halves
|
|
1874
|
+
// of what `start` returns, and the reason is the line under it.
|
|
1875
|
+
const { driver, done } = runtime.start(flow, { confidence: 0.9 });
|
|
1876
|
+
// **`driver.runId` is readable** (W66): the runtime minted this run's id, and every task id of the
|
|
1877
|
+
// run begins with it. Pass your own as `start`'s `runId` when you need to choose it. It is readable
|
|
1878
|
+
// the instant `start` returns and does not wait for `done`, which is what holding the PAIR buys
|
|
1879
|
+
// over holding the promise alone.
|
|
1880
|
+
if (!/^r\d+$/.test(driver.runId)) throw new Error(`expected a minted run id, got ${driver.runId}`);
|
|
1881
|
+
const state = await done;
|
|
1882
|
+
const pools = Object.keys(runtime.pools).sort();
|
|
1883
|
+
await runtime.shutdown();
|
|
1884
|
+
|
|
1885
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
1886
|
+
// One pool per type instance, keyed by the name the document uses — that is what a node's `type` names
|
|
1887
|
+
// — **and one more for every BUILT-IN node type**, which is `Transform` and nothing else today
|
|
1888
|
+
// (section 13). The built-ins come LAST, so adding one never renumbers yours; a `WorkerManager` that
|
|
1889
|
+
// never receives a task spawns no thread, measured at W109.
|
|
1890
|
+
if (pools.join() !== 'Ingest,Publish,Transform') throw new Error(`pools: ${pools.join()}`);
|
|
1891
|
+
// And the group the document declared was built and is the scheduler's, keyed as declared.
|
|
1892
|
+
if (Object.keys(runtime.groups).join() !== 'gate') throw new Error(`groups: ${Object.keys(runtime.groups).join()}`);
|
|
1893
|
+
```
|
|
1894
|
+
|
|
1895
|
+
**A custom group type is a FACTORY, and the runtime holds its output to what the scheduler CALLS.**
|
|
1896
|
+
The document names a group by KEY with a `type`; the code supplies `{ configSchema?, create(config,
|
|
1897
|
+
key) }` under that type name; the three built-ins are pre-registered in the same shape. What comes
|
|
1898
|
+
back is checked for ADR 0055's six members and refused by name otherwise. Read the counts off the
|
|
1899
|
+
object the runtime built — [W70](doc/work-items/w70-a-flow-that-decides-and-a-custom-group-on-the-runtime.md):
|
|
1900
|
+
|
|
1901
|
+
```ts run
|
|
1902
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
1903
|
+
import { type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
1904
|
+
import type { ResourceGroup } from 'fyflow-scheduler';
|
|
1905
|
+
|
|
1906
|
+
// A CUSTOM GROUP is the scheduler's own `ResourceGroup`, and the runtime holds a factory's output to
|
|
1907
|
+
// the six members the scheduler CALLS or a viewer READS — `canRun`, `onStart`, `onFinish`,
|
|
1908
|
+
// `getMetrics`, `id`, `type` — refusing a missing one at construction, by name (ADR 0055: the
|
|
1909
|
+
// interface marks two of those optional and the scheduler calls them unguarded, which is S10
|
|
1910
|
+
// upstream). This one counts what it sees. The constructor is written long-hand on purpose: a
|
|
1911
|
+
// parameter property is TypeScript Node 24 refuses to LOAD (RT-R9).
|
|
1912
|
+
class CountingGroup extends EventTarget implements ResourceGroup {
|
|
1913
|
+
readonly type = 'Counting';
|
|
1914
|
+
readonly id: string;
|
|
1915
|
+
private readonly limit: number;
|
|
1916
|
+
private active = 0;
|
|
1917
|
+
started = 0;
|
|
1918
|
+
finished = 0;
|
|
1919
|
+
constructor(id: string, limit: number) { super(); this.id = id; this.limit = limit; }
|
|
1920
|
+
canRun(): boolean { return this.active < this.limit; }
|
|
1921
|
+
onStart(): void { this.active += 1; this.started += 1; }
|
|
1922
|
+
onFinish(): void { this.active -= 1; this.finished += 1; }
|
|
1923
|
+
getMetrics(): { limit: number; running: number; available: number; utilization: number } {
|
|
1924
|
+
return { limit: this.limit, running: this.active, available: this.limit - this.active, utilization: this.active / this.limit };
|
|
1925
|
+
}
|
|
1926
|
+
}
|
|
1927
|
+
|
|
1928
|
+
// THE REGISTRY declares the group by KEY, with its TYPE by name and its config; the CODE supplies the
|
|
1929
|
+
// factory for that type name — `{ configSchema?, create(config, key) }`. The three built-ins,
|
|
1930
|
+
// `ConcurrentLimitGroup`, `RateLimitGroup` and `KeyedRateLimitGroup`, are pre-registered in the same
|
|
1931
|
+
// shape, so `"type": "ConcurrentLimitGroup"` needs no code at all. A node type names the groups its
|
|
1932
|
+
// pool's tasks acquire.
|
|
1933
|
+
const registry: RegistryDocument = {
|
|
1934
|
+
workers: { greeter: { url: '../../examples/getting-started/worker.ts' } },
|
|
1935
|
+
groups: { budget: { type: 'Counting', config: { limit: 1 } } },
|
|
1936
|
+
nodeTypes: { Metered: { worker: 'greeter', inline: true, groups: ['budget'] } },
|
|
1937
|
+
};
|
|
1938
|
+
const flow: FlowDefinition = {
|
|
1939
|
+
flowId: 'metered', version: 1, entry: 'greet',
|
|
1940
|
+
nodes: [
|
|
1941
|
+
{ id: 'greet', kind: 'task', type: 'Metered', settings: { say: 'world' }, edges: [{ to: 'shout' }] },
|
|
1942
|
+
{ id: 'shout', kind: 'task', type: 'Metered', settings: { say: '!' }, edges: [{ to: 'done' }] },
|
|
1943
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
1944
|
+
],
|
|
1945
|
+
};
|
|
1946
|
+
const runtime = createRuntime(registry, {
|
|
1947
|
+
groupTypes: {
|
|
1948
|
+
Counting: {
|
|
1949
|
+
configSchema: { type: 'object', required: ['limit'], properties: { limit: { type: 'integer', minimum: 1 } } },
|
|
1950
|
+
create: (config, key) => new CountingGroup(key, (config as { limit: number }).limit),
|
|
1951
|
+
},
|
|
1952
|
+
},
|
|
1953
|
+
}, { base: import.meta.url });
|
|
1954
|
+
|
|
1955
|
+
const state = await runtime.start(flow, { text: 'hello' }).done;
|
|
1956
|
+
await runtime.shutdown();
|
|
1957
|
+
|
|
1958
|
+
// The group the runtime built is the scheduler's, keyed as the document declared it — and it was
|
|
1959
|
+
// acquired and released once per ADMISSION of a task of the type that names it. Twice here, because
|
|
1960
|
+
// these two tasks are admitted once each; a retry, or a threaded pool that finds no free worker,
|
|
1961
|
+
// re-admits, so a budget is spent per admission and NOT per task (section 13).
|
|
1962
|
+
const budget = runtime.groups['budget'] as CountingGroup;
|
|
1963
|
+
// THREE observers, because `runtime.groups['budget']` is the KEY the document filed it under and
|
|
1964
|
+
// says nothing about the object: the counters the group kept, the metrics IT reports — `running`
|
|
1965
|
+
// back at zero is `onFinish` having run, which is the member ADR 0055 exists for — and its own
|
|
1966
|
+
// `id` and `type`, which a factory ignoring the key it is handed would get wrong in silence.
|
|
1967
|
+
const metrics = budget.getMetrics();
|
|
1968
|
+
console.log(state.outcome, budget.started, budget.finished, metrics.running, budget.id); // 'completed' 2 2 0 'budget'
|
|
1969
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
1970
|
+
if (budget.started !== 2 || budget.finished !== 2) throw new Error(`the group saw ${budget.started} starts and ${budget.finished} finishes`);
|
|
1971
|
+
if (metrics.running !== 0 || metrics.available !== 1) throw new Error(`the group reports ${metrics.running} running, ${metrics.available} available`);
|
|
1972
|
+
if (budget.id !== 'budget' || budget.type !== 'Counting') throw new Error(`the group calls itself ${budget.id} / ${budget.type}`);
|
|
1973
|
+
```
|
|
1974
|
+
|
|
1975
|
+
**Every refusal is at construction and named.** A worker a type names and the document does not
|
|
1976
|
+
declare, a pool whose type does not exist, a config that fails its schema, a schema keyword outside
|
|
1977
|
+
the built-in subset, a name declared twice — each is a `RegistryError` subclass naming the entry
|
|
1978
|
+
(section 9, section 13). Three more are the runtime's own: a **relative worker url with no `base`**,
|
|
1979
|
+
a group type with no factory, and a group factory returning an object the scheduler cannot use —
|
|
1980
|
+
`canRun`, `onStart`, `onFinish`, `getMetrics`, `id` and `type`, each refused by name. *(That list is
|
|
1981
|
+
wider than the scheduler's own interface marks required, and deliberately: measured at 0.5.2, a group
|
|
1982
|
+
with no `onStart` throws on its first task and one with no `onFinish` leaves every task naming it
|
|
1983
|
+
running for ever, while one with no `getMetrics` runs fine —
|
|
1984
|
+
[ADR 0055](doc/decisions/0055-a-group-factorys-output-is-held-to-what-the-scheduler-calls.md).)*
|
|
1985
|
+
|
|
1986
|
+
**If a flow transforms, tell the runtime about it.** A language adapter's worker half is written
|
|
1987
|
+
into a pool's config at construction, and only for the types whose nodes transform (section 13), so
|
|
1988
|
+
pass every flow the runtime will run: `createRuntime(registry, code, { base, flows: [flow] })`. A
|
|
1989
|
+
flow handed to `start` that transforms on a type no pool was built for is refused there, naming the
|
|
1990
|
+
type and the language, rather than failing every task of that node one run in.
|
|
1991
|
+
|
|
1992
|
+
**Thousands of runs at once: start them all on ONE runtime.** Every run `start` (or `resume`) opens
|
|
1993
|
+
shares the runtime's pools and its one scheduler, and the runtime ROUTES each settlement to the driver
|
|
1994
|
+
holding its run id — [ADR 0088](doc/decisions/0088-a-runtime-routes-each-settlement-to-the-driver-holding-its-run-id.md).
|
|
1995
|
+
A driver is handed its own run's settlements and nothing of any other run's, so N concurrent runs cost
|
|
1996
|
+
one delivery per settlement and the topology is **linear in N**; before it, every settlement reached
|
|
1997
|
+
every driver and one runtime cost N² (`examples/retention.ts topology N` measures either topology, one runtime or
|
|
1998
|
+
`own` per run, and ADR 0088 carries both figures). Two things are
|
|
1999
|
+
not routed, and both are yours to size:
|
|
2000
|
+
|
|
2001
|
+
- **A `Driver` you build by hand** on `runtime.scheduler` or on your own `FyflowScheduler` (the long
|
|
2002
|
+
way, below) subscribes to the scheduler itself and hears every settlement on it, dropping the ones
|
|
2003
|
+
that are not its run's (`ID-R4`). That is correct, and N of them on one scheduler is N² deliveries.
|
|
2004
|
+
For a handful of runs it does not matter; for thousands, go through `start`.
|
|
2005
|
+
- **A listener of your own** on `runtime.scheduler` — a dashboard, a logger — hears every event of
|
|
2006
|
+
every run. Key it on `detail.id` and do O(1) work per event, or N such listeners are N² again.
|
|
2007
|
+
|
|
2008
|
+
A task id that does not parse — work this library did not mint — still reaches EVERY live run's
|
|
2009
|
+
driver, which each report it through `onUnmintedTask` (section 9): nobody owns it, so everybody says so.
|
|
2010
|
+
|
|
2011
|
+
**A runtime keeps a run until its `done` settles, and then only its run id** —
|
|
2012
|
+
[ADR 0091](doc/decisions/0091-a-runtime-releases-a-run-when-its-done-settles-and-keeps-only-its-run-id.md).
|
|
2013
|
+
At that moment the runtime drops the run's driver from its routing, so one long-lived runtime's memory
|
|
2014
|
+
is bounded by its LIVE runs plus one id string per run it ever opened. A run that is `running` or
|
|
2015
|
+
parked `awaiting_action` is kept — `resolve()` works as before. **The runtime has no way to reach a
|
|
2016
|
+
past run, so a postmortem is what YOU keep**: the `RunHandle` (`driver.foldInputs()`, `snapshot()`,
|
|
2017
|
+
`checkpoint()`, the `RunState` `done` resolved with), or what your `onFoldInput` stored. Drop the
|
|
2018
|
+
handle and the run is gone. After `done`, a released run is handed nothing — not its own run id, not
|
|
2019
|
+
an unminted one — so a `stopped` run's tasks still in flight settle into no driver; listen on
|
|
2020
|
+
`runtime.scheduler` yourself if you want their results. **The id stays in use**: `start` or `resume`
|
|
2021
|
+
naming it on the same runtime throws `RunIdInUse`, because those late settlements carry it. And the
|
|
2022
|
+
SCHEDULER keeps every completed task in `scheduler.tasks` unless `maxCompletedTasks` caps it (its own
|
|
2023
|
+
`AGENTS.md`); `createRuntime` builds it uncapped, so pass `createScheduler` to cap it.
|
|
2024
|
+
|
|
2025
|
+
**The long way, and it is the same objects.** You can build the scheduler yourself and hand the
|
|
2026
|
+
`Driver` the registry sets — which is what the runtime does, and what ladder L2-L5 here do with a
|
|
2027
|
+
double. You supply the scheduler, and that is all you have to supply. Since
|
|
2028
|
+
[ADR 0048](doc/decisions/0048-the-scheduler-is-a-declared-dependency-reached-by-one-bare-specifier.md)
|
|
2029
|
+
this library **depends on `fyflow-scheduler`** and says so where your installer reads it — a
|
|
2030
|
+
`peerDependencies` entry, at the pin stated at the end of this section — and `createTask` is
|
|
2031
|
+
**optional**: the driver's default is `(config) => new FyflowTask(config)`, the object a
|
|
2032
|
+
`FyflowScheduler` consumes.
|
|
2033
|
+
|
|
2034
|
+
```ts run
|
|
2035
|
+
import { Driver, fold, isTerminal, liveToken, type FlowDefinition, type FoldInput, type RunState } from 'fyflow-orchestration';
|
|
2036
|
+
import { FyflowScheduler, FyflowTask, WorkerManager } from 'fyflow-scheduler';
|
|
2037
|
+
|
|
2038
|
+
// Your worker modules — here two this repository's own L6 rung runs, so this snippet is a real run
|
|
2039
|
+
// on a real scheduler rather than a sketch. **One module per worker**: a module's default export is
|
|
2040
|
+
// the class a `WorkerManager` instantiates, which is the scheduler's own contract.
|
|
2041
|
+
const ingestUrl = new URL('../../examples/workers/ingest.ts', import.meta.url).href;
|
|
2042
|
+
const publishUrl = new URL('../../examples/workers/publish.ts', import.meta.url).href;
|
|
2043
|
+
|
|
2044
|
+
const definition: FlowDefinition = {
|
|
2045
|
+
flowId: 'wired', version: 1, entry: 'ingest',
|
|
2046
|
+
nodes: [
|
|
2047
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'publish' }] },
|
|
2048
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
2049
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2050
|
+
],
|
|
2051
|
+
};
|
|
2052
|
+
|
|
2053
|
+
// The registry: one worker entry, and one node type per pool — the pools are built BY HAND here,
|
|
2054
|
+
// keyed by the type's name, which is what `addTask` routes on. The bootstrap helper builds pools
|
|
2055
|
+
// FROM entries like these (section 13) — envelope-fronted, the worker loaded by URL with its config
|
|
2056
|
+
// under `config.config` — where these two load the worker directly with `{ role }`; the helper's
|
|
2057
|
+
// call is the one a consumer copies, and this snippet keeps the raw scheduler in view.
|
|
2058
|
+
const registry = {
|
|
2059
|
+
workers: { ingest: { url: ingestUrl }, publish: { url: publishUrl } },
|
|
2060
|
+
nodeTypes: { IngestWorker: { worker: 'ingest' }, PublishWorker: { worker: 'publish' } },
|
|
2061
|
+
};
|
|
2062
|
+
const scheduler = new FyflowScheduler({
|
|
2063
|
+
IngestWorker: new WorkerManager(ingestUrl, { maxThreads: 1, inline: true }),
|
|
2064
|
+
PublishWorker: new WorkerManager(publishUrl, { maxThreads: 1, inline: true }),
|
|
2065
|
+
});
|
|
2066
|
+
|
|
2067
|
+
// A run is over when its OWN fold says so — never `scheduler.completed`, which fires every
|
|
2068
|
+
// time the scheduler drains (section 8). `onFoldInput` is the push half of `foldInputs()`, and
|
|
2069
|
+
// the outcome is read on a `queueMicrotask`, AFTER the driver's turn: a prefix cut mid-turn is
|
|
2070
|
+
// not a resting state, and section 8 says so.
|
|
2071
|
+
const inputs: FoldInput[] = [];
|
|
2072
|
+
const state = await new Promise<RunState>((resolve, reject) => {
|
|
2073
|
+
const driver = new Driver(definition, {
|
|
2074
|
+
runId: 'run-1',
|
|
2075
|
+
scheduler, // the real class, as it is — no cast (Q71)
|
|
2076
|
+
predicates: new Map(),
|
|
2077
|
+
...registry, // the types the flow names, and their workers (section 13)
|
|
2078
|
+
groups: Object.keys(scheduler.groups), // the resource groups a node may name (section 3)
|
|
2079
|
+
joinPolicy: liveToken,
|
|
2080
|
+
onFoldInput: (input) => {
|
|
2081
|
+
inputs.push(input);
|
|
2082
|
+
queueMicrotask(() => { const s = fold(inputs); if (isTerminal(s)) resolve(s); });
|
|
2083
|
+
},
|
|
2084
|
+
onRunFailure: (f) => reject(new Error(`${f.kind}: ${f.message}`)),
|
|
2085
|
+
});
|
|
2086
|
+
driver.start({ confidence: 0.9 });
|
|
2087
|
+
});
|
|
2088
|
+
const tasks = [...scheduler.tasks.values()]; // read BEFORE shutdown, which clears the map
|
|
2089
|
+
await scheduler.shutdown();
|
|
2090
|
+
|
|
2091
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
2092
|
+
// `nodes` is keyed by TASK id (`run-1:publish:-:0`), so look the node up by its id.
|
|
2093
|
+
const publish = Object.values(state.nodes).find((n) => n.nodeId === 'publish');
|
|
2094
|
+
if (publish?.state !== 'done') throw new Error(`publish never settled: ${publish?.state}`);
|
|
2095
|
+
// And the tasks the scheduler ran were the class's own — the driver built them, you did not.
|
|
2096
|
+
// The scheduler would have run a plain object too (it duck-types its task, with `attempts` left
|
|
2097
|
+
// undefined), which is why this is asserted rather than assumed.
|
|
2098
|
+
if (tasks.length !== 2 || !tasks.every((t) => t instanceof FyflowTask)) throw new Error('the default createTask did not build FyflowTasks');
|
|
2099
|
+
```
|
|
2100
|
+
|
|
2101
|
+
`nodeTypes` and `workers` are the sets the validator checks a definition against, and **a type's
|
|
2102
|
+
pool is keyed by the type's name** — so the object you hand the driver is the object you build the
|
|
2103
|
+
pools from, as above, rather than a second list; every task the driver dispatches carries the
|
|
2104
|
+
envelope `{ settings, input }` (section 13), which is what this repository's worker reads `input`
|
|
2105
|
+
from. Pass `createTask` only to run against a
|
|
2106
|
+
double — ladder L2-L5 here hand the driver `(c) => c` and read the plain object back — or to build
|
|
2107
|
+
something other than a plain `FyflowTask`.
|
|
2108
|
+
|
|
2109
|
+
**The `scheduler` option takes a `FyflowScheduler` as it is.** Until W57 it did not, under the one
|
|
2110
|
+
configuration most consumers have: `FyflowScheduler extends EventTarget`, so with the DOM lib in
|
|
2111
|
+
scope — TypeScript's default when `lib` is unset — its `addEventListener` was not assignable to
|
|
2112
|
+
`SchedulerLike`'s, and this section told you to write `scheduler as unknown as SchedulerLike`. That
|
|
2113
|
+
was [Q71](doc/questions/q71.md), a defect in our type and not in yours, and it is closed:
|
|
2114
|
+
`SchedulerLike`'s listener parameter is `object`, which is what makes the real class satisfy it,
|
|
2115
|
+
and every snippet in this document is typechecked under the DOM lib by `npm run check:docs`.
|
|
2116
|
+
|
|
2117
|
+
**A worker file can be TypeScript.** Node 24 runs it by stripping types, and Deno natively, so **one
|
|
2118
|
+
`.ts` worker serves Node and Deno**. *(The scheduler's own reference **used to say** a Node worker
|
|
2119
|
+
must be JavaScript, which was stale at Node 24 — [S9](doc/scheduler-upstream.md), raised 2026-09-07
|
|
2120
|
+
and **fixed in 0.5.2**. Its §3 now version-qualifies: a `.ts` worker works on **Node >= 22.18**, and
|
|
2121
|
+
below that it does not, so the old advice is right for that range and wrong for ours. This paragraph
|
|
2122
|
+
called the reference stale for nine days and then for one more, in the document above the sentence
|
|
2123
|
+
`D7` was built around.)* **Not the browser**: nothing there executes TypeScript, so
|
|
2124
|
+
a browser build needs the worker emitted as a bundled `.js` asset. `RT-R10` — ladder L6 runs in Node
|
|
2125
|
+
and Deno and not in the browser, and the browser rung is [Q52](doc/questions/q52.md).
|
|
2126
|
+
|
|
2127
|
+
**Pin the scheduler at 0.5.2.** This library's listener boundary is written against **0.3.0
|
|
2128
|
+
semantics, which neither 0.5.1 nor 0.5.2 changes**: `task.progress`'s detail changed shape at 0.3.0 (the id is
|
|
2129
|
+
`id`, not `taskId`), the second `task.user_action` dispatch site arrived there, and `task.timestamp`
|
|
2130
|
+
did not exist before it. **`core/FyflowScheduler.ts` is byte-identical between 0.3.0, 0.5.1 and 0.5.2**, so
|
|
2131
|
+
every one of those still holds; what 0.5.1 added is a `worker.setup.*` / `worker.teardown.*` event
|
|
2132
|
+
family this library subscribes to not at all. Install it from a registry — npm and JSR both carry it —
|
|
2133
|
+
and **never from a working tree**, which is
|
|
2134
|
+
[ADR 0029](doc/decisions/0029-the-scheduler-is-a-registry-dependency-and-nothing-reaches-the-working-tree.md)
|
|
2135
|
+
superseding [ADR 0013](doc/decisions/0013-the-scheduler-is-a-development-dependency-reached-from-the-working-tree.md)
|
|
2136
|
+
rules 3 and 5. **The manifests say the same thing**, since
|
|
2137
|
+
[ADR 0048](doc/decisions/0048-the-scheduler-is-a-declared-dependency-reached-by-one-bare-specifier.md):
|
|
2138
|
+
`package.json` names it in `peerDependencies` at exactly that version and `deno.json` maps it to
|
|
2139
|
+
`jsr:@fyflow/scheduler` at exactly that version, and `npm run check:scheduler`'s `D8` holds both to
|
|
2140
|
+
the pin the same way `D7` holds this sentence. *(This read **"pin at 0.3.0 … npm and JSR both stop at 0.2.1, so 0.3.0 is reached from
|
|
2141
|
+
a working tree"** — a live instruction to a consumer, wrong in the version AND in the mechanism, the
|
|
2142
|
+
second having been superseded on 2026-08-31. W36's conformity reader.)*
|
|
2143
|
+
|
|
2144
|
+
The worked flows on a real scheduler are in [`examples/`](examples/README.md), which counts them —
|
|
2145
|
+
including a routed-away branch, a lost branch, a human resolving a suspension, a refusal that stops
|
|
2146
|
+
a run, and a join's successor reducing the region's record through a declared transform. *(This
|
|
2147
|
+
sentence read **"Thirteen worked flows"** from the day it was written until W63, over a directory
|
|
2148
|
+
that had held twenty-two for two plans — a count in prose that no check reads, so it carries none
|
|
2149
|
+
now.)*
|
|
2150
|
+
|
|
2151
|
+
### The driver with no scheduler at all — the shape ladder L2 uses
|
|
2152
|
+
|
|
2153
|
+
*This was section 2, "Minimal working program", until
|
|
2154
|
+
[W69](doc/work-items/w69-a-getting-started-example-a-consumer-can-copy.md) — and since P18 it is the
|
|
2155
|
+
HARDER road, so it moved here byte for byte. It is still the one demonstration of a `Driver` with
|
|
2156
|
+
nothing underneath it: a fake scheduler in twelve lines, which is what ladder L2 runs against and
|
|
2157
|
+
what a unit test of your own routing wants.*
|
|
2158
|
+
|
|
2159
|
+
A flow is **data**. You supply the predicates and the task factory; the driver does the rest.
|
|
2160
|
+
|
|
2161
|
+
```ts run
|
|
2162
|
+
import { Driver, fold, liveToken, type FlowDefinition, type Predicate } from 'fyflow-orchestration';
|
|
2163
|
+
|
|
2164
|
+
const definition: FlowDefinition = {
|
|
2165
|
+
flowId: 'greet',
|
|
2166
|
+
version: 1,
|
|
2167
|
+
entry: 'ingest',
|
|
2168
|
+
nodes: [
|
|
2169
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'publish' }] },
|
|
2170
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
2171
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2172
|
+
],
|
|
2173
|
+
};
|
|
2174
|
+
|
|
2175
|
+
// A fake scheduler, so this snippet runs with no workers. In your program this is
|
|
2176
|
+
// `new FyflowScheduler({ ... })` — see section 12.
|
|
2177
|
+
//
|
|
2178
|
+
// **Key the listeners by event type.** The driver subscribes to three of them and
|
|
2179
|
+
// normalizes on the task's terminal STATE rather than the event name (`OR-R3`), so a
|
|
2180
|
+
// double that fans one settlement to every listener writes THREE records for it and
|
|
2181
|
+
// the run stops on a `DuplicateTaskId`. A real `EventTarget` does not do this; a
|
|
2182
|
+
// hand-written double is where it happens.
|
|
2183
|
+
const listeners = new Map<string, ((e: { detail: unknown }) => void)[]>();
|
|
2184
|
+
const added: { id: string }[] = [];
|
|
2185
|
+
const failures: string[] = [];
|
|
2186
|
+
const scheduler = {
|
|
2187
|
+
addTask(task: unknown) { added.push(task as { id: string }); },
|
|
2188
|
+
addEventListener(type: string, fn: (e: { detail: unknown }) => void) {
|
|
2189
|
+
listeners.set(type, [...(listeners.get(type) ?? []), fn]);
|
|
2190
|
+
},
|
|
2191
|
+
};
|
|
2192
|
+
const settle = (detail: unknown, type = 'task.completed') => {
|
|
2193
|
+
for (const fn of listeners.get(type) ?? []) fn({ detail });
|
|
2194
|
+
};
|
|
2195
|
+
|
|
2196
|
+
const driver = new Driver(definition, {
|
|
2197
|
+
runId: 'run-1',
|
|
2198
|
+
scheduler,
|
|
2199
|
+
createTask: (config) => config, // your `new FyflowTask(config)`
|
|
2200
|
+
predicates: new Map<string, Predicate>(),
|
|
2201
|
+
// Since v0.19 a task names a node TYPE and a type names a WORKER entry — section 13. The
|
|
2202
|
+
// validator and the driver require the entries to be DECLARED and load nothing; the URL is what
|
|
2203
|
+
// the bootstrap helper hands a real pool, so a fake scheduler's registry can carry any string.
|
|
2204
|
+
workers: { flow: { url: 'worker.js' } },
|
|
2205
|
+
nodeTypes: { IngestWorker: { worker: 'flow' }, PublishWorker: { worker: 'flow' } },
|
|
2206
|
+
joinPolicy: liveToken,
|
|
2207
|
+
// Section 9: a refusal raised inside a listener is REPORTED, never thrown.
|
|
2208
|
+
// Without this the run stops silently.
|
|
2209
|
+
onRunFailure: (f) => { failures.push(f.kind); },
|
|
2210
|
+
});
|
|
2211
|
+
|
|
2212
|
+
driver.start({ name: 'world' });
|
|
2213
|
+
console.log(added.map((t) => t.id)); // [ 'run-1:ingest:-:0' ]
|
|
2214
|
+
|
|
2215
|
+
// The scheduler reports a settlement; the driver routes.
|
|
2216
|
+
settle({ id: 'run-1:ingest:-:0', state: 'done', result: { greeting: 'hello world' } });
|
|
2217
|
+
settle({ id: 'run-1:publish:-:0', state: 'done', result: null });
|
|
2218
|
+
console.log(added.map((t) => t.id)); // [ 'run-1:ingest:-:0', 'run-1:publish:-:0' ]
|
|
2219
|
+
|
|
2220
|
+
const state = fold(driver.foldInputs());
|
|
2221
|
+
console.log(state.outcome); // 'completed'
|
|
2222
|
+
|
|
2223
|
+
if (added.map((t) => t.id).join() !== 'run-1:ingest:-:0,run-1:publish:-:0') {
|
|
2224
|
+
throw new Error(`wrong dispatches: ${added.map((t) => t.id).join()}`);
|
|
2225
|
+
}
|
|
2226
|
+
if (failures.length) throw new Error(`the run reported ${failures.join(', ')}`);
|
|
2227
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
2228
|
+
```
|
|
2229
|
+
|
|
2230
|
+
**Three things bite everyone.** The **run id is a constructor option**, not a `start()` argument,
|
|
2231
|
+
and it must match `[A-Za-z0-9_-]{1,64}`. A definition with a fan-out and no `joinPolicy` is
|
|
2232
|
+
**refused at construction**. A driver serves **exactly one run**, so `start()` is called at most
|
|
2233
|
+
once — **and a second driver on the same scheduler is fine**, because a settlement only affects the
|
|
2234
|
+
driver that dispatched it. Built by hand, each driver still HEARS every settlement and drops the
|
|
2235
|
+
others; through `runtime.start` it is only handed its own (ADR 0088, above). See section 9.
|
|
2236
|
+
|
|
2237
|
+
---
|
|
2238
|
+
|
|
2239
|
+
## 13. Node types — a registry DOCUMENT entry, a derivation to a pool, and the envelope
|
|
2240
|
+
|
|
2241
|
+
**Every `task` names a node type**, and may carry per-node **`settings`** and a **`pool`** —
|
|
2242
|
+
`spec/flow-definition.md` v0.19, [ADR 0049](doc/decisions/0049-a-node-type-is-a-registry-entry-and-its-worker-receives-an-envelope.md),
|
|
2243
|
+
[ADR 0054](doc/decisions/0054-every-task-node-names-a-type-and-a-type-is-a-derivation-to-pool-options.md),
|
|
2244
|
+
[W59](doc/work-items/w59-what-a-node-type-is.md), [W65](doc/work-items/w65-the-registry-document-and-a-node-type-is-a-derivation.md).
|
|
2245
|
+
**A node type is a REGISTRY entry and the definition stays data**: the registry is a DOCUMENT —
|
|
2246
|
+
`workers` (a module URL and the four JSON schemas that worker owns), `nodeTypes` (each naming a
|
|
2247
|
+
worker, with its pool's options, its groups, a config, a retry default and `settings` DEFAULTS),
|
|
2248
|
+
`pools` (named instances of a type), `groups` (built through group-type factories), plus the
|
|
2249
|
+
`predicates` and `languages` it names and your code supplies. **A type is a DERIVATION from an
|
|
2250
|
+
instance's config to pool options**; the document form is the identity derivation — what it
|
|
2251
|
+
declares is what its default pool is built from — and a class registered in code is any other, both
|
|
2252
|
+
passing the same checks. **Every type has a default pool, keyed by its name**; the flow names the
|
|
2253
|
+
type and nothing else — never a file, never a URL. *(Until v0.19 a task could name a `workerType`
|
|
2254
|
+
instead — a pool key, its worker handed the raw payload. That road is retired: `workerType` is an
|
|
2255
|
+
undeclared key, and a worker written for it reads under `input` what it read as its whole payload.)*
|
|
2256
|
+
|
|
2257
|
+
| | | |
|
|
2258
|
+
|---|---|---|
|
|
2259
|
+
| `type` | **required** | the name of an entry in `nodeTypes` whose `worker` the registry's `workers` declares (`FD-V32`, both clauses) — **or a BUILT-IN type, which no document declares and which names no worker** (`Transform`, below). A task with no `type` is `FD-V23` |
|
|
2260
|
+
| `pool` | optional | the name of an instance in the registry's `pools` whose `type` is this node's (`FD-V38`); absent, the type's DEFAULT instance. The type's own name is that default, stated explicitly |
|
|
2261
|
+
| `settings` | optional | a plain JSON object which — with the type's `settings` defaults merged UNDER it — the worker's **`settingsSchema`** accepts (`FD-V33`); **absent means the defaults alone**, `{}` when the type declares none, and the schema sees that. A schema is JSON Schema in the built-in subset (ADR 0053): a keyword outside it is refused at construction, never ignored |
|
|
2262
|
+
| `transform` | optional | `{ lang, expr }`: an expression, in a language the registry's **`languages`** set names, applied to the node's input BEFORE its worker sees it — below. `lang` unregistered is `FD-V35`, `expr` the language refuses is `FD-V36`, not `{ lang, expr }` is `FD-V37` |
|
|
2263
|
+
|
|
2264
|
+
### The one node type the library ships: a node that only TRANSFORMS
|
|
2265
|
+
|
|
2266
|
+
**`Transform` is a built-in node type, it names NO worker, and its task is its own `transform`** —
|
|
2267
|
+
`spec/flow-definition.md` v0.28,
|
|
2268
|
+
[ADR 0075](doc/decisions/0075-a-node-that-only-transforms-is-a-built-in-type-with-no-inner-worker.md),
|
|
2269
|
+
[W109](doc/work-items/w109-a-node-that-only-transforms.md). Write `{ "kind": "task", "type":
|
|
2270
|
+
"Transform", "transform": { "lang": …, "expr": … } }` and the node reshapes its input with no worker
|
|
2271
|
+
module, no worker entry and no node type of yours. **It is merged at every READ and stored in no
|
|
2272
|
+
registry**, so a flow may name it against a registry that declares no `nodeTypes` at all — which is what
|
|
2273
|
+
lets a document a tool EMITS carry a reshaping step without carrying code for it. Name your own type
|
|
2274
|
+
`Transform` and the registry refuses it by name, on every road: a built-in is never replaced. **It names no LANGUAGE**: `lang` is the
|
|
2275
|
+
node's, exactly as it is on any `transform`, so the adapter stays yours and no engine ships with the
|
|
2276
|
+
library.
|
|
2277
|
+
|
|
2278
|
+
**Use it last, and the reason is a cost.** A computation belongs at the cheapest tier that can hold it:
|
|
2279
|
+
|
|
2280
|
+
| tier | where it goes | what it costs |
|
|
2281
|
+
|---|---|---|
|
|
2282
|
+
| **selection** — *I need a value from another node* | **`needs`** on the consuming node (section 3) | **no task at all** |
|
|
2283
|
+
| **computation feeding a worker** — *reshape what THIS worker reads* | **`transform`** on that worker's own node | **no extra task** |
|
|
2284
|
+
| **computation with no consuming node** — *reshape for a join, a decision, an `end`* | **a `Transform` node** | **a real task**: measured at **+1 dispatch, +4 clones and +2 log records** per node, and inside a `map`, N of them |
|
|
2285
|
+
|
|
2286
|
+
Measured on a real `FyflowScheduler` at W109: a `map` whose branch is a `Transform` node feeding a
|
|
2287
|
+
worker runs **1.93x** the same `map` with the transform on the worker's own node, inline at N = 128,
|
|
2288
|
+
and a flat **~1.47x** threaded. *A reshaping task node is what
|
|
2289
|
+
[ADR 0040](doc/decisions/0040-what-a-node-can-reach-is-the-runs-own-outputs-and-a-node-id-names-them-relative-to-a-branch-path.md)
|
|
2290
|
+
and [ADR 0051](doc/decisions/0051-a-transform-is-a-declared-expression-in-a-registered-language-applied-in-the-types-worker-wrapper.md)
|
|
2291
|
+
exist to remove, so this is an escape hatch rather than a default — the ADR says so in those words.*
|
|
2292
|
+
|
|
2293
|
+
**Three things to know before you use one.** A node of this type with **no `transform` is refused at
|
|
2294
|
+
construction** by `FD-V46`, because the alternative is a task that returns what it was given. It
|
|
2295
|
+
**takes `settings`**, bound as `$settings` in the expression exactly as on any transform, so one type is
|
|
2296
|
+
reusable across flows. And **every arc LEAVING it is `unchecked`**: it declares no `outputSchema`
|
|
2297
|
+
because it has no worker, so section 15's authoring feedback cannot judge what comes next. What you get
|
|
2298
|
+
instead is a SAMPLE — `sampledOutput(registry, flow, nodeId)` returns the shape the node emitted on the
|
|
2299
|
+
producer's own example, the sample it came from, and the departing verdicts, which it returns rather
|
|
2300
|
+
than hides because a sample is evidence about one input and `fits` is a claim about every input
|
|
2301
|
+
([Q446](doc/questions/q446.md)).
|
|
2302
|
+
|
|
2303
|
+
**What EVERY task's worker receives is the ENVELOPE `{ settings, input }`** — `FD-C5`. `settings`
|
|
2304
|
+
is the node's effective settings — the merged object — as its own copy; **`input` is the value this
|
|
2305
|
+
node receives** — the previous node's result, the object `needs` describes, its own item as a map
|
|
2306
|
+
instance, the collection after a join. Write a worker to read `payload.settings` and
|
|
2307
|
+
`payload.input`, and nothing else changes about it.
|
|
2308
|
+
|
|
2309
|
+
```ts run
|
|
2310
|
+
import { validateDefinition, type FlowDefinition, type NodeTypeDeclaration, type Registry, type WorkerEntry } from 'fyflow-orchestration';
|
|
2311
|
+
|
|
2312
|
+
// The WORKER entries: a module URL, and the schemas the worker owns. `settingsSchema` is what
|
|
2313
|
+
// FD-V33 applies to every node of EVERY type running that entry — over the node's settings with
|
|
2314
|
+
// the type's defaults merged under, `{}` when neither says anything — so a schema that REQUIRES a
|
|
2315
|
+
// key refuses absence by name, and a type whose nodes carry no settings is written over an entry
|
|
2316
|
+
// that declares none (two entries here, one module each — since W67 a worker is a module). `configSchema` is held against every pool's
|
|
2317
|
+
// derived config at construction; `inputSchema` and `outputSchema` are declared and carried, enforced on ONE
|
|
2318
|
+
// value per run each (the entry payload at start(), a person's resolution at resolve() — FD-R4, W73)
|
|
2319
|
+
// and never per settlement — since v0.20 (ADR 0056) any schema may carry `examples`, held to it at
|
|
2320
|
+
// construction, `npm run check:workers` runs a worker on its input example (P19, W72), and since v0.21
|
|
2321
|
+
// `checkWiring(registry, flow)` gives every arc of a flow a verdict from the two (W73).
|
|
2322
|
+
const ingestUrl = new URL('../../examples/workers/ingest.ts', import.meta.url).href;
|
|
2323
|
+
const typedUrl = new URL('../../examples/workers/typed.ts', import.meta.url).href;
|
|
2324
|
+
const ingest: WorkerEntry = { url: ingestUrl };
|
|
2325
|
+
const scorer: WorkerEntry = {
|
|
2326
|
+
url: typedUrl,
|
|
2327
|
+
settingsSchema: { type: 'object', required: ['factor'], properties: { factor: { type: 'number', minimum: 0 } }, additionalProperties: false },
|
|
2328
|
+
};
|
|
2329
|
+
// Two node types, in the DOCUMENT form — the identity derivation: what each declares is what its
|
|
2330
|
+
// default pool is built from. `config` is every worker instance's constructor argument, `pool` the
|
|
2331
|
+
// WorkerManager options minus what the type states beside it, `groups` the POOL's groups (every
|
|
2332
|
+
// task of the type holds them), `retryPolicy` for every node of the type that declares none.
|
|
2333
|
+
const IngestWorker: NodeTypeDeclaration = { worker: 'ingest', inline: true };
|
|
2334
|
+
const Scorer: NodeTypeDeclaration = {
|
|
2335
|
+
worker: 'scorer',
|
|
2336
|
+
inline: true,
|
|
2337
|
+
pool: { maxThreads: 2 },
|
|
2338
|
+
groups: ['scorer'],
|
|
2339
|
+
retryPolicy: { maxRetries: 1, backoffMs: 0 },
|
|
2340
|
+
};
|
|
2341
|
+
|
|
2342
|
+
// Two nodes of ONE type with DIFFERENT settings — the whole point.
|
|
2343
|
+
const typed: FlowDefinition = {
|
|
2344
|
+
flowId: 'typed', version: 1, entry: 'ingest',
|
|
2345
|
+
nodes: [
|
|
2346
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'low' }] },
|
|
2347
|
+
{ id: 'low', kind: 'task', type: 'Scorer', settings: { factor: 10 }, edges: [{ to: 'high' }] },
|
|
2348
|
+
{ id: 'high', kind: 'task', type: 'Scorer', settings: { factor: 100 }, edges: [{ to: 'done' }] },
|
|
2349
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2350
|
+
],
|
|
2351
|
+
};
|
|
2352
|
+
|
|
2353
|
+
// The registry the driver reads — the same sections the bootstrap helper builds pools from (below),
|
|
2354
|
+
// so a type the flow names, the pool the scheduler holds and the schema the driver applies are one
|
|
2355
|
+
// object and not three copies. `groups` is the set of group KEYS a node or a type may name.
|
|
2356
|
+
const registry: Registry = { predicates: [], workers: { ingest, scorer }, nodeTypes: { IngestWorker, Scorer }, groups: ['scorer'] };
|
|
2357
|
+
const bad = validateDefinition(typed, registry);
|
|
2358
|
+
if (bad.length) throw new Error(`the typed flow is invalid: ${bad.map((f) => `${f.rule}@${f.at}`).join()}`);
|
|
2359
|
+
|
|
2360
|
+
// The refusals, each at construction and never on the one workload that hits it.
|
|
2361
|
+
const rules = (def: FlowDefinition, reg = registry) => validateDefinition(def, reg).map((f) => f.rule);
|
|
2362
|
+
const low = (node: object): FlowDefinition => ({ ...typed, nodes: [typed.nodes[0]!, node as never, typed.nodes[3]!] });
|
|
2363
|
+
// FD-V32 — a type the registry does not hold; `nodeTypes` OMITTED means NONE registered; and a
|
|
2364
|
+
// registered type whose WORKER the registry's `workers` does not declare.
|
|
2365
|
+
if (!rules(low({ id: 'low', kind: 'task', type: 'Nope', edges: [{ to: 'done' }] })).includes('FD-V32')) throw new Error('FD-V32');
|
|
2366
|
+
if (!rules(typed, { ...registry, nodeTypes: undefined }).includes('FD-V32')) throw new Error('FD-V32 with nodeTypes omitted');
|
|
2367
|
+
if (!rules(typed, { ...registry, workers: {} }).includes('FD-V32')) throw new Error('FD-V32: the type names a worker the registry lacks');
|
|
2368
|
+
// FD-V33 — settings the worker's schema refuses; and absent settings are the type's defaults — none
|
|
2369
|
+
// here, so `{}` — which this schema refuses because it requires `factor`.
|
|
2370
|
+
if (!rules(low({ id: 'low', kind: 'task', type: 'Scorer', settings: { factor: 'ten' }, edges: [{ to: 'done' }] })).includes('FD-V33')) throw new Error('FD-V33');
|
|
2371
|
+
if (!rules(low({ id: 'low', kind: 'task', type: 'Scorer', edges: [{ to: 'done' }] })).includes('FD-V33')) throw new Error('FD-V33 on absent settings');
|
|
2372
|
+
// FD-V38 — a `pool` that is not a registered instance of the node's type; the type's own name is
|
|
2373
|
+
// its default instance and is accepted.
|
|
2374
|
+
if (!rules(low({ id: 'low', kind: 'task', type: 'Scorer', settings: { factor: 1 }, pool: 'nope', edges: [{ to: 'done' }] })).includes('FD-V38')) throw new Error('FD-V38');
|
|
2375
|
+
if (rules(low({ id: 'low', kind: 'task', type: 'Scorer', settings: { factor: 1 }, pool: 'Scorer', edges: [{ to: 'done' }] })).length) throw new Error('the type`s own name is its default instance');
|
|
2376
|
+
// FD-V23 — there is no untyped road: a task without `type` is a task missing a required key.
|
|
2377
|
+
if (!rules(low({ id: 'low', kind: 'task', edges: [{ to: 'done' }] })).includes('FD-V23')) throw new Error('FD-V23: a task with no type');
|
|
2378
|
+
```
|
|
2379
|
+
|
|
2380
|
+
**Wiring it: `createRuntime`, or the bootstrap helper underneath it.** `fyflow-orchestration/scheduler`
|
|
2381
|
+
is a sub-path — the one module of this package that constructs scheduler objects, kept out of the
|
|
2382
|
+
root so the validator, the driver and the fold load no `WorkerManager` (`RT-R13`). **`createRuntime`
|
|
2383
|
+
is the call a consumer copies** (section 12): it does everything below from a registry DOCUMENT,
|
|
2384
|
+
including creating the group objects. `bootstrap` is what it uses for the pools, and it is exported
|
|
2385
|
+
for a consumer who builds their groups and their scheduler themselves. `bootstrap(registry, groups, options)` builds
|
|
2386
|
+
**one pool per type, keyed by the type's name, and one per named instance in `pools`**, each
|
|
2387
|
+
through the type's derivation and each running the package's envelope worker around your worker,
|
|
2388
|
+
and refuses **both ways** a type naming a group you gave no object for and an object no type names
|
|
2389
|
+
(the scheduler ignores an unknown group id in silence, at the pool level too — measured). Hand the
|
|
2390
|
+
scheduler its result, and the driver the SAME registry sections:
|
|
2391
|
+
|
|
2392
|
+
```ts check
|
|
2393
|
+
import { type FlowDefinition, type Languages, type NodeTypeDeclaration, type WorkerEntry } from 'fyflow-orchestration';
|
|
2394
|
+
import { bootstrap } from 'fyflow-orchestration/scheduler';
|
|
2395
|
+
import { ConcurrentLimitGroup, FyflowScheduler } from 'fyflow-scheduler';
|
|
2396
|
+
import { Driver, liveToken } from 'fyflow-orchestration';
|
|
2397
|
+
|
|
2398
|
+
declare const ingest: WorkerEntry;
|
|
2399
|
+
declare const scorer: WorkerEntry;
|
|
2400
|
+
declare const IngestWorker: NodeTypeDeclaration;
|
|
2401
|
+
declare const Scorer: NodeTypeDeclaration;
|
|
2402
|
+
declare const typed: FlowDefinition;
|
|
2403
|
+
declare const languages: Languages;
|
|
2404
|
+
|
|
2405
|
+
// The types' groups are given as OBJECTS here and only here; the helper returns them keyed as given.
|
|
2406
|
+
// The languages and the flows are what tells the helper WHICH typed pools need which adapter's
|
|
2407
|
+
// worker module — written into a pool's config only where a node of that type transforms (W61).
|
|
2408
|
+
const registry = { workers: { ingest, scorer }, nodeTypes: { IngestWorker, Scorer } };
|
|
2409
|
+
const built = bootstrap(registry, { scorer: new ConcurrentLimitGroup(4, 'scorer') }, { languages, flows: [typed] });
|
|
2410
|
+
const scheduler = new FyflowScheduler(built.workerPools, built.resourceGroups);
|
|
2411
|
+
const driver = new Driver(typed, {
|
|
2412
|
+
runId: 'run-1', scheduler, predicates: new Map(), joinPolicy: liveToken,
|
|
2413
|
+
...registry, // the SAME object the helper built every pool from
|
|
2414
|
+
groups: Object.keys(scheduler.groups), // and its groups, read back off the scheduler
|
|
2415
|
+
languages, // and the same languages
|
|
2416
|
+
});
|
|
2417
|
+
driver.start({ confidence: 0.9 });
|
|
2418
|
+
```
|
|
2419
|
+
|
|
2420
|
+
**What a type declares for its nodes, and how it merges.** A node's own `retryPolicy` wins, whole;
|
|
2421
|
+
then the type's; then the driver's `defaultRetryPolicy`. A type's `groups` are the pool's and the
|
|
2422
|
+
scheduler unions them with a node's own `workerGroups`, so nothing merges — and a task acquires its
|
|
2423
|
+
pool's group **once per ADMISSION**: a retry is admitted again, and so is a task that was admitted
|
|
2424
|
+
to a threaded pool and found no free worker (both measured at ladder L6), so a group's budget is
|
|
2425
|
+
spent per admission and not per task. Both are held at
|
|
2426
|
+
construction: a type's policy to `FD-V30` and a type's groups to the registry's `groups` under
|
|
2427
|
+
`FD-V31`, naming the type.
|
|
2428
|
+
|
|
2429
|
+
**A transform — what a node may DO to its input before its worker sees it** — `spec/flow-definition.md`
|
|
2430
|
+
v0.18, [ADR 0051](doc/decisions/0051-a-transform-is-a-declared-expression-in-a-registered-language-applied-in-the-types-worker-wrapper.md),
|
|
2431
|
+
[W61](doc/work-items/w61-what-a-transform-is.md). When the value a typed node receives is not the
|
|
2432
|
+
shape its worker reads — a join's collection where the worker wants a total, an item where it wants
|
|
2433
|
+
one field — the node declares `transform: { lang, expr }` instead of a reshaping task. **The language
|
|
2434
|
+
is a registry entry**, `Registry.languages`, the fifth set: `{ compile, worker }`, where `compile`
|
|
2435
|
+
runs on the main thread at construction and THROWS on an expression it refuses (so a SYNTAX error in
|
|
2436
|
+
the expression is `FD-V36` at `new Driver`, never a failure on the one workload that reaches the
|
|
2437
|
+
node — a misspelt KEY is not: it compiles and evaluates to `undefined`, which is P18's and P19's
|
|
2438
|
+
to catch),
|
|
2439
|
+
and `worker` is the URL of a module the envelope worker loads in `setup()` and evaluates with in
|
|
2440
|
+
`run()` — two halves, because a function cannot cross to a thread and a URL can. The expression sees
|
|
2441
|
+
the node's input as `$` and its settings as `$settings`; the worker receives `{ settings, input }`
|
|
2442
|
+
with `input` REPLACED by the result. **Evaluation happens in the type's worker wrapper, never in the
|
|
2443
|
+
driver**, so a transform is invisible to anything that reads what the driver dispatched. **A transform
|
|
2444
|
+
that cannot be evaluated FAILS THE TASK** — `TransformFailed`, naming the language, the expression and
|
|
2445
|
+
the language's own error code — and takes the two roads any worker throw takes: an `on: ["failed"]`
|
|
2446
|
+
edge on an `optional` node, `user_action` on a non-optional one. A result of `undefined` is not a
|
|
2447
|
+
failure. And a language module that failed to LOAD is not a worker error either: the wrapper
|
|
2448
|
+
remembers the failure and fails, at `run()`, exactly the tasks that name that language. `$now`,
|
|
2449
|
+
`$millis` and `$random` are unbound in every adapter: a transform is a function of what the run
|
|
2450
|
+
produced. The language this package ships is JSONata, as the sub-path `fyflow-orchestration/jsonata`
|
|
2451
|
+
— section 14; the snippet below uses a toy one so it loads no engine, and it is the same interface.
|
|
2452
|
+
|
|
2453
|
+
```ts run
|
|
2454
|
+
import { validateDefinition, type FlowDefinition, type LanguageAdapter, type NodeTypeDeclaration, type Registry, type WorkerEntry } from 'fyflow-orchestration';
|
|
2455
|
+
import { EnvelopeWorker, TransformFailed } from 'fyflow-orchestration/scheduler';
|
|
2456
|
+
|
|
2457
|
+
// A toy language: an expression is a dotted key path into the input, or `$settings.<path>`.
|
|
2458
|
+
// The REGISTRY half compiles synchronously and throws on anything else — what FD-V36 reports.
|
|
2459
|
+
// The WORKER half is a module URL (here a data: URL) whose default export compiles the same text
|
|
2460
|
+
// and evaluates it; `evaluate` may return a Promise, as JSONata's does.
|
|
2461
|
+
const GRAMMAR = /^(\$settings\.)?[A-Za-z0-9_]+(\.[A-Za-z0-9_]+)*$/;
|
|
2462
|
+
const workerSrc = `export default function compile(expr) {
|
|
2463
|
+
const segs = expr.split('.'); const fromSettings = segs[0] === '$settings'; const walk = fromSettings ? segs.slice(1) : segs;
|
|
2464
|
+
return { evaluate(input, settings) {
|
|
2465
|
+
let v = fromSettings ? settings : input;
|
|
2466
|
+
for (const k of walk) { if (v == null) return undefined; if (typeof v !== 'object') { const e = new Error('cannot read "' + k + '" of ' + typeof v); e.code = 'P0002'; throw e; } v = v[k]; }
|
|
2467
|
+
return v;
|
|
2468
|
+
} };
|
|
2469
|
+
}`;
|
|
2470
|
+
const path: LanguageAdapter = {
|
|
2471
|
+
compile: (expr) => { if (!GRAMMAR.test(expr)) throw new SyntaxError(`not a dotted key path: ${expr}`); },
|
|
2472
|
+
worker: `data:text/javascript,${encodeURIComponent(workerSrc)}`,
|
|
2473
|
+
};
|
|
2474
|
+
|
|
2475
|
+
// A worker that echoes its input, as a data: URL, and two types over it.
|
|
2476
|
+
const echo: WorkerEntry = { url: 'data:text/javascript,export default class W { run(p) { return p.input; } }' };
|
|
2477
|
+
const Scorer: NodeTypeDeclaration = { worker: 'echo' };
|
|
2478
|
+
const flow: FlowDefinition = {
|
|
2479
|
+
flowId: 'transformed', version: 1, entry: 'ingest',
|
|
2480
|
+
nodes: [
|
|
2481
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'score' }] },
|
|
2482
|
+
// The worker wants the confidence alone; the transform picks it out of the previous result.
|
|
2483
|
+
{ id: 'score', kind: 'task', type: 'Scorer', settings: { factor: 10 }, transform: { lang: 'path', expr: 'confidence' }, edges: [{ to: 'done' }] },
|
|
2484
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2485
|
+
],
|
|
2486
|
+
};
|
|
2487
|
+
const registry: Registry = { predicates: [], workers: { echo }, nodeTypes: { IngestWorker: { worker: 'echo' }, Scorer }, languages: { path } };
|
|
2488
|
+
const bad = validateDefinition(flow, registry);
|
|
2489
|
+
if (bad.length) throw new Error(`the transformed flow is invalid: ${bad.map((f) => `${f.rule}@${f.at}`).join()}`);
|
|
2490
|
+
|
|
2491
|
+
// The three refusals, each at construction.
|
|
2492
|
+
const rules = (node: object, reg = registry) => validateDefinition({ ...flow, nodes: [flow.nodes[0]!, node as never, flow.nodes[2]!] }, reg).map((f) => f.rule);
|
|
2493
|
+
const score = (transform: unknown, rest: object = { type: 'Scorer' }) => ({ id: 'score', kind: 'task', transform, edges: [{ to: 'done' }], ...rest });
|
|
2494
|
+
if (!rules(score({ lang: 'jsonata', expr: 'confidence' })).includes('FD-V35')) throw new Error('FD-V35: a language the registry does not name');
|
|
2495
|
+
if (!rules(score({ lang: 'path', expr: 'confidence' }), { ...registry, languages: undefined }).includes('FD-V35')) throw new Error('FD-V35 with languages omitted — absent means EMPTY');
|
|
2496
|
+
if (!rules(score({ lang: 'path', expr: '$sum(*.score' })).includes('FD-V36')) throw new Error('FD-V36: an expression the language`s compile refuses');
|
|
2497
|
+
if (!rules(score({ lang: 'path' })).includes('FD-V37')) throw new Error('FD-V37: not { lang, expr }');
|
|
2498
|
+
// And there is no untyped road for a transform to sit on: a node with no `type` is FD-V23, with the transform's own clauses applied beside it.
|
|
2499
|
+
if (!rules(score({ lang: 'path', expr: 'confidence' }, {})).includes('FD-V23')) throw new Error('FD-V23: a transform on a node with no type is a node with no type');
|
|
2500
|
+
|
|
2501
|
+
// And what the wrapper does with it — the class the bootstrap helper puts around your worker,
|
|
2502
|
+
// driven here directly. The helper writes `languages` into a typed pool's config for you when you
|
|
2503
|
+
// pass it the flows and the languages: `bootstrap(registry, groups, { languages: { path }, flows: [flow] })`.
|
|
2504
|
+
const w = new EnvelopeWorker({ worker: echo.url, languages: { path: path.worker } });
|
|
2505
|
+
await w.setup();
|
|
2506
|
+
const seen = await w.run({ settings: { factor: 10 }, input: { confidence: 0.9, text: 'a bb ccc' }, transform: { lang: 'path', expr: 'confidence' } });
|
|
2507
|
+
if (seen !== 0.9) throw new Error(`the worker received ${JSON.stringify(seen)} — not the transformed input`);
|
|
2508
|
+
const bound = await w.run({ settings: { factor: 10 }, input: { confidence: 0.9 }, transform: { lang: 'path', expr: '$settings.factor' } });
|
|
2509
|
+
if (bound !== 10) throw new Error('the node`s settings are bound as $settings');
|
|
2510
|
+
const plain = await w.run({ settings: { factor: 10 }, input: { confidence: 0.9 } });
|
|
2511
|
+
if (JSON.stringify(plain) !== JSON.stringify({ confidence: 0.9 })) throw new Error('no transform, the value received, untouched');
|
|
2512
|
+
let failed: unknown;
|
|
2513
|
+
try { await w.run({ settings: {}, input: { a: 5 }, transform: { lang: 'path', expr: 'a.b' } }); } catch (e) { failed = e; }
|
|
2514
|
+
if (!(failed instanceof TransformFailed) || failed.code !== 'P0002') throw new Error('a loud evaluation failure is a TransformFailed carrying the language`s own code — the TASK fails');
|
|
2515
|
+
await w.teardown();
|
|
2516
|
+
```
|
|
2517
|
+
|
|
2518
|
+
**What this is not.** What a type says about what its worker EXPECTS and RETURNS — input and
|
|
2519
|
+
output schemas, and a sample-based check across a transformed edge — was
|
|
2520
|
+
[P16](doc/plans/p16-what-a-node-type-is-and-what-the-library-owes-the-scheduler.md)'s plan after
|
|
2521
|
+
this one (P18) and is now [P19](doc/plans/p19-what-a-schema-buys-an-author.md)'s, answered in
|
|
2522
|
+
**section 15**: since v0.19 a worker DECLARES `inputSchema` and `outputSchema`, no RUN enforces
|
|
2523
|
+
them per settlement (that is OUT of P19 as a default) — since v0.21 the two values no static check
|
|
2524
|
+
can reach are held to them ONCE each, `FD-R4` (section 9's `BoundaryRefused`), and `checkWiring(registry, flow)`
|
|
2525
|
+
gives every arc of a flow one of five verdicts from them before any run (W73), with `pathsOf`,
|
|
2526
|
+
`tryTransform` and `signatureOf` beside it as the authoring assists (W74) — and since v0.20 any schema may carry
|
|
2527
|
+
`examples` — an annotation held to its schema at construction ([ADR 0056](doc/decisions/0056-an-example-is-an-annotation-validated-against-the-schema-it-sits-on.md),
|
|
2528
|
+
[W72](doc/work-items/w72-the-contract-an-example-beside-every-schema-and-a-worker-run-on-it.md)) —
|
|
2529
|
+
so that `npm run check:workers` can RUN a worker on its input example and validate what it returned
|
|
2530
|
+
against its output schema, which is an authoring-time check and never a driver behaviour; the
|
|
2531
|
+
adapter's optional `check` hook is the door left for the rest. A transform on a
|
|
2532
|
+
`map`'s `over`, on an edge or on a decision is refused, with the reasons in ADR 0051. The validator runs at construction and
|
|
2533
|
+
nowhere else: the driver holds your definition by reference, so a document you mutate after
|
|
2534
|
+
`new Driver` is dispatched as mutated, for every key ([Q259](doc/questions/q259.md)). Every one of
|
|
2535
|
+
the twenty-five worked flows in [`examples/`](examples/README.md) runs typed on a real scheduler
|
|
2536
|
+
since v0.19 — three of them with a declared `transform` (section 14), one at three arities, one
|
|
2537
|
+
whose transforms fail, and P18's own sample reducing a join's record for the node after it.
|
|
2538
|
+
*(Twenty-four until [W67](doc/work-items/w67-a-flow-from-documents-on-real-workers.md) added
|
|
2539
|
+
`triage.flow.json`, which is that plan's exit artifact: a flow and the runtime it needs from two
|
|
2540
|
+
JSON documents and a shim.)*
|
|
2541
|
+
|
|
2542
|
+
## 14. The JSONata adapter — `fyflow-orchestration/jsonata`
|
|
2543
|
+
|
|
2544
|
+
The language this package ships for a `transform` is **JSONata**, at the one exact version
|
|
2545
|
+
`package.json` names, as the sub-path
|
|
2546
|
+
`fyflow-orchestration/jsonata` — [ADR 0052](doc/decisions/0052-a-second-dependency-is-declared-where-its-one-sub-path-is-reached.md),
|
|
2547
|
+
[W62](doc/work-items/w62-the-jsonata-adapter.md). **The engine is a dependency of this package and
|
|
2548
|
+
is loaded by that sub-path alone**: the root and `fyflow-orchestration/scheduler` import none of it,
|
|
2549
|
+
so a consumer who never transforms pays the install and never the load, and a browser bundle carries
|
|
2550
|
+
the engine only where the sub-path is imported (237 KiB). Import it, put it in the registry's
|
|
2551
|
+
`languages` under the name a flow's `transform.lang` uses, and hand the same object to the
|
|
2552
|
+
bootstrap helper:
|
|
2553
|
+
|
|
2554
|
+
```ts check
|
|
2555
|
+
import { Driver, liveToken, type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
2556
|
+
import { bootstrap } from 'fyflow-orchestration/scheduler';
|
|
2557
|
+
import { jsonata } from 'fyflow-orchestration/jsonata';
|
|
2558
|
+
import { ConcurrentLimitGroup, FyflowScheduler } from 'fyflow-scheduler';
|
|
2559
|
+
|
|
2560
|
+
declare const registry: Pick<RegistryDocument, 'workers' | 'nodeTypes'>;
|
|
2561
|
+
declare const flow: FlowDefinition;
|
|
2562
|
+
|
|
2563
|
+
// The adapter is `{ compile, worker }`: `compile` runs on the main thread at `new Driver` and
|
|
2564
|
+
// `worker` is the URL of the SAME module, which the envelope worker imports inside the pool.
|
|
2565
|
+
const languages = { jsonata };
|
|
2566
|
+
const built = bootstrap(registry, { scorer: new ConcurrentLimitGroup(4, 'scorer') }, { languages, flows: [flow] });
|
|
2567
|
+
const scheduler = new FyflowScheduler(built.workerPools, built.resourceGroups);
|
|
2568
|
+
const driver = new Driver(flow, {
|
|
2569
|
+
runId: 'run-1', scheduler, predicates: new Map(), joinPolicy: liveToken,
|
|
2570
|
+
...registry, groups: Object.keys(scheduler.groups), languages,
|
|
2571
|
+
});
|
|
2572
|
+
driver.start({ items: [1, 2, 3] });
|
|
2573
|
+
```
|
|
2574
|
+
|
|
2575
|
+
**What an expression sees, and what it may not.** `$` is the node's input — the previous node's
|
|
2576
|
+
result, the object `needs` described, a map instance's own item, or a join's collection keyed by
|
|
2577
|
+
branch path — and the node's effective `settings` are bound as `$settings`, so one expression is
|
|
2578
|
+
parameterised per node. **Five names are unbound, by name, on every compiled expression**: `$now`,
|
|
2579
|
+
`$millis`, `$random` — the three every adapter must refuse, because a transform is a function of what
|
|
2580
|
+
the run produced and run state is a fold of the log — **`$shuffle`**, JSONata's own fourth, which
|
|
2581
|
+
draws from `Math.random()` (found by walking the engine's function table: of the 61 of its 63
|
|
2582
|
+
built-ins the walk can observe, nothing else differs between two evaluations of one input) — **and
|
|
2583
|
+
`$eval`**, for a different reason: it re-compiles its
|
|
2584
|
+
argument at RUN time, so a syntax error inside it would be a task failure carrying the engine's
|
|
2585
|
+
`D3120` that `FD-V36` never saw, and a transform is declared text refused at construction
|
|
2586
|
+
([Q265](doc/questions/q265.md)). Each throws naming itself and the task fails with
|
|
2587
|
+
`TransformFailed [UNBOUND]`; a computed key is read with `$lookup`, which recompiles nothing. **And the worker receives plain JSON values**: JSONata marks the
|
|
2588
|
+
arrays it builds with an own `sequence` property that survives `structuredClone` across a thread and
|
|
2589
|
+
that a deep-equal sees; the adapter strips it at every depth.
|
|
2590
|
+
|
|
2591
|
+
**Two things about JSONata itself that decide how you write a reduction.** A path over a join's
|
|
2592
|
+
record changes TYPE with the region's arity — `*.score` is an array at two matching branches, the
|
|
2593
|
+
scalar at one, absent at zero ([Q263](doc/questions/q263.md)) — so a reduction is written over the
|
|
2594
|
+
array form, `$sum([*.score])`, which is `0` for a join nothing reached. And a misspelt key is silent:
|
|
2595
|
+
`input.scroe` evaluates to `undefined` with no error, and the worker receives `undefined`, where
|
|
2596
|
+
`$sum("a")` is loud (`T0412`). Only a syntax error is refused at construction.
|
|
2597
|
+
|
|
2598
|
+
```ts run
|
|
2599
|
+
import { validateDefinition, type FlowDefinition, type NodeTypeDeclaration, type Registry, type WorkerEntry } from 'fyflow-orchestration';
|
|
2600
|
+
import { EnvelopeWorker, TransformFailed } from 'fyflow-orchestration/scheduler';
|
|
2601
|
+
import { jsonata } from 'fyflow-orchestration/jsonata';
|
|
2602
|
+
|
|
2603
|
+
const echo: WorkerEntry = { url: 'data:text/javascript,export default class W { run(p) { return p.input; } }' };
|
|
2604
|
+
const Scorer: NodeTypeDeclaration = { worker: 'echo' };
|
|
2605
|
+
const flow: FlowDefinition = {
|
|
2606
|
+
flowId: 'reduced', version: 1, entry: 'ingest',
|
|
2607
|
+
nodes: [
|
|
2608
|
+
{ id: 'ingest', kind: 'task', type: 'IngestWorker', edges: [{ to: 'split' }] },
|
|
2609
|
+
{ id: 'split', kind: 'fanout', branches: ['scoreA', 'scoreB'] },
|
|
2610
|
+
{ id: 'scoreA', kind: 'task', type: 'IngestWorker', edges: [{ to: 'collect' }] },
|
|
2611
|
+
{ id: 'scoreB', kind: 'task', type: 'IngestWorker', edges: [{ to: 'collect' }] },
|
|
2612
|
+
{ id: 'collect', kind: 'join', of: 'split', edges: [{ to: 'publish' }] },
|
|
2613
|
+
// The join's successor wants the total, not the record: the transform reduces it, no reshaping node.
|
|
2614
|
+
{ id: 'publish', kind: 'task', type: 'Scorer', settings: { factor: 2 }, transform: { lang: 'jsonata', expr: '$sum([*.score]) * $settings.factor' }, edges: [{ to: 'done' }] },
|
|
2615
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2616
|
+
],
|
|
2617
|
+
};
|
|
2618
|
+
const registry: Registry = { predicates: [], workers: { echo }, nodeTypes: { IngestWorker: { worker: 'echo' }, Scorer }, languages: { jsonata } };
|
|
2619
|
+
const bad = validateDefinition(flow, registry);
|
|
2620
|
+
if (bad.length) throw new Error(`the reduced flow is invalid: ${bad.map((f) => `${f.rule}@${f.at}`).join()}`);
|
|
2621
|
+
|
|
2622
|
+
// FD-V36 carries JSONata's own refusal — its code and the position of the missing token — at `new Driver`, never on a workload.
|
|
2623
|
+
const publish = (expr: string, reg = registry) => validateDefinition({ ...flow, nodes: flow.nodes.map((n) => (n.id === 'publish' ? { ...n, transform: { lang: 'jsonata', expr } } : n)) }, reg);
|
|
2624
|
+
const syntax = publish('$sum(*.score');
|
|
2625
|
+
if (syntax.map((f) => f.rule).join() !== 'FD-V36' || !syntax[0]!.detail.includes('S0203') || !syntax[0]!.detail.includes('position 12')) throw new Error(`FD-V36 with the engine's code and position: ${JSON.stringify(syntax)}`);
|
|
2626
|
+
// FD-V35 — the language is a registry entry: a registry without it refuses the flow, and an absent set means none.
|
|
2627
|
+
if (!publish('$sum([*.score])', { ...registry, languages: {} }).some((f) => f.rule === 'FD-V35')) throw new Error('FD-V35: jsonata not registered');
|
|
2628
|
+
|
|
2629
|
+
// The envelope worker — the class the bootstrap helper puts around your worker — with the adapter's worker half by URL.
|
|
2630
|
+
const w = new EnvelopeWorker({ worker: echo.url, languages: { jsonata: jsonata.worker } });
|
|
2631
|
+
await w.setup();
|
|
2632
|
+
const record = { '0': { score: 40 }, '1': { score: 20 } }; // JP-R12's collection, two branches
|
|
2633
|
+
const total = await w.run({ settings: { factor: 2 }, input: record, transform: { lang: 'jsonata', expr: '$sum([*.score]) * $settings.factor' } });
|
|
2634
|
+
if (total !== 120) throw new Error(`the worker received ${JSON.stringify(total)} — expected the reduction, 120`);
|
|
2635
|
+
const one = await w.run({ settings: {}, input: { '0': { score: 7 } }, transform: { lang: 'jsonata', expr: '$sum([*.score])' } });
|
|
2636
|
+
if (one !== 7) throw new Error('arity one, the same expression: 7');
|
|
2637
|
+
const none = await w.run({ settings: {}, input: {}, transform: { lang: 'jsonata', expr: '$sum([*.score])' } });
|
|
2638
|
+
if (none !== 0) throw new Error('a join nothing reached: 0, because the reduction is over the ARRAY form');
|
|
2639
|
+
let unbound: unknown;
|
|
2640
|
+
try { await w.run({ settings: {}, input: record, transform: { lang: 'jsonata', expr: '$now()' } }); } catch (e) { unbound = e; }
|
|
2641
|
+
if (!(unbound instanceof TransformFailed) || unbound.code !== 'UNBOUND') throw new Error('$now() is unbound: the TASK fails, naming it');
|
|
2642
|
+
await w.teardown();
|
|
2643
|
+
```
|
|
2644
|
+
|
|
2645
|
+
**How it is declared, and what that costs a consumer.** `package.json` names `jsonata` in
|
|
2646
|
+
`dependencies` at one EXACT version and `deno.json` maps it to `npm:jsonata@` that same version
|
|
2647
|
+
(there is no JSR package for it) — one pin, written twice, and `check:source` `S15` holds the two
|
|
2648
|
+
together and refuses a second field naming the engine;
|
|
2649
|
+
`S14` refuses any published file but `src/jsonata/**` importing the engine, and any form of the
|
|
2650
|
+
specifier there but the bare `jsonata`. An npm consumer installs one 834 KiB package with no
|
|
2651
|
+
dependencies of its own, whether or not they ever import the sub-path; a Deno consumer's
|
|
2652
|
+
`deno.json` resolves the mapping; a bundler includes the engine only in a bundle that imports the
|
|
2653
|
+
sub-path. **A bundled worker is your asset**: inside a bundle `import.meta.url` is the bundle's own
|
|
2654
|
+
URL ([Q261](doc/questions/q261.md)), so `jsonata.worker` — like the envelope worker itself — has to be
|
|
2655
|
+
served beside your bundle as a module of its own for a pool's thread to load it.
|
|
2656
|
+
|
|
2657
|
+
---
|
|
2658
|
+
|
|
2659
|
+
## 15. What a schema buys an AUTHOR — an authoring session
|
|
2660
|
+
|
|
2661
|
+
**The schemas in section 13 were never for the run.** A worker's `inputSchema` and `outputSchema`
|
|
2662
|
+
are held to the run's PAYLOAD once and to a person's RESOLUTION once each (`FD-R4`, section 9) and
|
|
2663
|
+
never per settlement — and neither of those is *two values per run*, because a fan-out or map entry
|
|
2664
|
+
validates one value per branch task it reaches and a run nobody parks validates no resolution at
|
|
2665
|
+
all. So what are they for? **They are for whoever is wiring the flow**, a person or an agent, and this section is
|
|
2666
|
+
the session that shows it: a flow written, a MISWIRE caught before anything runs, a transform
|
|
2667
|
+
written by a loop rather than guessed, and a worker typed against a signature generated from the
|
|
2668
|
+
declarations. Three functions do it, all exported from the package root, and none of them refuses
|
|
2669
|
+
anything or is reachable from a run. Two are pure over data; `tryTransform` is not, and the
|
|
2670
|
+
difference matters to a caller — it is `checkWiring`'s one dynamic `import()` of the language
|
|
2671
|
+
adapter's worker half, so it loads a module by URL and returns a promise:
|
|
2672
|
+
|
|
2673
|
+
| | answers |
|
|
2674
|
+
|---|---|
|
|
2675
|
+
| `pathsOf(schema)` | *what is there to reach for* — every path a value may carry, and which are guaranteed |
|
|
2676
|
+
| `tryTransform(registry, flow, node, transform)` | *does THIS expression fit* — one candidate, one verdict, on a real sample |
|
|
2677
|
+
| `signatureOf(name, worker)` | *what do I write my worker against* — TypeScript, as text |
|
|
2678
|
+
|
|
2679
|
+
The registry below is
|
|
2680
|
+
[`examples/getting-started/session-registry.ts`](examples/getting-started/session-registry.ts) — two
|
|
2681
|
+
workers whose schemas OPPOSE on purpose. `reader` returns `{ lines: string[], source }` and closes
|
|
2682
|
+
the object; `summarise` REQUIRES a `text` string and closes its own. Everything here runs on every
|
|
2683
|
+
`npm run gate`, and the same snippet is in `README.md`, held byte-equal by `check-docs`'s `X8`.
|
|
2684
|
+
|
|
2685
|
+
```ts run
|
|
2686
|
+
import { checkWiring, describePaths, pathsOf, signatureOf, tryTransform, type FlowDefinition } from 'fyflow-orchestration';
|
|
2687
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
2688
|
+
import { jsonata } from 'fyflow-orchestration/jsonata';
|
|
2689
|
+
import { SESSION_BASE, SESSION_REGISTRY, SESSION_WORKERS } from '../../examples/getting-started/session-registry.ts'; // in your program: './registry.ts', a module or a JSON file of your own
|
|
2690
|
+
|
|
2691
|
+
// The three things the check needs from a registry: the workers' schemas, the types that name them,
|
|
2692
|
+
// and the languages a transform may be written in. `createRuntime` takes the whole document below.
|
|
2693
|
+
const registry = { workers: SESSION_WORKERS, nodeTypes: SESSION_REGISTRY.nodeTypes, languages: { jsonata } };
|
|
2694
|
+
|
|
2695
|
+
// THE FLOW, as an author first writes it: read a batch, summarise it. It is a VALID document —
|
|
2696
|
+
// `validateDefinition` has nothing to say about it, because nothing here is about the grammar.
|
|
2697
|
+
const flow: FlowDefinition = {
|
|
2698
|
+
flowId: 'session', version: 1, entry: 'read',
|
|
2699
|
+
nodes: [
|
|
2700
|
+
{ id: 'read', kind: 'task', type: 'Reader', edges: [{ to: 'sum' }] },
|
|
2701
|
+
{ id: 'sum', kind: 'task', type: 'Summarise', edges: [{ to: 'done' }] },
|
|
2702
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2703
|
+
],
|
|
2704
|
+
};
|
|
2705
|
+
|
|
2706
|
+
// STEP 1 — the wiring check, before a worker has run once. Every arc gets one of five verdicts.
|
|
2707
|
+
const first = await checkWiring(registry, flow);
|
|
2708
|
+
console.log(first.counts); // { fits: 0, no: 1, ... }
|
|
2709
|
+
const arc = first.verdicts.find((v) => v.consumer === 'sum')!;
|
|
2710
|
+
console.log(`${arc.kind}: ${arc.why}`); // no: $: consumer requires "text", producer does not guarantee it
|
|
2711
|
+
if (arc.kind !== 'no') throw new Error(`the miswire is the point of this section: got ${arc.kind}`);
|
|
2712
|
+
|
|
2713
|
+
// STEP 2 — so what IS there? The producer's own output schema, enumerated. A leading `?` marks a
|
|
2714
|
+
// path a value may not carry at all; the rest are guaranteed by `required`.
|
|
2715
|
+
console.log(describePaths(pathsOf(SESSION_WORKERS.reader.outputSchema)));
|
|
2716
|
+
// $ object
|
|
2717
|
+
// $.lines array
|
|
2718
|
+
// ? $.lines[] string
|
|
2719
|
+
// $.source string
|
|
2720
|
+
|
|
2721
|
+
// STEP 3 — a first candidate. It is run on the producer's declared EXAMPLE, and the result is held
|
|
2722
|
+
// to the consumer's input schema, so the answer is about a value rather than about two declarations.
|
|
2723
|
+
const attempt = await tryTransform(registry, flow, 'sum', { lang: 'jsonata', expr: '{ "text": lines }' });
|
|
2724
|
+
console.log(`${attempt[0]!.kind}: ${attempt[0]!.why}`);
|
|
2725
|
+
// transform-fails: the transform's result on the example fails the consumer's inputSchema — $.text expected string, got array
|
|
2726
|
+
if (attempt[0]!.kind !== 'transform-fails') throw new Error('an array is not a string, and the loop says which');
|
|
2727
|
+
|
|
2728
|
+
// STEP 4 — ONE expression edit, and the same loop. `transform-passes` is never `fits`: it is a fact
|
|
2729
|
+
// about the sample the verdict names, which is why `sample.source` is on it (a declared example
|
|
2730
|
+
// here; where a schema declares none, a generated WITNESS, and the verdict says so).
|
|
2731
|
+
const good = { lang: 'jsonata', expr: '{ "text": $join(lines, " ") }' };
|
|
2732
|
+
const passed = await tryTransform(registry, flow, 'sum', good);
|
|
2733
|
+
console.log(`${passed[0]!.kind} on the ${passed[0]!.sample!.source}: ${JSON.stringify(passed[0]!.sample!.result)}`);
|
|
2734
|
+
if (passed[0]!.kind !== 'transform-passes') throw new Error(`expected the sample to pass, got ${passed[0]!.kind}`);
|
|
2735
|
+
|
|
2736
|
+
// STEP 5 — the signature the worker is written against, generated from the same three schemas.
|
|
2737
|
+
// `examples/getting-started/session-types.ts` is this text, checked in, and `summarise.ts` imports
|
|
2738
|
+
// it. A type holds the SHAPE: `words` is `number` here and its schema says `minimum: 0`.
|
|
2739
|
+
const signature = signatureOf('Summarise', SESSION_WORKERS.summarise);
|
|
2740
|
+
console.log(signature.split('\n').filter((l) => l.startsWith('export')).join('\n'));
|
|
2741
|
+
if (!signature.includes('"text": string;')) throw new Error('the required key is required in the type too');
|
|
2742
|
+
|
|
2743
|
+
// STEP 6 — and only now the runtime. Pools are built ONCE, so a flow whose transform was decided
|
|
2744
|
+
// during the session has to reach `createRuntime` as `flows`, or `start` throws `LanguageNotBuilt`
|
|
2745
|
+
// naming the type. Check, then build.
|
|
2746
|
+
const wired: FlowDefinition = { ...flow, nodes: flow.nodes.map((n) => (n.id === 'sum' ? { ...n, transform: good } : n)) };
|
|
2747
|
+
const runtime = createRuntime(SESSION_REGISTRY, { languages: { jsonata } }, { base: SESSION_BASE, flows: [wired] });
|
|
2748
|
+
const state = await runtime.start(wired, { source: 'inbox' }).done;
|
|
2749
|
+
await runtime.shutdown();
|
|
2750
|
+
console.log(state.outcome, Object.values(state.nodes).find((n) => n.nodeId === 'sum')?.result); // completed { words: 3 }
|
|
2751
|
+
|
|
2752
|
+
// The expected values are the DOCUMENT's, not literals beside the run: the consumer's own declared
|
|
2753
|
+
// output example, and the producer's declared line count. A registry whose examples stop describing
|
|
2754
|
+
// its workers is red here and in `test/definition/assist.test.ts`, which runs the same chain.
|
|
2755
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
2756
|
+
const got = Object.values(state.nodes).find((n) => n.nodeId === 'sum')?.result;
|
|
2757
|
+
if (JSON.stringify(got) !== JSON.stringify(SESSION_WORKERS.summarise.outputSchema.examples[0])) {
|
|
2758
|
+
throw new Error(`the run returned ${JSON.stringify(got)}, not the consumer's declared output example`);
|
|
2759
|
+
}
|
|
2760
|
+
if ((got as { words: number }).words !== SESSION_WORKERS.reader.outputSchema.examples[0].lines.length) {
|
|
2761
|
+
throw new Error(`the producer declares ${SESSION_WORKERS.reader.outputSchema.examples[0].lines.length} lines, joined into one text`);
|
|
2762
|
+
}
|
|
2763
|
+
const after = await checkWiring(registry, wired);
|
|
2764
|
+
if (after.counts.no !== 0) throw new Error('and the flow that ran has no unwired arc left');
|
|
2765
|
+
```
|
|
2766
|
+
|
|
2767
|
+
**What a verdict is and is not.** `fits` is a statement about two DECLARATIONS and is checked
|
|
2768
|
+
statically over the twelve constraining keywords of the subset. `transform-passes` is a statement
|
|
2769
|
+
about ONE VALUE — the producer's first declared example, or a generated witness where it declares
|
|
2770
|
+
none — and the verdict carries which it ran on and, over a region, the ARITY the sample had, because
|
|
2771
|
+
a transform's result changes type with the arity ([Q263](doc/questions/q263.md)). **A witness is not
|
|
2772
|
+
a proof**, and the reference says *the sample passed* rather than *fits* for exactly that reason.
|
|
2773
|
+
Two more the check knows and a document cannot say: an OPEN producer is read as never emitting a key
|
|
2774
|
+
it does not name **unless it REQUIRES that key**, which is a stated leniency and its one scope
|
|
2775
|
+
([Q305](doc/questions/q305.md); the scope is W100's, and `required` is what refutes the leniency's own
|
|
2776
|
+
premise); and a join successor's
|
|
2777
|
+
record is the check's own knowledge, because the subset cannot spell *any key, each of shape S*
|
|
2778
|
+
([Q306](doc/questions/q306.md)).
|
|
2779
|
+
|
|
2780
|
+
**One document per verdict** lives in [`spec/examples/wiring/`](spec/examples/wiring/), each named
|
|
2781
|
+
by the verdict it earns — `fits`, `no`, `transform-passes`, `transform-fails` and `unchecked`, the
|
|
2782
|
+
fifth being what the check says when a side declares no schema at all. `spec/flow-definition.md`
|
|
2783
|
+
section 7 records their verdicts and the suite compares the directory and the table both ways.
|
|
2784
|
+
|
|
2785
|
+
**And the generated signature is a RECIPE rather than enforcement.** `signatureOf` emits text from
|
|
2786
|
+
the subset with no dependency; `json-schema-to-typescript` is the richer consumer tool and reads the
|
|
2787
|
+
same schemas. **The two AGREE wherever a schema says something** — an open object with a property
|
|
2788
|
+
gets `[k: string]: unknown` from both — and they differ wherever it says LESS: an empty or
|
|
2789
|
+
annotation-only schema is `unknown` here and an OBJECT type there, and a closed object with no
|
|
2790
|
+
properties is `Record<string, never>` here and `{}` there, which in TypeScript means *any non-nullish
|
|
2791
|
+
value*. Both shapes are in the corpus. *A schema that names no `type` admits every type, and this
|
|
2792
|
+
emits the type that says so* — the same reading `pathsOf` gives it as `*`. *(This paragraph said the
|
|
2793
|
+
opposite — that the two differ on the OPEN object — in the reference, in the module header and in the
|
|
2794
|
+
item's own fact: two runs of one harness, one of which CONFIGURED the generator to close what the
|
|
2795
|
+
schema leaves open, and the later run's line count was kept beside the earlier run's conclusion.
|
|
2796
|
+
W74's conformity reader, from the author's own scratchpad output.)* Neither holds a
|
|
2797
|
+
`minimum`, a `maxLength` or a `minItems`: **a type catches the shape, and `npm run check:workers` —
|
|
2798
|
+
which RUNS each worker on its declared example and validates what came back — is what catches the
|
|
2799
|
+
rest.** Even that sees one value: a bound the code does not keep past the example is invisible to it
|
|
2800
|
+
and visible at every consumer the wiring check reads ([Q302](doc/questions/q302.md)).
|
|
2801
|
+
---
|
|
2802
|
+
|
|
2803
|
+
## 16. What a schema buys a RUN — holding a result to the schema its worker declares
|
|
2804
|
+
|
|
2805
|
+
**Section 15's last sentence is where this one starts.** A bound a worker's code does not keep past
|
|
2806
|
+
its declared example is invisible to every authoring-time tool here, and there is a second kind of
|
|
2807
|
+
gap beside it that no such tool could ever reach: **what the run CARRIES is not what the worker
|
|
2808
|
+
returned.** `OR-R2` clones every settled result into the log, and `structuredClone` copies a value's
|
|
2809
|
+
own **enumerable** string-keyed data and nothing else (`OR-R11`, section 9) — so a worker can return
|
|
2810
|
+
a value that satisfies its own `outputSchema` and the record can fail it.
|
|
2811
|
+
|
|
2812
|
+
**So a result may be held to that schema at run time, per settlement, and the answer is a READING.**
|
|
2813
|
+
Supply `onNonconformingResult` and every `done` settlement whose worker declares an `outputSchema`
|
|
2814
|
+
is checked; omit it and nothing is. **There is no boolean beside it — the callback's presence IS the
|
|
2815
|
+
switch**, because two members would be two sources that can disagree, and both disagreements are
|
|
2816
|
+
defects: a switch with no channel is a cost paid for a report nobody receives, and a channel with the
|
|
2817
|
+
switch off is a typed option silently dropped
|
|
2818
|
+
([ADR 0060](doc/decisions/0060-a-run-time-result-check-is-a-reading-and-never-a-refusal.md),
|
|
2819
|
+
`OR-R13`).
|
|
2820
|
+
|
|
2821
|
+
**A finding changes NOTHING about the run.** No record is written differently, no terminal state is
|
|
2822
|
+
authored, no dispatch moves, and the value reaches its successor unaltered — so the run completes
|
|
2823
|
+
exactly as it would with the check off. *You cannot audit a behaviour with an instrument that alters
|
|
2824
|
+
it*, and that is the whole reason this is not a refusal: what it buys you is **knowledge rather than
|
|
2825
|
+
protection**, and a value that fails its schema still flows downstream. The one way it can change a
|
|
2826
|
+
run is **your own callback throwing**, which is caught at the listener, reported through
|
|
2827
|
+
`onRunFailure` and stops the run — the treatment every callback here gets (section 9).
|
|
2828
|
+
|
|
2829
|
+
Three more things worth knowing before you switch it on:
|
|
2830
|
+
|
|
2831
|
+
| | |
|
|
2832
|
+
|---|---|
|
|
2833
|
+
| **`done` settlements only** | a failed or parked task has `result: null` (`OR-R2`), and an output schema no worker reached would refuse every one of them |
|
|
2834
|
+
| **settlements only, never a resolution** | a person's `done` value is held to the same schema at `resolve()`, once, as a REJECTED command (`FD-R4`, section 9). One value refused twice by two mechanisms with different answers is worse than either |
|
|
2835
|
+
| **a worker with no `outputSchema` is UNCHECKED** | *absent means unchecked*, exactly as `FD-R4` reads it |
|
|
2836
|
+
|
|
2837
|
+
*What it costs is a RATIO to the `structuredClone` the driver already pays on the same settlement,
|
|
2838
|
+
and on this repository's own corpus it stays UNDER it at every shape measured. The share FALLS as the
|
|
2839
|
+
value grows and RISES as the schema does — the clone is linear in the value and the check is linear in
|
|
2840
|
+
the schema, and `enum` is where a schema's size is unbounded, so one large `enum` puts the check OVER
|
|
2841
|
+
the clone. `spec/outcome-record.md` section 4 carries the figures with the method that took them. It
|
|
2842
|
+
is a ratio rather than a millisecond, deliberately, and the reason is visible in the one number
|
|
2843
|
+
`examples/l6.ts` PRINTS: over every settlement that rung makes, the same two values read about twice
|
|
2844
|
+
as dear in Node as in Deno. **A single figure would be a property of a machine**
|
|
2845
|
+
([Q179](doc/questions/q179.md), [Q335](doc/questions/q335.md)). (This paragraph said "about a third",
|
|
2846
|
+
which is the Node reading — W82's conformity reader, finding 12.)*
|
|
2847
|
+
|
|
2848
|
+
The worker below is
|
|
2849
|
+
[`examples/getting-started/drift.ts`](examples/getting-started/drift.ts) and it has **both** shapes
|
|
2850
|
+
in one return value, neither of which is a mistake anybody would call out in review. Everything here
|
|
2851
|
+
runs on every `npm run gate`. *(This sentence also said **and the same snippet is in `README.md`,
|
|
2852
|
+
held byte-equal by `check-docs`'s `X8`** until
|
|
2853
|
+
[W107](doc/work-items/w107-the-readme-is-for-a-person-who-will-use-the-library.md), which gave the
|
|
2854
|
+
three root documents one audience each
|
|
2855
|
+
([ADR 0073](doc/decisions/0073-the-readme-is-for-a-person-the-reference-is-for-an-agent.md)) and took
|
|
2856
|
+
this section's pairing out with the README section it mirrored. `X8` pairs three sections now — 2, 5
|
|
2857
|
+
and 15 — and each of those three still carries this claim and still earns it. Found by W107's
|
|
2858
|
+
conformity reader: the item edited every other document the change reaches and not this one, which is
|
|
2859
|
+
the document ADR 0073 designates as the agent's reference, about the mechanism the item changed.)*
|
|
2860
|
+
|
|
2861
|
+
```ts run
|
|
2862
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
2863
|
+
import { checkWiring, type FlowDefinition, type NonconformingResult, type RegistryDocument } from 'fyflow-orchestration';
|
|
2864
|
+
|
|
2865
|
+
// THE REGISTRY — one worker with both of its VALUE schemas declared, and one with none. The schemas
|
|
2866
|
+
// are the same objects the wiring check reads; what changes below is that a RUN reads them too.
|
|
2867
|
+
const registry: RegistryDocument = {
|
|
2868
|
+
workers: {
|
|
2869
|
+
drift: {
|
|
2870
|
+
url: '../../examples/getting-started/drift.ts', // in your program: './drift.ts'
|
|
2871
|
+
inputSchema: { type: 'object', properties: { text: { type: 'string' } }, examples: [{ text: 'alpha bb' }] },
|
|
2872
|
+
outputSchema: {
|
|
2873
|
+
type: 'object', required: ['words', 'load', 'label'], additionalProperties: false,
|
|
2874
|
+
properties: {
|
|
2875
|
+
words: { type: 'integer', minimum: 0 },
|
|
2876
|
+
load: { type: 'number', minimum: 0, maximum: 1 },
|
|
2877
|
+
label: { type: 'string' },
|
|
2878
|
+
},
|
|
2879
|
+
examples: [{ words: 2, load: 0.2, label: 'w2' }],
|
|
2880
|
+
},
|
|
2881
|
+
},
|
|
2882
|
+
inspect: { url: '../../examples/getting-started/inspect.ts' }, // no outputSchema: UNCHECKED
|
|
2883
|
+
},
|
|
2884
|
+
nodeTypes: { Drift: { worker: 'drift', inline: true }, Inspect: { worker: 'inspect', inline: true } },
|
|
2885
|
+
};
|
|
2886
|
+
|
|
2887
|
+
const flow: FlowDefinition = {
|
|
2888
|
+
flowId: 'drift', version: 1, entry: 'measure',
|
|
2889
|
+
nodes: [
|
|
2890
|
+
{ id: 'measure', kind: 'task', type: 'Drift', edges: [{ to: 'after' }] },
|
|
2891
|
+
{ id: 'after', kind: 'task', type: 'Inspect', edges: [{ to: 'done' }] },
|
|
2892
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
2893
|
+
],
|
|
2894
|
+
};
|
|
2895
|
+
|
|
2896
|
+
// EVERY authoring-time tool says this flow is fine, and both are right: one reads two SCHEMAS and
|
|
2897
|
+
// the other runs a worker on ONE declared example. Neither holds a record. The wiring check takes
|
|
2898
|
+
// the document's two relevant PARTS rather than the document (section 15).
|
|
2899
|
+
const wiring = await checkWiring({ workers: registry.workers, nodeTypes: registry.nodeTypes }, flow);
|
|
2900
|
+
console.log(wiring.counts.no); // 0
|
|
2901
|
+
if (wiring.counts.no !== 0) throw new Error('the static check has nothing to say about either defect');
|
|
2902
|
+
|
|
2903
|
+
// SUPPLY THE CALLBACK AND THE CHECK IS ON. There is no boolean beside it: its presence IS the
|
|
2904
|
+
// switch (ADR 0060). A finding is a READING — nothing is appended differently, no terminal state is
|
|
2905
|
+
// authored, no dispatch moves, and the value reaches the next node exactly as it would with the
|
|
2906
|
+
// check off.
|
|
2907
|
+
const found: NonconformingResult[] = [];
|
|
2908
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
2909
|
+
const state = await runtime.start(flow, { text: 'one two three four five six seven eight nine ten eleven' }, {
|
|
2910
|
+
onNonconformingResult: (f) => { found.push(f); },
|
|
2911
|
+
}).done;
|
|
2912
|
+
await runtime.shutdown();
|
|
2913
|
+
|
|
2914
|
+
console.log(state.outcome); // completed — a finding changes nothing
|
|
2915
|
+
// ONE report per settlement, carrying EVERY way that result failed, in document order.
|
|
2916
|
+
console.log(found.map((f) => `${f.nodeId} (${f.worker}): ${f.findings.map((x) => `${x.path} ${x.says}`).join('; ')}`));
|
|
2917
|
+
// [ 'measure (drift): $.label required; $.load above maximum 1' ]
|
|
2918
|
+
|
|
2919
|
+
if (state.outcome !== 'completed') throw new Error(`a finding is a reading: expected completed, got ${state.outcome}`);
|
|
2920
|
+
if (found.length !== 1) throw new Error(`one report per settlement, not one per finding: got ${found.length}`);
|
|
2921
|
+
if (found[0]?.worker !== 'drift' || found[0]?.type !== 'Drift') throw new Error(`a finding names the node, its type and the worker whose schema refused it: ${JSON.stringify(found[0])}`);
|
|
2922
|
+
|
|
2923
|
+
// `found` rather than `found[0]!`: a guard that raises a TypeError on an empty report is a CRASH and
|
|
2924
|
+
// not a detection, and every `includes` below would inherit it.
|
|
2925
|
+
const says = found.flatMap((f) => f.findings).map((x) => `${x.path} ${x.says}`).join('; ');
|
|
2926
|
+
// The BOUND is one the worker's own code does not keep past its declared domain: at the two words
|
|
2927
|
+
// its declared example carries, `load` is 0.2 and nothing is reported. `check-workers` runs that
|
|
2928
|
+
// example and is green; one example confirms a schema at ONE value.
|
|
2929
|
+
if (!says.includes('$.load above maximum 1')) throw new Error(`expected the bound to be reported, got ${says}`);
|
|
2930
|
+
// The MISSING KEY is not a defect in the worker at all. It returned `label`; `structuredClone`
|
|
2931
|
+
// copies own ENUMERABLE keys and nothing else (OR-R11), so the RECORD the run carries does not have
|
|
2932
|
+
// it — and what this check reads is the record. No authoring-time tool holds one.
|
|
2933
|
+
if (!says.includes('$.label required')) throw new Error(`expected the clone's loss to be reported, got ${says}`);
|
|
2934
|
+
|
|
2935
|
+
// AND THE VALUE FLOWED ON ANYWAY. `after` declares no `outputSchema`, so it is UNCHECKED; what it
|
|
2936
|
+
// reports is the key set it RECEIVED, which is the record's and not the worker's.
|
|
2937
|
+
const after = Object.values(state.nodes).find((n) => n.nodeId === 'after')?.result as { keys: string[] };
|
|
2938
|
+
console.log(after.keys); // [ 'load', 'words' ]
|
|
2939
|
+
if (after.keys.join() !== 'load,words') throw new Error(`the successor received ${after.keys.join()}`);
|
|
2940
|
+
|
|
2941
|
+
// OMIT THE CALLBACK AND NOTHING IS CHECKED — and the run is the same run. This is the property to
|
|
2942
|
+
// rely on: you can turn the reading on in one environment and off in another without changing what
|
|
2943
|
+
// the flow DOES, because an instrument that altered what it measures would be useless for auditing.
|
|
2944
|
+
const quiet = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
2945
|
+
const same = await quiet.start(flow, { text: 'one two three four five six seven eight nine ten eleven' }).done;
|
|
2946
|
+
await quiet.shutdown();
|
|
2947
|
+
const shape = (s: typeof state) => JSON.stringify(Object.values(s.nodes).map((n) => [n.nodeId, n.result]));
|
|
2948
|
+
if (same.outcome !== state.outcome || shape(same) !== shape(state)) {
|
|
2949
|
+
throw new Error('a finding is a reading: the same run with the check off must reach the same nodes with the same results');
|
|
2950
|
+
}
|
|
2951
|
+
```
|
|
2952
|
+
|
|
2953
|
+
**Where the two halves of this are proved, and it is not here.** Every guard in the snippet above
|
|
2954
|
+
can be made a tautology with `npm run check:docs` at exit 0 — `X4` counts a `throw` textually and
|
|
2955
|
+
refuses `if (false)` by name, and neither reaches `found.length !== found.length`
|
|
2956
|
+
([Q310](doc/questions/q310.md), measured on this section by W82's adequacy reader). **What holds the
|
|
2957
|
+
two claims is `examples/l6.ts`**, which runs the same two shapes as two separate workers on a real
|
|
2958
|
+
`FyflowScheduler` in both runtimes — `nonconforming — eleven words` and its control at two words,
|
|
2959
|
+
where the bound is kept and the clone's loss is reported anyway — and holds the check-ON run against
|
|
2960
|
+
the identical run with the check OFF, record for record. *A worker here made to keep its bound is
|
|
2961
|
+
green at this rung and red at that one.*
|
|
2962
|
+
## 17. Fanning out PER ITEM — a region inside a `map` region
|
|
2963
|
+
|
|
2964
|
+
**The shape this section teaches was refused at `Driver` construction until `0.4.0`.** *Score each item, then fan
|
|
2965
|
+
out per item* — a `map` over a list, and inside each of its branches a SECOND `map` over a list that
|
|
2966
|
+
branch computed — was refused at `Driver` construction, because a region's identity was its opener's
|
|
2967
|
+
node id and a `map` makes one opener reachable once per item. The workaround was the one this library
|
|
2968
|
+
exists to remove: a reshaping node and a second flow.
|
|
2969
|
+
|
|
2970
|
+
**A region INSTANCE is now the pair `(opener node id, the opener's own branch path)`**
|
|
2971
|
+
([ADR 0062](doc/decisions/0062-a-region-instance-is-its-opener-and-the-openers-own-branch-path.md)).
|
|
2972
|
+
Three things follow, and they are the whole of what you need:
|
|
2973
|
+
|
|
2974
|
+
| | |
|
|
2975
|
+
|---|---|
|
|
2976
|
+
| **`RunState.regions` is TWO-LEVEL** | `regions[openerId][openerPath]`. The opener's id is unchanged in meaning and in value; the second key is `-` for a region in no region, and inside a `map` region it is that map's item index |
|
|
2977
|
+
| **every `RegionEvent` carries `regionPath` beside `regionId`** | so an event says WHICH instance it is about. The same for `UnsatisfiableJoin` |
|
|
2978
|
+
| **a join's collection is per INSTANCE** | `JP-R12` keys it by branch path, and a branch inside an instance carries that instance's path as its own prefix — so two instances of one opener hold DISJOINT key sets and cannot collide |
|
|
2979
|
+
|
|
2980
|
+
**Both arities come from data, at both levels, and neither is in the document.** That is what
|
|
2981
|
+
separates a `map` from a `fanout`: `FD-V9` requires a fan-out's branch entries to be written down, so
|
|
2982
|
+
a fan-out's width is an authoring-time fact. Here the outer width is *how many orders the entry
|
|
2983
|
+
returned* and each inner width is *how many lines THAT order has* — different per instance.
|
|
2984
|
+
|
|
2985
|
+
**Writing it needs exactly one node between the two maps.** A `map`'s `over` names a key of what the
|
|
2986
|
+
previous node returned, and a `map`'s branch is a TASK (`FD-V25` refuses a map whose branch is a
|
|
2987
|
+
map) — so the outer branch entry is the node whose output carries the item's own list. No `needs`, no
|
|
2988
|
+
`transform`, no reshaping node and no second flow.
|
|
2989
|
+
|
|
2990
|
+
```ts run
|
|
2991
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
2992
|
+
import type { FlowDefinition, RegionEvent, RegistryDocument } from 'fyflow-orchestration';
|
|
2993
|
+
|
|
2994
|
+
const registry: RegistryDocument = {
|
|
2995
|
+
workers: {
|
|
2996
|
+
orders: { url: '../../examples/getting-started/orders.ts' }, // in your program: './orders.ts'
|
|
2997
|
+
lines: { url: '../../examples/getting-started/lines.ts' },
|
|
2998
|
+
tag: { url: '../../examples/getting-started/tag.ts' },
|
|
2999
|
+
},
|
|
3000
|
+
nodeTypes: {
|
|
3001
|
+
Orders: { worker: 'orders', inline: true },
|
|
3002
|
+
Lines: { worker: 'lines', inline: true },
|
|
3003
|
+
Tag: { worker: 'tag', inline: true },
|
|
3004
|
+
},
|
|
3005
|
+
};
|
|
3006
|
+
|
|
3007
|
+
// TWO MAPS, one inside the other's branch. `expand` is the single node between them: a map's `over`
|
|
3008
|
+
// names a key of what the PREVIOUS node returned, so the outer branch entry is what produces the
|
|
3009
|
+
// item's own list. `FD-V25` refuses a map whose branch is a map, which is why that node is required
|
|
3010
|
+
// and why it is the only one.
|
|
3011
|
+
const flow: FlowDefinition = {
|
|
3012
|
+
flowId: 'per-item', version: 1, entry: 'intake',
|
|
3013
|
+
nodes: [
|
|
3014
|
+
{ id: 'intake', kind: 'task', type: 'Orders', edges: [{ to: 'perOrder' }] },
|
|
3015
|
+
{ id: 'perOrder', kind: 'map', over: 'orders', branch: 'expand' },
|
|
3016
|
+
{ id: 'expand', kind: 'task', type: 'Lines', edges: [{ to: 'perLine' }] },
|
|
3017
|
+
{ id: 'perLine', kind: 'map', over: 'lines', branch: 'price' },
|
|
3018
|
+
{ id: 'price', kind: 'task', type: 'Tag', settings: { label: 'priced' }, edges: [{ to: 'lineJoin' }] },
|
|
3019
|
+
{ id: 'lineJoin', kind: 'join', of: 'perLine', edges: [{ to: 'orderJoin' }] },
|
|
3020
|
+
{ id: 'orderJoin', kind: 'join', of: 'perOrder', edges: [{ to: 'report' }] },
|
|
3021
|
+
{ id: 'report', kind: 'task', type: 'Tag', settings: { label: 'reported' }, edges: [{ to: 'done' }] },
|
|
3022
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
3023
|
+
],
|
|
3024
|
+
};
|
|
3025
|
+
|
|
3026
|
+
// A region event now says WHICH INSTANCE it is about. Before 0.4.0 it carried `regionId` alone,
|
|
3027
|
+
// which is the same string for every instance of one opener.
|
|
3028
|
+
const arrivals: string[] = [];
|
|
3029
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
3030
|
+
const state = await runtime.start(flow, { region: 'ab c' }, {
|
|
3031
|
+
onRegionEvent: (e: RegionEvent) => {
|
|
3032
|
+
if (e.type === 'branch.arrived') arrivals.push(`${e.regionId}@${e.regionPath} <- ${e.branchPath}`);
|
|
3033
|
+
},
|
|
3034
|
+
}).done;
|
|
3035
|
+
await runtime.shutdown();
|
|
3036
|
+
|
|
3037
|
+
console.log(state.outcome); // completed
|
|
3038
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
3039
|
+
|
|
3040
|
+
// ONE OPENER, TWO INSTANCES — keyed by the opener's own branch path, which inside `perOrder` is the
|
|
3041
|
+
// item index. `'ab c'` is two orders, so `perOrder` opened two branches and `perLine` was entered
|
|
3042
|
+
// twice; `ab` has two lines and `c` has one, and neither number is in the flow above.
|
|
3043
|
+
console.log(Object.keys(state.regions['perLine'] ?? {}).sort()); // [ '0', '1' ]
|
|
3044
|
+
console.log(Object.keys(state.regions['perOrder'] ?? {}).sort()); // [ '-' ]
|
|
3045
|
+
const instances = Object.keys(state.regions['perLine'] ?? {}).sort();
|
|
3046
|
+
if (instances.join() !== '0,1') throw new Error(`one opener, two instances: got ${instances.join()}`);
|
|
3047
|
+
if (Object.keys(state.regions['perOrder'] ?? {}).join() !== '-') throw new Error('perOrder is in no region, so its instance key is `-`');
|
|
3048
|
+
|
|
3049
|
+
// EACH INSTANCE HAS ITS OWN WIDTH AND ITS OWN ANSWER, and their branch keys are disjoint because a
|
|
3050
|
+
// branch inside an instance begins with that instance's path.
|
|
3051
|
+
const widths = instances.map((p) => Object.keys(state.regions['perLine']![p]!.settled).sort().join(' '));
|
|
3052
|
+
console.log(widths); // [ '0.0 0.1', '1.0' ]
|
|
3053
|
+
if (widths.join(' | ') !== '0.0 0.1 | 1.0') throw new Error(`each instance has its own width: got ${widths.join(' | ')}`);
|
|
3054
|
+
if (!instances.every((p) => state.regions['perLine']![p]!.joined === true)) throw new Error('both inner joins fired');
|
|
3055
|
+
|
|
3056
|
+
// AND THE EVENTS NAMED THE INSTANCE, not just the region.
|
|
3057
|
+
console.log(arrivals.sort());
|
|
3058
|
+
// [ 'perLine@0 <- 0.0', 'perLine@0 <- 0.1', 'perLine@1 <- 1.0', 'perOrder@- <- 0', 'perOrder@- <- 1' ]
|
|
3059
|
+
if (!arrivals.includes('perLine@0 <- 0.0') || !arrivals.includes('perLine@1 <- 1.0')) {
|
|
3060
|
+
throw new Error(`a region event names the instance: ${arrivals.sort().join(', ')}`);
|
|
3061
|
+
}
|
|
3062
|
+
|
|
3063
|
+
// WHAT THE OUTER JOIN CARRIES IS TWO LEVELS DEEP. `JP-R12`: one entry per branch that arrived, keyed
|
|
3064
|
+
// by branch path — and each of the outer entries IS that instance's own collection.
|
|
3065
|
+
const report = Object.values(state.nodes).find((n) => n.nodeId === 'report')?.result as Record<string, Record<string, { sku: string }>>;
|
|
3066
|
+
console.log(Object.keys(report).filter((k) => k !== 'reported').sort()); // [ '0', '1' ]
|
|
3067
|
+
console.log(report['0']!['0.1']!.sku); // ab-1
|
|
3068
|
+
if (report['0']!['0.1']!.sku !== 'ab-1' || report['1']!['1.0']!.sku !== 'c-0') {
|
|
3069
|
+
throw new Error(`the outer collection's values are the inner collections: ${JSON.stringify(report)}`);
|
|
3070
|
+
}
|
|
3071
|
+
```
|
|
3072
|
+
|
|
3073
|
+
**What is still refused, and by what.** `FD-V19` keeps one node out of two branch entries of one
|
|
3074
|
+
fan-out; `FD-V25` keeps a map's branch from being a map; `ID-R6` refuses a branch path across a map
|
|
3075
|
+
region with no item index; `ID-R1` refuses a duplicate dispatch. **Those four are what make the
|
|
3076
|
+
derived key total**, and they are why this cost no new refusal and no new error class.
|
|
3077
|
+
|
|
3078
|
+
**An empty inner list still stops the run.** A `map` over zero items opens no branch, so its join can
|
|
3079
|
+
never fire and the run is `stuck` (section 8) — one empty list anywhere makes the whole run stuck,
|
|
3080
|
+
and that ruling is unchanged by nesting.
|
|
3081
|
+
|
|
3082
|
+
**`0.4.0` is BREAKING in two ways and neither is `regionId`.** `RunState.regions` gaining its second
|
|
3083
|
+
level is a break for anyone reading it, and `UnsatisfiableJoin.regionPath` is a REQUIRED member on an
|
|
3084
|
+
interface a consumer may construct — so a test double reporting a dead join no longer compiles.
|
|
3085
|
+
`regionId` keeps its meaning AND its value, which is why no flow, no policy and no region name moves.
|
|
3086
|
+
|
|
3087
|
+
---
|
|
3088
|
+
|
|
3089
|
+
## 18. Saving a run and RESUMING it — append the events, resume from them
|
|
3090
|
+
|
|
3091
|
+
**A run can outlive the process that ran it, and what it needs is the events you were already
|
|
3092
|
+
handed.** The library defines the value and never where it goes
|
|
3093
|
+
([P23](doc/plans/p23-what-a-run-needs-to-outlive-its-process.md) decision 1): no file, no database,
|
|
3094
|
+
no serializer. A CHECKPOINT is `{ header, events }`
|
|
3095
|
+
([ADR 0064](doc/decisions/0064-a-checkpoint-is-the-whole-event-sequence-and-a-header-and-it-is-a-copy.md)) —
|
|
3096
|
+
the header is five facts the events cannot regenerate (`format`, `runId`, `flowId`, `version` and a
|
|
3097
|
+
copy of the entry payload), and `events` is every fold input so far, so **every event after a
|
|
3098
|
+
checkpoint is an append to it**. You never have to call `driver.checkpoint()` to keep one:
|
|
3099
|
+
|
|
3100
|
+
| | where you get it | what you do with it |
|
|
3101
|
+
|---|---|---|
|
|
3102
|
+
| the header | `runtime.start(…)` returns it as `header`, beside `driver` and `done` | write it once |
|
|
3103
|
+
| every event | `onFoldInput(event, position)` | write it AT `position` — the driver's index, never a count you start again in each process |
|
|
3104
|
+
| a resumed run | `runtime.resume(flow, { header, events }, options)` returns the same `{ driver, done, header }` | keep writing: it hands you positions from the STORED LENGTH on |
|
|
3105
|
+
|
|
3106
|
+
**Key the store by POSITION, and the reason is measured.** The first events are delivered WHILE
|
|
3107
|
+
`start` runs, so the header arrives after them — a store that must write its header first cannot —
|
|
3108
|
+
and a resumed run's first position is the length you stored, not 0, so an arrival counter per process
|
|
3109
|
+
overwrites the run it is resuming. ([W90](doc/work-items/w90-a-run-killed-and-resumed-on-real-workers.md)
|
|
3110
|
+
runs exactly that wrong store at L6 and it is refused `diverged` at position 0.) **The contract is
|
|
3111
|
+
`structuredClone`, not JSON**: a result is whatever survived `OR-R2`'s clone, which includes `Map`,
|
|
3112
|
+
`Set`, `Date`, `BigInt` and `-0` — pick a serializer that keeps them, or revive them.
|
|
3113
|
+
|
|
3114
|
+
```ts run
|
|
3115
|
+
import { createRuntime, RunIdInUse } from 'fyflow-orchestration/scheduler';
|
|
3116
|
+
import { fold, ResumeRefused } from 'fyflow-orchestration';
|
|
3117
|
+
import type { CheckpointHeader, FlowDefinition, FoldInput, RegistryDocument } from 'fyflow-orchestration';
|
|
3118
|
+
|
|
3119
|
+
// Section 17's flow and workers — ONE map inside another — so there is something in flight to lose.
|
|
3120
|
+
const registry: RegistryDocument = {
|
|
3121
|
+
workers: {
|
|
3122
|
+
orders: { url: '../../examples/getting-started/orders.ts' }, // in your program: './orders.ts'
|
|
3123
|
+
lines: { url: '../../examples/getting-started/lines.ts' },
|
|
3124
|
+
tag: { url: '../../examples/getting-started/tag.ts' },
|
|
3125
|
+
},
|
|
3126
|
+
nodeTypes: {
|
|
3127
|
+
Orders: { worker: 'orders', inline: true },
|
|
3128
|
+
Lines: { worker: 'lines', inline: true },
|
|
3129
|
+
Tag: { worker: 'tag', inline: true },
|
|
3130
|
+
},
|
|
3131
|
+
};
|
|
3132
|
+
const flow: FlowDefinition = {
|
|
3133
|
+
flowId: 'per-item', version: 1, entry: 'intake',
|
|
3134
|
+
nodes: [
|
|
3135
|
+
{ id: 'intake', kind: 'task', type: 'Orders', edges: [{ to: 'perOrder' }] },
|
|
3136
|
+
{ id: 'perOrder', kind: 'map', over: 'orders', branch: 'expand' },
|
|
3137
|
+
{ id: 'expand', kind: 'task', type: 'Lines', edges: [{ to: 'perLine' }] },
|
|
3138
|
+
{ id: 'perLine', kind: 'map', over: 'lines', branch: 'price' },
|
|
3139
|
+
{ id: 'price', kind: 'task', type: 'Tag', settings: { label: 'priced' }, edges: [{ to: 'lineJoin' }] },
|
|
3140
|
+
{ id: 'lineJoin', kind: 'join', of: 'perLine', edges: [{ to: 'orderJoin' }] },
|
|
3141
|
+
{ id: 'orderJoin', kind: 'join', of: 'perOrder', edges: [{ to: 'report' }] },
|
|
3142
|
+
{ id: 'report', kind: 'task', type: 'Tag', settings: { label: 'reported' }, edges: [{ to: 'done' }] },
|
|
3143
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
3144
|
+
],
|
|
3145
|
+
};
|
|
3146
|
+
|
|
3147
|
+
// YOUR STORE. The library never writes one: here it is an object in memory, in your program a
|
|
3148
|
+
// file, a table or a queue. It holds the HEADER once and every event AT ITS POSITION — the second
|
|
3149
|
+
// argument `onFoldInput` is handed — so a later process can go on appending to it.
|
|
3150
|
+
const store: { header?: CheckpointHeader; events: FoldInput[] } = { events: [] };
|
|
3151
|
+
|
|
3152
|
+
// ---- PROCESS ONE, which dies mid-run.
|
|
3153
|
+
let alive = true;
|
|
3154
|
+
let die!: () => void;
|
|
3155
|
+
const died = new Promise<void>((resolve) => { die = resolve; });
|
|
3156
|
+
const first = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
3157
|
+
const run = first.start(flow, { region: 'ab c' }, {
|
|
3158
|
+
onFoldInput: (event, position) => {
|
|
3159
|
+
if (!alive) return; // a dead process writes nothing
|
|
3160
|
+
store.events[position] = structuredClone(event); // your durable append, BY POSITION
|
|
3161
|
+
// THE CRASH, simulated: the process dies the instant the first `expand` settles — inside the
|
|
3162
|
+
// driver's turn, BEFORE it dispatches what that settlement causes.
|
|
3163
|
+
if (event.kind === 'outcome' && event.record.taskId.includes(':expand:')) { alive = false; die(); }
|
|
3164
|
+
},
|
|
3165
|
+
});
|
|
3166
|
+
// The header arrives on the handle, AFTER the first events: they were delivered while `start` ran.
|
|
3167
|
+
// A store that had to write its header first could not; one keyed by position does not care.
|
|
3168
|
+
store.header = run.header;
|
|
3169
|
+
await died;
|
|
3170
|
+
// `shutdown()` is cleanup, NOT the kill: it lets running work finish, and here that finishes the
|
|
3171
|
+
// whole run. What makes this a crash is the store — nothing after `alive = false` reached it.
|
|
3172
|
+
await first.shutdown();
|
|
3173
|
+
|
|
3174
|
+
const storedSettled = new Set(store.events.flatMap((e) => (e.kind === 'outcome' ? [e.record.taskId] : [])));
|
|
3175
|
+
const storedDispatched = new Set(store.events.flatMap((e) => (e.kind === 'dispatched' ? [e.taskId] : [])));
|
|
3176
|
+
console.log(`stored ${store.events.length} events, the last one the settlement the process died in`);
|
|
3177
|
+
|
|
3178
|
+
// ---- PROCESS TWO: a FRESH runtime, which is all a restarted process has. Resume from the store and
|
|
3179
|
+
// keep appending BY POSITION: a resumed run hands you positions from the stored length on, so the
|
|
3180
|
+
// same store holds the whole run again.
|
|
3181
|
+
const header = store.header!; // written once, by process one
|
|
3182
|
+
const second = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
3183
|
+
const resumed = second.resume(flow, { header, events: store.events.slice() }, {
|
|
3184
|
+
onFoldInput: (event, position) => { store.events[position] = structuredClone(event); },
|
|
3185
|
+
});
|
|
3186
|
+
const state = await resumed.done;
|
|
3187
|
+
|
|
3188
|
+
console.log(state.outcome); // completed
|
|
3189
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
3190
|
+
// The same answer a run nobody killed gives (section 17's):
|
|
3191
|
+
const report = Object.values(state.nodes).find((n) => n.nodeId === 'report')?.result as Record<string, Record<string, { sku: string }>>;
|
|
3192
|
+
if (report['0']!['0.1']!.sku !== 'ab-1' || report['1']!['1.0']!.sku !== 'c-0') {
|
|
3193
|
+
throw new Error(`the resumed run reaches the same report: ${JSON.stringify(report)}`);
|
|
3194
|
+
}
|
|
3195
|
+
|
|
3196
|
+
// WHAT RAN AGAIN — named by the `resumed` event, and recorded by the fold in `state.resumes`.
|
|
3197
|
+
const resumes = store.events.flatMap((e) => (e.kind === 'resumed' ? [e.taskIds] : []));
|
|
3198
|
+
if (resumes.length !== 1) throw new Error(`one resume is recorded as one \`resumed\` event: ${resumes.length}`);
|
|
3199
|
+
const again = resumes[0]!;
|
|
3200
|
+
console.log(again); // e.g. [ 'r1:expand:1:0', 'r1:price:0.0:0', 'r1:price:0.1:0' ]
|
|
3201
|
+
if (JSON.stringify(state.resumes) !== JSON.stringify(resumes)) throw new Error('RunState.resumes records it');
|
|
3202
|
+
// Nothing that SETTLED in the store runs again...
|
|
3203
|
+
if (again.some((id) => storedSettled.has(id))) throw new Error('a stored settlement is never re-run');
|
|
3204
|
+
// ...but work the store NEVER SAW does: the prices the dying settlement dispatched after the last
|
|
3205
|
+
// thing that was written. The re-run set is what was in flight at the END of the replay.
|
|
3206
|
+
const neverStored = again.filter((id) => !storedDispatched.has(id));
|
|
3207
|
+
console.log(neverStored); // the `price` tasks of the order whose `expand` the process died in
|
|
3208
|
+
if (neverStored.length === 0 || !neverStored.every((id) => id.includes(':price:'))) {
|
|
3209
|
+
throw new Error(`the lost tail's dispatches run again: ${JSON.stringify(neverStored)}`);
|
|
3210
|
+
}
|
|
3211
|
+
|
|
3212
|
+
// THE STORE HOLDS THE WHOLE RUN AGAIN: no gap at any position, and its fold IS the run.
|
|
3213
|
+
if (Object.keys(store.events).length !== store.events.length) throw new Error('a store keyed by position has no holes');
|
|
3214
|
+
if (JSON.stringify(fold(store.events)) !== JSON.stringify(state)) throw new Error('the store folds to the run');
|
|
3215
|
+
|
|
3216
|
+
// A run id belongs to the runtime that holds it: resuming the same run on `second` again is refused.
|
|
3217
|
+
try {
|
|
3218
|
+
second.resume(flow, { header, events: store.events.slice() });
|
|
3219
|
+
throw new Error('expected RunIdInUse');
|
|
3220
|
+
} catch (err) {
|
|
3221
|
+
if (!(err instanceof RunIdInUse)) throw err;
|
|
3222
|
+
}
|
|
3223
|
+
// And a checkpoint is resumed against the definition it was stored under, or not at all.
|
|
3224
|
+
const third = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
3225
|
+
try {
|
|
3226
|
+
third.resume({ ...flow, version: 2 }, { header, events: store.events.slice() });
|
|
3227
|
+
throw new Error('expected ResumeRefused');
|
|
3228
|
+
} catch (err) {
|
|
3229
|
+
if (!(err instanceof ResumeRefused) || err.reason !== 'flow') throw err;
|
|
3230
|
+
}
|
|
3231
|
+
await second.shutdown();
|
|
3232
|
+
await third.shutdown();
|
|
3233
|
+
```
|
|
3234
|
+
|
|
3235
|
+
**What a resume PROMISES, and what it does not** — [ADR 0065](doc/decisions/0065-a-resume-replays-the-stored-stimuli-refuses-what-it-cannot-regenerate-and-re-dispatches-what-was-in-flight.md), `spec/run-state.md` section 9:
|
|
3236
|
+
|
|
3237
|
+
- **It is AT-LEAST-ONCE, and your workers must tolerate running twice.** The resume REPLAYS the
|
|
3238
|
+
stored settlements and resolutions through the routing that produced them, with every dispatch
|
|
3239
|
+
held, and then hands every task still in flight to the new scheduler at the SAME task id. That set
|
|
3240
|
+
is what was in flight at the END of the replay — including a task that finished in the dead process
|
|
3241
|
+
whose settlement you had not stored yet, and a task the dying settlement dispatched after your last
|
|
3242
|
+
write, which the snippet shows the store never saw. The run records it: a `resumed` event naming
|
|
3243
|
+
those ids, and `state.resumes`.
|
|
3244
|
+
- **A re-run task gets a FRESH retry budget.** The log records a task's `attempts` only when it
|
|
3245
|
+
SETTLES, so the attempts a task spent before the crash are recorded nowhere — measured: a task with
|
|
3246
|
+
a budget of two attempts ran THREE times across one kill ([Q365](doc/questions/q365.md)).
|
|
3247
|
+
- **A task PARKED for a person is not in flight.** It stays parked, the resumed run is
|
|
3248
|
+
`awaiting_action`, and `resolve()` on the RESUMED driver completes it (section 7).
|
|
3249
|
+
- **The check sees only what you STORED.** The stored events must be a prefix of what the replay
|
|
3250
|
+
regenerates, and the first position where they are not is refused `diverged` — so a predicate or a
|
|
3251
|
+
flow you changed is caught where the stored run's road crosses it, and NOT where that road lies
|
|
3252
|
+
past your last event: there the resumed run takes the new road silently
|
|
3253
|
+
([Q372](doc/questions/q372.md)). Change a flow, change its `version`.
|
|
3254
|
+
- **The store is TRUSTED for what the resume feeds.** A stored settlement's result and a stored
|
|
3255
|
+
person's resolution are replayed as stored — nothing can check they are what the process saw
|
|
3256
|
+
([Q374](doc/questions/q374.md)). A store you do not trust is a store you should not resume from.
|
|
3257
|
+
- **`ResumeRefused` names its `reason`**: `format` (not a checkpoint this version writes), `run` (the
|
|
3258
|
+
header's run is not the driver's), `flow` (another flow id or `version` — there is no migration),
|
|
3259
|
+
`stopped` and `diverged` — the last two with the `index` of the position. **A STOPPED run is not
|
|
3260
|
+
resumable**: the refusal that stopped it would stop it again. A completed or `stuck` run resumes to
|
|
3261
|
+
itself and dispatches nothing.
|
|
3262
|
+
- **A refused resume may SPEND its run id.** Refused by its header, nothing exists and the id is still
|
|
3263
|
+
free; refused during its replay, the runtime built a driver for it — so that runtime refuses the id
|
|
3264
|
+
with `RunIdInUse` from then on, as it refuses any id it ever built a driver for, a finished run's
|
|
3265
|
+
included, long after it released the driver (section 12). Resume on a fresh runtime
|
|
3266
|
+
([Q371](doc/questions/q371.md)).
|
|
3267
|
+
- **`shutdown()` is not a kill.** It DRAINS: measured in node and deno, a run one settlement in
|
|
3268
|
+
completes inside it, while a retry waiting on its backoff never runs. It waits while tasks are RUNNING
|
|
3269
|
+
and for at most five seconds, so a run with longer work in flight, or only queued or blocked tasks, is
|
|
3270
|
+
left where the wait ended. A crash is what the snippet's
|
|
3271
|
+
`alive` flag stands for — the store stops, whatever the process was doing.
|
|
3272
|
+
|
|
3273
|
+
**`0.5.0` is BREAKING for a consumer of the fold, in two ways.** `FoldInput` has a new kind,
|
|
3274
|
+
`resumed`, so a `switch` that is exhaustive over `kind` no longer compiles, and `RunState` has a new
|
|
3275
|
+
cell, `resumes`, so a `RunState` you construct needs it. A run nobody resumes carries no `resumed`
|
|
3276
|
+
event and `resumes: []`.
|
|
3277
|
+
|
|
3278
|
+
---
|
|
3279
|
+
|
|
3280
|
+
## 19. What a node may REMEMBER across branches — a gate
|
|
3281
|
+
|
|
3282
|
+
**A branch cannot see its siblings, and that is usually what you want.** Two instances of one `map` are
|
|
3283
|
+
invisible to each other (section 3's reference rule), a join waits for all of them, and a worker's own
|
|
3284
|
+
memory is lost when its pool tears down or runs it on another thread. So a flow that must route each branch
|
|
3285
|
+
by what the run's OTHER branches have already seen — *keep the first file of each TV series and drop the
|
|
3286
|
+
rest, while every file gets a thumbnail* — had nowhere to keep that memory.
|
|
3287
|
+
([P24](doc/plans/p24-what-a-node-may-remember-across-a-runs-branches.md) is that ingest.)
|
|
3288
|
+
|
|
3289
|
+
**A `gate` is a decision WITH MEMORY, and the memory is the DRIVER's**
|
|
3290
|
+
([ADR 0066](doc/decisions/0066-a-gate-is-a-decision-with-memory-that-names-a-registered-type-and-wires-every-route-exactly-once.md),
|
|
3291
|
+
[ADR 0067](doc/decisions/0067-a-gate-runs-in-the-driver-its-memory-is-the-drivers-and-its-decision-is-recorded.md)):
|
|
3292
|
+
|
|
3293
|
+
| | |
|
|
3294
|
+
|---|---|
|
|
3295
|
+
| **it names a gate TYPE** | `distinct` is built in: routes `first` (its DEFAULT), `duplicate` and `nokey`, keyed by `settings.key` — a DOTTED path of object keys over the value that reached the gate, no index, no wildcard, no expression. A key a flow needs COMPUTED is computed by the task before the gate |
|
|
3296
|
+
| **every route is wired, exactly once** | an edge with no `route` carries the type's DEFAULT route, never every route; a DROP is an edge to an `end`, and each is refused at load if it is missing (`FD-V41`) |
|
|
3297
|
+
| **it computes nothing** | the branch leaves on the chosen route with EXACTLY the value that reached the gate |
|
|
3298
|
+
| **its memory is ONE per gate per RUN** | shared by every branch that reaches it, whatever map item or region instance it comes from — and it starts empty with the run and ends with it |
|
|
3299
|
+
| ***first* means first to ARRIVE in this run** | which varies between runs — so a test asserts that exactly ONE branch per key went `first`, never WHICH |
|
|
3300
|
+
| **every decision is recorded** | a `gated` fold input — the gate, the branch path, the route and the key — and `RunState.gated`, in the order the branches arrived |
|
|
3301
|
+
| **a resume REBUILDS the memory** | the replay asks the gate again for every stored settlement that reached it — the recorded decisions are REGENERATED and compared, never fed — so a run killed at any point resumes routing exactly as the run that was killed would have (section 18) |
|
|
3302
|
+
|
|
3303
|
+
```ts run
|
|
3304
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
3305
|
+
import type { FlowDefinition, GateType, RegistryDocument } from 'fyflow-orchestration';
|
|
3306
|
+
|
|
3307
|
+
const registry: RegistryDocument = {
|
|
3308
|
+
workers: {
|
|
3309
|
+
download: { url: '../../examples/workers/download.ts' }, // in your program: './download.ts'
|
|
3310
|
+
identify: { url: '../../examples/workers/identify.ts' }, // parses `Show.S01E02.mkv` into series `show`
|
|
3311
|
+
thumbnail: { url: '../../examples/workers/thumbnail.ts' },
|
|
3312
|
+
series: { url: '../../examples/workers/series.ts' }, // the per-series work; returns the file that opened it
|
|
3313
|
+
publish: { url: '../../examples/workers/publish.ts' },
|
|
3314
|
+
},
|
|
3315
|
+
nodeTypes: {
|
|
3316
|
+
Download: { worker: 'download', inline: true, pool: { maxThreads: 8 } },
|
|
3317
|
+
Identify: { worker: 'identify', inline: true, pool: { maxThreads: 8 } },
|
|
3318
|
+
Thumbnail: { worker: 'thumbnail', inline: true, pool: { maxThreads: 8 } },
|
|
3319
|
+
SeriesSetup: { worker: 'series', inline: true },
|
|
3320
|
+
Publish: { worker: 'publish', inline: true },
|
|
3321
|
+
},
|
|
3322
|
+
};
|
|
3323
|
+
|
|
3324
|
+
// A map over files. Each branch fetches its file and fans out BESIDE: a thumbnail for EVERY file, and `identify`,
|
|
3325
|
+
// which assigns the SERIES, then the gate. No task could know whether another file of its series came first — the
|
|
3326
|
+
// gate can, because its memory is the run's.
|
|
3327
|
+
const flow: FlowDefinition = {
|
|
3328
|
+
flowId: 'tv-ingest', version: 1, entry: 'episodes',
|
|
3329
|
+
nodes: [
|
|
3330
|
+
{ id: 'episodes', kind: 'map', over: 'files', branch: 'download' },
|
|
3331
|
+
{ id: 'download', kind: 'task', type: 'Download', edges: [{ to: 'beside' }] },
|
|
3332
|
+
{ id: 'beside', kind: 'fanout', branches: ['thumbnail', 'identify'] },
|
|
3333
|
+
{ id: 'thumbnail', kind: 'task', type: 'Thumbnail', edges: [{ to: 'perFile' }] },
|
|
3334
|
+
{ id: 'identify', kind: 'task', type: 'Identify', edges: [{ to: 'dedupe' }] },
|
|
3335
|
+
// ALL THREE of `distinct`'s routes wired, once each. The edge with no `route` carries its DEFAULT, `first`.
|
|
3336
|
+
{ id: 'dedupe', kind: 'gate', type: 'distinct', settings: { key: 'series.id' }, edges: [
|
|
3337
|
+
{ to: 'setupSeries' },
|
|
3338
|
+
{ route: 'duplicate', to: 'dropDuplicate' }, // a DROP is an edge to an `end`
|
|
3339
|
+
{ route: 'nokey', to: 'dropUnidentified' }, // `series.id` resolved to nothing: a name with no series in it
|
|
3340
|
+
] },
|
|
3341
|
+
{ id: 'setupSeries', kind: 'task', type: 'SeriesSetup', edges: [{ to: 'seriesReady' }] },
|
|
3342
|
+
{ id: 'seriesReady', kind: 'end', outcome: 'series-ready' },
|
|
3343
|
+
{ id: 'dropDuplicate', kind: 'end', outcome: 'duplicate' },
|
|
3344
|
+
{ id: 'dropUnidentified', kind: 'end', outcome: 'unidentified' },
|
|
3345
|
+
{ id: 'perFile', kind: 'join', of: 'beside', edges: [{ to: 'all' }] },
|
|
3346
|
+
{ id: 'all', kind: 'join', of: 'episodes', edges: [{ to: 'publish' }] },
|
|
3347
|
+
{ id: 'publish', kind: 'task', type: 'Publish', edges: [{ to: 'done' }] },
|
|
3348
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
3349
|
+
],
|
|
3350
|
+
};
|
|
3351
|
+
|
|
3352
|
+
const files = ['Dark.S01E01.mkv', 'Lost.S01E01.mkv', 'Dark.S01E02.mkv', 'trailer.mkv', 'Lost.S01E02.mkv']
|
|
3353
|
+
.map((name) => ({ name, sizeMB: 1 }));
|
|
3354
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url, flows: [flow] });
|
|
3355
|
+
const state = await runtime.start(flow, { files }).done;
|
|
3356
|
+
await runtime.shutdown();
|
|
3357
|
+
|
|
3358
|
+
console.log(state.outcome); // completed
|
|
3359
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
3360
|
+
|
|
3361
|
+
// EVERY DECISION, in the order the branches reached the gate. WHICH file of a series went `first` is whichever
|
|
3362
|
+
// ARRIVED first in this run — so assert the count per key, never the file.
|
|
3363
|
+
console.log(state.gated.map((d) => `${d.branchPath} ${d.route} ${d.key ?? '-'}`));
|
|
3364
|
+
// e.g. [ '0.1 first dark', '1.1 first lost', '2.1 duplicate dark', '3.1 nokey -', '4.1 duplicate lost' ]
|
|
3365
|
+
const opened = state.gated.filter((d) => d.route === 'first').map((d) => String(d.key)).sort();
|
|
3366
|
+
if (opened.join() !== 'dark,lost') throw new Error(`exactly ONE first per series: ${opened.join()}`);
|
|
3367
|
+
if (state.gated.filter((d) => d.route === 'duplicate').length !== 2) throw new Error('the other two identified files are duplicates');
|
|
3368
|
+
if (state.gated.filter((d) => d.route === 'nokey').length !== 1) throw new Error('the trailer carries no series');
|
|
3369
|
+
|
|
3370
|
+
// The per-series task ran for EXACTLY the branches the gate routed `first` — and received exactly what reached the
|
|
3371
|
+
// gate: `identify`'s whole result, the file name included, which neither the key nor the settings carries.
|
|
3372
|
+
const setups = Object.values(state.nodes).filter((n) => n.nodeId === 'setupSeries');
|
|
3373
|
+
const firsts = state.gated.filter((d) => d.route === 'first');
|
|
3374
|
+
if (setups.map((n) => n.branchPath).sort().join() !== firsts.map((d) => d.branchPath).sort().join()) {
|
|
3375
|
+
throw new Error('setupSeries ran for exactly the branches routed first');
|
|
3376
|
+
}
|
|
3377
|
+
for (const d of firsts) {
|
|
3378
|
+
const r = setups.find((n) => n.branchPath === d.branchPath)!.result as { series: string; openedBy: string };
|
|
3379
|
+
if (r.series !== d.key || r.openedBy !== files[Number(d.branchPath.split('.')[0])]!.name) {
|
|
3380
|
+
throw new Error(`what left the gate is what reached it: ${JSON.stringify(r)}`);
|
|
3381
|
+
}
|
|
3382
|
+
}
|
|
3383
|
+
|
|
3384
|
+
// A DROP IS AN `end`, so it is a path end like any other — and the thumbnail ran for EVERY file: each file's
|
|
3385
|
+
// `perFile` record holds its thumbnail, and nothing from the gate road, which ended at an `end` in every branch.
|
|
3386
|
+
console.log(state.pathEnds.map((p) => p.at).sort());
|
|
3387
|
+
// [ 'done', 'dropDuplicate', 'dropDuplicate', 'dropUnidentified', 'seriesReady', 'seriesReady' ]
|
|
3388
|
+
const published = Object.values(state.nodes).find((n) => n.nodeId === 'publish')!.result as
|
|
3389
|
+
{ joinPayload: Record<string, Record<string, { thumbnail?: string }>> };
|
|
3390
|
+
const thumbs = Object.entries(published.joinPayload).map(([i, perFile]) => perFile[`${i}.0`]?.thumbnail);
|
|
3391
|
+
if (thumbs.length !== files.length || thumbs.some((t) => t === undefined)) throw new Error(`a thumbnail per file: ${thumbs.join()}`);
|
|
3392
|
+
|
|
3393
|
+
// A GATE TYPE OF YOUR OWN is CODE, registered by name the way `distinct` is written. A route its `fn` returns that
|
|
3394
|
+
// it did not declare STOPS the run by name — `UndeclaredRoute` (`FD-R5`): no validator can see it, because only
|
|
3395
|
+
// running `fn` reveals it.
|
|
3396
|
+
const sloppy: GateType = { routes: ['keep', 'drop'], defaultRoute: 'keep', fn: () => 'maybe' };
|
|
3397
|
+
const filter: FlowDefinition = {
|
|
3398
|
+
flowId: 'sloppy-filter', version: 1, entry: 'download',
|
|
3399
|
+
nodes: [
|
|
3400
|
+
{ id: 'download', kind: 'task', type: 'Download', edges: [{ to: 'filter' }] },
|
|
3401
|
+
{ id: 'filter', kind: 'gate', type: 'sloppy', edges: [{ to: 'thumbnail' }, { route: 'drop', to: 'dropped' }] },
|
|
3402
|
+
{ id: 'thumbnail', kind: 'task', type: 'Thumbnail', edges: [{ to: 'done' }] },
|
|
3403
|
+
{ id: 'dropped', kind: 'end', outcome: 'dropped' },
|
|
3404
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
3405
|
+
],
|
|
3406
|
+
};
|
|
3407
|
+
const second = createRuntime(registry, { gateTypes: { sloppy } }, { base: import.meta.url, flows: [filter] });
|
|
3408
|
+
const kinds: string[] = [];
|
|
3409
|
+
const stopped = await second.start(filter, { name: 'Dark.S01E01.mkv', sizeMB: 1 }, { onRunFailure: (f) => kinds.push(f.kind) }).done;
|
|
3410
|
+
await second.shutdown();
|
|
3411
|
+
console.log(stopped.outcome, kinds); // stopped [ 'UndeclaredRoute' ]
|
|
3412
|
+
if (stopped.outcome !== 'stopped' || kinds.join() !== 'UndeclaredRoute' || stopped.gated.length !== 0) {
|
|
3413
|
+
throw new Error(`an undeclared route stops the run by name, recording nothing: ${stopped.outcome} ${kinds.join()}`);
|
|
3414
|
+
}
|
|
3415
|
+
```
|
|
3416
|
+
|
|
3417
|
+
**What the snippet cannot show, and you need:**
|
|
3418
|
+
|
|
3419
|
+
- **Test the COUNT, never the winner.** On an inline pool the first file to reach a gate is almost always the
|
|
3420
|
+
first file in the list, so a test that names it passes for the wrong reason and fails the day a pool is threaded
|
|
3421
|
+
([Q379](doc/questions/q379.md)). If a test must show that *first to arrive* is not *first in the list*, ARRANGE
|
|
3422
|
+
the arrival order — `examples/l6.ts` does it with the files' sizes, on a pool as wide as the map — and assert
|
|
3423
|
+
what the arrangement did beside the claim.
|
|
3424
|
+
- **The memory is per RUN, not per region instance.** A gate inside a nested `map` remembers across every outer
|
|
3425
|
+
item; there is no per-season scope. It forgets at the run's end — deduplicating across RUNS is your store's job.
|
|
3426
|
+
- **A gate's type must be the PURE function of its context** a decision predicate is asked to be
|
|
3427
|
+
([Q9](doc/questions/q9.md)): a resume replays every stored decision through `fn` to rebuild the memory, and a
|
|
3428
|
+
type that answered differently the second time is refused `diverged`. An ENTRY gate's type is also asked once more
|
|
3429
|
+
at `start()`, against an empty throwaway memory, to hold the payload to the target it chooses (`FD-R4`,
|
|
3430
|
+
[Q387](doc/questions/q387.md)).
|
|
3431
|
+
- **Keep your gate's memory in the gate.** A decision predicate that remembers in a closure looks like a gate and is
|
|
3432
|
+
not one: its memory lives in your process, a resume cannot rebuild it, and a run killed before its first decision
|
|
3433
|
+
resumes WRONG in silence ([Q386](doc/questions/q386.md)).
|
|
3434
|
+
- **`checkWiring` reads a route's GUARANTEE, where its type declares one.** A gate type may name the routes on
|
|
3435
|
+
which its key path resolved — `distinct` names `first` and `duplicate` — and what leaves the gate on such a
|
|
3436
|
+
route is what arrived with `required` added along the gate's key path, at every step. So a worker after `first`
|
|
3437
|
+
that REQUIRES the key `fits`. *It read `no` there until v0.25, which is [Q391](doc/questions/q391.md) and is
|
|
3438
|
+
[ADR 0068](doc/decisions/0068-a-route-guarantees-what-its-type-declares-and-the-library-derives-it-from-the-key-path.md).*
|
|
3439
|
+
- **A route that guarantees something else guarantees nothing a check can use.** A type declaring no
|
|
3440
|
+
`keyResolvedRoutes`, a gate naming no `key`, and a route the type does not name each pass the value through
|
|
3441
|
+
exactly as before — and the guarantee is PRESENCE along the path, never the leaf's TYPE, which is the type's own
|
|
3442
|
+
judgement (`distinct` wants a string or a finite number, and no static tool derives that).
|
|
3443
|
+
|
|
3444
|
+
**`0.6.0` is BREAKING in four ways, all of them for code that enumerates or matches.** `FlowNode` has a SIXTH kind,
|
|
3445
|
+
`GateNode` ([Q381](doc/questions/q381.md)), so a `switch` exhaustive over `node.kind` no longer compiles;
|
|
3446
|
+
`FoldInput` has a new kind, `gated`, so one exhaustive over `input.kind` no longer compiles either; `RunState` has a
|
|
3447
|
+
new cell, `gated`, so a `RunState` you construct needs it; and a `route` key on the edge of a node that is not a gate
|
|
3448
|
+
was refused as `FD-V17` and is refused as `FD-V41` — the same document refused under another rule id, so code
|
|
3449
|
+
matching a finding's `rule` moves. A flow with no gate records no `gated` input and has `gated: []`.
|
|
3450
|
+
|
|
3451
|
+
## 20. What a ROUTE buys an AUTHOR — wiring a gate before it runs
|
|
3452
|
+
|
|
3453
|
+
**Section 15 showed what a schema buys whoever is wiring a flow. A gate adds something a schema alone
|
|
3454
|
+
cannot say.** A gate routes by a KEY PATH into the value reaching it — `owner.id`, `series.id` — and two
|
|
3455
|
+
things follow that an author has to get right before anything runs:
|
|
3456
|
+
|
|
3457
|
+
1. **The path has to be one the producer can actually carry.** Misspell it and the key resolves for no
|
|
3458
|
+
branch, so every branch takes whichever route the gate's TYPE gives an unresolved key: the run
|
|
3459
|
+
completes, having done none of the work the gate guards. Nothing about the document is wrong —
|
|
3460
|
+
`owner.ident` is a perfectly good string — so the grammar accepts it, and step 5 below RUNS that
|
|
3461
|
+
case rather than describing it. *(Which route an unresolved key takes is the type's judgement, not
|
|
3462
|
+
the library's: `FD-V41` requires only that every route a type declares is wired exactly once, and a
|
|
3463
|
+
type need not declare a no-key route at all. `distinct` and the `quota` below both do.)*
|
|
3464
|
+
2. **On the routes where the path DID resolve, the value carries more than its producer promised.** A
|
|
3465
|
+
producer may declare `owner` optional; a branch leaving by a key-resolved route has an `owner`, or it
|
|
3466
|
+
would not have left that way. A consumer requiring it is correctly wired, and only the route says so.
|
|
3467
|
+
|
|
3468
|
+
**`checkWiring` answers both, and it answers them from the gate TYPE's own declaration**
|
|
3469
|
+
([ADR 0068](doc/decisions/0068-a-route-guarantees-what-its-type-declares-and-the-library-derives-it-from-the-key-path.md)):
|
|
3470
|
+
a type lists the routes on which its key path resolved — `keyResolvedRoutes` — and the library derives
|
|
3471
|
+
the guarantee from the path. `distinct` declares `first` and `duplicate`; the type below declares two of
|
|
3472
|
+
its three routes, so the third is the control sitting in the same report.
|
|
3473
|
+
|
|
3474
|
+
| the author writes | the check says |
|
|
3475
|
+
|---|---|
|
|
3476
|
+
| a key path the producer FORBIDS at any step | **`no`** on every arc leaving by a key-resolved route, naming the gate, the key, the STEP it stopped on, the word the schema itself uses, and the route |
|
|
3477
|
+
| a key path the producer describes | those arcs carry `required` along the path, so a consumer requiring it **`fits`** |
|
|
3478
|
+
| the same arc, with the type declaring no guarantee | **`no`** again — *consumer requires "owner", producer does not guarantee it* |
|
|
3479
|
+
| an arc leaving by a route the type does NOT call key-resolved | its own verdict, unchanged — the refusal is the ROUTE's and not the gate's |
|
|
3480
|
+
|
|
3481
|
+
**The type never walks the key path.** The library resolves it and hands the value over as `ctx.key`,
|
|
3482
|
+
which is why the library can derive what a route guarantees while the type decides what to do with it —
|
|
3483
|
+
including deciding that a path which resolved to the empty string is not a key, which is one of the two
|
|
3484
|
+
roads to `nokey` the run below takes.
|
|
3485
|
+
|
|
3486
|
+
**The registry, the gate type and the flow are
|
|
3487
|
+
[`examples/getting-started/quota-registry.ts`](examples/getting-started/quota-registry.ts)**, imported
|
|
3488
|
+
rather than written out here, for the reason that file's header gives: the session needs a second
|
|
3489
|
+
observer and a snippet cannot be one, so `test/definition/gate-session.test.ts` asserts the same
|
|
3490
|
+
verdicts from the same declarations.
|
|
3491
|
+
|
|
3492
|
+
```ts run
|
|
3493
|
+
import { checkWiring, distinct } from 'fyflow-orchestration';
|
|
3494
|
+
import type { GateType } from 'fyflow-orchestration';
|
|
3495
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
3496
|
+
// Everything the session claims lives in ONE module, so a TEST asserts the same verdicts from the same
|
|
3497
|
+
// declarations — `test/definition/gate-session.test.ts`. A snippet cannot be its own second observer.
|
|
3498
|
+
import { QUOTA, QUOTA_BASE, QUOTA_FILES, QUOTA_REGISTRY, QUOTA_WIRING, quotaFlow } from '../../examples/getting-started/quota-registry.ts';
|
|
3499
|
+
|
|
3500
|
+
// `report` has exactly ONE arc — the one leaving by `within`, the route the quota guards.
|
|
3501
|
+
const arc = async (key: string) => {
|
|
3502
|
+
const verdicts = (await checkWiring(QUOTA_WIRING, quotaFlow(key))).verdicts;
|
|
3503
|
+
const mine = verdicts.filter((v) => v.consumer === 'report');
|
|
3504
|
+
if (mine.length !== 1) throw new Error(`one arc into report, not ${mine.length}`);
|
|
3505
|
+
return mine[0]!;
|
|
3506
|
+
};
|
|
3507
|
+
|
|
3508
|
+
// ---- 1. the author misspells the key. Nothing is wrong with the DOCUMENT. ----
|
|
3509
|
+
const typo = await arc('ownr.id');
|
|
3510
|
+
console.log(typo.kind, '—', typo.why);
|
|
3511
|
+
// no — gate "quota" is keyed on "ownr.id", which the producer's schema FORBIDS at $: the schema forbids
|
|
3512
|
+
// every key it does not name (additionalProperties: false) and it names file, owner — so route "within"
|
|
3513
|
+
// can never be taken, and a run may route every branch by another route, doing none of the work this gate guards
|
|
3514
|
+
if (typo.kind !== 'no') throw new Error(`a forbidden key path is a wiring failure, got ${typo.kind}`);
|
|
3515
|
+
const NAMED = ['is keyed on "ownr.id"', 'FORBIDS at $:', 'it names file, owner', 'route "within"'];
|
|
3516
|
+
if (NAMED.length !== 4) throw new Error('the reason names FOUR things and each is asserted');
|
|
3517
|
+
for (const part of NAMED) {
|
|
3518
|
+
if (part === '' || !typo.why.includes(part)) throw new Error(`the reason names the gate, the key, the step, the schema's own words and the route: ${part}`);
|
|
3519
|
+
}
|
|
3520
|
+
|
|
3521
|
+
// ---- 2. one step in, the reason moves with it ----
|
|
3522
|
+
const deeper = await arc('owner.ident');
|
|
3523
|
+
if (!deeper.why.includes('FORBIDS at $.owner:') || !deeper.why.includes('it names id, name')) {
|
|
3524
|
+
throw new Error(`the STEP it stopped on is where to look: ${deeper.why}`);
|
|
3525
|
+
}
|
|
3526
|
+
|
|
3527
|
+
// ---- 3. the refusal belongs to the ROUTE. `skip` has two arcs and two verdicts at once. ----
|
|
3528
|
+
const both = (await checkWiring(QUOTA_WIRING, quotaFlow('ownr.id'))).verdicts.filter((v) => v.consumer === 'skip');
|
|
3529
|
+
console.log(both.map((v) => v.kind)); // [ 'no', 'fits' ]
|
|
3530
|
+
// A verdict carries no `route` field (Q410), so the ROUTE is read from the reason the refusal itself
|
|
3531
|
+
// gives: the `no` arc names `over`, and swapping which routes are key-resolved is caught here.
|
|
3532
|
+
const refused = both.filter((v) => v.kind === 'no');
|
|
3533
|
+
if (refused.length !== 1 || !refused[0]!.why.includes('route "over"')) {
|
|
3534
|
+
throw new Error('exactly the over arc is refused and its reason says so: ' + both.map((v) => v.kind + ' ' + v.why).join(' | '));
|
|
3535
|
+
}
|
|
3536
|
+
if (both.filter((v) => v.kind === 'fits').length !== 1) throw new Error('and the nokey arc, which the type does not call key-resolved, keeps its own verdict');
|
|
3537
|
+
|
|
3538
|
+
// ---- 4. spelled right, it fits — and the GUARANTEE is what makes it fit ----
|
|
3539
|
+
const right = await arc('owner.id');
|
|
3540
|
+
if (right.kind !== 'fits') throw new Error(`the path the producer describes: ${right.why}`);
|
|
3541
|
+
const { keyResolvedRoutes: _none, ...noGuarantee } = QUOTA;
|
|
3542
|
+
const blind = (await checkWiring({ ...QUOTA_WIRING, gateTypes: { quota: noGuarantee as GateType } }, quotaFlow('owner.id'))).verdicts;
|
|
3543
|
+
const unguaranteed = blind.find((v) => v.consumer === 'report')!;
|
|
3544
|
+
console.log(unguaranteed.kind, '—', unguaranteed.why); // no — $: consumer requires "owner", producer does not guarantee it
|
|
3545
|
+
if (unguaranteed.kind !== 'no' || !unguaranteed.why.includes('does not guarantee it')) {
|
|
3546
|
+
throw new Error('with no declared guarantee the same arc is `no`, which is what the declaration buys');
|
|
3547
|
+
}
|
|
3548
|
+
// The built-in type declares its own two, which is where this member came from.
|
|
3549
|
+
if ([...(distinct.keyResolvedRoutes ?? [])].join() !== 'first,duplicate') throw new Error('`distinct` declares first and duplicate');
|
|
3550
|
+
|
|
3551
|
+
// ---- 5. and it runs. ONE run, so the quota is shared by every branch. ----
|
|
3552
|
+
// `base` is the REGISTRY's own url, never the caller's: its worker urls are relative to the module that
|
|
3553
|
+
// declares them. Point it at the wrong base and the workers cannot load: every task of each such type fails
|
|
3554
|
+
// naming the worker URL and the load error — on a non-optional node it is parked user_action, so `done` waits
|
|
3555
|
+
// on a person (W141, Q266) — so this is a line to get right rather than to debug.
|
|
3556
|
+
const runtime = createRuntime(QUOTA_REGISTRY, { gateTypes: { quota: QUOTA } }, { base: QUOTA_BASE });
|
|
3557
|
+
const { done } = runtime.start(quotaFlow('owner.id'), { files: QUOTA_FILES });
|
|
3558
|
+
const state = await done;
|
|
3559
|
+
|
|
3560
|
+
console.log(state.outcome, state.gated.map((d) => `${d.key ?? '-'} ${d.route}`));
|
|
3561
|
+
// completed [ 'ada within', 'ada within', 'ada over', 'grace within', ' nokey', '- nokey' ]
|
|
3562
|
+
if (state.outcome !== 'completed') throw new Error(`expected completed, got ${state.outcome}`);
|
|
3563
|
+
const routes = state.gated.map((d) => d.route).sort();
|
|
3564
|
+
if (routes.join() !== 'nokey,nokey,over,within,within,within') throw new Error(`two within the quota per owner, ada's third over, and two roads to nokey: ${routes.join()}`);
|
|
3565
|
+
const over = state.gated.filter((d) => d.route === 'over');
|
|
3566
|
+
if (over.length !== 1 || over[0]!.key !== 'ada') throw new Error('the owner with three files is the one who goes over');
|
|
3567
|
+
// BOTH roads to `nokey`: a path that RESOLVES to the empty string, which the TYPE refuses, and a path
|
|
3568
|
+
// that does not resolve at all, which the library never resolved.
|
|
3569
|
+
const nokeys = state.gated.filter((d) => d.route === 'nokey').map((d) => (d.key === '' ? 'empty' : d.key === undefined ? 'unresolved' : `?${String(d.key)}`)).sort();
|
|
3570
|
+
if (nokeys.join() !== 'empty,unresolved') throw new Error(`the two roads to nokey are both taken: ${nokeys.join()}`);
|
|
3571
|
+
// And the work the guarantee licensed actually HAPPENED: the run completing is not evidence on its own.
|
|
3572
|
+
const did = Object.values(state.nodes).filter((n) => n.nodeId === 'report').map((n) => (n.result as { owner: string }).owner).sort();
|
|
3573
|
+
if (did.join() !== 'ada,ada,grace') throw new Error(`three reports ran, and each read the owner the route guaranteed: ${did.join()}`);
|
|
3574
|
+
|
|
3575
|
+
// ---- and the run the MISSPELLED key gives, which is what being wrong costs ----
|
|
3576
|
+
const wrong = await (runtime.start(quotaFlow('ownr.id'), { files: QUOTA_FILES })).done;
|
|
3577
|
+
await runtime.shutdown();
|
|
3578
|
+
console.log(wrong.outcome, wrong.gated.map((d) => d.route)); // completed [ 'nokey', 'nokey', ... ]
|
|
3579
|
+
if (wrong.outcome !== 'completed') throw new Error(`the run COMPLETES, which is what makes this worth checking: ${wrong.outcome}`);
|
|
3580
|
+
if (wrong.gated.some((d) => d.route !== 'nokey')) throw new Error('every branch takes the unresolved route');
|
|
3581
|
+
if (Object.values(wrong.nodes).some((n) => n.nodeId === 'report')) throw new Error('and NONE of the work the gate guards runs');
|
|
3582
|
+
```
|
|
3583
|
+
|
|
3584
|
+
**What the check does NOT say, each measured rather than assumed.** An **entry gate** takes no verdict at
|
|
3585
|
+
all — it has no arc, so a key path misspelled there is caught by nothing this report carries
|
|
3586
|
+
([Q416](doc/questions/q416.md)). A gate wired into a **join** is neither refined nor refused, because a
|
|
3587
|
+
join's record is built from what arrived AT the gate ([Q415](doc/questions/q415.md)). A container declared
|
|
3588
|
+
by `enum` or `const` rather than by `properties` can forbid a step in silence ([Q417](doc/questions/q417.md)).
|
|
3589
|
+
A verdict carries **no `route` field** ([Q410](doc/questions/q410.md)), so which route an arc left by is
|
|
3590
|
+
read from the reason a refusal gives rather than from the report's shape — which is why step 3 asserts
|
|
3591
|
+
the reason and not the order. And a type that declares **no** `keyResolvedRoutes` is asked no question
|
|
3592
|
+
about its key path at all — which is step 4's control, and the reason the declaration is worth writing.
|
|
3593
|
+
|
|
3594
|
+
## 21. A flow a UI could have emitted — what a document can say without code
|
|
3595
|
+
|
|
3596
|
+
**Sections 4 and 13 gave you the two halves: a `when` that may be DATA and a built-in type that only
|
|
3597
|
+
transforms. This is what they add up to.** A flow document plus a registry document, and the only
|
|
3598
|
+
TypeScript is the domain workers and one import of the language adapter the library itself ships.
|
|
3599
|
+
|
|
3600
|
+
**What that covers**, which is [P26](doc/plans/p26-what-a-flow-can-say-without-code.md) prediction 3
|
|
3601
|
+
and [W110](doc/work-items/w110-a-flow-a-ui-could-have-emitted.md)'s to answer:
|
|
3602
|
+
|
|
3603
|
+
| what a flow has to say | how, in data | what it costs in code |
|
|
3604
|
+
|---|---|---|
|
|
3605
|
+
| **sequence** | `edges: [{ to }]` | nothing |
|
|
3606
|
+
| **fan-out** at run-time arity | a `map` over a key | nothing |
|
|
3607
|
+
| **join** | a `join` naming its opener | nothing |
|
|
3608
|
+
| **selection** — give this node that node's output | `needs` on the consuming node (tier 1) | nothing |
|
|
3609
|
+
| **decision** | a `when`: a comparison or a flat `all`/`any` group | nothing |
|
|
3610
|
+
| the same decision, when the question is **outside the subset** | a `when` naming a **registered predicate** | **a function** — this is the half that is not data, which is P26's own framing |
|
|
3611
|
+
| **the join policy** a run is started with | nothing in the document: it is an argument to `runtime.start`, defaulted to `live-token` | the library's own, or a consumer's |
|
|
3612
|
+
| **computation feeding a worker** | `transform` on that worker's own node (tier 2) | the **language adapter** |
|
|
3613
|
+
| **computation feeding a DECISION or a join** | a `Transform` node (tier 3) | the **language adapter** |
|
|
3614
|
+
|
|
3615
|
+
**Two narrowings, both measured, because the table above is the interesting part and the honest part
|
|
3616
|
+
is underneath it.**
|
|
3617
|
+
|
|
3618
|
+
1. **Every `transform` names a `lang`, and no language ships registered.** `Registry.languages` is
|
|
3619
|
+
EMPTY when absent, so a transform against a registry naming none is `FD-V35`. The adapter is the
|
|
3620
|
+
library's own sub-path — `import { jsonata } from 'fyflow-orchestration/jsonata'` — so it is one
|
|
3621
|
+
import and one object entry with no logic in it, but it is not nothing and it is not a domain
|
|
3622
|
+
worker. **A flow that only SELECTS needs no adapter; a flow that COMPUTES does.**
|
|
3623
|
+
2. **A comparison reads ONE path and compares it to ONE literal.** It cannot name a second path and
|
|
3624
|
+
it cannot do arithmetic, so *is the mean below the threshold this run carries* is not a comparison.
|
|
3625
|
+
**The route around it is data**: a tier-three `Transform` node computes the answer and the `when`
|
|
3626
|
+
reads `outcome.result.<field>`. That is what the flow below does, and it is the one place the
|
|
3627
|
+
three-tier rule's *reshape for … a decision* is load-bearing rather than advisory — a `transform`
|
|
3628
|
+
transforms its node's INPUT and a decision reads its node's OUTPUT, so moving the computation onto
|
|
3629
|
+
the consuming node would leave the decision unable to see it.
|
|
3630
|
+
|
|
3631
|
+
> **A comparison whose `value` is meant to be a path is NOT refused** — `{ "op": "eq", "value":
|
|
3632
|
+
> "needs.cfg.threshold" }` is a perfectly good string comparison against the literal text, and it is
|
|
3633
|
+
> false for ever. The grammar catches a wrong ROOT (`FD-V43`) and nothing catches this
|
|
3634
|
+
> ([Q456](doc/questions/q456.md)). Write the literal, or compute the comparison in a `Transform` node.
|
|
3635
|
+
|
|
3636
|
+
> **This flow is `examples/flows/ui-emitted.flow.json` and this registry is
|
|
3637
|
+
> `examples/ui-registry.json`, with the SCHEMAS reduced to what these three verdicts turn on.** The
|
|
3638
|
+
> nodes, the edges, the `needs`, both transforms and both groups are the shipped document's; the
|
|
3639
|
+
> worker entries carry two schemas each instead of the corpus's full set, because the point here is
|
|
3640
|
+
> which arcs are checked and not what the corpus declares. **Nothing compares the two copies** — a
|
|
3641
|
+
> check would have to hold a TypeScript literal against a JSON document, where `X8` has byte-equality
|
|
3642
|
+
> to work with and this has none — so they are two copies and this sentence is what says so
|
|
3643
|
+
> ([Q458](doc/questions/q458.md)). The shipped pair is what ladder L6 RUNS, on real workers, in both
|
|
3644
|
+
> runtimes; this is what a reader can read.
|
|
3645
|
+
|
|
3646
|
+
```ts run
|
|
3647
|
+
import { checkWiring, validateDefinition, type FlowDefinition, type Registry } from 'fyflow-orchestration';
|
|
3648
|
+
// **The ONE import that is code**, and it is the library's own sub-path rather than anything you write.
|
|
3649
|
+
import { jsonata } from 'fyflow-orchestration/jsonata';
|
|
3650
|
+
|
|
3651
|
+
// ---- the flow a UI emits. No predicate names it, and nothing here is a function. ----
|
|
3652
|
+
const flow: FlowDefinition = {
|
|
3653
|
+
flowId: 'ui-emitted', version: 1, entry: 'intake',
|
|
3654
|
+
nodes: [
|
|
3655
|
+
{ id: 'intake', kind: 'task', type: 'SplitWorker', edges: [{ to: 'each' }] },
|
|
3656
|
+
{ id: 'each', kind: 'map', over: 'items', branch: 'rate' },
|
|
3657
|
+
{ id: 'rate', kind: 'task', type: 'ScoreWorker', edges: [{ to: 'gather' }] },
|
|
3658
|
+
{ id: 'gather', kind: 'join', of: 'each', edges: [{ to: 'rollup' }] },
|
|
3659
|
+
// TIER THREE — the computation a comparison cannot express, as a node with NO worker.
|
|
3660
|
+
{
|
|
3661
|
+
id: 'rollup', kind: 'task', type: 'Transform',
|
|
3662
|
+
needs: { joined: '.', cfg: 'intake' },
|
|
3663
|
+
transform: {
|
|
3664
|
+
lang: 'jsonata',
|
|
3665
|
+
expr: '( $s := [joined.*.score]; { "n": $count($s), "total": $sum($s), "mean": $sum($s) / $count($s), '
|
|
3666
|
+
+ '"weak": ($sum($s) / $count($s)) < (cfg.confidence * 100) } )',
|
|
3667
|
+
},
|
|
3668
|
+
edges: [
|
|
3669
|
+
// A flat GROUP, and both members decide: `weak` on one payload, `n` at one item.
|
|
3670
|
+
{ when: { all: [{ path: 'outcome.result.weak', op: 'eq', value: true }, { path: 'outcome.result.n', op: 'gte', value: 2 }] }, to: 'remediate' },
|
|
3671
|
+
{ when: { any: [{ path: 'outcome.result.n', op: 'eq', value: 1 }, { path: 'outcome.result.weak', op: 'eq', value: false }] }, to: 'publish' },
|
|
3672
|
+
// The third edge is a TRIPWIRE: a wrong group lands at a named `end` rather than falling through.
|
|
3673
|
+
{ to: 'stalled' },
|
|
3674
|
+
],
|
|
3675
|
+
},
|
|
3676
|
+
// TIER TWO — a transform on the node whose worker consumes it, so no extra task.
|
|
3677
|
+
{
|
|
3678
|
+
id: 'remediate', kind: 'task', type: 'WeighWorker',
|
|
3679
|
+
transform: { lang: 'jsonata', expr: '{ "item": { "word": "mean", "confidence": mean / 100 }, "run": { "confidence": 1 } }' },
|
|
3680
|
+
edges: [{ when: { path: 'outcome.result.score', op: 'lt', value: 20 }, to: 'escalated' }, { to: 'done' }],
|
|
3681
|
+
},
|
|
3682
|
+
{ id: 'publish', kind: 'task', type: 'PublishWorker', edges: [{ to: 'done' }] },
|
|
3683
|
+
{ id: 'escalated', kind: 'end', outcome: 'escalated-to-a-person' },
|
|
3684
|
+
{ id: 'stalled', kind: 'end', outcome: 'no-branch-matched' },
|
|
3685
|
+
{ id: 'done', kind: 'end', outcome: 'completed' },
|
|
3686
|
+
],
|
|
3687
|
+
};
|
|
3688
|
+
|
|
3689
|
+
// ---- the registry it names. `Transform` is NOT in it: a built-in is merged at every read. ----
|
|
3690
|
+
const registry: Registry = {
|
|
3691
|
+
// **EMPTY, and that is the claim.** A declared `when` resolves nothing against this.
|
|
3692
|
+
predicates: [],
|
|
3693
|
+
workers: {
|
|
3694
|
+
split: {
|
|
3695
|
+
url: './split.ts',
|
|
3696
|
+
outputSchema: {
|
|
3697
|
+
type: 'object', required: ['confidence', 'items'],
|
|
3698
|
+
properties: {
|
|
3699
|
+
confidence: { type: 'number' },
|
|
3700
|
+
// The ITEM's own shape, so the arc into the map is checked — without it that arc is
|
|
3701
|
+
// `unchecked` for a reason that has nothing to do with the transform node.
|
|
3702
|
+
items: { type: 'array', items: { type: 'object', required: ['word', 'confidence'], properties: { word: { type: 'string' }, confidence: { type: 'number' } } } },
|
|
3703
|
+
},
|
|
3704
|
+
},
|
|
3705
|
+
},
|
|
3706
|
+
score: {
|
|
3707
|
+
url: './score.ts',
|
|
3708
|
+
inputSchema: { type: 'object', properties: { word: { type: 'string' }, confidence: { type: 'number' } } },
|
|
3709
|
+
outputSchema: { type: 'object', required: ['score'], properties: { score: { type: 'number' } } },
|
|
3710
|
+
},
|
|
3711
|
+
weigh: { url: './weigh.ts', inputSchema: { type: 'object', properties: { item: { type: 'object' }, run: { type: 'object' } } } },
|
|
3712
|
+
publish: { url: './publish.ts' },
|
|
3713
|
+
},
|
|
3714
|
+
nodeTypes: {
|
|
3715
|
+
SplitWorker: { worker: 'split' }, ScoreWorker: { worker: 'score' },
|
|
3716
|
+
WeighWorker: { worker: 'weigh' }, PublishWorker: { worker: 'publish' },
|
|
3717
|
+
},
|
|
3718
|
+
// The one thing that is code: the adapter for the language the transforms name.
|
|
3719
|
+
languages: { jsonata },
|
|
3720
|
+
};
|
|
3721
|
+
|
|
3722
|
+
const findings = validateDefinition(flow, registry);
|
|
3723
|
+
console.log(findings.length); // 0
|
|
3724
|
+
if (findings.length !== 0) throw new Error(`a UI-emitted flow validates with NO predicate registered: ${JSON.stringify(findings)}`);
|
|
3725
|
+
|
|
3726
|
+
// **The registry really is empty** — an unused empty map would prove nothing (W110's careful-point 1).
|
|
3727
|
+
if ((registry.predicates as readonly string[]).length !== 0) throw new Error('no predicate is registered anywhere');
|
|
3728
|
+
|
|
3729
|
+
// ---- and what the author gets told about the wiring, which is where the residue is ----
|
|
3730
|
+
const report = await checkWiring(registry, flow);
|
|
3731
|
+
console.log(report.counts); // { fits: 1, no: 0, ..., unchecked: 3 }
|
|
3732
|
+
// `intake -> rate` is the one arc neither end of which is the transform node, and it FITS.
|
|
3733
|
+
const leaving = report.verdicts.filter((v) => v.producer === 'rollup').map((v) => v.kind);
|
|
3734
|
+
console.log(leaving); // [ 'unchecked', 'unchecked' ]
|
|
3735
|
+
if (leaving.length !== 2 || leaving.some((v) => v !== 'unchecked')) {
|
|
3736
|
+
throw new Error(`both arcs LEAVING the transform-only node are unchecked — it declares no outputSchema, which is Q446: ${JSON.stringify(leaving)}`);
|
|
3737
|
+
}
|
|
3738
|
+
// The arc INTO it is unchecked for the mirror reason — no inputSchema — so the residue is THREE arcs.
|
|
3739
|
+
if (report.counts.unchecked !== 3) throw new Error(`three arcs touch the node and all three are unchecked: ${report.counts.unchecked}`);
|
|
3740
|
+
// And the arcs that do not touch it are checked as any other arc is, so the residue is LOCAL to the node.
|
|
3741
|
+
if (report.counts.fits !== 1 || report.counts.no !== 0) throw new Error(`the rest of the flow is checked as usual: ${JSON.stringify(report.counts)}`);
|
|
3742
|
+
```
|
|
3743
|
+
|
|
3744
|
+
**What the check cannot tell you here, said rather than left to be discovered.** Every arc LEAVING a
|
|
3745
|
+
transform-only node is `unchecked`, because the node declares no `outputSchema` and a sample is
|
|
3746
|
+
evidence about one input where `fits` is a claim about every input
|
|
3747
|
+
([Q446](doc/questions/q446.md), and [ADR 0075](doc/decisions/0075-a-node-that-only-transforms-is-a-built-in-type-with-no-inner-worker.md)
|
|
3748
|
+
rule 6 is why it is not inferred). `sampledOutput` in the assists gives you the shape the node
|
|
3749
|
+
actually emitted on the producer's own example, with those verdicts beside it — so the residue is
|
|
3750
|
+
reported rather than hidden. And a declared predicate's PATH is checked against no schema at all, so
|
|
3751
|
+
a path naming a key the producer cannot emit answers *no match* for ever (ADR 0074 consequence 6).
|
|
3752
|
+
|
|
3753
|
+
> **A `Transform` node's realm and its groups are the scheduler's default and nothing else.** The
|
|
3754
|
+
> built-in declares no `inline`, no `groups` and no `pool`, and a named pool instance carries only
|
|
3755
|
+
> `type` and `config` — so of the four things ADR 0075 rule 3 lists, a named instance reaches
|
|
3756
|
+
> `config`, a `retryPolicy` goes on the NODE, and `inline` and `groups` are reachable by no road a
|
|
3757
|
+
> consumer can write ([Q457](doc/questions/q457.md)). Measured at W110.
|