fyflow-orchestration 0.13.0 → 0.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +146 -12
- package/README.md +1 -1
- package/dist/browser/chunks/{chunk-53EDI674.js → chunk-5KUYTCXI.js} +111 -8
- package/dist/browser/index.js +1 -1
- package/dist/browser/scheduler/index.js +17 -6
- package/dist/node/chunks/{chunk-53EDI674.js → chunk-5KUYTCXI.js} +111 -8
- package/dist/node/index.js +1 -1
- package/dist/node/scheduler/index.js +17 -6
- package/dist/types/src/definition/registry.d.ts +18 -10
- package/dist/types/src/definition/types.d.ts +25 -5
- package/dist/types/src/scheduler/runtime.d.ts +14 -10
- package/package.json +3 -3
package/AGENTS.md
CHANGED
|
@@ -2687,7 +2687,7 @@ flow, and the class says which document to open.
|
|
|
2687
2687
|
member and the group's key; `GroupNotBuildable` — a group type with no `create`; since W89, `RunIdInUse`
|
|
2688
2688
|
— a `start` or `resume` naming a run id that runtime has already built a driver for, a finished run's included (`RN-R25`, ADR 0091);
|
|
2689
2689
|
and, since W163, **`LimitKeyMissing`** — a `start` or `resume` given no `createTask`, of a flow a node of which reaches a
|
|
2690
|
-
group
|
|
2690
|
+
group a built-in keyed factory built — `KeyedRateLimitGroup`, or since spec v0.56 `KeyedConcurrentLimitGroup` (`FD-R6`, [ADR 0097](doc/decisions/0097-what-start-refuses-that-the-flow-does-not-decide.md), section 12), its `at` the node.
|
|
2691
2691
|
*(`LanguageNotBuilt` — a flow transforming on a type no pool was built for — was another until W162 removed it
|
|
2692
2692
|
with `createRuntime`'s `flows`: every typed pool carries every registered language since
|
|
2693
2693
|
[ADR 0096](doc/decisions/0096-a-runtime-is-its-registry-and-start-refuses-from-runtime-check.md), section 12.)* A bare
|
|
@@ -3237,7 +3237,7 @@ if (!haltedRun.haltedBy?.startsWith('UncloneableResult')) throw new Error(`halte
|
|
|
3237
3237
|
|---|---|
|
|
3238
3238
|
| flow-level **retry** — re-dispatching a NODE under a new attempt, possibly routing elsewhere | **task-level retry IS built** since [W58](doc/work-items/w58-what-a-node-says-about-its-own-task.md): declare `retryPolicy` on the node (section 3) and the scheduler retries inside the same task id, reported on the record's `attempts`. Re-dispatching the node is [Q16](doc/questions/q16.md)'s other half and the task id's fourth segment, and nothing mints one |
|
|
3239
3239
|
| **`runtime.register()`** — adding node types, pools, groups or languages to a RUNNING runtime | **declined** — [P37](doc/plans/p37-a-runtime-is-its-registry.md) decision 1, [ADR 0096](doc/decisions/0096-a-runtime-is-its-registry-and-start-refuses-from-runtime-check.md): `fyflow-scheduler` cannot add a pool after construction. A changed registry is a NEW runtime; a run on the old one finishes there or moves by stop-and-resume ([P34](doc/plans/p34-what-cancelling-stopping-and-pausing-a-run-do.md)). **A FLOW is not a registry entry**: `runtime.check(flow)`, then `runtime.start(flow, …)`, takes any flow the registry can run, at any time (section 12) |
|
|
3240
|
-
| **a `limitKey` from the FLOW** — a node or a document saying which key a task of a keyed group is counted under | **deferred** — [Q248](doc/questions/q248.md). Pass `start` a `createTask` that sets `limitKey` on the task it builds, or register a group type whose `create` builds a `KeyedRateLimitGroup` with a `keyFrom` (section 12). A node reaching a document's built-in keyed group with neither is refused before its run starts — `FD-R6`, `LimitKeyMissing` |
|
|
3240
|
+
| **a `limitKey` from the FLOW** — a node or a document saying which key a task of a keyed group is counted under | **deferred** — [Q248](doc/questions/q248.md). Pass `start` a `createTask` that sets `limitKey` on the task it builds, or register a group type whose `create` builds a `KeyedRateLimitGroup` or a `KeyedConcurrentLimitGroup` with a `keyFrom` (section 12). A node reaching a document's built-in keyed group with neither is refused before its run starts — `FD-R6`, `LimitKeyMissing` |
|
|
3241
3241
|
| **a `start()` that checks a TRANSFORMING task's payload** — the transform run first and its result held to the `inputSchema` | **no** — [ADR 0101](doc/decisions/0101-a-transforming-task-is-not-held-at-start.md): `start()` is synchronous and the transform runs only in the worker, so a task declaring one is UNCHECKED at `start` (`FD-R4`), as a worker declaring no `inputSchema` is. Call `givenVerdict(registry, flow, task, received)` — `tryTransform(…, { input: received })` on the declared transform — first, on each transforming target with the value it RECEIVES (section 15), then start |
|
|
3242
3242
|
| **redaction by `writeOnly`** — a schema annotation keeping a value out of what a run retains or reports | **no** — [ADR 0098](doc/decisions/0098-four-annotations-join-the-schema-subset-and-constrain-nothing.md) rule 4: `writeOnly`, like `readOnly`, `title` and `description`, is admitted as an annotation since v0.36, held to its shape and read by nothing after. Redact in your own code, before the value reaches a run (section 13) |
|
|
3243
3243
|
| **pool options on a CLASS type's instance**, and the scheduler's `idleCheckIntervalMs`, `requeueFailedTasks` and `maxWorkerRestarts` anywhere | **no** — [ADR 0099](doc/decisions/0099-a-named-instance-of-a-document-type-may-declare-its-own-pool-options.md) rules 4 and 7: a named instance of a DOCUMENT type may declare `pool` (section 13); a class computes its pool from the instance's `config` and key, so give it a `configSchema` knob. The three are refused as undeclared keys |
|
|
@@ -3344,7 +3344,7 @@ if (Object.keys(runtime.groups).join() !== 'gate') throw new Error(`groups: ${Ob
|
|
|
3344
3344
|
|
|
3345
3345
|
**A custom group type is a FACTORY, and the runtime holds its output to what the scheduler CALLS.**
|
|
3346
3346
|
The document names a group by KEY with a `type`; the code supplies `{ configSchema?, create(config,
|
|
3347
|
-
key) }` under that type name; the
|
|
3347
|
+
key) }` under that type name; the five built-ins are pre-registered in the same shape (section 13). What comes
|
|
3348
3348
|
back is checked for ADR 0055's six members and refused by name otherwise. Read the counts off the
|
|
3349
3349
|
object the runtime built — [W70](doc/work-items/w70-a-flow-that-decides-and-a-custom-group-on-the-runtime.md):
|
|
3350
3350
|
|
|
@@ -3376,8 +3376,9 @@ class CountingGroup extends EventTarget implements ResourceGroup {
|
|
|
3376
3376
|
}
|
|
3377
3377
|
|
|
3378
3378
|
// THE REGISTRY declares the group by KEY, with its TYPE by name and its config; the CODE supplies the
|
|
3379
|
-
// factory for that type name — `{ configSchema?, create(config, key) }`. The
|
|
3380
|
-
// `ConcurrentLimitGroup`, `RateLimitGroup
|
|
3379
|
+
// factory for that type name — `{ configSchema?, create(config, key) }`. The five built-ins,
|
|
3380
|
+
// `ConcurrentLimitGroup`, `RateLimitGroup`, `KeyedRateLimitGroup`, `UnitGroup` and
|
|
3381
|
+
// `KeyedConcurrentLimitGroup` (section 13), are pre-registered in the same
|
|
3381
3382
|
// shape, so `"type": "ConcurrentLimitGroup"` needs no code at all. A node type names the groups its
|
|
3382
3383
|
// pool's tasks acquire.
|
|
3383
3384
|
const registry: RegistryDocument = {
|
|
@@ -3444,14 +3445,14 @@ so a flow published after the runtime started runs on it, transforming on any ty
|
|
|
3444
3445
|
rules — every one but `FD-R6`, below — before any run id is spent, so the two cannot disagree: *register* a flow by checking it, then
|
|
3445
3446
|
start it. `check` leaves out what depends on a RUN rather than the flow — a payload `FD-R4` refuses,
|
|
3446
3447
|
and a run id in use. **One finding it returns is not a definition defect: `FD-R6`**
|
|
3447
|
-
([ADR 0097](doc/decisions/0097-what-start-refuses-that-the-flow-does-not-decide.md)) — a node reaching a group of
|
|
3448
|
+
([ADR 0097](doc/decisions/0097-what-start-refuses-that-the-flow-does-not-decide.md)) — a node reaching a group of a BUILT-IN keyed type — `KeyedRateLimitGroup`, or (spec v0.56) `KeyedConcurrentLimitGroup` (section 13) — through the
|
|
3448
3449
|
pool it runs on — the named instance its `pool` says, else its type's default — or its `workerGroups`. The runtime builds that group with no `keyFrom`, so the scheduler counts each task
|
|
3449
3450
|
under its `limitKey` — and nothing sets one unless you pass `start` a `createTask` that does, `(c) => new FyflowTask({
|
|
3450
3451
|
...c, limitKey: tenant })`; without one the scheduler's `addTask` throws. So `start` with no `createTask` throws
|
|
3451
3452
|
**`LimitKeyMissing`** for it — after any `InvalidDefinition`, before any run id is spent — and with one, `FD-R6` refuses
|
|
3452
3453
|
nothing (a factory that sets no key meets the scheduler's own throw). `check` cannot know what you will pass, so it
|
|
3453
3454
|
reports `FD-R6` as if you pass none: filter it out by `f.rule` if you start with a keying `createTask`. A group a CODE
|
|
3454
|
-
group type builds — a
|
|
3455
|
+
group type builds — a keyed group with a `keyFrom` — is never `FD-R6`. A pool's `residentGroups` is not read
|
|
3455
3456
|
here: a built-in keyed group held RESIDENT is refused earlier, at `createRuntime` (`DerivationRefused`, section 13,
|
|
3456
3457
|
[Q619](doc/questions/q619.md)). A key derived from the flow is
|
|
3457
3458
|
[Q248](doc/questions/q248.md), deferred. **And `createRuntime` refuses a malformed `defaultRetryPolicy`** —
|
|
@@ -3621,10 +3622,10 @@ anywhere else.)* This library's listener boundary is written against **0.3.0 sem
|
|
|
3621
3622
|
`task.progress`'s detail changed shape at 0.3.0 (the id is `id`, not `taskId`), the second
|
|
3622
3623
|
`task.user_action` dispatch site arrived there, and `task.timestamp` did not exist before it.
|
|
3623
3624
|
**`core/FyflowScheduler.ts` was byte-identical between 0.3.0, 0.5.1 and 0.5.2**; it is not at the
|
|
3624
|
-
current pin, and W157 re-read what this library rests on there rather than inferring it: the
|
|
3625
|
+
current pin, and W157 and W224 re-read what this library rests on there rather than inferring it: the
|
|
3625
3626
|
`user_action` pair is unchanged on both of its paths, and `task.progress` is built from the same
|
|
3626
3627
|
fields. What 0.5.1 added is a `worker.setup.*` / `worker.teardown.*` event family this library
|
|
3627
|
-
subscribes to not at all. **
|
|
3628
|
+
subscribes to not at all. **A sixth task state, `cancelled`, arrived at 0.6.0 and is still the pin's (re-read by W224), reached only through
|
|
3628
3629
|
`scheduler.cancel(id)`**, and a run hears it: a cancelled task ends its run `cancelled` and settles
|
|
3629
3630
|
its `done` (section 6, [P33](doc/plans/p33-what-a-run-does-when-a-task-ends-cancelled.md)) — on real
|
|
3630
3631
|
workers in ladder L6, a task cancelled while running and while queued — and `driver.cancel` asks for one on
|
|
@@ -3956,11 +3957,88 @@ a hand-written `NodeTypeDefinition`, or a copy of a resolved one, is a class to
|
|
|
3956
3957
|
**A `residentGroups` naming a group of the built-in `KeyedRateLimitGroup` type is refused at `createRuntime`** (v0.39,
|
|
3957
3958
|
[Q619](doc/questions/q619.md), W169) — `DerivationRefused`, naming the pool and the group, whether a type, a named
|
|
3958
3959
|
instance or your class's `pool()` put it there. The scheduler resolves no key for a RESIDENT group, so every worker
|
|
3959
|
-
holding it would be counted in ONE `""` bucket, in silence.
|
|
3960
|
-
|
|
3960
|
+
holding it would be counted in ONE `""` bucket, in silence. **A built-in `KeyedConcurrentLimitGroup` is refused the same
|
|
3961
|
+
way** (spec v0.56, W225) — the scheduler rejects one held resident at construction. Name a keyed group in `groups` instead — each task is then
|
|
3962
|
+
counted under its `limitKey` (`FD-R6`, section 12) — or hold a `ConcurrentLimitGroup` or `RateLimitGroup` resident (or a `UnitGroup`). A
|
|
3961
3963
|
group your own `code.groupTypes` factory builds is never refused. **BREAKING in 0.8.0**: such a document built a
|
|
3962
|
-
runtime at 0.7.0. **`bootstrap` called by hand is not covered** — it is handed group OBJECTS and cannot tell the
|
|
3964
|
+
runtime at orchestration 0.7.0. **`bootstrap` called by hand is not covered** — it is handed group OBJECTS and cannot tell the
|
|
3963
3965
|
built-in factory's keyed group from your own — as a `Driver` built by hand is not covered by `FD-R6`.
|
|
3966
|
+
|
|
3967
|
+
**Units with names, a limit per key, and a resident REQUEST — as data** — `spec/flow-definition.md` v0.56,
|
|
3968
|
+
[ADR 0143](doc/decisions/0143-unit-and-keyed-concurrent-groups-are-built-ins-and-a-resident-request-is-checked-against-the-group.md),
|
|
3969
|
+
[W225](doc/work-items/w225-unit-and-keyed-concurrent-groups-and-a-resident-request.md). Two more built-in group types, beside
|
|
3970
|
+
`ConcurrentLimitGroup`, `RateLimitGroup` and `KeyedRateLimitGroup`, each the scheduler's own class:
|
|
3971
|
+
|
|
3972
|
+
| `type` | `config` | built as |
|
|
3973
|
+
|---|---|---|
|
|
3974
|
+
| `UnitGroup` | `{ "units": 2 }` (units `0` and `1`) or `{ "units": ["gpu0", "gpu1"] }` — at least one, the names DISTINCT | `new UnitGroup(units, { id: key })` — a hold is granted named units; no `requestFrom` |
|
|
3975
|
+
| `KeyedConcurrentLimitGroup` | `{ "limit": 4, "limits": { "E:": 1 } }` — every per-key limit a positive integer | `new KeyedConcurrentLimitGroup(limit, { id: key, limitFor })`, `limitFor(k)` reading a COPY of `limits`, else `limit`; no `keyFrom` |
|
|
3976
|
+
|
|
3977
|
+
A repeated unit, or a per-key limit that is not a positive integer, is `GroupConfigRefused` naming the group. **A
|
|
3978
|
+
`KeyedConcurrentLimitGroup` is held to `KeyedRateLimitGroup`'s rules**: a node reaching one is `FD-R6` (section 12), and one
|
|
3979
|
+
held resident is refused, as above. **A `residentGroups` value may be a REQUEST** — `{ "gpu": { "cost": 1, "from":
|
|
3980
|
+
["gpu1"] } }`, `{ cost?, from?, prefer? }`: `from` the units a worker of the pool may hold, `prefer` a hint that never
|
|
3981
|
+
blocks, each a non-empty list of distinct ids (an integer for a `UnitGroup` declared by count). **A resident cost is a
|
|
3982
|
+
positive INTEGER** — a fraction, which the scheduler throws on, was accepted until spec v0.56. Every cost and request is
|
|
3983
|
+
held against the group it names when the runtime is built, `DerivationRefused` naming the pool and the group, each a pool
|
|
3984
|
+
the scheduler would throw on at construction: an id a `UnitGroup` lacks, a cost above `from`'s size or above a group's
|
|
3985
|
+
limit (a `RateLimitGroup`'s smallest window), and `from` / `prefer` on a `ConcurrentLimitGroup` or `RateLimitGroup`, which
|
|
3986
|
+
take no request — `{ cost }` alone is a cost for any group. A request to a group your `code.groupTypes` factory builds is
|
|
3987
|
+
handed on untouched WITHIN the request shape `{ cost?, from?, prefer? }`, for its own `checkRequest` — a request
|
|
3988
|
+
language beyond that shape is not declarable in a document. A TASK's request is its `units`, set by your `createTask`; nothing in a
|
|
3989
|
+
flow declares one.
|
|
3990
|
+
|
|
3991
|
+
```ts run
|
|
3992
|
+
import { createRuntime } from 'fyflow-orchestration/scheduler';
|
|
3993
|
+
import { DerivationRefused, GroupConfigRefused, type FlowDefinition, type RegistryDocument } from 'fyflow-orchestration';
|
|
3994
|
+
import { KeyedConcurrentLimitGroup, UnitGroup } from 'fyflow-scheduler';
|
|
3995
|
+
|
|
3996
|
+
// Two cards, a worker per card, and this pool pinned to the second; a per-disk limit with a slow USB disk.
|
|
3997
|
+
const registry = (resident: unknown, units: unknown = ['gpu0', 'gpu1']): RegistryDocument => ({
|
|
3998
|
+
workers: { ingest: { url: '../../examples/workers/ingest.ts' } },
|
|
3999
|
+
groups: {
|
|
4000
|
+
gpu: { type: 'UnitGroup', config: { units } },
|
|
4001
|
+
disk: { type: 'KeyedConcurrentLimitGroup', config: { limit: 4, limits: { 'E:': 1 } } },
|
|
4002
|
+
},
|
|
4003
|
+
nodeTypes: {
|
|
4004
|
+
Model: { worker: 'ingest', inline: true, pool: { idleTimeout: 1000, residentGroups: resident as never } },
|
|
4005
|
+
Copy: { worker: 'ingest', inline: true, groups: ['disk'] },
|
|
4006
|
+
},
|
|
4007
|
+
});
|
|
4008
|
+
|
|
4009
|
+
const runtime = createRuntime(registry({ gpu: { cost: 1, from: ['gpu1'] } }), {}, { base: import.meta.url });
|
|
4010
|
+
const gpu = runtime.groups['gpu'] as unknown as UnitGroup;
|
|
4011
|
+
const disk = runtime.groups['disk'] as unknown as KeyedConcurrentLimitGroup;
|
|
4012
|
+
if (!(gpu instanceof UnitGroup) || gpu.getFreeUnits().join() !== 'gpu0,gpu1') throw new Error('the scheduler`s own UnitGroup, both cards free');
|
|
4013
|
+
if (!(disk instanceof KeyedConcurrentLimitGroup) || disk.getKeyMetrics('E:').limit !== 1 || disk.getKeyMetrics('C:').limit !== 4) {
|
|
4014
|
+
throw new Error('the per-key limits: E: 1, any other disk 4');
|
|
4015
|
+
}
|
|
4016
|
+
// The pool is handed the request as declared — the scheduler grants this pool's workers gpu1 and nothing else.
|
|
4017
|
+
if (JSON.stringify(runtime.pools['Model']!.spec.residentGroups) !== '{"gpu":{"cost":1,"from":["gpu1"]}}') throw new Error('the request');
|
|
4018
|
+
|
|
4019
|
+
// A node reaching the keyed group needs a `limitKey`, which only your createTask sets: FD-R6, as for KeyedRateLimitGroup.
|
|
4020
|
+
const copy: FlowDefinition = {
|
|
4021
|
+
flowId: 'copy', version: 1, entry: 'c',
|
|
4022
|
+
nodes: [{ id: 'c', kind: 'task', type: 'Copy', edges: [{ to: 'done' }] }, { id: 'done', kind: 'end', outcome: 'completed' }],
|
|
4023
|
+
};
|
|
4024
|
+
if (runtime.check(copy).map((f) => f.rule).join() !== 'FD-R6') throw new Error('FD-R6 at the node reaching `disk`');
|
|
4025
|
+
await runtime.shutdown();
|
|
4026
|
+
|
|
4027
|
+
// What the scheduler would throw on at construction is refused here, naming the pool and the group.
|
|
4028
|
+
const refusal = (doc: RegistryDocument): Error => {
|
|
4029
|
+
try { createRuntime(doc, {}, { base: import.meta.url }); } catch (e) { return e as Error; }
|
|
4030
|
+
return new Error('expected a refusal'); // not a refusal class, so every check below throws
|
|
4031
|
+
};
|
|
4032
|
+
const missing = refusal(registry({ gpu: { from: ['gpu7'] } }));
|
|
4033
|
+
if (!(missing instanceof DerivationRefused) || !missing.message.includes('node type "Model"') || !missing.message.includes('unit "gpu7"')) throw new Error(missing.message);
|
|
4034
|
+
const tooMany = refusal(registry({ gpu: { cost: 2, from: ['gpu1'] } }));
|
|
4035
|
+
if (!(tooMany instanceof DerivationRefused) || !tooMany.message.includes('a cost of 2 from 1')) throw new Error(tooMany.message);
|
|
4036
|
+
const keyedResident = refusal(registry(['disk']));
|
|
4037
|
+
if (!(keyedResident instanceof DerivationRefused) || !keyedResident.message.includes('KeyedConcurrentLimitGroup')) throw new Error(keyedResident.message);
|
|
4038
|
+
const twice = refusal(registry(['gpu'], ['gpu0', 'gpu0']));
|
|
4039
|
+
if (!(twice instanceof GroupConfigRefused) || twice.at !== 'gpu') throw new Error(twice.message);
|
|
4040
|
+
```
|
|
4041
|
+
|
|
3964
4042
|
**Your worker's `run(payload, context)` gets the scheduler's own task context as its second argument** —
|
|
3965
4043
|
the same object the scheduler handed the envelope worker, so `context.signal` (an `AbortSignal`, aborted
|
|
3966
4044
|
when `scheduler.cancel(taskId)` is called for the task) and `context.emit(name, data)` (a `task.message`
|
|
@@ -3970,6 +4048,62 @@ pass `onTaskMessage` to `start` (since 0.8.0, section 8) and each message of the
|
|
|
3970
4048
|
its run, node, branch and attempts as fields — or listen for it on `runtime.scheduler` yourself, as before; what
|
|
3971
4049
|
cancelling a task does to its RUN is section 6's.
|
|
3972
4050
|
|
|
4051
|
+
**Which UNITS your worker holds reaches it in both places the scheduler puts it** — P46,
|
|
4052
|
+
[W226](doc/work-items/w226-grants-reach-a-types-worker-the-reply-and-p46s-close.md), held on real workers by ladder L6's
|
|
4053
|
+
*W226 exit* block, threaded and inline, in Node 24 and Deno. The constructor's second argument is the scheduler's own
|
|
4054
|
+
**worker context**, so **`workerContext.grants`** is what the worker's `residentGroups` granted it — a `UnitGroup`'s
|
|
4055
|
+
units as an array, `{ gpu: ['gpu1'] }` — set BEFORE the constructor runs, so `setup()` can load onto that card; keep the
|
|
4056
|
+
argument (a class extending the scheduler's `BaseWorker` passes it to `super` and reads `this.workerContext`). `run`'s
|
|
4057
|
+
second argument is the task's **`context`**, and **`context.grants`** is that attempt's: the task's own units plus its
|
|
4058
|
+
worker's resident ones — in the scheduler's words, *"its own winning on a clash"*, so a task holding the group its worker
|
|
4059
|
+
holds resident reads its own unit there — per attempt (a retry may get another unit, so read it every time), *"frozen one
|
|
4060
|
+
level down - a copy, on inline pools too"*, and `{}` when nothing was granted — a `ConcurrentLimitGroup` grants nothing. **A task's request is `units` on the task, and `units` set by your
|
|
4061
|
+
`createTask` reaches the scheduler as `limitKey` does**: `(c) => new FyflowTask({ ...c, units: { gpu: { from: ['gpu1']
|
|
4062
|
+
} } })`. A named instance's request reaches the scheduler the same way (section 13's `residentGroups` above); what `from`
|
|
4063
|
+
and `prefer` each mean is the scheduler's own.
|
|
4064
|
+
|
|
4065
|
+
```ts run
|
|
4066
|
+
import { createRuntime, type StartOptions } from 'fyflow-orchestration/scheduler';
|
|
4067
|
+
import type { FlowDefinition, RegistryDocument } from 'fyflow-orchestration';
|
|
4068
|
+
import { FyflowTask } from 'fyflow-scheduler';
|
|
4069
|
+
|
|
4070
|
+
// A type's worker reading both: the card its worker holds (read in the constructor) and what its task was granted.
|
|
4071
|
+
const worker = `data:text/javascript,${encodeURIComponent(`export default class {
|
|
4072
|
+
constructor(config, workerContext) { this.card = workerContext.grants.gpu?.[0] ?? null; }
|
|
4073
|
+
async setup() {}
|
|
4074
|
+
run(p, context) { return { card: this.card, task: context.grants }; } }`)}`;
|
|
4075
|
+
const registry: RegistryDocument = {
|
|
4076
|
+
workers: { model: { url: worker } },
|
|
4077
|
+
groups: { gpu: { type: 'UnitGroup', config: { units: ['gpu0', 'gpu1'] } } },
|
|
4078
|
+
nodeTypes: {
|
|
4079
|
+
Model: { worker: 'model', inline: true, pool: { residentGroups: { gpu: 1 } } }, // a card per WORKER
|
|
4080
|
+
Batch: { worker: 'model', inline: true, groups: ['gpu'] }, // a card per TASK
|
|
4081
|
+
},
|
|
4082
|
+
pools: { 'model-1': { type: 'Model', pool: { residentGroups: { gpu: { cost: 1, from: ['gpu1'] } } } } },
|
|
4083
|
+
};
|
|
4084
|
+
const seen = async (type: string, pool?: string, options: StartOptions = {}): Promise<string> => {
|
|
4085
|
+
const flow: FlowDefinition = { flowId: type, version: 1, entry: 't', nodes: [
|
|
4086
|
+
{ id: 't', kind: 'task', type, ...(pool === undefined ? {} : { pool }), edges: [{ to: 'done' }] },
|
|
4087
|
+
{ id: 'done', kind: 'end', outcome: 'completed' }] };
|
|
4088
|
+
const runtime = createRuntime(registry, {}, { base: import.meta.url }); // its own gpu, both cards free
|
|
4089
|
+
try {
|
|
4090
|
+
const run = runtime.start(flow, {}, options);
|
|
4091
|
+
if ((await run.done).outcome !== 'completed') return `${type} did not complete`; // a sentinel no check below accepts
|
|
4092
|
+
const results = run.driver.foldInputs().flatMap((i) => (i.kind === 'outcome' && i.record.kind === 'outcome' ? [i.record.result] : []));
|
|
4093
|
+
return JSON.stringify(results);
|
|
4094
|
+
} finally {
|
|
4095
|
+
await runtime.shutdown();
|
|
4096
|
+
}
|
|
4097
|
+
};
|
|
4098
|
+
// Resident: the worker holds a card from its constructor on, and its task sees it; the named instance's request reaches it.
|
|
4099
|
+
if (await seen('Model') !== '[{"card":"gpu0","task":{"gpu":["gpu0"]}}]') throw new Error('the lowest free card');
|
|
4100
|
+
if (await seen('Model', 'model-1') !== '[{"card":"gpu1","task":{"gpu":["gpu1"]}}]') throw new Error('the named instance`s request');
|
|
4101
|
+
// Per task: the worker holds nothing, and `units` set by createTask picks the card.
|
|
4102
|
+
const pinned: StartOptions = { createTask: (c) => new FyflowTask({ ...c, units: { gpu: { from: ['gpu1'] } } }) };
|
|
4103
|
+
if (await seen('Batch', undefined, pinned) !== '[{"card":null,"task":{"gpu":["gpu1"]}}]') throw new Error('units through createTask');
|
|
4104
|
+
if (await seen('Batch') !== '[{"card":null,"task":{"gpu":["gpu0"]}}]') throw new Error('no request, the lowest free card');
|
|
4105
|
+
```
|
|
4106
|
+
|
|
3973
4107
|
**A transform — what a node may DO to its input before its worker sees it** — `spec/flow-definition.md`
|
|
3974
4108
|
v0.18, [ADR 0051](doc/decisions/0051-a-transform-is-a-declared-expression-in-a-registered-language-applied-in-the-types-worker-wrapper.md),
|
|
3975
4109
|
[W61](doc/work-items/w61-what-a-transform-is.md). When the value a typed node receives is not the
|
package/README.md
CHANGED
|
@@ -17,7 +17,7 @@ worker returned, routing can **fan out** into parallel branches, and a **join**
|
|
|
17
17
|
The scheduler runs tasks; this library decides **which task runs next**. `fyflow-scheduler` has no
|
|
18
18
|
dependency graph of its own, so sequence, decision, fan-out and join live here.
|
|
19
19
|
|
|
20
|
-
Published on npm as `fyflow-orchestration` and on JSR as `@fyflow/orchestration`. The manifests read `0.
|
|
20
|
+
Published on npm as `fyflow-orchestration` and on JSR as `@fyflow/orchestration`. The manifests read `0.14.0`.
|
|
21
21
|
|
|
22
22
|
## Install
|
|
23
23
|
|
|
@@ -1684,11 +1684,38 @@ var GateTypeRefused = class extends RegistryError {
|
|
|
1684
1684
|
var BUILT_IN_GROUP_TYPES = {
|
|
1685
1685
|
ConcurrentLimitGroup: { configSchema: { type: "object", required: ["limit"], properties: { limit: { type: "integer", minimum: 1 } }, additionalProperties: false } },
|
|
1686
1686
|
RateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } },
|
|
1687
|
-
KeyedRateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } }
|
|
1687
|
+
KeyedRateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } },
|
|
1688
|
+
// v0.56: `units` a count (the units `0`…`n-1`) or a list of names — `type` may be a list in the subset (P46 fact 7), and
|
|
1689
|
+
// `minimum` reads a number and `minItems` / `items` an array, so each keyword holds its own form. Distinct: beside it.
|
|
1690
|
+
UnitGroup: { configSchema: { type: "object", required: ["units"], properties: { units: { type: ["integer", "array"], minimum: 1, minItems: 1, items: { type: "string", minLength: 1 } } }, additionalProperties: false } },
|
|
1691
|
+
// v0.56: `limits` an object of per-key limits; each value a positive integer is beside it (no schema-valued
|
|
1692
|
+
// `additionalProperties` in the subset).
|
|
1693
|
+
KeyedConcurrentLimitGroup: { configSchema: { type: "object", required: ["limit"], properties: { limit: { type: "integer", minimum: 1 }, limits: { type: "object" } }, additionalProperties: false } }
|
|
1688
1694
|
};
|
|
1689
1695
|
function RATE_WINDOWS() {
|
|
1690
1696
|
return { type: "array", minItems: 1, items: { type: "object", required: ["limit", "windowMs"], properties: { limit: { type: "integer", minimum: 1 }, windowMs: { type: "integer", minimum: 1 } }, additionalProperties: false } };
|
|
1691
1697
|
}
|
|
1698
|
+
var isPositiveInteger = (v) => typeof v === "number" && Number.isInteger(v) && v >= 1;
|
|
1699
|
+
function builtInConfigProblem(type, config) {
|
|
1700
|
+
if (type === "UnitGroup") {
|
|
1701
|
+
const units = config.units;
|
|
1702
|
+
if (Array.isArray(units)) {
|
|
1703
|
+
const seen = /* @__PURE__ */ new Set();
|
|
1704
|
+
for (const u of units) {
|
|
1705
|
+
if (seen.has(u)) return `$.units lists unit ${JSON.stringify(u)} twice \u2014 a UnitGroup's units are distinct`;
|
|
1706
|
+
seen.add(u);
|
|
1707
|
+
}
|
|
1708
|
+
}
|
|
1709
|
+
} else if (type === "KeyedConcurrentLimitGroup") {
|
|
1710
|
+
const limits = config.limits;
|
|
1711
|
+
if (limits !== void 0) {
|
|
1712
|
+
for (const [k, v] of Object.entries(limits)) {
|
|
1713
|
+
if (!isPositiveInteger(v)) return `$.limits[${q(k)}] must be a positive integer \u2014 a per-key limit, as \`limit\` is \u2014 got ${shown(v)}`;
|
|
1714
|
+
}
|
|
1715
|
+
}
|
|
1716
|
+
}
|
|
1717
|
+
return null;
|
|
1718
|
+
}
|
|
1692
1719
|
var DOCUMENT_KEYS = ["workers", "groups", "nodeTypes", "pools", "predicates", "languages"];
|
|
1693
1720
|
var WORKER_KEYS = ["url", "configSchema", "settingsSchema", "inputSchema", "outputSchema"];
|
|
1694
1721
|
var CHECKED_WORKER_SCHEMAS = ["configSchema", "settingsSchema"];
|
|
@@ -1728,9 +1755,12 @@ function mergePool(base, over) {
|
|
|
1728
1755
|
}
|
|
1729
1756
|
function copyResident(r) {
|
|
1730
1757
|
if (Array.isArray(r)) return [...r];
|
|
1731
|
-
if (isPlainObject2(r)) return
|
|
1758
|
+
if (isPlainObject2(r)) return Object.fromEntries(Object.entries(r).map(([k, v]) => [k, isPlainObject2(v) ? copyRequest(v) : v]));
|
|
1732
1759
|
return r;
|
|
1733
1760
|
}
|
|
1761
|
+
function copyRequest(v) {
|
|
1762
|
+
return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, Array.isArray(x) ? [...x] : x]));
|
|
1763
|
+
}
|
|
1734
1764
|
function nodeTypeFromDeclaration(decl, over) {
|
|
1735
1765
|
const declaredPool = mergePool(decl.pool ?? {}, over);
|
|
1736
1766
|
const def = {
|
|
@@ -1775,7 +1805,10 @@ function poolSpecProblem(v) {
|
|
|
1775
1805
|
if (r.some((g) => typeof g !== "string" || g.length === 0)) return `PoolSpec.residentGroups must be an array of group keys or a { key: cost } record, got ${JSON.stringify(r)}`;
|
|
1776
1806
|
if (new Set(r).size !== r.length) return "PoolSpec.residentGroups repeats a key";
|
|
1777
1807
|
} else if (isPlainObject2(r)) {
|
|
1778
|
-
for (const [k, c] of Object.entries(r))
|
|
1808
|
+
for (const [k, c] of Object.entries(r)) {
|
|
1809
|
+
const p = residentValueProblem(c);
|
|
1810
|
+
if (p !== null) return `PoolSpec.residentGroups[${q(k)}] ${p}`;
|
|
1811
|
+
}
|
|
1779
1812
|
} else return `PoolSpec.residentGroups must be an array of group keys or a { key: cost } record, got ${JSON.stringify(r)}`;
|
|
1780
1813
|
}
|
|
1781
1814
|
if (v.cancelGraceMs !== void 0 && !(typeof v.cancelGraceMs === "number" && v.cancelGraceMs >= 0)) {
|
|
@@ -1796,6 +1829,27 @@ function poolSpecProblem(v) {
|
|
|
1796
1829
|
}
|
|
1797
1830
|
return null;
|
|
1798
1831
|
}
|
|
1832
|
+
var RESIDENT_REQUEST_KEYS = ["cost", "from", "prefer"];
|
|
1833
|
+
function residentValueProblem(c) {
|
|
1834
|
+
if (typeof c === "number") return isPositiveInteger(c) ? null : `must be a cost > 0, an INTEGER (the scheduler's rule), or a request { ${RESIDENT_REQUEST_KEYS.map((k) => `${k}?`).join(", ")} }, got ${shown(c)}`;
|
|
1835
|
+
if (!isPlainObject2(c) || !isPlainPrototype(c)) {
|
|
1836
|
+
return `must be a cost (a positive integer) or a request { ${RESIDENT_REQUEST_KEYS.map((k) => `${k}?`).join(", ")} } \u2014 a plain object \u2014 got ${describeNonPlain(c)}`;
|
|
1837
|
+
}
|
|
1838
|
+
for (const k of Object.keys(c)) if (!RESIDENT_REQUEST_KEYS.includes(k)) return `is a request carrying undeclared key ${q(k)}; it declares ${RESIDENT_REQUEST_KEYS.join(", ")}`;
|
|
1839
|
+
if (ownKey(c, "cost") && !isPositiveInteger(c.cost)) return `.cost must be a positive integer, got ${shown(c.cost)}`;
|
|
1840
|
+
for (const f of ["from", "prefer"]) {
|
|
1841
|
+
if (!ownKey(c, f)) continue;
|
|
1842
|
+
const list = c[f];
|
|
1843
|
+
if (!Array.isArray(list) || list.length === 0) return `.${f} must be a non-empty list of unit ids, got ${shown(list)}`;
|
|
1844
|
+
for (const u of list) {
|
|
1845
|
+
if (!(typeof u === "string" && u.length > 0 || typeof u === "number" && Number.isInteger(u) && u >= 0)) {
|
|
1846
|
+
return `.${f} holds ${shown(u)} \u2014 a unit id is a non-empty string, or an integer >= 0 for a UnitGroup declared by count`;
|
|
1847
|
+
}
|
|
1848
|
+
}
|
|
1849
|
+
if (new Set(list).size !== list.length) return `.${f} repeats a unit id`;
|
|
1850
|
+
}
|
|
1851
|
+
return null;
|
|
1852
|
+
}
|
|
1799
1853
|
function isPlainPrototype(o) {
|
|
1800
1854
|
const proto = Object.getPrototypeOf(o);
|
|
1801
1855
|
return proto === Object.prototype || proto === null;
|
|
@@ -1826,6 +1880,45 @@ function groupsNamedBy(spec) {
|
|
|
1826
1880
|
function residentKeysOf(spec) {
|
|
1827
1881
|
return spec.residentGroups === void 0 ? [] : Array.isArray(spec.residentGroups) ? [...spec.residentGroups] : Object.keys(spec.residentGroups);
|
|
1828
1882
|
}
|
|
1883
|
+
function residentEntriesOf(spec) {
|
|
1884
|
+
const r = spec.residentGroups;
|
|
1885
|
+
return r === void 0 ? [] : Array.isArray(r) ? r.map((g) => [g, 1]) : Object.entries(r);
|
|
1886
|
+
}
|
|
1887
|
+
var KEYED_BUILT_INS = {
|
|
1888
|
+
KeyedRateLimitGroup: 'the scheduler resolves no key for a resident group, so every worker holding it is counted in ONE "" bucket, in silence',
|
|
1889
|
+
KeyedConcurrentLimitGroup: "the scheduler REJECTS a KeyedConcurrentLimitGroup in residentGroups when it is constructed: a worker holds its units across tasks, and so across keys, with no key to hold them under"
|
|
1890
|
+
};
|
|
1891
|
+
function residentAgainstGroupProblem(group, value) {
|
|
1892
|
+
if (!hasOwn(BUILT_IN_GROUP_TYPES, group.type)) return null;
|
|
1893
|
+
const request = isPlainObject2(value) ? value : void 0;
|
|
1894
|
+
const cost = typeof value === "number" ? value : request?.cost ?? 1;
|
|
1895
|
+
const named = `group ${q(group.key)}, of the built-in type ${q(group.type)}`;
|
|
1896
|
+
if (group.type === "UnitGroup") {
|
|
1897
|
+
const declared = group.config.units;
|
|
1898
|
+
const units = typeof declared === "number" ? Array.from({ length: declared }, (_, i) => i) : [...declared];
|
|
1899
|
+
const has = new Set(units);
|
|
1900
|
+
for (const f of ["from", "prefer"]) {
|
|
1901
|
+
for (const u of request?.[f] ?? []) {
|
|
1902
|
+
if (!has.has(u)) {
|
|
1903
|
+
return `${f} names unit ${shown(u)}, which ${named} does not have \u2014 its units are ${units.map((x) => shown(x)).join(", ")}` + (typeof declared === "number" ? " (declared by count: integers, never strings)" : "");
|
|
1904
|
+
}
|
|
1905
|
+
}
|
|
1906
|
+
}
|
|
1907
|
+
if (request?.from !== void 0 && cost > request.from.length) {
|
|
1908
|
+
return `a cost of ${cost} from ${request.from.length} candidate unit(s) in from can never be granted by ${named}`;
|
|
1909
|
+
}
|
|
1910
|
+
if (cost > units.length) return `a cost of ${cost} is above the ${units.length} unit(s) of ${named} \u2014 a worker of this pool could never start`;
|
|
1911
|
+
return null;
|
|
1912
|
+
}
|
|
1913
|
+
if (request !== void 0 && (ownKey(request, "from") || ownKey(request, "prefer"))) {
|
|
1914
|
+
return `a request naming ${["from", "prefer"].filter((f) => ownKey(request, f)).join(" and ")} on ${named}, which takes no request \u2014 from and prefer are a UnitGroup's (or a code group type's with a checkRequest); { cost } alone is a cost, for any group`;
|
|
1915
|
+
}
|
|
1916
|
+
const limit = group.type === "ConcurrentLimitGroup" ? group.config.limit : group.type === "RateLimitGroup" ? Math.min(...group.config.windows.map((w) => w.limit)) : void 0;
|
|
1917
|
+
if (limit !== void 0 && cost > limit) {
|
|
1918
|
+
return `a cost of ${cost} is above the limit of ${limit} of ${named}${group.type === "RateLimitGroup" ? " (its smallest window's)" : ""} \u2014 a worker of this pool could never start`;
|
|
1919
|
+
}
|
|
1920
|
+
return null;
|
|
1921
|
+
}
|
|
1829
1922
|
function poolOptionsOf(spec, inline, workerUrl, languages) {
|
|
1830
1923
|
const { config, groups, ...rest } = spec;
|
|
1831
1924
|
return {
|
|
@@ -1913,7 +2006,7 @@ function derivePool(key, typeName, def, worker, config, groupKeys, refuse, pool)
|
|
|
1913
2006
|
}
|
|
1914
2007
|
const resident = spec.residentGroups === void 0 ? 0 : Array.isArray(spec.residentGroups) ? spec.residentGroups.length : Object.keys(spec.residentGroups).length;
|
|
1915
2008
|
if (resident > 0 && spec.idleTimeout === 0) {
|
|
1916
|
-
throw new DerivationRefused(key, `${who}: residentGroups beside idleTimeout: 0 \u2014 a worker that never idles never releases a resident group, and a second pool contending for it waits for ever (measured
|
|
2009
|
+
throw new DerivationRefused(key, `${who}: residentGroups beside idleTimeout: 0 \u2014 a worker that never idles never releases a resident group, and a second pool contending for it waits for ever (measured at P18 fact 6 and re-measured at P46 fact 6); give the pool a positive idleTimeout`);
|
|
1917
2010
|
}
|
|
1918
2011
|
return { key, type: typeName, definition: def, worker, config, spec, ...pool !== void 0 ? { pool: copyPool(pool) } : {} };
|
|
1919
2012
|
}
|
|
@@ -2024,6 +2117,10 @@ function resolveRegistry(doc, code = {}) {
|
|
|
2024
2117
|
const fs = findings(gt.configSchema, inst.config ?? {});
|
|
2025
2118
|
if (fs.length) throw new GroupConfigRefused(key, `group ${q(key)}: config fails ${q(inst.type)}'s configSchema \u2014 ${describeFindings(fs)}`);
|
|
2026
2119
|
}
|
|
2120
|
+
if (hasOwn(BUILT_IN_GROUP_TYPES, inst.type)) {
|
|
2121
|
+
const p = builtInConfigProblem(inst.type, inst.config);
|
|
2122
|
+
if (p !== null) throw new GroupConfigRefused(key, `group ${q(key)}: config fails ${q(inst.type)}'s rule beside its configSchema \u2014 ${p}`);
|
|
2123
|
+
}
|
|
2027
2124
|
groups.set(key, { key, type: inst.type, groupType: gt, config: inst.config });
|
|
2028
2125
|
}
|
|
2029
2126
|
const groupKeys = new Set(groups.keys());
|
|
@@ -2062,10 +2159,16 @@ function resolveRegistry(doc, code = {}) {
|
|
|
2062
2159
|
pools.set(key, derivePool(key, inst.type, def, def.worker === void 0 ? void 0 : workers.get(def.worker), inst.config, groupKeys, ConfigRefused, inst.pool));
|
|
2063
2160
|
}
|
|
2064
2161
|
for (const p of pools.values()) {
|
|
2065
|
-
|
|
2066
|
-
|
|
2067
|
-
const
|
|
2068
|
-
|
|
2162
|
+
const who = p.key === p.type ? `node type ${q(p.type)}'s default instance` : `pool ${q(p.key)} (an instance of ${q(p.type)})`;
|
|
2163
|
+
for (const [g, value] of residentEntriesOf(p.spec)) {
|
|
2164
|
+
const group = groups.get(g);
|
|
2165
|
+
if (group === void 0) continue;
|
|
2166
|
+
const keyed = KEYED_BUILT_INS[group.type];
|
|
2167
|
+
if (hasOwn(KEYED_BUILT_INS, group.type) && keyed !== void 0) {
|
|
2168
|
+
throw new DerivationRefused(p.key, `${who}: the derived pool holds group ${q(g)}, of the built-in type ${q(group.type)}, in residentGroups \u2014 ${keyed} (Q619). Name ${q(g)} in groups, where each task is counted under its limitKey (FD-R6), or hold a ConcurrentLimitGroup or RateLimitGroup resident (or a UnitGroup)`);
|
|
2169
|
+
}
|
|
2170
|
+
const problem = residentAgainstGroupProblem(group, value);
|
|
2171
|
+
if (problem !== null) throw new DerivationRefused(p.key, `${who}: residentGroups[${q(g)}] \u2014 ${problem}`);
|
|
2069
2172
|
}
|
|
2070
2173
|
}
|
|
2071
2174
|
const predicateNames = doc.predicates ?? [];
|
package/dist/browser/index.js
CHANGED
|
@@ -152,7 +152,7 @@ import {
|
|
|
152
152
|
visibilityKey,
|
|
153
153
|
witness,
|
|
154
154
|
workersOf
|
|
155
|
-
} from "./chunks/chunk-
|
|
155
|
+
} from "./chunks/chunk-5KUYTCXI.js";
|
|
156
156
|
|
|
157
157
|
// src/definition/definition-schema.ts
|
|
158
158
|
var DRAFT_2020_12 = "https://json-schema.org/draft/2020-12/schema";
|
|
@@ -26,7 +26,7 @@ import {
|
|
|
26
26
|
retryPolicyProblem,
|
|
27
27
|
validateDefinition,
|
|
28
28
|
workersOf
|
|
29
|
-
} from "../chunks/chunk-
|
|
29
|
+
} from "../chunks/chunk-5KUYTCXI.js";
|
|
30
30
|
|
|
31
31
|
// src/scheduler/bootstrap.ts
|
|
32
32
|
import { WorkerManager } from "fyflow-scheduler";
|
|
@@ -83,7 +83,7 @@ function bootstrap(registry, groups = {}, options = {}) {
|
|
|
83
83
|
}
|
|
84
84
|
|
|
85
85
|
// src/scheduler/runtime.ts
|
|
86
|
-
import { ConcurrentLimitGroup, FyflowScheduler, KeyedRateLimitGroup, RateLimitGroup } from "fyflow-scheduler";
|
|
86
|
+
import { ConcurrentLimitGroup, FyflowScheduler, KeyedConcurrentLimitGroup, KeyedRateLimitGroup, RateLimitGroup, UnitGroup } from "fyflow-scheduler";
|
|
87
87
|
var RuntimeError = class extends Error {
|
|
88
88
|
/** The entry this refusal is about — a worker's key, a group's key, a type's name. `RegistryError`'s field, one layer out. */
|
|
89
89
|
at;
|
|
@@ -107,8 +107,18 @@ var KEYED_RULE = "FD-R6";
|
|
|
107
107
|
var BUILT_IN_GROUP_FACTORIES = {
|
|
108
108
|
ConcurrentLimitGroup: (config, key) => new ConcurrentLimitGroup(config.limit, key),
|
|
109
109
|
RateLimitGroup: (config, key) => new RateLimitGroup(config.windows, key),
|
|
110
|
-
KeyedRateLimitGroup: (config, key) => new KeyedRateLimitGroup(config.windows, { id: key })
|
|
110
|
+
KeyedRateLimitGroup: (config, key) => new KeyedRateLimitGroup(config.windows, { id: key }),
|
|
111
|
+
UnitGroup: (config, key) => {
|
|
112
|
+
const units = config.units;
|
|
113
|
+
return new UnitGroup(typeof units === "number" ? units : [...units], { id: key });
|
|
114
|
+
},
|
|
115
|
+
KeyedConcurrentLimitGroup: (config, key) => {
|
|
116
|
+
const { limit, limits } = config;
|
|
117
|
+
const own = new Map(Object.entries(limits ?? {}));
|
|
118
|
+
return new KeyedConcurrentLimitGroup(limit, { id: key, limitFor: (k) => own.get(k) ?? limit });
|
|
119
|
+
}
|
|
111
120
|
};
|
|
121
|
+
var KEYED_BUILT_IN_GROUP_TYPES = ["KeyedRateLimitGroup", "KeyedConcurrentLimitGroup"];
|
|
112
122
|
var GROUP_MEMBERS = [
|
|
113
123
|
{ name: "canRun", kind: "function", why: "the scheduler calls it before every admission; a group without it throws on the first task" },
|
|
114
124
|
{ name: "onStart", kind: "function", why: "the scheduler calls it unguarded at admission, though the interface marks it optional; a group without it throws on the first task (measured on 0.5.2)" },
|
|
@@ -210,8 +220,8 @@ function createRuntime(document, code = {}, options = {}) {
|
|
|
210
220
|
pools: poolsOf(registry.pools),
|
|
211
221
|
...registry.gateTypes !== void 0 ? { gateTypes: registry.gateTypes } : {}
|
|
212
222
|
};
|
|
213
|
-
const keyedByBuiltIn = /* @__PURE__ */ new
|
|
214
|
-
for (const [key, g] of resolved.groups) if (g.type
|
|
223
|
+
const keyedByBuiltIn = /* @__PURE__ */ new Map();
|
|
224
|
+
for (const [key, g] of resolved.groups) if (KEYED_BUILT_IN_GROUP_TYPES.includes(g.type)) keyedByBuiltIn.set(key, g.type);
|
|
215
225
|
const keyedFindings = (flow) => {
|
|
216
226
|
const nodes = flow?.nodes;
|
|
217
227
|
if (keyedByBuiltIn.size === 0 || typeof nodes?.[Symbol.iterator] !== "function") return [];
|
|
@@ -226,10 +236,11 @@ function createRuntime(document, code = {}, options = {}) {
|
|
|
226
236
|
const reached = [.../* @__PURE__ */ new Set([...fromPool, ...fromNode])].filter((g) => keyedByBuiltIn.has(g)).sort();
|
|
227
237
|
if (reached.length === 0) continue;
|
|
228
238
|
const at = typeof n.id === "string" ? n.id : null;
|
|
239
|
+
const types = [...new Set(reached.map((g) => keyedByBuiltIn.get(g)))].sort().join(" and ");
|
|
229
240
|
out.push({
|
|
230
241
|
rule: KEYED_RULE,
|
|
231
242
|
at,
|
|
232
|
-
detail: `node ${at === null ? "<unnamed>" : q(at)} reaches keyed group${reached.length > 1 ? "s" : ""} ${reached.map(q).join(", ")}, built by the built-in
|
|
243
|
+
detail: `node ${at === null ? "<unnamed>" : q(at)} reaches keyed group${reached.length > 1 ? "s" : ""} ${reached.map(q).join(", ")}, built by the built-in ${types} factory with no keyFrom \u2014 so every task of it needs a limitKey, which nothing sets unless start is given a createTask, and without one the scheduler's addTask throws "Missing limit key". Two roads work: pass start a createTask that sets limitKey on the task it builds, or declare the group with a code group type (code.groupTypes) whose create() builds a ${types} with a keyFrom that derives the key. A key derived from the flow itself is Q248, deferred`
|
|
233
244
|
});
|
|
234
245
|
}
|
|
235
246
|
return out;
|
|
@@ -1684,11 +1684,38 @@ var GateTypeRefused = class extends RegistryError {
|
|
|
1684
1684
|
var BUILT_IN_GROUP_TYPES = {
|
|
1685
1685
|
ConcurrentLimitGroup: { configSchema: { type: "object", required: ["limit"], properties: { limit: { type: "integer", minimum: 1 } }, additionalProperties: false } },
|
|
1686
1686
|
RateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } },
|
|
1687
|
-
KeyedRateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } }
|
|
1687
|
+
KeyedRateLimitGroup: { configSchema: { type: "object", required: ["windows"], properties: { windows: RATE_WINDOWS() }, additionalProperties: false } },
|
|
1688
|
+
// v0.56: `units` a count (the units `0`…`n-1`) or a list of names — `type` may be a list in the subset (P46 fact 7), and
|
|
1689
|
+
// `minimum` reads a number and `minItems` / `items` an array, so each keyword holds its own form. Distinct: beside it.
|
|
1690
|
+
UnitGroup: { configSchema: { type: "object", required: ["units"], properties: { units: { type: ["integer", "array"], minimum: 1, minItems: 1, items: { type: "string", minLength: 1 } } }, additionalProperties: false } },
|
|
1691
|
+
// v0.56: `limits` an object of per-key limits; each value a positive integer is beside it (no schema-valued
|
|
1692
|
+
// `additionalProperties` in the subset).
|
|
1693
|
+
KeyedConcurrentLimitGroup: { configSchema: { type: "object", required: ["limit"], properties: { limit: { type: "integer", minimum: 1 }, limits: { type: "object" } }, additionalProperties: false } }
|
|
1688
1694
|
};
|
|
1689
1695
|
function RATE_WINDOWS() {
|
|
1690
1696
|
return { type: "array", minItems: 1, items: { type: "object", required: ["limit", "windowMs"], properties: { limit: { type: "integer", minimum: 1 }, windowMs: { type: "integer", minimum: 1 } }, additionalProperties: false } };
|
|
1691
1697
|
}
|
|
1698
|
+
var isPositiveInteger = (v) => typeof v === "number" && Number.isInteger(v) && v >= 1;
|
|
1699
|
+
function builtInConfigProblem(type, config) {
|
|
1700
|
+
if (type === "UnitGroup") {
|
|
1701
|
+
const units = config.units;
|
|
1702
|
+
if (Array.isArray(units)) {
|
|
1703
|
+
const seen = /* @__PURE__ */ new Set();
|
|
1704
|
+
for (const u of units) {
|
|
1705
|
+
if (seen.has(u)) return `$.units lists unit ${JSON.stringify(u)} twice \u2014 a UnitGroup's units are distinct`;
|
|
1706
|
+
seen.add(u);
|
|
1707
|
+
}
|
|
1708
|
+
}
|
|
1709
|
+
} else if (type === "KeyedConcurrentLimitGroup") {
|
|
1710
|
+
const limits = config.limits;
|
|
1711
|
+
if (limits !== void 0) {
|
|
1712
|
+
for (const [k, v] of Object.entries(limits)) {
|
|
1713
|
+
if (!isPositiveInteger(v)) return `$.limits[${q(k)}] must be a positive integer \u2014 a per-key limit, as \`limit\` is \u2014 got ${shown(v)}`;
|
|
1714
|
+
}
|
|
1715
|
+
}
|
|
1716
|
+
}
|
|
1717
|
+
return null;
|
|
1718
|
+
}
|
|
1692
1719
|
var DOCUMENT_KEYS = ["workers", "groups", "nodeTypes", "pools", "predicates", "languages"];
|
|
1693
1720
|
var WORKER_KEYS = ["url", "configSchema", "settingsSchema", "inputSchema", "outputSchema"];
|
|
1694
1721
|
var CHECKED_WORKER_SCHEMAS = ["configSchema", "settingsSchema"];
|
|
@@ -1728,9 +1755,12 @@ function mergePool(base, over) {
|
|
|
1728
1755
|
}
|
|
1729
1756
|
function copyResident(r) {
|
|
1730
1757
|
if (Array.isArray(r)) return [...r];
|
|
1731
|
-
if (isPlainObject2(r)) return
|
|
1758
|
+
if (isPlainObject2(r)) return Object.fromEntries(Object.entries(r).map(([k, v]) => [k, isPlainObject2(v) ? copyRequest(v) : v]));
|
|
1732
1759
|
return r;
|
|
1733
1760
|
}
|
|
1761
|
+
function copyRequest(v) {
|
|
1762
|
+
return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, Array.isArray(x) ? [...x] : x]));
|
|
1763
|
+
}
|
|
1734
1764
|
function nodeTypeFromDeclaration(decl, over) {
|
|
1735
1765
|
const declaredPool = mergePool(decl.pool ?? {}, over);
|
|
1736
1766
|
const def = {
|
|
@@ -1775,7 +1805,10 @@ function poolSpecProblem(v) {
|
|
|
1775
1805
|
if (r.some((g) => typeof g !== "string" || g.length === 0)) return `PoolSpec.residentGroups must be an array of group keys or a { key: cost } record, got ${JSON.stringify(r)}`;
|
|
1776
1806
|
if (new Set(r).size !== r.length) return "PoolSpec.residentGroups repeats a key";
|
|
1777
1807
|
} else if (isPlainObject2(r)) {
|
|
1778
|
-
for (const [k, c] of Object.entries(r))
|
|
1808
|
+
for (const [k, c] of Object.entries(r)) {
|
|
1809
|
+
const p = residentValueProblem(c);
|
|
1810
|
+
if (p !== null) return `PoolSpec.residentGroups[${q(k)}] ${p}`;
|
|
1811
|
+
}
|
|
1779
1812
|
} else return `PoolSpec.residentGroups must be an array of group keys or a { key: cost } record, got ${JSON.stringify(r)}`;
|
|
1780
1813
|
}
|
|
1781
1814
|
if (v.cancelGraceMs !== void 0 && !(typeof v.cancelGraceMs === "number" && v.cancelGraceMs >= 0)) {
|
|
@@ -1796,6 +1829,27 @@ function poolSpecProblem(v) {
|
|
|
1796
1829
|
}
|
|
1797
1830
|
return null;
|
|
1798
1831
|
}
|
|
1832
|
+
var RESIDENT_REQUEST_KEYS = ["cost", "from", "prefer"];
|
|
1833
|
+
function residentValueProblem(c) {
|
|
1834
|
+
if (typeof c === "number") return isPositiveInteger(c) ? null : `must be a cost > 0, an INTEGER (the scheduler's rule), or a request { ${RESIDENT_REQUEST_KEYS.map((k) => `${k}?`).join(", ")} }, got ${shown(c)}`;
|
|
1835
|
+
if (!isPlainObject2(c) || !isPlainPrototype(c)) {
|
|
1836
|
+
return `must be a cost (a positive integer) or a request { ${RESIDENT_REQUEST_KEYS.map((k) => `${k}?`).join(", ")} } \u2014 a plain object \u2014 got ${describeNonPlain(c)}`;
|
|
1837
|
+
}
|
|
1838
|
+
for (const k of Object.keys(c)) if (!RESIDENT_REQUEST_KEYS.includes(k)) return `is a request carrying undeclared key ${q(k)}; it declares ${RESIDENT_REQUEST_KEYS.join(", ")}`;
|
|
1839
|
+
if (ownKey(c, "cost") && !isPositiveInteger(c.cost)) return `.cost must be a positive integer, got ${shown(c.cost)}`;
|
|
1840
|
+
for (const f of ["from", "prefer"]) {
|
|
1841
|
+
if (!ownKey(c, f)) continue;
|
|
1842
|
+
const list = c[f];
|
|
1843
|
+
if (!Array.isArray(list) || list.length === 0) return `.${f} must be a non-empty list of unit ids, got ${shown(list)}`;
|
|
1844
|
+
for (const u of list) {
|
|
1845
|
+
if (!(typeof u === "string" && u.length > 0 || typeof u === "number" && Number.isInteger(u) && u >= 0)) {
|
|
1846
|
+
return `.${f} holds ${shown(u)} \u2014 a unit id is a non-empty string, or an integer >= 0 for a UnitGroup declared by count`;
|
|
1847
|
+
}
|
|
1848
|
+
}
|
|
1849
|
+
if (new Set(list).size !== list.length) return `.${f} repeats a unit id`;
|
|
1850
|
+
}
|
|
1851
|
+
return null;
|
|
1852
|
+
}
|
|
1799
1853
|
function isPlainPrototype(o) {
|
|
1800
1854
|
const proto = Object.getPrototypeOf(o);
|
|
1801
1855
|
return proto === Object.prototype || proto === null;
|
|
@@ -1826,6 +1880,45 @@ function groupsNamedBy(spec) {
|
|
|
1826
1880
|
function residentKeysOf(spec) {
|
|
1827
1881
|
return spec.residentGroups === void 0 ? [] : Array.isArray(spec.residentGroups) ? [...spec.residentGroups] : Object.keys(spec.residentGroups);
|
|
1828
1882
|
}
|
|
1883
|
+
function residentEntriesOf(spec) {
|
|
1884
|
+
const r = spec.residentGroups;
|
|
1885
|
+
return r === void 0 ? [] : Array.isArray(r) ? r.map((g) => [g, 1]) : Object.entries(r);
|
|
1886
|
+
}
|
|
1887
|
+
var KEYED_BUILT_INS = {
|
|
1888
|
+
KeyedRateLimitGroup: 'the scheduler resolves no key for a resident group, so every worker holding it is counted in ONE "" bucket, in silence',
|
|
1889
|
+
KeyedConcurrentLimitGroup: "the scheduler REJECTS a KeyedConcurrentLimitGroup in residentGroups when it is constructed: a worker holds its units across tasks, and so across keys, with no key to hold them under"
|
|
1890
|
+
};
|
|
1891
|
+
function residentAgainstGroupProblem(group, value) {
|
|
1892
|
+
if (!hasOwn(BUILT_IN_GROUP_TYPES, group.type)) return null;
|
|
1893
|
+
const request = isPlainObject2(value) ? value : void 0;
|
|
1894
|
+
const cost = typeof value === "number" ? value : request?.cost ?? 1;
|
|
1895
|
+
const named = `group ${q(group.key)}, of the built-in type ${q(group.type)}`;
|
|
1896
|
+
if (group.type === "UnitGroup") {
|
|
1897
|
+
const declared = group.config.units;
|
|
1898
|
+
const units = typeof declared === "number" ? Array.from({ length: declared }, (_, i) => i) : [...declared];
|
|
1899
|
+
const has = new Set(units);
|
|
1900
|
+
for (const f of ["from", "prefer"]) {
|
|
1901
|
+
for (const u of request?.[f] ?? []) {
|
|
1902
|
+
if (!has.has(u)) {
|
|
1903
|
+
return `${f} names unit ${shown(u)}, which ${named} does not have \u2014 its units are ${units.map((x) => shown(x)).join(", ")}` + (typeof declared === "number" ? " (declared by count: integers, never strings)" : "");
|
|
1904
|
+
}
|
|
1905
|
+
}
|
|
1906
|
+
}
|
|
1907
|
+
if (request?.from !== void 0 && cost > request.from.length) {
|
|
1908
|
+
return `a cost of ${cost} from ${request.from.length} candidate unit(s) in from can never be granted by ${named}`;
|
|
1909
|
+
}
|
|
1910
|
+
if (cost > units.length) return `a cost of ${cost} is above the ${units.length} unit(s) of ${named} \u2014 a worker of this pool could never start`;
|
|
1911
|
+
return null;
|
|
1912
|
+
}
|
|
1913
|
+
if (request !== void 0 && (ownKey(request, "from") || ownKey(request, "prefer"))) {
|
|
1914
|
+
return `a request naming ${["from", "prefer"].filter((f) => ownKey(request, f)).join(" and ")} on ${named}, which takes no request \u2014 from and prefer are a UnitGroup's (or a code group type's with a checkRequest); { cost } alone is a cost, for any group`;
|
|
1915
|
+
}
|
|
1916
|
+
const limit = group.type === "ConcurrentLimitGroup" ? group.config.limit : group.type === "RateLimitGroup" ? Math.min(...group.config.windows.map((w) => w.limit)) : void 0;
|
|
1917
|
+
if (limit !== void 0 && cost > limit) {
|
|
1918
|
+
return `a cost of ${cost} is above the limit of ${limit} of ${named}${group.type === "RateLimitGroup" ? " (its smallest window's)" : ""} \u2014 a worker of this pool could never start`;
|
|
1919
|
+
}
|
|
1920
|
+
return null;
|
|
1921
|
+
}
|
|
1829
1922
|
function poolOptionsOf(spec, inline, workerUrl, languages) {
|
|
1830
1923
|
const { config, groups, ...rest } = spec;
|
|
1831
1924
|
return {
|
|
@@ -1913,7 +2006,7 @@ function derivePool(key, typeName, def, worker, config, groupKeys, refuse, pool)
|
|
|
1913
2006
|
}
|
|
1914
2007
|
const resident = spec.residentGroups === void 0 ? 0 : Array.isArray(spec.residentGroups) ? spec.residentGroups.length : Object.keys(spec.residentGroups).length;
|
|
1915
2008
|
if (resident > 0 && spec.idleTimeout === 0) {
|
|
1916
|
-
throw new DerivationRefused(key, `${who}: residentGroups beside idleTimeout: 0 \u2014 a worker that never idles never releases a resident group, and a second pool contending for it waits for ever (measured
|
|
2009
|
+
throw new DerivationRefused(key, `${who}: residentGroups beside idleTimeout: 0 \u2014 a worker that never idles never releases a resident group, and a second pool contending for it waits for ever (measured at P18 fact 6 and re-measured at P46 fact 6); give the pool a positive idleTimeout`);
|
|
1917
2010
|
}
|
|
1918
2011
|
return { key, type: typeName, definition: def, worker, config, spec, ...pool !== void 0 ? { pool: copyPool(pool) } : {} };
|
|
1919
2012
|
}
|
|
@@ -2024,6 +2117,10 @@ function resolveRegistry(doc, code = {}) {
|
|
|
2024
2117
|
const fs = findings(gt.configSchema, inst.config ?? {});
|
|
2025
2118
|
if (fs.length) throw new GroupConfigRefused(key, `group ${q(key)}: config fails ${q(inst.type)}'s configSchema \u2014 ${describeFindings(fs)}`);
|
|
2026
2119
|
}
|
|
2120
|
+
if (hasOwn(BUILT_IN_GROUP_TYPES, inst.type)) {
|
|
2121
|
+
const p = builtInConfigProblem(inst.type, inst.config);
|
|
2122
|
+
if (p !== null) throw new GroupConfigRefused(key, `group ${q(key)}: config fails ${q(inst.type)}'s rule beside its configSchema \u2014 ${p}`);
|
|
2123
|
+
}
|
|
2027
2124
|
groups.set(key, { key, type: inst.type, groupType: gt, config: inst.config });
|
|
2028
2125
|
}
|
|
2029
2126
|
const groupKeys = new Set(groups.keys());
|
|
@@ -2062,10 +2159,16 @@ function resolveRegistry(doc, code = {}) {
|
|
|
2062
2159
|
pools.set(key, derivePool(key, inst.type, def, def.worker === void 0 ? void 0 : workers.get(def.worker), inst.config, groupKeys, ConfigRefused, inst.pool));
|
|
2063
2160
|
}
|
|
2064
2161
|
for (const p of pools.values()) {
|
|
2065
|
-
|
|
2066
|
-
|
|
2067
|
-
const
|
|
2068
|
-
|
|
2162
|
+
const who = p.key === p.type ? `node type ${q(p.type)}'s default instance` : `pool ${q(p.key)} (an instance of ${q(p.type)})`;
|
|
2163
|
+
for (const [g, value] of residentEntriesOf(p.spec)) {
|
|
2164
|
+
const group = groups.get(g);
|
|
2165
|
+
if (group === void 0) continue;
|
|
2166
|
+
const keyed = KEYED_BUILT_INS[group.type];
|
|
2167
|
+
if (hasOwn(KEYED_BUILT_INS, group.type) && keyed !== void 0) {
|
|
2168
|
+
throw new DerivationRefused(p.key, `${who}: the derived pool holds group ${q(g)}, of the built-in type ${q(group.type)}, in residentGroups \u2014 ${keyed} (Q619). Name ${q(g)} in groups, where each task is counted under its limitKey (FD-R6), or hold a ConcurrentLimitGroup or RateLimitGroup resident (or a UnitGroup)`);
|
|
2169
|
+
}
|
|
2170
|
+
const problem = residentAgainstGroupProblem(group, value);
|
|
2171
|
+
if (problem !== null) throw new DerivationRefused(p.key, `${who}: residentGroups[${q(g)}] \u2014 ${problem}`);
|
|
2069
2172
|
}
|
|
2070
2173
|
}
|
|
2071
2174
|
const predicateNames = doc.predicates ?? [];
|
package/dist/node/index.js
CHANGED
|
@@ -152,7 +152,7 @@ import {
|
|
|
152
152
|
visibilityKey,
|
|
153
153
|
witness,
|
|
154
154
|
workersOf
|
|
155
|
-
} from "./chunks/chunk-
|
|
155
|
+
} from "./chunks/chunk-5KUYTCXI.js";
|
|
156
156
|
|
|
157
157
|
// src/definition/definition-schema.ts
|
|
158
158
|
var DRAFT_2020_12 = "https://json-schema.org/draft/2020-12/schema";
|
|
@@ -26,7 +26,7 @@ import {
|
|
|
26
26
|
retryPolicyProblem,
|
|
27
27
|
validateDefinition,
|
|
28
28
|
workersOf
|
|
29
|
-
} from "../chunks/chunk-
|
|
29
|
+
} from "../chunks/chunk-5KUYTCXI.js";
|
|
30
30
|
|
|
31
31
|
// src/scheduler/bootstrap.ts
|
|
32
32
|
import { WorkerManager } from "fyflow-scheduler";
|
|
@@ -83,7 +83,7 @@ function bootstrap(registry, groups = {}, options = {}) {
|
|
|
83
83
|
}
|
|
84
84
|
|
|
85
85
|
// src/scheduler/runtime.ts
|
|
86
|
-
import { ConcurrentLimitGroup, FyflowScheduler, KeyedRateLimitGroup, RateLimitGroup } from "fyflow-scheduler";
|
|
86
|
+
import { ConcurrentLimitGroup, FyflowScheduler, KeyedConcurrentLimitGroup, KeyedRateLimitGroup, RateLimitGroup, UnitGroup } from "fyflow-scheduler";
|
|
87
87
|
var RuntimeError = class extends Error {
|
|
88
88
|
/** The entry this refusal is about — a worker's key, a group's key, a type's name. `RegistryError`'s field, one layer out. */
|
|
89
89
|
at;
|
|
@@ -107,8 +107,18 @@ var KEYED_RULE = "FD-R6";
|
|
|
107
107
|
var BUILT_IN_GROUP_FACTORIES = {
|
|
108
108
|
ConcurrentLimitGroup: (config, key) => new ConcurrentLimitGroup(config.limit, key),
|
|
109
109
|
RateLimitGroup: (config, key) => new RateLimitGroup(config.windows, key),
|
|
110
|
-
KeyedRateLimitGroup: (config, key) => new KeyedRateLimitGroup(config.windows, { id: key })
|
|
110
|
+
KeyedRateLimitGroup: (config, key) => new KeyedRateLimitGroup(config.windows, { id: key }),
|
|
111
|
+
UnitGroup: (config, key) => {
|
|
112
|
+
const units = config.units;
|
|
113
|
+
return new UnitGroup(typeof units === "number" ? units : [...units], { id: key });
|
|
114
|
+
},
|
|
115
|
+
KeyedConcurrentLimitGroup: (config, key) => {
|
|
116
|
+
const { limit, limits } = config;
|
|
117
|
+
const own = new Map(Object.entries(limits ?? {}));
|
|
118
|
+
return new KeyedConcurrentLimitGroup(limit, { id: key, limitFor: (k) => own.get(k) ?? limit });
|
|
119
|
+
}
|
|
111
120
|
};
|
|
121
|
+
var KEYED_BUILT_IN_GROUP_TYPES = ["KeyedRateLimitGroup", "KeyedConcurrentLimitGroup"];
|
|
112
122
|
var GROUP_MEMBERS = [
|
|
113
123
|
{ name: "canRun", kind: "function", why: "the scheduler calls it before every admission; a group without it throws on the first task" },
|
|
114
124
|
{ name: "onStart", kind: "function", why: "the scheduler calls it unguarded at admission, though the interface marks it optional; a group without it throws on the first task (measured on 0.5.2)" },
|
|
@@ -210,8 +220,8 @@ function createRuntime(document, code = {}, options = {}) {
|
|
|
210
220
|
pools: poolsOf(registry.pools),
|
|
211
221
|
...registry.gateTypes !== void 0 ? { gateTypes: registry.gateTypes } : {}
|
|
212
222
|
};
|
|
213
|
-
const keyedByBuiltIn = /* @__PURE__ */ new
|
|
214
|
-
for (const [key, g] of resolved.groups) if (g.type
|
|
223
|
+
const keyedByBuiltIn = /* @__PURE__ */ new Map();
|
|
224
|
+
for (const [key, g] of resolved.groups) if (KEYED_BUILT_IN_GROUP_TYPES.includes(g.type)) keyedByBuiltIn.set(key, g.type);
|
|
215
225
|
const keyedFindings = (flow) => {
|
|
216
226
|
const nodes = flow?.nodes;
|
|
217
227
|
if (keyedByBuiltIn.size === 0 || typeof nodes?.[Symbol.iterator] !== "function") return [];
|
|
@@ -226,10 +236,11 @@ function createRuntime(document, code = {}, options = {}) {
|
|
|
226
236
|
const reached = [.../* @__PURE__ */ new Set([...fromPool, ...fromNode])].filter((g) => keyedByBuiltIn.has(g)).sort();
|
|
227
237
|
if (reached.length === 0) continue;
|
|
228
238
|
const at = typeof n.id === "string" ? n.id : null;
|
|
239
|
+
const types = [...new Set(reached.map((g) => keyedByBuiltIn.get(g)))].sort().join(" and ");
|
|
229
240
|
out.push({
|
|
230
241
|
rule: KEYED_RULE,
|
|
231
242
|
at,
|
|
232
|
-
detail: `node ${at === null ? "<unnamed>" : q(at)} reaches keyed group${reached.length > 1 ? "s" : ""} ${reached.map(q).join(", ")}, built by the built-in
|
|
243
|
+
detail: `node ${at === null ? "<unnamed>" : q(at)} reaches keyed group${reached.length > 1 ? "s" : ""} ${reached.map(q).join(", ")}, built by the built-in ${types} factory with no keyFrom \u2014 so every task of it needs a limitKey, which nothing sets unless start is given a createTask, and without one the scheduler's addTask throws "Missing limit key". Two roads work: pass start a createTask that sets limitKey on the task it builds, or declare the group with a code group type (code.groupTypes) whose create() builds a ${types} with a keyFrom that derives the key. A key derived from the flow itself is Q248, deferred`
|
|
233
244
|
});
|
|
234
245
|
}
|
|
235
246
|
return out;
|
|
@@ -61,10 +61,14 @@ export declare class ConfigRefused extends RegistryError {
|
|
|
61
61
|
/** A type's OWN config — its default instance's — fails its own `configSchema` or, derived, the worker's; the type has no valid default. */
|
|
62
62
|
export declare class TypeDefaultRefused extends RegistryError {
|
|
63
63
|
}
|
|
64
|
-
/** A group's `type` names no group type — none of the
|
|
64
|
+
/** A group's `type` names no group type — none of the built-ins (`BUILT_IN_GROUP_TYPES`), none registered in `code`. */
|
|
65
65
|
export declare class GroupTypeMissing extends RegistryError {
|
|
66
66
|
}
|
|
67
|
-
/**
|
|
67
|
+
/**
|
|
68
|
+
* A group's config fails its group type's `configSchema` — or, since v0.56 (ADR 0143 rules 1 and 2), a built-in's rule the
|
|
69
|
+
* schema subset cannot state: a `UnitGroup` listing a unit twice, a `KeyedConcurrentLimitGroup` per-key limit that is not
|
|
70
|
+
* a positive integer.
|
|
71
|
+
*/
|
|
68
72
|
export declare class GroupConfigRefused extends RegistryError {
|
|
69
73
|
}
|
|
70
74
|
/** One name, two definitions: a node type in the document AND in `code`, a pool named as a type, a group type named as a built-in. */
|
|
@@ -82,7 +86,9 @@ export declare class NameMissing extends RegistryError {
|
|
|
82
86
|
/**
|
|
83
87
|
* A derivation returned something that is not a `PoolSpec`, threw, or produced a pool the runtime refuses: a resident group
|
|
84
88
|
* beside `idleTimeout: 0`, or — since v0.39 ([Q619](../../doc/questions/q619.md) option B) — a group of the built-in
|
|
85
|
-
* `KeyedRateLimitGroup` type held in `residentGroups`, which the scheduler would count in one `""` bucket.
|
|
89
|
+
* `KeyedRateLimitGroup` type held in `residentGroups`, which the scheduler would count in one `""` bucket, and since v0.56 one
|
|
90
|
+
* of the built-in `KeyedConcurrentLimitGroup` type, which it rejects at construction; or, since v0.56 (ADR 0143 rules 4 and
|
|
91
|
+
* 5), a resident cost or request of the wrong shape, or one the built-in group it names refuses.
|
|
86
92
|
*/
|
|
87
93
|
export declare class DerivationRefused extends RegistryError {
|
|
88
94
|
}
|
|
@@ -98,12 +104,14 @@ export declare class DerivationRefused extends RegistryError {
|
|
|
98
104
|
export declare class GateTypeRefused extends RegistryError {
|
|
99
105
|
}
|
|
100
106
|
/**
|
|
101
|
-
* **The
|
|
102
|
-
* fact 2.
|
|
103
|
-
* `
|
|
107
|
+
* **The group types the runtime ships, by name and by config schema** — P18 decision 4 and
|
|
108
|
+
* fact 2; three until v0.56, five since (P46 decisions 3 and 5, ADR 0143). Their constructors differ
|
|
109
|
+
* (`ConcurrentLimitGroup(limit, id?)`, `RateLimitGroup(windows, id?)`, `KeyedRateLimitGroup(windows, options?)`,
|
|
110
|
+
* `UnitGroup(units, options?)`, `KeyedConcurrentLimitGroup(limit, options?)`), so each has a factory; the FACTORIES construct scheduler
|
|
104
111
|
* objects and live in the scheduler sub-path, beside `bootstrap` (W66; there is no `/runtime` sub-path — P16 decision 5's *one sub-path that constructs scheduler objects*). What this module needs of them is the NAME — a
|
|
105
112
|
* custom group type colliding with one is refused — and the SCHEMA its config is checked against, and
|
|
106
|
-
* both are data. `RateWindow` is `{ limit, windowMs }` verbatim from the scheduler's `.d.ts`.
|
|
113
|
+
* both are data. `RateWindow` is `{ limit, windowMs }` verbatim from the scheduler's `.d.ts`. **Two rules the
|
|
114
|
+
* subset cannot state** (P46 fact 7) are `builtInConfigProblem`'s, below, checked beside the schema.
|
|
107
115
|
*/
|
|
108
116
|
export declare const BUILT_IN_GROUP_TYPES: Readonly<Record<string, {
|
|
109
117
|
readonly configSchema: JsonSchema;
|
|
@@ -157,8 +165,8 @@ export declare function nodeTypeFromDeclaration(decl: NodeTypeDeclaration, over?
|
|
|
157
165
|
* **Why a derivation's output is not a `PoolSpec`, or `null`** — the check BOTH forms pass through
|
|
158
166
|
* (decision 2's *a class cannot compute around a check*). The scheduler coerces or ignores a wrong
|
|
159
167
|
* option in silence exactly as it does a retry policy (`FD-V30`'s measurements), so the shape is
|
|
160
|
-
* pinned here: numbers where numbers, a string list for `groups`, a string list or a
|
|
161
|
-
*
|
|
168
|
+
* pinned here: numbers where numbers, a string list for `groups`, a string list or a record of a cost
|
|
169
|
+
* or (v0.56) a request per key for `residentGroups` (`residentValueProblem`), and no key beside the eight.
|
|
162
170
|
*
|
|
163
171
|
* **`cancelGraceMs` and `messages` (v0.33, P33 decision 6, W159) are held to the SCHEDULER's own rules**, read
|
|
164
172
|
* at the pin in its `validateGraceMs` and `resolveMessageOptions`, where it throws at `new WorkerManager`:
|
|
@@ -193,7 +201,7 @@ export declare function poolOptionsOf(spec: PoolSpec, inline: boolean | undefine
|
|
|
193
201
|
* a document type has none — its knobs are the fields an instance may override), the derivation RUN,
|
|
194
202
|
* its output held to `PoolSpec`'s shape, the derived config against the WORKER's `configSchema`,
|
|
195
203
|
* every group it names present, and decision 9's refusal — a resident group beside `idleTimeout: 0`
|
|
196
|
-
* deadlocks a contended group (P18 fact 6, the scheduler reference's own sentence).
|
|
204
|
+
* deadlocks a contended group (P18 fact 6, re-measured at P46 fact 6; the scheduler reference's own sentence).
|
|
197
205
|
*
|
|
198
206
|
* **`pool` is a named instance's OWN pool options** (v0.37, ADR 0099): the instance is derived by its type's identity
|
|
199
207
|
* derivation with them merged over the type's declared ones (`nodeTypeFromDeclaration(declaration, pool)`), and the
|
|
@@ -328,6 +328,20 @@ export interface WorkerEntry {
|
|
|
328
328
|
/** What the worker's `run()` returns. Declared and carried; not enforced (P19). */
|
|
329
329
|
outputSchema?: JsonSchema;
|
|
330
330
|
}
|
|
331
|
+
/**
|
|
332
|
+
* **A resident group's REQUEST: which units a worker of the pool may hold** — `spec/flow-definition.md` v0.56, P46
|
|
333
|
+
* decision 4, [ADR 0143](../../doc/decisions/0143-unit-and-keyed-concurrent-groups-are-built-ins-and-a-resident-request-is-checked-against-the-group.md).
|
|
334
|
+
* The scheduler's own `ResidentGroupRequest`, written out rather than imported, as `PoolSpec`'s other fields are: `cost`
|
|
335
|
+
* a positive integer (absent: 1); `from` the candidate units, granted from these only; `prefer` a hint that never blocks.
|
|
336
|
+
* Each a non-empty list of distinct ids — a string, or an integer for a `UnitGroup` declared by count. `{ cost }` alone is
|
|
337
|
+
* a cost written as an object, for any group; `from` / `prefer` are for a group that takes a request (a `UnitGroup`, or a
|
|
338
|
+
* code group type with a `checkRequest`), and the registry holds them against a built-in group's declared units.
|
|
339
|
+
*/
|
|
340
|
+
export interface ResidentGroupRequest {
|
|
341
|
+
cost?: number;
|
|
342
|
+
from?: readonly (string | number)[];
|
|
343
|
+
prefer?: readonly (string | number)[];
|
|
344
|
+
}
|
|
331
345
|
/**
|
|
332
346
|
* **What a derivation RETURNS: what ONE `WorkerManager` needs, minus the two things the TYPE states
|
|
333
347
|
* beside it** (`worker`, `inline`) — decision 2. `config` is the worker's constructor argument and
|
|
@@ -341,8 +355,11 @@ export interface PoolSpec {
|
|
|
341
355
|
idleTimeout?: number;
|
|
342
356
|
/** Group KEYS every task of the pool acquires; the scheduler unions them with a node's `workerGroups`. */
|
|
343
357
|
groups?: readonly string[];
|
|
344
|
-
/**
|
|
345
|
-
|
|
358
|
+
/**
|
|
359
|
+
* Group keys held for the WORKER's lifetime rather than per task — a list (a cost of 1 each), or a record of a cost (a
|
|
360
|
+
* positive integer) or a {@link ResidentGroupRequest} per key (v0.56) — never beside `idleTimeout: 0` (decision 9).
|
|
361
|
+
*/
|
|
362
|
+
residentGroups?: readonly string[] | Readonly<Record<string, number | ResidentGroupRequest>>;
|
|
346
363
|
/**
|
|
347
364
|
* **Threaded pools: the ms a cancelled task's worker gets to stop by itself before it is forced** — the
|
|
348
365
|
* scheduler's own option, declarable since `spec/flow-definition.md` v0.33 (P33 decision 6, W159). Zero or
|
|
@@ -430,13 +447,16 @@ export interface PoolInstance {
|
|
|
430
447
|
}
|
|
431
448
|
/** A resource group the runtime constructs — an instance of a group TYPE, by name, with the config its factory's schema admits. */
|
|
432
449
|
export interface GroupInstance {
|
|
433
|
-
/**
|
|
450
|
+
/**
|
|
451
|
+
* One of the five built-ins (`ConcurrentLimitGroup`, `RateLimitGroup`, `KeyedRateLimitGroup`, and since v0.56
|
|
452
|
+
* `UnitGroup` and `KeyedConcurrentLimitGroup`) or a group type registered in `code`.
|
|
453
|
+
*/
|
|
434
454
|
type: string;
|
|
435
455
|
config?: unknown;
|
|
436
456
|
}
|
|
437
457
|
/**
|
|
438
|
-
* **A resource group is a FACTORY** — decision 4 and P18 fact 2: the scheduler's
|
|
439
|
-
* have
|
|
458
|
+
* **A resource group is a FACTORY** — decision 4 and P18 fact 2: the scheduler's group classes (five
|
|
459
|
+
* built in since v0.56) have different constructors, so each is a `create(config, key)` behind one shape, and a
|
|
440
460
|
* consumer registers a custom class the same way. `create` returns the scheduler's own `ResourceGroup`
|
|
441
461
|
* interface. **The runtime holds what it returns to SIX members** — `canRun`, `onStart` and
|
|
442
462
|
* `onFinish` because the scheduler calls them, `getMetrics`, `id` and `type` because the interface
|
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
*
|
|
14
14
|
* **What it does, in one sentence: `resolveRegistry` says what the document MEANS, and this module
|
|
15
15
|
* builds it.** The resolver checks every name, every schema and every derivation and constructs
|
|
16
|
-
* nothing (W65); here the
|
|
16
|
+
* nothing (W65); here the built-in group types (five since v0.56) get their factories, every group is created and
|
|
17
17
|
* held to what the scheduler actually calls, every worker URL is resolved against the consumer's
|
|
18
18
|
* `base`, `bootstrap` turns every derived instance into a pool, and the driver's registry is derived
|
|
19
19
|
* once and exposed as `runtime.registry` (decision 14).
|
|
@@ -27,7 +27,7 @@
|
|
|
27
27
|
* | omitted | the `.d.ts` says | what the task does |
|
|
28
28
|
* |---|---|---|
|
|
29
29
|
* | `onStart` | optional | THROWS `group.onStart is not a function` |
|
|
30
|
-
* | `onFinish` | optional | **never finishes** — `running` for ever,
|
|
30
|
+
* | `onFinish` | optional | **never finishes** — `running` for ever, and `stats` counts it `running` |
|
|
31
31
|
* | `getMetrics` | required | runs to `done` |
|
|
32
32
|
* | `canRun` | required | THROWS |
|
|
33
33
|
* | `type` | required | runs to `done` |
|
|
@@ -96,9 +96,9 @@ export declare class RunIdInUse extends RuntimeError {
|
|
|
96
96
|
}
|
|
97
97
|
/**
|
|
98
98
|
* **A `start` or a `resume` given no `createTask`, of a flow a node of which reaches a group the BUILT-IN
|
|
99
|
-
* `KeyedRateLimitGroup` factory built** — `FD-R6`,
|
|
99
|
+
* `KeyedRateLimitGroup` factory built — or, since v0.56, the built-in `KeyedConcurrentLimitGroup` factory** — `FD-R6`,
|
|
100
100
|
* [ADR 0097](../../doc/decisions/0097-what-start-refuses-that-the-flow-does-not-decide.md), P37 fact 1 and
|
|
101
|
-
* stop-and-ask (c).
|
|
101
|
+
* stop-and-ask (c). Each such factory gives the group no `keyFrom`, so its key is the task's `limitKey`, which nothing here
|
|
102
102
|
* sets — and the scheduler THROWS in `addTask` for a keyed group with no key. `at` is the first such node in the flow's
|
|
103
103
|
* order; the message carries every `FD-R6` finding `check` returned. Thrown before a run id is minted or a driver built,
|
|
104
104
|
* so the refused call costs nothing.
|
|
@@ -106,8 +106,12 @@ export declare class RunIdInUse extends RuntimeError {
|
|
|
106
106
|
export declare class LimitKeyMissing extends RuntimeError {
|
|
107
107
|
}
|
|
108
108
|
/**
|
|
109
|
-
* **The
|
|
110
|
-
* fact 2, re-run at W66): `(limit, id)`, `(windows, id)` and `(windows, { id })`, the third keyed.
|
|
109
|
+
* **The built-in group types' FACTORIES, and they are different constructors** (P18
|
|
110
|
+
* fact 2, re-run at W66): `(limit, id)`, `(windows, id)` and `(windows, { id })`, the third keyed — and since v0.56 (P46
|
|
111
|
+
* decisions 3 and 5, ADR 0143 rules 1 and 2) `UnitGroup(units, { id })` and `KeyedConcurrentLimitGroup(limit, { id,
|
|
112
|
+
* limitFor })`, the fifth keyed. **Neither keyed factory gives a `keyFrom`**, so each keys a task by its `limitKey` (`FD-R6`).
|
|
113
|
+
* `limitFor` reads a COPY of the document's `limits`, taken here, by OWN key — a later edit to the document moves no limit,
|
|
114
|
+
* and a key called `constructor` is not answered by `Object.prototype`. No `requestFrom`: a task's request is its `units`.
|
|
111
115
|
* `BUILT_IN_GROUP_TYPES` in `src/definition/registry.ts` carries their schemas and no factory,
|
|
112
116
|
* deliberately — the pure entry constructs nothing (`RT-R13`) — so the two halves meet here and are
|
|
113
117
|
* compared both ways by a test.
|
|
@@ -121,10 +125,10 @@ export declare const BUILT_IN_GROUP_FACTORIES: Readonly<Record<string, (config:
|
|
|
121
125
|
* **What a created group must carry.** The first three are what the scheduler CALLS on the acquire
|
|
122
126
|
* path, the last three what the interface requires and a viewer reads — the union, by the
|
|
123
127
|
* maintainer's decision at W66's declaration step. `EventTarget` is deliberately not required.
|
|
124
|
-
* **The `why` strings say "measured on 0.5.2" and stay dated**:
|
|
128
|
+
* **The `why` strings say "measured on 0.5.2" and stay dated**: W224 re-read the current pin's source
|
|
125
129
|
* rather than re-running the table above — `onStart`/`onFinish` are still optional in
|
|
126
|
-
* `ResourceGroup` (`groups/resourceGroup.ts:
|
|
127
|
-
* (`core/FyflowScheduler.ts:
|
|
130
|
+
* `ResourceGroup` (`groups/resourceGroup.ts:140`, `:147`) and still called unguarded
|
|
131
|
+
* (`core/FyflowScheduler.ts:1332`, `:1456`), so S10 stands unchanged.
|
|
128
132
|
*/
|
|
129
133
|
export declare const GROUP_MEMBERS: readonly {
|
|
130
134
|
readonly name: string;
|
|
@@ -395,7 +399,7 @@ export interface Runtime {
|
|
|
395
399
|
* rule 4, P37 stop-and-ask (b). `[]` means `start` will not refuse the flow; a non-empty result is what `start`
|
|
396
400
|
* throws, built from this same call (rule 5): the `FD-V` findings as `InvalidDefinition`, and — since
|
|
397
401
|
* [ADR 0097](../../doc/decisions/0097-what-start-refuses-that-the-flow-does-not-decide.md) — the `FD-R6` findings, a
|
|
398
|
-
* node reaching a group the built-in `KeyedRateLimitGroup` factory built, as `LimitKeyMissing` when `start` is given
|
|
402
|
+
* node reaching a group the built-in `KeyedRateLimitGroup` or (since v0.56) `KeyedConcurrentLimitGroup` factory built, as `LimitKeyMissing` when `start` is given
|
|
399
403
|
* no `createTask`. **`check` reads as `start` given none** (P37 stop-and-ask (c)), so a caller who will pass a
|
|
400
404
|
* `createTask` setting `limitKey` filters `FD-R6` out by its rule id. **It never throws for a flow's defect**: every
|
|
401
405
|
* finding is returned, as `validateDefinition` returns them, against this runtime's own registry. **Not included,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "fyflow-orchestration",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.14.0",
|
|
4
4
|
"description": "Flow orchestration over fyflow-scheduler: sequence, decision, fan-out and join",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"workflow",
|
|
@@ -127,12 +127,12 @@
|
|
|
127
127
|
"@cfworker/json-schema": "4.1.1",
|
|
128
128
|
"@types/node": "^24.13.3",
|
|
129
129
|
"esbuild": "^0.25.12",
|
|
130
|
-
"fyflow-scheduler": "0.
|
|
130
|
+
"fyflow-scheduler": "0.7.0",
|
|
131
131
|
"playwright": "^1.62.1",
|
|
132
132
|
"typescript": "^5.9.2"
|
|
133
133
|
},
|
|
134
134
|
"peerDependencies": {
|
|
135
|
-
"fyflow-scheduler": "0.
|
|
135
|
+
"fyflow-scheduler": "0.7.0"
|
|
136
136
|
},
|
|
137
137
|
"dependencies": {
|
|
138
138
|
"jsonata": "2.2.2"
|