scripts-orchestrator 3.0.0 → 3.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +136 -2
- package/index.js +101 -0
- package/lib/index.js +10 -0
- package/lib/orchestrator.aggregate.test.js +83 -0
- package/lib/orchestrator.concurrency.test.js +112 -0
- package/lib/orchestrator.js +146 -9
- package/lib/report-html.js +108 -9
- package/lib/report-html.test.js +120 -0
- package/lib/workspaces.js +376 -0
- package/lib/workspaces.test.js +339 -0
- package/package.json +1 -1
- package/scripts-orchestrator.config.js +8 -0
package/README.md
CHANGED
|
@@ -26,6 +26,7 @@ npm install --save-dev scripts-orchestrator
|
|
|
26
26
|
## Features
|
|
27
27
|
|
|
28
28
|
- **Parallel Execution**: Runs multiple commands concurrently for faster execution
|
|
29
|
+
- **Concurrency cap**: Bound how many commands a phase runs at once with `max_concurrency` / `--max-concurrency` (defaults to `auto` = CPU count − 1) so smaller machines aren't asked to host every command's toolchain simultaneously (v3.6+). A single phase can pin its own cap with a phase-level `max_concurrency` — e.g. `1` to serialise just that phase's commands while continuing past failures (v3.7+)
|
|
29
30
|
- **Sequential Mode**: Option to run all commands sequentially for low CPU machines
|
|
30
31
|
- **Dependency Management**: Handles command dependencies and ensures proper execution order
|
|
31
32
|
- **Background Processes**: Supports running commands in the background with health checks
|
|
@@ -41,6 +42,7 @@ npm install --save-dev scripts-orchestrator
|
|
|
41
42
|
- **Post-run hook**: Run a shell command after results are written via `post_run` config (v2.14+)
|
|
42
43
|
- **Run-state file**: Library-owned in-progress indicator for live dashboard integration (v2.14+)
|
|
43
44
|
- **Phase recommendations**: Memory-aware `--recommend` mode that proposes an optimal phase layout from a run's time/memory metrics (advisory, v2.15+)
|
|
45
|
+
- **npm workspace aggregation**: First-class workspace roll-up that discovers the npm workspaces in a repo and rolls each workspace's results JSON — plus the root run's global checks — into a single report. Drive it declaratively with the `aggregate` config key (in-process; v3.2+) or via the standalone `--aggregate` CLI mode (v3.1+)
|
|
44
46
|
|
|
45
47
|
## Configuration
|
|
46
48
|
|
|
@@ -416,6 +418,73 @@ This is particularly useful for:
|
|
|
416
418
|
- Debugging individual command failures
|
|
417
419
|
- Avoiding resource contention between commands
|
|
418
420
|
|
|
421
|
+
### Limiting Concurrency (`max_concurrency`)
|
|
422
|
+
|
|
423
|
+
Sequential mode is all-or-nothing. When a phase declares many parallel commands, running *every* one at once can overwhelm a smaller machine (each command may spin up its own Node/toolchain), while `--sequential` over-corrects by dropping to one at a time. `max_concurrency` is the middle ground: it caps how many of a phase's commands run **at once** without serialising everything.
|
|
424
|
+
|
|
425
|
+
- **`'auto'` (the default)** resolves to `max(1, cpuCount - 1)`, leaving one core for the OS/editor.
|
|
426
|
+
- **A positive integer** pins the cap to that exact number.
|
|
427
|
+
- **`0`, negative, or unparseable** values fall back to `auto`.
|
|
428
|
+
|
|
429
|
+
When the cap is greater than or equal to a phase's command count, behaviour is identical to unbounded parallel execution — so well-provisioned machines see no change. As each command finishes, the next queued one starts, keeping at most `max_concurrency` in flight. `--sequential` still wins (it is equivalent to a cap of 1).
|
|
430
|
+
|
|
431
|
+
#### Configuration
|
|
432
|
+
|
|
433
|
+
```js
|
|
434
|
+
export default {
|
|
435
|
+
max_concurrency: 'auto', // or a number like 4
|
|
436
|
+
phases: [
|
|
437
|
+
{
|
|
438
|
+
name: 'quality checks',
|
|
439
|
+
parallel: [
|
|
440
|
+
{ command: 'lint' },
|
|
441
|
+
{ command: 'typecheck' },
|
|
442
|
+
{ command: 'test' },
|
|
443
|
+
// ...more commands than the machine can run at once
|
|
444
|
+
],
|
|
445
|
+
},
|
|
446
|
+
],
|
|
447
|
+
};
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
#### CLI override
|
|
451
|
+
|
|
452
|
+
The `--max-concurrency` flag overrides the configured value for a single run:
|
|
453
|
+
|
|
454
|
+
```bash
|
|
455
|
+
# Run at most 3 commands per phase concurrently
|
|
456
|
+
npm run scripts-orchestrator -- --max-concurrency 3
|
|
457
|
+
|
|
458
|
+
# Force the auto cap (CPU count - 1) regardless of config
|
|
459
|
+
npm run scripts-orchestrator -- --max-concurrency auto
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
At the start of a parallel run the orchestrator logs the resolved cap, e.g. `🧮 Max concurrency: 3 (of 8 CPUs)`.
|
|
463
|
+
|
|
464
|
+
#### Per-phase override
|
|
465
|
+
|
|
466
|
+
A single phase can pin its own cap by setting `max_concurrency` on the phase itself. This is the right tool when one phase's commands share a single resource — a dev server, a GPU, a port — and must run one at a time, while every other phase keeps the global cap and its parallelism. The phase value resolves the same way as the global one (`'auto'` and invalid values fall back to CPU count − 1), and overrides both the configured global cap and any `--max-concurrency` flag for that phase only.
|
|
467
|
+
|
|
468
|
+
```js
|
|
469
|
+
export default {
|
|
470
|
+
max_concurrency: 'auto', // global default for every phase
|
|
471
|
+
phases: [
|
|
472
|
+
{ name: 'quality checks', parallel: [/* runs at the global cap */] },
|
|
473
|
+
{
|
|
474
|
+
name: 'browser suites',
|
|
475
|
+
max_concurrency: 1, // serialise just this phase (one shared dev server)
|
|
476
|
+
parallel: [
|
|
477
|
+
{ command: 'e2e:group-a' },
|
|
478
|
+
{ command: 'e2e:group-b' },
|
|
479
|
+
// ...each runs in turn; a failed group does NOT abort the others
|
|
480
|
+
],
|
|
481
|
+
},
|
|
482
|
+
],
|
|
483
|
+
};
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
A serial phase (`max_concurrency: 1`) still runs through the parallel path, so it **continues past a failed command** rather than stopping on the first failure — unlike the global `--sequential` flag, which both serialises everything and stops the phase at its first failure. When a phase's cap differs from the global one the orchestrator logs the override, e.g. `↳ phase concurrency: 1 (phase "browser suites" overrides the 3 default)`.
|
|
487
|
+
|
|
419
488
|
## Error Handling
|
|
420
489
|
|
|
421
490
|
- The script tracks failed and skipped commands
|
|
@@ -528,11 +597,76 @@ The hook receives two environment variables:
|
|
|
528
597
|
The hook runs synchronously and its exit code is logged but does not change the orchestrator's
|
|
529
598
|
own exit code.
|
|
530
599
|
|
|
531
|
-
**Typical use case:**
|
|
600
|
+
**Typical use case:** roll up a monorepo report after each workspace finishes (see
|
|
601
|
+
**npm workspace aggregation** below):
|
|
532
602
|
```javascript
|
|
533
|
-
post_run: '
|
|
603
|
+
post_run: 'npx scripts-orchestrator --aggregate ../../scripts-orchestrator-aggregate.config.js'
|
|
534
604
|
```
|
|
535
605
|
|
|
606
|
+
## npm workspace aggregation (v3.1+)
|
|
607
|
+
|
|
608
|
+
In a monorepo, each npm workspace can run its own orchestrator gate (writing its own
|
|
609
|
+
`json_results`), and a root run can run repo-wide "global" checks. The aggregator rolls all of
|
|
610
|
+
these into one report — no per-repo merge script required. Drive it the easy way with the
|
|
611
|
+
declarative [`aggregate` config key](#declarative-aggregate-config-key-recommended-v32) below, or
|
|
612
|
+
invoke the [`--aggregate` CLI mode](#--aggregate-cli-mode) directly.
|
|
613
|
+
|
|
614
|
+
It reads only artifacts the library itself writes — each scope's `json_results` and the
|
|
615
|
+
**run-state file** (`.scripts-orchestrator-run.json`) — so it needs no log scraping. The
|
|
616
|
+
run-state file tells it whether the run is still in flight (live report with auto-refresh) or
|
|
617
|
+
finished (static report). Each workspace section is classified as **OK / FAIL / RUNNING /
|
|
618
|
+
PENDING / STALE / INTERRUPTED / N/A** from its own results JSON and the run window.
|
|
619
|
+
|
|
620
|
+
### Declarative `aggregate` config key (recommended, v3.2+)
|
|
621
|
+
|
|
622
|
+
Rather than wiring a `periodic_hook` / `post_run` that shells out to `--aggregate`, set the
|
|
623
|
+
**`aggregate`** key in your orchestrator config and the library drives the roll-up **in-process** —
|
|
624
|
+
no subprocess spawned every interval, no dependency on `npx`/PATH resolution:
|
|
625
|
+
|
|
626
|
+
```javascript
|
|
627
|
+
// root run config — roll up periodically while running + once, static, at the end
|
|
628
|
+
aggregate: './scripts-orchestrator-aggregate.config.js', // or `true` for the built-in defaults
|
|
629
|
+
periodic_interval_ms: 45000, // cadence for the in-process roll-up
|
|
630
|
+
|
|
631
|
+
// each workspace gate config — refresh the roll-up as that workspace finishes
|
|
632
|
+
aggregate: '../../scripts-orchestrator-aggregate.config.js',
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
The value may be `true` (use defaults), a path to a config module (its `default` export is used as
|
|
636
|
+
the options below), or an options object inline. The library auto-detects whether the current run is
|
|
637
|
+
the **repo-root run** (it owns the periodic cadence and writes the final static report) or a
|
|
638
|
+
**fanned-out workspace run** (it refreshes the roll-up once, as that workspace finishes, leaving the
|
|
639
|
+
report in-progress because the root run is still live). On interrupt, the orchestrator writes one
|
|
640
|
+
final static roll-up itself.
|
|
641
|
+
|
|
642
|
+
### `--aggregate` CLI mode
|
|
643
|
+
|
|
644
|
+
The same roll-up is available as a standalone CLI mode (used internally by the declarative key, and
|
|
645
|
+
handy for manual/one-off rendering or legacy hook wiring). It is safe to fire repeatedly:
|
|
646
|
+
|
|
647
|
+
```bash
|
|
648
|
+
scripts-orchestrator --aggregate # use the built-in defaults
|
|
649
|
+
scripts-orchestrator --aggregate ./scripts-orchestrator-aggregate.config.js # override paths/title
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
The optional config module's `default` export may override any of these (all paths are
|
|
653
|
+
resolved against the auto-detected repo root unless absolute):
|
|
654
|
+
|
|
655
|
+
| Key | Default | Meaning |
|
|
656
|
+
| --- | --- | --- |
|
|
657
|
+
| `title` | `Workspaces Quality Report` | Report heading |
|
|
658
|
+
| `outJson` / `outHtml` | `logs/monorepo-quality-report.{json,html}` | Where the roll-up is written |
|
|
659
|
+
| `runStateFile` | `logs/.scripts-orchestrator-run.json` | Run-state file used to detect in-progress + run start |
|
|
660
|
+
| `rootResults` | `logs/scripts-orchestrator-logs/scripts-orchestrator-results.json` | Root run's results (source of global-check rows) |
|
|
661
|
+
| `globalResults` | `logs/scripts-orchestrator-logs/scripts-orchestrator-global-results.json` | Fallback source of global-check rows |
|
|
662
|
+
| `workspaceResults` | `logs/scripts-orchestrator-logs/scripts-orchestrator-results.json` | Per-workspace results path (relative to each workspace) |
|
|
663
|
+
| `globalPhase` / `workspacePhase` | `global quality checks` / `workspace quality gates` | Phase names used to split global rows from the fan-out row |
|
|
664
|
+
| `refreshSecs` | `5` | Auto-refresh cadence injected while the run is in progress |
|
|
665
|
+
| `exclude` | `[]` | Workspace directories (repo-root-relative) to omit |
|
|
666
|
+
|
|
667
|
+
The library also exports the building blocks for programmatic use:
|
|
668
|
+
`findRepoRoot`, `discoverWorkspaceDirs`, `aggregateWorkspacesReport`, `writeAggregateReport`.
|
|
669
|
+
|
|
536
670
|
## Git-Based Caching
|
|
537
671
|
|
|
538
672
|
The orchestrator automatically tracks the git commit hash and repository state to optimize execution:
|
package/index.js
CHANGED
|
@@ -35,6 +35,11 @@ const argv = yargs(hideBin(process.argv))
|
|
|
35
35
|
type: 'boolean',
|
|
36
36
|
description: 'Run all commands sequentially instead of in parallel (for low CPU machines)',
|
|
37
37
|
})
|
|
38
|
+
.option('max-concurrency', {
|
|
39
|
+
type: 'string',
|
|
40
|
+
description:
|
|
41
|
+
'Cap how many commands a phase runs at once: a positive integer, or \'auto\' (cpuCount - 1). Overrides config max_concurrency. Ignored under --sequential.',
|
|
42
|
+
})
|
|
38
43
|
.option('force', {
|
|
39
44
|
type: 'boolean',
|
|
40
45
|
description: 'Force execution even if git state is unchanged',
|
|
@@ -60,6 +65,11 @@ const argv = yargs(hideBin(process.argv))
|
|
|
60
65
|
description:
|
|
61
66
|
'Analyse an existing results JSON and print a memory-aware phase recommendation (no run).',
|
|
62
67
|
})
|
|
68
|
+
.option('aggregate', {
|
|
69
|
+
type: 'string',
|
|
70
|
+
description:
|
|
71
|
+
'Aggregate every npm workspace\'s results JSON into one roll-up report (no run). Pass an optional config path to override the default paths/title.',
|
|
72
|
+
})
|
|
63
73
|
.option('fanout', {
|
|
64
74
|
type: 'number',
|
|
65
75
|
description: 'Workspace fan-out (parallel gates sharing the host) used to size the --recommend budget. Default 1.',
|
|
@@ -148,6 +158,47 @@ if (argv.recommend != null) {
|
|
|
148
158
|
process.exit(0);
|
|
149
159
|
}
|
|
150
160
|
|
|
161
|
+
// --aggregate mode: roll up every npm workspace's results JSON (plus the root run's global
|
|
162
|
+
// checks) into a single report. Advisory of nothing — it just renders state already on disk, so
|
|
163
|
+
// it is safe to fire repeatedly (e.g. from the periodic_hook while the run is in flight, or once
|
|
164
|
+
// at the end). No orchestration run happens here.
|
|
165
|
+
if (argv.aggregate != null) {
|
|
166
|
+
const { writeAggregateReport } = await import('./lib/index.js');
|
|
167
|
+
let options = {};
|
|
168
|
+
const cfgArg = argv.aggregate;
|
|
169
|
+
if (cfgArg && cfgArg !== '-') {
|
|
170
|
+
const cfgPath = path.resolve(process.cwd(), cfgArg);
|
|
171
|
+
if (!fs.existsSync(cfgPath)) {
|
|
172
|
+
log.error(`Error: --aggregate config not found at ${cfgPath}`);
|
|
173
|
+
process.exit(1);
|
|
174
|
+
}
|
|
175
|
+
try {
|
|
176
|
+
options = (await import(new URL(`file://${cfgPath}`).href)).default ?? {};
|
|
177
|
+
} catch (err) {
|
|
178
|
+
log.error(`Error: failed to load --aggregate config: ${err.message}`);
|
|
179
|
+
process.exit(1);
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
// The orchestrator fires this hook a final time AFTER clearing its run-state, tagging it with
|
|
183
|
+
// SCRIPTS_ORCHESTRATOR_PERIODIC=final. Honour that as an explicit "run is over" signal so the
|
|
184
|
+
// final report is static (no auto-refresh) even if a stray marker lingers.
|
|
185
|
+
if (process.env.SCRIPTS_ORCHESTRATOR_PERIODIC === 'final') {
|
|
186
|
+
options = { ...options, inProgress: false };
|
|
187
|
+
}
|
|
188
|
+
try {
|
|
189
|
+
const { jsonPath, htmlPath } = writeAggregateReport(options);
|
|
190
|
+
if (process.env.ORCHESTRATOR_MERGE_QUIET !== '1') {
|
|
191
|
+
log.info(
|
|
192
|
+
`📄 Aggregated workspace report → ${path.relative(process.cwd(), jsonPath)} (+ ${path.basename(htmlPath)})`,
|
|
193
|
+
);
|
|
194
|
+
}
|
|
195
|
+
} catch (err) {
|
|
196
|
+
log.error(`Error: failed to aggregate workspace report: ${err.message}`);
|
|
197
|
+
process.exit(1);
|
|
198
|
+
}
|
|
199
|
+
process.exit(0);
|
|
200
|
+
}
|
|
201
|
+
|
|
151
202
|
// Extract arguments
|
|
152
203
|
const args = argv._;
|
|
153
204
|
const configPath = args[0] || './scripts-orchestrator.config.js';
|
|
@@ -207,6 +258,14 @@ const htmlResultsPath =
|
|
|
207
258
|
// post-run hook — shell command run after json_results written
|
|
208
259
|
const postRun = commandsConfig.post_run ?? null;
|
|
209
260
|
|
|
261
|
+
// Memory heat thresholds for the HTML report — fractions (0–1) of the run's peak memory above which
|
|
262
|
+
// a command is coloured amber (mid) / red (high). Config: `memory_heat: { mid: 0.33, high: 0.66 }`.
|
|
263
|
+
const memoryHeat = commandsConfig.memory_heat ?? null;
|
|
264
|
+
|
|
265
|
+
// Duration heat thresholds for the HTML report — fractions (0–1) of the run's slowest command above
|
|
266
|
+
// which a command is coloured amber (mid) / red (high). Config: `duration_heat: { mid: 0.33, high: 0.66 }`.
|
|
267
|
+
const durationHeat = commandsConfig.duration_heat ?? null;
|
|
268
|
+
|
|
210
269
|
// Periodic hook — shell command run on an interval WHILE the run is in flight (e.g. to roll up
|
|
211
270
|
// results into an aggregate report). Library owns only the cadence; the command is project-specific.
|
|
212
271
|
const periodicHook = commandsConfig.periodic_hook ?? null;
|
|
@@ -214,6 +273,34 @@ const periodicIntervalMs = Number(commandsConfig.periodic_interval_ms) > 0
|
|
|
214
273
|
? Number(commandsConfig.periodic_interval_ms)
|
|
215
274
|
: 45000;
|
|
216
275
|
|
|
276
|
+
// Declarative npm-workspace roll-up. `aggregate: true` uses library defaults; a string loads that
|
|
277
|
+
// config module's default export as writeAggregateReport options; an object is used verbatim. When
|
|
278
|
+
// set, the library drives the workspace aggregate IN-PROCESS — the repo-root run rolls up on the
|
|
279
|
+
// periodic cadence + once at the end (static), and a fanned-out workspace run rolls up once as it
|
|
280
|
+
// finishes. This is the declarative replacement for wiring periodic_hook / post_run to shell out
|
|
281
|
+
// to `scripts-orchestrator --aggregate`.
|
|
282
|
+
let aggregateOptions = null;
|
|
283
|
+
const aggregateCfg = commandsConfig.aggregate;
|
|
284
|
+
if (aggregateCfg != null && aggregateCfg !== false) {
|
|
285
|
+
if (aggregateCfg === true) {
|
|
286
|
+
aggregateOptions = {};
|
|
287
|
+
} else if (typeof aggregateCfg === 'string') {
|
|
288
|
+
const aggCfgPath = path.resolve(process.cwd(), aggregateCfg);
|
|
289
|
+
if (!fs.existsSync(aggCfgPath)) {
|
|
290
|
+
log.error(`Error: aggregate config not found at ${aggCfgPath}`);
|
|
291
|
+
process.exit(1);
|
|
292
|
+
}
|
|
293
|
+
try {
|
|
294
|
+
aggregateOptions = (await import(new URL(`file://${aggCfgPath}`).href)).default ?? {};
|
|
295
|
+
} catch (err) {
|
|
296
|
+
log.error(`Error: failed to load aggregate config: ${err.message}`);
|
|
297
|
+
process.exit(1);
|
|
298
|
+
}
|
|
299
|
+
} else if (typeof aggregateCfg === 'object') {
|
|
300
|
+
aggregateOptions = aggregateCfg;
|
|
301
|
+
}
|
|
302
|
+
}
|
|
303
|
+
|
|
217
304
|
// Set the log folder for the main orchestrator logs if specified
|
|
218
305
|
if (logFolder) {
|
|
219
306
|
log.setLogFolder(logFolder);
|
|
@@ -233,9 +320,18 @@ const orchestrator = new Orchestrator(
|
|
|
233
320
|
);
|
|
234
321
|
// wire post-run hook from config
|
|
235
322
|
orchestrator.postRun = postRun;
|
|
323
|
+
// wire memory heat thresholds from config (HTML report colouring)
|
|
324
|
+
orchestrator.memoryHeat = memoryHeat;
|
|
325
|
+
orchestrator.durationHeat = durationHeat;
|
|
236
326
|
// Wire periodic hook (cadence owned by the library)
|
|
237
327
|
orchestrator.periodicHook = periodicHook;
|
|
238
328
|
orchestrator.periodicIntervalMs = periodicIntervalMs;
|
|
329
|
+
// Wire declarative in-process workspace roll-up (takes the in-process path when set)
|
|
330
|
+
orchestrator.aggregateOptions = aggregateOptions;
|
|
331
|
+
// CLI --max-concurrency overrides the config's max_concurrency (resolved to a concrete cap).
|
|
332
|
+
if (argv.maxConcurrency != null && argv.maxConcurrency !== '') {
|
|
333
|
+
orchestrator.maxConcurrency = orchestrator._resolveMaxConcurrency(argv.maxConcurrency);
|
|
334
|
+
}
|
|
239
335
|
|
|
240
336
|
// Enhanced signal handlers
|
|
241
337
|
const handleSignal = async (signal) => {
|
|
@@ -248,6 +344,11 @@ const handleSignal = async (signal) => {
|
|
|
248
344
|
}
|
|
249
345
|
// clear run-state so dashboards know the run ended
|
|
250
346
|
orchestrator._clearRunState();
|
|
347
|
+
// Write one final static workspace roll-up so an interrupted run leaves a non-refreshing report
|
|
348
|
+
// (only when the declarative in-process aggregate is configured; a no-op otherwise).
|
|
349
|
+
if (orchestrator.aggregateOptions) {
|
|
350
|
+
orchestrator._firePeriodicHookFinal();
|
|
351
|
+
}
|
|
251
352
|
process.exit(1);
|
|
252
353
|
};
|
|
253
354
|
|
package/lib/index.js
CHANGED
|
@@ -13,6 +13,12 @@ import {
|
|
|
13
13
|
observedTimeline,
|
|
14
14
|
packPhases,
|
|
15
15
|
} from './recommend-phases.js';
|
|
16
|
+
import {
|
|
17
|
+
findRepoRoot,
|
|
18
|
+
discoverWorkspaceDirs,
|
|
19
|
+
aggregateWorkspacesReport,
|
|
20
|
+
writeAggregateReport,
|
|
21
|
+
} from './workspaces.js';
|
|
16
22
|
|
|
17
23
|
export {
|
|
18
24
|
Orchestrator,
|
|
@@ -28,5 +34,9 @@ export {
|
|
|
28
34
|
usableSteps,
|
|
29
35
|
observedTimeline,
|
|
30
36
|
packPhases,
|
|
37
|
+
findRepoRoot,
|
|
38
|
+
discoverWorkspaceDirs,
|
|
39
|
+
aggregateWorkspacesReport,
|
|
40
|
+
writeAggregateReport,
|
|
31
41
|
};
|
|
32
42
|
export default Orchestrator;
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
import { Orchestrator } from './orchestrator.js';
|
|
2
|
+
|
|
3
|
+
const baseConfig = (extra = {}) => ({
|
|
4
|
+
phases: [{ name: 'p', parallel: [{ command: 'build' }] }],
|
|
5
|
+
...extra,
|
|
6
|
+
});
|
|
7
|
+
|
|
8
|
+
// Build an orchestrator with the in-process aggregate wired, and stub the seams that would
|
|
9
|
+
// otherwise spawn processes / touch disk so we can assert the decision logic in isolation.
|
|
10
|
+
function makeOrch({ aggregateOptions = {}, periodicHook = null, isRoot = true } = {}) {
|
|
11
|
+
const orch = new Orchestrator(baseConfig());
|
|
12
|
+
orch.aggregateOptions = aggregateOptions;
|
|
13
|
+
orch.periodicHook = periodicHook;
|
|
14
|
+
orch._isRepoRootRun = () => isRoot;
|
|
15
|
+
// Capture aggregate fires instead of writing a report.
|
|
16
|
+
orch._fireAggregateCalls = [];
|
|
17
|
+
orch._fireAggregate = (inProgress) => orch._fireAggregateCalls.push(inProgress);
|
|
18
|
+
return orch;
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
// Minimal call-counter so the suite stays dependency-free (no jest globals under the ESM runner).
|
|
22
|
+
function counter() {
|
|
23
|
+
const fn = (...args) => { fn.calls.push(args); };
|
|
24
|
+
fn.calls = [];
|
|
25
|
+
return fn;
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
describe('Orchestrator in-process workspace aggregate', () => {
|
|
29
|
+
test('no aggregate configured: periodic machinery is inert', () => {
|
|
30
|
+
const orch = new Orchestrator(baseConfig());
|
|
31
|
+
const fireSpy = counter();
|
|
32
|
+
orch._firePeriodicTick = fireSpy;
|
|
33
|
+
orch._startPeriodicHook();
|
|
34
|
+
expect(orch._periodicTimer).toBeNull();
|
|
35
|
+
expect(fireSpy.calls).toHaveLength(0);
|
|
36
|
+
});
|
|
37
|
+
|
|
38
|
+
test('root run with aggregate: starts cadence with a prompt first roll-up', () => {
|
|
39
|
+
const orch = makeOrch({ isRoot: true });
|
|
40
|
+
const tickSpy = counter();
|
|
41
|
+
orch._firePeriodicTick = tickSpy;
|
|
42
|
+
orch._startPeriodicHook();
|
|
43
|
+
expect(tickSpy.calls).toHaveLength(1); // prompt first roll-up
|
|
44
|
+
expect(orch._periodicTimer).not.toBeNull();
|
|
45
|
+
orch._stopPeriodicHook(); // avoid a dangling interval
|
|
46
|
+
expect(orch._periodicTimer).toBeNull();
|
|
47
|
+
});
|
|
48
|
+
|
|
49
|
+
test('fanned-out workspace run with aggregate: no periodic cadence', () => {
|
|
50
|
+
const orch = makeOrch({ isRoot: false });
|
|
51
|
+
const tickSpy = counter();
|
|
52
|
+
orch._firePeriodicTick = tickSpy;
|
|
53
|
+
orch._startPeriodicHook();
|
|
54
|
+
expect(tickSpy.calls).toHaveLength(0);
|
|
55
|
+
expect(orch._periodicTimer).toBeNull();
|
|
56
|
+
});
|
|
57
|
+
|
|
58
|
+
test('periodic tick rolls up with auto-detected inProgress (undefined)', () => {
|
|
59
|
+
const orch = makeOrch({ isRoot: true });
|
|
60
|
+
orch._firePeriodicTick();
|
|
61
|
+
expect(orch._fireAggregateCalls).toEqual([undefined]);
|
|
62
|
+
});
|
|
63
|
+
|
|
64
|
+
test('final fire on the root run forces a static (inProgress=false) report', () => {
|
|
65
|
+
const orch = makeOrch({ isRoot: true });
|
|
66
|
+
orch._firePeriodicHookFinal();
|
|
67
|
+
expect(orch._fireAggregateCalls).toEqual([false]);
|
|
68
|
+
});
|
|
69
|
+
|
|
70
|
+
test('final fire on a workspace run leaves inProgress auto-detected (root still running)', () => {
|
|
71
|
+
const orch = makeOrch({ isRoot: false });
|
|
72
|
+
orch._firePeriodicHookFinal();
|
|
73
|
+
expect(orch._fireAggregateCalls).toEqual([undefined]);
|
|
74
|
+
});
|
|
75
|
+
|
|
76
|
+
test('a legacy periodic_hook takes precedence over the in-process aggregate', () => {
|
|
77
|
+
const orch = makeOrch({ isRoot: true, periodicHook: 'echo hi' });
|
|
78
|
+
// With a shell hook configured, the tick must NOT take the in-process path.
|
|
79
|
+
orch._periodicRunning = true; // make the shell path a no-op without spawning
|
|
80
|
+
orch._firePeriodicTick();
|
|
81
|
+
expect(orch._fireAggregateCalls).toEqual([]);
|
|
82
|
+
});
|
|
83
|
+
});
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
import os from 'os';
|
|
2
|
+
import { Orchestrator } from './orchestrator.js';
|
|
3
|
+
|
|
4
|
+
const baseConfig = (extra = {}) => ({
|
|
5
|
+
phases: [{ name: 'p', parallel: [{ command: 'build' }] }],
|
|
6
|
+
...extra,
|
|
7
|
+
});
|
|
8
|
+
|
|
9
|
+
const cpuAuto = () => Math.max(1, os.cpus().length - 1);
|
|
10
|
+
|
|
11
|
+
describe('Orchestrator max concurrency resolution', () => {
|
|
12
|
+
test('defaults to auto (cpuCount - 1) when nothing is configured', () => {
|
|
13
|
+
const orch = new Orchestrator(baseConfig());
|
|
14
|
+
expect(orch.maxConcurrency).toBe(cpuAuto());
|
|
15
|
+
});
|
|
16
|
+
|
|
17
|
+
test('config max_concurrency: \'auto\' resolves to cpuCount - 1', () => {
|
|
18
|
+
const orch = new Orchestrator(baseConfig({ max_concurrency: 'auto' }));
|
|
19
|
+
expect(orch.maxConcurrency).toBe(cpuAuto());
|
|
20
|
+
});
|
|
21
|
+
|
|
22
|
+
test('a positive integer is used verbatim (floored)', () => {
|
|
23
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: 4 })).maxConcurrency).toBe(4);
|
|
24
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: 3.9 })).maxConcurrency).toBe(3);
|
|
25
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: '6' })).maxConcurrency).toBe(6);
|
|
26
|
+
});
|
|
27
|
+
|
|
28
|
+
test('zero, negative, and unparseable values fall back to auto', () => {
|
|
29
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: 0 })).maxConcurrency).toBe(cpuAuto());
|
|
30
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: -2 })).maxConcurrency).toBe(cpuAuto());
|
|
31
|
+
expect(new Orchestrator(baseConfig({ max_concurrency: 'lots' })).maxConcurrency).toBe(cpuAuto());
|
|
32
|
+
});
|
|
33
|
+
|
|
34
|
+
test('legacy array config still resolves a cap (no max_concurrency key available)', () => {
|
|
35
|
+
const orch = new Orchestrator([{ command: 'a' }, { command: 'b' }]);
|
|
36
|
+
expect(orch.maxConcurrency).toBe(cpuAuto());
|
|
37
|
+
});
|
|
38
|
+
});
|
|
39
|
+
|
|
40
|
+
describe('Orchestrator._runWithConcurrency', () => {
|
|
41
|
+
test('preserves input order in the results array', async () => {
|
|
42
|
+
const orch = new Orchestrator(baseConfig());
|
|
43
|
+
const items = [10, 20, 30, 40];
|
|
44
|
+
const out = await orch._runWithConcurrency(items, 2, async (n) => n * 2);
|
|
45
|
+
expect(out).toEqual([20, 40, 60, 80]);
|
|
46
|
+
});
|
|
47
|
+
|
|
48
|
+
test('never exceeds the concurrency limit and still runs every item', async () => {
|
|
49
|
+
const orch = new Orchestrator(baseConfig());
|
|
50
|
+
let inFlight = 0;
|
|
51
|
+
let peak = 0;
|
|
52
|
+
const limit = 3;
|
|
53
|
+
const items = Array.from({ length: 12 }, (_, i) => i);
|
|
54
|
+
const worker = async (i) => {
|
|
55
|
+
inFlight += 1;
|
|
56
|
+
peak = Math.max(peak, inFlight);
|
|
57
|
+
// Yield a couple of microtasks so overlap actually happens.
|
|
58
|
+
await Promise.resolve();
|
|
59
|
+
await Promise.resolve();
|
|
60
|
+
inFlight -= 1;
|
|
61
|
+
return i;
|
|
62
|
+
};
|
|
63
|
+
const out = await orch._runWithConcurrency(items, limit, worker);
|
|
64
|
+
expect(out).toEqual(items);
|
|
65
|
+
expect(peak).toBeLessThanOrEqual(limit);
|
|
66
|
+
expect(peak).toBeGreaterThan(1); // proves it actually parallelised
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
test('limit >= item count behaves like Promise.all (all start immediately)', async () => {
|
|
70
|
+
const orch = new Orchestrator(baseConfig());
|
|
71
|
+
let inFlight = 0;
|
|
72
|
+
let peak = 0;
|
|
73
|
+
const items = [1, 2, 3];
|
|
74
|
+
await orch._runWithConcurrency(items, 10, async (n) => {
|
|
75
|
+
inFlight += 1;
|
|
76
|
+
peak = Math.max(peak, inFlight);
|
|
77
|
+
await Promise.resolve();
|
|
78
|
+
inFlight -= 1;
|
|
79
|
+
return n;
|
|
80
|
+
});
|
|
81
|
+
expect(peak).toBe(items.length);
|
|
82
|
+
});
|
|
83
|
+
|
|
84
|
+
test('empty input resolves to an empty array', async () => {
|
|
85
|
+
const orch = new Orchestrator(baseConfig());
|
|
86
|
+
await expect(orch._runWithConcurrency([], 4, async (n) => n)).resolves.toEqual([]);
|
|
87
|
+
});
|
|
88
|
+
});
|
|
89
|
+
|
|
90
|
+
describe('Orchestrator._phaseConcurrency (per-phase override)', () => {
|
|
91
|
+
test('falls back to the global cap when a phase has no max_concurrency', () => {
|
|
92
|
+
const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
|
|
93
|
+
expect(orch._phaseConcurrency({ name: 'p', parallel: [] })).toBe(5);
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
test('a phase max_concurrency overrides the global cap (both directions)', () => {
|
|
97
|
+
const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
|
|
98
|
+
expect(orch._phaseConcurrency({ name: 'serial', max_concurrency: 1 })).toBe(1);
|
|
99
|
+
expect(orch._phaseConcurrency({ name: 'wider', max_concurrency: 8 })).toBe(8);
|
|
100
|
+
});
|
|
101
|
+
|
|
102
|
+
test('a phase\'s \'auto\' resolves to cpuCount - 1 regardless of the global cap', () => {
|
|
103
|
+
const orch = new Orchestrator(baseConfig({ max_concurrency: 1 }));
|
|
104
|
+
expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 'auto' })).toBe(cpuAuto());
|
|
105
|
+
});
|
|
106
|
+
|
|
107
|
+
test('an invalid phase value falls back to auto, not to the global cap', () => {
|
|
108
|
+
const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
|
|
109
|
+
expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 0 })).toBe(cpuAuto());
|
|
110
|
+
expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 'lots' })).toBe(cpuAuto());
|
|
111
|
+
});
|
|
112
|
+
});
|