scripts-orchestrator 3.5.0 → 3.7.0

package/README.md

@@ -26,6 +26,7 @@ npm install --save-dev scripts-orchestrator
 ## Features
 
 - **Parallel Execution**: Runs multiple commands concurrently for faster execution
+- **Concurrency cap**: Bound how many commands a phase runs at once with `max_concurrency` / `--max-concurrency` (defaults to `auto` = CPU count − 1) so smaller machines aren't asked to host every command's toolchain simultaneously (v3.6+). A single phase can pin its own cap with a phase-level `max_concurrency` — e.g. `1` to serialise just that phase's commands while continuing past failures (v3.7+)
 - **Sequential Mode**: Option to run all commands sequentially for low CPU machines
 - **Dependency Management**: Handles command dependencies and ensures proper execution order
 - **Background Processes**: Supports running commands in the background with health checks
@@ -417,6 +418,73 @@ This is particularly useful for:
 - Debugging individual command failures
 - Avoiding resource contention between commands
 
+### Limiting Concurrency (`max_concurrency`)
+
+Sequential mode is all-or-nothing. When a phase declares many parallel commands, running *every* one at once can overwhelm a smaller machine (each command may spin up its own Node/toolchain), while `--sequential` over-corrects by dropping to one at a time. `max_concurrency` is the middle ground: it caps how many of a phase's commands run **at once** without serialising everything.
+
+- **`'auto'` (the default)** resolves to `max(1, cpuCount - 1)`, leaving one core for the OS/editor.
+- **A positive integer** pins the cap to that exact number.
+- **`0`, negative, or unparseable** values fall back to `auto`.
+
+When the cap is greater than or equal to a phase's command count, behaviour is identical to unbounded parallel execution — so well-provisioned machines see no change. As each command finishes, the next queued one starts, keeping at most `max_concurrency` in flight. `--sequential` still wins (it is equivalent to a cap of 1).
+
+#### Configuration
+
+```js
+export default {
+  max_concurrency: 'auto', // or a number like 4
+  phases: [
+    {
+      name: 'quality checks',
+      parallel: [
+        { command: 'lint' },
+        { command: 'typecheck' },
+        { command: 'test' },
+        // ...more commands than the machine can run at once
+      ],
+    },
+  ],
+};
+```
+
+#### CLI override
+
+The `--max-concurrency` flag overrides the configured value for a single run:
+
+```bash
+# Run at most 3 commands per phase concurrently
+npm run scripts-orchestrator -- --max-concurrency 3
+
+# Force the auto cap (CPU count - 1) regardless of config
+npm run scripts-orchestrator -- --max-concurrency auto
+```
+
+At the start of a parallel run the orchestrator logs the resolved cap, e.g. `🧮 Max concurrency: 3 (of 8 CPUs)`.
+
+#### Per-phase override
+
+A single phase can pin its own cap by setting `max_concurrency` on the phase itself. This is the right tool when one phase's commands share a single resource — a dev server, a GPU, a port — and must run one at a time, while every other phase keeps the global cap and its parallelism. The phase value resolves the same way as the global one (`'auto'` and invalid values fall back to CPU count − 1), and overrides both the configured global cap and any `--max-concurrency` flag for that phase only.
+
+```js
+export default {
+  max_concurrency: 'auto', // global default for every phase
+  phases: [
+    { name: 'quality checks', parallel: [/* runs at the global cap */] },
+    {
+      name: 'browser suites',
+      max_concurrency: 1, // serialise just this phase (one shared dev server)
+      parallel: [
+        { command: 'e2e:group-a' },
+        { command: 'e2e:group-b' },
+        // ...each runs in turn; a failed group does NOT abort the others
+      ],
+    },
+  ],
+};
+```
+
+A serial phase (`max_concurrency: 1`) still runs through the parallel path, so it **continues past a failed command** rather than stopping on the first failure — unlike the global `--sequential` flag, which both serialises everything and stops the phase at its first failure. When a phase's cap differs from the global one the orchestrator logs the override, e.g. `↳ phase concurrency: 1 (phase "browser suites" overrides the 3 default)`.
+
 ## Error Handling
 
 - The script tracks failed and skipped commands

package/index.js

@@ -35,6 +35,11 @@ const argv = yargs(hideBin(process.argv))
     type: 'boolean',
     description: 'Run all commands sequentially instead of in parallel (for low CPU machines)',
   })
+  .option('max-concurrency', {
+    type: 'string',
+    description:
+      'Cap how many commands a phase runs at once: a positive integer, or \'auto\' (cpuCount - 1). Overrides config max_concurrency. Ignored under --sequential.',
+  })
   .option('force', {
     type: 'boolean',
     description: 'Force execution even if git state is unchanged',
@@ -323,6 +328,10 @@ orchestrator.periodicHook = periodicHook;
 orchestrator.periodicIntervalMs = periodicIntervalMs;
 // Wire declarative in-process workspace roll-up (takes the in-process path when set)
 orchestrator.aggregateOptions = aggregateOptions;
+// CLI --max-concurrency overrides the config's max_concurrency (resolved to a concrete cap).
+if (argv.maxConcurrency != null && argv.maxConcurrency !== '') {
+  orchestrator.maxConcurrency = orchestrator._resolveMaxConcurrency(argv.maxConcurrency);
+}
 
 // Enhanced signal handlers
 const handleSignal = async (signal) => {

package/lib/orchestrator.concurrency.test.js

@@ -0,0 +1,112 @@
+import os from 'os';
+import { Orchestrator } from './orchestrator.js';
+
+const baseConfig = (extra = {}) => ({
+  phases: [{ name: 'p', parallel: [{ command: 'build' }] }],
+  ...extra,
+});
+
+const cpuAuto = () => Math.max(1, os.cpus().length - 1);
+
+describe('Orchestrator max concurrency resolution', () => {
+  test('defaults to auto (cpuCount - 1) when nothing is configured', () => {
+    const orch = new Orchestrator(baseConfig());
+    expect(orch.maxConcurrency).toBe(cpuAuto());
+  });
+
+  test('config max_concurrency: \'auto\' resolves to cpuCount - 1', () => {
+    const orch = new Orchestrator(baseConfig({ max_concurrency: 'auto' }));
+    expect(orch.maxConcurrency).toBe(cpuAuto());
+  });
+
+  test('a positive integer is used verbatim (floored)', () => {
+    expect(new Orchestrator(baseConfig({ max_concurrency: 4 })).maxConcurrency).toBe(4);
+    expect(new Orchestrator(baseConfig({ max_concurrency: 3.9 })).maxConcurrency).toBe(3);
+    expect(new Orchestrator(baseConfig({ max_concurrency: '6' })).maxConcurrency).toBe(6);
+  });
+
+  test('zero, negative, and unparseable values fall back to auto', () => {
+    expect(new Orchestrator(baseConfig({ max_concurrency: 0 })).maxConcurrency).toBe(cpuAuto());
+    expect(new Orchestrator(baseConfig({ max_concurrency: -2 })).maxConcurrency).toBe(cpuAuto());
+    expect(new Orchestrator(baseConfig({ max_concurrency: 'lots' })).maxConcurrency).toBe(cpuAuto());
+  });
+
+  test('legacy array config still resolves a cap (no max_concurrency key available)', () => {
+    const orch = new Orchestrator([{ command: 'a' }, { command: 'b' }]);
+    expect(orch.maxConcurrency).toBe(cpuAuto());
+  });
+});
+
+describe('Orchestrator._runWithConcurrency', () => {
+  test('preserves input order in the results array', async () => {
+    const orch = new Orchestrator(baseConfig());
+    const items = [10, 20, 30, 40];
+    const out = await orch._runWithConcurrency(items, 2, async (n) => n * 2);
+    expect(out).toEqual([20, 40, 60, 80]);
+  });
+
+  test('never exceeds the concurrency limit and still runs every item', async () => {
+    const orch = new Orchestrator(baseConfig());
+    let inFlight = 0;
+    let peak = 0;
+    const limit = 3;
+    const items = Array.from({ length: 12 }, (_, i) => i);
+    const worker = async (i) => {
+      inFlight += 1;
+      peak = Math.max(peak, inFlight);
+      // Yield a couple of microtasks so overlap actually happens.
+      await Promise.resolve();
+      await Promise.resolve();
+      inFlight -= 1;
+      return i;
+    };
+    const out = await orch._runWithConcurrency(items, limit, worker);
+    expect(out).toEqual(items);
+    expect(peak).toBeLessThanOrEqual(limit);
+    expect(peak).toBeGreaterThan(1); // proves it actually parallelised
+  });
+
+  test('limit >= item count behaves like Promise.all (all start immediately)', async () => {
+    const orch = new Orchestrator(baseConfig());
+    let inFlight = 0;
+    let peak = 0;
+    const items = [1, 2, 3];
+    await orch._runWithConcurrency(items, 10, async (n) => {
+      inFlight += 1;
+      peak = Math.max(peak, inFlight);
+      await Promise.resolve();
+      inFlight -= 1;
+      return n;
+    });
+    expect(peak).toBe(items.length);
+  });
+
+  test('empty input resolves to an empty array', async () => {
+    const orch = new Orchestrator(baseConfig());
+    await expect(orch._runWithConcurrency([], 4, async (n) => n)).resolves.toEqual([]);
+  });
+});
+
+describe('Orchestrator._phaseConcurrency (per-phase override)', () => {
+  test('falls back to the global cap when a phase has no max_concurrency', () => {
+    const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
+    expect(orch._phaseConcurrency({ name: 'p', parallel: [] })).toBe(5);
+  });
+
+  test('a phase max_concurrency overrides the global cap (both directions)', () => {
+    const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
+    expect(orch._phaseConcurrency({ name: 'serial', max_concurrency: 1 })).toBe(1);
+    expect(orch._phaseConcurrency({ name: 'wider', max_concurrency: 8 })).toBe(8);
+  });
+
+  test('a phase\'s \'auto\' resolves to cpuCount - 1 regardless of the global cap', () => {
+    const orch = new Orchestrator(baseConfig({ max_concurrency: 1 }));
+    expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 'auto' })).toBe(cpuAuto());
+  });
+
+  test('an invalid phase value falls back to auto, not to the global cap', () => {
+    const orch = new Orchestrator(baseConfig({ max_concurrency: 5 }));
+    expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 0 })).toBe(cpuAuto());
+    expect(orch._phaseConcurrency({ name: 'p', max_concurrency: 'lots' })).toBe(cpuAuto());
+  });
+});

package/lib/orchestrator.js

@@ -1,4 +1,5 @@
 import fs from 'fs';
+import os from 'os';
 import path from 'path';
 import { spawn, spawnSync } from 'child_process';
 import { processManager } from './process-manager.js';
@@ -28,6 +29,18 @@ export class Orchestrator {
     this.sequential = sequential;
     this.force = force;
     this.metrics = Array.isArray(metrics) ? metrics : [];
+    // Maximum number of commands a phase runs concurrently. Without this, a phase fires every
+    // enabled command at once (Promise.all) — fine on a big CI box, but a smaller machine can't
+    // sustain N heavy toolchains in parallel. `max_concurrency` caps the in-flight count:
+    //   - 'auto' (default) -> max(1, cpuCount - 1)
+    //   - a positive integer -> that exact cap
+    //   - 0 / negative / invalid -> treated as 'auto'
+    // `--sequential` still wins (it pins the cap to 1). When the cap >= a phase's command count the
+    // behaviour is identical to the old unbounded Promise.all, so big machines see no change.
+    // CLI `--max-concurrency` overrides the config value (wired in index.js).
+    this.maxConcurrency = this._resolveMaxConcurrency(
+      config && !Array.isArray(config) ? config.max_concurrency : undefined,
+    );
     // Global command prefix. Defaults to 'npm run' so existing configs keep working.
     // Set `command_prefix` to '' / false / null in the config to run commands verbatim
     // as regular shell commands. Per-command `shell: true` or `prefix` overrides this.
@@ -98,6 +111,52 @@ export class Orchestrator {
     return [];
   }
 
+  // Resolve a configured/CLI max_concurrency value to a concrete positive integer cap.
+  // 'auto' (or anything unparseable / <= 0) maps to max(1, cpuCount - 1); a positive number is
+  // floored and used verbatim. Kept side-effect free so it can be reused by the CLI override.
+  _resolveMaxConcurrency(value) {
+    const auto = Math.max(1, os.cpus().length - 1);
+    if (value == null || value === 'auto') return auto;
+    const n = Math.floor(Number(value));
+    if (!Number.isFinite(n) || n <= 0) return auto;
+    return n;
+  }
+
+  // Concurrency cap for a single phase. A phase may pin its own `max_concurrency` to
+  // run its commands at a different in-flight count than the rest of the run — e.g. a
+  // heavy phase whose commands share one resource (a single dev server, a GPU) sets
+  // `max_concurrency: 1` to serialise itself while every other phase keeps the global
+  // cap and its parallelism. Resolved the same way as the global value, so 'auto' and
+  // invalid entries fall back to cpuCount-1 (not silently to the global cap). Phase
+  // commands still run through the non-breaking parallel path, so a serial phase
+  // continues past a failed command rather than aborting its siblings.
+  _phaseConcurrency(phase) {
+    if (phase && phase.max_concurrency != null) {
+      return this._resolveMaxConcurrency(phase.max_concurrency);
+    }
+    return this.maxConcurrency;
+  }
+
+  // Run `items` through `worker` with at most `limit` in flight at once, preserving result order.
+  // This is the bounded-concurrency replacement for `Promise.all(items.map(worker))`: when
+  // `limit >= items.length` it is behaviourally identical (everything starts immediately), but a
+  // smaller limit keeps only `limit` tasks running and starts the next as each finishes. A worker
+  // that rejects rejects the whole batch (matching Promise.all semantics); executeCommand resolves
+  // rather than throws, so in practice failures surface as falsy results, not rejections.
+  async _runWithConcurrency(items, limit, worker) {
+    const results = new Array(items.length);
+    let next = 0;
+    const runNext = async () => {
+      while (next < items.length) {
+        const current = next++;
+        results[current] = await worker(items[current], current);
+      }
+    };
+    const pool = Array.from({ length: Math.min(limit, items.length) }, () => runNext());
+    await Promise.all(pool);
+    return results;
+  }
+
   _deriveEventsPath(jsonResultsPath) {
     if (!jsonResultsPath || jsonResultsPath === '-') return null;
     return jsonResultsPath.replace(/\.json$/, '') + '-events.ndjson';
@@ -694,16 +753,21 @@ export class Orchestrator {
             }
           }
         } else {
-          const tasks = this.config.map((commandConfig) =>
-            this.executeCommand(commandConfig),
+          const results = await this._runWithConcurrency(
+            this.config,
+            this.maxConcurrency,
+            (commandConfig) => this.executeCommand(commandConfig),
           );
-          const results = await Promise.all(tasks);
           hasFailures = results.some((result) => !result);
         }
       } else if (this.config.phases) {
         // New: Run phases sequentially, commands within phases in parallel or sequential based on flag
         if (this.sequential) {
           this.logger.info('🔄 Running in sequential mode');
+        } else {
+          this.logger.info(
+            `🧮 Max concurrency: ${this.maxConcurrency} (of ${os.cpus().length} CPUs) — commands per phase run at most this many at a time`,
+          );
         }
 
         for (const phase of this.config.phases) {
@@ -766,11 +830,22 @@ export class Orchestrator {
               }
             }
           } else {
-            // Run commands in parallel
-            const tasks = phase.parallel.map((commandConfig) =>
-              this.executeCommand(commandConfig, new Set(), phase.name),
+            // Run commands in parallel, but never more than the cap at once so a smaller
+            // machine isn't asked to host every command's toolchain simultaneously. The
+            // cap is per-phase: a phase may pin its own `max_concurrency` (e.g. force
+            // serial execution of commands that share one resource) without changing the
+            // limit any other phase runs at.
+            const phaseConcurrency = this._phaseConcurrency(phase);
+            if (phaseConcurrency !== this.maxConcurrency) {
+              this.logger.info(
+                `   ↳ phase concurrency: ${phaseConcurrency} (phase "${phase.name}" overrides the ${this.maxConcurrency} default)`,
+              );
+            }
+            results = await this._runWithConcurrency(
+              phase.parallel,
+              phaseConcurrency,
+              (commandConfig) => this.executeCommand(commandConfig, new Set(), phase.name),
             );
-            results = await Promise.all(tasks);
           }
 
           const phaseHasFailures = results.some((result) => !result);

package/lib/report-html.js

@@ -152,10 +152,12 @@ function renderCell(key, c, ctx) {
     return v != null ? durCellWrap(v, ctx.maxDuration, ctx.durationHeatThresholds) : '—';
   case 'memoryKb':
     return v != null ? memCellWrap(v, ctx.maxMemory, ctx.heatThresholds) : '—';
-  case 'logFile':
+  case 'logFile': {
+    const base = ctx.repoRoot || process.cwd();
     return v
-      ? `<a class="logref" href="file://${escapeHtml(path.resolve(process.cwd(), v))}" title="${escapeHtml(v)}"><code>${escapeHtml(v)}</code></a>`
+      ? `<a class="logref" href="file://${escapeHtml(path.resolve(base, v))}" title="${escapeHtml(v)}"><code>${escapeHtml(v)}</code></a>`
       : '—';
+  }
   default:
     if (v == null) return '—';
     return escapeHtml(typeof v === 'object' ? JSON.stringify(v) : v);
@@ -356,6 +358,7 @@ export function renderReportHtml(payload) {
   const ctx = {
     maxDuration: Math.max(0, ...allCommands.map((c) => c.durationMs || 0)),
     maxMemory: Math.max(0, ...allCommands.map((c) => c.memoryKb || 0)),
+    repoRoot: payload.repoRoot,
     // Heat thresholds travel in the payload so re-rendering a saved JSON (--render) honours them.
     heatThresholds: normalizeHeatThresholds(payload.memoryHeat),
     durationHeatThresholds: normalizeHeatThresholds(payload.durationHeat),

package/lib/workspaces.js

@@ -247,12 +247,26 @@ export function aggregateWorkspacesReport(options = {}) {
   const fanoutOk = fanoutCommands.length === 0 ? null : fanoutCommands.every((c) => c.success !== false);
 
   // Global quality checks section (commands run by the root orchestrator itself).
+  // Classify from the section's OWN commands, not the report-wide inProgress flag: the global
+  // checks finish before the workspace fan-out, so they must read OK as soon as they are done —
+  // even while other workspaces are still RUNNING in an in-progress snapshot.
   const globalCommands = collectGlobalCommands(opts, runStartedAt);
   const globalFailed = globalCommands.filter((c) => c.success === false).length;
+  const globalRunning = globalCommands.some((c) => c.success == null);
+  const globalState =
+    globalCommands.length === 0
+      ? { statusKind: 'muted', statusLabel: 'PENDING', success: null }
+      : globalRunning
+        ? { statusKind: 'running', statusLabel: 'RUNNING', success: null }
+        : globalFailed > 0
+          ? { statusKind: 'fail', statusLabel: 'FAIL', success: false }
+          : { statusKind: 'ok', statusLabel: 'OK', success: true };
   const sections = [
     {
       title: 'Global quality checks',
-      success: inProgress ? null : globalFailed === 0,
+      statusKind: globalState.statusKind,
+      statusLabel: globalState.statusLabel,
+      success: globalState.success,
       commands: globalCommands,
     },
   ];
@@ -277,15 +291,18 @@ export function aggregateWorkspacesReport(options = {}) {
     const st = classifyWorkspace({ noOp, exists, stale, success, inProgress, fanoutOk });
     if (st.state === 'fail' || st.state === 'interrupted') anyWorkspaceBad = true;
 
-    const showCommands = st.state === 'ok' || st.state === 'fail' || st.state === 'cached';
-    const commands = showCommands ? commandsWithRepoRelLogs(payload.commands, relDir) : [];
+    const showCommands =
+      st.state === 'ok' || st.state === 'fail' || st.state === 'cached' || st.state === 'running';
+    const commands = showCommands ? commandsWithRepoRelLogs(payload?.commands, relDir) : [];
 
     const note =
       commands.length === 0 && st.state === 'pending'
         ? 'Queued — not started yet this run'
         : commands.length === 0 && st.state === 'stale'
           ? 'No results for this run window (stale snapshot)'
-          : null;
+          : st.state === 'running' && commands.length > 0
+            ? 'In progress — partial command list'
+            : null;
 
     sections.push({
       title: name,
@@ -297,7 +314,7 @@ export function aggregateWorkspacesReport(options = {}) {
           : st.state === 'fail'
             ? false
             : null,
-      overallDurationMs: showCommands ? payload.overallDurationMs : undefined,
+      overallDurationMs: showCommands ? payload?.overallDurationMs : undefined,
       meta: { path: relDir, ...(note ? { note } : {}) },
       commands,
     });
@@ -306,12 +323,18 @@ export function aggregateWorkspacesReport(options = {}) {
   const rootOk = rootJson ? rootJson.success !== false : true;
   const success = inProgress ? null : rootOk && globalFailed === 0 && !anyWorkspaceBad;
 
+  const overallDurationMs =
+    inProgress && runStartedAt != null
+      ? Math.max(0, Date.now() - runStartedAt)
+      : rootJson?.overallDurationMs;
+
   return {
     title: opts.title,
     success,
     timestamp: new Date().toISOString(),
-    overallDurationMs: rootJson?.overallDurationMs,
+    overallDurationMs,
     inProgress,
+    repoRoot: toPosix(opts.repoRoot),
     sections,
     // Carry heat thresholds into the aggregate payload so the renderer honours them.
     ...(opts.memoryHeat ? { memoryHeat: opts.memoryHeat } : {}),

package/lib/workspaces.test.js

@@ -142,6 +142,7 @@ describe('aggregateWorkspacesReport', () => {
     expect(report.success).toBe(false);
     const global = report.sections.find((s) => s.title === 'Global quality checks');
     expect(global.success).toBe(false);
+    expect(global.statusLabel).toBe('FAIL');
   });
 
   test('in-progress run: success:null workspace is RUNNING, missing is PENDING', () => {
@@ -155,10 +156,45 @@ describe('aggregateWorkspacesReport', () => {
     const report = aggregateWorkspacesReport({ repoRoot });
     expect(report.inProgress).toBe(true);
     expect(report.success).toBeNull();
-    expect(report.sections.find((s) => s.title === 'apps/web').statusLabel).toBe('RUNNING');
+    const web = report.sections.find((s) => s.title === 'apps/web');
+    expect(web.statusLabel).toBe('RUNNING');
+    expect(web.commands).toHaveLength(1);
+    expect(web.commands[0].command).toBe('test');
+    expect(web.meta.note).toMatch(/partial command list/i);
     expect(report.sections.find((s) => s.title === 'packages/a').statusLabel).toBe('PENDING');
   });
 
+  test('global checks read OK as soon as they finish, even mid-run while a workspace is RUNNING', () => {
+    markRunning();
+    rootResults([{ command: 'lint', phase: 'global quality checks', success: true }], null);
+    makeWorkspace('apps/web', {
+      results: { success: null, timestamp: fresh(), commands: [{ command: 'test', success: null }] },
+    });
+
+    const report = aggregateWorkspacesReport({ repoRoot });
+    expect(report.inProgress).toBe(true);
+    const global = report.sections.find((s) => s.title === 'Global quality checks');
+    expect(global.statusLabel).toBe('OK');
+    expect(global.success).toBe(true);
+  });
+
+  test('global section is RUNNING while one of its own commands is still in flight', () => {
+    markRunning();
+    rootResults(
+      [
+        { command: 'lint', phase: 'global quality checks', success: true },
+        { command: 'i18n', phase: 'global quality checks', success: null, startedAt: fresh() },
+      ],
+      null,
+    );
+    makeWorkspace('apps/web'); // no results yet
+
+    const report = aggregateWorkspacesReport({ repoRoot });
+    const global = report.sections.find((s) => s.title === 'Global quality checks');
+    expect(global.statusLabel).toBe('RUNNING');
+    expect(global.success).toBeNull();
+  });
+
   test('after the run, a success:null results file reads as INTERRUPTED, not running', () => {
     rootResults([{ command: 'lint', phase: 'global quality checks', success: true }]);
     makeWorkspace('apps/web', {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "scripts-orchestrator",
-  "version": "3.5.0",
+  "version": "3.7.0",
   "description": "A powerful script orchestrator for running parallel commands with dependency management, background processes, and health checks",
   "main": "lib/index.js",
   "type": "module",