scripts-orchestrator 2.13.0 → 2.15.0

package/README.md

@@ -36,6 +36,11 @@ npm install --save-dev scripts-orchestrator
 - **Optional Phases**: Mark phases as optional and run them selectively
 - **Git-Based Caching**: Automatically skips execution when git state is unchanged
 - **Comprehensive Logging**: Detailed logging of command execution and results
+- **Incremental JSON results**: Live-updating `json_results` file as commands complete (v2.14+)
+- **NDJSON event stream**: Machine-readable per-command events for dashboards (v2.14+)
+- **Post-run hook**: Run a shell command after results are written via `post_run` config (v2.14+)
+- **Run-state file**: Library-owned in-progress indicator for live dashboard integration (v2.14+)
+- **Phase recommendations**: Memory-aware `--recommend` mode that proposes an optimal phase layout from a run's time/memory metrics (advisory, v2.15+)
 
 ## Configuration
 
@@ -410,6 +415,84 @@ export default {
 
 All logs (command logs, main orchestrator logs, and git cache) will be stored in the specified folder.
 
+## Live Dashboard Integration (v2.14+)
+
+### Incremental JSON results
+
+By default `json_results` is written only at the end of a run. From v2.14 onward the file is
+updated atomically (write-to-temp + rename) after **each command starts or completes**, so
+watchers always see a consistent snapshot:
+
+```json
+{
+  "success": null,
+  "timestamp": "2026-06-04T07:13:21.000Z",
+  "commands": [
+    { "command": "lint-ci", "phase": "lint", "success": true, "durationMs": 4200 },
+    { "command": "playwright_ci", "phase": "tests", "success": null, "startedAt": "2026-06-04T07:13:25.000Z" }
+  ]
+}
+```
+
+`"success": null` at the top level is the in-progress sentinel. It is replaced with `true` or
+`false` when `writeJsonResults` writes the final result.
+
+### NDJSON event stream
+
+Alongside `json_results`, the library writes one NDJSON line per event to
+`<json_results_basename>-events.ndjson`:
+
+```jsonl
+{"type":"command_start","timestamp":"...","command":"lint-ci","phase":"lint","scope":"workspace"}
+{"type":"command_end","timestamp":"...","command":"lint-ci","phase":"lint","success":true,"durationMs":4200}
+{"type":"run_end","timestamp":"...","success":true,"durationMs":12800}
+```
+
+Dashboard tools can `tail -f` this file or watch it with `fs.watch` to get real-time updates
+without parsing human-readable log lines.
+
+### Run-state file (A4)
+
+When `--logFolder` is specified, the library writes `{logFolder}/.scripts-orchestrator-run.json`
+at run start and removes it on run end:
+
+```json
+{
+  "startedAt": "2026-06-04T07:13:17.000Z",
+  "pid": 12345,
+  "phase": "tests",
+  "activeCommand": "playwright_ci"
+}
+```
+
+This file is the authoritative in-progress signal for live dashboards. Its absence means the run
+has finished (or never started).
+
+### Post-run hook (A7)
+
+Add `post_run` to your config to run a shell command **after** `json_results` is written and the
+run-state file is cleared, but **before** `process.exit()`:
+
+```javascript
+export default {
+  json_results: './logs/scripts-orchestrator-results.json',
+  post_run: 'node scripts/generate-report.js',  // called after every run
+  phases: [ /* ... */ ]
+};
+```
+
+The hook receives two environment variables:
+- `SCRIPTS_ORCHESTRATOR_SUCCESS=1` (or `0`) — whether the run succeeded
+- `SCRIPTS_ORCHESTRATOR_EXIT_CODE=0` (or `1`) — same, as a numeric exit code
+
+The hook runs synchronously and its exit code is logged but does not change the orchestrator's
+own exit code.
+
+**Typical use case:** trigger a monorepo rollup report after each workspace finishes:
+```javascript
+post_run: 'node ../../scripts/merge-orchestrator-report.js'
+```
+
 ## Git-Based Caching
 
 The orchestrator automatically tracks the git commit hash and repository state to optimize execution:
@@ -440,6 +523,50 @@ This is useful when you want to:
 - Debug issues without modifying the codebase
 - Override the cache in CI/CD pipelines
 
+## Phase Recommendations (advisory)
+
+When a run is executed with `metrics: ['time', 'memory']`, the results JSON records each command's
+`durationMs` and peak `memoryKb`. The `--recommend` mode reads that JSON and prints a **memory-aware
+phase recommendation**: it never runs anything and writes no files.
+
+```bash
+# Analyse an existing results JSON and print a suggested phase layout
+scripts-orchestrator --recommend ./logs/scripts-orchestrator-results.json
+
+# Size the budget for a machine running N gates in parallel (each gets 1/N of RAM and cores)
+scripts-orchestrator --recommend ./logs/scripts-orchestrator-results.json --fanout 3
+
+# Override the memory budget explicitly (MB) or change the RAM safety fraction
+scripts-orchestrator --recommend ./logs/results.json --budget-mb 8192
+scripts-orchestrator --recommend ./logs/results.json --mem-safety 0.7
+```
+
+It reports two things:
+
+1. **Observed timeline** — each phase's wall-clock (the longest step in it) and the concurrent peak
+   memory (Σ of member peaks), flagging any phase whose concurrent peak exceeds the host budget.
+2. **Recommended layout** — a [First-Fit-Decreasing](https://en.wikipedia.org/wiki/Bin_packing_problem)
+   bin-packing by duration that groups steps into sequential phases so each phase's concurrent peak
+   memory stays under `budget = totalmem × memSafety ÷ fanout` and its step count stays under
+   `coreShare = (cores − 2) ÷ fanout`. Long steps seed phases; short steps fill the gaps beneath them,
+   so the estimated makespan stays near the theoretical floor (the single longest step) without
+   oversubscribing RAM.
+
+The same logic is exported for programmatic use:
+
+```js
+import { recommendPhases, formatRecommendationReport } from 'scripts-orchestrator';
+
+const payload = JSON.parse(fs.readFileSync('./logs/results.json', 'utf8'));
+const rec = recommendPhases(payload, { fanout: 3 });
+console.log(formatRecommendationReport(rec));
+// rec.recommended.bins, rec.observed, rec.observedMakespanMs, rec.budgetBytes, …
+```
+
+This is advisory only — the budget is conservative (per-process peaks summed as if they coincide) and
+the packing does not model inter-phase data dependencies, so validate any suggested layout against a
+real run before adopting it.
+
 ## Exit Codes
 
 - `0`: All commands executed successfully
@@ -453,6 +580,7 @@ See [versions](./docs/versions.md)
 - Better UX to indicate what is happening
 - Tests to avoid regression
 - Run any shell command rather than assume the command is specified in package.json (? tentative)
+- Promote the advisory `--recommend` phase recommender into an opt-in automatic scheduler that packs each phase under a per-host memory budget at run time
 
 
 ## Disclaimer

package/index.js

@@ -51,10 +51,87 @@ const argv = yargs(hideBin(process.argv))
     type: 'string',
     description: 'Write HTML report to this path; use "-" for stdout only',
   })
+  .option('render', {
+    type: 'string',
+    description: 'Render an existing results JSON file to HTML (no run). Use with --html-results.',
+  })
+  .option('recommend', {
+    type: 'string',
+    description:
+      'Analyse an existing results JSON and print a memory-aware phase recommendation (R12, no run).',
+  })
+  .option('fanout', {
+    type: 'number',
+    description: 'Workspace fan-out (parallel gates sharing the host) used to size the --recommend budget. Default 1.',
+  })
+  .option('mem-safety', {
+    type: 'number',
+    description: 'Fraction of total RAM the --recommend budget may use (default 0.8).',
+  })
+  .option('budget-mb', {
+    type: 'number',
+    description: 'Override the --recommend memory budget with a fixed value in MB.',
+  })
   .help()
   .alias('h', 'help')
   .parse();
 
+// --render mode: turn an existing results JSON into HTML and exit (no orchestration run).
+// Keeps all HTML rendering in the library so consumers never reimplement it.
+if (argv.render != null) {
+  const { renderReportHtml } = await import('./lib/index.js');
+  const srcPath = path.resolve(process.cwd(), argv.render);
+  if (!fs.existsSync(srcPath)) {
+    log.error(`Error: --render source not found at ${srcPath}`);
+    process.exit(1);
+  }
+  let payload;
+  try {
+    payload = JSON.parse(fs.readFileSync(srcPath, 'utf8'));
+  } catch (err) {
+    log.error(`Error: failed to parse --render JSON: ${err.message}`);
+    process.exit(1);
+  }
+  const html = renderReportHtml(payload);
+  const out = argv.htmlResults ?? null;
+  if (out == null || out === '-') {
+    console.log(html);
+  } else {
+    const outPath = path.resolve(process.cwd(), out);
+    const tmpPath = outPath + '.tmp';
+    fs.mkdirSync(path.dirname(outPath), { recursive: true });
+    fs.writeFileSync(tmpPath, html, 'utf8');
+    fs.renameSync(tmpPath, outPath);
+    log.info(`📄 Rendered ${path.relative(process.cwd(), srcPath)} → ${path.relative(process.cwd(), outPath)}`);
+  }
+  process.exit(0);
+}
+
+// --recommend mode: analyse an existing results JSON and print a memory-aware phase
+// recommendation (R12). Advisory only — no orchestration run, no files written.
+if (argv.recommend != null) {
+  const { recommendPhases, formatRecommendationReport } = await import('./lib/index.js');
+  const srcPath = path.resolve(process.cwd(), argv.recommend);
+  if (!fs.existsSync(srcPath)) {
+    log.error(`Error: --recommend source not found at ${srcPath}`);
+    process.exit(1);
+  }
+  let payload;
+  try {
+    payload = JSON.parse(fs.readFileSync(srcPath, 'utf8'));
+  } catch (err) {
+    log.error(`Error: failed to parse --recommend JSON: ${err.message}`);
+    process.exit(1);
+  }
+  const rec = recommendPhases(payload, {
+    fanout: argv.fanout,
+    memSafety: argv.memSafety,
+    budgetMb: argv.budgetMb,
+  });
+  console.log(formatRecommendationReport(rec, { sourcePath: path.relative(process.cwd(), srcPath) }));
+  process.exit(0);
+}
+
 // Extract arguments
 const args = argv._;
 const configPath = args[0] || './scripts-orchestrator.config.js';
@@ -111,6 +188,16 @@ const htmlResultsPath =
     ? argv.htmlResults
     : (commandsConfig.html_results ?? commandsConfig.html_results_path ?? null);
 
+// A7: post-run hook — shell command run after json_results written
+const postRun = commandsConfig.post_run ?? null;
+
+// Periodic hook — shell command run on an interval WHILE the run is in flight (e.g. to roll up
+// results into an aggregate report). Library owns only the cadence; the command is project-specific.
+const periodicHook = commandsConfig.periodic_hook ?? null;
+const periodicIntervalMs = Number(commandsConfig.periodic_interval_ms) > 0
+  ? Number(commandsConfig.periodic_interval_ms)
+  : 45000;
+
 // Set the log folder for the main orchestrator logs if specified
 if (logFolder) {
   log.setLogFolder(logFolder);
@@ -128,15 +215,23 @@ const orchestrator = new Orchestrator(
   jsonResultsPath,
   htmlResultsPath,
 );
+// A7: wire post-run hook from config
+orchestrator.postRun = postRun;
+// Wire periodic hook (cadence owned by the library)
+orchestrator.periodicHook = periodicHook;
+orchestrator.periodicIntervalMs = periodicIntervalMs;
 
 // Enhanced signal handlers
 const handleSignal = async (signal) => {
   log.warn(`\nReceived ${signal} signal. Cleaning up...`);
+  orchestrator._stopPeriodicHook();
   try {
     await orchestrator.processManager.cleanup();
   } catch (error) {
     log.error(`Cleanup failed: ${error.message}`);
   }
+  // A4: clear run-state so dashboards know the run ended
+  orchestrator._clearRunState();
   process.exit(1);
 };
 

package/lib/index.js

@@ -3,6 +3,28 @@ import { ProcessManager } from './process-manager.js';
 import { HealthCheck } from './health-check.js';
 import { Logger } from './logger.js';
 import { GitCache } from './git-cache.js';
+import { renderReportHtml } from './report-html.js';
+import {
+  recommendPhases,
+  formatRecommendationReport,
+  computeBudget,
+  usableSteps,
+  observedTimeline,
+  packPhases,
+} from './recommend-phases.js';
 
-export { Orchestrator, ProcessManager, HealthCheck, Logger, GitCache };
-export default Orchestrator; 
+export {
+  Orchestrator,
+  ProcessManager,
+  HealthCheck,
+  Logger,
+  GitCache,
+  renderReportHtml,
+  recommendPhases,
+  formatRecommendationReport,
+  computeBudget,
+  usableSteps,
+  observedTimeline,
+  packPhases,
+};
+export default Orchestrator;