@omni-oss/task-bench 0.0.0-beta.1

package/README.md

@@ -0,0 +1,328 @@
+# @omni-oss/task-bench
+
+Generate configurable, minimal test workspaces and benchmark **task-execution
+overhead** across [omni](https://github.com/omni-oss/omni),
+[Turborepo](https://turbo.build), [Nx](https://nx.dev), and
+[moonrepo](https://moonrepo.dev).
+
+The goal is to isolate the cost of **project discovery** and **task caching**
+rather than the tasks themselves. Every generated task is fast, deterministic,
+and produces real output (logs + files) so each runner's cache and log-capture
+machinery is exercised. The four runners are configured as equivalently as
+possible from a single source of truth.
+
+## How it works
+
+For a given config the harness generates one workspace on disk containing:
+
+- `packages/<prefix><n>/` — minimal JS projects, each with:
+  - `package.json` (scripts + `workspace:*` deps encoding the project graph),
+  - `src/index.js` (a cache input),
+  - `task.mjs` (the shared, deterministic task runner),
+  - `project.omni.yaml` (omni), `project.json` (nx), and `moon.yml` (moon).
+- Root `workspace.omni.yaml`, `turbo.json`, `nx.json`, and `.moon/*.yml`
+  describing the same task graph (`tN` depends on `t(N-1)` and/or `^tN`) with
+  identical inputs (`package.json`, `task.mjs`, `src/**`) and outputs
+  (`dist/tN.*`). The workspace is also `git init`-ed and committed, which moon
+  requires to enable its cache (and is harmless/realistic for the others).
+
+It then benchmarks each enabled tool in two scenarios:
+
+- **cold** — caches + outputs wiped before each run (discovery + full exec).
+- **warm** — caches primed, so every task should be a cache hit. This is the
+  key metric for discovery + cache-restore overhead.
+
+### Fairness & correctness
+
+- **Verified cache hits.** Each task appends one line to an out-of-tree log
+  *only when it actually executes* (cache hits skip the process entirely).
+  The harness counts these to report a real, tool-agnostic cache-hit rate per
+  run, so "warm == all cached" is verified rather than assumed. A warm run that
+  is not 100% cached is flagged in the report. (Turbo's strict env mode is
+  handled via `globalPassThroughEnv` so the marker survives without affecting
+  cache keys.)
+- **Concurrency parity.** The same max-parallelism is passed to every runner
+  (`omni -c`, `turbo --concurrency`, `nx --parallel`); defaults to the CPU
+  count. Use `--concurrency` to pin it.
+- **Daemon handling.** By default each tool is allowed to use its persistent
+  daemon (Turbo's `turbod`, the Nx daemon) so warm runs reflect each tool's
+  best incremental performance; the daemon is warmed by an unmeasured prime run
+  and left alive across warm runs. Cold runs deliberately tear the daemon down
+  (and clear caches) so they include cold-start cost. Daemons are always
+  stopped as cleanup afterwards so nothing leaks. Pass `--no-daemon` to disable
+  daemons entirely (Turbo `--no-daemon`, `NX_DAEMON=false`); omni and moon have
+  no daemon.
+- **Statistics.** Reports median ± standard deviation; full per-run samples
+  (duration, exit code, executed-task count) are written to the JSON output.
+
+## CLI
+
+```sh
+# Generate a workspace only.
+task-bench generate -o /tmp/bench --projects 200 --tasks 4 --strategy layered
+
+# Generate, `bun install`, benchmark, and print a report.
+task-bench bench -o /tmp/bench --projects 200 --tasks 4 \
+    --strategy random --edge-probability 0.3 \
+    --cold-runs 3 --warm-runs 5 --json results.json
+
+# Benchmark an already-generated workspace.
+task-bench run -d /tmp/bench --tools omni,turbo --task t3
+
+# Run a whole preset of scenarios and summarize (see "Comprehensive suite").
+task-bench suite --preset full --json suite.json --md suite.md
+
+# Resolve a config and print it (no side effects).
+task-bench inspect --projects 50 --strategy chain
+```
+
+### Key options
+
+| Option | Description |
+| --- | --- |
+| `-o, --out <dir>` | Root dir to generate the workspace into. |
+| `--projects <n>` | Number of projects. |
+| `--tasks <n>` | Tasks per project (`t0..tN-1`). |
+| `--strategy <s>` | `isolated`, `chain`, `fan-out`, `layered`, or `random`. |
+| `--layers <n>` | Layers for the `layered` strategy. |
+| `--fanout <n>` | Max upstream deps per project. |
+| `--edge-probability <p>` | Edge probability for the `random` strategy. |
+| `--log-lines <n>` | Log lines printed per task. |
+| `--work <n>` | CPU work iterations per task. |
+| `--output-files <n>` | Output files written per task. |
+| `--tools <list>` | Comma-separated `omni,turbo,nx,moon`. |
+| `--turbo-version` / `--nx-version` / `--moon-version` / `--bun-version` | Pin the version of each tool to install. |
+| `--concurrency <n>` | Max parallel tasks, applied identically to every runner (default: CPU count). |
+| `--no-daemon` | Disable each tool's persistent daemon (Turbo, Nx). |
+| `--no-chain` / `--no-fan-upstream` | Disable intra/inter-project task deps. |
+| `--config <file>` | JSON config to use as a base for overrides. |
+
+## Library
+
+```ts
+import { generateWorkspace, runBenchmark, formatReport } from "@omni-oss/task-bench";
+
+await generateWorkspace("/tmp/bench", {
+    projects: 100,
+    tasksPerProject: 3,
+    dependency: { strategy: "layered", layers: 6, fanout: 3 },
+});
+
+const result = await runBenchmark("/tmp/bench", { warmRuns: 5 });
+console.log(formatReport(result));
+```
+
+## Comprehensive benchmark suite
+
+The `suite` command runs a **preset of scenarios** end-to-end (generate →
+install → benchmark each) and summarizes them into a combined report, with
+`--json` and/or `--md` output.
+
+```sh
+# List the built-in presets.
+task-bench suite --list
+#   quick | shapes | scale | density | daemon | full
+
+# Run a built-in preset and write both machine- and human-readable reports.
+task-bench suite --preset full --json suite.json --md suite.md
+
+# Override run parameters for every scenario in the preset.
+task-bench suite --preset scale --cold-runs 3 --warm-runs 5 --concurrency 8
+
+# Run only two tools, and keep the generated workspaces around.
+task-bench suite --preset shapes --tools omni,turbo --keep -o /tmp/suite
+
+# Run a custom scenario matrix from a JSON file.
+task-bench suite --file my-suite.json --md report.md
+```
+
+Built-in presets: **`quick`** (tiny smoke test), **`shapes`** (dependency-graph
+shapes at fixed scale), **`scale`** (50–600 projects), **`density`** (tasks per
+project), **`daemon`** (daemon on vs off), and **`full`** (shapes + scale +
+density). A custom preset file looks like:
+
+```jsonc
+{
+  "name": "my sweep",
+  "defaults": {
+    "config": { "tasksPerProject": 3, "dependency": { "strategy": "layered" } },
+    "run": { "concurrency": 8, "coldRuns": 2, "warmRuns": 3 }
+  },
+  "scenarios": [
+    { "name": "small", "config": { "projects": 50 } },
+    { "name": "large", "config": { "projects": 500 } },
+    { "name": "large-nodaemon", "config": { "projects": 500 }, "run": { "daemon": false } }
+  ]
+}
+```
+
+The Markdown report contains two summary matrices (warm + cold median per tool
+per scenario) followed by the detailed per-scenario tables; the JSON contains
+the full resolved config, run options, and per-run samples for every scenario.
+
+### Scripting it yourself
+
+If you prefer explicit shell control, the equivalent manual loop is:
+
+````bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Use the built CLI (`task-bench`) or the dev entry as below.
+BENCH="bun scripts/task-bench/src/cli/index.ts"
+OUT=/tmp/task-bench-suite
+RESULTS="$OUT/results"
+SUITE="$OUT/SUITE.md"
+mkdir -p "$RESULTS"
+: > "$SUITE"
+
+run() { # run <label> <bench|run> <args...>
+  local label="$1"; shift
+  echo "### $label" | tee -a "$SUITE"
+  echo '```' >> "$SUITE"
+  $BENCH "$@" --json "$RESULTS/$label.json" 2>/dev/null | tee -a "$SUITE"
+  echo '```' >> "$SUITE"
+}
+
+# 1) Dependency-shape sweep (fixed scale): how graph shape affects overhead.
+for strat in isolated chain fan-out layered random; do
+  run "shape-$strat" bench -o "$OUT/shape-$strat" \
+    --projects 120 --tasks 3 --strategy "$strat" \
+    --concurrency 8 --cold-runs 2 --warm-runs 3
+done
+
+# 2) Scale sweep (layered graph): how overhead grows with graph size.
+for n in 50 150 300 600; do
+  run "scale-$n" bench -o "$OUT/scale-$n" \
+    --projects "$n" --tasks 3 --strategy layered --layers 8 \
+    --concurrency 8 --cold-runs 2 --warm-runs 3
+done
+
+# 3) Task-density sweep (fixed project count).
+for t in 2 5 10; do
+  run "density-$t" bench -o "$OUT/density-$t" \
+    --projects 120 --tasks "$t" --strategy layered \
+    --concurrency 8 --cold-runs 2 --warm-runs 3
+done
+
+# 4) Daemon on vs off (reuse one installed workspace).
+run "daemon-on"  bench -o "$OUT/daemon" --projects 200 --tasks 3 \
+  --strategy layered --concurrency 8 --cold-runs 2 --warm-runs 3
+run "daemon-off" run  -d "$OUT/daemon" --no-daemon \
+  --concurrency 8 --cold-runs 2 --warm-runs 3
+
+echo "Suite written to $SUITE ; per-scenario JSON in $RESULTS"
+# rm -rf "$OUT"   # uncomment to clean up workspaces afterwards
+````
+
+Each JSON result contains per-run samples (`durationMs`, `exitCode`,
+`executed`), the verified `taskGraphSize`, and `stats` (min/max/mean/median/
+stddev) for both scenarios, so you can post-process or chart them however you
+like.
+
+### Sample results
+
+Collected on one machine (Linux, 8-way concurrency, 3 tasks/project, layered
+unless noted; `warm` = verified 100% cache hit, the discovery + cache-restore
+overhead metric). Absolute numbers are hardware-dependent — read the *ratios*.
+
+**Four-runner snapshot** (40 projects × 3 tasks, 120 nodes, layered):
+
+| tool | warm | cold |
+| --- | --- | --- |
+| omni | **76ms** | 990ms |
+| turbo | 135ms | 915ms |
+| nx | 434ms | 2.27s |
+| moon | 448ms | 1.45s |
+
+The larger sweeps below predate the moon addition (omni/turbo/nx only); rerun
+`task-bench suite --preset full` to regenerate them with all four runners.
+
+<!-- SUITE_RESULTS -->
+
+**Dependency-shape sweep** (120 projects × 3 tasks, 360 task-graph nodes):
+
+| strategy | omni warm | turbo warm | nx warm | omni cold | turbo cold | nx cold |
+| --- | --- | --- | --- | --- | --- | --- |
+| isolated | 292ms | **239ms** | 644ms | 3.27s | 2.52s | 8.22s |
+| chain    | 403ms | **231ms** | 1.09s | 5.36s | 4.91s | 9.09s |
+| fan-out  | 304ms | **246ms** | 678ms | 3.46s | 2.61s | 8.79s |
+| layered  | 252ms | **246ms** | 678ms | 3.21s | 2.63s | 8.71s |
+| random   | 284ms | **254ms** | 797ms | 3.59s | 3.12s | 8.72s |
+
+**Scale sweep** (layered, 3 tasks):
+
+| projects (nodes) | omni warm | turbo warm | nx warm | omni cold | turbo cold | nx cold |
+| --- | --- | --- | --- | --- | --- | --- |
+| 50 (150)  | **106ms** | 134ms | 478ms | 1.34s | 1.14s | 2.81s |
+| 150 (450) | 295ms | **287ms** | 764ms | 4.05s | 3.25s | 12.23s |
+| 300 (900) | 715ms | **523ms** | 1.17s | 8.62s | 6.44s | 38.68s |
+
+**Daemon on vs off** (200 projects × 3 tasks, 600 nodes):
+
+| mode | omni warm | turbo warm | nx warm | omni cold | turbo cold | nx cold |
+| --- | --- | --- | --- | --- | --- | --- |
+| daemon on  | 454ms | **361ms** | 915ms | 5.60s | 4.31s | 19.50s |
+| daemon off | 455ms | **359ms** | 924ms | 5.62s | 4.32s | 15.27s |
+
+**Observations**
+
+- **Warm overhead:** omni and turbo are close (turbo pulls slightly ahead as
+  scale grows); nx is consistently **~2.3–4.7× slower** on warm cache hits. At
+  small scale (50 projects) omni is fastest; from ~120 projects up turbo leads.
+- **Graph shape matters most for cold:** the `chain` strategy (deep, low
+  parallelism) roughly doubles cold time for omni/turbo and gives nx its worst
+  warm number, while `isolated`/`layered` (wide, parallel) are cheapest.
+- **Scale:** warm overhead grows roughly linearly with task-graph size for all
+  three; nx **cold** scales worst (38.7s at 300 projects vs 6–9s for the
+  others).
+- **Daemons:** on/off barely moves omni/turbo (omni has none; turbo warm is
+  unchanged) and nx warm is essentially flat — but nx **cold is faster with the
+  daemon off** (~15s vs ~19s) because daemon-mode cold pays a teardown/restart
+  each run. So nx's daemon does not materially reduce the per-invocation warm
+  overhead this benchmark isolates.
+
+> Reproduce with the script above; exact numbers vary by machine, the relative
+> ratios are the takeaway.
+
+## Tool adapters
+
+Each runner is a self-contained adapter (`src/tools/<tool>.ts`) that owns
+*everything* tool-specific, so the generator and the other tools stay
+decoupled. An adapter declares:
+
+- `supportedVersions` — semver ranges it supports. The version to install is
+  configurable (`versions.<tool>` / `--<tool>-version`); if it falls outside
+  the supported ranges the harness fails fast with a clear error. External
+  tools (omni) instead implement `detectVersion()` and validate the installed
+  binary.
+- `devDependencies(config)` — the npm packages it needs in the root
+  `package.json`.
+- `setup(ctx)` — derives and writes its own config from the generated
+  `projects` (e.g. `turbo.json`, `nx.json` + `project.json`, `.moon/*.yml` +
+  `moon.yml`, `workspace.omni.yaml` + `project.omni.yaml`).
+- `run` / `env` / `clearCaches` / `stopDaemon` — the runtime behavior.
+
+The generator only writes the neutral workspace (projects, `src/index.js`,
+`task.mjs`), then hands the project list to each enabled adapter. Adding a new
+runner is a single new file plus registering it in `src/tools/index.ts`.
+
+## Notes
+
+- The dependency graph is deterministic for a given `seed`.
+- Generated workspaces are self-contained and safe to place anywhere outside
+  this repository (each has its own `workspace.omni.yaml`).
+
+### Known limitations
+
+- Numbers are wall-clock process timings on a single machine; absolute values
+  vary by hardware, but the harness isolates *relative* discovery/caching
+  overhead.
+- Cold runs are fully fresh per tool (caches cleared; daemons torn down in
+  daemon mode), so cold includes each tool's start-up cost. Warm runs are the
+  primary overhead metric.
+- Task output is captured (piped) for all runners equally so every tool's
+  log-capture path is exercised.
+- moon only enables its cache inside a git repository, so generated workspaces
+  are committed automatically; benchmarking an existing workspace assumes it is
+  already a git repo.

package/dist/bench/index.d.ts

@@ -0,0 +1,82 @@
+import { Tool } from '../config';
+import { Stats } from './stats';
+export interface RunSample {
+    durationMs: number;
+    exitCode: number;
+    stdout: string;
+    stderr: string;
+    /** Number of tasks that actually executed (0 == a full cache hit). */
+    executed: number;
+    ok: boolean;
+}
+export interface ScenarioResult {
+    runs: number;
+    failures: number;
+    stats: Stats;
+    /** Median number of tasks that actually executed across the runs. */
+    executedMedian: number;
+}
+export interface ToolResult {
+    tool: Tool;
+    task: string;
+    /** Size of the executed task graph (tasks run on a cold, uncached run). */
+    taskGraphSize: number;
+    cold: ScenarioResult;
+    warm: ScenarioResult;
+    error?: string;
+}
+export interface BenchmarkResult {
+    rootDir: string;
+    task: string;
+    projects: number;
+    tasksPerProject: number;
+    concurrency: number;
+    daemon: boolean;
+    /** Resolved version of each benchmarked tool (detected for omni). */
+    versions: Record<string, string | null>;
+    generatedAt: string;
+    tools: ToolResult[];
+}
+export type BenchEvent = {
+    kind: "tool-start";
+    tool: Tool;
+} | {
+    kind: "scenario";
+    tool: Tool;
+    scenario: "cold" | "warm";
+    run: number;
+    total: number;
+    sample: RunSample;
+} | {
+    kind: "tool-error";
+    tool: Tool;
+    error: string;
+} | {
+    kind: "tool-unsuccessful";
+    tool: Tool;
+    sample: RunSample;
+};
+export interface RunBenchmarkOptions {
+    tools?: Tool[] | undefined;
+    task?: string | undefined;
+    coldRuns?: number | undefined;
+    warmRuns?: number | undefined;
+    concurrency?: number | undefined;
+    /** Allow each tool's persistent daemon (default true). */
+    daemon?: boolean | undefined;
+    onEvent?: ((event: BenchEvent) => void) | undefined;
+}
+/**
+ * Benchmark the enabled tools against an already-generated workspace.
+ *
+ * For each tool we measure two scenarios:
+ *   - cold: caches + outputs wiped before every run (discovery + full exec + cache write)
+ *   - warm: caches primed, so ideally every task is a cache hit
+ *           (isolates discovery + cache-restore overhead)
+ *
+ * A tool-agnostic execution counter (see the generated task runner) records how
+ * many tasks actually ran, so warm-run cache effectiveness is *verified* rather
+ * than assumed. Concurrency is pinned identically across all runners.
+ */
+export declare function runBenchmark(rootDir: string, options?: RunBenchmarkOptions): Promise<BenchmarkResult>;
+//# sourceMappingURL=index.d.ts.map

package/dist/bench/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/bench/index.ts"],"names":[],"mappings":"AAMA,OAAO,EAAuB,KAAK,IAAI,EAAE,MAAM,WAAW,CAAC;AAG3D,OAAO,EAAgB,KAAK,KAAK,EAAE,MAAM,SAAS,CAAC;AAEnD,MAAM,WAAW,SAAS;IACtB,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,sEAAsE;IACtE,QAAQ,EAAE,MAAM,CAAC;IACjB,EAAE,EAAE,OAAO,CAAC;CACf;AAED,MAAM,WAAW,cAAc;IAC3B,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,KAAK,CAAC;IACb,qEAAqE;IACrE,cAAc,EAAE,MAAM,CAAC;CAC1B;AAED,MAAM,WAAW,UAAU;IACvB,IAAI,EAAE,IAAI,CAAC;IACX,IAAI,EAAE,MAAM,CAAC;IACb,2EAA2E;IAC3E,aAAa,EAAE,MAAM,CAAC;IACtB,IAAI,EAAE,cAAc,CAAC;IACrB,IAAI,EAAE,cAAc,CAAC;IACrB,KAAK,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,eAAe;IAC5B,OAAO,EAAE,MAAM,CAAC;IAChB,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,MAAM,CAAC;IACjB,eAAe,EAAE,MAAM,CAAC;IACxB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,EAAE,OAAO,CAAC;IAChB,qEAAqE;IACrE,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,IAAI,CAAC,CAAC;IACxC,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,UAAU,EAAE,CAAC;CACvB;AAED,MAAM,MAAM,UAAU,GAChB;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,IAAI,EAAE,IAAI,CAAA;CAAE,GAClC;IACI,IAAI,EAAE,UAAU,CAAC;IACjB,IAAI,EAAE,IAAI,CAAC;IACX,QAAQ,EAAE,MAAM,GAAG,MAAM,CAAC;IAC1B,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,SAAS,CAAC;CACrB,GACD;IAAE,IAAI,EAAE,YAAY,CAAC;IAAC,IAAI,EAAE,IAAI,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,GACjD;IACI,IAAI,EAAE,mBAAmB,CAAC;IAC1B,IAAI,EAAE,IAAI,CAAC;IACX,MAAM,EAAE,SAAS,CAAC;CACrB,CAAC;AAER,MAAM,WAAW,mBAAmB;IAChC,KAAK,CAAC,EAAE,IAAI,EAAE,GAAG,SAAS,CAAC;IAC3B,IAAI,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC1B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9B,WAAW,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjC,0DAA0D;IAC1D,MAAM,CAAC,EAAE,OAAO,GAAG,SAAS,CAAC;IAC7B,OAAO,CAAC,EAAE,CAAC,CAAC,KAAK,EAAE,UAAU,KAAK,IAAI,CAAC,GAAG,SAAS,CAAC;CACvD;AAuED;;;;;;;;;;;GAWG;AACH,wBAAsB,YAAY,CAC9B,OAAO,EAAE,MAAM,EACf,OAAO,GAAE,mBAAwB,GAClC,OAAO,CAAC,eAAe,CAAC,CA8J1B"}

package/dist/bench/install.d.ts

@@ -0,0 +1,5 @@
+/** Install dependencies in a generated workspace with bun. */
+export declare function installWorkspace(dir: string, opts?: {
+    quiet?: boolean;
+}): Promise<void>;
+//# sourceMappingURL=install.d.ts.map

package/dist/bench/install.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"install.d.ts","sourceRoot":"","sources":["../../src/bench/install.ts"],"names":[],"mappings":"AAEA,8DAA8D;AAC9D,wBAAsB,gBAAgB,CAClC,GAAG,EAAE,MAAM,EACX,IAAI,GAAE;IAAE,KAAK,CAAC,EAAE,OAAO,CAAA;CAAO,GAC/B,OAAO,CAAC,IAAI,CAAC,CAKf"}

package/dist/bench/report.d.ts

@@ -0,0 +1,9 @@
+import { BenchmarkResult } from './index';
+/**
+ * Render a human-readable comparison table plus a short takeaway. The warm
+ * column is the key metric: with all tasks cached it approximates each tool's
+ * discovery + cache-restore overhead. The warm-cache-hit column verifies that
+ * assumption held (should be 100%).
+ */
+export declare function formatReport(result: BenchmarkResult): string;
+//# sourceMappingURL=report.d.ts.map

package/dist/bench/report.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"report.d.ts","sourceRoot":"","sources":["../../src/bench/report.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,eAAe,EAAc,MAAM,SAAS,CAAC;AA6B3D;;;;;GAKG;AACH,wBAAgB,YAAY,CAAC,MAAM,EAAE,eAAe,GAAG,MAAM,CAkG5D"}

package/dist/bench/stats.d.ts

@@ -0,0 +1,12 @@
+/** Summary statistics for a set of timing samples (all in milliseconds). */
+export interface Stats {
+    samples: number[];
+    min: number;
+    max: number;
+    mean: number;
+    median: number;
+    stddev: number;
+}
+export declare function computeStats(samples: number[]): Stats;
+export declare function formatMs(ms: number): string;
+//# sourceMappingURL=stats.d.ts.map

package/dist/bench/stats.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"stats.d.ts","sourceRoot":"","sources":["../../src/bench/stats.ts"],"names":[],"mappings":"AAAA,4EAA4E;AAC5E,MAAM,WAAW,KAAK;IAClB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;CAClB;AAED,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,EAAE,GAAG,KAAK,CAsBrD;AAED,wBAAgB,QAAQ,CAAC,EAAE,EAAE,MAAM,GAAG,MAAM,CAG3C"}

package/dist/cli/index.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+export {};
+//# sourceMappingURL=index.d.ts.map

package/dist/cli/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/cli/index.ts"],"names":[],"mappings":""}

package/dist/config.d.ts

@@ -0,0 +1,91 @@
+import { z } from 'zod';
+/**
+ * The supported inter-project dependency graph shapes. These control how much
+ * of a task graph a runner has to walk during discovery / scheduling, which is
+ * what we are trying to measure.
+ */
+export declare const DEPENDENCY_STRATEGIES: readonly ["isolated", "chain", "fan-out", "layered", "random"];
+export declare const DependencyStrategySchema: z.ZodEnum<{
+    isolated: "isolated";
+    chain: "chain";
+    "fan-out": "fan-out";
+    layered: "layered";
+    random: "random";
+}>;
+export type DependencyStrategy = z.infer<typeof DependencyStrategySchema>;
+/** The task runners we know how to generate configuration for and benchmark. */
+export declare const TOOLS: readonly ["omni", "turbo", "nx", "moon"];
+export declare const ToolSchema: z.ZodEnum<{
+    omni: "omni";
+    turbo: "turbo";
+    nx: "nx";
+    moon: "moon";
+}>;
+export type Tool = z.infer<typeof ToolSchema>;
+export declare const DependencyConfigSchema: z.ZodPrefault<z.ZodObject<{
+    strategy: z.ZodDefault<z.ZodEnum<{
+        isolated: "isolated";
+        chain: "chain";
+        "fan-out": "fan-out";
+        layered: "layered";
+        random: "random";
+    }>>;
+    layers: z.ZodDefault<z.ZodNumber>;
+    fanout: z.ZodDefault<z.ZodNumber>;
+    edgeProbability: z.ZodDefault<z.ZodNumber>;
+}, z.core.$strip>>;
+export declare const TaskConfigSchema: z.ZodPrefault<z.ZodObject<{
+    logLines: z.ZodDefault<z.ZodNumber>;
+    workIterations: z.ZodDefault<z.ZodNumber>;
+    outputFiles: z.ZodDefault<z.ZodNumber>;
+    chainWithinProject: z.ZodDefault<z.ZodBoolean>;
+    fanUpstream: z.ZodDefault<z.ZodBoolean>;
+}, z.core.$strip>>;
+export declare const VersionsConfigSchema: z.ZodPrefault<z.ZodObject<{
+    turbo: z.ZodDefault<z.ZodString>;
+    nx: z.ZodDefault<z.ZodString>;
+    moon: z.ZodDefault<z.ZodString>;
+    bun: z.ZodDefault<z.ZodString>;
+}, z.core.$strip>>;
+export declare const HarnessConfigSchema: z.ZodObject<{
+    seed: z.ZodDefault<z.ZodNumber>;
+    projectPrefix: z.ZodDefault<z.ZodString>;
+    projects: z.ZodDefault<z.ZodNumber>;
+    tasksPerProject: z.ZodDefault<z.ZodNumber>;
+    dependency: z.ZodPrefault<z.ZodObject<{
+        strategy: z.ZodDefault<z.ZodEnum<{
+            isolated: "isolated";
+            chain: "chain";
+            "fan-out": "fan-out";
+            layered: "layered";
+            random: "random";
+        }>>;
+        layers: z.ZodDefault<z.ZodNumber>;
+        fanout: z.ZodDefault<z.ZodNumber>;
+        edgeProbability: z.ZodDefault<z.ZodNumber>;
+    }, z.core.$strip>>;
+    task: z.ZodPrefault<z.ZodObject<{
+        logLines: z.ZodDefault<z.ZodNumber>;
+        workIterations: z.ZodDefault<z.ZodNumber>;
+        outputFiles: z.ZodDefault<z.ZodNumber>;
+        chainWithinProject: z.ZodDefault<z.ZodBoolean>;
+        fanUpstream: z.ZodDefault<z.ZodBoolean>;
+    }, z.core.$strip>>;
+    tools: z.ZodDefault<z.ZodArray<z.ZodEnum<{
+        omni: "omni";
+        turbo: "turbo";
+        nx: "nx";
+        moon: "moon";
+    }>>>;
+    versions: z.ZodPrefault<z.ZodObject<{
+        turbo: z.ZodDefault<z.ZodString>;
+        nx: z.ZodDefault<z.ZodString>;
+        moon: z.ZodDefault<z.ZodString>;
+        bun: z.ZodDefault<z.ZodString>;
+    }, z.core.$strip>>;
+}, z.core.$strip>;
+export type HarnessConfig = z.infer<typeof HarnessConfigSchema>;
+export type HarnessConfigInput = z.input<typeof HarnessConfigSchema>;
+/** Parse and fill defaults for a (possibly partial) harness config. */
+export declare function resolveConfig(input?: HarnessConfigInput): HarnessConfig;
+//# sourceMappingURL=config.d.ts.map

package/dist/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAExB;;;;GAIG;AACH,eAAO,MAAM,qBAAqB,gEAMxB,CAAC;AAEX,eAAO,MAAM,wBAAwB;;;;;;EAAgC,CAAC;AACtE,MAAM,MAAM,kBAAkB,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,wBAAwB,CAAC,CAAC;AAE1E,gFAAgF;AAChF,eAAO,MAAM,KAAK,0CAA2C,CAAC;AAC9D,eAAO,MAAM,UAAU;;;;;EAAgB,CAAC;AACxC,MAAM,MAAM,IAAI,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,UAAU,CAAC,CAAC;AAE9C,eAAO,MAAM,sBAAsB;;;;;;;;;;;kBA0BlB,CAAC;AAElB,eAAO,MAAM,gBAAgB;;;;;;kBAmCZ,CAAC;AAElB,eAAO,MAAM,oBAAoB;;;;;kBAOhB,CAAC;AAElB,eAAO,MAAM,mBAAmB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;iBAgC9B,CAAC;AAEH,MAAM,MAAM,aAAa,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,mBAAmB,CAAC,CAAC;AAChE,MAAM,MAAM,kBAAkB,GAAG,CAAC,CAAC,KAAK,CAAC,OAAO,mBAAmB,CAAC,CAAC;AAErE,uEAAuE;AACvE,wBAAgB,aAAa,CAAC,KAAK,CAAC,EAAE,kBAAkB,GAAG,aAAa,CAEvE"}

package/dist/generate/index.d.ts

@@ -0,0 +1,19 @@
+import { HarnessConfig, HarnessConfigInput } from '../config';
+import { ProjectNode } from '../graph';
+export interface GenerateResult {
+    rootDir: string;
+    config: HarnessConfig;
+    projects: ProjectNode[];
+    /** Every file written, workspace-relative. */
+    files: string[];
+}
+/**
+ * Generate a complete benchmark workspace at `rootDir`. Existing contents at
+ * `rootDir` are removed first so the workspace is reproducible.
+ *
+ * The generator only produces the neutral, tool-agnostic workspace (projects,
+ * sources, task runners). Each enabled tool then writes its own configuration
+ * via its adapter, keeping the tools fully decoupled.
+ */
+export declare function generateWorkspace(rootDir: string, input?: HarnessConfigInput): Promise<GenerateResult>;
+//# sourceMappingURL=index.d.ts.map

package/dist/generate/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/generate/index.ts"],"names":[],"mappings":"AAGA,OAAO,EACH,KAAK,aAAa,EAClB,KAAK,kBAAkB,EAE1B,MAAM,WAAW,CAAC;AACnB,OAAO,EAAc,KAAK,WAAW,EAAa,MAAM,UAAU,CAAC;AAInE,MAAM,WAAW,cAAc;IAC3B,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,aAAa,CAAC;IACtB,QAAQ,EAAE,WAAW,EAAE,CAAC;IACxB,8CAA8C;IAC9C,KAAK,EAAE,MAAM,EAAE,CAAC;CACnB;AAsHD;;;;;;;GAOG;AACH,wBAAsB,iBAAiB,CACnC,OAAO,EAAE,MAAM,EACf,KAAK,CAAC,EAAE,kBAAkB,GAC3B,OAAO,CAAC,cAAc,CAAC,CAmEzB"}

package/dist/generate/templates.d.ts

@@ -0,0 +1,20 @@
+import { HarnessConfig } from '../config';
+import { ProjectNode } from '../graph';
+/**
+ * The minimal source file each project ships. It exists so that every runner
+ * has a real input file to hash for cache keys.
+ */
+export declare function sourceFile(project: ProjectNode): string;
+/**
+ * The task runner every project uses. It is intentionally tiny but does real
+ * work that exercises each runner's caching + log-capture machinery:
+ *   - reads its own source (a cache input),
+ *   - performs a deterministic CPU loop,
+ *   - prints a configurable number of log lines to stdout,
+ *   - writes deterministic output file(s) into dist/ (cache outputs).
+ *
+ * Determinism matters: identical inputs => identical outputs => cache hits on
+ * warm runs, which is exactly what we want to measure.
+ */
+export declare function taskRunner(config: HarnessConfig, project: ProjectNode): string;
+//# sourceMappingURL=templates.d.ts.map

package/dist/generate/templates.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"templates.d.ts","sourceRoot":"","sources":["../../src/generate/templates.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAC/C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,UAAU,CAAC;AAE5C;;;GAGG;AACH,wBAAgB,UAAU,CAAC,OAAO,EAAE,WAAW,GAAG,MAAM,CASvD;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,UAAU,CACtB,MAAM,EAAE,aAAa,EACrB,OAAO,EAAE,WAAW,GACrB,MAAM,CAsDR"}

package/dist/graph.d.ts

@@ -0,0 +1,21 @@
+import { HarnessConfig } from './config';
+/** A single generated project and the upstream projects it depends on. */
+export interface ProjectNode {
+    /** Zero-based index in the generated set. */
+    index: number;
+    /** Package name, e.g. `bench-p0007`. */
+    name: string;
+    /** Workspace-relative POSIX directory, e.g. `packages/bench-p0007`. */
+    dir: string;
+    /** Indices of upstream projects this project depends on. */
+    dependencies: number[];
+}
+export declare function projectName(config: HarnessConfig, index: number): string;
+/**
+ * Build the full project graph for a config. Dependencies always point to
+ * lower indices, guaranteeing an acyclic graph.
+ */
+export declare function buildGraph(config: HarnessConfig): ProjectNode[];
+/** The task names generated for every project: `t0`, `t1`, ... */
+export declare function taskNames(config: HarnessConfig): string[];
+//# sourceMappingURL=graph.d.ts.map

package/dist/graph.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"graph.d.ts","sourceRoot":"","sources":["../src/graph.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAE9C,0EAA0E;AAC1E,MAAM,WAAW,WAAW;IACxB,6CAA6C;IAC7C,KAAK,EAAE,MAAM,CAAC;IACd,wCAAwC;IACxC,IAAI,EAAE,MAAM,CAAC;IACb,uEAAuE;IACvE,GAAG,EAAE,MAAM,CAAC;IACZ,4DAA4D;IAC5D,YAAY,EAAE,MAAM,EAAE,CAAC;CAC1B;AAmBD,wBAAgB,WAAW,CAAC,MAAM,EAAE,aAAa,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAGxE;AA+DD;;;GAGG;AACH,wBAAgB,UAAU,CAAC,MAAM,EAAE,aAAa,GAAG,WAAW,EAAE,CAa/D;AAED,kEAAkE;AAClE,wBAAgB,SAAS,CAAC,MAAM,EAAE,aAAa,GAAG,MAAM,EAAE,CAEzD"}

package/dist/index.d.ts

@@ -0,0 +1,16 @@
+export * from './bench';
+export { installWorkspace } from './bench/install';
+export { formatReport } from './bench/report';
+export type { Stats } from './bench/stats';
+export { computeStats, formatMs } from './bench/stats';
+export * from './config';
+export type { GenerateResult } from './generate';
+export { generateWorkspace } from './generate';
+export * from './graph';
+export type { RunSuiteOptions, SuiteEvent, SuiteResult, SuiteScenarioResult, } from './suite';
+export { runSuite } from './suite';
+export * from './suite/preset';
+export { formatSuiteMarkdown } from './suite/report';
+export type { GenerationContext, RunInvocation, ToolAdapter, ToolContext, WorkspaceWriter, } from './tools';
+export { assertSupportedVersion, getAdapter, getAdapters, resolveToolVersions, } from './tools';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,SAAS,CAAC;AACxB,OAAO,EAAE,gBAAgB,EAAE,MAAM,iBAAiB,CAAC;AACnD,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC;AAC9C,YAAY,EAAE,KAAK,EAAE,MAAM,eAAe,CAAC;AAC3C,OAAO,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,eAAe,CAAC;AACvD,cAAc,UAAU,CAAC;AACzB,YAAY,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AACjD,OAAO,EAAE,iBAAiB,EAAE,MAAM,YAAY,CAAC;AAC/C,cAAc,SAAS,CAAC;AACxB,YAAY,EACR,eAAe,EACf,UAAU,EACV,WAAW,EACX,mBAAmB,GACtB,MAAM,SAAS,CAAC;AACjB,OAAO,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AACnC,cAAc,gBAAgB,CAAC;AAC/B,OAAO,EAAE,mBAAmB,EAAE,MAAM,gBAAgB,CAAC;AACrD,YAAY,EACR,iBAAiB,EACjB,aAAa,EACb,WAAW,EACX,WAAW,EACX,eAAe,GAClB,MAAM,SAAS,CAAC;AACjB,OAAO,EACH,sBAAsB,EACtB,UAAU,EACV,WAAW,EACX,mBAAmB,GACtB,MAAM,SAAS,CAAC"}