ppef 1.2.0 → 1.3.0

package/README.md

@@ -45,11 +45,275 @@ npx tsx --test src/path/to/file.test.ts
 
 CLI (after build):
 ```bash
-ppef run          # Execute experiments
-ppef validate     # Validate configuration
-ppef plan         # Dry-run execution plan
-ppef aggregate    # Post-process results
-ppef evaluate     # Run evaluators on results
+ppef experiment.json   # Run experiment (default command)
+ppef run config.json   # Explicit run command
+ppef validate          # Validate configuration
+ppef plan              # Dry-run execution plan
+ppef aggregate         # Post-process results
+ppef evaluate          # Run evaluators on results
+```
+
+## Quick Start
+
+Create a minimal experiment with three files and a config:
+
+**experiment.json**
+```json
+{
+  "experiment": {
+    "name": "string-length",
+    "description": "Compare string length implementations"
+  },
+  "executor": {
+    "repetitions": 3
+  },
+  "suts": [
+    {
+      "id": "builtin-length",
+      "module": "./sut.mjs",
+      "exportName": "createSut",
+      "registration": {
+        "name": "Built-in .length",
+        "version": "1.0.0",
+        "role": "primary"
+      }
+    }
+  ],
+  "cases": [
+    {
+      "id": "hello-world",
+      "module": "./case.mjs",
+      "exportName": "createCase"
+    }
+  ],
+  "metricsExtractor": {
+    "module": "./metrics.mjs",
+    "exportName": "extract"
+  },
+  "output": {
+    "path": "./results"
+  }
+}
+```
+
+**sut.mjs** — System Under Test factory
+```js
+export function createSut() {
+  return {
+    id: "builtin-length",
+    config: {},
+    run: async (input) => ({ length: input.text.length }),
+  };
+}
+```
+
+**case.mjs** — Test case definition
+```js
+export function createCase() {
+  return {
+    case: {
+      caseId: "hello-world",
+      caseClass: "basic",
+      name: "Hello World",
+      version: "1.0.0",
+      inputs: { text: "hello world" },
+    },
+    getInput: async () => ({ text: "hello world" }),
+    getInputs: () => ({ text: "hello world" }),
+  };
+}
+```
+
+**metrics.mjs** — Metrics extractor
+```js
+export function extract(result) {
+  return { length: result.length ?? 0 };
+}
+```
+
+Run it:
+```bash
+npx ppef experiment.json
+```
+
+## Workflows
+
+The typical pipeline chains CLI commands: validate, run, aggregate, then evaluate.
+
+```
+ppef validate config.json
+    → ppef run config.json
+        → ppef aggregate results.json
+            → ppef evaluate aggregates.json -t claims -c claims.json
+```
+
+### 1. Validate Configuration
+
+Check an experiment config for errors before running:
+
+```bash
+ppef validate experiment.json
+```
+
+### 2. Preview Execution Plan
+
+See what would run without executing (SUTs x cases x repetitions):
+
+```bash
+ppef plan experiment.json
+```
+
+### 3. Run an Experiment
+
+Execute all SUTs against all cases with worker thread isolation:
+
+```bash
+ppef run experiment.json
+ppef run experiment.json -o ./output -j 4 --verbose
+ppef run experiment.json --unsafe-in-process  # No worker isolation (debugging only)
+```
+
+The output directory contains a results JSON and (by default) an aggregates JSON.
+
+### 4. Aggregate Results
+
+Compute summary statistics, pairwise comparisons, and rankings from raw results:
+
+```bash
+ppef aggregate results.json
+ppef aggregate results.json -o aggregates.json --compute-comparisons
+```
+
+### 5. Evaluate Results
+
+Run evaluators against aggregated (or raw) results. Each evaluator type takes a JSON config file.
+
+#### Claims — Test Explicit Hypotheses
+
+Test whether SUT A outperforms baseline B on a given metric with statistical significance:
+
+```bash
+ppef evaluate aggregates.json -t claims -c claims.json -v
+```
+
+**claims.json**:
+```json
+{
+  "claims": [
+    {
+      "claimId": "C001",
+      "description": "Primary has greater accuracy than baseline",
+      "sut": "primary-sut",
+      "baseline": "baseline-sut",
+      "metric": "accuracy",
+      "direction": "greater",
+      "scope": "global"
+    }
+  ],
+  "significanceLevel": 0.05
+}
+```
+
+#### Metrics — Threshold, Baseline, and Range Criteria
+
+Evaluate metrics against fixed thresholds, baselines, or target ranges:
+
+```bash
+ppef evaluate aggregates.json -t metrics -c metrics-config.json
+```
+
+**metrics-config.json**:
+```json
+{
+  "criteria": [
+    {
+      "criterionId": "exec-time",
+      "description": "Execution time under 1000ms",
+      "type": "threshold",
+      "metric": "executionTime",
+      "sut": "*",
+      "threshold": { "operator": "lt", "value": 1000 }
+    },
+    {
+      "criterionId": "f1-range",
+      "description": "F1 score in [0.8, 1.0]",
+      "type": "target-range",
+      "metric": "f1Score",
+      "sut": "*",
+      "targetRange": { "min": 0.8, "max": 1.0, "minInclusive": true, "maxInclusive": true }
+    }
+  ]
+}
+```
+
+#### Robustness — Sensitivity Under Perturbations
+
+Measure how performance degrades under perturbations at varying intensity levels:
+
+```bash
+ppef evaluate results.json -t robustness -c robustness-config.json
+```
+
+**robustness-config.json**:
+```json
+{
+  "metrics": ["executionTime", "accuracy"],
+  "perturbations": ["edge-removal", "noise", "seed-shift"],
+  "intensityLevels": [0.1, 0.2, 0.3, 0.4, 0.5],
+  "runsPerLevel": 10
+}
+```
+
+#### Output Formats
+
+All evaluators support JSON and LaTeX output:
+
+```bash
+ppef evaluate aggregates.json -t claims -c claims.json -f latex
+ppef evaluate aggregates.json -t metrics -c metrics.json -f json -o results.json
+```
+
+### Inline Evaluators
+
+Evaluator configs can be embedded directly in the experiment config via the optional `evaluators` field, making the config self-contained:
+
+```json
+{
+  "experiment": { "name": "my-experiment" },
+  "executor": { "repetitions": 10 },
+  "suts": [ ... ],
+  "cases": [ ... ],
+  "metricsExtractor": { ... },
+  "output": { "path": "./results" },
+  "evaluators": [
+    {
+      "type": "claims",
+      "config": {
+        "claims": [ ... ]
+      }
+    }
+  ]
+}
+```
+
+### JSON Schema Validation
+
+Experiment configs can reference the generated schema for IDE autocompletion:
+
+```json
+{
+  "$schema": "./ppef.schema.json",
+  "experiment": { ... }
+}
+```
+
+Standalone evaluator configs reference schema `$defs`:
+
+```json
+{
+  "$schema": "./ppef.schema.json#/$defs/ClaimsEvaluatorConfig",
+  "claims": [ ... ]
+}
 ```
 
 ## Architecture
@@ -114,7 +378,7 @@ Available subpaths: `ppef/types`, `ppef/registry`, `ppef/executor`, `ppef/collec
 - Node.js native test runner (`node:test` + `node:assert`) — not Vitest/Jest
 - Coverage via c8 (text + html + json-summary in `./coverage/`)
 - Conventional commits enforced via commitlint + husky
-- Semantic release from all branches
+- Semantic release from main branch
 - No `any` types — use `unknown` with type guards
 - Executor produces deterministic `runId` via SHA-256 hash of inputs
 

package/dist/__tests__/cli/evaluate-command.integration.test.js

@@ -268,6 +268,66 @@ describe("evaluate command - integration tests", () => {
             assert.ok(mockLogger.calls.some((call) => call.startsWith("error:") && call.includes("aggregate")));
         });
     });
+    describe("exploratory evaluation", () => {
+        it("should evaluate exploratory analysis successfully", async () => {
+            const aggregatesFile = "/mock/aggregates.json";
+            const configFile = "/mock/config.json";
+            const mockFS = new MockFS({
+                [aggregatesFile]: createMockAggregatesFile(),
+                [configFile]: JSON.stringify({ metrics: ["accuracy"] }),
+            });
+            const mockLogger = new SpyLogger();
+            let exitCode;
+            const mockExit = (code) => {
+                exitCode = code;
+                throw new Error(`Exit ${code}`);
+            };
+            try {
+                await executeEvaluate(aggregatesFile, {
+                    type: "exploratory",
+                    config: configFile,
+                    verbose: true,
+                }, { logger: mockLogger, fileSystem: mockFS, processExit: mockExit });
+            }
+            catch (e) {
+                if (exitCode !== undefined) {
+                    const errorLogs = mockLogger.calls.filter((call) => call.startsWith("error:"));
+                    assert.fail(`processExit was called with code ${exitCode}. Logged errors:\n${errorLogs.join("\n")}`);
+                }
+                throw e;
+            }
+            assert.strictEqual(exitCode, undefined, "processExit should not be called on success");
+            assert.ok(mockLogger.calls.some((call) => call.startsWith("info:Evaluation complete:")));
+        });
+        it("should handle empty config for exploratory", async () => {
+            const aggregatesFile = "/mock/aggregates.json";
+            const configFile = "/mock/config.json";
+            const mockFS = new MockFS({
+                [aggregatesFile]: createMockAggregatesFile(),
+                [configFile]: JSON.stringify({}),
+            });
+            const mockLogger = new SpyLogger();
+            let exitCode;
+            const mockExit = (code) => {
+                exitCode = code;
+                throw new Error(`Exit ${code}`);
+            };
+            try {
+                await executeEvaluate(aggregatesFile, {
+                    type: "exploratory",
+                    config: configFile,
+                }, { logger: mockLogger, fileSystem: mockFS, processExit: mockExit });
+            }
+            catch (e) {
+                if (exitCode !== undefined) {
+                    const errorLogs = mockLogger.calls.filter((call) => call.startsWith("error:"));
+                    assert.fail(`processExit was called with code ${exitCode}. Logged errors:\n${errorLogs.join("\n")}`);
+                }
+                throw e;
+            }
+            assert.strictEqual(exitCode, undefined, "processExit should not be called on success");
+        });
+    });
     describe("robustness evaluation", () => {
         it("should require raw results for robustness", async () => {
             const aggregatesFile = "/mock/aggregates.json";

package/dist/__tests__/cli/evaluate-command.integration.test.js.map

@@ -1 +1 @@
-{"version":3,"file":"evaluate-command.integration.test.js","sourceRoot":"","sources":["../../../src/__tests__/cli/evaluate-command.integration.test.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,WAAW,CAAC;AACzC,OAAO,EAAE,MAAM,IAAI,MAAM,EAAE,MAAM,aAAa,CAAC;AAG/C,OAAO,EAAE,eAAe,EAAE,MAAM,gCAAgC,CAAC;AAEjE,OAAO,EAAE,mBAAmB,EAAE,sBAAsB,EAAE,MAAM,oBAAoB,CAAC;AAEjF;;GAEG;AAEH,SAAS,wBAAwB;IAChC,MAAM,UAAU,GAAuB;QACtC,mBAAmB,CAAC,aAAa,EAAE,SAAS,EAAE,SAAS,EAAE;YACxD,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;SACpD,CAAC;QACF,mBAAmB,CAAC,cAAc,EAAE,UAAU,EAAE,SAAS,EAAE;YAC1D,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;SACpD,CAAC;KACF,CAAC;IAEF,OAAO,IAAI,CAAC,SAAS,CAAC,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;AAChD,CAAC;AAED,SAAS,sBAAsB;IAC9B,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,MAAM,EAAE;YACP;gBACC,OAAO,EAAE,MAAM;gBACf,WAAW,EAAE,YAAY;gBACzB,GAAG,EAAE,aAAa;gBAClB,QAAQ,EAAE,cAAc;gBACxB,MAAM,EAAE,UAAU;gBAClB,SAAS,EAAE,SAAS;gBACpB,KAAK,EAAE,QAAQ;aACf;SACD;KACD,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED,SAAS,uBAAuB;IAC/B,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,QAAQ,EAAE;YACT;gBACC,WAAW,EAAE,MAAM;gBACnB,WAAW,EAAE,gBAAgB;gBAC7B,IAAI,EAAE,WAAW;gBACjB,MAAM,EAAE,UAAU;gBAClB,GAAG,EAAE,aAAa;gBAClB,SAAS,EAAE,EAAE,QAAQ,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE;aAC1C;SACD;KACD,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED,SAAS,0BAA0B;IAClC,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,OAAO,EAAE,CAAC,UAAU,CAAC;QACrB,aAAa,EAAE,CAAC,OAAO,CAAC;KACxB,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED;;GAEG;AACH,MAAM,SAAS;IACd,KAAK,GAAa,EAAE,CAAC;IAErB,MAAM,CAAC,OAAe;QACrB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,UAAU,OAAO,EAAE,CAAC,CAAC;IACtC,CAAC;IAED,SAAS,CAAC,OAAe;QACxB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,aAAa,OAAO,EAAE,CAAC,CAAC;IACzC,CAAC;IAED,IAAI,CAAC,OAAe;QACnB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAe;QACpB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,OAAO,EAAE,CAAC,CAAC;IACrC,CAAC;IAED,KAAK,CAAC,OAAe;QACpB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,OAAO,EAAE,CAAC,CAAC;IACrC,CAAC;IAED,IAAI,CAAC,OAAe;QACnB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IACpC,CAAC;IAED,WAAW,CAAC,QAAiB;QAC5B,SAAS;IACV,CAAC;IAED,OAAO,CAAC,OAAe;QACtB,OAAO,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IAC/C,CAAC;IAED,QAAQ,CAAC,OAAe;QACvB,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC;IACvF,CAAC;CACD;AAED;;GAEG;AACH,MAAM,MAAM;IACH,KAAK,GAAG,IAAI,GAAG,EAAkB,CAAC;IAE1C,YAAY,KAA6B;QACxC,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,OAAO,CAAC,EAAE,EAAE;YACjD,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;QAC/B,CAAC,CAAC,CAAC;IACJ,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,IAAY,EAAE,SAAiB;QAC7C,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACrC,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;YAC3B,MAAM,IAAI,KAAK,CAAC,mBAAmB,IAAI,EAAE,CAAC,CAAC;QAC5C,CAAC;QACD,OAAO,OAAO,CAAC;IAChB,CAAC;IAED,KAAK,CAAC,SAAS,CAAC,IAAY,EAAE,IAAY,EAAE,SAAiB;QAC5D,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC5B,CAAC;CACD;AAED,QAAQ,CAAC,sCAAsC,EAAE,GAAG,EAAE;IACrD,QAAQ,CAAC,mBAAmB,EAAE,GAAG,EAAE;QAClC,EAAE,CAAC,qCAAqC,EAAE,KAAK,IAAI,EAAE;YACpD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,sBAAsB,EAAE;aACtC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,sDAAsD;YACtD,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,IAAI,EAAE,EAC9D,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,2EAA2E;gBAC3E,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;YACvF,kDAAkD;YAClD,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,2BAA2B,CAAC,CAAC,CAAC,CAAC;QAC1F,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,iCAAiC,EAAE,KAAK,IAAI,EAAE;YAChD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,sBAAsB,EAAE;aACtC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,EAC/C,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;QACxF,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,yCAAyC,EAAE,KAAK,IAAI,EAAE;YACxD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;aACnD,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,EAC/C,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,mBAAmB,CAAC,CAAC,CAAC;QACrD,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,oBAAoB,EAAE,GAAG,EAAE;QACnC,EAAE,CAAC,sCAAsC,EAAE,KAAK,IAAI,EAAE;YACrD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,uBAAuB,EAAE;aACvC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd;oBACC,IAAI,EAAE,SAAkB;oBACxB,MAAM,EAAE,UAAU;oBAClB,OAAO,EAAE,IAAI;iBACb,EACD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;YACvF,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,2BAA2B,CAAC,CAAC,CAAC,CAAC;QAC1F,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,gBAAgB,EAAE,GAAG,EAAE;QAC/B,EAAE,CAAC,4CAA4C,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC,EAAE,CAAC,CAAC;YAC9B,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,oBAAoB,EACpB,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;QACvE,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,uCAAuC,EAAE,KAAK,IAAI,EAAE;YACtD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,eAAe;aACjC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,0CAA0C,EAAE,KAAK,IAAI,EAAE;YACzD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;aACjD,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CACR,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC,CACxF,CAAC;QACH,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,uBAAuB,EAAE,GAAG,EAAE;QACtC,EAAE,CAAC,2CAA2C,EAAE,KAAK,IAAI,EAAE;YAC1D,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC;oBAChC,UAAU,EAAE;wBACX,mBAAmB,CAAC,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE;4BAClD,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,CAAC,CAAC;yBACxC,CAAC;qBACF;iBACD,CAAC;gBACF,CAAC,UAAU,CAAC,EAAE,0BAA0B,EAAE;aAC1C,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,YAAqB,EAAE,MAAM,EAAE,UAAU,EAAE,EACnD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,oCAAoC;YACrC,CAAC,CAAC,CAAC;YAEH,uBAAuB;YACvB,IAAI,QAAQ,KAAK,CAAC,EAAE,CAAC;gBACpB,MAAM,CAAC,IAAI,CACV,sCAAsC,QAAQ,iBAAiB,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAC5F,CAAC;YACH,CAAC;YAED,0BAA0B;YAC1B,MAAM,kBAAkB,GAAG,UAAU,CAAC,KAAK,CAAC,IAAI,CAC/C,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,aAAa,CAAC,CACnE,CAAC;YACF,IAAI,CAAC,kBAAkB,EAAE,CAAC;gBACzB,MAAM,CAAC,IAAI,CACV,8DAA8D,UAAU,CAAC,KAAK;qBAC5E,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC;qBACrC,IAAI,CAAC,IAAI,CAAC,EAAE,CACd,CAAC;YACH,CAAC;QACF,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;AACJ,CAAC,CAAC,CAAC"}
+{"version":3,"file":"evaluate-command.integration.test.js","sourceRoot":"","sources":["../../../src/__tests__/cli/evaluate-command.integration.test.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAEH,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,WAAW,CAAC;AACzC,OAAO,EAAE,MAAM,IAAI,MAAM,EAAE,MAAM,aAAa,CAAC;AAG/C,OAAO,EAAE,eAAe,EAAE,MAAM,gCAAgC,CAAC;AAEjE,OAAO,EAAE,mBAAmB,EAAE,sBAAsB,EAAE,MAAM,oBAAoB,CAAC;AAEjF;;GAEG;AAEH,SAAS,wBAAwB;IAChC,MAAM,UAAU,GAAuB;QACtC,mBAAmB,CAAC,aAAa,EAAE,SAAS,EAAE,SAAS,EAAE;YACxD,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;SACpD,CAAC;QACF,mBAAmB,CAAC,cAAc,EAAE,UAAU,EAAE,SAAS,EAAE;YAC1D,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;SACpD,CAAC;KACF,CAAC;IAEF,OAAO,IAAI,CAAC,SAAS,CAAC,EAAE,UAAU,EAAE,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC;AAChD,CAAC;AAED,SAAS,sBAAsB;IAC9B,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,MAAM,EAAE;YACP;gBACC,OAAO,EAAE,MAAM;gBACf,WAAW,EAAE,YAAY;gBACzB,GAAG,EAAE,aAAa;gBAClB,QAAQ,EAAE,cAAc;gBACxB,MAAM,EAAE,UAAU;gBAClB,SAAS,EAAE,SAAS;gBACpB,KAAK,EAAE,QAAQ;aACf;SACD;KACD,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED,SAAS,uBAAuB;IAC/B,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,QAAQ,EAAE;YACT;gBACC,WAAW,EAAE,MAAM;gBACnB,WAAW,EAAE,gBAAgB;gBAC7B,IAAI,EAAE,WAAW;gBACjB,MAAM,EAAE,UAAU;gBAClB,GAAG,EAAE,aAAa;gBAClB,SAAS,EAAE,EAAE,QAAQ,EAAE,KAAK,EAAE,KAAK,EAAE,GAAG,EAAE;aAC1C;SACD;KACD,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED,SAAS,0BAA0B;IAClC,OAAO,IAAI,CAAC,SAAS,CACpB;QACC,OAAO,EAAE,CAAC,UAAU,CAAC;QACrB,aAAa,EAAE,CAAC,OAAO,CAAC;KACxB,EACD,IAAI,EACJ,CAAC,CACD,CAAC;AACH,CAAC;AAED;;GAEG;AACH,MAAM,SAAS;IACd,KAAK,GAAa,EAAE,CAAC;IAErB,MAAM,CAAC,OAAe;QACrB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,UAAU,OAAO,EAAE,CAAC,CAAC;IACtC,CAAC;IAED,SAAS,CAAC,OAAe;QACxB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,aAAa,OAAO,EAAE,CAAC,CAAC;IACzC,CAAC;IAED,IAAI,CAAC,OAAe;QACnB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAe;QACpB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,OAAO,EAAE,CAAC,CAAC;IACrC,CAAC;IAED,KAAK,CAAC,OAAe;QACpB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,OAAO,EAAE,CAAC,CAAC;IACrC,CAAC;IAED,IAAI,CAAC,OAAe;QACnB,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IACpC,CAAC;IAED,WAAW,CAAC,QAAiB;QAC5B,SAAS;IACV,CAAC;IAED,OAAO,CAAC,OAAe;QACtB,OAAO,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,QAAQ,OAAO,EAAE,CAAC,CAAC;IAC/C,CAAC;IAED,QAAQ,CAAC,OAAe;QACvB,OAAO,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC;IACvF,CAAC;CACD;AAED;;GAEG;AACH,MAAM,MAAM;IACH,KAAK,GAAG,IAAI,GAAG,EAAkB,CAAC;IAE1C,YAAY,KAA6B;QACxC,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,OAAO,CAAC,EAAE,EAAE;YACjD,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;QAC/B,CAAC,CAAC,CAAC;IACJ,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,IAAY,EAAE,SAAiB;QAC7C,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACrC,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;YAC3B,MAAM,IAAI,KAAK,CAAC,mBAAmB,IAAI,EAAE,CAAC,CAAC;QAC5C,CAAC;QACD,OAAO,OAAO,CAAC;IAChB,CAAC;IAED,KAAK,CAAC,SAAS,CAAC,IAAY,EAAE,IAAY,EAAE,SAAiB;QAC5D,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC5B,CAAC;CACD;AAED,QAAQ,CAAC,sCAAsC,EAAE,GAAG,EAAE;IACrD,QAAQ,CAAC,mBAAmB,EAAE,GAAG,EAAE;QAClC,EAAE,CAAC,qCAAqC,EAAE,KAAK,IAAI,EAAE;YACpD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,sBAAsB,EAAE;aACtC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,sDAAsD;YACtD,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,OAAO,EAAE,IAAI,EAAE,EAC9D,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,2EAA2E;gBAC3E,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;YACvF,kDAAkD;YAClD,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,2BAA2B,CAAC,CAAC,CAAC,CAAC;QAC1F,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,iCAAiC,EAAE,KAAK,IAAI,EAAE;YAChD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,sBAAsB,EAAE;aACtC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,EAC/C,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;QACxF,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,yCAAyC,EAAE,KAAK,IAAI,EAAE;YACxD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,QAAQ,EAAE,CAAC;aACnD,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,MAAM,EAAE,UAAU,EAAE,EAC/C,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,mBAAmB,CAAC,CAAC,CAAC;QACrD,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,oBAAoB,EAAE,GAAG,EAAE;QACnC,EAAE,CAAC,sCAAsC,EAAE,KAAK,IAAI,EAAE;YACrD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,uBAAuB,EAAE;aACvC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd;oBACC,IAAI,EAAE,SAAkB;oBACxB,MAAM,EAAE,UAAU;oBAClB,OAAO,EAAE,IAAI;iBACb,EACD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;YACvF,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,2BAA2B,CAAC,CAAC,CAAC,CAAC;QAC1F,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,gBAAgB,EAAE,GAAG,EAAE;QAC/B,EAAE,CAAC,4CAA4C,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC,EAAE,CAAC,CAAC;YAC9B,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,oBAAoB,EACpB,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC;QACvE,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,uCAAuC,EAAE,KAAK,IAAI,EAAE;YACtD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,eAAe;aACjC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,0CAA0C,EAAE,KAAK,IAAI,EAAE;YACzD,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;aACjD,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,QAAiB,EAAE,EAC3B,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,WAAW;YACZ,CAAC,CAAC,CAAC;YAEH,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC;YAChC,MAAM,CAAC,EAAE,CACR,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC,CACxF,CAAC;QACH,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,wBAAwB,EAAE,GAAG,EAAE;QACvC,EAAE,CAAC,mDAAmD,EAAE,KAAK,IAAI,EAAE;YAClE,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,OAAO,EAAE,CAAC,UAAU,CAAC,EAAE,CAAC;aACvD,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd;oBACC,IAAI,EAAE,aAAsB;oBAC5B,MAAM,EAAE,UAAU;oBAClB,OAAO,EAAE,IAAI;iBACb,EACD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;YACvF,MAAM,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,2BAA2B,CAAC,CAAC,CAAC,CAAC;QAC1F,CAAC,CAAC,CAAC;QAEH,EAAE,CAAC,4CAA4C,EAAE,KAAK,IAAI,EAAE;YAC3D,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,wBAAwB,EAAE;gBAC5C,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EAAE,CAAC;aAChC,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAA4B,CAAC;YACjC,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,IAAI,CAAC;gBACJ,MAAM,eAAe,CACpB,cAAc,EACd;oBACC,IAAI,EAAE,aAAsB;oBAC5B,MAAM,EAAE,UAAU;iBAClB,EACD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC;YACH,CAAC;YAAC,OAAO,CAAC,EAAE,CAAC;gBACZ,IAAI,QAAQ,KAAK,SAAS,EAAE,CAAC;oBAC5B,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC;oBAC/E,MAAM,CAAC,IAAI,CACV,oCAAoC,QAAQ,qBAAqB,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CACvF,CAAC;gBACH,CAAC;gBACD,MAAM,CAAC,CAAC;YACT,CAAC;YAED,MAAM,CAAC,WAAW,CAAC,QAAQ,EAAE,SAAS,EAAE,6CAA6C,CAAC,CAAC;QACxF,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;IAEH,QAAQ,CAAC,uBAAuB,EAAE,GAAG,EAAE;QACtC,EAAE,CAAC,2CAA2C,EAAE,KAAK,IAAI,EAAE;YAC1D,MAAM,cAAc,GAAG,uBAAuB,CAAC;YAC/C,MAAM,UAAU,GAAG,mBAAmB,CAAC;YACvC,MAAM,MAAM,GAAG,IAAI,MAAM,CAAC;gBACzB,CAAC,cAAc,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC;oBAChC,UAAU,EAAE;wBACX,mBAAmB,CAAC,OAAO,EAAE,SAAS,EAAE,SAAS,EAAE;4BAClD,QAAQ,EAAE,sBAAsB,CAAC,CAAC,IAAI,CAAC,CAAC;yBACxC,CAAC;qBACF;iBACD,CAAC;gBACF,CAAC,UAAU,CAAC,EAAE,0BAA0B,EAAE;aAC1C,CAAC,CAAC;YACH,MAAM,UAAU,GAAG,IAAI,SAAS,EAAE,CAAC;YACnC,IAAI,QAAQ,GAAG,CAAC,CAAC;YACjB,MAAM,QAAQ,GAAG,CAAC,IAAY,EAAE,EAAE;gBACjC,QAAQ,GAAG,IAAI,CAAC;gBAChB,MAAM,IAAI,KAAK,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC;YACjC,CAAC,CAAC;YAEF,MAAM,eAAe,CACpB,cAAc,EACd,EAAE,IAAI,EAAE,YAAqB,EAAE,MAAM,EAAE,UAAU,EAAE,EACnD,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,CACjE,CAAC,KAAK,CAAC,GAAG,EAAE;gBACZ,oCAAoC;YACrC,CAAC,CAAC,CAAC;YAEH,uBAAuB;YACvB,IAAI,QAAQ,KAAK,CAAC,EAAE,CAAC;gBACpB,MAAM,CAAC,IAAI,CACV,sCAAsC,QAAQ,iBAAiB,UAAU,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAC5F,CAAC;YACH,CAAC;YAED,0BAA0B;YAC1B,MAAM,kBAAkB,GAAG,UAAU,CAAC,KAAK,CAAC,IAAI,CAC/C,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,aAAa,CAAC,CACnE,CAAC;YACF,IAAI,CAAC,kBAAkB,EAAE,CAAC;gBACzB,MAAM,CAAC,IAAI,CACV,8DAA8D,UAAU,CAAC,KAAK;qBAC5E,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC;qBACrC,IAAI,CAAC,IAAI,CAAC,EAAE,CACd,CAAC;YACH,CAAC;QACF,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC,CAAC;AACJ,CAAC,CAAC,CAAC"}

package/dist/__tests__/examples.integration.test.d.ts

@@ -0,0 +1,8 @@
+/**
+ * Example Integration Tests
+ *
+ * Runs example experiments via the CLI binary to verify
+ * end-to-end functionality. Requires `pnpm build` first.
+ */
+export {};
+//# sourceMappingURL=examples.integration.test.d.ts.map

package/dist/__tests__/examples.integration.test.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"examples.integration.test.d.ts","sourceRoot":"","sources":["../../src/__tests__/examples.integration.test.ts"],"names":[],"mappings":"AAAA;;;;;GAKG"}

package/dist/__tests__/examples.integration.test.js

@@ -0,0 +1,236 @@
+/**
+ * Example Integration Tests
+ *
+ * Runs example experiments via the CLI binary to verify
+ * end-to-end functionality. Requires `pnpm build` first.
+ */
+import { describe, it, before, after } from "node:test";
+import { strict as assert } from "node:assert";
+import { execFile } from "node:child_process";
+import { mkdtemp, rm, readdir, readFile, access } from "node:fs/promises";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import { promisify } from "node:util";
+const execFileAsync = promisify(execFile);
+const PROJECT_ROOT = join(import.meta.dirname, "..", "..");
+const BIN_PATH = join(PROJECT_ROOT, "bin", "ppef.mjs");
+const EXAMPLES_DIR = join(PROJECT_ROOT, "examples");
+/**
+ * Verify dist/ exists before running tests.
+ */
+async function assertDistExists() {
+    try {
+        await access(join(PROJECT_ROOT, "dist"));
+    }
+    catch {
+        throw new Error("dist/ not found — run `pnpm build` before running integration tests");
+    }
+}
+describe("Example Integration Tests", () => {
+    let tempDir;
+    before(async () => {
+        await assertDistExists();
+        tempDir = await mkdtemp(join(tmpdir(), "ppef-examples-"));
+    });
+    after(async () => {
+        if (tempDir) {
+            await rm(tempDir, { recursive: true, force: true });
+        }
+    });
+    describe("string-length example", () => {
+        it("runs end-to-end and produces results", async () => {
+            const outputDir = join(tempDir, "string-length-results");
+            const configPath = join(EXAMPLES_DIR, "string-length", "experiment.json");
+            // Read the config and override output path
+            const configContent = JSON.parse(await readFile(configPath, "utf-8"));
+            configContent.output.path = outputDir;
+            // Write modified config to temp dir
+            const tempConfig = join(tempDir, "experiment.json");
+            const { writeFile: writeFileFs } = await import("node:fs/promises");
+            await writeFileFs(tempConfig, JSON.stringify(configContent));
+            // Copy example modules to temp dir so relative paths resolve
+            const { copyFile } = await import("node:fs/promises");
+            await copyFile(join(EXAMPLES_DIR, "string-length", "sut.mjs"), join(tempDir, "sut.mjs"));
+            await copyFile(join(EXAMPLES_DIR, "string-length", "case.mjs"), join(tempDir, "case.mjs"));
+            await copyFile(join(EXAMPLES_DIR, "string-length", "metrics.mjs"), join(tempDir, "metrics.mjs"));
+            const { stdout } = await execFileAsync("node", [BIN_PATH, "run", tempConfig, "--unsafe-in-process"], {
+                cwd: tempDir,
+                timeout: 30000,
+            });
+            assert.ok(stdout.length > 0, "Expected CLI output");
+            // Verify output directory exists with results
+            const files = await readdir(outputDir);
+            assert.ok(files.length > 0, `Expected result files in ${outputDir}, found none`);
+            // Find a results JSON file and verify structure
+            const jsonFiles = files.filter((f) => f.endsWith(".json"));
+            assert.ok(jsonFiles.length > 0, "Expected at least one JSON result file");
+            const resultContent = JSON.parse(await readFile(join(outputDir, jsonFiles[0]), "utf-8"));
+            // Verify results have the expected structure
+            assert.ok(resultContent.results ?? resultContent.aggregates, "Expected results or aggregates in output");
+        });
+        it("validates config successfully", async () => {
+            const configPath = join(EXAMPLES_DIR, "string-length", "experiment.json");
+            const { stdout } = await execFileAsync("node", [BIN_PATH, "validate", configPath], {
+                cwd: PROJECT_ROOT,
+                timeout: 10000,
+            });
+            assert.ok(stdout.length > 0, "Expected validation output");
+        });
+        it("dry-run produces no output files", async () => {
+            const outputDir = join(tempDir, "dry-run-results");
+            const configPath = join(EXAMPLES_DIR, "string-length", "experiment.json");
+            // Read config and override output path
+            const configContent = JSON.parse(await readFile(configPath, "utf-8"));
+            configContent.output.path = outputDir;
+            const tempConfig = join(tempDir, "dry-run-experiment.json");
+            const { writeFile: writeFileFs } = await import("node:fs/promises");
+            await writeFileFs(tempConfig, JSON.stringify(configContent));
+            // Copy modules
+            // (already copied from previous test, but be explicit)
+            const { copyFile } = await import("node:fs/promises");
+            for (const file of ["sut.mjs", "case.mjs", "metrics.mjs"]) {
+                await copyFile(join(EXAMPLES_DIR, "string-length", file), join(tempDir, file));
+            }
+            await execFileAsync("node", [BIN_PATH, "plan", tempConfig], {
+                cwd: tempDir,
+                timeout: 10000,
+            });
+            // Verify no output directory was created
+            try {
+                await access(outputDir);
+                assert.fail("Expected output directory to not exist after dry-run");
+            }
+            catch {
+                // Expected — directory should not exist
+            }
+        });
+    });
+    describe("two-sut evaluation pipeline", () => {
+        let twoSutOutputDir;
+        let twoSutAggregatesPath;
+        it("runs two-SUT experiment end-to-end", async () => {
+            twoSutOutputDir = join(tempDir, "two-sut-results");
+            const configPath = join(EXAMPLES_DIR, "string-length", "experiment-two-suts.json");
+            // Read config and override output path
+            const configContent = JSON.parse(await readFile(configPath, "utf-8"));
+            configContent.output.path = twoSutOutputDir;
+            const tempConfig = join(tempDir, "experiment-two-suts.json");
+            const { writeFile: writeFileFs, copyFile } = await import("node:fs/promises");
+            await writeFileFs(tempConfig, JSON.stringify(configContent));
+            // Copy all required modules
+            for (const file of [
+                "sut.mjs",
+                "sut-spread.mjs",
+                "case.mjs",
+                "case-unicode.mjs",
+                "metrics.mjs",
+            ]) {
+                await copyFile(join(EXAMPLES_DIR, "string-length", file), join(tempDir, file));
+            }
+            const { stdout } = await execFileAsync("node", [BIN_PATH, "run", tempConfig, "--unsafe-in-process"], {
+                cwd: tempDir,
+                timeout: 30000,
+            });
+            assert.ok(stdout.length > 0, "Expected CLI output");
+            // Find the aggregates file
+            const files = await readdir(twoSutOutputDir);
+            const aggregatesFiles = files.filter((f) => f.includes("aggregates"));
+            assert.ok(aggregatesFiles.length > 0, "Expected aggregates file");
+            twoSutAggregatesPath = join(twoSutOutputDir, aggregatesFiles[0]);
+            // Verify aggregates structure
+            const aggContent = JSON.parse(await readFile(twoSutAggregatesPath, "utf-8"));
+            assert.ok(Array.isArray(aggContent.aggregates), "Expected aggregates array in output");
+            assert.ok(aggContent.aggregates.length >= 2, "Expected at least 2 aggregated results");
+        });
+        it("evaluates claims on two-SUT results", async () => {
+            assert.ok(twoSutAggregatesPath, "Aggregates path must be set by previous test");
+            const claimsConfig = join(EXAMPLES_DIR, "string-length", "eval-claims.json");
+            const claimsOutput = join(tempDir, "claims-output.json");
+            const { stdout } = await execFileAsync("node", [
+                BIN_PATH,
+                "evaluate",
+                twoSutAggregatesPath,
+                "-t",
+                "claims",
+                "-c",
+                claimsConfig,
+                "-o",
+                claimsOutput,
+                "-v",
+            ], {
+                cwd: tempDir,
+                timeout: 15000,
+            });
+            assert.ok(stdout.length > 0, "Expected CLI output");
+            const output = JSON.parse(await readFile(claimsOutput, "utf-8"));
+            assert.strictEqual(output.type, "claims", "Expected claims evaluation type");
+        });
+        it("evaluates exploratory analysis on two-SUT results", async () => {
+            assert.ok(twoSutAggregatesPath, "Aggregates path must be set by previous test");
+            const exploratoryConfig = join(EXAMPLES_DIR, "string-length", "eval-exploratory.json");
+            const exploratoryOutput = join(tempDir, "exploratory-output.json");
+            const { stdout } = await execFileAsync("node", [
+                BIN_PATH,
+                "evaluate",
+                twoSutAggregatesPath,
+                "-t",
+                "exploratory",
+                "-c",
+                exploratoryConfig,
+                "-o",
+                exploratoryOutput,
+                "-v",
+            ], {
+                cwd: tempDir,
+                timeout: 15000,
+            });
+            assert.ok(stdout.length > 0, "Expected CLI output");
+            const output = JSON.parse(await readFile(exploratoryOutput, "utf-8"));
+            assert.strictEqual(output.type, "exploratory", "Expected exploratory evaluation type");
+            assert.ok(output.data.rankings, "Expected rankings in exploratory output");
+            assert.ok(Array.isArray(output.data.pairwiseComparisons), "Expected pairwiseComparisons array");
+        });
+        it("evaluates metrics on two-SUT results", async () => {
+            assert.ok(twoSutAggregatesPath, "Aggregates path must be set by previous test");
+            const metricsConfig = join(EXAMPLES_DIR, "metrics-only", "eval-config.json");
+            const metricsOutput = join(tempDir, "metrics-output.json");
+            const { stdout } = await execFileAsync("node", [
+                BIN_PATH,
+                "evaluate",
+                twoSutAggregatesPath,
+                "-t",
+                "metrics",
+                "-c",
+                metricsConfig,
+                "-o",
+                metricsOutput,
+                "-v",
+            ], {
+                cwd: tempDir,
+                timeout: 15000,
+            });
+            assert.ok(stdout.length > 0, "Expected CLI output");
+            const output = JSON.parse(await readFile(metricsOutput, "utf-8"));
+            assert.strictEqual(output.type, "metrics", "Expected metrics evaluation type");
+        });
+    });
+    describe("invalid config", () => {
+        it("rejects malformed config with non-zero exit code", async () => {
+            const invalidConfig = join(tempDir, "invalid.json");
+            const { writeFile: writeFileFs } = await import("node:fs/promises");
+            await writeFileFs(invalidConfig, JSON.stringify({ invalid: true }));
+            try {
+                await execFileAsync("node", [BIN_PATH, invalidConfig], {
+                    cwd: tempDir,
+                    timeout: 10000,
+                });
+                assert.fail("Expected non-zero exit code for invalid config");
+            }
+            catch (error) {
+                // execFile rejects on non-zero exit code
+                assert.ok(error, "Expected error for invalid config");
+            }
+        });
+    });
+});
+//# sourceMappingURL=examples.integration.test.js.map