@cogitator-ai/evals 0.12.0 → 0.13.1

package/README.md

@@ -13,7 +13,7 @@ pnpm add @cogitator-ai/core  # agent targets and LLM-as-judge metrics
 pnpm add -D vitest  # describeEval from @cogitator-ai/evals/vitest
 ```
 
-Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
+Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Experiments](https://cogitator.app/docs/evals/experiments), [Reporters](https://cogitator.app/docs/evals/reporters).
 
 ## Features
 
@@ -26,7 +26,8 @@ Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogi
 - **Trials and pass@k** - run each case several times and score how often it passes with `passAtK()`
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
 - **A/B Testing** - EvalComparison with McNemar's test (exact for few discordant pairs), the paired t-test or Wilcoxon signed-rank, Holm's correction across metrics, bootstrap confidence intervals, effect sizes, latency and cost comparisons and pairwise judging
-- **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
+- **Experiment Tracking** - named runs with metadata saved as versioned run files, a `RunStore` directory of runs, `compareRuns()` with the A/B statistics, per-case regressions and improvements, and warnings when datasets, prompt versions, metrics or judges differ. Agent cases keep the `traceId` and versioned `prompt` of their runs
+- **8 Reporters** - console (colored table), JSON, CSV, JUnit XML, Markdown (pull request comments), self-contained HTML, GitHub Actions annotations and job summary, CI (exit code on failure), plus `reportComparison()` for comparisons
 - **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
 - **Baseline Workflow** — Save baselines, compare against them, catch regressions in CI
@@ -86,7 +87,7 @@ const suite = new EvalSuite({
 
 `suite.run(options?: EvalRunOptions)` takes `{ maxCases?: number, signal?: AbortSignal }`: with `maxCases` only the first N cases of the dataset run, and the results, aggregates, assertions and `stats.total` cover just those cases. Aborting `signal` cancels the attempts in flight, starts no further case, and `run()` rejects with the signal's reason.
 
-A case whose every attempt failed is not scored: its empty output costs no judge calls and counts as no 0, the aggregates cover the cases that answered (`aggregated.<metric>.count` says how many), and `maxErrorRate(rate)` fails a run where too many cases errored. An agent target's case result carries `truncated`, `iterationLimitReached` and `paused` when the run ended that way.
+A case whose every attempt failed is not scored: its empty output costs no judge calls and counts as no 0, the aggregates cover the cases that answered (`aggregated.<metric>.count` says how many), and `maxErrorRate(rate)` fails a run where too many cases errored. An agent target's case result carries `truncated`, `iterationLimitReached` and `paused` when the run ended that way (`paused` for a run waiting on tool approvals or on a question one of its tools asked with `context.interrupt()`).
 
 A failure never drops a case. A metric that throws or returns a score that is not a finite number (such as NaN) scores 0 for that case with the reason in `score.error`, and an `onProgress` callback that throws is reported as a process warning.
 
@@ -574,14 +575,16 @@ Access full suite results via `result.baseline` and `result.challenger`. `compar
 
 Call `result.report()` after a suite run to output results.
 
-| Reporter  | Output                                                               |
-| --------- | -------------------------------------------------------------------- |
-| `console` | Colored table with metrics, assertions, summary                      |
-| `json`    | Writes `eval-report.json` (configurable path)                        |
-| `csv`     | Writes `eval-report.csv` (configurable path)                         |
-| `junit`   | Writes `eval-report.xml`, a JUnit report (configurable path)         |
-| `github`  | GitHub Actions annotations, plus a job summary in Actions            |
-| `ci`      | Compact verdict, sets `process.exitCode = 1` when an assertion fails |
+| Reporter   | Output                                                                |
+| ---------- | --------------------------------------------------------------------- |
+| `console`  | Colored table with metrics, assertions, summary                       |
+| `json`     | Writes `eval-report.json` (configurable path)                         |
+| `csv`      | Writes `eval-report.csv` (configurable path)                          |
+| `junit`    | Writes `eval-report.xml`, a JUnit report (configurable path)          |
+| `github`   | GitHub Actions annotations, plus a job summary in Actions             |
+| `ci`       | Compact verdict, sets `process.exitCode = 1` when an assertion fails  |
+| `markdown` | Writes `eval-report.md`, a compact summary for a pull request comment |
+| `html`     | Writes `eval-report.html`, a self-contained page for light and dark   |
 
 ```typescript
 const result = await suite.run();
@@ -592,9 +595,11 @@ result.report(['console', 'json', 'csv', 'junit'], { path: './reports/eval' });
 result.report(['github', 'ci']);
 ```
 
-The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
+The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`, `.md`, `.html`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
 
 - **junit** writes one `<testsuite name="eval">` with a `<testcase>` per case (a `<failure>` when it errored, its scores in `<system-out>`) and a `<testcase>` per assertion (a `<failure>` when it did not pass). Everything is XML-escaped, and characters XML 1.0 does not allow are replaced.
+- **markdown** writes the totals (with the trials per case when there are several), each metric's mean and cases, the assertions with pass or fail, and the failures of every case and trial. Every value is escaped, so no answer breaks the tables.
+- **html** writes one page with inline CSS and no scripts or external assets: totals, metrics, assertions and every case with its scores, status, turns, trials, trace id and prompt version. Every value is HTML-escaped.
 - **github** prints `::error` for each failed assertion and errored case and `::warning` for each failed metric score. When `GITHUB_STEP_SUMMARY` is set it also appends a Markdown summary with the totals, the aggregated metrics (mean, p50, p95, cases scored) and the assertions.
 
 Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output. The json report keeps each case's `toolCalls`, `toolResults`, `retrieved`, `turns` and `trials`, and the console and github reports show the trials per case.
@@ -636,6 +641,41 @@ it('books a table', async () => {
 
 ---
 
+## Experiment Tracking
+
+Give a suite a `name` and JSON `metadata`, save each run as a run file, and compare runs later, across commits or prompt versions.
+
+```typescript
+import { EvalSuite, RunStore, compareRuns, reportComparison } from '@cogitator-ai/evals';
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [exactMatch(), faithfulness()],
+  judge: { model: 'google/gemini-3.5-flash-lite', cogitator },
+  name: 'support-agent',
+  metadata: { branch: process.env.GITHUB_HEAD_REF, commit: process.env.GITHUB_SHA },
+});
+
+const result = await suite.run();
+const store = new RunStore('./eval-runs');
+const base = store.latest({ name: 'support-agent' });
+store.save(result); // ./eval-runs/<runId>.json, as result.saveRun('./eval-runs') writes it
+
+if (base) {
+  const comparison = compareRuns(base, result, { seed: 42 });
+  reportComparison(comparison, ['console', 'markdown'], { path: './reports/comparison' });
+}
+```
+
+- **Run files.** `result.saveRun(target)` writes JSON with `format: 'cogitator.eval-run'` and `version: 1`: `runId`, `startedAt` and `finishedAt`, `name`, `metadata`, the dataset fingerprint (SHA-256 of the canonical JSON of the cases run), the config (metric names, judge model, temperature and samples, trials, concurrency, target kind, agent name and model), the prompt versions used, every case result with its `caseKey`, the aggregates, assertions and stats. A directory target (or one ending in `/`) gets `<runId>.json`. Functions, Cogitator instances and keys are never written. `result.toRun()` returns the same object, and `loadRun(path)` reads one back, validated with Zod.
+- **Traces and prompts.** Agent cases keep the `traceId` of their run (`RunResult.trace.traceId`) and the versioned instructions in `prompt` (`{ key, versionId?, version?, abTest? }`, set when the Cogitator versions instructions with `cogitator.prompts`). Each turn of a multi-turn case keeps its own and the case keeps the last turn's, each trial its own.
+- **RunStore.** `new RunStore(dir, { onSkip? })` keeps run files in a directory: `save(result | run)`, `list(filter?)` newest first with id, name, metadata, times and metric means, `load(id)`, and `latest(filter?)`, where a filter is `{ name }` or a predicate. `list()` and `latest()` skip `.json` entries they cannot use (a directory, a file that is not JSON, a run file of another version or edited by hand) and report each to `onSkip`, a process warning by default, while `load(id)` throws for them. A run is validated before it is written, so a hand-made `runId` cannot lead outside the directory.
+- **compareRuns(base, head, { alpha?, test?, seed?, metrics? }).** Cases are matched by `caseKey` (SHA-256 of `{ input, turns, expected }`), the others listed in `added` and `removed`. Each metric both runs have gets the statistics of `EvalComparison`: means, `difference`, `ci`, `effectSize`, `pValue`, `adjustedPValue` (Holm across the compared metrics), `significant` and `winner` (`head`, `base` or `tie`), plus the `improved` and `regressed` cases. An errored side counts as 0, a case both runs errored on is dropped. `warnings` say when the datasets, the prompt versions, the metrics or the judge differ, and name a metric left out because no matched case was scored on it.
+- **reportComparison(comparison, 'markdown' | 'html' | 'console', { path? }).** Renders a `compareRuns()` or `EvalComparison` result: the winner, a metric table with difference, CI, p and adjusted p and significance, the regressions and improvements, and the warnings. The default path is `eval-comparison`.
+
+---
+
 ## Builder API
 
 `EvalBuilder` provides a fluent interface for constructing eval suites.
@@ -720,6 +760,10 @@ result.report('ci');
 | `EvalComparison` | A/B testing runner with statistical significance                    |
 | `calibrateJudge` | Agreement of a judge metric with labelled examples                  |
 | `EvalBuilder`    | Fluent builder for EvalSuite                                        |
+| `compareRuns`    | Compares two saved runs or suite results case by case               |
+| `RunStore`       | Directory of run files: save, list, load, latest                    |
+| `loadRun`        | Reads and validates a run file                                      |
+| `caseKey`        | The key a case is matched by across runs                            |
 | `Dataset`        | Immutable dataset with from/fromJsonl/fromCsv/filter/sample/shuffle |
 | `loadJsonl`      | Low-level JSONL file loader                                         |
 | `loadCsv`        | Low-level CSV file loader                                           |
@@ -767,25 +811,30 @@ result.report('ci');
 
 ### Reporters
 
-| Export                             | Description                              |
-| ---------------------------------- | ---------------------------------------- |
-| `report`                           | Dispatch to one or more reporter types   |
-| `describeEval` (`/vitest` entry)   | Run a suite as a Vitest describe block   |
-| `evalMatchers` (`/matchers` entry) | Matchers for `expect.extend()`, Vitest 5 |
+| Export                                 | Description                                      |
+| -------------------------------------- | ------------------------------------------------ |
+| `report`                               | Dispatch to one or more reporter types           |
+| `reportComparison`                     | Render a comparison as console, Markdown or HTML |
+| `comparisonMarkdown`, `comparisonHtml` | The Markdown or HTML of a comparison as a string |
+| `describeEval` (`/vitest` entry)       | Run a suite as a Vitest describe block           |
+| `evalMatchers` (`/matchers` entry)     | Matchers for `expect.extend()`, Vitest 5         |
 
-`report` takes `console`, `json`, `csv`, `junit`, `github` and `ci`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
+`report` takes `console`, `json`, `csv`, `junit`, `github`, `ci`, `markdown` and `html`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
 
 ### Types
 
-| Export             | Description                                                              |
-| ------------------ | ------------------------------------------------------------------------ |
-| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency` |
-| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`  |
-| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers              |
-| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on      |
-| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`              |
-| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`               |
-| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `error?`   |
+| Export             | Description                                                                                   |
+| ------------------ | --------------------------------------------------------------------------------------------- |
+| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency`                      |
+| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`                       |
+| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers                                   |
+| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on                           |
+| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`                                   |
+| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`                                    |
+| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `traceId?`, `prompt?`, `error?` |
+| `EvalPrompt`       | Versioned instructions of an agent run: `key`, `versionId?`, `version?`, `abTest?`            |
+| `EvalRun`          | A saved run, as `loadRun()` returns it                                                        |
+| `RunComparison`    | What `compareRuns()` returns                                                                  |
 
 ### Statistics
 

package/dist/comparison-stats.d.ts

@@ -0,0 +1,104 @@
+import type { EvalComparisonConfig } from './schema.js';
+/**
+ * The test a comparison ran: McNemar's (exact below 25 discordant pairs, chi-square with
+ * continuity correction from 25), the paired t-test, Wilcoxon signed-rank, or none with fewer
+ * than 2 cases
+ */
+export type ComparisonTest = 'mcnemar' | 'mcnemar-exact' | 't' | 'wilcoxon' | 'none';
+export type ComparisonWinner = 'baseline' | 'challenger' | 'tie';
+export interface MetricComparison {
+    /** Mean score over the compared cases, NaN when `n` is 0 */
+    baseline: number;
+    challenger: number;
+    /** Cases compared: those at least one side scored, an errored side counting as a 0 */
+    n: number;
+    /** Mean of the paired differences, challenger minus baseline, NaN when `n` is 0 */
+    difference: number;
+    /** Percentile bootstrap confidence interval of `difference` at `1 - alpha` */
+    ci: {
+        low: number;
+        high: number;
+    };
+    /** Cohen's d of the paired differences (dz), NaN with fewer than 2 cases */
+    effectSize: number;
+    test: ComparisonTest;
+    /** The test's own p-value */
+    pValue: number;
+    /** `pValue` after Holm's correction across the compared metrics */
+    adjustedPValue: number;
+    /** `adjustedPValue` is below `alpha` */
+    significant: boolean;
+    /**
+     * The higher side when significant, by the test's direction: the sign of `difference`, except
+     * for Wilcoxon's test, where it is the side with the larger rank sum and may differ from it
+     */
+    winner: ComparisonWinner;
+}
+/**
+ * Latency or cost of the two sides, paired over the cases both answered, where lower is better.
+ * Not corrected for multiple comparisons and not part of the metrics' vote on the winner
+ */
+export interface ResourceComparison {
+    /** Mean per case, NaN when no case pairs up */
+    baseline: number;
+    challenger: number;
+    n: number;
+    /** Challenger minus baseline: below 0 when the challenger is cheaper or faster */
+    difference: number;
+    test: ComparisonTest;
+    pValue: number;
+    significant: boolean;
+    /** The lower side when significant, by the test's direction as for `MetricComparison.winner` */
+    winner: ComparisonWinner;
+}
+/** The statistical settings of a comparison */
+export type ComparisonStatsConfig = Pick<EvalComparisonConfig, 'alpha' | 'test' | 'seed'>;
+/** A case result as the comparison reads it: its error, its scores and its trials */
+export interface ScoredTrial {
+    error?: string;
+    scores: ReadonlyArray<{
+        name: string;
+        score: number;
+    }>;
+    trials?: readonly ScoredTrial[];
+}
+/** The two sides of one case, either missing when that side has no such case */
+export type CasePair<T extends ScoredTrial = ScoredTrial> = readonly [T | undefined, T | undefined];
+/** How the score of a case moved from one side to the other */
+export interface CaseChange {
+    /** Zero based position of the case in the pairs compared */
+    index: number;
+    /** The case input on one line, shortened */
+    input: string;
+    /** Score of the first side (the baseline, or the base run) */
+    before: number;
+    /** Score of the second side (the challenger, or the head run) */
+    after: number;
+    /** `after` minus `before` */
+    delta: number;
+}
+/** A score moves when it changes by more than this */
+export declare const SCORE_EPSILON = 1e-9;
+/**
+ * Compares each metric on its paired scores, then adjusts the p-values with Holm's correction
+ * across the metrics and decides each metric's winner
+ */
+export declare function compareMetrics(paired: ReadonlyMap<string, readonly [number[], number[]]>, config: ComparisonStatsConfig): Record<string, MetricComparison>;
+export declare function compareResource(baseline: number[], challenger: number[], config: ComparisonStatsConfig): ResourceComparison;
+/** The side more metrics name as their winner, a tie when as many name each */
+export declare function determineOverallWinner(metrics: Record<string, Pick<MetricComparison, 'winner'>>): ComparisonWinner;
+/** The names of the metrics any trial of the cases was scored on, in the order first seen */
+export declare function scoredMetricNames(results: Iterable<ScoredTrial>): Set<string>;
+/** A case's score of a metric: the mean over its trials the metric scored, none when it scored none */
+export declare function caseScore(result: ScoredTrial | undefined, metricName: string): number | undefined;
+/** The scores of a metric for each pair, side by side, as `pairedCaseScores` pairs them */
+export declare function pairedScores(pairs: readonly CasePair[], metricName: string): [number[], number[]];
+/**
+ * The cases whose score of a metric rose or fell from the first side to the second by more than
+ * `SCORE_EPSILON`, the largest moves first
+ */
+export declare function caseChanges(pairs: readonly CasePair[], metricName: string, inputOf: (index: number) => string): {
+    improved: CaseChange[];
+    regressed: CaseChange[];
+};
+//# sourceMappingURL=comparison-stats.d.ts.map

package/dist/comparison-stats.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"comparison-stats.d.ts","sourceRoot":"","sources":["../src/comparison-stats.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,UAAU,CAAC;AAWrD;;;;GAIG;AACH,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,eAAe,GAAG,GAAG,GAAG,UAAU,GAAG,MAAM,CAAC;AAErF,MAAM,MAAM,gBAAgB,GAAG,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;AAEjE,MAAM,WAAW,gBAAgB;IAC/B,4DAA4D;IAC5D,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,mFAAmF;IACnF,UAAU,EAAE,MAAM,CAAC;IACnB,8EAA8E;IAC9E,EAAE,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,6BAA6B;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,mEAAmE;IACnE,cAAc,EAAE,MAAM,CAAC;IACvB,wCAAwC;IACxC,WAAW,EAAE,OAAO,CAAC;IACrB;;;OAGG;IACH,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,+CAA+C;IAC/C,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,CAAC,EAAE,MAAM,CAAC;IACV,kFAAkF;IAClF,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,gGAAgG;IAChG,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED,+CAA+C;AAC/C,MAAM,MAAM,qBAAqB,GAAG,IAAI,CAAC,oBAAoB,EAAE,OAAO,GAAG,MAAM,GAAG,MAAM,CAAC,CAAC;AAE1F,qFAAqF;AACrF,MAAM,WAAW,WAAW;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,aAAa,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACvD,MAAM,CAAC,EAAE,SAAS,WAAW,EAAE,CAAC;CACjC;AAED,gFAAgF;AAChF,MAAM,MAAM,QAAQ,CAAC,CAAC,SAAS,WAAW,GAAG,WAAW,IAAI,SAAS,CAAC,CAAC,GAAG,SAAS,EAAE,CAAC,GAAG,SAAS,CAAC,CAAC;AAEpG,+DAA+D;AAC/D,MAAM,WAAW,UAAU;IACzB,4DAA4D;IAC5D,KAAK,EAAE,MAAM,CAAC;IACd,4CAA4C;IAC5C,KAAK,EAAE,MAAM,CAAC;IACd,8DAA8D;IAC9D,MAAM,EAAE,MAAM,CAAC;IACf,iEAAiE;IACjE,KAAK,EAAE,MAAM,CAAC;IACd,6BAA6B;IAC7B,KAAK,EAAE,MAAM,CAAC;CACf;AAED,sDAAsD;AACtD,eAAO,MAAM,aAAa,OAAO,CAAC;AAgFlC;;;GAGG;AACH,wBAAgB,cAAc,CAC5B,MAAM,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,EAC1D,MAAM,EAAE,qBAAqB,GAC5B,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAmBlC;AAED,wBAAgB,eAAe,CAC7B,QAAQ,EAAE,MAAM,EAAE,EAClB,UAAU,EAAE,MAAM,EAAE,EACpB,MAAM,EAAE,qBAAqB,GAC5B,kBAAkB,CAcpB;AAED,+EAA+E;AAC/E,wBAAgB,sBAAsB,CACpC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,IAAI,CAAC,gBAAgB,EAAE,QAAQ,CAAC,CAAC,GACxD,gBAAgB,CAYlB;AAED,6FAA6F;AAC7F,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,QAAQ,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC,MAAM,CAAC,CAQ7E;AAED,uGAAuG;AACvG,wBAAgB,SAAS,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,EAAE,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAUjG;AAqBD,2FAA2F;AAC3F,wBAAgB,YAAY,CAAC,KAAK,EAAE,SAAS,QAAQ,EAAE,EAAE,UAAU,EAAE,MAAM,GAAG,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,CAAC,CAGjG;AAED;;;GAGG;AACH,wBAAgB,WAAW,CACzB,KAAK,EAAE,SAAS,QAAQ,EAAE,EAC1B,UAAU,EAAE,MAAM,EAClB,OAAO,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,GACjC;IAAE,QAAQ,EAAE,UAAU,EAAE,CAAC;IAAC,SAAS,EAAE,UAAU,EAAE,CAAA;CAAE,CAYrD"}

package/dist/comparison-stats.js

@@ -0,0 +1,179 @@
+import { pairedTTest } from './stats/t-test.js';
+import { mcnemarExact, mcnemarsTest } from './stats/mcnemar.js';
+import { wilcoxonSignedRank } from './stats/wilcoxon.js';
+import { bootstrapCI } from './stats/bootstrap.js';
+import { cohensD } from './stats/effect-size.js';
+import { holm } from './stats/holm.js';
+import { mean } from './stats/percentiles.js';
+import { pairedDifference } from './stats/precision.js';
+import { truncateInput } from './reporters/format.js';
+/** A score moves when it changes by more than this */
+export const SCORE_EPSILON = 1e-9;
+function isBinary(scores) {
+    return scores.every((s) => s === 0 || s === 1);
+}
+/** Below this many discordant pairs McNemar's test is exact */
+const EXACT_MCNEMAR_BELOW = 25;
+function pairedTest(baseline, challenger, continuous, binaryAllowed) {
+    const direction = Math.sign(mean(differencesOf(baseline, challenger)));
+    if (baseline.length < 2)
+        return { test: 'none', pValue: 1, direction };
+    if (binaryAllowed && isBinary(baseline) && isBinary(challenger)) {
+        let b = 0;
+        let c = 0;
+        for (let i = 0; i < baseline.length; i++) {
+            if (baseline[i] === 1 && challenger[i] === 0)
+                b++;
+            if (baseline[i] === 0 && challenger[i] === 1)
+                c++;
+        }
+        return b + c < EXACT_MCNEMAR_BELOW
+            ? { test: 'mcnemar-exact', pValue: mcnemarExact(b, c).pValue, direction }
+            : { test: 'mcnemar', pValue: mcnemarsTest(b, c).pValue, direction };
+    }
+    if (continuous === 'wilcoxon') {
+        const { statistic, n, pValue } = wilcoxonSignedRank(baseline, challenger);
+        return { test: 'wilcoxon', pValue, direction: Math.sign((n * (n + 1)) / 4 - statistic) };
+    }
+    return { test: 't', pValue: pairedTTest(baseline, challenger).pValue, direction };
+}
+/** Challenger minus baseline per case, without float noise */
+function differencesOf(baseline, challenger) {
+    return challenger.map((score, i) => pairedDifference(score, baseline[i]));
+}
+function meanOrNaN(values) {
+    return values.length === 0 ? Number.NaN : mean(values);
+}
+function compareMetric(name, baselineScores, challengerScores, config) {
+    const differences = differencesOf(baselineScores, challengerScores);
+    const { low, high } = bootstrapCI(differences, {
+        confidence: 1 - config.alpha,
+        ...(config.seed !== undefined && { seed: `${String(config.seed)}:${name}` }),
+    });
+    return {
+        baseline: meanOrNaN(baselineScores),
+        challenger: meanOrNaN(challengerScores),
+        n: baselineScores.length,
+        difference: meanOrNaN(differences),
+        ci: { low, high },
+        effectSize: cohensD(differences),
+        ...pairedTest(baselineScores, challengerScores, config.test, true),
+    };
+}
+/**
+ * Compares each metric on its paired scores, then adjusts the p-values with Holm's correction
+ * across the metrics and decides each metric's winner
+ */
+export function compareMetrics(paired, config) {
+    const names = [...paired.keys()];
+    const compared = names.map((name) => {
+        const [baselineScores, challengerScores] = paired.get(name);
+        return compareMetric(name, baselineScores, challengerScores, config);
+    });
+    const adjusted = holm(compared.map((metric) => metric.pValue));
+    const decided = {};
+    for (const [at, name] of names.entries()) {
+        const { direction, ...metric } = compared[at];
+        const significant = adjusted[at] < config.alpha && direction !== 0;
+        decided[name] = {
+            ...metric,
+            adjustedPValue: adjusted[at],
+            significant,
+            winner: !significant ? 'tie' : direction > 0 ? 'challenger' : 'baseline',
+        };
+    }
+    return decided;
+}
+export function compareResource(baseline, challenger, config) {
+    const difference = meanOrNaN(differencesOf(baseline, challenger));
+    const { test, pValue, direction } = pairedTest(baseline, challenger, config.test, false);
+    const significant = pValue < config.alpha && direction !== 0;
+    return {
+        baseline: meanOrNaN(baseline),
+        challenger: meanOrNaN(challenger),
+        n: baseline.length,
+        difference,
+        test,
+        pValue,
+        significant,
+        winner: !significant ? 'tie' : direction < 0 ? 'challenger' : 'baseline',
+    };
+}
+/** The side more metrics name as their winner, a tie when as many name each */
+export function determineOverallWinner(metrics) {
+    let baselineWins = 0;
+    let challengerWins = 0;
+    for (const mc of Object.values(metrics)) {
+        if (mc.winner === 'baseline')
+            baselineWins++;
+        if (mc.winner === 'challenger')
+            challengerWins++;
+    }
+    if (challengerWins > baselineWins)
+        return 'challenger';
+    if (baselineWins > challengerWins)
+        return 'baseline';
+    return 'tie';
+}
+/** The names of the metrics any trial of the cases was scored on, in the order first seen */
+export function scoredMetricNames(results) {
+    const names = new Set();
+    for (const r of results) {
+        for (const trial of r.trials ?? [r]) {
+            for (const s of trial.scores)
+                names.add(s.name);
+        }
+    }
+    return names;
+}
+/** A case's score of a metric: the mean over its trials the metric scored, none when it scored none */
+export function caseScore(result, metricName) {
+    if (!result)
+        return undefined;
+    const trials = result.trials ?? [result];
+    if (trials.every((trial) => trial.error !== undefined))
+        return undefined;
+    const scores = trials.flatMap((trial) => trial.error !== undefined
+        ? [0]
+        : trial.scores.filter((s) => s.name === metricName).map((s) => s.score));
+    return scores.length === 0 ? undefined : mean(scores);
+}
+/**
+ * The scores of a metric for each pair, side by side, with the index of the pair. A case one side
+ * errored on counts as a 0 for that side, since a target that fails loses the case, and a case
+ * both sides errored on is left out, telling nothing about either
+ */
+function pairedCaseScores(pairs, metricName) {
+    const scored = [];
+    for (const [index, [first, second]] of pairs.entries()) {
+        const before = caseScore(first, metricName);
+        const after = caseScore(second, metricName);
+        if (before === undefined && after === undefined)
+            continue;
+        scored.push({ index, before: before ?? 0, after: after ?? 0 });
+    }
+    return scored;
+}
+/** The scores of a metric for each pair, side by side, as `pairedCaseScores` pairs them */
+export function pairedScores(pairs, metricName) {
+    const scored = pairedCaseScores(pairs, metricName);
+    return [scored.map((s) => s.before), scored.map((s) => s.after)];
+}
+/**
+ * The cases whose score of a metric rose or fell from the first side to the second by more than
+ * `SCORE_EPSILON`, the largest moves first
+ */
+export function caseChanges(pairs, metricName, inputOf) {
+    const improved = [];
+    const regressed = [];
+    for (const { index, before, after } of pairedCaseScores(pairs, metricName)) {
+        const delta = pairedDifference(after, before);
+        if (Math.abs(delta) <= SCORE_EPSILON)
+            continue;
+        const change = { index, input: truncateInput(inputOf(index)), before, after, delta };
+        (delta > 0 ? improved : regressed).push(change);
+    }
+    const byMove = (a, b) => Math.abs(b.delta) - Math.abs(a.delta) || a.index - b.index;
+    return { improved: improved.sort(byMove), regressed: regressed.sort(byMove) };
+}
+//# sourceMappingURL=comparison-stats.js.map

package/dist/comparison-stats.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"comparison-stats.js","sourceRoot":"","sources":["../src/comparison-stats.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC7C,OAAO,EAAE,YAAY,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAC7D,OAAO,EAAE,kBAAkB,EAAE,MAAM,kBAAkB,CAAC;AACtD,OAAO,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAChD,OAAO,EAAE,OAAO,EAAE,MAAM,qBAAqB,CAAC;AAC9C,OAAO,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACpC,OAAO,EAAE,IAAI,EAAE,MAAM,qBAAqB,CAAC;AAC3C,OAAO,EAAE,gBAAgB,EAAE,MAAM,mBAAmB,CAAC;AACrD,OAAO,EAAE,aAAa,EAAE,MAAM,oBAAoB,CAAC;AAkFnD,sDAAsD;AACtD,MAAM,CAAC,MAAM,aAAa,GAAG,IAAI,CAAC;AAElC,SAAS,QAAQ,CAAC,MAAgB;IAChC,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;AACjD,CAAC;AAED,+DAA+D;AAC/D,MAAM,mBAAmB,GAAG,EAAE,CAAC;AAY/B,SAAS,UAAU,CACjB,QAAkB,EAClB,UAAoB,EACpB,UAA4B,EAC5B,aAAsB;IAEtB,MAAM,SAAS,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,aAAa,CAAC,QAAQ,EAAE,UAAU,CAAC,CAAC,CAAC,CAAC;IACvE,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC;IACvE,IAAI,aAAa,IAAI,QAAQ,CAAC,QAAQ,CAAC,IAAI,QAAQ,CAAC,UAAU,CAAC,EAAE,CAAC;QAChE,IAAI,CAAC,GAAG,CAAC,CAAC;QACV,IAAI,CAAC,GAAG,CAAC,CAAC;QACV,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,QAAQ,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACzC,IAAI,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC;gBAAE,CAAC,EAAE,CAAC;YAClD,IAAI,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,UAAU,CAAC,CAAC,CAAC,KAAK,CAAC;gBAAE,CAAC,EAAE,CAAC;QACpD,CAAC;QACD,OAAO,CAAC,GAAG,CAAC,GAAG,mBAAmB;YAChC,CAAC,CAAC,EAAE,IAAI,EAAE,eAAe,EAAE,MAAM,EAAE,YAAY,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,EAAE,SAAS,EAAE;YACzE,CAAC,CAAC,EAAE,IAAI,EAAE,SAAS,EAAE,MAAM,EAAE,YAAY,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,EAAE,SAAS,EAAE,CAAC;IACxE,CAAC;IACD,IAAI,UAAU,KAAK,UAAU,EAAE,CAAC;QAC9B,MAAM,EAAE,SAAS,EAAE,CAAC,EAAE,MAAM,EAAE,GAAG,kBAAkB,CAAC,QAAQ,EAAE,UAAU,CAAC,CAAC;QAC1E,OAAO,EAAE,IAAI,EAAE,UAAU,EAAE,MAAM,EAAE,SAAS,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,SAAS,CAAC,EAAE,CAAC;IAC3F,CAAC;IACD,OAAO,EAAE,IAAI,EAAE,GAAG,EAAE,MAAM,EAAE,WAAW,CAAC,QAAQ,EAAE,UAAU,CAAC,CAAC,MAAM,EAAE,SAAS,EAAE,CAAC;AACpF,CAAC;AAED,8DAA8D;AAC9D,SAAS,aAAa,CAAC,QAAkB,EAAE,UAAoB;IAC7D,OAAO,UAAU,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,EAAE,EAAE,CAAC,gBAAgB,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AAC5E,CAAC;AAMD,SAAS,SAAS,CAAC,MAAgB;IACjC,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;AACzD,CAAC;AAED,SAAS,aAAa,CACpB,IAAY,EACZ,cAAwB,EACxB,gBAA0B,EAC1B,MAA6B;IAE7B,MAAM,WAAW,GAAG,aAAa,CAAC,cAAc,EAAE,gBAAgB,CAAC,CAAC;IACpE,MAAM,EAAE,GAAG,EAAE,IAAI,EAAE,GAAG,WAAW,CAAC,WAAW,EAAE;QAC7C,UAAU,EAAE,CAAC,GAAG,MAAM,CAAC,KAAK;QAC5B,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,SAAS,IAAI,EAAE,IAAI,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,EAAE,CAAC;KAC7E,CAAC,CAAC;IACH,OAAO;QACL,QAAQ,EAAE,SAAS,CAAC,cAAc,CAAC;QACnC,UAAU,EAAE,SAAS,CAAC,gBAAgB,CAAC;QACvC,CAAC,EAAE,cAAc,CAAC,MAAM;QACxB,UAAU,EAAE,SAAS,CAAC,WAAW,CAAC;QAClC,EAAE,EAAE,EAAE,GAAG,EAAE,IAAI,EAAE;QACjB,UAAU,EAAE,OAAO,CAAC,WAAW,CAAC;QAChC,GAAG,UAAU,CAAC,cAAc,EAAE,gBAAgB,EAAE,MAAM,CAAC,IAAI,EAAE,IAAI,CAAC;KACnE,CAAC;AACJ,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,cAAc,CAC5B,MAA0D,EAC1D,MAA6B;IAE7B,MAAM,KAAK,GAAG,CAAC,GAAG,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IACjC,MAAM,QAAQ,GAAG,KAAK,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;QAClC,MAAM,CAAC,cAAc,EAAE,gBAAgB,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,IAAI,CAAE,CAAC;QAC7D,OAAO,aAAa,CAAC,IAAI,EAAE,cAAc,EAAE,gBAAgB,EAAE,MAAM,CAAC,CAAC;IACvE,CAAC,CAAC,CAAC;IACH,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC;IAC/D,MAAM,OAAO,GAAqC,EAAE,CAAC;IACrD,KAAK,MAAM,CAAC,EAAE,EAAE,IAAI,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;QACzC,MAAM,EAAE,SAAS,EAAE,GAAG,MAAM,EAAE,GAAG,QAAQ,CAAC,EAAE,CAAC,CAAC;QAC9C,MAAM,WAAW,GAAG,QAAQ,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,KAAK,IAAI,SAAS,KAAK,CAAC,CAAC;QACnE,OAAO,CAAC,IAAI,CAAC,GAAG;YACd,GAAG,MAAM;YACT,cAAc,EAAE,QAAQ,CAAC,EAAE,CAAC;YAC5B,WAAW;YACX,MAAM,EAAE,CAAC,WAAW,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,UAAU;SACzE,CAAC;IACJ,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,MAAM,UAAU,eAAe,CAC7B,QAAkB,EAClB,UAAoB,EACpB,MAA6B;IAE7B,MAAM,UAAU,GAAG,SAAS,CAAC,aAAa,CAAC,QAAQ,EAAE,UAAU,CAAC,CAAC,CAAC;IAClE,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,SAAS,EAAE,GAAG,UAAU,CAAC,QAAQ,EAAE,UAAU,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;IACzF,MAAM,WAAW,GAAG,MAAM,GAAG,MAAM,CAAC,KAAK,IAAI,SAAS,KAAK,CAAC,CAAC;IAC7D,OAAO;QACL,QAAQ,EAAE,SAAS,CAAC,QAAQ,CAAC;QAC7B,UAAU,EAAE,SAAS,CAAC,UAAU,CAAC;QACjC,CAAC,EAAE,QAAQ,CAAC,MAAM;QAClB,UAAU;QACV,IAAI;QACJ,MAAM;QACN,WAAW;QACX,MAAM,EAAE,CAAC,WAAW,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,UAAU;KACzE,CAAC;AACJ,CAAC;AAED,+EAA+E;AAC/E,MAAM,UAAU,sBAAsB,CACpC,OAAyD;IAEzD,IAAI,YAAY,GAAG,CAAC,CAAC;IACrB,IAAI,cAAc,GAAG,CAAC,CAAC;IAEvB,KAAK,MAAM,EAAE,IAAI,MAAM,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;QACxC,IAAI,EAAE,CAAC,MAAM,KAAK,UAAU;YAAE,YAAY,EAAE,CAAC;QAC7C,IAAI,EAAE,CAAC,MAAM,KAAK,YAAY;YAAE,cAAc,EAAE,CAAC;IACnD,CAAC;IAED,IAAI,cAAc,GAAG,YAAY;QAAE,OAAO,YAAY,CAAC;IACvD,IAAI,YAAY,GAAG,cAAc;QAAE,OAAO,UAAU,CAAC;IACrD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,6FAA6F;AAC7F,MAAM,UAAU,iBAAiB,CAAC,OAA8B;IAC9D,MAAM,KAAK,GAAG,IAAI,GAAG,EAAU,CAAC;IAChC,KAAK,MAAM,CAAC,IAAI,OAAO,EAAE,CAAC;QACxB,KAAK,MAAM,KAAK,IAAI,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC;YACpC,KAAK,MAAM,CAAC,IAAI,KAAK,CAAC,MAAM;gBAAE,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,uGAAuG;AACvG,MAAM,UAAU,SAAS,CAAC,MAA+B,EAAE,UAAkB;IAC3E,IAAI,CAAC,MAAM;QAAE,OAAO,SAAS,CAAC;IAC9B,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,IAAI,CAAC,MAAM,CAAC,CAAC;IACzC,IAAI,MAAM,CAAC,KAAK,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,KAAK,KAAK,SAAS,CAAC;QAAE,OAAO,SAAS,CAAC;IACzE,MAAM,MAAM,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,CACtC,KAAK,CAAC,KAAK,KAAK,SAAS;QACvB,CAAC,CAAC,CAAC,CAAC,CAAC;QACL,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAC1E,CAAC;IACF,OAAO,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;AACxD,CAAC;AAED;;;;GAIG;AACH,SAAS,gBAAgB,CACvB,KAA0B,EAC1B,UAAkB;IAElB,MAAM,MAAM,GAA4D,EAAE,CAAC;IAC3E,KAAK,MAAM,CAAC,KAAK,EAAE,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC,IAAI,KAAK,CAAC,OAAO,EAAE,EAAE,CAAC;QACvD,MAAM,MAAM,GAAG,SAAS,CAAC,KAAK,EAAE,UAAU,CAAC,CAAC;QAC5C,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,EAAE,UAAU,CAAC,CAAC;QAC5C,IAAI,MAAM,KAAK,SAAS,IAAI,KAAK,KAAK,SAAS;YAAE,SAAS;QAC1D,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,IAAI,CAAC,EAAE,KAAK,EAAE,KAAK,IAAI,CAAC,EAAE,CAAC,CAAC;IACjE,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,2FAA2F;AAC3F,MAAM,UAAU,YAAY,CAAC,KAA0B,EAAE,UAAkB;IACzE,MAAM,MAAM,GAAG,gBAAgB,CAAC,KAAK,EAAE,UAAU,CAAC,CAAC;IACnD,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC;AACnE,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,WAAW,CACzB,KAA0B,EAC1B,UAAkB,EAClB,OAAkC;IAElC,MAAM,QAAQ,GAAiB,EAAE,CAAC;IAClC,MAAM,SAAS,GAAiB,EAAE,CAAC;IACnC,KAAK,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,IAAI,gBAAgB,CAAC,KAAK,EAAE,UAAU,CAAC,EAAE,CAAC;QAC3E,MAAM,KAAK,GAAG,gBAAgB,CAAC,KAAK,EAAE,MAAM,CAAC,CAAC;QAC9C,IAAI,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,aAAa;YAAE,SAAS;QAC/C,MAAM,MAAM,GAAG,EAAE,KAAK,EAAE,KAAK,EAAE,aAAa,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;QACrF,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAClD,CAAC;IACD,MAAM,MAAM,GAAG,CAAC,CAAa,EAAE,CAAa,EAAE,EAAE,CAC9C,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC;IAC7D,OAAO,EAAE,QAAQ,EAAE,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,SAAS,EAAE,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC;AAChF,CAAC"}

package/dist/eval-comparison.d.ts

@@ -3,6 +3,7 @@ import type { MetricFn, StatisticalMetricFn } from './metrics/types.js';
 import type { JudgeConfigInput } from './schema.js';
 import { Dataset } from './datasets/index.js';
 import type { PairwiseOptions, PairwiseResult } from './pairwise.js';
+import type { ComparisonWinner, MetricComparison, ResourceComparison } from './comparison-stats.js';
 export interface EvalComparisonOptions {
     dataset: Dataset;
     targets: {
@@ -48,57 +49,7 @@ export interface EvalComparisonRunOptions {
     /** Cancels both runs; `run()` rejects with the signal's reason */
     signal?: AbortSignal;
 }
-/**
- * The test a comparison ran: McNemar's (exact below 25 discordant pairs, chi-square with
- * continuity correction from 25), the paired t-test, Wilcoxon signed-rank, or none with fewer
- * than 2 cases
- */
-export type ComparisonTest = 'mcnemar' | 'mcnemar-exact' | 't' | 'wilcoxon' | 'none';
-export type ComparisonWinner = 'baseline' | 'challenger' | 'tie';
-export interface MetricComparison {
-    baseline: number;
-    challenger: number;
-    /** Cases compared: those at least one side scored, an errored side counting as a 0 */
-    n: number;
-    /** Mean of the paired differences, challenger minus baseline */
-    difference: number;
-    /** Percentile bootstrap confidence interval of `difference` at `1 - alpha` */
-    ci: {
-        low: number;
-        high: number;
-    };
-    /** Cohen's d of the paired differences (dz), NaN with fewer than 2 cases */
-    effectSize: number;
-    test: ComparisonTest;
-    /** The test's own p-value */
-    pValue: number;
-    /** `pValue` after Holm's correction across the compared metrics */
-    adjustedPValue: number;
-    /** `adjustedPValue` is below `alpha` */
-    significant: boolean;
-    /**
-     * The higher side when significant, by the test's direction: the sign of `difference`, except
-     * for Wilcoxon's test, where it is the side with the larger rank sum and may differ from it
-     */
-    winner: ComparisonWinner;
-}
-/**
- * Latency or cost of the two sides, paired over the cases both answered, where lower is better.
- * Not corrected for multiple comparisons and not part of the metrics' vote on the winner
- */
-export interface ResourceComparison {
-    /** Mean per case, NaN when no case pairs up */
-    baseline: number;
-    challenger: number;
-    n: number;
-    /** Challenger minus baseline: below 0 when the challenger is cheaper or faster */
-    difference: number;
-    test: ComparisonTest;
-    pValue: number;
-    significant: boolean;
-    /** The lower side when significant, by the test's direction as for `MetricComparison.winner` */
-    winner: ComparisonWinner;
-}
+export type { ComparisonTest, ComparisonWinner, MetricComparison, ResourceComparison, } from './comparison-stats.js';
 export interface ComparisonResult {
     summary: {
         winner: ComparisonWinner;

package/dist/eval-comparison.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAiC,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACpG,OAAO,KAAK,EAAwC,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEvF,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAIrC,OAAO,KAAK,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAWlE,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,gEAAgE;IAChE,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB;;;OAGG;IACH,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;;OAGG;IACH,IAAI,CAAC,EAAE,GAAG,GAAG,UAAU,CAAC;IACxB,4FAA4F;IAC5F,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IACvB;;;OAGG;IACH,QAAQ,CAAC,EAAE,eAAe,CAAC;IAC3B,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED;;;;GAIG;AACH,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,eAAe,GAAG,GAAG,GAAG,UAAU,GAAG,MAAM,CAAC;AAErF,MAAM,MAAM,gBAAgB,GAAG,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;AAEjE,MAAM,WAAW,gBAAgB;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,gEAAgE;IAChE,UAAU,EAAE,MAAM,CAAC;IACnB,8EAA8E;IAC9E,EAAE,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,6BAA6B;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,mEAAmE;IACnE,cAAc,EAAE,MAAM,CAAC;IACvB,wCAAwC;IACxC,WAAW,EAAE,OAAO,CAAC;IACrB;;;OAGG;IACH,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,+CAA+C;IAC/C,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,CAAC,EAAE,MAAM,CAAC;IACV,kFAAkF;IAClF,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,gGAAgG;IAChG,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,gBAAgB,CAAC;QACzB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;QAC1C,yEAAyE;QACzE,OAAO,EAAE,kBAAkB,CAAC;QAC5B,sFAAsF;QACtF,IAAI,CAAC,EAAE,kBAAkB,CAAC;KAC3B,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;IAC5B,gEAAgE;IAChE,QAAQ,CAAC,EAAE,cAAc,CAAC;CAC3B;AAuID,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;IAC7C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuB;IAC9C,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAe;gBAElC,IAAI,EAAE,qBAAqB;IAiCjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CAwG7E"}
+{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAoB,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACvF,OAAO,KAAK,EAAwC,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEvF,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAIrC,OAAO,KAAK,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAUlE,OAAO,KAAK,EAEV,gBAAgB,EAChB,gBAAgB,EAChB,kBAAkB,EACnB,MAAM,oBAAoB,CAAC;AAE5B,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,gEAAgE;IAChE,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB;;;OAGG;IACH,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;;OAGG;IACH,IAAI,CAAC,EAAE,GAAG,GAAG,UAAU,CAAC;IACxB,4FAA4F;IAC5F,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IACvB;;;OAGG;IACH,QAAQ,CAAC,EAAE,eAAe,CAAC;IAC3B,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,YAAY,EACV,cAAc,EACd,gBAAgB,EAChB,gBAAgB,EAChB,kBAAkB,GACnB,MAAM,oBAAoB,CAAC;AAE5B,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,gBAAgB,CAAC;QACzB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;QAC1C,yEAAyE;QACzE,OAAO,EAAE,kBAAkB,CAAC;QAC5B,sFAAsF;QACtF,IAAI,CAAC,EAAE,kBAAkB,CAAC;KAC3B,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;IAC5B,gEAAgE;IAChE,QAAQ,CAAC,EAAE,cAAc,CAAC;CAC3B;AAED,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;IAC7C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuB;IAC9C,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAe;gBAElC,IAAI,EAAE,qBAAqB;IAiCjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CA8F7E"}