@cogitator-ai/evals 0.11.0 → 0.13.0

package/README.md

@@ -13,20 +13,21 @@ pnpm add @cogitator-ai/core  # agent targets and LLM-as-judge metrics
 pnpm add -D vitest  # describeEval from @cogitator-ai/evals/vitest
 ```
 
-Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
+Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Experiments](https://cogitator.app/docs/evals/experiments), [Reporters](https://cogitator.app/docs/evals/reporters).
 
 ## Features
 
 - **EvalSuite** — Run datasets against agents or plain functions with configurable concurrency, timeouts, and retries
 - **8 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, tokenF1, regex, jsonSchema (Zod)
-- **6 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, custom llmMetric
+- **7 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, gEval, custom llmMetric with a free prompt or a discrete rubric. Prompts keep every untrusted value in escaped tags, `judge.samples` averages several verdicts, and `calibrateJudge()` checks a judge against labels
 - **4 Statistical Metrics** - latency, cost, tokenUsage with full percentile breakdowns, and passAtK over trials
 - **Custom Metrics** — `metric()` factory for anything domain-specific
 - **Agent Evaluation** - `toolCalls()` and `trajectory()` against expected tool calls, `retrievalRecall()`, `retrievalPrecision()`, `retrievalMRR()` and the `contextRelevance()` judge against expected passages, multi-turn cases on one conversation
 - **Trials and pass@k** - run each case several times and score how often it passes with `passAtK()`
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
-- **A/B Testing** — EvalComparison with paired t-test and McNemar's test for statistical significance
-- **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
+- **A/B Testing** - EvalComparison with McNemar's test (exact for few discordant pairs), the paired t-test or Wilcoxon signed-rank, Holm's correction across metrics, bootstrap confidence intervals, effect sizes, latency and cost comparisons and pairwise judging
+- **Experiment Tracking** - named runs with metadata saved as versioned run files, a `RunStore` directory of runs, `compareRuns()` with the A/B statistics, per-case regressions and improvements, and warnings when datasets, prompt versions, metrics or judges differ. Agent cases keep the `traceId` and versioned `prompt` of their runs
+- **8 Reporters** - console (colored table), JSON, CSV, JUnit XML, Markdown (pull request comments), self-contained HTML, GitHub Actions annotations and job summary, CI (exit code on failure), plus `reportComparison()` for comparisons
 - **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
 - **Baseline Workflow** — Save baselines, compare against them, catch regressions in CI
@@ -235,7 +236,8 @@ Metrics scored by an LLM judge (0.0 to 1.0). They require a `judge` config on th
 | `coherence`        | Logical structure and readability            |
 | `helpfulness`      | Practical usefulness to the user             |
 | `contextRelevance` | Whether the retrieved passages fit the input |
-| `llmMetric`        | Custom prompt, you define the criteria       |
+| `gEval`            | G-Eval: evaluation steps, then a scale       |
+| `llmMetric`        | Custom prompt or rubric, your criteria       |
 
 ```typescript
 import { faithfulness, relevance, llmMetric } from '@cogitator-ai/evals';
@@ -263,7 +265,43 @@ Dataset.from([{ input: 'Write the lede', context: { dossier: dossierText } }]);
 
 When the run retrieved passages (see [Retrieval](#retrieval)), `faithfulness()` checks the response against them instead of `context`. `contextRelevance()` judges the input against the retrieved passages alone, or against `context` when nothing was retrieved, and scores 0 without a judge call when the case has neither.
 
-The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
+The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency?, samples? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
+
+`llmMetric` also takes a discrete `rubric` instead of a free prompt: the judge picks one level, and the score is normalized to 0 - 1 by the lowest and highest level. A rubric needs at least two levels with distinct scores, and a verdict that is not a level counts as a judge error.
+
+```typescript
+llmMetric({
+  name: 'correctness',
+  prompt: 'Grade whether the response answers the question correctly.', // optional with a rubric
+  rubric: [
+    { score: 1, description: 'wrong, or does not answer' },
+    { score: 3, description: 'partly correct' },
+    { score: 5, description: 'fully correct' },
+  ],
+});
+```
+
+`gEval({ name, criteria, steps?, scale? })` follows G-Eval: the judge scores each case on `scale` (`[1, 5]` by default, normalized to 0 - 1) following evaluation steps. Without `steps`, the judge writes them from `criteria` once per metric instance, before the first case, and every later case reuses them. A failed generation scores the cases waiting on it as judge errors, and the next case tries again. The original G-Eval weights each score by the probability of its token, which not every provider exposes, so set `judge.samples` (with a `temperature` above 0) to average several verdicts instead.
+
+```typescript
+gEval({
+  name: 'clarity',
+  criteria: 'A beginner can follow the explanation, and it names the key idea first.',
+});
+```
+
+Every judge prompt keeps the untrusted values of a case (input, response, expected answer, context, retrieved passages, the turns of a conversation) inside XML-like tags such as `<input>` and `<response>`, with `<`, `>` and `&` escaped, so an answer cannot close its tag or open another, and the judge is told that tag contents are data to evaluate, never instructions. An answer that tells the judge to give it full marks is judged as content.
+
+`judge.samples` asks the judge that many times per case and metric (1 by default) and scores the mean of the verdicts it could read. The score's `details` give the spread (`mean of 3 verdicts, spread 0.6..0.9`), `metadata.samples` each verdict, and `usage` the summed cost. With `cache`, each sample is cached under its own key.
+
+`calibrateJudge(metric, examples, options?)` checks a judge against labels people gave: it scores each `{ input, output, expected?, context?, label }` (labels from 0 to 1) and returns `{ n, errors, pearson, spearman, mae, kappa, perCase }`, `kappa` being Cohen's kappa of pass and fail at `threshold` (0.5 by default). Pass a metric bound with `bindJudgeContext`, or `judge` with a `cogitator` to bind it. Examples the metric fails on are reported in `perCase` and left out of the figures.
+
+```typescript
+const calibration = await calibrateJudge(correctness, labelled, {
+  judge: { model: 'openai/gpt-6.1-sol', cogitator: cog },
+});
+console.log(calibration.spearman, calibration.kappa);
+```
 
 The judge answers with `{ "score", "reasoning" }` JSON. A verdict in a code fence or with the score as a numeric string is read as well, and an answer without JSON falls back to its last labelled score (`Score: 0.4`, `score is 4/5`), never to a stray number in the prose. When no verdict can be read, or the judge call fails, the metric scores 0 with the reason in `score.error`. The judge's usage is reported on each score and counted in `stats.judgeCost`.
 
@@ -505,6 +543,30 @@ for (const [name, mc] of Object.entries(result.summary.metrics)) {
 }
 ```
 
+Binary metrics (every score 0 or 1) take McNemar's test, exact below 25 discordant pairs and chi-square with continuity correction from 25. Other metrics take the paired t-test, or Wilcoxon's signed-rank test with `test: 'wilcoxon'` (zero differences dropped, exact up to 20 pairs, normal approximation above). Paired differences are rounded to 12 significant digits, so float noise neither splits ties nor hides a zero. The winner follows the test's direction: the higher mean for McNemar's and the t-test, the larger rank sum for Wilcoxon's, which can disagree with `difference`, the mean difference. The p-values are corrected with Holm's method across the compared metrics: `pValue` is the test's own, `adjustedPValue` the corrected one, and `significant` means `adjustedPValue < alpha` (`alpha` 0.05 by default). Each metric also reports `difference` (the mean of challenger minus baseline), `ci` (a percentile bootstrap interval of the paired differences at `1 - alpha`, the same on every run with `seed`), `effectSize` (Cohen's d of the paired differences) and the `test` it ran.
+
+`result.summary.latency` and `result.summary.cost` compare milliseconds and USD per case, paired over the cases both sides answered, where lower is better: means, `difference`, `pValue` and `winner`. `cost` is there only when both sides reported usage. Neither takes part in the metrics' vote on the overall winner.
+
+With `pairwise`, the judge also reads both answers of each case and says which is better, twice, the second time with the answers swapped. Agreeing verdicts give the case to that side, disagreeing ones or a tie make it a tie, a side that errored loses the case without a judge call, and a case both sides errored on is left out. A judge failure counts as a tie and is reported.
+
+```typescript
+const comparison = new EvalComparison({
+  dataset,
+  targets: { baseline, challenger },
+  metrics: [exactMatch()],
+  judge: { model: 'openai/gpt-6.1-sol', cogitator: cog },
+  pairwise: { criteria: 'Which answer is more accurate.' },
+  test: 'wilcoxon',
+  alpha: 0.05,
+  seed: 42,
+});
+
+const { pairwise } = await comparison.run();
+// { name, wins: { baseline, challenger }, ties, cases, winRate, judgeErrors, cost, perCase }
+```
+
+`winRate` is the challenger's wins over the decided cases, NaN when none was decided.
+
 Access full suite results via `result.baseline` and `result.challenger`. `comparison.run({ signal })` cancels both runs.
 
 ---
@@ -513,14 +575,16 @@ Access full suite results via `result.baseline` and `result.challenger`. `compar
 
 Call `result.report()` after a suite run to output results.
 
-| Reporter  | Output                                                               |
-| --------- | -------------------------------------------------------------------- |
-| `console` | Colored table with metrics, assertions, summary                      |
-| `json`    | Writes `eval-report.json` (configurable path)                        |
-| `csv`     | Writes `eval-report.csv` (configurable path)                         |
-| `junit`   | Writes `eval-report.xml`, a JUnit report (configurable path)         |
-| `github`  | GitHub Actions annotations, plus a job summary in Actions            |
-| `ci`      | Compact verdict, sets `process.exitCode = 1` when an assertion fails |
+| Reporter   | Output                                                                |
+| ---------- | --------------------------------------------------------------------- |
+| `console`  | Colored table with metrics, assertions, summary                       |
+| `json`     | Writes `eval-report.json` (configurable path)                         |
+| `csv`      | Writes `eval-report.csv` (configurable path)                          |
+| `junit`    | Writes `eval-report.xml`, a JUnit report (configurable path)          |
+| `github`   | GitHub Actions annotations, plus a job summary in Actions             |
+| `ci`       | Compact verdict, sets `process.exitCode = 1` when an assertion fails  |
+| `markdown` | Writes `eval-report.md`, a compact summary for a pull request comment |
+| `html`     | Writes `eval-report.html`, a self-contained page for light and dark   |
 
 ```typescript
 const result = await suite.run();
@@ -531,9 +595,11 @@ result.report(['console', 'json', 'csv', 'junit'], { path: './reports/eval' });
 result.report(['github', 'ci']);
 ```
 
-The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
+The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`, `.md`, `.html`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
 
 - **junit** writes one `<testsuite name="eval">` with a `<testcase>` per case (a `<failure>` when it errored, its scores in `<system-out>`) and a `<testcase>` per assertion (a `<failure>` when it did not pass). Everything is XML-escaped, and characters XML 1.0 does not allow are replaced.
+- **markdown** writes the totals (with the trials per case when there are several), each metric's mean and cases, the assertions with pass or fail, and the failures of every case and trial. Every value is escaped, so no answer breaks the tables.
+- **html** writes one page with inline CSS and no scripts or external assets: totals, metrics, assertions and every case with its scores, status, turns, trials, trace id and prompt version. Every value is HTML-escaped.
 - **github** prints `::error` for each failed assertion and errored case and `::warning` for each failed metric score. When `GITHUB_STEP_SUMMARY` is set it also appends a Markdown summary with the totals, the aggregated metrics (mean, p50, p95, cases scored) and the assertions.
 
 Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output. The json report keeps each case's `toolCalls`, `toolResults`, `retrieved`, `turns` and `trials`, and the console and github reports show the trials per case.
@@ -575,6 +641,41 @@ it('books a table', async () => {
 
 ---
 
+## Experiment Tracking
+
+Give a suite a `name` and JSON `metadata`, save each run as a run file, and compare runs later, across commits or prompt versions.
+
+```typescript
+import { EvalSuite, RunStore, compareRuns, reportComparison } from '@cogitator-ai/evals';
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [exactMatch(), faithfulness()],
+  judge: { model: 'google/gemini-3.5-flash-lite', cogitator },
+  name: 'support-agent',
+  metadata: { branch: process.env.GITHUB_HEAD_REF, commit: process.env.GITHUB_SHA },
+});
+
+const result = await suite.run();
+const store = new RunStore('./eval-runs');
+const base = store.latest({ name: 'support-agent' });
+store.save(result); // ./eval-runs/<runId>.json, as result.saveRun('./eval-runs') writes it
+
+if (base) {
+  const comparison = compareRuns(base, result, { seed: 42 });
+  reportComparison(comparison, ['console', 'markdown'], { path: './reports/comparison' });
+}
+```
+
+- **Run files.** `result.saveRun(target)` writes JSON with `format: 'cogitator.eval-run'` and `version: 1`: `runId`, `startedAt` and `finishedAt`, `name`, `metadata`, the dataset fingerprint (SHA-256 of the canonical JSON of the cases run), the config (metric names, judge model, temperature and samples, trials, concurrency, target kind, agent name and model), the prompt versions used, every case result with its `caseKey`, the aggregates, assertions and stats. A directory target (or one ending in `/`) gets `<runId>.json`. Functions, Cogitator instances and keys are never written. `result.toRun()` returns the same object, and `loadRun(path)` reads one back, validated with Zod.
+- **Traces and prompts.** Agent cases keep the `traceId` of their run (`RunResult.trace.traceId`) and the versioned instructions in `prompt` (`{ key, versionId?, version?, abTest? }`, set when the Cogitator versions instructions with `cogitator.prompts`). Each turn of a multi-turn case keeps its own and the case keeps the last turn's, each trial its own.
+- **RunStore.** `new RunStore(dir, { onSkip? })` keeps run files in a directory: `save(result | run)`, `list(filter?)` newest first with id, name, metadata, times and metric means, `load(id)`, and `latest(filter?)`, where a filter is `{ name }` or a predicate. `list()` and `latest()` skip `.json` entries they cannot use (a directory, a file that is not JSON, a run file of another version or edited by hand) and report each to `onSkip`, a process warning by default, while `load(id)` throws for them. A run is validated before it is written, so a hand-made `runId` cannot lead outside the directory.
+- **compareRuns(base, head, { alpha?, test?, seed?, metrics? }).** Cases are matched by `caseKey` (SHA-256 of `{ input, turns, expected }`), the others listed in `added` and `removed`. Each metric both runs have gets the statistics of `EvalComparison`: means, `difference`, `ci`, `effectSize`, `pValue`, `adjustedPValue` (Holm across the compared metrics), `significant` and `winner` (`head`, `base` or `tie`), plus the `improved` and `regressed` cases. An errored side counts as 0, a case both runs errored on is dropped. `warnings` say when the datasets, the prompt versions, the metrics or the judge differ, and name a metric left out because no matched case was scored on it.
+- **reportComparison(comparison, 'markdown' | 'html' | 'console', { path? }).** Renders a `compareRuns()` or `EvalComparison` result: the winner, a metric table with difference, CI, p and adjusted p and significance, the regressions and improvements, and the warnings. The default path is `eval-comparison`.
+
+---
+
 ## Builder API
 
 `EvalBuilder` provides a fluent interface for constructing eval suites.
@@ -657,7 +758,12 @@ result.report('ci');
 | ---------------- | ------------------------------------------------------------------- |
 | `EvalSuite`      | Main evaluation runner                                              |
 | `EvalComparison` | A/B testing runner with statistical significance                    |
+| `calibrateJudge` | Agreement of a judge metric with labelled examples                  |
 | `EvalBuilder`    | Fluent builder for EvalSuite                                        |
+| `compareRuns`    | Compares two saved runs or suite results case by case               |
+| `RunStore`       | Directory of run files: save, list, load, latest                    |
+| `loadRun`        | Reads and validates a run file                                      |
+| `caseKey`        | The key a case is matched by across runs                            |
 | `Dataset`        | Immutable dataset with from/fromJsonl/fromCsv/filter/sample/shuffle |
 | `loadJsonl`      | Low-level JSONL file loader                                         |
 | `loadCsv`        | Low-level CSV file loader                                           |
@@ -680,7 +786,8 @@ result.report('ci');
 | `coherence`          | LLM Judge     | Logical structure                                                  |
 | `helpfulness`        | LLM Judge     | Practical usefulness                                               |
 | `contextRelevance`   | LLM Judge     | Relevance of the retrieved passages to the input                   |
-| `llmMetric`          | LLM Judge     | Custom judge prompt                                                |
+| `gEval`              | LLM Judge     | Evaluation steps from criteria, then a score on a scale            |
+| `llmMetric`          | LLM Judge     | Custom judge prompt or discrete rubric                             |
 | `toolCalls`          | Agent         | Tool calls against `expectedToolCalls`, F1, recall or precision    |
 | `trajectory`         | Agent         | Sequence of tool names, exact, in-order or any-order               |
 | `retrievalRecall`    | Agent         | Share of `expectedPassages` retrieved                              |
@@ -704,37 +811,49 @@ result.report('ci');
 
 ### Reporters
 
-| Export                             | Description                              |
-| ---------------------------------- | ---------------------------------------- |
-| `report`                           | Dispatch to one or more reporter types   |
-| `describeEval` (`/vitest` entry)   | Run a suite as a Vitest describe block   |
-| `evalMatchers` (`/matchers` entry) | Matchers for `expect.extend()`, Vitest 5 |
+| Export                                 | Description                                      |
+| -------------------------------------- | ------------------------------------------------ |
+| `report`                               | Dispatch to one or more reporter types           |
+| `reportComparison`                     | Render a comparison as console, Markdown or HTML |
+| `comparisonMarkdown`, `comparisonHtml` | The Markdown or HTML of a comparison as a string |
+| `describeEval` (`/vitest` entry)       | Run a suite as a Vitest describe block           |
+| `evalMatchers` (`/matchers` entry)     | Matchers for `expect.extend()`, Vitest 5         |
 
-`report` takes `console`, `json`, `csv`, `junit`, `github` and `ci`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
+`report` takes `console`, `json`, `csv`, `junit`, `github`, `ci`, `markdown` and `html`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
 
 ### Types
 
-| Export             | Description                                                              |
-| ------------------ | ------------------------------------------------------------------------ |
-| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency` |
-| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`  |
-| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers              |
-| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on      |
-| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`              |
-| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`               |
-| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `error?`   |
+| Export             | Description                                                                                   |
+| ------------------ | --------------------------------------------------------------------------------------------- |
+| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency`                      |
+| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`                       |
+| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers                                   |
+| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on                           |
+| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`                                   |
+| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`                                    |
+| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `traceId?`, `prompt?`, `error?` |
+| `EvalPrompt`       | Versioned instructions of an agent run: `key`, `versionId?`, `version?`, `abTest?`            |
+| `EvalRun`          | A saved run, as `loadRun()` returns it                                                        |
+| `RunComparison`    | What `compareRuns()` returns                                                                  |
 
 ### Statistics
 
-| Export         | Description                                               |
-| -------------- | --------------------------------------------------------- |
-| `pairedTTest`  | Paired t-test for continuous metric comparison            |
-| `mcnemarsTest` | McNemar's test for binary metric comparison               |
-| `mean`         | Arithmetic mean                                           |
-| `median`       | Median value                                              |
-| `stdDev`       | Sample standard deviation                                 |
-| `percentile`   | Arbitrary percentile                                      |
-| `aggregate`    | Full stats: mean, median, min, max, stdDev, p50, p95, p99 |
+| Export               | Description                                                     |
+| -------------------- | --------------------------------------------------------------- |
+| `pairedTTest`        | Paired t-test for continuous metric comparison                  |
+| `wilcoxonSignedRank` | Wilcoxon signed-rank test, zeros dropped, exact up to 20 pairs  |
+| `mcnemarsTest`       | McNemar's test (chi-square, continuity corrected) for binary    |
+| `mcnemarExact`       | Exact McNemar's test, the binomial test of the discordant pairs |
+| `holm`               | Holm-Bonferroni adjusted p-values                               |
+| `bootstrapCI`        | Seeded percentile bootstrap confidence interval of the mean     |
+| `cohensD`            | Cohen's d of paired differences (dz)                            |
+| `pearson`            | Pearson's correlation                                           |
+| `spearman`           | Spearman's rank correlation                                     |
+| `mean`               | Arithmetic mean                                                 |
+| `median`             | Median value                                                    |
+| `stdDev`             | Sample standard deviation                                       |
+| `percentile`         | Arbitrary percentile                                            |
+| `aggregate`          | Full stats: mean, median, min, max, stdDev, p50, p95, p99       |
 
 ### Agent Tools
 

package/dist/calibrate-judge.d.ts

@@ -0,0 +1,53 @@
+import type { MetricFn } from './metrics/types.js';
+import type { JudgeConfigInput } from './schema.js';
+/** An answer labelled by a person, with the score the metric should give it, from 0 to 1 */
+export interface CalibrationExample {
+    input: string;
+    output: string;
+    expected?: string;
+    context?: Record<string, unknown>;
+    label: number;
+}
+export interface CalibrateJudgeOptions {
+    /** Where a score or label counts as a pass for `kappa`, 0.5 by default */
+    threshold?: number;
+    /** Binds an unbound judge metric, run by `judge.cogitator` */
+    judge?: JudgeConfigInput;
+    /** Examples scored at once, 5 by default */
+    concurrency?: number;
+}
+export interface CalibrationCase {
+    /** Position of the example */
+    index: number;
+    label: number;
+    /** The metric's score, absent when it failed */
+    score?: number;
+    details?: string;
+    error?: string;
+}
+export interface JudgeCalibration {
+    /** Examples the metric scored, those it failed on left out of the figures */
+    n: number;
+    /** Examples the metric failed on */
+    errors: number;
+    /** Pearson's correlation of scores and labels, NaN when either does not vary */
+    pearson: number;
+    /** Spearman's rank correlation of scores and labels */
+    spearman: number;
+    /** Mean absolute error of the scores against the labels */
+    mae: number;
+    /**
+     * Cohen's kappa of pass and fail, a score or label passing at `threshold` or more. NaN when
+     * agreement by chance is certain, every score and label being on one side
+     */
+    kappa: number;
+    perCase: CalibrationCase[];
+}
+/**
+ * How well a metric, usually an LLM judge, agrees with labels people gave: it scores each
+ * labelled example and compares the scores with the labels. Bind a judge metric first with
+ * `bindJudgeContext`, or pass `judge` with a `cogitator`. Examples the metric fails on are
+ * reported and left out of the figures
+ */
+export declare function calibrateJudge(metric: MetricFn, examples: readonly CalibrationExample[], options?: CalibrateJudgeOptions): Promise<JudgeCalibration>;
+//# sourceMappingURL=calibrate-judge.d.ts.map

package/dist/calibrate-judge.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"calibrate-judge.d.ts","sourceRoot":"","sources":["../src/calibrate-judge.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAkB,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAIhE,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAKjD,4FAA4F;AAC5F,MAAM,WAAW,kBAAkB;IACjC,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAClC,KAAK,EAAE,MAAM,CAAC;CACf;AAED,MAAM,WAAW,qBAAqB;IACpC,0EAA0E;IAC1E,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,4CAA4C;IAC5C,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,eAAe;IAC9B,8BAA8B;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,gDAAgD;IAChD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,6EAA6E;IAC7E,CAAC,EAAE,MAAM,CAAC;IACV,oCAAoC;IACpC,MAAM,EAAE,MAAM,CAAC;IACf,gFAAgF;IAChF,OAAO,EAAE,MAAM,CAAC;IAChB,uDAAuD;IACvD,QAAQ,EAAE,MAAM,CAAC;IACjB,2DAA2D;IAC3D,GAAG,EAAE,MAAM,CAAC;IACZ;;;OAGG;IACH,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,eAAe,EAAE,CAAC;CAC5B;AA6ED;;;;;GAKG;AACH,wBAAsB,cAAc,CAClC,MAAM,EAAE,QAAQ,EAChB,QAAQ,EAAE,SAAS,kBAAkB,EAAE,EACvC,OAAO,GAAE,qBAA0B,GAClC,OAAO,CAAC,gBAAgB,CAAC,CAyC3B"}

package/dist/calibrate-judge.js

@@ -0,0 +1,114 @@
+import { z } from 'zod';
+import { isLLMMetric } from './eval-suite.js';
+import { nonFiniteScoreError } from './metrics/types.js';
+import { bindJudgeContext, judgeContextFor } from './metrics/llm-judge.js';
+import { JudgeConfigSchema } from './schema.js';
+import { parseOptions } from './options.js';
+import { pearson, spearman } from './stats/correlation.js';
+import { mean } from './stats/percentiles.js';
+const CalibrationExampleSchema = z.object({
+    input: z.string(),
+    output: z.string(),
+    expected: z.string().optional(),
+    context: z.record(z.string(), z.unknown()).optional(),
+    label: z.number().min(0).max(1),
+});
+const CalibrateJudgeOptionsSchema = z.object({
+    threshold: z.number().min(0).max(1).default(0.5),
+    concurrency: z.number().int().positive().default(5),
+});
+/** Cohen's kappa of two binary raters */
+function binaryKappa(a, b) {
+    const n = a.length;
+    if (n === 0)
+        return Number.NaN;
+    let agree = 0;
+    let aPass = 0;
+    let bPass = 0;
+    for (let i = 0; i < n; i++) {
+        if (a[i] === b[i])
+            agree++;
+        if (a[i])
+            aPass++;
+        if (b[i])
+            bPass++;
+    }
+    const observed = agree / n;
+    const chance = (aPass / n) * (bPass / n) + (1 - aPass / n) * (1 - bPass / n);
+    return chance === 1 ? Number.NaN : (observed - chance) / (1 - chance);
+}
+function bindMetric(metric, judge) {
+    if (!isLLMMetric(metric))
+        return metric;
+    if (!judge) {
+        throw new Error('calibrateJudge needs options.judge to run an unbound judge metric');
+    }
+    const config = JudgeConfigSchema.parse(judge);
+    if (!config.cogitator) {
+        throw new Error('calibrateJudge needs judge.cogitator to run the judge');
+    }
+    return bindJudgeContext(metric, judgeContextFor(config.cogitator, config));
+}
+async function scoreExample(metric, example, index) {
+    const result = {
+        case: {
+            input: example.input,
+            ...(example.expected !== undefined && { expected: example.expected }),
+            ...(example.context !== undefined && { context: example.context }),
+        },
+        output: example.output,
+        duration: 0,
+    };
+    try {
+        const scored = await metric(result);
+        const invalid = scored.error ?? nonFiniteScoreError(scored.score);
+        if (invalid !== undefined)
+            return { index, label: example.label, error: invalid };
+        return {
+            index,
+            label: example.label,
+            score: scored.score,
+            ...(scored.details !== undefined && { details: scored.details }),
+        };
+    }
+    catch (error) {
+        return {
+            index,
+            label: example.label,
+            error: error instanceof Error ? error.message : String(error),
+        };
+    }
+}
+/**
+ * How well a metric, usually an LLM judge, agrees with labels people gave: it scores each
+ * labelled example and compares the scores with the labels. Bind a judge metric first with
+ * `bindJudgeContext`, or pass `judge` with a `cogitator`. Examples the metric fails on are
+ * reported and left out of the figures
+ */
+export async function calibrateJudge(metric, examples, options = {}) {
+    const { threshold, concurrency } = parseOptions(CalibrateJudgeOptionsSchema, { threshold: options.threshold, concurrency: options.concurrency }, 'calibrateJudge');
+    const parsed = parseOptions(z.array(CalibrationExampleSchema), examples, 'calibrateJudge examples');
+    const bound = bindMetric(metric, options.judge);
+    const perCase = new Array(parsed.length);
+    let next = 0;
+    const worker = async () => {
+        while (next < parsed.length) {
+            const at = next++;
+            perCase[at] = await scoreExample(bound, parsed[at], at);
+        }
+    };
+    await Promise.all(Array.from({ length: Math.min(concurrency, parsed.length) }, worker));
+    const scored = perCase.filter((one) => one.score !== undefined);
+    const scores = scored.map((one) => one.score);
+    const labels = scored.map((one) => one.label);
+    return {
+        n: scored.length,
+        errors: perCase.length - scored.length,
+        pearson: pearson(scores, labels),
+        spearman: spearman(scores, labels),
+        mae: scored.length === 0 ? Number.NaN : mean(scored.map((one) => Math.abs(one.score - one.label))),
+        kappa: binaryKappa(scores.map((score) => score >= threshold), labels.map((label) => label >= threshold)),
+        perCase,
+    };
+}
+//# sourceMappingURL=calibrate-judge.js.map

package/dist/calibrate-judge.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"calibrate-judge.js","sourceRoot":"","sources":["../src/calibrate-judge.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AACxB,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAE3C,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,gBAAgB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AACxE,OAAO,EAAE,iBAAiB,EAAE,MAAM,UAAU,CAAC;AAE7C,OAAO,EAAE,YAAY,EAAE,MAAM,WAAW,CAAC;AACzC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,qBAAqB,CAAC;AAiD3C,MAAM,wBAAwB,GAAG,CAAC,CAAC,MAAM,CAAC;IACxC,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE;IACjB,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE;IAClB,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;IAC/B,OAAO,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC,QAAQ,EAAE;IACrD,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;CAChC,CAAC,CAAC;AAEH,MAAM,2BAA2B,GAAG,CAAC,CAAC,MAAM,CAAC;IAC3C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC;IAChD,WAAW,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;CACpD,CAAC,CAAC;AAEH,yCAAyC;AACzC,SAAS,WAAW,CAAC,CAAqB,EAAE,CAAqB;IAC/D,MAAM,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC;IACnB,IAAI,CAAC,KAAK,CAAC;QAAE,OAAO,MAAM,CAAC,GAAG,CAAC;IAC/B,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC3B,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;QAC3B,IAAI,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;IACpB,CAAC;IACD,MAAM,QAAQ,GAAG,KAAK,GAAG,CAAC,CAAC;IAC3B,MAAM,MAAM,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,CAAC,CAAC;IAC7E,OAAO,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC;AACxE,CAAC;AAED,SAAS,UAAU,CAAC,MAAgB,EAAE,KAAmC;IACvE,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC;QAAE,OAAO,MAAM,CAAC;IACxC,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,MAAM,IAAI,KAAK,CAAC,mEAAmE,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,MAAM,GAAG,iBAAiB,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;IAC9C,IAAI,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC;QACtB,MAAM,IAAI,KAAK,CAAC,uDAAuD,CAAC,CAAC;IAC3E,CAAC;IACD,OAAO,gBAAgB,CAAC,MAAM,EAAE,eAAe,CAAC,MAAM,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC;AAC7E,CAAC;AAED,KAAK,UAAU,YAAY,CACzB,MAAgB,EAChB,OAAkD,EAClD,KAAa;IAEb,MAAM,MAAM,GAAmB;QAC7B,IAAI,EAAE;YACJ,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,IAAI,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC;YACrE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,SAAS,IAAI,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,CAAC;SACnE;QACD,MAAM,EAAE,OAAO,CAAC,MAAM;QACtB,QAAQ,EAAE,CAAC;KACZ,CAAC;IACF,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,MAAM,CAAC,CAAC;QACpC,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,IAAI,mBAAmB,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QAClE,IAAI,OAAO,KAAK,SAAS;YAAE,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;QAClF,OAAO;YACL,KAAK;YACL,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,KAAK,EAAE,MAAM,CAAC,KAAK;YACnB,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,IAAI,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC;SACjE,CAAC;IACJ,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO;YACL,KAAK;YACL,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,KAAK,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC;SAC9D,CAAC;IACJ,CAAC;AACH,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,MAAgB,EAChB,QAAuC,EACvC,UAAiC,EAAE;IAEnC,MAAM,EAAE,SAAS,EAAE,WAAW,EAAE,GAAG,YAAY,CAC7C,2BAA2B,EAC3B,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,WAAW,EAAE,OAAO,CAAC,WAAW,EAAE,EAClE,gBAAgB,CACjB,CAAC;IACF,MAAM,MAAM,GAAG,YAAY,CACzB,CAAC,CAAC,KAAK,CAAC,wBAAwB,CAAC,EACjC,QAAQ,EACR,yBAAyB,CAC1B,CAAC;IACF,MAAM,KAAK,GAAG,UAAU,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IAEhD,MAAM,OAAO,GAAG,IAAI,KAAK,CAAkB,MAAM,CAAC,MAAM,CAAC,CAAC;IAC1D,IAAI,IAAI,GAAG,CAAC,CAAC;IACb,MAAM,MAAM,GAAG,KAAK,IAAI,EAAE;QACxB,OAAO,IAAI,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC;YAC5B,MAAM,EAAE,GAAG,IAAI,EAAE,CAAC;YAClB,OAAO,CAAC,EAAE,CAAC,GAAG,MAAM,YAAY,CAAC,KAAK,EAAE,MAAM,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC,CAAC;IACF,MAAM,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,MAAM,CAAC,MAAM,CAAC,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IAExF,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAC3B,CAAC,GAAG,EAA8C,EAAE,CAAC,GAAG,CAAC,KAAK,KAAK,SAAS,CAC7E,CAAC;IACF,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IAC9C,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IAC9C,OAAO;QACL,CAAC,EAAE,MAAM,CAAC,MAAM;QAChB,MAAM,EAAE,OAAO,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM;QACtC,OAAO,EAAE,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC;QAChC,QAAQ,EAAE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;QAClC,GAAG,EACD,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC;QAC/F,KAAK,EAAE,WAAW,CAChB,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,SAAS,CAAC,EACzC,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,SAAS,CAAC,CAC1C;QACD,OAAO;KACR,CAAC;AACJ,CAAC"}

package/dist/comparison-stats.d.ts

@@ -0,0 +1,104 @@
+import type { EvalComparisonConfig } from './schema.js';
+/**
+ * The test a comparison ran: McNemar's (exact below 25 discordant pairs, chi-square with
+ * continuity correction from 25), the paired t-test, Wilcoxon signed-rank, or none with fewer
+ * than 2 cases
+ */
+export type ComparisonTest = 'mcnemar' | 'mcnemar-exact' | 't' | 'wilcoxon' | 'none';
+export type ComparisonWinner = 'baseline' | 'challenger' | 'tie';
+export interface MetricComparison {
+    /** Mean score over the compared cases, NaN when `n` is 0 */
+    baseline: number;
+    challenger: number;
+    /** Cases compared: those at least one side scored, an errored side counting as a 0 */
+    n: number;
+    /** Mean of the paired differences, challenger minus baseline, NaN when `n` is 0 */
+    difference: number;
+    /** Percentile bootstrap confidence interval of `difference` at `1 - alpha` */
+    ci: {
+        low: number;
+        high: number;
+    };
+    /** Cohen's d of the paired differences (dz), NaN with fewer than 2 cases */
+    effectSize: number;
+    test: ComparisonTest;
+    /** The test's own p-value */
+    pValue: number;
+    /** `pValue` after Holm's correction across the compared metrics */
+    adjustedPValue: number;
+    /** `adjustedPValue` is below `alpha` */
+    significant: boolean;
+    /**
+     * The higher side when significant, by the test's direction: the sign of `difference`, except
+     * for Wilcoxon's test, where it is the side with the larger rank sum and may differ from it
+     */
+    winner: ComparisonWinner;
+}
+/**
+ * Latency or cost of the two sides, paired over the cases both answered, where lower is better.
+ * Not corrected for multiple comparisons and not part of the metrics' vote on the winner
+ */
+export interface ResourceComparison {
+    /** Mean per case, NaN when no case pairs up */
+    baseline: number;
+    challenger: number;
+    n: number;
+    /** Challenger minus baseline: below 0 when the challenger is cheaper or faster */
+    difference: number;
+    test: ComparisonTest;
+    pValue: number;
+    significant: boolean;
+    /** The lower side when significant, by the test's direction as for `MetricComparison.winner` */
+    winner: ComparisonWinner;
+}
+/** The statistical settings of a comparison */
+export type ComparisonStatsConfig = Pick<EvalComparisonConfig, 'alpha' | 'test' | 'seed'>;
+/** A case result as the comparison reads it: its error, its scores and its trials */
+export interface ScoredTrial {
+    error?: string;
+    scores: ReadonlyArray<{
+        name: string;
+        score: number;
+    }>;
+    trials?: readonly ScoredTrial[];
+}
+/** The two sides of one case, either missing when that side has no such case */
+export type CasePair<T extends ScoredTrial = ScoredTrial> = readonly [T | undefined, T | undefined];
+/** How the score of a case moved from one side to the other */
+export interface CaseChange {
+    /** Zero based position of the case in the pairs compared */
+    index: number;
+    /** The case input on one line, shortened */
+    input: string;
+    /** Score of the first side (the baseline, or the base run) */
+    before: number;
+    /** Score of the second side (the challenger, or the head run) */
+    after: number;
+    /** `after` minus `before` */
+    delta: number;
+}
+/** A score moves when it changes by more than this */
+export declare const SCORE_EPSILON = 1e-9;
+/**
+ * Compares each metric on its paired scores, then adjusts the p-values with Holm's correction
+ * across the metrics and decides each metric's winner
+ */
+export declare function compareMetrics(paired: ReadonlyMap<string, readonly [number[], number[]]>, config: ComparisonStatsConfig): Record<string, MetricComparison>;
+export declare function compareResource(baseline: number[], challenger: number[], config: ComparisonStatsConfig): ResourceComparison;
+/** The side more metrics name as their winner, a tie when as many name each */
+export declare function determineOverallWinner(metrics: Record<string, Pick<MetricComparison, 'winner'>>): ComparisonWinner;
+/** The names of the metrics any trial of the cases was scored on, in the order first seen */
+export declare function scoredMetricNames(results: Iterable<ScoredTrial>): Set<string>;
+/** A case's score of a metric: the mean over its trials the metric scored, none when it scored none */
+export declare function caseScore(result: ScoredTrial | undefined, metricName: string): number | undefined;
+/** The scores of a metric for each pair, side by side, as `pairedCaseScores` pairs them */
+export declare function pairedScores(pairs: readonly CasePair[], metricName: string): [number[], number[]];
+/**
+ * The cases whose score of a metric rose or fell from the first side to the second by more than
+ * `SCORE_EPSILON`, the largest moves first
+ */
+export declare function caseChanges(pairs: readonly CasePair[], metricName: string, inputOf: (index: number) => string): {
+    improved: CaseChange[];
+    regressed: CaseChange[];
+};
+//# sourceMappingURL=comparison-stats.d.ts.map

package/dist/comparison-stats.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"comparison-stats.d.ts","sourceRoot":"","sources":["../src/comparison-stats.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,UAAU,CAAC;AAWrD;;;;GAIG;AACH,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,eAAe,GAAG,GAAG,GAAG,UAAU,GAAG,MAAM,CAAC;AAErF,MAAM,MAAM,gBAAgB,GAAG,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;AAEjE,MAAM,WAAW,gBAAgB;IAC/B,4DAA4D;IAC5D,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,mFAAmF;IACnF,UAAU,EAAE,MAAM,CAAC;IACnB,8EAA8E;IAC9E,EAAE,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,6BAA6B;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,mEAAmE;IACnE,cAAc,EAAE,MAAM,CAAC;IACvB,wCAAwC;IACxC,WAAW,EAAE,OAAO,CAAC;IACrB;;;OAGG;IACH,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,+CAA+C;IAC/C,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,CAAC,EAAE,MAAM,CAAC;IACV,kFAAkF;IAClF,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,gGAAgG;IAChG,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED,+CAA+C;AAC/C,MAAM,MAAM,qBAAqB,GAAG,IAAI,CAAC,oBAAoB,EAAE,OAAO,GAAG,MAAM,GAAG,MAAM,CAAC,CAAC;AAE1F,qFAAqF;AACrF,MAAM,WAAW,WAAW;IAC1B,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,aAAa,CAAC;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACvD,MAAM,CAAC,EAAE,SAAS,WAAW,EAAE,CAAC;CACjC;AAED,gFAAgF;AAChF,MAAM,MAAM,QAAQ,CAAC,CAAC,SAAS,WAAW,GAAG,WAAW,IAAI,SAAS,CAAC,CAAC,GAAG,SAAS,EAAE,CAAC,GAAG,SAAS,CAAC,CAAC;AAEpG,+DAA+D;AAC/D,MAAM,WAAW,UAAU;IACzB,4DAA4D;IAC5D,KAAK,EAAE,MAAM,CAAC;IACd,4CAA4C;IAC5C,KAAK,EAAE,MAAM,CAAC;IACd,8DAA8D;IAC9D,MAAM,EAAE,MAAM,CAAC;IACf,iEAAiE;IACjE,KAAK,EAAE,MAAM,CAAC;IACd,6BAA6B;IAC7B,KAAK,EAAE,MAAM,CAAC;CACf;AAED,sDAAsD;AACtD,eAAO,MAAM,aAAa,OAAO,CAAC;AAgFlC;;;GAGG;AACH,wBAAgB,cAAc,CAC5B,MAAM,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,EAC1D,MAAM,EAAE,qBAAqB,GAC5B,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAmBlC;AAED,wBAAgB,eAAe,CAC7B,QAAQ,EAAE,MAAM,EAAE,EAClB,UAAU,EAAE,MAAM,EAAE,EACpB,MAAM,EAAE,qBAAqB,GAC5B,kBAAkB,CAcpB;AAED,+EAA+E;AAC/E,wBAAgB,sBAAsB,CACpC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,IAAI,CAAC,gBAAgB,EAAE,QAAQ,CAAC,CAAC,GACxD,gBAAgB,CAYlB;AAED,6FAA6F;AAC7F,wBAAgB,iBAAiB,CAAC,OAAO,EAAE,QAAQ,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC,MAAM,CAAC,CAQ7E;AAED,uGAAuG;AACvG,wBAAgB,SAAS,CAAC,MAAM,EAAE,WAAW,GAAG,SAAS,EAAE,UAAU,EAAE,MAAM,GAAG,MAAM,GAAG,SAAS,CAUjG;AAqBD,2FAA2F;AAC3F,wBAAgB,YAAY,CAAC,KAAK,EAAE,SAAS,QAAQ,EAAE,EAAE,UAAU,EAAE,MAAM,GAAG,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,CAAC,CAGjG;AAED;;;GAGG;AACH,wBAAgB,WAAW,CACzB,KAAK,EAAE,SAAS,QAAQ,EAAE,EAC1B,UAAU,EAAE,MAAM,EAClB,OAAO,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,MAAM,GACjC;IAAE,QAAQ,EAAE,UAAU,EAAE,CAAC;IAAC,SAAS,EAAE,UAAU,EAAE,CAAA;CAAE,CAYrD"}