@cogitator-ai/evals 0.10.0 → 0.12.0

package/README.md

@@ -18,14 +18,14 @@ Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogi
 ## Features
 
 - **EvalSuite** — Run datasets against agents or plain functions with configurable concurrency, timeouts, and retries
-- **7 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, regex, jsonSchema (Zod)
-- **6 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, custom llmMetric
+- **8 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, tokenF1, regex, jsonSchema (Zod)
+- **7 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, gEval, custom llmMetric with a free prompt or a discrete rubric. Prompts keep every untrusted value in escaped tags, `judge.samples` averages several verdicts, and `calibrateJudge()` checks a judge against labels
 - **4 Statistical Metrics** - latency, cost, tokenUsage with full percentile breakdowns, and passAtK over trials
 - **Custom Metrics** — `metric()` factory for anything domain-specific
 - **Agent Evaluation** - `toolCalls()` and `trajectory()` against expected tool calls, `retrievalRecall()`, `retrievalPrecision()`, `retrievalMRR()` and the `contextRelevance()` judge against expected passages, multi-turn cases on one conversation
 - **Trials and pass@k** - run each case several times and score how often it passes with `passAtK()`
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
-- **A/B Testing** — EvalComparison with paired t-test and McNemar's test for statistical significance
+- **A/B Testing** - EvalComparison with McNemar's test (exact for few discordant pairs), the paired t-test or Wilcoxon signed-rank, Holm's correction across metrics, bootstrap confidence intervals, effect sizes, latency and cost comparisons and pairwise judging
 - **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
 - **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
@@ -176,6 +176,7 @@ Metrics that compare output against expected values using string operations or s
 | `containsAll` | Fraction of configured keywords found          | No                  |
 | `containsAny` | Whether any configured keyword is found        | No                  |
 | `similarity`  | Levenshtein distance similarity score (0 to 1) | Yes                 |
+| `tokenF1`     | Token-level F1 score (precision and recall)    | Yes                 |
 | `regex`       | Output matches a regex pattern                 | No                  |
 | `jsonSchema`  | Output is valid JSON matching a Zod schema     | No                  |
 
@@ -186,6 +187,7 @@ import {
   containsAll,
   containsAny,
   similarity,
+  tokenF1,
   regex,
   jsonSchema,
   threshold,
@@ -198,6 +200,7 @@ const metrics = [
   containsAll(['security', 'privacy', 'compliance']),
   containsAny(['approved', 'accepted']),
   similarity(),
+  tokenF1(),
   regex(/\d{4}-\d{2}-\d{2}/),
   jsonSchema(z.object({ answer: z.string(), confidence: z.number() })),
 ];
@@ -217,6 +220,8 @@ const metrics = [
 
 `containsAll()` scores the fraction of its keywords found and reports any missing keywords in `details`. `containsAny()` scores 1 when at least one keyword is found and 0 otherwise. Keywords use substring matching, so `cat` matches `concatenate`. Both metrics are case-insensitive by default, accept `caseSensitive` and `name` options, ignore duplicate keywords, and reject empty lists or empty and whitespace-only keywords when the metric is created.
 
+`tokenF1()` computes token overlap using multiset counts so duplicate tokens are handled correctly, returning the harmonic mean of precision and recall. It lowercases unless `caseSensitive: true`, strips punctuation, removes standalone articles (`a`, `an`, `the`), splits on whitespace, scores 1 for both empty strings, and reports precision and recall in `details` whenever the score is below 1.
+
 `regex()` ignores the `g` and `y` flags, which would make identical outputs alternate between pass and fail.
 
 ### LLM-as-Judge
@@ -230,7 +235,8 @@ Metrics scored by an LLM judge (0.0 to 1.0). They require a `judge` config on th
 | `coherence`        | Logical structure and readability            |
 | `helpfulness`      | Practical usefulness to the user             |
 | `contextRelevance` | Whether the retrieved passages fit the input |
-| `llmMetric`        | Custom prompt, you define the criteria       |
+| `gEval`            | G-Eval: evaluation steps, then a scale       |
+| `llmMetric`        | Custom prompt or rubric, your criteria       |
 
 ```typescript
 import { faithfulness, relevance, llmMetric } from '@cogitator-ai/evals';
@@ -258,7 +264,43 @@ Dataset.from([{ input: 'Write the lede', context: { dossier: dossierText } }]);
 
 When the run retrieved passages (see [Retrieval](#retrieval)), `faithfulness()` checks the response against them instead of `context`. `contextRelevance()` judges the input against the retrieved passages alone, or against `context` when nothing was retrieved, and scores 0 without a judge call when the case has neither.
 
-The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
+The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency?, samples? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
+
+`llmMetric` also takes a discrete `rubric` instead of a free prompt: the judge picks one level, and the score is normalized to 0 - 1 by the lowest and highest level. A rubric needs at least two levels with distinct scores, and a verdict that is not a level counts as a judge error.
+
+```typescript
+llmMetric({
+  name: 'correctness',
+  prompt: 'Grade whether the response answers the question correctly.', // optional with a rubric
+  rubric: [
+    { score: 1, description: 'wrong, or does not answer' },
+    { score: 3, description: 'partly correct' },
+    { score: 5, description: 'fully correct' },
+  ],
+});
+```
+
+`gEval({ name, criteria, steps?, scale? })` follows G-Eval: the judge scores each case on `scale` (`[1, 5]` by default, normalized to 0 - 1) following evaluation steps. Without `steps`, the judge writes them from `criteria` once per metric instance, before the first case, and every later case reuses them. A failed generation scores the cases waiting on it as judge errors, and the next case tries again. The original G-Eval weights each score by the probability of its token, which not every provider exposes, so set `judge.samples` (with a `temperature` above 0) to average several verdicts instead.
+
+```typescript
+gEval({
+  name: 'clarity',
+  criteria: 'A beginner can follow the explanation, and it names the key idea first.',
+});
+```
+
+Every judge prompt keeps the untrusted values of a case (input, response, expected answer, context, retrieved passages, the turns of a conversation) inside XML-like tags such as `<input>` and `<response>`, with `<`, `>` and `&` escaped, so an answer cannot close its tag or open another, and the judge is told that tag contents are data to evaluate, never instructions. An answer that tells the judge to give it full marks is judged as content.
+
+`judge.samples` asks the judge that many times per case and metric (1 by default) and scores the mean of the verdicts it could read. The score's `details` give the spread (`mean of 3 verdicts, spread 0.6..0.9`), `metadata.samples` each verdict, and `usage` the summed cost. With `cache`, each sample is cached under its own key.
+
+`calibrateJudge(metric, examples, options?)` checks a judge against labels people gave: it scores each `{ input, output, expected?, context?, label }` (labels from 0 to 1) and returns `{ n, errors, pearson, spearman, mae, kappa, perCase }`, `kappa` being Cohen's kappa of pass and fail at `threshold` (0.5 by default). Pass a metric bound with `bindJudgeContext`, or `judge` with a `cogitator` to bind it. Examples the metric fails on are reported in `perCase` and left out of the figures.
+
+```typescript
+const calibration = await calibrateJudge(correctness, labelled, {
+  judge: { model: 'openai/gpt-6.1-sol', cogitator: cog },
+});
+console.log(calibration.spearman, calibration.kappa);
+```
 
 The judge answers with `{ "score", "reasoning" }` JSON. A verdict in a code fence or with the score as a numeric string is read as well, and an answer without JSON falls back to its last labelled score (`Score: 0.4`, `score is 4/5`), never to a stray number in the prose. When no verdict can be read, or the judge call fails, the metric scores 0 with the reason in `score.error`. The judge's usage is reported on each score and counted in `stats.judgeCost`.
 
@@ -500,6 +542,30 @@ for (const [name, mc] of Object.entries(result.summary.metrics)) {
 }
 ```
 
+Binary metrics (every score 0 or 1) take McNemar's test, exact below 25 discordant pairs and chi-square with continuity correction from 25. Other metrics take the paired t-test, or Wilcoxon's signed-rank test with `test: 'wilcoxon'` (zero differences dropped, exact up to 20 pairs, normal approximation above). Paired differences are rounded to 12 significant digits, so float noise neither splits ties nor hides a zero. The winner follows the test's direction: the higher mean for McNemar's and the t-test, the larger rank sum for Wilcoxon's, which can disagree with `difference`, the mean difference. The p-values are corrected with Holm's method across the compared metrics: `pValue` is the test's own, `adjustedPValue` the corrected one, and `significant` means `adjustedPValue < alpha` (`alpha` 0.05 by default). Each metric also reports `difference` (the mean of challenger minus baseline), `ci` (a percentile bootstrap interval of the paired differences at `1 - alpha`, the same on every run with `seed`), `effectSize` (Cohen's d of the paired differences) and the `test` it ran.
+
+`result.summary.latency` and `result.summary.cost` compare milliseconds and USD per case, paired over the cases both sides answered, where lower is better: means, `difference`, `pValue` and `winner`. `cost` is there only when both sides reported usage. Neither takes part in the metrics' vote on the overall winner.
+
+With `pairwise`, the judge also reads both answers of each case and says which is better, twice, the second time with the answers swapped. Agreeing verdicts give the case to that side, disagreeing ones or a tie make it a tie, a side that errored loses the case without a judge call, and a case both sides errored on is left out. A judge failure counts as a tie and is reported.
+
+```typescript
+const comparison = new EvalComparison({
+  dataset,
+  targets: { baseline, challenger },
+  metrics: [exactMatch()],
+  judge: { model: 'openai/gpt-6.1-sol', cogitator: cog },
+  pairwise: { criteria: 'Which answer is more accurate.' },
+  test: 'wilcoxon',
+  alpha: 0.05,
+  seed: 42,
+});
+
+const { pairwise } = await comparison.run();
+// { name, wins: { baseline, challenger }, ties, cases, winRate, judgeErrors, cost, perCase }
+```
+
+`winRate` is the challenger's wins over the decided cases, NaN when none was decided.
+
 Access full suite results via `result.baseline` and `result.challenger`. `comparison.run({ signal })` cancels both runs.
 
 ---
@@ -652,6 +718,7 @@ result.report('ci');
 | ---------------- | ------------------------------------------------------------------- |
 | `EvalSuite`      | Main evaluation runner                                              |
 | `EvalComparison` | A/B testing runner with statistical significance                    |
+| `calibrateJudge` | Agreement of a judge metric with labelled examples                  |
 | `EvalBuilder`    | Fluent builder for EvalSuite                                        |
 | `Dataset`        | Immutable dataset with from/fromJsonl/fromCsv/filter/sample/shuffle |
 | `loadJsonl`      | Low-level JSONL file loader                                         |
@@ -667,6 +734,7 @@ result.report('ci');
 | `containsAll`        | Deterministic | Fraction of keywords found                                         |
 | `containsAny`        | Deterministic | Any configured keyword found                                       |
 | `similarity`         | Deterministic | Levenshtein distance similarity score                              |
+| `tokenF1`            | Deterministic | Token-level F1 score                                               |
 | `regex`              | Deterministic | Regex pattern match                                                |
 | `jsonSchema`         | Deterministic | Zod schema validation                                              |
 | `faithfulness`       | LLM Judge     | Factual accuracy                                                   |
@@ -674,7 +742,8 @@ result.report('ci');
 | `coherence`          | LLM Judge     | Logical structure                                                  |
 | `helpfulness`        | LLM Judge     | Practical usefulness                                               |
 | `contextRelevance`   | LLM Judge     | Relevance of the retrieved passages to the input                   |
-| `llmMetric`          | LLM Judge     | Custom judge prompt                                                |
+| `gEval`              | LLM Judge     | Evaluation steps from criteria, then a score on a scale            |
+| `llmMetric`          | LLM Judge     | Custom judge prompt or discrete rubric                             |
 | `toolCalls`          | Agent         | Tool calls against `expectedToolCalls`, F1, recall or precision    |
 | `trajectory`         | Agent         | Sequence of tool names, exact, in-order or any-order               |
 | `retrievalRecall`    | Agent         | Share of `expectedPassages` retrieved                              |
@@ -720,15 +789,22 @@ result.report('ci');
 
 ### Statistics
 
-| Export         | Description                                               |
-| -------------- | --------------------------------------------------------- |
-| `pairedTTest`  | Paired t-test for continuous metric comparison            |
-| `mcnemarsTest` | McNemar's test for binary metric comparison               |
-| `mean`         | Arithmetic mean                                           |
-| `median`       | Median value                                              |
-| `stdDev`       | Sample standard deviation                                 |
-| `percentile`   | Arbitrary percentile                                      |
-| `aggregate`    | Full stats: mean, median, min, max, stdDev, p50, p95, p99 |
+| Export               | Description                                                     |
+| -------------------- | --------------------------------------------------------------- |
+| `pairedTTest`        | Paired t-test for continuous metric comparison                  |
+| `wilcoxonSignedRank` | Wilcoxon signed-rank test, zeros dropped, exact up to 20 pairs  |
+| `mcnemarsTest`       | McNemar's test (chi-square, continuity corrected) for binary    |
+| `mcnemarExact`       | Exact McNemar's test, the binomial test of the discordant pairs |
+| `holm`               | Holm-Bonferroni adjusted p-values                               |
+| `bootstrapCI`        | Seeded percentile bootstrap confidence interval of the mean     |
+| `cohensD`            | Cohen's d of paired differences (dz)                            |
+| `pearson`            | Pearson's correlation                                           |
+| `spearman`           | Spearman's rank correlation                                     |
+| `mean`               | Arithmetic mean                                                 |
+| `median`             | Median value                                                    |
+| `stdDev`             | Sample standard deviation                                       |
+| `percentile`         | Arbitrary percentile                                            |
+| `aggregate`          | Full stats: mean, median, min, max, stdDev, p50, p95, p99       |
 
 ### Agent Tools
 

package/dist/calibrate-judge.d.ts

@@ -0,0 +1,53 @@
+import type { MetricFn } from './metrics/types.js';
+import type { JudgeConfigInput } from './schema.js';
+/** An answer labelled by a person, with the score the metric should give it, from 0 to 1 */
+export interface CalibrationExample {
+    input: string;
+    output: string;
+    expected?: string;
+    context?: Record<string, unknown>;
+    label: number;
+}
+export interface CalibrateJudgeOptions {
+    /** Where a score or label counts as a pass for `kappa`, 0.5 by default */
+    threshold?: number;
+    /** Binds an unbound judge metric, run by `judge.cogitator` */
+    judge?: JudgeConfigInput;
+    /** Examples scored at once, 5 by default */
+    concurrency?: number;
+}
+export interface CalibrationCase {
+    /** Position of the example */
+    index: number;
+    label: number;
+    /** The metric's score, absent when it failed */
+    score?: number;
+    details?: string;
+    error?: string;
+}
+export interface JudgeCalibration {
+    /** Examples the metric scored, those it failed on left out of the figures */
+    n: number;
+    /** Examples the metric failed on */
+    errors: number;
+    /** Pearson's correlation of scores and labels, NaN when either does not vary */
+    pearson: number;
+    /** Spearman's rank correlation of scores and labels */
+    spearman: number;
+    /** Mean absolute error of the scores against the labels */
+    mae: number;
+    /**
+     * Cohen's kappa of pass and fail, a score or label passing at `threshold` or more. NaN when
+     * agreement by chance is certain, every score and label being on one side
+     */
+    kappa: number;
+    perCase: CalibrationCase[];
+}
+/**
+ * How well a metric, usually an LLM judge, agrees with labels people gave: it scores each
+ * labelled example and compares the scores with the labels. Bind a judge metric first with
+ * `bindJudgeContext`, or pass `judge` with a `cogitator`. Examples the metric fails on are
+ * reported and left out of the figures
+ */
+export declare function calibrateJudge(metric: MetricFn, examples: readonly CalibrationExample[], options?: CalibrateJudgeOptions): Promise<JudgeCalibration>;
+//# sourceMappingURL=calibrate-judge.d.ts.map

package/dist/calibrate-judge.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"calibrate-judge.d.ts","sourceRoot":"","sources":["../src/calibrate-judge.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAkB,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAIhE,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAKjD,4FAA4F;AAC5F,MAAM,WAAW,kBAAkB;IACjC,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAClC,KAAK,EAAE,MAAM,CAAC;CACf;AAED,MAAM,WAAW,qBAAqB;IACpC,0EAA0E;IAC1E,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8DAA8D;IAC9D,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,4CAA4C;IAC5C,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,eAAe;IAC9B,8BAA8B;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,MAAM,CAAC;IACd,gDAAgD;IAChD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,6EAA6E;IAC7E,CAAC,EAAE,MAAM,CAAC;IACV,oCAAoC;IACpC,MAAM,EAAE,MAAM,CAAC;IACf,gFAAgF;IAChF,OAAO,EAAE,MAAM,CAAC;IAChB,uDAAuD;IACvD,QAAQ,EAAE,MAAM,CAAC;IACjB,2DAA2D;IAC3D,GAAG,EAAE,MAAM,CAAC;IACZ;;;OAGG;IACH,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,eAAe,EAAE,CAAC;CAC5B;AA6ED;;;;;GAKG;AACH,wBAAsB,cAAc,CAClC,MAAM,EAAE,QAAQ,EAChB,QAAQ,EAAE,SAAS,kBAAkB,EAAE,EACvC,OAAO,GAAE,qBAA0B,GAClC,OAAO,CAAC,gBAAgB,CAAC,CAyC3B"}

package/dist/calibrate-judge.js

@@ -0,0 +1,114 @@
+import { z } from 'zod';
+import { isLLMMetric } from './eval-suite.js';
+import { nonFiniteScoreError } from './metrics/types.js';
+import { bindJudgeContext, judgeContextFor } from './metrics/llm-judge.js';
+import { JudgeConfigSchema } from './schema.js';
+import { parseOptions } from './options.js';
+import { pearson, spearman } from './stats/correlation.js';
+import { mean } from './stats/percentiles.js';
+const CalibrationExampleSchema = z.object({
+    input: z.string(),
+    output: z.string(),
+    expected: z.string().optional(),
+    context: z.record(z.string(), z.unknown()).optional(),
+    label: z.number().min(0).max(1),
+});
+const CalibrateJudgeOptionsSchema = z.object({
+    threshold: z.number().min(0).max(1).default(0.5),
+    concurrency: z.number().int().positive().default(5),
+});
+/** Cohen's kappa of two binary raters */
+function binaryKappa(a, b) {
+    const n = a.length;
+    if (n === 0)
+        return Number.NaN;
+    let agree = 0;
+    let aPass = 0;
+    let bPass = 0;
+    for (let i = 0; i < n; i++) {
+        if (a[i] === b[i])
+            agree++;
+        if (a[i])
+            aPass++;
+        if (b[i])
+            bPass++;
+    }
+    const observed = agree / n;
+    const chance = (aPass / n) * (bPass / n) + (1 - aPass / n) * (1 - bPass / n);
+    return chance === 1 ? Number.NaN : (observed - chance) / (1 - chance);
+}
+function bindMetric(metric, judge) {
+    if (!isLLMMetric(metric))
+        return metric;
+    if (!judge) {
+        throw new Error('calibrateJudge needs options.judge to run an unbound judge metric');
+    }
+    const config = JudgeConfigSchema.parse(judge);
+    if (!config.cogitator) {
+        throw new Error('calibrateJudge needs judge.cogitator to run the judge');
+    }
+    return bindJudgeContext(metric, judgeContextFor(config.cogitator, config));
+}
+async function scoreExample(metric, example, index) {
+    const result = {
+        case: {
+            input: example.input,
+            ...(example.expected !== undefined && { expected: example.expected }),
+            ...(example.context !== undefined && { context: example.context }),
+        },
+        output: example.output,
+        duration: 0,
+    };
+    try {
+        const scored = await metric(result);
+        const invalid = scored.error ?? nonFiniteScoreError(scored.score);
+        if (invalid !== undefined)
+            return { index, label: example.label, error: invalid };
+        return {
+            index,
+            label: example.label,
+            score: scored.score,
+            ...(scored.details !== undefined && { details: scored.details }),
+        };
+    }
+    catch (error) {
+        return {
+            index,
+            label: example.label,
+            error: error instanceof Error ? error.message : String(error),
+        };
+    }
+}
+/**
+ * How well a metric, usually an LLM judge, agrees with labels people gave: it scores each
+ * labelled example and compares the scores with the labels. Bind a judge metric first with
+ * `bindJudgeContext`, or pass `judge` with a `cogitator`. Examples the metric fails on are
+ * reported and left out of the figures
+ */
+export async function calibrateJudge(metric, examples, options = {}) {
+    const { threshold, concurrency } = parseOptions(CalibrateJudgeOptionsSchema, { threshold: options.threshold, concurrency: options.concurrency }, 'calibrateJudge');
+    const parsed = parseOptions(z.array(CalibrationExampleSchema), examples, 'calibrateJudge examples');
+    const bound = bindMetric(metric, options.judge);
+    const perCase = new Array(parsed.length);
+    let next = 0;
+    const worker = async () => {
+        while (next < parsed.length) {
+            const at = next++;
+            perCase[at] = await scoreExample(bound, parsed[at], at);
+        }
+    };
+    await Promise.all(Array.from({ length: Math.min(concurrency, parsed.length) }, worker));
+    const scored = perCase.filter((one) => one.score !== undefined);
+    const scores = scored.map((one) => one.score);
+    const labels = scored.map((one) => one.label);
+    return {
+        n: scored.length,
+        errors: perCase.length - scored.length,
+        pearson: pearson(scores, labels),
+        spearman: spearman(scores, labels),
+        mae: scored.length === 0 ? Number.NaN : mean(scored.map((one) => Math.abs(one.score - one.label))),
+        kappa: binaryKappa(scores.map((score) => score >= threshold), labels.map((label) => label >= threshold)),
+        perCase,
+    };
+}
+//# sourceMappingURL=calibrate-judge.js.map

package/dist/calibrate-judge.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"calibrate-judge.js","sourceRoot":"","sources":["../src/calibrate-judge.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AACxB,OAAO,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAE3C,OAAO,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACtD,OAAO,EAAE,gBAAgB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AACxE,OAAO,EAAE,iBAAiB,EAAE,MAAM,UAAU,CAAC;AAE7C,OAAO,EAAE,YAAY,EAAE,MAAM,WAAW,CAAC;AACzC,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,MAAM,qBAAqB,CAAC;AACxD,OAAO,EAAE,IAAI,EAAE,MAAM,qBAAqB,CAAC;AAiD3C,MAAM,wBAAwB,GAAG,CAAC,CAAC,MAAM,CAAC;IACxC,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE;IACjB,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE;IAClB,QAAQ,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;IAC/B,OAAO,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC,QAAQ,EAAE;IACrD,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;CAChC,CAAC,CAAC;AAEH,MAAM,2BAA2B,GAAG,CAAC,CAAC,MAAM,CAAC;IAC3C,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC;IAChD,WAAW,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC;CACpD,CAAC,CAAC;AAEH,yCAAyC;AACzC,SAAS,WAAW,CAAC,CAAqB,EAAE,CAAqB;IAC/D,MAAM,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC;IACnB,IAAI,CAAC,KAAK,CAAC;QAAE,OAAO,MAAM,CAAC,GAAG,CAAC;IAC/B,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC3B,IAAI,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;QAC3B,IAAI,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;QAClB,IAAI,CAAC,CAAC,CAAC,CAAC;YAAE,KAAK,EAAE,CAAC;IACpB,CAAC;IACD,MAAM,QAAQ,GAAG,KAAK,GAAG,CAAC,CAAC;IAC3B,MAAM,MAAM,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,CAAC,CAAC;IAC7E,OAAO,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC;AACxE,CAAC;AAED,SAAS,UAAU,CAAC,MAAgB,EAAE,KAAmC;IACvE,IAAI,CAAC,WAAW,CAAC,MAAM,CAAC;QAAE,OAAO,MAAM,CAAC;IACxC,IAAI,CAAC,KAAK,EAAE,CAAC;QACX,MAAM,IAAI,KAAK,CAAC,mEAAmE,CAAC,CAAC;IACvF,CAAC;IACD,MAAM,MAAM,GAAG,iBAAiB,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;IAC9C,IAAI,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC;QACtB,MAAM,IAAI,KAAK,CAAC,uDAAuD,CAAC,CAAC;IAC3E,CAAC;IACD,OAAO,gBAAgB,CAAC,MAAM,EAAE,eAAe,CAAC,MAAM,CAAC,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC;AAC7E,CAAC;AAED,KAAK,UAAU,YAAY,CACzB,MAAgB,EAChB,OAAkD,EAClD,KAAa;IAEb,MAAM,MAAM,GAAmB;QAC7B,IAAI,EAAE;YACJ,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,GAAG,CAAC,OAAO,CAAC,QAAQ,KAAK,SAAS,IAAI,EAAE,QAAQ,EAAE,OAAO,CAAC,QAAQ,EAAE,CAAC;YACrE,GAAG,CAAC,OAAO,CAAC,OAAO,KAAK,SAAS,IAAI,EAAE,OAAO,EAAE,OAAO,CAAC,OAAO,EAAE,CAAC;SACnE;QACD,MAAM,EAAE,OAAO,CAAC,MAAM;QACtB,QAAQ,EAAE,CAAC;KACZ,CAAC;IACF,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,MAAM,CAAC,CAAC;QACpC,MAAM,OAAO,GAAG,MAAM,CAAC,KAAK,IAAI,mBAAmB,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QAClE,IAAI,OAAO,KAAK,SAAS;YAAE,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,KAAK,EAAE,OAAO,EAAE,CAAC;QAClF,OAAO;YACL,KAAK;YACL,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,KAAK,EAAE,MAAM,CAAC,KAAK;YACnB,GAAG,CAAC,MAAM,CAAC,OAAO,KAAK,SAAS,IAAI,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAAC;SACjE,CAAC;IACJ,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,OAAO;YACL,KAAK;YACL,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,KAAK,EAAE,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC;SAC9D,CAAC;IACJ,CAAC;AACH,CAAC;AAED;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,cAAc,CAClC,MAAgB,EAChB,QAAuC,EACvC,UAAiC,EAAE;IAEnC,MAAM,EAAE,SAAS,EAAE,WAAW,EAAE,GAAG,YAAY,CAC7C,2BAA2B,EAC3B,EAAE,SAAS,EAAE,OAAO,CAAC,SAAS,EAAE,WAAW,EAAE,OAAO,CAAC,WAAW,EAAE,EAClE,gBAAgB,CACjB,CAAC;IACF,MAAM,MAAM,GAAG,YAAY,CACzB,CAAC,CAAC,KAAK,CAAC,wBAAwB,CAAC,EACjC,QAAQ,EACR,yBAAyB,CAC1B,CAAC;IACF,MAAM,KAAK,GAAG,UAAU,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC;IAEhD,MAAM,OAAO,GAAG,IAAI,KAAK,CAAkB,MAAM,CAAC,MAAM,CAAC,CAAC;IAC1D,IAAI,IAAI,GAAG,CAAC,CAAC;IACb,MAAM,MAAM,GAAG,KAAK,IAAI,EAAE;QACxB,OAAO,IAAI,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC;YAC5B,MAAM,EAAE,GAAG,IAAI,EAAE,CAAC;YAClB,OAAO,CAAC,EAAE,CAAC,GAAG,MAAM,YAAY,CAAC,KAAK,EAAE,MAAM,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;QAC1D,CAAC;IACH,CAAC,CAAC;IACF,MAAM,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,GAAG,CAAC,WAAW,EAAE,MAAM,CAAC,MAAM,CAAC,EAAE,EAAE,MAAM,CAAC,CAAC,CAAC;IAExF,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,CAC3B,CAAC,GAAG,EAA8C,EAAE,CAAC,GAAG,CAAC,KAAK,KAAK,SAAS,CAC7E,CAAC;IACF,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IAC9C,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;IAC9C,OAAO;QACL,CAAC,EAAE,MAAM,CAAC,MAAM;QAChB,MAAM,EAAE,OAAO,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM;QACtC,OAAO,EAAE,OAAO,CAAC,MAAM,EAAE,MAAM,CAAC;QAChC,QAAQ,EAAE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;QAClC,GAAG,EACD,MAAM,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC;QAC/F,KAAK,EAAE,WAAW,CAChB,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,SAAS,CAAC,EACzC,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,IAAI,SAAS,CAAC,CAC1C;QACD,OAAO;KACR,CAAC;AACJ,CAAC"}

package/dist/datasets/dataset.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"dataset.d.ts","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAI1C,qBAAa,OAAO;IAClB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAsB;IAE7C,OAAO;IAIP,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO;WAI1B,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;WAKzC,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAKpD,IAAI,MAAM,IAAI,MAAM,CAEnB;IAED,IAAI,KAAK,IAAI,SAAS,QAAQ,EAAE,CAE/B;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,QAAQ,CAAC,QAAQ,CAAC;IAavC,MAAM,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,QAAQ,KAAK,OAAO,GAAG,OAAO;IAI7C;;;OAGG;IACH,MAAM,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;IAYpE,yEAAyE;IACzE,OAAO,CAAC,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;CAG3D"}
+{"version":3,"file":"dataset.d.ts","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAK1C,qBAAa,OAAO;IAClB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAsB;IAE7C,OAAO;IAIP,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO;WAI1B,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;WAKzC,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAKpD,IAAI,MAAM,IAAI,MAAM,CAEnB;IAED,IAAI,KAAK,IAAI,SAAS,QAAQ,EAAE,CAE/B;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,QAAQ,CAAC,QAAQ,CAAC;IAavC,MAAM,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,QAAQ,KAAK,OAAO,GAAG,OAAO;IAI7C;;;OAGG;IACH,MAAM,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;IAYpE,yEAAyE;IACzE,OAAO,CAAC,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;CAG3D"}

package/dist/datasets/dataset.js

@@ -1,6 +1,7 @@
 import { EvalCaseSchema } from '../schema.js';
 import { loadJsonl } from './jsonl-loader.js';
 import { loadCsv } from './csv-loader.js';
+import { seededRandom } from '../stats/random.js';
 export class Dataset {
     _cases;
     constructor(cases) {
@@ -47,12 +48,12 @@ export class Dataset {
             return new Dataset([]);
         }
         const count = Math.min(n, this._cases.length);
-        const indices = shuffled(Array.from({ length: this._cases.length }, (_, i) => i), randomOf(options.seed));
+        const indices = shuffled(Array.from({ length: this._cases.length }, (_, i) => i), seededRandom(options.seed));
         return new Dataset(indices.slice(0, count).map((i) => this._cases[i]));
     }
     /** The cases in random order, the same order on every run with `seed` */
     shuffle(options = {}) {
-        return new Dataset(shuffled([...this._cases], randomOf(options.seed)));
+        return new Dataset(shuffled([...this._cases], seededRandom(options.seed)));
     }
 }
 /** Shuffles `items` in place (Fisher-Yates) with the random numbers `random` gives */
@@ -63,29 +64,4 @@ function shuffled(items, random) {
     }
     return items;
 }
-/**
- * `Math.random`, or a mulberry32 generator seeded by the hash of `seed`, so every number and
- * string seeds its own sequence (`1.2` and `1.7`, `-1` and `2 ** 32 - 1` alike)
- */
-function randomOf(seed) {
-    if (seed === undefined)
-        return Math.random;
-    let state = hashSeed(String(seed));
-    return () => {
-        state = (state + 0x6d2b79f5) >>> 0;
-        let t = state;
-        t = Math.imul(t ^ (t >>> 15), t | 1);
-        t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
-        return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
-    };
-}
-/** FNV-1a of a string, to seed the generator */
-function hashSeed(seed) {
-    let hash = 0x811c9dc5;
-    for (let i = 0; i < seed.length; i++) {
-        hash ^= seed.charCodeAt(i);
-        hash = Math.imul(hash, 0x01000193) >>> 0;
-    }
-    return hash;
-}
 //# sourceMappingURL=dataset.js.map

package/dist/datasets/dataset.js.map

@@ -1 +1 @@
-{"version":3,"file":"dataset.js","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAE3C,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAC3C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,MAAM,OAAO,OAAO;IACD,MAAM,CAAsB;IAE7C,YAAoB,KAAiB;QACnC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,CAAC,IAAI,CAAC,KAAiB;QAC3B,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,IAAY;QACjC,MAAM,KAAK,GAAG,MAAM,SAAS,CAAC,IAAI,CAAC,CAAC;QACpC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,IAAY;QAC/B,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC;QAClC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,IAAI,MAAM;QACR,OAAO,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;IAC5B,CAAC;IAED,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC;QACf,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,OAAO;YACL,IAAI;gBACF,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC;oBACzB,OAAO,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;gBAChD,CAAC;gBACD,OAAO,EAAE,KAAK,EAAE,SAAkB,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;YACnD,CAAC;SACF,CAAC;IACJ,CAAC;IAED,MAAM,CAAC,EAA4B;QACjC,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC7C,CAAC;IAED;;;OAGG;IACH,MAAM,CAAC,CAAS,EAAE,UAAsC,EAAE;QACxD,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACvC,OAAO,IAAI,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,CAAC;QACD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC9C,MAAM,OAAO,GAAG,QAAQ,CACtB,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EACvD,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CACvB,CAAC;QACF,OAAO,IAAI,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,yEAAyE;IACzE,OAAO,CAAC,UAAsC,EAAE;QAC9C,OAAO,IAAI,OAAO,CAAC,QAAQ,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;CACF;AAED,sFAAsF;AACtF,SAAS,QAAQ,CAAI,KAAU,EAAE,MAAoB;IACnD,KAAK,IAAI,CAAC,GAAG,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1C,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QACzC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,KAAK,CAAC,CAAC,CAAE,CAAC,CAAC;IAChD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED;;;GAGG;AACH,SAAS,QAAQ,CAAC,IAAiC;IACjD,IAAI,IAAI,KAAK,SAAS;QAAE,OAAO,IAAI,CAAC,MAAM,CAAC;IAC3C,IAAI,KAAK,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACnC,OAAO,GAAG,EAAE;QACV,KAAK,GAAG,CAAC,KAAK,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC;QACnC,IAAI,CAAC,GAAG,KAAK,CAAC;QACd,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;QACrC,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC;QAC1C,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,UAAU,CAAC;IAC/C,CAAC,CAAC;AACJ,CAAC;AAED,gDAAgD;AAChD,SAAS,QAAQ,CAAC,IAAY;IAC5B,IAAI,IAAI,GAAG,UAAU,CAAC;IACtB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,IAAI,IAAI,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;QAC3B,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,UAAU,CAAC,KAAK,CAAC,CAAC;IAC3C,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}
+{"version":3,"file":"dataset.js","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAE3C,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAC3C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,OAAO,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAE/C,MAAM,OAAO,OAAO;IACD,MAAM,CAAsB;IAE7C,YAAoB,KAAiB;QACnC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,CAAC,IAAI,CAAC,KAAiB;QAC3B,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,IAAY;QACjC,MAAM,KAAK,GAAG,MAAM,SAAS,CAAC,IAAI,CAAC,CAAC;QACpC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,IAAY;QAC/B,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC;QAClC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,IAAI,MAAM;QACR,OAAO,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;IAC5B,CAAC;IAED,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC;QACf,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,OAAO;YACL,IAAI;gBACF,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC;oBACzB,OAAO,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;gBAChD,CAAC;gBACD,OAAO,EAAE,KAAK,EAAE,SAAkB,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;YACnD,CAAC;SACF,CAAC;IACJ,CAAC;IAED,MAAM,CAAC,EAA4B;QACjC,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC7C,CAAC;IAED;;;OAGG;IACH,MAAM,CAAC,CAAS,EAAE,UAAsC,EAAE;QACxD,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACvC,OAAO,IAAI,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,CAAC;QACD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC9C,MAAM,OAAO,GAAG,QAAQ,CACtB,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EACvD,YAAY,CAAC,OAAO,CAAC,IAAI,CAAC,CAC3B,CAAC;QACF,OAAO,IAAI,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,yEAAyE;IACzE,OAAO,CAAC,UAAsC,EAAE;QAC9C,OAAO,IAAI,OAAO,CAAC,QAAQ,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAC7E,CAAC;CACF;AAED,sFAAsF;AACtF,SAAS,QAAQ,CAAI,KAAU,EAAE,MAAoB;IACnD,KAAK,IAAI,CAAC,GAAG,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1C,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QACzC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,KAAK,CAAC,CAAC,CAAE,CAAC,CAAC;IAChD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/eval-comparison.d.ts

@@ -2,6 +2,7 @@ import type { EvalTarget, EvalSuiteResult } from './eval-suite.js';
 import type { MetricFn, StatisticalMetricFn } from './metrics/types.js';
 import type { JudgeConfigInput } from './schema.js';
 import { Dataset } from './datasets/index.js';
+import type { PairwiseOptions, PairwiseResult } from './pairwise.js';
 export interface EvalComparisonOptions {
     dataset: Dataset;
     targets: {
@@ -10,6 +11,7 @@ export interface EvalComparisonOptions {
     };
     metrics?: MetricFn[];
     statisticalMetrics?: StatisticalMetricFn[];
+    /** The judge of LLM metrics and of `pairwise`, on both sides */
     judge?: JudgeConfigInput;
     concurrency?: number;
     timeout?: number;
@@ -19,6 +21,23 @@ export interface EvalComparisonOptions {
      * mean over its scored trials, and `passAtK()` statistical metrics see every trial
      */
     trials?: number;
+    /**
+     * Significance level, 0.05 by default. A metric is significant when its p-value, adjusted with
+     * Holm's correction across the compared metrics, is below it, and its `ci` covers `1 - alpha`
+     */
+    alpha?: number;
+    /**
+     * Test of metrics that are not binary, and of latency and cost: `t` (paired t-test, the
+     * default) or `wilcoxon` (Wilcoxon signed-rank). Binary metrics always take McNemar's test
+     */
+    test?: 't' | 'wilcoxon';
+    /** Seeds the bootstrap of each metric's `ci`, so the intervals are the same on every run */
+    seed?: number | string;
+    /**
+     * Asks the judge which side answered each case better, both ways round, into
+     * `result.pairwise`. Needs `judge`, run by `judge.cogitator` or an agent target's `cogitator`
+     */
+    pairwise?: PairwiseOptions;
     onProgress?: (progress: {
         target: string;
         completed: number;
@@ -29,25 +48,75 @@ export interface EvalComparisonRunOptions {
     /** Cancels both runs; `run()` rejects with the signal's reason */
     signal?: AbortSignal;
 }
+/**
+ * The test a comparison ran: McNemar's (exact below 25 discordant pairs, chi-square with
+ * continuity correction from 25), the paired t-test, Wilcoxon signed-rank, or none with fewer
+ * than 2 cases
+ */
+export type ComparisonTest = 'mcnemar' | 'mcnemar-exact' | 't' | 'wilcoxon' | 'none';
+export type ComparisonWinner = 'baseline' | 'challenger' | 'tie';
 export interface MetricComparison {
     baseline: number;
     challenger: number;
     /** Cases compared: those at least one side scored, an errored side counting as a 0 */
     n: number;
+    /** Mean of the paired differences, challenger minus baseline */
+    difference: number;
+    /** Percentile bootstrap confidence interval of `difference` at `1 - alpha` */
+    ci: {
+        low: number;
+        high: number;
+    };
+    /** Cohen's d of the paired differences (dz), NaN with fewer than 2 cases */
+    effectSize: number;
+    test: ComparisonTest;
+    /** The test's own p-value */
+    pValue: number;
+    /** `pValue` after Holm's correction across the compared metrics */
+    adjustedPValue: number;
+    /** `adjustedPValue` is below `alpha` */
+    significant: boolean;
+    /**
+     * The higher side when significant, by the test's direction: the sign of `difference`, except
+     * for Wilcoxon's test, where it is the side with the larger rank sum and may differ from it
+     */
+    winner: ComparisonWinner;
+}
+/**
+ * Latency or cost of the two sides, paired over the cases both answered, where lower is better.
+ * Not corrected for multiple comparisons and not part of the metrics' vote on the winner
+ */
+export interface ResourceComparison {
+    /** Mean per case, NaN when no case pairs up */
+    baseline: number;
+    challenger: number;
+    n: number;
+    /** Challenger minus baseline: below 0 when the challenger is cheaper or faster */
+    difference: number;
+    test: ComparisonTest;
     pValue: number;
     significant: boolean;
-    winner: 'baseline' | 'challenger' | 'tie';
+    /** The lower side when significant, by the test's direction as for `MetricComparison.winner` */
+    winner: ComparisonWinner;
 }
 export interface ComparisonResult {
     summary: {
-        winner: 'baseline' | 'challenger' | 'tie';
+        winner: ComparisonWinner;
         metrics: Record<string, MetricComparison>;
+        /** Milliseconds per case, the mean over the answered trials of a case */
+        latency: ResourceComparison;
+        /** USD per case, over the cases both sides reported usage on. Absent when none did */
+        cost?: ResourceComparison;
     };
     baseline: EvalSuiteResult;
     challenger: EvalSuiteResult;
+    /** The judge's pairwise verdicts, with the `pairwise` option */
+    pairwise?: PairwiseResult;
 }
 export declare class EvalComparison {
     private readonly opts;
+    private readonly config;
+    private readonly pairwiseJudge?;
     constructor(opts: EvalComparisonOptions);
     run(options?: EvalComparisonRunOptions): Promise<ComparisonResult>;
 }

package/dist/eval-comparison.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAiC,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACpG,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAKrC,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,WAAW,gBAAgB;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;CAC3C;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;QAC1C,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;KAC3C,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;CAC7B;AAmED,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;gBAEjC,IAAI,EAAE,qBAAqB;IAUjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CA2E7E"}
+{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAiC,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACpG,OAAO,KAAK,EAAwC,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEvF,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAIrC,OAAO,KAAK,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAWlE,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,gEAAgE;IAChE,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB;;;OAGG;IACH,KAAK,CAAC,EAAE,MAAM,CAAC;IACf;;;OAGG;IACH,IAAI,CAAC,EAAE,GAAG,GAAG,UAAU,CAAC;IACxB,4FAA4F;IAC5F,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IACvB;;;OAGG;IACH,QAAQ,CAAC,EAAE,eAAe,CAAC;IAC3B,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED;;;;GAIG;AACH,MAAM,MAAM,cAAc,GAAG,SAAS,GAAG,eAAe,GAAG,GAAG,GAAG,UAAU,GAAG,MAAM,CAAC;AAErF,MAAM,MAAM,gBAAgB,GAAG,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;AAEjE,MAAM,WAAW,gBAAgB;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,gEAAgE;IAChE,UAAU,EAAE,MAAM,CAAC;IACnB,8EAA8E;IAC9E,EAAE,EAAE;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;IAClC,4EAA4E;IAC5E,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,6BAA6B;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,mEAAmE;IACnE,cAAc,EAAE,MAAM,CAAC;IACvB,wCAAwC;IACxC,WAAW,EAAE,OAAO,CAAC;IACrB;;;OAGG;IACH,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED;;;GAGG;AACH,MAAM,WAAW,kBAAkB;IACjC,+CAA+C;IAC/C,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,CAAC,EAAE,MAAM,CAAC;IACV,kFAAkF;IAClF,UAAU,EAAE,MAAM,CAAC;IACnB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,gGAAgG;IAChG,MAAM,EAAE,gBAAgB,CAAC;CAC1B;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,gBAAgB,CAAC;QACzB,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;QAC1C,yEAAyE;QACzE,OAAO,EAAE,kBAAkB,CAAC;QAC5B,sFAAsF;QACtF,IAAI,CAAC,EAAE,kBAAkB,CAAC;KAC3B,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;IAC5B,gEAAgE;IAChE,QAAQ,CAAC,EAAE,cAAc,CAAC;CAC3B;AAuID,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;IAC7C,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAuB;IAC9C,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAe;gBAElC,IAAI,EAAE,qBAAqB;IAiCjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CAwG7E"}