@cogitator-ai/evals 0.9.0 → 0.11.0

package/README.md

@@ -13,19 +13,21 @@ pnpm add @cogitator-ai/core  # agent targets and LLM-as-judge metrics
 pnpm add -D vitest  # describeEval from @cogitator-ai/evals/vitest
 ```
 
-Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
+Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
 
 ## Features
 
 - **EvalSuite** — Run datasets against agents or plain functions with configurable concurrency, timeouts, and retries
-- **7 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, regex, jsonSchema (Zod)
-- **5 LLM-as-Judge Metrics** — faithfulness, relevance, coherence, helpfulness, custom llmMetric
-- **3 Statistical Metrics** — latency, cost, tokenUsage with full percentile breakdowns
+- **8 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, tokenF1, regex, jsonSchema (Zod)
+- **6 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, custom llmMetric
+- **4 Statistical Metrics** - latency, cost, tokenUsage with full percentile breakdowns, and passAtK over trials
 - **Custom Metrics** — `metric()` factory for anything domain-specific
+- **Agent Evaluation** - `toolCalls()` and `trajectory()` against expected tool calls, `retrievalRecall()`, `retrievalPrecision()`, `retrievalMRR()` and the `contextRelevance()` judge against expected passages, multi-turn cases on one conversation
+- **Trials and pass@k** - run each case several times and score how often it passes with `passAtK()`
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
 - **A/B Testing** — EvalComparison with paired t-test and McNemar's test for statistical significance
 - **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
-- **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions
+- **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
 - **Baseline Workflow** — Save baselines, compare against them, catch regressions in CI
 - **Zod Validation** — Type-safe configuration with runtime checks
@@ -64,7 +66,7 @@ result.report('console');
 result.saveBaseline('./baseline.json');
 ```
 
-A target is either `{ fn: (input, { signal, case, attempt }) => Promise<string> }` or `{ agent, cogitator }`, which runs the agent with `cogitator.run(agent, { input, context, signal })` and records its usage for the statistical metrics. Defaults: `concurrency` 5, `timeout` 30 000 ms (minimum 1 000), `retries` 0.
+A target is either `{ fn: (input, { signal, case, attempt, turn, history }) => Promise<string> }` or `{ agent, cogitator, retrieval? }`, which runs the agent with `cogitator.run(agent, { input, context, signal, onToolResult })` and records its usage, tool calls, tool results and retrieved passages (see [Agent Evaluation](#agent-evaluation)). Defaults: `concurrency` 5, `timeout` 30 000 ms (minimum 1 000), `retries` 0, `trials` 1.
 
 A case that throws or exceeds `timeout` is retried up to `retries` times. A timed-out attempt is cancelled through its `signal`: an agent target's run is aborted, and a `fn` target should pass the signal on to its model call. The suite waits for the cancelled attempt to stop (at most another `timeout`) before it retries, so attempts of one case never overlap and `concurrency` holds. If every attempt fails, its result has an empty `output` and an `error` message describing the last failure.
 
@@ -90,21 +92,21 @@ A failure never drops a case. A metric that throws or returns a score that is no
 
 `result.stats` holds:
 
-| Field          | Meaning                                                      |
-| -------------- | ------------------------------------------------------------ |
-| `total`        | Cases run                                                    |
-| `errors`       | Cases whose every attempt failed or timed out                |
-| `metricErrors` | Metric scores that failed and count as 0                     |
-| `duration`     | Wall time of the run in ms                                   |
-| `cost`         | USD spent, `targetCost + judgeCost`                          |
-| `targetCost`   | USD the target spent, over every attempt that reported usage |
-| `judgeCost`    | USD the LLM judge spent                                      |
+| Field          | Meaning                                                                                                                |
+| -------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `total`        | Cases run                                                                                                              |
+| `errors`       | Cases whose every attempt failed or timed out (with several trials, a case counts as errored when every trial errored) |
+| `metricErrors` | Metric scores that failed and count as 0, over every trial                                                             |
+| `duration`     | Wall time of the run in ms                                                                                             |
+| `cost`         | USD spent, `targetCost + judgeCost`                                                                                    |
+| `targetCost`   | USD the target spent, over every attempt and trial that reported usage                                                 |
+| `judgeCost`    | USD the LLM judge spent                                                                                                |
 
 ---
 
 ## Datasets
 
-Datasets are immutable collections of eval cases. Each case has an `input`, optional `expected`, optional `context`, and optional `metadata`.
+Datasets are immutable collections of eval cases. Each case has an `input`, optional `expected`, optional `context`, and optional `metadata`. Agent cases may also have `turns` (further user turns run on the same conversation), `expectedToolCalls` (`{ name, arguments? }[]`) and `expectedPassages` (ids or texts the retrieval should surface).
 
 ### From inline data
 
@@ -138,7 +140,13 @@ Requires `papaparse` as an optional dependency.
 const dataset = await Dataset.fromCsv('./evals/qa.csv');
 ```
 
-CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `context.*`.
+CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `context.*`, and `turns`, `expectedToolCalls` and `expectedPassages`, whose cells hold JSON arrays (CSV-quoted, an empty cell leaves the field out):
+
+```csv
+input,turns,expectedToolCalls,expectedPassages
+Book a table,"[""For two, please""]","[{""name"":""book_table"",""arguments"":{""guests"":2}}]",
+How long do refunds take?,,"[{""name"":""rag_search""}]","[""refund is issued within 5 days""]"
+```
 
 Both loaders name the line of the file a case comes from when the case schema refuses it (`Invalid case at line 3: input: ...`), the CSV loader counting quoted fields that span lines. The CSV loader splits on commas and throws on a malformed row, a quote never closed for one, with its line.
 
@@ -168,6 +176,7 @@ Metrics that compare output against expected values using string operations or s
 | `containsAll` | Fraction of configured keywords found          | No                  |
 | `containsAny` | Whether any configured keyword is found        | No                  |
 | `similarity`  | Levenshtein distance similarity score (0 to 1) | Yes                 |
+| `tokenF1`     | Token-level F1 score (precision and recall)    | Yes                 |
 | `regex`       | Output matches a regex pattern                 | No                  |
 | `jsonSchema`  | Output is valid JSON matching a Zod schema     | No                  |
 
@@ -178,6 +187,7 @@ import {
   containsAll,
   containsAny,
   similarity,
+  tokenF1,
   regex,
   jsonSchema,
   threshold,
@@ -190,6 +200,7 @@ const metrics = [
   containsAll(['security', 'privacy', 'compliance']),
   containsAny(['approved', 'accepted']),
   similarity(),
+  tokenF1(),
   regex(/\d{4}-\d{2}-\d{2}/),
   jsonSchema(z.object({ answer: z.string(), confidence: z.number() })),
 ];
@@ -209,19 +220,22 @@ const metrics = [
 
 `containsAll()` scores the fraction of its keywords found and reports any missing keywords in `details`. `containsAny()` scores 1 when at least one keyword is found and 0 otherwise. Keywords use substring matching, so `cat` matches `concatenate`. Both metrics are case-insensitive by default, accept `caseSensitive` and `name` options, ignore duplicate keywords, and reject empty lists or empty and whitespace-only keywords when the metric is created.
 
+`tokenF1()` computes token overlap using multiset counts so duplicate tokens are handled correctly, returning the harmonic mean of precision and recall. It lowercases unless `caseSensitive: true`, strips punctuation, removes standalone articles (`a`, `an`, `the`), splits on whitespace, scores 1 for both empty strings, and reports precision and recall in `details` whenever the score is below 1.
+
 `regex()` ignores the `g` and `y` flags, which would make identical outputs alternate between pass and fail.
 
 ### LLM-as-Judge
 
 Metrics scored by an LLM judge (0.0 to 1.0). They require a `judge` config on the suite and a Cogitator to run the judge agent: `judge.cogitator`, or the `cogitator` of an agent target. Without one, the suite throws when it is created.
 
-| Metric         | Evaluates                               |
-| -------------- | --------------------------------------- |
-| `faithfulness` | Factual accuracy relative to input      |
-| `relevance`    | How on-topic the response is            |
-| `coherence`    | Logical structure and readability       |
-| `helpfulness`  | Practical usefulness to the user        |
-| `llmMetric`    | Custom prompt — you define the criteria |
+| Metric             | Evaluates                                    |
+| ------------------ | -------------------------------------------- |
+| `faithfulness`     | Factual accuracy relative to input           |
+| `relevance`        | How on-topic the response is                 |
+| `coherence`        | Logical structure and readability            |
+| `helpfulness`      | Practical usefulness to the user             |
+| `contextRelevance` | Whether the retrieved passages fit the input |
+| `llmMetric`        | Custom prompt, you define the criteria       |
 
 ```typescript
 import { faithfulness, relevance, llmMetric } from '@cogitator-ai/evals';
@@ -247,6 +261,8 @@ The judge reads the case's `context` too, so `faithfulness()` can check a respon
 Dataset.from([{ input: 'Write the lede', context: { dossier: dossierText } }]);
 ```
 
+When the run retrieved passages (see [Retrieval](#retrieval)), `faithfulness()` checks the response against them instead of `context`. `contextRelevance()` judges the input against the retrieved passages alone, or against `context` when nothing was retrieved, and scores 0 without a judge call when the case has neither.
+
 The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
 
 The judge answers with `{ "score", "reasoning" }` JSON. A verdict in a code fence or with the score as a numeric string is read as well, and an answer without JSON falls back to its last labelled score (`Score: 0.4`, `score is 4/5`), never to a stray number in the prose. When no verdict can be read, or the judge call fails, the metric scores 0 with the reason in `score.error`. The judge's usage is reported on each score and counted in `stats.judgeCost`.
@@ -268,7 +284,7 @@ const suite = new EvalSuite({
 
 ### Custom
 
-Build domain-specific metrics with the `metric()` factory.
+Build domain-specific metrics with the `metric()` factory. `evaluate` gets `input`, `output`, `expected` and `context`, the run's `toolCalls`, `toolResults`, `retrieved` and `turns`, and the whole case result as `result`.
 
 ```typescript
 import { metric } from '@cogitator-ai/evals';
@@ -292,6 +308,108 @@ Scores are automatically clamped to [0, 1]. A score that is not a finite number
 
 ---
 
+## Agent Evaluation
+
+An agent target records what the agent did, not only what it answered. Each case result carries `toolCalls` (every call of the run), `toolResults` (`{ callId, name, result?, error? }`, the raw values the run's `onToolResult` reported) and `retrieved` (`{ id?, content, score?, source? }` passages the tool results surfaced), summed over every turn of the case.
+
+### Tool calls and trajectory
+
+```typescript
+import { EvalSuite, Dataset, toolCalls, trajectory } from '@cogitator-ai/evals';
+
+const dataset = Dataset.from([
+  {
+    input: 'What is the weather in Paris?',
+    expectedToolCalls: [{ name: 'get_weather', arguments: { city: 'Paris' } }],
+  },
+]);
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [toolCalls(), trajectory({ order: 'in-order' })],
+});
+```
+
+`toolCalls({ arguments?, score? })` pairs the actual calls with the expected ones as multisets, each expected call taking at most one actual call. `arguments: 'subset'` (default) wants the expected arguments in the call with equal values, other arguments allowed, `'exact'` wants equal arguments and `'ignore'` compares names only. An expected call without `arguments` matches on its name. `score` is `'f1'` (default), `'recall'` or `'precision'`, and `details` lists the missing and unexpected calls.
+
+`trajectory({ order? })` compares the sequence of tool names: `'exact'` (default) scores 1 for the same sequence and 0 otherwise, `'in-order'` scores the longest common subsequence over the longer sequence, and `'any-order'` the names both share over the longer sequence.
+
+Both score 0 with `details: 'no expected tool calls provided'` on a case without `expectedToolCalls`, as `exactMatch()` does without `expected`. A case expecting no calls (`expectedToolCalls: []`) scores 1 when the agent made none.
+
+### Retrieval
+
+```typescript
+import {
+  retrievalRecall,
+  retrievalPrecision,
+  retrievalMRR,
+  contextRelevance,
+} from '@cogitator-ai/evals';
+import { createSearchTool } from '@cogitator-ai/rag';
+
+const agent = new Agent({ name: 'kb', model, tools: [tool(createSearchTool(pipeline))] });
+
+const suite = new EvalSuite({
+  dataset: Dataset.from([
+    { input: 'How long do refunds take?', expectedPassages: ['refund is issued within 5 days'] },
+  ]),
+  target: { agent, cogitator },
+  metrics: [retrievalRecall(), retrievalPrecision(), retrievalMRR(), contextRelevance()],
+  judge: { model: 'google/gemini-3.5-flash-lite' },
+});
+```
+
+The target's `retrieval` reads the passages out of each successful tool result. The default, `ragSearchPassages`, reads the `rag_search` tool of `@cogitator-ai/rag` (`{ results: [{ content, chunkId, score, source, ... }] }`, the `id` of a result or else its `chunkId` as the passage id) and any other result of that shape. Pass your own for other tools:
+
+```typescript
+target: {
+  agent,
+  cogitator,
+  retrieval: ({ name, result }) =>
+    name === 'search_docs' ? (result as Doc[]).map((d) => ({ id: d.slug, content: d.body })) : undefined,
+},
+```
+
+A retrieved passage matches an expected one by an equal `id`, by an equal text once case and whitespace are normalized, or when one normalized text contains the other and the contained one has at least three words, so a short id or a single word never matches every passage that happens to hold it. `retrievalRecall()` scores the share of expected passages retrieved, `retrievalPrecision()` the share of retrieved passages expected, and `retrievalMRR()` the reciprocal rank of the first match (passages rank in the order they came). They score 0 on a case without `expectedPassages`, and a case expecting none scores 1 when nothing was retrieved.
+
+### Multi-turn cases
+
+```typescript
+const cogitator = new Cogitator({ llm, memory: { adapter: 'memory' } });
+
+Dataset.from([
+  {
+    input: 'My name is Ada.',
+    turns: ['What is my name?'],
+    expected: 'Ada',
+  },
+]);
+```
+
+An agent case with `turns` runs `input`, then each turn, on one thread of its own (`threadId` `eval-<uuid>`, a fresh one for every attempt and trial). The Cogitator's memory carries the conversation from turn to turn, so it needs a memory adapter. The case `output` is the last answer, `toolCalls`, `toolResults`, `retrieved` and `usage` cover every turn, and `turns` lists each `{ input, output, toolCalls? }`. A turn that fails fails the case with its error, `turns` then ending with the failed one, and a cancelled or timed-out case starts no further turn. LLM judges read a multi-turn case as the conversation, each turn's input and answer, with the last user message marked as the one the response answers. A `fn` target is called once per turn, with `context.turn` (0 for `input`) and `context.history` (the earlier `{ input, output }` turns).
+
+### Trials and pass@k
+
+```typescript
+import { passAtK, exactMatch, threshold } from '@cogitator-ai/evals';
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [exactMatch()],
+  statisticalMetrics: [passAtK('exactMatch', { k: 1 }), passAtK('exactMatch', { k: 5 })],
+  assertions: [threshold('passAtK(exactMatch,5)', 0.9)],
+  trials: 5,
+});
+```
+
+`trials` (a positive integer, 1 by default) runs each case that many times. Trials share the suite's concurrency, timeout, retries, abort signal and `maxCases`, and each is scored on its own. The case result is the first trial that answered (the first one when every trial errored), so it carries an error only when all trials failed, with every trial in `trials` (each with its `scores`), and the aggregate of a metric takes the mean of a case's scored trials as that case's score. `stats.errors` counts a case as errored when every trial errored, while costs and `metricErrors` cover every trial. `latency()`, `cost()` and `tokenUsage()` see every trial as one value each, `passAtK()` sees the cases with their trials. `onProgress` fires once per case, after its last trial. `EvalComparison` takes `trials` too, comparing each case's mean over its trials.
+
+`passAtK(metric, { k, threshold = 1 })`, named `passAtK(<metric>,<k>)`, estimates per case the chance that at least one of `k` trials scores `threshold` or more, with the unbiased estimator `1 - C(n - c, k) / C(n, k)` (`n` trials, `c` passing), and averages it over the cases with at least one scored trial. An errored trial counts as a failure. The suite throws when it is created if `k` exceeds `trials` or the metric is not among its `metrics`.
+
+---
+
 ## Assertions
 
 Assertions check aggregated metrics after a suite run and produce pass/fail results.
@@ -374,6 +492,7 @@ const comparison = new EvalComparison({
   },
   metrics: [exactMatch(), contains()],
   concurrency: 5,
+  trials: 1,
 });
 
 const result = await comparison.run();
@@ -417,7 +536,7 @@ The `path` is a base path: each file reporter adds its own extension (`.json`, `
 - **junit** writes one `<testsuite name="eval">` with a `<testcase>` per case (a `<failure>` when it errored, its scores in `<system-out>`) and a `<testcase>` per assertion (a `<failure>` when it did not pass). Everything is XML-escaped, and characters XML 1.0 does not allow are replaced.
 - **github** prints `::error` for each failed assertion and errored case and `::warning` for each failed metric score. When `GITHUB_STEP_SUMMARY` is set it also appends a Markdown summary with the totals, the aggregated metrics (mean, p50, p95, cases scored) and the assertions.
 
-Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output.
+Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output. The json report keeps each case's `toolCalls`, `toolResults`, `retrieved`, `turns` and `trials`, and the console and github reports show the trials per case.
 
 ### Vitest
 
@@ -431,6 +550,27 @@ const qa = describeEval('qa agent', { suite, run: { maxCases: 50 }, timeout: 5 *
 afterAll(() => qa.result().report('junit', { path: './reports/qa' }));
 ```
 
+`evalMatchers` adds assertions on results to Vitest's `expect`, typed through a module augmentation of `Assertion` and `AsymmetricMatchersContaining` (Vitest 5):
+
+```typescript
+import { expect, it } from 'vitest';
+import { evalMatchers } from '@cogitator-ai/evals/matchers';
+
+expect.extend(evalMatchers);
+
+it('books a table', async () => {
+  const result = await suite.run();
+  const [booking] = result.results;
+  expect(booking).toHaveCalledTool('book_table', { guests: 2 });
+  expect(booking).toFollowTrajectory(['find_restaurant', 'book_table'], { order: 'in-order' });
+  expect(booking).toHaveRetrieved('doc:menu');
+  expect(booking).toScoreAtLeast('toolCalls', 1);
+  expect(result).toScoreAtLeast('trajectory', 0.8);
+});
+```
+
+`toHaveCalledTool(name, args?)` checks a case result's calls, `args` as a subset of a call's arguments. `toFollowTrajectory(names, { order? })` wants exactly those calls in that order by default, the calls in that order with others between them for `'in-order'`, and each of them in any order for `'any-order'`. `toHaveRetrieved(idOrText)` matches passages as the retrieval metrics do. `toScoreAtLeast(metric, min)` reads a case result's score or a suite result's aggregated mean, and throws when the metric is missing.
+
 `describeEval(name, { suite, run?, timeout? })` registers a `describe` block. The suite runs once in `beforeAll` (`timeout` defaults to 10 minutes, after which the run is cancelled), each case of `suite.cases(run?.maxCases)` gets a test that fails with the case error, and an `assertions` test lists every failed assertion. When the run itself fails, every test fails with that cause. `result()` returns the `EvalSuiteResult` once the run has finished.
 
 ---
@@ -461,6 +601,7 @@ const suite = new EvalBuilder()
   .withConcurrency(10)
   .withTimeout(60_000)
   .withRetries(2)
+  .withTrials(3)
   .onProgress(({ completed, total }) => {
     console.log(`${completed}/${total}`);
   })
@@ -524,24 +665,33 @@ result.report('ci');
 
 ### Metrics
 
-| Export         | Type          | Description                           |
-| -------------- | ------------- | ------------------------------------- |
-| `exactMatch`   | Deterministic | Exact string match                    |
-| `contains`     | Deterministic | Substring match                       |
-| `containsAll`  | Deterministic | Fraction of keywords found            |
-| `containsAny`  | Deterministic | Any configured keyword found          |
-| `similarity`   | Deterministic | Levenshtein distance similarity score |
-| `regex`        | Deterministic | Regex pattern match                   |
-| `jsonSchema`   | Deterministic | Zod schema validation                 |
-| `faithfulness` | LLM Judge     | Factual accuracy                      |
-| `relevance`    | LLM Judge     | Topical relevance                     |
-| `coherence`    | LLM Judge     | Logical structure                     |
-| `helpfulness`  | LLM Judge     | Practical usefulness                  |
-| `llmMetric`    | LLM Judge     | Custom judge prompt                   |
-| `latency`      | Statistical   | Response time percentiles             |
-| `cost`         | Statistical   | Token cost aggregation                |
-| `tokenUsage`   | Statistical   | Input/output token counts             |
-| `metric`       | Custom        | Factory for domain-specific metrics   |
+| Export               | Type          | Description                                                        |
+| -------------------- | ------------- | ------------------------------------------------------------------ |
+| `exactMatch`         | Deterministic | Exact string match                                                 |
+| `contains`           | Deterministic | Substring match                                                    |
+| `containsAll`        | Deterministic | Fraction of keywords found                                         |
+| `containsAny`        | Deterministic | Any configured keyword found                                       |
+| `similarity`         | Deterministic | Levenshtein distance similarity score                              |
+| `tokenF1`            | Deterministic | Token-level F1 score                                               |
+| `regex`              | Deterministic | Regex pattern match                                                |
+| `jsonSchema`         | Deterministic | Zod schema validation                                              |
+| `faithfulness`       | LLM Judge     | Factual accuracy                                                   |
+| `relevance`          | LLM Judge     | Topical relevance                                                  |
+| `coherence`          | LLM Judge     | Logical structure                                                  |
+| `helpfulness`        | LLM Judge     | Practical usefulness                                               |
+| `contextRelevance`   | LLM Judge     | Relevance of the retrieved passages to the input                   |
+| `llmMetric`          | LLM Judge     | Custom judge prompt                                                |
+| `toolCalls`          | Agent         | Tool calls against `expectedToolCalls`, F1, recall or precision    |
+| `trajectory`         | Agent         | Sequence of tool names, exact, in-order or any-order               |
+| `retrievalRecall`    | Agent         | Share of `expectedPassages` retrieved                              |
+| `retrievalPrecision` | Agent         | Share of retrieved passages expected                               |
+| `retrievalMRR`       | Agent         | Reciprocal rank of the first expected passage                      |
+| `latency`            | Statistical   | Response time percentiles                                          |
+| `cost`               | Statistical   | Token cost aggregation                                             |
+| `tokenUsage`         | Statistical   | Input/output token counts                                          |
+| `passAtK`            | Statistical   | pass@k of a metric over the trials                                 |
+| `metric`             | Custom        | Factory for domain-specific metrics                                |
+| `ragSearchPassages`  | Helper        | The default `retrieval` of agent targets, for `rag_search` results |
 
 ### Assertions
 
@@ -554,10 +704,11 @@ result.report('ci');
 
 ### Reporters
 
-| Export                           | Description                            |
-| -------------------------------- | -------------------------------------- |
-| `report`                         | Dispatch to one or more reporter types |
-| `describeEval` (`/vitest` entry) | Run a suite as a Vitest describe block |
+| Export                             | Description                              |
+| ---------------------------------- | ---------------------------------------- |
+| `report`                           | Dispatch to one or more reporter types   |
+| `describeEval` (`/vitest` entry)   | Run a suite as a Vitest describe block   |
+| `evalMatchers` (`/matchers` entry) | Matchers for `expect.extend()`, Vitest 5 |
 
 `report` takes `console`, `json`, `csv`, `junit`, `github` and `ci`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
 
@@ -569,6 +720,9 @@ result.report('ci');
 | `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`  |
 | `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers              |
 | `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on      |
+| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`              |
+| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`               |
+| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `error?`   |
 
 ### Statistics
 

package/dist/datasets/csv-loader.d.ts

@@ -1,3 +1,8 @@
 import type { EvalCase } from '../schema.js';
+/**
+ * Cases from a CSV file with an `input` column, an optional `expected` one, `metadata.<key>` and
+ * `context.<key>` columns, and `turns`, `expectedToolCalls` and `expectedPassages` columns whose
+ * cells hold JSON arrays (an empty cell leaves the field out)
+ */
 export declare function loadCsv(path: string): Promise<EvalCase[]>;
 //# sourceMappingURL=csv-loader.d.ts.map

package/dist/datasets/csv-loader.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CA8E/D"}
+{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAK1C;;;;GAIG;AACH,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CAwF/D"}

package/dist/datasets/csv-loader.js

@@ -1,6 +1,13 @@
 import { readFile } from 'node:fs/promises';
 import { EvalCaseSchema } from '../schema.js';
 import { describeIssues } from './issues.js';
+/** Columns whose cells hold JSON, such as `["follow-up question"]` for `turns` */
+const JSON_FIELDS = ['turns', 'expectedToolCalls', 'expectedPassages'];
+/**
+ * Cases from a CSV file with an `input` column, an optional `expected` one, `metadata.<key>` and
+ * `context.<key>` columns, and `turns`, `expectedToolCalls` and `expectedPassages` columns whose
+ * cells hold JSON arrays (an empty cell leaves the field out)
+ */
 export async function loadCsv(path) {
     let Papa;
     try {
@@ -51,6 +58,17 @@ export async function loadCsv(path) {
         if (row.expected !== undefined && row.expected !== '') {
             evalCase.expected = row.expected;
         }
+        for (const field of JSON_FIELDS) {
+            const cell = row[field];
+            if (cell === undefined || cell.trim() === '')
+                continue;
+            try {
+                evalCase[field] = JSON.parse(cell);
+            }
+            catch {
+                throw new Error(`Invalid case at line ${line}: ${field}: not valid JSON`);
+            }
+        }
         const metadata = {};
         const context = {};
         for (const [key, value] of Object.entries(row)) {

package/dist/datasets/csv-loader.js.map

@@ -1 +1 @@
-{"version":3,"file":"csv-loader.js","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAG1C,MAAM,CAAC,KAAK,UAAU,OAAO,CAAC,IAAY;IACxC,IAAI,IAAgC,CAAC;IACrC,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,CAAC,MAAM,MAAM,CAAC,WAAW,CAAC,CAExC,CAAC;QACF,IAAI,GAAG,MAAM,CAAC,OAAO,IAAI,MAAM,CAAC;IAClC,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,KAAK,CACb,+EAA+E,CAChF,CAAC;IACJ,CAAC;IAED,MAAM,OAAO,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACvE,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,MAA4B,CAAC;IACjC,MAAM,IAAI,GAAyD,EAAE,CAAC;IACtE,IAAI,QAAQ,GAAG,CAAC,CAAC;IACjB,IAAI,SAA6B,CAAC;IAClC,IAAI,CAAC,KAAK,CAAW,OAAO,EAAE;QAC5B,SAAS,EAAE,GAAG;QACd,cAAc,EAAE,IAAI;QACpB,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;YACvB,MAAM,IAAI,GAAG,MAAM,CAAC,YAAY,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,CAAC;YACrD,QAAQ,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC;YAC9B,MAAM,CAAC,KAAK,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;YAC9B,IAAI,KAAK,EAAE,CAAC;gBACV,SAAS,GAAG,uBAAuB,IAAI,KAAK,KAAK,CAAC,OAAO,EAAE,CAAC;gBAC5D,MAAM,CAAC,KAAK,EAAE,CAAC;gBACf,OAAO;YACT,CAAC;YACD,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC;gBACrB,OAAO;YACT,CAAC;YACD,MAAM,GAAG,GAA2B,EAAE,CAAC;YACvC,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;gBAC3B,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;gBAC9B,IAAI,KAAK,KAAK,SAAS;oBAAE,GAAG,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC;YAC9C,CAAC,CAAC,CAAC;YACH,IAAI,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC;QAC3B,CAAC;KACF,CAAC,CAAC;IAEH,IAAI,SAAS;QAAE,MAAM,IAAI,KAAK,CAAC,SAAS,CAAC,CAAC;IAC1C,IAAI,CAAC,MAAM,EAAE,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;IACrD,CAAC;IAED,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,EAAE,EAAE;QAChC,MAAM,QAAQ,GAA4B,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,GAAG,CAAC,QAAQ,KAAK,EAAE,EAAE,CAAC;YACtD,QAAQ,CAAC,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC;QACnC,CAAC;QAED,MAAM,QAAQ,GAA4B,EAAE,CAAC;QAC7C,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,OAAO,IAAI,GAAG,KAAK,UAAU;gBAAE,SAAS;YAEpD,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBAChC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAClD,CAAC;iBAAM,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;gBACtC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAChD,CAAC;QACH,CAAC;QAED,IAAI,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACnE,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,OAAO,GAAG,OAAO,CAAC;QAEhE,MAAM,OAAO,GAAG,cAAc,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QACnD,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,wBAAwB,IAAI,KAAK,cAAc,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACpF,CAAC;QACD,OAAO,OAAO,CAAC,IAAI,CAAC;IACtB,CAAC,CAAC,CAAC;AACL,CAAC;AAED,0FAA0F;AAC1F,SAAS,YAAY,CAAC,OAAe,EAAE,MAAc;IACnD,IAAI,EAAE,GAAG,MAAM,CAAC;IAChB,OAAO,EAAE,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,CAAC;QAAE,EAAE,EAAE,CAAC;IACnF,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,wFAAwF;AACxF,SAAS,UAAU,CAAC,OAAe;IACjC,MAAM,MAAM,GAAa,CAAC,CAAC,CAAC,CAAC;IAC7B,KAAK,IAAI,EAAE,GAAG,CAAC,EAAE,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,EAAE,EAAE,CAAC;QAC3C,MAAM,IAAI,GAAG,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,GAAG,CAAC,CAAC,KAAK,IAAI;YAAE,SAAS;QACxD,IAAI,IAAI,KAAK,IAAI,IAAI,IAAI,KAAK,IAAI;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;IAC1D,CAAC;IACD,OAAO,CAAC,MAAM,EAAE,EAAE;QAChB,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,IAAI,IAAI,GAAG,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC;QAC7B,OAAO,GAAG,GAAG,IAAI,EAAE,CAAC;YAClB,MAAM,GAAG,GAAG,CAAC,GAAG,GAAG,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;YAClC,IAAI,MAAM,CAAC,GAAG,CAAE,IAAI,MAAM;gBAAE,GAAG,GAAG,GAAG,CAAC;;gBACjC,IAAI,GAAG,GAAG,GAAG,CAAC,CAAC;QACtB,CAAC;QACD,OAAO,GAAG,GAAG,CAAC,CAAC;IACjB,CAAC,CAAC;AACJ,CAAC"}
+{"version":3,"file":"csv-loader.js","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAG1C,kFAAkF;AAClF,MAAM,WAAW,GAAG,CAAC,OAAO,EAAE,mBAAmB,EAAE,kBAAkB,CAAU,CAAC;AAEhF;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,OAAO,CAAC,IAAY;IACxC,IAAI,IAAgC,CAAC;IACrC,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,CAAC,MAAM,MAAM,CAAC,WAAW,CAAC,CAExC,CAAC;QACF,IAAI,GAAG,MAAM,CAAC,OAAO,IAAI,MAAM,CAAC;IAClC,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,KAAK,CACb,+EAA+E,CAChF,CAAC;IACJ,CAAC;IAED,MAAM,OAAO,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACvE,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,MAA4B,CAAC;IACjC,MAAM,IAAI,GAAyD,EAAE,CAAC;IACtE,IAAI,QAAQ,GAAG,CAAC,CAAC;IACjB,IAAI,SAA6B,CAAC;IAClC,IAAI,CAAC,KAAK,CAAW,OAAO,EAAE;QAC5B,SAAS,EAAE,GAAG;QACd,cAAc,EAAE,IAAI;QACpB,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;YACvB,MAAM,IAAI,GAAG,MAAM,CAAC,YAAY,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,CAAC;YACrD,QAAQ,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC;YAC9B,MAAM,CAAC,KAAK,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;YAC9B,IAAI,KAAK,EAAE,CAAC;gBACV,SAAS,GAAG,uBAAuB,IAAI,KAAK,KAAK,CAAC,OAAO,EAAE,CAAC;gBAC5D,MAAM,CAAC,KAAK,EAAE,CAAC;gBACf,OAAO;YACT,CAAC;YACD,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC;gBACrB,OAAO;YACT,CAAC;YACD,MAAM,GAAG,GAA2B,EAAE,CAAC;YACvC,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;gBAC3B,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;gBAC9B,IAAI,KAAK,KAAK,SAAS;oBAAE,GAAG,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC;YAC9C,CAAC,CAAC,CAAC;YACH,IAAI,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC;QAC3B,CAAC;KACF,CAAC,CAAC;IAEH,IAAI,SAAS;QAAE,MAAM,IAAI,KAAK,CAAC,SAAS,CAAC,CAAC;IAC1C,IAAI,CAAC,MAAM,EAAE,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;IACrD,CAAC;IAED,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,EAAE,EAAE;QAChC,MAAM,QAAQ,GAA4B,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,GAAG,CAAC,QAAQ,KAAK,EAAE,EAAE,CAAC;YACtD,QAAQ,CAAC,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC;QACnC,CAAC;QAED,KAAK,MAAM,KAAK,IAAI,WAAW,EAAE,CAAC;YAChC,MAAM,IAAI,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC;YACxB,IAAI,IAAI,KAAK,SAAS,IAAI,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE;gBAAE,SAAS;YACvD,IAAI,CAAC;gBACH,QAAQ,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAY,CAAC;YAChD,CAAC;YAAC,MAAM,CAAC;gBACP,MAAM,IAAI,KAAK,CAAC,wBAAwB,IAAI,KAAK,KAAK,kBAAkB,CAAC,CAAC;YAC5E,CAAC;QACH,CAAC;QAED,MAAM,QAAQ,GAA4B,EAAE,CAAC;QAC7C,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,OAAO,IAAI,GAAG,KAAK,UAAU;gBAAE,SAAS;YAEpD,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBAChC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAClD,CAAC;iBAAM,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;gBACtC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAChD,CAAC;QACH,CAAC;QAED,IAAI,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACnE,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,OAAO,GAAG,OAAO,CAAC;QAEhE,MAAM,OAAO,GAAG,cAAc,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QACnD,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,wBAAwB,IAAI,KAAK,cAAc,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACpF,CAAC;QACD,OAAO,OAAO,CAAC,IAAI,CAAC;IACtB,CAAC,CAAC,CAAC;AACL,CAAC;AAED,0FAA0F;AAC1F,SAAS,YAAY,CAAC,OAAe,EAAE,MAAc;IACnD,IAAI,EAAE,GAAG,MAAM,CAAC;IAChB,OAAO,EAAE,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,CAAC;QAAE,EAAE,EAAE,CAAC;IACnF,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,wFAAwF;AACxF,SAAS,UAAU,CAAC,OAAe;IACjC,MAAM,MAAM,GAAa,CAAC,CAAC,CAAC,CAAC;IAC7B,KAAK,IAAI,EAAE,GAAG,CAAC,EAAE,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,EAAE,EAAE,CAAC;QAC3C,MAAM,IAAI,GAAG,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,GAAG,CAAC,CAAC,KAAK,IAAI;YAAE,SAAS;QACxD,IAAI,IAAI,KAAK,IAAI,IAAI,IAAI,KAAK,IAAI;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;IAC1D,CAAC;IACD,OAAO,CAAC,MAAM,EAAE,EAAE;QAChB,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,IAAI,IAAI,GAAG,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC;QAC7B,OAAO,GAAG,GAAG,IAAI,EAAE,CAAC;YAClB,MAAM,GAAG,GAAG,CAAC,GAAG,GAAG,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;YAClC,IAAI,MAAM,CAAC,GAAG,CAAE,IAAI,MAAM;gBAAE,GAAG,GAAG,GAAG,CAAC;;gBACjC,IAAI,GAAG,GAAG,GAAG,CAAC,CAAC;QACtB,CAAC;QACD,OAAO,GAAG,GAAG,CAAC,CAAC;IACjB,CAAC,CAAC;AACJ,CAAC"}

package/dist/eval-builder.d.ts

@@ -14,6 +14,7 @@ export declare class EvalBuilder {
     private _concurrency?;
     private _timeout?;
     private _retries?;
+    private _trials?;
     private _onProgress?;
     withDataset(dataset: Dataset): this;
     withTarget(target: EvalTarget): this;
@@ -24,6 +25,8 @@ export declare class EvalBuilder {
     withConcurrency(n: number): this;
     withTimeout(ms: number): this;
     withRetries(n: number): this;
+    /** Runs each case `n` times, see `EvalSuiteOptions.trials` */
+    withTrials(n: number): this;
     onProgress(fn: EvalSuiteOptions['onProgress']): this;
     build(): EvalSuite;
 }

package/dist/eval-builder.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"eval-builder.d.ts","sourceRoot":"","sources":["../src/eval-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAe,MAAM,cAAc,CAAC;AACtD,OAAO,KAAK,EAAE,UAAU,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AACjE,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,KAAK,EAAE,QAAQ,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACrE,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AACjD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAEhD,qBAAa,WAAW;IACtB,OAAO,CAAC,QAAQ,CAAC,CAAU;IAC3B,OAAO,CAAC,OAAO,CAAC,CAAa;IAC7B,OAAO,CAAC,QAAQ,CAAkB;IAClC,OAAO,CAAC,mBAAmB,CAA6B;IACxD,OAAO,CAAC,MAAM,CAAC,CAAmB;IAClC,OAAO,CAAC,WAAW,CAAqB;IACxC,OAAO,CAAC,YAAY,CAAC,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,CAAS;IAC1B,OAAO,CAAC,QAAQ,CAAC,CAAS;IAC1B,OAAO,CAAC,WAAW,CAAC,CAAiC;IAErD,WAAW,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAKnC,UAAU,CAAC,MAAM,EAAE,UAAU,GAAG,IAAI;IAKpC,WAAW,CAAC,OAAO,EAAE,QAAQ,EAAE,GAAG,IAAI;IAKtC,sBAAsB,CAAC,OAAO,EAAE,mBAAmB,EAAE,GAAG,IAAI;IAK5D,SAAS,CAAC,MAAM,EAAE,gBAAgB,GAAG,IAAI;IAKzC,cAAc,CAAC,UAAU,EAAE,WAAW,EAAE,GAAG,IAAI;IAK/C,eAAe,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAKhC,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,IAAI;IAK7B,WAAW,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAK5B,UAAU,CAAC,EAAE,EAAE,gBAAgB,CAAC,YAAY,CAAC,GAAG,IAAI;IAKpD,KAAK,IAAI,SAAS;CA8BnB"}
+{"version":3,"file":"eval-builder.d.ts","sourceRoot":"","sources":["../src/eval-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAe,MAAM,cAAc,CAAC;AACtD,OAAO,KAAK,EAAE,UAAU,EAAE,gBAAgB,EAAE,MAAM,cAAc,CAAC;AACjE,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AACrC,OAAO,KAAK,EAAE,QAAQ,EAAE,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACrE,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AACjD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAEhD,qBAAa,WAAW;IACtB,OAAO,CAAC,QAAQ,CAAC,CAAU;IAC3B,OAAO,CAAC,OAAO,CAAC,CAAa;IAC7B,OAAO,CAAC,QAAQ,CAAkB;IAClC,OAAO,CAAC,mBAAmB,CAA6B;IACxD,OAAO,CAAC,MAAM,CAAC,CAAmB;IAClC,OAAO,CAAC,WAAW,CAAqB;IACxC,OAAO,CAAC,YAAY,CAAC,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,CAAS;IAC1B,OAAO,CAAC,QAAQ,CAAC,CAAS;IAC1B,OAAO,CAAC,OAAO,CAAC,CAAS;IACzB,OAAO,CAAC,WAAW,CAAC,CAAiC;IAErD,WAAW,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAKnC,UAAU,CAAC,MAAM,EAAE,UAAU,GAAG,IAAI;IAKpC,WAAW,CAAC,OAAO,EAAE,QAAQ,EAAE,GAAG,IAAI;IAKtC,sBAAsB,CAAC,OAAO,EAAE,mBAAmB,EAAE,GAAG,IAAI;IAK5D,SAAS,CAAC,MAAM,EAAE,gBAAgB,GAAG,IAAI;IAKzC,cAAc,CAAC,UAAU,EAAE,WAAW,EAAE,GAAG,IAAI;IAK/C,eAAe,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAKhC,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,IAAI;IAK7B,WAAW,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAK5B,8DAA8D;IAC9D,UAAU,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAK3B,UAAU,CAAC,EAAE,EAAE,gBAAgB,CAAC,YAAY,CAAC,GAAG,IAAI;IAKpD,KAAK,IAAI,SAAS;CA+BnB"}

package/dist/eval-builder.js

@@ -9,6 +9,7 @@ export class EvalBuilder {
     _concurrency;
     _timeout;
     _retries;
+    _trials;
     _onProgress;
     withDataset(dataset) {
         this._dataset = dataset;
@@ -46,6 +47,11 @@ export class EvalBuilder {
         this._retries = n;
         return this;
     }
+    /** Runs each case `n` times, see `EvalSuiteOptions.trials` */
+    withTrials(n) {
+        this._trials = n;
+        return this;
+    }
     onProgress(fn) {
         this._onProgress = fn;
         return this;
@@ -71,6 +77,7 @@ export class EvalBuilder {
             concurrency: this._concurrency,
             timeout: this._timeout,
             retries: this._retries,
+            trials: this._trials,
             onProgress: this._onProgress,
         };
         return new EvalSuite(opts);

package/dist/eval-builder.js.map

@@ -1 +1 @@
-{"version":3,"file":"eval-builder.js","sourceRoot":"","sources":["../src/eval-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAOtD,MAAM,OAAO,WAAW;IACd,QAAQ,CAAW;IACnB,OAAO,CAAc;IACrB,QAAQ,GAAe,EAAE,CAAC;IAC1B,mBAAmB,GAA0B,EAAE,CAAC;IAChD,MAAM,CAAoB;IAC1B,WAAW,GAAkB,EAAE,CAAC;IAChC,YAAY,CAAU;IACtB,QAAQ,CAAU;IAClB,QAAQ,CAAU;IAClB,WAAW,CAAkC;IAErD,WAAW,CAAC,OAAgB;QAC1B,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,UAAU,CAAC,MAAkB;QAC3B,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,OAAmB;QAC7B,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,sBAAsB,CAAC,OAA8B;QACnD,IAAI,CAAC,mBAAmB,GAAG,OAAO,CAAC;QACnC,OAAO,IAAI,CAAC;IACd,CAAC;IAED,SAAS,CAAC,MAAwB;QAChC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;QACrB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,cAAc,CAAC,UAAyB;QACtC,IAAI,CAAC,WAAW,GAAG,UAAU,CAAC;QAC9B,OAAO,IAAI,CAAC;IACd,CAAC;IAED,eAAe,CAAC,CAAS;QACvB,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,EAAU;QACpB,IAAI,CAAC,QAAQ,GAAG,EAAE,CAAC;QACnB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,CAAS;QACnB,IAAI,CAAC,QAAQ,GAAG,CAAC,CAAC;QAClB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,UAAU,CAAC,EAAkC;QAC3C,IAAI,CAAC,WAAW,GAAG,EAAE,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,KAAK;QACH,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC;YACnB,MAAM,IAAI,KAAK,CAAC,yCAAyC,CAAC,CAAC;QAC7D,CAAC;QAED,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,MAAM,IAAI,KAAK,CAAC,uCAAuC,CAAC,CAAC;QAC3D,CAAC;QAED,MAAM,aAAa,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QACtD,IAAI,aAAa,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC;YAClC,MAAM,IAAI,KAAK,CAAC,yDAAyD,CAAC,CAAC;QAC7E,CAAC;QAED,MAAM,IAAI,GAAqB;YAC7B,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,MAAM,EAAE,IAAI,CAAC,OAAO;YACpB,OAAO,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS;YAC7D,kBAAkB,EAChB,IAAI,CAAC,mBAAmB,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC,CAAC,SAAS;YAC5E,KAAK,EAAE,IAAI,CAAC,MAAM;YAClB,UAAU,EAAE,IAAI,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS;YACtE,WAAW,EAAE,IAAI,CAAC,YAAY;YAC9B,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,UAAU,EAAE,IAAI,CAAC,WAAW;SAC7B,CAAC;QAEF,OAAO,IAAI,SAAS,CAAC,IAAI,CAAC,CAAC;IAC7B,CAAC;CACF"}
+{"version":3,"file":"eval-builder.js","sourceRoot":"","sources":["../src/eval-builder.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAOtD,MAAM,OAAO,WAAW;IACd,QAAQ,CAAW;IACnB,OAAO,CAAc;IACrB,QAAQ,GAAe,EAAE,CAAC;IAC1B,mBAAmB,GAA0B,EAAE,CAAC;IAChD,MAAM,CAAoB;IAC1B,WAAW,GAAkB,EAAE,CAAC;IAChC,YAAY,CAAU;IACtB,QAAQ,CAAU;IAClB,QAAQ,CAAU;IAClB,OAAO,CAAU;IACjB,WAAW,CAAkC;IAErD,WAAW,CAAC,OAAgB;QAC1B,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,UAAU,CAAC,MAAkB;QAC3B,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,OAAmB;QAC7B,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;QACxB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,sBAAsB,CAAC,OAA8B;QACnD,IAAI,CAAC,mBAAmB,GAAG,OAAO,CAAC;QACnC,OAAO,IAAI,CAAC;IACd,CAAC;IAED,SAAS,CAAC,MAAwB;QAChC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;QACrB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,cAAc,CAAC,UAAyB;QACtC,IAAI,CAAC,WAAW,GAAG,UAAU,CAAC;QAC9B,OAAO,IAAI,CAAC;IACd,CAAC;IAED,eAAe,CAAC,CAAS;QACvB,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,EAAU;QACpB,IAAI,CAAC,QAAQ,GAAG,EAAE,CAAC;QACnB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,WAAW,CAAC,CAAS;QACnB,IAAI,CAAC,QAAQ,GAAG,CAAC,CAAC;QAClB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,8DAA8D;IAC9D,UAAU,CAAC,CAAS;QAClB,IAAI,CAAC,OAAO,GAAG,CAAC,CAAC;QACjB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,UAAU,CAAC,EAAkC;QAC3C,IAAI,CAAC,WAAW,GAAG,EAAE,CAAC;QACtB,OAAO,IAAI,CAAC;IACd,CAAC;IAED,KAAK;QACH,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC;YACnB,MAAM,IAAI,KAAK,CAAC,yCAAyC,CAAC,CAAC;QAC7D,CAAC;QAED,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,MAAM,IAAI,KAAK,CAAC,uCAAuC,CAAC,CAAC;QAC3D,CAAC;QAED,MAAM,aAAa,GAAG,IAAI,CAAC,QAAQ,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QACtD,IAAI,aAAa,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,CAAC;YAClC,MAAM,IAAI,KAAK,CAAC,yDAAyD,CAAC,CAAC;QAC7E,CAAC;QAED,MAAM,IAAI,GAAqB;YAC7B,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,MAAM,EAAE,IAAI,CAAC,OAAO;YACpB,OAAO,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,SAAS;YAC7D,kBAAkB,EAChB,IAAI,CAAC,mBAAmB,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,mBAAmB,CAAC,CAAC,CAAC,SAAS;YAC5E,KAAK,EAAE,IAAI,CAAC,MAAM;YAClB,UAAU,EAAE,IAAI,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS;YACtE,WAAW,EAAE,IAAI,CAAC,YAAY;YAC9B,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,OAAO,EAAE,IAAI,CAAC,QAAQ;YACtB,MAAM,EAAE,IAAI,CAAC,OAAO;YACpB,UAAU,EAAE,IAAI,CAAC,WAAW;SAC7B,CAAC;QAEF,OAAO,IAAI,SAAS,CAAC,IAAI,CAAC,CAAC;IAC7B,CAAC;CACF"}

package/dist/eval-comparison.d.ts

@@ -14,6 +14,11 @@ export interface EvalComparisonOptions {
     concurrency?: number;
     timeout?: number;
     retries?: number;
+    /**
+     * Runs of each case on each side, as the suite's `trials`. A metric is compared on each case's
+     * mean over its scored trials, and `passAtK()` statistical metrics see every trial
+     */
+    trials?: number;
     onProgress?: (progress: {
         target: string;
         completed: number;

package/dist/eval-comparison.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAe,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AAClF,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAKrC,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,WAAW,gBAAgB;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;CAC3C;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;QAC1C,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;KAC3C,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;CAC7B;AAmED,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;gBAEjC,IAAI,EAAE,qBAAqB;IASjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CAuE7E"}
+{"version":3,"file":"eval-comparison.d.ts","sourceRoot":"","sources":["../src/eval-comparison.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,cAAc,CAAC;AAChE,OAAO,KAAK,EAAE,QAAQ,EAAiC,mBAAmB,EAAE,MAAM,iBAAiB,CAAC;AACpG,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAKrC,MAAM,WAAW,qBAAqB;IACpC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE;QACP,QAAQ,EAAE,UAAU,CAAC;QACrB,UAAU,EAAE,UAAU,CAAC;KACxB,CAAC;IACF,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC;IACrB,kBAAkB,CAAC,EAAE,mBAAmB,EAAE,CAAC;IAC3C,KAAK,CAAC,EAAE,gBAAgB,CAAC;IACzB,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,CAAC,QAAQ,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;CACvF;AAED,MAAM,WAAW,wBAAwB;IACvC,kEAAkE;IAClE,MAAM,CAAC,EAAE,WAAW,CAAC;CACtB;AAED,MAAM,WAAW,gBAAgB;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,sFAAsF;IACtF,CAAC,EAAE,MAAM,CAAC;IACV,MAAM,EAAE,MAAM,CAAC;IACf,WAAW,EAAE,OAAO,CAAC;IACrB,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;CAC3C;AAED,MAAM,WAAW,gBAAgB;IAC/B,OAAO,EAAE;QACP,MAAM,EAAE,UAAU,GAAG,YAAY,GAAG,KAAK,CAAC;QAC1C,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,CAAC;KAC3C,CAAC;IACF,QAAQ,EAAE,eAAe,CAAC;IAC1B,UAAU,EAAE,eAAe,CAAC;CAC7B;AAmED,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAwB;gBAEjC,IAAI,EAAE,qBAAqB;IAUjC,GAAG,CAAC,OAAO,GAAE,wBAA6B,GAAG,OAAO,CAAC,gBAAgB,CAAC;CA2E7E"}

package/dist/eval-comparison.js

@@ -69,11 +69,12 @@ export class EvalComparison {
             concurrency: opts.concurrency,
             timeout: opts.timeout,
             retries: opts.retries,
+            trials: opts.trials,
         });
         this.opts = opts;
     }
     async run(options = {}) {
-        const { dataset, targets, metrics, statisticalMetrics, judge, concurrency, timeout, retries, onProgress, } = this.opts;
+        const { dataset, targets, metrics, statisticalMetrics, judge, concurrency, timeout, retries, trials, onProgress, } = this.opts;
         const sharedConfig = {
             dataset,
             metrics,
@@ -82,6 +83,7 @@ export class EvalComparison {
             concurrency,
             timeout,
             retries,
+            trials,
         };
         const baselineSuite = new EvalSuite({
             ...sharedConfig,
@@ -104,8 +106,10 @@ export class EvalComparison {
         const metricNames = new Set();
         for (const result of [baselineResult, challengerResult]) {
             for (const r of result.results) {
-                for (const s of r.scores)
-                    metricNames.add(s.name);
+                for (const trial of r.trials ?? [r]) {
+                    for (const s of trial.scores)
+                        metricNames.add(s.name);
+                }
             }
         }
         const metricComparisons = {};
@@ -123,13 +127,25 @@ export class EvalComparison {
         };
     }
 }
+/** A case's score of a metric: the mean over its trials the metric scored, none when it scored none */
+function caseScore(result, metricName) {
+    if (!result)
+        return undefined;
+    const trials = result.trials ?? [result];
+    if (trials.every((trial) => trial.error !== undefined))
+        return undefined;
+    const scores = trials.flatMap((trial) => trial.error !== undefined
+        ? [0]
+        : trial.scores.filter((s) => s.name === metricName).map((s) => s.score));
+    return scores.length === 0 ? undefined : mean(scores);
+}
 /**
  * The scores of a metric for each case, side by side. A case one side errored on counts as a 0
  * for that side, since a target that fails loses the case, and a case both sides errored on is
  * left out, telling nothing about either
  */
 function pairedScores(baseline, challenger, metricName) {
-    const scoreOf = (result, at) => result.results[at]?.scores.find((s) => s.name === metricName)?.score;
+    const scoreOf = (result, at) => caseScore(result.results[at], metricName);
     const baselineScores = [];
     const challengerScores = [];
     for (let at = 0; at < baseline.results.length; at++) {