@cogitator-ai/evals 0.8.0 → 0.10.0

package/README.md

@@ -10,20 +10,24 @@ pnpm add @cogitator-ai/evals
 # Optional dependencies
 pnpm add papaparse  # CSV dataset loading
 pnpm add @cogitator-ai/core  # agent targets and LLM-as-judge metrics
+pnpm add -D vitest  # describeEval from @cogitator-ai/evals/vitest
 ```
 
-Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
+Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Agents](https://cogitator.app/docs/evals/agents), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
 
 ## Features
 
 - **EvalSuite** — Run datasets against agents or plain functions with configurable concurrency, timeouts, and retries
 - **7 Deterministic Metrics** - exactMatch, contains, containsAll, containsAny, similarity, regex, jsonSchema (Zod)
-- **5 LLM-as-Judge Metrics** — faithfulness, relevance, coherence, helpfulness, custom llmMetric
-- **3 Statistical Metrics** — latency, cost, tokenUsage with full percentile breakdowns
+- **6 LLM-as-Judge Metrics** - faithfulness, relevance, coherence, helpfulness, contextRelevance, custom llmMetric
+- **4 Statistical Metrics** - latency, cost, tokenUsage with full percentile breakdowns, and passAtK over trials
 - **Custom Metrics** — `metric()` factory for anything domain-specific
+- **Agent Evaluation** - `toolCalls()` and `trajectory()` against expected tool calls, `retrievalRecall()`, `retrievalPrecision()`, `retrievalMRR()` and the `contextRelevance()` judge against expected passages, multi-turn cases on one conversation
+- **Trials and pass@k** - run each case several times and score how often it passes with `passAtK()`
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
 - **A/B Testing** — EvalComparison with paired t-test and McNemar's test for statistical significance
-- **4 Reporters** — console (colored table), JSON, CSV, CI (exit code on failure)
+- **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
+- **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions, and `evalMatchers` from `@cogitator-ai/evals/matchers` (Vitest 5) adds `toHaveCalledTool`, `toFollowTrajectory`, `toHaveRetrieved` and `toScoreAtLeast`
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
 - **Baseline Workflow** — Save baselines, compare against them, catch regressions in CI
 - **Zod Validation** — Type-safe configuration with runtime checks
@@ -62,7 +66,7 @@ result.report('console');
 result.saveBaseline('./baseline.json');
 ```
 
-A target is either `{ fn: (input, { signal, case, attempt }) => Promise<string> }` or `{ agent, cogitator }`, which runs the agent with `cogitator.run(agent, { input, context, signal })` and records its usage for the statistical metrics. Defaults: `concurrency` 5, `timeout` 30 000 ms (minimum 1 000), `retries` 0.
+A target is either `{ fn: (input, { signal, case, attempt, turn, history }) => Promise<string> }` or `{ agent, cogitator, retrieval? }`, which runs the agent with `cogitator.run(agent, { input, context, signal, onToolResult })` and records its usage, tool calls, tool results and retrieved passages (see [Agent Evaluation](#agent-evaluation)). Defaults: `concurrency` 5, `timeout` 30 000 ms (minimum 1 000), `retries` 0, `trials` 1.
 
 A case that throws or exceeds `timeout` is retried up to `retries` times. A timed-out attempt is cancelled through its `signal`: an agent target's run is aborted, and a `fn` target should pass the signal on to its model call. The suite waits for the cancelled attempt to stop (at most another `timeout`) before it retries, so attempts of one case never overlap and `concurrency` holds. If every attempt fails, its result has an empty `output` and an `error` message describing the last failure.
 
@@ -82,25 +86,27 @@ const suite = new EvalSuite({
 
 `suite.run(options?: EvalRunOptions)` takes `{ maxCases?: number, signal?: AbortSignal }`: with `maxCases` only the first N cases of the dataset run, and the results, aggregates, assertions and `stats.total` cover just those cases. Aborting `signal` cancels the attempts in flight, starts no further case, and `run()` rejects with the signal's reason.
 
+A case whose every attempt failed is not scored: its empty output costs no judge calls and counts as no 0, the aggregates cover the cases that answered (`aggregated.<metric>.count` says how many), and `maxErrorRate(rate)` fails a run where too many cases errored. An agent target's case result carries `truncated`, `iterationLimitReached` and `paused` when the run ended that way.
+
 A failure never drops a case. A metric that throws or returns a score that is not a finite number (such as NaN) scores 0 for that case with the reason in `score.error`, and an `onProgress` callback that throws is reported as a process warning.
 
 `result.stats` holds:
 
-| Field          | Meaning                                                      |
-| -------------- | ------------------------------------------------------------ |
-| `total`        | Cases run                                                    |
-| `errors`       | Cases whose every attempt failed or timed out                |
-| `metricErrors` | Metric scores that failed and count as 0                     |
-| `duration`     | Wall time of the run in ms                                   |
-| `cost`         | USD spent, `targetCost + judgeCost`                          |
-| `targetCost`   | USD the target spent, over every attempt that reported usage |
-| `judgeCost`    | USD the LLM judge spent                                      |
+| Field          | Meaning                                                                                                                |
+| -------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `total`        | Cases run                                                                                                              |
+| `errors`       | Cases whose every attempt failed or timed out (with several trials, a case counts as errored when every trial errored) |
+| `metricErrors` | Metric scores that failed and count as 0, over every trial                                                             |
+| `duration`     | Wall time of the run in ms                                                                                             |
+| `cost`         | USD spent, `targetCost + judgeCost`                                                                                    |
+| `targetCost`   | USD the target spent, over every attempt and trial that reported usage                                                 |
+| `judgeCost`    | USD the LLM judge spent                                                                                                |
 
 ---
 
 ## Datasets
 
-Datasets are immutable collections of eval cases. Each case has an `input`, optional `expected`, optional `context`, and optional `metadata`.
+Datasets are immutable collections of eval cases. Each case has an `input`, optional `expected`, optional `context`, and optional `metadata`. Agent cases may also have `turns` (further user turns run on the same conversation), `expectedToolCalls` (`{ name, arguments? }[]`) and `expectedPassages` (ids or texts the retrieval should surface).
 
 ### From inline data
 
@@ -134,7 +140,15 @@ Requires `papaparse` as an optional dependency.
 const dataset = await Dataset.fromCsv('./evals/qa.csv');
 ```
 
-CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `context.*`.
+CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `context.*`, and `turns`, `expectedToolCalls` and `expectedPassages`, whose cells hold JSON arrays (CSV-quoted, an empty cell leaves the field out):
+
+```csv
+input,turns,expectedToolCalls,expectedPassages
+Book a table,"[""For two, please""]","[{""name"":""book_table"",""arguments"":{""guests"":2}}]",
+How long do refunds take?,,"[{""name"":""rag_search""}]","[""refund is issued within 5 days""]"
+```
+
+Both loaders name the line of the file a case comes from when the case schema refuses it (`Invalid case at line 3: input: ...`), the CSV loader counting quoted fields that span lines. The CSV loader splits on commas and throws on a malformed row, a quote never closed for one, with its line.
 
 ### Transformations
 
@@ -142,6 +156,7 @@ CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `co
 const filtered = dataset.filter((c) => c.expected !== undefined);
 const sampled = dataset.sample(50);
 const shuffled = dataset.shuffle();
+const nightly = dataset.sample(50, { seed: 'nightly' }); // the same cases on every run
 ```
 
 All transformations return new `Dataset` instances — the original is never mutated.
@@ -208,13 +223,14 @@ const metrics = [
 
 Metrics scored by an LLM judge (0.0 to 1.0). They require a `judge` config on the suite and a Cogitator to run the judge agent: `judge.cogitator`, or the `cogitator` of an agent target. Without one, the suite throws when it is created.
 
-| Metric         | Evaluates                               |
-| -------------- | --------------------------------------- |
-| `faithfulness` | Factual accuracy relative to input      |
-| `relevance`    | How on-topic the response is            |
-| `coherence`    | Logical structure and readability       |
-| `helpfulness`  | Practical usefulness to the user        |
-| `llmMetric`    | Custom prompt — you define the criteria |
+| Metric             | Evaluates                                    |
+| ------------------ | -------------------------------------------- |
+| `faithfulness`     | Factual accuracy relative to input           |
+| `relevance`        | How on-topic the response is                 |
+| `coherence`        | Logical structure and readability            |
+| `helpfulness`      | Practical usefulness to the user             |
+| `contextRelevance` | Whether the retrieved passages fit the input |
+| `llmMetric`        | Custom prompt, you define the criteria       |
 
 ```typescript
 import { faithfulness, relevance, llmMetric } from '@cogitator-ai/evals';
@@ -240,7 +256,9 @@ The judge reads the case's `context` too, so `faithfulness()` can check a respon
 Dataset.from([{ input: 'Write the lede', context: { dossier: dossierText } }]);
 ```
 
-The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator? }`, with `temperature` defaulting to `0`.
+When the run retrieved passages (see [Retrieval](#retrieval)), `faithfulness()` checks the response against them instead of `context`. `contextRelevance()` judges the input against the retrieved passages alone, or against `context` when nothing was retrieved, and scores 0 without a judge call when the case has neither.
+
+The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
 
 The judge answers with `{ "score", "reasoning" }` JSON. A verdict in a code fence or with the score as a numeric string is read as well, and an answer without JSON falls back to its last labelled score (`Score: 0.4`, `score is 4/5`), never to a stray number in the prose. When no verdict can be read, or the judge call fails, the metric scores 0 with the reason in `score.error`. The judge's usage is reported on each score and counted in `stats.judgeCost`.
 
@@ -261,7 +279,7 @@ const suite = new EvalSuite({
 
 ### Custom
 
-Build domain-specific metrics with the `metric()` factory.
+Build domain-specific metrics with the `metric()` factory. `evaluate` gets `input`, `output`, `expected` and `context`, the run's `toolCalls`, `toolResults`, `retrieved` and `turns`, and the whole case result as `result`.
 
 ```typescript
 import { metric } from '@cogitator-ai/evals';
@@ -285,6 +303,108 @@ Scores are automatically clamped to [0, 1]. A score that is not a finite number
 
 ---
 
+## Agent Evaluation
+
+An agent target records what the agent did, not only what it answered. Each case result carries `toolCalls` (every call of the run), `toolResults` (`{ callId, name, result?, error? }`, the raw values the run's `onToolResult` reported) and `retrieved` (`{ id?, content, score?, source? }` passages the tool results surfaced), summed over every turn of the case.
+
+### Tool calls and trajectory
+
+```typescript
+import { EvalSuite, Dataset, toolCalls, trajectory } from '@cogitator-ai/evals';
+
+const dataset = Dataset.from([
+  {
+    input: 'What is the weather in Paris?',
+    expectedToolCalls: [{ name: 'get_weather', arguments: { city: 'Paris' } }],
+  },
+]);
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [toolCalls(), trajectory({ order: 'in-order' })],
+});
+```
+
+`toolCalls({ arguments?, score? })` pairs the actual calls with the expected ones as multisets, each expected call taking at most one actual call. `arguments: 'subset'` (default) wants the expected arguments in the call with equal values, other arguments allowed, `'exact'` wants equal arguments and `'ignore'` compares names only. An expected call without `arguments` matches on its name. `score` is `'f1'` (default), `'recall'` or `'precision'`, and `details` lists the missing and unexpected calls.
+
+`trajectory({ order? })` compares the sequence of tool names: `'exact'` (default) scores 1 for the same sequence and 0 otherwise, `'in-order'` scores the longest common subsequence over the longer sequence, and `'any-order'` the names both share over the longer sequence.
+
+Both score 0 with `details: 'no expected tool calls provided'` on a case without `expectedToolCalls`, as `exactMatch()` does without `expected`. A case expecting no calls (`expectedToolCalls: []`) scores 1 when the agent made none.
+
+### Retrieval
+
+```typescript
+import {
+  retrievalRecall,
+  retrievalPrecision,
+  retrievalMRR,
+  contextRelevance,
+} from '@cogitator-ai/evals';
+import { createSearchTool } from '@cogitator-ai/rag';
+
+const agent = new Agent({ name: 'kb', model, tools: [tool(createSearchTool(pipeline))] });
+
+const suite = new EvalSuite({
+  dataset: Dataset.from([
+    { input: 'How long do refunds take?', expectedPassages: ['refund is issued within 5 days'] },
+  ]),
+  target: { agent, cogitator },
+  metrics: [retrievalRecall(), retrievalPrecision(), retrievalMRR(), contextRelevance()],
+  judge: { model: 'google/gemini-3.5-flash-lite' },
+});
+```
+
+The target's `retrieval` reads the passages out of each successful tool result. The default, `ragSearchPassages`, reads the `rag_search` tool of `@cogitator-ai/rag` (`{ results: [{ content, chunkId, score, source, ... }] }`, the `id` of a result or else its `chunkId` as the passage id) and any other result of that shape. Pass your own for other tools:
+
+```typescript
+target: {
+  agent,
+  cogitator,
+  retrieval: ({ name, result }) =>
+    name === 'search_docs' ? (result as Doc[]).map((d) => ({ id: d.slug, content: d.body })) : undefined,
+},
+```
+
+A retrieved passage matches an expected one by an equal `id`, by an equal text once case and whitespace are normalized, or when one normalized text contains the other and the contained one has at least three words, so a short id or a single word never matches every passage that happens to hold it. `retrievalRecall()` scores the share of expected passages retrieved, `retrievalPrecision()` the share of retrieved passages expected, and `retrievalMRR()` the reciprocal rank of the first match (passages rank in the order they came). They score 0 on a case without `expectedPassages`, and a case expecting none scores 1 when nothing was retrieved.
+
+### Multi-turn cases
+
+```typescript
+const cogitator = new Cogitator({ llm, memory: { adapter: 'memory' } });
+
+Dataset.from([
+  {
+    input: 'My name is Ada.',
+    turns: ['What is my name?'],
+    expected: 'Ada',
+  },
+]);
+```
+
+An agent case with `turns` runs `input`, then each turn, on one thread of its own (`threadId` `eval-<uuid>`, a fresh one for every attempt and trial). The Cogitator's memory carries the conversation from turn to turn, so it needs a memory adapter. The case `output` is the last answer, `toolCalls`, `toolResults`, `retrieved` and `usage` cover every turn, and `turns` lists each `{ input, output, toolCalls? }`. A turn that fails fails the case with its error, `turns` then ending with the failed one, and a cancelled or timed-out case starts no further turn. LLM judges read a multi-turn case as the conversation, each turn's input and answer, with the last user message marked as the one the response answers. A `fn` target is called once per turn, with `context.turn` (0 for `input`) and `context.history` (the earlier `{ input, output }` turns).
+
+### Trials and pass@k
+
+```typescript
+import { passAtK, exactMatch, threshold } from '@cogitator-ai/evals';
+
+const suite = new EvalSuite({
+  dataset,
+  target: { agent, cogitator },
+  metrics: [exactMatch()],
+  statisticalMetrics: [passAtK('exactMatch', { k: 1 }), passAtK('exactMatch', { k: 5 })],
+  assertions: [threshold('passAtK(exactMatch,5)', 0.9)],
+  trials: 5,
+});
+```
+
+`trials` (a positive integer, 1 by default) runs each case that many times. Trials share the suite's concurrency, timeout, retries, abort signal and `maxCases`, and each is scored on its own. The case result is the first trial that answered (the first one when every trial errored), so it carries an error only when all trials failed, with every trial in `trials` (each with its `scores`), and the aggregate of a metric takes the mean of a case's scored trials as that case's score. `stats.errors` counts a case as errored when every trial errored, while costs and `metricErrors` cover every trial. `latency()`, `cost()` and `tokenUsage()` see every trial as one value each, `passAtK()` sees the cases with their trials. `onProgress` fires once per case, after its last trial. `EvalComparison` takes `trials` too, comparing each case's mean over its trials.
+
+`passAtK(metric, { k, threshold = 1 })`, named `passAtK(<metric>,<k>)`, estimates per case the chance that at least one of `k` trials scores `threshold` or more, with the unbiased estimator `1 - C(n - c, k) / C(n, k)` (`n` trials, `c` passing), and averages it over the cases with at least one scored trial. An errored trial counts as a failure. The suite throws when it is created if `k` exceeds `trials` or the metric is not among its `metrics`.
+
+---
+
 ## Assertions
 
 Assertions check aggregated metrics after a suite run and produce pass/fail results.
@@ -306,6 +426,16 @@ const assertions = [
 
 Latency, cost and tokenUsage metrics (and names ending in `Duration` or `Latency`) are lower-is-better: the assertion passes when the value is at or below the threshold. `noRegression` uses the same rule.
 
+### maxErrorRate
+
+```typescript
+import { maxErrorRate } from '@cogitator-ai/evals';
+
+const assertions = [threshold('exactMatch', 0.9), maxErrorRate(0.05)];
+```
+
+Passes while at most `rate` (0 to 1) of the cases errored. Errored cases are not scored, so pair metric thresholds with it.
+
 ### noRegression
 
 Compares current results against a saved baseline file.
@@ -357,6 +487,7 @@ const comparison = new EvalComparison({
   },
   metrics: [exactMatch(), contains()],
   concurrency: 5,
+  trials: 1,
 });
 
 const result = await comparison.run();
@@ -377,22 +508,66 @@ Access full suite results via `result.baseline` and `result.challenger`. `compar
 
 Call `result.report()` after a suite run to output results.
 
-| Reporter  | Output                                          |
-| --------- | ----------------------------------------------- |
-| `console` | Colored table with metrics, assertions, summary |
-| `json`    | Writes `eval-report.json` (configurable path)   |
-| `csv`     | Writes `eval-report.csv` (configurable path)    |
-| `ci`      | Compact output, `process.exit(1)` on failure    |
+| Reporter  | Output                                                               |
+| --------- | -------------------------------------------------------------------- |
+| `console` | Colored table with metrics, assertions, summary                      |
+| `json`    | Writes `eval-report.json` (configurable path)                        |
+| `csv`     | Writes `eval-report.csv` (configurable path)                         |
+| `junit`   | Writes `eval-report.xml`, a JUnit report (configurable path)         |
+| `github`  | GitHub Actions annotations, plus a job summary in Actions            |
+| `ci`      | Compact verdict, sets `process.exitCode = 1` when an assertion fails |
 
 ```typescript
 const result = await suite.run();
 
 result.report('console');
 result.report('json', { path: './reports/eval.json' });
-result.report(['console', 'json', 'csv']);
-result.report('ci');
+result.report(['console', 'json', 'csv', 'junit'], { path: './reports/eval' });
+result.report(['github', 'ci']);
+```
+
+The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
+
+- **junit** writes one `<testsuite name="eval">` with a `<testcase>` per case (a `<failure>` when it errored, its scores in `<system-out>`) and a `<testcase>` per assertion (a `<failure>` when it did not pass). Everything is XML-escaped, and characters XML 1.0 does not allow are replaced.
+- **github** prints `::error` for each failed assertion and errored case and `::warning` for each failed metric score. When `GITHUB_STEP_SUMMARY` is set it also appends a Markdown summary with the totals, the aggregated metrics (mean, p50, p95, cases scored) and the assertions.
+
+Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output. The json report keeps each case's `toolCalls`, `toolResults`, `retrieved`, `turns` and `trials`, and the console and github reports show the trials per case.
+
+### Vitest
+
+```typescript
+// eval/qa.eval.test.ts
+import { afterAll } from 'vitest';
+import { describeEval } from '@cogitator-ai/evals/vitest';
+
+const qa = describeEval('qa agent', { suite, run: { maxCases: 50 }, timeout: 5 * 60_000 });
+
+afterAll(() => qa.result().report('junit', { path: './reports/qa' }));
+```
+
+`evalMatchers` adds assertions on results to Vitest's `expect`, typed through a module augmentation of `Assertion` and `AsymmetricMatchersContaining` (Vitest 5):
+
+```typescript
+import { expect, it } from 'vitest';
+import { evalMatchers } from '@cogitator-ai/evals/matchers';
+
+expect.extend(evalMatchers);
+
+it('books a table', async () => {
+  const result = await suite.run();
+  const [booking] = result.results;
+  expect(booking).toHaveCalledTool('book_table', { guests: 2 });
+  expect(booking).toFollowTrajectory(['find_restaurant', 'book_table'], { order: 'in-order' });
+  expect(booking).toHaveRetrieved('doc:menu');
+  expect(booking).toScoreAtLeast('toolCalls', 1);
+  expect(result).toScoreAtLeast('trajectory', 0.8);
+});
 ```
 
+`toHaveCalledTool(name, args?)` checks a case result's calls, `args` as a subset of a call's arguments. `toFollowTrajectory(names, { order? })` wants exactly those calls in that order by default, the calls in that order with others between them for `'in-order'`, and each of them in any order for `'any-order'`. `toHaveRetrieved(idOrText)` matches passages as the retrieval metrics do. `toScoreAtLeast(metric, min)` reads a case result's score or a suite result's aggregated mean, and throws when the metric is missing.
+
+`describeEval(name, { suite, run?, timeout? })` registers a `describe` block. The suite runs once in `beforeAll` (`timeout` defaults to 10 minutes, after which the run is cancelled), each case of `suite.cases(run?.maxCases)` gets a test that fails with the case error, and an `assertions` test lists every failed assertion. When the run itself fails, every test fails with that cause. `result()` returns the `EvalSuiteResult` once the run has finished.
+
 ---
 
 ## Builder API
@@ -421,6 +596,7 @@ const suite = new EvalBuilder()
   .withConcurrency(10)
   .withTimeout(60_000)
   .withRetries(2)
+  .withTrials(3)
   .onProgress(({ completed, total }) => {
     console.log(`${completed}/${total}`);
   })
@@ -484,38 +660,63 @@ result.report('ci');
 
 ### Metrics
 
-| Export         | Type          | Description                           |
-| -------------- | ------------- | ------------------------------------- |
-| `exactMatch`   | Deterministic | Exact string match                    |
-| `contains`     | Deterministic | Substring match                       |
-| `containsAll`  | Deterministic | Fraction of keywords found            |
-| `containsAny`  | Deterministic | Any configured keyword found          |
-| `similarity`   | Deterministic | Levenshtein distance similarity score |
-| `regex`        | Deterministic | Regex pattern match                   |
-| `jsonSchema`   | Deterministic | Zod schema validation                 |
-| `faithfulness` | LLM Judge     | Factual accuracy                      |
-| `relevance`    | LLM Judge     | Topical relevance                     |
-| `coherence`    | LLM Judge     | Logical structure                     |
-| `helpfulness`  | LLM Judge     | Practical usefulness                  |
-| `llmMetric`    | LLM Judge     | Custom judge prompt                   |
-| `latency`      | Statistical   | Response time percentiles             |
-| `cost`         | Statistical   | Token cost aggregation                |
-| `tokenUsage`   | Statistical   | Input/output token counts             |
-| `metric`       | Custom        | Factory for domain-specific metrics   |
+| Export               | Type          | Description                                                        |
+| -------------------- | ------------- | ------------------------------------------------------------------ |
+| `exactMatch`         | Deterministic | Exact string match                                                 |
+| `contains`           | Deterministic | Substring match                                                    |
+| `containsAll`        | Deterministic | Fraction of keywords found                                         |
+| `containsAny`        | Deterministic | Any configured keyword found                                       |
+| `similarity`         | Deterministic | Levenshtein distance similarity score                              |
+| `regex`              | Deterministic | Regex pattern match                                                |
+| `jsonSchema`         | Deterministic | Zod schema validation                                              |
+| `faithfulness`       | LLM Judge     | Factual accuracy                                                   |
+| `relevance`          | LLM Judge     | Topical relevance                                                  |
+| `coherence`          | LLM Judge     | Logical structure                                                  |
+| `helpfulness`        | LLM Judge     | Practical usefulness                                               |
+| `contextRelevance`   | LLM Judge     | Relevance of the retrieved passages to the input                   |
+| `llmMetric`          | LLM Judge     | Custom judge prompt                                                |
+| `toolCalls`          | Agent         | Tool calls against `expectedToolCalls`, F1, recall or precision    |
+| `trajectory`         | Agent         | Sequence of tool names, exact, in-order or any-order               |
+| `retrievalRecall`    | Agent         | Share of `expectedPassages` retrieved                              |
+| `retrievalPrecision` | Agent         | Share of retrieved passages expected                               |
+| `retrievalMRR`       | Agent         | Reciprocal rank of the first expected passage                      |
+| `latency`            | Statistical   | Response time percentiles                                          |
+| `cost`               | Statistical   | Token cost aggregation                                             |
+| `tokenUsage`         | Statistical   | Input/output token counts                                          |
+| `passAtK`            | Statistical   | pass@k of a metric over the trials                                 |
+| `metric`             | Custom        | Factory for domain-specific metrics                                |
+| `ragSearchPassages`  | Helper        | The default `retrieval` of agent targets, for `rag_search` results |
 
 ### Assertions
 
 | Export         | Description                                    |
 | -------------- | ---------------------------------------------- |
 | `threshold`    | Enforce min/max on metric mean                 |
+| `maxErrorRate` | Fail a run where too many cases errored        |
 | `noRegression` | Compare against saved baseline                 |
 | `assertion`    | Custom assertion with arbitrary check function |
 
 ### Reporters
 
-| Export   | Description                            |
-| -------- | -------------------------------------- |
-| `report` | Dispatch to one or more reporter types |
+| Export                             | Description                              |
+| ---------------------------------- | ---------------------------------------- |
+| `report`                           | Dispatch to one or more reporter types   |
+| `describeEval` (`/vitest` entry)   | Run a suite as a Vitest describe block   |
+| `evalMatchers` (`/matchers` entry) | Matchers for `expect.extend()`, Vitest 5 |
+
+`report` takes `console`, `json`, `csv`, `junit`, `github` and `ci`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
+
+### Types
+
+| Export             | Description                                                              |
+| ------------------ | ------------------------------------------------------------------------ |
+| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency` |
+| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`  |
+| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers              |
+| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on      |
+| `ExpectedToolCall` | A tool call a case expects: `name` and optional `arguments`              |
+| `Passage`          | A retrieved passage: `id?`, `content`, `score?`, `source?`               |
+| `EvalTurnResult`   | A turn of a multi-turn case: `input`, `output`, `toolCalls?`, `error?`   |
 
 ### Statistics
 

package/dist/assertions/error-rate.d.ts

@@ -0,0 +1,8 @@
+import type { AssertionFn } from './index.js';
+/**
+ * Passes while at most `rate` (0 to 1) of the cases errored, every attempt failed or timed out.
+ * Errored cases are not scored, so a metric's mean says nothing about them: pair metric
+ * thresholds with this to catch a run where the target mostly failed.
+ */
+export declare function maxErrorRate(rate: number): AssertionFn;
+//# sourceMappingURL=error-rate.d.ts.map

package/dist/assertions/error-rate.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"error-rate.d.ts","sourceRoot":"","sources":["../../src/assertions/error-rate.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE3C;;;;GAIG;AACH,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,GAAG,WAAW,CAoBtD"}

package/dist/assertions/error-rate.js

@@ -0,0 +1,27 @@
+/**
+ * Passes while at most `rate` (0 to 1) of the cases errored, every attempt failed or timed out.
+ * Errored cases are not scored, so a metric's mean says nothing about them: pair metric
+ * thresholds with this to catch a run where the target mostly failed.
+ */
+export function maxErrorRate(rate) {
+    if (!Number.isFinite(rate) || rate < 0 || rate > 1) {
+        throw new Error(`maxErrorRate takes a rate from 0 to 1, got ${rate}`);
+    }
+    const name = `maxErrorRate(${rate})`;
+    const percent = (value) => `${Math.round(value * 1000) / 10}%`;
+    return (_aggregated, stats) => {
+        const errors = stats.errors ?? 0;
+        const actual = stats.total === 0 ? 0 : errors / stats.total;
+        const passed = actual <= rate;
+        return {
+            name,
+            passed,
+            message: passed
+                ? `${errors} of ${stats.total} cases errored, at most ${percent(rate)} allowed`
+                : `${errors} of ${stats.total} cases errored (${percent(actual)}), expected at most ${percent(rate)}`,
+            actual,
+            expected: rate,
+        };
+    };
+}
+//# sourceMappingURL=error-rate.js.map

package/dist/assertions/error-rate.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"error-rate.js","sourceRoot":"","sources":["../../src/assertions/error-rate.ts"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,UAAU,YAAY,CAAC,IAAY;IACvC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,IAAI,GAAG,CAAC,IAAI,IAAI,GAAG,CAAC,EAAE,CAAC;QACnD,MAAM,IAAI,KAAK,CAAC,8CAA8C,IAAI,EAAE,CAAC,CAAC;IACxE,CAAC;IACD,MAAM,IAAI,GAAG,gBAAgB,IAAI,GAAG,CAAC;IACrC,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,EAAE,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,CAAC;IACvE,OAAO,CAAC,WAAW,EAAE,KAAK,EAAE,EAAE;QAC5B,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,IAAI,CAAC,CAAC;QACjC,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC;QAC5D,MAAM,MAAM,GAAG,MAAM,IAAI,IAAI,CAAC;QAC9B,OAAO;YACL,IAAI;YACJ,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,MAAM,OAAO,KAAK,CAAC,KAAK,2BAA2B,OAAO,CAAC,IAAI,CAAC,UAAU;gBAC/E,CAAC,CAAC,GAAG,MAAM,OAAO,KAAK,CAAC,KAAK,mBAAmB,OAAO,CAAC,MAAM,CAAC,uBAAuB,OAAO,CAAC,IAAI,CAAC,EAAE;YACvG,MAAM;YACN,QAAQ,EAAE,IAAI;SACf,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/assertions/index.d.ts

@@ -8,6 +8,8 @@ export interface AggregatedMetric {
     p50: number;
     p95: number;
     p99: number;
+    /** Cases the metric was scored on: errored cases are not scored */
+    count?: number;
     /** Extra figures a statistical metric reports, such as the total cost */
     metadata?: Record<string, unknown>;
 }
@@ -39,4 +41,5 @@ export type AssertionFn = (aggregated: Record<string, AggregatedMetric>, stats:
 export { threshold } from './threshold.js';
 export { noRegression } from './regression.js';
 export { assertion } from './custom.js';
+export { maxErrorRate } from './error-rate.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/assertions/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACpC;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,OAAO,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,sCAAsC;AACtC,MAAM,WAAW,SAAS;IACxB,gBAAgB;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,gFAAgF;IAChF,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,0EAA0E;IAC1E,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,4CAA4C;IAC5C,QAAQ,EAAE,MAAM,CAAC;IACjB,+CAA+C;IAC/C,IAAI,EAAE,MAAM,CAAC;IACb,mEAAmE;IACnE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8BAA8B;IAC9B,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,MAAM,WAAW,GAAG,CACxB,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,EAC5C,KAAK,EAAE,SAAS,KACb,eAAe,CAAC;AAErB,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,mEAAmE;IACnE,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACpC;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,OAAO,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,sCAAsC;AACtC,MAAM,WAAW,SAAS;IACxB,gBAAgB;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,gFAAgF;IAChF,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,0EAA0E;IAC1E,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,4CAA4C;IAC5C,QAAQ,EAAE,MAAM,CAAC;IACjB,+CAA+C;IAC/C,IAAI,EAAE,MAAM,CAAC;IACb,mEAAmE;IACnE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8BAA8B;IAC9B,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,MAAM,WAAW,GAAG,CACxB,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,EAC5C,KAAK,EAAE,SAAS,KACb,eAAe,CAAC;AAErB,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AACrC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC"}

package/dist/assertions/index.js

@@ -1,4 +1,5 @@
 export { threshold } from './threshold.js';
 export { noRegression } from './regression.js';
 export { assertion } from './custom.js';
+export { maxErrorRate } from './error-rate.js';
 //# sourceMappingURL=index.js.map

package/dist/assertions/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AA6CA,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AA+CA,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AACrC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC"}

package/dist/assertions/regression.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"regression.d.ts","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAG3C,wBAAgB,YAAY,CAAC,YAAY,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,GAAG,WAAW,CAuF7F"}
+{"version":3,"file":"regression.d.ts","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAG3C,wBAAgB,YAAY,CAAC,YAAY,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,GAAG,WAAW,CA0F7F"}

package/dist/assertions/regression.js

@@ -38,7 +38,9 @@ export function noRegression(baselinePath, opts) {
                 return {
                     name: 'noRegression',
                     passed: false,
-                    message: `'${metric}' = ${actual} is not a finite number (baseline ${baselineValue})`,
+                    message: agg.count === 0
+                        ? `'${metric}' has no scored cases, every case errored (baseline ${baselineValue})`
+                        : `'${metric}' = ${actual} is not a finite number (baseline ${baselineValue})`,
                     actual,
                     expected: baselineValue,
                 };

package/dist/assertions/regression.js.map

@@ -1 +1 @@
-{"version":3,"file":"regression.js","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,SAAS,CAAC;AAE9B,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,UAAU,YAAY,CAAC,YAAoB,EAAE,IAA6B;IAC9E,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,IAAI,QAAiC,CAAC;QACtC,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,EAAE,CAAC,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;YACnD,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;YACxC,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC;YACnC,CAAC;YACD,QAAQ,GAAG,MAAiC,CAAC;QAC/C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,iCAAiC,YAAY,EAAE;aACzD,CAAC;QACJ,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,EAAE,SAAS,IAAI,IAAI,CAAC;QAC1C,IAAI,SAAS,GAAG,CAAC,CAAC;QAElB,KAAK,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/D,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;YAC/B,IAAI,CAAC,GAAG;gBAAE,SAAS;YAEnB,SAAS,EAAE,CAAC;YACZ,MAAM,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC;YAExB,IAAI,OAAO,aAAa,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC;gBACzE,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,sBAAsB,MAAM,QAAQ,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,4CAA4C;oBACtH,MAAM;iBACP,CAAC;YACJ,CAAC;YAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC7B,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,IAAI,MAAM,OAAO,MAAM,qCAAqC,aAAa,GAAG;oBACrF,MAAM;oBACN,QAAQ,EAAE,aAAa;iBACxB,CAAC;YACJ,CAAC;YACD,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,CAAC;YAE1C,IAAI,WAAW,EAAE,CAAC;gBAChB,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;QACH,CAAC;QAED,IAAI,SAAS,KAAK,CAAC,EAAE,CAAC;YACpB,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,2EAA2E;aACrF,CAAC;QACJ,CAAC;QAED,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE,0CAA0C;SACpD,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}
+{"version":3,"file":"regression.js","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,SAAS,CAAC;AAE9B,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,UAAU,YAAY,CAAC,YAAoB,EAAE,IAA6B;IAC9E,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,IAAI,QAAiC,CAAC;QACtC,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,EAAE,CAAC,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;YACnD,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;YACxC,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC;YACnC,CAAC;YACD,QAAQ,GAAG,MAAiC,CAAC;QAC/C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,iCAAiC,YAAY,EAAE;aACzD,CAAC;QACJ,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,EAAE,SAAS,IAAI,IAAI,CAAC;QAC1C,IAAI,SAAS,GAAG,CAAC,CAAC;QAElB,KAAK,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/D,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;YAC/B,IAAI,CAAC,GAAG;gBAAE,SAAS;YAEnB,SAAS,EAAE,CAAC;YACZ,MAAM,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC;YAExB,IAAI,OAAO,aAAa,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC;gBACzE,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,sBAAsB,MAAM,QAAQ,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,4CAA4C;oBACtH,MAAM;iBACP,CAAC;YACJ,CAAC;YAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC7B,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EACL,GAAG,CAAC,KAAK,KAAK,CAAC;wBACb,CAAC,CAAC,IAAI,MAAM,uDAAuD,aAAa,GAAG;wBACnF,CAAC,CAAC,IAAI,MAAM,OAAO,MAAM,qCAAqC,aAAa,GAAG;oBAClF,MAAM;oBACN,QAAQ,EAAE,aAAa;iBACxB,CAAC;YACJ,CAAC;YACD,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,CAAC;YAE1C,IAAI,WAAW,EAAE,CAAC;gBAChB,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;QACH,CAAC;QAED,IAAI,SAAS,KAAK,CAAC,EAAE,CAAC;YACpB,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,2EAA2E;aACrF,CAAC;QACJ,CAAC;QAED,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE,0CAA0C;SACpD,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/assertions/threshold.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAoB,MAAM,SAAS,CAAC;AAoB7D,wBAAgB,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,WAAW,CAoCxE"}
+{"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAoB,MAAM,SAAS,CAAC;AAoB7D,wBAAgB,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,WAAW,CAuCxE"}

package/dist/assertions/threshold.js

@@ -23,10 +23,13 @@ export function threshold(metricName, value) {
             };
         }
         if (!Number.isFinite(actual)) {
+            const unscored = aggregated[metricName.split('.')[0]]?.count === 0;
             return {
                 name: `threshold(${metricName})`,
                 passed: false,
-                message: `${metricName} = ${actual} is not a finite number`,
+                message: unscored
+                    ? `${metricName} has no scored cases: every case errored`
+                    : `${metricName} = ${actual} is not a finite number`,
                 actual,
                 expected: value,
             };

package/dist/assertions/threshold.js.map

@@ -1 +1 @@
-{"version":3,"file":"threshold.js","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,SAAS,OAAO,CACd,UAA4C,EAC5C,IAAY;IAEZ,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC9B,MAAM,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACpC,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,KAAK,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;IAEnE,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAA2B,CAAC;IACjD,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC1B,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/D,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;AACrC,CAAC;AAED,MAAM,UAAU,SAAS,CAAC,UAAkB,EAAE,KAAa;IACzD,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,OAAO,CAAC,UAAU,EAAE,UAAU,CAAC,CAAC;QAEjE,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,WAAW,UAAU,mCAAmC;aAClE,CAAC;QACJ,CAAC;QAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC7B,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,GAAG,UAAU,MAAM,MAAM,yBAAyB;gBAC3D,MAAM;gBACN,QAAQ,EAAE,KAAK;aAChB,CAAC;QACJ,CAAC;QAED,MAAM,WAAW,GAAG,aAAa,CAAC,UAAU,CAAC,CAAC;QAC9C,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC;QAC/D,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAE5C,OAAO;YACL,IAAI,EAAE,aAAa,UAAU,GAAG;YAChC,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,IAAI,SAAS,IAAI,KAAK,EAAE;gBACnD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,cAAc,SAAS,IAAI,KAAK,EAAE;YAC/D,MAAM;YACN,QAAQ,EAAE,KAAK;SAChB,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}
+{"version":3,"file":"threshold.js","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,SAAS,OAAO,CACd,UAA4C,EAC5C,IAAY;IAEZ,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC9B,MAAM,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACpC,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,KAAK,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;IAEnE,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAA2B,CAAC;IACjD,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC1B,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/D,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;AACrC,CAAC;AAED,MAAM,UAAU,SAAS,CAAC,UAAkB,EAAE,KAAa;IACzD,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,OAAO,CAAC,UAAU,EAAE,UAAU,CAAC,CAAC;QAEjE,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,WAAW,UAAU,mCAAmC;aAClE,CAAC;QACJ,CAAC;QAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC7B,MAAM,QAAQ,GAAG,UAAU,CAAC,UAAU,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAE,CAAC,EAAE,KAAK,KAAK,CAAC,CAAC;YACpE,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,QAAQ;oBACf,CAAC,CAAC,GAAG,UAAU,0CAA0C;oBACzD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,yBAAyB;gBACtD,MAAM;gBACN,QAAQ,EAAE,KAAK;aAChB,CAAC;QACJ,CAAC;QAED,MAAM,WAAW,GAAG,aAAa,CAAC,UAAU,CAAC,CAAC;QAC9C,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC;QAC/D,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAE5C,OAAO;YACL,IAAI,EAAE,aAAa,UAAU,GAAG;YAChC,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,IAAI,SAAS,IAAI,KAAK,EAAE;gBACnD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,cAAc,SAAS,IAAI,KAAK,EAAE;YAC/D,MAAM;YACN,QAAQ,EAAE,KAAK;SAChB,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/datasets/csv-loader.d.ts

@@ -1,3 +1,8 @@
 import type { EvalCase } from '../schema.js';
+/**
+ * Cases from a CSV file with an `input` column, an optional `expected` one, `metadata.<key>` and
+ * `context.<key>` columns, and `turns`, `expectedToolCalls` and `expectedPassages` columns whose
+ * cells hold JSON arrays (an empty cell leaves the field out)
+ */
 export declare function loadCsv(path: string): Promise<EvalCase[]>;
 //# sourceMappingURL=csv-loader.d.ts.map

package/dist/datasets/csv-loader.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CA6C/D"}
+{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAK1C;;;;GAIG;AACH,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CAwF/D"}