nexus-ai-pro 1.14.0 → 1.15.0

package/API_STABILITY.md

@@ -355,6 +355,23 @@ subpaths are public and follow the 1.x rules. None is exported from the root imp
 - `AgentLoop`, `tool()`, and `ToolExecutor` keep their behaviour and are re-exported from
   `nexus-ai-pro/agent` as well as the root.
 
+## Evaluation stage (1.15.0)
+
+`nexus-ai-pro/evaluate` is public and follows the 1.x rules, and is not exported from the root.
+`Dataset`, `DatasetExample`, `Experiment`, `ExampleResult`, `MetricSummary`, and the comparison shapes
+are normalized; new optional fields and new bundled evaluators may be added in a minor release.
+
+- A dataset version is derived from example content. The hash itself is an implementation detail, but
+  the guarantee is not: identical examples produce identical versions, and any change produces a
+  different one.
+- `compareExperiments()` is statistical. Its intervals come from a seeded paired bootstrap, so the
+  same inputs give the same verdict; the exact interval bounds may change if the method improves, and
+  a verdict is a judgement about evidence rather than a compatibility guarantee.
+- `evaluate()` reports a target's failure as a failed example and an evaluator's failure as a failed
+  measurement; neither throws.
+- The existing `EvalRunner` and `MediaEvalRunner` keep their APIs unchanged. Rebuilding them on this
+  entry point is a later, internal change.
+
 ## Deprecation process
 
 Deprecated APIs are marked with `@deprecated` in declarations and described in the changelog. Removals

package/CHANGELOG.md

@@ -4,6 +4,40 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+## [1.15.0] - 2026-09-21
+
+Evaluation. One entry point for evaluating anything, datasets versioned by their content, experiments
+that can be compared with a verdict rather than a vibe, and the loop from a production failure back to
+a permanent regression test.
+
+### Added
+
+- **`nexus-ai-pro/evaluate`.** `evaluate(target, dataset, evaluators, options)` accepts any target: a
+  completion, an agent, a graph, an image operation, or plain code.
+  - Examples run concurrently, with repetitions for a target that is not deterministic, a timeout per
+    example, and per-metric statistics including a 95% interval.
+  - A target that throws is recorded as a failed example; an evaluator that throws is recorded as a
+    failed measurement. Neither voids the experiment.
+- **Datasets versioned by content.** Change an example and the version changes with it, so two
+  experiments are comparable only when they ran over the same data. `MemoryDatasetStore` and
+  `FileDatasetStore` (one reviewable JSON file per version), plus `splitOf()` for train and test
+  splits.
+- **`datasetFromTraces()`.** Turns recorded runs into examples, each keeping the run it came from, so
+  yesterday's production failure becomes tomorrow's regression test.
+- **Bundled evaluators.** `exactMatch`, `contains`, `mustNotMatch`, `completed`, `underLatency`,
+  `embeddingSimilarity` through any injected embedder, `pairwise` for side-by-side judgements, and
+  `trajectory`, which scores how an answer was reached rather than only what it said. `passRate` and
+  `totalCost` summarize a whole experiment.
+- **`compareExperiments()` with a real verdict.** A paired bootstrap over per-example differences
+  gives each metric a 95% interval; a metric whose interval spans zero is `unchanged` rather than an
+  improvement. The bootstrap is seeded, so a CI gate cannot flip between runs. The report names the
+  examples that moved most, the failures that are new, the ones that were fixed, and any dataset
+  version mismatch. `formatComparison()` prints it for a pull request.
+- **`evaluateOnline()`.** Scores sampled production runs, writes the scores back as trace feedback so
+  alert rules can watch them, and sends uncertain results to people.
+- **`AnnotationQueue`.** Review work with a rubric, claims that expire so nothing is stranded,
+  consensus across several reviewers, and `toExamples()` to turn reviewed items into dataset examples.
+
 ## [1.14.0] - 2026-09-20
 
 The memory, agent, and MCP work planned for 1.13.0 ships here together with tracing, so no 1.13.0 was
@@ -717,7 +751,8 @@ shared policy vocabulary is in place for a future release that revisits this.
 - URL fetching rejects unsafe private-network targets and limits response reads.
 - CLI and audit findings no longer disclose detected secret values.
 
-[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.14.0...HEAD
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.15.0...HEAD
+[1.15.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.14.0...v1.15.0
 [1.14.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.12.0...v1.14.0
 [1.12.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.11.0...v1.12.0
 [1.11.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v1.10.0...v1.11.0

package/README.md

@@ -724,6 +724,7 @@ The Model Context Protocol, both directions, implemented directly rather than th
 ```ts
 import { McpClient, createStdioTransport } from 'nexus-ai-pro/mcp';
 import { Tracer, MemoryTraceStore } from 'nexus-ai-pro/tracing';
+import { evaluate, createDataset, compareExperiments } from 'nexus-ai-pro/evaluate';
 
 const client = new McpClient(createStdioTransport({ command: 'npx', args: ['-y', 'some-mcp-server'] }));
 const agent = createAgent({ client: ai, tools: await client.toNexusTools({ prefix: 'files' }) });
@@ -810,6 +811,93 @@ setInterval(() => void alerts.evaluate(), 60_000);
 An alert names the rule, the measured value, and three runs that contributed, so the next step is
 reading them rather than starting an investigation.
 
+## Evaluation
+
+One entry point for evaluating anything: a completion, an agent, a graph, an image operation, or
+plain code. A dataset of examples, a target that turns an example into an output, evaluators that
+score it, and a stored experiment a later run can be compared against.
+
+```ts
+import { evaluate, createDataset, exactMatch, contains, trajectory, passRate } from 'nexus-ai-pro/evaluate';
+
+const dataset = createDataset({
+  name: 'support-questions',
+  examples: [
+    { id: 'refund', inputs: { question: 'how do I get a refund?' }, expected: 'Open the order and choose refund.' },
+    { id: 'hours', inputs: { question: 'when are you open?' }, expected: 'Weekdays, nine to five.' },
+  ],
+});
+
+const experiment = await evaluate(
+  (inputs) => agent.invoke(agentInput(inputs.question)),
+  dataset,
+  [contains(['refund']), trajectory({ expected: ['search_orders'] })],
+  { repetitions: 3, concurrency: 4, summary: [passRate()], store: experiments },
+);
+```
+
+**A dataset is versioned by its content**, so two experiments are comparable only when they really
+ran over the same examples. Change an example and the version changes with it.
+
+**Datasets can come from production.** `datasetFromTraces({ store, query: { status: 'error' } })`
+turns recorded runs into examples, each keeping the run it came from — the request that failed
+yesterday becomes tomorrow's regression test.
+
+**Evaluators are ordinary functions.** Bundled: `exactMatch`, `contains`, `mustNotMatch`, `completed`,
+`underLatency`, `embeddingSimilarity` through any embedder, `pairwise` for side-by-side judgements,
+and `trajectory`, which scores *how* an answer was reached — an agent that gets the right answer by
+calling the refund tool three times is not working. The LLM judge from `nexus-ai-pro/evals/judge`
+plugs in as one more evaluator.
+
+### Was the change real?
+
+```ts
+import { compareExperiments, formatComparison } from 'nexus-ai-pro/evaluate';
+
+const comparison = compareExperiments(baseline, candidate);
+console.log(formatComparison(comparison));
+if (comparison.regressed) process.exit(1);
+```
+
+A mean that moves from 0.81 to 0.83 says nothing on its own; with twenty examples that is noise, and
+shipping on it makes a quality gate a coin toss. The verdict comes from a **paired bootstrap** over
+the per-example differences, which assumes nothing about how the scores are distributed, and the
+pairing removes the variation caused by the examples themselves. A metric whose interval spans zero
+is reported as `unchanged`, not as an improvement. The bootstrap is seeded, so CI gives the same
+verdict twice.
+
+The report names the examples that moved most, the failures that are new, and the ones that were
+fixed. A comparison across different dataset versions says so rather than pretending it is like for
+like.
+
+### The runs you did not think of
+
+```ts
+import { evaluateOnline, AnnotationQueue } from 'nexus-ai-pro/evaluate';
+
+const queue = new AnnotationQueue({
+  rubric: [{ key: 'helpful', prompt: 'Did this answer the question?', type: 'boolean' }],
+  consensus: 2,
+});
+
+await evaluateOnline({
+  store: traceStore,
+  evaluators: [mustNotMatch([/i cannot help/i])],
+  sampleRate: 0.1,
+  reviewQueue: queue,
+  reviewWhen: (scores) => scores.some((score) => score.passed === false),
+});
+```
+
+A dataset tells you whether a change works on the cases you thought of; online evaluation tells you
+how it is doing on the rest. Scores are written back as feedback on the run, so an alert rule can
+watch them, and anything uncertain goes to a person.
+
+**The review queue keeps track of what people owe you.** Claims expire, so a reviewer who closes the
+tab does not strand an item; `consensus` lets two reviewers see the same item when one opinion is not
+enough; and `toExamples()` turns reviewed items into dataset examples, which closes the loop from a
+production failure to a permanent regression test.
+
 ## Provider Batch Tiers
 
 Both OpenAI and Anthropic sell an asynchronous tier at roughly half price, in exchange for a
@@ -2094,6 +2182,7 @@ only, so it never appears in an import graph.
 | `nexus-ai-pro/realtime/openai-webrtc` | 46 KB | 8% | none |
 | `nexus-ai-pro/security` | 40 KB | 7% | +3.4 MB |
 | `nexus-ai-pro/models` | 35 KB | 6% | none |
+| `nexus-ai-pro/evaluate` | 33 KB | 5% | none |
 | `nexus-ai-pro/images/inputs` | 31 KB | 5% | none |
 | `nexus-ai-pro/realtime/openai-websocket` | 29 KB | 5% | none |
 | `nexus-ai-pro/tracing` | 26 KB | 4% | none |

package/dist/evaluate/compare.d.ts

@@ -0,0 +1,64 @@
+import type { EvaluationScore, Experiment } from '../types/evaluate.js';
+export interface MetricComparison {
+    key: string;
+    baseline: number;
+    candidate: number;
+    delta: number;
+    /** 95% interval for the delta, from a paired bootstrap. Excludes zero when the change is real. */
+    ci95: [number, number];
+    /** `better`, `worse`, or `unchanged` — unchanged when the interval spans zero. */
+    verdict: 'better' | 'worse' | 'unchanged';
+}
+export interface ExampleComparison {
+    exampleId: string;
+    key: string;
+    baseline: number;
+    candidate: number;
+    delta: number;
+}
+export interface ExperimentComparison {
+    baseline: {
+        id: string;
+        name: string;
+    };
+    candidate: {
+        id: string;
+        name: string;
+    };
+    /** Present when the two experiments did not run over the same dataset version. */
+    datasetMismatch?: {
+        baseline: string;
+        candidate: string;
+    };
+    metrics: MetricComparison[];
+    /** The examples that moved most, worst first. */
+    regressions: ExampleComparison[];
+    improvements: ExampleComparison[];
+    /** True when any metric got worse beyond noise, or a new error appeared. */
+    regressed: boolean;
+    newErrors: string[];
+    fixedErrors: string[];
+}
+export interface CompareOptions {
+    /** Metrics where a lower number is better, such as latency or cost. */
+    lowerIsBetter?: readonly string[];
+    /** Bootstrap resamples used for the interval. Defaults to 2,000. */
+    resamples?: number;
+    /** Deterministic sampling, so a comparison in CI is reproducible. Defaults to a fixed seed. */
+    seed?: number;
+    /** Examples listed in `regressions` and `improvements`. Defaults to 10. */
+    topExamples?: number;
+}
+/**
+ * Compares two experiments and says whether the change is real.
+ *
+ * A mean that moved from 0.81 to 0.83 says nothing on its own — with twenty examples that is noise,
+ * and shipping on it is how a quality gate becomes a coin toss. The interval comes from a paired
+ * bootstrap over the per-example deltas, which needs no assumption about how the scores are
+ * distributed, and pairing is what removes the variation caused by the examples themselves.
+ */
+export declare function compareExperiments(baseline: Experiment, candidate: Experiment, options?: CompareOptions): ExperimentComparison;
+/** A comparison as text, for a pull-request comment or a CI log. */
+export declare function formatComparison(comparison: ExperimentComparison): string;
+/** Summary scores as a map, for asserting on one in a test or a gate. */
+export declare function scoreMap(scores: EvaluationScore[]): Record<string, number>;

package/dist/evaluate/compare.js

@@ -0,0 +1,148 @@
+import { stats } from './run.js';
+/**
+ * Compares two experiments and says whether the change is real.
+ *
+ * A mean that moved from 0.81 to 0.83 says nothing on its own — with twenty examples that is noise,
+ * and shipping on it is how a quality gate becomes a coin toss. The interval comes from a paired
+ * bootstrap over the per-example deltas, which needs no assumption about how the scores are
+ * distributed, and pairing is what removes the variation caused by the examples themselves.
+ */
+export function compareExperiments(baseline, candidate, options = {}) {
+    const lowerIsBetter = new Set(options.lowerIsBetter ?? ['latency', 'total-cost', 'cost']);
+    const baselineScores = index(baseline.results);
+    const candidateScores = index(candidate.results);
+    const keys = [
+        ...new Set([...baselineScores.keys(), ...candidateScores.keys()].map((entry) => entry.split('␟')[1])),
+    ];
+    const metrics = [];
+    const exampleDeltas = [];
+    for (const key of keys) {
+        const pairs = [];
+        for (const [entry, value] of baselineScores) {
+            const [exampleId, metric] = entry.split('␟');
+            if (metric !== key)
+                continue;
+            const other = candidateScores.get(entry);
+            if (other === undefined)
+                continue;
+            pairs.push({ exampleId, baseline: value, candidate: other });
+        }
+        if (pairs.length === 0)
+            continue;
+        const deltas = pairs.map((pair) => pair.candidate - pair.baseline);
+        const meanDelta = stats(deltas).mean;
+        const ci95 = bootstrapInterval(deltas, options.resamples ?? 2_000, options.seed ?? 12_345);
+        const improved = lowerIsBetter.has(key) ? meanDelta < 0 : meanDelta > 0;
+        const significant = ci95[0] > 0 || ci95[1] < 0;
+        metrics.push({
+            key,
+            baseline: stats(pairs.map((pair) => pair.baseline)).mean,
+            candidate: stats(pairs.map((pair) => pair.candidate)).mean,
+            delta: meanDelta,
+            ci95,
+            verdict: !significant ? 'unchanged' : improved ? 'better' : 'worse',
+        });
+        for (const pair of pairs) {
+            if (pair.candidate === pair.baseline)
+                continue;
+            exampleDeltas.push({
+                exampleId: pair.exampleId,
+                key,
+                baseline: pair.baseline,
+                candidate: pair.candidate,
+                delta: lowerIsBetter.has(key) ? pair.baseline - pair.candidate : pair.candidate - pair.baseline,
+            });
+        }
+    }
+    const failed = (experiment) => new Set(experiment.results.filter((result) => result.error).map((result) => result.exampleId));
+    const baselineFailures = failed(baseline);
+    const candidateFailures = failed(candidate);
+    const newErrors = [...candidateFailures].filter((id) => !baselineFailures.has(id));
+    const fixedErrors = [...baselineFailures].filter((id) => !candidateFailures.has(id));
+    const top = options.topExamples ?? 10;
+    return {
+        baseline: { id: baseline.id, name: baseline.name },
+        candidate: { id: candidate.id, name: candidate.name },
+        ...(baseline.dataset.version === candidate.dataset.version
+            ? {}
+            : { datasetMismatch: { baseline: baseline.dataset.version, candidate: candidate.dataset.version } }),
+        metrics,
+        regressions: exampleDeltas
+            .filter((item) => item.delta < 0)
+            .sort((a, b) => a.delta - b.delta)
+            .slice(0, top),
+        improvements: exampleDeltas
+            .filter((item) => item.delta > 0)
+            .sort((a, b) => b.delta - a.delta)
+            .slice(0, top),
+        regressed: metrics.some((metric) => metric.verdict === 'worse') || newErrors.length > 0,
+        newErrors,
+        fixedErrors,
+    };
+}
+/** A comparison as text, for a pull-request comment or a CI log. */
+export function formatComparison(comparison) {
+    const lines = [`${comparison.candidate.name} vs ${comparison.baseline.name}`];
+    if (comparison.datasetMismatch) {
+        lines.push(`  ! dataset differs: ${comparison.datasetMismatch.baseline} vs ${comparison.datasetMismatch.candidate}; the comparison is not like for like`);
+    }
+    for (const metric of comparison.metrics) {
+        const arrow = metric.verdict === 'better' ? '▲' : metric.verdict === 'worse' ? '▼' : '=';
+        lines.push(`  ${arrow} ${metric.key}: ${round(metric.baseline)} → ${round(metric.candidate)} (${signed(metric.delta)}, 95% CI ${round(metric.ci95[0])}..${round(metric.ci95[1])})`);
+    }
+    if (comparison.newErrors.length > 0)
+        lines.push(`  ! new failures: ${comparison.newErrors.join(', ')}`);
+    if (comparison.fixedErrors.length > 0)
+        lines.push(`  + fixed failures: ${comparison.fixedErrors.join(', ')}`);
+    for (const example of comparison.regressions.slice(0, 5)) {
+        lines.push(`  - ${example.exampleId} ${example.key}: ${round(example.baseline)} → ${round(example.candidate)}`);
+    }
+    return lines.join('\n');
+}
+function index(results) {
+    // Repetitions of one example are averaged first, so a noisy target does not dominate the pairing.
+    const grouped = new Map();
+    for (const result of results) {
+        for (const score of result.scores) {
+            const key = `${result.exampleId}␟${score.key}`;
+            grouped.set(key, [...(grouped.get(key) ?? []), score.score]);
+        }
+    }
+    return new Map([...grouped.entries()].map(([key, values]) => [key, stats(values).mean]));
+}
+/** Paired bootstrap: resample the per-example deltas and take the middle 95% of the means. */
+function bootstrapInterval(deltas, resamples, seed) {
+    if (deltas.length < 2)
+        return [0, 0];
+    const random = mulberry32(seed);
+    const means = [];
+    for (let sample = 0; sample < resamples; sample += 1) {
+        let total = 0;
+        for (let index = 0; index < deltas.length; index += 1) {
+            total += deltas[Math.floor(random() * deltas.length)];
+        }
+        means.push(total / deltas.length);
+    }
+    means.sort((a, b) => a - b);
+    return [means[Math.floor(resamples * 0.025)], means[Math.floor(resamples * 0.975)]];
+}
+/** A small deterministic generator, so a CI comparison gives the same verdict twice. */
+function mulberry32(seed) {
+    let state = seed >>> 0;
+    return () => {
+        state = (state + 0x6d_2b_79_f5) >>> 0;
+        let t = Math.imul(state ^ (state >>> 15), 1 | state);
+        t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
+        return ((t ^ (t >>> 14)) >>> 0) / 4_294_967_296;
+    };
+}
+function round(value) {
+    return Number.isInteger(value) ? String(value) : value.toFixed(3);
+}
+function signed(value) {
+    return `${value >= 0 ? '+' : ''}${round(value)}`;
+}
+/** Summary scores as a map, for asserting on one in a test or a gate. */
+export function scoreMap(scores) {
+    return Object.fromEntries(scores.map((score) => [score.key, score.score]));
+}

package/dist/evaluate/datasets.d.ts

@@ -0,0 +1,79 @@
+import type { Dataset, DatasetExample, DatasetStore, Experiment, ExperimentStore } from '../types/evaluate.js';
+import type { Run, RunQuery, TraceStore } from '../types/tracing.js';
+export interface CreateDatasetOptions<I, O> {
+    name: string;
+    examples: Array<Omit<DatasetExample<I, O>, 'id'> & {
+        id?: string;
+    }>;
+    description?: string;
+    tags?: string[];
+    /** Defaults to a hash of the examples, so identical content is the same version. */
+    version?: string;
+    now?: () => Date;
+}
+/**
+ * Builds a dataset, versioned by its content.
+ *
+ * A version derived from the examples means two experiments can be compared only when they really
+ * ran over the same data: change an example and the version changes with it, rather than silently
+ * invalidating every earlier comparison.
+ */
+export declare function createDataset<I = unknown, O = unknown>(options: CreateDatasetOptions<I, O>): Dataset<I, O>;
+export declare function contentVersion(examples: readonly DatasetExample[]): string;
+/** Examples of one split, as a dataset in its own right. */
+export declare function splitOf<I, O>(dataset: Dataset<I, O>, split: string): Dataset<I, O>;
+export interface FromTracesOptions {
+    name: string;
+    store: TraceStore;
+    query?: RunQuery;
+    /** Turns a run into an example. Defaults to its inputs and outputs. */
+    toExample?: (run: Run) => Omit<DatasetExample, 'id'> | undefined;
+    limit?: number;
+}
+/**
+ * Builds a dataset from recorded production runs.
+ *
+ * The most valuable examples are the ones that already happened: the request that failed, the answer
+ * a user marked wrong. Each example keeps `sourceRunId`, so a result can always be traced back to
+ * the run it came from.
+ */
+export declare function datasetFromTraces(options: FromTracesOptions): Promise<Dataset>;
+/** Datasets in memory, keyed by name and version. */
+export declare class MemoryDatasetStore implements DatasetStore {
+    private readonly datasets;
+    save(dataset: Dataset): void;
+    get(name: string, version?: string): Dataset | undefined;
+    list(): Array<{
+        name: string;
+        versions: string[];
+    }>;
+}
+/**
+ * Datasets as JSON files in a directory, one file per version.
+ *
+ * Readable, diffable, and reviewable: a dataset belongs in version control next to the code it
+ * tests, which a database row does not allow.
+ */
+export declare class FileDatasetStore implements DatasetStore {
+    private readonly directory;
+    constructor(directory: string);
+    save(dataset: Dataset): Promise<void>;
+    get(name: string, version?: string): Promise<Dataset | undefined>;
+    list(): Promise<Array<{
+        name: string;
+        versions: string[];
+    }>>;
+    private read;
+    private fileOf;
+}
+/** Experiments in memory, newest first. */
+export declare class MemoryExperimentStore implements ExperimentStore {
+    private readonly experiments;
+    save(experiment: Experiment): void;
+    get(id: string): Experiment | undefined;
+    list(filter?: {
+        name?: string;
+        dataset?: string;
+        limit?: number;
+    }): Experiment[];
+}

package/dist/evaluate/datasets.js

@@ -0,0 +1,144 @@
+import { createHash } from 'node:crypto';
+import { mkdir, readFile, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+/**
+ * Builds a dataset, versioned by its content.
+ *
+ * A version derived from the examples means two experiments can be compared only when they really
+ * ran over the same data: change an example and the version changes with it, rather than silently
+ * invalidating every earlier comparison.
+ */
+export function createDataset(options) {
+    const examples = options.examples.map((example, index) => ({
+        ...example,
+        id: example.id ?? `ex-${index + 1}`,
+    }));
+    return {
+        name: options.name,
+        version: options.version ?? contentVersion(examples),
+        ...(options.description ? { description: options.description } : {}),
+        examples,
+        ...(options.tags ? { tags: options.tags } : {}),
+        createdAt: (options.now ?? (() => new Date()))().toISOString(),
+    };
+}
+export function contentVersion(examples) {
+    const hash = createHash('sha256');
+    for (const example of examples) {
+        hash.update(JSON.stringify({ id: example.id, inputs: example.inputs, expected: example.expected }));
+    }
+    return `v${hash.digest('hex').slice(0, 12)}`;
+}
+/** Examples of one split, as a dataset in its own right. */
+export function splitOf(dataset, split) {
+    const examples = dataset.examples.filter((example) => example.split === split);
+    return { ...dataset, name: `${dataset.name}:${split}`, version: contentVersion(examples), examples };
+}
+/**
+ * Builds a dataset from recorded production runs.
+ *
+ * The most valuable examples are the ones that already happened: the request that failed, the answer
+ * a user marked wrong. Each example keeps `sourceRunId`, so a result can always be traced back to
+ * the run it came from.
+ */
+export async function datasetFromTraces(options) {
+    const runs = await options.store.query({ limit: options.limit ?? 100, ...options.query });
+    const examples = [];
+    for (const run of runs) {
+        const example = options.toExample?.(run) ?? { inputs: run.inputs, expected: run.outputs };
+        if (example)
+            examples.push({ ...example, id: run.id, sourceRunId: run.id });
+    }
+    return createDataset({ name: options.name, examples, description: 'Built from recorded runs' });
+}
+/** Datasets in memory, keyed by name and version. */
+export class MemoryDatasetStore {
+    datasets = new Map();
+    save(dataset) {
+        this.datasets.set(`${dataset.name}@${dataset.version}`, dataset);
+    }
+    get(name, version) {
+        if (version)
+            return this.datasets.get(`${name}@${version}`);
+        const versions = [...this.datasets.values()].filter((dataset) => dataset.name === name);
+        return versions.sort((a, b) => b.createdAt.localeCompare(a.createdAt))[0];
+    }
+    list() {
+        const byName = new Map();
+        for (const dataset of this.datasets.values()) {
+            byName.set(dataset.name, [...(byName.get(dataset.name) ?? []), dataset.version]);
+        }
+        return [...byName.entries()].map(([name, versions]) => ({ name, versions }));
+    }
+}
+/**
+ * Datasets as JSON files in a directory, one file per version.
+ *
+ * Readable, diffable, and reviewable: a dataset belongs in version control next to the code it
+ * tests, which a database row does not allow.
+ */
+export class FileDatasetStore {
+    directory;
+    constructor(directory) {
+        this.directory = directory;
+    }
+    async save(dataset) {
+        await mkdir(this.directory, { recursive: true });
+        await writeFile(this.fileOf(dataset.name, dataset.version), `${JSON.stringify(dataset, null, 2)}\n`, 'utf8');
+    }
+    async get(name, version) {
+        if (version)
+            return this.read(this.fileOf(name, version));
+        const versions = (await this.list()).find((entry) => entry.name === name)?.versions ?? [];
+        const newest = (await Promise.all(versions.map((item) => this.read(this.fileOf(name, item)))))
+            .filter((dataset) => dataset !== undefined)
+            .sort((a, b) => b.createdAt.localeCompare(a.createdAt));
+        return newest[0];
+    }
+    async list() {
+        const { readdir } = await import('node:fs/promises');
+        let files;
+        try {
+            files = await readdir(this.directory);
+        }
+        catch {
+            return [];
+        }
+        const byName = new Map();
+        for (const file of files) {
+            const match = /^(.*)@(.*)\.json$/.exec(file);
+            if (!match)
+                continue;
+            const [, name, version] = match;
+            byName.set(decodeURIComponent(name), [...(byName.get(decodeURIComponent(name)) ?? []), version]);
+        }
+        return [...byName.entries()].map(([name, versions]) => ({ name, versions }));
+    }
+    async read(file) {
+        try {
+            return JSON.parse(await readFile(file, 'utf8'));
+        }
+        catch {
+            return undefined;
+        }
+    }
+    fileOf(name, version) {
+        return path.join(this.directory, `${encodeURIComponent(name)}@${version}.json`);
+    }
+}
+/** Experiments in memory, newest first. */
+export class MemoryExperimentStore {
+    experiments = [];
+    save(experiment) {
+        this.experiments.unshift(experiment);
+    }
+    get(id) {
+        return this.experiments.find((experiment) => experiment.id === id);
+    }
+    list(filter = {}) {
+        return this.experiments
+            .filter((experiment) => (!filter.name || experiment.name === filter.name) &&
+            (!filter.dataset || experiment.dataset.name === filter.dataset))
+            .slice(0, filter.limit ?? 50);
+    }
+}