@cogitator-ai/evals 0.8.0 → 0.9.0

package/README.md

@@ -10,6 +10,7 @@ pnpm add @cogitator-ai/evals
 # Optional dependencies
 pnpm add papaparse  # CSV dataset loading
 pnpm add @cogitator-ai/core  # agent targets and LLM-as-judge metrics
+pnpm add -D vitest  # describeEval from @cogitator-ai/evals/vitest
 ```
 
 Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogitator.app/docs/evals/datasets), [Metrics](https://cogitator.app/docs/evals/metrics), [Assertions](https://cogitator.app/docs/evals/assertions), [Comparison](https://cogitator.app/docs/evals/comparison), [Reporters](https://cogitator.app/docs/evals/reporters).
@@ -23,7 +24,8 @@ Website docs: [Evals](https://cogitator.app/docs/evals), [Datasets](https://cogi
 - **Custom Metrics** — `metric()` factory for anything domain-specific
 - **Assertions** — threshold, noRegression, custom assertion with auto-detection of lower-is-better metrics
 - **A/B Testing** — EvalComparison with paired t-test and McNemar's test for statistical significance
-- **4 Reporters** — console (colored table), JSON, CSV, CI (exit code on failure)
+- **6 Reporters** - console (colored table), JSON, CSV, JUnit XML, GitHub Actions annotations and job summary, CI (exit code on failure)
+- **Vitest** - `describeEval` from `@cogitator-ai/evals/vitest` runs a suite as a test file, one test per case plus one for the assertions
 - **Builder API** — Fluent `EvalBuilder` for composable eval pipelines
 - **Baseline Workflow** — Save baselines, compare against them, catch regressions in CI
 - **Zod Validation** — Type-safe configuration with runtime checks
@@ -82,6 +84,8 @@ const suite = new EvalSuite({
 
 `suite.run(options?: EvalRunOptions)` takes `{ maxCases?: number, signal?: AbortSignal }`: with `maxCases` only the first N cases of the dataset run, and the results, aggregates, assertions and `stats.total` cover just those cases. Aborting `signal` cancels the attempts in flight, starts no further case, and `run()` rejects with the signal's reason.
 
+A case whose every attempt failed is not scored: its empty output costs no judge calls and counts as no 0, the aggregates cover the cases that answered (`aggregated.<metric>.count` says how many), and `maxErrorRate(rate)` fails a run where too many cases errored. An agent target's case result carries `truncated`, `iterationLimitReached` and `paused` when the run ended that way.
+
 A failure never drops a case. A metric that throws or returns a score that is not a finite number (such as NaN) scores 0 for that case with the reason in `score.error`, and an `onProgress` callback that throws is reported as a process warning.
 
 `result.stats` holds:
@@ -136,12 +140,15 @@ const dataset = await Dataset.fromCsv('./evals/qa.csv');
 
 CSV must have an `input` column. Optional columns: `expected`, `metadata.*`, `context.*`.
 
+Both loaders name the line of the file a case comes from when the case schema refuses it (`Invalid case at line 3: input: ...`), the CSV loader counting quoted fields that span lines. The CSV loader splits on commas and throws on a malformed row, a quote never closed for one, with its line.
+
 ### Transformations
 
 ```typescript
 const filtered = dataset.filter((c) => c.expected !== undefined);
 const sampled = dataset.sample(50);
 const shuffled = dataset.shuffle();
+const nightly = dataset.sample(50, { seed: 'nightly' }); // the same cases on every run
 ```
 
 All transformations return new `Dataset` instances — the original is never mutated.
@@ -240,7 +247,7 @@ The judge reads the case's `context` too, so `faithfulness()` can check a respon
 Dataset.from([{ input: 'Write the lede', context: { dossier: dossierText } }]);
 ```
 
-The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator? }`, with `temperature` defaulting to `0`.
+The `judge` option is a `JudgeConfigInput`: `{ model, temperature?, maxTokens?, cogitator?, cache?, concurrency? }`, with `temperature` defaulting to `0`. `cache: true` reuses a verdict for the same model, settings and prompt in memory (a `JudgeCache` with `get` and `set` keeps them across runs), and `concurrency` caps the judge calls running at once.
 
 The judge answers with `{ "score", "reasoning" }` JSON. A verdict in a code fence or with the score as a numeric string is read as well, and an answer without JSON falls back to its last labelled score (`Score: 0.4`, `score is 4/5`), never to a stray number in the prose. When no verdict can be read, or the judge call fails, the metric scores 0 with the reason in `score.error`. The judge's usage is reported on each score and counted in `stats.judgeCost`.
 
@@ -306,6 +313,16 @@ const assertions = [
 
 Latency, cost and tokenUsage metrics (and names ending in `Duration` or `Latency`) are lower-is-better: the assertion passes when the value is at or below the threshold. `noRegression` uses the same rule.
 
+### maxErrorRate
+
+```typescript
+import { maxErrorRate } from '@cogitator-ai/evals';
+
+const assertions = [threshold('exactMatch', 0.9), maxErrorRate(0.05)];
+```
+
+Passes while at most `rate` (0 to 1) of the cases errored. Errored cases are not scored, so pair metric thresholds with it.
+
 ### noRegression
 
 Compares current results against a saved baseline file.
@@ -377,22 +394,45 @@ Access full suite results via `result.baseline` and `result.challenger`. `compar
 
 Call `result.report()` after a suite run to output results.
 
-| Reporter  | Output                                          |
-| --------- | ----------------------------------------------- |
-| `console` | Colored table with metrics, assertions, summary |
-| `json`    | Writes `eval-report.json` (configurable path)   |
-| `csv`     | Writes `eval-report.csv` (configurable path)    |
-| `ci`      | Compact output, `process.exit(1)` on failure    |
+| Reporter  | Output                                                               |
+| --------- | -------------------------------------------------------------------- |
+| `console` | Colored table with metrics, assertions, summary                      |
+| `json`    | Writes `eval-report.json` (configurable path)                        |
+| `csv`     | Writes `eval-report.csv` (configurable path)                         |
+| `junit`   | Writes `eval-report.xml`, a JUnit report (configurable path)         |
+| `github`  | GitHub Actions annotations, plus a job summary in Actions            |
+| `ci`      | Compact verdict, sets `process.exitCode = 1` when an assertion fails |
 
 ```typescript
 const result = await suite.run();
 
 result.report('console');
 result.report('json', { path: './reports/eval.json' });
-result.report(['console', 'json', 'csv']);
-result.report('ci');
+result.report(['console', 'json', 'csv', 'junit'], { path: './reports/eval' });
+result.report(['github', 'ci']);
+```
+
+The `path` is a base path: each file reporter adds its own extension (`.json`, `.csv`, `.xml`). `ci` runs last and does not end the process, so output flushes and later hooks still run before Node exits with code 1.
+
+- **junit** writes one `<testsuite name="eval">` with a `<testcase>` per case (a `<failure>` when it errored, its scores in `<system-out>`) and a `<testcase>` per assertion (a `<failure>` when it did not pass). Everything is XML-escaped, and characters XML 1.0 does not allow are replaced.
+- **github** prints `::error` for each failed assertion and errored case and `::warning` for each failed metric score. When `GITHUB_STEP_SUMMARY` is set it also appends a Markdown summary with the totals, the aggregated metrics (mean, p50, p95, cases scored) and the assertions.
+
+Agent run flags (`truncated`, `iterationLimitReached`, `paused`) show up per case in the json, csv and junit reports, and each aggregated metric's `count` (cases it was scored on) in the console, json and github output.
+
+### Vitest
+
+```typescript
+// eval/qa.eval.test.ts
+import { afterAll } from 'vitest';
+import { describeEval } from '@cogitator-ai/evals/vitest';
+
+const qa = describeEval('qa agent', { suite, run: { maxCases: 50 }, timeout: 5 * 60_000 });
+
+afterAll(() => qa.result().report('junit', { path: './reports/qa' }));
 ```
 
+`describeEval(name, { suite, run?, timeout? })` registers a `describe` block. The suite runs once in `beforeAll` (`timeout` defaults to 10 minutes, after which the run is cancelled), each case of `suite.cases(run?.maxCases)` gets a test that fails with the case error, and an `assertions` test lists every failed assertion. When the run itself fails, every test fails with that cause. `result()` returns the `EvalSuiteResult` once the run has finished.
+
 ---
 
 ## Builder API
@@ -508,14 +548,27 @@ result.report('ci');
 | Export         | Description                                    |
 | -------------- | ---------------------------------------------- |
 | `threshold`    | Enforce min/max on metric mean                 |
+| `maxErrorRate` | Fail a run where too many cases errored        |
 | `noRegression` | Compare against saved baseline                 |
 | `assertion`    | Custom assertion with arbitrary check function |
 
 ### Reporters
 
-| Export   | Description                            |
-| -------- | -------------------------------------- |
-| `report` | Dispatch to one or more reporter types |
+| Export                           | Description                            |
+| -------------------------------- | -------------------------------------- |
+| `report`                         | Dispatch to one or more reporter types |
+| `describeEval` (`/vitest` entry) | Run a suite as a Vitest describe block |
+
+`report` takes `console`, `json`, `csv`, `junit`, `github` and `ci`. `describeEval` takes `DescribeEvalOptions` and returns a `DescribeEvalHandle`.
+
+### Types
+
+| Export             | Description                                                              |
+| ------------------ | ------------------------------------------------------------------------ |
+| `JudgeConfigInput` | The `judge` option: model, settings, `cogitator`, `cache`, `concurrency` |
+| `JudgeCache`       | Where verdicts are kept across runs: `get(key)` and `set(key, verdict)`  |
+| `AggregatedMetric` | Aggregate of a metric, with `count` for the cases it covers              |
+| `MetricComparison` | A metric of a comparison, with `n` for the cases it was compared on      |
 
 ### Statistics
 

package/dist/assertions/error-rate.d.ts

@@ -0,0 +1,8 @@
+import type { AssertionFn } from './index.js';
+/**
+ * Passes while at most `rate` (0 to 1) of the cases errored, every attempt failed or timed out.
+ * Errored cases are not scored, so a metric's mean says nothing about them: pair metric
+ * thresholds with this to catch a run where the target mostly failed.
+ */
+export declare function maxErrorRate(rate: number): AssertionFn;
+//# sourceMappingURL=error-rate.d.ts.map

package/dist/assertions/error-rate.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"error-rate.d.ts","sourceRoot":"","sources":["../../src/assertions/error-rate.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAE3C;;;;GAIG;AACH,wBAAgB,YAAY,CAAC,IAAI,EAAE,MAAM,GAAG,WAAW,CAoBtD"}

package/dist/assertions/error-rate.js

@@ -0,0 +1,27 @@
+/**
+ * Passes while at most `rate` (0 to 1) of the cases errored, every attempt failed or timed out.
+ * Errored cases are not scored, so a metric's mean says nothing about them: pair metric
+ * thresholds with this to catch a run where the target mostly failed.
+ */
+export function maxErrorRate(rate) {
+    if (!Number.isFinite(rate) || rate < 0 || rate > 1) {
+        throw new Error(`maxErrorRate takes a rate from 0 to 1, got ${rate}`);
+    }
+    const name = `maxErrorRate(${rate})`;
+    const percent = (value) => `${Math.round(value * 1000) / 10}%`;
+    return (_aggregated, stats) => {
+        const errors = stats.errors ?? 0;
+        const actual = stats.total === 0 ? 0 : errors / stats.total;
+        const passed = actual <= rate;
+        return {
+            name,
+            passed,
+            message: passed
+                ? `${errors} of ${stats.total} cases errored, at most ${percent(rate)} allowed`
+                : `${errors} of ${stats.total} cases errored (${percent(actual)}), expected at most ${percent(rate)}`,
+            actual,
+            expected: rate,
+        };
+    };
+}
+//# sourceMappingURL=error-rate.js.map

package/dist/assertions/error-rate.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"error-rate.js","sourceRoot":"","sources":["../../src/assertions/error-rate.ts"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,UAAU,YAAY,CAAC,IAAY;IACvC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,IAAI,GAAG,CAAC,IAAI,IAAI,GAAG,CAAC,EAAE,CAAC;QACnD,MAAM,IAAI,KAAK,CAAC,8CAA8C,IAAI,EAAE,CAAC,CAAC;IACxE,CAAC;IACD,MAAM,IAAI,GAAG,gBAAgB,IAAI,GAAG,CAAC;IACrC,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,EAAE,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,IAAI,CAAC,GAAG,EAAE,GAAG,CAAC;IACvE,OAAO,CAAC,WAAW,EAAE,KAAK,EAAE,EAAE;QAC5B,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,IAAI,CAAC,CAAC;QACjC,MAAM,MAAM,GAAG,KAAK,CAAC,KAAK,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,MAAM,GAAG,KAAK,CAAC,KAAK,CAAC;QAC5D,MAAM,MAAM,GAAG,MAAM,IAAI,IAAI,CAAC;QAC9B,OAAO;YACL,IAAI;YACJ,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,MAAM,OAAO,KAAK,CAAC,KAAK,2BAA2B,OAAO,CAAC,IAAI,CAAC,UAAU;gBAC/E,CAAC,CAAC,GAAG,MAAM,OAAO,KAAK,CAAC,KAAK,mBAAmB,OAAO,CAAC,MAAM,CAAC,uBAAuB,OAAO,CAAC,IAAI,CAAC,EAAE;YACvG,MAAM;YACN,QAAQ,EAAE,IAAI;SACf,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/assertions/index.d.ts

@@ -8,6 +8,8 @@ export interface AggregatedMetric {
     p50: number;
     p95: number;
     p99: number;
+    /** Cases the metric was scored on: errored cases are not scored */
+    count?: number;
     /** Extra figures a statistical metric reports, such as the total cost */
     metadata?: Record<string, unknown>;
 }
@@ -39,4 +41,5 @@ export type AssertionFn = (aggregated: Record<string, AggregatedMetric>, stats:
 export { threshold } from './threshold.js';
 export { noRegression } from './regression.js';
 export { assertion } from './custom.js';
+export { maxErrorRate } from './error-rate.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/assertions/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACpC;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,OAAO,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,sCAAsC;AACtC,MAAM,WAAW,SAAS;IACxB,gBAAgB;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,gFAAgF;IAChF,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,0EAA0E;IAC1E,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,4CAA4C;IAC5C,QAAQ,EAAE,MAAM,CAAC;IACjB,+CAA+C;IAC/C,IAAI,EAAE,MAAM,CAAC;IACb,mEAAmE;IACnE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8BAA8B;IAC9B,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,MAAM,WAAW,GAAG,CACxB,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,EAC5C,KAAK,EAAE,SAAS,KACb,eAAe,CAAC;AAErB,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,GAAG,EAAE,MAAM,CAAC;IACZ,mEAAmE;IACnE,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,yEAAyE;IACzE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACpC;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,OAAO,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,sCAAsC;AACtC,MAAM,WAAW,SAAS;IACxB,gBAAgB;IAChB,KAAK,EAAE,MAAM,CAAC;IACd,gFAAgF;IAChF,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,0EAA0E;IAC1E,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,4CAA4C;IAC5C,QAAQ,EAAE,MAAM,CAAC;IACjB,+CAA+C;IAC/C,IAAI,EAAE,MAAM,CAAC;IACb,mEAAmE;IACnE,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8BAA8B;IAC9B,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,MAAM,WAAW,GAAG,CACxB,UAAU,EAAE,MAAM,CAAC,MAAM,EAAE,gBAAgB,CAAC,EAC5C,KAAK,EAAE,SAAS,KACb,eAAe,CAAC;AAErB,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AACrC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC"}

package/dist/assertions/index.js

@@ -1,4 +1,5 @@
 export { threshold } from './threshold.js';
 export { noRegression } from './regression.js';
 export { assertion } from './custom.js';
+export { maxErrorRate } from './error-rate.js';
 //# sourceMappingURL=index.js.map

package/dist/assertions/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AA6CA,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/assertions/index.ts"],"names":[],"mappings":"AA+CA,OAAO,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AACxC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AACrC,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC"}

package/dist/assertions/regression.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"regression.d.ts","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAG3C,wBAAgB,YAAY,CAAC,YAAY,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,GAAG,WAAW,CAuF7F"}
+{"version":3,"file":"regression.d.ts","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,SAAS,CAAC;AAG3C,wBAAgB,YAAY,CAAC,YAAY,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE;IAAE,SAAS,CAAC,EAAE,MAAM,CAAA;CAAE,GAAG,WAAW,CA0F7F"}

package/dist/assertions/regression.js

@@ -38,7 +38,9 @@ export function noRegression(baselinePath, opts) {
                 return {
                     name: 'noRegression',
                     passed: false,
-                    message: `'${metric}' = ${actual} is not a finite number (baseline ${baselineValue})`,
+                    message: agg.count === 0
+                        ? `'${metric}' has no scored cases, every case errored (baseline ${baselineValue})`
+                        : `'${metric}' = ${actual} is not a finite number (baseline ${baselineValue})`,
                     actual,
                     expected: baselineValue,
                 };

package/dist/assertions/regression.js.map

@@ -1 +1 @@
-{"version":3,"file":"regression.js","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,SAAS,CAAC;AAE9B,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,UAAU,YAAY,CAAC,YAAoB,EAAE,IAA6B;IAC9E,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,IAAI,QAAiC,CAAC;QACtC,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,EAAE,CAAC,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;YACnD,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;YACxC,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC;YACnC,CAAC;YACD,QAAQ,GAAG,MAAiC,CAAC;QAC/C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,iCAAiC,YAAY,EAAE;aACzD,CAAC;QACJ,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,EAAE,SAAS,IAAI,IAAI,CAAC;QAC1C,IAAI,SAAS,GAAG,CAAC,CAAC;QAElB,KAAK,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/D,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;YAC/B,IAAI,CAAC,GAAG;gBAAE,SAAS;YAEnB,SAAS,EAAE,CAAC;YACZ,MAAM,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC;YAExB,IAAI,OAAO,aAAa,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC;gBACzE,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,sBAAsB,MAAM,QAAQ,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,4CAA4C;oBACtH,MAAM;iBACP,CAAC;YACJ,CAAC;YAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC7B,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,IAAI,MAAM,OAAO,MAAM,qCAAqC,aAAa,GAAG;oBACrF,MAAM;oBACN,QAAQ,EAAE,aAAa;iBACxB,CAAC;YACJ,CAAC;YACD,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,CAAC;YAE1C,IAAI,WAAW,EAAE,CAAC;gBAChB,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;QACH,CAAC;QAED,IAAI,SAAS,KAAK,CAAC,EAAE,CAAC;YACpB,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,2EAA2E;aACrF,CAAC;QACJ,CAAC;QAED,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE,0CAA0C;SACpD,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}
+{"version":3,"file":"regression.js","sourceRoot":"","sources":["../../src/assertions/regression.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,SAAS,CAAC;AAE9B,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,MAAM,UAAU,YAAY,CAAC,YAAoB,EAAE,IAA6B;IAC9E,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,IAAI,QAAiC,CAAC;QACtC,IAAI,CAAC;YACH,MAAM,GAAG,GAAG,EAAE,CAAC,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;YACnD,MAAM,MAAM,GAAY,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;YACxC,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,MAAM,KAAK,IAAI,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC3E,MAAM,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC;YACnC,CAAC;YACD,QAAQ,GAAG,MAAiC,CAAC;QAC/C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,iCAAiC,YAAY,EAAE;aACzD,CAAC;QACJ,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,EAAE,SAAS,IAAI,IAAI,CAAC;QAC1C,IAAI,SAAS,GAAG,CAAC,CAAC;QAElB,KAAK,MAAM,CAAC,MAAM,EAAE,aAAa,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,CAAC;YAC/D,MAAM,GAAG,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;YAC/B,IAAI,CAAC,GAAG;gBAAE,SAAS;YAEnB,SAAS,EAAE,CAAC;YACZ,MAAM,MAAM,GAAG,GAAG,CAAC,IAAI,CAAC;YAExB,IAAI,OAAO,aAAa,KAAK,QAAQ,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,aAAa,CAAC,EAAE,CAAC;gBACzE,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EAAE,sBAAsB,MAAM,QAAQ,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,4CAA4C;oBACtH,MAAM;iBACP,CAAC;YACJ,CAAC;YAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC7B,OAAO;oBACL,IAAI,EAAE,cAAc;oBACpB,MAAM,EAAE,KAAK;oBACb,OAAO,EACL,GAAG,CAAC,KAAK,KAAK,CAAC;wBACb,CAAC,CAAC,IAAI,MAAM,uDAAuD,aAAa,GAAG;wBACnF,CAAC,CAAC,IAAI,MAAM,OAAO,MAAM,qCAAqC,aAAa,GAAG;oBAClF,MAAM;oBACN,QAAQ,EAAE,aAAa;iBACxB,CAAC;YACJ,CAAC;YACD,MAAM,WAAW,GAAG,aAAa,CAAC,MAAM,CAAC,CAAC;YAE1C,IAAI,WAAW,EAAE,CAAC;gBAChB,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,MAAM,KAAK,GAAG,aAAa,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC;gBAC9C,IAAI,MAAM,GAAG,KAAK,EAAE,CAAC;oBACnB,OAAO;wBACL,IAAI,EAAE,cAAc;wBACpB,MAAM,EAAE,KAAK;wBACb,OAAO,EAAE,kBAAkB,MAAM,MAAM,MAAM,MAAM,KAAK,cAAc,aAAa,eAAe,SAAS,GAAG,GAAG,IAAI;wBACrH,MAAM;wBACN,QAAQ,EAAE,KAAK;qBAChB,CAAC;gBACJ,CAAC;YACH,CAAC;QACH,CAAC;QAED,IAAI,SAAS,KAAK,CAAC,EAAE,CAAC;YACpB,OAAO;gBACL,IAAI,EAAE,cAAc;gBACpB,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,2EAA2E;aACrF,CAAC;QACJ,CAAC;QAED,OAAO;YACL,IAAI,EAAE,cAAc;YACpB,MAAM,EAAE,IAAI;YACZ,OAAO,EAAE,0CAA0C;SACpD,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/assertions/threshold.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAoB,MAAM,SAAS,CAAC;AAoB7D,wBAAgB,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,WAAW,CAoCxE"}
+{"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,WAAW,EAAoB,MAAM,SAAS,CAAC;AAoB7D,wBAAgB,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,WAAW,CAuCxE"}

package/dist/assertions/threshold.js

@@ -23,10 +23,13 @@ export function threshold(metricName, value) {
             };
         }
         if (!Number.isFinite(actual)) {
+            const unscored = aggregated[metricName.split('.')[0]]?.count === 0;
             return {
                 name: `threshold(${metricName})`,
                 passed: false,
-                message: `${metricName} = ${actual} is not a finite number`,
+                message: unscored
+                    ? `${metricName} has no scored cases: every case errored`
+                    : `${metricName} = ${actual} is not a finite number`,
                 actual,
                 expected: value,
             };

package/dist/assertions/threshold.js.map

@@ -1 +1 @@
-{"version":3,"file":"threshold.js","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,SAAS,OAAO,CACd,UAA4C,EAC5C,IAAY;IAEZ,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC9B,MAAM,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACpC,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,KAAK,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;IAEnE,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAA2B,CAAC;IACjD,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC1B,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/D,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;AACrC,CAAC;AAED,MAAM,UAAU,SAAS,CAAC,UAAkB,EAAE,KAAa;IACzD,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,OAAO,CAAC,UAAU,EAAE,UAAU,CAAC,CAAC;QAEjE,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,WAAW,UAAU,mCAAmC;aAClE,CAAC;QACJ,CAAC;QAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC7B,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,GAAG,UAAU,MAAM,MAAM,yBAAyB;gBAC3D,MAAM;gBACN,QAAQ,EAAE,KAAK;aAChB,CAAC;QACJ,CAAC;QAED,MAAM,WAAW,GAAG,aAAa,CAAC,UAAU,CAAC,CAAC;QAC9C,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC;QAC/D,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAE5C,OAAO;YACL,IAAI,EAAE,aAAa,UAAU,GAAG;YAChC,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,IAAI,SAAS,IAAI,KAAK,EAAE;gBACnD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,cAAc,SAAS,IAAI,KAAK,EAAE;YAC/D,MAAM;YACN,QAAQ,EAAE,KAAK;SAChB,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}
+{"version":3,"file":"threshold.js","sourceRoot":"","sources":["../../src/assertions/threshold.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE5C,SAAS,OAAO,CACd,UAA4C,EAC5C,IAAY;IAEZ,MAAM,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAC9B,MAAM,MAAM,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IACpC,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,KAAK,EAAE,MAAM,CAAC,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;IAEnE,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAA2B,CAAC;IACjD,MAAM,GAAG,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IAC1B,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,KAAK,EAAE,CAAC,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;IAE/D,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC;AACrC,CAAC;AAED,MAAM,UAAU,SAAS,CAAC,UAAkB,EAAE,KAAa;IACzD,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,EAAE;QAC5B,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,GAAG,OAAO,CAAC,UAAU,EAAE,UAAU,CAAC,CAAC;QAEjE,IAAI,CAAC,KAAK,EAAE,CAAC;YACX,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,WAAW,UAAU,mCAAmC;aAClE,CAAC;QACJ,CAAC;QAED,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC7B,MAAM,QAAQ,GAAG,UAAU,CAAC,UAAU,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAE,CAAC,EAAE,KAAK,KAAK,CAAC,CAAC;YACpE,OAAO;gBACL,IAAI,EAAE,aAAa,UAAU,GAAG;gBAChC,MAAM,EAAE,KAAK;gBACb,OAAO,EAAE,QAAQ;oBACf,CAAC,CAAC,GAAG,UAAU,0CAA0C;oBACzD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,yBAAyB;gBACtD,MAAM;gBACN,QAAQ,EAAE,KAAK;aAChB,CAAC;QACJ,CAAC;QAED,MAAM,WAAW,GAAG,aAAa,CAAC,UAAU,CAAC,CAAC;QAC9C,MAAM,MAAM,GAAG,WAAW,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,MAAM,IAAI,KAAK,CAAC;QAC/D,MAAM,SAAS,GAAG,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC;QAE5C,OAAO;YACL,IAAI,EAAE,aAAa,UAAU,GAAG;YAChC,MAAM;YACN,OAAO,EAAE,MAAM;gBACb,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,IAAI,SAAS,IAAI,KAAK,EAAE;gBACnD,CAAC,CAAC,GAAG,UAAU,MAAM,MAAM,cAAc,SAAS,IAAI,KAAK,EAAE;YAC/D,MAAM;YACN,QAAQ,EAAE,KAAK;SAChB,CAAC;IACJ,CAAC,CAAC;AACJ,CAAC"}

package/dist/datasets/csv-loader.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CA6C/D"}
+{"version":3,"file":"csv-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CA8E/D"}

package/dist/datasets/csv-loader.js

@@ -1,22 +1,52 @@
 import { readFile } from 'node:fs/promises';
 import { EvalCaseSchema } from '../schema.js';
+import { describeIssues } from './issues.js';
 export async function loadCsv(path) {
     let Papa;
     try {
-        Papa = await import('papaparse');
+        const loaded = (await import('papaparse'));
+        Papa = loaded.default ?? loaded;
     }
     catch {
         throw new Error('papaparse is required for CSV loading. Install it with: npm install papaparse');
     }
-    const content = await readFile(path, 'utf-8');
-    const { data, meta } = Papa.parse(content, {
-        header: true,
+    const content = (await readFile(path, 'utf-8')).replace(/^\uFEFF/, '');
+    const lineOf = lineFinder(content);
+    let fields;
+    const rows = [];
+    let rowStart = 0;
+    let malformed;
+    Papa.parse(content, {
+        delimiter: ',',
         skipEmptyLines: true,
+        step: (result, parser) => {
+            const line = lineOf(firstContent(content, rowStart));
+            rowStart = result.meta.cursor;
+            const [error] = result.errors;
+            if (error) {
+                malformed = `Invalid CSV at line ${line}: ${error.message}`;
+                parser.abort();
+                return;
+            }
+            if (!fields) {
+                fields = result.data;
+                return;
+            }
+            const row = {};
+            fields.forEach((field, at) => {
+                const value = result.data[at];
+                if (value !== undefined)
+                    row[field] = value;
+            });
+            rows.push({ row, line });
+        },
     });
-    if (!meta.fields?.includes('input')) {
+    if (malformed)
+        throw new Error(malformed);
+    if (!fields?.includes('input')) {
         throw new Error('CSV must have an "input" column');
     }
-    return data.map((row) => {
+    return rows.map(({ row, line }) => {
         const evalCase = { input: row.input };
         if (row.expected !== undefined && row.expected !== '') {
             evalCase.expected = row.expected;
@@ -37,7 +67,41 @@ export async function loadCsv(path) {
             evalCase.metadata = metadata;
         if (Object.keys(context).length > 0)
             evalCase.context = context;
-        return EvalCaseSchema.parse(evalCase);
+        const checked = EvalCaseSchema.safeParse(evalCase);
+        if (!checked.success) {
+            throw new Error(`Invalid case at line ${line}: ${describeIssues(checked.error)}`);
+        }
+        return checked.data;
     });
 }
+/** Where the text after `offset` starts, past the line breaks of blank lines before it */
+function firstContent(content, offset) {
+    let at = offset;
+    while (at < content.length && (content[at] === '\n' || content[at] === '\r'))
+        at++;
+    return at;
+}
+/** The 1-based line of an offset in `content`, a line ending at `\r\n`, `\r` or `\n` */
+function lineFinder(content) {
+    const starts = [0];
+    for (let at = 0; at < content.length; at++) {
+        const char = content[at];
+        if (char === '\r' && content[at + 1] === '\n')
+            continue;
+        if (char === '\n' || char === '\r')
+            starts.push(at + 1);
+    }
+    return (offset) => {
+        let low = 0;
+        let high = starts.length - 1;
+        while (low < high) {
+            const mid = (low + high + 1) >> 1;
+            if (starts[mid] <= offset)
+                low = mid;
+            else
+                high = mid - 1;
+        }
+        return low + 1;
+    };
+}
 //# sourceMappingURL=csv-loader.js.map

package/dist/datasets/csv-loader.js.map

@@ -1 +1 @@
-{"version":3,"file":"csv-loader.js","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAG3C,MAAM,CAAC,KAAK,UAAU,OAAO,CAAC,IAAY;IACxC,IAAI,IAAgC,CAAC;IACrC,IAAI,CAAC;QACH,IAAI,GAAG,MAAM,MAAM,CAAC,WAAW,CAAC,CAAC;IACnC,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,KAAK,CACb,+EAA+E,CAChF,CAAC;IACJ,CAAC;IAED,MAAM,OAAO,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;IAC9C,MAAM,EAAE,IAAI,EAAE,IAAI,EAAE,GAAG,IAAI,CAAC,KAAK,CAAyB,OAAO,EAAE;QACjE,MAAM,EAAE,IAAI;QACZ,cAAc,EAAE,IAAI;KACrB,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QACpC,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;IACrD,CAAC;IAED,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE;QACtB,MAAM,QAAQ,GAA4B,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,GAAG,CAAC,QAAQ,KAAK,EAAE,EAAE,CAAC;YACtD,QAAQ,CAAC,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC;QACnC,CAAC;QAED,MAAM,QAAQ,GAA4B,EAAE,CAAC;QAC7C,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,OAAO,IAAI,GAAG,KAAK,UAAU;gBAAE,SAAS;YAEpD,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBAChC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAClD,CAAC;iBAAM,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;gBACtC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAChD,CAAC;QACH,CAAC;QAED,IAAI,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACnE,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,OAAO,GAAG,OAAO,CAAC;QAEhE,OAAO,cAAc,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;IACxC,CAAC,CAAC,CAAC;AACL,CAAC"}
+{"version":3,"file":"csv-loader.js","sourceRoot":"","sources":["../../src/datasets/csv-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAG1C,MAAM,CAAC,KAAK,UAAU,OAAO,CAAC,IAAY;IACxC,IAAI,IAAgC,CAAC;IACrC,IAAI,CAAC;QACH,MAAM,MAAM,GAAG,CAAC,MAAM,MAAM,CAAC,WAAW,CAAC,CAExC,CAAC;QACF,IAAI,GAAG,MAAM,CAAC,OAAO,IAAI,MAAM,CAAC;IAClC,CAAC;IAAC,MAAM,CAAC;QACP,MAAM,IAAI,KAAK,CACb,+EAA+E,CAChF,CAAC;IACJ,CAAC;IAED,MAAM,OAAO,GAAG,CAAC,MAAM,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACvE,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,MAA4B,CAAC;IACjC,MAAM,IAAI,GAAyD,EAAE,CAAC;IACtE,IAAI,QAAQ,GAAG,CAAC,CAAC;IACjB,IAAI,SAA6B,CAAC;IAClC,IAAI,CAAC,KAAK,CAAW,OAAO,EAAE;QAC5B,SAAS,EAAE,GAAG;QACd,cAAc,EAAE,IAAI;QACpB,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE;YACvB,MAAM,IAAI,GAAG,MAAM,CAAC,YAAY,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC,CAAC;YACrD,QAAQ,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC;YAC9B,MAAM,CAAC,KAAK,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;YAC9B,IAAI,KAAK,EAAE,CAAC;gBACV,SAAS,GAAG,uBAAuB,IAAI,KAAK,KAAK,CAAC,OAAO,EAAE,CAAC;gBAC5D,MAAM,CAAC,KAAK,EAAE,CAAC;gBACf,OAAO;YACT,CAAC;YACD,IAAI,CAAC,MAAM,EAAE,CAAC;gBACZ,MAAM,GAAG,MAAM,CAAC,IAAI,CAAC;gBACrB,OAAO;YACT,CAAC;YACD,MAAM,GAAG,GAA2B,EAAE,CAAC;YACvC,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,EAAE,EAAE,EAAE;gBAC3B,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;gBAC9B,IAAI,KAAK,KAAK,SAAS;oBAAE,GAAG,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC;YAC9C,CAAC,CAAC,CAAC;YACH,IAAI,CAAC,IAAI,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC,CAAC;QAC3B,CAAC;KACF,CAAC,CAAC;IAEH,IAAI,SAAS;QAAE,MAAM,IAAI,KAAK,CAAC,SAAS,CAAC,CAAC;IAC1C,IAAI,CAAC,MAAM,EAAE,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,CAAC,CAAC;IACrD,CAAC;IAED,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,IAAI,EAAE,EAAE,EAAE;QAChC,MAAM,QAAQ,GAA4B,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;QAE/D,IAAI,GAAG,CAAC,QAAQ,KAAK,SAAS,IAAI,GAAG,CAAC,QAAQ,KAAK,EAAE,EAAE,CAAC;YACtD,QAAQ,CAAC,QAAQ,GAAG,GAAG,CAAC,QAAQ,CAAC;QACnC,CAAC;QAED,MAAM,QAAQ,GAA4B,EAAE,CAAC;QAC7C,MAAM,OAAO,GAA4B,EAAE,CAAC;QAE5C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAC/C,IAAI,GAAG,KAAK,OAAO,IAAI,GAAG,KAAK,UAAU;gBAAE,SAAS;YAEpD,IAAI,GAAG,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;gBAChC,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAClD,CAAC;iBAAM,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;gBACtC,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,KAAK,CAAC;YAChD,CAAC;QACH,CAAC;QAED,IAAI,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACnE,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,GAAG,CAAC;YAAE,QAAQ,CAAC,OAAO,GAAG,OAAO,CAAC;QAEhE,MAAM,OAAO,GAAG,cAAc,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QACnD,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,wBAAwB,IAAI,KAAK,cAAc,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACpF,CAAC;QACD,OAAO,OAAO,CAAC,IAAI,CAAC;IACtB,CAAC,CAAC,CAAC;AACL,CAAC;AAED,0FAA0F;AAC1F,SAAS,YAAY,CAAC,OAAe,EAAE,MAAc;IACnD,IAAI,EAAE,GAAG,MAAM,CAAC;IAChB,OAAO,EAAE,GAAG,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,CAAC;QAAE,EAAE,EAAE,CAAC;IACnF,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,wFAAwF;AACxF,SAAS,UAAU,CAAC,OAAe;IACjC,MAAM,MAAM,GAAa,CAAC,CAAC,CAAC,CAAC;IAC7B,KAAK,IAAI,EAAE,GAAG,CAAC,EAAE,EAAE,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,EAAE,EAAE,CAAC;QAC3C,MAAM,IAAI,GAAG,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,IAAI,IAAI,KAAK,IAAI,IAAI,OAAO,CAAC,EAAE,GAAG,CAAC,CAAC,KAAK,IAAI;YAAE,SAAS;QACxD,IAAI,IAAI,KAAK,IAAI,IAAI,IAAI,KAAK,IAAI;YAAE,MAAM,CAAC,IAAI,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC;IAC1D,CAAC;IACD,OAAO,CAAC,MAAM,EAAE,EAAE;QAChB,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,IAAI,IAAI,GAAG,MAAM,CAAC,MAAM,GAAG,CAAC,CAAC;QAC7B,OAAO,GAAG,GAAG,IAAI,EAAE,CAAC;YAClB,MAAM,GAAG,GAAG,CAAC,GAAG,GAAG,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC;YAClC,IAAI,MAAM,CAAC,GAAG,CAAE,IAAI,MAAM;gBAAE,GAAG,GAAG,GAAG,CAAC;;gBACjC,IAAI,GAAG,GAAG,GAAG,CAAC,CAAC;QACtB,CAAC;QACD,OAAO,GAAG,GAAG,CAAC,CAAC;IACjB,CAAC,CAAC;AACJ,CAAC"}

package/dist/datasets/dataset.d.ts

@@ -9,7 +9,16 @@ export declare class Dataset {
     get cases(): readonly EvalCase[];
     [Symbol.iterator](): Iterator<EvalCase>;
     filter(fn: (c: EvalCase) => boolean): Dataset;
-    sample(n: number): Dataset;
-    shuffle(): Dataset;
+    /**
+     * `n` cases picked at random, all of them when `n` is at least the size. With `seed` the pick
+     * is the same on every run, so a sampled eval compares like with like
+     */
+    sample(n: number, options?: {
+        seed?: number | string;
+    }): Dataset;
+    /** The cases in random order, the same order on every run with `seed` */
+    shuffle(options?: {
+        seed?: number | string;
+    }): Dataset;
 }
 //# sourceMappingURL=dataset.d.ts.map

package/dist/datasets/dataset.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"dataset.d.ts","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAI1C,qBAAa,OAAO;IAClB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAsB;IAE7C,OAAO;IAIP,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO;WAI1B,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;WAKzC,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAKpD,IAAI,MAAM,IAAI,MAAM,CAEnB;IAED,IAAI,KAAK,IAAI,SAAS,QAAQ,EAAE,CAE/B;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,QAAQ,CAAC,QAAQ,CAAC;IAavC,MAAM,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,QAAQ,KAAK,OAAO,GAAG,OAAO;IAI7C,MAAM,CAAC,CAAC,EAAE,MAAM,GAAG,OAAO;IAa1B,OAAO,IAAI,OAAO;CAQnB"}
+{"version":3,"file":"dataset.d.ts","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAI1C,qBAAa,OAAO;IAClB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAsB;IAE7C,OAAO;IAIP,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO;WAI1B,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;WAKzC,OAAO,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAKpD,IAAI,MAAM,IAAI,MAAM,CAEnB;IAED,IAAI,KAAK,IAAI,SAAS,QAAQ,EAAE,CAE/B;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC,IAAI,QAAQ,CAAC,QAAQ,CAAC;IAavC,MAAM,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,QAAQ,KAAK,OAAO,GAAG,OAAO;IAI7C;;;OAGG;IACH,MAAM,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;IAYpE,yEAAyE;IACzE,OAAO,CAAC,OAAO,GAAE;QAAE,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAA;KAAO,GAAG,OAAO;CAG3D"}

package/dist/datasets/dataset.js

@@ -38,25 +38,54 @@ export class Dataset {
     filter(fn) {
         return new Dataset(this._cases.filter(fn));
     }
-    sample(n) {
+    /**
+     * `n` cases picked at random, all of them when `n` is at least the size. With `seed` the pick
+     * is the same on every run, so a sampled eval compares like with like
+     */
+    sample(n, options = {}) {
         if (n <= 0 || this._cases.length === 0) {
             return new Dataset([]);
         }
         const count = Math.min(n, this._cases.length);
-        const indices = Array.from({ length: this._cases.length }, (_, i) => i);
-        for (let i = indices.length - 1; i > 0; i--) {
-            const j = Math.floor(Math.random() * (i + 1));
-            [indices[i], indices[j]] = [indices[j], indices[i]];
-        }
+        const indices = shuffled(Array.from({ length: this._cases.length }, (_, i) => i), randomOf(options.seed));
         return new Dataset(indices.slice(0, count).map((i) => this._cases[i]));
     }
-    shuffle() {
-        const arr = [...this._cases];
-        for (let i = arr.length - 1; i > 0; i--) {
-            const j = Math.floor(Math.random() * (i + 1));
-            [arr[i], arr[j]] = [arr[j], arr[i]];
-        }
-        return new Dataset(arr);
+    /** The cases in random order, the same order on every run with `seed` */
+    shuffle(options = {}) {
+        return new Dataset(shuffled([...this._cases], randomOf(options.seed)));
+    }
+}
+/** Shuffles `items` in place (Fisher-Yates) with the random numbers `random` gives */
+function shuffled(items, random) {
+    for (let i = items.length - 1; i > 0; i--) {
+        const j = Math.floor(random() * (i + 1));
+        [items[i], items[j]] = [items[j], items[i]];
+    }
+    return items;
+}
+/**
+ * `Math.random`, or a mulberry32 generator seeded by the hash of `seed`, so every number and
+ * string seeds its own sequence (`1.2` and `1.7`, `-1` and `2 ** 32 - 1` alike)
+ */
+function randomOf(seed) {
+    if (seed === undefined)
+        return Math.random;
+    let state = hashSeed(String(seed));
+    return () => {
+        state = (state + 0x6d2b79f5) >>> 0;
+        let t = state;
+        t = Math.imul(t ^ (t >>> 15), t | 1);
+        t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
+        return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
+    };
+}
+/** FNV-1a of a string, to seed the generator */
+function hashSeed(seed) {
+    let hash = 0x811c9dc5;
+    for (let i = 0; i < seed.length; i++) {
+        hash ^= seed.charCodeAt(i);
+        hash = Math.imul(hash, 0x01000193) >>> 0;
     }
+    return hash;
 }
 //# sourceMappingURL=dataset.js.map

package/dist/datasets/dataset.js.map

@@ -1 +1 @@
-{"version":3,"file":"dataset.js","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAE3C,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAC3C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,MAAM,OAAO,OAAO;IACD,MAAM,CAAsB;IAE7C,YAAoB,KAAiB;QACnC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,CAAC,IAAI,CAAC,KAAiB;QAC3B,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,IAAY;QACjC,MAAM,KAAK,GAAG,MAAM,SAAS,CAAC,IAAI,CAAC,CAAC;QACpC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,IAAY;QAC/B,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC;QAClC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,IAAI,MAAM;QACR,OAAO,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;IAC5B,CAAC;IAED,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC;QACf,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,OAAO;YACL,IAAI;gBACF,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC;oBACzB,OAAO,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;gBAChD,CAAC;gBACD,OAAO,EAAE,KAAK,EAAE,SAAkB,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;YACnD,CAAC;SACF,CAAC;IACJ,CAAC;IAED,MAAM,CAAC,EAA4B;QACjC,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC7C,CAAC;IAED,MAAM,CAAC,CAAS;QACd,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACvC,OAAO,IAAI,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,CAAC;QACD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC9C,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC;QACxE,KAAK,IAAI,CAAC,GAAG,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YAC5C,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;YAC9C,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC;QACtD,CAAC;QACD,OAAO,IAAI,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,OAAO;QACL,MAAM,GAAG,GAAG,CAAC,GAAG,IAAI,CAAC,MAAM,CAAe,CAAC;QAC3C,KAAK,IAAI,CAAC,GAAG,GAAG,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YACxC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;YAC9C,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QACtC,CAAC;QACD,OAAO,IAAI,OAAO,CAAC,GAAG,CAAC,CAAC;IAC1B,CAAC;CACF"}
+{"version":3,"file":"dataset.js","sourceRoot":"","sources":["../../src/datasets/dataset.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAE3C,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAC3C,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAEvC,MAAM,OAAO,OAAO;IACD,MAAM,CAAsB;IAE7C,YAAoB,KAAiB;QACnC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,MAAM,CAAC,IAAI,CAAC,KAAiB;QAC3B,OAAO,IAAI,OAAO,CAAC,KAAK,CAAC,CAAC;IAC5B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,IAAY;QACjC,MAAM,KAAK,GAAG,MAAM,SAAS,CAAC,IAAI,CAAC,CAAC;QACpC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,MAAM,CAAC,KAAK,CAAC,OAAO,CAAC,IAAY;QAC/B,MAAM,KAAK,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC;QAClC,OAAO,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IAC7B,CAAC;IAED,IAAI,MAAM;QACR,OAAO,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;IAC5B,CAAC;IAED,IAAI,KAAK;QACP,OAAO,IAAI,CAAC,MAAM,CAAC;IACrB,CAAC;IAED,CAAC,MAAM,CAAC,QAAQ,CAAC;QACf,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,MAAM,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC;QAC1B,OAAO;YACL,IAAI;gBACF,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC;oBACzB,OAAO,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;gBAChD,CAAC;gBACD,OAAO,EAAE,KAAK,EAAE,SAAkB,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;YACnD,CAAC;SACF,CAAC;IACJ,CAAC;IAED,MAAM,CAAC,EAA4B;QACjC,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;IAC7C,CAAC;IAED;;;OAGG;IACH,MAAM,CAAC,CAAS,EAAE,UAAsC,EAAE;QACxD,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACvC,OAAO,IAAI,OAAO,CAAC,EAAE,CAAC,CAAC;QACzB,CAAC;QACD,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;QAC9C,MAAM,OAAO,GAAG,QAAQ,CACtB,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,EACvD,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CACvB,CAAC;QACF,OAAO,IAAI,OAAO,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;IAED,yEAAyE;IACzE,OAAO,CAAC,UAAsC,EAAE;QAC9C,OAAO,IAAI,OAAO,CAAC,QAAQ,CAAC,CAAC,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACzE,CAAC;CACF;AAED,sFAAsF;AACtF,SAAS,QAAQ,CAAI,KAAU,EAAE,MAAoB;IACnD,KAAK,IAAI,CAAC,GAAG,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC1C,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QACzC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAE,EAAE,KAAK,CAAC,CAAC,CAAE,CAAC,CAAC;IAChD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED;;;GAGG;AACH,SAAS,QAAQ,CAAC,IAAiC;IACjD,IAAI,IAAI,KAAK,SAAS;QAAE,OAAO,IAAI,CAAC,MAAM,CAAC;IAC3C,IAAI,KAAK,GAAG,QAAQ,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC;IACnC,OAAO,GAAG,EAAE;QACV,KAAK,GAAG,CAAC,KAAK,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC;QACnC,IAAI,CAAC,GAAG,KAAK,CAAC;QACd,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;QACrC,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC,CAAC;QAC1C,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,UAAU,CAAC;IAC/C,CAAC,CAAC;AACJ,CAAC;AAED,gDAAgD;AAChD,SAAS,QAAQ,CAAC,IAAY;IAC5B,IAAI,IAAI,GAAG,UAAU,CAAC;IACtB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,IAAI,IAAI,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;QAC3B,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE,UAAU,CAAC,KAAK,CAAC,CAAC;IAC3C,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC"}

package/dist/datasets/issues.d.ts

@@ -0,0 +1,4 @@
+import type { z } from 'zod';
+/** What a case breaks of the case schema, one `field: problem` per issue */
+export declare function describeIssues(error: z.ZodError): string;
+//# sourceMappingURL=issues.d.ts.map

package/dist/datasets/issues.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"issues.d.ts","sourceRoot":"","sources":["../../src/datasets/issues.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAE7B,4EAA4E;AAC5E,wBAAgB,cAAc,CAAC,KAAK,EAAE,CAAC,CAAC,QAAQ,GAAG,MAAM,CAIxD"}

package/dist/datasets/issues.js

@@ -0,0 +1,7 @@
+/** What a case breaks of the case schema, one `field: problem` per issue */
+export function describeIssues(error) {
+    return error.issues
+        .map((issue) => `${issue.path.length > 0 ? issue.path.join('.') : '(case)'}: ${issue.message}`)
+        .join('; ');
+}
+//# sourceMappingURL=issues.js.map

package/dist/datasets/issues.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"issues.js","sourceRoot":"","sources":["../../src/datasets/issues.ts"],"names":[],"mappings":"AAEA,4EAA4E;AAC5E,MAAM,UAAU,cAAc,CAAC,KAAiB;IAC9C,OAAO,KAAK,CAAC,MAAM;SAChB,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,GAAG,KAAK,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,KAAK,KAAK,CAAC,OAAO,EAAE,CAAC;SAC9F,IAAI,CAAC,IAAI,CAAC,CAAC;AAChB,CAAC"}

package/dist/datasets/jsonl-loader.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"jsonl-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/jsonl-loader.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CAyBjE"}
+{"version":3,"file":"jsonl-loader.d.ts","sourceRoot":"","sources":["../../src/datasets/jsonl-loader.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,wBAAsB,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,EAAE,CAAC,CA6BjE"}

package/dist/datasets/jsonl-loader.js

@@ -1,6 +1,7 @@
 import { createReadStream } from 'node:fs';
 import { createInterface } from 'node:readline';
 import { EvalCaseSchema } from '../schema.js';
+import { describeIssues } from './issues.js';
 export async function loadJsonl(path) {
     const cases = [];
     let lineNumber = 0;
@@ -20,7 +21,11 @@ export async function loadJsonl(path) {
         catch {
             throw new Error(`Invalid JSON at line ${lineNumber}: ${trimmed}`);
         }
-        cases.push(EvalCaseSchema.parse(parsed));
+        const checked = EvalCaseSchema.safeParse(parsed);
+        if (!checked.success) {
+            throw new Error(`Invalid case at line ${lineNumber}: ${describeIssues(checked.error)}`);
+        }
+        cases.push(checked.data);
     }
     return cases;
 }

package/dist/datasets/jsonl-loader.js.map

@@ -1 +1 @@
-{"version":3,"file":"jsonl-loader.js","sourceRoot":"","sources":["../../src/datasets/jsonl-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAC;AAC3C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAChD,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAG3C,MAAM,CAAC,KAAK,UAAU,SAAS,CAAC,IAAY;IAC1C,MAAM,KAAK,GAAe,EAAE,CAAC;IAC7B,IAAI,UAAU,GAAG,CAAC,CAAC;IAEnB,MAAM,EAAE,GAAG,eAAe,CAAC;QACzB,KAAK,EAAE,gBAAgB,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,OAAO,EAAE,CAAC;QACpD,SAAS,EAAE,QAAQ;KACpB,CAAC,CAAC;IAEH,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,EAAE,EAAE,CAAC;QAC5B,UAAU,EAAE,CAAC;QACb,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC;QAC5B,IAAI,CAAC,OAAO;YAAE,SAAS;QAEvB,IAAI,MAAe,CAAC;QACpB,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;QAC/B,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,IAAI,KAAK,CAAC,wBAAwB,UAAU,KAAK,OAAO,EAAE,CAAC,CAAC;QACpE,CAAC;QAED,KAAK,CAAC,IAAI,CAAC,cAAc,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC;IAC3C,CAAC;IAED,OAAO,KAAK,CAAC;AACf,CAAC"}
+{"version":3,"file":"jsonl-loader.js","sourceRoot":"","sources":["../../src/datasets/jsonl-loader.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,SAAS,CAAC;AAC3C,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAChD,OAAO,EAAE,cAAc,EAAE,MAAM,WAAW,CAAC;AAC3C,OAAO,EAAE,cAAc,EAAE,MAAM,UAAU,CAAC;AAG1C,MAAM,CAAC,KAAK,UAAU,SAAS,CAAC,IAAY;IAC1C,MAAM,KAAK,GAAe,EAAE,CAAC;IAC7B,IAAI,UAAU,GAAG,CAAC,CAAC;IAEnB,MAAM,EAAE,GAAG,eAAe,CAAC;QACzB,KAAK,EAAE,gBAAgB,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,OAAO,EAAE,CAAC;QACpD,SAAS,EAAE,QAAQ;KACpB,CAAC,CAAC;IAEH,IAAI,KAAK,EAAE,MAAM,IAAI,IAAI,EAAE,EAAE,CAAC;QAC5B,UAAU,EAAE,CAAC;QACb,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,EAAE,CAAC;QAC5B,IAAI,CAAC,OAAO;YAAE,SAAS;QAEvB,IAAI,MAAe,CAAC;QACpB,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;QAC/B,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,IAAI,KAAK,CAAC,wBAAwB,UAAU,KAAK,OAAO,EAAE,CAAC,CAAC;QACpE,CAAC;QAED,MAAM,OAAO,GAAG,cAAc,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC;QACjD,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE,CAAC;YACrB,MAAM,IAAI,KAAK,CAAC,wBAAwB,UAAU,KAAK,cAAc,CAAC,OAAO,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAC1F,CAAC;QACD,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3B,CAAC;IAED,OAAO,KAAK,CAAC;AACf,CAAC"}

package/dist/eval-comparison.d.ts

@@ -27,6 +27,8 @@ export interface EvalComparisonRunOptions {
 export interface MetricComparison {
     baseline: number;
     challenger: number;
+    /** Cases compared: those at least one side scored, an errored side counting as a 0 */
+    n: number;
     pValue: number;
     significant: boolean;
     winner: 'baseline' | 'challenger' | 'tie';