@graphorin/evals 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -1,5 +1,35 @@
 # @graphorin/evals
 
+## 0.7.0
+
+### Minor Changes
+
+- [#154](https://github.com/o-stepper/graphorin/pull/154) [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04) Thanks [@o-stepper](https://github.com/o-stepper)! - W-072: every export map's `import` condition becomes `default`, and the Node floor rises to `>=22.12.0`.
+
+  CJS consumers previously hit a bewildering `ERR_PACKAGE_PATH_NOT_EXPORTED` instead of a clear ESM-only signal. With the `default` condition, plain `require('@graphorin/core')` works via Node's stable `require(esm)` - which shipped in 22.12, hence the engines bump across every workspace manifest (packages, examples, benchmarks, docs; enforced by the widened mvp-readiness sweep). No dual-instance hazard: there is no CJS build, `require()` returns the same ESM module instance. ESM consumers are unaffected (`default` serves both paths; `types` stays first). The pack gate now runs attw under the full `node16` profile (was `esm-only`) and adds a runtime `require(esm)` smoke against the packed tarballs. Installs on Node 22.0-22.11 with `engine-strict` will refuse - upgrade Node (see the migration guide).
+
+- [#153](https://github.com/o-stepper/graphorin/pull/153) [`832f22e`](https://github.com/o-stepper/graphorin/commit/832f22e570b8c3175c1adeb4c150070cbd131534) Thanks [@o-stepper](https://github.com/o-stepper)! - LOCOMO loader fidelity (W-022): `MemoryEvalTurn` gains an optional `speaker` field carrying the dataset-native speaker NAME alongside the two-role mapping - most LOCOMO questions reference speakers by name, and the LongMemEval benchmark runner now renders `<speaker>: ...` in both the full-context prompt and the ingested memory text. Numeric LOCOMO reference answers (e.g. `2022` - 6 of the 1986 QA pairs) are stringified instead of silently collapsing to `expected: ''`; QA pairs with no reference answer at all are SKIPPED rather than emitted, so the LLM judge never grades against an empty reference (documented in the module docs; this reduces the case count for such pairs). Any locally seeded LOCOMO baselines should be re-seeded - the ingested text changed; the committed stub-fixture CI baseline is unaffected.
+
+### Patch Changes
+
+- [#164](https://github.com/o-stepper/graphorin/pull/164) [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00) Thanks [@o-stepper](https://github.com/o-stepper)! - TSDoc `{@link}` hygiene sweep (W-130): all 55 broken links found by TypeDoc's now-enabled `validation.invalidLink` are fixed - two resolved to their real targets (`GraphorinMCPError` was misnamed `MCPError`), the rest (cross-package, `import()`-form, unexported-constant, and DOM-type references that have never rendered as hrefs) converted to plain inline code. The docs build now fails on any new broken `{@link}` via a scoped gate.
+
+- [#164](https://github.com/o-stepper/graphorin/pull/164) [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00) Thanks [@o-stepper](https://github.com/o-stepper)! - Tarballs now ship `src/` so the published `dist/**/*.d.ts.map` files actually work (W-136): the maps referenced `../src/*.ts` that the `files` whitelist excluded, so go-to-definition fell back into `.d.ts` and the shipped maps were dead weight. The pack gate gains a `map-integrity` leg: every source referenced by a shipped map must resolve inside the tarball (or be embedded via `sourcesContent`), with an anti-vacuous guard - a package whose tsdown config emits declaration maps must contain a non-zero number of `.d.ts.map` files, so a cache-restored dist that silently dropped maps fails the gate instead of passing vacuously. `mvp-readiness` now requires `src` in every publishable `files` array.
+
+- Updated dependencies [[`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`832f22e`](https://github.com/o-stepper/graphorin/commit/832f22e570b8c3175c1adeb4c150070cbd131534), [`832f22e`](https://github.com/o-stepper/graphorin/commit/832f22e570b8c3175c1adeb4c150070cbd131534), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`832f22e`](https://github.com/o-stepper/graphorin/commit/832f22e570b8c3175c1adeb4c150070cbd131534), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`4ee256e`](https://github.com/o-stepper/graphorin/commit/4ee256e30fe9190cef6c48dc6785464757707156), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`32bbd03`](https://github.com/o-stepper/graphorin/commit/32bbd03b588136a355e4b5ad6ac5e19b36b4d8ab), [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00), [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00), [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00), [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00), [`764239b`](https://github.com/o-stepper/graphorin/commit/764239b97e0e0202442e91272583f13adeb12d00), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04), [`fe98522`](https://github.com/o-stepper/graphorin/commit/fe98522ce2477c9a7dc09029f9dcfdb1f7c9aa04)]:
+  - @graphorin/core@0.7.0
+  - @graphorin/observability@0.7.0
+
+## 0.6.1
+
+### Patch Changes
+
+- [#142](https://github.com/o-stepper/graphorin/pull/142) [`436d6ca`](https://github.com/o-stepper/graphorin/commit/436d6ca5ebbd16df094e915682d3915c279a8430) Thanks [@o-stepper](https://github.com/o-stepper)! - Version constants and version-bearing strings now derive from each package's manifest at build time (`VERSION = pkg.version`; writer ids, client/server info, OTLP framework attributes, build-info metrics interpolate it). No behavioral change at the current version: the rendered strings are byte-identical. A release bump no longer edits source; the new `check-version-consistency` gate fails any reintroduced hardcoded framework version.
+
+- Updated dependencies [[`826ee59`](https://github.com/o-stepper/graphorin/commit/826ee5933ad38693b0dd2f20a110abfecba7d23d), [`436d6ca`](https://github.com/o-stepper/graphorin/commit/436d6ca5ebbd16df094e915682d3915c279a8430)]:
+  - @graphorin/observability@0.6.1
+  - @graphorin/core@0.6.1
+
 ## 0.6.0
 
 ### Minor Changes

package/README.md

@@ -7,14 +7,14 @@
 > bounded concurrency, and regression detection that compares the
 > current run against a stored baseline.
 >
-> Project Graphorin · v0.6.0 · MIT License · © 2026 Oleksiy Stepurenko ·
+> Project Graphorin · v0.7.0 · MIT License · © 2026 Oleksiy Stepurenko ·
 > <https://github.com/o-stepper/graphorin>
 
 ---
 
 ## Status
 
-- **Published:** v0.6.0 (optional sub-pack; the full orchestrator is
+- **Published:** v0.7.0 (optional sub-pack; the full orchestrator is
   decoupled from `@graphorin/observability` per RB-17 / DEC-152).
 
 ---
@@ -103,6 +103,25 @@ const report = await runEvals({
 });
 ```
 
+Every report summary carries a Wilson 95% confidence interval for the
+pass rate (`summary.passRateCi`), and, when `iterations > 1`,
+`summary.passHatK` - the pass^k estimate over the per-case iteration
+outcomes (the probability that all k iterations of a case pass).
+
+---
+
+## Statistics
+
+Sample-size-aware statistics ship as plain exported functions:
+
+```ts
+import { mean, sampleStddev, wilsonInterval, passHatK, pairedPassSignificance } from '@graphorin/evals';
+
+wilsonInterval(18, 20);            // { lo, hi } - 95% CI for 18/20 passes
+passHatK(outcomesByCase);          // pass^k over `-iter-N` outcome groups
+pairedPassSignificance(a, b);      // McNemar paired test between two runs
+```
+
 ---
 
 ## Regression detection
@@ -116,6 +135,9 @@ const regression = detectRegressions(report, baseline, {
   maxPassRateDropPct: 5,
   maxAvgScoreDrop: 0.05,
   maxAvgDurationIncreaseMs: 250,
+  // Opt-in: veto pass-rate-drop findings the McNemar paired test
+  // cannot distinguish from noise at the given alpha (default 0.05).
+  requireSignificance: true,
 });
 if (regression.hasRegressions) {
   for (const f of regression.findings) {
@@ -125,6 +147,10 @@ if (regression.hasRegressions) {
 exitOnFailures(report, regression);
 ```
 
+Pass-rate findings are annotated with the paired regressed / improved
+case counts and the McNemar p-value, so a 3-point drop on 20 cases
+reads differently from the same drop on 2000.
+
 ---
 
 ## Multi-format report writing
@@ -154,4 +180,4 @@ MIT © 2026 Oleksiy Stepurenko
 
 ---
 
-**Project Graphorin** · v0.6.0 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>
+**Project Graphorin** · v0.7.0 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>

package/dist/index.d.ts

@@ -60,8 +60,7 @@ import { PairedSignificance, mean, pairedPassSignificance, passByBaseCase, passH
  *
  * @packageDocumentation
  */
-/** Canonical version constant. Mirrors the `package.json` version. */
-declare const VERSION = "0.6.0";
+declare const VERSION: string;
 //#endregion
 export { type AgentLike, type ArgumentValidityOptions, type Case, type CorrectToolSelectedOptions, type Dataset, type EvalCaseResult, type EvalReport, type ExactMatchOptions, type FinalStateCorrectOptions, type FromTracesOptions, type JsonPathOptions, type LlmJudgeOptions, type LoadCsvOptions, type LoadDmrOptions, type LoadJsonlOptions, type LoadLocomoOptions, type LoadLongMemEvalOptions, type MemoryEvalAbility, type MemoryEvalInput, type MemoryEvalSession, type MemoryEvalTurn, type PairedSignificance, type PrebuiltScorerOptions, type PredicateOptions, type ProgressEvent, type RecoveryAfterErrorOptions, type RedundantCallDetectionOptions, type RegexMatchOptions, type RegressionFinding, type RegressionOptions, type RegressionReport, type ReporterFormat, type RunEvalOptions, type RunOptions, type ScoreResult, type Scorer, type TraceEvent, type Trajectory, type TrajectoryToolCall, VERSION, type WriteReportsOptions, type WrittenReport, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, mean, pairedPassSignificance, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, passByBaseCase, passHatK, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, sampleStddev, scoreContract, stripIterationSuffix, toxicityScorer, wilsonInterval, writeReports };
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cAgCa,OAAA"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cAkCa"}

package/dist/index.js

@@ -1,3 +1,4 @@
+import { version } from "./package.js";
 import { mean, pairedPassSignificance, passByBaseCase, passHatK, sampleStddev, stripIterationSuffix, wilsonInterval } from "./stats.js";
 import { detectRegressions } from "./regression.js";
 import { renderHtmlReport } from "./reporters/html.js";
@@ -57,8 +58,8 @@ import { redundantCallDetection } from "./scorers/trajectory/redundant-call-dete
 *
 * @packageDocumentation
 */
-/** Canonical version constant. Mirrors the `package.json` version. */
-const VERSION = "0.6.0";
+/** Canonical version constant, derived from `package.json` at build time. */
+const VERSION = version;
 
 //#endregion
 export { VERSION, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, mean, pairedPassSignificance, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, passByBaseCase, passHatK, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, sampleStddev, scoreContract, stripIterationSuffix, toxicityScorer, wilsonInterval, writeReports };

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","names":[],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/evals - eval framework for the Graphorin framework.\n *\n * Builds on the eval primitives shipped from `@graphorin/observability`\n * (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled\n * from the core observability bundle so consumers do not pay the\n * dataset / reporter cost when only the inline runner is needed.\n *\n * ```ts\n * import {\n *   runEvals,\n *   loadJsonlDataset,\n *   exactMatch,\n *   renderTerminalReport,\n *   exitOnFailures,\n * } from '@graphorin/evals';\n *\n * const dataset = await loadJsonlDataset('./fixtures/golden.jsonl');\n * const report = await runEvals({\n *   agent,\n *   dataset,\n *   scorers: [exactMatch()],\n *   concurrency: 4,\n * });\n * console.log(renderTerminalReport(report));\n * exitOnFailures(report);\n * ```\n *\n * @packageDocumentation\n */\n\n/** Canonical version constant. Mirrors the `package.json` version. */\nexport const VERSION = '0.6.0';\n\nexport {\n  detectRegressions,\n  exitOnFailures,\n  type ReporterFormat,\n  type WriteReportsOptions,\n  type WrittenReport,\n  writeReports,\n} from './cli/index.js';\nexport type {\n  FromTracesOptions,\n  LoadDmrOptions,\n  LoadLocomoOptions,\n  LoadLongMemEvalOptions,\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n  TraceEvent,\n} from './loaders/index.js';\nexport {\n  fromIterable,\n  type LoadCsvOptions,\n  type LoadJsonlOptions,\n  loadCsvDataset,\n  loadDatasetFromTraces,\n  loadDmrDataset,\n  loadJsonlDataset,\n  loadLocomoDataset,\n  loadLongMemEvalDataset,\n  parseCsv,\n  parseDmr,\n  parseJsonl,\n  parseLocomo,\n  parseLongMemEval,\n} from './loaders/index.js';\nexport { detectRegressions as detectRegressionsFromReports } from './regression.js';\nexport {\n  renderHtmlReport,\n  renderJsonReport,\n  renderJunitReport,\n  renderMarkdownReport,\n  renderTerminalReport,\n} from './reporters/index.js';\nexport { runEvals } from './runner.js';\nexport {\n  type ArgumentValidityOptions,\n  argumentValidity,\n  type CorrectToolSelectedOptions,\n  correctToolSelected,\n  type ExactMatchOptions,\n  exactMatch,\n  type FinalStateCorrectOptions,\n  factualityScorer,\n  fenceForJudge,\n  finalStateCorrect,\n  helpfulnessScorer,\n  type JsonPathOptions,\n  jsonPath,\n  type LlmJudgeOptions,\n  llmJudge,\n  type PrebuiltScorerOptions,\n  type PredicateOptions,\n  parseScore,\n  predicate,\n  type RecoveryAfterErrorOptions,\n  type RedundantCallDetectionOptions,\n  type RegexMatchOptions,\n  recoveryAfterError,\n  redundantCallDetection,\n  regexMatch,\n  scoreContract,\n  type Trajectory,\n  type TrajectoryToolCall,\n  toxicityScorer,\n} from './scorers/index.js';\nexport {\n  mean,\n  type PairedSignificance,\n  pairedPassSignificance,\n  passByBaseCase,\n  passHatK,\n  sampleStddev,\n  stripIterationSuffix,\n  wilsonInterval,\n} from './stats.js';\nexport type {\n  AgentLike,\n  Case,\n  Dataset,\n  EvalCaseResult,\n  EvalReport,\n  ProgressEvent,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n  RunEvalOptions,\n  RunOptions,\n  ScoreResult,\n  Scorer,\n} from './types.js';\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAgCA,MAAa,UAAU"}
+{"version":3,"file":"index.js","names":["VERSION: string","pkg.version"],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/evals - eval framework for the Graphorin framework.\n *\n * Builds on the eval primitives shipped from `@graphorin/observability`\n * (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled\n * from the core observability bundle so consumers do not pay the\n * dataset / reporter cost when only the inline runner is needed.\n *\n * ```ts\n * import {\n *   runEvals,\n *   loadJsonlDataset,\n *   exactMatch,\n *   renderTerminalReport,\n *   exitOnFailures,\n * } from '@graphorin/evals';\n *\n * const dataset = await loadJsonlDataset('./fixtures/golden.jsonl');\n * const report = await runEvals({\n *   agent,\n *   dataset,\n *   scorers: [exactMatch()],\n *   concurrency: 4,\n * });\n * console.log(renderTerminalReport(report));\n * exitOnFailures(report);\n * ```\n *\n * @packageDocumentation\n */\n\n/** Canonical version constant, derived from `package.json` at build time. */\nimport pkg from '../package.json' with { type: 'json' };\n\nexport const VERSION: string = pkg.version;\n\nexport {\n  detectRegressions,\n  exitOnFailures,\n  type ReporterFormat,\n  type WriteReportsOptions,\n  type WrittenReport,\n  writeReports,\n} from './cli/index.js';\nexport type {\n  FromTracesOptions,\n  LoadDmrOptions,\n  LoadLocomoOptions,\n  LoadLongMemEvalOptions,\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n  TraceEvent,\n} from './loaders/index.js';\nexport {\n  fromIterable,\n  type LoadCsvOptions,\n  type LoadJsonlOptions,\n  loadCsvDataset,\n  loadDatasetFromTraces,\n  loadDmrDataset,\n  loadJsonlDataset,\n  loadLocomoDataset,\n  loadLongMemEvalDataset,\n  parseCsv,\n  parseDmr,\n  parseJsonl,\n  parseLocomo,\n  parseLongMemEval,\n} from './loaders/index.js';\nexport { detectRegressions as detectRegressionsFromReports } from './regression.js';\nexport {\n  renderHtmlReport,\n  renderJsonReport,\n  renderJunitReport,\n  renderMarkdownReport,\n  renderTerminalReport,\n} from './reporters/index.js';\nexport { runEvals } from './runner.js';\nexport {\n  type ArgumentValidityOptions,\n  argumentValidity,\n  type CorrectToolSelectedOptions,\n  correctToolSelected,\n  type ExactMatchOptions,\n  exactMatch,\n  type FinalStateCorrectOptions,\n  factualityScorer,\n  fenceForJudge,\n  finalStateCorrect,\n  helpfulnessScorer,\n  type JsonPathOptions,\n  jsonPath,\n  type LlmJudgeOptions,\n  llmJudge,\n  type PrebuiltScorerOptions,\n  type PredicateOptions,\n  parseScore,\n  predicate,\n  type RecoveryAfterErrorOptions,\n  type RedundantCallDetectionOptions,\n  type RegexMatchOptions,\n  recoveryAfterError,\n  redundantCallDetection,\n  regexMatch,\n  scoreContract,\n  type Trajectory,\n  type TrajectoryToolCall,\n  toxicityScorer,\n} from './scorers/index.js';\nexport {\n  mean,\n  type PairedSignificance,\n  pairedPassSignificance,\n  passByBaseCase,\n  passHatK,\n  sampleStddev,\n  stripIterationSuffix,\n  wilsonInterval,\n} from './stats.js';\nexport type {\n  AgentLike,\n  Case,\n  Dataset,\n  EvalCaseResult,\n  EvalReport,\n  ProgressEvent,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n  RunEvalOptions,\n  RunOptions,\n  ScoreResult,\n  Scorer,\n} from './types.js';\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAkCA,MAAaA,UAAkBC"}

package/dist/loaders/locomo.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"locomo.d.ts","names":[],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":[],"mappings":";;;;;;UA6CiB,iBAAA;;;;;;;;;;;;;;;iBAgBK,iBAAA,UACX,oBACR,QAAQ,QAAQ;;;;;;;iBAmBH,WAAA,gBAA2B,cAAc,KAAK"}
+{"version":3,"file":"locomo.d.ts","names":[],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":[],"mappings":";;;;;;UAuDiB,iBAAA;;;;;;;;;;;;;;;iBAgBK,iBAAA,UACX,oBACR,QAAQ,QAAQ;;;;;;;iBAmBH,WAAA,gBAA2B,cAAc,KAAK"}

package/dist/loaders/locomo.js

@@ -31,6 +31,16 @@ import { readFile } from "node:fs/promises";
 * (adversarial), everything else (`3` open-domain, `4` single-hop)
 * →info-extraction.
 *
+* Answer handling (W-022): LOCOMO answers may be numbers (e.g. `2022`)
+* - they are stringified, never silently coerced to `''`. A QA pair
+* with NO answer at all (neither `answer` nor `adversarial_answer`) is
+* SKIPPED rather than emitted with an empty `expected`: the LLM judge
+* grading against an empty reference produces meaningless verdicts, so
+* a missing reference must reduce the case count, not poison scores.
+* Turns carry the dataset-native `speaker` name alongside the
+* two-role mapping (~99 percent of LOCOMO questions reference speakers
+* by name).
+*
 * @packageDocumentation
 */
 /**
@@ -79,7 +89,8 @@ function appendSampleCases(sample, index, out) {
 		if (question === void 0) continue;
 		const category = typeof item.category === "number" ? item.category : void 0;
 		const ability = mapCategory(category);
-		const answer = asString(item.answer) ?? asString(item.adversarial_answer) ?? "";
+		const answer = answerToText(item.answer) ?? answerToText(item.adversarial_answer);
+		if (answer === void 0) continue;
 		const evidence = asArray(item.evidence).filter((e) => typeof e === "string");
 		out.push({
 			id: `${sampleId}-q${q}`,
@@ -121,7 +132,8 @@ function extractSessions(conversation) {
 			turns.push({
 				role,
 				content,
-				...date !== void 0 ? { timestamp: date } : {}
+				...date !== void 0 ? { timestamp: date } : {},
+				...speaker !== void 0 ? { speaker } : {}
 			});
 		}
 		sessions.push({
@@ -143,6 +155,16 @@ function mapCategory(category) {
 function asString(value) {
 	return typeof value === "string" ? value : void 0;
 }
+/**
+* LOCOMO reference answers may be strings OR bare numbers (6 of the
+* 1986 QA pairs ship e.g. `2022`). Stringify finite numbers instead of
+* dropping them to `''` (W-022); a blank string counts as "no answer"
+* so the judge never grades against an empty reference.
+*/
+function answerToText(value) {
+	if (typeof value === "string") return value.trim().length > 0 ? value : void 0;
+	if (typeof value === "number" && Number.isFinite(value)) return String(value);
+}
 function asArray(value) {
 	return Array.isArray(value) ? value : [];
 }

package/dist/loaders/locomo.js.map

@@ -1 +1 @@
-{"version":3,"file":"locomo.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","indices: number[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]","role: 'user' | 'assistant'"],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":["/**\n * Real LOCOMO dataset loader (arXiv:2402.17753 -\n * https://snap-research.github.io/locomo/). LOCOMO ships as a JSON\n * **array** of samples; each sample bundles one multi-session\n * conversation with many QA pairs over it:\n *\n * ```jsonc\n * {\n *   \"sample_id\": \"conv-26\",\n *   \"conversation\": {\n *     \"speaker_a\": \"Caroline\",\n *     \"speaker_b\": \"Melanie\",\n *     \"session_1_date_time\": \"1:56 pm on 8 May, 2023\",\n *     \"session_1\": [ { \"speaker\": \"Caroline\", \"dia_id\": \"D1:1\", \"text\": \"...\" }, ... ],\n *     \"session_2_date_time\": \"...\",\n *     \"session_2\": [ ... ]\n *   },\n *   \"qa\": [\n *     { \"question\": \"...\", \"answer\": \"...\", \"evidence\": [\"D1:1\"], \"category\": 1 },\n *     { \"question\": \"...\", \"category\": 5, \"adversarial_answer\": \"Not mentioned ...\" }\n *   ]\n * }\n * ```\n *\n * Each QA pair becomes one {@link MemoryEvalInput} case that shares the\n * sample's sessions. LOCOMO categories map to abilities as:\n * `1`→multi-session (multi-hop), `2`→temporal, `5`→abstention\n * (adversarial), everything else (`3` open-domain, `4` single-hop)\n * →info-extraction.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLocomoOptions {\n  /** Local path to the LOCOMO JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LOCOMO JSON file and return a fully-materialised\n * {@link Dataset} - one case per QA pair, scored against the reference\n * answer string (LLM-judge \"J\").\n *\n * @stable\n */\nexport async function loadLocomoDataset(\n  options: LoadLocomoOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLocomo(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'locomo',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLocomo(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LOCOMO dataset must be a JSON array of samples.');\n  }\n  const samples = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < samples.length; i++) {\n    const sample = samples[i];\n    if (sample === null || typeof sample !== 'object') {\n      throw new Error(`[graphorin/evals] LOCOMO sample ${i} is not an object.`);\n    }\n    appendSampleCases(sample as Record<string, unknown>, i, out);\n  }\n  return out;\n}\n\nfunction appendSampleCases(\n  sample: Record<string, unknown>,\n  index: number,\n  out: Case<MemoryEvalInput, string>[],\n): void {\n  const sampleId = asString(sample.sample_id) ?? `locomo-${index}`;\n  const conversation = isRecord(sample.conversation) ? sample.conversation : {};\n  const sessions = extractSessions(conversation);\n  const qa = asArray(sample.qa);\n  for (let q = 0; q < qa.length; q++) {\n    const item = qa[q];\n    if (!isRecord(item)) continue;\n    const question = asString(item.question);\n    if (question === undefined) continue;\n    const category = typeof item.category === 'number' ? item.category : undefined;\n    const ability = mapCategory(category);\n    const answer = asString(item.answer) ?? asString(item.adversarial_answer) ?? '';\n    const evidence = asArray(item.evidence).filter((e): e is string => typeof e === 'string');\n    out.push({\n      id: `${sampleId}-q${q}`,\n      input: { haystackSessions: sessions, question, ability },\n      expected: answer,\n      metadata: {\n        datasetName: 'locomo',\n        sampleId,\n        ability,\n        ...(category !== undefined ? { category } : {}),\n        evidence,\n      },\n    });\n  }\n}\n\nfunction extractSessions(conversation: Record<string, unknown>): MemoryEvalSession[] {\n  const speakerA = asString(conversation.speaker_a);\n  const indices: number[] = [];\n  for (const key of Object.keys(conversation)) {\n    const m = /^session_(\\d+)$/.exec(key);\n    if (m?.[1] !== undefined) indices.push(Number.parseInt(m[1], 10));\n  }\n  indices.sort((a, b) => a - b);\n  const sessions: MemoryEvalSession[] = [];\n  for (const n of indices) {\n    const raw = asArray(conversation[`session_${n}`]);\n    const date = asString(conversation[`session_${n}_date_time`]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of raw) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.text) ?? asString(t.content);\n      if (content === undefined) continue;\n      const speaker = asString(t.speaker);\n      const role: 'user' | 'assistant' =\n        speakerA !== undefined && speaker !== undefined && speaker !== speakerA\n          ? 'assistant'\n          : 'user';\n      turns.push({ role, content, ...(date !== undefined ? { timestamp: date } : {}) });\n    }\n    sessions.push({ id: `session_${n}`, turns });\n  }\n  return sessions;\n}\n\n/** Maps a LOCOMO numeric `category` onto a {@link MemoryEvalAbility}. */\nfunction mapCategory(category: number | undefined): MemoryEvalAbility {\n  switch (category) {\n    case 1:\n      return 'multi-session';\n    case 2:\n      return 'temporal';\n    case 5:\n      return 'abstention';\n    default:\n      return 'info-extraction';\n  }\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA6DA,eAAsB,kBACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,YADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CAClB;EAG7B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,YAAY,MAA4D;CACtF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,oEAAoE;CAEtF,MAAM,UAAU;CAChB,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,QAAQ,QAAQ,KAAK;EACvC,MAAM,SAAS,QAAQ;AACvB,MAAI,WAAW,QAAQ,OAAO,WAAW,SACvC,OAAM,IAAI,MAAM,mCAAmC,EAAE,oBAAoB;AAE3E,oBAAkB,QAAmC,GAAG,IAAI;;AAE9D,QAAO;;AAGT,SAAS,kBACP,QACA,OACA,KACM;CACN,MAAM,WAAW,SAAS,OAAO,UAAU,IAAI,UAAU;CAEzD,MAAM,WAAW,gBADI,SAAS,OAAO,aAAa,GAAG,OAAO,eAAe,EAAE,CAC/B;CAC9C,MAAM,KAAK,QAAQ,OAAO,GAAG;AAC7B,MAAK,IAAI,IAAI,GAAG,IAAI,GAAG,QAAQ,KAAK;EAClC,MAAM,OAAO,GAAG;AAChB,MAAI,CAAC,SAAS,KAAK,CAAE;EACrB,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OAAW;EAC5B,MAAM,WAAW,OAAO,KAAK,aAAa,WAAW,KAAK,WAAW;EACrE,MAAM,UAAU,YAAY,SAAS;EACrC,MAAM,SAAS,SAAS,KAAK,OAAO,IAAI,SAAS,KAAK,mBAAmB,IAAI;EAC7E,MAAM,WAAW,QAAQ,KAAK,SAAS,CAAC,QAAQ,MAAmB,OAAO,MAAM,SAAS;AACzF,MAAI,KAAK;GACP,IAAI,GAAG,SAAS,IAAI;GACpB,OAAO;IAAE,kBAAkB;IAAU;IAAU;IAAS;GACxD,UAAU;GACV,UAAU;IACR,aAAa;IACb;IACA;IACA,GAAI,aAAa,SAAY,EAAE,UAAU,GAAG,EAAE;IAC9C;IACD;GACF,CAAC;;;AAIN,SAAS,gBAAgB,cAA4D;CACnF,MAAM,WAAW,SAAS,aAAa,UAAU;CACjD,MAAMC,UAAoB,EAAE;AAC5B,MAAK,MAAM,OAAO,OAAO,KAAK,aAAa,EAAE;EAC3C,MAAM,IAAI,kBAAkB,KAAK,IAAI;AACrC,MAAI,IAAI,OAAO,OAAW,SAAQ,KAAK,OAAO,SAAS,EAAE,IAAI,GAAG,CAAC;;AAEnE,SAAQ,MAAM,GAAG,MAAM,IAAI,EAAE;CAC7B,MAAMC,WAAgC,EAAE;AACxC,MAAK,MAAM,KAAK,SAAS;EACvB,MAAM,MAAM,QAAQ,aAAa,WAAW,KAAK;EACjD,MAAM,OAAO,SAAS,aAAa,WAAW,EAAE,aAAa;EAC7D,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,KAAK;AACnB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ;AACvD,OAAI,YAAY,OAAW;GAC3B,MAAM,UAAU,SAAS,EAAE,QAAQ;GACnC,MAAMC,OACJ,aAAa,UAAa,YAAY,UAAa,YAAY,WAC3D,cACA;AACN,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAAG,CAAC;;AAEnF,WAAS,KAAK;GAAE,IAAI,WAAW;GAAK;GAAO,CAAC;;AAE9C,QAAO;;;AAIT,SAAS,YAAY,UAAiD;AACpE,SAAQ,UAAR;EACE,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,QACE,QAAO;;;AAIb,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}
+{"version":3,"file":"locomo.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","indices: number[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]","role: 'user' | 'assistant'"],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":["/**\n * Real LOCOMO dataset loader (arXiv:2402.17753 -\n * https://snap-research.github.io/locomo/). LOCOMO ships as a JSON\n * **array** of samples; each sample bundles one multi-session\n * conversation with many QA pairs over it:\n *\n * ```jsonc\n * {\n *   \"sample_id\": \"conv-26\",\n *   \"conversation\": {\n *     \"speaker_a\": \"Caroline\",\n *     \"speaker_b\": \"Melanie\",\n *     \"session_1_date_time\": \"1:56 pm on 8 May, 2023\",\n *     \"session_1\": [ { \"speaker\": \"Caroline\", \"dia_id\": \"D1:1\", \"text\": \"...\" }, ... ],\n *     \"session_2_date_time\": \"...\",\n *     \"session_2\": [ ... ]\n *   },\n *   \"qa\": [\n *     { \"question\": \"...\", \"answer\": \"...\", \"evidence\": [\"D1:1\"], \"category\": 1 },\n *     { \"question\": \"...\", \"category\": 5, \"adversarial_answer\": \"Not mentioned ...\" }\n *   ]\n * }\n * ```\n *\n * Each QA pair becomes one {@link MemoryEvalInput} case that shares the\n * sample's sessions. LOCOMO categories map to abilities as:\n * `1`→multi-session (multi-hop), `2`→temporal, `5`→abstention\n * (adversarial), everything else (`3` open-domain, `4` single-hop)\n * →info-extraction.\n *\n * Answer handling (W-022): LOCOMO answers may be numbers (e.g. `2022`)\n * - they are stringified, never silently coerced to `''`. A QA pair\n * with NO answer at all (neither `answer` nor `adversarial_answer`) is\n * SKIPPED rather than emitted with an empty `expected`: the LLM judge\n * grading against an empty reference produces meaningless verdicts, so\n * a missing reference must reduce the case count, not poison scores.\n * Turns carry the dataset-native `speaker` name alongside the\n * two-role mapping (~99 percent of LOCOMO questions reference speakers\n * by name).\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLocomoOptions {\n  /** Local path to the LOCOMO JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LOCOMO JSON file and return a fully-materialised\n * {@link Dataset} - one case per QA pair, scored against the reference\n * answer string (LLM-judge \"J\").\n *\n * @stable\n */\nexport async function loadLocomoDataset(\n  options: LoadLocomoOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLocomo(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'locomo',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLocomo(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LOCOMO dataset must be a JSON array of samples.');\n  }\n  const samples = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < samples.length; i++) {\n    const sample = samples[i];\n    if (sample === null || typeof sample !== 'object') {\n      throw new Error(`[graphorin/evals] LOCOMO sample ${i} is not an object.`);\n    }\n    appendSampleCases(sample as Record<string, unknown>, i, out);\n  }\n  return out;\n}\n\nfunction appendSampleCases(\n  sample: Record<string, unknown>,\n  index: number,\n  out: Case<MemoryEvalInput, string>[],\n): void {\n  const sampleId = asString(sample.sample_id) ?? `locomo-${index}`;\n  const conversation = isRecord(sample.conversation) ? sample.conversation : {};\n  const sessions = extractSessions(conversation);\n  const qa = asArray(sample.qa);\n  for (let q = 0; q < qa.length; q++) {\n    const item = qa[q];\n    if (!isRecord(item)) continue;\n    const question = asString(item.question);\n    if (question === undefined) continue;\n    const category = typeof item.category === 'number' ? item.category : undefined;\n    const ability = mapCategory(category);\n    const answer = answerToText(item.answer) ?? answerToText(item.adversarial_answer);\n    // No reference answer at all: skip the pair (see module docs, W-022).\n    if (answer === undefined) continue;\n    const evidence = asArray(item.evidence).filter((e): e is string => typeof e === 'string');\n    out.push({\n      id: `${sampleId}-q${q}`,\n      input: { haystackSessions: sessions, question, ability },\n      expected: answer,\n      metadata: {\n        datasetName: 'locomo',\n        sampleId,\n        ability,\n        ...(category !== undefined ? { category } : {}),\n        evidence,\n      },\n    });\n  }\n}\n\nfunction extractSessions(conversation: Record<string, unknown>): MemoryEvalSession[] {\n  const speakerA = asString(conversation.speaker_a);\n  const indices: number[] = [];\n  for (const key of Object.keys(conversation)) {\n    const m = /^session_(\\d+)$/.exec(key);\n    if (m?.[1] !== undefined) indices.push(Number.parseInt(m[1], 10));\n  }\n  indices.sort((a, b) => a - b);\n  const sessions: MemoryEvalSession[] = [];\n  for (const n of indices) {\n    const raw = asArray(conversation[`session_${n}`]);\n    const date = asString(conversation[`session_${n}_date_time`]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of raw) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.text) ?? asString(t.content);\n      if (content === undefined) continue;\n      const speaker = asString(t.speaker);\n      const role: 'user' | 'assistant' =\n        speakerA !== undefined && speaker !== undefined && speaker !== speakerA\n          ? 'assistant'\n          : 'user';\n      turns.push({\n        role,\n        content,\n        ...(date !== undefined ? { timestamp: date } : {}),\n        // W-022: keep the dataset-native name; the role mapping stays\n        // for cross-dataset machinery compatibility.\n        ...(speaker !== undefined ? { speaker } : {}),\n      });\n    }\n    sessions.push({ id: `session_${n}`, turns });\n  }\n  return sessions;\n}\n\n/** Maps a LOCOMO numeric `category` onto a {@link MemoryEvalAbility}. */\nfunction mapCategory(category: number | undefined): MemoryEvalAbility {\n  switch (category) {\n    case 1:\n      return 'multi-session';\n    case 2:\n      return 'temporal';\n    case 5:\n      return 'abstention';\n    default:\n      return 'info-extraction';\n  }\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\n/**\n * LOCOMO reference answers may be strings OR bare numbers (6 of the\n * 1986 QA pairs ship e.g. `2022`). Stringify finite numbers instead of\n * dropping them to `''` (W-022); a blank string counts as \"no answer\"\n * so the judge never grades against an empty reference.\n */\nfunction answerToText(value: unknown): string | undefined {\n  if (typeof value === 'string') return value.trim().length > 0 ? value : undefined;\n  if (typeof value === 'number' && Number.isFinite(value)) return String(value);\n  return undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAuEA,eAAsB,kBACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,YADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CAClB;EAG7B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,YAAY,MAA4D;CACtF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,oEAAoE;CAEtF,MAAM,UAAU;CAChB,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,QAAQ,QAAQ,KAAK;EACvC,MAAM,SAAS,QAAQ;AACvB,MAAI,WAAW,QAAQ,OAAO,WAAW,SACvC,OAAM,IAAI,MAAM,mCAAmC,EAAE,oBAAoB;AAE3E,oBAAkB,QAAmC,GAAG,IAAI;;AAE9D,QAAO;;AAGT,SAAS,kBACP,QACA,OACA,KACM;CACN,MAAM,WAAW,SAAS,OAAO,UAAU,IAAI,UAAU;CAEzD,MAAM,WAAW,gBADI,SAAS,OAAO,aAAa,GAAG,OAAO,eAAe,EAAE,CAC/B;CAC9C,MAAM,KAAK,QAAQ,OAAO,GAAG;AAC7B,MAAK,IAAI,IAAI,GAAG,IAAI,GAAG,QAAQ,KAAK;EAClC,MAAM,OAAO,GAAG;AAChB,MAAI,CAAC,SAAS,KAAK,CAAE;EACrB,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OAAW;EAC5B,MAAM,WAAW,OAAO,KAAK,aAAa,WAAW,KAAK,WAAW;EACrE,MAAM,UAAU,YAAY,SAAS;EACrC,MAAM,SAAS,aAAa,KAAK,OAAO,IAAI,aAAa,KAAK,mBAAmB;AAEjF,MAAI,WAAW,OAAW;EAC1B,MAAM,WAAW,QAAQ,KAAK,SAAS,CAAC,QAAQ,MAAmB,OAAO,MAAM,SAAS;AACzF,MAAI,KAAK;GACP,IAAI,GAAG,SAAS,IAAI;GACpB,OAAO;IAAE,kBAAkB;IAAU;IAAU;IAAS;GACxD,UAAU;GACV,UAAU;IACR,aAAa;IACb;IACA;IACA,GAAI,aAAa,SAAY,EAAE,UAAU,GAAG,EAAE;IAC9C;IACD;GACF,CAAC;;;AAIN,SAAS,gBAAgB,cAA4D;CACnF,MAAM,WAAW,SAAS,aAAa,UAAU;CACjD,MAAMC,UAAoB,EAAE;AAC5B,MAAK,MAAM,OAAO,OAAO,KAAK,aAAa,EAAE;EAC3C,MAAM,IAAI,kBAAkB,KAAK,IAAI;AACrC,MAAI,IAAI,OAAO,OAAW,SAAQ,KAAK,OAAO,SAAS,EAAE,IAAI,GAAG,CAAC;;AAEnE,SAAQ,MAAM,GAAG,MAAM,IAAI,EAAE;CAC7B,MAAMC,WAAgC,EAAE;AACxC,MAAK,MAAM,KAAK,SAAS;EACvB,MAAM,MAAM,QAAQ,aAAa,WAAW,KAAK;EACjD,MAAM,OAAO,SAAS,aAAa,WAAW,EAAE,aAAa;EAC7D,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,KAAK;AACnB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ;AACvD,OAAI,YAAY,OAAW;GAC3B,MAAM,UAAU,SAAS,EAAE,QAAQ;GACnC,MAAMC,OACJ,aAAa,UAAa,YAAY,UAAa,YAAY,WAC3D,cACA;AACN,SAAM,KAAK;IACT;IACA;IACA,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAGjD,GAAI,YAAY,SAAY,EAAE,SAAS,GAAG,EAAE;IAC7C,CAAC;;AAEJ,WAAS,KAAK;GAAE,IAAI,WAAW;GAAK;GAAO,CAAC;;AAE9C,QAAO;;;AAIT,SAAS,YAAY,UAAiD;AACpE,SAAQ,UAAR;EACE,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,QACE,QAAO;;;AAIb,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;;;;;;;AAS7C,SAAS,aAAa,OAAoC;AACxD,KAAI,OAAO,UAAU,SAAU,QAAO,MAAM,MAAM,CAAC,SAAS,IAAI,QAAQ;AACxE,KAAI,OAAO,UAAU,YAAY,OAAO,SAAS,MAAM,CAAE,QAAO,OAAO,MAAM;;AAI/E,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}

package/dist/loaders/memory-eval.d.ts

@@ -25,6 +25,14 @@ interface MemoryEvalTurn {
   readonly content: string;
   /** Dataset-native (often ISO-8601) timestamp, when the dataset provides one. */
   readonly timestamp?: string;
+  /**
+   * Dataset-native speaker NAME (e.g. LOCOMO's `"Melanie"`), when the
+   * dataset provides one (W-022). Distinct from `role`: two-speaker
+   * datasets map onto user/assistant for machinery compatibility, but
+   * most LOCOMO questions reference the speakers by name, so the
+   * system under test must see the names in the ingested text.
+   */
+  readonly speaker?: string;
 }
 /**
  * One prior session in the haystack the memory system must ingest

package/dist/loaders/memory-eval.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"memory-eval.d.ts","names":[],"sources":["../../src/loaders/memory-eval.ts"],"sourcesContent":[],"mappings":";;AAsBA;AAaA;AAYA;AAYA;;;;;;;;;;;;;;;;;UArCiB,cAAA;;;;;;;;;;;;UAaA,iBAAA;;kBAEC,cAAc;;;;;;;;;KAUpB,iBAAA;;;;;;UAYK,eAAA;;6BAEY,cAAc;;;;;;;;;qBAStB"}
+{"version":3,"file":"memory-eval.d.ts","names":[],"sources":["../../src/loaders/memory-eval.ts"],"sourcesContent":[],"mappings":";;AAsBA;AAqBA;AAYA;AAYA;;;;;;;;;;;;;;;;;UA7CiB,cAAA;;;;;;;;;;;;;;;;;;;;UAqBA,iBAAA;;kBAEC,cAAc;;;;;;;;;KAUpB,iBAAA;;;;;;UAYK,eAAA;;6BAEY,cAAc;;;;;;;;;qBAStB"}

package/dist/package.js

@@ -0,0 +1,6 @@
+//#region package.json
+var version = "0.7.0";
+
+//#endregion
+export { version };
+//# sourceMappingURL=package.js.map

package/dist/package.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"package.js","names":[],"sources":["../package.json"],"sourcesContent":["{\n  \"name\": \"@graphorin/evals\",\n  \"version\": \"0.7.0\",\n  \"description\": \"Eval framework for the Graphorin framework. Ships scorer libraries (`code/` exact-match, regex, JSON path, predicates; `llm/` LLM-judge; `prebuilt/` toxicity / factuality / helpfulness), dataset loaders (JSONL, CSV, from-traces, generic iterable), reporters (terminal, markdown, JSON, JUnit XML), a parallel runner with `concurrency` config, and regression detection that compares the current run against a stored baseline. Builds on the eval primitives shipped from `@graphorin/observability` (RB-17 / DEC-152 - full orchestrator lives here, post-MVP, decoupled from the core observability bundle so consumers do not pay the dataset / reporter cost when only the inline runner is needed). Created and maintained by Oleksiy Stepurenko.\",\n  \"license\": \"MIT\",\n  \"author\": \"Oleksiy Stepurenko\",\n  \"homepage\": \"https://github.com/o-stepper/graphorin/tree/main/packages/evals\",\n  \"repository\": {\n    \"type\": \"git\",\n    \"url\": \"git+https://github.com/o-stepper/graphorin.git\",\n    \"directory\": \"packages/evals\"\n  },\n  \"bugs\": {\n    \"url\": \"https://github.com/o-stepper/graphorin/issues\"\n  },\n  \"keywords\": [\n    \"graphorin\",\n    \"ai\",\n    \"agents\",\n    \"framework\",\n    \"evals\",\n    \"evaluation\",\n    \"scorers\",\n    \"llm-judge\",\n    \"datasets\",\n    \"reporters\",\n    \"regression-detection\"\n  ],\n  \"type\": \"module\",\n  \"engines\": {\n    \"node\": \">=22.12.0\"\n  },\n  \"main\": \"./dist/index.js\",\n  \"module\": \"./dist/index.js\",\n  \"types\": \"./dist/index.d.ts\",\n  \"sideEffects\": false,\n  \"exports\": {\n    \".\": {\n      \"types\": \"./dist/index.d.ts\",\n      \"default\": \"./dist/index.js\"\n    },\n    \"./scorers\": {\n      \"types\": \"./dist/scorers/index.d.ts\",\n      \"default\": \"./dist/scorers/index.js\"\n    },\n    \"./loaders\": {\n      \"types\": \"./dist/loaders/index.d.ts\",\n      \"default\": \"./dist/loaders/index.js\"\n    },\n    \"./reporters\": {\n      \"types\": \"./dist/reporters/index.d.ts\",\n      \"default\": \"./dist/reporters/index.js\"\n    },\n    \"./cli\": {\n      \"types\": \"./dist/cli/index.d.ts\",\n      \"default\": \"./dist/cli/index.js\"\n    },\n    \"./package.json\": \"./package.json\"\n  },\n  \"files\": [\n    \"dist\",\n    \"src\",\n    \"README.md\",\n    \"CHANGELOG.md\",\n    \"LICENSE\"\n  ],\n  \"scripts\": {\n    \"build\": \"tsdown\",\n    \"typecheck\": \"tsc --noEmit && tsc -p tsconfig.tests.json\",\n    \"test\": \"vitest run\",\n    \"lint\": \"biome check .\",\n    \"clean\": \"rimraf dist .turbo *.tsbuildinfo\"\n  },\n  \"dependencies\": {\n    \"@graphorin/core\": \"workspace:*\",\n    \"@graphorin/observability\": \"workspace:*\"\n  },\n  \"publishConfig\": {\n    \"access\": \"public\",\n    \"provenance\": true\n  }\n}\n"],"mappings":";cAEa"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@graphorin/evals",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Eval framework for the Graphorin framework. Ships scorer libraries (`code/` exact-match, regex, JSON path, predicates; `llm/` LLM-judge; `prebuilt/` toxicity / factuality / helpfulness), dataset loaders (JSONL, CSV, from-traces, generic iterable), reporters (terminal, markdown, JSON, JUnit XML), a parallel runner with `concurrency` config, and regression detection that compares the current run against a stored baseline. Builds on the eval primitives shipped from `@graphorin/observability` (RB-17 / DEC-152 - full orchestrator lives here, post-MVP, decoupled from the core observability bundle so consumers do not pay the dataset / reporter cost when only the inline runner is needed). Created and maintained by Oleksiy Stepurenko.",
   "license": "MIT",
   "author": "Oleksiy Stepurenko",
@@ -28,7 +28,7 @@
   ],
   "type": "module",
   "engines": {
-    "node": ">=22.0.0"
+    "node": ">=22.12.0"
   },
   "main": "./dist/index.js",
   "module": "./dist/index.js",
@@ -37,35 +37,36 @@
   "exports": {
     ".": {
       "types": "./dist/index.d.ts",
-      "import": "./dist/index.js"
+      "default": "./dist/index.js"
     },
     "./scorers": {
       "types": "./dist/scorers/index.d.ts",
-      "import": "./dist/scorers/index.js"
+      "default": "./dist/scorers/index.js"
     },
     "./loaders": {
       "types": "./dist/loaders/index.d.ts",
-      "import": "./dist/loaders/index.js"
+      "default": "./dist/loaders/index.js"
     },
     "./reporters": {
       "types": "./dist/reporters/index.d.ts",
-      "import": "./dist/reporters/index.js"
+      "default": "./dist/reporters/index.js"
     },
     "./cli": {
       "types": "./dist/cli/index.d.ts",
-      "import": "./dist/cli/index.js"
+      "default": "./dist/cli/index.js"
     },
     "./package.json": "./package.json"
   },
   "files": [
     "dist",
+    "src",
     "README.md",
     "CHANGELOG.md",
     "LICENSE"
   ],
   "dependencies": {
-    "@graphorin/core": "0.6.0",
-    "@graphorin/observability": "0.6.0"
+    "@graphorin/core": "0.7.0",
+    "@graphorin/observability": "0.7.0"
   },
   "publishConfig": {
     "access": "public",

package/src/cli/index.ts

@@ -0,0 +1,124 @@
+/**
+ * CLI integration helpers. Convenience wrappers that combine the
+ * runner + a reporter + an exit-code mapping so consumer scripts can
+ * stay short.
+ *
+ * Typical use from a `package.json` script:
+ *
+ * ```jsonc
+ * {
+ *   "scripts": {
+ *     "eval": "node ./scripts/run-evals.mjs"
+ *   }
+ * }
+ * ```
+ *
+ * Where `run-evals.mjs` looks like:
+ *
+ * ```ts
+ * import { runEvals, exitOnFailures, renderTerminalReport } from '@graphorin/evals';
+ *
+ * const report = await runEvals({...});
+ * console.log(renderTerminalReport(report));
+ * exitOnFailures(report);
+ * ```
+ *
+ * @packageDocumentation
+ */
+
+import { writeFile } from 'node:fs/promises';
+import process from 'node:process';
+
+import type { EvalReport } from '@graphorin/observability/eval';
+
+import { detectRegressions } from '../regression.js';
+import {
+  renderHtmlReport,
+  renderJsonReport,
+  renderJunitReport,
+  renderMarkdownReport,
+  renderTerminalReport,
+} from '../reporters/index.js';
+import type { RegressionOptions, RegressionReport } from '../types.js';
+
+/**
+ * Set `process.exitCode` to `1` when at least one case failed, or
+ * when a regression report contains findings. Uses `exitCode` rather
+ * than `process.exit` so other async tasks finish cleanly.
+ *
+ * @stable
+ */
+export function exitOnFailures<I, O>(
+  report: EvalReport<I, O>,
+  regression?: RegressionReport<I, O>,
+): void {
+  if (report.summary.failed > 0) process.exitCode = 1;
+  if (regression?.hasRegressions) process.exitCode = 1;
+}
+
+/**
+ * Reporter ids accepted by {@link writeReports}.
+ *
+ * @stable
+ */
+export type ReporterFormat = 'terminal' | 'markdown' | 'json' | 'junit' | 'html';
+
+/** @stable */
+export interface WriteReportsOptions<I, O> {
+  readonly report: EvalReport<I, O>;
+  readonly outDir?: string;
+  readonly formats: ReadonlyArray<ReporterFormat>;
+  readonly basename?: string;
+}
+
+/** @stable */
+export interface WrittenReport {
+  readonly format: ReporterFormat;
+  readonly path: string;
+  readonly bytes: number;
+}
+
+/**
+ * Render the report in every requested format and write each one to a
+ * file. Returns the manifest of written files.
+ *
+ * @stable
+ */
+export async function writeReports<I, O>(
+  options: WriteReportsOptions<I, O>,
+): Promise<ReadonlyArray<WrittenReport>> {
+  const out: WrittenReport[] = [];
+  const dir = options.outDir ?? '.';
+  const basename = options.basename ?? 'eval-report';
+  for (const format of options.formats) {
+    const { extension, body } = renderForFormat(format, options.report);
+    const path = `${dir}/${basename}.${extension}`;
+    await writeFile(path, body, 'utf8');
+    out.push({ format, path, bytes: Buffer.byteLength(body, 'utf8') });
+  }
+  return out;
+}
+
+function renderForFormat<I, O>(
+  format: ReporterFormat,
+  report: EvalReport<I, O>,
+): { extension: string; body: string } {
+  switch (format) {
+    case 'terminal':
+      return { extension: 'txt', body: renderTerminalReport(report) };
+    case 'markdown':
+      return { extension: 'md', body: renderMarkdownReport(report) };
+    case 'json':
+      return { extension: 'json', body: renderJsonReport(report, { pretty: true }) };
+    case 'junit':
+      return { extension: 'xml', body: renderJunitReport(report) };
+    case 'html':
+      return { extension: 'html', body: renderHtmlReport(report) };
+    default: {
+      const exhaustive: never = format;
+      throw new Error(`unknown reporter format '${exhaustive as string}'`);
+    }
+  }
+}
+
+export { detectRegressions, type RegressionOptions };

package/src/index.ts

@@ -0,0 +1,136 @@
+/**
+ * @graphorin/evals - eval framework for the Graphorin framework.
+ *
+ * Builds on the eval primitives shipped from `@graphorin/observability`
+ * (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled
+ * from the core observability bundle so consumers do not pay the
+ * dataset / reporter cost when only the inline runner is needed.
+ *
+ * ```ts
+ * import {
+ *   runEvals,
+ *   loadJsonlDataset,
+ *   exactMatch,
+ *   renderTerminalReport,
+ *   exitOnFailures,
+ * } from '@graphorin/evals';
+ *
+ * const dataset = await loadJsonlDataset('./fixtures/golden.jsonl');
+ * const report = await runEvals({
+ *   agent,
+ *   dataset,
+ *   scorers: [exactMatch()],
+ *   concurrency: 4,
+ * });
+ * console.log(renderTerminalReport(report));
+ * exitOnFailures(report);
+ * ```
+ *
+ * @packageDocumentation
+ */
+
+/** Canonical version constant, derived from `package.json` at build time. */
+import pkg from '../package.json' with { type: 'json' };
+
+export const VERSION: string = pkg.version;
+
+export {
+  detectRegressions,
+  exitOnFailures,
+  type ReporterFormat,
+  type WriteReportsOptions,
+  type WrittenReport,
+  writeReports,
+} from './cli/index.js';
+export type {
+  FromTracesOptions,
+  LoadDmrOptions,
+  LoadLocomoOptions,
+  LoadLongMemEvalOptions,
+  MemoryEvalAbility,
+  MemoryEvalInput,
+  MemoryEvalSession,
+  MemoryEvalTurn,
+  TraceEvent,
+} from './loaders/index.js';
+export {
+  fromIterable,
+  type LoadCsvOptions,
+  type LoadJsonlOptions,
+  loadCsvDataset,
+  loadDatasetFromTraces,
+  loadDmrDataset,
+  loadJsonlDataset,
+  loadLocomoDataset,
+  loadLongMemEvalDataset,
+  parseCsv,
+  parseDmr,
+  parseJsonl,
+  parseLocomo,
+  parseLongMemEval,
+} from './loaders/index.js';
+export { detectRegressions as detectRegressionsFromReports } from './regression.js';
+export {
+  renderHtmlReport,
+  renderJsonReport,
+  renderJunitReport,
+  renderMarkdownReport,
+  renderTerminalReport,
+} from './reporters/index.js';
+export { runEvals } from './runner.js';
+export {
+  type ArgumentValidityOptions,
+  argumentValidity,
+  type CorrectToolSelectedOptions,
+  correctToolSelected,
+  type ExactMatchOptions,
+  exactMatch,
+  type FinalStateCorrectOptions,
+  factualityScorer,
+  fenceForJudge,
+  finalStateCorrect,
+  helpfulnessScorer,
+  type JsonPathOptions,
+  jsonPath,
+  type LlmJudgeOptions,
+  llmJudge,
+  type PrebuiltScorerOptions,
+  type PredicateOptions,
+  parseScore,
+  predicate,
+  type RecoveryAfterErrorOptions,
+  type RedundantCallDetectionOptions,
+  type RegexMatchOptions,
+  recoveryAfterError,
+  redundantCallDetection,
+  regexMatch,
+  scoreContract,
+  type Trajectory,
+  type TrajectoryToolCall,
+  toxicityScorer,
+} from './scorers/index.js';
+export {
+  mean,
+  type PairedSignificance,
+  pairedPassSignificance,
+  passByBaseCase,
+  passHatK,
+  sampleStddev,
+  stripIterationSuffix,
+  wilsonInterval,
+} from './stats.js';
+export type {
+  AgentLike,
+  Case,
+  Dataset,
+  EvalCaseResult,
+  EvalReport,
+  ProgressEvent,
+  RegressionFinding,
+  RegressionOptions,
+  RegressionReport,
+  RunEvalOptions,
+  RunOptions,
+  ScoreResult,
+  Scorer,
+} from './types.js';