@graphorin/evals 0.5.0 → 0.6.1

package/CHANGELOG.md

@@ -1,5 +1,39 @@
 # @graphorin/evals
 
+## 0.6.1
+
+### Patch Changes
+
+- [#142](https://github.com/o-stepper/graphorin/pull/142) [`436d6ca`](https://github.com/o-stepper/graphorin/commit/436d6ca5ebbd16df094e915682d3915c279a8430) Thanks [@o-stepper](https://github.com/o-stepper)! - Version constants and version-bearing strings now derive from each package's manifest at build time (`VERSION = pkg.version`; writer ids, client/server info, OTLP framework attributes, build-info metrics interpolate it). No behavioral change at the current version: the rendered strings are byte-identical. A release bump no longer edits source; the new `check-version-consistency` gate fails any reintroduced hardcoded framework version.
+
+- Updated dependencies [[`826ee59`](https://github.com/o-stepper/graphorin/commit/826ee5933ad38693b0dd2f20a110abfecba7d23d), [`436d6ca`](https://github.com/o-stepper/graphorin/commit/436d6ca5ebbd16df094e915682d3915c279a8430)]:
+  - @graphorin/observability@0.6.1
+  - @graphorin/core@0.6.1
+
+## 0.6.0
+
+### Minor Changes
+
+- [#138](https://github.com/o-stepper/graphorin/pull/138) [`6293a25`](https://github.com/o-stepper/graphorin/commit/6293a2531c5c9265294da22fb365a08f2bdf445a) Thanks [@o-stepper](https://github.com/o-stepper)! - OTel GenAI alignment + honest eval statistics (audit 2026-07-04 Wave E, cluster E8).
+
+  - Span names follow the GenAI semconv `{operation} {target}` shape when attributes carry the target (`chat <model>`, `execute_tool <tool>`, `invoke_agent <agent>`); the operation-mapping table and `GenAIOperationName` gain `invoke_agent` to match what the runtime actually emits; `agent.step` spans carry `gen_ai.agent.name`; pricing's `listMissingModels` reads the current `gen_ai.provider.name` attribute first (deprecated `gen_ai.system` kept as fallback).
+  - New `@graphorin/evals` stats module: `mean`/`sampleStddev`, Wilson 95% interval, `passHatK` over `-iter-N` outcomes, and McNemar paired significance. `runEvals` summaries always carry `passRateCi` and (under `iterations > 1`) `passHatK`; `detectRegressions` annotates pass-rate-drop findings with the paired regressed/improved counts + p-value, and opt-in `requireSignificance` vetoes drops the paired test cannot distinguish from noise - the fixed tolerance alone was sample-size blind.
+
+### Patch Changes
+
+- Updated dependencies [[`32f20c1`](https://github.com/o-stepper/graphorin/commit/32f20c110f184f8cef7eec85bf39f5f07c886cb6), [`4f850d9`](https://github.com/o-stepper/graphorin/commit/4f850d9bc0a05d6256c59c5117b010336fcb41d3), [`4f850d9`](https://github.com/o-stepper/graphorin/commit/4f850d9bc0a05d6256c59c5117b010336fcb41d3), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`6293a25`](https://github.com/o-stepper/graphorin/commit/6293a2531c5c9265294da22fb365a08f2bdf445a)]:
+  - @graphorin/core@0.6.0
+  - @graphorin/observability@0.6.0
+
+## 0.5.0
+
+First version published to the npm registry (with Sigstore build
+provenance). The 0.2.0, 0.3.0, and 0.4.0 versions were internal lockstep
+milestones and were never published. All `@graphorin/*` packages release
+lockstep at the same version; the full release notes for 0.2.0-0.5.0 live
+in the repository-level
+[CHANGELOG](https://github.com/o-stepper/graphorin/blob/main/CHANGELOG.md).
+
 ## 0.1.0
 
 ### Minor Changes

package/README.md

@@ -7,14 +7,14 @@
 > bounded concurrency, and regression detection that compares the
 > current run against a stored baseline.
 >
-> Project Graphorin · v0.5.0 · MIT License · © 2026 Oleksiy Stepurenko ·
+> Project Graphorin · v0.6.1 · MIT License · © 2026 Oleksiy Stepurenko ·
 > <https://github.com/o-stepper/graphorin>
 
 ---
 
 ## Status
 
-- **Published:** v0.5.0 (optional sub-pack; the full orchestrator is
+- **Published:** v0.6.1 (optional sub-pack; the full orchestrator is
   decoupled from `@graphorin/observability` per RB-17 / DEC-152).
 
 ---
@@ -62,7 +62,7 @@ exitOnFailures(report);
 | `code/`     | `exactMatch`, `regexMatch`, `jsonPath`, `predicate` | Pure-code grading. No provider call.|
 | `llm/`      | `llmJudge`                                          | LLM-as-judge. Default `gpt-4o-mini`-class judge with `temperature: 0`. |
 | `prebuilt/` | `toxicityScorer`, `factualityScorer`, `helpfulnessScorer` | Wrap `llmJudge` with a project-tested rubric. |
-| `trajectory/` | `correctToolSelected`, `argumentValidity`, `redundantCallDetection`, `recoveryAfterError`, `finalStateCorrect` | Pure-code, offline scorers over a `Trajectory` (the tool calls a harness made). Measure harness reliability — tool selection, argument validity, redundant work, error recovery, goal state. |
+| `trajectory/` | `correctToolSelected`, `argumentValidity`, `redundantCallDetection`, `recoveryAfterError`, `finalStateCorrect` | Pure-code, offline scorers over a `Trajectory` (the tool calls a harness made). Measure harness reliability - tool selection, argument validity, redundant work, error recovery, goal state. |
 
 ---
 
@@ -103,6 +103,25 @@ const report = await runEvals({
 });
 ```
 
+Every report summary carries a Wilson 95% confidence interval for the
+pass rate (`summary.passRateCi`), and, when `iterations > 1`,
+`summary.passHatK` - the pass^k estimate over the per-case iteration
+outcomes (the probability that all k iterations of a case pass).
+
+---
+
+## Statistics
+
+Sample-size-aware statistics ship as plain exported functions:
+
+```ts
+import { mean, sampleStddev, wilsonInterval, passHatK, pairedPassSignificance } from '@graphorin/evals';
+
+wilsonInterval(18, 20);            // { lo, hi } - 95% CI for 18/20 passes
+passHatK(outcomesByCase);          // pass^k over `-iter-N` outcome groups
+pairedPassSignificance(a, b);      // McNemar paired test between two runs
+```
+
 ---
 
 ## Regression detection
@@ -116,15 +135,22 @@ const regression = detectRegressions(report, baseline, {
   maxPassRateDropPct: 5,
   maxAvgScoreDrop: 0.05,
   maxAvgDurationIncreaseMs: 250,
+  // Opt-in: veto pass-rate-drop findings the McNemar paired test
+  // cannot distinguish from noise at the given alpha (default 0.05).
+  requireSignificance: true,
 });
 if (regression.hasRegressions) {
   for (const f of regression.findings) {
-    console.error(`regression — ${f.kind}: ${f.message}`);
+    console.error(`regression - ${f.kind}: ${f.message}`);
   }
 }
 exitOnFailures(report, regression);
 ```
 
+Pass-rate findings are annotated with the paired regressed / improved
+case counts and the McNemar p-value, so a 3-point drop on 20 cases
+reads differently from the same drop on 2000.
+
 ---
 
 ## Multi-format report writing
@@ -144,7 +170,7 @@ await writeReports({
 
 ## Related decisions
 
-- DEC-152 — Eval split: keep evaluation interfaces in `@graphorin/observability`, ship the full eval framework as `@graphorin/evals`.
+- DEC-152 - Eval split: keep evaluation interfaces in `@graphorin/observability`, ship the full eval framework as `@graphorin/evals`.
 
 ---
 
@@ -154,4 +180,4 @@ MIT © 2026 Oleksiy Stepurenko
 
 ---
 
-**Project Graphorin** · v0.5.0 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>
+**Project Graphorin** · v0.6.1 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>

package/dist/index.d.ts

@@ -27,13 +27,14 @@ import { CorrectToolSelectedOptions, correctToolSelected } from "./scorers/traje
 import { FinalStateCorrectOptions, finalStateCorrect } from "./scorers/trajectory/final-state-correct.js";
 import { RecoveryAfterErrorOptions, recoveryAfterError } from "./scorers/trajectory/recovery-after-error.js";
 import { RedundantCallDetectionOptions, redundantCallDetection } from "./scorers/trajectory/redundant-call-detection.js";
+import { PairedSignificance, mean, pairedPassSignificance, passByBaseCase, passHatK, sampleStddev, stripIterationSuffix, wilsonInterval } from "./stats.js";
 
 //#region src/index.d.ts
 /**
- * @graphorin/evals — eval framework for the Graphorin framework.
+ * @graphorin/evals - eval framework for the Graphorin framework.
  *
  * Builds on the eval primitives shipped from `@graphorin/observability`
- * (RB-17 / DEC-152) — full orchestrator lives here, post-MVP, decoupled
+ * (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled
  * from the core observability bundle so consumers do not pay the
  * dataset / reporter cost when only the inline runner is needed.
  *
@@ -59,8 +60,7 @@ import { RedundantCallDetectionOptions, redundantCallDetection } from "./scorers
  *
  * @packageDocumentation
  */
-/** Canonical version constant. Mirrors the `package.json` version. */
-declare const VERSION = "0.5.0";
+declare const VERSION: string;
 //#endregion
-export { type AgentLike, type ArgumentValidityOptions, type Case, type CorrectToolSelectedOptions, type Dataset, type EvalCaseResult, type EvalReport, type ExactMatchOptions, type FinalStateCorrectOptions, type FromTracesOptions, type JsonPathOptions, type LlmJudgeOptions, type LoadCsvOptions, type LoadDmrOptions, type LoadJsonlOptions, type LoadLocomoOptions, type LoadLongMemEvalOptions, type MemoryEvalAbility, type MemoryEvalInput, type MemoryEvalSession, type MemoryEvalTurn, type PrebuiltScorerOptions, type PredicateOptions, type ProgressEvent, type RecoveryAfterErrorOptions, type RedundantCallDetectionOptions, type RegexMatchOptions, type RegressionFinding, type RegressionOptions, type RegressionReport, type ReporterFormat, type RunEvalOptions, type RunOptions, type ScoreResult, type Scorer, type TraceEvent, type Trajectory, type TrajectoryToolCall, VERSION, type WriteReportsOptions, type WrittenReport, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, scoreContract, toxicityScorer, writeReports };
+export { type AgentLike, type ArgumentValidityOptions, type Case, type CorrectToolSelectedOptions, type Dataset, type EvalCaseResult, type EvalReport, type ExactMatchOptions, type FinalStateCorrectOptions, type FromTracesOptions, type JsonPathOptions, type LlmJudgeOptions, type LoadCsvOptions, type LoadDmrOptions, type LoadJsonlOptions, type LoadLocomoOptions, type LoadLongMemEvalOptions, type MemoryEvalAbility, type MemoryEvalInput, type MemoryEvalSession, type MemoryEvalTurn, type PairedSignificance, type PrebuiltScorerOptions, type PredicateOptions, type ProgressEvent, type RecoveryAfterErrorOptions, type RedundantCallDetectionOptions, type RegexMatchOptions, type RegressionFinding, type RegressionOptions, type RegressionReport, type ReporterFormat, type RunEvalOptions, type RunOptions, type ScoreResult, type Scorer, type TraceEvent, type Trajectory, type TrajectoryToolCall, VERSION, type WriteReportsOptions, type WrittenReport, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, mean, pairedPassSignificance, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, passByBaseCase, passHatK, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, sampleStddev, scoreContract, stripIterationSuffix, toxicityScorer, wilsonInterval, writeReports };
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAgCa,cAAA,OAAA,GAAO,OAAA"}
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;cAkCa"}

package/dist/index.js

@@ -1,3 +1,5 @@
+import { version } from "./package.js";
+import { mean, pairedPassSignificance, passByBaseCase, passHatK, sampleStddev, stripIterationSuffix, wilsonInterval } from "./stats.js";
 import { detectRegressions } from "./regression.js";
 import { renderHtmlReport } from "./reporters/html.js";
 import { renderJsonReport } from "./reporters/json.js";
@@ -27,10 +29,10 @@ import { redundantCallDetection } from "./scorers/trajectory/redundant-call-dete
 
 //#region src/index.ts
 /**
-* @graphorin/evals — eval framework for the Graphorin framework.
+* @graphorin/evals - eval framework for the Graphorin framework.
 *
 * Builds on the eval primitives shipped from `@graphorin/observability`
-* (RB-17 / DEC-152) — full orchestrator lives here, post-MVP, decoupled
+* (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled
 * from the core observability bundle so consumers do not pay the
 * dataset / reporter cost when only the inline runner is needed.
 *
@@ -56,9 +58,9 @@ import { redundantCallDetection } from "./scorers/trajectory/redundant-call-dete
 *
 * @packageDocumentation
 */
-/** Canonical version constant. Mirrors the `package.json` version. */
-const VERSION = "0.5.0";
+/** Canonical version constant, derived from `package.json` at build time. */
+const VERSION = version;
 
 //#endregion
-export { VERSION, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, scoreContract, toxicityScorer, writeReports };
+export { VERSION, argumentValidity, correctToolSelected, detectRegressions, detectRegressions as detectRegressionsFromReports, exactMatch, exitOnFailures, factualityScorer, fenceForJudge, finalStateCorrect, fromIterable, helpfulnessScorer, jsonPath, llmJudge, loadCsvDataset, loadDatasetFromTraces, loadDmrDataset, loadJsonlDataset, loadLocomoDataset, loadLongMemEvalDataset, mean, pairedPassSignificance, parseCsv, parseDmr, parseJsonl, parseLocomo, parseLongMemEval, parseScore, passByBaseCase, passHatK, predicate, recoveryAfterError, redundantCallDetection, regexMatch, renderHtmlReport, renderJsonReport, renderJunitReport, renderMarkdownReport, renderTerminalReport, runEvals, sampleStddev, scoreContract, stripIterationSuffix, toxicityScorer, wilsonInterval, writeReports };
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","names":[],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/evals — eval framework for the Graphorin framework.\n *\n * Builds on the eval primitives shipped from `@graphorin/observability`\n * (RB-17 / DEC-152) — full orchestrator lives here, post-MVP, decoupled\n * from the core observability bundle so consumers do not pay the\n * dataset / reporter cost when only the inline runner is needed.\n *\n * ```ts\n * import {\n *   runEvals,\n *   loadJsonlDataset,\n *   exactMatch,\n *   renderTerminalReport,\n *   exitOnFailures,\n * } from '@graphorin/evals';\n *\n * const dataset = await loadJsonlDataset('./fixtures/golden.jsonl');\n * const report = await runEvals({\n *   agent,\n *   dataset,\n *   scorers: [exactMatch()],\n *   concurrency: 4,\n * });\n * console.log(renderTerminalReport(report));\n * exitOnFailures(report);\n * ```\n *\n * @packageDocumentation\n */\n\n/** Canonical version constant. Mirrors the `package.json` version. */\nexport const VERSION = '0.5.0';\n\nexport {\n  detectRegressions,\n  exitOnFailures,\n  type ReporterFormat,\n  type WriteReportsOptions,\n  type WrittenReport,\n  writeReports,\n} from './cli/index.js';\nexport type {\n  FromTracesOptions,\n  LoadDmrOptions,\n  LoadLocomoOptions,\n  LoadLongMemEvalOptions,\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n  TraceEvent,\n} from './loaders/index.js';\nexport {\n  fromIterable,\n  type LoadCsvOptions,\n  type LoadJsonlOptions,\n  loadCsvDataset,\n  loadDatasetFromTraces,\n  loadDmrDataset,\n  loadJsonlDataset,\n  loadLocomoDataset,\n  loadLongMemEvalDataset,\n  parseCsv,\n  parseDmr,\n  parseJsonl,\n  parseLocomo,\n  parseLongMemEval,\n} from './loaders/index.js';\nexport { detectRegressions as detectRegressionsFromReports } from './regression.js';\nexport {\n  renderHtmlReport,\n  renderJsonReport,\n  renderJunitReport,\n  renderMarkdownReport,\n  renderTerminalReport,\n} from './reporters/index.js';\nexport { runEvals } from './runner.js';\nexport {\n  type ArgumentValidityOptions,\n  argumentValidity,\n  type CorrectToolSelectedOptions,\n  correctToolSelected,\n  type ExactMatchOptions,\n  exactMatch,\n  type FinalStateCorrectOptions,\n  factualityScorer,\n  fenceForJudge,\n  finalStateCorrect,\n  helpfulnessScorer,\n  type JsonPathOptions,\n  jsonPath,\n  type LlmJudgeOptions,\n  llmJudge,\n  type PrebuiltScorerOptions,\n  type PredicateOptions,\n  parseScore,\n  predicate,\n  type RecoveryAfterErrorOptions,\n  type RedundantCallDetectionOptions,\n  type RegexMatchOptions,\n  recoveryAfterError,\n  redundantCallDetection,\n  regexMatch,\n  scoreContract,\n  type Trajectory,\n  type TrajectoryToolCall,\n  toxicityScorer,\n} from './scorers/index.js';\nexport type {\n  AgentLike,\n  Case,\n  Dataset,\n  EvalCaseResult,\n  EvalReport,\n  ProgressEvent,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n  RunEvalOptions,\n  RunOptions,\n  ScoreResult,\n  Scorer,\n} from './types.js';\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAgCA,MAAa,UAAU"}
+{"version":3,"file":"index.js","names":["VERSION: string","pkg.version"],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/evals - eval framework for the Graphorin framework.\n *\n * Builds on the eval primitives shipped from `@graphorin/observability`\n * (RB-17 / DEC-152) - full orchestrator lives here, post-MVP, decoupled\n * from the core observability bundle so consumers do not pay the\n * dataset / reporter cost when only the inline runner is needed.\n *\n * ```ts\n * import {\n *   runEvals,\n *   loadJsonlDataset,\n *   exactMatch,\n *   renderTerminalReport,\n *   exitOnFailures,\n * } from '@graphorin/evals';\n *\n * const dataset = await loadJsonlDataset('./fixtures/golden.jsonl');\n * const report = await runEvals({\n *   agent,\n *   dataset,\n *   scorers: [exactMatch()],\n *   concurrency: 4,\n * });\n * console.log(renderTerminalReport(report));\n * exitOnFailures(report);\n * ```\n *\n * @packageDocumentation\n */\n\n/** Canonical version constant, derived from `package.json` at build time. */\nimport pkg from '../package.json' with { type: 'json' };\n\nexport const VERSION: string = pkg.version;\n\nexport {\n  detectRegressions,\n  exitOnFailures,\n  type ReporterFormat,\n  type WriteReportsOptions,\n  type WrittenReport,\n  writeReports,\n} from './cli/index.js';\nexport type {\n  FromTracesOptions,\n  LoadDmrOptions,\n  LoadLocomoOptions,\n  LoadLongMemEvalOptions,\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n  TraceEvent,\n} from './loaders/index.js';\nexport {\n  fromIterable,\n  type LoadCsvOptions,\n  type LoadJsonlOptions,\n  loadCsvDataset,\n  loadDatasetFromTraces,\n  loadDmrDataset,\n  loadJsonlDataset,\n  loadLocomoDataset,\n  loadLongMemEvalDataset,\n  parseCsv,\n  parseDmr,\n  parseJsonl,\n  parseLocomo,\n  parseLongMemEval,\n} from './loaders/index.js';\nexport { detectRegressions as detectRegressionsFromReports } from './regression.js';\nexport {\n  renderHtmlReport,\n  renderJsonReport,\n  renderJunitReport,\n  renderMarkdownReport,\n  renderTerminalReport,\n} from './reporters/index.js';\nexport { runEvals } from './runner.js';\nexport {\n  type ArgumentValidityOptions,\n  argumentValidity,\n  type CorrectToolSelectedOptions,\n  correctToolSelected,\n  type ExactMatchOptions,\n  exactMatch,\n  type FinalStateCorrectOptions,\n  factualityScorer,\n  fenceForJudge,\n  finalStateCorrect,\n  helpfulnessScorer,\n  type JsonPathOptions,\n  jsonPath,\n  type LlmJudgeOptions,\n  llmJudge,\n  type PrebuiltScorerOptions,\n  type PredicateOptions,\n  parseScore,\n  predicate,\n  type RecoveryAfterErrorOptions,\n  type RedundantCallDetectionOptions,\n  type RegexMatchOptions,\n  recoveryAfterError,\n  redundantCallDetection,\n  regexMatch,\n  scoreContract,\n  type Trajectory,\n  type TrajectoryToolCall,\n  toxicityScorer,\n} from './scorers/index.js';\nexport {\n  mean,\n  type PairedSignificance,\n  pairedPassSignificance,\n  passByBaseCase,\n  passHatK,\n  sampleStddev,\n  stripIterationSuffix,\n  wilsonInterval,\n} from './stats.js';\nexport type {\n  AgentLike,\n  Case,\n  Dataset,\n  EvalCaseResult,\n  EvalReport,\n  ProgressEvent,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n  RunEvalOptions,\n  RunOptions,\n  ScoreResult,\n  Scorer,\n} from './types.js';\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAkCA,MAAaA,UAAkBC"}

package/dist/loaders/dmr.js

@@ -2,7 +2,7 @@ import { readFile } from "node:fs/promises";
 
 //#region src/loaders/dmr.ts
 /**
-* DMR (Deep Memory Retrieval) loader — the MemGPT/Letta continuity set
+* DMR (Deep Memory Retrieval) loader - the MemGPT/Letta continuity set
 * derived from Multi-Session Chat (arXiv:2310.08560). DMR's published
 * shape is less standardised than LongMemEval/LOCOMO, so this loader is
 * deliberately tolerant: it accepts a JSON **array** of items, each

package/dist/loaders/dmr.js.map

@@ -1 +1 @@
-{"version":3,"file":"dmr.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]"],"sources":["../../src/loaders/dmr.ts"],"sourcesContent":["/**\n * DMR (Deep Memory Retrieval) loader — the MemGPT/Letta continuity set\n * derived from Multi-Session Chat (arXiv:2310.08560). DMR's published\n * shape is less standardised than LongMemEval/LOCOMO, so this loader is\n * deliberately tolerant: it accepts a JSON **array** of items, each\n * exposing a `question`, an `answer`, and the prior conversation under\n * either `sessions` or `haystack_sessions`. Sessions may be a bare\n * array of turns or an object wrapping the turns under\n * `turns` / `messages` / `dialog`; a turn may use `role` or `speaker`\n * and `content` or `text`.\n *\n * ```jsonc\n * {\n *   \"id\": \"dmr-1\",\n *   \"question\": \"What is the user's sister's name?\",\n *   \"answer\": \"Anna\",\n *   \"sessions\": [\n *     { \"id\": \"s1\", \"turns\": [ { \"role\": \"user\", \"content\": \"...\" } ] },\n *     [ { \"speaker\": \"assistant\", \"text\": \"...\" } ]\n *   ]\n * }\n * ```\n *\n * Every DMR case is treated as `multi-session` (its defining ability).\n * The exact field names should be reconciled against the downloaded\n * file the first time `scripts/fetch-eval-datasets.mjs` is run.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type { MemoryEvalInput, MemoryEvalSession, MemoryEvalTurn } from './memory-eval.js';\n\n/** @stable */\nexport interface LoadDmrOptions {\n  /** Local path to the DMR JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a DMR JSON file and return a fully-materialised {@link Dataset}\n * of multi-session retrieval cases scored against the reference answer.\n *\n * @stable\n */\nexport async function loadDmrDataset(\n  options: LoadDmrOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseDmr(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'dmr',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseDmr(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] DMR dataset must be a JSON array of items.');\n  }\n  const items = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < items.length; i++) {\n    const item = items[i];\n    if (!isRecord(item)) {\n      throw new Error(`[graphorin/evals] DMR item ${i} is not an object.`);\n    }\n    const question = asString(item.question);\n    if (question === undefined) {\n      throw new Error(`[graphorin/evals] DMR item ${i} is missing a 'question'.`);\n    }\n    const sessions = coerceSessions(item.sessions ?? item.haystack_sessions);\n    out.push({\n      id: asString(item.id) ?? `dmr-${i}`,\n      input: { haystackSessions: sessions, question, ability: 'multi-session' },\n      expected: asString(item.answer) ?? '',\n      metadata: { datasetName: 'dmr', ability: 'multi-session' },\n    });\n  }\n  return out;\n}\n\nfunction coerceSessions(value: unknown): MemoryEvalSession[] {\n  const rawSessions = asArray(value);\n  const sessions: MemoryEvalSession[] = [];\n  for (let s = 0; s < rawSessions.length; s++) {\n    const raw = rawSessions[s];\n    const rawTurns = Array.isArray(raw)\n      ? (raw as ReadonlyArray<unknown>)\n      : isRecord(raw)\n        ? asArray(raw.turns ?? raw.messages ?? raw.dialog)\n        : [];\n    const id = isRecord(raw) ? (asString(raw.id) ?? `session-${s}`) : `session-${s}`;\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of rawTurns) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.content) ?? asString(t.text);\n      if (content === undefined) continue;\n      const speaker = asString(t.role) ?? asString(t.speaker);\n      const role = speaker === 'assistant' ? 'assistant' : 'user';\n      const timestamp = asString(t.timestamp);\n      turns.push({ role, content, ...(timestamp !== undefined ? { timestamp } : {}) });\n    }\n    sessions.push({ id, turns });\n  }\n  return sessions;\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAoDA,eAAsB,eACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,SADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CACrB;EAG1B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,SAAS,MAA4D;CACnF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,+DAA+D;CAEjF,MAAM,QAAQ;CACd,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;EACrC,MAAM,OAAO,MAAM;AACnB,MAAI,CAAC,SAAS,KAAK,CACjB,OAAM,IAAI,MAAM,8BAA8B,EAAE,oBAAoB;EAEtE,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OACf,OAAM,IAAI,MAAM,8BAA8B,EAAE,2BAA2B;EAE7E,MAAM,WAAW,eAAe,KAAK,YAAY,KAAK,kBAAkB;AACxE,MAAI,KAAK;GACP,IAAI,SAAS,KAAK,GAAG,IAAI,OAAO;GAChC,OAAO;IAAE,kBAAkB;IAAU;IAAU,SAAS;IAAiB;GACzE,UAAU,SAAS,KAAK,OAAO,IAAI;GACnC,UAAU;IAAE,aAAa;IAAO,SAAS;IAAiB;GAC3D,CAAC;;AAEJ,QAAO;;AAGT,SAAS,eAAe,OAAqC;CAC3D,MAAM,cAAc,QAAQ,MAAM;CAClC,MAAMC,WAAgC,EAAE;AACxC,MAAK,IAAI,IAAI,GAAG,IAAI,YAAY,QAAQ,KAAK;EAC3C,MAAM,MAAM,YAAY;EACxB,MAAM,WAAW,MAAM,QAAQ,IAAI,GAC9B,MACD,SAAS,IAAI,GACX,QAAQ,IAAI,SAAS,IAAI,YAAY,IAAI,OAAO,GAChD,EAAE;EACR,MAAM,KAAK,SAAS,IAAI,GAAI,SAAS,IAAI,GAAG,IAAI,WAAW,MAAO,WAAW;EAC7E,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,UAAU;AACxB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,QAAQ,IAAI,SAAS,EAAE,KAAK;AACvD,OAAI,YAAY,OAAW;GAE3B,MAAM,QADU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ,MAC9B,cAAc,cAAc;GACrD,MAAM,YAAY,SAAS,EAAE,UAAU;AACvC,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,cAAc,SAAY,EAAE,WAAW,GAAG,EAAE;IAAG,CAAC;;AAElF,WAAS,KAAK;GAAE;GAAI;GAAO,CAAC;;AAE9B,QAAO;;AAGT,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}
+{"version":3,"file":"dmr.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]"],"sources":["../../src/loaders/dmr.ts"],"sourcesContent":["/**\n * DMR (Deep Memory Retrieval) loader - the MemGPT/Letta continuity set\n * derived from Multi-Session Chat (arXiv:2310.08560). DMR's published\n * shape is less standardised than LongMemEval/LOCOMO, so this loader is\n * deliberately tolerant: it accepts a JSON **array** of items, each\n * exposing a `question`, an `answer`, and the prior conversation under\n * either `sessions` or `haystack_sessions`. Sessions may be a bare\n * array of turns or an object wrapping the turns under\n * `turns` / `messages` / `dialog`; a turn may use `role` or `speaker`\n * and `content` or `text`.\n *\n * ```jsonc\n * {\n *   \"id\": \"dmr-1\",\n *   \"question\": \"What is the user's sister's name?\",\n *   \"answer\": \"Anna\",\n *   \"sessions\": [\n *     { \"id\": \"s1\", \"turns\": [ { \"role\": \"user\", \"content\": \"...\" } ] },\n *     [ { \"speaker\": \"assistant\", \"text\": \"...\" } ]\n *   ]\n * }\n * ```\n *\n * Every DMR case is treated as `multi-session` (its defining ability).\n * The exact field names should be reconciled against the downloaded\n * file the first time `scripts/fetch-eval-datasets.mjs` is run.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type { MemoryEvalInput, MemoryEvalSession, MemoryEvalTurn } from './memory-eval.js';\n\n/** @stable */\nexport interface LoadDmrOptions {\n  /** Local path to the DMR JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a DMR JSON file and return a fully-materialised {@link Dataset}\n * of multi-session retrieval cases scored against the reference answer.\n *\n * @stable\n */\nexport async function loadDmrDataset(\n  options: LoadDmrOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseDmr(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'dmr',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseDmr(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] DMR dataset must be a JSON array of items.');\n  }\n  const items = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < items.length; i++) {\n    const item = items[i];\n    if (!isRecord(item)) {\n      throw new Error(`[graphorin/evals] DMR item ${i} is not an object.`);\n    }\n    const question = asString(item.question);\n    if (question === undefined) {\n      throw new Error(`[graphorin/evals] DMR item ${i} is missing a 'question'.`);\n    }\n    const sessions = coerceSessions(item.sessions ?? item.haystack_sessions);\n    out.push({\n      id: asString(item.id) ?? `dmr-${i}`,\n      input: { haystackSessions: sessions, question, ability: 'multi-session' },\n      expected: asString(item.answer) ?? '',\n      metadata: { datasetName: 'dmr', ability: 'multi-session' },\n    });\n  }\n  return out;\n}\n\nfunction coerceSessions(value: unknown): MemoryEvalSession[] {\n  const rawSessions = asArray(value);\n  const sessions: MemoryEvalSession[] = [];\n  for (let s = 0; s < rawSessions.length; s++) {\n    const raw = rawSessions[s];\n    const rawTurns = Array.isArray(raw)\n      ? (raw as ReadonlyArray<unknown>)\n      : isRecord(raw)\n        ? asArray(raw.turns ?? raw.messages ?? raw.dialog)\n        : [];\n    const id = isRecord(raw) ? (asString(raw.id) ?? `session-${s}`) : `session-${s}`;\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of rawTurns) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.content) ?? asString(t.text);\n      if (content === undefined) continue;\n      const speaker = asString(t.role) ?? asString(t.speaker);\n      const role = speaker === 'assistant' ? 'assistant' : 'user';\n      const timestamp = asString(t.timestamp);\n      turns.push({ role, content, ...(timestamp !== undefined ? { timestamp } : {}) });\n    }\n    sessions.push({ id, turns });\n  }\n  return sessions;\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAoDA,eAAsB,eACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,SADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CACrB;EAG1B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,SAAS,MAA4D;CACnF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,+DAA+D;CAEjF,MAAM,QAAQ;CACd,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;EACrC,MAAM,OAAO,MAAM;AACnB,MAAI,CAAC,SAAS,KAAK,CACjB,OAAM,IAAI,MAAM,8BAA8B,EAAE,oBAAoB;EAEtE,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OACf,OAAM,IAAI,MAAM,8BAA8B,EAAE,2BAA2B;EAE7E,MAAM,WAAW,eAAe,KAAK,YAAY,KAAK,kBAAkB;AACxE,MAAI,KAAK;GACP,IAAI,SAAS,KAAK,GAAG,IAAI,OAAO;GAChC,OAAO;IAAE,kBAAkB;IAAU;IAAU,SAAS;IAAiB;GACzE,UAAU,SAAS,KAAK,OAAO,IAAI;GACnC,UAAU;IAAE,aAAa;IAAO,SAAS;IAAiB;GAC3D,CAAC;;AAEJ,QAAO;;AAGT,SAAS,eAAe,OAAqC;CAC3D,MAAM,cAAc,QAAQ,MAAM;CAClC,MAAMC,WAAgC,EAAE;AACxC,MAAK,IAAI,IAAI,GAAG,IAAI,YAAY,QAAQ,KAAK;EAC3C,MAAM,MAAM,YAAY;EACxB,MAAM,WAAW,MAAM,QAAQ,IAAI,GAC9B,MACD,SAAS,IAAI,GACX,QAAQ,IAAI,SAAS,IAAI,YAAY,IAAI,OAAO,GAChD,EAAE;EACR,MAAM,KAAK,SAAS,IAAI,GAAI,SAAS,IAAI,GAAG,IAAI,WAAW,MAAO,WAAW;EAC7E,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,UAAU;AACxB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,QAAQ,IAAI,SAAS,EAAE,KAAK;AACvD,OAAI,YAAY,OAAW;GAE3B,MAAM,QADU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ,MAC9B,cAAc,cAAc;GACrD,MAAM,YAAY,SAAS,EAAE,UAAU;AACvC,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,cAAc,SAAY,EAAE,WAAW,GAAG,EAAE;IAAG,CAAC;;AAElF,WAAS,KAAK;GAAE;GAAI;GAAO,CAAC;;AAE9B,QAAO;;AAGT,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}

package/dist/loaders/from-traces.js

@@ -2,7 +2,7 @@ import { readFile } from "node:fs/promises";
 
 //#region src/loaders/from-traces.ts
 /**
-* `loadDatasetFromTraces` — distil a dataset from the framework's
+* `loadDatasetFromTraces` - distil a dataset from the framework's
 * replay log. The caller supplies the JSONL replay file and a small
 * extraction function that pulls the `(input, output)` pair out of
 * each event group; the loader handles the JSONL parsing + grouping

package/dist/loaders/from-traces.js.map

@@ -1 +1 @@
-{"version":3,"file":"from-traces.js","names":["parsed: unknown","event: TraceEvent","cases: Case<I, O>[]"],"sources":["../../src/loaders/from-traces.ts"],"sourcesContent":["/**\n * `loadDatasetFromTraces` — distil a dataset from the framework's\n * replay log. The caller supplies the JSONL replay file and a small\n * extraction function that pulls the `(input, output)` pair out of\n * each event group; the loader handles the JSONL parsing + grouping\n * by `runId`.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\n/** @stable */\nexport interface TraceEvent {\n  readonly runId: string;\n  readonly type: string;\n  readonly payload: Readonly<Record<string, unknown>>;\n  readonly timestamp?: string;\n}\n\n/** @stable */\nexport interface FromTracesOptions<I, O> {\n  /** Distil one `Case<I, O>` from every group of events sharing a `runId`. */\n  readonly extract: (events: ReadonlyArray<TraceEvent>) => Case<I, O> | null;\n  /** Optional name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/** @stable */\nexport async function loadDatasetFromTraces<I, O>(\n  path: string,\n  options: FromTracesOptions<I, O>,\n): Promise<Dataset<I, O>> {\n  const text = await readFile(path, 'utf8');\n  return groupAndExtract<I, O>(text, options);\n}\n\n/**\n * Pure parser for the trace JSONL format. Exported so tests can\n * exercise the extraction without touching the filesystem.\n *\n * @stable\n */\nexport function groupAndExtract<I, O>(\n  text: string,\n  options: FromTracesOptions<I, O>,\n): Dataset<I, O> {\n  const lines = text.split(/\\r?\\n/);\n  const groups = new Map<string, TraceEvent[]>();\n  for (const raw of lines) {\n    if (raw.trim().length === 0) continue;\n    let parsed: unknown;\n    try {\n      parsed = JSON.parse(raw);\n    } catch {\n      continue;\n    }\n    if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) continue;\n    const obj = parsed as Record<string, unknown>;\n    const runId = typeof obj.runId === 'string' ? obj.runId : null;\n    const type = typeof obj.type === 'string' ? obj.type : null;\n    if (runId === null || type === null) continue;\n    const payload =\n      obj.payload !== null && typeof obj.payload === 'object' && !Array.isArray(obj.payload)\n        ? (obj.payload as Readonly<Record<string, unknown>>)\n        : ({} as Readonly<Record<string, unknown>>);\n    const event: TraceEvent = {\n      runId,\n      type,\n      payload,\n      ...(typeof obj.timestamp === 'string' ? { timestamp: obj.timestamp } : {}),\n    };\n    const existing = groups.get(runId);\n    if (existing === undefined) groups.set(runId, [event]);\n    else existing.push(event);\n  }\n  const cases: Case<I, O>[] = [];\n  for (const events of groups.values()) {\n    const sample = options.extract(events);\n    if (sample !== null) cases.push(sample);\n  }\n  const meta: Dataset<I, O>['metadata'] = {\n    ...(options.name !== undefined ? { name: options.name } : {}),\n    ...(options.description !== undefined ? { description: options.description } : {}),\n    createdAt: new Date(),\n  };\n  return { cases, metadata: meta };\n}\n"],"mappings":";;;;;;;;;;;;;AAiCA,eAAsB,sBACpB,MACA,SACwB;AAExB,QAAO,gBADM,MAAM,SAAS,MAAM,OAAO,EACN,QAAQ;;;;;;;;AAS7C,SAAgB,gBACd,MACA,SACe;CACf,MAAM,QAAQ,KAAK,MAAM,QAAQ;CACjC,MAAM,yBAAS,IAAI,KAA2B;AAC9C,MAAK,MAAM,OAAO,OAAO;AACvB,MAAI,IAAI,MAAM,CAAC,WAAW,EAAG;EAC7B,IAAIA;AACJ,MAAI;AACF,YAAS,KAAK,MAAM,IAAI;UAClB;AACN;;AAEF,MAAI,WAAW,QAAQ,OAAO,WAAW,YAAY,MAAM,QAAQ,OAAO,CAAE;EAC5E,MAAM,MAAM;EACZ,MAAM,QAAQ,OAAO,IAAI,UAAU,WAAW,IAAI,QAAQ;EAC1D,MAAM,OAAO,OAAO,IAAI,SAAS,WAAW,IAAI,OAAO;AACvD,MAAI,UAAU,QAAQ,SAAS,KAAM;EAKrC,MAAMC,QAAoB;GACxB;GACA;GACA,SANA,IAAI,YAAY,QAAQ,OAAO,IAAI,YAAY,YAAY,CAAC,MAAM,QAAQ,IAAI,QAAQ,GACjF,IAAI,UACJ,EAAE;GAKP,GAAI,OAAO,IAAI,cAAc,WAAW,EAAE,WAAW,IAAI,WAAW,GAAG,EAAE;GAC1E;EACD,MAAM,WAAW,OAAO,IAAI,MAAM;AAClC,MAAI,aAAa,OAAW,QAAO,IAAI,OAAO,CAAC,MAAM,CAAC;MACjD,UAAS,KAAK,MAAM;;CAE3B,MAAMC,QAAsB,EAAE;AAC9B,MAAK,MAAM,UAAU,OAAO,QAAQ,EAAE;EACpC,MAAM,SAAS,QAAQ,QAAQ,OAAO;AACtC,MAAI,WAAW,KAAM,OAAM,KAAK,OAAO;;AAOzC,QAAO;EAAE;EAAO,UALwB;GACtC,GAAI,QAAQ,SAAS,SAAY,EAAE,MAAM,QAAQ,MAAM,GAAG,EAAE;GAC5D,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EAC+B"}
+{"version":3,"file":"from-traces.js","names":["parsed: unknown","event: TraceEvent","cases: Case<I, O>[]"],"sources":["../../src/loaders/from-traces.ts"],"sourcesContent":["/**\n * `loadDatasetFromTraces` - distil a dataset from the framework's\n * replay log. The caller supplies the JSONL replay file and a small\n * extraction function that pulls the `(input, output)` pair out of\n * each event group; the loader handles the JSONL parsing + grouping\n * by `runId`.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\n/** @stable */\nexport interface TraceEvent {\n  readonly runId: string;\n  readonly type: string;\n  readonly payload: Readonly<Record<string, unknown>>;\n  readonly timestamp?: string;\n}\n\n/** @stable */\nexport interface FromTracesOptions<I, O> {\n  /** Distil one `Case<I, O>` from every group of events sharing a `runId`. */\n  readonly extract: (events: ReadonlyArray<TraceEvent>) => Case<I, O> | null;\n  /** Optional name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/** @stable */\nexport async function loadDatasetFromTraces<I, O>(\n  path: string,\n  options: FromTracesOptions<I, O>,\n): Promise<Dataset<I, O>> {\n  const text = await readFile(path, 'utf8');\n  return groupAndExtract<I, O>(text, options);\n}\n\n/**\n * Pure parser for the trace JSONL format. Exported so tests can\n * exercise the extraction without touching the filesystem.\n *\n * @stable\n */\nexport function groupAndExtract<I, O>(\n  text: string,\n  options: FromTracesOptions<I, O>,\n): Dataset<I, O> {\n  const lines = text.split(/\\r?\\n/);\n  const groups = new Map<string, TraceEvent[]>();\n  for (const raw of lines) {\n    if (raw.trim().length === 0) continue;\n    let parsed: unknown;\n    try {\n      parsed = JSON.parse(raw);\n    } catch {\n      continue;\n    }\n    if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) continue;\n    const obj = parsed as Record<string, unknown>;\n    const runId = typeof obj.runId === 'string' ? obj.runId : null;\n    const type = typeof obj.type === 'string' ? obj.type : null;\n    if (runId === null || type === null) continue;\n    const payload =\n      obj.payload !== null && typeof obj.payload === 'object' && !Array.isArray(obj.payload)\n        ? (obj.payload as Readonly<Record<string, unknown>>)\n        : ({} as Readonly<Record<string, unknown>>);\n    const event: TraceEvent = {\n      runId,\n      type,\n      payload,\n      ...(typeof obj.timestamp === 'string' ? { timestamp: obj.timestamp } : {}),\n    };\n    const existing = groups.get(runId);\n    if (existing === undefined) groups.set(runId, [event]);\n    else existing.push(event);\n  }\n  const cases: Case<I, O>[] = [];\n  for (const events of groups.values()) {\n    const sample = options.extract(events);\n    if (sample !== null) cases.push(sample);\n  }\n  const meta: Dataset<I, O>['metadata'] = {\n    ...(options.name !== undefined ? { name: options.name } : {}),\n    ...(options.description !== undefined ? { description: options.description } : {}),\n    createdAt: new Date(),\n  };\n  return { cases, metadata: meta };\n}\n"],"mappings":";;;;;;;;;;;;;AAiCA,eAAsB,sBACpB,MACA,SACwB;AAExB,QAAO,gBADM,MAAM,SAAS,MAAM,OAAO,EACN,QAAQ;;;;;;;;AAS7C,SAAgB,gBACd,MACA,SACe;CACf,MAAM,QAAQ,KAAK,MAAM,QAAQ;CACjC,MAAM,yBAAS,IAAI,KAA2B;AAC9C,MAAK,MAAM,OAAO,OAAO;AACvB,MAAI,IAAI,MAAM,CAAC,WAAW,EAAG;EAC7B,IAAIA;AACJ,MAAI;AACF,YAAS,KAAK,MAAM,IAAI;UAClB;AACN;;AAEF,MAAI,WAAW,QAAQ,OAAO,WAAW,YAAY,MAAM,QAAQ,OAAO,CAAE;EAC5E,MAAM,MAAM;EACZ,MAAM,QAAQ,OAAO,IAAI,UAAU,WAAW,IAAI,QAAQ;EAC1D,MAAM,OAAO,OAAO,IAAI,SAAS,WAAW,IAAI,OAAO;AACvD,MAAI,UAAU,QAAQ,SAAS,KAAM;EAKrC,MAAMC,QAAoB;GACxB;GACA;GACA,SANA,IAAI,YAAY,QAAQ,OAAO,IAAI,YAAY,YAAY,CAAC,MAAM,QAAQ,IAAI,QAAQ,GACjF,IAAI,UACJ,EAAE;GAKP,GAAI,OAAO,IAAI,cAAc,WAAW,EAAE,WAAW,IAAI,WAAW,GAAG,EAAE;GAC1E;EACD,MAAM,WAAW,OAAO,IAAI,MAAM;AAClC,MAAI,aAAa,OAAW,QAAO,IAAI,OAAO,CAAC,MAAM,CAAC;MACjD,UAAS,KAAK,MAAM;;CAE3B,MAAMC,QAAsB,EAAE;AAC9B,MAAK,MAAM,UAAU,OAAO,QAAQ,EAAE;EACpC,MAAM,SAAS,QAAQ,QAAQ,OAAO;AACtC,MAAI,WAAW,KAAM,OAAM,KAAK,OAAO;;AAOzC,QAAO;EAAE;EAAO,UALwB;GACtC,GAAI,QAAQ,SAAS,SAAY,EAAE,MAAM,QAAQ,MAAM,GAAG,EAAE;GAC5D,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EAC+B"}

package/dist/loaders/iterable.js.map

@@ -1 +1 @@
-{"version":3,"file":"iterable.js","names":[],"sources":["../../src/loaders/iterable.ts"],"sourcesContent":["/**\n * `fromIterable` — wrap an in-memory array / iterable as a\n * {@link Dataset}. Useful for tests + ad-hoc data.\n *\n * @packageDocumentation\n */\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\n/** @stable */\nexport function fromIterable<I, O = unknown>(\n  cases: Iterable<Case<I, O>> | ReadonlyArray<Case<I, O>>,\n  options: { readonly name?: string; readonly description?: string } = {},\n): Dataset<I, O> {\n  const arr = Array.isArray(cases) ? (cases as ReadonlyArray<Case<I, O>>) : Array.from(cases);\n  const meta: Dataset<I, O>['metadata'] = {\n    ...(options.name !== undefined ? { name: options.name } : {}),\n    ...(options.description !== undefined ? { description: options.description } : {}),\n    createdAt: new Date(),\n  };\n  return { cases: arr, metadata: meta };\n}\n"],"mappings":";;AAUA,SAAgB,aACd,OACA,UAAqE,EAAE,EACxD;AAOf,QAAO;EAAE,OANG,MAAM,QAAQ,MAAM,GAAI,QAAsC,MAAM,KAAK,MAAM;EAMtE,UALmB;GACtC,GAAI,QAAQ,SAAS,SAAY,EAAE,MAAM,QAAQ,MAAM,GAAG,EAAE;GAC5D,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACoC"}
+{"version":3,"file":"iterable.js","names":[],"sources":["../../src/loaders/iterable.ts"],"sourcesContent":["/**\n * `fromIterable` - wrap an in-memory array / iterable as a\n * {@link Dataset}. Useful for tests + ad-hoc data.\n *\n * @packageDocumentation\n */\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\n/** @stable */\nexport function fromIterable<I, O = unknown>(\n  cases: Iterable<Case<I, O>> | ReadonlyArray<Case<I, O>>,\n  options: { readonly name?: string; readonly description?: string } = {},\n): Dataset<I, O> {\n  const arr = Array.isArray(cases) ? (cases as ReadonlyArray<Case<I, O>>) : Array.from(cases);\n  const meta: Dataset<I, O>['metadata'] = {\n    ...(options.name !== undefined ? { name: options.name } : {}),\n    ...(options.description !== undefined ? { description: options.description } : {}),\n    createdAt: new Date(),\n  };\n  return { cases: arr, metadata: meta };\n}\n"],"mappings":";;AAUA,SAAgB,aACd,OACA,UAAqE,EAAE,EACxD;AAOf,QAAO;EAAE,OANG,MAAM,QAAQ,MAAM,GAAI,QAAsC,MAAM,KAAK,MAAM;EAMtE,UALmB;GACtC,GAAI,QAAQ,SAAS,SAAY,EAAE,MAAM,QAAQ,MAAM,GAAG,EAAE;GAC5D,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACoC"}

package/dist/loaders/locomo.d.ts

@@ -14,7 +14,7 @@ interface LoadLocomoOptions {
 }
 /**
  * Read a LOCOMO JSON file and return a fully-materialised
- * {@link Dataset} — one case per QA pair, scored against the reference
+ * {@link Dataset} - one case per QA pair, scored against the reference
  * answer string (LLM-judge "J").
  *
  * @stable

package/dist/loaders/locomo.js

@@ -2,7 +2,7 @@ import { readFile } from "node:fs/promises";
 
 //#region src/loaders/locomo.ts
 /**
-* Real LOCOMO dataset loader (arXiv:2402.17753 —
+* Real LOCOMO dataset loader (arXiv:2402.17753 -
 * https://snap-research.github.io/locomo/). LOCOMO ships as a JSON
 * **array** of samples; each sample bundles one multi-session
 * conversation with many QA pairs over it:
@@ -35,7 +35,7 @@ import { readFile } from "node:fs/promises";
 */
 /**
 * Read a LOCOMO JSON file and return a fully-materialised
-* {@link Dataset} — one case per QA pair, scored against the reference
+* {@link Dataset} - one case per QA pair, scored against the reference
 * answer string (LLM-judge "J").
 *
 * @stable

package/dist/loaders/locomo.js.map

@@ -1 +1 @@
-{"version":3,"file":"locomo.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","indices: number[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]","role: 'user' | 'assistant'"],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":["/**\n * Real LOCOMO dataset loader (arXiv:2402.17753 —\n * https://snap-research.github.io/locomo/). LOCOMO ships as a JSON\n * **array** of samples; each sample bundles one multi-session\n * conversation with many QA pairs over it:\n *\n * ```jsonc\n * {\n *   \"sample_id\": \"conv-26\",\n *   \"conversation\": {\n *     \"speaker_a\": \"Caroline\",\n *     \"speaker_b\": \"Melanie\",\n *     \"session_1_date_time\": \"1:56 pm on 8 May, 2023\",\n *     \"session_1\": [ { \"speaker\": \"Caroline\", \"dia_id\": \"D1:1\", \"text\": \"...\" }, ... ],\n *     \"session_2_date_time\": \"...\",\n *     \"session_2\": [ ... ]\n *   },\n *   \"qa\": [\n *     { \"question\": \"...\", \"answer\": \"...\", \"evidence\": [\"D1:1\"], \"category\": 1 },\n *     { \"question\": \"...\", \"category\": 5, \"adversarial_answer\": \"Not mentioned ...\" }\n *   ]\n * }\n * ```\n *\n * Each QA pair becomes one {@link MemoryEvalInput} case that shares the\n * sample's sessions. LOCOMO categories map to abilities as:\n * `1`→multi-session (multi-hop), `2`→temporal, `5`→abstention\n * (adversarial), everything else (`3` open-domain, `4` single-hop)\n * →info-extraction.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLocomoOptions {\n  /** Local path to the LOCOMO JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LOCOMO JSON file and return a fully-materialised\n * {@link Dataset} — one case per QA pair, scored against the reference\n * answer string (LLM-judge \"J\").\n *\n * @stable\n */\nexport async function loadLocomoDataset(\n  options: LoadLocomoOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLocomo(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'locomo',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLocomo(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LOCOMO dataset must be a JSON array of samples.');\n  }\n  const samples = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < samples.length; i++) {\n    const sample = samples[i];\n    if (sample === null || typeof sample !== 'object') {\n      throw new Error(`[graphorin/evals] LOCOMO sample ${i} is not an object.`);\n    }\n    appendSampleCases(sample as Record<string, unknown>, i, out);\n  }\n  return out;\n}\n\nfunction appendSampleCases(\n  sample: Record<string, unknown>,\n  index: number,\n  out: Case<MemoryEvalInput, string>[],\n): void {\n  const sampleId = asString(sample.sample_id) ?? `locomo-${index}`;\n  const conversation = isRecord(sample.conversation) ? sample.conversation : {};\n  const sessions = extractSessions(conversation);\n  const qa = asArray(sample.qa);\n  for (let q = 0; q < qa.length; q++) {\n    const item = qa[q];\n    if (!isRecord(item)) continue;\n    const question = asString(item.question);\n    if (question === undefined) continue;\n    const category = typeof item.category === 'number' ? item.category : undefined;\n    const ability = mapCategory(category);\n    const answer = asString(item.answer) ?? asString(item.adversarial_answer) ?? '';\n    const evidence = asArray(item.evidence).filter((e): e is string => typeof e === 'string');\n    out.push({\n      id: `${sampleId}-q${q}`,\n      input: { haystackSessions: sessions, question, ability },\n      expected: answer,\n      metadata: {\n        datasetName: 'locomo',\n        sampleId,\n        ability,\n        ...(category !== undefined ? { category } : {}),\n        evidence,\n      },\n    });\n  }\n}\n\nfunction extractSessions(conversation: Record<string, unknown>): MemoryEvalSession[] {\n  const speakerA = asString(conversation.speaker_a);\n  const indices: number[] = [];\n  for (const key of Object.keys(conversation)) {\n    const m = /^session_(\\d+)$/.exec(key);\n    if (m?.[1] !== undefined) indices.push(Number.parseInt(m[1], 10));\n  }\n  indices.sort((a, b) => a - b);\n  const sessions: MemoryEvalSession[] = [];\n  for (const n of indices) {\n    const raw = asArray(conversation[`session_${n}`]);\n    const date = asString(conversation[`session_${n}_date_time`]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of raw) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.text) ?? asString(t.content);\n      if (content === undefined) continue;\n      const speaker = asString(t.speaker);\n      const role: 'user' | 'assistant' =\n        speakerA !== undefined && speaker !== undefined && speaker !== speakerA\n          ? 'assistant'\n          : 'user';\n      turns.push({ role, content, ...(date !== undefined ? { timestamp: date } : {}) });\n    }\n    sessions.push({ id: `session_${n}`, turns });\n  }\n  return sessions;\n}\n\n/** Maps a LOCOMO numeric `category` onto a {@link MemoryEvalAbility}. */\nfunction mapCategory(category: number | undefined): MemoryEvalAbility {\n  switch (category) {\n    case 1:\n      return 'multi-session';\n    case 2:\n      return 'temporal';\n    case 5:\n      return 'abstention';\n    default:\n      return 'info-extraction';\n  }\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA6DA,eAAsB,kBACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,YADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CAClB;EAG7B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,YAAY,MAA4D;CACtF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,oEAAoE;CAEtF,MAAM,UAAU;CAChB,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,QAAQ,QAAQ,KAAK;EACvC,MAAM,SAAS,QAAQ;AACvB,MAAI,WAAW,QAAQ,OAAO,WAAW,SACvC,OAAM,IAAI,MAAM,mCAAmC,EAAE,oBAAoB;AAE3E,oBAAkB,QAAmC,GAAG,IAAI;;AAE9D,QAAO;;AAGT,SAAS,kBACP,QACA,OACA,KACM;CACN,MAAM,WAAW,SAAS,OAAO,UAAU,IAAI,UAAU;CAEzD,MAAM,WAAW,gBADI,SAAS,OAAO,aAAa,GAAG,OAAO,eAAe,EAAE,CAC/B;CAC9C,MAAM,KAAK,QAAQ,OAAO,GAAG;AAC7B,MAAK,IAAI,IAAI,GAAG,IAAI,GAAG,QAAQ,KAAK;EAClC,MAAM,OAAO,GAAG;AAChB,MAAI,CAAC,SAAS,KAAK,CAAE;EACrB,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OAAW;EAC5B,MAAM,WAAW,OAAO,KAAK,aAAa,WAAW,KAAK,WAAW;EACrE,MAAM,UAAU,YAAY,SAAS;EACrC,MAAM,SAAS,SAAS,KAAK,OAAO,IAAI,SAAS,KAAK,mBAAmB,IAAI;EAC7E,MAAM,WAAW,QAAQ,KAAK,SAAS,CAAC,QAAQ,MAAmB,OAAO,MAAM,SAAS;AACzF,MAAI,KAAK;GACP,IAAI,GAAG,SAAS,IAAI;GACpB,OAAO;IAAE,kBAAkB;IAAU;IAAU;IAAS;GACxD,UAAU;GACV,UAAU;IACR,aAAa;IACb;IACA;IACA,GAAI,aAAa,SAAY,EAAE,UAAU,GAAG,EAAE;IAC9C;IACD;GACF,CAAC;;;AAIN,SAAS,gBAAgB,cAA4D;CACnF,MAAM,WAAW,SAAS,aAAa,UAAU;CACjD,MAAMC,UAAoB,EAAE;AAC5B,MAAK,MAAM,OAAO,OAAO,KAAK,aAAa,EAAE;EAC3C,MAAM,IAAI,kBAAkB,KAAK,IAAI;AACrC,MAAI,IAAI,OAAO,OAAW,SAAQ,KAAK,OAAO,SAAS,EAAE,IAAI,GAAG,CAAC;;AAEnE,SAAQ,MAAM,GAAG,MAAM,IAAI,EAAE;CAC7B,MAAMC,WAAgC,EAAE;AACxC,MAAK,MAAM,KAAK,SAAS;EACvB,MAAM,MAAM,QAAQ,aAAa,WAAW,KAAK;EACjD,MAAM,OAAO,SAAS,aAAa,WAAW,EAAE,aAAa;EAC7D,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,KAAK;AACnB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ;AACvD,OAAI,YAAY,OAAW;GAC3B,MAAM,UAAU,SAAS,EAAE,QAAQ;GACnC,MAAMC,OACJ,aAAa,UAAa,YAAY,UAAa,YAAY,WAC3D,cACA;AACN,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAAG,CAAC;;AAEnF,WAAS,KAAK;GAAE,IAAI,WAAW;GAAK;GAAO,CAAC;;AAE9C,QAAO;;;AAIT,SAAS,YAAY,UAAiD;AACpE,SAAQ,UAAR;EACE,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,QACE,QAAO;;;AAIb,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}
+{"version":3,"file":"locomo.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","indices: number[]","sessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]","role: 'user' | 'assistant'"],"sources":["../../src/loaders/locomo.ts"],"sourcesContent":["/**\n * Real LOCOMO dataset loader (arXiv:2402.17753 -\n * https://snap-research.github.io/locomo/). LOCOMO ships as a JSON\n * **array** of samples; each sample bundles one multi-session\n * conversation with many QA pairs over it:\n *\n * ```jsonc\n * {\n *   \"sample_id\": \"conv-26\",\n *   \"conversation\": {\n *     \"speaker_a\": \"Caroline\",\n *     \"speaker_b\": \"Melanie\",\n *     \"session_1_date_time\": \"1:56 pm on 8 May, 2023\",\n *     \"session_1\": [ { \"speaker\": \"Caroline\", \"dia_id\": \"D1:1\", \"text\": \"...\" }, ... ],\n *     \"session_2_date_time\": \"...\",\n *     \"session_2\": [ ... ]\n *   },\n *   \"qa\": [\n *     { \"question\": \"...\", \"answer\": \"...\", \"evidence\": [\"D1:1\"], \"category\": 1 },\n *     { \"question\": \"...\", \"category\": 5, \"adversarial_answer\": \"Not mentioned ...\" }\n *   ]\n * }\n * ```\n *\n * Each QA pair becomes one {@link MemoryEvalInput} case that shares the\n * sample's sessions. LOCOMO categories map to abilities as:\n * `1`→multi-session (multi-hop), `2`→temporal, `5`→abstention\n * (adversarial), everything else (`3` open-domain, `4` single-hop)\n * →info-extraction.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLocomoOptions {\n  /** Local path to the LOCOMO JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LOCOMO JSON file and return a fully-materialised\n * {@link Dataset} - one case per QA pair, scored against the reference\n * answer string (LLM-judge \"J\").\n *\n * @stable\n */\nexport async function loadLocomoDataset(\n  options: LoadLocomoOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLocomo(text);\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? 'locomo',\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLocomo(text: string): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LOCOMO dataset must be a JSON array of samples.');\n  }\n  const samples = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < samples.length; i++) {\n    const sample = samples[i];\n    if (sample === null || typeof sample !== 'object') {\n      throw new Error(`[graphorin/evals] LOCOMO sample ${i} is not an object.`);\n    }\n    appendSampleCases(sample as Record<string, unknown>, i, out);\n  }\n  return out;\n}\n\nfunction appendSampleCases(\n  sample: Record<string, unknown>,\n  index: number,\n  out: Case<MemoryEvalInput, string>[],\n): void {\n  const sampleId = asString(sample.sample_id) ?? `locomo-${index}`;\n  const conversation = isRecord(sample.conversation) ? sample.conversation : {};\n  const sessions = extractSessions(conversation);\n  const qa = asArray(sample.qa);\n  for (let q = 0; q < qa.length; q++) {\n    const item = qa[q];\n    if (!isRecord(item)) continue;\n    const question = asString(item.question);\n    if (question === undefined) continue;\n    const category = typeof item.category === 'number' ? item.category : undefined;\n    const ability = mapCategory(category);\n    const answer = asString(item.answer) ?? asString(item.adversarial_answer) ?? '';\n    const evidence = asArray(item.evidence).filter((e): e is string => typeof e === 'string');\n    out.push({\n      id: `${sampleId}-q${q}`,\n      input: { haystackSessions: sessions, question, ability },\n      expected: answer,\n      metadata: {\n        datasetName: 'locomo',\n        sampleId,\n        ability,\n        ...(category !== undefined ? { category } : {}),\n        evidence,\n      },\n    });\n  }\n}\n\nfunction extractSessions(conversation: Record<string, unknown>): MemoryEvalSession[] {\n  const speakerA = asString(conversation.speaker_a);\n  const indices: number[] = [];\n  for (const key of Object.keys(conversation)) {\n    const m = /^session_(\\d+)$/.exec(key);\n    if (m?.[1] !== undefined) indices.push(Number.parseInt(m[1], 10));\n  }\n  indices.sort((a, b) => a - b);\n  const sessions: MemoryEvalSession[] = [];\n  for (const n of indices) {\n    const raw = asArray(conversation[`session_${n}`]);\n    const date = asString(conversation[`session_${n}_date_time`]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of raw) {\n      if (!isRecord(t)) continue;\n      const content = asString(t.text) ?? asString(t.content);\n      if (content === undefined) continue;\n      const speaker = asString(t.speaker);\n      const role: 'user' | 'assistant' =\n        speakerA !== undefined && speaker !== undefined && speaker !== speakerA\n          ? 'assistant'\n          : 'user';\n      turns.push({ role, content, ...(date !== undefined ? { timestamp: date } : {}) });\n    }\n    sessions.push({ id: `session_${n}`, turns });\n  }\n  return sessions;\n}\n\n/** Maps a LOCOMO numeric `category` onto a {@link MemoryEvalAbility}. */\nfunction mapCategory(category: number | undefined): MemoryEvalAbility {\n  switch (category) {\n    case 1:\n      return 'multi-session';\n    case 2:\n      return 'temporal';\n    case 5:\n      return 'abstention';\n    default:\n      return 'info-extraction';\n  }\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n\nfunction isRecord(value: unknown): value is Record<string, unknown> {\n  return value !== null && typeof value === 'object' && !Array.isArray(value);\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA6DA,eAAsB,kBACpB,SAC2C;AAG3C,QAAO;EACL,OAFY,YADD,MAAM,SAAS,QAAQ,MAAM,OAAO,CAClB;EAG7B,UAAU;GACR,MAAM,QAAQ,QAAQ;GACtB,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,YAAY,MAA4D;CACtF,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,oEAAoE;CAEtF,MAAM,UAAU;CAChB,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,QAAQ,QAAQ,KAAK;EACvC,MAAM,SAAS,QAAQ;AACvB,MAAI,WAAW,QAAQ,OAAO,WAAW,SACvC,OAAM,IAAI,MAAM,mCAAmC,EAAE,oBAAoB;AAE3E,oBAAkB,QAAmC,GAAG,IAAI;;AAE9D,QAAO;;AAGT,SAAS,kBACP,QACA,OACA,KACM;CACN,MAAM,WAAW,SAAS,OAAO,UAAU,IAAI,UAAU;CAEzD,MAAM,WAAW,gBADI,SAAS,OAAO,aAAa,GAAG,OAAO,eAAe,EAAE,CAC/B;CAC9C,MAAM,KAAK,QAAQ,OAAO,GAAG;AAC7B,MAAK,IAAI,IAAI,GAAG,IAAI,GAAG,QAAQ,KAAK;EAClC,MAAM,OAAO,GAAG;AAChB,MAAI,CAAC,SAAS,KAAK,CAAE;EACrB,MAAM,WAAW,SAAS,KAAK,SAAS;AACxC,MAAI,aAAa,OAAW;EAC5B,MAAM,WAAW,OAAO,KAAK,aAAa,WAAW,KAAK,WAAW;EACrE,MAAM,UAAU,YAAY,SAAS;EACrC,MAAM,SAAS,SAAS,KAAK,OAAO,IAAI,SAAS,KAAK,mBAAmB,IAAI;EAC7E,MAAM,WAAW,QAAQ,KAAK,SAAS,CAAC,QAAQ,MAAmB,OAAO,MAAM,SAAS;AACzF,MAAI,KAAK;GACP,IAAI,GAAG,SAAS,IAAI;GACpB,OAAO;IAAE,kBAAkB;IAAU;IAAU;IAAS;GACxD,UAAU;GACV,UAAU;IACR,aAAa;IACb;IACA;IACA,GAAI,aAAa,SAAY,EAAE,UAAU,GAAG,EAAE;IAC9C;IACD;GACF,CAAC;;;AAIN,SAAS,gBAAgB,cAA4D;CACnF,MAAM,WAAW,SAAS,aAAa,UAAU;CACjD,MAAMC,UAAoB,EAAE;AAC5B,MAAK,MAAM,OAAO,OAAO,KAAK,aAAa,EAAE;EAC3C,MAAM,IAAI,kBAAkB,KAAK,IAAI;AACrC,MAAI,IAAI,OAAO,OAAW,SAAQ,KAAK,OAAO,SAAS,EAAE,IAAI,GAAG,CAAC;;AAEnE,SAAQ,MAAM,GAAG,MAAM,IAAI,EAAE;CAC7B,MAAMC,WAAgC,EAAE;AACxC,MAAK,MAAM,KAAK,SAAS;EACvB,MAAM,MAAM,QAAQ,aAAa,WAAW,KAAK;EACjD,MAAM,OAAO,SAAS,aAAa,WAAW,EAAE,aAAa;EAC7D,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,KAAK;AACnB,OAAI,CAAC,SAAS,EAAE,CAAE;GAClB,MAAM,UAAU,SAAS,EAAE,KAAK,IAAI,SAAS,EAAE,QAAQ;AACvD,OAAI,YAAY,OAAW;GAC3B,MAAM,UAAU,SAAS,EAAE,QAAQ;GACnC,MAAMC,OACJ,aAAa,UAAa,YAAY,UAAa,YAAY,WAC3D,cACA;AACN,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAAG,CAAC;;AAEnF,WAAS,KAAK;GAAE,IAAI,WAAW;GAAK;GAAO,CAAC;;AAE9C,QAAO;;;AAIT,SAAS,YAAY,UAAiD;AACpE,SAAQ,UAAR;EACE,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,KAAK,EACH,QAAO;EACT,QACE,QAAO;;;AAIb,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE;;AAGtE,SAAS,SAAS,OAAkD;AAClE,QAAO,UAAU,QAAQ,OAAO,UAAU,YAAY,CAAC,MAAM,QAAQ,MAAM"}

package/dist/loaders/longmemeval.js

@@ -2,7 +2,7 @@ import { readFile } from "node:fs/promises";
 
 //#region src/loaders/longmemeval.ts
 /**
-* LongMemEval dataset loader (ICLR 2025, arXiv:2410.10813 —
+* LongMemEval dataset loader (ICLR 2025, arXiv:2410.10813 -
 * https://github.com/xiaowu0162/LongMemEval). LongMemEval ships as a
 * JSON **array** of question objects:
 *

package/dist/loaders/longmemeval.js.map

@@ -1 +1 @@
-{"version":3,"file":"longmemeval.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","haystackSessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]"],"sources":["../../src/loaders/longmemeval.ts"],"sourcesContent":["/**\n * LongMemEval dataset loader (ICLR 2025, arXiv:2410.10813 —\n * https://github.com/xiaowu0162/LongMemEval). LongMemEval ships as a\n * JSON **array** of question objects:\n *\n * ```jsonc\n * {\n *   \"question_id\": \"abc_abs\",          // `_abs` suffix marks abstention\n *   \"question_type\": \"temporal-reasoning\",\n *   \"question\": \"Where did I live in 2021?\",\n *   \"answer\": \"Berlin\",\n *   \"question_date\": \"2023/05/20 (Sat) 14:00\",\n *   \"haystack_session_ids\": [\"s1\", \"s2\"],\n *   \"haystack_dates\": [\"2021/03/01 ...\", \"2023/01/05 ...\"],\n *   \"haystack_sessions\": [\n *     [ { \"role\": \"user\", \"content\": \"...\" }, { \"role\": \"assistant\", \"content\": \"...\", \"has_answer\": true } ],\n *     [ ... ]\n *   ]\n * }\n * ```\n *\n * The three parallel arrays (`haystack_session_ids` / `haystack_dates`\n * / `haystack_sessions`) are zipped by index; one date applies to every\n * turn in its session.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLongMemEvalOptions {\n  /** Local path to the dataset JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Which release this file is. `'S'` (~115 sessions) is the default target. */\n  readonly variant?: 'S' | 'M';\n  /** When set, keep only cases whose mapped ability is in this list. */\n  readonly abilities?: ReadonlyArray<MemoryEvalAbility>;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LongMemEval JSON file and return a fully-materialised\n * {@link Dataset} of {@link MemoryEvalInput} cases scored against the\n * reference `answer` string.\n *\n * @stable\n */\nexport async function loadLongMemEvalDataset(\n  options: LoadLongMemEvalOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLongMemEval(text, options.abilities);\n  const variantSuffix = options.variant !== undefined ? `_${options.variant.toLowerCase()}` : '';\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? `longmemeval${variantSuffix}`,\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLongMemEval(\n  text: string,\n  abilities?: ReadonlyArray<MemoryEvalAbility>,\n): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LongMemEval dataset must be a JSON array of questions.');\n  }\n  const rows = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < rows.length; i++) {\n    const row = rows[i];\n    if (row === null || typeof row !== 'object') {\n      throw new Error(`[graphorin/evals] LongMemEval row ${i} is not an object.`);\n    }\n    const mapped = mapRow(row as Record<string, unknown>, i);\n    if (abilities !== undefined && !abilities.includes(mapped.input.ability ?? 'info-extraction')) {\n      continue;\n    }\n    out.push(mapped);\n  }\n  return out;\n}\n\nfunction mapRow(row: Record<string, unknown>, index: number): Case<MemoryEvalInput, string> {\n  const question = asString(row.question);\n  if (question === undefined) {\n    throw new Error(`[graphorin/evals] LongMemEval row ${index} is missing a 'question'.`);\n  }\n  const questionId = asString(row.question_id) ?? `longmemeval-${index}`;\n  const questionType = asString(row.question_type) ?? '';\n  const askedAt = asString(row.question_date);\n  const ability = mapAbility(questionType, questionId);\n\n  const rawSessions = asArray(row.haystack_sessions);\n  const sessionIds = asArray(row.haystack_session_ids);\n  const sessionDates = asArray(row.haystack_dates);\n  const haystackSessions: MemoryEvalSession[] = [];\n  for (let s = 0; s < rawSessions.length; s++) {\n    const rawTurns = asArray(rawSessions[s]);\n    const date = asString(sessionDates[s]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of rawTurns) {\n      if (t === null || typeof t !== 'object') continue;\n      const content = asString((t as Record<string, unknown>).content);\n      if (content === undefined) continue;\n      const role = (t as Record<string, unknown>).role === 'assistant' ? 'assistant' : 'user';\n      turns.push({ role, content, ...(date !== undefined ? { timestamp: date } : {}) });\n    }\n    haystackSessions.push({ id: asString(sessionIds[s]) ?? `session-${s}`, turns });\n  }\n\n  return {\n    id: questionId,\n    input: {\n      haystackSessions,\n      question,\n      ...(askedAt !== undefined ? { askedAt } : {}),\n      ability,\n    },\n    expected: asString(row.answer) ?? '',\n    metadata: { datasetName: 'longmemeval', ability, questionType },\n  };\n}\n\n/** Maps a LongMemEval `question_type` / `_abs` suffix onto a {@link MemoryEvalAbility}. */\nfunction mapAbility(questionType: string, questionId: string): MemoryEvalAbility {\n  const qt = questionType.toLowerCase();\n  if (questionId.endsWith('_abs') || qt.includes('abstention')) return 'abstention';\n  if (qt.includes('temporal')) return 'temporal';\n  if (qt.includes('knowledge-update') || qt.includes('knowledge_update')) return 'knowledge-update';\n  if (qt.includes('multi-session') || qt.includes('multi_session')) return 'multi-session';\n  return 'info-extraction';\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA4DA,eAAsB,uBACpB,SAC2C;CAE3C,MAAM,QAAQ,iBADD,MAAM,SAAS,QAAQ,MAAM,OAAO,EACZ,QAAQ,UAAU;CACvD,MAAM,gBAAgB,QAAQ,YAAY,SAAY,IAAI,QAAQ,QAAQ,aAAa,KAAK;AAC5F,QAAO;EACL;EACA,UAAU;GACR,MAAM,QAAQ,QAAQ,cAAc;GACpC,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,iBACd,MACA,WAC8C;CAC9C,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,2EAA2E;CAE7F,MAAM,OAAO;CACb,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,KAAK,QAAQ,KAAK;EACpC,MAAM,MAAM,KAAK;AACjB,MAAI,QAAQ,QAAQ,OAAO,QAAQ,SACjC,OAAM,IAAI,MAAM,qCAAqC,EAAE,oBAAoB;EAE7E,MAAM,SAAS,OAAO,KAAgC,EAAE;AACxD,MAAI,cAAc,UAAa,CAAC,UAAU,SAAS,OAAO,MAAM,WAAW,kBAAkB,CAC3F;AAEF,MAAI,KAAK,OAAO;;AAElB,QAAO;;AAGT,SAAS,OAAO,KAA8B,OAA8C;CAC1F,MAAM,WAAW,SAAS,IAAI,SAAS;AACvC,KAAI,aAAa,OACf,OAAM,IAAI,MAAM,qCAAqC,MAAM,2BAA2B;CAExF,MAAM,aAAa,SAAS,IAAI,YAAY,IAAI,eAAe;CAC/D,MAAM,eAAe,SAAS,IAAI,cAAc,IAAI;CACpD,MAAM,UAAU,SAAS,IAAI,cAAc;CAC3C,MAAM,UAAU,WAAW,cAAc,WAAW;CAEpD,MAAM,cAAc,QAAQ,IAAI,kBAAkB;CAClD,MAAM,aAAa,QAAQ,IAAI,qBAAqB;CACpD,MAAM,eAAe,QAAQ,IAAI,eAAe;CAChD,MAAMC,mBAAwC,EAAE;AAChD,MAAK,IAAI,IAAI,GAAG,IAAI,YAAY,QAAQ,KAAK;EAC3C,MAAM,WAAW,QAAQ,YAAY,GAAG;EACxC,MAAM,OAAO,SAAS,aAAa,GAAG;EACtC,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,UAAU;AACxB,OAAI,MAAM,QAAQ,OAAO,MAAM,SAAU;GACzC,MAAM,UAAU,SAAU,EAA8B,QAAQ;AAChE,OAAI,YAAY,OAAW;GAC3B,MAAM,OAAQ,EAA8B,SAAS,cAAc,cAAc;AACjF,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAAG,CAAC;;AAEnF,mBAAiB,KAAK;GAAE,IAAI,SAAS,WAAW,GAAG,IAAI,WAAW;GAAK;GAAO,CAAC;;AAGjF,QAAO;EACL,IAAI;EACJ,OAAO;GACL;GACA;GACA,GAAI,YAAY,SAAY,EAAE,SAAS,GAAG,EAAE;GAC5C;GACD;EACD,UAAU,SAAS,IAAI,OAAO,IAAI;EAClC,UAAU;GAAE,aAAa;GAAe;GAAS;GAAc;EAChE;;;AAIH,SAAS,WAAW,cAAsB,YAAuC;CAC/E,MAAM,KAAK,aAAa,aAAa;AACrC,KAAI,WAAW,SAAS,OAAO,IAAI,GAAG,SAAS,aAAa,CAAE,QAAO;AACrE,KAAI,GAAG,SAAS,WAAW,CAAE,QAAO;AACpC,KAAI,GAAG,SAAS,mBAAmB,IAAI,GAAG,SAAS,mBAAmB,CAAE,QAAO;AAC/E,KAAI,GAAG,SAAS,gBAAgB,IAAI,GAAG,SAAS,gBAAgB,CAAE,QAAO;AACzE,QAAO;;AAGT,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE"}
+{"version":3,"file":"longmemeval.js","names":["parsed: unknown","out: Case<MemoryEvalInput, string>[]","haystackSessions: MemoryEvalSession[]","turns: MemoryEvalTurn[]"],"sources":["../../src/loaders/longmemeval.ts"],"sourcesContent":["/**\n * LongMemEval dataset loader (ICLR 2025, arXiv:2410.10813 -\n * https://github.com/xiaowu0162/LongMemEval). LongMemEval ships as a\n * JSON **array** of question objects:\n *\n * ```jsonc\n * {\n *   \"question_id\": \"abc_abs\",          // `_abs` suffix marks abstention\n *   \"question_type\": \"temporal-reasoning\",\n *   \"question\": \"Where did I live in 2021?\",\n *   \"answer\": \"Berlin\",\n *   \"question_date\": \"2023/05/20 (Sat) 14:00\",\n *   \"haystack_session_ids\": [\"s1\", \"s2\"],\n *   \"haystack_dates\": [\"2021/03/01 ...\", \"2023/01/05 ...\"],\n *   \"haystack_sessions\": [\n *     [ { \"role\": \"user\", \"content\": \"...\" }, { \"role\": \"assistant\", \"content\": \"...\", \"has_answer\": true } ],\n *     [ ... ]\n *   ]\n * }\n * ```\n *\n * The three parallel arrays (`haystack_session_ids` / `haystack_dates`\n * / `haystack_sessions`) are zipped by index; one date applies to every\n * turn in its session.\n *\n * @packageDocumentation\n */\n\nimport { readFile } from 'node:fs/promises';\n\nimport type { Case, Dataset } from '@graphorin/observability/eval';\n\nimport type {\n  MemoryEvalAbility,\n  MemoryEvalInput,\n  MemoryEvalSession,\n  MemoryEvalTurn,\n} from './memory-eval.js';\n\n/** @stable */\nexport interface LoadLongMemEvalOptions {\n  /** Local path to the dataset JSON (under `benchmarks/.datasets/`). */\n  readonly path: string;\n  /** Which release this file is. `'S'` (~115 sessions) is the default target. */\n  readonly variant?: 'S' | 'M';\n  /** When set, keep only cases whose mapped ability is in this list. */\n  readonly abilities?: ReadonlyArray<MemoryEvalAbility>;\n  /** Optional dataset name surfaced in `Dataset.metadata.name`. */\n  readonly name?: string;\n  /** Optional description surfaced in `Dataset.metadata.description`. */\n  readonly description?: string;\n}\n\n/**\n * Read a LongMemEval JSON file and return a fully-materialised\n * {@link Dataset} of {@link MemoryEvalInput} cases scored against the\n * reference `answer` string.\n *\n * @stable\n */\nexport async function loadLongMemEvalDataset(\n  options: LoadLongMemEvalOptions,\n): Promise<Dataset<MemoryEvalInput, string>> {\n  const text = await readFile(options.path, 'utf8');\n  const cases = parseLongMemEval(text, options.abilities);\n  const variantSuffix = options.variant !== undefined ? `_${options.variant.toLowerCase()}` : '';\n  return {\n    cases,\n    metadata: {\n      name: options.name ?? `longmemeval${variantSuffix}`,\n      ...(options.description !== undefined ? { description: options.description } : {}),\n      createdAt: new Date(),\n    },\n  };\n}\n\n/**\n * Pure parser. Exported so tests can exercise the mapping without\n * touching the filesystem.\n *\n * @stable\n */\nexport function parseLongMemEval(\n  text: string,\n  abilities?: ReadonlyArray<MemoryEvalAbility>,\n): ReadonlyArray<Case<MemoryEvalInput, string>> {\n  const parsed: unknown = JSON.parse(text);\n  if (!Array.isArray(parsed)) {\n    throw new Error('[graphorin/evals] LongMemEval dataset must be a JSON array of questions.');\n  }\n  const rows = parsed as ReadonlyArray<unknown>;\n  const out: Case<MemoryEvalInput, string>[] = [];\n  for (let i = 0; i < rows.length; i++) {\n    const row = rows[i];\n    if (row === null || typeof row !== 'object') {\n      throw new Error(`[graphorin/evals] LongMemEval row ${i} is not an object.`);\n    }\n    const mapped = mapRow(row as Record<string, unknown>, i);\n    if (abilities !== undefined && !abilities.includes(mapped.input.ability ?? 'info-extraction')) {\n      continue;\n    }\n    out.push(mapped);\n  }\n  return out;\n}\n\nfunction mapRow(row: Record<string, unknown>, index: number): Case<MemoryEvalInput, string> {\n  const question = asString(row.question);\n  if (question === undefined) {\n    throw new Error(`[graphorin/evals] LongMemEval row ${index} is missing a 'question'.`);\n  }\n  const questionId = asString(row.question_id) ?? `longmemeval-${index}`;\n  const questionType = asString(row.question_type) ?? '';\n  const askedAt = asString(row.question_date);\n  const ability = mapAbility(questionType, questionId);\n\n  const rawSessions = asArray(row.haystack_sessions);\n  const sessionIds = asArray(row.haystack_session_ids);\n  const sessionDates = asArray(row.haystack_dates);\n  const haystackSessions: MemoryEvalSession[] = [];\n  for (let s = 0; s < rawSessions.length; s++) {\n    const rawTurns = asArray(rawSessions[s]);\n    const date = asString(sessionDates[s]);\n    const turns: MemoryEvalTurn[] = [];\n    for (const t of rawTurns) {\n      if (t === null || typeof t !== 'object') continue;\n      const content = asString((t as Record<string, unknown>).content);\n      if (content === undefined) continue;\n      const role = (t as Record<string, unknown>).role === 'assistant' ? 'assistant' : 'user';\n      turns.push({ role, content, ...(date !== undefined ? { timestamp: date } : {}) });\n    }\n    haystackSessions.push({ id: asString(sessionIds[s]) ?? `session-${s}`, turns });\n  }\n\n  return {\n    id: questionId,\n    input: {\n      haystackSessions,\n      question,\n      ...(askedAt !== undefined ? { askedAt } : {}),\n      ability,\n    },\n    expected: asString(row.answer) ?? '',\n    metadata: { datasetName: 'longmemeval', ability, questionType },\n  };\n}\n\n/** Maps a LongMemEval `question_type` / `_abs` suffix onto a {@link MemoryEvalAbility}. */\nfunction mapAbility(questionType: string, questionId: string): MemoryEvalAbility {\n  const qt = questionType.toLowerCase();\n  if (questionId.endsWith('_abs') || qt.includes('abstention')) return 'abstention';\n  if (qt.includes('temporal')) return 'temporal';\n  if (qt.includes('knowledge-update') || qt.includes('knowledge_update')) return 'knowledge-update';\n  if (qt.includes('multi-session') || qt.includes('multi_session')) return 'multi-session';\n  return 'info-extraction';\n}\n\nfunction asString(value: unknown): string | undefined {\n  return typeof value === 'string' ? value : undefined;\n}\n\nfunction asArray(value: unknown): ReadonlyArray<unknown> {\n  return Array.isArray(value) ? (value as ReadonlyArray<unknown>) : [];\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AA4DA,eAAsB,uBACpB,SAC2C;CAE3C,MAAM,QAAQ,iBADD,MAAM,SAAS,QAAQ,MAAM,OAAO,EACZ,QAAQ,UAAU;CACvD,MAAM,gBAAgB,QAAQ,YAAY,SAAY,IAAI,QAAQ,QAAQ,aAAa,KAAK;AAC5F,QAAO;EACL;EACA,UAAU;GACR,MAAM,QAAQ,QAAQ,cAAc;GACpC,GAAI,QAAQ,gBAAgB,SAAY,EAAE,aAAa,QAAQ,aAAa,GAAG,EAAE;GACjF,2BAAW,IAAI,MAAM;GACtB;EACF;;;;;;;;AASH,SAAgB,iBACd,MACA,WAC8C;CAC9C,MAAMA,SAAkB,KAAK,MAAM,KAAK;AACxC,KAAI,CAAC,MAAM,QAAQ,OAAO,CACxB,OAAM,IAAI,MAAM,2EAA2E;CAE7F,MAAM,OAAO;CACb,MAAMC,MAAuC,EAAE;AAC/C,MAAK,IAAI,IAAI,GAAG,IAAI,KAAK,QAAQ,KAAK;EACpC,MAAM,MAAM,KAAK;AACjB,MAAI,QAAQ,QAAQ,OAAO,QAAQ,SACjC,OAAM,IAAI,MAAM,qCAAqC,EAAE,oBAAoB;EAE7E,MAAM,SAAS,OAAO,KAAgC,EAAE;AACxD,MAAI,cAAc,UAAa,CAAC,UAAU,SAAS,OAAO,MAAM,WAAW,kBAAkB,CAC3F;AAEF,MAAI,KAAK,OAAO;;AAElB,QAAO;;AAGT,SAAS,OAAO,KAA8B,OAA8C;CAC1F,MAAM,WAAW,SAAS,IAAI,SAAS;AACvC,KAAI,aAAa,OACf,OAAM,IAAI,MAAM,qCAAqC,MAAM,2BAA2B;CAExF,MAAM,aAAa,SAAS,IAAI,YAAY,IAAI,eAAe;CAC/D,MAAM,eAAe,SAAS,IAAI,cAAc,IAAI;CACpD,MAAM,UAAU,SAAS,IAAI,cAAc;CAC3C,MAAM,UAAU,WAAW,cAAc,WAAW;CAEpD,MAAM,cAAc,QAAQ,IAAI,kBAAkB;CAClD,MAAM,aAAa,QAAQ,IAAI,qBAAqB;CACpD,MAAM,eAAe,QAAQ,IAAI,eAAe;CAChD,MAAMC,mBAAwC,EAAE;AAChD,MAAK,IAAI,IAAI,GAAG,IAAI,YAAY,QAAQ,KAAK;EAC3C,MAAM,WAAW,QAAQ,YAAY,GAAG;EACxC,MAAM,OAAO,SAAS,aAAa,GAAG;EACtC,MAAMC,QAA0B,EAAE;AAClC,OAAK,MAAM,KAAK,UAAU;AACxB,OAAI,MAAM,QAAQ,OAAO,MAAM,SAAU;GACzC,MAAM,UAAU,SAAU,EAA8B,QAAQ;AAChE,OAAI,YAAY,OAAW;GAC3B,MAAM,OAAQ,EAA8B,SAAS,cAAc,cAAc;AACjF,SAAM,KAAK;IAAE;IAAM;IAAS,GAAI,SAAS,SAAY,EAAE,WAAW,MAAM,GAAG,EAAE;IAAG,CAAC;;AAEnF,mBAAiB,KAAK;GAAE,IAAI,SAAS,WAAW,GAAG,IAAI,WAAW;GAAK;GAAO,CAAC;;AAGjF,QAAO;EACL,IAAI;EACJ,OAAO;GACL;GACA;GACA,GAAI,YAAY,SAAY,EAAE,SAAS,GAAG,EAAE;GAC5C;GACD;EACD,UAAU,SAAS,IAAI,OAAO,IAAI;EAClC,UAAU;GAAE,aAAa;GAAe;GAAS;GAAc;EAChE;;;AAIH,SAAS,WAAW,cAAsB,YAAuC;CAC/E,MAAM,KAAK,aAAa,aAAa;AACrC,KAAI,WAAW,SAAS,OAAO,IAAI,GAAG,SAAS,aAAa,CAAE,QAAO;AACrE,KAAI,GAAG,SAAS,WAAW,CAAE,QAAO;AACpC,KAAI,GAAG,SAAS,mBAAmB,IAAI,GAAG,SAAS,mBAAmB,CAAE,QAAO;AAC/E,KAAI,GAAG,SAAS,gBAAgB,IAAI,GAAG,SAAS,gBAAgB,CAAE,QAAO;AACzE,QAAO;;AAGT,SAAS,SAAS,OAAoC;AACpD,QAAO,OAAO,UAAU,WAAW,QAAQ;;AAG7C,SAAS,QAAQ,OAAwC;AACvD,QAAO,MAAM,QAAQ,MAAM,GAAI,QAAmC,EAAE"}

package/dist/loaders/memory-eval.d.ts

@@ -9,7 +9,7 @@
  * reference answer string, graded by the `llmJudge` "J" scorer.
  *
  * This module is intentionally **type-only** and carries no dependency
- * on `@graphorin/memory` — the concrete system-under-test agent lives
+ * on `@graphorin/memory` - the concrete system-under-test agent lives
  * in the benchmark package so `@graphorin/evals` stays a generic
  * harness.
  *

package/dist/package.js

@@ -0,0 +1,6 @@
+//#region package.json
+var version = "0.6.1";
+
+//#endregion
+export { version };
+//# sourceMappingURL=package.js.map

package/dist/package.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"package.js","names":[],"sources":["../package.json"],"sourcesContent":["{\n  \"name\": \"@graphorin/evals\",\n  \"version\": \"0.6.1\",\n  \"description\": \"Eval framework for the Graphorin framework. Ships scorer libraries (`code/` exact-match, regex, JSON path, predicates; `llm/` LLM-judge; `prebuilt/` toxicity / factuality / helpfulness), dataset loaders (JSONL, CSV, from-traces, generic iterable), reporters (terminal, markdown, JSON, JUnit XML), a parallel runner with `concurrency` config, and regression detection that compares the current run against a stored baseline. Builds on the eval primitives shipped from `@graphorin/observability` (RB-17 / DEC-152 - full orchestrator lives here, post-MVP, decoupled from the core observability bundle so consumers do not pay the dataset / reporter cost when only the inline runner is needed). Created and maintained by Oleksiy Stepurenko.\",\n  \"license\": \"MIT\",\n  \"author\": \"Oleksiy Stepurenko\",\n  \"homepage\": \"https://github.com/o-stepper/graphorin/tree/main/packages/evals\",\n  \"repository\": {\n    \"type\": \"git\",\n    \"url\": \"git+https://github.com/o-stepper/graphorin.git\",\n    \"directory\": \"packages/evals\"\n  },\n  \"bugs\": {\n    \"url\": \"https://github.com/o-stepper/graphorin/issues\"\n  },\n  \"keywords\": [\n    \"graphorin\",\n    \"ai\",\n    \"agents\",\n    \"framework\",\n    \"evals\",\n    \"evaluation\",\n    \"scorers\",\n    \"llm-judge\",\n    \"datasets\",\n    \"reporters\",\n    \"regression-detection\"\n  ],\n  \"type\": \"module\",\n  \"engines\": {\n    \"node\": \">=22.0.0\"\n  },\n  \"main\": \"./dist/index.js\",\n  \"module\": \"./dist/index.js\",\n  \"types\": \"./dist/index.d.ts\",\n  \"sideEffects\": false,\n  \"exports\": {\n    \".\": {\n      \"types\": \"./dist/index.d.ts\",\n      \"import\": \"./dist/index.js\"\n    },\n    \"./scorers\": {\n      \"types\": \"./dist/scorers/index.d.ts\",\n      \"import\": \"./dist/scorers/index.js\"\n    },\n    \"./loaders\": {\n      \"types\": \"./dist/loaders/index.d.ts\",\n      \"import\": \"./dist/loaders/index.js\"\n    },\n    \"./reporters\": {\n      \"types\": \"./dist/reporters/index.d.ts\",\n      \"import\": \"./dist/reporters/index.js\"\n    },\n    \"./cli\": {\n      \"types\": \"./dist/cli/index.d.ts\",\n      \"import\": \"./dist/cli/index.js\"\n    },\n    \"./package.json\": \"./package.json\"\n  },\n  \"files\": [\n    \"dist\",\n    \"README.md\",\n    \"CHANGELOG.md\",\n    \"LICENSE\"\n  ],\n  \"scripts\": {\n    \"build\": \"tsdown\",\n    \"typecheck\": \"tsc --noEmit && tsc -p tsconfig.tests.json\",\n    \"test\": \"vitest run\",\n    \"lint\": \"biome check .\",\n    \"clean\": \"rimraf dist .turbo *.tsbuildinfo\"\n  },\n  \"dependencies\": {\n    \"@graphorin/core\": \"workspace:*\",\n    \"@graphorin/observability\": \"workspace:*\"\n  },\n  \"publishConfig\": {\n    \"access\": \"public\",\n    \"provenance\": true\n  }\n}\n"],"mappings":";cAEa"}

package/dist/regression.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"regression.d.ts","names":[],"sources":["../src/regression.ts"],"sourcesContent":[],"mappings":";;;;;;;;;AAiCG,iBAJa,iBAIb,CAAA,CAAA,EAAA,CAAA,CAAA,CAAA,OAAA,EAHQ,UAGR,CAHmB,CAGnB,EAHsB,CAGtB,CAAA,EAAA,QAAA,EAFS,UAET,CAFoB,CAEpB,EAFuB,CAEvB,CAAA,EAAA,OAAA,CAAA,EADQ,iBACR,CAAA,EAAA,gBAAA,CAAiB,CAAjB,EAAoB,CAApB,CAAA"}
+{"version":3,"file":"regression.d.ts","names":[],"sources":["../src/regression.ts"],"sourcesContent":[],"mappings":";;;;;;;;;AAkCG,iBAJa,iBAIb,CAAA,CAAA,EAAA,CAAA,CAAA,CAAA,OAAA,EAHQ,UAGR,CAHmB,CAGnB,EAHsB,CAGtB,CAAA,EAAA,QAAA,EAFS,UAET,CAFoB,CAEpB,EAFuB,CAEvB,CAAA,EAAA,OAAA,CAAA,EADQ,iBACR,CAAA,EAAA,gBAAA,CAAiB,CAAjB,EAAoB,CAApB,CAAA"}

package/dist/regression.js

@@ -1,4 +1,13 @@
+import { pairedPassSignificance, passByBaseCase } from "./stats.js";
+
 //#region src/regression.ts
+/**
+* Compares two `EvalReport`s - typically the freshly-produced report
+* vs a stored baseline - and lists every metric that regressed beyond
+* the configured tolerance.
+*
+* @packageDocumentation
+*/
 const DEFAULT_PASS_RATE_DROP_PCT = 5;
 const DEFAULT_AVG_SCORE_DROP = .05;
 const DEFAULT_AVG_DURATION_INCREASE_MS = Number.POSITIVE_INFINITY;
@@ -15,15 +24,21 @@ function detectRegressions(current, baseline, options = {}) {
 	const currentPassRate = total(current) === 0 ? 0 : current.summary.passed / total(current);
 	const baselinePassRate = total(baseline) === 0 ? 0 : baseline.summary.passed / total(baseline);
 	const passRateDropPct = (baselinePassRate - currentPassRate) * 100;
-	if (passRateDropPct > maxPassRateDropPct) findings.push({
-		kind: "pass-rate-drop",
-		message: `pass rate dropped by ${passRateDropPct.toFixed(2)}% (${(currentPassRate * 100).toFixed(2)}% < baseline ${(baselinePassRate * 100).toFixed(2)}% − tolerance ${maxPassRateDropPct.toFixed(2)}%).`,
-		delta: -passRateDropPct
-	});
+	if (passRateDropPct > maxPassRateDropPct) {
+		const significance = pairedPassSignificance(passByBaseCase(caseOutcomes(current)), passByBaseCase(caseOutcomes(baseline)));
+		const havePairs = significance.pairs > 0;
+		const alpha = options.significanceAlpha ?? .05;
+		if (!(options.requireSignificance === true && havePairs && significance.pValue >= alpha)) findings.push({
+			kind: "pass-rate-drop",
+			message: `pass rate dropped by ${passRateDropPct.toFixed(2)}% (${(currentPassRate * 100).toFixed(2)}% < baseline ${(baselinePassRate * 100).toFixed(2)}% - tolerance ${maxPassRateDropPct.toFixed(2)}%)` + (havePairs ? ` [paired: ${significance.regressed} regressed / ${significance.improved} improved over ${significance.pairs} shared case(s), McNemar p=${significance.pValue.toFixed(4)}]` : "") + ".",
+			delta: -passRateDropPct,
+			...havePairs ? { pValue: significance.pValue } : {}
+		});
+	}
 	const durationDelta = current.summary.avgDurationMs - baseline.summary.avgDurationMs;
 	if (durationDelta > maxAvgDurationIncreaseMs) findings.push({
 		kind: "avg-duration-increase",
-		message: `avg duration increased by ${durationDelta.toFixed(2)} ms (${current.summary.avgDurationMs.toFixed(2)} ms > baseline ${baseline.summary.avgDurationMs.toFixed(2)} ms − tolerance ${maxAvgDurationIncreaseMs} ms).`,
+		message: `avg duration increased by ${durationDelta.toFixed(2)} ms (${current.summary.avgDurationMs.toFixed(2)} ms > baseline ${baseline.summary.avgDurationMs.toFixed(2)} ms - tolerance ${maxAvgDurationIncreaseMs} ms).`,
 		delta: durationDelta
 	});
 	for (const [scorer, baselineRow] of Object.entries(baseline.summary.byScorer)) {
@@ -42,7 +57,7 @@ function detectRegressions(current, baseline, options = {}) {
 			if (drop > maxAvgScoreDrop) findings.push({
 				kind: "avg-score-drop",
 				scorer,
-				message: `'${scorer}' avg score dropped by ${drop.toFixed(4)} (${currentRow.avgScore.toFixed(4)} < baseline ${baselineRow.avgScore.toFixed(4)} − tolerance ${maxAvgScoreDrop}).`,
+				message: `'${scorer}' avg score dropped by ${drop.toFixed(4)} (${currentRow.avgScore.toFixed(4)} < baseline ${baselineRow.avgScore.toFixed(4)} - tolerance ${maxAvgScoreDrop}).`,
 				delta: -drop
 			});
 		}
@@ -57,6 +72,13 @@ function detectRegressions(current, baseline, options = {}) {
 function total(r) {
 	return r.summary.total;
 }
+/** Per-iteration case outcomes (a case passes when every scorer passed). */
+function caseOutcomes(r) {
+	return r.results.map((c) => ({
+		caseId: c.caseId,
+		pass: c.scores.every((s) => s.result.pass)
+	}));
+}
 
 //#endregion
 export { detectRegressions };

package/dist/regression.js.map

@@ -1 +1 @@
-{"version":3,"file":"regression.js","names":["findings: RegressionFinding[]"],"sources":["../src/regression.ts"],"sourcesContent":["/**\n * Compares two `EvalReport`s — typically the freshly-produced report\n * vs a stored baseline — and lists every metric that regressed beyond\n * the configured tolerance.\n *\n * @packageDocumentation\n */\n\nimport type {\n  EvalReport,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n} from './types.js';\n\nconst DEFAULT_PASS_RATE_DROP_PCT = 5;\nconst DEFAULT_AVG_SCORE_DROP = 0.05;\n// EB-4: the avg-duration gate is OPT-IN (default off). Absolute wall-clock\n// budgets are environment-sensitive — a baseline recorded on a fast\n// workstation vs a loaded CI runner, or real LLM-latency jitter of whole\n// seconds, would flag spurious \"regressions\". Callers that genuinely want a\n// duration budget pass an explicit finite `maxAvgDurationIncreaseMs`.\nconst DEFAULT_AVG_DURATION_INCREASE_MS = Number.POSITIVE_INFINITY;\n\n/**\n * Detect regressions between `current` and `baseline` reports.\n *\n * @stable\n */\nexport function detectRegressions<I, O>(\n  current: EvalReport<I, O>,\n  baseline: EvalReport<I, O>,\n  options: RegressionOptions = {},\n): RegressionReport<I, O> {\n  const maxPassRateDropPct = options.maxPassRateDropPct ?? DEFAULT_PASS_RATE_DROP_PCT;\n  const maxAvgScoreDrop = options.maxAvgScoreDrop ?? DEFAULT_AVG_SCORE_DROP;\n  const maxAvgDurationIncreaseMs =\n    options.maxAvgDurationIncreaseMs ?? DEFAULT_AVG_DURATION_INCREASE_MS;\n  const findings: RegressionFinding[] = [];\n\n  const currentPassRate = total(current) === 0 ? 0 : current.summary.passed / total(current);\n  const baselinePassRate = total(baseline) === 0 ? 0 : baseline.summary.passed / total(baseline);\n  const passRateDropPct = (baselinePassRate - currentPassRate) * 100;\n  if (passRateDropPct > maxPassRateDropPct) {\n    findings.push({\n      kind: 'pass-rate-drop',\n      message:\n        `pass rate dropped by ${passRateDropPct.toFixed(2)}% ` +\n        `(${(currentPassRate * 100).toFixed(2)}% < baseline ${(baselinePassRate * 100).toFixed(2)}% ` +\n        `− tolerance ${maxPassRateDropPct.toFixed(2)}%).`,\n      delta: -passRateDropPct,\n    });\n  }\n\n  const durationDelta = current.summary.avgDurationMs - baseline.summary.avgDurationMs;\n  if (durationDelta > maxAvgDurationIncreaseMs) {\n    findings.push({\n      kind: 'avg-duration-increase',\n      message:\n        `avg duration increased by ${durationDelta.toFixed(2)} ms ` +\n        `(${current.summary.avgDurationMs.toFixed(2)} ms > baseline ` +\n        `${baseline.summary.avgDurationMs.toFixed(2)} ms − tolerance ${maxAvgDurationIncreaseMs} ms).`,\n      delta: durationDelta,\n    });\n  }\n\n  for (const [scorer, baselineRow] of Object.entries(baseline.summary.byScorer)) {\n    const currentRow = current.summary.byScorer[scorer];\n    if (currentRow === undefined) {\n      findings.push({\n        kind: 'scorer-removed',\n        scorer,\n        message: `scorer '${scorer}' is in the baseline but missing from the current run.`,\n        delta: Number.NaN,\n      });\n      continue;\n    }\n    if (typeof baselineRow.avgScore === 'number' && typeof currentRow.avgScore === 'number') {\n      const drop = baselineRow.avgScore - currentRow.avgScore;\n      if (drop > maxAvgScoreDrop) {\n        findings.push({\n          kind: 'avg-score-drop',\n          scorer,\n          message:\n            `'${scorer}' avg score dropped by ${drop.toFixed(4)} ` +\n            `(${currentRow.avgScore.toFixed(4)} < baseline ${baselineRow.avgScore.toFixed(4)} ` +\n            `− tolerance ${maxAvgScoreDrop}).`,\n          delta: -drop,\n        });\n      }\n    }\n  }\n\n  return {\n    hasRegressions: findings.length > 0,\n    findings,\n    current,\n    baseline,\n  };\n}\n\nfunction total<I, O>(r: EvalReport<I, O>): number {\n  return r.summary.total;\n}\n"],"mappings":";AAeA,MAAM,6BAA6B;AACnC,MAAM,yBAAyB;AAM/B,MAAM,mCAAmC,OAAO;;;;;;AAOhD,SAAgB,kBACd,SACA,UACA,UAA6B,EAAE,EACP;CACxB,MAAM,qBAAqB,QAAQ,sBAAsB;CACzD,MAAM,kBAAkB,QAAQ,mBAAmB;CACnD,MAAM,2BACJ,QAAQ,4BAA4B;CACtC,MAAMA,WAAgC,EAAE;CAExC,MAAM,kBAAkB,MAAM,QAAQ,KAAK,IAAI,IAAI,QAAQ,QAAQ,SAAS,MAAM,QAAQ;CAC1F,MAAM,mBAAmB,MAAM,SAAS,KAAK,IAAI,IAAI,SAAS,QAAQ,SAAS,MAAM,SAAS;CAC9F,MAAM,mBAAmB,mBAAmB,mBAAmB;AAC/D,KAAI,kBAAkB,mBACpB,UAAS,KAAK;EACZ,MAAM;EACN,SACE,wBAAwB,gBAAgB,QAAQ,EAAE,CAAC,MAC9C,kBAAkB,KAAK,QAAQ,EAAE,CAAC,gBAAgB,mBAAmB,KAAK,QAAQ,EAAE,CAAC,gBAC3E,mBAAmB,QAAQ,EAAE,CAAC;EAC/C,OAAO,CAAC;EACT,CAAC;CAGJ,MAAM,gBAAgB,QAAQ,QAAQ,gBAAgB,SAAS,QAAQ;AACvE,KAAI,gBAAgB,yBAClB,UAAS,KAAK;EACZ,MAAM;EACN,SACE,6BAA6B,cAAc,QAAQ,EAAE,CAAC,OAClD,QAAQ,QAAQ,cAAc,QAAQ,EAAE,CAAC,iBAC1C,SAAS,QAAQ,cAAc,QAAQ,EAAE,CAAC,kBAAkB,yBAAyB;EAC1F,OAAO;EACR,CAAC;AAGJ,MAAK,MAAM,CAAC,QAAQ,gBAAgB,OAAO,QAAQ,SAAS,QAAQ,SAAS,EAAE;EAC7E,MAAM,aAAa,QAAQ,QAAQ,SAAS;AAC5C,MAAI,eAAe,QAAW;AAC5B,YAAS,KAAK;IACZ,MAAM;IACN;IACA,SAAS,WAAW,OAAO;IAC3B,OAAO;IACR,CAAC;AACF;;AAEF,MAAI,OAAO,YAAY,aAAa,YAAY,OAAO,WAAW,aAAa,UAAU;GACvF,MAAM,OAAO,YAAY,WAAW,WAAW;AAC/C,OAAI,OAAO,gBACT,UAAS,KAAK;IACZ,MAAM;IACN;IACA,SACE,IAAI,OAAO,yBAAyB,KAAK,QAAQ,EAAE,CAAC,IAChD,WAAW,SAAS,QAAQ,EAAE,CAAC,cAAc,YAAY,SAAS,QAAQ,EAAE,CAAC,eAClE,gBAAgB;IACjC,OAAO,CAAC;IACT,CAAC;;;AAKR,QAAO;EACL,gBAAgB,SAAS,SAAS;EAClC;EACA;EACA;EACD;;AAGH,SAAS,MAAY,GAA6B;AAChD,QAAO,EAAE,QAAQ"}
+{"version":3,"file":"regression.js","names":["findings: RegressionFinding[]"],"sources":["../src/regression.ts"],"sourcesContent":["/**\n * Compares two `EvalReport`s - typically the freshly-produced report\n * vs a stored baseline - and lists every metric that regressed beyond\n * the configured tolerance.\n *\n * @packageDocumentation\n */\n\nimport { pairedPassSignificance, passByBaseCase } from './stats.js';\nimport type {\n  EvalReport,\n  RegressionFinding,\n  RegressionOptions,\n  RegressionReport,\n} from './types.js';\n\nconst DEFAULT_PASS_RATE_DROP_PCT = 5;\nconst DEFAULT_AVG_SCORE_DROP = 0.05;\n// EB-4: the avg-duration gate is OPT-IN (default off). Absolute wall-clock\n// budgets are environment-sensitive - a baseline recorded on a fast\n// workstation vs a loaded CI runner, or real LLM-latency jitter of whole\n// seconds, would flag spurious \"regressions\". Callers that genuinely want a\n// duration budget pass an explicit finite `maxAvgDurationIncreaseMs`.\nconst DEFAULT_AVG_DURATION_INCREASE_MS = Number.POSITIVE_INFINITY;\n\n/**\n * Detect regressions between `current` and `baseline` reports.\n *\n * @stable\n */\nexport function detectRegressions<I, O>(\n  current: EvalReport<I, O>,\n  baseline: EvalReport<I, O>,\n  options: RegressionOptions = {},\n): RegressionReport<I, O> {\n  const maxPassRateDropPct = options.maxPassRateDropPct ?? DEFAULT_PASS_RATE_DROP_PCT;\n  const maxAvgScoreDrop = options.maxAvgScoreDrop ?? DEFAULT_AVG_SCORE_DROP;\n  const maxAvgDurationIncreaseMs =\n    options.maxAvgDurationIncreaseMs ?? DEFAULT_AVG_DURATION_INCREASE_MS;\n  const findings: RegressionFinding[] = [];\n\n  const currentPassRate = total(current) === 0 ? 0 : current.summary.passed / total(current);\n  const baselinePassRate = total(baseline) === 0 ? 0 : baseline.summary.passed / total(baseline);\n  const passRateDropPct = (baselinePassRate - currentPassRate) * 100;\n  if (passRateDropPct > maxPassRateDropPct) {\n    // E8 (evals-05/08): the fixed tolerance is sample-size blind, so pair the\n    // shared cases and run McNemar's test. The p-value is always attached for\n    // the report; it VETOES the finding only under opt-in requireSignificance.\n    const significance = pairedPassSignificance(\n      passByBaseCase(caseOutcomes(current)),\n      passByBaseCase(caseOutcomes(baseline)),\n    );\n    const havePairs = significance.pairs > 0;\n    const alpha = options.significanceAlpha ?? 0.05;\n    const vetoed =\n      options.requireSignificance === true && havePairs && significance.pValue >= alpha;\n    if (!vetoed) {\n      findings.push({\n        kind: 'pass-rate-drop',\n        message:\n          `pass rate dropped by ${passRateDropPct.toFixed(2)}% ` +\n          `(${(currentPassRate * 100).toFixed(2)}% < baseline ${(baselinePassRate * 100).toFixed(2)}% ` +\n          `- tolerance ${maxPassRateDropPct.toFixed(2)}%)` +\n          (havePairs\n            ? ` [paired: ${significance.regressed} regressed / ${significance.improved} improved over ${significance.pairs} shared case(s), McNemar p=${significance.pValue.toFixed(4)}]`\n            : '') +\n          '.',\n        delta: -passRateDropPct,\n        ...(havePairs ? { pValue: significance.pValue } : {}),\n      });\n    }\n  }\n\n  const durationDelta = current.summary.avgDurationMs - baseline.summary.avgDurationMs;\n  if (durationDelta > maxAvgDurationIncreaseMs) {\n    findings.push({\n      kind: 'avg-duration-increase',\n      message:\n        `avg duration increased by ${durationDelta.toFixed(2)} ms ` +\n        `(${current.summary.avgDurationMs.toFixed(2)} ms > baseline ` +\n        `${baseline.summary.avgDurationMs.toFixed(2)} ms - tolerance ${maxAvgDurationIncreaseMs} ms).`,\n      delta: durationDelta,\n    });\n  }\n\n  for (const [scorer, baselineRow] of Object.entries(baseline.summary.byScorer)) {\n    const currentRow = current.summary.byScorer[scorer];\n    if (currentRow === undefined) {\n      findings.push({\n        kind: 'scorer-removed',\n        scorer,\n        message: `scorer '${scorer}' is in the baseline but missing from the current run.`,\n        delta: Number.NaN,\n      });\n      continue;\n    }\n    if (typeof baselineRow.avgScore === 'number' && typeof currentRow.avgScore === 'number') {\n      const drop = baselineRow.avgScore - currentRow.avgScore;\n      if (drop > maxAvgScoreDrop) {\n        findings.push({\n          kind: 'avg-score-drop',\n          scorer,\n          message:\n            `'${scorer}' avg score dropped by ${drop.toFixed(4)} ` +\n            `(${currentRow.avgScore.toFixed(4)} < baseline ${baselineRow.avgScore.toFixed(4)} ` +\n            `- tolerance ${maxAvgScoreDrop}).`,\n          delta: -drop,\n        });\n      }\n    }\n  }\n\n  return {\n    hasRegressions: findings.length > 0,\n    findings,\n    current,\n    baseline,\n  };\n}\n\nfunction total<I, O>(r: EvalReport<I, O>): number {\n  return r.summary.total;\n}\n\n/** Per-iteration case outcomes (a case passes when every scorer passed). */\nfunction caseOutcomes<I, O>(\n  r: EvalReport<I, O>,\n): ReadonlyArray<{ readonly caseId: string; readonly pass: boolean }> {\n  return r.results.map((c) => ({\n    caseId: c.caseId,\n    pass: c.scores.every((s) => s.result.pass),\n  }));\n}\n"],"mappings":";;;;;;;;;;AAgBA,MAAM,6BAA6B;AACnC,MAAM,yBAAyB;AAM/B,MAAM,mCAAmC,OAAO;;;;;;AAOhD,SAAgB,kBACd,SACA,UACA,UAA6B,EAAE,EACP;CACxB,MAAM,qBAAqB,QAAQ,sBAAsB;CACzD,MAAM,kBAAkB,QAAQ,mBAAmB;CACnD,MAAM,2BACJ,QAAQ,4BAA4B;CACtC,MAAMA,WAAgC,EAAE;CAExC,MAAM,kBAAkB,MAAM,QAAQ,KAAK,IAAI,IAAI,QAAQ,QAAQ,SAAS,MAAM,QAAQ;CAC1F,MAAM,mBAAmB,MAAM,SAAS,KAAK,IAAI,IAAI,SAAS,QAAQ,SAAS,MAAM,SAAS;CAC9F,MAAM,mBAAmB,mBAAmB,mBAAmB;AAC/D,KAAI,kBAAkB,oBAAoB;EAIxC,MAAM,eAAe,uBACnB,eAAe,aAAa,QAAQ,CAAC,EACrC,eAAe,aAAa,SAAS,CAAC,CACvC;EACD,MAAM,YAAY,aAAa,QAAQ;EACvC,MAAM,QAAQ,QAAQ,qBAAqB;AAG3C,MAAI,EADF,QAAQ,wBAAwB,QAAQ,aAAa,aAAa,UAAU,OAE5E,UAAS,KAAK;GACZ,MAAM;GACN,SACE,wBAAwB,gBAAgB,QAAQ,EAAE,CAAC,MAC9C,kBAAkB,KAAK,QAAQ,EAAE,CAAC,gBAAgB,mBAAmB,KAAK,QAAQ,EAAE,CAAC,gBAC3E,mBAAmB,QAAQ,EAAE,CAAC,OAC5C,YACG,aAAa,aAAa,UAAU,eAAe,aAAa,SAAS,iBAAiB,aAAa,MAAM,6BAA6B,aAAa,OAAO,QAAQ,EAAE,CAAC,KACzK,MACJ;GACF,OAAO,CAAC;GACR,GAAI,YAAY,EAAE,QAAQ,aAAa,QAAQ,GAAG,EAAE;GACrD,CAAC;;CAIN,MAAM,gBAAgB,QAAQ,QAAQ,gBAAgB,SAAS,QAAQ;AACvE,KAAI,gBAAgB,yBAClB,UAAS,KAAK;EACZ,MAAM;EACN,SACE,6BAA6B,cAAc,QAAQ,EAAE,CAAC,OAClD,QAAQ,QAAQ,cAAc,QAAQ,EAAE,CAAC,iBAC1C,SAAS,QAAQ,cAAc,QAAQ,EAAE,CAAC,kBAAkB,yBAAyB;EAC1F,OAAO;EACR,CAAC;AAGJ,MAAK,MAAM,CAAC,QAAQ,gBAAgB,OAAO,QAAQ,SAAS,QAAQ,SAAS,EAAE;EAC7E,MAAM,aAAa,QAAQ,QAAQ,SAAS;AAC5C,MAAI,eAAe,QAAW;AAC5B,YAAS,KAAK;IACZ,MAAM;IACN;IACA,SAAS,WAAW,OAAO;IAC3B,OAAO;IACR,CAAC;AACF;;AAEF,MAAI,OAAO,YAAY,aAAa,YAAY,OAAO,WAAW,aAAa,UAAU;GACvF,MAAM,OAAO,YAAY,WAAW,WAAW;AAC/C,OAAI,OAAO,gBACT,UAAS,KAAK;IACZ,MAAM;IACN;IACA,SACE,IAAI,OAAO,yBAAyB,KAAK,QAAQ,EAAE,CAAC,IAChD,WAAW,SAAS,QAAQ,EAAE,CAAC,cAAc,YAAY,SAAS,QAAQ,EAAE,CAAC,eAClE,gBAAgB;IACjC,OAAO,CAAC;IACT,CAAC;;;AAKR,QAAO;EACL,gBAAgB,SAAS,SAAS;EAClC;EACA;EACA;EACD;;AAGH,SAAS,MAAY,GAA6B;AAChD,QAAO,EAAE,QAAQ;;;AAInB,SAAS,aACP,GACoE;AACpE,QAAO,EAAE,QAAQ,KAAK,OAAO;EAC3B,QAAQ,EAAE;EACV,MAAM,EAAE,OAAO,OAAO,MAAM,EAAE,OAAO,KAAK;EAC3C,EAAE"}