evalcore 2.1.0__tar.gz → 2.3.0__tar.gz

{evalcore-2.1.0 → evalcore-2.3.0}/CHANGELOG.md

@@ -6,6 +6,52 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-08-08
+
+A run now describes its own scores.
+
+### Added
+- `RunResult.graders`, a map of grader name to `models.GraderInfo`
+  (`category`, `scale`), filled in by the runner from the graders it built.
+  A `Score` names the grader that emitted it and nothing else, so a run could
+  not previously answer for itself: a `run.json` written by one CI step and
+  published by a later one needed the suite file alongside it, and without one
+  every grader reported `unknown` and a suite with a judge could not be
+  published at all. Run-grain rather than score-grain - one entry per grader,
+  not two fields repeated across a couple of hundred rows.
+- `models.GraderInfo`.
+- `register` also records the category on the decorated class as
+  `grader_category`, so a grader instance can answer for itself. The runner
+  reads it the same way it already collects `judge_version`.
+
+### Changed
+- `store.score_rows` takes the grader category and judge scale from
+  `run.graders`. `grader_types` and `judge_scales` still override it, and are
+  how a run written before this release - whose map is empty - publishes
+  correctly. `grader_lookups` is unchanged and still builds them from a suite.
+
+## [2.2.0] - 2026-08-08
+
+### Added
+- `store.ScoreExporter`, a `runtime_checkable` Protocol naming the seam
+  `JsonlOutboxExporter` already occupied. `store.py` has always said to
+  "replace this class with a database client implementing the same
+  `export_scores` method", but the contract was a docstring sentence and
+  `**kwargs`, so an implementation had to duck-type a private shape. An
+  exporter belongs in the package that owns the store it targets - it is the
+  store that knows its own column types, null policy, and transport - and
+  swapping one for another is now a constructor line at the call site, so an
+  offline run and a live one share a code path.
+
+### Fixed
+- `examples/quickstart/graders.py` still used the one-argument `register` and
+  raised `TypeError` on import, so 2.1.0 shipped with its own bundled example
+  broken. `just test` does not run the example; `just test-all` does.
+- The README's Python API example called a `.export()` that does not exist and
+  passed `RunResult`s to `compare.compare` and `render_scorecard`, which take
+  `Scorecard`s. It now runs verbatim, and shows `grader_lookups` feeding the
+  exporter.
+
 ## [2.1.0] - 2026-08-07
 
 A grader declares what kind of check it is at registration, so a consumer
@@ -174,7 +220,9 @@ by semantic versioning: a breaking change to either means a 2.0.
   rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
   column-store outbox, and content-hash provenance.
 
-[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.1.0...HEAD
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.3.0...HEAD
+[2.3.0]: https://github.com/scottpmiller/evalcore/compare/2.2.0...2.3.0
+[2.2.0]: https://github.com/scottpmiller/evalcore/compare/2.1.0...2.2.0
 [2.1.0]: https://github.com/scottpmiller/evalcore/compare/2.0.0...2.1.0
 [2.0.0]: https://github.com/scottpmiller/evalcore/compare/1.0.0...2.0.0
 [1.0.0]: https://github.com/scottpmiller/evalcore/compare/0.3.0...1.0.0

{evalcore-2.1.0 → evalcore-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalcore
-Version: 2.1.0
+Version: 2.3.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -502,13 +502,23 @@ candidate = runner.run_suite_sync(suite, 'candidate', mode='replay',
                                   revision='abc123', created_at=now)
 # (async context: `await runner.run_suite(...)` is the same call)
 
-result = compare.compare(baseline, candidate, suite.thresholds)
-print(report.render_scorecard(candidate))
+# run_suite returns a RunResult: the scorecard plus every per-sample
+# result behind it. compare and the reporters take the scorecard.
+result = compare.compare(
+    baseline.scorecard, candidate.scorecard, suite.thresholds
+)
+print(report.render_scorecard(candidate.scorecard))
 print(report.render_comparison(result))
 
-store.write_scorecard('candidate.scorecard.json', candidate)
+store.write_scorecard('candidate.scorecard.json', candidate.scorecard)
 store.write_comparison('comparison.json', result)
-store.JsonlOutboxExporter('outbox.jsonl').export(candidate)
+
+# The rows. A run carries what its graders are, so nothing else is needed.
+# The baseline half exports without the comparison, since it was not itself
+# gated.
+exporter = store.JsonlOutboxExporter('outbox.jsonl')
+exporter.export_scores(baseline)
+exporter.export_scores(candidate, result, baseline_run_id=baseline.run_id)
 
 raise SystemExit(0 if result.verdict != 'fail' else 1)
 ```
@@ -784,6 +794,27 @@ table without a mapping layer. A store that forbids nullable columns fills
 those nulls in at ingest, on its side of the seam. Swap the exporter for a real
 database client without touching the runner or any consumer.
 
+`ScoreExporter` is that seam, named: any object with an `export_scores(run,
+comparison, **kwargs)` method returning a count. `JsonlOutboxExporter` is the
+built-in one; an exporter that publishes to a real store belongs in the
+package that owns that store, since it is the store that knows its own column
+types, null policy, and transport. Swapping one for the other is a
+constructor line, so an offline run and a live one share a code path:
+
+```python
+exporter = store.JsonlOutboxExporter(path)   # offline
+exporter = KafkaOutboxExporter(...)          # live, from another package
+exporter.export_scores(run, comparison)
+```
+
+A `RunResult` carries `graders`, a map of grader name to its category and
+judge scale, so it describes its own scores and an exporter needs nothing but
+the run. That matters most for an artifact: a `run.json` written by one CI
+step and published by a later one used to need the suite file alongside it,
+and without one every grader reported `unknown`. `grader_types` and
+`judge_scales` still override the map, which is how a run written before
+2.3.0 publishes correctly - `grader_lookups(suite.graders)` builds them.
+
 ---
 
 ## The two extension seams (recap)

{evalcore-2.1.0 → evalcore-2.3.0}/README.md

@@ -470,13 +470,23 @@ candidate = runner.run_suite_sync(suite, 'candidate', mode='replay',
                                   revision='abc123', created_at=now)
 # (async context: `await runner.run_suite(...)` is the same call)
 
-result = compare.compare(baseline, candidate, suite.thresholds)
-print(report.render_scorecard(candidate))
+# run_suite returns a RunResult: the scorecard plus every per-sample
+# result behind it. compare and the reporters take the scorecard.
+result = compare.compare(
+    baseline.scorecard, candidate.scorecard, suite.thresholds
+)
+print(report.render_scorecard(candidate.scorecard))
 print(report.render_comparison(result))
 
-store.write_scorecard('candidate.scorecard.json', candidate)
+store.write_scorecard('candidate.scorecard.json', candidate.scorecard)
 store.write_comparison('comparison.json', result)
-store.JsonlOutboxExporter('outbox.jsonl').export(candidate)
+
+# The rows. A run carries what its graders are, so nothing else is needed.
+# The baseline half exports without the comparison, since it was not itself
+# gated.
+exporter = store.JsonlOutboxExporter('outbox.jsonl')
+exporter.export_scores(baseline)
+exporter.export_scores(candidate, result, baseline_run_id=baseline.run_id)
 
 raise SystemExit(0 if result.verdict != 'fail' else 1)
 ```
@@ -752,6 +762,27 @@ table without a mapping layer. A store that forbids nullable columns fills
 those nulls in at ingest, on its side of the seam. Swap the exporter for a real
 database client without touching the runner or any consumer.
 
+`ScoreExporter` is that seam, named: any object with an `export_scores(run,
+comparison, **kwargs)` method returning a count. `JsonlOutboxExporter` is the
+built-in one; an exporter that publishes to a real store belongs in the
+package that owns that store, since it is the store that knows its own column
+types, null policy, and transport. Swapping one for the other is a
+constructor line, so an offline run and a live one share a code path:
+
+```python
+exporter = store.JsonlOutboxExporter(path)   # offline
+exporter = KafkaOutboxExporter(...)          # live, from another package
+exporter.export_scores(run, comparison)
+```
+
+A `RunResult` carries `graders`, a map of grader name to its category and
+judge scale, so it describes its own scores and an exporter needs nothing but
+the run. That matters most for an artifact: a `run.json` written by one CI
+step and published by a later one used to need the suite file alongside it,
+and without one every grader reported `unknown`. `grader_types` and
+`judge_scales` still override the map, which is how a run written before
+2.3.0 publishes correctly - `grader_lookups(suite.graders)` builds them.
+
 ---
 
 ## The two extension seams (recap)

{evalcore-2.1.0 → evalcore-2.3.0}/examples/quickstart/graders.py

@@ -26,7 +26,7 @@ _ACK = re.compile(
 )
 
 
-@base.register('acknowledges_customer')
+@base.register('acknowledges_customer', base.GraderType.HEURISTIC)
 class AcknowledgesCustomer:
     """The reply must acknowledge the customer, not just brush them off.
 
@@ -59,7 +59,7 @@ class AcknowledgesCustomer:
         ]
 
 
-@base.register('distinct_reply_rate')
+@base.register('distinct_reply_rate', base.GraderType.HEURISTIC)
 class DistinctReplyRate:
     """Fraction of cases whose reply is unique (mode-collapse check).
 

{evalcore-2.1.0 → evalcore-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "2.1.0"
+version = "2.3.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-2.1.0 → evalcore-2.3.0}/src/evalcore/graders/base.py

@@ -94,6 +94,10 @@ def register(
             raise ConfigError(f'grader type {type_name!r} already registered')
         _REGISTRY[type_name] = cls
         _CATEGORIES[type_name] = GraderType(category)
+        # Also on the class, so an instance can answer for itself. The runner
+        # reads it off the graders it built to make the run self-describing,
+        # the same way it collects `judge_version`.
+        cls.grader_category = GraderType(category)
         return cls
 
     return _decorate

{evalcore-2.1.0 → evalcore-2.3.0}/src/evalcore/models.py

@@ -162,6 +162,28 @@ class Scorecard(pydantic.BaseModel):
     metrics: dict[str, MetricValue] = pydantic.Field(default_factory=dict)
 
 
+class GraderInfo(pydantic.BaseModel):
+    """What a grader is, as opposed to what it scored.
+
+    A ``Score`` names the grader that emitted it and nothing else, so a run
+    that did not carry this could not describe its own scores: publishing it
+    needed the suite file alongside it, and a ``run.json`` handed to a later
+    CI step without one reported every grader as ``unknown``.
+
+    Run-grain, not score-grain: one entry per grader rather than the same two
+    fields repeated on a couple of hundred rows.
+    """
+
+    #: The registered category, a ``graders.GraderType`` value. A plain `str`
+    #: here because `graders.base` imports this module, so this module cannot
+    #: import it back; ``GraderType`` is a ``StrEnum`` and compares equal.
+    category: str = 'unknown'
+    #: The scale raw judge points sit on, 0 for a grader that is not a judge.
+    #: Normalization is points / scale, so the points are unreadable without
+    #: it.
+    scale: int = 0
+
+
 class RunResult(pydantic.BaseModel):
     """Everything one run produced: the scorecard plus per-sample results.
 
@@ -174,12 +196,18 @@ class RunResult(pydantic.BaseModel):
     over the whole run. The scorecard keeps their values but not which grader
     emitted them or what it reported, so they are retained here too - a
     results-store row needs the grader attribution.
+
+    ``graders`` maps grader name to what that grader is, so the run is
+    self-describing: an exporter needs nothing but the run. Empty on runs
+    written before 2.3.0, which is why exporters still accept the lookups as
+    a fallback.
     """
 
     run_id: str
     scorecard: Scorecard
     results: list[CaseResult] = pydantic.Field(default_factory=list)
     aggregate_scores: list[Score] = pydantic.Field(default_factory=list)
+    graders: dict[str, GraderInfo] = pydantic.Field(default_factory=dict)
 
 
 class Rating(pydantic.BaseModel):

{evalcore-2.1.0 → evalcore-2.3.0}/src/evalcore/runner.py

@@ -84,6 +84,35 @@ def _uuid7() -> uuid.UUID:
     )
 
 
+def _grader_info(*grader_groups) -> dict[str, models.GraderInfo]:
+    """What each grader is, so the run can describe its own scores.
+
+    Read off the grader instances rather than the suite config, so a run is
+    publishable from the artifact alone - no suite file alongside it. Both
+    attributes are duck-typed the way ``judge_version`` is: ``register`` puts
+    ``grader_category`` on the class, and only a judge has a ``scale``.
+
+    Args:
+        grader_groups: The per-case and aggregate grader lists.
+
+    Returns:
+        Grader name to its category and judge scale.
+
+    """
+    return {
+        grader.name: models.GraderInfo(
+            category=str(
+                getattr(
+                    grader, 'grader_category', graders_base.GraderType.UNKNOWN
+                )
+            ),
+            scale=int(getattr(grader, 'scale', 0) or 0),
+        )
+        for group in grader_groups
+        for grader in group
+    }
+
+
 def _aggregate_metrics(
     results: list[models.CaseResult], agg_scores: list[models.Score]
 ) -> dict[str, models.MetricValue]:
@@ -321,6 +350,7 @@ async def run_suite(
         scorecard=scorecard,
         results=results,
         aggregate_scores=agg_scores,
+        graders=_grader_info(per_case_graders, aggregate_graders),
     )
 
 

{evalcore-2.1.0 → evalcore-2.3.0}/src/evalcore/store.py

@@ -32,6 +32,7 @@ than model attribute names, so ``project`` is emitted as ``application``,
 
 import json
 import pathlib
+import typing
 
 from evalcore import graders, models
 
@@ -401,8 +402,12 @@ def score_rows(
 
     ``comparison`` stamps the run-grain gate columns and the per-metric ``win``
     and ``guardrail`` fields; without it the run reads as ungated.
-    ``grader_types`` and ``judge_scales`` are grader-name lookups the caller
-    builds from the suite, since a ``Score`` carries neither.
+    The grader category and judge scale come from ``run.graders``, which the
+    runner fills in - a ``Score`` carries neither, and a run that could not
+    answer for itself was not publishable from its own artifact.
+    ``grader_types`` and ``judge_scales`` override it, and are how a run
+    written before 2.3.0 (whose map is empty) still publishes correctly;
+    :func:`grader_lookups` builds them from the suite.
     """
     key = _run_key(run.scorecard)
     gate = _gate(comparison, baseline_run_id)
@@ -411,8 +416,16 @@ def score_rows(
         if comparison
         else {}
     )
-    types = grader_types or {}
-    scales = judge_scales or {}
+    # The run describes its own graders as of 2.3.0. The lookups remain for
+    # runs written before that, which carry an empty map - a `run.json` on
+    # disk outlives the release that wrote it. An explicit lookup still wins,
+    # so a caller can correct a run it did not produce.
+    types = {name: info.category for name, info in run.graders.items()}
+    scales = {
+        name: info.scale for name, info in run.graders.items() if info.scale
+    }
+    types.update(grader_types or {})
+    scales.update(judge_scales or {})
     rows: list[dict] = []
     scored: set[str] = set()
 
@@ -511,12 +524,56 @@ def score_rows(
     return rows
 
 
+@typing.runtime_checkable
+class ScoreExporter(typing.Protocol):
+    """Where a run's score rows go.
+
+    The seam between the engine and a results store. :func:`score_rows`
+    produces the rows; an exporter decides what happens to them. Implement
+    this to publish to a real store - a Kafka producer, a database client -
+    without the runner or any consumer changing.
+
+    An implementation lives with the store it targets, not here. It is the
+    store that knows its own column types, its null policy, and its
+    transport; the engine only knows the row.
+
+    Swapping one for another is a constructor line at the call site, so an
+    offline run and a live one share the same code path::
+
+        exporter = JsonlOutboxExporter(path)      # offline
+        exporter = KafkaOutboxExporter(...)       # live, from another package
+        exporter.export_scores(run, comparison, grader_types=..., ...)
+
+    """
+
+    def export_scores(
+        self,
+        run: models.RunResult,
+        comparison: models.Comparison | None = None,
+        **kwargs,
+    ) -> int:
+        """Export a run's rows and return how many were exported.
+
+        Args:
+            run: The completed run.
+            comparison: The gate result, on the candidate half of a gate;
+                left off the baseline half, which was not itself gated.
+            kwargs: Passed through to :func:`score_rows` - ``baseline_run_id``,
+                ``grader_types`` and ``judge_scales``.
+
+        Returns:
+            The number of rows exported.
+
+        """
+        ...
+
+
 class JsonlOutboxExporter:
     """Append score rows to a JSONL outbox for a shipper to drain.
 
     A no-network stand-in for direct ingestion: real deployments point a
-    shipper at this file, or replace this class with a database client
-    implementing the same ``export_scores`` method.
+    shipper at this file, or swap in another :class:`ScoreExporter` that
+    writes to the store directly.
     """
 
     def __init__(self, outbox_path: str | pathlib.Path):

{evalcore-2.1.0 → evalcore-2.3.0}/tests/test_store.py

@@ -338,6 +338,92 @@ class GraderLookupTests(unittest.TestCase):
                 name = 'non_empty'
 
 
+class SelfDescribingRunTests(unittest.TestCase):
+    """A run carries what its scores mean, so an artifact is publishable."""
+
+    @staticmethod
+    def _run_with_graders():
+        run = _run()
+        run.graders = {
+            'det': models.GraderInfo(category='heuristic'),
+            'j': models.GraderInfo(category='llm_as_judge', scale=7),
+            'cls': models.GraderInfo(category='statistical'),
+        }
+        return run
+
+    def test_the_run_supplies_the_lookups(self):
+        rows = store.score_rows(self._run_with_graders())
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+        self.assertEqual(types['j'], 'llm_as_judge')
+
+    def test_a_judge_scale_comes_off_the_run(self):
+        rows = store.score_rows(self._run_with_graders())
+        judged = {r['judge_scale'] for r in rows if r['grader'] == 'j'}
+        plain = {r['judge_scale'] for r in rows if r['grader'] == 'det'}
+        self.assertEqual(judged, {7})
+        self.assertEqual(plain, {0})
+
+    def test_an_explicit_lookup_still_wins(self):
+        """So a caller can correct a run it did not produce."""
+        rows = store.score_rows(
+            self._run_with_graders(), grader_types={'det': 'statistical'}
+        )
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'statistical')
+
+    def test_a_run_from_before_the_field_still_publishes(self):
+        """The map is empty on runs written by older versions, so the
+        lookups have to stay."""
+        run = _run()
+        self.assertEqual(run.graders, {})
+        rows = store.score_rows(run, grader_types={'det': 'heuristic'})
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+
+    def test_it_survives_the_json_round_trip(self):
+        """The whole point: no suite file alongside the artifact."""
+        with tempfile.TemporaryDirectory() as tmp:
+            path = pathlib.Path(tmp) / 'run.json'
+            store.write_run(path, self._run_with_graders())
+            reloaded = store.read_run(path)
+        self.assertEqual(reloaded.graders['j'].scale, 7)
+        rows = store.score_rows(reloaded)
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+        self.assertEqual(types['j'], 'llm_as_judge')
+
+
+class ScoreExporterProtocolTests(unittest.TestCase):
+    """The seam another package implements to publish to a real store."""
+
+    def test_the_builtin_exporter_satisfies_it(self):
+        exporter = store.JsonlOutboxExporter('/dev/null')
+        self.assertIsInstance(exporter, store.ScoreExporter)
+
+    def test_an_outside_implementation_satisfies_it(self):
+        """A store-side exporter: same method, its own transport."""
+
+        class Collecting:
+            def __init__(self):
+                self.rows = []
+
+            def export_scores(self, run, comparison=None, **kwargs):
+                self.rows = store.score_rows(run, comparison, **kwargs)
+                return len(self.rows)
+
+        exporter = Collecting()
+        self.assertIsInstance(exporter, store.ScoreExporter)
+        self.assertEqual(exporter.export_scores(_run()), len(exporter.rows))
+
+    def test_missing_the_method_does_not(self):
+        class NotAnExporter:
+            def export(self, run):  # pragma: no cover - never called
+                return 0
+
+        self.assertNotIsInstance(NotAnExporter(), store.ScoreExporter)
+
+
 class ExporterTests(unittest.TestCase):
     def test_export_scores(self):
         with tempfile.TemporaryDirectory() as tmp:

{evalcore-2.1.0 → evalcore-2.3.0}/uv.lock

@@ -188,7 +188,7 @@ wheels = [
 
 [[package]]
 name = "evalcore"
-version = "2.1.0"
+version = "2.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "pydantic" },