evalcore 2.2.0__tar.gz → 2.3.0__tar.gz

{evalcore-2.2.0 → evalcore-2.3.0}/CHANGELOG.md

@@ -6,6 +6,30 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.3.0] - 2026-08-08
+
+A run now describes its own scores.
+
+### Added
+- `RunResult.graders`, a map of grader name to `models.GraderInfo`
+  (`category`, `scale`), filled in by the runner from the graders it built.
+  A `Score` names the grader that emitted it and nothing else, so a run could
+  not previously answer for itself: a `run.json` written by one CI step and
+  published by a later one needed the suite file alongside it, and without one
+  every grader reported `unknown` and a suite with a judge could not be
+  published at all. Run-grain rather than score-grain - one entry per grader,
+  not two fields repeated across a couple of hundred rows.
+- `models.GraderInfo`.
+- `register` also records the category on the decorated class as
+  `grader_category`, so a grader instance can answer for itself. The runner
+  reads it the same way it already collects `judge_version`.
+
+### Changed
+- `store.score_rows` takes the grader category and judge scale from
+  `run.graders`. `grader_types` and `judge_scales` still override it, and are
+  how a run written before this release - whose map is empty - publishes
+  correctly. `grader_lookups` is unchanged and still builds them from a suite.
+
 ## [2.2.0] - 2026-08-08
 
 ### Added
@@ -196,7 +220,8 @@ by semantic versioning: a breaking change to either means a 2.0.
   rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
   column-store outbox, and content-hash provenance.
 
-[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.2.0...HEAD
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.3.0...HEAD
+[2.3.0]: https://github.com/scottpmiller/evalcore/compare/2.2.0...2.3.0
 [2.2.0]: https://github.com/scottpmiller/evalcore/compare/2.1.0...2.2.0
 [2.1.0]: https://github.com/scottpmiller/evalcore/compare/2.0.0...2.1.0
 [2.0.0]: https://github.com/scottpmiller/evalcore/compare/1.0.0...2.0.0

{evalcore-2.2.0 → evalcore-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalcore
-Version: 2.2.0
+Version: 2.3.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -513,19 +513,12 @@ print(report.render_comparison(result))
 store.write_scorecard('candidate.scorecard.json', candidate.scorecard)
 store.write_comparison('comparison.json', result)
 
-# The rows. grader_lookups supplies the grader category and judge scale a
-# Score doesn't carry; the baseline half exports without the comparison,
-# since it was not itself gated.
-types, scales = store.grader_lookups(suite.graders)
+# The rows. A run carries what its graders are, so nothing else is needed.
+# The baseline half exports without the comparison, since it was not itself
+# gated.
 exporter = store.JsonlOutboxExporter('outbox.jsonl')
-exporter.export_scores(baseline, grader_types=types, judge_scales=scales)
-exporter.export_scores(
-    candidate,
-    result,
-    baseline_run_id=baseline.run_id,
-    grader_types=types,
-    judge_scales=scales,
-)
+exporter.export_scores(baseline)
+exporter.export_scores(candidate, result, baseline_run_id=baseline.run_id)
 
 raise SystemExit(0 if result.verdict != 'fail' else 1)
 ```
@@ -811,9 +804,17 @@ constructor line, so an offline run and a live one share a code path:
 ```python
 exporter = store.JsonlOutboxExporter(path)   # offline
 exporter = KafkaOutboxExporter(...)          # live, from another package
-exporter.export_scores(run, comparison, grader_types=..., judge_scales=...)
+exporter.export_scores(run, comparison)
 ```
 
+A `RunResult` carries `graders`, a map of grader name to its category and
+judge scale, so it describes its own scores and an exporter needs nothing but
+the run. That matters most for an artifact: a `run.json` written by one CI
+step and published by a later one used to need the suite file alongside it,
+and without one every grader reported `unknown`. `grader_types` and
+`judge_scales` still override the map, which is how a run written before
+2.3.0 publishes correctly - `grader_lookups(suite.graders)` builds them.
+
 ---
 
 ## The two extension seams (recap)

{evalcore-2.2.0 → evalcore-2.3.0}/README.md

@@ -481,19 +481,12 @@ print(report.render_comparison(result))
 store.write_scorecard('candidate.scorecard.json', candidate.scorecard)
 store.write_comparison('comparison.json', result)
 
-# The rows. grader_lookups supplies the grader category and judge scale a
-# Score doesn't carry; the baseline half exports without the comparison,
-# since it was not itself gated.
-types, scales = store.grader_lookups(suite.graders)
+# The rows. A run carries what its graders are, so nothing else is needed.
+# The baseline half exports without the comparison, since it was not itself
+# gated.
 exporter = store.JsonlOutboxExporter('outbox.jsonl')
-exporter.export_scores(baseline, grader_types=types, judge_scales=scales)
-exporter.export_scores(
-    candidate,
-    result,
-    baseline_run_id=baseline.run_id,
-    grader_types=types,
-    judge_scales=scales,
-)
+exporter.export_scores(baseline)
+exporter.export_scores(candidate, result, baseline_run_id=baseline.run_id)
 
 raise SystemExit(0 if result.verdict != 'fail' else 1)
 ```
@@ -779,9 +772,17 @@ constructor line, so an offline run and a live one share a code path:
 ```python
 exporter = store.JsonlOutboxExporter(path)   # offline
 exporter = KafkaOutboxExporter(...)          # live, from another package
-exporter.export_scores(run, comparison, grader_types=..., judge_scales=...)
+exporter.export_scores(run, comparison)
 ```
 
+A `RunResult` carries `graders`, a map of grader name to its category and
+judge scale, so it describes its own scores and an exporter needs nothing but
+the run. That matters most for an artifact: a `run.json` written by one CI
+step and published by a later one used to need the suite file alongside it,
+and without one every grader reported `unknown`. `grader_types` and
+`judge_scales` still override the map, which is how a run written before
+2.3.0 publishes correctly - `grader_lookups(suite.graders)` builds them.
+
 ---
 
 ## The two extension seams (recap)

{evalcore-2.2.0 → evalcore-2.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "2.2.0"
+version = "2.3.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-2.2.0 → evalcore-2.3.0}/src/evalcore/graders/base.py

@@ -94,6 +94,10 @@ def register(
             raise ConfigError(f'grader type {type_name!r} already registered')
         _REGISTRY[type_name] = cls
         _CATEGORIES[type_name] = GraderType(category)
+        # Also on the class, so an instance can answer for itself. The runner
+        # reads it off the graders it built to make the run self-describing,
+        # the same way it collects `judge_version`.
+        cls.grader_category = GraderType(category)
         return cls
 
     return _decorate

{evalcore-2.2.0 → evalcore-2.3.0}/src/evalcore/models.py

@@ -162,6 +162,28 @@ class Scorecard(pydantic.BaseModel):
     metrics: dict[str, MetricValue] = pydantic.Field(default_factory=dict)
 
 
+class GraderInfo(pydantic.BaseModel):
+    """What a grader is, as opposed to what it scored.
+
+    A ``Score`` names the grader that emitted it and nothing else, so a run
+    that did not carry this could not describe its own scores: publishing it
+    needed the suite file alongside it, and a ``run.json`` handed to a later
+    CI step without one reported every grader as ``unknown``.
+
+    Run-grain, not score-grain: one entry per grader rather than the same two
+    fields repeated on a couple of hundred rows.
+    """
+
+    #: The registered category, a ``graders.GraderType`` value. A plain `str`
+    #: here because `graders.base` imports this module, so this module cannot
+    #: import it back; ``GraderType`` is a ``StrEnum`` and compares equal.
+    category: str = 'unknown'
+    #: The scale raw judge points sit on, 0 for a grader that is not a judge.
+    #: Normalization is points / scale, so the points are unreadable without
+    #: it.
+    scale: int = 0
+
+
 class RunResult(pydantic.BaseModel):
     """Everything one run produced: the scorecard plus per-sample results.
 
@@ -174,12 +196,18 @@ class RunResult(pydantic.BaseModel):
     over the whole run. The scorecard keeps their values but not which grader
     emitted them or what it reported, so they are retained here too - a
     results-store row needs the grader attribution.
+
+    ``graders`` maps grader name to what that grader is, so the run is
+    self-describing: an exporter needs nothing but the run. Empty on runs
+    written before 2.3.0, which is why exporters still accept the lookups as
+    a fallback.
     """
 
     run_id: str
     scorecard: Scorecard
     results: list[CaseResult] = pydantic.Field(default_factory=list)
     aggregate_scores: list[Score] = pydantic.Field(default_factory=list)
+    graders: dict[str, GraderInfo] = pydantic.Field(default_factory=dict)
 
 
 class Rating(pydantic.BaseModel):

{evalcore-2.2.0 → evalcore-2.3.0}/src/evalcore/runner.py

@@ -84,6 +84,35 @@ def _uuid7() -> uuid.UUID:
     )
 
 
+def _grader_info(*grader_groups) -> dict[str, models.GraderInfo]:
+    """What each grader is, so the run can describe its own scores.
+
+    Read off the grader instances rather than the suite config, so a run is
+    publishable from the artifact alone - no suite file alongside it. Both
+    attributes are duck-typed the way ``judge_version`` is: ``register`` puts
+    ``grader_category`` on the class, and only a judge has a ``scale``.
+
+    Args:
+        grader_groups: The per-case and aggregate grader lists.
+
+    Returns:
+        Grader name to its category and judge scale.
+
+    """
+    return {
+        grader.name: models.GraderInfo(
+            category=str(
+                getattr(
+                    grader, 'grader_category', graders_base.GraderType.UNKNOWN
+                )
+            ),
+            scale=int(getattr(grader, 'scale', 0) or 0),
+        )
+        for group in grader_groups
+        for grader in group
+    }
+
+
 def _aggregate_metrics(
     results: list[models.CaseResult], agg_scores: list[models.Score]
 ) -> dict[str, models.MetricValue]:
@@ -321,6 +350,7 @@ async def run_suite(
         scorecard=scorecard,
         results=results,
         aggregate_scores=agg_scores,
+        graders=_grader_info(per_case_graders, aggregate_graders),
     )
 
 

{evalcore-2.2.0 → evalcore-2.3.0}/src/evalcore/store.py

@@ -402,8 +402,12 @@ def score_rows(
 
     ``comparison`` stamps the run-grain gate columns and the per-metric ``win``
     and ``guardrail`` fields; without it the run reads as ungated.
-    ``grader_types`` and ``judge_scales`` are grader-name lookups the caller
-    builds from the suite, since a ``Score`` carries neither.
+    The grader category and judge scale come from ``run.graders``, which the
+    runner fills in - a ``Score`` carries neither, and a run that could not
+    answer for itself was not publishable from its own artifact.
+    ``grader_types`` and ``judge_scales`` override it, and are how a run
+    written before 2.3.0 (whose map is empty) still publishes correctly;
+    :func:`grader_lookups` builds them from the suite.
     """
     key = _run_key(run.scorecard)
     gate = _gate(comparison, baseline_run_id)
@@ -412,8 +416,16 @@ def score_rows(
         if comparison
         else {}
     )
-    types = grader_types or {}
-    scales = judge_scales or {}
+    # The run describes its own graders as of 2.3.0. The lookups remain for
+    # runs written before that, which carry an empty map - a `run.json` on
+    # disk outlives the release that wrote it. An explicit lookup still wins,
+    # so a caller can correct a run it did not produce.
+    types = {name: info.category for name, info in run.graders.items()}
+    scales = {
+        name: info.scale for name, info in run.graders.items() if info.scale
+    }
+    types.update(grader_types or {})
+    scales.update(judge_scales or {})
     rows: list[dict] = []
     scored: set[str] = set()
 

{evalcore-2.2.0 → evalcore-2.3.0}/tests/test_store.py

@@ -338,6 +338,62 @@ class GraderLookupTests(unittest.TestCase):
                 name = 'non_empty'
 
 
+class SelfDescribingRunTests(unittest.TestCase):
+    """A run carries what its scores mean, so an artifact is publishable."""
+
+    @staticmethod
+    def _run_with_graders():
+        run = _run()
+        run.graders = {
+            'det': models.GraderInfo(category='heuristic'),
+            'j': models.GraderInfo(category='llm_as_judge', scale=7),
+            'cls': models.GraderInfo(category='statistical'),
+        }
+        return run
+
+    def test_the_run_supplies_the_lookups(self):
+        rows = store.score_rows(self._run_with_graders())
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+        self.assertEqual(types['j'], 'llm_as_judge')
+
+    def test_a_judge_scale_comes_off_the_run(self):
+        rows = store.score_rows(self._run_with_graders())
+        judged = {r['judge_scale'] for r in rows if r['grader'] == 'j'}
+        plain = {r['judge_scale'] for r in rows if r['grader'] == 'det'}
+        self.assertEqual(judged, {7})
+        self.assertEqual(plain, {0})
+
+    def test_an_explicit_lookup_still_wins(self):
+        """So a caller can correct a run it did not produce."""
+        rows = store.score_rows(
+            self._run_with_graders(), grader_types={'det': 'statistical'}
+        )
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'statistical')
+
+    def test_a_run_from_before_the_field_still_publishes(self):
+        """The map is empty on runs written by older versions, so the
+        lookups have to stay."""
+        run = _run()
+        self.assertEqual(run.graders, {})
+        rows = store.score_rows(run, grader_types={'det': 'heuristic'})
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+
+    def test_it_survives_the_json_round_trip(self):
+        """The whole point: no suite file alongside the artifact."""
+        with tempfile.TemporaryDirectory() as tmp:
+            path = pathlib.Path(tmp) / 'run.json'
+            store.write_run(path, self._run_with_graders())
+            reloaded = store.read_run(path)
+        self.assertEqual(reloaded.graders['j'].scale, 7)
+        rows = store.score_rows(reloaded)
+        types = {row['grader']: row['grader_type'] for row in rows}
+        self.assertEqual(types['det'], 'heuristic')
+        self.assertEqual(types['j'], 'llm_as_judge')
+
+
 class ScoreExporterProtocolTests(unittest.TestCase):
     """The seam another package implements to publish to a real store."""
 

{evalcore-2.2.0 → evalcore-2.3.0}/uv.lock

@@ -188,7 +188,7 @@ wheels = [
 
 [[package]]
 name = "evalcore"
-version = "2.2.0"
+version = "2.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "pydantic" },