evalcore 1.0.0__tar.gz → 2.1.0__tar.gz

evalcore-2.1.0/CHANGELOG.md

@@ -0,0 +1,183 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/), and the project follows
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [2.1.0] - 2026-08-07
+
+A grader declares what kind of check it is at registration, so a consumer
+plug-in is categorised the same way a built-in is.
+
+Shipped as a minor despite the signature change below. `register` is public,
+so the 1.0.0 policy would call this a major; it goes out as 2.1.0 as a
+deliberate exception, because the break is a one-line edit per grader that
+fails loudly at import.
+
+### Changed
+- **Breaking:** `graders.base.register` takes a required second argument,
+  `category`, a `graders.GraderType`. Every `@base.register('foo')` becomes
+  `@base.register('foo', base.GraderType.HEURISTIC)` or whichever member
+  applies; omitting it is a `TypeError` at import. Required rather than
+  defaulted on purpose - it is the only source of a row's `grader_type`, and
+  a default would be the value every grader forgets to override.
+- `store.grader_lookups` reads the category from the registry instead of a
+  closed table of built-in type names, so a plug-in that declares
+  `HEURISTIC` reports `heuristic` where it used to report `unknown`. Rows for
+  consumer graders change value in the `grader_type` column; nothing about
+  the row shape changes.
+
+### Added
+- `graders.GraderType`, a `StrEnum` over the closed set the results store's
+  `grader_type` column accepts: `unknown`, `heuristic`, `statistical`,
+  `llm_as_judge`, `trajectory`, `human`. A `StrEnum` so it needs no
+  serializer of its own on the way to a row.
+- `graders.category_of`, the registry lookup behind `grader_lookups`.
+
+### Removed
+- `store._GRADER_TYPES`, the private table the categories used to live in.
+  Keeping it alongside the registration argument would mean two sources for
+  one fact and a precedence rule between them.
+
+## [2.0.0] - 2026-07-31
+
+Breaks both the public API and the outbox row shape, so it's a major per the
+1.0.0 policy.
+
+### Changed
+- **Breaking:** a sample is identified by `sample_hash` - a content digest of
+  the output it produced - rather than by the ordinal `sample_idx`. A row now
+  names the exact response behind it. The field is renamed on `CaseResult`,
+  `Rating`, `Preference`, `PairwiseOutcome` and `PairwiseAgreementCase`, and in
+  the outbox rows. Two runs of the same case never share a hash, so anything
+  comparing runs aligns on `case_id` and sample order; pairwise and the ranking
+  app anchor a pair on the `variant_a` side's digest so human and judge picks
+  still join. **Ratings and preferences files written by 1.x do not carry a
+  hash and will not join to a run** - re-collect them, or backfill the field.
+- **Breaking:** outbox keys track the store's column names: `created_at` is
+  emitted as `started_at`, `n_cases` as `case_count`, `n_samples` as
+  `sample_count`.
+- **Breaking:** `store.read_checkpoint_results` is now
+  `store.read_checkpoint_samples` and returns `(ordinal, result)` pairs.
+  Checkpoint lines nest the result under `result` and tag it with `sample`, so
+  resume knows which samples are still owed. 1.x checkpoints cannot be resumed.
+
+### Fixed
+- Resuming a run with `concurrency > 1` could skip a sample and re-run another.
+  A concurrent run checkpoints in completion order, so an interrupt leaves a
+  hole rather than a clean prefix; resume now reruns the samples that are
+  actually missing. On a deterministic target the re-run collided with a digest
+  already recorded, so the store collapsed two samples into one row.
+- The pairwise judge and the side-by-side ranking app filtered samples
+  differently - non-empty content vs. no error - so with `n_samples > 1` and an
+  error on either side they paired A's n-th sample against different B samples.
+  `agreement` joins the two on A's digest, so it scored two different
+  comparisons as one. Both now go through
+  `pairwise.comparable_samples`, which requires a successful invocation *and*
+  resolvable content. Two behaviour changes fall out: an errored output is no
+  longer judged even when its content ref still resolves, and a sample with
+  empty content is no longer shown to a rater as a blank panel.
+
+### Changed (internal)
+- `pairwise._content_map` is now `pairwise.comparable_samples` and is the one
+  place that decides whether a sample can take part in a comparison.
+
+### Removed
+- `docs/clickhouse-schema.{sql,html}`. The outbox targets a flat row shape, not
+  one vendor's DDL, and the file documented a specific deployment - database
+  name, ingestion topology, sample data - none of which the engine needs. A
+  store with column types the feed does not match maps the rows at its own
+  boundary.
+
+## [1.0.0] - 2026-07-28
+
+First stable release. The public API and the outbox row shape are now covered
+by semantic versioning: a breaking change to either means a 2.0.
+
+### Changed
+- **Breaking:** outbox rows no longer carry `input_tokens`, `output_tokens` or
+  `cost`. `Output.tokens` is an open dict an adapter may put anything in, and
+  usage accounting is captured outside the results store.
+- **Breaking:** the run-grain `revision` key is emitted as
+  `application_revision`, which is its column name in the store.
+
+### Added
+- `docs/clickhouse-schema.sql`: the ClickHouse schema the outbox rows target -
+  the written `evaluation_scores` table, the row grammar as `CONSTRAINT`s, and
+  the canonical scorecard query. Also rendered as
+  `docs/clickhouse-schema.html`. (Both removed again in 2.0.0.)
+
+## [0.3.0] - 2026-07-28
+
+### Changed
+- **Breaking:** the results-store outbox is now one feed at
+  `(run, case, sample, grader, metric)` grain, targeting a single flat
+  `evaluation_scores` table with the run trend and the invocation grain as
+  plain views over it. `store.scorecard_rows` and
+  `JsonlOutboxExporter.export` are removed; use `store.score_rows` and
+  `export_scores`. `--export-scores` is now an alias for `--export`.
+- **Breaking:** a missing measurement is emitted as `null` rather than the
+  `(value=0, has_value=false)` sentinel pair, and the `has_value` / `has_stdev`
+  companion keys are gone. The store columns are `Nullable`, and a real `0.0` is
+  a meaningful score.
+- **Breaking:** outbox row keys are the store column names, so `project` is
+  emitted as `application`, `mode` as `adapter_mode`, `created_at` as
+  `timestamp`, and `Output.error` as `is_error` plus `error_text`.
+- **Breaking:** outbox rows no longer carry `model_id` or `prompt_version`. Both
+  are `variant.knobs.get(...)` projections and travel inside `variant_knobs`.
+- **Breaking:** `run_id` is now a dashed UUIDv7 rather than `uuid4().hex`, so it
+  parses as a ClickHouse `UUID` and carries its own creation time. Uses
+  `uuid.uuid7()` on 3.14 and an RFC 9562 implementation on 3.11 through 3.13.
+- Outbox rows now carry the gate: `gate_verdict`, `gate_win`,
+  `baseline_run_id`, `baseline_variant`, `win_baseline`, `win_candidate`,
+  `win_delta` and `gate_summary` at run grain, plus `win`, `guardrail` and
+  `guardrail_gap` on the metric each one refers to. Pass the `Comparison` to
+  `score_rows`; without it a run reads as ungated.
+
+### Added
+- `RunResult.aggregate_scores`, retaining the `kind='aggregate'` scores so a
+  store row keeps the grader that emitted them and what it reported. The
+  scorecard kept only their values.
+- `store.grader_lookups`, mapping grader names to a category and to a judge
+  scale from a suite's grader specs, since a `Score` carries neither.
+- Three outbox row shapes that previously had no representation: an aggregate
+  metric (no `case_id`), an invocation that failed before any grader ran (no
+  `grader` or `metric`), and a metric a guardrail or the win metric names but
+  never scored (null value).
+
+## [0.2.0] - 2026-07-16
+
+### Changed
+- **Breaking:** the import package and CLI are now `evalcore` (were `evalkit`).
+  Update `import evalkit` to `import evalcore` and the `evalkit` command to
+  `evalcore`. The distribution name (`evalcore`) is unchanged.
+- Lowered the minimum Python to **3.11** (was 3.14).
+
+### Added
+- `evalcore.__version__`.
+- Public `evalcore.adapters.expand_env` for `${VAR}` expansion in custom
+  adapters (replaces the private `adapters._env` module).
+- Exception hierarchy: `EvalcoreError` (base) and `ConfigError` (also a
+  `ValueError`, so existing handlers keep working).
+- Top-level convenience entry points: `load_suite`, `load_cases`, `run_suite`,
+  `run_suite_sync`.
+- HTML rendering for `sweep` and `pairwise` reports, and `--report` /
+  `--report-out` on those CLI commands.
+
+## [0.1.0] - 2026-07-16
+
+- Initial public release: adapters (http/replay), graders (deterministic,
+  numeric, classification, LLM judge + panel), runner (N-sampling, concurrency,
+  retries, checkpoint/resume), compare/gate, sweep, pairwise, blind human
+  rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
+  column-store outbox, and content-hash provenance.
+
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/2.1.0...HEAD
+[2.1.0]: https://github.com/scottpmiller/evalcore/compare/2.0.0...2.1.0
+[2.0.0]: https://github.com/scottpmiller/evalcore/compare/1.0.0...2.0.0
+[1.0.0]: https://github.com/scottpmiller/evalcore/compare/0.3.0...1.0.0
+[0.3.0]: https://github.com/scottpmiller/evalcore/compare/0.2.0...0.3.0
+[0.2.0]: https://github.com/scottpmiller/evalcore/compare/0.1.0...0.2.0
+[0.1.0]: https://github.com/scottpmiller/evalcore/releases/tag/0.1.0

{evalcore-1.0.0 → evalcore-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalcore
-Version: 1.0.0
+Version: 2.1.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -695,7 +695,7 @@ shorthand; a repeatable `--view label:kind:ref` gives explicit control.
 Sessions are **resumable** (a rater only sees items they haven't scored).
 **Blinding is enforced server-side** - the queue payload carries an opaque
 item id and never the run/variant/model; ratings map back to
-`(run_id, case_id, sample_idx)` only on the server. Ratings land in a JSONL
+`(run_id, case_id, sample_hash)` only on the server. Ratings land in a JSONL
 file (`models.Rating`) that is the **open interchange format**: any external
 tool or spreadsheet export in the same shape feeds `agreement` too.
 
@@ -774,13 +774,15 @@ scored value, the invocation it came from, the gate verdict, and the full
 reproducibility key (incl. `run_id`), so a multi-tenant trend table can
 filter/group on any dimension without joins. Nothing derived is written: a
 run's scorecard is a read-time aggregation over the same rows, so it cannot
-disagree with the scores behind it. A missing measurement is `null` (those
-columns are `Nullable`, and a real `0.0` is a meaningful score); `passed` is
-the tri-state string `'true'|'false'|'null'`, matching its `Enum8`. Row keys
-are the store's column names, so a JSONEachRow-style feed maps onto the schema
-in
-[`docs/clickhouse-schema.sql`](docs/clickhouse-schema.sql). Swap the exporter
-for a real database client without touching the runner or any consumer.
+disagree with the scores behind it. A measurement that does not exist is
+`null`, not `0` - a real `0.0` is a meaningful score, and `avg`/`sum`/`count`
+skip nulls, so no query has to remember a filter; `passed` is the tri-state
+string `'true'|'false'|'null'`, since a judge has no pass line by design and
+that is a third state rather than a missing value. Row keys are column names
+rather than model attribute names, so a JSONEachRow-style feed lands in a flat
+table without a mapping layer. A store that forbids nullable columns fills
+those nulls in at ingest, on its side of the seam. Swap the exporter for a real
+database client without touching the runner or any consumer.
 
 ---
 
@@ -827,19 +829,18 @@ src/evalcore/
 tests/               engine unit tests
 examples/quickstart  a runnable consumer that doubles as an implementation test
 docs/design.md       the design overview
-docs/clickhouse-schema.sql  the results-store schema the outbox rows target
 ```
 
 ## Status
 
-1.0. The public API and the outbox row shape are stable; a breaking change to
-either means a 2.0. Built: deterministic + classification + **LLM-judge**
+2.0. The public API and the outbox row shape are stable; a breaking change to
+either means a 3.0. Built: deterministic + classification + **LLM-judge**
 (rubric scoring; single judge or a Claude/GPT **panel** with per-dimension
 means, per-judge overalls, inter-judge disagreement flagging, and
 image/screenshot inputs) graders, http/replay/browser adapters, runner
 (N-sampling, optional concurrency, per-sample `RunResult` + `run_id` +
-variance), comparison/gate, JSON + run + outbox store (one score-grain feed,
-`docs/clickhouse-schema.sql`), **N-way sweeps
+variance), comparison/gate, JSON + run + outbox store (one score-grain feed),
+**N-way sweeps
 + counterbalanced pairwise A-vs-B win-rate** (`sweep`/`pairwise`), **blind
 human-rating + side-by-side ranking web apps** with judge↔human agreement and
 human-vs-judge pairwise agreement (`rate`/`agreement`, `rank`/`preferences`),

{evalcore-1.0.0 → evalcore-2.1.0}/README.md

@@ -663,7 +663,7 @@ shorthand; a repeatable `--view label:kind:ref` gives explicit control.
 Sessions are **resumable** (a rater only sees items they haven't scored).
 **Blinding is enforced server-side** - the queue payload carries an opaque
 item id and never the run/variant/model; ratings map back to
-`(run_id, case_id, sample_idx)` only on the server. Ratings land in a JSONL
+`(run_id, case_id, sample_hash)` only on the server. Ratings land in a JSONL
 file (`models.Rating`) that is the **open interchange format**: any external
 tool or spreadsheet export in the same shape feeds `agreement` too.
 
@@ -742,13 +742,15 @@ scored value, the invocation it came from, the gate verdict, and the full
 reproducibility key (incl. `run_id`), so a multi-tenant trend table can
 filter/group on any dimension without joins. Nothing derived is written: a
 run's scorecard is a read-time aggregation over the same rows, so it cannot
-disagree with the scores behind it. A missing measurement is `null` (those
-columns are `Nullable`, and a real `0.0` is a meaningful score); `passed` is
-the tri-state string `'true'|'false'|'null'`, matching its `Enum8`. Row keys
-are the store's column names, so a JSONEachRow-style feed maps onto the schema
-in
-[`docs/clickhouse-schema.sql`](docs/clickhouse-schema.sql). Swap the exporter
-for a real database client without touching the runner or any consumer.
+disagree with the scores behind it. A measurement that does not exist is
+`null`, not `0` - a real `0.0` is a meaningful score, and `avg`/`sum`/`count`
+skip nulls, so no query has to remember a filter; `passed` is the tri-state
+string `'true'|'false'|'null'`, since a judge has no pass line by design and
+that is a third state rather than a missing value. Row keys are column names
+rather than model attribute names, so a JSONEachRow-style feed lands in a flat
+table without a mapping layer. A store that forbids nullable columns fills
+those nulls in at ingest, on its side of the seam. Swap the exporter for a real
+database client without touching the runner or any consumer.
 
 ---
 
@@ -795,19 +797,18 @@ src/evalcore/
 tests/               engine unit tests
 examples/quickstart  a runnable consumer that doubles as an implementation test
 docs/design.md       the design overview
-docs/clickhouse-schema.sql  the results-store schema the outbox rows target
 ```
 
 ## Status
 
-1.0. The public API and the outbox row shape are stable; a breaking change to
-either means a 2.0. Built: deterministic + classification + **LLM-judge**
+2.0. The public API and the outbox row shape are stable; a breaking change to
+either means a 3.0. Built: deterministic + classification + **LLM-judge**
 (rubric scoring; single judge or a Claude/GPT **panel** with per-dimension
 means, per-judge overalls, inter-judge disagreement flagging, and
 image/screenshot inputs) graders, http/replay/browser adapters, runner
 (N-sampling, optional concurrency, per-sample `RunResult` + `run_id` +
-variance), comparison/gate, JSON + run + outbox store (one score-grain feed,
-`docs/clickhouse-schema.sql`), **N-way sweeps
+variance), comparison/gate, JSON + run + outbox store (one score-grain feed),
+**N-way sweeps
 + counterbalanced pairwise A-vs-B win-rate** (`sweep`/`pairwise`), **blind
 human-rating + side-by-side ranking web apps** with judge↔human agreement and
 human-vs-judge pairwise agreement (`rate`/`agreement`, `rank`/`preferences`),

{evalcore-1.0.0 → evalcore-2.1.0}/docs/design.md

@@ -127,19 +127,32 @@ Scorecards and comparisons serialize to JSON for CI artifacts and local files.
 For trend tracking, results can land in a column store (e.g. ClickHouse) keyed
 by `project`/`suite`. Rather than couple the engine to any particular database
 driver, `store.JsonlOutboxExporter` flattens a run into a stable, flat row
-shape and writes JSONL to an **outbox** a separate shipper drains:
-
-- one **metric** row per scorecard metric (`metric, value, stdev, metric_kind,
-  n`), and
-- one **per-case score** row (`run_id, case_id, sample_idx, grader, metric,
-  value, passed, detail`).
-
-Both repeat the full reproducibility key so a multi-tenant trend table can
-filter/group on any dimension without joins. The rows use a no-`Nullable`
-convention that maps cleanly onto a column store (a missing value is the
-sentinel pair `(value=0, has_value=false)`; `passed` is the tri-state string
-`'true'|'false'|'null'`). Swap the exporter for a real database client without
-touching the runner or any consumer.
+shape and writes JSONL to an **outbox** a separate shipper drains: one feed at
+`(run, case, sample, grader, metric)` grain, one row per score
+(`run_id, case_id, sample_hash, grader, metric, metric_kind, value, passed,
+detail`), carrying the invocation it came from and the gate verdict alongside.
+
+Nothing derived is written. A run's scorecard and its trend are read-time
+aggregations over the same rows, so they cannot disagree with the scores behind
+them.
+
+Every row repeats the full reproducibility key so a multi-tenant trend table can
+filter/group on any dimension without joins.
+
+A measurement that does not exist is `null`. A real `0.0` is a meaningful score -
+what a failing deterministic check earns - so filling an absent one in with 0
+would make the two unreadable apart, and `avg`/`sum`/`count` skip nulls anyway,
+so no query has to remember a filter. `metric_kind` carries the engine's word
+verbatim and says nothing about presence, so a metric some cases could not score
+still reads as one metric; `'none'` is reserved for the two row shapes that hold
+no score at all. `passed` is the tri-state string
+`'true'|'false'|'null'`, since a judge has no pass line by design: a third
+state, not a missing value.
+
+A store that forbids nullable columns is free to fill those nulls in at ingest.
+That mapping belongs at its boundary, not in the engine, which keeps absence
+representable all the way out. Swap the exporter for a real database client
+without touching the runner or any consumer.
 
 ## 7. Provenance
 

{evalcore-1.0.0 → evalcore-2.1.0}/justfile

@@ -1,4 +1,4 @@
-# evalkit - a generic, consumer-agnostic eval engine. Standard uv project.
+# evalcore - a generic, consumer-agnostic eval engine. Standard uv project.
 
 # Sync the environment (editable install + dev/extras).
 sync:
@@ -23,7 +23,7 @@ lint:
 
 # Run the quickstart suite offline against recorded fixtures.
 example:
-    uv run evalkit --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
+    uv run evalcore --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
 
 # Run the quickstart suite through the Python API (no CLI), offline.
 example-api:
@@ -31,4 +31,4 @@ example-api:
 
 # Head-to-head A-vs-B win-rate over the quickstart suite (offline).
 example-pairwise:
-    uv run evalkit --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay
+    uv run evalcore --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay

{evalcore-1.0.0 → evalcore-2.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "1.0.0"
+version = "2.1.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-1.0.0 → evalcore-2.1.0}/src/evalcore/graders/__init__.py

@@ -20,5 +20,14 @@ from evalcore.graders import (
     judge,
     numeric,
 )
+from evalcore.graders.base import GraderType, category_of
 
-__all__ = ['base', 'classification', 'deterministic', 'judge', 'numeric']
+__all__ = [
+    'GraderType',
+    'base',
+    'category_of',
+    'classification',
+    'deterministic',
+    'judge',
+    'numeric',
+]

evalcore-2.1.0/src/evalcore/graders/base.py

@@ -0,0 +1,145 @@
+"""Grader protocols and the type registry.
+
+A grader spec is a plain dict from the suite config: ``{type, name, ...}``.
+``build_graders`` turns a list of specs into grader instances, split into the
+per-case and aggregate buckets the runner needs.
+"""
+
+import enum
+import typing
+
+from evalcore import models
+from evalcore.errors import ConfigError
+
+
+class GraderType(enum.StrEnum):
+    """What kind of check a grader performs, not which one.
+
+    A closed set, unlike the registry's ``type`` names, which any consumer
+    may extend. Declared once per grader at registration and read back by
+    ``store.grader_lookups``; a ``StrEnum`` so it needs no serializer of its
+    own on the way to a row.
+
+    The same set is spelled out in three other places, all of which have to
+    change together: ``GraderType`` in ``internal-eval-results``, the
+    ``grader_type`` ``Enum8`` in that repo's ``schema.sql``, and the deployed
+    DDL in ``schemata/clickhouse`` on GHE, which is the source of truth.
+
+    ``UNKNOWN`` exists for a producer with no registry behind it. Nothing in
+    evalcore emits it: ``register`` requires a category, so a grader that
+    reaches a suite has always declared one.
+
+    """
+
+    UNKNOWN = 'unknown'
+    HEURISTIC = 'heuristic'
+    STATISTICAL = 'statistical'
+    LLM_AS_JUDGE = 'llm_as_judge'
+    TRAJECTORY = 'trajectory'
+    HUMAN = 'human'
+
+
+@typing.runtime_checkable
+class Grader(typing.Protocol):
+    """Per-case grader. Scores are averaged across cases by the runner."""
+
+    name: str
+
+    def grade(
+        self, case: models.Case, output: models.Output
+    ) -> list[models.Score]: ...
+
+
+@typing.runtime_checkable
+class AggregateGrader(typing.Protocol):
+    """Whole-run grader for set-level metrics (P/R/F1, win-rate, ...)."""
+
+    name: str
+
+    def aggregate(
+        self, results: list[models.CaseResult]
+    ) -> list[models.Score]: ...
+
+
+_REGISTRY: dict[str, type] = {}
+
+_CATEGORIES: dict[str, GraderType] = {}
+
+
+def register(
+    type_name: str, category: GraderType
+) -> typing.Callable[[type], type]:
+    """Class decorator registering a grader under a suite-config ``type``.
+
+    ``category`` is required rather than defaulting, because it is the only
+    source of the row's ``grader_type`` and a default would be the value
+    every grader forgets to override. A grader's category belongs to its
+    implementation, not to a suite's use of it, so it is declared here and
+    not in the suite config.
+
+    Args:
+        type_name: The ``type`` a suite spec names to select this grader.
+        category: What kind of check it performs.
+
+    Returns:
+        The decorator.
+
+    Raises:
+        ConfigError: If ``type_name`` is already registered.
+
+    """
+
+    def _decorate(cls: type) -> type:
+        if type_name in _REGISTRY:
+            raise ConfigError(f'grader type {type_name!r} already registered')
+        _REGISTRY[type_name] = cls
+        _CATEGORIES[type_name] = GraderType(category)
+        return cls
+
+    return _decorate
+
+
+def category_of(type_name: str) -> GraderType:
+    """Return the category a grader type registered under.
+
+    Args:
+        type_name: The suite spec's ``type``.
+
+    Returns:
+        The declared category, or ``UNKNOWN`` for a type no plug-in has
+        registered. A suite naming one cannot run - ``build_graders``
+        raises - so ``UNKNOWN`` only reaches a row when a caller builds
+        rows without loading the plug-ins that produced them.
+
+    """
+    return _CATEGORIES.get(type_name, GraderType.UNKNOWN)
+
+
+def build_graders(
+    specs: list[dict],
+) -> tuple[list[Grader], list[AggregateGrader]]:
+    """Instantiate grader specs, partitioned into per-case and aggregate.
+
+    Each spec's ``type`` selects a registered class; remaining keys (minus
+    ``type``) are passed as keyword arguments to its constructor.
+    """
+    per_case: list[Grader] = []
+    aggregate: list[AggregateGrader] = []
+    for spec in specs:
+        spec = dict(spec)
+        type_name = spec.pop('type')
+        if type_name not in _REGISTRY:
+            raise ConfigError(
+                f'unknown grader type {type_name!r}; '
+                f'known: {sorted(_REGISTRY)}'
+            )
+        grader = _REGISTRY[type_name](**spec)
+        if isinstance(grader, AggregateGrader):
+            aggregate.append(grader)
+        elif isinstance(grader, Grader):
+            per_case.append(grader)
+        else:  # pragma: no cover - defensive
+            raise TypeError(
+                f'{type_name!r} is neither Grader nor AggregateGrader'
+            )
+    return per_case, aggregate

{evalcore-1.0.0 → evalcore-2.1.0}/src/evalcore/graders/classification.py

@@ -19,7 +19,7 @@ def _safe_div(numerator: float, denominator: float) -> float:
     return numerator / denominator if denominator else 0.0
 
 
-@base.register('classification')
+@base.register('classification', base.GraderType.STATISTICAL)
 class Classification:
     """Binary precision/recall/F1 + FN/FP rates over a labeled dataset."""
 

{evalcore-1.0.0 → evalcore-2.1.0}/src/evalcore/graders/deterministic.py

@@ -32,7 +32,7 @@ def _score(name: str, metric: str, case_id: str, ok: bool, detail: str):
     )
 
 
-@base.register('max_chars')
+@base.register('max_chars', base.GraderType.HEURISTIC)
 class MaxChars:
     """Assert a text field is at most ``maximum`` characters long."""
 
@@ -58,7 +58,7 @@ class MaxChars:
         ]
 
 
-@base.register('regex_absent')
+@base.register('regex_absent', base.GraderType.HEURISTIC)
 class RegexAbsent:
     """Assert a text field does NOT match ``pattern`` (e.g. no tokens)."""
 
@@ -78,7 +78,7 @@ class RegexAbsent:
         return [_score(self.name, self.name, case.id, ok, detail)]
 
 
-@base.register('regex_present')
+@base.register('regex_present', base.GraderType.HEURISTIC)
 class RegexPresent:
     """Assert a text field matches EVERY pattern in ``patterns`` (all-of).
 
@@ -111,7 +111,7 @@ class RegexPresent:
         return [_score(self.name, self.name, case.id, ok, detail)]
 
 
-@base.register('non_empty')
+@base.register('non_empty', base.GraderType.HEURISTIC)
 class NonEmpty:
     """Assert a field resolves to a non-empty value."""
 

{evalcore-1.0.0 → evalcore-2.1.0}/src/evalcore/graders/judge.py

@@ -289,7 +289,7 @@ def _load_images(refs_values: list) -> list[dict]:
     return images
 
 
-@base.register('llm_judge')
+@base.register('llm_judge', base.GraderType.LLM_AS_JUDGE)
 class RubricJudge:
     """Score an output's text on rubric dimensions with an LLM judge/panel.
 

{evalcore-1.0.0 → evalcore-2.1.0}/src/evalcore/graders/numeric.py

@@ -48,7 +48,7 @@ def _as_float(value) -> float | None:
         return None
 
 
-@base.register('numeric')
+@base.register('numeric', base.GraderType.HEURISTIC)
 class Numeric:
     """Surface numeric output fields as metrics, optionally range-checked."""