evalcore 1.0.0__tar.gz → 2.0.0__tar.gz

{evalcore-1.0.0 → evalcore-2.0.0}/CHANGELOG.md

@@ -6,6 +6,56 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.0.0] - 2026-07-31
+
+Breaks both the public API and the outbox row shape, so it's a major per the
+1.0.0 policy.
+
+### Changed
+- **Breaking:** a sample is identified by `sample_hash` - a content digest of
+  the output it produced - rather than by the ordinal `sample_idx`. A row now
+  names the exact response behind it. The field is renamed on `CaseResult`,
+  `Rating`, `Preference`, `PairwiseOutcome` and `PairwiseAgreementCase`, and in
+  the outbox rows. Two runs of the same case never share a hash, so anything
+  comparing runs aligns on `case_id` and sample order; pairwise and the ranking
+  app anchor a pair on the `variant_a` side's digest so human and judge picks
+  still join. **Ratings and preferences files written by 1.x do not carry a
+  hash and will not join to a run** - re-collect them, or backfill the field.
+- **Breaking:** outbox keys track the store's column names: `created_at` is
+  emitted as `started_at`, `n_cases` as `case_count`, `n_samples` as
+  `sample_count`.
+- **Breaking:** `store.read_checkpoint_results` is now
+  `store.read_checkpoint_samples` and returns `(ordinal, result)` pairs.
+  Checkpoint lines nest the result under `result` and tag it with `sample`, so
+  resume knows which samples are still owed. 1.x checkpoints cannot be resumed.
+
+### Fixed
+- Resuming a run with `concurrency > 1` could skip a sample and re-run another.
+  A concurrent run checkpoints in completion order, so an interrupt leaves a
+  hole rather than a clean prefix; resume now reruns the samples that are
+  actually missing. On a deterministic target the re-run collided with a digest
+  already recorded, so the store collapsed two samples into one row.
+- The pairwise judge and the side-by-side ranking app filtered samples
+  differently - non-empty content vs. no error - so with `n_samples > 1` and an
+  error on either side they paired A's n-th sample against different B samples.
+  `agreement` joins the two on A's digest, so it scored two different
+  comparisons as one. Both now go through
+  `pairwise.comparable_samples`, which requires a successful invocation *and*
+  resolvable content. Two behaviour changes fall out: an errored output is no
+  longer judged even when its content ref still resolves, and a sample with
+  empty content is no longer shown to a rater as a blank panel.
+
+### Changed (internal)
+- `pairwise._content_map` is now `pairwise.comparable_samples` and is the one
+  place that decides whether a sample can take part in a comparison.
+
+### Removed
+- `docs/clickhouse-schema.{sql,html}`. The outbox targets a flat row shape, not
+  one vendor's DDL, and the file documented a specific deployment - database
+  name, ingestion topology, sample data - none of which the engine needs. A
+  store with column types the feed does not match maps the rows at its own
+  boundary.
+
 ## [1.0.0] - 2026-07-28
 
 First stable release. The public API and the outbox row shape are now covered
@@ -20,9 +70,9 @@ by semantic versioning: a breaking change to either means a 2.0.
 
 ### Added
 - `docs/clickhouse-schema.sql`: the ClickHouse schema the outbox rows target -
-  the written `evaluation_scores` table, the run-grain `evaluations` view over
-  it, the row grammar as `CONSTRAINT`s, and the trend queries. Also rendered as
-  `docs/clickhouse-schema.html`.
+  the written `evaluation_scores` table, the row grammar as `CONSTRAINT`s, and
+  the canonical scorecard query. Also rendered as
+  `docs/clickhouse-schema.html`. (Both removed again in 2.0.0.)
 
 ## [0.3.0] - 2026-07-28
 

{evalcore-1.0.0 → evalcore-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalcore
-Version: 1.0.0
+Version: 2.0.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore
@@ -695,7 +695,7 @@ shorthand; a repeatable `--view label:kind:ref` gives explicit control.
 Sessions are **resumable** (a rater only sees items they haven't scored).
 **Blinding is enforced server-side** - the queue payload carries an opaque
 item id and never the run/variant/model; ratings map back to
-`(run_id, case_id, sample_idx)` only on the server. Ratings land in a JSONL
+`(run_id, case_id, sample_hash)` only on the server. Ratings land in a JSONL
 file (`models.Rating`) that is the **open interchange format**: any external
 tool or spreadsheet export in the same shape feeds `agreement` too.
 
@@ -774,13 +774,15 @@ scored value, the invocation it came from, the gate verdict, and the full
 reproducibility key (incl. `run_id`), so a multi-tenant trend table can
 filter/group on any dimension without joins. Nothing derived is written: a
 run's scorecard is a read-time aggregation over the same rows, so it cannot
-disagree with the scores behind it. A missing measurement is `null` (those
-columns are `Nullable`, and a real `0.0` is a meaningful score); `passed` is
-the tri-state string `'true'|'false'|'null'`, matching its `Enum8`. Row keys
-are the store's column names, so a JSONEachRow-style feed maps onto the schema
-in
-[`docs/clickhouse-schema.sql`](docs/clickhouse-schema.sql). Swap the exporter
-for a real database client without touching the runner or any consumer.
+disagree with the scores behind it. A measurement that does not exist is
+`null`, not `0` - a real `0.0` is a meaningful score, and `avg`/`sum`/`count`
+skip nulls, so no query has to remember a filter; `passed` is the tri-state
+string `'true'|'false'|'null'`, since a judge has no pass line by design and
+that is a third state rather than a missing value. Row keys are column names
+rather than model attribute names, so a JSONEachRow-style feed lands in a flat
+table without a mapping layer. A store that forbids nullable columns fills
+those nulls in at ingest, on its side of the seam. Swap the exporter for a real
+database client without touching the runner or any consumer.
 
 ---
 
@@ -827,19 +829,18 @@ src/evalcore/
 tests/               engine unit tests
 examples/quickstart  a runnable consumer that doubles as an implementation test
 docs/design.md       the design overview
-docs/clickhouse-schema.sql  the results-store schema the outbox rows target
 ```
 
 ## Status
 
-1.0. The public API and the outbox row shape are stable; a breaking change to
-either means a 2.0. Built: deterministic + classification + **LLM-judge**
+2.0. The public API and the outbox row shape are stable; a breaking change to
+either means a 3.0. Built: deterministic + classification + **LLM-judge**
 (rubric scoring; single judge or a Claude/GPT **panel** with per-dimension
 means, per-judge overalls, inter-judge disagreement flagging, and
 image/screenshot inputs) graders, http/replay/browser adapters, runner
 (N-sampling, optional concurrency, per-sample `RunResult` + `run_id` +
-variance), comparison/gate, JSON + run + outbox store (one score-grain feed,
-`docs/clickhouse-schema.sql`), **N-way sweeps
+variance), comparison/gate, JSON + run + outbox store (one score-grain feed),
+**N-way sweeps
 + counterbalanced pairwise A-vs-B win-rate** (`sweep`/`pairwise`), **blind
 human-rating + side-by-side ranking web apps** with judge↔human agreement and
 human-vs-judge pairwise agreement (`rate`/`agreement`, `rank`/`preferences`),

{evalcore-1.0.0 → evalcore-2.0.0}/README.md

@@ -663,7 +663,7 @@ shorthand; a repeatable `--view label:kind:ref` gives explicit control.
 Sessions are **resumable** (a rater only sees items they haven't scored).
 **Blinding is enforced server-side** - the queue payload carries an opaque
 item id and never the run/variant/model; ratings map back to
-`(run_id, case_id, sample_idx)` only on the server. Ratings land in a JSONL
+`(run_id, case_id, sample_hash)` only on the server. Ratings land in a JSONL
 file (`models.Rating`) that is the **open interchange format**: any external
 tool or spreadsheet export in the same shape feeds `agreement` too.
 
@@ -742,13 +742,15 @@ scored value, the invocation it came from, the gate verdict, and the full
 reproducibility key (incl. `run_id`), so a multi-tenant trend table can
 filter/group on any dimension without joins. Nothing derived is written: a
 run's scorecard is a read-time aggregation over the same rows, so it cannot
-disagree with the scores behind it. A missing measurement is `null` (those
-columns are `Nullable`, and a real `0.0` is a meaningful score); `passed` is
-the tri-state string `'true'|'false'|'null'`, matching its `Enum8`. Row keys
-are the store's column names, so a JSONEachRow-style feed maps onto the schema
-in
-[`docs/clickhouse-schema.sql`](docs/clickhouse-schema.sql). Swap the exporter
-for a real database client without touching the runner or any consumer.
+disagree with the scores behind it. A measurement that does not exist is
+`null`, not `0` - a real `0.0` is a meaningful score, and `avg`/`sum`/`count`
+skip nulls, so no query has to remember a filter; `passed` is the tri-state
+string `'true'|'false'|'null'`, since a judge has no pass line by design and
+that is a third state rather than a missing value. Row keys are column names
+rather than model attribute names, so a JSONEachRow-style feed lands in a flat
+table without a mapping layer. A store that forbids nullable columns fills
+those nulls in at ingest, on its side of the seam. Swap the exporter for a real
+database client without touching the runner or any consumer.
 
 ---
 
@@ -795,19 +797,18 @@ src/evalcore/
 tests/               engine unit tests
 examples/quickstart  a runnable consumer that doubles as an implementation test
 docs/design.md       the design overview
-docs/clickhouse-schema.sql  the results-store schema the outbox rows target
 ```
 
 ## Status
 
-1.0. The public API and the outbox row shape are stable; a breaking change to
-either means a 2.0. Built: deterministic + classification + **LLM-judge**
+2.0. The public API and the outbox row shape are stable; a breaking change to
+either means a 3.0. Built: deterministic + classification + **LLM-judge**
 (rubric scoring; single judge or a Claude/GPT **panel** with per-dimension
 means, per-judge overalls, inter-judge disagreement flagging, and
 image/screenshot inputs) graders, http/replay/browser adapters, runner
 (N-sampling, optional concurrency, per-sample `RunResult` + `run_id` +
-variance), comparison/gate, JSON + run + outbox store (one score-grain feed,
-`docs/clickhouse-schema.sql`), **N-way sweeps
+variance), comparison/gate, JSON + run + outbox store (one score-grain feed),
+**N-way sweeps
 + counterbalanced pairwise A-vs-B win-rate** (`sweep`/`pairwise`), **blind
 human-rating + side-by-side ranking web apps** with judge↔human agreement and
 human-vs-judge pairwise agreement (`rate`/`agreement`, `rank`/`preferences`),

{evalcore-1.0.0 → evalcore-2.0.0}/docs/design.md

@@ -127,19 +127,32 @@ Scorecards and comparisons serialize to JSON for CI artifacts and local files.
 For trend tracking, results can land in a column store (e.g. ClickHouse) keyed
 by `project`/`suite`. Rather than couple the engine to any particular database
 driver, `store.JsonlOutboxExporter` flattens a run into a stable, flat row
-shape and writes JSONL to an **outbox** a separate shipper drains:
-
-- one **metric** row per scorecard metric (`metric, value, stdev, metric_kind,
-  n`), and
-- one **per-case score** row (`run_id, case_id, sample_idx, grader, metric,
-  value, passed, detail`).
-
-Both repeat the full reproducibility key so a multi-tenant trend table can
-filter/group on any dimension without joins. The rows use a no-`Nullable`
-convention that maps cleanly onto a column store (a missing value is the
-sentinel pair `(value=0, has_value=false)`; `passed` is the tri-state string
-`'true'|'false'|'null'`). Swap the exporter for a real database client without
-touching the runner or any consumer.
+shape and writes JSONL to an **outbox** a separate shipper drains: one feed at
+`(run, case, sample, grader, metric)` grain, one row per score
+(`run_id, case_id, sample_hash, grader, metric, metric_kind, value, passed,
+detail`), carrying the invocation it came from and the gate verdict alongside.
+
+Nothing derived is written. A run's scorecard and its trend are read-time
+aggregations over the same rows, so they cannot disagree with the scores behind
+them.
+
+Every row repeats the full reproducibility key so a multi-tenant trend table can
+filter/group on any dimension without joins.
+
+A measurement that does not exist is `null`. A real `0.0` is a meaningful score -
+what a failing deterministic check earns - so filling an absent one in with 0
+would make the two unreadable apart, and `avg`/`sum`/`count` skip nulls anyway,
+so no query has to remember a filter. `metric_kind` carries the engine's word
+verbatim and says nothing about presence, so a metric some cases could not score
+still reads as one metric; `'none'` is reserved for the two row shapes that hold
+no score at all. `passed` is the tri-state string
+`'true'|'false'|'null'`, since a judge has no pass line by design: a third
+state, not a missing value.
+
+A store that forbids nullable columns is free to fill those nulls in at ingest.
+That mapping belongs at its boundary, not in the engine, which keeps absence
+representable all the way out. Swap the exporter for a real database client
+without touching the runner or any consumer.
 
 ## 7. Provenance
 

{evalcore-1.0.0 → evalcore-2.0.0}/justfile

@@ -1,4 +1,4 @@
-# evalkit - a generic, consumer-agnostic eval engine. Standard uv project.
+# evalcore - a generic, consumer-agnostic eval engine. Standard uv project.
 
 # Sync the environment (editable install + dev/extras).
 sync:
@@ -23,7 +23,7 @@ lint:
 
 # Run the quickstart suite offline against recorded fixtures.
 example:
-    uv run evalkit --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
+    uv run evalcore --plugins examples.quickstart.graders gate --suite examples/quickstart/suite.yaml --mode replay
 
 # Run the quickstart suite through the Python API (no CLI), offline.
 example-api:
@@ -31,4 +31,4 @@ example-api:
 
 # Head-to-head A-vs-B win-rate over the quickstart suite (offline).
 example-pairwise:
-    uv run evalkit --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay
+    uv run evalcore --plugins examples.quickstart.graders pairwise --suite examples/quickstart/suite.yaml --a baseline --b candidate --mode replay

{evalcore-1.0.0 → evalcore-2.0.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "1.0.0"
+version = "2.0.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-1.0.0 → evalcore-2.0.0}/src/evalcore/models.py

@@ -99,11 +99,18 @@ class Score(pydantic.BaseModel):
 
 
 class CaseResult(pydantic.BaseModel):
-    """The output + per-case scores for one (case, sample)."""
+    """The output + per-case scores for one (case, sample).
+
+    ``sample_hash`` is a content digest of the output this sample produced,
+    minted by the runner. It identifies the sample by what came back rather
+    than by position, so a row is traceable to the exact response behind it.
+    Two runs of the same case do not share a hash - anything comparing runs
+    aligns on ``case.id`` and sample order, not on this field.
+    """
 
     case: Case
     variant_name: str
-    sample_idx: int
+    sample_hash: str
     output: Output
     scores: list[Score] = pydantic.Field(default_factory=list)
 
@@ -182,11 +189,14 @@ class Rating(pydantic.BaseModel):
     ingestible from any external tool. ``scores`` maps rubric dimension ->
     an integer on the same 1..scale the judge used, so human and judge are
     directly comparable.
+
+    ``sample_hash`` is the rated output's digest from
+    :class:`CaseResult`, so a rating points at the exact response a human saw.
     """
 
     run_id: str
     case_id: str
-    sample_idx: int = 0
+    sample_hash: str = ''
     rater: str
     scores: dict[str, int] = pydantic.Field(default_factory=dict)
     rated_at: str | None = None
@@ -202,10 +212,16 @@ class Preference(pydantic.BaseModel):
     counterbalances left/right per rater and un-blinds server-side, so a
     stored ``'a'`` always means ``variant_a`` won regardless of which side it
     was shown on.
+
+    A comparison spans two runs, whose outputs hash differently, so
+    ``sample_hash`` carries the ``variant_a`` side's digest as the pair's
+    identity. :class:`PairwiseOutcome` anchors on the same side, which is what
+    lets the two join in
+    :func:`~evalcore.rating.compute_pairwise_agreement`.
     """
 
     case_id: str
-    sample_idx: int = 0
+    sample_hash: str = ''
     variant_a: str
     variant_b: str
     rater: str
@@ -270,10 +286,14 @@ class SweepResult(pydantic.BaseModel):
 
 
 class PairwiseOutcome(pydantic.BaseModel):
-    """The head-to-head result for one case (counterbalanced for order)."""
+    """The head-to-head result for one case (counterbalanced for order).
+
+    ``sample_hash`` is the ``variant_a`` output's digest, the same side
+    :class:`Preference` anchors on.
+    """
 
     case_id: str
-    sample_idx: int = 0
+    sample_hash: str = ''
     winner: typing.Literal['a', 'b', 'tie'] = 'tie'
     detail: str | None = None
 
@@ -339,7 +359,7 @@ class PairwiseAgreementCase(pydantic.BaseModel):
     """Human-panel vs LLM-judge winner for one case."""
 
     case_id: str
-    sample_idx: int = 0
+    sample_hash: str = ''
     human: typing.Literal['a', 'b', 'tie'] = 'tie'
     judge: typing.Literal['a', 'b', 'tie'] = 'tie'
     agree: bool = False

{evalcore-1.0.0 → evalcore-2.0.0}/src/evalcore/pairwise.py

@@ -4,8 +4,15 @@ Rubric scoring asks "how good is this output, 1..5"; pairwise asks the
 sharper question "is A better than B for this case?" and reports A's
 win-rate. It is a cross-variant operation the per-variant runner can't
 express as a grader (it needs both variants' output for the same case at
-once), so it lives here: align two runs by (case_id, sample_idx), ask a
-judge to pick a winner per case, aggregate.
+once), so it lives here: align two runs by case, ask a judge to pick a winner
+per case, aggregate.
+
+Alignment is by ``case_id`` and then sample order within the case. It cannot
+key on ``sample_hash``: that is a digest of the output, and the whole point of
+a comparison is that the two variants produced different output, so the hashes
+never match. A case's samples are repeat draws of one input, so pairing the
+n-th of A with the n-th of B is as meaningful as any other pairing. Each
+outcome records A's hash, so a pair traces back to a concrete response.
 
 Position bias (LLMs favour whichever option they see first) is handled by
 **counterbalancing**: each pair is judged in both orders and a pick that
@@ -207,21 +214,43 @@ def build_pairwise_client(mode: str, config: dict) -> PairwiseClient:
     return AnthropicPairwiseClient(model)
 
 
-def _content_map(
-    run: models.RunResult, content_ref: str
-) -> dict[tuple[str, int], tuple[str, models.CaseResult]]:
-    out: dict[tuple[str, int], tuple[str, models.CaseResult]] = {}
+def comparable_samples(
+    run: models.RunResult, content_ref: str | None
+) -> dict[str, list[tuple[str, models.CaseResult]]]:
+    """``case_id -> [(content, result)]`` for the samples a comparison can use.
+
+    A sample is comparable when the invocation succeeded and - if a
+    ``content_ref`` is given - that ref resolves to non-empty text. An errored
+    output has nothing meaningful to judge or to show a rater, and empty
+    content gives a judge nothing to weigh and a human an empty panel.
+    ``content`` is ``''`` when no ``content_ref`` was asked for.
+
+    Every path that pairs two runs has to filter through here, because
+    alignment within a case is positional: a sample one side keeps and the
+    other drops shifts every pair after it. The LLM judge would then compare
+    A's n-th sample against B's n-th while the human panel compared that same
+    A sample against a different B, and
+    :func:`~evalcore.rating.compute_pairwise_agreement` joins the two on A's
+    digest - so it would score two different comparisons as if they were one.
+    """
+    out: dict[str, list[tuple[str, models.CaseResult]]] = {}
     for result in run.results:
-        ctx = {
-            'input': result.case.input,
-            'expected': result.case.expected or {},
-            'output': result.output.fields,
-            'case': result.case.model_dump(),
-            'artifacts': result.output.artifacts,
-        }
-        content = refs.resolve_ref(ctx, content_ref)
-        if isinstance(content, str) and content:
-            out[result.case.id, result.sample_idx] = (content, result)
+        if result.output.error:
+            continue
+        content = ''
+        if content_ref:
+            ctx = {
+                'input': result.case.input,
+                'expected': result.case.expected or {},
+                'output': result.output.fields,
+                'case': result.case.model_dump(),
+                'artifacts': result.output.artifacts,
+            }
+            resolved = refs.resolve_ref(ctx, content_ref)
+            if not (isinstance(resolved, str) and resolved):
+                continue
+            content = resolved
+        out.setdefault(result.case.id, []).append((content, result))
     return out
 
 
@@ -249,15 +278,23 @@ async def judge_pairwise(
     judge_version: str = 'v1',
 ) -> models.PairwiseResult:
     """Compare two runs case-by-case and report A's win-rate."""
-    a_map = _content_map(run_a, content_ref)
-    b_map = _content_map(run_b, content_ref)
+    a_map = comparable_samples(run_a, content_ref)
+    b_map = comparable_samples(run_b, content_ref)
     context_refs = context_refs or {}
 
+    # Comparable pairs: only cases both variants produced, and within a case
+    # the n-th sample of A against the n-th of B. strict=False because an
+    # uneven sample count is expected - a case that errored on one side has
+    # fewer usable outputs there, and the extras have nothing to compare to.
+    pairs = [
+        (case_id, a_side, b_side)
+        for case_id in sorted(a_map.keys() & b_map.keys())
+        for a_side, b_side in zip(a_map[case_id], b_map[case_id], strict=False)
+    ]
+
     outcomes: list[models.PairwiseOutcome] = []
     a_wins = b_wins = ties = 0
-    for key in sorted(a_map.keys() & b_map.keys()):
-        content_a, result_a = a_map[key]
-        content_b, _ = b_map[key]
+    for case_id, (content_a, result_a), (content_b, _) in pairs:
         context = {
             label: refs.resolve_ref(
                 {
@@ -299,7 +336,10 @@ async def judge_pairwise(
             ties += 1
         outcomes.append(
             models.PairwiseOutcome(
-                case_id=key[0], sample_idx=key[1], winner=winner, detail=detail
+                case_id=case_id,
+                sample_hash=result_a.sample_hash,
+                winner=winner,
+                detail=detail,
             )
         )