evalcore 0.2.0__tar.gz → 0.3.0__tar.gz

evalcore-0.3.0/CHANGELOG.md

@@ -0,0 +1,77 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/), and the project follows
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.3.0] - 2026-07-28
+
+### Changed
+- **Breaking:** the results-store outbox is now one feed at
+  `(run, case, sample, grader, metric)` grain, targeting a single flat
+  `evaluation_scores` table with the run trend and the invocation grain as
+  plain views over it. `store.scorecard_rows` and
+  `JsonlOutboxExporter.export` are removed; use `store.score_rows` and
+  `export_scores`. `--export-scores` is now an alias for `--export`.
+- **Breaking:** a missing measurement is emitted as `null` rather than the
+  `(value=0, has_value=false)` sentinel pair, and the `has_value` / `has_stdev`
+  companion keys are gone. The store columns are `Nullable`, and a real `0.0` is
+  a meaningful score.
+- **Breaking:** outbox row keys are the store column names, so `project` is
+  emitted as `application`, `mode` as `adapter_mode`, `created_at` as
+  `timestamp`, and `Output.error` as `is_error` plus `error_text`.
+- **Breaking:** outbox rows no longer carry `model_id` or `prompt_version`. Both
+  are `variant.knobs.get(...)` projections and travel inside `variant_knobs`.
+- **Breaking:** `run_id` is now a dashed UUIDv7 rather than `uuid4().hex`, so it
+  parses as a ClickHouse `UUID` and carries its own creation time. Uses
+  `uuid.uuid7()` on 3.14 and an RFC 9562 implementation on 3.11 through 3.13.
+- Outbox rows now carry the gate: `gate_verdict`, `gate_win`,
+  `baseline_run_id`, `baseline_variant`, `win_baseline`, `win_candidate`,
+  `win_delta` and `gate_summary` at run grain, plus `win`, `guardrail` and
+  `guardrail_gap` on the metric each one refers to. Pass the `Comparison` to
+  `score_rows`; without it a run reads as ungated.
+
+### Added
+- `RunResult.aggregate_scores`, retaining the `kind='aggregate'` scores so a
+  store row keeps the grader that emitted them and what it reported. The
+  scorecard kept only their values.
+- `store.grader_lookups`, mapping grader names to a category and to a judge
+  scale from a suite's grader specs, since a `Score` carries neither.
+- Three outbox row shapes that previously had no representation: an aggregate
+  metric (no `case_id`), an invocation that failed before any grader ran (no
+  `grader` or `metric`), and a metric a guardrail or the win metric names but
+  never scored (null value).
+
+## [0.2.0] - 2026-07-16
+
+### Changed
+- **Breaking:** the import package and CLI are now `evalcore` (were `evalkit`).
+  Update `import evalkit` to `import evalcore` and the `evalkit` command to
+  `evalcore`. The distribution name (`evalcore`) is unchanged.
+- Lowered the minimum Python to **3.11** (was 3.14).
+
+### Added
+- `evalcore.__version__`.
+- Public `evalcore.adapters.expand_env` for `${VAR}` expansion in custom
+  adapters (replaces the private `adapters._env` module).
+- Exception hierarchy: `EvalcoreError` (base) and `ConfigError` (also a
+  `ValueError`, so existing handlers keep working).
+- Top-level convenience entry points: `load_suite`, `load_cases`, `run_suite`,
+  `run_suite_sync`.
+- HTML rendering for `sweep` and `pairwise` reports, and `--report` /
+  `--report-out` on those CLI commands.
+
+## [0.1.0] - 2026-07-16
+
+- Initial public release: adapters (http/replay), graders (deterministic,
+  numeric, classification, LLM judge + panel), runner (N-sampling, concurrency,
+  retries, checkpoint/resume), compare/gate, sweep, pairwise, blind human
+  rating + ranking with judge agreement, Markdown/HTML reporters, JSON +
+  column-store outbox, and content-hash provenance.
+
+[Unreleased]: https://github.com/scottpmiller/evalcore/compare/0.3.0...HEAD
+[0.3.0]: https://github.com/scottpmiller/evalcore/compare/0.2.0...0.3.0
+[0.2.0]: https://github.com/scottpmiller/evalcore/compare/0.1.0...0.2.0
+[0.1.0]: https://github.com/scottpmiller/evalcore/releases/tag/0.1.0

{evalcore-0.2.0 → evalcore-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalcore
-Version: 0.2.0
+Version: 0.3.0
 Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
 Project-URL: Homepage, https://github.com/scottpmiller/evalcore
 Project-URL: Repository, https://github.com/scottpmiller/evalcore

{evalcore-0.2.0 → evalcore-0.3.0}/docs/design.md

@@ -20,7 +20,7 @@ adapter + datasets + graders config," not "fork the framework." If it leaks,
 every new consumer reopens the core.
 
 This shape is well-trodden - promptfoo, OpenAI Evals, Braintrust, and LangSmith
-all converge on the same pipeline (case → target → graders → comparison →
+all converge on the same pipeline (case -> target -> graders -> comparison ->
 gate). evalcore is not inventing the abstraction; it is drawing the line so the
 engine stays reusable across unrelated systems.
 
@@ -28,21 +28,21 @@ engine stays reusable across unrelated systems.
 
 | Concern | **Engine** | **Consumer** |
 |---|---|---|
-| Case / suite schema + loading | ✅ | uses it |
-| Runner: sampling (N), concurrency, live/replay modes | ✅ | picks mode/config |
-| Target-adapter protocol + built-ins (HTTP, replay) | ✅ | configures/extends |
-| Variant model (baseline vs candidate knobs) | ✅ opaque dict | defines its knobs |
-| Grader protocol + registry | ✅ | registers its graders |
-| Generic graders: deterministic, LLM-judge, classification, numeric | ✅ | configures + writes custom |
-| Comparison / regression engine (win metric, guardrails) | ✅ | sets thresholds |
-| Gate decision (comparison → pass/warn/fail) | ✅ | sets policy |
-| Results store + row schema / outbox | ✅ | gets a `project` namespace |
-| Reporters (Markdown / HTML) | ✅ | gets them free |
-| Datasets (cases + labels) | ❌ | ✅ owns |
-| Judge **rubrics** | ❌ | ✅ owns |
-| Custom graders / adapters | ❌ | ✅ owns |
-| Thresholds / guardrail config | ❌ | ✅ owns |
-| Which suite runs on which change trigger | ❌ | ✅ owns |
+| Case / suite schema + loading | yes | uses it |
+| Runner: sampling (N), concurrency, live/replay modes | yes | picks mode/config |
+| Target-adapter protocol + built-ins (HTTP, replay) | yes | configures/extends |
+| Variant model (baseline vs candidate knobs) | yes opaque dict | defines its knobs |
+| Grader protocol + registry | yes | registers its graders |
+| Generic graders: deterministic, LLM-judge, classification, numeric | yes | configures + writes custom |
+| Comparison / regression engine (win metric, guardrails) | yes | sets thresholds |
+| Gate decision (comparison -> pass/warn/fail) | yes | sets policy |
+| Results store + row schema / outbox | yes | gets a `project` namespace |
+| Reporters (Markdown / HTML) | yes | gets them free |
+| Datasets (cases + labels) | no | yes owns |
+| Judge **rubrics** | no | yes owns |
+| Custom graders / adapters | no | yes owns |
+| Thresholds / guardrail config | no | yes owns |
+| Which suite runs on which change trigger | no | yes owns |
 
 Everything in the left column is consumer-agnostic. The right column is what a
 new team writes.
@@ -66,12 +66,12 @@ Eight concepts, none of which mention any particular system:
   own for anything else.
 - **Grader** - `grade(case, output) -> [Score]` where
   `Score = {metric, value, passed?, detail}`. Registry-based; tiered
-  (deterministic → numeric → classification → LLM judge).
-- **Run** - execute `suite × variant` over all cases × N samples → outputs +
+  (deterministic -> numeric -> classification -> LLM judge).
+- **Run** - execute `suite × variant` over all cases × N samples -> outputs +
   scores, persisted as a `RunResult`.
 - **Scorecard** - aggregated scores for one run (mean/stdev per metric).
-- **Comparison + Gate** - candidate scorecard vs baseline → deltas, guardrail
-  checks, win metric → `pass | warn | fail`.
+- **Comparison + Gate** - candidate scorecard vs baseline -> deltas, guardrail
+  checks, win metric -> `pass | warn | fail`.
 
 ```python
 class TargetAdapter(typing.Protocol):

{evalcore-0.2.0 → evalcore-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "evalcore"
-version = "0.2.0"
+version = "0.3.0"
 description = "A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs."
 readme = "README.md"
 requires-python = ">=3.11"

{evalcore-0.2.0 → evalcore-0.3.0}/src/evalcore/cli.py

@@ -160,19 +160,28 @@ def _cmd_gate(args: argparse.Namespace) -> int:
     )
     _emit_report(args, reporters.wrap_document(rep, body))
 
-    if args.export:
-        exporter = store.JsonlOutboxExporter(args.export)
-        exporter.export(baseline.scorecard)
-        exporter.export(candidate.scorecard)
-        print(f'\nexported scorecards -> {args.export}', file=sys.stderr)
-    if args.export_scores:
-        exporter = store.JsonlOutboxExporter(args.export_scores)
-        rows = exporter.export_scores(baseline)
-        rows += exporter.export_scores(candidate)
-        print(
-            f'exported {rows} score rows -> {args.export_scores}',
-            file=sys.stderr,
+    # One feed now, so --export and --export-scores name the same rows;
+    # --export-scores is kept as an alias for existing invocations.
+    outboxes = dict.fromkeys(
+        path for path in (args.export, args.export_scores) if path
+    )
+    if outboxes:
+        types, scales = store.grader_lookups(suite.graders)
+    for outbox in outboxes:
+        exporter = store.JsonlOutboxExporter(outbox)
+        # The baseline half of a pair is not itself gated, so it carries no
+        # comparison; the candidate carries the verdict and the guardrails.
+        rows = exporter.export_scores(
+            baseline, grader_types=types, judge_scales=scales
+        )
+        rows += exporter.export_scores(
+            candidate,
+            result,
+            baseline_run_id=baseline.run_id,
+            grader_types=types,
+            judge_scales=scales,
         )
+        print(f'\nexported {rows} store rows -> {outbox}', file=sys.stderr)
 
     return 0 if result.verdict != 'fail' else 1
 
@@ -201,12 +210,16 @@ def _cmd_sweep(args: argparse.Namespace) -> int:
         args, reporters.wrap_document(rep, reporters.render_sweep(rep, result))
     )
     if args.export:
+        types, scales = store.grader_lookups(suite.graders)
         exporter = store.JsonlOutboxExporter(args.export)
-        for run in runs:
-            exporter.export(run.scorecard)
+        rows = sum(
+            exporter.export_scores(
+                run, grader_types=types, judge_scales=scales
+            )
+            for run in runs
+        )
         print(
-            f'\nexported {len(runs)} scorecards -> {args.export}',
-            file=sys.stderr,
+            f'\nexported {rows} store rows -> {args.export}', file=sys.stderr
         )
     return 0
 
@@ -438,11 +451,13 @@ def build_parser() -> argparse.ArgumentParser:
         help='grader mode when it differs from --mode '
         '(e.g. live judges on replayed data)',
     )
-    gate.add_argument('--export', help='append scorecards to a JSONL outbox')
+    gate.add_argument(
+        '--export', help='append results-store rows to a JSONL outbox'
+    )
     gate.add_argument(
         '--export-scores',
         dest='export_scores',
-        help='append per-case score rows to a JSONL outbox (eval_scores)',
+        help='alias for --export, kept for existing invocations',
     )
     gate.add_argument(
         '--revision', help='opaque provenance id stamped on both scorecards'
@@ -474,7 +489,9 @@ def build_parser() -> argparse.ArgumentParser:
         help='grader mode when it differs from --mode '
         '(e.g. live judges on replayed data)',
     )
-    sweep.add_argument('--export', help='append scorecards to a JSONL outbox')
+    sweep.add_argument(
+        '--export', help='append results-store rows to a JSONL outbox'
+    )
     sweep.add_argument('--revision')
     sweep.add_argument(
         '--report',

{evalcore-0.2.0 → evalcore-0.3.0}/src/evalcore/models.py

@@ -162,11 +162,17 @@ class RunResult(pydantic.BaseModel):
     one entry per (case, sample) with the full output, artifacts, and
     per-case scores. Persisting it is what makes transcript review,
     human rating, and judge-agreement analysis possible after the fact.
+
+    ``aggregate_scores`` holds the ``kind='aggregate'`` scores computed once
+    over the whole run. The scorecard keeps their values but not which grader
+    emitted them or what it reported, so they are retained here too - a
+    results-store row needs the grader attribution.
     """
 
     run_id: str
     scorecard: Scorecard
     results: list[CaseResult] = pydantic.Field(default_factory=list)
+    aggregate_scores: list[Score] = pydantic.Field(default_factory=list)
 
 
 class Rating(pydantic.BaseModel):

{evalcore-0.2.0 → evalcore-0.3.0}/src/evalcore/runner.py

@@ -13,7 +13,9 @@ concurrent calls.
 
 import asyncio
 import inspect
+import secrets
 import statistics
+import time
 import uuid
 
 from evalcore import loader, models, store
@@ -47,6 +49,25 @@ async def _invoke_with_retry(
         attempt += 1
 
 
+def _uuid7() -> uuid.UUID:
+    """A UUIDv7 (RFC 9562): 48-bit millisecond timestamp, then random.
+
+    Time-ordered, so a run id carries its own creation time and ids sort in
+    the order the runs happened. ``uuid.uuid7`` is 3.14; this is the fallback
+    for 3.11 through 3.13.
+    """
+    if hasattr(uuid, 'uuid7'):
+        return uuid.uuid7()
+    ms = int(time.time() * 1000) & 0xFFFFFFFFFFFF
+    return uuid.UUID(
+        int=(ms << 80)
+        | (0x7 << 76)
+        | (secrets.randbits(12) << 64)
+        | (0b10 << 62)
+        | secrets.randbits(62)
+    )
+
+
 def _aggregate_metrics(
     results: list[models.CaseResult], agg_scores: list[models.Score]
 ) -> dict[str, models.MetricValue]:
@@ -140,7 +161,7 @@ async def run_suite(
 
     cases = loader.load_cases(suite.dataset)
     dataset_hash = loader.dataset_hash(cases)
-    run_id = uuid.uuid4().hex
+    run_id = str(_uuid7())
 
     # Resume: reuse already-completed (case, sample) results from a checkpoint
     # of the *same* eval; otherwise start (or restart) the checkpoint fresh.
@@ -272,7 +293,10 @@ async def run_suite(
         metrics=_aggregate_metrics(results, agg_scores),
     )
     return models.RunResult(
-        run_id=run_id, scorecard=scorecard, results=results
+        run_id=run_id,
+        scorecard=scorecard,
+        results=results,
+        aggregate_scores=agg_scores,
     )