evalcore 0.1.0__py3-none-any.whl

evalcore-0.1.0.dist-info/METADATA

@@ -0,0 +1,828 @@
+Metadata-Version: 2.4
+Name: evalcore
+Version: 0.1.0
+Summary: A generic, consumer-agnostic evaluation engine for prompt, model, and API outputs.
+Project-URL: Homepage, https://github.com/scottpmiller/evalcore
+Project-URL: Repository, https://github.com/scottpmiller/evalcore
+Project-URL: Issues, https://github.com/scottpmiller/evalcore/issues
+Author: AWeber Communications
+License-Expression: BSD-3-Clause
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.14
+Requires-Dist: pydantic<3,>=2.7
+Requires-Dist: pyyaml<7,>=6
+Provides-Extra: http
+Requires-Dist: httpx>=0.27; extra == 'http'
+Provides-Extra: judge
+Requires-Dist: anthropic>=0.40; extra == 'judge'
+Requires-Dist: openai>=1.40; extra == 'judge'
+Description-Content-Type: text/markdown
+
+# evalkit
+
+A small, **consumer-agnostic** evaluation engine for prompt, model, and API
+outputs. It detects regressions and improvements as prompts and models change,
+by scoring a candidate against a baseline over a fixed dataset and applying
+guardrails + a headline win metric to produce a gate verdict.
+
+evalkit knows nothing about any particular system under test. A consumer
+supplies four things; evalkit supplies everything else:
+
+| Consumer provides (data + small plug-ins) | evalkit provides |
+| --- | --- |
+| an **adapter** config (how to call the system + the knobs a variant sets) | runner (N-sampling), comparison/regression engine |
+| **datasets** (cases with opaque `input`/`expected` blobs) | grader registry + generic graders |
+| **graders** + judge rubrics | results store (JSON + column-store outbox) |
+| a **suite + thresholds** config | Markdown reporting, CLI, gate exit codes |
+
+See `docs/design.md` for the full design. A complete runnable consumer lives
+in `examples/quickstart/` (a support-reply eval: custom adapter + custom
+graders + deterministic/classification/LLM-judge checks, runnable fully
+offline) and doubles as an end-to-end usage reference.
+
+## Install
+
+```bash
+pip install evalcore          # distribution name; imports as `evalkit`
+```
+
+The PyPI package is **`evalcore`** (the name `evalkit` was taken); the import
+package and CLI are still `evalkit` (`import evalkit`, `evalkit --help`). Extras:
+`evalcore[http]`, `evalcore[judge]`.
+
+## Develop
+
+Standard [uv](https://docs.astral.sh/uv/) project; recipes via
+[just](https://github.com/casey/just):
+
+```bash
+just sync          # editable install + all extras + dev deps
+just test          # engine unit tests (coverage over src/)
+just test-example  # run the quickstart example assertions offline
+just example       # quickstart suite: scorecards + verdict (replay)
+just example-api   # quickstart suite through the Python API (replay)
+```
+
+Extras: `http` (httpx, needed only for live adapter runs) and `judge`
+(anthropic + openai SDKs, needed only for the live LLM judge). Replay/offline
+runs need none of them.
+
+---
+
+# Writing a consumer
+
+A consumer is a directory of data files plus (optionally) a small plug-in
+module. Nothing about your system leaks into the engine; everything below
+lives in *your* tree (your repo, or a directory under `examples/` here):
+
+```
+my_service/
+  suite.yaml                     # the suite: adapter + graders + variants + thresholds
+  graders.py                     # optional: custom graders/adapters (plug-in module)
+  datasets/<name>/v1/cases/*.yaml
+  fixtures/replay.yaml           # recorded outputs (offline runs / CI)
+  fixtures/judge.yaml            # recorded judgments (offline judge)
+```
+
+The five steps, in dependency order:
+
+## 1. Cases (the dataset)
+
+A dataset is a directory containing `cases/`, holding one YAML (or JSON)
+file per case. Every field except `id` is **opaque to the engine** — only
+your adapter and graders interpret `input` and `expected`:
+
+```yaml
+# datasets/support_reply/v1/cases/refund_request.yaml
+id: refund_request          # optional; defaults to the filename stem
+labels: {category: billing} # optional metadata, useful for slicing later
+input:                      # whatever YOUR adapter needs to call the system
+  ticket_text: 'I was charged twice for March...'
+  customer_tier: pro
+expected:                   # optional ground truth for graders
+  intent: refund
+```
+
+`loader.load_cases(dataset_dir)` reads every `cases/*.yaml|yml|json` in
+sorted filename order and validates them into `models.Case`. Version the
+dataset by directory (`v1`, `v2`, ...) and set `dataset_version` in the
+suite to match; the engine also computes a `dataset_hash` at run time (see
+"Provenance" below) so an edited case can't hide behind an unbumped version.
+
+## 2. The adapter (how to call your system)
+
+The adapter is the single seam to the system under test:
+
+```python
+class TargetAdapter(typing.Protocol):
+    async def invoke(self, case: Case, variant: Variant) -> Output: ...
+```
+
+It receives one `Case` and one `Variant` and must return an
+`Output` — never raise for a failed invocation; set `Output.error` instead
+so graders can count it (see `errors` below). Set `Output.retryable = True`
+alongside a *transient* error (a 429, a 5xx, a network timeout) and the
+runner's retry loop will back off and try again (see "Retries" below); leave
+it False for terminal failures (a bad request, a malformed response) so the
+run doesn't burn attempts on something a retry can't fix.
+
+### The built-in `http` adapter
+
+Declares the whole call as data in the suite file:
+
+```yaml
+adapter:
+  type: http
+  base_url: ${MY_SERVICE_BASE_URL}     # ${VAR} expands from the environment
+  path: /reply
+  method: POST                          # default POST
+  timeout: 30.0
+  headers:
+    Content-Type: application/json
+    Authorization: ${MY_SERVICE_JWT}    # header dropped entirely if VAR unset
+  body:                                 # template rendered per (case, variant)
+    ticket: $input.ticket_text          # $-strings are references, resolved
+    tier: $input.customer_tier          #   against the case/variant; anything
+    model: $variant.model               #   else passes through literally
+  extract:                              # Output.fields <- dotted paths into
+    reply: choices.0.text               #   the JSON response (list indices ok)
+    intent: analysis.intent
+```
+
+Reference roots available in `body`: `$input.*`, `$expected.*`,
+`$variant.*` (the variant's knobs), `$case.*` (the whole case). A path that
+doesn't resolve degrades to `null` rather than erroring. HTTP failures,
+non-2xx statuses, and non-JSON bodies all become `Output.error` values with
+the latency still recorded.
+
+### Custom adapters
+
+When the built-in isn't enough — auth dances, response post-processing,
+non-HTTP targets — register your own under a config `type` and select it in
+the suite. Subclassing the http adapter is often the shortest path.
+`examples/quickstart/adapter.py` is a worked example: it turns each case's
+input into structured `Output.fields` your graders can score:
+
+```python
+from evalkit import models
+from evalkit.adapters import base, http
+
+@base.register('my_service_json')
+class MyAdapter(http.HTTPAdapter):
+    async def invoke(self, case, variant):
+        output = await super().invoke(case, variant)
+        ...post-process output.fields...
+        return output
+```
+
+Constructor kwargs come from the suite's `adapter:` mapping (everything
+except `type`). Load the module at run time with `--plugins my_service.graders`
+(CLI) or a plain `import` (Python API) — registration happens on import.
+
+An adapter need not be HTTP-backed: it can grade *what a deployed system
+already did* by reading from an observability store — turning an aggregated
+result row into `Output.fields`.
+
+An adapter that holds resources (a browser, an injected session, pooled
+connections) may expose an optional async `aclose()`; the runner calls it
+after the run, even on failure — so a browser-automation adapter (e.g. one
+driving Playwright) can open its context once and tear it down cleanly.
+
+## 3. Variants (what's being compared)
+
+A variant is a named dict of **knobs** — opaque to the engine, interpreted
+by your adapter (usually via `$variant.*` refs in the body template):
+
+```yaml
+variants:
+  baseline:
+    model: claude-haiku-4-5
+    prompt_version: v3
+  candidate:
+    model: claude-sonnet-4-6
+    prompt_version: v4
+```
+
+Two knob names get special treatment: `model` and `prompt_version` are
+lifted onto the scorecard as `model_id` / `prompt_version` so results are
+self-describing in the store. Everything else is yours.
+
+## 4. Graders (how outputs are scored)
+
+Two protocols; a class implements one or the other and the runner sorts
+them into buckets automatically:
+
+```python
+class Grader(typing.Protocol):            # per-case; scores averaged
+    name: str
+    def grade(self, case, output) -> list[Score]: ...   # may be async
+
+class AggregateGrader(typing.Protocol):   # whole-run; scores stored as-is
+    name: str
+    def aggregate(self, results: list[CaseResult]) -> list[Score]: ...
+```
+
+Suite config is a list of `{type, ...kwargs}` specs; `type` selects a
+registered class, the rest becomes constructor kwargs. Built-ins:
+
+```yaml
+graders:
+  # --- deterministic per-case checks (emit 1.0/0.0 + passed) -------------
+  - type: max_chars          # len(field) <= maximum
+    name: length_ok          # `name` doubles as the metric name
+    field: output.reply
+    maximum: 400
+  - type: regex_absent       # field must NOT match pattern
+    name: no_pii
+    field: output.reply
+    pattern: '\b\d{3}-\d{2}-\d{4}\b'
+  - type: regex_present      # field must match EVERY pattern (all-of)
+    name: required_markup
+    field: output.html
+    patterns: ['<form', 'type="email"']
+  - type: non_empty          # field must be truthy
+    field: output.reply
+
+  # --- numeric (per-case): promote numeric output fields to metrics ------
+  - type: numeric            # each field -> a scorecard metric (its mean)
+    fields:
+      - {ref: output.tool_error_rate, max: 0.09}  # bounded -> pass/fail too
+      - {ref: output.hallucination_rate, max: 0.02}
+      - output.cost_per_request                    # unbounded -> measurement
+
+  # --- classification (aggregate): P/R/F1 + FN/FP rates ------------------
+  - type: classification
+    name: intent_detection
+    predicted_ref: output.intent     # what the system produced
+    expected_ref: expected.intent    # the human-authored answer key
+    positive_labels: [refund]        # the class you must not miss
+    negative_labels: [question, complaint]
+
+  # --- LLM judge (per-case): rubric scoring, 1..scale -> 0..1 ------------
+  - type: llm_judge
+    name: quality
+    content_ref: output.reply        # the text to judge
+    scale: 5
+    dimensions:
+      - {key: empathy, description: "Acknowledges the customer's situation."}
+      - {key: accuracy, description: 'Consistent with the ticket facts.'}
+    rubric: |                        # optional free-text rubric for the judge
+      Judge as a support-quality reviewer...
+    context_refs:                    # extra context shown to the judge
+      ticket: input.ticket_text
+    model: claude-sonnet-4-6         # single-judge shorthand (live mode)
+    judge_version: v1                # bump on ANY judge change; re-baseline
+    replay_path: fixtures/judge.yaml # recorded judgments (replay mode)
+```
+
+Grader field selectors resolve against roots `input`, `expected`, `output`
+(the adapter's extracted fields), `case`, and `artifacts` (the output's
+saved files, e.g. `artifacts.screenshot`).
+
+The `numeric` grader is what turns adapter-extracted numbers into scorecard
+metrics: only `Score`s reach a scorecard, so a value the adapter merely put in
+`Output.fields` (a cost, an error rate) needs a grader to promote it. Each
+field's metric name defaults to the ref's leaf (`output.cost` -> `cost`); add
+`min`/`max` to also emit per-case pass/fail. Absent or non-numeric fields
+degrade to `null`. `compare`'s guardrails and a `win_metric` with
+`win_higher_is_better: false` then gate on these directly — e.g. gating a
+`generation_cost` or `tool_error_rate` alongside quality judges.
+
+The judge runs live (`AnthropicJudgeClient` forced tool call, or
+`OpenAIJudgeClient` `json_schema` — both temperature 0, needing the `judge`
+extra plus `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`) or offline
+(`ReplayJudgeClient`), chosen by the run mode like the adapter. Each
+dimension becomes a metric `<name>.<key>` plus a `<name>.overall` mean.
+
+**Panel + images.** Replace the single `model`/`replay_path` with a
+`judges:` list to run a **panel** — each judge scores independently:
+
+```yaml
+  - type: llm_judge
+    name: quality
+    content_ref: output.html
+    dimensions: [ ... ]
+    image_refs:                      # images shown to judges (live only)
+      screenshot: artifacts.screenshot
+    disagreement_threshold: 2        # raw-point spread that flags a case
+    judges:
+      - {key: claude, provider: anthropic, model: claude-sonnet-4-6, replay_path: fixtures/judge_claude.yaml}
+      - {key: gpt,    provider: openai,    model: 'openai:gpt-4o',    replay_path: fixtures/judge_gpt.yaml}
+```
+
+A panel emits, on top of the per-dimension panel means and `<name>.overall`:
+`<name>.<judge>.overall` (each judge's own mean, so a systematically
+generous judge is visible), `<name>.disagreement` (mean inter-judge spread
+in raw points), and `<name>.flagged` (1.0 when any dimension's spread
+reaches `disagreement_threshold` — averaged across cases, the fraction a
+human should review). `image_refs` resolve to file paths (from
+`output.artifacts`) or inline `{media_type, data}`; images are sent live
+only, so replay stays offline. A single judge emits none of the panel-only
+metrics, so existing single-judge suites are unchanged (a panel is the natural
+fit for judging rendered screenshots with a Claude+GPT pair, for instance).
+
+Custom graders register exactly like adapters — see
+`examples/quickstart/graders.py` for one of each kind (a per-case
+keyword check and a whole-run distinctness check). A grader that needs to
+know the run mode (live vs replay) can expose `set_mode(mode: str)`; the
+runner calls it before the run starts.
+
+## 5. Fixtures (offline / CI runs)
+
+Replay mode swaps the configured adapter for recorded outputs, keyed by
+case id and variant name — the whole pipeline then runs with no network,
+keys, or deployed service:
+
+```yaml
+# fixtures/replay.yaml
+refund_request:
+  baseline:  {reply: 'We can refund...', intent: refund}
+  candidate: {reply: 'Refund issued...', intent: refund}
+billing_question:
+  baseline:  {error: 'HTTP 502'}        # recorded failures work too
+```
+
+Judge fixtures are keyed by the **exact content string under judgment**, so
+different variants (different text) deterministically get different scores:
+
+```yaml
+# fixtures/judge.yaml
+'We can refund...':
+  scores: {empathy: 4, accuracy: 5}
+  rationale: Correct but a little curt.
+```
+
+## 6. The suite + thresholds (the gate policy)
+
+Ties it all together. Paths (`dataset`, `replay_fixtures`, grader
+`replay_path`) resolve relative to the suite file, so the suite runs from
+any working directory:
+
+```yaml
+project: my-service          # store namespace
+suite: support_reply         # suite name within the project
+dataset: datasets/support_reply/v1
+dataset_version: v1
+mode_default: http           # 'replay' to default offline
+replay_fixtures: fixtures/replay.yaml
+adapter: {...}               # step 2
+graders: [...]               # step 4
+variants: {...}              # step 3
+n_samples: 1                 # invocations per case (sampling)
+concurrency: 1               # max concurrent (case, sample) invocations
+retry:                       # transient-failure retry (default: no retry)
+  max_attempts: 3            # total tries per invocation (1 = off)
+  backoff_base: 0.5          # seconds; delay = base * 2**(attempt-1)
+  backoff_max: 30.0          # per-sleep cap
+  jitter: 0.1                # +/- fractional randomization
+
+thresholds:
+  win_metric: quality.overall      # the ONE headline signal
+  win_higher_is_better: true
+  win_min_delta: 0.02              # dead band: |delta| <= this -> neutral
+  on_regression: warn              # or 'fail' to hard-gate the win metric
+  variants: {baseline: baseline, candidate: candidate}   # gate defaults
+  guardrails:                      # hard constraints on the CANDIDATE
+    - metric: false_negative_rate
+      max: 0.10                    # absolute ceiling
+      must_not_increase: true      # ...and no worse than baseline
+    - metric: no_pii
+      min: 1.0                     # absolute floor (pass-rates: 'all passed')
+    - metric: errors
+      max: 0                       # any failed invocation fails the gate
+```
+
+Guardrail rules compose: `max`, `min`, `must_not_increase`,
+`must_not_decrease`. A guardrail whose metric is missing on the candidate
+fails closed. Pick guardrails for the failures that must never ship, and
+one win metric for the improvement you're hunting; everything else is
+reported informationally.
+
+## 7. Running it
+
+**CLI** (plug-ins first, so custom types register):
+
+```bash
+# one variant -> scorecard (optionally saved)
+evalkit --plugins my_service.graders run \
+    --suite my_service/suite.yaml --variant candidate --mode replay \
+    --out candidate.scorecard.json --revision "$GIT_SHA"
+
+# the CI workhorse: run baseline+candidate, compare, exit 1 on 'fail'
+evalkit --plugins my_service.graders gate \
+    --suite my_service/suite.yaml --mode replay \
+    --export outbox.jsonl --revision "$GIT_SHA"
+
+# re-compare two previously saved runs (accepts --out scorecards OR
+# --run-out run files)
+evalkit compare --suite my_service/suite.yaml \
+    --baseline old.run.json --candidate new.run.json
+```
+
+`gate` picks variant names from `thresholds.variants`, falling back to
+variants literally named `baseline`/`candidate`. `--revision` is an opaque
+provenance id (git SHA, image digest, release label — whatever your world
+uses; the engine never interprets it).
+
+`run`, `compare`, and `gate` take `--report markdown` (default) or
+`--report html` for a standalone, self-contained report document (a CI
+artifact or PR attachment). Reporters are a registry seam like adapters and
+graders — register a custom format with `evalkit.reporters.base.register`
+and select it by name, e.g. `--report pdf`.
+
+**The change loop — run, change, run, compare.** To measure whether a
+change (a prompt edit, a new model, a frontend PR) helped or regressed,
+run the *same variant* before and after and compare the two saved runs:
+
+```bash
+evalkit ... run --variant candidate --run-out before.run.json --revision before
+# ... make the change (edit the prompt, point at the PR build, swap the model) ...
+evalkit ... run --variant candidate --run-out after.run.json  --revision after
+evalkit compare --suite my_service/suite.yaml \
+    --baseline before.run.json --candidate after.run.json      # deltas + verdict
+```
+
+The comparison's guardrails + win metric then read as "did the change
+regress?" For nondeterministic targets (LLMs, browsers) raise `n_samples`
+so each metric is a mean (with `stdev`) over several generations — a single
+run per side makes a small delta indistinguishable from run-to-run noise,
+and `win_min_delta` is your noise floor.
+
+**Python API** — everything the CLI does is a library call; the full worked
+version is `examples/quickstart/run_eval.py`:
+
+```python
+import my_service.graders  # noqa: F401  (registers custom types)
+from evalkit import compare, loader, report, runner, store
+
+suite = loader.load_suite('my_service/suite.yaml')
+baseline = runner.run_suite_sync(suite, 'baseline', mode='replay',
+                                 revision='abc123', created_at=now)
+candidate = runner.run_suite_sync(suite, 'candidate', mode='replay',
+                                  revision='abc123', created_at=now)
+# (async context: `await runner.run_suite(...)` is the same call)
+
+result = compare.compare(baseline, candidate, suite.thresholds)
+print(report.render_scorecard(candidate))
+print(report.render_comparison(result))
+
+store.write_scorecard('candidate.scorecard.json', candidate)
+store.write_comparison('comparison.json', result)
+store.JsonlOutboxExporter('outbox.jsonl').export(candidate)
+
+raise SystemExit(0 if result.verdict != 'fail' else 1)
+```
+
+---
+
+# Interpreting results
+
+## The scorecard
+
+One scorecard per (suite × variant) run. Header first:
+
+```
+### my-service/support_reply - `candidate`
+- model: `claude-sonnet-4-6`  mode: `replay`  dataset: `v1`  cases: 6x2
+```
+
+`cases: 6x2` = 6 cases × `n_samples` 2 → **12 observations** behind every
+number below it. Then one row per metric; there are three families, read
+differently:
+
+**Pass-rates** (deterministic + custom per-case graders). Each output
+scored 1.0 or 0.0; the scorecard shows the mean. `1.0000` = every
+observation passed; `0.9167` = 11 of 12. The metric name is the grader's
+`name`.
+
+**Judge scores** (`quality.empathy`, ..., `quality.overall`). Each output
+rubric-scored 1..scale by the pinned judge, normalized to 0..1, averaged
+across observations. `overall` is the per-output mean of the dimensions,
+then averaged. An output the judge couldn't score (errored invocation,
+missing content) contributes *nothing* — it is excluded from the mean, not
+counted as zero — so always read judge means alongside `errors`.
+
+**Set-level aggregates** (classification + custom aggregate graders).
+Computed once over the whole run from a confusion matrix. The
+`classification` grader maps predicted/expected labels to
+positive/negative via your configured label sets, with three rules: an
+errored output increments `errors` and is excluded; a label resolving to
+nothing is an error too; an **unlisted** label counts as *negative*, so a
+stray verdict can never masquerade as a catch. Then:
+
+| metric | formula | question it answers |
+| --- | --- | --- |
+| `precision` | TP/(TP+FP) | of what it flagged positive, how much really was? |
+| `recall` | TP/(TP+FN) | of the real positives, how many did it catch? |
+| `f1` | 2PR/(P+R) | single-number balance of the two |
+| `false_negative_rate` | FN/(FN+TP) | misses, as a fraction of real positives |
+| `false_positive_rate` | FP/(FP+TN) | false alarms, as a fraction of real negatives |
+| `accuracy` | (TP+TN)/all | overall fraction correct — flatters on imbalanced data; never guardrail it |
+| `support_positive` / `support_negative` | TP+FN / TN+FP | the denominators: how much evidence backs the rates |
+| `errors` | count | invocations that failed or produced no usable label |
+
+FNR and FPR are first-class (not just `1-recall`) because they're the
+operational failure modes gates hang guardrails on: FNR = "a positive
+slipped through", FPR = "a negative got blocked". They have different
+denominators, so they stay honest on imbalanced datasets where `accuracy`
+lies. `support_*` doubles as a drift alarm: if it changes between runs on
+the same `dataset_hash`, extraction or labels broke. `errors` is a raw
+count and worth a `max: 0` guardrail — errored results are excluded from
+every rate, so without it a variant that crashes on its hardest cases
+would look *better*.
+
+In JSON/outbox form each metric carries `kind` (`mean` vs `aggregate`) and
+`n`. Downstream tooling may re-average `mean` metrics across runs (weighted
+by `n`) but must never average two `aggregate` values (the mean of two F1s
+is not the combined F1).
+
+## The comparison
+
+```
+## **PASS** - my-service/support_reply
+`candidate` vs `baseline` - quality.overall neutral
+```
+
+The badge is the verdict; the tail is why. The delta table lists every
+metric with baseline / candidate / delta; the row marked
+**(improved | regressed | neutral)** is the configured win metric. Its call
+uses the dead band: `|delta| <= win_min_delta` → **neutral** — deliberate
+protection against celebrating (or reverting on) noise from small samples.
+
+The **Guardrails** section shows each rule as `[ok]` or `[BREACH]` with the
+measured value. Verdict logic, in order:
+
+1. any guardrail breach → **FAIL** (regardless of the win metric);
+2. else win metric regressed → **WARN** (or **FAIL** if
+   `on_regression: fail`);
+3. else → **PASS**.
+
+`gate` (and the example driver) exit non-zero exactly on **FAIL**, so the
+verdict drops straight into CI. A **PASS with neutral win** is a perfectly
+good outcome — it means "no regression, no proven improvement".
+
+## Provenance (trusting a number later)
+
+Every scorecard (and every outbox row) carries the full reproducibility
+key: `project, suite, variant, dataset_version, model_id, prompt_version,
+judge_version, revision, suite_hash, dataset_hash, mode, created_at`.
+
+The declared versions state *intent*; the engine-computed content hashes
+prove it: `suite_hash` digests the raw suite file, `dataset_hash` digests
+the loaded cases (order-independent, formatting-independent). Two runs
+whose hashes match evaluated the same config over the same data — if a
+metric moved, the *system under test* moved. If a hash changed, the eval
+itself changed and the comparison is apples-to-oranges: re-baseline.
+`revision` ties the run to whatever provenance scheme you use (commit,
+image digest, release label). A judge model/prompt/scale change is also a
+re-baseline event — bump `judge_version`; the runner lifts each judge
+grader's pin (`key@version`, a panel joins them) onto `Scorecard.judge_version`
+so it rides the reproducibility key, and it stays recorded on every judge
+score's `detail` too.
+
+## Per-sample results
+
+`runner.run_suite` returns a **`RunResult`** — the scorecard plus every
+per-sample `CaseResult` (the output, its `artifacts`, and its scores). The
+scorecard is the aggregate; the results are the ground truth it was folded
+from. Persist the whole thing with `store.write_run` (CLI: `run --run-out`)
+so transcript review, human rating, and judge-agreement analysis can read
+back individual generations without re-running the suite. Every run gets a
+`run_id` (a UUID) that threads onto the scorecard and every store row.
+
+With `n_samples > 1`, each `mean` metric also carries a `stdev` over its
+observations, so repeat-generation spread is visible, not just the average.
+Set `concurrency: N` in the suite to run invocations concurrently (the
+adapter and per-case graders must then tolerate concurrent calls).
+
+**Retries.** Live targets fail transiently — a rate limit, a 5xx, a dropped
+connection. A `retry:` block (above) makes the runner re-invoke the adapter
+with exponential backoff (`backoff_base * 2**(attempt-1)`, capped at
+`backoff_max`, ± `jitter`) when — and only when — the adapter marks the
+failure `Output.retryable`. The built-in `http` adapter flags 429/5xx/network
+errors and leaves other 4xx terminal; a custom adapter sets the flag for
+whatever its transient failures are. The default (`max_attempts: 1`) is a
+no-op, so existing suites are unchanged. Retries hold their concurrency slot
+while backing off, so a rate-limited target naturally applies backpressure.
+
+The **LLM judge** honors the same `retry:` policy: a transient judge-client
+error (a 429/5xx/timeout raised by the Anthropic/OpenAI SDK) backs off and
+retries, and only a sustained failure surfaces.
+
+**Resume.** For long live runs, pass `run --checkpoint run.ckpt`: the runner
+appends each `(case, sample)` result to that JSONL file as it completes, so an
+interrupted run (Ctrl-C, crash, spot-instance reclaim) leaves a valid partial
+trail. Re-run with `--resume` and it reuses the recorded results and invokes
+only what's missing, reusing the original `run_id`:
+
+```bash
+evalkit run --suite suite.yaml --variant candidate --checkpoint run.ckpt
+# ... interrupted after 40/100 cases ...
+evalkit run --suite suite.yaml --variant candidate --checkpoint run.ckpt --resume
+```
+
+The checkpoint's meta line records `suite_hash`/`dataset_hash`, so a resume
+against a changed suite, dataset, or variant refuses rather than mixing
+incompatible results — delete the checkpoint to start over (which `--resume`
+also does implicitly when the file is absent). A checkpointed `(case, sample)`
+is treated as done whether it succeeded or errored; to redo just the failures,
+drop their lines from the checkpoint first.
+
+## Human rating & judge calibration
+
+An LLM judge is only trustworthy as a win metric once you've checked it
+tracks human taste. evalkit closes that loop over the persisted runs:
+
+```bash
+# blind rating web app over one or more saved runs (repeat --run to blind
+# across variants: the browser never sees which model produced an output)
+evalkit rate --run cand.run.json --run base.run.json \
+    --ratings ratings.jsonl --dimensions visual_design,copy_quality \
+    --content-ref output.html            # or a screenshot via artifacts.*
+
+# how well the judge agreed with the humans, per dimension
+evalkit agreement --run cand.run.json --ratings ratings.jsonl \
+    --dimensions visual_design,copy_quality --judge-name quality
+```
+
+`rate` serves a dependency-free localhost page: a seeded-shuffled queue and
+1..scale buttons per dimension. It renders each item as **typed panels**
+derived from the output — `image`/`pdf` artifacts, `html` (sandboxed iframe
+with a rendered/source toggle), `json`, or `text` — so plain-text
+or JSON results render with zero config, and any number of artifacts become
+that many panels. `--content-ref`/`--screenshot-ref` are the common
+shorthand; a repeatable `--view label:kind:ref` gives explicit control.
+Sessions are **resumable** (a rater only sees items they haven't scored).
+**Blinding is enforced server-side** — the queue payload carries an opaque
+item id and never the run/variant/model; ratings map back to
+`(run_id, case_id, sample_idx)` only on the server. Ratings land in a JSONL
+file (`models.Rating`) that is the **open interchange format**: any external
+tool or spreadsheet export in the same shape feeds `agreement` too.
+
+`agreement` reports, per dimension and overall, the mean-absolute-error and
+correlation between the per-case human mean and the judge's score (both on
+0..1). Low MAE + high correlation is the green light to trust that judge
+dimension as a win metric; a panel's `flagged` cases (see the judge panel
+above) are the natural first items to route through `rate`.
+
+**Side-by-side preference (`rank`/`preferences`).** `rate` scores each output
+in isolation; `rank` is its A-vs-B analog — the human counterpart of
+`pairwise`. It shows both variants' outputs for the same case as neutral
+"Option 1"/"Option 2" columns and the rater picks a winner overall and per
+dimension:
+
+```bash
+# blind side-by-side ranking web app over two saved runs
+evalkit rank --run-a base.run.json --run-b cand.run.json \
+    --preferences prefs.jsonl --dimensions visual_design,copy_quality \
+    --content-ref output.html
+
+# human A-vs-B win-rate (overall + per dimension) from the collected file
+evalkit preferences --run-a base.run.json --run-b cand.run.json \
+    --preferences prefs.jsonl --report html --report-out prefs.html
+```
+
+Left/right sides are **shuffled per rater and un-blinded server-side**, so a
+stored pick is always in *variant* terms (`variant_a`/`variant_b`) regardless
+of which side it was shown on — position bias counterbalances across raters
+exactly like `pairwise`'s order swap. Sessions are resumable and picks land
+in a JSONL file (`models.Preference`), the **open A-vs-B interchange format**.
+`preferences` aggregates it (ties count half); to check the LLM pairwise judge
+against the human panel, pass the same file to `pairwise --preferences
+prefs.jsonl`, which appends a per-case human-vs-judge agreement table — the
+head-to-head calibration gate.
+
+**Live judges on recorded data.** `--mode` drives the adapter; `--judge-mode`
+drives the graders independently (default: same as `--mode`). So
+`--mode replay --judge-mode live` re-scores recorded outputs with the live
+judge panel — no regeneration — which is how you iterate on a rubric or
+re-record judge fixtures cheaply.
+
+## Sweeps & pairwise win-rate
+
+`compare`/`gate` answer "candidate vs baseline". Two commands go beyond that:
+
+```bash
+# N-way: run every variant (or a subset) and rank them by the win metric
+evalkit sweep --suite suite.yaml --mode replay          # or --variants a,b,c
+
+# A-vs-B: a judge picks a winner per case -> A's win-rate
+evalkit pairwise --suite suite.yaml --a baseline --b candidate --mode replay
+```
+
+`sweep` prints a ranked leaderboard plus a full metric × variant matrix (a
+model × prompt-version grid is just several named variants) — reusing the
+per-variant runner unchanged; it's pure orchestration + tabulation.
+
+`pairwise` is the sharper subjective signal: instead of scoring each output
+in isolation, a judge is shown **both** variants' outputs for the same case
+and picks a winner, and evalkit reports A's win-rate (ties count half).
+Order is **counterbalanced** — each pair is judged both ways and a pick that
+flips when you swap the order collapses to a tie, so position bias can't
+manufacture a winner. Configure it under `thresholds.pairwise` (`content_ref`,
+`model`/`replay_path`, optional `rubric`/`context_refs`); it runs live
+(Anthropic or OpenAI) or offline against recorded, order-independent
+judgments, like the rubric judge. `examples/quickstart/suite.yaml` has an
+offline pairwise config.
+
+## The outbox
+
+`JsonlOutboxExporter` flattens results into JSONL for a column-store shipper
+(e.g. ClickHouse) to drain, in a flat `eval_runs`/`eval_scores` shape:
+`export(scorecard)` writes one **metric** row each (`metric, value, stdev,
+metric_kind, n`), and `export_scores(run)` writes one **per-case score** row
+each (`case_id, sample_idx, grader, metric, value, passed, detail`). Both
+repeat the full reproducibility key (incl. `run_id`) so a multi-tenant
+trend table can filter/group on any dimension without joins. Swap the
+exporter for a real database client without touching the runner or any
+consumer. The rows use a no-`Nullable` convention (a missing value is the
+sentinel pair `(value=0, has_value=false)`; `passed` is the tri-state string
+`'true'|'false'|'null'`), so a JSONEachRow-style feed maps straight onto a
+flat schema.
+
+---
+
+## The two extension seams (recap)
+
+```python
+class TargetAdapter(typing.Protocol):       # how to call the system under test
+    async def invoke(self, case, variant) -> Output: ...
+
+class Grader(typing.Protocol):              # per-case (averaged)
+    def grade(self, case, output) -> list[Score]: ...
+class AggregateGrader(typing.Protocol):     # whole-run (P/R/F1, win-rate)
+    def aggregate(self, results) -> list[Score]: ...
+```
+
+Built-ins: `http` + `replay` adapters; `classification`, `max_chars`,
+`regex_absent`, `regex_present`, `non_empty`, and `llm_judge` graders; and
+`markdown` + `html` reporters (single scorecards and comparative
+comparisons; pick one with `--report`). Register more with
+`evalkit.adapters.base.register` / `evalkit.graders.base.register` /
+`evalkit.reporters.base.register` and load them with `--plugins your.module`
+(CLI) or a plain import (Python API). If onboarding a new consumer ever
+requires touching `src/evalkit/`, that's an abstraction leak — fix the engine
+seam, don't fork it.
+
+## Layout
+
+```
+src/evalkit/
+  models.py      Case, Variant, Output, Score, Scorecard, Comparison (opaque-blob based)
+  refs.py        $ref resolution (the only thing that opens a consumer's blobs)
+  loader.py      suite + dataset loading, content hashes (YAML/JSON; suite-relative paths)
+  adapters/      target seam - http, replay
+  graders/       grader seam - deterministic, classification, llm_judge
+  runner.py      suite x variant -> RunResult (scorecard + per-sample results)
+  compare.py     candidate vs baseline -> guardrails + win -> verdict
+  sweep.py       run N variants -> ranked leaderboard (metric x variant)
+  pairwise.py    A-vs-B judging -> counterbalanced win-rate
+  store.py       scorecard/run JSON + column-store outbox + ratings/prefs JSONL
+  rating.py      blind rating + side-by-side ranking web apps + agreement
+  reporters/     report seam - markdown / html (scorecard, comparison, ...)
+  report.py      Markdown renderers (scorecard / comparison / sweep / pairwise / ...)
+  cli.py         run - compare - gate - sweep - pairwise - rate - rank - report - ...
+tests/               engine unit tests
+examples/quickstart  a runnable consumer that doubles as an implementation test
+docs/design.md       the design overview
+```
+
+## Status
+
+MVP. Built: deterministic + classification + **LLM-judge** (rubric scoring;
+single judge or a Claude/GPT **panel** with per-dimension means, per-judge
+overalls, inter-judge disagreement flagging, and image/screenshot inputs)
+graders, http/replay/browser adapters, runner (N-sampling, optional
+concurrency, per-sample `RunResult` + `run_id` + variance), comparison/gate,
+JSON + run + outbox store (metric and per-case-score rows), **N-way sweeps
++ counterbalanced pairwise A-vs-B win-rate** (`sweep`/`pairwise`), **blind
+human-rating + side-by-side ranking web apps** with judge↔human agreement and
+human-vs-judge pairwise agreement (`rate`/`agreement`, `rank`/`preferences`),
+**pluggable reporters** (`markdown`/`html`, `--report`), provenance
+(`revision` + suite/dataset content hashes), typed package (`py.typed`).
+
+Known gaps / next:
+
+- A real column-store client (the row shape + JSONL outbox exporter stand
+  in, in `store.py`).
+- Cost/token capture: `Output.tokens`/`cost` fields exist but nothing
+  populates them (an adapter must fill them from whatever usage its target
+  reports).
+- Run robustness lands: retry with exponential backoff on transient failures
+  for both the adapter (suite `retry:` + `Output.retryable`) and the LLM judge
+  client, plus idempotent mid-run resume from a `run --checkpoint`.
+
+## Releasing
+
+Releases publish to PyPI as **`evalcore`** via
+[Trusted Publishing](https://docs.pypi.org/trusted-publishers/) (OIDC) — no API
+tokens are stored. One-time setup on PyPI: add a *pending publisher* for project
+`evalcore` pointing at owner `scottpmiller`, repo `evalcore`, workflow
+`publish.yml`, environment `pypi`. Then to cut a release: bump `version` in
+`pyproject.toml`, tag it, and publish a GitHub Release — `.github/workflows/publish.yml`
+builds the sdist + wheel and uploads them. (Point it at TestPyPI first for a dry run.)