evalpair 0.1.0__tar.gz

evalpair-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kameswara Sai Srikar Manda
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalpair-0.1.0/PKG-INFO

@@ -0,0 +1,189 @@
+Metadata-Version: 2.4
+Name: evalpair
+Version: 0.1.0
+Summary: Run an LLM eval suite against a local or hosted model, cache every call, and compare two runs item by item with bootstrap intervals and a measured noise floor. CLI and MCP server.
+Keywords: llm,evaluation,eval,benchmark,bootstrap,ollama,mcp,cli
+Author: Kameswara Sai Srikar Manda
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Dist: mcp>=2.3,<3
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/Srikarmk/evalpair
+Project-URL: Source, https://github.com/Srikarmk/evalpair
+Project-URL: Issues, https://github.com/Srikarmk/evalpair/issues
+Description-Content-Type: text/markdown
+
+# evalpair
+
+<!-- mcp-name: io.github.Srikarmk/evalpair -->
+
+Run an LLM eval suite against a model, cache every call, and compare two runs item by item: did the new
+model, prompt or quantization actually change anything, or is the difference noise?
+
+Most eval tools print two scores side by side. That reads a 2-point gap on 160 questions as a result, when
+it usually isn't one. evalpair pairs the two runs by item, puts a bootstrap interval on the difference, counts
+the items that got worse and better, and can measure a **noise floor**: the same model asked the same
+questions twice. In my own studies the same prompt sent twice at temperature 0 changed up to 3% of answers,
+so an answer-change rate only means something above that.
+
+It's a CLI and an MCP server, so an agent can run an eval and read the verdict itself. It only calls the
+model server you point it at, and keeps everything in one local folder.
+
+## Quick start
+
+```bash
+uvx evalpair suites
+uvx evalpair run gsm8k-160 -m llama3.1:8b --repeat --label llama
+uvx evalpair run gsm8k-160 -m qwen3:8b --think off --label qwen3
+uvx evalpair compare llama qwen3
+```
+
+`run` prints the score with a 95% interval and the first failures, unparsed and truncated replies first
+(those are usually the parser's or the token budget's fault, not the model's). `compare` prints both scores,
+the paired difference with its interval, how many items got worse and better, the noise floor, and ends with
+a plain-language verdict: whether the score difference is distinguishable from zero, and whether the answers
+moved more than the same model asked twice.
+
+A real comparison: Ollama's default 4-bit Llama 3.1 8B against its 2-bit build, on an M5 Pro laptop
+(`--num-ctx 4096`; about 40 minutes for both runs):
+
+```
+$ evalpair compare llama-q4 llama-q2
+A  llama-q4: 84.4% [79.4%, 90.0%]
+B  llama-q2: 60.6% [53.1%, 68.1%]
+B − A        -23.8 pp [-31.9 pp, -16.2 pp]
+changed      40.6% [33.1%, 48.1%] of 160 items (43 worse in B, 5 better)
+floor (A)    0.0% [0.0%, 0.0%]; excess +40.6 pp [+33.1 pp, +48.1 pp]
+
+B scores lower than A: -23.8 pp [-31.9 pp, -16.2 pp], paired over 160 items. Answers changed on 40.6% of
+items (43 got worse, 5 better). The noise floor (the same model asked twice) is 0.0% for A, so the change
+is beyond noise.
+  worse in b: gsm8k-0259  '168' → '52'
+  ...
+```
+
+Both scores are the ones the quantization study measured for these builds on the same items.
+
+Interrupt a run and run it again: every reply is cached, so it picks up where it stopped.
+
+## As an MCP server
+
+```bash
+claude mcp add evalpair -- uvx evalpair mcp
+```
+
+or in any MCP client's config: `{"command": "uvx", "args": ["evalpair", "mcp"]}`.
+
+| Tool | What it does |
+|---|---|
+| `list_suites` | Suites you can run: name, item count, checker, what it measures |
+| `run_eval` | Run a suite against one model and save the run (calls the model; cached replies are reused) |
+| `list_runs` | Saved runs, newest first |
+| `show_run` | A run's score with its interval, its noise floor, and the end of failed replies |
+| `compare_runs` | Two runs paired by item: difference with an interval, flips each way, change vs. noise floor, examples |
+
+Everything except `run_eval` only reads saved runs.
+
+## Backends
+
+- **Ollama** (default): `--backend ollama`, server at `http://localhost:11434`. The run records the model's
+  digest, so you can tell later exactly which build was tested. `--think on|off` for models that think,
+  `--num-ctx` for the context window.
+- **Any OpenAI-compatible server**: `--backend openai --base-url http://localhost:8000/v1` for vLLM,
+  llama.cpp's server or LM Studio, or a hosted API. A bearer token is read from `EVALPAIR_API_KEY` (or
+  `OPENAI_API_KEY`) and never written to disk.
+
+Temperature defaults to 0 and the seed to 7. Both are part of the cache key, along with the server URL,
+model, messages, token budget and thinking setting.
+
+## Suites
+
+Three are bundled. They're the item sets from my Measuring LLM Systems studies, with those studies' prompts
+word for word, so a run can be checked against the published numbers:
+
+| Suite | Items | Checker | Source |
+|---|---|---|---|
+| `gsm8k-160` | 160 | `number` | GSM8K test (MIT) |
+| `mmlu-hs-160` | 160 | `choice`, stratified by subject | MMLU, four high-school subjects (MIT) |
+| `e2e-extract-150` | 150 | `json_fields` | E2E NLG restaurant records (CC BY-SA 4.0) |
+
+To make your own, create a folder with two files, in `~/.evalpair/suites/<name>/` or anywhere you then pass
+by path:
+
+```toml
+# suite.toml
+description = "Capitals, answered in one word"
+checker = "exact"
+max_tokens = 64
+system = "Answer in one word."           # optional
+user = "What is the capital of {country}?"
+group = "region"                         # optional: stratify the bootstrap by this item field
+```
+
+```jsonl
+{"id": "fr", "country": "France", "region": "Europe", "gold": "Paris"}
+{"id": "pe", "country": "Peru", "region": "Americas", "gold": ["Lima", "Ciudad de los Reyes"]}
+```
+
+`{field}` in a prompt is filled from the item; a list of strings becomes lettered options (`A. …`), anything
+else is shown as indented JSON. Use `{{` and `}}` for literal braces.
+
+| Checker | Scores | Gold |
+|---|---|---|
+| `number` | the number after "Answer:", else `\boxed{}`, else the last number | a number |
+| `choice` | the letter after "Answer:", with fallbacks (letters set by `checker_options.letters`) | a letter |
+| `exact` | the text after "Answer:", or the last line, normalised | a string or list of strings |
+| `contains` | whether a gold string appears as whole words | a string or list |
+| `regex` | the last match of `checker_options.pattern` (first group) | a string or list |
+| `json_fields` | the share of the gold object's fields the reply's JSON gets right (null must be null) | an object |
+
+Checkers are deterministic code: no LLM grades another LLM here. The `number`, `choice` and `json_fields`
+parsers come from the studies, and re-scoring all 9,870 replies from the quantization study's main run with them gives
+the study's own scores on every one (`scripts/check_against_studies.py`).
+
+## How the numbers are computed
+
+- **Unit of resampling: the item.** 2,000 bootstrap draws, percentile intervals, seeded, so the same runs
+  always give the same interval. With a `group`, items are resampled within each group.
+- **Comparisons are paired** over the items both runs answered, using the first pass of each. If the two
+  runs used different versions of a suite (the hash of its spec and items differs), the comparison says so.
+- **Noise floor**: with `--repeat`, every item is sent a second time as a separate call (its own cache entry).
+  The floor is the share of items whose parsed answer changed between the passes. A comparison calls a change
+  "beyond noise" only if, per item, (changed between runs) minus (changed between the passes) has an interval
+  above zero for every run that has a floor.
+- Items are taken in file order, so `--limit 50` is the same 50 items every time and limited runs still pair up.
+
+## Where things live
+
+`$EVALPAIR_HOME`, or `~/.evalpair`:
+
+```
+cache.sqlite           every reply, keyed by request
+runs/<run>/meta.json   suite and its hash, model, digest, backend, options, counts, timings
+runs/<run>/rows.jsonl  one line per item and pass: reply, parsed answer, score, tokens
+suites/<name>/         your suites
+```
+
+## Limits
+
+- One model per run, chat-style prompts, one turn. No tool use, multi-turn or constrained decoding yet.
+- Deterministic checkers only, so open-ended answers need a suite whose answer can be checked by code.
+- The noise floor measures run-to-run variation on the same server. It doesn't cover a different
+  machine or a server update.
+
+## License
+
+MIT for the code. The bundled suites keep their sources' licenses (MIT for GSM8K and MMLU, CC BY-SA 4.0 for
+E2E NLG); each `suite.toml` names its license and source.

evalpair-0.1.0/README.md

@@ -0,0 +1,162 @@
+# evalpair
+
+<!-- mcp-name: io.github.Srikarmk/evalpair -->
+
+Run an LLM eval suite against a model, cache every call, and compare two runs item by item: did the new
+model, prompt or quantization actually change anything, or is the difference noise?
+
+Most eval tools print two scores side by side. That reads a 2-point gap on 160 questions as a result, when
+it usually isn't one. evalpair pairs the two runs by item, puts a bootstrap interval on the difference, counts
+the items that got worse and better, and can measure a **noise floor**: the same model asked the same
+questions twice. In my own studies the same prompt sent twice at temperature 0 changed up to 3% of answers,
+so an answer-change rate only means something above that.
+
+It's a CLI and an MCP server, so an agent can run an eval and read the verdict itself. It only calls the
+model server you point it at, and keeps everything in one local folder.
+
+## Quick start
+
+```bash
+uvx evalpair suites
+uvx evalpair run gsm8k-160 -m llama3.1:8b --repeat --label llama
+uvx evalpair run gsm8k-160 -m qwen3:8b --think off --label qwen3
+uvx evalpair compare llama qwen3
+```
+
+`run` prints the score with a 95% interval and the first failures, unparsed and truncated replies first
+(those are usually the parser's or the token budget's fault, not the model's). `compare` prints both scores,
+the paired difference with its interval, how many items got worse and better, the noise floor, and ends with
+a plain-language verdict: whether the score difference is distinguishable from zero, and whether the answers
+moved more than the same model asked twice.
+
+A real comparison: Ollama's default 4-bit Llama 3.1 8B against its 2-bit build, on an M5 Pro laptop
+(`--num-ctx 4096`; about 40 minutes for both runs):
+
+```
+$ evalpair compare llama-q4 llama-q2
+A  llama-q4: 84.4% [79.4%, 90.0%]
+B  llama-q2: 60.6% [53.1%, 68.1%]
+B − A        -23.8 pp [-31.9 pp, -16.2 pp]
+changed      40.6% [33.1%, 48.1%] of 160 items (43 worse in B, 5 better)
+floor (A)    0.0% [0.0%, 0.0%]; excess +40.6 pp [+33.1 pp, +48.1 pp]
+
+B scores lower than A: -23.8 pp [-31.9 pp, -16.2 pp], paired over 160 items. Answers changed on 40.6% of
+items (43 got worse, 5 better). The noise floor (the same model asked twice) is 0.0% for A, so the change
+is beyond noise.
+  worse in b: gsm8k-0259  '168' → '52'
+  ...
+```
+
+Both scores are the ones the quantization study measured for these builds on the same items.
+
+Interrupt a run and run it again: every reply is cached, so it picks up where it stopped.
+
+## As an MCP server
+
+```bash
+claude mcp add evalpair -- uvx evalpair mcp
+```
+
+or in any MCP client's config: `{"command": "uvx", "args": ["evalpair", "mcp"]}`.
+
+| Tool | What it does |
+|---|---|
+| `list_suites` | Suites you can run: name, item count, checker, what it measures |
+| `run_eval` | Run a suite against one model and save the run (calls the model; cached replies are reused) |
+| `list_runs` | Saved runs, newest first |
+| `show_run` | A run's score with its interval, its noise floor, and the end of failed replies |
+| `compare_runs` | Two runs paired by item: difference with an interval, flips each way, change vs. noise floor, examples |
+
+Everything except `run_eval` only reads saved runs.
+
+## Backends
+
+- **Ollama** (default): `--backend ollama`, server at `http://localhost:11434`. The run records the model's
+  digest, so you can tell later exactly which build was tested. `--think on|off` for models that think,
+  `--num-ctx` for the context window.
+- **Any OpenAI-compatible server**: `--backend openai --base-url http://localhost:8000/v1` for vLLM,
+  llama.cpp's server or LM Studio, or a hosted API. A bearer token is read from `EVALPAIR_API_KEY` (or
+  `OPENAI_API_KEY`) and never written to disk.
+
+Temperature defaults to 0 and the seed to 7. Both are part of the cache key, along with the server URL,
+model, messages, token budget and thinking setting.
+
+## Suites
+
+Three are bundled. They're the item sets from my Measuring LLM Systems studies, with those studies' prompts
+word for word, so a run can be checked against the published numbers:
+
+| Suite | Items | Checker | Source |
+|---|---|---|---|
+| `gsm8k-160` | 160 | `number` | GSM8K test (MIT) |
+| `mmlu-hs-160` | 160 | `choice`, stratified by subject | MMLU, four high-school subjects (MIT) |
+| `e2e-extract-150` | 150 | `json_fields` | E2E NLG restaurant records (CC BY-SA 4.0) |
+
+To make your own, create a folder with two files, in `~/.evalpair/suites/<name>/` or anywhere you then pass
+by path:
+
+```toml
+# suite.toml
+description = "Capitals, answered in one word"
+checker = "exact"
+max_tokens = 64
+system = "Answer in one word."           # optional
+user = "What is the capital of {country}?"
+group = "region"                         # optional: stratify the bootstrap by this item field
+```
+
+```jsonl
+{"id": "fr", "country": "France", "region": "Europe", "gold": "Paris"}
+{"id": "pe", "country": "Peru", "region": "Americas", "gold": ["Lima", "Ciudad de los Reyes"]}
+```
+
+`{field}` in a prompt is filled from the item; a list of strings becomes lettered options (`A. …`), anything
+else is shown as indented JSON. Use `{{` and `}}` for literal braces.
+
+| Checker | Scores | Gold |
+|---|---|---|
+| `number` | the number after "Answer:", else `\boxed{}`, else the last number | a number |
+| `choice` | the letter after "Answer:", with fallbacks (letters set by `checker_options.letters`) | a letter |
+| `exact` | the text after "Answer:", or the last line, normalised | a string or list of strings |
+| `contains` | whether a gold string appears as whole words | a string or list |
+| `regex` | the last match of `checker_options.pattern` (first group) | a string or list |
+| `json_fields` | the share of the gold object's fields the reply's JSON gets right (null must be null) | an object |
+
+Checkers are deterministic code: no LLM grades another LLM here. The `number`, `choice` and `json_fields`
+parsers come from the studies, and re-scoring all 9,870 replies from the quantization study's main run with them gives
+the study's own scores on every one (`scripts/check_against_studies.py`).
+
+## How the numbers are computed
+
+- **Unit of resampling: the item.** 2,000 bootstrap draws, percentile intervals, seeded, so the same runs
+  always give the same interval. With a `group`, items are resampled within each group.
+- **Comparisons are paired** over the items both runs answered, using the first pass of each. If the two
+  runs used different versions of a suite (the hash of its spec and items differs), the comparison says so.
+- **Noise floor**: with `--repeat`, every item is sent a second time as a separate call (its own cache entry).
+  The floor is the share of items whose parsed answer changed between the passes. A comparison calls a change
+  "beyond noise" only if, per item, (changed between runs) minus (changed between the passes) has an interval
+  above zero for every run that has a floor.
+- Items are taken in file order, so `--limit 50` is the same 50 items every time and limited runs still pair up.
+
+## Where things live
+
+`$EVALPAIR_HOME`, or `~/.evalpair`:
+
+```
+cache.sqlite           every reply, keyed by request
+runs/<run>/meta.json   suite and its hash, model, digest, backend, options, counts, timings
+runs/<run>/rows.jsonl  one line per item and pass: reply, parsed answer, score, tokens
+suites/<name>/         your suites
+```
+
+## Limits
+
+- One model per run, chat-style prompts, one turn. No tool use, multi-turn or constrained decoding yet.
+- Deterministic checkers only, so open-ended answers need a suite whose answer can be checked by code.
+- The noise floor measures run-to-run variation on the same server. It doesn't cover a different
+  machine or a server update.
+
+## License
+
+MIT for the code. The bundled suites keep their sources' licenses (MIT for GSM8K and MMLU, CC BY-SA 4.0 for
+E2E NLG); each `suite.toml` names its license and source.

evalpair-0.1.0/pyproject.toml

@@ -0,0 +1,60 @@
+[project]
+name = "evalpair"
+version = "0.1.0"
+description = "Run an LLM eval suite against a local or hosted model, cache every call, and compare two runs item by item with bootstrap intervals and a measured noise floor. CLI and MCP server."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = [
+    "llm",
+    "evaluation",
+    "eval",
+    "benchmark",
+    "bootstrap",
+    "ollama",
+    "mcp",
+    "cli",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["mcp>=2.3,<3"]
+
+[[project.authors]]
+name = "Kameswara Sai Srikar Manda"
+
+[project.urls]
+Homepage = "https://github.com/Srikarmk/evalpair"
+Source = "https://github.com/Srikarmk/evalpair"
+Issues = "https://github.com/Srikarmk/evalpair/issues"
+
+[project.scripts]
+evalpair = "evalpair.cli:main"
+
+[dependency-groups]
+dev = [
+    "pytest>=8",
+    "anyio>=4",
+]
+
+[build-system]
+requires = ["uv_build>=0.9,<0.14"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120

evalpair-0.1.0/pyproject.toml.orig

@@ -0,0 +1,46 @@
+[project]
+name = "evalpair"
+version = "0.1.0"
+description = "Run an LLM eval suite against a local or hosted model, cache every call, and compare two runs item by item with bootstrap intervals and a measured noise floor. CLI and MCP server."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Kameswara Sai Srikar Manda" }]
+keywords = ["llm", "evaluation", "eval", "benchmark", "bootstrap", "ollama", "mcp", "cli"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["mcp>=2.3,<3"]
+
+[project.urls]
+Homepage = "https://github.com/Srikarmk/evalpair"
+Source = "https://github.com/Srikarmk/evalpair"
+Issues = "https://github.com/Srikarmk/evalpair/issues"
+
+[project.scripts]
+evalpair = "evalpair.cli:main"
+
+[dependency-groups]
+dev = ["pytest>=8", "anyio>=4"]
+
+[build-system]
+requires = ["uv_build>=0.9,<0.14"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120

evalpair-0.1.0/src/evalpair/__init__.py

@@ -0,0 +1,3 @@
+"""Run an LLM eval suite, cache every call, and compare two runs item by item with intervals."""
+
+__version__ = "0.1.0"

evalpair-0.1.0/src/evalpair/__main__.py

@@ -0,0 +1,4 @@
+from .cli import main
+
+if __name__ == "__main__":
+    main()

evalpair-0.1.0/src/evalpair/analysis.py

@@ -0,0 +1,141 @@
+"""Summaries of one run and comparisons of two, built from saved rows only (no model calls).
+
+A comparison pairs the two runs by item id over the items both answered (first pass). It reports the
+score difference with a paired bootstrap interval, how many items flipped each way, how often the answer
+changed at all, and, when a run was made with a repeat pass, that run's own answer-change rate: the noise
+floor. A change rate between two runs only means something above the floor.
+"""
+
+from __future__ import annotations
+
+from .stats import excludes_zero, mean, mean_ci
+
+EXCERPT = 400
+
+
+def _pp(x: float) -> str:
+    return f"{100 * x:+.1f} pp"
+
+
+def _pc(x: float) -> str:
+    return f"{100 * x:.1f}%"
+
+
+def _ci(ci: dict, pct=_pc) -> dict:
+    return {**{k: round(v, 4) for k, v in ci.items() if k != "n"}, "n": ci["n"],
+            "text": f"{pct(ci['mean'])} [{pct(ci['lo'])}, {pct(ci['hi'])}]" if ci["n"] else "no items"}
+
+
+def by_pass(rows: list[dict], p: int) -> dict[str, dict]:
+    return {r["id"]: r for r in rows if r["pass"] == p}
+
+
+def excerpt(text: str, n: int = EXCERPT) -> str:
+    text = text or ""
+    return text if len(text) <= n else "…" + text[-n:]
+
+
+def noise_floor(rows: list[dict]) -> dict | None:
+    """Share of items whose parsed answer changed between pass 1 and the repeat pass, and the score
+    change between them. None if the run had no repeat."""
+    a, b = by_pass(rows, 1), by_pass(rows, 2)
+    ids = [i for i in a if i in b]
+    if not ids:
+        return None
+    changed = [float(a[i]["answer"] != b[i]["answer"]) for i in ids]
+    groups = [a[i]["group"] for i in ids]
+    return {"items": len(ids), "answers_changed": int(sum(changed)),
+            "change_rate": _ci(mean_ci(changed, groups)),
+            "score_change": _ci(mean_ci([b[i]["score"] - a[i]["score"] for i in ids], groups), _pp)}
+
+
+def summarize(meta: dict, rows: list[dict], failures: int = 5) -> dict:
+    first = list(by_pass(rows, 1).values())
+    groups = [r["group"] for r in first]
+    wrong = [r for r in first if r["score"] < 1]
+    out = {
+        "run": meta["run"], "suite": meta["suite"], "model": meta["model"], "backend": meta["backend"],
+        "items": len(first), "score": _ci(mean_ci([r["score"] for r in first], groups)),
+        "no_answer": sum(r["answer"] is None for r in first), "truncated": sum(r["truncated"] for r in first),
+        "mean_output_tokens": round(mean(r["output_tokens"] for r in first), 1),
+        "noise_floor": noise_floor(rows),
+        "calls": meta.get("calls"), "cache_hits": meta.get("cache_hits"), "finished": meta.get("finished"),
+    }
+    if any(r["group"] is not None for r in first):
+        per = {}
+        for g in sorted({r["group"] for r in first}, key=str):
+            rs = [r for r in first if r["group"] == g]
+            per[str(g)] = {"items": len(rs), "score": round(mean(r["score"] for r in rs), 4)}
+        out["by_group"] = per
+    # unparsed and truncated replies first: they are usually the checker's or the budget's fault, not the model's
+    wrong.sort(key=lambda r: (r["answer"] is not None, not r["truncated"], r["id"]))
+    out["failures"] = [{"id": r["id"], "answer": r["answer"], "score": r["score"], "how": r["how"],
+                        "truncated": r["truncated"], "reply_end": excerpt(r["text"])} for r in wrong[:failures]]
+    return out
+
+
+def compare(meta_a: dict, rows_a: list[dict], meta_b: dict, rows_b: list[dict], examples: int = 5) -> dict:
+    a, b = by_pass(rows_a, 1), by_pass(rows_b, 1)
+    ids = [i for i in a if i in b]
+    notes = []
+    if meta_a.get("suite_hash") != meta_b.get("suite_hash"):
+        notes.append("The two runs used different versions of the suite (prompt, checker or items differ); "
+                     "only items with the same id are paired, and a difference may come from the suite.")
+    only = len(a) - len(ids), len(b) - len(ids)
+    if any(only):
+        notes.append(f"Paired on the {len(ids)} items both runs answered ({only[0]} only in A, {only[1]} only in B).")
+    if not ids:
+        return {"a": meta_a["run"], "b": meta_b["run"], "items": 0, "notes": notes + ["No items in common."]}
+    groups = [a[i]["group"] for i in ids]
+    sa, sb = [a[i]["score"] for i in ids], [b[i]["score"] for i in ids]
+    diff = mean_ci([y - x for x, y in zip(sa, sb)], groups)
+    worse = [i for i in ids if b[i]["score"] < a[i]["score"]]
+    better = [i for i in ids if b[i]["score"] > a[i]["score"]]
+    changed = [float(a[i]["answer"] != b[i]["answer"]) for i in ids]
+    out = {
+        "a": {"run": meta_a["run"], "model": meta_a["model"], "score": _ci(mean_ci(sa, groups))},
+        "b": {"run": meta_b["run"], "model": meta_b["model"], "score": _ci(mean_ci(sb, groups))},
+        "items": len(ids),
+        "difference": _ci(diff, _pp),
+        "worse_in_b": len(worse), "better_in_b": len(better),
+        "answers_changed": _ci(mean_ci(changed, groups)),
+    }
+    floors = {}
+    for name, rows in (("a", rows_a), ("b", rows_b)):
+        rep, p1 = by_pass(rows, 2), by_pass(rows, 1)
+        common = [i for i in ids if i in rep]
+        if common:
+            # per item: changed between the runs, minus changed between this run's own two passes
+            excess = [float(a[i]["answer"] != b[i]["answer"]) - float(p1[i]["answer"] != rep[i]["answer"])
+                      for i in common]
+            floors[name] = {"items": len(common),
+                            "change_rate": _ci(mean_ci([float(p1[i]["answer"] != rep[i]["answer"]) for i in common],
+                                                       [a[i]["group"] for i in common])),
+                            "excess_over_floor": _ci(mean_ci(excess, [a[i]["group"] for i in common]), _pp)}
+    out["noise_floor"] = floors or None
+    if excludes_zero(diff):
+        verdict = (f"B scores {'higher' if diff['mean'] > 0 else 'lower'} than A: "
+                   f"{_pp(diff['mean'])} [{_pp(diff['lo'])}, {_pp(diff['hi'])}], paired over {len(ids)} items.")
+    else:
+        verdict = (f"No clear difference in score: {_pp(diff['mean'])} [{_pp(diff['lo'])}, {_pp(diff['hi'])}], "
+                   f"an interval that includes zero, over {len(ids)} items.")
+    ch = out["answers_changed"]
+    verdict += f" Answers changed on {_pc(ch['mean'])} of items ({len(worse)} got worse, {len(better)} better)."
+    if floors:
+        # beyond noise only if it clears every floor measured, so a quiet run can't hide a noisy one
+        beyond = all(f["excess_over_floor"]["lo"] > 0 for f in floors.values())
+        rates = " and ".join(f"{_pc(f['change_rate']['mean'])} for {k.upper()}" for k, f in floors.items())
+        verdict += (f" The noise floor (the same model asked twice) is {rates}, so "
+                    + ("the change is beyond noise." if beyond else "the change is not clearly beyond noise."))
+    else:
+        verdict += " Neither run has a repeat pass, so there is no noise floor to compare against (run with repeat)."
+    out["verdict"] = verdict
+    out["notes"] = notes
+
+    def ex_rows(ids_):
+        return [{"id": i, "a_answer": a[i]["answer"], "b_answer": b[i]["answer"],
+                 "a_score": a[i]["score"], "b_score": b[i]["score"],
+                 "a_reply_end": excerpt(a[i]["text"], 240), "b_reply_end": excerpt(b[i]["text"], 240)}
+                for i in ids_[:examples]]
+    out["examples"] = {"worse_in_b": ex_rows(worse), "better_in_b": ex_rows(better)}
+    return out