honestboard 0.1.0__tar.gz

honestboard-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jeffery Opoku and David Banahene
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

honestboard-0.1.0/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: honestboard
+Version: 0.1.0
+Summary: Statistical checks for AI leaderboards: a selective record test with online certification, rank intervals, a confidence set of the best system and a winner's-curse correction, from per-task results.
+Author: David Banahene
+Author-email: Jeffery Opoku <jeffery.opoku01@utrgv.edu>
+License: MIT
+Project-URL: Homepage, https://github.com/jopoku16/honestboard
+Project-URL: Repository, https://github.com/jopoku16/honestboard
+Project-URL: Live page, https://jopoku16.github.io/honestboard/
+Project-URL: Issues, https://github.com/jopoku16/honestboard/issues
+Keywords: leaderboard,benchmark,evaluation,selective inference,winner's curse,AI agents,statistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Requires-Dist: scipy>=1.9
+Dynamic: license-file
+
+# honestboard
+
+Honest uncertainty for agent leaderboards. Give it the per-task results of every system on a benchmark and it tells you:
+
+- **Is this SOTA claim real?** An exact paired test of a new record against the record it replaced.
+- **What can this benchmark resolve?** The smallest difference it detects with 80% power, and how many tasks a one-point gap would need.
+- **How sure is the ranking?** Rank intervals and the probability each system is the best.
+- **Who could still be the best?** A 95% confidence set of the best system.
+- **What is the winner's score, corrected?** The top score on a leaderboard is biased upward because it was picked for being highest. `honestboard` removes most of that selection bias.
+- **How many record claims survive?** An audit of every record step in a leaderboard's history, with a Holm correction.
+- **Certify records as they arrive.** `certify_chain` runs a sequential rule along the submission history: the exact sign-test p-value conditional on the record event (with a per-date factor for submissions that share a date), at levels set by alpha-spending or by the online fallback rule (family-wise error over the whole chain, any number of records, no independence assumption), by tenure-indexed spending on the certified baseline, or by LORD++ (online FDR; certified records earn back budget), against the leaderboard holder or against the last certified record.
+
+It needs per-task outcomes (which tasks each system solved), not just the headline score. A checkout of the SWE-bench `experiments` repository loads with one call (`from_swebench_experiments`); any other leaderboard works once you write a board file (format below).
+
+## Quick start
+
+```python
+from honestboard import Board
+
+b = Board.from_json("swebench_verified.json")   # board files may carry a "clusters" map (task -> repository)
+
+b.compare("20250928_trae_doubao_seed_code", "20251205_sonar-foundation-agent_claude-opus-4-5")
+b.rank_summary(top=10)              # 95% rank intervals and top-rank frequency
+b.confidence_set_best()             # systems that cannot be ruled out as best
+b.record_audit()                    # every record step: selective exact test (with the per-date factor), Holm-adjusted
+b.record_lags()                     # how far back a record is distinguishable
+from honestboard import certify_chain
+certify_chain(b, rule="lord", baseline="certified")   # sequential certification in date order
+b.resolution()                      # minimum detectable difference, tasks needed for 1 and 2 points
+b.winner_adjusted()                 # selection-corrected score for the leader (iterated bootstrap)
+b.check_claim("some-system-name")   # the main facts about one system in one call
+```
+
+Command line:
+
+```
+python -m honestboard report swebench_verified.json
+python -m honestboard check swebench_verified.json 20251205_sonar-foundation-agent_claude-opus-4-5
+python -m honestboard pr-check experiments/evaluation/verified experiments/evaluation/verified/NEW_SUBMISSION --tasks verified_ids.json
+```
+
+`pr-check` reads a checkout of a leaderboard repository and prints a Markdown report on one submission (rank interval, margin over the current record, a paired test of the claim, what the board can resolve). `examples/leaderboard-pr-check.yml` shows how a leaderboard repository can post that report on every pull request.
+
+Board file (JSON): `{"tasks": [...ids...], "systems": [{"name": "...", "date": "20251205", "resolved": [...ids...]}], "clusters": {task: group}}`.
+For repeated trials use `"rates": {task: share_passed}` instead of `"resolved"`. A CSV with header `name,date,<task ids>` and 0/1 or rate cells also works. `clusters` is optional; `cluster=True` needs it and raises an error without it.
+
+## Methods
+
+| Question | Method |
+|---|---|
+| Is B better than A? | Exact McNemar test (0/1 outcomes) or paired sign-flip permutation test (rates): one-sided, two-sided and the selective p-value `p_selective` (the one-sided tail conditional on B having scored higher, the record test) |
+| Rank uncertainty | Task bootstrap; tied systems share a rank; top-rank frequency counts ties fractionally. `cluster=True` resamples repositories instead of tasks |
+| Confidence set of the best | Paired-bootstrap multiple comparisons with the best over all systems (max studentised difference); coverage is approximate |
+| Record audit | Selective exact paired test of each new record against the previous record (`sided="selective"`, default; `"two"` and `"one"` as sensitivities), Holm-adjusted over the steps. `tie_order` handles same-date submissions best-first (default, fewest steps) or worst-first; under best-first the challenger is the best of its date, so its p-value is multiplied by the size of that date group (`date_correction=True`, default), which the guarantees need |
+| Resolution | Normal approximation from the paired standard deviation between neighbouring top systems; checked by simulation |
+| Winner's score | Iterated plug-in bootstrap bias correction (default). The exact conditional (polyhedral) method is `winner_conditional`; two published baselines are `winner_hybrid` (the hybrid of Andrews, Kitagawa and McCloskey, 2024: conditional, with the truncation set bounded by a simultaneous projection interval, so never unbounded) and `winner_zoom` (the zoom interval of Zrnic and Fithian, 2025: simultaneous over the plausible winners only; corrects the interval, not the estimate) |
+| Record chains over time | `certify_chain`: a selective p-value given the record event, at levels from an alpha-spending series, the online fallback rule (`rule="fallback"`: a certified record passes its level to the next one) or, for `baseline="certified"` (the leader moves only on certification), the tenure-indexed rule (`rule="tenure"`); all three with family-wise error control over the chain proved under the task-wise null, any dependence. LORD++ (online FDR) is calibrated by simulation. See the paper |
+
+## Limits you should know about
+
+- It measures **task-sampling luck only**. Developers tuning on a public benchmark and run-to-run randomness of agents are outside the model, so the real overstatement of a winner is larger than the correction removes.
+- Tasks are treated as independent draws. Tasks from the same repository are correlated; `cluster=True` is a rough sensitivity check (with few clusters it is not reliable and the paired tests still ignore clusters).
+- The record test conditions on the record event (the selective p-value); it is exact for a fixed holder under the task-wise null and conservative when the previous record is a running maximum (under a global null it rejects well under 5% of record steps). A record that does not pass is not necessarily worse. It cannot be told apart from the previous record with these tasks. The task-level test treats tasks as independent; where tasks share a repository, `cluster=True` is the cautious count.
+- The winner correction removes most of the bias in simulations but not all of it, much less under a shift to other repositories, and its interval is approximate (below nominal at small sample sizes).
+- The conditional method is exact under normality but gives an unbounded answer when the winner ties or nearly ties the runner-up.
+- The record chain depends on how submissions with the same date are ordered.
+
+## Install and test
+
+```
+pip install honestboard   # from PyPI (numpy and scipy only); or, from a checkout of this folder:
+pip install -e .
+python -m pytest tests
+```
+
+MIT licensed. If you use `honestboard`, please cite the paper (see `CITATION.cff`).

honestboard-0.1.0/README.md

@@ -0,0 +1,76 @@
+# honestboard
+
+Honest uncertainty for agent leaderboards. Give it the per-task results of every system on a benchmark and it tells you:
+
+- **Is this SOTA claim real?** An exact paired test of a new record against the record it replaced.
+- **What can this benchmark resolve?** The smallest difference it detects with 80% power, and how many tasks a one-point gap would need.
+- **How sure is the ranking?** Rank intervals and the probability each system is the best.
+- **Who could still be the best?** A 95% confidence set of the best system.
+- **What is the winner's score, corrected?** The top score on a leaderboard is biased upward because it was picked for being highest. `honestboard` removes most of that selection bias.
+- **How many record claims survive?** An audit of every record step in a leaderboard's history, with a Holm correction.
+- **Certify records as they arrive.** `certify_chain` runs a sequential rule along the submission history: the exact sign-test p-value conditional on the record event (with a per-date factor for submissions that share a date), at levels set by alpha-spending or by the online fallback rule (family-wise error over the whole chain, any number of records, no independence assumption), by tenure-indexed spending on the certified baseline, or by LORD++ (online FDR; certified records earn back budget), against the leaderboard holder or against the last certified record.
+
+It needs per-task outcomes (which tasks each system solved), not just the headline score. A checkout of the SWE-bench `experiments` repository loads with one call (`from_swebench_experiments`); any other leaderboard works once you write a board file (format below).
+
+## Quick start
+
+```python
+from honestboard import Board
+
+b = Board.from_json("swebench_verified.json")   # board files may carry a "clusters" map (task -> repository)
+
+b.compare("20250928_trae_doubao_seed_code", "20251205_sonar-foundation-agent_claude-opus-4-5")
+b.rank_summary(top=10)              # 95% rank intervals and top-rank frequency
+b.confidence_set_best()             # systems that cannot be ruled out as best
+b.record_audit()                    # every record step: selective exact test (with the per-date factor), Holm-adjusted
+b.record_lags()                     # how far back a record is distinguishable
+from honestboard import certify_chain
+certify_chain(b, rule="lord", baseline="certified")   # sequential certification in date order
+b.resolution()                      # minimum detectable difference, tasks needed for 1 and 2 points
+b.winner_adjusted()                 # selection-corrected score for the leader (iterated bootstrap)
+b.check_claim("some-system-name")   # the main facts about one system in one call
+```
+
+Command line:
+
+```
+python -m honestboard report swebench_verified.json
+python -m honestboard check swebench_verified.json 20251205_sonar-foundation-agent_claude-opus-4-5
+python -m honestboard pr-check experiments/evaluation/verified experiments/evaluation/verified/NEW_SUBMISSION --tasks verified_ids.json
+```
+
+`pr-check` reads a checkout of a leaderboard repository and prints a Markdown report on one submission (rank interval, margin over the current record, a paired test of the claim, what the board can resolve). `examples/leaderboard-pr-check.yml` shows how a leaderboard repository can post that report on every pull request.
+
+Board file (JSON): `{"tasks": [...ids...], "systems": [{"name": "...", "date": "20251205", "resolved": [...ids...]}], "clusters": {task: group}}`.
+For repeated trials use `"rates": {task: share_passed}` instead of `"resolved"`. A CSV with header `name,date,<task ids>` and 0/1 or rate cells also works. `clusters` is optional; `cluster=True` needs it and raises an error without it.
+
+## Methods
+
+| Question | Method |
+|---|---|
+| Is B better than A? | Exact McNemar test (0/1 outcomes) or paired sign-flip permutation test (rates): one-sided, two-sided and the selective p-value `p_selective` (the one-sided tail conditional on B having scored higher, the record test) |
+| Rank uncertainty | Task bootstrap; tied systems share a rank; top-rank frequency counts ties fractionally. `cluster=True` resamples repositories instead of tasks |
+| Confidence set of the best | Paired-bootstrap multiple comparisons with the best over all systems (max studentised difference); coverage is approximate |
+| Record audit | Selective exact paired test of each new record against the previous record (`sided="selective"`, default; `"two"` and `"one"` as sensitivities), Holm-adjusted over the steps. `tie_order` handles same-date submissions best-first (default, fewest steps) or worst-first; under best-first the challenger is the best of its date, so its p-value is multiplied by the size of that date group (`date_correction=True`, default), which the guarantees need |
+| Resolution | Normal approximation from the paired standard deviation between neighbouring top systems; checked by simulation |
+| Winner's score | Iterated plug-in bootstrap bias correction (default). The exact conditional (polyhedral) method is `winner_conditional`; two published baselines are `winner_hybrid` (the hybrid of Andrews, Kitagawa and McCloskey, 2024: conditional, with the truncation set bounded by a simultaneous projection interval, so never unbounded) and `winner_zoom` (the zoom interval of Zrnic and Fithian, 2025: simultaneous over the plausible winners only; corrects the interval, not the estimate) |
+| Record chains over time | `certify_chain`: a selective p-value given the record event, at levels from an alpha-spending series, the online fallback rule (`rule="fallback"`: a certified record passes its level to the next one) or, for `baseline="certified"` (the leader moves only on certification), the tenure-indexed rule (`rule="tenure"`); all three with family-wise error control over the chain proved under the task-wise null, any dependence. LORD++ (online FDR) is calibrated by simulation. See the paper |
+
+## Limits you should know about
+
+- It measures **task-sampling luck only**. Developers tuning on a public benchmark and run-to-run randomness of agents are outside the model, so the real overstatement of a winner is larger than the correction removes.
+- Tasks are treated as independent draws. Tasks from the same repository are correlated; `cluster=True` is a rough sensitivity check (with few clusters it is not reliable and the paired tests still ignore clusters).
+- The record test conditions on the record event (the selective p-value); it is exact for a fixed holder under the task-wise null and conservative when the previous record is a running maximum (under a global null it rejects well under 5% of record steps). A record that does not pass is not necessarily worse. It cannot be told apart from the previous record with these tasks. The task-level test treats tasks as independent; where tasks share a repository, `cluster=True` is the cautious count.
+- The winner correction removes most of the bias in simulations but not all of it, much less under a shift to other repositories, and its interval is approximate (below nominal at small sample sizes).
+- The conditional method is exact under normality but gives an unbounded answer when the winner ties or nearly ties the runner-up.
+- The record chain depends on how submissions with the same date are ordered.
+
+## Install and test
+
+```
+pip install honestboard   # from PyPI (numpy and scipy only); or, from a checkout of this folder:
+pip install -e .
+python -m pytest tests
+```
+
+MIT licensed. If you use `honestboard`, please cite the paper (see `CITATION.cff`).

honestboard-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "honestboard"
+version = "0.1.0"
+description = "Statistical checks for AI leaderboards: a selective record test with online certification, rank intervals, a confidence set of the best system and a winner's-curse correction, from per-task results."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [
+  { name = "Jeffery Opoku", email = "jeffery.opoku01@utrgv.edu" },
+  { name = "David Banahene" },
+]
+keywords = ["leaderboard", "benchmark", "evaluation", "selective inference", "winner's curse", "AI agents", "statistics"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["numpy>=1.22", "scipy>=1.9"]
+
+[project.urls]
+Homepage = "https://github.com/jopoku16/honestboard"
+Repository = "https://github.com/jopoku16/honestboard"
+"Live page" = "https://jopoku16.github.io/honestboard/"
+Issues = "https://github.com/jopoku16/honestboard/issues"
+
+[project.scripts]
+honestboard = "honestboard.__main__:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

honestboard-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

honestboard-0.1.0/src/honestboard/__init__.py

@@ -0,0 +1,9 @@
+"""honestboard: honest uncertainty for agent leaderboards."""
+from .chain import certify_chain, fallback_level, selective_sign_pvalue
+from .core import Board
+from .loaders import board_with_submission, from_swebench_experiments
+from .selective import simultaneous_quantile, truncation_limits, winner_adjusted, winner_hybrid, winner_zoom
+
+__all__ = ["Board", "board_with_submission", "certify_chain", "fallback_level", "from_swebench_experiments", "selective_sign_pvalue",
+           "simultaneous_quantile", "truncation_limits", "winner_adjusted", "winner_hybrid", "winner_zoom"]
+__version__ = "0.1.0"

honestboard-0.1.0/src/honestboard/__main__.py

@@ -0,0 +1,131 @@
+"""Command line: python -m honestboard report BOARD.json | check BOARD.json SYSTEM"""
+import argparse
+import sys
+
+from .core import Board
+
+
+def _load(path):
+    return Board.from_csv(path) if path.endswith(".csv") else Board.from_json(path)
+
+
+def _pct(x):
+    return f"{100 * x:.1f}%"
+
+
+def cmd_report(args):
+    b = _load(args.board)
+    print(f"{b.K} systems, {b.T} tasks\n")
+    rs = b.rank_summary(top=args.top, cluster=args.cluster)
+    print("Leaders with 95% rank intervals and top-rank frequency:")
+    for r in rs:
+        print(f"  {r['name'][:52]:52s} {_pct(r['score']):>6s}  rank [{r['rank_lo']:.0f}, {r['rank_hi']:.0f}]  top-rank freq.={r['p_best']:.2f}")
+    cs = b.confidence_set_best(cluster=args.cluster)
+    print(f"\nConfidence set of the best (95%): {cs['size']} of {b.K} systems cannot be ruled out as best.")
+    if b.dates:
+        a = b.record_audit()
+        print(f"\nRecord claims: {a['n_steps']} steps, {a['n_significant_raw']} beat the previous record at the 5% level "
+              f"(selective exact paired test, conditional on the record event), {a['n_significant_holm']} after Holm correction.")
+        r = b.resolution()
+        print(f"Resolution: this board detects differences of about {100 * r['mdd']:.1f} points (80% power); "
+              f"resolving a 1-point gap would take about {r['tasks_for_1pt']:,.0f} tasks.")
+    w = b.winner_adjusted(cluster=args.cluster)
+    print(f"\nWinner {w['name']}: reported {_pct(w['observed'])}, selection-adjusted estimate {_pct(w['estimate'])}, "
+          f"95% interval [{_pct(w['ci'][0])}, {_pct(w['ci'][1])}] (naive [{_pct(w['naive_ci'][0])}, {_pct(w['naive_ci'][1])}]).")
+    c = w["conditional"]
+    print(f"  conditional method: estimate {_pct(c['estimate'])}, interval [{_pct(c['ci'][0])}, {_pct(c['ci'][1])}] "
+          f"(unbounded when the winner's margin is about zero)")
+    h = b.winner_hybrid(cluster=args.cluster, w=w["winner"])
+    z = b.winner_zoom(cluster=args.cluster, w=w["winner"])
+    print(f"  hybrid (Andrews, Kitagawa and McCloskey): estimate {_pct(h['estimate'])}, interval [{_pct(h['ci'][0])}, {_pct(h['ci'][1])}]"
+          + (" (undefined at an exact tie for first)" if h["degenerate"] else ""))
+    print(f"  zoom interval (Zrnic and Fithian): [{_pct(z['ci'][0])}, {_pct(z['ci'][1])}] over {z['n_plausible']} plausible winners")
+
+
+def cmd_check(args):
+    b = _load(args.board)
+    r = b.check_claim(args.system, cluster=args.cluster)
+    print(f"{r['system']}: score {_pct(r['score'])}, observed rank {r['rank_observed']}, "
+          f"95% rank interval {r['rank_interval_95']}, top-rank freq.={r['p_best']:.2f}")
+    print(f"Inside the 95% confidence set of the best: {r['in_confidence_set_of_best']} (set size {r['confidence_set_size']})")
+    c = r["vs_best_other"]
+    print(f"Paired test, {c['b']} vs {c['a']}: difference {100 * c['diff']:+.1f} pts, one-sided exact p={c['p_one_sided']:.3f}")
+    if "winner_adjusted" in r:
+        w = r["winner_adjusted"]
+        print(f"Selection-adjusted score: {_pct(w['estimate'])}, 95% interval [{_pct(w['ci'][0])}, {_pct(w['ci'][1])}] "
+              f"(reported {_pct(w['observed'])})")
+
+
+def cmd_pr_check(args):
+    """Markdown report on one new submission, suitable for a pull-request comment."""
+    import json
+    import numpy as np
+    from .loaders import board_with_submission
+    tasks = None
+    if args.tasks:
+        with open(args.tasks, encoding="utf-8") as f:
+            tasks = json.load(f)
+    b, k = board_with_submission(args.eval_dir, args.new_dir, tasks=tasks)
+    if tasks is None:
+        print("_Warning: no --tasks file was given, so tasks that no submission solved are not counted and every score is overstated._\n")
+    if b.K < 2:
+        print(f"### honestboard check: `{b.names[k]}`\n\n- First submission on this board: score {_pct(b.scores()[k])} on {b.T:,} tasks; nothing to compare with yet.")
+        return
+    sc = b.scores()
+    others = [j for j in np.argsort(-sc, kind="stable") if j != k]
+    prev = others[0]
+    res = b.resolution()
+    rs = {r["name"]: r for r in b.rank_summary(B=4000)}
+    me = rs[b.names[k]]
+    margin = sc[k] - sc[prev]
+    lines = [f"### honestboard check: `{b.names[k]}`", ""]
+    lines.append(f"- Score: **{_pct(sc[k])}** on {b.T:,} tasks; observed rank {1 + int((sc > sc[k]).sum())} of {b.K}; "
+                 f"95% rank interval {me['rank_lo']:.0f} to {me['rank_hi']:.0f}; top-rank frequency {me['p_best']:.2f}.")
+    if margin > 1e-12:
+        c = b.compare(b.names[prev], b.names[k], B=3000)
+        verdict = "beats" if c["p_selective"] <= 0.05 else "cannot be told apart from"
+        lines.append(f"- Current record: `{b.names[prev]}` at {_pct(sc[prev])}. This submission is {100 * margin:+.1f} points above it "
+                     f"(selective exact paired p, conditional on it scoring higher = {c['p_selective']:.3f}), so it **{verdict}** the current record at the 5% level.")
+        if abs(100 * margin) < 100 * res["mdd"]:
+            lines.append(f"- This board detects differences of about {100 * res['mdd']:.1f} points with 80% power, so a margin of {100 * margin:.1f} points is below its resolution.")
+        w = b.winner_adjusted(B=3000)
+        lines.append(f"- Selection-adjusted score for the new leader: {_pct(w['estimate'])} (95% interval {_pct(w['ci'][0])} to {_pct(w['ci'][1])}).")
+    elif abs(margin) <= 1e-12:
+        lines.append(f"- Current record: `{b.names[prev]}` at {_pct(sc[prev])}. This submission ties it exactly and does not set a new record.")
+    else:
+        c = b.compare(b.names[k], b.names[prev], B=3000)
+        lines.append(f"- Current record: `{b.names[prev]}` at {_pct(sc[prev])}. This submission is {100 * -margin:.1f} points below it and does not set a new record "
+                     f"(one-sided p that the record is better = {c['p_one_sided']:.3f}).")
+    lines.append("")
+    lines.append("_Task sampling only; single runs; see the honestboard documentation for limits._")
+    print("\n".join(lines))
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="honestboard")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    p = sub.add_parser("pr-check", help="check one new submission against a leaderboard checkout and print Markdown")
+    p.add_argument("eval_dir", help="evaluation folder of the leaderboard checkout, for example experiments/evaluation/verified")
+    p.add_argument("new_dir", help="the new submission folder")
+    p.add_argument("--tasks", help="JSON file with the benchmark's full task id list (recommended)")
+    p.set_defaults(fn=cmd_pr_check)
+    for name, fn in (("report", cmd_report), ("check", cmd_check)):
+        p = sub.add_parser(name)
+        p.add_argument("board")
+        if name == "report":
+            p.add_argument("--top", type=int, default=10)
+        else:
+            p.add_argument("system")
+        p.add_argument("--cluster", action="store_true",
+                       help="resample task clusters (needs a 'clusters' map in the board file) instead of tasks")
+        p.set_defaults(fn=fn)
+    args = ap.parse_args(argv)
+    try:
+        args.fn(args)
+    except ValueError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 2
+
+
+if __name__ == "__main__":
+    sys.exit(main())

honestboard-0.1.0/src/honestboard/chain.py

@@ -0,0 +1,244 @@
+"""Sequential certification of record claims on a leaderboard.
+
+A leaderboard tests a submission only when it scores higher than the current record. Three things make a naive
+test wrong: the record event itself (selection on the sign of the difference), the holder being a running
+maximum, and the unknown, data-dependent number of records that will be tested. This module provides:
+
+* `selective_sign_pvalue`: the exact sign-test p-value conditional on the record event,
+  P(X >= n_b | X > n/2) with X ~ Bin(n, 1/2), n = n_a + n_b discordant tasks. Under equal ability it is
+  super-uniform given that the step is a record (for a fixed holder); a holder that is a running maximum makes
+  it conservative.
+* `selective_signflip_pvalue`: the same idea for per-task rates, by Monte Carlo sign flips.
+* `alpha_spending_level`: alpha_i = alpha * 6 / (pi^2 i^2) for the i-th record. The family-wise error over the
+  whole chain is at most alpha for ANY number of records and ANY dependence, by the union bound, provided each
+  step's p-value is valid given that the step occurred.
+* `lord_level`: LORD++ (Ramdas, Yang, Wainwright and Jordan, 2017), an online false-discovery-rate rule in which
+  each certified record earns back testing budget, so late records remain testable. Its guarantee is proved
+  for independent p-values; with shared tasks the p-values are dependent, so it is checked by simulation.
+* `certify_chain`: run a rule along a board's submission history, either against the leaderboard holder or
+  against the last certified record (a baseline that moves only when a record is certified).
+"""
+from __future__ import annotations
+
+import numpy as np
+from scipy import stats
+
+from . import selective
+
+
+# ---- per-step p-values ----------------------------------------------
+def selective_sign_pvalue(n_b: int, n_a: int) -> float:
+    """P(X >= n_b | X > n/2), X ~ Bin(n, 1/2). Equals 1 unless n_b > n_a."""
+    n = int(n_a + n_b)
+    if n == 0 or n_b <= n_a:
+        return 1.0
+    tail = stats.binom.sf(n_b - 1, n, 0.5)          # P(X >= n_b)
+    cond = stats.binom.sf(n // 2, n, 0.5)           # P(X > n/2)
+    return float(min(1.0, tail / cond))
+
+
+SIGNFLIP_DRAWS = 200000
+
+
+def signflip_pvalues(d, rng=None, draws=SIGNFLIP_DRAWS, chunk=20000):
+    """Rates version of the paired tests: one-sided, two-sided and selective p-values for the sum S of randomly
+    signed per-task differences d, by Monte Carlo with `draws` sign patterns (generated in chunks to bound memory).
+    The selective p-value is P(S >= s_obs | S > 0). With 2e5 patterns about 1e5 fall in {S > 0}, so a p-value near
+    0.001 has a Monte Carlo standard error of about 1e-4. `compare` and `certify_chain` draw the patterns from
+    the same seed convention (seed + 7*b + a for the indices of holder a and challenger b), so the audit and the
+    certification agree to the last digit. Returns (p_one_sided, p_two_sided, p_selective)."""
+    d = np.asarray(d, dtype=float)
+    obs = d.sum()
+    rng = np.random.default_rng(0) if rng is None else rng
+    pos = hit = hit2 = 0
+    done = 0
+    while done < draws:
+        m = min(chunk, draws - done)
+        null = rng.choice([-1.0, 1.0], size=(m, d.size)) @ d
+        hit += int((null >= obs - 1e-12).sum())
+        hit2 += int((np.abs(null) >= abs(obs) - 1e-12).sum())
+        pos += int((null > 1e-12).sum())
+        done += m
+    p1 = (1 + hit) / (1 + draws)
+    p2 = (1 + hit2) / (1 + draws)
+    ps = 1.0 if (obs <= 1e-12 or pos == 0) else min(1.0, (1 + hit) / (1 + pos))
+    return float(p1), float(p2), float(ps)
+
+
+def selective_signflip_pvalue(d, rng=None, draws=SIGNFLIP_DRAWS) -> float:
+    """Rates version of the record test: P(S >= s_obs | S > 0) under random sign flips of the per-task differences d."""
+    return signflip_pvalues(d, rng, draws)[2]
+
+
+def polyhedral_step_pvalue(p_hat, sigma, b, a, earlier, lattice=0.0) -> float:
+    """Selective p-value under p_hat ~ N(theta, sigma) for H0: theta_b <= theta_a, conditional on the record
+    event {p_b > p_a} and on a having been the holder {p_a >= p_j for every earlier j != a}. `lattice` (1/T for
+    pass counts) applies a continuity correction; without it, ties with the closest competitor put the statistic
+    on the truncation boundary and the p-value collapses."""
+    K = len(p_hat)
+    rows, rhs = [], []
+    r = np.zeros(K); r[a] = 1.0; r[b] = -1.0
+    rows.append(r); rhs.append(0.0)                 # p_a - p_b <= 0
+    for j in earlier:
+        if j == a:
+            continue
+        r = np.zeros(K); r[j] = 1.0; r[a] = -1.0
+        rows.append(r); rhs.append(0.0)             # p_j - p_a <= 0
+    eta = np.zeros(K); eta[b] = 1.0; eta[a] = -1.0
+    return selective.polyhedral_pvalue(np.asarray(p_hat, float), sigma, eta, np.array(rows), np.array(rhs), lattice=lattice)
+
+
+# ---- test levels ----------------------------------------------------
+def alpha_spending_level(i: int, alpha: float) -> float:
+    """Level for the i-th record (i = 1, 2, ...); the levels sum to alpha."""
+    return alpha * 6.0 / (np.pi ** 2 * i ** 2)
+
+
+def lord_gamma(j: int, kind: str = "quadratic") -> float:
+    """A spending sequence with sum at most 1. 'quadratic': 6 / (pi^2 j^2), which sums to 1 and suits chains of
+    tens of records. 'log': the sequence published with LORD++, 0.0722 log(max(j, 2)) / (j e^{sqrt(log j)}), which
+    sums to 1 only in the limit and is built for very long streams."""
+    if j < 1:
+        return 0.0
+    if kind == "quadratic":
+        return float(6.0 / (np.pi ** 2 * j ** 2))
+    if j == 1:
+        return float(0.0722 * np.log(2))
+    return float(0.0722 * np.log(max(j, 2)) / (j * np.exp(np.sqrt(np.log(j)))))
+
+
+def lord_level(t: int, alpha: float, w0: float, rejection_times, kind: str = "quadratic") -> float:
+    """LORD++ level for the t-th test given the times of earlier rejections (1-indexed test counts)."""
+    lvl = lord_gamma(t, kind) * w0
+    if rejection_times:
+        lvl += (alpha - w0) * lord_gamma(t - rejection_times[0], kind)
+        for tau in rejection_times[1:]:
+            lvl += alpha * lord_gamma(t - tau, kind)
+    return float(lvl)
+
+
+# ---- the procedure --------------------------------------------------
+def _submission_order(board, tie_order="best_first"):
+    if board.dates is None:
+        raise ValueError("certification needs dates")
+    undated = [n for n, d in zip(board.names, board.dates) if not d]
+    if undated:
+        raise ValueError(f"{len(undated)} systems have no date (first: {undated[0]})")
+    sc = board.scores()
+    sign = -1.0 if tie_order == "best_first" else 1.0
+    return sorted(range(board.K), key=lambda k: (board.dates[k], sign * sc[k]))
+
+
+def _step_pvalue(board, a, b, pvalue, sigma, earlier, seed):
+    if pvalue == "polyhedral":
+        return polyhedral_step_pvalue(board.scores(), sigma, b, a, earlier, lattice=1.0 / board.T)
+    ya, yb = board.Y[a], board.Y[b]
+    if board.is_binary:
+        n_b = int(((yb == 1) & (ya == 0)).sum())
+        n_a = int(((ya == 1) & (yb == 0)).sum())
+        return selective_sign_pvalue(n_b, n_a)
+    return selective_signflip_pvalue(yb.astype(float) - ya.astype(float), np.random.default_rng(seed + 7 * b + a))
+
+
+def fallback_level(t: int, alpha: float, prev_level: float, prev_certified: bool) -> float:
+    """Online fallback (Wiens 2003; Tian and Ramdas 2021): the t-th record is tested at its spending level plus the
+    level of the previous record if that record was certified, so a run of certified records accumulates budget.
+    Family-wise error over the chain is at most alpha for any dependence, by the same argument as for spending:
+    on the event that step t is the first null step certified, every step in the run that feeds it is non-null, and
+    the runs feeding different null steps are disjoint, so the levels available to null steps sum to at most alpha."""
+    return float(alpha_spending_level(t, alpha) + (prev_level if prev_certified else 0.0))
+
+
+def tenure_level(alpha: float, n_certified: int, dates_since_holder: int) -> float:
+    """Tenure-indexed spending for the certified baseline: alpha * gamma(j) * gamma(m) with j the index of the
+    current holder's tenure (one plus the number of certifications so far) and m the number of distinct submission
+    dates since the holder's date, gamma(k) = 6/(pi^2 k^2). Both indices are fixed by the submission calendar and
+    the earlier history, which is what the chain-wise guarantee needs; several submissions on one date share the
+    slot and are handled by the per-date correction. The price is that budget is spent on every date, tested or
+    not."""
+    return float(alpha * (6.0 / (np.pi ** 2 * n_certified ** 2)) * (6.0 / (np.pi ** 2 * dates_since_holder ** 2)))
+
+
+def certify_chain(board, alpha=0.05, rule="lord", pvalue="exact", baseline="leader", tie_order="best_first", w0=None, seed=0,
+                  date_correction=True):
+    """Walk the submissions in date order and certify records.
+
+    rule: 'unadjusted' (level alpha every step), 'holm' (retrospective, over the realised chain),
+          'spending' (alpha * 6 / (pi^2 i^2) for the i-th tested record; family-wise control for any number of
+          records), 'fallback' (spending plus the level of the previous record when it was certified; the same
+          family-wise guarantee with more power along runs of genuine records), 'tenure' (certified baseline only:
+          spending indexed by the holder's tenure and the dates since its date, which carries the chain-wise
+          guarantee for that baseline), 'lord' (LORD++ online FDR).
+    pvalue: 'exact' (sign test conditional on the record event; sign flips for rates) or 'polyhedral'
+            (normal approximation, conditional on the record event and on the holder having been the maximum).
+    baseline: 'leader' tests each leaderboard record against the record it replaced; 'certified' tests every
+              submission that beats the last certified record against that record (the baseline moves only when
+              a record is certified).
+    date_correction: under the best-first ordering a tested submission was selected among the submissions of its
+              date; the p-value is multiplied by the size of that group (a Bonferroni factor), which the chain-wise
+              guarantees need. The worst-first ordering is a bound and is not corrected."""
+    if rule not in ("unadjusted", "holm", "spending", "fallback", "tenure", "lord"):
+        raise ValueError("rule must be 'unadjusted', 'holm', 'spending', 'fallback', 'tenure' or 'lord'")
+    if pvalue not in ("exact", "polyhedral"):
+        raise ValueError("pvalue must be 'exact' or 'polyhedral'")
+    if baseline not in ("leader", "certified"):
+        raise ValueError("baseline must be 'leader' or 'certified'")
+    if rule == "tenure" and baseline != "certified":
+        raise ValueError("the tenure rule is defined for the certified baseline")
+    w0 = alpha / 2 if w0 is None else w0
+    order = _submission_order(board, tie_order)
+    sc = board.scores()
+    sigma = board.sigma() if pvalue == "polyhedral" else None
+    correct = bool(date_correction and tie_order == "best_first")
+    date_rank = {}                                     # distinct dates in order, for the tenure index
+    for k in order:
+        date_rank.setdefault(board.dates[k], len(date_rank))
+    holder = order[0]
+    certified_holder = order[0]
+    steps, rejections, t = [], [], 0
+    prev_level, prev_cert = 0.0, False
+    for pos, k in enumerate(order[1:], start=1):
+        ref = holder if baseline == "leader" else certified_holder
+        if sc[k] > sc[ref] + 1e-12:
+            t += 1
+            p_raw = _step_pvalue(board, ref, k, pvalue, sigma, order[:pos], seed)
+            g = board.date_group_size(k) if correct else 1
+            p = min(1.0, g * p_raw)
+            if rule == "unadjusted":
+                level = alpha
+            elif rule == "spending":
+                level = alpha_spending_level(t, alpha)
+            elif rule == "fallback":
+                level = fallback_level(t, alpha, prev_level, prev_cert)
+            elif rule == "tenure":
+                level = tenure_level(alpha, len(rejections) + 1, max(1, date_rank[board.dates[k]] - date_rank[board.dates[ref]]))
+            elif rule == "lord":
+                level = lord_level(t, alpha, w0, rejections)
+            else:
+                level = None
+            cert = (p <= level) if level is not None else None
+            prev_level, prev_cert = (level or 0.0), bool(cert)
+            if cert:
+                rejections.append(t)
+                certified_holder = k
+            steps.append({"test": t, "position": pos, "new": board.names[k], "reference": board.names[ref],
+                          "margin": float(sc[k] - sc[ref]), "p": float(p), "p_uncorrected": float(p_raw), "date_group": g,
+                          "level": level, "certified": cert})
+        if sc[k] > sc[holder] + 1e-12:
+            holder = k
+    if rule == "holm":
+        m = len(steps)
+        ps = np.array([s["p"] for s in steps]) if m else np.array([])
+        rej = np.zeros(m, dtype=bool)
+        for rank, idx in enumerate(np.argsort(ps)):
+            if ps[idx] <= alpha / (m - rank):
+                rej[idx] = True
+            else:
+                break
+        for s, r in zip(steps, rej):
+            s["certified"] = bool(r)
+            s["level"] = None
+    return {"rule": rule, "pvalue": pvalue, "baseline": baseline, "alpha": alpha, "date_correction": correct, "n_tests": len(steps),
+            "n_certified": int(sum(1 for s in steps if s["certified"])),
+            "certified": [s["new"] for s in steps if s["certified"]], "steps": steps,
+            "final_certified_holder": board.names[certified_holder]}