auditable 0.3.0__tar.gz → 0.4.0__tar.gz

auditable-0.4.0/CHANGELOG.md

@@ -0,0 +1,42 @@
+# Changelog
+
+## 0.4.0 (2026-11-14)
+
+0.4.0 adds leaderboards: `auditable.bench` reruns the tables of the CatchBench board on the same populations and protocols, and places any new method on them.
+
+### Added
+
+- `auditable.bench`: `leaderboard(task_id, methods=None, extra=())` scores methods on a task and returns a ranked `Board` (`print`, `to_markdown()`, `to_dict()`, `compare_golden()`, `rank_of()`); `extra=[MyMethod]` adds a method that is not in the registry. A method whose extra is missing is listed as skipped with the extra named, and a method that raises is listed as an error while the rest of the board is still scored. Oracle rows, which read the labels, are shown as references and never ranked; control rows are ranked and marked.
+- Six tasks, each reproducing a table of the CatchBench 0.1.3 board row by row: `detect/swegym` (376 runs) and `detect/tau` (660 runs), run-level failure detection by ROC-AUC; `live/swegym` and `live/tau`, early warning on 25, 50, 75 and 100 percent prefixes; `localize/whoandwhen` (126 failed runs), decisive-step localization by top-1, top-3 and MRR, including 31 LLM-judge rows replayed from CatchBench's committed prediction caches without an API call; and `pre/over_privilege` (1,187 configurations), configuration audit by F1, precision and recall, overall and per source. The populations are read through agentpile at the dataset revisions CatchBench pins, with the selection rules of GRADE commit 3839a57; the protocols are ported from CatchBench's adapters.
+- `run.lr_spline_flat`, the flexible size control: a cubic spline of the four GRADE `flat` counts, then the board's logistic regression, ported from the `size (spline)` arm of CatchBench's size audits. It joins the `detect` and `live` boards as a control. On `detect/swegym` it ranks second, above both GRADE dependency regressions; on `detect/tau` it ranks below `run.lr_flatdep`. The library now holds 44 methods.
+- Golden files with the board values of every row and the SHA-256 of the board they come from; the spline control's value comes from CatchBench's committed size audits instead, with a 0.001 tolerance. `python -m auditable.bench <task> --golden` compares a run with them and fails on a mismatch, on an error row, on a golden row left unchecked (skipped, missing a metric, absent from the task's default rows, or in a golden file that is missing), and on a run that compared no golden value; `--allow-skipped` accepts rows skipped for a missing extra. Three rows differ on macOS arm64 by a known platform effect and are reported as platform values; `--exact` counts them as mismatches, and the weekly Linux run uses it.
+- `python -m auditable.bench`, the command-line entry: lists the tasks, prints a board, or emits it as Markdown or JSON.
+- The `bench` extra (`pyod`, `scikit-learn`, `numpy`, `networkx`, `agentpile[hf]`); the graph rows also need `graph-ad`.
+- CI: a `bench-core` job runs the bench unit tests, which download nothing, a weekly `bench golden` workflow reruns every task on Linux against its golden file, and a weekly `bench platforms` workflow installs the package from a clean checkout on Linux, macOS, and Windows (Python 3.10 and 3.13) with the README's graph-ad steps and checks three tasks against their golden files.
+
+## 0.3.0 (2026-10-25)
+
+0.3.0 turns `auditable` into a library of auditing methods for AI agents, with one interface per family and the ability to read public agent traces through agentpile.
+
+### Added
+
+- `auditable.methods`: 43 methods in four families, each with one interface. The `config` family (8 methods) audits an agent configuration before a run, `online` (5) raises causal alarms while a run streams, and `run` (16) and `step` (14) detect a failed run and localize its decisive step afterwards. A registry (`list_methods`, `get_method`) returns each method's `MethodCard`, which records its fit scope, fidelity to its source, required fields, extras and citations. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench; every method that reproduces a CatchBench 0.1.2 board row matches it sample by sample.
+- `auditable.featurize`: trace normalization for step lists, run dicts, typed graphs, `SessionGraph` and `TraceView` inputs, and the features the methods compute (the GRADE structural layers, the Who&When step matrix, the step dependency graph, PyG encodings).
+- `auditable.llm`: a `prompt -> reply` backend protocol, `FixedResponseBackend` for tests and replays, `OpenAICompatibleBackend` (standard library only), and a prediction cache.
+- `auditable.from_agentpile`, `configs_from_agentpile`, `step_targets`, `run_targets` and `auditable.datasets.load_traces` read agentpile records (public datasets, trace files, captured runs) into the methods' inputs and labels. `convention="grade"` with `deps="grade"` reproduces the steps, dependencies and graphs of GRADE's converters (commit 3839a57) on seven corpora; `convention="generic"` (the default) makes one step per message and one per tool call.
+- Extras `tabular-ad`, `graph-ad` and `agentpile`; the core install still has no third-party dependency.
+- `docs/methods-contract.md`, the contract a method follows and the behaviours a pinned consumer such as CatchBench relies on.
+
+### Behaviour of the Conversion
+
+- In `convention="generic"`, content given as a list of parts (dicts with a string `type`, possibly mixed with bare strings) keeps its text and writes every other part (an image, a document) as `[<type>]`, dropping that part's other fields; `meta["report"]` counts those parts and says how each step of the record maps into the trace.
+- `convention="grade"` raises `ValueError` for a run from a dataset revision other than the one its projection was checked on.
+- A run with subagent trajectories is refused by default; the error names the two ways to read it, `from_agentpile(run.subagents, ...)` and `subagents="skip"`.
+
+### Deprecated
+
+- `ActionGate`, `ReferenceLedger` and `DataAuditor` emit a `DeprecationWarning` when constructed and will be removed in a later release.
+
+## Earlier Releases
+
+Releases up to 0.2.1 are listed on [PyPI](https://pypi.org/project/auditable/#history) and as [GitHub tags](https://github.com/yzhao062/auditable/tags).

{auditable-0.3.0 → auditable-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: auditable
-Version: 0.3.0
+Version: 0.4.0
 Summary: A library of auditing methods for AI agents: configuration audits, run-level failure detection, step-level fault localization, and online monitors, behind one API per family.
 Project-URL: Homepage, https://github.com/yzhao062/auditable
 Project-URL: Repository, https://github.com/yzhao062/auditable
@@ -27,6 +27,12 @@ Requires-Dist: agentpile[hf]<0.4,>=0.3; extra == 'agentpile'
 Requires-Dist: numpy>=1.21; extra == 'agentpile'
 Provides-Extra: anomaly
 Requires-Dist: pyod>=1.1.0; extra == 'anomaly'
+Provides-Extra: bench
+Requires-Dist: agentpile[hf]<0.4,>=0.3; extra == 'bench'
+Requires-Dist: networkx>=3.0; extra == 'bench'
+Requires-Dist: numpy>=1.21; extra == 'bench'
+Requires-Dist: pyod>=1.1.0; extra == 'bench'
+Requires-Dist: scikit-learn>=1.0; extra == 'bench'
 Provides-Extra: dev
 Requires-Dist: build>=1.0; extra == 'dev'
 Requires-Dist: pyod>=1.1.0; extra == 'dev'
@@ -73,20 +79,114 @@ Description-Content-Type: text/markdown
 
 # auditable
 
-**A library of auditing methods for AI agents: configuration audits before a run, online monitors while it runs, and failure detection and fault localization after it, with one interface per family.**
+**A library of auditing methods for AI agents, with a leaderboard for each task: configuration audits before a run, online monitors while it runs, and failure detection and fault localization after it.**
 
 [![PyPI](https://img.shields.io/pypi/v/auditable.svg)](https://pypi.org/project/auditable/)
 [![Python](https://img.shields.io/pypi/pyversions/auditable.svg)](https://pypi.org/project/auditable/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/yzhao062/auditable/blob/main/LICENSE)
 [![CI](https://github.com/yzhao062/auditable/actions/workflows/ci.yml/badge.svg)](https://github.com/yzhao062/auditable/actions/workflows/ci.yml)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yzhao062/auditable/blob/main/examples/leaderboard.ipynb)
 
-[Quick Start](#quick-start) · [Real Agent Traces](#real-agent-traces) · [Methods](#methods) · [Install](#install) · [Reproducing CatchBench](#reproducing-catchbench) · [Adding a Method](#adding-a-method) · [Citation](#citation)
+[Leaderboard](#leaderboard) · [Put Your Method on the Board](#put-your-method-on-the-board) · [Tasks](#tasks) · [Quick Start](#quick-start) · [Real Agent Traces](#real-agent-traces) · [Methods](#methods) · [Install](#install) · [Adding a Method](#adding-a-method) · [Citation](#citation)
 
 </div>
 
+`auditable` is a library of auditing methods for AI agents with a leaderboard for each task: one command reruns the [CatchBench](https://github.com/yzhao062/catchbench) board on your machine, and a method you write gets a rank on the same population and protocol.
+
+## Leaderboard
+
+Did the run fail? The board below ranks 17 methods by ROC-AUC on 376 SWE-Gym OpenHands runs, 188 of which failed. One command reruns it on the population and protocol of the CatchBench board:
+
+```bash
+pip install "auditable[bench]"
+python -m auditable.bench detect/swegym
+```
+
+| Rank | Method | Algorithm | ROC-AUC |
+|---|---|---|---|
+| 1 | `run.pygod_gaan` | GAAN adversarial detector (PyGOD) | 0.850 |
+| 2 | `run.lr_spline_flat` | Logistic regression on a cubic spline of run size, a control | 0.832 |
+| 3 | `run.gsafeguard_lite` | G-Safeguard-Lite (supervised graph classifier) | 0.828 |
+| 4 | `run.lr_full` | Logistic regression on all raw layers (GRADE full) | 0.819 |
+| 5 | `run.lr_flatdep` | Logistic regression on size and dependency (GRADE flatdep) | 0.804 |
+| 6 | `run.guardian_lite` | GUARDIAN-Lite (attribute-reconstruction autoencoder) | 0.767 |
+| 7 | `run.pyod_ecod` | ECOD on run size | 0.765 |
+| 8 | `run.pygod_conad` | CONAD contrastive graph autoencoder (PyGOD) | 0.750 |
+| 9 | `run.lr_flat` | Logistic regression on run size (GRADE flat), the size baseline | 0.663 |
+| 10 | `run.pyod_copod` | COPOD on run size | 0.625 |
+| 11 | `run.pygod_anomalydae` | AnomalyDAE dual autoencoder (PyGOD) | 0.592 |
+| 12 | `run.pyod_lof` | Local outlier factor on run size | 0.584 |
+| 13 | `run.pyod_iforest` | Isolation Forest on run size | 0.571 |
+| 14 | `run.pygod_dominant` | DOMINANT graph autoencoder (PyGOD) | 0.547 |
+| 15 | `run.random` | Random run scores, a control | 0.483 |
+| 16 | `run.pyod_knn` | KNN distance on run size | 0.446 |
+| 17 | `run.pyod_hbos` | HBOS on run size | 0.319 |
+
+PyGOD's GAAN, a literature baseline, leads this board at 0.850. Second, at 0.832, is the flexible size control `run.lr_spline_flat`: a cubic spline of the same four run-size counts the linear size baseline `run.lr_flat` reads (0.663). Run size alone, fitted without a linear restriction, ranks above G-Safeguard-Lite and above both GRADE dependency regressions (0.819 and 0.804), so this board does not show that dependency structure predicts failure beyond size, the reading GRADE retired on 2026-09-08. Read a new row against both size controls; on tau-bench the order changes, and `run.lr_flatdep` (0.665) ranks above the spline control (0.631).
+
+The 16 CatchBench rows equal the CatchBench board, which was computed on Linux; the spline control is not a board row, and its reference value is the `size (spline)` arm of CatchBench's committed size audit. On macOS, `run.pyod_lof` prints 0.582 instead of 0.584, and `--golden` reports that as a platform value and not as a mismatch. The first run downloads the data through agentpile; later runs load SWE-Gym from the cache in about 3 seconds, and the full board takes about 70 seconds on a laptop CPU, of which PyGOD GAAN is the slowest row at about 37 seconds. The graph rows need `auditable[graph-ad]` and the PyG sampling backend ([Install](#install)); without them those rows are listed as skipped with the extra named, and the rest of the board is still scored.
+
+```bash
+python -m auditable.bench                                                    # list the tasks
+python -m auditable.bench detect/swegym --methods run.lr_flat run.pyod_ecod  # score chosen rows only
+python -m auditable.bench detect/swegym --golden                             # exit 1 if a row differs from the CatchBench board
+python -m auditable.bench detect/swegym --markdown                           # a Markdown table; --json prints JSON
+```
+
+## Put Your Method on the Board
+
+A method is a class with a `MethodCard`. Pass it as `extra=` and `leaderboard` scores it beside the default rows, under the protocol of its kind: a supervised detector gets the mean ROC-AUC over five seeds of stratified 5-fold cross-validation with a fresh estimator per fold, and an unsupervised one gets a single `fit_score` over the whole population. The snippet below adds a random forest on run size, a deliberately plain method:
+
+```python
+from sklearn.ensemble import RandomForestClassifier
+
+from auditable.base import BaseRunDetector, MethodCard
+from auditable.bench import leaderboard
+from auditable.featurize import grade_layer_matrix
+
+
+class ForestOnSize(BaseRunDetector):
+    card = MethodCard(method_id="run.forest_flat", name="Random forest on run size", family="run",
+                      fit_scope="inductive", supervised=True, output_kind="score", fidelity="original")
+
+    def fit(self, runs, y=None):
+        self.model_ = RandomForestClassifier(200, random_state=0)
+        self.model_.fit(grade_layer_matrix(list(runs), "flat"), y)
+        self.decision_scores_ = self.decision_function(runs)
+        return self
+
+    def decision_function(self, runs):
+        return self.model_.predict_proba(grade_layer_matrix(list(runs), "flat"))[:, 1]
+
+
+board = leaderboard("detect/swegym", extra=[ForestOnSize])
+print(board.rank_of("run.forest_flat"), board["run.forest_flat"].scores)
+```
+
+The last line prints the rank of the new row among all rows and its scores. `print(board)` shows the full table, with the CatchBench value in brackets beside each row that has one, and `board.to_markdown()`, `board.to_dict()` and `board.compare_golden()` export it. `methods=[...]` scores only the rows you name, which is quick while you iterate (`leaderboard("detect/swegym", methods=["run.lr_flat", ForestOnSize])`), and `Entry(method, kwargs={...}, label="...", key="...")` adds a row for a variant with constructor arguments (give each variant its own key, because a row that keeps the method id is compared with that method's CatchBench value). A method that raises is listed as an error and the rest of the board is still scored. The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions for the four families.
+
+## Tasks
+
+Six tasks ship with 0.4.0. Each fixes a population and a protocol, so every method on a board is scored on the same data in the same way. `detect` scores the run detectors, `localize` the step locators, `live` the online monitors (run detectors also take part, scored on prefixes of the run), and `pre` the configuration auditors.
+
+| Task | Question | Population | Metric |
+|---|---|---|---|
+| `detect/swegym` | Did the run fail? | SWE-Gym OpenHands trajectories, one scaffold: 376 runs, 188 failed | ROC-AUC |
+| `detect/tau` | Did the run fail? | tau-bench: 660 runs, 363 failed | ROC-AUC |
+| `localize/whoandwhen` | Which step was the decisive mistake? | Who&When: 126 failed multi-agent runs with a human mistake step | top-1, top-3, MRR (mean reciprocal rank) |
+| `live/swegym` | Can the failure be caught early? | SWE-Gym runs, read as prefixes of 25, 50, 75 and 100 percent of their steps | mean prefix ROC-AUC |
+| `live/tau` | Can the failure be caught early? | tau-bench runs, read as prefixes of 25, 50, 75 and 100 percent of their steps | mean prefix ROC-AUC |
+| `pre/over_privilege` | Is the agent configuration over-privileged before it runs? | 1,187 configurations from six sources | F1, precision, recall |
+
+Every task id works with the commands above: `python -m auditable.bench <task>` reruns the board, and `leaderboard("<task>", extra=[...])` places a method on it. The LLM judges on `localize/whoandwhen` and `pre/over_privilege` are replayed from committed caches, so those boards need no API call.
+
+The CatchBench board shows how the other tasks read. On Who&When localization, the best LLM judges (all-at-once) reach top-1 of about 0.42 to 0.45 against 0.119 for random, and the structural blast rule ties the position prior at 0.159 top-1. The tau-bench detection board is led by `run.lr_flatdep` and `run.lr_full`, tied at 0.665, against 0.619 for the size baseline. For over-privilege, the best rule scanner `config.owasp_asi_combined` reaches F1 0.654, flagging every configuration reaches 0.601, and an LLM judge reaches 0.695.
+
+## What Is in the Library
+
 `auditable` collects methods that audit LLM agents at three points: an agent configuration before it runs (PRE), a run while it streams (LIVE), and a finished run (POST). Methods are grouped into four families, and each family has one interface, in the way PyOD gives outlier detectors one interface. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN, G-Safeguard, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench.
 
-[CatchBench](https://github.com/yzhao062/catchbench) is the benchmark for these methods: the tasks, corpora, evaluation protocols, and metrics live there. Every method here that appears on the CatchBench board reproduces its row sample by sample, and the full board reruns byte-identical with those rows computed by this library ([Reproducing CatchBench](#reproducing-catchbench)).
+[CatchBench](https://github.com/yzhao062/catchbench) is the paper and benchmark whose board the leaderboards reproduce: `auditable.bench` carries its populations and evaluation protocols, and every method here that appears on its board reproduces its row sample by sample ([Reproducing CatchBench](#reproducing-catchbench)). [agentpile](https://github.com/yzhao062/agentpile) is the converter that reads public agent-trace datasets into the one record the methods consume ([Real Agent Traces](#real-agent-traces)).
 
 ## Quick Start
 
@@ -211,6 +311,7 @@ Configuration rules and the step-by-step and binary-search LLM judges run on the
 | Supervised | `run.lr_flat` | Logistic regression on run size (GRADE flat) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
 | Supervised | `run.lr_flatdep` | Logistic regression on size and dependency (GRADE flatdep) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
 | Supervised | `run.lr_full` | Logistic regression on all raw layers (GRADE full) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
+| Supervised | `run.lr_spline_flat` | Logistic regression on a cubic spline of run size (flexible size control) | [CatchBench (2026)](https://arxiv.org/abs/2608.22808) |
 | PyOD detector | `run.pyod_copod` | COPOD on run size | [COPOD (Li et al., 2020)](https://arxiv.org/abs/2009.09463) |
 | PyOD detector | `run.pyod_ecod` | ECOD on run size | [ECOD (Li et al., 2022)](https://arxiv.org/abs/2201.00382) |
 | PyOD detector | `run.pyod_hbos` | HBOS on run size | HBOS (Goldstein and Dengel, 2012) |
@@ -253,15 +354,16 @@ Configuration rules and the step-by-step and binary-search LLM judges run on the
 
 ```bash
 pip install auditable                  # core: the registry and the configuration rules, no third-party dependencies
+pip install "auditable[bench]"         # the leaderboards: PyOD, scikit-learn, NumPy, NetworkX, and agentpile with its Hugging Face loaders
 pip install "auditable[tabular-ad]"    # run, step, and online methods on trace features (NumPy, scikit-learn, PyOD, NetworkX)
 pip install "auditable[agentpile]"     # read public datasets, trace files and captured runs through agentpile
 ```
 
-The examples, tests and benchmark harnesses are in the repository, not the wheel; to run them, install from a clone with `pip install -e ".[dev,tabular-ad]"`.
+The examples, tests and benchmark harnesses are in the repository, not the wheel; to run them, install from a clone with `pip install -e ".[dev,tabular-ad,bench]"`.
 
 The LLM judges take any `prompt -> reply` function as their backend; `OpenAICompatibleBackend` reaches an OpenAI-compatible endpoint with the standard library alone.
 
-The graph methods (PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite) need torch, PyTorch Geometric, PyGOD, and a sampling backend built for the installed torch, which pip cannot resolve from the extra. Install torch and the matching `pyg-lib` wheel first:
+The graph methods (PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite), and so the graph rows of the leaderboards, need torch, PyTorch Geometric, PyGOD, and a sampling backend built for the installed torch, which pip cannot resolve from the extra; they need Python 3.10 or newer, since torch 2.12.1 and `pyg-lib` publish no wheels for 3.9. Install torch and the matching `pyg-lib` wheel first (the same steps on Linux, macOS on Apple silicon, and Windows):
 
 ```bash
 pip install "torch==2.12.1" --index-url https://download.pytorch.org/whl/cpu
@@ -270,7 +372,7 @@ pip install "auditable[graph-ad]"
 python -c "import torch_geometric.typing as t; assert t.WITH_PYG_LIB"
 ```
 
-Where torch is already installed, as on Colab, keep it: skip the first line, and in the `pyg-lib` URL replace `2.12.1+cpu` with the installed version, which `python -c "import torch; print(torch.__version__)"` prints (for example `2.11.0+cu130`); the pinned CPU build would replace Colab's torch and break its preinstalled torchvision. The graph-ad install was checked on Colab CPU and T4 runtimes with Colab's torch, and on Linux aarch64 (DGX Spark) with the pinned one. The PyGOD methods run on the CPU (`gpu=-1`) even when a GPU is present.
+Where torch is already installed, as on Colab, keep it: skip the first line, and in the `pyg-lib` URL replace `2.12.1+cpu` with the installed version, which `python -c "import torch; print(torch.__version__)"` prints; the pinned CPU build would replace Colab's torch and break its preinstalled torchvision. PyG does not publish wheels for every patch release, so when pip finds no `pyg_lib` at that URL, use the `.0` release with the same CUDA tag: Colab's torch `2.14.1+cu130` takes `https://data.pyg.org/whl/torch-2.14.0+cu130.html`. The graph-ad install was checked for 0.4.0 on a Colab T4 runtime with its torch 2.14.1+cu130, and for 0.3.0 on Colab CPU and T4 runtimes with Colab's torch and on Linux aarch64 (DGX Spark) with the pinned one; a weekly workflow (`bench-platforms.yml`) repeats the pinned steps from a clean install on Linux, macOS, and Windows and checks three boards against their golden files. The PyGOD methods run on the CPU (`gpu=-1`) even when a GPU is present.
 
 ## Traces and Inputs
 
@@ -280,7 +382,7 @@ Methods read traces through `auditable.featurize.normalize_trace`, which accepts
 
 ## Reproducing CatchBench
 
-[`benchmarks/catchbench_parity`](https://github.com/yzhao062/auditable/tree/main/benchmarks/catchbench_parity) checks the library against CatchBench 0.1.2 and GRADE at commit `3839a57`, the versions the CatchBench board pins. It is a development harness and is not part of the wheel.
+`python -m auditable.bench <task> --golden` reruns a CatchBench board with the methods of this library and exits 1 if a row differs from the board ([Leaderboard](#leaderboard)). The parity harness [`benchmarks/catchbench_parity`](https://github.com/yzhao062/auditable/tree/main/benchmarks/catchbench_parity) remains for development and is not part of the wheel. It compares every migrated method with its CatchBench or GRADE original sample by sample, at CatchBench 0.1.2 and GRADE commit `3839a57`, the last versions whose board computes those entries with its own code.
 
 ```bash
 export GRADE_DIR=/path/to/grade   # checked out at 3839a57
@@ -291,11 +393,9 @@ python benchmarks/catchbench_parity/run_parity.py board --perturb --out perturbe
 python benchmarks/catchbench_parity/run_parity.py consumption board.txt perturbed.txt
 ```
 
-The checks compare every migrated method with its CatchBench or GRADE original on the real corpora, sample by sample and bit for bit: per-run scores, per-step scores, out-of-fold predictions, configuration findings, and the features underneath them. The board run replaces CatchBench's method factories with library adapters, so the main tables and every breakdown and robustness table are computed by the library, while the original scoring functions of migrated methods are replaced by functions that raise. Its output must match the board computed before migration byte for byte. The consumption run perturbs every library output and confirms that each migrated row changes and no other row does.
-
 ## Adding a Method
 
-Subclass the family base class in `auditable.base`, give it a `MethodCard`, and register it with `@register`. The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions: input forms, label alignment for supervised step methods, output order, optional dependencies, the tests a method needs, and how to add a parity check when the method reproduces a CatchBench row.
+Subclass the family base class in `auditable.base`, give it a `MethodCard`, and register it with `@register`, which makes it reachable by id (`get_method`, `--methods`) and adds it to the default rows of the detect and pre boards. To try a method first, pass it to `leaderboard(task, extra=[...])` without registering ([Put Your Method on the Board](#put-your-method-on-the-board)). The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions: input forms, label alignment for supervised step methods, output order, optional dependencies, the tests a method needs, and how to add a parity check when the method reproduces a CatchBench row.
 
 ## Records and Graph Analysis
 
@@ -309,14 +409,26 @@ Records are content-addressed and hash-chained, and that property is limited. A
 
 ## Related Projects
 
-- [CatchBench](https://github.com/yzhao062/catchbench): the benchmark for these methods (arXiv:2608.22808).
-- [agentpile](https://github.com/yzhao062/agentpile): loads public agent-trace datasets, trace files and captured runs into one record, which `from_agentpile` reads.
+- [CatchBench](https://github.com/yzhao062/catchbench): the paper and benchmark whose board the leaderboards reproduce (arXiv:2608.22808).
+- [agentpile](https://github.com/yzhao062/agentpile): the converter that loads public agent-trace datasets, trace files and captured runs into one record, which `from_agentpile` reads.
 - [GRADE](https://arxiv.org/abs/2606.22741): the typed two-layer graph of agent execution and dependency that the structural features read.
 - [awesome-auditable-ai](https://github.com/yzhao062/awesome-auditable-ai): a reading list on AI agent reliability and auditing.
 
 ## Citation
 
-If you use `auditable`, please cite the work behind the methods you use (each `MethodCard.cite`) and, for the structural features and the benchmark, GRADE and CatchBench:
+If you use `auditable`, please cite Auditable Agents:
+
+```bibtex
+@misc{nian2026auditable,
+  title         = {Auditable Agents},
+  author        = {Nian, Yi and Yuan, Aojie and Zhang, Haiyue and Li, Jiate and Li, Li and Hu, Xiyang and Wei, Hua and Xiao, Xiongye and Xiao, Chaowei and Zhao, Yue},
+  year          = {2026},
+  eprint        = {2604.05485},
+  archivePrefix = {arXiv}
+}
+```
+
+If you use the structural features or the benchmark, please also cite GRADE and CatchBench, and for each method, the work behind it (its `MethodCard.cite`):
 
 ```bibtex
 @article{zhao2026grade,

{auditable-0.3.0 → auditable-0.4.0}/README.md

@@ -9,20 +9,114 @@
 
 # auditable
 
-**A library of auditing methods for AI agents: configuration audits before a run, online monitors while it runs, and failure detection and fault localization after it, with one interface per family.**
+**A library of auditing methods for AI agents, with a leaderboard for each task: configuration audits before a run, online monitors while it runs, and failure detection and fault localization after it.**
 
 [![PyPI](https://img.shields.io/pypi/v/auditable.svg)](https://pypi.org/project/auditable/)
 [![Python](https://img.shields.io/pypi/pyversions/auditable.svg)](https://pypi.org/project/auditable/)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/yzhao062/auditable/blob/main/LICENSE)
 [![CI](https://github.com/yzhao062/auditable/actions/workflows/ci.yml/badge.svg)](https://github.com/yzhao062/auditable/actions/workflows/ci.yml)
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yzhao062/auditable/blob/main/examples/leaderboard.ipynb)
 
-[Quick Start](#quick-start) · [Real Agent Traces](#real-agent-traces) · [Methods](#methods) · [Install](#install) · [Reproducing CatchBench](#reproducing-catchbench) · [Adding a Method](#adding-a-method) · [Citation](#citation)
+[Leaderboard](#leaderboard) · [Put Your Method on the Board](#put-your-method-on-the-board) · [Tasks](#tasks) · [Quick Start](#quick-start) · [Real Agent Traces](#real-agent-traces) · [Methods](#methods) · [Install](#install) · [Adding a Method](#adding-a-method) · [Citation](#citation)
 
 </div>
 
+`auditable` is a library of auditing methods for AI agents with a leaderboard for each task: one command reruns the [CatchBench](https://github.com/yzhao062/catchbench) board on your machine, and a method you write gets a rank on the same population and protocol.
+
+## Leaderboard
+
+Did the run fail? The board below ranks 17 methods by ROC-AUC on 376 SWE-Gym OpenHands runs, 188 of which failed. One command reruns it on the population and protocol of the CatchBench board:
+
+```bash
+pip install "auditable[bench]"
+python -m auditable.bench detect/swegym
+```
+
+| Rank | Method | Algorithm | ROC-AUC |
+|---|---|---|---|
+| 1 | `run.pygod_gaan` | GAAN adversarial detector (PyGOD) | 0.850 |
+| 2 | `run.lr_spline_flat` | Logistic regression on a cubic spline of run size, a control | 0.832 |
+| 3 | `run.gsafeguard_lite` | G-Safeguard-Lite (supervised graph classifier) | 0.828 |
+| 4 | `run.lr_full` | Logistic regression on all raw layers (GRADE full) | 0.819 |
+| 5 | `run.lr_flatdep` | Logistic regression on size and dependency (GRADE flatdep) | 0.804 |
+| 6 | `run.guardian_lite` | GUARDIAN-Lite (attribute-reconstruction autoencoder) | 0.767 |
+| 7 | `run.pyod_ecod` | ECOD on run size | 0.765 |
+| 8 | `run.pygod_conad` | CONAD contrastive graph autoencoder (PyGOD) | 0.750 |
+| 9 | `run.lr_flat` | Logistic regression on run size (GRADE flat), the size baseline | 0.663 |
+| 10 | `run.pyod_copod` | COPOD on run size | 0.625 |
+| 11 | `run.pygod_anomalydae` | AnomalyDAE dual autoencoder (PyGOD) | 0.592 |
+| 12 | `run.pyod_lof` | Local outlier factor on run size | 0.584 |
+| 13 | `run.pyod_iforest` | Isolation Forest on run size | 0.571 |
+| 14 | `run.pygod_dominant` | DOMINANT graph autoencoder (PyGOD) | 0.547 |
+| 15 | `run.random` | Random run scores, a control | 0.483 |
+| 16 | `run.pyod_knn` | KNN distance on run size | 0.446 |
+| 17 | `run.pyod_hbos` | HBOS on run size | 0.319 |
+
+PyGOD's GAAN, a literature baseline, leads this board at 0.850. Second, at 0.832, is the flexible size control `run.lr_spline_flat`: a cubic spline of the same four run-size counts the linear size baseline `run.lr_flat` reads (0.663). Run size alone, fitted without a linear restriction, ranks above G-Safeguard-Lite and above both GRADE dependency regressions (0.819 and 0.804), so this board does not show that dependency structure predicts failure beyond size, the reading GRADE retired on 2026-09-08. Read a new row against both size controls; on tau-bench the order changes, and `run.lr_flatdep` (0.665) ranks above the spline control (0.631).
+
+The 16 CatchBench rows equal the CatchBench board, which was computed on Linux; the spline control is not a board row, and its reference value is the `size (spline)` arm of CatchBench's committed size audit. On macOS, `run.pyod_lof` prints 0.582 instead of 0.584, and `--golden` reports that as a platform value and not as a mismatch. The first run downloads the data through agentpile; later runs load SWE-Gym from the cache in about 3 seconds, and the full board takes about 70 seconds on a laptop CPU, of which PyGOD GAAN is the slowest row at about 37 seconds. The graph rows need `auditable[graph-ad]` and the PyG sampling backend ([Install](#install)); without them those rows are listed as skipped with the extra named, and the rest of the board is still scored.
+
+```bash
+python -m auditable.bench                                                    # list the tasks
+python -m auditable.bench detect/swegym --methods run.lr_flat run.pyod_ecod  # score chosen rows only
+python -m auditable.bench detect/swegym --golden                             # exit 1 if a row differs from the CatchBench board
+python -m auditable.bench detect/swegym --markdown                           # a Markdown table; --json prints JSON
+```
+
+## Put Your Method on the Board
+
+A method is a class with a `MethodCard`. Pass it as `extra=` and `leaderboard` scores it beside the default rows, under the protocol of its kind: a supervised detector gets the mean ROC-AUC over five seeds of stratified 5-fold cross-validation with a fresh estimator per fold, and an unsupervised one gets a single `fit_score` over the whole population. The snippet below adds a random forest on run size, a deliberately plain method:
+
+```python
+from sklearn.ensemble import RandomForestClassifier
+
+from auditable.base import BaseRunDetector, MethodCard
+from auditable.bench import leaderboard
+from auditable.featurize import grade_layer_matrix
+
+
+class ForestOnSize(BaseRunDetector):
+    card = MethodCard(method_id="run.forest_flat", name="Random forest on run size", family="run",
+                      fit_scope="inductive", supervised=True, output_kind="score", fidelity="original")
+
+    def fit(self, runs, y=None):
+        self.model_ = RandomForestClassifier(200, random_state=0)
+        self.model_.fit(grade_layer_matrix(list(runs), "flat"), y)
+        self.decision_scores_ = self.decision_function(runs)
+        return self
+
+    def decision_function(self, runs):
+        return self.model_.predict_proba(grade_layer_matrix(list(runs), "flat"))[:, 1]
+
+
+board = leaderboard("detect/swegym", extra=[ForestOnSize])
+print(board.rank_of("run.forest_flat"), board["run.forest_flat"].scores)
+```
+
+The last line prints the rank of the new row among all rows and its scores. `print(board)` shows the full table, with the CatchBench value in brackets beside each row that has one, and `board.to_markdown()`, `board.to_dict()` and `board.compare_golden()` export it. `methods=[...]` scores only the rows you name, which is quick while you iterate (`leaderboard("detect/swegym", methods=["run.lr_flat", ForestOnSize])`), and `Entry(method, kwargs={...}, label="...", key="...")` adds a row for a variant with constructor arguments (give each variant its own key, because a row that keeps the method id is compared with that method's CatchBench value). A method that raises is listed as an error and the rest of the board is still scored. The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions for the four families.
+
+## Tasks
+
+Six tasks ship with 0.4.0. Each fixes a population and a protocol, so every method on a board is scored on the same data in the same way. `detect` scores the run detectors, `localize` the step locators, `live` the online monitors (run detectors also take part, scored on prefixes of the run), and `pre` the configuration auditors.
+
+| Task | Question | Population | Metric |
+|---|---|---|---|
+| `detect/swegym` | Did the run fail? | SWE-Gym OpenHands trajectories, one scaffold: 376 runs, 188 failed | ROC-AUC |
+| `detect/tau` | Did the run fail? | tau-bench: 660 runs, 363 failed | ROC-AUC |
+| `localize/whoandwhen` | Which step was the decisive mistake? | Who&When: 126 failed multi-agent runs with a human mistake step | top-1, top-3, MRR (mean reciprocal rank) |
+| `live/swegym` | Can the failure be caught early? | SWE-Gym runs, read as prefixes of 25, 50, 75 and 100 percent of their steps | mean prefix ROC-AUC |
+| `live/tau` | Can the failure be caught early? | tau-bench runs, read as prefixes of 25, 50, 75 and 100 percent of their steps | mean prefix ROC-AUC |
+| `pre/over_privilege` | Is the agent configuration over-privileged before it runs? | 1,187 configurations from six sources | F1, precision, recall |
+
+Every task id works with the commands above: `python -m auditable.bench <task>` reruns the board, and `leaderboard("<task>", extra=[...])` places a method on it. The LLM judges on `localize/whoandwhen` and `pre/over_privilege` are replayed from committed caches, so those boards need no API call.
+
+The CatchBench board shows how the other tasks read. On Who&When localization, the best LLM judges (all-at-once) reach top-1 of about 0.42 to 0.45 against 0.119 for random, and the structural blast rule ties the position prior at 0.159 top-1. The tau-bench detection board is led by `run.lr_flatdep` and `run.lr_full`, tied at 0.665, against 0.619 for the size baseline. For over-privilege, the best rule scanner `config.owasp_asi_combined` reaches F1 0.654, flagging every configuration reaches 0.601, and an LLM judge reaches 0.695.
+
+## What Is in the Library
+
 `auditable` collects methods that audit LLM agents at three points: an agent configuration before it runs (PRE), a run while it streams (LIVE), and a finished run (POST). Methods are grouped into four families, and each family has one interface, in the way PyOD gives outlier detectors one interface. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN, G-Safeguard, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench.
 
-[CatchBench](https://github.com/yzhao062/catchbench) is the benchmark for these methods: the tasks, corpora, evaluation protocols, and metrics live there. Every method here that appears on the CatchBench board reproduces its row sample by sample, and the full board reruns byte-identical with those rows computed by this library ([Reproducing CatchBench](#reproducing-catchbench)).
+[CatchBench](https://github.com/yzhao062/catchbench) is the paper and benchmark whose board the leaderboards reproduce: `auditable.bench` carries its populations and evaluation protocols, and every method here that appears on its board reproduces its row sample by sample ([Reproducing CatchBench](#reproducing-catchbench)). [agentpile](https://github.com/yzhao062/agentpile) is the converter that reads public agent-trace datasets into the one record the methods consume ([Real Agent Traces](#real-agent-traces)).
 
 ## Quick Start
 
@@ -147,6 +241,7 @@ Configuration rules and the step-by-step and binary-search LLM judges run on the
 | Supervised | `run.lr_flat` | Logistic regression on run size (GRADE flat) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
 | Supervised | `run.lr_flatdep` | Logistic regression on size and dependency (GRADE flatdep) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
 | Supervised | `run.lr_full` | Logistic regression on all raw layers (GRADE full) | [GRADE (Zhao, 2026)](https://arxiv.org/abs/2606.22741) |
+| Supervised | `run.lr_spline_flat` | Logistic regression on a cubic spline of run size (flexible size control) | [CatchBench (2026)](https://arxiv.org/abs/2608.22808) |
 | PyOD detector | `run.pyod_copod` | COPOD on run size | [COPOD (Li et al., 2020)](https://arxiv.org/abs/2009.09463) |
 | PyOD detector | `run.pyod_ecod` | ECOD on run size | [ECOD (Li et al., 2022)](https://arxiv.org/abs/2201.00382) |
 | PyOD detector | `run.pyod_hbos` | HBOS on run size | HBOS (Goldstein and Dengel, 2012) |
@@ -189,15 +284,16 @@ Configuration rules and the step-by-step and binary-search LLM judges run on the
 
 ```bash
 pip install auditable                  # core: the registry and the configuration rules, no third-party dependencies
+pip install "auditable[bench]"         # the leaderboards: PyOD, scikit-learn, NumPy, NetworkX, and agentpile with its Hugging Face loaders
 pip install "auditable[tabular-ad]"    # run, step, and online methods on trace features (NumPy, scikit-learn, PyOD, NetworkX)
 pip install "auditable[agentpile]"     # read public datasets, trace files and captured runs through agentpile
 ```
 
-The examples, tests and benchmark harnesses are in the repository, not the wheel; to run them, install from a clone with `pip install -e ".[dev,tabular-ad]"`.
+The examples, tests and benchmark harnesses are in the repository, not the wheel; to run them, install from a clone with `pip install -e ".[dev,tabular-ad,bench]"`.
 
 The LLM judges take any `prompt -> reply` function as their backend; `OpenAICompatibleBackend` reaches an OpenAI-compatible endpoint with the standard library alone.
 
-The graph methods (PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite) need torch, PyTorch Geometric, PyGOD, and a sampling backend built for the installed torch, which pip cannot resolve from the extra. Install torch and the matching `pyg-lib` wheel first:
+The graph methods (PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite), and so the graph rows of the leaderboards, need torch, PyTorch Geometric, PyGOD, and a sampling backend built for the installed torch, which pip cannot resolve from the extra; they need Python 3.10 or newer, since torch 2.12.1 and `pyg-lib` publish no wheels for 3.9. Install torch and the matching `pyg-lib` wheel first (the same steps on Linux, macOS on Apple silicon, and Windows):
 
 ```bash
 pip install "torch==2.12.1" --index-url https://download.pytorch.org/whl/cpu
@@ -206,7 +302,7 @@ pip install "auditable[graph-ad]"
 python -c "import torch_geometric.typing as t; assert t.WITH_PYG_LIB"
 ```
 
-Where torch is already installed, as on Colab, keep it: skip the first line, and in the `pyg-lib` URL replace `2.12.1+cpu` with the installed version, which `python -c "import torch; print(torch.__version__)"` prints (for example `2.11.0+cu130`); the pinned CPU build would replace Colab's torch and break its preinstalled torchvision. The graph-ad install was checked on Colab CPU and T4 runtimes with Colab's torch, and on Linux aarch64 (DGX Spark) with the pinned one. The PyGOD methods run on the CPU (`gpu=-1`) even when a GPU is present.
+Where torch is already installed, as on Colab, keep it: skip the first line, and in the `pyg-lib` URL replace `2.12.1+cpu` with the installed version, which `python -c "import torch; print(torch.__version__)"` prints; the pinned CPU build would replace Colab's torch and break its preinstalled torchvision. PyG does not publish wheels for every patch release, so when pip finds no `pyg_lib` at that URL, use the `.0` release with the same CUDA tag: Colab's torch `2.14.1+cu130` takes `https://data.pyg.org/whl/torch-2.14.0+cu130.html`. The graph-ad install was checked for 0.4.0 on a Colab T4 runtime with its torch 2.14.1+cu130, and for 0.3.0 on Colab CPU and T4 runtimes with Colab's torch and on Linux aarch64 (DGX Spark) with the pinned one; a weekly workflow (`bench-platforms.yml`) repeats the pinned steps from a clean install on Linux, macOS, and Windows and checks three boards against their golden files. The PyGOD methods run on the CPU (`gpu=-1`) even when a GPU is present.
 
 ## Traces and Inputs
 
@@ -216,7 +312,7 @@ Methods read traces through `auditable.featurize.normalize_trace`, which accepts
 
 ## Reproducing CatchBench
 
-[`benchmarks/catchbench_parity`](https://github.com/yzhao062/auditable/tree/main/benchmarks/catchbench_parity) checks the library against CatchBench 0.1.2 and GRADE at commit `3839a57`, the versions the CatchBench board pins. It is a development harness and is not part of the wheel.
+`python -m auditable.bench <task> --golden` reruns a CatchBench board with the methods of this library and exits 1 if a row differs from the board ([Leaderboard](#leaderboard)). The parity harness [`benchmarks/catchbench_parity`](https://github.com/yzhao062/auditable/tree/main/benchmarks/catchbench_parity) remains for development and is not part of the wheel. It compares every migrated method with its CatchBench or GRADE original sample by sample, at CatchBench 0.1.2 and GRADE commit `3839a57`, the last versions whose board computes those entries with its own code.
 
 ```bash
 export GRADE_DIR=/path/to/grade   # checked out at 3839a57
@@ -227,11 +323,9 @@ python benchmarks/catchbench_parity/run_parity.py board --perturb --out perturbe
 python benchmarks/catchbench_parity/run_parity.py consumption board.txt perturbed.txt
 ```
 
-The checks compare every migrated method with its CatchBench or GRADE original on the real corpora, sample by sample and bit for bit: per-run scores, per-step scores, out-of-fold predictions, configuration findings, and the features underneath them. The board run replaces CatchBench's method factories with library adapters, so the main tables and every breakdown and robustness table are computed by the library, while the original scoring functions of migrated methods are replaced by functions that raise. Its output must match the board computed before migration byte for byte. The consumption run perturbs every library output and confirms that each migrated row changes and no other row does.
-
 ## Adding a Method
 
-Subclass the family base class in `auditable.base`, give it a `MethodCard`, and register it with `@register`. The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions: input forms, label alignment for supervised step methods, output order, optional dependencies, the tests a method needs, and how to add a parity check when the method reproduces a CatchBench row.
+Subclass the family base class in `auditable.base`, give it a `MethodCard`, and register it with `@register`, which makes it reachable by id (`get_method`, `--methods`) and adds it to the default rows of the detect and pre boards. To try a method first, pass it to `leaderboard(task, extra=[...])` without registering ([Put Your Method on the Board](#put-your-method-on-the-board)). The [method contract](https://github.com/yzhao062/auditable/blob/main/docs/methods-contract.md) lists the conventions: input forms, label alignment for supervised step methods, output order, optional dependencies, the tests a method needs, and how to add a parity check when the method reproduces a CatchBench row.
 
 ## Records and Graph Analysis
 
@@ -245,14 +339,26 @@ Records are content-addressed and hash-chained, and that property is limited. A
 
 ## Related Projects
 
-- [CatchBench](https://github.com/yzhao062/catchbench): the benchmark for these methods (arXiv:2608.22808).
-- [agentpile](https://github.com/yzhao062/agentpile): loads public agent-trace datasets, trace files and captured runs into one record, which `from_agentpile` reads.
+- [CatchBench](https://github.com/yzhao062/catchbench): the paper and benchmark whose board the leaderboards reproduce (arXiv:2608.22808).
+- [agentpile](https://github.com/yzhao062/agentpile): the converter that loads public agent-trace datasets, trace files and captured runs into one record, which `from_agentpile` reads.
 - [GRADE](https://arxiv.org/abs/2606.22741): the typed two-layer graph of agent execution and dependency that the structural features read.
 - [awesome-auditable-ai](https://github.com/yzhao062/awesome-auditable-ai): a reading list on AI agent reliability and auditing.
 
 ## Citation
 
-If you use `auditable`, please cite the work behind the methods you use (each `MethodCard.cite`) and, for the structural features and the benchmark, GRADE and CatchBench:
+If you use `auditable`, please cite Auditable Agents:
+
+```bibtex
+@misc{nian2026auditable,
+  title         = {Auditable Agents},
+  author        = {Nian, Yi and Yuan, Aojie and Zhang, Haiyue and Li, Jiate and Li, Li and Hu, Xiyang and Wei, Hua and Xiao, Xiongye and Xiao, Chaowei and Zhao, Yue},
+  year          = {2026},
+  eprint        = {2604.05485},
+  archivePrefix = {arXiv}
+}
+```
+
+If you use the structural features or the benchmark, please also cite GRADE and CatchBench, and for each method, the work behind it (its `MethodCard.cite`):
 
 ```bibtex
 @article{zhao2026grade,

{auditable-0.3.0 → auditable-0.4.0}/examples/README.md

@@ -1,5 +1,35 @@
 # auditable Examples
 
+## Leaderboards: Rerun the Board and Add Your Method
+
+`auditable.bench` scores auditing methods on frozen populations with frozen protocols, ported from [CatchBench](https://github.com/yzhao062/catchbench), the paper and benchmark whose board these leaderboards reproduce. One call reruns a board on your machine, and a method you write gets a rank on the same population and protocol as every other row. The data come through [agentpile](https://github.com/yzhao062/agentpile), which reads public agent-trace datasets into one record.
+
+```
+pip install "auditable[bench]"
+python examples/example_leaderboard.py
+python examples/example_add_your_method.py
+```
+
+`example_leaderboard.py` prints the `detect/swegym` board: did the run fail, on 376 SWE-Gym OpenHands runs (188 failed), scored by ROC-AUC. It then restricts the board to three tabular rows (`run.lr_flat`, `run.lr_flatdep`, `run.pyod_ecod`) with `methods=[...]` and prints the full board as a Markdown table. The first call loads the runs through agentpile, and the full 17-row board takes about 70 seconds on a laptop CPU. A row whose pip extra is missing is listed as skipped with the extra named, and a row that raises is listed as an error, while the rest of the board is still scored. The graph rows (PyGOD, GUARDIAN-Lite, G-Safeguard-Lite) need `auditable[graph-ad]` plus the PyG sampling backend described in the Install section of the main README.
+
+`example_add_your_method.py` defines one new run detector in about 20 lines: a logistic regression on the number of steps and the longest dependency span, written as a `BaseRunDetector` subclass with a full `MethodCard`. The script passes the class to `leaderboard(..., extra=[StepsAndSpan])` next to the size baseline `run.lr_flat`, `run.lr_flatdep`, `run.pyod_ecod`, and the random control, then prints the board and the rank of the new row. Copy the class, change `features` and the estimator, and the board scores your idea under the same cross-validation as every supervised row.
+
+The same boards run from a shell:
+
+```
+python -m auditable.bench                                   # list the tasks
+python -m auditable.bench detect/swegym --methods run.lr_flat run.pyod_ecod
+python -m auditable.bench detect/swegym --golden            # exit 1 if a row differs from the CatchBench board
+```
+
+`--golden` compares each row with the CatchBench board and prints a known platform difference without failing: on macOS `run.pyod_lof` scores 0.582 where the Linux board prints 0.584. Pass another task id to score another task, and name only methods of the task's family (the live tasks take `run` and `online` methods), because a method of another family is listed as skipped.
+
+Read a board with its baselines in view. The top row of `detect/swegym` is PyGOD's GAAN detector, a literature baseline, and `run.lr_flat` is the size baseline that belongs beside every dependency method. A linear size baseline cannot separate dependency shape from a nonlinear function of size, and GRADE retired the reading that the dependency layers predict failure beyond run size on 2026-09-08. A rank is one measurement on one population and protocol.
+
+`leaderboard.ipynb` runs the same two steps in a notebook, with the board, the golden check, and a template detector, and it opens in Colab.
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yzhao062/auditable/blob/main/examples/leaderboard.ipynb)
+
 ## The Method Library
 
 `example_methods.py` runs one method from each family of `auditable.methods` (a

auditable-0.4.0/examples/example_add_your_method.py

@@ -0,0 +1,68 @@
+"""Add your own method to a board: score a new run detector on SWE-Gym next to the paper's rows.
+
+Run from a checkout after installing the bench extra:
+
+    pip install "auditable[bench]"
+    python examples/example_add_your_method.py
+
+The detector is a logistic regression on two hand-made features of the typed run graph: the
+number of steps and the longest dependency span (how far back any step reaches). The board
+scores it as it scores every supervised method, by mean ROC-AUC over five seeds of
+stratified 5-fold cross-validation with a fresh estimator per fold.
+
+The script prints the small board (the size baseline ``run.lr_flat``, the dependency
+regression ``run.lr_flatdep``, ``run.pyod_ecod``, the random control, and the new row) and
+the rank and score of the new row beside the size baseline. Read any gain over
+``run.lr_flat`` with care. The longest span grows with run length, and GRADE retired on
+2026-09-08 the reading that dependency structure predicts failure beyond run size, so a
+claim for a new method needs a comparison with a size-only model fitted as flexibly as
+yours. A rank describes a method on one population and protocol; it does not explain why
+the method works.
+"""
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import StandardScaler
+
+from auditable.base import BaseRunDetector, MethodCard
+from auditable.bench import leaderboard
+from auditable.featurize import normalize_trace, run_max_span
+
+
+class StepsAndSpan(BaseRunDetector):
+    """Run length and the longest dependency span, weighed jointly by a linear model."""
+
+    card = MethodCard(
+        method_id="run.lr_steps_span", name="Logistic regression on step count and longest span",
+        family="run", fit_scope="inductive", supervised=True, output_kind="score", fidelity="original",
+        requires=frozenset({"deps"}), extras=("tabular-ad",),
+        summary="Balanced logistic regression on the log step count and the longest dependency span.")
+
+    def features(self, runs):
+        # log1p tames the long tail of run length; the board calls this once and indexes the matrix per fold.
+        return np.array([[np.log1p(len(normalize_trace(g))), run_max_span(g)] for g in runs])
+
+    def fit(self, X, y=None):
+        X = X if getattr(X, "ndim", 0) == 2 else self.features(X)
+        self.pipeline_ = make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced")).fit(X, y)
+        self.decision_scores_ = self.pipeline_.decision_function(X)
+        return self
+
+    def decision_function(self, X):
+        return self.pipeline_.decision_function(X if getattr(X, "ndim", 0) == 2 else self.features(X))
+
+
+def main():
+    # The size baseline and the paper's dependency and unsupervised rows stay on the board so the
+    # new row is read against them; extra= adds the new row to those named in methods=.
+    board = leaderboard("detect/swegym", methods=["run.lr_flat", "run.lr_flatdep", "run.pyod_ecod", "run.random"],
+                        extra=[StepsAndSpan])
+    print(board)
+    key = StepsAndSpan.card.method_id
+    print(f"\n{key} ranks {board.rank_of(key)} of {len(board.ranked())}, "
+          f"ROC-AUC {board[key].scores['roc_auc']:.3f} against {board['run.lr_flat'].scores['roc_auc']:.3f} "
+          f"for the size baseline")
+
+
+if __name__ == "__main__":
+    main()