evaluma 0.2.0__tar.gz → 0.3.0__tar.gz

{evaluma-0.2.0 → evaluma-0.3.0}/.github/workflows/ci.yml

@@ -34,21 +34,9 @@ jobs:
       - run: ruff format --check .
       - run: ty check
 
-  docs:
-    runs-on: ubuntu-latest
-    env:
-      IPYTHONDIR: /tmp/ipython
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-      - run: pip install -e ".[docs]"
-      - run: sphinx-build -W -b html docs docs/_build/html
-
   publish:
     if: startsWith(github.ref, 'refs/tags/v')
-    needs: [test, lint, docs]
+    needs: [test, lint]
     runs-on: ubuntu-latest
     environment: pypi
     permissions:

{evaluma-0.2.0 → evaluma-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evaluma
-Version: 0.2.0
+Version: 0.3.0
 Summary: ML benchmark ranking tools — IQM, Bayesian pairwise, and Dolan-Moré performance profiles
 Author-email: Nils Lehmann <nils.lehmann24@gmail.com>
 License: Apache 2.0
@@ -58,11 +58,16 @@ Dynamic: license-file
 
 # evaluma
 
-A small Python package for comparing machine learning models across benchmark suites. Given a CSV of per-model, per-dataset scores, evaluma can compute three complementary views of the results:
+A small Python package for comparing machine learning models across benchmark suites. Given a CSV of per-model, per-dataset scores, evaluma computes eight complementary views of the results:
 
+- **Aggregate ranking** — point-estimate ranking via trimmed mean, mean, or median
 - **IQM ranking** — interquartile mean with bootstrapped confidence intervals, following [Agarwal et al. (2021)](https://arxiv.org/abs/2108.13264)
+- **ELO ranking** — MLE ELO ratings from pairwise head-to-head battles with bootstrap CIs and a win-rate matrix, following [Erickson et al. (2025)](https://arxiv.org/abs/2506.16791)
+- **Improvability ranking** — mean percent error reduction each model needs to match the per-dataset best, in raw error space, following [TabArena](https://arxiv.org/abs/2506.16791) / [BeyondArena](https://arxiv.org/abs/2606.30410)
 - **Bayesian pairwise comparison** — posterior probabilities that model A beats model B (or is practically equivalent), via [baycomp](https://github.com/janezd/baycomp)
-- **Dolan-Moré performance profiles** — cumulative distribution of performance ratios and area-under-profile scores
+- **Frequentist comparison** — Friedman + Nemenyi (all-pairs) or Wilcoxon + Holm (reference model), following [Demšar (2006)](https://jmlr.org/papers/v7/demsar06a.html)
+- **Dolan-Moré performance profiles** — cumulative distribution of performance ratios and area-under-profile scores, following [Dolan & Moré (2002)](https://doi.org/10.1007/s101070100263)
+- **Rank sensitivity** — Kendall τ-b with bootstrap CI measuring whether rankings hold across two experimental conditions, following [Kendall (1945)](https://doi.org/10.1093/biomet/33.3.239)
 
 
 ## Documentation
@@ -99,36 +104,62 @@ bench = evaluma.load_df(
     score="score",
 )
 
-# IQM ranking with 95% bootstrap CI
+# Point-estimate aggregate ranking (trimmed mean)
+agg = bench.aggregate_ranking()
+print(agg.table)
+
+# IQM ranking with 95% bootstrap CI (requires seed column)
 iqm = bench.iqm_ranking()
 print(iqm.table)
 fig = iqm.plot()
 fig.savefig("iqm.png")
 
+# ELO ranking with win-rate matrix
+elo = bench.elo_ranking()
+print(elo.table)
+fig = elo.plot_winrate()
+
+# Improvability ranking (mean % error reduction needed to match the best)
+imp = bench.improvability_ranking()
+print(imp.table)
+fig = imp.plot()
+
 # Bayesian pairwise probabilities
 bayes = bench.bayesian_comparison()
 print(bayes.table)
 
+# Frequentist comparison (Friedman + Nemenyi)
+freq = bench.frequentist_comparison()
+print(freq.table)
+
 # Dolan-Moré performance profiles
 profiles = bench.performance_profiles()
 fig = profiles.plot()
+
+# Rank sensitivity across two conditions
+bench_b = evaluma.load_df("results_b.csv", model="model", dataset="dataset",
+                          metric="metric", score="score")
+sens = bench.rank_sensitivity(bench_b, cond_a="condA", cond_b="condB")
+print(f"Kendall τ = {sens.tau:.3f}, 95% CI = {sens.tau_ci}")
 ```
 
 ### CLI
 
 ```bash
-# Run all three analyses and write six output files
+# Run aggregate, Bayesian, frequentist, and performance-profile analyses
 evaluma report results.csv \
     --model model --dataset dataset --metric metric --score score \
     --output results/
 
 # Individual subcommands
-evaluma rank    results.csv --model model --dataset dataset --metric metric --score score --output results/
-evaluma compare results.csv --model model --dataset dataset --metric metric --score score --output results/
-evaluma profiles results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma aggregate   results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma rank        results.csv --model model --dataset dataset --metric metric --score score --seed seed --output results/
+evaluma compare     results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma frequentist results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma profiles    results.csv --model model --dataset dataset --metric metric --score score --output results/
 ```
 
-Each subcommand writes a `.csv` table and a `.png` figure to `--output`.
+Each subcommand writes a `.csv` table and a `.png` figure to `--output`. ELO ranking and rank sensitivity are available via the Python API only.
 
 ### Column mapping
 
@@ -238,6 +269,15 @@ also cite the works of the underlying methods and frameworks used:
   year      = {2021},
 }
 
+@inproceedings{erickson2025tabarena,
+  title     = {{TabArena}: A Living Benchmark for Machine Learning on Tabular Data},
+  author    = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej
+               and Holzm{\"u}ller, David and Mutalik Desai, Prabhant
+               and Salinas, David and Hutter, Frank},
+  booktitle = {Advances in Neural Information Processing Systems},
+  year      = {2025},
+}
+
 @article{benavoli2017time,
   title   = {Time for a Change: a Tutorial for Comparing Multiple Classifiers
              Through Bayesian Analysis},
@@ -250,6 +290,15 @@ also cite the works of the underlying methods and frameworks used:
   year    = {2017},
 }
 
+@article{demsar2006statistical,
+  title   = {Statistical Comparisons of Classifiers over Multiple Data Sets},
+  author  = {Dem{\v{s}}ar, Janez},
+  journal = {Journal of Machine Learning Research},
+  volume  = {7},
+  pages   = {1--30},
+  year    = {2006},
+}
+
 @article{dolan2002benchmarking,
   title   = {Benchmarking Optimization Software with Performance Profiles},
   author  = {Dolan, Elizabeth D. and Mor{\'e}, Jorge J.},
@@ -258,4 +307,15 @@ also cite the works of the underlying methods and frameworks used:
   pages   = {201--213},
   year    = {2002},
 }
+
+@article{kendall1945treatment,
+  title   = {The Treatment of Ties in Ranking Problems},
+  author  = {Kendall, Maurice G.},
+  journal = {Biometrika},
+  volume  = {33},
+  number  = {3},
+  pages   = {239--251},
+  year    = {1945},
+  doi     = {10.1093/biomet/33.3.239},
+}
 ```

{evaluma-0.2.0 → evaluma-0.3.0}/README.md

@@ -13,11 +13,16 @@
 
 # evaluma
 
-A small Python package for comparing machine learning models across benchmark suites. Given a CSV of per-model, per-dataset scores, evaluma can compute three complementary views of the results:
+A small Python package for comparing machine learning models across benchmark suites. Given a CSV of per-model, per-dataset scores, evaluma computes eight complementary views of the results:
 
+- **Aggregate ranking** — point-estimate ranking via trimmed mean, mean, or median
 - **IQM ranking** — interquartile mean with bootstrapped confidence intervals, following [Agarwal et al. (2021)](https://arxiv.org/abs/2108.13264)
+- **ELO ranking** — MLE ELO ratings from pairwise head-to-head battles with bootstrap CIs and a win-rate matrix, following [Erickson et al. (2025)](https://arxiv.org/abs/2506.16791)
+- **Improvability ranking** — mean percent error reduction each model needs to match the per-dataset best, in raw error space, following [TabArena](https://arxiv.org/abs/2506.16791) / [BeyondArena](https://arxiv.org/abs/2606.30410)
 - **Bayesian pairwise comparison** — posterior probabilities that model A beats model B (or is practically equivalent), via [baycomp](https://github.com/janezd/baycomp)
-- **Dolan-Moré performance profiles** — cumulative distribution of performance ratios and area-under-profile scores
+- **Frequentist comparison** — Friedman + Nemenyi (all-pairs) or Wilcoxon + Holm (reference model), following [Demšar (2006)](https://jmlr.org/papers/v7/demsar06a.html)
+- **Dolan-Moré performance profiles** — cumulative distribution of performance ratios and area-under-profile scores, following [Dolan & Moré (2002)](https://doi.org/10.1007/s101070100263)
+- **Rank sensitivity** — Kendall τ-b with bootstrap CI measuring whether rankings hold across two experimental conditions, following [Kendall (1945)](https://doi.org/10.1093/biomet/33.3.239)
 
 
 ## Documentation
@@ -54,36 +59,62 @@ bench = evaluma.load_df(
     score="score",
 )
 
-# IQM ranking with 95% bootstrap CI
+# Point-estimate aggregate ranking (trimmed mean)
+agg = bench.aggregate_ranking()
+print(agg.table)
+
+# IQM ranking with 95% bootstrap CI (requires seed column)
 iqm = bench.iqm_ranking()
 print(iqm.table)
 fig = iqm.plot()
 fig.savefig("iqm.png")
 
+# ELO ranking with win-rate matrix
+elo = bench.elo_ranking()
+print(elo.table)
+fig = elo.plot_winrate()
+
+# Improvability ranking (mean % error reduction needed to match the best)
+imp = bench.improvability_ranking()
+print(imp.table)
+fig = imp.plot()
+
 # Bayesian pairwise probabilities
 bayes = bench.bayesian_comparison()
 print(bayes.table)
 
+# Frequentist comparison (Friedman + Nemenyi)
+freq = bench.frequentist_comparison()
+print(freq.table)
+
 # Dolan-Moré performance profiles
 profiles = bench.performance_profiles()
 fig = profiles.plot()
+
+# Rank sensitivity across two conditions
+bench_b = evaluma.load_df("results_b.csv", model="model", dataset="dataset",
+                          metric="metric", score="score")
+sens = bench.rank_sensitivity(bench_b, cond_a="condA", cond_b="condB")
+print(f"Kendall τ = {sens.tau:.3f}, 95% CI = {sens.tau_ci}")
 ```
 
 ### CLI
 
 ```bash
-# Run all three analyses and write six output files
+# Run aggregate, Bayesian, frequentist, and performance-profile analyses
 evaluma report results.csv \
     --model model --dataset dataset --metric metric --score score \
     --output results/
 
 # Individual subcommands
-evaluma rank    results.csv --model model --dataset dataset --metric metric --score score --output results/
-evaluma compare results.csv --model model --dataset dataset --metric metric --score score --output results/
-evaluma profiles results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma aggregate   results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma rank        results.csv --model model --dataset dataset --metric metric --score score --seed seed --output results/
+evaluma compare     results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma frequentist results.csv --model model --dataset dataset --metric metric --score score --output results/
+evaluma profiles    results.csv --model model --dataset dataset --metric metric --score score --output results/
 ```
 
-Each subcommand writes a `.csv` table and a `.png` figure to `--output`.
+Each subcommand writes a `.csv` table and a `.png` figure to `--output`. ELO ranking and rank sensitivity are available via the Python API only.
 
 ### Column mapping
 
@@ -193,6 +224,15 @@ also cite the works of the underlying methods and frameworks used:
   year      = {2021},
 }
 
+@inproceedings{erickson2025tabarena,
+  title     = {{TabArena}: A Living Benchmark for Machine Learning on Tabular Data},
+  author    = {Erickson, Nick and Purucker, Lennart and Tschalzev, Andrej
+               and Holzm{\"u}ller, David and Mutalik Desai, Prabhant
+               and Salinas, David and Hutter, Frank},
+  booktitle = {Advances in Neural Information Processing Systems},
+  year      = {2025},
+}
+
 @article{benavoli2017time,
   title   = {Time for a Change: a Tutorial for Comparing Multiple Classifiers
              Through Bayesian Analysis},
@@ -205,6 +245,15 @@ also cite the works of the underlying methods and frameworks used:
   year    = {2017},
 }
 
+@article{demsar2006statistical,
+  title   = {Statistical Comparisons of Classifiers over Multiple Data Sets},
+  author  = {Dem{\v{s}}ar, Janez},
+  journal = {Journal of Machine Learning Research},
+  volume  = {7},
+  pages   = {1--30},
+  year    = {2006},
+}
+
 @article{dolan2002benchmarking,
   title   = {Benchmarking Optimization Software with Performance Profiles},
   author  = {Dolan, Elizabeth D. and Mor{\'e}, Jorge J.},
@@ -213,4 +262,15 @@ also cite the works of the underlying methods and frameworks used:
   pages   = {201--213},
   year    = {2002},
 }
+
+@article{kendall1945treatment,
+  title   = {The Treatment of Ties in Ranking Problems},
+  author  = {Kendall, Maurice G.},
+  journal = {Biometrika},
+  volume  = {33},
+  number  = {3},
+  pages   = {239--251},
+  year    = {1945},
+  doi     = {10.1093/biomet/33.3.239},
+}
 ```

{evaluma-0.2.0 → evaluma-0.3.0}/docs/overview.md

@@ -1,11 +1,14 @@
 # Overview
 
-Given a collection of models evaluated on multiple datasets, the question "which model is best?" is harder than it sounds. Simple averages obscure robustness, ignore statistical uncertainty, and make it difficult to compare methods across papers. However, several works in the literature and different competition benchmarks have explored various schemas to come up with informative rankings. **evaluma** is designed to be an accessible entry point to build such rankings for your benchmark evaluations providing four complementary analyses from a single input format:
+Given a collection of models evaluated on multiple datasets, the question "which model is best?" is harder than it sounds. Simple averages obscure robustness, ignore statistical uncertainty, and make it difficult to compare methods across papers. However, several works in the literature and different competition benchmarks have explored various schemas to come up with informative rankings. **evaluma** is designed to be an accessible entry point to build such rankings for your benchmark evaluations providing six complementary analyses from a single input format:
 
 - **Aggregate ranking** — point-estimate ranking via trimmed mean, mean, or median
 - **IQM ranking** — robust central tendency with bootstrap confidence intervals (requires multiple seeds per model/dataset)
+- **ELO ranking** — MLE ELO ratings from pairwise head-to-head battles with bootstrap CIs and a win-rate matrix
 - **Bayesian comparison** — probability that one model outperforms another
+- **Frequentist comparison** — Friedman + Nemenyi (all-pairs) or Wilcoxon + Holm (reference model) significance tests
 - **Performance profiles** — cumulative distribution showing how often each model comes within a factor of the best
+- **Rank sensitivity** — Kendall τ-b measuring whether rankings hold across two experimental conditions
 
 This page walks through installing the package, preparing your results CSV, and running the analyses.
 
@@ -65,7 +68,7 @@ bench = evaluma.load_csv(
 )
 ```
 
-The `Benchmark` object holds your data and provides four analysis methods. Each returns a result object with a `.table` (pandas DataFrame) and a `.plot()` method:
+The `Benchmark` object holds your data and provides analysis methods. Each returns a result object with a `.table` (pandas DataFrame) and a `.plot()` method:
 
 ### Aggregate ranking
 
@@ -91,6 +94,19 @@ iqm.plot()
 
 The IQM discards the top and bottom 25% of per-dataset scores before averaging, making it resistant to outliers. The bootstrap CIs are stratified — seeds are resampled independently within each dataset. **Requires multiple seeds** — pass `seed="seed_column"` to `load_csv()` when loading. Use `aggregate_ranking()` for single-run data. See the [IQM Tutorial](./tutorials/iqm_ranking.md) for a more in-depth example.
 
+### ELO ranking
+
+Compute MLE ELO ratings from pairwise head-to-head battles across datasets:
+
+```python
+elo = bench.elo_ranking()
+print(elo.table)        # model, ELO, CI_low, CI_high
+elo.plot()              # horizontal bar chart with 95% CI
+elo.plot_winrate()      # M×M win-rate heatmap
+```
+
+Each dataset contributes equally via sample weighting. Bootstrap CIs resample battles within each dataset. `tie_threshold` treats near-equal scores as draws; `calibration_model` anchors one model to ELO 1000. See the [ELO Ranking Tutorial](./tutorials/elo_ranking.md) for a more in-depth example.
+
 ### Bayesian pairwise comparison
 
 Compute posterior probabilities for every model pair:
@@ -103,6 +119,22 @@ bayes.plot()
 
 For each pair (A, B), the output gives `p_a_better`, `p_equiv`, and `p_b_better` — the probabilities that A is better, equivalent (within a practical equivalence region), or worse than B. See the [Bayesian Comparison](./tutorials/bayesian_comparison.md) for a more in-depth example.
 
+### Frequentist comparison
+
+Run Friedman + Nemenyi (all-pairs) or Wilcoxon + Holm (reference model) significance tests:
+
+```python
+# All-pairs: Friedman omnibus then Nemenyi post-hoc
+freq = bench.frequentist_comparison()
+print(freq.table)
+freq.plot()    # critical-difference diagram
+
+# Reference mode: Wilcoxon + Holm against one baseline
+freq_ref = bench.frequentist_comparison(reference="ModelA")
+```
+
+Requires at least 5 datasets. See the [Frequentist Comparison Tutorial](./tutorials/frequentist_comparison.md) for a more in-depth example.
+
 ### Performance profiles
 
 Plot how often each model achieves near-best performance across datasets:
@@ -115,6 +147,24 @@ profiles.plot()
 
 The profile curve shows, for each performance ratio τ ≥ 1, the fraction of datasets where a model's score is within τ of the best. A curve that rises faster means the model is closer to best more often. See the [Performance Profile Tutorial](./tutorials/performance_profiles.md) for a more in-depth example.
 
+### Rank sensitivity
+
+Measure whether model rankings hold when you change an experimental condition (e.g. optimizer, augmentation policy, normalization choice):
+
+```python
+from evaluma import load_csv
+
+# Load two conditions separately, or use condition_col to split a single DataFrame
+bench_b = load_csv("results_b.csv", model="model", dataset="dataset",
+                   metric="metric", score="score")
+sens = bench.rank_sensitivity(bench_b, cond_a="condA", cond_b="condB")
+print(f"Kendall τ = {sens.tau:.3f},  95% CI = {sens.tau_ci}")
+print(sens.table)   # per-model delta_rank, sorted by |delta_rank|
+sens.plot()         # rank scatter with identity line
+```
+
+Alternatively, pass `condition_col=` to `load_df` to get a `BenchmarkGroup` and call `.rank_sensitivity("condA", "condB")` directly. See the [Rank Sensitivity Tutorial](./tutorials/rank_sensitivity.md) for a more in-depth example.
+
 ## Column mapping
 
 If your CSV uses different column names, pass them explicitly:
@@ -131,7 +181,7 @@ bench = evaluma.load_csv(
 
 ## CLI quickstart
 
-Run all three point-estimate analyses and write CSV and PNG outputs to the current directory:
+Run aggregate, Bayesian, frequentist, and performance-profile analyses, writing CSV and PNG outputs:
 
 ```bash
 evaluma report results.csv
@@ -149,10 +199,15 @@ Save outputs to a specific directory:
 evaluma report results.csv --output ./results/
 ```
 
-Individual subcommands are also available: `evaluma rank` (IQM with seeds), `evaluma aggregate`, `evaluma compare`, and `evaluma profiles`. The CLI supports the same column mapping with `--model`, `--dataset`, `--metric`, and `--score` flags, or via a YAML config file passed with `--config`.
+Individual subcommands are also available: `evaluma aggregate`, `evaluma rank` (IQM, requires `--seed`), `evaluma compare` (Bayesian), `evaluma frequentist`, and `evaluma profiles`. The CLI supports the same column mapping with `--model`, `--dataset`, `--metric`, and `--score` flags, or via a YAML config file passed with `--config`.
+
+ELO ranking and rank sensitivity are available via the Python API only.
 
 ## Next steps
 
 - [IQM Ranking tutorial](tutorials/iqm_ranking) — deeper dive with worked examples
+- [ELO Ranking tutorial](tutorials/elo_ranking) — head-to-head battles, win-rate matrix, and calibration
 - [Bayesian Comparison tutorial](tutorials/bayesian_comparison) — interpreting posterior probabilities
+- [Frequentist Comparison tutorial](tutorials/frequentist_comparison) — Friedman, Nemenyi, Wilcoxon, and CD diagrams
 - [Performance Profiles tutorial](tutorials/performance_profiles) — understanding the Dolan-Moré framework
+- [Rank Sensitivity tutorial](tutorials/rank_sensitivity) — measuring ranking stability across conditions

{evaluma-0.2.0 → evaluma-0.3.0}/docs/references.md

@@ -11,6 +11,15 @@ Deep Reinforcement Learning at the Edge of the Statistical Precipice.
 
 ---
 
+**ELO Ranking**
+
+Erickson, N., Purucker, L., Tschalzev, A., Holzmüller, D., Mutalik Desai, P., Salinas, D., & Hutter, F. (2025).
+TabArena: A Living Benchmark for Machine Learning on Tabular Data.
+*Advances in Neural Information Processing Systems* (Datasets and Benchmarks, Spotlight).
+<https://arxiv.org/abs/2506.16791>
+
+---
+
 **Bayesian Pairwise Comparison**
 
 Benavoli, A., Corani, G., Demšar, J., & Zaffalon, M. (2017).
@@ -20,6 +29,15 @@ Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesia
 
 ---
 
+**Frequentist Comparison**
+
+Demšar, J. (2006).
+Statistical Comparisons of Classifiers over Multiple Data Sets.
+*Journal of Machine Learning Research*, 7, 1–30.
+<https://jmlr.org/papers/v7/demsar06a.html>
+
+---
+
 **Dolan-Moré Performance Profiles**
 
 Dolan, E. D., & Moré, J. J. (2002).
@@ -29,6 +47,15 @@ Benchmarking Optimization Software with Performance Profiles.
 
 ---
 
+**Rank Sensitivity**
+
+Kendall, M. G. (1945).
+The Treatment of Ties in Ranking Problems.
+*Biometrika*, 33(3), 239–251.
+<https://doi.org/10.1093/biomet/33.3.239>
+
+---
+
 ## Libraries
 
 **baycomp**