errorbars 0.1.2__tar.gz

errorbars-0.1.2/.gitignore

@@ -0,0 +1,22 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+.venv/
+venv/
+dist/
+build/
+*.egg
+
+node_modules/
+web/dist/
+web/.vite/
+npm-debug.log*
+
+.DS_Store
+*.swp
+web/*.tsbuildinfo

errorbars-0.1.2/CHANGELOG.md

@@ -0,0 +1,74 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## [0.1.2] - 2026-10-01
+
+### Added
+
+- Published to PyPI: `pip install "errorbars[cli]"`. The README's images and
+  links are rewritten to absolute URLs at build time so they work on the
+  project page.
+- `errorbars --version`.
+- `py.typed`, so type checkers use the package's annotations (the `Typing ::
+  Typed` classifier was already declared).
+
+## [0.1.1] - 2026-09-30
+
+### Fixed
+
+- The paired test's p-value and CI used the normal distribution, and the
+  README called that "somewhat conservative" for small n; it is the opposite
+  (at 8 degrees of freedom, a statistic of 2.2 gave p = 0.028 instead of
+  0.059). Paired comparisons now use Student's t with n - 1 degrees of freedom
+  and match `scipy.stats.ttest_rel` to 1e-7, and clustered SEs (the clustered
+  CI in `summarize`, the clustered paired CI and p-value, and so the
+  leaderboard's Holm-corrected tests) use t with G - 1 degrees of freedom for
+  G clusters. No scipy at runtime: the t tail comes from a continued-fraction
+  incomplete beta function. On the bundled example the headline pair's paired
+  p moves from 0.1116 to 0.1131, its clustered p from 0.1154 to 0.1235, and
+  its Holm-adjusted p from 0.298 to 0.322; groups are unchanged.
+  With a single cluster, where the clustered SE falls back to the plain one,
+  the clustered test keeps the n - 1 reference too.
+
+- A non-finite score (`nan`, `inf`) was accepted and propagated into every
+  mean, CI and p-value; on a leaderboard the NaN model sorted to rank 1.
+  Loading now fails with the row number.
+- A repeated (model, question, sample) row was counted as another question,
+  inflating n and shrinking standard errors. Loading now fails with both row
+  numbers and says to give repeated generations distinct `sample` values.
+- `power` answered "2 questions" for a baseline of 0 or 1 (where p(1-p) = 0)
+  and accepted targets above 100% (`--baseline 0.98 --delta 0.05`). The
+  baseline must be strictly between 0 and 1, baseline + delta can't exceed
+  1, and a continuous-metric variance must be positive; the web calculator's
+  effect-size slider is capped at 1 - baseline to match.
+- `compare`/`summarize` with an unknown model name said "fewer than 2 shared
+  question_ids" or "no rows"; they now name the missing model and list the
+  models in the file.
+- Install hints for the optional extras named `errorbars` on PyPI, where it
+  isn't published; they now name the dependency or the Git URL.
+
+## [0.1.0] - 2026-09-24
+
+Initial release.
+
+- `summarize`: CLT, Wilson, and bootstrap confidence intervals; clustered
+  standard errors with design effect and ICC; within/between-question
+  variance decomposition for repeated samples.
+- `compare`: paired mean difference, SE, CI, p-value; correlation and
+  variance reduction from pairing; clustered paired SE; exact McNemar test
+  for binary scores.
+- `leaderboard`: per-model CIs, Holm-corrected pairwise paired tests,
+  maximal-clique grouping of statistically indistinguishable models, SVG and
+  (optional) matplotlib forest plots.
+- `power`: questions needed for a target effect/power, and minimum
+  detectable effect for a given n, accounting for pairing correlation,
+  repeated sampling, and cluster design effect.
+- CLI (`errorbars summarize|compare|leaderboard|power|import`) with `rich`
+  tables and `--json` output.
+- Adapters: `errorbars import lm-eval|inspect` converts
+  lm-evaluation-harness `--log_samples` JSONL or an Inspect AI `.eval` log
+  into the canonical format, verified against real output from lm-eval
+  0.4.13 and inspect-ai 0.3.268.
+- Web calculator (Vite + TypeScript) mirroring the Python power formulas,
+  with shared JSON test vectors.

errorbars-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anton Soloviev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

errorbars-0.1.2/PKG-INFO

@@ -0,0 +1,297 @@
+Metadata-Version: 2.5
+Name: errorbars
+Version: 0.1.2
+Summary: Error bars for LLM evals: standard errors, clustering, paired comparisons, leaderboards, and power analysis.
+Project-URL: Homepage, https://antonsoo.github.io/errorbars/
+Project-URL: Repository, https://github.com/antonsoo/errorbars
+Project-URL: Issues, https://github.com/antonsoo/errorbars/issues
+Author-email: Anton Soloviev <anton@praviel.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmarking,confidence-intervals,evaluation,language-models,llm-evals,statistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Provides-Extra: all
+Requires-Dist: matplotlib>=3.7; extra == 'all'
+Requires-Dist: pandas>=1.5; extra == 'all'
+Requires-Dist: rich>=13.0; extra == 'all'
+Provides-Extra: cli
+Requires-Dist: rich>=13.0; extra == 'cli'
+Provides-Extra: inspect
+Requires-Dist: agent-client-protocol<1.0,>=0.12; extra == 'inspect'
+Requires-Dist: httpx<1.0,>=0.27; extra == 'inspect'
+Requires-Dist: inspect-ai>=0.3.268; extra == 'inspect'
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == 'pandas'
+Provides-Extra: plot
+Requires-Dist: matplotlib>=3.7; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# errorbars
+
+**Error bars for LLM evals. Know whether model B is actually better, or you're reading noise.**
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/antonsoo/errorbars/blob/main/LICENSE)
+[![Live demo](https://img.shields.io/badge/live%20demo-calculator-2454a6)](https://antonsoo.github.io/errorbars/)
+[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-workbench-ffd21e)](https://huggingface.co/spaces/antonsoloviev/errorbars)
+
+LLM eval results get reported as bare accuracies — "model B scored 71.2%, up from 69.7%" — with
+no error bar, no significance test, and no accounting for the fact that the questions came in
+correlated groups. On a 500-question benchmark, a 1.5-point gain is very often noise: at a 50%
+baseline you'd need roughly a 9-point gap to reliably detect anything at 80% power, α=0.05
+(`errorbars power --n 500 --baseline 0.5` → minimum detectable effect ≈ 8.9 points). Evan Miller's
+"[Adding Error Bars to Evals: A Statistical Approach to Language Model
+Evaluations](https://arxiv.org/abs/2411.00640)" (arXiv:2411.00640, 2024) lays out the right
+practice — CLT and clustered standard errors, variance reduction through pairing, power analysis —
+and this package turns it into a one-liner.
+
+![errorbars leaderboard on a synthetic clustered benchmark](https://raw.githubusercontent.com/antonsoo/errorbars/main/docs/assets/leaderboard-terminal.png)
+
+## Why this exists
+
+Run `errorbars leaderboard` on a synthetic benchmark (below) and one apparent 7-point win —
+`tuned-70b` over `baseline-70b`, 0.690 vs. 0.620 — is not statistically significant once it has a
+proper error bar: the paired test already gives p = 0.11. Accounting for the fact that questions
+come 5-to-a-passage, and Holm-correcting across all 6 pairwise comparisons on the board, pushes
+that to p = 0.32. A naive leaderboard that ranks by bare accuracy would still have shipped the gap
+as a win. The full walkthrough, with every number copied from a real command, is in
+[`examples/README.md`](https://github.com/antonsoo/errorbars/blob/main/examples/README.md).
+
+## Quickstart
+
+```bash
+pip install "errorbars[cli]"
+errorbars power --delta 0.03 --baseline 0.5
+```
+
+```
+Questions needed: 4361
+```
+
+That's how many questions you'd need to reliably detect a 3-point accuracy gap at a 50% baseline,
+with the defaults (80% power, α=0.05, no pairing correlation, no clustering). Clone the repo to
+try it against real per-item scores:
+
+```bash
+git clone https://github.com/antonsoo/errorbars && cd errorbars
+errorbars leaderboard examples/data/reading_comprehension.csv
+```
+
+## Features
+
+- **`summarize`** — mean, SE, and 95% CI (CLT by default, Wilson for small-n binary scores,
+  bootstrap on request); clustered SE with design effect and ICC when a `cluster_id` column is
+  present; within/between-question variance decomposition when a `sample` column is present.
+- **`compare`** — paired mean difference, SE, CI, p-value; the correlation between the two models'
+  per-question scores and how much pairing shrank the SE vs. an unpaired comparison; a
+  cluster-robust paired SE/p-value when clusters are present; exact McNemar test for binary scores.
+- **`leaderboard`** — every model with its CI, Holm-corrected pairwise paired tests, and groups of
+  statistically indistinguishable models (maximal cliques of the "not significantly different"
+  graph); a forest plot (SVG, no dependency; matplotlib if installed).
+- **`power`** — number of questions needed to detect an effect δ at a given α and power, or the
+  minimum detectable effect for a given n, accounting for pairing correlation, repeated sampling,
+  and cluster design effect.
+- **CLI** — `errorbars summarize|compare|leaderboard|power|import`, `rich` tables by default,
+  `--json` for scripting.
+- **Adapters** — `errorbars import lm-eval|inspect` converts lm-evaluation-harness `--log_samples`
+  output or an Inspect AI `.eval` log into the canonical format (see below).
+- **Web calculator** — a static "how many eval questions do I need?" power calculator
+  ([live demo](https://antonsoo.github.io/errorbars/)), whose TypeScript formulas are checked
+  against the Python ones by the test suite.
+- Typed Python ≥3.10, `numpy` the only runtime dependency; `pandas` and `matplotlib` are optional
+  extras; `scipy`/`statsmodels` are test-only oracles, never imported at runtime.
+
+## Usage
+
+```bash
+errorbars summarize examples/data/reading_comprehension.csv --model tuned-70b
+```
+
+```
+    summarize: tuned-70b
+┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric ┃            value ┃
+┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n      │              200 │
+│ mean   │           0.6900 │
+│ SE     │           0.0328 │
+│ 95% CI │ [0.6257, 0.7543] │
+│ method │              clt │
+└────────┴──────────────────┘
+       clustering diagnostics
+┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric        ┃            value ┃
+┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n clusters    │               40 │
+│ ICC           │           0.0710 │
+│ design effect │            1.284 │
+│ clustered SE  │           0.0372 │
+│ clustered CI  │ [0.6148, 0.7652] │
+└───────────────┴──────────────────┘
+```
+
+```bash
+errorbars compare examples/data/reading_comprehension.csv \
+  --model-a tuned-70b --model-b baseline-70b
+```
+
+```
+mean diff (A - B)                 0.0700
+paired SE                         0.0440
+95% CI                  [-0.0167, 0.1567]
+p-value                           0.1131
+correlation(A, B)                 0.1434
+variance reduction from pairing     14.3%
+McNemar exact p-value             0.1405
+```
+
+```bash
+errorbars power --delta 0.05 --baseline 0.65 --rho 0.14 --cluster-deff 1.28
+```
+
+```
+Questions needed: 1573
+```
+
+Every number above is copied verbatim from running these commands against
+`examples/data/reading_comprehension.csv` (synthetic — see below). The full leaderboard output and
+the reasoning behind each step are in [`examples/README.md`](https://github.com/antonsoo/errorbars/blob/main/examples/README.md).
+
+### Input format
+
+Long-format CSV or JSONL, one row per observation:
+
+| question_id | cluster_id (optional) | model | score | sample (optional) |
+|---|---|---|---|---|
+| q1 | passage-003 | tuned-70b | 1 | |
+
+Column names are configurable (`--question-col`, `--cluster-col`, etc., or `ColumnMap` in Python).
+Scores can be binary (0/1) or continuous. `cluster_id` groups correlated questions (e.g. several
+questions per reading passage); `sample` marks repeated generations of the same question.
+
+### Importing from lm-evaluation-harness or Inspect AI
+
+`errorbars import` converts either tool's own log format into the canonical CSV above:
+
+```bash
+# lm-evaluation-harness: lm_eval run --model <...> --tasks <...> --log_samples --output_path <dir>
+errorbars import lm-eval runs/<dir>/samples_<task>_<timestamp>.jsonl \
+  --model my-model-name -o converted.csv
+
+# Inspect AI: inspect eval <task> --model <...>  (writes a .eval log by default)
+errorbars import inspect logs/<run>.eval -o converted.csv
+```
+
+`--metric` (lm-eval) picks which computed metric to use as the score when a task reports more than
+one (e.g. `acc` vs. `acc_norm`); it defaults to the first one. `--scorer` (Inspect) does the same
+for tasks with multiple scorers. Inspect epochs (`--epochs N`, repeated sampling of the same input)
+land in the `sample` column automatically.
+
+Both adapters were built and tested against real, unedited output — not from memory: **lm-eval
+0.4.13** (`lm_eval run --model dummy --tasks copa --limit 20 --log_samples ...`, plus a
+multi-metric `arc_easy` run) and **inspect-ai 0.3.268** (a 5-sample task through the built-in
+`mockllm/model` provider). The exact log files are committed as test fixtures
+(`tests/fixtures/samples_*.jsonl`, `tests/fixtures/inspect_tiny_qa.eval`) and re-parsed in
+`tests/test_adapters.py` on every run. The Inspect adapter reads logs with Inspect's own
+`inspect_ai.log.read_eval_log` and `inspect_ai.scorer.value_to_float` rather than hand-parsing its
+binary `.eval` format, and needs the `inspect` extra:
+`pip install "errorbars[inspect]"`. The lm-eval
+adapter has no extra dependency — `--log_samples` is already plain JSONL.
+
+## How it works
+
+Every statistic is implemented from scratch on `numpy` + the standard library
+(`statistics.NormalDist` for normal quantiles; Student-t tails from a continued-fraction incomplete
+beta function) — no scipy or statsmodels at runtime.
+Full derivations with references are in [`docs/formulas.md`](https://github.com/antonsoo/errorbars/blob/main/docs/formulas.md):
+
+1. CLT and Wilson confidence intervals for a mean
+2. Cluster-robust standard errors (the CR1 sandwich estimator, matching
+   `statsmodels`' `cov_type="cluster"`)
+3. Intraclass correlation and Kish's design effect
+4. Within/between-question variance decomposition for repeated sampling
+5. Paired comparisons, variance reduction from pairing, and exact McNemar
+6. Holm-Bonferroni correction and maximal-clique grouping for leaderboards
+7. Power analysis (sample size and minimum detectable effect) under pairing and clustering
+
+## Accuracy and limitations
+
+- Cluster-robust SE matches `statsmodels`' `OLS(..., cov_type="cluster")` to 1e-9 on both balanced
+  and unbalanced cluster sizes; Wilson intervals match `statsmodels.stats.proportion_confint` to
+  1e-9; the paired t-test matches `scipy.stats.ttest_rel` to 1e-7 at any n, and McNemar's exact
+  test is cross-checked against `statsmodels`.
+  See `tests/test_stats_vs_oracles.py` and `tests/test_compare_vs_oracles.py`.
+- Monte Carlo coverage tests (`tests/test_coverage_montecarlo.py`) confirm nominal 95% CIs cover
+  the true parameter close to 95% of the time — for CLT, Wilson, bootstrap, and cluster-robust
+  intervals — with a tolerance sized to the trial count so it won't flake.
+- Paired comparisons use Student's t with n − 1 degrees of freedom, like `scipy.stats.ttest_rel`,
+  and clustered SEs use t with G − 1 for G clusters (Cameron & Miller 2015), computed without
+  scipy. With few clusters that interval is honest but wide; the per-model `clt` interval stays
+  normal-based, which is only right once n is in the dozens.
+- The power formula's samples-per-question adjustment assumes all single-sample variance is
+  decoding noise (see `docs/formulas.md` §11 for why, and the caveat on when this is optimistic).
+  It's a planning tool for before you run the eval; for a post-hoc measurement with the true
+  within/between-question split, use `summarize` on data with a `sample` column instead.
+- Leaderboard groups are maximal cliques of the "not significantly different" graph, which is the
+  statistically direct approach — it can produce a model in more than one group, unlike a
+  minimal-letters heuristic (e.g. R's `multcompView`).
+- 168 Python tests, 605 TypeScript tests (cross-checking the JS formulas against Python-generated
+  vectors), all passing on this box (14 vCPU WSL2 Linux, 48 GB RAM; Python 3.12.3, Node 26.7.0)
+  as of 2026-10-01.
+
+## Web calculator
+
+[`web/`](https://github.com/antonsoo/errorbars/blob/main/web/) is a static Vite + TypeScript "how many eval questions do I need?" calculator
+([live demo](https://antonsoo.github.io/errorbars/)). Its power formulas
+(`web/src/power.ts`, `web/src/normal.ts`) are a direct port of `src/errorbars/power.py`; Python
+generates JSON test vectors (`scripts/export_test_vectors.py` → `web/test-vectors.json`) that the
+TypeScript test suite checks against, so the two implementations can't silently drift apart.
+
+![Power calculator](https://raw.githubusercontent.com/antonsoo/errorbars/main/docs/assets/calculator-hero.png)
+
+## Development
+
+```bash
+uv sync --group dev --extra all
+uv run pytest
+uv run ruff check .
+uv run mypy src/errorbars
+```
+
+See [`CONTRIBUTING.md`](https://github.com/antonsoo/errorbars/blob/main/CONTRIBUTING.md) for the web calculator's dev loop and guidelines.
+
+## Contributing
+
+Issues and PRs welcome. Any new statistic needs a test against an independent oracle (reference
+library, closed form, or Monte Carlo simulation) — see `tests/` for the pattern.
+
+## Citations
+
+```bibtex
+@misc{miller2024addingerrorbarsevals,
+  title  = {Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations},
+  author = {Evan Miller},
+  year   = {2024},
+  eprint = {2411.00640},
+  archivePrefix = {arXiv},
+  primaryClass  = {stat.AP},
+  url    = {https://arxiv.org/abs/2411.00640}
+}
+```
+
+## License
+
+[MIT](https://github.com/antonsoo/errorbars/blob/main/LICENSE) © 2026 Anton Soloviev
+
+---
+
+<sub>Part of [Officina](https://antonsoo.github.io/officina/), a set of small open-source tools by [Anton Soloviev](https://github.com/antonsoo).</sub>

errorbars-0.1.2/README.md

@@ -0,0 +1,259 @@
+# errorbars
+
+**Error bars for LLM evals. Know whether model B is actually better, or you're reading noise.**
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+[![Live demo](https://img.shields.io/badge/live%20demo-calculator-2454a6)](https://antonsoo.github.io/errorbars/)
+[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-workbench-ffd21e)](https://huggingface.co/spaces/antonsoloviev/errorbars)
+
+LLM eval results get reported as bare accuracies — "model B scored 71.2%, up from 69.7%" — with
+no error bar, no significance test, and no accounting for the fact that the questions came in
+correlated groups. On a 500-question benchmark, a 1.5-point gain is very often noise: at a 50%
+baseline you'd need roughly a 9-point gap to reliably detect anything at 80% power, α=0.05
+(`errorbars power --n 500 --baseline 0.5` → minimum detectable effect ≈ 8.9 points). Evan Miller's
+"[Adding Error Bars to Evals: A Statistical Approach to Language Model
+Evaluations](https://arxiv.org/abs/2411.00640)" (arXiv:2411.00640, 2024) lays out the right
+practice — CLT and clustered standard errors, variance reduction through pairing, power analysis —
+and this package turns it into a one-liner.
+
+![errorbars leaderboard on a synthetic clustered benchmark](docs/assets/leaderboard-terminal.png)
+
+## Why this exists
+
+Run `errorbars leaderboard` on a synthetic benchmark (below) and one apparent 7-point win —
+`tuned-70b` over `baseline-70b`, 0.690 vs. 0.620 — is not statistically significant once it has a
+proper error bar: the paired test already gives p = 0.11. Accounting for the fact that questions
+come 5-to-a-passage, and Holm-correcting across all 6 pairwise comparisons on the board, pushes
+that to p = 0.32. A naive leaderboard that ranks by bare accuracy would still have shipped the gap
+as a win. The full walkthrough, with every number copied from a real command, is in
+[`examples/README.md`](examples/README.md).
+
+## Quickstart
+
+```bash
+pip install "errorbars[cli]"
+errorbars power --delta 0.03 --baseline 0.5
+```
+
+```
+Questions needed: 4361
+```
+
+That's how many questions you'd need to reliably detect a 3-point accuracy gap at a 50% baseline,
+with the defaults (80% power, α=0.05, no pairing correlation, no clustering). Clone the repo to
+try it against real per-item scores:
+
+```bash
+git clone https://github.com/antonsoo/errorbars && cd errorbars
+errorbars leaderboard examples/data/reading_comprehension.csv
+```
+
+## Features
+
+- **`summarize`** — mean, SE, and 95% CI (CLT by default, Wilson for small-n binary scores,
+  bootstrap on request); clustered SE with design effect and ICC when a `cluster_id` column is
+  present; within/between-question variance decomposition when a `sample` column is present.
+- **`compare`** — paired mean difference, SE, CI, p-value; the correlation between the two models'
+  per-question scores and how much pairing shrank the SE vs. an unpaired comparison; a
+  cluster-robust paired SE/p-value when clusters are present; exact McNemar test for binary scores.
+- **`leaderboard`** — every model with its CI, Holm-corrected pairwise paired tests, and groups of
+  statistically indistinguishable models (maximal cliques of the "not significantly different"
+  graph); a forest plot (SVG, no dependency; matplotlib if installed).
+- **`power`** — number of questions needed to detect an effect δ at a given α and power, or the
+  minimum detectable effect for a given n, accounting for pairing correlation, repeated sampling,
+  and cluster design effect.
+- **CLI** — `errorbars summarize|compare|leaderboard|power|import`, `rich` tables by default,
+  `--json` for scripting.
+- **Adapters** — `errorbars import lm-eval|inspect` converts lm-evaluation-harness `--log_samples`
+  output or an Inspect AI `.eval` log into the canonical format (see below).
+- **Web calculator** — a static "how many eval questions do I need?" power calculator
+  ([live demo](https://antonsoo.github.io/errorbars/)), whose TypeScript formulas are checked
+  against the Python ones by the test suite.
+- Typed Python ≥3.10, `numpy` the only runtime dependency; `pandas` and `matplotlib` are optional
+  extras; `scipy`/`statsmodels` are test-only oracles, never imported at runtime.
+
+## Usage
+
+```bash
+errorbars summarize examples/data/reading_comprehension.csv --model tuned-70b
+```
+
+```
+    summarize: tuned-70b
+┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric ┃            value ┃
+┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n      │              200 │
+│ mean   │           0.6900 │
+│ SE     │           0.0328 │
+│ 95% CI │ [0.6257, 0.7543] │
+│ method │              clt │
+└────────┴──────────────────┘
+       clustering diagnostics
+┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric        ┃            value ┃
+┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n clusters    │               40 │
+│ ICC           │           0.0710 │
+│ design effect │            1.284 │
+│ clustered SE  │           0.0372 │
+│ clustered CI  │ [0.6148, 0.7652] │
+└───────────────┴──────────────────┘
+```
+
+```bash
+errorbars compare examples/data/reading_comprehension.csv \
+  --model-a tuned-70b --model-b baseline-70b
+```
+
+```
+mean diff (A - B)                 0.0700
+paired SE                         0.0440
+95% CI                  [-0.0167, 0.1567]
+p-value                           0.1131
+correlation(A, B)                 0.1434
+variance reduction from pairing     14.3%
+McNemar exact p-value             0.1405
+```
+
+```bash
+errorbars power --delta 0.05 --baseline 0.65 --rho 0.14 --cluster-deff 1.28
+```
+
+```
+Questions needed: 1573
+```
+
+Every number above is copied verbatim from running these commands against
+`examples/data/reading_comprehension.csv` (synthetic — see below). The full leaderboard output and
+the reasoning behind each step are in [`examples/README.md`](examples/README.md).
+
+### Input format
+
+Long-format CSV or JSONL, one row per observation:
+
+| question_id | cluster_id (optional) | model | score | sample (optional) |
+|---|---|---|---|---|
+| q1 | passage-003 | tuned-70b | 1 | |
+
+Column names are configurable (`--question-col`, `--cluster-col`, etc., or `ColumnMap` in Python).
+Scores can be binary (0/1) or continuous. `cluster_id` groups correlated questions (e.g. several
+questions per reading passage); `sample` marks repeated generations of the same question.
+
+### Importing from lm-evaluation-harness or Inspect AI
+
+`errorbars import` converts either tool's own log format into the canonical CSV above:
+
+```bash
+# lm-evaluation-harness: lm_eval run --model <...> --tasks <...> --log_samples --output_path <dir>
+errorbars import lm-eval runs/<dir>/samples_<task>_<timestamp>.jsonl \
+  --model my-model-name -o converted.csv
+
+# Inspect AI: inspect eval <task> --model <...>  (writes a .eval log by default)
+errorbars import inspect logs/<run>.eval -o converted.csv
+```
+
+`--metric` (lm-eval) picks which computed metric to use as the score when a task reports more than
+one (e.g. `acc` vs. `acc_norm`); it defaults to the first one. `--scorer` (Inspect) does the same
+for tasks with multiple scorers. Inspect epochs (`--epochs N`, repeated sampling of the same input)
+land in the `sample` column automatically.
+
+Both adapters were built and tested against real, unedited output — not from memory: **lm-eval
+0.4.13** (`lm_eval run --model dummy --tasks copa --limit 20 --log_samples ...`, plus a
+multi-metric `arc_easy` run) and **inspect-ai 0.3.268** (a 5-sample task through the built-in
+`mockllm/model` provider). The exact log files are committed as test fixtures
+(`tests/fixtures/samples_*.jsonl`, `tests/fixtures/inspect_tiny_qa.eval`) and re-parsed in
+`tests/test_adapters.py` on every run. The Inspect adapter reads logs with Inspect's own
+`inspect_ai.log.read_eval_log` and `inspect_ai.scorer.value_to_float` rather than hand-parsing its
+binary `.eval` format, and needs the `inspect` extra:
+`pip install "errorbars[inspect]"`. The lm-eval
+adapter has no extra dependency — `--log_samples` is already plain JSONL.
+
+## How it works
+
+Every statistic is implemented from scratch on `numpy` + the standard library
+(`statistics.NormalDist` for normal quantiles; Student-t tails from a continued-fraction incomplete
+beta function) — no scipy or statsmodels at runtime.
+Full derivations with references are in [`docs/formulas.md`](docs/formulas.md):
+
+1. CLT and Wilson confidence intervals for a mean
+2. Cluster-robust standard errors (the CR1 sandwich estimator, matching
+   `statsmodels`' `cov_type="cluster"`)
+3. Intraclass correlation and Kish's design effect
+4. Within/between-question variance decomposition for repeated sampling
+5. Paired comparisons, variance reduction from pairing, and exact McNemar
+6. Holm-Bonferroni correction and maximal-clique grouping for leaderboards
+7. Power analysis (sample size and minimum detectable effect) under pairing and clustering
+
+## Accuracy and limitations
+
+- Cluster-robust SE matches `statsmodels`' `OLS(..., cov_type="cluster")` to 1e-9 on both balanced
+  and unbalanced cluster sizes; Wilson intervals match `statsmodels.stats.proportion_confint` to
+  1e-9; the paired t-test matches `scipy.stats.ttest_rel` to 1e-7 at any n, and McNemar's exact
+  test is cross-checked against `statsmodels`.
+  See `tests/test_stats_vs_oracles.py` and `tests/test_compare_vs_oracles.py`.
+- Monte Carlo coverage tests (`tests/test_coverage_montecarlo.py`) confirm nominal 95% CIs cover
+  the true parameter close to 95% of the time — for CLT, Wilson, bootstrap, and cluster-robust
+  intervals — with a tolerance sized to the trial count so it won't flake.
+- Paired comparisons use Student's t with n − 1 degrees of freedom, like `scipy.stats.ttest_rel`,
+  and clustered SEs use t with G − 1 for G clusters (Cameron & Miller 2015), computed without
+  scipy. With few clusters that interval is honest but wide; the per-model `clt` interval stays
+  normal-based, which is only right once n is in the dozens.
+- The power formula's samples-per-question adjustment assumes all single-sample variance is
+  decoding noise (see `docs/formulas.md` §11 for why, and the caveat on when this is optimistic).
+  It's a planning tool for before you run the eval; for a post-hoc measurement with the true
+  within/between-question split, use `summarize` on data with a `sample` column instead.
+- Leaderboard groups are maximal cliques of the "not significantly different" graph, which is the
+  statistically direct approach — it can produce a model in more than one group, unlike a
+  minimal-letters heuristic (e.g. R's `multcompView`).
+- 168 Python tests, 605 TypeScript tests (cross-checking the JS formulas against Python-generated
+  vectors), all passing on this box (14 vCPU WSL2 Linux, 48 GB RAM; Python 3.12.3, Node 26.7.0)
+  as of 2026-10-01.
+
+## Web calculator
+
+[`web/`](web/) is a static Vite + TypeScript "how many eval questions do I need?" calculator
+([live demo](https://antonsoo.github.io/errorbars/)). Its power formulas
+(`web/src/power.ts`, `web/src/normal.ts`) are a direct port of `src/errorbars/power.py`; Python
+generates JSON test vectors (`scripts/export_test_vectors.py` → `web/test-vectors.json`) that the
+TypeScript test suite checks against, so the two implementations can't silently drift apart.
+
+![Power calculator](docs/assets/calculator-hero.png)
+
+## Development
+
+```bash
+uv sync --group dev --extra all
+uv run pytest
+uv run ruff check .
+uv run mypy src/errorbars
+```
+
+See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the web calculator's dev loop and guidelines.
+
+## Contributing
+
+Issues and PRs welcome. Any new statistic needs a test against an independent oracle (reference
+library, closed form, or Monte Carlo simulation) — see `tests/` for the pattern.
+
+## Citations
+
+```bibtex
+@misc{miller2024addingerrorbarsevals,
+  title  = {Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations},
+  author = {Evan Miller},
+  year   = {2024},
+  eprint = {2411.00640},
+  archivePrefix = {arXiv},
+  primaryClass  = {stat.AP},
+  url    = {https://arxiv.org/abs/2411.00640}
+}
+```
+
+## License
+
+[MIT](LICENSE) © 2026 Anton Soloviev
+
+---
+
+<sub>Part of [Officina](https://antonsoo.github.io/officina/), a set of small open-source tools by [Anton Soloviev](https://github.com/antonsoo).</sub>