errorbars 0.1.2__py3-none-any.whl

errorbars-0.1.2.dist-info/METADATA

@@ -0,0 +1,297 @@
+Metadata-Version: 2.5
+Name: errorbars
+Version: 0.1.2
+Summary: Error bars for LLM evals: standard errors, clustering, paired comparisons, leaderboards, and power analysis.
+Project-URL: Homepage, https://antonsoo.github.io/errorbars/
+Project-URL: Repository, https://github.com/antonsoo/errorbars
+Project-URL: Issues, https://github.com/antonsoo/errorbars/issues
+Author-email: Anton Soloviev <anton@praviel.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmarking,confidence-intervals,evaluation,language-models,llm-evals,statistics
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Provides-Extra: all
+Requires-Dist: matplotlib>=3.7; extra == 'all'
+Requires-Dist: pandas>=1.5; extra == 'all'
+Requires-Dist: rich>=13.0; extra == 'all'
+Provides-Extra: cli
+Requires-Dist: rich>=13.0; extra == 'cli'
+Provides-Extra: inspect
+Requires-Dist: agent-client-protocol<1.0,>=0.12; extra == 'inspect'
+Requires-Dist: httpx<1.0,>=0.27; extra == 'inspect'
+Requires-Dist: inspect-ai>=0.3.268; extra == 'inspect'
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == 'pandas'
+Provides-Extra: plot
+Requires-Dist: matplotlib>=3.7; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# errorbars
+
+**Error bars for LLM evals. Know whether model B is actually better, or you're reading noise.**
+
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/antonsoo/errorbars/blob/main/LICENSE)
+[![Live demo](https://img.shields.io/badge/live%20demo-calculator-2454a6)](https://antonsoo.github.io/errorbars/)
+[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-workbench-ffd21e)](https://huggingface.co/spaces/antonsoloviev/errorbars)
+
+LLM eval results get reported as bare accuracies — "model B scored 71.2%, up from 69.7%" — with
+no error bar, no significance test, and no accounting for the fact that the questions came in
+correlated groups. On a 500-question benchmark, a 1.5-point gain is very often noise: at a 50%
+baseline you'd need roughly a 9-point gap to reliably detect anything at 80% power, α=0.05
+(`errorbars power --n 500 --baseline 0.5` → minimum detectable effect ≈ 8.9 points). Evan Miller's
+"[Adding Error Bars to Evals: A Statistical Approach to Language Model
+Evaluations](https://arxiv.org/abs/2411.00640)" (arXiv:2411.00640, 2024) lays out the right
+practice — CLT and clustered standard errors, variance reduction through pairing, power analysis —
+and this package turns it into a one-liner.
+
+![errorbars leaderboard on a synthetic clustered benchmark](https://raw.githubusercontent.com/antonsoo/errorbars/main/docs/assets/leaderboard-terminal.png)
+
+## Why this exists
+
+Run `errorbars leaderboard` on a synthetic benchmark (below) and one apparent 7-point win —
+`tuned-70b` over `baseline-70b`, 0.690 vs. 0.620 — is not statistically significant once it has a
+proper error bar: the paired test already gives p = 0.11. Accounting for the fact that questions
+come 5-to-a-passage, and Holm-correcting across all 6 pairwise comparisons on the board, pushes
+that to p = 0.32. A naive leaderboard that ranks by bare accuracy would still have shipped the gap
+as a win. The full walkthrough, with every number copied from a real command, is in
+[`examples/README.md`](https://github.com/antonsoo/errorbars/blob/main/examples/README.md).
+
+## Quickstart
+
+```bash
+pip install "errorbars[cli]"
+errorbars power --delta 0.03 --baseline 0.5
+```
+
+```
+Questions needed: 4361
+```
+
+That's how many questions you'd need to reliably detect a 3-point accuracy gap at a 50% baseline,
+with the defaults (80% power, α=0.05, no pairing correlation, no clustering). Clone the repo to
+try it against real per-item scores:
+
+```bash
+git clone https://github.com/antonsoo/errorbars && cd errorbars
+errorbars leaderboard examples/data/reading_comprehension.csv
+```
+
+## Features
+
+- **`summarize`** — mean, SE, and 95% CI (CLT by default, Wilson for small-n binary scores,
+  bootstrap on request); clustered SE with design effect and ICC when a `cluster_id` column is
+  present; within/between-question variance decomposition when a `sample` column is present.
+- **`compare`** — paired mean difference, SE, CI, p-value; the correlation between the two models'
+  per-question scores and how much pairing shrank the SE vs. an unpaired comparison; a
+  cluster-robust paired SE/p-value when clusters are present; exact McNemar test for binary scores.
+- **`leaderboard`** — every model with its CI, Holm-corrected pairwise paired tests, and groups of
+  statistically indistinguishable models (maximal cliques of the "not significantly different"
+  graph); a forest plot (SVG, no dependency; matplotlib if installed).
+- **`power`** — number of questions needed to detect an effect δ at a given α and power, or the
+  minimum detectable effect for a given n, accounting for pairing correlation, repeated sampling,
+  and cluster design effect.
+- **CLI** — `errorbars summarize|compare|leaderboard|power|import`, `rich` tables by default,
+  `--json` for scripting.
+- **Adapters** — `errorbars import lm-eval|inspect` converts lm-evaluation-harness `--log_samples`
+  output or an Inspect AI `.eval` log into the canonical format (see below).
+- **Web calculator** — a static "how many eval questions do I need?" power calculator
+  ([live demo](https://antonsoo.github.io/errorbars/)), whose TypeScript formulas are checked
+  against the Python ones by the test suite.
+- Typed Python ≥3.10, `numpy` the only runtime dependency; `pandas` and `matplotlib` are optional
+  extras; `scipy`/`statsmodels` are test-only oracles, never imported at runtime.
+
+## Usage
+
+```bash
+errorbars summarize examples/data/reading_comprehension.csv --model tuned-70b
+```
+
+```
+    summarize: tuned-70b
+┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric ┃            value ┃
+┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n      │              200 │
+│ mean   │           0.6900 │
+│ SE     │           0.0328 │
+│ 95% CI │ [0.6257, 0.7543] │
+│ method │              clt │
+└────────┴──────────────────┘
+       clustering diagnostics
+┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ metric        ┃            value ┃
+┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ n clusters    │               40 │
+│ ICC           │           0.0710 │
+│ design effect │            1.284 │
+│ clustered SE  │           0.0372 │
+│ clustered CI  │ [0.6148, 0.7652] │
+└───────────────┴──────────────────┘
+```
+
+```bash
+errorbars compare examples/data/reading_comprehension.csv \
+  --model-a tuned-70b --model-b baseline-70b
+```
+
+```
+mean diff (A - B)                 0.0700
+paired SE                         0.0440
+95% CI                  [-0.0167, 0.1567]
+p-value                           0.1131
+correlation(A, B)                 0.1434
+variance reduction from pairing     14.3%
+McNemar exact p-value             0.1405
+```
+
+```bash
+errorbars power --delta 0.05 --baseline 0.65 --rho 0.14 --cluster-deff 1.28
+```
+
+```
+Questions needed: 1573
+```
+
+Every number above is copied verbatim from running these commands against
+`examples/data/reading_comprehension.csv` (synthetic — see below). The full leaderboard output and
+the reasoning behind each step are in [`examples/README.md`](https://github.com/antonsoo/errorbars/blob/main/examples/README.md).
+
+### Input format
+
+Long-format CSV or JSONL, one row per observation:
+
+| question_id | cluster_id (optional) | model | score | sample (optional) |
+|---|---|---|---|---|
+| q1 | passage-003 | tuned-70b | 1 | |
+
+Column names are configurable (`--question-col`, `--cluster-col`, etc., or `ColumnMap` in Python).
+Scores can be binary (0/1) or continuous. `cluster_id` groups correlated questions (e.g. several
+questions per reading passage); `sample` marks repeated generations of the same question.
+
+### Importing from lm-evaluation-harness or Inspect AI
+
+`errorbars import` converts either tool's own log format into the canonical CSV above:
+
+```bash
+# lm-evaluation-harness: lm_eval run --model <...> --tasks <...> --log_samples --output_path <dir>
+errorbars import lm-eval runs/<dir>/samples_<task>_<timestamp>.jsonl \
+  --model my-model-name -o converted.csv
+
+# Inspect AI: inspect eval <task> --model <...>  (writes a .eval log by default)
+errorbars import inspect logs/<run>.eval -o converted.csv
+```
+
+`--metric` (lm-eval) picks which computed metric to use as the score when a task reports more than
+one (e.g. `acc` vs. `acc_norm`); it defaults to the first one. `--scorer` (Inspect) does the same
+for tasks with multiple scorers. Inspect epochs (`--epochs N`, repeated sampling of the same input)
+land in the `sample` column automatically.
+
+Both adapters were built and tested against real, unedited output — not from memory: **lm-eval
+0.4.13** (`lm_eval run --model dummy --tasks copa --limit 20 --log_samples ...`, plus a
+multi-metric `arc_easy` run) and **inspect-ai 0.3.268** (a 5-sample task through the built-in
+`mockllm/model` provider). The exact log files are committed as test fixtures
+(`tests/fixtures/samples_*.jsonl`, `tests/fixtures/inspect_tiny_qa.eval`) and re-parsed in
+`tests/test_adapters.py` on every run. The Inspect adapter reads logs with Inspect's own
+`inspect_ai.log.read_eval_log` and `inspect_ai.scorer.value_to_float` rather than hand-parsing its
+binary `.eval` format, and needs the `inspect` extra:
+`pip install "errorbars[inspect]"`. The lm-eval
+adapter has no extra dependency — `--log_samples` is already plain JSONL.
+
+## How it works
+
+Every statistic is implemented from scratch on `numpy` + the standard library
+(`statistics.NormalDist` for normal quantiles; Student-t tails from a continued-fraction incomplete
+beta function) — no scipy or statsmodels at runtime.
+Full derivations with references are in [`docs/formulas.md`](https://github.com/antonsoo/errorbars/blob/main/docs/formulas.md):
+
+1. CLT and Wilson confidence intervals for a mean
+2. Cluster-robust standard errors (the CR1 sandwich estimator, matching
+   `statsmodels`' `cov_type="cluster"`)
+3. Intraclass correlation and Kish's design effect
+4. Within/between-question variance decomposition for repeated sampling
+5. Paired comparisons, variance reduction from pairing, and exact McNemar
+6. Holm-Bonferroni correction and maximal-clique grouping for leaderboards
+7. Power analysis (sample size and minimum detectable effect) under pairing and clustering
+
+## Accuracy and limitations
+
+- Cluster-robust SE matches `statsmodels`' `OLS(..., cov_type="cluster")` to 1e-9 on both balanced
+  and unbalanced cluster sizes; Wilson intervals match `statsmodels.stats.proportion_confint` to
+  1e-9; the paired t-test matches `scipy.stats.ttest_rel` to 1e-7 at any n, and McNemar's exact
+  test is cross-checked against `statsmodels`.
+  See `tests/test_stats_vs_oracles.py` and `tests/test_compare_vs_oracles.py`.
+- Monte Carlo coverage tests (`tests/test_coverage_montecarlo.py`) confirm nominal 95% CIs cover
+  the true parameter close to 95% of the time — for CLT, Wilson, bootstrap, and cluster-robust
+  intervals — with a tolerance sized to the trial count so it won't flake.
+- Paired comparisons use Student's t with n − 1 degrees of freedom, like `scipy.stats.ttest_rel`,
+  and clustered SEs use t with G − 1 for G clusters (Cameron & Miller 2015), computed without
+  scipy. With few clusters that interval is honest but wide; the per-model `clt` interval stays
+  normal-based, which is only right once n is in the dozens.
+- The power formula's samples-per-question adjustment assumes all single-sample variance is
+  decoding noise (see `docs/formulas.md` §11 for why, and the caveat on when this is optimistic).
+  It's a planning tool for before you run the eval; for a post-hoc measurement with the true
+  within/between-question split, use `summarize` on data with a `sample` column instead.
+- Leaderboard groups are maximal cliques of the "not significantly different" graph, which is the
+  statistically direct approach — it can produce a model in more than one group, unlike a
+  minimal-letters heuristic (e.g. R's `multcompView`).
+- 168 Python tests, 605 TypeScript tests (cross-checking the JS formulas against Python-generated
+  vectors), all passing on this box (14 vCPU WSL2 Linux, 48 GB RAM; Python 3.12.3, Node 26.7.0)
+  as of 2026-10-01.
+
+## Web calculator
+
+[`web/`](https://github.com/antonsoo/errorbars/blob/main/web/) is a static Vite + TypeScript "how many eval questions do I need?" calculator
+([live demo](https://antonsoo.github.io/errorbars/)). Its power formulas
+(`web/src/power.ts`, `web/src/normal.ts`) are a direct port of `src/errorbars/power.py`; Python
+generates JSON test vectors (`scripts/export_test_vectors.py` → `web/test-vectors.json`) that the
+TypeScript test suite checks against, so the two implementations can't silently drift apart.
+
+![Power calculator](https://raw.githubusercontent.com/antonsoo/errorbars/main/docs/assets/calculator-hero.png)
+
+## Development
+
+```bash
+uv sync --group dev --extra all
+uv run pytest
+uv run ruff check .
+uv run mypy src/errorbars
+```
+
+See [`CONTRIBUTING.md`](https://github.com/antonsoo/errorbars/blob/main/CONTRIBUTING.md) for the web calculator's dev loop and guidelines.
+
+## Contributing
+
+Issues and PRs welcome. Any new statistic needs a test against an independent oracle (reference
+library, closed form, or Monte Carlo simulation) — see `tests/` for the pattern.
+
+## Citations
+
+```bibtex
+@misc{miller2024addingerrorbarsevals,
+  title  = {Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations},
+  author = {Evan Miller},
+  year   = {2024},
+  eprint = {2411.00640},
+  archivePrefix = {arXiv},
+  primaryClass  = {stat.AP},
+  url    = {https://arxiv.org/abs/2411.00640}
+}
+```
+
+## License
+
+[MIT](https://github.com/antonsoo/errorbars/blob/main/LICENSE) © 2026 Anton Soloviev
+
+---
+
+<sub>Part of [Officina](https://antonsoo.github.io/officina/), a set of small open-source tools by [Anton Soloviev](https://github.com/antonsoo).</sub>

errorbars-0.1.2.dist-info/RECORD

@@ -0,0 +1,17 @@
+errorbars/__init__.py,sha256=CRp6GSNgD1xpOmJLHMrAxylNPVnKH6EmhTmfAME_Ol0,547
+errorbars/cli.py,sha256=ubQF9prXpRBAq02cgL1IXP1GJBzDm8BsrvtzSAQz5gY,17448
+errorbars/compare.py,sha256=OZ8clRwaL6ek6KTBRN57_Q3utoIEMKULCKsY97dV-p4,6602
+errorbars/io.py,sha256=a7R4HDQqh5jVkXFLuy8EP5nG-cnCTh1AYxxjE0Xwub0,7615
+errorbars/leaderboard.py,sha256=K19pskSFBqMqr03yC1BIof1EVjg5xDeBhCtCGTsQK3k,6492
+errorbars/plot.py,sha256=NoWT40_OfzdveHLqEw8CPV8HGUcGB98eS3HgA2GA48M,5041
+errorbars/power.py,sha256=1GNpBxR5VpW6xGQLEhIppepjowxp2rQi9aLiqH4m7tM,5429
+errorbars/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+errorbars/stats.py,sha256=lo0x4TY0Y2UCjLO-3i3Q5WFEZnd-LauDLhpXy763YSM,11807
+errorbars/adapters/__init__.py,sha256=XhWjHhf5OQrODBysnjKDAu_gjX_gY9qCVc6m0B5j1Z8,77
+errorbars/adapters/inspect_ai.py,sha256=5_IACOHXvpF5sJjxj7SzSLlKcBUaFmJskRoGoaU4UvQ,3189
+errorbars/adapters/lm_eval.py,sha256=BOrt30jp5TkiVMRifw9x3I2kkX9XzsXoDTJFhkLQE9o,4023
+errorbars-0.1.2.dist-info/METADATA,sha256=FwiEtauE3MkQuEYwjiTVEOzv4Vsg1wfSdeurfaA38yc,14774
+errorbars-0.1.2.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+errorbars-0.1.2.dist-info/entry_points.txt,sha256=4tLng6AuvkO_nJeSf4KDkgsGXsEqQcZGkIcjkdmG45E,49
+errorbars-0.1.2.dist-info/licenses/LICENSE,sha256=4mFEvoZz1pwXUSatjc2e14QxRk8LlxILeZ3HXsfttkc,1071
+errorbars-0.1.2.dist-info/RECORD,,

errorbars-0.1.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

errorbars-0.1.2.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+errorbars = errorbars.cli:main

errorbars-0.1.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anton Soloviev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.