evalstats 0.2.4__tar.gz → 0.2.6__tar.gz

evalstats-0.2.6/PKG-INFO

@@ -0,0 +1,434 @@
+Metadata-Version: 2.4
+Name: evalstats
+Version: 0.2.6
+Summary: Statistically sane analysis methods for comparing AI model and prompt performance.
+Author: Ian Arawjo
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ianarawjo/evalstats
+Project-URL: Repository, https://github.com/ianarawjo/evalstats
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Requires-Dist: scipy>=1.9
+Requires-Dist: pandas>=1.5
+Requires-Dist: matplotlib>=3.6
+Requires-Dist: scikit-posthocs
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: plotext
+Provides-Extra: interactive
+Requires-Dist: plotly>=5.0; extra == "interactive"
+Provides-Extra: models
+Requires-Dist: choix>=0.3; extra == "models"
+Provides-Extra: lmm
+Requires-Dist: pymer4>=0.9; extra == "lmm"
+Requires-Dist: pyarrow>=14.0; extra == "lmm"
+Requires-Dist: great_tables; extra == "lmm"
+Requires-Dist: joblib; extra == "lmm"
+Requires-Dist: rpy2; extra == "lmm"
+Requires-Dist: polars; extra == "lmm"
+Requires-Dist: scikit-learn; extra == "lmm"
+Requires-Dist: formulae; extra == "lmm"
+Provides-Extra: xlsx
+Requires-Dist: openpyxl>=3.0; extra == "xlsx"
+Provides-Extra: all
+Requires-Dist: plotly>=5.0; extra == "all"
+Requires-Dist: choix>=0.3; extra == "all"
+Requires-Dist: openpyxl>=3.0; extra == "all"
+Requires-Dist: pymer4>=0.9; extra == "all"
+Requires-Dist: pyarrow>=14.0; extra == "all"
+Requires-Dist: great_tables; extra == "all"
+Requires-Dist: joblib; extra == "all"
+Requires-Dist: rpy2; extra == "all"
+Requires-Dist: polars; extra == "all"
+Requires-Dist: scikit-learn; extra == "all"
+Requires-Dist: formulae; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# evalstats
+
+[![PyPI](https://img.shields.io/pypi/v/evalstats)](https://pypi.org/project/evalstats/)
+[![License](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](pyproject.toml)
+
+Rigorous statistical analysis for LLM evaluations: from model and prompt comparisons to statistical tests resilient to LLM judge bias, including in small-sample data regimes.
+
+`evalstats` helps you answer questions like:
+- Is Prompt A actually better than Prompt B, or just slightly luckier on this dataset?
+- Does Model A beat Model B, or only under a specific prompt phrasing?
+- Are my performance differences large enough to be meaningful, or just noise?
+- How stable are scores across runs, evaluators, or inputs?
+- Can I trust my LLM-judge scores, or do they need correcting against human labels first?
+
+You give `evalstats` your benchmark data, and it runs statistically appropriate analyses that quantify uncertainty and provide confidence bounds on your claims, in two main ways:
+
+- **Comparisons**: compare models, prompts, or both at once, and get 95% confidence intervals, pairwise significance tests, and multi-run sensitivity analyses. `evalstats` picks well-calibrated methods by default, backed by simulations, and was built specifically for small-sample datasets (N<100); it will output stats down to 15 samples. See [Recommended Methods](#recommended-methods).
+- **PPI-corrected inference**: use a small set of human labels to correct bias in noisy LLM-judge scores, so your means, confidence intervals, and hypothesis-test p-values stay calibrated in the face of LLM judge bias. See [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests).
+
+Scientists can use our PPI-corrected statistical tests for **mixed human-AI subject studies**, where some observations are human-labeled and the rest are graded by an LLM judge. `evalstats.tests` gives you LLM-judge-bias-corrected versions of:
+
+- t-test (`ttest`, independent or paired; Welch's by default, or Student's equal-variance via `equal_var=True`)
+- Mann–Whitney U (`mannwhitney`)
+- Wilcoxon signed-rank (`wilcoxon`)
+- One-way ANOVA (`anova_oneway`, independent or repeated-measures)
+- Friedman test (`friedman`, repeated-measures rank-based)
+- Kruskal-Wallis (`kruskalwallis`, independent-groups rank-based)
+
+As long as items for human labeling were sampled at random from the full dataset, p-values stay calibrated even when the LLM judge is biased or miscalibrated, validated via extensive Monte Carlo simulations (see [`simulations/harness`](simulations/harness)). To the best of our knowledge, `evalstats` provides the only known implementations of PPI-corrected rank-based nonparametric tests like Wilcoxon.
+
+> [!IMPORTANT]
+> We are actively building out this project. A paper with the full methodology and simulation-backed validation behind every default is forthcoming: see [Recommended Methods](#recommended-methods) and [Citation](#citation). In the meantime, the [Stats for LLM Evals guide](https://statsforevals.com/) covers the same material in web form. If there's something you'd like to see, let us know by raising an Issue.
+
+## Contents
+
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [See it in action](#see-it-in-action)
+- [Recommended Methods](#recommended-methods)
+- [Python API](#python-api)
+- [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests)
+- [CLI Reference](#cli-reference)
+- [Examples](#examples)
+- [Mixed effects models (LMM)](#mixed-effects-models-lmm)
+- [Reproducibility: Monte Carlo simulations](#reproducibility-monte-carlo-simulations)
+- [Motivation](#motivation)
+- [Development and Contributions](#development-and-contributions)
+- [Citation](#citation)
+- [License](#license)
+
+## Installation
+
+```bash
+pip install evalstats
+```
+
+For Excel (`.xlsx`) input support: `pip install "evalstats[xlsx]"`. For every optional extra (including mixed-effects/LMM support): `pip install "evalstats[all]"`.
+
+## Quick start
+
+From the command line, `evalstats` can read a CSV or Excel file directly and print a statistical summary:
+
+```bash
+evalstats analyze results.csv
+```
+
+The input file should have a prompt/template column, an item/input column, and a score column (model, run, and evaluator columns are optional). See the [column alias table](#python-api) below, or run `evalstats analyze --help` for the full option and alias list.
+
+From Python, the main entry point is `load_from()` + `compare()`:
+
+```python
+import pandas as pd
+import evalstats as es
+
+df = pd.read_csv("results.csv")  # columns: prompt, item, score (model optional)
+
+evaldata = es.load_from(df)
+evaldata.summary()  # inspect detected structure/column assignments before analyzing
+
+result = es.compare(evaldata, factors="prompt")
+result.summary()  # full terminal report: CIs, pairwise tests, rank probabilities
+```
+
+See [Python API](#python-api) for the full data-format and `compare()` reference.
+
+## See it in action
+
+Running `es.compare(evaldata, factors="prompt")` then `result.summary()` prints a full statistical report to the terminal: confidence interval line plots, pairwise comparisons, and per-input stability across runs. Below: a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators).
+
+![Example terminal output](docs/example-output.png)
+
+From this we can see Minimal and Instructive are the most promising candidates, but it's statistically unclear which is better. Chain-of-thought gives the least consistent outputs across runs.
+
+Comparing models and prompts at once, `evalstats` colors in a 4-way tie between four model-prompt combinations:
+
+![Example terminal output with colors](docs/terminal-output-example.jpg)
+
+You can also plot within notebook environments. `plot_point_estimates` shows each template's absolute mean score with marginal confidence intervals:
+
+![Mean advantage plot](docs/mean_advantage.png)
+
+And LLMs are stochastic at temperature > 0: the "noise plot" visualizes (in)stability across runs for the same input:
+
+![Per-input noise across runs](docs/per-input-noise.png)
+
+## Recommended Methods
+
+`evalstats.compare()` defaults to `method="auto"`, which picks a well-calibrated statistical method based on your data's estimand, data type, and sample size. These defaults come from an extensive Monte Carlo simulation study across eval data types, sample sizes, and comparison setups, cross-checked against real LLM eval data, summarized in the two decision trees below. **Boxed methods are the default; gray notes give the multi-run variant and conservative alternatives.**
+
+![Decision tree for selecting a 95% confidence interval method](docs/decision-tree-ci.png)
+
+![Decision tree for selecting a p-value method or FWER correction](docs/decision-tree-pvalue.png)
+
+A paper with the full methodology, simulation results, and justification behind each recommendation is forthcoming (see [Citation](#citation)). Until then, see the [Which Method?](https://statsforevals.com/which-method.html) page on the `evalstats` site for the web version of these trees, and [`simulations/harness`](simulations/harness) to reproduce the underlying simulations yourself.
+
+## Python API
+
+`evalstats` expects **long-format** data: one row per (item, score) observation, plus whichever axis you want to compare (`model`, `prompt`, or both) and optionally `run` for repeated runs. Only `item` and `score` are strictly required; you need at least one of `model`/`prompt` too, whichever you pass to `compare(factors=...)`. `load_from()` auto-detects each column's role by matching its name (case-insensitively) against this table:
+
+| Role     | Canonical name | Recognized aliases                    | Required?                                          |
+|----------|-----------------|----------------------------------------|------------------------------------------------------|
+| model    | `model`         | `model_label`, `model_name`            | Optional: needed to compare models (`factors="model"`) |
+| prompt   | `prompt`        | `template`, `prompt_template`          | Optional: needed to compare prompts (`factors="prompt"`) |
+| item     | `item`          | `input`, `example`, `id`, `input_label`| Yes                                                  |
+| score    | `score`         | `value`, `result`, `metric`            | Yes                                                  |
+| run      | `run`           | `seed`, `repeat`, `run_id`, `trial`    | Optional: add if you have repeated runs per (model/prompt, item) |
+
+For example, a minimal CSV comparing prompts:
+
+| prompt      | item | score |
+|-------------|------|-------|
+| Minimal     | q1   | 0.82  |
+| Instructive | q1   | 0.91  |
+| Minimal     | q2   | 0.75  |
+| Instructive | q2   | 0.88  |
+
+If your columns don't match any alias above, remap them explicitly: `es.load_from(df, col_map={"llm": "model", "variant": "prompt", "q_id": "item"})`.
+
+`compare()` also handles:
+
+- **Comparing models**: `factors="model"`
+- **Factorial designs** (model × prompt): `factors=["model", "prompt"]` (routes to an LMM backend)
+- **Filtering**: any keyword matching a column name acts as a row filter, e.g. `es.compare(evaldata, factors="model", split="test")`
+- **PPI-corrected inference** for noisy LLM-judge scores: see [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests)
+
+The returned `result` is a `ComparisonResult`. Besides `.summary()`, it has `.to_frame()` / `.to_dict()` for programmatic access, `.plot(method="forest" | "bar" | "cd" | "pareto")` for charts, and `.disagreements()` to surface the items entities disagree on most.
+
+<details>
+<summary><strong>Advanced: raw score arrays (low-level engine)</strong></summary>
+
+`compare()` is a wrapper around a lower-level engine, `analyze()`, which operates directly on `BenchmarkResult` / `MultiModelBenchmark` objects (numpy score arrays) rather than a DataFrame. Reach for this path only if you already have scores as arrays and don't want to build a DataFrame first. Most use cases should use `compare()` above.
+
+```python
+import numpy as np
+import evalstats as estats
+
+# Example raw scores for 4 templates × 3 inputs (single run, single evaluator)
+your_scores = [
+    [0.91, 0.88, 0.86],
+    [0.90, 0.89, 0.84],
+    [0.85, 0.82, 0.80],
+    [0.79, 0.76, 0.74],
+]
+n_templates, n_inputs = 4, 3
+
+# scores shape: (n_templates, n_inputs, n_runs, n_evaluators)
+scores = np.array(your_scores).reshape(n_templates, n_inputs, 1, 1)
+
+result = estats.BenchmarkResult(
+    scores=scores,
+    template_labels=["Minimal", "Instructive", "Few-shot", "Chain-of-thought"],
+    input_labels=[f"input_{i}" for i in range(n_inputs)],
+)
+
+analysis = estats.analyze(result, reference="grand_mean", n_bootstrap=5_000)
+analysis.summary()  # same terminal report as ComparisonResult.summary()
+```
+
+If you want this lower-level path from a DataFrame (e.g. to inspect the raw `BenchmarkResult` object, or fine-tune `strict_complete_design`), use `from_dataframe()` instead of `load_from()`. It returns the array-based `BenchmarkResult` / `MultiModelBenchmark` plus an optional `DataLoadReport`, a data-quality log of coercions/repairs made while parsing:
+
+```python
+import evalstats as estats
+
+benchmark, load_report = estats.from_dataframe(
+    df, format="auto", repair=True, strict_complete_design=True, return_report=True,
+)
+for line in load_report.to_lines():
+    print(line)
+
+analysis = estats.analyze(benchmark)
+analysis.summary()
+```
+
+To visualize absolute prompt performance directly from a `BenchmarkResult`, bypassing `analyze()`:
+
+```python
+fig = estats.plot_point_estimates(result)
+fig.savefig("mean_performance.png", dpi=150, bbox_inches="tight")
+```
+
+</details>
+
+## PPI-Corrected Inference (Means, CIs, and Tests)
+
+PPI (Prediction-Powered Inference) lets you use lots of cheap LLM judgments plus a smaller set of human labels to correct measurement error from the LLM judge, giving you corrected estimates and uncertainty that better reflect what you'd have gotten from a fully human-labeled study (Angelopoulos et al., 2023). Most corrections use PPIBoot (bootstrap variant of PPI; Zrnic, 2024), battle-tested via simulations (see [`simulations/sim_type_i_calibration.py`](simulations/sim_type_i_calibration.py)).
+
+> [!IMPORTANT]
+> **Which items get a human label must be chosen uniformly at random.** PPI correction assumes the labeled subset is representative of the full dataset. If your labeling process instead targets specific items — e.g. "always double-check the borderline or highest-scoring responses" — that's missing-not-at-random (MNAR) selection on the outcome, and PPI correction can stay badly miscalibrated **no matter how many items you label**. This isn't ordinary small-sample noise that more labels fixes; confirmed in simulation to persist from 15 up through 300 labeled items out of 400. See `evalstats.ppi.correct`'s docstring for the full analysis, and use `evalstats label` (below) to draw a compliant random sample.
+
+### Comparing models with corrected LLM judge evals via `compare(..., alignment=...)`
+
+```python
+import evalstats as es
+
+# Dataframe columns include: model  item  llm_score  human_score (NaN for unlabeled rows)
+evaldata = es.load_from(df)
+
+alignment = es.judge_alignment(evaldata, llm_metric="llm_score", human_groundtruth="human_score")
+
+result = es.compare(
+    evaldata, factors="model", metric="llm_score", alignment={"llm_score": alignment},
+)
+result.summary()
+```
+
+### T-test PPI-correction via `evalstats.tests.ttest`
+
+```python
+import evalstats as es
+
+res = es.tests.ttest(
+    a=llm_a, b=llm_b,
+    a_lab=human_a,  # same length as llm_a, NaN where unlabeled
+    b_lab=human_b,  # same length as llm_b, NaN where unlabeled
+    paired=False, print_result=False,
+)
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+<details>
+<summary><strong>More PPI-corrected tests: Mann-Whitney U, Wilcoxon, one-way ANOVA</strong></summary>
+
+**Mann-Whitney U** (`evalstats.tests.mannwhitney`): nonparametric two-group comparison based on relative ranks rather than assuming normally distributed scores:
+
+```python
+res = es.tests.mannwhitney(x=llm_x, y=llm_y, x_lab=human_x, y_lab=human_y, print_result=False)
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+**Wilcoxon signed-rank** (`evalstats.tests.wilcoxon`): nonparametric paired test for matched observations (before/after, A/B on the same items, etc.):
+
+```python
+res = es.tests.wilcoxon(x=llm_before, y=llm_after, x_lab=human_before, y_lab=human_after, print_result=False)
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+**One-way ANOVA** (`evalstats.tests.anova_oneway`): more than two groups; pass `repeated=True` for repeated-measures (same subjects across conditions):
+
+```python
+res = es.tests.anova_oneway(
+    llm_g1, llm_g2, llm_g3,
+    groups_lab=[human_g1, human_g2, human_g3], repeated=False, print_result=False,
+)
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+</details>
+
+## CLI Reference
+
+```bash
+evalstats analyze results.csv       # full statistical report from a CSV/XLSX file
+evalstats label results.csv         # draw a random, MCAR-compliant sample of items for human labeling
+```
+
+`evalstats label` picks a uniformly random sample of items per condition (respecting the PPI sample-size floors: 15 minimum, 30 recommended) and writes a CSV/XLSX with a `human_<metric>` column ready for grading, the safe way to build the labeled subset `alignment=`/`*_lab` needs above. Run `evalstats label --help` for the full option list.
+
+## Examples
+
+`examples/` has 25+ runnable, self-contained scripts covering common workflows: synthetic and OpenAI-backed benchmarks, multi-run comparisons, PPI-corrected judge alignment, factorial designs, and reliability/robustness demos. From the repository root:
+
+```bash
+python examples/synthetic_mean_advantage.py      # no API key needed
+python examples/sentiment.py                     # OpenAI sentiment benchmark
+python examples/sentiment_multirun.py             # captures run-to-run variability
+python examples/compare_models_multirun.py        # multi-model comparison across prompts
+python examples/compare_alignment_ppi.py           # PPI-corrected judge comparison
+```
+
+OpenAI-powered examples require `OPENAI_API_KEY` set in your environment, but the model calls are easy to swap for whichever provider you prefer.
+
+## Mixed effects models (LMM)
+
+> [!IMPORTANT]
+> Mixed effects analysis is experimental, currently offering only graceful handling of missing data (assumed reasonably random). Use `method="lmm"` if you need robustness to missing (`NaN`) cells; factor decomposition across multiple input factors is planned.
+
+`evalstats` supports mixed-effects models (`score ~ template + (1|input)`) for missing data and multi-factor decomposition. The default backend is pure-Python `statsmodels`, no extra setup required:
+
+```python
+analysis = estats.analyze(result, method="lmm")
+```
+
+This fits the model with REML, computes Wald CIs via the delta method, and estimates rank distributions by parametric simulation.
+
+<details>
+<summary><strong>Optional backend: pymer4 (requires R)</strong></summary>
+
+For Satterthwaite degrees of freedom and `emmeans`-based pairwise contrasts (R's gold standard for mixed models), pass `backend="pymer4"`:
+
+```python
+analysis = estats.analyze(result, method="lmm", backend="pymer4")
+```
+
+This requires a working R installation with:
+
+```r
+install.packages(c("lme4", "emmeans", "tibble", "broom", "broom.mixed", "lmerTest", "report", "car"))
+```
+
+Then `pip install "evalstats[lmm]"`. If your environment needs manual dependency pinning, this is the tested equivalent:
+
+```bash
+pip install "pymer4>=0.9" great_tables joblib rpy2 polars scikit-learn formulae pyarrow
+```
+
+Installation details may differ on your system.
+
+</details>
+
+## Reproducibility: Monte Carlo simulations
+
+Claims in this README and on the [`evalstats` site](https://statsforevals.com/) like "verified in our simulations" are backed by a runnable harness in [`simulations/harness/`](simulations/harness):
+
+```bash
+python -m simulations.harness.cli --list-cases
+python -m simulations.harness.cli --official-tests
+python -m simulations.harness.cli ci_single --reps 50 --sizes 10 20
+python -m simulations.harness.cli pvalues --mode ppi --tests ttest wilcoxon anova_rep
+```
+
+`--official-tests` brings up a CLI to run each case's canonical, full-scale preset, writing results plus a `manifest.json` (args, output paths, key metrics, pass/fail) to `simulations/out/official_<timestamp>/`. See [`simulations/harness/README.md`](simulations/harness/README.md) for the full case list and scenario library. Note that each simulation can take a long time to run. Even parallelized across many cores, official-scale runs can take hours.
+
+- `ci_single` / `ci_paired`: coverage and width of confidence interval methods across synthetic distributions and real benchmark data (OpenEval, Inspect AI).
+- `pvalues --mode pairwise` / `--mode multiarm`: Type-I error and power for pairwise and multi-arm comparisons, including FWER correction strategies.
+- `pvalues --mode ppi`: Type-I error calibration and power for every PPI-corrected test in `evalstats.tests`, swept across judge-bias severity, label fraction, and MNAR-labeling scenarios.
+
+## Motivation
+
+Most eval tools in the LLM evaluation space don't help users perform *any* statistical tests. They present bar charts of average performance, and developers glance at the chart and decide "prompt/model A is better than B." But was it really? Relying on bar charts and averages alone can easily lead to erroneous conclusions: B might be more robust than A, or perform better on an important data subset, or there might not be enough data to conclude either way.
+
+People do evals this way because they don't have the time, tools, or statistical knowledge to do better. Often they don't even know there's a better way. `evalstats` aims to rectify this with simple, powerful defaults: throw us your data, and we'll run the stats and plot the results for you.
+
+## Development and Contributions
+
+For package build, release validation, and maintainer workflows, see [DEVELOPMENT.md](DEVELOPMENT.md).
+
+We welcome contributions, especially refinements to our statistical methods. If you're proposing a new correction, CI method, or a fix to an existing one, battle-test it against the [simulation harness](#reproducibility-monte-carlo-simulations) first. Add or extend a scenario and confirm your change holds up on Type-I error and power, not just on the case that motivated it, before opening a PR.
+
+## Citation
+
+`evalstats` doesn't have a paper yet. One covering the full simulation-backed method validation is forthcoming. Until then, please cite the GitHub repository:
+
+```bibtex
+@software{arawjo_evalstats,
+  author  = {Arawjo, Ian},
+  title   = {evalstats: Statistically Sound Analysis for LLM Evaluations},
+  url     = {https://github.com/ianarawjo/evalstats},
+  year    = {2026}
+}
+```
+
+or in prose: Ian Arawjo, *evalstats* (GitHub: [ianarawjo/evalstats](https://github.com/ianarawjo/evalstats)).
+
+## License
+
+This repository uses two licenses:
+
+- **`evalstats` package** (everything outside `website/`): [MIT](LICENSE).
+- **Stats for Evals Website** (everything in `website/`): [CC BY-NC-ND 4.0](website/LICENSE). You may share it with attribution non-commercially, but commercial use and derivative works are not permitted.