evalstats 0.2.0__tar.gz → 0.2.4__tar.gz

evalstats-0.2.4/PKG-INFO

@@ -0,0 +1,548 @@
+Metadata-Version: 2.4
+Name: evalstats
+Version: 0.2.4
+Summary: Statistically sane analysis methods for comparing AI model and prompt performance.
+Author: Ian Arawjo
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ianarawjo/evalstats
+Project-URL: Repository, https://github.com/ianarawjo/evalstats
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Requires-Dist: scipy>=1.9
+Requires-Dist: pandas>=1.5
+Requires-Dist: matplotlib>=3.6
+Requires-Dist: scikit-posthocs
+Requires-Dist: statsmodels>=0.14
+Requires-Dist: plotext
+Provides-Extra: interactive
+Requires-Dist: plotly>=5.0; extra == "interactive"
+Provides-Extra: models
+Requires-Dist: choix>=0.3; extra == "models"
+Provides-Extra: lmm
+Requires-Dist: pymer4>=0.9; extra == "lmm"
+Requires-Dist: pyarrow>=14.0; extra == "lmm"
+Requires-Dist: great_tables; extra == "lmm"
+Requires-Dist: joblib; extra == "lmm"
+Requires-Dist: rpy2; extra == "lmm"
+Requires-Dist: polars; extra == "lmm"
+Requires-Dist: scikit-learn; extra == "lmm"
+Requires-Dist: formulae; extra == "lmm"
+Provides-Extra: xlsx
+Requires-Dist: openpyxl>=3.0; extra == "xlsx"
+Provides-Extra: all
+Requires-Dist: plotly>=5.0; extra == "all"
+Requires-Dist: choix>=0.3; extra == "all"
+Requires-Dist: openpyxl>=3.0; extra == "all"
+Requires-Dist: pymer4>=0.9; extra == "all"
+Requires-Dist: pyarrow>=14.0; extra == "all"
+Requires-Dist: great_tables; extra == "all"
+Requires-Dist: joblib; extra == "all"
+Requires-Dist: rpy2; extra == "all"
+Requires-Dist: polars; extra == "all"
+Requires-Dist: scikit-learn; extra == "all"
+Requires-Dist: formulae; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# evalstats
+
+Rigorous statistical analysis for LLM evaluations, from model and prompt comparisons to statistical tests resilient to LLM judge bias, including in small sample data regimes.
+
+`evalstats` helps you answer questions like:
+ - Is Prompt A actually better than Prompt B, or just slightly luckier on this dataset?
+ - Does Model A beat Model B, or only under a specific prompt phrasing?
+ - How sensitive is model performance to prompt wording?
+ - Are my performance differences large enough to be meaningful, or just noise?
+ - How stable are scores across runs, evaluators, or inputs?
+ - Can I trust my LLM-judge scores, or do they need correcting against human labels first?
+
+You give `evalstats` your benchmark data, and it runs statistically appropriate analyses that quantify uncertainty and provide confidence bounds on your claims. It does this in two main ways:
+
+- **Comparisons**: Comparing models, prompts, or both at once (or any other thing you're comparing, like agent harnesses), and get 95% confidence intervals, pairwise significance tests, and multi-run sensitivity analyses. `evalstats` guides you toward best practices and choose well-calibrated methods and procedures by default, backed by simulations, and was built specifically to fill the gap of statistical knowledge for small-sample size datasets N<100; it will output stats as long as there are at least 15 samples. See [Statistics](#statistics). 
+- **PPI-corrected inference**: using a small set of human labels to correct bias in noisy LLM-judge scores, so your means, confidence intervals, and hypothesis tests p-values are accurately calibrated in the face of LLM judge bias. This builds on prediction-powered inference (PPI). See [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests).
+
+In particular, scientists can use our PPI-corrected statistical tests to analyze data for **mixed human-AI subject studies**, where some observations are human-labeled and the rest are graded by an LLM judge. Use `evalstats.tests` directly for LLM-judge-bias-corrected versions of:
+
+- t-test (`ttest`, independent or paired; Welch's by default, or Student's equal-variance via `equal_var=True`)
+- Mann–Whitney U (`mannwhitney`)
+- Wilcoxon signed-rank (`wilcoxon`)
+- One-way ANOVA (`anova_oneway`, independent or repeated-measures)
+- Friedman test (`friedman`, repeated-measures rank-based)
+- Kruskal-Wallis (`kruskalwallis`, independent-groups rank-based)
+
+As long as the items for human labeling were sampled at random from the full dataset, p-values will stay calibrated even when the LLM judge is biased or miscalibrated. These corrections are validated via extensive Monte Carlo simulations (see `simulations/harness`). To the best of our knowledge, `evalstats` provides the only known implementations of PPI-corrected rank-based nonparametric tests like Wilcoxon. 
+
+As well, 
+
+> [!IMPORTANT]
+> We are actively building out this project, both the website/guide and the package.
+> Aside from the package itself, there is a "learning" guide in `website/` which I am building out and will return to after writing up the simulations. This will include simulation- and research-backed examples of statistics for LLM evals, as well as example code (which will, obviously, use `evalstats`, but the lessons hold regardless of implementation). 
+> If there's something you'd like to see, or guidance on a specific topic, let us know
+> by raising an Issue.
+
+## Sample output
+
+Running `es.compare(evaldata, factors="prompt")` and then `result.summary()` prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and per-input stability across runs (how stable the model is across multiple runs for the same input). Below is example excerpt from an analysis of a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators):
+
+![Example terminal output](docs/example-output.png)
+
+From this output, we can see that Minimal and Instructive are the most promising candidates, but it is statistically unclear which is better. We also see that Chain-of-thought gives the least consistent outputs across multiple runs for the same inputs, compared to the other methods. 
+
+In the most recent version of `evalstats`, there's also helpful colors to help you see
+this information. For instance, comparing models and prompts at the same time,
+`evalstats` shows a 4-way tie between four combinations of model-prompt: 
+
+![Example terminal output with colors](docs/terminal-output-example.jpg)
+
+You can also plot within notebook environments (although this feature is being actively built out over time and the least developed at the moment). The `plot_point_estimates` function produces a chart showing each template's absolute mean score with marginal confidence intervals:
+
+![Mean advantage plot](docs/mean_advantage.png)
+
+## Statistics
+
+The specific statistical tests that `evalstats.compare()` runs (via the lower-level `analyze()` engine underneath it) are:
+
+- **All pairwise prompt comparisons (paired by input)** via `all_pairwise(...)`:
+    - Computes mean or median difference (mean by default), bootstrapped 95% confidence interval, and p-value for every prompt template pair.
+    - Comparison method defaults to `method="auto"`:
+        - **Smoothed bootstrap with a Gaussian KDE** (`method="smooth_bootstrap"`) in situations of non-binary data. It has been verified in our simulations that for eval-type data and small sample sizes especially, smoothed is superior to the other bootstrap methods considered (percentile, BCa, Bayesian).
+        - **Bayesian pairwise from [`bayes_evals`](https://github.com/sambowyer/bayes_evals/tree/main) and McNemar's test**: Default methods for binary scores (0 or 1 only). Our simulations showed Bayesian pairwise was superior to bootstrap at small N. Note that Bayesian methods should technically be called credible intervals, but they estimate the confidence interval very closely.  
+    - Multiple-comparisons correction for p-values (defaults to **Benjamini–Hochberg (fdr_bh)**). 
+    - Also reports Wilcoxon signed-rank test p-value, in case you need it for people familiar with that test, although p-values from bootstrapped CIs are more robust
+
+- **Bootstrap rank distribution** via `bootstrap_ranks(...)`:
+    - Estimates each prompt template’s `P(best)` and expected rank among the full list of prompt templates.
+
+- **Point estimates** via `robustness_metrics(...)`:
+    - Descriptive stats like mean, median, std, CV, IQR, CVaR-10, and key percentiles.
+    - Marginal confidence intervals on absolute means/medians.
+
+If your benchmark includes repeated runs (`R >= 3`), bootstrap-based analyses above use a **two-level nested bootstrap** (resample inputs, then runs within each input) so run-to-run stochasticity is propagated into CIs and rankings. In that case, `analyze()` also returns a seed/input variance decomposition via `seed_variance_decomposition(...)`.
+
+If you set `method="lmm"`, `analyze()` switches to a mixed-effects path (`score ~ template + (1|input)`) with Wald CIs and parametric rank distributions. By default this uses `statsmodels` (pure Python, no additional setup required); pass `backend="pymer4"` to use R's lme4/emmeans instead (requires a separate R installation — see below). **Mixed effects model support is more experimental at the moment.**
+
+## Installation and Quick start CLI
+
+```bash
+pip install evalstats
+```
+
+For Excel (`.xlsx`) input support:
+
+```bash
+pip install "evalstats[xlsx]"
+```
+
+For all optional extras (including mixed-effects/LMM support):
+
+```bash
+pip install "evalstats[all]"
+```
+
+From the command line, `evalstats` can read a CSV or Excel file directly and print a statistical summary:
+
+```bash
+evalstats analyze results.csv
+```
+
+The input file should have a prompt/template column, an item/input column, and a score column (model, run, and evaluator columns are optional) — see the column alias table in the [Python API](#python-api) section below for recognized names. Run `evalstats analyze --help` for the full list of options and supported column aliases.
+
+For more complex statistical analysis with mixed effects models, use `method="lmm"`. The default `statsmodels` backend works out of the box; for the optional R-based backend, see below.
+
+## Python API
+
+The main entry point is `load_from()` + `compare()`: parse your data once into an `EvalResults` object, then run comparisons against it.
+
+```python
+import pandas as pd
+import evalstats as es
+
+df = pd.read_csv("results.csv")  # columns: prompt, item, score (model optional)
+
+evaldata = es.load_from(df)
+evaldata.summary()  # inspect detected structure/column assignments before analyzing
+
+result = es.compare(evaldata, factors="prompt")
+result.summary()  # full terminal report: CIs, pairwise tests, rank probabilities
+```
+
+`evalstats` expects **long-format** data: one row per (item, score) observation, plus whichever axis you want to compare — `model`, `prompt`, or both — and optionally `run` for repeated runs. Only `item` and `score` are strictly required; you need at least one of `model`/`prompt` too, whichever you pass to `compare(factors=...)`. `load_from()` auto-detects each column's role by matching its name (case-insensitively) against this table:
+
+| Role     | Canonical name | Recognized aliases                    | Required?                                          |
+|----------|-----------------|----------------------------------------|------------------------------------------------------|
+| model    | `model`         | `model_label`, `model_name`            | Optional — needed to compare models (`factors="model"`) |
+| prompt   | `prompt`        | `template`, `prompt_template`          | Optional — needed to compare prompts (`factors="prompt"`) |
+| item     | `item`          | `input`, `example`, `id`, `input_label`| Yes                                                  |
+| score    | `score`         | `value`, `result`, `metric`            | Yes                                                  |
+| run      | `run`           | `seed`, `repeat`, `run_id`, `trial`    | Optional — add if you have repeated runs per (model/prompt, item) |
+
+For example, a minimal CSV comparing prompts:
+
+| prompt      | item | score |
+|-------------|------|-------|
+| Minimal     | q1   | 0.82  |
+| Instructive | q1   | 0.91  |
+| Minimal     | q2   | 0.75  |
+| Instructive | q2   | 0.88  |
+
+If your columns don't match any of the aliases above, remap them explicitly with `col_map`:
+
+```python
+evaldata = es.load_from(df, col_map={"llm": "model", "variant": "prompt", "q_id": "item"})
+```
+
+`compare()` also handles:
+
+- **Comparing models**: `factors="model"`
+- **Factorial designs** (model × prompt): `factors=["model", "prompt"]` (routes to an LMM backend)
+- **Filtering**: any keyword matching a column name acts as a row filter, e.g. `es.compare(evaldata, factors="model", split="test")`
+- **PPI-corrected inference** for noisy LLM-judge scores against a smaller human-labeled subset — see [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests) below
+
+The returned `result` is a `ComparisonResult`. Besides `.summary()`, it has `.to_frame()` / `.to_dict()` for programmatic access, `.plot(method="bar" | "forest" | "cd")` for charts, and `.disagreements()` to surface the items entities disagree on most.
+
+### Advanced: raw score arrays (low-level engine)
+
+`compare()` is a wrapper around a lower-level engine, `analyze()`, which operates directly on `BenchmarkResult` / `MultiModelBenchmark` objects (numpy score arrays) rather than a DataFrame. Reach for this path only if you already have scores as arrays and don't want to build a DataFrame first — most use cases should use `compare()` above.
+
+```python
+import numpy as np
+import evalstats as estats
+
+# Example raw scores for 4 templates × 3 inputs (single run, single evaluator)
+your_scores = [
+    [0.91, 0.88, 0.86],
+    [0.90, 0.89, 0.84],
+    [0.85, 0.82, 0.80],
+    [0.79, 0.76, 0.74],
+]
+n_templates = 4
+n_inputs = 3
+
+# scores shape: (n_templates, n_inputs, n_runs, n_evaluators)
+# For a single evaluator and single run, shape is (N, M, 1, 1)
+scores = np.array(your_scores).reshape(n_templates, n_inputs, 1, 1)
+
+result = estats.BenchmarkResult(
+    scores=scores,
+    template_labels=["Minimal", "Instructive", "Few-shot", "Chain-of-thought"],
+    input_labels=[f"input_{i}" for i in range(n_inputs)],
+)
+
+analysis = estats.analyze(result, reference="grand_mean", n_bootstrap=5_000)
+analysis.summary()  # same terminal report as ComparisonResult.summary()
+```
+
+If you want this lower-level path from a DataFrame (e.g. to inspect the raw `BenchmarkResult` object, or to fine-tune `strict_complete_design`), use `from_dataframe()` instead of `load_from()`. It returns the array-based `BenchmarkResult` / `MultiModelBenchmark` that `analyze()` expects, plus an optional `DataLoadReport` — a data-quality log of coercions/repairs made while parsing (not a statistical report):
+
+```python
+import evalstats as estats
+
+benchmark, load_report = estats.from_dataframe(
+    df,
+    format="auto",                  # auto / wide / long
+    repair=True,                     # average duplicate cells + fill partial run slots
+    strict_complete_design=True,     # set False to keep NaNs
+    return_report=True,
+)
+
+for line in load_report.to_lines():
+    print(line)
+
+analysis = estats.analyze(benchmark)
+analysis.summary()
+```
+
+To visualize absolute prompt performance directly from a `BenchmarkResult`, bypassing `analyze()` (use `result.plot()` above instead if you're on the `compare()` path):
+
+```python
+fig = estats.plot_point_estimates(result)
+fig.savefig("mean_performance.png", dpi=150, bbox_inches="tight")
+```
+
+## PPI-Corrected Inference (Means, CIs, and Tests)
+
+`evalstats` supports PPI-corrected inference for means, confidence intervals, and common statistical tests.
+
+PPI (Prediction-Powered Inference) lets you use lots of cheap LLM
+judgments plus a smaller set of human labels to correct measurement error from the LLM
+judge. This gives you corrected estimates and uncertainty that better reflect what you
+would have gotten from a fully human-labeled study (Angelopoulos et al., 2023).
+
+Most PPI correction methods use PPIBoot (bootstrap variant of PPI; Zrnic, 2024).
+Implemented corrections have been battle-tested via simulations (see `simulations/sim_type_i_calibration.py`).
+
+> **Important: which items get a human label must be chosen uniformly at
+> random.** PPI correction assumes the labeled subset is representative of
+> the full dataset. If your labeling process instead targets specific items
+> — e.g. "always double-check the borderline or highest-scoring responses,"
+> a common real-world review habit — that's missing-not-at-random (MNAR)
+> selection on the outcome, and PPI correction can stay badly miscalibrated
+> **no matter how many items you label**. This isn't ordinary small-sample
+> noise that more labels fixes; it was confirmed in simulation to persist
+> from 15 up through 300 labeled items out of 400. See
+> `evalstats.ppi.correct`'s docstring for the full analysis. If you can't
+> guarantee random labeling, treat any PPI-corrected result here with
+> caution regardless of the reported CI/p-value.
+
+### Example: Comparing models with corrected LLM judge evals via `compare(..., alignment=...)`
+
+```python
+import evalstats as es
+
+# Dataframe columns include:
+#  model   item    llm_score  human_score (NaN for unlabeled rows)
+evaldata = es.load_from(df)
+
+# Compute alignment between LLM and human judges
+alignment = es.validate_alignment(
+    evaldata,
+    llm_metric="llm_score",
+    human_groundtruth="human_score",
+)
+
+# Compare models, using PPI to correct for bias/misalignment with human graders
+result = es.compare(
+    evaldata,
+    factors="model",
+    metric="llm_score",
+    alignment={"llm_score": alignment},
+)
+
+result.summary()
+```
+
+### Example: T-test PPI-correction via `evalstats.tests.ttest`
+
+Use this for a t-test of mean differences between two groups (or two paired
+conditions when `paired=True`).
+
+```python
+import evalstats as es
+
+res = es.tests.ttest(
+    a=llm_a,
+    b=llm_b,
+    a_lab=human_a,  # same length as llm_a, NaN where unlabeled
+    b_lab=human_b,  # same length as llm_b, NaN where unlabeled
+    paired=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Mann-Whitney U test PPI-correction via `evalstats.tests.mannwhitney`
+
+Use this for a Mann-Whitney U test, a nonparametric two-group comparison based
+on relative ranks rather than assuming normally distributed scores.
+
+```python
+import evalstats as es
+
+res = es.tests.mannwhitney(
+    x=llm_x,
+    y=llm_y,
+    x_lab=human_x,
+    y_lab=human_y,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Wilcoxon signed-ranks test PPI-correction via `evalstats.tests.wilcoxon` (paired)
+
+Use this for a Wilcoxon signed-rank test, a nonparametric paired test for
+matched observations (before/after, A/B on the same items, etc.).
+
+```python
+import evalstats as es
+
+res = es.tests.wilcoxon(
+    x=llm_before,
+    y=llm_after,
+    x_lab=human_before,
+    y_lab=human_after,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: One-way ANOVA PPI-correction via `evalstats.tests.anova_oneway`
+
+Use this for one-way ANOVA when comparing more than two groups, with
+`repeated=True` for repeated-measures (same subjects across conditions).
+
+```python
+import evalstats as es
+
+res = es.tests.anova_oneway(
+    llm_g1,
+    llm_g2,
+    llm_g3,
+    groups_lab=[human_g1, human_g2, human_g3],
+    repeated=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+## Motivation
+
+Most eval tools in the LLM evaluation space don't help users perform _any_ statistical tests, let alone showcase variances in performance between prompts or models. They instead present bar charts of average performance. Developers then glance at the bar chart and decide that "prompt/model A is better than B." But was it really?
+
+Relying purely on bar charts and averages can very, very easily lead to erroneous conclusions—B might actually be more robust than A, or B performs well on an important subset of data, or there's not enough data to conclude one way or the other.
+
+Why do people do evals this way? Well, they don't have the time, tools, or knowledge on how to do it better—frequently, they don't even know there's a better way.
+
+`evalstats` aims to rectify this with simple, powerful defaults—just throw us your data and we'll run the stats and plot the results for you. Upstream applications, like LLM observability platforms, could take `evalstats` results and plot them in their own front-ends. Prompt optimization tools could also use `evalstats` to decide, e.g., when to cull a candidate prompt and how to present results to users.
+
+## Examples
+
+### Is one prompt "better" than others? Quantify uncertainty
+
+When you have scores for multiple prompt templates across a set of inputs, `evalstats` computes bootstrapped 95% confidence intervals and pairwise significance tests so you can see not just which prompt scored highest on average, but how certain you can be about that ranking. It plots these to the terminal so you can check at a glance:
+
+![Comparing across prompts output](docs/compare-prompts-output.png)
+
+### Comparing across models while accounting for prompt sensitivity
+
+A common failure mode in LLM benchmarking, both in academic papers and practitioner evaluations, is testing each model with a single prompt template and reporting the resulting scores as if they reflect stable model capabilities. In reality, model rankings can flip under semantically equivalent paraphrases of the same instruction. A benchmark result that says "Model A beats Model B" may be an artifact of prompt phrasing, not a meaningful capability difference. 
+
+Here, we can see the difference between OpenAI's `gpt-4.1-nano` and MistralAI's `ministral-8b-2512` on a small sentiment classification benchmark, quantified by bootstrapped 95% confidence intervals:
+
+![Comparing across models output](docs/compare-models-output.png)
+
+In this run, multiple prompt template variations were considered, making this result more robust than trying a single prompt and calling it a day.
+
+### How stable is the performance across runs?
+
+LLMs are stochastic at temperature>0. Will the performance stay similar, even upon multiple runs for the same inputs? `evalstats` offers a helpful "noise plot" which visualizes (in)stability across runs:
+
+![Per-input noise across runs](docs/per-input-noise.png)
+
+## Running Example Scripts
+
+We provide multiple standalone example scripts that rig up a simple benchmark, collect LLM responses, and run analyses over them. From the repository root, run any example script directly:
+
+```bash
+python examples/synthetic_mean_advantage.py
+```
+
+Additional examples:
+
+```bash
+# OpenAI sentiment benchmark (single run)
+python examples/sentiment.py
+
+# Multi-run variant (captures run-to-run variability)
+python examples/sentiment_multirun.py
+
+# Multi-model comparison across prompt templates
+python examples/compare_models_multirun.py
+
+# Manual API call walkthrough
+python examples/sentiment_manual_api_calls.py
+```
+
+OpenAI-powered examples require `OPENAI_API_KEY` set in your environment. But, you can easily swap out the model calls to whatever model you prefer. 
+
+## Mixed effects models (LMM)
+
+> [!IMPORTANT]
+> Mixed effects analysis is experimental, and currently offers only the advantage of gracefully
+> dealing with missing data. In the future, we plan to add factor decomposition across multiple inputs.
+> We recommend only using `method="lmm"` if you need robustness to missing data (`NaN`). Keep
+> in mind that missing data must be reasonably random (i.e., like sampling from a larger distribution).
+
+`evalstats` supports mixed-effects models (`score ~ template + (1|input)`) for:
+- Missing data in inputs (some score cells are `NaN`)
+- Factor decomposition when multiple input factors are present
+
+### Default backend: statsmodels (pure Python)
+
+No extra setup required — `statsmodels` is included in the standard `pip install evalstats`. Simply pass `method="lmm"`:
+
+```python
+analysis = estats.analyze(result, method="lmm")
+```
+
+`evalstats` fits the model with REML, computes Wald CIs via the delta method, and estimates rank distributions by parametric simulation.
+
+### Optional backend: pymer4 (requires R)
+
+For Satterthwaite degrees of freedom and `emmeans`-based pairwise contrasts (R's gold standard for mixed models), pass `backend="pymer4"`:
+
+```python
+analysis = estats.analyze(result, method="lmm", backend="pymer4")
+```
+
+This requires a working R installation with the following packages:
+
+```r
+install.packages(c(
+    "lme4",
+    "emmeans",
+    "tibble",
+    "broom",
+    "broom.mixed",
+    "lmerTest",
+    "report",
+    "car"
+))
+```
+
+Then install the Python LMM extra:
+
+```bash
+pip install "evalstats[lmm]"
+```
+
+> [!NOTE]
+> If your environment needs manual dependency pinning, this is the tested equivalent:
+>
+> ```bash
+> pip install "pymer4>=0.9" great_tables joblib rpy2 polars scikit-learn formulae pyarrow
+> ```
+
+Installation details may differ on your system.
+
+## Reproducibility: Monte Carlo simulations
+
+Claims in this README like "verified in our simulations" are backed by a runnable simulation harness in `simulations/harness/` of this package. We engineered these simulations so that you can run these yourself. For instance: 
+
+```bash
+python -m simulations.harness.cli --list-cases
+python -m simulations.harness.cli --official-tests
+python -m simulations.harness.cli ci_single --reps 50 --sizes 10 20
+python -m simulations.harness.cli pvalues --mode ppi --tests ttest wilcoxon anova_rep
+```
+
+`--official-tests will bring up a CLI with options to run specific tests. Each runs each case's canonical, full-scale preset and writes results plus a `manifest.json` (args, output paths, key metrics, pass/fail) to `simulations/out/official_<timestamp>/`. See [`simulations/harness/README.md`](simulations/harness/README.md) for the full case list, scenario library, and verification methodology against the original standalone scripts. Note that *each* simulation can take *very long* to run; even on a MacBook Pro with an M4 Max chip and 64GB RAM, with computation paralellized across 16 CPU cores, it often takes many hours.
+- `ci_single` / `ci_paired` — coverage and width of confidence interval methods (bootstrap, smoothed bootstrap, Bayesian, Wilson, etc.) across synthetic distributions and real benchmark data (OpenEval, Inspect AI).
+- `pvalues --mode pairwise` / `--mode multiarm` — Type-I error and power for pairwise and multi-arm comparisons, including multiple-comparisons correction strategies.
+- `pvalues --mode ppi` — Type-I error calibration and power for every PPI-corrected test in `evalstats.tests`, swept across judge-bias severity, label fraction, and MNAR-labeling scenarios.
+
+
+## Development and Contributions
+
+For package build, release validation, and maintainer workflows, see [DEVELOPMENT.md](DEVELOPMENT.md).
+
+We welcome contributions, especially refinements to our statistical methods. If you're proposing a new correction, CI method, or a fix to an existing one, we encourage battle-testing it against the [simulation harness](#reproducibility-monte-carlo-simulations) first. Please add or extend a scenario and confirm your change holds up on Type-I error and power, not just on the case that motivated it, before opening a PR. The `evalstats` repository already offers a rigorous, expansive synthetic suite that generally has held up against real data. 
+
+## License
+
+This repository uses two licenses:
+
+- **`evalstats` package** (everything outside `website/`) — [MIT](LICENSE).
+- **Stats for Evals Website** (everything in `website/`) — [CC BY-NC-ND 4.0](website/LICENSE). You may share it with attribution non-commercially, but commercial use and derivative works are not permitted.