evalstats 0.2.0__tar.gz → 0.2.2__tar.gz

{evalstats-0.2.0 → evalstats-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: evalstats
-Version: 0.2.0
+Version: 0.2.2
 Summary: Statistically sane analysis methods for comparing AI model and prompt performance.
 Author: Ian Arawjo
 License-Expression: MIT
@@ -81,7 +81,7 @@ the lessons hold regardless of implementation).
 
 ## Sample output
 
-Running `estats.analyze()` and then `estats.print_analysis_summary(analysis)` prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and per-input stability across runs (how stable the model is across multiple runs for the same input). Below is example excerpt from an analysis of a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators):
+Running `es.compare(evaldata, factors="prompt")` and then `result.summary()` prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and per-input stability across runs (how stable the model is across multiple runs for the same input). Below is example excerpt from an analysis of a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators):
 
 ![Example terminal output](docs/example-output.png)
 
@@ -99,10 +99,10 @@ You can also plot within notebook environments (although this feature is being a
 
 ## Statistics
 
-The specific statistical tests the `evalstats.analyze()` method runs are:
+The specific statistical tests that `evalstats.compare()` runs (via the lower-level `analyze()` engine underneath it) are:
 
 - **All pairwise prompt comparisons (paired by input)** via `all_pairwise(...)`:
-    - Computes mean or median difference (mean by default), bootstrapped 99% confidence interval, and p-value for every prompt template pair.
+    - Computes mean or median difference (mean by default), bootstrapped 95% confidence interval, and p-value for every prompt template pair.
     - Comparison method defaults to `method="auto"`:
         - **Smoothed bootstrap with a Gaussian KDE** (`method="smooth_bootstrap"`) in situations of non-binary data. It has been verified in our simulations that for eval-type data and small sample sizes especially, smoothed is superior to the other bootstrap methods considered (percentile, BCa, Bayesian).
         - **Bayesian pairwise from [`bayes_evals`](https://github.com/sambowyer/bayes_evals/tree/main) and McNemar's test**: Default methods for binary scores (0 or 1 only). Our simulations showed Bayesian pairwise was superior to bootstrap at small N. Note that Bayesian methods should technically be called credible intervals, but they estimate the confidence interval very closely.  
@@ -144,13 +144,64 @@ From the command line, `evalstats` can read a CSV or Excel file directly and pri
 evalstats analyze results.csv
 ```
 
-The input file should have columns `template`, `input`, and `score` (run and evaluator columns are optional). Run `evalstats analyze --help` for the full list of options and supported column aliases.
+The input file should have a prompt/template column, an item/input column, and a score column (model, run, and evaluator columns are optional) — see the column alias table in the [Python API](#python-api) section below for recognized names. Run `evalstats analyze --help` for the full list of options and supported column aliases.
 
 For more complex statistical analysis with mixed effects models, use `method="lmm"`. The default `statsmodels` backend works out of the box; for the optional R-based backend, see below.
 
 ## Python API
 
-`evalstats` main use case is as a Python API, which provides a similar entry point, the `analyze()` function. Simply pass your benchmark data in the correct format, and pass it to `analyze` to get a battery of results:
+The main entry point is `load_from()` + `compare()`: parse your data once into an `EvalResults` object, then run comparisons against it.
+
+```python
+import pandas as pd
+import evalstats as es
+
+df = pd.read_csv("results.csv")  # columns: prompt, item, score (model optional)
+
+evaldata = es.load_from(df)
+evaldata.summary()  # inspect detected structure/column assignments before analyzing
+
+result = es.compare(evaldata, factors="prompt")
+result.summary()  # full terminal report: CIs, pairwise tests, rank probabilities
+```
+
+`evalstats` expects **long-format** data: one row per (item, score) observation, plus whichever axis you want to compare — `model`, `prompt`, or both — and optionally `run` for repeated runs. Only `item` and `score` are strictly required; you need at least one of `model`/`prompt` too, whichever you pass to `compare(factors=...)`. `load_from()` auto-detects each column's role by matching its name (case-insensitively) against this table:
+
+| Role     | Canonical name | Recognized aliases                    | Required?                                          |
+|----------|-----------------|----------------------------------------|------------------------------------------------------|
+| model    | `model`         | `model_label`, `model_name`            | Optional — needed to compare models (`factors="model"`) |
+| prompt   | `prompt`        | `template`, `prompt_template`          | Optional — needed to compare prompts (`factors="prompt"`) |
+| item     | `item`          | `input`, `example`, `id`, `input_label`| Yes                                                  |
+| score    | `score`         | `value`, `result`, `metric`            | Yes                                                  |
+| run      | `run`           | `seed`, `repeat`, `run_id`, `trial`    | Optional — add if you have repeated runs per (model/prompt, item) |
+
+For example, a minimal CSV comparing prompts:
+
+| prompt      | item | score |
+|-------------|------|-------|
+| Minimal     | q1   | 0.82  |
+| Instructive | q1   | 0.91  |
+| Minimal     | q2   | 0.75  |
+| Instructive | q2   | 0.88  |
+
+If your columns don't match any of the aliases above, remap them explicitly with `col_map`:
+
+```python
+evaldata = es.load_from(df, col_map={"llm": "model", "variant": "prompt", "q_id": "item"})
+```
+
+`compare()` also handles:
+
+- **Comparing models**: `factors="model"`
+- **Factorial designs** (model × prompt): `factors=["model", "prompt"]` (routes to an LMM backend)
+- **Filtering**: any keyword matching a column name acts as a row filter, e.g. `es.compare(evaldata, factors="model", split="test")`
+- **PPI-corrected inference** for noisy LLM-judge scores against a smaller human-labeled subset — see [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests) below
+
+The returned `result` is a `ComparisonResult`. Besides `.summary()`, it has `.to_frame()` / `.to_dict()` for programmatic access, `.plot(method="bar" | "forest" | "cd")` for charts, and `.disagreements()` to surface the items entities disagree on most.
+
+### Advanced: raw score arrays (low-level engine)
+
+`compare()` is a wrapper around a lower-level engine, `analyze()`, which operates directly on `BenchmarkResult` / `MultiModelBenchmark` objects (numpy score arrays) rather than a DataFrame. Reach for this path only if you already have scores as arrays and don't want to build a DataFrame first — most use cases should use `compare()` above.
 
 ```python
 import numpy as np
@@ -177,10 +228,10 @@ result = estats.BenchmarkResult(
 )
 
 analysis = estats.analyze(result, reference="grand_mean", n_bootstrap=5_000)
-estats.print_analysis_summary(analysis)
+analysis.summary()  # same terminal report as ComparisonResult.summary()
 ```
 
-If your source data is already in a pandas DataFrame (possibly with noisy values), you can parse it directly and inspect a coercion report:
+If you want this lower-level path from a DataFrame (e.g. to inspect the raw `BenchmarkResult` object, or to fine-tune `strict_complete_design`), use `from_dataframe()` instead of `load_from()`. It returns the array-based `BenchmarkResult` / `MultiModelBenchmark` that `analyze()` expects, plus an optional `DataLoadReport` — a data-quality log of coercions/repairs made while parsing (not a statistical report):
 
 ```python
 import evalstats as estats
@@ -197,15 +248,133 @@ for line in load_report.to_lines():
     print(line)
 
 analysis = estats.analyze(benchmark)
+analysis.summary()
 ```
 
-To visualize absolute prompt performance with bootstrapped 99% confidence intervals:
+To visualize absolute prompt performance directly from a `BenchmarkResult`, bypassing `analyze()` (use `result.plot()` above instead if you're on the `compare()` path):
 
 ```python
 fig = estats.plot_point_estimates(result)
 fig.savefig("mean_performance.png", dpi=150, bbox_inches="tight")
 ```
 
+## PPI-Corrected Inference (Means, CIs, and Tests)
+
+`evalstats` supports PPI-corrected inference for means, confidence intervals, and common statistical tests.
+
+PPI (Prediction-Powered Inference) lets you use lots of cheap LLM
+judgments plus a smaller set of human labels to correct measurement error from the LLM
+judge. This gives you corrected estimates and uncertainty that better reflect what you
+would have gotten from a fully human-labeled study (Angelopoulos et al., 2023).
+
+Most PPI correction methods use PPIBoot (bootstrap variant of PPI; Zrnic, 2024).
+Implemented corrections have been battle-tested via simulations (see `simulations/sim_type_i_calibration.py`).
+
+### Example: Comparing models with corrected LLM judge evals via `compare(..., alignment=...)`
+
+```python
+import evalstats as es
+
+# Dataframe columns include:
+#  model   item    llm_score  human_score (NaN for unlabeled rows)
+evaldata = es.load_from(df)
+
+# Compute alignment between LLM and human judges
+alignment = es.validate_alignment(
+    evaldata,
+    llm_metric="llm_score",
+    human_groundtruth="human_score",
+)
+
+# Compare models, using PPI to correct for bias/misalignment with human graders
+result = es.compare(
+    evaldata,
+    factors="model",
+    metric="llm_score",
+    alignment={"llm_score": alignment},
+)
+
+result.summary()
+```
+
+### Example: T-test PPI-correction via `evalstats.tests.ttest`
+
+Use this for a t-test of mean differences between two groups (or two paired
+conditions when `paired=True`).
+
+```python
+import evalstats as es
+
+res = es.tests.ttest(
+    a=llm_a,
+    b=llm_b,
+    a_lab=human_a,  # same length as llm_a, NaN where unlabeled
+    b_lab=human_b,  # same length as llm_b, NaN where unlabeled
+    paired=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Mann-Whitney U test PPI-correction via `evalstats.tests.mannwhitney`
+
+Use this for a Mann-Whitney U test, a nonparametric two-group comparison based
+on relative ranks rather than assuming normally distributed scores.
+
+```python
+import evalstats as es
+
+res = es.tests.mannwhitney(
+    x=llm_x,
+    y=llm_y,
+    x_lab=human_x,
+    y_lab=human_y,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Wilcoxon signed-ranks test PPI-correction via `evalstats.tests.wilcoxon` (paired)
+
+Use this for a Wilcoxon signed-rank test, a nonparametric paired test for
+matched observations (before/after, A/B on the same items, etc.).
+
+```python
+import evalstats as es
+
+res = es.tests.wilcoxon(
+    x=llm_before,
+    y=llm_after,
+    x_lab=human_before,
+    y_lab=human_after,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: One-way ANOVA PPI-correction via `evalstats.tests.anova_oneway`
+
+Use this for one-way ANOVA when comparing more than two groups, with
+`repeated=True` for repeated-measures (same subjects across conditions).
+
+```python
+import evalstats as es
+
+res = es.tests.anova_oneway(
+    llm_g1,
+    llm_g2,
+    llm_g3,
+    groups_lab=[human_g1, human_g2, human_g3],
+    repeated=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
 ## Motivation
 
 Most eval tools in the LLM evaluation space don't help users perform _any_ statistical tests, let alone showcase variances in performance between prompts or models. They instead present bar charts of average performance. Developers then glance at the bar chart and decide that "prompt/model A is better than B." But was it really?
@@ -220,7 +389,7 @@ Why do people do evals this way? Well, they don't have the time, tools, or knowl
 
 ### Is one prompt "better" than others? Quantify uncertainty
 
-When you have scores for multiple prompt templates across a set of inputs, `evalstats` computes bootstrapped 99% confidence intervals and pairwise significance tests so you can see not just which prompt scored highest on average, but how certain you can be about that ranking. It plots these to the terminal so you can check at a glance:
+When you have scores for multiple prompt templates across a set of inputs, `evalstats` computes bootstrapped 95% confidence intervals and pairwise significance tests so you can see not just which prompt scored highest on average, but how certain you can be about that ranking. It plots these to the terminal so you can check at a glance:
 
 ![Comparing across prompts output](docs/compare-prompts-output.png)
 
@@ -228,7 +397,7 @@ When you have scores for multiple prompt templates across a set of inputs, `eval
 
 A common failure mode in LLM benchmarking, both in academic papers and practitioner evaluations, is testing each model with a single prompt template and reporting the resulting scores as if they reflect stable model capabilities. In reality, model rankings can flip under semantically equivalent paraphrases of the same instruction. A benchmark result that says "Model A beats Model B" may be an artifact of prompt phrasing, not a meaningful capability difference. 
 
-Here, we can see the difference between OpenAI's `gpt-4.1-nano` and MistralAI's `ministral-8b-2512` on a small sentiment classification benchmark, quantified by bootstrapped 99% confidence intervals:
+Here, we can see the difference between OpenAI's `gpt-4.1-nano` and MistralAI's `ministral-8b-2512` on a small sentiment classification benchmark, quantified by bootstrapped 95% confidence intervals:
 
 ![Comparing across models output](docs/compare-models-output.png)
 

{evalstats-0.2.0 → evalstats-0.2.2}/README.md

@@ -27,7 +27,7 @@ the lessons hold regardless of implementation).
 
 ## Sample output
 
-Running `estats.analyze()` and then `estats.print_analysis_summary(analysis)` prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and per-input stability across runs (how stable the model is across multiple runs for the same input). Below is example excerpt from an analysis of a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators):
+Running `es.compare(evaldata, factors="prompt")` and then `result.summary()` prints a full statistical report to the terminal, including confidence interval line plots, pairwise comparisons between prompt templates, and per-input stability across runs (how stable the model is across multiple runs for the same input). Below is example excerpt from an analysis of a 4-template sentiment-classification benchmark (GPT-4.1-nano, 27 inputs, 3 runs, 3 evaluators):
 
 ![Example terminal output](docs/example-output.png)
 
@@ -45,10 +45,10 @@ You can also plot within notebook environments (although this feature is being a
 
 ## Statistics
 
-The specific statistical tests the `evalstats.analyze()` method runs are:
+The specific statistical tests that `evalstats.compare()` runs (via the lower-level `analyze()` engine underneath it) are:
 
 - **All pairwise prompt comparisons (paired by input)** via `all_pairwise(...)`:
-    - Computes mean or median difference (mean by default), bootstrapped 99% confidence interval, and p-value for every prompt template pair.
+    - Computes mean or median difference (mean by default), bootstrapped 95% confidence interval, and p-value for every prompt template pair.
     - Comparison method defaults to `method="auto"`:
         - **Smoothed bootstrap with a Gaussian KDE** (`method="smooth_bootstrap"`) in situations of non-binary data. It has been verified in our simulations that for eval-type data and small sample sizes especially, smoothed is superior to the other bootstrap methods considered (percentile, BCa, Bayesian).
         - **Bayesian pairwise from [`bayes_evals`](https://github.com/sambowyer/bayes_evals/tree/main) and McNemar's test**: Default methods for binary scores (0 or 1 only). Our simulations showed Bayesian pairwise was superior to bootstrap at small N. Note that Bayesian methods should technically be called credible intervals, but they estimate the confidence interval very closely.  
@@ -90,13 +90,64 @@ From the command line, `evalstats` can read a CSV or Excel file directly and pri
 evalstats analyze results.csv
 ```
 
-The input file should have columns `template`, `input`, and `score` (run and evaluator columns are optional). Run `evalstats analyze --help` for the full list of options and supported column aliases.
+The input file should have a prompt/template column, an item/input column, and a score column (model, run, and evaluator columns are optional) — see the column alias table in the [Python API](#python-api) section below for recognized names. Run `evalstats analyze --help` for the full list of options and supported column aliases.
 
 For more complex statistical analysis with mixed effects models, use `method="lmm"`. The default `statsmodels` backend works out of the box; for the optional R-based backend, see below.
 
 ## Python API
 
-`evalstats` main use case is as a Python API, which provides a similar entry point, the `analyze()` function. Simply pass your benchmark data in the correct format, and pass it to `analyze` to get a battery of results:
+The main entry point is `load_from()` + `compare()`: parse your data once into an `EvalResults` object, then run comparisons against it.
+
+```python
+import pandas as pd
+import evalstats as es
+
+df = pd.read_csv("results.csv")  # columns: prompt, item, score (model optional)
+
+evaldata = es.load_from(df)
+evaldata.summary()  # inspect detected structure/column assignments before analyzing
+
+result = es.compare(evaldata, factors="prompt")
+result.summary()  # full terminal report: CIs, pairwise tests, rank probabilities
+```
+
+`evalstats` expects **long-format** data: one row per (item, score) observation, plus whichever axis you want to compare — `model`, `prompt`, or both — and optionally `run` for repeated runs. Only `item` and `score` are strictly required; you need at least one of `model`/`prompt` too, whichever you pass to `compare(factors=...)`. `load_from()` auto-detects each column's role by matching its name (case-insensitively) against this table:
+
+| Role     | Canonical name | Recognized aliases                    | Required?                                          |
+|----------|-----------------|----------------------------------------|------------------------------------------------------|
+| model    | `model`         | `model_label`, `model_name`            | Optional — needed to compare models (`factors="model"`) |
+| prompt   | `prompt`        | `template`, `prompt_template`          | Optional — needed to compare prompts (`factors="prompt"`) |
+| item     | `item`          | `input`, `example`, `id`, `input_label`| Yes                                                  |
+| score    | `score`         | `value`, `result`, `metric`            | Yes                                                  |
+| run      | `run`           | `seed`, `repeat`, `run_id`, `trial`    | Optional — add if you have repeated runs per (model/prompt, item) |
+
+For example, a minimal CSV comparing prompts:
+
+| prompt      | item | score |
+|-------------|------|-------|
+| Minimal     | q1   | 0.82  |
+| Instructive | q1   | 0.91  |
+| Minimal     | q2   | 0.75  |
+| Instructive | q2   | 0.88  |
+
+If your columns don't match any of the aliases above, remap them explicitly with `col_map`:
+
+```python
+evaldata = es.load_from(df, col_map={"llm": "model", "variant": "prompt", "q_id": "item"})
+```
+
+`compare()` also handles:
+
+- **Comparing models**: `factors="model"`
+- **Factorial designs** (model × prompt): `factors=["model", "prompt"]` (routes to an LMM backend)
+- **Filtering**: any keyword matching a column name acts as a row filter, e.g. `es.compare(evaldata, factors="model", split="test")`
+- **PPI-corrected inference** for noisy LLM-judge scores against a smaller human-labeled subset — see [PPI-Corrected Inference](#ppi-corrected-inference-means-cis-and-tests) below
+
+The returned `result` is a `ComparisonResult`. Besides `.summary()`, it has `.to_frame()` / `.to_dict()` for programmatic access, `.plot(method="bar" | "forest" | "cd")` for charts, and `.disagreements()` to surface the items entities disagree on most.
+
+### Advanced: raw score arrays (low-level engine)
+
+`compare()` is a wrapper around a lower-level engine, `analyze()`, which operates directly on `BenchmarkResult` / `MultiModelBenchmark` objects (numpy score arrays) rather than a DataFrame. Reach for this path only if you already have scores as arrays and don't want to build a DataFrame first — most use cases should use `compare()` above.
 
 ```python
 import numpy as np
@@ -123,10 +174,10 @@ result = estats.BenchmarkResult(
 )
 
 analysis = estats.analyze(result, reference="grand_mean", n_bootstrap=5_000)
-estats.print_analysis_summary(analysis)
+analysis.summary()  # same terminal report as ComparisonResult.summary()
 ```
 
-If your source data is already in a pandas DataFrame (possibly with noisy values), you can parse it directly and inspect a coercion report:
+If you want this lower-level path from a DataFrame (e.g. to inspect the raw `BenchmarkResult` object, or to fine-tune `strict_complete_design`), use `from_dataframe()` instead of `load_from()`. It returns the array-based `BenchmarkResult` / `MultiModelBenchmark` that `analyze()` expects, plus an optional `DataLoadReport` — a data-quality log of coercions/repairs made while parsing (not a statistical report):
 
 ```python
 import evalstats as estats
@@ -143,15 +194,133 @@ for line in load_report.to_lines():
     print(line)
 
 analysis = estats.analyze(benchmark)
+analysis.summary()
 ```
 
-To visualize absolute prompt performance with bootstrapped 99% confidence intervals:
+To visualize absolute prompt performance directly from a `BenchmarkResult`, bypassing `analyze()` (use `result.plot()` above instead if you're on the `compare()` path):
 
 ```python
 fig = estats.plot_point_estimates(result)
 fig.savefig("mean_performance.png", dpi=150, bbox_inches="tight")
 ```
 
+## PPI-Corrected Inference (Means, CIs, and Tests)
+
+`evalstats` supports PPI-corrected inference for means, confidence intervals, and common statistical tests.
+
+PPI (Prediction-Powered Inference) lets you use lots of cheap LLM
+judgments plus a smaller set of human labels to correct measurement error from the LLM
+judge. This gives you corrected estimates and uncertainty that better reflect what you
+would have gotten from a fully human-labeled study (Angelopoulos et al., 2023).
+
+Most PPI correction methods use PPIBoot (bootstrap variant of PPI; Zrnic, 2024).
+Implemented corrections have been battle-tested via simulations (see `simulations/sim_type_i_calibration.py`).
+
+### Example: Comparing models with corrected LLM judge evals via `compare(..., alignment=...)`
+
+```python
+import evalstats as es
+
+# Dataframe columns include:
+#  model   item    llm_score  human_score (NaN for unlabeled rows)
+evaldata = es.load_from(df)
+
+# Compute alignment between LLM and human judges
+alignment = es.validate_alignment(
+    evaldata,
+    llm_metric="llm_score",
+    human_groundtruth="human_score",
+)
+
+# Compare models, using PPI to correct for bias/misalignment with human graders
+result = es.compare(
+    evaldata,
+    factors="model",
+    metric="llm_score",
+    alignment={"llm_score": alignment},
+)
+
+result.summary()
+```
+
+### Example: T-test PPI-correction via `evalstats.tests.ttest`
+
+Use this for a t-test of mean differences between two groups (or two paired
+conditions when `paired=True`).
+
+```python
+import evalstats as es
+
+res = es.tests.ttest(
+    a=llm_a,
+    b=llm_b,
+    a_lab=human_a,  # same length as llm_a, NaN where unlabeled
+    b_lab=human_b,  # same length as llm_b, NaN where unlabeled
+    paired=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Mann-Whitney U test PPI-correction via `evalstats.tests.mannwhitney`
+
+Use this for a Mann-Whitney U test, a nonparametric two-group comparison based
+on relative ranks rather than assuming normally distributed scores.
+
+```python
+import evalstats as es
+
+res = es.tests.mannwhitney(
+    x=llm_x,
+    y=llm_y,
+    x_lab=human_x,
+    y_lab=human_y,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: Wilcoxon signed-ranks test PPI-correction via `evalstats.tests.wilcoxon` (paired)
+
+Use this for a Wilcoxon signed-rank test, a nonparametric paired test for
+matched observations (before/after, A/B on the same items, etc.).
+
+```python
+import evalstats as es
+
+res = es.tests.wilcoxon(
+    x=llm_before,
+    y=llm_after,
+    x_lab=human_before,
+    y_lab=human_after,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
+### Example: One-way ANOVA PPI-correction via `evalstats.tests.anova_oneway`
+
+Use this for one-way ANOVA when comparing more than two groups, with
+`repeated=True` for repeated-measures (same subjects across conditions).
+
+```python
+import evalstats as es
+
+res = es.tests.anova_oneway(
+    llm_g1,
+    llm_g2,
+    llm_g3,
+    groups_lab=[human_g1, human_g2, human_g3],
+    repeated=False,
+    print_result=False,
+)
+
+print(res.p_value, res.corrected_p_value, res.corrected_ci)
+```
+
 ## Motivation
 
 Most eval tools in the LLM evaluation space don't help users perform _any_ statistical tests, let alone showcase variances in performance between prompts or models. They instead present bar charts of average performance. Developers then glance at the bar chart and decide that "prompt/model A is better than B." But was it really?
@@ -166,7 +335,7 @@ Why do people do evals this way? Well, they don't have the time, tools, or knowl
 
 ### Is one prompt "better" than others? Quantify uncertainty
 
-When you have scores for multiple prompt templates across a set of inputs, `evalstats` computes bootstrapped 99% confidence intervals and pairwise significance tests so you can see not just which prompt scored highest on average, but how certain you can be about that ranking. It plots these to the terminal so you can check at a glance:
+When you have scores for multiple prompt templates across a set of inputs, `evalstats` computes bootstrapped 95% confidence intervals and pairwise significance tests so you can see not just which prompt scored highest on average, but how certain you can be about that ranking. It plots these to the terminal so you can check at a glance:
 
 ![Comparing across prompts output](docs/compare-prompts-output.png)
 
@@ -174,7 +343,7 @@ When you have scores for multiple prompt templates across a set of inputs, `eval
 
 A common failure mode in LLM benchmarking, both in academic papers and practitioner evaluations, is testing each model with a single prompt template and reporting the resulting scores as if they reflect stable model capabilities. In reality, model rankings can flip under semantically equivalent paraphrases of the same instruction. A benchmark result that says "Model A beats Model B" may be an artifact of prompt phrasing, not a meaningful capability difference. 
 
-Here, we can see the difference between OpenAI's `gpt-4.1-nano` and MistralAI's `ministral-8b-2512` on a small sentiment classification benchmark, quantified by bootstrapped 99% confidence intervals:
+Here, we can see the difference between OpenAI's `gpt-4.1-nano` and MistralAI's `ministral-8b-2512` on a small sentiment classification benchmark, quantified by bootstrapped 95% confidence intervals:
 
 ![Comparing across models output](docs/compare-models-output.png)
 

{evalstats-0.2.0 → evalstats-0.2.2}/evalstats/__init__.py

@@ -1,5 +1,6 @@
 """evalstats: Statistical analysis and visualization for prompt benchmarking."""
 
+# ── Core types and analysis engine ───────────────────────────────────────────
 from evalstats.core.types import BenchmarkResult, MultiModelBenchmark
 from evalstats.core.paired import pairwise_differences, all_pairwise, vs_baseline, friedman_nemenyi, FriedmanResult
 from evalstats.core.ranking import bootstrap_ranks
@@ -24,17 +25,34 @@ from evalstats.vis.scoreboard import plot_accuracy_bar
 from evalstats.io import from_dataframe, DataLoadReport
 from evalstats.core.resampling import bayes_binary_ci_1d, bayes_paired_diff_ci
 from evalstats.core import bayes_evals
-from evalstats.compare import (
-    compare_prompts,
-    compare_models,
-    CompareReport,
-    EntityStats,
-)
 from evalstats.config import set_alpha_ci, get_alpha_ci
 
-__version__ = "0.2.0"
+# ── High-level spec API ───────────────────────────────────────────────────────
+# Must come after all other imports: importing evalstats.api triggers
+# evalstats.compare submodule registration, which would shadow a bare
+# "compare" name if it were imported before the submodule.
+from evalstats.loader import load_from, EvalResults, EvalLoadError
+from evalstats.api import compare, compare_models, compare_prompts, ComparisonResult
+from evalstats.alignment import validate_alignment, AlignmentResult
+from evalstats import ppi
+from evalstats import tests
+
+__version__ = "0.2.2"
 
 __all__ = [
+    # High-level spec API
+    "load_from",
+    "validate_alignment",
+    "AlignmentResult",
+    "ppi",
+    "tests",
+    "EvalResults",
+    "EvalLoadError",
+    "compare",
+    "compare_models",
+    "compare_prompts",
+    "ComparisonResult",
+    # Core types
     "BenchmarkResult",
     "MultiModelBenchmark",
     "pairwise_differences",
@@ -62,10 +80,6 @@ __all__ = [
     "bayes_binary_ci_1d",
     "bayes_paired_diff_ci",
     "bayes_evals",
-    "compare_prompts",
-    "compare_models",
-    "CompareReport",
-    "EntityStats",
     "analyze_factorial",
     "set_alpha_ci",
     "get_alpha_ci",