evalaudit 0.0.1__tar.gz

evalaudit-0.0.1/.github/workflows/tests.yml

@@ -0,0 +1,20 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q

evalaudit-0.0.1/.gitignore

@@ -0,0 +1,18 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.env
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.coverage
+htmlcov/
+.ipynb_checkpoints/
+.DS_Store
+.vscode/
+.idea/
+data/raw/
+outputs/
+*.jsonl

evalaudit-0.0.1/CLAUDE.md

@@ -0,0 +1,39 @@
+# CLAUDE.md — evalaudit project rules
+
+## Design invariants
+
+- Every public function returns a frozen dataclass from `_types.py`.
+- Every return type has a `summary()` that produces a plain-English verdict.
+- Never return a bare p-value. Always report effect size plus confidence interval.
+- Dependencies are numpy, scipy, and pandas only.
+
+## Testing
+
+- Tests are written before implementations and must not be edited to make code pass.
+- Coverage tests (simulation-based) verify that intervals cover at the claimed rate.
+
+## Implementation style
+
+- Vectorise bootstraps in numpy. Draw index matrices in one call and compute row statistics — no Python loops over resamples.
+- Use `rng.integers(0, n, size=(n_boot, n))` for bootstrap index matrices.
+
+## Build order
+
+scores → compare → agreement → pairwise → judge → power → audit
+
+The core of the package is **agreement**, **judge**, and **pairwise** — these are the point, not extras. scores and compare are the foundation they stand on.
+
+## Commands
+
+- Install dev: `pip install -e ".[dev]"`
+- Run tests: `pytest -q`
+
+## Git
+Never run `git commit`, `git push`, or any history-rewriting command.
+Make changes and show the diff. The user reviews and commits.
+
+## Writing style
+Write plainly, the way a knowledgeable person would talk. No em-dashes.
+No "not X, but Y" constructions. No colon-then-reveal sentences.
+Short, direct sentences over clever phrasing. This applies to docstrings,
+README text, commit messages, and the summary() strings.

evalaudit-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hezron Mokaya
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalaudit-0.0.1/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.5
+Name: evalaudit
+Version: 0.0.1
+Summary: Statistical validity checks for human-graded AI evaluations. Rater agreement, judge validation, and whether your result holds up.
+Project-URL: Homepage, https://github.com/ondibahezron-glitch/evalaudit
+Project-URL: Issues, https://github.com/ondibahezron-glitch/evalaudit/issues
+Author-email: Hezron Mokaya <ondibahezron@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: annotation-quality,bradley-terry,evaluation,inter-rater-reliability,krippendorff,llm,llm-as-judge,statistics
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.22
+Requires-Dist: pandas>=1.5
+Requires-Dist: scipy>=1.9
+Provides-Extra: dev
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# evalaudit
+
+[![Tests](https://github.com/ondibahezron-glitch/evalaudit/actions/workflows/tests.yml/badge.svg)](https://github.com/ondibahezron-glitch/evalaudit/actions/workflows/tests.yml)
+
+**Statistical validity checks for human-graded AI evaluations.**
+
+Three experts graded 200 items. One of them disagreed with the other two on
+almost everything. Nobody noticed, because nobody computed agreement — and
+half the items were only graded once, so it could not have been computed.
+
+That evaluation produced a number. The number is now in a slide deck.
+
+`evalaudit` is the set of checks that should have run first.
+
+---
+
+## Start here
+
+```python
+from evalaudit import score_ci
+
+wins = [1] * 58 + [0] * 42
+print(score_ci(wins).summary())
+```
+
+```
+58.0% pass rate (95% CI: 48.2%-67.2%, n=100). The interval spans
+19 points; treat differences smaller than that as unresolved.
+```
+
+The interval crosses 50%. On this data the new model might be worse than the
+old one. That evaluation cannot support the decision being made with it.
+
+---
+
+## Why this exists
+
+`scipy` computes McNemar's test. `sklearn` computes Cohen's kappa. Several
+libraries now put bootstrapped intervals on automated eval scores.
+
+What is missing is anything built for the **human** side of evaluation: the
+graders, the rubric, the LLM judge standing in for both. That is where the
+expensive failures happen, and it is where almost no tooling exists.
+
+Every function returns an object with a `.summary()`. The numbers are the
+input to the audit. The sentence is the output.
+
+Two rules the library follows throughout:
+
+- **Never report a bare p-value.** Always an effect size and an interval.
+- **State what the data cannot support**, not only what it can.
+
+---
+
+## What it checks
+
+| Module | Question |
+|---|---|
+| `agreement` | Do your graders agree? Which grader is the outlier? Which items are ambiguous rather than hard? |
+| `judge` | Does your LLM judge track your humans — and on which slices does it stop? Is it biased by position or length? |
+| `pairwise` | In a blind pairwise comparison, which models are actually separable? |
+| `power` | Could this many comparisons ever have detected the effect you care about? |
+| `scores` | Is there an interval around your headline number, and how wide? |
+| `compare` | Does the margin between two systems survive a paired test? |
+| `audit` | All of the above, as a report ranked by what changes the conclusion. |
+
+The first four are the point. The last three are the foundation they stand on.
+
+---
+
+## Status
+
+Early. `scores` is implemented. `agreement`, `judge` and `pairwise` are
+specified and landing over the coming weeks.
+
+---
+
+## Coverage
+
+A statistics package earns trust by showing its intervals cover at the rate
+they claim. Every interval here is tested by simulation: generate 1,000
+datasets with a known truth, confirm the 95% interval contains it about 95% of
+the time. Those tests run in CI and you can read them in `tests/`.
+
+---
+
+## Related
+
+[`promptstats`](https://pypi.org/project/promptstats/) covers the automated
+side well — prompt sensitivity, model comparison across prompt variations,
+bootstrapped intervals on benchmark scores. If your evaluation has no human
+graders in it, start there.
+
+---
+
+## Install
+
+```bash
+git clone https://github.com/ondibahezron-glitch/evalaudit.git
+cd evalaudit
+pip install -e ".[dev]"
+```
+
+Python 3.9+, numpy, scipy, pandas. Nothing else. PyPI package coming soon.
+
+---
+
+## Licence
+
+MIT.

evalaudit-0.0.1/README.md

@@ -0,0 +1,110 @@
+# evalaudit
+
+[![Tests](https://github.com/ondibahezron-glitch/evalaudit/actions/workflows/tests.yml/badge.svg)](https://github.com/ondibahezron-glitch/evalaudit/actions/workflows/tests.yml)
+
+**Statistical validity checks for human-graded AI evaluations.**
+
+Three experts graded 200 items. One of them disagreed with the other two on
+almost everything. Nobody noticed, because nobody computed agreement — and
+half the items were only graded once, so it could not have been computed.
+
+That evaluation produced a number. The number is now in a slide deck.
+
+`evalaudit` is the set of checks that should have run first.
+
+---
+
+## Start here
+
+```python
+from evalaudit import score_ci
+
+wins = [1] * 58 + [0] * 42
+print(score_ci(wins).summary())
+```
+
+```
+58.0% pass rate (95% CI: 48.2%-67.2%, n=100). The interval spans
+19 points; treat differences smaller than that as unresolved.
+```
+
+The interval crosses 50%. On this data the new model might be worse than the
+old one. That evaluation cannot support the decision being made with it.
+
+---
+
+## Why this exists
+
+`scipy` computes McNemar's test. `sklearn` computes Cohen's kappa. Several
+libraries now put bootstrapped intervals on automated eval scores.
+
+What is missing is anything built for the **human** side of evaluation: the
+graders, the rubric, the LLM judge standing in for both. That is where the
+expensive failures happen, and it is where almost no tooling exists.
+
+Every function returns an object with a `.summary()`. The numbers are the
+input to the audit. The sentence is the output.
+
+Two rules the library follows throughout:
+
+- **Never report a bare p-value.** Always an effect size and an interval.
+- **State what the data cannot support**, not only what it can.
+
+---
+
+## What it checks
+
+| Module | Question |
+|---|---|
+| `agreement` | Do your graders agree? Which grader is the outlier? Which items are ambiguous rather than hard? |
+| `judge` | Does your LLM judge track your humans — and on which slices does it stop? Is it biased by position or length? |
+| `pairwise` | In a blind pairwise comparison, which models are actually separable? |
+| `power` | Could this many comparisons ever have detected the effect you care about? |
+| `scores` | Is there an interval around your headline number, and how wide? |
+| `compare` | Does the margin between two systems survive a paired test? |
+| `audit` | All of the above, as a report ranked by what changes the conclusion. |
+
+The first four are the point. The last three are the foundation they stand on.
+
+---
+
+## Status
+
+Early. `scores` is implemented. `agreement`, `judge` and `pairwise` are
+specified and landing over the coming weeks.
+
+---
+
+## Coverage
+
+A statistics package earns trust by showing its intervals cover at the rate
+they claim. Every interval here is tested by simulation: generate 1,000
+datasets with a known truth, confirm the 95% interval contains it about 95% of
+the time. Those tests run in CI and you can read them in `tests/`.
+
+---
+
+## Related
+
+[`promptstats`](https://pypi.org/project/promptstats/) covers the automated
+side well — prompt sensitivity, model comparison across prompt variations,
+bootstrapped intervals on benchmark scores. If your evaluation has no human
+graders in it, start there.
+
+---
+
+## Install
+
+```bash
+git clone https://github.com/ondibahezron-glitch/evalaudit.git
+cd evalaudit
+pip install -e ".[dev]"
+```
+
+Python 3.9+, numpy, scipy, pandas. Nothing else. PyPI package coming soon.
+
+---
+
+## Licence
+
+MIT.

evalaudit-0.0.1/evalaudit/__init__.py

@@ -0,0 +1,7 @@
+"""evalaudit: statistical validity checks for AI evaluations."""
+
+__version__ = "0.1.0"
+
+from .scores import score_ci
+
+__all__ = ["score_ci"]

evalaudit-0.0.1/evalaudit/_types.py

@@ -0,0 +1,66 @@
+"""Return types for evalaudit.
+
+Every public function returns one of these. Each carries the numbers and a
+``summary()`` that states, in plain English, what the numbers do and do not
+support. The summary is the point of the library: scipy already computes
+McNemar, but nothing tells a product manager their result does not hold.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+def _pct(x: float) -> str:
+    return f"{x * 100:.1f}%"
+
+
+@dataclass(frozen=True)
+class ScoreCI:
+    """Confidence interval on a single eval score."""
+
+    estimate: float
+    ci_low: float
+    ci_high: float
+    n: int
+    method: str
+    binary: bool
+    confidence: float = 0.95
+
+    @property
+    def margin(self) -> float:
+        """Half-width of the interval."""
+        return (self.ci_high - self.ci_low) / 2
+
+    @property
+    def width(self) -> float:
+        return self.ci_high - self.ci_low
+
+    def summary(self) -> str:
+        conf = f"{self.confidence * 100:.0f}%"
+        if self.binary:
+            head = (
+                f"{_pct(self.estimate)} pass rate "
+                f"({conf} CI: {_pct(self.ci_low)}-{_pct(self.ci_high)}, n={self.n})."
+            )
+            span = f"{self.width * 100:.0f} points"
+        else:
+            head = (
+                f"Mean score {self.estimate:.3f} "
+                f"({conf} CI: {self.ci_low:.3f}-{self.ci_high:.3f}, n={self.n})."
+            )
+            span = f"{self.width:.3f}"
+
+        tail = (
+            f" The interval spans {span}; treat differences smaller than that "
+            f"as unresolved."
+        )
+        if self.n < 30:
+            tail += (
+                f" With only {self.n} observations this estimate is weak "
+                f"regardless of the point value."
+            )
+        return head + tail
+
+    def __str__(self) -> str:  # pragma: no cover
+        return self.summary()

evalaudit-0.0.1/evalaudit/scores.py

@@ -0,0 +1,136 @@
+"""Confidence intervals on a single eval score.
+
+The most common missing piece in published eval results: a pass rate with no
+interval around it. At n=50 the interval is roughly +/- 10 points, which is
+usually wider than the difference being claimed.
+"""
+
+from __future__ import annotations
+
+from typing import Optional, Sequence
+
+import numpy as np
+from scipy import stats as _stats
+
+from ._types import ScoreCI
+
+__all__ = ["score_ci"]
+
+_VALID_METHODS = {"auto", "wilson", "bootstrap", "t"}
+
+
+def score_ci(
+    scores: Sequence[float],
+    method: str = "auto",
+    confidence: float = 0.95,
+    n_boot: int = 10000,
+    seed: Optional[int] = None,
+) -> ScoreCI:
+    """Confidence interval on an eval score.
+
+    Parameters
+    ----------
+    scores
+        Binary (0/1) or continuous values, one per evaluated item.
+    method
+        ``"auto"`` picks Wilson for binary data and bootstrap for continuous.
+        ``"wilson"`` is the score interval for a proportion. Use it rather than
+        the normal approximation, which is visibly wrong at small n and near
+        the boundaries, and can return bounds outside [0, 1].
+        ``"bootstrap"`` is the percentile bootstrap of the mean.
+        ``"t"`` is the Student-t interval on the mean.
+    confidence
+        Nominal coverage, default 0.95.
+    n_boot
+        Bootstrap resamples. 1000 is enough for a consulting report;
+        the default is generous.
+    seed
+        Seeds the bootstrap so results are reproducible. Always set this in
+        anything you publish.
+
+    Returns
+    -------
+    ScoreCI
+
+    Examples
+    --------
+    >>> score_ci([1] * 42 + [0] * 8).summary()
+    '84.0% pass rate (95% CI: 71.5%-91.7%, n=50). ...'
+    """
+    x = np.asarray(scores, dtype=float)
+    n = len(x)
+    if n == 0:
+        raise ValueError("scores must not be empty")
+    if method not in _VALID_METHODS:
+        raise ValueError(f"method must be one of {_VALID_METHODS}, got {method!r}")
+
+    binary = _is_binary(x)
+
+    if method == "auto":
+        method = "wilson" if binary else "bootstrap"
+
+    estimate = float(np.mean(x))
+
+    if method == "wilson":
+        k = int(np.sum(x))
+        ci_low, ci_high = _wilson(k, n, confidence)
+    elif method == "bootstrap":
+        ci_low, ci_high = _bootstrap_mean(x, confidence, n_boot, seed)
+    else:  # "t"
+        alpha = 1 - confidence
+        sem = float(np.std(x, ddof=1) / np.sqrt(n))
+        t_crit = _stats.t.ppf(1 - alpha / 2, df=n - 1)
+        ci_low = estimate - t_crit * sem
+        ci_high = estimate + t_crit * sem
+
+    return ScoreCI(
+        estimate=estimate,
+        ci_low=ci_low,
+        ci_high=ci_high,
+        n=n,
+        method=method,
+        binary=binary,
+        confidence=confidence,
+    )
+
+
+def _is_binary(x: np.ndarray) -> bool:
+    """True when every value is 0 or 1."""
+    return bool(np.all((x == 0) | (x == 1)))
+
+
+def _wilson(k: int, n: int, confidence: float) -> tuple[float, float]:
+    """Wilson score interval for a proportion.
+
+        (p + z^2/2n  +/-  z * sqrt( p(1-p)/n + z^2/4n^2 )) / (1 + z^2/n)
+
+    Clip to [0, 1] at the end.
+    """
+    p = k / n
+    z = _stats.norm.ppf(1 - (1 - confidence) / 2)
+    z2 = z * z
+    denom = 1 + z2 / n
+    centre = (p + z2 / (2 * n)) / denom
+    margin = (z / denom) * np.sqrt(p * (1 - p) / n + z2 / (4 * n * n))
+    lo = float(np.clip(centre - margin, 0.0, 1.0))
+    hi = float(np.clip(centre + margin, 0.0, 1.0))
+    return lo, hi
+
+
+def _bootstrap_mean(
+    x: np.ndarray, confidence: float, n_boot: int, seed: Optional[int]
+) -> tuple[float, float]:
+    """Percentile bootstrap of the mean.
+
+    Vectorise this. Draw an (n_boot, n) index matrix with
+    ``rng.integers(0, n, size=(n_boot, n))`` and take row means in one pass.
+    A Python loop here is the difference between instant and slow on a laptop.
+    """
+    rng = np.random.default_rng(seed)
+    n = len(x)
+    idx = rng.integers(0, n, size=(n_boot, n))
+    means = x[idx].mean(axis=1)
+    alpha = 1 - confidence
+    lo = float(np.percentile(means, 100 * alpha / 2))
+    hi = float(np.percentile(means, 100 * (1 - alpha / 2)))
+    return lo, hi

evalaudit-0.0.1/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "evalaudit"
+version = "0.0.1"
+description = "Statistical validity checks for human-graded AI evaluations. Rater agreement, judge validation, and whether your result holds up."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Hezron Mokaya", email = "ondibahezron@gmail.com" }]
+keywords = ["llm", "evaluation", "statistics", "inter-rater-reliability", "krippendorff", "bradley-terry", "llm-as-judge", "annotation-quality"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "numpy>=1.22",
+    "scipy>=1.9",
+    "pandas>=1.5",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "pytest-cov"]
+
+[project.urls]
+Homepage = "https://github.com/ondibahezron-glitch/evalaudit"
+Issues = "https://github.com/ondibahezron-glitch/evalaudit/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["evalaudit"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

evalaudit-0.0.1/tests/test_scores.py

@@ -0,0 +1,181 @@
+"""Tests for evalaudit.scores.
+
+Written before the implementation. Make these pass.
+
+The fixtures in test_wilson_known_values are exact Wilson score intervals
+computed independently; if your implementation returns these, it is correct.
+"""
+
+import numpy as np
+import pytest
+
+from evalaudit.scores import score_ci
+
+
+# --------------------------------------------------------------------------
+# Wilson interval, binary data
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize(
+    "k, n, expected_low, expected_high",
+    [
+        (42, 50, 0.714858, 0.916626),
+        (9, 10, 0.595850, 0.982124),
+        (0, 20, 0.000000, 0.161125),
+        (20, 20, 0.838875, 1.000000),
+        (58, 100, 0.482065, 0.672016),
+    ],
+)
+def test_wilson_known_values(k, n, expected_low, expected_high):
+    scores = [1] * k + [0] * (n - k)
+    r = score_ci(scores, method="wilson")
+    assert r.ci_low == pytest.approx(expected_low, abs=1e-5)
+    assert r.ci_high == pytest.approx(expected_high, abs=1e-5)
+    assert r.estimate == pytest.approx(k / n)
+    assert r.n == n
+
+
+def test_wilson_never_leaves_unit_interval():
+    """The normal approximation goes below 0 here. Wilson must not."""
+    r = score_ci([0] * 30 + [1], method="wilson")
+    assert r.ci_low >= 0.0
+    assert r.ci_high <= 1.0
+
+
+def test_auto_picks_wilson_for_binary():
+    r = score_ci([1, 0, 1, 1, 0, 1, 1, 1, 0, 1])
+    assert r.method == "wilson"
+    assert r.binary is True
+
+
+def test_auto_picks_bootstrap_for_continuous():
+    r = score_ci([0.2, 0.5, 0.9, 0.4, 0.7, 0.3], seed=0)
+    assert r.method == "bootstrap"
+    assert r.binary is False
+
+
+# --------------------------------------------------------------------------
+# Bootstrap, continuous data
+# --------------------------------------------------------------------------
+
+def test_bootstrap_is_deterministic_with_seed():
+    xs = np.random.default_rng(1).normal(3.0, 1.0, 60)
+    a = score_ci(xs, method="bootstrap", seed=42, n_boot=2000)
+    b = score_ci(xs, method="bootstrap", seed=42, n_boot=2000)
+    assert a.ci_low == b.ci_low
+    assert a.ci_high == b.ci_high
+
+
+def test_bootstrap_brackets_the_mean():
+    xs = np.random.default_rng(2).normal(4.2, 0.8, 200)
+    r = score_ci(xs, method="bootstrap", seed=7)
+    assert r.ci_low < r.estimate < r.ci_high
+    assert r.estimate == pytest.approx(float(np.mean(xs)))
+
+
+def test_interval_narrows_with_more_data():
+    rng = np.random.default_rng(3)
+    small = score_ci(rng.normal(0, 1, 40), seed=1)
+    large = score_ci(rng.normal(0, 1, 4000), seed=1)
+    assert large.width < small.width
+
+
+# --------------------------------------------------------------------------
+# Coverage. This is the credibility argument for the whole package.
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("true_p, n", [(0.5, 50), (0.9, 50), (0.3, 200)])
+def test_wilson_coverage(true_p, n):
+    """Across 1000 simulated datasets the 95% interval should cover the
+    true rate about 95% of the time. Wilson under-covers slightly at the
+    extremes, so the band is deliberately generous."""
+    rng = np.random.default_rng(11)
+    covered = 0
+    trials = 1000
+    for _ in range(trials):
+        sample = rng.binomial(1, true_p, n)
+        r = score_ci(sample, method="wilson")
+        if r.ci_low <= true_p <= r.ci_high:
+            covered += 1
+    rate = covered / trials
+    assert 0.92 <= rate <= 0.98, f"coverage was {rate:.3f}"
+
+
+def test_bootstrap_coverage():
+    rng = np.random.default_rng(12)
+    true_mean = 3.0
+    covered = 0
+    trials = 400
+    for _ in range(trials):
+        sample = rng.normal(true_mean, 1.0, 80)
+        r = score_ci(sample, method="bootstrap", n_boot=1000, seed=int(rng.integers(1e6)))
+        if r.ci_low <= true_mean <= r.ci_high:
+            covered += 1
+    rate = covered / trials
+    assert 0.91 <= rate <= 0.99, f"coverage was {rate:.3f}"
+
+
+# --------------------------------------------------------------------------
+# The summary is the product
+# --------------------------------------------------------------------------
+
+def test_summary_mentions_estimate_and_bounds():
+    r = score_ci([1] * 42 + [0] * 8, method="wilson")
+    s = r.summary()
+    assert "84.0%" in s
+    assert "71.5%" in s or "71.4%" in s
+    assert "n=50" in s
+
+
+def test_summary_warns_on_small_n():
+    r = score_ci([1, 1, 0, 1, 1])
+    assert "5 observations" in r.summary()
+
+
+# --------------------------------------------------------------------------
+# Input handling
+# --------------------------------------------------------------------------
+
+def test_rejects_empty():
+    with pytest.raises(ValueError):
+        score_ci([])
+
+
+def test_rejects_unknown_method():
+    with pytest.raises(ValueError):
+        score_ci([1, 0, 1], method="nonsense")
+
+
+def test_accepts_numpy_list_and_series():
+    import pandas as pd
+    base = [1, 0, 1, 1, 0]
+    a = score_ci(base, method="wilson")
+    b = score_ci(np.array(base), method="wilson")
+    c = score_ci(pd.Series(base), method="wilson")
+    assert a.ci_low == pytest.approx(b.ci_low) == pytest.approx(c.ci_low)
+
+
+# --------------------------------------------------------------------------
+# Non-default confidence levels
+# --------------------------------------------------------------------------
+
+def test_confidence_level_affects_width():
+    """A hardcoded z=1.96 would pass every 0.95 test. This catches that."""
+    data = [1] * 42 + [0] * 8
+
+    r90 = score_ci(data, method="wilson", confidence=0.90)
+    r95 = score_ci(data, method="wilson", confidence=0.95)
+    r99 = score_ci(data, method="wilson", confidence=0.99)
+
+    assert r90.width < r95.width < r99.width
+
+    assert "90%" in r90.summary()
+    assert "95%" in r95.summary()
+    assert "99%" in r99.summary()
+
+    xs = np.random.default_rng(0).normal(5.0, 1.0, 80)
+    b90 = score_ci(xs, method="bootstrap", confidence=0.90, seed=1)
+    b95 = score_ci(xs, method="bootstrap", confidence=0.95, seed=1)
+    b99 = score_ci(xs, method="bootstrap", confidence=0.99, seed=1)
+
+    assert b90.width < b95.width < b99.width