reliability-lab 0.1.2__tar.gz → 0.1.3__tar.gz

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ## Unreleased
 
+## 0.1.3
+
+Faster bootstrap.
+
+- Every scheme is faster. With 2,000 resamples of 5,000 units and 5 coders (ordinal, 3 estimates):
+  - units, percentile: 1.4 s → 0.5 s;
+  - units, BCa: 6.4 s → 0.5 s, and `compare(method="bca")`: 18 s → 0.5 s;
+  - kalpha: 5.1 s → 2.8 s;
+  - coders: 8.2 s → 1.4 s, and two_way: 10 s → 2.1 s.
+- BCa intervals no longer go through `scipy.stats.bootstrap`. The jackknife leaves each cluster out in closed form, instead of recomputing alpha from k × k resample weights, so BCa now runs on large studies: 20,000 units take 3.7 s, where they previously ran out of memory. The resamples and the interval are still the ones `scipy.stats.bootstrap` gives, and the tests check this.
+- The random draws are unchanged: a given `random_state` gives the same intervals as in 0.1.2, up to floating-point rounding.
+- The README links to the docs and examples on GitHub, so the links work on PyPI. The project links on PyPI include the changelog.
+
 ## 0.1.2
 
 Generalising over coders.

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/PKG-INFO

@@ -1,9 +1,10 @@
 Metadata-Version: 2.5
 Name: reliability-lab
-Version: 0.1.2
+Version: 0.1.3
 Summary: Study-design library for the reliability of LLM and human judgments, built on Krippendorff's alpha.
 Project-URL: Repository, https://github.com/Sathapana/reliability-lab
 Project-URL: Issues, https://github.com/Sathapana/reliability-lab/issues
+Project-URL: Changelog, https://github.com/Sathapana/reliability-lab/blob/main/CHANGELOG.md
 Author: Sathapana Chawananon
 License-Expression: BSD-3-Clause
 License-File: LICENSE
@@ -30,7 +31,7 @@ Description-Content-Type: text/markdown
 
 # reliability-lab
 
-**Status: alpha (0.1.2).** The API may still change between releases.
+**Status: alpha (0.1.3).** The API may still change between releases.
 
 reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
 
@@ -45,7 +46,7 @@ reliability-lab is a study-design library for measuring the reliability of LLM a
 - versioned JSON and Parquet reports with provenance;
 - LLM judge workflows: a trace sidecar of judge configurations, declared retry handling, and evidence specific to each design (run agreement, order swaps and position bias).
 
-Mixed human-LLM panels are next on the roadmap ([design document](docs/design/framing-v2.md)).
+Mixed human-LLM panels are next on the roadmap ([design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
 
 > **Reliability is not validity.** Coders can agree consistently and still be wrong in the same way, for example because of a flawed rubric or a bias they share. When results come from different designs (repeatability, inter-model agreement, human panels), they are reported as different quantities and are never pooled.
 
@@ -111,7 +112,7 @@ print(jk.summary())  # interval, standard error, and the most influential coders
 
 When coders are resampled, the copies of one coder are never paired with each other, since a coder always agrees with itself. These schemes need at least 3 coders and give percentile intervals.
 
-Which scheme to trust, and when, is measured by simulation in the [bootstrap coverage guidance](docs/guidance/bootstrap-coverage.md).
+Which scheme to trust, and when, is measured by simulation in the [bootstrap coverage guidance](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md).
 
 To compare two configurations of a judge, for example two prompts, select each one's rows. Both must cover the same units. Every replicate resamples the same units for both, and the interval is on the difference:
 
@@ -150,7 +151,7 @@ dev2, test2 = later_study.apply_manifest("split.json")  # same units, new judge
 test.manifest.assert_no_leakage(few_shot_units)  # raises LeakageError
 ```
 
-See [`examples/splits.py`](examples/splits.py) for a full run. Releases are described in [docs/releasing.md](docs/releasing.md).
+See [`examples/splits.py`](https://github.com/Sathapana/reliability-lab/blob/main/examples/splits.py) for a full run. Releases are described in [docs/releasing.md](https://github.com/Sathapana/reliability-lab/blob/main/docs/releasing.md).
 
 ## LLM judge workflows
 
@@ -190,7 +191,7 @@ result.trace_findings  # confounds and caveats from the trace
 
 The single-configuration designs (repeatability, prompt robustness, order robustness) can hold several configurations to compare, such as an old and a new judge. Each configuration is checked on its own. `alpha()` refuses to pool them; `study.compare(a={"condition_id": "old"}, b={"condition_id": "new"})` reports each one's alpha and diagnostics, with a paired interval on the difference.
 
-Retried calls are never resolved silently. Number the attempts with an `attempt` column, then declare `retries="last"` or `retries="first_observed"`. Coverage then reports error rates both per attempt and for the attempts that count. See [`examples/llm_judge.py`](examples/llm_judge.py).
+Retried calls are never resolved silently. Number the attempts with an `attempt` column, then declare `retries="last"` or `retries="first_observed"`. Coverage then reports error rates both per attempt and for the attempts that count. See [`examples/llm_judge.py`](https://github.com/Sathapana/reliability-lab/blob/main/examples/llm_judge.py).
 
 ## Licence
 

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/README.md

@@ -1,6 +1,6 @@
 # reliability-lab
 
-**Status: alpha (0.1.2).** The API may still change between releases.
+**Status: alpha (0.1.3).** The API may still change between releases.
 
 reliability-lab is a study-design library for measuring the reliability of LLM and human judgments. It is built on Krippendorff's alpha. Alpha implementations already exist in Python. This package adds the workflow around the coefficient:
 
@@ -15,7 +15,7 @@ reliability-lab is a study-design library for measuring the reliability of LLM a
 - versioned JSON and Parquet reports with provenance;
 - LLM judge workflows: a trace sidecar of judge configurations, declared retry handling, and evidence specific to each design (run agreement, order swaps and position bias).
 
-Mixed human-LLM panels are next on the roadmap ([design document](docs/design/framing-v2.md)).
+Mixed human-LLM panels are next on the roadmap ([design document](https://github.com/Sathapana/reliability-lab/blob/main/docs/design/framing-v2.md)).
 
 > **Reliability is not validity.** Coders can agree consistently and still be wrong in the same way, for example because of a flawed rubric or a bias they share. When results come from different designs (repeatability, inter-model agreement, human panels), they are reported as different quantities and are never pooled.
 
@@ -81,7 +81,7 @@ print(jk.summary())  # interval, standard error, and the most influential coders
 
 When coders are resampled, the copies of one coder are never paired with each other, since a coder always agrees with itself. These schemes need at least 3 coders and give percentile intervals.
 
-Which scheme to trust, and when, is measured by simulation in the [bootstrap coverage guidance](docs/guidance/bootstrap-coverage.md).
+Which scheme to trust, and when, is measured by simulation in the [bootstrap coverage guidance](https://github.com/Sathapana/reliability-lab/blob/main/docs/guidance/bootstrap-coverage.md).
 
 To compare two configurations of a judge, for example two prompts, select each one's rows. Both must cover the same units. Every replicate resamples the same units for both, and the interval is on the difference:
 
@@ -120,7 +120,7 @@ dev2, test2 = later_study.apply_manifest("split.json")  # same units, new judge
 test.manifest.assert_no_leakage(few_shot_units)  # raises LeakageError
 ```
 
-See [`examples/splits.py`](examples/splits.py) for a full run. Releases are described in [docs/releasing.md](docs/releasing.md).
+See [`examples/splits.py`](https://github.com/Sathapana/reliability-lab/blob/main/examples/splits.py) for a full run. Releases are described in [docs/releasing.md](https://github.com/Sathapana/reliability-lab/blob/main/docs/releasing.md).
 
 ## LLM judge workflows
 
@@ -160,7 +160,7 @@ result.trace_findings  # confounds and caveats from the trace
 
 The single-configuration designs (repeatability, prompt robustness, order robustness) can hold several configurations to compare, such as an old and a new judge. Each configuration is checked on its own. `alpha()` refuses to pool them; `study.compare(a={"condition_id": "old"}, b={"condition_id": "new"})` reports each one's alpha and diagnostics, with a paired interval on the difference.
 
-Retried calls are never resolved silently. Number the attempts with an `attempt` column, then declare `retries="last"` or `retries="first_observed"`. Coverage then reports error rates both per attempt and for the attempts that count. See [`examples/llm_judge.py`](examples/llm_judge.py).
+Retried calls are never resolved silently. Number the attempts with an `attempt` column, then declare `retries="last"` or `retries="first_observed"`. Coverage then reports error rates both per attempt and for the attempts that count. See [`examples/llm_judge.py`](https://github.com/Sathapana/reliability-lab/blob/main/examples/llm_judge.py).
 
 ## Licence
 

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/pyproject.toml

@@ -44,6 +44,7 @@ parquet = ["pyarrow>=15"]
 [project.urls]
 Repository = "https://github.com/Sathapana/reliability-lab"
 Issues = "https://github.com/Sathapana/reliability-lab/issues"
+Changelog = "https://github.com/Sathapana/reliability-lab/blob/main/CHANGELOG.md"
 
 [dependency-groups]
 dev = [

reliability_lab-0.1.3/src/reliability_lab/_version.py

@@ -0,0 +1 @@
+__version__ = "0.1.3"

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/src/reliability_lab/inference.py

@@ -23,15 +23,14 @@ configurations, so the interval falls on the difference alpha_b - alpha_a.
 
 from __future__ import annotations
 
-import warnings
-from collections.abc import Callable, Mapping
+from collections.abc import Mapping
 from dataclasses import dataclass
 from typing import Any
 
 import numpy as np
 import numpy.typing as npt
 import pandas as pd
-from scipy import sparse, stats
+from scipy import sparse, special, stats
 
 from reliability_lab.alpha import ORDINAL, Distance, compute_alpha, per_unit_coincidences
 from reliability_lab.data import preview
@@ -218,17 +217,41 @@ def replicate_alphas(
     distance: Distance,
 ) -> FloatArray:
     """Alpha for each row of cluster weights; NaN where alpha is undefined."""
-    w = np.atleast_2d(np.asarray(weights, dtype=float))
+    w = np.atleast_2d(np.asarray(weights))
     n_values = len(values)
+    n_flat = n_values * n_values
+    # A dense product is one BLAS call; large value domains stay sparse.
+    dense = clusters.shape[0] * n_flat <= BATCH_FLOATS
+    contributions = clusters.toarray() if dense else sparse.csr_array(clusters.T)
     out = np.empty(len(w))
-    batch = max(1, BATCH_FLOATS // max(n_values * n_values, 1))
+    batch = max(1, BATCH_FLOATS // max(n_flat, clusters.shape[0], 1))
     for start in range(0, len(w), batch):
-        wb = w[start : start + batch]
-        o = np.asarray(clusters.T @ wb.T, dtype=float).T.reshape(len(wb), n_values, n_values)
+        wb = np.asarray(w[start : start + batch], dtype=float)
+        if dense:
+            flat = wb @ contributions
+        else:
+            flat = np.asarray(contributions @ np.asfortranarray(wb.T)).T
+        o = flat.reshape(len(wb), n_values, n_values)
         out[start : start + len(wb)] = alphas_from_coincidences(o, values, distance)
     return out
 
 
+def jackknife_alphas(
+    clusters: sparse.csr_array, values: tuple[Any, ...], distance: Distance
+) -> FloatArray:
+    """Alpha without each cluster in turn: the coincidences less that cluster's."""
+    n_values = len(values)
+    n_flat = n_values * n_values
+    total = np.asarray(clusters.sum(axis=0), dtype=float).ravel()
+    out = np.empty(clusters.shape[0])
+    batch = max(1, BATCH_FLOATS // max(n_flat, 1))
+    for start in range(0, clusters.shape[0], batch):
+        part = clusters[start : start + batch].toarray()
+        o = (total - part).reshape(len(part), n_values, n_values)
+        out[start : start + len(part)] = alphas_from_coincidences(o, values, distance)
+    return out
+
+
 def alphas_from_coincidences(
     o: FloatArray, values: tuple[Any, ...], distance: Distance
 ) -> FloatArray:
@@ -285,25 +308,26 @@ def two_way_alphas(
     ``(n_u n_u' - diag n_u) / (m_u - 1)``.
     """
     v = np.atleast_2d(np.asarray(coder_weights, dtype=float))
-    n_units, _, n_values = onehot.shape
-    w = (
-        np.ones((len(v), n_units))
-        if unit_weights is None
-        else np.atleast_2d(np.asarray(unit_weights, dtype=float))
-    )
+    n_units, n_coders, n_values = onehot.shape
+    w = None if unit_weights is None else np.atleast_2d(np.asarray(unit_weights))
+    # Every sum over coders is a matrix product with the coder weights.
+    by_coder = onehot.transpose(1, 0, 2).reshape(n_coders, n_units * n_values)
+    rated = onehot.sum(axis=2).T
     out = np.empty(len(v))
     per_replicate = max(n_units * n_values * 2, n_values * n_values, 1)
     batch = max(1, BATCH_FLOATS // per_replicate)
     for start in range(0, len(v), batch):
-        vb, wb = v[start : start + batch], w[start : start + batch]
-        counts = np.einsum("ujc,rj->ruc", onehot, vb)
-        squares = np.einsum("ujc,rj->ruc", onehot, vb * vb)
-        m = counts.sum(axis=2)
-        pairs = m * m - squares.sum(axis=2)
+        vb = v[start : start + batch]
+        v2 = vb * vb
+        wb = 1.0 if w is None else np.asarray(w[start : start + batch], dtype=float)
+        counts = (vb @ by_coder).reshape(len(vb), n_units, n_values)
+        squares = (v2 @ by_coder).reshape(len(vb), n_units, n_values)
+        m = vb @ rated
+        pairs = m * m - v2 @ rated
         with np.errstate(divide="ignore", invalid="ignore"):
             scale = np.where(pairs > 0, wb * m / pairs, 0.0)
-        o = np.einsum("ru,ruc,rud->rcd", scale, counts, counts)
-        diagonal = np.einsum("ru,ruc->rc", scale, squares)
+        o = (counts * scale[:, :, None]).transpose(0, 2, 1) @ counts
+        diagonal = (scale[:, None, :] @ squares)[:, 0, :]
         o[:, np.arange(n_values), np.arange(n_values)] -= diagonal
         out[start : start + len(vb)] = alphas_from_coincidences(o, values, distance)
     return out
@@ -389,12 +413,14 @@ def kalpha_replicates(pool: KalphaPool, draws: IntArray, redraws: IntArray) -> F
     ``redraws`` (replicates x len(second_pick)) replaces a unit's second draw when
     it repeats the first.
     """
-    d = np.array(draws, dtype=np.intp)
-    if len(pool.second_pick):
-        first = d[:, pool.second_pick - 1]
-        second = d[:, pool.second_pick]
-        d[:, pool.second_pick] = np.where(first == second, redraws, second)
-    alpha = 1.0 - (pool.errors[d] * pool.weights).sum(axis=1)
+    d = np.asarray(draws, dtype=np.intp)
+    sp = pool.second_pick
+    alpha = 1.0 - np.take(pool.errors, d) @ pool.weights
+    if len(sp):
+        # Repeats are rare, so their redraws correct the sums afterwards.
+        rows, cols = np.nonzero(d[:, sp - 1] == d[:, sp])
+        redrawn = pool.errors[np.asarray(redraws)[rows, cols]] - pool.errors[d[rows, sp[cols]]]
+        np.subtract.at(alpha, rows, redrawn * pool.weights[sp[cols]])
     return np.asarray(np.maximum(alpha, -1.0), dtype=float)
 
 
@@ -452,37 +478,34 @@ def percentile_interval(
 def bca_interval(
     name: str,
     point: float,
-    statistic: Callable[[IntArray], FloatArray],
-    n_clusters: int,
-    n_resamples: int,
+    distribution: FloatArray,
+    theta_hat: float,
+    jackknife: FloatArray,
     confidence: float,
-    seed: int,
     alpha_min: float | None = None,
 ) -> Interval:
-    """BCa interval through ``scipy.stats.bootstrap`` over resampled cluster indices."""
-
-    def vectorised(idx: IntArray, axis: int = -1) -> FloatArray:
-        return statistic(np.moveaxis(np.asarray(idx, dtype=np.intp), axis, -1))
-
-    # scipy's own warning about a degenerate BCa is replaced by this library's
-    # BootstrapWarning, which names the estimate and the fallback.
-    with np.errstate(divide="ignore", invalid="ignore"), warnings.catch_warnings():
-        warnings.simplefilter("ignore", stats.DegenerateDataWarning)
-        res = stats.bootstrap(
-            (np.arange(n_clusters),),
-            vectorised,
-            n_resamples=n_resamples,
-            confidence_level=confidence,
-            method="BCa",
-            vectorized=True,
-            rng=np.random.default_rng(seed),
-        )
-    distribution = np.asarray(res.bootstrap_distribution, dtype=float)
+    """BCa interval (Efron and Tibshirani 1993, 14.3 and 15.4), as ``scipy.stats.bootstrap`` does.
+
+    ``theta_hat`` is the statistic on the full sample, computed like the replicates,
+    and ``jackknife`` the statistic without each cluster in turn.
+    """
+    n_resamples = len(distribution)
     failed = np.isnan(distribution)
-    low, high = float(res.confidence_interval.low), float(res.confidence_interval.high)
-    reason = None
-    if not (np.isfinite(low) and np.isfinite(high)):
-        reason = "the BCa interval is undefined (undefined replicates or a degenerate jackknife)"
+    z_alpha = float(special.ndtri((1.0 - confidence) / 2.0))
+    with np.errstate(divide="ignore", invalid="ignore"):
+        below_hat = (distribution < theta_hat).sum() + (distribution <= theta_hat).sum()
+        z0 = special.ndtri(below_hat / (2 * n_resamples))
+        u = jackknife.mean() - jackknife
+        a_hat = (u**3).sum() / (6.0 * (u**2).sum() ** 1.5)
+        num = z0 + np.array([z_alpha, -z_alpha])
+        levels = special.ndtr(z0 + num / (1.0 - a_hat * num))
+    low = high = np.nan
+    reason: str | None = (
+        "the BCa interval is undefined (undefined replicates or a degenerate jackknife)"
+    )
+    if not failed.any() and np.isfinite(levels).all():
+        low, high = (float(q) for q in np.quantile(distribution, levels))
+        reason = None
     defined = distribution[~failed]
     below = float((defined < alpha_min).mean()) if alpha_min is not None and len(defined) else None
     return Interval(
@@ -490,6 +513,30 @@ def bca_interval(
     )
 
 
+def bca_weights(seed: int, n_clusters: int, n_resamples: int) -> npt.NDArray[np.int64]:
+    """Cluster weights of the BCa resamples (n x clusters).
+
+    The indices are the ones ``scipy.stats.bootstrap`` draws from the same seed,
+    drawn in batches (which leaves the stream unchanged) and counted.
+    """
+    rng = np.random.default_rng(seed)
+    batch = max(1, BATCH_FLOATS // max(n_clusters, 1))
+    parts = []
+    for start in range(0, n_resamples, batch):
+        size = min(batch, n_resamples - start)
+        idx = rng.integers(0, n_clusters, (size, n_clusters))
+        parts.append(weights_from_indices(idx, n_clusters))
+    return np.concatenate(parts)
+
+
+def bca_parts(
+    clusters: sparse.csr_array, values: tuple[Any, ...], distance: Distance
+) -> tuple[float, FloatArray]:
+    """The full-sample statistic and the leave-one-cluster-out jackknife for BCa."""
+    theta_hat = replicate_alphas(clusters, np.ones(clusters.shape[0]), values, distance)[0]
+    return float(theta_hat), jackknife_alphas(clusters, values, distance)
+
+
 def weights_from_indices(idx: IntArray, n_clusters: int) -> npt.NDArray[np.int64]:
     """Count cluster indices along the last axis: (..., k) -> (prod(...), n_clusters)."""
     flat = np.asarray(idx, dtype=np.intp).reshape(-1, np.shape(idx)[-1])
@@ -498,17 +545,6 @@ def weights_from_indices(idx: IntArray, n_clusters: int) -> npt.NDArray[np.int64
     return counts.reshape(len(flat), n_clusters)
 
 
-def _alpha_statistic(
-    clusters: sparse.csr_array, n_clusters: int, values: tuple[Any, ...], distance: Distance
-) -> Callable[[IntArray], FloatArray]:
-    def statistic(idx: IntArray) -> FloatArray:
-        lead = np.shape(idx)[:-1]
-        w = weights_from_indices(idx, n_clusters)
-        return replicate_alphas(clusters, w, values, distance).reshape(lead)
-
-    return statistic
-
-
 # -- entry points ---------------------------------------------------------------------
 
 
@@ -624,35 +660,27 @@ def bootstrap(
             recorded.append(emit(BootstrapWarning, level_warning, stacklevel=3))
         if method == "bca" and strata:
             raise DesignError(
-                "BCa intervals cannot resample within strata (scipy.stats.bootstrap has no "
-                "strata); use method='percentile'"
+                "BCa intervals cannot resample within strata; use method='percentile'"
             )
         plan = cluster_plan(sample, level_used, strata)
         n_clusters = plan.n_clusters
         weights = (
             draw_weights(np.random.default_rng(seed), plan, n_resamples)
             if method == "percentile"
-            else None
+            else bca_weights(seed, n_clusters, n_resamples)
         )
         for name in names:
             values = sample.inputs[name].values
             clusters = cluster_coincidences(sample, name, plan)
-            if weights is not None:
-                distribution = replicate_alphas(clusters, weights, values, sample.distance)
+            distribution = replicate_alphas(clusters, weights, values, sample.distance)
+            if method == "percentile":
                 intervals[name] = percentile_interval(
                     name, points[name], distribution, confidence, alpha_min
                 )
             else:
-                statistic = _alpha_statistic(clusters, n_clusters, values, sample.distance)
+                theta_hat, jackknife = bca_parts(clusters, values, sample.distance)
                 intervals[name] = bca_interval(
-                    name,
-                    points[name],
-                    statistic,
-                    n_clusters,
-                    n_resamples,
-                    confidence,
-                    seed,
-                    alpha_min,
+                    name, points[name], distribution, theta_hat, jackknife, confidence, alpha_min
                 )
         notes.append(UNITS_NOTE.format(level=level_used, within=" within strata" if strata else ""))
         if len(names) > 1:
@@ -795,7 +823,7 @@ def compare(
     weights = (
         draw_weights(np.random.default_rng(seed), plan_a, n_resamples)
         if method == "percentile"
-        else None
+        else bca_weights(seed, plan_a.n_clusters, n_resamples)
     )
 
     arm_intervals: dict[str, dict[str, Interval]] = {"a": {}, "b": {}}
@@ -806,30 +834,22 @@ def compare(
         clusters_a = cluster_coincidences(sample_a, name, plan_a)
         clusters_b = cluster_coincidences(sample_b, name, plan_b)
         values_a, values_b = sample_a.inputs[name].values, sample_b.inputs[name].values
-        if weights is not None:
-            dist_a = replicate_alphas(clusters_a, weights, values_a, sample_a.distance)
-            dist_b = replicate_alphas(clusters_b, weights, values_b, sample_b.distance)
+        dist_a = replicate_alphas(clusters_a, weights, values_a, sample_a.distance)
+        dist_b = replicate_alphas(clusters_b, weights, values_b, sample_b.distance)
+        if method == "percentile":
             arm_intervals["a"][name] = percentile_interval(name, point_a, dist_a, confidence)
             arm_intervals["b"][name] = percentile_interval(name, point_b, dist_b, confidence)
             differences[name] = percentile_interval(
                 name, point_b - point_a, dist_b - dist_a, confidence
             )
             continue
-        k = plan_a.n_clusters
-        stat_a = _alpha_statistic(clusters_a, k, values_a, sample_a.distance)
-        stat_b = _alpha_statistic(clusters_b, k, values_b, sample_b.distance)
-
-        def stat_diff(
-            idx: IntArray,
-            stat_a: Callable[[IntArray], FloatArray] = stat_a,
-            stat_b: Callable[[IntArray], FloatArray] = stat_b,
-        ) -> FloatArray:
-            return stat_b(idx) - stat_a(idx)
-
-        args = (k, n_resamples, confidence, seed)
-        arm_intervals["a"][name] = bca_interval(name, point_a, stat_a, *args)
-        arm_intervals["b"][name] = bca_interval(name, point_b, stat_b, *args)
-        differences[name] = bca_interval(name, point_b - point_a, stat_diff, *args)
+        theta_a, jack_a = bca_parts(clusters_a, values_a, sample_a.distance)
+        theta_b, jack_b = bca_parts(clusters_b, values_b, sample_b.distance)
+        arm_intervals["a"][name] = bca_interval(name, point_a, dist_a, theta_a, jack_a, confidence)
+        arm_intervals["b"][name] = bca_interval(name, point_b, dist_b, theta_b, jack_b, confidence)
+        differences[name] = bca_interval(
+            name, point_b - point_a, dist_b - dist_a, theta_b - theta_a, jack_b - jack_a, confidence
+        )
 
     recorded.extend(_interval_warnings(differences, n_resamples))
     within = " within strata" if strata else ""

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/tests/test_bca.py

@@ -6,10 +6,11 @@ from fixtures.krippendorff2011 import FOUR_CODERS_MISSING
 from helpers import human_study
 from scipy import stats
 
-from reliability_lab import BootstrapWarning, DesignError
+from reliability_lab import AlphaResult, BootstrapWarning, DesignError
 from reliability_lab.inference import (
     cluster_coincidences,
     cluster_plan,
+    jackknife_alphas,
     replicate_alphas,
     weights_from_indices,
 )
@@ -17,6 +18,35 @@ from reliability_lab.inference import (
 F4 = FOUR_CODERS_MISSING
 
 
+def direct_scipy_bca(
+    result: AlphaResult, name: str, n_resamples: int, seed: int
+) -> tuple[float, float]:
+    """The BCa interval of a direct ``scipy.stats.bootstrap`` call over cluster indices."""
+    sample = result._sample
+    assert sample is not None
+    plan = cluster_plan(sample, "unit", strata=False)
+    clusters = cluster_coincidences(sample, name, plan)
+    values = sample.inputs[name].values
+
+    def statistic(idx: np.ndarray, axis: int = -1) -> np.ndarray:
+        idx = np.moveaxis(idx, axis, -1)
+        w = weights_from_indices(idx, plan.n_clusters)
+        return replicate_alphas(clusters, w, values, sample.distance).reshape(idx.shape[:-1])
+
+    with np.errstate(divide="ignore", invalid="ignore"), warnings.catch_warnings():
+        warnings.simplefilter("ignore", stats.DegenerateDataWarning)
+        direct = stats.bootstrap(
+            (np.arange(plan.n_clusters),),
+            statistic,
+            n_resamples=n_resamples,
+            confidence_level=0.95,
+            method="BCa",
+            vectorized=True,
+            rng=np.random.default_rng(seed),
+        )
+    return float(direct.confidence_interval.low), float(direct.confidence_interval.high)
+
+
 def test_bca_matches_a_direct_scipy_call() -> None:
     result = human_study(F4["matrix"], "interval", F4["values"]).alpha()
     with warnings.catch_warnings():
@@ -25,30 +55,63 @@ def test_bca_matches_a_direct_scipy_call() -> None:
     assert not boot.experimental
     assert "[experimental]" not in boot.summary()
 
+    low, high = direct_scipy_bca(result, "as_missing", 999, 5)
+    interval = boot["as_missing"]
+    assert interval.low == pytest.approx(low)
+    assert interval.high == pytest.approx(high)
+    assert interval.low < interval.point < interval.high
+
+
+@pytest.mark.filterwarnings("ignore::reliability_lab.CategorySkewWarning")
+@pytest.mark.filterwarnings("ignore::reliability_lab.BootstrapWarning")
+@pytest.mark.parametrize(
+    ("matrix", "n_resamples", "seed"),
+    [
+        # Undefined replicates: scipy gives no interval.
+        ({"a": ["x"] * 19 + ["y"], "b": ["x"] * 20}, 500, 1),
+        ({"a": ["x"] * 9 + ["y"], "b": ["x"] * 9 + ["y"]}, 50, 3),
+        # Every replicate defined in a short run, undefined ones in a long run.
+        (
+            {"a": ["x"] * 5 + ["y"] * 5, "b": ["x"] * 4 + ["y"] * 6, "c": ["x"] * 6 + ["y"] * 4},
+            50,
+            4,
+        ),
+        (
+            {"a": ["x"] * 5 + ["y"] * 5, "b": ["x"] * 4 + ["y"] * 6, "c": ["x"] * 6 + ["y"] * 4},
+            500,
+            4,
+        ),
+    ],
+)
+def test_bca_matches_scipy_on_degenerate_data(
+    matrix: dict[str, list[str]], n_resamples: int, seed: int
+) -> None:
+    result = human_study(matrix, "nominal", ["x", "y"]).alpha()
+    boot = result.bootstrap(
+        scheme="units", method="bca", n_resamples=n_resamples, random_state=seed
+    )
+    low, high = direct_scipy_bca(result, "as_missing", n_resamples, seed)
+    interval = boot["as_missing"]
+    np.testing.assert_allclose([interval.low, interval.high], [low, high], equal_nan=True)
+    assert interval.defined == np.isfinite(low)
+
+
+@pytest.mark.parametrize("budget", [None, 20])
+def test_jackknife_alphas_leave_out_each_cluster(
+    budget: int | None, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    if budget is not None:
+        monkeypatch.setattr("reliability_lab.inference.BATCH_FLOATS", budget)
+    result = human_study(F4["matrix"], "interval", F4["values"]).alpha()
     sample = result._sample
     assert sample is not None
     plan = cluster_plan(sample, "unit", strata=False)
     clusters = cluster_coincidences(sample, "as_missing", plan)
     values = sample.inputs["as_missing"].values
-
-    def statistic(idx: np.ndarray, axis: int = -1) -> np.ndarray:
-        idx = np.moveaxis(idx, axis, -1)
-        w = weights_from_indices(idx, plan.n_clusters)
-        return replicate_alphas(clusters, w, values, sample.distance).reshape(idx.shape[:-1])
-
-    direct = stats.bootstrap(
-        (np.arange(plan.n_clusters),),
-        statistic,
-        n_resamples=999,
-        confidence_level=0.95,
-        method="BCa",
-        vectorized=True,
-        rng=np.random.default_rng(5),
-    )
-    interval = boot["as_missing"]
-    assert interval.low == pytest.approx(direct.confidence_interval.low)
-    assert interval.high == pytest.approx(direct.confidence_interval.high)
-    assert interval.low < interval.point < interval.high
+    k = plan.n_clusters
+    left_out = replicate_alphas(clusters, np.ones((k, k)) - np.eye(k), values, sample.distance)
+    jackknife = jackknife_alphas(clusters, values, sample.distance)
+    np.testing.assert_allclose(jackknife, left_out, atol=1e-12, equal_nan=True)
 
 
 @pytest.mark.filterwarnings("ignore::reliability_lab.CategorySkewWarning")

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/tests/test_compare.py

@@ -1,12 +1,22 @@
 import json
+from collections.abc import Callable
 from typing import Any
 
 import numpy as np
 import pandas as pd
 import pytest
 from reference import reference_alpha
+from scipy import stats
 
 from reliability_lab import DesignError, RatingFrame, ReliabilityStudy
+from reliability_lab.inference import (
+    ClusterPlan,
+    StudySample,
+    cluster_coincidences,
+    cluster_plan,
+    replicate_alphas,
+    weights_from_indices,
+)
 
 pytestmark = pytest.mark.filterwarnings("ignore::reliability_lab.CategorySkewWarning")
 
@@ -135,3 +145,44 @@ def test_bca_comparison() -> None:
     assert not [w for w in diff.warnings if w.category == "BootstrapWarning"]
     d = diff["as_missing"]
     assert d.low <= d.point <= d.high
+
+
+def test_bca_comparison_matches_direct_scipy_calls() -> None:
+    diff = study_of(panel_matrix()).compare(A, B, method="bca", n_resamples=500, random_state=2)
+    sample_a, sample_b = diff.result_a._sample, diff.result_b._sample
+    assert sample_a is not None and sample_b is not None
+    plan_a = cluster_plan(sample_a, "unit", strata=False)
+    plan_b = cluster_plan(sample_b, "unit", strata=False, universe=plan_a.labels)
+    k = plan_a.n_clusters
+
+    def arm(sample: StudySample, plan: ClusterPlan) -> Callable[[np.ndarray], np.ndarray]:
+        clusters = cluster_coincidences(sample, "as_missing", plan)
+        values = sample.inputs["as_missing"].values
+
+        def statistic(idx: np.ndarray, axis: int = -1) -> np.ndarray:
+            idx = np.moveaxis(idx, axis, -1)
+            w = weights_from_indices(idx, k)
+            return replicate_alphas(clusters, w, values, sample.distance).reshape(idx.shape[:-1])
+
+        return statistic
+
+    stat_a, stat_b = arm(sample_a, plan_a), arm(sample_b, plan_b)
+
+    def stat_diff(idx: np.ndarray, axis: int = -1) -> np.ndarray:
+        return stat_b(idx, axis) - stat_a(idx, axis)
+
+    for statistic, interval in [
+        (stat_a, diff.arm_a["as_missing"]),
+        (stat_b, diff.arm_b["as_missing"]),
+        (stat_diff, diff["as_missing"]),
+    ]:
+        direct = stats.bootstrap(
+            (np.arange(k),),
+            statistic,
+            n_resamples=500,
+            method="BCa",
+            vectorized=True,
+            rng=np.random.default_rng(2),
+        )
+        assert interval.low == pytest.approx(direct.confidence_interval.low)
+        assert interval.high == pytest.approx(direct.confidence_interval.high)

{reliability_lab-0.1.2 → reliability_lab-0.1.3}/tests/test_inference_units.py

@@ -76,6 +76,24 @@ def test_vectorised_units_scheme_matches_naive_loop(level: str, resample: str) -
     np.testing.assert_allclose(fast, slow, atol=1e-12, equal_nan=True)
 
 
+@pytest.mark.parametrize("level", LEVELS)
+def test_sparse_batched_product_matches_naive_loop(
+    level: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    # A tiny budget keeps the cluster contributions sparse, with a few replicates per batch.
+    monkeypatch.setattr("reliability_lab.inference.BATCH_FLOATS", 50)
+    matrix, study = nested_study(level)
+    sample = study.alpha()._sample
+    assert sample is not None
+    plan = cluster_plan(sample, "unit", strata=False)
+    weights = draw_weights(np.random.default_rng(11), plan, 40)
+    clusters = cluster_coincidences(sample, "as_missing", plan)
+    fast = replicate_alphas(clusters, weights, sample.inputs["as_missing"].values, sample.distance)
+    mapping = {int(label): i for i, label in enumerate(sample.unit_labels)}
+    slow = naive_units_alphas(matrix, level, VALUES, mapping, weights)
+    np.testing.assert_allclose(fast, slow, atol=1e-12, equal_nan=True)
+
+
 def test_custom_marginal_dependent_distance_matches_naive_loop() -> None:
     custom = Distance("my_ordinal", ORDINAL.fn, depends_on_marginals=True)
     matrix, study = nested_study(custom)

reliability_lab-0.1.2/src/reliability_lab/_version.py

@@ -1 +0,0 @@
-__version__ = "0.1.2"