camt-test 0.1.0__tar.gz

camt_test-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aqib Gul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

camt_test-0.1.0/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: camt-test
+Version: 0.1.0
+Summary: CAMT, the clustered AUC maximum test: patient-clustered, selection-adjusted test that the best of K models beats a baseline, with an audit of published claims
+Author: Aqib Gul
+License: MIT
+Keywords: AUC,clustered data,winner's curse,multiple comparisons,model comparison,medical AI
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: scipy>=1.9
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# camt: the clustered AUC maximum test (CAMT)
+
+Tests of the claim that **the best of K models has a higher AUC than a baseline** on a test set that contains several images or
+records per patient, and a **summary-statistics audit** of published claims of this kind.
+
+Two errors act together when the winner among K candidate models is compared with a baseline on one shared test set:
+
+1. observations from one patient are correlated, so a test that treats every image as independent understates the uncertainty;
+2. the winner is chosen among K candidates, so its apparent advantage is inflated even if all models are equally good.
+
+`camt` implements CAMT, a test that accounts for both. Labels may vary within a patient. The method is described in the accompanying
+manuscript (in preparation).
+
+Status: research software, version 0.1.0.
+
+## Installation
+
+```bash
+pip install camt-test      # from PyPI; the import name is `camt`
+pip install .            # or from the repository root; requires Python >= 3.9, numpy and scipy
+pip install ".[test]"    # also installs pytest
+```
+
+## Quick start
+
+```python
+from camt import camt_test
+
+# scores:   array (n_images, 1 + K), one column per model; column 0 is the baseline
+# labels:   array (n_images,) of 0/1 image-level labels
+# patients: array (n_images,) of patient identifiers (the clusters)
+result = camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=1)
+print(result.summary())
+```
+
+```
+best of 5 candidates vs baseline (column 0); best = column 1
+  AUC best 0.841 vs baseline 0.782 (difference +0.058)
+  500 images, 100 patients (47 with positives, 53 with negatives, 0 with both)
+  p (patient-clustered, adjusted for K) = 0.1685
+  p (pruned) = 0.1585 | clustering only = 0.0490 | selection only = 0.0310 | naive = 0.0068
+```
+
+In this simulated example no candidate is truly better, and the naive analysis reports p = 0.007. A complete script is in
+`examples/quickstart.py`.
+
+### Which p-value to report
+
+| attribute | meaning |
+|---|---|
+| `p_value` | **the proposed test**: patient-clustered, adjusted for K, with the small-class calibration. Use this one. |
+| `p_value_pruned` | as above, but candidates that are clearly worse than the baseline are dropped from the null distribution. More power when many candidates are poor; valid pointwise only. |
+| `p_naive` | what an image-level test of the best candidate would give (for comparison). |
+| `p_clustering_only` | patient-clustered, but no adjustment for K. |
+| `p_selection_only` | adjusted for K, but images treated as independent. |
+
+All p-values are one-sided; the null hypothesis is that no candidate is better than the baseline.
+
+## Functions
+
+| function | purpose |
+|---|---|
+| `camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=None, influence_fn=None)` | the test; returns a `CamtResult` |
+| `design_effect(scores, labels, patients, baseline=0)` | Kish cluster size, design effect and within-patient correlation of the AUC influence function, for each model and for each difference from the baseline |
+| `threshold_influence(thresholds, kind="balanced")` | influence function for sensitivity, specificity or balanced accuracy at operating points fixed in advance; pass it as `influence_fn` |
+| `audit_claim(p_naive, patients, images, K, rho=...)` | adjust a published p-value for clustering and for K, using only summary numbers |
+| `sensitivity_table(...)` | `audit_claim` over a range of within-patient correlations |
+| `hanley_mcneil_se`, `rho_from_patient_level_ci` | helpers for the audit |
+
+### Metrics at a fixed operating point
+
+For balanced accuracy at thresholds chosen in advance (for example on the training patients):
+
+```python
+from camt import camt_test, threshold_influence
+result = camt_test(scores, labels, patients, influence_fn=threshold_influence(thresholds, kind="balanced"))
+```
+
+Thresholds estimated on the test sample itself are not covered.
+
+### Auditing a published claim
+
+```python
+from camt import audit_claim
+a = audit_claim(p_naive=0.0004, patients=40, images=3900, K=10, rho=0.1)
+a["p_bonferroni"]   # adjusted p-value if the within-patient correlation is 0.1 and 10 models were tried
+a["rho_star"]       # correlation above which the clustered p-value alone exceeds 0.05
+```
+
+The within-patient correlation cannot be recovered from summary data, so report the result over a range of values
+(`sensitivity_table`) and treat `rho_star` as the quantity of interest. The audit is valid if the correlation is not understated and
+K is the true number of models tried.
+
+## Assumptions and limits
+
+- **K must be the true number of candidates examined**, and the baseline must be fixed in advance. Pre-specifying one primary model
+  (K = 1) removes the selection cost.
+- The target is the **fitted models** on patients like the test patients, not the training algorithm. Test patients must be disjoint
+  from the training data.
+- The test assumes independent patients, a bounded number of images per patient and a moderate number of candidates.
+- The test can be **liberal** when fewer than about 20 patients belong to the smaller class and many candidates are screened, and
+  when a few patients contribute most of the images. The function emits a warning in both situations. The cut-off for the second
+  warning (fewer than 100 effective clusters, that is, images divided by the Kish cluster size) is a heuristic and has not been
+  validated.
+- Candidates whose scores give a zero-variance difference from the baseline are dropped.
+- Metrics whose threshold is estimated on the test sample are not covered.
+
+## Reproducing the behaviour of the test
+
+```bash
+pytest                                          # unit tests, about 25 seconds
+python examples/reproduce_size.py               # false-positive rates in simulation (a few minutes)
+python examples/reproduce_size.py --reps 200    # quick version
+```
+
+`examples/reproduce_size.py` simulates test sets in which all models are equal and prints the false-positive rate of the naive,
+clustering-only, selection-only and proposed procedures for K = 1, 5 and 20, at a nominal level of 5%.
+
+## Citation
+
+If you use this software, please cite it (see `CITATION.cff`) together with the accompanying manuscript:
+
+> Gul A. Valid inference for "best of K" model comparisons on patient-clustered diagnostic data. Manuscript in preparation.
+
+The archived release and its DOI are given in `CITATION.cff` once available.
+
+## Licence
+
+MIT. See `LICENSE`.

camt_test-0.1.0/README.md

@@ -0,0 +1,129 @@
+# camt: the clustered AUC maximum test (CAMT)
+
+Tests of the claim that **the best of K models has a higher AUC than a baseline** on a test set that contains several images or
+records per patient, and a **summary-statistics audit** of published claims of this kind.
+
+Two errors act together when the winner among K candidate models is compared with a baseline on one shared test set:
+
+1. observations from one patient are correlated, so a test that treats every image as independent understates the uncertainty;
+2. the winner is chosen among K candidates, so its apparent advantage is inflated even if all models are equally good.
+
+`camt` implements CAMT, a test that accounts for both. Labels may vary within a patient. The method is described in the accompanying
+manuscript (in preparation).
+
+Status: research software, version 0.1.0.
+
+## Installation
+
+```bash
+pip install camt-test      # from PyPI; the import name is `camt`
+pip install .            # or from the repository root; requires Python >= 3.9, numpy and scipy
+pip install ".[test]"    # also installs pytest
+```
+
+## Quick start
+
+```python
+from camt import camt_test
+
+# scores:   array (n_images, 1 + K), one column per model; column 0 is the baseline
+# labels:   array (n_images,) of 0/1 image-level labels
+# patients: array (n_images,) of patient identifiers (the clusters)
+result = camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=1)
+print(result.summary())
+```
+
+```
+best of 5 candidates vs baseline (column 0); best = column 1
+  AUC best 0.841 vs baseline 0.782 (difference +0.058)
+  500 images, 100 patients (47 with positives, 53 with negatives, 0 with both)
+  p (patient-clustered, adjusted for K) = 0.1685
+  p (pruned) = 0.1585 | clustering only = 0.0490 | selection only = 0.0310 | naive = 0.0068
+```
+
+In this simulated example no candidate is truly better, and the naive analysis reports p = 0.007. A complete script is in
+`examples/quickstart.py`.
+
+### Which p-value to report
+
+| attribute | meaning |
+|---|---|
+| `p_value` | **the proposed test**: patient-clustered, adjusted for K, with the small-class calibration. Use this one. |
+| `p_value_pruned` | as above, but candidates that are clearly worse than the baseline are dropped from the null distribution. More power when many candidates are poor; valid pointwise only. |
+| `p_naive` | what an image-level test of the best candidate would give (for comparison). |
+| `p_clustering_only` | patient-clustered, but no adjustment for K. |
+| `p_selection_only` | adjusted for K, but images treated as independent. |
+
+All p-values are one-sided; the null hypothesis is that no candidate is better than the baseline.
+
+## Functions
+
+| function | purpose |
+|---|---|
+| `camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=None, influence_fn=None)` | the test; returns a `CamtResult` |
+| `design_effect(scores, labels, patients, baseline=0)` | Kish cluster size, design effect and within-patient correlation of the AUC influence function, for each model and for each difference from the baseline |
+| `threshold_influence(thresholds, kind="balanced")` | influence function for sensitivity, specificity or balanced accuracy at operating points fixed in advance; pass it as `influence_fn` |
+| `audit_claim(p_naive, patients, images, K, rho=...)` | adjust a published p-value for clustering and for K, using only summary numbers |
+| `sensitivity_table(...)` | `audit_claim` over a range of within-patient correlations |
+| `hanley_mcneil_se`, `rho_from_patient_level_ci` | helpers for the audit |
+
+### Metrics at a fixed operating point
+
+For balanced accuracy at thresholds chosen in advance (for example on the training patients):
+
+```python
+from camt import camt_test, threshold_influence
+result = camt_test(scores, labels, patients, influence_fn=threshold_influence(thresholds, kind="balanced"))
+```
+
+Thresholds estimated on the test sample itself are not covered.
+
+### Auditing a published claim
+
+```python
+from camt import audit_claim
+a = audit_claim(p_naive=0.0004, patients=40, images=3900, K=10, rho=0.1)
+a["p_bonferroni"]   # adjusted p-value if the within-patient correlation is 0.1 and 10 models were tried
+a["rho_star"]       # correlation above which the clustered p-value alone exceeds 0.05
+```
+
+The within-patient correlation cannot be recovered from summary data, so report the result over a range of values
+(`sensitivity_table`) and treat `rho_star` as the quantity of interest. The audit is valid if the correlation is not understated and
+K is the true number of models tried.
+
+## Assumptions and limits
+
+- **K must be the true number of candidates examined**, and the baseline must be fixed in advance. Pre-specifying one primary model
+  (K = 1) removes the selection cost.
+- The target is the **fitted models** on patients like the test patients, not the training algorithm. Test patients must be disjoint
+  from the training data.
+- The test assumes independent patients, a bounded number of images per patient and a moderate number of candidates.
+- The test can be **liberal** when fewer than about 20 patients belong to the smaller class and many candidates are screened, and
+  when a few patients contribute most of the images. The function emits a warning in both situations. The cut-off for the second
+  warning (fewer than 100 effective clusters, that is, images divided by the Kish cluster size) is a heuristic and has not been
+  validated.
+- Candidates whose scores give a zero-variance difference from the baseline are dropped.
+- Metrics whose threshold is estimated on the test sample are not covered.
+
+## Reproducing the behaviour of the test
+
+```bash
+pytest                                          # unit tests, about 25 seconds
+python examples/reproduce_size.py               # false-positive rates in simulation (a few minutes)
+python examples/reproduce_size.py --reps 200    # quick version
+```
+
+`examples/reproduce_size.py` simulates test sets in which all models are equal and prints the false-positive rate of the naive,
+clustering-only, selection-only and proposed procedures for K = 1, 5 and 20, at a nominal level of 5%.
+
+## Citation
+
+If you use this software, please cite it (see `CITATION.cff`) together with the accompanying manuscript:
+
+> Gul A. Valid inference for "best of K" model comparisons on patient-clustered diagnostic data. Manuscript in preparation.
+
+The archived release and its DOI are given in `CITATION.cff` once available.
+
+## Licence
+
+MIT. See `LICENSE`.

camt_test-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "camt-test"
+version = "0.1.0"
+description = "CAMT, the clustered AUC maximum test: patient-clustered, selection-adjusted test that the best of K models beats a baseline, with an audit of published claims"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "Aqib Gul" }]
+dependencies = ["numpy>=1.23", "scipy>=1.9"]
+keywords = ["AUC", "clustered data", "winner's curse", "multiple comparisons", "model comparison", "medical AI"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering",
+]
+
+[project.optional-dependencies]
+test = ["pytest"]
+
+# [project.urls]
+# Repository = "add the repository URL here"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

camt_test-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

camt_test-0.1.0/src/camt/__init__.py

@@ -0,0 +1,7 @@
+"""camt: the clustered AUC maximum test (CAMT), a patient-clustered, selection-adjusted test of "the best of K models beats a baseline"."""
+from .audit import audit_claim, hanley_mcneil_se, rho_from_patient_level_ci, sensitivity_table
+from .core import CamtResult, auc_influence, camt_test, design_effect, threshold_influence
+
+__version__ = "0.1.0"
+__all__ = ["camt_test", "CamtResult", "design_effect", "auc_influence", "threshold_influence", "audit_claim",
+           "sensitivity_table", "hanley_mcneil_se", "rho_from_patient_level_ci", "__version__"]

camt_test-0.1.0/src/camt/audit.py

@@ -0,0 +1,56 @@
+"""Summary-statistics audit of a published "best of K models beats the baseline" claim (see the accompanying manuscript).
+
+Inputs are the winner's naive one-sided p-value, the numbers of patients and images, and the number of models tried, K.
+"""
+from __future__ import annotations
+
+import numpy as np
+from scipy.stats import norm
+from scipy.stats import t as tdist
+
+__all__ = ["audit_claim", "sensitivity_table", "hanley_mcneil_se", "rho_from_patient_level_ci"]
+
+
+def audit_claim(p_naive, patients, images, K, rho=0.2, pos_patients=None, neg_patients=None, sided="one", alpha=0.05, image_sq_sum=None):
+    """Adjust a published p-value for clustering (given rho) and for the number of models tried.
+
+    ``sided="two"`` halves a two-sided p-value. ``image_sq_sum`` is the sum of squared images per patient if known (unequal
+    cluster sizes); otherwise equal cluster sizes are assumed. Passing the numbers of positive and negative patients switches
+    on the small-class t-calibration.
+    """
+    z = norm.isf(p_naive if sided == "one" else p_naive / 2)
+    m = images / patients if image_sq_sum is None else image_sq_sum / images
+    nu = None if pos_patients is None else max(min(pos_patients, neg_patients) - 1, 1)
+    sf = (lambda x: tdist.sf(x, nu)) if nu else norm.sf
+    crit = tdist.isf(alpha, nu) if nu else norm.isf(alpha)
+    deff = 1 + (m - 1) * rho
+    p_c = float(sf(z / np.sqrt(deff)))
+    rho_star = max(((z / crit) ** 2 - 1) / (m - 1), 0.0) if m > 1 else np.inf
+    return dict(z_naive=float(z), m=float(m), deff=float(deff), p_cluster=p_c, p_bonferroni=min(1.0, K * p_c), p_independent=1 - (1 - p_c) ** K,
+                rho_star=float(rho_star), K_star_bonferroni=int(alpha // p_c) if p_c > 0 else np.inf,
+                K_star_independent=int(np.log(1 - alpha) // np.log(1 - p_c)) if 0 < p_c < alpha else (0 if p_c >= alpha else np.inf))
+
+
+def sensitivity_table(p_naive, patients, images, K, rhos=(0.0, 0.05, 0.1, 0.2, 0.3, 0.5), **kw):
+    """List of :func:`audit_claim` results over a range of within-patient correlations."""
+    return [dict(rho=r, **audit_claim(p_naive, patients, images, K, rho=r, **kw)) for r in rhos]
+
+
+def hanley_mcneil_se(auc, n_pos_images, n_neg_images):
+    """Image-level (independence) standard error of a single AUC from summary numbers (Hanley and McNeil 1982)."""
+    q1, q2 = auc / (2 - auc), 2 * auc ** 2 / (1 + auc)
+    var = (auc * (1 - auc) + (n_pos_images - 1) * (q1 - auc ** 2) + (n_neg_images - 1) * (q2 - auc ** 2)) / (n_pos_images * n_neg_images)
+    return float(np.sqrt(var))
+
+
+def rho_from_patient_level_ci(auc, ci_low, ci_high, n_pos_images, n_neg_images, patients, images, image_sq_sum=None):
+    """Within-patient correlation of a SINGLE model's AUC implied by a reported patient-level 95% interval.
+
+    On BreakHis this overstates the correlation by about 1.5-fold, because the Hanley-McNeil standard error is smaller than
+    the DeLong one, and the correlation for a paired difference is usually lower than that of a single AUC. Use it as a loose
+    upper bound, not as an estimate.
+    """
+    se_rep = (ci_high - ci_low) / (2 * 1.96)
+    deff = (se_rep / hanley_mcneil_se(auc, n_pos_images, n_neg_images)) ** 2
+    m = images / patients if image_sq_sum is None else image_sq_sum / images
+    return float(max((deff - 1) / (m - 1), 0.0)), float(deff)

camt_test-0.1.0/src/camt/core.py

@@ -0,0 +1,224 @@
+"""Patient-clustered, selection-adjusted test of "the best of K models beats a baseline" (see the accompanying manuscript)."""
+from __future__ import annotations
+
+import warnings
+from dataclasses import dataclass, field
+
+import numpy as np
+from scipy.stats import norm, rankdata
+from scipy.stats import t as tdist
+
+__all__ = ["CamtResult", "camt_test", "auc_influence", "threshold_influence", "design_effect"]
+
+MIN_MINORITY_PATIENTS = 20          # below this, with many candidates, the test is liberal (paper, Section 5.3)
+MIN_EFFECTIVE_CLUSTERS = 100        # heuristic flag for very unequal cluster sizes (paper, Section 6.2); not a validated threshold
+
+
+def auc_influence(scores, labels):
+    """AUC of every column of ``scores`` and the per-image influence function (n x M). Ties receive average ranks."""
+    S = np.asarray(scores, dtype=float)
+    y = np.asarray(labels).astype(int)
+    pos = y == 1
+    n1, n0 = pos.sum(), (~pos).sum()
+    r_all = rankdata(S, axis=0)
+    V = np.empty_like(S, dtype=float)
+    V[pos] = (r_all[pos] - rankdata(S[pos], axis=0)) / n0
+    V[~pos] = 1 - (r_all[~pos] - rankdata(S[~pos], axis=0)) / n1
+    theta = V[pos].mean(0)
+    infl = np.where(pos[:, None], (V - theta) / n1, (V - theta) / n0)
+    return theta, infl
+
+
+def threshold_influence(thresholds, kind="balanced"):
+    """Influence-function builder for a metric at operating points fixed in advance (one threshold per model column).
+
+    ``kind`` is "balanced" (mean of sensitivity and specificity), "sensitivity" or "specificity". The thresholds must not be
+    estimated from the test sample; thresholds chosen on the test sample add a quantile term that is not covered.
+    Pass the returned function as ``influence_fn`` to :func:`camt_test`.
+    """
+    thr = np.asarray(thresholds, dtype=float)
+
+    def fn(S, y):
+        pos = y == 1
+        n1, n0 = pos.sum(), (~pos).sum()
+        flag = S > thr[None, :]
+        sens = flag[pos].mean(0)
+        spec = (~flag[~pos]).mean(0)
+        w1, w0 = {"balanced": (0.5, 0.5), "sensitivity": (1.0, 0.0), "specificity": (0.0, 1.0)}[kind]
+        theta = w1 * sens + w0 * spec
+        infl = np.zeros_like(S, dtype=float)
+        infl[pos] = w1 * (flag[pos] - sens) / n1
+        infl[~pos] = w0 * ((~flag[~pos]) - spec) / n0
+        return theta, infl
+
+    return fn
+
+
+def _max_pvalue(G, se, z_obs, rng, n_boot, chunk=500):
+    """P(max_k sum_c xi_c G_ck / se_k >= z_obs) for independent standard normal xi_c."""
+    Gs = G / se
+    hits, done = 0, 0
+    while done < n_boot:
+        b = min(chunk, n_boot - done)
+        hits += ((rng.standard_normal((b, G.shape[0])) @ Gs).max(1) >= z_obs).sum()
+        done += b
+    return (hits + 1) / (n_boot + 1)
+
+
+@dataclass
+class CamtResult:
+    """Result of :func:`camt_test`. All p-values are one-sided (H0: no candidate is better than the baseline)."""
+
+    p_value: float                      # patient-clustered, adjusted for the number of candidates, small-class calibration
+    p_value_pruned: float               # same, with clearly inferior candidates dropped from the null (pointwise valid only)
+    p_naive: float                      # images treated as independent, best candidate, no adjustment for K
+    p_clustering_only: float            # patient-clustered standard error, best candidate, no adjustment for K
+    p_selection_only: float             # adjusted for K, but images treated as independent
+    best: int                           # column index of the candidate with the largest clustered z-statistic
+    auc: np.ndarray                     # AUC (or other metric) of every column
+    baseline: int
+    candidates: list                    # column indices of the candidates that entered the test
+    difference: np.ndarray              # D_k for the candidates
+    se_clustered: np.ndarray
+    se_image: np.ndarray
+    n_images: int
+    n_patients: int
+    n_positive_patients: int
+    n_negative_patients: int
+    n_mixed_patients: int
+    kish_cluster_size: float
+    effective_clusters: float
+    se_ratio: float
+    warnings: list = field(default_factory=list)
+
+    def summary(self) -> str:
+        lines = [f"best of {len(self.candidates)} candidates vs baseline (column {self.baseline}); best = column {self.best}",
+                 f"  AUC best {self.auc[self.best]:.3f} vs baseline {self.auc[self.baseline]:.3f} "
+                 f"(difference {self.auc[self.best] - self.auc[self.baseline]:+.3f})",
+                 f"  {self.n_images} images, {self.n_patients} patients ({self.n_positive_patients} with positives, "
+                 f"{self.n_negative_patients} with negatives, {self.n_mixed_patients} with both)",
+                 f"  p (patient-clustered, adjusted for K) = {self.p_value:.4f}",
+                 f"  p (pruned) = {self.p_value_pruned:.4f} | clustering only = {self.p_clustering_only:.4f} | "
+                 f"selection only = {self.p_selection_only:.4f} | naive = {self.p_naive:.4f}"]
+        lines += ["  WARNING: " + w for w in self.warnings]
+        return "\n".join(lines)
+
+
+def _check_inputs(scores, labels, patients, baseline):
+    S = np.asarray(scores, dtype=float)
+    if S.ndim != 2 or S.shape[1] < 2:
+        raise ValueError("scores must be a 2-D array (images x models) with the baseline and at least one candidate")
+    y = np.asarray(labels)
+    if y.shape != (S.shape[0],) or not np.isin(y, (0, 1)).all():
+        raise ValueError("labels must be a 1-D array of 0/1 values, one per image")
+    pid = np.asarray(patients)
+    if pid.shape != (S.shape[0],):
+        raise ValueError("patients must have one id per image")
+    if not np.isfinite(S).all():
+        raise ValueError("scores contain NaN or infinite values")
+    if not 0 <= baseline < S.shape[1]:
+        raise ValueError("baseline must be a column index of scores")
+    return S, y.astype(int), pid
+
+
+def camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=None, influence_fn=None, warn=True) -> CamtResult:
+    """Test whether the best of K candidate models has a higher AUC than the baseline.
+
+    Parameters
+    ----------
+    scores : array (n_images, 1 + K). One column per model; ``baseline`` is the index of the baseline column.
+    labels : array (n_images,) of 0/1 image-level labels. Labels may vary within a patient.
+    patients : array (n_images,) of patient identifiers (the clusters).
+    n_boot : number of multiplier-bootstrap draws (at least 4,000 is recommended).
+    seed : seed or ``numpy.random.Generator``.
+    influence_fn : optional replacement for the AUC influence function, for example from :func:`threshold_influence`.
+    """
+    S, y, pid = _check_inputs(scores, labels, patients, baseline)
+    rng = np.random.default_rng(seed)
+    _, cid = np.unique(pid, return_inverse=True)
+    P, n = cid.max() + 1, len(y)
+    pos = y == 1
+    has_pos = np.zeros(P, bool)
+    has_pos[cid[pos]] = True
+    has_neg = np.zeros(P, bool)
+    has_neg[cid[~pos]] = True
+    n1c, n0c = int(has_pos.sum()), int(has_neg.sum())
+    mixed = int((has_pos & has_neg).sum())
+    if min(n1c, n0c) < 2:
+        raise ValueError("need at least 2 patients with images of each class")
+
+    theta, infl = (influence_fn or auc_influence)(S, y)
+    cand = [j for j in range(S.shape[1]) if j != baseline]
+    D = theta[cand] - theta[baseline]
+    ID = infl[:, cand] - infl[:, [baseline]]
+    g_pos = np.zeros((P, len(cand)))
+    g_neg = np.zeros((P, len(cand)))
+    np.add.at(g_pos, cid[pos], ID[pos])
+    np.add.at(g_neg, cid[~pos], ID[~pos])
+    gs = np.sqrt(n1c / (n1c - 1)) * g_pos + np.sqrt(n0c / (n0c - 1)) * g_neg
+    se_c = np.sqrt((gs ** 2).sum(0))
+    se_i = np.sqrt(n / (n - 1) * (ID ** 2).sum(0))
+    ok = (se_c > 1e-12) & (se_i > 1e-12)
+    if not ok.any():
+        raise ValueError("no candidate differs from the baseline (zero-variance differences)")
+    if not ok.all():
+        cand = [c for c, keep_ in zip(cand, ok) if keep_]
+        D, ID, gs, se_c, se_i = D[ok], ID[:, ok], gs[:, ok], se_c[ok], se_i[ok]
+    z_c, z_i = D / se_c, D / se_i
+    ratio = tdist.ppf(0.95, max(min(n1c, n0c) - 1, 1)) / norm.ppf(0.95)       # small-class calibration
+
+    p_sel = _max_pvalue(ID, se_i, z_i.max(), rng, n_boot)
+    p_main = _max_pvalue(gs, se_c, z_c.max() / ratio, rng, n_boot)
+    keep = z_c >= -np.sqrt(2 * np.log(np.log(max(P, 3))))
+    p_pruned = _max_pvalue(gs[:, keep], se_c[keep], z_c.max() / ratio, rng, n_boot) if keep.any() else 1.0
+
+    sizes = np.bincount(cid)
+    kish = float((sizes ** 2).sum() / sizes.sum())
+    eff = float(n / kish)
+    notes = []
+    if len(cand) > 1 and min(n1c, n0c) < MIN_MINORITY_PATIENTS:
+        notes.append(f"only {min(n1c, n0c)} patients in the smaller class with {len(cand)} candidates: the test is liberal in this regime; "
+                     "treat the result as exploratory")
+    if eff < MIN_EFFECTIVE_CLUSTERS:
+        notes.append(f"effective number of clusters is about {eff:.0f} (Kish cluster size {kish:.0f}); with very unequal cluster sizes the "
+                     "test was liberal in the OCT example of the paper; treat the result with caution")
+    if warn:
+        for w in notes:
+            warnings.warn(w, stacklevel=2)
+    best = int(np.argmax(z_c))
+    return CamtResult(
+        p_value=float(p_main), p_value_pruned=float(p_pruned), p_naive=float(norm.sf(z_i.max())), p_clustering_only=float(norm.sf(z_c.max())),
+        p_selection_only=float(p_sel), best=cand[best], auc=theta, baseline=baseline, candidates=cand, difference=D, se_clustered=se_c,
+        se_image=se_i, n_images=n, n_patients=int(P), n_positive_patients=n1c, n_negative_patients=n0c, n_mixed_patients=mixed,
+        kish_cluster_size=kish, effective_clusters=eff, se_ratio=float(np.mean(se_c / se_i)), warnings=notes)
+
+
+def design_effect(scores, labels, patients, baseline=0):
+    """Design effect and within-patient correlation of the AUC influence function.
+
+    Returns a dict with the Kish cluster size ``m`` and, for the AUC of each model and for the difference of each model from the
+    baseline, the design effect (clustered variance divided by image-level variance) and rho = (DEFF - 1) / (m - 1).
+    """
+    S, y, pid = _check_inputs(scores, labels, patients, baseline)
+    _, cid = np.unique(pid, return_inverse=True)
+    P, n = cid.max() + 1, len(y)
+    sizes = np.bincount(cid)
+    m = float((sizes ** 2).sum() / sizes.sum())
+    pos = y == 1
+    h1, h0 = np.zeros(P, bool), np.zeros(P, bool)
+    h1[cid[pos]] = True
+    h0[cid[~pos]] = True
+    a1, a0 = h1.sum() / (h1.sum() - 1), h0.sum() / (h0.sum() - 1)
+    _, infl = auc_influence(S, y)
+    out = []
+    others = [j for j in range(S.shape[1]) if j != baseline]
+    for I in (infl, infl[:, others] - infl[:, [baseline]]):
+        gp, gn = np.zeros((P, I.shape[1])), np.zeros((P, I.shape[1]))
+        np.add.at(gp, cid[pos], I[pos])
+        np.add.at(gn, cid[~pos], I[~pos])
+        se_c2 = ((np.sqrt(a1) * gp + np.sqrt(a0) * gn) ** 2).sum(0)
+        se_i2 = n / (n - 1) * (I ** 2).sum(0)
+        out.append(se_c2 / se_i2)
+    rho = lambda d: (d - 1) / (m - 1) if m > 1 else np.full_like(d, np.nan)
+    return dict(kish_cluster_size=m, deff_auc=out[0], rho_auc=rho(out[0]), deff_difference=out[1], rho_difference=rho(out[1]),
+                difference_columns=others)

camt_test-0.1.0/src/camt_test.egg-info/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: camt-test
+Version: 0.1.0
+Summary: CAMT, the clustered AUC maximum test: patient-clustered, selection-adjusted test that the best of K models beats a baseline, with an audit of published claims
+Author: Aqib Gul
+License: MIT
+Keywords: AUC,clustered data,winner's curse,multiple comparisons,model comparison,medical AI
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: scipy>=1.9
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# camt: the clustered AUC maximum test (CAMT)
+
+Tests of the claim that **the best of K models has a higher AUC than a baseline** on a test set that contains several images or
+records per patient, and a **summary-statistics audit** of published claims of this kind.
+
+Two errors act together when the winner among K candidate models is compared with a baseline on one shared test set:
+
+1. observations from one patient are correlated, so a test that treats every image as independent understates the uncertainty;
+2. the winner is chosen among K candidates, so its apparent advantage is inflated even if all models are equally good.
+
+`camt` implements CAMT, a test that accounts for both. Labels may vary within a patient. The method is described in the accompanying
+manuscript (in preparation).
+
+Status: research software, version 0.1.0.
+
+## Installation
+
+```bash
+pip install camt-test      # from PyPI; the import name is `camt`
+pip install .            # or from the repository root; requires Python >= 3.9, numpy and scipy
+pip install ".[test]"    # also installs pytest
+```
+
+## Quick start
+
+```python
+from camt import camt_test
+
+# scores:   array (n_images, 1 + K), one column per model; column 0 is the baseline
+# labels:   array (n_images,) of 0/1 image-level labels
+# patients: array (n_images,) of patient identifiers (the clusters)
+result = camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=1)
+print(result.summary())
+```
+
+```
+best of 5 candidates vs baseline (column 0); best = column 1
+  AUC best 0.841 vs baseline 0.782 (difference +0.058)
+  500 images, 100 patients (47 with positives, 53 with negatives, 0 with both)
+  p (patient-clustered, adjusted for K) = 0.1685
+  p (pruned) = 0.1585 | clustering only = 0.0490 | selection only = 0.0310 | naive = 0.0068
+```
+
+In this simulated example no candidate is truly better, and the naive analysis reports p = 0.007. A complete script is in
+`examples/quickstart.py`.
+
+### Which p-value to report
+
+| attribute | meaning |
+|---|---|
+| `p_value` | **the proposed test**: patient-clustered, adjusted for K, with the small-class calibration. Use this one. |
+| `p_value_pruned` | as above, but candidates that are clearly worse than the baseline are dropped from the null distribution. More power when many candidates are poor; valid pointwise only. |
+| `p_naive` | what an image-level test of the best candidate would give (for comparison). |
+| `p_clustering_only` | patient-clustered, but no adjustment for K. |
+| `p_selection_only` | adjusted for K, but images treated as independent. |
+
+All p-values are one-sided; the null hypothesis is that no candidate is better than the baseline.
+
+## Functions
+
+| function | purpose |
+|---|---|
+| `camt_test(scores, labels, patients, baseline=0, n_boot=4000, seed=None, influence_fn=None)` | the test; returns a `CamtResult` |
+| `design_effect(scores, labels, patients, baseline=0)` | Kish cluster size, design effect and within-patient correlation of the AUC influence function, for each model and for each difference from the baseline |
+| `threshold_influence(thresholds, kind="balanced")` | influence function for sensitivity, specificity or balanced accuracy at operating points fixed in advance; pass it as `influence_fn` |
+| `audit_claim(p_naive, patients, images, K, rho=...)` | adjust a published p-value for clustering and for K, using only summary numbers |
+| `sensitivity_table(...)` | `audit_claim` over a range of within-patient correlations |
+| `hanley_mcneil_se`, `rho_from_patient_level_ci` | helpers for the audit |
+
+### Metrics at a fixed operating point
+
+For balanced accuracy at thresholds chosen in advance (for example on the training patients):
+
+```python
+from camt import camt_test, threshold_influence
+result = camt_test(scores, labels, patients, influence_fn=threshold_influence(thresholds, kind="balanced"))
+```
+
+Thresholds estimated on the test sample itself are not covered.
+
+### Auditing a published claim
+
+```python
+from camt import audit_claim
+a = audit_claim(p_naive=0.0004, patients=40, images=3900, K=10, rho=0.1)
+a["p_bonferroni"]   # adjusted p-value if the within-patient correlation is 0.1 and 10 models were tried
+a["rho_star"]       # correlation above which the clustered p-value alone exceeds 0.05
+```
+
+The within-patient correlation cannot be recovered from summary data, so report the result over a range of values
+(`sensitivity_table`) and treat `rho_star` as the quantity of interest. The audit is valid if the correlation is not understated and
+K is the true number of models tried.
+
+## Assumptions and limits
+
+- **K must be the true number of candidates examined**, and the baseline must be fixed in advance. Pre-specifying one primary model
+  (K = 1) removes the selection cost.
+- The target is the **fitted models** on patients like the test patients, not the training algorithm. Test patients must be disjoint
+  from the training data.
+- The test assumes independent patients, a bounded number of images per patient and a moderate number of candidates.
+- The test can be **liberal** when fewer than about 20 patients belong to the smaller class and many candidates are screened, and
+  when a few patients contribute most of the images. The function emits a warning in both situations. The cut-off for the second
+  warning (fewer than 100 effective clusters, that is, images divided by the Kish cluster size) is a heuristic and has not been
+  validated.
+- Candidates whose scores give a zero-variance difference from the baseline are dropped.
+- Metrics whose threshold is estimated on the test sample are not covered.
+
+## Reproducing the behaviour of the test
+
+```bash
+pytest                                          # unit tests, about 25 seconds
+python examples/reproduce_size.py               # false-positive rates in simulation (a few minutes)
+python examples/reproduce_size.py --reps 200    # quick version
+```
+
+`examples/reproduce_size.py` simulates test sets in which all models are equal and prints the false-positive rate of the naive,
+clustering-only, selection-only and proposed procedures for K = 1, 5 and 20, at a nominal level of 5%.
+
+## Citation
+
+If you use this software, please cite it (see `CITATION.cff`) together with the accompanying manuscript:
+
+> Gul A. Valid inference for "best of K" model comparisons on patient-clustered diagnostic data. Manuscript in preparation.
+
+The archived release and its DOI are given in `CITATION.cff` once available.
+
+## Licence
+
+MIT. See `LICENSE`.

camt_test-0.1.0/src/camt_test.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+pyproject.toml
+src/camt/__init__.py
+src/camt/audit.py
+src/camt/core.py
+src/camt_test.egg-info/PKG-INFO
+src/camt_test.egg-info/SOURCES.txt
+src/camt_test.egg-info/dependency_links.txt
+src/camt_test.egg-info/requires.txt
+src/camt_test.egg-info/top_level.txt
+tests/test_camt.py

camt_test-0.1.0/src/camt_test.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

camt_test-0.1.0/src/camt_test.egg-info/requires.txt

@@ -0,0 +1,5 @@
+numpy>=1.23
+scipy>=1.9
+
+[test]
+pytest

camt_test-0.1.0/src/camt_test.egg-info/top_level.txt

@@ -0,0 +1 @@
+camt

camt_test-0.1.0/tests/test_camt.py

@@ -0,0 +1,94 @@
+import warnings
+
+import numpy as np
+import pytest
+from helpers import simulate
+
+from camt import audit_claim, camt_test, design_effect, sensitivity_table, threshold_influence
+
+
+def test_reference_values():
+    """Deterministic quantities are fixed; bootstrap p-values are Monte Carlo estimates and are checked with a tolerance."""
+    S, y, pid = simulate(seed=3)
+    r = camt_test(S, y, pid, n_boot=4000, seed=11, warn=False)
+    assert r.best == 5
+    assert (r.n_images, r.n_patients) == (320, 80)
+    np.testing.assert_allclose(r.auc, [0.8193139098, 0.8127741228, 0.8387374687, 0.8297305764, 0.7961701128, 0.8412437343], atol=1e-9)
+    assert r.se_clustered[0] == pytest.approx(0.0343591316, abs=1e-9)
+    assert r.se_image[0] == pytest.approx(0.0265136743, abs=1e-9)
+    assert r.p_naive == pytest.approx(0.20525702527235268, abs=1e-12)
+    assert r.p_clustering_only == pytest.approx(0.28082648001716126, abs=1e-12)
+    assert r.p_value == pytest.approx(0.613, abs=0.04)
+    assert r.p_selection_only == pytest.approx(0.497, abs=0.04)
+    assert r.kish_cluster_size == pytest.approx(4.0)
+    d = design_effect(S, y, pid)
+    np.testing.assert_allclose(d["deff_difference"], [1.679363, 1.87628, 1.987149, 2.466094, 2.01072], atol=1e-5)
+
+
+def test_detects_a_large_edge_and_not_a_null():
+    S, y, pid = simulate(P=200, edge=0.8, seed=1)
+    assert camt_test(S, y, pid, n_boot=1000, seed=0, warn=False).p_value < 0.05
+    S0, y0, pid0 = simulate(P=200, edge=0.0, seed=1)
+    r0 = camt_test(S0, y0, pid0, n_boot=1000, seed=0, warn=False)
+    assert r0.p_value >= r0.p_clustering_only            # adjusting for K can only raise the p-value
+
+
+def test_size_is_near_nominal_under_the_null():
+    reps = 120
+    rej = sum(camt_test(*simulate(P=100, K=5, seed=100 + s), n_boot=500, seed=s, warn=False).p_value < 0.05 for s in range(reps))
+    assert rej / reps < 0.13                              # nominal 5%, Monte Carlo SE about 2 points
+
+
+def test_balanced_accuracy_influence_runs():
+    S, y, pid = simulate(P=120, edge=0.8, seed=5)
+    r = camt_test(S, y, pid, n_boot=500, seed=0, influence_fn=threshold_influence(np.median(S, axis=0)), warn=False)
+    assert 0 <= r.p_value <= 1
+
+
+def test_input_validation():
+    S, y, pid = simulate()
+    with pytest.raises(ValueError):
+        camt_test(S[:, :1], y, pid)
+    with pytest.raises(ValueError):
+        camt_test(S, y + 2, pid)
+    S_bad = S.copy()
+    S_bad[0, 0] = np.nan
+    with pytest.raises(ValueError):
+        camt_test(S_bad, y, pid)
+    with pytest.raises(ValueError):
+        camt_test(S, np.zeros_like(y), pid)          # no positives
+
+
+def test_warns_for_a_small_minority_class():
+    S, y, pid = simulate(P=60, K=5)
+    y = np.where(pid < 5, 1, 0)                           # five positive patients only
+    with pytest.warns(UserWarning):
+        camt_test(S, y, pid, n_boot=200, seed=0)
+
+
+def test_no_warning_for_a_well_posed_problem():
+    S, y, pid = simulate(P=200, K=3, seed=2)
+    with warnings.catch_warnings():
+        warnings.simplefilter("error")
+        camt_test(S, y, pid, n_boot=200, seed=0)
+
+
+def test_design_effect_is_one_for_one_image_per_patient():
+    S, y, pid = simulate(P=300, m=1)
+    d = design_effect(S, y, pid)
+    assert d["kish_cluster_size"] == pytest.approx(1.0)
+    assert np.allclose(d["deff_auc"], 1.0, atol=0.25)
+
+
+def test_audit_worked_example():
+    a = audit_claim(0.0004, 40, 3900, 10, rho=0.1, pos_patients=28, neg_patients=12)
+    assert a["m"] == pytest.approx(97.5)
+    assert a["p_cluster"] > 0.0004 and a["p_bonferroni"] >= a["p_cluster"]
+    tab = sensitivity_table(0.0004, 40, 3900, 10)
+    assert all(tab[i]["p_cluster"] <= tab[i + 1]["p_cluster"] for i in range(len(tab) - 1))   # p rises with rho
+
+
+def test_audit_two_sided_halves_p():
+    one = audit_claim(0.01, 100, 100, 1)
+    two = audit_claim(0.02, 100, 100, 1, sided="two")
+    assert one["z_naive"] == pytest.approx(two["z_naive"])