evalsuite-python 0.1.0a1__tar.gz → 0.1.0a2__tar.gz

evalsuite_python-0.1.0a2/CHANGELOG.md

@@ -0,0 +1,43 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0a2]
+
+### Added
+
+- Model comparison (`es.compare`): per-model confidence intervals from paired bootstrap resamples, pairwise
+  tests (McNemar for accuracy, DeLong for binary ROC AUC, paired bootstrap otherwise), multiple-comparison
+  correction, best model per metric, and summary/pandas/Markdown/LaTeX/JSON export.
+- Confidence intervals: `bootstrap_ci` (percentile, basic, BCa; stratified and reproducible),
+  `proportion_ci` and `accuracy_ci` (Wilson, Clopper-Pearson, normal), `roc_auc_ci` (DeLong).
+- Paired tests: `mcnemar_test`, `delong_test`, `paired_bootstrap_test`.
+- Effect sizes: `cohens_d` (independent and paired), `hedges_g`, `cliffs_delta`; `adjust_pvalues`
+  (Holm, Bonferroni, Benjamini-Hochberg, Benjamini-Yekutieli).
+- Reference tests against statsmodels and SciPy, brute-force DeLong checks and coverage simulations.
+- Python 3.14 support and CI.
+
+### Fixed
+
+- 0.1.0a1 installed an `evalsuite` console command although the CLI is not implemented yet, so the command
+  failed with `ModuleNotFoundError`. The entry point is removed until the CLI ships.
+
+## [0.1.0a1]
+
+First alpha, published to reserve the name and test the release pipeline. Distribution name
+`evalsuite-python` (`pip install --pre evalsuite-python`), imported as `evalsuite`.
+
+### Added
+
+- Core: input validation with actionable errors, exception hierarchy, immutable result objects with
+  JSON/pandas/Markdown/LaTeX export, metric registry (`list_metrics`, `metric_info`), and an evaluation
+  context that computes the confusion matrix once per evaluation.
+- Classification metrics for binary, multiclass and multilabel targets with all averaging modes and
+  sample weights.
+- Regression metrics for single- and multi-output targets with sample weights.
+- `evaluate()` high-level API with task inference and default metric sets.
+- Reference tests against scikit-learn and property-based tests; Python 3.9 to 3.13.

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0a2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.1.0a1
+Version: 0.1.0a2
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -23,6 +23,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Typing :: Typed
@@ -42,6 +43,7 @@ Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
 Requires-Dist: scikit-learn>=1.2; extra == 'dev'
+Requires-Dist: statsmodels>=0.13; extra == 'dev'
 Requires-Dist: twine; extra == 'dev'
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.5; extra == 'plot'
@@ -109,6 +111,28 @@ es.metric_info("classification.mcc").formula  # documentation
 es.list_metrics("regression")
 ```
 
+## Comparing models
+
+```python
+result = es.compare(
+    y_true,
+    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
+    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
+    random_state=0,
+)
+print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
+result.to_latex(label="tab:models")
+
+es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
+es.accuracy_ci(y_true, y_pred)  # Wilson interval
+es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
+es.mcnemar_test(y_true, pred_a, pred_b)
+```
+
+Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared
+with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
+p-values are adjusted for multiple comparisons (Holm by default).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0a2}/README.md

@@ -60,6 +60,28 @@ es.metric_info("classification.mcc").formula  # documentation
 es.list_metrics("regression")
 ```
 
+## Comparing models
+
+```python
+result = es.compare(
+    y_true,
+    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
+    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
+    random_state=0,
+)
+print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
+result.to_latex(label="tab:models")
+
+es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
+es.accuracy_ci(y_true, y_pred)  # Wilson interval
+es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
+es.mcnemar_test(y_true, pred_a, pred_b)
+```
+
+Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared
+with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
+p-values are adjusted for multiple comparisons (Holm by default).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0a2}/pyproject.toml

@@ -25,6 +25,7 @@ classifiers = [
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
   "Topic :: Scientific/Engineering",
   "Topic :: Scientific/Engineering :: Artificial Intelligence",
   "Typing :: Typed",
@@ -39,6 +40,7 @@ dev = [
   "pytest-cov>=4",
   "hypothesis>=6.80",
   "scikit-learn>=1.2",
+  "statsmodels>=0.13",
   "ruff>=0.6",
   "mypy>=1.10",
   "pandas-stubs",
@@ -54,8 +56,7 @@ Source = "https://github.com/mkcs28/evalsuite-python"
 Issues = "https://github.com/mkcs28/evalsuite-python/issues"
 Changelog = "https://github.com/mkcs28/evalsuite-python/blob/main/CHANGELOG.md"
 
-[project.scripts]
-evalsuite = "evalsuite.cli.main:main"
+# [project.scripts] evalsuite = "evalsuite.cli.main:main" is added together with the CLI itself.
 
 [tool.hatch.version]
 path = "src/evalsuite/version.py"

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0a2}/src/evalsuite/__init__.py

@@ -5,7 +5,7 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import classification, regression
+from . import classification, regression, stats
 from .api import evaluate
 from .classification import (
     accuracy,
@@ -59,9 +59,42 @@ from .regression import (
     rse,
     smape,
 )
+from .stats import (
+    ComparisonResult,
+    ConfidenceInterval,
+    TestResult,
+    accuracy_ci,
+    adjust_pvalues,
+    bootstrap_ci,
+    cliffs_delta,
+    cohens_d,
+    compare,
+    delong_test,
+    hedges_g,
+    mcnemar_test,
+    paired_bootstrap_test,
+    proportion_ci,
+    roc_auc_ci,
+)
 from .version import __version__
 
 __all__ = [
+    "stats",
+    "roc_auc_ci",
+    "proportion_ci",
+    "paired_bootstrap_test",
+    "mcnemar_test",
+    "hedges_g",
+    "delong_test",
+    "compare",
+    "cohens_d",
+    "cliffs_delta",
+    "bootstrap_ci",
+    "adjust_pvalues",
+    "accuracy_ci",
+    "TestResult",
+    "ConfidenceInterval",
+    "ComparisonResult",
     "EvalSuiteError",
     "EvaluationResult",
     "InputValidationError",

evalsuite_python-0.1.0a2/src/evalsuite/stats/__init__.py

@@ -0,0 +1,25 @@
+"""Statistical evaluation: confidence intervals, paired tests, effect sizes, multiple comparisons, compare()."""
+
+from .compare import ComparisonResult, compare
+from .effect import adjust_pvalues, cliffs_delta, cohens_d, hedges_g
+from .intervals import accuracy_ci, bootstrap_ci, proportion_ci, roc_auc_ci
+from .paired import delong_test, mcnemar_test, paired_bootstrap_test
+from .results import ConfidenceInterval, TestResult
+
+__all__ = [
+    "ComparisonResult",
+    "ConfidenceInterval",
+    "TestResult",
+    "accuracy_ci",
+    "adjust_pvalues",
+    "bootstrap_ci",
+    "cliffs_delta",
+    "cohens_d",
+    "compare",
+    "delong_test",
+    "hedges_g",
+    "mcnemar_test",
+    "paired_bootstrap_test",
+    "proportion_ci",
+    "roc_auc_ci",
+]

evalsuite_python-0.1.0a2/src/evalsuite/stats/_resolve.py

@@ -0,0 +1,90 @@
+"""Resolve metric names to functions and call them on index subsets."""
+
+from __future__ import annotations
+
+import contextlib
+import inspect
+from collections.abc import Callable
+from typing import Any, Optional
+
+import numpy as np
+from numpy.typing import NDArray
+
+from ..core.exceptions import InputValidationError
+from ..core.validation import target_type, to_numpy
+
+MetricFn = Callable[..., Any]
+
+
+def resolve_metric(metric: Any) -> tuple[MetricFn, str]:
+    """A metric function (``es.f1``) or its name (``"f1"``, ``"classification.f1"``)."""
+    if callable(metric):
+        return metric, getattr(metric, "__name__", "metric")
+    if isinstance(metric, str):
+        import evalsuite
+
+        name = metric.split(".")[-1]
+        fn = getattr(evalsuite, name, None)
+        if callable(fn) and name in {m.split(".")[-1] for m in evalsuite.list_metrics()}:
+            return fn, name
+        raise InputValidationError(
+            f"Unknown metric '{metric}'. Use a metric function or a name from list_metrics()."
+        )
+    raise InputValidationError("metric must be a metric function (e.g. es.f1) or its name (e.g. 'f1').")
+
+
+def is_categorical(y: NDArray[Any]) -> bool:
+    try:
+        return target_type(y) in ("binary", "multiclass", "multilabel")
+    except Exception:
+        return False
+
+
+class MetricCall:
+    """Evaluate ``fn(y_true[idx], second[idx], **kwargs)`` repeatedly on index subsets."""
+
+    def __init__(
+        self,
+        fn: MetricFn,
+        y_true: Any,
+        y_pred: Any = None,
+        y_prob: Any = None,
+        sample_weight: Any = None,
+        kwargs: Optional[dict[str, Any]] = None,
+    ) -> None:
+        if (y_pred is None) == (y_prob is None):
+            raise InputValidationError(
+                "Pass exactly one of y_pred (label/value metrics) or y_prob (probability metrics)."
+            )
+        self.fn = fn
+        self.y_true = to_numpy(y_true, "y_true", allow_2d=True)
+        second = y_pred if y_pred is not None else y_prob
+        self.second = to_numpy(second, "y_pred" if y_pred is not None else "y_prob", allow_2d=True)
+        if self.second.shape[0] != self.y_true.shape[0]:
+            raise InputValidationError(
+                f"y_true and {'y_pred' if y_pred is not None else 'y_prob'} must contain the same number of "
+                f"observations. Received {self.y_true.shape[0]} and {self.second.shape[0]}."
+            )
+        self.n = self.y_true.shape[0]
+        self.weight = None if sample_weight is None else to_numpy(sample_weight, "sample_weight")
+        self.kwargs = dict(kwargs or {})
+        params = inspect.signature(fn).parameters
+        self.categorical = is_categorical(self.y_true)
+        # Keep every class in every resample so per-class and averaged metrics stay comparable.
+        if self.categorical and "labels" in params and "labels" not in self.kwargs and self.y_true.ndim == 1:
+            present = self.y_true if y_pred is None else np.concatenate([self.y_true, self.second])
+            with contextlib.suppress(TypeError):  # incomparable labels: the metric reports it
+                self.kwargs["labels"] = np.unique(present)
+        if self.weight is not None and "sample_weight" not in params:
+            raise InputValidationError(f"{getattr(fn, '__name__', 'This metric')} does not accept sample_weight.")
+
+    def __call__(self, idx: Optional[NDArray[np.int64]] = None) -> float:
+        if idx is None:
+            yt, s, w = self.y_true, self.second, self.weight
+        else:
+            yt, s = self.y_true[idx], self.second[idx]
+            w = None if self.weight is None else self.weight[idx]
+        kw = dict(self.kwargs)
+        if w is not None:
+            kw["sample_weight"] = w
+        return float(self.fn(yt, s, **kw))

evalsuite_python-0.1.0a2/src/evalsuite/stats/compare.py

@@ -0,0 +1,323 @@
+"""compare(): evaluate several models on the same test set with intervals and paired tests."""
+
+from __future__ import annotations
+
+import itertools
+import json
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass, field
+from types import MappingProxyType
+from typing import TYPE_CHECKING, Any, Optional, cast
+
+import numpy as np
+
+from ..core.exceptions import InputValidationError
+from ..core.registry import metric_info
+from ..core.result import _json_safe, _latex_escape, _latex_table
+from ..core.types import ArrayLike
+from ..core.validation import to_numpy
+from ._resolve import MetricCall, is_categorical, resolve_metric
+from .effect import adjust_pvalues
+from .intervals import _check_level, _check_resamples, bootstrap_distribution, resample_indices
+from .paired import delong_test, mcnemar_test
+
+if TYPE_CHECKING:
+    import pandas as pd
+
+__all__ = ["ComparisonResult", "compare"]
+
+_DEFAULT_CLF = ["accuracy", "f1", "mcc"]
+_DEFAULT_CLF_PROB = ["roc_auc", "brier_score"]
+_DEFAULT_REG = ["mae", "rmse", "r2"]
+
+
+def _p(value: float) -> str:
+    return "<0.0001" if value < 1e-4 else f"={value:.4f}"
+
+
+def _info(name: str) -> Any:
+    for cat in ("classification", "regression"):
+        try:
+            return metric_info(f"{cat}.{name}")
+        except KeyError:
+            continue
+    return None
+
+
+def _needs_prob(name: str) -> bool:
+    info = _info(name)
+    return info is not None and "y_prob" in info.input_requirements
+
+
+@dataclass(frozen=True, eq=False)
+class ComparisonResult:
+    """Per-model estimates with confidence intervals, and pairwise tests with adjusted p-values."""
+
+    models: tuple[str, ...]
+    metrics: tuple[str, ...]
+    estimates: tuple[Mapping[str, Any], ...]
+    tests: tuple[Mapping[str, Any], ...]
+    settings: Mapping[str, Any] = field(default_factory=dict)
+
+    def __post_init__(self) -> None:
+        object.__setattr__(self, "estimates", tuple(MappingProxyType(dict(r)) for r in self.estimates))
+        object.__setattr__(self, "tests", tuple(MappingProxyType(dict(r)) for r in self.tests))
+        object.__setattr__(self, "settings", MappingProxyType(dict(self.settings)))
+
+    def estimate(self, model: str, metric: str) -> Mapping[str, Any]:
+        for row in self.estimates:
+            if row["model"] == model and row["metric"] == metric:
+                return row
+        raise KeyError(f"No estimate for model={model!r}, metric={metric!r}.")
+
+    def best(self, metric: str) -> str:
+        """Model with the best point estimate (direction from the metric registry)."""
+        rows = [r for r in self.estimates if r["metric"] == metric]
+        if not rows:
+            raise KeyError(f"Metric {metric!r} was not compared. Compared: {', '.join(self.metrics)}.")
+        info = _info(metric)
+        higher = True if info is None or info.higher_is_better is None else info.higher_is_better
+        pick = max if higher else min
+        return str(pick(rows, key=lambda r: r["estimate"])["model"])
+
+    def summary(self, *, digits: int = 3) -> str:
+        level = round(self.settings["level"] * 100, 6)
+        lines = [
+            f"EvalSuite model comparison (n={self.settings['n_samples']}, {level:g}% CIs, "
+            f"{self.settings['n_resamples']} paired bootstrap resamples, {self.settings['correction']} correction)"
+        ]
+        width = max(len(m) for m in self.models)
+        for metric in self.metrics:
+            lines.append(f"\n{metric}  (best: {self.best(metric)})")
+            for row in (r for r in self.estimates if r["metric"] == metric):
+                lines.append(
+                    f"  {row['model']:<{width}}  {row['estimate']:.{digits}f} "
+                    f"({row['low']:.{digits}f}–{row['high']:.{digits}f})"
+                )
+            for t in (t for t in self.tests if t["metric"] == metric):
+                mark = " *" if t["significant"] else ""
+                lines.append(
+                    f"  {t['model_a']} − {t['model_b']}: {t['difference']:+.{digits}f} "
+                    f"[{t['test']}, p={t['p_value']:.4f}, adjusted p={t['p_adjusted']:.4f}]{mark}"
+                )
+        lines.append(f"\n* significant at α = {self.settings['alpha']} after correction")
+        return "\n".join(lines)
+
+    def __repr__(self) -> str:
+        return self.summary()
+
+    def to_dict(self) -> dict[str, Any]:
+        return cast(
+            "dict[str, Any]",
+            _json_safe(
+                {
+                    "models": list(self.models),
+                    "metrics": list(self.metrics),
+                    "estimates": [dict(r) for r in self.estimates],
+                    "tests": [dict(r) for r in self.tests],
+                    "settings": dict(self.settings),
+                }
+            ),
+        )
+
+    def to_json(self, *, indent: Optional[int] = 2) -> str:
+        return json.dumps(self.to_dict(), indent=indent, allow_nan=False)
+
+    def to_dataframe(self, which: str = "estimates") -> pd.DataFrame:
+        """``"estimates"`` (one row per model and metric) or ``"tests"`` (one row per comparison)."""
+        import pandas as pd
+
+        if which not in ("estimates", "tests"):
+            raise InputValidationError("which must be 'estimates' or 'tests'.")
+        frame: pd.DataFrame = pd.DataFrame(
+            [dict(r) for r in (self.estimates if which == "estimates" else self.tests)]
+        )
+        return frame
+
+    def _cell(self, model: str, metric: str, digits: int) -> str:
+        r = self.estimate(model, metric)
+        return f"{r['estimate']:.{digits}f} ({r['low']:.{digits}f}–{r['high']:.{digits}f})"
+
+    def to_markdown(self, *, digits: int = 3) -> str:
+        """Models as rows, metrics as columns: ``estimate (low–high)``; best value per metric in bold."""
+        header = "| Model | " + " | ".join(self.metrics) + " |"
+        sep = "| --- |" + " ---: |" * len(self.metrics)
+        rows = []
+        for model in self.models:
+            cells = []
+            for metric in self.metrics:
+                c = self._cell(model, metric, digits)
+                cells.append(f"**{c}**" if self.best(metric) == model else c)
+            rows.append(f"| {model} | " + " | ".join(cells) + " |")
+        return "\n".join([header, sep, *rows])
+
+    def to_latex(self, *, digits: int = 3, caption: Optional[str] = None, label: Optional[str] = None) -> str:
+        """Publication table: models × metrics with confidence intervals; best value per metric in bold."""
+        header = ["Model", *(_latex_escape(m) for m in self.metrics)]
+        rows = []
+        for model in self.models:
+            cells = [_latex_escape(model)]
+            for metric in self.metrics:
+                c = self._cell(model, metric, digits).replace("–", "--")
+                cells.append(rf"\textbf{{{c}}}" if self.best(metric) == model else c)
+            rows.append(cells)
+        level = round(self.settings["level"] * 100, 6)
+        cap = caption or f"Model comparison: estimate ({level:g}% CI)."  # escaped once by _latex_table
+        return _latex_table(header, rows, caption=cap, label=label)
+
+
+def compare(
+    y_true: ArrayLike,
+    predictions: Optional[Mapping[str, ArrayLike]] = None,
+    *,
+    probabilities: Optional[Mapping[str, ArrayLike]] = None,
+    metrics: Optional[Sequence[str]] = None,
+    baseline: Optional[str] = None,
+    level: float = 0.95,
+    alpha: float = 0.05,
+    n_resamples: int = 1000,
+    correction: str = "holm",
+    random_state: Optional[int] = None,
+    stratify: Optional[bool] = None,
+    sample_weight: Optional[ArrayLike] = None,
+    metric_kwargs: Optional[Mapping[str, Mapping[str, Any]]] = None,
+) -> ComparisonResult:
+    """Compare models evaluated on the same test set.
+
+    ``predictions`` and/or ``probabilities`` map model names to that model's predicted labels/values or
+    probabilities. Every model is resampled on the same bootstrap indices (paired), giving a percentile
+    confidence interval per model and metric and a paired test per pair of models:
+
+    * accuracy: McNemar's test (exact when there are fewer than 25 discordant pairs);
+    * binary ROC AUC: DeLong's test;
+    * everything else: paired bootstrap test.
+
+    Pairs are every model against ``baseline`` if given, otherwise all pairs. p-values are adjusted within
+    each metric (``correction``: "holm", "bonferroni", "bh", "by"); ``significant`` uses the adjusted
+    p-value and ``alpha``. ``metric_kwargs`` passes options per metric, e.g. ``{"f1": {"average": "macro"}}``.
+    """
+    level = _check_level(level)
+    n_resamples = _check_resamples(n_resamples)
+    predictions = dict(predictions or {})
+    probabilities = dict(probabilities or {})
+    names = list(dict.fromkeys([*predictions, *probabilities]))
+    if len(names) < 2:
+        raise InputValidationError("compare() needs at least two models (a dict of name -> predictions).")
+    if baseline is not None and baseline not in names:
+        raise InputValidationError(f"baseline={baseline!r} is not one of the models: {', '.join(names)}.")
+    yt = to_numpy(y_true, "y_true", allow_2d=True)
+    categorical = is_categorical(yt)
+    if metrics is None:
+        if categorical:
+            metric_names = list(_DEFAULT_CLF) if predictions else []
+            if probabilities:
+                metric_names += _DEFAULT_CLF_PROB if np.unique(yt).shape[0] <= 2 and yt.ndim == 1 else ["roc_auc"]
+        else:
+            metric_names = list(_DEFAULT_REG)
+    else:
+        metric_names = [resolve_metric(m)[1] for m in metrics]
+    kwargs_by_metric = {k: dict(v) for k, v in (metric_kwargs or {}).items()}
+
+    calls: dict[tuple[str, str], MetricCall] = {}
+    for metric in metric_names:
+        fn, _ = resolve_metric(metric)
+        prob = _needs_prob(metric)
+        source = probabilities if prob else predictions
+        for model in names:
+            if model not in source:
+                kind = "probabilities" if prob else "predictions"
+                raise InputValidationError(f"Metric '{metric}' needs {kind} for model '{model}'.")
+            calls[(model, metric)] = MetricCall(
+                fn,
+                yt,
+                None if prob else source[model],
+                source[model] if prob else None,
+                sample_weight,
+                kwargs_by_metric.get(metric),
+            )
+        # identical label set for every model of this metric
+        labs = [calls[(m, metric)].kwargs.get("labels") for m in names]
+        if all(lab is not None for lab in labs):
+            shared = labs[0]
+            for lab in labs[1:]:
+                shared = np.union1d(shared, lab)
+            for m in names:
+                calls[(m, metric)].kwargs["labels"] = shared
+
+    rng = np.random.default_rng(random_state)
+    use_strata = categorical if stratify is None else bool(stratify)
+    idx = resample_indices(yt.shape[0], n_resamples, rng, yt if use_strata else None)
+    a = (1 - level) / 2
+    boot: dict[tuple[str, str], np.ndarray] = {}
+    estimates: list[dict[str, Any]] = []
+    for metric in metric_names:
+        for model in names:
+            call = calls[(model, metric)]
+            dist, failed = bootstrap_distribution(call, idx)
+            boot[(model, metric)] = dist
+            low, high = np.nanquantile(dist, [a, 1 - a])
+            estimates.append(
+                {
+                    "model": model,
+                    "metric": metric,
+                    "estimate": call(),
+                    "low": float(low),
+                    "high": float(high),
+                    "failed_resamples": int(failed),
+                }
+            )
+
+    pairs = (
+        [(m, baseline) for m in names if m != baseline]
+        if baseline is not None
+        else list(itertools.combinations(names, 2))
+    )
+    tests: list[dict[str, Any]] = []
+    for metric in metric_names:
+        block: list[dict[str, Any]] = []
+        for ma, mb in pairs:
+            diff_dist = boot[(ma, metric)] - boot[(mb, metric)]
+            diff_dist = diff_dist[~np.isnan(diff_dist)]
+            observed = calls[(ma, metric)]() - calls[(mb, metric)]()
+            low, high = np.quantile(diff_dist, [a, 1 - a])
+            if metric == "accuracy" and sample_weight is None:
+                t = mcnemar_test(yt, predictions[ma], predictions[mb])
+                test_name, stat, p = t.test, t.statistic, t.p_value
+            elif metric == "roc_auc" and sample_weight is None and yt.ndim == 1 and np.unique(yt).shape[0] == 2:
+                t = delong_test(yt, probabilities[ma], probabilities[mb])
+                test_name, stat, p = t.test, t.statistic, t.p_value
+            else:
+                extreme = np.sum(np.abs(diff_dist - observed) >= abs(observed))
+                test_name, stat, p = "paired-bootstrap", observed, (1 + extreme) / (diff_dist.shape[0] + 1)
+            block.append(
+                {
+                    "metric": metric,
+                    "model_a": ma,
+                    "model_b": mb,
+                    "difference": float(observed),
+                    "ci_low": float(low),
+                    "ci_high": float(high),
+                    "test": test_name,
+                    "statistic": float(stat),
+                    "p_value": float(p),
+                }
+            )
+        if block:
+            adjusted = adjust_pvalues([r["p_value"] for r in block], method=correction)  # type: ignore[arg-type]
+            for r, pa in zip(block, adjusted):
+                r["p_adjusted"] = float(pa)
+                r["significant"] = bool(pa < alpha)
+        tests.extend(block)
+
+    settings = {
+        "n_samples": int(yt.shape[0]),
+        "level": level,
+        "alpha": alpha,
+        "n_resamples": n_resamples,
+        "correction": correction,
+        "random_state": random_state,
+        "stratified": use_strata,
+        "baseline": baseline,
+        "ci_method": "bootstrap-percentile (paired resamples)",
+    }
+    return ComparisonResult(tuple(names), tuple(metric_names), tuple(estimates), tuple(tests), settings)