evalsuite-python 0.3.0__tar.gz → 0.3.1__tar.gz

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/CHANGELOG.md

@@ -6,6 +6,35 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.3.1] - 2026-10-09
+
+Quality-assurance release: release pipeline, packaging checks, edge-case fixes and an overall benchmark.
+No API was removed or renamed; code written for 0.3.0 runs unchanged.
+
+### Fixed
+- Labels that mix numbers and strings (`[0, "a"]`, or string `y_true` with integer `y_pred`) now raise
+  `InputValidationError` instead of being silently compared as text.
+- `rmse` no longer overflows to `inf` when errors exceed ~1e154; the value is computed with rescaling.
+- `bootstrap_ci`, `paired_bootstrap_test` and `compare` raise `StatisticalTestError` for fewer than two
+  observations instead of returning a zero-width interval.
+- A 2-D mask given as a nested Python list (`[[1, 1], [0, 0]]`) is read as one image, as a NumPy array is.
+- Release workflow: the GitHub release step no longer fails when the release already exists (for example
+  when the tag was created from the GitHub UI); it attaches the files instead.
+
+### Added
+- Benchmarks: an overall summary first (per suite and in total: cases, agreement with the reference,
+  how many are faster, geometric-mean and range of speed-ups) and rows in alphabetical order;
+  `BenchmarkResult.overall()`, `overall_markdown()` and `sorted_rows()`; `overall` in the JSON export.
+- Release workflow: refuses a tag that does not match the package version or a version already on PyPI,
+  checks the built files, uses `--verify-tag`, and can be run by hand to attach a version's PyPI files
+  (hash-verified) to its GitHub release.
+- CI: wheel and sdist installed in clean environments with `pip check` and a smoke test that must give the
+  same results for both, a distribution check (metadata, forbidden files, credentials), a minimum-dependency
+  job (NumPy 1.22, SciPy 1.8, pandas 1.4), README examples run as tests, and repository hygiene tests
+  (no credentials, no unsafe calls, least-privilege workflows, single version source).
+- QA tests against scikit-learn and statsmodels for the edge cases in the release checklist.
+- `SECURITY.md` with the reporting process.
+
 ## [0.3.0] - 2026-10-09
 
 Computer vision (the v0.3.0 roadmap), with comparison, plots, reporting and benchmarks extended to it.

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.3.0
+Version: 0.3.1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -49,7 +49,7 @@ Requires-Dist: pycocotools>=2.0.7; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
-Requires-Dist: scikit-learn>=1.2; extra == 'dev'
+Requires-Dist: scikit-learn>=1.4; extra == 'dev'
 Requires-Dist: statsmodels>=0.13; extra == 'dev'
 Requires-Dist: twine; extra == 'dev'
 Provides-Extra: plot
@@ -70,7 +70,7 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.3.0).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
+> **Status: stable (0.3.1).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -283,24 +283,28 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 ## Performance
 
 Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). Every result agrees with the reference to floating-point rounding (largest difference
-1.4e-14).
-
-| Case | n | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
-| --- | ---: | --- | ---: | ---: | ---: |
-| 8 binary label metrics via `evaluate()` | 1,000,000 | scikit-learn | 30.2 | 1020.2 | **33.8×** |
-| macro F1, 10 classes | 1,000,000 | scikit-learn | 22.4 | 128.7 | **5.8×** |
-| ROC AUC, binary | 1,000,000 | scikit-learn | 173.2 | 300.6 | **1.7×** |
-| MAE, MSE, RMSE, R² via `evaluate()` | 1,000,000 | scikit-learn | 19.1 | 9.7 | 0.51× |
-| sensitivity, specificity, LR+, LR− | 1,000,000 | scikit-learn | 66.2 | 392.8 | **5.9×** |
-| calibration slope and intercept | 1,000,000 | statsmodels | 178.0 | 1014.8 | **5.7×** |
-| decision curve, 99 thresholds | 1,000,000 | NumPy loop | 155.2 | 174.3 | **1.1×** |
-| diagnostic report (7 CIs) | 1,000,000 | statsmodels | 16.6 | 4.6 | 0.28× |
-| Welch t-test | 1,000,000 | SciPy | 14.5 | 7.3 | 0.51× |
-| Hochberg correction | 1,000,000 | statsmodels | 75.4 | 81.9 | **1.1×** |
-| segmentation Dice and IoU per class | 1,000,000 px | scikit-learn | 40.2 | 282.0 | **7.0×** |
-| COCO detection evaluation (12 numbers) | 1,000 images | pycocotools | 967.4 | 980.6 | **1.0×** |
-| Hausdorff distance | 50 images | SciPy | 26.2 | 20.8 | 0.79× |
+Linux x86_64). **Overall: 45 of 45 rows agree with the reference** (largest
+difference 2.8e-14), 32 are faster, and the geometric-mean speed-up across all
+15 cases and three sizes is **2.22×**. At 1,000,000 samples (pixels for
+segmentation; 1,000 images for detection and 50 for Hausdorff), in alphabetical order:
+
+| Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
+| --- | --- | ---: | ---: | ---: |
+| 10 classes: macro F1 | scikit-learn | 29.3 | 130.8 | **4.5×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 39.5 | 1084.1 | **27.4×** |
+| binary: ROC AUC | scikit-learn | 195.0 | 348.3 | **1.8×** |
+| calibration: slope and intercept | statsmodels | 193.0 | 1068.0 | **5.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 23.5 | 4.7 | 0.20× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 69.1 | 414.0 | **6.0×** |
+| decision curve: 99 thresholds | NumPy loop | 156.3 | 181.6 | **1.2×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 817.3 | 983.7 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 82.0 | 108.0 | **1.3×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 23.1 | 10.8 | 0.47× |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 43.0 | 290.0 | **6.8×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 27.0 | 20.2 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.5 | 0.4 | 0.88× |
+| statistics: Mann–Whitney U | SciPy | 349.7 | 350.2 | **1.0×** |
+| statistics: Welch t-test | SciPy | 14.5 | 7.2 | 0.49× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/README.md

@@ -10,7 +10,7 @@
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.3.0).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
+> **Status: stable (0.3.1).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -223,24 +223,28 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 ## Performance
 
 Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). Every result agrees with the reference to floating-point rounding (largest difference
-1.4e-14).
-
-| Case | n | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
-| --- | ---: | --- | ---: | ---: | ---: |
-| 8 binary label metrics via `evaluate()` | 1,000,000 | scikit-learn | 30.2 | 1020.2 | **33.8×** |
-| macro F1, 10 classes | 1,000,000 | scikit-learn | 22.4 | 128.7 | **5.8×** |
-| ROC AUC, binary | 1,000,000 | scikit-learn | 173.2 | 300.6 | **1.7×** |
-| MAE, MSE, RMSE, R² via `evaluate()` | 1,000,000 | scikit-learn | 19.1 | 9.7 | 0.51× |
-| sensitivity, specificity, LR+, LR− | 1,000,000 | scikit-learn | 66.2 | 392.8 | **5.9×** |
-| calibration slope and intercept | 1,000,000 | statsmodels | 178.0 | 1014.8 | **5.7×** |
-| decision curve, 99 thresholds | 1,000,000 | NumPy loop | 155.2 | 174.3 | **1.1×** |
-| diagnostic report (7 CIs) | 1,000,000 | statsmodels | 16.6 | 4.6 | 0.28× |
-| Welch t-test | 1,000,000 | SciPy | 14.5 | 7.3 | 0.51× |
-| Hochberg correction | 1,000,000 | statsmodels | 75.4 | 81.9 | **1.1×** |
-| segmentation Dice and IoU per class | 1,000,000 px | scikit-learn | 40.2 | 282.0 | **7.0×** |
-| COCO detection evaluation (12 numbers) | 1,000 images | pycocotools | 967.4 | 980.6 | **1.0×** |
-| Hausdorff distance | 50 images | SciPy | 26.2 | 20.8 | 0.79× |
+Linux x86_64). **Overall: 45 of 45 rows agree with the reference** (largest
+difference 2.8e-14), 32 are faster, and the geometric-mean speed-up across all
+15 cases and three sizes is **2.22×**. At 1,000,000 samples (pixels for
+segmentation; 1,000 images for detection and 50 for Hausdorff), in alphabetical order:
+
+| Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
+| --- | --- | ---: | ---: | ---: |
+| 10 classes: macro F1 | scikit-learn | 29.3 | 130.8 | **4.5×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 39.5 | 1084.1 | **27.4×** |
+| binary: ROC AUC | scikit-learn | 195.0 | 348.3 | **1.8×** |
+| calibration: slope and intercept | statsmodels | 193.0 | 1068.0 | **5.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 23.5 | 4.7 | 0.20× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 69.1 | 414.0 | **6.0×** |
+| decision curve: 99 thresholds | NumPy loop | 156.3 | 181.6 | **1.2×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 817.3 | 983.7 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 82.0 | 108.0 | **1.3×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 23.1 | 10.8 | 0.47× |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 43.0 | 290.0 | **6.8×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 27.0 | 20.2 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.5 | 0.4 | 0.88× |
+| statistics: Mann–Whitney U | SciPy | 349.7 | 350.2 | **1.0×** |
+| statistics: Welch t-test | SciPy | 14.5 | 7.2 | 0.49× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/pyproject.toml

@@ -43,7 +43,7 @@ dev = [
   "pytest>=7",
   "pytest-cov>=4",
   "hypothesis>=6.80",
-  "scikit-learn>=1.2",
+  "scikit-learn>=1.4",
   "statsmodels>=0.13",
   "matplotlib>=3.5",
   "pillow>=9",

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/benchmarks.py

@@ -17,8 +17,10 @@ whose reference library is not installed is timed for EvalSuite only.
 from __future__ import annotations
 
 import contextlib
+import importlib
 import json
 import platform
+import re
 import time
 import tracemalloc
 import warnings
@@ -371,6 +373,28 @@ def _clinical_cases(n: int, rng: np.random.Generator) -> list[Case]:
     return [(name, ref, es_fn, own if own is not None else refs.get(name)) for name, ref, es_fn, own in cases]
 
 
+_GROUPS = (
+    ("Classification and regression", ("binary", "10 classes", "regression")),
+    ("Clinical, calibration and statistics", ("clinical", "calibration", "decision", "statistics", "multiple")),
+    ("Segmentation and object detection", ("segmentation", "detection")),
+)
+MATCH_TOLERANCE = 1e-9
+
+
+def benchmark_group(case: str) -> str:
+    """The suite a benchmark case belongs to, by its name."""
+    lowered = case.lower()
+    for title, prefixes in _GROUPS:
+        if lowered.startswith(prefixes):
+            return title
+    return "Other"
+
+
+def _geomean(values: Sequence[float]) -> Optional[float]:
+    vals = [v for v in values if v is not None and v > 0]
+    return float(np.exp(np.mean(np.log(vals)))) if vals else None
+
+
 @dataclass(frozen=True, eq=False)
 class BenchmarkResult:
     """Rows of timings (milliseconds), peak memory (MiB) and agreement, plus the environment they ran in."""
@@ -382,6 +406,60 @@ class BenchmarkResult:
         object.__setattr__(self, "rows", tuple(MappingProxyType(dict(r)) for r in self.rows))
         object.__setattr__(self, "environment", MappingProxyType(dict(self.environment)))
 
+    def sorted_rows(self) -> list[Any]:
+        """Rows in alphabetical order of case, then by size."""
+        return sorted(self.rows, key=lambda r: (r["case"].lower(), r["n"]))
+
+    def overall(self) -> list[dict[str, Any]]:
+        """One summary row per suite and one for everything: cases, how many agree with the reference
+        (max |difference| <= 1e-9), and the geometric-mean, smallest and largest speed-up."""
+
+        def summarise(name: str, rows: Sequence[Any]) -> dict[str, Any]:
+            compared = [r for r in rows if r["speedup"] is not None]
+            speedups = [r["speedup"] for r in compared]
+            return {
+                "group": name,
+                "cases": len({re.sub(r"\s*\(\d+ images?\)", "", r["case"]) for r in rows}),
+                "rows": len(rows),
+                "compared": len(compared),
+                "matching": sum(
+                    r["max_abs_diff"] is not None and r["max_abs_diff"] <= MATCH_TOLERANCE for r in rows
+                ),
+                "faster": sum(sp > 1 for sp in speedups),
+                "geomean_speedup": _geomean(speedups),
+                "min_speedup": min(speedups) if speedups else None,
+                "max_speedup": max(speedups) if speedups else None,
+            }
+
+        groups = sorted({benchmark_group(r["case"]) for r in self.rows})
+        out = [summarise(g, [r for r in self.rows if benchmark_group(r["case"]) == g]) for g in groups]
+        out.append(summarise("Overall", self.rows))
+        return out
+
+    _OVERALL_TITLES = ("Suite", "Cases", "Rows", "Match reference", "Faster", "Geo-mean speed-up", "Range")
+
+    def _overall_cells(self) -> list[list[str]]:
+        def x(v: Optional[float]) -> str:
+            return "–" if v is None else f"{v:.2f}×"
+
+        return [
+            [
+                o["group"],
+                str(o["cases"]),
+                str(o["rows"]),
+                f"{o['matching']}/{o['compared']}",
+                f"{o['faster']}/{o['compared']}",
+                x(o["geomean_speedup"]),
+                "–" if o["min_speedup"] is None else f"{x(o['min_speedup'])}–{x(o['max_speedup'])}",
+            ]
+            for o in self.overall()
+        ]
+
+    def overall_markdown(self) -> str:
+        """The overall summary as a Markdown table."""
+        lines = ["| " + " | ".join(self._OVERALL_TITLES) + " |", "| --- |" + " ---: |" * 6]
+        return "\n".join(lines + ["| " + " | ".join(c) + " |" for c in self._overall_cells()])
+
     def _cells(self, digits: int) -> list[list[str]]:
         def f(v: Any, d: int = digits) -> str:
             return "–" if v is None else f"{v:.{d}f}"
@@ -398,7 +476,7 @@ class BenchmarkResult:
                 f(r.get("reference_peak_mb"), 2),
                 "–" if r["max_abs_diff"] is None else f"{r['max_abs_diff']:.1e}",
             ]
-            for r in self.rows
+            for r in self.sorted_rows()
         ]
 
     _TITLES = (
@@ -430,7 +508,14 @@ class BenchmarkResult:
             + f" | {env['machine']} | fastest of {env['repeat']} runs"
         )
         note = "Speed-up > 1 means EvalSuite is faster. Max |difference| compares EvalSuite with the reference."
-        return "\n".join([head, "", line(self._TITLES), *(line(c) for c in cells), "", note])
+        ocells = self._overall_cells()
+        owidths = [max(len(t), *(len(c[i]) for c in ocells)) for i, t in enumerate(self._OVERALL_TITLES)]
+
+        def oline(c: Sequence[str]) -> str:
+            return "  ".join(x.ljust(owidths[i]) if i == 0 else x.rjust(owidths[i]) for i, x in enumerate(c))
+
+        overall = ["Overall", oline(self._OVERALL_TITLES), *(oline(c) for c in ocells), ""]
+        return "\n".join([head, "", *overall, line(self._TITLES), *(line(c) for c in cells), "", note])
 
     def __repr__(self) -> str:
         return self.summary()
@@ -438,7 +523,13 @@ class BenchmarkResult:
     def to_dict(self) -> dict[str, Any]:
         return cast(
             "dict[str, Any]",
-            _json_safe({"environment": dict(self.environment), "rows": [dict(r) for r in self.rows]}),
+            _json_safe(
+                {
+                    "environment": dict(self.environment),
+                    "overall": self.overall(),
+                    "rows": [dict(r) for r in self.rows],
+                }
+            ),
         )
 
     def to_json(self, *, indent: Optional[int] = 2) -> str:
@@ -514,7 +605,7 @@ def run_benchmarks(
     if compare_sklearn:
         for mod in versions:
             with contextlib.suppress(ImportError):
-                __import__(mod)
+                importlib.import_module(mod)  # fixed names only: sklearn, statsmodels, pycocotools
                 from importlib.metadata import version as _dist_version
 
                 versions[mod] = _dist_version("scikit-learn" if mod == "sklearn" else mod)

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/core/validation.py

@@ -45,9 +45,25 @@ def to_numpy(x: ArrayLike, name: str, *, allow_2d: bool = False) -> NDArray[Any]
         raise InputValidationError(f"{name} must be {expected}; received an array with shape {arr.shape}.")
     if arr.shape[0] == 0:
         raise InputValidationError(f"{name} is empty. Provide at least one observation.")
+    if (
+        arr.ndim == 1
+        and arr.dtype.kind in "US"
+        and isinstance(x, (list, tuple))
+        and not all(isinstance(v, (str, bytes)) for v in x)
+    ):
+        # NumPy silently turns [0, "a"] into ["0", "a"]; a mix of numbers and strings is almost always a bug.
+        raise InputValidationError(f"{name} mixes numbers and strings; use one label type throughout.")
     return arr
 
 
+def _label_kind(arr: NDArray[Any]) -> str:
+    if arr.dtype.kind in "US":
+        return "string"
+    if arr.dtype.kind in "biuf":
+        return "number"
+    return "other"
+
+
 def check_finite(arr: NDArray[Any], name: str) -> None:
     if np.issubdtype(arr.dtype, np.number) and not np.all(np.isfinite(arr)):
         n_nan = int(np.isnan(arr).sum())
@@ -126,6 +142,11 @@ def resolve_labels(
         if np.unique(lab).shape[0] != lab.shape[0]:
             raise InputValidationError("labels contains duplicates.")
         return lab
+    if y_pred is not None and {_label_kind(y_true), _label_kind(y_pred)} == {"string", "number"}:
+        raise InputValidationError(
+            f"y_true has {_label_kind(y_true)} labels but y_pred has {_label_kind(y_pred)} labels; "
+            "use the same label type in both (for example map class names to integers first)."
+        )
     present = y_true if y_pred is None else np.concatenate([y_true, y_pred])
     try:
         return unique_labels(present)

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/regression/metrics.py

@@ -202,7 +202,15 @@ def rmse(
 ) -> MetricResult:
     """Root mean squared error (square root taken per output, then averaged)."""
     inp = _inputs(y_true, y_pred, sample_weight)
-    return _finish(np.sqrt(inp.mean(inp.error**2)), inp, multioutput, "rmse", "RMSE")
+    with np.errstate(over="ignore"):
+        per_output = np.sqrt(inp.mean(inp.error**2))
+    if not np.all(np.isfinite(per_output)):
+        # Errors above ~1e154 overflow when squared although the RMSE itself is representable:
+        # rescale by the largest error and square again.
+        scale = np.abs(inp.error).max(axis=0)
+        scale = np.where(np.isfinite(scale) & (scale > 0), scale, 1.0)
+        per_output = scale * np.sqrt(inp.mean((inp.error / scale) ** 2))
+    return _finish(per_output, inp, multioutput, "rmse", "RMSE")
 
 
 def _r2_per_output(inp: _Inputs) -> FloatArray:

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/stats/_resolve.py

@@ -10,7 +10,7 @@ from typing import Any, Optional
 import numpy as np
 from numpy.typing import NDArray
 
-from ..core.exceptions import InputValidationError
+from ..core.exceptions import InputValidationError, StatisticalTestError
 from ..core.validation import target_type, to_numpy
 
 MetricFn = Callable[..., Any]
@@ -89,6 +89,11 @@ class MetricCall:
                 f"observations. Received {self.y_true.shape[0]} and {self.second.shape[0]}."
             )
         self.n = self.y_true.shape[0]
+        if self.n < 2:
+            raise StatisticalTestError(
+                f"Resampling needs at least 2 observations; got {self.n}. An interval or test from a single "
+                "observation would have zero width and no meaning."
+            )
         self.weight = None if sample_weight is None else to_numpy(sample_weight, "sample_weight")
         self.kwargs = dict(kwargs or {})
         params = inspect.signature(fn).parameters

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/version.py

@@ -1,3 +1,3 @@
 """Package version (single source of truth, read by the build backend)."""
 
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/src/evalsuite/vision/segmentation.py

@@ -84,9 +84,17 @@ _REF_HD = (
 
 # ---- input handling ------------------------------------------------------------------------------
 def _as_images(masks: Masks, name: str) -> list[NDArray[Any]]:
+    images = None
     if isinstance(masks, (list, tuple)) or (isinstance(masks, np.ndarray) and masks.dtype == object):
         images = [np.asarray(m) for m in masks]
-    else:
+        if (
+            images
+            and isinstance(masks, (list, tuple))
+            and all(m.ndim == 1 for m in images)
+            and not any(isinstance(m, np.ndarray) for m in masks)
+        ):
+            images = None  # a nested Python list of rows, e.g. [[1, 1], [0, 0]], is one 2-D mask
+    if images is None:
         arr = np.asarray(masks)
         if arr.ndim < 2:
             raise InputValidationError(

evalsuite_python-0.3.1/tests/integration/test_qa.py

@@ -0,0 +1,311 @@
+"""Release QA: edge cases against reference libraries, documented conventions, exports, reproducibility.
+
+Each test names the QA item it covers. Regression tests for defects fixed in 0.3.1 are marked "0.3.1 fix".
+"""
+
+from __future__ import annotations
+
+import html
+import json
+import math
+import warnings
+
+import numpy as np
+import pytest
+
+import evalsuite as es
+
+sk = pytest.importorskip("sklearn.metrics")
+
+
+# ---------------------------------------------------------------- classification vs scikit-learn
+def _binary(seed: int, n: int = 300, prevalence: float = 0.5):
+    rng = np.random.default_rng(seed)
+    y = (rng.random(n) < prevalence).astype(int)
+    p = np.clip(0.35 * y + rng.random(n) * 0.65, 0, 1)
+    return y, (p >= 0.5).astype(int), p
+
+
+@pytest.mark.parametrize("prevalence", [0.5, 0.05])  # balanced and imbalanced
+@pytest.mark.parametrize("seed", [0, 1])
+def test_binary_metrics_match_sklearn(seed: int, prevalence: float) -> None:
+    y, pred, prob = _binary(seed, prevalence=prevalence)
+    pairs = {
+        es.accuracy: sk.accuracy_score,
+        es.precision: lambda a, b: sk.precision_score(a, b, zero_division=0),
+        es.recall: sk.recall_score,
+        es.f1: lambda a, b: sk.f1_score(a, b, zero_division=0),
+        es.mcc: sk.matthews_corrcoef,
+        es.balanced_accuracy: sk.balanced_accuracy_score,
+        es.sensitivity: sk.recall_score,
+        es.ppv: lambda a, b: sk.precision_score(a, b, zero_division=0),
+    }
+    for ours, ref in pairs.items():
+        assert float(ours(y, pred)) == pytest.approx(ref(y, pred), abs=1e-12), ours.__name__
+    assert float(es.specificity(y, pred)) == pytest.approx(sk.recall_score(y, pred, pos_label=0))
+    assert float(es.npv(y, pred)) == pytest.approx(sk.precision_score(y, pred, pos_label=0, zero_division=0))
+    assert float(es.roc_auc(y, prob)) == pytest.approx(sk.roc_auc_score(y, prob), abs=1e-12)
+    np.testing.assert_array_equal(es.confusion_matrix(y, pred), sk.confusion_matrix(y, pred))
+
+
+@pytest.mark.parametrize("average", ["macro", "micro", "weighted"])
+def test_multiclass_with_missing_predicted_class_matches_sklearn(average: str) -> None:
+    y = np.array([0, 1, 2, 2, 1, 0, 2, 2])
+    pred = np.array([0, 1, 1, 1, 1, 0, 1, 0])  # class 2 is never predicted
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        for ours, ref in [(es.precision, sk.precision_score), (es.recall, sk.recall_score), (es.f1, sk.f1_score)]:
+            expected = ref(y, pred, average=average, zero_division=0)
+            assert float(ours(y, pred, average=average)) == pytest.approx(expected, abs=1e-12)
+
+
+def test_multiclass_auc_follows_sorted_label_order_for_string_labels() -> None:
+    y = np.array(["cat", "dog", "bird", "dog", "cat", "bird"])
+    prob = np.array(
+        [[0.1, 0.8, 0.1], [0.1, 0.2, 0.7], [0.7, 0.2, 0.1], [0.2, 0.2, 0.6], [0.2, 0.6, 0.2], [0.5, 0.3, 0.2]]
+    )  # columns: bird, cat, dog
+    assert float(es.roc_auc(y, prob)) == pytest.approx(sk.roc_auc_score(y, prob, multi_class="ovr"))
+
+
+def test_zero_division_policy_is_documented_and_configurable() -> None:
+    y, pred = [0, 1, 1], [0, 0, 0]  # no predicted positives
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        assert float(es.precision(y, pred)) == 0.0
+        assert float(es.precision(y, pred, zero_division=1.0)) == 1.0
+    assert float(es.recall([0, 1, 1], [1, 0, 0])) == 0.0  # zero true positives
+
+
+@pytest.mark.parametrize(
+    ("call", "match"),
+    [
+        (lambda: es.accuracy([], []), "empty"),
+        (lambda: es.accuracy([0, np.nan], [0, 1]), "NaN"),
+        (lambda: es.roc_auc([0, 1], [0.2, np.inf]), "infinite"),
+        (lambda: es.roc_auc([0, 1], [0.2, 1.4]), r"\[0, 1\]"),
+        (lambda: es.accuracy([0, 1], [0]), "same number"),
+        (lambda: es.roc_auc([0, 1, 2], np.full((3, 2), 0.5)), "columns"),
+        (lambda: es.roc_auc([0, 1], np.full((2, 2, 2), 0.5)), "1-D or 2-D"),
+        (lambda: es.roc_auc([0, 1, 2], np.full((3, 3), 0.5)), "sum to 1"),
+    ],
+)
+def test_invalid_classification_inputs_raise_clear_errors(call, match: str) -> None:
+    with pytest.raises(es.InputValidationError, match=match):
+        call()
+
+
+def test_single_class_target_auc_is_an_error_and_mcc_follows_sklearn() -> None:
+    with pytest.raises(es.MetricInputError, match="no negative"):
+        es.roc_auc([1, 1, 1], [0.2, 0.5, 0.9])
+    assert float(es.mcc([1, 1, 1], [1, 1, 1])) == sk.matthews_corrcoef([1, 1, 1], [1, 1, 1])
+
+
+def test_mixed_number_and_string_labels_are_rejected() -> None:  # 0.3.1 fix
+    with pytest.raises(es.InputValidationError, match="mixes numbers and strings"):
+        es.accuracy([0, "a"], [0, 1])
+    with pytest.raises(es.InputValidationError, match="string labels but y_pred has number"):
+        es.accuracy(np.array(["a", "b"]), np.array([0, 1]))
+    assert float(es.accuracy(["a", "b"], ["a", "a"])) == 0.5  # all strings stay fine
+
+
+# ---------------------------------------------------------------- regression vs scikit-learn
+@pytest.mark.parametrize("dtype", [np.int64, np.float32, np.float64])
+def test_regression_matches_sklearn_for_int_and_float_inputs(dtype) -> None:
+    rng = np.random.default_rng(3)
+    y = (rng.normal(50, 10, 200)).astype(dtype)
+    p = (y + rng.normal(0, 3, 200)).astype(dtype)
+    yf, pf = y.astype(np.float64), p.astype(np.float64)
+    assert float(es.mae(y, p)) == pytest.approx(sk.mean_absolute_error(yf, pf), rel=1e-12)
+    assert float(es.mse(y, p)) == pytest.approx(sk.mean_squared_error(yf, pf), rel=1e-12)
+    assert float(es.rmse(y, p)) == pytest.approx(math.sqrt(sk.mean_squared_error(yf, pf)), rel=1e-12)
+    assert float(es.r2(y, p)) == pytest.approx(sk.r2_score(yf, pf), rel=1e-12)
+    assert isinstance(es.mse(y, p).value, float)
+
+
+def test_multioutput_regression_matches_sklearn() -> None:
+    rng = np.random.default_rng(4)
+    y = rng.normal(size=(100, 3))
+    p = y + rng.normal(0, 0.5, (100, 3))
+    np.testing.assert_allclose(
+        es.mae(y, p, multioutput="raw_values").value, sk.mean_absolute_error(y, p, multioutput="raw_values")
+    )
+    assert float(es.r2(y, p)) == pytest.approx(sk.r2_score(y, p))
+
+
+def test_degenerate_regression_inputs() -> None:
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        assert float(es.r2([2, 2, 2], [2, 2, 3])) == sk.r2_score([2, 2, 2], [2, 2, 3])  # constant target
+        assert float(es.r2([2, 2, 2], [2, 2, 2])) == 1.0
+    assert float(es.mae([1.5], [1.0])) == 0.5  # single sample
+    for bad, match in [(([], []), "empty"), (([1, np.nan], [1, 2]), "NaN"), (([1, np.inf], [1, 2]), "infinite")]:
+        with pytest.raises(es.InputValidationError, match=match):
+            es.mse(*bad)
+
+
+def test_rmse_does_not_overflow_for_large_values() -> None:  # 0.3.1 fix
+    value = float(es.rmse([1e154, 2e154], [0.0, 0.0]))
+    assert math.isfinite(value)
+    assert value == pytest.approx(math.sqrt((1 + 4) / 2) * 1e154, rel=1e-12)
+    assert float(es.rmse([1.0, 3.0], [0.0, 0.0])) == pytest.approx(math.sqrt(5))  # unchanged path
+
+
+# ---------------------------------------------------------------- statistics
+def test_resampling_needs_two_observations() -> None:  # 0.3.1 fix
+    with pytest.raises(es.StatisticalTestError, match="at least 2 observations"):
+        es.bootstrap_ci("accuracy", [1], [1])
+    with pytest.raises(es.StatisticalTestError, match="at least 2 observations"):
+        es.paired_bootstrap_test("accuracy", [1], [1], [0])
+
+
+def test_randomised_procedures_are_reproducible_with_a_seed() -> None:
+    y, pred, _ = _binary(5)
+    a = es.bootstrap_ci("f1", y, pred, n_resamples=300, random_state=7)
+    b = es.bootstrap_ci("f1", y, pred, n_resamples=300, random_state=7)
+    assert (a.low, a.high) == (b.low, b.high)
+    c1 = es.compare(y, {"A": pred, "B": 1 - pred}, n_resamples=200, random_state=1)
+    c2 = es.compare(y, {"A": pred, "B": 1 - pred}, n_resamples=200, random_state=1)
+    assert c1.to_json() == c2.to_json()
+
+
+def test_multiple_testing_matches_statsmodels() -> None:
+    mt = pytest.importorskip("statsmodels.stats.multitest")
+    p = np.array([0.001, 0.008, 0.039, 0.041, 0.042, 0.06, 0.074, 0.205, 0.5])
+    for ours, theirs in [
+        ("bonferroni", "bonferroni"),
+        ("holm", "holm"),
+        ("hochberg", "simes-hochberg"),
+        ("bh", "fdr_bh"),
+        ("by", "fdr_by"),
+    ]:
+        np.testing.assert_allclose(es.adjust_pvalues(p, method=ours), mt.multipletests(p, method=theirs)[1])
+
+
+# ---------------------------------------------------------------- clinical
+def test_ppv_and_npv_follow_bayes_with_prevalence() -> None:
+    y, pred, _ = _binary(6, n=2000, prevalence=0.2)
+    se, sp = float(es.sensitivity(y, pred)), float(es.specificity(y, pred))
+    prev = y.mean()
+    assert float(es.ppv(y, pred)) == pytest.approx(se * prev / (se * prev + (1 - sp) * (1 - prev)))
+    assert float(es.npv(y, pred)) == pytest.approx(sp * (1 - prev) / (sp * (1 - prev) + (1 - se) * prev))
+    assert float(es.lr_positive(y, pred)) == pytest.approx(se / (1 - sp))
+    assert float(es.lr_negative(y, pred)) == pytest.approx((1 - se) / sp)
+
+
+def test_decision_curve_net_benefit_and_reference_strategies() -> None:
+    y, _, prob = _binary(8, n=500, prevalence=0.3)
+    dca = es.decision_curve(y, {"m": prob}, thresholds=[0.1, 0.25, 0.5])
+    prev = y.mean()
+    for i, t in enumerate(dca.thresholds):
+        pos = prob >= t
+        tp, fp = np.sum(pos & (y == 1)), np.sum(pos & (y == 0))
+        expected = tp / y.size - fp / y.size * t / (1 - t)
+        assert dca.net_benefit["m"][i] == pytest.approx(expected)
+        assert dca.treat_all[i] == pytest.approx(prev - (1 - prev) * t / (1 - t))
+        assert dca.treat_none[i] == 0.0
+
+
+# ---------------------------------------------------------------- segmentation conventions
+def test_segmentation_empty_masks_absent_classes_and_ignore_index() -> None:
+    empty = np.zeros((4, 4), int)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        per_class = es.dice(empty, empty, num_classes=3, average=None).value
+    assert per_class[0] == 1.0 and np.isnan(per_class[1:]).all()  # absent classes are NaN, not 0
+    assert float(es.dice(empty, empty, num_classes=3, empty=1.0)) == 1.0  # explicit policy
+    t = np.array([[1, 1, 255], [0, 0, 255]])
+    p = np.array([[1, 0, 1], [0, 0, 0]])
+    assert float(es.pixel_accuracy(t, p, ignore_index=255)) == pytest.approx(3 / 4)
+
+
+def test_per_image_and_dataset_aggregation_differ_as_documented() -> None:
+    a = np.zeros((2, 10, 10), int)
+    b = a.copy()
+    a[0, :5] = 1
+    b[0, :5] = 1  # image 0 perfect
+    a[1, 0, 0] = 1  # image 1: one pixel, missed
+    dataset = float(es.dice(a, b, average=None, aggregate="dataset").value[1])
+    image = float(es.dice(a, b, average=None, aggregate="image").value[1])
+    assert dataset == pytest.approx(2 * 50 / (51 + 50))
+    assert image == pytest.approx((1.0 + 0.0) / 2)
+
+
+# ---------------------------------------------------------------- exports and API
+UNTRUSTED = '<script>alert("x")</script> & café'
+
+
+def _reports():
+    rng = np.random.default_rng(0)
+    yc = rng.integers(0, 2, 60)
+    masks = rng.integers(0, 2, (2, 8, 8))
+    det = [{"boxes": [[0, 0, 10, 10]], "labels": [1]}]
+    pred = [{"boxes": [[0, 0, 10, 10]], "labels": [1], "scores": [0.9]}]
+    return {
+        "classification": es.classification_report(*[np.where(v == 0, UNTRUSTED, "β-class") for v in (yc, yc)]),
+        "segmentation": es.segmentation_report(masks, masks, class_names={0: UNTRUSTED, 1: "β-class"}),
+        "detection": es.detection_report(det, pred),
+        "evaluation": es.evaluate(yc, yc),
+        "diagnostic": es.diagnostic_report(yc, yc),
+    }
+
+
+@pytest.mark.parametrize("kind", ["classification", "segmentation", "detection", "evaluation", "diagnostic"])
+def test_every_export_format_is_valid_and_safe(kind: str, tmp_path) -> None:
+    report = _reports()[kind]
+    data = json.loads(report.to_json())  # valid JSON (NaN is written as null)
+    assert isinstance(data, dict)
+    page = report.to_html()
+    assert "<script>" not in page
+    if UNTRUSTED in str(getattr(report, "rows", "")) or kind == "classification":
+        assert html.escape(UNTRUSTED) in page or "&lt;script&gt;" in page
+    for ext in ("csv", "md", "tex", "json", "html"):
+        out = tmp_path / f"r.{ext}"
+        report.save(out)
+        assert out.read_text(encoding="utf-8").strip()
+
+
+def test_json_export_preserves_values_and_non_finite() -> None:
+    masks = np.zeros((1, 4, 4), int)
+    masks[0, 0, 0] = 1
+    pred = np.zeros_like(masks)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        rep = es.segmentation_report(masks, pred)
+    data = json.loads(rep.to_json())
+    assert "NaN" not in rep.to_json() and "Infinity" not in rep.to_json()
+    assert data["values"]["miou"] == pytest.approx(rep["miou"])
+
+
+def test_public_api_names_are_stable() -> None:
+    required = [
+        "evaluate",
+        "compare",
+        "bootstrap_ci",
+        "accuracy",
+        "f1",
+        "roc_auc",
+        "mae",
+        "rmse",
+        "r2",
+        "diagnostic_report",
+        "decision_curve",
+        "calibration_report",
+        "t_test",
+        "adjust_pvalues",
+        "dice",
+        "iou",
+        "miou",
+        "hausdorff_distance",
+        "segmentation_report",
+        "detection_report",
+        "mean_average_precision",
+        "box_iou",
+        "from_coco",
+        "plot",
+        "list_metrics",
+        "metric_info",
+    ]
+    missing = [name for name in required if not hasattr(es, name)]
+    assert not missing
+    assert isinstance(es.accuracy([0, 1], [0, 1]), es.MetricResult)
+    assert isinstance(es.t_test([1.0, 2.0, 3.0], [2.0, 3.0, 4.0]), es.TestResult)

evalsuite_python-0.3.1/tests/integration/test_readme_examples.py

@@ -0,0 +1,122 @@
+"""Every Python example in README.md runs against the package.
+
+Placeholder lines (code containing ``...``) are skipped; the variables the examples assume are provided by
+fixtures matched to the README section the example sits in.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import warnings
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+import evalsuite as es
+
+pytest.importorskip("matplotlib")
+import matplotlib
+
+matplotlib.use("Agg")
+
+README = Path(__file__).resolve().parents[2] / "README.md"
+
+
+def _blocks() -> list[tuple[str, str]]:
+    text = README.read_text(encoding="utf-8")
+    out, heading = [], ""
+    for match in re.finditer(r"^(#+ [^\n]*)$|^```python\n(.*?)^```", text, flags=re.M | re.S):
+        if match.group(1):
+            heading = match.group(1).lstrip("# ").strip()
+        else:
+            out.append((heading, match.group(2)))
+    return out
+
+
+def _runnable(code: str) -> str:
+    lines = []
+    for line in code.splitlines():
+        if "..." in line.split("#", 1)[0]:
+            continue  # placeholder such as [x1, y1, ...] or es.compare(...)
+        lines.append(line)
+    return "\n".join(lines)
+
+
+def _namespace(tmp_path: Path) -> dict:
+    rng = np.random.default_rng(0)
+    y = rng.integers(0, 2, 200)
+
+    def prob(strength: float) -> np.ndarray:
+        return 1 / (1 + np.exp(-(strength * (y - 0.5) + rng.normal(0, 1, 200))))
+
+    probs = {k: prob(s) for k, s in {"lr": 2.0, "rf": 1.6, "gb": 1.2, "a": 2.0, "b": 1.0}.items()}
+    preds = {k: (v >= 0.5).astype(int) for k, v in probs.items()}
+    masks = rng.integers(0, 2, (3, 24, 24))
+    boxes = [{"boxes": [[0, 0, 10, 10], [20, 20, 40, 40]], "labels": [1, 2]}]
+    dets = [{"boxes": [[1, 0, 10, 11], [19, 20, 40, 41]], "labels": [1, 2], "scores": [0.9, 0.8]}]
+    coco_gt = {
+        "images": [{"id": 1, "width": 50, "height": 50}],
+        "annotations": [
+            {"id": 1, "image_id": 1, "category_id": 1, "bbox": [0, 0, 10, 10], "area": 100, "iscrowd": 0}
+        ],
+        "categories": [{"id": 1, "name": "cell"}],
+    }
+    (tmp_path / "instances_val.json").write_text(json.dumps(coco_gt))
+    (tmp_path / "detections.json").write_text(
+        json.dumps([{"image_id": 1, "category_id": 1, "bbox": [0, 0, 10, 10], "score": 0.9}])
+    )
+    g = [rng.normal(m, 1, 30) for m in (0, 0.3, 0.6)]
+    return {
+        "es": es,
+        "y_true": y,
+        "y_pred": preds["lr"],
+        "y_prob": probs["lr"],
+        "prob": probs["lr"],
+        **{f"pred_{k}": v for k, v in preds.items()},
+        **{f"prob_{k}": v for k, v in probs.items()},
+        "scores_a": g[0],
+        "scores_b": g[1],
+        "scores_c": g[2],
+        "fold_scores_a": g[0],
+        "fold_scores_b": g[1],
+        "a": g[0],
+        "b": g[1],
+        "g1": g[0],
+        "g2": g[1],
+        "g3": g[2],
+        "residuals": g[0],
+        "table": np.array([[20, 10], [8, 25]]),
+        "p_values": [0.01, 0.04, 0.03, 0.2],
+        "y_reg": g[0],
+        "pred_reg": g[0] + rng.normal(0, 0.2, 30),
+        "masks": masks,
+        "image": rng.random((24, 24)),
+        "boxes_a": [[0, 0, 10, 10]],
+        "boxes_b": [[0, 0, 5, 5]],
+        "_vision": {"masks": masks, "boxes": boxes, "dets": dets},
+    }
+
+
+EXAMPLES = _blocks()
+
+
+@pytest.mark.parametrize(("heading", "code"), EXAMPLES, ids=[f"{i}-{h[:30]}" for i, (h, _) in enumerate(EXAMPLES)])
+def test_readme_example_runs(heading: str, code: str, tmp_path, monkeypatch) -> None:
+    monkeypatch.chdir(tmp_path)
+    ns = _namespace(tmp_path)
+    vision = ns.pop("_vision")
+    if heading.lower().startswith("segmentation"):
+        ns["y_true"], ns["y_pred"] = vision["masks"], np.roll(vision["masks"], 1, axis=2)
+    if heading.lower().startswith("object detection"):
+        ns["y_true"], ns["y_pred"] = vision["boxes"], vision["dets"]
+    if heading.lower().startswith("classification report") or heading.lower().startswith("plots"):
+        ns["y_pred"] = (ns["y_prob"] >= 0.5).astype(int)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        exec(compile(_runnable(code), f"README.md [{heading}]", "exec"), ns)  # noqa: S102
+
+
+def test_readme_has_examples() -> None:
+    assert len(EXAMPLES) >= 8

evalsuite_python-0.3.1/tests/integration/test_repo_hygiene.py

@@ -0,0 +1,97 @@
+"""Security and maintenance checks on the repository itself (run in CI on every push)."""
+
+from __future__ import annotations
+
+import re
+import subprocess
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parents[2]
+SRC = ROOT / "src" / "evalsuite"
+
+SECRET = re.compile(
+    r"(github_pat_[A-Za-z0-9_]{20,}|gh[pousr]_[A-Za-z0-9]{30,}|pypi-AgEI[A-Za-z0-9_-]{20,}|AKIA[0-9A-Z]{16}|"
+    r"-----BEGIN [A-Z ]*PRIVATE KEY-----|xox[baprs]-[A-Za-z0-9-]{10,}|sk-[A-Za-z0-9]{32,})"
+)
+
+
+def _tracked_files() -> list[Path]:
+    try:
+        out = subprocess.run(
+            ["git", "ls-files"],  # noqa: S607
+            cwd=ROOT,
+            capture_output=True,
+            text=True,
+            check=True,
+        ).stdout
+        return [ROOT / line for line in out.splitlines() if line]
+    except (OSError, subprocess.CalledProcessError):  # sdist without .git: scan what is there
+        return [p for p in ROOT.rglob("*") if p.is_file() and "__pycache__" not in p.parts]
+
+
+def test_no_credentials_in_repository() -> None:
+    leaks = []
+    for path in _tracked_files():
+        if not path.is_file() or path.stat().st_size > 2_000_000:
+            continue
+        text = path.read_text(encoding="utf-8", errors="ignore")
+        if SECRET.search(text):
+            leaks.append(str(path.relative_to(ROOT)))
+    assert not leaks, f"possible credentials committed: {leaks}"
+
+
+@pytest.mark.parametrize(
+    "pattern",
+    [
+        r"\bpickle\b",
+        r"(?<![\w.])eval\(",
+        r"(?<![\w.])exec\(",
+        r"\bsubprocess\b",
+        r"shell\s*=\s*True",
+        r"yaml\.load\(",
+        r"allow_pickle\s*=\s*True",
+        r"__import__\(",
+    ],
+)
+def test_no_unsafe_calls_in_package(pattern: str) -> None:
+    hits = [
+        f"{p.relative_to(ROOT)}:{i}"
+        for p in SRC.rglob("*.py")
+        for i, line in enumerate(p.read_text(encoding="utf-8").splitlines(), 1)
+        if re.search(pattern, line.split("#", 1)[0])
+    ]
+    assert not hits, f"unsafe call {pattern!r} in {hits}"
+
+
+def test_numpy_loads_never_allow_pickle() -> None:
+    for p in SRC.rglob("*.py"):
+        for line in p.read_text(encoding="utf-8").splitlines():
+            if "np.load(" in line:
+                assert "allow_pickle=False" in line, f"{p.name}: {line.strip()}"
+
+
+def test_workflows_use_least_privilege() -> None:
+    workflows = list((ROOT / ".github" / "workflows").glob("*.yml"))
+    if not workflows:
+        pytest.skip("workflows are not shipped in the sdist")
+    for wf in workflows:
+        text = wf.read_text(encoding="utf-8")
+        assert re.search(r"^permissions:\s*\n\s+contents: read", text, re.M), (
+            f"{wf.name}: default must be read-only"
+        )
+        assert "pull_request_target" not in text, (
+            f"{wf.name}: pull_request_target runs untrusted code with secrets"
+        )
+        assert "password:" not in text and "PYPI_API_TOKEN" not in text, f"{wf.name}: use Trusted Publishing"
+
+
+def test_single_version_source() -> None:
+    pyproject = (ROOT / "pyproject.toml").read_text(encoding="utf-8")
+    assert 'dynamic = ["version"]' in pyproject and 'path = "src/evalsuite/version.py"' in pyproject
+    import evalsuite
+
+    version = re.search(r'__version__ = "(.+)"', (SRC / "version.py").read_text()).group(1)
+    assert evalsuite.__version__ == version
+    assert f"## [{version}]" in (ROOT / "CHANGELOG.md").read_text(encoding="utf-8")

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/tests/output/test_benchmarks.py

@@ -46,3 +46,25 @@ def test_v020_benchmarks_agree_with_their_references() -> None:
         assert row["reference_ms"] > 0 and row["max_abs_diff"] < 1e-9, row["case"]
     assert "statsmodels" in b.summary().splitlines()[0]
     assert len(run_benchmarks(sizes=(300,), repeat=1, suite="all").rows) == 15
+
+
+def test_overall_summary_and_alphabetical_rows() -> None:
+    b = run_benchmarks(sizes=(300,), repeat=1, suite="all")
+    overall = b.overall()
+    assert [o["group"] for o in overall] == [
+        "Classification and regression",
+        "Clinical, calibration and statistics",
+        "Segmentation and object detection",
+        "Overall",
+    ]
+    total = overall[-1]
+    assert total["rows"] == 15 == sum(o["rows"] for o in overall[:-1])
+    assert total["matching"] == total["compared"]  # every compared case agrees with its reference
+    if total["compared"]:
+        assert total["min_speedup"] <= total["geomean_speedup"] <= total["max_speedup"]
+    cases = [r["case"].lower() for r in b.sorted_rows()]
+    assert cases == sorted(cases)
+    text = b.summary()
+    assert text.index("Overall") < text.index("Case")  # the overall table comes first
+    assert b.overall_markdown().startswith("| Suite |")
+    assert json.loads(b.to_json())["overall"][-1]["group"] == "Overall"

{evalsuite_python-0.3.0 → evalsuite_python-0.3.1}/tests/vision/test_edges.py

@@ -124,3 +124,13 @@ def test_evaluate_points_vision_inputs_to_the_right_function() -> None:
         es.evaluate([{"boxes": [[0, 0, 1, 1]], "labels": [1]}], [{"boxes": [], "labels": []}])
     with pytest.raises(es.UnsupportedTaskError, match="segmentation_report"):
         es.evaluate(np.zeros((2, 3, 3), int), np.zeros((2, 3, 3), int))
+
+
+def test_nested_list_is_one_2d_mask() -> None:  # 0.3.1 fix
+    as_list = es.dice([[1, 1], [0, 0]], [[1, 0], [0, 0]], average=None).value
+    as_array = es.dice(np.array([[1, 1], [0, 0]]), np.array([[1, 0], [0, 0]]), average=None).value
+    np.testing.assert_array_equal(as_list, as_array)
+    stacked = es.dice([[[1, 1], [0, 0]], [[1, 0], [0, 0]]], [[[1, 1], [0, 0]], [[1, 0], [0, 0]]])
+    assert float(stacked) == 1.0  # a list of 2-D masks is still a list of images
+    with pytest.raises(es.InputValidationError):
+        es.dice([1, 0, 1], [1, 0, 1])