evalsuite-python 0.2.1__tar.gz → 0.3.0__tar.gz

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/CHANGELOG.md

@@ -6,6 +6,36 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-10-09
+
+Computer vision (the v0.3.0 roadmap), with comparison, plots, reporting and benchmarks extended to it.
+
+### Added
+- Segmentation: `dice`, `iou`, `miou`, `pixel_accuracy`, `mean_pixel_accuracy`, `boundary_iou`,
+  `hausdorff_distance` (HD and HD95, anisotropic `spacing`), `average_surface_distance`,
+  `segmentation_confusion`, `per_image_scores` and `segmentation_report`. 2-D and 3-D masks, lists of
+  differently sized masks, `ignore_index`, `aggregate="dataset"|"image"`, undefined classes reported as NaN.
+- Object detection: `box_iou`, `detection_report` (the 12 COCO numbers plus AP per class),
+  `mean_average_precision`, `average_precision_detection` (COCO or VOC interpolation), `detection_pr_curve`,
+  `from_coco`. Matches pycocotools exactly, including crowd regions, area ranges and detection limits.
+- Comparison for vision: `compare`, `bootstrap_ci` and `paired_bootstrap_test` resample images for
+  segmentation masks and per-image detections.
+- Plots: `es.plot.segmentation` (prediction fill vs truth outline), `es.plot.per_class` (any per-class
+  result, a segmentation report or a detection report), `es.plot.detection_pr`.
+- CLI: `evalsuite segmentation` (.npy/.npz or a folder of mask images) and `evalsuite detection` (COCO JSON).
+- Benchmarks: `--suite vision` against scikit-learn, SciPy and pycocotools.
+- `vision` extra (Pillow, for reading mask image folders); `CITATION.cff`.
+
+### Changed
+- `evaluate()` points segmentation masks and detection annotations to the right functions instead of
+  failing with a shape error.
+- CI and release workflows use the Node 24 versions of the GitHub actions.
+
+### Fixed
+- Calibration slope and intercept no longer emit an overflow warning before reporting a perfectly
+  separated outcome.
+- The CLI no longer fails on consoles that cannot print Unicode (Windows code pages).
+
 ## [0.2.1] - 2026-10-09
 
 ### Added

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.2.1
+Version: 0.3.0
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -11,7 +11,7 @@ Author: Manoj Kumar C S, Nikhil D Bharadwaj
 Maintainer: Manoj Kumar C S, Nikhil D Bharadwaj
 License-Expression: MIT
 License-File: LICENSE
-Keywords: calibration,classification,clinical,diagnostic accuracy,evaluation,machine learning,metrics,regression,reproducibility,statistical tests,statistics
+Keywords: calibration,classification,clinical,diagnostic accuracy,evaluation,mAP,machine learning,metrics,object detection,regression,reproducibility,segmentation,statistical tests,statistics
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Healthcare Industry
@@ -27,6 +27,7 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
 Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
 Classifier: Typing :: Typed
 Requires-Python: >=3.9
@@ -35,13 +36,16 @@ Requires-Dist: pandas>=1.4
 Requires-Dist: scipy>=1.8
 Provides-Extra: all
 Requires-Dist: matplotlib>=3.5; extra == 'all'
+Requires-Dist: pillow>=9; extra == 'all'
 Provides-Extra: dev
 Requires-Dist: build; extra == 'dev'
 Requires-Dist: hypothesis>=6.80; extra == 'dev'
 Requires-Dist: matplotlib>=3.5; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: pillow>=9; extra == 'dev'
 Requires-Dist: pip-audit; extra == 'dev'
+Requires-Dist: pycocotools>=2.0.7; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
@@ -50,6 +54,8 @@ Requires-Dist: statsmodels>=0.13; extra == 'dev'
 Requires-Dist: twine; extra == 'dev'
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.5; extra == 'plot'
+Provides-Extra: vision
+Requires-Dist: pillow>=9; extra == 'vision'
 Description-Content-Type: text/markdown
 
 # EvalSuite
@@ -61,10 +67,10 @@ Description-Content-Type: text/markdown
 
 **Unified, reproducible evaluation for machine learning and research.**
 
-EvalSuite brings classification, regression, clinical and statistical evaluation (with segmentation and
-object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
+EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
+evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.2.1).** Every item on the 0.1.0 and 0.2.0 roadmaps is implemented and verified.
+> **Status: stable (0.3.0).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -181,6 +187,45 @@ es.adjust_pvalues(p_values, method="hochberg")  # also holm, bonferroni, bh, by
 Every test returns a `TestResult` with the statistic, p-value and an effect size, computed with SciPy and
 checked against SciPy and statsmodels in the test suite.
 
+## Segmentation
+
+```python
+# label masks: one 2-D image, or images on the first axis (N, H, W) / (N, D, H, W), or a list of masks
+es.dice(y_true, y_pred)  # macro over classes, pixel counts summed over the dataset
+es.iou(y_true, y_pred, average=None)  # per class; classes absent from both masks are NaN, not 0
+es.miou(y_true, y_pred, ignore_index=255)
+es.dice(y_true, y_pred, aggregate="image")  # mean of per-image scores (medical imaging convention)
+es.boundary_iou(y_true, y_pred)  # Cheng et al. 2021
+es.hausdorff_distance(y_true, y_pred, percentile=95, spacing=(0.8, 0.8))  # HD95 in mm
+
+report = es.segmentation_report(y_true, y_pred, class_names={0: "background", 1: "liver"})
+print(report)  # mIoU, Dice, pixel accuracy, Boundary IoU, HD95, ASSD + per-class table
+es.plot.segmentation(image, y_true[0], y_pred[0])  # prediction fill, truth outline
+es.plot.per_class(report, metric="iou")  # per-class bars (also takes a detection report)
+```
+
+## Object detection
+
+```python
+y_true = [{"boxes": [[x1, y1, x2, y2], ...], "labels": [3, ...]}, ...]  # one dict per image
+y_pred = [{"boxes": [...], "labels": [...], "scores": [...]}, ...]
+
+report = es.detection_report(y_true, y_pred)  # the 12 COCO numbers + AP per class
+report["map"], report["map_50"], report["mar_100"]
+es.mean_average_precision(y_true, y_pred, iou_threshold=0.5)  # mAP@.50
+es.average_precision_detection(y_true, y_pred, interpolation="voc")  # per class, VOC-style
+es.box_iou(boxes_a, boxes_b, box_format="xywh")
+y_true, y_pred = es.from_coco("instances_val.json", "detections.json")
+es.plot.detection_pr(y_true, y_pred)
+```
+
+The COCO protocol (crowd regions, area ranges, max detections, 101-point interpolation) matches
+`pycocotools` to the last digit in the test suite.
+
+Models can be compared over the same images with intervals and paired tests, as for every other task:
+`es.compare(y_true, {"unet": masks_a, "deeplab": masks_b})` resamples images; for detection it compares
+mAP.
+
 ## Classification report
 
 ```python
@@ -225,6 +270,8 @@ evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
 evalsuite diagnostic predictions.csv --y-true label --y-pred pred   # sensitivity, LR+, DOR... with CIs
 evalsuite calibration predictions.csv --y-true label --y-prob prob  # slope, intercept, ECE, HL
 evalsuite plot decision predictions.csv --y-true label --y-prob prob -o dca.png
+evalsuite segmentation true_masks.npy pred_masks.npy --ignore-index 255 --plot per_class.png
+evalsuite detection instances_val.json detections.json --plot pr_curves.png
 evalsuite metrics --category clinical
 evalsuite info classification.mcc
 evalsuite benchmark --quick
@@ -251,6 +298,9 @@ Linux x86_64). Every result agrees with the reference to floating-point rounding
 | diagnostic report (7 CIs) | 1,000,000 | statsmodels | 16.6 | 4.6 | 0.28× |
 | Welch t-test | 1,000,000 | SciPy | 14.5 | 7.3 | 0.51× |
 | Hochberg correction | 1,000,000 | statsmodels | 75.4 | 81.9 | **1.1×** |
+| segmentation Dice and IoU per class | 1,000,000 px | scikit-learn | 40.2 | 282.0 | **7.0×** |
+| COCO detection evaluation (12 numbers) | 1,000 images | pycocotools | 967.4 | 980.6 | **1.0×** |
+| Hausdorff distance | 50 images | SciPy | 26.2 | 20.8 | 0.79× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows
@@ -279,6 +329,13 @@ Kruskal–Wallis, Friedman, Shapiro–Wilk, χ², Fisher's exact; effect sizes (
 delta, Cramér's V); multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini–Hochberg,
 Benjamini–Yekutieli).
 
+**Segmentation** (2-D and 3-D label masks; `ignore_index`; dataset or per-image aggregation): Dice, IoU,
+mIoU, pixel accuracy, mean pixel accuracy, Boundary IoU, Hausdorff distance and HD95, average symmetric
+surface distance (with pixel spacing), confusion matrix and a full report.
+
+**Object detection**: box IoU (xyxy, xywh, cxcywh), COCO mAP@[.50:.95], mAP@.50, mAP@.75, mAP and mAR by
+object size, AP per class with COCO or VOC interpolation, precision-recall curves, COCO file import.
+
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/README.md

@@ -7,10 +7,10 @@
 
 **Unified, reproducible evaluation for machine learning and research.**
 
-EvalSuite brings classification, regression, clinical and statistical evaluation (with segmentation and
-object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
+EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
+evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.2.1).** Every item on the 0.1.0 and 0.2.0 roadmaps is implemented and verified.
+> **Status: stable (0.3.0).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -127,6 +127,45 @@ es.adjust_pvalues(p_values, method="hochberg")  # also holm, bonferroni, bh, by
 Every test returns a `TestResult` with the statistic, p-value and an effect size, computed with SciPy and
 checked against SciPy and statsmodels in the test suite.
 
+## Segmentation
+
+```python
+# label masks: one 2-D image, or images on the first axis (N, H, W) / (N, D, H, W), or a list of masks
+es.dice(y_true, y_pred)  # macro over classes, pixel counts summed over the dataset
+es.iou(y_true, y_pred, average=None)  # per class; classes absent from both masks are NaN, not 0
+es.miou(y_true, y_pred, ignore_index=255)
+es.dice(y_true, y_pred, aggregate="image")  # mean of per-image scores (medical imaging convention)
+es.boundary_iou(y_true, y_pred)  # Cheng et al. 2021
+es.hausdorff_distance(y_true, y_pred, percentile=95, spacing=(0.8, 0.8))  # HD95 in mm
+
+report = es.segmentation_report(y_true, y_pred, class_names={0: "background", 1: "liver"})
+print(report)  # mIoU, Dice, pixel accuracy, Boundary IoU, HD95, ASSD + per-class table
+es.plot.segmentation(image, y_true[0], y_pred[0])  # prediction fill, truth outline
+es.plot.per_class(report, metric="iou")  # per-class bars (also takes a detection report)
+```
+
+## Object detection
+
+```python
+y_true = [{"boxes": [[x1, y1, x2, y2], ...], "labels": [3, ...]}, ...]  # one dict per image
+y_pred = [{"boxes": [...], "labels": [...], "scores": [...]}, ...]
+
+report = es.detection_report(y_true, y_pred)  # the 12 COCO numbers + AP per class
+report["map"], report["map_50"], report["mar_100"]
+es.mean_average_precision(y_true, y_pred, iou_threshold=0.5)  # mAP@.50
+es.average_precision_detection(y_true, y_pred, interpolation="voc")  # per class, VOC-style
+es.box_iou(boxes_a, boxes_b, box_format="xywh")
+y_true, y_pred = es.from_coco("instances_val.json", "detections.json")
+es.plot.detection_pr(y_true, y_pred)
+```
+
+The COCO protocol (crowd regions, area ranges, max detections, 101-point interpolation) matches
+`pycocotools` to the last digit in the test suite.
+
+Models can be compared over the same images with intervals and paired tests, as for every other task:
+`es.compare(y_true, {"unet": masks_a, "deeplab": masks_b})` resamples images; for detection it compares
+mAP.
+
 ## Classification report
 
 ```python
@@ -171,6 +210,8 @@ evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
 evalsuite diagnostic predictions.csv --y-true label --y-pred pred   # sensitivity, LR+, DOR... with CIs
 evalsuite calibration predictions.csv --y-true label --y-prob prob  # slope, intercept, ECE, HL
 evalsuite plot decision predictions.csv --y-true label --y-prob prob -o dca.png
+evalsuite segmentation true_masks.npy pred_masks.npy --ignore-index 255 --plot per_class.png
+evalsuite detection instances_val.json detections.json --plot pr_curves.png
 evalsuite metrics --category clinical
 evalsuite info classification.mcc
 evalsuite benchmark --quick
@@ -197,6 +238,9 @@ Linux x86_64). Every result agrees with the reference to floating-point rounding
 | diagnostic report (7 CIs) | 1,000,000 | statsmodels | 16.6 | 4.6 | 0.28× |
 | Welch t-test | 1,000,000 | SciPy | 14.5 | 7.3 | 0.51× |
 | Hochberg correction | 1,000,000 | statsmodels | 75.4 | 81.9 | **1.1×** |
+| segmentation Dice and IoU per class | 1,000,000 px | scikit-learn | 40.2 | 282.0 | **7.0×** |
+| COCO detection evaluation (12 numbers) | 1,000 images | pycocotools | 967.4 | 980.6 | **1.0×** |
+| Hausdorff distance | 50 images | SciPy | 26.2 | 20.8 | 0.79× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows
@@ -225,6 +269,13 @@ Kruskal–Wallis, Friedman, Shapiro–Wilk, χ², Fisher's exact; effect sizes (
 delta, Cramér's V); multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini–Hochberg,
 Benjamini–Yekutieli).
 
+**Segmentation** (2-D and 3-D label masks; `ignore_index`; dataset or per-image aggregation): Dice, IoU,
+mIoU, pixel accuracy, mean pixel accuracy, Boundary IoU, Hausdorff distance and HD95, average symmetric
+surface distance (with pixel spacing), confusion matrix and a full report.
+
+**Object detection**: box IoU (xyxy, xywh, cxcywh), COCO mAP@[.50:.95], mAP@.50, mAP@.75, mAP and mAR by
+object size, AP per class with COCO or VOC interpolation, precision-recall curves, COCO file import.
+
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/pyproject.toml

@@ -12,12 +12,13 @@ license-files = ["LICENSE"]
 requires-python = ">=3.9"
 authors = [{ name = "Manoj Kumar C S" }, { name = "Nikhil D Bharadwaj" }]
 maintainers = [{ name = "Manoj Kumar C S" }, { name = "Nikhil D Bharadwaj" }]
-keywords = ["evaluation", "metrics", "clinical", "diagnostic accuracy", "calibration", "statistical tests", "machine learning", "statistics", "classification", "regression", "reproducibility"]
+keywords = ["evaluation", "metrics", "segmentation", "object detection", "mAP", "clinical", "diagnostic accuracy", "calibration", "statistical tests", "machine learning", "statistics", "classification", "regression", "reproducibility"]
 classifiers = [
   "Development Status :: 5 - Production/Stable",
   "Intended Audience :: Science/Research",
   "Intended Audience :: Healthcare Industry",
   "Topic :: Scientific/Engineering :: Medical Science Apps.",
+  "Topic :: Scientific/Engineering :: Image Recognition",
   "Intended Audience :: Developers",
   "Operating System :: OS Independent",
   "Programming Language :: Python :: 3",
@@ -36,7 +37,8 @@ dependencies = ["numpy>=1.22", "scipy>=1.8", "pandas>=1.4"]
 
 [project.optional-dependencies]
 plot = ["matplotlib>=3.5"]
-all = ["matplotlib>=3.5"]
+vision = ["pillow>=9"]
+all = ["matplotlib>=3.5", "pillow>=9"]
 dev = [
   "pytest>=7",
   "pytest-cov>=4",
@@ -44,6 +46,8 @@ dev = [
   "scikit-learn>=1.2",
   "statsmodels>=0.13",
   "matplotlib>=3.5",
+  "pillow>=9",
+  "pycocotools>=2.0.7",
   "ruff>=0.6",
   "mypy>=1.10",
   "pandas-stubs",

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/src/evalsuite/__init__.py

@@ -5,7 +5,7 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import calibration, classification, clinical, plot, regression, stats
+from . import calibration, classification, clinical, plot, regression, stats, vision
 from .api import evaluate
 from .calibration import (
     CalibrationReport,
@@ -111,8 +111,49 @@ from .stats import (
     wilcoxon_test,
 )
 from .version import __version__
+from .vision import (
+    DetectionReport,
+    SegmentationReport,
+    average_precision_detection,
+    average_surface_distance,
+    boundary_iou,
+    box_iou,
+    detection_pr_curve,
+    detection_report,
+    dice,
+    from_coco,
+    hausdorff_distance,
+    iou,
+    mean_average_precision,
+    mean_pixel_accuracy,
+    miou,
+    per_image_scores,
+    pixel_accuracy,
+    segmentation_confusion,
+    segmentation_report,
+)
 
 __all__ = [
+    "SegmentationReport",
+    "segmentation_report",
+    "detection_pr_curve",
+    "vision",
+    "DetectionReport",
+    "average_precision_detection",
+    "average_surface_distance",
+    "boundary_iou",
+    "box_iou",
+    "detection_report",
+    "dice",
+    "from_coco",
+    "hausdorff_distance",
+    "iou",
+    "mean_average_precision",
+    "mean_pixel_accuracy",
+    "miou",
+    "per_image_scores",
+    "pixel_accuracy",
+    "segmentation_confusion",
     "CalibrationReport",
     "calibration_report",
     "calibration",

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/src/evalsuite/api.py

@@ -149,6 +149,16 @@ def evaluate(
     >>> round(r["accuracy"], 2)
     0.75
     """
+    if isinstance(y_true, (list, tuple)) and y_true and isinstance(y_true[0], dict):
+        raise UnsupportedTaskError(
+            "y_true looks like object detection annotations (one dict per image); use "
+            "evalsuite.detection_report(y_true, y_pred) or evalsuite.mean_average_precision(...)."
+        )
+    if np.ndim(y_true) >= 3:
+        raise UnsupportedTaskError(
+            "y_true has 3 or more dimensions, which looks like segmentation masks (images first); use "
+            "evalsuite.segmentation_report(y_true, y_pred) or evalsuite.dice / evalsuite.iou."
+        )
     task = task or _infer_task(y_true, y_prob)
     if task == "classification":
         return _evaluate_classification(

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/src/evalsuite/benchmarks.py

@@ -6,7 +6,8 @@ quantities on the same data, and the largest absolute difference between their r
 speed is never shown for numbers that disagree.
 
 References: scikit-learn for classification and regression (``suite="core"``); scikit-learn, statsmodels
-and SciPy for the v0.2.0 clinical, calibration and statistics functions (``suite="clinical"``). A case
+and SciPy for the v0.2.0 clinical, calibration and statistics functions (``suite="clinical"``);
+scikit-learn, SciPy and pycocotools for segmentation and detection (``suite="vision"``). A case
 whose reference library is not installed is timed for EvalSuite only.
 
     >>> from evalsuite.benchmarks import run_benchmarks
@@ -120,6 +121,145 @@ def _core_cases(n: int, rng: np.random.Generator) -> list[Case]:
     return [(name, ref, es_fn, sk_fn) for (name, ref, es_fn, _), sk_fn in zip(cases, sk)]
 
 
+def _vision_cases(n: int, rng: np.random.Generator) -> list[Case]:
+    """v0.3.0: segmentation overlap and surface distance (n = pixels) and COCO detection (n / 1000 images)."""
+    import evalsuite as es
+
+    side = 64
+    n_img = max(1, n // (side * side))
+    k = 5
+    yy, xx = np.ogrid[:side, :side]
+    true = np.zeros((n_img, side, side), dtype=np.int64)
+    for i in range(n_img):
+        for c in range(1, k):
+            cy, cx, r = rng.integers(8, side - 8), rng.integers(8, side - 8), rng.integers(4, 14)
+            true[i][(yy - cy) ** 2 + (xx - cx) ** 2 <= r * r] = c
+    pred = np.roll(true, 1, axis=2)
+    noise = rng.random(pred.shape) < 0.02
+    pred[noise] = rng.integers(0, k, int(noise.sum()))
+    labels = list(range(k))
+
+    def es_overlap() -> list[float]:
+        d = np.asarray(es.dice(true, pred, average=None).value)
+        j = np.asarray(es.iou(true, pred, average=None).value)
+        return [*d, *j]
+
+    n_hd = min(n_img, 50)
+
+    def es_hd() -> list[float]:
+        return [float(es.hausdorff_distance(true[i], pred[i], labels=[k - 1])) for i in range(n_hd)]
+
+    n_det = max(10, n // 1000)
+    y_true, y_pred = [], []
+    for _ in range(n_det):
+        m = int(rng.integers(1, 8))
+        xy = rng.uniform(0, 500, (m, 2))
+        wh = rng.uniform(8, 160, (m, 2))
+        boxes = np.column_stack([xy, xy + wh])
+        lab = rng.integers(1, 6, m)
+        y_true.append({"boxes": boxes, "labels": lab, "iscrowd": np.zeros(m, int)})
+        jitter = boxes + rng.normal(0, 4, boxes.shape)
+        jitter[:, 2:] = np.maximum(jitter[:, 2:], jitter[:, :2] + 1)
+        extra = rng.uniform(0, 500, (3, 2))
+        fp = np.column_stack([extra, extra + rng.uniform(10, 80, (3, 2))])
+        y_pred.append(
+            {
+                "boxes": np.vstack([jitter, fp]),
+                "labels": np.r_[lab, rng.integers(1, 6, 3)],
+                "scores": rng.random(m + 3),
+            }
+        )
+
+    def es_map() -> list[float]:
+        r = es.detection_report(y_true, y_pred)
+        return [r["map"], r["map_50"], r["map_75"], r["mar_100"]]
+
+    cases: list[Case] = [
+        ("segmentation: Dice and IoU per class (n = pixels)", "scikit-learn", es_overlap, None),
+        (f"segmentation: Hausdorff distance ({n_hd} image{'s' if n_hd != 1 else ''})", "SciPy", es_hd, None),
+        (f"detection: COCO evaluation ({n_det} images)", "pycocotools", es_map, None),
+    ]
+    refs: dict[str, Callable[[], Any]] = {}
+    try:
+        import sklearn.metrics as skm
+
+        def sk_overlap() -> list[float]:
+            ft, fp_ = true.ravel(), pred.ravel()
+            return [
+                *skm.f1_score(ft, fp_, labels=labels, average=None),
+                *skm.jaccard_score(ft, fp_, labels=labels, average=None),
+            ]
+
+        refs[cases[0][0]] = sk_overlap
+    except ImportError:
+        pass
+    from scipy import ndimage
+    from scipy.spatial.distance import directed_hausdorff
+
+    def scipy_hd() -> list[float]:
+        out = []
+        for i in range(n_hd):
+            pts = []
+            for m_ in (true[i] == k - 1, pred[i] == k - 1):
+                er = ndimage.binary_erosion(m_, structure=ndimage.generate_binary_structure(2, 1), border_value=0)
+                pts.append(np.argwhere(m_ & ~er).astype(float))
+            out.append(max(directed_hausdorff(pts[0], pts[1])[0], directed_hausdorff(pts[1], pts[0])[0]))
+        return out
+
+    refs[cases[1][0]] = scipy_hd
+    try:
+        import contextlib as _ctx
+        import io
+
+        from pycocotools.coco import COCO  # type: ignore[import-untyped]
+        from pycocotools.cocoeval import COCOeval  # type: ignore[import-untyped]
+
+        def coco_map() -> list[float]:
+            images, anns, dets, aid = [], [], [], 1
+            for i, (t, p) in enumerate(zip(y_true, y_pred)):
+                images.append({"id": i + 1})
+                for b, c in zip(t["boxes"], t["labels"]):
+                    w, h = b[2] - b[0], b[3] - b[1]
+                    anns.append(
+                        {
+                            "id": aid,
+                            "image_id": i + 1,
+                            "category_id": int(c),
+                            "bbox": [b[0], b[1], w, h],
+                            "area": w * h,
+                            "iscrowd": 0,
+                        }
+                    )
+                    aid += 1
+                for b, c, sc in zip(p["boxes"], p["labels"], p["scores"]):
+                    dets.append(
+                        {
+                            "image_id": i + 1,
+                            "category_id": int(c),
+                            "bbox": [b[0], b[1], b[2] - b[0], b[3] - b[1]],
+                            "score": float(sc),
+                        }
+                    )
+            with _ctx.redirect_stdout(io.StringIO()):
+                gt = COCO()
+                gt.dataset = {
+                    "images": images,
+                    "annotations": anns,
+                    "categories": [{"id": c} for c in range(1, 6)],
+                }
+                gt.createIndex()
+                ev = COCOeval(gt, gt.loadRes(dets), "bbox")
+                ev.evaluate()
+                ev.accumulate()
+                ev.summarize()
+            return [ev.stats[0], ev.stats[1], ev.stats[2], ev.stats[8]]
+
+        refs[cases[2][0]] = coco_map
+    except ImportError:
+        pass
+    return [(name, ref, es_fn, refs.get(name)) for name, ref, es_fn, _ in cases]
+
+
 def _clinical_cases(n: int, rng: np.random.Generator) -> list[Case]:
     """v0.2.0: diagnostic accuracy, calibration, decision curves and statistical tests."""
     import evalsuite as es
@@ -286,6 +426,7 @@ class BenchmarkResult:
             + (f" | scikit-learn {env['sklearn']}" if env.get("sklearn") else "")
             + (f" | statsmodels {env['statsmodels']}" if env.get("statsmodels") else "")
             + (f" | SciPy {env['scipy']}" if env.get("scipy") else "")
+            + (f" | pycocotools {env['pycocotools']}" if env.get("pycocotools") else "")
             + f" | {env['machine']} | fastest of {env['repeat']} runs"
         )
         note = "Speed-up > 1 means EvalSuite is faster. Max |difference| compares EvalSuite with the reference."
@@ -354,7 +495,8 @@ def run_benchmarks(
     """Time and memory for evaluation workloads at each size, against a reference implementation.
 
     ``suite``: ``"core"`` (classification and regression vs scikit-learn), ``"clinical"`` (v0.2.0 clinical,
-    calibration and statistics vs scikit-learn, statsmodels, SciPy) or ``"all"`` (default).
+    calibration and statistics vs scikit-learn, statsmodels, SciPy), ``"vision"`` (segmentation and COCO
+    detection vs scikit-learn, SciPy, pycocotools) or ``"all"`` (default).
     ``compare_sklearn=False`` times EvalSuite alone. Rows keep ``sklearn_ms``/``sklearn_peak_mb`` for rows
     whose reference is scikit-learn, for compatibility with 0.1.x.
     """
@@ -364,16 +506,24 @@ def run_benchmarks(
 
     if repeat < 1:
         raise ValueError("repeat must be at least 1.")
-    if suite not in ("all", "core", "clinical"):
-        raise ValueError("suite must be 'all', 'core' or 'clinical'.")
+    if suite not in ("all", "core", "clinical", "vision"):
+        raise ValueError("suite must be 'all', 'core', 'clinical' or 'vision'.")
     rng = np.random.default_rng(random_state)
     rows: list[dict[str, Any]] = []
-    versions: dict[str, Optional[str]] = {"sklearn": None, "statsmodels": None}
+    versions: dict[str, Optional[str]] = {"sklearn": None, "statsmodels": None, "pycocotools": None}
     if compare_sklearn:
         for mod in versions:
             with contextlib.suppress(ImportError):
-                versions[mod] = __import__(mod).__version__
-    builders = {"core": [_core_cases], "clinical": [_clinical_cases], "all": [_core_cases, _clinical_cases]}[suite]
+                __import__(mod)
+                from importlib.metadata import version as _dist_version
+
+                versions[mod] = _dist_version("scikit-learn" if mod == "sklearn" else mod)
+    builders = {
+        "core": [_core_cases],
+        "clinical": [_clinical_cases],
+        "vision": [_vision_cases],
+        "all": [_core_cases, _clinical_cases, _vision_cases],
+    }[suite]
     for n in sizes:
         for build in builders:
             for name, ref_name, es_fn, ref_fn in build(int(n), rng):
@@ -410,6 +560,7 @@ def run_benchmarks(
         "scipy": scipy.__version__ if suite != "core" else None,
         "sklearn": versions["sklearn"],
         "statsmodels": versions["statsmodels"] if suite != "core" else None,
+        "pycocotools": versions["pycocotools"] if suite in ("all", "vision") else None,
         "machine": f"{platform.system()} {platform.machine()}",
         "repeat": repeat,
         "suite": suite,

{evalsuite_python-0.2.1 → evalsuite_python-0.3.0}/src/evalsuite/calibration.py

@@ -128,7 +128,7 @@ def logistic_fit(
     beta = np.zeros(design.shape[1])
     for _ in range(max_iter):
         eta = design @ beta + off
-        mu = 1 / (1 + np.exp(-eta))
+        mu = 1 / (1 + np.exp(-np.clip(eta, -700, 700)))
         grad = design.T @ (w * (y - mu))
         hess = (design * (w * mu * (1 - mu))[:, None]).T @ design
         try: