evalsuite-python 0.1.0a2__tar.gz → 0.1.0b1__tar.gz

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/CHANGELOG.md

@@ -6,6 +6,29 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.1.0b1]
+
+Feature-complete for 0.1.0.
+
+### Added
+
+- Plots (`es.plot`, optional `[plot]` extra): ROC, precision-recall, calibration, confusion matrix, residuals /
+  predicted-vs-true, and model-comparison forest plots. Values come from EvalSuite's metrics; several models
+  get distinct colours and line styles. Importing `evalsuite` never imports matplotlib.
+- Reporting: `classification_report` (per-class precision, recall, F1, specificity, support; accuracy, micro,
+  macro and weighted averages; matches scikit-learn); `to_html()` and `to_csv()` on every result; `save(path)`
+  choosing the format from the extension (.json .csv .md .tex .html .txt).
+- `calibration_curve` and `expected_calibration_error` (matches scikit-learn's calibration curve).
+- Command line: `evalsuite evaluate | report | compare | plot | metrics | info | benchmark`, and
+  `python -m evalsuite`. Reads CSV, TSV, Parquet and JSON; clear one-line errors with exit code 2.
+- Benchmarks (`evalsuite.benchmarks.run_benchmarks`, `evalsuite benchmark`): time and peak memory against
+  scikit-learn, with a check that both libraries return the same numbers. Results in `BENCHMARKS.md`.
+
+### Changed
+
+- Regression `evaluate()` validates inputs once for all metrics and uses a faster unweighted mean (2× faster
+  on large arrays).
+
 ## [0.1.0a2]
 
 ### Added

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.1.0a2
+Version: 0.1.0b1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -12,7 +12,7 @@ Maintainer: Manoj Kumar C S
 License-Expression: MIT
 License-File: LICENSE
 Keywords: classification,evaluation,machine learning,metrics,regression,reproducibility,statistics
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
@@ -36,6 +36,7 @@ Requires-Dist: matplotlib>=3.5; extra == 'all'
 Provides-Extra: dev
 Requires-Dist: build; extra == 'dev'
 Requires-Dist: hypothesis>=6.80; extra == 'dev'
+Requires-Dist: matplotlib>=3.5; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: pandas-stubs; extra == 'dev'
 Requires-Dist: pip-audit; extra == 'dev'
@@ -61,7 +62,7 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: in development (v0.1.0 in progress).** The API may change before 0.1.0.
+> **Status: beta (0.1.0b1).** Feature-complete for 0.1.0; the stable release follows once this beta is verified.
 
 ## Installation
 
@@ -133,12 +134,67 @@ Every model is evaluated on the same bootstrap resamples, so differences are pai
 with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
 p-values are adjusted for multiple comparisons (Holm by default).
 
+## Classification report
+
+```python
+report = es.classification_report(y_true, y_pred)
+print(report)  # per-class precision, recall, F1, specificity, support + averages
+report.save("report.html")  # also .csv .md .tex .json .txt
+```
+
+## Plots
+
+```bash
+pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
+```
+
+```python
+es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
+es.plot.pr(y_true, prob)  # AP and the prevalence line
+es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
+es.plot.confusion_matrix(y_true, y_pred, normalize="true")
+es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
+es.plot.comparison(es.compare(...))  # forest plot with CIs
+```
+
+Each function returns a matplotlib `Axes` (pass `ax=` to draw into your own figure). The numbers shown are
+computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours
+*and* line styles, so figures stay readable in greyscale print.
+
+## Exports
+
+Every result (`evaluate`, `classification_report`, `compare`, single metrics) exports to `summary()`,
+`to_json()`, `to_csv()`, `to_dataframe()`, `to_markdown()`, `to_latex()` and `to_html()`, and `save(path)` picks
+the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.
+
+## Command line
+
+```bash
+evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
+evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
+evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
+    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
+evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
+evalsuite metrics --category classification
+evalsuite info classification.mcc
+evalsuite benchmark --quick
+```
+
+Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` or the `-o` extension
+(text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.
+
+## Performance
+
+`evaluate()` validates inputs once and computes the confusion matrix once for all metrics: about 10× faster
+than the equivalent separate scikit-learn calls, with identical results. See [BENCHMARKS.md](BENCHMARKS.md).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;
 sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard,
 MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary,
-one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy.
+one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
+calibration curve and expected calibration error.
 
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/README.md

@@ -10,7 +10,7 @@
 EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: in development (v0.1.0 in progress).** The API may change before 0.1.0.
+> **Status: beta (0.1.0b1).** Feature-complete for 0.1.0; the stable release follows once this beta is verified.
 
 ## Installation
 
@@ -82,12 +82,67 @@ Every model is evaluated on the same bootstrap resamples, so differences are pai
 with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
 p-values are adjusted for multiple comparisons (Holm by default).
 
+## Classification report
+
+```python
+report = es.classification_report(y_true, y_pred)
+print(report)  # per-class precision, recall, F1, specificity, support + averages
+report.save("report.html")  # also .csv .md .tex .json .txt
+```
+
+## Plots
+
+```bash
+pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
+```
+
+```python
+es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
+es.plot.pr(y_true, prob)  # AP and the prevalence line
+es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
+es.plot.confusion_matrix(y_true, y_pred, normalize="true")
+es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
+es.plot.comparison(es.compare(...))  # forest plot with CIs
+```
+
+Each function returns a matplotlib `Axes` (pass `ax=` to draw into your own figure). The numbers shown are
+computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours
+*and* line styles, so figures stay readable in greyscale print.
+
+## Exports
+
+Every result (`evaluate`, `classification_report`, `compare`, single metrics) exports to `summary()`,
+`to_json()`, `to_csv()`, `to_dataframe()`, `to_markdown()`, `to_latex()` and `to_html()`, and `save(path)` picks
+the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.
+
+## Command line
+
+```bash
+evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
+evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
+evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
+    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
+evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
+evalsuite metrics --category classification
+evalsuite info classification.mcc
+evalsuite benchmark --quick
+```
+
+Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` or the `-o` extension
+(text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.
+
+## Performance
+
+`evaluate()` validates inputs once and computes the confusion matrix once for all metrics: about 10× faster
+than the equivalent separate scikit-learn calls, with identical results. See [BENCHMARKS.md](BENCHMARKS.md).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;
 sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard,
 MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary,
-one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy.
+one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
+calibration curve and expected calibration error.
 
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/pyproject.toml

@@ -14,7 +14,7 @@ authors = [{ name = "Manoj Kumar C S" }]
 maintainers = [{ name = "Manoj Kumar C S" }]
 keywords = ["evaluation", "metrics", "machine learning", "statistics", "classification", "regression", "reproducibility"]
 classifiers = [
-  "Development Status :: 3 - Alpha",
+  "Development Status :: 4 - Beta",
   "Intended Audience :: Science/Research",
   "Intended Audience :: Developers",
   "Operating System :: OS Independent",
@@ -41,6 +41,7 @@ dev = [
   "hypothesis>=6.80",
   "scikit-learn>=1.2",
   "statsmodels>=0.13",
+  "matplotlib>=3.5",
   "ruff>=0.6",
   "mypy>=1.10",
   "pandas-stubs",
@@ -56,7 +57,8 @@ Source = "https://github.com/mkcs28/evalsuite-python"
 Issues = "https://github.com/mkcs28/evalsuite-python/issues"
 Changelog = "https://github.com/mkcs28/evalsuite-python/blob/main/CHANGELOG.md"
 
-# [project.scripts] evalsuite = "evalsuite.cli.main:main" is added together with the CLI itself.
+[project.scripts]
+evalsuite = "evalsuite.cli.main:main"
 
 [tool.hatch.version]
 path = "src/evalsuite/version.py"

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/src/evalsuite/__init__.py

@@ -5,15 +5,17 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import classification, regression, stats
+from . import classification, plot, regression, stats
 from .api import evaluate
 from .classification import (
     accuracy,
     average_precision,
     balanced_accuracy,
     brier_score,
+    calibration_curve,
     cohen_kappa,
     confusion_matrix,
+    expected_calibration_error,
     f1,
     fbeta,
     hamming_loss,
@@ -59,6 +61,7 @@ from .regression import (
     rse,
     smape,
 )
+from .reporting import ClassificationReport, classification_report
 from .stats import (
     ComparisonResult,
     ConfidenceInterval,
@@ -79,6 +82,11 @@ from .stats import (
 from .version import __version__
 
 __all__ = [
+    "plot",
+    "expected_calibration_error",
+    "calibration_curve",
+    "classification_report",
+    "ClassificationReport",
     "stats",
     "roc_auc_ci",
     "proportion_ci",

evalsuite_python-0.1.0b1/src/evalsuite/__main__.py

@@ -0,0 +1,5 @@
+"""``python -m evalsuite`` runs the command-line interface."""
+
+from .cli.main import main
+
+raise SystemExit(main())

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/src/evalsuite/api.py

@@ -232,6 +232,28 @@ def _evaluate_regression(
                 f"Unknown regression metric(s): {', '.join(unknown)}. Available: {', '.join(_REGRESSION)}."
             )
     out: dict[str, MetricResult] = {}
+    shared = reg._Inputs(y_true, y_pred, sample_weight)  # validate once for every metric
+    token = reg._SHARED.set((id(y_true), id(y_pred), id(sample_weight), shared))
+    try:
+        _regression_loop(names, out, y_true, y_pred, sample_weight)
+    finally:
+        reg._SHARED.reset(token)
+    return EvaluationResult(
+        task="regression",
+        metrics=out,
+        n_samples=yt.shape[0],
+        target_type="continuous",
+        metadata=_metadata(weighted=sample_weight is not None, outputs=yt.shape[1] if multi else 1),
+    )
+
+
+def _regression_loop(
+    names: list[str],
+    out: dict[str, MetricResult],
+    y_true: ArrayLike,
+    y_pred: ArrayLike,
+    sample_weight: Optional[ArrayLike],
+) -> None:
     for name in names:
         fn = _REGRESSION[name]
         if name in _NO_WEIGHTS:
@@ -240,10 +262,3 @@ def _evaluate_regression(
             out[name] = fn(y_true, y_pred)
         else:
             out[name] = fn(y_true, y_pred, sample_weight=sample_weight)
-    return EvaluationResult(
-        task="regression",
-        metrics=out,
-        n_samples=yt.shape[0],
-        target_type="continuous",
-        metadata=_metadata(weighted=sample_weight is not None, outputs=yt.shape[1] if multi else 1),
-    )

evalsuite_python-0.1.0b1/src/evalsuite/benchmarks.py

@@ -0,0 +1,272 @@
+"""Speed and memory benchmarks, against scikit-learn when it is installed.
+
+Each case times the fastest of ``repeat`` runs (after one warm-up) and measures peak traced memory with
+``tracemalloc`` (NumPy reports its allocations to it). Both libraries compute the same metrics on the same
+data, and the largest absolute difference between their results is reported, so speed is never shown for
+numbers that disagree.
+
+    >>> from evalsuite.benchmarks import run_benchmarks
+    >>> print(run_benchmarks(sizes=(10_000,), repeat=3))  # doctest: +SKIP
+"""
+
+from __future__ import annotations
+
+import json
+import platform
+import time
+import tracemalloc
+import warnings
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass, field
+from types import MappingProxyType
+from typing import TYPE_CHECKING, Any, Optional, cast
+
+import numpy as np
+
+from .core.result import _json_safe, _latex_escape, _latex_table
+
+if TYPE_CHECKING:
+    import pandas as pd
+
+__all__ = ["BenchmarkResult", "run_benchmarks"]
+
+_HEADER = (
+    "case",
+    "n",
+    "evalsuite_ms",
+    "sklearn_ms",
+    "speedup",
+    "evalsuite_peak_mb",
+    "sklearn_peak_mb",
+    "max_abs_diff",
+)
+
+
+def _measure(fn: Callable[[], Any], repeat: int) -> tuple[float, float, Any]:
+    """(fastest seconds, peak MiB, result)."""
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        result = fn()  # warm-up
+        best = float("inf")
+        for _ in range(repeat):
+            t0 = time.perf_counter()
+            fn()
+            best = min(best, time.perf_counter() - t0)
+        tracemalloc.start()
+        try:
+            fn()
+            _, peak = tracemalloc.get_traced_memory()
+        finally:
+            tracemalloc.stop()
+    return best, peak / 2**20, result
+
+
+def _cases(n: int, rng: np.random.Generator) -> list[tuple[str, Callable[[], Any], Optional[Callable[[], Any]]]]:
+    import evalsuite as es
+
+    y = rng.integers(0, 2, n)
+    p = np.where(rng.random(n) < 0.8, y, 1 - y)
+    prob = 1 / (1 + np.exp(-(2.0 * (y - 0.5) + rng.normal(0, 1, n))))
+    yk = rng.integers(0, 10, n)
+    pk = np.where(rng.random(n) < 0.7, yk, rng.integers(0, 10, n))
+    yr = rng.normal(10, 3, n)
+    pr = yr + rng.normal(0, 1, n)
+    names = ["accuracy", "balanced_accuracy", "precision", "recall", "f1", "specificity", "mcc", "cohen_kappa"]
+
+    def es_binary() -> list[float]:
+        r = es.evaluate(y, p, metrics=names)
+        return [float(r[m]) for m in names]
+
+    def es_reg() -> list[float]:
+        r = es.evaluate(yr, pr, metrics=["mae", "mse", "rmse", "r2"])
+        return [float(r[m]) for m in ("mae", "mse", "rmse", "r2")]
+
+    cases: list[tuple[str, Callable[[], Any], Optional[Callable[[], Any]]]] = [
+        ("binary: 8 label metrics via evaluate()", es_binary, None),
+        ("10 classes: macro F1", lambda: [float(es.f1(yk, pk, average="macro"))], None),
+        ("binary: ROC AUC", lambda: [float(es.roc_auc(y, prob))], None),
+        ("regression: MAE, MSE, RMSE, R² via evaluate()", es_reg, None),
+    ]
+    try:
+        import sklearn.metrics as skm  # type: ignore[import-untyped]
+    except ImportError:
+        return cases
+
+    def sk_binary() -> list[float]:
+        return [
+            skm.accuracy_score(y, p),
+            skm.balanced_accuracy_score(y, p),
+            skm.precision_score(y, p),
+            skm.recall_score(y, p),
+            skm.f1_score(y, p),
+            skm.recall_score(y, p, pos_label=0),
+            skm.matthews_corrcoef(y, p),
+            skm.cohen_kappa_score(y, p),
+        ]
+
+    def sk_reg() -> list[float]:
+        mse = skm.mean_squared_error(yr, pr)
+        return [skm.mean_absolute_error(yr, pr), mse, float(np.sqrt(mse)), skm.r2_score(yr, pr)]
+
+    sk = [sk_binary, lambda: [skm.f1_score(yk, pk, average="macro")], lambda: [skm.roc_auc_score(y, prob)], sk_reg]
+    return [(name, es_fn, sk_fn) for (name, es_fn, _), sk_fn in zip(cases, sk)]
+
+
+@dataclass(frozen=True, eq=False)
+class BenchmarkResult:
+    """Rows of timings (milliseconds), peak memory (MiB) and agreement, plus the environment they ran in."""
+
+    rows: tuple[Any, ...]
+    environment: Any = field(default_factory=dict)
+
+    def __post_init__(self) -> None:
+        object.__setattr__(self, "rows", tuple(MappingProxyType(dict(r)) for r in self.rows))
+        object.__setattr__(self, "environment", MappingProxyType(dict(self.environment)))
+
+    def _cells(self, digits: int) -> list[list[str]]:
+        def f(v: Any, d: int = digits) -> str:
+            return "–" if v is None else f"{v:.{d}f}"
+
+        return [
+            [
+                r["case"],
+                f"{r['n']:,}",
+                f(r["evalsuite_ms"]),
+                f(r["sklearn_ms"]),
+                "–" if r["speedup"] is None else f"{r['speedup']:.2f}×",
+                f(r["evalsuite_peak_mb"], 2),
+                f(r["sklearn_peak_mb"], 2),
+                "–" if r["max_abs_diff"] is None else f"{r['max_abs_diff']:.1e}",
+            ]
+            for r in self.rows
+        ]
+
+    _TITLES = (
+        "Case",
+        "n",
+        "EvalSuite (ms)",
+        "scikit-learn (ms)",
+        "Speed-up",
+        "EvalSuite peak (MiB)",
+        "scikit-learn peak (MiB)",
+        "Max |difference|",
+    )
+
+    def summary(self, *, digits: int = 3) -> str:
+        cells = self._cells(digits)
+        widths = [max(len(t), *(len(c[i]) for c in cells)) for i, t in enumerate(self._TITLES)]
+
+        def line(c: Sequence[str]) -> str:
+            return "  ".join(x.ljust(widths[i]) if i == 0 else x.rjust(widths[i]) for i, x in enumerate(c))
+
+        env = self.environment
+        head = (
+            f"EvalSuite {env['evalsuite']} benchmarks | Python {env['python']} | NumPy {env['numpy']}"
+            + (f" | scikit-learn {env['sklearn']}" if env.get("sklearn") else "")
+            + f" | {env['machine']} | fastest of {env['repeat']} runs"
+        )
+        note = "Speed-up > 1 means EvalSuite is faster. Max |difference| compares the two libraries' results."
+        return "\n".join([head, "", line(self._TITLES), *(line(c) for c in cells), "", note])
+
+    def __repr__(self) -> str:
+        return self.summary()
+
+    def to_dict(self) -> dict[str, Any]:
+        return cast(
+            "dict[str, Any]",
+            _json_safe({"environment": dict(self.environment), "rows": [dict(r) for r in self.rows]}),
+        )
+
+    def to_json(self, *, indent: Optional[int] = 2) -> str:
+        return json.dumps(self.to_dict(), indent=indent, allow_nan=False)
+
+    def to_dataframe(self) -> pd.DataFrame:
+        import pandas as pd
+
+        frame: pd.DataFrame = pd.DataFrame([dict(r) for r in self.rows])
+        return frame
+
+    def to_csv(self, path: Optional[str] = None) -> str:
+        from .core.export import csv_text
+
+        rows = [[r[h] if r[h] is not None else float("nan") for h in _HEADER] for r in self.rows]
+        text = csv_text(list(_HEADER), rows)
+        if path is not None:
+            with open(path, "w", encoding="utf-8", newline="") as fh:
+                fh.write(text)
+        return text
+
+    def to_markdown(self, *, digits: int = 3) -> str:
+        lines = ["| " + " | ".join(self._TITLES) + " |", "| --- |" + " ---: |" * (len(self._TITLES) - 1)]
+        return "\n".join(lines + ["| " + " | ".join(c) + " |" for c in self._cells(digits)])
+
+    def to_latex(self, *, digits: int = 3, caption: Optional[str] = None, label: Optional[str] = None) -> str:
+        return _latex_table(
+            [_latex_escape(t) for t in self._TITLES],
+            [[_latex_escape(x) for x in c] for c in self._cells(digits)],
+            caption=caption or "EvalSuite benchmarks.",
+            label=label,
+        )
+
+    def to_html(self, *, digits: int = 3, full: bool = False) -> str:
+        from .core.export import html_document, html_table
+
+        table = html_table(list(self._TITLES), self._cells(digits), caption="Benchmarks")
+        env = self.environment
+        meta = f"Python {env['python']}, NumPy {env['numpy']}, {env['machine']}"
+        return html_document("EvalSuite benchmarks", table, meta) if full else table
+
+
+def run_benchmarks(
+    sizes: Sequence[int] = (1_000, 100_000, 1_000_000),
+    *,
+    repeat: int = 5,
+    compare_sklearn: bool = True,
+    random_state: Optional[int] = 0,
+) -> BenchmarkResult:
+    """Time and memory for common evaluation workloads at each size, against scikit-learn if installed."""
+    import evalsuite as es
+
+    if repeat < 1:
+        raise ValueError("repeat must be at least 1.")
+    rng = np.random.default_rng(random_state)
+    rows: list[dict[str, Any]] = []
+    sk_version: Optional[str] = None
+    if compare_sklearn:
+        try:
+            import sklearn
+
+            sk_version = sklearn.__version__
+        except ImportError:
+            compare_sklearn = False
+    for n in sizes:
+        for name, es_fn, sk_fn in _cases(int(n), rng):
+            es_t, es_mem, es_val = _measure(es_fn, repeat)
+            row: dict[str, Any] = {
+                "case": name,
+                "n": int(n),
+                "evalsuite_ms": es_t * 1000,
+                "evalsuite_peak_mb": es_mem,
+                "sklearn_ms": None,
+                "sklearn_peak_mb": None,
+                "speedup": None,
+                "max_abs_diff": None,
+            }
+            if compare_sklearn and sk_fn is not None:
+                sk_t, sk_mem, sk_val = _measure(sk_fn, repeat)
+                row.update(
+                    sklearn_ms=sk_t * 1000,
+                    sklearn_peak_mb=sk_mem,
+                    speedup=sk_t / es_t if es_t else None,
+                    max_abs_diff=float(np.max(np.abs(np.asarray(es_val, float) - np.asarray(sk_val, float)))),
+                )
+            rows.append(row)
+    env = {
+        "evalsuite": es.__version__,
+        "python": platform.python_version(),
+        "numpy": np.__version__,
+        "sklearn": sk_version,
+        "machine": f"{platform.system()} {platform.machine()}",
+        "repeat": repeat,
+    }
+    return BenchmarkResult(tuple(rows), env)

{evalsuite_python-0.1.0a2 → evalsuite_python-0.1.0b1}/src/evalsuite/classification/__init__.py

@@ -5,8 +5,10 @@ from .metrics import (
     average_precision,
     balanced_accuracy,
     brier_score,
+    calibration_curve,
     cohen_kappa,
     confusion_matrix,
+    expected_calibration_error,
     f1,
     fbeta,
     hamming_loss,
@@ -28,8 +30,10 @@ __all__ = [
     "average_precision",
     "balanced_accuracy",
     "brier_score",
+    "calibration_curve",
     "cohen_kappa",
     "confusion_matrix",
+    "expected_calibration_error",
     "f1",
     "fbeta",
     "hamming_loss",