evalsuite-python 0.1.0a1__tar.gz → 0.1.0b1__tar.gz

evalsuite_python-0.1.0b1/CHANGELOG.md

@@ -0,0 +1,66 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0b1]
+
+Feature-complete for 0.1.0.
+
+### Added
+
+- Plots (`es.plot`, optional `[plot]` extra): ROC, precision-recall, calibration, confusion matrix, residuals /
+  predicted-vs-true, and model-comparison forest plots. Values come from EvalSuite's metrics; several models
+  get distinct colours and line styles. Importing `evalsuite` never imports matplotlib.
+- Reporting: `classification_report` (per-class precision, recall, F1, specificity, support; accuracy, micro,
+  macro and weighted averages; matches scikit-learn); `to_html()` and `to_csv()` on every result; `save(path)`
+  choosing the format from the extension (.json .csv .md .tex .html .txt).
+- `calibration_curve` and `expected_calibration_error` (matches scikit-learn's calibration curve).
+- Command line: `evalsuite evaluate | report | compare | plot | metrics | info | benchmark`, and
+  `python -m evalsuite`. Reads CSV, TSV, Parquet and JSON; clear one-line errors with exit code 2.
+- Benchmarks (`evalsuite.benchmarks.run_benchmarks`, `evalsuite benchmark`): time and peak memory against
+  scikit-learn, with a check that both libraries return the same numbers. Results in `BENCHMARKS.md`.
+
+### Changed
+
+- Regression `evaluate()` validates inputs once for all metrics and uses a faster unweighted mean (2× faster
+  on large arrays).
+
+## [0.1.0a2]
+
+### Added
+
+- Model comparison (`es.compare`): per-model confidence intervals from paired bootstrap resamples, pairwise
+  tests (McNemar for accuracy, DeLong for binary ROC AUC, paired bootstrap otherwise), multiple-comparison
+  correction, best model per metric, and summary/pandas/Markdown/LaTeX/JSON export.
+- Confidence intervals: `bootstrap_ci` (percentile, basic, BCa; stratified and reproducible),
+  `proportion_ci` and `accuracy_ci` (Wilson, Clopper-Pearson, normal), `roc_auc_ci` (DeLong).
+- Paired tests: `mcnemar_test`, `delong_test`, `paired_bootstrap_test`.
+- Effect sizes: `cohens_d` (independent and paired), `hedges_g`, `cliffs_delta`; `adjust_pvalues`
+  (Holm, Bonferroni, Benjamini-Hochberg, Benjamini-Yekutieli).
+- Reference tests against statsmodels and SciPy, brute-force DeLong checks and coverage simulations.
+- Python 3.14 support and CI.
+
+### Fixed
+
+- 0.1.0a1 installed an `evalsuite` console command although the CLI is not implemented yet, so the command
+  failed with `ModuleNotFoundError`. The entry point is removed until the CLI ships.
+
+## [0.1.0a1]
+
+First alpha, published to reserve the name and test the release pipeline. Distribution name
+`evalsuite-python` (`pip install --pre evalsuite-python`), imported as `evalsuite`.
+
+### Added
+
+- Core: input validation with actionable errors, exception hierarchy, immutable result objects with
+  JSON/pandas/Markdown/LaTeX export, metric registry (`list_metrics`, `metric_info`), and an evaluation
+  context that computes the confusion matrix once per evaluation.
+- Classification metrics for binary, multiclass and multilabel targets with all averaging modes and
+  sample weights.
+- Regression metrics for single- and multi-output targets with sample weights.
+- `evaluate()` high-level API with task inference and default metric sets.
+- Reference tests against scikit-learn and property-based tests; Python 3.9 to 3.13.

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0b1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.1.0a1
+Version: 0.1.0b1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -12,7 +12,7 @@ Maintainer: Manoj Kumar C S
 License-Expression: MIT
 License-File: LICENSE
 Keywords: classification,evaluation,machine learning,metrics,regression,reproducibility,statistics
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
@@ -23,6 +23,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Typing :: Typed
@@ -35,6 +36,7 @@ Requires-Dist: matplotlib>=3.5; extra == 'all'
 Provides-Extra: dev
 Requires-Dist: build; extra == 'dev'
 Requires-Dist: hypothesis>=6.80; extra == 'dev'
+Requires-Dist: matplotlib>=3.5; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: pandas-stubs; extra == 'dev'
 Requires-Dist: pip-audit; extra == 'dev'
@@ -42,6 +44,7 @@ Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
 Requires-Dist: scikit-learn>=1.2; extra == 'dev'
+Requires-Dist: statsmodels>=0.13; extra == 'dev'
 Requires-Dist: twine; extra == 'dev'
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.5; extra == 'plot'
@@ -59,7 +62,7 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: in development (v0.1.0 in progress).** The API may change before 0.1.0.
+> **Status: beta (0.1.0b1).** Feature-complete for 0.1.0; the stable release follows once this beta is verified.
 
 ## Installation
 
@@ -109,12 +112,89 @@ es.metric_info("classification.mcc").formula  # documentation
 es.list_metrics("regression")
 ```
 
+## Comparing models
+
+```python
+result = es.compare(
+    y_true,
+    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
+    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
+    random_state=0,
+)
+print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
+result.to_latex(label="tab:models")
+
+es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
+es.accuracy_ci(y_true, y_pred)  # Wilson interval
+es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
+es.mcnemar_test(y_true, pred_a, pred_b)
+```
+
+Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared
+with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
+p-values are adjusted for multiple comparisons (Holm by default).
+
+## Classification report
+
+```python
+report = es.classification_report(y_true, y_pred)
+print(report)  # per-class precision, recall, F1, specificity, support + averages
+report.save("report.html")  # also .csv .md .tex .json .txt
+```
+
+## Plots
+
+```bash
+pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
+```
+
+```python
+es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
+es.plot.pr(y_true, prob)  # AP and the prevalence line
+es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
+es.plot.confusion_matrix(y_true, y_pred, normalize="true")
+es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
+es.plot.comparison(es.compare(...))  # forest plot with CIs
+```
+
+Each function returns a matplotlib `Axes` (pass `ax=` to draw into your own figure). The numbers shown are
+computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours
+*and* line styles, so figures stay readable in greyscale print.
+
+## Exports
+
+Every result (`evaluate`, `classification_report`, `compare`, single metrics) exports to `summary()`,
+`to_json()`, `to_csv()`, `to_dataframe()`, `to_markdown()`, `to_latex()` and `to_html()`, and `save(path)` picks
+the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.
+
+## Command line
+
+```bash
+evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
+evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
+evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
+    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
+evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
+evalsuite metrics --category classification
+evalsuite info classification.mcc
+evalsuite benchmark --quick
+```
+
+Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` or the `-o` extension
+(text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.
+
+## Performance
+
+`evaluate()` validates inputs once and computes the confusion matrix once for all metrics: about 10× faster
+than the equivalent separate scikit-learn calls, with identical results. See [BENCHMARKS.md](BENCHMARKS.md).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;
 sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard,
 MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary,
-one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy.
+one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
+calibration curve and expected calibration error.
 
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0b1}/README.md

@@ -10,7 +10,7 @@
 EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: in development (v0.1.0 in progress).** The API may change before 0.1.0.
+> **Status: beta (0.1.0b1).** Feature-complete for 0.1.0; the stable release follows once this beta is verified.
 
 ## Installation
 
@@ -60,12 +60,89 @@ es.metric_info("classification.mcc").formula  # documentation
 es.list_metrics("regression")
 ```
 
+## Comparing models
+
+```python
+result = es.compare(
+    y_true,
+    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
+    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
+    random_state=0,
+)
+print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
+result.to_latex(label="tab:models")
+
+es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
+es.accuracy_ci(y_true, y_pred)  # Wilson interval
+es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
+es.mcnemar_test(y_true, pred_a, pred_b)
+```
+
+Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared
+with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
+p-values are adjusted for multiple comparisons (Holm by default).
+
+## Classification report
+
+```python
+report = es.classification_report(y_true, y_pred)
+print(report)  # per-class precision, recall, F1, specificity, support + averages
+report.save("report.html")  # also .csv .md .tex .json .txt
+```
+
+## Plots
+
+```bash
+pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
+```
+
+```python
+es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
+es.plot.pr(y_true, prob)  # AP and the prevalence line
+es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
+es.plot.confusion_matrix(y_true, y_pred, normalize="true")
+es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
+es.plot.comparison(es.compare(...))  # forest plot with CIs
+```
+
+Each function returns a matplotlib `Axes` (pass `ax=` to draw into your own figure). The numbers shown are
+computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours
+*and* line styles, so figures stay readable in greyscale print.
+
+## Exports
+
+Every result (`evaluate`, `classification_report`, `compare`, single metrics) exports to `summary()`,
+`to_json()`, `to_csv()`, `to_dataframe()`, `to_markdown()`, `to_latex()` and `to_html()`, and `save(path)` picks
+the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.
+
+## Command line
+
+```bash
+evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
+evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
+evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
+    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
+evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
+evalsuite metrics --category classification
+evalsuite info classification.mcc
+evalsuite benchmark --quick
+```
+
+Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` or the `-o` extension
+(text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.
+
+## Performance
+
+`evaluate()` validates inputs once and computes the confusion matrix once for all metrics: about 10× faster
+than the equivalent separate scikit-learn calls, with identical results. See [BENCHMARKS.md](BENCHMARKS.md).
+
 ## Metrics in this release
 
 **Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;
 sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard,
 MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary,
-one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy.
+one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
+calibration curve and expected calibration error.
 
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0b1}/pyproject.toml

@@ -14,7 +14,7 @@ authors = [{ name = "Manoj Kumar C S" }]
 maintainers = [{ name = "Manoj Kumar C S" }]
 keywords = ["evaluation", "metrics", "machine learning", "statistics", "classification", "regression", "reproducibility"]
 classifiers = [
-  "Development Status :: 3 - Alpha",
+  "Development Status :: 4 - Beta",
   "Intended Audience :: Science/Research",
   "Intended Audience :: Developers",
   "Operating System :: OS Independent",
@@ -25,6 +25,7 @@ classifiers = [
   "Programming Language :: Python :: 3.11",
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
   "Topic :: Scientific/Engineering",
   "Topic :: Scientific/Engineering :: Artificial Intelligence",
   "Typing :: Typed",
@@ -39,6 +40,8 @@ dev = [
   "pytest-cov>=4",
   "hypothesis>=6.80",
   "scikit-learn>=1.2",
+  "statsmodels>=0.13",
+  "matplotlib>=3.5",
   "ruff>=0.6",
   "mypy>=1.10",
   "pandas-stubs",

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0b1}/src/evalsuite/__init__.py

@@ -5,15 +5,17 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import classification, regression
+from . import classification, plot, regression, stats
 from .api import evaluate
 from .classification import (
     accuracy,
     average_precision,
     balanced_accuracy,
     brier_score,
+    calibration_curve,
     cohen_kappa,
     confusion_matrix,
+    expected_calibration_error,
     f1,
     fbeta,
     hamming_loss,
@@ -59,9 +61,48 @@ from .regression import (
     rse,
     smape,
 )
+from .reporting import ClassificationReport, classification_report
+from .stats import (
+    ComparisonResult,
+    ConfidenceInterval,
+    TestResult,
+    accuracy_ci,
+    adjust_pvalues,
+    bootstrap_ci,
+    cliffs_delta,
+    cohens_d,
+    compare,
+    delong_test,
+    hedges_g,
+    mcnemar_test,
+    paired_bootstrap_test,
+    proportion_ci,
+    roc_auc_ci,
+)
 from .version import __version__
 
 __all__ = [
+    "plot",
+    "expected_calibration_error",
+    "calibration_curve",
+    "classification_report",
+    "ClassificationReport",
+    "stats",
+    "roc_auc_ci",
+    "proportion_ci",
+    "paired_bootstrap_test",
+    "mcnemar_test",
+    "hedges_g",
+    "delong_test",
+    "compare",
+    "cohens_d",
+    "cliffs_delta",
+    "bootstrap_ci",
+    "adjust_pvalues",
+    "accuracy_ci",
+    "TestResult",
+    "ConfidenceInterval",
+    "ComparisonResult",
     "EvalSuiteError",
     "EvaluationResult",
     "InputValidationError",

evalsuite_python-0.1.0b1/src/evalsuite/__main__.py

@@ -0,0 +1,5 @@
+"""``python -m evalsuite`` runs the command-line interface."""
+
+from .cli.main import main
+
+raise SystemExit(main())

{evalsuite_python-0.1.0a1 → evalsuite_python-0.1.0b1}/src/evalsuite/api.py

@@ -232,6 +232,28 @@ def _evaluate_regression(
                 f"Unknown regression metric(s): {', '.join(unknown)}. Available: {', '.join(_REGRESSION)}."
             )
     out: dict[str, MetricResult] = {}
+    shared = reg._Inputs(y_true, y_pred, sample_weight)  # validate once for every metric
+    token = reg._SHARED.set((id(y_true), id(y_pred), id(sample_weight), shared))
+    try:
+        _regression_loop(names, out, y_true, y_pred, sample_weight)
+    finally:
+        reg._SHARED.reset(token)
+    return EvaluationResult(
+        task="regression",
+        metrics=out,
+        n_samples=yt.shape[0],
+        target_type="continuous",
+        metadata=_metadata(weighted=sample_weight is not None, outputs=yt.shape[1] if multi else 1),
+    )
+
+
+def _regression_loop(
+    names: list[str],
+    out: dict[str, MetricResult],
+    y_true: ArrayLike,
+    y_pred: ArrayLike,
+    sample_weight: Optional[ArrayLike],
+) -> None:
     for name in names:
         fn = _REGRESSION[name]
         if name in _NO_WEIGHTS:
@@ -240,10 +262,3 @@ def _evaluate_regression(
             out[name] = fn(y_true, y_pred)
         else:
             out[name] = fn(y_true, y_pred, sample_weight=sample_weight)
-    return EvaluationResult(
-        task="regression",
-        metrics=out,
-        n_samples=yt.shape[0],
-        target_type="continuous",
-        metadata=_metadata(weighted=sample_weight is not None, outputs=yt.shape[1] if multi else 1),
-    )