evalsuite-python 0.1.2__tar.gz → 0.2.0__tar.gz

{evalsuite_python-0.1.2 → evalsuite_python-0.2.0}/CHANGELOG.md

@@ -6,6 +6,25 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-10-08
+
+Clinical and statistical evaluation (the v0.2.0 roadmap).
+
+### Added
+- Clinical: `sensitivity`, `ppv`, `lr_positive`, `lr_negative`, `diagnostic_odds_ratio`, `youden_j`,
+  `net_benefit`; `diagnostic_report` with confidence intervals for every measure (Wilson or Clopper–Pearson
+  for proportions, log method for likelihood ratios, Woolf for the DOR, Wald for Youden's J);
+  `decision_curve` (net benefit, treat all, treat none, useful threshold range).
+- Calibration: `maximum_calibration_error`, `calibration_slope`, `calibration_intercept`,
+  `hosmer_lemeshow`, `calibration_report`.
+- Statistical tests: `t_test` (Welch/Student), `paired_t_test`, `mann_whitney_test`, `wilcoxon_test`,
+  `kruskal_wallis_test`, `friedman_test`, `shapiro_wilk_test`, `chi_square_test`, `fisher_exact_test`, each
+  with an effect size; `cramers_v` (optional bias correction); Hochberg correction in `adjust_pvalues` and
+  `compare`.
+- `es.plot.decision_curve`; CLI commands `evalsuite diagnostic`, `evalsuite calibration` and
+  `evalsuite plot decision`.
+- Validated against statsmodels (GLM, Table2x2, proportion_confint, multipletests, CompareMeans) and SciPy.
+
 ## [0.1.2] - 2026-10-08
 
 ### Changed

{evalsuite_python-0.1.2 → evalsuite_python-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.1.2
+Version: 0.2.0
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -11,9 +11,10 @@ Author: Manoj Kumar C S, Nikhil D Bharadwaj
 Maintainer: Manoj Kumar C S, Nikhil D Bharadwaj
 License-Expression: MIT
 License-File: LICENSE
-Keywords: classification,evaluation,machine learning,metrics,regression,reproducibility,statistics
+Keywords: calibration,classification,clinical,diagnostic accuracy,evaluation,machine learning,metrics,regression,reproducibility,statistical tests,statistics
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Healthcare Industry
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
@@ -26,6 +27,7 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
 Classifier: Typing :: Typed
 Requires-Python: >=3.9
 Requires-Dist: numpy>=1.22
@@ -59,10 +61,10 @@ Description-Content-Type: text/markdown
 
 **Unified, reproducible evaluation for machine learning and research.**
 
-EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
+EvalSuite brings classification, regression, clinical and statistical evaluation (with segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: stable (0.1.2).** Every item on the 0.1.0 roadmap is implemented and verified.
+> **Status: stable (0.2.0).** Every item on the 0.1.0 and 0.2.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -134,6 +136,51 @@ Every model is evaluated on the same bootstrap resamples, so differences are pai
 with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
 p-values are adjusted for multiple comparisons (Holm by default).
 
+## Clinical evaluation
+
+```python
+report = es.diagnostic_report(y_true, y_pred)  # binary test vs reference standard
+print(report)
+# Sensitivity, specificity, PPV, NPV (Wilson CIs), LR+ and LR− (log CIs, Simel 1991),
+# diagnostic odds ratio (Woolf), Youden's J, accuracy and prevalence
+
+es.lr_positive(y_true, y_pred)
+es.youden_j(y_true, y_pred)
+
+dca = es.decision_curve(y_true, {"model": y_prob})  # net benefit vs treat all / treat none
+dca.useful_range()  # thresholds where the model beats both
+es.plot.decision_curve(y_true, {"model": y_prob})
+```
+
+Ratios that divide by zero are `inf` or NaN with a warning, never 0.
+
+## Calibration
+
+```python
+es.calibration_report(y_true, y_prob)  # Brier, ECE, MCE, intercept, slope, Hosmer–Lemeshow
+es.calibration_slope(y_true, y_prob)  # ideal 1; < 1 means predictions are too extreme
+es.calibration_intercept(y_true, y_prob)  # ideal 0 (calibration-in-the-large)
+es.hosmer_lemeshow(y_true, y_prob, n_groups=10)
+```
+
+## Statistical tests
+
+```python
+es.t_test(scores_a, scores_b)  # Welch by default; mean difference with CI and Cohen's d
+es.paired_t_test(fold_scores_a, fold_scores_b)
+es.wilcoxon_test(fold_scores_a, fold_scores_b)  # with matched-pairs rank-biserial r
+es.mann_whitney_test(a, b)  # with rank-biserial r
+es.friedman_test(scores_a, scores_b, scores_c)  # with Kendall's W
+es.kruskal_wallis_test(g1, g2, g3)
+es.shapiro_wilk_test(residuals)
+es.chi_square_test(table)  # with Cramér's V
+es.fisher_exact_test([[8, 2], [1, 5]])
+es.adjust_pvalues(p_values, method="hochberg")  # also holm, bonferroni, bh, by
+```
+
+Every test returns a `TestResult` with the statistic, p-value and an effect size, computed with SciPy and
+checked against SciPy and statsmodels in the test suite.
+
 ## Classification report
 
 ```python
@@ -175,7 +222,10 @@ evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
 evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
     --prob lr=p_lr --prob rf=p_rf --plot comparison.png
 evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
-evalsuite metrics --category classification
+evalsuite diagnostic predictions.csv --y-true label --y-pred pred   # sensitivity, LR+, DOR... with CIs
+evalsuite calibration predictions.csv --y-true label --y-prob prob  # slope, intercept, ECE, HL
+evalsuite plot decision predictions.csv --y-true label --y-prob prob -o dca.png
+evalsuite metrics --category clinical
 evalsuite info classification.mcc
 evalsuite benchmark --quick
 ```
@@ -213,6 +263,19 @@ MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matr
 one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
 calibration curve and expected calibration error.
 
+**Clinical** (binary; `pos_label`; sample weights): sensitivity, specificity, PPV, NPV, positive and
+negative likelihood ratios, diagnostic odds ratio, Youden's J, net benefit and decision curves, and a
+diagnostic report with confidence intervals for all of them.
+
+**Calibration**: calibration curve, Brier score, expected and maximum calibration error, calibration slope
+and intercept, Hosmer–Lemeshow test.
+
+**Statistics**: confidence intervals (bootstrap percentile/basic/BCa, Wilson, Clopper–Pearson, DeLong),
+paired tests (McNemar, DeLong, paired bootstrap), t-tests (Welch, Student, paired), Mann–Whitney, Wilcoxon,
+Kruskal–Wallis, Friedman, Shapiro–Wilk, χ², Fisher's exact; effect sizes (Cohen's d, Hedges' g, Cliff's
+delta, Cramér's V); multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini–Hochberg,
+Benjamini–Yekutieli).
+
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.

{evalsuite_python-0.1.2 → evalsuite_python-0.2.0}/README.md

@@ -7,10 +7,10 @@
 
 **Unified, reproducible evaluation for machine learning and research.**
 
-EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
+EvalSuite brings classification, regression, clinical and statistical evaluation (with segmentation and
 object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
 
-> **Status: stable (0.1.2).** Every item on the 0.1.0 roadmap is implemented and verified.
+> **Status: stable (0.2.0).** Every item on the 0.1.0 and 0.2.0 roadmaps is implemented and verified.
 
 ## Installation
 
@@ -82,6 +82,51 @@ Every model is evaluated on the same bootstrap resamples, so differences are pai
 with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
 p-values are adjusted for multiple comparisons (Holm by default).
 
+## Clinical evaluation
+
+```python
+report = es.diagnostic_report(y_true, y_pred)  # binary test vs reference standard
+print(report)
+# Sensitivity, specificity, PPV, NPV (Wilson CIs), LR+ and LR− (log CIs, Simel 1991),
+# diagnostic odds ratio (Woolf), Youden's J, accuracy and prevalence
+
+es.lr_positive(y_true, y_pred)
+es.youden_j(y_true, y_pred)
+
+dca = es.decision_curve(y_true, {"model": y_prob})  # net benefit vs treat all / treat none
+dca.useful_range()  # thresholds where the model beats both
+es.plot.decision_curve(y_true, {"model": y_prob})
+```
+
+Ratios that divide by zero are `inf` or NaN with a warning, never 0.
+
+## Calibration
+
+```python
+es.calibration_report(y_true, y_prob)  # Brier, ECE, MCE, intercept, slope, Hosmer–Lemeshow
+es.calibration_slope(y_true, y_prob)  # ideal 1; < 1 means predictions are too extreme
+es.calibration_intercept(y_true, y_prob)  # ideal 0 (calibration-in-the-large)
+es.hosmer_lemeshow(y_true, y_prob, n_groups=10)
+```
+
+## Statistical tests
+
+```python
+es.t_test(scores_a, scores_b)  # Welch by default; mean difference with CI and Cohen's d
+es.paired_t_test(fold_scores_a, fold_scores_b)
+es.wilcoxon_test(fold_scores_a, fold_scores_b)  # with matched-pairs rank-biserial r
+es.mann_whitney_test(a, b)  # with rank-biserial r
+es.friedman_test(scores_a, scores_b, scores_c)  # with Kendall's W
+es.kruskal_wallis_test(g1, g2, g3)
+es.shapiro_wilk_test(residuals)
+es.chi_square_test(table)  # with Cramér's V
+es.fisher_exact_test([[8, 2], [1, 5]])
+es.adjust_pvalues(p_values, method="hochberg")  # also holm, bonferroni, bh, by
+```
+
+Every test returns a `TestResult` with the statistic, p-value and an effect size, computed with SciPy and
+checked against SciPy and statsmodels in the test suite.
+
 ## Classification report
 
 ```python
@@ -123,7 +168,10 @@ evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
 evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
     --prob lr=p_lr --prob rf=p_rf --plot comparison.png
 evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
-evalsuite metrics --category classification
+evalsuite diagnostic predictions.csv --y-true label --y-pred pred   # sensitivity, LR+, DOR... with CIs
+evalsuite calibration predictions.csv --y-true label --y-prob prob  # slope, intercept, ECE, HL
+evalsuite plot decision predictions.csv --y-true label --y-prob prob -o dca.png
+evalsuite metrics --category clinical
 evalsuite info classification.mcc
 evalsuite benchmark --quick
 ```
@@ -161,6 +209,19 @@ MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matr
 one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
 calibration curve and expected calibration error.
 
+**Clinical** (binary; `pos_label`; sample weights): sensitivity, specificity, PPV, NPV, positive and
+negative likelihood ratios, diagnostic odds ratio, Youden's J, net benefit and decision curves, and a
+diagnostic report with confidence intervals for all of them.
+
+**Calibration**: calibration curve, Brier score, expected and maximum calibration error, calibration slope
+and intercept, Hosmer–Lemeshow test.
+
+**Statistics**: confidence intervals (bootstrap percentile/basic/BCa, Wilson, Clopper–Pearson, DeLong),
+paired tests (McNemar, DeLong, paired bootstrap), t-tests (Welch, Student, paired), Mann–Whitney, Wilcoxon,
+Kruskal–Wallis, Friedman, Shapiro–Wilk, χ², Fisher's exact; effect sizes (Cohen's d, Hedges' g, Cliff's
+delta, Cramér's V); multiple-testing corrections (Bonferroni, Holm, Hochberg, Benjamini–Hochberg,
+Benjamini–Yekutieli).
+
 **Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.

{evalsuite_python-0.1.2 → evalsuite_python-0.2.0}/pyproject.toml

@@ -12,10 +12,12 @@ license-files = ["LICENSE"]
 requires-python = ">=3.9"
 authors = [{ name = "Manoj Kumar C S" }, { name = "Nikhil D Bharadwaj" }]
 maintainers = [{ name = "Manoj Kumar C S" }, { name = "Nikhil D Bharadwaj" }]
-keywords = ["evaluation", "metrics", "machine learning", "statistics", "classification", "regression", "reproducibility"]
+keywords = ["evaluation", "metrics", "clinical", "diagnostic accuracy", "calibration", "statistical tests", "machine learning", "statistics", "classification", "regression", "reproducibility"]
 classifiers = [
   "Development Status :: 5 - Production/Stable",
   "Intended Audience :: Science/Research",
+  "Intended Audience :: Healthcare Industry",
+  "Topic :: Scientific/Engineering :: Medical Science Apps.",
   "Intended Audience :: Developers",
   "Operating System :: OS Independent",
   "Programming Language :: Python :: 3",

{evalsuite_python-0.1.2 → evalsuite_python-0.2.0}/src/evalsuite/__init__.py

@@ -5,8 +5,16 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import classification, plot, regression, stats
+from . import calibration, classification, clinical, plot, regression, stats
 from .api import evaluate
+from .calibration import (
+    CalibrationReport,
+    calibration_intercept,
+    calibration_report,
+    calibration_slope,
+    hosmer_lemeshow,
+    maximum_calibration_error,
+)
 from .classification import (
     accuracy,
     average_precision,
@@ -31,6 +39,19 @@ from .classification import (
     specificity,
     top_k_accuracy,
 )
+from .clinical import (
+    DecisionCurve,
+    DiagnosticReport,
+    decision_curve,
+    diagnostic_odds_ratio,
+    diagnostic_report,
+    lr_negative,
+    lr_positive,
+    net_benefit,
+    ppv,
+    sensitivity,
+    youden_j,
+)
 from .core.exceptions import (
     EvalSuiteError,
     InputValidationError,
@@ -69,19 +90,58 @@ from .stats import (
     accuracy_ci,
     adjust_pvalues,
     bootstrap_ci,
+    chi_square_test,
     cliffs_delta,
     cohens_d,
     compare,
+    cramers_v,
     delong_test,
+    fisher_exact_test,
+    friedman_test,
     hedges_g,
+    kruskal_wallis_test,
+    mann_whitney_test,
     mcnemar_test,
     paired_bootstrap_test,
+    paired_t_test,
     proportion_ci,
     roc_auc_ci,
+    shapiro_wilk_test,
+    t_test,
+    wilcoxon_test,
 )
 from .version import __version__
 
 __all__ = [
+    "CalibrationReport",
+    "calibration_report",
+    "calibration",
+    "clinical",
+    "calibration_intercept",
+    "calibration_slope",
+    "hosmer_lemeshow",
+    "maximum_calibration_error",
+    "DecisionCurve",
+    "DiagnosticReport",
+    "decision_curve",
+    "diagnostic_odds_ratio",
+    "diagnostic_report",
+    "lr_negative",
+    "lr_positive",
+    "net_benefit",
+    "ppv",
+    "sensitivity",
+    "youden_j",
+    "chi_square_test",
+    "cramers_v",
+    "fisher_exact_test",
+    "friedman_test",
+    "kruskal_wallis_test",
+    "mann_whitney_test",
+    "paired_t_test",
+    "shapiro_wilk_test",
+    "t_test",
+    "wilcoxon_test",
     "plot",
     "expected_calibration_error",
     "calibration_curve",