evalsuite-python 0.1.0__py3-none-any.whl

evalsuite_python-0.1.0.dist-info/METADATA

@@ -0,0 +1,247 @@
+Metadata-Version: 2.5
+Name: evalsuite-python
+Version: 0.1.0
+Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
+Project-URL: Homepage, https://evalsuite-nine.vercel.app
+Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
+Project-URL: Source, https://github.com/mkcs28/evalsuite-python
+Project-URL: Issues, https://github.com/mkcs28/evalsuite-python/issues
+Project-URL: Changelog, https://github.com/mkcs28/evalsuite-python/blob/main/CHANGELOG.md
+Author: Manoj Kumar C S
+Maintainer: Manoj Kumar C S
+License-Expression: MIT
+License-File: LICENSE
+Keywords: classification,evaluation,machine learning,metrics,regression,reproducibility,statistics
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.22
+Requires-Dist: pandas>=1.4
+Requires-Dist: scipy>=1.8
+Provides-Extra: all
+Requires-Dist: matplotlib>=3.5; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: hypothesis>=6.80; extra == 'dev'
+Requires-Dist: matplotlib>=3.5; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: pip-audit; extra == 'dev'
+Requires-Dist: pytest-cov>=4; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: scikit-learn>=1.2; extra == 'dev'
+Requires-Dist: statsmodels>=0.13; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: plot
+Requires-Dist: matplotlib>=3.5; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# EvalSuite
+
+[![CI](https://github.com/mkcs28/evalsuite-python/actions/workflows/ci.yml/badge.svg)](https://github.com/mkcs28/evalsuite-python/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/evalsuite-python)](https://pypi.org/project/evalsuite-python/)
+[![Python](https://img.shields.io/pypi/pyversions/evalsuite-python)](https://pypi.org/project/evalsuite-python/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+**Unified, reproducible evaluation for machine learning and research.**
+
+EvalSuite brings classification and regression metrics (with clinical, statistical, segmentation and
+object-detection evaluation on the roadmap) into one consistent, validated, documented framework.
+
+> **Status: stable (0.1.0).** Every item on the 0.1.0 roadmap is implemented and verified.
+
+## Installation
+
+```bash
+pip install evalsuite-python
+```
+
+The package is installed as `evalsuite-python` and imported as `evalsuite`:
+
+```python
+import evalsuite as es
+```
+
+## Why EvalSuite
+
+- **One consistent API.** Every metric returns a result object that behaves like a number and exports to
+  JSON, pandas, Markdown and LaTeX.
+- **Explicit conventions.** Averaging, label order, the positive class and zero-division behaviour are stated
+  and recorded in every result, never silently assumed.
+- **Validated.** Each metric is tested against scikit-learn where definitions coincide, plus property-based
+  tests and edge cases.
+- **Documented.** Every metric carries its definition, formula, range, input requirements and references,
+  available programmatically through `metric_info()`.
+- **Efficient.** `evaluate()` validates inputs once and computes the confusion matrix once for all metrics.
+- **Lightweight.** Requires only NumPy, SciPy and pandas.
+
+## Quick start
+
+```python
+import evalsuite as es
+
+y_true = [0, 1, 1, 0, 1, 0]
+y_pred = [0, 1, 0, 0, 1, 1]
+y_prob = [0.1, 0.9, 0.4, 0.2, 0.8, 0.6]
+
+result = es.evaluate(y_true, y_pred, y_prob=y_prob)
+print(result.summary())
+
+result["f1"]  # MetricResult(f1=0.666667)
+f"{result['mcc']:.3f}"  # '0.333'
+result.to_latex(caption="Test-set performance")
+result.to_dataframe()
+
+es.f1(y_true, y_pred)  # individual metrics
+es.roc_auc(y_true, y_prob)
+es.metric_info("classification.mcc").formula  # documentation
+es.list_metrics("regression")
+```
+
+## Comparing models
+
+```python
+result = es.compare(
+    y_true,
+    {"logistic": pred_lr, "forest": pred_rf, "boosting": pred_gb},
+    probabilities={"logistic": prob_lr, "forest": prob_rf, "boosting": prob_gb},
+    random_state=0,
+)
+print(result.summary())  # estimates with 95% CIs, paired tests, Holm-adjusted p-values
+result.to_latex(label="tab:models")
+
+es.bootstrap_ci("f1", y_true, y_pred, average="macro", random_state=0)  # BCa interval for any metric
+es.accuracy_ci(y_true, y_pred)  # Wilson interval
+es.delong_test(y_true, prob_a, prob_b)  # two correlated AUCs
+es.mcnemar_test(y_true, pred_a, pred_b)
+```
+
+Every model is evaluated on the same bootstrap resamples, so differences are paired. Accuracy is compared
+with McNemar's test, binary ROC AUC with DeLong's test and other metrics with a paired bootstrap test;
+p-values are adjusted for multiple comparisons (Holm by default).
+
+## Classification report
+
+```python
+report = es.classification_report(y_true, y_pred)
+print(report)  # per-class precision, recall, F1, specificity, support + averages
+report.save("report.html")  # also .csv .md .tex .json .txt
+```
+
+## Plots
+
+```bash
+pip install "evalsuite-python[plot]"   # adds matplotlib; importing evalsuite never loads it
+```
+
+```python
+es.plot.roc(y_true, {"logistic": prob_lr, "forest": prob_rf})  # AUC in the legend
+es.plot.pr(y_true, prob)  # AP and the prevalence line
+es.plot.calibration(y_true, prob)  # reliability diagram, ECE, Brier
+es.plot.confusion_matrix(y_true, y_pred, normalize="true")
+es.plot.residuals(y_reg, pred_reg)  # or kind="predicted"
+es.plot.comparison(es.compare(...))  # forest plot with CIs
+```
+
+Each function returns a matplotlib `Axes` (pass `ax=` to draw into your own figure). The numbers shown are
+computed with EvalSuite's metrics, so plots and tables always agree. Several models get distinct colours
+*and* line styles, so figures stay readable in greyscale print.
+
+## Exports
+
+Every result (`evaluate`, `classification_report`, `compare`, single metrics) exports to `summary()`,
+`to_json()`, `to_csv()`, `to_dataframe()`, `to_markdown()`, `to_latex()` and `to_html()`, and `save(path)` picks
+the format from the extension. HTML pages are standalone (inline CSS, no scripts) and escape all text.
+
+## Command line
+
+```bash
+evalsuite evaluate predictions.csv --y-true label --y-pred pred --y-prob prob
+evalsuite report predictions.csv --y-true label --y-pred pred -o report.html
+evalsuite compare predictions.csv --y-true label --pred lr=pred_lr --pred rf=pred_rf \
+    --prob lr=p_lr --prob rf=p_rf --plot comparison.png
+evalsuite plot roc predictions.csv --y-true label --y-prob prob -o roc.png
+evalsuite metrics --category classification
+evalsuite info classification.mcc
+evalsuite benchmark --quick
+```
+
+Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` or the `-o` extension
+(text, json, csv, markdown, latex, html). Errors are reported in one line with exit code 2.
+
+## Performance
+
+Benchmarked against scikit-learn on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
+scikit-learn 1.9, Linux x86_64). Every result agrees with scikit-learn to floating-point rounding
+(largest difference 1.1e-16).
+
+| Case | n | EvalSuite (ms) | scikit-learn (ms) | Speed-up | Peak memory EvalSuite / sklearn (MiB) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 8 binary label metrics via `evaluate()` | 1,000 | 0.38 | 11.70 | **31.2×** | 0.04 / 0.05 |
+| 8 binary label metrics via `evaluate()` | 100,000 | 10.9 | 116.9 | **10.7×** | 3.2 / 3.1 |
+| 8 binary label metrics via `evaluate()` | 1,000,000 | 108.8 | 1043.6 | **9.6×** | 31.5 / 30.5 |
+| macro F1, 10 classes | 1,000,000 | 88.4 | 139.3 | **1.58×** | 30.5 / 21.8 |
+| ROC AUC, binary | 1,000,000 | 247.4 | 352.5 | **1.42×** | 91.6 / 76.3 |
+| MAE, MSE, RMSE, R² via `evaluate()` | 1,000 | 0.12 | 0.90 | **7.2×** | 0.03 / 0.02 |
+| MAE, MSE, RMSE, R² via `evaluate()` | 1,000,000 | 29.3 | 20.1 | 0.69× | 22.9 / 15.3 |
+
+`evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where the
+speed-up comes from. Large regression arrays are slower because EvalSuite checks every value for NaN,
+infinity, shape and dtype before computing. Reproduce on your machine with `evalsuite benchmark`; full
+table and notes in
+[BENCHMARKS.md](https://github.com/mkcs28/evalsuite-python/blob/main/BENCHMARKS.md).
+
+## Metrics in this release
+
+**Classification** (binary, multiclass, multilabel; micro/macro/weighted/samples/per-class averaging;
+sample weights): accuracy, balanced accuracy, precision, recall, specificity, NPV, F1, F-beta, Jaccard,
+MCC, Cohen's kappa (unweighted, linear, quadratic), Hamming loss, confusion matrix, ROC AUC (binary,
+one-vs-rest, one-vs-one), average precision, ROC and PR curves, log loss, Brier score, top-k accuracy,
+calibration curve and expected calibration error.
+
+**Regression** (single and multi-output; sample weights): MAE, MSE, RMSE, R², adjusted R², MAPE, sMAPE,
+MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
+loss, Huber loss, relative absolute error, relative squared error.
+
+## Conventions
+
+- `average="auto"` resolves to `"binary"` for binary targets and `"macro"` otherwise; the resolved value
+  is stored in `result.params["average"]`.
+- Labels are sorted unless you pass `labels=[...]`; that order defines per-class outputs and the columns
+  of 2-D `y_prob`.
+- Undefined ratios (zero denominators) return 0 **with an `UndefinedMetricWarning`**; pass
+  `zero_division=np.nan` to propagate NaN, or `0`/`1` to choose silently.
+- Domain violations raise clear errors instead of being patched over (for example MAPE with zero targets).
+
+## Development
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+pytest --cov=evalsuite
+ruff check . && ruff format --check . && mypy
+```
+
+## Links
+
+- PyPI: https://pypi.org/project/evalsuite-python/
+- Website and documentation: https://evalsuite-nine.vercel.app
+- Website source: https://github.com/mkcs28/evalsuite
+
+## License
+
+MIT. See [LICENSE](LICENSE).

evalsuite_python-0.1.0.dist-info/RECORD

@@ -0,0 +1,35 @@
+evalsuite/__init__.py,sha256=giC_4Uq1bnYCuDuw7lxuxDAp5vFxN7RLkdNTnBt-pPM,3114
+evalsuite/__main__.py,sha256=Nq90_8VlUN1mif-AVgX-kZRL63bd94F5-F7F9UkVZD4,117
+evalsuite/api.py,sha256=ggEc_LM7guktX6pnApYIf-n8AHHMujdiWriqMy6JZdE,9578
+evalsuite/benchmarks.py,sha256=6y0K6-5vwIc7D3ACItQXLSqVCRX6EvqWdVPttVlBkNU,10018
+evalsuite/plot.py,sha256=QqQgcEOjIGkpMSwfgXjO2PzOaLBmj8-dTZrzlqzOHFw,11416
+evalsuite/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+evalsuite/reporting.py,sha256=-LISv9vvEnltT26GtzLjVlxOCcAwmjv4-HyB8_1UF5o,9615
+evalsuite/version.py,sha256=NhOfuA4GsTmRzoTsxKIfwdcZ0oQ5D93WpPm-V0Nw8ck,98
+evalsuite/classification/__init__.py,sha256=ErBIXbBGnhfW6v-1Cn5SegiERoyxNlmO1tEKcLM15mA,833
+evalsuite/classification/_common.py,sha256=3ANV33ix_mnOzvOAG8glyJO-iYVhhTGO0S3YG0ZddB8,4543
+evalsuite/classification/metrics.py,sha256=8-RPjzuFUKJYlXndrDv6kXlzRdx6aR2lHp7mzKsas2g,36969
+evalsuite/cli/__init__.py,sha256=1x48hORLcg_Is2X08A7CPBT6VDJ1QL94zaJP1HTFxUY,96
+evalsuite/cli/main.py,sha256=ez_P7SKuuOin9E5CAQ0orA5GSHn8sE-oH248RiFsgpI,15021
+evalsuite/core/__init__.py,sha256=oDHl5Dc2GoZf1tCtJueMQ2KcmOq7zhzMZT9XF4uVC6U,93
+evalsuite/core/context.py,sha256=gvRpH3b-S2oAgAGW9W0XWnBVbuydQP3UB9rgWQNM60g,7874
+evalsuite/core/exceptions.py,sha256=9eZI_Lmgwc9fjkSUY9s3fwjAb7KoqgAlcYetrHRhbBg,1503
+evalsuite/core/export.py,sha256=6M_ktQzW3E7usOrwqv5f5zOEdoi48JD6CvgyRpDkGvM,4303
+evalsuite/core/registry.py,sha256=3l2_o2PlckuKY89HxfnxPLf-sDaCe_-McUcAcUmXusk,2884
+evalsuite/core/result.py,sha256=L_fQZRGRgVgzII2jZb_jpRBK2jy-CdO5u9s-6pUk9fE,14927
+evalsuite/core/types.py,sha256=r6bhUk7CYAcrlQzjmf-6xWQsNqO3EOipZD-e-2yAJTI,720
+evalsuite/core/validation.py,sha256=EIAd6m5DIguFM4wLUNVissEYqvjylYEuLGrPq8hOUpw,8350
+evalsuite/regression/__init__.py,sha256=t2XEHUw6e5tyWBHcEeiog2WWd-wwaxf4m5AIX8dJyng,571
+evalsuite/regression/metrics.py,sha256=jVdzmegdpBdyJNCSm1dFR1E-9VHDmQmTVtyrqyjk3KY,22459
+evalsuite/stats/__init__.py,sha256=PcYS9tZh-IptSXx3sX7shmUUdNYsQuD-G7Xkgb3FHiY,745
+evalsuite/stats/_resolve.py,sha256=cK6qdwM7P_mSjzHDEVcV4SekBS58VGGYQ8PuKptgqQs,3768
+evalsuite/stats/compare.py,sha256=DmC53p_IlL5ecP_7sGEYV5fszKjLIyN3dN0JMUp_ri4,17746
+evalsuite/stats/effect.py,sha256=O0-pxY_6o25G3w7ymDar3xb_dtCOO0kG0sLv-A_Yb0Q,4884
+evalsuite/stats/intervals.py,sha256=w2bLuFt3p-TTN_ALH1uKPmDMLcTzBBoV2YvcP-ENAOk,13709
+evalsuite/stats/paired.py,sha256=YK3AFndChKeCEpK512u56VtRlKvhlGdkFLHHYe77zmc,8244
+evalsuite/stats/results.py,sha256=nqXHUiW8l0fQJkbMiunU6Gz-2be1NApZ6Fif2B1K_Yw,4407
+evalsuite_python-0.1.0.dist-info/METADATA,sha256=PMXY0G7f1HRp5SK9mBIMnm8cFBY29t80b1M3QN2ym5s,10743
+evalsuite_python-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+evalsuite_python-0.1.0.dist-info/entry_points.txt,sha256=YYIwisn4L4sQt3ntzbM0_Z4yZE2KCu3GJ6KfWL1kIwc,54
+evalsuite_python-0.1.0.dist-info/licenses/LICENSE,sha256=GGh6ACzs6Gih4IlfbWxzHvkNKC1E9EQ3eVWW0Njpafk,1072
+evalsuite_python-0.1.0.dist-info/RECORD,,

evalsuite_python-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any

evalsuite_python-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+evalsuite = evalsuite.cli.main:main

evalsuite_python-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Manoj Kumar C S
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.