evalsuite-python 0.4.0__tar.gz → 0.4.1__tar.gz

{evalsuite_python-0.4.0 → evalsuite_python-0.4.1}/CHANGELOG.md

@@ -6,6 +6,24 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.4.1] - 2026-10-09
+
+Benchmarks for every metric. No API changed.
+
+### Added
+- `run_benchmarks(suite="metrics")` and `evalsuite benchmark --suite metrics`: one row for each of the 147
+  registered metrics and statistics functions, compared with a reference library (scikit-learn, SciPy,
+  statsmodels, pycocotools, sacreBLEU, rouge-score, NLTK, pycocoevalcap, ranx, krippendorff, choix, POT,
+  jsonschema) where one exists, otherwise with an independent NumPy / standard-library implementation of
+  the textbook formula; learned, judge-dependent and randomised procedures are timed alone. All 128
+  comparisons agree to 1e-9. A test keeps the suite in step with the registry.
+- `scripts/bench_metrics_report.py` renders the per-metric table for BENCHMARKS.md and the website.
+
+### Changed
+- `net_benefit` at a single threshold computes in one pass instead of sorting (about 3× faster).
+- `perplexity` and `cross_entropy` validate the pooled tokens once instead of each sequence separately
+  (5–9× faster for many short sequences); errors still name the offending sequence.
+
 ## [0.4.0] - 2026-10-09
 
 LLM evaluation (the v0.4.0 roadmap). No existing API changed.

{evalsuite_python-0.4.0 → evalsuite_python-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.4.0
+Version: 0.4.1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -82,7 +82,7 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.0).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
+> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
 > (except SPICE, which needs a Java scene-graph parser).
 > **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
 
@@ -343,6 +343,12 @@ alphabetical order:
 | text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
 | text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
+independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+measurements faster). Per-metric table in BENCHMARKS.md.
+
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows
 below 1× pay for input validation and the extra intervals and effect sizes EvalSuite reports. Reproduce on

{evalsuite_python-0.4.0 → evalsuite_python-0.4.1}/README.md

@@ -10,7 +10,7 @@
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.0).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
+> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
 > (except SPICE, which needs a Java scene-graph parser).
 > **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
 
@@ -271,6 +271,12 @@ alphabetical order:
 | text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
 | text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
+independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+measurements faster). Per-metric table in BENCHMARKS.md.
+
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows
 below 1× pay for input validation and the extra intervals and effect sizes EvalSuite reports. Reproduce on