evalsuite-python 0.3.1__tar.gz → 0.4.0__tar.gz

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/CHANGELOG.md

@@ -6,6 +6,45 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-10-09
+
+LLM evaluation (the v0.4.0 roadmap). No existing API changed.
+
+### Added
+- `es.text` and top-level functions for language-model evaluation; 69 new registered metrics, each checked
+  against a reference implementation where one exists:
+  - Text generation: BLEU, sentence BLEU, chrF/chrF++, TER (sacreBLEU), ROUGE-1/2/L/Lsum (rouge-score),
+    METEOR (NLTK, synonyms pluggable), CIDEr-D (pycocoevalcap), perplexity, cross-entropy, Distinct-n,
+    Self-BLEU, MAUVE (mauve-text divergence frontier, NumPy k-means).
+  - Semantic: BERTScore, embedding cosine / Euclidean / Manhattan, MoverScore (exact transport, = POT), and
+    `model_score` to run COMET, BLEURT, BARTScore, AlignScore or any scorer with intervals and comparison.
+  - Factuality: faithfulness, hallucination (unsupported-claim) rate, groundedness, citation precision and
+    recall (ALCE), claim-verification accuracy, knowledge consistency, answer correctness and relevance
+    (RAGAS), abstention accuracy, exact match and token F1 (SQuAD).
+  - LLM-as-a-judge: win rate, Bradley–Terry (= choix), Elo, Krippendorff's alpha (= krippendorff), Fleiss'
+    kappa (= statsmodels), judge–human agreement, position consistency, verbosity and self-preference bias,
+    rubric scores.
+  - Reasoning: pass@k, majority-vote accuracy, answer extraction and benchmark accuracy (GSM8K, MATH,
+    multiple choice).
+  - RAG: Precision/Recall/Hit rate@k, MRR, MAP, NDCG (= ranx), context precision, recall and relevance,
+    latency summary, task success rate, failure attribution.
+  - Structured output: JSON validity, JSON Schema compliance (built-in validator, agrees with jsonschema),
+    XML validity, required-field accuracy, tool selection / argument accuracy and tool-call F1, API-call
+    success, instruction compliance, constraint satisfaction, format compliance, multi-turn retention,
+    extra-content rate.
+- `es.text_report` and `evalsuite text` (CLI) for the reference-based text metrics in one report.
+- `bootstrap_ci`, `compare` and the paired tests resample whole items (texts, reference lists, rankings) for
+  these metrics, so corpus metrics such as BLEU get proper intervals.
+- Benchmark suite `llm` against sacreBLEU, rouge-score, NLTK, ranx, krippendorff and jsonschema.
+- `llm` extra (`nltk`) for METEOR's default Porter stemmer.
+- Plots: `es.plot.ratings` (Bradley–Terry or Elo leaderboard with bootstrap intervals), `es.plot.win_matrix`
+  (pairwise win rates) and `es.plot.text_scores` (per-example score distributions by system).
+- Benchmark overall summary counts cases without their size suffixes.
+
+### Not included
+- SPICE (needs a Java scene-graph parser); BLEURT, COMET, BARTScore and AlignScore run through
+  `es.model_score` with your model rather than being bundled.
+
 ## [0.3.1] - 2026-10-09
 
 Quality-assurance release: release pipeline, packaging checks, edge-case fixes and an overall benchmark.

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.3.1
+Version: 0.4.0
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -36,22 +36,34 @@ Requires-Dist: pandas>=1.4
 Requires-Dist: scipy>=1.8
 Provides-Extra: all
 Requires-Dist: matplotlib>=3.5; extra == 'all'
+Requires-Dist: nltk>=3.8; extra == 'all'
 Requires-Dist: pillow>=9; extra == 'all'
 Provides-Extra: dev
 Requires-Dist: build; extra == 'dev'
+Requires-Dist: choix>=0.3; extra == 'dev'
 Requires-Dist: hypothesis>=6.80; extra == 'dev'
+Requires-Dist: jsonschema>=4.18; extra == 'dev'
+Requires-Dist: krippendorff>=0.6; extra == 'dev'
 Requires-Dist: matplotlib>=3.5; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: nltk>=3.8; extra == 'dev'
 Requires-Dist: pandas-stubs; extra == 'dev'
 Requires-Dist: pillow>=9; extra == 'dev'
 Requires-Dist: pip-audit; extra == 'dev'
+Requires-Dist: pot>=0.9; (python_version < '3.14') and extra == 'dev'
+Requires-Dist: pycocoevalcap>=1.2; extra == 'dev'
 Requires-Dist: pycocotools>=2.0.7; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ranx>=0.3.19; (python_version < '3.14') and extra == 'dev'
+Requires-Dist: rouge-score>=0.1.2; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: sacrebleu>=2.4; extra == 'dev'
 Requires-Dist: scikit-learn>=1.4; extra == 'dev'
 Requires-Dist: statsmodels>=0.13; extra == 'dev'
 Requires-Dist: twine; extra == 'dev'
+Provides-Extra: llm
+Requires-Dist: nltk>=3.8; extra == 'llm'
 Provides-Extra: plot
 Requires-Dist: matplotlib>=3.5; extra == 'plot'
 Provides-Extra: vision
@@ -65,12 +77,14 @@ Description-Content-Type: text/markdown
 [![Python](https://img.shields.io/pypi/pyversions/evalsuite-python)](https://pypi.org/project/evalsuite-python/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 
-**Unified, reproducible evaluation for machine learning and research.**
+**Unified, reproducible evaluation for machine learning, LLM and research.**
 
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.3.1).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
+> **Status: stable (0.4.0).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
+> (except SPICE, which needs a Java scene-graph parser).
+> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
 
 ## Installation
 
@@ -226,6 +240,22 @@ Models can be compared over the same images with intervals and paired tests, as
 `es.compare(y_true, {"unet": masks_a, "deeplab": masks_b})` resamples images; for detection it compares
 mAP.
 
+## LLM evaluation
+
+```python
+es.bleu(references, predictions)  # sacreBLEU-identical; also chrf, ter, rouge_l, meteor, cider
+es.text_report(references, predictions)  # BLEU, chrF(++), TER, ROUGE, METEOR, EM, token F1 at once
+es.bertscore(ref_token_embs, pred_token_embs)  # from your encoder's embeddings; also mauve, moverscore
+es.model_score(refs, preds, scorer=comet_fn)  # any learned metric or judge, with intervals and compare
+es.faithfulness(claim_verdicts)  # also hallucination_rate, citation_recall, answer_correctness
+es.bradley_terry(comparisons, scale="elo")  # also win_rate, elo_ratings, krippendorff_alpha
+es.ndcg_at_k(relevant, retrieved, k=10)  # also mrr, context_precision, context_recall
+es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_compliance_rate
+es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
+es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
+es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+```
+
 ## Classification report
 
 ```python
@@ -283,28 +313,35 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 ## Performance
 
 Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 45 of 45 rows agree with the reference** (largest
-difference 2.8e-14), 32 are faster, and the geometric-mean speed-up across all
-15 cases and three sizes is **2.22×**. At 1,000,000 samples (pixels for
-segmentation; 1,000 images for detection and 50 for Hausdorff), in alphabetical order:
+Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
+difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
+21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
+alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 29.3 | 130.8 | **4.5×** |
-| binary: 8 label metrics via evaluate() | scikit-learn | 39.5 | 1084.1 | **27.4×** |
-| binary: ROC AUC | scikit-learn | 195.0 | 348.3 | **1.8×** |
-| calibration: slope and intercept | statsmodels | 193.0 | 1068.0 | **5.5×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 23.5 | 4.7 | 0.20× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 69.1 | 414.0 | **6.0×** |
-| decision curve: 99 thresholds | NumPy loop | 156.3 | 181.6 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 817.3 | 983.7 | **1.2×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 82.0 | 108.0 | **1.3×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 23.1 | 10.8 | 0.47× |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 43.0 | 290.0 | **6.8×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 27.0 | 20.2 | 0.75× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.5 | 0.4 | 0.88× |
-| statistics: Mann–Whitney U | SciPy | 349.7 | 350.2 | **1.0×** |
-| statistics: Welch t-test | SciPy | 14.5 | 7.2 | 0.49× |
+| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
+| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
+| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
+| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
+| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
+| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/README.md

@@ -5,12 +5,14 @@
 [![Python](https://img.shields.io/pypi/pyversions/evalsuite-python)](https://pypi.org/project/evalsuite-python/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 
-**Unified, reproducible evaluation for machine learning and research.**
+**Unified, reproducible evaluation for machine learning, LLM and research.**
 
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.3.1).** Every item on the 0.1.0, 0.2.0 and 0.3.0 roadmaps is implemented and verified.
+> **Status: stable (0.4.0).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
+> (except SPICE, which needs a Java scene-graph parser).
+> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
 
 ## Installation
 
@@ -166,6 +168,22 @@ Models can be compared over the same images with intervals and paired tests, as
 `es.compare(y_true, {"unet": masks_a, "deeplab": masks_b})` resamples images; for detection it compares
 mAP.
 
+## LLM evaluation
+
+```python
+es.bleu(references, predictions)  # sacreBLEU-identical; also chrf, ter, rouge_l, meteor, cider
+es.text_report(references, predictions)  # BLEU, chrF(++), TER, ROUGE, METEOR, EM, token F1 at once
+es.bertscore(ref_token_embs, pred_token_embs)  # from your encoder's embeddings; also mauve, moverscore
+es.model_score(refs, preds, scorer=comet_fn)  # any learned metric or judge, with intervals and compare
+es.faithfulness(claim_verdicts)  # also hallucination_rate, citation_recall, answer_correctness
+es.bradley_terry(comparisons, scale="elo")  # also win_rate, elo_ratings, krippendorff_alpha
+es.ndcg_at_k(relevant, retrieved, k=10)  # also mrr, context_precision, context_recall
+es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_compliance_rate
+es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
+es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
+es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+```
+
 ## Classification report
 
 ```python
@@ -223,28 +241,35 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 ## Performance
 
 Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 45 of 45 rows agree with the reference** (largest
-difference 2.8e-14), 32 are faster, and the geometric-mean speed-up across all
-15 cases and three sizes is **2.22×**. At 1,000,000 samples (pixels for
-segmentation; 1,000 images for detection and 50 for Hausdorff), in alphabetical order:
+Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
+difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
+21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
+alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 29.3 | 130.8 | **4.5×** |
-| binary: 8 label metrics via evaluate() | scikit-learn | 39.5 | 1084.1 | **27.4×** |
-| binary: ROC AUC | scikit-learn | 195.0 | 348.3 | **1.8×** |
-| calibration: slope and intercept | statsmodels | 193.0 | 1068.0 | **5.5×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 23.5 | 4.7 | 0.20× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 69.1 | 414.0 | **6.0×** |
-| decision curve: 99 thresholds | NumPy loop | 156.3 | 181.6 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 817.3 | 983.7 | **1.2×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 82.0 | 108.0 | **1.3×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 23.1 | 10.8 | 0.47× |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 43.0 | 290.0 | **6.8×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 27.0 | 20.2 | 0.75× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.5 | 0.4 | 0.88× |
-| statistics: Mann–Whitney U | SciPy | 349.7 | 350.2 | **1.0×** |
-| statistics: Welch t-test | SciPy | 14.5 | 7.2 | 0.49× |
+| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
+| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
+| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
+| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
+| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
+| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
 of the speed-up comes from. Hypothesis tests use SciPy underneath, so they match its speed at best; rows

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/pyproject.toml

@@ -38,7 +38,8 @@ dependencies = ["numpy>=1.22", "scipy>=1.8", "pandas>=1.4"]
 [project.optional-dependencies]
 plot = ["matplotlib>=3.5"]
 vision = ["pillow>=9"]
-all = ["matplotlib>=3.5", "pillow>=9"]
+llm = ["nltk>=3.8"]
+all = ["matplotlib>=3.5", "pillow>=9", "nltk>=3.8"]
 dev = [
   "pytest>=7",
   "pytest-cov>=4",
@@ -48,6 +49,15 @@ dev = [
   "matplotlib>=3.5",
   "pillow>=9",
   "pycocotools>=2.0.7",
+  "nltk>=3.8",
+  "sacrebleu>=2.4",
+  "rouge-score>=0.1.2",
+  "pycocoevalcap>=1.2",
+  "jsonschema>=4.18",
+  "choix>=0.3",
+  "krippendorff>=0.6",
+  "pot>=0.9; python_version < '3.14'",
+  "ranx>=0.3.19; python_version < '3.14'",
   "ruff>=0.6",
   "mypy>=1.10",
   "pandas-stubs",
@@ -99,6 +109,12 @@ files = ["src/evalsuite"]
 module = ["scipy.*", "matplotlib.*"]
 ignore_missing_imports = true
 
+[[tool.mypy.overrides]]
+# optional reference libraries used only by the benchmarks and METEOR's default stemmer
+module = ["sacrebleu.*", "rouge_score.*", "nltk.*", "ranx.*", "krippendorff.*", "jsonschema.*", "pycocotools.*"]
+ignore_missing_imports = true
+follow_untyped_imports = false
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 addopts = "-q --strict-markers"

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/src/evalsuite/__init__.py

@@ -5,7 +5,7 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import calibration, classification, clinical, plot, regression, stats, vision
+from . import calibration, classification, clinical, plot, regression, stats, text, vision
 from .api import evaluate
 from .calibration import (
     CalibrationReport,
@@ -110,6 +110,8 @@ from .stats import (
     t_test,
     wilcoxon_test,
 )
+from .text import *  # noqa: F403  (every LLM-evaluation function, listed in text.__all__)
+from .text import __all__ as _text_all
 from .version import __version__
 from .vision import (
     DetectionReport,
@@ -134,6 +136,7 @@ from .vision import (
 )
 
 __all__ = [
+    "text",
     "SegmentationReport",
     "segmentation_report",
     "detection_pr_curve",
@@ -258,3 +261,4 @@ __all__ = [
     "specificity",
     "top_k_accuracy",
 ]
+__all__ += list(_text_all)

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/src/evalsuite/benchmarks.py

@@ -213,8 +213,8 @@ def _vision_cases(n: int, rng: np.random.Generator) -> list[Case]:
         import contextlib as _ctx
         import io
 
-        from pycocotools.coco import COCO  # type: ignore[import-untyped]
-        from pycocotools.cocoeval import COCOeval  # type: ignore[import-untyped]
+        from pycocotools.coco import COCO
+        from pycocotools.cocoeval import COCOeval
 
         def coco_map() -> list[float]:
             images, anns, dets, aid = [], [], [], 1
@@ -262,6 +262,155 @@ def _vision_cases(n: int, rng: np.random.Generator) -> list[Case]:
     return [(name, ref, es_fn, refs.get(name)) for name, ref, es_fn, _ in cases]
 
 
+_LLM_VOCAB = (  # noqa: SIM905
+    "the a cat dog sat on mat quickly model models data run running ran 2.5 3,000 well-known state-of-the-art "
+    "( ) , . ! ? don't U.S. e-mail Hello WORLD naive cafe results show improves baseline evaluation"
+).split()
+
+
+def _llm_cases(n: int, rng: np.random.Generator) -> list[Case]:
+    """v0.4.0: text generation, retrieval, rater agreement and structured output (n / 100 examples)."""
+    import evalsuite as es
+
+    m = max(10, n // 100)
+    preds, refs_ = [], []
+    for _ in range(m):
+        base = list(rng.choice(_LLM_VOCAB, int(rng.integers(5, 25))))
+        keep = [w for w in base if rng.random() > 0.2] + list(rng.choice(_LLM_VOCAB, int(rng.integers(0, 4))))
+        preds.append(" ".join(keep))
+        refs_.append(" ".join(base))
+    queries = [
+        (set(rng.choice(200, int(rng.integers(1, 6)), replace=False).tolist()), rng.permutation(200)[:20].tolist())
+        for _ in range(m)
+    ]
+    rel, ret = [q[0] for q in queries], [q[1] for q in queries]
+    ratings = rng.integers(1, 6, (4, m)).astype(float)
+    ratings[rng.random(ratings.shape) < 0.1] = np.nan
+    schema = {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "minLength": 1},
+            "age": {"type": "integer", "minimum": 0},
+            "tags": {"type": "array", "items": {"type": "string"}, "uniqueItems": True},
+        },
+        "required": ["name", "age"],
+        "additionalProperties": False,
+    }
+    docs = []
+    for i in range(m):
+        d: dict[str, Any] = {"name": f"n{i}", "age": int(rng.integers(-2, 90)), "tags": ["a", "b"]}
+        if rng.random() < 0.1:
+            d["extra"] = 1
+        if rng.random() < 0.1:
+            del d["name"]
+        docs.append(d)
+    import json as _json
+
+    docs_text = [_json.dumps(d) for d in docs]
+
+    cases: list[Case] = [
+        (
+            f"text: corpus BLEU and chrF ({m} sentences)",
+            "sacreBLEU",
+            lambda: [float(es.bleu(refs_, preds)), float(es.chrf(refs_, preds))],
+            None,
+        ),
+        (
+            f"text: ROUGE-1, ROUGE-2, ROUGE-L ({m} sentences)",
+            "rouge-score",
+            lambda: [
+                float(es.rouge_1(refs_, preds)),
+                float(es.rouge_2(refs_, preds)),
+                float(es.rouge_l(refs_, preds)),
+            ],
+            None,
+        ),
+        (
+            f"text: METEOR, exact and stem matches ({m} sentences)",
+            "NLTK",
+            lambda: [float(es.meteor(refs_, preds))],
+            None,
+        ),
+        (
+            f"retrieval: MRR, MAP@20, NDCG@10 ({m} queries)",
+            "ranx",
+            lambda: [
+                float(es.mrr(rel, ret)),
+                float(es.mean_average_precision_at_k(rel, ret, k=20)),
+                float(es.ndcg_at_k(rel, ret, k=10)),
+            ],
+            None,
+        ),
+        (
+            f"agreement: Krippendorff's alpha, interval (4 raters × {m} items)",
+            "krippendorff",
+            lambda: [float(es.krippendorff_alpha(ratings, level="interval"))],
+            None,
+        ),
+        (
+            f"structured: JSON Schema compliance ({m} documents)",
+            "jsonschema",
+            lambda: [float(es.json_schema_compliance(docs_text, schema))],
+            None,
+        ),
+    ]
+    refs: dict[str, Callable[[], Any]] = {}
+    with contextlib.suppress(ImportError):
+        import sacrebleu
+
+        refs[cases[0][0]] = lambda: [
+            sacrebleu.corpus_bleu(preds, [refs_], force=True).score,
+            sacrebleu.corpus_chrf(preds, [refs_]).score,
+        ]
+    with contextlib.suppress(ImportError):
+        from rouge_score import rouge_scorer
+
+        def rouge_ref() -> list[float]:
+            sc = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"])
+            out = [sc.score(r, p) for r, p in zip(refs_, preds)]
+            return [float(np.mean([o[k].fmeasure for o in out])) for k in ("rouge1", "rouge2", "rougeL")]
+
+        refs[cases[1][0]] = rouge_ref
+    with contextlib.suppress(ImportError):
+        from nltk.translate.meteor_score import meteor_score
+
+        class _NoWordNet:
+            def synsets(self, _w: str) -> list[Any]:
+                return []
+
+        refs[cases[2][0]] = lambda: [
+            float(
+                np.mean([meteor_score([r.split()], p.split(), wordnet=_NoWordNet()) for r, p in zip(refs_, preds)])
+            )
+        ]
+    with contextlib.suppress(ImportError):
+        from ranx import Qrels, Run
+        from ranx import evaluate as ranx_evaluate
+
+        def ranx_ref() -> list[float]:
+            qrels = Qrels({f"q{i}": {f"d{d}": 1 for d in r} for i, r in enumerate(rel)})
+            run = Run({f"q{i}": {f"d{d}": float(len(r) - j) for j, d in enumerate(r)} for i, r in enumerate(ret)})
+            res = ranx_evaluate(qrels, run, ["mrr", "map@20", "ndcg@10"])
+            return [float(res["mrr"]), float(res["map@20"]), float(res["ndcg@10"])]
+
+        refs[cases[3][0]] = ranx_ref
+    with contextlib.suppress(ImportError):
+        import krippendorff
+
+        refs[cases[4][0]] = lambda: [
+            float(krippendorff.alpha(reliability_data=ratings, level_of_measurement="interval"))
+        ]
+    with contextlib.suppress(ImportError):
+        import jsonschema
+
+        def jsonschema_ref() -> list[float]:
+            v = jsonschema.Draft202012Validator(schema)
+            return [float(np.mean([v.is_valid(_json.loads(t)) for t in docs_text]))]
+
+        refs[cases[5][0]] = jsonschema_ref
+    return [(name, ref, es_fn, refs.get(name)) for name, ref, es_fn, _ in cases]
+
+
 def _clinical_cases(n: int, rng: np.random.Generator) -> list[Case]:
     """v0.2.0: diagnostic accuracy, calibration, decision curves and statistical tests."""
     import evalsuite as es
@@ -377,6 +526,7 @@ _GROUPS = (
     ("Classification and regression", ("binary", "10 classes", "regression")),
     ("Clinical, calibration and statistics", ("clinical", "calibration", "decision", "statistics", "multiple")),
     ("Segmentation and object detection", ("segmentation", "detection")),
+    ("LLM evaluation", ("text", "retrieval", "agreement", "structured")),
 )
 MATCH_TOLERANCE = 1e-9
 
@@ -419,7 +569,7 @@ class BenchmarkResult:
             speedups = [r["speedup"] for r in compared]
             return {
                 "group": name,
-                "cases": len({re.sub(r"\s*\(\d+ images?\)", "", r["case"]) for r in rows}),
+                "cases": len({re.sub(r"\s*\((?=[^)]*\d)[^)]*\)$", "", r["case"]) for r in rows}),
                 "rows": len(rows),
                 "compared": len(compared),
                 "matching": sum(
@@ -597,8 +747,8 @@ def run_benchmarks(
 
     if repeat < 1:
         raise ValueError("repeat must be at least 1.")
-    if suite not in ("all", "core", "clinical", "vision"):
-        raise ValueError("suite must be 'all', 'core', 'clinical' or 'vision'.")
+    if suite not in ("all", "core", "clinical", "vision", "llm"):
+        raise ValueError("suite must be 'all', 'core', 'clinical', 'vision' or 'llm'.")
     rng = np.random.default_rng(random_state)
     rows: list[dict[str, Any]] = []
     versions: dict[str, Optional[str]] = {"sklearn": None, "statsmodels": None, "pycocotools": None}
@@ -613,7 +763,8 @@ def run_benchmarks(
         "core": [_core_cases],
         "clinical": [_clinical_cases],
         "vision": [_vision_cases],
-        "all": [_core_cases, _clinical_cases, _vision_cases],
+        "llm": [_llm_cases],
+        "all": [_core_cases, _clinical_cases, _vision_cases, _llm_cases],
     }[suite]
     for n in sizes:
         for build in builders:

{evalsuite_python-0.3.1 → evalsuite_python-0.4.0}/src/evalsuite/cli/main.py

@@ -297,6 +297,24 @@ def cmd_detection(args: argparse.Namespace) -> int:
     return 0
 
 
+def cmd_text(args: argparse.Namespace) -> int:
+    import evalsuite as es
+
+    df = read_table(args.file)
+    preds = [str(v) for v in column(df, args.prediction, "--prediction")]
+    ref_cols = [column(df, c, "--reference") for c in args.reference]
+    refs = [[str(col[i]) for col in ref_cols if not _missing(col[i])] for i in range(len(preds))]
+    if any(not r for r in refs):
+        raise CLIError("Every row needs at least one non-empty reference.")
+    report = es.text_report(refs, preds, metrics=metric_list(args.metrics))
+    emit(report, args)
+    return 0
+
+
+def _missing(v: Any) -> bool:
+    return v is None or (isinstance(v, float) and v != v) or (isinstance(v, str) and not v.strip())
+
+
 def cmd_compare(args: argparse.Namespace) -> int:
     import evalsuite as es
 
@@ -506,6 +524,18 @@ def build_parser() -> argparse.ArgumentParser:
     add_output(p, digits=3)
     p.set_defaults(func=cmd_detection)
 
+    p = sub.add_parser(
+        "text", help="text generation: BLEU, chrF, chrF++, TER, ROUGE, METEOR, exact match, token F1"
+    )
+    p.add_argument("file", help="CSV, TSV, Parquet or JSON-lines file with one example per row ('-' for stdin)")
+    p.add_argument("--prediction", required=True, help="column with the model output")
+    p.add_argument(
+        "--reference", required=True, action="append", help="column with a reference (repeat for several)"
+    )
+    p.add_argument("--metrics", help="comma-separated subset, e.g. bleu,chrf,rouge_l")
+    add_output(p)
+    p.set_defaults(func=cmd_text)
+
     p = sub.add_parser(
         "plot", help="save a ROC, PR, calibration, decision-curve, confusion-matrix or residual plot"
     )
@@ -529,7 +559,7 @@ def build_parser() -> argparse.ArgumentParser:
     p.add_argument("--repeat", type=int, default=5, help="timed repetitions per case; the fastest is reported")
     p.add_argument(
         "--suite",
-        choices=("all", "core", "clinical", "vision"),
+        choices=("all", "core", "clinical", "vision", "llm"),
         default="all",
         help="core: classification/regression; clinical: clinical, calibration, tests; vision: segmentation, "
         "detection (default all)",