evalsuite-python 0.4.1__tar.gz → 0.5.1__tar.gz

{evalsuite_python-0.4.1 → evalsuite_python-0.5.1}/CHANGELOG.md

@@ -6,6 +6,56 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.5.1] - 2026-10-10
+
+### Added
+- `evalsuite benchmark --suite llmsys` (and `run_benchmarks(suite="llmsys")`): v0.5 LLM-systems workloads
+  against radon, the codebleu package, scikit-learn, jsonschema, SciPy and NumPy; included in `--suite all`
+  (27 cases, 81 measurements, all agreeing with their reference).
+
+### Changed
+- LLM-systems metrics accept NumPy arrays without converting them to Python lists first (much faster on
+  large inputs); `confidence_accuracy_correlation` computes Spearman from the ranks it already has.
+
+## [0.5.0] - 2026-10-10
+
+LLM systems (the v0.5.0 roadmap: 8 areas, 85 items) and SPICE. No existing API changed.
+
+### Added
+- `es.llmsys` (all functions also at top level): 77 new registered metrics. Items on the roadmap that an
+  existing function already covers (ECE, MCE, Brier, log loss, abstention, pass@k, BLEU/chrF/COMET,
+  ROUGE-L/BERTScore, tool-call precision/recall, Recall@k/MRR) are documented as such.
+  - Safety: harmful-response / unsafe-compliance, refusal and appropriate refusal, over-refusal, jailbreak /
+    prompt-injection attack success, red-team success@k, expected maximum toxicity and toxicity probability
+    (RealToxicityPrompts) with per-attribute scores, CrowS-Pairs stereotype preference, WEAT effect size with
+    a permutation test, PII leakage (built-in e-mail, phone, Luhn-checked card, IPv4, SSN detectors plus
+    protected strings; `detect_pii`), memorization exposure (Carlini et al.), policy violations.
+  - Robustness: adversarial and typo-noise robustness (`add_typos`), OOD accuracy, distribution-shift drop
+    with a Welch interval (= SciPy), paraphrase consistency, counterfactual invariance violations, response
+    stability, contradiction rate, failure / timeout / error rate, recovery rate, prompt sensitivity, accuracy
+    slope against context length.
+  - Uncertainty: adaptive calibration error, risk-coverage curve, AURC and E-AURC, selective risk, risk at
+    coverage, coverage at risk, confidence AUROC (= scikit-learn).
+  - Agents: task completion and progress, invalid tool calls checked against each tool's JSON Schema,
+    execution failures, tool-use efficiency, steps per task, plan adherence (LCS), joint goal and slot
+    accuracy, tool-failure recovery, repeated-call and loop rates, human intervention, cost per success.
+  - Multilingual: language-ID accuracy and macro-F1, bitext mining (cosine or margin), language parity,
+    code-switching robustness, z-normalised direct assessment, cultural appropriateness, language consistency,
+    cross-lingual consistency.
+  - Code: unit-test pass rate, syntax validity (parsed, never run), static-analysis violations per KLOC,
+    execution success, test coverage, patch acceptance and bug reproduction, SWE-bench resolved rate,
+    security findings by severity, EffiBench NET / NMU, CodeBLEU (n-gram terms identical to the `codebleu`
+    package; syntax and data-flow terms from Python's `ast`), cyclomatic complexity and maintainability index
+    (identical to radon on every module of EvalSuite and radon).
+  - Long context: accuracy by length with effective length (RULER), needle-in-a-haystack grid, accuracy by
+    position, lost-in-the-middle gap, context utilization, summary coverage and factual consistency,
+    compression ratio, citation coverage, cross-document consistency.
+  - Efficiency: TTFT, TPOT, latency percentiles, throughput, token usage, cost (with cached-token pricing),
+    resource utilization, energy per request, requests per second, availability and error budget.
+- `es.spice`: SPICE from scene-graph tuples or any `parser`, with exact, WordNet or custom synonym matching
+  and object / attribute / relation F-scores. Runs without Java.
+- Benchmarks: `suite="metrics"` covers all 225 metrics and statistics functions; new "LLM systems" group.
+
 ## [0.4.1] - 2026-10-09
 
 Benchmarks for every metric. No API changed.

{evalsuite_python-0.4.1 → evalsuite_python-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.4.1
+Version: 0.5.1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -55,6 +55,7 @@ Requires-Dist: pycocoevalcap>=1.2; extra == 'dev'
 Requires-Dist: pycocotools>=2.0.7; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: radon>=6; extra == 'dev'
 Requires-Dist: ranx>=0.3.19; (python_version < '3.14') and extra == 'dev'
 Requires-Dist: rouge-score>=0.1.2; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
@@ -82,9 +83,9 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
-> (except SPICE, which needs a Java scene-graph parser).
-> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
+> **Status: stable (0.5.1).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
+> multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
 ## Installation
 
@@ -254,6 +255,32 @@ es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_com
 es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
 es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
 es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+es.spice(reference_tuples, candidate_tuples)  # scene-graph F-score; parser= for captions, synonyms="wordnet"
+```
+
+## LLM systems
+
+Safety, robustness, uncertainty, agents, multilingual, code, long context and serving cost. The verdicts
+these take (harmful or not, test passed or not, confidence) come from your own judge, classifier or sandbox;
+EvalSuite turns them into rates with Wilson intervals and breakdowns. PII detection, CodeBLEU, cyclomatic
+complexity and the maintainability index (identical to radon) are computed from the text itself, and
+EvalSuite never executes generated code.
+
+```python
+import evalsuite as es
+
+es.harmful_response_rate([True, False, False, True], harmful_prompt=[True, True, False, True])
+es.over_refusal_rate(refused=[True, False, True], should_refuse=[True, False, False])
+es.pii_leakage_rate(["mail me at a@b.io", "nothing here"])  # e-mail, phone, Luhn-checked cards, IPs
+es.paraphrase_consistency([["Paris", "paris"], ["4", "5", "4"]])
+es.aurc(correct=[True, True, False, True], confidence=[0.9, 0.8, 0.7, 0.4])  # also selective_risk
+es.plan_adherence(plans=[["search", "read", "answer"]], executed=[["search", "answer"]])
+es.language_parity({"en": [1, 1, 0, 1], "sw": [1, 0, 0, 1]})
+es.codebleu(["def add(a, b):\n    return a + b"], ["def add(x, y):\n    return x + y"])
+es.code_complexity(["def f(x):\n    return 1 if x else 0"])  # cyclomatic complexity + maintainability
+es.needle_in_haystack(correct=[1, 0, 1, 1], context_lengths=[4000, 4000, 8000, 8000], depths=[0, 0.5, 0, 0.5])
+es.time_to_first_token(request_times=[0.0, 1.0], first_token_times=[0.21, 1.35])
+es.inference_cost([1200, 800], [300, 150], input_price=3.0, output_price=15.0)  # prices per 1M tokens
 ```
 
 ## Classification report
@@ -312,41 +339,47 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 
 ## Performance
 
-Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
-difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
-21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+Benchmarked against reference implementations on the same data (fastest of 5 runs; EvalSuite 0.5.1, Python 3.13,
+NumPy 2.5, Linux x86_64). **Overall: 81 of 81 rows agree with the reference** (largest
+difference 2.5e-14), 56 are faster, and the geometric-mean speed-up across all
+27 cases and three sizes is **1.89×**. At 1,000,000 samples (pixels for
 segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
 alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
-| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
-| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
-| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
-| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
-| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
-| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
-| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
-| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
-| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
-| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
-| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
-| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
-
-**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
-statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
-independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+| 10 classes: macro F1 | scikit-learn | 41.5 | 137.6 | **3.3×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 4.5 | 4.5 | **1.0×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 49.4 | 1056.8 | **21.4×** |
+| binary: ROC AUC | scikit-learn | 200.3 | 338.5 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 344.3 | 1541.7 | **4.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 33.5 | 9.6 | 0.29× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 137.9 | 438.1 | **3.2×** |
+| decision curve: 99 thresholds | NumPy loop | 172.1 | 236.2 | **1.4×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 1147.9 | 1351.0 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 91.9 | 98.4 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 32.0 | 22.6 | 0.71× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 248.6 | 897.0 | **3.6×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 51.6 | 289.5 | **5.6×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 31.9 | 23.8 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.3 | 0.3 | 0.95× |
+| statistics: Mann–Whitney U | SciPy | 390.3 | 425.8 | **1.1×** |
+| statistics: Welch t-test | SciPy | 37.7 | 16.3 | 0.43× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 159.4 | 371.7 | **2.3×** |
+| systems: CodeBLEU n-gram terms (1000 programs) | codebleu | 740.3 | 791.9 | **1.1×** |
+| systems: confidence AUROC (n answers) | scikit-learn | 194.4 | 327.7 | **1.7×** |
+| systems: distribution-shift drop, Welch CI (n scores) | SciPy | 13.3 | 16.9 | **1.3×** |
+| systems: invalid tool calls vs JSON Schema (10000 tasks) | jsonschema | 145.0 | 333.2 | **2.3×** |
+| systems: latency p50 / p95 / p99 (n requests) | NumPy | 41.0 | 19.7 | 0.48× |
+| systems: maintainability index (1000 programs) | radon | 138.3 | 168.7 | **1.2×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2726.2 | 3121.0 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 507.4 | 590.3 | **1.2×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1147.3 | 1010.7 | 0.88× |
+
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
+independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.67×** (139 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
@@ -387,6 +420,29 @@ object size, AP per class with COCO or VOC interpolation, precision-recall curve
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.
 
+**LLM evaluation** (0.4.0): BLEU, sentence BLEU, chrF/chrF++, TER, ROUGE-1/2/L/Lsum, METEOR, CIDEr-D, SPICE,
+perplexity, cross-entropy, Distinct-n, Self-BLEU, MAUVE, BERTScore, embedding similarity, MoverScore, learned
+metrics via `model_score`; faithfulness, hallucination rate, groundedness, citation precision/recall, claim
+verification, knowledge consistency, answer correctness/relevance, abstention, exact match, token F1; win
+rate, Bradley–Terry, Elo, Krippendorff's alpha, Fleiss' kappa, judge agreement and biases, rubric scores;
+pass@k, majority vote, benchmark accuracy; Precision/Recall/Hit rate@k, MRR, MAP, NDCG, context precision,
+recall and relevance; JSON / JSON Schema / XML validity, required fields, tool calls, instruction following.
+
+**LLM systems** (0.5.0): harmful-response, refusal, over-refusal, attack-success and red-team success rates,
+expected maximum toxicity, CrowS-Pairs stereotype preference, WEAT, PII leakage, memorization exposure,
+policy violations; adversarial, typo-noise, OOD and code-switching robustness, distribution-shift drop,
+paraphrase / counterfactual / cross-lingual consistency, response stability, contradiction rate, failure and
+recovery rates, prompt sensitivity, truncation sensitivity; adaptive calibration error, risk-coverage curve,
+AURC and E-AURC, selective risk, risk at coverage, coverage at risk, confidence AUROC; task completion,
+invalid tool calls, tool-use efficiency, steps per task, plan adherence, state tracking, tool-failure
+recovery, loop rate, human intervention, cost per task; language ID, bitext mining, language parity, direct
+assessment, cultural appropriateness, language consistency; unit-test pass rate, syntax validity, static
+analysis and security findings, execution success, test coverage, patch acceptance, SWE-bench resolved rate,
+CodeBLEU, cyclomatic complexity and maintainability index, runtime efficiency; long-context accuracy by
+length, needle-in-a-haystack grid, position accuracy, lost-in-the-middle gap, context utilization, summary
+coverage, compression ratio, citation coverage, cross-document consistency; TTFT, TPOT, latency percentiles,
+throughput, token usage, cost, resource utilization, energy per request, requests per second, availability.
+
 ## Conventions
 
 - `average="auto"` resolves to `"binary"` for binary targets and `"macro"` otherwise; the resolved value

{evalsuite_python-0.4.1 → evalsuite_python-0.5.1}/README.md

@@ -10,9 +10,9 @@
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
-> (except SPICE, which needs a Java scene-graph parser).
-> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
+> **Status: stable (0.5.1).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
+> multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
 ## Installation
 
@@ -182,6 +182,32 @@ es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_com
 es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
 es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
 es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+es.spice(reference_tuples, candidate_tuples)  # scene-graph F-score; parser= for captions, synonyms="wordnet"
+```
+
+## LLM systems
+
+Safety, robustness, uncertainty, agents, multilingual, code, long context and serving cost. The verdicts
+these take (harmful or not, test passed or not, confidence) come from your own judge, classifier or sandbox;
+EvalSuite turns them into rates with Wilson intervals and breakdowns. PII detection, CodeBLEU, cyclomatic
+complexity and the maintainability index (identical to radon) are computed from the text itself, and
+EvalSuite never executes generated code.
+
+```python
+import evalsuite as es
+
+es.harmful_response_rate([True, False, False, True], harmful_prompt=[True, True, False, True])
+es.over_refusal_rate(refused=[True, False, True], should_refuse=[True, False, False])
+es.pii_leakage_rate(["mail me at a@b.io", "nothing here"])  # e-mail, phone, Luhn-checked cards, IPs
+es.paraphrase_consistency([["Paris", "paris"], ["4", "5", "4"]])
+es.aurc(correct=[True, True, False, True], confidence=[0.9, 0.8, 0.7, 0.4])  # also selective_risk
+es.plan_adherence(plans=[["search", "read", "answer"]], executed=[["search", "answer"]])
+es.language_parity({"en": [1, 1, 0, 1], "sw": [1, 0, 0, 1]})
+es.codebleu(["def add(a, b):\n    return a + b"], ["def add(x, y):\n    return x + y"])
+es.code_complexity(["def f(x):\n    return 1 if x else 0"])  # cyclomatic complexity + maintainability
+es.needle_in_haystack(correct=[1, 0, 1, 1], context_lengths=[4000, 4000, 8000, 8000], depths=[0, 0.5, 0, 0.5])
+es.time_to_first_token(request_times=[0.0, 1.0], first_token_times=[0.21, 1.35])
+es.inference_cost([1200, 800], [300, 150], input_price=3.0, output_price=15.0)  # prices per 1M tokens
 ```
 
 ## Classification report
@@ -240,41 +266,47 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 
 ## Performance
 
-Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
-difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
-21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+Benchmarked against reference implementations on the same data (fastest of 5 runs; EvalSuite 0.5.1, Python 3.13,
+NumPy 2.5, Linux x86_64). **Overall: 81 of 81 rows agree with the reference** (largest
+difference 2.5e-14), 56 are faster, and the geometric-mean speed-up across all
+27 cases and three sizes is **1.89×**. At 1,000,000 samples (pixels for
 segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
 alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
-| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
-| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
-| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
-| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
-| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
-| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
-| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
-| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
-| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
-| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
-| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
-| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
-
-**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
-statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
-independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+| 10 classes: macro F1 | scikit-learn | 41.5 | 137.6 | **3.3×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 4.5 | 4.5 | **1.0×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 49.4 | 1056.8 | **21.4×** |
+| binary: ROC AUC | scikit-learn | 200.3 | 338.5 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 344.3 | 1541.7 | **4.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 33.5 | 9.6 | 0.29× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 137.9 | 438.1 | **3.2×** |
+| decision curve: 99 thresholds | NumPy loop | 172.1 | 236.2 | **1.4×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 1147.9 | 1351.0 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 91.9 | 98.4 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 32.0 | 22.6 | 0.71× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 248.6 | 897.0 | **3.6×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 51.6 | 289.5 | **5.6×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 31.9 | 23.8 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.3 | 0.3 | 0.95× |
+| statistics: Mann–Whitney U | SciPy | 390.3 | 425.8 | **1.1×** |
+| statistics: Welch t-test | SciPy | 37.7 | 16.3 | 0.43× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 159.4 | 371.7 | **2.3×** |
+| systems: CodeBLEU n-gram terms (1000 programs) | codebleu | 740.3 | 791.9 | **1.1×** |
+| systems: confidence AUROC (n answers) | scikit-learn | 194.4 | 327.7 | **1.7×** |
+| systems: distribution-shift drop, Welch CI (n scores) | SciPy | 13.3 | 16.9 | **1.3×** |
+| systems: invalid tool calls vs JSON Schema (10000 tasks) | jsonschema | 145.0 | 333.2 | **2.3×** |
+| systems: latency p50 / p95 / p99 (n requests) | NumPy | 41.0 | 19.7 | 0.48× |
+| systems: maintainability index (1000 programs) | radon | 138.3 | 168.7 | **1.2×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2726.2 | 3121.0 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 507.4 | 590.3 | **1.2×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1147.3 | 1010.7 | 0.88× |
+
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
+independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.67×** (139 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
@@ -315,6 +347,29 @@ object size, AP per class with COCO or VOC interpolation, precision-recall curve
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.
 
+**LLM evaluation** (0.4.0): BLEU, sentence BLEU, chrF/chrF++, TER, ROUGE-1/2/L/Lsum, METEOR, CIDEr-D, SPICE,
+perplexity, cross-entropy, Distinct-n, Self-BLEU, MAUVE, BERTScore, embedding similarity, MoverScore, learned
+metrics via `model_score`; faithfulness, hallucination rate, groundedness, citation precision/recall, claim
+verification, knowledge consistency, answer correctness/relevance, abstention, exact match, token F1; win
+rate, Bradley–Terry, Elo, Krippendorff's alpha, Fleiss' kappa, judge agreement and biases, rubric scores;
+pass@k, majority vote, benchmark accuracy; Precision/Recall/Hit rate@k, MRR, MAP, NDCG, context precision,
+recall and relevance; JSON / JSON Schema / XML validity, required fields, tool calls, instruction following.
+
+**LLM systems** (0.5.0): harmful-response, refusal, over-refusal, attack-success and red-team success rates,
+expected maximum toxicity, CrowS-Pairs stereotype preference, WEAT, PII leakage, memorization exposure,
+policy violations; adversarial, typo-noise, OOD and code-switching robustness, distribution-shift drop,
+paraphrase / counterfactual / cross-lingual consistency, response stability, contradiction rate, failure and
+recovery rates, prompt sensitivity, truncation sensitivity; adaptive calibration error, risk-coverage curve,
+AURC and E-AURC, selective risk, risk at coverage, coverage at risk, confidence AUROC; task completion,
+invalid tool calls, tool-use efficiency, steps per task, plan adherence, state tracking, tool-failure
+recovery, loop rate, human intervention, cost per task; language ID, bitext mining, language parity, direct
+assessment, cultural appropriateness, language consistency; unit-test pass rate, syntax validity, static
+analysis and security findings, execution success, test coverage, patch acceptance, SWE-bench resolved rate,
+CodeBLEU, cyclomatic complexity and maintainability index, runtime efficiency; long-context accuracy by
+length, needle-in-a-haystack grid, position accuracy, lost-in-the-middle gap, context utilization, summary
+coverage, compression ratio, citation coverage, cross-document consistency; TTFT, TPOT, latency percentiles,
+throughput, token usage, cost, resource utilization, energy per request, requests per second, availability.
+
 ## Conventions
 
 - `average="auto"` resolves to `"binary"` for binary targets and `"macro"` otherwise; the resolved value

{evalsuite_python-0.4.1 → evalsuite_python-0.5.1}/pyproject.toml

@@ -56,6 +56,7 @@ dev = [
   "jsonschema>=4.18",
   "choix>=0.3",
   "krippendorff>=0.6",
+  "radon>=6",
   "pot>=0.9; python_version < '3.14'",
   "ranx>=0.3.19; python_version < '3.14'",
   "ruff>=0.6",
@@ -111,7 +112,7 @@ ignore_missing_imports = true
 
 [[tool.mypy.overrides]]
 # optional reference libraries used only by the benchmarks and METEOR's default stemmer
-module = ["sacrebleu.*", "rouge_score.*", "nltk.*", "ranx.*", "krippendorff.*", "jsonschema.*", "pycocotools.*"]
+module = ["sacrebleu.*", "rouge_score.*", "nltk.*", "ranx.*", "krippendorff.*", "jsonschema.*", "pycocotools.*", "codebleu.*", "radon.*", "pycocoevalcap.*", "ot.*", "choix.*"]
 ignore_missing_imports = true
 follow_untyped_imports = false
 

{evalsuite_python-0.4.1 → evalsuite_python-0.5.1}/src/evalsuite/__init__.py

@@ -5,7 +5,7 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import calibration, classification, clinical, plot, regression, stats, text, vision
+from . import calibration, classification, clinical, llmsys, plot, regression, stats, text, vision
 from .api import evaluate
 from .calibration import (
     CalibrationReport,
@@ -63,6 +63,8 @@ from .core.exceptions import (
 )
 from .core.registry import MetricInfo, list_metrics, metric_info
 from .core.result import EvaluationResult, MetricResult
+from .llmsys import *  # noqa: F403  (v0.5.0 LLM-systems functions, listed in llmsys.__all__)
+from .llmsys import __all__ as _llmsys_all
 from .regression import (
     adjusted_r2,
     explained_variance,
@@ -137,6 +139,7 @@ from .vision import (
 
 __all__ = [
     "text",
+    "llmsys",
     "SegmentationReport",
     "segmentation_report",
     "detection_pr_curve",
@@ -262,3 +265,4 @@ __all__ = [
     "top_k_accuracy",
 ]
 __all__ += list(_text_all)
+__all__ += list(_llmsys_all)