evalsuite-python 0.4.1__tar.gz → 0.5.0__tar.gz

{evalsuite_python-0.4.1 → evalsuite_python-0.5.0}/CHANGELOG.md

@@ -6,6 +6,45 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-10-10
+
+LLM systems (the v0.5.0 roadmap: 8 areas, 85 items) and SPICE. No existing API changed.
+
+### Added
+- `es.llmsys` (all functions also at top level): 77 new registered metrics. Items on the roadmap that an
+  existing function already covers (ECE, MCE, Brier, log loss, abstention, pass@k, BLEU/chrF/COMET,
+  ROUGE-L/BERTScore, tool-call precision/recall, Recall@k/MRR) are documented as such.
+  - Safety: harmful-response / unsafe-compliance, refusal and appropriate refusal, over-refusal, jailbreak /
+    prompt-injection attack success, red-team success@k, expected maximum toxicity and toxicity probability
+    (RealToxicityPrompts) with per-attribute scores, CrowS-Pairs stereotype preference, WEAT effect size with
+    a permutation test, PII leakage (built-in e-mail, phone, Luhn-checked card, IPv4, SSN detectors plus
+    protected strings; `detect_pii`), memorization exposure (Carlini et al.), policy violations.
+  - Robustness: adversarial and typo-noise robustness (`add_typos`), OOD accuracy, distribution-shift drop
+    with a Welch interval (= SciPy), paraphrase consistency, counterfactual invariance violations, response
+    stability, contradiction rate, failure / timeout / error rate, recovery rate, prompt sensitivity, accuracy
+    slope against context length.
+  - Uncertainty: adaptive calibration error, risk-coverage curve, AURC and E-AURC, selective risk, risk at
+    coverage, coverage at risk, confidence AUROC (= scikit-learn).
+  - Agents: task completion and progress, invalid tool calls checked against each tool's JSON Schema,
+    execution failures, tool-use efficiency, steps per task, plan adherence (LCS), joint goal and slot
+    accuracy, tool-failure recovery, repeated-call and loop rates, human intervention, cost per success.
+  - Multilingual: language-ID accuracy and macro-F1, bitext mining (cosine or margin), language parity,
+    code-switching robustness, z-normalised direct assessment, cultural appropriateness, language consistency,
+    cross-lingual consistency.
+  - Code: unit-test pass rate, syntax validity (parsed, never run), static-analysis violations per KLOC,
+    execution success, test coverage, patch acceptance and bug reproduction, SWE-bench resolved rate,
+    security findings by severity, EffiBench NET / NMU, CodeBLEU (n-gram terms identical to the `codebleu`
+    package; syntax and data-flow terms from Python's `ast`), cyclomatic complexity and maintainability index
+    (identical to radon on every module of EvalSuite and radon).
+  - Long context: accuracy by length with effective length (RULER), needle-in-a-haystack grid, accuracy by
+    position, lost-in-the-middle gap, context utilization, summary coverage and factual consistency,
+    compression ratio, citation coverage, cross-document consistency.
+  - Efficiency: TTFT, TPOT, latency percentiles, throughput, token usage, cost (with cached-token pricing),
+    resource utilization, energy per request, requests per second, availability and error budget.
+- `es.spice`: SPICE from scene-graph tuples or any `parser`, with exact, WordNet or custom synonym matching
+  and object / attribute / relation F-scores. Runs without Java.
+- Benchmarks: `suite="metrics"` covers all 225 metrics and statistics functions; new "LLM systems" group.
+
 ## [0.4.1] - 2026-10-09
 
 Benchmarks for every metric. No API changed.

{evalsuite_python-0.4.1 → evalsuite_python-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.4.1
+Version: 0.5.0
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -55,6 +55,7 @@ Requires-Dist: pycocoevalcap>=1.2; extra == 'dev'
 Requires-Dist: pycocotools>=2.0.7; extra == 'dev'
 Requires-Dist: pytest-cov>=4; extra == 'dev'
 Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: radon>=6; extra == 'dev'
 Requires-Dist: ranx>=0.3.19; (python_version < '3.14') and extra == 'dev'
 Requires-Dist: rouge-score>=0.1.2; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
@@ -82,9 +83,9 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
-> (except SPICE, which needs a Java scene-graph parser).
-> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
+> **Status: stable (0.5.0).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
+> multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
 ## Installation
 
@@ -254,6 +255,32 @@ es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_com
 es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
 es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
 es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+es.spice(reference_tuples, candidate_tuples)  # scene-graph F-score; parser= for captions, synonyms="wordnet"
+```
+
+## LLM systems
+
+Safety, robustness, uncertainty, agents, multilingual, code, long context and serving cost. The verdicts
+these take (harmful or not, test passed or not, confidence) come from your own judge, classifier or sandbox;
+EvalSuite turns them into rates with Wilson intervals and breakdowns. PII detection, CodeBLEU, cyclomatic
+complexity and the maintainability index (identical to radon) are computed from the text itself, and
+EvalSuite never executes generated code.
+
+```python
+import evalsuite as es
+
+es.harmful_response_rate([True, False, False, True], harmful_prompt=[True, True, False, True])
+es.over_refusal_rate(refused=[True, False, True], should_refuse=[True, False, False])
+es.pii_leakage_rate(["mail me at a@b.io", "nothing here"])  # e-mail, phone, Luhn-checked cards, IPs
+es.paraphrase_consistency([["Paris", "paris"], ["4", "5", "4"]])
+es.aurc(correct=[True, True, False, True], confidence=[0.9, 0.8, 0.7, 0.4])  # also selective_risk
+es.plan_adherence(plans=[["search", "read", "answer"]], executed=[["search", "answer"]])
+es.language_parity({"en": [1, 1, 0, 1], "sw": [1, 0, 0, 1]})
+es.codebleu(["def add(a, b):\n    return a + b"], ["def add(x, y):\n    return x + y"])
+es.code_complexity(["def f(x):\n    return 1 if x else 0"])  # cyclomatic complexity + maintainability
+es.needle_in_haystack(correct=[1, 0, 1, 1], context_lengths=[4000, 4000, 8000, 8000], depths=[0, 0.5, 0, 0.5])
+es.time_to_first_token(request_times=[0.0, 1.0], first_token_times=[0.21, 1.35])
+es.inference_cost([1200, 800], [300, 150], input_price=3.0, output_price=15.0)  # prices per 1M tokens
 ```
 
 ## Classification report
@@ -343,10 +370,10 @@ alphabetical order:
 | text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
 | text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
-**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
-statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
-independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
+independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.65×** (137 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
@@ -387,6 +414,29 @@ object size, AP per class with COCO or VOC interpolation, precision-recall curve
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.
 
+**LLM evaluation** (0.4.0): BLEU, sentence BLEU, chrF/chrF++, TER, ROUGE-1/2/L/Lsum, METEOR, CIDEr-D, SPICE,
+perplexity, cross-entropy, Distinct-n, Self-BLEU, MAUVE, BERTScore, embedding similarity, MoverScore, learned
+metrics via `model_score`; faithfulness, hallucination rate, groundedness, citation precision/recall, claim
+verification, knowledge consistency, answer correctness/relevance, abstention, exact match, token F1; win
+rate, Bradley–Terry, Elo, Krippendorff's alpha, Fleiss' kappa, judge agreement and biases, rubric scores;
+pass@k, majority vote, benchmark accuracy; Precision/Recall/Hit rate@k, MRR, MAP, NDCG, context precision,
+recall and relevance; JSON / JSON Schema / XML validity, required fields, tool calls, instruction following.
+
+**LLM systems** (0.5.0): harmful-response, refusal, over-refusal, attack-success and red-team success rates,
+expected maximum toxicity, CrowS-Pairs stereotype preference, WEAT, PII leakage, memorization exposure,
+policy violations; adversarial, typo-noise, OOD and code-switching robustness, distribution-shift drop,
+paraphrase / counterfactual / cross-lingual consistency, response stability, contradiction rate, failure and
+recovery rates, prompt sensitivity, truncation sensitivity; adaptive calibration error, risk-coverage curve,
+AURC and E-AURC, selective risk, risk at coverage, coverage at risk, confidence AUROC; task completion,
+invalid tool calls, tool-use efficiency, steps per task, plan adherence, state tracking, tool-failure
+recovery, loop rate, human intervention, cost per task; language ID, bitext mining, language parity, direct
+assessment, cultural appropriateness, language consistency; unit-test pass rate, syntax validity, static
+analysis and security findings, execution success, test coverage, patch acceptance, SWE-bench resolved rate,
+CodeBLEU, cyclomatic complexity and maintainability index, runtime efficiency; long-context accuracy by
+length, needle-in-a-haystack grid, position accuracy, lost-in-the-middle gap, context utilization, summary
+coverage, compression ratio, citation coverage, cross-document consistency; TTFT, TPOT, latency percentiles,
+throughput, token usage, cost, resource utilization, energy per request, requests per second, availability.
+
 ## Conventions
 
 - `average="auto"` resolves to `"binary"` for binary targets and `"macro"` otherwise; the resolved value

{evalsuite_python-0.4.1 → evalsuite_python-0.5.0}/README.md

@@ -10,9 +10,9 @@
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.4.1).** Every item on the 0.1.0, 0.2.0, 0.3.0 and 0.4.0 roadmaps is implemented and verified
-> (except SPICE, which needs a Java scene-graph parser).
-> **LLM evaluation is available from 0.4.0**; LLM systems, safety and operations are planned for 0.5.0.
+> **Status: stable (0.5.0).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
+> multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
 ## Installation
 
@@ -182,6 +182,32 @@ es.json_schema_compliance(outputs, schema)  # also tool_call_f1, instruction_com
 es.pass_at_k(n_samples, n_correct, k=10)  # also benchmark_accuracy(..., style="gsm8k")
 es.compare(references, {"a": preds_a, "b": preds_b}, metrics=["bleu", "rouge_l"])  # paired over examples
 es.plot.ratings(comparisons)  # Bradley–Terry leaderboard with intervals; also win_matrix, text_scores
+es.spice(reference_tuples, candidate_tuples)  # scene-graph F-score; parser= for captions, synonyms="wordnet"
+```
+
+## LLM systems
+
+Safety, robustness, uncertainty, agents, multilingual, code, long context and serving cost. The verdicts
+these take (harmful or not, test passed or not, confidence) come from your own judge, classifier or sandbox;
+EvalSuite turns them into rates with Wilson intervals and breakdowns. PII detection, CodeBLEU, cyclomatic
+complexity and the maintainability index (identical to radon) are computed from the text itself, and
+EvalSuite never executes generated code.
+
+```python
+import evalsuite as es
+
+es.harmful_response_rate([True, False, False, True], harmful_prompt=[True, True, False, True])
+es.over_refusal_rate(refused=[True, False, True], should_refuse=[True, False, False])
+es.pii_leakage_rate(["mail me at a@b.io", "nothing here"])  # e-mail, phone, Luhn-checked cards, IPs
+es.paraphrase_consistency([["Paris", "paris"], ["4", "5", "4"]])
+es.aurc(correct=[True, True, False, True], confidence=[0.9, 0.8, 0.7, 0.4])  # also selective_risk
+es.plan_adherence(plans=[["search", "read", "answer"]], executed=[["search", "answer"]])
+es.language_parity({"en": [1, 1, 0, 1], "sw": [1, 0, 0, 1]})
+es.codebleu(["def add(a, b):\n    return a + b"], ["def add(x, y):\n    return x + y"])
+es.code_complexity(["def f(x):\n    return 1 if x else 0"])  # cyclomatic complexity + maintainability
+es.needle_in_haystack(correct=[1, 0, 1, 1], context_lengths=[4000, 4000, 8000, 8000], depths=[0, 0.5, 0, 0.5])
+es.time_to_first_token(request_times=[0.0, 1.0], first_token_times=[0.21, 1.35])
+es.inference_cost([1200, 800], [300, 150], input_price=3.0, output_price=15.0)  # prices per 1M tokens
 ```
 
 ## Classification report
@@ -271,10 +297,10 @@ alphabetical order:
 | text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
 | text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
 
-**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 147 registered metrics and
-statistics functions at 10,000 and 100,000 samples. 79 are compared with a reference library, 49 with an
-independent textbook formula and 19 (learned, judge-dependent or randomised) are timed alone; **all 128
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.76×** (123 of 158
+**Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
+statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
+independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.65×** (137 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most
@@ -315,6 +341,29 @@ object size, AP per class with COCO or VOC interpolation, precision-recall curve
 MSLE, RMSLE, median absolute error, explained variance, max error, mean bias error, quantile (pinball)
 loss, Huber loss, relative absolute error, relative squared error.
 
+**LLM evaluation** (0.4.0): BLEU, sentence BLEU, chrF/chrF++, TER, ROUGE-1/2/L/Lsum, METEOR, CIDEr-D, SPICE,
+perplexity, cross-entropy, Distinct-n, Self-BLEU, MAUVE, BERTScore, embedding similarity, MoverScore, learned
+metrics via `model_score`; faithfulness, hallucination rate, groundedness, citation precision/recall, claim
+verification, knowledge consistency, answer correctness/relevance, abstention, exact match, token F1; win
+rate, Bradley–Terry, Elo, Krippendorff's alpha, Fleiss' kappa, judge agreement and biases, rubric scores;
+pass@k, majority vote, benchmark accuracy; Precision/Recall/Hit rate@k, MRR, MAP, NDCG, context precision,
+recall and relevance; JSON / JSON Schema / XML validity, required fields, tool calls, instruction following.
+
+**LLM systems** (0.5.0): harmful-response, refusal, over-refusal, attack-success and red-team success rates,
+expected maximum toxicity, CrowS-Pairs stereotype preference, WEAT, PII leakage, memorization exposure,
+policy violations; adversarial, typo-noise, OOD and code-switching robustness, distribution-shift drop,
+paraphrase / counterfactual / cross-lingual consistency, response stability, contradiction rate, failure and
+recovery rates, prompt sensitivity, truncation sensitivity; adaptive calibration error, risk-coverage curve,
+AURC and E-AURC, selective risk, risk at coverage, coverage at risk, confidence AUROC; task completion,
+invalid tool calls, tool-use efficiency, steps per task, plan adherence, state tracking, tool-failure
+recovery, loop rate, human intervention, cost per task; language ID, bitext mining, language parity, direct
+assessment, cultural appropriateness, language consistency; unit-test pass rate, syntax validity, static
+analysis and security findings, execution success, test coverage, patch acceptance, SWE-bench resolved rate,
+CodeBLEU, cyclomatic complexity and maintainability index, runtime efficiency; long-context accuracy by
+length, needle-in-a-haystack grid, position accuracy, lost-in-the-middle gap, context utilization, summary
+coverage, compression ratio, citation coverage, cross-document consistency; TTFT, TPOT, latency percentiles,
+throughput, token usage, cost, resource utilization, energy per request, requests per second, availability.
+
 ## Conventions
 
 - `average="auto"` resolves to `"binary"` for binary targets and `"macro"` otherwise; the resolved value

{evalsuite_python-0.4.1 → evalsuite_python-0.5.0}/pyproject.toml

@@ -56,6 +56,7 @@ dev = [
   "jsonschema>=4.18",
   "choix>=0.3",
   "krippendorff>=0.6",
+  "radon>=6",
   "pot>=0.9; python_version < '3.14'",
   "ranx>=0.3.19; python_version < '3.14'",
   "ruff>=0.6",

{evalsuite_python-0.4.1 → evalsuite_python-0.5.0}/src/evalsuite/__init__.py

@@ -5,7 +5,7 @@
 >>> print(result.summary())  # doctest: +SKIP
 """
 
-from . import calibration, classification, clinical, plot, regression, stats, text, vision
+from . import calibration, classification, clinical, llmsys, plot, regression, stats, text, vision
 from .api import evaluate
 from .calibration import (
     CalibrationReport,
@@ -63,6 +63,8 @@ from .core.exceptions import (
 )
 from .core.registry import MetricInfo, list_metrics, metric_info
 from .core.result import EvaluationResult, MetricResult
+from .llmsys import *  # noqa: F403  (v0.5.0 LLM-systems functions, listed in llmsys.__all__)
+from .llmsys import __all__ as _llmsys_all
 from .regression import (
     adjusted_r2,
     explained_variance,
@@ -137,6 +139,7 @@ from .vision import (
 
 __all__ = [
     "text",
+    "llmsys",
     "SegmentationReport",
     "segmentation_report",
     "detection_pr_curve",
@@ -262,3 +265,4 @@ __all__ = [
     "top_k_accuracy",
 ]
 __all__ += list(_text_all)
+__all__ += list(_llmsys_all)