evalsuite-python 0.5.0__tar.gz → 0.5.1__tar.gz

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/CHANGELOG.md

@@ -6,6 +6,17 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.5.1] - 2026-10-10
+
+### Added
+- `evalsuite benchmark --suite llmsys` (and `run_benchmarks(suite="llmsys")`): v0.5 LLM-systems workloads
+  against radon, the codebleu package, scikit-learn, jsonschema, SciPy and NumPy; included in `--suite all`
+  (27 cases, 81 measurements, all agreeing with their reference).
+
+### Changed
+- LLM-systems metrics accept NumPy arrays without converting them to Python lists first (much faster on
+  large inputs); `confidence_accuracy_correlation` computes Spearman from the ranks it already has.
+
 ## [0.5.0] - 2026-10-10
 
 LLM systems (the v0.5.0 roadmap: 8 areas, 85 items) and SPICE. No existing API changed.

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: evalsuite-python
-Version: 0.5.0
+Version: 0.5.1
 Summary: A unified Python framework for machine-learning, clinical, statistical, segmentation, object-detection, uncertainty, and model evaluation.
 Project-URL: Homepage, https://evalsuite-nine.vercel.app
 Project-URL: Documentation, https://evalsuite-nine.vercel.app/docs
@@ -83,7 +83,7 @@ Description-Content-Type: text/markdown
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.5.0).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **Status: stable (0.5.1).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
 > **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
 > multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
@@ -339,41 +339,47 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 
 ## Performance
 
-Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
-difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
-21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+Benchmarked against reference implementations on the same data (fastest of 5 runs; EvalSuite 0.5.1, Python 3.13,
+NumPy 2.5, Linux x86_64). **Overall: 81 of 81 rows agree with the reference** (largest
+difference 2.5e-14), 56 are faster, and the geometric-mean speed-up across all
+27 cases and three sizes is **1.89×**. At 1,000,000 samples (pixels for
 segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
 alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
-| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
-| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
-| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
-| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
-| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
-| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
-| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
-| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
-| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
-| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
-| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
-| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
+| 10 classes: macro F1 | scikit-learn | 41.5 | 137.6 | **3.3×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 4.5 | 4.5 | **1.0×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 49.4 | 1056.8 | **21.4×** |
+| binary: ROC AUC | scikit-learn | 200.3 | 338.5 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 344.3 | 1541.7 | **4.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 33.5 | 9.6 | 0.29× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 137.9 | 438.1 | **3.2×** |
+| decision curve: 99 thresholds | NumPy loop | 172.1 | 236.2 | **1.4×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 1147.9 | 1351.0 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 91.9 | 98.4 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 32.0 | 22.6 | 0.71× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 248.6 | 897.0 | **3.6×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 51.6 | 289.5 | **5.6×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 31.9 | 23.8 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.3 | 0.3 | 0.95× |
+| statistics: Mann–Whitney U | SciPy | 390.3 | 425.8 | **1.1×** |
+| statistics: Welch t-test | SciPy | 37.7 | 16.3 | 0.43× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 159.4 | 371.7 | **2.3×** |
+| systems: CodeBLEU n-gram terms (1000 programs) | codebleu | 740.3 | 791.9 | **1.1×** |
+| systems: confidence AUROC (n answers) | scikit-learn | 194.4 | 327.7 | **1.7×** |
+| systems: distribution-shift drop, Welch CI (n scores) | SciPy | 13.3 | 16.9 | **1.3×** |
+| systems: invalid tool calls vs JSON Schema (10000 tasks) | jsonschema | 145.0 | 333.2 | **2.3×** |
+| systems: latency p50 / p95 / p99 (n requests) | NumPy | 41.0 | 19.7 | 0.48× |
+| systems: maintainability index (1000 programs) | radon | 138.3 | 168.7 | **1.2×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2726.2 | 3121.0 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 507.4 | 590.3 | **1.2×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1147.3 | 1010.7 | 0.88× |
 
 **Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
 statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
 independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.65×** (137 of 172
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.67×** (139 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/README.md

@@ -10,7 +10,7 @@
 EvalSuite brings classification, regression, clinical, statistical, segmentation and object-detection
 evaluation into one consistent, validated, documented framework.
 
-> **Status: stable (0.5.0).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
+> **Status: stable (0.5.1).** Every item on the 0.1.0–0.5.0 roadmaps is implemented and verified, including SPICE.
 > **LLM systems arrive in 0.5.0**: safety, robustness, calibration and uncertainty, agents and tool use,
 > multilingual, code generation, long context and summarization, and inference efficiency and cost.
 
@@ -266,41 +266,47 @@ Input files can be CSV, TSV, Parquet or JSON. Output format follows `--format` o
 
 ## Performance
 
-Benchmarked against reference implementations on the same data (fastest of 5 runs; Python 3.12, NumPy 2.5,
-Linux x86_64). **Overall: 63 of 63 rows agree with the reference** (largest
-difference 1.8e-14), 42 are faster, and the geometric-mean speed-up across all
-21 cases and three sizes is **2.12×**. At 1,000,000 samples (pixels for
+Benchmarked against reference implementations on the same data (fastest of 5 runs; EvalSuite 0.5.1, Python 3.13,
+NumPy 2.5, Linux x86_64). **Overall: 81 of 81 rows agree with the reference** (largest
+difference 2.5e-14), 56 are faster, and the geometric-mean speed-up across all
+27 cases and three sizes is **1.89×**. At 1,000,000 samples (pixels for
 segmentation; 1,000 images for detection and 50 for Hausdorff; 10,000 examples for the LLM cases), in
 alphabetical order:
 
 | Case | Reference | EvalSuite (ms) | Reference (ms) | Speed-up |
 | --- | --- | ---: | ---: | ---: |
-| 10 classes: macro F1 | scikit-learn | 23.8 | 135.5 | **5.7×** |
-| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 3.5 | 3.4 | 0.96× |
-| binary: 8 label metrics via evaluate() | scikit-learn | 28.5 | 1023.8 | **36.0×** |
-| binary: ROC AUC | scikit-learn | 187.4 | 327.3 | **1.7×** |
-| calibration: slope and intercept | statsmodels | 205.7 | 1173.7 | **5.7×** |
-| clinical: diagnostic report (7 CIs) | statsmodels | 19.6 | 4.9 | 0.25× |
-| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 75.2 | 411.5 | **5.5×** |
-| decision curve: 99 thresholds | NumPy loop | 165.0 | 201.9 | **1.2×** |
-| detection: COCO evaluation (1000 images) | pycocotools | 838.7 | 848.6 | **1.0×** |
-| multiple testing: Hochberg (n p-values) | statsmodels | 72.1 | 79.0 | **1.1×** |
-| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 22.9 | 10.6 | 0.47× |
-| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 170.4 | 861.0 | **5.1×** |
-| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 34.3 | 256.0 | **7.5×** |
-| segmentation: Hausdorff distance (50 images) | SciPy | 24.2 | 19.8 | 0.82× |
-| statistics: Cramér's V (5×5 table) | SciPy | 0.2 | 0.2 | **1.0×** |
-| statistics: Mann–Whitney U | SciPy | 353.5 | 340.6 | 0.96× |
-| statistics: Welch t-test | SciPy | 16.1 | 8.3 | 0.52× |
-| structured: JSON Schema compliance (10000 documents) | jsonschema | 144.6 | 330.5 | **2.3×** |
-| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2789.8 | 3051.8 | **1.1×** |
-| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 488.1 | 626.7 | **1.3×** |
-| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1290.1 | 1095.6 | 0.85× |
+| 10 classes: macro F1 | scikit-learn | 41.5 | 137.6 | **3.3×** |
+| agreement: Krippendorff's alpha, interval (4 raters × 10000 items) | krippendorff | 4.5 | 4.5 | **1.0×** |
+| binary: 8 label metrics via evaluate() | scikit-learn | 49.4 | 1056.8 | **21.4×** |
+| binary: ROC AUC | scikit-learn | 200.3 | 338.5 | **1.7×** |
+| calibration: slope and intercept | statsmodels | 344.3 | 1541.7 | **4.5×** |
+| clinical: diagnostic report (7 CIs) | statsmodels | 33.5 | 9.6 | 0.29× |
+| clinical: sensitivity, specificity, LR+, LR− | scikit-learn | 137.9 | 438.1 | **3.2×** |
+| decision curve: 99 thresholds | NumPy loop | 172.1 | 236.2 | **1.4×** |
+| detection: COCO evaluation (1000 images) | pycocotools | 1147.9 | 1351.0 | **1.2×** |
+| multiple testing: Hochberg (n p-values) | statsmodels | 91.9 | 98.4 | **1.1×** |
+| regression: MAE, MSE, RMSE, R² via evaluate() | scikit-learn | 32.0 | 22.6 | 0.71× |
+| retrieval: MRR, MAP@20, NDCG@10 (10000 queries) | ranx | 248.6 | 897.0 | **3.6×** |
+| segmentation: Dice and IoU per class (n = pixels) | scikit-learn | 51.6 | 289.5 | **5.6×** |
+| segmentation: Hausdorff distance (50 images) | SciPy | 31.9 | 23.8 | 0.75× |
+| statistics: Cramér's V (5×5 table) | SciPy | 0.3 | 0.3 | 0.95× |
+| statistics: Mann–Whitney U | SciPy | 390.3 | 425.8 | **1.1×** |
+| statistics: Welch t-test | SciPy | 37.7 | 16.3 | 0.43× |
+| structured: JSON Schema compliance (10000 documents) | jsonschema | 159.4 | 371.7 | **2.3×** |
+| systems: CodeBLEU n-gram terms (1000 programs) | codebleu | 740.3 | 791.9 | **1.1×** |
+| systems: confidence AUROC (n answers) | scikit-learn | 194.4 | 327.7 | **1.7×** |
+| systems: distribution-shift drop, Welch CI (n scores) | SciPy | 13.3 | 16.9 | **1.3×** |
+| systems: invalid tool calls vs JSON Schema (10000 tasks) | jsonschema | 145.0 | 333.2 | **2.3×** |
+| systems: latency p50 / p95 / p99 (n requests) | NumPy | 41.0 | 19.7 | 0.48× |
+| systems: maintainability index (1000 programs) | radon | 138.3 | 168.7 | **1.2×** |
+| text: corpus BLEU and chrF (10000 sentences) | sacreBLEU | 2726.2 | 3121.0 | **1.1×** |
+| text: METEOR, exact and stem matches (10000 sentences) | NLTK | 507.4 | 590.3 | **1.2×** |
+| text: ROUGE-1, ROUGE-2, ROUGE-L (10000 sentences) | rouge-score | 1147.3 | 1010.7 | 0.88× |
 
 **Every metric is benchmarked too** (`evalsuite benchmark --suite metrics`): all 225 registered metrics and
 statistics functions at 10,000 and 100,000 samples. 86 are compared with a reference library, 118 with an
 independent textbook formula and 21 (learned, judge-dependent or randomised) are timed alone; **all 204
-comparisons agree**, and against the libraries the geometric-mean speed-up is **2.65×** (137 of 172
+comparisons agree**, and against the libraries the geometric-mean speed-up is **2.67×** (139 of 172
 measurements faster). Per-metric table in BENCHMARKS.md.
 
 `evaluate()` validates inputs once and builds the confusion matrix once for all metrics, which is where most

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/pyproject.toml

@@ -112,7 +112,7 @@ ignore_missing_imports = true
 
 [[tool.mypy.overrides]]
 # optional reference libraries used only by the benchmarks and METEOR's default stemmer
-module = ["sacrebleu.*", "rouge_score.*", "nltk.*", "ranx.*", "krippendorff.*", "jsonschema.*", "pycocotools.*"]
+module = ["sacrebleu.*", "rouge_score.*", "nltk.*", "ranx.*", "krippendorff.*", "jsonschema.*", "pycocotools.*", "codebleu.*", "radon.*", "pycocoevalcap.*", "ot.*", "choix.*"]
 ignore_missing_imports = true
 follow_untyped_imports = false
 

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/_bench_metrics.py

@@ -903,7 +903,7 @@ def _llm(c: _Cases, n: int, rng: np.random.Generator) -> None:
 
     cider_ref: Optional[Callable[[], float]] = None
     with contextlib.suppress(ImportError):
-        from pycocoevalcap.cider.cider import Cider  # type: ignore[import-untyped]
+        from pycocoevalcap.cider.cider import Cider
 
         def _cider() -> float:
             res = {i: [p] for i, p in enumerate(preds)}

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/benchmarks.py

@@ -524,6 +524,117 @@ def _clinical_cases(n: int, rng: np.random.Generator) -> list[Case]:
     return [(name, ref, es_fn, own if own is not None else refs.get(name)) for name, ref, es_fn, own in cases]
 
 
+def _llmsys_cases(n: int, rng: np.random.Generator) -> list[Case]:
+    """v0.5.0: LLM systems workloads (code quality, selective prediction, agent tool calls, shift, latency)."""
+    import json as _json
+
+    import evalsuite as es
+
+    from ._bench_metrics import _CODE_SAMPLES
+
+    m = max(10, n // 100)
+    progs = [str(rng.choice(np.array(_CODE_SAMPLES))) for _ in range(max(4, m // 10))]
+    refs_c = progs
+    preds_c = [p.replace("a", "x").replace("s", "t") for p in progs]
+    conf = rng.uniform(0, 1, n)
+    correct = rng.random(n) < conf
+    tools = {
+        t: {"type": "object", "properties": {"q": {"type": "string"}, "k": {"type": "integer"}}, "required": ["q"]}
+        for t in ("search", "calc", "email")
+    }
+    trajs = [
+        [
+            {
+                "name": str(rng.choice(["search", "calc", "email", "bad"])),
+                "arguments": {"q": "x"} if rng.random() > 0.1 else {"k": 1},
+            }
+            for _ in range(int(rng.integers(1, 6)))
+        ]
+        for _ in range(m)
+    ]
+    src, tgt = rng.normal(0.8, 0.1, n), rng.normal(0.75, 0.12, n)
+    lat = rng.lognormal(0, 0.5, n)
+
+    def es_shift() -> list[float]:
+        r = es.distribution_shift_drop(src, tgt)
+        return [r.params["ci_low"], r.params["ci_high"]]
+
+    def es_lat() -> list[float]:
+        r = es.latency_percentiles(lat)
+        return [r.params["p50"], r.params["p95"], r.params["p99"]]
+
+    cases: list[Case] = [
+        (
+            f"systems: maintainability index ({len(progs)} programs)",
+            "radon",
+            lambda: [float(es.code_complexity(progs).params["maintainability_index"])],
+            None,
+        ),
+        (
+            f"systems: CodeBLEU n-gram terms ({len(progs)} programs)",
+            "codebleu",
+            lambda: [es.codebleu(refs_c, preds_c).params[k] for k in ("ngram_match", "weighted_ngram_match")],
+            None,
+        ),
+        (
+            "systems: confidence AUROC (n answers)",
+            "scikit-learn",
+            lambda: [float(es.confidence_accuracy_correlation(correct, conf))],
+            None,
+        ),
+        (
+            f"systems: invalid tool calls vs JSON Schema ({m} tasks)",
+            "jsonschema",
+            lambda: [float(es.invalid_tool_call_rate(trajs, tools))],
+            None,
+        ),
+        ("systems: distribution-shift drop, Welch CI (n scores)", "SciPy", es_shift, None),
+        (
+            "systems: latency p50 / p95 / p99 (n requests)",
+            "NumPy",
+            es_lat,
+            lambda: np.percentile(lat, [50, 95, 99]).tolist(),
+        ),
+    ]
+    refs: dict[str, Callable[[], Any]] = {}
+    with contextlib.suppress(ImportError):
+        from radon.metrics import mi_visit
+
+        refs[cases[0][0]] = lambda: [float(np.mean([mi_visit(p, True) for p in progs]))]
+    with contextlib.suppress(Exception):
+        from codebleu import calc_codebleu
+
+        def cb() -> list[float]:
+            r = calc_codebleu([[x] for x in refs_c], preds_c, lang="python")
+            return [r["ngram_match_score"], r["weighted_ngram_match_score"]]
+
+        cb()
+        refs[cases[1][0]] = cb
+    with contextlib.suppress(ImportError):
+        import sklearn.metrics as skm
+
+        refs[cases[2][0]] = lambda: [float(skm.roc_auc_score(correct, conf))]
+    with contextlib.suppress(ImportError):
+        import jsonschema
+
+        def js_ref() -> list[float]:
+            flags = []
+            for traj in trajs:
+                for c in traj:
+                    sch = tools.get(str(c["name"]))
+                    flags.append(sch is None or not jsonschema.Draft202012Validator(sch).is_valid(c["arguments"]))
+            return [float(np.mean(flags))]
+
+        refs[cases[3][0]] = js_ref
+    from scipy import stats as _st
+
+    res = _st.ttest_ind(src[:3], tgt[:3], equal_var=False)
+    if hasattr(res, "confidence_interval"):
+        refs[cases[4][0]] = lambda: list(_st.ttest_ind(src, tgt, equal_var=False).confidence_interval(0.95))
+    del _json
+    return [(name, ref, es_fn, own if own is not None else refs.get(name)) for name, ref, es_fn, own in cases]
+
+
 _GROUPS = (
     ("Classification and regression", ("binary", "10 classes", "regression", "classification")),
     ("Clinical, calibration and statistics", ("clinical", "calibration", "decision", "statistics", "multiple")),
@@ -532,6 +643,7 @@ _GROUPS = (
     (
         "LLM systems",
         (
+            "systems:",
             "safety.",
             "robustness.",
             "uncertainty.",
@@ -765,8 +877,8 @@ def run_benchmarks(
 
     if repeat < 1:
         raise ValueError("repeat must be at least 1.")
-    if suite not in ("all", "core", "clinical", "vision", "llm", "metrics"):
-        raise ValueError("suite must be 'all', 'core', 'clinical', 'vision', 'llm' or 'metrics'.")
+    if suite not in ("all", "core", "clinical", "vision", "llm", "llmsys", "metrics"):
+        raise ValueError("suite must be 'all', 'core', 'clinical', 'vision', 'llm', 'llmsys' or 'metrics'.")
     rng = np.random.default_rng(random_state)
     rows: list[dict[str, Any]] = []
     versions: dict[str, Optional[str]] = {"sklearn": None, "statsmodels": None, "pycocotools": None}
@@ -782,7 +894,8 @@ def run_benchmarks(
         "clinical": [_clinical_cases],
         "vision": [_vision_cases],
         "llm": [_llm_cases],
-        "all": [_core_cases, _clinical_cases, _vision_cases, _llm_cases],
+        "llmsys": [_llmsys_cases],
+        "all": [_core_cases, _clinical_cases, _vision_cases, _llm_cases, _llmsys_cases],
         "metrics": [metric_cases],
     }[suite]
     for n in sizes:

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/cli/main.py

@@ -559,11 +559,11 @@ def build_parser() -> argparse.ArgumentParser:
     p.add_argument("--repeat", type=int, default=5, help="timed repetitions per case; the fastest is reported")
     p.add_argument(
         "--suite",
-        choices=("all", "core", "clinical", "vision", "llm", "metrics"),
+        choices=("all", "core", "clinical", "vision", "llm", "llmsys", "metrics"),
         default="all",
         help="core: classification/regression; clinical: clinical, calibration, tests; vision: segmentation, "
-        "detection; llm: text, retrieval, agreement, structured output; metrics: one row per metric and "
-        "statistics function (default all)",
+        "detection; llm: text, retrieval, agreement, structured output; llmsys: v0.5.0 LLM systems; "
+        "metrics: one row per metric and statistics function (default all)",
     )
     p.add_argument("--no-sklearn", action="store_true", help="time EvalSuite only (skip reference libraries)")
     p.add_argument("--seed", type=int, default=0)

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/llmsys/_common.py

@@ -31,6 +31,14 @@ def seq(x: Any, name: str) -> list[Any]:
 
 def bools(x: Any, name: str) -> np.ndarray:
     """A 1-D boolean array from booleans or 0/1 values (anything else is rejected, never coerced)."""
+    if isinstance(x, np.ndarray) and x.ndim == 1 and x.size and x.dtype.kind in "biuf":
+        if x.dtype.kind == "b":
+            return x.astype(bool, copy=False)
+        bad = ~np.isin(x, (0, 1))
+        if bad.any():
+            i = int(np.flatnonzero(bad)[0])
+            raise InputValidationError(f"{name}[{i}] must be a boolean (or 0/1); got {x[i]!r}.")
+        return x.astype(bool)
     items = seq(x, name)
     out = np.empty(len(items), dtype=bool)
     for i, v in enumerate(items):
@@ -45,7 +53,7 @@ def bools(x: Any, name: str) -> np.ndarray:
 
 def floats(x: Any, name: str, *, lo: Optional[float] = None, hi: Optional[float] = None) -> np.ndarray:
     """A finite 1-D float array, optionally bounded."""
-    items = seq(x, name)
+    items: Any = x if isinstance(x, np.ndarray) and x.size and x.dtype.kind in "biuf" else seq(x, name)
     try:
         a = np.asarray(items, dtype=np.float64)
     except (TypeError, ValueError) as exc:

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/llmsys/uncertainty.py

@@ -215,8 +215,12 @@ def confidence_accuracy_correlation(correct: Any, confidence: Any) -> MetricResu
     if np.ptp(p) == 0:
         pb = sp = float("nan")
     else:
-        pb = float(stats.pearsonr(c.astype(float), p)[0])
-        sp = float(stats.spearmanr(c.astype(float), p)[0])
+        cf = c.astype(float)
+        pb = float(np.corrcoef(cf, p)[0, 1])
+        # Spearman = Pearson correlation of average ranks; the ranks of p are already computed
+        n0 = c.size - n1
+        rc = np.where(c, n0 + (n1 + 1) / 2, (n0 + 1) / 2)
+        sp = float(np.corrcoef(rc, ranks)[0, 1])
     return MetricResult(
         "confidence_accuracy_correlation",
         "Confidence–accuracy AUROC",

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/src/evalsuite/version.py

@@ -1,3 +1,3 @@
 """Package version (single source of truth, read by the build backend)."""
 
-__version__ = "0.5.0"
+__version__ = "0.5.1"

{evalsuite_python-0.5.0 → evalsuite_python-0.5.1}/tests/output/test_benchmarks.py

@@ -45,7 +45,7 @@ def test_v020_benchmarks_agree_with_their_references() -> None:
     for row in b.rows:
         assert row["reference_ms"] > 0 and row["max_abs_diff"] < 1e-9, row["case"]
     assert "statsmodels" in b.summary().splitlines()[0]
-    assert len(run_benchmarks(sizes=(300,), repeat=1, suite="all").rows) == 21
+    assert len(run_benchmarks(sizes=(300,), repeat=1, suite="all").rows) == 27
 
 
 def test_overall_summary_and_alphabetical_rows() -> None:
@@ -55,11 +55,12 @@ def test_overall_summary_and_alphabetical_rows() -> None:
         "Classification and regression",
         "Clinical, calibration and statistics",
         "LLM evaluation",
+        "LLM systems",
         "Segmentation and object detection",
         "Overall",
     ]
     total = overall[-1]
-    assert total["rows"] == 21 == sum(o["rows"] for o in overall[:-1])
+    assert total["rows"] == 27 == sum(o["rows"] for o in overall[:-1])
     assert total["matching"] == total["compared"]  # every compared case agrees with its reference
     if total["compared"]:
         assert total["min_speedup"] <= total["geomean_speedup"] <= total["max_speedup"]
@@ -114,3 +115,11 @@ def test_cases_needing_a_missing_optional_dependency_are_skipped(monkeypatch) ->
     monkeypatch.setattr(es, "meteor", missing)
     b = benchmarks.run_benchmarks(sizes=(500,), repeat=1, suite="llm")
     assert not any("METEOR" in r["case"] for r in b.rows) and b.rows
+
+
+def test_llmsys_benchmarks_agree_with_their_references() -> None:
+    b = run_benchmarks(sizes=(2000,), repeat=1, suite="llmsys")
+    assert len(b.rows) == 6
+    for row in b.rows:
+        if row["max_abs_diff"] is not None:
+            assert row["max_abs_diff"] < 1e-9, row["case"]