eval-toolkit 1.0.1__tar.gz → 1.0.2__tar.gz

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/CHANGELOG.md

@@ -5,6 +5,55 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.2] — 2026-05-26 — #76 cleanup batch closes (RC2 + RC3 + F-metrics-1/3/4)
+
+Closes the GH #76 v1.0.1 cleanup tracker. All 6 items shipped across
+v1.0.1 (RC4) and v1.0.2 (this release). All P3, all NON-BREAKING.
+
+### Changed (Tier-2 ADDITIVE: contract clarification only)
+
+- **RC2** (#76) — `SimilarityStrategy` Protocol promoted from
+  "pre-v0.7 internal interface" (prose framing only) to formal
+  10th strict Tier-2 Protocol per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).
+  Aligns prose surfaces (README, extending.md, strict_tier2_protocols.md,
+  api/protocols.md, ADR 0004 §D6, roadmap.md) with the contract
+  already locked in `tests/golden/public_api/snapshot.json` +
+  `src/eval_toolkit/__init__.py:_EXPORTS` since v1.0.0. **No code
+  change — documentation-only reconciliation.** Strict-Tier-2 count
+  goes 9 → 10 (+ 1 opt-in `Versioned`).
+
+### Fixed
+
+- **RC3** (#76) — `tests/test_harness_folded.py::test_evaluate_folded_reseed_splitter_varies_partitions`
+  test hardening. Previous assertions covered count + key existence
+  only; a regression silently reusing the splitter (R8-C1 pre-fix
+  behavior) could still pass. v1.0.2 adds row-content comparison:
+  replays `reseed_splitter` against the splitter for `seed=1` vs
+  `seed=2` and asserts fold-0 test partitions differ via feature-text
+  set membership (robust to `_slice_subset`'s `reset_index(drop=True)`
+  via stable text-column identifiers).
+
+- **F-metrics-1** (#76) — `brier_score` docstring input-domain clarity.
+  Added explicit "Input domain" Notes subsection clarifying binary
+  labels in `{0, 1}` + calibrated probabilities in `[0, 1]` are
+  required; raw logits or unbounded ranking scores pass the finiteness
+  check but produce out-of-range MSE that misrepresents calibration
+  quality. Includes calibration-applying recipe pointer.
+
+- **F-metrics-3** (#76) — `expected_calibration_error` docstring
+  uniform-scores note. Added explicit Notes subsection documenting
+  that constant `y_score` returns 0.0 (per-bin formula trivially
+  satisfied) but is semantically misleading — uninformative scorers
+  look "perfectly calibrated" despite zero discriminative power.
+  Callers should filter constant inputs before ECE.
+
+- **F-metrics-4** (#76) — `brier_score` docstring single-class
+  edge-case explicit. Added Notes subsection with closed-form
+  expressions for all-zeros (`BS = mean(p²)`) and all-ones
+  (`BS = mean((1-p)²)`) cases. Explicit confirmation that
+  per-slice degenerate-class evaluation is supported (unlike
+  PR-AUC / ROC-AUC).
+
 ## [1.0.1] — 2026-05-25 — audit_citation_alignment + RC4 docs polish
 
 First v1.x patch release. Ships the `audit_citation_alignment` validator

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.0.1
+Version: 1.0.2
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -115,7 +115,8 @@ format changes.
 ├─ Tier 2 ─ Protocol-based orchestration ────────────────┤
 │  Scorer / SliceAwareScorer / LeakageCheck / Splitter   │
 │  ThresholdSelector / DatasetLoader / MetricSpec        │
-│  MetaLearner / Probe / TextTransform (9 strict)        │
+│  MetaLearner / Probe / TextTransform /                 │
+│  SimilarityStrategy (10 strict)                        │
 │  Versioned (opt-in: per-object versions in manifest)   │
 ├─ Tier 1 ─ Functional core ─────────────────────────────┤
 │  pr_auc / roc_auc / ECE variants / Brier / bootstrap_ci│

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/README.md

@@ -32,7 +32,8 @@ format changes.
 ├─ Tier 2 ─ Protocol-based orchestration ────────────────┤
 │  Scorer / SliceAwareScorer / LeakageCheck / Splitter   │
 │  ThresholdSelector / DatasetLoader / MetricSpec        │
-│  MetaLearner / Probe / TextTransform (9 strict)        │
+│  MetaLearner / Probe / TextTransform /                 │
+│  SimilarityStrategy (10 strict)                        │
 │  Versioned (opt-in: per-object versions in manifest)   │
 ├─ Tier 1 ─ Functional core ─────────────────────────────┤
 │  pr_auc / roc_auc / ECE variants / Brier / bootstrap_ci│

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.0.1"
+__version__ = "1.0.2"

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/src/eval_toolkit/metrics.py

@@ -792,6 +792,20 @@ def expected_calibration_error(
     empirical positive rate in the bin, and :math:`\\mathrm{conf}` is the
     mean predicted score.
 
+    **Uniform / uninformative scores** (F-metrics-3 v1.0.2 clarity pass):
+    when ``y_score`` is constant (e.g., ``[0.5] * n`` — an uninformative
+    detector), this function returns ``0.0`` regardless of the true label
+    distribution. That's technically correct per the formula —
+    :math:`|\\mathrm{acc}(B_m) - \\mathrm{conf}(B_m)|` measures bin-level
+    calibration, and a single occupied bin with ``conf = base rate``
+    achieves perfect calibration locally. But it is semantically
+    misleading: an uninformative scorer looks "perfectly calibrated"
+    even though it has zero discriminative power. **Callers should
+    detect and filter uninformative inputs before passing to ECE** —
+    e.g., reject when ``np.unique(y_score).size == 1`` or when the
+    score variance is below a domain-specific threshold. Use
+    :func:`brier_score` or :func:`pr_auc` for resolution-aware metrics.
+
     References
     ----------
     .. [1] DeGroot, M. H. & Fienberg, S. E. "The comparison and evaluation of
@@ -1240,6 +1254,30 @@ def brier_score(
     -----
     .. math:: \mathrm{BS} = \frac{1}{n} \sum_i (p_i - y_i)^2
 
+    **Input domain** (F-metrics-1 v1.0.2 clarity pass): ``y_true`` must
+    be binary labels in ``{0, 1}`` (other label values raise
+    ``ValueError``). ``y_score`` must be calibrated probabilities in
+    ``[0, 1]`` — raw logits or unbounded ranking scores will pass the
+    finiteness check but produce an out-of-range MSE that misrepresents
+    calibration quality. If your scorer produces logits, apply
+    sigmoid / softmax / a fitted calibrator (see
+    :mod:`eval_toolkit.calibration`) before passing to ``brier_score``.
+
+    **Single-class behavior** (F-metrics-4 v1.0.2 clarity pass): unlike
+    PR-AUC / ROC-AUC, ``brier_score`` is well-defined when ``y_true``
+    is all-zeros or all-ones — it degenerates to the MSE around the
+    constant class label. Specifically:
+
+    - All-zeros: :math:`\mathrm{BS} = \frac{1}{n} \sum_i p_i^2` —
+      forecasting any positive probability incurs squared-error loss.
+    - All-ones: :math:`\mathrm{BS} = \frac{1}{n} \sum_i (1 - p_i)^2`
+      — forecasting low probability incurs squared-error loss.
+
+    This is the deliberate Brier-as-strict-proper-scoring-rule behavior
+    (Brier 1950). Per-slice degenerate-class evaluation is supported
+    via the ``empty_strategy`` parameter for ``n=0`` only; non-empty
+    single-class slices score normally.
+
     See Also
     --------
     eval_toolkit.metrics.brier_decomposition :

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/tests/golden/public_api/snapshot.json

@@ -1373,7 +1373,7 @@
       "doc_first_line": "str(object='') -> str",
       "kind": "value",
       "type": "str",
-      "value": "'1.0.1'"
+      "value": "'1.0.2'"
     },
     "apply_operating_points": {
       "doc_first_line": "Apply fitted thresholds to a mixed-class or single-class target slice.",

{eval_toolkit-1.0.1 → eval_toolkit-1.0.2}/tests/test_harness_folded.py

@@ -162,6 +162,29 @@ def test_evaluate_folded_reseed_splitter_varies_partitions() -> None:
     assert "seed=1/fold=0" in fold_ids
     assert "seed=2/fold=0" in fold_ids
 
+    # R10-RC3 v1.0.2 hardening (#76): the previous assertions covered
+    # COUNT + key existence but did NOT verify the actual partition
+    # indices differ across seeds — a regression that silently reused
+    # the splitter (R8-C1 pre-fix behavior) could still pass. Directly
+    # verify the reseed_splitter callback yields different partitions
+    # by replaying it against the splitter.
+    splitter = StratifiedKFoldSplitter(k=2, seed=42)
+    splits_seed_1 = list(dataclasses.replace(splitter, seed=1).iter_folds(parent, groups=None))
+    splits_seed_2 = list(dataclasses.replace(splitter, seed=2).iter_folds(parent, groups=None))
+    # _slice_subset resets the child df index to [0..n-1], so compare
+    # the underlying text feature values instead (stable across the
+    # reset_index drop). Each child slice's `text` column carries the
+    # original row labels.
+    fold_0_test_texts_seed_1 = set(splits_seed_1[0]["test"].df["text"].tolist())
+    fold_0_test_texts_seed_2 = set(splits_seed_2[0]["test"].df["text"].tolist())
+    # Different seeds → different fold-0 test partitions (the whole
+    # point of reseed_splitter).
+    assert fold_0_test_texts_seed_1 != fold_0_test_texts_seed_2, (
+        "reseed_splitter callback failed to vary partitions: "
+        f"seed=1 fold=0 texts={sorted(fold_0_test_texts_seed_1)[:5]}... "
+        f"seed=2 fold=0 texts={sorted(fold_0_test_texts_seed_2)[:5]}..."
+    )
+
 
 @pytest.mark.unit
 def test_evaluate_folded_single_seed_no_deprecation_warning() -> None: