eval-toolkit 0.41.0__tar.gz → 0.42.0__tar.gz

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/CHANGELOG.md

@@ -7,6 +7,58 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.42.0] — 2026-05-19 — fit_isotonic_binary completes 4-calibrator family (closes #44)
+
+Final element of the binary scalar-prob calibrator family started by
+`fit_temperature_binary` (v0.35.0). All four now uniformly return
+`(params, apply)`:
+
+| Function | Params | Shipped |
+|---|---|---|
+| `fit_temperature_binary` | `(T,)` — single float | v0.35.0 |
+| `fit_isotonic_binary`    | `None` — non-parametric | **v0.42.0** |
+| `fit_platt_binary`       | `(a, b)` | v0.40.0 |
+| `fit_beta_binary`        | `(a, b, c)` | v0.40.0 |
+
+Consumer code can now iterate the family with a single shape, used
+to distinguish parametric from non-parametric via
+`if params is not None`:
+
+```text
+CALIBRATORS = {
+    "temperature": fit_temperature_binary,
+    "isotonic":    fit_isotonic_binary,
+    "platt":       fit_platt_binary,
+    "beta":        fit_beta_binary,
+}
+for name, fit_fn in CALIBRATORS.items():
+    params, apply = fit_fn(y_val, p_val)
+    calibrated = apply(p_test)
+    if params is not None:
+        manifest.record(f"{name}_params", params)
+```
+
+This matches the consumer's calibration-battery pattern in
+`prompt-injection-detection-prototype` (their ADR-056 supersedes
+ADR-023 to adopt the canonical `(params, apply)` shape across the
+full 4-calibrator audit battery).
+
+### Added
+
+- **`eval_toolkit.fit_isotonic_binary(y_true, y_score) -> (None,
+  apply)`** — thin wrapper over `fit_isotonic_calibrator`. The
+  `None` in the params slot encodes "non-parametric" (isotonic
+  regression is a monotone step function, no scalar params to log).
+- 6 new unit tests in `tests/test_calibration_binary_adapters.py`
+  including a 4-calibrator family-iteration integration test that
+  verifies the `None`-vs-tuple convention.
+
+### Protocol stability
+
+Additive only. No Tier-2 Protocol shape edits. v0.42 is minor 3 of
+consecutive-without-Protocol-changes (v0.40 + v0.41 + v0.42). Gate 2
+stays MET.
+
 ## [0.41.0] — 2026-05-18 — Croissant end-to-end (closes #42, v1.0 Gate 4 MET)
 
 Closes v1.0 readiness Gate 4 — "Croissant interop verified end-to-end."

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.41.0
+Version: 0.42.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/src/eval_toolkit/__init__.py

@@ -85,6 +85,7 @@ _EXPORTS: dict[str, str] = {
     "bayes_optimal_threshold": "eval_toolkit.calibration",
     "fit_beta_binary": "eval_toolkit.calibration",
     "fit_beta_calibrator": "eval_toolkit.calibration",
+    "fit_isotonic_binary": "eval_toolkit.calibration",
     "fit_isotonic_calibrator": "eval_toolkit.calibration",
     "fit_platt_binary": "eval_toolkit.calibration",
     "fit_platt_calibrator": "eval_toolkit.calibration",

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.41.0"
+__version__ = "0.42.0"

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/src/eval_toolkit/calibration.py

@@ -55,6 +55,7 @@ __all__ = [
     "bayes_optimal_threshold",
     "fit_beta_binary",
     "fit_beta_calibrator",
+    "fit_isotonic_binary",
     "fit_isotonic_calibrator",
     "fit_platt_binary",
     "fit_platt_calibrator",
@@ -1294,6 +1295,83 @@ def fit_beta_binary(
     return (a, b, c), apply
 
 
+def fit_isotonic_binary(
+    y_true: np.ndarray, y_score: np.ndarray
+) -> tuple[None, Callable[[np.ndarray], np.ndarray]]:
+    r"""Binary-probability adapter for :func:`fit_isotonic_calibrator`.
+
+    Mirror of :func:`fit_temperature_binary` / :func:`fit_platt_binary`
+    / :func:`fit_beta_binary`: returns ``(None, apply)``. Isotonic
+    regression is non-parametric — there are no introspectable scalar
+    parameters to log alongside the apply callable — so the params
+    slot is :obj:`None`.
+
+    The ``None``-in-params slot makes "non-parametric" unambiguous
+    while preserving the canonical ``(params_tuple, apply)`` shape
+    shared by the four binary scalar-prob calibrators. Consumer code
+    can iterate over the full family with one idiom:
+
+    .. code-block:: text
+
+        CALIBRATORS = {
+            "temperature": fit_temperature_binary,
+            "isotonic":    fit_isotonic_binary,
+            "platt":       fit_platt_binary,
+            "beta":        fit_beta_binary,
+        }
+        for name, fit_fn in CALIBRATORS.items():
+            params, apply = fit_fn(y_val, p_val)
+            calibrated = apply(p_test)
+            if params is not None:
+                manifest.record(f"{name}_params", params)
+
+    Added v0.42.0 (closes #44) to complete the binary scalar-prob
+    calibrator family started by ``fit_temperature_binary`` (v0.35.0).
+
+    Parameters
+    ----------
+    y_true : np.ndarray, shape (n,)
+        Binary validation labels in ``{0, 1}``.
+    y_score : np.ndarray, shape (n,)
+        Validation predicted probabilities of class 1, in [0, 1].
+
+    Returns
+    -------
+    tuple
+        ``(None, apply)`` — ``None`` in the params slot (isotonic is
+        non-parametric); ``apply`` maps probabilities through the
+        fitted monotonic step function.
+
+    Raises
+    ------
+    ValueError
+        On shape mismatch, empty input, non-finite scores, or
+        single-class ``y_true`` (propagated from
+        :func:`fit_isotonic_calibrator`).
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> rng = np.random.default_rng(0)
+    >>> n = 500
+    >>> y_val = rng.binomial(1, 0.3, size=n).astype(int)
+    >>> p_val = np.clip(y_val * 0.6 + rng.normal(0, 0.2, n), 0.01, 0.99)
+    >>> params, apply = fit_isotonic_binary(y_val, p_val)
+    >>> params is None
+    True
+    >>> apply(np.array([0.1, 0.5, 0.9])).shape == (3,)
+    True
+
+    See Also
+    --------
+    fit_isotonic_calibrator : underlying non-parametric fitter.
+    fit_temperature_binary : 1-parameter sibling.
+    fit_platt_binary : 2-parameter sibling.
+    fit_beta_binary : 3-parameter sibling.
+    """
+    return None, fit_isotonic_calibrator(y_true, y_score)
+
+
 def fit_temperature_oracle(
     y_true: np.ndarray, y_score: np.ndarray
 ) -> tuple[float, Callable[[np.ndarray], np.ndarray]]:

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/tests/golden/public_api/snapshot.json

@@ -136,6 +136,7 @@
     "file_sha256",
     "fit_beta_binary",
     "fit_beta_calibrator",
+    "fit_isotonic_binary",
     "fit_isotonic_calibrator",
     "fit_operating_points",
     "fit_platt_binary",
@@ -1036,7 +1037,7 @@
       "doc_first_line": "str(object='') -> str",
       "kind": "value",
       "type": "str",
-      "value": "'0.41.0'"
+      "value": "'0.42.0'"
     },
     "apply_operating_points": {
       "doc_first_line": "Apply fitted thresholds to a mixed-class or single-class target slice.",
@@ -1208,6 +1209,11 @@
       "kind": "function",
       "signature": "(y_true: 'np.ndarray', y_score: 'np.ndarray') -> 'Callable[[np.ndarray], np.ndarray]'"
     },
+    "fit_isotonic_binary": {
+      "doc_first_line": "Binary-probability adapter for :func:`fit_isotonic_calibrator`.",
+      "kind": "function",
+      "signature": "(y_true: 'np.ndarray', y_score: 'np.ndarray') -> 'tuple[None, Callable[[np.ndarray], np.ndarray]]'"
+    },
     "fit_isotonic_calibrator": {
       "doc_first_line": "Niculescu-Mizil & Caruana 2005 [#nm05]_ isotonic regression.",
       "kind": "function",

{eval_toolkit-0.41.0 → eval_toolkit-0.42.0}/tests/test_calibration_binary_adapters.py

@@ -12,6 +12,8 @@ import pytest
 from eval_toolkit import (
     fit_beta_binary,
     fit_beta_calibrator,
+    fit_isotonic_binary,
+    fit_isotonic_calibrator,
     fit_platt_binary,
     fit_platt_calibrator,
     fit_temperature_binary,
@@ -160,6 +162,66 @@ def test_beta_binary_apply_rejects_non_finite_test_scores(
         apply(np.array([0.5, np.nan, 0.8]))
 
 
+# ---------- fit_isotonic_binary ----------
+
+
+@pytest.mark.unit
+def test_isotonic_binary_returns_none_params_and_apply(
+    synthetic_binary_data: tuple[np.ndarray, np.ndarray],
+) -> None:
+    """Isotonic is non-parametric → params slot is None."""
+    y, p = synthetic_binary_data
+    params, apply = fit_isotonic_binary(y, p)
+    assert params is None
+    assert callable(apply)
+
+
+@pytest.mark.unit
+def test_isotonic_binary_apply_returns_same_shape(
+    synthetic_binary_data: tuple[np.ndarray, np.ndarray],
+) -> None:
+    y, p = synthetic_binary_data
+    _, apply = fit_isotonic_binary(y, p)
+    test = np.array([0.1, 0.5, 0.9])
+    out = apply(test)
+    assert out.shape == test.shape
+    assert (out >= 0.0).all() and (out <= 1.0).all()
+
+
+@pytest.mark.unit
+def test_isotonic_binary_apply_matches_underlying_calibrator(
+    synthetic_binary_data: tuple[np.ndarray, np.ndarray],
+) -> None:
+    """fit_isotonic_binary apply should match fit_isotonic_calibrator output."""
+    y, p = synthetic_binary_data
+    _, apply = fit_isotonic_binary(y, p)
+    canonical_apply = fit_isotonic_calibrator(y, p)
+    test = np.linspace(0.05, 0.95, 20)
+    np.testing.assert_allclose(apply(test), canonical_apply(test))
+
+
+@pytest.mark.unit
+def test_isotonic_binary_rejects_single_class() -> None:
+    y_single = np.zeros(50, dtype=int)
+    p = np.random.default_rng(0).uniform(0.0, 1.0, 50)
+    with pytest.raises(ValueError):
+        fit_isotonic_binary(y_single, p)
+
+
+@pytest.mark.unit
+def test_isotonic_binary_monotone(
+    synthetic_binary_data: tuple[np.ndarray, np.ndarray],
+) -> None:
+    """Isotonic regression is monotone non-decreasing in the score."""
+    y, p = synthetic_binary_data
+    _, apply = fit_isotonic_binary(y, p)
+    test = np.linspace(0.05, 0.95, 50)
+    out = apply(test)
+    # Allow tiny numerical noise but enforce non-decreasing trend
+    deltas = np.diff(out)
+    assert (deltas >= -1e-12).all(), "isotonic should be non-decreasing"
+
+
 # ---------- consistency across the calibrator family ----------
 
 
@@ -167,14 +229,15 @@ def test_beta_binary_apply_rejects_non_finite_test_scores(
 def test_all_four_binary_adapters_have_consistent_shape(
     synthetic_binary_data: tuple[np.ndarray, np.ndarray],
 ) -> None:
-    """temperature_binary, platt_binary, beta_binary all return ``(params, apply)``.
+    """temperature, isotonic, platt, beta all return ``(params, apply)``.
 
     Documents the audit-battery contract for the 4-calibrator pattern.
     """
     y, p = synthetic_binary_data
-    # All return (params, apply); apply is a callable taking (n,) -> (n,)
+    # All return (params, apply); apply is a callable taking (n,) -> (n,).
     for name, fitter in [
         ("temperature", fit_temperature_binary),
+        ("isotonic", fit_isotonic_binary),
         ("platt", fit_platt_binary),
         ("beta", fit_beta_binary),
     ]:
@@ -185,6 +248,38 @@ def test_all_four_binary_adapters_have_consistent_shape(
         assert (out >= 0.0).all() and (out <= 1.0).all(), f"{name}: output not in [0,1]"
 
 
+@pytest.mark.unit
+def test_consumer_idiom_iterating_all_four_calibrators(
+    synthetic_binary_data: tuple[np.ndarray, np.ndarray],
+) -> None:
+    """End-to-end consumer idiom: iterate the 4-element family with one shape.
+
+    Matches the calibration-battery pattern in
+    ``prompt-injection-detection-prototype/src/eval/calibration_battery.py``
+    (ADR-056). The ``params is not None`` check distinguishes parametric
+    from non-parametric in a single conditional.
+    """
+    y, p = synthetic_binary_data
+    fitters = {
+        "temperature": fit_temperature_binary,
+        "isotonic": fit_isotonic_binary,
+        "platt": fit_platt_binary,
+        "beta": fit_beta_binary,
+    }
+    p_test = np.linspace(0.05, 0.95, 30)
+    recorded_params: dict[str, object] = {}
+    calibrated: dict[str, np.ndarray] = {}
+    for name, fit_fn in fitters.items():
+        params, apply = fit_fn(y, p)
+        calibrated[name] = apply(p_test)
+        if params is not None:
+            recorded_params[name] = params
+    # Three of four have inspectable params; isotonic is None.
+    assert set(recorded_params.keys()) == {"temperature", "platt", "beta"}
+    # All four produced calibrated outputs of matching shape.
+    assert all(out.shape == p_test.shape for out in calibrated.values())
+
+
 @pytest.mark.unit
 def test_platt_binary_params_are_pair(
     synthetic_binary_data: tuple[np.ndarray, np.ndarray],