eval-toolkit 0.34.0__tar.gz → 0.37.0__tar.gz

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/.gitignore

@@ -22,6 +22,7 @@ wheels/
 .coverage.*
 htmlcov/
 coverage.xml
+coverage.json
 .hypothesis/
 
 # Type-checker / linter caches

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/CHANGELOG.md

@@ -7,6 +7,146 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.37.0] — 2026-05-18 — TokenizationLeakageCheck + per-module coverage floors
+
+Two-issue bundle (#35 + #37) plus housekeeping closure of stale items
+(PR #27, #38) that turned out to have been resolved in v0.33.x without
+being checked off. Roadmap refresh in `3d40796` (this minor's
+predecessor commit) replaced the version-keyed candidate list with
+issue-keyed tracking, so this class of stale-roadmap bug shouldn't
+recur.
+
+### Added
+
+- **`eval_toolkit.leakage.TokenizationLeakageCheck`** — new within-split
+  `LeakageCheck` that dedups on tokenizer output rather than raw text.
+  Catches encoding-obfuscated dupes that survive
+  `NormalizedFormLeakageCheck` but collapse to identical `input_ids`
+  under a transformer's BPE / SentencePiece / WordPiece tokenizer.
+  Accepts any `Callable[[str], Mapping[str, object]]` returning HF-style
+  output with an `"input_ids"` key — does **not** import `transformers`
+  itself; consumers pass an already-instantiated tokenizer. Default
+  severity `"error"` (mirrors `NormalizedFormLeakageCheck`). Closes #35.
+- New optional install extra **`[transformers]`** (`transformers>=4.0`).
+  Intentionally **not** in `[all]` / `[dev]` — mirrors the `[embeddings]`
+  precedent from v0.33.1 to keep contributor setup small (transformers
+  transitively pulls torch ~700MB).
+
+### Test
+
+- **Per-module coverage floors restored.** `scripts/check_module_floors.py`
+  enforces an 85 % per-file floor (coverage.py natively only ships
+  global `--fail-under`). Hooked into `make coverage` via a post-pytest
+  invocation. Closes #37.
+- **`# pragma: no cover` on optional-dep-active paths** in `seeds.py`
+  (torch) and `embeddings.py` (sentence-transformers). Reflects the
+  reality that these branches execute in user code, not CI. Both
+  modules now report 100 % coverage; previously sat at ~70 % which
+  obscured per-module floor enforcement.
+
+### Fixed
+
+- **`make coverage` Makefile parity with PR CI.** PR #27 (external
+  contributor @leno23, draft) proposed adding `-m "not monte_carlo and
+  not benchmark"` to the `coverage` target. Audit found the same fix
+  had landed in v0.33.0 commit `9e375a8` ahead of the PR being filed;
+  closed PR #27 as superseded with thanks. No change in this release.
+
+### Closed (already-resolved)
+
+- **#38 — CI doctests for `paths.py` / `provenance.py` / `seeds.py` /
+  `docs.py`.** All four modules were added to `.doctest-modules` in
+  `a26fd44` (2026-05-14, v0.32.x era); 7 doctests collected across the
+  named modules in current CI. Closed as already-resolved.
+
+### Test coverage
+
+Test count 1376 → 1387 (+11). Aggregate 95.65 % → 95.69 %. All 28
+modules ≥ 90 % individually post-pragma.
+
+## [0.36.0] — 2026-05-18 — harness parallelization (#29, #30) + Node 24 actions
+
+Wires the v0.34.0 unified parallelism pattern into the harness evaluation
+loop. `evaluate()` and `evaluate_folded()` now accept an `n_jobs` kwarg
+(default `1` preserves bit-identical sequential behavior); under
+`n_jobs != 1`, the `(slice × scorer)` work-unit loop in
+`_score_all_slices` and the `(spec × scorer)` fit phase in
+`_attach_transferred_operating_points` dispatch through joblib loky via
+the existing `_parallel.parallel_map` helper.
+
+### Added
+
+- `evaluate(..., n_jobs: int = 1)` and `evaluate_folded(..., n_jobs: int = 1)`
+  — keyword-only kwarg per Principle #3 of `methodology/parallelism.md`.
+  `n_jobs=1` (default) runs the existing pure-Python sequential loop
+  (Principle #4 — bit-identical to v0.35). `n_jobs > 1` uses joblib loky;
+  `n_jobs=-1` uses all cores; `n_jobs=0` is rejected. Closes #29, #30.
+- Strict-pickle Scorer sniff at `evaluate()` entry when `n_jobs != 1`:
+  raises a clean `TypeError` referencing
+  `methodology/parallelism.md#scorer-picklability` with the underlying
+  pickle error attached. Reuses the v0.35 ADR contract; no new exception
+  class. Catches non-picklable scorers up front rather than relying on
+  joblib's more permissive cloudpickle path (which would silently absorb
+  closures and obscure the contract documented in v0.35).
+
+### Internal
+
+- New module-scope step functions `_score_one_pair` and
+  `_fit_one_op_point_pair` in `harness.py` (picklable; required by loky).
+- `_score_all_slices` and `_attach_transferred_operating_points`
+  refactored to use flat work-unit dispatch via `parallel_map`.
+
+### Tests
+
+- New `tests/test_harness_parallelism.py` (7 tests): bit-identical
+  reproducibility across `n_jobs=1` vs `n_jobs=2` for `evaluate`
+  (basic, paired-diffs, operating-points), `evaluate_folded`,
+  picklability rejection (closure scorer), `n_jobs=0` rejection,
+  `n_jobs=-1` smoke. All 66 harness tests pass (7 new + 59 existing).
+
+### Infrastructure
+
+- Bumped `actions/upload-artifact` and `actions/download-artifact` from
+  `@v5` → `@v6` across `publish.yml` / `nightly-mc.yml` /
+  `nightly-benchmarks.yml`. The v6 majors run on Node.js 24
+  (GitHub deprecates Node 20 actions from 2026-06-02). Other pinned
+  actions (`checkout@v6`, `setup-uv@v8.1.0`, `codeql-action@v3`,
+  `deploy-pages@v4`, `upload-pages-artifact@v3`) were not flagged in
+  the v0.35 publish annotation and are deferred to a separate audit.
+
+## [0.35.0] — 2026-05-18 — `fit_temperature_binary` + Scorer picklability ADR
+
+Small, additive release. Adds a binary-classification calibration helper
+that lets consumers drop the ~50 LOC scalar-proba adapter many were
+carrying, plus a design ADR that unblocks the v0.36 harness / operating-
+point parallelization work (#29, #30) without re-litigating picklability.
+
+### Added
+
+- `eval_toolkit.fit_temperature_binary(y_true, y_score)` — scalar-proba
+  adapter for the multi-class `fit_temperature` fitter. Converts `(n,)`
+  probabilities of class 1 to a 2-column logit array via clipped logit
+  (`[0, logit(p)]` so softmax row 1 reproduces `p`), delegates to the
+  deployment-quality fitter, and returns `(T_opt, apply)` where
+  `apply: (n,) -> (n,)` does scalar-in / scalar-out T-scaling. Unlike
+  `fit_temperature_oracle`, no warning — the contract assumes val / test
+  separation (deployment-quality calibration, not fit-on-test). Closes
+  #28.
+
+### Documentation
+
+- `docs/source/methodology/parallelism.md` — new `## Scorer picklability`
+  sub-section documenting the Scorer protocol's picklability contract
+  for `n_jobs > 1` usage. Includes worked picklable / broken-closure /
+  fix examples plus a list of common non-picklable patterns to watch for
+  in user-supplied Scorers (closures, lambdas on instances, local-scope
+  classes, attributes holding live sockets / file handles). Anchors on
+  the existing v0.34.0 `parallel_map` pickle sniff + `TypeError`
+  channel — no new exception class. Unblocks v0.36 implementation of
+  #29 and #30.
+- `eval_toolkit.protocols.Scorer` docstring — Notes block pointing at
+  the new methodology section.
+
 ## [0.34.0] — 2026-05-17 — Phase 4 stats unblockers + unified parallelism + cookbook (BREAKING)
 
 Closes all 7 open backlog issues in one consumer-closing release. Also

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.34.0
+Version: 0.37.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -69,6 +69,8 @@ Requires-Dist: matplotlib>=3.8; extra == 'plotting'
 Requires-Dist: pillow>=10.0; extra == 'plotting'
 Provides-Extra: property
 Requires-Dist: hypothesis>=6.100; extra == 'property'
+Provides-Extra: transformers
+Requires-Dist: transformers>=4.0; extra == 'transformers'
 Provides-Extra: validation
 Provides-Extra: yaml
 Requires-Dist: pyyaml>=6.0; extra == 'yaml'

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/pyproject.toml

@@ -56,6 +56,13 @@ parquet = ["pyarrow>=15.0"]
 # setup small. The canonical semantic-dedup recipe (all-MiniLM-L6-v2 +
 # cosine@0.80) is what this factory pre-wires for callers.
 embeddings = ["sentence-transformers>=3.0"]
+# v0.37.0: TokenizationLeakageCheck — HF-tokenizer-aware dedup.
+# transformers transitively pulls torch + tokenizers (~700MB) so we
+# follow the [embeddings] precedent: opt-in only, NOT in [all] / [dev].
+# Consumers pass an already-instantiated tokenizer callable; the check
+# itself does not import transformers, so the optional install is
+# strictly for callers wanting AutoTokenizer.from_pretrained(...).
+transformers = ["transformers>=4.0"]
 # DEPRECATED (announced v0.30.1, removal v0.33.0).
 #
 # Retained as a transitive no-op so `pip install eval-toolkit[validation]`

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/src/eval_toolkit/__init__.py

@@ -87,6 +87,7 @@ _EXPORTS: dict[str, str] = {
     "fit_isotonic_calibrator": "eval_toolkit.calibration",
     "fit_platt_calibrator": "eval_toolkit.calibration",
     "fit_temperature": "eval_toolkit.calibration",
+    "fit_temperature_binary": "eval_toolkit.calibration",
     "fit_temperature_oracle": "eval_toolkit.calibration",
     "reliability_curve": "eval_toolkit.calibration",
     "reliability_diagram_data": "eval_toolkit.calibration",
@@ -146,6 +147,7 @@ _EXPORTS: dict[str, str] = {
     "NearDuplicateCheck": "eval_toolkit.leakage",
     "NormalizedFormLeakageCheck": "eval_toolkit.leakage",
     "TemporalLeakageCheck": "eval_toolkit.leakage",
+    "TokenizationLeakageCheck": "eval_toolkit.leakage",
     "run_leakage_checks": "eval_toolkit.leakage",
     # --- loaders ---
     "DataFrameLoader": "eval_toolkit.loaders",

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.34.0"
+__version__ = "0.37.0"

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/src/eval_toolkit/calibration.py

@@ -57,6 +57,7 @@ __all__ = [
     "fit_isotonic_calibrator",
     "fit_platt_calibrator",
     "fit_temperature",
+    "fit_temperature_binary",
     "fit_temperature_oracle",
     "maximum_calibration_error",
     "reliability_curve",
@@ -1038,6 +1039,102 @@ def _negative_log_likelihood(t: float, logits: np.ndarray, labels: np.ndarray) -
     return float(-log_probs[np.arange(len(labels)), labels].mean())
 
 
+def fit_temperature_binary(
+    y_true: np.ndarray,
+    y_score: np.ndarray,
+    *,
+    bounds: tuple[float, float] = (0.05, 20.0),
+) -> tuple[float, Callable[[np.ndarray], np.ndarray]]:
+    r"""Binary-probability adapter for :func:`fit_temperature` (Guo et al. 2017 [#guo]_).
+
+    Fits a scalar T > 0 on *validation* probabilities of class 1 and returns
+    both T and a callable that applies the same T-scaling to test
+    probabilities. Internally:
+
+    1. Clips ``y_score`` to ``[1e-7, 1-1e-7]`` for finite logit inversion.
+    2. Builds a 2-column logit array ``[0, logit(p)]`` so softmax row 1
+       reproduces ``p`` exactly.
+    3. Delegates to :func:`fit_temperature` for the bounded NLL minimization.
+    4. Returns ``(T, apply)`` where ``apply(p_test) = sigmoid(logit(p_test)/T)``.
+
+    Unlike :func:`fit_temperature_oracle`, this does NOT emit a warning — the
+    contract is that ``y_true`` / ``y_score`` come from a held-out validation
+    set and ``apply`` is invoked on a separate test set (deployment-quality
+    calibration, not fit-on-test).
+
+    Parameters
+    ----------
+    y_true : np.ndarray, shape (n,)
+        Binary validation labels in {0, 1}.
+    y_score : np.ndarray, shape (n,)
+        Validation predicted probabilities of class 1, in [0, 1]. Values at
+        the extremes are clipped to ``[1e-7, 1 - 1e-7]``.
+    bounds : tuple of float, optional
+        ``(lo, hi)`` bracket for T. Default ``(0.05, 20.0)``, matches
+        :func:`fit_temperature`.
+
+    Returns
+    -------
+    tuple
+        ``(T_optimal, apply)`` where ``apply: (n,) -> (n,)`` maps any input
+        probability array through :math:`\sigma(\mathrm{logit}(p) / T)`.
+
+    Raises
+    ------
+    ValueError
+        On shape mismatch, empty input, non-finite scores, or single-class
+        ``y_true``.
+    RuntimeError
+        If the bounded scalar optimizer fails to converge.
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> rng = np.random.default_rng(0)
+    >>> n = 500
+    >>> y_val = rng.binomial(1, 0.3, size=n).astype(int)
+    >>> p_val = np.clip(y_val * 0.6 + rng.normal(0, 0.2, n), 0.01, 0.99)
+    >>> T, apply = fit_temperature_binary(y_val, p_val)
+    >>> T > 0
+    True
+    >>> p_test = np.array([0.1, 0.5, 0.9])
+    >>> apply(p_test).shape == (3,)
+    True
+
+    See Also
+    --------
+    fit_temperature : underlying multi-class fitter (operates on 2-col logits)
+    fit_temperature_oracle : diagnostic-only variant that fits T on the same
+        probabilities it scores
+
+    References
+    ----------
+    .. [#guo] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. "On
+       calibration of modern neural networks." ICML 2017. arXiv:1706.04599.
+    """
+    y_true_arr, y_score_arr = _validate_calibrator_inputs(y_true, y_score)
+
+    # Build 2-col logits [0, logit(p)] so softmax([0, logit(p)])[1] == p exactly.
+    s_clipped = np.clip(y_score_arr, _SCORE_CLIP_LO, _SCORE_CLIP_HI)
+    logit_pos = np.log(s_clipped / (1.0 - s_clipped))
+    val_logits_2col = np.column_stack([np.zeros_like(logit_pos), logit_pos])
+
+    result = fit_temperature(val_logits_2col, y_true_arr, bounds=bounds)
+    t_optimal = float(result["temperature"])
+
+    def apply(scores: np.ndarray) -> np.ndarray:
+        arr = np.asarray(scores, dtype=float).ravel()
+        if not np.isfinite(arr).all():
+            raise ValueError("scores contains NaN or inf")
+        clipped = np.clip(arr, _SCORE_CLIP_LO, _SCORE_CLIP_HI)
+        logit = np.log(clipped / (1.0 - clipped))
+        scaled = logit / t_optimal
+        out: np.ndarray = (1.0 / (1.0 + np.exp(-scaled))).astype(float)
+        return out
+
+    return t_optimal, apply
+
+
 def fit_temperature_oracle(
     y_true: np.ndarray, y_score: np.ndarray
 ) -> tuple[float, Callable[[np.ndarray], np.ndarray]]:

{eval_toolkit-0.34.0 → eval_toolkit-0.37.0}/src/eval_toolkit/embeddings.py

@@ -88,15 +88,18 @@ def make_minilm_embedder(
             "Install via: pip install eval-toolkit[embeddings]"
         ) from e
 
-    _logger.debug(
+    # sentence-transformers-active path: excluded from CI coverage
+    # because [embeddings] is intentionally kept out of [dev]/[all]
+    # (transitive torch cost ~700MB per the v0.33.1 design note).
+    _logger.debug(  # pragma: no cover
         "loading SentenceTransformer model_id=%s device=%s batch_size=%d",
         model_id,
         device,
         batch_size,
     )
-    model = SentenceTransformer(model_id, device=device)
+    model = SentenceTransformer(model_id, device=device)  # pragma: no cover
 
-    def embedder(texts: Sequence[str]) -> np.ndarray:
+    def embedder(texts: Sequence[str]) -> np.ndarray:  # pragma: no cover
         result = model.encode(
             list(texts),
             convert_to_numpy=True,
@@ -105,4 +108,4 @@ def make_minilm_embedder(
         )
         return np.asarray(result, dtype=np.float64)
 
-    return embedder
+    return embedder  # pragma: no cover