eval-toolkit 0.33.1__tar.gz → 0.35.0__tar.gz

{eval_toolkit-0.33.1 → eval_toolkit-0.35.0}/CHANGELOG.md

@@ -7,6 +7,142 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.35.0] — 2026-05-18 — `fit_temperature_binary` + Scorer picklability ADR
+
+Small, additive release. Adds a binary-classification calibration helper
+that lets consumers drop the ~50 LOC scalar-proba adapter many were
+carrying, plus a design ADR that unblocks the v0.36 harness / operating-
+point parallelization work (#29, #30) without re-litigating picklability.
+
+### Added
+
+- `eval_toolkit.fit_temperature_binary(y_true, y_score)` — scalar-proba
+  adapter for the multi-class `fit_temperature` fitter. Converts `(n,)`
+  probabilities of class 1 to a 2-column logit array via clipped logit
+  (`[0, logit(p)]` so softmax row 1 reproduces `p`), delegates to the
+  deployment-quality fitter, and returns `(T_opt, apply)` where
+  `apply: (n,) -> (n,)` does scalar-in / scalar-out T-scaling. Unlike
+  `fit_temperature_oracle`, no warning — the contract assumes val / test
+  separation (deployment-quality calibration, not fit-on-test). Closes
+  #28.
+
+### Documentation
+
+- `docs/source/methodology/parallelism.md` — new `## Scorer picklability`
+  sub-section documenting the Scorer protocol's picklability contract
+  for `n_jobs > 1` usage. Includes worked picklable / broken-closure /
+  fix examples plus a list of common non-picklable patterns to watch for
+  in user-supplied Scorers (closures, lambdas on instances, local-scope
+  classes, attributes holding live sockets / file handles). Anchors on
+  the existing v0.34.0 `parallel_map` pickle sniff + `TypeError`
+  channel — no new exception class. Unblocks v0.36 implementation of
+  #29 and #30.
+- `eval_toolkit.protocols.Scorer` docstring — Notes block pointing at
+  the new methodology section.
+
+## [0.34.0] — 2026-05-17 — Phase 4 stats unblockers + unified parallelism + cookbook (BREAKING)
+
+Closes all 7 open backlog issues in one consumer-closing release. Also
+lands the toolkit's first unified parallelism story: a shared internal
+`parallel_map` helper + `n_jobs` kwarg on all 5 public bootstrap
+functions. Future iterations will mechanically extend the same helper to
+harness + operating-points (follow-up issues filed).
+
+### Breaking changes
+
+- **`eval_toolkit.bootstrap.mde_from_ci`**: parameter renamed from
+  `paired` to `ci` and type widened to `BootstrapCI | PairedBootstrapCI`
+  (was `PairedBootstrapCI`-only). Positional callers unaffected; keyword
+  callers must update:
+  ```python
+  mde_from_ci(paired=x)  # v0.33.x and earlier
+  mde_from_ci(ci=x)      # v0.34.0+
+  mde_from_ci(x)         # positional form, unchanged
+  ```
+  This is a one-time exception to the repo's 2-minor-version deprecation
+  warning policy (justification + criteria recorded in
+  [`docs/source/DEPRECATION.md`](docs/source/DEPRECATION.md#one-time-exceptions-to-the-2-minor-version-warning-policy)).
+  Notification issues filed on the 2 known toolkit consumers
+  (`prompt-injection-detection-submission`, `post-transformers`). Audit
+  confirms both use positional form — zero actual breakage in practice.
+
+### Added
+
+- `eval_toolkit.bootstrap.block_bootstrap_on_folds` — CV-aware sibling
+  to `cv_clt_ci`; resamples K folds with replacement; returns
+  `BootstrapCI(method="block_bootstrap")`. The A-008 sensitivity-check
+  pattern (block-bootstrap halfwidth / cv_clt halfwidth > 1.5 flags
+  LODO non-exchangeability) is the prototypical use. Closes #21.
+- `eval_toolkit.RecallAtFprResult` (frozen dataclass) +
+  `eval_toolkit.recall_at_fpr(y_true, y_score, target_fpr)` — one-shot
+  recall + actual_fpr + FP/TN at the smallest threshold meeting FPR ≤
+  target. Use `.to_dict()` for JSON / pandas-row integration. Closes #9.
+- New optional `n_jobs: int = 1` kwarg on 5 bootstrap functions:
+  `bootstrap_ci`, `paired_bootstrap_diff`, `paired_bootstrap_ece_diff`,
+  `paired_bootstrap_op_point_diff`, `paired_mde` (via `_bootstrap_t_ci`
+  internal helper). Backed by a new internal `_parallel.parallel_map`
+  helper (joblib loky; not exported). `n_jobs > 1` reproduces `n_jobs=1`
+  result bit-for-bit for the same seed (via
+  `np.random.SeedSequence.spawn`). Lambda metrics rejected at call time
+  with helpful `TypeError`. `n_jobs > os.cpu_count()` is auto-capped
+  with WARNING log; `n_jobs=0` raises `ValueError`. Closes #17.
+- 6 new pages in `docs/source/examples/`:
+  - **Cookbook** (closes #19): `nested_seed_split.md`,
+    `callable_embedder_dedup.md`, `cross_corpus_contamination_scan.md`.
+  - **Plotting walkthroughs**: `plot_roc_curve_walkthrough.md`,
+    `plot_pareto_frontier_walkthrough.md`,
+    `plot_slice_metric_heatmap_walkthrough.md` (backfills the v0.33.0
+    docs gap).
+- `docs/source/methodology/parallelism.md` — design rationale + caller
+  contract for the toolkit-wide parallelism story. Documents the 6
+  design principles (single backend, single helper, opt-in per-fn,
+  default sequential, reproducibility via SeedSequence, picklability
+  surface) and the checklist for adding `n_jobs` to a new function.
+
+### Changed
+
+- `eval_toolkit.bootstrap.mde_from_ci` now accepts
+  `BootstrapCI | PairedBootstrapCI` (was paired-only). See **Breaking
+  changes** above. Closes #20.
+- `eval_toolkit.build_manifest` gains `config_path: Path | str | None`
+  kwarg; when supplied, `config_hash` is computed as
+  `sha256(Path(config_path).read_bytes()).hexdigest()` — capturing the
+  exact YAML file bytes including comments + key ordering (which the
+  default canonical-JSON path strips during parse). Default behavior
+  preserved when `config_path is None`. Closes #10.
+
+### Internal
+
+- New `src/eval_toolkit/_parallel.py` (internal; not exported) — single
+  source of truth for parallelism. Future per-function `n_jobs`
+  additions will reuse this helper. The toolkit's first INFO-level
+  log site is here (once-per-process guidance log when `n_jobs=1` AND
+  iteration count ≥ 1000). New `tests/test_parallel.py` covers smart-
+  default semantics + reproducibility contract.
+- New golden test `tests/golden/test_dedup_holdout_calibration.py`
+  exercising 3 deterministic `SimilarityStrategy` variants against a
+  migrated 50-pair adversarial fixture at thresholds {0.75, 0.80, 0.85}
+  (strict snapshot at `tests/golden/data/dedup_holdout_expected.json`)
+  plus an `EmbeddingCosineStrategy` soft-bound check (FPR < 0.5,
+  FNR < 0.5 at threshold 0.80) gated by `pytest.importorskip` +
+  `@pytest.mark.slow`. Refresh helper at
+  `scripts/refresh_dedup_holdout.py`. Closes #18.
+- `CONTRIBUTING.md` + `docs/source/repo-strategy.md` updated with
+  explicit "Parallelism" section codifying the new pattern (was an
+  implicit anti-pattern before; v0.34.0 codifies the new opt-in design).
+- `docs/source/DEPRECATION.md` extended with a "One-time exceptions"
+  section documenting the `mde_from_ci` rename + criteria future
+  exceptions must satisfy.
+- RNG-stream note: the 5 wired bootstrap fns now derive per-resample
+  seeds via `np.random.SeedSequence(seed).spawn(n_resamples)` instead
+  of sequential calls on a single `Generator`. The bootstrap output is
+  *statistically equivalent* (both are valid bootstraps) but the exact
+  numerical CI bounds for the same caller-supplied `seed` will differ
+  slightly from v0.33.x. Existing tests use behavioral assertions
+  (`overlaps_zero`, `delta`, etc.) that are robust to the RNG-stream
+  change; tests that pin exact CI bounds (e.g., consumer golden tests)
+  may need regen on upgrade.
+
 ## [0.33.1] — 2026-05-17 — MiniLM convenience embedder
 
 Closes the last open item in the v0.33 milestone (deferred from v0.33.0

{eval_toolkit-0.33.1 → eval_toolkit-0.35.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.33.1
+Version: 0.35.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-0.33.1 → eval_toolkit-0.35.0}/pyproject.toml

@@ -164,7 +164,7 @@ warn_no_return = true
 strict_equality = true
 
 [[tool.mypy.overrides]]
-module = ["scipy.*", "sklearn.*", "matplotlib.*", "pandas.*", "yaml.*", "sentence_transformers.*"]
+module = ["scipy.*", "sklearn.*", "matplotlib.*", "pandas.*", "yaml.*", "sentence_transformers.*", "joblib.*"]
 ignore_missing_imports = true
 
 [tool.pytest.ini_options]

{eval_toolkit-0.33.1 → eval_toolkit-0.35.0}/src/eval_toolkit/__init__.py

@@ -62,6 +62,7 @@ _EXPORTS: dict[str, str] = {
     "PairedBootstrapCI": "eval_toolkit.bootstrap",
     "ThresholdedMetricFn": "eval_toolkit.bootstrap",
     "ThresholdFn": "eval_toolkit.bootstrap",
+    "block_bootstrap_on_folds": "eval_toolkit.bootstrap",
     "bonferroni_correct": "eval_toolkit.bootstrap",
     "bootstrap_ci": "eval_toolkit.bootstrap",
     "correct_p_values": "eval_toolkit.bootstrap",
@@ -86,6 +87,7 @@ _EXPORTS: dict[str, str] = {
     "fit_isotonic_calibrator": "eval_toolkit.calibration",
     "fit_platt_calibrator": "eval_toolkit.calibration",
     "fit_temperature": "eval_toolkit.calibration",
+    "fit_temperature_binary": "eval_toolkit.calibration",
     "fit_temperature_oracle": "eval_toolkit.calibration",
     "reliability_curve": "eval_toolkit.calibration",
     "reliability_diagram_data": "eval_toolkit.calibration",
@@ -252,6 +254,7 @@ _EXPORTS: dict[str, str] = {
     "CISafeThresholdSelector": "eval_toolkit.thresholds",
     "CostSensitiveSelector": "eval_toolkit.thresholds",
     "MaxF1Selector": "eval_toolkit.thresholds",
+    "RecallAtFprResult": "eval_toolkit.thresholds",
     "TargetFPRSelector": "eval_toolkit.thresholds",
     "TargetPrecisionSelector": "eval_toolkit.thresholds",
     "TargetRecallSelector": "eval_toolkit.thresholds",
@@ -259,6 +262,7 @@ _EXPORTS: dict[str, str] = {
     "ThresholdSelector": "eval_toolkit.thresholds",
     "WilsonInterval": "eval_toolkit.thresholds",
     "YoudenJSelector": "eval_toolkit.thresholds",
+    "recall_at_fpr": "eval_toolkit.thresholds",
     "select_threshold": "eval_toolkit.thresholds",
     "wilson_interval": "eval_toolkit.thresholds",
 }

eval_toolkit-0.35.0/src/eval_toolkit/_parallel.py

@@ -0,0 +1,129 @@
+"""Toolkit-internal parallel-map helper. Single source of truth for parallelism.
+
+This module is *internal* (not exported via ``__all__`` or
+``__init__._EXPORTS``); future per-function ``n_jobs`` additions across the
+toolkit call into this helper rather than each inventing their own
+parallelism backend.
+
+See ``docs/source/methodology/parallelism.md`` for the design rationale +
+caller contract (reproducibility, picklability, smart defaults).
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import pickle
+from collections.abc import Callable, Iterable, Sized
+
+_logger = logging.getLogger(__name__)
+
+_GUIDANCE_THRESHOLD = 1000
+
+_GUIDANCE_EMITTED = False
+
+
+def parallel_map[T, R](
+    fn: Callable[[T], R],
+    items: Iterable[T],
+    *,
+    n_jobs: int = 1,
+    description: str = "work",
+) -> list[R]:
+    """Map ``fn`` over ``items``; parallel when ``n_jobs != 1``.
+
+    Design contract (see ``docs/source/methodology/parallelism.md``):
+
+    - ``n_jobs == 1`` (default) — pure-Python serial; preserves tracebacks.
+      If ``len(items) >= 1000``, emits an INFO log on the first qualifying
+      call per Python process suggesting ``n_jobs > 1`` (silent thereafter).
+    - ``n_jobs == -1`` — joblib loky with all cores.
+    - ``n_jobs > 1`` — joblib loky; values exceeding ``os.cpu_count()``
+      are silently capped (with a WARNING log) to avoid CPU-frying.
+    - ``n_jobs == 0`` — raises ``ValueError`` (likely a typo for 1 or -1).
+    - ``fn`` MUST be picklable when ``n_jobs != 1`` (lambdas and closures
+      over local state are rejected at call time with a helpful
+      ``TypeError``).
+    - Reproducibility: caller is responsible for deterministic per-item
+      state (use ``np.random.SeedSequence(seed).spawn(n)`` for resample
+      loops so ``n_jobs > 1`` produces identical results to ``n_jobs == 1``
+      for the same seed).
+
+    Parameters
+    ----------
+    fn : Callable[[T], R]
+        Picklable callable to apply to each item. Lambdas and closures
+        over local state are rejected when ``n_jobs != 1``.
+    items : Iterable[T]
+        Work items. Materialised internally; pass any iterable.
+    n_jobs : int, optional
+        Default 1 (sequential). Set to -1 for all cores, or a positive int.
+        ``n_jobs=0`` is rejected (use 1 or -1).
+    description : str, optional
+        Used in log messages and error messages for context (e.g.,
+        ``"paired bootstrap"``).
+
+    Returns
+    -------
+    list[R]
+        Results in the order of ``items``.
+
+    Raises
+    ------
+    ValueError
+        If ``n_jobs == 0``.
+    TypeError
+        If ``n_jobs != 1`` and ``fn`` is not picklable.
+
+    Examples
+    --------
+    >>> def square(x): return x * x
+    >>> parallel_map(square, [1, 2, 3], n_jobs=1)
+    [1, 4, 9]
+    """
+    if n_jobs == 0:
+        raise ValueError(
+            f"n_jobs=0 is not allowed for {description}; use 1 (sequential), "
+            "-1 (all cores), or a positive integer."
+        )
+
+    items_list = items if isinstance(items, list) else list(items)
+    n_items = len(items_list) if isinstance(items_list, Sized) else 0
+
+    if n_jobs == 1:
+        global _GUIDANCE_EMITTED
+        if n_items >= _GUIDANCE_THRESHOLD and not _GUIDANCE_EMITTED:
+            _logger.info(
+                "%s: running %d items sequentially (n_jobs=1). For parallel "
+                "speedup set n_jobs > 1 (typical wall-clock 3-5x on 8 cores). "
+                "(Shown once per process.)",
+                description,
+                n_items,
+            )
+            _GUIDANCE_EMITTED = True
+        return [fn(item) for item in items_list]
+
+    if n_jobs > 0:
+        cpu_count = os.cpu_count() or 1
+        if n_jobs > cpu_count:
+            _logger.warning(
+                "%s: capping n_jobs from %d to %d (os.cpu_count()).",
+                description,
+                n_jobs,
+                cpu_count,
+            )
+            n_jobs = cpu_count
+
+    try:
+        pickle.dumps(fn)
+    except (pickle.PicklingError, AttributeError, TypeError) as e:
+        raise TypeError(
+            f"parallel_map of {description}: callable is not picklable "
+            f"(lambdas and closures over local state are not supported "
+            f"with n_jobs != 1). Define a named top-level function. "
+            f"Underlying error: {e}"
+        ) from e
+
+    from joblib import Parallel, delayed  # noqa: PLC0415
+
+    return list(Parallel(n_jobs=n_jobs, backend="loky")(delayed(fn)(item) for item in items_list))

{eval_toolkit-0.33.1 → eval_toolkit-0.35.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.33.1"
+__version__ = "0.35.0"