eval-toolkit 0.33.0__tar.gz → 0.34.0__tar.gz

{eval_toolkit-0.33.0 → eval_toolkit-0.34.0}/CHANGELOG.md

@@ -7,6 +7,146 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.34.0] — 2026-05-17 — Phase 4 stats unblockers + unified parallelism + cookbook (BREAKING)
+
+Closes all 7 open backlog issues in one consumer-closing release. Also
+lands the toolkit's first unified parallelism story: a shared internal
+`parallel_map` helper + `n_jobs` kwarg on all 5 public bootstrap
+functions. Future iterations will mechanically extend the same helper to
+harness + operating-points (follow-up issues filed).
+
+### Breaking changes
+
+- **`eval_toolkit.bootstrap.mde_from_ci`**: parameter renamed from
+  `paired` to `ci` and type widened to `BootstrapCI | PairedBootstrapCI`
+  (was `PairedBootstrapCI`-only). Positional callers unaffected; keyword
+  callers must update:
+  ```python
+  mde_from_ci(paired=x)  # v0.33.x and earlier
+  mde_from_ci(ci=x)      # v0.34.0+
+  mde_from_ci(x)         # positional form, unchanged
+  ```
+  This is a one-time exception to the repo's 2-minor-version deprecation
+  warning policy (justification + criteria recorded in
+  [`docs/source/DEPRECATION.md`](docs/source/DEPRECATION.md#one-time-exceptions-to-the-2-minor-version-warning-policy)).
+  Notification issues filed on the 2 known toolkit consumers
+  (`prompt-injection-detection-submission`, `post-transformers`). Audit
+  confirms both use positional form — zero actual breakage in practice.
+
+### Added
+
+- `eval_toolkit.bootstrap.block_bootstrap_on_folds` — CV-aware sibling
+  to `cv_clt_ci`; resamples K folds with replacement; returns
+  `BootstrapCI(method="block_bootstrap")`. The A-008 sensitivity-check
+  pattern (block-bootstrap halfwidth / cv_clt halfwidth > 1.5 flags
+  LODO non-exchangeability) is the prototypical use. Closes #21.
+- `eval_toolkit.RecallAtFprResult` (frozen dataclass) +
+  `eval_toolkit.recall_at_fpr(y_true, y_score, target_fpr)` — one-shot
+  recall + actual_fpr + FP/TN at the smallest threshold meeting FPR ≤
+  target. Use `.to_dict()` for JSON / pandas-row integration. Closes #9.
+- New optional `n_jobs: int = 1` kwarg on 5 bootstrap functions:
+  `bootstrap_ci`, `paired_bootstrap_diff`, `paired_bootstrap_ece_diff`,
+  `paired_bootstrap_op_point_diff`, `paired_mde` (via `_bootstrap_t_ci`
+  internal helper). Backed by a new internal `_parallel.parallel_map`
+  helper (joblib loky; not exported). `n_jobs > 1` reproduces `n_jobs=1`
+  result bit-for-bit for the same seed (via
+  `np.random.SeedSequence.spawn`). Lambda metrics rejected at call time
+  with helpful `TypeError`. `n_jobs > os.cpu_count()` is auto-capped
+  with WARNING log; `n_jobs=0` raises `ValueError`. Closes #17.
+- 6 new pages in `docs/source/examples/`:
+  - **Cookbook** (closes #19): `nested_seed_split.md`,
+    `callable_embedder_dedup.md`, `cross_corpus_contamination_scan.md`.
+  - **Plotting walkthroughs**: `plot_roc_curve_walkthrough.md`,
+    `plot_pareto_frontier_walkthrough.md`,
+    `plot_slice_metric_heatmap_walkthrough.md` (backfills the v0.33.0
+    docs gap).
+- `docs/source/methodology/parallelism.md` — design rationale + caller
+  contract for the toolkit-wide parallelism story. Documents the 6
+  design principles (single backend, single helper, opt-in per-fn,
+  default sequential, reproducibility via SeedSequence, picklability
+  surface) and the checklist for adding `n_jobs` to a new function.
+
+### Changed
+
+- `eval_toolkit.bootstrap.mde_from_ci` now accepts
+  `BootstrapCI | PairedBootstrapCI` (was paired-only). See **Breaking
+  changes** above. Closes #20.
+- `eval_toolkit.build_manifest` gains `config_path: Path | str | None`
+  kwarg; when supplied, `config_hash` is computed as
+  `sha256(Path(config_path).read_bytes()).hexdigest()` — capturing the
+  exact YAML file bytes including comments + key ordering (which the
+  default canonical-JSON path strips during parse). Default behavior
+  preserved when `config_path is None`. Closes #10.
+
+### Internal
+
+- New `src/eval_toolkit/_parallel.py` (internal; not exported) — single
+  source of truth for parallelism. Future per-function `n_jobs`
+  additions will reuse this helper. The toolkit's first INFO-level
+  log site is here (once-per-process guidance log when `n_jobs=1` AND
+  iteration count ≥ 1000). New `tests/test_parallel.py` covers smart-
+  default semantics + reproducibility contract.
+- New golden test `tests/golden/test_dedup_holdout_calibration.py`
+  exercising 3 deterministic `SimilarityStrategy` variants against a
+  migrated 50-pair adversarial fixture at thresholds {0.75, 0.80, 0.85}
+  (strict snapshot at `tests/golden/data/dedup_holdout_expected.json`)
+  plus an `EmbeddingCosineStrategy` soft-bound check (FPR < 0.5,
+  FNR < 0.5 at threshold 0.80) gated by `pytest.importorskip` +
+  `@pytest.mark.slow`. Refresh helper at
+  `scripts/refresh_dedup_holdout.py`. Closes #18.
+- `CONTRIBUTING.md` + `docs/source/repo-strategy.md` updated with
+  explicit "Parallelism" section codifying the new pattern (was an
+  implicit anti-pattern before; v0.34.0 codifies the new opt-in design).
+- `docs/source/DEPRECATION.md` extended with a "One-time exceptions"
+  section documenting the `mde_from_ci` rename + criteria future
+  exceptions must satisfy.
+- RNG-stream note: the 5 wired bootstrap fns now derive per-resample
+  seeds via `np.random.SeedSequence(seed).spawn(n_resamples)` instead
+  of sequential calls on a single `Generator`. The bootstrap output is
+  *statistically equivalent* (both are valid bootstraps) but the exact
+  numerical CI bounds for the same caller-supplied `seed` will differ
+  slightly from v0.33.x. Existing tests use behavioral assertions
+  (`overlaps_zero`, `delta`, etc.) that are robust to the RNG-stream
+  change; tests that pin exact CI bounds (e.g., consumer golden tests)
+  may need regen on upgrade.
+
+## [0.33.1] — 2026-05-17 — MiniLM convenience embedder
+
+Closes the last open item in the v0.33 milestone (deferred from v0.33.0
+per the planned split). Ships the canonical semantic-dedup recipe
+(`sentence-transformers/all-MiniLM-L6-v2` at cosine ≥ 0.80, per ADR-027)
+pre-wired for `EmbeddingCosineStrategy` so consumers stop reinventing the
+embedder-wrapping boilerplate.
+
+No breaking changes. Public API gains 1 new export
+(`make_minilm_embedder`) and 1 new optional dependency extra
+(`[embeddings]`). Existing `EmbeddingCosineStrategy` callers that already
+ship their own embedder are unaffected.
+
+### Added
+
+- `eval_toolkit.embeddings.make_minilm_embedder` — factory returning a
+  `Callable[[Sequence[str]], np.ndarray]` that loads
+  `sentence-transformers/all-MiniLM-L6-v2` (configurable), memoises model
+  loads via `functools.lru_cache(maxsize=8)`, and emits `(n, 384)`
+  `float64` embeddings ready for `EmbeddingCosineStrategy`. Raises a
+  helpful `ImportError` with the install hint when the optional dep is
+  absent. Closes #3.
+- New optional dependency extra `[embeddings]` →
+  `sentence-transformers>=3.0`. Intentionally **not** in `[all]` / `[dev]`
+  because the transitive `torch` install (~700MB) would balloon
+  contributor setup.
+
+### Internal
+
+- `docs/source/api/embeddings.md` Sphinx page added (autosummary stub);
+  wired into the API toctree alongside the other module pages.
+- `docs/source/api/plotting.md` autosummary backfilled with
+  `plot_roc_curve`, `plot_pareto_frontier`, `plot_slice_metric_heatmap`
+  (missed in v0.33.0).
+- `tool.mypy.overrides` extended with `sentence_transformers.*` (matches
+  the existing pattern for untyped third-party libs).
+
 ## [0.33.0] — 2026-05-17 — Plotting batch + ax= parity + CI quality-of-life
 
 Consumer-unblocking release: closes the four upstream-gap TODOs in

{eval_toolkit-0.33.0 → eval_toolkit-0.34.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.33.0
+Version: 0.34.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -60,6 +60,8 @@ Requires-Dist: sphinx-autodoc-typehints>=2.0; extra == 'docs'
 Requires-Dist: sphinx-copybutton>=0.5; extra == 'docs'
 Requires-Dist: sphinx-design>=0.6; extra == 'docs'
 Requires-Dist: sphinx>=7.3; extra == 'docs'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=3.0; extra == 'embeddings'
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=15.0; extra == 'parquet'
 Provides-Extra: plotting

{eval_toolkit-0.33.0 → eval_toolkit-0.34.0}/pyproject.toml

@@ -50,6 +50,12 @@ plotting = ["matplotlib>=3.8", "pillow>=10.0"]
 property = ["hypothesis>=6.100"]
 yaml = ["pyyaml>=6.0"]
 parquet = ["pyarrow>=15.0"]
+# v0.33.1: MiniLM convenience embedder for EmbeddingCosineStrategy.
+# sentence-transformers transitively pulls torch + tokenizers (~700MB).
+# Intentionally NOT in [all] / [dev] — opt-in only to keep contributor
+# setup small. The canonical semantic-dedup recipe (all-MiniLM-L6-v2 +
+# cosine@0.80) is what this factory pre-wires for callers.
+embeddings = ["sentence-transformers>=3.0"]
 # DEPRECATED (announced v0.30.1, removal v0.33.0).
 #
 # Retained as a transitive no-op so `pip install eval-toolkit[validation]`
@@ -158,7 +164,7 @@ warn_no_return = true
 strict_equality = true
 
 [[tool.mypy.overrides]]
-module = ["scipy.*", "sklearn.*", "matplotlib.*", "pandas.*", "yaml.*"]
+module = ["scipy.*", "sklearn.*", "matplotlib.*", "pandas.*", "yaml.*", "sentence_transformers.*", "joblib.*"]
 ignore_missing_imports = true
 
 [tool.pytest.ini_options]

{eval_toolkit-0.33.0 → eval_toolkit-0.34.0}/src/eval_toolkit/__init__.py

@@ -62,6 +62,7 @@ _EXPORTS: dict[str, str] = {
     "PairedBootstrapCI": "eval_toolkit.bootstrap",
     "ThresholdedMetricFn": "eval_toolkit.bootstrap",
     "ThresholdFn": "eval_toolkit.bootstrap",
+    "block_bootstrap_on_folds": "eval_toolkit.bootstrap",
     "bonferroni_correct": "eval_toolkit.bootstrap",
     "bootstrap_ci": "eval_toolkit.bootstrap",
     "correct_p_values": "eval_toolkit.bootstrap",
@@ -117,6 +118,8 @@ _EXPORTS: dict[str, str] = {
     "render_files": "eval_toolkit.docs",
     "render_text": "eval_toolkit.docs",
     "walk_path": "eval_toolkit.docs",
+    # --- embeddings ---
+    "make_minilm_embedder": "eval_toolkit.embeddings",
     # --- evidence ---
     "AggregateEvidence": "eval_toolkit.evidence",
     "EvidenceAxis": "eval_toolkit.evidence",
@@ -250,6 +253,7 @@ _EXPORTS: dict[str, str] = {
     "CISafeThresholdSelector": "eval_toolkit.thresholds",
     "CostSensitiveSelector": "eval_toolkit.thresholds",
     "MaxF1Selector": "eval_toolkit.thresholds",
+    "RecallAtFprResult": "eval_toolkit.thresholds",
     "TargetFPRSelector": "eval_toolkit.thresholds",
     "TargetPrecisionSelector": "eval_toolkit.thresholds",
     "TargetRecallSelector": "eval_toolkit.thresholds",
@@ -257,6 +261,7 @@ _EXPORTS: dict[str, str] = {
     "ThresholdSelector": "eval_toolkit.thresholds",
     "WilsonInterval": "eval_toolkit.thresholds",
     "YoudenJSelector": "eval_toolkit.thresholds",
+    "recall_at_fpr": "eval_toolkit.thresholds",
     "select_threshold": "eval_toolkit.thresholds",
     "wilson_interval": "eval_toolkit.thresholds",
 }

eval_toolkit-0.34.0/src/eval_toolkit/_parallel.py

@@ -0,0 +1,129 @@
+"""Toolkit-internal parallel-map helper. Single source of truth for parallelism.
+
+This module is *internal* (not exported via ``__all__`` or
+``__init__._EXPORTS``); future per-function ``n_jobs`` additions across the
+toolkit call into this helper rather than each inventing their own
+parallelism backend.
+
+See ``docs/source/methodology/parallelism.md`` for the design rationale +
+caller contract (reproducibility, picklability, smart defaults).
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import pickle
+from collections.abc import Callable, Iterable, Sized
+
+_logger = logging.getLogger(__name__)
+
+_GUIDANCE_THRESHOLD = 1000
+
+_GUIDANCE_EMITTED = False
+
+
+def parallel_map[T, R](
+    fn: Callable[[T], R],
+    items: Iterable[T],
+    *,
+    n_jobs: int = 1,
+    description: str = "work",
+) -> list[R]:
+    """Map ``fn`` over ``items``; parallel when ``n_jobs != 1``.
+
+    Design contract (see ``docs/source/methodology/parallelism.md``):
+
+    - ``n_jobs == 1`` (default) — pure-Python serial; preserves tracebacks.
+      If ``len(items) >= 1000``, emits an INFO log on the first qualifying
+      call per Python process suggesting ``n_jobs > 1`` (silent thereafter).
+    - ``n_jobs == -1`` — joblib loky with all cores.
+    - ``n_jobs > 1`` — joblib loky; values exceeding ``os.cpu_count()``
+      are silently capped (with a WARNING log) to avoid CPU-frying.
+    - ``n_jobs == 0`` — raises ``ValueError`` (likely a typo for 1 or -1).
+    - ``fn`` MUST be picklable when ``n_jobs != 1`` (lambdas and closures
+      over local state are rejected at call time with a helpful
+      ``TypeError``).
+    - Reproducibility: caller is responsible for deterministic per-item
+      state (use ``np.random.SeedSequence(seed).spawn(n)`` for resample
+      loops so ``n_jobs > 1`` produces identical results to ``n_jobs == 1``
+      for the same seed).
+
+    Parameters
+    ----------
+    fn : Callable[[T], R]
+        Picklable callable to apply to each item. Lambdas and closures
+        over local state are rejected when ``n_jobs != 1``.
+    items : Iterable[T]
+        Work items. Materialised internally; pass any iterable.
+    n_jobs : int, optional
+        Default 1 (sequential). Set to -1 for all cores, or a positive int.
+        ``n_jobs=0`` is rejected (use 1 or -1).
+    description : str, optional
+        Used in log messages and error messages for context (e.g.,
+        ``"paired bootstrap"``).
+
+    Returns
+    -------
+    list[R]
+        Results in the order of ``items``.
+
+    Raises
+    ------
+    ValueError
+        If ``n_jobs == 0``.
+    TypeError
+        If ``n_jobs != 1`` and ``fn`` is not picklable.
+
+    Examples
+    --------
+    >>> def square(x): return x * x
+    >>> parallel_map(square, [1, 2, 3], n_jobs=1)
+    [1, 4, 9]
+    """
+    if n_jobs == 0:
+        raise ValueError(
+            f"n_jobs=0 is not allowed for {description}; use 1 (sequential), "
+            "-1 (all cores), or a positive integer."
+        )
+
+    items_list = items if isinstance(items, list) else list(items)
+    n_items = len(items_list) if isinstance(items_list, Sized) else 0
+
+    if n_jobs == 1:
+        global _GUIDANCE_EMITTED
+        if n_items >= _GUIDANCE_THRESHOLD and not _GUIDANCE_EMITTED:
+            _logger.info(
+                "%s: running %d items sequentially (n_jobs=1). For parallel "
+                "speedup set n_jobs > 1 (typical wall-clock 3-5x on 8 cores). "
+                "(Shown once per process.)",
+                description,
+                n_items,
+            )
+            _GUIDANCE_EMITTED = True
+        return [fn(item) for item in items_list]
+
+    if n_jobs > 0:
+        cpu_count = os.cpu_count() or 1
+        if n_jobs > cpu_count:
+            _logger.warning(
+                "%s: capping n_jobs from %d to %d (os.cpu_count()).",
+                description,
+                n_jobs,
+                cpu_count,
+            )
+            n_jobs = cpu_count
+
+    try:
+        pickle.dumps(fn)
+    except (pickle.PicklingError, AttributeError, TypeError) as e:
+        raise TypeError(
+            f"parallel_map of {description}: callable is not picklable "
+            f"(lambdas and closures over local state are not supported "
+            f"with n_jobs != 1). Define a named top-level function. "
+            f"Underlying error: {e}"
+        ) from e
+
+    from joblib import Parallel, delayed  # noqa: PLC0415
+
+    return list(Parallel(n_jobs=n_jobs, backend="loky")(delayed(fn)(item) for item in items_list))

{eval_toolkit-0.33.0 → eval_toolkit-0.34.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.33.0"
+__version__ = "0.34.0"