eval-toolkit 0.27.2__tar.gz → 0.28.0__tar.gz

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/.gitignore

@@ -40,3 +40,6 @@ coverage.xml
 
 # Claude Code project settings (machine-local)
 .claude/
+
+# mkdocs build output (Section E.1 v0.28.0)
+/site/

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/CHANGELOG.md

@@ -7,6 +7,188 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.28.0] — 2026-05-15 — temporalcv cross-pollination bundle
+
+Six-section bundle adopting the highest-value patterns from the
+sibling `temporalcv` project plus public-repo polish + hosted docs.
+Major additions: `PurgedKFoldSplitter` for label-overlap-protected
+cross-validation, nightly Monte Carlo bootstrap CI calibration
+testing, 6-example documentation gallery, a hosted mkdocs-material
+docs site with MathJax + tikzjax for full LaTeX + TikZ rendering,
+SECURITY.md + CITATION.cff for public-repo polish, and a
+documentary mutmut audit cataloguing math-kernel test strength.
+
+### Added
+
+- Section F (mutmut audit, from temporalcv-cross-pollination bundle):
+  added `docs/internals/mutmut_audit.md` — documentary code-analysis
+  audit of the 5 math kernel modules (`metrics`, `bootstrap`,
+  `calibration`, `operating_points`, `thresholds`). Per Q10=A
+  acceptance (audit-only, no kill-rate target), the deliverable is
+  a catalog of likely surviving mutant patterns per module + an
+  assessment of whether the existing test suite would catch them.
+  Identifies 3 specific high-leverage gaps for future work:
+  (a) calibration fit-vs-eval data isolation, (b) BCa degenerate-
+  jackknife fallback assertion strengthening, (c) `empty_strategy`
+  default lock-in tests. Programmatic mutmut run deferred: mutmut
+  3.5.0 has a config-parsing bug in our env where `tests_dir =
+  "tests/"` is splat character-by-character — revisit with mutmut
+  v4 or cosmic-ray. Re-run instructions captured in the audit doc.
+
+- Section E.2 (mkdocs link cleanup, from temporalcv-cross-pollination
+  bundle): fixed 30+ broken relative links across 18 documentation
+  files. Pattern: docs that link to `../src/eval_toolkit/<X>.py`
+  (works on GitHub render but breaks in mkdocs) now point at the
+  auto-generated API reference page (`api/<X>.md`). CHANGELOG.md
+  references (also outside the docs tree) repointed to absolute
+  GitHub URLs. Down from 93 warnings to 1: the remaining
+  `griffe: π : float` is a documented tool limitation — griffe
+  doesn't parse Unicode parameter names; the project's STYLE.md
+  intentionally allows Unicode in math kernels (`π`, `α`, etc.).
+  Also patched `harness.py` RunResult docstring: replaced the Sphinx
+  `.. versionchanged::` directive with a NumPy "Notes" section so
+  mkdocstrings renders it cleanly. `mkdocs build --strict` would
+  fail on the 1 remaining griffe warning, so the docs.yml workflow
+  intentionally runs without `--strict`. The link-cleanup deliverable
+  is complete; the source-docstring + methodology enrichment passes
+  originally scoped for E.2 are deferred (existing docstrings already
+  carry References + LaTeX where it matters; methodology pages are
+  already strong content-wise — only the link structure needed fixing).
+
+- Section E.1 (hosted documentation site, from
+  temporalcv-cross-pollination bundle): new mkdocs-material site at
+  `https://brandon-behring.github.io/eval-toolkit/`, auto-generated
+  from existing Markdown docs + `mkdocstrings`-rendered API reference.
+  - `mkdocs.yml` configures the material theme (auto light/dark,
+    tabs nav, code-copy buttons, full-text search) with MathJax v3 +
+    tikzjax loaded from CDN for full LaTeX + TikZ rendering
+    (per Q12=B).
+  - `docs/index.md` — site landing page
+  - `docs/api/index.md` — curated API landing organized by README's
+    three-tier architecture (Tier 1 functional core, Tier 2 protocol
+    orchestration, Tier 3 reproducibility scaffolding); per Q8=C.
+  - `docs/api/<module>.md` — 22 per-module auto-gen stubs invoking
+    `::: eval_toolkit.<module>` mkdocstrings directives.
+  - `docs/javascripts/mathjax-config.js` — MathJax v3 init script
+    matching mkdocs-material's pymdownx.arithmatex (generic: true).
+  - `.github/workflows/docs.yml` deploys to GitHub Pages on every
+    push to main + every tag push. Single-version site (no `mike`,
+    per Q11=A).
+  - `[docs]` optional extra added to `pyproject.toml` listing
+    mkdocs-material, mkdocstrings[python], pymdown-extensions.
+  - `pyproject.urls.Documentation` repointed at the hosted-docs URL.
+  - README badge added: `Docs` linking to the GitHub Pages site.
+  - `.gitignore` extended to exclude the mkdocs build output (`/site/`).
+  - **Known follow-up**: 30+ relative-link warnings in
+    `docs/methodology/*.md` files (links to `../../src/...` and
+    `../../CHANGELOG.md`). Workflow temporarily runs without
+    `--strict`; Section E.2 will fix these and re-enable strict mode.
+
+- Section D (public-repo polish from temporalcv-cross-pollination bundle):
+  added `SECURITY.md` (security disclosure policy with response SLAs,
+  scope, and reporter-credit policy); added `CITATION.cff` (machine-
+  readable academic citation metadata, exposing the GitHub web UI
+  "Cite this repository" button — methodology-relevant primary
+  references listed for `bootstrap_ci`, `brier_score`,
+  `fit_platt_calibrator`, `delong_roc_variance`, `PurgedKFoldSplitter`).
+  Added four trust-set badges to README (CI status, PyPI version,
+  Python ≥3.13, License MIT). Extended `pyproject.urls` with a
+  `Documentation` key pointing at `docs/getting-started.md` (the
+  hosted-docs URL replaces this in Section E.1). Module-docstring
+  audit across all 22 `src/eval_toolkit/*.py` modules — all already
+  carry adequate module-level docstrings; no patches needed.
+
+- Section C (example gallery from temporalcv-cross-pollination bundle):
+  six new minimal worked examples in `docs/examples/`, each one
+  concept per file, Sybil-validated end-to-end in CI:
+  - `metrics_and_bootstrap.md` — `pr_auc` / `roc_auc` / `brier_score`
+    + `bootstrap_ci` (BCa vs percentile)
+  - `evaluate_harness.md` — slice-aware `evaluate(...)` with two
+    scorers, `write_run_result(...)`, JSON schema validation
+  - `calibration.md` — Platt + isotonic recalibration, ECE before/after
+  - `leakage_detection.md` — `ExactDuplicateCheck` +
+    `NormalizedFormLeakageCheck` + `LabelConflictCheck` on a
+    contaminated train/test pair
+  - `claims_and_gates.md` — `EvidenceGate` composition (metric
+    threshold + minimum slice size) for release-decision gating
+  - `paired_comparison.md` — `paired_bootstrap_diff` for two-scorer
+    significance + `mde_from_ci` for power analysis
+  - `index.md` — examples landing page mapping each example to the
+    capability it demonstrates + the minimum extras required
+  Total: 28 sybil-validated code blocks. Each is the headline-import
+  → usable-output minimum surface; together they cover the public API
+  surface a new user needs to be productive.
+
+- Section B (PurgedKFold splitter from temporalcv-cross-pollination bundle):
+  `PurgedKFoldSplitter(n_splits, purge_gap, embargo_pct, time_col)` and a
+  standalone `compute_label_overlap(t_train, t_test, horizon)` helper, both
+  now public via `from eval_toolkit import ...`. Time-aware k-fold with
+  explicit purge gap straddling the test fold + post-test embargo —
+  prevents label-window leakage when labels have a forward horizon
+  (e.g., H-step forward returns). The standalone helper audits arbitrary
+  train/test overlap independent of the splitter. Adapted from López de
+  Prado (2018) Chapter 7 via temporalcv's `cv_financial.py`; API names
+  preserved verbatim for cross-library muscle memory. Public-API
+  drift-guard snapshot regenerated for the two new exports.
+
+### Internal
+
+- Section A (Monte Carlo bootstrap CI calibration, from temporalcv-cross-pollination
+  bundle): added `tests/test_bootstrap_calibration_mc.py` (slow-marker) that runs
+  500-replicate MC validation of `bootstrap_ci` coverage + bias across 5 cases
+  (pr_auc / roc_auc × balanced / imbalanced × n=200 / n=1000 × BCa / percentile
+  method). Asserts empirical coverage ∈ [0.90, 0.99] for nominal 95% CIs and
+  |bias| < 0.05. Complements Tier 1's golden tests: goldens pin exact numerical
+  output (drift detection), MC tests validate that the math is correct (a buggy
+  implementation producing self-consistent wrong values fails MC but passes
+  goldens). Also added CI width-scaling test (width should shrink as ~1/√n).
+  New workflow `.github/workflows/nightly-mc.yml` triggers this suite weekly
+  on Sundays at 03:00 UTC (plus `workflow_dispatch` for manual runs). Harness
+  pattern adapted from temporalcv's `tests/conftest.py` Monte Carlo helpers.
+
+- Test coverage (Tier 1 — math kernel correctness + integration backbone):
+  added end-to-end pipeline tests (`tests/test_pipeline_e2e.py`) that
+  exercise loader → `evaluate` → `write_run_result` → JSON schema
+  validation for `DataFrameLoader` and `SingleSliceLoader` (incl.
+  paired-diffs path). Extended `tests/test_metrics_props.py` with
+  Brier-score bounds + label/score inversion symmetry properties. Added
+  bootstrap CI golden tests (`tests/test_bootstrap_golden.py`, fixture at
+  `tests/golden/bootstrap_ci/cases.json`) pinning BCa/percentile output
+  on 6 canonical stress points (balanced, imbalanced 5%, small-n=10,
+  tied scores) to ±1e-9. Expanded the `golden` pytest marker doc.
+
+- Test coverage (Tier 3 — resilience moat): added multi-slice
+  fault-injection tests (`tests/test_harness_fault_injection.py`) that
+  exercise `on_scorer_error="record"` across three slices where the
+  scorer succeeds on the middle one and fails on the outer two —
+  asserts per-(slice, scorer) independence (no error-state bleed) plus
+  a healthy-vs-faulting scorer parity check against a no-fault control.
+  Added exactness tests for `TargetFPRSelector`
+  (`tests/test_thresholds.py`): analytical answer on
+  perfectly-separable data plus a golden-style pinned threshold value
+  for a canonical (n=500, seed=42) overlapping distribution across
+  target FPRs 0.01 / 0.05 / 0.10 / 0.20, with a monotonicity invariant.
+  Added calibration determinism tests
+  (`tests/test_calibration_determinism.py`): same `(y, score)` produces
+  bit-identical Platt fit `a`/`b` parameters and isotonic transform
+  output across runs, parametrized over 1% / 50% / 99% positive
+  prevalence. Added NaN/+inf/-inf rejection tests for `pr_auc`,
+  `roc_auc`, `brier_score` to `tests/test_metrics_props.py` —
+  parametrized; locks the input-validation contract.
+
+- Test coverage (Tier 2 — public-contract + integration breadth):
+  added public-API drift guard (`tests/test_public_api.py`, fixture at
+  `tests/golden/public_api/snapshot.json`) that snapshots all 199 names
+  in `eval_toolkit.__all__` with signatures, class bases, first
+  docstring lines, and primitive-value summaries. Drift now requires an
+  explicit golden-regeneration commit. Extended
+  `tests/test_pipeline_e2e.py` with `ParquetGlobLoader` round-trip
+  (synthetic parquet → glob → load → evaluate → schema-validate; gated
+  on `pyarrow`). Extended `tests/test_artifacts.py` with four manifest
+  v2↔v3 dispatcher tests: v3 well-formed accepted; v3 missing
+  `contamination_flags` rejected; v3 with unknown enum value rejected;
+  v2 payloads still routed to v2 schema (no eager v3 demotion).
+
 ## [0.27.2] — 2026-05-15 — fix base-install pandas import
 
 Base install of `eval-toolkit` (no extras) was broken in 0.27.1: every

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/PKG-INFO

@@ -1,8 +1,9 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.27.2
+Version: 0.28.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
+Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
 Project-URL: Repository, https://github.com/brandon-behring/eval-toolkit.git
 Project-URL: Issues, https://github.com/brandon-behring/eval-toolkit/issues
 Project-URL: Changelog, https://github.com/brandon-behring/eval-toolkit/blob/main/CHANGELOG.md
@@ -49,6 +50,10 @@ Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: pyyaml>=6.0; extra == 'dev'
 Requires-Dist: ruff>=0.5; extra == 'dev'
 Requires-Dist: sybil>=10.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.24; extra == 'docs'
+Requires-Dist: pymdown-extensions>=10.7; extra == 'docs'
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=15.0; extra == 'parquet'
 Provides-Extra: plotting
@@ -63,6 +68,12 @@ Description-Content-Type: text/markdown
 
 # eval-toolkit
 
+[![CI](https://github.com/brandon-behring/eval-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/brandon-behring/eval-toolkit/actions/workflows/ci.yml)
+[![Docs](https://github.com/brandon-behring/eval-toolkit/actions/workflows/docs.yml/badge.svg)](https://brandon-behring.github.io/eval-toolkit/)
+[![PyPI version](https://img.shields.io/pypi/v/eval-toolkit.svg)](https://pypi.org/project/eval-toolkit/)
+[![Python ≥3.13](https://img.shields.io/badge/python-%E2%89%A53.13-blue.svg)](https://pypi.org/project/eval-toolkit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
+
 A **methodology-aware evaluation harness for binary classification**:
 metrics, bootstrap CIs, calibration, leakage detection, splitting,
 threshold selection, dataset loading, reproducibility manifests, and a

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/README.md

@@ -1,5 +1,11 @@
 # eval-toolkit
 
+[![CI](https://github.com/brandon-behring/eval-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/brandon-behring/eval-toolkit/actions/workflows/ci.yml)
+[![Docs](https://github.com/brandon-behring/eval-toolkit/actions/workflows/docs.yml/badge.svg)](https://brandon-behring.github.io/eval-toolkit/)
+[![PyPI version](https://img.shields.io/pypi/v/eval-toolkit.svg)](https://pypi.org/project/eval-toolkit/)
+[![Python ≥3.13](https://img.shields.io/badge/python-%E2%89%A53.13-blue.svg)](https://pypi.org/project/eval-toolkit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
+
 A **methodology-aware evaluation harness for binary classification**:
 metrics, bootstrap CIs, calibration, leakage detection, splitting,
 threshold selection, dataset loading, reproducibility manifests, and a

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/pyproject.toml

@@ -54,6 +54,15 @@ parquet = ["pyarrow>=15.0"]
 # resolve cleanly; jsonschema is now in the base deps. Remove in v0.17.0
 # after a deprecation window if no downstream consumer complains.
 validation = []
+# v0.28.0 (Section E.1): docs site via mkdocs-material + mkdocstrings.
+# Per Q4=B in the temporalcv-cross-pollination plan; LaTeX via MathJax v3
+# (configured in docs/javascripts/mathjax-config.js) and tikzjax loaded
+# from CDN in mkdocs.yml.
+docs = [
+    "mkdocs-material>=9.5",
+    "mkdocstrings[python]>=0.24",
+    "pymdown-extensions>=10.7",
+]
 # `all` references the sub-extras directly (PEP 685 self-reference). This
 # avoids drift: adding a dep to e.g. `dataframe` no longer requires a
 # mirroring edit here.
@@ -72,6 +81,7 @@ dev = [
 
 [project.urls]
 Homepage = "https://github.com/brandon-behring/eval-toolkit"
+Documentation = "https://brandon-behring.github.io/eval-toolkit/"
 Repository = "https://github.com/brandon-behring/eval-toolkit.git"
 Issues = "https://github.com/brandon-behring/eval-toolkit/issues"
 Changelog = "https://github.com/brandon-behring/eval-toolkit/blob/main/CHANGELOG.md"
@@ -146,8 +156,9 @@ markers = [
     "unit: Sklearn-reference and analytical correctness tests",
     "property: Hypothesis property-based invariant tests",
     "smoke: End-to-end smoke tests",
-    "golden: Snapshot tests for deterministic outputs (docs.py)",
+    "golden: Snapshot tests for deterministic outputs (docs renderer, bootstrap CI numerical pins, public API surface).",
     "slow: Tests > 2s (bootstrap-t studentized, multi-seed K-fold). Opt out with `pytest -m 'not slow'`.",
+    "monte_carlo: Monte Carlo calibration suite (~14 min). Skipped in PR CI; runs only in the nightly-mc workflow via `-m monte_carlo`.",
 ]
 
 [tool.coverage.run]

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/src/eval_toolkit/__init__.py

@@ -183,10 +183,12 @@ _EXPORTS: dict[str, str] = {
     "GroupKFoldSplitter": "eval_toolkit.splits",
     "HoldoutSplitter": "eval_toolkit.splits",
     "PoolBuilder": "eval_toolkit.splits",
+    "PurgedKFoldSplitter": "eval_toolkit.splits",
     "SourceDisjointKFoldSplitter": "eval_toolkit.splits",
     "Splitter": "eval_toolkit.splits",
     "StratifiedKFoldSplitter": "eval_toolkit.splits",
     "TimeSeriesSplitter": "eval_toolkit.splits",
+    "compute_label_overlap": "eval_toolkit.splits",
     "iter_folds_with_pool": "eval_toolkit.splits",
     "DEFAULT_DEDUP_THRESHOLD": "eval_toolkit.text_dedup",
     "DedupReport": "eval_toolkit.text_dedup",

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.27.2"
+__version__ = "0.28.0"

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/src/eval_toolkit/harness.py

@@ -185,9 +185,11 @@ class RunResult:
         JSON schema version. ``"v1"`` for v0.7.0+; downstream parsers gate
         on this.
 
-    .. versionchanged:: 0.7.0
-        Added ``by_fold``, ``fold_summary``, ``schema_version`` (additive,
-        defaults empty / ``"v1"`` — backward compatible).
+    Notes
+    -----
+    Changed in 0.7.0: added ``by_fold``, ``fold_summary``,
+    ``schema_version`` (additive, defaults empty / ``"v1"`` — backward
+    compatible).
     """
 
     run_id: str

{eval_toolkit-0.27.2 → eval_toolkit-0.28.0}/src/eval_toolkit/splits.py

@@ -36,10 +36,12 @@ __all__ = [
     "GroupKFoldSplitter",
     "HoldoutSplitter",
     "PoolBuilder",
+    "PurgedKFoldSplitter",
     "SourceDisjointKFoldSplitter",
     "Splitter",
     "StratifiedKFoldSplitter",
     "TimeSeriesSplitter",
+    "compute_label_overlap",
     "iter_folds_with_pool",
 ]
 
@@ -513,3 +515,283 @@ def iter_folds_with_pool(
         # PoolBuilder's keys (train, val, possibly more) take precedence;
         # test is reattached from the Splitter.
         yield {**built, "test": test}
+
+
+# ---------------------------------------------------------------------------
+# Purged K-fold for label-overlap protection (v0.28.0)
+#
+# Adapted from temporalcv (Behring 2026) for the financial / forecasting
+# label-overlap case: when labels use future data (e.g., H-day forward
+# returns), train and test folds can overlap in their LABEL windows even
+# when their FEATURE windows don't. Purging drops a band of training
+# samples within ``purge_gap`` of each test fold; embargo drops an
+# additional fraction of n samples bordering each test fold.
+# ---------------------------------------------------------------------------
+
+
+def compute_label_overlap(
+    t_train: np.ndarray,
+    t_test: np.ndarray,
+    horizon: int,
+) -> np.ndarray:
+    r"""Boolean ``(n_train, n_test)`` matrix: True where label windows overlap.
+
+    For h-step forward labels, the label at time ``t`` depends on the data
+    at times ``[t, t+h]``. Two samples ``t_train[i]`` and ``t_test[j]``
+    have label-window overlap if their windows share at least one time
+    point — equivalently, if ``|t_train[i] - t_test[j]| < horizon``.
+
+    Use this to audit whether a given train/test split has any label
+    leakage. Standalone helper; does NOT require a particular splitter.
+
+    Parameters
+    ----------
+    t_train : np.ndarray, shape (n_train,)
+        Time indices of the training set (any sortable numeric type).
+    t_test : np.ndarray, shape (n_test,)
+        Time indices of the test set.
+    horizon : int
+        Label horizon (e.g., ``5`` for 5-step forward returns). Must be
+        non-negative; ``horizon=0`` means no overlap is possible.
+
+    Returns
+    -------
+    np.ndarray, shape (n_train, n_test), dtype bool
+        Entry ``(i, j)`` is ``True`` iff
+        ``|t_train[i] - t_test[j]| < horizon``.
+
+    Raises
+    ------
+    ValueError
+        If ``horizon`` is negative.
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> t_train = np.array([0, 1, 5, 6])
+    >>> t_test = np.array([3, 4])
+    >>> overlap = compute_label_overlap(t_train, t_test, horizon=3)
+    >>> overlap
+    array([[False, False],
+           [ True, False],
+           [ True,  True],
+           [False,  True]])
+    >>> # Sample 0 (t=0): no overlap with test (|0-3|=3, |0-4|=4 ≥ horizon)
+    >>> # Sample 1 (t=1): overlaps test[0]=3 (|1-3|=2 < 3)
+    >>> # Sample 2 (t=5): overlaps both (|5-3|=2, |5-4|=1)
+    >>> # Sample 3 (t=6): overlaps test[1]=4 (|6-4|=2 < 3)
+
+    Notes
+    -----
+    The check is **symmetric in time**: ``|t_train - t_test| < horizon``
+    treats overlap in either temporal direction equally. For strictly
+    forward-only label overlap (train-before-test), filter the result
+    with ``(t_test[None, :] - t_train[:, None]) > 0``.
+
+    For h-step forward labels: label at time t covers ``[t, t+h)``, so
+    two labels at times ``t1, t2`` share data iff their intervals
+    overlap, which holds iff ``|t1 - t2| < h``.
+
+    References
+    ----------
+    .. [1] López de Prado, M. (2018). "Advances in Financial Machine
+           Learning." Wiley. Chapter 7: Cross-Validation in Finance.
+    """
+    if horizon < 0:
+        raise ValueError(f"horizon must be >= 0, got {horizon}")
+    if horizon == 0:
+        return np.zeros((len(t_train), len(t_test)), dtype=bool)
+    t_train_arr = np.asarray(t_train)
+    t_test_arr = np.asarray(t_test)
+    # Outer absolute difference: (n_train, n_test)
+    dist = np.abs(t_train_arr[:, None] - t_test_arr[None, :])
+    overlap: np.ndarray = dist < horizon
+    return overlap
+
+
+def _apply_purge_embargo(
+    test_idx: np.ndarray,
+    n_samples: int,
+    purge_gap: int,
+    embargo_pct: float,
+) -> np.ndarray:
+    """Build a training-index array excluding the test fold + purge + embargo.
+
+    The test fold's indices are contiguous (TimeSeriesSplit-style); purging
+    drops `[test_min - purge_gap, test_max + purge_gap]` from training;
+    embargo drops an additional `floor(embargo_pct * n_samples)` indices
+    after the test fold (one-sided: protects the post-test region from
+    label-window leakage when labels are forward-looking).
+
+    Adapted from temporalcv's ``_apply_purge_and_embargo`` but vectorized
+    (no Python-level set/loop) and asymmetric-by-default (embargo only on
+    the post-test side, matching López de Prado's original definition).
+    """
+    test_min = int(np.min(test_idx))
+    test_max = int(np.max(test_idx))
+    purge_start = max(0, test_min - purge_gap)
+    purge_end = min(n_samples, test_max + 1 + purge_gap)
+    n_embargo = int(embargo_pct * n_samples)
+    embargo_end = min(n_samples, test_max + 1 + n_embargo)
+
+    full_idx = np.arange(n_samples)
+    # Mask out: the test fold itself + purge band on both sides + post-test embargo
+    keep = np.ones(n_samples, dtype=bool)
+    keep[purge_start:purge_end] = False  # zeroes out test + purge band
+    keep[test_max + 1 : embargo_end] = False  # post-test embargo
+    return full_idx[keep]
+
+
+@dataclass(frozen=True, slots=True)
+class PurgedKFoldSplitter:
+    r"""Time-aware k-fold with explicit purge gap + post-test embargo.
+
+    Pattern from López de Prado (2018) Ch. 7: when labels have a forward
+    lookahead (e.g., H-step returns), train and test folds can overlap in
+    their **label windows** even when their **feature windows** don't.
+    Standard k-fold leaks information through this overlap. PurgedKFold
+    drops a ``purge_gap``-sample band straddling each test fold's boundary
+    plus a post-test ``embargo_pct * n`` window — preventing both
+    backward and forward label-overlap leakage.
+
+    Implements the :class:`Splitter` Protocol, yielding
+    ``{"train": EvalSlice, "test": EvalSlice}`` dicts.
+
+    Parameters
+    ----------
+    n_splits : int, optional
+        Number of folds. Default 5. Must be ≥ 2.
+    purge_gap : int, optional
+        Samples to drop on each side of every test fold's boundary.
+        Default 0 (no purging — equivalent to vanilla TimeSeriesSplit).
+        For h-step forward labels, ``purge_gap=h`` is the canonical choice.
+    embargo_pct : float, optional
+        Additional embargo as a fraction of total ``n``, applied **after**
+        each test fold (one-sided, López de Prado convention). Default
+        0.0. Typical: 0.01 (1%).
+    time_col : str or None, optional
+        Column carrying a sortable timestamp. If set, the parent slice is
+        sorted by this column before splitting. ``None`` assumes the slice
+        is already in temporal order. Default ``"timestamp"``.
+
+    Raises
+    ------
+    ValueError
+        At construction time if ``n_splits < 2`` or ``purge_gap < 0`` or
+        ``embargo_pct ∉ [0, 1)``.
+    KeyError
+        At ``iter_folds`` time if ``time_col`` is set but not present in
+        the slice DataFrame.
+
+    Examples
+    --------
+    >>> import pandas as pd
+    >>> from eval_toolkit.harness import EvalSlice
+    >>> from eval_toolkit.splits import PurgedKFoldSplitter
+    >>> df = pd.DataFrame({
+    ...     "text": [f"row{i}" for i in range(50)],
+    ...     "label": [i % 2 for i in range(50)],
+    ...     "t": list(range(50)),
+    ... })
+    >>> parent = EvalSlice(name="all", df=df)
+    >>> spl = PurgedKFoldSplitter(n_splits=5, purge_gap=2, embargo_pct=0.02, time_col="t")
+    >>> folds = list(spl.iter_folds(parent))
+    >>> len(folds)
+    5
+    >>> sorted(folds[0].keys())
+    ['test', 'train']
+
+    Notes
+    -----
+    **Two units in one signature**: ``purge_gap`` is an absolute count of
+    samples (int) while ``embargo_pct`` is a fraction (float). This
+    mirrors López de Prado / temporalcv conventions verbatim — users
+    moving between libraries see the same parameter names. Use the
+    standalone helper :func:`compute_label_overlap` to size ``purge_gap``
+    for a known label horizon.
+
+    See Also
+    --------
+    eval_toolkit.splits.compute_label_overlap :
+        Audit label-window overlap between arbitrary train/test sets.
+    eval_toolkit.splits.TimeSeriesSplitter :
+        Time-aware k-fold without purging — use when labels have no
+        lookahead horizon.
+
+    References
+    ----------
+    .. [1] López de Prado, M. (2018). "Advances in Financial Machine
+           Learning." Wiley. Chapter 7.
+    """
+
+    n_splits: int = 5
+    purge_gap: int = 0
+    embargo_pct: float = 0.0
+    time_col: str | None = "timestamp"
+
+    def __post_init__(self) -> None:
+        """Validate parameters."""
+        if self.n_splits < 2:
+            raise ValueError(f"n_splits must be >= 2, got {self.n_splits}")
+        if self.purge_gap < 0:
+            raise ValueError(f"purge_gap must be >= 0, got {self.purge_gap}")
+        if not 0.0 <= self.embargo_pct < 1.0:
+            raise ValueError(f"embargo_pct must be in [0, 1), got {self.embargo_pct}")
+
+    def iter_folds(
+        self,
+        slice_: EvalSlice,
+        *,
+        groups: np.ndarray | None = None,
+    ) -> Iterator[dict[str, EvalSlice]]:
+        """Yield ``n_splits`` fold dicts with purge + embargo applied.
+
+        Raises
+        ------
+        KeyError
+            If ``self.time_col`` is set but not present in ``slice_.df``.
+        """
+        if self.time_col is not None:
+            if self.time_col not in slice_.df.columns:
+                raise KeyError(
+                    f"time_col {self.time_col!r} not in slice columns " f"{list(slice_.df.columns)}"
+                )
+            sorted_df = slice_.df.sort_values(self.time_col).reset_index(drop=True)
+            sorted_slice = EvalSlice(
+                name=slice_.name,
+                df=sorted_df,
+                description=slice_.description,
+                feature_col=slice_.feature_col,
+                label_col=slice_.label_col,
+                strata_col=slice_.strata_col,
+            )
+        else:
+            sorted_slice = slice_
+
+        n_samples = len(sorted_slice.df)
+        if self.n_splits >= n_samples:
+            raise ValueError(f"n_splits ({self.n_splits}) must be < n_samples ({n_samples})")
+
+        # Fold sizes (mirrors TimeSeriesSplit / temporalcv: trailing folds
+        # absorb the remainder)
+        fold_sizes = np.full(self.n_splits, n_samples // self.n_splits)
+        fold_sizes[: n_samples % self.n_splits] += 1
+
+        current = 0
+        for fold_size in fold_sizes:
+            test_idx = np.arange(current, current + fold_size)
+            train_idx = _apply_purge_embargo(
+                test_idx,
+                n_samples=n_samples,
+                purge_gap=self.purge_gap,
+                embargo_pct=self.embargo_pct,
+            )
+            yield {
+                "train": _slice_subset(sorted_slice, train_idx, "train"),
+                "test": _slice_subset(sorted_slice, test_idx, "test"),
+            }
+            current += fold_size
+
+    def get_n_splits(self, slice_: EvalSlice) -> int:
+        """Return ``self.n_splits``."""
+        return self.n_splits

eval_toolkit-0.28.0/tests/golden/bootstrap_ci/cases.json

@@ -0,0 +1,50 @@
+{
+  "pr_auc_balanced_n200_BCa_seed42": {
+    "ci_high": 0.911879605,
+    "ci_low": 0.786756306,
+    "confidence": 0.95,
+    "method": "BCa",
+    "n_resamples": 500,
+    "point_estimate": 0.857436357
+  },
+  "pr_auc_balanced_n200_percentile_seed42": {
+    "ci_high": 0.920789971,
+    "ci_low": 0.796620442,
+    "confidence": 0.95,
+    "method": "percentile",
+    "n_resamples": 500,
+    "point_estimate": 0.857436357
+  },
+  "pr_auc_imbalanced_p05_n200_BCa_seed42": {
+    "ci_high": 0.585177068,
+    "ci_low": 0.104087605,
+    "confidence": 0.95,
+    "method": "BCa",
+    "n_resamples": 500,
+    "point_estimate": 0.277566582
+  },
+  "pr_auc_n10_percentile_seed42": {
+    "ci_high": 1.0,
+    "ci_low": 0.530416667,
+    "confidence": 0.95,
+    "method": "percentile",
+    "n_resamples": 500,
+    "point_estimate": 0.858333333
+  },
+  "roc_auc_balanced_n200_BCa_seed42": {
+    "ci_high": 0.911376276,
+    "ci_low": 0.811418319,
+    "confidence": 0.95,
+    "method": "BCa",
+    "n_resamples": 500,
+    "point_estimate": 0.8632
+  },
+  "roc_auc_tied_n40_BCa_seed42": {
+    "ci_high": 0.9725,
+    "ci_low": 0.797211394,
+    "confidence": 0.95,
+    "method": "BCa",
+    "n_resamples": 500,
+    "point_estimate": 0.915
+  }
+}