eval-toolkit 0.44.0__tar.gz → 0.46.0__tar.gz

{eval_toolkit-0.44.0 → eval_toolkit-0.46.0}/.gitignore

@@ -39,6 +39,12 @@ coverage.json
 # Logs
 *.log
 
+# Local environment overrides (machine-local credentials / config)
+.env.local
+
+# Mutation-testing output (mutmut / cargo-mutants — local run artifacts)
+mutants/
+
 # Claude Code project settings (machine-local)
 .claude/
 

{eval_toolkit-0.44.0 → eval_toolkit-0.46.0}/CHANGELOG.md

@@ -5,6 +5,128 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.46.0] — 2026-05-21 — Scorecard: primary v1.0 metric surface (closes #36)
+
+Second minor of the staggered v0.45 → v0.46 → v0.47 → v0.48 → v1.0 sequence.
+**Soft-breaking** — existing top-level scalar metric imports still work but
+emit `DeprecationWarning` (hard-removed at v0.47).
+
+See `docs/source/migration/v0.46.md` for the full consumer migration guide and
+`docs/source/adr/0002-scorecard-as-primary-metric-surface.md` for the
+decision record.
+
+### Added
+
+- **`eval_toolkit.scorecard(y_true, y_score, metrics=[...], bootstrap=True)`**
+  — primary v1.0 metric surface. Single call computes multiple threshold-free
+  metrics + bootstrap CIs on one slice; returns a `Scorecard` (read-only
+  `Mapping[str, MetricResult]`). Type-safe dict-subscript access; status-aware
+  cells; per-cell error isolation.
+- **`MetricSpec` Protocol** — v1.0 Tier-2 contract; `name: str` +
+  `compute(y_true, y_score) -> float`. Custom user specs satisfy structurally.
+- **`MetricResult`** frozen dataclass — `value: float | None`, `status:
+  Literal["ok", "skipped", "error"]`, `reason: str`, `ci: BootstrapCI | None`.
+  Reuses the existing `MetricState` vocabulary from `artifacts.py:30-61`.
+- **`Scorecard`** read-only `Mapping[str, MetricResult]` — `to_dict()`
+  JSON-friendly, `to_pandas()` one-row DataFrame (lazy pandas import).
+- **`eval_toolkit.metric_specs`** namespace submodule with threshold-free
+  first-party specs:
+  - `pr_auc`, `roc_auc`, `brier` — module-level singletons (identity stable).
+  - `ece(n_bins, strategy)` — LRU-cached factory (identity stable per kwargs).
+- **`SINGLE_CLASS_INCOMPATIBLE_METRICS`** extended with `pr_auc` / `roc_auc`
+  aliases (alongside existing `auroc` / `auprc`) so the v0.46 scorecard
+  surface and the v0.39 harness paths both produce correct skipped-status
+  behavior. Non-breaking; doctest + unit tests added.
+- **`docs/source/adr/0002-scorecard-as-primary-metric-surface.md`** —
+  decision record covering single-surface rationale, threshold-free scope,
+  Tier-2 Protocol commitment, and v2.0 trigger conditions.
+- **`docs/source/migration/v0.46.md`** — consumer migration guide with
+  side-by-side recipes for every common pattern.
+
+### Deprecated
+
+The following 8 top-level scalar imports emit `DeprecationWarning` and will
+be hard-removed at v0.47.0. Use `scorecard()` + `metric_specs` or the
+`eval_toolkit.metrics` submodule path (internal API, no warning).
+
+- `pr_auc`, `roc_auc`, `brier_score`
+- `expected_calibration_error`
+- `expected_calibration_error_debiased`
+- `expected_calibration_error_equal_mass`
+- `expected_calibration_error_l2`
+- `expected_calibration_error_l2_debiased`
+
+### Audit findings integrated (Round 5)
+
+Per `docs/source/audit_findings.md`:
+
+- **F1** (scorecard threshold semantics) — addressed by Decision R: ship
+  threshold-free first-party specs only at v0.46. Threshold-dependent
+  metrics (F1, accuracy, precision, recall) deferred to v1.x with explicit
+  operating-point provenance.
+- **F2** (scorecard cell-state semantics) — addressed by Decision S: reuse
+  existing `MetricState` (`ok`/`skipped`/`error`) vocabulary.
+- **F4** (deprecation shim must extend the lazy resolver, not replace it) —
+  addressed: `__getattr__` deprecation branch sits between `__version__`
+  short-circuit and the base `_EXPORTS` lookup; tagged with BEGIN/END
+  TRANSITIONAL markers for clean v0.47 removal. Tests guard that every
+  remaining `_EXPORTS` symbol still resolves.
+- **X.2 precondition** — `is_metric_defined_for_slice` aliases shipped
+  ahead of v0.46 (PR #62).
+
+### Protocol stability
+
+Tier-2 streak continues: 7 of 7 consecutive minors (v0.40–v0.46) without
+method-shape edits to any existing Tier-2 Protocol. `MetricSpec` is a NEW
+Tier-2 Protocol added at v0.46; freezes at v1.0.
+
+## [0.45.0] — 2026-05-21 — Stacking: MetaLearner Protocol + LogisticStacker (closes #52)
+
+First minor of the staggered v0.45 → v0.46 → v0.47 → v0.48 → v1.0 sequence
+(per the v1.0 plan at `~/.claude/plans/evaluate-all-the-work-twinkly-kite.md`).
+Non-breaking — purely additive. No Protocol shape edits to the existing 6
+Tier-2 contracts (Gate 2 streak continues: 6 of 6 consecutive minors without
+Protocol-shape changes).
+
+### Added
+
+- `eval_toolkit.stacking` — new module providing the `MetaLearner` Protocol
+  and one reference impl, `LogisticStacker`, for combining outputs from
+  multiple binary detectors into a calibrated ensemble. Wraps
+  `sklearn.linear_model.LogisticRegression` with a stacker-shaped public API
+  (sklearn-style `fit(score_matrix, y)`, `predict(score_matrix)`,
+  `predict_proba(score_matrix)`, plus `coef_` / `classes_` / `intercept_`
+  attributes). No new dependencies — `scikit-learn` is already core since
+  v0.27. Closes #52.
+- `MetaLearner` Protocol — `@runtime_checkable`; sklearn-shape contract
+  taking a `(n_samples, n_detectors)` score matrix. Sized as a v1.0 Tier-2
+  contract per the v1.0 plan Decision M (tiered stability — strict freeze at
+  v1.0; additive subprotocols permitted in minor releases). Mirrors the
+  `Probe` Protocol pattern from v0.43.
+- `LogisticStacker` reference impl — configurable C, fit_intercept,
+  class_weight, penalty, solver, max_iter, random_state. Class-weight default
+  `"balanced"` for the common imbalanced-detection setting. Composes with the
+  4-binary-calibrator family (v0.40 + v0.42) via `fit_platt_binary` /
+  `fit_isotonic_binary` chaining on stacked output.
+- 24-test coverage in `tests/test_stacking.py`: Protocol satisfaction (both
+  structural and duck-typed), shape contracts (3-detector × 500-sample
+  fixtures), regularization behavior (C, L1 penalty), signal ordering,
+  calibration chaining (Platt + Isotonic), bootstrap CI on stacker output
+  (Audit F6a-aware — uses correct `BootstrapCI.ci_low/ci_high` attribute
+  names), determinism under fixed `random_state`, hypothesis property on
+  signal monotonicity, input validation (shape mismatch, single-class,
+  non-finite, unfit, wrong-n-detectors).
+- `docs/source/examples/stacking.md` — myst-nb worked example: 3 synthetic
+  detectors with descending signal-to-noise, stacker fit, post-stacking
+  Platt calibration. Cites Wolpert 1992 + Breiman 1996.
+
+### Notes
+
+- Sklearn 1.8+ deprecates `LogisticRegression(penalty=...)` in favor of
+  `l1_ratio`. The public `LogisticStacker(penalty=...)` API is preserved;
+  internal sklearn-side migration to `l1_ratio` will land when sklearn 1.10
+  lands and the warning becomes more visible. No user-facing impact.
+
 ## [0.44.0] — 2026-05-19 — Defenses + losses: Spotlighting variants + RecallAtLowFPR (closes #50, #51)
 
 ### Added

{eval_toolkit-0.44.0 → eval_toolkit-0.46.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.44.0
+Version: 0.46.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

eval_toolkit-0.46.0/docs/source/adr/README.md

@@ -0,0 +1,76 @@
+# Architecture Decision Records
+
+This directory captures architecturally-significant decisions that shape
+`eval-toolkit`'s long-term design. ADRs are immutable historical records —
+once accepted, a decision is not edited in place; if it changes, a new ADR
+supersedes it.
+
+## When to file an ADR
+
+File a new ADR when a decision:
+
+- **Locks in an interface or shape** that future code is expected to
+  conform to (e.g., "metrics return type", "Protocol vs ABC").
+- **Closes off alternatives** that were seriously considered, so the
+  reasoning isn't lost.
+- **Carries cost** to reverse (e.g., a public-API contract that promises
+  stability across a release line).
+
+Routine refactors, bug fixes, and internal-only patterns do not need ADRs —
+the commit message + CHANGELOG entry are enough.
+
+## Numbering
+
+Sequential, zero-padded: `0001-flat-module-layout.md`,
+`0002-scorecard-as-primary-metric-surface.md`, etc. Number is assigned
+at the time of writing; if two ADRs are drafted in parallel, the second
+to merge takes the next number.
+
+## Format
+
+Each ADR uses this skeleton (loosely based on MADR — Markdown ADR — without
+the heavyweight template):
+
+```markdown
+# ADR NNNN: Title
+
+**Status:** Proposed | Accepted | Superseded by ADR-MMMM
+**Date:** YYYY-MM-DD
+**Deciders:** (names or roles)
+
+## Context
+
+What's the situation that requires a decision? What constraints are at play?
+
+## Decision
+
+What did we decide?
+
+## Consequences
+
+What follows from this decision? (Both positive and negative.)
+
+## Alternatives considered
+
+What else was on the table, and why wasn't it chosen?
+
+## Trigger to revisit
+
+What would have to change for this decision to be reopened?
+(Optional but useful — keeps the ADR self-documenting.)
+```
+
+## Cross-references
+
+- [`docs/RELEASING.md`](../../RELEASING.md) — release-flow process; ADRs
+  are typically drafted as part of release prep.
+- [`docs/source/roadmap.md`](../roadmap.md) — long-term direction;
+  ADRs explain how individual roadmap decisions were made.
+
+## Index
+
+(Updated as ADRs are added.)
+
+| # | Title | Status | Date |
+|---|---|---|---|
+| _none yet_ | | | |

{eval_toolkit-0.44.0 → eval_toolkit-0.46.0}/src/eval_toolkit/__init__.py

@@ -193,20 +193,18 @@ _EXPORTS: dict[str, str] = {
     "SINGLE_CLASS_INCOMPATIBLE_METRICS": "eval_toolkit.metrics",
     "ThresholdResult": "eval_toolkit.metrics",
     "brier_decomposition": "eval_toolkit.metrics",
-    "brier_score": "eval_toolkit.metrics",
-    "expected_calibration_error": "eval_toolkit.metrics",
-    "expected_calibration_error_debiased": "eval_toolkit.metrics",
-    "expected_calibration_error_equal_mass": "eval_toolkit.metrics",
-    "expected_calibration_error_l2": "eval_toolkit.metrics",
-    "expected_calibration_error_l2_debiased": "eval_toolkit.metrics",
+    # `brier_score`, `pr_auc`, `roc_auc`, and the 5 ECE variants removed from
+    # `_EXPORTS` at v0.46 (Decision L). They remain reachable at the top
+    # level via the `__getattr__` deprecation branch (emits
+    # `DeprecationWarning`; branch removed at v0.47) and via the metrics
+    # submodule (`from eval_toolkit.metrics import pr_auc` — internal API
+    # per ADR 0002, not part of the v1.0 stability contract).
     "headline_metrics": "eval_toolkit.metrics",
     "is_metric_defined_for_slice": "eval_toolkit.metrics",
     "metrics_at_threshold": "eval_toolkit.metrics",
-    "pr_auc": "eval_toolkit.metrics",
     "precision_at_prior": "eval_toolkit.metrics",
     "quantile_stratified_pr_auc": "eval_toolkit.metrics",
     "quantile_stratified_report": "eval_toolkit.metrics",
-    "roc_auc": "eval_toolkit.metrics",
     "score_distribution_summary": "eval_toolkit.metrics",
     "single_class_threshold_metrics": "eval_toolkit.metrics",
     "stratified_recall": "eval_toolkit.metrics",
@@ -294,15 +292,70 @@ _EXPORTS: dict[str, str] = {
     "recall_at_fpr": "eval_toolkit.thresholds",
     "select_threshold": "eval_toolkit.thresholds",
     "wilson_interval": "eval_toolkit.thresholds",
+    "LogisticStacker": "eval_toolkit.stacking",
+    "MetaLearner": "eval_toolkit.stacking",
+    "MetricResult": "eval_toolkit._scorecard",
+    "MetricSpec": "eval_toolkit._scorecard",
+    "Scorecard": "eval_toolkit._scorecard",
+    "scorecard": "eval_toolkit._scorecard",
 }
 
 __all__ = ["__version__", *_EXPORTS.keys()]
 
 
+# ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
+# At v0.46 the scalar metric functions left the top-level `_EXPORTS` map (above)
+# in favor of the `scorecard()` surface (Decision A). To give the consumer one
+# release of overlap before the hard removal at v0.47, the names below remain
+# reachable via the package-level `__getattr__` (which delegates to the
+# `eval_toolkit.metrics` submodule) but emit a `DeprecationWarning` on first
+# lookup pointing at the new API.
+#
+# WHY THIS IS A BRANCH, NOT A REPLACEMENT (Audit F4 — Round 5):
+# `__getattr__` below is the load-bearing lazy export resolver for every name
+# in `_EXPORTS`. The deprecation branch is a discrete `if name in
+# _DEPRECATED_SCALARS` check ABOVE the resolver — the resolver's existing
+# behavior for non-deprecated names is unchanged. At v0.47 we delete this
+# transitional block and the resolver continues to work for every remaining
+# `_EXPORTS` entry.
+_DEPRECATED_SCALARS: frozenset[str] = frozenset(
+    {
+        "pr_auc",
+        "roc_auc",
+        "brier_score",
+        "expected_calibration_error",
+        "expected_calibration_error_debiased",
+        "expected_calibration_error_equal_mass",
+        "expected_calibration_error_l2",
+        "expected_calibration_error_l2_debiased",
+    }
+)
+# ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
+
+
 def __getattr__(name: str) -> Any:
     """Resolve public symbols lazily."""
     if name == "__version__":
         return __version__
+    # ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
+    if name in _DEPRECATED_SCALARS:
+        import warnings
+
+        warnings.warn(
+            f"eval_toolkit.{name} is deprecated and will be removed in v0.47. "
+            f"Use `scorecard(y, s, metrics=[metric_specs.{_scorecard_spec_for(name)}])"
+            f'["{_scorecard_spec_for(name)}"].value` instead, or "import from the'
+            f" `eval_toolkit.metrics` submodule directly (internal API).",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        module = import_module("eval_toolkit.metrics")
+        value = getattr(module, name)
+        # Do NOT cache in globals() — repeated lookups should keep re-warning
+        # (one warning per call site, modulo Python's default
+        # DeprecationWarning de-duplication).
+        return value
+    # ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
     module_name = _EXPORTS.get(name)
     if module_name is None:
         raise AttributeError(f"module 'eval_toolkit' has no attribute {name!r}")
@@ -312,6 +365,29 @@ def __getattr__(name: str) -> Any:
     return value
 
 
+# ── BEGIN TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
+def _scorecard_spec_for(deprecated_name: str) -> str:
+    """Map a deprecated-scalar name to its `metric_specs` replacement name.
+
+    Used only inside the v0.46 deprecation warning message. Returns the
+    closest equivalent first-party spec name where one exists; falls back
+    to the original name for ECE variants whose exact-match spec isn't in
+    the v0.46 first-party namespace (e.g., the L2 / debiased variants —
+    callers either implement a custom `MetricSpec` or stay on the
+    submodule path).
+    """
+    return {
+        "pr_auc": "pr_auc",
+        "roc_auc": "roc_auc",
+        "brier_score": "brier",
+        "expected_calibration_error": "ece(n_bins=10)",
+        "expected_calibration_error_equal_mass": 'ece(n_bins=10, strategy="quantile")',
+    }.get(deprecated_name, deprecated_name)
+
+
+# ── END TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
+
+
 def __dir__() -> list[str]:
     """Expose lazy public symbols to introspection."""
     return sorted(__all__)