eval-toolkit 0.45.0__tar.gz → 0.46.0__tar.gz

{eval_toolkit-0.45.0 → eval_toolkit-0.46.0}/CHANGELOG.md

@@ -5,6 +5,81 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.46.0] — 2026-05-21 — Scorecard: primary v1.0 metric surface (closes #36)
+
+Second minor of the staggered v0.45 → v0.46 → v0.47 → v0.48 → v1.0 sequence.
+**Soft-breaking** — existing top-level scalar metric imports still work but
+emit `DeprecationWarning` (hard-removed at v0.47).
+
+See `docs/source/migration/v0.46.md` for the full consumer migration guide and
+`docs/source/adr/0002-scorecard-as-primary-metric-surface.md` for the
+decision record.
+
+### Added
+
+- **`eval_toolkit.scorecard(y_true, y_score, metrics=[...], bootstrap=True)`**
+  — primary v1.0 metric surface. Single call computes multiple threshold-free
+  metrics + bootstrap CIs on one slice; returns a `Scorecard` (read-only
+  `Mapping[str, MetricResult]`). Type-safe dict-subscript access; status-aware
+  cells; per-cell error isolation.
+- **`MetricSpec` Protocol** — v1.0 Tier-2 contract; `name: str` +
+  `compute(y_true, y_score) -> float`. Custom user specs satisfy structurally.
+- **`MetricResult`** frozen dataclass — `value: float | None`, `status:
+  Literal["ok", "skipped", "error"]`, `reason: str`, `ci: BootstrapCI | None`.
+  Reuses the existing `MetricState` vocabulary from `artifacts.py:30-61`.
+- **`Scorecard`** read-only `Mapping[str, MetricResult]` — `to_dict()`
+  JSON-friendly, `to_pandas()` one-row DataFrame (lazy pandas import).
+- **`eval_toolkit.metric_specs`** namespace submodule with threshold-free
+  first-party specs:
+  - `pr_auc`, `roc_auc`, `brier` — module-level singletons (identity stable).
+  - `ece(n_bins, strategy)` — LRU-cached factory (identity stable per kwargs).
+- **`SINGLE_CLASS_INCOMPATIBLE_METRICS`** extended with `pr_auc` / `roc_auc`
+  aliases (alongside existing `auroc` / `auprc`) so the v0.46 scorecard
+  surface and the v0.39 harness paths both produce correct skipped-status
+  behavior. Non-breaking; doctest + unit tests added.
+- **`docs/source/adr/0002-scorecard-as-primary-metric-surface.md`** —
+  decision record covering single-surface rationale, threshold-free scope,
+  Tier-2 Protocol commitment, and v2.0 trigger conditions.
+- **`docs/source/migration/v0.46.md`** — consumer migration guide with
+  side-by-side recipes for every common pattern.
+
+### Deprecated
+
+The following 8 top-level scalar imports emit `DeprecationWarning` and will
+be hard-removed at v0.47.0. Use `scorecard()` + `metric_specs` or the
+`eval_toolkit.metrics` submodule path (internal API, no warning).
+
+- `pr_auc`, `roc_auc`, `brier_score`
+- `expected_calibration_error`
+- `expected_calibration_error_debiased`
+- `expected_calibration_error_equal_mass`
+- `expected_calibration_error_l2`
+- `expected_calibration_error_l2_debiased`
+
+### Audit findings integrated (Round 5)
+
+Per `docs/source/audit_findings.md`:
+
+- **F1** (scorecard threshold semantics) — addressed by Decision R: ship
+  threshold-free first-party specs only at v0.46. Threshold-dependent
+  metrics (F1, accuracy, precision, recall) deferred to v1.x with explicit
+  operating-point provenance.
+- **F2** (scorecard cell-state semantics) — addressed by Decision S: reuse
+  existing `MetricState` (`ok`/`skipped`/`error`) vocabulary.
+- **F4** (deprecation shim must extend the lazy resolver, not replace it) —
+  addressed: `__getattr__` deprecation branch sits between `__version__`
+  short-circuit and the base `_EXPORTS` lookup; tagged with BEGIN/END
+  TRANSITIONAL markers for clean v0.47 removal. Tests guard that every
+  remaining `_EXPORTS` symbol still resolves.
+- **X.2 precondition** — `is_metric_defined_for_slice` aliases shipped
+  ahead of v0.46 (PR #62).
+
+### Protocol stability
+
+Tier-2 streak continues: 7 of 7 consecutive minors (v0.40–v0.46) without
+method-shape edits to any existing Tier-2 Protocol. `MetricSpec` is a NEW
+Tier-2 Protocol added at v0.46; freezes at v1.0.
+
 ## [0.45.0] — 2026-05-21 — Stacking: MetaLearner Protocol + LogisticStacker (closes #52)
 
 First minor of the staggered v0.45 → v0.46 → v0.47 → v0.48 → v1.0 sequence

{eval_toolkit-0.45.0 → eval_toolkit-0.46.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.45.0
+Version: 0.46.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-0.45.0 → eval_toolkit-0.46.0}/src/eval_toolkit/__init__.py

@@ -193,20 +193,18 @@ _EXPORTS: dict[str, str] = {
     "SINGLE_CLASS_INCOMPATIBLE_METRICS": "eval_toolkit.metrics",
     "ThresholdResult": "eval_toolkit.metrics",
     "brier_decomposition": "eval_toolkit.metrics",
-    "brier_score": "eval_toolkit.metrics",
-    "expected_calibration_error": "eval_toolkit.metrics",
-    "expected_calibration_error_debiased": "eval_toolkit.metrics",
-    "expected_calibration_error_equal_mass": "eval_toolkit.metrics",
-    "expected_calibration_error_l2": "eval_toolkit.metrics",
-    "expected_calibration_error_l2_debiased": "eval_toolkit.metrics",
+    # `brier_score`, `pr_auc`, `roc_auc`, and the 5 ECE variants removed from
+    # `_EXPORTS` at v0.46 (Decision L). They remain reachable at the top
+    # level via the `__getattr__` deprecation branch (emits
+    # `DeprecationWarning`; branch removed at v0.47) and via the metrics
+    # submodule (`from eval_toolkit.metrics import pr_auc` — internal API
+    # per ADR 0002, not part of the v1.0 stability contract).
     "headline_metrics": "eval_toolkit.metrics",
     "is_metric_defined_for_slice": "eval_toolkit.metrics",
     "metrics_at_threshold": "eval_toolkit.metrics",
-    "pr_auc": "eval_toolkit.metrics",
     "precision_at_prior": "eval_toolkit.metrics",
     "quantile_stratified_pr_auc": "eval_toolkit.metrics",
     "quantile_stratified_report": "eval_toolkit.metrics",
-    "roc_auc": "eval_toolkit.metrics",
     "score_distribution_summary": "eval_toolkit.metrics",
     "single_class_threshold_metrics": "eval_toolkit.metrics",
     "stratified_recall": "eval_toolkit.metrics",
@@ -296,15 +294,68 @@ _EXPORTS: dict[str, str] = {
     "wilson_interval": "eval_toolkit.thresholds",
     "LogisticStacker": "eval_toolkit.stacking",
     "MetaLearner": "eval_toolkit.stacking",
+    "MetricResult": "eval_toolkit._scorecard",
+    "MetricSpec": "eval_toolkit._scorecard",
+    "Scorecard": "eval_toolkit._scorecard",
+    "scorecard": "eval_toolkit._scorecard",
 }
 
 __all__ = ["__version__", *_EXPORTS.keys()]
 
 
+# ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
+# At v0.46 the scalar metric functions left the top-level `_EXPORTS` map (above)
+# in favor of the `scorecard()` surface (Decision A). To give the consumer one
+# release of overlap before the hard removal at v0.47, the names below remain
+# reachable via the package-level `__getattr__` (which delegates to the
+# `eval_toolkit.metrics` submodule) but emit a `DeprecationWarning` on first
+# lookup pointing at the new API.
+#
+# WHY THIS IS A BRANCH, NOT A REPLACEMENT (Audit F4 — Round 5):
+# `__getattr__` below is the load-bearing lazy export resolver for every name
+# in `_EXPORTS`. The deprecation branch is a discrete `if name in
+# _DEPRECATED_SCALARS` check ABOVE the resolver — the resolver's existing
+# behavior for non-deprecated names is unchanged. At v0.47 we delete this
+# transitional block and the resolver continues to work for every remaining
+# `_EXPORTS` entry.
+_DEPRECATED_SCALARS: frozenset[str] = frozenset(
+    {
+        "pr_auc",
+        "roc_auc",
+        "brier_score",
+        "expected_calibration_error",
+        "expected_calibration_error_debiased",
+        "expected_calibration_error_equal_mass",
+        "expected_calibration_error_l2",
+        "expected_calibration_error_l2_debiased",
+    }
+)
+# ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
+
+
 def __getattr__(name: str) -> Any:
     """Resolve public symbols lazily."""
     if name == "__version__":
         return __version__
+    # ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
+    if name in _DEPRECATED_SCALARS:
+        import warnings
+
+        warnings.warn(
+            f"eval_toolkit.{name} is deprecated and will be removed in v0.47. "
+            f"Use `scorecard(y, s, metrics=[metric_specs.{_scorecard_spec_for(name)}])"
+            f'["{_scorecard_spec_for(name)}"].value` instead, or "import from the'
+            f" `eval_toolkit.metrics` submodule directly (internal API).",
+            DeprecationWarning,
+            stacklevel=2,
+        )
+        module = import_module("eval_toolkit.metrics")
+        value = getattr(module, name)
+        # Do NOT cache in globals() — repeated lookups should keep re-warning
+        # (one warning per call site, modulo Python's default
+        # DeprecationWarning de-duplication).
+        return value
+    # ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
     module_name = _EXPORTS.get(name)
     if module_name is None:
         raise AttributeError(f"module 'eval_toolkit' has no attribute {name!r}")
@@ -314,6 +365,29 @@ def __getattr__(name: str) -> Any:
     return value
 
 
+# ── BEGIN TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
+def _scorecard_spec_for(deprecated_name: str) -> str:
+    """Map a deprecated-scalar name to its `metric_specs` replacement name.
+
+    Used only inside the v0.46 deprecation warning message. Returns the
+    closest equivalent first-party spec name where one exists; falls back
+    to the original name for ECE variants whose exact-match spec isn't in
+    the v0.46 first-party namespace (e.g., the L2 / debiased variants —
+    callers either implement a custom `MetricSpec` or stay on the
+    submodule path).
+    """
+    return {
+        "pr_auc": "pr_auc",
+        "roc_auc": "roc_auc",
+        "brier_score": "brier",
+        "expected_calibration_error": "ece(n_bins=10)",
+        "expected_calibration_error_equal_mass": 'ece(n_bins=10, strategy="quantile")',
+    }.get(deprecated_name, deprecated_name)
+
+
+# ── END TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
+
+
 def __dir__() -> list[str]:
     """Expose lazy public symbols to introspection."""
     return sorted(__all__)