eval-toolkit 0.46.1__tar.gz → 0.47.0__tar.gz

{eval_toolkit-0.46.1 → eval_toolkit-0.47.0}/CHANGELOG.md

@@ -5,6 +5,140 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.47.0] — 2026-05-21 — Sweep unification + TextTransform + advanced-6 + cleanup + Round 6 follow-on
+
+Second BREAKING minor of the staggered v0.45 → v0.46 → v0.46.1 → v0.47 →
+v0.48 → v1.0 release sequence (plan
+``~/.claude/plans/evaluate-all-the-work-twinkly-kite.md``, Step 3).
+
+Closes:
+
+- The v0.43 CHANGELOG forward-look re: advanced-6 character-injection
+  techniques (Decision Q11→11.3 — "12-technique suite + new sweep API
+  in one migration step").
+- Round 6 audit follow-on items per Decision R6-E (R6-A docstring,
+  R6-B duplicate name guard, R6-C to_pandas schema, R6-D Protocol
+  method-shape drift guard, R6-F5 narrow except, R6-F6 plan/roadmap
+  refresh, R6-H make_spec_name helper) — see ``docs/source/audit_findings.md``
+  for the per-finding ledger.
+
+### Removed (BREAKING)
+
+- **Top-level scalar metric names** (``eval_toolkit.pr_auc``,
+  ``eval_toolkit.roc_auc``, ``eval_toolkit.brier_score``, all 5
+  ``expected_calibration_*`` variants) — the v0.46 ``__getattr__``
+  deprecation shim has been deleted. These names now raise
+  ``AttributeError`` at the top-level. Migration: use ``scorecard(...)``
+  with ``metric_specs`` (primary) OR import from the
+  ``eval_toolkit.metrics`` submodule (internal API per ADR 0002).
+  (Decision L; plan §4D.)
+- **Module-level ``adversarial.sweep`` + ``preprocessing.sweep``** —
+  consolidated into the top-level :func:`sweep` (Decision D + plan §4C).
+  Parity tests in Sub-PR 4 of this release proved 1:1 output equivalence
+  on the neutral subset.
+- **``adversarial.character_injection`` + ``preprocessing.spotlighting``
+  ``SimpleNamespace`` shortcuts** — removed (Decision N + plan §4E).
+  The 12 adversarial dataclasses + the 3 preprocessing variants + the
+  underlying functional API are the only public paths.
+- **``adversarial.CharacterInjectionStrategy``** per-module Protocol —
+  removed. The top-level :class:`TextTransform` Protocol (Decision K)
+  is the single canonical contract; all 12 character-injection
+  dataclasses + 3 preprocessing variants satisfy it structurally.
+
+### Added
+
+- **``TextTransform`` Protocol** (top-level; ``eval_toolkit.protocols`` module).
+  Decision K + Audit R5-F3 (Codex Round 5): unifies the "name + transform(text)"
+  shape across preprocessing (defence) and adversarial (attack) strategies so
+  the v0.47 top-level :func:`sweep` (next sub-PR) can mix them in one call. The
+  9th strict Tier-2 Protocol per ADR 0003.
+- **Advanced 6 character-injection techniques** (plan §4F, Decision Q11→11.3) —
+  closes the v0.43.0 CHANGELOG forward-look that referenced these as "scheduled
+  for v0.43.1" (a version that never shipped). Each satisfies the top-level
+  :class:`TextTransform` Protocol structurally; all are frozen + ``slots=True``
+  dataclasses with deterministic behaviour under their ``seed`` kwarg where
+  applicable:
+
+    - :class:`BidiRTLInjection` — wrap input in ``U+202E … U+202C``
+      RIGHT-TO-LEFT OVERRIDE block.
+    - :class:`TagStrippingInjection` — strip HTML/XML-like ``<…>`` tags
+      (idempotent).
+    - :class:`SynonymSubstitution` — replace whitelisted prompt-injection-
+      relevant function words / verbs with semantic-preserving synonyms.
+    - :class:`TokenSplitting` — insert a single space inside long enough
+      words; forces subword tokenizers to re-segment.
+    - :class:`UnicodeNormalization` — NFC / NFD / NFKC / NFKD; default NFKC
+      folds compatibility chars (e.g., fullwidth ``ＡＢＣ`` → ``ABC``).
+    - :class:`InvisibleCharsInjection` — sample from the 5-element invisible-
+      code-point set (ZWSP, ZWNJ, ZWJ, word joiner, BOM) — distinct from the
+      single-codepoint :class:`ZeroWidthSpaceInjection`.
+
+  Also exported: ``ADVANCED_TECHNIQUES`` (6-tuple) and ``ALL_TECHNIQUES``
+  (12-tuple = core 6 + advanced 6).
+- **Top-level :func:`sweep`** — single ``TextTransform`` enumeration entry
+  point (Decision K + Decision D + Audit R5-F3). Replaces the per-module
+  ``adversarial.sweep`` + ``preprocessing.sweep`` (those are removed in a
+  subsequent sub-PR of this release). New contract:
+
+    - ``sweep(strategies, texts)`` → neutral DataFrame with ``text_id`` /
+      ``variant`` / ``transformed_text`` columns. Pure text-transform
+      enumeration; defence + attack strategies compose freely.
+    - ``sweep(..., scorer=...)`` → also emits ``original_score`` /
+      ``transformed_score`` columns (single batched scorer call per
+      strategy, not per-row).
+    - ``sweep(..., scorer=..., attack_threshold=t)`` → also emits ``asr``
+      (per-row attack-success flag). Explicit threshold REQUIRED to
+      materialize ``asr``; no magic ``threshold=0.5`` default.
+      ``attack_threshold`` without ``scorer`` raises ``ValueError``.
+
+  Parity tests against the existing module-level sweeps ship in this
+  sub-PR (``tests/test_sweep.py``) and prove the v0.47 consolidation
+  produces identical transformed-text rows for the 6 core character-
+  injection techniques + the 3 spotlighting variants.
+- **3 preprocessing dataclasses** (``DelimitVariant``, ``DatamarkVariant``,
+  ``EncodeVariant``) in :mod:`eval_toolkit.preprocessing`. Frozen +
+  ``slots=True`` thin wrappers over the existing :func:`delimit` /
+  :func:`datamark` / :func:`encode` functions. Closes Audit R5-F3
+  (Codex Round 5) — prior to this commit, ``preprocessing.__all__`` exported
+  only functions, so the "concrete classes satisfy ``TextTransform``
+  structurally" claim only held on the adversarial side. Now both sides
+  share the dataclass-strategy shape.
+- ``metric_specs.make_spec_name(prefix, **kwargs)`` canonicalization helper
+  for custom parameterized :class:`MetricSpec` implementations. Alphabetized
+  kwargs joined by underscore — same convention the v0.46 ECE factory uses.
+  Lands in ``metric_specs.__all__`` only; **not** top-level ``__all__`` per
+  Decision R6-H. (Closes Round 6 Gemini R6-F4.)
+
+### Changed (Round 6 follow-on)
+
+- ``scorecard()`` now raises ``ValueError`` when two :class:`MetricSpec`
+  instances in the ``metrics`` list share a ``name``. Forces caller
+  disambiguation; the ``Mapping[str, MetricResult]`` contract never silently
+  drops a cell. Error message reports both indices. (Decision R6-B; closes
+  Round 6 Codex R6-F3.)
+- ``scorecard(seed=None)`` docstring rewritten to document the deterministic-
+  by-default contract (``None`` is treated as ``seed=0``). No behavior
+  change; v0.46 documented the wrong contract. (Decision R6-A; closes Round 6
+  Codex R6-F4 + Gemini R6-F1.)
+- ``_evaluate_spec()`` exception catches narrowed: ``MemoryError``,
+  ``RecursionError``, ``KeyboardInterrupt``, and ``SystemExit`` now propagate
+  out of ``scorecard()`` instead of being captured as a ``status="error"``
+  cell. Per-cell isolation remains for ordinary application errors.
+  (Decision R6-F5; closes Round 6 Gemini R6-F5.)
+- ``Scorecard.to_pandas()`` MultiIndex schema extended with two new inner-
+  field columns: ``n_resamples`` (int / NaN sentinel) and ``method``
+  (string / ``""`` sentinel). The DataFrame view is now lossless against
+  :meth:`BootstrapCI.to_dict` — trace provenance (resample count + CI
+  method) no longer drops at the DataFrame boundary. Callers indexing the
+  MultiIndex by name keep working; callers indexing by position must
+  re-check column offsets. (Decision R6-C; closes Round 6 Gemini R6-F3.)
+- ``tests/test_public_api.py`` drift guard now captures method signatures
+  for ``typing.Protocol`` classes in ``__all__`` (a ``protocol_methods``
+  sub-entry in the snapshot). Together with a Tier-2 coverage test, this
+  actually enforces the strict method-shape stability ADR 0003 promises
+  for the 9 Tier-2 Protocols. (Decision R6-D; closes Round 6 Codex R6-F5.)
+  Public-API golden regenerated alongside this change.
+
 ## [0.46.1] — 2026-05-21 — Round 6 hotfix: ECE strategy validation + deprecation warning content
 
 Hotfix release per **Decision Q** (data correctness regression + time-sensitive

{eval_toolkit-0.46.1 → eval_toolkit-0.47.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.46.1
+Version: 0.47.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -215,7 +215,7 @@ pip install "eval-toolkit[all]"                 # everything
 
 ```python
 import numpy as np
-from eval_toolkit import pr_auc, roc_auc, expected_calibration_error
+from eval_toolkit.metrics import pr_auc, roc_auc, expected_calibration_error
 
 rng = np.random.default_rng(42)
 y = rng.integers(0, 2, size=200)
@@ -230,7 +230,8 @@ print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")
 ### Bootstrap confidence intervals
 
 ```python
-from eval_toolkit import bootstrap_ci, paired_bootstrap_diff, pr_auc
+from eval_toolkit import bootstrap_ci, paired_bootstrap_diff
+from eval_toolkit.metrics import pr_auc
 
 ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, seed=42)
 print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")
@@ -244,8 +245,10 @@ print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")
 ### Temperature scaling (Guo et al. 2017)
 
 ```python
+import numpy as np
 from eval_toolkit import fit_temperature
 
+rng = np.random.default_rng(42)
 logits = rng.normal(size=(500, 2))
 labels = (logits[:, 1] > logits[:, 0]).astype(int)
 result = fit_temperature(logits, labels)

{eval_toolkit-0.46.1 → eval_toolkit-0.47.0}/README.md

@@ -132,7 +132,7 @@ pip install "eval-toolkit[all]"                 # everything
 
 ```python
 import numpy as np
-from eval_toolkit import pr_auc, roc_auc, expected_calibration_error
+from eval_toolkit.metrics import pr_auc, roc_auc, expected_calibration_error
 
 rng = np.random.default_rng(42)
 y = rng.integers(0, 2, size=200)
@@ -147,7 +147,8 @@ print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")
 ### Bootstrap confidence intervals
 
 ```python
-from eval_toolkit import bootstrap_ci, paired_bootstrap_diff, pr_auc
+from eval_toolkit import bootstrap_ci, paired_bootstrap_diff
+from eval_toolkit.metrics import pr_auc
 
 ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, seed=42)
 print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")
@@ -161,8 +162,10 @@ print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")
 ### Temperature scaling (Guo et al. 2017)
 
 ```python
+import numpy as np
 from eval_toolkit import fit_temperature
 
+rng = np.random.default_rng(42)
 logits = rng.normal(size=(500, 2))
 labels = (logits[:, 1] > logits[:, 0]).astype(int)
 result = fit_temperature(logits, labels)

{eval_toolkit-0.46.1 → eval_toolkit-0.47.0}/src/eval_toolkit/__init__.py

@@ -31,22 +31,36 @@ _logging.getLogger("eval_toolkit").addHandler(_logging.NullHandler())
 # tests/golden/public_api/ reads dict keys + values, not comments.
 _EXPORTS: dict[str, str] = {
     # --- adversarial ---
+    "ADVANCED_TECHNIQUES": "eval_toolkit.adversarial",
+    "ALL_TECHNIQUES": "eval_toolkit.adversarial",
+    "BidiRTLInjection": "eval_toolkit.adversarial",
     "CORE_TECHNIQUES": "eval_toolkit.adversarial",
     "CaseRandomization": "eval_toolkit.adversarial",
-    "CharacterInjectionStrategy": "eval_toolkit.adversarial",
     "DiacriticInjection": "eval_toolkit.adversarial",
     "HomoglyphSubstitution": "eval_toolkit.adversarial",
+    "InvisibleCharsInjection": "eval_toolkit.adversarial",
     "PunctuationInjection": "eval_toolkit.adversarial",
+    "SynonymSubstitution": "eval_toolkit.adversarial",
+    "TagStrippingInjection": "eval_toolkit.adversarial",
+    "TokenSplitting": "eval_toolkit.adversarial",
+    "UnicodeNormalization": "eval_toolkit.adversarial",
     "WhitespaceInjection": "eval_toolkit.adversarial",
     "ZeroWidthSpaceInjection": "eval_toolkit.adversarial",
-    "character_injection": "eval_toolkit.adversarial",
+    # CharacterInjectionStrategy + character_injection SimpleNamespace
+    # removed at v0.47 (Decision N + plan §4E). TextTransform Protocol +
+    # the 12 concrete dataclasses are now the only public path.
     # --- losses ---
     "RecallAtLowFPR": "eval_toolkit.losses",
     # --- preprocessing ---
+    # `spotlighting` SimpleNamespace removed at v0.47 (Decision N + plan §4E).
+    # The 3 Variant dataclasses + the underlying functional API are the
+    # only public path.
+    "DatamarkVariant": "eval_toolkit.preprocessing",
+    "DelimitVariant": "eval_toolkit.preprocessing",
+    "EncodeVariant": "eval_toolkit.preprocessing",
     "datamark": "eval_toolkit.preprocessing",
     "delimit": "eval_toolkit.preprocessing",
     "encode": "eval_toolkit.preprocessing",
-    "spotlighting": "eval_toolkit.preprocessing",
     # --- probes ---
     "ActivationDeltaProbe": "eval_toolkit.probes",
     "ActivationExtractor": "eval_toolkit.probes",
@@ -247,6 +261,7 @@ _EXPORTS: dict[str, str] = {
     "PredictionReader": "eval_toolkit.protocols",
     "Scorer": "eval_toolkit.protocols",
     "SliceAwareScorer": "eval_toolkit.protocols",
+    "TextTransform": "eval_toolkit.protocols",
     "Versioned": "eval_toolkit.protocols",
     # --- seeds ---
     "set_global_seeds": "eval_toolkit.seeds",
@@ -298,61 +313,28 @@ _EXPORTS: dict[str, str] = {
     "MetricSpec": "eval_toolkit._scorecard",
     "Scorecard": "eval_toolkit._scorecard",
     "scorecard": "eval_toolkit._scorecard",
+    # --- sweep (top-level v0.47 unification — Decision K + Decision D) ---
+    "sweep": "eval_toolkit._sweep",
 }
 
 __all__ = ["__version__", *_EXPORTS.keys()]
 
 
-# ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
-# At v0.46 the scalar metric functions left the top-level `_EXPORTS` map (above)
-# in favor of the `scorecard()` surface (Decision A). To give the consumer one
-# release of overlap before the hard removal at v0.47, the names below remain
-# reachable via the package-level `__getattr__` (which delegates to the
-# `eval_toolkit.metrics` submodule) but emit a `DeprecationWarning` on first
-# lookup pointing at the new API.
-#
-# WHY THIS IS A BRANCH, NOT A REPLACEMENT (Audit F4 — Round 5):
-# `__getattr__` below is the load-bearing lazy export resolver for every name
-# in `_EXPORTS`. The deprecation branch is a discrete `if name in
-# _DEPRECATED_SCALARS` check ABOVE the resolver — the resolver's existing
-# behavior for non-deprecated names is unchanged. At v0.47 we delete this
-# transitional block and the resolver continues to work for every remaining
-# `_EXPORTS` entry.
-_DEPRECATED_SCALARS: frozenset[str] = frozenset(
-    {
-        "pr_auc",
-        "roc_auc",
-        "brier_score",
-        "expected_calibration_error",
-        "expected_calibration_error_debiased",
-        "expected_calibration_error_equal_mass",
-        "expected_calibration_error_l2",
-        "expected_calibration_error_l2_debiased",
-    }
-)
-# ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
-
-
 def __getattr__(name: str) -> Any:
-    """Resolve public symbols lazily."""
+    """Resolve public symbols lazily.
+
+    v0.47 cleanup (Decision L): the BEGIN/END TRANSITIONAL DEPRECATION
+    BRANCH that v0.46 inserted in front of the resolver — together with the
+    ``_DEPRECATED_SCALARS`` frozenset and the ``_deprecation_warning_for``
+    helper — has been removed. The lazy resolver below is the v0.46 base
+    behavior; with the transitional block gone, deprecated v0.45 scalar names
+    (``pr_auc``, ``roc_auc``, ``brier_score``, the 5 ``expected_calibration_*``
+    variants) now raise :class:`AttributeError` cleanly. Submodule-level
+    access (e.g., ``from eval_toolkit.metrics import pr_auc``) is unaffected
+    per Decision C / ADR 0002.
+    """
     if name == "__version__":
         return __version__
-    # ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
-    if name in _DEPRECATED_SCALARS:
-        import warnings
-
-        warnings.warn(
-            _deprecation_warning_for(name),
-            DeprecationWarning,
-            stacklevel=2,
-        )
-        module = import_module("eval_toolkit.metrics")
-        value = getattr(module, name)
-        # Do NOT cache in globals() — repeated lookups should keep re-warning
-        # (one warning per call site, modulo Python's default
-        # DeprecationWarning de-duplication).
-        return value
-    # ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
     module_name = _EXPORTS.get(name)
     if module_name is None:
         raise AttributeError(f"module 'eval_toolkit' has no attribute {name!r}")
@@ -362,113 +344,6 @@ def __getattr__(name: str) -> Any:
     return value
 
 
-# ── BEGIN TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
-#
-# Per Round 6 audit (Codex R6-F2 + Gemini R6-F2; Decisions R6-F + R6-G):
-# - For deprecated scalars with a first-party `metric_specs` equivalent, the
-#   warning emits an EXECUTABLE scorecard snippet (factory expression + the
-#   correct encoded scorecard key, not the factory call string).
-# - For the 3 ECE variants without a `metric_specs` equivalent
-#   (expected_calibration_error_debiased / _l2 / _l2_debiased), the warning
-#   instead points at the submodule path per Decision R6-G — no first-party
-#   replacement is shipped at v0.47.
-# - ECE `n_bins=10` preserves the pre-v0.46 default (verified at
-#   `metrics.py:730-734`) — Decision R6-F. A migration note explains that
-#   the v0.46+ `metric_specs.ece()` factory defaults to `n_bins=15` (matching
-#   Hines et al.) and how to opt in.
-_FirstParty = tuple[str, str]  # (factory_expression, scorecard_key)
-"""Type alias for a deprecated-scalar that has a metric_specs replacement.
-
-The factory expression is what the user types after ``metric_specs.``; the
-scorecard key is the literal string that indexes ``Scorecard``.
-"""
-
-
-_FIRST_PARTY_REPLACEMENTS: dict[str, _FirstParty] = {
-    "pr_auc": ("pr_auc", "pr_auc"),
-    "roc_auc": ("roc_auc", "roc_auc"),
-    "brier_score": ("brier", "brier"),
-    # ECE variants: use n_bins=10 (pre-v0.46 default per Decision R6-F).
-    # The migration note in the warning text explains how to switch to
-    # n_bins=15 if the user wants the v0.46+ metric_specs.ece() default.
-    "expected_calibration_error": (
-        "ece(n_bins=10)",
-        "ece_n_bins_10_strategy_uniform",
-    ),
-    "expected_calibration_error_equal_mass": (
-        'ece(n_bins=10, strategy="quantile")',
-        "ece_n_bins_10_strategy_quantile",
-    ),
-}
-"""Names that have a first-party metric_specs replacement at v0.46.
-
-The 3 ECE variants NOT in this map (_debiased, _l2, _l2_debiased) get the
-submodule-path warning template instead (Decision R6-G).
-"""
-
-
-def _deprecation_warning_for(name: str) -> str:
-    """Render the DeprecationWarning message for a deprecated scalar name.
-
-    Branches on whether ``name`` has a first-party `metric_specs` replacement
-    (Decision R6-G):
-
-    - First-party (5 names): scorecard snippet with the correct encoded key
-      (Decision R6-F).
-    - Submodule-only (3 ECE variants): point at the submodule path per
-      Decision R6-G.
-
-    The first-party variants for ECE include a migration note explaining the
-    new ``metric_specs.ece()`` factory default of ``n_bins=15`` so users can
-    opt in to the new convention; the snippet itself uses ``n_bins=10`` for
-    bit-identical pre-v0.46 math (Decision R6-F).
-
-    Parameters
-    ----------
-    name : str
-        A name in ``_DEPRECATED_SCALARS``.
-
-    Returns
-    -------
-    str
-        The warning message, ready to pass to ``warnings.warn``.
-    """
-    first_party = _FIRST_PARTY_REPLACEMENTS.get(name)
-    if first_party is not None:
-        factory_expr, scorecard_key = first_party
-        msg = (
-            f"eval_toolkit.{name} is deprecated and will be removed in v0.47. "
-            f"For the same math, use:\n"
-            f"    scorecard(y, s, metrics=[metric_specs.{factory_expr}])"
-            f'["{scorecard_key}"].value\n'
-            f"Or import from the eval_toolkit.metrics submodule directly "
-            f"(internal API per ADR 0002 — stable across v1.x, subject to "
-            f"refactor in major versions)."
-        )
-        # ECE-specific migration note about the n_bins default change.
-        if name.startswith("expected_calibration_error"):
-            msg += (
-                "\nNote: the v0.46+ metric_specs.ece() factory defaults to "
-                "n_bins=15 (matching Hines et al.); the n_bins=10 in this "
-                "snippet preserves the pre-v0.46 math. Pass n_bins=15 to use "
-                "the new convention."
-            )
-        return msg
-    # Decision R6-G: 3 ECE variants without first-party replacements →
-    # submodule path only.
-    return (
-        f"eval_toolkit.{name} is deprecated and will be removed in v0.47. "
-        f"This variant is NOT in v0.46+ metric_specs. Use:\n"
-        f"    from eval_toolkit.metrics import {name}\n"
-        f"(internal API per ADR 0002 — stable across v1.x, subject to "
-        f"refactor in major versions). Or contribute the variant to "
-        f"metric_specs if you use it regularly."
-    )
-
-
-# ── END TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
-
-
 def __dir__() -> list[str]:
     """Expose lazy public symbols to introspection."""
     return sorted(__all__)

{eval_toolkit-0.46.1 → eval_toolkit-0.47.0}/src/eval_toolkit/_scorecard.py

@@ -261,10 +261,17 @@ class Scorecard(Mapping[str, MetricResult]):
         ``ImportError`` with an install hint when pandas is missing.
 
         The DataFrame has 1 row (one slice) and a 2-level column index:
-        outer = metric name, inner = field name in
-        ``{"value", "status", "reason", "ci_low", "ci_high", "confidence"}``.
-        ``ci_low`` / ``ci_high`` / ``confidence`` are ``NaN`` / ``""`` when
-        no CI is present.
+        outer = metric name, inner = field name in ``{"value", "status",
+        "reason", "ci_low", "ci_high", "confidence", "n_resamples",
+        "method"}``. CI-related columns (``ci_low``, ``ci_high``,
+        ``confidence``, ``n_resamples``, ``method``) are sentinel-valued
+        (``NaN`` for numeric, ``""`` for string) when no CI is present
+        (status="skipped" / "error", or bootstrap=False).
+
+        Decision R6-C (Round 6 audit, Gemini F3): the v0.47 expansion adds
+        ``n_resamples`` + ``method`` so the schema is lossless against
+        :meth:`BootstrapCI.to_dict` — trace provenance no longer drops in
+        the DataFrame view.
         """
         try:
             import pandas as pd
@@ -285,6 +292,8 @@ class Scorecard(Mapping[str, MetricResult]):
                     (name, "ci_low"),
                     (name, "ci_high"),
                     (name, "confidence"),
+                    (name, "n_resamples"),
+                    (name, "method"),
                 ]
             )
             values.extend(
@@ -295,6 +304,8 @@ class Scorecard(Mapping[str, MetricResult]):
                     result.ci.ci_low if result.ci is not None else float("nan"),
                     result.ci.ci_high if result.ci is not None else float("nan"),
                     result.ci.confidence if result.ci is not None else float("nan"),
+                    result.ci.n_resamples if result.ci is not None else float("nan"),
+                    result.ci.method if result.ci is not None else "",
                 ]
             )
 
@@ -341,7 +352,13 @@ def scorecard(
     confidence : float, optional
         Two-sided CI level ∈ ``(0, 1)``. Default ``0.95``.
     seed : int or None, optional
-        Bootstrap RNG seed. Default ``None`` (non-deterministic).
+        Bootstrap RNG seed. Default ``None``, which is treated as ``seed=0``
+        for reproducibility — eval-toolkit's evaluation pipelines are
+        deterministic by default. Pass an explicit integer to control the
+        bootstrap RNG; pass a value derived from
+        ``np.random.SeedSequence().entropy`` for non-deterministic sampling.
+        Decision R6-A (Round 6 audit) locked the deterministic-by-default
+        contract; the prior docstring framing was incorrect.
 
     Returns
     -------
@@ -410,6 +427,7 @@ def scorecard(
         confidence=confidence,
         bootstrap=bootstrap,
     )
+    _validate_unique_spec_names(metrics)
 
     is_single_class = bool(np.unique(y_true_arr).size < 2)
 
@@ -450,6 +468,11 @@ def _evaluate_spec(
 
     try:
         point = float(spec.compute(y_true, y_score))
+    except (MemoryError, RecursionError, KeyboardInterrupt, SystemExit):
+        # Process-exhaustion / user-interrupt signals must propagate;
+        # per-cell isolation is for application-level errors only.
+        # Decision R6-F5 (Round 6 audit, Gemini).
+        raise
     except Exception as exc:  # noqa: BLE001 — broad catch is intentional (per-cell isolation)
         return MetricResult(
             value=None,
@@ -469,6 +492,9 @@ def _evaluate_spec(
             confidence=confidence,
             seed=seed if seed is not None else 0,
         )
+    except (MemoryError, RecursionError, KeyboardInterrupt, SystemExit):
+        # Same R6-F5 invariant for the bootstrap path.
+        raise
     except Exception as exc:  # noqa: BLE001
         # Point estimate succeeded; the bootstrap couldn't (e.g., n < 10
         # floor from bootstrap.py:198, BCa degeneracy, etc.). Record the
@@ -507,3 +533,21 @@ def _validate_scorecard_inputs(
             raise ValueError(f"n_resamples must be >= 1 when bootstrap=True; got {n_resamples}")
         if not 0.0 < confidence < 1.0:
             raise ValueError(f"confidence must be in (0, 1); got {confidence}")
+
+
+def _validate_unique_spec_names(metrics: Sequence[MetricSpec]) -> None:
+    """Reject duplicate MetricSpec.name values in a single scorecard() call.
+
+    Locked by Decision R6-B (Round 6 audit, Codex R6-F3): silent last-wins
+    overwrite is not a documented Mapping[str, MetricResult] contract. Force
+    the caller to disambiguate so we never lose data on user error.
+    """
+    seen: dict[str, int] = {}
+    for i, spec in enumerate(metrics):
+        if spec.name in seen:
+            raise ValueError(
+                f"Duplicate MetricSpec name {spec.name!r} at index {i} "
+                f"(previously at index {seen[spec.name]}); each spec must have a "
+                f"unique name for the Scorecard Mapping[str, MetricResult] contract."
+            )
+        seen[spec.name] = i