eval-toolkit 0.46.0__tar.gz → 0.47.0__tar.gz

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/.gitignore

@@ -45,6 +45,16 @@ coverage.json
 # Mutation-testing output (mutmut / cargo-mutants — local run artifacts)
 mutants/
 
+# Local audit artifacts (Round 5+ Gate 3 LLM cross-review packets + reports).
+# The canonical prompt lives at ~/.claude/plans/gate3-audit-prompt.md and the
+# canonical findings ledger lives at docs/source/audit_findings.md; per-run
+# raw model outputs are author-local working copies.
+# Tracked: per-round briefing files (`gate3-audit-round-<N>.md`).
+# Untracked: prompt template, generic report, per-round report files.
+gate3-audit-prompt.md
+gate3-audit-report.md
+gate3-audit-round-*-report.md
+
 # Claude Code project settings (machine-local)
 .claude/
 

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/CHANGELOG.md

@@ -5,6 +5,205 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.47.0] — 2026-05-21 — Sweep unification + TextTransform + advanced-6 + cleanup + Round 6 follow-on
+
+Second BREAKING minor of the staggered v0.45 → v0.46 → v0.46.1 → v0.47 →
+v0.48 → v1.0 release sequence (plan
+``~/.claude/plans/evaluate-all-the-work-twinkly-kite.md``, Step 3).
+
+Closes:
+
+- The v0.43 CHANGELOG forward-look re: advanced-6 character-injection
+  techniques (Decision Q11→11.3 — "12-technique suite + new sweep API
+  in one migration step").
+- Round 6 audit follow-on items per Decision R6-E (R6-A docstring,
+  R6-B duplicate name guard, R6-C to_pandas schema, R6-D Protocol
+  method-shape drift guard, R6-F5 narrow except, R6-F6 plan/roadmap
+  refresh, R6-H make_spec_name helper) — see ``docs/source/audit_findings.md``
+  for the per-finding ledger.
+
+### Removed (BREAKING)
+
+- **Top-level scalar metric names** (``eval_toolkit.pr_auc``,
+  ``eval_toolkit.roc_auc``, ``eval_toolkit.brier_score``, all 5
+  ``expected_calibration_*`` variants) — the v0.46 ``__getattr__``
+  deprecation shim has been deleted. These names now raise
+  ``AttributeError`` at the top-level. Migration: use ``scorecard(...)``
+  with ``metric_specs`` (primary) OR import from the
+  ``eval_toolkit.metrics`` submodule (internal API per ADR 0002).
+  (Decision L; plan §4D.)
+- **Module-level ``adversarial.sweep`` + ``preprocessing.sweep``** —
+  consolidated into the top-level :func:`sweep` (Decision D + plan §4C).
+  Parity tests in Sub-PR 4 of this release proved 1:1 output equivalence
+  on the neutral subset.
+- **``adversarial.character_injection`` + ``preprocessing.spotlighting``
+  ``SimpleNamespace`` shortcuts** — removed (Decision N + plan §4E).
+  The 12 adversarial dataclasses + the 3 preprocessing variants + the
+  underlying functional API are the only public paths.
+- **``adversarial.CharacterInjectionStrategy``** per-module Protocol —
+  removed. The top-level :class:`TextTransform` Protocol (Decision K)
+  is the single canonical contract; all 12 character-injection
+  dataclasses + 3 preprocessing variants satisfy it structurally.
+
+### Added
+
+- **``TextTransform`` Protocol** (top-level; ``eval_toolkit.protocols`` module).
+  Decision K + Audit R5-F3 (Codex Round 5): unifies the "name + transform(text)"
+  shape across preprocessing (defence) and adversarial (attack) strategies so
+  the v0.47 top-level :func:`sweep` (next sub-PR) can mix them in one call. The
+  9th strict Tier-2 Protocol per ADR 0003.
+- **Advanced 6 character-injection techniques** (plan §4F, Decision Q11→11.3) —
+  closes the v0.43.0 CHANGELOG forward-look that referenced these as "scheduled
+  for v0.43.1" (a version that never shipped). Each satisfies the top-level
+  :class:`TextTransform` Protocol structurally; all are frozen + ``slots=True``
+  dataclasses with deterministic behaviour under their ``seed`` kwarg where
+  applicable:
+
+    - :class:`BidiRTLInjection` — wrap input in ``U+202E … U+202C``
+      RIGHT-TO-LEFT OVERRIDE block.
+    - :class:`TagStrippingInjection` — strip HTML/XML-like ``<…>`` tags
+      (idempotent).
+    - :class:`SynonymSubstitution` — replace whitelisted prompt-injection-
+      relevant function words / verbs with semantic-preserving synonyms.
+    - :class:`TokenSplitting` — insert a single space inside long enough
+      words; forces subword tokenizers to re-segment.
+    - :class:`UnicodeNormalization` — NFC / NFD / NFKC / NFKD; default NFKC
+      folds compatibility chars (e.g., fullwidth ``ＡＢＣ`` → ``ABC``).
+    - :class:`InvisibleCharsInjection` — sample from the 5-element invisible-
+      code-point set (ZWSP, ZWNJ, ZWJ, word joiner, BOM) — distinct from the
+      single-codepoint :class:`ZeroWidthSpaceInjection`.
+
+  Also exported: ``ADVANCED_TECHNIQUES`` (6-tuple) and ``ALL_TECHNIQUES``
+  (12-tuple = core 6 + advanced 6).
+- **Top-level :func:`sweep`** — single ``TextTransform`` enumeration entry
+  point (Decision K + Decision D + Audit R5-F3). Replaces the per-module
+  ``adversarial.sweep`` + ``preprocessing.sweep`` (those are removed in a
+  subsequent sub-PR of this release). New contract:
+
+    - ``sweep(strategies, texts)`` → neutral DataFrame with ``text_id`` /
+      ``variant`` / ``transformed_text`` columns. Pure text-transform
+      enumeration; defence + attack strategies compose freely.
+    - ``sweep(..., scorer=...)`` → also emits ``original_score`` /
+      ``transformed_score`` columns (single batched scorer call per
+      strategy, not per-row).
+    - ``sweep(..., scorer=..., attack_threshold=t)`` → also emits ``asr``
+      (per-row attack-success flag). Explicit threshold REQUIRED to
+      materialize ``asr``; no magic ``threshold=0.5`` default.
+      ``attack_threshold`` without ``scorer`` raises ``ValueError``.
+
+  Parity tests against the existing module-level sweeps ship in this
+  sub-PR (``tests/test_sweep.py``) and prove the v0.47 consolidation
+  produces identical transformed-text rows for the 6 core character-
+  injection techniques + the 3 spotlighting variants.
+- **3 preprocessing dataclasses** (``DelimitVariant``, ``DatamarkVariant``,
+  ``EncodeVariant``) in :mod:`eval_toolkit.preprocessing`. Frozen +
+  ``slots=True`` thin wrappers over the existing :func:`delimit` /
+  :func:`datamark` / :func:`encode` functions. Closes Audit R5-F3
+  (Codex Round 5) — prior to this commit, ``preprocessing.__all__`` exported
+  only functions, so the "concrete classes satisfy ``TextTransform``
+  structurally" claim only held on the adversarial side. Now both sides
+  share the dataclass-strategy shape.
+- ``metric_specs.make_spec_name(prefix, **kwargs)`` canonicalization helper
+  for custom parameterized :class:`MetricSpec` implementations. Alphabetized
+  kwargs joined by underscore — same convention the v0.46 ECE factory uses.
+  Lands in ``metric_specs.__all__`` only; **not** top-level ``__all__`` per
+  Decision R6-H. (Closes Round 6 Gemini R6-F4.)
+
+### Changed (Round 6 follow-on)
+
+- ``scorecard()`` now raises ``ValueError`` when two :class:`MetricSpec`
+  instances in the ``metrics`` list share a ``name``. Forces caller
+  disambiguation; the ``Mapping[str, MetricResult]`` contract never silently
+  drops a cell. Error message reports both indices. (Decision R6-B; closes
+  Round 6 Codex R6-F3.)
+- ``scorecard(seed=None)`` docstring rewritten to document the deterministic-
+  by-default contract (``None`` is treated as ``seed=0``). No behavior
+  change; v0.46 documented the wrong contract. (Decision R6-A; closes Round 6
+  Codex R6-F4 + Gemini R6-F1.)
+- ``_evaluate_spec()`` exception catches narrowed: ``MemoryError``,
+  ``RecursionError``, ``KeyboardInterrupt``, and ``SystemExit`` now propagate
+  out of ``scorecard()`` instead of being captured as a ``status="error"``
+  cell. Per-cell isolation remains for ordinary application errors.
+  (Decision R6-F5; closes Round 6 Gemini R6-F5.)
+- ``Scorecard.to_pandas()`` MultiIndex schema extended with two new inner-
+  field columns: ``n_resamples`` (int / NaN sentinel) and ``method``
+  (string / ``""`` sentinel). The DataFrame view is now lossless against
+  :meth:`BootstrapCI.to_dict` — trace provenance (resample count + CI
+  method) no longer drops at the DataFrame boundary. Callers indexing the
+  MultiIndex by name keep working; callers indexing by position must
+  re-check column offsets. (Decision R6-C; closes Round 6 Gemini R6-F3.)
+- ``tests/test_public_api.py`` drift guard now captures method signatures
+  for ``typing.Protocol`` classes in ``__all__`` (a ``protocol_methods``
+  sub-entry in the snapshot). Together with a Tier-2 coverage test, this
+  actually enforces the strict method-shape stability ADR 0003 promises
+  for the 9 Tier-2 Protocols. (Decision R6-D; closes Round 6 Codex R6-F5.)
+  Public-API golden regenerated alongside this change.
+
+## [0.46.1] — 2026-05-21 — Round 6 hotfix: ECE strategy validation + deprecation warning content
+
+Hotfix release per **Decision Q** (data correctness regression + time-sensitive
+warning content) + **Decision R6-E** (scope: R6-F1 + R6-F2 only; R6-A docstring
+rolls forward to v0.47). All other Round 6 findings dispositioned to v0.47.0.
+
+See [`docs/source/audit_findings.md`](docs/source/audit_findings.md) Round 6 for
+the full disposition ledger.
+
+### Fixed
+
+- **`metric_specs.ece(strategy=<value>)` strategy validation** (Round 6 Codex
+  R6-F1). Prior to v0.46.1, an invalid strategy string (e.g.
+  `metric_specs.ece(strategy="typo")`) silently dispatched to quantile ECE and
+  returned a `scorecard()` cell with `status="ok"` under an invalid encoded key
+  (`"ece_n_bins_15_strategy_typo"`) — wrong-by-design data correctness path.
+  Verified by Codex via runtime probe. Now both the `ece()` factory and
+  `_EceSpec.compute()` raise:
+  ```
+  ValueError: ECE strategy must be 'uniform' or 'quantile'; got 'typo'
+  ```
+  Defence-in-depth: the factory validates eagerly (before LRU cache hit) AND
+  `compute()` validates at the compute boundary so direct construction of
+  `_EceSpec(strategy="typo")` (bypassing the factory) also raises.
+
+- **Deprecation warning content for all 5 ECE variants** (Round 6 Codex R6-F2 +
+  Gemini R6-F2, with Decisions R6-F + R6-G). The v0.46.0 `__getattr__`
+  deprecation shim's warning messages produced broken migration snippets:
+  - For `expected_calibration_error` + `expected_calibration_error_equal_mass`:
+    the suggested `Scorecard` lookup key was the factory-call expression
+    (`"ece(n_bins=10)"`) instead of the encoded spec name
+    (`"ece_n_bins_10_strategy_uniform"`). Now uses the correct encoded key.
+  - For `expected_calibration_error_debiased` / `_l2` / `_l2_debiased`: these
+    variants are not in the v0.46 `metric_specs` namespace (Decision R6-G;
+    research-completeness primitives, deferred to v1.x if user demand
+    surfaces). Their warnings now point at the submodule path
+    (`from eval_toolkit.metrics import expected_calibration_error_debiased`)
+    instead of an unconstructable scorecard snippet.
+  - Pre-v0.46 default verification: Gemini's report claimed
+    `expected_calibration_error` defaulted to `n_bins=15`; verified against
+    `metrics.py:730-734` that the actual default is `n_bins=10`. Per Decision
+    R6-F, warning snippets use `n_bins=10` to preserve bit-identical pre-v0.46
+    math + add a migration note explaining the new `metric_specs.ece()` factory
+    default of `n_bins=15` (matching Hines et al.).
+
+### Tests
+
+- `tests/test_scorecard.py`: 4 new tests for ECE strategy validation
+  (parametrized factory-rejection + compute-defence-in-depth).
+- `tests/test_deprecated_scalars_shim.py`: 4 new test classes — verify each
+  warning contains correct factory expression + encoded scorecard key, ECE
+  warnings carry the n_bins=10/15 migration note, submodule-only warnings cite
+  `eval_toolkit.metrics` path, and the snippet in each first-party warning is
+  EXECUTABLE (parses + runs against synthetic data + produces ok-status cell).
+
+### Rolled forward to v0.47 (Decision R6-E)
+
+- R6-A `seed=None` docstring fix (non-blocker per Decision Q).
+- R6-F3 duplicate `MetricSpec.name` rejection.
+- R6-F5 (Codex) Protocol method-shape drift guard.
+- R6-F3 (Gemini) `Scorecard.to_pandas()` schema expansion.
+- R6-F4 (Gemini) `make_spec_name()` helper.
+- R6-F5 (Gemini) narrow `_evaluate_spec()` exception catch.
+- R6-F6 (Codex) plan + roadmap state-drift refresh.
+
 ## [0.46.0] — 2026-05-21 — Scorecard: primary v1.0 metric surface (closes #36)
 
 Second minor of the staggered v0.45 → v0.46 → v0.47 → v0.48 → v1.0 sequence.

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 0.46.0
+Version: 0.47.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -215,7 +215,7 @@ pip install "eval-toolkit[all]"                 # everything
 
 ```python
 import numpy as np
-from eval_toolkit import pr_auc, roc_auc, expected_calibration_error
+from eval_toolkit.metrics import pr_auc, roc_auc, expected_calibration_error
 
 rng = np.random.default_rng(42)
 y = rng.integers(0, 2, size=200)
@@ -230,7 +230,8 @@ print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")
 ### Bootstrap confidence intervals
 
 ```python
-from eval_toolkit import bootstrap_ci, paired_bootstrap_diff, pr_auc
+from eval_toolkit import bootstrap_ci, paired_bootstrap_diff
+from eval_toolkit.metrics import pr_auc
 
 ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, seed=42)
 print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")
@@ -244,8 +245,10 @@ print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")
 ### Temperature scaling (Guo et al. 2017)
 
 ```python
+import numpy as np
 from eval_toolkit import fit_temperature
 
+rng = np.random.default_rng(42)
 logits = rng.normal(size=(500, 2))
 labels = (logits[:, 1] > logits[:, 0]).astype(int)
 result = fit_temperature(logits, labels)

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/README.md

@@ -132,7 +132,7 @@ pip install "eval-toolkit[all]"                 # everything
 
 ```python
 import numpy as np
-from eval_toolkit import pr_auc, roc_auc, expected_calibration_error
+from eval_toolkit.metrics import pr_auc, roc_auc, expected_calibration_error
 
 rng = np.random.default_rng(42)
 y = rng.integers(0, 2, size=200)
@@ -147,7 +147,8 @@ print(f"ECE (10 bins): {expected_calibration_error(y, s, n_bins=10):.3f}")
 ### Bootstrap confidence intervals
 
 ```python
-from eval_toolkit import bootstrap_ci, paired_bootstrap_diff, pr_auc
+from eval_toolkit import bootstrap_ci, paired_bootstrap_diff
+from eval_toolkit.metrics import pr_auc
 
 ci = bootstrap_ci(y, s, pr_auc, n_resamples=1000, seed=42)
 print(f"PR-AUC: {ci.point_estimate:.3f}  95% CI: [{ci.ci_low:.3f}, {ci.ci_high:.3f}]")
@@ -161,8 +162,10 @@ print(f"Δ PR-AUC: {diff.delta:.3f}  overlaps zero: {diff.overlaps_zero}")
 ### Temperature scaling (Guo et al. 2017)
 
 ```python
+import numpy as np
 from eval_toolkit import fit_temperature
 
+rng = np.random.default_rng(42)
 logits = rng.normal(size=(500, 2))
 labels = (logits[:, 1] > logits[:, 0]).astype(int)
 result = fit_temperature(logits, labels)

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/docs/source/adr/README.md

@@ -73,4 +73,6 @@ What would have to change for this decision to be reopened?
 
 | # | Title | Status | Date |
 |---|---|---|---|
-| _none yet_ | | | |
+| [0001](0001-flat-module-layout.md) | Flat single-file modules through v1.x | Accepted | 2026-05-21 |
+| [0002](0002-scorecard-as-primary-metric-surface.md) | `scorecard()` as the primary v1.0 metric surface | Accepted | 2026-05-21 |
+| [0003](0003-stability-contract-and-gate3-methodology.md) | v1.0 stability contract + Gate 3 methodology | Accepted | 2026-05-21 |

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/src/eval_toolkit/__init__.py

@@ -31,22 +31,36 @@ _logging.getLogger("eval_toolkit").addHandler(_logging.NullHandler())
 # tests/golden/public_api/ reads dict keys + values, not comments.
 _EXPORTS: dict[str, str] = {
     # --- adversarial ---
+    "ADVANCED_TECHNIQUES": "eval_toolkit.adversarial",
+    "ALL_TECHNIQUES": "eval_toolkit.adversarial",
+    "BidiRTLInjection": "eval_toolkit.adversarial",
     "CORE_TECHNIQUES": "eval_toolkit.adversarial",
     "CaseRandomization": "eval_toolkit.adversarial",
-    "CharacterInjectionStrategy": "eval_toolkit.adversarial",
     "DiacriticInjection": "eval_toolkit.adversarial",
     "HomoglyphSubstitution": "eval_toolkit.adversarial",
+    "InvisibleCharsInjection": "eval_toolkit.adversarial",
     "PunctuationInjection": "eval_toolkit.adversarial",
+    "SynonymSubstitution": "eval_toolkit.adversarial",
+    "TagStrippingInjection": "eval_toolkit.adversarial",
+    "TokenSplitting": "eval_toolkit.adversarial",
+    "UnicodeNormalization": "eval_toolkit.adversarial",
     "WhitespaceInjection": "eval_toolkit.adversarial",
     "ZeroWidthSpaceInjection": "eval_toolkit.adversarial",
-    "character_injection": "eval_toolkit.adversarial",
+    # CharacterInjectionStrategy + character_injection SimpleNamespace
+    # removed at v0.47 (Decision N + plan §4E). TextTransform Protocol +
+    # the 12 concrete dataclasses are now the only public path.
     # --- losses ---
     "RecallAtLowFPR": "eval_toolkit.losses",
     # --- preprocessing ---
+    # `spotlighting` SimpleNamespace removed at v0.47 (Decision N + plan §4E).
+    # The 3 Variant dataclasses + the underlying functional API are the
+    # only public path.
+    "DatamarkVariant": "eval_toolkit.preprocessing",
+    "DelimitVariant": "eval_toolkit.preprocessing",
+    "EncodeVariant": "eval_toolkit.preprocessing",
     "datamark": "eval_toolkit.preprocessing",
     "delimit": "eval_toolkit.preprocessing",
     "encode": "eval_toolkit.preprocessing",
-    "spotlighting": "eval_toolkit.preprocessing",
     # --- probes ---
     "ActivationDeltaProbe": "eval_toolkit.probes",
     "ActivationExtractor": "eval_toolkit.probes",
@@ -247,6 +261,7 @@ _EXPORTS: dict[str, str] = {
     "PredictionReader": "eval_toolkit.protocols",
     "Scorer": "eval_toolkit.protocols",
     "SliceAwareScorer": "eval_toolkit.protocols",
+    "TextTransform": "eval_toolkit.protocols",
     "Versioned": "eval_toolkit.protocols",
     # --- seeds ---
     "set_global_seeds": "eval_toolkit.seeds",
@@ -298,64 +313,28 @@ _EXPORTS: dict[str, str] = {
     "MetricSpec": "eval_toolkit._scorecard",
     "Scorecard": "eval_toolkit._scorecard",
     "scorecard": "eval_toolkit._scorecard",
+    # --- sweep (top-level v0.47 unification — Decision K + Decision D) ---
+    "sweep": "eval_toolkit._sweep",
 }
 
 __all__ = ["__version__", *_EXPORTS.keys()]
 
 
-# ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
-# At v0.46 the scalar metric functions left the top-level `_EXPORTS` map (above)
-# in favor of the `scorecard()` surface (Decision A). To give the consumer one
-# release of overlap before the hard removal at v0.47, the names below remain
-# reachable via the package-level `__getattr__` (which delegates to the
-# `eval_toolkit.metrics` submodule) but emit a `DeprecationWarning` on first
-# lookup pointing at the new API.
-#
-# WHY THIS IS A BRANCH, NOT A REPLACEMENT (Audit F4 — Round 5):
-# `__getattr__` below is the load-bearing lazy export resolver for every name
-# in `_EXPORTS`. The deprecation branch is a discrete `if name in
-# _DEPRECATED_SCALARS` check ABOVE the resolver — the resolver's existing
-# behavior for non-deprecated names is unchanged. At v0.47 we delete this
-# transitional block and the resolver continues to work for every remaining
-# `_EXPORTS` entry.
-_DEPRECATED_SCALARS: frozenset[str] = frozenset(
-    {
-        "pr_auc",
-        "roc_auc",
-        "brier_score",
-        "expected_calibration_error",
-        "expected_calibration_error_debiased",
-        "expected_calibration_error_equal_mass",
-        "expected_calibration_error_l2",
-        "expected_calibration_error_l2_debiased",
-    }
-)
-# ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
-
-
 def __getattr__(name: str) -> Any:
-    """Resolve public symbols lazily."""
+    """Resolve public symbols lazily.
+
+    v0.47 cleanup (Decision L): the BEGIN/END TRANSITIONAL DEPRECATION
+    BRANCH that v0.46 inserted in front of the resolver — together with the
+    ``_DEPRECATED_SCALARS`` frozenset and the ``_deprecation_warning_for``
+    helper — has been removed. The lazy resolver below is the v0.46 base
+    behavior; with the transitional block gone, deprecated v0.45 scalar names
+    (``pr_auc``, ``roc_auc``, ``brier_score``, the 5 ``expected_calibration_*``
+    variants) now raise :class:`AttributeError` cleanly. Submodule-level
+    access (e.g., ``from eval_toolkit.metrics import pr_auc``) is unaffected
+    per Decision C / ADR 0002.
+    """
     if name == "__version__":
         return __version__
-    # ── BEGIN TRANSITIONAL DEPRECATION BRANCH (Decision L; REMOVE AT v0.47) ──
-    if name in _DEPRECATED_SCALARS:
-        import warnings
-
-        warnings.warn(
-            f"eval_toolkit.{name} is deprecated and will be removed in v0.47. "
-            f"Use `scorecard(y, s, metrics=[metric_specs.{_scorecard_spec_for(name)}])"
-            f'["{_scorecard_spec_for(name)}"].value` instead, or "import from the'
-            f" `eval_toolkit.metrics` submodule directly (internal API).",
-            DeprecationWarning,
-            stacklevel=2,
-        )
-        module = import_module("eval_toolkit.metrics")
-        value = getattr(module, name)
-        # Do NOT cache in globals() — repeated lookups should keep re-warning
-        # (one warning per call site, modulo Python's default
-        # DeprecationWarning de-duplication).
-        return value
-    # ── END TRANSITIONAL DEPRECATION (Decision L; REMOVE AT v0.47) ──
     module_name = _EXPORTS.get(name)
     if module_name is None:
         raise AttributeError(f"module 'eval_toolkit' has no attribute {name!r}")
@@ -365,29 +344,6 @@ def __getattr__(name: str) -> Any:
     return value
 
 
-# ── BEGIN TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
-def _scorecard_spec_for(deprecated_name: str) -> str:
-    """Map a deprecated-scalar name to its `metric_specs` replacement name.
-
-    Used only inside the v0.46 deprecation warning message. Returns the
-    closest equivalent first-party spec name where one exists; falls back
-    to the original name for ECE variants whose exact-match spec isn't in
-    the v0.46 first-party namespace (e.g., the L2 / debiased variants —
-    callers either implement a custom `MetricSpec` or stay on the
-    submodule path).
-    """
-    return {
-        "pr_auc": "pr_auc",
-        "roc_auc": "roc_auc",
-        "brier_score": "brier",
-        "expected_calibration_error": "ece(n_bins=10)",
-        "expected_calibration_error_equal_mass": 'ece(n_bins=10, strategy="quantile")',
-    }.get(deprecated_name, deprecated_name)
-
-
-# ── END TRANSITIONAL DEPRECATION HELPER (Decision L; REMOVE AT v0.47) ──
-
-
 def __dir__() -> list[str]:
     """Expose lazy public symbols to introspection."""
     return sorted(__all__)

{eval_toolkit-0.46.0 → eval_toolkit-0.47.0}/src/eval_toolkit/_scorecard.py

@@ -261,10 +261,17 @@ class Scorecard(Mapping[str, MetricResult]):
         ``ImportError`` with an install hint when pandas is missing.
 
         The DataFrame has 1 row (one slice) and a 2-level column index:
-        outer = metric name, inner = field name in
-        ``{"value", "status", "reason", "ci_low", "ci_high", "confidence"}``.
-        ``ci_low`` / ``ci_high`` / ``confidence`` are ``NaN`` / ``""`` when
-        no CI is present.
+        outer = metric name, inner = field name in ``{"value", "status",
+        "reason", "ci_low", "ci_high", "confidence", "n_resamples",
+        "method"}``. CI-related columns (``ci_low``, ``ci_high``,
+        ``confidence``, ``n_resamples``, ``method``) are sentinel-valued
+        (``NaN`` for numeric, ``""`` for string) when no CI is present
+        (status="skipped" / "error", or bootstrap=False).
+
+        Decision R6-C (Round 6 audit, Gemini F3): the v0.47 expansion adds
+        ``n_resamples`` + ``method`` so the schema is lossless against
+        :meth:`BootstrapCI.to_dict` — trace provenance no longer drops in
+        the DataFrame view.
         """
         try:
             import pandas as pd
@@ -285,6 +292,8 @@ class Scorecard(Mapping[str, MetricResult]):
                     (name, "ci_low"),
                     (name, "ci_high"),
                     (name, "confidence"),
+                    (name, "n_resamples"),
+                    (name, "method"),
                 ]
             )
             values.extend(
@@ -295,6 +304,8 @@ class Scorecard(Mapping[str, MetricResult]):
                     result.ci.ci_low if result.ci is not None else float("nan"),
                     result.ci.ci_high if result.ci is not None else float("nan"),
                     result.ci.confidence if result.ci is not None else float("nan"),
+                    result.ci.n_resamples if result.ci is not None else float("nan"),
+                    result.ci.method if result.ci is not None else "",
                 ]
             )
 
@@ -341,7 +352,13 @@ def scorecard(
     confidence : float, optional
         Two-sided CI level ∈ ``(0, 1)``. Default ``0.95``.
     seed : int or None, optional
-        Bootstrap RNG seed. Default ``None`` (non-deterministic).
+        Bootstrap RNG seed. Default ``None``, which is treated as ``seed=0``
+        for reproducibility — eval-toolkit's evaluation pipelines are
+        deterministic by default. Pass an explicit integer to control the
+        bootstrap RNG; pass a value derived from
+        ``np.random.SeedSequence().entropy`` for non-deterministic sampling.
+        Decision R6-A (Round 6 audit) locked the deterministic-by-default
+        contract; the prior docstring framing was incorrect.
 
     Returns
     -------
@@ -410,6 +427,7 @@ def scorecard(
         confidence=confidence,
         bootstrap=bootstrap,
     )
+    _validate_unique_spec_names(metrics)
 
     is_single_class = bool(np.unique(y_true_arr).size < 2)
 
@@ -450,6 +468,11 @@ def _evaluate_spec(
 
     try:
         point = float(spec.compute(y_true, y_score))
+    except (MemoryError, RecursionError, KeyboardInterrupt, SystemExit):
+        # Process-exhaustion / user-interrupt signals must propagate;
+        # per-cell isolation is for application-level errors only.
+        # Decision R6-F5 (Round 6 audit, Gemini).
+        raise
     except Exception as exc:  # noqa: BLE001 — broad catch is intentional (per-cell isolation)
         return MetricResult(
             value=None,
@@ -469,6 +492,9 @@ def _evaluate_spec(
             confidence=confidence,
             seed=seed if seed is not None else 0,
         )
+    except (MemoryError, RecursionError, KeyboardInterrupt, SystemExit):
+        # Same R6-F5 invariant for the bootstrap path.
+        raise
     except Exception as exc:  # noqa: BLE001
         # Point estimate succeeded; the bootstrap couldn't (e.g., n < 10
         # floor from bootstrap.py:198, BCa degeneracy, etc.). Record the
@@ -507,3 +533,21 @@ def _validate_scorecard_inputs(
             raise ValueError(f"n_resamples must be >= 1 when bootstrap=True; got {n_resamples}")
         if not 0.0 < confidence < 1.0:
             raise ValueError(f"confidence must be in (0, 1); got {confidence}")
+
+
+def _validate_unique_spec_names(metrics: Sequence[MetricSpec]) -> None:
+    """Reject duplicate MetricSpec.name values in a single scorecard() call.
+
+    Locked by Decision R6-B (Round 6 audit, Codex R6-F3): silent last-wins
+    overwrite is not a documented Mapping[str, MetricResult] contract. Force
+    the caller to disambiguate so we never lose data on user error.
+    """
+    seen: dict[str, int] = {}
+    for i, spec in enumerate(metrics):
+        if spec.name in seen:
+            raise ValueError(
+                f"Duplicate MetricSpec name {spec.name!r} at index {i} "
+                f"(previously at index {seen[spec.name]}); each spec must have a "
+                f"unique name for the Scorecard Mapping[str, MetricResult] contract."
+            )
+        seen[spec.name] = i