eval-toolkit 1.2.0__tar.gz → 1.3.0__tar.gz

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/CHANGELOG.md

@@ -5,6 +5,120 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0] — 2026-05-26 — `audit_value_bindings` cross-detector list-grammar pairing rules (closes #81)
+
+Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).
+Closes [#81](https://github.com/brandon-behring/eval-toolkit/issues/81)
+— consumer-feedback follow-on after v1.2.0's adoption at
+`prompt-injection-detection-submission@v1.3.12` (4 residual
+warnings, all cross-detector list-grammar or metric-axis
+confusion). Introduces **Layer 3 — pairing rules** as the third
+correctness layer alongside ADR 0005's identity + scope model
+(see new [ADR 0006](docs/source/adr/0006-pairing-rules-for-cross-detector-list-grammar.md)).
+
+Consumer-side dogfood result: **4 → 0 warnings**. Combined with
+v1.1.0 + v1.2.0, **100% reduction vs the pre-fix v1.0.5 baseline**
+on the consumer's writeup (95 → 0).
+
+### Added — `audit_value_bindings.py` Layer 3 pairing rules
+
+All four rules activate ONLY when `scope="narrative"`. Legacy
+`scope="all"` callers see zero behavior change. No new public
+kwargs; keyword sets are hardcoded module-level `frozenset`
+constants.
+
+- **Pattern A — `"for {detector}"` postfix override.** When a
+  candidate value is followed (within +50 chars) by `"for
+  {detector_alias}"` AND no other value lies between, the
+  postfix is authoritative: confirms pairing for this binding
+  OR skips if it names a different canonical detector.
+  Intervening-value check uses the v1.1.0 exclusion-ranges
+  infrastructure (CI brackets like `[0.286, 0.301]` don't count
+  as intervening values).
+- **Pattern B — `"{detector}'s"` possessive override.** Same
+  mechanics; scans −80 chars before the value. Last possessive
+  in the pre-window is authoritative if its end is within 30
+  chars of the value start. Catches both immediate `"frozen
+  probe's 0.515"` and short-clause `"LoRA's ... AUROC is 0.383"`.
+- **Pattern C — group-subject suppression.** When prose contains
+  `"for the {trained|frozen|baseline|all|both|other} detectors"`
+  within ±60 chars of the value AND on the same side of any
+  sentence boundary, the value is suppressed (it refers to a
+  multi-detector group statement that doesn't bind to a single
+  canonical detector). Multi-detector inference deferred to v1.4.0+
+  per ADR 0006.
+- **Pattern D — metric-axis nearest-pairing.** Symmetric to
+  detector-axis pairing. Pre-collects ALL metric positions per
+  file (across `metric_aliases` keys, not just binding-derived
+  metrics). Requires the NEAREST metric to the value to be THIS
+  binding's metric. Catches prose like `"AUPRC delta suggests:
+  ... AUROC is 0.383"` where the wider window-based metric
+  proximity check picks up the wrong metric.
+
+### Internal changes (no public API impact)
+
+- New module-level constants:
+  - `_GROUP_SUBJECT_KEYWORDS: frozenset[str]` — group adjectives.
+  - `_GROUP_SUBJECT_PATTERN: re.Pattern[str]` — compiled regex
+    matching `"for the {kw} detectors?"`.
+- New private helpers:
+  - `_build_postfix_pattern(detector_aliases, detector_keys)` —
+    per-call regex builder for Pattern A.
+  - `_build_possessive_pattern(detector_aliases, detector_keys)` —
+    per-call regex builder for Pattern B.
+- `metric_patterns` build extended to use the union of
+  `binding-derived` and `metric_aliases.keys()` so Pattern D can
+  pair against unbound-but-aliased metrics.
+- Inner loop reordered to apply C-suppress → Pattern A → Pattern B
+  before proximity-based detector pairing. Pattern A/B record a
+  `pairing_confirmed_pos` that BYPASSES proximity when the override
+  confirms THIS binding's detector.
+- Pattern D added as a separate check after the existing
+  metric_close proximity test.
+
+### Dogfood evidence (compounded across the cycle)
+
+| Release | Configuration | Warnings on consumer HEAD | Reduction vs v1.0.5 |
+|---|---|---|---|
+| v1.0.5 | Legacy 2-tuple, no scope | 95 | — |
+| v1.1.0 | BindingKey + scope='narrative' content-type | 23 | -76% |
+| v1.2.0 | + T1–T4 context filters | 7 | -93% |
+| **v1.3.0** | + Pattern A/B/C/D pairing rules | **0** | **-100%** |
+
+### Consumer adoption path
+
+`prompt-injection-detection-submission` and other consumers using
+`scope="narrative"` get the v1.3.0 pairing rules automatically with
+no code change. Recommended migration:
+
+1. Re-pin `eval-toolkit>=1.3.0,<2` (additive; no consumer code
+   change required).
+2. **HARD-gate promotion is now credible.** With 0 residual
+   warnings, `audit_value_bindings` can be promoted from SOFT to
+   HARD (failing CI on violations) bundled with
+   `audit_citation_alignment` per the consumer's v1.3.8
+   bundled-promotion plan.
+
+### Tests
+
+43 in `tests/test_audit_value_bindings.py` (36 from v1.2.0 + 7
+new for Pattern A/B/C/D + unknown-alias fall-through + scope='all'
+backward-compat + combined dogfood). All pass. Public API
+snapshot regenerated for `__version__` bump only (no signature
+changes).
+
+### Out of scope (deferred)
+
+- **Multi-detector inference for Pattern C** — replace
+  suppression with explicit iteration over implied group
+  detectors. ~250 LOC; v1.4.0+ candidate if consumer demand
+  emerges.
+- **Enumeration parsing** — `"X scored Y, Z, W for A, B, C
+  respectively"` patterns. Not in #81; v1.4.0+ if needed.
+- **Markdown AST parsing** (ADR 0005 §A4) — v2.0 territory.
+- **Public kwargs for pairing-rule keyword extension** — YAGNI;
+  add in v1.3.x patch if demand emerges.
+
 ## [1.2.0] — 2026-05-26 — `audit_value_bindings` context-aware noise reduction (consumer-feedback follow-on to #80)
 
 Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.2.0
+Version: 1.3.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/docs/source/adr/README.md

@@ -78,3 +78,4 @@ What would have to change for this decision to be reopened?
 | [0003](0003-stability-contract-and-gate3-methodology.md) | v1.0 stability contract + Gate 3 methodology | Accepted | 2026-05-21 |
 | [0004](0004-naming-conventions.md) | Naming conventions for modules, classes, and parameters | Accepted | 2026-05-23 |
 | [0005](0005-structured-keys-for-audit-validators.md) | Structured keys over positional tuples for canonical-identity types in audit validators | Accepted | 2026-05-26 |
+| [0006](0006-pairing-rules-for-cross-detector-list-grammar.md) | Pairing rules for cross-detector list-grammar in audit validators | Accepted | 2026-05-26 |

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.2.0"
+__version__ = "1.3.0"

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/src/eval_toolkit/audit_value_bindings.py

@@ -154,6 +154,51 @@ _DELTA_PATTERN: re.Pattern[str] = _compile_keyword_pattern(_DELTA_KEYWORDS)
 _FLOOR_PATTERN: re.Pattern[str] = _compile_keyword_pattern(_FLOOR_KEYWORDS)
 
 
+# v1.3.0 Layer 3 (pairing rules) per ADR 0006. Three rules that
+# override or suppress the proximity-based detector pairing under
+# explicit grammar cues:
+#
+# - Pattern A: "for {detector}" postfix → re-pair value to that
+#   detector (override). Built per-call via _build_postfix_pattern
+#   since it depends on the consumer's detector_aliases dict.
+# - Pattern B: "{detector}'s ... is {value}" possessive → re-pair
+#   value to the possessor (override). Built per-call via
+#   _build_possessive_pattern.
+# - Pattern C: "for the {trained|frozen|baseline|all|both|other}
+#   detectors" group subject → suppress the candidate value entirely
+#   (it's a group statement that doesn't bind to a single detector).
+#   Pattern is detector-independent so it compiles once at module
+#   load.
+
+# Group-subject adjectives that introduce a multi-detector statement.
+# When prose says "for the trained detectors", the following value
+# refers to a GROUP (LoRA + TF-IDF + ... whatever bindings exist),
+# not a single canonical detector. The validator can't infer which
+# specific detectors own the group value with positional heuristics,
+# so v1.3.0 suppresses the candidate rather than attempting multi-
+# detector inference (a v1.4.0+ candidate per ADR 0006).
+_GROUP_SUBJECT_KEYWORDS: frozenset[str] = frozenset(
+    {
+        "trained",
+        "frozen",
+        "baseline",
+        "all",
+        "both",
+        "other",
+    }
+)
+
+# Module-level: detector-independent group-subject regex. Matches
+# "for the {trained|frozen|...} detectors" (with optional "the"; both
+# singular and plural "detector"/"detectors" tolerated).
+_GROUP_SUBJECT_PATTERN: re.Pattern[str] = re.compile(
+    r"\bfor\s+(?:the\s+)?(?:"
+    + "|".join(sorted(re.escape(kw) for kw in _GROUP_SUBJECT_KEYWORDS))
+    + r")\s+detectors?\b",
+    re.IGNORECASE,
+)
+
+
 @dataclass(frozen=True)
 class BindingKey:
     """Canonical identity for a `(detector, metric, slice)` measurement.
@@ -469,7 +514,13 @@ def validate_reader_value_bindings(
     slice_aliases_dict: dict[str, Sequence[str]] = dict(slice_aliases) if slice_aliases else {}
 
     detector_keys = sorted({k.detector for k in canonical_bindings})
-    metric_keys = sorted({k.metric for k in canonical_bindings})
+    # v1.3.0 Pattern D (metric-axis nearest-pairing) requires knowing
+    # ALL metrics that might appear in prose, not just bound metrics —
+    # e.g., when prose mentions AUROC near a value but only AUPRC is
+    # bound, Pattern D needs the AUROC pattern to correctly pair the
+    # value with the right metric. Union of binding metrics +
+    # consumer-supplied metric_aliases keys.
+    metric_keys = sorted({k.metric for k in canonical_bindings} | set(metric_aliases_dict.keys()))
     # Only compile slice patterns for non-"any" slice keys; "any"
     # signals legacy 2-tuple semantics (no slice scoping).
     slice_keys = sorted({k.slice for k in canonical_bindings if k.slice != "any"})
@@ -488,6 +539,43 @@ def validate_reader_value_bindings(
     }
     value_re = re.compile(value_pattern)
 
+    # v1.3.0 Layer 3 pairing rules (per ADR 0006). Built per-call
+    # because Patterns A and B depend on the consumer's detector
+    # aliases. `None` when scope="all" (legacy; rules don't fire).
+    postfix_pat: re.Pattern[str] | None = (
+        _build_postfix_pattern(detector_aliases_dict, detector_keys)
+        if scope == "narrative"
+        else None
+    )
+    possessive_pat: re.Pattern[str] | None = (
+        _build_possessive_pattern(detector_aliases_dict, detector_keys)
+        if scope == "narrative"
+        else None
+    )
+
+    # Inverse-alias index: alias-regex (string form) → canonical key.
+    # Used to resolve a matched postfix/possessive alias-group back to
+    # the canonical detector for override resolution. Each alias regex
+    # is keyed verbatim; the resolution path tries each canonical key's
+    # alias list + canonical-name fallback.
+    def _resolve_canonical_from_alias_match(alias_text: str) -> str | None:
+        """Return the canonical detector key whose pattern matched ``alias_text``.
+
+        Iterates the per-detector patterns and tries to match the
+        alias_text. Uses re.IGNORECASE for consistency with the
+        outer postfix/possessive patterns. First-match wins (the
+        OR-build above means there's only one canonical key per
+        match anyway in practice).
+        """
+        for det_key in detector_keys:
+            det_pat = detector_patterns[det_key]
+            # det_pat is the alias OR pattern from _build_pattern,
+            # case-insensitive. fullmatch on the alias_text checks
+            # whether this alias belongs to det_key's set.
+            if det_pat.fullmatch(alias_text):
+                return det_key
+        return None
+
     violations: list[Violation] = []
     matched: list[Match] = []
     matched_keys: set[BindingKey] = set()
@@ -542,6 +630,19 @@ def validate_reader_value_bindings(
                 slice_positions.append((s_match.start(), s_key))
         slice_positions.sort()
 
+        # v1.3.0 Pattern D — metric-axis nearest-pairing (Layer 3 per
+        # ADR 0006, narrative-scope only). Pre-collect ALL metric
+        # positions so each value can be paired with its NEAREST
+        # metric mention (text-order). Catches the case where prose
+        # mentions BOTH metrics ("AUPRC delta suggests: AUROC 0.383")
+        # and the validator's window-based metric proximity check
+        # picks up the wrong metric. Symmetric to detector pairing.
+        metric_positions: list[tuple[int, str]] = []  # (position, canonical_metric)
+        for m_key, m_re in metric_patterns.items():
+            for m_match in m_re.finditer(text):
+                metric_positions.append((m_match.start(), m_key))
+        metric_positions.sort()
+
         # For each canonical binding, look in each file for triples.
         for canonical_key, expected in canonical_bindings.items():
             det_key = canonical_key.detector
@@ -615,18 +716,110 @@ def validate_reader_value_bindings(
                     ):
                         continue
 
-                    # Cross-detector disambiguation: require the current
-                    # det_key to be the detector paired with this value
-                    # by the text-order rule (last detector before; else
-                    # first detector after). Avoids cross-contamination
-                    # on multi-detector prose like "TF-IDF achieves
-                    # 0.971, while LoRA reaches 0.974".
-                    detector_match = _nearest_canonical_key(
-                        detector_positions, val_start_in_full, max_distance_chars
-                    )
-                    if detector_match is None or detector_match[0] != det_key:
-                        continue
-                    paired_det_pos = detector_match[1]
+                    # v1.3.0 Pattern C — group-subject suppression
+                    # (narrative-scope only). When prose says "for the
+                    # {trained|frozen|baseline|all|both|other}
+                    # detectors" within ±60 chars of the value AND on
+                    # the same side of any sentence boundary, the
+                    # value refers to a multi-detector group statement
+                    # that doesn't bind to a single canonical detector.
+                    # Suppress the candidate (v1.4.0+ may attempt
+                    # multi-detector inference per ADR 0006).
+                    if scope == "narrative":
+                        gs_start = max(0, val_start_in_full - 60)
+                        gs_end = min(len(text), val_start_in_full + len(val_str) + 60)
+                        gs_match = _GROUP_SUBJECT_PATTERN.search(text, gs_start, gs_end)
+                        if gs_match is not None and not _crosses_sentence_boundary(
+                            gs_match.start(), val_start_in_full, sentence_positions
+                        ):
+                            continue
+
+                    # v1.3.0 Pattern A / B — Layer 3 pairing-rule
+                    # OVERRIDES (narrative-scope only). When a postfix
+                    # or possessive explicitly names a detector, the
+                    # override is AUTHORITATIVE — it confirms or
+                    # rejects the binding without falling through to
+                    # the proximity-based detector pairing below.
+                    #
+                    # - postfix_confirmed_pos / possessive_confirmed_pos:
+                    #   the character position of the override match,
+                    #   used as the effective "paired detector
+                    #   position" for downstream T4 (sentence-
+                    #   boundary) check.
+                    # - If postfix/possessive_canonical == det_key:
+                    #   confirmed; bypass proximity.
+                    # - If != det_key AND is in bindings: skip (the
+                    #   other detector's loop iteration claims it).
+                    # - If doesn't resolve / no match: fall through
+                    #   to proximity-based pairing.
+                    pairing_confirmed_pos: int | None = None
+
+                    # Pattern A — "for {detector}" postfix
+                    if postfix_pat is not None:
+                        val_end = val_start_in_full + len(val_str)
+                        pf_match = postfix_pat.search(text, val_end, min(len(text), val_end + 50))
+                        if pf_match is not None:
+                            # Intervening-value guard: prose like
+                            # "X 0.971 versus 0.293 for LoRA" — the
+                            # "for LoRA" postfix belongs to 0.293,
+                            # not 0.971. CI brackets like `[0.283,
+                            # 0.298]` are excluded from intervening
+                            # consideration via the existing
+                            # excluded_ranges (v1.1.0 scope filter):
+                            # values inside brackets aren't real
+                            # binding-candidate intervening values.
+                            intervening: re.Match[str] | None = None
+                            for m in value_re.finditer(text, val_end, pf_match.start()):
+                                if not (
+                                    excluded_ranges and _is_excluded(m.start(), excluded_ranges)
+                                ):
+                                    intervening = m
+                                    break
+                            if intervening is None:
+                                postfix_canonical = _resolve_canonical_from_alias_match(
+                                    pf_match.group(1)
+                                )
+                                if postfix_canonical is not None:
+                                    if postfix_canonical != det_key:
+                                        continue
+                                    pairing_confirmed_pos = pf_match.start()
+
+                    # Pattern B — possessive `'s` (only if Pattern A
+                    # didn't already confirm). Find the LAST possessive
+                    # in the −80 char pre-window; if its end is within
+                    # 30 chars of the value start, apply override.
+                    if pairing_confirmed_pos is None and possessive_pat is not None:
+                        ps_matches = list(
+                            possessive_pat.finditer(
+                                text, max(0, val_start_in_full - 80), val_start_in_full
+                            )
+                        )
+                        if ps_matches:
+                            ps_match = ps_matches[-1]
+                            if val_start_in_full - ps_match.end() <= 30:
+                                possessive_canonical = _resolve_canonical_from_alias_match(
+                                    ps_match.group(1)
+                                )
+                                if possessive_canonical is not None:
+                                    if possessive_canonical != det_key:
+                                        continue
+                                    pairing_confirmed_pos = ps_match.start()
+
+                    # Detector pairing: when a Layer 3 override
+                    # confirmed the binding (pairing_confirmed_pos
+                    # set), skip the proximity check — the postfix /
+                    # possessive is authoritative. Otherwise, fall
+                    # back to the text-order proximity rule (last
+                    # detector before; else first detector after).
+                    if pairing_confirmed_pos is not None:
+                        paired_det_pos = pairing_confirmed_pos
+                    else:
+                        detector_match = _nearest_canonical_key(
+                            detector_positions, val_start_in_full, max_distance_chars
+                        )
+                        if detector_match is None or detector_match[0] != det_key:
+                            continue
+                        paired_det_pos = detector_match[1]
 
                     # v1.2.0 T4 (narrative-scope only): reject the
                     # detector-value pair if a sentence boundary lies
@@ -646,6 +839,22 @@ def validate_reader_value_bindings(
                     if not met_close:
                         continue
 
+                    # v1.3.0 Pattern D — metric-axis nearest-pairing
+                    # (narrative-scope only). Require the NEAREST
+                    # metric mention to the value (by text-order
+                    # last-before-first-after) to be THIS binding's
+                    # canonical metric. Catches prose like "than the
+                    # AUPRC delta suggests: LoRA's pooled OOD AUROC
+                    # is 0.383" where the AUPRC mention from the
+                    # delta clause is within window of 0.383 but
+                    # AUROC is the metric semantically owning it.
+                    if scope == "narrative":
+                        metric_match = _nearest_canonical_key(
+                            metric_positions, val_start_in_full, max_distance_chars
+                        )
+                        if metric_match is not None and metric_match[0] != met_key:
+                            continue
+
                     # Slice disambiguation: when the canonical key is
                     # slice-scoped (slice != "any"), pair the value
                     # with the NEAREST slice mention by the same
@@ -1019,6 +1228,79 @@ def _has_keyword_in_window(
     return bool(pattern.search(text, start, end))
 
 
+def _build_postfix_pattern(
+    detector_aliases: Mapping[str, Sequence[str]],
+    detector_keys: Sequence[str],
+) -> re.Pattern[str] | None:
+    """Build a regex matching `"for {detector_alias}"` postfix constructs.
+
+    v1.3.0 Pattern A (Layer 3 pairing rule per ADR 0006). Used to
+    re-pair a candidate value with the detector named in a "for X"
+    postfix (e.g., ``"0.291 [...] for TF-IDF + LR"`` binds 0.291 to
+    TF-IDF + LR via the postfix, overriding proximity-based pairing).
+
+    Each alias is paired with its canonical key in a single named-group
+    OR pattern; the capture group reveals which detector matched. The
+    canonical-key-as-fallback ensures the canonical name itself is
+    matched even if no alias regex is provided for that detector.
+
+    Returns None if there are no detectors to build patterns for
+    (empty bindings).
+    """
+    if not detector_keys:
+        return None
+    alts: list[str] = []
+    for det_key in detector_keys:
+        # Canonical name as a literal alternative + all alias regexes
+        # (which may themselves contain regex syntax like `\+`).
+        parts = [re.escape(det_key)] + list(detector_aliases.get(det_key, ()))
+        # Each detector's parts collapse into a non-capturing group.
+        alts.append("(?:" + "|".join(parts) + ")")
+    # The outer capture group reveals which detector token matched.
+    # The text-order rule means the first alternative wins per Python
+    # re semantics, which is fine for our use case.
+    return re.compile(
+        r"\bfor\s+(?:the\s+)?(" + "|".join(alts) + r")(?=[\s,;.)\]]|$)",
+        re.IGNORECASE,
+    )
+
+
+def _build_possessive_pattern(
+    detector_aliases: Mapping[str, Sequence[str]],
+    detector_keys: Sequence[str],
+) -> re.Pattern[str] | None:
+    """Build a regex matching `"{detector_alias}'s"` possessive markers.
+
+    v1.3.0 Pattern B (Layer 3 pairing rule per ADR 0006). The
+    possessive ``'s`` construction is a strong binding signal that
+    isn't captured by detector-alias regex matching directly (alias
+    patterns don't typically include the apostrophe). Re-pairs the
+    candidate value with the possessor detector.
+
+    The pattern matches JUST the possessive marker (``{alias}'s``);
+    binding-claim proximity is enforced at the call site (the
+    inner loop's Pattern B block requires the LAST possessive
+    in the pre-window to END within 30 chars of the value, which
+    covers both `"frozen probe's 0.515"` (immediate) and
+    `"LoRA's pooled OOD AUROC is 0.383"` (5-token clause).
+
+    Returns None if there are no detectors (empty bindings).
+    """
+    if not detector_keys:
+        return None
+    alts: list[str] = []
+    for det_key in detector_keys:
+        parts = [re.escape(det_key)] + list(detector_aliases.get(det_key, ()))
+        alts.append("(?:" + "|".join(parts) + ")")
+    # Match `{alias}'s` (ASCII apostrophe or typographic ’s). Tight
+    # — proximity to the value is enforced at the call site via
+    # `match.end()` against the value position.
+    return re.compile(
+        r"(" + "|".join(alts) + r")[’']s\b",
+        re.IGNORECASE,
+    )
+
+
 def _build_pattern(
     canonical: str,
     aliases: Sequence[str],

{eval_toolkit-1.2.0 → eval_toolkit-1.3.0}/tests/golden/public_api/snapshot.json

@@ -1429,7 +1429,7 @@
       "doc_first_line": "str(object='') -> str",
       "kind": "value",
       "type": "str",
-      "value": "'1.2.0'"
+      "value": "'1.3.0'"
     },
     "apply_operating_points": {
       "doc_first_line": "Apply fitted thresholds to a mixed-class or single-class target slice.",