eval-toolkit 1.1.0__tar.gz → 1.3.0__tar.gz

{eval_toolkit-1.1.0 → eval_toolkit-1.3.0}/CHANGELOG.md

@@ -5,6 +5,255 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0] — 2026-05-26 — `audit_value_bindings` cross-detector list-grammar pairing rules (closes #81)
+
+Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).
+Closes [#81](https://github.com/brandon-behring/eval-toolkit/issues/81)
+— consumer-feedback follow-on after v1.2.0's adoption at
+`prompt-injection-detection-submission@v1.3.12` (4 residual
+warnings, all cross-detector list-grammar or metric-axis
+confusion). Introduces **Layer 3 — pairing rules** as the third
+correctness layer alongside ADR 0005's identity + scope model
+(see new [ADR 0006](docs/source/adr/0006-pairing-rules-for-cross-detector-list-grammar.md)).
+
+Consumer-side dogfood result: **4 → 0 warnings**. Combined with
+v1.1.0 + v1.2.0, **100% reduction vs the pre-fix v1.0.5 baseline**
+on the consumer's writeup (95 → 0).
+
+### Added — `audit_value_bindings.py` Layer 3 pairing rules
+
+All four rules activate ONLY when `scope="narrative"`. Legacy
+`scope="all"` callers see zero behavior change. No new public
+kwargs; keyword sets are hardcoded module-level `frozenset`
+constants.
+
+- **Pattern A — `"for {detector}"` postfix override.** When a
+  candidate value is followed (within +50 chars) by `"for
+  {detector_alias}"` AND no other value lies between, the
+  postfix is authoritative: confirms pairing for this binding
+  OR skips if it names a different canonical detector.
+  Intervening-value check uses the v1.1.0 exclusion-ranges
+  infrastructure (CI brackets like `[0.286, 0.301]` don't count
+  as intervening values).
+- **Pattern B — `"{detector}'s"` possessive override.** Same
+  mechanics; scans −80 chars before the value. Last possessive
+  in the pre-window is authoritative if its end is within 30
+  chars of the value start. Catches both immediate `"frozen
+  probe's 0.515"` and short-clause `"LoRA's ... AUROC is 0.383"`.
+- **Pattern C — group-subject suppression.** When prose contains
+  `"for the {trained|frozen|baseline|all|both|other} detectors"`
+  within ±60 chars of the value AND on the same side of any
+  sentence boundary, the value is suppressed (it refers to a
+  multi-detector group statement that doesn't bind to a single
+  canonical detector). Multi-detector inference deferred to v1.4.0+
+  per ADR 0006.
+- **Pattern D — metric-axis nearest-pairing.** Symmetric to
+  detector-axis pairing. Pre-collects ALL metric positions per
+  file (across `metric_aliases` keys, not just binding-derived
+  metrics). Requires the NEAREST metric to the value to be THIS
+  binding's metric. Catches prose like `"AUPRC delta suggests:
+  ... AUROC is 0.383"` where the wider window-based metric
+  proximity check picks up the wrong metric.
+
+### Internal changes (no public API impact)
+
+- New module-level constants:
+  - `_GROUP_SUBJECT_KEYWORDS: frozenset[str]` — group adjectives.
+  - `_GROUP_SUBJECT_PATTERN: re.Pattern[str]` — compiled regex
+    matching `"for the {kw} detectors?"`.
+- New private helpers:
+  - `_build_postfix_pattern(detector_aliases, detector_keys)` —
+    per-call regex builder for Pattern A.
+  - `_build_possessive_pattern(detector_aliases, detector_keys)` —
+    per-call regex builder for Pattern B.
+- `metric_patterns` build extended to use the union of
+  `binding-derived` and `metric_aliases.keys()` so Pattern D can
+  pair against unbound-but-aliased metrics.
+- Inner loop reordered to apply C-suppress → Pattern A → Pattern B
+  before proximity-based detector pairing. Pattern A/B record a
+  `pairing_confirmed_pos` that BYPASSES proximity when the override
+  confirms THIS binding's detector.
+- Pattern D added as a separate check after the existing
+  metric_close proximity test.
+
+### Dogfood evidence (compounded across the cycle)
+
+| Release | Configuration | Warnings on consumer HEAD | Reduction vs v1.0.5 |
+|---|---|---|---|
+| v1.0.5 | Legacy 2-tuple, no scope | 95 | — |
+| v1.1.0 | BindingKey + scope='narrative' content-type | 23 | -76% |
+| v1.2.0 | + T1–T4 context filters | 7 | -93% |
+| **v1.3.0** | + Pattern A/B/C/D pairing rules | **0** | **-100%** |
+
+### Consumer adoption path
+
+`prompt-injection-detection-submission` and other consumers using
+`scope="narrative"` get the v1.3.0 pairing rules automatically with
+no code change. Recommended migration:
+
+1. Re-pin `eval-toolkit>=1.3.0,<2` (additive; no consumer code
+   change required).
+2. **HARD-gate promotion is now credible.** With 0 residual
+   warnings, `audit_value_bindings` can be promoted from SOFT to
+   HARD (failing CI on violations) bundled with
+   `audit_citation_alignment` per the consumer's v1.3.8
+   bundled-promotion plan.
+
+### Tests
+
+43 in `tests/test_audit_value_bindings.py` (36 from v1.2.0 + 7
+new for Pattern A/B/C/D + unknown-alias fall-through + scope='all'
+backward-compat + combined dogfood). All pass. Public API
+snapshot regenerated for `__version__` bump only (no signature
+changes).
+
+### Out of scope (deferred)
+
+- **Multi-detector inference for Pattern C** — replace
+  suppression with explicit iteration over implied group
+  detectors. ~250 LOC; v1.4.0+ candidate if consumer demand
+  emerges.
+- **Enumeration parsing** — `"X scored Y, Z, W for A, B, C
+  respectively"` patterns. Not in #81; v1.4.0+ if needed.
+- **Markdown AST parsing** (ADR 0005 §A4) — v2.0 territory.
+- **Public kwargs for pairing-rule keyword extension** — YAGNI;
+  add in v1.3.x patch if demand emerges.
+
+## [1.2.0] — 2026-05-26 — `audit_value_bindings` context-aware noise reduction (consumer-feedback follow-on to #80)
+
+Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).
+Consumer-feedback follow-on after v1.1.0's adoption at
+`prompt-injection-detection-submission@v1.3.11`. The v1.1.0
+slice-axis fix achieved 62% noise reduction (96 → 36 warnings) on
+the consumer's writeup; the residual 36 were positional-heuristic
+limitations [ADR 0005](docs/source/adr/0005-structured-keys-for-audit-validators.md)
+named as "Future work (deferred)" for v1.2.0+. This release
+addresses 81% of that residual (36 → 7) via four context-aware
+extensions to `scope="narrative"`. Combined with v1.1.0,
+**93% total noise reduction** vs the pre-fix v1.0.5 baseline.
+
+### Added — `audit_value_bindings.py` context-aware narrative filters
+
+All four filters activate ONLY when `scope="narrative"`. Legacy
+`scope="all"` callers see zero behavior change (Tier-1 ADDITIVE).
+No new public kwargs; no signature drift; the keyword lists are
+hardcoded module-level `frozenset` constants. Issue [#80](https://github.com/brandon-behring/eval-toolkit/issues/80)'s
+acceptance criterion was ≤5 warnings; v1.2.0 hits 7 (close to the
+target; the remaining 7 are pure cross-detector list-grammar cases
+that require parser-level work — see "Out of scope" below).
+
+- **T1: Delta-context filter.** Suppresses values that are
+  comparative magnitudes rather than binding claims. Two
+  sub-filters:
+  - Sign-prefix skip: values immediately preceded by `+` or `-`
+    (negative-magnitude markers like `-0.071 AUPRC`,
+    `+0.073 lift`) are dropped.
+  - Delta-keyword skip: values within 30 chars AFTER a
+    delta-marker token are dropped. The before-only window
+    prevents mis-firing on prose like `"frozen probe's 0.515
+    (delta -0.132)"` where the `"delta"` token refers to the
+    following `-0.132`, not the preceding `0.515`.
+
+  Keyword list (`_DELTA_KEYWORDS`, hardcoded frozenset):
+  `delta`, `drop`, `drops`, `lift`, `lifts`, `gap`, `margin`,
+  `regresses`, `improves`, `beats`, `exceeds`, `trails`,
+  `underperforms`, `vs`, `versus`, `below`. Excluded:
+  `against`, `above`, `ahead`, `behind` (too ambiguous; common
+  comparison prepositions in legitimate binding prose).
+
+- **T2: Floor-context filter.** Suppresses values near random-
+  baseline / floor mentions. Window is asymmetric (50 chars
+  before, 5 chars after) because floor mentions canonically
+  precede the value (`"random AUPRC is 0.374"`).
+
+  Keyword list (`_FLOOR_KEYWORDS`): `random`, `floor`, `chance`,
+  `trivial`. Intentionally narrow — `baseline`, `prior`,
+  `majority` excluded because they have legitimate non-floor
+  senses (`"TF-IDF baseline"`, `"prior work"`). Multi-word
+  patterns like `"below the prevalence baseline of 0.374"` are
+  caught by T1's `"below"` keyword instead.
+
+- **T3: Consume-on-match within sentence.** After a value
+  produces a Match for `(detector, metric, slice)`, subsequent
+  values for the same canonical binding in the same sentence are
+  suppressed. Catches dense multi-detector enumerations like
+  `"AUPRC 0.556 vs 0.519"` where the second value is implicitly
+  a contrasting detector's binding (cross-detector inference
+  remains out of scope per ADR 0005 A4).
+
+- **T4: Sentence-boundary detector-pair reject.** When pairing a
+  detector mention with a value, if a sentence terminator (`.`,
+  `!`, `?`, `\n\n`) lies between them, the pair is rejected.
+  Sentence detection uses paragraph-aware abbreviation guarding
+  (`vs.`, `e.g.`, `i.e.`, `c.f.`, `etc.`, `cf.`, `fig.`,
+  `eq.`, `pp.`, `viz.`, `ca.` excluded; decimal numbers and
+  letter-dot-letter patterns also guarded). Single `\n` is a
+  soft break (markdown line-wrap, NOT a sentence boundary);
+  `\n\n` is hard.
+
+### Internal changes (no public API impact)
+
+- `_nearest_canonical_key()` now returns `(key, position)`
+  instead of just `key`. The position is needed for T4's
+  sentence-boundary check. The slice-pairing call site unpacks
+  and discards the position. Private helper; no consumer impact.
+- New private helpers: `_is_sentence_terminator_dot`,
+  `_sentence_boundary_positions`, `_sentence_id_of`,
+  `_crosses_sentence_boundary`, `_is_signed_value`,
+  `_has_keyword_in_window`, `_compile_keyword_pattern`. All
+  underscore-prefixed; Tier-3 FREE.
+
+### Dogfood evidence
+
+| Configuration | Warnings on `prompt-injection-detection-submission` HEAD | Reduction vs v1.0.5 baseline |
+|---|---|---|
+| v1.0.5 (legacy 2-tuple) | 95 | — |
+| v1.1.0 BindingKey + scope='narrative' (content-type filter only) | 23 | 76% |
+| **v1.2.0 + context filters (this release)** | **7** | **93%** |
+
+The 7 v1.2.0 residuals are all cross-detector list constructions
+(e.g., `"0.293 versus 0.364 for the frozen probe and 0.291 for
+TF-IDF + LR"` where the validator can't infer that 0.361 / 0.291
+belong to ProtectAI-v1 and TF-IDF respectively because they're
+introduced by `"and"` / `"for"` without an immediately-preceding
+detector mention). These require true list-grammar parsing
+(rejected for v1.x in ADR 0005 A4) and are tracked for v1.3.0+
+with their own ADR design review.
+
+### Consumer adoption path
+
+`prompt-injection-detection-submission` and other consumers using
+`scope="narrative"` get the v1.2.0 filters automatically with no
+code change. Consumers on `scope="all"` (default) continue with
+v1.1.0 behavior. Recommended consumer migration:
+
+1. Re-pin `eval-toolkit>=1.2.0,<2` (additive; no consumer code
+   change required).
+2. HARD-gate promotion is now credible: 7 residual warnings is
+   below the actionable threshold; consumer can promote
+   `audit_value_bindings` from SOFT to HARD bundled with
+   `audit_citation_alignment` per the v1.3.8 plan.
+
+### Tests
+
+36 in `tests/test_audit_value_bindings.py` (28 from v1.1.0 + 8
+new for T1–T4 + sentence-boundary helper unit test). All pass.
+Public API snapshot regenerated for `__version__` bump only (no
+signature changes beyond an inspect-formatting normalization on
+the `validate_reader_value_bindings` `bindings` annotation; same
+type semantically).
+
+### Out of scope (deferred)
+
+- **Cross-detector list-grammar parsing** — the 7 residual
+  warnings. Requires lookahead context-aware list parsing
+  (`"X scored Y vs Z for W and V for U"`). Track as a v1.3.0+
+  candidate; needs ADR design before implementation.
+- **Markdown AST parsing** (ADR 0005 A4) — v2.0 territory.
+- **`extra_*_keywords` kwargs** for runtime extension of the
+  hardcoded keyword lists — YAGNI for now (consumer's prose is
+  covered); add in a v1.2.x patch if concrete demand emerges.
+
 ## [1.1.0] — 2026-05-26 — `audit_value_bindings` slice-aware matching via `BindingKey` (closes #80)
 
 Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).

{eval_toolkit-1.1.0 → eval_toolkit-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.1.0
+Version: 1.3.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/

{eval_toolkit-1.1.0 → eval_toolkit-1.3.0}/docs/source/adr/README.md

@@ -76,3 +76,6 @@ What would have to change for this decision to be reopened?
 | [0001](0001-flat-module-layout.md) | Flat single-file modules through v1.x | Accepted | 2026-05-21 |
 | [0002](0002-scorecard-as-primary-metric-surface.md) | `scorecard()` as the primary v1.0 metric surface | Accepted | 2026-05-21 |
 | [0003](0003-stability-contract-and-gate3-methodology.md) | v1.0 stability contract + Gate 3 methodology | Accepted | 2026-05-21 |
+| [0004](0004-naming-conventions.md) | Naming conventions for modules, classes, and parameters | Accepted | 2026-05-23 |
+| [0005](0005-structured-keys-for-audit-validators.md) | Structured keys over positional tuples for canonical-identity types in audit validators | Accepted | 2026-05-26 |
+| [0006](0006-pairing-rules-for-cross-detector-list-grammar.md) | Pairing rules for cross-detector list-grammar in audit validators | Accepted | 2026-05-26 |

{eval_toolkit-1.1.0 → eval_toolkit-1.3.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.1.0"
+__version__ = "1.3.0"