eval-toolkit 1.3.0__tar.gz → 1.5.0__tar.gz

{eval_toolkit-1.3.0 → eval_toolkit-1.5.0}/CHANGELOG.md

@@ -5,6 +5,145 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5.0] — 2026-05-29 — Tier-2 `eda` layer (#83) + schema-aware `HFDatasetsLoader` (#85)
+
+Tier-2 / `loaders` ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md) — backward-compatible.
+
+- **`eda` Job-1 integrity gate (#83):** `audit_dataset` / `DataAudit` / `SplitSummary` + the
+  `class_balance` / `no_cross_split_leakage` / `context_window_fit` gates + the §B2 obfuscation
+  prevalence module.
+- **schema-aware `HFDatasetsLoader` (#85):** load real-world dataset schemas without column
+  guessing — `feature_cols` + `feature_join` (join multiple columns into one feature; NaN-safe),
+  `label_map` (remap raw labels → int; fail-fast `ValueError` lists unmapped values), `revision`
+  (pin the HF dataset SHA). All new params default to the prior behavior; a missing feature/label
+  column raises `KeyError` listing the observed columns.
+
+## [1.4.0] — 2026-05-26 — `audit_citation_alignment` Layer 2 + Layer 3 (closes #82); shared `_narrative` helpers (ADR 0007)
+
+Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).
+Closes [#82](https://github.com/brandon-behring/eval-toolkit/issues/82)
+— consumer-feedback follow-on after v1.3.0 closed
+`audit_value_bindings`. The consumer's 188 residual warnings on
+`audit_citation_alignment` were the same architectural-class gap
+(missing Layer 2 + Layer 3 context-awareness) that
+`audit_value_bindings` worked through over v1.1.0 → v1.3.0.
+
+Introduces [ADR 0007](docs/source/adr/0007-three-layer-architecture-for-audit-validators.md)
+codifying the **three-layer correctness model** (identity + scope +
+pairing) as the canonical architecture for ALL `audit_*` validators
+in the family. ADR 0005 + ADR 0006 were originally validator-
+specific; ADR 0007 generalizes.
+
+### Added — `audit_citation_alignment` Layer 2 + Layer 3 (closes #82)
+
+- **`scope: Literal["all", "narrative"] = "all"`** kwarg on
+  `validate_citations(...)`. Default `"all"` preserves v1.0.1 /
+  v1.3.x behavior exactly (Tier-1 ADDITIVE; byte-identical legacy
+  semantics).
+- **Pattern β (Layer 2)** — under `scope="narrative"`, citations
+  inside markdown table rows, bracketed expressions, and fenced
+  code blocks are excluded. Mirrors `audit_value_bindings`'s
+  Layer 2 from v1.1.0. Closes ~67 of the consumer's residual
+  warnings (SPEC_SHEET.md table rows).
+- **Pattern γ (Layer 3)** — the category-keyword extraction window
+  for a citation is bounded by the SENTENCE containing the
+  citation, not by a ±N-line window. Uses
+  `_sentence_boundary_positions` (paragraph-aware, abbreviation-
+  guarded) from `_narrative`. Catches the consumer's dense
+  multi-clause sentences where keywords from prior clauses pull
+  through.
+- **Pattern α (Layer 3)** — when MULTIPLE ADR citations appear in
+  the same sentence (e.g.,
+  `"per ADR-025 + ADR-021 + ADR-034 + ADR-045"`), the validator
+  switches from first-match-wins category check to multi-category
+  set membership. Each ADR's actual category is accepted if it's
+  in the SET of categories matched by the sentence's keywords —
+  not just the dominant first-match. Catches the dense multi-ADR
+  list pattern where each ADR addresses a different topic.
+
+### Refactor — Shared `_narrative.py` helpers
+
+Per ADR 0007, narrative-prose helpers are extracted to private flat
+module `src/eval_toolkit/_narrative.py` (consistent with ADR 0001's
+`_rng.py` / `_parallel.py` / `_sweep.py` precedent — flat-module
+compliant, private/underscore-prefixed). Both validators import:
+
+- Keyword frozensets: `_DELTA_KEYWORDS`, `_FLOOR_KEYWORDS`,
+  `_GROUP_SUBJECT_KEYWORDS`, `_ABBREV_BEFORE_DOT`.
+- Compiled patterns: `_DELTA_PATTERN`, `_FLOOR_PATTERN`,
+  `_GROUP_SUBJECT_PATTERN`.
+- Helpers: `_build_exclusion_ranges`, `_is_excluded`,
+  `_is_sentence_terminator_dot`, `_sentence_boundary_positions`,
+  `_sentence_id_of`, `_crosses_sentence_boundary`,
+  `_is_signed_value`, `_has_keyword_in_window`,
+  `_compile_keyword_pattern`.
+
+`audit_value_bindings.py` updated to import these from `_narrative`
+instead of defining inline. **Signature-preserving refactor**: all
+43 existing `audit_value_bindings` tests pass UNCHANGED. The
+private helpers are non-public, so no Tier-1 STRICT impact.
+
+### Dogfood result
+
+| Configuration | Warnings on `prompt-injection-detection-submission` HEAD | Reduction |
+|---|---|---|
+| v1.3.0 (`audit_citation_alignment` with scope='all') | 188 | — (baseline) |
+| **v1.4.0 (`scope='narrative'`)** | **37** | **80%** |
+
+Verified locally via `.scratch/dogfood_v1_4_0_citation.py`
+(monkey-patched consumer call with `scope="narrative"`).
+
+The residual 37 are a mix of:
+- **Real misalignments** consumer should triage (e.g., `ADR-025`
+  cited for a threshold claim when ADR-025 is the cost ADR — could
+  be a wrong-ADR bug or a multi-topic ADR not captured by the
+  consumer's category-keyword map).
+- **Single-topic sentences** where the first-match category
+  inferred from the sentence genuinely differs from the ADR's
+  actual category. The multi-topic Pattern α fallback only fires
+  when ≥2 categories match; single-topic prose stays on the
+  legacy first-match check.
+- **Edge cases** requiring parser-level understanding of how an
+  ADR's scope intersects with a multi-clause sentence's topics.
+
+The original #82 acceptance criterion was ≤20 warnings (the
+filer's estimate of "genuinely ambiguous citations"). v1.4.0
+hits 37 — above the target but a 5× reduction overall. The
+remaining gap requires either (a) consumer-side expansion of
+`CATEGORY_KEYWORDS` to capture multi-topic ADRs, (b) consumer
+prose adjustments for the real misalignments, or (c) future
+v1.4.x refinements to the validator's heuristic. Consumer
+HARD-gate promotion remains a judgment call — the residual 37
+includes some real misalignments worth fixing.
+
+### Consumer adoption path
+
+Consumer (`prompt-injection-detection-submission`):
+1. Re-pin `eval-toolkit>=1.4.0,<2`.
+2. Add `scope="narrative"` to their `validate_citations(...)` call
+   in `scripts/audit_citation_alignment.py`.
+3. **Bundled HARD-gate promotion** of BOTH `audit_value_bindings`
+   AND `audit_citation_alignment` now credible per the v1.3.8
+   plan. Promotes from SOFT to HARD in their next v1.3.X release.
+
+### Tests
+
+61 across the audit-validator suite (43 audit_value_bindings + 18
+audit_citation_alignment; 6 new for v1.4.0 — Pattern α / β / γ /
+scope='all' backward-compat / shared-helpers / combined dogfood).
+All pass. Public API snapshot regenerated for `__version__` bump
++ `validate_citations` signature with new `scope` kwarg.
+
+### Out of scope (deferred)
+
+- **`audit_sister_doc_concept_drift` Layer 2 / Layer 3** — embedding-
+  based validator (v1.0.4); different false-positive surface. Add
+  layers only if consumer demand emerges.
+- **Public helper promotion** (`eval_toolkit.audit_narrative`) —
+  YAGNI per ADR 0007 §A2.
+- **Configurable category-keyword-window extension kwargs** —
+  YAGNI; add in v1.4.x patch if demand emerges.
+
 ## [1.3.0] — 2026-05-26 — `audit_value_bindings` cross-detector list-grammar pairing rules (closes #81)
 
 Tier-1 ADDITIVE per [ADR 0003](docs/source/adr/0003-stability-contract-and-gate3-methodology.md).

{eval_toolkit-1.3.0 → eval_toolkit-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eval-toolkit
-Version: 1.3.0
+Version: 1.5.0
 Summary: Reusable evaluation contracts for binary classification: metrics, bootstrap CIs, calibration, artifacts, and evidence gates.
 Project-URL: Homepage, https://github.com/brandon-behring/eval-toolkit
 Project-URL: Documentation, https://brandon-behring.github.io/eval-toolkit/
@@ -60,6 +60,9 @@ Requires-Dist: sphinx-autodoc-typehints>=2.0; extra == 'docs'
 Requires-Dist: sphinx-copybutton>=0.5; extra == 'docs'
 Requires-Dist: sphinx-design>=0.6; extra == 'docs'
 Requires-Dist: sphinx>=7.3; extra == 'docs'
+Provides-Extra: eda
+Requires-Dist: matplotlib>=3.8; extra == 'eda'
+Requires-Dist: pandas>=2.0; extra == 'eda'
 Provides-Extra: embeddings
 Requires-Dist: sentence-transformers>=3.0; extra == 'embeddings'
 Provides-Extra: losses

{eval_toolkit-1.3.0 → eval_toolkit-1.5.0}/docs/source/adr/README.md

@@ -79,3 +79,4 @@ What would have to change for this decision to be reopened?
 | [0004](0004-naming-conventions.md) | Naming conventions for modules, classes, and parameters | Accepted | 2026-05-23 |
 | [0005](0005-structured-keys-for-audit-validators.md) | Structured keys over positional tuples for canonical-identity types in audit validators | Accepted | 2026-05-26 |
 | [0006](0006-pairing-rules-for-cross-detector-list-grammar.md) | Pairing rules for cross-detector list-grammar in audit validators | Accepted | 2026-05-26 |
+| [0007](0007-three-layer-architecture-for-audit-validators.md) | Three-layer architecture for audit validators (family-wide) | Accepted | 2026-05-26 |

{eval_toolkit-1.3.0 → eval_toolkit-1.5.0}/pyproject.toml

@@ -74,6 +74,14 @@ probes = ["torch>=2.0", "transformers>=4.40"]
 # (granular extras — losses callers should not have to install the larger
 # transformers stack). Shares the torch version pin with [probes].
 losses = ["torch>=2.0"]
+# v1.5.0 (feat/eda-data-audit): eval_toolkit.eda Job-1 integrity-gate layer.
+# Tier-2 surface (ADR 0003) — torch-free by design. pandas powers the
+# DataFrameLoader reuse path; matplotlib is reserved for the EDA layer's
+# future profiling plots. Intentionally NO sentence-transformers / torch:
+# the near-dup / cross-split checks use the lexical TfidfCosineStrategy and
+# token-length quantiles take a caller-supplied tokenizer (no transformers
+# import in this module). NOT folded into [all] / [dev] — opt-in only.
+eda = ["pandas>=2.0", "matplotlib>=3.8"]
 # NO-OP extra kept for backward compatibility (R3 at v0.49.0).
 #
 # jsonschema>=4.21 moved to base deps at v0.16.0; this extra has been a

eval_toolkit-1.5.0/src/eval_toolkit/_narrative.py

@@ -0,0 +1,425 @@
+"""Shared narrative-prose helpers for the `audit_*` validator family.
+
+This private flat module hosts the Layer 2 (scope) + Layer 3 (pairing-
+rule) building blocks that emerged from the v1.1.0 → v1.3.0 cycle of
+``audit_value_bindings`` and are reused by ``audit_citation_alignment``
+at v1.4.0+. Per ADR 0007, the three-layer correctness model (identity
++ scope + pairing) applies family-wide; this module is the canonical
+home for the prose-pattern primitives that the scope + pairing layers
+build on.
+
+Design notes:
+
+- **Private flat module** (underscore-prefixed name): matches ADR 0001's
+  `_rng.py` / `_parallel.py` / `_sweep.py` precedent. Not in the
+  package's public ``_EXPORTS`` resolver; consumers import via
+  ``eval_toolkit.audit_*`` modules, which in turn import from here.
+- **Helpers preserve their exact signatures from audit_value_bindings.py**
+  (v1.1.0–v1.3.0 vintage) — extraction is a signature-preserving
+  refactor. All 43 existing audit_value_bindings tests continue to
+  pass unchanged.
+- **Keyword frozensets are audit_value_bindings-specific** (delta /
+  floor / group-subject keywords are about value-binding prose, not
+  citation prose). Other validators that need similar lists define
+  their own constants. The SHARED parts are the regex-compilation
+  utility and the structural helpers (exclusion ranges, sentence
+  boundaries) that are validator-agnostic.
+
+Cross-references:
+- ADR 0005 — identity layer (BindingKey)
+- ADR 0006 — pairing layer (audit_value_bindings-specific)
+- ADR 0007 — three-layer architecture, family-wide
+"""
+
+from __future__ import annotations
+
+import bisect
+import re
+from collections.abc import Sequence
+
+# ---------------------------------------------------------------------------
+# Module-level keyword sets and compiled patterns.
+# Specific to audit_value_bindings' Layer 2 filters (T1/T2/C); kept here
+# so the validator can `from eval_toolkit._narrative import ...` rather
+# than maintaining two copies. Other audit_* validators define their
+# own keyword sets analogously.
+# ---------------------------------------------------------------------------
+
+
+# _DELTA_KEYWORDS: case-insensitive whole-token markers indicating a
+# value is a paired-delta or comparative magnitude, not a binding claim.
+# T1 filter suppresses candidate values when any of these appears within
+# ±30 chars of the value position (under scope="narrative").
+_DELTA_KEYWORDS: frozenset[str] = frozenset(
+    {
+        # Unambiguous delta nouns/verbs (consumer prose patterns):
+        "delta",
+        "drop",
+        "drops",
+        "lift",
+        "lifts",
+        "gap",
+        "margin",
+        # Comparison verbs that signal "this is a relative magnitude":
+        "regresses",
+        "improves",
+        "beats",
+        "exceeds",
+        "trails",
+        "underperforms",
+        # "vs"/"versus" intentionally INCLUDED — they're the canonical
+        # delta separator in consumer prose ("AUPRC 0.556 vs 0.519").
+        # The before-only window keeps these tight: "X vs Y" fires on
+        # Y (preceded by "vs"), not X. T3 also catches the same-sentence
+        # duplicate-binding flag separately.
+        "vs",
+        "versus",
+        # Comparison directions — kept under before-only window so
+        # "drops -0.071 below" suppresses -0.071 (sign also catches),
+        # but "0.515 (delta -0.132)" doesn't suppress 0.515 ("delta"
+        # is AFTER 0.515).
+        # Excluded: "against", "above", "ahead", "behind" — too
+        # ambiguous; common comparison prepositions that appear in
+        # legitimate binding claims.
+        "below",
+    }
+)
+
+# _FLOOR_KEYWORDS: markers indicating a value is a random-baseline or
+# floor reference, not a detector binding. T2 filter suppresses
+# candidate values when any of these appears within −50 / +5 chars
+# (asymmetric: floor mentions canonically precede the value, e.g.,
+# "random AUPRC is 0.374").
+#
+# Intentionally narrow: "baseline", "prior", "majority" are EXCLUDED
+# because they have legitimate non-floor senses ("TF-IDF baseline",
+# "prior work", "majority of detectors"). The consumer's prose
+# patterns with these words ("below the prevalence baseline of 0.374")
+# are caught by T1 via "below"/"above" instead — the comparative
+# preposition is the reliable signal, not the noun.
+_FLOOR_KEYWORDS: frozenset[str] = frozenset(
+    {
+        "random",
+        "floor",
+        "chance",
+        "trivial",
+    }
+)
+
+# _ABBREV_BEFORE_DOT: tokens that should NOT trigger a sentence
+# boundary when followed by `.`. The multi-letter pattern (e.g., i.e.,
+# c.f.) is handled separately via letter-dot-letter detection.
+_ABBREV_BEFORE_DOT: frozenset[str] = frozenset(
+    {
+        "vs",
+        "etc",
+        "cf",
+        "fig",
+        "eq",
+        "pp",
+        "viz",
+        "ca",
+    }
+)
+
+
+def _compile_keyword_pattern(keywords: frozenset[str]) -> re.Pattern[str]:
+    """Compile case-insensitive word-boundary OR regex matching any keyword."""
+    parts = sorted(re.escape(kw) for kw in keywords)
+    return re.compile(r"\b(?:" + "|".join(parts) + r")\b", re.IGNORECASE)
+
+
+_DELTA_PATTERN: re.Pattern[str] = _compile_keyword_pattern(_DELTA_KEYWORDS)
+_FLOOR_PATTERN: re.Pattern[str] = _compile_keyword_pattern(_FLOOR_KEYWORDS)
+
+
+# Group-subject adjectives that introduce a multi-detector statement.
+# When prose says "for the trained detectors", the following value
+# refers to a GROUP (LoRA + TF-IDF + ... whatever bindings exist),
+# not a single canonical detector. The validator can't infer which
+# specific detectors own the group value with positional heuristics,
+# so v1.3.0 suppresses the candidate rather than attempting multi-
+# detector inference (a v1.4.0+ candidate per ADR 0006).
+_GROUP_SUBJECT_KEYWORDS: frozenset[str] = frozenset(
+    {
+        "trained",
+        "frozen",
+        "baseline",
+        "all",
+        "both",
+        "other",
+    }
+)
+
+# Module-level: detector-independent group-subject regex. Matches
+# "for the {trained|frozen|...} detectors" (with optional "the"; both
+# singular and plural "detector"/"detectors" tolerated).
+_GROUP_SUBJECT_PATTERN: re.Pattern[str] = re.compile(
+    r"\bfor\s+(?:the\s+)?(?:"
+    + "|".join(sorted(re.escape(kw) for kw in _GROUP_SUBJECT_KEYWORDS))
+    + r")\s+detectors?\b",
+    re.IGNORECASE,
+)
+
+
+# ---------------------------------------------------------------------------
+# Layer 2: content-type filtering helpers (`scope="narrative"`).
+# Used by audit_value_bindings (v1.1.0+) and audit_citation_alignment
+# (v1.4.0+) to exclude markdown table rows, bracketed expressions, and
+# fenced code blocks from candidate-value / candidate-citation matching.
+# ---------------------------------------------------------------------------
+
+
+def _build_exclusion_ranges(
+    text: str,
+    line_starts: Sequence[int],
+) -> list[tuple[int, int]]:
+    """Compute sorted character ranges that ``scope="narrative"`` excludes.
+
+    Excluded content types (per the lint-scope design discussion in
+    ADR 0005):
+
+    - **Markdown table rows**: lines starting with optional whitespace
+      then ``|``. Tables are structured data audited via different
+      mechanisms (e.g., direct results-table verification), not via
+      narrative-prose binding-claim checks. Values in cells are
+      typically inline statistics (multiple metrics per row), and the
+      validator's positional heuristics can't disambiguate them.
+    - **Bracketed expressions** ``[...]``: confidence intervals,
+      reference markers, ranges. The numeric content inside brackets
+      is not a point-estimate claim; the validator should not flag it.
+    - **Fenced code blocks**: triple-backtick blocks contain code or
+      literal data, not narrative claims.
+
+    Returns a sorted list of ``(start, end)`` character intervals
+    (half-open) for use with :func:`_is_excluded`.
+    """
+    excluded: list[tuple[int, int]] = []
+    in_code_block = False
+    code_block_start = 0
+    n_lines = len(line_starts)
+    for line_idx in range(n_lines):
+        line_start = line_starts[line_idx]
+        line_end = line_starts[line_idx + 1] if line_idx + 1 < n_lines else len(text)
+        line = text[line_start:line_end]
+
+        # Triple-backtick code-fence toggle. The fence line itself is
+        # also part of the excluded range (so values aren't matched
+        # from within the fence marker, though that's unlikely).
+        stripped = line.lstrip()
+        if stripped.startswith("```"):
+            if not in_code_block:
+                in_code_block = True
+                code_block_start = line_start
+            else:
+                in_code_block = False
+                excluded.append((code_block_start, line_end))
+            continue
+        if in_code_block:
+            # Lines inside a code block are folded into the outer
+            # range emitted at the closing fence; no per-line emission.
+            continue
+
+        # Markdown table row.
+        if stripped.startswith("|"):
+            excluded.append((line_start, line_end))
+            continue
+
+        # Bracketed expressions on this line. Multiple `[...]` allowed.
+        # Nested brackets are rare in measurement prose; first close
+        # wins.
+        i = 0
+        while True:
+            open_idx = line.find("[", i)
+            if open_idx == -1:
+                break
+            close_idx = line.find("]", open_idx + 1)
+            if close_idx == -1:
+                break
+            excluded.append((line_start + open_idx, line_start + close_idx + 1))
+            i = close_idx + 1
+
+    # Handle unterminated code block (defensive: treat rest of file as
+    # excluded). Sort by start position.
+    if in_code_block:
+        excluded.append((code_block_start, len(text)))
+    excluded.sort()
+    return excluded
+
+
+def _is_excluded(pos: int, excluded: Sequence[tuple[int, int]]) -> bool:
+    """Return True if ``pos`` falls inside any excluded range.
+
+    Uses binary search on the sorted ranges. Half-open semantics: a
+    range ``(start, end)`` excludes positions ``start <= pos < end``.
+    """
+    if not excluded:
+        return False
+    # Find rightmost range with start <= pos.
+    lo, hi = 0, len(excluded)
+    while lo < hi:
+        mid = (lo + hi) // 2
+        if excluded[mid][0] <= pos:
+            lo = mid + 1
+        else:
+            hi = mid
+    if lo == 0:
+        return False
+    start, end = excluded[lo - 1]
+    return start <= pos < end
+
+
+# ---------------------------------------------------------------------------
+# Sentence-boundary detection (paragraph-aware, abbreviation-guarded).
+# Used by v1.2.0 T3/T4 in audit_value_bindings and the v1.4.0 Layer 3
+# rule γ in audit_citation_alignment (sentence-boundary respect for
+# category-keyword window).
+# ---------------------------------------------------------------------------
+
+
+def _is_sentence_terminator_dot(text: str, dot_pos: int) -> bool:
+    """Return True if the dot at ``dot_pos`` terminates a sentence.
+
+    False positives the abbreviation guard catches:
+
+    - Decimal numbers (digit-dot-digit): ``0.5``, ``§5.2``.
+    - Letter-dot-letter-dot patterns: ``e.g.``, ``i.e.``, ``c.f.``.
+    - Single-token abbreviations preceding the dot (whitespace- /
+      punctuation-separated): ``vs.``, ``etc.``, ``cf.``, ``fig.``,
+      ``eq.``, ``pp.``, ``viz.``, ``ca.``. See ``_ABBREV_BEFORE_DOT``.
+    """
+    n = len(text)
+    prev_char = text[dot_pos - 1] if dot_pos > 0 else ""
+    next_char = text[dot_pos + 1] if dot_pos + 1 < n else ""
+    # Decimal: digit-dot-digit.
+    if prev_char.isdigit() and next_char.isdigit():
+        return False
+    # Letter-dot-letter-dot pattern, dot is the SECOND dot in "x.y."
+    if (
+        dot_pos >= 3
+        and prev_char.isalpha()
+        and text[dot_pos - 2] == "."
+        and text[dot_pos - 3].isalpha()
+    ):
+        return False
+    # Letter-dot-letter-dot pattern, dot is the FIRST dot in "x.y."
+    if dot_pos + 2 < n and next_char.isalpha() and text[dot_pos + 2] == ".":
+        return False
+    # Single-token abbreviation preceding the dot.
+    j = dot_pos - 1
+    while j >= 0 and text[j].isalpha():
+        j -= 1
+    word = text[j + 1 : dot_pos].lower()
+    return word not in _ABBREV_BEFORE_DOT
+
+
+def _sentence_boundary_positions(text: str) -> list[int]:
+    """Return sorted character positions where each sentence STARTS.
+
+    Hard breaks (sentence terminators):
+
+    - ``!`` and ``?`` always terminate.
+    - ``.`` terminates unless the abbreviation guard
+      (:func:`_is_sentence_terminator_dot`) returns False.
+    - ``\\n\\n`` (paragraph break) terminates.
+
+    Soft breaks (NOT sentence boundaries):
+
+    - Single ``\\n`` (markdown line-wrap mid-sentence).
+    - ``;`` (semicolons in dense list constructions).
+    - ``:`` (colons preceding list items or definitions).
+
+    The first sentence starts at position 0. Subsequent sentence starts
+    are recorded at the first non-whitespace character after a hard
+    break. Used by audit_value_bindings T3/T4 and
+    audit_citation_alignment Layer 3 rule γ.
+    """
+    positions = [0]
+    n = len(text)
+    i = 0
+    while i < n:
+        ch = text[i]
+        boundary = False
+        skip = 1
+        if ch in "!?" or ch == "." and _is_sentence_terminator_dot(text, i):
+            boundary = True
+        elif ch == "\n" and i + 1 < n and text[i + 1] == "\n":
+            boundary = True
+            skip = 2
+        if boundary:
+            j = i + skip
+            while j < n and text[j].isspace():
+                j += 1
+            if j < n and j > positions[-1]:
+                positions.append(j)
+            i = max(j, i + skip)
+        else:
+            i += 1
+    return positions
+
+
+def _sentence_id_of(pos: int, sentence_positions: Sequence[int]) -> int:
+    """Return the zero-based sentence index containing ``pos``.
+
+    Uses binary search over the sorted ``sentence_positions``. Returns
+    ``0`` for any position before the first sentence start.
+    """
+    if not sentence_positions:
+        return 0
+    idx = bisect.bisect_right(sentence_positions, pos) - 1
+    return max(0, idx)
+
+
+def _crosses_sentence_boundary(pos_a: int, pos_b: int, sentence_positions: Sequence[int]) -> bool:
+    """Return True if a sentence boundary lies strictly between ``pos_a`` and ``pos_b``.
+
+    Sentence-boundary positions are derived from
+    :func:`_sentence_boundary_positions`. Used by audit_value_bindings
+    T4 (reject (detector, value) pairs across a sentence boundary)
+    and audit_citation_alignment Layer 3 rule γ (the category-keyword
+    extraction window for an ADR citation must not cross a sentence
+    boundary).
+    """
+    if not sentence_positions:
+        return False
+    lo = min(pos_a, pos_b)
+    hi = max(pos_a, pos_b)
+    idx = bisect.bisect_right(sentence_positions, lo)
+    return idx < len(sentence_positions) and sentence_positions[idx] <= hi
+
+
+# ---------------------------------------------------------------------------
+# Value-context helpers (used by audit_value_bindings T1/T2; kept here
+# for any future audit_* validator that wants the same primitives).
+# ---------------------------------------------------------------------------
+
+
+def _is_signed_value(text: str, val_start: int) -> bool:
+    """True if the value at ``val_start`` is immediately preceded by ``+`` or ``-``.
+
+    The sign marker indicates a paired-delta or comparative magnitude
+    (e.g., ``-0.071`` AUPRC delta), not a binding claim. T1 filter
+    skips these under ``scope="narrative"``.
+    """
+    return val_start > 0 and text[val_start - 1] in "+-"
+
+
+def _has_keyword_in_window(
+    text: str,
+    val_start: int,
+    pattern: re.Pattern[str],
+    before_chars: int,
+    after_chars: int,
+) -> bool:
+    """True if ``pattern`` matches anywhere in the character window around ``val_start``.
+
+    Used by audit_value_bindings T1 (delta keywords) and T2 (floor
+    keywords) to detect context cues near a candidate value.
+    ``before_chars`` and ``after_chars`` control the asymmetric
+    window — floor mentions typically PRECEDE the value (e.g.,
+    "random AUPRC is 0.374"), while delta mentions can be on either
+    side.
+    """
+    start = max(0, val_start - before_chars)
+    end = min(len(text), val_start + after_chars)
+    return bool(pattern.search(text, start, end))

{eval_toolkit-1.3.0 → eval_toolkit-1.5.0}/src/eval_toolkit/_version.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "1.3.0"
+__version__ = "1.5.0"