euredact 0.3.2__tar.gz → 0.3.3__tar.gz

{euredact-0.3.2 → euredact-0.3.3}/CHANGELOG.md

@@ -1,5 +1,100 @@
 # Changelog
 
+## 0.3.3 (2026-07-29)
+
+### Fixed
+
+- **A shorter validated match could re-cut a longer one and leak the
+  remainder.** Under `countries` `["NL"]` the Dutch national-ID pattern
+  validates on `194.232.104`, outranked the IP address `194.232.104.77`, and
+  left `.77` in the output. The TypeScript SDK did the same with no country
+  declared at all.
+
+  Span length now outranks every other signal, including validation. Priority
+  still decides between candidates of *equal* length, which is where it was
+  always meant to apply — a valid IBAN beats a coincidental match on the same
+  characters. Masking more than necessary is recoverable; masking less is not.
+
+- **`countries` could change which spans were found, not just how they were
+  labelled.** Promotion depends on whether the document corroborates a
+  country, so the declared country decided which of two overlapping candidates
+  won. Two cases were found, the second only after fixing the first.
+
+  Ordering between different spans is now country-blind by construction —
+  length, then the span's structural tier, then leftmost — and `(length, start)`
+  identifies a span uniquely, so the country-aware signals can only choose the
+  **label within a span**, never its extent. The invariant holds by design
+  rather than by test.
+
+- **A bare string for `countries` silently discarded every detection.**
+  `countries="NL"` is iterable, so it was walked into the codes `"N"` and `"L"`.
+  Neither resolves, so nothing was declared — and every detection carrying a
+  country came back flagged out of scope, while the redacted text still looked
+  correct. The README tells callers to filter on exactly that field, so a
+  documented pipeline kept none of them. It failed toward "no PII here", from a
+  one-character typo.
+
+  Both SDKs now raise `TypeError` naming the argument and showing the fix, on
+  every entry point. A wrong *code* still only warns: raising there would invite
+  callers to wrap redaction in try/except and skip it. A wrong *type* is a
+  programming error with no correct interpretation to fall back on.
+
+- **The generic phone pattern claimed fragments of rejected identifiers.**
+  `Rijksregisternummer: 85.03.19-284.73` was reported as a `PHONE`. The Belgian
+  national-number pattern matched the whole value and failed its checksum; the
+  separator-tolerant phone pattern then took characters 3-14 of it.
+
+  A candidate covering only *part* of a rejected identifier is a fragment of
+  that identifier, not a separate entity, and is now dropped. An equal span is
+  left alone: that is two schemes competing for the whole value, and demoting
+  those is what previously handed a Swedish phone number to a Danish CPR.
+
+  Fragment detection ignores the declared country by design. It removes
+  candidates, so making it country-aware would let `countries` change which
+  spans are found — the invariant in `test_invariant_generation.py`.
+
+  No measurable cost: recall, precision and type-correct rate are unchanged on
+  all 152,300 corpus documents.
+
+- **Identifiers glued to a non-ASCII letter were missed.** Python's `\b` is
+  Unicode-aware, so it saw no boundary between a Cyrillic or accented letter
+  and a digit: `ЕГН7523169263`, `PESELŁ44051401359`, `čísloř7401011233`,
+  `Steuerß12345678911` and `kodasž38605181232` were all detected by the
+  TypeScript SDK and silently missed by this one.
+
+  `\b` is now compiled to the *union* of the Unicode and ASCII readings.
+  Swapping to ASCII-only — the obvious fix — trades one silent miss for
+  another: it drops Greek and Cyrillic e-mail local parts, which are
+  deliberately supported. `re.ASCII` is unusable for the same reason, since it
+  would also narrow `\w`.
+
+  Costs about 20% latency on short documents from the added lookarounds.
+  Accuracy on the corpus is unchanged.
+
+### Added
+
+- **`make check` and `make verify`.** One entry point for every check.
+  `make check` is what CI runs; `make verify` adds the corpus checks CI cannot
+  run, because the corpus lives outside the repository.
+
+  - `tests/sweep.py` — structural properties over ~187,000 documents: offsets
+    index the text, spans do not overlap, detection is deterministic, the cache
+    is transparent, and no country argument changes which spans are found. Both
+    ranking bugs above were found here; the twenty-document version that runs in
+    CI showed nothing.
+  - `scripts/parity.py` — do both SDKs mask the same *characters*, over whole
+    corpora. Conformance vectors pin named cases; this is the broad counterpart,
+    and it is how a 19,014-character gap was found that no vector showed.
+
+
+### Known issues
+
+- A checksum-invalid identifier occupying a span no other detector claims can
+  still be reported under the wrong type — `7401011234` with `countries` `["CZ"]`
+  is masked as a Romanian `PHONE` at `countryConfidence` 0. The span *is*
+  redacted; only the label is wrong. The fragment rule above does not apply
+  because the spans are equal rather than nested.
+
 ## 0.3.2 (2026-07-29)
 
 ### Changed

{euredact-0.3.2 → euredact-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: euredact
-Version: 0.3.2
+Version: 0.3.3
 Summary: European PII redaction SDK — rule engine for 31 countries
 Project-URL: Documentation, https://github.com/euredact/euredact#readme
 Project-URL: Repository, https://github.com/euredact/euredact

{euredact-0.3.2 → euredact-0.3.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "euredact"
-version = "0.3.2"
+version = "0.3.3"
 description = "European PII redaction SDK — rule engine for 31 countries"
 requires-python = ">=3.10"
 license = "Apache-2.0"

{euredact-0.3.2 → euredact-0.3.3}/src/euredact/__init__.py

@@ -22,7 +22,7 @@ from __future__ import annotations
 
 from typing import Iterator
 
-__version__ = "0.3.2"
+__version__ = "0.3.3"
 
 from euredact.rules.bic_registry import set_bic_registry
 from euredact.rules.context import DocumentContext

{euredact-0.3.2 → euredact-0.3.3}/src/euredact/rules/engine.py

@@ -39,6 +39,31 @@ def _validate_custom_pattern(pattern: str) -> None:
         )
 
 
+def check_country_arg(value: object, param: str) -> None:
+    """Reject a bare string where a list of country codes is expected.
+
+    ``countries="NL"`` is iterable, so Python walks it into the codes ``"N"``
+    and ``"L"``. Neither resolves, so nothing is declared — and every detection
+    that *does* carry a country is then flagged ``out_of_scope``. A caller
+    following the documented pattern of filtering on that field silently keeps
+    none of them, while ``redacted_text`` still looks perfectly correct. The
+    failure direction is "no PII here", from a one-character typo.
+
+    A wrong *code* is data and only warns (see ``CountryRegistry.warn_unknown``)
+    — raising there would invite callers to wrap redaction in try/except and
+    skip it. A wrong *type* is a programming error, surfaces immediately, and
+    has no correct interpretation to fall back on.
+    """
+    if isinstance(value, (str, bytes)):
+        shown = value.decode(errors="replace") if isinstance(value, bytes) else value
+        raise TypeError(
+            f"{param} must be a list of country codes, not a bare string. "
+            f"Pass {param}=[{shown!r}] rather than {param}={shown!r} — a string "
+            f"is iterated character by character, which silently declares "
+            f"nothing and flags every detection out_of_scope."
+        )
+
+
 class RuleEngine:
     """The core rule-based PII detection engine.
 
@@ -135,6 +160,9 @@ class RuleEngine:
         It is scored alongside this text's own evidence but is **not** returned,
         so a caller accumulating evidence never counts it twice.
         """
+        check_country_arg(country_codes, "countries")
+        check_country_arg(country_hint, "country_hint")
+
         raw_matches = self.generate_candidates(text)
 
         # The declared countries are a scoring signal from here on: they rank
@@ -226,16 +254,50 @@ class RuleEngine:
         # phone number — and demoting on it cost every PHONE candidate in the
         # document its rank, which is how a Dutch mobile came back as a Danish
         # NATIONAL_ID even after the digits were correctly ranked.
-        # Zones are per entity type. A failed checksum means "these digits are
-        # not a valid X" — that is evidence against X, not against the span.
-        # Demoting every type at once meant Sweden's own personnummer checksum
-        # failing on 0708787668 demoted the Swedish *phone* candidate for the
-        # same digits, handing the span to a Danish CPR that happened to
-        # validate. Worth 878 mistyped phone numbers on 30,000 documents.
+        # Two rules, because a failed checksum carries two different messages
+        # depending on how the spans line up.
+        #
+        # Same type, any containment: "these digits are not a valid X" is
+        # evidence against X, not against the span. Demoting *every* type on an
+        # equal span meant Sweden's own personnummer checksum failing on
+        # 0708787668 demoted the Swedish *phone* candidate for the same digits,
+        # handing the span to a Danish CPR that happened to validate — 878
+        # mistyped phone numbers per 30,000 documents.
+        #
+        # Any type, strict subset: a candidate covering only *part* of a
+        # rejected identifier is a fragment of it, not a different entity. The
+        # generic separator-tolerant phone pattern claimed characters 3-14 of
+        # the rejected Belgian national number in
+        # "Rijksregisternummer: 85.03.19-284.73" and reported it as a PHONE.
+        # This is the case the original suppression zones were written for —
+        # licence-plate fragments inside an invalid IBAN — which the per-type
+        # split above had given up along with the over-reach.
+        #
+        # An equal span is two schemes competing for the whole value; a strict
+        # subset is one scheme picking over another's wreckage.
+        corroborated: list[tuple[int, int, object]] = [
+            (start, end, etype)
+            for start, end, cc, etype in failed_spans
+            if not country_scores or cc in ("SHARED", "CUSTOM") or cc in country_scores
+        ]
+        corroborated.sort()
+
+        # Fragment detection deliberately ignores corroboration and uses every
+        # failed span, because it *removes* candidates and must therefore be
+        # country-blind. Gating it on the document's countries made the set of
+        # spans found depend on `countries=`, which is precisely invariant I1 —
+        # caught by tests/test_invariant_generation.py within seconds of being
+        # written. Demotion may be country-aware because it only reorders; a
+        # demoted candidate still claims a span nothing else wants.
+        #
+        # Kept unmerged: merging adjacent spans would invent strict-subset
+        # relationships that neither original span had.
+        all_failed = sorted((start, end) for start, end, _cc, _et in failed_spans)
+        strict_starts = [f[0] for f in all_failed]
+
         by_type: dict[object, list[tuple[int, int]]] = {}
-        for start, end, cc, etype in failed_spans:
-            if not country_scores or cc in ("SHARED", "CUSTOM") or cc in country_scores:
-                by_type.setdefault(etype, []).append((start, end))
+        for start, end, etype in corroborated:
+            by_type.setdefault(etype, []).append((start, end))
 
         # Merge overlapping zones per type and index them for O(log n)
         # containment checks instead of O(n) per match.
@@ -274,7 +336,7 @@ class RuleEngine:
         # (priority, in_scope, start, end, match, prebuilt). in_scope ranks
         # candidates within a priority tier; it never removes any.
         candidates: list[
-            tuple[int, float, int, int, int, RawMatch | None, Detection | None]
+            tuple[int, int, float, int, int, int, RawMatch | None, Detection | None]
         ] = []
         for match, has_valid_validator in validated:
             # A validator-less match inside a failed-validation span used to be
@@ -295,9 +357,31 @@ class RuleEngine:
                     zone_starts, zone_ends = zone
                     idx = bisect.bisect_right(zone_starts, match.start) - 1
                     demoted = idx >= 0 and zone_ends[idx] >= match.end
+            # Strict subset of a rejected identifier, whatever its type: the
+            # candidate covers only part of something already recognised and
+            # rejected, so it is a fragment of that, not a separate entity.
+            #
+            # Dropped rather than demoted. Demotion cannot help here: nothing
+            # else wants a fragment, so a demoted candidate still wins its span
+            # by default — the generic phone pattern kept claiming characters
+            # 3-14 of a rejected Belgian national number. Dropping is safe in a
+            # way that whole-span deletion was not, because the characters stay
+            # covered by whatever claims the enclosing span, or by nothing —
+            # and "nothing" is the right answer for the tail of a rejected
+            # identifier.
+            fragment = False
+            if match.pattern_def.validator is None and not demoted:
+                hi = bisect.bisect_right(strict_starts, match.start)
+                for zs, ze in all_failed[:hi]:
+                    if match.end <= ze and (match.start > zs or match.end < ze):
+                        fragment = True
+                        break
+            if fragment:
+                continue
 
             if demoted:
                 priority = -1
+                blind_priority = -1
             elif has_valid_validator:
                 # A passing checksum normally outranks everything. But a
                 # checksum only says the digits fit *some* national scheme, and
@@ -321,28 +405,32 @@ class RuleEngine:
                     or match.country_code in country_scores
                 )
                 priority = 3 if vouched else 1
+                blind_priority = 3
             elif match.country_code == "CUSTOM":
                 priority = 2
+                blind_priority = 2
             elif match.pattern_def.entity_type == EntityType.POSTAL_CODE:
                 # A bare digit run is the weakest evidence in the engine and
                 # must never re-cut a span a structured detector claims
                 # (PHONE, SSN, NATIONAL_ID, BANK_ACCOUNT, VAT...). Postal
                 # codes take unclaimed spans only, whatever the span lengths.
                 priority = 0
+                blind_priority = 0
             else:
                 priority = 1
+                blind_priority = 1
 
             in_scope = 1 if (declared is None or match.country_code in declared) else 0
             country_score = country_scores.get(match.country_code, 0.0)
             candidates.append(
-                (priority, country_score, in_scope,
+                (blind_priority, priority, country_score, in_scope,
                  match.start, match.end, match, None)
             )
 
         # Structural detectors (JSON field names, CSV headers). These arrive as
         # finished Detections, carry no RawMatch, and are not suppressed.
         for d in detect_structural_dob(text):
-            candidates.append((1, 0.0, 1, d.start, d.end, None, d))
+            candidates.append((1, 1, 0.0, 1, d.start, d.end, None, d))
 
         # Deduplicate: validated > custom > regex-only, then longer wins
         detections = self._deduplicate(text, candidates, declared)
@@ -366,7 +454,7 @@ class RuleEngine:
     def _deduplicate(
         text: str,
         candidates: list[
-            tuple[int, float, int, int, int, RawMatch | None, Detection | None]
+            tuple[int, int, float, int, int, int, RawMatch | None, Detection | None]
         ],
         declared: set[str] | None = None,
     ) -> list[Detection]:
@@ -388,24 +476,51 @@ class RuleEngine:
         if not candidates:
             return []
 
-        # Sort by (priority, span_length) descending. Stable, so ties fall back
-        # to pattern registration order, as before.
+        # Ordering between *different* spans is country-blind; the country
+        # only ever picks the winner among candidates covering the *same* span.
+        #
+        # That split is what makes invariant I1 structural rather than merely
+        # tested. Country-aware promotion had twice decided which span won:
+        # under countries=["NL"] a validated national ID outranked a longer IP
+        # address and exposed ".77", and two equal-length candidates at
+        # different offsets swapped places. Both were found by sweeping the
+        # corpus, not by reasoning, and both are impossible now — the key below
+        # is country-blind up to and including `start`, and (length, start)
+        # already identifies a span uniquely.
+        #
+        # Ordering, in words: longest first (a shorter match winning re-cuts a
+        # longer one and leaks the remainder), then the structural tier, then
+        # leftmost. Only after a span is fixed do the country-aware fields
+        # break the tie, deciding the label rather than the extent.
+        #
+        # The structural tier is a property of the *span*, not of the candidate:
+        # the best tier anything claiming those exact characters can offer. Used
+        # per candidate it would also discriminate *within* a span, where the
+        # country-aware tier is supposed to have the final word — that silently
+        # disabled country inference entirely, handing every ambiguous value to
+        # whichever foreign scheme happened to validate.
+        span_tier: dict[tuple[int, int], int] = {}
+        for c in candidates:
+            key = (c[4], c[5])
+            if c[0] > span_tier.get(key, -2):
+                span_tier[key] = c[0]
+
         sorted_cands = sorted(
             candidates,
-            # Span length outranks scope: preferring the declared country over
-            # the longest match can truncate an entity. Measured: with
-            # countries=["BE"], the Belgian phone pattern claims 11 of the 14
-            # characters of "06 12 34 56 78" and leaves three digits exposed.
-            # Scope only breaks ties between candidates of equal length, so it
-            # can change WHICH country is attributed, never WHAT is masked.
-            key=lambda c: (c[0], c[4] - c[3], c[1], c[2]),
-            reverse=True,
+            key=lambda c: (
+                -(c[5] - c[4]),                # longest first    } country-blind
+                -span_tier[(c[4], c[5])],      # best tier here   } — decides
+                c[4],                          # leftmost         } WHICH span
+                -c[1],                         # country priority } country-aware
+                -c[2],                         # country evidence } — decides the
+                -c[3],                         # declared scope   } LABEL only
+            ),
         )
         result: list[Detection] = []
         occupied: set[int] = set()
         span_verdicts: dict[tuple[object, int, int], bool] = {}
 
-        for _priority, _score, in_scope, start, end, match, prebuilt in sorted_cands:
+        for _bp, _prio, _score, in_scope, start, end, match, prebuilt in sorted_cands:
             if any(p in occupied for p in range(start, end)):
                 continue
 

{euredact-0.3.2 → euredact-0.3.3}/src/euredact/rules/matchers.py

@@ -90,6 +90,68 @@ def _silence_stderr() -> Iterator[None]:
         os.close(saved_fd)
 
 
+# Python's \b is Unicode-aware: a Cyrillic or accented letter counts as a word
+# character, so "\b\d{10}\b" does not match the digits in "ЕГН7523169263". JS's
+# \b is ASCII-only, so Node matched and redacted exactly those cases and Python
+# did not — five of five, across the alphabets this engine targets.
+#
+# Which rule is "right" is arguable: Python's is self-consistent, JS's is
+# script-dependent. But for a redaction tool the argument is settled by the
+# failure direction. Python's principled rule leaves a national ID glued to a
+# non-ASCII letter unredacted; JS's limitation catches it. Over-detection is
+# recoverable, a silent miss is not.
+#
+# So \b is rewritten at compile time rather than in 303 pattern sources.
+# re.ASCII is not usable here: it would also narrow \w, breaking Unicode e-mail
+# local parts, which are deliberately supported.
+#
+# The rewrite is a *union* of both readings, not a swap to the ASCII one.
+# Neither alone is sufficient, and swapping regresses the other direction:
+#
+#   "ЕГН7523169263"           needs the ASCII reading   (Н and 7 are both
+#                             word characters to Python, so no boundary)
+#   "Mail: αλέξης@example.gr" needs the Unicode reading (α is not an ASCII
+#                             word character, so no boundary)
+#
+# Taking either one alone trades one silent miss for another — swapping to
+# ASCII-only was measured to drop the Greek address entirely. The union fires
+# wherever *either* reading sees a boundary, which is the over-detecting
+# direction on both counts and matches what Node already does on all seven
+# cases.
+_ASCII_WORD_BOUNDARY = (
+    r"(?:\b|(?<![0-9A-Za-z_])(?=[0-9A-Za-z_])|(?<=[0-9A-Za-z_])(?![0-9A-Za-z_]))"
+)
+
+
+def _ascii_word_boundaries(pattern: str) -> str:
+    """Rewrite every ``\b`` in *pattern* to the union boundary above.
+
+    Skips escaped backslashes and character classes, where ``\b`` means a
+    backspace rather than a boundary. Neither occurs in the shipped patterns
+    today; the scan is here so that adding one cannot silently corrupt it.
+    """
+    out: list[str] = []
+    i = 0
+    in_class = False
+    while i < len(pattern):
+        ch = pattern[i]
+        if ch == "\\" and i + 1 < len(pattern):
+            nxt = pattern[i + 1]
+            if nxt == "b" and not in_class:
+                out.append(_ASCII_WORD_BOUNDARY)
+            else:
+                out.append(pattern[i:i + 2])
+            i += 2
+            continue
+        if ch == "[":
+            in_class = True
+        elif ch == "]":
+            in_class = False
+        out.append(ch)
+        i += 1
+    return "".join(out)
+
+
 def _max_match_width(pattern: str) -> int:
     """Longest string *pattern* can match, or a huge number if unbounded.
 
@@ -149,14 +211,14 @@ class MultiPatternMatcher:
 
     def add_pattern(self, pattern_def: PatternDef, country_code: str) -> None:
         """Add a single pattern."""
-        compiled = re.compile(pattern_def.pattern, re.UNICODE)
+        compiled = re.compile(_ascii_word_boundaries(pattern_def.pattern), re.UNICODE)
         self._patterns.append((compiled, pattern_def, country_code))
         self._compiled = False
 
     def add_country(self, config: CountryConfig) -> None:
         """Add a country's patterns."""
         for pdef in config.patterns:
-            compiled = re.compile(pdef.pattern, re.UNICODE)
+            compiled = re.compile(_ascii_word_boundaries(pdef.pattern), re.UNICODE)
             self._patterns.append((compiled, pdef, config.code))
         self._compiled = False
 
@@ -176,6 +238,9 @@ class MultiPatternMatcher:
         codes and secrets), and those that can match further than the window
         overlap, which the sliding window could otherwise straddle.
         """
+        # The *original* source goes to RE2, not the rewritten one: RE2's \b is
+        # already ASCII-only, so the semantics match, while the rewrite uses
+        # lookbehind that RE2 cannot compile.
         automaton = _re2.Set.SearchSet()
         self._re2_slot = []
         added = 0

{euredact-0.3.2 → euredact-0.3.3}/src/euredact/sdk.py

@@ -10,7 +10,7 @@ from typing import Iterator
 from euredact.cache import ResultCache
 from euredact.normalizer import map_offset_to_original, normalize
 from euredact.rules.context import DocumentContext
-from euredact.rules.engine import RuleEngine
+from euredact.rules.engine import RuleEngine, check_country_arg
 from euredact.rules.evidence import weights_to_ranking
 from euredact.types import EntityType, RedactResult
 
@@ -112,7 +112,12 @@ class EuRedact:
                 engine applies keyword and structural (JSON/CSV header)
                 checks before emitting a date detection.
         """
-        # Step 0: Input size guard
+        # Step 0: argument and input-size guards. The country check runs here
+        # as well as in the engine so that a bare string is rejected before any
+        # work happens, and on every entry point that funnels through redact().
+        check_country_arg(countries, "countries")
+        check_country_arg(country_hint, "country_hint")
+
         if len(text) > self._max_input_length:
             raise ValueError(
                 f"Input text length ({len(text):,} chars) exceeds the maximum "