pseudonymize 0.2.0__tar.gz → 0.2.1__tar.gz

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/.agents/PRIVATE_RELEASE_PLAN.md

@@ -92,7 +92,24 @@ PyPI upload verified:
 Status:
 ```
 
-### 0.2.0: Optional local ML
+### 0.2.1: Performance and default policy patches
+
+Implementation started: 2026-08-22
+Starting commit: aabba9a
+Starting PyPI version: 0.2.0
+Baseline downloads day/week/month: 0/0/0
+Baseline stars/forks/watchers: 0/0/0
+Files changed: 13
+Compatibility tests added: Yes
+Artifact smoke environments: Win local isolated wheel
+Known limitations: 
+Deferred items: 
+Release commit: TBD
+Tag: v0.2.1
+PyPI upload verified: Pending workflow
+7-day metrics: 
+28-day metrics: 
+Status: Ready
 
 Implementation started: 2026-08-21
 Starting commit: 3205c22ef6143aa8ae38fa2db9eea64fc237777d

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/.github/workflows/package.yml

@@ -49,16 +49,17 @@ jobs:
       - name: Install and audit wheel
         if: runner.os != 'Windows'
         run: |
-          uv pip install --python .wheel-venv/bin/python dist/pseudonymize-0.2.0-py3-none-any.whl
+          uv pip install --python .wheel-venv/bin/python dist/*.whl
           uv pip check --python .wheel-venv/bin/python
-          .wheel-venv/bin/python -I scripts/audit_install.py --version 0.2.0
+          .wheel-venv/bin/python -I scripts/audit_install.py
           .wheel-venv/bin/python -I scripts/smoke_wheel.py
           .wheel-venv/bin/pseudonymize detectors
       - name: Install and audit wheel on Windows
         if: runner.os == 'Windows'
+        shell: bash
         run: |
-          uv pip install --python .wheel-venv\Scripts\python.exe dist\pseudonymize-0.2.0-py3-none-any.whl
-          uv pip check --python .wheel-venv\Scripts\python.exe
-          .wheel-venv\Scripts\python.exe -I scripts\audit_install.py --version 0.2.0
-          .wheel-venv\Scripts\python.exe -I scripts\smoke_wheel.py
-          .wheel-venv\Scripts\pseudonymize.exe detectors
+          uv pip install --python .wheel-venv/Scripts/python.exe dist/*.whl
+          uv pip check --python .wheel-venv/Scripts/python.exe
+          .wheel-venv/Scripts/python.exe -I scripts/audit_install.py
+          .wheel-venv/Scripts/python.exe -I scripts/smoke_wheel.py
+          .wheel-venv/Scripts/pseudonymize.exe detectors

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/CHANGELOG.md

@@ -4,6 +4,33 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
 
 ## [Unreleased]
 
+## [0.2.1] - 2026-08-22
+
+### Changed
+
+- `Policy.default()` now includes `URL_CREDENTIAL` and `SECRET`, so the documented secret and
+  URL-credential detectors act without opting into `Policy.strict()` or `Policy.llm()`. Callers
+  that relied on secrets passing through the default policy must configure an explicit
+  `Policy(entity_types=...)`.
+- Overlap resolution now uses an ordered-interval scan instead of a quadratic pairwise check,
+  and detected spans are replaced with a single-pass segment join. Texts with thousands of
+  detections process in milliseconds instead of seconds.
+
+### Fixed
+
+- `LocalONNXPIIBackend` now merges contiguous token predictions into whole entity spans, so
+  multi-word and subword-split names receive a single alias instead of one alias per token.
+- ONNX test artifacts downloaded during testing are now verified against pinned SHA-256
+  checksums.
+- URL credential detection now covers the whole userinfo section when it contains extra `@`
+  characters and no longer swallows the URL fragment into sensitive query values.
+- Overlap resolution now ranks URL credentials above emails, so a `user:password@host` userinfo
+  can no longer lose its `user:` prefix to an overlapping email match.
+- The `ml` extra now declares `numpy`, which the ONNX backend imports directly, and no longer
+  pulls in the unused `huggingface-hub` dependency.
+- The optional-import guard in the ONNX backend now type-checks under strict mypy.
+- Normalization now maps generic URL schemes to `http` or `https` for safer alias sharing.
+
 ## [0.2.0] - 2026-08-21
 
 ### Added
@@ -135,7 +162,8 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
 - Deterministic processing now requires `mode="deterministic"` in addition to a key.
 - `redact()` now emits `[REDACTED]` by default; generic mode emits typed placeholders.
 
-[Unreleased]: https://github.com/ma2za/pseudonymize/compare/v0.2.0...HEAD
+[Unreleased]: https://github.com/ma2za/pseudonymize/compare/v0.2.1...HEAD
+[0.2.1]: https://github.com/ma2za/pseudonymize/compare/v0.2.0...v0.2.1
 [0.2.0]: https://github.com/ma2za/pseudonymize/releases/tag/v0.2.0
 [0.1.0]: https://github.com/ma2za/pseudonymize/releases/tag/v0.1.0
 [0.1.0rc1]: https://github.com/ma2za/pseudonymize/releases/tag/v0.1.0rc1

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pseudonymize
-Version: 0.2.0
+Version: 0.2.1
 Summary: Local-first PII pseudonymization for text, structured data, and LLM payloads.
 Project-URL: Homepage, https://github.com/ma2za/pseudonymize
 Project-URL: Documentation, https://ma2za.github.io/pseudonymize/

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/docs/policies.md

@@ -1,9 +1,10 @@
 # Policies
 
-`Policy.default()` enables structured identifiers plus person, organization, and location entities
-when a configured backend can detect them. `Policy.strict()` also enables secrets and URL
-credentials with a lower threshold. `Policy.llm()` enables all available entities.
-`Policy.financial()` limits detection to IBANs and payment cards.
+`Policy.default()` enables structured identifiers, secrets, URL credentials, and person,
+organization, and location entities when a configured backend can detect them.
+`Policy.strict()` enables the same entities with a lower confidence threshold. `Policy.llm()`
+enables all available entities. `Policy.financial()` limits detection to IBANs and payment
+cards.
 
 Paths are dot-separated segments. `*` matches one segment, so `messages.*.content` matches list
 indices. Exclusions win over inclusions. Dictionary keys are not transformed.

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/docs/releasing.md

@@ -29,10 +29,18 @@ After the first upload, verify that the pending publisher became an ordinary pro
    uv run ruff check .
    uv run mypy
    uv run pytest
-   uv run mkdocs build --strict
+   uv run python -m mkdocs build --strict
    uv build
    uv run twine check dist/*
    uv run python scripts/verify_release.py
+   
+   # Verify the built wheel locally in an isolated environment (simulating CI)
+   uv venv --python 3.14 .wheel-venv
+   uv pip install --python .wheel-venv/bin/python dist/*.whl
+   uv pip check --python .wheel-venv/bin/python
+   .wheel-venv/bin/python -I scripts/audit_install.py
+   .wheel-venv/bin/python -I scripts/smoke_wheel.py
+   .wheel-venv/bin/pseudonymize detectors
    ```
 
 4. Open a pull request and merge only after every required check passes.

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pseudonymize"
-version = "0.2.0"
+version = "0.2.1"
 description = "Local-first PII pseudonymization for text, structured data, and LLM payloads."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -91,7 +91,7 @@ minversion = "8"
 testpaths = ["tests"]
 pythonpath = ["."]
 python_files = ["test_*.py", "benchmark_*.py"]
-addopts = ["--strict-config", "--strict-markers", "-ra", "--cov=pseudonymize", "--cov-report=term-missing", "--cov-branch", "--cov-fail-under=99.36"]
+addopts = ["--strict-config", "--strict-markers", "-ra", "--cov=pseudonymize", "--cov-report=term-missing", "--cov-branch", "--cov-fail-under=99.36", "--basetemp=.pytest_temp"]
 markers = ["benchmark: performance benchmark", "integration: integration test", "slow: deliberately slow test"]
 filterwarnings = ["error"]
 

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/scripts/audit_install.py

@@ -27,11 +27,18 @@ def _blocked_network(*arguments: object, **keywords: object) -> None:
 
 def main() -> None:
     parser = argparse.ArgumentParser()
-    parser.add_argument("--version", required=True)
+    parser.add_argument("--version", required=False)
     arguments = parser.parse_args()
 
+    expected_version = arguments.version
+    if not expected_version:
+        import tomllib
+
+        with open("pyproject.toml", "rb") as stream:
+            expected_version = tomllib.load(stream)["project"]["version"]
+
     installed = distribution("pseudonymize")
-    if installed.version != arguments.version:
+    if installed.version != expected_version:
         raise RuntimeError("installed version does not match release")
     if installed.requires:
         required_deps = [req for req in installed.requires if "extra ==" not in req]

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/backends/ml/onnx.py

@@ -25,6 +25,20 @@ else:
         ort = None
         Tokenizer = None
 
+_LABEL_SUFFIXES: tuple[tuple[tuple[str, ...], EntityType], ...] = (
+    # Standard CoNLL-03 suffixes plus Ai4Privacy fine-grained labels.
+    (("PER", "FIRSTNAME", "LASTNAME", "MIDDLENAME"), EntityType.PERSON),
+    (("ORG", "COMPANYNAME"), EntityType.ORGANIZATION),
+    (("LOC", "CITY", "STATE", "COUNTY", "STREET", "ZIPCODE"), EntityType.LOCATION),
+)
+
+
+def _entity_type_for(label: str) -> EntityType | None:
+    for suffixes, entity_type in _LABEL_SUFFIXES:
+        if label.endswith(suffixes):
+            return entity_type
+    return None
+
 
 class LocalONNXPIIBackend(DetectionBackend):
     def __init__(
@@ -111,55 +125,44 @@ class LocalONNXPIIBackend(DetectionBackend):
 
             predictions = np.argmax(logits, axis=-1)
 
-            detections = []
+            # Token predictions are merged into entity spans: subword continuations
+            # (zero gap) and same-type tokens separated by one whitespace character
+            # collapse into a single detection so that "John Smith" is one PERSON.
+            spans: list[tuple[EntityType, int, int]] = []
 
             for idx, label_id in enumerate(predictions):
-                if label_id == 0 or self._id2label is None:
-                    continue
-
-                label_str = self._id2label.get(label_id)
+                label_str = (self._id2label or {}).get(int(label_id))
                 if not label_str or label_str == "O":
                     continue
-
-                entity_type = None
-
-                # Standard CoNLL-03 tags
-                if label_str.endswith("PER"):
-                    entity_type = EntityType.PERSON
-                elif label_str.endswith("ORG"):
-                    entity_type = EntityType.ORGANIZATION
-                elif label_str.endswith("LOC"):
-                    entity_type = EntityType.LOCATION
-
-                # Ai4Privacy tags mapping
-                elif any(label_str.endswith(s) for s in ("FIRSTNAME", "LASTNAME", "MIDDLENAME")):
-                    entity_type = EntityType.PERSON
-                elif label_str.endswith("COMPANYNAME"):
-                    entity_type = EntityType.ORGANIZATION
-                elif any(
-                    label_str.endswith(s) for s in ("CITY", "STATE", "COUNTY", "STREET", "ZIPCODE")
-                ):
-                    entity_type = EntityType.LOCATION
-
-                if entity_type:
-                    start, end = encoding.offsets[idx]
-                    if start == 0 and end == 0 and idx not in (0, len(predictions) - 1):
-                        continue
-                    if start == end:
+                entity_type = _entity_type_for(label_str)
+                if entity_type is None:
+                    continue
+                start, end = encoding.offsets[idx]
+                if start >= end:
+                    continue
+                if spans:
+                    previous_type, previous_start, previous_end = spans[-1]
+                    gap = block.text[previous_end:start]
+                    if (
+                        entity_type is previous_type
+                        and previous_end <= start <= previous_end + 1
+                        and not gap.strip()
+                    ):
+                        spans[-1] = (previous_type, previous_start, end)
                         continue
-
-                    detections.append(
-                        Detection(
-                            entity_type=entity_type,
-                            start=start,
-                            end=end,
-                            confidence=0.99,
-                            backend=self.name,
-                            detector="onnx",
-                        )
-                    )
-
-            return tuple(detections)
+                spans.append((entity_type, start, end))
+
+            return tuple(
+                Detection(
+                    entity_type=entity_type,
+                    start=start,
+                    end=end,
+                    confidence=0.99,
+                    backend=self.name,
+                    detector="onnx",
+                )
+                for entity_type, start, end in spans
+            )
 
         except Exception as e:
             raise BackendExecutionError(f"ONNX PII inference failed: {e}") from e

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/detectors/url.py

@@ -10,6 +10,11 @@ _SENSITIVE_QUERY_NAMES = frozenset(
 )
 
 
+def _authority_end(url: str, authority_start: int) -> int:
+    indexes = (url.find(character, authority_start) for character in "/?#")
+    return min((index for index in indexes if index != -1), default=len(url))
+
+
 @dataclass(frozen=True, slots=True)
 class UrlDetector:
     name: str = "url"
@@ -21,7 +26,8 @@ class UrlDetector:
             parsed = urllib.parse.urlsplit(url)
             if parsed.username is not None or parsed.password is not None:
                 authority_start = url.find("//") + 2
-                at = url.find("@", authority_start)
+                authority_end = _authority_end(url, authority_start)
+                at = url.rfind("@", authority_start, authority_end)
                 detections.append(
                     Detection(
                         EntityType.URL_CREDENTIAL,
@@ -34,8 +40,10 @@ class UrlDetector:
             query_start = url.find("?") + 1
             if query_start == 0:
                 continue
+            fragment_start = url.find("#", query_start)
+            query_end = fragment_start if fragment_start != -1 else len(url)
             cursor = query_start
-            for item in url[query_start:].split("&"):
+            for item in url[query_start:query_end].split("&"):
                 name, separator, value = item.partition("=")
                 if separator and urllib.parse.unquote_plus(name).lower() in _SENSITIVE_QUERY_NAMES:
                     value_start = cursor + len(name) + 1

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/engine.py

@@ -265,10 +265,15 @@ class Pseudonymizer:
             self.transformer.render(entity, alias)
             for entity, alias in zip(entities, aliases, strict=True)
         )
-        output = text
-        for entity, token in reversed(tuple(zip(entities, tokens, strict=True))):
+        segments: list[str] = []
+        cursor = 0
+        for entity, token in zip(entities, tokens, strict=True):
             detection = entity.detection
-            output = output[: detection.start] + token + output[detection.end :]
+            segments.append(text[cursor : detection.start])
+            segments.append(token)
+            cursor = detection.end
+        segments.append(text[cursor:])
+        output = "".join(segments)
         replacements = _replacement_reports(entities, tokens)
         reports.extend(_replacement_detection_reports(block, replacements))
         mapping = _mapping(text, entities, aliases, tokens) if include_mapping else None
@@ -287,10 +292,10 @@ class Pseudonymizer:
         if isinstance(data, str):
             block_id = f"block-{block_counter[0]:06d}"
             block_counter[0] += 1
-            block = ContentBlock(block_id, data, JSONPathLocation(path))
             if not self.policy.allows_path(tuple(str(part) for part in path)):
                 statistics.blocks_processed += 1
                 return data
+            block = ContentBlock(block_id, data, JSONPathLocation(path))
             return self._process_block(block, context, False, statistics, reports).text
         if data is None or isinstance(data, (bool, int, float)):
             return data

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/formats.py

@@ -242,7 +242,7 @@ def _json_replacements(document: Document) -> dict[tuple[str | int, ...], str]:
 
 def _render_json(document: Document, state: _AdapterState) -> str:
     value = _replace_json_strings(
-        _require_json_content(state.content),
+        cast(JSONValue, state.content),
         (),
         _json_replacements(document),
     )
@@ -250,32 +250,26 @@ def _render_json(document: Document, state: _AdapterState) -> str:
 
 
 def _extract_jsonl(text: str, decoded: _DecodedText) -> tuple[Document, tuple[JSONValue, ...]]:
-    if not text:
-        values: tuple[JSONValue, ...] = ()
-    else:
-        lines = text.splitlines()
-        values_list: list[JSONValue] = []
-        blocks: list[ContentBlock] = []
-        for line_index, line in enumerate(lines):
-            if not line.strip():
-                raise _JSONLLineError(line_index + 1)
-            try:
-                value = _load_json(line)
-            except (TypeError, ValueError):
-                raise _JSONLLineError(line_index + 1) from None
-            values_list.append(value)
-            _string_blocks(
-                value,
-                (line_index,),
-                blocks,
-                prefix=f"line-{line_index:06d}-block",
-            )
-        values = tuple(values_list)
-        return (
-            Document("file", tuple(blocks), _metadata(FileFormat.JSONL, decoded)),
-            values,
+    values: list[JSONValue] = []
+    blocks: list[ContentBlock] = []
+    for line_index, line in enumerate(text.splitlines()):
+        if not line.strip():
+            raise _JSONLLineError(line_index + 1)
+        try:
+            value = _load_json(line)
+        except (TypeError, ValueError):
+            raise _JSONLLineError(line_index + 1) from None
+        values.append(value)
+        _string_blocks(
+            value,
+            (line_index,),
+            blocks,
+            prefix=f"line-{line_index:06d}-block",
         )
-    return Document("file", (), _metadata(FileFormat.JSONL, decoded)), values
+    return (
+        Document("file", tuple(blocks), _metadata(FileFormat.JSONL, decoded)),
+        tuple(values),
+    )
 
 
 def _validate_document(document: Document, original: Document) -> None:
@@ -289,7 +283,7 @@ def _validate_document(document: Document, original: Document) -> None:
 
 
 def _render_jsonl(document: Document, state: _AdapterState) -> str:
-    values = _require_jsonl_content(state.content)
+    values = cast(tuple[JSONValue, ...], state.content)
     replacements = _json_replacements(document)
     rendered = (
         json.dumps(
@@ -332,7 +326,7 @@ def _read_csv(text: str) -> tuple[tuple[str, ...], ...]:
 
 
 def _render_csv(document: Document, state: _AdapterState) -> str:
-    rows = _require_csv_content(state.content)
+    rows = cast(tuple[tuple[str, ...], ...], state.content)
     replacements = {
         (
             cast(CSVCellLocation, block.location).row,
@@ -347,15 +341,3 @@ def _render_csv(document: Document, state: _AdapterState) -> str:
     output = io.StringIO(newline="")
     csv.writer(output, dialect="excel", lineterminator="\r\n").writerows(transformed)
     return output.getvalue()
-
-
-def _require_json_content(content: object) -> JSONValue:
-    return cast(JSONValue, content)
-
-
-def _require_jsonl_content(content: object) -> tuple[JSONValue, ...]:
-    return cast(tuple[JSONValue, ...], content)
-
-
-def _require_csv_content(content: object) -> tuple[tuple[str, ...], ...]:
-    return cast(tuple[tuple[str, ...], ...], content)

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/normalization.py

@@ -11,12 +11,11 @@ def normalize(value: str, entity_type: EntityType) -> str:
     if entity_type is EntityType.EMAIL:
         local, separator, domain = value.rpartition("@")
         return f"{local}{separator}{domain.lower()}"
-    if entity_type in {EntityType.IBAN, EntityType.PAYMENT_CARD, EntityType.PHONE}:
+    if entity_type is EntityType.IBAN:
+        return "".join(value.split()).upper()
+    if entity_type in {EntityType.PAYMENT_CARD, EntityType.PHONE}:
         prefix = "+" if entity_type is EntityType.PHONE and value.startswith("+") else ""
-        compact = "".join(character for character in value if character.isdigit())
-        if entity_type is EntityType.IBAN:
-            return "".join(value.split()).upper()
-        return prefix + compact
+        return prefix + "".join(character for character in value if character.isdigit())
     if entity_type is EntityType.IP_ADDRESS:
         return ipaddress.ip_address(value.strip("[]")).compressed
     return value

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/policy.py

@@ -13,6 +13,7 @@ _STRUCTURED = frozenset(
         EntityType.PAYMENT_CARD,
     }
 )
+_CREDENTIALS = frozenset({EntityType.URL_CREDENTIAL, EntityType.SECRET})
 _SEMANTIC = frozenset({EntityType.PERSON, EntityType.ORGANIZATION, EntityType.LOCATION})
 
 
@@ -24,7 +25,7 @@ class NetworkPolicy(StrEnum):
 
 @dataclass(frozen=True, slots=True)
 class Policy:
-    entity_types: Set[EntityType] = _STRUCTURED | _SEMANTIC
+    entity_types: Set[EntityType] = _STRUCTURED | _CREDENTIALS | _SEMANTIC
     minimum_confidence: float = 0.8
     detector_priority: tuple[str, ...] = ()
     include_paths: tuple[str, ...] = ()

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/src/pseudonymize/spans.py

@@ -1,3 +1,4 @@
+import bisect
 from collections.abc import Iterable
 
 from pseudonymize.result import Detection, EntityType
@@ -5,9 +6,12 @@ from pseudonymize.result import Detection, EntityType
 _ENTITY_PRIORITY = {
     EntityType.PAYMENT_CARD: 70,
     EntityType.IBAN: 70,
+    # URL credentials outrank emails: a password such as "s3cret" followed by
+    # "@host" also matches the email pattern, and letting the email span win
+    # would leave the "user:" part of the userinfo unmasked.
+    EntityType.URL_CREDENTIAL: 65,
     EntityType.EMAIL: 60,
     EntityType.IP_ADDRESS: 60,
-    EntityType.URL_CREDENTIAL: 50,
     EntityType.SECRET: 50,
     EntityType.PHONE: 40,
     EntityType.PERSON: 30,
@@ -35,12 +39,16 @@ def resolve_overlaps(
             detection.backend,
         ),
     )
+    # Accepted spans are kept sorted and non-overlapping, so a candidate can
+    # only collide with the span immediately before its insertion point.
+    starts: list[int] = []
+    ends: list[int] = []
     selected: list[Detection] = []
     for detection in ranked:
-        if any(
-            detection.start < existing.end and existing.start < detection.end
-            for existing in selected
-        ):
+        index = bisect.bisect_left(starts, detection.end)
+        if index and ends[index - 1] > detection.start:
             continue
+        starts.insert(index, detection.start)
+        ends.insert(index, detection.end)
         selected.append(detection)
     return tuple(sorted(selected, key=lambda detection: (detection.start, detection.end)))

{pseudonymize-0.2.0 → pseudonymize-0.2.1}/tests/unit/backends/test_onnx.py

@@ -1,3 +1,4 @@
+import hashlib
 import urllib.request
 from pathlib import Path
 from typing import Any
@@ -15,27 +16,39 @@ MODEL_URL_BASE = (
     "https://huggingface.co/onnx-community/distilbert_finetuned_ai4privacy_v2-ONNX/resolve/main/"
 )
 MODEL_FILES = {
-    "config.json": "config.json",
-    "tokenizer.json": "tokenizer.json",
-    "model_int8.onnx": "onnx/model_int8.onnx",
+    "config.json": (
+        "config.json",
+        "5155e76f303c68ee15d8f01b580550c062cebfbb42dda3f5f3698b2d75424216",
+    ),
+    "tokenizer.json": (
+        "tokenizer.json",
+        "cb374d6bc042c22455946f4e09a89d29882a199fdaf8fb25be00dc8b8857a448",
+    ),
+    "model_int8.onnx": (
+        "onnx/model_int8.onnx",
+        "6faa1d7f5b54140bbba18ba87480e11073927b5fff16f69558bd51058a05b305",
+    ),
 }
 
 
-def download_file(url: str, dest: Path) -> None:
-    if dest.exists():
-        return
-    dest.parent.mkdir(parents=True, exist_ok=True)
-    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})  # noqa: S310
-    with urllib.request.urlopen(req) as response, open(dest, "wb") as f:  # noqa: S310
-        f.write(response.read())
+def download_file(url: str, dest: Path, sha256: str) -> None:
+    if not dest.exists():
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})  # noqa: S310
+        with urllib.request.urlopen(req) as response, open(dest, "wb") as f:  # noqa: S310
+            f.write(response.read())
+    digest = hashlib.sha256(dest.read_bytes()).hexdigest()
+    if digest != sha256:
+        dest.unlink()
+        raise RuntimeError(f"checksum mismatch for {dest.name}: {digest}")
 
 
 @pytest.fixture(scope="session")
 def distilbert_artifacts() -> tuple[Path, Path, Path]:
     paths = []
-    for local_name, remote_path in MODEL_FILES.items():
+    for local_name, (remote_path, sha256) in MODEL_FILES.items():
         dest = CACHE_DIR / local_name
-        download_file(MODEL_URL_BASE + remote_path, dest)
+        download_file(MODEL_URL_BASE + remote_path, dest, sha256)
         paths.append(dest)
     # Returns (config, tokenizer, model)
     return (paths[0], paths[1], paths[2])
@@ -166,19 +179,15 @@ def test_ml_detect_real_inference_returns_meaningful_detections(
     # Sort detections by offset for deterministic assertion
     sorted_detections = sorted(detections, key=lambda d: d.start)
 
-    # We expect a good DistilBERT ML model to find:
-    # PERSON: "John" (0,4), "Smith" (5,10) - (backend yields token-by-token right now)
-    # LOC: "Seattle" (61,68), "Washington" (70,80)
-
-    # Note: Since the backend returns detections at the token level,
-    # we rigorously assert the exact subword spans mapped correctly to their entity type
-    assert len(sorted_detections) >= 4
+    # Token predictions merge into entity spans, so we expect:
+    # PERSON: "John Smith" (0,10) as one span, subwords and the space included
+    # LOC: "Seattle" (61,68) and "Washington" (70,80) kept apart by the comma
+    assert len(sorted_detections) >= 3
 
     # Let's map out the exact expected strings for the entities found
     found_entities = [(d.entity_type, text[d.start : d.end]) for d in sorted_detections]
 
-    assert (EntityType.PERSON, "John") in found_entities
-    assert (EntityType.PERSON, "Smith") in found_entities
+    assert (EntityType.PERSON, "John Smith") in found_entities
     assert (EntityType.LOCATION, "Seattle") in found_entities
     assert (EntityType.LOCATION, "Washington") in found_entities
 
@@ -250,6 +259,28 @@ def test_ml_detect_real_inference_returns_meaningful_detections(
     assert EntityType.LOCATION in found_types
 
 
+def test_ml_engine_shares_one_alias_per_merged_entity(
+    distilbert_artifacts: tuple[Path, Path, Path],
+) -> None:
+    from pseudonymize import Pseudonymizer
+
+    config_path, tokenizer_path, model_path = distilbert_artifacts
+    backend = LocalONNXPIIBackend(
+        model_path=model_path, tokenizer_path=tokenizer_path, config_path=config_path
+    )
+    engine = Pseudonymizer(backends=[backend])
+    result = engine.process("John Smith emailed John Smith from Seattle.")
+
+    assert "John" not in result.text
+    assert "Smith" not in result.text
+    person_tokens = {
+        replacement.token
+        for replacement in result.replacements
+        if replacement.detection.entity_type is EntityType.PERSON
+    }
+    assert person_tokens == {"<PERSON_1>"}
+
+
 def test_ml_detect_raises_on_inference_failure(
     distilbert_artifacts: tuple[Path, Path, Path], monkeypatch: pytest.MonkeyPatch
 ) -> None: