rsna-anonymizer 19.1.0.dev4__tar.gz → 19.1.0.dev5__tar.gz

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/CHANGELOG.md

@@ -6,6 +6,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [19.1.0.dev5]
+- PyPI development pre-release of 19.1 on branch `dev` (install with `uv pip install --pre rsna-anonymizer`)
+- After metadata PHI removal (Series View Metadata Save and AI Batch), Dataset Original Description and Embedded PHI columns refresh from cleaned Study/Series Description and removed spans; lookup CSV adds CleanedStudyDescription and CleanedSeriesDescription after the sticky Study/Series Description columns
+- AI Batch Embedded PHI: Pixel PHI / Metadata / Reports path checkboxes; Match strictness and Whitelist stay visible when Pixel PHI is off
+- Series View windowing keeps PET window widths below one, with shared auto-window helpers and projection/default-window alignment
+
 ## [19.1.0.dev4]
 - PyPI development pre-release of 19.1 on branch `dev` (install with `uv pip install --pre rsna-anonymizer`)
 - PyPI source archive and wheel ship the installable packages. Pixel PHI experiment output is no longer stored in the repository. Help-site screenshots and the curated DICOM fixtures stay in git.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rsna-anonymizer
-Version: 19.1.0.dev4
+Version: 19.1.0.dev5
 Summary: RSNA DICOM Anonymizer
 Project-URL: Homepage, https://github.com/RSNA/anonymizer
 Author-email: Algorhythm Software <michael@dx.life>
@@ -62,7 +62,7 @@ Other languages: [Deutsch](readme.de.md), [Español](readme.es.md), [Français](
 [![PyPI](https://img.shields.io/pypi/v/rsna-anonymizer.svg)](https://pypi.org/project/rsna-anonymizer/)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
 
-**Version 19.1.0.dev4** is the current 19.1 development pre-release on the `dev` branch (`uv pip install --pre rsna-anonymizer`). Stable **19.0.10** remains on `master`. Requires **Python 3.11–3.14** with **tkinter** built in. Prefer **3.12** for the widest Tk/AI wheel coverage; avoid **3.14.1** (torchvision excludes that patch).
+**Version 19.1.0.dev5** is the current 19.1 development pre-release on the `dev` branch (`uv pip install --pre rsna-anonymizer`). Stable **19.0.10** remains on `master`. Requires **Python 3.11–3.14** with **tkinter** built in. Prefer **3.12** for the widest Tk/AI wheel coverage; avoid **3.14.1** (torchvision excludes that patch).
 
 ## Install
 
@@ -88,7 +88,7 @@ uv --version
 
 ### 2. Install Python with tkinter, then the app
 
-The desktop UI needs **tkinter**. Install a Python that ships with it, then create the venv and install **19.1.0.dev4** (`--pre`):
+The desktop UI needs **tkinter**. Install a Python that ships with it, then create the venv and install **19.1.0.dev5** (`--pre`):
 
 | Platform | Ensure tkinter is available |
 | --- | --- |
@@ -101,7 +101,7 @@ uv python install 3.12                     # or: 3.11 / 3.13 / 3.14
 uv venv rsna-anonymizer --python 3.12      # or: --python 3.11 / 3.13 / 3.14
 source rsna-anonymizer/bin/activate        # Windows: rsna-anonymizer\Scripts\activate
 
-uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
+uv pip install --pre rsna-anonymizer       # version 19.1.0.dev5
 ```
 
 ### 3. Verify the install
@@ -109,7 +109,7 @@ uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
 ```bash
 python --version          # 3.11.x–3.14.x
 python -m tkinter         # a small Tk window must open — required for the UI
-rsna-anonymizer --version # should report 19.1.0.dev4
+rsna-anonymizer --version # should report 19.1.0.dev5
 ```
 
 If `python -m tkinter` fails, the venv’s Python was built without Tk: fix the platform step above, recreate the venv, and reinstall.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "rsna-anonymizer"
-version = "19.1.0.dev4"
+version = "19.1.0.dev5"
 description = "RSNA DICOM Anonymizer"
 readme = "readme.md"
 license = { text = "Apache-2.0" }

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/readme.de.md

@@ -7,7 +7,7 @@
 [![PyPI](https://img.shields.io/pypi/v/rsna-anonymizer.svg)](https://pypi.org/project/rsna-anonymizer/)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
 
-**Version 19.1.0.dev4** ist die aktuelle 19.1-Entwicklungs-Vorabversion auf dem Branch `dev` (`uv pip install --pre rsna-anonymizer`). Stabil bleibt **19.0.10** auf `master`. Benötigt **Python 3.11–3.14** mit eingebautem **tkinter**. Empfohlen ist **3.12**; **3.14.1** vermeiden (torchvision schließt diesen Patch aus).
+**Version 19.1.0.dev5** ist die aktuelle 19.1-Entwicklungs-Vorabversion auf dem Branch `dev` (`uv pip install --pre rsna-anonymizer`). Stabil bleibt **19.0.10** auf `master`. Benötigt **Python 3.11–3.14** mit eingebautem **tkinter**. Empfohlen ist **3.12**; **3.14.1** vermeiden (torchvision schließt diesen Patch aus).
 
 ## Installation
 
@@ -33,7 +33,7 @@ uv --version
 
 ### 2. Python mit tkinter und die App installieren
 
-Die Desktop-Oberfläche braucht **tkinter**. Installieren Sie ein Python mit tkinter, erstellen Sie die venv und installieren Sie **19.1.0.dev4** (`--pre`):
+Die Desktop-Oberfläche braucht **tkinter**. Installieren Sie ein Python mit tkinter, erstellen Sie die venv und installieren Sie **19.1.0.dev5** (`--pre`):
 
 | Plattform | tkinter verfügbar machen |
 | --- | --- |
@@ -46,7 +46,7 @@ uv python install 3.12                     # or: 3.11 / 3.13 / 3.14
 uv venv rsna-anonymizer --python 3.12      # or: --python 3.11 / 3.13 / 3.14
 source rsna-anonymizer/bin/activate        # Windows: rsna-anonymizer\Scripts\activate
 
-uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
+uv pip install --pre rsna-anonymizer       # version 19.1.0.dev5
 ```
 
 ### 3. Installation prüfen
@@ -54,7 +54,7 @@ uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
 ```bash
 python --version          # 3.11.x–3.14.x
 python -m tkinter         # a small Tk window must open — required for the UI
-rsna-anonymizer --version # should report 19.1.0.dev4
+rsna-anonymizer --version # should report 19.1.0.dev5
 ```
 
 Wenn `python -m tkinter` fehlschlägt, wurde das Python der venv ohne Tk gebaut: Plattformschritt oben korrigieren, venv neu anlegen und neu installieren.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/readme.es.md

@@ -7,7 +7,7 @@
 [![PyPI](https://img.shields.io/pypi/v/rsna-anonymizer.svg)](https://pypi.org/project/rsna-anonymizer/)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
 
-**La versión 19.1.0.dev4** es la prelanzamiento de desarrollo 19.1 actual en la rama `dev` (`uv pip install --pre rsna-anonymizer`). La estable **19.0.10** permanece en `master`. Requiere **Python 3.11–3.14** con **tkinter** incluido. Se recomienda **3.12**; evite **3.14.1** (torchvision excluye ese parche).
+**La versión 19.1.0.dev5** es la prelanzamiento de desarrollo 19.1 actual en la rama `dev` (`uv pip install --pre rsna-anonymizer`). La estable **19.0.10** permanece en `master`. Requiere **Python 3.11–3.14** con **tkinter** incluido. Se recomienda **3.12**; evite **3.14.1** (torchvision excluye ese parche).
 
 ## Instalación
 
@@ -33,7 +33,7 @@ uv --version
 
 ### 2. Instalar Python con tkinter y la aplicación
 
-La interfaz de escritorio necesita **tkinter**. Instale un Python que lo incluya, cree el venv e instale **19.1.0.dev4** (`--pre`):
+La interfaz de escritorio necesita **tkinter**. Instale un Python que lo incluya, cree el venv e instale **19.1.0.dev5** (`--pre`):
 
 | Plataforma | Asegurar tkinter |
 | --- | --- |
@@ -46,7 +46,7 @@ uv python install 3.12                     # or: 3.11 / 3.13 / 3.14
 uv venv rsna-anonymizer --python 3.12      # or: --python 3.11 / 3.13 / 3.14
 source rsna-anonymizer/bin/activate        # Windows: rsna-anonymizer\Scripts\activate
 
-uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
+uv pip install --pre rsna-anonymizer       # version 19.1.0.dev5
 ```
 
 ### 3. Verificar la instalación
@@ -54,7 +54,7 @@ uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
 ```bash
 python --version          # 3.11.x–3.14.x
 python -m tkinter         # a small Tk window must open — required for the UI
-rsna-anonymizer --version # should report 19.1.0.dev4
+rsna-anonymizer --version # should report 19.1.0.dev5
 ```
 
 Si `python -m tkinter` falla, el Python del venv se compiló sin Tk: corrija el paso de la plataforma, recree el venv y reinstale.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/readme.fr.md

@@ -7,7 +7,7 @@
 [![PyPI](https://img.shields.io/pypi/v/rsna-anonymizer.svg)](https://pypi.org/project/rsna-anonymizer/)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
 
-**La version 19.1.0.dev4** est la préversion de développement 19.1 actuelle sur la branche `dev` (`uv pip install --pre rsna-anonymizer`). La stable **19.0.10** reste sur `master`. Nécessite **Python 3.11–3.14** avec **tkinter** intégré. Préférez **3.12** ; évitez **3.14.1** (torchvision exclut ce correctif).
+**La version 19.1.0.dev5** est la préversion de développement 19.1 actuelle sur la branche `dev` (`uv pip install --pre rsna-anonymizer`). La stable **19.0.10** reste sur `master`. Nécessite **Python 3.11–3.14** avec **tkinter** intégré. Préférez **3.12** ; évitez **3.14.1** (torchvision exclut ce correctif).
 
 ## Installation
 
@@ -33,7 +33,7 @@ uv --version
 
 ### 2. Installer Python avec tkinter, puis l’application
 
-L’interface de bureau a besoin de **tkinter**. Installez un Python qui l’inclut, créez le venv et installez **19.1.0.dev4** (`--pre`) :
+L’interface de bureau a besoin de **tkinter**. Installez un Python qui l’inclut, créez le venv et installez **19.1.0.dev5** (`--pre`) :
 
 | Plateforme | Assurer la disponibilité de tkinter |
 | --- | --- |
@@ -46,7 +46,7 @@ uv python install 3.12                     # or: 3.11 / 3.13 / 3.14
 uv venv rsna-anonymizer --python 3.12      # or: --python 3.11 / 3.13 / 3.14
 source rsna-anonymizer/bin/activate        # Windows: rsna-anonymizer\Scripts\activate
 
-uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
+uv pip install --pre rsna-anonymizer       # version 19.1.0.dev5
 ```
 
 ### 3. Vérifier l’installation
@@ -54,7 +54,7 @@ uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
 ```bash
 python --version          # 3.11.x–3.14.x
 python -m tkinter         # a small Tk window must open — required for the UI
-rsna-anonymizer --version # should report 19.1.0.dev4
+rsna-anonymizer --version # should report 19.1.0.dev5
 ```
 
 Si `python -m tkinter` échoue, le Python du venv a été compilé sans Tk : corrigez l’étape plateforme ci-dessus, recréez le venv et réinstallez.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/readme.md

@@ -9,7 +9,7 @@ Other languages: [Deutsch](readme.de.md), [Español](readme.es.md), [Français](
 [![PyPI](https://img.shields.io/pypi/v/rsna-anonymizer.svg)](https://pypi.org/project/rsna-anonymizer/)
 [![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue.svg)](https://www.python.org/downloads/)
 
-**Version 19.1.0.dev4** is the current 19.1 development pre-release on the `dev` branch (`uv pip install --pre rsna-anonymizer`). Stable **19.0.10** remains on `master`. Requires **Python 3.11–3.14** with **tkinter** built in. Prefer **3.12** for the widest Tk/AI wheel coverage; avoid **3.14.1** (torchvision excludes that patch).
+**Version 19.1.0.dev5** is the current 19.1 development pre-release on the `dev` branch (`uv pip install --pre rsna-anonymizer`). Stable **19.0.10** remains on `master`. Requires **Python 3.11–3.14** with **tkinter** built in. Prefer **3.12** for the widest Tk/AI wheel coverage; avoid **3.14.1** (torchvision excludes that patch).
 
 ## Install
 
@@ -35,7 +35,7 @@ uv --version
 
 ### 2. Install Python with tkinter, then the app
 
-The desktop UI needs **tkinter**. Install a Python that ships with it, then create the venv and install **19.1.0.dev4** (`--pre`):
+The desktop UI needs **tkinter**. Install a Python that ships with it, then create the venv and install **19.1.0.dev5** (`--pre`):
 
 | Platform | Ensure tkinter is available |
 | --- | --- |
@@ -48,7 +48,7 @@ uv python install 3.12                     # or: 3.11 / 3.13 / 3.14
 uv venv rsna-anonymizer --python 3.12      # or: --python 3.11 / 3.13 / 3.14
 source rsna-anonymizer/bin/activate        # Windows: rsna-anonymizer\Scripts\activate
 
-uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
+uv pip install --pre rsna-anonymizer       # version 19.1.0.dev5
 ```
 
 ### 3. Verify the install
@@ -56,7 +56,7 @@ uv pip install --pre rsna-anonymizer       # version 19.1.0.dev4
 ```bash
 python --version          # 3.11.x–3.14.x
 python -m tkinter         # a small Tk window must open — required for the UI
-rsna-anonymizer --version # should report 19.1.0.dev4
+rsna-anonymizer --version # should report 19.1.0.dev5
 ```
 
 If `python -m tkinter` fails, the venv’s Python was built without Tk: fix the platform step above, recreate the venv, and reinstall.

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/src/anonymizer/controller/ai/batch_process.py

@@ -1298,9 +1298,15 @@ def _scrub_series_descriptors(
     dcm_paths: Sequence[Path],
     replace_mode: str = "blank",
     project_dir: Path | None = None,
+    anon_model: AnonymizerModel | None = None,
     on_log_detail: Callable[[str], None] | None = None,
 ) -> tuple[int, list[str]]:
-    """Run Clean Descriptors on each instance; return (modified_count, removed texts)."""
+    """Run Clean Descriptors on each instance; return (modified_count, removed texts).
+
+    When anon_model is set, removed spans are merged into Instance.embedded_phi so
+    the Dataset series row Embedded PHI column updates after the batch refresh.
+    """
+    del series_path  # retained for call-site clarity / future series-level hooks
     mode = "synth" if str(replace_mode).strip().lower() == "synth" else "blank"
     modified_count = 0
     all_removed: list[str] = []
@@ -1317,6 +1323,20 @@ def _scrub_series_descriptors(
         if changed:
             modified_count += 1
             all_removed.extend(removed)
+            if anon_model is not None:
+                if removed:
+                    apply_instance_embedded_phi_for_dcm(
+                        anon_model, dcm_path, removed, merge=True
+                    )
+                # Keep Dataset Original Description in sync with cleaned tags.
+                try:
+                    ds = dcmread(str(dcm_path), stop_before_pixels=True, force=True)
+                    anon_model.sync_descriptions_from_dataset(ds)
+                except Exception:
+                    logger.exception(
+                        "Failed to sync Dataset descriptions after Clean Descriptors for %s",
+                        dcm_path,
+                    )
             if on_log_detail is not None and removed:
                 on_log_detail(f"{dcm_path.name}: " + ", ".join(removed[:8]))
     return modified_count, all_removed
@@ -1386,6 +1406,7 @@ def _apply_remove_pixel_phi_series(
                     dcm_paths=sorted(get_dcm_files(series_path)),
                     replace_mode=metadata_replace_mode,
                     project_dir=project_dir,
+                    anon_model=anon_model,
                     on_log_detail=on_log_detail,
                 )
             except Exception as exc:  # noqa: BLE001
@@ -1540,6 +1561,7 @@ def _apply_remove_pixel_phi_series(
                 dcm_paths=dcm_paths,
                 replace_mode=metadata_replace_mode,
                 project_dir=project_dir,
+                anon_model=anon_model,
                 on_log_detail=on_log_detail,
             )
             all_removed_texts.extend(meta_removed)

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/src/anonymizer/controller/ai/embedded_phi/metadata/detect.py

@@ -6,8 +6,9 @@ Explicit workflow (steps 1–5; apply continues 6–7)
 2. Compile NER input (Tag Name: value + separators) — see ner_compile.py.
 3. OpenMed on that string (chunk with optional overlap when over char cap).
 4. Remap spans via the compile offset map onto original Content.
-5. Post-NER: whitelist drop (always includes DICOM VR codes), anon drop,
-   drop whole-cell pure-numeric false IDs, for/caret/Dr. expand, date +
+5. Post-NER: pixel decide on each span (NER score floor + whitelist-vs-NER
+   contest; DICOM VR codes and clinical keep terms in the keep list), anon
+   drop, drop whole-cell pure-numeric false IDs, for/caret/Dr. expand, date +
    descriptor PHI pattern fill (IDs, SSN-like, at/by sites, phone, Dr.).
 
 Public API
@@ -60,7 +61,11 @@ from anonymizer.controller.ai.embedded_phi.ner.radiology_filters import (
     drop_measurement_digit_id_entities,
     is_anon_identity_tag,
 )
-from anonymizer.controller.ai.embedded_phi.pixel.ocr_pipeline import PhiMatchMode, thresholds_for
+from anonymizer.controller.ai.embedded_phi.pixel.ocr_pipeline import (
+    PhiMatchMode,
+    phi_accepted_scores,
+    thresholds_for,
+)
 from anonymizer.controller.ai.embedded_phi.pixel.whitelist_io import load_modality_phi_match_mode
 from anonymizer.controller.ai.embedded_phi.report.process_report_text import resolve_dataset_whitelist
 from anonymizer.controller.ai.embedded_phi.term_similarity import similarity_term_match
@@ -121,9 +126,9 @@ def detect_metadata_phi(
         locale: NER locale pack.
         whitelist: Terms masked after NER. None loads resolve_dataset_whitelist.
             An explicit list (including empty) is used as given.
-        match_mode: PhiMatchMode for whitelist similarity on an imaging file.
-            None keeps the floor from resolve_dataset_whitelist. A report
-            keeps the floor from that resolver.
+        match_mode: PhiMatchMode for whitelist similarity and NER score floors
+            on an imaging file. None loads the modality preset (STANDARD when
+            unset).
         project_dir: Project storage dir holding per-modality whitelists
             and the compiled *-anonymizer.script (@safe skip).
         modality: Imaging modality code passed to resolve_dataset_whitelist.
@@ -144,7 +149,6 @@ def detect_metadata_phi(
     if whitelist is not None:
         base_terms = list(whitelist)
         effective_match = match_mode or load_modality_phi_match_mode(project_dir, modality)
-        similarity = thresholds_for(effective_match).whitelist_similarity
     else:
         resolved = resolve_dataset_whitelist(
             project_dir,
@@ -154,17 +158,20 @@ def detect_metadata_phi(
             modality=modality,
         )
         base_terms = list(resolved.terms)
-        similarity = resolved.similarity
-        if resolved.source == "modality" and match_mode is not None:
-            similarity = thresholds_for(match_mode).whitelist_similarity
+        if match_mode is not None:
+            effective_match = match_mode
+        else:
+            code = modality or (resolved.modalities[0] if resolved.modalities else None)
+            effective_match = load_modality_phi_match_mode(project_dir, code)
         logger.info(
-            "Clean Descriptors whitelist source=%s modalities=%s terms=%d similarity=%.2f",
+            "Clean Descriptors whitelist source=%s modalities=%s terms=%d mode=%s",
             resolved.source,
             ",".join(resolved.modalities) if resolved.modalities else "—",
             len(resolved.terms),
-            similarity,
+            effective_match.value,
         )
     effective_whitelist = _with_dicom_vr_allow_terms(base_terms)
+    similarity = thresholds_for(effective_match).whitelist_similarity
     min_length_ratio = 0.70
 
     ops = (
@@ -263,7 +270,7 @@ def detect_metadata_phi(
             ents,
             whitelist=effective_whitelist,
             anon_values=anon,
-            similarity=similarity,
+            match_mode=effective_match,
             min_length_ratio=min_length_ratio,
         )
         whitelist_dropped += w_drop
@@ -306,9 +313,9 @@ def detect_metadata_phi(
         out[text] = tuple(pattern)
 
     logger.info(
-        "Clean Descriptors step 5/7 post-NER whitelist_drop=%d anon_drop=%d "
+        "Clean Descriptors step 5/7 post-NER decide_drop=%d anon_drop=%d "
         "pure_numeric_whole_cell_drop=%d expand=%d pattern_fills=%d "
-        "with_phi=%d whitelist_terms=%d similarity=%.2f",
+        "with_phi=%d whitelist_terms=%d mode=%s wl>=%.2f ner>=%.2f",
         whitelist_dropped,
         anon_dropped,
         numeric_dropped,
@@ -316,7 +323,9 @@ def detect_metadata_phi(
         pattern_added,
         len(out),
         len(effective_whitelist),
+        effective_match.value,
         similarity,
+        thresholds_for(effective_match).ner_score,
     )
     return out
 
@@ -504,14 +513,15 @@ def _drop_whitelist_and_anon(
     *,
     whitelist: Sequence[str],
     anon_values: Sequence[str] | set[str] | frozenset[str],
-    similarity: float,
+    match_mode: PhiMatchMode,
     min_length_ratio: float,
 ) -> tuple[list[Entity], int, int]:
-    """Drop post-NER spans that match whitelist terms or anon header surfaces."""
+    """Drop post-NER spans via pixel decide (NER floor + whitelist contest) or anon."""
     anon = frozenset(str(v).strip() for v in anon_values if str(v).strip())
     terms = [str(t).strip() for t in whitelist if str(t).strip()]
+    thr = thresholds_for(match_mode)
     kept: list[Entity] = []
-    w_drop = 0
+    decide_drop = 0
     a_drop = 0
     for ent in ents:
         span = text[ent.start : ent.end].strip()
@@ -524,23 +534,38 @@ def _drop_whitelist_and_anon(
                 span[:60],
             )
             continue
+        wl_score: float | None = None
+        wl_term: str | None = None
         if terms:
             hit = similarity_term_match(
                 span,
                 terms,
-                threshold=similarity,
+                threshold=thr.whitelist_similarity,
                 min_length_ratio=min_length_ratio,
             )
-            if hit.matched:
-                w_drop += 1
-                logger.info(
-                    "Clean Descriptors step 5/7 drop whitelist span=%r term=%r",
-                    span[:60],
-                    hit.term,
-                )
-                continue
+            if hit.term is not None:
+                wl_score = float(hit.similarity)
+                wl_term = hit.term
+        if not phi_accepted_scores(
+            float(ent.score),
+            has_label=bool(str(ent.label or "").strip()),
+            whitelist_score=wl_score,
+            mode=match_mode,
+        ):
+            decide_drop += 1
+            logger.info(
+                "Clean Descriptors step 5/7 drop decide span=%r label=%s "
+                "ner=%.3f wl=%s term=%r mode=%s",
+                span[:60],
+                ent.label,
+                float(ent.score),
+                f"{wl_score:.3f}" if wl_score is not None else "—",
+                wl_term,
+                match_mode.value,
+            )
+            continue
         kept.append(ent)
-    return kept, w_drop, a_drop
+    return kept, decide_drop, a_drop
 
 
 def _post_ner_pattern_entities(text: str) -> list[Entity]:

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/src/anonymizer/controller/ai/embedded_phi/ner/entity.py

@@ -15,8 +15,9 @@ class Entity:
     """One PHI span: product label plus character offsets into the source string.
 
     score is the model's softmax confidence for raw_label (fine-grained HF
-    class), not for the mapped product label. No score threshold is applied —
-    see module README. UI hover shows label only.
+    class), not for the mapped product label. The NER engine keeps every mapped
+    span; pixel Embedded PHI and Clean Descriptors decide apply PhiMatchMode
+    floors and the whitelist-vs-NER contest. UI hover shows label only.
 
     replacement: when set (metadata PHI tagged with a DICOM keyword as
     label), replace_entities always writes this string — the anonymized

{rsna_anonymizer-19.1.0.dev4 → rsna_anonymizer-19.1.0.dev5}/src/anonymizer/controller/ai/embedded_phi/pixel/ocr_pipeline.py

@@ -13,9 +13,24 @@ PhiMatchMode
 PhiMatchThresholds
     whitelist_similarity and ner_score for one mode.
 
+WHITELIST_CONFIDENT_SCORE
+    Whitelist similarity above this always wins over NER (0.8).
+
+WHITELIST_OVER_NER_MARGIN
+    Weaker whitelist wins when ner.score < whitelist.score - this margin (0.1).
+
 thresholds_for(mode: PhiMatchMode) -> PhiMatchThresholds
     Preset floors for a match mode.
 
+whitelist_score_meets_floor(whitelist_score: float | None, mode: PhiMatchMode) -> bool
+    True when whitelist_score meets the mode similarity floor.
+
+whitelist_overrides_ner_scores(ner_score: float | None, whitelist_score: float | None, mode: PhiMatchMode) -> bool
+    Pixel/metadata decide: confident whitelist wins, or weaker hit with margin.
+
+phi_accepted_scores(ner_score: float | None, *, has_label: bool, whitelist_score: float | None, mode: PhiMatchMode) -> bool
+    True when NER meets the mode floor and whitelist does not override.
+
 is_whitelisted(t: OCRText, mode: PhiMatchMode) -> bool
     True when whitelist.score is set and meets the mode similarity floor.
 
@@ -108,9 +123,9 @@ _NER_JOIN_MAX_SKIP_CHARS = 2
 _NER_TRAILING_SEPARATORS = frozenset(":-/\\|;,.")
 _PIXEL_NAME_LABELS = frozenset({"FIRST_NAME", "LAST_NAME"})
 # A whitelist score above this always wins over NER.
-_WHITELIST_CONFIDENT_SCORE = 0.8
+WHITELIST_CONFIDENT_SCORE = 0.8
 # Weaker hits override NER only when ner.score < whitelist.score - this margin.
-_WHITELIST_OVER_NER_MARGIN = 0.1
+WHITELIST_OVER_NER_MARGIN = 0.1
 # XR (CR/DX) and US burn-in clocks: any separators → HH:MM:SS for NER.
 _CLOCK_NORMALIZE_MODALITIES = frozenset({*XR_MODALITIES, Modality.US})
 # Separators: non-digits and/or digit 8 (EasyOCR often reads ':' as '8'), mixed ok.
@@ -152,37 +167,81 @@ def thresholds_for(mode: PhiMatchMode) -> PhiMatchThresholds:
     return _MATCH_PRESETS[mode]
 
 
-def is_whitelisted(t: OCRText, mode: PhiMatchMode) -> bool:
-    """True when whitelist.score is set and score >= mode.whitelist_similarity."""
-    thr = thresholds_for(mode)
-    return t.whitelist.score is not None and t.whitelist.score >= thr.whitelist_similarity
+def whitelist_score_meets_floor(
+    whitelist_score: float | None, mode: PhiMatchMode
+) -> bool:
+    """True when whitelist_score meets the mode similarity floor."""
+    if whitelist_score is None:
+        return False
+    return float(whitelist_score) >= thresholds_for(mode).whitelist_similarity
 
 
-def _whitelist_overrides_ner(t: OCRText, mode: PhiMatchMode) -> bool:
+def whitelist_overrides_ner_scores(
+    ner_score: float | None,
+    whitelist_score: float | None,
+    mode: PhiMatchMode,
+) -> bool:
     """
-    True when the box meets the whitelist floor and whitelist wins.
+    True when whitelist meets the mode floor and wins over NER.
 
     A whitelist score above 0.8 always wins. A weaker hit wins when NER is
     absent or ner.score < whitelist.score - 0.1. Close scores on a fuzzy
     resemblance stay PHI.
 
     Args:
-        t: Annotated OCR box.
+        ner_score: Model confidence for the PHI span, or None when unlabeled.
+        whitelist_score: Best keep-term similarity for the span, or None.
         mode: Current PhiMatchMode floors.
 
     Returns:
-        True when callers should treat the box as whitelist chrome, not PHI.
+        True when callers should treat the span as keep-list chrome, not PHI.
     """
-    if not is_whitelisted(t, mode):
-        return False
-    wl_score = t.whitelist.score
-    if wl_score is None:
+    if not whitelist_score_meets_floor(whitelist_score, mode):
         return False
-    if float(wl_score) > _WHITELIST_CONFIDENT_SCORE:
+    assert whitelist_score is not None
+    if float(whitelist_score) > WHITELIST_CONFIDENT_SCORE:
         return True
-    if t.ner.label is None or t.ner.score is None:
+    if ner_score is None:
         return True
-    return float(t.ner.score) < float(wl_score) - _WHITELIST_OVER_NER_MARGIN
+    return float(ner_score) < float(whitelist_score) - WHITELIST_OVER_NER_MARGIN
+
+
+def phi_accepted_scores(
+    ner_score: float | None,
+    *,
+    has_label: bool,
+    whitelist_score: float | None,
+    mode: PhiMatchMode,
+) -> bool:
+    """
+    True when NER meets the mode floor and whitelist does not override.
+
+    Shared by pixel boxes and Clean Descriptors metadata spans.
+
+    Args:
+        ner_score: Model confidence for the PHI span, or None.
+        has_label: True when a product PHI label is set.
+        whitelist_score: Best keep-term similarity for the span, or None.
+        mode: Current PhiMatchMode floors.
+
+    Returns:
+        True when callers should treat the span as Embedded PHI.
+    """
+    thr = thresholds_for(mode)
+    if not has_label or ner_score is None or float(ner_score) < thr.ner_score:
+        return False
+    return not whitelist_overrides_ner_scores(ner_score, whitelist_score, mode)
+
+
+def is_whitelisted(t: OCRText, mode: PhiMatchMode) -> bool:
+    """True when whitelist.score is set and score >= mode.whitelist_similarity."""
+    return whitelist_score_meets_floor(t.whitelist.score, mode)
+
+
+def _whitelist_overrides_ner(t: OCRText, mode: PhiMatchMode) -> bool:
+    """OCRText wrapper for whitelist_overrides_ner_scores."""
+    ner_score = None if t.ner.label is None else t.ner.score
+    return whitelist_overrides_ner_scores(ner_score, t.whitelist.score, mode)
 
 
 def phi_accepted(t: OCRText, mode: PhiMatchMode) -> bool:
@@ -198,14 +257,12 @@ def phi_accepted(t: OCRText, mode: PhiMatchMode) -> bool:
     Returns:
         True when callers should treat the box as Embedded PHI.
     """
-    thr = thresholds_for(mode)
-    if (
-        t.ner.label is None
-        or t.ner.score is None
-        or t.ner.score < thr.ner_score
-    ):
-        return False
-    return not _whitelist_overrides_ner(t, mode)
+    return phi_accepted_scores(
+        t.ner.score,
+        has_label=t.ner.label is not None,
+        whitelist_score=t.whitelist.score,
+        mode=mode,
+    )
 
 
 def accepted_phi_entities(t: OCRText, mode: PhiMatchMode) -> tuple[tuple[str, str], ...]: