gdpr-pseudonymizer 2.1.3__tar.gz → 2.2.0__tar.gz

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gdpr-pseudonymizer
-Version: 2.1.3
+Version: 2.2.0
 Summary: CLI and GUI tool for GDPR-compliant pseudonymization of French text documents using NLP-based entity detection and reversible mapping
 License: MIT
 License-File: LICENSE
@@ -30,7 +30,7 @@ Provides-Extra: gui
 Provides-Extra: pdf
 Requires-Dist: PySide6-Essentials (>=6.7.0,<6.8.0) ; (python_version >= "3.10" and python_version < "3.13") and (extra == "gui" or extra == "all")
 Requires-Dist: PyYAML (>=6.0,<7.0)
-Requires-Dist: SQLAlchemy (>=2.0.0,<3.0.0)
+Requires-Dist: SQLAlchemy (>=2.0.0,<2.1.0)
 Requires-Dist: click (>=8.0.0,<8.2.0)
 Requires-Dist: cryptography (>=50.0.1,<51.0.0)
 Requires-Dist: markdown-it-py (>=3.0.0,<4.0.0)
@@ -85,10 +85,14 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
 
 | Platform | File | Notes |
 |----------|------|-------|
-| **Windows** | `gdpr-pseudonymizer-2.1.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
-| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
-| **macOS (Intel)** | `gdpr-pseudonymizer-2.1.0-macos-x86_64.dmg` | Open DMG, drag to Applications. |
-| **Linux** | `gdpr-pseudonymizer-2.1.0-linux.AppImage` | `chmod +x` then run. |
+| **Windows** | `gdpr-pseudonymizer-2.1.3-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
+| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.3-macos-arm64.dmg` | Open DMG, drag to Applications. |
+| **Linux** | `gdpr-pseudonymizer-2.1.3-linux.AppImage` | `chmod +x` then run. |
+
+> **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
+> Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
+> `pip install gdpr-pseudonymizer` instead — the CLI and GUI both work on Intel.
+
 
 ### Platform Notes
 
@@ -370,7 +374,8 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
 |----------|----------|-----------|--------|-------|
 | **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
 | **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
-| **Hybrid + expanded patterns** | 31.79% | 19.49% | 85.15% | Story 7.5 (current) |
+| **Hybrid + expanded patterns** | 31.79% | 25.38% | 42.54% | Story 7.5 |
+| **Span-bleed fixes** | 32.34% | 26.75% | 40.88% | 2026-09-29 (current) — recall drop is a ground-truth artefact, see QA report |
 
 **Accuracy trajectory:** spaCy-only baseline → hybrid approach with annotation cleanup, expanded regex patterns, and French geography dictionary doubled F1 score. Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
 
@@ -389,7 +394,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 - ✅ **Entity-by-type grouping** - Review PERSON → ORG → LOCATION in logical order
 - ✅ **Context display** - See 10 words before/after each entity with highlighting
 - ✅ **Confidence scores** - Color-coded confidence from spaCy NER (green >80%, yellow 60-80%, red <60%)
-- ✅ **Keyboard shortcuts** - Single-key actions: [Space] Confirm, [R] Reject, [E] Modify, [A] Add, [C] Change pseudonym
+- ✅ **Keyboard shortcuts** - Single-key actions: [Space] Confirm, [R] Reject, [E] Modify, [T] Change type, [A] Add, [C] Change pseudonym
 - ✅ **Batch operations** - Accept/reject all entities of a type at once (Shift+A/R) with entity count feedback
 - ✅ **Context cycling indicator** - Dot indicator (`● ○ ○ ○ ○`) shows current context position; `[Press X to cycle]` hint improves discoverability
 - ✅ **Help overlay** - Press [H] for full command reference

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/README.md

@@ -36,10 +36,14 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
 
 | Platform | File | Notes |
 |----------|------|-------|
-| **Windows** | `gdpr-pseudonymizer-2.1.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
-| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
-| **macOS (Intel)** | `gdpr-pseudonymizer-2.1.0-macos-x86_64.dmg` | Open DMG, drag to Applications. |
-| **Linux** | `gdpr-pseudonymizer-2.1.0-linux.AppImage` | `chmod +x` then run. |
+| **Windows** | `gdpr-pseudonymizer-2.1.3-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
+| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.3-macos-arm64.dmg` | Open DMG, drag to Applications. |
+| **Linux** | `gdpr-pseudonymizer-2.1.3-linux.AppImage` | `chmod +x` then run. |
+
+> **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
+> Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
+> `pip install gdpr-pseudonymizer` instead — the CLI and GUI both work on Intel.
+
 
 ### Platform Notes
 
@@ -321,7 +325,8 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
 |----------|----------|-----------|--------|-------|
 | **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
 | **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
-| **Hybrid + expanded patterns** | 31.79% | 19.49% | 85.15% | Story 7.5 (current) |
+| **Hybrid + expanded patterns** | 31.79% | 25.38% | 42.54% | Story 7.5 |
+| **Span-bleed fixes** | 32.34% | 26.75% | 40.88% | 2026-09-29 (current) — recall drop is a ground-truth artefact, see QA report |
 
 **Accuracy trajectory:** spaCy-only baseline → hybrid approach with annotation cleanup, expanded regex patterns, and French geography dictionary doubled F1 score. Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
 
@@ -340,7 +345,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 - ✅ **Entity-by-type grouping** - Review PERSON → ORG → LOCATION in logical order
 - ✅ **Context display** - See 10 words before/after each entity with highlighting
 - ✅ **Confidence scores** - Color-coded confidence from spaCy NER (green >80%, yellow 60-80%, red <60%)
-- ✅ **Keyboard shortcuts** - Single-key actions: [Space] Confirm, [R] Reject, [E] Modify, [A] Add, [C] Change pseudonym
+- ✅ **Keyboard shortcuts** - Single-key actions: [Space] Confirm, [R] Reject, [E] Modify, [T] Change type, [A] Add, [C] Change pseudonym
 - ✅ **Batch operations** - Accept/reject all entities of a type at once (Shift+A/R) with entity count feedback
 - ✅ **Context cycling indicator** - Dot indicator (`● ○ ○ ○ ○`) shows current context position; `[Press X to cycle]` hint improves discoverability
 - ✅ **Help overlay** - Press [H] for full command reference

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/cli/commands/batch.py

@@ -37,6 +37,7 @@ from gdpr_pseudonymizer.cli.progress import ETAColumn, ProgressTracker
 from gdpr_pseudonymizer.cli.validators import (
     ensure_database,
     parse_entity_type_filter,
+    validate_model_or_exit,
     validate_theme_or_exit,
 )
 from gdpr_pseudonymizer.core.document_processor import DocumentProcessor
@@ -438,6 +439,7 @@ def batch_command(
 
         # Validate theme
         validate_theme_or_exit(effective_theme)
+        validate_model_or_exit(effective_model)
 
         # Get passphrase
         resolved_passphrase = resolve_passphrase(

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/cli/commands/process.py

@@ -25,6 +25,7 @@ from gdpr_pseudonymizer.cli.passphrase import resolve_passphrase
 from gdpr_pseudonymizer.cli.validators import (
     ensure_database,
     parse_entity_type_filter,
+    validate_model_or_exit,
     validate_theme_or_exit,
 )
 from gdpr_pseudonymizer.core.document_processor import DocumentProcessor
@@ -161,6 +162,7 @@ def process_command(
 
         # Validate theme
         validate_theme_or_exit(effective_theme)
+        validate_model_or_exit(effective_model)
 
         # Get passphrase (with warning if --passphrase flag used)
         passphrase = resolve_passphrase(cli_passphrase=passphrase)

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/cli/config.py

@@ -22,6 +22,7 @@ from gdpr_pseudonymizer.exceptions import (
     ConfigValidationError,
     PassphraseInConfigError,
 )
+from gdpr_pseudonymizer.nlp.model_names import is_spacy_package_name
 
 
 @dataclass
@@ -73,6 +74,21 @@ VALID_LOG_LEVELS = ["DEBUG", "INFO", "WARNING", "ERROR"]
 VALID_MODELS = ["spacy"]
 
 
+def is_valid_model(model: str) -> bool:
+    """Check if a model value is accepted.
+
+    "spacy" selects the default French model; any well-formed spaCy package
+    name (e.g., "en_core_web_trf") selects that model.
+
+    Args:
+        model: Model value from config or CLI
+
+    Returns:
+        True if the value is accepted
+    """
+    return model in VALID_MODELS or is_spacy_package_name(model)
+
+
 def get_default_config() -> AppConfig:
     """Get default application configuration.
 
@@ -117,10 +133,11 @@ def validate_config_dict(config_dict: dict[str, Any], source: str = "config") ->
             )
 
         model = pseudonymization.get("model")
-        if model is not None and model not in VALID_MODELS:
+        if model is not None and not is_valid_model(str(model)):
             raise ConfigValidationError(
                 f"Invalid model '{model}' in {source}. "
-                f"Valid models: {', '.join(VALID_MODELS)}"
+                f"Valid models: {', '.join(VALID_MODELS)}, "
+                f"or a spaCy package name (e.g. en_core_web_trf)"
             )
 
     # Validate logging level

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/cli/main.py

@@ -220,7 +220,9 @@ def _process(
         None,
         "--model",
         "-m",
-        help=_("NLP model name (spacy). Default from config."),
+        help=_(
+            "NLP model: 'spacy' (default French model) or a spaCy package name, e.g. en_core_web_trf. Default from config."
+        ),
     ),
     db_path: Optional[str] = typer.Option(
         None,
@@ -287,7 +289,9 @@ def _batch(
         None,
         "--model",
         "-m",
-        help=_("NLP model name (spacy). Default from config."),
+        help=_(
+            "NLP model: 'spacy' (default French model) or a spaCy package name, e.g. en_core_web_trf. Default from config."
+        ),
     ),
     db_path: Optional[str] = typer.Option(
         None,

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/cli/validators.py

@@ -12,6 +12,7 @@ from typing import Optional
 from rich.console import Console
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
+from gdpr_pseudonymizer.cli.config import VALID_MODELS, is_valid_model
 from gdpr_pseudonymizer.cli.formatters import (
     ErrorCode,
     format_error_message,
@@ -302,6 +303,22 @@ def validate_theme_or_exit(theme: str) -> None:
         sys.exit(1)
 
 
+def validate_model_or_exit(model: str) -> None:
+    """Validate NLP model value and exit(1) if invalid.
+
+    Args:
+        model: "spacy" or a spaCy package name (e.g., "en_core_web_trf")
+    """
+    if not is_valid_model(model):
+        format_error_message(
+            "Invalid Model",
+            f"Model '{model}' is not recognized.",
+            f"Use {', '.join(VALID_MODELS)} (default French model) or a spaCy "
+            f"package name, e.g. en_core_web_trf",
+        )
+        sys.exit(1)
+
+
 def ensure_database(db_path: str, passphrase: str, console: Console) -> None:
     """Initialize the database if it does not exist, with a progress spinner.
 

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/core/document_processor.py

@@ -27,6 +27,7 @@ from gdpr_pseudonymizer.data.repositories.mapping_repository import (
 )
 from gdpr_pseudonymizer.nlp.entity_detector import DetectedEntity
 from gdpr_pseudonymizer.nlp.hybrid_detector import HybridDetector
+from gdpr_pseudonymizer.nlp.model_names import resolve_spacy_model
 from gdpr_pseudonymizer.pseudonym.assignment_engine import (
     CompositionalPseudonymEngine,
 )
@@ -132,7 +133,8 @@ class DocumentProcessor:
             db_path: Path to SQLite database file
             passphrase: Encryption passphrase for database access
             theme: Pseudonym library theme (neutral/star_wars/lotr)
-            model_name: NLP model name (spacy)
+            model_name: NLP model: "spacy" (default French model) or a spaCy
+                package name (e.g., "en_core_web_trf")
             notifier: Optional callback for user-facing messages.
                      Decouples core from CLI presentation layer.
 
@@ -162,7 +164,9 @@ class DocumentProcessor:
             OSError: If spaCy model not installed
         """
         if self._detector is None:
-            self._detector = HybridDetector()
+            self._detector = HybridDetector(
+                default_model=resolve_spacy_model(self.model_name)
+            )
         return self._detector
 
     def _detect_and_filter_entities(
@@ -1100,21 +1104,21 @@ class DocumentProcessor:
         Returns:
             Model version (e.g., "fr_core_news_lg-3.8.0")
         """
-        if self.model_name == "spacy":
-            try:
-                detector = self._get_detector()
-                if hasattr(detector, "nlp") and detector.nlp is not None:
-                    # Extract model name and version from spaCy meta
-                    meta = detector.nlp.meta
-                    return (
-                        f"{meta.get('name', 'unknown')}-{meta.get('version', '0.0.0')}"
-                    )
-            except (OSError, AttributeError, ImportError) as e:
-                logger.warning(
-                    "model_version_unavailable",
-                    error_type=type(e).__name__,
-                )
-        return f"{self.model_name}-unknown"
+        model = resolve_spacy_model(self.model_name)
+        try:
+            info = self._get_detector().spacy_detector.get_model_info()
+            if isinstance(info, dict) and "version" in info:
+                # spaCy meta names omit the language prefix ("core_news_lg")
+                lang, name = info.get("language", ""), info.get("name", "")
+                if lang and not name.startswith(f"{lang}_"):
+                    name = f"{lang}_{name}"
+                return f"{name}-{info['version']}"
+        except (OSError, AttributeError, ImportError) as e:
+            logger.warning(
+                "model_version_unavailable",
+                error_type=type(e).__name__,
+            )
+        return f"{model}-unknown"
 
     def _log_failed_operation(
         self,

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/data/models.py

@@ -45,7 +45,6 @@ class Entity(Base):
         DateTime,
         nullable=False,
         default=datetime.utcnow,
-        insert_default=datetime.utcnow,
     )
     gender: Mapped[str | None] = mapped_column(
         String, nullable=True
@@ -54,9 +53,7 @@ class Entity(Base):
     theme: Mapped[str] = mapped_column(String, nullable=False)  # neutral/star_wars/lotr
 
     # Validation support fields
-    is_ambiguous: Mapped[bool] = mapped_column(
-        Boolean, nullable=False, default=False, insert_default=False
-    )
+    is_ambiguous: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
     ambiguity_reason: Mapped[str | None] = mapped_column(String, nullable=True)
 
     def __init__(self, **kwargs: Any) -> None:
@@ -84,7 +81,6 @@ class Operation(Base):
         DateTime,
         nullable=False,
         default=datetime.utcnow,
-        insert_default=datetime.utcnow,
     )
     operation_type: Mapped[str] = mapped_column(
         String, nullable=False
@@ -132,7 +128,6 @@ class Metadata(Base):
         DateTime,
         nullable=False,
         default=datetime.utcnow,
-        insert_default=datetime.utcnow,
         onupdate=datetime.utcnow,
     )
 

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/locale/fr/LC_MESSAGES/cli.po

@@ -64,8 +64,8 @@ msgstr "Chemin du fichier de sortie (par défaut : <entrée>_pseudonymized.ext)"
 msgid "Pseudonym library theme (neutral/star_wars/lotr). Default from config."
 msgstr "Thème de la bibliothèque de pseudonymes (neutral/star_wars/lotr). Par défaut depuis la configuration."
 
-msgid "NLP model name (spacy). Default from config."
-msgstr "Nom du modèle NLP (spacy). Par défaut depuis la configuration."
+msgid "NLP model: 'spacy' (default French model) or a spaCy package name, e.g. en_core_web_trf. Default from config."
+msgstr "Modèle NLP : « spacy » (modèle français par défaut) ou un nom de paquet spaCy, p. ex. en_core_web_trf. Par défaut depuis la configuration."
 
 msgid "Database file path. Default from config."
 msgstr "Chemin du fichier de base de données. Par défaut depuis la configuration."

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/nlp/hybrid_detector.py

@@ -10,7 +10,10 @@ Implements EntityDetector interface using a hybrid approach:
 
 from __future__ import annotations
 
+import re
+
 from gdpr_pseudonymizer.nlp.entity_detector import DetectedEntity, EntityDetector
+from gdpr_pseudonymizer.nlp.model_names import DEFAULT_SPACY_MODEL
 from gdpr_pseudonymizer.nlp.regex_matcher import RegexMatcher
 from gdpr_pseudonymizer.nlp.spacy_detector import SpaCyDetector
 from gdpr_pseudonymizer.utils.french_patterns import strip_french_titles
@@ -38,10 +41,16 @@ class HybridDetector(EntityDetector):
         regex_matcher: RegexMatcher instance for pattern-based detection
     """
 
-    def __init__(self) -> None:
-        """Initialize hybrid detector with spaCy and regex components."""
+    def __init__(self, default_model: str = DEFAULT_SPACY_MODEL) -> None:
+        """Initialize hybrid detector with spaCy and regex components.
+
+        Args:
+            default_model: spaCy model loaded lazily on first detection when
+                load_model() was not called (e.g., "en_core_web_trf")
+        """
         self.spacy_detector = SpaCyDetector()
         self.regex_matcher = RegexMatcher()
+        self.default_model = default_model
         self._model_loaded = False
 
     def load_model(self, model_name: str) -> None:
@@ -87,14 +96,20 @@ class HybridDetector(EntityDetector):
 
         if not self._model_loaded:
             # Lazy load with default model
-            logger.warning("hybrid_detector_lazy_loading_model")
-            self.load_model("fr_core_news_lg")
+            logger.warning(
+                "hybrid_detector_lazy_loading_model", model=self.default_model
+            )
+            self.load_model(self.default_model)
 
         # Step 1: spaCy NER
         spacy_entities = self.spacy_detector.detect_entities(text)
         for entity in spacy_entities:
             entity.source = "spacy"
 
+        # Trim non-name tokens (timestamps, punctuation) off span edges before
+        # merging, so dedup and mapping lookup see the bare name
+        spacy_entities = self._trim_entity_boundaries(spacy_entities)
+
         logger.debug("spacy_detection_complete", entities_found=len(spacy_entities))
 
         # Step 2: Regex pattern matching (pass spaCy Doc for POS disambiguation)
@@ -236,6 +251,82 @@ class HybridDetector(EntityDetector):
 
         return False
 
+    @staticmethod
+    def _is_edge_junk_token(token: str, entity_type: str) -> bool:
+        """Check if a span-edge token cannot be part of an entity name.
+
+        Tokens with letters are always kept. PERSON names never contain
+        letterless tokens, so any is junk. ORG/LOCATION keep bare numbers
+        ("Studio 54", "Paris 2024") and only lose timestamps and punctuation.
+
+        Args:
+            token: Whitespace-delimited token at the edge of an entity span
+            entity_type: PERSON, LOCATION, or ORG
+
+        Returns:
+            True if the token should be trimmed off the span
+        """
+        if any(char.isalpha() for char in token):
+            return False
+        if entity_type == "PERSON":
+            return True
+        is_timestamp = ":" in token and any(char.isdigit() for char in token)
+        is_punctuation = not any(char.isalnum() for char in token)
+        return is_timestamp or is_punctuation
+
+    def _trim_entity_boundaries(
+        self, entities: list[DetectedEntity]
+    ) -> list[DetectedEntity]:
+        """Trim timestamps and punctuation that NER models glue to entity spans.
+
+        Transformer models (e.g. en_core_web_trf) often draw spans too wide on
+        transcripts: "Aino Virtanen        24:85:01" comes back as one PERSON.
+        Since mappings are keyed on the full entity text, every timestamp would
+        otherwise create a new entity with its own pseudonym.
+
+        Args:
+            entities: List of detected entities
+
+        Returns:
+            Entities with edge junk trimmed; entities that are only junk are dropped
+        """
+        trimmed = []
+        for entity in entities:
+            tokens = list(re.finditer(r"\S+", entity.text))
+            first, last = 0, len(tokens) - 1
+            while first <= last and self._is_edge_junk_token(
+                tokens[first].group(), entity.entity_type
+            ):
+                first += 1
+            while last >= first and self._is_edge_junk_token(
+                tokens[last].group(), entity.entity_type
+            ):
+                last -= 1
+
+            if first > last:
+                logger.debug(
+                    "junk_only_entity_filtered",
+                    text=entity.text,
+                    entity_type=entity.entity_type,
+                    reason="no_name_tokens",
+                )
+                continue
+
+            start = tokens[first].start()
+            end = tokens[last].end()
+            if start != 0 or end != len(entity.text):
+                logger.debug(
+                    "entity_boundary_trimmed",
+                    original=entity.text,
+                    trimmed=entity.text[start:end],
+                )
+                entity.start_pos += start
+                entity.end_pos = entity.start_pos + (end - start)
+                entity.text = entity.text[start:end]
+            trimmed.append(entity)
+
+        return trimmed
+
     def _filter_title_only_entities(
         self, entities: list[DetectedEntity]
     ) -> list[DetectedEntity]:

gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/nlp/model_names.py

@@ -0,0 +1,46 @@
+"""
+spaCy model name resolution
+
+Lightweight on purpose: imported by CLI config validation, so it must not
+import spaCy.
+"""
+
+from __future__ import annotations
+
+import re
+
+DEFAULT_SPACY_MODEL = "fr_core_news_lg"
+
+# spaCy package names: language code + underscore-separated parts
+# (fr_core_news_lg, en_core_web_trf, xx_ent_wiki_sm). Strict on purpose: the
+# name is passed to `python -m spacy download` when the model is missing.
+_SPACY_PACKAGE_NAME = re.compile(r"^[a-z]{2,3}(_[a-z0-9]+)+$")
+
+
+def is_spacy_package_name(name: str) -> bool:
+    """Check if a string is a well-formed spaCy model package name.
+
+    Args:
+        name: Candidate model name (e.g., "en_core_web_trf")
+
+    Returns:
+        True if the name looks like a spaCy package name
+    """
+    return bool(_SPACY_PACKAGE_NAME.match(name))
+
+
+def resolve_spacy_model(model_name: str) -> str:
+    """Map a configured model value to the spaCy package to load.
+
+    The legacy backend name "spacy" (and any other non-package value) maps to
+    the default French model, so existing configurations keep working.
+
+    Args:
+        model_name: Value from config or --model (e.g., "spacy", "en_core_web_trf")
+
+    Returns:
+        spaCy package name to load
+    """
+    if is_spacy_package_name(model_name):
+        return model_name
+    return DEFAULT_SPACY_MODEL

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/nlp/regex_matcher.py

@@ -121,6 +121,9 @@ class RegexMatcher:
                             "entity_type": config.get("entity_type", "PERSON"),
                             "confidence": config.get("confidence", 0.5),
                             "description": pattern_def.get("description", ""),
+                            "require_known_first_name": bool(
+                                config.get("require_known_first_name", False)
+                            ),
                         }
                     )
                 except re.error as e:
@@ -169,6 +172,11 @@ class RegexMatcher:
             for pattern_def in pattern_list:
                 matches = pattern_def["regex"].finditer(text)
                 for match in matches:
+                    if pattern_def[
+                        "require_known_first_name"
+                    ] and not self._has_known_first_name(match):
+                        continue
+
                     # Extract entity text (full match or last capturing group)
                     entity_text = match.group(0)
                     start_pos = match.start()
@@ -198,6 +206,29 @@ class RegexMatcher:
         logger.debug("regex_matching_complete", entities_found=len(entities))
         return entities
 
+    def _has_known_first_name(self, match: re.Match[str]) -> bool:
+        """Check that the match's last group contains a known first name.
+
+        Guards "Last, First" matching: without it, any "Word, Word" pair
+        ("Merci, Jean", "Thanks, Aino", "Weber, Directeur") becomes a PERSON.
+        Compound first names pass if any part is known ("Jean-Marc").
+        Without a loaded name dictionary the match is kept.
+
+        Args:
+            match: Regex match whose last capturing group is the first name
+
+        Returns:
+            True if the first name is in the name dictionary (or no dictionary)
+        """
+        if self.name_dictionary is None:
+            return True
+        first_name = match.group(match.re.groups)
+        return any(
+            self.name_dictionary.is_first_name(part)
+            for part in first_name.split("-")
+            if part
+        )
+
     def _is_full_names_enabled(self) -> bool:
         """Check if full names pattern matching is enabled."""
         pattern_config = self._config.get("patterns", {})

{gdpr_pseudonymizer-2.1.3 → gdpr_pseudonymizer-2.2.0}/gdpr_pseudonymizer/resources/detection_patterns.yaml

@@ -1,4 +1,6 @@
 # Regex Pattern Library for French Entity Detection
+# Letter classes cover Latin-1 + Latin Extended-A (í ñ ø å ş ı ł č š ž ő...),
+# not only French accents, so non-French names are not cut mid-word.
 # Each pattern category can be enabled/disabled for hybrid detection
 
 patterns:
@@ -8,8 +10,8 @@ patterns:
     confidence: 0.85
     entity_type: PERSON
     patterns:
-      - pattern: '\b(M\.|Mme|Mlle|Dr\.|Pr\.|Maître|Me\.?)\s+([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+(?:\s+[A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+)?)'
-        description: "Matches titles + names: M. Dupont, Dr. Marie Dubois, Mme Laurent, Maître Dupont, Me Mercier"
+      - pattern: '\b(M\.|Mme|Mlle|Dr\.|Pr\.|Maître|Me\.?)\s+([A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+(?:-[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)*(?:\s+[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+(?:-[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)*)?)'
+        description: "Matches titles + names: M. Dupont, Dr. Marie Dubois, Mme Laurent, Maître Dupont, Me Mercier, Dr. Marja-Liisa Mikkola"
         examples:
           - "M. Dupont"
           - "Mme Laurent"
@@ -25,7 +27,7 @@ patterns:
     confidence: 0.80
     entity_type: PERSON
     patterns:
-      - pattern: '\b([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+-[A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+)\b'
+      - pattern: '\b([A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+-[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)\b'
         description: "Matches hyphenated first names: Jean-Pierre, Marie-Claire"
         examples:
           - "Jean-Pierre"
@@ -38,8 +40,11 @@ patterns:
     enabled: true
     confidence: 0.80
     entity_type: PERSON
+    # First name (after the comma) must be in the name dictionary, otherwise
+    # any "Word, Word" pair matches ("Merci, Jean", "Weber, Directeur")
+    require_known_first_name: true
     patterns:
-      - pattern: '\b([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+(?:-[A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+)?)\s*,\s*(?!Cabinet|Société|Association|Fondation|Institut|Groupe|Consortium|Fédération|Entreprise|Compagnie|Direction|Service|Département|Commission|Comité|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Ligue)([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+(?:-[A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+)?)'
+      - pattern: '\b([A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+(?:-[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)?)\s*,\s*(?!Cabinet|Société|Association|Fondation|Institut|Groupe|Consortium|Fédération|Entreprise|Compagnie|Direction|Service|Département|Commission|Comité|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Ligue)([A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+(?:-[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)?)'
         description: "Last, First format (e.g., Dubois, Jean-Marc) with exclusion of ORG/function words"
         examples:
           - "Dubois, Jean-Marc"
@@ -52,7 +57,7 @@ patterns:
     confidence: 0.65
     entity_type: LOCATION
     patterns:
-      - pattern: '\b(à|en|dans|près de|ville de)\s+([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+(?:\s+[A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][a-zàâäéèêëïîôöùûü]+)?)'
+      - pattern: '\b(à|en|dans|près de|ville de)\s+([A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+(?:\s+[A-ZÀ-ÖØ-ÞĀ-ſ][a-zß-öø-ÿĀ-ſ]+)?)'
         description: "Matches location indicators: à Paris, en France, près de Lyon"
         examples:
           - "à Paris"
@@ -67,7 +72,7 @@ patterns:
     confidence: 0.70
     entity_type: ORG
     patterns:
-      - pattern: '\b([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][A-Za-zàâäéèêëïîôöùûü\s&,''\.\-]+?)\s+(SA|SARL|SAS|SASU|EURL|SNC|SCM|SCI|GIE|EI|SCOP|SEL|Association|Fondation|Institut|Groupe|Consortium|Fédération|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Comité|Commission|Ligue)\b'
+      - pattern: '\b([A-ZÀ-ÖØ-ÞĀ-ſ][A-Za-zß-öø-ÿĀ-ſ\s&,''\.\-]+?)\s+(SA|SARL|SAS|SASU|EURL|SNC|SCM|SCI|GIE|EI|SCOP|SEL|Association|Fondation|Institut|Groupe|Consortium|Fédération|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Comité|Commission|Ligue)\b'
         description: "Organization with legal form suffix (30 forms)"
         examples:
           - "TechCorp SA"
@@ -78,7 +83,7 @@ patterns:
           - "Solutions SCOP"
           - "TechCorp Association"
           - "Marie Curie Fondation"
-      - pattern: '\b(Société|Entreprise|Cabinet|Groupe|Compagnie|Association|Fondation|Institut|Consortium|Fédération|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Comité|Commission|Ligue)\s+([A-ZÀÂÄÉÈÊËÏÎÔÙÛÜ][A-Za-zàâäéèêëïîôöùûü\s&,\-]+)'
+      - pattern: '\b(Société|Entreprise|Cabinet|Groupe|Compagnie|Association|Fondation|Institut|Consortium|Fédération|Syndicat|Chambre|Mutuelle|Coopérative|Ordre|Caisse|Union|Confédération|Agence|Comité|Commission|Ligue)\s+([A-ZÀ-ÖØ-ÞĀ-ſ][A-Za-zß-öø-ÿĀ-ſ\s&,\-]+)'
         description: "Organization with prefix keyword (22 prefixes)"
         examples:
           - "Société TechCorp"