gdpr-pseudonymizer 2.2.0__tar.gz → 2.3.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/PKG-INFO +87 -32
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/README.md +83 -28
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/batch.py +53 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/cli/commands/delete_mapping.py +375 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/destroy_table.py +9 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/import_mappings.py +41 -7
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/list_mappings.py +9 -4
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/process.py +7 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/cli/commands/replace_pseudonym.py +300 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/validate_mappings.py +83 -11
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/main.py +54 -1
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/core/custom_pseudonym.py +1060 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/core/document_processor.py +555 -53
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/core/legacy_names.py +690 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/core/mapping_key.py +227 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/core/saved_own_words.py +549 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/data/database.py +872 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/models.py +10 -2
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/repositories/audit_repository.py +7 -5
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/data/repositories/mapping_repository.py +1329 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/exceptions.py +42 -1
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/accessibility/focus_manager.py +16 -5
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/app.py +15 -2
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/en.qm +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/en.ts +177 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/fr.ts +171 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/main_window.py +59 -1
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/gui/models/validation_state.py +1622 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/batch.py +221 -22
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/database.py +313 -12
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/home.py +47 -4
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/processing.py +19 -1
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/validation.py +353 -36
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/entity_panel.py +5 -3
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/passphrase_dialog.py +33 -2
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/toast.py +2 -1
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/batch_worker.py +60 -3
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/database_worker.py +92 -8
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/detection_worker.py +40 -9
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/finalization_worker.py +21 -5
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/processing_worker.py +3 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/locale/fr/LC_MESSAGES/cli.mo +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/locale/fr/LC_MESSAGES/cli.po +16 -2
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/nlp/hybrid_detector.py +2790 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/regex_matcher.py +9 -2
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/pseudonym/assignment_engine.py +12 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/pseudonym/library_manager.py +72 -15
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/__init__.py +5 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/detection_patterns.yaml +10 -8
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/resources/location_noise_filter.yaml +74 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/resources/org_role_filter.yaml +138 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/resources/person_boundaries.yaml +64 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/resources/place_abbreviations.yaml +14 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/pseudonyms/lotr.json +318 -469
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/pseudonyms/neutral.json +146 -182
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/pseudonyms/star_wars.json +358 -656
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/resources/salutations.yaml +17 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/file_handler.py +19 -7
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/tabular_reader.py +5 -2
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/tabular_writer.py +5 -3
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/utils/text_form.py +79 -0
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/validation/models.py +830 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/validation/ui.py +20 -4
- gdpr_pseudonymizer-2.3.1/gdpr_pseudonymizer/validation/workflow.py +1082 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/pyproject.toml +11 -11
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/cli/commands/delete_mapping.py +0 -283
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/data/database.py +0 -285
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/data/repositories/mapping_repository.py +0 -545
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/gui/models/validation_state.py +0 -568
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/locale/fr/LC_MESSAGES/cli.mo +0 -0
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/nlp/hybrid_detector.py +0 -491
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/validation/models.py +0 -453
- gdpr_pseudonymizer-2.2.0/gdpr_pseudonymizer/validation/workflow.py +0 -578
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/LICENSE +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/config_show.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/export.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/init.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/list_entities.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/commands/stats.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/config.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/formatters.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/i18n.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/passphrase.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/progress.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/validation_stub.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/cli/validators.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/core/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/core/naive_processor.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/encryption.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/naive_data.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/repositories/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/data/repositories/metadata_repository.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/accessibility/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/accessibility/shortcuts.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/config.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/dialogs/shortcuts_help.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/error_handler.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/fr.qm +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/i18n/translator.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/models/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/app.ico +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/icon_16.png +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/icon_256.png +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/icon_32.png +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/icon_48.png +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/icons/icon_512.png +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/themes/dark.qss +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/themes/high-contrast.qss +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/resources/themes/light.qss +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/results.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/settings.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/screens/stub.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/confirm_dialog.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/drop_zone.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/entity_editor.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/widgets/step_indicator.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/model_manager.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/gui/workers/signals.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/locale/en/LC_MESSAGES/cli.mo +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/locale/en/LC_MESSAGES/cli.po +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/entity_detector.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/entity_grouping.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/geography_dictionary.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/model_names.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/name_dictionary.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/spacy_detector.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/nlp/stanza_detector.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/pseudonym/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/pseudonym/gender_detector.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/pseudonym/neutral_id_generator.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/french_gender_lookup.json +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/french_geography.json +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/resources/french_names.json +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/config_manager.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/french_patterns.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/utils/logger.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/validation/__init__.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/validation/actions.py +0 -0
- {gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.1}/gdpr_pseudonymizer/validation/context_precomputer.py +0 -0
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: gdpr-pseudonymizer
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.3.1
|
|
4
4
|
Summary: CLI and GUI tool for GDPR-compliant pseudonymization of French text documents using NLP-based entity detection and reversible mapping
|
|
5
5
|
License: MIT
|
|
6
6
|
License-File: LICENSE
|
|
7
7
|
Keywords: gdpr,pseudonymization,nlp,privacy,data-protection
|
|
8
8
|
Author: GDPR Pseudonymizer Team
|
|
9
|
-
Requires-Python: >=3.
|
|
9
|
+
Requires-Python: >=3.11,<3.14
|
|
10
10
|
Classifier: Development Status :: 5 - Production/Stable
|
|
11
11
|
Classifier: Intended Audience :: Developers
|
|
12
12
|
Classifier: Intended Audience :: Legal Industry
|
|
@@ -15,7 +15,6 @@ Classifier: License :: OSI Approved :: MIT License
|
|
|
15
15
|
Classifier: Natural Language :: French
|
|
16
16
|
Classifier: Operating System :: OS Independent
|
|
17
17
|
Classifier: Programming Language :: Python :: 3
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
19
18
|
Classifier: Programming Language :: Python :: 3.11
|
|
20
19
|
Classifier: Programming Language :: Python :: 3.12
|
|
21
20
|
Classifier: Programming Language :: Python :: 3.13
|
|
@@ -28,11 +27,12 @@ Provides-Extra: excel
|
|
|
28
27
|
Provides-Extra: formats
|
|
29
28
|
Provides-Extra: gui
|
|
30
29
|
Provides-Extra: pdf
|
|
31
|
-
Requires-Dist: PySide6-Essentials (>=6.
|
|
30
|
+
Requires-Dist: PySide6-Essentials (>=6.8.3,<6.9.0) ; (python_version >= "3.11" and python_version < "3.14") and (extra == "gui" or extra == "all")
|
|
32
31
|
Requires-Dist: PyYAML (>=6.0,<7.0)
|
|
33
32
|
Requires-Dist: SQLAlchemy (>=2.0.0,<2.1.0)
|
|
34
33
|
Requires-Dist: click (>=8.0.0,<8.2.0)
|
|
35
34
|
Requires-Dist: cryptography (>=50.0.1,<51.0.0)
|
|
35
|
+
Requires-Dist: defusedxml (>=0.7.1) ; extra == "excel" or extra == "formats" or extra == "all"
|
|
36
36
|
Requires-Dist: markdown-it-py (>=3.0.0,<4.0.0)
|
|
37
37
|
Requires-Dist: openpyxl (>=3.1.0,<4.0.0) ; extra == "excel" or extra == "formats" or extra == "all"
|
|
38
38
|
Requires-Dist: pdfplumber (>=0.11.0,<0.12.0) ; extra == "pdf" or extra == "formats" or extra == "all"
|
|
@@ -63,7 +63,30 @@ Transform sensitive French documents for safe AI analysis with local processing,
|
|
|
63
63
|
|
|
64
64
|
---
|
|
65
65
|
|
|
66
|
-
## What's New in v2.
|
|
66
|
+
## What's New in v2.3
|
|
67
|
+
|
|
68
|
+
- **Names you add are replaced everywhere** — A name you add by hand (`[A]` in the command line, "Ajouter comme entité" in the desktop app) is now replaced at every place it appears, also where it is written in capitals or in capitals without accents ("ZORBAL", "HELENE"), never in a word written in small letters ("Petit" leaves "petit"). A name of two words or more is found whatever its accents and capitals.
|
|
69
|
+
- **Corrections cover the whole name** — Correcting a name or changing its type now applies at every place, in the desktop app as in the command line, in Excel and CSV cells too, and with the full name you type: correcting "Élorine" to "Élorine du Prélis" no longer leaves "du Prélis" in clear.
|
|
70
|
+
- **Your own pseudonym is used** — A pseudonym you choose ("Changer le pseudonyme", `[C]`) is now written in the file and kept for later documents. Before, a generated one was used instead.
|
|
71
|
+
- **Complete erasure** — `delete-mapping "Name"` deletes every saved pseudonym of that name: each type, and each way it was written with capitals or accents. The erasure log no longer keeps the erased name, and the names earlier versions wrote there are removed the first time v2.3 opens the database.
|
|
72
|
+
- **Safer batch runs** — In desktop batch review, each document keeps its own decisions and result when you move between documents, and a running batch can no longer mix with another run. When several documents of a `batch` meet the same new name at the same moment, it gets one pseudonym.
|
|
73
|
+
- **No name left in clear without warning** — Names you neither accepted nor rejected are announced before the file is written. When the detector finds no new name, the command line asks whether to add one it missed.
|
|
74
|
+
- **Same name, two types** — "Zorbaville" the town and "Zorbaville" the company now get separate pseudonyms. Your mapping database is converted once, the first time v2.3 opens it; after that, do not open it with v2.2 or older (copy it first if you may need to go back).
|
|
75
|
+
- **More precise detection** — One candidate per name instead of fragments; job titles (CTO, DRH, "VP Europe") are no longer offered as organisations; names no longer run into the next line; fewer false places; names with particles ("van der", "Le") stay whole; people in greetings, organisations followed by their country, short forms and aliases defined in a contract are now offered; accents stored as a separate mark are handled. On a blind final exam of 30 unseen documents, detection scores F1 61.4% (see [Detection accuracy](#detection-accuracy-epic-10-october-2026)).
|
|
76
|
+
|
|
77
|
+
<details>
|
|
78
|
+
<summary>v2.2 highlights</summary>
|
|
79
|
+
|
|
80
|
+
- **Choose the spaCy model** — `--model` / `pseudonymization.model` accept any spaCy package name (e.g. `en_core_web_trf` for English); `spacy` still selects the French `fr_core_news_lg`. The model only drives NLP detection: regex patterns, dictionaries and pseudonym libraries stay French. See the [CLI Reference](docs/CLI-REFERENCE.md).
|
|
81
|
+
- **Retype entities during CLI validation (`[T]`)** — Switch a detected entity between PERSON, LOCATION and ORG for every occurrence, instead of rejecting and re-adding it
|
|
82
|
+
- **Cleaner entity spans** — Editing with `[E]` no longer erases neighbouring text; "Last, First" matching now requires a known first name; timestamps and stray punctuation are trimmed from NER span edges; names with non-French letters ("María", "Yılmaz") and hyphenated names after titles are detected whole
|
|
83
|
+
- **Fresh `pip install` fixed** — v2.1.3 crashed on import when pip resolved SQLAlchemy 2.1; SQLAlchemy is now capped below 2.1
|
|
84
|
+
- **Audit log records the real model** — e.g. `fr_core_news_lg-3.8.0` instead of `spacy-unknown`
|
|
85
|
+
|
|
86
|
+
</details>
|
|
87
|
+
|
|
88
|
+
<details>
|
|
89
|
+
<summary>v2.1 highlights</summary>
|
|
67
90
|
|
|
68
91
|
- **Validate-once-per-entity** — Accept or reject one entity occurrence and it applies to all same-text occurrences in the document (productivity boost for repeated names)
|
|
69
92
|
- **Excel/CSV Format Support** — Process `.xlsx` and `.csv` files with cell-aware pseudonymization for HR/compliance use cases (`pip install gdpr-pseudonymizer[excel]`)
|
|
@@ -71,6 +94,8 @@ Transform sensitive French documents for safe AI analysis with local processing,
|
|
|
71
94
|
- **NER Accuracy Improvements** — Expanded ORG detection patterns, POS-tag disambiguation for geography matching; LOCATION false-negative rate reduced from 27% to 13%
|
|
72
95
|
- **GUI Discoverability** — F1 keyboard shortcuts help dialog with all shortcut groups, database path persistence across sessions, "Hide confirmed" toggle
|
|
73
96
|
|
|
97
|
+
</details>
|
|
98
|
+
|
|
74
99
|
<!-- TODO: Add GUI screenshot -->
|
|
75
100
|
|
|
76
101
|
**Upgrade:** `pip install --upgrade gdpr-pseudonymizer[gui,excel]`
|
|
@@ -85,9 +110,9 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
|
|
|
85
110
|
|
|
86
111
|
| Platform | File | Notes |
|
|
87
112
|
|----------|------|-------|
|
|
88
|
-
| **Windows** | `gdpr-pseudonymizer-2.
|
|
89
|
-
| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.
|
|
90
|
-
| **Linux** | `gdpr-pseudonymizer-2.
|
|
113
|
+
| **Windows** | `gdpr-pseudonymizer-2.3.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
|
|
114
|
+
| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.3.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
|
|
115
|
+
| **Linux** | `gdpr-pseudonymizer-2.3.0-linux.AppImage` | `chmod +x` then run. |
|
|
91
116
|
|
|
92
117
|
> **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
|
|
93
118
|
> Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
|
|
@@ -130,7 +155,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
|
|
|
130
155
|
- ✅ **Zero telemetry** - No analytics, crash reporting, or external communication
|
|
131
156
|
|
|
132
157
|
### 🤝 **AI + Human Verification**
|
|
133
|
-
- ✅ **Hybrid detection** -
|
|
158
|
+
- ✅ **Hybrid detection** - NLP + regex + geography dictionary: F1 61.4% on 30 unseen documents (CI run 38382553368). Improving this is a priority — see [Technical Details](#-technical-details)
|
|
134
159
|
- ✅ **Mandatory validation** - You review and confirm all entities (ensures 100% accuracy)
|
|
135
160
|
- ✅ **Fast validation UI** - Rich CLI interface with keyboard shortcuts, <2 min per document
|
|
136
161
|
- ✅ **Smart workflow** - Entity-by-type grouping (PERSON → ORG → LOCATION) with context display
|
|
@@ -165,7 +190,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
|
|
|
165
190
|
|
|
166
191
|
## 🚀 Quick Start
|
|
167
192
|
|
|
168
|
-
**Status:** 🎉 **v2.
|
|
193
|
+
**Status:** 🎉 **v2.3.0** (November 2026) — Detection Precision, Corrections at Every Place & Complete Erasure
|
|
169
194
|
|
|
170
195
|
### Getting Started
|
|
171
196
|
|
|
@@ -187,23 +212,27 @@ pip install gdpr-pseudonymizer[excel]
|
|
|
187
212
|
pip install gdpr-pseudonymizer[formats]
|
|
188
213
|
```
|
|
189
214
|
|
|
190
|
-
### What v2.
|
|
215
|
+
### What v2.3 Delivers
|
|
191
216
|
|
|
192
217
|
- 🖥️ **Desktop GUI** — Visual entity validation with drag-and-drop, batch dashboard, and database management
|
|
193
|
-
- 📦 **Standalone executables** — Windows .exe, macOS .dmg, Linux AppImage — no Python required
|
|
218
|
+
- 📦 **Standalone executables** — Windows .exe, macOS .dmg (Apple Silicon), Linux AppImage — no Python required
|
|
194
219
|
- ♿ **WCAG 2.1 AA accessibility** — Keyboard navigation, screen reader, high contrast mode
|
|
195
220
|
- 🌐 **French UI** — Complete FR/EN interface with live language switching
|
|
196
|
-
- 🤖 **AI-assisted detection** — Hybrid NLP + regex
|
|
221
|
+
- 🤖 **AI-assisted detection** — Hybrid NLP + regex detection: F1 61.4% (recall 60.6%) on 30 unseen documents by a different author (CI run 38382553368)
|
|
197
222
|
- ✅ **Mandatory human verification** — You review and confirm all entities (ensures 100% accuracy)
|
|
198
223
|
- 🔒 **100% local processing** — Your data never leaves your machine
|
|
199
224
|
- 📄 **PDF/DOCX support** — Process PDF and DOCX files directly (optional extras)
|
|
200
225
|
- 📊 **Excel/CSV support** — Process .xlsx and .csv files with cell-aware pseudonymization (optional extra: `[excel]`)
|
|
201
226
|
- 🆔 **Neutral ID theme** — Counter-based identifiers (PERSON-001, LIEU-001) for formal/legal contexts
|
|
202
227
|
- 🎯 **NER accuracy** — LOCATION false-negative rate reduced from 27% to 13% via regex expansion & POS disambiguation
|
|
228
|
+
- 🧠 **Choice of spaCy model** — French by default; any spaCy model for detection (e.g. `--model en_core_web_trf`)
|
|
229
|
+
- 🏷️ **Entity retyping** — Change an entity's type (PERSON/LOCATION/ORG) during CLI validation with `[T]`
|
|
230
|
+
- ✍️ **Corrections at every place** — Names you add, correct or retype are replaced wherever they appear, also where written in capitals
|
|
231
|
+
- 🗑️ **Complete erasure** — `delete-mapping` erases every saved pseudonym of a name, and the erasure log keeps no name
|
|
203
232
|
|
|
204
|
-
**What v2.
|
|
233
|
+
**What v2.3 does NOT deliver:**
|
|
205
234
|
- ❌ Fully automatic "set and forget" processing
|
|
206
|
-
- ❌ 85%+ AI accuracy (current:
|
|
235
|
+
- ❌ 85%+ AI accuracy (current: F1 61.4% on unseen documents, 82.6% on our development corpus — we plan to raise it)
|
|
207
236
|
- ❌ Optional validation mode (validation is mandatory)
|
|
208
237
|
|
|
209
238
|
### Roadmap
|
|
@@ -214,10 +243,14 @@ pip install gdpr-pseudonymizer[formats]
|
|
|
214
243
|
|
|
215
244
|
**v2.0 (Q1 2026):** Desktop GUI, standalone executables, WCAG AA accessibility, French UI, batch validation, core hardening
|
|
216
245
|
|
|
217
|
-
**v2.1 (Q1 2026)
|
|
246
|
+
**v2.1 (Q1 2026):** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
|
|
247
|
+
|
|
248
|
+
**v2.2 (Q3 2026):** Choice of spaCy model, entity retyping in CLI validation, entity-span accuracy fixes, fresh-install fix (SQLAlchemy < 2.1)
|
|
249
|
+
|
|
250
|
+
**v2.3 (Q4 2026) — CURRENT RELEASE:** Detection precision (Epic 10), benchmark repair and blind final exam, names added or corrected replaced at every place, custom pseudonyms used, complete erasure with no name in the erasure log, safer batch runs
|
|
218
251
|
|
|
219
252
|
**v3.0 (2027+):** NLP accuracy & automation
|
|
220
|
-
- Fine-tuned French NER model (70-85% F1 target, up from
|
|
253
|
+
- Fine-tuned French NER model (70-85% F1 target, up from 61.4% on unseen documents today)
|
|
221
254
|
- Optional `--no-validate` flag for high-confidence workflows
|
|
222
255
|
- Confidence-based auto-processing (85%+ F1 target)
|
|
223
256
|
- Multi-language support (English, Spanish, German)
|
|
@@ -229,7 +262,7 @@ pip install gdpr-pseudonymizer[formats]
|
|
|
229
262
|
See [Installation Guide](https://liochandayo.github.io/RGPDpseudonymizer/installation/) for detailed platform-specific instructions.
|
|
230
263
|
|
|
231
264
|
### Prerequisites
|
|
232
|
-
- **Python 3.
|
|
265
|
+
- **Python 3.11, 3.12 or 3.13** (validated in CI/CD — 3.14+ not yet tested)
|
|
233
266
|
|
|
234
267
|
### Install from PyPI (Recommended)
|
|
235
268
|
|
|
@@ -366,6 +399,20 @@ GDPR_PSEUDO_LANG=fr gdpr-pseudo --help
|
|
|
366
399
|
|
|
367
400
|
## 🔬 Technical Details
|
|
368
401
|
|
|
402
|
+
### Detection accuracy (Epic 10, October 2026)
|
|
403
|
+
|
|
404
|
+
**On unseen documents: F1 61.4%** (95% range 59.3–63.6%), recall 60.6%, precision 62.3%. This is a blind final exam: 30 French documents written by a different author in messy formats (email threads with quoted replies, PDF-style line breaks, tables flattened to text, mixed case), scored once (CI run 38382553368). By type: PERSON 69.2% F1, LOCATION 68.7%, ORG 34.8%; organisations remain the weak point.
|
|
405
|
+
|
|
406
|
+
**For context:** 82.6% F1 on the development corpus (25 documents) and 78.1% on a held-out set (6 documents) (CI run 38212303870). Both are closer to the documents the rules were built on.
|
|
407
|
+
|
|
408
|
+
How the final exam was made, and its limits:
|
|
409
|
+
- An LLM (GPT) wrote the 30 documents from a brief that named the hard cases to include but quoted no corpus text.
|
|
410
|
+
- Two models (GPT and Gemini) annotated them independently, and the maintainer settled their disagreements.
|
|
411
|
+
- The set was frozen before the last detection change, then scored once.
|
|
412
|
+
- The documents are synthetic, so real documents may score differently.
|
|
413
|
+
|
|
414
|
+
Details: [docs/qa/ner-accuracy-report.md](docs/qa/ner-accuracy-report.md).
|
|
415
|
+
|
|
369
416
|
### NLP Library Selection (Story 1.2 - Completed)
|
|
370
417
|
|
|
371
418
|
After comprehensive benchmarking on 25 French interview/business documents (1,737 annotated entities):
|
|
@@ -373,11 +420,18 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
|
|
|
373
420
|
| Approach | F1 Score | Precision | Recall | Notes |
|
|
374
421
|
|----------|----------|-----------|--------|-------|
|
|
375
422
|
| **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
|
|
376
|
-
| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
|
|
377
|
-
| **Hybrid
|
|
378
|
-
|
|
423
|
+
| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 (see scoring note) |
|
|
424
|
+
| **Hybrid (v2.2.0)** | **58.53%** | 48.42% | 73.98% | Scored the way the app normalizes entities |
|
|
425
|
+
|
|
426
|
+
By entity type (v2.2.0): PERSON 72.9% F1, LOCATION 45.0%, ORG 12.9%. Organisation detection is the weak point: job titles such as CTO, DPO or RSSI are often mistaken for companies.
|
|
427
|
+
|
|
428
|
+
**How these numbers are scored (corrected October 2026).** The app strips titles and prepositions before pseudonymizing ("Mme Isabelle Moreau" is handled as "Isabelle Moreau", "à Paris" as "Paris"), and the benchmark now scores detections the same way. Until then the committed scorer compared raw text, so every correctly detected "Mme Isabelle Moreau" counted as a miss plus a false alarm. Under that strict scoring the same detector measured ~32% F1, a figure briefly published on this page in October 2026; it understated real-use detection. The Story 5.3 figure (59.97%) appears to have been measured with title-aware matching and is consistent with v2.2.0's 58.53%. Each number in the table above was measured with `pytest tests/accuracy -m accuracy -s` on the version it names.
|
|
429
|
+
|
|
430
|
+
**Read these numbers with two caveats.** They come from one 25-document French interview/business corpus, and part of its ground truth was auto-annotated with the tool's own regex patterns, which skews results in both directions. And they measure *pre-detection* only: mandatory validation is what makes the final output correct.
|
|
431
|
+
|
|
432
|
+
**Epic 10 (October 2026)** repaired that benchmark against written annotation guidelines, added a held-out set and the blind final exam above, and improved organisation detection. A fine-tuned French NER model remains the v3.0 target (70-85% F1).
|
|
379
433
|
|
|
380
|
-
**Accuracy trajectory:**
|
|
434
|
+
**Accuracy trajectory:** Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
|
|
381
435
|
|
|
382
436
|
**Approved Solution:**
|
|
383
437
|
- ✅ **Hybrid approach** (NLP + regex + geography dictionary + POS disambiguation)
|
|
@@ -417,12 +471,12 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
417
471
|
|
|
418
472
|
| Component | Technology | Version | Purpose |
|
|
419
473
|
|-----------|------------|---------|---------|
|
|
420
|
-
| **Runtime** | Python | 3.
|
|
474
|
+
| **Runtime** | Python | 3.11-3.13 | Validated in CI/CD (3.14+ not yet tested) |
|
|
421
475
|
| **NLP Library** | spaCy | 3.8.0 | French entity detection (fr_core_news_lg) |
|
|
422
476
|
| **CLI Framework** | Typer | 0.9+ | Command-line interface |
|
|
423
477
|
| **Database** | SQLite | 3.35+ | Local mapping table storage with WAL mode |
|
|
424
478
|
| **Encryption** | cryptography (AESSIV) | 44.0+ | AES-256-SIV encryption for sensitive fields (PBKDF2 key derivation, passphrase-protected) |
|
|
425
|
-
| **ORM** | SQLAlchemy | 2.0
|
|
479
|
+
| **ORM** | SQLAlchemy | 2.0.x (<2.1) | Database abstraction and session management |
|
|
426
480
|
| **Desktop GUI** | PySide6 | 6.7+ | Desktop application (optional: `pip install gdpr-pseudonymizer[gui]`) |
|
|
427
481
|
| **Validation UI** | rich | 13.7+ | Interactive CLI entity review |
|
|
428
482
|
| **Keyboard Input** | readchar | 4.2+ | Single-keypress capture for validation UI |
|
|
@@ -438,7 +492,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
438
492
|
**Long answer:**
|
|
439
493
|
1. **GDPR defensibility** - Human verification provides legal audit trail
|
|
440
494
|
2. **Zero false negatives** - AI misses entities, humans catch them (100% coverage)
|
|
441
|
-
3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach
|
|
495
|
+
3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach: F1 61.4% and recall 60.6% on unseen documents, 82.6% F1 on our development corpus; organisations are the weak point; improvement planned)
|
|
442
496
|
4. **Better than alternatives:**
|
|
443
497
|
- ✅ **vs Manual redaction:** 50%+ faster (AI pre-detection)
|
|
444
498
|
- ✅ **vs Cloud services:** 100% local processing (no data leakage)
|
|
@@ -524,13 +578,13 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
524
578
|
|
|
525
579
|
## 🛠️ Development Status
|
|
526
580
|
|
|
527
|
-
**Epics 1-7 Complete** — v2.
|
|
581
|
+
**Epics 1-7 and 10 Complete** — current release v2.3.0 (November 2026) closed Epic 10: detection precision, corrections at every place, complete erasure. v2.1.0 (March 2026) closed Epic 7.
|
|
528
582
|
|
|
529
583
|
- ✅ **Epic 1:** Foundation & NLP Validation (9 stories) — spaCy integration, validation UI, hybrid detection, entity deduplication
|
|
530
584
|
- ✅ **Epic 2:** Core Pseudonymization Engine (9 stories) — pseudonym libraries, encryption, audit logging, batch processing, GDPR 1:1 mapping
|
|
531
585
|
- ✅ **Epic 3:** CLI Interface & Batch Processing (7 stories) — 8 CLI commands, progress reporting, config files, parallel batch, UX polish
|
|
532
586
|
- ✅ **Epic 4:** Launch Readiness (8 stories) — LLM utility validation, cross-platform testing, documentation, NER accuracy suite, performance validation, beta feedback integration, codebase refactoring, launch preparation
|
|
533
|
-
- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy
|
|
587
|
+
- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy work (annotation cleanup, expanded regex, geography dictionary), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
|
|
534
588
|
- ✅ **Epic 6:** v2.0 Desktop GUI & Broader Accessibility (9 stories) — PySide6 desktop application, visual entity validation, batch GUI, i18n, WCAG AA, standalone executables
|
|
535
589
|
- ✅ Story 6.1: UX Architecture & GUI Framework Selection
|
|
536
590
|
- ✅ Story 6.2: GUI Application Foundation (main window, theming, home screen, settings, 77 GUI tests)
|
|
@@ -552,6 +606,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
552
606
|
- ✅ Story 7.5: NER Accuracy Regex Expansion & POS Disambiguation
|
|
553
607
|
- ✅ Story 7.6: Quality & Compatibility Integration Tests
|
|
554
608
|
- ✅ Story 7.7: v2.1 Release Preparation
|
|
609
|
+
- ✅ **Epic 10:** v2.3 Detection Precision Quick Wins — benchmark repair and held-out set, blind final exam with confidence ranges, overlap and boundary fixes, fewer false places, greetings and organisation-plus-country handling, type-aware mappings, names added or corrected replaced at every place, custom pseudonyms, erasure log without names, batch fixes, v2.3 release
|
|
555
610
|
- **Total:** 60 stories, 1670+ tests, 86%+ coverage, all quality gates green
|
|
556
611
|
|
|
557
612
|
---
|
|
@@ -614,7 +669,7 @@ This project is licensed under the [MIT License](LICENSE).
|
|
|
614
669
|
- ⚠️ Test thoroughly before production use
|
|
615
670
|
|
|
616
671
|
**Current limitations:**
|
|
617
|
-
- AI detection:
|
|
672
|
+
- AI detection: F1 61.4% on unseen documents (82.6% on our development corpus), not 85%+; improving it is planned
|
|
618
673
|
- Validation required for ALL documents (not optional)
|
|
619
674
|
- French documents only (English, Spanish, etc. in future versions)
|
|
620
675
|
- Text-based formats: .txt, .md, .pdf, .docx, .xlsx, .csv (PDF/DOCX/Excel require optional extras: `pip install gdpr-pseudonymizer[formats]`)
|
|
@@ -702,7 +757,7 @@ poetry run pytest tests/integration/test_validation_workflow_integration.py -v
|
|
|
702
757
|
- **Performance tests:** 19 tests validating all NFR targets — single-document benchmarks (NFR1), entity-detection benchmarks, batch performance (NFR2), memory profiling (NFR4), startup time (NFR5), stability/error rate (NFR6), stress testing (Story 4.5)
|
|
703
758
|
- **Current coverage:** 86%+ across all modules (100% for progress module, 91.41% for AuditRepository)
|
|
704
759
|
- **Total tests:** 1670+ tests
|
|
705
|
-
- **CI/CD:** Tests run on Python 3.
|
|
760
|
+
- **CI/CD:** Tests run on Python 3.11-3.13 across Windows, macOS, and Linux
|
|
706
761
|
- **Quality gates:** All pass (Black, Ruff, mypy, pytest)
|
|
707
762
|
|
|
708
763
|
### Key Integration Test Scenarios
|
|
@@ -735,7 +790,7 @@ The integration test suite covers:
|
|
|
735
790
|
|
|
736
791
|
| Metric | Value | Status |
|
|
737
792
|
|--------|-------|--------|
|
|
738
|
-
| **Development Progress** | v2.
|
|
793
|
+
| **Development Progress** | v2.3.0 | ✅ Epics 1-7 and 10 complete |
|
|
739
794
|
| **Stories Complete** | 60 (Epics 1-7) | ✅ All epics complete |
|
|
740
795
|
| **LLM Utility (NFR10)** | 4.27/5.0 (85.4%) | ✅ PASSED (threshold: 80%) |
|
|
741
796
|
| **Installation Success (NFR3)** | 87.5% (7/8 platforms) | ✅ PASSED (threshold: 85%) |
|
|
@@ -743,7 +798,7 @@ The integration test suite covers:
|
|
|
743
798
|
| **Critical Bugs Found** | 1 (Story 2.8) | ✅ RESOLVED - Epic 3 Unblocked |
|
|
744
799
|
| **Test Corpus Size** | 25 docs, 1,737 entities | ✅ Complete (post-cleanup) |
|
|
745
800
|
| **NLP Accuracy (Baseline)** | 29.5% F1 (spaCy only) | ✅ Measured (Story 1.2) |
|
|
746
|
-
| **Hybrid Accuracy (NLP+Regex)** |
|
|
801
|
+
| **Hybrid Accuracy (NLP+Regex)** | 61.4% F1 on unseen documents (final exam, CI run 38382553368); 82.6% development corpus, 78.1% held-out (CI run 38212303870) | ⚠️ Below target — improvement planned |
|
|
747
802
|
| **Final Accuracy (AI+Human)** | 100% (validated) | 🎯 By Design |
|
|
748
803
|
| **Pseudonym Libraries** | 3 themes (2,426 names + 240 locations + 588 orgs) | ✅ Stories 2.1, 3.0, 4.6 Complete |
|
|
749
804
|
| **Compositional Matching** | Operational (component reuse + title stripping + compound names) | ✅ Stories 2.2, 2.3 Complete |
|
|
@@ -775,5 +830,5 @@ The integration test suite covers:
|
|
|
775
830
|
|
|
776
831
|
---
|
|
777
832
|
|
|
778
|
-
**Last Updated:** 2026-03
|
|
833
|
+
**Last Updated:** 2026-11-03 (v2.3.0 — detection precision, corrections at every place, complete erasure)
|
|
779
834
|
|
|
@@ -14,7 +14,30 @@ Transform sensitive French documents for safe AI analysis with local processing,
|
|
|
14
14
|
|
|
15
15
|
---
|
|
16
16
|
|
|
17
|
-
## What's New in v2.
|
|
17
|
+
## What's New in v2.3
|
|
18
|
+
|
|
19
|
+
- **Names you add are replaced everywhere** — A name you add by hand (`[A]` in the command line, "Ajouter comme entité" in the desktop app) is now replaced at every place it appears, also where it is written in capitals or in capitals without accents ("ZORBAL", "HELENE"), never in a word written in small letters ("Petit" leaves "petit"). A name of two words or more is found whatever its accents and capitals.
|
|
20
|
+
- **Corrections cover the whole name** — Correcting a name or changing its type now applies at every place, in the desktop app as in the command line, in Excel and CSV cells too, and with the full name you type: correcting "Élorine" to "Élorine du Prélis" no longer leaves "du Prélis" in clear.
|
|
21
|
+
- **Your own pseudonym is used** — A pseudonym you choose ("Changer le pseudonyme", `[C]`) is now written in the file and kept for later documents. Before, a generated one was used instead.
|
|
22
|
+
- **Complete erasure** — `delete-mapping "Name"` deletes every saved pseudonym of that name: each type, and each way it was written with capitals or accents. The erasure log no longer keeps the erased name, and the names earlier versions wrote there are removed the first time v2.3 opens the database.
|
|
23
|
+
- **Safer batch runs** — In desktop batch review, each document keeps its own decisions and result when you move between documents, and a running batch can no longer mix with another run. When several documents of a `batch` meet the same new name at the same moment, it gets one pseudonym.
|
|
24
|
+
- **No name left in clear without warning** — Names you neither accepted nor rejected are announced before the file is written. When the detector finds no new name, the command line asks whether to add one it missed.
|
|
25
|
+
- **Same name, two types** — "Zorbaville" the town and "Zorbaville" the company now get separate pseudonyms. Your mapping database is converted once, the first time v2.3 opens it; after that, do not open it with v2.2 or older (copy it first if you may need to go back).
|
|
26
|
+
- **More precise detection** — One candidate per name instead of fragments; job titles (CTO, DRH, "VP Europe") are no longer offered as organisations; names no longer run into the next line; fewer false places; names with particles ("van der", "Le") stay whole; people in greetings, organisations followed by their country, short forms and aliases defined in a contract are now offered; accents stored as a separate mark are handled. On a blind final exam of 30 unseen documents, detection scores F1 61.4% (see [Detection accuracy](#detection-accuracy-epic-10-october-2026)).
|
|
27
|
+
|
|
28
|
+
<details>
|
|
29
|
+
<summary>v2.2 highlights</summary>
|
|
30
|
+
|
|
31
|
+
- **Choose the spaCy model** — `--model` / `pseudonymization.model` accept any spaCy package name (e.g. `en_core_web_trf` for English); `spacy` still selects the French `fr_core_news_lg`. The model only drives NLP detection: regex patterns, dictionaries and pseudonym libraries stay French. See the [CLI Reference](docs/CLI-REFERENCE.md).
|
|
32
|
+
- **Retype entities during CLI validation (`[T]`)** — Switch a detected entity between PERSON, LOCATION and ORG for every occurrence, instead of rejecting and re-adding it
|
|
33
|
+
- **Cleaner entity spans** — Editing with `[E]` no longer erases neighbouring text; "Last, First" matching now requires a known first name; timestamps and stray punctuation are trimmed from NER span edges; names with non-French letters ("María", "Yılmaz") and hyphenated names after titles are detected whole
|
|
34
|
+
- **Fresh `pip install` fixed** — v2.1.3 crashed on import when pip resolved SQLAlchemy 2.1; SQLAlchemy is now capped below 2.1
|
|
35
|
+
- **Audit log records the real model** — e.g. `fr_core_news_lg-3.8.0` instead of `spacy-unknown`
|
|
36
|
+
|
|
37
|
+
</details>
|
|
38
|
+
|
|
39
|
+
<details>
|
|
40
|
+
<summary>v2.1 highlights</summary>
|
|
18
41
|
|
|
19
42
|
- **Validate-once-per-entity** — Accept or reject one entity occurrence and it applies to all same-text occurrences in the document (productivity boost for repeated names)
|
|
20
43
|
- **Excel/CSV Format Support** — Process `.xlsx` and `.csv` files with cell-aware pseudonymization for HR/compliance use cases (`pip install gdpr-pseudonymizer[excel]`)
|
|
@@ -22,6 +45,8 @@ Transform sensitive French documents for safe AI analysis with local processing,
|
|
|
22
45
|
- **NER Accuracy Improvements** — Expanded ORG detection patterns, POS-tag disambiguation for geography matching; LOCATION false-negative rate reduced from 27% to 13%
|
|
23
46
|
- **GUI Discoverability** — F1 keyboard shortcuts help dialog with all shortcut groups, database path persistence across sessions, "Hide confirmed" toggle
|
|
24
47
|
|
|
48
|
+
</details>
|
|
49
|
+
|
|
25
50
|
<!-- TODO: Add GUI screenshot -->
|
|
26
51
|
|
|
27
52
|
**Upgrade:** `pip install --upgrade gdpr-pseudonymizer[gui,excel]`
|
|
@@ -36,9 +61,9 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
|
|
|
36
61
|
|
|
37
62
|
| Platform | File | Notes |
|
|
38
63
|
|----------|------|-------|
|
|
39
|
-
| **Windows** | `gdpr-pseudonymizer-2.
|
|
40
|
-
| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.
|
|
41
|
-
| **Linux** | `gdpr-pseudonymizer-2.
|
|
64
|
+
| **Windows** | `gdpr-pseudonymizer-2.3.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
|
|
65
|
+
| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.3.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
|
|
66
|
+
| **Linux** | `gdpr-pseudonymizer-2.3.0-linux.AppImage` | `chmod +x` then run. |
|
|
42
67
|
|
|
43
68
|
> **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
|
|
44
69
|
> Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
|
|
@@ -81,7 +106,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
|
|
|
81
106
|
- ✅ **Zero telemetry** - No analytics, crash reporting, or external communication
|
|
82
107
|
|
|
83
108
|
### 🤝 **AI + Human Verification**
|
|
84
|
-
- ✅ **Hybrid detection** -
|
|
109
|
+
- ✅ **Hybrid detection** - NLP + regex + geography dictionary: F1 61.4% on 30 unseen documents (CI run 38382553368). Improving this is a priority — see [Technical Details](#-technical-details)
|
|
85
110
|
- ✅ **Mandatory validation** - You review and confirm all entities (ensures 100% accuracy)
|
|
86
111
|
- ✅ **Fast validation UI** - Rich CLI interface with keyboard shortcuts, <2 min per document
|
|
87
112
|
- ✅ **Smart workflow** - Entity-by-type grouping (PERSON → ORG → LOCATION) with context display
|
|
@@ -116,7 +141,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
|
|
|
116
141
|
|
|
117
142
|
## 🚀 Quick Start
|
|
118
143
|
|
|
119
|
-
**Status:** 🎉 **v2.
|
|
144
|
+
**Status:** 🎉 **v2.3.0** (November 2026) — Detection Precision, Corrections at Every Place & Complete Erasure
|
|
120
145
|
|
|
121
146
|
### Getting Started
|
|
122
147
|
|
|
@@ -138,23 +163,27 @@ pip install gdpr-pseudonymizer[excel]
|
|
|
138
163
|
pip install gdpr-pseudonymizer[formats]
|
|
139
164
|
```
|
|
140
165
|
|
|
141
|
-
### What v2.
|
|
166
|
+
### What v2.3 Delivers
|
|
142
167
|
|
|
143
168
|
- 🖥️ **Desktop GUI** — Visual entity validation with drag-and-drop, batch dashboard, and database management
|
|
144
|
-
- 📦 **Standalone executables** — Windows .exe, macOS .dmg, Linux AppImage — no Python required
|
|
169
|
+
- 📦 **Standalone executables** — Windows .exe, macOS .dmg (Apple Silicon), Linux AppImage — no Python required
|
|
145
170
|
- ♿ **WCAG 2.1 AA accessibility** — Keyboard navigation, screen reader, high contrast mode
|
|
146
171
|
- 🌐 **French UI** — Complete FR/EN interface with live language switching
|
|
147
|
-
- 🤖 **AI-assisted detection** — Hybrid NLP + regex
|
|
172
|
+
- 🤖 **AI-assisted detection** — Hybrid NLP + regex detection: F1 61.4% (recall 60.6%) on 30 unseen documents by a different author (CI run 38382553368)
|
|
148
173
|
- ✅ **Mandatory human verification** — You review and confirm all entities (ensures 100% accuracy)
|
|
149
174
|
- 🔒 **100% local processing** — Your data never leaves your machine
|
|
150
175
|
- 📄 **PDF/DOCX support** — Process PDF and DOCX files directly (optional extras)
|
|
151
176
|
- 📊 **Excel/CSV support** — Process .xlsx and .csv files with cell-aware pseudonymization (optional extra: `[excel]`)
|
|
152
177
|
- 🆔 **Neutral ID theme** — Counter-based identifiers (PERSON-001, LIEU-001) for formal/legal contexts
|
|
153
178
|
- 🎯 **NER accuracy** — LOCATION false-negative rate reduced from 27% to 13% via regex expansion & POS disambiguation
|
|
179
|
+
- 🧠 **Choice of spaCy model** — French by default; any spaCy model for detection (e.g. `--model en_core_web_trf`)
|
|
180
|
+
- 🏷️ **Entity retyping** — Change an entity's type (PERSON/LOCATION/ORG) during CLI validation with `[T]`
|
|
181
|
+
- ✍️ **Corrections at every place** — Names you add, correct or retype are replaced wherever they appear, also where written in capitals
|
|
182
|
+
- 🗑️ **Complete erasure** — `delete-mapping` erases every saved pseudonym of a name, and the erasure log keeps no name
|
|
154
183
|
|
|
155
|
-
**What v2.
|
|
184
|
+
**What v2.3 does NOT deliver:**
|
|
156
185
|
- ❌ Fully automatic "set and forget" processing
|
|
157
|
-
- ❌ 85%+ AI accuracy (current:
|
|
186
|
+
- ❌ 85%+ AI accuracy (current: F1 61.4% on unseen documents, 82.6% on our development corpus — we plan to raise it)
|
|
158
187
|
- ❌ Optional validation mode (validation is mandatory)
|
|
159
188
|
|
|
160
189
|
### Roadmap
|
|
@@ -165,10 +194,14 @@ pip install gdpr-pseudonymizer[formats]
|
|
|
165
194
|
|
|
166
195
|
**v2.0 (Q1 2026):** Desktop GUI, standalone executables, WCAG AA accessibility, French UI, batch validation, core hardening
|
|
167
196
|
|
|
168
|
-
**v2.1 (Q1 2026)
|
|
197
|
+
**v2.1 (Q1 2026):** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
|
|
198
|
+
|
|
199
|
+
**v2.2 (Q3 2026):** Choice of spaCy model, entity retyping in CLI validation, entity-span accuracy fixes, fresh-install fix (SQLAlchemy < 2.1)
|
|
200
|
+
|
|
201
|
+
**v2.3 (Q4 2026) — CURRENT RELEASE:** Detection precision (Epic 10), benchmark repair and blind final exam, names added or corrected replaced at every place, custom pseudonyms used, complete erasure with no name in the erasure log, safer batch runs
|
|
169
202
|
|
|
170
203
|
**v3.0 (2027+):** NLP accuracy & automation
|
|
171
|
-
- Fine-tuned French NER model (70-85% F1 target, up from
|
|
204
|
+
- Fine-tuned French NER model (70-85% F1 target, up from 61.4% on unseen documents today)
|
|
172
205
|
- Optional `--no-validate` flag for high-confidence workflows
|
|
173
206
|
- Confidence-based auto-processing (85%+ F1 target)
|
|
174
207
|
- Multi-language support (English, Spanish, German)
|
|
@@ -180,7 +213,7 @@ pip install gdpr-pseudonymizer[formats]
|
|
|
180
213
|
See [Installation Guide](https://liochandayo.github.io/RGPDpseudonymizer/installation/) for detailed platform-specific instructions.
|
|
181
214
|
|
|
182
215
|
### Prerequisites
|
|
183
|
-
- **Python 3.
|
|
216
|
+
- **Python 3.11, 3.12 or 3.13** (validated in CI/CD — 3.14+ not yet tested)
|
|
184
217
|
|
|
185
218
|
### Install from PyPI (Recommended)
|
|
186
219
|
|
|
@@ -317,6 +350,20 @@ GDPR_PSEUDO_LANG=fr gdpr-pseudo --help
|
|
|
317
350
|
|
|
318
351
|
## 🔬 Technical Details
|
|
319
352
|
|
|
353
|
+
### Detection accuracy (Epic 10, October 2026)
|
|
354
|
+
|
|
355
|
+
**On unseen documents: F1 61.4%** (95% range 59.3–63.6%), recall 60.6%, precision 62.3%. This is a blind final exam: 30 French documents written by a different author in messy formats (email threads with quoted replies, PDF-style line breaks, tables flattened to text, mixed case), scored once (CI run 38382553368). By type: PERSON 69.2% F1, LOCATION 68.7%, ORG 34.8%; organisations remain the weak point.
|
|
356
|
+
|
|
357
|
+
**For context:** 82.6% F1 on the development corpus (25 documents) and 78.1% on a held-out set (6 documents) (CI run 38212303870). Both are closer to the documents the rules were built on.
|
|
358
|
+
|
|
359
|
+
How the final exam was made, and its limits:
|
|
360
|
+
- An LLM (GPT) wrote the 30 documents from a brief that named the hard cases to include but quoted no corpus text.
|
|
361
|
+
- Two models (GPT and Gemini) annotated them independently, and the maintainer settled their disagreements.
|
|
362
|
+
- The set was frozen before the last detection change, then scored once.
|
|
363
|
+
- The documents are synthetic, so real documents may score differently.
|
|
364
|
+
|
|
365
|
+
Details: [docs/qa/ner-accuracy-report.md](docs/qa/ner-accuracy-report.md).
|
|
366
|
+
|
|
320
367
|
### NLP Library Selection (Story 1.2 - Completed)
|
|
321
368
|
|
|
322
369
|
After comprehensive benchmarking on 25 French interview/business documents (1,737 annotated entities):
|
|
@@ -324,11 +371,18 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
|
|
|
324
371
|
| Approach | F1 Score | Precision | Recall | Notes |
|
|
325
372
|
|----------|----------|-----------|--------|-------|
|
|
326
373
|
| **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
|
|
327
|
-
| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
|
|
328
|
-
| **Hybrid
|
|
329
|
-
|
|
374
|
+
| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 (see scoring note) |
|
|
375
|
+
| **Hybrid (v2.2.0)** | **58.53%** | 48.42% | 73.98% | Scored the way the app normalizes entities |
|
|
376
|
+
|
|
377
|
+
By entity type (v2.2.0): PERSON 72.9% F1, LOCATION 45.0%, ORG 12.9%. Organisation detection is the weak point: job titles such as CTO, DPO or RSSI are often mistaken for companies.
|
|
378
|
+
|
|
379
|
+
**How these numbers are scored (corrected October 2026).** The app strips titles and prepositions before pseudonymizing ("Mme Isabelle Moreau" is handled as "Isabelle Moreau", "à Paris" as "Paris"), and the benchmark now scores detections the same way. Until then the committed scorer compared raw text, so every correctly detected "Mme Isabelle Moreau" counted as a miss plus a false alarm. Under that strict scoring the same detector measured ~32% F1, a figure briefly published on this page in October 2026; it understated real-use detection. The Story 5.3 figure (59.97%) appears to have been measured with title-aware matching and is consistent with v2.2.0's 58.53%. Each number in the table above was measured with `pytest tests/accuracy -m accuracy -s` on the version it names.
|
|
380
|
+
|
|
381
|
+
**Read these numbers with two caveats.** They come from one 25-document French interview/business corpus, and part of its ground truth was auto-annotated with the tool's own regex patterns, which skews results in both directions. And they measure *pre-detection* only: mandatory validation is what makes the final output correct.
|
|
382
|
+
|
|
383
|
+
**Epic 10 (October 2026)** repaired that benchmark against written annotation guidelines, added a held-out set and the blind final exam above, and improved organisation detection. A fine-tuned French NER model remains the v3.0 target (70-85% F1).
|
|
330
384
|
|
|
331
|
-
**Accuracy trajectory:**
|
|
385
|
+
**Accuracy trajectory:** Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
|
|
332
386
|
|
|
333
387
|
**Approved Solution:**
|
|
334
388
|
- ✅ **Hybrid approach** (NLP + regex + geography dictionary + POS disambiguation)
|
|
@@ -368,12 +422,12 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
368
422
|
|
|
369
423
|
| Component | Technology | Version | Purpose |
|
|
370
424
|
|-----------|------------|---------|---------|
|
|
371
|
-
| **Runtime** | Python | 3.
|
|
425
|
+
| **Runtime** | Python | 3.11-3.13 | Validated in CI/CD (3.14+ not yet tested) |
|
|
372
426
|
| **NLP Library** | spaCy | 3.8.0 | French entity detection (fr_core_news_lg) |
|
|
373
427
|
| **CLI Framework** | Typer | 0.9+ | Command-line interface |
|
|
374
428
|
| **Database** | SQLite | 3.35+ | Local mapping table storage with WAL mode |
|
|
375
429
|
| **Encryption** | cryptography (AESSIV) | 44.0+ | AES-256-SIV encryption for sensitive fields (PBKDF2 key derivation, passphrase-protected) |
|
|
376
|
-
| **ORM** | SQLAlchemy | 2.0
|
|
430
|
+
| **ORM** | SQLAlchemy | 2.0.x (<2.1) | Database abstraction and session management |
|
|
377
431
|
| **Desktop GUI** | PySide6 | 6.7+ | Desktop application (optional: `pip install gdpr-pseudonymizer[gui]`) |
|
|
378
432
|
| **Validation UI** | rich | 13.7+ | Interactive CLI entity review |
|
|
379
433
|
| **Keyboard Input** | readchar | 4.2+ | Single-keypress capture for validation UI |
|
|
@@ -389,7 +443,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
389
443
|
**Long answer:**
|
|
390
444
|
1. **GDPR defensibility** - Human verification provides legal audit trail
|
|
391
445
|
2. **Zero false negatives** - AI misses entities, humans catch them (100% coverage)
|
|
392
|
-
3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach
|
|
446
|
+
3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach: F1 61.4% and recall 60.6% on unseen documents, 82.6% F1 on our development corpus; organisations are the weak point; improvement planned)
|
|
393
447
|
4. **Better than alternatives:**
|
|
394
448
|
- ✅ **vs Manual redaction:** 50%+ faster (AI pre-detection)
|
|
395
449
|
- ✅ **vs Cloud services:** 100% local processing (no data leakage)
|
|
@@ -475,13 +529,13 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
475
529
|
|
|
476
530
|
## 🛠️ Development Status
|
|
477
531
|
|
|
478
|
-
**Epics 1-7 Complete** — v2.
|
|
532
|
+
**Epics 1-7 and 10 Complete** — current release v2.3.0 (November 2026) closed Epic 10: detection precision, corrections at every place, complete erasure. v2.1.0 (March 2026) closed Epic 7.
|
|
479
533
|
|
|
480
534
|
- ✅ **Epic 1:** Foundation & NLP Validation (9 stories) — spaCy integration, validation UI, hybrid detection, entity deduplication
|
|
481
535
|
- ✅ **Epic 2:** Core Pseudonymization Engine (9 stories) — pseudonym libraries, encryption, audit logging, batch processing, GDPR 1:1 mapping
|
|
482
536
|
- ✅ **Epic 3:** CLI Interface & Batch Processing (7 stories) — 8 CLI commands, progress reporting, config files, parallel batch, UX polish
|
|
483
537
|
- ✅ **Epic 4:** Launch Readiness (8 stories) — LLM utility validation, cross-platform testing, documentation, NER accuracy suite, performance validation, beta feedback integration, codebase refactoring, launch preparation
|
|
484
|
-
- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy
|
|
538
|
+
- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy work (annotation cleanup, expanded regex, geography dictionary), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
|
|
485
539
|
- ✅ **Epic 6:** v2.0 Desktop GUI & Broader Accessibility (9 stories) — PySide6 desktop application, visual entity validation, batch GUI, i18n, WCAG AA, standalone executables
|
|
486
540
|
- ✅ Story 6.1: UX Architecture & GUI Framework Selection
|
|
487
541
|
- ✅ Story 6.2: GUI Application Foundation (main window, theming, home screen, settings, 77 GUI tests)
|
|
@@ -503,6 +557,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
|
|
|
503
557
|
- ✅ Story 7.5: NER Accuracy Regex Expansion & POS Disambiguation
|
|
504
558
|
- ✅ Story 7.6: Quality & Compatibility Integration Tests
|
|
505
559
|
- ✅ Story 7.7: v2.1 Release Preparation
|
|
560
|
+
- ✅ **Epic 10:** v2.3 Detection Precision Quick Wins — benchmark repair and held-out set, blind final exam with confidence ranges, overlap and boundary fixes, fewer false places, greetings and organisation-plus-country handling, type-aware mappings, names added or corrected replaced at every place, custom pseudonyms, erasure log without names, batch fixes, v2.3 release
|
|
506
561
|
- **Total:** 60 stories, 1670+ tests, 86%+ coverage, all quality gates green
|
|
507
562
|
|
|
508
563
|
---
|
|
@@ -565,7 +620,7 @@ This project is licensed under the [MIT License](LICENSE).
|
|
|
565
620
|
- ⚠️ Test thoroughly before production use
|
|
566
621
|
|
|
567
622
|
**Current limitations:**
|
|
568
|
-
- AI detection:
|
|
623
|
+
- AI detection: F1 61.4% on unseen documents (82.6% on our development corpus), not 85%+; improving it is planned
|
|
569
624
|
- Validation required for ALL documents (not optional)
|
|
570
625
|
- French documents only (English, Spanish, etc. in future versions)
|
|
571
626
|
- Text-based formats: .txt, .md, .pdf, .docx, .xlsx, .csv (PDF/DOCX/Excel require optional extras: `pip install gdpr-pseudonymizer[formats]`)
|
|
@@ -653,7 +708,7 @@ poetry run pytest tests/integration/test_validation_workflow_integration.py -v
|
|
|
653
708
|
- **Performance tests:** 19 tests validating all NFR targets — single-document benchmarks (NFR1), entity-detection benchmarks, batch performance (NFR2), memory profiling (NFR4), startup time (NFR5), stability/error rate (NFR6), stress testing (Story 4.5)
|
|
654
709
|
- **Current coverage:** 86%+ across all modules (100% for progress module, 91.41% for AuditRepository)
|
|
655
710
|
- **Total tests:** 1670+ tests
|
|
656
|
-
- **CI/CD:** Tests run on Python 3.
|
|
711
|
+
- **CI/CD:** Tests run on Python 3.11-3.13 across Windows, macOS, and Linux
|
|
657
712
|
- **Quality gates:** All pass (Black, Ruff, mypy, pytest)
|
|
658
713
|
|
|
659
714
|
### Key Integration Test Scenarios
|
|
@@ -686,7 +741,7 @@ The integration test suite covers:
|
|
|
686
741
|
|
|
687
742
|
| Metric | Value | Status |
|
|
688
743
|
|--------|-------|--------|
|
|
689
|
-
| **Development Progress** | v2.
|
|
744
|
+
| **Development Progress** | v2.3.0 | ✅ Epics 1-7 and 10 complete |
|
|
690
745
|
| **Stories Complete** | 60 (Epics 1-7) | ✅ All epics complete |
|
|
691
746
|
| **LLM Utility (NFR10)** | 4.27/5.0 (85.4%) | ✅ PASSED (threshold: 80%) |
|
|
692
747
|
| **Installation Success (NFR3)** | 87.5% (7/8 platforms) | ✅ PASSED (threshold: 85%) |
|
|
@@ -694,7 +749,7 @@ The integration test suite covers:
|
|
|
694
749
|
| **Critical Bugs Found** | 1 (Story 2.8) | ✅ RESOLVED - Epic 3 Unblocked |
|
|
695
750
|
| **Test Corpus Size** | 25 docs, 1,737 entities | ✅ Complete (post-cleanup) |
|
|
696
751
|
| **NLP Accuracy (Baseline)** | 29.5% F1 (spaCy only) | ✅ Measured (Story 1.2) |
|
|
697
|
-
| **Hybrid Accuracy (NLP+Regex)** |
|
|
752
|
+
| **Hybrid Accuracy (NLP+Regex)** | 61.4% F1 on unseen documents (final exam, CI run 38382553368); 82.6% development corpus, 78.1% held-out (CI run 38212303870) | ⚠️ Below target — improvement planned |
|
|
698
753
|
| **Final Accuracy (AI+Human)** | 100% (validated) | 🎯 By Design |
|
|
699
754
|
| **Pseudonym Libraries** | 3 themes (2,426 names + 240 locations + 588 orgs) | ✅ Stories 2.1, 3.0, 4.6 Complete |
|
|
700
755
|
| **Compositional Matching** | Operational (component reuse + title stripping + compound names) | ✅ Stories 2.2, 2.3 Complete |
|
|
@@ -726,4 +781,4 @@ The integration test suite covers:
|
|
|
726
781
|
|
|
727
782
|
---
|
|
728
783
|
|
|
729
|
-
**Last Updated:** 2026-03
|
|
784
|
+
**Last Updated:** 2026-11-03 (v2.3.0 — detection precision, corrections at every place, complete erasure)
|