gdpr-pseudonymizer 2.2.0__tar.gz → 2.3.0__tar.gz

{gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gdpr-pseudonymizer
-Version: 2.2.0
+Version: 2.3.0
 Summary: CLI and GUI tool for GDPR-compliant pseudonymization of French text documents using NLP-based entity detection and reversible mapping
 License: MIT
 License-File: LICENSE
@@ -63,7 +63,16 @@ Transform sensitive French documents for safe AI analysis with local processing,
 
 ---
 
-## What's New in v2.1
+## What's New in v2.2
+
+- **Choose the spaCy model** — `--model` / `pseudonymization.model` accept any spaCy package name (e.g. `en_core_web_trf` for English); `spacy` still selects the French `fr_core_news_lg`. The model only drives NLP detection: regex patterns, dictionaries and pseudonym libraries stay French. See the [CLI Reference](docs/CLI-REFERENCE.md).
+- **Retype entities during CLI validation (`[T]`)** — Switch a detected entity between PERSON, LOCATION and ORG for every occurrence, instead of rejecting and re-adding it
+- **Cleaner entity spans** — Editing with `[E]` no longer erases neighbouring text; "Last, First" matching now requires a known first name; timestamps and stray punctuation are trimmed from NER span edges; names with non-French letters ("María", "Yılmaz") and hyphenated names after titles are detected whole
+- **Fresh `pip install` fixed** — v2.1.3 crashed on import when pip resolved SQLAlchemy 2.1; SQLAlchemy is now capped below 2.1
+- **Audit log records the real model** — e.g. `fr_core_news_lg-3.8.0` instead of `spacy-unknown`
+
+<details>
+<summary>v2.1 highlights</summary>
 
 - **Validate-once-per-entity** — Accept or reject one entity occurrence and it applies to all same-text occurrences in the document (productivity boost for repeated names)
 - **Excel/CSV Format Support** — Process `.xlsx` and `.csv` files with cell-aware pseudonymization for HR/compliance use cases (`pip install gdpr-pseudonymizer[excel]`)
@@ -71,6 +80,8 @@ Transform sensitive French documents for safe AI analysis with local processing,
 - **NER Accuracy Improvements** — Expanded ORG detection patterns, POS-tag disambiguation for geography matching; LOCATION false-negative rate reduced from 27% to 13%
 - **GUI Discoverability** — F1 keyboard shortcuts help dialog with all shortcut groups, database path persistence across sessions, "Hide confirmed" toggle
 
+</details>
+
 <!-- TODO: Add GUI screenshot -->
 
 **Upgrade:** `pip install --upgrade gdpr-pseudonymizer[gui,excel]`
@@ -85,9 +96,9 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
 
 | Platform | File | Notes |
 |----------|------|-------|
-| **Windows** | `gdpr-pseudonymizer-2.1.3-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
-| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.3-macos-arm64.dmg` | Open DMG, drag to Applications. |
-| **Linux** | `gdpr-pseudonymizer-2.1.3-linux.AppImage` | `chmod +x` then run. |
+| **Windows** | `gdpr-pseudonymizer-2.2.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
+| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.2.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
+| **Linux** | `gdpr-pseudonymizer-2.2.0-linux.AppImage` | `chmod +x` then run. |
 
 > **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
 > Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
@@ -130,7 +141,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
 - ✅ **Zero telemetry** - No analytics, crash reporting, or external communication
 
 ### 🤝 **AI + Human Verification**
-- ✅ **Hybrid detection** - AI pre-detects ~60% of entities (NLP + regex + geography dictionary)
+- ✅ **Hybrid detection** - NLP + regex + geography dictionary: F1 61.4% on 30 unseen documents (CI run 38382553368). Improving this is a priority — see [Technical Details](#-technical-details)
 - ✅ **Mandatory validation** - You review and confirm all entities (ensures 100% accuracy)
 - ✅ **Fast validation UI** - Rich CLI interface with keyboard shortcuts, <2 min per document
 - ✅ **Smart workflow** - Entity-by-type grouping (PERSON → ORG → LOCATION) with context display
@@ -165,7 +176,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
 
 ## 🚀 Quick Start
 
-**Status:** 🎉 **v2.1.0** (March 2026) — GUI Polish, Excel/CSV Support & NER Accuracy
+**Status:** 🎉 **v2.2.0** (September 2026) — spaCy Model Choice, Entity Retyping & Span Fixes
 
 ### Getting Started
 
@@ -187,23 +198,25 @@ pip install gdpr-pseudonymizer[excel]
 pip install gdpr-pseudonymizer[formats]
 ```
 
-### What v2.1 Delivers
+### What v2.2 Delivers
 
 - 🖥️ **Desktop GUI** — Visual entity validation with drag-and-drop, batch dashboard, and database management
-- 📦 **Standalone executables** — Windows .exe, macOS .dmg, Linux AppImage — no Python required
+- 📦 **Standalone executables** — Windows .exe, macOS .dmg (Apple Silicon), Linux AppImage — no Python required
 - ♿ **WCAG 2.1 AA accessibility** — Keyboard navigation, screen reader, high contrast mode
 - 🌐 **French UI** — Complete FR/EN interface with live language switching
-- 🤖 **AI-assisted detection** — Hybrid NLP + regex detects ~60% of entities automatically
+- 🤖 **AI-assisted detection** — Hybrid NLP + regex detection: F1 61.4% (recall 60.6%) on 30 unseen documents by a different author (CI run 38382553368)
 - ✅ **Mandatory human verification** — You review and confirm all entities (ensures 100% accuracy)
 - 🔒 **100% local processing** — Your data never leaves your machine
 - 📄 **PDF/DOCX support** — Process PDF and DOCX files directly (optional extras)
 - 📊 **Excel/CSV support** — Process .xlsx and .csv files with cell-aware pseudonymization (optional extra: `[excel]`)
 - 🆔 **Neutral ID theme** — Counter-based identifiers (PERSON-001, LIEU-001) for formal/legal contexts
 - 🎯 **NER accuracy** — LOCATION false-negative rate reduced from 27% to 13% via regex expansion & POS disambiguation
+- 🧠 **Choice of spaCy model** — French by default; any spaCy model for detection (e.g. `--model en_core_web_trf`)
+- 🏷️ **Entity retyping** — Change an entity's type (PERSON/LOCATION/ORG) during CLI validation with `[T]`
 
-**What v2.1 does NOT deliver:**
+**What v2.2 does NOT deliver:**
 - ❌ Fully automatic "set and forget" processing
-- ❌ 85%+ AI accuracy (current: ~60% F1 with hybrid approach)
+- ❌ 85%+ AI accuracy (current: F1 61.4% on unseen documents, 82.6% on our development corpus — we plan to raise it)
 - ❌ Optional validation mode (validation is mandatory)
 
 ### Roadmap
@@ -214,10 +227,12 @@ pip install gdpr-pseudonymizer[formats]
 
 **v2.0 (Q1 2026):** Desktop GUI, standalone executables, WCAG AA accessibility, French UI, batch validation, core hardening
 
-**v2.1 (Q1 2026) — CURRENT RELEASE:** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
+**v2.1 (Q1 2026):** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
+
+**v2.2 (Q3 2026) — CURRENT RELEASE:** Choice of spaCy model, entity retyping in CLI validation, entity-span accuracy fixes, fresh-install fix (SQLAlchemy < 2.1)
 
 **v3.0 (2027+):** NLP accuracy & automation
-- Fine-tuned French NER model (70-85% F1 target, up from ~60%)
+- Fine-tuned French NER model (70-85% F1 target, up from 61.4% on unseen documents today)
 - Optional `--no-validate` flag for high-confidence workflows
 - Confidence-based auto-processing (85%+ F1 target)
 - Multi-language support (English, Spanish, German)
@@ -366,6 +381,20 @@ GDPR_PSEUDO_LANG=fr gdpr-pseudo --help
 
 ## 🔬 Technical Details
 
+### Detection accuracy (Epic 10, October 2026)
+
+**On unseen documents: F1 61.4%** (95% range 59.3–63.6%), recall 60.6%, precision 62.3%. This is a blind final exam: 30 French documents written by a different author in messy formats (email threads with quoted replies, PDF-style line breaks, tables flattened to text, mixed case), scored once (CI run 38382553368). By type: PERSON 69.2% F1, LOCATION 68.7%, ORG 34.8%; organisations remain the weak point.
+
+**For context:** 82.6% F1 on the development corpus (25 documents) and 78.1% on a held-out set (6 documents) (CI run 38212303870). Both are closer to the documents the rules were built on.
+
+How the final exam was made, and its limits:
+- An LLM (GPT) wrote the 30 documents from a brief that named the hard cases to include but quoted no corpus text.
+- Two models (GPT and Gemini) annotated them independently, and the maintainer settled their disagreements.
+- The set was frozen before the last detection change, then scored once.
+- The documents are synthetic, so real documents may score differently.
+
+Details: [docs/qa/ner-accuracy-report.md](docs/qa/ner-accuracy-report.md).
+
 ### NLP Library Selection (Story 1.2 - Completed)
 
 After comprehensive benchmarking on 25 French interview/business documents (1,737 annotated entities):
@@ -373,11 +402,18 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
 | Approach | F1 Score | Precision | Recall | Notes |
 |----------|----------|-----------|--------|-------|
 | **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
-| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
-| **Hybrid + expanded patterns** | 31.79% | 25.38% | 42.54% | Story 7.5 |
-| **Span-bleed fixes** | 32.34% | 26.75% | 40.88% | 2026-09-29 (current) — recall drop is a ground-truth artefact, see QA report |
+| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 (see scoring note) |
+| **Hybrid (v2.2.0)** | **58.53%** | 48.42% | 73.98% | Scored the way the app normalizes entities |
+
+By entity type (v2.2.0): PERSON 72.9% F1, LOCATION 45.0%, ORG 12.9%. Organisation detection is the weak point: job titles such as CTO, DPO or RSSI are often mistaken for companies.
+
+**How these numbers are scored (corrected October 2026).** The app strips titles and prepositions before pseudonymizing ("Mme Isabelle Moreau" is handled as "Isabelle Moreau", "à Paris" as "Paris"), and the benchmark now scores detections the same way. Until then the committed scorer compared raw text, so every correctly detected "Mme Isabelle Moreau" counted as a miss plus a false alarm. Under that strict scoring the same detector measured ~32% F1, a figure briefly published on this page in October 2026; it understated real-use detection. The Story 5.3 figure (59.97%) appears to have been measured with title-aware matching and is consistent with v2.2.0's 58.53%. Each number in the table above was measured with `pytest tests/accuracy -m accuracy -s` on the version it names.
+
+**Read these numbers with two caveats.** They come from one 25-document French interview/business corpus, and part of its ground truth was auto-annotated with the tool's own regex patterns, which skews results in both directions. And they measure *pre-detection* only: mandatory validation is what makes the final output correct.
+
+**Epic 10 (October 2026)** repaired that benchmark against written annotation guidelines, added a held-out set and the blind final exam above, and improved organisation detection. A fine-tuned French NER model remains the v3.0 target (70-85% F1).
 
-**Accuracy trajectory:** spaCy-only baseline → hybrid approach with annotation cleanup, expanded regex patterns, and French geography dictionary doubled F1 score. Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
+**Accuracy trajectory:** Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
 
 **Approved Solution:**
 - ✅ **Hybrid approach** (NLP + regex + geography dictionary + POS disambiguation)
@@ -422,7 +458,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 | **CLI Framework** | Typer | 0.9+ | Command-line interface |
 | **Database** | SQLite | 3.35+ | Local mapping table storage with WAL mode |
 | **Encryption** | cryptography (AESSIV) | 44.0+ | AES-256-SIV encryption for sensitive fields (PBKDF2 key derivation, passphrase-protected) |
-| **ORM** | SQLAlchemy | 2.0+ | Database abstraction and session management |
+| **ORM** | SQLAlchemy | 2.0.x (<2.1) | Database abstraction and session management |
 | **Desktop GUI** | PySide6 | 6.7+ | Desktop application (optional: `pip install gdpr-pseudonymizer[gui]`) |
 | **Validation UI** | rich | 13.7+ | Interactive CLI entity review |
 | **Keyboard Input** | readchar | 4.2+ | Single-keypress capture for validation UI |
@@ -438,7 +474,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 **Long answer:**
 1. **GDPR defensibility** - Human verification provides legal audit trail
 2. **Zero false negatives** - AI misses entities, humans catch them (100% coverage)
-3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach reaches ~60%)
+3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach: F1 61.4% and recall 60.6% on unseen documents, 82.6% F1 on our development corpus; organisations are the weak point; improvement planned)
 4. **Better than alternatives:**
    - ✅ **vs Manual redaction:** 50%+ faster (AI pre-detection)
    - ✅ **vs Cloud services:** 100% local processing (no data leakage)
@@ -524,13 +560,13 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 
 ## 🛠️ Development Status
 
-**Epics 1-7 Complete** — v2.1.0 (March 2026). GUI polish, Excel/CSV support, NER accuracy improvements.
+**Epics 1-7 Complete** — current release v2.2.0 (September 2026): spaCy model choice, entity retyping, entity-span fixes. v2.1.0 (March 2026) closed Epic 7.
 
 - ✅ **Epic 1:** Foundation & NLP Validation (9 stories) — spaCy integration, validation UI, hybrid detection, entity deduplication
 - ✅ **Epic 2:** Core Pseudonymization Engine (9 stories) — pseudonym libraries, encryption, audit logging, batch processing, GDPR 1:1 mapping
 - ✅ **Epic 3:** CLI Interface & Batch Processing (7 stories) — 8 CLI commands, progress reporting, config files, parallel batch, UX polish
 - ✅ **Epic 4:** Launch Readiness (8 stories) — LLM utility validation, cross-platform testing, documentation, NER accuracy suite, performance validation, beta feedback integration, codebase refactoring, launch preparation
-- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy improvements (F1 29.74% → 59.97%), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
+- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy work (annotation cleanup, expanded regex, geography dictionary), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
 - ✅ **Epic 6:** v2.0 Desktop GUI & Broader Accessibility (9 stories) — PySide6 desktop application, visual entity validation, batch GUI, i18n, WCAG AA, standalone executables
   - ✅ Story 6.1: UX Architecture & GUI Framework Selection
   - ✅ Story 6.2: GUI Application Foundation (main window, theming, home screen, settings, 77 GUI tests)
@@ -614,7 +650,7 @@ This project is licensed under the [MIT License](LICENSE).
 - ⚠️ Test thoroughly before production use
 
 **Current limitations:**
-- AI detection: ~60% F1 baseline (not 85%+)
+- AI detection: F1 61.4% on unseen documents (82.6% on our development corpus), not 85%+; improving it is planned
 - Validation required for ALL documents (not optional)
 - French documents only (English, Spanish, etc. in future versions)
 - Text-based formats: .txt, .md, .pdf, .docx, .xlsx, .csv (PDF/DOCX/Excel require optional extras: `pip install gdpr-pseudonymizer[formats]`)
@@ -735,7 +771,7 @@ The integration test suite covers:
 
 | Metric | Value | Status |
 |--------|-------|--------|
-| **Development Progress** | v2.1.0 | ✅ Epics 1-7 complete |
+| **Development Progress** | v2.2.0 | ✅ Epics 1-7 complete |
 | **Stories Complete** | 60 (Epics 1-7) | ✅ All epics complete |
 | **LLM Utility (NFR10)** | 4.27/5.0 (85.4%) | ✅ PASSED (threshold: 80%) |
 | **Installation Success (NFR3)** | 87.5% (7/8 platforms) | ✅ PASSED (threshold: 85%) |
@@ -743,7 +779,7 @@ The integration test suite covers:
 | **Critical Bugs Found** | 1 (Story 2.8) | ✅ RESOLVED - Epic 3 Unblocked |
 | **Test Corpus Size** | 25 docs, 1,737 entities | ✅ Complete (post-cleanup) |
 | **NLP Accuracy (Baseline)** | 29.5% F1 (spaCy only) | ✅ Measured (Story 1.2) |
-| **Hybrid Accuracy (NLP+Regex)** | 59.97% F1 (+30.23pp vs baseline) | ✅ Story 5.3 Complete |
+| **Hybrid Accuracy (NLP+Regex)** | 61.4% F1 on unseen documents (final exam, CI run 38382553368); 82.6% development corpus, 78.1% held-out (CI run 38212303870) | ⚠️ Below target — improvement planned |
 | **Final Accuracy (AI+Human)** | 100% (validated) | 🎯 By Design |
 | **Pseudonym Libraries** | 3 themes (2,426 names + 240 locations + 588 orgs) | ✅ Stories 2.1, 3.0, 4.6 Complete |
 | **Compositional Matching** | Operational (component reuse + title stripping + compound names) | ✅ Stories 2.2, 2.3 Complete |
@@ -775,5 +811,5 @@ The integration test suite covers:
 
 ---
 
-**Last Updated:** 2026-03-17 (v2.1.0 — GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, 1670+ tests)
+**Last Updated:** 2026-10-01 (v2.2.0 — spaCy model choice, entity retyping, entity-span fixes)
 

{gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.0}/README.md

@@ -14,7 +14,16 @@ Transform sensitive French documents for safe AI analysis with local processing,
 
 ---
 
-## What's New in v2.1
+## What's New in v2.2
+
+- **Choose the spaCy model** — `--model` / `pseudonymization.model` accept any spaCy package name (e.g. `en_core_web_trf` for English); `spacy` still selects the French `fr_core_news_lg`. The model only drives NLP detection: regex patterns, dictionaries and pseudonym libraries stay French. See the [CLI Reference](docs/CLI-REFERENCE.md).
+- **Retype entities during CLI validation (`[T]`)** — Switch a detected entity between PERSON, LOCATION and ORG for every occurrence, instead of rejecting and re-adding it
+- **Cleaner entity spans** — Editing with `[E]` no longer erases neighbouring text; "Last, First" matching now requires a known first name; timestamps and stray punctuation are trimmed from NER span edges; names with non-French letters ("María", "Yılmaz") and hyphenated names after titles are detected whole
+- **Fresh `pip install` fixed** — v2.1.3 crashed on import when pip resolved SQLAlchemy 2.1; SQLAlchemy is now capped below 2.1
+- **Audit log records the real model** — e.g. `fr_core_news_lg-3.8.0` instead of `spacy-unknown`
+
+<details>
+<summary>v2.1 highlights</summary>
 
 - **Validate-once-per-entity** — Accept or reject one entity occurrence and it applies to all same-text occurrences in the document (productivity boost for repeated names)
 - **Excel/CSV Format Support** — Process `.xlsx` and `.csv` files with cell-aware pseudonymization for HR/compliance use cases (`pip install gdpr-pseudonymizer[excel]`)
@@ -22,6 +31,8 @@ Transform sensitive French documents for safe AI analysis with local processing,
 - **NER Accuracy Improvements** — Expanded ORG detection patterns, POS-tag disambiguation for geography matching; LOCATION false-negative rate reduced from 27% to 13%
 - **GUI Discoverability** — F1 keyboard shortcuts help dialog with all shortcut groups, database path persistence across sessions, "Hide confirmed" toggle
 
+</details>
+
 <!-- TODO: Add GUI screenshot -->
 
 **Upgrade:** `pip install --upgrade gdpr-pseudonymizer[gui,excel]`
@@ -36,9 +47,9 @@ Pre-built standalone executables are available for Windows, macOS, and Linux. No
 
 | Platform | File | Notes |
 |----------|------|-------|
-| **Windows** | `gdpr-pseudonymizer-2.1.3-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
-| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.1.3-macos-arm64.dmg` | Open DMG, drag to Applications. |
-| **Linux** | `gdpr-pseudonymizer-2.1.3-linux.AppImage` | `chmod +x` then run. |
+| **Windows** | `gdpr-pseudonymizer-2.2.0-windows-setup.exe` | Run the installer. Adds Start Menu shortcut. |
+| **macOS (Apple Silicon)** | `gdpr-pseudonymizer-2.2.0-macos-arm64.dmg` | Open DMG, drag to Applications. |
+| **Linux** | `gdpr-pseudonymizer-2.2.0-linux.AppImage` | `chmod +x` then run. |
 
 > **Intel Macs:** no DMG is published. GitHub retired its Intel macOS runners, and an
 > Apple Silicon runner cannot cross-build a working x86_64 bundle. Install with
@@ -81,7 +92,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
 - ✅ **Zero telemetry** - No analytics, crash reporting, or external communication
 
 ### 🤝 **AI + Human Verification**
-- ✅ **Hybrid detection** - AI pre-detects ~60% of entities (NLP + regex + geography dictionary)
+- ✅ **Hybrid detection** - NLP + regex + geography dictionary: F1 61.4% on 30 unseen documents (CI run 38382553368). Improving this is a priority — see [Technical Details](#-technical-details)
 - ✅ **Mandatory validation** - You review and confirm all entities (ensures 100% accuracy)
 - ✅ **Fast validation UI** - Rich CLI interface with keyboard shortcuts, <2 min per document
 - ✅ **Smart workflow** - Entity-by-type grouping (PERSON → ORG → LOCATION) with context display
@@ -116,7 +127,7 @@ GDPR Pseudonymizer is a **privacy-first CLI and GUI tool** that combines AI effi
 
 ## 🚀 Quick Start
 
-**Status:** 🎉 **v2.1.0** (March 2026) — GUI Polish, Excel/CSV Support & NER Accuracy
+**Status:** 🎉 **v2.2.0** (September 2026) — spaCy Model Choice, Entity Retyping & Span Fixes
 
 ### Getting Started
 
@@ -138,23 +149,25 @@ pip install gdpr-pseudonymizer[excel]
 pip install gdpr-pseudonymizer[formats]
 ```
 
-### What v2.1 Delivers
+### What v2.2 Delivers
 
 - 🖥️ **Desktop GUI** — Visual entity validation with drag-and-drop, batch dashboard, and database management
-- 📦 **Standalone executables** — Windows .exe, macOS .dmg, Linux AppImage — no Python required
+- 📦 **Standalone executables** — Windows .exe, macOS .dmg (Apple Silicon), Linux AppImage — no Python required
 - ♿ **WCAG 2.1 AA accessibility** — Keyboard navigation, screen reader, high contrast mode
 - 🌐 **French UI** — Complete FR/EN interface with live language switching
-- 🤖 **AI-assisted detection** — Hybrid NLP + regex detects ~60% of entities automatically
+- 🤖 **AI-assisted detection** — Hybrid NLP + regex detection: F1 61.4% (recall 60.6%) on 30 unseen documents by a different author (CI run 38382553368)
 - ✅ **Mandatory human verification** — You review and confirm all entities (ensures 100% accuracy)
 - 🔒 **100% local processing** — Your data never leaves your machine
 - 📄 **PDF/DOCX support** — Process PDF and DOCX files directly (optional extras)
 - 📊 **Excel/CSV support** — Process .xlsx and .csv files with cell-aware pseudonymization (optional extra: `[excel]`)
 - 🆔 **Neutral ID theme** — Counter-based identifiers (PERSON-001, LIEU-001) for formal/legal contexts
 - 🎯 **NER accuracy** — LOCATION false-negative rate reduced from 27% to 13% via regex expansion & POS disambiguation
+- 🧠 **Choice of spaCy model** — French by default; any spaCy model for detection (e.g. `--model en_core_web_trf`)
+- 🏷️ **Entity retyping** — Change an entity's type (PERSON/LOCATION/ORG) during CLI validation with `[T]`
 
-**What v2.1 does NOT deliver:**
+**What v2.2 does NOT deliver:**
 - ❌ Fully automatic "set and forget" processing
-- ❌ 85%+ AI accuracy (current: ~60% F1 with hybrid approach)
+- ❌ 85%+ AI accuracy (current: F1 61.4% on unseen documents, 82.6% on our development corpus — we plan to raise it)
 - ❌ Optional validation mode (validation is mandatory)
 
 ### Roadmap
@@ -165,10 +178,12 @@ pip install gdpr-pseudonymizer[formats]
 
 **v2.0 (Q1 2026):** Desktop GUI, standalone executables, WCAG AA accessibility, French UI, batch validation, core hardening
 
-**v2.1 (Q1 2026) — CURRENT RELEASE:** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
+**v2.1 (Q1 2026):** GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, keyboard shortcuts help
+
+**v2.2 (Q3 2026) — CURRENT RELEASE:** Choice of spaCy model, entity retyping in CLI validation, entity-span accuracy fixes, fresh-install fix (SQLAlchemy < 2.1)
 
 **v3.0 (2027+):** NLP accuracy & automation
-- Fine-tuned French NER model (70-85% F1 target, up from ~60%)
+- Fine-tuned French NER model (70-85% F1 target, up from 61.4% on unseen documents today)
 - Optional `--no-validate` flag for high-confidence workflows
 - Confidence-based auto-processing (85%+ F1 target)
 - Multi-language support (English, Spanish, German)
@@ -317,6 +332,20 @@ GDPR_PSEUDO_LANG=fr gdpr-pseudo --help
 
 ## 🔬 Technical Details
 
+### Detection accuracy (Epic 10, October 2026)
+
+**On unseen documents: F1 61.4%** (95% range 59.3–63.6%), recall 60.6%, precision 62.3%. This is a blind final exam: 30 French documents written by a different author in messy formats (email threads with quoted replies, PDF-style line breaks, tables flattened to text, mixed case), scored once (CI run 38382553368). By type: PERSON 69.2% F1, LOCATION 68.7%, ORG 34.8%; organisations remain the weak point.
+
+**For context:** 82.6% F1 on the development corpus (25 documents) and 78.1% on a held-out set (6 documents) (CI run 38212303870). Both are closer to the documents the rules were built on.
+
+How the final exam was made, and its limits:
+- An LLM (GPT) wrote the 30 documents from a brief that named the hard cases to include but quoted no corpus text.
+- Two models (GPT and Gemini) annotated them independently, and the maintainer settled their disagreements.
+- The set was frozen before the last detection change, then scored once.
+- The documents are synthetic, so real documents may score differently.
+
+Details: [docs/qa/ner-accuracy-report.md](docs/qa/ner-accuracy-report.md).
+
 ### NLP Library Selection (Story 1.2 - Completed)
 
 After comprehensive benchmarking on 25 French interview/business documents (1,737 annotated entities):
@@ -324,11 +353,18 @@ After comprehensive benchmarking on 25 French interview/business documents (1,73
 | Approach | F1 Score | Precision | Recall | Notes |
 |----------|----------|-----------|--------|-------|
 | **spaCy only** `fr_core_news_lg` | 29.5% | 27.0% | 32.7% | Story 1.2 baseline |
-| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 |
-| **Hybrid + expanded patterns** | 31.79% | 25.38% | 42.54% | Story 7.5 |
-| **Span-bleed fixes** | 32.34% | 26.75% | 40.88% | 2026-09-29 (current) — recall drop is a ground-truth artefact, see QA report |
+| **Hybrid** (spaCy + regex) | 59.97% | 48.17% | 79.45% | Story 5.3 (see scoring note) |
+| **Hybrid (v2.2.0)** | **58.53%** | 48.42% | 73.98% | Scored the way the app normalizes entities |
+
+By entity type (v2.2.0): PERSON 72.9% F1, LOCATION 45.0%, ORG 12.9%. Organisation detection is the weak point: job titles such as CTO, DPO or RSSI are often mistaken for companies.
+
+**How these numbers are scored (corrected October 2026).** The app strips titles and prepositions before pseudonymizing ("Mme Isabelle Moreau" is handled as "Isabelle Moreau", "à Paris" as "Paris"), and the benchmark now scores detections the same way. Until then the committed scorer compared raw text, so every correctly detected "Mme Isabelle Moreau" counted as a miss plus a false alarm. Under that strict scoring the same detector measured ~32% F1, a figure briefly published on this page in October 2026; it understated real-use detection. The Story 5.3 figure (59.97%) appears to have been measured with title-aware matching and is consistent with v2.2.0's 58.53%. Each number in the table above was measured with `pytest tests/accuracy -m accuracy -s` on the version it names.
+
+**Read these numbers with two caveats.** They come from one 25-document French interview/business corpus, and part of its ground truth was auto-annotated with the tool's own regex patterns, which skews results in both directions. And they measure *pre-detection* only: mandatory validation is what makes the final output correct.
+
+**Epic 10 (October 2026)** repaired that benchmark against written annotation guidelines, added a held-out set and the blind final exam above, and improved organisation detection. A fine-tuned French NER model remains the v3.0 target (70-85% F1).
 
-**Accuracy trajectory:** spaCy-only baseline → hybrid approach with annotation cleanup, expanded regex patterns, and French geography dictionary doubled F1 score. Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
+**Accuracy trajectory:** Story 7.5 added 12 ORG pattern keywords, POS-tag disambiguation for geography matching, and 7 international locations — reducing LOCATION false-negative rate from 27.42% to 12.90%.
 
 **Approved Solution:**
 - ✅ **Hybrid approach** (NLP + regex + geography dictionary + POS disambiguation)
@@ -373,7 +409,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 | **CLI Framework** | Typer | 0.9+ | Command-line interface |
 | **Database** | SQLite | 3.35+ | Local mapping table storage with WAL mode |
 | **Encryption** | cryptography (AESSIV) | 44.0+ | AES-256-SIV encryption for sensitive fields (PBKDF2 key derivation, passphrase-protected) |
-| **ORM** | SQLAlchemy | 2.0+ | Database abstraction and session management |
+| **ORM** | SQLAlchemy | 2.0.x (<2.1) | Database abstraction and session management |
 | **Desktop GUI** | PySide6 | 6.7+ | Desktop application (optional: `pip install gdpr-pseudonymizer[gui]`) |
 | **Validation UI** | rich | 13.7+ | Interactive CLI entity review |
 | **Keyboard Input** | readchar | 4.2+ | Single-keypress capture for validation UI |
@@ -389,7 +425,7 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 **Long answer:**
 1. **GDPR defensibility** - Human verification provides legal audit trail
 2. **Zero false negatives** - AI misses entities, humans catch them (100% coverage)
-3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach reaches ~60%)
+3. **Current NLP limitations** - French models on interview/business docs: 29.5% F1 out-of-box (hybrid approach: F1 61.4% and recall 60.6% on unseen documents, 82.6% F1 on our development corpus; organisations are the weak point; improvement planned)
 4. **Better than alternatives:**
    - ✅ **vs Manual redaction:** 50%+ faster (AI pre-detection)
    - ✅ **vs Cloud services:** 100% local processing (no data leakage)
@@ -475,13 +511,13 @@ The validation UI provides an intuitive keyboard-driven interface for reviewing
 
 ## 🛠️ Development Status
 
-**Epics 1-7 Complete** — v2.1.0 (March 2026). GUI polish, Excel/CSV support, NER accuracy improvements.
+**Epics 1-7 Complete** — current release v2.2.0 (September 2026): spaCy model choice, entity retyping, entity-span fixes. v2.1.0 (March 2026) closed Epic 7.
 
 - ✅ **Epic 1:** Foundation & NLP Validation (9 stories) — spaCy integration, validation UI, hybrid detection, entity deduplication
 - ✅ **Epic 2:** Core Pseudonymization Engine (9 stories) — pseudonym libraries, encryption, audit logging, batch processing, GDPR 1:1 mapping
 - ✅ **Epic 3:** CLI Interface & Batch Processing (7 stories) — 8 CLI commands, progress reporting, config files, parallel batch, UX polish
 - ✅ **Epic 4:** Launch Readiness (8 stories) — LLM utility validation, cross-platform testing, documentation, NER accuracy suite, performance validation, beta feedback integration, codebase refactoring, launch preparation
-- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy improvements (F1 29.74% → 59.97%), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
+- ✅ **Epic 5:** Quick Wins & GDPR Compliance (7 stories) — GDPR Article 17 erasure, gender-aware pseudonyms, NER accuracy work (annotation cleanup, expanded regex, geography dictionary), French documentation translation, PDF/DOCX support, CLI polish & benchmarks, v1.1 release
 - ✅ **Epic 6:** v2.0 Desktop GUI & Broader Accessibility (9 stories) — PySide6 desktop application, visual entity validation, batch GUI, i18n, WCAG AA, standalone executables
   - ✅ Story 6.1: UX Architecture & GUI Framework Selection
   - ✅ Story 6.2: GUI Application Foundation (main window, theming, home screen, settings, 77 GUI tests)
@@ -565,7 +601,7 @@ This project is licensed under the [MIT License](LICENSE).
 - ⚠️ Test thoroughly before production use
 
 **Current limitations:**
-- AI detection: ~60% F1 baseline (not 85%+)
+- AI detection: F1 61.4% on unseen documents (82.6% on our development corpus), not 85%+; improving it is planned
 - Validation required for ALL documents (not optional)
 - French documents only (English, Spanish, etc. in future versions)
 - Text-based formats: .txt, .md, .pdf, .docx, .xlsx, .csv (PDF/DOCX/Excel require optional extras: `pip install gdpr-pseudonymizer[formats]`)
@@ -686,7 +722,7 @@ The integration test suite covers:
 
 | Metric | Value | Status |
 |--------|-------|--------|
-| **Development Progress** | v2.1.0 | ✅ Epics 1-7 complete |
+| **Development Progress** | v2.2.0 | ✅ Epics 1-7 complete |
 | **Stories Complete** | 60 (Epics 1-7) | ✅ All epics complete |
 | **LLM Utility (NFR10)** | 4.27/5.0 (85.4%) | ✅ PASSED (threshold: 80%) |
 | **Installation Success (NFR3)** | 87.5% (7/8 platforms) | ✅ PASSED (threshold: 85%) |
@@ -694,7 +730,7 @@ The integration test suite covers:
 | **Critical Bugs Found** | 1 (Story 2.8) | ✅ RESOLVED - Epic 3 Unblocked |
 | **Test Corpus Size** | 25 docs, 1,737 entities | ✅ Complete (post-cleanup) |
 | **NLP Accuracy (Baseline)** | 29.5% F1 (spaCy only) | ✅ Measured (Story 1.2) |
-| **Hybrid Accuracy (NLP+Regex)** | 59.97% F1 (+30.23pp vs baseline) | ✅ Story 5.3 Complete |
+| **Hybrid Accuracy (NLP+Regex)** | 61.4% F1 on unseen documents (final exam, CI run 38382553368); 82.6% development corpus, 78.1% held-out (CI run 38212303870) | ⚠️ Below target — improvement planned |
 | **Final Accuracy (AI+Human)** | 100% (validated) | 🎯 By Design |
 | **Pseudonym Libraries** | 3 themes (2,426 names + 240 locations + 588 orgs) | ✅ Stories 2.1, 3.0, 4.6 Complete |
 | **Compositional Matching** | Operational (component reuse + title stripping + compound names) | ✅ Stories 2.2, 2.3 Complete |
@@ -726,4 +762,4 @@ The integration test suite covers:
 
 ---
 
-**Last Updated:** 2026-03-17 (v2.1.0 — GUI polish, Excel/CSV support, neutral ID theme, NER accuracy improvements, 1670+ tests)
+**Last Updated:** 2026-10-01 (v2.2.0 — spaCy model choice, entity retyping, entity-span fixes)

{gdpr_pseudonymizer-2.2.0 → gdpr_pseudonymizer-2.3.0}/gdpr_pseudonymizer/cli/commands/batch.py

@@ -41,6 +41,8 @@ from gdpr_pseudonymizer.cli.validators import (
     validate_theme_or_exit,
 )
 from gdpr_pseudonymizer.core.document_processor import DocumentProcessor
+from gdpr_pseudonymizer.data.database import open_database
+from gdpr_pseudonymizer.exceptions import DatabaseConversionError
 from gdpr_pseudonymizer.utils.logger import configure_logging, get_logger
 
 # Configure logging
@@ -451,6 +453,10 @@ def batch_command(
         # Initialize database if needed
         ensure_database(effective_db_path, resolved_passphrase, console)
 
+        # Open it once before any document: a pre-10.7 file is converted here,
+        # once, before any worker opens it (Story 10.7, AC3)
+        _convert_database_before_workers(effective_db_path, resolved_passphrase)
+
         # Determine processing mode based on workers parameter
         actual_workers = min(cpu_count(), effective_workers, 8)
         actual_workers = max(1, actual_workers)
@@ -714,6 +720,35 @@ def batch_command(
         sys.exit(2)
 
 
+def _convert_database_before_workers(db_path: str, passphrase: str) -> None:
+    """Open the database once before any document is processed (Story 10.7).
+
+    A database made before Story 10.7 is converted by its first open, so it
+    is converted here, once, before the workers start. A conversion that
+    cannot be made stops the run before any document, with its message. Any
+    other open error is left to the per-document processing, as before.
+
+    Args:
+        db_path: Path to the database (created by ensure_database if missing)
+        passphrase: Database passphrase
+    """
+    if not Path(db_path).exists():
+        return
+    try:
+        with open_database(db_path, passphrase):
+            pass
+    except DatabaseConversionError as e:
+        format_error_message(
+            "Database Conversion Failed",
+            str(e),
+            "Make the database file and its folder writable, free some disk "
+            "space or close other programs using it, then run the command again.",
+        )
+        sys.exit(1)
+    except Exception as e:
+        logger.info("batch_database_precheck_skipped", error_type=type(e).__name__)
+
+
 def _display_batch_summary(result: BatchResult) -> None:
     """Display batch processing summary report.