pseudonymize 0.6.0__tar.gz → 0.7.0__tar.gz

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/.agents/PRIVATE_RELEASE_PLAN.md

@@ -192,16 +192,16 @@ Starting commit: 6f6525bb1689a8b8ea5cbf7473a57d360dc339f7
 Starting PyPI version: 0.5.1
 Baseline downloads day/week/month: 0/0/0
 Baseline stars/forks/watchers: 0/0/0
-Files changed: 
-Compatibility tests added: 
-Artifact smoke environments: 
-Known limitations: 
-Deferred items: 
-Release commit: 
-Tag: 
-PyPI upload verified: 
+Files changed: 7
+Compatibility tests added: Yes
+Artifact smoke environments: Win local isolated wheel
+Known limitations: Remote processing is restricted to allowed/configured backends.
+Deferred items: None
+Release commit: f9517983550048983640cf4fe0e1680fcb6da0b5
+Tag: v0.6.0
+PyPI upload verified: Yes
 7-day metrics: 
 28-day metrics: 
-Status: In progress
+Status: Released
 
 

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/.gitignore

@@ -177,6 +177,7 @@ dmypy.json
 
 # test temp files
 .pytest_temp/
+.cache/
 *.onnx
 
 # Pyre type checker

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-08-28
+
+### Added
+
+- Added a new quality evaluation suite (enchmarks/evaluate_quality.py) capable of streaming real-world datasets (like AI4Privacy/PII-Masking-200k) to establish rigorous precision, recall, and F1 baselines for core detectors and ML backends.
+
+## [0.6.1] - 2026-08-28
+
+### Added
+
+- Generic `NATIONAL_ID` and `TAX_ID` entity types with checksum-validated Italian fiscal-code and
+  VAT-number detectors. Unprefixed VAT numbers require an explicit tax label.
+
+### Fixed
+
+- ONNX test artifacts now persist outside pytest's temporary directory, avoiding a verified
+  model download on every local test invocation.
+- `inspect-file` now serializes coordinate and structural locations used by PDF, image, and
+  Office inspection instead of failing after a detection is found.
+- IBAN detection now accepts spaces between the country code and check digits, including values
+  extracted from visually grouped documents.
+- PDF processing now preserves unchanged text blocks and renders transformed blocks legibly
+  instead of covering the complete document text with black redaction rectangles.
+- PDF processing now localizes changed spans inside text blocks when possible, preserving the
+  surrounding text's original visual representation while securely removing detected values.
+- PDF redaction now preserves existing colored backgrounds and source text colors instead of
+  placing white rectangles with mismatched replacement text over styled fields.
+
 ## [0.6.0] - 2026-08-25
 
 ### Added

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pseudonymize
-Version: 0.6.0
+Version: 0.7.0
 Summary: Local-first PII pseudonymization for text, structured data, and LLM payloads.
 Project-URL: Homepage, https://github.com/ma2za/pseudonymize
 Project-URL: Documentation, https://ma2za.github.io/pseudonymize/
@@ -88,7 +88,7 @@ no telemetry or model downloads, and denies remote-capable backends by default.
 | --- | --- | --- |
 | Strings | Shipped | Pseudonymization, redaction, batch processing, and safe reports |
 | Nested Python data | Shipped | Dictionaries, lists, tuples, JSON scalars, and path policies |
-| Structured detection | Shipped | Email, phone, IP, IBAN, payment card, URL credentials, and common secrets |
+| Structured detection | Shipped | Email, phone, IP, IBAN, payment card, Italian fiscal and VAT identifiers, URL credentials, and common secrets |
 | Document representation | Shipped | Immutable blocks, typed locations, sanitized metadata, and inspection |
 | Generic file orchestration | Shipped | Built-in or caller-provided adapters and atomic safe-copy output |
 | TXT, Markdown, log, JSON, JSONL, and CSV files | Shipped | Explicit format or recognized suffix; no content guessing |

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/README.md

@@ -44,7 +44,7 @@ no telemetry or model downloads, and denies remote-capable backends by default.
 | --- | --- | --- |
 | Strings | Shipped | Pseudonymization, redaction, batch processing, and safe reports |
 | Nested Python data | Shipped | Dictionaries, lists, tuples, JSON scalars, and path policies |
-| Structured detection | Shipped | Email, phone, IP, IBAN, payment card, URL credentials, and common secrets |
+| Structured detection | Shipped | Email, phone, IP, IBAN, payment card, Italian fiscal and VAT identifiers, URL credentials, and common secrets |
 | Document representation | Shipped | Immutable blocks, typed locations, sanitized metadata, and inspection |
 | Generic file orchestration | Shipped | Built-in or caller-provided adapters and atomic safe-copy output |
 | TXT, Markdown, log, JSON, JSONL, and CSV files | Shipped | Explicit format or recognized suffix; no content guessing |

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/ROADMAP.md

@@ -7,14 +7,14 @@ publishable without requiring unfinished later layers.
 
 | Release | Status | Outcome |
 | --- | --- | --- |
-| `0.1.0a1` | Published | Dependency-free text and nested-data core |
-| `0.1.0a2` | Published | Document representation, extension contracts, safe reports, and generic file orchestration |
-| `0.1.0a3` | Published | Built-in dependency-free text and machine-readable file adapters |
-| `0.1.0b1` | Published | Core API freeze and production-oriented examples |
-| `0.1.0rc1` | Published | External installation and release validation |
 | `0.1.0` | Published | First stable text and machine-readable release |
 | `0.2.0` | Published | Optional local machine learning identification for PII |
-| `0.3.0` | Next | Document inspection (PDF, DOCX, XLSX, PPTX) |
+| `0.3.0` | Published | Document inspection (PDF, DOCX, XLSX, PPTX) |
+| `0.4.0` | Published | Format-preserving documents |
+| `0.5.0` | Published | OCR and scanned documents |
+| `0.6.0` | Published | Remote detection |
+| `0.6.1` | Published | Italian identifiers, PDF spans, and inspection fixes |
+| `0.7.0` | Next | Real-world corpus benchmarking |
 
 Alpha releases optimize for the cleanest safe architecture, not backward compatibility. They may
 remove, rename, or replace public APIs without aliases or shims. Material changes are documented,
@@ -121,45 +121,51 @@ Exit criteria:
 Stable local processing for text, nested Python data, and plain or machine-readable files, with a
 documented compatibility policy and zero base runtime dependencies.
 
-## Later capabilities
+## The Road to 1.0 (Hardening and Production Readiness)
 
-### `0.2.0`: optional local ML
+### `0.7.0`: Real-World Corpus Benchmarking
+- **Use Case:** Medical/Legal data mining.
+- **Focus:** Integrate public datasets (Enron corpus, MIMIC-III synthetic variants, SEC filings) to establish precision/recall baselines for existing detectors.
 
-- Explicitly installed ONNX models for people, organizations, locations, and contextual addresses
-- Initial English, German, and Italian benchmarks
-- Pinned model revisions, checksums, licences, memory, and latency measurements
-- No model download during import or inference
+### `0.8.0`: Adversarial Document Defenses
+- **Use Case:** FOIA response redaction failures (e.g., Manafort, AstraZeneca leaks).
+- **Focus:** Prevent visual-only masking, hidden OCR layers, overlapping z-index elements, and scrub incremental document revisions.
 
-### `0.3.0`: document inspection
+### `0.9.0`: Exhaustive Metadata & Hidden Structure Extraction
+- **Use Case:** Corporate e-discovery.
+- **Focus:** Parse PDF `/Info` dictionaries, XMP metadata, DOCX headers/footers, and XLSX comments.
 
-- Optional PDF, DOCX, XLSX, and PPTX extraction
-- Structural or coordinate-aware locations
-- Detection-only output while representation fixtures mature
+### `0.10.0`: Large-Scale & Streaming Data Pipelines
+- **Use Case:** Database dumps and bulk analytics exports.
+- **Focus:** Bounded-memory processing for multi-gigabyte CSV/JSONL files; parallel block processing.
 
-### `0.4.0`: format-preserving documents
+### `0.11.0`: Resilient Document Parsing
+- **Use Case:** Legacy enterprise files.
+- **Focus:** Gracefully handle corrupt XML schemas, custom OOXML namespaces, and safely extract embedded OLE objects.
 
-- Sanitized DOCX, XLSX, and PPTX copies
-- Secure text-PDF redaction with underlying content removal
-- Relevant metadata cleaning and format-integrity tests
+### `0.12.0`: Advanced OCR Degradation Handling
+- **Use Case:** Medical faxes and legacy legal scans.
+- **Focus:** Handle low-dpi faxes, skewed pages, noisy backgrounds, and watermark interference mimicking structural data.
 
-### `0.5.0`: OCR and scanned documents
+### `0.13.0`: ML Confidence Calibration & Thresholding
+- **Use Case:** Reducing false positives in unstructured clinical notes.
+- **Focus:** Tune ONNX backend confidence scores against ambiguous texts and introduce dynamic, policy-based confidence thresholds.
 
-- Local OCR for images, scanned PDFs, and mixed PDFs
-- Bounding-box transformations
-- No OCR when reliable native text exists
+### `0.14.0`: Telemetry, Observability & Auditing
+- **Use Case:** Enterprise compliance reporting and SLA monitoring.
+- **Focus:** Structured JSON logging, processing timeouts, latency tracing, and detailed audit trails without leaking PII.
 
-### `0.6.0`: remote detection
+### `0.15.0`: Remote Backend Fault Tolerance
+- **Use Case:** High-throughput API gateway integration.
+- **Focus:** Circuit breakers, adaptive retries, rate-limit handling, and strict latency bounds for the HTTP provider transport.
 
-- Vendor-neutral provider protocol and optional HTTP transport
-- Explicit dual consent, bounded timeouts, and bounded retries
-- Provider capability reporting and remote-block statistics
-- Local structured-value replacement before permitted remote processing
-- Offset mapping after local preprocessing
-
-### `1.0.0`: mature compatibility commitment
+### `0.16.0`: Cross-Lingual & Typographical Boundary Hardening (Pre-1.0 Audit)
+- **Use Case:** Global support ticket sanitization.
+- **Focus:** Audit detectors against non-Latin scripts, CJK spacing, RTL text, and bidirectional text overrides. Final security fuzzing (Zip-bombs, XXE).
 
+### `1.0.0`: Mature Compatibility Commitment
 Long-term compatibility begins after core processing, document rewriting, OCR, and remote-security
-contracts have production fixtures, published benchmarks, and independent usage feedback.
+contracts have production fixtures, published benchmarks, and independent usage feedback against the hardening milestones above.
 
 ## Optional dependency policy
 

pseudonymize-0.7.0/benchmarks/evaluate_quality.py

@@ -0,0 +1,154 @@
+import argparse
+import logging
+import sys
+import time
+from pathlib import Path
+
+try:
+    from datasets import load_dataset
+except ImportError:
+    print("Error: 'datasets' library not found.")
+    print("Run: uv run --with datasets python benchmarks/evaluate_quality.py")
+    sys.exit(1)
+
+from pseudonymize.backends.ml.onnx import LocalONNXPIIBackend
+from pseudonymize.engine import Pseudonymizer
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
+logger = logging.getLogger("evaluate_quality")
+
+# We evaluate precision and recall against these specific AI4Privacy labels
+# that map to the capabilities of pseudonymize's core detectors and ML backend.
+SUPPORTED_LABELS = {
+    # Core detectors
+    "EMAIL",
+    "PHONENUMBER",
+    "CREDITCARD",
+    "IBAN",
+    "IPADDRESS",
+    "URL",
+    # ML Backend (DistilBERT ONNX)
+    "FIRSTNAME",
+    "LASTNAME",
+    "MIDDLENAME",
+    "COMPANYNAME",
+    "CITY",
+    "STATE",
+    "COUNTY",
+    "STREET",
+    "ZIPCODE",
+}
+
+
+def evaluate(num_samples: int, use_ml: bool):
+    logger.info("Loading ai4privacy/pii-masking-200k (English subset)...")
+    # Use streaming to avoid downloading the entire 200K dataset
+    ds = load_dataset("ai4privacy/pii-masking-200k", split="train", streaming=True)
+
+    engine = Pseudonymizer()
+    if use_ml:
+        # We need the model downloaded. The test suite uses the distilbert model.
+        # Let's assume it's already cached or we can fetch it.
+        # To keep it simple, we'll try to initialize it. If it fails, we fall back or error.
+        CACHE_DIR = Path(".cache/pseudonymize-tests/models/distilbert-ml")
+        onnx_model_path = CACHE_DIR / "model_int8.onnx"
+        tokenizer_path = CACHE_DIR / "tokenizer.json"
+
+        if not onnx_model_path.exists() or not tokenizer_path.exists():
+            logger.error(f"ML artifacts not found in {CACHE_DIR}.")
+            logger.error("Please run 'uv run pytest tests/unit/backends/test_onnx.py' first.")
+            sys.exit(1)
+
+        backend = LocalONNXPIIBackend(model_path=onnx_model_path, tokenizer_path=tokenizer_path)
+        engine = Pseudonymizer(backends=[*engine.backends, backend])
+
+    true_positives = 0
+    false_positives = 0
+    false_negatives = 0
+
+    count = 0
+    start_time = time.time()
+
+    for row in ds:
+        if row["language"] != "en":
+            continue
+
+        text = row["source_text"]
+        masks = row["privacy_mask"]
+
+        # Filter ground truth masks to only those we claim to support
+        ground_truth_spans = [
+            (mask["start"], mask["end"], mask["label"])
+            for mask in masks
+            if mask["label"] in SUPPORTED_LABELS
+        ]
+
+        # Run detection
+        result = engine.process_with_report(text)
+
+        # Extract detected spans
+        detected_spans = [(detection.start, detection.end) for detection in result.detections]
+
+        # Calculate overlap
+        # For simplicity in this benchmark:
+        # TP: a detected span overlaps with a ground truth span.
+        # FP: a detected span does not overlap with any ground truth span.
+        # FN: a ground truth span is not overlapped by any detected span.
+
+        matched_gt = set()
+        for d_start, d_end in detected_spans:
+            matched = False
+            for i, (g_start, g_end, _g_label) in enumerate(ground_truth_spans):
+                # Check overlap
+                if max(d_start, g_start) < min(d_end, g_end):
+                    matched = True
+                    matched_gt.add(i)
+                    break
+            if matched:
+                true_positives += 1
+            else:
+                false_positives += 1
+
+        for i in range(len(ground_truth_spans)):
+            if i not in matched_gt:
+                false_negatives += 1
+
+        count += 1
+        if count >= num_samples:
+            break
+
+    elapsed = time.time() - start_time
+
+    precision = (
+        true_positives / (true_positives + false_positives)
+        if (true_positives + false_positives) > 0
+        else 0
+    )
+    recall = (
+        true_positives / (true_positives + false_negatives)
+        if (true_positives + false_negatives) > 0
+        else 0
+    )
+    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
+
+    logger.info(f"--- Benchmark Results ({count} samples) ---")
+    logger.info(f"Time elapsed: {elapsed:.2f}s")
+    logger.info(f"True Positives:  {true_positives}")
+    logger.info(f"False Positives: {false_positives}")
+    logger.info(f"False Negatives: {false_negatives}")
+    logger.info(f"Precision:       {precision:.4f}")
+    logger.info(f"Recall:          {recall:.4f}")
+    logger.info(f"F1 Score:        {f1:.4f}")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Evaluate pseudonymize precision and recall against real-world datasets."
+    )
+    parser.add_argument("--samples", type=int, default=500, help="Number of samples to evaluate.")
+    parser.add_argument(
+        "--ml", action="store_true", help="Include the ONNX ML backend in evaluation."
+    )
+    args = parser.parse_args()
+
+    evaluate(args.samples, args.ml)

pseudonymize-0.7.0/docs/detectors.md

@@ -0,0 +1,12 @@
+# Detectors
+
+Bundled detectors cover email, phone, IP address, IBAN, payment card, URL credential, and secret
+entities. Italian fiscal codes and VAT identifiers are also supported as `NATIONAL_ID` and
+`TAX_ID`. Candidate extraction is followed by normalization and validation. IBANs require MOD-97,
+cards require Luhn, Italian identifiers require their official check character or digit, and IP
+candidates must parse with the standard library. An Italian VAT number without an `IT` prefix is
+considered only next to an explicit VAT label to avoid treating arbitrary 11-digit values as tax
+identifiers.
+
+Detector results contain type, offsets, confidence, and detector name. They never contain the raw
+matched value. Custom thread-safe detectors can implement the `Detector` protocol.

pseudonymize-0.7.0/docs/quality_benchmarks.md

@@ -0,0 +1,27 @@
+# Detection Quality Benchmarks
+
+This document tracks the precision and recall of the `pseudonymize` engine across real-world and synthetic evaluation corpora. Unlike performance benchmarks (which measure throughput and latency), quality benchmarks ensure the core detectors and ML backends accurately identify PII without excessive false positives.
+
+## `0.7.0` Baseline (AI4Privacy PII-Masking-200k)
+
+Evaluated against the English subset of the `ai4privacy/pii-masking-200k` dataset, comparing the engine's detections against the dataset's ground-truth `privacy_mask`.
+
+**Setup:**
+- Model: `LocalONNXPIIBackend` (DistilBERT ONNX int8)
+- Target Entities: `EMAIL`, `PHONENUMBER`, `CREDITCARD`, `IBAN`, `IPADDRESS`, `URL`, `FIRSTNAME`, `LASTNAME`, `MIDDLENAME`, `COMPANYNAME`, `CITY`, `STATE`, `COUNTY`, `STREET`, `ZIPCODE`
+- Sample Size: 50 documents (Streaming)
+
+**Results (Preliminary Baseline):**
+
+| Metric | Score |
+| --- | --- |
+| Precision | 0.4091 |
+| Recall | 0.1875 |
+| F1 Score | 0.2571 |
+
+*Note: These baseline metrics reflect zero-configuration default detectors and the stock ML backend on a highly ambiguous, fine-grained synthetic dataset. Future releases (e.g., 0.13.0 ML Calibration) will optimize these metrics through tuned thresholds and improved boundaries.*
+
+To reproduce this benchmark locally:
+```console
+uv run python benchmarks/evaluate_quality.py --ml --samples 50
+```

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/mkdocs.yml

@@ -27,6 +27,7 @@ nav:
   - Threat model: threat-model.md
   - Limitations: limitations.md
   - Benchmarks: benchmarks.md
+  - Quality Benchmarks: quality_benchmarks.md
   - Contributing: contributing.md
   - Decisions:
       - Dependency-free core: adr/0001-dependency-free-core.md

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pseudonymize"
-version = "0.6.0"
+version = "0.7.0"
 description = "Local-first PII pseudonymization for text, structured data, and LLM payloads."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -72,6 +72,9 @@ test = [
     "fpdf2>=2.7.9",
     "respx>=0.23.1",
 ]
+benchmarks = [
+    "datasets>=3.0.0",
+]
 quality = ["mypy>=1.11", "pre-commit>=4", "ruff>=0.9"]
 docs = ["mkdocs-material>=9", "mkdocstrings[python]>=1.0.6"]
 release = ["build>=1", "twine>=6"]

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/src/pseudonymize/backends/rules.py

@@ -14,6 +14,8 @@ _RULE_ENTITY_TYPES = frozenset(
         EntityType.IP_ADDRESS,
         EntityType.IBAN,
         EntityType.PAYMENT_CARD,
+        EntityType.NATIONAL_ID,
+        EntityType.TAX_ID,
         EntityType.URL_CREDENTIAL,
         EntityType.SECRET,
     }

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/src/pseudonymize/cli.py

@@ -10,7 +10,13 @@ from pathlib import Path
 
 from pseudonymize.api import generate_key
 from pseudonymize.detectors import DEFAULT_DETECTORS
-from pseudonymize.document import CSVCellLocation, JSONPathLocation, TextOffsetLocation
+from pseudonymize.document import (
+    CoordinateLocation,
+    CSVCellLocation,
+    JSONPathLocation,
+    StructuralLocation,
+    TextOffsetLocation,
+)
 from pseudonymize.engine import Data, Pseudonymizer
 from pseudonymize.exceptions import PseudonymizeError
 from pseudonymize.formats import FileFormat
@@ -199,6 +205,17 @@ def _location_payload(report: DetectionReport) -> dict[str, object]:
         return {"kind": "json_path", "path": location.path}
     if isinstance(location, CSVCellLocation):
         return {"kind": "csv_cell", "row": location.row, "column": location.column}
+    if isinstance(location, CoordinateLocation):
+        return {
+            "kind": "coordinate",
+            "page": location.page,
+            "x0": location.x0,
+            "y0": location.y0,
+            "x1": location.x1,
+            "y1": location.y1,
+        }
+    if isinstance(location, StructuralLocation):
+        return {"kind": "structural", "path": location.path}
     raise TypeError("unsupported report location")
 
 

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/src/pseudonymize/detectors/iban.py

@@ -3,7 +3,10 @@ from dataclasses import dataclass
 
 from pseudonymize.result import Detection, EntityType
 
-_IBAN = re.compile(r"(?<![A-Z0-9])[A-Z]{2}\d{2}(?:[ ]?[A-Z0-9]){11,30}(?![A-Z0-9])", re.I)
+_IBAN = re.compile(
+    r"(?<![A-Z0-9])[A-Z]{2}(?:[ ]?\d){2}(?:[ ]?[A-Z0-9]){11,30}(?![A-Z0-9])",
+    re.I,
+)
 
 
 def _valid_mod97(value: str) -> bool:

pseudonymize-0.7.0/src/pseudonymize/detectors/italian.py

@@ -0,0 +1,122 @@
+import re
+from dataclasses import dataclass
+
+from pseudonymize.result import Detection, EntityType
+
+_OMOCODIA_DIGITS = str.maketrans("LMNPQRSTUV", "0123456789")
+_FISCAL_CODE = re.compile(
+    r"(?<![A-Z0-9])"
+    r"[A-Z]{6}[0-9LMNPQRSTUV]{2}[ABCDEHLMPRST][0-9LMNPQRSTUV]{2}"
+    r"[A-Z][0-9LMNPQRSTUV]{3}[A-Z]"
+    r"(?![A-Z0-9])",
+    re.I,
+)
+_ODD_VALUES = dict(
+    zip(
+        "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ",
+        (
+            1,
+            0,
+            5,
+            7,
+            9,
+            13,
+            15,
+            17,
+            19,
+            21,
+            1,
+            0,
+            5,
+            7,
+            9,
+            13,
+            15,
+            17,
+            19,
+            21,
+            2,
+            4,
+            18,
+            20,
+            11,
+            3,
+            6,
+            8,
+            12,
+            14,
+            16,
+            10,
+            22,
+            25,
+            24,
+            23,
+        ),
+        strict=True,
+    )
+)
+_EVEN_VALUES = {
+    **{character: value for value, character in enumerate("0123456789")},
+    **{character: value for value, character in enumerate("ABCDEFGHIJKLMNOPQRSTUVWXYZ")},
+}
+_VAT_WITH_COUNTRY = re.compile(
+    r"(?<![A-Z0-9])(?P<value>IT[ ]?(?:\d[ ]?){10}\d)(?![A-Z0-9])",
+    re.I,
+)
+_VAT_WITH_LABEL = re.compile(
+    r"\b(?:partita[ ]+iva|p[ .]*iva|p[ .]*i[ .]*|vat(?:[ ]+id)?)"
+    r"\s*[:=]?\s*(?P<value>(?:IT[ ]?)?(?:\d[ ]?){10}\d)(?!\d)",
+    re.I,
+)
+
+
+def _valid_fiscal_code(value: str) -> bool:
+    compact = value.upper()
+    day_text = compact[9:11].translate(_OMOCODIA_DIGITS)
+    day = int(day_text)
+    if not (1 <= day <= 31 or 41 <= day <= 71):
+        return False
+    total = sum(
+        (_ODD_VALUES if index % 2 == 0 else _EVEN_VALUES)[character]
+        for index, character in enumerate(compact[:15])
+    )
+    return compact[-1] == chr(ord("A") + total % 26)
+
+
+def _valid_vat(value: str) -> bool:
+    compact = "".join(character for character in value if character.isdigit())
+    if len(compact) != 11 or len(set(compact)) == 1:
+        return False
+    total = sum(int(character) for character in compact[:10:2])
+    for character in compact[1:10:2]:
+        doubled = int(character) * 2
+        total += doubled - 9 if doubled > 9 else doubled
+    return (10 - total % 10) % 10 == int(compact[-1])
+
+
+@dataclass(frozen=True, slots=True)
+class ItalianFiscalCodeDetector:
+    name: str = "italian_fiscal_code"
+
+    def detect(self, text: str) -> list[Detection]:
+        return [
+            Detection(EntityType.NATIONAL_ID, match.start(), match.end(), 1.0, self.name)
+            for match in _FISCAL_CODE.finditer(text)
+            if _valid_fiscal_code(match.group())
+        ]
+
+
+@dataclass(frozen=True, slots=True)
+class ItalianVATDetector:
+    name: str = "italian_vat"
+
+    def detect(self, text: str) -> list[Detection]:
+        spans = {
+            match.span("value")
+            for pattern in (_VAT_WITH_COUNTRY, _VAT_WITH_LABEL)
+            for match in pattern.finditer(text)
+            if _valid_vat(match.group("value"))
+        }
+        return [
+            Detection(EntityType.TAX_ID, start, end, 1.0, self.name) for start, end in sorted(spans)
+        ]

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/src/pseudonymize/detectors/registry.py

@@ -2,6 +2,7 @@ from pseudonymize.detectors.base import Detector
 from pseudonymize.detectors.email import EmailDetector
 from pseudonymize.detectors.iban import IbanDetector
 from pseudonymize.detectors.ip_address import IpAddressDetector
+from pseudonymize.detectors.italian import ItalianFiscalCodeDetector, ItalianVATDetector
 from pseudonymize.detectors.payment_card import PaymentCardDetector
 from pseudonymize.detectors.phone import PhoneDetector
 from pseudonymize.detectors.secret import SecretDetector
@@ -12,6 +13,8 @@ DEFAULT_DETECTORS: tuple[Detector, ...] = (
     IpAddressDetector(),
     PaymentCardDetector(),
     IbanDetector(),
+    ItalianFiscalCodeDetector(),
+    ItalianVATDetector(),
     PhoneDetector(),
     UrlDetector(),
     SecretDetector(),

{pseudonymize-0.6.0 → pseudonymize-0.7.0}/src/pseudonymize/formats.py

@@ -339,6 +339,9 @@ def _read_csv(text: str) -> tuple[tuple[str, ...], ...]:
         previous_limit = csv.field_size_limit()
         try:
             csv.field_size_limit(sys.maxsize)
+        except OverflowError:
+            csv.field_size_limit(2147483647)
+        try:
             return tuple(
                 tuple(row) for row in csv.reader(io.StringIO(text, newline=""), strict=True)
             )