SqueakyCleanText 0.6.1__tar.gz → 0.7.1__tar.gz

{squeakycleantext-0.6.1 → squeakycleantext-0.7.1}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.6.1
+Version: 0.7.1
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
-License: MIT
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Repository, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Issues, https://github.com/rhnfzl/SqueakyCleanText/issues
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -32,7 +32,6 @@ Requires-Dist: stop-words>=2025.11.4
 Requires-Dist: emoji>=2.14
 Requires-Dist: ftfy>=6.3
 Requires-Dist: Unidecode>=1.4
-Requires-Dist: beautifulsoup4>=4.14
 Requires-Dist: onnxruntime>=1.24.2
 Requires-Dist: tokenizers>=0.22.2
 Requires-Dist: huggingface-hub>=1.5.0
@@ -56,8 +55,6 @@ Provides-Extra: presidio
 Requires-Dist: presidio-analyzer>=2.2.360; extra == "presidio"
 Provides-Extra: classify
 Requires-Dist: gliclass>=0.1.16; extra == "classify"
-Provides-Extra: classify-onnx
-Requires-Dist: onnxruntime>=1.24.2; extra == "classify-onnx"
 Provides-Extra: all-ner
 Requires-Dist: torch>=2.6.0; extra == "all-ner"
 Requires-Dist: transformers>=4.48; extra == "all-ner"
@@ -73,6 +70,11 @@ Requires-Dist: rapidfuzz>=3.12; extra == "dev"
 Provides-Extra: test
 Requires-Dist: coverage>=7.8; extra == "test"
 Requires-Dist: pytest-cov>=6.0; extra == "test"
+Provides-Extra: mcp
+Requires-Dist: fastmcp>=4.0.5; extra == "mcp"
+Provides-Extra: ocr
+Requires-Dist: pillow>=12.3.0; extra == "ocr"
+Requires-Dist: pytesseract>=0.3.13; extra == "ocr"
 Dynamic: license-file
 
 <div align="center">
@@ -82,7 +84,7 @@ Dynamic: license-file
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -179,6 +181,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -304,7 +308,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -588,11 +592,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -615,6 +622,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -653,6 +661,121 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -690,6 +813,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency

{squeakycleantext-0.6.1 → squeakycleantext-0.7.1}/README.md

@@ -5,7 +5,7 @@
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -102,6 +102,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -227,7 +229,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -511,11 +513,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -538,6 +543,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -576,6 +582,121 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -613,6 +734,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency

{squeakycleantext-0.6.1 → squeakycleantext-0.7.1}/SqueakyCleanText.egg-info/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.6.1
+Version: 0.7.1
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
-License: MIT
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Repository, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Issues, https://github.com/rhnfzl/SqueakyCleanText/issues
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -32,7 +32,6 @@ Requires-Dist: stop-words>=2025.11.4
 Requires-Dist: emoji>=2.14
 Requires-Dist: ftfy>=6.3
 Requires-Dist: Unidecode>=1.4
-Requires-Dist: beautifulsoup4>=4.14
 Requires-Dist: onnxruntime>=1.24.2
 Requires-Dist: tokenizers>=0.22.2
 Requires-Dist: huggingface-hub>=1.5.0
@@ -56,8 +55,6 @@ Provides-Extra: presidio
 Requires-Dist: presidio-analyzer>=2.2.360; extra == "presidio"
 Provides-Extra: classify
 Requires-Dist: gliclass>=0.1.16; extra == "classify"
-Provides-Extra: classify-onnx
-Requires-Dist: onnxruntime>=1.24.2; extra == "classify-onnx"
 Provides-Extra: all-ner
 Requires-Dist: torch>=2.6.0; extra == "all-ner"
 Requires-Dist: transformers>=4.48; extra == "all-ner"
@@ -73,6 +70,11 @@ Requires-Dist: rapidfuzz>=3.12; extra == "dev"
 Provides-Extra: test
 Requires-Dist: coverage>=7.8; extra == "test"
 Requires-Dist: pytest-cov>=6.0; extra == "test"
+Provides-Extra: mcp
+Requires-Dist: fastmcp>=4.0.5; extra == "mcp"
+Provides-Extra: ocr
+Requires-Dist: pillow>=12.3.0; extra == "ocr"
+Requires-Dist: pytesseract>=0.3.13; extra == "ocr"
 Dynamic: license-file
 
 <div align="center">
@@ -82,7 +84,7 @@ Dynamic: license-file
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -179,6 +181,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -304,7 +308,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -588,11 +592,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -615,6 +622,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -653,6 +661,121 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -690,6 +813,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency