SqueakyCleanText 0.6.0__tar.gz → 0.7.0__tar.gz

{squeakycleantext-0.6.0 → squeakycleantext-0.7.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.6.0
+Version: 0.7.0
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
-License: MIT
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Repository, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Issues, https://github.com/rhnfzl/SqueakyCleanText/issues
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -32,7 +32,6 @@ Requires-Dist: stop-words>=2025.11.4
 Requires-Dist: emoji>=2.14
 Requires-Dist: ftfy>=6.3
 Requires-Dist: Unidecode>=1.4
-Requires-Dist: beautifulsoup4>=4.14
 Requires-Dist: onnxruntime>=1.24.2
 Requires-Dist: tokenizers>=0.22.2
 Requires-Dist: huggingface-hub>=1.5.0
@@ -56,8 +55,6 @@ Provides-Extra: presidio
 Requires-Dist: presidio-analyzer>=2.2.360; extra == "presidio"
 Provides-Extra: classify
 Requires-Dist: gliclass>=0.1.16; extra == "classify"
-Provides-Extra: classify-onnx
-Requires-Dist: onnxruntime>=1.24.2; extra == "classify-onnx"
 Provides-Extra: all-ner
 Requires-Dist: torch>=2.6.0; extra == "all-ner"
 Requires-Dist: transformers>=4.48; extra == "all-ner"
@@ -73,6 +70,11 @@ Requires-Dist: rapidfuzz>=3.12; extra == "dev"
 Provides-Extra: test
 Requires-Dist: coverage>=7.8; extra == "test"
 Requires-Dist: pytest-cov>=6.0; extra == "test"
+Provides-Extra: mcp
+Requires-Dist: fastmcp>=4.0.5; extra == "mcp"
+Provides-Extra: ocr
+Requires-Dist: pillow>=12.3.0; extra == "ocr"
+Requires-Dist: pytesseract>=0.3.13; extra == "ocr"
 Dynamic: license-file
 
 <div align="center">
@@ -82,7 +84,7 @@ Dynamic: license-file
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -179,6 +181,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -230,6 +234,21 @@ cfg = TextCleanerConfig(
 cleaner = TextCleaner(cfg=cfg)
 ```
 
+#### Language Specification
+
+All language parameters accept Lingua names (`'ENGLISH'`), ISO 639-1 (`'en'`), or ISO 639-3 (`'eng'`) codes:
+
+```python
+# Pin to one language (skip auto-detection)
+cfg = TextCleanerConfig(language='de', check_ner_process=False)
+
+# Restrict detection to specific languages (auto-detect among them)
+cfg = TextCleanerConfig(language=('en', 'nl', 'de'), check_ner_process=False)
+
+# Add extra languages for detection
+cfg = TextCleanerConfig(extra_languages=('fr', 'pt'), check_ner_process=False)
+```
+
 ### GLiNER: Zero-Shot Custom NER
 
 Use [GLiNER](https://github.com/urchade/GLiNER) to recognize any entity type without retraining:
@@ -638,6 +657,116 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -675,6 +804,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency

{squeakycleantext-0.6.0 → squeakycleantext-0.7.0}/README.md

@@ -5,7 +5,7 @@
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -102,6 +102,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -153,6 +155,21 @@ cfg = TextCleanerConfig(
 cleaner = TextCleaner(cfg=cfg)
 ```
 
+#### Language Specification
+
+All language parameters accept Lingua names (`'ENGLISH'`), ISO 639-1 (`'en'`), or ISO 639-3 (`'eng'`) codes:
+
+```python
+# Pin to one language (skip auto-detection)
+cfg = TextCleanerConfig(language='de', check_ner_process=False)
+
+# Restrict detection to specific languages (auto-detect among them)
+cfg = TextCleanerConfig(language=('en', 'nl', 'de'), check_ner_process=False)
+
+# Add extra languages for detection
+cfg = TextCleanerConfig(extra_languages=('fr', 'pt'), check_ner_process=False)
+```
+
 ### GLiNER: Zero-Shot Custom NER
 
 Use [GLiNER](https://github.com/urchade/GLiNER) to recognize any entity type without retraining:
@@ -561,6 +578,116 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -598,6 +725,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency

{squeakycleantext-0.6.0 → squeakycleantext-0.7.0}/SqueakyCleanText.egg-info/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.6.0
+Version: 0.7.0
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
-License: MIT
+License-Expression: MIT
 Project-URL: Homepage, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Repository, https://github.com/rhnfzl/SqueakyCleanText
 Project-URL: Issues, https://github.com/rhnfzl/SqueakyCleanText/issues
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
-Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Operating System :: OS Independent
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -32,7 +32,6 @@ Requires-Dist: stop-words>=2025.11.4
 Requires-Dist: emoji>=2.14
 Requires-Dist: ftfy>=6.3
 Requires-Dist: Unidecode>=1.4
-Requires-Dist: beautifulsoup4>=4.14
 Requires-Dist: onnxruntime>=1.24.2
 Requires-Dist: tokenizers>=0.22.2
 Requires-Dist: huggingface-hub>=1.5.0
@@ -56,8 +55,6 @@ Provides-Extra: presidio
 Requires-Dist: presidio-analyzer>=2.2.360; extra == "presidio"
 Provides-Extra: classify
 Requires-Dist: gliclass>=0.1.16; extra == "classify"
-Provides-Extra: classify-onnx
-Requires-Dist: onnxruntime>=1.24.2; extra == "classify-onnx"
 Provides-Extra: all-ner
 Requires-Dist: torch>=2.6.0; extra == "all-ner"
 Requires-Dist: transformers>=4.48; extra == "all-ner"
@@ -73,6 +70,11 @@ Requires-Dist: rapidfuzz>=3.12; extra == "dev"
 Provides-Extra: test
 Requires-Dist: coverage>=7.8; extra == "test"
 Requires-Dist: pytest-cov>=6.0; extra == "test"
+Provides-Extra: mcp
+Requires-Dist: fastmcp>=4.0.5; extra == "mcp"
+Provides-Extra: ocr
+Requires-Dist: pillow>=12.3.0; extra == "ocr"
+Requires-Dist: pytesseract>=0.3.13; extra == "ocr"
 Dynamic: license-file
 
 <div align="center">
@@ -82,7 +84,7 @@ Dynamic: license-file
 [![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
 [![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
-[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13-blue)](https://pypi.org/project/squeakycleantext/)
+[![Python Versions](https://img.shields.io/badge/Python-3.11%20|%203.12%20|%203.13%20|%203.14-blue)](https://pypi.org/project/squeakycleantext/)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
 A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
@@ -179,6 +181,8 @@ The base install uses **ONNX Runtime** for NER inference - no PyTorch or Transfo
 | Synthetic | `pip install SqueakyCleanText[synthetic]` | Faker-based synthetic replacement (realistic fake values instead of `<TAG>` tokens) |
 | Presidio | `pip install SqueakyCleanText[presidio]` | Presidio-analyzer for `presidio_gliner` backend |
 | Classify | `pip install SqueakyCleanText[classify]` | GLiClass document-level pre-classification |
+| MCP | `pip install SqueakyCleanText[mcp]` | FastMCP server with text and JSON cleaning tools |
+| OCR | `pip install SqueakyCleanText[ocr]` | Pillow image redaction and Tesseract OCR adapter |
 | All NER | `pip install SqueakyCleanText[all-ner]` | All NER backends combined |
 | Development | `pip install SqueakyCleanText[dev]` | Testing and linting tools |
 
@@ -230,6 +234,21 @@ cfg = TextCleanerConfig(
 cleaner = TextCleaner(cfg=cfg)
 ```
 
+#### Language Specification
+
+All language parameters accept Lingua names (`'ENGLISH'`), ISO 639-1 (`'en'`), or ISO 639-3 (`'eng'`) codes:
+
+```python
+# Pin to one language (skip auto-detection)
+cfg = TextCleanerConfig(language='de', check_ner_process=False)
+
+# Restrict detection to specific languages (auto-detect among them)
+cfg = TextCleanerConfig(language=('en', 'nl', 'de'), check_ner_process=False)
+
+# Add extra languages for detection
+cfg = TextCleanerConfig(extra_languages=('fr', 'pt'), check_ner_process=False)
+```
+
 ### GLiNER: Zero-Shot Custom NER
 
 Use [GLiNER](https://github.com/urchade/GLiNER) to recognize any entity type without retraining:
@@ -638,6 +657,116 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 
 </details>
 
+## Privacy Engine
+
+Applications can apply versioned policy rules and inspect each decision.
+This makes privacy behavior testable across services, datasets, and model upgrades.
+
+```python
+from sct import EntityRule, PrivacyPolicy, TextCleaner, TextCleanerConfig
+
+policy = PrivacyPolicy(
+    name="support-export",
+    version="1",
+    rules={
+        "PERSON": EntityRule(action="redact", min_score=0.8),
+        "ORGANISATION": EntityRule(action="keep"),
+    },
+    allowlist=frozenset({"Jane Public"}),
+)
+
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(ner_mode="pii"),
+    privacy_policy=policy,
+    include_entities=True,
+)
+result = cleaner.process("Contact Maria Chen for help.")
+
+print(result.metadata["findings"])
+print(result.metadata["detector"])
+```
+
+Reversible output is pseudonymized personal data. Anyone with the token map can
+restore the source values. Keep token maps outside logs and cleaned datasets.
+
+Use a `TokenStore` implementation to keep maps behind opaque references:
+
+```python
+from sct import InMemoryTokenStore, TextCleaner, TextCleanerConfig
+
+store = InMemoryTokenStore()
+cleaner = TextCleaner(
+    cfg=TextCleanerConfig(replacement_mode="reversible"),
+    token_store=store,
+)
+result = cleaner.process("Maria lives in Utrecht.")
+reference = result.metadata["anon_map_reference"]
+```
+
+`InMemoryTokenStore` supports tests and short sessions. Production systems
+should implement `TokenStore` with encryption, access control, expiry, and audit logs.
+
+### Boundary adapters
+
+The base package includes JSON, DataFrame, retrieval, command-line, and streaming adapters:
+
+```python
+from sct import RAGChunk, process_json, process_rag_chunks
+
+cleaned_json = process_json(cleaner, {"message": "Email a@example.com"})
+cleaned_chunks = process_rag_chunks(
+    cleaner,
+    [RAGChunk(id="1", text="Call +1-202-555-0147", metadata={})],
+)
+```
+
+```sh
+printf 'Email maria@example.com' | sct-clean --no-ner
+printf 'Email maria@example.com' | sct-clean --no-ner \
+  --replacement-mode reversible --token-map tokens.json
+pip install 'SqueakyCleanText[mcp]'
+sct-mcp
+```
+
+`BufferedStreamingRedactor` holds a full document for cross-chunk NER safety.
+`WindowedStreamingRedactor` provides bounded latency for contact tokens.
+
+The OCR adapter accepts any `OCRProvider`. The included Tesseract provider also
+needs a local Tesseract executable. Browser export writes an ONNX Runtime Web
+manifest, but it does not copy model files or Python language detection.
+
+Corpus owners can call `assess_reidentification_risk` to measure k-anonymity
+for selected quasi-identifiers. Reports contain counts only, not source values.
+
+## PII Evaluation
+
+Model changes need ground-truth privacy evaluation, not only backend agreement.
+The evaluation harness reports exact entity metrics by label, language, domain,
+and named slice.
+
+```sh
+python -m scripts.evaluate_pii \
+  --dataset evaluation/pii_starter.jsonl \
+  --predictions path/to/model-predictions.jsonl \
+  --model model-name-and-revision \
+  --output evaluation/results/model-name.json
+```
+
+The included starter corpus is synthetic and only checks the evaluation path.
+See [`evaluation/README.md`](evaluation/README.md) for schemas and release rules.
+
+Pinned candidates live in `sct.model_candidates`. Run one candidate with:
+
+```sh
+python -m scripts.benchmark_models \
+  --dataset evaluation/pii_starter.jsonl \
+  --candidate gliner2-pii \
+  --output evaluation/results/gliner2-pii.json
+```
+
+Published third-party results are reference data only. `assess_candidate`
+requires local recall, exact-span F1, latency, and ONNX parity before adoption.
+
 ## Architecture
 
 SqueakyCleanText processes text through a configurable pipeline of sequential steps:
@@ -675,6 +804,23 @@ Each step is toggled by a `TextCleanerConfig` field. The pipeline is built once
 
 ## What's New
 
+**v0.7.0**
+- **Privacy policies**: Apply versioned entity rules, allowlists, thresholds, and redaction actions.
+- **Structured audit findings**: Inspect entity decisions, detector backends, and pinned model revisions.
+- **Safer reversible processing**: Restore contact values and keep token maps behind an optional `TokenStore`.
+- **Boundary adapters**: Clean JSON, DataFrame-like objects, RAG chunks, standard input, and MCP tool requests.
+- **Privacy evaluation**: Score exact PII spans by label, language, domain, and custom slice.
+- **Model adoption gates**: Compare pinned candidates using recall, F1, latency, and ONNX parity.
+- **Extended media support**: Redact OCR regions, process bounded streams, and export browser deployment manifests.
+- **Corpus risk indicators**: Measure k-anonymity for user-selected quasi-identifiers.
+- **Supply-chain improvements**: Pin model revisions, publish through PyPI Trusted Publishing, and ship standard lockfiles.
+- **Python 3.14**: Added CI coverage alongside Python 3.11 through 3.13.
+
+Migration notes:
+- `gliclass_onnx=True` now raises an error because the upstream package has no native ONNX loader.
+- HTML processing now honors `replace_with_html` instead of silently stripping tags.
+- Structured adapters reject reversible mode unless each token map can be retained safely.
+
 **v0.6.0**
 - **PII detection mode** (`ner_mode='pii'`): auto-configures GLiNER with 60+ PII entity labels (personal, financial, healthcare, identity, digital)
 - **Synthetic replacement** (`replacement_mode='synthetic'`): Faker-generated realistic values instead of `<TAG>` placeholders, with per-document consistency

{squeakycleantext-0.6.0 → squeakycleantext-0.7.0}/SqueakyCleanText.egg-info/SOURCES.txt

@@ -5,11 +5,23 @@ setup.py
 SqueakyCleanText.egg-info/PKG-INFO
 SqueakyCleanText.egg-info/SOURCES.txt
 SqueakyCleanText.egg-info/dependency_links.txt
+SqueakyCleanText.egg-info/entry_points.txt
 SqueakyCleanText.egg-info/requires.txt
 SqueakyCleanText.egg-info/top_level.txt
 sct/__init__.py
+sct/adapters.py
+sct/browser.py
+sct/cli.py
 sct/config.py
+sct/evaluation.py
+sct/mcp_server.py
+sct/model_benchmark.py
+sct/model_candidates.py
+sct/ocr.py
+sct/privacy.py
+sct/risk.py
 sct/sct.py
+sct/streaming.py
 sct/utils/__init__.py
 sct/utils/anonymization_map.py
 sct/utils/constants.py
@@ -26,4 +38,16 @@ sct/utils/special.py
 sct/utils/stopwords.py
 sct/utils/synthetic.py
 sct/utils/torch_pipeline.py
-tests/test_sct.py
+tests/test_adapters.py
+tests/test_browser.py
+tests/test_cli.py
+tests/test_contracts.py
+tests/test_evaluation.py
+tests/test_functional.py
+tests/test_mcp_server.py
+tests/test_model_benchmarks.py
+tests/test_ocr.py
+tests/test_privacy.py
+tests/test_risk.py
+tests/test_sct.py
+tests/test_streaming.py

squeakycleantext-0.7.0/SqueakyCleanText.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+sct-clean = sct.cli:main
+sct-mcp = sct.mcp_server:main