SqueakyCleanText 0.7.0__tar.gz → 0.7.1__tar.gz

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.7.0
+Version: 0.7.1
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
 License-Expression: MIT
@@ -308,7 +308,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -592,11 +592,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -619,6 +622,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -686,6 +690,11 @@ print(result.metadata["findings"])
 print(result.metadata["detector"])
 ```
 
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
 Reversible output is pseudonymized personal data. Anyone with the token map can
 restore the source values. Keep token maps outside logs and cleaned datasets.
 

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/README.md

@@ -229,7 +229,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -513,11 +513,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -540,6 +543,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -607,6 +611,11 @@ print(result.metadata["findings"])
 print(result.metadata["detector"])
 ```
 
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
 Reversible output is pseudonymized personal data. Anyone with the token map can
 restore the source values. Keep token maps outside logs and cleaned datasets.
 

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/SqueakyCleanText.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: SqueakyCleanText
-Version: 0.7.0
+Version: 0.7.1
 Summary: Text preprocessing & PII anonymization pipeline for NLP/ML: ONNX NER ensemble, language detection, stopword removal, and configurable token replacement.
 Author: Rehan Fazal
 License-Expression: MIT
@@ -308,7 +308,7 @@ lm_text, stat_text, lang = cleaner.process(
 # Entities are anonymized: names, SSNs, emails, dates of birth, and 50+ more PII types
 ```
 
-PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets threshold to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
+PII mode auto-configures: `ner_backend='gliner'`, uses [`knowledgator/gliner-pii-base-v1.0`](https://huggingface.co/knowledgator/gliner-pii-base-v1.0), sets `gliner_threshold` and the final `ner_confidence_threshold` to 0.3 (recall-focused), and expands positional tags. User-provided values always take priority.
 
 **Alternative PII models** (pass as `gliner_model`):
 
@@ -592,11 +592,14 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `check_replace_html` | `True` | Strip/replace HTML tags |
 | `check_replace_urls` | `True` | Replace URLs with token |
 | `check_replace_emails` | `True` | Replace emails with token |
+| `check_replace_ips` | `True` | Replace IPv4/IPv6 addresses (runs before dates, phones, years) |
+| `replace_ips_public_only` | `False` | Keep private, loopback and reserved IPs; replace only public ones |
 | `check_replace_years` | `True` | Replace years (1900-2099) |
 | `check_replace_dates` | `False` | Replace full dates (ISO 8601, month names) |
 | `check_fuzzy_replace_dates` | `False` | Fuzzy match misspelled months (requires `[fuzzy]`) |
 | `check_replace_phone_numbers` | `True` | Replace phone numbers |
 | `check_replace_numbers` | `True` | Replace standalone numbers |
+| `replace_numbers_in_words` | `False` | Also replace digits inside Latin/Greek/Cyrillic words (`GPT4` to `GPT<NUMBER>`) |
 | `check_replace_currency_symbols` | `True` | Replace currency symbols |
 | `check_ner_process` | `True` | Run NER entity recognition |
 | `check_remove_isolated_letters` | `True` | Remove single letters |
@@ -619,6 +622,7 @@ new_cfg = dataclasses.replace(cfg, check_ner_process=False)
 | `replace_with_url` | `"<URL>"` |
 | `replace_with_html` | `"<HTML>"` |
 | `replace_with_email` | `"<EMAIL>"` |
+| `replace_with_ips` | `"<IP>"` |
 | `replace_with_years` | `"<YEAR>"` |
 | `replace_with_dates` | `"<DATE>"` |
 | `replace_with_phone_numbers` | `"<PHONE>"` |
@@ -686,6 +690,11 @@ print(result.metadata["findings"])
 print(result.metadata["detector"])
 ```
 
+Finding offsets (`start`, `end`) and NER entries in the reversible map are
+document-relative positions in the post-regex text: the text after the regex
+stages replaced URLs, emails, phones and the like. They are not offsets into the
+raw input.
+
 Reversible output is pseudonymized personal data. Anyone with the token map can
 restore the source values. Keep token maps outside logs and cleaned datasets.
 

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/SqueakyCleanText.egg-info/SOURCES.txt

@@ -39,8 +39,11 @@ sct/utils/stopwords.py
 sct/utils/synthetic.py
 sct/utils/torch_pipeline.py
 tests/test_adapters.py
+tests/test_benchmark_release.py
 tests/test_browser.py
+tests/test_chunking.py
 tests/test_cli.py
+tests/test_config_state.py
 tests/test_contracts.py
 tests/test_evaluation.py
 tests/test_functional.py
@@ -48,6 +51,7 @@ tests/test_mcp_server.py
 tests/test_model_benchmarks.py
 tests/test_ocr.py
 tests/test_privacy.py
+tests/test_regex_boundaries.py
 tests/test_risk.py
 tests/test_sct.py
 tests/test_streaming.py

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/__init__.py

@@ -49,7 +49,7 @@ from sct.streaming import (
 from sct.utils.anonymization_map import AnonymizationMap, MapEntry
 from sct.utils.process_result import ProcessResult
 
-__version__ = "0.7.0"
+__version__ = "0.7.1"
 __all__ = [
     "TextCleaner", "TextCleanerConfig",
     "PII_LABELS", "PII_LABEL_MAP",

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/adapters.py

@@ -6,6 +6,7 @@ from typing import Any, Iterable, Mapping, Optional
 
 from sct.privacy import EntityFinding
 from sct.sct import TextCleaner
+from sct.utils import constants
 
 
 @dataclass(frozen=True)
@@ -75,10 +76,16 @@ def is_sensitive_result(
         and result.lm_text != source
     ):
         return True
+    # The HTML step rewrites marker-like tags already in the source, so they
+    # cannot offset markers emitted later (``<EMAIL> Alice`` -> ``<PERSON>``).
+    baseline = source
+    if cleaner.cfg.check_replace_html:
+        baseline = constants.HTML_REGEX.sub("", source)
     configured_markers = (
         cleaner.cfg.replace_with_url,
         cleaner.cfg.replace_with_html,
         cleaner.cfg.replace_with_email,
+        cleaner.cfg.replace_with_ips,
         cleaner.cfg.replace_with_years,
         cleaner.cfg.replace_with_dates,
         cleaner.cfg.replace_with_phone_numbers,
@@ -88,14 +95,15 @@ def is_sensitive_result(
     if any(
         isinstance(marker, str)
         and marker
-        and marker in result.lm_text
-        and marker not in source
+        and result.lm_text.count(marker) > baseline.count(marker)
         for marker in configured_markers
     ):
         return True
-    return bool(
-        re.search(r"<[A-Z][A-Z0-9_]*>", result.lm_text)
-        and not re.search(r"<[A-Z][A-Z0-9_]*>", source)
+    # Compare counts, not presence: a source that already holds a marker
+    # still gains one per replacement.
+    marker_pattern = r"<[A-Z][A-Z0-9_]*>"
+    return len(re.findall(marker_pattern, result.lm_text)) > len(
+        re.findall(marker_pattern, baseline)
     )
 
 
@@ -122,9 +130,10 @@ def process_json(cleaner: TextCleaner, value: Any) -> StructuredResult:
             return result.lm_text
         if isinstance(item, dict):
             processed = {}
-            for key, child in item.items():
+            for index, (key, child) in enumerate(item.items()):
+                # The raw key may itself be PII, so its path uses the index.
                 processed_key = (
-                    visit(key, path + ("@key", key))
+                    visit(key, path + ("@key", index))
                     if isinstance(key, str) else key
                 )
                 if processed_key in processed:

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/browser.py

@@ -7,6 +7,7 @@ from typing import Any
 
 from sct.config import TextCleanerConfig
 from sct.utils import constants
+from sct.utils.contact import numbers_regex
 
 
 def _rule(pattern, replacement, enabled: bool) -> dict[str, Any]:
@@ -30,6 +31,51 @@ def build_browser_manifest(config: TextCleanerConfig) -> dict[str, Any]:
             "Browser export currently supports placeholder replacement only"
         )
 
+    if config.check_replace_ips and config.replace_ips_public_only:
+        raise ValueError(
+            "Browser export does not support replace_ips_public_only=True"
+        )
+
+    # The browser runs rules in order with nothing protecting emitted tokens,
+    # so a digit in one ("<EMAIL1>") would be rewritten by the number rules.
+    tokens = (
+        config.replace_with_html, config.replace_with_url, config.replace_with_email,
+        config.replace_with_ips, config.replace_with_dates,
+        config.replace_with_phone_numbers, config.replace_with_years,
+    )
+    if any(ch.isdigit() for token in tokens for ch in token):
+        raise ValueError("Browser export requires replacement tokens without digits")
+    if config.check_replace_currency_symbols and any(
+        constants.CURRENCY_REGEX.search(token) for token in tokens
+    ):
+        raise ValueError(
+            "Browser export requires replacement tokens without currency symbols"
+        )
+
+    # Same reason for any token a later enabled rule would match
+    # (an HTML token holding an email or URL, a URL token holding an email).
+    ordered = [
+        ("html", constants.HTML_REGEX, config.replace_with_html, config.check_replace_html),
+        ("url", constants.URL_REGEX, config.replace_with_url, config.check_replace_urls),
+        ("email", constants.EMAIL_REGEX, config.replace_with_email, config.check_replace_emails),
+        ("ip", constants.IP_REGEX, config.replace_with_ips, config.check_replace_ips),
+        ("date", constants.DATE_REGEX, config.replace_with_dates, config.check_replace_dates),
+        ("phone", constants.PHONE_REGEX, config.replace_with_phone_numbers,
+         config.check_replace_phone_numbers),
+        ("year", constants.YEAR_REGEX, config.replace_with_years, config.check_replace_years),
+        ("number", numbers_regex(config.replace_numbers_in_words), config.replace_with_numbers,
+         config.check_replace_numbers),
+        ("currency", constants.CURRENCY_REGEX, config.replace_with_currency_symbols,
+         config.check_replace_currency_symbols),
+    ]
+    for i, (name, _, token, enabled) in enumerate(ordered):
+        for later, pattern, _, later_enabled in ordered[i + 1:]:
+            if enabled and later_enabled and pattern.search(token):
+                raise ValueError(
+                    f"Browser export: the {name} replacement token would be "
+                    f"rewritten by the later {later} rule"
+                )
+
     model_revisions = config.ner_model_revisions or {}
     unpinned_models = sorted(
         key
@@ -54,7 +100,7 @@ def build_browser_manifest(config: TextCleanerConfig) -> dict[str, Any]:
         "runtime": "onnxruntime-web",
         "models": models,
         "rule_order": [
-            "html", "url", "email", "date", "year", "phone", "number",
+            "html", "url", "email", "ip", "date", "phone", "year", "number",
             "currency",
         ],
         "rules": {
@@ -73,6 +119,11 @@ def build_browser_manifest(config: TextCleanerConfig) -> dict[str, Any]:
                 config.replace_with_email,
                 config.check_replace_emails,
             ),
+            "ip": _rule(
+                constants.IP_REGEX,
+                config.replace_with_ips,
+                config.check_replace_ips,
+            ),
             "date": _rule(
                 constants.DATE_REGEX,
                 config.replace_with_dates,
@@ -89,7 +140,7 @@ def build_browser_manifest(config: TextCleanerConfig) -> dict[str, Any]:
                 config.check_replace_phone_numbers,
             ),
             "number": _rule(
-                constants.NUMBERS_REGEX,
+                numbers_regex(config.replace_numbers_in_words),
                 config.replace_with_numbers,
                 config.check_replace_numbers,
             ),

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/config.py

@@ -287,6 +287,12 @@ class TextCleanerConfig:
     # Computed at __post_init__ — union of all language sources
     supported_languages: frozenset = frozenset()
 
+    # Added in 0.7.1; kept last so existing positional arguments keep their meaning.
+    check_replace_ips: bool = True
+    replace_ips_public_only: bool = False  # True keeps private/loopback/reserved IPs
+    replace_with_ips: str = "<IP>"
+    replace_numbers_in_words: bool = False  # True also replaces digits glued to Latin/Greek/Cyrillic letters (GPT4)
+
     def __post_init__(self):
         if self.gliclass_onnx:
             raise ValueError(
@@ -357,6 +363,11 @@ class TextCleanerConfig:
                 object.__setattr__(self, 'gliner_labels', PII_LABELS)
             if self.gliner_threshold == 0.4:
                 object.__setattr__(self, 'gliner_threshold', PII_DEFAULT_THRESHOLD)
+            # The final ensemble filter must not drop what GLiNER kept.
+            if self.ner_confidence_threshold == 0.85:
+                object.__setattr__(
+                    self, 'ner_confidence_threshold', self.gliner_threshold,
+                )
             if self.gliner_label_map is None:
                 object.__setattr__(self, 'gliner_label_map', PII_LABEL_MAP)
             # Expand positional_tags to include all PII label tags
@@ -439,7 +450,17 @@ class TextCleanerConfig:
                     f"ner_models must include {sorted(REQUIRED_NER_KEYS)}, "
                     f"missing: {sorted(missing)}"
                 )
-            merged = {**DEFAULT_NER_MODELS, **self.ner_models}
+            # A map of exactly LANG_KEYS matching ner_models_list is a normalized
+            # legacy/default config (replace(), copy): keep it exact. Decided by
+            # content only, so a dict and a mapping proxy give the same config.
+            derived = (
+                set(self.ner_models) == set(LANG_KEYS)
+                and tuple(self.ner_models[k] for k in LANG_KEYS) == self.ner_models_list
+            )
+            merged = (
+                dict(self.ner_models) if derived
+                else {**DEFAULT_NER_MODELS, **self.ner_models}
+            )
             object.__setattr__(self, 'ner_models', MappingProxyType(merged))
             object.__setattr__(
                 self, 'ner_models_list',
@@ -532,6 +553,21 @@ class TextCleanerConfig:
                             f"{sorted(self.supported_languages)}"
                         )
 
+    def __getstate__(self):
+        # Restore the exact normalized state; never re-run __post_init__.
+        state = dict(self.__dict__)
+        frozen = [k for k, v in state.items() if isinstance(v, MappingProxyType)]
+        for key in frozen:
+            state[key] = dict(state[key])
+        return state, frozen
+
+    def __setstate__(self, packed):
+        state, frozen = packed
+        for key, value in state.items():
+            if key in frozen:
+                value = MappingProxyType(value)
+            object.__setattr__(self, key, value)
+
 
 def _config_from_module_globals() -> TextCleanerConfig:
     """Build a TextCleanerConfig from the current module-level variable state.
@@ -546,12 +582,15 @@ def _config_from_module_globals() -> TextCleanerConfig:
         check_replace_html=m.CHECK_REPLACE_HTML,
         check_replace_urls=m.CHECK_REPLACE_URLS,
         check_replace_emails=m.CHECK_REPLACE_EMAILS,
+        check_replace_ips=getattr(m, 'CHECK_REPLACE_IPS', True),
+        replace_ips_public_only=getattr(m, 'REPLACE_IPS_PUBLIC_ONLY', False),
         check_replace_years=m.CHECK_REPLACE_YEARS,
         check_replace_dates=getattr(m, 'CHECK_REPLACE_DATES', False),
         check_fuzzy_replace_dates=getattr(m, 'CHECK_FUZZY_REPLACE_DATES', False),
         fuzzy_date_score_cutoff=getattr(m, 'FUZZY_DATE_SCORE_CUTOFF', 85),
         check_replace_phone_numbers=m.CHECK_REPLACE_PHONE_NUMBERS,
         check_replace_numbers=m.CHECK_REPLACE_NUMBERS,
+        replace_numbers_in_words=getattr(m, 'REPLACE_NUMBERS_IN_WORDS', False),
         check_replace_currency_symbols=m.CHECK_REPLACE_CURRENCY_SYMBOLS,
         check_ner_process=m.CHECK_NER_PROCESS,
         check_remove_isolated_letters=m.CHECK_REMOVE_ISOLATED_LETTERS,
@@ -569,6 +608,7 @@ def _config_from_module_globals() -> TextCleanerConfig:
         replace_with_url=m.REPLACE_WITH_URL,
         replace_with_html=m.REPLACE_WITH_HTML,
         replace_with_email=m.REPLACE_WITH_EMAIL,
+        replace_with_ips=getattr(m, 'REPLACE_WITH_IPS', '<IP>'),
         replace_with_years=m.REPLACE_WITH_YEARS,
         replace_with_dates=getattr(m, 'REPLACE_WITH_DATES', '<DATE>'),
         replace_with_phone_numbers=m.REPLACE_WITH_PHONE_NUMBERS,
@@ -597,12 +637,15 @@ CHECK_TO_ASCII_UNICODE = True
 CHECK_REPLACE_HTML = True
 CHECK_REPLACE_URLS = True
 CHECK_REPLACE_EMAILS = True
+CHECK_REPLACE_IPS = True
+REPLACE_IPS_PUBLIC_ONLY = False
 CHECK_REPLACE_YEARS = True
 CHECK_REPLACE_DATES = False
 CHECK_FUZZY_REPLACE_DATES = False
 FUZZY_DATE_SCORE_CUTOFF = 85
 CHECK_REPLACE_PHONE_NUMBERS = True
 CHECK_REPLACE_NUMBERS = True
+REPLACE_NUMBERS_IN_WORDS = False
 CHECK_REPLACE_CURRENCY_SYMBOLS = True
 CHECK_NER_PROCESS = True
 CHECK_REMOVE_ISOLATED_LETTERS = True
@@ -621,6 +664,7 @@ CHECK_REMOVE_EMOJI = False
 REPLACE_WITH_URL = "<URL>"
 REPLACE_WITH_HTML = "<HTML>"
 REPLACE_WITH_EMAIL = "<EMAIL>"
+REPLACE_WITH_IPS = "<IP>"
 REPLACE_WITH_YEARS = "<YEAR>"
 REPLACE_WITH_DATES = "<DATE>"
 REPLACE_WITH_PHONE_NUMBERS = "<PHONE>"

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/model_benchmark.py

@@ -56,6 +56,7 @@ def candidate_config(candidate: ModelCandidate) -> TextCleanerConfig:
         "check_replace_html": False,
         "check_replace_urls": False,
         "check_replace_emails": False,
+        "check_replace_ips": False,
         "check_replace_years": False,
         "check_replace_dates": False,
         "check_replace_phone_numbers": False,
@@ -82,6 +83,7 @@ def candidate_config(candidate: ModelCandidate) -> TextCleanerConfig:
             gliner_labels=PII_LABELS,
             gliner_label_map=PII_LABEL_MAP,
             gliner_threshold=0.3,
+            ner_confidence_threshold=0.3,
             gliner_onnx=candidate.supports_onnx,
         )
     if candidate.backend == "torch":

{squeakycleantext-0.7.0 → squeakycleantext-0.7.1}/sct/privacy.py

@@ -42,7 +42,11 @@ class InMemoryTokenStore:
 
 @dataclass(frozen=True)
 class DetectedEntity:
-    """One entity produced by a detector."""
+    """One entity produced by a detector.
+
+    ``start``/``end`` index the post-regex text (after the regex replacement
+    stages ran), not the raw input passed to ``process()``.
+    """
 
     entity_type: str
     score: float