pii-eraser 0.1.0__tar.gz

pii_eraser-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+.venv/
+__pycache__/
+*.egg-info/
+dist/
+.pytest_cache/
+.ruff_cache/

pii_eraser-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 37lab OÜ
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pii_eraser-0.1.0/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.5
+Name: pii-eraser
+Version: 0.1.0
+Summary: Find personal data and secrets in text and code, replace them with stable placeholders, locally.
+Project-URL: Homepage, https://37lab.ai
+Project-URL: Repository, https://github.com/tkenaz/pii-eraser
+Author-email: 37lab OÜ <mv@37lab.ai>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anonymization,gdpr,llm,pii,privacy,redaction,secrets
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Security
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: gliner
+Requires-Dist: gliner>=0.2.20; extra == 'gliner'
+Description-Content-Type: text/markdown
+
+# pii-eraser
+
+Find personal data and secrets in text and code. Replace them with stable
+placeholders. Do it locally, before the text goes to a model, a log or a vendor.
+
+```
+Contact Jane Doe <jane.doe@example.org>, key AKIA…      →   Contact [NAME_1] <[EMAIL_1]>, key [SECRET_1]
+```
+
+- **Local.** Regex rules and an optional NER model run on your machine. No network.
+- **Stable.** A session keeps one placeholder table across many texts:
+  `Jane Doe` is `[NAME_1]` in the diff, in every file an agent reads afterwards,
+  and in the next file, even where no rule fires again.
+- **Code-aware.** The code profile drops rules that misfire on source (dates in
+  changelogs, numbers that look like phones) and checks card numbers (Luhn) and
+  IBANs (mod-97). Line counts never change, so line numbers in a review still hold.
+- **Reversible for the owner.** `session.restore(text)` puts the originals back
+  into a model's answer, on your side.
+
+## Install
+
+```bash
+pip install pii-eraser              # regex rules only, no dependencies
+pip install 'pii-eraser[gliner]'    # + GLiNER NER for names and organisations (~400 MB model)
+```
+
+## Use
+
+```python
+from pii_eraser import Eraser
+
+eraser = Eraser.for_prose()  # documents, emails, MR descriptions
+result = eraser.erase("Name: Jane Doe, jane.doe@example.org")
+result.text  # 'Name: [NAME_1], [EMAIL_1]'
+result.mapping  # {'[NAME_1]': 'Jane Doe', '[EMAIL_1]': 'jane.doe@example.org'}
+
+session = Eraser.for_code().session()  # many texts, one table
+session.erase(diff_text)
+session.erase(file_the_agent_read)
+answer = session.restore(model_answer)
+```
+
+With NER:
+
+```python
+from pii_eraser.detectors.gliner import GlinerDetector
+
+Eraser.for_prose(GlinerDetector())  # NER hits are replaced
+Eraser.for_code(GlinerDetector())  # NER hits are only reported in result.signals
+```
+
+On source code an NER model tags variable names and authors as people. In the
+code profile its hits are signals for a human, not replacements.
+
+## CLI
+
+```bash
+pii-eraser contract.txt > contract.clean.txt
+git diff origin/main | pii-eraser --profile code --report --fail-on-secret > /dev/null
+```
+
+The report on stderr lists kind, line and rule, never the original value, so it
+is safe for CI logs. `--fail-on-secret` exits 1 when a secret is found.
+
+## What it finds
+
+| Kind | Examples |
+|---|---|
+| `SECRET` | private key blocks, AWS / GitHub / GitLab / Slack / Anthropic / OpenAI / Google / Stripe keys, JWTs, passwords in URLs, `password = "…"` / `API_KEY=…` assignments, long random quoted strings |
+| `EMAIL`, `PHONE` | emails; international, US, Swiss, German phone formats |
+| `NAME` | labelled names, titles, salutations, signature blocks, contract parties; with NER: any person or organisation |
+| `ADDRESS` | US street / city / state / ZIP, UK postcodes, German streets, labelled addresses |
+| `FINANCIAL` | payment cards (Luhn), IBAN (mod-97), account, routing, tax ids |
+| `SSN`, `NATIONAL_ID`, `DOB`, `MRN`, `HEALTH_PLAN_ID` | US SSN, Swiss AHV, Spanish NIE/DNI, UK NI, passports, licences, dates of birth, medical record and insurance numbers (EN/DE) |
+
+References to secrets are left alone: `${DB_PASSWORD}`, `<your-token>`,
+`os.environ['SECRET']`, `changeme`.
+
+## Limits
+
+Detection is heuristic. Rules miss formats they don't know; NER misses rare
+names and catches some non-names. Treat the output as reduced exposure, not as
+anonymisation in the legal sense, and keep a secret scanner such as gitleaks in
+your pipeline: a leaked key must be rotated, not just masked.
+
+## Origin
+
+The personal-data rules and NER filters come from the scrubber of
+37lab's contract analyzer, tuned on real contracts, medical notes and business
+email. The secret rules, sessions and the code profile were added for agentic
+code review in [semantic-diff](https://github.com/tkenaz/semantic_diff).
+
+## License
+
+MIT — [37lab](https://37lab.ai). The default NER model
+`urchade/gliner_medium-v2.1` is Apache-2.0.

pii_eraser-0.1.0/README.md

@@ -0,0 +1,96 @@
+# pii-eraser
+
+Find personal data and secrets in text and code. Replace them with stable
+placeholders. Do it locally, before the text goes to a model, a log or a vendor.
+
+```
+Contact Jane Doe <jane.doe@example.org>, key AKIA…      →   Contact [NAME_1] <[EMAIL_1]>, key [SECRET_1]
+```
+
+- **Local.** Regex rules and an optional NER model run on your machine. No network.
+- **Stable.** A session keeps one placeholder table across many texts:
+  `Jane Doe` is `[NAME_1]` in the diff, in every file an agent reads afterwards,
+  and in the next file, even where no rule fires again.
+- **Code-aware.** The code profile drops rules that misfire on source (dates in
+  changelogs, numbers that look like phones) and checks card numbers (Luhn) and
+  IBANs (mod-97). Line counts never change, so line numbers in a review still hold.
+- **Reversible for the owner.** `session.restore(text)` puts the originals back
+  into a model's answer, on your side.
+
+## Install
+
+```bash
+pip install pii-eraser              # regex rules only, no dependencies
+pip install 'pii-eraser[gliner]'    # + GLiNER NER for names and organisations (~400 MB model)
+```
+
+## Use
+
+```python
+from pii_eraser import Eraser
+
+eraser = Eraser.for_prose()  # documents, emails, MR descriptions
+result = eraser.erase("Name: Jane Doe, jane.doe@example.org")
+result.text  # 'Name: [NAME_1], [EMAIL_1]'
+result.mapping  # {'[NAME_1]': 'Jane Doe', '[EMAIL_1]': 'jane.doe@example.org'}
+
+session = Eraser.for_code().session()  # many texts, one table
+session.erase(diff_text)
+session.erase(file_the_agent_read)
+answer = session.restore(model_answer)
+```
+
+With NER:
+
+```python
+from pii_eraser.detectors.gliner import GlinerDetector
+
+Eraser.for_prose(GlinerDetector())  # NER hits are replaced
+Eraser.for_code(GlinerDetector())  # NER hits are only reported in result.signals
+```
+
+On source code an NER model tags variable names and authors as people. In the
+code profile its hits are signals for a human, not replacements.
+
+## CLI
+
+```bash
+pii-eraser contract.txt > contract.clean.txt
+git diff origin/main | pii-eraser --profile code --report --fail-on-secret > /dev/null
+```
+
+The report on stderr lists kind, line and rule, never the original value, so it
+is safe for CI logs. `--fail-on-secret` exits 1 when a secret is found.
+
+## What it finds
+
+| Kind | Examples |
+|---|---|
+| `SECRET` | private key blocks, AWS / GitHub / GitLab / Slack / Anthropic / OpenAI / Google / Stripe keys, JWTs, passwords in URLs, `password = "…"` / `API_KEY=…` assignments, long random quoted strings |
+| `EMAIL`, `PHONE` | emails; international, US, Swiss, German phone formats |
+| `NAME` | labelled names, titles, salutations, signature blocks, contract parties; with NER: any person or organisation |
+| `ADDRESS` | US street / city / state / ZIP, UK postcodes, German streets, labelled addresses |
+| `FINANCIAL` | payment cards (Luhn), IBAN (mod-97), account, routing, tax ids |
+| `SSN`, `NATIONAL_ID`, `DOB`, `MRN`, `HEALTH_PLAN_ID` | US SSN, Swiss AHV, Spanish NIE/DNI, UK NI, passports, licences, dates of birth, medical record and insurance numbers (EN/DE) |
+
+References to secrets are left alone: `${DB_PASSWORD}`, `<your-token>`,
+`os.environ['SECRET']`, `changeme`.
+
+## Limits
+
+Detection is heuristic. Rules miss formats they don't know; NER misses rare
+names and catches some non-names. Treat the output as reduced exposure, not as
+anonymisation in the legal sense, and keep a secret scanner such as gitleaks in
+your pipeline: a leaked key must be rotated, not just masked.
+
+## Origin
+
+The personal-data rules and NER filters come from the scrubber of
+37lab's contract analyzer, tuned on real contracts, medical notes and business
+email. The secret rules, sessions and the code profile were added for agentic
+code review in [semantic-diff](https://github.com/tkenaz/semantic_diff).
+
+## License
+
+MIT — [37lab](https://37lab.ai). The default NER model
+`urchade/gliner_medium-v2.1` is Apache-2.0.

pii_eraser-0.1.0/pii_eraser/__init__.py

@@ -0,0 +1,9 @@
+"""pii-eraser: find personal data and secrets in text and replace them with
+stable placeholders, locally, before the text goes anywhere."""
+
+from pii_eraser.eraser import Eraser, ErasingSession, Stage
+from pii_eraser.types import EraseResult, Kind, Span
+
+__version__ = "0.1.0"
+
+__all__ = ["Eraser", "ErasingSession", "Stage", "EraseResult", "Kind", "Span", "__version__"]

pii_eraser-0.1.0/pii_eraser/cli.py

@@ -0,0 +1,60 @@
+"""pii-eraser FILE|-  →  erased text on stdout, a report on stderr.
+
+The report names kind, line and rule, never the original value: it is meant
+for CI logs.
+"""
+
+import argparse
+import json
+import sys
+
+from pii_eraser.eraser import Eraser
+from pii_eraser.types import EraseResult, Kind
+
+
+def _line_of(text: str, offset: int) -> int:
+    return text.count("\n", 0, offset) + 1
+
+
+def _report(original: str, result: EraseResult) -> dict:
+    def row(span):
+        return {"kind": span.kind.value, "line": _line_of(original, span.start), "rule": span.source}
+
+    return {
+        "found": [row(s) for s in result.found],
+        "signals": [row(s) for s in result.signals],
+        "secrets": sum(1 for s in result.found if s.kind is Kind.SECRET),
+    }
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(prog="pii-eraser", description=__doc__.split("\n")[0])
+    parser.add_argument("path", nargs="?", default="-", help="file to read, '-' for stdin")
+    parser.add_argument("--profile", choices=("prose", "code"), default="prose")
+    parser.add_argument("--ner", action="store_true", help="add GLiNER (pip install 'pii-eraser[gliner]')")
+    parser.add_argument("--report", action="store_true", help="JSON report on stderr")
+    parser.add_argument("--fail-on-secret", action="store_true", help="exit 1 when a secret is found")
+    args = parser.parse_args(argv)
+
+    text = sys.stdin.read() if args.path == "-" else open(args.path, encoding="utf-8").read()
+
+    ner = None
+    if args.ner:
+        from pii_eraser.detectors.gliner import GlinerDetector
+
+        ner = GlinerDetector()
+    eraser = Eraser.for_code(ner) if args.profile == "code" else Eraser.for_prose(ner)
+    result = eraser.erase(text)
+
+    sys.stdout.write(result.text)
+    report = _report(text, result)
+    if args.report:
+        print(json.dumps(report, indent=2), file=sys.stderr)
+    if args.fail_on_secret and report["secrets"]:
+        print(f"pii-eraser: {report['secrets']} secret(s) found", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

pii_eraser-0.1.0/pii_eraser/detectors/__init__.py

@@ -0,0 +1,4 @@
+from pii_eraser.detectors.base import Detector
+from pii_eraser.detectors.regex import RegexDetector, Rule
+
+__all__ = ["Detector", "RegexDetector", "Rule"]

pii_eraser-0.1.0/pii_eraser/detectors/base.py

@@ -0,0 +1,13 @@
+"""The one interface every detector implements."""
+
+from typing import List, Protocol
+
+from pii_eraser.types import Span
+
+
+class Detector(Protocol):
+    name: str
+
+    def detect(self, text: str) -> List[Span]:
+        """Return candidate spans with absolute offsets into `text`."""
+        ...

pii_eraser-0.1.0/pii_eraser/detectors/gliner.py

@@ -0,0 +1,335 @@
+"""NER detection with GLiNER (zero-shot labels, runs locally, ~400 MB model).
+
+Ported from the contract-analyzer scrubber. Install with `pip install
+pii-eraser[gliner]`; the model downloads on first use.
+"""
+
+import logging
+from typing import Dict, List, Optional, Sequence
+
+from pii_eraser.types import Kind, Span
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_MODEL = "urchade/gliner_medium-v2.1"  # Apache-2.0
+
+LABEL_TO_KIND: Dict[str, Kind] = {
+    "person": Kind.NAME,
+    "organization": Kind.NAME,
+    "company": Kind.NAME,
+    "corporation": Kind.NAME,
+    "government": Kind.NAME,
+    "county": Kind.NAME,
+    "city": Kind.NAME,
+    "municipality": Kind.NAME,
+    "location": Kind.ADDRESS,
+    "address": Kind.ADDRESS,
+    "ssn": Kind.SSN,
+    "tax_id": Kind.FINANCIAL,
+    "passport_number": Kind.NATIONAL_ID,
+    "drivers_license": Kind.NATIONAL_ID,
+    "bank_account": Kind.FINANCIAL,
+    "routing_number": Kind.FINANCIAL,
+    "credit_card": Kind.FINANCIAL,
+    "phone_number": Kind.PHONE,
+    "email": Kind.EMAIL,
+    "date_of_birth": Kind.DOB,
+}
+
+DEFAULT_LABELS = list(LABEL_TO_KIND)
+
+# Role nouns, pronouns and bare titles that NER tags as person/organization but
+# that never identify anyone. Scrubbing them corrupts quotes ("[NAME_5] owning
+# 95% of the Shares") and a downstream model then reports the corruption.
+GENERIC_TERMS = frozenset(
+    {
+        "client",
+        "contractor",
+        "vendor",
+        "supplier",
+        "customer",
+        "employee",
+        "employees",
+        "employer",
+        "party",
+        "parties",
+        "company",
+        "corporation",
+        "llc",
+        "inc",
+        "ltd",
+        "agreement",
+        "contract",
+        "document",
+        "plaintiff",
+        "defendant",
+        "attorney",
+        "counsel",
+        "licensor",
+        "licensee",
+        "lessor",
+        "lessee",
+        "buyer",
+        "seller",
+        "purchaser",
+        "shareholder",
+        "shareholders",
+        "stockholder",
+        "stockholders",
+        "member",
+        "members",
+        "director",
+        "directors",
+        "officer",
+        "officers",
+        "authorized officer",
+        "founder",
+        "founders",
+        "partner",
+        "partners",
+        "trustee",
+        "beneficiary",
+        "executor",
+        "administrator",
+        "guarantor",
+        "borrower",
+        "lender",
+        "investor",
+        "investors",
+        "signatory",
+        "signatories",
+        "witness",
+        "board",
+        "board of directors",
+        "the board",
+        "person",
+        "persons",
+        "individual",
+        "individuals",
+        "entity",
+        "entities",
+        "the undersigned",
+        "undersigned",
+        "each party",
+        "either party",
+        "i",
+        "we",
+        "you",
+        "he",
+        "she",
+        "they",
+        "heirs",
+        "successors",
+        "assigns",
+        "spouse",
+        "affiliate",
+        "affiliates",
+        "cto",
+        "ceo",
+        "cfo",
+        "coo",
+        "cio",
+        "president",
+        "chairman",
+        "chairperson",
+        "secretary",
+        "treasurer",
+        "general counsel",
+        "managing director",
+        "email address",
+        "e-mail address",
+        "facsimile number",
+        "fax number",
+        "telephone number",
+        "phone number",
+        "mobile number",
+        "capital account",
+        "bank name",
+        "financial institution",
+        "account number",
+        "registered address",
+        "principal place of business",
+        "transferee",
+        "transferees",
+        "transferor",
+        "addressee",
+        "recipient",
+        "recipients",
+        "someone",
+        "anyone",
+        "competitor",
+        "competitors",
+        "representative",
+        "representatives",
+        "court",
+        "courts",
+        "natural person",
+        "human being",
+        "personal representative",
+        "personal representatives",
+        "prevailing party",
+        # software roles and words NER tags as people in code and commit messages
+        "user",
+        "users",
+        "admin",
+        "author",
+        "maintainer",
+        "owner",
+        "reviewer",
+        "developer",
+        "service",
+        "server",
+        "client id",
+    }
+)
+
+# Words that do not make a phrase a name: "New Shareholder", "Each party".
+_STOP_MODIFIERS = frozenset(
+    {
+        "new",
+        "each",
+        "every",
+        "any",
+        "such",
+        "said",
+        "the",
+        "a",
+        "an",
+        "all",
+        "other",
+        "proposed",
+        "purported",
+        "prevailing",
+        "transferring",
+        "natural",
+        "personal",
+        "authorized",
+        "issuing",
+        "federal",
+        "state",
+        "or",
+        "and",
+        "of",
+        "to",
+        "this",
+        "that",
+        "no",
+        "non",
+        "former",
+        "future",
+        "existing",
+    }
+)
+
+_NOT_NAME_WORDS = GENERIC_TERMS | _STOP_MODIFIERS
+
+_NAMEY_LABELS = frozenset(
+    {
+        "person",
+        "organization",
+        "company",
+        "corporation",
+        "government",
+        "county",
+        "city",
+        "municipality",
+    }
+)
+
+_LABEL_WORDS = {
+    "ssn": {"ssn", "social security", "social security number"},
+    "tax_id": {"tax id", "tax", "ein"},
+    "passport_number": {"passport", "passport number"},
+    "drivers_license": {"driver's license", "drivers license", "driver license", "dl"},
+    "bank_account": {"bank account", "account", "bank"},
+    "routing_number": {"routing", "routing number"},
+    "phone_number": {"phone", "telephone", "tel", "mobile", "cell"},
+    "email": {"email", "e-mail"},
+    "address": {"address"},
+    "date_of_birth": {"dob", "date of birth", "birthdate"},
+}
+
+
+def is_generic_name_phrase(text: str, label: str) -> bool:
+    """True when a person/org hit holds no actual name: only role nouns, stop
+    modifiers and lowercase words. 'transferring Shareholder' -> True,
+    'Tunnel King Studios Inc.' -> False."""
+    if label not in _NAMEY_LABELS:
+        return False
+    for token in text.split():
+        bare = token.strip(".,;:()\"'’").rstrip(".")
+        if bare and bare[0].isupper() and bare.lower() not in _NOT_NAME_WORDS:
+            return False
+    return True
+
+
+def is_noise(text: str, label: str) -> bool:
+    low = text.lower().strip().strip(".,;:")
+    if low in GENERIC_TERMS:
+        return True
+    for article in ("the ", "a ", "an ", "such ", "said ", "any ", "each ", "every "):
+        if low.startswith(article) and low[len(article) :] in GENERIC_TERMS:
+            return True
+    return is_generic_name_phrase(text, label) or low in _LABEL_WORDS.get(label, ())
+
+
+class GlinerDetector:
+    name = "gliner"
+
+    def __init__(
+        self,
+        model_id: str = DEFAULT_MODEL,
+        labels: Optional[Sequence[str]] = None,
+        # 0.5: below it the model floods the table with generic phrases scored
+        # as names; rare surnames it under-scores are left to the regex rules.
+        threshold: float = 0.5,
+        chunk_size: int = 1500,  # characters; the model truncates near 384 tokens
+        overlap: int = 200,
+        model=None,  # injected for tests
+    ):
+        self.model_id = model_id
+        self.labels = list(labels or DEFAULT_LABELS)
+        self.threshold = threshold
+        self.chunk_size = chunk_size
+        self.overlap = overlap
+        self._model = model
+
+    def _load(self):
+        if self._model is None:
+            try:
+                from gliner import GLiNER
+            except ImportError as e:
+                raise ImportError("GLiNER is not installed: pip install 'pii-eraser[gliner]'") from e
+            logger.info("loading GLiNER model %s", self.model_id)
+            self._model = GLiNER.from_pretrained(self.model_id)
+        return self._model
+
+    def detect(self, text: str) -> List[Span]:
+        model = self._load()
+        spans: Dict[tuple, Span] = {}
+        start = 0
+        while start < len(text):
+            end = min(start + self.chunk_size, len(text))
+            chunk = text[start:end]
+            for entity in model.predict_entities(chunk, self.labels, threshold=self.threshold):
+                label, value = entity["label"], entity["text"]
+                if is_noise(value, label):
+                    continue
+                local = entity.get("start")
+                if local is None or chunk[local : local + len(value)] != value:
+                    local = chunk.find(value)
+                    if local == -1:
+                        continue
+                abs_start = start + local
+                spans[(abs_start, value)] = Span(
+                    start=abs_start,
+                    end=abs_start + len(value),
+                    text=value,
+                    kind=LABEL_TO_KIND.get(label, Kind.NAME),
+                    confidence=float(entity["score"]),
+                    source=f"gliner:{label}",
+                )
+            if end >= len(text):
+                break
+            start = end - self.overlap
+        return list(spans.values())