dpdp-scrub 0.0.1__tar.gz

dpdp_scrub-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 dpdp-scrub contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dpdp_scrub-0.0.1/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: dpdp-scrub
+Version: 0.0.1
+Summary: Detect, validate, and redact Indian PII (Aadhaar, PAN, GSTIN, UPI, and more) — self-hosted, checksum-validated, DPDP-ready.
+Author: Palkush Dave
+License: MIT
+Keywords: pii,dpdp,aadhaar,pan,gstin,privacy,redaction,india
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Security
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Dynamic: license-file
+
+# dpdp-scrub
+
+**Detect, validate, and redact Indian PII before it reaches an LLM — self-hosted, checksum-validated, DPDP-ready.**
+
+Your org keeps the keys; the LLM gets the placeholders.
+
+```
+IN  : Sir aadhaar 2345 6789 0124 hai, call karo 9876543210 pe, PAN ABCPE1234F
+LLM : Sir aadhaar XXXX XXXX 0124 hai, call karo <PHONE_IN_1> pe, PAN <PAN_1>
+BACK: Done! 9876543210 pe call schedule ho gaya.        ← rehydrated, agent-side
+```
+
+## Why this exists
+
+Every Indian company wiring LLMs into support, sales, and ops is streaming
+Aadhaar numbers, PANs, GSTINs, UPI IDs, and phone numbers to third-party model
+providers — and copying them into logs, vector DBs, and fine-tuning sets.
+The DPDP Act's substantive obligations become enforceable in **May 2027**
+(penalties up to ₹250 crore); UIDAI's Aadhaar-masking rules and RBI's card
+rules already apply.
+
+The redaction loop itself is well-established engineering (Presidio, Google
+DLP). What didn't exist is **eyes that can see Indian PII** — validated,
+context-aware, and open. Google Cloud DLP ships 3 Indian infoTypes, AWS
+Comprehend 4, Presidio 6. dpdp-scrub detects **15**, most with mathematical
+or reference-data validation:
+
+| Entity | Validation |
+|---|---|
+| Aadhaar / Aadhaar VID | **Verhoeff checksum** + first-digit rule |
+| GSTIN | **mod-36 check character** + embedded-PAN structure + state code |
+| Credit/debit card | **Luhn checksum** |
+| PAN | holder-type structure rule |
+| UPI ID | handle verified against **known PSP list** (`@ybl`, `@oksbi`, …) |
+| IFSC, phone, email, voter ID, passport | structural + context scoring |
+| Vehicle registration | state/RTO code verified against reference list (incl. BH series) |
+| Bank account, EPF UAN, ABHA | **context-required** (surface only with evidence like "A/c", "UAN") |
+
+The context engine is what separates a phone number from an order ID:
+
+```
+order 9876543210 ka status batao   →  nothing        (suppressed by "order")
+call karo 9876543210 pe            →  PHONE_IN 0.99  (boosted by "call")
+mera a/c 34512345678901 hai        →  BANK_ACCOUNT   (earned by "a/c")
+code 34512345678901 likha hai      →  nothing
+```
+
+## Quick start
+
+```python
+from dpdp_scrub import Scrubber
+
+s = Scrubber()
+
+result = s.scrub("mera PAN ABCPE1234F hai, call 9876543210", session_id="ticket-42")
+result.text    # "mera PAN <PAN_1> hai, call <PHONE_IN_1>"
+result.spans   # typed spans: entity, offsets, confidence
+
+reply = call_your_llm(result.text)          # provider never sees real values
+s.rehydrate(reply, "ticket-42")             # real values restored, your side only
+```
+
+Detection only:
+
+```python
+from dpdp_scrub import detect
+detect("payment 9876543210@ybl pe karo")
+# [Span(entity='UPI_ID', start=8, end=22, ..., confidence=0.92)]
+```
+
+Zero runtime dependencies. Python 3.9+.
+
+## Trust model
+
+```
+Ring 0: the vault (your Redis/memory)   — real values, session-scoped, droppable
+Ring 1: your org's services             — placeholders + scoped right to resolve
+Ring 2: LLM providers, logs, vector DBs — placeholders only, no resolution path
+```
+
+- Same value → same placeholder within a session (multi-turn consistency).
+- `vault.drop(session)` makes every old copy of its placeholders permanently
+  unresolvable — retroactive redaction for logs you already wrote.
+- No cloud, no telemetry, no phoning home. This is a library, not a service:
+  **your data never leaves your infrastructure.**
+
+Per-entity actions are policy: `tokenize` (reversible), `mask` (UIDAI-format
+`XXXX XXXX 0124` for Aadhaar, last-4 for cards), `redact` (ABHA health IDs by
+default), `ignore`.
+
+## Benchmark
+
+IndicPII-Bench v0 (in `bench/`): 500 synthetic messages — Hinglish, English,
+Devanagari, OCR-noise spacing, multi-entity — with 20% traps (order IDs,
+invoice refs, phones-in-txn-context). Presidio runs at its best
+configuration — its IN_* recognizers explicitly registered (they are shipped
+but **not loaded by default**). F1, value-level matching:
+
+| entity | dpdp-scrub | presidio |
+|---|---|---|
+| AADHAAR | **1.000** | 1.000 |
+| GSTIN | **1.000** | 0.971 |
+| PAN | **1.000** | 0.383 |
+| PHONE_IN | **1.000** | 0.602 |
+| UPI_ID | **1.000** | 0.000 |
+| IFSC | **1.000** | 0.000 |
+| EMAIL | **1.000** | 1.000 |
+| **OVERALL** | **1.000** | 0.595 |
+| trap false positives | **0/100** | 200/100 |
+
+Honest caveats: this is our own v0 corpus (templated, synthetic, Hinglish-
+heavy) — a perfect self-score mainly means the corpus is still too easy, and
+it will get harder (Devanagari, OCR noise, multi-entity). Presidio was not
+built for code-mixed text; the 107 trap FPs come from it having no context
+suppression (e.g. any 10-digit number near "txn" flags as phone). Reproduce:
+`python3 bench/compare.py 500`.
+
+## Status
+
+Pre-alpha, moving fast. Built so far: 15 entities, checksum validators,
+context-scoring engine, scrub/rehydrate loop with session vault — 86 tests,
+all synthetic data. Roadmap:
+
+- [ ] YAML policy profiles (`dpdp-strict`, `uidai-mask`, `audit-only`) + audit log
+- [ ] `scrub_json()` for structured payloads
+- [ ] Redis vault backend
+- [ ] LiteLLM / LangChain / Presidio adapters
+- [ ] **IndicPII-Bench** — a public benchmark for Indian PII detection
+- [ ] Indic-script & code-mixed NER for names/addresses (AI4Bharat IndicNER)
+
+## Contributing
+
+The reference-data files are designed for one-line PRs: new UPI PSP handles,
+RTO codes, context words (English/Hindi/regional). Entity requests welcome —
+include format, any checksum, and public documentation.
+
+All test data must be synthetic (checksum-generated) or published samples.
+Never commit real PII, including your own.
+
+## License
+
+MIT

dpdp_scrub-0.0.1/README.md

@@ -0,0 +1,142 @@
+# dpdp-scrub
+
+**Detect, validate, and redact Indian PII before it reaches an LLM — self-hosted, checksum-validated, DPDP-ready.**
+
+Your org keeps the keys; the LLM gets the placeholders.
+
+```
+IN  : Sir aadhaar 2345 6789 0124 hai, call karo 9876543210 pe, PAN ABCPE1234F
+LLM : Sir aadhaar XXXX XXXX 0124 hai, call karo <PHONE_IN_1> pe, PAN <PAN_1>
+BACK: Done! 9876543210 pe call schedule ho gaya.        ← rehydrated, agent-side
+```
+
+## Why this exists
+
+Every Indian company wiring LLMs into support, sales, and ops is streaming
+Aadhaar numbers, PANs, GSTINs, UPI IDs, and phone numbers to third-party model
+providers — and copying them into logs, vector DBs, and fine-tuning sets.
+The DPDP Act's substantive obligations become enforceable in **May 2027**
+(penalties up to ₹250 crore); UIDAI's Aadhaar-masking rules and RBI's card
+rules already apply.
+
+The redaction loop itself is well-established engineering (Presidio, Google
+DLP). What didn't exist is **eyes that can see Indian PII** — validated,
+context-aware, and open. Google Cloud DLP ships 3 Indian infoTypes, AWS
+Comprehend 4, Presidio 6. dpdp-scrub detects **15**, most with mathematical
+or reference-data validation:
+
+| Entity | Validation |
+|---|---|
+| Aadhaar / Aadhaar VID | **Verhoeff checksum** + first-digit rule |
+| GSTIN | **mod-36 check character** + embedded-PAN structure + state code |
+| Credit/debit card | **Luhn checksum** |
+| PAN | holder-type structure rule |
+| UPI ID | handle verified against **known PSP list** (`@ybl`, `@oksbi`, …) |
+| IFSC, phone, email, voter ID, passport | structural + context scoring |
+| Vehicle registration | state/RTO code verified against reference list (incl. BH series) |
+| Bank account, EPF UAN, ABHA | **context-required** (surface only with evidence like "A/c", "UAN") |
+
+The context engine is what separates a phone number from an order ID:
+
+```
+order 9876543210 ka status batao   →  nothing        (suppressed by "order")
+call karo 9876543210 pe            →  PHONE_IN 0.99  (boosted by "call")
+mera a/c 34512345678901 hai        →  BANK_ACCOUNT   (earned by "a/c")
+code 34512345678901 likha hai      →  nothing
+```
+
+## Quick start
+
+```python
+from dpdp_scrub import Scrubber
+
+s = Scrubber()
+
+result = s.scrub("mera PAN ABCPE1234F hai, call 9876543210", session_id="ticket-42")
+result.text    # "mera PAN <PAN_1> hai, call <PHONE_IN_1>"
+result.spans   # typed spans: entity, offsets, confidence
+
+reply = call_your_llm(result.text)          # provider never sees real values
+s.rehydrate(reply, "ticket-42")             # real values restored, your side only
+```
+
+Detection only:
+
+```python
+from dpdp_scrub import detect
+detect("payment 9876543210@ybl pe karo")
+# [Span(entity='UPI_ID', start=8, end=22, ..., confidence=0.92)]
+```
+
+Zero runtime dependencies. Python 3.9+.
+
+## Trust model
+
+```
+Ring 0: the vault (your Redis/memory)   — real values, session-scoped, droppable
+Ring 1: your org's services             — placeholders + scoped right to resolve
+Ring 2: LLM providers, logs, vector DBs — placeholders only, no resolution path
+```
+
+- Same value → same placeholder within a session (multi-turn consistency).
+- `vault.drop(session)` makes every old copy of its placeholders permanently
+  unresolvable — retroactive redaction for logs you already wrote.
+- No cloud, no telemetry, no phoning home. This is a library, not a service:
+  **your data never leaves your infrastructure.**
+
+Per-entity actions are policy: `tokenize` (reversible), `mask` (UIDAI-format
+`XXXX XXXX 0124` for Aadhaar, last-4 for cards), `redact` (ABHA health IDs by
+default), `ignore`.
+
+## Benchmark
+
+IndicPII-Bench v0 (in `bench/`): 500 synthetic messages — Hinglish, English,
+Devanagari, OCR-noise spacing, multi-entity — with 20% traps (order IDs,
+invoice refs, phones-in-txn-context). Presidio runs at its best
+configuration — its IN_* recognizers explicitly registered (they are shipped
+but **not loaded by default**). F1, value-level matching:
+
+| entity | dpdp-scrub | presidio |
+|---|---|---|
+| AADHAAR | **1.000** | 1.000 |
+| GSTIN | **1.000** | 0.971 |
+| PAN | **1.000** | 0.383 |
+| PHONE_IN | **1.000** | 0.602 |
+| UPI_ID | **1.000** | 0.000 |
+| IFSC | **1.000** | 0.000 |
+| EMAIL | **1.000** | 1.000 |
+| **OVERALL** | **1.000** | 0.595 |
+| trap false positives | **0/100** | 200/100 |
+
+Honest caveats: this is our own v0 corpus (templated, synthetic, Hinglish-
+heavy) — a perfect self-score mainly means the corpus is still too easy, and
+it will get harder (Devanagari, OCR noise, multi-entity). Presidio was not
+built for code-mixed text; the 107 trap FPs come from it having no context
+suppression (e.g. any 10-digit number near "txn" flags as phone). Reproduce:
+`python3 bench/compare.py 500`.
+
+## Status
+
+Pre-alpha, moving fast. Built so far: 15 entities, checksum validators,
+context-scoring engine, scrub/rehydrate loop with session vault — 86 tests,
+all synthetic data. Roadmap:
+
+- [ ] YAML policy profiles (`dpdp-strict`, `uidai-mask`, `audit-only`) + audit log
+- [ ] `scrub_json()` for structured payloads
+- [ ] Redis vault backend
+- [ ] LiteLLM / LangChain / Presidio adapters
+- [ ] **IndicPII-Bench** — a public benchmark for Indian PII detection
+- [ ] Indic-script & code-mixed NER for names/addresses (AI4Bharat IndicNER)
+
+## Contributing
+
+The reference-data files are designed for one-line PRs: new UPI PSP handles,
+RTO codes, context words (English/Hindi/regional). Entity requests welcome —
+include format, any checksum, and public documentation.
+
+All test data must be synthetic (checksum-generated) or published samples.
+Never commit real PII, including your own.
+
+## License
+
+MIT

dpdp_scrub-0.0.1/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "dpdp-scrub"
+version = "0.0.1"
+description = "Detect, validate, and redact Indian PII (Aadhaar, PAN, GSTIN, UPI, and more) — self-hosted, checksum-validated, DPDP-ready."
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.9"
+authors = [{ name = "Palkush Dave" }]
+keywords = ["pii", "dpdp", "aadhaar", "pan", "gstin", "privacy", "redaction", "india"]
+classifiers = [
+    "Development Status :: 2 - Pre-Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Security",
+    "Topic :: Text Processing",
+]
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "ruff>=0.4", "mypy>=1.10"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 100
+src = ["src", "tests"]

dpdp_scrub-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

dpdp_scrub-0.0.1/src/dpdp_scrub/__init__.py

@@ -0,0 +1,23 @@
+"""dpdp-scrub: detect, validate, and redact Indian PII.
+
+Self-hosted, checksum-validated, DPDP-ready. The LLM never sees the real value;
+your systems still do.
+"""
+
+__version__ = "0.0.1"
+
+from . import validators
+from .detectors import detect
+from .scrubber import Scrubber, ScrubResult
+from .spans import Span
+from .vault import MemoryVault
+
+__all__ = [
+    "validators",
+    "detect",
+    "Scrubber",
+    "ScrubResult",
+    "MemoryVault",
+    "Span",
+    "__version__",
+]

dpdp_scrub-0.0.1/src/dpdp_scrub/data.py

@@ -0,0 +1,41 @@
+"""Reference data for detectors.
+
+These are public identifiers of payment infrastructure (PSP handles), not PII.
+Community-maintainable: adding a new UPI handle is a one-line PR.
+"""
+
+# Known UPI PSP handles (the part after '@' in a VPA). Not exhaustive —
+# curated from NPCI's ecosystem. The handle list is what separates a UPI ID
+# from an email-shaped string, so keep entries lowercase and dot-free.
+UPI_PSP_HANDLES = frozenset(
+    {
+        # PhonePe
+        "ybl", "ibl", "axl",
+        # Google Pay
+        "okaxis", "okhdfcbank", "okicici", "oksbi",
+        # Paytm
+        "paytm", "ptyes", "ptaxis", "pthdfc", "ptsbi",
+        # Amazon Pay
+        "apl", "yapl", "rapl",
+        # WhatsApp Pay
+        "waaxis", "wahdfcbank", "waicici", "wasbi",
+        # Bank / app handles
+        "upi", "sbi", "hdfcbank", "icici", "axisbank", "barodampay",
+        "fbl", "idfcbank", "indus", "kotak", "dbs", "yesbank", "rbl",
+        "aubank", "bandhan", "federal", "uboi", "cnrb", "pnb", "boi",
+        # Apps
+        "freecharge", "mobikwik", "airtel", "jio", "slice", "jupiteraxis",
+        "naviaxis", "niyoicici", "tapicici", "timecosmos", "zoicici",
+    }
+)
+
+# RTO state/UT codes for vehicle registration plates, plus the BH (Bharat)
+# series. Includes legacy codes still on the road (OR, UA, TS).
+VEHICLE_STATE_CODES = frozenset(
+    {
+        "AN", "AP", "AR", "AS", "BR", "CG", "CH", "DD", "DL", "DN", "GA",
+        "GJ", "HP", "HR", "JH", "JK", "KA", "KL", "LA", "LD", "MH", "ML",
+        "MN", "MP", "MZ", "NL", "OD", "OR", "PB", "PY", "RJ", "SK", "TN",
+        "TR", "TS", "TG", "UK", "UA", "UP", "WB",
+    }
+)