piiscrub 0.1.0__tar.gz

piiscrub-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Omkar Pathak
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

piiscrub-0.1.0/PKG-INFO

@@ -0,0 +1,198 @@
+Metadata-Version: 2.4
+Name: piiscrub
+Version: 0.1.0
+Summary: A blazing-fast, lightweight Python library and CLI tool designed to scrub Personally Identifiable Information (PII)
+Author: Omkar Pathak
+Project-URL: Homepage, https://github.com/OmkarPathak/cleanslate
+Project-URL: Issues, https://github.com/OmkarPathak/cleanslate/issues
+Keywords: pii,scrub,anonymization,privacy,llm,rag
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Security
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: faker
+Dynamic: license-file
+
+# PiiScrub
+
+A blazing-fast, lightweight Python library and CLI tool designed to scrub Personally Identifiable Information (PII) from datasets for LLM training and RAG pipelines.
+
+## Features
+
+- **Maximum Speed & Zero Dependencies:** Relies exclusively on Python's standard library. No `pandas`, `spaCy`, or other heavy external packages.
+- **Deterministic Validation:** Raw regex matches for high-risk entities (like credit cards and IPs) pass algorithmic checksums (e.g., Luhn algorithm, octet range checks) before being flagged to eliminate false positives.
+- **Pre-compiled Regex:** All regular expressions are compiled at the module level using `re.compile()` for O(1) setup time during execution.
+- **Large Dataset Streaming:** Features `scrub_stream` and `extract_stream` to process massive datasets chunk-by-chunk without hitting Out-Of-Memory limit.
+- **Multi-Core Parallel Processing:** Leverage multiple CPU cores to scrub large files at blazing speed using `--parallel`.
+- **Pre-Bundled Compliance Profiles:** Quickly target specific standards like `hipaa`, `pci-dss`, or `gdpr` using the `--profile` flag.
+- **Compliance Auditing & Metric Reports:** Generate detailed JSON reports with statistics on redacted entities and execution time using `--report`.
+- **High-Value Secret Detection:** Added parsing to locate critical assets like AWS Access Keys, GitHub Tokens, and RSA Private Keys out of the box.
+- **Deterministic Hashing:** Replace PII with deterministic SHA-256 hashes instead of generic tags to track uniqueness without leaking data.
+- **Synthetic Data Generation:** Replace real PII with realistic "fake" data using the `faker` library (beta).
+- **Configuration File Support:** Manage complex settings via `piiscrub.json` instead of long CLI commands.
+- **Custom Pattern Injection:** Dynamically inject your own regex patterns and validators directly into the engine without modifying the core library.
+- **Allowlist Support:** Explicitly bypass scrubbing for public figures, system emails, or company identifiers to prevent false positives.
+
+## Supported Entities
+
+- **Global:**
+  - `EMAIL`
+  - `PHONE_GENERIC` (international)
+  - `CREDIT_CARD` (13-16 digits with Luhn algorithm validation)
+  - `IPV4` (validation ensuring all octets <= 255)
+  - `IPV6`
+- **US Specific:**
+  - `US_SSN`
+- **India Specific:**
+  - `IN_AADHAAR` (12 digits, cannot start with 0 or 1)
+  - `IN_PAN` (5 uppercase letters, 4 digits, 1 uppercase letter)
+- **Secrets & Credentials (V2):**
+  - `AWS_ACCESS_KEY`
+  - `GITHUB_TOKEN`
+  - `RSA_PRIVATE_KEY`
+
+## Installation
+
+```bash
+pip install .
+```
+
+## CLI Usage
+
+### Extract PII
+```bash
+piiscrub extract --text "My email is test@example.com"
+piiscrub extract --file text.txt
+```
+
+### Scrub PII
+```bash
+piiscrub scrub --text "My email is test@example.com"
+piiscrub scrub --file text.txt
+
+# Use deterministic hashing instead of standard tags
+piiscrub scrub --text "My email is test@example.com" --style hash
+# Output: My email is <EMAIL_a1517717>
+
+# Bypass scrubbing for specific public strings
+piiscrub scrub --text "Contact support@example.com or user@example.com" --allowlist support@example.com
+# Output: Contact support@example.com or <EMAIL>
+
+# Inject Custom Pattern from the CLI
+piiscrub scrub --text "This is employee EMP-99881 and email a@b.com" --custom-pattern EMP_ID "\bEMP-\d{5}\b" --entities EMP_ID EMAIL
+# Output: This is employee <EMP_ID> and email <EMAIL>
+
+# Synthetic Data Generation
+piiscrub scrub --text "Contact me at omkar@example.com" --style synthetic
+# Output: Contact me at victoria12@gmail.com
+```
+
+### Advanced Features
+
+#### 1. Configuration File (`piiscrub.json`)
+You can define a `piiscrub.json` file in your working directory to simplify your commands:
+
+```json
+{
+    "style": "hash",
+    "entities": ["EMAIL", "PHONE_GENERIC"],
+    "allowlist": ["support@mycompany.com"],
+    "custom_patterns": {
+        "ORDER_ID": "ORD-\\d{5}"
+    }
+}
+```
+
+Now just run:
+```bash
+piiscrub scrub --file data.txt
+```
+
+#### 2. Parallel Processing
+For large files, use multi-core processing:
+
+```bash
+piiscrub scrub --file large_dataset.txt --parallel --output cleaned.txt
+```
+> [!TIP]
+> Parallel mode automatically handles file I/O efficiently and defaults to using all available CPU cores.
+
+#### 3. Pre-Bundled Compliance Profiles
+Quickly target common privacy standards without remembering every entity name:
+
+```bash
+# Scrub only PCI-DSS related data (Credit Cards)
+piiscrub scrub --file transactions.txt --profile pci-dss
+
+# Scrub HIPAA related data (SSN, Phone, Email, IP)
+piiscrub scrub --file medical_records.txt --profile hipaa
+```
+
+Available profiles: `pci-dss`, `hipaa`, `gdpr`, `strict`.
+
+#### 4. Compliance Auditing & Metric Reports
+Data compliance teams can generate a statistical summary of the scrubbing process as proof of redaction:
+
+```bash
+piiscrub scrub --file sensitive_data.txt --report audit.json
+```
+
+**Sample `audit.json` output:**
+```json
+{
+    "command": "scrub",
+    "total_lines_processed": 5000,
+    "execution_time_seconds": 1.25,
+    "entities_redacted": {
+        "EMAIL": 142,
+        "CREDIT_CARD": 12,
+        "PHONE_GENERIC": 5
+    },
+    "style": "tag"
+}
+```
+
+### Stream Processing
+For extremely large files (e.g. LLM corpus data logs):
+```bash
+piiscrub scrub --file huge_dataset.jsonl --stream > scrubbed.jsonl
+piiscrub extract --file huge_dataset.jsonl --stream > entities.json
+```
+
+## Library Usage
+
+```python
+from piiscrub.core import PiiScrub
+import re
+
+# Initialize with custom generic entities or pattern injection!
+custom_patterns = {
+    "INTERNAL_ID": re.compile(r"\bEMP-\d{5}\b")
+}
+cs = PiiScrub(
+    entities=["EMAIL", "CREDIT_CARD", "INTERNAL_ID"], 
+    custom_patterns=custom_patterns,
+    allowlist=["public@example.com"]
+)
+
+code = "Contact test@example.com for info on EMP-12345."
+
+# Extract entities
+extracted = cs.extract_entities(code)
+print(extracted)
+# {'EMAIL': ['test@example.com'], 'INTERNAL_ID': ['EMP-12345']}
+
+# Scrub entities using hashing
+scrubbed_code = cs.scrub_text(code, replacement_style="hash")
+print(scrubbed_code)
+# Contact <EMAIL_a1517717> for info on <INTERNAL_ID_b5fb38c3>.
+```

piiscrub-0.1.0/README.md

@@ -0,0 +1,174 @@
+# PiiScrub
+
+A blazing-fast, lightweight Python library and CLI tool designed to scrub Personally Identifiable Information (PII) from datasets for LLM training and RAG pipelines.
+
+## Features
+
+- **Maximum Speed & Zero Dependencies:** Relies exclusively on Python's standard library. No `pandas`, `spaCy`, or other heavy external packages.
+- **Deterministic Validation:** Raw regex matches for high-risk entities (like credit cards and IPs) pass algorithmic checksums (e.g., Luhn algorithm, octet range checks) before being flagged to eliminate false positives.
+- **Pre-compiled Regex:** All regular expressions are compiled at the module level using `re.compile()` for O(1) setup time during execution.
+- **Large Dataset Streaming:** Features `scrub_stream` and `extract_stream` to process massive datasets chunk-by-chunk without hitting Out-Of-Memory limit.
+- **Multi-Core Parallel Processing:** Leverage multiple CPU cores to scrub large files at blazing speed using `--parallel`.
+- **Pre-Bundled Compliance Profiles:** Quickly target specific standards like `hipaa`, `pci-dss`, or `gdpr` using the `--profile` flag.
+- **Compliance Auditing & Metric Reports:** Generate detailed JSON reports with statistics on redacted entities and execution time using `--report`.
+- **High-Value Secret Detection:** Added parsing to locate critical assets like AWS Access Keys, GitHub Tokens, and RSA Private Keys out of the box.
+- **Deterministic Hashing:** Replace PII with deterministic SHA-256 hashes instead of generic tags to track uniqueness without leaking data.
+- **Synthetic Data Generation:** Replace real PII with realistic "fake" data using the `faker` library (beta).
+- **Configuration File Support:** Manage complex settings via `piiscrub.json` instead of long CLI commands.
+- **Custom Pattern Injection:** Dynamically inject your own regex patterns and validators directly into the engine without modifying the core library.
+- **Allowlist Support:** Explicitly bypass scrubbing for public figures, system emails, or company identifiers to prevent false positives.
+
+## Supported Entities
+
+- **Global:**
+  - `EMAIL`
+  - `PHONE_GENERIC` (international)
+  - `CREDIT_CARD` (13-16 digits with Luhn algorithm validation)
+  - `IPV4` (validation ensuring all octets <= 255)
+  - `IPV6`
+- **US Specific:**
+  - `US_SSN`
+- **India Specific:**
+  - `IN_AADHAAR` (12 digits, cannot start with 0 or 1)
+  - `IN_PAN` (5 uppercase letters, 4 digits, 1 uppercase letter)
+- **Secrets & Credentials (V2):**
+  - `AWS_ACCESS_KEY`
+  - `GITHUB_TOKEN`
+  - `RSA_PRIVATE_KEY`
+
+## Installation
+
+```bash
+pip install .
+```
+
+## CLI Usage
+
+### Extract PII
+```bash
+piiscrub extract --text "My email is test@example.com"
+piiscrub extract --file text.txt
+```
+
+### Scrub PII
+```bash
+piiscrub scrub --text "My email is test@example.com"
+piiscrub scrub --file text.txt
+
+# Use deterministic hashing instead of standard tags
+piiscrub scrub --text "My email is test@example.com" --style hash
+# Output: My email is <EMAIL_a1517717>
+
+# Bypass scrubbing for specific public strings
+piiscrub scrub --text "Contact support@example.com or user@example.com" --allowlist support@example.com
+# Output: Contact support@example.com or <EMAIL>
+
+# Inject Custom Pattern from the CLI
+piiscrub scrub --text "This is employee EMP-99881 and email a@b.com" --custom-pattern EMP_ID "\bEMP-\d{5}\b" --entities EMP_ID EMAIL
+# Output: This is employee <EMP_ID> and email <EMAIL>
+
+# Synthetic Data Generation
+piiscrub scrub --text "Contact me at omkar@example.com" --style synthetic
+# Output: Contact me at victoria12@gmail.com
+```
+
+### Advanced Features
+
+#### 1. Configuration File (`piiscrub.json`)
+You can define a `piiscrub.json` file in your working directory to simplify your commands:
+
+```json
+{
+    "style": "hash",
+    "entities": ["EMAIL", "PHONE_GENERIC"],
+    "allowlist": ["support@mycompany.com"],
+    "custom_patterns": {
+        "ORDER_ID": "ORD-\\d{5}"
+    }
+}
+```
+
+Now just run:
+```bash
+piiscrub scrub --file data.txt
+```
+
+#### 2. Parallel Processing
+For large files, use multi-core processing:
+
+```bash
+piiscrub scrub --file large_dataset.txt --parallel --output cleaned.txt
+```
+> [!TIP]
+> Parallel mode automatically handles file I/O efficiently and defaults to using all available CPU cores.
+
+#### 3. Pre-Bundled Compliance Profiles
+Quickly target common privacy standards without remembering every entity name:
+
+```bash
+# Scrub only PCI-DSS related data (Credit Cards)
+piiscrub scrub --file transactions.txt --profile pci-dss
+
+# Scrub HIPAA related data (SSN, Phone, Email, IP)
+piiscrub scrub --file medical_records.txt --profile hipaa
+```
+
+Available profiles: `pci-dss`, `hipaa`, `gdpr`, `strict`.
+
+#### 4. Compliance Auditing & Metric Reports
+Data compliance teams can generate a statistical summary of the scrubbing process as proof of redaction:
+
+```bash
+piiscrub scrub --file sensitive_data.txt --report audit.json
+```
+
+**Sample `audit.json` output:**
+```json
+{
+    "command": "scrub",
+    "total_lines_processed": 5000,
+    "execution_time_seconds": 1.25,
+    "entities_redacted": {
+        "EMAIL": 142,
+        "CREDIT_CARD": 12,
+        "PHONE_GENERIC": 5
+    },
+    "style": "tag"
+}
+```
+
+### Stream Processing
+For extremely large files (e.g. LLM corpus data logs):
+```bash
+piiscrub scrub --file huge_dataset.jsonl --stream > scrubbed.jsonl
+piiscrub extract --file huge_dataset.jsonl --stream > entities.json
+```
+
+## Library Usage
+
+```python
+from piiscrub.core import PiiScrub
+import re
+
+# Initialize with custom generic entities or pattern injection!
+custom_patterns = {
+    "INTERNAL_ID": re.compile(r"\bEMP-\d{5}\b")
+}
+cs = PiiScrub(
+    entities=["EMAIL", "CREDIT_CARD", "INTERNAL_ID"], 
+    custom_patterns=custom_patterns,
+    allowlist=["public@example.com"]
+)
+
+code = "Contact test@example.com for info on EMP-12345."
+
+# Extract entities
+extracted = cs.extract_entities(code)
+print(extracted)
+# {'EMAIL': ['test@example.com'], 'INTERNAL_ID': ['EMP-12345']}
+
+# Scrub entities using hashing
+scrubbed_code = cs.scrub_text(code, replacement_style="hash")
+print(scrubbed_code)
+# Contact <EMAIL_a1517717> for info on <INTERNAL_ID_b5fb38c3>.
+```

piiscrub-0.1.0/piiscrub/__init__.py

@@ -0,0 +1,4 @@
+"""
+PiiScrub - Fast, lightweight PII scrubbing library.
+"""
+__version__ = "0.1.0"

piiscrub-0.1.0/piiscrub/cli.py

@@ -0,0 +1,175 @@
+import argparse
+import sys
+import re
+import json
+import os
+import time
+from piiscrub.core import PiiScrub
+
+def get_text_from_args(args) -> str:
+    if args.text is not None:
+        return args.text
+    elif args.file is not None:
+        try:
+            with open(args.file, "r", encoding="utf-8") as f:
+                return f.read()
+        except IOError as e:
+            print(f"Error reading file {args.file}: {e}", file=sys.stderr)
+            sys.exit(1)
+    else:
+        print("Error: Must provide either --text or --file.", file=sys.stderr)
+        sys.exit(1)
+
+def load_config(config_path=None):
+    """Load configuration from a JSON file."""
+    path = config_path or "piiscrub.json"
+    if os.path.exists(path):
+        try:
+            with open(path, "r", encoding="utf-8") as f:
+                return json.load(f)
+        except (IOError, json.JSONDecodeError) as e:
+            print(f"Warning: Could not load config file {path}: {e}", file=sys.stderr)
+    return {}
+
+def main():
+    parser = argparse.ArgumentParser(description="PiiScrub - PII Scrubbing and Extraction Tool")
+    subparsers = parser.add_subparsers(dest="command", help="Command to run: 'scrub' or 'extract'")
+    subparsers.required = True
+
+    # Common arguments for both scrub and extract
+    parent_parser = argparse.ArgumentParser(add_help=False)
+    group = parent_parser.add_mutually_exclusive_group(required=True)
+    group.add_argument("--text", type=str, help="Raw text string to process")
+    group.add_argument("--file", type=str, help="Path to text file to process")
+    parent_parser.add_argument("--entities", type=str, nargs="+", help="Specific entities to target (e.g., EMAIL CREDIT_CARD)")
+    parent_parser.add_argument("--allowlist", type=str, nargs="+", help="Specific strings to bypass scrubbing (e.g., support@example.com)")
+    parent_parser.add_argument("--custom-pattern", nargs=2, action="append", metavar=("NAME", "REGEX"), help="Inject a custom regex pattern. Can be used multiple times.")
+    parent_parser.add_argument("--stream", action="store_true", help="Process the file chunk-by-chunk.")
+    parent_parser.add_argument("--parallel", action="store_true", help="Process the file in parallel using multiple cores.")
+    parent_parser.add_argument("--config", type=str, help="Path to piiscrub.json configuration file.")
+    parent_parser.add_argument("--report", type=str, help="Path to save the JSON audit report.")
+    parent_parser.add_argument("--profile", type=str, help="Compliance profile to use (e.g., pci-dss, hipaa, gdpr, strict)")
+
+    # Extract subcommand
+    parser_extract = subparsers.add_parser("extract", parents=[parent_parser], help="Extract PII entities from text")
+    
+    # Scrub subcommand
+    parser_scrub = subparsers.add_parser("scrub", parents=[parent_parser], help="Scrub PII entities from text")
+    parser_scrub.add_argument("--style", type=str, choices=["tag", "redacted", "hash", "synthetic"], help="Replacement style: 'tag', 'redacted', 'hash', or 'synthetic'")
+    parser_scrub.add_argument("--output", type=str, help="Output file path (recommended for large files or parallel mode)")
+
+    args = parser.parse_args()
+
+    # Load config file if present
+    config = load_config(args.config)
+    
+    # Merge config with args (CLI args take precedence)
+    entities = args.entities or config.get("entities")
+    allowlist = args.allowlist or config.get("allowlist")
+    parallel = args.parallel or config.get("parallel", False)
+    profile = args.profile or config.get("profile")
+    style = (getattr(args, "style", None) or config.get("style", "tag"))
+
+    # Process custom patterns from CLI
+    custom_patterns_dict = {}
+    if args.custom_pattern:
+        for name, pattern_str in args.custom_pattern:
+            try:
+                custom_patterns_dict[name] = re.compile(pattern_str)
+            except re.error as e:
+                print(f"Error compiling regex for {name}: {e}", file=sys.stderr)
+                sys.exit(1)
+                
+    # Merge custom patterns from config
+    config_patterns = config.get("custom_patterns", {})
+    for name, pattern_str in config_patterns.items():
+        if name not in custom_patterns_dict:
+            try:
+                custom_patterns_dict[name] = re.compile(pattern_str)
+            except re.error as e:
+                print(f"Error compiling config regex for {name}: {e}", file=sys.stderr)
+
+    # Initialize Core Engine
+    cs = PiiScrub(
+        entities=entities, 
+        profile=profile,
+        allowlist=allowlist,
+        custom_patterns=custom_patterns_dict if custom_patterns_dict else None
+    )
+
+    if (args.stream or parallel) and not args.file:
+        print("Error: --stream or --parallel requires --file.", file=sys.stderr)
+        sys.exit(1)
+
+    start_time = time.time()
+    total_lines = 0
+
+    if parallel and args.command == "scrub":
+        output_path = args.output or (args.file + ".scrubbed")
+        print(f"Processing in parallel... saving to {output_path}")
+        cs.scrub_file_parallel(args.file, output_path, replacement_style=style)
+        # We can count lines by reading the file or getting it from parallel scrub (if we update it)
+        # For now, let's keep it simple and just report execution time and entities
+        execution_time = time.time() - start_time
+        if args.report:
+            report = {
+                "command": args.command,
+                "execution_time_seconds": round(execution_time, 4),
+                "entities_redacted": cs.get_stats(),
+                "style": style
+            }
+            with open(args.report, "w", encoding="utf-8") as f_rep:
+                json.dump(report, f_rep, indent=4)
+        return
+
+    if args.stream:
+        # Streaming logic for files
+        try:
+            with open(args.file, "r", encoding="utf-8") as f:
+                if args.command == "extract":
+                    results = cs.extract_stream(f)
+                    print(json.dumps(results, indent=2))
+                elif args.command == "scrub":
+                    for scrubbed_line in cs.scrub_stream(f, replacement_style=style):
+                        total_lines += 1
+                        if args.output:
+                            with open(args.output, "a", encoding="utf-8") as f_out:
+                                f_out.write(scrubbed_line)
+                        else:
+                            sys.stdout.write(scrubbed_line)
+        except IOError as e:
+            print(f"Error reading file {args.file}: {e}", file=sys.stderr)
+            sys.exit(1)
+    else:
+        # Traditional in-memory logic
+        text = get_text_from_args(args)
+        total_lines = len(text.splitlines())
+        
+        if args.command == "extract":
+            results = cs.extract_entities(text)
+            print(json.dumps(results, indent=2))
+            
+        elif args.command == "scrub":
+            result = cs.scrub_text(text, replacement_style=style)
+            if args.output:
+                with open(args.output, "w", encoding="utf-8") as f_out:
+                    f_out.write(result)
+            else:
+                print(result)
+
+    execution_time = time.time() - start_time
+    if args.report:
+        report = {
+            "command": args.command,
+            "total_lines_processed": total_lines,
+            "execution_time_seconds": round(execution_time, 4),
+            "entities_found" if args.command == "extract" else "entities_redacted": cs.get_stats(),
+        }
+        if args.command == "scrub":
+            report["style"] = style
+            
+        with open(args.report, "w", encoding="utf-8") as f_rep:
+            json.dump(report, f_rep, indent=4)
+
+if __name__ == "__main__":
+    main()