lethe-cli 0.1.1__tar.gz

lethe_cli-0.1.1/.gitignore

@@ -0,0 +1,11 @@
+.DS_Store
+__pycache__/
+*.pyc
+.pytest_cache/
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+.env

lethe_cli-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Marco Kotrotsos
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

lethe_cli-0.1.1/PKG-INFO

@@ -0,0 +1,78 @@
+Metadata-Version: 2.4
+Name: lethe-cli
+Version: 0.1.1
+Summary: Data anonymization CLI tool
+Author: Marco Kotrotsos
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Security
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: faker<35,>=25.0
+Requires-Dist: pandas<3,>=2.0
+Requires-Dist: presidio-analyzer<3,>=2.2
+Requires-Dist: presidio-anonymizer<3,>=2.2
+Requires-Dist: rich<14,>=13.0
+Requires-Dist: spacy<4,>=3.7
+Requires-Dist: typer[all]<1,>=0.12
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: trf
+Requires-Dist: spacy[transformers]; extra == 'trf'
+Description-Content-Type: text/markdown
+
+# Lethe
+
+Data anonymization CLI for structured files. Detect and replace PII in CSV, TSV, and plain text using Presidio and spaCy NER, with Faker-generated replacements that stay consistent across your dataset.
+
+## Install
+
+```bash
+pip install lethe-cli
+python -m spacy download en_core_web_trf
+```
+
+For a faster, lighter model instead of the transformer:
+
+```bash
+python -m spacy download en_core_web_sm
+```
+
+## Usage
+
+### Anonymize
+
+Replace detected PII with consistent fake values:
+
+```bash
+lethe anonymize data.csv -o anonymized.csv
+lethe anonymize data.csv --model sm --threshold 0.7
+lethe anonymize notes.txt -o clean.txt --locale nl_NL
+```
+
+### Multiply
+
+Generate synthetic rows from an existing dataset:
+
+```bash
+lethe multiply data.csv --factor 5 -o expanded.csv
+lethe multiply data.csv --factor 10 --sanitize --seed 42
+```
+
+## Options
+
+Run `lethe anonymize --help` or `lethe multiply --help` for the full list of options.
+
+## License
+
+MIT

lethe_cli-0.1.1/README.md

@@ -0,0 +1,45 @@
+# Lethe
+
+Data anonymization CLI for structured files. Detect and replace PII in CSV, TSV, and plain text using Presidio and spaCy NER, with Faker-generated replacements that stay consistent across your dataset.
+
+## Install
+
+```bash
+pip install lethe-cli
+python -m spacy download en_core_web_trf
+```
+
+For a faster, lighter model instead of the transformer:
+
+```bash
+python -m spacy download en_core_web_sm
+```
+
+## Usage
+
+### Anonymize
+
+Replace detected PII with consistent fake values:
+
+```bash
+lethe anonymize data.csv -o anonymized.csv
+lethe anonymize data.csv --model sm --threshold 0.7
+lethe anonymize notes.txt -o clean.txt --locale nl_NL
+```
+
+### Multiply
+
+Generate synthetic rows from an existing dataset:
+
+```bash
+lethe multiply data.csv --factor 5 -o expanded.csv
+lethe multiply data.csv --factor 10 --sanitize --seed 42
+```
+
+## Options
+
+Run `lethe anonymize --help` or `lethe multiply --help` for the full list of options.
+
+## License
+
+MIT

lethe_cli-0.1.1/docs/architecture.md

@@ -0,0 +1,157 @@
+# Architecture and Inner Workings
+
+## Overview
+
+Lethe processes CSV files through a four-stage pipeline: **Parse**, **Scan**, **Replace**, **Write**. Each stage is designed around a single principle: process columns, not rows. Structured data has PII concentrated in specific columns (names, emails, SSNs), while other columns (IDs, timestamps, statuses) can be skipped entirely. This column-first approach avoids wasting NLP resources on data that is obviously not PII.
+
+## Pipeline
+
+```
+Input CSV
+  |
+  v
+CsvReader.read_chunks()          # Stream rows in chunks of N (default 5000)
+  |
+  v  (for each chunk)
+PiiScanner.scan_chunk()           # Analyze each column independently
+  |
+  v
+Replacer.replace_chunk()          # Swap PII cells with consistent fakes
+  |
+  v
+CsvWriter.write_chunk()           # Append to output file
+```
+
+The pipeline is orchestrated by `pipeline.py`, which wires these stages together and displays a progress bar via Rich.
+
+## Chunked Streaming
+
+Large files are never loaded entirely into memory. The `CsvReader` uses pandas' `chunksize` parameter to yield DataFrames of N rows at a time. The `CsvWriter` appends each processed chunk to the output file. This means Lethe can handle files larger than available RAM, the memory footprint is roughly proportional to `chunk_size * number_of_columns`.
+
+## PII Scanner
+
+The scanner (`scanner/engine.py`) is the core of Lethe. It wraps Presidio's `AnalyzerEngine` and adds two layers on top: column heuristics and confidence boosting. For each column in a chunk, the scanner runs three strategies:
+
+### Strategy 1: Column Heuristics
+
+Before running any NLP, the scanner examines the column name. A column named `first_name` is almost certainly a person's name. A column named `id` or `created_at` is almost certainly not PII.
+
+The heuristic classifier (`scanner/column_heuristics.py`) returns one of three outcomes:
+
+- **SKIP**: The column is definitely not PII (IDs, timestamps, booleans, statuses, amounts). The scanner skips the entire column, saving all NLP computation.
+- **PII type hint**: The column name matches a known PII pattern (e.g., `email` -> `EMAIL_ADDRESS`, `phone` -> `PHONE_NUMBER`). This hint is used later to boost confidence scores.
+- **None**: The column name is ambiguous. The scanner relies entirely on NLP and regex detection.
+
+**Skip patterns** cover:
+- Identifiers: `id`, `uuid`, `pk`, `key`, `guid`
+- Timestamps: `created_at`, `updated_on`, `timestamp`, `modified_date`
+- Booleans: `is_active`, `has_permission`, `enabled`
+- Numerics: `count`, `total`, `amount`, `price`, `score`, `age`
+- Statuses: `status`, `state`, `type`, `category`, `role`
+
+**PII hint patterns** cover:
+- `PERSON`: name, first_name, last_name, surname, customer_name
+- `EMAIL_ADDRESS`: email, mail, email_addr
+- `PHONE_NUMBER`: phone, mobile, cell, tel, fax
+- `LOCATION`: address, street, city, zip, postal, country
+- `US_SSN`: ssn, social_security
+- `CREDIT_CARD`: credit_card, card_num, cc_num
+- `IBAN_CODE`: iban
+- `IP_ADDRESS`: ip_addr, ip, remote_ip, client_ip
+- And more (driver's licenses, passports, dates of birth)
+
+### Strategy 2: Presidio Analysis
+
+For each non-skipped cell, the scanner calls Presidio's `AnalyzerEngine.analyze()`. Presidio combines two detection methods internally:
+
+1. **Regex pattern recognizers**: Built-in patterns for emails, phone numbers, credit cards, SSNs, and more. Lethe also registers three custom recognizers for IBAN codes, UK National Insurance Numbers (NINO), and Dutch BSN numbers.
+
+2. **NER (Named Entity Recognition)**: A spaCy NLP model identifies entities like person names and locations that cannot be caught by regex. The model choice (`trf` vs `sm`) affects accuracy here:
+   - `en_core_web_trf`: Transformer-based, highest accuracy, slower. Best for production use where you cannot afford to miss PII.
+   - `en_core_web_sm`: Small CNN-based model, faster, slightly less accurate on ambiguous names.
+
+Each recognizer returns results with a confidence score between 0 and 1.
+
+### Strategy 3: Confidence Boosting
+
+After Presidio returns its results, the confidence module (`scanner/confidence.py`) applies two adjustments based on the column heuristic:
+
+1. **Boost matching results**: If the heuristic said this column is `PHONE_NUMBER` and Presidio also detected `PHONE_NUMBER`, the score gets boosted by 0.25 (capped at 1.0). This pushes borderline detections over the threshold.
+
+2. **Synthesize missing results**: If the heuristic said `PERSON` but Presidio returned nothing (common for unusual names), the system synthesizes a result with a score of 0.4. The column name itself is strong evidence, this prevents names from slipping through because the NER model did not recognize them.
+
+### Result Selection
+
+After boosting, the scanner picks the highest-scoring result per cell. If that score meets or exceeds the configured threshold (default 0.35), the cell is marked as PII with its detected entity type.
+
+## Session Index and Mapping
+
+The `SessionIndex` (`mapping/session_index.py`) is a dictionary that maps `(entity_type, original_value)` tuples to fake replacements. When the replacer encounters "John Smith" in a `PERSON` column, it checks the index:
+
+- **First encounter**: Generate a fake name via Faker (e.g., "Nicholas Quinn"), store the mapping, return the fake.
+- **Subsequent encounters**: Return "Nicholas Quinn" immediately, O(1) lookup.
+
+This guarantees **cross-row and cross-table consistency**. If "John Smith" appears in both `customers.csv` and `orders.csv` (processed in the same session), it maps to the same fake name. This preserves referential integrity, foreign key relationships still work in the anonymized data.
+
+Each entity type maps to a specific Faker generator:
+
+| Entity Type | Faker Method | Example Output |
+|---|---|---|
+| `PERSON` | `name()` | "Nicholas Quinn" |
+| `EMAIL_ADDRESS` | `email()` | "sanchezmichelle@example.net" |
+| `PHONE_NUMBER` | `phone_number()` | "+1-555-384-2918" |
+| `LOCATION` | `address()` | "742 Pine Street, Apt 3" |
+| `US_SSN` | `ssn()` | "746-44-8807" |
+| `CREDIT_CARD` | `credit_card_number()` | "4532015112830366" |
+| `IP_ADDRESS` | `ipv4()` | "192.168.42.7" |
+| `IBAN_CODE` | `iban()` | "DE89370400440532013000" |
+| `DATE_TIME` | `date()` | "1994-07-15" |
+
+The `--locale` option controls the style of generated data. With `nl_NL`, Faker produces Dutch names and addresses. With `de_DE`, German ones.
+
+The `--seed` option makes the Faker output deterministic. Same seed + same input = identical output, useful for reproducible test pipelines.
+
+## Replacer
+
+The `Replacer` (`replacer/engine.py`) is intentionally simple. For each cell flagged as PII by the scanner, it:
+
+1. Reads the original cell value
+2. Calls `SessionIndex.get_or_create(entity_type, original)`
+3. Writes the fake value back
+
+This is **whole-cell replacement**, the entire cell content is swapped. This design is correct for structured CSV data where PII typically occupies the full cell (a name is the whole cell, not embedded in a sentence). Free-text span replacement, where PII is embedded in longer text, is planned for a future phase.
+
+## Module Map
+
+```
+src/lethe/
+  config.py                      # LetheConfig frozen dataclass
+  pipeline.py                    # Orchestrator: ties all stages together
+  cli.py                         # Typer CLI, thin wrapper over pipeline
+  parsers/
+    base.py                      # ChunkedReader / ChunkedWriter protocols
+    csv_parser.py                # pandas-based CSV I/O with chunking
+  scanner/
+    engine.py                    # PiiScanner: Presidio + heuristics + boosting
+    column_heuristics.py         # Column name -> SKIP / PII type / None
+    pattern_recognizers.py       # Custom regex: IBAN, UK NINO, NL BSN
+    confidence.py                # Score boosting and result synthesis
+  mapping/
+    session_index.py             # (entity_type, original) -> fake, via Faker
+  replacer/
+    engine.py                    # Whole-cell swap using session index
+```
+
+## Design Decisions
+
+**Why build on Presidio, not around it?**
+Presidio already handles entity resolution, deduplication, and spaCy NER integration. Reimplementing these would be error-prone and hard to maintain. Instead, Lethe registers custom recognizers into Presidio's `RecognizerRegistry` and lets Presidio handle the heavy lifting.
+
+**Why not use Presidio's AnonymizerEngine?**
+Presidio's anonymizer is designed for free-text spans, replacing "John" within "Dear John, your order is ready." Our data is structured CSV where PII is the entire cell. Our `Replacer` + `SessionIndex` approach is simpler, gives cross-table consistency for free, and avoids span-offset arithmetic.
+
+**Why column-first processing?**
+In structured data, PII type is determined by the column, not the row. All values in a `first_name` column are names. By processing column-by-column, we can skip entire columns via heuristics (saving all NLP cost for that column) and use the column name as a confidence signal.
+
+**Why `en_core_web_trf` by default?**
+Accuracy is the top priority. Missing PII in production data is worse than being slow. The transformer model catches ambiguous names and locations that the small model misses. For development and testing, `--model sm` provides a fast alternative.

lethe_cli-0.1.1/docs/cli.md

@@ -0,0 +1,111 @@
+# CLI Reference
+
+## Installation
+
+```bash
+pip install -e .
+python -m spacy download en_core_web_sm
+```
+
+For the transformer model (higher accuracy, slower):
+
+```bash
+pip install -e ".[trf]"
+python -m spacy download en_core_web_trf
+```
+
+## Commands
+
+### `lethe anonymize`
+
+Anonymize PII in a CSV file.
+
+```
+lethe anonymize <INPUT_FILE> [OPTIONS]
+```
+
+**Arguments:**
+
+| Argument | Description |
+|---|---|
+| `INPUT_FILE` | Path to the CSV file to anonymize. Must exist and be readable. |
+
+**Options:**
+
+| Option | Short | Default | Description |
+|---|---|---|---|
+| `--output` | `-o` | `<input>_anonymized.csv` | Output file path. When omitted, appends `_anonymized` to the input filename. |
+| `--model` | `-m` | `trf` | NLP model to use. `trf` = transformer (accurate), `sm` = small (fast). |
+| `--threshold` | `-t` | `0.35` | Minimum confidence score to classify a cell as PII. Lower values catch more PII but risk false positives. Higher values are more conservative. |
+| `--chunk-size` | | `5000` | Number of rows per processing chunk. Controls memory usage for large files. |
+| `--locale` | | `en_US` | Faker locale for generating replacement values. Affects name styles, address formats, phone patterns. |
+| `--seed` | | None | Random seed for reproducible output. Same seed + same input = identical anonymized output. |
+
+## Examples
+
+### Basic usage
+
+```bash
+lethe anonymize customers.csv
+```
+
+Output: `customers_anonymized.csv` in the same directory.
+
+### Specify output path
+
+```bash
+lethe anonymize customers.csv -o /tmp/safe_customers.csv
+```
+
+### Fast mode with small model
+
+```bash
+lethe anonymize customers.csv --model sm
+```
+
+Uses `en_core_web_sm` instead of the transformer model. Significantly faster, slightly less accurate on ambiguous names and locations.
+
+### Reproducible output
+
+```bash
+lethe anonymize customers.csv --seed 42
+```
+
+Running this twice on the same input produces identical output. Useful for deterministic test pipelines.
+
+### Tuning detection sensitivity
+
+```bash
+# Aggressive: catch anything that might be PII
+lethe anonymize customers.csv --threshold 0.2
+
+# Conservative: only replace high-confidence PII
+lethe anonymize customers.csv --threshold 0.7
+```
+
+### Large files with limited memory
+
+```bash
+lethe anonymize huge_dataset.csv --chunk-size 1000
+```
+
+Processes 1000 rows at a time instead of the default 5000. Lower values use less memory, higher values are slightly faster.
+
+### Dutch locale for replacement data
+
+```bash
+lethe anonymize klanten.csv --locale nl_NL
+```
+
+Generated fake names, addresses, and phone numbers will follow Dutch conventions.
+
+### Combining options
+
+```bash
+lethe anonymize data/export.csv \
+  -o data/export_safe.csv \
+  --model sm \
+  --threshold 0.3 \
+  --locale de_DE \
+  --seed 123
+```