PII-discovery 0.1.0__py3-none-any.whl

pii_discovery-0.1.0.dist-info/METADATA

@@ -0,0 +1,173 @@
+Metadata-Version: 2.4
+Name: PII-discovery
+Version: 0.1.0
+Summary: Local-first discovery of candidate PII, PHI, and financial data in common file formats
+Author: Pratik P. Lahudkar
+License-Expression: MIT
+Keywords: pii,phi,privacy,data-discovery,sensitive-data,structured-data,unstructured-data,forensics
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tqdm<5,>=4.66
+Provides-Extra: formats
+Requires-Dist: openpyxl<4,>=3.1; extra == "formats"
+Requires-Dist: pypdf<7,>=5; extra == "formats"
+Requires-Dist: python-docx<2,>=1.1; extra == "formats"
+Provides-Extra: ner
+Requires-Dist: spacy<4,>=3.7; extra == "ner"
+Provides-Extra: all
+Requires-Dist: openpyxl<4,>=3.1; extra == "all"
+Requires-Dist: pypdf<7,>=5; extra == "all"
+Requires-Dist: python-docx<2,>=1.1; extra == "all"
+Requires-Dist: spacy<4,>=3.7; extra == "all"
+Requires-Dist: tqdm<5,>=4.66; extra == "all"
+Provides-Extra: maintainer
+Requires-Dist: build<2,>=1.2; extra == "maintainer"
+Requires-Dist: twine<7,>=5; extra == "maintainer"
+Dynamic: license-file
+
+# PII-discovery
+
+**Local-first discovery of candidate PII, PHI, and financial data across structured, semi-structured, and text-based files.**
+
+PII-discovery is a Python library and command-line scanner for locating potentially sensitive values in supported local files. It combines structured-field and key-name matching with value patterns, explicit labels, selected validation rules, nearby text context, and optional spaCy named-entity recognition.
+
+The scanner reads source files locally and writes reports to a directory you choose. It does not redact values or modify source files. Findings include exact candidate values and unredacted context so an analyst can locate and review each match.
+
+PII-discovery is a discovery aid. It does not certify that a dataset is free of sensitive information, guarantee zero false positives, verify a person's identity, or establish regulatory compliance.
+
+## Capabilities
+
+- Scans structured, semi-structured, and unstructured text in supported local files.
+- Uses embedded header and key aliases, regular expressions, explicit label capture, selected format and checksum validation, and source context.
+- Optionally identifies PERSON and ORG candidates with spaCy NER.
+- Reports exact values, source locations, detection evidence, confidence/severity scores, and prefix/suffix context.
+- Keeps structured header findings separate from value-level findings.
+- Includes a progress bar through tqdm.
+- Embeds a reference taxonomy of 651 data-type entries across 27 categories. The taxonomy supports classification and matching; it does not mean every entry has a dedicated value detector.
+
+## Installation
+
+Install the base package from PyPI:
+
+~~~~powershell
+python -m pip install pii-discovery
+~~~~
+
+The base installation includes the command-line scanner and progress display. Optional readers for Excel, PDF, and Word documents can be installed with:
+
+~~~~powershell
+python -m pip install "pii-discovery[formats]"
+~~~~
+
+To enable spaCy named-entity recognition, install the NER extra and its language model:
+
+~~~~powershell
+python -m pip install "pii-discovery[ner]"
+python -m spacy download en_core_web_sm
+~~~~
+
+Install all optional integrations with `python -m pip install "pii-discovery[all]"`. The spaCy language model is installed separately. If spaCy or the requested model is unavailable, the scanner warns and continues without NER.
+
+## Supported input formats
+
+The current readers support:
+
+- Delimited text, including CSV and TSV
+- JSON and JSON Lines
+- XML and HTML
+- Plain text, logs, and Markdown
+- SQLite databases, read through a read-only connection
+- EML email messages, ICS calendars, and VCF files
+- XLSX and XLSM workbooks
+- PDF files with extractable embedded text
+- DOCX paragraphs and tables
+
+Some formats require the `formats` extra. XLSX scanning reads cached cell values and does not calculate formulas. PDF scanning extracts embedded text and does not OCR scanned pages. Images, audio, video, external databases, hidden document metadata, and password-protected content are outside the current scanner's coverage.
+
+## Quick start
+
+Scan a directory recursively and write reports to a separate output directory:
+
+~~~~powershell
+pii-discovery ./source_data --output ./pii_report
+~~~~
+
+Scan individual files and set a higher minimum candidate score:
+
+~~~~powershell
+pii-discovery ./customers.csv ./case_notes.txt --output ./case_report --min-confidence 0.70
+~~~~
+
+Use the command's help option to see every setting. Key controls include:
+
+| Option | Default | Purpose |
+| --- | ---: | --- |
+| `--output` | `pii_discovery_report` | Report destination |
+| `--no-recursive` | Off | Scan only files immediately inside each input directory |
+| `--min-confidence` | `0.55` | Minimum candidate score to report |
+| `--context-chars` | `70` | Maximum prefix and suffix context length in findings |
+| `--max-file-mb` | `100` | Per-file size limit |
+| `--spacy-model` | `en_core_web_sm` | spaCy model used for optional PERSON/ORG NER |
+| `--no-spacy` | Off | Disable spaCy NER |
+
+Confidence and severity values are evidence scores assigned by the implementation. They are not calibrated probabilities that a match is true.
+
+## Python API
+
+The primary programmatic entry point is `build_report`:
+
+~~~~python
+from pathlib import Path
+
+from pii_discovery import build_report
+
+summary = build_report(
+    inputs=[Path("source_data")],
+    output_dir=Path("pii_report"),
+    recursive=True,
+    min_confidence=0.55,
+    context_chars=70,
+    max_file_mb=100,
+    spacy_model="en_core_web_sm",
+)
+
+print(summary)
+~~~~
+
+Pass `spacy_model=None` to disable NER. The function returns a summary dictionary and writes the reports to `output_dir`.
+
+## Reports
+
+The report directory contains:
+
+- `findings.csv` — value-level candidate findings, including exact unredacted values in `value_preview`, source locations, detection evidence, and unredacted prefix/suffix context.
+- `structured_fields.csv` — sensitive column and key names detected from structured sources, kept separate from value-level findings.
+- `summary_by_category.csv`, `summary_by_file.csv`, and `summary_by_pii_type.csv` — finding counts by category, source file, and mapped catalog type.
+- `file_inventory.csv` — discovered files, scan status, values scanned, finding count, and parser errors.
+- `document_indicators.csv` — document-level terms reported separately from value findings.
+- `pii_type_catalog.csv`, `classification_legend.csv`, and `regulatory_terminology.csv` — embedded reference material used in classification and reporting.
+- `report.md` — scan totals, summaries, interpretation notes, and known limitations.
+
+The scanner does not redact values. Findings and reports may contain personal, health, financial, identity, or investigative information. Store and share them only under appropriate access controls and evidence-handling procedures.
+
+## Detection scope and limitations
+
+Detection combines header/key aliases, embedded taxonomy terms, regular expressions, explicit label capture, selected format or checksum checks, nearby context, and optional spaCy PERSON/ORG NER. Document-level keywords are reported separately and are not treated as proof that a value was found.
+
+Not every catalog item has a dedicated value detector. Names, addresses, health content, dates, and jurisdiction-specific identifiers can depend heavily on labels, language, context, and source quality. NER candidates are statistical signals, not proof of identity or personal-data status. The scanner does not query government, financial, carrier, or other issuing registries to authenticate values.
+
+No automated detector can guarantee complete recall or zero false positives across all languages, jurisdictions, source layouts, and data conditions. Validate results against an approved, representative, adjudicated dataset and have a qualified analyst review findings before relying on them in an investigation. Do not use this package as the sole basis for a legal, employment, financial, or other high-impact decision.
+
+The scanner processes files locally and does not upload source contents to a remote detection service. It is not a forensic acquisition tool and does not create a chain-of-custody record.
+
+## License
+
+MIT. See the included `LICENSE` file.

pii_discovery-0.1.0.dist-info/RECORD

@@ -0,0 +1,7 @@
+pii_discovery.py,sha256=nojDOUoLIKuGoCrhvrkzZNlqw-rNzu4uajIoTIlMN2Q,274460
+pii_discovery-0.1.0.dist-info/licenses/LICENSE,sha256=h1apmRfDXrqh9ci-m4AS-jtF1u4op7Q9KwZFTCHY438,1075
+pii_discovery-0.1.0.dist-info/METADATA,sha256=0GM9gHSKVD9KkxTLI2G84ORiP4KoYz9NCQdRqjtjUTI,8880
+pii_discovery-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+pii_discovery-0.1.0.dist-info/entry_points.txt,sha256=Qb6gUv7nYRQL4mTG_JycL25Njtvd2NkJNvM5DDphGrE,53
+pii_discovery-0.1.0.dist-info/top_level.txt,sha256=sx_4AE5f5hfTKOyIeOB5s9QoP8kG6JvFxgo2XdHmy08,14
+pii_discovery-0.1.0.dist-info/RECORD,,

pii_discovery-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

pii_discovery-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+pii-discovery = pii_discovery:main

pii_discovery-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pratik P. Lahudkar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pii_discovery-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+pii_discovery