local-redactor 0.3.0__tar.gz

local_redactor-0.3.0/PKG-INFO

@@ -0,0 +1,887 @@
+Metadata-Version: 2.4
+Name: local-redactor
+Version: 0.3.0
+Summary: Local command-line tool to redact PII and DevOps secrets from text files and images, built on Microsoft Presidio.
+Author: Munish Mehta
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/mumehta/local-redact
+Project-URL: Repository, https://github.com/mumehta/local-redact
+Keywords: presidio,redaction,pii,secrets,devops,ocr
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Security
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: presidio-analyzer==2.2.364
+Requires-Dist: presidio-anonymizer==2.2.364
+Requires-Dist: presidio-image-redactor==0.0.60
+Requires-Dist: pillow
+Requires-Dist: pytesseract
+Provides-Extra: dev
+Requires-Dist: pytest==8.4.2; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+
+# Local Redact
+
+[![CI](https://github.com/mumehta/local-redact/actions/workflows/ci.yml/badge.svg)](https://github.com/mumehta/local-redact/actions/workflows/ci.yml)
+
+A local command-line redaction tool that removes personally identifiable information (PII), DevOps secrets, API keys, tokens, credentials, and other sensitive values from text files and images. It is built on top of [Microsoft Presidio](https://github.com/microsoft/presidio).
+
+The Python distribution is named `local-redactor` (on PyPI); the installed command is `redact`.
+
+The goal of this project is to provide a simple command such as:
+
+```bash
+redact sensitive-file.txt
+redact screenshot.png
+```
+
+which creates sanitized copies that are safer to share in public forums, GitHub issues, support tickets, AI tools, documentation, and other external systems.
+
+The project is designed to run locally so that the original sensitive data does not need to be uploaded to a third-party redaction service.
+
+It runs on **Windows, macOS, and Linux**. The tool installs as a proper Python package with a `redact` console command, so the same command works identically on every operating system. Only the system prerequisites (Tesseract OCR, the spaCy model) differ per OS.
+
+## Image Redaction Example
+
+| Before | After |
+| --- | --- |
+| ![Unredacted DevOps secrets screenshot](tests/fixtures/devops-secrets.png) | ![Redacted DevOps secrets screenshot](tests/fixtures/devops-secrets.redacted.png) |
+
+## Related Posts
+
+- [LinkedIn post about Local Redactor](https://lnkd.in/p/ghje9qJn)
+- [Redact Sensitive Data Locally Before Sharing With AI](https://www.munish-mehta.com/post/redact-sensitive-data-locally-before-sharing-with-ai/)
+- [Before You Paste Logs Into AI, Redact Them Locally](https://medium.com/cyber-threat-diaries/before-you-paste-logs-into-ai-redact-them-locally-9aa56c023b1a)
+
+---
+
+## Why This Project Exists
+
+Engineers regularly need to share:
+
+- Application logs
+- Terminal output
+- Configuration files
+- JSON/YAML
+- Error messages
+- Screenshots
+- Debugging information
+- Infrastructure configuration
+- Support diagnostics
+
+These files can unintentionally contain sensitive information such as:
+
+- Names
+- Email addresses
+- Phone numbers
+- IP addresses
+- URLs
+- Account identifiers
+- Authentication tokens
+- API keys
+- Cloud credentials
+- Private keys
+- Connection strings
+
+Manually finding and redacting every sensitive value is slow and error-prone.
+
+This project provides a local automated redaction layer before information is shared externally.
+
+---
+
+# Current Status
+
+The current version supports text-based files using:
+
+- Presidio Analyzer
+- Presidio Anonymizer
+- spaCy
+- `en_core_web_lg`
+
+It also supports PNG/JPG/JPEG image redaction using:
+
+- Presidio Image Redactor
+- Tesseract OCR
+- pytesseract
+
+DevOps-specific secret detection is implemented with custom recognizers for common infrastructure credentials, API keys, tokens, private keys, and connection strings.
+
+The project ships as an installable Python package (`local-redactor`) with a `redact` console entry point.
+
+---
+
+# Architecture
+
+## Text Redaction
+
+```text
+Input file
+    |
+    v
+Presidio Analyzer
+    |
+    +-- Pattern recognizers
+    |
+    +-- spaCy NLP model
+    |
+    v
+Detected entities
+    |
+    v
+Presidio Anonymizer
+    |
+    v
+Redacted output file
+```
+
+Example:
+
+```text
+user-pii.txt
+        |
+        v
+Presidio Analyzer
+        |
+        v
+PERSON
+EMAIL_ADDRESS
+PHONE_NUMBER
+        |
+        v
+Presidio Anonymizer
+        |
+        v
+user-pii.redacted.txt
+```
+
+## Image Redaction
+
+```text
+Screenshot
+    |
+    v
+Tesseract OCR
+    |
+    v
+Extracted text + coordinates
+    |
+    v
+Presidio Analyzer
+    |
+    v
+Sensitive entities
+    |
+    v
+Presidio Image Redactor
+    |
+    v
+Opaque redaction boxes
+    |
+    v
+screenshot.redacted.png
+```
+
+---
+
+# Prerequisites
+
+Two things are required on every operating system and **cannot** be installed by
+`pip`, so they are installed per-OS below:
+
+1. **Python 3.9 or newer**
+2. **Tesseract OCR** (only needed for image redaction)
+
+A third requirement, the spaCy language model, is installed with the same
+command on every OS after the Python package is installed (see
+[Installation](#installation)).
+
+Pick your operating system:
+
+- [Windows prerequisites](#windows-prerequisites)
+- [macOS prerequisites](#macos-prerequisites)
+- [Linux (Ubuntu/Debian) prerequisites](#linux-ubuntudebian-prerequisites)
+
+---
+
+## Windows prerequisites
+
+### Python
+
+Install from [python.org](https://www.python.org/downloads/) or with winget:
+
+```powershell
+winget install -e --id Python.Python.3.12
+python --version
+```
+
+### Tesseract OCR
+
+Tesseract is a native system dependency and is **not installed by pip**.
+
+```powershell
+winget install -e --id tesseract-ocr.tesseract
+```
+
+A typical installation location is:
+
+```text
+C:\Program Files\Tesseract-OCR
+```
+
+Verify:
+
+```powershell
+tesseract --version
+tesseract --list-langs
+```
+
+At minimum, the English language model (`eng`) should be listed.
+
+If Windows cannot find `tesseract` after installation, either add
+`C:\Program Files\Tesseract-OCR` to your `PATH`, or point the redactor at the
+binary directly with the `TESSERACT_CMD` environment variable:
+
+```powershell
+$env:TESSERACT_CMD = "C:\Program Files\Tesseract-OCR\tesseract.exe"
+```
+
+---
+
+## macOS prerequisites
+
+### Python
+
+macOS ships with Python 3, but a dedicated install via
+[Homebrew](https://brew.sh/) is recommended:
+
+```bash
+brew install python
+python3 --version
+```
+
+### Tesseract OCR
+
+```bash
+brew install tesseract
+tesseract --version
+tesseract --list-langs
+```
+
+Homebrew places `tesseract` on your `PATH` automatically. If you installed it
+somewhere non-standard, point the redactor at it explicitly:
+
+```bash
+export TESSERACT_CMD="/opt/homebrew/bin/tesseract"
+```
+
+---
+
+## Linux (Ubuntu/Debian) prerequisites
+
+### Python
+
+```bash
+sudo apt update
+sudo apt install -y python3 python3-venv python3-pip
+python3 --version
+```
+
+### Tesseract OCR
+
+```bash
+sudo apt install -y tesseract-ocr
+tesseract --version
+tesseract --list-langs
+```
+
+For other distributions, use the equivalent package
+(for example `sudo dnf install tesseract` on Fedora). If `tesseract` is not on
+your `PATH`, set `TESSERACT_CMD` to its full path:
+
+```bash
+export TESSERACT_CMD="/usr/bin/tesseract"
+```
+
+---
+
+# Installation
+
+The steps below are the same on every OS once the
+[prerequisites](#prerequisites) are in place. Windows users can run the same
+commands in PowerShell (adjusting only the virtual-environment activation line,
+noted below).
+
+## 1. Clone the repository
+
+```bash
+git clone https://github.com/mumehta/local-redact.git
+cd local-redact
+```
+
+## 2. Create and activate a virtual environment
+
+A dedicated virtual environment keeps Presidio, spaCy, OpenCV, OCR libraries,
+and their dependencies isolated from system-wide Python packages.
+
+Create it:
+
+```bash
+python3 -m venv .venv
+```
+
+Activate it:
+
+**macOS / Linux**
+
+```bash
+source .venv/bin/activate
+```
+
+**Windows (PowerShell)**
+
+```powershell
+.\.venv\Scripts\Activate.ps1
+```
+
+Your prompt should now be prefixed with `(.venv)`.
+
+## 3. Upgrade pip
+
+```bash
+python -m pip install --upgrade pip
+```
+
+## 4. Install the package
+
+Install the project (and its pinned dependencies) into the virtual environment.
+This also creates the `redact` command.
+
+```bash
+python -m pip install .
+```
+
+For development (editable install plus test dependencies):
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+## 5. Install the spaCy English model
+
+This command is identical on every OS:
+
+```bash
+python -m spacy download en_core_web_lg
+```
+
+Verify:
+
+```bash
+python -m spacy validate
+```
+
+`en_core_web_lg` should be listed as compatible.
+
+## 6. Verify the installation
+
+```bash
+redact --help
+```
+
+You should see the CLI usage. Optionally verify the underlying engines:
+
+```bash
+python -c "from presidio_analyzer import AnalyzerEngine; a=AnalyzerEngine(); print([r.entity_type for r in a.analyze(text='My name is John Smith and my email is john@example.com', language='en')])"
+python -c "import pytesseract; print(pytesseract.get_tesseract_version())"
+```
+
+---
+
+# The `redact` Command
+
+Installing the package with `pip install .` (or `pip install -e .`) creates a
+real `redact` executable on every operating system:
+
+- On **Windows**, pip generates `redact.exe` in the environment's `Scripts`
+  directory.
+- On **macOS/Linux**, pip generates a `redact` executable in the environment's
+  `bin` directory.
+
+No hand-written wrapper script is required. This replaces the older Windows-only
+`redact.cmd` approach (see [Legacy Windows wrapper](#legacy-windows-wrapper) if
+you still want a global command that does not require activating the virtual
+environment).
+
+## Install globally with pipx (recommended for everyday use)
+
+[pipx](https://pipx.pypa.io/) installs the command into an isolated environment
+and puts `redact` on your `PATH`, so you never have to activate a virtual
+environment to use it. This works the same on Windows, macOS, and Linux.
+
+Install the published release from PyPI:
+
+```bash
+pipx install local-redactor
+```
+
+Or install from a local clone (for unreleased changes):
+
+```bash
+pipx install .
+```
+
+Then, from anywhere:
+
+```bash
+redact application.log
+redact screenshot.png
+```
+
+You still need Tesseract and the spaCy model installed as described in
+[Prerequisites](#prerequisites) and [Installation](#installation). When using
+pipx, the spaCy model must be downloaded into the pipx-managed environment for
+`local-redactor` (pipx isolates each app, so a model installed elsewhere is not
+visible to it):
+
+```bash
+pipx runpip local-redactor -- python -m spacy download en_core_web_lg
+```
+
+---
+
+# Using the Redactor
+
+With the virtual environment activated (or after `pipx install`), run:
+
+```bash
+redact ./example.txt
+```
+
+The tool creates:
+
+```text
+example.redacted.txt
+```
+
+The original file is left unchanged.
+
+## Display detected entities
+
+```bash
+redact ./example.txt --show-detections
+```
+
+Example:
+
+```text
+Detections:
+
+PERSON               score=0.85 position=11:21
+EMAIL_ADDRESS        score=1.00 position=38:54
+PHONE_NUMBER         score=0.75 position=71:86
+```
+
+This is useful when testing detection accuracy.
+
+## Specify an output file
+
+```bash
+redact ./example.txt -o ./safe-to-share.txt
+```
+
+## Overwrite an existing redacted file
+
+By default, the tool will not overwrite an existing output file. Use `--force`
+when intentional overwriting is required:
+
+```bash
+redact ./example.txt --force
+```
+
+## Image redaction
+
+```bash
+redact screenshot.png
+```
+
+produces:
+
+```text
+screenshot.redacted.png
+```
+
+Supported image formats: `.png`, `.jpg`, `.jpeg`.
+
+If Tesseract is not installed or not on your `PATH`, the tool prints a clear
+error with the correct install command for your OS. You can also point it at a
+specific Tesseract binary with the `TESSERACT_CMD` environment variable.
+
+---
+
+# Supported Text Files
+
+The current implementation supports text-based formats including:
+
+```text
+.txt
+.log
+.json
+.yaml
+.yml
+.env
+.conf
+.config
+.ini
+.xml
+.csv
+.md
+```
+
+Files are expected to contain UTF-8 text.
+
+Structured formats such as JSON, YAML, XML, `.env`, and `.ini` are redacted
+as text. The tool preserves useful structure in many common cases, but it
+does not parse and reserialize these formats yet, so review generated output
+before using it as machine-readable configuration.
+
+---
+
+# Example
+
+Input:
+
+```text
+My name is John Smith.
+My email address is john@example.com.
+My phone number is +61 412 345 678.
+```
+
+Run:
+
+```bash
+redact example.txt
+```
+
+Output (`example.redacted.txt`):
+
+```text
+My name is <PERSON>.
+My email address is <EMAIL_ADDRESS>.
+My phone number is <PHONE_NUMBER>.
+```
+
+---
+
+# Dependency Management
+
+Dependencies are declared in `pyproject.toml`:
+
+- Runtime dependencies live under `[project].dependencies`.
+- Development/test dependencies live under
+  `[project.optional-dependencies].dev` and are installed with
+  `pip install -e ".[dev]"`.
+
+Two supporting files remain for convenience:
+
+## `requirements.txt`
+
+The direct runtime dependencies, mirroring `pyproject.toml`, for environments
+that prefer a plain requirements file:
+
+```text
+presidio-analyzer==2.2.364
+presidio-anonymizer==2.2.364
+presidio-image-redactor==0.0.60
+```
+
+## `requirements-lock.txt`
+
+Captures the complete known-working Python environment, including transitive
+dependencies. Regenerate it with:
+
+```bash
+python -m pip freeze > requirements-lock.txt
+```
+
+Use it when an exact environment needs to be reproduced:
+
+```bash
+python -m pip install -r requirements-lock.txt
+```
+
+---
+
+# Development And Tests
+
+Install the project with development dependencies:
+
+```bash
+python -m pip install -e ".[dev]"
+```
+
+Run the automated tests:
+
+```bash
+python -m pytest
+```
+
+The tests use synthetic PII and credential-shaped fixtures only. Image tests use
+fakes/mocks and do not require Tesseract to be installed.
+
+---
+
+# System Dependencies
+
+Some dependencies cannot be represented in `pyproject.toml` and are installed
+per-OS (see [Prerequisites](#prerequisites)):
+
+```text
+Tesseract OCR 5.x
+```
+
+The spaCy language model is also installed separately (same command on all
+operating systems):
+
+```bash
+python -m spacy download en_core_web_lg
+```
+
+---
+
+# Security Considerations
+
+## Automated redaction is not a security guarantee
+
+The output of this tool should **not automatically be assumed safe for public disclosure**.
+
+PII and secret detection systems can produce:
+
+- False positives
+- False negatives
+- Incorrect entity boundaries
+- OCR errors
+- Unrecognized credential formats
+
+Review highly sensitive output before publishing it.
+
+## DevOps Secrets
+
+Standard Presidio recognizers are primarily designed for PII.
+
+Infrastructure and DevOps material may contain secrets that are not detected by default, including:
+
+```text
+AWS_ACCESS_KEY_ID
+AWS_SECRET_ACCESS_KEY
+
+ghp_...
+github_pat_...
+
+Authorization: Bearer ...
+
+JWT tokens
+
+client_secret=...
+
+password=...
+
+N8N_ENCRYPTION_KEY=...
+
+privkey:...
+
+nlpriv:...
+
+Kubernetes secrets
+
+database connection strings
+
+SSH/private keys
+
+OAuth credentials
+
+cloud provider credentials
+```
+
+Custom recognizers cover many of these patterns, but manually inspect infrastructure-related output before sharing it publicly.
+
+## Current Limitations
+
+- Files are processed one at a time; directory/batch redaction is not implemented yet.
+- Text files must be UTF-8 encoded.
+- JSON, YAML, XML, `.env`, `.ini`, and similar files are processed as text, not parsed as structured data.
+- OCR accuracy depends on screenshot quality, font size, contrast, and layout.
+- Detection is best-effort and can miss unfamiliar token formats or redact too much context.
+
+---
+
+# Legacy Windows wrapper
+
+Before the project was packaged with a console entry point, Windows users
+exposed a global `redact` command with a small `.cmd` wrapper that called the
+virtual environment's Python directly. This is **no longer necessary** — use
+`pip install .` or `pipx install .` instead, which produce a `redact` command
+on every OS.
+
+The wrapper is documented here only for historical reference. If you still want
+a wrapper that invokes the project without activating the virtual environment,
+create `C:\Users\<username>\bin\redact.cmd`:
+
+```bat
+@echo off
+"C:\path\to\local-redact\.venv\Scripts\python.exe" -m presidio_redactor.cli %*
+```
+
+and add `C:\Users\<username>\bin` to your user `PATH`.
+
+---
+
+# Repository Structure
+
+```text
+local-redact/
+|
++-- .gitignore
++-- README.md
++-- pyproject.toml               # packaging, deps, `redact` entry point
++-- requirements.txt
++-- requirements-lock.txt
++-- requirements-dev.txt
+|
++-- src/
+|   +-- presidio_redactor/
+|       +-- __init__.py
+|       +-- cli.py               # argparse + main(); the `redact` command
+|       +-- text.py              # text redaction pipeline
+|       +-- image.py             # image redaction + Tesseract resolution
+|       +-- recognizers/
+|           +-- __init__.py
+|           +-- devops.py        # custom DevOps/secret recognizers
+|
++-- tests/
+|   +-- fixtures/
+|   +-- test_devops_recognizers.py
+|   +-- test_image_redaction.py
+|   +-- test_text_redaction.py
+|   +-- test_tesseract_resolution.py
+|
++-- user-pii.txt                 # synthetic local example
++-- user-pii.redacted.txt        # synthetic redacted example
+|
++-- .venv/                       # ignored by Git
+```
+
+---
+
+# Planned Features
+
+Future development includes:
+
+- Content-based file type detection
+- Broader OAuth token detection
+- Broader GCP/Azure credential detection
+- GitLab token detection
+- Format-aware Kubernetes Secret parsing
+- Batch directory redaction
+- Dry-run mode
+- Configurable entity selection
+- Confidence thresholds
+- Windows Explorer "Redact before sharing" integration
+- Publishing to PyPI
+
+---
+
+# Development Roadmap
+
+The recommended implementation order is:
+
+```text
+1. Text PII redaction                 DONE
+        |
+2. Global redact command              DONE
+        |
+3. Tesseract installation             DONE
+        |
+4. Image redaction                    DONE
+        |
+5. DevOps secret recognizers          DONE
+        |
+6. Automated synthetic tests          DONE
+        |
+7. Cross-platform support             DONE
+        |
+8. Python CLI packaging               DONE
+        |
+9. Batch redaction
+        |
+10. Windows Explorer integration
+```
+
+---
+
+# Git Safety
+
+> **Test-data notice:** All names, email addresses, phone numbers, passwords,
+> tokens, credentials, connection strings, and other sensitive-looking values
+> committed in `tests/fixtures/`, `user-pii.txt`, and
+> `user-pii.redacted.txt` are synthetic dummy data. They are deliberately
+> shaped like real PII and secrets to exercise the redaction pipeline. They
+> are not valid credentials and are not associated with real accounts.
+> Automated secret scanners may still flag these fixtures because their
+> formats intentionally resemble real credentials.
+
+Never commit real sensitive data simply to test the redactor.
+
+Avoid committing:
+
+```text
+.env
+real application logs
+credentials
+private keys
+access tokens
+unredacted screenshots
+production configuration
+customer information
+personal information
+```
+
+Use synthetic test data instead.
+
+Do not rely on `.gitignore` as a security boundary. A file that has already been committed remains in Git history even if it is subsequently added to `.gitignore`.
+
+---
+
+# Privacy Model
+
+The primary design principle of this project is:
+
+> Sensitive source material should remain local wherever possible.
+
+Unlike an online redaction service, the local pipeline processes source files on the user's machine.
+
+However, users remain responsible for validating that redaction was successful before publishing or transmitting the resulting files.
+
+---
+
+# License
+
+Choose and add an appropriate open-source license before distributing the project publicly.
+
+For a small open-source utility of this type, the MIT License is one possible option.
+
+---
+
+# Acknowledgements
+
+This project builds on:
+
+- Microsoft Presidio
+- spaCy
+- Tesseract OCR
+- pytesseract
+
+These projects provide the underlying PII detection, natural-language processing, and optical character recognition capabilities.