py-secret-scan 3.0.1__tar.gz

py_secret_scan-3.0.1/.gitignore

@@ -0,0 +1,12 @@
+# Default ignored files
+/shelf/
+/workspace.xml
+# Ignored default folder with query files
+/queries/
+# Datasource local storage ignored files
+/dataSources/
+/dataSources.local.xml
+# Editor-based HTTP Client requests
+/httpRequests/
+/.venv/
+__pycache__/

py_secret_scan-3.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JMartynov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

py_secret_scan-3.0.1/PKG-INFO

@@ -0,0 +1,662 @@
+Metadata-Version: 2.4
+Name: py-secret-scan
+Version: 3.0.1
+Summary: A local security tool to detect secrets and PII in LLM prompts, code, and logs.
+Project-URL: Homepage, https://github.com/JMartynov/secret-scan
+Project-URL: Repository, https://github.com/JMartynov/secret-scan
+Author: JMartynov
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.8
+Requires-Dist: faker
+Requires-Dist: google-re2
+Requires-Dist: pyahocorasick
+Requires-Dist: pydantic<2.0
+Requires-Dist: pyyaml==6.0.3
+Requires-Dist: regex
+Requires-Dist: ruamel-yaml
+Description-Content-Type: text/markdown
+
+# LLM Secrets Leak Detector
+
+[![Self Secret Scan](https://github.com/JMartynov/secret-scan/actions/workflows/self-scan.yml/badge.svg)](https://github.com/JMartynov/secret-scan/actions/workflows/self-scan.yml)
+
+**LLM Secrets Leak Detector** is a security tool designed to prevent accidental exposure of sensitive data when interacting with Large Language Models (LLMs).
+
+Modern AI development workflows frequently involve sending code snippets, configuration files, logs, and debugging output to language models. In many cases developers unintentionally include sensitive information such as API keys, database credentials, private tokens, or internal infrastructure details.
+
+This project detects those secrets **before they leave the developer’s environment**.
+
+The system scans prompts, responses, logs, and source code to identify potential secrets and warns the user when confidential data may be exposed.
+
+---
+
+# Problem
+
+LLM-assisted development has dramatically increased developer productivity. However, it also introduced a new security risk.
+
+Developers regularly paste entire code blocks, configuration files, or logs into AI assistants to ask questions like:
+
+> “Here is my code, can you help debug it?”
+
+These inputs often contain secrets such as:
+
+* API keys
+* database credentials
+* private tokens
+* authentication secrets
+* internal infrastructure URLs
+* encryption keys
+* JWT tokens
+
+Example input that might be leaked:
+
+```
+OPENAI_API_KEY=sk-abc123
+DATABASE_URL=postgres://user:pass@db
+JWT_SECRET=super-secret-token
+```
+
+Once sent to an external LLM service, this data may:
+
+* appear in provider logs
+* be stored for debugging
+* violate compliance policies
+* leak sensitive infrastructure details
+
+The exposure of secrets in software artifacts has been increasing rapidly, with millions of credentials discovered in public repositories in recent years. ([arXiv][1])
+
+Security teams now treat **secret detection as a critical part of modern development pipelines**.
+
+---
+
+# Solution
+
+LLM Secrets Leak Detector automatically scans AI interaction data and identifies potential secrets before they are transmitted.
+
+The tool analyzes:
+
+* prompts sent to LLMs
+* LLM responses
+* application logs
+* code snippets
+* configuration files
+
+When a potential secret is detected, the tool generates a warning describing:
+
+* the secret type
+* location
+* severity level
+
+Example output:
+
+```
+⚠ Secrets detected
+
+Type: OpenAI API Key
+Location: line 3
+Risk: HIGH
+
+Type: Database credentials
+Location: line 4
+Risk: CRITICAL
+```
+
+This allows developers to **remove or redact sensitive information before it reaches an external AI system.**
+
+---
+
+# Core Detection Approach
+
+The detection engine uses a layered strategy similar to modern secret detection systems.
+
+Most secret scanners rely on three complementary techniques:
+
+1. **Pattern Matching (Regex)**
+   Identifies secrets with known formats such as AWS keys or GitHub tokens.
+
+2. **Entropy Analysis**
+   Detects strings that appear random, which is typical for cryptographic tokens.
+
+3. **Contextual Analysis**
+   Reduces false positives by analyzing surrounding code and variable names. ([gitguardian.com][2])
+
+Combining these methods significantly improves accuracy.
+
+---
+
+# Secret Types Detected
+
+The scanner detects over **180 classes** of sensitive data, including:
+
+### API Keys
+
+Examples:
+
+```
+sk-xxxxxxxxxxxxxxxx
+AIzaSyxxxxxxxxxxxx
+```
+
+### Cloud Credentials
+
+Examples:
+
+```
+AWS_ACCESS_KEY_ID
+AWS_SECRET_ACCESS_KEY
+AZURE_TOKEN
+```
+
+### Version Control Tokens
+
+Examples:
+
+```
+ghp_xxxxxxxxxxxxxxxxx
+glpat_xxxxxxxxxxxxx
+```
+
+### Authentication Secrets
+
+Examples:
+
+```
+JWT_SECRET
+SESSION_SECRET
+PRIVATE_KEY
+```
+
+### Database Credentials
+
+Examples:
+
+```
+postgres://user:password@host
+mysql://root:pass@db
+```
+
+### Cryptographic Material
+
+Examples:
+
+```
+-----BEGIN PRIVATE KEY-----
+```
+
+---
+
+# Feature Matrix
+
+The **LLM Secrets Leak Detector** provides a comprehensive suite of features designed for security, performance, and developer experience.
+
+| Category | Feature | Status | Implementation Details |
+| :--- | :--- | :--- | :--- |
+| **Detection Engines** | **Regex Matching (RE2)** | ✅ | Primary engine using `google-re2`.<br>Fast, linear-time matching. |
+| | **Regex Matching (Legacy)** | ✅ | Fallback to `regex` (Python) for complex patterns.<br>ReDoS protection. |
+| | **Entropy Analysis** | ✅ | Shannon entropy scoring for random-looking tokens (min 20 chars). |
+| | **Contextual Heuristics** | ✅ | Identifies secrets based on surrounding keywords like `prod`, `password`, `key`. Supports multi-lingual conversational intent matching (English, Spanish, French, German). |
+| | **Rule-based Logic** | ✅ | 1750+ rules loaded from `data/` (Expanded 2026). |
+| **Input Sources** | **File Scanning** | ✅ | Scans local files with UTF-8 support.<br>Error handling. |
+| | **Stdin / Piped Input** | ✅ | Real-time processing of piped data (e.g., `cat log \| ./run.sh`). |
+| | **Direct Text** | ✅ | Via `--text` flag for quick prompt validation. |
+| | **Streaming** | ✅ | Optimized line-by-line generator for low-latency processing. |
+| **Obfuscation** | **Redact** | ✅ | Masks the middle of secrets (e.g., `AKIA...CDEF`). |
+| | **Hash** | ✅ | Consistent SHA-256 hashing (first 12 chars) for safe debugging. |
+| | **Synthetic** | ✅ | [NEW] Realistic fake data generation (AWS, GitHub, Emails) using `Faker`. |
+| **Safety & Performance** | **Keyword Filtering** | ✅ | Uses `ahocorasick-rs` automaton (with SIMD) to skip rules missing their required keywords. |
+| | **Parallel Scanning** | ✅ | [NEW] Utilizes `ProcessPoolExecutor` for high-speed historical audits and multi-file directory scans. |
+| | **Commit Caching** | ✅ | [NEW] Incremental scanning using `.secretscan_cache` to skip verified SHAs. |
+| | **Zero-Copy Scanning**| ✅ | Uses `mmap` mapping with chunk overlaps for gigabyte-scale logs. |
+| | **ReDoS Protection** | ✅ | `SIGALRM` timeouts (1s) for non-RE2 regex execution. |
+| | **Input Truncation** | ✅ | Blocks capped at 1MB characters to prevent memory exhaustion. |
+| | **Deduplication** | ✅ | Merges overlapping findings.<br>Prioritizes longest matches. |
+| | **Force All Scan** | ✅ | `--force-scan-all` bypasses keyword filters so every line is scored. |
+| **Reporting & UI** | **Surgical Highlighting** | ✅ | [NEW] ANSI-colored context lines with the secret highlighted in red. |
+| | **Remediation Hints** | ✅ | [NEW] Actionable advice with links to official provider documentation. |
+| | **Colorized Output** | ✅ | ANSI colors for risk levels (Red=High, Yellow=Medium, Blue=Low). |
+| | **Report Formats** | ✅ | `Summary` (counts only).<br>`Short` (redacted).<br>`Full` (raw secrets + context).<br>`SARIF` (GitHub Code Scanning). |
+| | **CI/CD Friendly** | ✅ | `--nocolors` flag.<br>Standard exit codes for automation. |
+| **Testing & Dev** | **BDD Acceptance** | ✅ | 25 scenarios in `acceptance.feature` (including Git workflows) using `pytest-bdd`. |
+| | **Performance Bench** | ✅ | [NEW] Automated suite to verify caching and parallelization gains. |
+| | **Unit Testing** | ✅ | Comprehensive suite for core logic (detector, obfuscator, cli). |
+| | **Synthetic Corpus** | ✅ | `generate_test_data.py` creates a balanced test set from rules. |
+| | **Rule Deduplication** | ✅ | `tools/deduplicate_rules.py` keeps the catalog clean before release. |
+
+## Git & CI/CD Integration
+
+The detector is now natively aware of Git lifecycles, allowing for surgical scans of changes rather than entire files.
+
+### 🛠 Git Scanning Modes
+
+```bash
+# Scan staged changes (perfect for pre-commit hooks)
+./run.sh --git-staged --mode fast
+
+# Scan unstaged changes in the working directory
+./run.sh --git-working
+
+# Scan the diff between a feature branch and main (PR audits)
+./run.sh --git-branch origin/main --format sarif
+
+# Deep audit of repository history (Parallelized & Cached)
+./run.sh --history --since "1 month ago" --max-commits 100
+```
+
+### 🏎 Performance & Scalability
+
+- **Parallel Execution**: Large-scale historical audits and multi-file directory scans automatically utilize multiple CPU cores for regex and entropy analysis.
+- **Commit Caching**: The engine maintains a `.secretscan_cache` to track verified "clean" commits, reducing redundant scan times by up to 90% in incremental audits.
+- **Modes**: Choose between `fast` (optimized for <1s hooks), `balanced` (standard dev), and `deep` (thorough CI audits).
+
+---
+
+## Surgical Highlighting & Remediation
+
+When a secret is detected, the terminal output provides immediate visual context and actionable fix instructions.
+
+```text
+⚠ Secrets detected: 1
+- HIGH: 1
+
+Type: stripe_api_key
+Location: line 1
+Risk: HIGH
+Suggestion: Rotate this Stripe API key immediately in your dashboard. See: https://stripe.com/docs/keys#api-key-rotation
+Context: config process result Stripe secret: [SECRET_HIGHLIGHTED_IN_RED]
+```
+
+Remediation hints now include direct links to official security guides for AWS, GitHub, Stripe, and Google Cloud to guide developers through the revocation and rotation process.
+
+### Natural Language Contextual Matching
+
+The detection engine uses a 100-character context window to detect natural language conversational intents, such as `Here is my prod api key:`, which are common when interacting with LLMs. This feature is fully multi-lingual, boosting confidence scores when intent is detected in English, Spanish, French, or German.
+
+---
+
+## Extended Infrastructure Mode
+
+The latest feature expansion brings the infrastructure-focused taxonomy front and center:
+
+* `data/infrastructure` now houses rules for credit cards, IBANs/SEPA references, national ID numbers, and other high-risk identifiers.
+* Entropy-aware scoring plus overlap resolution lets structured infrastructure matches win over generic keywords or high-entropy heuristics.
+* The CLI `--force-scan-all` option ensures legacy logs that omit keywords still get evaluated (see the new acceptance scenario for this mode).
+* Dedicated tests cover deduped rules, synthetic obfuscation, and the expanded dataset to ensure the library stays precise.
+
+## Development Utilities
+
+Keep the catalog healthy with the accompanying tools:
+
+* `tools/migrate_patterns.py` normalizes schema fields, adds entropy defaults, and maps external categories to the in-tree taxonomy.
+* `tools/generate_test_data.py` rebuilds the base64-encoded `data/*/test_data.json` files from regexes so every rule ships with reproducible samples.
+* `tools/deduplicate_rules.py` merges duplicate patterns across categories before rules ship.
+* Use `tools/regex_lint.py`, `tools/run_safe_regex.py`, and `tools/run_redoctor.py` to guard against ReDoS, syntax drift, and schema regressions.
+
+Run `pytest tests/test_acceptance.py::test_force_scan_keywordless` before releasing to exercise the keywordless mode.
+
+---
+
+# Pattern Database
+
+The detection engine can leverage large open-source pattern databases containing thousands of secret signatures.
+
+For example, open datasets include **over 1600 regular expressions** that detect API keys, tokens, passwords, and other credentials across hundreds of services. ([GitHub][3])
+
+This allows the scanner to stay updated with newly introduced API key formats.
+
+---
+
+# Project Goals
+
+The project focuses on **protecting AI workflows** rather than traditional repository scanning.
+
+Key design goals:
+
+### AI-first security
+
+Detect secrets inside:
+
+* LLM prompts
+* chat transcripts
+* agent logs
+* debugging sessions
+
+### Developer-first experience
+
+The tool integrates directly into developer workflows without requiring complex configuration.
+
+### Local processing
+
+All scanning occurs locally to ensure no data leaves the environment.
+
+### Fast feedback
+
+Secrets should be detected immediately during development.
+
+---
+
+# Core Components
+
+The system is composed of several modules.
+
+### Detection Engine
+
+Responsible for identifying potential secrets using:
+
+* regex pattern matching
+* entropy scoring
+* context heuristics
+
+### Pattern Database
+
+A continuously updated collection of secret signatures.
+
+Includes patterns for:
+
+* API providers
+* cloud platforms
+* CI/CD tokens
+* authentication systems
+
+### Scanner Interface
+
+The scanner processes different input sources:
+
+* text prompts
+* log streams
+* source files
+* application outputs
+
+### Reporting System
+
+Findings are returned as structured results including:
+
+* secret type
+* location
+* confidence score
+* risk level
+* risk score (0-100)
+
+The CLI produces clear, color-coded output highlighting the location, risk level (HIGH, MEDIUM, LOW), and an Advanced Risk Score (0-100) of detected secrets. The risk score is determined by a weighted heuristic that incorporates regex confidence, contextual proximity bonuses, and entropy adjustments. The `report.py` module manages deduplication and formatting. Use `--format sarif` for CI/CD integration.
+
+You can tune sensitivity and filter out low-confidence noise by using the `--min-score` flag (e.g., `--min-score 70`).
+
+---
+
+# Architecture
+
+The architecture prioritizes simplicity and speed.
+
+```
+Input Sources
+    │
+    │
+    ▼
+Preprocessing Layer
+    │
+    │
+    ▼
+Detection Engine
+    ├── Regex Matching
+    ├── Entropy Detection
+    └── Context Analysis
+    │
+    ▼
+Secret Classification
+    │
+    ▼
+Security Report
+```
+
+---
+
+# Example Detection
+
+Input text:
+
+```
+Here is my configuration:
+
+DATABASE_URL=postgres://admin:password@localhost
+```
+
+Output:
+
+```
+Secrets detected:
+
+[1] OpenAI API Key
+location: line 3
+risk: HIGH
+
+[2] Database Credentials
+location: line 4
+risk: CRITICAL
+```
+
+---
+
+## Installation
+
+```bash
+# Install from PyPI (Recommended)
+pip install py-secret-scan
+
+# Run
+secret-scan example_file.txt
+```
+
+### Developer Installation
+
+```bash
+# Create virtual environment
+python3 -m venv .venv
+source .venv/bin/activate
+
+# Install in editable mode
+pip install -e .
+```
+
+Or scan text directly:
+```bash
+# Standard scan
+secret-scan --text "My API key is AIzaSy-12345"
+
+# Force scan all lines (bypasses keyword filters)
+secret-scan --force-scan-all .
+```
+
+### Data Obfuscation & Masking
+
+You can redact sensitive data from logs or prompts while preserving the rest of the text. This is useful for sanitizing data before sharing it with an LLM or for safe debugging.
+
+Enable obfuscation with the `--obfuscate` flag:
+
+```bash
+# Default mode: redact (Redacts middle of the secret)
+# Input: "My key is ghp_1234567890abcdefghijklmnopqrstuvwx"
+# Output: "My key is ghp_...uvwx"
+cat logs.txt | secret-scan --obfuscate
+```
+
+Choose different obfuscation strategies with `--obfuscate-mode`:
+
+#### 1. `redact` (Default)
+Partial masking that keeps the prefix/suffix for context but hides the sensitive core.
+* **Example:** `AKIA...CDEF`
+
+#### 2. `hash`
+Replaces secrets with a consistent, short SHA-256 hash. Identical secrets will result in identical hashes, which is crucial for debugging data flows without seeing the actual values.
+* **Example:** `[HASHED_d8c7b92f4a19]`
+
+#### 3. `synthetic` (Recommended for LLM Prompts)
+Replaces secrets with realistic-looking fake data that matches the original format (using the `Faker` library). This allows LLMs to still "understand" the structure of your data (e.g., seeing a fake AWS key where a real one was) without exposing real credentials.
+* **Example (AWS ID):** `AKIAJ7O2N6M4L9K0P8R1`
+* **Example (GitHub Token):** `ghp_zXyWvUtSrQpOnMlKjIhGfEdCbA9876543210`
+* **Example (Email):** `fake_user@example.org`
+
+```bash
+# Use synthetic mode for realistic placeholders
+secret-scan --obfuscate --obfuscate-mode synthetic logs.txt
+```
+
+### Custom CLI helpers
+
+The repository ships with a few convenience commands:
+
+* `./run.sh <file>` is deprecated, use `secret-scan <file>` directly.
+* `secret-scan --text "<string>"` runs the scanner on an inline string (useful when building prompts before sending them to an LLM).
+* `python tools/generate_test_data.py` rebuilds `data/test_data.json` from `data/rules.json` and should be rerun whenever the rule set changes.
+
+## Usage Example
+
+```
+$ secret-scan test_file.py
+
+⚠ Secrets detected: 1
+- CRITICAL: 1
+
+Type: Database Credentials
+Location: line 10
+Risk: CRITICAL
+Content: post...ocal (redacted)
+```
+
+# Test Data & Custom Cases
+
+Every rule in `data/rules.json` maps to a base64-encoded positive and negative sample under `data/test_data.json`. `tools/generate_test_data.py` drives the data:
+
+* It loads each regex, runs it through `exrex` (with length caps) to emit matches, and encodes them so the detector tests operate on identical strings as production data.
+* Negatives are hand-crafted near-misses that resemble real-world secrets but should not trigger a hit.
+* Rules listed in `STRICT_RULES` bypass the default `encode_str` mutation because even inserting `DUMMY_IGNORE` would break the required format.
+* Custom helpers generate valid payloads for the trickiest patterns (`auth0_domain_url`, `skybiometry_api_key`, `okta_api_domain_url`, `facebook_oauth_id`, `linemessaging_api_key`, `nethunt_api_key`) so the detector still sees legal samples even though those regexes restrict character sets or lengths tightly.
+
+Run `python tools/generate_test_data.py` after any rule changes; it prints progress every 100 rules and overwrites `data/test_data.json` with the refreshed corpus that powers the pytest suite.
+
+
+# Use Cases
+
+### AI Application Development
+
+Developers building:
+
+* chatbots
+* RAG pipelines
+* AI agents
+* coding assistants
+
+can scan prompts before sending them to LLM APIs.
+
+---
+
+### Security Auditing
+
+Security teams can analyze:
+
+* prompt logs
+* application logs
+* LLM interaction history
+
+to ensure no secrets were exposed.
+
+---
+
+### Compliance
+
+Organizations can enforce policies preventing sensitive information from being sent to external AI providers.
+
+---
+
+### DevSecOps Integration
+
+The scanner can be integrated into:
+
+* CI/CD pipelines
+* AI gateways
+* API proxies
+* developer tooling
+
+---
+
+## PII Detection
+
+The scanner now supports detecting Personal Identifiable Information (PII) including emails, phone numbers, credit cards, and SSNs.
+
+Enable PII detection with the `--pii` flag:
+
+```bash
+# Scan a file for secrets and PII
+secret-scan --pii example_file.txt
+
+# Limit PII scanning to specific regions (e.g., US only for SSNs and US phone numbers)
+secret-scan --pii --pii-region US example_file.txt
+```
+
+PII findings are integrated into the multi-tier reporting system, where highly structured secrets (Tier 1) take precedence over contextual or generic entropy hits.
+
+---
+
+# Target Users
+
+### AI Developers
+
+Engineers building LLM-powered applications.
+
+### Security Engineers
+
+Teams responsible for application security reviews.
+
+### AI Startups
+
+Companies working with prompt engineering and LLM pipelines.
+
+---
+
+# Roadmap
+
+The project evolves in several stages.
+
+### CLI Scanner
+
+A lightweight command-line tool for scanning prompts and logs.
+
+### API Service
+
+A service that allows AI systems to validate prompts before sending them to LLM providers.
+
+### Developer Tooling
+
+Integration with:
+
+* IDE plugins
+* Git hooks
+* CI pipelines
+
+### Enterprise Security Platform
+
+Future capabilities may include:
+
+* real-time prompt filtering
+* AI data loss prevention (DLP)
+* secret monitoring across AI infrastructure
+
+---
+
+# Why This Matters
+
+AI-assisted development dramatically increases the speed of coding and debugging, but it also increases the risk of accidentally exposing sensitive data.
+
+Developers frequently paste large blocks of code or logs into AI systems without reviewing them for secrets.
+
+LLM Secrets Leak Detector provides a safety layer that prevents confidential data from leaving the organization.
+
+---
+
+# License
+
+MIT License
+
+
+[1]: https://arxiv.org/abs/2307.00714?utm_source=chatgpt.com "A Comparative Study of Software Secrets Reporting by Secret Detection Tools"
+[2]: https://www.gitguardian.com/solutions/secrets-detection?utm_source=chatgpt.com "Secrets Detection: Scan Code for Exposed API Keys and Credentials | GitGuardian"
+[3]: https://github.com/mazen160/secrets-patterns-db?utm_source=chatgpt.com "GitHub - mazen160/secrets-patterns-db: Secrets Patterns DB: The largest open-source Database for detecting secrets, API keys, passwords, tokens, and more."