safe-pip-scanner 1.4.0__tar.gz

safe_pip_scanner-1.4.0/CHANGELOG.md

@@ -0,0 +1,308 @@
+# Changelog
+
+All notable changes to safe-pip are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+---
+
+## [1.4.0] — 2026-06-14
+
+### 🎯 Scan Accuracy Improvements
+
+#### Self-referential Info findings suppressed
+- Scanning `requests` showed `● Info  Network call in __init__.py: requests.get(` — flagging
+  the requests package for using `requests.get()` internally is a false positive.
+- **Fix** (`code_scanner.py`): Info findings whose text references the scanned package's
+  own name followed by a dot are filtered out. e.g. "requests.get" while scanning "requests".
+
+#### `burst releases` tag no longer shown for trusted packages
+- Well-known packages (numpy, requests, django, boto3) have many release bursts by nature.
+  Showing `[red]burst releases[/red]` for them was noise that undermined trust in the signal.
+- **Fix** (`display.py`): burst and dormancy-break tags are suppressed when `score=0` and
+  `decision=INSTALL` — i.e. packages the scorer has fully cleared. The signals remain in
+  the underlying data and still fire for suspicious packages.
+
+#### Author field fixed for modern PyPI metadata
+- All packages showed `author: unknown` because PyPI's v2 API moved the author to
+  `author_email` as `"Display Name <email>"` format, and `maintainers[]` list.
+- First fix extracted the wrong part (`me` from `Kenneth Reitz <me@kennethreitz.org>`).
+- **Final fix** (`scanner.py`): takes the display name *before* `<`, requires `<` to be
+  present (ignores bare email addresses), then falls back to `maintainers[0].name`.
+- Result: `requests` now shows `author: Kenneth Reitz` instead of `author: me`.
+
+### 🛡 Threat Intelligence Expansion
+
+#### Threat feed: 66 → 143 entries (+77)
+New attack families: boto3/AWS (credential theft), SQLAlchemy, click/rich/typer,
+pydantic/fastapi, paramiko/fabric (infrastructure), pytest/CI tools (supply chain),
+pyyaml, pillow/opencv, discord bots, and confirmed malicious packages from documented
+PyPI incidents (xmrig, ascii2text, discordspy, ultrarequests, pptest, dpp-discord).
+
+#### KNOWN_DANGEROUS: 5 → 15 entries (+10)
+Added: `ecdsa` (WARN — timing attacks), `rsa` (WARN — CVE-2020-25658), `colourama`,
+`python3-dateutil`, `ascii2text`, `ultrarequests`, `xmrig`, `discordspy`, `pptest`.
+
+#### Scoring fix: KNOWN_DANGEROUS floor applied after reputation bonuses
+- `ecdsa` and `rsa` were scoring 26 (INSTALL) because a long-standing-reputation bonus
+  (−5) was subtracting from the floor (31), pulling them below the WARN threshold.
+- **Fix**: floor is now re-applied after all bonuses, guaranteeing `score ≥ 31` → WARN.
+
+### ⚙ Infrastructure
+
+#### Claude model updated to `claude-sonnet-4-6`
+- Was pinned to `claude-sonnet-4-5`. Updated to current release.
+
+#### AI prompt: false-positive avoidance rules added
+- Added explicit instructions to the AI scoring prompt:
+  - Skip self-referential findings (e.g. `requests.get` inside `requests`)
+  - Don't penalise HTTP client libraries for making HTTP calls
+  - Don't penalise deployment tools for subprocess usage
+  - `import winreg` in cross-platform packages = INFO only, not a risk signal
+
+---
+
+
+## [1.3.0] — 2026-06-13
+
+### 🐛 Bug Fixes
+
+#### Removed `requests` as a runtime dependency — critical fix
+- `safe-pip` exited immediately on startup if `requests` was not installed, even
+  though `scanner.py` already used stdlib `urllib` for all HTTP calls.
+- **Root cause**: `osv.py`, `pypistats.py`, and `code_scanner.py` all imported
+  `requests` directly, and `__init__.py` listed it as a hard required dep.
+- **Fix**: all three modules converted to `urllib.request` / `urllib.error`.
+  `requests` is no longer imported anywhere in the package.
+- **Fix**: removed `"requests"` from `_REQUIRED` in `__init__.py`.
+- **Fix**: `doctor` command now lists `requests` as *optional*.
+- After this fix the following all work with `requests` uninstalled:
+  ```
+  safe-pip scan requests
+  safe-pip scan flask --deps
+  safe-pip update-db
+  python -m safe_pip scan flask
+  safe-pip watch status
+  ```
+
+#### Decision reason no longer shows Info-level findings
+- `Decision: ✓ INSTALL  Network call in __init__.py: requests.get(` was appearing
+  for the `requests` package — an internal diagnostic surfacing as the headline verdict.
+- **Fix**: `local_scorer.py` now excludes `category="Info"` findings when selecting
+  the decision reason. Only real risk signals (Reputation, CVE, Code, Maintenance)
+  are eligible. Info findings still appear in the full findings list.
+- After fix: `Decision: ✓ INSTALL  'requests' is a well-known, widely-trusted PyPI package`
+
+#### Cache hits now display `(cached)` instead of replaying the original scan time
+- Repeated scans showed identical times (e.g. `6.64s` three times), making it appear
+  the cache wasn't working. The cache **was** working — the stored `elapsed` from the
+  first scan was being redisplayed on every hit.
+- **Fix** (`scanner.py`): cache hits return a shallow copy with `elapsed` set to the
+  actual lookup time and `from_cache: true` added.
+- **Fix** (`display.py`): when `from_cache` is true, the header shows `(cached)`;
+  live scans show `X.XXs` (minimum `0.01s`).
+- Result:
+  ```
+  Scan 1:  requests v2.34.2  score 0/100  LOW  ✓ INSTALL  6.64s
+  Scan 2:  requests v2.34.2  score 0/100  LOW  ✓ INSTALL  (cached)
+  Scan 3:  requests v2.34.2  score 0/100  LOW  ✓ INSTALL  (cached)
+  ```
+
+#### `--no-cache` flag now forwarded correctly from `scan` command
+- `safe-pip scan requests --no-cache` was silently ignored — parsed but never passed
+  to `_run_scan`. Fixed: `_run_scan(pkg, cfg, scan_output, no_cache=no_cache)`.
+
+#### Cache write errors now logged at WARNING level
+- `set_cached()` swallowed all write errors silently (`log.debug`), masking failures.
+- Changed to `log.warning` so errors surface under `--debug`.
+- `json.dumps()` now called before opening the file to avoid corrupt partial writes.
+
+#### Removed duplicate `scan_cache` import inside `scan()`
+- `scanner.py` imported `cache_key, get_cached, set_cached` at module level and again
+  inside `scan()`. The redundant inner import was removed.
+
+### ✨ New
+
+#### `safe-pip cache debug <package>` — cache self-diagnostic
+- Diagnose cache state for any package:
+  ```
+  safe-pip cache debug requests
+  ```
+  Shows: cache directory, key, file path, file age, TTL validity, and whether
+  `get_cached()` returns a hit. If the file exists but parsing fails, reports
+  the exact JSON error.
+
+---
+
+## [1.1.0] — 2026-06-02
+
+### 🐛 False Positive Fixes
+
+#### Dormancy break — no longer a CODE WARNING for established packages
+- Packages with ≥10 releases before a dormancy gap (e.g. `numpy`) now generate
+  an **INFO** finding instead of a `medium` warning.  Established packages that
+  go quiet for a year then revive are almost never malicious.
+- New message: *"likely legitimate revival of established package"* vs the old
+  generic *"possible maintainer takeover"* which was triggering for `numpy` and
+  other long-lived libraries.
+
+#### Hardcoded IP — OID/ASN.1 version strings no longer flagged
+- Improved IP-detection regex: first octet must now be ≥ 20 to avoid matching
+  X.509 OID strings like `2.5.29.9` found in `cryptography` and similar TLS libs.
+- Added context-aware downgrade: if the surrounding line contains ASN/OID/X.509
+  context keywords, or if the second octet is ≤ 9 (typical OID), the finding is
+  downgraded to **INFO** — shown but never scored.
+- Eliminates false positives: `beautifulsoup4` `dammit.py: 13.2.3.1` and
+  `cryptography` `_oid.py: 2.5.29.9` now produce at most an INFO note.
+
+### ✨ New Features
+
+#### "Did you mean?" for typosquats
+- When a package is detected as a likely typosquat, the scan output now shows:
+  `Did you mean: requests  (run: pip install requests)`
+- Available in both Rich terminal and JSON output (`did_you_mean` field).
+
+#### Suggested replacement for known deprecated/unsafe packages
+- `pycrypto` → `pycryptodome`, `nose` → `pytest`, `mock` → `unittest.mock`, and
+  10 other known-deprecated packages now display:
+  `Suggested replacement: pycryptodome  (run: pip install pycryptodome)`
+- Available in both Rich terminal and JSON output (`suggested_replacement` field).
+
+#### JSON output (`--output json` / `safe-pip scan --output json`)
+- JSON output now includes two new fields:
+  - `"did_you_mean"`: canonical name if typosquat detected, else `null`
+  - `"suggested_replacement"`: replacement package name if known, else `null`
+
+#### `--fail-on-high` for CI/CD pipelines (`scan-file`)
+- `safe-pip scan-file requirements.txt --fail-on-high`
+- **Exit 0** = all packages are LOW or MEDIUM risk (pass)
+- **Exit 1** = at least one package is HIGH risk (BLOCK)
+- Complements the existing `--fail-on-warn` (exit 1 on any WARN or BLOCK).
+- Documented in README CI/CD section with GitHub Actions example.
+
+#### HTML report (`--html report.html`)
+- `safe-pip scan-file requirements.txt --html report.html`
+- Generates a self-contained dark-theme HTML report with:
+  - Summary cards (safe / warned / blocked / total)
+  - Horizontal bar chart of risk distribution
+  - Full results table with score, verdict, CVE count, top finding, decision
+  - "Did you mean?" and "Suggested replacement" inline in the table
+- No external dependencies — fully self-contained single file.
+
+#### `--fail-on-high` wired into `batch` command too
+- `batch` now accepts `--fail-on-high` independently of `--fail-on-warn`.
+- Previously `scan-file --fail-on-high` incorrectly conflated both flags; now:
+  - `--fail-on-warn` → exit 1 on WARN **or** BLOCK
+  - `--fail-on-high` → exit 1 on BLOCK only (WARN = exit 0)
+  - Both flags can be combined for custom gate logic.
+
+#### `--json` shorthand on `scan`
+- `safe-pip scan requests --json` is now equivalent to `safe-pip scan requests --output json`.
+- Cleaner for shell pipelines: `safe-pip scan requests --json | jq .decision`
+
+#### `--deps` — dependency tree analysis
+- `safe-pip scan requests --deps` scans declared dependencies (one level deep).
+- Reads `requires_dist` from PyPI metadata, extracts package names, and runs
+  a full scan on each transitive dependency.
+- Rich terminal output shows a tree:
+  ```
+  Dependencies of requests:
+  ├─ urllib3   score 5   ✓ INSTALL
+  ├─ certifi   score 3   ✓ INSTALL
+  └─ idna      score 4   ✓ INSTALL
+  ```
+- JSON output emits a separate `{"deps_of": "requests", "dependencies": [...]}` object.
+- Returns exit 1 if any dependency is risky (respects `--fail-on-warn`/`--fail-on-high`).
+
+---
+
+
+- New top-level command that downloads and caches:
+  - Typosquat blocklist
+  - Malicious package list with CVE mappings
+  - Top-8000 PyPI packages feed
+- Progress bar with per-source entry counts.
+- Replaces `safe-pip update` for users who want explicit control over DB freshness.
+
+---
+
+
+
+### 🎉 First stable release
+
+#### Core scanner
+- **No API key required** — full functionality with local rule-based scorer
+- **7-stage pipeline**: PyPI metadata → typosquat → CVE → download stats → release anomaly → static code analysis → risk scoring
+- **PackageNotFoundError** — non-existent clean packages forwarded to real pip automatically
+- **Claude AI upgrade path** — set `ANTHROPIC_API_KEY` for enhanced analysis
+
+#### Typosquat detection
+- Levenshtein distance engine against 8,000+ top PyPI packages (live feed, 24h cache)
+- Keyboard-proximity, homoglyph, vowel-swap, and affix-padding checks
+- 70+ curated known-malicious entries (requestss, colourama, langchian, etc.)
+- `safe-pip update` — refresh threat feed on demand
+
+#### Static code analysis (NEW)
+- Downloads wheel/sdist **without installing** and scans Python source
+- Detects: encoded payloads, reverse shells, data exfiltration, credential leaks
+- Detects: hardcoded external IPs, dynamic exec/eval, install hook abuse
+- INFO-level findings for legitimate patterns (ctypes, requests.get, private IPs)
+
+#### Release anomaly detection (NEW)
+- Burst pattern detection (3+ releases within 24 hours)
+- High velocity scoring (>1 release/day)
+- Dormancy break detection (inactive years → sudden activity = possible takeover)
+- Maintainer reputation scoring
+
+#### CVE analysis
+- Real-time OSV.dev lookups — critical/high/medium/low severity
+- Skips OSV for packages not on PyPI (eliminates false positives)
+- Version-aware: only flags CVEs affecting the actual installed version
+
+#### SBOM generation (NEW)
+- `safe-pip sbom` — CycloneDX 1.4 JSON output
+- Accepted by GitHub Dependency Review, FOSSA, Snyk, and enterprise tools
+- Includes PURL, license, score, verdict, findings per component
+
+#### Watch mode
+- `safe-pip watch enable` — intercepts `pip install` system-wide
+- **PowerShell**: function alias in profile (PS5 + PS7)
+- **CMD**: pip.bat written beside pip.exe
+- **Admin CMD**: UAC elevation to install system-level shim
+- `safe-pip watch disable` — clean removal from all locations
+- `safe-pip watch status` — shows coverage per terminal type
+
+#### Dashboard
+- `safe-pip dashboard` — local browser UI at localhost:7676
+- Stats: total scans, verdict breakdown, blocked count, average score
+- Charts: verdict donut, 30-day timeline (install/warn/block)
+- Table: sortable, filterable, searchable scan history
+- Auto-refreshes every 60 seconds
+
+#### CLI improvements
+- `safe-pip scan requirements.txt` — auto-detects requirements files
+- `safe-pip scan-file requirements.txt` — explicit shorthand
+- Package aliases: `sklearn→scikit-learn`, `cv2→opencv-python`, `PIL→pillow`, etc.
+- Clean error messages — no tracebacks shown to users
+- `--output rich|plain|json` on all scan commands
+- `--sarif FILE` — SARIF 2.1.0 output for GitHub Security tab
+
+#### False positive fixes
+- `requests.get()` → INFO only (not scored)
+- `import ctypes` → INFO only (not scored)
+- `0.0.0.0`, `127.x`, `10.x`, `192.168.x`, RFC1918 → excluded from IP detection
+- `compile(expression, "<string>")` → INFO (Python built-in math eval)
+- `tensorflow-gpu`, `tensorflow-cpu`, `tensorflow-intel` → trusted (official Google)
+- Release anomalies suppressed for trusted packages (numpy, etc.)
+
+#### Test suite
+- **605 tests** across 12 test files
+- `tests/safe_packages.txt` — 31 trusted packages verified INSTALL
+- `tests/malicious_packages.txt` — 22 attack packages verified BLOCK
+- `tests/edge_cases.txt` — aliases, deprecated-legit, CVE-warning packages
+- Integration tests cover full pipeline end-to-end
+
+---
+
+## [0.x.x] — Pre-release development
+
+Internal development iterations — see git history.

safe_pip_scanner-1.4.0/CONTRIBUTING.md

@@ -0,0 +1,93 @@
+# Contributing to safe-pip
+
+Thank you for your interest in contributing! This document covers how to set
+up a development environment, run tests, and submit changes.
+
+---
+
+## Development setup
+
+```bash
+# Clone and install in editable mode with dev dependencies
+git clone https://github.com/safe-pip/safe-pip
+cd safe-pip
+pip install -e ".[dev]"
+
+# Set your Anthropic API key (required for scanner tests that call Claude)
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+## Running tests
+
+```bash
+# All tests
+pytest
+
+# Specific file
+pytest tests/test_cli.py -v
+
+# With coverage
+pip install pytest-cov
+pytest --cov=safe_pip --cov-report=term-missing
+```
+
+## Linting
+
+```bash
+pip install ruff
+ruff check safe_pip/ tests/ --select E,F,W --ignore E501
+```
+
+## Project structure
+
+```
+safe_pip/
+├── __init__.py      # version
+├── cli.py           # Click CLI — all commands
+├── scanner.py       # Core pipeline: PyPI → typosquat → OSV → AI
+├── typosquat.py     # Levenshtein similarity engine + threat DB
+├── osv.py           # OSV CVE database client
+├── pypistats.py     # pypistats.org download stats client
+├── policy.py        # Rule evaluation engine + config loader
+├── display.py       # Rich terminal UI
+├── db.py            # SQLite persistence layer
+├── sarif.py         # SARIF 2.1.0 report generator
+└── py.typed         # PEP 561 marker
+
+tests/
+├── test_build.py    # Package metadata + CLI smoke tests
+├── test_cli.py      # CLI command tests (Click test runner)
+└── test_db.py       # SQLite layer tests (in-memory DB)
+```
+
+## Adding a new scanning signal
+
+1. Add your logic to `scanner.py` (fetch data) or `typosquat.py` (similarity)
+2. Pass the result into `_build_prompt()` so Claude sees it
+3. Add a corresponding policy `condition` string to `policy.py`
+4. Add a stage label to `display.STAGE_LABELS`
+5. Write tests in `tests/test_cli.py` or a new `tests/test_<module>.py`
+
+## Adding a new CLI command
+
+1. Define a new `@main.command()` in `cli.py`
+2. Add it to the `--help` assertion in `tests/test_build.py::TestCLIIntegration::test_help_flag`
+3. Document it in `README.md` and `CHANGELOG.md`
+
+## Submitting a pull request
+
+1. Fork the repo and create a feature branch: `git checkout -b feat/my-feature`
+2. Make your changes and ensure all tests pass: `pytest`
+3. Ensure lint passes: `ruff check safe_pip/ tests/ --select E,F,W --ignore E501`
+4. Update `CHANGELOG.md` under `[Unreleased]`
+5. Open a PR — the CI pipeline will run automatically
+
+## Reporting bugs
+
+Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md).
+Please include the output of `safe-pip --version` and the full error message.
+
+## Security issues
+
+Do **not** open a public issue for security vulnerabilities. Email the
+maintainers directly — contact details are in the GitHub repository.

safe_pip_scanner-1.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 safe-pip contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

safe_pip_scanner-1.4.0/MANIFEST.in

@@ -0,0 +1,7 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+include safe_pip/py.typed
+recursive-include tests *.py
+recursive-include .github *