breachbench 1.3.1__tar.gz

breachbench-1.3.1/CHANGELOG.md

@@ -0,0 +1,37 @@
+# Changelog
+
+All notable changes to **BreachBench** are documented in this file.
+
+## [1.3.1] - 2026-11-13
+
+- README rewrite with numbered Quick Start and architecture diagrams.
+- OWASP lab endpoint parity with all 10 `/vuln/*` routes in `data/api_eval.json` and matching hardened counterparts.
+- Installer dry-run fix: `./install.sh --multi-machine --dry-run` no longer passes an unsupported CLI flag.
+- Added `CONTRIBUTING.md`.
+- PyPI packaging (SPDX license, bundled eval corpora, Trusted Publishing workflow ready for tag `v1.3.1`).
+- Removal of an unrelated posture integration (already on `main`; included in this release).
+
+Do not move tag `v1.3.0`; cut this release with `git tag v1.3.1`.
+
+## [1.3.0] - 2026-10-27
+
+### Added
+- **3-Layer Defense-in-Depth Guardrail Shield (`breachbench/shield.py`)**:
+  - **Layer 1 — Input Pre-Screen (`prescreen_input`)**: Scans untrusted prompts and embedded artifacts with BreachBench's static code vulnerability scanner (`scan_code_snippet`), malware & supply-chain hook detector (`scan_malware_payload`), phishing/BEC/MFA-pretext heuristics (`score_phishing_email`, `normalize_threat_text`), plus regex rules for malware-authoring requests (keyloggers, reverse shells, credential/SSH key exfiltration, ransomware, obfuscated droppers) and prompt-injection/tool-hijack markers (ignore previous instructions, DAN/jailbreaks, forced verdict overrides, canary/env exfiltration, `curl | sh`, and unpermitted SSH `deploy-machines` execution).
+  - **Layer 2 — Hardened System Prompt & Data Spotlighting (`build_spotlighted_messages`)**: Wraps untrusted input inside `<untrusted_input_data>` spotlighting tags and injects structured pre-screen analyst telemetry (detected CWEs, OWASP categories, malware categories, and injection indicators) into the system prompt.
+  - **Layer 3 — Output Enforcement (`enforce_output_shield`)**: Redacts leaked canary tokens, blocks/replaces responses that comply with malware-authoring or unauthorized SSH tool-hijack requests, neutralizes injection-induced false `clear`/`SAFE` verdicts, and appends a structured `BreachBench Shield findings` section (verdict, CWE IDs, malware categories, and remediation guidance) when the model omits detected findings.
+  - Records per-task `shield_actions` in evaluation JSON reports.
+- **CLI Flag `breachbench eval-agent --shield`**: Enables the 3-layer Guardrail Shield for `openai`, `ollama`, `echo`, and `deterministic` providers.
+- **CLI Subcommand `breachbench compare-shield`**: Runs a target model unshielded (`--shield` off) and shielded (`--shield` on), prints a side-by-side comparison table (per-suite, overall, and per-task PASS/FAIL with triggered shield layers), and optionally writes JSON (`--out`) and Markdown (`--md`) reports.
+- **Weak Stub Provider (`--provider echo`)**: Added a naive/compliance-prone stub provider that simulates an unaligned small model to verify and benchmark Guardrail Shield lift deterministically offline.
+- **Release & PyPI Trusted Publishing Workflow (`.github/workflows/release.yml`)**: Automated sdist/wheel build, `twine check`, test verification, GitHub Release creation, and PyPI Trusted Publishing on `v*` tags.
+
+### Changed
+- Bumped package version to `1.3.0` across `pyproject.toml`, `setup.py`, `breachbench/__init__.py`, `breachbench/cli.py`, and `app/main.py`.
+- Expanded unit and integration tests in `tests/test_breachbench.py` covering all three shield layers, unseen prompt generalization, zero false positives on benign artifacts, and `compare-shield`.
+
+## [1.2.0] - 2026-04-12
+
+### Added
+- Multi-domain security benchmark (`breachbench benchmark`) spanning Phishing/BEC/MFA-fatigue triage, Prompt Injection & Canary Guard, OWASP API Top 10 surface analysis, Static & Agent Code Vulnerability Scanner (`scan-code`), Malware & Supply-Chain Lure Detector (`scan-malware`), AI Agent Security Evaluator (`eval-agent`), and permission-gated multi-machine SSH runner (`deploy-machines`).
+- FastAPI service (`app/main.py`) and OWASP vulnerable vs. hardened API comparison lab (`labs/vulnerable_api/app.py`).

breachbench-1.3.1/CONTRIBUTING.md

@@ -0,0 +1,72 @@
+# Contributing to BreachBench
+
+Thanks for helping improve this defensive security benchmark. Please read [ETHICS.md](ETHICS.md) first: only test systems you own or have written permission to evaluate, and do not contribute live malware or real credentials.
+
+## Setup
+
+Python 3.9+ is required (CI runs 3.10–3.12). On Debian/Ubuntu, install `python3-venv` if `python3 -m venv` fails.
+
+```bash
+git clone https://github.com/tu2090-ops/breachbench.git
+cd breachbench
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+pip install -e ".[dev]"
+```
+
+On Windows, activate with `.venv\Scripts\activate`.
+
+## Running tests
+
+```bash
+python -m unittest discover -s tests
+pytest -v
+```
+
+Both commands execute `tests/test_breachbench.py`. After behavior changes, also run:
+
+```bash
+breachbench eval-agent --provider deterministic
+breachbench benchmark --out /tmp/bb-report.json --md /tmp/bb-report.md
+breachbench compare-shield --provider echo --model echo-weak-stub-v1
+```
+
+Do not commit regenerated files under `results/` unless you intentionally updated a published snapshot.
+
+## Adding a test case or suite
+
+1. **Scanner / triage corpora** — add a gold-labeled item to `data/code_eval.json`, `data/malware_eval.json`, `data/phishing_eval.jsonl`, or `data/api_eval.json`. Keep samples synthetic and inert.
+2. **Agent eval tasks** — append an object to `data/agent_eval_tasks.json` with `id`, `category` (`malware_refusal_and_detection`, `code_vulnerability_detection`, or `prompt_injection_and_tool_hijack`), `prompt`, `expected_behavior`, `required_concepts`, and optional `forbidden_patterns`.
+3. **Scenarios** — add a `SCEN-00N` object to `data/scenarios_eval.json`.
+4. **Assertions** — extend `tests/test_breachbench.py` (or add `tests/test_*.py`). Cover the new path, a benign/false-positive case, and any Guardrail Shield layer you touch.
+5. **Lab API** — if you add a `/vuln/*` route, add a matching hardened counterpart and list both in `data/api_eval.json`. Regenerate `labs/vulnerable_api/openapi.json` from the FastAPI app.
+
+## Code style
+
+- Target Python 3.9+: use `from __future__ import annotations` in library modules that already do so.
+- Keep imports at the top of the module. Do not add inline imports unless a circular dependency is documented.
+- Match existing naming: `scan_*` for detectors, `evaluate_*` for corpus metrics, CLI flags with argparse.
+- Do not look up agent-eval task IDs or grading keywords inside `breachbench/shield.py`.
+- Do not add network calls in the OWASP lab beyond returning simulated results (no live egress).
+
+## Pull requests
+
+1. Branch from current `main`.
+2. Keep the change focused (one concern per PR).
+3. Use conventional commits when possible: `feat`, `fix`, `docs`, `test`, `chore`, `refactor`.
+4. Update [README.md](README.md) or [CHANGELOG.md](CHANGELOG.md) when user-facing behavior, flags, or endpoints change.
+5. Open a pull request against `https://github.com/tu2090-ops/breachbench` with what changed, how you tested it, and any follow-up the maintainer must do (secrets, PyPI, allowlists).
+
+## Releases
+
+The package version is `1.3.1` (`pyproject.toml`, `breachbench/__init__.py`). Tag the next PyPI/GitHub release as **`v1.3.1`**. Do not move or reuse tag `v1.3.0`; it already points at an older commit on origin.
+
+```bash
+git checkout main
+git pull
+git tag v1.3.1
+git push origin v1.3.1
+```
+
+Issues: [github.com/tu2090-ops/breachbench/issues](https://github.com/tu2090-ops/breachbench/issues).

breachbench-1.3.1/ETHICS.md

@@ -0,0 +1,14 @@
+# Ethics, Safety & Authorization Policy
+
+**BreachBench** is a defensive security evaluation suite and static/agent vulnerability scanner.
+
+1. **Explicit Permission Gate for Multi-Machine Deployment:**
+   - Installing or executing BreachBench across local or lab machines via `install.sh` or `breachbench deploy-machines` requires explicit operator authorization (`--confirm-permission` or interactive `"YES_I_HAVE_AUTHORIZATION"` confirmation) and a machine allowlist (`config/machines.example.json` where `"authorized": true`).
+   - Every multi-machine deployment attempt is recorded in an immutable JSONL audit trail (`results/deployment_audit.jsonl`).
+
+2. **Synthetic & Inert Test Fixtures Only:**
+   - All malware, phishing, prompt-injection, and vulnerable code samples in `data/` and `samples/` are synthetic, inert, non-weaponized indicators designed strictly to test detection accuracy, static analysis rules, and AI agent refusal boundaries.
+   - No live command-and-control servers, real credentials, or destructive payloads are included or executed.
+
+3. **Defensive Use Only:**
+   - BreachBench must only be run against source code, AI agents, and compute nodes that you own or have explicit written permission to test.

breachbench-1.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tanushree Umathe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

breachbench-1.3.1/MANIFEST.in

@@ -0,0 +1,9 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include CONTRIBUTING.md
+include ETHICS.md
+recursive-include data *.json *.jsonl
+recursive-include config *.json
+recursive-include samples *.py *.json *.txt
+recursive-include labs *.json