vamp-llm-probe 1.4.0__tar.gz

vamp_llm_probe-1.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 VampSecure Studios
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vamp_llm_probe-1.4.0/PKG-INFO

@@ -0,0 +1,363 @@
+Metadata-Version: 2.4
+Name: vamp-llm-probe
+Version: 1.4.0
+Summary: LLM API security auditor — prompt injection, jailbreak, ASCII smuggling, data exfiltration, OWASP LLM Top 10 mapping
+Author-email: VampSecure Studios <contact@vampsecurestudios.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Vampsecure-Labs/vamp-llm-probe
+Project-URL: Repository, https://github.com/Vampsecure-Labs/vamp-llm-probe
+Keywords: llm,security,prompt-injection,jailbreak,ascii-smuggling,owasp,pentest,ai-security
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Information Technology
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Security
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: fpdf2>=2.7
+Dynamic: license-file
+
+# vamp-llm-probe
+
+![Python 3.8+](https://img.shields.io/badge/Python-3.8%2B-blue?style=flat-square)
+![Version](https://img.shields.io/badge/version-1.3.0-dc143c?style=flat-square)
+![License MIT](https://img.shields.io/badge/License-MIT-green?style=flat-square)
+![VampSecure Labs](https://img.shields.io/badge/VampSecure-Labs-red?style=flat-square)
+
+Security auditor for language model inference API endpoints. Sends crafted HTTP requests to detect vulnerabilities without relying on any AI SDK — only `aiohttp`, `asyncio`, and the standard library.
+
+> **For authorized use only.** Run this tool exclusively against endpoints you own or have explicit written permission to test.
+
+---
+
+## Features
+
+- **6 audit phases** covering reconnaissance, prompt injection, restriction bypass, data extraction, access controls and **adversarial dataset red team**
+- **Truly bilingual detection** — refusal and compliance heuristics cover both English and Spanish; models responding in Spanish are correctly evaluated regardless of the prompt language
+- **Bundled adversarial datasets** — 666 jailbreaks (EN) + 50 injection vectors (ES) + 30 jailbreaks (ES) + 210 injection prompts (EN) + 390 forbidden questions (13 content-policy categories)
+- **6-subtest Phase 6**: A (injection EN), B (jailbreak EN), C (forbidden questions), A_es (injection ES), B_es (jailbreak ES), **D (ASCII smuggling)**
+- **ASCII smuggling detection** — active (subtest D sends Unicode Tags payloads) and passive (scans every Phase 2 response for hidden Unicode Tags characters in output)
+- **No AI SDK dependency** — pure HTTP-level testing via `aiohttp`
+- **Async execution** — parallel requests for rate-limiting tests
+- **Structured findings** with severity levels (CRITICAL / HIGH / MEDIUM / LOW / INFO)
+- **OWASP mapping** — every finding is automatically tagged with the corresponding [OWASP LLM Top 10 2025](https://owasp.org/www-project-top-10-for-large-language-model-applications/) and [OWASP Agentic AI Top 10 2026](https://owasp.org/www-project-agentic-ai-threats/) categories; visible in JSON output and HTML reports
+- **Professional reports** — JSON (machine-readable) and HTML (client-delivery) with OWASP tags on each finding card
+- **Exit codes** suitable for CI/CD pipeline integration
+- **Auto-detection** of API format and available models
+
+---
+
+## Installation
+
+```bash
+pip install -r requirements.txt
+```
+
+Requirements: Python 3.8+ and `aiohttp>=3.9.0`.
+
+---
+
+## Usage
+
+### Basic scan
+
+```bash
+python3 vamp_llm_probe.py --endpoint http://localhost:11434
+```
+
+### With API key and specific model
+
+```bash
+python3 vamp_llm_probe.py \
+  --endpoint http://api.example.com \
+  --api-key sk-your-api-key \
+  --model llama3:8b
+```
+
+### Full engagement with HTML report
+
+```bash
+python3 vamp_llm_probe.py \
+  --endpoint http://inference.internal:8080 \
+  --api-key Bearer_TOKEN \
+  --client "Empresa SL" \
+  --engagement "Pentest Infraestructura 2026-Q3" \
+  --auditor "VampSecure Labs" \
+  --output results.json \
+  --report-html report.html \
+  --verbose
+```
+
+### Activate dataset red team (Phase 6)
+
+```bash
+python3 vamp_llm_probe.py \
+  --endpoint http://localhost:11434 \
+  --dataset \
+  --dataset-sample 30
+```
+
+### Dataset with category filter (forbidden questions)
+
+```bash
+python3 vamp_llm_probe.py \
+  --endpoint http://localhost:11434 \
+  --dataset \
+  --dataset-sample 20 \
+  --dataset-categories "Malware,Illegal Activity,Physical Harm"
+```
+
+### Skip jailbreak phase (restricted environments)
+
+```bash
+python3 vamp_llm_probe.py \
+  --endpoint http://localhost:11434 \
+  --no-jailbreak
+```
+
+---
+
+## CLI Arguments
+
+| Argument | Description | Default |
+|---|---|---|
+| `--endpoint URL` | Base URL of the endpoint to audit (required) | — |
+| `--api-key KEY` | Bearer authorization key | (none) |
+| `--model MODELO` | Model name for inference tests | auto-detect |
+| `--timeout N` | Request timeout in seconds | 30 |
+| `--output FILE` | Save results as JSON to FILE | (none) |
+| `--report-html FILE` | Generate professional HTML report | (none) |
+| `--client NOMBRE` | Client name for the report cover | Confidencial |
+| `--engagement DESC` | Engagement description | (none) |
+| `--auditor NOMBRE` | Auditor name for the report | VampSecure Labs |
+| `--no-jailbreak` | Skip Phase 3 (restriction bypass) | false |
+| `--verbose` | Verbose mode — shows HTTP traces | false |
+| `--dataset` | Activate Phase 6: adversarial dataset red team | false |
+| `--dataset-sample N` | Prompts per dataset type to test | 15 |
+| `--dataset-categories CATS` | Comma-separated forbidden question categories to test | all |
+
+**Available forbidden question categories:** `Illegal Activity`, `Hate Speech`, `Malware`, `Physical Harm`, `Economic Harm`, `Fraud`, `Pornography`, `Political Lobbying`, `Privacy Violence`, `Legal Opinion`, `Financial Advice`, `Health Consultation`, `Gov Decision`
+
+---
+
+## Audit Phases & Findings
+
+| Phase | Name | Findings | Severity |
+|---|---|---|---|
+| 1 | Endpoint Reconnaissance | LLM-001..009 | CRITICAL to INFO |
+| 2 | Prompt Injection | LLM-010..029 | CRITICAL to HIGH |
+| 3 | Restriction Bypass Attempts | LLM-030..049 | HIGH to MEDIUM |
+| 4 | Data Extraction & Leaks | LLM-050..069 | CRITICAL to MEDIUM |
+| 5 | Access Controls & Behavior | LLM-070..089 | HIGH to LOW |
+| 6 | Adversarial Dataset Red Team | LLM-100..139 | HIGH |
+
+### Phase 1 — Endpoint Reconnaissance
+
+| Finding | Title | Severity |
+|---|---|---|
+| LLM-001 | Endpoint exposes model list without authentication | CRITICAL |
+| LLM-002 | Web management interface publicly accessible | MEDIUM |
+| LLM-003 | Server version exposed in headers or response | INFO |
+| LLM-004 | Multiple administrative routes accessible | MEDIUM |
+| LLM-005 | Inference endpoint accessible without authentication | CRITICAL |
+
+### Phase 2 — Prompt Injection
+
+Tests 10 crafted prompt injection payloads including direct overrides, role substitution, JSON format overrides, indirect HTML injection, multilingual overrides, zero-width space evasion, developer-mode unlocking, and token-separator injection.
+
+### Phase 3 — Restriction Bypass Attempts
+
+Tests 8 bypass techniques: Base64-encoded instructions, unrestricted roleplay, query fragmentation, emoji/token obfuscation, language-switch overrides (English, French), and continuation-text technique.
+
+### Phase 4 — Data Extraction & Leaks
+
+| Finding | Title | Severity |
+|---|---|---|
+| LLM-050..052 | System prompt extraction attempts | HIGH |
+| LLM-053 | Filesystem listing request | HIGH |
+| LLM-054 | API key / credential extraction | CRITICAL |
+| LLM-055 | Network configuration extraction | HIGH |
+| LLM-056 | SSRF vector via prompt injection | CRITICAL |
+| LLM-057 | Environment variable extraction | HIGH |
+
+### Phase 5 — Access Controls & Behavior
+
+| Finding | Title | Severity |
+|---|---|---|
+| LLM-070 | No rate limiting on inference endpoint | HIGH |
+| LLM-071 | Context overflow accepted without limits | MEDIUM |
+| LLM-072 | Poor handling of non-existent models | MEDIUM |
+| LLM-073 | Malformed JSON does not return HTTP 400 | LOW |
+| LLM-074 | Permissive CORS on inference endpoint | MEDIUM |
+| LLM-075 | Missing HTTP security headers | LOW |
+
+### Phase 6 — Adversarial Dataset Red Team (`--dataset`)
+
+Uses five bundled datasets — three in English (from [TrustAI-laboratory/Learn-Prompt-Hacking](https://github.com/TrustAI-laboratory/Learn-Prompt-Hacking)) and two curated in Spanish by VampSecure Labs — plus a dynamic ASCII smuggling generator:
+
+| Subtest | Dataset | Size | Language | Detection Method |
+|---|---|---|---|---|
+| A | `injection_prompts.csv` | 210 real injection vectors | EN | Model responds substantively without refusal to task-redirect prompts |
+| B | `jailbreak_prompts.csv` | 666 real jailbreak techniques | EN | Model response contains compliance markers without refusal patterns |
+| C | `forbidden_questions.csv` | 390 questions across 13 policy categories | EN | Model provides >150-char substantive answer without refusal |
+| A_es | `injection_prompts_es.csv` | 50 curated injection vectors | ES | Same bilingual heuristic — Spanish refusal phrases also checked |
+| B_es | `jailbreak_prompts_es.csv` | 30 jailbreak prompts | ES | Spanish compliance markers (NEXUS/MÁXIMO/ALFA/SOMBRA/modo sin filtros…) detected |
+| D | `_generar_payloads_ascii_smuggling()` | 30 generated payloads | EN | Model executes hidden instructions encoded in Unicode Tags (U+E0000-U+E007F) |
+
+The detection engine is **language-aware**: `_REFUSAL_PATTERNS` covers 30+ English and Spanish refusal phrases; `_COMPLIANCE_MARKERS` covers 30+ English and Spanish compliance indicators. A Spanish model refusing in Spanish counts as a refusal; a Spanish model adopting an unrestricted persona in Spanish counts as a jailbreak success.
+
+Findings LLM-100+ are generated dynamically. Each finding includes the exact prompt and model response as evidence.
+
+### ASCII Smuggling Vector
+
+ASCII smuggling exploits the **Unicode Tags block (U+E0000–U+E007F)** — an invisible copy of printable ASCII. These characters are not rendered on screen but are processed by LLMs, allowing hidden instructions to be embedded in content that appears clean to a human reviewer.
+
+Microsoft published an analysis on 3 Sep 2026 showing the technique is actively used in phishing to evade email security filters: [ASCII Smuggling Crosses Over from AI Prompt Injection to Phishing Evasion](https://www.microsoft.com/en-us/security/blog/2026/09/03/ascii-smuggling-crosses-over-from-ai-prompt-injection-to-phishing-evasion/).
+
+**vamp-llm-probe detects this vector in two ways:**
+
+1. **Active (Subtest D)** — sends 30 payloads where innocent-looking visible text contains Unicode Tags–encoded jailbreak instructions. A CRITICAL finding is raised if the model executes the hidden instruction.
+2. **Passive (Phase 2)** — every response from the inference endpoint is scanned for Unicode Tags characters. A HIGH finding is raised if the endpoint itself returns invisible characters (which could inject hidden instructions into downstream clients).
+
+Legitimate exceptions — the English, Scottish, and Welsh flag emoji — are excluded from detection (they encode their subdivision tags using this same Unicode block).
+
+---
+
+## OWASP Mapping
+
+Every finding produced by vamp-llm-probe is automatically tagged with the corresponding OWASP categories before the report is generated. Tags appear in the JSON output (`finding.tags`) and as blue badges in the HTML report.
+
+### OWASP LLM Top 10 — 2025
+
+| Tag | Category |
+|---|---|
+| `OWASP-LLM01` | Prompt Injection |
+| `OWASP-LLM02` | Sensitive Information Disclosure |
+| `OWASP-LLM05` | Improper Output Handling |
+| `OWASP-LLM06` | Excessive Agency |
+| `OWASP-LLM07` | System Prompt Leakage |
+| `OWASP-LLM10` | Unbounded Consumption |
+
+### OWASP Agentic AI Top 10 — 2026
+
+| Tag | Category |
+|---|---|
+| `OWASP-AGENT04` | Context Manipulation |
+| `OWASP-AGENT06` | Intent Breaking & Goal Hijacking |
+| `OWASP-AGENT07` | Data Exfiltration via Agents |
+| `OWASP-AGENT09` | Resource Overuse |
+
+### Finding-to-OWASP mapping
+
+| Finding range | Phase | OWASP tags |
+|---|---|---|
+| LLM-001..009 | Endpoint Reconnaissance | `LLM06` (+ `LLM02` if LLM-003) |
+| LLM-010..029 | Prompt Injection + passive ASCII scan | `LLM01` `AGENT04` `AGENT06` |
+| LLM-030..049 | Restriction Bypass / Jailbreak | `LLM01` `AGENT06` |
+| LLM-050..069 | Data Extraction & Leaks | `LLM02` `LLM07` `AGENT07` |
+| LLM-070 | Rate limiting absent | `LLM10` `AGENT09` |
+| LLM-071..073 | Output handling issues | `LLM05` |
+| LLM-074..089 | CORS / security headers | `LLM06` |
+| LLM-100..199 | Adversarial dataset red team | `LLM01` `AGENT06` |
+| LLM-ASCII-* | ASCII smuggling active (Subtest D) | `LLM01` `AGENT04` |
+
+---
+
+## Bundled Datasets
+
+```
+vamp-llm-probe/payloads/
+├── jailbreak_prompts.csv           # 666 real jailbreaks EN (verazuo/jailbreak_llms)
+├── injection_prompts.csv           # 210 injection prompts EN (TrustAI curated)
+├── forbidden_questions.csv         # 390 questions × 13 policy categories (TrustAI)
+├── injection_prompts_es.csv        # 50 injection vectors ES (VSL curated)
+├── jailbreak_prompts_es.csv        # 30 jailbreak prompts ES (VSL curated)
+└── ascii_smuggling_payloads.json   # Source instructions for ASCII smuggling Subtest D (VSL)
+```
+
+All datasets are offline and self-contained. No external requests are made at runtime. The English datasets are sourced from TrustAI-laboratory/Learn-Prompt-Hacking; the Spanish datasets were curated by VampSecure Labs to cover native Spanish-language attack vectors not present in the original corpus.
+
+---
+
+## Exit Codes
+
+| Code | Meaning |
+|---|---|
+| `0` | No critical findings (MEDIUM, LOW, or INFO only) |
+| `1` | HIGH severity findings detected |
+| `2` | CRITICAL severity findings detected |
+
+Use these codes in CI/CD pipelines to gate deployments:
+
+```bash
+python3 vamp_llm_probe.py --endpoint "$ENDPOINT" --dataset || {
+  echo "Security findings detected — blocking deployment"
+  exit 1
+}
+```
+
+---
+
+## Output Formats
+
+### JSON (`--output results.json`)
+
+Machine-readable structured output following the VSL standard schema:
+
+```json
+{
+  "schema_version": "1.0",
+  "generated": "2026-08-12 12:00 UTC",
+  "meta": { "tool": "vamp-llm-probe", "tool_version": "1.3.0", ... },
+  "summary": { "total": 5, "by_severity": { "CRITICAL": 2, "HIGH": 1, ... } },
+  "findings": [ { "id": "LLM-001", "severity": "CRITICAL", ... } ]
+}
+```
+
+### HTML (`--report-html report.html`)
+
+Professional client-delivery report with:
+- Cover page with engagement details
+- Executive summary with risk distribution chart
+- Findings table with severity color coding
+- Detailed finding cards with evidence and remediation
+
+---
+
+## Project Structure
+
+```
+vamp-llm-probe/
+├── vamp_llm_probe.py    # Main auditor (6 phases, bilingual detection, ASCII smuggling)
+├── vampsec_report.py    # Unified reporting module (VSL shared)
+├── payloads/            # Adversarial datasets (Phase 6)
+│   ├── jailbreak_prompts.csv           # EN — 666 jailbreaks
+│   ├── injection_prompts.csv           # EN — 210 injection vectors
+│   ├── forbidden_questions.csv         # EN — 390 forbidden questions
+│   ├── injection_prompts_es.csv        # ES — 50 injection vectors (VSL)
+│   ├── jailbreak_prompts_es.csv        # ES — 30 jailbreak prompts (VSL)
+│   └── ascii_smuggling_payloads.json   # ASCII smuggling source instructions (VSL)
+├── requirements.txt
+├── .gitignore
+└── README.md
+```
+
+---
+
+## License
+
+MIT License — see individual file headers for copyright details.
+
+---
+
+© VampSecure Studios — VampSecure Labs Security Research Division  
+Authorized use only in environments with explicit written permission.