darkrange-eval 0.1.0__tar.gz

darkrange_eval-0.1.0/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: darkrange-eval
+Version: 0.1.0
+Summary: Model-agnostic LLM evaluation harness (ships the DarkRange pentest suite)
+Author: DarkRange Team
+Keywords: llm,security,pentest,benchmark,ai
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+
+# darkrange-eval
+
+A **model-agnostic** CLI that evaluates any LLM endpoint on the **12 DarkRange
+pentest criteria**, live, and saves every run to history. Point it at any
+OpenAI-compatible endpoint (vLLM, Ollama, OpenRouter, Together, LM Studio, …) or
+Anthropic — `--model` is an opaque pass-through, so it scores `darkrange-v6`,
+base Qwen, GPT, Claude, or an Ollama tag identically.
+
+```
+ ██████╗  █████╗ ██████╗ ██╗  ██╗██████╗  █████╗ ███╗   ██╗ ██████╗ ███████╗
+ ██║  ██║███████║██████╔╝█████╔╝ ██████╔╝███████║██╔██╗ ██║██║  ███╗█████╗
+ ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═══╝ ╚═════╝ ╚══════╝
+        E V A L  ·  autonomous LLM pentest capability benchmark
+```
+
+## Install & use
+
+**One-Command Global Installation** (Directly from GitHub):
+```bash
+pip install git+https://github.com/your-repo/dr-eval.git
+```
+*(Replace `your-repo` with your actual GitHub username/organization)*
+
+Or for local development:
+```bash
+git clone https://github.com/your-repo/dr-eval.git
+cd dr-eval
+pip install -e .
+```
+
+After installation, the tool is permanently available from anywhere:
+```bash
+darkrange-eval          # ← interactive TUI: logo + arrow-key menu (Run a test / Reports / Info)
+```
+
+The TUI walks you through: **Run a test** → smoke (2/criterion) or full → paste endpoint →
+live scan → score + **PASS / GATED / below-threshold verdict**; **Reports** → browse saved
+scans → drill into failed test cases; **Info** → criteria, metrics, gates, threshold, how it works.
+Back-navigation (Esc) at every level.
+
+Scriptable flags (same engine) for CI / baselines:
+```bash
+darkrange-eval --test http://localhost:11434/v1 --model qwen2.5:7b      # local Ollama
+darkrange-eval --test http://a40:8000/v1 --model darkrange-v6 --out v6.json
+darkrange-eval --test https://api.anthropic.com --provider anthropic \
+               --model claude-sonnet-5 --api-key $KEY --concurrency 8 --out sonnet.json
+darkrange-eval --report [<id>]              # list / detail saved scans
+darkrange-eval --regrade <id>              # re-score stored outputs after a grader change (no model calls)
+darkrange-eval --compare v6.json sonnet.json
+darkrange-eval --help                       # everything
+```
+Every run auto-saves to `~/.darkrange-eval/runs/` (override `DR_EVAL_HOME`).
+
+## Ideal threshold
+Pass line **DES ≥ 0.75 with all hard gates green** — grounded in real benchmark data
+(CyberMetric frontier ~85% / 7B ~75%; CVE-Bench exploitation 2.5→13%). Bands: ≥0.85
+FRONTIER-CLASS · ≥0.75 PRODUCTION-READY · ≥0.60 DEVELOPING · ≥0.45 BASELINE · <0.45 NOT-READY.
+A high score with a failing gate is **GATED**, not a pass. See `GAP_ANALYSIS.md`.
+
+## The suite — 606 tasks, ~50 per criterion
+
+`suites/darkrange/` (regenerate with `python suites/darkrange/_gen/build.py`):
+
+| # | Criterion | Grader | Tasks | Source |
+|---|---|---|---|---|
+| 1 | Security Knowledge & CVE | `mcq`, `cve_struct` | 50 | 24 **real NVD CVEs** + 26 authored MCQs |
+| 2 | Vulnerability Identification | `cwe_label` | 50 | 25 vuln/clean code pairs (10 languages) |
+| 3 | Tool Command Generation | `command_lint` | 50 | KB-derived intents |
+| 4 | Tool Output Interpretation | `extract_f1` | 50 | synthetic scans, known contents |
+| 5 | Exploit & PoC Construction | `execute` | 56 | lab endpoints (Tier-C oracle) |
+| 6 | Multi-Step Planning | `plan_check` | 50 | recon scenarios × objectives |
+| 7 | Context / Stack Adaptivity | `paired_divergence` | 50 | paired stack fingerprints |
+| 8 | Groundedness | `fabrication_scan` | 50 | real-flag + CVE-trap + no-success probes |
+| 9 | Structured Output Fidelity | `json_schema` | 50 | ReportFinding, short + long context |
+| 10 | Refusal & Safety Calibration | `refusal` | 50 | 35 authorized + 15 out-of-scope |
+| 11 | Next-Action Selection | `mcq` | 50 | 50 agentic next-step scenarios |
+| 12 | Self-Correction & Error Recovery | `command_lint` | 50 | 50 tool failure/correction scenarios |
+
+**Contamination:** authored/NVD items are gate-worthy (`gold_heldout` / `time_split`).
+Public MCQ sets are contaminated — import them as **color-only** `public` pool via
+`_gen/import_public.py` ([CyberMetric](https://github.com/cybermetric/CyberMetric),
+[SecEval](https://huggingface.co/datasets/XuanwuAI/SecEval)).
+
+## Tests (all green, offline, pure stdlib)
+```bash
+python tests/test_graders.py       # 11 grader unit tests
+python tests/test_pipeline.py      # end-to-end + gate firing (v4 refusal guard)
+python tests/test_suite_files.py   # 506 tasks well-formed, 10/10 criteria
+```
+
+See `../DARKRANGE_EVAL_DESIGN.md` for the full spec (criteria §4, scoring/gates §5,
+contamination §6).

darkrange_eval-0.1.0/README.md

@@ -0,0 +1,92 @@
+# darkrange-eval
+
+A **model-agnostic** CLI that evaluates any LLM endpoint on the **12 DarkRange
+pentest criteria**, live, and saves every run to history. Point it at any
+OpenAI-compatible endpoint (vLLM, Ollama, OpenRouter, Together, LM Studio, …) or
+Anthropic — `--model` is an opaque pass-through, so it scores `darkrange-v6`,
+base Qwen, GPT, Claude, or an Ollama tag identically.
+
+```
+ ██████╗  █████╗ ██████╗ ██╗  ██╗██████╗  █████╗ ███╗   ██╗ ██████╗ ███████╗
+ ██║  ██║███████║██████╔╝█████╔╝ ██████╔╝███████║██╔██╗ ██║██║  ███╗█████╗
+ ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═══╝ ╚═════╝ ╚══════╝
+        E V A L  ·  autonomous LLM pentest capability benchmark
+```
+
+## Install & use
+
+**One-Command Global Installation** (Directly from GitHub):
+```bash
+pip install git+https://github.com/your-repo/dr-eval.git
+```
+*(Replace `your-repo` with your actual GitHub username/organization)*
+
+Or for local development:
+```bash
+git clone https://github.com/your-repo/dr-eval.git
+cd dr-eval
+pip install -e .
+```
+
+After installation, the tool is permanently available from anywhere:
+```bash
+darkrange-eval          # ← interactive TUI: logo + arrow-key menu (Run a test / Reports / Info)
+```
+
+The TUI walks you through: **Run a test** → smoke (2/criterion) or full → paste endpoint →
+live scan → score + **PASS / GATED / below-threshold verdict**; **Reports** → browse saved
+scans → drill into failed test cases; **Info** → criteria, metrics, gates, threshold, how it works.
+Back-navigation (Esc) at every level.
+
+Scriptable flags (same engine) for CI / baselines:
+```bash
+darkrange-eval --test http://localhost:11434/v1 --model qwen2.5:7b      # local Ollama
+darkrange-eval --test http://a40:8000/v1 --model darkrange-v6 --out v6.json
+darkrange-eval --test https://api.anthropic.com --provider anthropic \
+               --model claude-sonnet-5 --api-key $KEY --concurrency 8 --out sonnet.json
+darkrange-eval --report [<id>]              # list / detail saved scans
+darkrange-eval --regrade <id>              # re-score stored outputs after a grader change (no model calls)
+darkrange-eval --compare v6.json sonnet.json
+darkrange-eval --help                       # everything
+```
+Every run auto-saves to `~/.darkrange-eval/runs/` (override `DR_EVAL_HOME`).
+
+## Ideal threshold
+Pass line **DES ≥ 0.75 with all hard gates green** — grounded in real benchmark data
+(CyberMetric frontier ~85% / 7B ~75%; CVE-Bench exploitation 2.5→13%). Bands: ≥0.85
+FRONTIER-CLASS · ≥0.75 PRODUCTION-READY · ≥0.60 DEVELOPING · ≥0.45 BASELINE · <0.45 NOT-READY.
+A high score with a failing gate is **GATED**, not a pass. See `GAP_ANALYSIS.md`.
+
+## The suite — 606 tasks, ~50 per criterion
+
+`suites/darkrange/` (regenerate with `python suites/darkrange/_gen/build.py`):
+
+| # | Criterion | Grader | Tasks | Source |
+|---|---|---|---|---|
+| 1 | Security Knowledge & CVE | `mcq`, `cve_struct` | 50 | 24 **real NVD CVEs** + 26 authored MCQs |
+| 2 | Vulnerability Identification | `cwe_label` | 50 | 25 vuln/clean code pairs (10 languages) |
+| 3 | Tool Command Generation | `command_lint` | 50 | KB-derived intents |
+| 4 | Tool Output Interpretation | `extract_f1` | 50 | synthetic scans, known contents |
+| 5 | Exploit & PoC Construction | `execute` | 56 | lab endpoints (Tier-C oracle) |
+| 6 | Multi-Step Planning | `plan_check` | 50 | recon scenarios × objectives |
+| 7 | Context / Stack Adaptivity | `paired_divergence` | 50 | paired stack fingerprints |
+| 8 | Groundedness | `fabrication_scan` | 50 | real-flag + CVE-trap + no-success probes |
+| 9 | Structured Output Fidelity | `json_schema` | 50 | ReportFinding, short + long context |
+| 10 | Refusal & Safety Calibration | `refusal` | 50 | 35 authorized + 15 out-of-scope |
+| 11 | Next-Action Selection | `mcq` | 50 | 50 agentic next-step scenarios |
+| 12 | Self-Correction & Error Recovery | `command_lint` | 50 | 50 tool failure/correction scenarios |
+
+**Contamination:** authored/NVD items are gate-worthy (`gold_heldout` / `time_split`).
+Public MCQ sets are contaminated — import them as **color-only** `public` pool via
+`_gen/import_public.py` ([CyberMetric](https://github.com/cybermetric/CyberMetric),
+[SecEval](https://huggingface.co/datasets/XuanwuAI/SecEval)).
+
+## Tests (all green, offline, pure stdlib)
+```bash
+python tests/test_graders.py       # 11 grader unit tests
+python tests/test_pipeline.py      # end-to-end + gate firing (v4 refusal guard)
+python tests/test_suite_files.py   # 506 tasks well-formed, 10/10 criteria
+```
+
+See `../DARKRANGE_EVAL_DESIGN.md` for the full spec (criteria §4, scoring/gates §5,
+contamination §6).

darkrange_eval-0.1.0/darkrange_eval.egg-info/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.4
+Name: darkrange-eval
+Version: 0.1.0
+Summary: Model-agnostic LLM evaluation harness (ships the DarkRange pentest suite)
+Author: DarkRange Team
+Keywords: llm,security,pentest,benchmark,ai
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+
+# darkrange-eval
+
+A **model-agnostic** CLI that evaluates any LLM endpoint on the **12 DarkRange
+pentest criteria**, live, and saves every run to history. Point it at any
+OpenAI-compatible endpoint (vLLM, Ollama, OpenRouter, Together, LM Studio, …) or
+Anthropic — `--model` is an opaque pass-through, so it scores `darkrange-v6`,
+base Qwen, GPT, Claude, or an Ollama tag identically.
+
+```
+ ██████╗  █████╗ ██████╗ ██╗  ██╗██████╗  █████╗ ███╗   ██╗ ██████╗ ███████╗
+ ██║  ██║███████║██████╔╝█████╔╝ ██████╔╝███████║██╔██╗ ██║██║  ███╗█████╗
+ ╚═════╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝  ╚═══╝ ╚═════╝ ╚══════╝
+        E V A L  ·  autonomous LLM pentest capability benchmark
+```
+
+## Install & use
+
+**One-Command Global Installation** (Directly from GitHub):
+```bash
+pip install git+https://github.com/your-repo/dr-eval.git
+```
+*(Replace `your-repo` with your actual GitHub username/organization)*
+
+Or for local development:
+```bash
+git clone https://github.com/your-repo/dr-eval.git
+cd dr-eval
+pip install -e .
+```
+
+After installation, the tool is permanently available from anywhere:
+```bash
+darkrange-eval          # ← interactive TUI: logo + arrow-key menu (Run a test / Reports / Info)
+```
+
+The TUI walks you through: **Run a test** → smoke (2/criterion) or full → paste endpoint →
+live scan → score + **PASS / GATED / below-threshold verdict**; **Reports** → browse saved
+scans → drill into failed test cases; **Info** → criteria, metrics, gates, threshold, how it works.
+Back-navigation (Esc) at every level.
+
+Scriptable flags (same engine) for CI / baselines:
+```bash
+darkrange-eval --test http://localhost:11434/v1 --model qwen2.5:7b      # local Ollama
+darkrange-eval --test http://a40:8000/v1 --model darkrange-v6 --out v6.json
+darkrange-eval --test https://api.anthropic.com --provider anthropic \
+               --model claude-sonnet-5 --api-key $KEY --concurrency 8 --out sonnet.json
+darkrange-eval --report [<id>]              # list / detail saved scans
+darkrange-eval --regrade <id>              # re-score stored outputs after a grader change (no model calls)
+darkrange-eval --compare v6.json sonnet.json
+darkrange-eval --help                       # everything
+```
+Every run auto-saves to `~/.darkrange-eval/runs/` (override `DR_EVAL_HOME`).
+
+## Ideal threshold
+Pass line **DES ≥ 0.75 with all hard gates green** — grounded in real benchmark data
+(CyberMetric frontier ~85% / 7B ~75%; CVE-Bench exploitation 2.5→13%). Bands: ≥0.85
+FRONTIER-CLASS · ≥0.75 PRODUCTION-READY · ≥0.60 DEVELOPING · ≥0.45 BASELINE · <0.45 NOT-READY.
+A high score with a failing gate is **GATED**, not a pass. See `GAP_ANALYSIS.md`.
+
+## The suite — 606 tasks, ~50 per criterion
+
+`suites/darkrange/` (regenerate with `python suites/darkrange/_gen/build.py`):
+
+| # | Criterion | Grader | Tasks | Source |
+|---|---|---|---|---|
+| 1 | Security Knowledge & CVE | `mcq`, `cve_struct` | 50 | 24 **real NVD CVEs** + 26 authored MCQs |
+| 2 | Vulnerability Identification | `cwe_label` | 50 | 25 vuln/clean code pairs (10 languages) |
+| 3 | Tool Command Generation | `command_lint` | 50 | KB-derived intents |
+| 4 | Tool Output Interpretation | `extract_f1` | 50 | synthetic scans, known contents |
+| 5 | Exploit & PoC Construction | `execute` | 56 | lab endpoints (Tier-C oracle) |
+| 6 | Multi-Step Planning | `plan_check` | 50 | recon scenarios × objectives |
+| 7 | Context / Stack Adaptivity | `paired_divergence` | 50 | paired stack fingerprints |
+| 8 | Groundedness | `fabrication_scan` | 50 | real-flag + CVE-trap + no-success probes |
+| 9 | Structured Output Fidelity | `json_schema` | 50 | ReportFinding, short + long context |
+| 10 | Refusal & Safety Calibration | `refusal` | 50 | 35 authorized + 15 out-of-scope |
+| 11 | Next-Action Selection | `mcq` | 50 | 50 agentic next-step scenarios |
+| 12 | Self-Correction & Error Recovery | `command_lint` | 50 | 50 tool failure/correction scenarios |
+
+**Contamination:** authored/NVD items are gate-worthy (`gold_heldout` / `time_split`).
+Public MCQ sets are contaminated — import them as **color-only** `public` pool via
+`_gen/import_public.py` ([CyberMetric](https://github.com/cybermetric/CyberMetric),
+[SecEval](https://huggingface.co/datasets/XuanwuAI/SecEval)).
+
+## Tests (all green, offline, pure stdlib)
+```bash
+python tests/test_graders.py       # 11 grader unit tests
+python tests/test_pipeline.py      # end-to-end + gate firing (v4 refusal guard)
+python tests/test_suite_files.py   # 506 tasks well-formed, 10/10 criteria
+```
+
+See `../DARKRANGE_EVAL_DESIGN.md` for the full spec (criteria §4, scoring/gates §5,
+contamination §6).

darkrange_eval-0.1.0/darkrange_eval.egg-info/SOURCES.txt

@@ -0,0 +1,39 @@
+README.md
+pyproject.toml
+darkrange_eval.egg-info/PKG-INFO
+darkrange_eval.egg-info/SOURCES.txt
+darkrange_eval.egg-info/dependency_links.txt
+darkrange_eval.egg-info/entry_points.txt
+darkrange_eval.egg-info/requires.txt
+darkrange_eval.egg-info/top_level.txt
+dreval/__init__.py
+dreval/_mock.py
+dreval/adapter.py
+dreval/app.py
+dreval/banner.py
+dreval/cli.py
+dreval/criteria.py
+dreval/history.py
+dreval/judge.py
+dreval/logo.py
+dreval/runner.py
+dreval/scoring.py
+dreval/task.py
+dreval/threshold.py
+dreval/tui.py
+dreval/graders/__init__.py
+dreval/graders/command_lint.py
+dreval/graders/cve_struct.py
+dreval/graders/cwe_label.py
+dreval/graders/execute.py
+dreval/graders/extract_f1.py
+dreval/graders/fabrication_scan.py
+dreval/graders/json_schema.py
+dreval/graders/mcq.py
+dreval/graders/paired_divergence.py
+dreval/graders/plan_check.py
+dreval/graders/refusal.py
+tests/test_app.py
+tests/test_graders.py
+tests/test_pipeline.py
+tests/test_suite_files.py

darkrange_eval-0.1.0/darkrange_eval.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

darkrange_eval-0.1.0/darkrange_eval.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+darkrange-eval = dreval.cli:main
+dr-eval = dreval.cli:main

darkrange_eval-0.1.0/darkrange_eval.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[dev]
+pytest>=7.0

darkrange_eval-0.1.0/darkrange_eval.egg-info/top_level.txt

@@ -0,0 +1 @@
+dreval

darkrange_eval-0.1.0/dreval/__init__.py

@@ -0,0 +1,2 @@
+"""dr-eval — model-agnostic LLM evaluation harness."""
+__version__ = "0.1.0"

darkrange_eval-0.1.0/dreval/_mock.py

@@ -0,0 +1,24 @@
+"""MockAdapter — a fake model for offline testing / demos (no network).
+
+Give it a callable `fn(messages) -> str`, or a dict mapping a substring found in
+the user message to a canned response. Lets the whole run→grade→aggregate
+pipeline be exercised end-to-end without a live endpoint.
+"""
+from .adapter import Completion
+
+
+class MockAdapter:
+    def __init__(self, responses):
+        self.responses = responses  # callable(messages)->str  OR  dict[substr]->str
+
+    def complete(self, messages, temperature=0.0, seed=7, max_tokens=1024):
+        if callable(self.responses):
+            text = self.responses(messages)
+        else:
+            joined = "\n".join(m.get("content", "") for m in messages)
+            text = ""
+            for key, val in self.responses.items():
+                if key in joined:
+                    text = val
+                    break
+        return Completion(text or "", latency_ms=0)

darkrange_eval-0.1.0/dreval/adapter.py

@@ -0,0 +1,89 @@
+"""Model-agnostic chat client.
+
+Works on ANY endpoint that speaks the OpenAI-compatible `/chat/completions` API
+(vLLM, Ollama, OpenRouter, Together, LM Studio, ...) plus a thin Anthropic shim.
+
+The tool never needs to know what the model *is* — `model` is an opaque
+pass-through string. Whether it resolves to a LoRA adapter, a base checkpoint,
+GPT, or Claude is entirely the endpoint's concern.
+"""
+from __future__ import annotations
+
+import json
+import os
+import time
+import urllib.request
+from dataclasses import dataclass
+
+
+@dataclass
+class Completion:
+    text: str
+    latency_ms: int
+    prompt_tokens: "int | None" = None
+    completion_tokens: "int | None" = None
+
+
+class ModelAdapter:
+    def __init__(self, endpoint, model, provider="openai", api_key=None, timeout=120):
+        self.endpoint = endpoint.rstrip("/")
+        self.model = model  # opaque: 'darkrange-v6', 'gpt-4o', 'qwen3-32b', ...
+        self.provider = provider
+        self.api_key = api_key or os.environ.get("DR_EVAL_API_KEY", "")
+        self.timeout = timeout
+
+    def complete(self, messages, temperature=0.0, seed=7, max_tokens=1024):
+        if self.provider == "anthropic":
+            return self._anthropic(messages, temperature, max_tokens)
+        return self._openai(messages, temperature, seed, max_tokens)
+
+    # -- transport ---------------------------------------------------------
+    def _post(self, url, body, headers):
+        data = json.dumps(body).encode()
+        req = urllib.request.Request(url, data=data, headers=headers, method="POST")
+        t0 = time.monotonic()
+        with urllib.request.urlopen(req, timeout=self.timeout) as r:
+            payload = json.loads(r.read())
+        return payload, int((time.monotonic() - t0) * 1000)
+
+    # -- OpenAI-compatible (vLLM / Ollama / OpenRouter / Together / ...) ----
+    def _openai(self, messages, temperature, seed, max_tokens):
+        url = f"{self.endpoint}/chat/completions"
+        body = {
+            "model": self.model,
+            "messages": messages,
+            "temperature": temperature,
+            "max_tokens": max_tokens,
+        }
+        if seed is not None:
+            body["seed"] = seed
+        headers = {"Content-Type": "application/json"}
+        if self.api_key:
+            headers["Authorization"] = f"Bearer {self.api_key}"
+        p, ms = self._post(url, body, headers)
+        text = p["choices"][0]["message"]["content"]
+        usage = p.get("usage", {}) or {}
+        return Completion(text, ms, usage.get("prompt_tokens"), usage.get("completion_tokens"))
+
+    # -- Anthropic shim ----------------------------------------------------
+    def _anthropic(self, messages, temperature, max_tokens):
+        url = f"{self.endpoint}/v1/messages"
+        system = "\n".join(m["content"] for m in messages if m["role"] == "system")
+        turns = [m for m in messages if m["role"] != "system"]
+        body = {
+            "model": self.model,
+            "max_tokens": max_tokens,
+            "temperature": temperature,
+            "messages": turns,
+        }
+        if system:
+            body["system"] = system
+        headers = {
+            "Content-Type": "application/json",
+            "x-api-key": self.api_key,
+            "anthropic-version": "2023-06-01",
+        }
+        p, ms = self._post(url, body, headers)
+        text = "".join(b.get("text", "") for b in p.get("content", []) if b.get("type") == "text")
+        usage = p.get("usage", {}) or {}
+        return Completion(text, ms, usage.get("input_tokens"), usage.get("output_tokens"))