securevibebench 0.2.0__tar.gz

securevibebench-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Junkai Chen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

securevibebench-0.2.0/MANIFEST.in

@@ -0,0 +1,11 @@
+include LICENSE THIRD_PARTY_NOTICES.md README.md uv.lock
+include build_backend.py
+recursive-include evaluation *.py *.yaml *.sh *.md
+include evaluation/resources.json evaluation/mini-source.tar.gz
+exclude evaluation/resources.tar.gz
+recursive-include tools *.py
+recursive-include docs *.md
+recursive-include tests *.py
+prune evaluation/mini-swe-agent
+prune dev
+global-exclude __pycache__ *.py[cod] .env .env.*

securevibebench-0.2.0/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.4
+Name: securevibebench
+Version: 0.2.0
+Summary: Generate and evaluate secure coding agents on SecureVibeBench
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/iCSawyer/SecureVibeBench
+Project-URL: Documentation, https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md
+Project-URL: Issues, https://github.com/iCSawyer/SecureVibeBench/issues
+Project-URL: Dataset, https://huggingface.co/datasets/iCSawyer/SecureVibeBench
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: THIRD_PARTY_NOTICES.md
+Requires-Dist: harbor<0.26.0,>=0.25.0
+Requires-Dist: mini-swe-agent==2.4.6
+Requires-Dist: huggingface-hub>=0.34.3
+Requires-Dist: docker>=7.1.0
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: python-dotenv>=1.1.1
+Dynamic: license-file
+
+# SecureVibeBench
+
+<p align="left">
+    <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2">
+    <a href="https://2026.aclweb.org/"><img src="https://img.shields.io/badge/Venue-ACL%202026-2ea44f.svg?style=for-the-badge" alt="Venue: ACL 2026"></a>
+    <a href="https://arxiv.org/abs/2509.22097"><img src="https://img.shields.io/badge/arXiv-2509.22097-b31b1b.svg?style=for-the-badge" alt="arXiv"></a>
+    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a>
+    <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a>
+    <!-- <a href="https://chen-junkai.github.io/securevibebench-leaderboard/"><img src="https://img.shields.io/badge/Leaderboard-Online-2ea44f.svg?style=for-the-badge" alt="Leaderboard"></a> -->
+</p>
+
+> [!IMPORTANT]
+> **SecureVibeBench is now v0.2!** Try `pip install securevibebench`!
+>
+> We refer to the original release and evaluation workflow as **v0.1**, and **results obtained with v0.2 may differ from v0.1.** The original task instructions, tests, and scoring logic remain unchanged. Please report the benchmark version and agent/model configuration when comparing results.
+
+<p align="left">
+    ✨&nbsp;<a href="#nav-news">News</a>
+    | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
+    | 🔭&nbsp;<a href="#nav-overview">Overview</a>
+    | 📚&nbsp;<a href="#nav-citation">Citation</a>
+    | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
+</p>
+
+<a id="nav-news"></a>
+
+## ✨ News
+
+- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as **v0.1**; results may differ between versions.
+<!-- - **[2026-06-06]** 🏆 We released the online [Leaderboard](https://chen-junkai.github.io/securevibebench-leaderboard/) for SecureVibeBench.
+- **[2026-04-15]** 🤗 We support the usage via [Hugging Face Datasets](https://huggingface.co/datasets/iCSawyer/SecureVibeBench). 
+- **[2026-04-11]** 🚀 We released code and data for SecureVibeBench. -->
+- **[2026-04-07]** 🎉 Our paper has been accepted to **ACL 2026**.
+
+
+<a id="nav-quick-start"></a>
+
+## 🛠️ Quick Start
+
+Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.
+
+### Install
+
+We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments. pip also works; uv is not installed as a package dependency.
+
+```bash
+# Run without installing into your current environment
+uvx securevibebench --help
+
+# Or install into an active virtual environment
+uv pip install securevibebench
+
+# Or with pip
+pip install securevibebench
+```
+
+Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.
+
+### For development
+
+```bash
+git clone https://github.com/iCSawyer/SecureVibeBench.git
+cd SecureVibeBench
+uv sync  # editable install + development tools
+```
+
+Use `uv run securevibebench eval ...` for the development checkout, or activate `.venv/bin/activate`.
+
+Without uv, use `python tools/install.py --dev --installer pip` in an active virtual environment.
+
+### Run evaluation
+
+```bash
+export OPENAI_API_KEY='<your-api-key>'
+export MODEL='openai/<model-name>'
+
+# With uv, without a persistent installation
+uvx securevibebench eval --model "$MODEL" --task 992
+
+# Or after installing the package
+securevibebench eval --model "$MODEL" --task 992
+```
+
+Omit `--task 992` to run all 105 tasks. Static analysis is disabled by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. Results are saved to `runs/<job>/summary.json` under the current working directory.
+
+> [!NOTE]
+> For custom endpoints, development setup, and more details, see the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md). Run `securevibebench eval --help` for evaluation options.
+
+
+<a id="nav-overview"></a>
+
+## 🔭 Overview
+
+SecureVibeBench is the **first** SWE-bench-level benchmark for secure vibe coding of agents, consisting of 105 C/C++ coding tasks sourced from real vulnerabilities ([OSS-Fuzz](https://github.com/google/oss-fuzz)/[ARVO](https://arxiv.org/abs/2408.02153)) covering various projects.
+
+![Overview](https://raw.githubusercontent.com/iCSawyer/SecureVibeBench/main/assets/overview.png)
+
+For each task in SecureVibeBench, we **reconstruct the real scenario where a human developer introduced a vulnerability** into the codebase, and then ask the agent to **implement the same requirements** and to see if the agent will also introduce the same vulnerability or not (and maybe new security issues as well).
+
+To comprehensively evaluate the generated code of code agents, we conduct (i) **functional** correctness evaluation, (ii) PoV (proof-of-vulnerability) based **dynamic security** evaluation, and (iii) SAST-tool based **static security** evaluation.
+
+> [!IMPORTANT]
+> **Why SecureVibeBench?**
+>
+> - First SWE-bench-level, peer-reviewed benchmark for secure vibe coding.
+> - Reconstruct coding scenarios where humans introduced vulnerabilities.
+> - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.
+
+
+<a id="nav-citation"></a>
+
+## 📚 Citation
+
+If you find SecureVibeBench useful, please cite:
+
+```bibtex
+@inproceedings{chen2026securevibebench,
+  title     = "{S}ecure{V}ibe{B}ench: Benchmarking Secure Vibe Coding of {AI} Agents via Reconstructing Vulnerability-Introducing Scenarios",
+  author    = "Chen, Junkai and Huang, Huihui and Lyu, Yunbo and An, Junwen and
+               Shi, Jieke and Yang, Chengran and Zhang, Ting and Tian, Haoye and
+               Li, Yikun and Li, Zhenhao and Zhou, Xin and Hu, Xing and Lo, David",
+  editor    = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
+  booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
+  year      = "2026", month = jul, pages = "24144--24168",
+  address   = "San Diego, California, United States",
+  publisher = "Association for Computational Linguistics",
+  url       = "https://aclanthology.org/2026.acl-long.1107/",
+  ISBN      = "979-8-89176-390-6"
+}
+```
+
+<a id="nav-acknowledgments"></a>
+
+## 🙏 Acknowledgments
+
+Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](https://arxiv.org/abs/2408.02153).
+
+```bibtex
+@misc{mei2024arvoatlasreproduciblevulnerabilities,
+  title         = {ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software},
+  author        = {Xiang Mei and Pulkit Singh Singaria and Jordi Del Castillo and Haoran Xi and
+                   Abdelouahab and Benchikh and Tiffany Bao and Ruoyu Wang and Yan Shoshitaishvili and
+                   Adam Doupé and Hammond Pearce and Brendan Dolan-Gavitt},
+  year          = {2024}, eprint = {2408.02153},
+  archivePrefix = {arXiv}, primaryClass = {cs.CR},
+  url           = {https://arxiv.org/abs/2408.02153}
+}
+```
+
+```bibtex
+@article{serebryany2017oss,
+  title  = {$\{$OSS-Fuzz$\}$-Google's continuous fuzzing service for open source software},
+  author = {Serebryany, Kostya},
+  year   = {2017}
+}
+```

securevibebench-0.2.0/README.md

@@ -0,0 +1,156 @@
+# SecureVibeBench
+
+<p align="left">
+    <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2">
+    <a href="https://2026.aclweb.org/"><img src="https://img.shields.io/badge/Venue-ACL%202026-2ea44f.svg?style=for-the-badge" alt="Venue: ACL 2026"></a>
+    <a href="https://arxiv.org/abs/2509.22097"><img src="https://img.shields.io/badge/arXiv-2509.22097-b31b1b.svg?style=for-the-badge" alt="arXiv"></a>
+    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a>
+    <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a>
+    <!-- <a href="https://chen-junkai.github.io/securevibebench-leaderboard/"><img src="https://img.shields.io/badge/Leaderboard-Online-2ea44f.svg?style=for-the-badge" alt="Leaderboard"></a> -->
+</p>
+
+> [!IMPORTANT]
+> **SecureVibeBench is now v0.2!** Try `pip install securevibebench`!
+>
+> We refer to the original release and evaluation workflow as **v0.1**, and **results obtained with v0.2 may differ from v0.1.** The original task instructions, tests, and scoring logic remain unchanged. Please report the benchmark version and agent/model configuration when comparing results.
+
+<p align="left">
+    ✨&nbsp;<a href="#nav-news">News</a>
+    | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
+    | 🔭&nbsp;<a href="#nav-overview">Overview</a>
+    | 📚&nbsp;<a href="#nav-citation">Citation</a>
+    | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
+</p>
+
+<a id="nav-news"></a>
+
+## ✨ News
+
+- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as **v0.1**; results may differ between versions.
+<!-- - **[2026-06-06]** 🏆 We released the online [Leaderboard](https://chen-junkai.github.io/securevibebench-leaderboard/) for SecureVibeBench.
+- **[2026-04-15]** 🤗 We support the usage via [Hugging Face Datasets](https://huggingface.co/datasets/iCSawyer/SecureVibeBench). 
+- **[2026-04-11]** 🚀 We released code and data for SecureVibeBench. -->
+- **[2026-04-07]** 🎉 Our paper has been accepted to **ACL 2026**.
+
+
+<a id="nav-quick-start"></a>
+
+## 🛠️ Quick Start
+
+Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.
+
+### Install
+
+We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments. pip also works; uv is not installed as a package dependency.
+
+```bash
+# Run without installing into your current environment
+uvx securevibebench --help
+
+# Or install into an active virtual environment
+uv pip install securevibebench
+
+# Or with pip
+pip install securevibebench
+```
+
+Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.
+
+### For development
+
+```bash
+git clone https://github.com/iCSawyer/SecureVibeBench.git
+cd SecureVibeBench
+uv sync  # editable install + development tools
+```
+
+Use `uv run securevibebench eval ...` for the development checkout, or activate `.venv/bin/activate`.
+
+Without uv, use `python tools/install.py --dev --installer pip` in an active virtual environment.
+
+### Run evaluation
+
+```bash
+export OPENAI_API_KEY='<your-api-key>'
+export MODEL='openai/<model-name>'
+
+# With uv, without a persistent installation
+uvx securevibebench eval --model "$MODEL" --task 992
+
+# Or after installing the package
+securevibebench eval --model "$MODEL" --task 992
+```
+
+Omit `--task 992` to run all 105 tasks. Static analysis is disabled by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. Results are saved to `runs/<job>/summary.json` under the current working directory.
+
+> [!NOTE]
+> For custom endpoints, development setup, and more details, see the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md). Run `securevibebench eval --help` for evaluation options.
+
+
+<a id="nav-overview"></a>
+
+## 🔭 Overview
+
+SecureVibeBench is the **first** SWE-bench-level benchmark for secure vibe coding of agents, consisting of 105 C/C++ coding tasks sourced from real vulnerabilities ([OSS-Fuzz](https://github.com/google/oss-fuzz)/[ARVO](https://arxiv.org/abs/2408.02153)) covering various projects.
+
+![Overview](https://raw.githubusercontent.com/iCSawyer/SecureVibeBench/main/assets/overview.png)
+
+For each task in SecureVibeBench, we **reconstruct the real scenario where a human developer introduced a vulnerability** into the codebase, and then ask the agent to **implement the same requirements** and to see if the agent will also introduce the same vulnerability or not (and maybe new security issues as well).
+
+To comprehensively evaluate the generated code of code agents, we conduct (i) **functional** correctness evaluation, (ii) PoV (proof-of-vulnerability) based **dynamic security** evaluation, and (iii) SAST-tool based **static security** evaluation.
+
+> [!IMPORTANT]
+> **Why SecureVibeBench?**
+>
+> - First SWE-bench-level, peer-reviewed benchmark for secure vibe coding.
+> - Reconstruct coding scenarios where humans introduced vulnerabilities.
+> - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.
+
+
+<a id="nav-citation"></a>
+
+## 📚 Citation
+
+If you find SecureVibeBench useful, please cite:
+
+```bibtex
+@inproceedings{chen2026securevibebench,
+  title     = "{S}ecure{V}ibe{B}ench: Benchmarking Secure Vibe Coding of {AI} Agents via Reconstructing Vulnerability-Introducing Scenarios",
+  author    = "Chen, Junkai and Huang, Huihui and Lyu, Yunbo and An, Junwen and
+               Shi, Jieke and Yang, Chengran and Zhang, Ting and Tian, Haoye and
+               Li, Yikun and Li, Zhenhao and Zhou, Xin and Hu, Xing and Lo, David",
+  editor    = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
+  booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
+  year      = "2026", month = jul, pages = "24144--24168",
+  address   = "San Diego, California, United States",
+  publisher = "Association for Computational Linguistics",
+  url       = "https://aclanthology.org/2026.acl-long.1107/",
+  ISBN      = "979-8-89176-390-6"
+}
+```
+
+<a id="nav-acknowledgments"></a>
+
+## 🙏 Acknowledgments
+
+Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](https://arxiv.org/abs/2408.02153).
+
+```bibtex
+@misc{mei2024arvoatlasreproduciblevulnerabilities,
+  title         = {ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software},
+  author        = {Xiang Mei and Pulkit Singh Singaria and Jordi Del Castillo and Haoran Xi and
+                   Abdelouahab and Benchikh and Tiffany Bao and Ruoyu Wang and Yan Shoshitaishvili and
+                   Adam Doupé and Hammond Pearce and Brendan Dolan-Gavitt},
+  year          = {2024}, eprint = {2408.02153},
+  archivePrefix = {arXiv}, primaryClass = {cs.CR},
+  url           = {https://arxiv.org/abs/2408.02153}
+}
+```
+
+```bibtex
+@article{serebryany2017oss,
+  title  = {$\{$OSS-Fuzz$\}$-Google's continuous fuzzing service for open source software},
+  author = {Serebryany, Kostya},
+  year   = {2017}
+}
+```

securevibebench-0.2.0/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,12 @@
+# Bundled resources
+
+SecureVibeBench installs the original dataset and evaluator from
+[iCSawyer/SecureVibeBench](https://huggingface.co/datasets/iCSawyer/SecureVibeBench)
+and bundles the source of [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent),
+both licensed under the MIT license. Source distributions include the agent's
+license in `mini-source.tar.gz`. Installed resources include both license texts
+as `dataset/LICENSE` and `mini-swe-agent/LICENSE.md`.
+
+The installed `resources.json` records the exact upstream commits and file
+checksums. Dataset instructions, tests, and scoring code are preserved verbatim;
+the host verifier is stored in `dataset/evaluator/` for packaging.

securevibebench-0.2.0/build_backend.py

@@ -0,0 +1,77 @@
+"""PEP 517 backend that fetches the pinned HF dataset during installation."""
+
+from __future__ import annotations
+
+import gzip
+import hashlib
+import importlib.util
+import io
+import json
+from pathlib import Path, PurePosixPath
+import tarfile
+import tempfile
+
+from setuptools import build_meta as _backend
+
+
+ROOT = Path(__file__).resolve().parent
+
+
+def _prepare_resources() -> None:
+    package = ROOT / "evaluation"
+    release = json.loads((package / "resources.json").read_text())
+    spec = importlib.util.spec_from_file_location("svb_build_bootstrap", package / "bootstrap.py")
+    bootstrap = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(bootstrap)
+    snapshot = bootstrap.download_dataset(release=release)
+
+    with tarfile.open(package / "mini-source.tar.gz", "r:gz") as source:
+        mini = {}
+        for member in source.getmembers():
+            if not member.isfile() or not member.name.startswith("mini-swe-agent/"):
+                raise ValueError(f"Invalid bundled agent source: {member.name}")
+            mini[member.name] = source.extractfile(member).read()
+
+    with tempfile.TemporaryDirectory(prefix=".svb-build-", dir=ROOT) as temporary:
+        archive = Path(temporary) / "resources.tar.gz"
+        with archive.open("wb") as raw, gzip.GzipFile(filename="", mode="wb", fileobj=raw, mtime=0) as zipped:
+            with tarfile.open(fileobj=zipped, mode="w") as tar:
+                for name, expected in sorted(release["files_sha256"].items()):
+                    relative = PurePosixPath(name)
+                    if relative.is_absolute() or ".." in relative.parts:
+                        raise ValueError(f"Unsafe resource path: {name}")
+                    if name.startswith("dataset/evaluator/"):
+                        path = snapshot / "habour/dev" / name.removeprefix("dataset/evaluator/")
+                        content = path.read_bytes()
+                    elif name.startswith("dataset/"):
+                        content = (snapshot / "habour" / name.removeprefix("dataset/")).read_bytes()
+                    else:
+                        content = mini[name]
+                    if hashlib.sha256(content).hexdigest() != expected:
+                        raise ValueError(f"Resource checksum mismatch: {name}")
+                    info = tarfile.TarInfo(name)
+                    info.size, info.mode = len(content), 0o644
+                    tar.addfile(info, io.BytesIO(content))
+        # Compression output can vary with Python/zlib versions. File checksums
+        # above pin the content; record the archive produced by this build.
+        release["archive_sha256"] = hashlib.sha256(archive.read_bytes()).hexdigest()
+        archive.replace(package / "resources.tar.gz")
+        (package / "resources.json").write_text(json.dumps(release, indent=2) + "\n")
+
+
+def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
+    _prepare_resources()
+    return _backend.build_wheel(wheel_directory, config_settings, metadata_directory)
+
+
+def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
+    _prepare_resources()
+    return _backend.build_editable(wheel_directory, config_settings, metadata_directory)
+
+
+build_sdist = _backend.build_sdist
+get_requires_for_build_sdist = _backend.get_requires_for_build_sdist
+get_requires_for_build_wheel = _backend.get_requires_for_build_wheel
+get_requires_for_build_editable = _backend.get_requires_for_build_editable
+prepare_metadata_for_build_wheel = _backend.prepare_metadata_for_build_wheel
+prepare_metadata_for_build_editable = _backend.prepare_metadata_for_build_editable

securevibebench-0.2.0/docs/README.md

@@ -0,0 +1,77 @@
+# Usage Guide
+
+Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon. We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/); pip also works. uv is not a host package dependency.
+
+## Install
+
+```bash
+# Run in an isolated environment managed by uv
+uvx securevibebench --help
+
+# Or install into an active virtual environment
+uv pip install securevibebench
+
+# Or use pip
+pip install securevibebench
+```
+
+Create a virtual environment first if needed: `uv venv` or `python -m venv .venv`, then `source .venv/bin/activate`.
+
+Installation downloads the pinned 105-task dataset and original evaluator from Hugging Face. pip/uv installs mini-swe-agent as a dependency. Docker images and container tools are prepared when evaluation starts.
+
+## Source installation and development
+
+From a source checkout:
+
+```bash
+python tools/install.py
+```
+
+The installer uses uv when available, otherwise the current Python's pip. Use `--installer pip` to choose pip explicitly.
+
+For editable source and development tools:
+
+```bash
+uv sync
+# Or with pip, in an active virtual environment
+python tools/install.py --dev --installer pip
+```
+
+The uv path creates `.venv`; activate it with `source .venv/bin/activate`. The pip path installs into the current environment.
+
+## Evaluate
+
+```bash
+export OPENAI_API_KEY='<your-api-key>'
+securevibebench eval --model 'openai/<model-name>' --task 992
+```
+
+Omit `--task 992` to run all tasks, or repeat `--task` to select several. Without a persistent install, use `uvx securevibebench eval ...`. Other providers use their own prefix and credential variable, such as `anthropic/<model-name>` and `ANTHROPIC_API_KEY`.
+
+For an OpenAI-compatible endpoint:
+
+```bash
+export MODEL_API_KEY='<your-endpoint-api-key>'
+securevibebench eval --model 'openai/<model-name>' --task 992 \
+  --base-url 'http://<model-server>:8000/v1' \
+  --api-key-name MODEL_API_KEY \
+  --api-format openai_chat_completions
+```
+
+The endpoint must be reachable from task containers. Set the key to `unused` if authentication is disabled. SAST is disabled by default. To enable it, pass `--run-sast TRUE` and provide Semgrep configuration and credentials such as `SEMGREP_APP_TOKEN`. See `securevibebench eval --help` for options.
+
+## Results
+
+Results are written to `runs/<job>/summary.json` under the current working directory, alongside trajectories, patches, and original reports. Functional and PoV security scores remain independent and retain the original scoring rules.
+
+Downloaded data is cached in `~/.cache/securevibebench/huggingface/`; installed resources are extracted to `~/.cache/securevibebench/resources/`. Both respect `XDG_CACHE_HOME`. The package's `resources.json` records upstream commits and checksums. No `dev/` directory is required.
+
+## Developer checks
+
+```bash
+uv run python -m unittest discover -s tests -v
+uv build --sdist
+uv run twine check dist/*.tar.gz
+```
+
+With pip, use `python -m unittest discover -s tests -v`, `python -m build --sdist`, and `python -m twine check dist/*.tar.gz` instead. Publish the source archive (`dist/*.tar.gz`) to PyPI. Development dependencies are installed by `tools/install.py --dev`. To refresh release resources, run `python tools/bundle_resources.py` (requires Git).

securevibebench-0.2.0/evaluation/README.md

@@ -0,0 +1,6 @@
+# Evaluation
+
+Run SecureVibeBench with `securevibebench eval --model '<provider/model>'`.
+
+See the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md)
+for installation, development setup, model configuration, and results.

securevibebench-0.2.0/evaluation/__init__.py

@@ -0,0 +1 @@
+"""HF-sourced tasks and evaluation, orchestrated by official Harbor agents."""

securevibebench-0.2.0/evaluation/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())