securevibebench 0.2.0__tar.gz → 0.2.2__tar.gz

{securevibebench-0.2.0 → securevibebench-0.2.2}/MANIFEST.in

@@ -1,5 +1,4 @@
-include LICENSE THIRD_PARTY_NOTICES.md README.md uv.lock
-include build_backend.py
+include LICENSE README.md uv.lock
 recursive-include evaluation *.py *.yaml *.sh *.md
 include evaluation/resources.json evaluation/mini-source.tar.gz
 exclude evaluation/resources.tar.gz

{securevibebench-0.2.0/securevibebench.egg-info → securevibebench-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: securevibebench
-Version: 0.2.0
+Version: 0.2.2
 Summary: Generate and evaluate secure coding agents on SecureVibeBench
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/iCSawyer/SecureVibeBench
@@ -16,10 +16,8 @@ Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-License-File: THIRD_PARTY_NOTICES.md
+License-File: docs/THIRD_PARTY_NOTICES.md
 Requires-Dist: harbor<0.26.0,>=0.25.0
-Requires-Dist: mini-swe-agent==2.4.6
-Requires-Dist: huggingface-hub>=0.34.3
 Requires-Dist: docker>=7.1.0
 Requires-Dist: pyyaml>=6.0.2
 Requires-Dist: python-dotenv>=1.1.1
@@ -28,18 +26,23 @@ Dynamic: license-file
 # SecureVibeBench
 
 <p align="left">
-    <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2">
+    <!-- <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2"> -->
     <a href="https://2026.aclweb.org/"><img src="https://img.shields.io/badge/Venue-ACL%202026-2ea44f.svg?style=for-the-badge" alt="Venue: ACL 2026"></a>
     <a href="https://arxiv.org/abs/2509.22097"><img src="https://img.shields.io/badge/arXiv-2509.22097-b31b1b.svg?style=for-the-badge" alt="arXiv"></a>
-    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a>
-    <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a>
+    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhuggingface.co%2Fapi%2Fdatasets%2FiCSawyer%2FSecureVibeBench%3Fexpand%255B%255D%3DdownloadsAllTime&query=%24.downloadsAllTime&label=Downloads&color=FFD21F&style=for-the-badge&logo=huggingface" alt="Hugging Face Downloads"></a>
+    <!-- <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a> -->
+    <!-- <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a> -->
     <!-- <a href="https://chen-junkai.github.io/securevibebench-leaderboard/"><img src="https://img.shields.io/badge/Leaderboard-Online-2ea44f.svg?style=for-the-badge" alt="Leaderboard"></a> -->
 </p>
 
+
+
+
 > [!IMPORTANT]
-> **SecureVibeBench is now v0.2!** Try `pip install securevibebench`!
+> **SecureVibeBench is now v0.2!** Just try `pip install -U securevibebench`!
 >
-> We refer to the original release and evaluation workflow as **v0.1**, and **results obtained with v0.2 may differ from v0.1.** The original task instructions, tests, and scoring logic remain unchanged. Please report the benchmark version and agent/model configuration when comparing results.
+> We are updating the evaluation criteria and workflow currently, so **the results obtained may differ from v0.1**.
+> Please report any usage issues you encouter via Issues or my [email](mailto:junkaichen@smu.edu.sg), and we will investigate and fix as soon as possible. 🙏
 
 <p align="left">
     ✨&nbsp;<a href="#nav-news">News</a>
@@ -53,7 +56,7 @@ Dynamic: license-file
 
 ## ✨ News
 
-- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as **v0.1**; results may differ between versions.
+- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model evaluation. The original version is now referred to as **v0.1**.
 <!-- - **[2026-06-06]** 🏆 We released the online [Leaderboard](https://chen-junkai.github.io/securevibebench-leaderboard/) for SecureVibeBench.
 - **[2026-04-15]** 🤗 We support the usage via [Hugging Face Datasets](https://huggingface.co/datasets/iCSawyer/SecureVibeBench). 
 - **[2026-04-11]** 🚀 We released code and data for SecureVibeBench. -->
@@ -64,24 +67,21 @@ Dynamic: license-file
 
 ## 🛠️ Quick Start
 
-Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.
+Requirements: Python 3.12+, Linux, and Docker.
 
 ### Install
 
-We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments. pip also works; uv is not installed as a package dependency.
+We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments.
 
 ```bash
-# Run without installing into your current environment
-uvx securevibebench --help
-
-# Or install into an active virtual environment
-uv pip install securevibebench
-
+uv pip install -U securevibebench
+```
+```bash
 # Or with pip
-pip install securevibebench
+pip install -U securevibebench
 ```
 
-Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.
+Task containers will be prepared when the real evaluation starts.
 
 ### For development
 
@@ -91,24 +91,19 @@ cd SecureVibeBench
 uv sync  # editable install + development tools
 ```
 
-Use `uv run securevibebench eval ...` for the development checkout, or activate `.venv/bin/activate`.
-
-Without uv, use `python tools/install.py --dev --installer pip` in an active virtual environment.
-
 ### Run evaluation
 
 ```bash
 export OPENAI_API_KEY='<your-api-key>'
 export MODEL='openai/<model-name>'
 
-# With uv, without a persistent installation
-uvx securevibebench eval --model "$MODEL" --task 992
-
-# Or after installing the package
 securevibebench eval --model "$MODEL" --task 992
+# securevibebench eval --model "$MODEL" # all 105 tasks
 ```
 
-Omit `--task 992` to run all 105 tasks. Static analysis is disabled by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. Results are saved to `runs/<job>/summary.json` under the current working directory.
+Results are saved to `runs/<job>/summary.json` under the current working directory.
+
+We disable static vulnerability analysis by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. 
 
 > [!NOTE]
 > For custom endpoints, development setup, and more details, see the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md). Run `securevibebench eval --help` for evaluation options.
@@ -138,7 +133,7 @@ To comprehensively evaluate the generated code of code agents, we conduct (i) **
 
 ## 📚 Citation
 
-If you find SecureVibeBench useful, please cite:
+If you find our project useful, please consider citing our paper: 
 
 ```bibtex
 @inproceedings{chen2026securevibebench,
@@ -160,7 +155,7 @@ If you find SecureVibeBench useful, please cite:
 
 ## 🙏 Acknowledgments
 
-Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](https://arxiv.org/abs/2408.02153).
+Our work builds on the following excellent works, and please consider citing them: [OSS-Fuzz](https://github.com/google/oss-fuzz), [ARVO](https://arxiv.org/abs/2408.02153), [habour](https://github.com/harbor-framework/harbor), and [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent).
 
 ```bibtex
 @misc{mei2024arvoatlasreproduciblevulnerabilities,
@@ -181,3 +176,23 @@ Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](htt
   year   = {2017}
 }
 ```
+
+```bibtex
+@software{Harbor_Framework,
+author = {{Harbor Framework Team}},
+title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
+year = {2026},
+doi = {10.5281/zenodo.20953922},
+url = {https://doi.org/10.5281/zenodo.20953922}
+}
+```
+
+```bibtex
+@inproceedings{yang2024sweagent,
+  title={{SWE}-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
+  author={John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press},
+  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
+  year={2024},
+  url={https://arxiv.org/abs/2405.15793}
+}
+```

{securevibebench-0.2.0 → securevibebench-0.2.2}/README.md

@@ -1,18 +1,23 @@
 # SecureVibeBench
 
 <p align="left">
-    <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2">
+    <!-- <img src="https://img.shields.io/badge/Version-v0.2-blue.svg?style=for-the-badge" alt="Version: v0.2"> -->
     <a href="https://2026.aclweb.org/"><img src="https://img.shields.io/badge/Venue-ACL%202026-2ea44f.svg?style=for-the-badge" alt="Venue: ACL 2026"></a>
     <a href="https://arxiv.org/abs/2509.22097"><img src="https://img.shields.io/badge/arXiv-2509.22097-b31b1b.svg?style=for-the-badge" alt="arXiv"></a>
-    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a>
-    <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a>
+    <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhuggingface.co%2Fapi%2Fdatasets%2FiCSawyer%2FSecureVibeBench%3Fexpand%255B%255D%3DdownloadsAllTime&query=%24.downloadsAllTime&label=Downloads&color=FFD21F&style=for-the-badge&logo=huggingface" alt="Hugging Face Downloads"></a>
+    <!-- <a href="https://huggingface.co/datasets/iCSawyer/SecureVibeBench"><img src="https://img.shields.io/badge/HF-SecureVibeBench-FFD21F.svg?style=for-the-badge&logo=huggingface" alt="Hugging Face: SecureVibeBench"></a> -->
+    <!-- <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="License MIT"></a> -->
     <!-- <a href="https://chen-junkai.github.io/securevibebench-leaderboard/"><img src="https://img.shields.io/badge/Leaderboard-Online-2ea44f.svg?style=for-the-badge" alt="Leaderboard"></a> -->
 </p>
 
+
+
+
 > [!IMPORTANT]
-> **SecureVibeBench is now v0.2!** Try `pip install securevibebench`!
+> **SecureVibeBench is now v0.2!** Just try `pip install -U securevibebench`!
 >
-> We refer to the original release and evaluation workflow as **v0.1**, and **results obtained with v0.2 may differ from v0.1.** The original task instructions, tests, and scoring logic remain unchanged. Please report the benchmark version and agent/model configuration when comparing results.
+> We are updating the evaluation criteria and workflow currently, so **the results obtained may differ from v0.1**.
+> Please report any usage issues you encouter via Issues or my [email](mailto:junkaichen@smu.edu.sg), and we will investigate and fix as soon as possible. 🙏
 
 <p align="left">
     ✨&nbsp;<a href="#nav-news">News</a>
@@ -26,7 +31,7 @@
 
 ## ✨ News
 
-- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as **v0.1**; results may differ between versions.
+- **[2026-10-29]** We updated to **v0.2**, introducing a PyPI package, Harbor-format tasks, and unified model evaluation. The original version is now referred to as **v0.1**.
 <!-- - **[2026-06-06]** 🏆 We released the online [Leaderboard](https://chen-junkai.github.io/securevibebench-leaderboard/) for SecureVibeBench.
 - **[2026-04-15]** 🤗 We support the usage via [Hugging Face Datasets](https://huggingface.co/datasets/iCSawyer/SecureVibeBench). 
 - **[2026-04-11]** 🚀 We released code and data for SecureVibeBench. -->
@@ -37,24 +42,21 @@
 
 ## 🛠️ Quick Start
 
-Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.
+Requirements: Python 3.12+, Linux, and Docker.
 
 ### Install
 
-We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments. pip also works; uv is not installed as a package dependency.
+We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments.
 
 ```bash
-# Run without installing into your current environment
-uvx securevibebench --help
-
-# Or install into an active virtual environment
-uv pip install securevibebench
-
+uv pip install -U securevibebench
+```
+```bash
 # Or with pip
-pip install securevibebench
+pip install -U securevibebench
 ```
 
-Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.
+Task containers will be prepared when the real evaluation starts.
 
 ### For development
 
@@ -64,24 +66,19 @@ cd SecureVibeBench
 uv sync  # editable install + development tools
 ```
 
-Use `uv run securevibebench eval ...` for the development checkout, or activate `.venv/bin/activate`.
-
-Without uv, use `python tools/install.py --dev --installer pip` in an active virtual environment.
-
 ### Run evaluation
 
 ```bash
 export OPENAI_API_KEY='<your-api-key>'
 export MODEL='openai/<model-name>'
 
-# With uv, without a persistent installation
-uvx securevibebench eval --model "$MODEL" --task 992
-
-# Or after installing the package
 securevibebench eval --model "$MODEL" --task 992
+# securevibebench eval --model "$MODEL" # all 105 tasks
 ```
 
-Omit `--task 992` to run all 105 tasks. Static analysis is disabled by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. Results are saved to `runs/<job>/summary.json` under the current working directory.
+Results are saved to `runs/<job>/summary.json` under the current working directory.
+
+We disable static vulnerability analysis by default. Enable it with `--run-sast TRUE` and configure Semgrep credentials such as `SEMGREP_APP_TOKEN`. 
 
 > [!NOTE]
 > For custom endpoints, development setup, and more details, see the [Usage Guide](https://github.com/iCSawyer/SecureVibeBench/blob/main/docs/README.md). Run `securevibebench eval --help` for evaluation options.
@@ -111,7 +108,7 @@ To comprehensively evaluate the generated code of code agents, we conduct (i) **
 
 ## 📚 Citation
 
-If you find SecureVibeBench useful, please cite:
+If you find our project useful, please consider citing our paper: 
 
 ```bibtex
 @inproceedings{chen2026securevibebench,
@@ -133,7 +130,7 @@ If you find SecureVibeBench useful, please cite:
 
 ## 🙏 Acknowledgments
 
-Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](https://arxiv.org/abs/2408.02153).
+Our work builds on the following excellent works, and please consider citing them: [OSS-Fuzz](https://github.com/google/oss-fuzz), [ARVO](https://arxiv.org/abs/2408.02153), [habour](https://github.com/harbor-framework/harbor), and [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent).
 
 ```bibtex
 @misc{mei2024arvoatlasreproduciblevulnerabilities,
@@ -154,3 +151,23 @@ Our work builds on [OSS-Fuzz](https://github.com/google/oss-fuzz) and [ARVO](htt
   year   = {2017}
 }
 ```
+
+```bibtex
+@software{Harbor_Framework,
+author = {{Harbor Framework Team}},
+title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
+year = {2026},
+doi = {10.5281/zenodo.20953922},
+url = {https://doi.org/10.5281/zenodo.20953922}
+}
+```
+
+```bibtex
+@inproceedings{yang2024sweagent,
+  title={{SWE}-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
+  author={John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press},
+  booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
+  year={2024},
+  url={https://arxiv.org/abs/2405.15793}
+}
+```

{securevibebench-0.2.0 → securevibebench-0.2.2}/docs/README.md

@@ -17,7 +17,7 @@ pip install securevibebench
 
 Create a virtual environment first if needed: `uv venv` or `python -m venv .venv`, then `source .venv/bin/activate`.
 
-Installation downloads the pinned 105-task dataset and original evaluator from Hugging Face. pip/uv installs mini-swe-agent as a dependency. Docker images and container tools are prepared when evaluation starts.
+Installation downloads the pinned 105-task dataset and original evaluator from Hugging Face. mini-swe-agent is installed inside task containers when evaluation starts; it is not installed on the host. Docker images and container tools are prepared automatically.
 
 ## Source installation and development
 
@@ -60,11 +60,27 @@ securevibebench eval --model 'openai/<model-name>' --task 992 \
 
 The endpoint must be reachable from task containers. Set the key to `unused` if authentication is disabled. SAST is disabled by default. To enable it, pass `--run-sast TRUE` and provide Semgrep configuration and credentials such as `SEMGREP_APP_TOKEN`. See `securevibebench eval --help` for options.
 
+## Agent installation cache
+
+Each task container needs uv, a managed Python, and mini-swe-agent. The first task downloads and installs them, then saves the result to `.cache/securevibebench/agent-toolchain/` in the current working directory; later tasks restore that archive instead of downloading again (about 25 seconds instead of several minutes). Archives are keyed by the container's `$HOME`, CPU architecture, and C library, so images that cannot share one get their own. If a restore fails, the task installs from scratch. Delete the directory to force a fresh install.
+
+## Interrupt and resume
+
+Progress is printed as one line per task phase. Ctrl-C, `SIGTERM`, or `SIGHUP` stops the run, removes its task and evaluator containers, and keeps finished tasks in `summary.json` (`"status": "interrupted"`). Continue later with the saved settings; finished tasks are not rerun:
+
+```bash
+securevibebench eval --resume runs/<job>
+```
+
+Resuming also removes containers left by a run that was killed outright (for example with `SIGKILL`).
+
 ## Results
 
-Results are written to `runs/<job>/summary.json` under the current working directory, alongside trajectories, patches, and original reports. Functional and PoV security scores remain independent and retain the original scoring rules.
+Results are written to `runs/<job>/summary.json` under the current working directory, alongside trajectories, patches, and original reports. The summary is updated after every finished task. Functional and PoV security scores remain independent and retain the original scoring rules.
+
+Since v0.2.1 the functional-test parser reads parallel `ctest -j` output per test. The Wireshark tasks interleave output from concurrent tests, and the original parser could pair a test with another test's status, misreporting passes and failures. Functional results for those tasks can therefore differ from v0.2.0 and v0.1; other tasks are unaffected. Locally cached baselines are rebuilt automatically.
 
-Downloaded data is cached in `~/.cache/securevibebench/huggingface/`; installed resources are extracted to `~/.cache/securevibebench/resources/`. Both respect `XDG_CACHE_HOME`. The package's `resources.json` records upstream commits and checksums. No `dev/` directory is required.
+Runtime caches live next to `runs/` in the current working directory: resources and functional-test baselines in `.cache/securevibebench/resources/`, and the agent toolchain in `.cache/securevibebench/agent-toolchain/`. The dataset downloaded during installation is cached in `~/.cache/securevibebench/huggingface/` (respects `XDG_CACHE_HOME`). The package's `resources.json` records upstream commits and checksums. No `dev/` directory is required.
 
 ## Developer checks
 
@@ -74,4 +90,4 @@ uv build --sdist
 uv run twine check dist/*.tar.gz
 ```
 
-With pip, use `python -m unittest discover -s tests -v`, `python -m build --sdist`, and `python -m twine check dist/*.tar.gz` instead. Publish the source archive (`dist/*.tar.gz`) to PyPI. Development dependencies are installed by `tools/install.py --dev`. To refresh release resources, run `python tools/bundle_resources.py` (requires Git).
+With pip, use `python -m unittest discover -s tests -v`, `python -m build --sdist`, and `python -m twine check dist/*.tar.gz` instead. Pushing a `v<version>` tag publishes the source archive to PyPI through `.github/workflows/package.yml`. Development dependencies are installed by `tools/install.py --dev`. To refresh release resources, run `python tools/bundle_resources.py` (requires Git).

securevibebench-0.2.2/docs/changeLog.md

@@ -0,0 +1,25 @@
+# Change Log
+
+## Roadmap
+
+- [ ] **Cascaded LLM-as-a-judge.** Add an LLM judge as a final stage after the functional checks. Its verdicts would be reported separately so they don't change the existing scores.
+
+## v0.2.2
+
+- Minor changes: the README install commands use `pip install -U`, the roadmap moved into this change log, and repository files were reorganized. Evaluation is unchanged.
+
+## v0.2.1
+
+- **Bug fix: functional evaluation of task 10172 and the other Wireshark tasks.** Their test scripts run `ctest -j`, which interleaves output from concurrent tests. The parser could attach a test name to another test's status and also counted summaries such as `OK (skipped=2)` as tests. For example, a 10172 patch that passed every test scored `functional = 0`. The parser now reads each test's output on its own. Functional scores for these tasks can differ from v0.2.0 and v0.1; other tasks are unaffected. The dataset is pinned to HF `habour-preview` commit `5fa48dd`.
+- Interrupting a run with Ctrl-C, `SIGTERM`, or `SIGHUP` now removes its containers and keeps finished tasks in `summary.json`. Continue the run with `securevibebench eval --resume runs/<job>`.
+- `summary.json` is updated after each task, and progress prints one line per task phase.
+- The agent toolchain is cached across tasks. Runtime caches (resources, baselines, agent toolchain) now live in `.cache/securevibebench/` under the working directory, next to `runs/`.
+- mini-swe-agent runs only inside task containers and is no longer installed on the host.
+- Added `securevibebench --version`.
+
+## v0.2.0
+
+- First PyPI release: `pip install securevibebench` installs all 105 tasks and the original evaluator from Hugging Face. The tasks use the Harbor format.
+- A single command, `securevibebench eval`, generates code with mini-swe-agent and evaluates it, and works with custom OpenAI-compatible endpoints.
+- SAST is disabled by default; enable it with `--run-sast TRUE`.
+- Task instructions, tests, and scoring logic are unchanged from v0.1, but results can differ because the agent and evaluation workflow changed.

{securevibebench-0.2.0 → securevibebench-0.2.2}/evaluation/cli.py

@@ -3,20 +3,30 @@
 from __future__ import annotations
 
 import argparse
+from importlib.metadata import PackageNotFoundError, version
 import sys
 
 from . import run
 
 
+def package_version() -> str:
+    try:
+        return version("securevibebench")
+    except PackageNotFoundError:
+        return "unknown"
+
+
 def build_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="securevibebench", description="Run secure coding agent benchmarks.",
     )
+    parser.add_argument("--version", action="version", version=f"%(prog)s {package_version()}")
     commands = parser.add_subparsers(dest="command", required=True)
-    commands.add_parser(
+    evaluation = commands.add_parser(
         "eval", help="Generate and evaluate code on SecureVibeBench tasks",
         description=run.__doc__, parents=[run.build_parser()], add_help=False,
     )
+    evaluation.set_defaults(usage_error=evaluation.error)
     return parser
 
 
@@ -26,4 +36,4 @@ def main(argv: list[str] | None = None) -> int:
     if not arguments:
         parser.print_help()
         return 0
-    return run.evaluate(parser.parse_args(arguments))
+    return run.evaluate(run.check_arguments(parser.parse_args(arguments)))

{securevibebench-0.2.0 → securevibebench-0.2.2}/evaluation/editable_mini.py

@@ -6,19 +6,30 @@ version detection, and ATIF conversion are inherited from Harbor's integration.
 
 from __future__ import annotations
 
+import hashlib
 import json
+import os
 from pathlib import Path
 import shlex
 import shutil
 import tempfile
 from typing import Any
 
-from harbor.agents.installed.mini_swe_agent import MiniSweAgent, MiniSweAgentOptions
+from harbor.agents.installed.mini_swe_agent import (
+    _MINI_SWE_AGENT_PYTHON_VERSION, MiniSweAgent, MiniSweAgentOptions,
+)
 from harbor.environments.base import BaseEnvironment
 from pydantic import Field
 
 
 REMOTE_SOURCE = "/installed-agent/mini-swe-agent-source"
+REMOTE_TOOLCHAIN = "/tmp/securevibebench-agent-toolchain.tar.gz"
+# uv, its managed Python, and the mini-swe-agent tool environment, relative to $HOME.
+TOOLCHAIN_PATHS = ".local/bin .local/share/uv"
+AGENT_PATH = (
+    'if [ -f "$HOME/.local/bin/env" ]; then . "$HOME/.local/bin/env"; fi; '
+    'export PATH="$HOME/.local/bin:$PATH"; '
+)
 
 
 def copy_source(source: Path, target: Path) -> None:
@@ -59,22 +70,71 @@ print(json.dumps({{"version": dist.version, "editable": True, "url": direct["url
 
 class EditableMiniSweAgentOptions(MiniSweAgentOptions):
     source_dir: str = Field(description="Local official mini-swe-agent source checkout to install editable")
+    toolchain_cache_dir: str | None = Field(
+        default=None, description="Host directory reusing the installed agent toolchain across tasks")
 
 
 class EditableMiniSweAgent(MiniSweAgent):
     options_model = EditableMiniSweAgentOptions
     options: EditableMiniSweAgentOptions
 
-    def __init__(self, *, source_dir: str | Path, **kwargs: Any) -> None:
+    def __init__(self, *, source_dir: str | Path, toolchain_cache_dir: str | Path | None = None,
+                 **kwargs: Any) -> None:
         self.source_dir = Path(source_dir).resolve()
         for relative in ("pyproject.toml", "README.md", "LICENSE.md", "src/minisweagent/__init__.py"):
             if not (self.source_dir / relative).is_file():
                 raise ValueError(f"Incomplete mini-swe-agent checkout: {self.source_dir / relative}")
-        super().__init__(source_dir=str(self.source_dir), **kwargs)
+        self.toolchain_cache_dir = Path(toolchain_cache_dir).resolve() if toolchain_cache_dir else None
+        super().__init__(source_dir=str(self.source_dir),
+                         toolchain_cache_dir=str(self.toolchain_cache_dir) if self.toolchain_cache_dir else None,
+                         **kwargs)
+
+    async def toolchain_archive(self, environment: BaseEnvironment) -> Path | None:
+        """Cache key: binaries and wheels only run where $HOME, arch, and libc match."""
+        if self.toolchain_cache_dir is None:
+            return None
+        probe = await self.exec_as_agent(
+            environment, command='printf "%s|%s|%s" "$HOME" "$(uname -m)" "$(ldd --version 2>&1 | head -n 1)"')
+        key = hashlib.sha256(f"{probe.stdout}|{self._version}|{_MINI_SWE_AGENT_PYTHON_VERSION}".encode())
+        return self.toolchain_cache_dir / f"mini-swe-agent-{self._version}-{key.hexdigest()[:16]}.tar.gz"
+
+    async def restore_toolchain(self, environment: BaseEnvironment, archive: Path | None) -> bool:
+        if archive is None or not archive.is_file():
+            return False
+        try:
+            await self.ensure_system_dependencies(environment, ("bash", "git"))
+            await environment.upload_file(source_path=archive, target_path=REMOTE_TOOLCHAIN)
+            await self.exec_as_agent(environment, command=(
+                f'tar -xzf {REMOTE_TOOLCHAIN} -C "$HOME" && rm -f {REMOTE_TOOLCHAIN} && '
+                'if ! grep -q \'export PATH="$HOME/.local/bin:$PATH"\' "$HOME/.bashrc" 2>/dev/null; then'
+                '  echo \'export PATH="$HOME/.local/bin:$PATH"\' >> "$HOME/.bashrc"; fi; '
+                + AGENT_PATH + "mini-swe-agent --help >/dev/null"))
+        except Exception as error:
+            self.logger.warning(f"Cached agent toolchain {archive} is unusable, reinstalling: {error}")
+            return False
+        return True
+
+    async def save_toolchain(self, environment: BaseEnvironment, archive: Path | None) -> None:
+        if archive is None:
+            return
+        try:
+            await self.exec_as_agent(
+                environment, command=f'tar -czf {REMOTE_TOOLCHAIN} -C "$HOME" {TOOLCHAIN_PATHS}')
+            archive.parent.mkdir(parents=True, exist_ok=True)
+            partial = archive.with_name(f".{archive.name}.{os.getpid()}.{id(self)}")
+            await environment.download_file(source_path=REMOTE_TOOLCHAIN, target_path=partial)
+            await self.exec_as_agent(environment, command=f"rm -f {REMOTE_TOOLCHAIN}")
+            # Concurrent trials may race to save the same key; either copy is complete.
+            os.replace(partial, archive)
+        except Exception as error:
+            self.logger.warning(f"Cannot cache the agent toolchain: {error}")
 
     async def install(self, environment: BaseEnvironment) -> None:
-        # Reuse upstream bootstrap rather than maintaining a second installer.
-        await super().install(environment)
+        archive = await self.toolchain_archive(environment)
+        if not await self.restore_toolchain(environment, archive):
+            # Reuse upstream bootstrap rather than maintaining a second installer.
+            await super().install(environment)
+            await self.save_toolchain(environment, archive)
         setup_dir = self.logs_dir / "setup"
         setup_dir.mkdir(parents=True, exist_ok=True)
         with tempfile.TemporaryDirectory(prefix="mini-source-", dir=setup_dir) as temporary: