securevibebench 0.2.1__tar.gz → 0.2.2__tar.gz

{securevibebench-0.2.1 → securevibebench-0.2.2}/MANIFEST.in

@@ -1,5 +1,4 @@
-include LICENSE THIRD_PARTY_NOTICES.md README.md uv.lock
-include build_backend.py
+include LICENSE README.md uv.lock
 recursive-include evaluation *.py *.yaml *.sh *.md
 include evaluation/resources.json evaluation/mini-source.tar.gz
 exclude evaluation/resources.tar.gz

{securevibebench-0.2.1/securevibebench.egg-info → securevibebench-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: securevibebench
-Version: 0.2.1
+Version: 0.2.2
 Summary: Generate and evaluate secure coding agents on SecureVibeBench
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/iCSawyer/SecureVibeBench
@@ -16,7 +16,7 @@ Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-License-File: THIRD_PARTY_NOTICES.md
+License-File: docs/THIRD_PARTY_NOTICES.md
 Requires-Dist: harbor<0.26.0,>=0.25.0
 Requires-Dist: docker>=7.1.0
 Requires-Dist: pyyaml>=6.0.2
@@ -39,7 +39,7 @@ Dynamic: license-file
 
 
 > [!IMPORTANT]
-> **SecureVibeBench is now v0.2!** Just try `pip install securevibebench`!
+> **SecureVibeBench is now v0.2!** Just try `pip install -U securevibebench`!
 >
 > We are updating the evaluation criteria and workflow currently, so **the results obtained may differ from v0.1**.
 > Please report any usage issues you encouter via Issues or my [email](mailto:junkaichen@smu.edu.sg), and we will investigate and fix as soon as possible. 🙏
@@ -48,7 +48,6 @@ Dynamic: license-file
     ✨&nbsp;<a href="#nav-news">News</a>
     | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
     | 🔭&nbsp;<a href="#nav-overview">Overview</a>
-    | 🗺️&nbsp;<a href="#nav-roadmap">Roadmap</a>
     | 📚&nbsp;<a href="#nav-citation">Citation</a>
     | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
 </p>
@@ -75,11 +74,11 @@ Requirements: Python 3.12+, Linux, and Docker.
 We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments.
 
 ```bash
-uv pip install securevibebench
+uv pip install -U securevibebench
 ```
 ```bash
 # Or with pip
-pip install securevibebench
+pip install -U securevibebench
 ```
 
 Task containers will be prepared when the real evaluation starts.
@@ -130,13 +129,6 @@ To comprehensively evaluate the generated code of code agents, we conduct (i) **
 > - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.
 
 
-<a id="nav-roadmap"></a>
-
-## 🗺️ Roadmap
-
-- [ ] **Cascaded LLM-as-a-judge.** Add an LLM judge as a final stage after the functional checks. Its verdicts would be reported separately so they don't change the existing scores.
-
-
 <a id="nav-citation"></a>
 
 ## 📚 Citation

{securevibebench-0.2.1 → securevibebench-0.2.2}/README.md

@@ -14,7 +14,7 @@
 
 
 > [!IMPORTANT]
-> **SecureVibeBench is now v0.2!** Just try `pip install securevibebench`!
+> **SecureVibeBench is now v0.2!** Just try `pip install -U securevibebench`!
 >
 > We are updating the evaluation criteria and workflow currently, so **the results obtained may differ from v0.1**.
 > Please report any usage issues you encouter via Issues or my [email](mailto:junkaichen@smu.edu.sg), and we will investigate and fix as soon as possible. 🙏
@@ -23,7 +23,6 @@
     ✨&nbsp;<a href="#nav-news">News</a>
     | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
     | 🔭&nbsp;<a href="#nav-overview">Overview</a>
-    | 🗺️&nbsp;<a href="#nav-roadmap">Roadmap</a>
     | 📚&nbsp;<a href="#nav-citation">Citation</a>
     | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
 </p>
@@ -50,11 +49,11 @@ Requirements: Python 3.12+, Linux, and Docker.
 We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments.
 
 ```bash
-uv pip install securevibebench
+uv pip install -U securevibebench
 ```
 ```bash
 # Or with pip
-pip install securevibebench
+pip install -U securevibebench
 ```
 
 Task containers will be prepared when the real evaluation starts.
@@ -105,13 +104,6 @@ To comprehensively evaluate the generated code of code agents, we conduct (i) **
 > - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.
 
 
-<a id="nav-roadmap"></a>
-
-## 🗺️ Roadmap
-
-- [ ] **Cascaded LLM-as-a-judge.** Add an LLM judge as a final stage after the functional checks. Its verdicts would be reported separately so they don't change the existing scores.
-
-
 <a id="nav-citation"></a>
 
 ## 📚 Citation

{securevibebench-0.2.1 → securevibebench-0.2.2}/docs/README.md

@@ -90,4 +90,4 @@ uv build --sdist
 uv run twine check dist/*.tar.gz
 ```
 
-With pip, use `python -m unittest discover -s tests -v`, `python -m build --sdist`, and `python -m twine check dist/*.tar.gz` instead. Publish the source archive (`dist/*.tar.gz`) to PyPI. Development dependencies are installed by `tools/install.py --dev`. To refresh release resources, run `python tools/bundle_resources.py` (requires Git).
+With pip, use `python -m unittest discover -s tests -v`, `python -m build --sdist`, and `python -m twine check dist/*.tar.gz` instead. Pushing a `v<version>` tag publishes the source archive to PyPI through `.github/workflows/package.yml`. Development dependencies are installed by `tools/install.py --dev`. To refresh release resources, run `python tools/bundle_resources.py` (requires Git).

{securevibebench-0.2.1 → securevibebench-0.2.2}/docs/changeLog.md

@@ -1,5 +1,13 @@
 # Change Log
 
+## Roadmap
+
+- [ ] **Cascaded LLM-as-a-judge.** Add an LLM judge as a final stage after the functional checks. Its verdicts would be reported separately so they don't change the existing scores.
+
+## v0.2.2
+
+- Minor changes: the README install commands use `pip install -U`, the roadmap moved into this change log, and repository files were reorganized. Evaluation is unchanged.
+
 ## v0.2.1
 
 - **Bug fix: functional evaluation of task 10172 and the other Wireshark tasks.** Their test scripts run `ctest -j`, which interleaves output from concurrent tests. The parser could attach a test name to another test's status and also counted summaries such as `OK (skipped=2)` as tests. For example, a 10172 patch that passed every test scored `functional = 0`. The parser now reads each test's output on its own. Functional scores for these tasks can differ from v0.2.0 and v0.1; other tasks are unaffected. The dataset is pinned to HF `habour-preview` commit `5fa48dd`.

{securevibebench-0.2.1 → securevibebench-0.2.2}/pyproject.toml

@@ -1,16 +1,16 @@
 [build-system]
 requires = ["setuptools>=77"]
 build-backend = "build_backend"
-backend-path = ["."]
+backend-path = ["tools"]
 
 [project]
 name = "securevibebench"
-version = "0.2.1"
+version = "0.2.2"
 description = "Generate and evaluate secure coding agents on SecureVibeBench"
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
-license-files = ["LICENSE", "THIRD_PARTY_NOTICES.md"]
+license-files = ["LICENSE", "docs/THIRD_PARTY_NOTICES.md"]
 classifiers = [
     "Development Status :: 4 - Beta",
     "Intended Audience :: Science/Research",

{securevibebench-0.2.1 → securevibebench-0.2.2/securevibebench.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: securevibebench
-Version: 0.2.1
+Version: 0.2.2
 Summary: Generate and evaluate secure coding agents on SecureVibeBench
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/iCSawyer/SecureVibeBench
@@ -16,7 +16,7 @@ Classifier: Topic :: Software Development :: Testing
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-License-File: THIRD_PARTY_NOTICES.md
+License-File: docs/THIRD_PARTY_NOTICES.md
 Requires-Dist: harbor<0.26.0,>=0.25.0
 Requires-Dist: docker>=7.1.0
 Requires-Dist: pyyaml>=6.0.2
@@ -39,7 +39,7 @@ Dynamic: license-file
 
 
 > [!IMPORTANT]
-> **SecureVibeBench is now v0.2!** Just try `pip install securevibebench`!
+> **SecureVibeBench is now v0.2!** Just try `pip install -U securevibebench`!
 >
 > We are updating the evaluation criteria and workflow currently, so **the results obtained may differ from v0.1**.
 > Please report any usage issues you encouter via Issues or my [email](mailto:junkaichen@smu.edu.sg), and we will investigate and fix as soon as possible. 🙏
@@ -48,7 +48,6 @@ Dynamic: license-file
     ✨&nbsp;<a href="#nav-news">News</a>
     | 🛠️&nbsp;<a href="#nav-quick-start">Quick Start</a>
     | 🔭&nbsp;<a href="#nav-overview">Overview</a>
-    | 🗺️&nbsp;<a href="#nav-roadmap">Roadmap</a>
     | 📚&nbsp;<a href="#nav-citation">Citation</a>
     | 🙏&nbsp;<a href="#nav-acknowledgments">Acknowledgments</a>
 </p>
@@ -75,11 +74,11 @@ Requirements: Python 3.12+, Linux, and Docker.
 We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) for managing Python environments.
 
 ```bash
-uv pip install securevibebench
+uv pip install -U securevibebench
 ```
 ```bash
 # Or with pip
-pip install securevibebench
+pip install -U securevibebench
 ```
 
 Task containers will be prepared when the real evaluation starts.
@@ -130,13 +129,6 @@ To comprehensively evaluate the generated code of code agents, we conduct (i) **
 > - No other secure coding benchmark considering (i) functional correctness, (ii) PoV-based evaluation, and (iii) SAST tool-based new security issue detection.
 
 
-<a id="nav-roadmap"></a>
-
-## 🗺️ Roadmap
-
-- [ ] **Cascaded LLM-as-a-judge.** Add an LLM judge as a final stage after the functional checks. Its verdicts would be reported separately so they don't change the existing scores.
-
-
 <a id="nav-citation"></a>
 
 ## 📚 Citation

{securevibebench-0.2.1 → securevibebench-0.2.2}/securevibebench.egg-info/SOURCES.txt

@@ -1,11 +1,10 @@
 LICENSE
 MANIFEST.in
 README.md
-THIRD_PARTY_NOTICES.md
-build_backend.py
 pyproject.toml
 uv.lock
 docs/README.md
+docs/THIRD_PARTY_NOTICES.md
 docs/changeLog.md
 evaluation/__init__.py
 evaluation/__main__.py
@@ -15,12 +14,9 @@ evaluation/editable_mini.py
 evaluation/harbor.yaml
 evaluation/hf_verifier.py
 evaluation/mini-source.tar.gz
-evaluation/prepare.sh
 evaluation/resources.json
 evaluation/resources.py
 evaluation/run.py
-evaluation/run.sh
-evaluation/setup.sh
 securevibebench.egg-info/PKG-INFO
 securevibebench.egg-info/SOURCES.txt
 securevibebench.egg-info/dependency_links.txt
@@ -29,6 +25,6 @@ securevibebench.egg-info/requires.txt
 securevibebench.egg-info/top_level.txt
 tests/test_bootstrap.py
 tests/test_package.py
+tools/build_backend.py
 tools/bundle_resources.py
-tools/download_dataset.py
 tools/install.py

{securevibebench-0.2.1 → securevibebench-0.2.2/tools}/build_backend.py

@@ -14,7 +14,7 @@ import tempfile
 from setuptools import build_meta as _backend
 
 
-ROOT = Path(__file__).resolve().parent
+ROOT = Path(__file__).resolve().parents[1]
 
 
 def _prepare_resources() -> None:

{securevibebench-0.2.1 → securevibebench-0.2.2}/uv.lock

@@ -2321,7 +2321,7 @@ wheels = [
 
 [[package]]
 name = "securevibebench"
-version = "0.2.1"
+version = "0.2.2"
 source = { editable = "." }
 dependencies = [
     { name = "docker" },

securevibebench-0.2.1/evaluation/prepare.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-REPO_ROOT="$(dirname -- "$SCRIPT_DIR")"
-exec python3 "$REPO_ROOT/tools/install.py" "$@"

securevibebench-0.2.1/evaluation/run.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-if [[ "${1:-}" == eval ]]; then
-    shift
-fi
-if [[ $# -gt 0 && "$1" != -* ]]; then
-    MODEL_NAME="$1"
-    shift
-    set -- --model "$MODEL_NAME" "$@"
-fi
-exec python3 -m securevibebench_evaluation eval "$@"

securevibebench-0.2.1/evaluation/setup.sh

@@ -1,4 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
-exec bash "$SCRIPT_DIR/prepare.sh" "$@"

securevibebench-0.2.1/tools/download_dataset.py

@@ -1,13 +0,0 @@
-#!/usr/bin/env python3
-"""Download and verify the pinned HF dataset for release maintenance."""
-
-import importlib.util
-from pathlib import Path
-
-path = Path(__file__).resolve().parents[1] / "evaluation/bootstrap.py"
-spec = importlib.util.spec_from_file_location("svb_bootstrap", path)
-module = importlib.util.module_from_spec(spec)
-spec.loader.exec_module(module)
-
-if __name__ == "__main__":
-    raise SystemExit(module.main(download_only=True))