featurebench 0.1.0__tar.gz

featurebench-0.1.0/.gitignore

@@ -0,0 +1,56 @@
+# Custom
+download_cache/*
+scripts/*
+cases_filter/*
+.vscode/
+*output/
+*outputs/
+*log/
+*logs/
+featurebench/resources/repos
+featurebench/resources/constants/*
+!featurebench/resources/constants/python_example.py
+!featurebench/resources/constants/full_images.txt
+!featurebench/resources/constants/lite_images.txt
+config.toml
+tmp/
+runs/
+OpenHands/
+.cursor/*
+.github/skills/
+ms-swift/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv

featurebench-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

featurebench-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik R Narasimhan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

featurebench-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: featurebench
+Version: 0.1.0
+Summary: FeatureBench Pipeline - A test-driven data generation pipeline for building and evaluating feature-level coding benchmarks
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: datasets>=3.3.3
+Requires-Dist: docker>=5.0.3
+Requires-Dist: filelock>=3.20.0
+Requires-Dist: huggingface-hub>=0.26.5
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: matplotlib>=3.10.7
+Requires-Dist: openai>=2.6.1
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: pyarrow>=18.1.0
+Requires-Dist: pytest>=8.4.2
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: rich>=13.0.0
+Requires-Dist: scipy>=1.16.3
+Requires-Dist: seaborn>=0.13.2
+Requires-Dist: toml>=0.10.2
+Requires-Dist: tqdm>=4.67.1
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="docs/pics/logo.png" style="height: 10em" alt="logo" />
+</p>
+
+<p align="center">
+  <a href="https://arxiv.org/abs/2602.10975"><img src="https://img.shields.io/badge/arXiv-2602.10975-b31b1b.svg" alt="arXiv"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License"></a>
+  <a href="https://hub.docker.com/u/libercoders"><img src="https://img.shields.io/badge/DockerHub-Images-blue.svg" alt="DockerHub"></a>
+  <a href="https://huggingface.co/datasets/LiberCoders/FeatureBench"><img src="https://img.shields.io/badge/HuggingFace-datasets-yellow.svg" alt="HuggingFace"></a>
+  <a href="https://LiberCoders.github.io/FeatureBench/"><img src="https://img.shields.io/badge/Leaderboard-view-purple.svg" alt="Leaderboard"></a>
+</p>
+
+---
+
+FeatureBench is a test-driven data generation and evaluation pipeline for feature-level coding benchmarks.
+It provides a unified CLI to run inference, evaluation, and dataset generation.
+
+## 📰 News
+
+🎁 **2026.02.06**: We now support one-click inference for mainstream agent frameworks, including **OpenHands, Claude Code, Codex, Gemini CLI, and mini-swe-agent**. All supported agent frameworks can be found [here](featurebench/infer/agents/). We have also open-sourced the FeatureBench **data pipeline**.
+
+## 🚀 Quickstart
+
+**Prerequisites:**
+- [uv](https://docs.astral.sh/uv/getting-started/installation/) for Python environment management
+- [docker](https://docs.docker.com/engine/install/) for reproducible builds and evaluation
+
+```bash
+# pypi
+pip install featurebench
+# or uv add featurebench
+
+# local
+git clone https://github.com/LiberCoders/FeatureBench.git
+cd FeatureBench
+uv sync
+```
+
+**Configure:**
+```bash
+cp config_example.toml config.toml
+```
+See [docs/config.md](docs/config.md) for a comprehensive reference (harness, infer, data pipeline) with examples.
+
+**Optional: pre-pull images to reduce network variance:**
+```bash
+fb pull --mode lite                 # lite split image list (13 images)
+fb pull --mode full                 # full split image list (24 images)
+fb pull --mode /path/to/images.txt  # one image name per line
+
+# full list: featurebench/resources/constants/full_images.txt
+# lite list: featurebench/resources/constants/lite_images.txt
+```
+
+**Run inference:**
+```bash
+fb infer \
+    --config-path config.toml \
+    --agent mini_swe_agent \
+    --model openai/qwen3-coder-480b-a35b-instruct \
+    --split lite
+```
+
+**Run evaluation:**
+```bash
+fb eval \
+    -p runs/<timestamp>/output.jsonl \
+    --split lite
+```
+
+## 🧭 CLI Overview
+
+`fb` provides three core commands:
+- `fb infer` runs `featurebench.infer.run_infer` (docs: [docs/infer_cli_arg.md](docs/infer_cli_arg.md))
+- `fb eval` runs `featurebench.harness.run_evaluation` (docs: [docs/harness_cli_arg.md](docs/harness_cli_arg.md))
+- `fb data` runs `featurebench.pipeline` (docs: [docs/pipeline.md](docs/pipeline.md))
+
+## ✍️ Citation
+
+If you found FeatureBench useful, please cite us as:
+
+```bibtex
+@misc{zhou2026featurebenchbenchmarkingagenticcoding,
+      title={FeatureBench: Benchmarking Agentic Coding for Complex Feature Development}, 
+      author={Qixing Zhou and Jiacheng Zhang and Haiyang Wang and Rui Hao and Jiahe Wang and Minghao Han and Yuxue Yang and Shuzhe Wu and Feiyang Pan and Lue Fan and Dandan Tu and Zhaoxiang Zhang},
+      year={2026},
+      eprint={2602.10975},
+      archivePrefix={arXiv},
+      primaryClass={cs.SE},
+      url={https://arxiv.org/abs/2602.10975}, 
+}
+```
+
+## 📧 Contact
+
+If you have any questions, feel free to contact [qixingzhou1125@gmail.com](mailto:qixingzhou1125@gmail.com) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).

featurebench-0.1.0/README.md

@@ -0,0 +1,96 @@
+<p align="center">
+  <img src="docs/pics/logo.png" style="height: 10em" alt="logo" />
+</p>
+
+<p align="center">
+  <a href="https://arxiv.org/abs/2602.10975"><img src="https://img.shields.io/badge/arXiv-2602.10975-b31b1b.svg" alt="arXiv"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="License"></a>
+  <a href="https://hub.docker.com/u/libercoders"><img src="https://img.shields.io/badge/DockerHub-Images-blue.svg" alt="DockerHub"></a>
+  <a href="https://huggingface.co/datasets/LiberCoders/FeatureBench"><img src="https://img.shields.io/badge/HuggingFace-datasets-yellow.svg" alt="HuggingFace"></a>
+  <a href="https://LiberCoders.github.io/FeatureBench/"><img src="https://img.shields.io/badge/Leaderboard-view-purple.svg" alt="Leaderboard"></a>
+</p>
+
+---
+
+FeatureBench is a test-driven data generation and evaluation pipeline for feature-level coding benchmarks.
+It provides a unified CLI to run inference, evaluation, and dataset generation.
+
+## 📰 News
+
+🎁 **2026.02.06**: We now support one-click inference for mainstream agent frameworks, including **OpenHands, Claude Code, Codex, Gemini CLI, and mini-swe-agent**. All supported agent frameworks can be found [here](featurebench/infer/agents/). We have also open-sourced the FeatureBench **data pipeline**.
+
+## 🚀 Quickstart
+
+**Prerequisites:**
+- [uv](https://docs.astral.sh/uv/getting-started/installation/) for Python environment management
+- [docker](https://docs.docker.com/engine/install/) for reproducible builds and evaluation
+
+```bash
+# pypi
+pip install featurebench
+# or uv add featurebench
+
+# local
+git clone https://github.com/LiberCoders/FeatureBench.git
+cd FeatureBench
+uv sync
+```
+
+**Configure:**
+```bash
+cp config_example.toml config.toml
+```
+See [docs/config.md](docs/config.md) for a comprehensive reference (harness, infer, data pipeline) with examples.
+
+**Optional: pre-pull images to reduce network variance:**
+```bash
+fb pull --mode lite                 # lite split image list (13 images)
+fb pull --mode full                 # full split image list (24 images)
+fb pull --mode /path/to/images.txt  # one image name per line
+
+# full list: featurebench/resources/constants/full_images.txt
+# lite list: featurebench/resources/constants/lite_images.txt
+```
+
+**Run inference:**
+```bash
+fb infer \
+    --config-path config.toml \
+    --agent mini_swe_agent \
+    --model openai/qwen3-coder-480b-a35b-instruct \
+    --split lite
+```
+
+**Run evaluation:**
+```bash
+fb eval \
+    -p runs/<timestamp>/output.jsonl \
+    --split lite
+```
+
+## 🧭 CLI Overview
+
+`fb` provides three core commands:
+- `fb infer` runs `featurebench.infer.run_infer` (docs: [docs/infer_cli_arg.md](docs/infer_cli_arg.md))
+- `fb eval` runs `featurebench.harness.run_evaluation` (docs: [docs/harness_cli_arg.md](docs/harness_cli_arg.md))
+- `fb data` runs `featurebench.pipeline` (docs: [docs/pipeline.md](docs/pipeline.md))
+
+## ✍️ Citation
+
+If you found FeatureBench useful, please cite us as:
+
+```bibtex
+@misc{zhou2026featurebenchbenchmarkingagenticcoding,
+      title={FeatureBench: Benchmarking Agentic Coding for Complex Feature Development}, 
+      author={Qixing Zhou and Jiacheng Zhang and Haiyang Wang and Rui Hao and Jiahe Wang and Minghao Han and Yuxue Yang and Shuzhe Wu and Feiyang Pan and Lue Fan and Dandan Tu and Zhaoxiang Zhang},
+      year={2026},
+      eprint={2602.10975},
+      archivePrefix={arXiv},
+      primaryClass={cs.SE},
+      url={https://arxiv.org/abs/2602.10975}, 
+}
+```
+
+## 📧 Contact
+
+If you have any questions, feel free to contact [qixingzhou1125@gmail.com](mailto:qixingzhou1125@gmail.com) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).

featurebench-0.1.0/config_example.toml

@@ -0,0 +1,106 @@
+# ==============================================================================
+# Global Environment Variables (injected into each container at runtime)
+# ==============================================================================
+[env_vars]
+HF_ENDPOINT=""
+HF_TOKEN=""
+GITHUB_TOKEN=""
+
+# ==============================================================================
+# DataPipeline Configuration
+# ==============================================================================
+
+# LLM Basic Configuration
+[llm_config]
+llm_name="" # e.g., "azure-gpt-5-fy0828" / "local"
+llm_temperature=0.0
+llm_max_tokens=65536    # Maximum output tokens (-1 = no limit)
+timeout=180
+
+[llm.azure-gpt-5-fy0828]
+model = "azure/gpt-5-fy0828"
+api_key = ""
+base_url = ""
+
+[llm.claude-sonnet-4-20250514]
+model = "claude-sonnet-4-20250514"
+api_key = ""
+base_url = [
+    "",
+    "",
+]
+
+[llm.local]
+backend = "vllm"                                     # Backend type: vllm
+api_key = ""                                         # Optional: API key passed to vLLM server
+base_url = "http://localhost:8080/v1"                # vLLM server URL
+
+# ==============================================================================
+# Inference Agent Configuration
+# ==============================================================================
+
+# Inference runtime settings
+[infer]
+# Optional: Host cache directory for agent downloads; will be mounted to /download inside containers
+# Set to empty to disable cache mount (each container will re-download/install)
+# Use an absolute path here
+download_cache_dir="/path/to/FeatureBench/download_cache"
+
+# OpenHands Agent Configuration
+[infer_config.openhands]
+# Required: LLM API Key (supports OpenAI/Anthropic/Azure, etc.)
+LLM_API_KEY = ""
+# Optional: LLM API Base URL
+LLM_BASE_URL = ""
+# Optional: Lock OpenHands version (leave empty to use latest)
+OPENHANDS_VERSION = "0.62.0"
+# Optional: Reasoning effort for OpenAI o-series models
+LLM_REASONING_EFFORT = ""
+# Azure API Version (required for Azure models only)
+LLM_API_VERSION = ""
+# Optional: OpenHands agent max iterations (step limit). Upstream default is 500.
+OPENHANDS_MAX_ITERATIONS = ""
+# Optional: Save llm completions if true
+SAVE_COMPLETIONS = false
+# Optional: Enable OpenHands condensation (default false). Set true to enable condensers.
+ENABLE_CONDENSER = false
+# Optional: Render infer.log mode (compact|full)
+INFER_LOG_RENDER_MODE = "full"
+
+# Claude Code Agent Configuration
+[infer_config.claude_code]
+# Required: Anthropic API Key
+ANTHROPIC_API_KEY = ""
+# Optional: Custom API Base URL
+ANTHROPIC_BASE_URL = ""
+# Optional: Lock Claude Code version (leave empty to use latest)
+CLAUDE_CODE_VERSION = ""
+
+# Gemini CLI Agent Configuration
+[infer_config.gemini_cli]
+GEMINI_API_KEY = ""
+GOOGLE_GEMINI_BASE_URL = ""
+# Optional: Lock Gemini CLI version (leave empty to use latest)
+GEMINI_CLI_VERSION = ""
+
+# Codex Agent Configuration (Azure OpenAI supported)
+[infer_config.codex]
+# Required: OpenAI API Key
+OPENAI_API_KEY = ""
+# Optional: Custom API Base URL
+OPENAI_BASE_URL = ""
+# Optional: Reasoning effort level for Codex reasoning models; empty defaults to "medium"
+CODEX_REASONING_EFFORT = ""
+# Optional: Lock Codex version (leave empty to use latest)
+CODEX_VERSION = ""
+
+# mini_swe_agent Configuration
+[infer_config.mini_swe_agent]
+# Required: Unified mini_swe_agent API key
+MSWEA_API_KEY = ""
+# Optional: Base URL passed through to provider envs
+MSWEA_BASE_URL = ""
+# Optional: Cost tracking mode (default behavior in FB adapter is ignore_errors)
+MSWEA_COST_TRACKING = ""
+# Optional: Lock mini_swe_agent version (leave empty to use latest)
+MINI_SWE_AGENT_VERSION = ""

featurebench-0.1.0/docs/config.md

@@ -0,0 +1,208 @@
+# FeatureBench Configuration Guide: config.toml
+
+FeatureBench has three major workflows:
+
+- **harness (evaluation)**: runs tests on inference outputs and produces a report.
+- **infer (inference)**: runs different agents on FeatureBench benchmark tasks to generate patches.
+- **data pipeline (dataset generation)**: builds FeatureBench-style tasks automatically from real repositories.
+
+The repo-root `config.toml` is the single configuration file that manages what all three workflows need at runtime.
+
+This guide uses the example configuration [config_example.toml](../config_example.toml) as the source of truth. It explains every field and how to use it, organized into **harness / infer / data pipeline**, and provides examples.
+
+## Quickstart
+
+Copy the example config:
+
+```bash
+cp config_example.toml config.toml
+```
+
+### [env_vars] (global environment variables used by harness / infer / pipeline)
+
+`[env_vars]` is the global environment variable section. Values here will be injected into every runnable container.
+
+Example:
+
+```toml
+[env_vars]
+HF_ENDPOINT = ""
+HF_TOKEN = ""
+GITHUB_TOKEN = ""
+```
+
+---
+
+## A. Harness
+
+The core idea of harness is: read the `output.jsonl` produced by infer, apply each patch to the corresponding repository image, run tests, and generate evaluation results.
+
+When harness loads the HuggingFace dataset during evaluation, it uses `HF_ENDPOINT` from `[env_vars]`.
+
+---
+
+## B. Infer
+
+The core idea of infer is: run an agent inside a container (e.g. `claude_code/openhands/codex/mini_swe_agent/...`) and let it generate a patch for each task.
+
+How infer reads `config.toml`:
+
+- Load global `[env_vars]` first
+- Then load `[infer]` (inference runtime settings)
+- Then load the current agent’s `[infer_config.<agent>]`
+- Merge `[env_vars]` and `[infer_config.<agent>]`, and inject them into the inference container as environment variables
+
+### B1. [infer]
+
+```toml
+[infer]
+download_cache_dir = "/abs/path/to/FeatureBench/download_cache"
+```
+
+- `download_cache_dir`: host-side cache directory that will be mounted into the container at `/download`.
+  - Purpose: speed up agent installs/downloads (e.g. npm cache).
+  - Empty means no mount (each container downloads on its own; usually slower).
+
+### B2. [infer_config.openhands]
+
+```toml
+[infer_config.openhands]
+LLM_API_KEY = ""
+LLM_BASE_URL = ""     # Optional
+LLM_API_VERSION = ""  # Optional: Azure only
+OPENHANDS_VERSION = "" # Optional: pin OpenHands version (empty usually means default/latest)
+SAVE_COMPLETIONS = false      # Optional: whether to save LLM completions (true/false)
+INFER_LOG_RENDER_MODE = "compact" # Optional: compact|full for infer.log rendering
+
+LLM_REASONING_EFFORT = ""     # Optional: Reasoning effort for OpenAI o-series models
+OPENHANDS_MAX_ITERATIONS = "" # Optional: OpenHands agent max iterations (step limit). Upstream default is 500.
+
+```
+
+Example 1: OpenHands + OpenAI
+
+```toml
+[infer_config.openhands]
+LLM_API_KEY = "<your-openai-api-key>"
+LLM_BASE_URL = ""            # Fill if needed
+LLM_API_VERSION = ""
+OPENHANDS_VERSION = "0.62.0"
+SAVE_COMPLETIONS = false
+```
+
+Example 2: OpenHands + Azure OpenAI
+
+```toml
+[infer_config.openhands]
+LLM_API_KEY = "<your-azure-api-key>"
+LLM_BASE_URL = "https://<resource>.openai.azure.com"
+LLM_API_VERSION = "2024-xx-xx"
+OPENHANDS_VERSION = "0.62.0"
+SAVE_COMPLETIONS = false
+```
+
+### B3. [infer_config.claude_code]
+
+```toml
+[infer_config.claude_code]
+ANTHROPIC_API_KEY = ""    # Required
+ANTHROPIC_BASE_URL = ""   # Optional
+```
+
+### B4. [infer_config.gemini_cli]
+
+```toml
+[infer_config.gemini_cli]
+GEMINI_API_KEY = ""         # Required
+GOOGLE_GEMINI_BASE_URL = "" # Optional
+```
+
+### B5. [infer_config.codex]
+
+```toml
+[infer_config.codex]
+OPENAI_API_KEY = ""         # Required
+OPENAI_BASE_URL = ""        # Optional
+CODEX_REASONING_EFFORT = "" # Optional: empty defaults to medium
+```
+
+Example 1: Codex + OpenAI
+
+```toml
+[infer_config.codex]
+OPENAI_API_KEY = "<your-openai-api-key>"
+OPENAI_BASE_URL = ""
+CODEX_REASONING_EFFORT = "medium"
+```
+
+Example 2: Codex + Azure OpenAI
+
+```toml
+[infer_config.codex]
+OPENAI_API_KEY = "<your-azure-api-key>"
+OPENAI_BASE_URL = "https://<resource>.openai.azure.com"
+CODEX_REASONING_EFFORT = "high"
+```
+
+### B6. [infer_config.mini_swe_agent]
+
+```toml
+[infer_config.mini_swe_agent]
+MSWEA_API_KEY = ""
+MSWEA_BASE_URL = ""
+MSWEA_COST_TRACKING = ""
+MINI_SWE_AGENT_VERSION = ""
+```
+
+---
+
+## C. Data Pipeline
+
+the core idea of the data pipeline is: call an LLM + access repositories/platforms to automatically generate or process data (tasks, prompts, test files, etc.).
+
+It mainly depends on two types of config:
+
+- `[env_vars]`: platform/repo access related
+- `[llm_config]` + `[llm.<name>]`: select an LLM configuration for the pipeline
+
+### C1. [llm_config]
+
+```toml
+[llm_config]
+llm_name = ""         # Select the name of one [llm.<name>]
+llm_temperature = 0.0  # Default temperature (some models/backends may ignore it, e.g. some GPT-5 paths)
+llm_max_tokens = 65536 # Max output tokens; set -1 to "not send max_tokens/max_completion_tokens", backend decides the default behavior
+timeout = 180          # Request timeout (seconds)
+```
+
+### C2. [llm.\<name>]
+
+Example (Azure OpenAI):
+
+```toml
+[llm.azure-o3-fy1]
+model = "azure/o3-fy1"
+api_key = "..."
+base_url = "https://<resource>.openai.azure.com"
+api_version = "2025-04-01-preview"
+```
+
+Example (Anthropic):
+
+```toml
+[llm.claude-sonnet-4-20250514]
+model = "claude-sonnet-4-20250514"
+api_key = "..."
+base_url = [
+  "https://...",
+  "https://..."
+]   # Supports multiple URLs; will retry the list on failure
+```
+
+Example (local vLLM):
+
+```toml
+[llm.local]
+backend = "vllm"
+base_url = "http://localhost:8080/v1"
+```