featurebench 0.2.1__tar.gz → 0.2.2__tar.gz

{featurebench-0.2.1 → featurebench-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: featurebench
-Version: 0.2.1
+Version: 0.2.2
 Summary: FeatureBench Pipeline - A test-driven data generation pipeline for building and evaluating feature-level coding benchmarks
 License-File: LICENSE
 Requires-Python: >=3.12
@@ -41,6 +41,8 @@ It provides a unified CLI to run inference, evaluation, and dataset generation.
 
 ## 📰 News
 
+📦 **2026.08.24**: We released **FeatureBench Dataset V1.1**, primarily correcting inconsistencies and omissions in task problem statements.
+
 📊 **2026.05.18**: We added **lite split** evaluation results for frontier models including **GPT-5.5, Claude Opus 4.7, DeepSeek-V4, GLM-5.1, Kimi-2.6, Mimo-V2.5-Pro**, and more to the leaderboard.
 
 🚀 **2026.03.27**: We released the **fast split** containing 100 instances (a subset of full split). These instances require no GPU and are optimized for rapid evaluation. On an Intel Xeon Platinum 8457C with 944GB RAM, the average evaluation time per instance using gold patches is **57.2 seconds**.
@@ -121,6 +123,7 @@ fb pull --mode /path/to/images.txt  # one image name per line
 ```bash
 fb infer \
     --config-path config.toml \
+    --data-version v1.1 \
     --agent mini_swe_agent \
     --model openai/qwen3-coder-480b-a35b-instruct \
     --split fast
@@ -156,4 +159,4 @@ If you found FeatureBench useful, please cite us as:
 
 ## 📧 Contact
 
-If you have any questions, feel free to contact [qixingzhou1125@gmail.com](mailto:qixingzhou1125@gmail.com) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).
+If you have any questions, feel free to contact [zhouqixing@stu.cqu.edu.cn](mailto:zhouqixing@stu.cqu.edu.cn) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).

{featurebench-0.2.1 → featurebench-0.2.2}/README.md

@@ -17,6 +17,8 @@ It provides a unified CLI to run inference, evaluation, and dataset generation.
 
 ## 📰 News
 
+📦 **2026.08.24**: We released **FeatureBench Dataset V1.1**, primarily correcting inconsistencies and omissions in task problem statements.
+
 📊 **2026.05.18**: We added **lite split** evaluation results for frontier models including **GPT-5.5, Claude Opus 4.7, DeepSeek-V4, GLM-5.1, Kimi-2.6, Mimo-V2.5-Pro**, and more to the leaderboard.
 
 🚀 **2026.03.27**: We released the **fast split** containing 100 instances (a subset of full split). These instances require no GPU and are optimized for rapid evaluation. On an Intel Xeon Platinum 8457C with 944GB RAM, the average evaluation time per instance using gold patches is **57.2 seconds**.
@@ -97,6 +99,7 @@ fb pull --mode /path/to/images.txt  # one image name per line
 ```bash
 fb infer \
     --config-path config.toml \
+    --data-version v1.1 \
     --agent mini_swe_agent \
     --model openai/qwen3-coder-480b-a35b-instruct \
     --split fast
@@ -132,4 +135,4 @@ If you found FeatureBench useful, please cite us as:
 
 ## 📧 Contact
 
-If you have any questions, feel free to contact [qixingzhou1125@gmail.com](mailto:qixingzhou1125@gmail.com) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).
+If you have any questions, feel free to contact [zhouqixing@stu.cqu.edu.cn](mailto:zhouqixing@stu.cqu.edu.cn) or [zjcheng2022@gmail.com](mailto:zjcheng2022@gmail.com).

{featurebench-0.2.1 → featurebench-0.2.2}/config_example.toml

@@ -6,6 +6,14 @@ HF_ENDPOINT=""
 HF_TOKEN=""
 GITHUB_TOKEN=""
 
+# ==============================================================================
+# Hugging Face Dataset
+# ==============================================================================
+[dataset]
+# Dataset tag, branch, or commit SHA used by both `fb infer` and `fb eval`.
+# The CLI flag `--data-version` overrides this value.
+revision="v1.1"
+
 # ==============================================================================
 # DataPipeline Configuration
 # ==============================================================================
@@ -109,4 +117,4 @@ MSWEA_BASE_URL = ""
 # Optional: Cost tracking mode (default behavior in FB adapter is ignore_errors)
 MSWEA_COST_TRACKING = ""
 # Optional: Lock mini_swe_agent version (leave empty to use latest)
-MINI_SWE_AGENT_VERSION = ""
+MINI_SWE_AGENT_VERSION = ""

{featurebench-0.2.1 → featurebench-0.2.2}/docs/config.md

@@ -31,13 +31,25 @@ HF_TOKEN = ""
 GITHUB_TOKEN = ""
 ```
 
+### [dataset] (dataset version used by harness / infer)
+
+```toml
+[dataset]
+revision = "v1.1"
+```
+
+- `revision`: Hugging Face dataset tag, branch, or commit SHA used by both
+  `fb infer` and `fb eval`. The default is `v1.1`.
+- `--data-version` overrides this value for a single command.
+
 ---
 
 ## A. Harness
 
 The core idea of harness is: read the `output.jsonl` produced by infer, apply each patch to the corresponding repository image, run tests, and generate evaluation results.
 
-When harness loads the HuggingFace dataset during evaluation, it uses `HF_ENDPOINT` from `[env_vars]`.
+When harness loads the HuggingFace dataset during evaluation, it uses `HF_ENDPOINT`
+from `[env_vars]` and the pinned revision from `[dataset]`.
 
 ---
 
@@ -208,4 +220,4 @@ Example (local vLLM):
 [llm.local]
 backend = "vllm"
 base_url = "http://localhost:8080/v1"
-```
+```

{featurebench-0.2.1 → featurebench-0.2.2}/docs/harness_cli_arg.md

@@ -51,6 +51,11 @@ fb eval \
   HuggingFace dataset repo name (e.g., `LiberCoders/FeatureBench`).  
   Default: `LiberCoders/FeatureBench`.
 
+- `--data-version`
+  Hugging Face dataset tag, branch, or commit SHA.
+  Normally inferred from the inference run's `run_metadata.json`.
+  If unavailable, falls back to `[dataset].revision` or `v1.1`.
+
 - `--split`  
   Dataset split name (e.g., `lite`, `full`).  
   Default: `full`.

{featurebench-0.2.1 → featurebench-0.2.2}/docs/infer_cli_arg.md

@@ -6,6 +6,7 @@ This document describes all CLI arguments supported by `fb infer`.
 
 ```bash
 fb infer \
+  --data-version v1.1 \
   --agent gemini_cli \
   --model gemini-3-pro-preview
 ```
@@ -53,6 +54,12 @@ flags can override metadata (see the argument list below).
   HuggingFace dataset repo name (e.g., `LiberCoders/FeatureBench`).  
   Default: `LiberCoders/FeatureBench` in non-resume mode.
 
+- `--data-version`
+  Hugging Face dataset tag, branch, or commit SHA.
+  Overrides `[dataset].revision` from `config.toml`; default: `v1.1`.
+  Saved into `run_metadata.json`; resume mode keeps the recorded dataset version.
+  This is separate from `--version`, which controls the agent software version.
+
 - `--split`  
   Dataset split name (e.g., `lite`, `full`).  
   Default: `full` in non-resume mode. In resume mode, uses metadata.
@@ -166,4 +173,4 @@ runs/{timestamp}/
             ├── infer.log         # Agent execution log
             ├── run.log           # Runtime log
             └── patch.diff        # Generated patch
-```
+```

{featurebench-0.2.1 → featurebench-0.2.2}/featurebench/cli.py

@@ -31,7 +31,7 @@ def _print_help(stream=None) -> None:
         "  data   Run data pipeline (supports featurebench-pipeline args)\n"
         "\n"
         "Examples:\n"
-        "  fb infer --agent mini_swe_agent --model openai/gpt-4o --split fast\n"
+        "  fb infer --data-version v1.1 --agent mini_swe_agent --model openai/gpt-4o --split fast\n"
         "  fb eval --predictions-path runs/xxx/output.jsonl --split fast\n"
         "  fb pull --mode fast\n"
         "  fb data --config-path constants/python_new.py --output-dir runs/data\n"
@@ -128,4 +128,4 @@ def main(argv: Sequence[str] | None = None) -> int:
 
 
 if __name__ == "__main__":
-    raise SystemExit(main())
+    raise SystemExit(main())

{featurebench-0.2.1 → featurebench-0.2.2}/featurebench/harness/run_evaluation.py

@@ -74,7 +74,13 @@ from featurebench.harness.utils import (
     parse_repo_settings,
 )
 
-from featurebench.infer.config import InferConfigLoader
+from featurebench.infer.config import (
+    DEFAULT_DATASET_REVISION,
+    InferConfigLoader,
+    configure_hf_access,
+    resolve_hf_dataset_revision,
+    verify_loaded_dataset_revision,
+)
 from featurebench.infer.gpu_scheduler import GpuLease, GpuScheduler, detect_host_gpu_ids, parse_gpu_id_list
 
 # Global console for rich output
@@ -747,6 +753,7 @@ def load_dataset_from_hf(
     split: str,
     dataset: str,
     config_path: str | None = None,
+    dataset_revision: str | None = None,
 ) -> pd.DataFrame:
     """
     Load dataset from HuggingFace using config.toml settings.
@@ -766,21 +773,39 @@ def load_dataset_from_hf(
         resolved_path = Path(config_path).expanduser() if config_path else None
         config_loader = InferConfigLoader(resolved_path)
         env_vars = config_loader.env_vars
-        hf_token = env_vars.get("HF_TOKEN")
+        hf_token, hf_endpoint = configure_hf_access(env_vars)
+        configured_revision = config_loader.get_dataset_revision()
     except FileNotFoundError:
         console.print("[yellow]Warning: config.toml not found, using defaults[/]")
-        hf_token = os.environ.get('HF_TOKEN', None)
+        hf_token, hf_endpoint = configure_hf_access({})
+        configured_revision = DEFAULT_DATASET_REVISION
+
+    requested_revision = str(dataset_revision or "").strip()
+    effective_revision = requested_revision or configured_revision
 
     console.print(f"[bold blue]Loading dataset from HuggingFace...[/]")
     console.print(f"[dim]Repository: {dataset}[/]")
     console.print(f"[dim]Split: {split}[/]")
+    console.print(f"[dim]Version: {effective_revision}[/]")
 
     if hf_token:
         console.print("[dim]Using HuggingFace token from config[/]")
 
     try:
+        resolved_revision = resolve_hf_dataset_revision(
+            dataset,
+            effective_revision,
+            hf_token,
+            hf_endpoint,
+        )
         # Load the specified split
-        hf_dataset = load_dataset(dataset, split=split, token=hf_token)
+        hf_dataset = load_dataset(
+            dataset,
+            split=split,
+            token=hf_token,
+            revision=resolved_revision,
+        )
+        verify_loaded_dataset_revision(hf_dataset, resolved_revision)
         df = pd.DataFrame(hf_dataset)
 
         # Determine level from instance_id's last segment (e.g., "xxx.lv1" -> level 1)
@@ -809,6 +834,28 @@ def load_dataset_from_hf(
         raise
 
 
+def resolve_eval_dataset_revision(
+    cli_revision: str | None,
+    run_metadata: dict | None,
+) -> str | None:
+    """Resolve the dataset revision for evaluation.
+
+    An explicit CLI value wins. Otherwise, evaluation follows the revision
+    recorded by inference. Runs created before revision tracking belong to the
+    original v1.0 dataset. When no run metadata is available, the dataset
+    loader falls back to the configured/default revision.
+    """
+    requested_revision = str(cli_revision or "").strip()
+    if requested_revision:
+        return requested_revision
+
+    if run_metadata:
+        recorded_revision = str(run_metadata.get("dataset_revision") or "").strip()
+        return recorded_revision or "v1.0"
+
+    return None
+
+
 def parse_args() -> argparse.Namespace:
     """Parse command line arguments."""
     parser = argparse.ArgumentParser(
@@ -874,6 +921,17 @@ def parse_args() -> argparse.Namespace:
         help="HuggingFace dataset repo name (e.g., 'LiberCoders/FeatureBench')",
     )
 
+    parser.add_argument(
+        "--data-version",
+        dest="dataset_version",
+        type=str,
+        default=None,
+        help=(
+            "Hugging Face dataset version (tag, branch, or commit SHA). "
+            "Default: infer from run_metadata.json, then [dataset].revision, then v1.1"
+        ),
+    )
+
     parser.add_argument(
         "--split",
         type=str,
@@ -956,15 +1014,23 @@ def main():
     # we keep FAIL_TO_PASS test files visible during Level 1 patch application to avoid
     # "patch touches deleted test file" apply failures.
     white_enabled = False
+    run_metadata = None
     try:
         metadata_path = output_dir / "run_metadata.json"
-        if metadata_path.exists():
+        if not use_gold_patch and metadata_path.exists():
             with open(metadata_path, "r", encoding="utf-8") as f:
                 meta = json.load(f)
-            if isinstance(meta, dict) and meta.get("white_box") is True:
-                white_enabled = True
+            if isinstance(meta, dict):
+                run_metadata = meta
+                white_enabled = meta.get("white_box") is True
     except Exception:
         white_enabled = False
+        run_metadata = None
+
+    dataset_revision = resolve_eval_dataset_revision(
+        args.dataset_version,
+        run_metadata,
+    )
 
     # Print startup banner
     console.print()
@@ -996,7 +1062,13 @@ def main():
     console.print()
 
     # Load dataset
-    dataset = load_dataset_from_hf(console, args.split, args.dataset, args.config_path)
+    dataset = load_dataset_from_hf(
+        console,
+        args.split,
+        args.dataset,
+        args.config_path,
+        dataset_revision,
+    )
 
     # Load predictions
     console.print(f"[bold blue]Loading predictions...[/]")
@@ -1221,4 +1293,4 @@ def main():
 
 
 if __name__ == "__main__":
-    raise SystemExit(main())
+    raise SystemExit(main())

{featurebench-0.2.1 → featurebench-0.2.2}/featurebench/infer/agents/mini_swe_agent.py

@@ -109,6 +109,14 @@ echo "mini-swe-agent installation complete"
             )
         except Exception:
             pass
+        try:
+            self.cm.copy_from_container(
+                container,
+                "/root/.config/mini-swe-agent/last_mini_run.traj.json",
+                log_dir / "traj.json",
+            )
+        except Exception:
+            pass
         return True
 
     def failure_hook(self, container, log_file: Path) -> None:
@@ -122,6 +130,14 @@ echo "mini-swe-agent installation complete"
             )
         except Exception:
             pass
+        try:
+            self.cm.copy_from_container(
+                container,
+                "/root/.config/mini-swe-agent/last_mini_run.traj.json",
+                log_dir / "traj.json",
+            )
+        except Exception:
+            pass
 
     def get_run_command(self, instruction: str) -> str:
         """Get the command to run mini-swe-agent."""

{featurebench-0.2.1 → featurebench-0.2.2}/featurebench/infer/config.py

@@ -5,8 +5,75 @@ Configuration loader for the inference module.
 import logging
 import os
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple, Union
+
 import toml
+from huggingface_hub import HfApi
+
+
+DEFAULT_DATASET_REVISION = "v1.1"
+
+
+def configure_hf_access(env_vars: Dict[str, Any]) -> Tuple[Union[str, bool, None], Optional[str]]:
+    """Apply configured Hub access settings and return normalized values."""
+    configured_endpoint = str(env_vars.get("HF_ENDPOINT") or "").strip()
+    endpoint = configured_endpoint or str(os.environ.get("HF_ENDPOINT") or "").strip()
+    if endpoint:
+        os.environ["HF_ENDPOINT"] = endpoint
+        # ``datasets.config.HF_ENDPOINT`` is initialized at import time, so update
+        # it as well when callers load their config after importing datasets.
+        import datasets
+
+        datasets.config.HF_ENDPOINT = endpoint
+
+    if "HF_TOKEN" in env_vars:
+        configured_token = str(env_vars.get("HF_TOKEN") or "").strip()
+        token: Union[str, bool, None] = configured_token or False
+    else:
+        token = str(os.environ.get("HF_TOKEN") or "").strip() or None
+
+    return token, endpoint or None
+
+
+def resolve_hf_dataset_revision(
+    dataset: str,
+    revision: str,
+    token: Union[str, bool, None],
+    endpoint: Optional[str],
+) -> str:
+    """Resolve a Hub dataset revision to its immutable commit SHA."""
+    try:
+        info = HfApi(endpoint=endpoint, token=token).dataset_info(
+            repo_id=dataset,
+            revision=revision,
+        )
+    except Exception as exc:
+        raise RuntimeError(
+            f"Unable to resolve dataset revision '{revision}' for '{dataset}'. "
+            "Refusing to use an unverified cached dataset."
+        ) from exc
+
+    resolved_revision = str(info.sha or "").strip()
+    if not resolved_revision:
+        raise RuntimeError(
+            f"Hugging Face returned no commit SHA for '{dataset}' at '{revision}'"
+        )
+    return resolved_revision
+
+
+def verify_loaded_dataset_revision(dataset: Any, resolved_revision: str) -> None:
+    """Reject Hugging Face's fallback to a cache from another revision."""
+    cache_files = getattr(dataset, "cache_files", None) or []
+    cache_paths = [
+        Path(item["filename"])
+        for item in cache_files
+        if isinstance(item, dict) and item.get("filename")
+    ]
+    if cache_paths and any(resolved_revision not in path.parts for path in cache_paths):
+        raise RuntimeError(
+            f"Loaded dataset cache does not match resolved commit {resolved_revision}. "
+            "Refusing to use a different dataset version."
+        )
 
 
 class InferConfigLoader:
@@ -75,6 +142,12 @@ class InferConfigLoader:
             return Path(path).expanduser()
         return None
 
+    def get_dataset_revision(self) -> str:
+        """Get the pinned Hugging Face dataset revision."""
+        dataset_cfg = self._config.get("dataset", {}) or {}
+        revision = str(dataset_cfg.get("revision") or "").strip()
+        return revision or DEFAULT_DATASET_REVISION
+
     def get_agent_env_vars(self, agent_name: str) -> Dict[str, str]:
         """
         Get environment variables for a specific agent.
@@ -153,20 +226,15 @@ class DatasetLoader:
     
     def _setup_hf_env(self) -> None:
         """Set up HuggingFace environment variables."""
-        env_vars = self.config.env_vars
-        
-        if env_vars.get("HF_ENDPOINT"):
-            os.environ["HF_ENDPOINT"] = env_vars["HF_ENDPOINT"]
-        
-        if env_vars.get("HF_TOKEN"):
-            os.environ["HF_TOKEN"] = env_vars["HF_TOKEN"]
+        configure_hf_access(self.config.env_vars)
     
     def load_dataset(
         self,
         dataset: str,
         split: Optional[str] = None,
         levels: Optional[List[int]] = None,
-        task_ids: Optional[List[str]] = None
+        task_ids: Optional[List[str]] = None,
+        revision: Optional[str] = None,
     ) -> List[Dict[str, Any]]:
         """
         Load dataset from HuggingFace.
@@ -177,6 +245,8 @@ class DatasetLoader:
             levels: List of levels to filter (1, 2). If None, loads all levels.
                    Level is determined by the last segment of instance_id (e.g., "lv1", "lv2").
             task_ids: Specific task IDs to filter. If None, loads all tasks.
+            revision: Hugging Face dataset revision (tag, branch, or commit SHA).
+                      If omitted, uses ``[dataset].revision`` or ``v1.1``.
             
         Returns:
             List of task instances as dictionaries
@@ -193,7 +263,8 @@ class DatasetLoader:
         from datasets import load_dataset
         
         env_vars = self.config.env_vars
-        hf_token = env_vars.get("HF_TOKEN")
+        hf_token, hf_endpoint = configure_hf_access(env_vars)
+        dataset_revision = str(revision or "").strip() or self.config.get_dataset_revision()
 
         dataset_name = str(dataset or "").strip()
         if not dataset_name:
@@ -205,11 +276,19 @@ class DatasetLoader:
         all_instances = []
         
         try:
+            resolved_revision = resolve_hf_dataset_revision(
+                dataset_name,
+                dataset_revision,
+                hf_token,
+                hf_endpoint,
+            )
             dataset = load_dataset(
                 dataset_name,
                 split=split,
-                token=hf_token
+                token=hf_token,
+                revision=resolved_revision,
             )
+            verify_loaded_dataset_revision(dataset, resolved_revision)
             
             for item in dataset:
                 item_dict = dict(item)

{featurebench-0.2.1 → featurebench-0.2.2}/featurebench/infer/models.py

@@ -152,6 +152,8 @@ class InferConfig:
     model: str
     # HuggingFace dataset repo name (e.g., "LiberCoders/FeatureBench")
     dataset: str = "LiberCoders/FeatureBench"
+    # Hugging Face dataset tag, branch, or commit SHA. None defers to config.toml/default.
+    dataset_revision: Optional[str] = None
     n_concurrent: int = 1
     n_attempts: int = 1
     task_ids: Optional[List[str]] = None
@@ -191,6 +193,7 @@ class InferConfig:
             "agent": self.agent,
             "model": self.model,
             "dataset": self.dataset,
+            "dataset_revision": self.dataset_revision,
             "n_concurrent": self.n_concurrent,
             "n_attempts": self.n_attempts,
             "task_ids": self.task_ids,
@@ -257,6 +260,7 @@ class RunMetadata:
     task_ids: List[str]
     output_path: str
     start_time: str
+    dataset_revision: str = "v1.1"
     timeout: int = 7200
     proxy_port: Optional[int] = None
     runtime_proxy: Optional[bool] = None
@@ -284,6 +288,7 @@ class RunMetadata:
             "agent": self.agent,
             "model": self.model,
             "dataset": self.dataset,
+            "dataset_revision": self.dataset_revision,
             "n_concurrent": self.n_concurrent,
             "n_attempts": self.n_attempts,
             "task_ids": self.task_ids,
@@ -321,6 +326,8 @@ class RunMetadata:
         """Load metadata from a JSON file."""
         with open(path, "r", encoding="utf-8") as f:
             data = json.load(f)
+        # Runs created before dataset versioning used the original Hub release.
+        data.setdefault("dataset_revision", "v1.0")
         return cls(**data)
 
 
@@ -358,4 +365,4 @@ class TaskPaths:
     
     def ensure_dirs(self) -> None:
         """Create all necessary directories."""
-        self.attempt_dir.mkdir(parents=True, exist_ok=True)
+        self.attempt_dir.mkdir(parents=True, exist_ok=True)