benchflow 0.7.10.dev2406__tar.gz → 0.7.11__tar.gz

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/CHANGELOG.md

@@ -2,6 +2,87 @@
 
 ## [Unreleased]
 
+## 0.7.11 — 2026-10-16
+
+### Added
+
+- **`--separate-verifier-include PATH` carries outputs outside the workspace
+  into a separate verifier sandbox.** Only the workspace (`sandbox.workdir`,
+  else the image's `WORKDIR`) crossed, so a task whose agent writes graded
+  files elsewhere scored 0 with no error: FrontierPhysics
+  `kinked-ladder-polymer-scattering` (workspace `/app`, outputs in `/root`)
+  went from 19/19 in shared mode to 0/19 with `--separate-verifier`, and 14
+  of its 56 tasks grade files outside the workspace. Each include path
+  (repeatable; SDK and config `separate_verifier_include`) is captured beside
+  the workspace into `evidence/artifacts/`, hash-checked, and laid over the
+  verifier image's copy at the same path; pseudo filesystems and roots such as
+  `/`, `/usr` and `/home` are refused before any sandbox starts.
+  `config.json` records `verifier_include`, and the transfer inventory lists
+  each path as captured, missing or already in the workspace.
+- **A separate verifier lists files the agent left outside everything that
+  crosses.** When the workspace is frozen, the regular files written during
+  the agent phase outside the workspace and include paths (skipping system,
+  scratch and provider trees, package caches and the agent's home) are logged
+  as a warning with a count per directory and examples, and recorded under
+  `outside_workspace_writes` in `verifier-sandbox/verifier-sandbox.json`.
+  Advisory: it never fails a rollout.
+
+### Fixed
+
+- **The verifier ignores pip's and git's per-user config under `/root`.**
+  The verifier runs as root with `HOME=/root`, the agent's workspace in many
+  tasks, so `~/.config/pip/pip.conf` or `~/.gitconfig` there were whatever the
+  agent left, in both modes (a separate verifier receives the `/root`
+  workspace too). `VERIFIER_ENV` now sets `PIP_CONFIG_FILE=/dev/null` and
+  `GIT_CONFIG_GLOBAL=/dev/null` (git 2.32+; `/etc/gitconfig` still applies),
+  and drops `PIP_NO_USER_CONFIG`, which is not a pip option and did nothing.
+  A task that needs either config back sets it in its verifier `env`.
+- **Package-manager caches stay out of workspace evidence.** `.cache/pip`,
+  `.cache/uv` and npm's `_cacache`, `_npx` and `_logs` below a captured root
+  are recorded as `package_cache` exclusions instead of captured. With a
+  `/root` workspace the harness's own installs fill them as root before the
+  agent starts: on a FrontierPhysics ion-trap-design run they were 97% of the
+  745 MB the separate verifier received, and of the rubric review's upload.
+- **`mcp` 1.30.0 in `uv.lock`** for CVE-2026-59951 (was 1.28.1), which failed
+  the `pip-audit` job and, through it, `main`'s release gate.
+
+## 0.7.10 — 2026-10-16
+
+### Added
+
+- **Separate verifier sandboxes run on Docker and Daytona.** A task that sets
+  `[verifier] sandbox_mode = "separate"` (Harbor `environment_mode`) or
+  declares `[verifier.sandbox]` was refused with "separate verifier
+  sandboxes are parsed but not executed". It now runs: after the agent stops,
+  the workspace is frozen into `evidence/`, and a fresh sandbox built from
+  the verifier image (`[verifier.sandbox].docker_image`, `[sandbox]
+  .docker_image`, `tests/Dockerfile`, then the task's own
+  `environment/Dockerfile`) receives only those hash-checked bytes before the
+  normal hardening and `test.sh` run there. Nothing else from the agent's
+  sandbox reaches the verifier. `bench eval run --separate-verifier` (SDK and
+  config `separate_verifier`) forces the mode for every task of a run without
+  editing them, so task digests do not change. A transfer or sandbox failure
+  is a verifier infrastructure error, never a score;
+  `verifier-sandbox/verifier-sandbox.json` records the image source, sandbox
+  id, transfer inventory and timing, and `config.json` records
+  `verifier_sandbox_mode`. Ported from release/0.8's separate verifier,
+  without its declared-artifact and `/logs/artifacts` transfer, which main
+  does not have.
+
+### Fixed
+
+- **Root helpers no longer import modules from the agent's workspace.** The
+  helpers BenchFlow runs as root after the agent stops (disk reclaim, the
+  verifier PATH check, pytest-plugin discovery, the external-symlink purge,
+  the workspace-restore fallback, workspace evidence capture and its
+  reviewer-side admission) ran plain `python3 -c` from the sandbox's default
+  working directory, which is the agent's workspace, so a `json.py` or
+  `glob.py` left there was imported as root before the verifier. They now run
+  `python3 -I`; the `.pth`/`sitecustomize.py` purge keeps its environment
+  but runs from `/`. Pytest-plugin discovery therefore scans only the
+  interpreter's site-packages: declare a plugin installed elsewhere in
+  `verifier.pytest_plugins`.
+
 ## 0.7.9 — 2026-10-14
 
 ### Fixed

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.10.dev2406
+Version: 0.7.11
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.10.dev2406"
+version = "0.7.11"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/_utils/scoring.py

@@ -225,6 +225,10 @@ def classify_verifier_error(verifier_error: str | None) -> str | None:
     if not verifier_error:
         return None
     lower = verifier_error.lower()
+    if lower.startswith("separate verifier "):
+        # The verifier sandbox never received the agent's outputs (or never
+        # started): nothing was assessed.
+        return VERIFIER_INFRA
     if "verifier crashed" in verifier_error:
         if contains_verifier_dep_install_marker(lower):
             return VERIFIER_DEP_INSTALL

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/_utils/yaml_loader.py

@@ -125,6 +125,8 @@ def rollout_config_from_dict(
         ),
         context_root=raw.get("context_root"),
         base_image_override=raw.get("base_image_override"),
+        separate_verifier=bool(raw.get("separate_verifier", False)),
+        separate_verifier_include=tuple(raw.get("separate_verifier_include") or ()),
         agent=raw.get("agent", "claude-agent-acp"),
         model=raw.get("model"),
         reasoning_effort=raw.get("reasoning_effort"),

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/cli/main.py

@@ -75,6 +75,7 @@ from benchflow.eval_plan import EvalCreateRequest, EvalPlanError, build_eval_pla
 from benchflow.evaluation import DEFAULT_AGENT, effective_model
 from benchflow.sandbox.providers import providers_phrase
 from benchflow.skill_policy import SKILL_MODE_NO_SKILL
+from benchflow.task.verifier_sandbox import normalize_include_paths
 
 if TYPE_CHECKING:
     from benchflow.eval_plan import EvalPlan
@@ -446,6 +447,29 @@ def eval_run(
             ),
         ),
     ] = None,
+    separate_verifier: Annotated[
+        bool,
+        typer.Option(
+            "--separate-verifier",
+            help=(
+                "Run each verifier in a fresh sandbox built from the task image "
+                "that receives only the frozen workspace, as if every task set "
+                "\\[verifier].sandbox_mode = 'separate'. Docker and Daytona only."
+            ),
+        ),
+    ] = False,
+    separate_verifier_include: Annotated[
+        list[str] | None,
+        typer.Option(
+            "--separate-verifier-include",
+            help=(
+                "A path outside the workspace that a separate verifier sandbox "
+                "receives too, at the same path, for tasks whose agent writes "
+                "graded files there (e.g. /root when the workspace is /app). "
+                "Repeatable."
+            ),
+        ),
+    ] = None,
     skills_dir: Annotated[
         Path | None,
         typer.Option("--skills-dir", help="Skills directory to deploy"),
@@ -652,6 +676,12 @@ def eval_run(
         # the TTY Rich dashboard (see cli._live_progress.progress_enabled).
         os.environ["BENCHFLOW_NO_PROGRESS"] = "1"
 
+    try:
+        include_paths = normalize_include_paths(separate_verifier_include or ())
+    except ValueError as exc:
+        print_error(f"{exc}")
+        raise typer.Exit(1) from None
+
     request = EvalCreateRequest(
         config_file=config_file,
         tasks_dir=tasks_dir,
@@ -677,6 +707,8 @@ def eval_run(
         sandbox_setup_timeout=sandbox_setup_timeout,
         context_root=context_root,
         base_image_override=base_image_override,
+        separate_verifier=separate_verifier,
+        separate_verifier_include=include_paths,
         skills_dir=skills_dir,
         skill_mode=skill_mode,
         skill_creator_dir=skill_creator_dir,
@@ -1019,6 +1051,17 @@ def _run_config_file_eval(plan: "EvalPlan") -> None:
         # run-config file was a no-op.
         if plan.eval_config_override is not None:
             j._config.config_override = plan.eval_config_override
+        # --separate-verifier only ever strengthens isolation, so it wins over
+        # a config file that leaves it off.
+        if plan.request.separate_verifier:
+            j._config.separate_verifier = True
+        if plan.request.separate_verifier_include:
+            j._config.separate_verifier_include = normalize_include_paths(
+                [
+                    *j._config.separate_verifier_include,
+                    *plan.request.separate_verifier_include,
+                ]
+            )
     except subprocess.CalledProcessError as e:
         # A source.repo clone/fetch failure (git exits non-zero) otherwise escapes
         # as a raw traceback — it is not a config-parse error, so give it its own

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/eval_plan.py

@@ -103,6 +103,8 @@ class EvalCreateRequest:
     sandbox_setup_timeout: int = 120
     context_root: Path | None = None
     base_image_override: str | None = None
+    separate_verifier: bool = False
+    separate_verifier_include: tuple[str, ...] = ()
     skills_dir: Path | None = None
     skill_mode: str = SKILL_MODE_NO_SKILL
     skill_creator_dir: Path | None = None
@@ -201,6 +203,8 @@ class EvalPlan:
             sandbox_setup_timeout=req.sandbox_setup_timeout,
             context_root=str(req.context_root) if req.context_root else None,
             base_image_override=req.base_image_override,
+            separate_verifier=req.separate_verifier,
+            separate_verifier_include=req.separate_verifier_include,
             skills_dir=str(req.skills_dir) if req.skills_dir else None,
             skill_mode=req.skill_mode,
             skill_creator_dir=(

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/eval_sharding.py

@@ -134,6 +134,8 @@ def _config_payload(
         "agent_idle_timeout": config.agent_idle_timeout,
         "context_root": config.context_root,
         "base_image_override": config.base_image_override,
+        "separate_verifier": config.separate_verifier,
+        "separate_verifier_include": list(config.separate_verifier_include),
         "exclude_tasks": sorted(config.exclude_tasks),
         "include_tasks": sorted(shard.task_names),
         "skill_mode": config.skill_mode,

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/eval_worker.py

@@ -69,6 +69,8 @@ def _evaluation_config(raw: dict[str, Any]) -> EvaluationConfig:
         agent_idle_timeout=raw.get("agent_idle_timeout"),
         context_root=raw.get("context_root"),
         base_image_override=raw.get("base_image_override"),
+        separate_verifier=bool(raw.get("separate_verifier", False)),
+        separate_verifier_include=tuple(raw.get("separate_verifier_include") or ()),
         exclude_tasks=set(raw.get("exclude_tasks") or []),
         include_tasks=set(raw.get("include_tasks") or []),
         skill_mode=raw.get("skill_mode") or SKILL_MODE_NO_SKILL,

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/evaluation.py

@@ -501,6 +501,8 @@ class EvaluationConfig:
     agent_idle_timeout: int | None = 600
     context_root: str | None = None
     base_image_override: str | None = None
+    separate_verifier: bool = False
+    separate_verifier_include: tuple[str, ...] = ()
     exclude_tasks: set[str] = field(default_factory=set)
     include_tasks: set[str] = field(default_factory=set)
     skill_mode: str = SKILL_MODE_NO_SKILL
@@ -910,6 +912,8 @@ class Evaluation:
             ),
             context_root=raw.get("context_root"),
             base_image_override=raw.get("base_image_override"),
+            separate_verifier=bool(raw.get("separate_verifier", False)),
+            separate_verifier_include=tuple(raw.get("separate_verifier_include") or ()),
             exclude_tasks=exclude,
             include_tasks=include,
             skill_mode=raw.get("skill_mode", SKILL_MODE_NO_SKILL),
@@ -1007,6 +1011,8 @@ class Evaluation:
             ),
             context_root=raw.get("context_root"),
             base_image_override=raw.get("base_image_override"),
+            separate_verifier=bool(raw.get("separate_verifier", False)),
+            separate_verifier_include=tuple(raw.get("separate_verifier_include") or ()),
             include_tasks=include,
             exclude_tasks=exclude,
             skill_mode=raw.get("skill_mode", SKILL_MODE_NO_SKILL),
@@ -1284,6 +1290,8 @@ class Evaluation:
             agent_idle_timeout=cfg.agent_idle_timeout,
             context_root=cfg.context_root,
             base_image_override=cfg.base_image_override,
+            separate_verifier=cfg.separate_verifier,
+            separate_verifier_include=cfg.separate_verifier_include,
             skill_mode=skill_mode,
             skill_creator_dir=cfg.skill_creator_dir,
             self_gen_no_internet=cfg.self_gen_no_internet,
@@ -1345,6 +1353,8 @@ class Evaluation:
             agent_idle_timeout=cfg.agent_idle_timeout,
             context_root=cfg.context_root,
             base_image_override=cfg.base_image_override,
+            separate_verifier=cfg.separate_verifier,
+            separate_verifier_include=cfg.separate_verifier_include,
             skill_mode=cfg.skill_mode,
             skill_creator_dir=cfg.skill_creator_dir,
             self_gen_no_internet=cfg.self_gen_no_internet,

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/review/evidence.py

@@ -30,6 +30,19 @@ from benchflow.task.config import ArtifactConfig
 
 logger = logging.getLogger(__name__)
 
+# Package-manager caches, matched as path components below a captured root.
+# With a /root workspace, the harness's own installs (codex-acp through npm,
+# tools through uv) fill these as root before the agent starts, and they can
+# be most of the workspace's bytes. A cache is never an output: the review and
+# the separate verifier sandbox both do without it.
+PACKAGE_CACHE_PATHS = (
+    ".cache/pip",
+    ".cache/uv",
+    ".npm/_cacache",
+    ".npm/_logs",
+    ".npm/_npx",
+)
+
 
 class EvidenceError(RuntimeError):
     """Required evidence could not be captured or verified completely."""
@@ -51,7 +64,9 @@ class EvidenceExclusion(BaseModel):
     model_config = ConfigDict(extra="forbid", frozen=True)
 
     original_path: str
-    reason: Literal["credential", "sandbox_runtime", "special_file", "task_exclude"]
+    reason: Literal[
+        "credential", "sandbox_runtime", "package_cache", "special_file", "task_exclude"
+    ]
 
 
 class ArtifactEvidence(BaseModel):
@@ -125,6 +140,8 @@ def excluded(path):
     # but a root that itself lies under such a directory is still captured.
     if reason is None and any(component("/" + relative, rule) for rule in rules["sandbox_runtime"]):
         reason = "sandbox_runtime"
+    if reason is None and any(component("/" + relative, rule) for rule in rules["package_cache"]):
+        reason = "package_cache"
     if reason is None and any(fnmatch.fnmatchcase(relative, rule) or fnmatch.fnmatchcase(path.name, rule) for rule in rules["exclude"]):
         reason = "task_exclude"
     if reason:
@@ -346,9 +363,9 @@ async def capture_workspace(
     ``destination`` must not exist. It receives ``workspace/`` and
     ``manifest.json`` together after successful validation. Capture limits are
     hard errors: this API never labels a truncated tree complete. Credential
-    files, provider runtime state (``SANDBOX_RUNTIME_STATE_PATHS``) and
-    sockets, FIFOs or device nodes stay out of the bundle, each recorded in
-    ``manifest.exclusions``.
+    files, provider runtime state (``SANDBOX_RUNTIME_STATE_PATHS``), package
+    manager caches (``PACKAGE_CACHE_PATHS``) and sockets, FIFOs or device
+    nodes stay out of the bundle, each recorded in ``manifest.exclusions``.
     """
     if destination.exists() or destination.is_symlink():
         raise EvidenceError(f"Evidence destination already exists: {destination}")
@@ -361,6 +378,7 @@ async def capture_workspace(
         {
             "credentials": CREDENTIAL_EVIDENCE_PATHS,
             "sandbox_runtime": SANDBOX_RUNTIME_STATE_PATHS,
+            "package_cache": PACKAGE_CACHE_PATHS,
             "exclude": list(exclude),
             "paths": list(excluded_paths),
         }
@@ -368,6 +386,7 @@ async def capture_workspace(
     command = shlex.join(
         [
             "python3",
+            "-I",
             "-c",
             _CAPTURE_SCRIPT,
             workspace,
@@ -475,7 +494,9 @@ async def validate_uploaded_workspace(
     env: Sandbox, workspace: str, manifest_path: str, *, timeout_sec: int = 600
 ) -> None:
     """Reject missing or changed files in the reviewer sandbox after upload."""
-    command = shlex.join(["python3", "-c", _ADMISSION_SCRIPT, workspace, manifest_path])
+    command = shlex.join(
+        ["python3", "-I", "-c", _ADMISSION_SCRIPT, workspace, manifest_path]
+    )
     result = await env.exec(command, user="root", timeout_sec=timeout_sec)
     if result.return_code:
         raise EvidenceError(
@@ -574,7 +595,7 @@ async def install_uploaded_workspace(
     The destination must be fresh; a failed installation is never admissible.
     """
     command = shlex.join(
-        ["python3", "-c", _UNPACK_SCRIPT, archive_path, workspace, manifest_path]
+        ["python3", "-I", "-c", _UNPACK_SCRIPT, archive_path, workspace, manifest_path]
     )
     result = await env.exec(command, user="root", timeout_sec=timeout_sec)
     if result.return_code:
@@ -646,6 +667,7 @@ async def capture_task_evidence(
             command = shlex.join(
                 [
                     "python3",
+                    "-I",
                     "-c",
                     _ARTIFACT_PROBE_SCRIPT,
                     manifest.workspace,

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/rollout/__init__.py

@@ -124,8 +124,10 @@ from benchflow.rollout._results import (
 from benchflow.rollout._review import (
     capture_terminal_workspace,
     finish_terminal_review,
+    mark_agent_phase_start,
     prepare_terminal_review,
 )
+from benchflow.rollout._separate_verifier import run_separate_verifier
 from benchflow.rollout._setup import (
     _agent_launch_with_web_policy as _agent_launch_with_web_policy,
 )
@@ -219,6 +221,10 @@ from benchflow.skill_policy import (
     task_bundled_skills_dir,
 )
 from benchflow.skill_policy import SKILL_MODE_WITH_SKILL as SKILL_MODE_WITH_SKILL
+from benchflow.task.verifier_sandbox import (
+    apply_separate_verifier_flag,
+    separate_verifier_requested,
+)
 from benchflow.trajectories._capture import (
     TrajectoryWriter,
     _capture_session_trajectory,
@@ -770,6 +776,10 @@ class Rollout:
         # an export-time infra failure ("connection lost") as the agent's own
         # infra_failure category in dashboards.
         self._export_error: str | None = None
+        # Separate verifier only: the sandbox clock as the agent phase began,
+        # and the files written outside everything that crosses since then.
+        self._agent_phase_started: float | None = None
+        self._outside_workspace_writes: dict | None = None
         # Single bag for the four parallel diagnostic fields the old code
         # carried as separate attrs (issue #503). Each callsite that used
         # to assign to one of those slots now calls ``self._diagnostics.set(...)``
@@ -967,6 +977,23 @@ class Rollout:
                 self._task.config, cfg.config_override
             )
 
+        if cfg.separate_verifier:
+            # --separate-verifier: this run's verifier gets its own sandbox,
+            # the task's files (and digest) stay as they are. Refused here,
+            # before any sandbox exists, where that sandbox cannot run.
+            self._task.config = apply_separate_verifier_flag(
+                self._task.config, sandbox=cfg.environment, task_dir=cfg.task_path
+            )
+
+        if cfg.separate_verifier_include and not separate_verifier_requested(
+            self._task.config
+        ):
+            logger.warning(
+                "--separate-verifier-include has no effect for %s: its verifier "
+                "runs in the agent's sandbox",
+                self._task.name,
+            )
+
         prepare_terminal_review(self)
 
         self._disallow_web_tools = (
@@ -1106,6 +1133,16 @@ class Rollout:
             started_at=self._started_at,
             agent_env=self._agent_env,
             base_image_override=cfg.base_image_override,
+            verifier_sandbox_mode=(
+                "separate"
+                if separate_verifier_requested(getattr(self._task, "config", None))
+                else "shared"
+            ),
+            verifier_include=(
+                cfg.separate_verifier_include
+                if separate_verifier_requested(getattr(self._task, "config", None))
+                else ()
+            ),
             usage_tracking=cfg.usage_tracking.with_env_defaults(),
             concurrency=cfg.concurrency,
             agent_idle_timeout=cfg.agent_idle_timeout,
@@ -1224,6 +1261,7 @@ class Rollout:
                     self._env, cfg.generated_skills_root, cfg.sandbox_user
                 )
             await self._planes.lockdown_paths(self._env, self._effective_locked)
+            await mark_agent_phase_start(self)
             self._phase = "installed"
             return
 
@@ -1291,6 +1329,7 @@ class Rollout:
                 self._env, cfg.generated_skills_root, cfg.sandbox_user
             )
         await self._planes.lockdown_paths(self._env, self._effective_locked)
+        await mark_agent_phase_start(self)
 
         self._phase = "installed"
 
@@ -1899,6 +1938,13 @@ class Rollout:
                     f"Using scraped trajectory ({len(scraped)} events) — UNTRUSTED"
                 )
 
+        if separate_verifier_requested(getattr(self._task, "config", None)):
+            # The verifier never runs in the agent's sandbox: a fresh one gets
+            # only the frozen workspace.
+            self._rewards, self._verifier_error = await run_separate_verifier(self)
+            self._phase = "verified"
+            return self._rewards
+
         await _publish_trajectory_for_verifier(
             self._env, self._trajectory, self._rollout_paths.agent_dir
         )
@@ -1933,6 +1979,14 @@ class Rollout:
         Returns (rewards, verifier_output, verifier_error). The final
         verify() still does full hardening.
         """
+        if separate_verifier_requested(getattr(self._task, "config", None)):
+            # Soft verification would upload tests/ into the agent's sandbox.
+            return (
+                None,
+                None,
+                "soft_verify is unavailable: the task uses a separate verifier "
+                "sandbox, so tests never enter the agent's sandbox",
+            )
         self._rollout_paths.verifier_dir.mkdir(parents=True, exist_ok=True)
         # Clean verifier output dir — chmod 777 so non-root verifier processes can write.
         # Keep /app present for task/verifier paths that still use the legacy
@@ -2221,6 +2275,9 @@ class Rollout:
                     user="root",
                     timeout_sec=10,
                 )
+                # That harness step writes /root/.gitconfig: restart the agent
+                # phase clock so it is not listed as the solver's output.
+                await mark_agent_phase_start(self)
                 self._trajectory, self._agent_name = await _run_oracle(
                     self._env, cfg.task_path, self._timeout, sandbox_user=None
                 )

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/rollout/_config.py

@@ -166,6 +166,12 @@ class RolloutConfig:
     generated_skills_root: str = GENERATED_SKILLS_ROOT
     self_gen_no_internet: bool = False
     skip_verify: bool = False
+    # Run the verifier in a fresh sandbox that receives only the frozen
+    # workspace, whatever the task declares (``--separate-verifier``).
+    separate_verifier: bool = False
+    # Paths outside the workspace that a separate verifier sandbox receives
+    # too, at the same path (``--separate-verifier-include``).
+    separate_verifier_include: tuple[str, ...] = ()
     reviewer: ReviewerConfig = field(default_factory=ReviewerConfig)
     purpose: Literal["task", "reviewer"] = "task"
     parent_rollout: str | None = None
@@ -183,6 +189,7 @@ class RolloutConfig:
 
     def __post_init__(self) -> None:
         from benchflow._utils.config import normalize_agent_idle_timeout
+        from benchflow.task.verifier_sandbox import normalize_include_paths
 
         if not isinstance(self.task_path, Path):
             self.task_path = Path(self.task_path)
@@ -197,6 +204,9 @@ class RolloutConfig:
             self.base_image_override = base_image
         if self.skills_dir is not None and not isinstance(self.skills_dir, Path):
             self.skills_dir = Path(self.skills_dir)
+        self.separate_verifier_include = normalize_include_paths(
+            self.separate_verifier_include or ()
+        )
         self.skill_mode = normalize_skill_mode(self.skill_mode)
         if self.artifact_skill_mode is not None:
             self.artifact_skill_mode = normalize_skill_mode(self.artifact_skill_mode)

{benchflow-0.7.10.dev2406 → benchflow-0.7.11}/src/benchflow/rollout/_results.py

@@ -16,6 +16,7 @@ from __future__ import annotations
 
 import json
 import logging
+from collections.abc import Sequence
 from datetime import datetime
 from pathlib import Path
 from typing import Any, Literal
@@ -121,6 +122,8 @@ def _write_config(
     started_at: datetime,
     agent_env: dict[str, str],
     base_image_override: str | None = None,
+    verifier_sandbox_mode: str = "shared",
+    verifier_include: Sequence[str] = (),
     reasoning_effort: str | None = None,
     usage_tracking: UsageTrackingConfig | None = None,
     concurrency: int | None = None,
@@ -170,6 +173,8 @@ def _write_config(
         else effective_install_timeout(agent, sandbox_setup_timeout),
         "context_root": str(context_root) if context_root else None,
         "base_image_override": base_image_override,
+        "verifier_sandbox_mode": verifier_sandbox_mode,
+        "verifier_include": list(verifier_include),
         "timeout_sec": timeout,
         "concurrency": concurrency,
         "agent_idle_timeout_sec": agent_idle_timeout,