benchflow 0.7.6.dev2266__tar.gz → 0.7.7.dev2305__tar.gz

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## [Unreleased]
 
+### Added
+- **`network_mode: denylist` blocks a list of URLs and hosts for the agent on
+  Docker and Daytona.** The task keeps internet access; `blocked_urls` and
+  `blocked_hosts` are enforced by a root-owned loopback proxy behind the
+  sandbox-user firewall, hosted search tools are switched off per harness,
+  and every refused request lands in `trajectory/egress_denylist.jsonl`.
+  Other backends refuse the mode at preflight. (#1113)
+
+## 0.7.6 — 2026-09-04
+
 ### Added
 - **ACP rollout directories render as an interactive reviewer page.**
   `bench eval view <rollout>` now assembles a JSON payload (normalized
@@ -42,6 +52,48 @@
   timestamps.** Steps gain a `+m:ss` offset chip and tool calls a duration
   chip whenever events carry `ts` / `started_at` / `finished_at` fields;
   with today's captures (which carry none) nothing changes visually.
+- **Z.ai Coding Plan routing.** Coding Plan subscriptions route through the
+  provider layer, with validated generation parameters forwarded and explicit
+  provider endpoints preserved by the LiteLLM proxy. (#1074)
+- **`bench eval run` publishes job artifacts and eval results.**
+  `--publish-bucket` syncs a job directory to a HuggingFace storage bucket (an
+  alternative to `--publish-hf`'s dataset-repo upload), and
+  `--eval-results-model/-dataset/-task` opens a community eval-results PR
+  (`.eval_results/*.yaml`) on a model's HF repo, scored from the run's mean
+  reward. (#1035)
+
+### Fixed
+
+- **`claude-fable-5-1` can run: the `claude-agent-acp` pin moves 0.40.0 to
+  0.73.0.** The old pin bundled `@anthropic-ai/claude-agent-sdk` 0.3.160, and
+  the model rejects Claude Code older than 2.1.251 with
+  `claude_code_version_too_old` (HTTP 400) — so every rollout failed on its
+  first API call. The new pin bundles sdk 0.3.257, and the `set_config_option`
+  `"model"` / `"effort"` wiring is re-verified against it by
+  `tests/test_acp_pinned_protocol_guard.py`. (#1086)
+- **`codex-acp` dispatch modernized.** The shim pin moves to 1.6.0 and
+  reasoning effort travels inside the model id (`modelId[effort]`) through
+  `session/set_model`, rather than a config option the shim rejects. (#1044)
+- **ACP usage and terminal evidence survive an agent timeout**, instead of
+  being dropped with the timed-out turn. (#1080)
+- **A pending tool call can no longer defer the idle watchdog without
+  bound** — the deferral is capped. (#1066)
+- **`bench eval` resume re-runs infra-retryable verifier-errored tasks**
+  rather than treating the infrastructure failure as a settled result. (#1063)
+- **Sandbox hardening execs draw on the verifier-setup budget**, so hardening
+  no longer competes with the agent's own time. (#1062)
+- **`--trials > 1` without `--matrix` is rejected up front.** (#1064)
+- **Healthy native ACP subscription results are preserved.** (#1049)
+- **The agent judge parses native edit targets.** (#1069)
+- **Non-dict rewards no longer break completed-outcome classification.** (#1054)
+- **Codex honors proxy-owned model selection.** (#1076)
+- **Gemini routing corrections.** ACP model ids fixed and wrapped ids
+  normalized, Gemma routing supported with pass-through usage captured, every
+  Google key alias routed through the proxy, Google Gemini gateway routes
+  normalized, and headless runs trust the sandbox workspace.
+- **LiteLLM provisions the Vertex sandbox runtime.** (#985)
+- **OpenClaw caps supported model output tokens.**
+- **RestrictedPython updated to 8.3.**
 
 ## 0.7.5 — 2026-08-19
 

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.6.dev2266
+Version: 0.7.7.dev2305
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.6.dev2266"
+version = "0.7.7.dev2305"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/agents/install.py

@@ -99,7 +99,7 @@ def _owner_from_home(home: str) -> str | None:
 
 
 def _policy_home_dirs(agent: str, agent_cfg: AgentConfig) -> list[str]:
-    """Agent home dirs a no-web setup command may create."""
+    """Agent home dirs a web-policy setup command may create."""
     dirs = set(agent_cfg.home_dirs)
     for owned_path in agent_cfg.disallow_web_tools_owned_paths:
         if not owned_path.startswith("$HOME/"):
@@ -265,15 +265,21 @@ async def apply_web_tool_policy(
     home: str,
     *,
     disallow: bool,
+    disallow_hosted_search: bool = False,
 ) -> None:
-    """Apply an agent-specific hard web-tool disable in the agent home."""
-    if not disallow or not agent_cfg or not agent_cfg.disallow_web_tools_setup_cmd:
+    """Apply the agent's no-web or hosted-search-off setup command in the agent home."""
+    if not agent_cfg:
+        return
+    if disallow:
+        policy, setup_cmd = "no-web", agent_cfg.disallow_web_tools_setup_cmd
+    elif disallow_hosted_search:
+        policy, setup_cmd = "hosted-search", agent_cfg.disallow_hosted_search_setup_cmd
+    else:
+        return
+    if not setup_cmd:
         return
 
-    cmd = (
-        f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; "
-        f"{agent_cfg.disallow_web_tools_setup_cmd}"
-    )
+    cmd = f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; {setup_cmd}"
     owner = _owner_from_home(home)
     if owner:
         q_owner = shlex.quote(owner)
@@ -296,7 +302,7 @@ async def apply_web_tool_policy(
         if stderr:
             details.append(f"stderr: {stderr}")
         raise RuntimeError(
-            f"Failed to apply no-web policy for {agent}: {'; '.join(details)}"
+            f"Failed to apply {policy} policy for {agent}: {'; '.join(details)}"
         )
 
 

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/agents/manifest.py

@@ -89,6 +89,8 @@ _SHIM_ONLY = frozenset(
         "disallow_web_tools_setup_cmd",
         "disallow_web_tools_owned_paths",
         "disallow_web_tools_launch_suffix",
+        "disallow_hosted_search_setup_cmd",
+        "disallow_hosted_search_launch_suffix",
         "task_mcp_transport",
         "task_mcp_config_path",
     }

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/agents/registry.py

@@ -502,6 +502,12 @@ class AgentConfig:
     # String appended to launch_cmd when BenchFlow's no-web policy is active.
     # Use for agents whose supported toggle is a launch/config override.
     disallow_web_tools_launch_suffix: str = ""
+    # Shell snippet that switches off hosted (provider-side) search tools when
+    # the denylist egress policy is active; local fetch tools stay on because
+    # they go through the egress proxy. Reuses disallow_web_tools_owned_paths.
+    disallow_hosted_search_setup_cmd: str = ""
+    # String appended to launch_cmd when the denylist egress policy is active.
+    disallow_hosted_search_launch_suffix: str = ""
     # How task-declared MCP servers are delivered to the agent:
     # "acp" sends them in session/new; "native-config" writes an agent-specific
     # config file before launch (for agents whose ACP server drops/reformats
@@ -511,6 +517,15 @@ class AgentConfig:
     task_mcp_config_path: str = ""
 
 
+_GEMINI_EXCLUDE_WEB_TOOLS_CMD = _json_settings_merge(
+    "$BENCHFLOW_AGENT_HOME/.gemini/settings.json",
+    'd.setdefault("tools",{}).setdefault("exclude",[]);'
+    '[d["tools"]["exclude"].append(t) for t in '
+    '["google_web_search","web_fetch"] '
+    'if t not in d["tools"]["exclude"]]',
+)
+
+
 # Agent registry — all supported agents
 AGENTS: dict[str, AgentConfig] = {
     "claude-agent-acp": AgentConfig(
@@ -552,6 +567,12 @@ AGENTS: dict[str, AgentConfig] = {
             'if t not in d["permissions"]["deny"]]',
         ),
         disallow_web_tools_owned_paths=["$HOME/.claude"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.claude/settings.json",
+            'd.setdefault("permissions",{}).setdefault("deny",[]);'
+            '[d["permissions"]["deny"].append(t) for t in ["WebSearch"] '
+            'if t not in d["permissions"]["deny"]]',
+        ),
         supports_acp_set_model=False,
         acp_model_config_id="model",
         acp_effort_config_id="effort",
@@ -650,6 +671,7 @@ AGENTS: dict[str, AgentConfig] = {
             ],
         ),
         disallow_web_tools_launch_suffix=" -c tools.web_search=false",
+        disallow_hosted_search_launch_suffix=" -c tools.web_search=false",
     ),
     "gemini": AgentConfig(
         name="gemini",
@@ -694,14 +716,11 @@ AGENTS: dict[str, AgentConfig] = {
                 ),
             ],
         ),
-        disallow_web_tools_setup_cmd=_json_settings_merge(
-            "$BENCHFLOW_AGENT_HOME/.gemini/settings.json",
-            'd.setdefault("tools",{}).setdefault("exclude",[]);'
-            '[d["tools"]["exclude"].append(t) for t in '
-            '["google_web_search","web_fetch"] '
-            'if t not in d["tools"]["exclude"]]',
-        ),
+        disallow_web_tools_setup_cmd=_GEMINI_EXCLUDE_WEB_TOOLS_CMD,
         disallow_web_tools_owned_paths=["$HOME/.gemini"],
+        # web_fetch tries the hosted urlContext path before any local fetch, so
+        # the egress proxy cannot filter it; the denylist excludes both tools.
+        disallow_hosted_search_setup_cmd=_GEMINI_EXCLUDE_WEB_TOOLS_CMD,
     ),
     "opencode": AgentConfig(
         name="opencode",
@@ -730,6 +749,10 @@ AGENTS: dict[str, AgentConfig] = {
             'd.setdefault("tools",{})["webfetch"]=False',
         ),
         disallow_web_tools_owned_paths=["$HOME/.config/opencode"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.config/opencode/opencode.json",
+            'd.setdefault("tools",{})["websearch"]=False',
+        ),
     ),
     "mimo": AgentConfig(
         name="mimo",
@@ -779,6 +802,10 @@ AGENTS: dict[str, AgentConfig] = {
             'd.setdefault("tools",{})["webfetch"]=False',
         ),
         disallow_web_tools_owned_paths=["$HOME/.config/mimocode"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.config/mimocode/mimocode.json",
+            'd.setdefault("tools",{})["websearch"]=False',
+        ),
     ),
     "harvey-lab-harness": AgentConfig(
         name="harvey-lab-harness",
@@ -1183,6 +1210,8 @@ def _acpx_wrap(config: AgentConfig) -> AgentConfig:
         disallow_web_tools_setup_cmd=config.disallow_web_tools_setup_cmd,
         disallow_web_tools_owned_paths=config.disallow_web_tools_owned_paths,
         disallow_web_tools_launch_suffix=config.disallow_web_tools_launch_suffix,
+        disallow_hosted_search_setup_cmd=config.disallow_hosted_search_setup_cmd,
+        disallow_hosted_search_launch_suffix=config.disallow_hosted_search_launch_suffix,
         task_mcp_transport=config.task_mcp_transport,
         task_mcp_config_path=config.task_mcp_config_path,
     )
@@ -1377,6 +1406,8 @@ def register_agent(
     disallow_web_tools_setup_cmd: str = "",
     disallow_web_tools_owned_paths: list[str] | None = None,
     disallow_web_tools_launch_suffix: str = "",
+    disallow_hosted_search_setup_cmd: str = "",
+    disallow_hosted_search_launch_suffix: str = "",
 ) -> AgentConfig:
     """Register a custom agent at runtime.
 
@@ -1416,6 +1447,8 @@ def register_agent(
         disallow_web_tools_setup_cmd=disallow_web_tools_setup_cmd,
         disallow_web_tools_owned_paths=disallow_web_tools_owned_paths or [],
         disallow_web_tools_launch_suffix=disallow_web_tools_launch_suffix,
+        disallow_hosted_search_setup_cmd=disallow_hosted_search_setup_cmd,
+        disallow_hosted_search_launch_suffix=disallow_hosted_search_launch_suffix,
     )
     AGENTS[name] = config
     AGENT_INSTALLERS[name] = install_cmd

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/contracts/planes.py

@@ -8,10 +8,13 @@ the composition edge so the kernel does not import those concrete modules.
 from __future__ import annotations
 
 from pathlib import Path
-from typing import Any, Protocol, runtime_checkable
+from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
 
 from benchflow.environment.manifest import EnvironmentManifest
 
+if TYPE_CHECKING:
+    from benchflow.sandbox.egress_denylist import EgressDenylist
+
 
 class LiveUsageGateway(Protocol):
     """A provider gateway process that reports live cumulative token usage.
@@ -33,7 +36,13 @@ class LiveUsageGateway(Protocol):
 class RolloutPlanes(Protocol):
     """Concrete-plane operations the rollout kernel needs."""
 
-    def agent_launch(self, agent: str, *, disallow_web_tools: bool) -> str: ...
+    def agent_launch(
+        self,
+        agent: str,
+        *,
+        disallow_web_tools: bool,
+        disallow_hosted_search: bool = False,
+    ) -> str: ...
     def agent_config(self, agent: str) -> Any: ...
 
     def resolve_agent_env(
@@ -102,6 +111,10 @@ class RolloutPlanes(Protocol):
     async def link_skill_paths(self, *args: Any, **kwargs: Any) -> None: ...
     async def ensure_litellm_runtime(self, *args: Any, **kwargs: Any) -> Any: ...
     async def stop_provider_runtime(self, runtime: Any) -> None: ...
+    async def start_egress_denylist(
+        self, env: Any, sandbox_user: str | None, denylist: EgressDenylist
+    ) -> None: ...
+    async def stop_egress_denylist(self, env: Any, rollout_dir: Path) -> None: ...
     def extract_usage(self, runtime: Any) -> dict[str, Any]: ...
     async def connect_acp(self, *args: Any, **kwargs: Any) -> Any: ...
     async def execute_prompts(self, *args: Any, **kwargs: Any) -> Any: ...

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/rollout/__init__.py

@@ -149,6 +149,7 @@ from benchflow.rollout._setup import _start_env_and_upload as _start_env_and_upl
 from benchflow.rollout._setup import (
     _task_disallows_internet as _task_disallows_internet,
 )
+from benchflow.rollout._setup import _task_egress_denylist as _task_egress_denylist
 from benchflow.rollout._setup import _verify_rollout as _verify_rollout
 from benchflow.rollout._skills import (
     _resolve_skill_creator_root as _resolve_skill_creator_root,
@@ -197,6 +198,7 @@ from benchflow.rollout.task_runtime import TaskRuntimeConfig as TaskRuntimeConfi
 from benchflow.rollout.task_runtime import TaskRuntimeResult as TaskRuntimeResult
 from benchflow.rollout_branch import ChildRunner
 from benchflow.rollout_branch import branch as _branch_engine
+from benchflow.sandbox.egress_denylist import EgressDenylist, denylist_agent_env
 from benchflow.sandbox.metadata import persist_sandbox_info
 from benchflow.scenes import compile_scenes_to_steps
 from benchflow.scenes import scene_step_prompt as scene_step_prompt
@@ -658,6 +660,8 @@ class Rollout:
         self._timing: dict[str, float] = {}
         self._effective_locked: list[str] = []
         self._disallow_web_tools: bool = False
+        self._egress_denylist: EgressDenylist | None = None
+        self._disallow_hosted_search: bool = False
         self._effective_skills_dir: Path | None = None
         self._effective_skills_sandbox_dir: str | None = None
         # Task dir actually deployed: a temp copy (self._task_tmp) when
@@ -952,6 +956,14 @@ class Rollout:
         self._disallow_web_tools = (
             _task_disallows_internet(self._task) or cfg.self_gen_no_internet
         ) and cfg.primary_agent != "oracle"
+        self._egress_denylist = (
+            None
+            if self._disallow_web_tools or cfg.primary_agent == "oracle"
+            else _task_egress_denylist(self._task)
+        )
+        if self._egress_denylist is not None and not cfg.sandbox_user:
+            raise ValueError("network_mode='denylist' requires a sandbox_user")
+        self._disallow_hosted_search = self._egress_denylist is not None
         self._agent_env = _apply_web_policy(
             self._planes.resolve_agent_env(
                 cfg.primary_agent, cfg.primary_model, cfg.agent_env
@@ -973,6 +985,7 @@ class Rollout:
         self._agent_launch = self._planes.agent_launch(
             cfg.primary_agent,
             disallow_web_tools=self._disallow_web_tools,
+            disallow_hosted_search=self._disallow_hosted_search,
         )
 
         # Copy task dir to temp when Dockerfile mutations are needed
@@ -1231,6 +1244,7 @@ class Rollout:
             self._agent_cfg,
             cred_home,
             disallow=self._disallow_web_tools,
+            disallow_hosted_search=self._disallow_hosted_search,
         )
         await self._planes.snapshot_build_config(self._env, workspace=self._agent_cwd)
         await self._planes.seed_verifier_workspace(
@@ -1274,11 +1288,18 @@ class Rollout:
             return cfg.session_factory
         return None
 
+    async def _start_egress_denylist(self, denylist: EgressDenylist) -> None:
+        """(Re)start the egress proxy before an ACP connection; a restored sandbox has none running."""
+        await self._planes.start_egress_denylist(
+            self._env, self._config.sandbox_user, denylist
+        )
+
     async def connect(self) -> None:
         """Open an ACP connection to the agent. Can be called multiple times."""
         cfg = self._config
         rollout_dir = self._require_rollout_dir()
         t0 = datetime.now()
+        egress_denylist = getattr(self, "_egress_denylist", None)
 
         (
             self._agent_env,
@@ -1295,11 +1316,16 @@ class Rollout:
             sandbox_setup_timeout=cfg.sandbox_setup_timeout,
             required_skill_names=getattr(self, "_required_skill_names", ()),
             live_trajectory_path=rollout_dir / "trajectory" / "llm_trajectory.jsonl",
-            force_sandbox_local=getattr(self, "_disallow_web_tools", False),
+            force_sandbox_local=getattr(self, "_disallow_web_tools", False)
+            or egress_denylist is not None,
         )
+        if egress_denylist is not None:
+            self._agent_env = denylist_agent_env(self._agent_env)
         sf_entrypoint = self._session_factory_entrypoint(cfg.primary_agent)
         self._is_session_factory = sf_entrypoint is not None
         if sf_entrypoint is not None:
+            if egress_denylist is not None:
+                raise RuntimeError("network_mode='denylist' requires an ACP agent")
             (
                 self._acp_client,
                 self._session,
@@ -1317,6 +1343,8 @@ class Rollout:
                 agent_cwd=self._agent_cwd,
             )
         else:
+            if egress_denylist is not None:
+                await self._start_egress_denylist(egress_denylist)
             (
                 self._acp_client,
                 self._session,
@@ -2012,6 +2040,17 @@ class Rollout:
             finally:
                 self._usage_runtime = None
 
+        rollout_dir = getattr(self, "_rollout_dir", None)
+        if (
+            getattr(self, "_egress_denylist", None) is not None
+            and self._env is not None
+            and rollout_dir is not None
+        ):
+            try:
+                await self._planes.stop_egress_denylist(self._env, rollout_dir)
+            except Exception as e:
+                logger.warning(f"Egress denylist proxy stop failed: {e}")
+
         self._finalize_usage_metrics()
         self._enforce_required_usage_tracking()
 
@@ -2272,9 +2311,16 @@ class Rollout:
         if disallow_web_tools is None:
             disallow_web_tools = _task_disallows_internet(getattr(self, "_task", None))
         disallow_web_tools = bool(disallow_web_tools and role.agent != "oracle")
+        egress_denylist = (
+            None
+            if disallow_web_tools or role.agent == "oracle"
+            else _task_egress_denylist(getattr(self, "_task", None))
+        )
+        disallow_hosted_search = egress_denylist is not None
         agent_launch = self._planes.agent_launch(
             role.agent,
             disallow_web_tools=disallow_web_tools,
+            disallow_hosted_search=disallow_hosted_search,
         )
         agent_env = _apply_web_policy(
             self._planes.resolve_agent_env(
@@ -2296,8 +2342,10 @@ class Rollout:
             sandbox_setup_timeout=cfg.sandbox_setup_timeout,
             required_skill_names=getattr(self, "_required_skill_names", ()),
             live_trajectory_path=rollout_dir / "trajectory" / "llm_trajectory.jsonl",
-            force_sandbox_local=disallow_web_tools,
+            force_sandbox_local=disallow_web_tools or disallow_hosted_search,
         )
+        if egress_denylist is not None:
+            agent_env = denylist_agent_env(agent_env)
 
         role_agent_differs = role.agent != cfg.primary_agent
         needs_role_credentials = (
@@ -2342,6 +2390,7 @@ class Rollout:
                 agent_cfg,
                 cred_home,
                 disallow=disallow_web_tools,
+                disallow_hosted_search=disallow_hosted_search,
             )
 
         self._agent_launch = agent_launch
@@ -2349,6 +2398,8 @@ class Rollout:
         sf_entrypoint = self._session_factory_entrypoint(role.agent)
         self._is_session_factory = sf_entrypoint is not None
         if sf_entrypoint is not None:
+            if egress_denylist is not None:
+                raise RuntimeError("network_mode='denylist' requires an ACP agent")
             (
                 self._acp_client,
                 self._session,
@@ -2368,6 +2419,8 @@ class Rollout:
                 agent_cwd=self._agent_cwd,
             )
         else:
+            if egress_denylist is not None:
+                await self._start_egress_denylist(egress_denylist)
             (
                 self._acp_client,
                 self._session,

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/rollout/_setup.py

@@ -42,6 +42,7 @@ from benchflow.rewards.validation import (
     validate_reward_map,
 )
 from benchflow.rollout._results import _DIAG_TRUNCATE
+from benchflow.sandbox.egress_denylist import EgressDenylist, egress_denylist_for
 from benchflow.trajectories.types import redact_acp_trajectory_jsonl
 
 logger = logging.getLogger(__name__)
@@ -55,6 +56,11 @@ def _task_disallows_internet(task: Any) -> bool:
     return getattr(env_config, "allow_internet", True) is False
 
 
+def _task_egress_denylist(task: Any) -> EgressDenylist | None:
+    """Return the egress denylist the task's sandbox config declares, if any."""
+    return egress_denylist_for(getattr(getattr(task, "config", None), "sandbox", None))
+
+
 def _read_task_instruction(task_path: Path) -> str:
     """Read the agent-facing instruction from legacy files or ``task.md``."""
     document_path = task_path / "task.md"

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/rollout/task_runtime.py

@@ -159,6 +159,11 @@ class TaskRuntime:
         rollout = await Rollout.create(self.config.to_rollout_config())
         try:
             await rollout.setup()
+            if getattr(rollout, "_egress_denylist", None) is not None:
+                raise RuntimeError(
+                    "network_mode='denylist' requires an ACP agent rollout; "
+                    "the bash primitive runs without the egress proxy"
+                )
             await rollout.start()
             await rollout.install_agent()
         except BaseException:

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/rollout_planes.py

@@ -31,6 +31,11 @@ from benchflow.providers.runtime import (
     extract_usage,
     stop_provider_runtime,
 )
+from benchflow.sandbox.egress_denylist import (
+    EgressDenylist,
+    start_egress_denylist,
+    stop_egress_denylist,
+)
 from benchflow.sandbox.lockdown import (
     _resolve_locked_paths,
     _seed_verifier_workspace,
@@ -53,13 +58,21 @@ from benchflow.sandbox.setup import (
 class DefaultRolloutPlanes:
     """Default bindings for the four concrete planes."""
 
-    def agent_launch(self, agent: str, *, disallow_web_tools: bool) -> str:
+    def agent_launch(
+        self,
+        agent: str,
+        *,
+        disallow_web_tools: bool,
+        disallow_hosted_search: bool = False,
+    ) -> str:
         launch = AGENT_LAUNCH.get(agent, agent)
-        if not disallow_web_tools:
-            return launch
         agent_cfg = AGENTS.get(agent)
-        if agent_cfg and agent_cfg.disallow_web_tools_launch_suffix:
+        if agent_cfg is None:
+            return launch
+        if disallow_web_tools:
             return launch + agent_cfg.disallow_web_tools_launch_suffix
+        if disallow_hosted_search:
+            return launch + agent_cfg.disallow_hosted_search_launch_suffix
         return launch
 
     def agent_config(self, agent: str) -> Any:
@@ -179,6 +192,14 @@ class DefaultRolloutPlanes:
     async def stop_provider_runtime(self, runtime: Any) -> None:
         await stop_provider_runtime(runtime)
 
+    async def start_egress_denylist(
+        self, env: Any, sandbox_user: str | None, denylist: EgressDenylist
+    ) -> None:
+        await start_egress_denylist(env, sandbox_user, denylist)
+
+    async def stop_egress_denylist(self, env: Any, rollout_dir: Path) -> None:
+        await stop_egress_denylist(env, rollout_dir)
+
     def extract_usage(self, runtime: Any) -> dict[str, Any]:
         return extract_usage(runtime)
 

{benchflow-0.7.6.dev2266 → benchflow-0.7.7.dev2305}/src/benchflow/sandbox/_compose.py

@@ -9,6 +9,7 @@ COMPOSE_BASE_PATH = COMPOSE_DIR / "docker-compose-base.yaml"
 COMPOSE_BUILD_PATH = COMPOSE_DIR / "docker-compose-build.yaml"
 COMPOSE_PREBUILT_PATH = COMPOSE_DIR / "docker-compose-prebuilt.yaml"
 COMPOSE_NO_NETWORK_PATH = COMPOSE_DIR / "docker-compose-no-network.yaml"
+COMPOSE_NET_ADMIN_PATH = COMPOSE_DIR / "docker-compose-net-admin.yaml"
 
 # Back-off delays for retrying a `compose up` that hit a daemon-side network
 # create/attach race. Shared by the host docker.py path and the Daytona DinD

benchflow-0.7.7.dev2305/src/benchflow/sandbox/_compose_files/docker-compose-net-admin.yaml

@@ -0,0 +1,4 @@
+services:
+  main:
+    cap_add:
+      - NET_ADMIN