benchflow 0.7.6.dev2255__tar.gz → 0.7.7__tar.gz

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/CHANGELOG.md

@@ -2,6 +2,41 @@
 
 ## [Unreleased]
 
+## 0.7.7 — 2026-09-09
+
+### Added
+- **`network_mode: denylist` blocks a list of URLs and hosts for the agent on
+  Docker and Daytona.** The task keeps internet access; `blocked_urls` and
+  `blocked_hosts` are enforced by a root-owned loopback proxy behind the
+  sandbox-user firewall, hosted search tools are switched off per harness,
+  and every refused request lands in `trajectory/egress_denylist.jsonl`.
+  Other backends refuse the mode at preflight. (#1113)
+
+### Fixed
+
+- **Denylist egress no longer cuts the agent off from its own model.** The
+  controller registers the running LiteLLM gateway's exact
+  `127.0.0.1:<port>` endpoint with the proxy, so clients that ignore
+  `NO_PROXY` (Gemini's Undici `ProxyAgent` tunnels even plain HTTP) still
+  reach it; the exception comes from the live gateway, never task metadata or
+  agent-supplied environment, and every other private destination stays
+  blocked. Reconnects re-register the current port. (#1118)
+- **The denylist is a shared sandbox policy, not a per-harness one.** Every
+  supported ACP harness — custom registrations included — gets the same proxy,
+  certificates, and UID firewall on primary connections and later roles alike,
+  independent of harness name, model id, or provider. (#1118)
+- **`codex-acp` sessions are configured through native ACP settings.** Hosted
+  search is switched off via `CODEX_CONFIG.web_search` (the CLI ignores `-c`
+  overrides), and Codex runs in `agent-full-access` session mode when
+  BenchFlow has already selected a non-root sandbox user, so its bubblewrap
+  sandbox is not nested inside Docker or Daytona — BenchFlow's own user,
+  filesystem restrictions, proxy, and firewall still apply. (#1118)
+- **`cryptography>=44` is a core dependency.** Denylist egress mints TLS
+  certificates on both Docker and Daytona, so the pin moved out of the
+  `sandbox-agentcore` extra. (#1118)
+
+## 0.7.6 — 2026-09-04
+
 ### Added
 - **ACP rollout directories render as an interactive reviewer page.**
   `bench eval view <rollout>` now assembles a JSON payload (normalized
@@ -42,6 +77,48 @@
   timestamps.** Steps gain a `+m:ss` offset chip and tool calls a duration
   chip whenever events carry `ts` / `started_at` / `finished_at` fields;
   with today's captures (which carry none) nothing changes visually.
+- **Z.ai Coding Plan routing.** Coding Plan subscriptions route through the
+  provider layer, with validated generation parameters forwarded and explicit
+  provider endpoints preserved by the LiteLLM proxy. (#1074)
+- **`bench eval run` publishes job artifacts and eval results.**
+  `--publish-bucket` syncs a job directory to a HuggingFace storage bucket (an
+  alternative to `--publish-hf`'s dataset-repo upload), and
+  `--eval-results-model/-dataset/-task` opens a community eval-results PR
+  (`.eval_results/*.yaml`) on a model's HF repo, scored from the run's mean
+  reward. (#1035)
+
+### Fixed
+
+- **`claude-fable-5-1` can run: the `claude-agent-acp` pin moves 0.40.0 to
+  0.73.0.** The old pin bundled `@anthropic-ai/claude-agent-sdk` 0.3.160, and
+  the model rejects Claude Code older than 2.1.251 with
+  `claude_code_version_too_old` (HTTP 400) — so every rollout failed on its
+  first API call. The new pin bundles sdk 0.3.257, and the `set_config_option`
+  `"model"` / `"effort"` wiring is re-verified against it by
+  `tests/test_acp_pinned_protocol_guard.py`. (#1086)
+- **`codex-acp` dispatch modernized.** The shim pin moves to 1.6.0 and
+  reasoning effort travels inside the model id (`modelId[effort]`) through
+  `session/set_model`, rather than a config option the shim rejects. (#1044)
+- **ACP usage and terminal evidence survive an agent timeout**, instead of
+  being dropped with the timed-out turn. (#1080)
+- **A pending tool call can no longer defer the idle watchdog without
+  bound** — the deferral is capped. (#1066)
+- **`bench eval` resume re-runs infra-retryable verifier-errored tasks**
+  rather than treating the infrastructure failure as a settled result. (#1063)
+- **Sandbox hardening execs draw on the verifier-setup budget**, so hardening
+  no longer competes with the agent's own time. (#1062)
+- **`--trials > 1` without `--matrix` is rejected up front.** (#1064)
+- **Healthy native ACP subscription results are preserved.** (#1049)
+- **The agent judge parses native edit targets.** (#1069)
+- **Non-dict rewards no longer break completed-outcome classification.** (#1054)
+- **Codex honors proxy-owned model selection.** (#1076)
+- **Gemini routing corrections.** ACP model ids fixed and wrapped ids
+  normalized, Gemma routing supported with pass-through usage captured, every
+  Google key alias routed through the proxy, Google Gemini gateway routes
+  normalized, and headless runs trust the sandbox workspace.
+- **LiteLLM provisions the Vertex sandbox runtime.** (#985)
+- **OpenClaw caps supported model output tokens.**
+- **RestrictedPython updated to 8.3.**
 
 ## 0.7.5 — 2026-08-19
 

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.6.dev2255
+Version: 0.7.7
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -20,6 +20,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.12
 Requires-Dist: agent-client-protocol>=0.10
 Requires-Dist: anyio>=4.0
+Requires-Dist: cryptography>=44
 Requires-Dist: google-cloud-aiplatform<2.0,>=1.133.0
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: litellm[proxy]==1.91.0
@@ -58,7 +59,6 @@ Requires-Dist: openai>=1.40; extra == 'judge'
 Provides-Extra: sandbox-agentcore
 Requires-Dist: bedrock-agentcore>=1.18; extra == 'sandbox-agentcore'
 Requires-Dist: boto3>=1.43.31; extra == 'sandbox-agentcore'
-Requires-Dist: cryptography>=44; extra == 'sandbox-agentcore'
 Requires-Dist: pathspec>=0.12; extra == 'sandbox-agentcore'
 Provides-Extra: sandbox-daytona
 Requires-Dist: daytona>=0.184.0; extra == 'sandbox-daytona'

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.6.dev2255"
+version = "0.7.7"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -12,6 +12,8 @@ dependencies = [
     "pydantic>=2.7",
     "pyyaml>=6.0",
     "rich>=13.0",
+    # Denylist egress mints TLS certificates on both Docker and Daytona.
+    "cryptography>=44",
     # Dataset registry bench_version checks (PEP 440 specifier sets).
     "packaging>=24",
     "litellm[proxy]==1.91.0",
@@ -100,10 +102,6 @@ sandbox-agentcore = [
     # (InvokeAgentRuntimeCommand). The boto3 floor is the first release that
     # models the bedrock-agentcore/-control services.
     "bedrock-agentcore>=1.18",
-    # Used directly by the sealed AgentCore transport (RSA-OAEP key wrap +
-    # AES-CTR/HMAC). It otherwise arrives only transitively via
-    # litellm[proxy]; declaring it here keeps the dependency honest.
-    "cryptography>=44",
     "boto3>=1.43.31",
     # Docker-compatible build-context filtering without requiring a daemon.
     "pathspec>=0.12",

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/acp/runtime.py

@@ -621,6 +621,7 @@ async def connect_acp(
         agent_env,
         model=model,
         reasoning_effort=reasoning_effort,
+        sandboxed=bool(sandbox_user),
     )
     agent_env = await _prepare_openhands_direct_execution(
         env,

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/agents/codex_config.py

@@ -91,11 +91,23 @@ def apply_codex_launch_config(
     *,
     model: str | None,
     reasoning_effort: str | None,
+    sandboxed: bool = False,
 ) -> tuple[dict[str, str], bool]:
-    """Apply launch-owned effort and report whether config owns model selection."""
+    """Configure the adapter's sandbox, web policy and launch-owned model effort."""
     if agent != "codex-acp":
         return agent_env, False
-    config = _parse_codex_config(agent_env.get(CODEX_CONFIG_ENV))
+    updated_env = dict(agent_env)
+    if sandboxed:
+        # BenchFlow's non-root sandbox and UID firewall own isolation. The
+        # adapter defaults to workspace-write, whose nested bwrap fails on
+        # Docker/Daytona before tools run. CODEX_CONFIG.sandbox_mode does not
+        # fix this: codex-acp overrides it with its session's AgentMode.
+        updated_env.setdefault("INITIAL_AGENT_MODE", "agent-full-access")
+    disable_search = any(
+        agent_env.get(key) == "1"
+        for key in ("BENCHFLOW_DISALLOW_WEB_TOOLS", "BENCHFLOW_EGRESS_DENYLIST")
+    )
+    config = _parse_codex_config(agent_env.get(CODEX_CONFIG_ENV), strict=disable_search)
     provider_model = agent_env.get(_PROVIDER_MODEL_ENV)
     owns_model = bool(
         model
@@ -105,14 +117,15 @@ def apply_codex_launch_config(
         and config is not None
         and config.get("model") == provider_model
     )
-    if not owns_model or not reasoning_effort:
-        return agent_env, owns_model
-
-    assert config is not None
-    updated_env = dict(agent_env)
-    config["model_reasoning_effort"] = reasoning_effort
-    updated_env[CODEX_CONFIG_ENV] = json.dumps(config, separators=(",", ":"))
-    return updated_env, True
+    if config is not None and (disable_search or (owns_model and reasoning_effort)):
+        if disable_search:
+            # codex-acp 1.6.0 ignores CLI -c flags; its supported CODEX_CONFIG
+            # is forwarded to the Codex app-server's thread configuration.
+            config["web_search"] = "disabled"
+        if owns_model and reasoning_effort:
+            config["model_reasoning_effort"] = reasoning_effort
+        updated_env[CODEX_CONFIG_ENV] = json.dumps(config, separators=(",", ":"))
+    return (updated_env if updated_env != agent_env else agent_env), owns_model
 
 
 def _apply_codex_default_auth_request(

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/agents/install.py

@@ -99,7 +99,7 @@ def _owner_from_home(home: str) -> str | None:
 
 
 def _policy_home_dirs(agent: str, agent_cfg: AgentConfig) -> list[str]:
-    """Agent home dirs a no-web setup command may create."""
+    """Agent home dirs a web-policy setup command may create."""
     dirs = set(agent_cfg.home_dirs)
     for owned_path in agent_cfg.disallow_web_tools_owned_paths:
         if not owned_path.startswith("$HOME/"):
@@ -265,15 +265,21 @@ async def apply_web_tool_policy(
     home: str,
     *,
     disallow: bool,
+    disallow_hosted_search: bool = False,
 ) -> None:
-    """Apply an agent-specific hard web-tool disable in the agent home."""
-    if not disallow or not agent_cfg or not agent_cfg.disallow_web_tools_setup_cmd:
+    """Apply the agent's no-web or hosted-search-off setup command in the agent home."""
+    if not agent_cfg:
+        return
+    if disallow:
+        policy, setup_cmd = "no-web", agent_cfg.disallow_web_tools_setup_cmd
+    elif disallow_hosted_search:
+        policy, setup_cmd = "hosted-search", agent_cfg.disallow_hosted_search_setup_cmd
+    else:
+        return
+    if not setup_cmd:
         return
 
-    cmd = (
-        f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; "
-        f"{agent_cfg.disallow_web_tools_setup_cmd}"
-    )
+    cmd = f"export BENCHFLOW_AGENT_HOME={shlex.quote(home)}; {setup_cmd}"
     owner = _owner_from_home(home)
     if owner:
         q_owner = shlex.quote(owner)
@@ -296,7 +302,7 @@ async def apply_web_tool_policy(
         if stderr:
             details.append(f"stderr: {stderr}")
         raise RuntimeError(
-            f"Failed to apply no-web policy for {agent}: {'; '.join(details)}"
+            f"Failed to apply {policy} policy for {agent}: {'; '.join(details)}"
         )
 
 

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/agents/manifest.py

@@ -89,6 +89,8 @@ _SHIM_ONLY = frozenset(
         "disallow_web_tools_setup_cmd",
         "disallow_web_tools_owned_paths",
         "disallow_web_tools_launch_suffix",
+        "disallow_hosted_search_setup_cmd",
+        "disallow_hosted_search_launch_suffix",
         "task_mcp_transport",
         "task_mcp_config_path",
     }

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/agents/registry.py

@@ -502,6 +502,12 @@ class AgentConfig:
     # String appended to launch_cmd when BenchFlow's no-web policy is active.
     # Use for agents whose supported toggle is a launch/config override.
     disallow_web_tools_launch_suffix: str = ""
+    # Shell snippet that switches off hosted (provider-side) search tools when
+    # the denylist egress policy is active; local fetch tools stay on because
+    # they go through the egress proxy. Reuses disallow_web_tools_owned_paths.
+    disallow_hosted_search_setup_cmd: str = ""
+    # String appended to launch_cmd when the denylist egress policy is active.
+    disallow_hosted_search_launch_suffix: str = ""
     # How task-declared MCP servers are delivered to the agent:
     # "acp" sends them in session/new; "native-config" writes an agent-specific
     # config file before launch (for agents whose ACP server drops/reformats
@@ -511,6 +517,15 @@ class AgentConfig:
     task_mcp_config_path: str = ""
 
 
+_GEMINI_EXCLUDE_WEB_TOOLS_CMD = _json_settings_merge(
+    "$BENCHFLOW_AGENT_HOME/.gemini/settings.json",
+    'd.setdefault("tools",{}).setdefault("exclude",[]);'
+    '[d["tools"]["exclude"].append(t) for t in '
+    '["google_web_search","web_fetch"] '
+    'if t not in d["tools"]["exclude"]]',
+)
+
+
 # Agent registry — all supported agents
 AGENTS: dict[str, AgentConfig] = {
     "claude-agent-acp": AgentConfig(
@@ -552,6 +567,12 @@ AGENTS: dict[str, AgentConfig] = {
             'if t not in d["permissions"]["deny"]]',
         ),
         disallow_web_tools_owned_paths=["$HOME/.claude"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.claude/settings.json",
+            'd.setdefault("permissions",{}).setdefault("deny",[]);'
+            '[d["permissions"]["deny"].append(t) for t in ["WebSearch"] '
+            'if t not in d["permissions"]["deny"]]',
+        ),
         supports_acp_set_model=False,
         acp_model_config_id="model",
         acp_effort_config_id="effort",
@@ -649,7 +670,8 @@ AGENTS: dict[str, AgentConfig] = {
                 HostAuthFile("~/.codex/auth.json", "{home}/.codex/auth.json"),
             ],
         ),
-        disallow_web_tools_launch_suffix=" -c tools.web_search=false",
+        # codex-acp ignores CLI -c flags. codex_config applies the web policy
+        # through CODEX_CONFIG before launching the adapter.
     ),
     "gemini": AgentConfig(
         name="gemini",
@@ -694,14 +716,11 @@ AGENTS: dict[str, AgentConfig] = {
                 ),
             ],
         ),
-        disallow_web_tools_setup_cmd=_json_settings_merge(
-            "$BENCHFLOW_AGENT_HOME/.gemini/settings.json",
-            'd.setdefault("tools",{}).setdefault("exclude",[]);'
-            '[d["tools"]["exclude"].append(t) for t in '
-            '["google_web_search","web_fetch"] '
-            'if t not in d["tools"]["exclude"]]',
-        ),
+        disallow_web_tools_setup_cmd=_GEMINI_EXCLUDE_WEB_TOOLS_CMD,
         disallow_web_tools_owned_paths=["$HOME/.gemini"],
+        # web_fetch tries the hosted urlContext path before any local fetch, so
+        # the egress proxy cannot filter it; the denylist excludes both tools.
+        disallow_hosted_search_setup_cmd=_GEMINI_EXCLUDE_WEB_TOOLS_CMD,
     ),
     "opencode": AgentConfig(
         name="opencode",
@@ -730,6 +749,10 @@ AGENTS: dict[str, AgentConfig] = {
             'd.setdefault("tools",{})["webfetch"]=False',
         ),
         disallow_web_tools_owned_paths=["$HOME/.config/opencode"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.config/opencode/opencode.json",
+            'd.setdefault("tools",{})["websearch"]=False',
+        ),
     ),
     "mimo": AgentConfig(
         name="mimo",
@@ -779,6 +802,10 @@ AGENTS: dict[str, AgentConfig] = {
             'd.setdefault("tools",{})["webfetch"]=False',
         ),
         disallow_web_tools_owned_paths=["$HOME/.config/mimocode"],
+        disallow_hosted_search_setup_cmd=_json_settings_merge(
+            "$BENCHFLOW_AGENT_HOME/.config/mimocode/mimocode.json",
+            'd.setdefault("tools",{})["websearch"]=False',
+        ),
     ),
     "harvey-lab-harness": AgentConfig(
         name="harvey-lab-harness",
@@ -1183,6 +1210,8 @@ def _acpx_wrap(config: AgentConfig) -> AgentConfig:
         disallow_web_tools_setup_cmd=config.disallow_web_tools_setup_cmd,
         disallow_web_tools_owned_paths=config.disallow_web_tools_owned_paths,
         disallow_web_tools_launch_suffix=config.disallow_web_tools_launch_suffix,
+        disallow_hosted_search_setup_cmd=config.disallow_hosted_search_setup_cmd,
+        disallow_hosted_search_launch_suffix=config.disallow_hosted_search_launch_suffix,
         task_mcp_transport=config.task_mcp_transport,
         task_mcp_config_path=config.task_mcp_config_path,
     )
@@ -1377,6 +1406,8 @@ def register_agent(
     disallow_web_tools_setup_cmd: str = "",
     disallow_web_tools_owned_paths: list[str] | None = None,
     disallow_web_tools_launch_suffix: str = "",
+    disallow_hosted_search_setup_cmd: str = "",
+    disallow_hosted_search_launch_suffix: str = "",
 ) -> AgentConfig:
     """Register a custom agent at runtime.
 
@@ -1416,6 +1447,8 @@ def register_agent(
         disallow_web_tools_setup_cmd=disallow_web_tools_setup_cmd,
         disallow_web_tools_owned_paths=disallow_web_tools_owned_paths or [],
         disallow_web_tools_launch_suffix=disallow_web_tools_launch_suffix,
+        disallow_hosted_search_setup_cmd=disallow_hosted_search_setup_cmd,
+        disallow_hosted_search_launch_suffix=disallow_hosted_search_launch_suffix,
     )
     AGENTS[name] = config
     AGENT_INSTALLERS[name] = install_cmd

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/contracts/planes.py

@@ -8,10 +8,13 @@ the composition edge so the kernel does not import those concrete modules.
 from __future__ import annotations
 
 from pathlib import Path
-from typing import Any, Protocol, runtime_checkable
+from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
 
 from benchflow.environment.manifest import EnvironmentManifest
 
+if TYPE_CHECKING:
+    from benchflow.sandbox.egress_denylist import EgressDenylist
+
 
 class LiveUsageGateway(Protocol):
     """A provider gateway process that reports live cumulative token usage.
@@ -33,7 +36,13 @@ class LiveUsageGateway(Protocol):
 class RolloutPlanes(Protocol):
     """Concrete-plane operations the rollout kernel needs."""
 
-    def agent_launch(self, agent: str, *, disallow_web_tools: bool) -> str: ...
+    def agent_launch(
+        self,
+        agent: str,
+        *,
+        disallow_web_tools: bool,
+        disallow_hosted_search: bool = False,
+    ) -> str: ...
     def agent_config(self, agent: str) -> Any: ...
 
     def resolve_agent_env(
@@ -102,6 +111,15 @@ class RolloutPlanes(Protocol):
     async def link_skill_paths(self, *args: Any, **kwargs: Any) -> None: ...
     async def ensure_litellm_runtime(self, *args: Any, **kwargs: Any) -> Any: ...
     async def stop_provider_runtime(self, runtime: Any) -> None: ...
+    async def start_egress_denylist(
+        self,
+        env: Any,
+        sandbox_user: str | None,
+        denylist: EgressDenylist,
+        *,
+        model_gateway_url: str | None = None,
+    ) -> None: ...
+    async def stop_egress_denylist(self, env: Any, rollout_dir: Path) -> None: ...
     def extract_usage(self, runtime: Any) -> dict[str, Any]: ...
     async def connect_acp(self, *args: Any, **kwargs: Any) -> Any: ...
     async def execute_prompts(self, *args: Any, **kwargs: Any) -> Any: ...

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/contracts/user.py

@@ -62,7 +62,7 @@ class FunctionUser(BaseUser):
         self,
         fn: Callable[
             [int, str, RoundResult | None],
-            str | None | Awaitable[str | None],
+            str | Awaitable[str | None] | None,
         ],
     ) -> None:
         self._fn = fn

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/evaluation.py

@@ -64,6 +64,7 @@ from benchflow._utils.scoring import (
     classify_verifier_error,
     count_audit_outcomes,
     count_score_outcomes,
+    extract_reward,
     mean_scored_reward,
     pass_rate,
     pass_rate_excl_errors,
@@ -431,8 +432,7 @@ def _classify_completed_outcomes(
     """
     passed = failed = errored = 0
     for r in completed.values():
-        rewards = r.get("rewards") if isinstance(r, dict) else None
-        reward = rewards.get("reward") if rewards else None
+        reward = extract_reward(r) if isinstance(r, dict) else None
         if reward == 1:
             passed += 1
         elif reward is not None:
@@ -1107,6 +1107,17 @@ class Evaluation:
                 r = json.loads(rfile.read_text())
                 task = r["task_name"]
                 if r.get("rewards") is not None or r.get("verifier_error"):
+                    rewards = r.get("rewards")
+                    if rewards is not None and not isinstance(rewards, dict):
+                        logger.warning(
+                            "Malformed rewards field in %s for task %r: "
+                            "expected dict or null, got %s %r — "
+                            "treating as no reward (task will count as errored)",
+                            rfile,
+                            task,
+                            type(rewards).__name__,
+                            rewards,
+                        )
                     rank = (
                         r.get("rewards") is not None,
                         rfile.stat().st_mtime,

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/rollout/__init__.py

@@ -149,6 +149,7 @@ from benchflow.rollout._setup import _start_env_and_upload as _start_env_and_upl
 from benchflow.rollout._setup import (
     _task_disallows_internet as _task_disallows_internet,
 )
+from benchflow.rollout._setup import _task_egress_denylist as _task_egress_denylist
 from benchflow.rollout._setup import _verify_rollout as _verify_rollout
 from benchflow.rollout._skills import (
     _resolve_skill_creator_root as _resolve_skill_creator_root,
@@ -197,6 +198,7 @@ from benchflow.rollout.task_runtime import TaskRuntimeConfig as TaskRuntimeConfi
 from benchflow.rollout.task_runtime import TaskRuntimeResult as TaskRuntimeResult
 from benchflow.rollout_branch import ChildRunner
 from benchflow.rollout_branch import branch as _branch_engine
+from benchflow.sandbox.egress_denylist import EgressDenylist, denylist_agent_env
 from benchflow.sandbox.metadata import persist_sandbox_info
 from benchflow.scenes import compile_scenes_to_steps
 from benchflow.scenes import scene_step_prompt as scene_step_prompt
@@ -658,6 +660,8 @@ class Rollout:
         self._timing: dict[str, float] = {}
         self._effective_locked: list[str] = []
         self._disallow_web_tools: bool = False
+        self._egress_denylist: EgressDenylist | None = None
+        self._disallow_hosted_search: bool = False
         self._effective_skills_dir: Path | None = None
         self._effective_skills_sandbox_dir: str | None = None
         # Task dir actually deployed: a temp copy (self._task_tmp) when
@@ -952,6 +956,14 @@ class Rollout:
         self._disallow_web_tools = (
             _task_disallows_internet(self._task) or cfg.self_gen_no_internet
         ) and cfg.primary_agent != "oracle"
+        self._egress_denylist = (
+            None
+            if self._disallow_web_tools or cfg.primary_agent == "oracle"
+            else _task_egress_denylist(self._task)
+        )
+        if self._egress_denylist is not None and not cfg.sandbox_user:
+            raise ValueError("network_mode='denylist' requires a sandbox_user")
+        self._disallow_hosted_search = self._egress_denylist is not None
         self._agent_env = _apply_web_policy(
             self._planes.resolve_agent_env(
                 cfg.primary_agent, cfg.primary_model, cfg.agent_env
@@ -973,6 +985,7 @@ class Rollout:
         self._agent_launch = self._planes.agent_launch(
             cfg.primary_agent,
             disallow_web_tools=self._disallow_web_tools,
+            disallow_hosted_search=self._disallow_hosted_search,
         )
 
         # Copy task dir to temp when Dockerfile mutations are needed
@@ -1231,6 +1244,7 @@ class Rollout:
             self._agent_cfg,
             cred_home,
             disallow=self._disallow_web_tools,
+            disallow_hosted_search=self._disallow_hosted_search,
         )
         await self._planes.snapshot_build_config(self._env, workspace=self._agent_cwd)
         await self._planes.seed_verifier_workspace(
@@ -1274,11 +1288,22 @@ class Rollout:
             return cfg.session_factory
         return None
 
+    async def _start_egress_denylist(self, denylist: EgressDenylist) -> None:
+        """(Re)start the egress proxy before an ACP connection; a restored sandbox has none running."""
+        runtime = self._usage_runtime
+        await self._planes.start_egress_denylist(
+            self._env,
+            self._config.sandbox_user,
+            denylist,
+            model_gateway_url=runtime.agent_base_url if runtime is not None else None,
+        )
+
     async def connect(self) -> None:
         """Open an ACP connection to the agent. Can be called multiple times."""
         cfg = self._config
         rollout_dir = self._require_rollout_dir()
         t0 = datetime.now()
+        egress_denylist = getattr(self, "_egress_denylist", None)
 
         (
             self._agent_env,
@@ -1295,11 +1320,16 @@ class Rollout:
             sandbox_setup_timeout=cfg.sandbox_setup_timeout,
             required_skill_names=getattr(self, "_required_skill_names", ()),
             live_trajectory_path=rollout_dir / "trajectory" / "llm_trajectory.jsonl",
-            force_sandbox_local=getattr(self, "_disallow_web_tools", False),
+            force_sandbox_local=getattr(self, "_disallow_web_tools", False)
+            or egress_denylist is not None,
         )
+        if egress_denylist is not None:
+            self._agent_env = denylist_agent_env(self._agent_env)
         sf_entrypoint = self._session_factory_entrypoint(cfg.primary_agent)
         self._is_session_factory = sf_entrypoint is not None
         if sf_entrypoint is not None:
+            if egress_denylist is not None:
+                raise RuntimeError("network_mode='denylist' requires an ACP agent")
             (
                 self._acp_client,
                 self._session,
@@ -1317,6 +1347,8 @@ class Rollout:
                 agent_cwd=self._agent_cwd,
             )
         else:
+            if egress_denylist is not None:
+                await self._start_egress_denylist(egress_denylist)
             (
                 self._acp_client,
                 self._session,
@@ -2012,6 +2044,17 @@ class Rollout:
             finally:
                 self._usage_runtime = None
 
+        rollout_dir = getattr(self, "_rollout_dir", None)
+        if (
+            getattr(self, "_egress_denylist", None) is not None
+            and self._env is not None
+            and rollout_dir is not None
+        ):
+            try:
+                await self._planes.stop_egress_denylist(self._env, rollout_dir)
+            except Exception as e:
+                logger.warning(f"Egress denylist proxy stop failed: {e}")
+
         self._finalize_usage_metrics()
         self._enforce_required_usage_tracking()
 
@@ -2272,9 +2315,16 @@ class Rollout:
         if disallow_web_tools is None:
             disallow_web_tools = _task_disallows_internet(getattr(self, "_task", None))
         disallow_web_tools = bool(disallow_web_tools and role.agent != "oracle")
+        egress_denylist = (
+            None
+            if disallow_web_tools or role.agent == "oracle"
+            else _task_egress_denylist(getattr(self, "_task", None))
+        )
+        disallow_hosted_search = egress_denylist is not None
         agent_launch = self._planes.agent_launch(
             role.agent,
             disallow_web_tools=disallow_web_tools,
+            disallow_hosted_search=disallow_hosted_search,
         )
         agent_env = _apply_web_policy(
             self._planes.resolve_agent_env(
@@ -2296,8 +2346,10 @@ class Rollout:
             sandbox_setup_timeout=cfg.sandbox_setup_timeout,
             required_skill_names=getattr(self, "_required_skill_names", ()),
             live_trajectory_path=rollout_dir / "trajectory" / "llm_trajectory.jsonl",
-            force_sandbox_local=disallow_web_tools,
+            force_sandbox_local=disallow_web_tools or disallow_hosted_search,
         )
+        if egress_denylist is not None:
+            agent_env = denylist_agent_env(agent_env)
 
         role_agent_differs = role.agent != cfg.primary_agent
         needs_role_credentials = (
@@ -2342,6 +2394,7 @@ class Rollout:
                 agent_cfg,
                 cred_home,
                 disallow=disallow_web_tools,
+                disallow_hosted_search=disallow_hosted_search,
             )
 
         self._agent_launch = agent_launch
@@ -2349,6 +2402,8 @@ class Rollout:
         sf_entrypoint = self._session_factory_entrypoint(role.agent)
         self._is_session_factory = sf_entrypoint is not None
         if sf_entrypoint is not None:
+            if egress_denylist is not None:
+                raise RuntimeError("network_mode='denylist' requires an ACP agent")
             (
                 self._acp_client,
                 self._session,
@@ -2368,6 +2423,8 @@ class Rollout:
                 agent_cwd=self._agent_cwd,
             )
         else:
+            if egress_denylist is not None:
+                await self._start_egress_denylist(egress_denylist)
             (
                 self._acp_client,
                 self._session,

{benchflow-0.7.6.dev2255 → benchflow-0.7.7}/src/benchflow/rollout/_setup.py

@@ -42,6 +42,7 @@ from benchflow.rewards.validation import (
     validate_reward_map,
 )
 from benchflow.rollout._results import _DIAG_TRUNCATE
+from benchflow.sandbox.egress_denylist import EgressDenylist, egress_denylist_for
 from benchflow.trajectories.types import redact_acp_trajectory_jsonl
 
 logger = logging.getLogger(__name__)
@@ -55,6 +56,11 @@ def _task_disallows_internet(task: Any) -> bool:
     return getattr(env_config, "allow_internet", True) is False
 
 
+def _task_egress_denylist(task: Any) -> EgressDenylist | None:
+    """Return the egress denylist the task's sandbox config declares, if any."""
+    return egress_denylist_for(getattr(getattr(task, "config", None), "sandbox", None))
+
+
 def _read_task_instruction(task_path: Path) -> str:
     """Read the agent-facing instruction from legacy files or ``task.md``."""
     document_path = task_path / "task.md"