benchflow 0.6.5__tar.gz → 0.6.7.dev1798__tar.gz

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/CHANGELOG.md

@@ -2,6 +2,99 @@
 
 ## [Unreleased]
 
+### Fixed
+- Isolated pull-request and post-test integration concurrency groups so a
+  credential-free workflow completion cannot cancel an active PR rollout.
+
+## 0.6.6 — 2026-08-04
+
+### Added
+- **Apple Container and Amazon Bedrock AgentCore sandboxes.** Apple Silicon
+  users can run supported single-container arm64 tasks through Apple's
+  Virtualization.framework, while `--sandbox agentcore` builds task-specific
+  AWS runtimes with lease-aware cleanup for supported public-network,
+  single-container arm64 tasks. (#936, #937)
+- **Rubric review (`bench review`).** Detached agentic grading of finished
+  rollouts against a `rubric.json` — rubric contract **v0.1**: an object with
+  a `criteria` list, each entry carrying a `name` (structured-output field), a
+  `description` (author documentation, never shown to the reviewer), and
+  `guidance` (the grading contract). The document carries no in-file
+  version key. One
+  reviewer agent per rollout runs as an ordinary rollout of a throwaway
+  wrapper task on a digest-pinned multi-architecture base image and no
+  task-authored Dockerfile
+  (AgentCore still builds a derived runtime image), reads a read-only evidence
+  copy of the rollout and, when admitted from a trusted `--tasks-root` with a
+  verified digest, its task, and answers every criterion with `pass` / `fail` /
+  `not_applicable` plus an explanation. The wrapper's own reward means only
+  "the reviewer produced a structurally valid result"; graded outcomes land
+  in `review_report.json`. Reviews never modify a reviewed rollout's rewards
+  or `result.json`. Rubric resolution: `-r` > the task's own
+  `verifier/rubric.json` > a built-in default (`reward_hacking`,
+  `task_specification`). `--passing` / `--failing` filter the rollouts under
+  review; a job-level prose summary aggregates multi-rollout runs. The
+  default reviewer harness is `opencode` (pinned to `1.18.11`).
+  Evidence mounts at `/evidence` outside the agent workdir (root-owned,
+  unwritable by the reviewer), symlinks are dropped rather than
+  dereferenced, task skills, shipped rubrics, and cumulative provider-history
+  trajectories are excluded while the canonical ACP trajectory is retained;
+  dropped ACP tool observations and generic tool titles are repaired from
+  exact-ID events in the trusted provider capture,
+  reviewer egress is gateway-scoped via the sandbox lockdown flag, artifact
+  consumption is pinned to each invocation's unique runtime leaf, and the
+  job summary is a deterministic aggregation.
+- **Sealed AgentCore uploads.** Every AgentCore **upload and staged
+  environment** is now encrypted end-to-end: the sandbox generates a
+  keypair, only the public key appears in command output, payloads travel
+  as AES-256-CTR ciphertext with an HMAC-SHA256 tag over IV and
+  ciphertext (verified before decryption), and the decrypted key never
+  appears in command text. Fixes provider credentials from
+  `launch_config.json` and command environments being recoverable from
+  the runtime's CloudWatch command log; the generated wrapper image
+  installs `openssl` when missing. Downloads are not sealed: they return
+  file contents as base64 through command output, so they must only carry
+  non-secret run artifacts.
+- **`RolloutConfig.uploads`.** Generic post-start host→sandbox uploads
+  (directory or file → absolute sandbox path), used by rubric review to
+  deliver evidence into prebuilt-image sandboxes and available to any caller
+  whose task data is not baked into the image.
+- **Native TRL tool-calling SFT export.** `bench train convert --format
+  trl-sft` emits conversational prompt/completion rows with a `tools` column,
+  excludes OpenCode title/summary helper calls, and accepts rollout trees,
+  canonical `results.jsonl`, or existing TRL JSONL. `bench train validate
+  --format trl-sft` can render rows with a pinned tokenizer and fail closed on
+  missing assistant masks or overlength samples. Tokenizer-aware
+  `--context-policy message-window` keeps the harness/task prefix and the
+  longest complete recent assistant/tool suffix when exact rows exceed the
+  student context window. (#925)
+- **LLM call-purpose provenance.** Captured LLM exchanges retain provider/model
+  metadata and classify agent, title, summary, compaction, and helper calls;
+  `results.jsonl` carries the metadata on each trajectory step. (#925)
+- **Live trajectory streaming and token logprobs.** Redacted LLM trajectory
+  snapshots are written during active rollouts, and training runs can opt into
+  sampled-token logprobs with `BENCHFLOW_CAPTURE_TOKEN_LOGPROBS=1`. (#922, #926)
+
+### Changed
+- OpenHands supports requested reasoning effort through its LiteLLM request
+  body and now fails closed if completed provider exchanges are missing from
+  trainer trajectories. The built-in OpenCode harness is version-pinned for
+  reproducibility. (#921, #931)
+
+### Fixed
+- Restored the documented `test` → live `integration-light` → internal-preview
+  publication chain for successful `main` pushes, pinned to the exact tested
+  commit and fail-closed against untrusted workflow sources.
+- Fixed OpenHands startup outside root-owned workdirs, OpenCode gateway setup
+  in non-Python task images, Qwen3.5 TRL SFT tokenization, and incomplete agent
+  skill-path validation. (#919, #924, #929, #932)
+- Moved recursive review evidence work and blocking platform probes off the
+  async event loop so concurrent reviews and rollouts retain responsive
+  scheduling and timeouts.
+- Remove staged Docker and Apple Container credential files when live process
+  launch fails before the agent can source and unlink them.
+- Corrected release-facing authentication, sandbox, trajectory-artifact,
+  branching, and provider documentation to match the shipped interfaces.
+
 ## 0.6.5 — 2026-07-10
 
 ### Added

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: benchflow
-Version: 0.6.5
+Version: 0.6.7.dev1798
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -21,7 +21,7 @@ Requires-Python: >=3.12
 Requires-Dist: agent-client-protocol>=0.10
 Requires-Dist: anyio>=4.0
 Requires-Dist: httpx>=0.27.0
-Requires-Dist: litellm[proxy]==1.89.0
+Requires-Dist: litellm[proxy]==1.91.0
 Requires-Dist: packaging>=24
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
@@ -34,6 +34,7 @@ Provides-Extra: deepagents
 Requires-Dist: deepagents>=0.6; extra == 'deepagents'
 Requires-Dist: langchain-openai>=0.2; extra == 'deepagents'
 Provides-Extra: dev
+Requires-Dist: pathspec>=0.12; extra == 'dev'
 Requires-Dist: pre-commit>=3.7; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
 Requires-Dist: pytest>=9.0.3; extra == 'dev'
@@ -43,6 +44,11 @@ Provides-Extra: judge
 Requires-Dist: anthropic>=0.40; extra == 'judge'
 Requires-Dist: google-genai>=1.0; extra == 'judge'
 Requires-Dist: openai>=1.40; extra == 'judge'
+Provides-Extra: sandbox-agentcore
+Requires-Dist: bedrock-agentcore>=1.18; extra == 'sandbox-agentcore'
+Requires-Dist: boto3>=1.43.31; extra == 'sandbox-agentcore'
+Requires-Dist: cryptography>=44; extra == 'sandbox-agentcore'
+Requires-Dist: pathspec>=0.12; extra == 'sandbox-agentcore'
 Provides-Extra: sandbox-daytona
 Requires-Dist: daytona>=0.184.0; extra == 'sandbox-daytona'
 Requires-Dist: tenacity>=8.0; extra == 'sandbox-daytona'
@@ -53,7 +59,7 @@ Provides-Extra: train
 Requires-Dist: transformers==5.6.2; extra == 'train'
 Provides-Extra: trl
 Requires-Dist: datasets>=2.19.0; extra == 'trl'
-Requires-Dist: trl>=0.24.0; extra == 'trl'
+Requires-Dist: trl<2,>=1.8.0; extra == 'trl'
 Description-Content-Type: text/markdown
 
 <div align="center">
@@ -77,9 +83,9 @@ BenchFlow is a universal environment framework: it runs AI agents against task e
 - **Loop strategies** — wrap any agent in a `--loop-strategy` (`verify-retry`, `self-review`); every rollout captures a per-iteration reward + token trajectory, so you can plot capability against cost (can a cheap model + loops match an expensive one at equal token spend?)
 - **`task.md` tasks** — one file (YAML frontmatter + prompt body) replaces the split `task.toml` + `instruction.md` layout; author with `bench tasks init` / `check` / `migrate` / `export`
 - **Hosted environments** — run external PrimeIntellect / Verifiers environments through `--source-env`, without converting them to BenchFlow tasks
-- **Sandboxes** — Docker locally, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments
+- **Sandboxes** — Docker locally, Apple Container on Apple Silicon Macs, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments, and AgentCore for AWS-hosted runtimes
 - **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
-- **Training-ready output** — every scored rollout emits ATIF (`trainer/atif.json`) and ADP (`trainer/adp.jsonl`) trajectory records next to the Verifiers/ORS (OpenReward) reward record
+- **Training-ready output** — scored rollouts emit a Verifiers/ORS reward record and best-effort ATIF (`trainer/atif.json`) / ADP (`trainer/adp.jsonl`) conversions; ATIF is omitted when the trajectory is empty, and conversion errors are reported in the rollout result
 
 ## Quickstart
 
@@ -88,14 +94,14 @@ BenchFlow is a universal environment framework: it runs AI agents against task e
 uv tool install --python 3.12 --upgrade benchflow
 
 # Run a benchmark: any task source, any ACP agent, any sandbox
-export GEMINI_API_KEY=...            # or claude login / codex --login for subscription auth
+export GEMINI_API_KEY=...            # or claude auth login / codex login for subscription auth
 bench eval run \
     --source-repo benchflow-ai/skillsbench --source-path tasks \
     --agent gemini --model gemini-3.1-flash-lite-preview \
     --sandbox docker
 ```
 
-Each run writes a per-task `result.json` (rewards + trajectory + token usage) and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
+Each run writes a per-task `result.json` (rewards, trajectory summary, and token usage), full events under `trajectory/`, and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
 
 ## Install
 
@@ -110,7 +116,7 @@ uv tool install --python 3.12 --upgrade benchflow
   in the install command so `uv` does not resolve an older Python-compatible
   package that lacks the CLI entrypoints.
 - If you see `Executables already exist: bench, benchflow`, re-run with `uv tool install --python 3.12 --upgrade --force benchflow` to replace stale entrypoints from an older install.
-- For Daytona or Modal extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.
+- For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.
 
 Internal users wanting the newest preview from `main` install the [internal preview channel](./docs/release.md) (`uv tool install --python 3.12 --prerelease allow --upgrade benchflow`).
 
@@ -118,7 +124,7 @@ Internal users wanting the newest preview from `main` install the [internal prev
 `--python 3.12` flag lets it provision a compatible interpreter for the tool
 install. Set `DAYTONA_API_KEY` for Daytona or configure Modal auth for Modal;
 export an agent API key (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, …) or use
-subscription auth (`claude login` / `codex --login`). Provider-prefixed models
+subscription auth (`claude auth login` / `codex login`). Provider-prefixed models
 may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` +
 `AZURE_API_ENDPOINT`.
 

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/README.md

@@ -19,9 +19,9 @@ BenchFlow is a universal environment framework: it runs AI agents against task e
 - **Loop strategies** — wrap any agent in a `--loop-strategy` (`verify-retry`, `self-review`); every rollout captures a per-iteration reward + token trajectory, so you can plot capability against cost (can a cheap model + loops match an expensive one at equal token spend?)
 - **`task.md` tasks** — one file (YAML frontmatter + prompt body) replaces the split `task.toml` + `instruction.md` layout; author with `bench tasks init` / `check` / `migrate` / `export`
 - **Hosted environments** — run external PrimeIntellect / Verifiers environments through `--source-env`, without converting them to BenchFlow tasks
-- **Sandboxes** — Docker locally, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments
+- **Sandboxes** — Docker locally, Apple Container on Apple Silicon Macs, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments, and AgentCore for AWS-hosted runtimes
 - **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
-- **Training-ready output** — every scored rollout emits ATIF (`trainer/atif.json`) and ADP (`trainer/adp.jsonl`) trajectory records next to the Verifiers/ORS (OpenReward) reward record
+- **Training-ready output** — scored rollouts emit a Verifiers/ORS reward record and best-effort ATIF (`trainer/atif.json`) / ADP (`trainer/adp.jsonl`) conversions; ATIF is omitted when the trajectory is empty, and conversion errors are reported in the rollout result
 
 ## Quickstart
 
@@ -30,14 +30,14 @@ BenchFlow is a universal environment framework: it runs AI agents against task e
 uv tool install --python 3.12 --upgrade benchflow
 
 # Run a benchmark: any task source, any ACP agent, any sandbox
-export GEMINI_API_KEY=...            # or claude login / codex --login for subscription auth
+export GEMINI_API_KEY=...            # or claude auth login / codex login for subscription auth
 bench eval run \
     --source-repo benchflow-ai/skillsbench --source-path tasks \
     --agent gemini --model gemini-3.1-flash-lite-preview \
     --sandbox docker
 ```
 
-Each run writes a per-task `result.json` (rewards + trajectory + token usage) and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
+Each run writes a per-task `result.json` (rewards, trajectory summary, and token usage), full events under `trajectory/`, and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
 
 ## Install
 
@@ -52,7 +52,7 @@ uv tool install --python 3.12 --upgrade benchflow
   in the install command so `uv` does not resolve an older Python-compatible
   package that lacks the CLI entrypoints.
 - If you see `Executables already exist: bench, benchflow`, re-run with `uv tool install --python 3.12 --upgrade --force benchflow` to replace stale entrypoints from an older install.
-- For Daytona or Modal extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.
+- For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.
 
 Internal users wanting the newest preview from `main` install the [internal preview channel](./docs/release.md) (`uv tool install --python 3.12 --prerelease allow --upgrade benchflow`).
 
@@ -60,7 +60,7 @@ Internal users wanting the newest preview from `main` install the [internal prev
 `--python 3.12` flag lets it provision a compatible interpreter for the tool
 install. Set `DAYTONA_API_KEY` for Daytona or configure Modal auth for Modal;
 export an agent API key (`GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, …) or use
-subscription auth (`claude login` / `codex --login`). Provider-prefixed models
+subscription auth (`claude auth login` / `codex login`). Provider-prefixed models
 may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` +
 `AZURE_API_ENDPOINT`.
 

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.5"
+version = "0.6.7.dev1798"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -14,7 +14,7 @@ dependencies = [
     "rich>=13.0",
     # Dataset registry bench_version checks (PEP 440 specifier sets).
     "packaging>=24",
-    "litellm[proxy]==1.89.0",
+    "litellm[proxy]==1.91.0",
     "tomli-w>=1.0",
     "typer>=0.9",
 ]
@@ -41,6 +41,9 @@ classifiers = [
 [project.optional-dependencies]
 dev = [
     "pre-commit>=3.7",
+    # AgentCore's CodeBuild path must reproduce Docker's ignore rules in tests
+    # even when the optional AWS backend is not installed.
+    "pathspec>=0.12",
     "pytest>=9.0.3",
     "pytest-asyncio>=0.24.0",
     "ruff>=0.7.0",
@@ -52,7 +55,9 @@ train = [
     "transformers==5.6.2",
 ]
 trl = [
-    "trl>=0.24.0",
+    # Native tool-calling prompt/completion rows, assistant-only masks, and
+    # training chat-template patches used by `bench train validate --format trl-sft`.
+    "trl>=1.8.0,<2",
     "datasets>=2.19.0",
 ]
 sandbox-daytona = [
@@ -68,6 +73,21 @@ sandbox-modal = [
     "modal>=0.73",
     "tenacity>=8.0",
 ]
+sandbox-agentcore = [
+    # Bedrock AgentCore Runtime microVMs. The SDK supplies the interactive
+    # shell (`open_shell`) used as the ACP transport; boto3 supplies the
+    # control plane (CreateAgentRuntime) and the command plane
+    # (InvokeAgentRuntimeCommand). The boto3 floor is the first release that
+    # models the bedrock-agentcore/-control services.
+    "bedrock-agentcore>=1.18",
+    # Used directly by the sealed AgentCore transport (RSA-OAEP key wrap +
+    # AES-CTR/HMAC). It otherwise arrives only transitively via
+    # litellm[proxy]; declaring it here keeps the dependency honest.
+    "cryptography>=44",
+    "boto3>=1.43.31",
+    # Docker-compatible build-context filtering without requiring a daemon.
+    "pathspec>=0.12",
+]
 bedrock = [
     "boto3>=1.40",
 ]

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/src/benchflow/__init__.py

@@ -60,6 +60,14 @@ from benchflow.monitor import (
     MonitorNotImplementedError,
     MonitorResult,
 )
+from benchflow.review import (
+    ReviewReport,
+    ReviewRubricError,
+    run_reviews,
+)
+from benchflow.review import Rubric as ReviewRubric
+from benchflow.review import RubricCriterion as ReviewRubricCriterion
+from benchflow.review import load_rubric as load_review_rubric
 
 # Rewards plane. Reward is the canonical node-based contract
 # (``score(node) -> VerifyResult``); RewardFunc is the legacy path-based shape
@@ -160,6 +168,12 @@ __all__ = [
     "load_rubric",
     "load_rubric_json",
     "load_rubric_toml",
+    "ReviewReport",
+    "ReviewRubric",
+    "ReviewRubricCriterion",
+    "ReviewRubricError",
+    "load_review_rubric",
+    "run_reviews",
     "Sandbox",
     "SandboxExecResult",
     "SandboxImage",

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/src/benchflow/_utils/scoring.py

@@ -216,7 +216,10 @@ def contains_verifier_dep_install_marker(text: str) -> bool:
 
 
 def _looks_like_verifier_infra_error(error: str) -> bool:
-    return any(
+    # Verifier execution uses the same sandbox transport as agent execution.
+    # Preserve retry parity for transport failures such as Daytona losing its
+    # session command between execution phases.
+    return _looks_like_infra_error(error) or any(
         marker in error
         for marker in (
             "failed to add tests directory",

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/src/benchflow/_utils/task_authoring/__init__.py

@@ -143,6 +143,32 @@ def task_digest(task_dir: Path) -> str:
     return f"sha256:{digest.hexdigest()}"
 
 
+def _check_review_rubric(verifier_dir: Path, *, verifier_label: str) -> list[str]:
+    """Validate a review ``rubric.json`` when the task ships one.
+
+    Review rubrics are ``{"criteria": [{name, description, guidance}]}``
+    JSON documents. A ``rubric.json`` that is not shaped like that (for
+    example an llm-judge rubric owned by the verifier-strategy machinery)
+    is not checked here.
+    """
+    from benchflow.review.config import (
+        ReviewRubricError,
+        is_review_rubric_file,
+        load_rubric,
+    )
+
+    candidate = verifier_dir / "rubric.json"
+    if not candidate.is_file() or not is_review_rubric_file(candidate):
+        # Absent, unreadable, or another rubric dialect (e.g. an llm-judge
+        # rubric with id/match_criteria entries) — not ours to validate.
+        return []
+    try:
+        load_rubric(candidate)
+    except ReviewRubricError as e:
+        return [f"{verifier_label}/rubric.json invalid: {e}"]
+    return []
+
+
 def check_task(
     task_dir: Path,
     *,
@@ -241,6 +267,7 @@ def check_task(
                 verifier_label=verifier_label,
             )
         )
+        issues.extend(_check_review_rubric(verifier_dir, verifier_label=verifier_label))
     else:
         issues.append(
             "Missing verifier/ directory (or legacy tests/; verifier needs "

{benchflow-0.6.5 → benchflow-0.6.7.dev1798}/src/benchflow/acp/runtime.py

@@ -25,6 +25,7 @@ from pathlib import Path
 
 from benchflow.acp.client import ACPClient
 from benchflow.acp.container_transport import ContainerTransport
+from benchflow.acp.selection import selected_acp_transport
 from benchflow.acp.types import McpServerSpec
 from benchflow.agents.protocol import ACPSessionAdapter
 from benchflow.agents.providers import (
@@ -38,9 +39,13 @@ from benchflow.diagnostics import (
     AgentPromptTimeoutError,
     IdleTimeoutDiagnostic,
     IdleTimeoutError,
+    TransportClosedDiagnostic,
+    TransportClosedError,
+)
+from benchflow.sandbox.lockdown import (
+    build_priv_drop_cmd,
+    enforce_agent_egress_firewall,
 )
-from benchflow.sandbox.lockdown import build_priv_drop_cmd
-from benchflow.sandbox.process import DaytonaProcess, DaytonaPtyProcess, DockerProcess
 from benchflow.trajectories._capture import _capture_session_trajectory
 
 # Re-exported for backwards compatibility — tests and downstream code
@@ -51,6 +56,7 @@ __all__ = [
     "IdleTimeoutError",
     "connect_acp",
     "execute_prompts",
+    "selected_acp_transport",
 ]
 
 logger = logging.getLogger(__name__)
@@ -59,6 +65,84 @@ logger = logging.getLogger(__name__)
 _ACP_CONNECT_MAX_RETRIES = 3
 _ACP_CONNECT_BASE_DELAY = 2.0
 _PROMPT_CANCEL_DRAIN_TIMEOUT_SEC = 0.25
+_ACP_HANDSHAKE_TIMEOUT_SEC = 60
+_OPENHANDS_DISABLE_SUBAGENTS_ENV = "BENCHFLOW_OPENHANDS_DISABLE_SUBAGENTS"
+
+
+async def _prepare_openhands_direct_execution(
+    env,
+    *,
+    agent: str,
+    agent_env: dict[str, str],
+) -> dict[str, str]:
+    """Patch OpenHands as root before a sandbox-user privilege drop.
+
+    OpenHands is installed under ``/root/.local/share/uv/tools`` and exposed to
+    the sandbox user through a read-only symlink.  Running the opt-in patch from
+    the launch command only worked for tasks whose workspace was ``/root``,
+    because sandbox-user setup happened to chown that whole directory.  Tasks
+    rooted elsewhere (for example ``/app``) exited before ACP initialization.
+
+    Apply the same narrow patch through the environment's root execution plane,
+    then disable the duplicate launch-time patch for this process.  The caller's
+    environment mapping is copied so recorded run provenance still reflects the
+    requested opt-in value.
+    """
+    if agent != "openhands" or agent_env.get(_OPENHANDS_DISABLE_SUBAGENTS_ENV) != "1":
+        return agent_env
+
+    patch_cmd = (
+        'export PATH="$HOME/.local/bin:$PATH"; '
+        'OH_BIN="$(command -v openhands)"; '
+        '[ -n "$OH_BIN" ] || { echo "Cannot locate OpenHands executable" >&2; exit 127; }; '
+        'OH_PY="$(dirname "$(readlink -f "$OH_BIN")")/python"; '
+        '[ -x "$OH_PY" ] || { '
+        'echo "Cannot locate OpenHands tool interpreter" >&2; exit 127; }; '
+        '"$OH_PY" -c \'from pathlib import Path; '
+        "import openhands_cli.utils as u; "
+        "p=Path(u.__file__); s=p.read_text(); "
+        'old="        Tool(name=task_tool_name),\\n"; '
+        'new="        # BenchFlow: delegation disabled for this run.\\n"; '
+        "assert old in s or new in s; "
+        "p.write_text(s.replace(old,new,1)) if old in s else None; "
+        "assert new in p.read_text()'"
+    )
+    result = await env.exec(patch_cmd, user="root", timeout_sec=30)
+    return_code = getattr(
+        result,
+        "return_code",
+        getattr(result, "exit_code", 1),
+    )
+    if return_code != 0:
+        stdout = (getattr(result, "stdout", "") or "").strip()
+        stderr = (getattr(result, "stderr", "") or "").strip()
+        detail = stderr or stdout or "no output"
+        raise RuntimeError(
+            "Failed to prepare OpenHands direct-execution mode as root "
+            f"(exit code {return_code}): {detail[:2000]}"
+        )
+
+    launch_env = dict(agent_env)
+    launch_env[_OPENHANDS_DISABLE_SUBAGENTS_ENV] = "0"
+    logger.info("Prepared OpenHands direct-execution mode before privilege drop")
+    return launch_env
+
+
+async def _wait_for_acp_handshake(awaitable, *, phase: str):
+    try:
+        return await asyncio.wait_for(awaitable, timeout=_ACP_HANDSHAKE_TIMEOUT_SEC)
+    except TimeoutError as e:
+        msg = (
+            f"ACP {phase} timed out after {_ACP_HANDSHAKE_TIMEOUT_SEC}s "
+            "before the first prompt"
+        )
+        raise TransportClosedError(
+            msg,
+            TransportClosedDiagnostic(
+                raw_message=msg,
+                transport_diagnosis=f"acp_{phase}_timeout",
+            ),
+        ) from e
 
 
 # models.dev provider inference — used when acp_model_format="provider/model"
@@ -453,6 +537,12 @@ async def connect_acp(
 
     Retries with exponential backoff on ConnectionError (Daytona SSH storms).
     """
+    agent_env = await _prepare_openhands_direct_execution(
+        env,
+        agent=agent,
+        agent_env=agent_env,
+    )
+
     # Resolve agent binary path for non-docker environments
     if environment != "docker":
         which_result = await env.exec(
@@ -481,24 +571,13 @@ async def connect_acp(
             await asyncio.sleep(delay)
 
         try:
-            if environment == "docker":
-                live_proc = DockerProcess.from_sandbox_env(env)
-            elif environment == "daytona":
-                if agent == "gemini":
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
-                    logger.info("Using SSH transport for Gemini on Daytona")
-                else:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for Daytona sandbox")
-            else:
-                is_dind = hasattr(env, "_strategy") and hasattr(
-                    env._strategy, "_compose_cmd"
-                )
-                if is_dind:
-                    live_proc = await DaytonaPtyProcess.from_sandbox_env(env)
-                    logger.info("Using PTY transport for DinD compose task")
-                else:
-                    live_proc = await DaytonaProcess.from_sandbox_env(env)
+            # The sandbox owns its transport: it knows how to reach into
+            # itself, so there is no provider-name branch here. Backends with
+            # no live-process transport raise an actionable NotImplementedError
+            # from BaseSandbox.live_process rather than silently receiving
+            # another backend's transport (Modal previously fell through to
+            # DaytonaProcess and failed deep inside SSH setup).
+            live_proc = await env.live_process(agent=agent)
 
             agent_log = rollout_dir / "agent" / f"{agent.replace('-', '_')}.txt"
             transport = ContainerTransport(
@@ -511,15 +590,18 @@ async def connect_acp(
             acp_client = ACPClient(transport)
             await acp_client.connect()
 
-            init_result = await asyncio.wait_for(acp_client.initialize(), timeout=60)
+            init_result = await _wait_for_acp_handshake(
+                acp_client.initialize(),
+                phase="initialize",
+            )
             agent_name = (
                 init_result.agent_info.name if init_result.agent_info else agent
             )
             logger.info(f"ACP agent: {agent_name}")
 
-            session = await asyncio.wait_for(
+            session = await _wait_for_acp_handshake(
                 acp_client.session_new(cwd=agent_cwd, mcp_servers=mcp_servers),
-                timeout=60,
+                phase="session_new",
             )
             logger.info(f"Session: {session.session_id}")
             break
@@ -552,6 +634,7 @@ async def connect_acp(
             agent_env=agent_env,
             reasoning_effort=reasoning_effort,
         )
+        await enforce_agent_egress_firewall(env, sandbox_user, agent_env)
     except Exception:
         with contextlib.suppress(Exception):
             await acp_client.close()

benchflow-0.6.7.dev1798/src/benchflow/acp/selection.py

@@ -0,0 +1,30 @@
+"""Pure ACP transport selection helpers shared by runtime and artifacts."""
+
+import logging
+import os
+
+logger = logging.getLogger(__name__)
+
+_DAYTONA_ACP_TRANSPORT_ENV = "BENCHFLOW_DAYTONA_ACP_TRANSPORT"
+_DAYTONA_ACP_TRANSPORTS = {"pty", "ssh"}
+
+
+def daytona_acp_transport() -> str:
+    value = os.environ.get(_DAYTONA_ACP_TRANSPORT_ENV, "pty").strip().lower()
+    if value in _DAYTONA_ACP_TRANSPORTS:
+        return value
+    logger.warning(
+        "Invalid %s=%r; using PTY transport",
+        _DAYTONA_ACP_TRANSPORT_ENV,
+        value,
+    )
+    return "pty"
+
+
+def selected_acp_transport(*, agent: str, environment: str) -> str:
+    """Return the concrete agent transport selected for artifact provenance."""
+    if environment == "docker":
+        return "docker-stdio"
+    if environment == "daytona":
+        return "ssh" if agent == "gemini" else daytona_acp_transport()
+    return "provider-default"