benchflow 0.3.3__tar.gz → 0.3.4__tar.gz

{benchflow-0.3.3 → benchflow-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: benchflow
-Version: 0.3.3
+Version: 0.3.4
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -52,7 +52,7 @@ BenchFlow runs AI agents against benchmark tasks in sandboxed environments. Sing
 
 - **Any ACP agent** — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, OpenClaw, Pi, or your own
 - **Single + multi + progressive** — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python `BaseUser` callback
-- **Sandbox backends** — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
+- **Sandboxes** — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
 - **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
 
 ## Install
@@ -70,7 +70,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | If you want to… | Read |
 |------------------|------|
 | Run an eval on an existing task | [Getting started](./docs/getting-started.md) |
-| Understand Trial / Scene / Role / Verifier | [Concepts](./docs/concepts.md) |
+| Understand Rollout / Scene / Role / Verifier | [Concepts](./docs/concepts.md) |
 | Author a new task | [Task authoring](./docs/task-authoring.md) |
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
@@ -99,12 +99,12 @@ Run any benchmark via the CLI:
 
 ```bash
 # From a YAML config
-bench eval create -f benchmarks/skillsbench-claude-glm51.yaml
+bench eval create --config benchmarks/skillsbench-claude-glm51.yaml
 
 # Inline — mirrors the YAML source fields
 bench eval create \
     --source-repo benchflow-ai/skillsbench --source-path tasks \
-    -a gemini -m gemini-3.1-flash-lite-preview -e daytona -c 64
+    --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64
 ```
 
 Repos are cloned and cached locally under `.cache/datasets/` on first use.
@@ -116,7 +116,7 @@ SkillsBench when you need its lockfile to point at the newest BenchFlow commit.
 
 ## Featured
 
-- **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round trial: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). Also benchflow's [Harbor #1316](https://github.com/harbor-ai/harbor/issues/1316) parity answer for the no-second-LLM case. See [Progressive disclosure](./docs/progressive-disclosure.md).
+- **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). Also benchflow's [Harbor #1316](https://github.com/harbor-ai/harbor/issues/1316) parity answer for the no-second-LLM case. See [Progressive disclosure](./docs/progressive-disclosure.md).
 
 ## Research artifacts
 

{benchflow-0.3.3 → benchflow-0.3.4}/README.md

@@ -15,7 +15,7 @@ BenchFlow runs AI agents against benchmark tasks in sandboxed environments. Sing
 
 - **Any ACP agent** — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, OpenClaw, Pi, or your own
 - **Single + multi + progressive** — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python `BaseUser` callback
-- **Sandbox backends** — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
+- **Sandboxes** — Docker locally, Daytona for parallel cloud runs, Modal for serverless/GPU-backed task environments
 - **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
 
 ## Install
@@ -33,7 +33,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | If you want to… | Read |
 |------------------|------|
 | Run an eval on an existing task | [Getting started](./docs/getting-started.md) |
-| Understand Trial / Scene / Role / Verifier | [Concepts](./docs/concepts.md) |
+| Understand Rollout / Scene / Role / Verifier | [Concepts](./docs/concepts.md) |
 | Author a new task | [Task authoring](./docs/task-authoring.md) |
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
@@ -62,12 +62,12 @@ Run any benchmark via the CLI:
 
 ```bash
 # From a YAML config
-bench eval create -f benchmarks/skillsbench-claude-glm51.yaml
+bench eval create --config benchmarks/skillsbench-claude-glm51.yaml
 
 # Inline — mirrors the YAML source fields
 bench eval create \
     --source-repo benchflow-ai/skillsbench --source-path tasks \
-    -a gemini -m gemini-3.1-flash-lite-preview -e daytona -c 64
+    --agent gemini --model gemini-3.1-flash-lite-preview --sandbox daytona --concurrency 64
 ```
 
 Repos are cloned and cached locally under `.cache/datasets/` on first use.
@@ -79,7 +79,7 @@ SkillsBench when you need its lockfile to point at the newest BenchFlow commit.
 
 ## Featured
 
-- **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round trial: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). Also benchflow's [Harbor #1316](https://github.com/harbor-ai/harbor/issues/1316) parity answer for the no-second-LLM case. See [Progressive disclosure](./docs/progressive-disclosure.md).
+- **Progressive disclosure on SWE-bench Pro** — the `BaseUser` abstraction drives a multi-round rollout: terse round-0 prompt → failing-test hints → full spec. 5/5 oracle on Daytona, runnable demo at [`docs/examples/swebench_pro_progressive_disclosure.ipynb`](./docs/examples/swebench_pro_progressive_disclosure.ipynb). Also benchflow's [Harbor #1316](https://github.com/harbor-ai/harbor/issues/1316) parity answer for the no-second-LLM case. See [Progressive disclosure](./docs/progressive-disclosure.md).
 
 ## Research artifacts
 

{benchflow-0.3.3 → benchflow-0.3.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.3.3"
+version = "0.3.4"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.3.3 → benchflow-0.3.4}/src/benchflow/__init__.py

@@ -3,8 +3,8 @@
 Re-exports environment APIs and adds:
 - ACP client for multi-turn agent communication
 - Trajectory capture (HTTP proxy, OTel collector, ACP native)
-- SDK for programmatic usage
-- Job orchestration with retries and concurrency
+- Rollout lifecycle for single-task execution
+- Evaluation orchestration with retries and concurrency
 - Metrics collection and aggregation
 """
 
@@ -25,10 +25,18 @@ from harbor import (
 
 # benchflow's additions
 from benchflow._env_setup import stage_dockerfile_deps
-from benchflow._scene import MailboxTransport, Message, MessageTransport, Role, Scene
+from benchflow._scene import MailboxTransport, Message, MessageTransport, SceneRole
+from benchflow._scene import Scene as SceneRuntime
 from benchflow._snapshot import list_snapshots, restore, snapshot
+from benchflow._types import Role, Scene, Turn
 from benchflow.acp.client import ACPClient
 from benchflow.acp.session import ACPSession
+from benchflow.adapters import (
+    InspectAdapter,
+    ORSAdapter,
+    to_inspect_task,
+    to_ors_reward,
+)
 from benchflow.agents.registry import (
     AGENTS,
     get_agent,
@@ -43,34 +51,89 @@ from benchflow.environments import (
     detect_services_from_dockerfile,
     register_service,
 )
-from benchflow.job import Job, JobConfig, JobResult, RetryConfig
+from benchflow.evaluation import (
+    Evaluation,
+    EvaluationConfig,
+    EvaluationResult,
+    RetryConfig,
+)
 from benchflow.metrics import BenchmarkMetrics, collect_metrics
-from benchflow.models import AgentInstallError, AgentTimeoutError, RunResult
+from benchflow.models import AgentInstallError, AgentTimeoutError, RolloutResult
+
+# Rewards protocol (v0.4 — composable Rubric + RewardFunc)
+from benchflow.rewards import (
+    CodeExecRewardFunc,
+    Criterion,
+    JudgeConfig,
+    LLMJudgeRewardFunc,
+    RewardEvent,
+    RewardFunc,
+    Rubric,
+    RubricConfig,
+    ScoringConfig,
+    StringMatchRewardFunc,
+    TestRewardFunc,
+    VerifyResult,
+    load_rubric_toml,
+)
+from benchflow.rollout import Rollout, RolloutConfig
 from benchflow.runtime import (
     Agent,
     Environment,
     Runtime,
     RuntimeConfig,
     RuntimeResult,
-    run,  # bf.run(agent, env) — the primary 0.3 API
-)
+    run,
+)  # bf.run() — supports Agent, RolloutConfig, and str calling conventions
+
+# Sandbox protocol (v0.4 — parallel types, Harbor not yet removed)
+from benchflow.sandbox import ExecResult as SandboxExecResult
+from benchflow.sandbox import ImageBuilder, ImageConfig, ImageRef, Sandbox
 from benchflow.sdk import SDK
 from benchflow.skills import SkillInfo, discover_skills, install_skill, parse_skill
 from benchflow.trajectories.otel import OTelCollector
 from benchflow.trajectories.proxy import TrajectoryProxy
 from benchflow.trajectories.types import Trajectory
-from benchflow.trial import Role as TrialRole
-from benchflow.trial import Scene as TrialScene
-from benchflow.trial import Trial, TrialConfig, Turn
 from benchflow.trial_yaml import trial_config_from_yaml
 from benchflow.user import BaseUser, FunctionUser, PassthroughUser, RoundResult
 
+# Backward-compat aliases
+Trial = Rollout
+TrialConfig = RolloutConfig
+TrialRole = Role
+TrialScene = Scene
+RunResult = RolloutResult
+Job = Evaluation
+JobConfig = EvaluationConfig
+JobResult = EvaluationResult
+
 # Public API surface. Anything not in this list is implementation detail and
 # may change without notice. Names are grouped by source module to match the
 # imports above and to make it obvious to a future agent which module owns
 # what.
 __all__ = [
     "__version__",
+    # Rewards protocol (v0.4)
+    "Rubric",
+    "RewardFunc",
+    "RewardEvent",
+    "VerifyResult",
+    "TestRewardFunc",
+    "LLMJudgeRewardFunc",
+    "StringMatchRewardFunc",
+    "CodeExecRewardFunc",
+    # Rubric config (ENG-55)
+    "Criterion",
+    "JudgeConfig",
+    "RubricConfig",
+    "ScoringConfig",
+    "load_rubric_toml",
+    # Sandbox protocol (v0.4)
+    "Sandbox",
+    "SandboxExecResult",
+    "ImageBuilder",
+    "ImageConfig",
+    "ImageRef",
     # Harbor re-exports
     "BaseAgent",
     "BaseEnvironment",
@@ -89,28 +152,38 @@ __all__ = [
     "is_vertex_model",
     "list_agents",
     "register_agent",
-    # Job orchestration
+    # Evaluation orchestration (new names)
+    "Evaluation",
+    "EvaluationConfig",
+    "EvaluationResult",
+    "RetryConfig",
+    # Backward-compat aliases for Job
     "Job",
     "JobConfig",
     "JobResult",
-    "RetryConfig",
     # Metrics
     "BenchmarkMetrics",
     "collect_metrics",
     # Models / errors
     "AgentInstallError",
     "AgentTimeoutError",
+    "RolloutResult",
     "RunResult",
-    # Runtime (0.3 primary API)
+    # Runtime (0.3 compat)
     "Agent",
     "Environment",
     "Runtime",
     "RuntimeConfig",
     "RuntimeResult",
+    # Single entry point
     "run",
-    # Multi-agent scene
-    "Scene",
+    # Canonical declarative types (_types.py — ENG-47)
     "Role",
+    "Scene",
+    "Turn",
+    # Multi-agent scene runtime
+    "SceneRole",
+    "SceneRuntime",
     "Message",
     "MessageTransport",
     "MailboxTransport",
@@ -118,12 +191,14 @@ __all__ = [
     "snapshot",
     "restore",
     "list_snapshots",
-    # Trial (decomposed lifecycle)
+    # Rollout (single execution path — ENG-46)
+    "Rollout",
+    "RolloutConfig",
+    # Backward-compat aliases for Trial
     "Trial",
     "TrialConfig",
     "TrialRole",
     "TrialScene",
-    "Turn",
     "trial_config_from_yaml",
     # User abstraction (progressive disclosure)
     "BaseUser",
@@ -147,11 +222,25 @@ __all__ = [
     "OTelCollector",
     "TrajectoryProxy",
     "Trajectory",
+    # External adapters (ENG-51)
+    "InspectAdapter",
+    "ORSAdapter",
+    "to_inspect_task",
+    "to_ors_reward",
 ]
 
 
 def __getattr__(name: str):
     """Fall through to harbor for names not explicitly re-exported."""
+    # Let Python's normal submodule resolution handle subpackages first.
+    import importlib
+
+    try:
+        return importlib.import_module(f"benchflow.{name}")
+    except ModuleNotFoundError as e:
+        if e.name != f"benchflow.{name}":
+            raise
+
     import harbor
 
     if hasattr(harbor, name):

{benchflow-0.3.3 → benchflow-0.3.4}/src/benchflow/_agent_env.py

@@ -163,7 +163,7 @@ def inject_vertex_credentials(agent_env: dict[str, str], model: str) -> None:
     if "GOOGLE_CLOUD_PROJECT" not in agent_env:
         raise ValueError(
             f"GOOGLE_CLOUD_PROJECT required for Vertex AI model {model!r}. "
-            f"Export it or pass via --ae GOOGLE_CLOUD_PROJECT=<project>"
+            f"Export it or pass via --agent-env GOOGLE_CLOUD_PROJECT=<project>"
         )
 
 

{benchflow-0.3.3 → benchflow-0.3.4}/src/benchflow/_agent_setup.py

@@ -243,6 +243,7 @@ async def deploy_skills(
                 logger.warning(f"Skills dir not found: {skills_path}")
         else:
             logger.info("Skills already injected via Dockerfile")
+            effective_skills = "/skills"
 
     # Distribute to agent-specific discovery paths
     if agent_cfg is not None:

benchflow-0.3.4/src/benchflow/_run.py

@@ -0,0 +1,34 @@
+"""Single entry point: ``bf.run(RolloutConfig) → RolloutResult``."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+from benchflow.models import RolloutResult
+
+if TYPE_CHECKING:
+    from benchflow.rollout import RolloutConfig
+
+
+async def run(config: RolloutConfig | None = None, **kwargs: Any) -> RolloutResult:
+    """Execute a single rollout and return the result.
+
+    This is the canonical entry point for running an agent on a task::
+
+        import benchflow as bf
+        result = await bf.run(RolloutConfig(task_path=..., agent=...))
+
+    If *config* is None, a :class:`RolloutConfig` is built from *kwargs*
+    via :meth:`RolloutConfig.from_legacy`.
+    """
+    from benchflow.rollout import Rollout
+    from benchflow.rollout import RolloutConfig as _RolloutConfig
+
+    if config is None:
+        config = _RolloutConfig.from_legacy(**kwargs)
+    if config.skill_mode == "self-gen":
+        from benchflow.self_gen import run_self_gen
+
+        return await run_self_gen(config)
+    rollout = await Rollout.create(config)
+    return await rollout.run()

{benchflow-0.3.3 → benchflow-0.3.4}/src/benchflow/_scene.py

@@ -14,12 +14,27 @@ reads the outbox after each agent exits, routes through the transport,
 and injects into the next agent's prompt.
 
 0.3 scope: exactly 2 roles, sequential execution, mailbox transport only.
+
+.. rubric:: Capability boundary (ENG-50)
+
+BenchFlow's scene scheduler handles **turn sequencing** and **message
+routing** only.  It does *not* manage:
+
+* Agent-internal loops — the agent decides when/how to iterate.
+* Tool protocols — tool-use is a per-agent capability.
+* Agent-as-tool invocation — if agent A wants to call agent B as a
+  tool, agent A is responsible for reaching B's endpoint (all roles
+  share the sandbox, so localhost networking is available).
+
+The scheduler ensures each role runs in order and that outbox messages
+are delivered.  Everything beyond that is the agent's responsibility.
 """
 
 from __future__ import annotations
 
 import json
 import logging
+import shlex
 import uuid
 from abc import ABC, abstractmethod
 from collections.abc import Callable
@@ -37,7 +52,13 @@ logger = logging.getLogger(__name__)
 
 
 @dataclass
-class Role:
+class SceneRole:
+    """Runtime role for multi-agent scenes (internal to _scene.py).
+
+    Unlike the canonical ``Role`` in ``_types.py``, this carries the
+    instruction text and tool list needed by the scene scheduler.
+    """
+
     name: str
     agent: str
     model: str
@@ -106,7 +127,7 @@ class Scene:
 
     def __init__(
         self,
-        roles: dict[str, Role],
+        roles: dict[str, SceneRole],
         transport: MessageTransport | None = None,
         max_rounds: int = 10,
     ) -> None:
@@ -158,7 +179,7 @@ class Scene:
     def is_done(self) -> bool:
         return self._done or self._round >= self.max_rounds
 
-    def build_prompt_for_role(self, role: Role, inbox: list[Message]) -> str:
+    def build_prompt_for_role(self, role: SceneRole, inbox: list[Message]) -> str:
         """Build the prompt for a role, injecting any pending messages."""
         parts = [role.instruction]
         if inbox:
@@ -190,7 +211,8 @@ class Scene:
         ]
         messages = []
         for fpath in files:
-            cat_result = await env.exec(f"cat {fpath}")
+            qpath = shlex.quote(fpath)
+            cat_result = await env.exec(f"cat {qpath}")
             try:
                 data = json.loads(cat_result.stdout or "{}")
                 recipient = data.get("to", "")
@@ -212,7 +234,7 @@ class Scene:
                     messages.append(msg)
             except json.JSONDecodeError:
                 logger.warning(f"[Scene] invalid JSON in outbox file: {fpath}")
-            await env.exec(f"rm -f {fpath}")
+            await env.exec(f"rm -f {qpath}")
         return messages
 
     # ------------------------------------------------------------------
@@ -287,3 +309,7 @@ class Scene:
         ]
         path.write_text("\n".join(lines) + "\n" if lines else "")
         logger.info(f"Scene trajectory saved: {len(self.trajectory)} messages → {path}")
+
+
+# Backward-compat alias — existing code imports ``Role`` from this module.
+Role = SceneRole

{benchflow-0.3.3 → benchflow-0.3.4}/src/benchflow/_snapshot.py

@@ -1,10 +1,10 @@
 """Sandbox snapshot/restore — filesystem-level checkpointing.
 
 Provides snapshot(name) -> ref and restore(ref) for any environment that
-supports env.exec().  Works on both Docker and Daytona backends.
+supports env.exec().  Works on both Docker and Daytona sandboxes.
 
 Implementation: tar the workspace directory into /tmp/.benchflow_snapshots/.
-In-place restore by clearing and untarring.  Backend-agnostic — no Docker
+In-place restore by clearing and untarring.  Sandbox-agnostic — no Docker
 daemon or Daytona snapshot API required.
 
 For 0.3+, Daytona's _experimental_create_snapshot can be swapped in as an
@@ -13,6 +13,8 @@ filesystem approach covers the rewind use case and is provable now.
 """
 
 import logging
+import re as _re
+import shlex
 from pathlib import PurePosixPath
 
 logger = logging.getLogger(__name__)
@@ -26,16 +28,14 @@ async def snapshot(env, name: str, workspace: str = "/app") -> str:
     Returns a reference string suitable for restore() and for recording
     in trial metadata / rewards.jsonl.
     """
-    import re
-
-    if not re.match(r"^[a-zA-Z0-9_-]+$", name):
+    if not _re.match(r"^[a-zA-Z0-9_-]+$", name):
         raise ValueError(
             f"Snapshot name must be alphanumeric/dash/underscore, got: {name!r}"
         )
     await env.exec(f"mkdir -p {_SNAP_DIR}")
     snap_path = f"{_SNAP_DIR}/{name}.tar.gz"
     result = await env.exec(
-        f"tar czf {snap_path} -C {workspace} .",
+        f"tar czf {shlex.quote(snap_path)} -C {shlex.quote(workspace)} .",
         timeout_sec=120,
     )
     if result.return_code != 0:
@@ -54,12 +54,19 @@ async def restore(env, ref: str, workspace: str = "/app") -> None:
     if len(parts) != 3 or parts[0] != "fs":
         raise ValueError(f"invalid snapshot ref: {ref}")
     snap_path = parts[2]
-    check = await env.exec(f"test -f {snap_path} && echo ok || echo missing")
+    # Validate snap_path: must be under _SNAP_DIR and a .tar.gz file
+    if not snap_path.startswith(_SNAP_DIR + "/") or not snap_path.endswith(".tar.gz"):
+        raise ValueError(f"invalid snapshot ref: path must be under {_SNAP_DIR}")
+    if ".." in snap_path.split("/"):
+        raise ValueError("invalid snapshot ref: path traversal not allowed")
+    check = await env.exec(
+        f"test -f {shlex.quote(snap_path)} && echo ok || echo missing"
+    )
     if "missing" in (check.stdout or ""):
         raise FileNotFoundError(f"snapshot not found: {snap_path}")
     result = await env.exec(
-        f"rm -rf {workspace}/* {workspace}/.[!.]* 2>/dev/null; "
-        f"tar xzf {snap_path} -C {workspace}",
+        f"rm -rf {shlex.quote(workspace)}/* {shlex.quote(workspace)}/.[!.]* 2>/dev/null; "
+        f"tar xzf {shlex.quote(snap_path)} -C {shlex.quote(workspace)}",
         timeout_sec=120,
     )
     if result.return_code != 0:

benchflow-0.3.4/src/benchflow/_types.py

@@ -0,0 +1,94 @@
+"""Canonical Scene / Role / Turn data types for benchflow trials.
+
+These are the *declarative* types — they describe what a trial *will* do.
+Runtime classes (e.g. ``_scene.Scene``) consume these but are not defined
+here.
+
+Merged from the duplicate definitions that lived in ``trial.py`` and
+``_scene.py`` prior to ENG-47.
+
+.. rubric:: Capability boundary (ENG-50)
+
+BenchFlow provides **sandbox + instruction + observation** infrastructure.
+It does **not** orchestrate agent-internal loops, tool protocols, or
+agent-as-tool invocations. Those are per-agent capabilities declared via
+:pyattr:`Role.capabilities` and implemented by the agent itself.
+
+Concretely:
+
+* BenchFlow sets up the sandbox (Docker / Daytona / Modal), injects
+  environment variables and skills, and wires up the ACP transport.
+* BenchFlow ensures sandbox networking allows inter-agent communication
+  (all roles in a scene share a sandbox, so localhost is reachable).
+* The ``capabilities`` field on :class:`Role` is a **declaration** — it
+  tells downstream tooling / dashboards what the agent supports, but
+  BenchFlow itself does not act on it.
+* Agent-as-tool is a per-agent capability.  BenchFlow does not invoke
+  agents on behalf of other agents; the calling agent uses its own
+  native tool-use to reach a companion agent's endpoint.
+* Loop management is entirely the agent's responsibility.  BenchFlow
+  scenes define *turns* (prompts), not iteration.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from pathlib import Path
+
+
+@dataclass
+class Role:
+    """One agent participant in a scene.
+
+    The ``capabilities`` field is a declarative list of strings describing
+    what the agent natively supports (e.g. ``["tool-use", "loop",
+    "agent-as-tool"]``).  BenchFlow records these in trial metadata but
+    does **not** act on them — the agent itself is responsible for
+    implementing whatever capabilities it advertises.
+    """
+
+    name: str
+    agent: str
+    model: str | None = None
+    env: dict[str, str] = field(default_factory=dict)
+    timeout_sec: int | None = None  # None = inherit from task.toml
+    idle_timeout_sec: int | None = None
+    skills_dir: str | Path | None = None
+    capabilities: list[str] | None = None  # e.g. ["tool-use", "agent-as-tool", "loop"]
+
+
+@dataclass
+class Turn:
+    """One prompt in a scene. *role* selects which Role acts."""
+
+    role: str
+    prompt: str | None = None  # None = expand from instruction.md
+
+
+@dataclass
+class Scene:
+    """One interaction region — roles take turns executing prompts."""
+
+    name: str = "default"
+    roles: list[Role] = field(default_factory=list)
+    turns: list[Turn] = field(default_factory=list)
+    skills_dir: str | Path | None = None
+    parallel_group: str | None = None  # scenes with same group execute concurrently
+
+    @classmethod
+    def single(
+        cls,
+        *,
+        agent: str,
+        model: str | None = None,
+        prompts: list[str | None] | None = None,
+        role_name: str = "agent",
+        skills_dir: str | Path | None = None,
+    ) -> Scene:
+        """Shortcut for single-agent, single-role scene."""
+        prompts = prompts or [None]
+        return cls(
+            roles=[Role(name=role_name, agent=agent, model=model)],
+            turns=[Turn(role=role_name, prompt=p) for p in prompts],
+            skills_dir=skills_dir,
+        )

benchflow-0.3.4/src/benchflow/adapters/__init__.py

@@ -0,0 +1,25 @@
+"""External framework adapters for BenchFlow types.
+
+These are pure format converters — they translate BenchFlow's ``Scene``,
+``Rubric``, ``VerifyResult``, and ``RewardEvent`` types into dicts that
+follow the conventions of external eval frameworks.  No external SDK is
+required.
+
+Supported targets:
+
+* **Inspect AI** — ``InspectAdapter`` / ``to_inspect_task``
+* **ORS (OpenReward)** — ``ORSAdapter`` / ``to_ors_reward``
+
+To add a new adapter, create a module under ``benchflow.adapters`` and
+re-export its public symbols here.
+"""
+
+from benchflow.adapters.inspect_ai import InspectAdapter, to_inspect_task
+from benchflow.adapters.ors import ORSAdapter, to_ors_reward
+
+__all__ = [
+    "InspectAdapter",
+    "ORSAdapter",
+    "to_inspect_task",
+    "to_ors_reward",
+]