benchflow 0.7.8.dev2347__tar.gz → 0.7.9.dev2399__tar.gz

{benchflow-0.7.8.dev2347 → benchflow-0.7.9.dev2399}/CHANGELOG.md

@@ -2,6 +2,113 @@
 
 ## [Unreleased]
 
+### Fixed
+
+- **Codex knows `gpt-6.1-sol`: the `codex-acp` pin moves from 1.13.1 to
+  2.0.1.** Codex takes a model's tool surface from the model catalog bundled
+  with its `@openai/codex` dependency. codex 0.156.1 (behind 1.13.1) has no
+  `gpt-6.1-sol`, so a `gpt-6.1-sol` rollout logged "Model metadata for
+  `gpt-6.1-sol` not found. Defaulting to fallback metadata" and was offered
+  only the fallback function tools: no code mode, no `apply_patch`, no
+  multi-agent tools (the #1145 symptom again), and `session/set_model` to a
+  `gpt-6.1-sol[...]` id failed. codex-acp 2.0.1 depends on `@openai/codex`
+  ^0.159.1 (npm resolves 0.159.2 today; both catalogs list `gpt-6.1-sol` with
+  efforts low to max and ultra, code mode only, multi-agent v2, a 272k
+  context). On a live docker rollout through the LiteLLM proxy (Azure),
+  `gpt-6.1-sol` at effort `max` now gets code mode (`exec`, `wait`),
+  `clock.sleep` and the collaboration tools (`spawn_agent`, `wait_agent`,
+  `send_message`, `followup_task`, `interrupt_agent`, `list_agents`), and
+  `session/set_model` accepts `gpt-6.1-sol[max]` for direct providers.
+- **Codex command output stays in the trajectory under codex-acp 2.x.**
+  codex-acp 2.0 (its tool call contract) stopped putting a finished
+  command's output in `rawOutput.formatted_output` for a client that declares
+  no terminal capability, which is BenchFlow: a shell command now ends with no
+  `rawOutput` and a read / search / list command with only `{"exit_code": n}`,
+  while the output arrives as `_meta.terminal_output_delta` chunks and the
+  exit code in `_meta.terminal_exit`. `ToolCallRecord.raw_output` rebuilds the
+  1.x `{"formatted_output", "exit_code"}` shape from those chunks (an
+  agent-sent `formatted_output` still wins), so captured trajectories keep
+  every command's output. The trajectory viewer now shows it for shell
+  commands too: it rendered Codex's `terminal` content block (a terminal id)
+  instead of the command's raw output, on codex-acp 1.x as well.
+- **Known regression: the `gpt-5.4` family falls back under codex-acp 2.0.1.**
+  codex 0.158 dropped `gpt-5.4` from its catalog, and no codex release lists
+  both `gpt-5.4` and `gpt-6.1-sol`. On 2.0.1, `gpt-5.4`, `gpt-5.4-mini` and
+  `gpt-5.4-nano` log "Model metadata ... not found" and lose `apply_patch` and
+  `tool_search`. This affects the codex-acp integration model
+  (`.github/integration/scope_defaults.yml`, `gpt-5.4-nano`) and the
+  `openai/gpt-5.4-mini` examples; a model the catalog knows (for example
+  `gpt-5.5` or `gpt-5.6-luna`) keeps the full surface.
+- **Codex gets its real model id, and a codex that knows it (#1145).** Under
+  a BenchFlow provider (LiteLLM proxy) the `codex-acp` thread was started with
+  the proxy alias as the model (`benchflow-azure-foundry-openai-gpt-5.6-luna`).
+  Codex resolves model metadata by slug, warned "Model metadata for `...` not
+  found. Defaulting to fallback metadata", and offered a reduced tool surface
+  on every rollout: eight function tools, no native `apply_patch`, no code
+  mode, no multi-agent tools, where native `codex exec` on the same provider
+  offers all of them. `CODEX_CONFIG.model` now names the bare slug (which the
+  proxy already serves next to the alias), the launch-config writer accepts
+  it when applying the reasoning effort, and the `codex-acp` pin moves to
+  1.13.1 (codex 0.156.1) because 0.148 has no metadata for `gpt-6-astra`
+  even under the bare slug. Verified on hello-world rollouts: bare slug +
+  0.156.1 gives `gpt-6-astra` code mode + `apply_patch` + `spawn_agent` /
+  `wait_agent` / `send_message` / `followup_task`, and `gpt-5.6-luna` code
+  mode + `apply_patch`, matching the native CLI's tool lists for both.
+
+### Added
+
+- **Native Google Antigravity CLI agent (`antigravity`, alias `agy`).** The
+  Antigravity CLI replaced the hosted Gemini CLI in mid-2026 but ships no ACP
+  mode, so BenchFlow bundles an ACP shim over its headless `stream-json`
+  protocol. The registry entry pins the native `agy` release (SHA-512
+  verified, no Node.js), runs in Gemini API-key mode, streams tool calls with
+  raw input/output and per-turn token usage, honors `--model` and
+  `--reasoning-effort` over ACP (agy requires an effort; the default is
+  `high`), discovers skills from `~/.gemini/config/skills` and
+  `<workspace>/.agents/skills` (a `SKILL.md` read is recorded as the skill
+  invocation), delivers task MCP servers through
+  `~/.gemini/config/mcp_config.json`, routes API-key runs through the LiteLLM
+  Gemini pass-through like the `gemini` agent, and enforces the no-web and
+  denylist policies with `PreToolUse` deny hooks.
+
+## 0.7.8 — 2026-09-14
+
+### Added
+
+- **Automatic rubric review and scoring.** Tasks with a weighted `rubric.json`
+  run a separate reviewer after verification. Passing requires all required
+  tests and rubric blockers to pass; the final reward is the weighted quality
+  score. Solver evidence and reviewer runs are retained, and `bench eval score`
+  can retry an interrupted review without rerunning the solver. (#1126)
+- **Rubric results in the trajectory viewer.** The Rubric tab displays criterion
+  judgments, explanations, blocker outcomes, and weighted scores. (#1102)
+- **Viewer themes and syntax highlighting.** Trajectory pages support a dark
+  theme and highlighted tool content. (#1098)
+
+### Fixed
+
+- **Automatic rubric review works on Daytona when the task workspace is
+  `/root`.** Daytona's daemon keeps its session tree in `/root/.daytona` (the
+  entrypoint's FIFOs plus every session command's script, log and exit code),
+  and evidence capture aborted on the first FIFO, so the rollout ended in a
+  scoring error with `rewards: null` although its verifier had run. Capture now
+  leaves that tree out as `sandbox_runtime` and records any other socket, FIFO
+  or device node as a `special_file` exclusion instead of aborting. Capture
+  limits, escaping symlinks and concurrent changes still fail closed. (#1128)
+- **Automatic review supports shell-only task images.** Required Python
+  capture tools are provisioned before the solver starts, so a successful
+  shell task does not lose its review to a missing interpreter. (#1127)
+- **Denylist enforcement closes CONNECT and HTTP Host bypasses.** TLS identity
+  and HTTP authority are checked against the allowed destination on Docker and
+  Daytona. (#1122)
+- **IPv6 egress rules are installed when procfs reports zero-size files.**
+  Sandbox firewall setup reads the kernel data instead of relying on the
+  reported file size. (#1124)
+- **Buffered ACP prompts respect their deadlines.** Timeout handling no longer
+  repeats captured provider history. (#1125)
+- **Tool captures retain their content and raw input/output.** ACP trajectories
+  preserve observations needed for later review. (#1100)
+
 ## 0.7.7 — 2026-09-09
 
 ### Added

{benchflow-0.7.8.dev2347 → benchflow-0.7.9.dev2399}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.8.dev2347
+Version: 0.7.9.dev2399
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -23,7 +23,7 @@ Requires-Dist: anyio>=4.0
 Requires-Dist: cryptography>=44
 Requires-Dist: google-cloud-aiplatform<2.0,>=1.133.0
 Requires-Dist: httpx>=0.27.0
-Requires-Dist: litellm[proxy]==1.91.0
+Requires-Dist: litellm[proxy]==1.91.5
 Requires-Dist: packaging>=24
 Requires-Dist: pydantic>=2.7
 Requires-Dist: pyyaml>=6.0
@@ -67,7 +67,7 @@ Provides-Extra: sandbox-modal
 Requires-Dist: modal>=0.73; extra == 'sandbox-modal'
 Requires-Dist: tenacity>=8.0; extra == 'sandbox-modal'
 Provides-Extra: train
-Requires-Dist: transformers==5.6.2; extra == 'train'
+Requires-Dist: transformers==5.10.4; extra == 'train'
 Provides-Extra: trl
 Requires-Dist: datasets>=2.19.0; extra == 'trl'
 Requires-Dist: trl<2,>=1.8.0; extra == 'trl'

{benchflow-0.7.8.dev2347 → benchflow-0.7.9.dev2399}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.8.dev2347"
+version = "0.7.9.dev2399"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -16,7 +16,7 @@ dependencies = [
     "cryptography>=44",
     # Dataset registry bench_version checks (PEP 440 specifier sets).
     "packaging>=24",
-    "litellm[proxy]==1.91.0",
+    "litellm[proxy]==1.91.5",
     "google-cloud-aiplatform>=1.133.0,<2.0",
     "tomli-w>=1.0",
     "typer>=0.9",
@@ -74,7 +74,7 @@ dev = [
 train = [
     # Used by the Prime-RL SFT wrapper for local token-window staging before
     # handing JSONL rows to Prime-RL.
-    "transformers==5.6.2",
+    "transformers==5.10.4",
 ]
 trl = [
     # Native tool-calling prompt/completion rows, assistant-only masks, and

{benchflow-0.7.8.dev2347 → benchflow-0.7.9.dev2399}/src/benchflow/acp/runtime.py

@@ -43,6 +43,7 @@ from benchflow.agents.registry import (
     ACPX_KEY_PREFIX,
     AGENTS,
     OPENCODE_PROXY_PROVIDER_ID,
+    routes_gemini_natively,
 )
 from benchflow.diagnostics import (
     AgentPromptTimeoutDiagnostic,
@@ -276,14 +277,17 @@ def _format_acp_model(model: str, agent: str) -> str:
     models.dev provider prefix when the agent requires it.
     """
     bare = strip_provider_prefix(model)
-    # Gemini CLI accepts bare Google model IDs. ``google/`` is a models.dev
-    # provider prefix rather than a registered BenchFlow provider, so the
-    # generic normalizer intentionally leaves it alone.
-    if agent.removeprefix(ACPX_KEY_PREFIX) == "gemini" and model.startswith(
+    agent_cfg = AGENTS.get(agent)
+    # Gemini-native agents (Gemini CLI, Antigravity CLI) take bare Google model
+    # IDs. ``google/`` is a models.dev provider prefix rather than a registered
+    # BenchFlow provider, so the generic normalizer intentionally leaves it
+    # alone. An acpx runtime key is only registered once resolve_agent_key has
+    # run, so fall back to the wrapped agent's own entry.
+    native_cfg = agent_cfg or AGENTS.get(agent.removeprefix(ACPX_KEY_PREFIX))
+    if routes_gemini_natively(native_cfg) and model.startswith(
         ("google/gemini-", "google/gemma-")
     ):
         bare = model.removeprefix("google/")
-    agent_cfg = AGENTS.get(agent)
     if agent_cfg and agent_cfg.acp_model_format == "registered-provider/model":
         # Proxy mode: BenchFlow's LiteLLM proxy serves the model under the alias
         # "benchflow-…", which the pi-acp launcher registers under the "litellm"

{benchflow-0.7.8.dev2347 → benchflow-0.7.9.dev2399}/src/benchflow/acp/session.py

@@ -163,9 +163,12 @@ class ToolCallRecord:
         self.status = ToolCallStatus.PENDING
         self.content: list[dict] = []
         # ACP ``rawInput`` / ``rawOutput``: the agent's own view of the call.
-        # codex-acp puts the command and its output here and nowhere else.
+        # codex-acp puts the command here; its output lands in ``raw_output``
+        # (see the property for codex-acp 2.x, which streams it in ``_meta``).
         self.raw_input: object | None = None
-        self.raw_output: object | None = None
+        self._raw_output: object | None = None
+        self._terminal_output: list[str] = []
+        self._terminal_exit: dict | None = None
         self.started_at = datetime.now()
         self.finished_at: datetime | None = None
 
@@ -182,12 +185,51 @@ class ToolCallRecord:
         ):
             self.finished_at = datetime.now()
 
+    @property
+    def raw_output(self) -> object | None:
+        """The agent's ``rawOutput``, completed from terminal chunks if it lacks the output.
+
+        codex-acp 1.x ended a command with ``rawOutput`` ``{"formatted_output",
+        "exit_code"}``. codex-acp 2.x (its tool call contract, upstream #530)
+        sends a client that declares no terminal capability the output only as
+        ``_meta.terminal_output_delta`` chunks and the exit code only in
+        ``_meta.terminal_exit``: a shell command ends with no ``rawOutput``, a
+        read / search / list command with ``{"exit_code": n}``. Rebuild the 1.x
+        shape from those chunks so the trajectory and viewer keep the output;
+        an agent-sent ``formatted_output`` always wins.
+        """
+        raw = self._raw_output
+        if not self._terminal_output and self._terminal_exit is None:
+            return raw
+        if raw is None:
+            raw = {}
+        if not isinstance(raw, dict) or "formatted_output" in raw:
+            return raw
+        folded = {"formatted_output": "".join(self._terminal_output), **raw}
+        if self._terminal_exit is not None:
+            folded["exit_code"] = self._terminal_exit.get("exit_code")
+        folded.setdefault("exit_code", None)
+        return folded
+
+    @raw_output.setter
+    def raw_output(self, value: object | None) -> None:
+        self._raw_output = value
+
     def absorb_raw_io(self, update: dict) -> None:
-        """Keep the latest ``rawInput`` / ``rawOutput`` an update carries."""
+        """Keep the latest ``rawInput`` / ``rawOutput`` and the terminal chunks an update carries."""
         if update.get("rawInput") is not None:
             self.raw_input = update["rawInput"]
         if update.get("rawOutput") is not None:
-            self.raw_output = update["rawOutput"]
+            self._raw_output = update["rawOutput"]
+        meta = update.get("_meta")
+        if not isinstance(meta, dict):
+            return
+        chunk = meta.get("terminal_output_delta")
+        if isinstance(chunk, dict) and isinstance(chunk.get("data"), str):
+            self._terminal_output.append(chunk["data"])
+        terminal_exit = meta.get("terminal_exit")
+        if isinstance(terminal_exit, dict):
+            self._terminal_exit = terminal_exit
 
 
 class ACPSession: