benchflow 0.6.7.dev1845__tar.gz → 0.6.8__tar.gz

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/CHANGELOG.md

@@ -2,6 +2,68 @@
 
 ## [Unreleased]
 
+## 0.6.8 — 2026-08-15
+
+### Added
+- **Public trajectory contribution.** `bench traj upload PATH` validates and
+  structurally redacts trajectory JSONL, creates a content-addressed manifest,
+  and uploads through the built-in public broker. Azure Blob quarantine,
+  versioning, short-lived create-only user-delegation SAS grants, and an
+  event-driven fail-closed validator keep untrusted captures out of the
+  community namespace until hashes, sizes, strict JSONL (including duplicate-key
+  and non-finite-number rejection), and artifact/manifest secret scans pass.
+  Replaying an ingested digest is a no-op; trusted operators can opt into direct
+  Azure upload with the `azure` extra and `--direct`.
+
+### Changed
+- **BREAKING (task.md): the `environment:` frontmatter key is renamed to
+  `sandbox:`.** The native task-config surface now accepts only `sandbox:`
+  (plus `verifier.sandbox:` for the verifier's separate sandbox spec and
+  `verifier.sandbox_mode:` for shared/separate selection); `environment:`,
+  `verifier.environment:`, and `verifier.environment_mode:` no longer
+  validate and fail with an actionable message naming the rename. **The
+  one-line fix for existing task.md files is renaming the key.**
+  Legacy/Harbor `task.toml` imports are unaffected: the toml loader
+  converts `[environment]`, `[verifier.environment]`, and
+  `environment_mode` to the `sandbox` spellings (declaring both spellings
+  in one file is an error), and `bench tasks export` emits the inverse —
+  a stock-Harbor `[environment]`-spelled `task.toml`. All native emitters
+  — `model_dump_toml`, `bench tasks migrate`, task scaffolding,
+  skill-eval/trace/adapter task generation, rubric-review wrappers — now
+  write `sandbox`. Python API: the compat property
+  `TaskConfig.environment` is removed (use `TaskConfig.sandbox`),
+  `VerifierConfig.environment`/`environment_mode` became
+  `sandbox`/`sandbox_mode`, and the `VerifierEnvironmentMode` enum is now
+  `VerifierSandboxMode`. The Environment plane
+  (`--environment-manifest`, `benchflow.environment.manifest`, the
+  eval-config `environment:` docker/daytona selector) is a different
+  subsystem and is unchanged.
+
+### Fixed
+- **Coherent integration coverage for code-and-fixture changes.** The
+  credential-free fixture job now tests pull-request source together with its
+  task fixtures, while the secret-bearing smoke job retains trusted fixtures;
+  the release gate requires both results. (#969)
+- **Retryable Daytona transport failures stay retryable.** Transient SDK
+  transport errors are stamped while their vendor type is still available,
+  empty exception messages retain useful type information, and permanent
+  errors remain outside the retry policy. (#970)
+- **Live token counters no longer freeze behind callback-log reads.** Larger,
+  bounded ranged reads distinguish EOF from read failure, expose lag, and keep
+  terminal phase labels from moving backwards. (#971)
+- **Fresh OpenClaw installs and GPT-5.4 calls use compatible limits.** OpenClaw
+  is pinned to a Node-compatible runtime, and raw plus ACP-alias GPT-5.4 model
+  IDs clamp output tokens to the provider's 128,000-token limit. (#977, #986)
+- **Anthropic Vertex routes include their required runtime.** The gateway now
+  installs the Google Cloud Vertex AI SDK used by LiteLLM's Anthropic Vertex
+  path. (#978)
+- **ACP subprocess diagnostics survive normal teardown.** A bounded, redacted
+  stderr tail is retained even when stdout remains open; stderr drain failures
+  cannot mask the structured transport diagnosis, and repeated process close
+  calls are idempotent. (#980)
+
+## 0.6.7 — 2026-08-09
+
 ### Added
 - **Built-in environment registry.** The committed env-axis pins
   (`env0@prod`, `env0@outage`) moved from `benchmarks/_environments/` into
@@ -10,13 +72,71 @@
   `pip install benchflow` with no checkout and no env vars.
   `$BENCHFLOW_ENV_REGISTRY`, when set, still wins entirely; resolution
   stays content-addressed (sha256 logged), and unknown names now error
-  listing the available specs.
+  listing the available specs. (#961)
 - **Console progress heartbeat.** Single-concurrency eval runs print a
   throttled progress line (`… 6.2min, 12 tool calls (last: …)`) about every
   45 seconds while the agent works, so a long prompt is distinguishable from
   a hang. The heartbeat is auto-gated off for multi-concurrency jobs;
   `bench eval run --quiet` suppresses it, and `BENCHFLOW_PROGRESS=on`/`off`
   overrides the auto-gate. (#951)
+- **Live per-task activity in the eval dashboard.** Under a TTY the
+  running-now table gains an activity column ("38 calls · last:
+  file_editor", plus tokens once a usage snapshot exists), polled from the
+  ACP session's existing heartbeat counters. The agents-manifest autoload
+  also clones quietly — one "Cloning …" summary line instead of raw git
+  progress. (#956)
+- **Phase labels and per-task failure reasons.** The activity cell is never
+  blank while a row exists: sandbox create, agent install, and verify show
+  dim phase labels ("creating sandbox…", "installing agent…",
+  "verifying…"). The final block prints one dim "✗ task: reason" line per
+  failed task — verifier error first, else a compact reward/metric
+  breakdown, else the reward — capped at 5 lines. `--quiet` now silences
+  the dashboard as well as the heartbeat. (#957)
+- **Failure reasons mined from verifier artifacts.** When a displayed
+  failure would read as a bare "reward X", the CLI reads a small bounded
+  artifact from the rollout's own verifier dir — the CTRF report (first
+  failed test plus its assertion line) or a tail of `test-stdout.txt` — and
+  a dim "(details: …/verifier)" pointer names the artifact directory.
+  Artifacts resolve by the recorded rollout name, never by glob. (#959)
+- **Fractional rewards in console summaries.** Per-task lines carry the
+  scored reward (`✗ task (reward=0.30, tools=47)`), the Score line renders
+  ", mean reward 0.30" alongside the binarized counts, and
+  `EvaluationResult` / `summary.json` gain a `mean_reward` field (mean over
+  scored rollouts; errors excluded, not zeroed). (#960)
+- **Full failure counts on per-task console lines.** A CTRF report with
+  more than one failed test rolls up as " (+N more failure(s); P/T checks
+  passed)" after the first failure — the count suffix is never truncated
+  away — and parametrized test names keep their `[param]` ids whenever the
+  report carries them. (#962)
+- **Live token usage in the dashboard footer.** The footer sums completed
+  tasks' trusted telemetry plus every running rollout's live ACP session
+  usage, so spend is visible while the run executes (usage lands per
+  completed prompt; cost stays scoring-time from the gateway log).
+  Single-tool agents' activity cell drops the redundant "last:" suffix in
+  favor of "38 calls · 412.0k tok" once tokens are available. (#963)
+- **env0-shaped failure breakdowns.** The failure-reason tiers understand
+  metrics nested one level under `metrics` / `details`, pair
+  `<name>_found` with `<name>_total` into fractions ("deadlines 1/5",
+  lowest-signal first), probe `verifier/reward.json` on disk between the
+  CTRF and stdout tiers, and every failure block with on-disk verifier
+  artifacts gets one "(details: …)" pointer. (#964)
+- **Mid-prompt live token usage via the gateway's live capture.** The #963
+  live tokens stepped forward only when an ACP prompt completed, so a
+  single-prompt rollout showed "— tokens" for its whole agent phase. The
+  proxy runtime's existing live-capture loop (which already tails the
+  sandbox gateway's callback log every second) now also accumulates
+  provider tokens into an O(1) counter, and the dashboard reconciles the
+  two non-decreasing live signals as max(ACP, gateway) — display-only,
+  still replaced by the trusted scoring import at completion, and any
+  gateway-side failure degrades to the ACP-only behavior. (#965)
+
+- **Compact flow-style arrays in emitted task.md frontmatter.** Short
+  scalar-only lists render on one line (`tags: [parsing, nlp]`) instead of
+  multiline bullets, so hand-written flow arrays survive `bench tasks
+  migrate` / normalize round-trips. The style predicate measures each list
+  with the real YAML emitter (so it can never disagree with PyYAML's own
+  quoting) and falls back to block style for long, nested, or multiline
+  items. Parsing is unchanged — both styles were always accepted. (#967)
 
 ### Changed
 - Migrated the clawsbench `archive-amazon-shipping` task to the native
@@ -51,6 +171,27 @@
   environment now fails loud when it declares services but none are startable
   in the image, instead of passing a vacuous readiness gate and dying at the
   verifier. (#954)
+- Pointed the built-in `env0@prod` / `env0@outage` pins at the org-owned
+  `ghcr.io/benchflow-ai/env0:0.2.0` base image. The wheel-shipped pins named a
+  personal Docker Hub image while their own comments and the environment docs
+  declared the ghcr image authoritative; `base_image` is recorded as rollout
+  provenance, so every env0 result carried the wrong base. Verified on Daytona
+  (env0 `auth-least-privilege-summary`, 8/8 services ready, reward 1.0).
+- Restored the `env0@outage` perturbation (gmail and slack removed relative
+  to `env0@prod`) that the #954 upstream sync had erased by mirroring the
+  full service list into both pins, and corrected the pin header's stale
+  slack port. (#955)
+- **ACP protocol JSON glued to PTY shell noise now decodes.** On PTY
+  transports, an agent's initialize response could arrive on the same line
+  as the shell prompt (ANSI/OSC-prefixed), fail the strict whole-line JSON
+  parse, and get filed as noise — the handshake then "timed out" at any
+  window with the answer sitting in the agent log. The PTY-facing transport
+  now retries from the first `{` and once more after an ANSI scrub, but
+  only until the first successfully decoded protocol message per
+  connection; afterwards the strict contract rules, so log-echoed envelopes
+  cannot impersonate protocol traffic mid-session. The pre-prompt handshake
+  window is also env-configurable via `BENCHFLOW_ACP_HANDSHAKE_TIMEOUT`
+  (seconds; default 60). (#958)
 
 ## 0.6.6 — 2026-08-04
 

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: benchflow
-Version: 0.6.7.dev1845
+Version: 0.6.8
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -20,6 +20,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.12
 Requires-Dist: agent-client-protocol>=0.10
 Requires-Dist: anyio>=4.0
+Requires-Dist: google-cloud-aiplatform<2.0,>=1.133.0
 Requires-Dist: httpx>=0.27.0
 Requires-Dist: litellm[proxy]==1.91.0
 Requires-Dist: packaging>=24
@@ -28,6 +29,15 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.0
 Requires-Dist: tomli-w>=1.0
 Requires-Dist: typer>=0.9
+Provides-Extra: azure
+Requires-Dist: azure-identity>=1.16; extra == 'azure'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'azure'
+Provides-Extra: azure-service
+Requires-Dist: azure-data-tables>=12.5; extra == 'azure-service'
+Requires-Dist: azure-identity>=1.16; extra == 'azure-service'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'azure-service'
+Requires-Dist: azure-storage-queue>=12.9; extra == 'azure-service'
+Requires-Dist: uvicorn>=0.30; extra == 'azure-service'
 Provides-Extra: bedrock
 Requires-Dist: boto3>=1.40; extra == 'bedrock'
 Provides-Extra: deepagents
@@ -145,6 +155,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
 | Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) |
+| Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) |
 | Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) |
 | Use public vs internal preview SDK releases | [Release channels](./docs/release.md) |
 | CLI flags + commands | [CLI reference](./docs/reference/cli.md) |

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/README.md

@@ -81,6 +81,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
 | Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) |
+| Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) |
 | Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) |
 | Use public vs internal preview SDK releases | [Release channels](./docs/release.md) |
 | CLI flags + commands | [CLI reference](./docs/reference/cli.md) |

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.7.dev1845"
+version = "0.6.8"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -15,6 +15,7 @@ dependencies = [
     # Dataset registry bench_version checks (PEP 440 specifier sets).
     "packaging>=24",
     "litellm[proxy]==1.91.0",
+    "google-cloud-aiplatform>=1.133.0,<2.0",
     "tomli-w>=1.0",
     "typer>=0.9",
 ]
@@ -39,6 +40,20 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
+azure = [
+    # Direct Azure Blob trajectory uploads (bench traj upload --direct).
+    # Broker-mode uploads need neither package.
+    "azure-storage-blob>=12.19",
+    "azure-identity>=1.16",
+]
+azure-service = [
+    # Dependencies baked into the public trajectory broker/validator image.
+    "azure-storage-blob>=12.19",
+    "azure-identity>=1.16",
+    "azure-data-tables>=12.5",
+    "azure-storage-queue>=12.9",
+    "uvicorn>=0.30",
+]
 dev = [
     "pre-commit>=3.7",
     # AgentCore's CodeBuild path must reproduce Docker's ignore rules in tests

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/_utils/config_override.py

@@ -144,8 +144,9 @@ def apply_config_override(config: Any, overlay: dict[str, Any] | None) -> Any:
 
     # Merge against the FIELD-NAME dump (``by_alias=False``) so overlays use the
     # canonical field names (``agent``, ``sandbox``, …); ``populate_by_name`` lets
-    # re-validation accept them. ``by_alias=True`` would force callers to write the
-    # alias (e.g. ``environment`` for ``sandbox``) — the bug this avoids.
+    # re-validation accept them. (``sandbox`` is now the only spelling — the
+    # legacy ``environment`` alias was removed in the rename — but ``oracle``
+    # still serializes via alias, so the field-name dump stays load-bearing.)
     merged = deep_merge(config.model_dump(by_alias=False), overlay)
     patched = TaskConfig.model_validate(merged)
     logger.debug(

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/_utils/scoring.py

@@ -4,7 +4,10 @@ import math
 from collections.abc import Iterable, Mapping
 from typing import Any, Literal
 
-from benchflow.diagnostics import DIAGNOSTIC_REASON_IDLE_TIMEOUT
+from benchflow.diagnostics import (
+    DIAGNOSTIC_REASON_IDLE_TIMEOUT,
+    TRANSIENT_SANDBOX_TRANSPORT_MARKER,
+)
 
 # Error category constants
 INSTALL_FAILED = "install_failure"
@@ -174,6 +177,14 @@ def _looks_like_infra_error(error: str) -> bool:
             "connection reset",
             "connection refused",
             "broken pipe",
+            # Stamped by TransientSandboxTransportError at the sandbox-SDK
+            # boundary, where the vendor exception type is still available to
+            # judge. One marker covers every provider call — exec, upload,
+            # download, stat — instead of one entry per vendor message
+            # prefix, which would be endless and silently incomplete.
+            TRANSIENT_SANDBOX_TRANSPORT_MARKER,
+            # Predates the marker above and is kept for strings rebuilt
+            # outside that boundary (e.g. a verifier error re-raised as text).
             "failed to get session command",
             "sandbox not found",
             "workspace not found",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/_utils/task_authoring/scaffolding.py

@@ -333,7 +333,7 @@ timeout_sec = 300
 [verifier]
 timeout_sec = 120
 
-[environment]
+[sandbox]
 cpus = 1
 memory_mb = 2048
 """)
@@ -412,7 +412,7 @@ agent:
   timeout_sec: 300
 verifier:
   timeout_sec: 120
-environment:
+sandbox:
   cpus: 1
   memory_mb: 2048
 ---

benchflow-0.6.8/src/benchflow/_utils/text.py

@@ -0,0 +1,86 @@
+"""Plain-text rendering for console and log lines.
+
+Two helpers live here: :func:`truncate_end`, which shortens a message
+without letting a sliced token read as a whole word, and
+:func:`describe_exception`, which renders an exception so the resulting
+line is never detail-free.
+
+The eval console renders one line per rollout, e.g.::
+
+    [ERR] my-task (tools=0) (Docker compose command failed ...)
+
+Error messages routinely embed task names and paths, and a bare
+``msg[:n]`` slice can cut one of those tokens in half — ``environment
+authored-task`` rendered as ``environment auth`` — which reads as a
+different, complete name.  ``truncate_end`` is the canonical fix: the cut
+only ever removes the tail, backs up to a word boundary when one exists,
+and is marked with an ellipsis so a shortened message can never
+masquerade as a full one.
+"""
+
+from __future__ import annotations
+
+
+def truncate_end(message: str, limit: int) -> str:
+    """Shorten ``message`` to at most ``limit`` characters, ending in ``…``.
+
+    The kept text is always a verbatim prefix of the original (interior
+    words are never dropped), backed up to the previous word boundary so
+    a sliced token cannot read as a complete word.  A single token longer
+    than the budget is still cut, with the ellipsis marking the cut.
+    Messages within budget are returned unchanged.
+    """
+    if len(message) <= limit:
+        return message
+    if limit <= 1:
+        return "…" if limit == 1 else ""
+    kept = message[: limit - 1]
+    if message[limit - 1] != " ":
+        head, sep, _partial = kept.rpartition(" ")
+        if sep:
+            kept = head
+    return kept.rstrip() + "…"
+
+
+def describe_exception(exc: BaseException) -> str:
+    """Render ``exc`` as a one-line description that is never detail-free.
+
+    ``f"{exc}"`` keeps only ``str(exc)``, which for some SDK errors carries
+    no information at all. The Daytona SDK wraps every toolbox call as
+    ``"<prefix>: " + str(underlying)``, and httpx raises its timeout and
+    connection errors with an *empty* message — so a read timeout on
+    ``execute_session_command`` stringifies to the bare
+    ``"Failed to execute session command: "``, a message whose detail after
+    the colon is empty. The exception *class* (``DaytonaTimeoutError`` vs
+    ``DaytonaConnectionError``) is then the only surviving evidence of what
+    actually went wrong, and plain interpolation discards it.
+
+    Lead with the class name so "the exec timed out" can never again be
+    indistinguishable from "the connection dropped". The trailing
+    ``status_code``/``error_code`` fields serve the *other* shape of SDK
+    failure — an OpenAPI exception carrying an HTTP response — and are
+    absent by construction for the detail-free transport case above, which
+    never reaches a response at all.
+    """
+    name = type(exc).__name__
+    message = str(exc).strip()
+    # Approximation: this tests the whole message, not specifically the
+    # detail after a wrapper prefix — a message that legitimately ends in a
+    # colon is annotated too. That is harmless, and the alternative means
+    # knowing every vendor's prefix.
+    if message.endswith(":"):
+        message = f"{message} (no detail)"
+    described = f"{name}: {message}" if message else f"{name} (no message)"
+    details: list[str] = []
+    status_code = getattr(exc, "status_code", None)
+    # Both fields are "absent" when falsy, but only ``status_code`` has a
+    # meaningful zero-ish value to protect (no HTTP status is 0, yet an
+    # explicit 0 should still surface as evidence of a malformed response).
+    if status_code is not None:
+        details.append(f"status_code={status_code}")
+    error_code = getattr(exc, "error_code", None)
+    if error_code:
+        details.append(f"error_code={error_code}")
+    if details:
+        described = f"{described} [{', '.join(details)}]"
+    return described

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/acp/container_transport.py

@@ -7,6 +7,7 @@ from typing import Any, TextIO
 
 from benchflow.sandbox.process import LiveProcess
 from benchflow.sandbox.process._base import _ANSI_CSI_RE, _ANSI_OSC_RE
+from benchflow.trajectories.types import redact_trajectory_text
 
 from .transport import Transport, decode_json_rpc_message
 
@@ -138,3 +139,7 @@ class ContainerTransport(Transport):
             self._agent_log_file.close()
             self._agent_log_file = None
         await self._cp.close()
+        stderr = getattr(self._cp, "stderr_tail", "")
+        if isinstance(stderr, str) and stderr and self._agent_log_path:
+            with self._agent_log_path.open("a") as agent_log:
+                agent_log.write(redact_trajectory_text(stderr))

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/acp/runtime.py

@@ -549,7 +549,7 @@ async def connect_acp(
     dormant and the auto-approve policy ran unconditionally (#382 follow-up).
 
     ``mcp_servers`` are the task's configured MCP servers (mapped from
-    ``[[environment.mcp_servers]]``); they are attached to the ACP session at
+    ``[[sandbox.mcp_servers]]``); they are attached to the ACP session at
     ``session/new`` so the agent can reach them. ``None`` attaches none.
 
     Retries with exponential backoff on ConnectionError (Daytona SSH storms).

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/adapters/_toolathlon.py

@@ -508,7 +508,7 @@ def _toolathlon_task_toml(
         "metadata": metadata,
         "agent": {"timeout_sec": 1800.0},
         "verifier": {"timeout_sec": 900.0},
-        "environment": environment,
+        "sandbox": environment,
     }
 
 

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/adapters/harbor.py

@@ -15,12 +15,14 @@ BenchFlow-native shape:
     └── tests/             # test.sh — the verifier
 
 This adapter is consequently a thin *normalizer*: it loads the foreign
-``task.toml`` through the native :class:`TaskConfig` validator (which already
-handles Harbor's ``[environment]``-keyed sandbox section and the
-``version`` -> ``schema_version`` rename), reads ``instruction.md``, and
-records the build/solution/test files under their native relative paths. No
-field remapping is needed — Harbor *is* the native format, which is exactly
-what makes Terminal-Bench backward-compatible through this edge.
+``task.toml`` through :func:`~benchflow.task.imports.import_task_config_toml`,
+whose toml import boundary translates Harbor's ``[environment]``-keyed
+sandbox section to the native ``sandbox`` key and handles the ``version`` ->
+``schema_version`` rename before native :class:`TaskConfig` validation. It
+then reads ``instruction.md`` and records the build/solution/test files under
+their native relative paths. Beyond that spelling translation no field
+remapping is needed — Harbor is structurally the native format, which is
+exactly what makes Terminal-Bench backward-compatible through this edge.
 """
 
 from __future__ import annotations

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/adapters/source.py

@@ -243,7 +243,7 @@ def _materialize_mcp_atlas(ctx: _SourceContext, output_dir: Path) -> None:
                         "MCP_ATLAS_JUDGE_MODEL": "${MCP_ATLAS_JUDGE_MODEL:-qwen/qwen-plus}",
                     },
                 },
-                "environment": {
+                "sandbox": {
                     "cpus": 4,
                     "memory_mb": 8192,
                     "storage_mb": 10240,

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/agents/openclaw_acp_shim.py

@@ -49,6 +49,17 @@ _PARAM_MAP = {
 }
 
 
+def _default_max_tokens(model: str) -> int | None:
+    # OpenClaw derives an invalid ~172k default for GPT-5.4; ACP sends a
+    # BenchFlow-generated LiteLLM alias in normal runs.
+    model = model.removeprefix("openai/")
+    if model == "gpt-5.4" or (
+        model.startswith("benchflow-") and model.endswith("-gpt-5.4")
+    ):
+        return 128000
+    return None
+
+
 # ── ACP stdio I/O ─────────────────────────────────────────────────────────────
 
 
@@ -671,6 +682,7 @@ def main():
 
         elif method == "session/set_model":
             model = params.get("modelId", "")
+            requested_model = model
             # A provider-resolution / config-write failure here must NOT crash the
             # shim: an unhandled exception exits rc=1, which benchflow sees as the
             # ACP transport dying mid-set_model ("Process closed stdout (rc=1)") —
@@ -726,6 +738,25 @@ def main():
                             check=True,
                             timeout=10,
                         )
+                max_tokens = _default_max_tokens(requested_model)
+                configured_max_tokens = os.environ.get("BENCHFLOW_MODEL_MAX_TOKENS")
+                if max_tokens is not None and (
+                    not configured_max_tokens
+                    or not configured_max_tokens.isdigit()
+                    or int(configured_max_tokens) > max_tokens
+                ):
+                    subprocess.run(
+                        [
+                            _OPENCLAW_BIN,
+                            "config",
+                            "set",
+                            _PARAM_MAP["BENCHFLOW_MODEL_MAX_TOKENS"],
+                            str(max_tokens),
+                        ],
+                        capture_output=True,
+                        check=True,
+                        timeout=10,
+                    )
             except Exception as exc:
                 diag = (
                     f"[openclaw-acp-shim] set_model setup failed, continuing: {exc!r}"

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/agents/registry.py

@@ -53,6 +53,8 @@ import shlex
 from dataclasses import dataclass, field
 from pathlib import Path
 
+from benchflow._utils.text import describe_exception
+
 
 def _install_python_script(container_path: str, source: str) -> str:
     """Shell snippet that ensures python3 and writes `source` to container_path.
@@ -124,9 +126,7 @@ _JS_AGENT_PATH = (
     f"{_BENCHFLOW_BIN_PREFIX}:{_BENCHFLOW_JS_AGENT_PREFIX}/bin:"
     f"{_BENCHFLOW_NODE_PREFIX}/bin:$PATH"
 )
-# Node >=22.19 is required by current openclaw (the JS agents install
-# @latest); keep this pin at or above that floor or the openclaw ACP
-# bootstrap aborts at its runtime version check (BF-10).
+# Node 22.20.0 supports OpenClaw 2026.6.9. Keep their pin pair in sync.
 _NODE_INSTALL = (
     "export DEBIAN_FRONTEND=noninteractive; "
     f"BF_NODE_DIR={_BENCHFLOW_NODE_PREFIX}; "
@@ -583,7 +583,7 @@ AGENTS: dict[str, AgentConfig] = {
         description="OpenClaw agent via ACP shim — model set at runtime via --model",
         skill_paths=["$HOME/.claude/skills", "$WORKSPACE/skills"],
         install_cmd=(
-            f"{_js_agent_install('openclaw', 'openclaw')} && "
+            f"{_js_agent_install('openclaw', 'openclaw@2026.6.9')} && "
             # Configure: auto-approve tools (no model — set at runtime via ACP set_model)
             "mkdir -p ~/.openclaw && "
             'echo \'{"version":1,"defaults":{"allow_all":true}}\''
@@ -1469,7 +1469,7 @@ def _load_agent_plugin_packages() -> None:
     try:
         eps = entry_points(group="benchflow.agents")
     except Exception as exc:  # pragma: no cover - metadata backend quirks
-        FAILED_AGENT_PLUGINS["<entry-point-scan>"] = f"{type(exc).__name__}: {exc}"
+        FAILED_AGENT_PLUGINS["<entry-point-scan>"] = describe_exception(exc)
         logging.getLogger(__name__).warning(
             "benchflow.agents entry-point scan failed; ALL agent plugins are "
             "disabled: %s",
@@ -1483,7 +1483,7 @@ def _load_agent_plugin_packages() -> None:
             if callable(loaded):
                 loaded()
         except Exception as exc:
-            FAILED_AGENT_PLUGINS[ep.name] = f"{type(exc).__name__}: {exc}"
+            FAILED_AGENT_PLUGINS[ep.name] = describe_exception(exc)
             logging.getLogger(__name__).warning(
                 "benchflow.agents plugin %r failed to load (some or all of its "
                 "agents may be unavailable or partially registered): %s",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/cli/_live_progress.py

@@ -135,13 +135,21 @@ def _fmt_tokens(n: int) -> str:
 # while the session counters are unavailable — before the agent session
 # exists and after it is torn down — so sandbox create, agent install, and
 # the verifier read as work, not as a hang.
+#
+# The labels must be monotonic across a run: the row is a progress indicator,
+# so stepping back to an earlier stage reads as the run having restarted.
+# "executed" maps to the agent stage, not the verifier: since verify() marks
+# "verifying" at its own entry, the only stretch that renders under "executed"
+# is the inside of disconnect() (session already dropped, agent process being
+# killed) — labelling that "verifying…" made the row show "verifying…", then
+# "running agent…" (disconnect's "installed"), then "verifying…" again.
 _PHASE_LABELS = {
     "created": "creating sandbox…",
     "setup": "creating sandbox…",
     "started": "installing agent…",
     "installed": "running agent…",
     "connected": "running agent…",
-    "executed": "verifying…",
+    "executed": "running agent…",
     "verifying": "verifying…",
     "verified": "cleaning up…",
     "cleaned": "cleaning up…",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8}/src/benchflow/cli/main.py

@@ -57,6 +57,7 @@ from benchflow.cli.sandbox import register_sandbox
 from benchflow.cli.skills import register_skills
 from benchflow.cli.tasks import register_tasks
 from benchflow.cli.train import register_train
+from benchflow.cli.traj import register_traj
 from benchflow.eval_plan import EvalCreateRequest, EvalPlanError, build_eval_plan
 from benchflow.evaluation import DEFAULT_AGENT, effective_model
 from benchflow.sandbox.providers import providers_phrase
@@ -1242,6 +1243,7 @@ register_continue(eval_app, alias_app=app)
 register_skills(app)
 register_review(app)
 register_tasks(app)
+register_traj(app)
 register_train(app)
 register_hub(app)
 register_agent(app)