benchflow 0.6.7.dev1845__tar.gz → 0.6.8.dev1861__tar.gz

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 ## [Unreleased]
 
+### Changed
+- **BREAKING (task.md): the `environment:` frontmatter key is renamed to
+  `sandbox:`.** The native task-config surface now accepts only `sandbox:`
+  (plus `verifier.sandbox:` for the verifier's separate sandbox spec and
+  `verifier.sandbox_mode:` for shared/separate selection); `environment:`,
+  `verifier.environment:`, and `verifier.environment_mode:` no longer
+  validate and fail with an actionable message naming the rename. **The
+  one-line fix for existing task.md files is renaming the key.**
+  Legacy/Harbor `task.toml` imports are unaffected: the toml loader
+  converts `[environment]`, `[verifier.environment]`, and
+  `environment_mode` to the `sandbox` spellings (declaring both spellings
+  in one file is an error), and `bench tasks export` emits the inverse —
+  a stock-Harbor `[environment]`-spelled `task.toml`. All native emitters
+  — `model_dump_toml`, `bench tasks migrate`, task scaffolding,
+  skill-eval/trace/adapter task generation, rubric-review wrappers — now
+  write `sandbox`. Python API: the compat property
+  `TaskConfig.environment` is removed (use `TaskConfig.sandbox`),
+  `VerifierConfig.environment`/`environment_mode` became
+  `sandbox`/`sandbox_mode`, and the `VerifierEnvironmentMode` enum is now
+  `VerifierSandboxMode`. The Environment plane
+  (`--environment-manifest`, `benchflow.environment.manifest`, the
+  eval-config `environment:` docker/daytona selector) is a different
+  subsystem and is unchanged.
+
+## 0.6.7 — 2026-08-09
+
 ### Added
 - **Built-in environment registry.** The committed env-axis pins
   (`env0@prod`, `env0@outage`) moved from `benchmarks/_environments/` into
@@ -10,13 +36,71 @@
   `pip install benchflow` with no checkout and no env vars.
   `$BENCHFLOW_ENV_REGISTRY`, when set, still wins entirely; resolution
   stays content-addressed (sha256 logged), and unknown names now error
-  listing the available specs.
+  listing the available specs. (#961)
 - **Console progress heartbeat.** Single-concurrency eval runs print a
   throttled progress line (`… 6.2min, 12 tool calls (last: …)`) about every
   45 seconds while the agent works, so a long prompt is distinguishable from
   a hang. The heartbeat is auto-gated off for multi-concurrency jobs;
   `bench eval run --quiet` suppresses it, and `BENCHFLOW_PROGRESS=on`/`off`
   overrides the auto-gate. (#951)
+- **Live per-task activity in the eval dashboard.** Under a TTY the
+  running-now table gains an activity column ("38 calls · last:
+  file_editor", plus tokens once a usage snapshot exists), polled from the
+  ACP session's existing heartbeat counters. The agents-manifest autoload
+  also clones quietly — one "Cloning …" summary line instead of raw git
+  progress. (#956)
+- **Phase labels and per-task failure reasons.** The activity cell is never
+  blank while a row exists: sandbox create, agent install, and verify show
+  dim phase labels ("creating sandbox…", "installing agent…",
+  "verifying…"). The final block prints one dim "✗ task: reason" line per
+  failed task — verifier error first, else a compact reward/metric
+  breakdown, else the reward — capped at 5 lines. `--quiet` now silences
+  the dashboard as well as the heartbeat. (#957)
+- **Failure reasons mined from verifier artifacts.** When a displayed
+  failure would read as a bare "reward X", the CLI reads a small bounded
+  artifact from the rollout's own verifier dir — the CTRF report (first
+  failed test plus its assertion line) or a tail of `test-stdout.txt` — and
+  a dim "(details: …/verifier)" pointer names the artifact directory.
+  Artifacts resolve by the recorded rollout name, never by glob. (#959)
+- **Fractional rewards in console summaries.** Per-task lines carry the
+  scored reward (`✗ task (reward=0.30, tools=47)`), the Score line renders
+  ", mean reward 0.30" alongside the binarized counts, and
+  `EvaluationResult` / `summary.json` gain a `mean_reward` field (mean over
+  scored rollouts; errors excluded, not zeroed). (#960)
+- **Full failure counts on per-task console lines.** A CTRF report with
+  more than one failed test rolls up as " (+N more failure(s); P/T checks
+  passed)" after the first failure — the count suffix is never truncated
+  away — and parametrized test names keep their `[param]` ids whenever the
+  report carries them. (#962)
+- **Live token usage in the dashboard footer.** The footer sums completed
+  tasks' trusted telemetry plus every running rollout's live ACP session
+  usage, so spend is visible while the run executes (usage lands per
+  completed prompt; cost stays scoring-time from the gateway log).
+  Single-tool agents' activity cell drops the redundant "last:" suffix in
+  favor of "38 calls · 412.0k tok" once tokens are available. (#963)
+- **env0-shaped failure breakdowns.** The failure-reason tiers understand
+  metrics nested one level under `metrics` / `details`, pair
+  `<name>_found` with `<name>_total` into fractions ("deadlines 1/5",
+  lowest-signal first), probe `verifier/reward.json` on disk between the
+  CTRF and stdout tiers, and every failure block with on-disk verifier
+  artifacts gets one "(details: …)" pointer. (#964)
+- **Mid-prompt live token usage via the gateway's live capture.** The #963
+  live tokens stepped forward only when an ACP prompt completed, so a
+  single-prompt rollout showed "— tokens" for its whole agent phase. The
+  proxy runtime's existing live-capture loop (which already tails the
+  sandbox gateway's callback log every second) now also accumulates
+  provider tokens into an O(1) counter, and the dashboard reconciles the
+  two non-decreasing live signals as max(ACP, gateway) — display-only,
+  still replaced by the trusted scoring import at completion, and any
+  gateway-side failure degrades to the ACP-only behavior. (#965)
+
+- **Compact flow-style arrays in emitted task.md frontmatter.** Short
+  scalar-only lists render on one line (`tags: [parsing, nlp]`) instead of
+  multiline bullets, so hand-written flow arrays survive `bench tasks
+  migrate` / normalize round-trips. The style predicate measures each list
+  with the real YAML emitter (so it can never disagree with PyYAML's own
+  quoting) and falls back to block style for long, nested, or multiline
+  items. Parsing is unchanged — both styles were always accepted. (#967)
 
 ### Changed
 - Migrated the clawsbench `archive-amazon-shipping` task to the native
@@ -51,6 +135,27 @@
   environment now fails loud when it declares services but none are startable
   in the image, instead of passing a vacuous readiness gate and dying at the
   verifier. (#954)
+- Pointed the built-in `env0@prod` / `env0@outage` pins at the org-owned
+  `ghcr.io/benchflow-ai/env0:0.2.0` base image. The wheel-shipped pins named a
+  personal Docker Hub image while their own comments and the environment docs
+  declared the ghcr image authoritative; `base_image` is recorded as rollout
+  provenance, so every env0 result carried the wrong base. Verified on Daytona
+  (env0 `auth-least-privilege-summary`, 8/8 services ready, reward 1.0).
+- Restored the `env0@outage` perturbation (gmail and slack removed relative
+  to `env0@prod`) that the #954 upstream sync had erased by mirroring the
+  full service list into both pins, and corrected the pin header's stale
+  slack port. (#955)
+- **ACP protocol JSON glued to PTY shell noise now decodes.** On PTY
+  transports, an agent's initialize response could arrive on the same line
+  as the shell prompt (ANSI/OSC-prefixed), fail the strict whole-line JSON
+  parse, and get filed as noise — the handshake then "timed out" at any
+  window with the answer sitting in the agent log. The PTY-facing transport
+  now retries from the first `{` and once more after an ANSI scrub, but
+  only until the first successfully decoded protocol message per
+  connection; afterwards the strict contract rules, so log-echoed envelopes
+  cannot impersonate protocol traffic mid-session. The pre-prompt handshake
+  window is also env-configurable via `BENCHFLOW_ACP_HANDSHAKE_TIMEOUT`
+  (seconds; default 60). (#958)
 
 ## 0.6.6 — 2026-08-04
 

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: benchflow
-Version: 0.6.7.dev1845
+Version: 0.6.8.dev1861
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.7.dev1845"
+version = "0.6.8.dev1861"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/_utils/config_override.py

@@ -144,8 +144,9 @@ def apply_config_override(config: Any, overlay: dict[str, Any] | None) -> Any:
 
     # Merge against the FIELD-NAME dump (``by_alias=False``) so overlays use the
     # canonical field names (``agent``, ``sandbox``, …); ``populate_by_name`` lets
-    # re-validation accept them. ``by_alias=True`` would force callers to write the
-    # alias (e.g. ``environment`` for ``sandbox``) — the bug this avoids.
+    # re-validation accept them. (``sandbox`` is now the only spelling — the
+    # legacy ``environment`` alias was removed in the rename — but ``oracle``
+    # still serializes via alias, so the field-name dump stays load-bearing.)
     merged = deep_merge(config.model_dump(by_alias=False), overlay)
     patched = TaskConfig.model_validate(merged)
     logger.debug(

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/_utils/scoring.py

@@ -4,7 +4,10 @@ import math
 from collections.abc import Iterable, Mapping
 from typing import Any, Literal
 
-from benchflow.diagnostics import DIAGNOSTIC_REASON_IDLE_TIMEOUT
+from benchflow.diagnostics import (
+    DIAGNOSTIC_REASON_IDLE_TIMEOUT,
+    TRANSIENT_SANDBOX_TRANSPORT_MARKER,
+)
 
 # Error category constants
 INSTALL_FAILED = "install_failure"
@@ -174,6 +177,14 @@ def _looks_like_infra_error(error: str) -> bool:
             "connection reset",
             "connection refused",
             "broken pipe",
+            # Stamped by TransientSandboxTransportError at the sandbox-SDK
+            # boundary, where the vendor exception type is still available to
+            # judge. One marker covers every provider call — exec, upload,
+            # download, stat — instead of one entry per vendor message
+            # prefix, which would be endless and silently incomplete.
+            TRANSIENT_SANDBOX_TRANSPORT_MARKER,
+            # Predates the marker above and is kept for strings rebuilt
+            # outside that boundary (e.g. a verifier error re-raised as text).
             "failed to get session command",
             "sandbox not found",
             "workspace not found",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/_utils/task_authoring/scaffolding.py

@@ -333,7 +333,7 @@ timeout_sec = 300
 [verifier]
 timeout_sec = 120
 
-[environment]
+[sandbox]
 cpus = 1
 memory_mb = 2048
 """)
@@ -412,7 +412,7 @@ agent:
   timeout_sec: 300
 verifier:
   timeout_sec: 120
-environment:
+sandbox:
   cpus: 1
   memory_mb: 2048
 ---

benchflow-0.6.8.dev1861/src/benchflow/_utils/text.py

@@ -0,0 +1,86 @@
+"""Plain-text rendering for console and log lines.
+
+Two helpers live here: :func:`truncate_end`, which shortens a message
+without letting a sliced token read as a whole word, and
+:func:`describe_exception`, which renders an exception so the resulting
+line is never detail-free.
+
+The eval console renders one line per rollout, e.g.::
+
+    [ERR] my-task (tools=0) (Docker compose command failed ...)
+
+Error messages routinely embed task names and paths, and a bare
+``msg[:n]`` slice can cut one of those tokens in half — ``environment
+authored-task`` rendered as ``environment auth`` — which reads as a
+different, complete name.  ``truncate_end`` is the canonical fix: the cut
+only ever removes the tail, backs up to a word boundary when one exists,
+and is marked with an ellipsis so a shortened message can never
+masquerade as a full one.
+"""
+
+from __future__ import annotations
+
+
+def truncate_end(message: str, limit: int) -> str:
+    """Shorten ``message`` to at most ``limit`` characters, ending in ``…``.
+
+    The kept text is always a verbatim prefix of the original (interior
+    words are never dropped), backed up to the previous word boundary so
+    a sliced token cannot read as a complete word.  A single token longer
+    than the budget is still cut, with the ellipsis marking the cut.
+    Messages within budget are returned unchanged.
+    """
+    if len(message) <= limit:
+        return message
+    if limit <= 1:
+        return "…" if limit == 1 else ""
+    kept = message[: limit - 1]
+    if message[limit - 1] != " ":
+        head, sep, _partial = kept.rpartition(" ")
+        if sep:
+            kept = head
+    return kept.rstrip() + "…"
+
+
+def describe_exception(exc: BaseException) -> str:
+    """Render ``exc`` as a one-line description that is never detail-free.
+
+    ``f"{exc}"`` keeps only ``str(exc)``, which for some SDK errors carries
+    no information at all. The Daytona SDK wraps every toolbox call as
+    ``"<prefix>: " + str(underlying)``, and httpx raises its timeout and
+    connection errors with an *empty* message — so a read timeout on
+    ``execute_session_command`` stringifies to the bare
+    ``"Failed to execute session command: "``, a message whose detail after
+    the colon is empty. The exception *class* (``DaytonaTimeoutError`` vs
+    ``DaytonaConnectionError``) is then the only surviving evidence of what
+    actually went wrong, and plain interpolation discards it.
+
+    Lead with the class name so "the exec timed out" can never again be
+    indistinguishable from "the connection dropped". The trailing
+    ``status_code``/``error_code`` fields serve the *other* shape of SDK
+    failure — an OpenAPI exception carrying an HTTP response — and are
+    absent by construction for the detail-free transport case above, which
+    never reaches a response at all.
+    """
+    name = type(exc).__name__
+    message = str(exc).strip()
+    # Approximation: this tests the whole message, not specifically the
+    # detail after a wrapper prefix — a message that legitimately ends in a
+    # colon is annotated too. That is harmless, and the alternative means
+    # knowing every vendor's prefix.
+    if message.endswith(":"):
+        message = f"{message} (no detail)"
+    described = f"{name}: {message}" if message else f"{name} (no message)"
+    details: list[str] = []
+    status_code = getattr(exc, "status_code", None)
+    # Both fields are "absent" when falsy, but only ``status_code`` has a
+    # meaningful zero-ish value to protect (no HTTP status is 0, yet an
+    # explicit 0 should still surface as evidence of a malformed response).
+    if status_code is not None:
+        details.append(f"status_code={status_code}")
+    error_code = getattr(exc, "error_code", None)
+    if error_code:
+        details.append(f"error_code={error_code}")
+    if details:
+        described = f"{described} [{', '.join(details)}]"
+    return described

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/acp/runtime.py

@@ -549,7 +549,7 @@ async def connect_acp(
     dormant and the auto-approve policy ran unconditionally (#382 follow-up).
 
     ``mcp_servers`` are the task's configured MCP servers (mapped from
-    ``[[environment.mcp_servers]]``); they are attached to the ACP session at
+    ``[[sandbox.mcp_servers]]``); they are attached to the ACP session at
     ``session/new`` so the agent can reach them. ``None`` attaches none.
 
     Retries with exponential backoff on ConnectionError (Daytona SSH storms).

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/adapters/_toolathlon.py

@@ -508,7 +508,7 @@ def _toolathlon_task_toml(
         "metadata": metadata,
         "agent": {"timeout_sec": 1800.0},
         "verifier": {"timeout_sec": 900.0},
-        "environment": environment,
+        "sandbox": environment,
     }
 
 

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/adapters/harbor.py

@@ -15,12 +15,14 @@ BenchFlow-native shape:
     └── tests/             # test.sh — the verifier
 
 This adapter is consequently a thin *normalizer*: it loads the foreign
-``task.toml`` through the native :class:`TaskConfig` validator (which already
-handles Harbor's ``[environment]``-keyed sandbox section and the
-``version`` -> ``schema_version`` rename), reads ``instruction.md``, and
-records the build/solution/test files under their native relative paths. No
-field remapping is needed — Harbor *is* the native format, which is exactly
-what makes Terminal-Bench backward-compatible through this edge.
+``task.toml`` through :func:`~benchflow.task.imports.import_task_config_toml`,
+whose toml import boundary translates Harbor's ``[environment]``-keyed
+sandbox section to the native ``sandbox`` key and handles the ``version`` ->
+``schema_version`` rename before native :class:`TaskConfig` validation. It
+then reads ``instruction.md`` and records the build/solution/test files under
+their native relative paths. Beyond that spelling translation no field
+remapping is needed — Harbor is structurally the native format, which is
+exactly what makes Terminal-Bench backward-compatible through this edge.
 """
 
 from __future__ import annotations

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/adapters/source.py

@@ -243,7 +243,7 @@ def _materialize_mcp_atlas(ctx: _SourceContext, output_dir: Path) -> None:
                         "MCP_ATLAS_JUDGE_MODEL": "${MCP_ATLAS_JUDGE_MODEL:-qwen/qwen-plus}",
                     },
                 },
-                "environment": {
+                "sandbox": {
                     "cpus": 4,
                     "memory_mb": 8192,
                     "storage_mb": 10240,

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/agents/registry.py

@@ -53,6 +53,8 @@ import shlex
 from dataclasses import dataclass, field
 from pathlib import Path
 
+from benchflow._utils.text import describe_exception
+
 
 def _install_python_script(container_path: str, source: str) -> str:
     """Shell snippet that ensures python3 and writes `source` to container_path.
@@ -1469,7 +1471,7 @@ def _load_agent_plugin_packages() -> None:
     try:
         eps = entry_points(group="benchflow.agents")
     except Exception as exc:  # pragma: no cover - metadata backend quirks
-        FAILED_AGENT_PLUGINS["<entry-point-scan>"] = f"{type(exc).__name__}: {exc}"
+        FAILED_AGENT_PLUGINS["<entry-point-scan>"] = describe_exception(exc)
         logging.getLogger(__name__).warning(
             "benchflow.agents entry-point scan failed; ALL agent plugins are "
             "disabled: %s",
@@ -1483,7 +1485,7 @@ def _load_agent_plugin_packages() -> None:
             if callable(loaded):
                 loaded()
         except Exception as exc:
-            FAILED_AGENT_PLUGINS[ep.name] = f"{type(exc).__name__}: {exc}"
+            FAILED_AGENT_PLUGINS[ep.name] = describe_exception(exc)
             logging.getLogger(__name__).warning(
                 "benchflow.agents plugin %r failed to load (some or all of its "
                 "agents may be unavailable or partially registered): %s",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/cli/_live_progress.py

@@ -135,13 +135,21 @@ def _fmt_tokens(n: int) -> str:
 # while the session counters are unavailable — before the agent session
 # exists and after it is torn down — so sandbox create, agent install, and
 # the verifier read as work, not as a hang.
+#
+# The labels must be monotonic across a run: the row is a progress indicator,
+# so stepping back to an earlier stage reads as the run having restarted.
+# "executed" maps to the agent stage, not the verifier: since verify() marks
+# "verifying" at its own entry, the only stretch that renders under "executed"
+# is the inside of disconnect() (session already dropped, agent process being
+# killed) — labelling that "verifying…" made the row show "verifying…", then
+# "running agent…" (disconnect's "installed"), then "verifying…" again.
 _PHASE_LABELS = {
     "created": "creating sandbox…",
     "setup": "creating sandbox…",
     "started": "installing agent…",
     "installed": "running agent…",
     "connected": "running agent…",
-    "executed": "verifying…",
+    "executed": "running agent…",
     "verifying": "verifying…",
     "verified": "cleaning up…",
     "cleaned": "cleaning up…",

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/continue_run/orchestrator.py

@@ -30,6 +30,7 @@ from dataclasses import dataclass
 from pathlib import Path
 from typing import Any, cast
 
+from benchflow._utils.text import describe_exception
 from benchflow.continue_run.replay_proxy import ReplayProxy, ReplayRouter
 from benchflow.continue_run.run_folder import RunFolder, RunFolderError, load_run_folder
 from benchflow.continue_run.sandbox_proxy import (
@@ -442,7 +443,7 @@ async def _safe_sandbox_continuation_teardown(
         try:
             await awaitable
         except Exception as exc:
-            message = f"{label}: {exc}"
+            message = f"{label}: {describe_exception(exc)}"
             errors.append(message)
             logger.warning("Continuation teardown step failed: %s", message)
 

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/demo_task/task.md

@@ -24,7 +24,7 @@ verifier:
     cleanup_conftests: true
 agent:
   timeout_sec: 120.0
-environment:
+sandbox:
   network_mode: public
   build_timeout_sec: 600.0
   os: linux

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/diagnostics.py

@@ -477,6 +477,33 @@ class RolloutDiagnostics:
         return d if isinstance(d, TransportClosedDiagnostic) else None
 
 
+TRANSIENT_SANDBOX_TRANSPORT_MARKER = "transient sandbox transport failure"
+
+
+class TransientSandboxTransportError(ConnectionError):
+    """A sandbox-provider API call failed at the transport layer.
+
+    Raised at the SDK boundary, where the vendor exception *type* is still
+    alive, and carrying one stable marker so downstream classification never
+    has to keep a census of vendor message prefixes.
+
+    This exists because the vendor prefix is the wrong thing to match on.
+    The Daytona SDK stamps a different prefix per method — "Failed to
+    execute session command: ", "Failed to upload files: ", "Failed to get
+    file info: " — while the *failure* is identical in every case: an httpx
+    timeout or disconnect against the toolbox API, with an empty message.
+    Enumerating those prefixes in the classifier means one entry per vendor
+    method, silently incomplete the moment the SDK adds or renames one, and
+    it also over-matches: a permanent 401/400/409 shares the prefix but is
+    not transient. Deciding at the raise site, where
+    ``_is_daytona_transient_retry_error`` can inspect the real class, keeps
+    both halves honest.
+    """
+
+    def __init__(self, detail: str) -> None:
+        super().__init__(f"{TRANSIENT_SANDBOX_TRANSPORT_MARKER}: {detail}")
+
+
 # Summary / check_results helpers driven by the registry
 
 
@@ -514,6 +541,8 @@ __all__ = [
     "IdleTimeoutError",
     "TransportClosedError",
     "RolloutDiagnostics",
+    "TRANSIENT_SANDBOX_TRANSPORT_MARKER",
+    "TransientSandboxTransportError",
     "summary_warning",
     "format_issue_for_field",
 ]

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/environment/_registry/env0@outage.toml

@@ -9,10 +9,13 @@
 # resolves on a bare pip install with no $BENCHFLOW_ENV_REGISTRY set:
 #
 #   bench eval create --tasks-dir <task> --environment-manifest env0@outage ...
+#
+# NOTE: ghcr.io/benchflow-ai/env0:0.2.0 is amd64-only — run env0 on Daytona
+# (x86_64), not local Docker on Apple Silicon.
 
 [environment]
 name           = "env-0"
-base_image     = "xdotli/env0-base:latest"
+base_image     = "ghcr.io/benchflow-ai/env0:0.2.0"
 owns_lifecycle = false
 isolation      = "per_task"
 
@@ -22,13 +25,9 @@ mechanism = "image"
 [environment.readiness]
 timeout_sec = 60
 
-# Services below are copied VERBATIM from the upstream env-0.toml pin
-# (2026-08-09 sync). The previous pin had drifted: bare CLI names
-# (auth/gmail/...) that do not exist in ghcr.io/benchflow-ai/env0:0.2.0
-# images (which ship mock-auth/mock-gmail/... console scripts), plus a
-# phantom slack service and shifted ports — so ManifestEnvironment
-# probed, found ZERO startable services, and readiness passed vacuously
-# while every rollout died at the verifier with connection-refused.
+# Services below are copied VERBATIM from the upstream env-0.toml pin,
+# minus this variant's deliberate outage (mock-gmail and mock-slack
+# removed).
 
 [[environment.services]]
 name    = "mock-auth"

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/environment/_registry/env0@prod.toml

@@ -1,7 +1,7 @@
 # env0@prod — pinned, content-addressed Environment-plane manifest for env-0.
 #
 # This is a built-in registry entry, shipped inside the benchflow wheel
-# (benchflow/environment/_registry/) so PR #790's env-axis pin resolves on a
+# (benchflow/environment/_registry/) so the env-axis pin resolves on a
 # bare pip install — no checkout, no env vars:
 #
 #   bench eval create --tasks-dir <env0 tasks> --environment-manifest env0@prod ...
@@ -9,16 +9,18 @@
 # Set $BENCHFLOW_ENV_REGISTRY to a local directory of name@version.toml files
 # to override the built-in registry entirely.
 #
-# Mirrors benchflow-ai/env0 tasks/_manifests/env-0.toml (the source of truth). env-0's per-task images build
-# FROM the shared base and bake seed data; benchflow's ManifestEnvironment
-# probes each service's CLI and starts only the ones installed in the per-task
-# image, gating the agent on each service's /health.
+# Mirrors benchflow-ai/env0 tasks/_manifests/env-0.toml (the source of truth).
+# env-0's per-task images build FROM the shared base and bake seed data;
+# benchflow's ManifestEnvironment probes each service's CLI and starts only
+# the ones installed in the per-task image, gating the agent on each
+# service's /health.
 #
-# NOTE: xdotli/env0-base:latest is amd64-only — run env0 on Daytona (x86_64).
+# NOTE: ghcr.io/benchflow-ai/env0:0.2.0 is amd64-only — run env0 on Daytona
+# (x86_64), not local Docker on Apple Silicon.
 
 [environment]
 name           = "env-0"
-base_image     = "xdotli/env0-base:latest"
+base_image     = "ghcr.io/benchflow-ai/env0:0.2.0"
 owns_lifecycle = false
 isolation      = "per_task"
 
@@ -28,13 +30,7 @@ mechanism = "image"
 [environment.readiness]
 timeout_sec = 60
 
-# Services below are copied VERBATIM from the upstream env-0.toml pin
-# (2026-08-09 sync). The previous pin had drifted: bare CLI names
-# (auth/gmail/...) that do not exist in ghcr.io/benchflow-ai/env0:0.2.0
-# images (which ship mock-auth/mock-gmail/... console scripts), plus a
-# phantom slack service and shifted ports — so ManifestEnvironment
-# probed, found ZERO startable services, and readiness passed vacuously
-# while every rollout died at the verifier with connection-refused.
+# Services below are copied VERBATIM from the upstream env-0.toml pin.
 
 [[environment.services]]
 name    = "mock-auth"

{benchflow-0.6.7.dev1845 → benchflow-0.6.8.dev1861}/src/benchflow/experimental/mcp/reviewer_server.py

@@ -9,7 +9,7 @@ with a clean tool-call interface that prevents reward hacking (reviewer
 never has write access to /app/).
 
 Usage in task.toml:
-    [[environment.mcp_servers]]
+    [[sandbox.mcp_servers]]
     name = "reviewer"
     transport = "streamable-http"
     url = "http://localhost:8100/mcp"