benchflow 0.6.7.dev1823__tar.gz → 0.6.7.dev1827__tar.gz

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/CHANGELOG.md

@@ -2,9 +2,47 @@
 
 ## [Unreleased]
 
+### Added
+- **Console progress heartbeat.** Single-concurrency eval runs print a
+  throttled progress line (`… 6.2min, 12 tool calls (last: …)`) about every
+  45 seconds while the agent works, so a long prompt is distinguishable from
+  a hang. The heartbeat is auto-gated off for multi-concurrency jobs;
+  `bench eval run --quiet` suppresses it, and `BENCHFLOW_PROGRESS=on`/`off`
+  overrides the auto-gate. (#951)
+
+### Changed
+- Migrated the clawsbench `archive-amazon-shipping` task to the native
+  `task.md` package layout (`bench tasks migrate --remove-legacy`), and a
+  clawsbench run launched without `--environment-manifest` now fails with an
+  actionable verifier message naming the exact flag to pass instead of an
+  opaque connection error. (#952)
+
 ### Fixed
 - Isolated pull-request and post-test integration concurrency groups so a
   credential-free workflow completion cannot cancel an active PR rollout.
+- **Host-side hard rollout deadline.** An await wedged below the phase-level
+  watchdogs (e.g. a Daytona PTY teardown on a dead websocket) could freeze an
+  entire eval job. Every rollout attempt now runs under a host-side hard
+  deadline derived from the task's own phase budgets plus a fixed margin; a
+  trip returns a retryable infra-error result and abandons the sandbox after
+  a bounded cleanup grace. Override with `BENCHFLOW_ROLLOUT_HARD_DEADLINE`
+  (seconds; `off`/`none`/`0` disables). Verifier timeouts that produced zero
+  output are retried once — that signature is an exec-layer wedge, not a slow
+  verifier. Agent-sent reasoning parameters are also forwarded through the
+  LLM gateway, and DeepSeek routes natively with `reasoning_effort` passed
+  verbatim. (#949)
+- Quieter Daytona teardown: successful runs end at the score line — the
+  atexit client cleanup no longer prints a cancelled-reader traceback after
+  `✓ Score`, and benign engineio PTY-disconnect errors are silenced. (#951)
+- Accept the Harbor 1.3 `[task] version` field in task configs. It is
+  informational and stored verbatim; the strict schema previously rejected
+  it, which made every Harbor-1.3 curated task unloadable. (#953)
+- Synced the committed `env0@prod` / `env0@outage` environment pins with the
+  upstream `benchflow-ai/env0` manifest (the pins had drifted to stale
+  service CLI names, a phantom service, and shifted ports), and a manifest
+  environment now fails loud when it declares services but none are startable
+  in the image, instead of passing a vacuous readiness gate and dying at the
+  verifier. (#954)
 
 ## 0.6.6 — 2026-08-04
 

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: benchflow
-Version: 0.6.7.dev1823
+Version: 0.6.7.dev1827
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.7.dev1823"
+version = "0.6.7.dev1827"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/src/benchflow/_utils/benchmark_repos.py

@@ -95,11 +95,21 @@ def _looks_like_commit_sha(ref: str) -> bool:
 
 def _checkout_fetched_ref(repo_root: Path, ref: str) -> None:
     subprocess.run(
-        ["git", "-C", str(repo_root), "fetch", "--depth", "1", "origin", ref],
+        [
+            "git",
+            "-C",
+            str(repo_root),
+            "fetch",
+            "--quiet",
+            "--depth",
+            "1",
+            "origin",
+            ref,
+        ],
         check=True,
     )
     subprocess.run(
-        ["git", "-C", str(repo_root), "checkout", "--detach", "FETCH_HEAD"],
+        ["git", "-C", str(repo_root), "checkout", "--quiet", "--detach", "FETCH_HEAD"],
         check=True,
     )
 
@@ -155,7 +165,10 @@ def _clone_repo_unlocked(org: str, repo: str, ref: str | None = None) -> Path:
     try:
         if clone_tmp.exists():
             shutil.rmtree(clone_tmp)
-        cmd = ["git", "clone", "--depth", "1"]
+        # --quiet: raw clone progress ("remote: Counting objects: …") on the
+        # console pre-dashboard is noise; the "Cloning …" log line above is
+        # the one-line summary. Errors still print (git keeps stderr on fail).
+        cmd = ["git", "clone", "--quiet", "--depth", "1"]
         if ref and not _looks_like_commit_sha(ref):
             cmd.extend(["--branch", ref])
         cmd.extend([url, str(clone_tmp)])

benchflow-0.6.7.dev1827/src/benchflow/_utils/live_activity.py

@@ -0,0 +1,53 @@
+"""Process-global registry of live rollouts for the eval dashboard.
+
+The Rich dashboard (``cli/_live_progress.py``) renders in the same process as
+the engine's rollouts, so surfacing per-task activity needs no new event
+plumbing: the engine registers each live Rollout under its task name and the
+dashboard polls the rollout's ACP session counters — the exact ones the
+console heartbeat (``ACPSession._maybe_log_progress``) logs — at render time.
+Best-effort by contract: every reader failure degrades to "no activity",
+never to a render error.
+"""
+
+from __future__ import annotations
+
+import threading
+from typing import Any
+
+_lock = threading.Lock()
+# Task name -> live Rollout. Task basenames are unique keys by engine-wide
+# invariant: the whole run pipeline (resume filtering, result keying — see
+# Evaluation's `d.name not in completed` plan filter) is keyed by basename,
+# so each name is scheduled at most once at a time.
+_live: dict[str, Any] = {}
+
+
+def register(task_name: str, rollout: Any) -> None:
+    """Expose a live rollout to the dashboard under its task name."""
+    with _lock:
+        _live[task_name] = rollout
+
+
+def unregister(task_name: str) -> None:
+    with _lock:
+        _live.pop(task_name, None)
+
+
+def activity(task_name: str) -> tuple[int, str, int | None] | None:
+    """(tool calls, last tool title, total tokens) for a running task.
+
+    None until the task's agent session exists — including non-ACP
+    (session-factory) agents, which never grow an ACP client. The dig into
+    client/session state lives on ``Rollout.activity_snapshot()`` (typed,
+    owner-side); the except here only guards renders racing rollout teardown.
+    """
+    with _lock:
+        rollout = _live.get(task_name)
+    if rollout is None:
+        return None
+    try:
+        return rollout.activity_snapshot()
+    except Exception:
+        # Dashboard reads race a live rollout; degrading to "no activity" is
+        # always preferable to perturbing the render.
+        return None

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/src/benchflow/acp/session.py

@@ -221,6 +221,20 @@ class ACPSession:
             # which is otherwise easy to miss in a 64-concurrency log.
             logger.error(f"ACPSession on_change callback failed: {e}")
 
+    def progress_snapshot(self) -> tuple[int, str]:
+        """(tool-call count, last tool title) — the console heartbeat's
+        counters, also polled by the live eval dashboard's activity cell.
+
+        The title is the raw first line (untruncated — display width belongs
+        to each render site); ``split`` not ``splitlines`` so a whitespace-only
+        title strips to ``""`` instead of raising on an empty line list.
+        """
+        title = ""
+        if self.tool_calls:
+            last = self.tool_calls[-1]
+            title = (last.title or last.kind or "").strip().split("\n", 1)[0]
+        return len(self.tool_calls), title
+
     def _maybe_log_progress(self) -> None:
         if not self._progress_enabled or self._prompt_started_at is None:
             return
@@ -229,12 +243,10 @@ class ACPSession:
             return
         self._last_progress_at = now
         elapsed_min = (now - self._prompt_started_at) / 60.0
-        line = f"  … {elapsed_min:.1f}min, {len(self.tool_calls)} tool calls"
-        if self.tool_calls:
-            last = self.tool_calls[-1]
-            title = (last.title or last.kind or "").strip().splitlines()[0][:60]
-            if title:
-                line += f" (last: {title})"
+        calls, title = self.progress_snapshot()
+        line = f"  … {elapsed_min:.1f}min, {calls} tool calls"
+        if title:
+            line += f" (last: {title[:60]})"
         logger.info(line)
 
     def record_user_prompt(self, text: str) -> None:

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/src/benchflow/cli/_live_progress.py

@@ -27,6 +27,7 @@ from rich.markup import escape
 from rich.table import Table
 from rich.text import Text
 
+from benchflow._utils import live_activity
 from benchflow.cli._shared import console
 from benchflow.usage_tracking import is_trusted_usage_source
 
@@ -127,6 +128,27 @@ def _fmt_tokens(n: int) -> str:
     return str(n)
 
 
+def _activity_cell(name: str) -> str:
+    """Per-task activity for the "running now" table, e.g.
+    ``38 calls · last: file_editor``.
+
+    Polls the live rollout's ACP session counters — the same ones the console
+    heartbeat logs, which the Live display otherwise mutes — via the
+    live-activity registry. Empty until the agent session exists. Tokens appear
+    only once the session has a usage snapshot (i.e. after a completed prompt).
+    """
+    snap = live_activity.activity(name)
+    if snap is None:
+        return ""
+    calls, last_title, tokens = snap
+    cell = f"{calls} calls"
+    if tokens:
+        cell += f" · {_fmt_tokens(tokens)} tok"
+    if last_title:
+        cell += f" · last: {last_title[:30]}"
+    return cell
+
+
 class LiveEvalProgress:
     """A live ``Live``-rendered dashboard, driven by the engine's progress hooks.
 
@@ -295,17 +317,23 @@ class LiveEvalProgress:
             )
             tbl.add_column("running now", no_wrap=True)
             tbl.add_column("elapsed", justify="right")
+            tbl.add_column("activity", no_wrap=True)
             now = time.monotonic()
             for name in sorted(running, key=lambda n: running[n])[:_MAX_RUNNING_ROWS]:
                 # Text() so a task name containing Rich markup (`[` is legal in
                 # SkillsBench dir names) is rendered literally, not parsed as
                 # markup — a MarkupError here escapes __rich__ and aborts the
                 # CLI on live-context exit, violating this module's "a render
-                # bug can never perturb a run" contract.
-                tbl.add_row(Text(name), _fmt_dur(now - running[name]))
+                # bug can never perturb a run" contract. Same for the activity
+                # cell, which carries agent-authored tool titles.
+                tbl.add_row(
+                    Text(name),
+                    _fmt_dur(now - running[name]),
+                    Text(_activity_cell(name), style="dim"),
+                )
             extra = len(running) - _MAX_RUNNING_ROWS
             if extra > 0:
-                tbl.add_row(f"… {extra} more", "")
+                tbl.add_row(f"… {extra} more", "", "")
             parts.append(tbl)
 
         # Footer: pass-rate (excl errors) + token/cost economics. Show "—" (not

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/src/benchflow/evaluation.py

@@ -1250,11 +1250,20 @@ class Evaluation:
 
             return await run_self_gen(rollout_config)
         rollout = await Rollout.create(rollout_config)
-        # Rollout.run() enforces its own host-side hard deadline against
-        # awaits wedged below the phase-level timeouts — see
-        # benchflow.rollout._deadline. A trip surfaces here as a normal
-        # infra-retryable error result.
-        return await rollout.run()
+        # Expose the live rollout to the eval dashboard's activity cell —
+        # a same-process poll of the session's heartbeat counters, see
+        # benchflow._utils.live_activity.
+        from benchflow._utils import live_activity
+
+        live_activity.register(task_dir.name, rollout)
+        try:
+            # Rollout.run() enforces its own host-side hard deadline against
+            # awaits wedged below the phase-level timeouts — see
+            # benchflow.rollout._deadline. A trip surfaces here as a normal
+            # infra-retryable error result.
+            return await rollout.run()
+        finally:
+            live_activity.unregister(task_dir.name)
 
     async def _run_single_task_legacy(
         self, task_dir: Path, cfg: EvaluationConfig

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/src/benchflow/rollout/__init__.py

@@ -755,6 +755,23 @@ class Rollout:
     def acp_client(self) -> Any:
         return self._acp_client
 
+    def activity_snapshot(self) -> tuple[int, str, int | None] | None:
+        """(tool calls, last tool title, total tokens) for the eval
+        dashboard's activity cell, or None before the agent session exists.
+
+        Rollout owns the client/session dig so a rename breaks here — in
+        typed, tested code — instead of silently blanking the dashboard cell
+        (see benchflow._utils.live_activity). Session-factory agents have no
+        ACP client and always return None.
+        """
+        session = self._acp_client.session if self._acp_client else None
+        if session is None:
+            return None
+        calls, last_title = session.progress_snapshot()
+        usage = session.latest_usage_totals()
+        tokens = usage.get("total_tokens") if usage else None
+        return calls, last_title, tokens
+
     @property
     def trajectory(self) -> list[dict]:
         return self._trajectory
@@ -2401,7 +2418,11 @@ class Rollout:
         TrajectoryWriter(
             self._rollout_dir / "trajectory" / "acp_trajectory.jsonl"
         ).write_final(self._trajectory)
-        logger.warning(
+        # info, not warning: trajectory repair is evidence-mutation an auditor
+        # should find at default (non-TTY/CI) verbosity, but as a warning it
+        # survived the live dashboard's WARNING+ replay and printed between
+        # teardown and the score line (dogfood 2026-08-09).
+        logger.info(
             "Repaired %d lossy ACP tool event(s) from trusted provider capture",
             repaired,
         )

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/tests/test_cli_live_progress.py

@@ -92,6 +92,85 @@ def test_trusted_telemetry_accumulates():
     assert d._covered == 2
 
 
+class _FakeSession:
+    def progress_snapshot(self):
+        return 38, "file_editor"
+
+    def latest_usage_totals(self):
+        return {"total_tokens": 1500}
+
+
+def test_activity_cell_polls_live_session_counters():
+    # Per-task activity in the running-now table (fresh-user dogfood
+    # 2026-08-09): the heartbeat's counters must reach the dashboard, since
+    # the Live mutes the logged heartbeat line itself.
+    from benchflow._utils import live_activity
+    from benchflow.cli._live_progress import _activity_cell
+
+    live_activity.register(
+        "edit-pdf", SimpleNamespace(activity_snapshot=lambda: (38, "file_editor", 1500))
+    )
+    try:
+        assert _activity_cell("edit-pdf") == "38 calls · 1.5k tok · last: file_editor"
+    finally:
+        live_activity.unregister("edit-pdf")
+    # Unregistered tasks (and pre-session rollouts) degrade to empty, not raise.
+    assert _activity_cell("edit-pdf") == ""
+
+
+def test_rollout_activity_snapshot_reads_acp_session():
+    # The client/session dig lives on Rollout (typed, owner-side) so a rename
+    # of session counters breaks HERE instead of silently blanking the cell.
+    from benchflow.rollout import Rollout
+
+    connected = SimpleNamespace(_acp_client=SimpleNamespace(session=_FakeSession()))
+    assert Rollout.activity_snapshot(connected) == (38, "file_editor", 1500)
+    # Pre-connect (and session-factory) rollouts have no client -> None.
+    assert Rollout.activity_snapshot(SimpleNamespace(_acp_client=None)) is None
+
+
+def test_dashboard_renders_activity_for_registered_running_task():
+    # End-to-end through __rich__: a registered running task's activity must
+    # appear in the rendered panel — reverting the table wiring fails this.
+    import io
+
+    from benchflow._utils import live_activity
+
+    live_activity.register(
+        "edit-pdf", SimpleNamespace(activity_snapshot=lambda: (38, "file_editor", None))
+    )
+    try:
+        d = _dash()
+        d.on_plan(total=1, done=0, remaining=1)
+        d.on_task_start("edit-pdf")
+        out = Console(file=io.StringIO(), width=120)
+        out.print(d.__rich__())
+        text = out.file.getvalue()
+        assert "38 calls" in text
+        assert "file_editor" in text
+    finally:
+        live_activity.unregister("edit-pdf")
+
+
+def test_activity_cell_empty_before_session_exists():
+    from benchflow._utils import live_activity
+    from benchflow.cli._live_progress import _activity_cell
+
+    # Rollout registered but not yet connected (no ACP client / session):
+    # the cell stays empty and the render is unperturbed.
+    live_activity.register(
+        "warming-up", SimpleNamespace(activity_snapshot=lambda: None)
+    )
+    try:
+        assert _activity_cell("warming-up") == ""
+        d = _dash()
+        d.on_plan(total=1, done=0, remaining=1)
+        d.on_task_start("warming-up")
+        d.__rich__()  # builds the running-now table row; must not raise
+    finally:
+        live_activity.unregister("warming-up")
+
+
 def test_progress_enabled_respects_tty_and_optout(monkeypatch):
     tty = SimpleNamespace(is_terminal=True)
     notty = SimpleNamespace(is_terminal=False)

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/tests/test_console_progress.py

@@ -75,6 +75,28 @@ def test_heartbeat_silent_when_disabled(monkeypatch, caplog):
     assert not [r for r in caplog.records if "tool calls" in r.message]
 
 
+def test_progress_snapshot_exposes_heartbeat_counters(monkeypatch):
+    """The dashboard's activity cell polls the same counters the line logs."""
+    monkeypatch.setenv("BENCHFLOW_PROGRESS", "off")
+    s = ACPSession("sid")
+    assert s.progress_snapshot() == (0, "")
+    _drive_tool_call(s, "tc1")
+    assert s.progress_snapshot() == (1, "IPython cell")
+
+
+def test_progress_snapshot_whitespace_only_title(monkeypatch):
+    """A whitespace-only title strips to "" instead of raising IndexError
+
+    (truthy title -> strip() -> "" -> splitlines() == [] under the old code).
+    """
+    monkeypatch.setenv("BENCHFLOW_PROGRESS", "off")
+    s = ACPSession("sid")
+    s.handle_update(
+        {"sessionUpdate": "tool_call", "toolCallId": "tc1", "title": "   ", "kind": ""}
+    )
+    assert s.progress_snapshot() == (1, "")
+
+
 def test_heartbeat_silent_outside_prompt(monkeypatch, caplog):
     """No heartbeat before the first prompt or after mark_prompt_end."""
     monkeypatch.setenv("BENCHFLOW_PROGRESS", "on")

{benchflow-0.6.7.dev1823 → benchflow-0.6.7.dev1827}/tests/test_task_download.py

@@ -51,6 +51,7 @@ def test_skillsbench_alias_clones_main_branch(tmp_path, monkeypatch):
         [
             "git",
             "clone",
+            "--quiet",
             "--depth",
             "1",
             "--branch",
@@ -733,13 +734,24 @@ def test_resolve_source_with_sha_ref_fetches_after_clone(tmp_path, monkeypatch):
         [
             "git",
             "clone",
+            "--quiet",
             "--depth",
             "1",
             "https://github.com/org/repo.git",
             str(clone_tmp),
         ],
-        ["git", "-C", str(clone_tmp), "fetch", "--depth", "1", "origin", sha_ref],
-        ["git", "-C", str(clone_tmp), "checkout", "--detach", "FETCH_HEAD"],
+        [
+            "git",
+            "-C",
+            str(clone_tmp),
+            "fetch",
+            "--quiet",
+            "--depth",
+            "1",
+            "origin",
+            sha_ref,
+        ],
+        ["git", "-C", str(clone_tmp), "checkout", "--quiet", "--detach", "FETCH_HEAD"],
     ]
 
 
@@ -760,8 +772,8 @@ def test_resolve_source_with_ref_refreshes_cached_checkout(tmp_path, monkeypatch
     resolve_source("org/repo", path="tasks", ref="main")
 
     assert calls == [
-        ["git", "-C", str(cache), "fetch", "--depth", "1", "origin", "main"],
-        ["git", "-C", str(cache), "checkout", "--detach", "FETCH_HEAD"],
+        ["git", "-C", str(cache), "fetch", "--quiet", "--depth", "1", "origin", "main"],
+        ["git", "-C", str(cache), "checkout", "--quiet", "--detach", "FETCH_HEAD"],
     ]