benchflow 0.7.2.dev1998__tar.gz → 0.7.3__tar.gz

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 ## [Unreleased]
 
+## 0.7.3 — 2026-08-16
+
+### Added
+- **`bench traj upload` waits out short rate-limit responses instead of
+  failing.** When the contribution service answers 429 with a short
+  `Retry-After`, the handshake now sleeps it out with jitter and retries up
+  to three times (two-minute cap per wait), so a crowd of simultaneous
+  contributors self-heals instead of surfacing errors. Longer waits still
+  fail fast with the actionable retry-after message. (#1027)
+
+### Changed
+- **The contribution service rate-limits per contributor, not per venue.**
+  Upload budgets are token buckets keyed on contributor identity with a
+  wide per-IP abuse backstop, refill continuously, and answer 429 with
+  seconds-until-next-token instead of the remainder of the clock hour, so
+  many contributors behind one NAT no longer starve each other. Contended
+  bucket updates back off with jitter rather than shedding simultaneous
+  crowds. (#1027, #1028)
+
+## 0.7.2 — 2026-08-16
+
 ### Added
 - **The upload preview itemizes what redaction masked, by kind.** The
   redactor now categorizes every replacement by the rule that fired — API
@@ -34,13 +55,42 @@
 - **Trajectory uploads are tagged with the session's repository by
   default.** Unless `--source-id` is given, `bench traj upload` reads the
   session's recorded working directory (Claude `cwd` events, Codex
-  `session_meta`), resolves its git `origin` remote (invocation directory as
-  fallback), and stores `repo/<owner>/<name>` as the manifest source id,
-  printing `Repo: owner/name (use --no-repo to omit)`. `--no-repo` opts out
+  `session_meta`), resolves its git `origin` remote, and stores
+  `repo/<owner>/<name>` as the manifest source id, printing
+  `Repo: owner/name (from session cwd /path; use --no-repo to omit)` (the
+  local path is terminal output only, never uploaded). `--no-repo` opts out
   — the `benchflow-traj-upload` skill now surfaces the detected tag during
   the confirm step so contributors can decline it for private repos — and
   undetectable repos fall back silently to the path-derived source id.
 
+### Fixed
+- **The repo tag derives only from the session's own recorded cwd.** The
+  initial repo-tagging implementation (#1015) fell back to the upload
+  invocation directory's git remote when the session cwd yielded nothing; a
+  collector-side audit showed this mis-attributes provenance — a session
+  recorded in a non-repo directory, uploaded from the benchflow checkout,
+  was tagged `repo/benchflow-ai/benchflow` (two community-dataset entries
+  carry the mis-tag). The fallback is removed: no session cwd, a missing
+  directory, or no GitHub remote now mean no repo tag, exactly like
+  `--no-repo`.
+- **The trajectory viewer header no longer shows `?` badges on real Claude
+  Code sessions.** `bench eval view` on a `~/.claude` session JSONL rendered
+  `model: ?`, `session: ?...`, `claude code: ?`, and `total cost: $0.0000`
+  because the header only read a `type: system` event that real session
+  files don't contain. The header now derives its metadata from what the
+  file actually carries (first assistant event's `message.model`, per-event
+  `version` / `sessionId`, the filename stem as a session fallback) and
+  hides any badge whose value is unknown — including the cost badge when no
+  event carries cost data. Presentation only.
+- **Upload progress no longer claims the broker is waking up.** A warm
+  retry printed `Uploading… the first request can take a minute while the
+  service wakes up` even when the service was already up. The line is now
+  `Uploading… this can take up to a minute; retries are safe.`
+- **`bench traj setup --list` no longer wraps session paths mid-token.**
+  Each hit prints index/source/time, then the path on its own line, then
+  the snippet, via plain `print` so Rich does not split a long JSONL path
+  and break copy-paste.
+
 ### Changed
 - **Trajectory viewer tool calls are color-coded and backgrounds are light.**
   Each tool kind now gets a muted GitHub-label-style accent on its name pill

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.2.dev1998
+Version: 0.7.3
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.2.dev1998"
+version = "0.7.3"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/src/benchflow/cli/traj.py

@@ -292,17 +292,23 @@ def register_traj(app: typer.Typer) -> None:
 def _run_upload(options: _UploadOptions) -> None:
     prompted = options.path is None
     path = options.path or _prompt_for_path()
-    repo_slug: str | None = None
+    detected: _DetectedRepo | None = None
     if options.source_id is not None:
         source_id = options.source_id
     else:
         if options.repo:
-            repo_slug = _detect_repo_slug(path)
-        source_id = f"repo/{repo_slug}" if repo_slug else default_source_id(path)
-    if repo_slug:
-        # Contributor-visible metadata: surface the tag so private-repo
-        # sessions can opt out before anything leaves the machine.
-        console.print(f"Repo: {repo_slug} (use --no-repo to omit)")
+            detected = _detect_repo_slug(path)
+        source_id = f"repo/{detected.slug}" if detected else default_source_id(path)
+    if detected:
+        # Contributor-visible metadata: surface the tag and where it came
+        # from so private-repo sessions can opt out before anything leaves
+        # the machine. The local path stays terminal-only; the uploaded
+        # source id is just repo/<owner>/<name>. Plain print keeps the line
+        # one physical line (no Rich wrapping) so the path stays selectable.
+        print(
+            f"Repo: {detected.slug} "
+            f"(from session cwd {detected.session_cwd}; use --no-repo to omit)"
+        )
     destination = _resolve_destination(options)
 
     with (
@@ -345,10 +351,9 @@ def _run_upload(options: _UploadOptions) -> None:
             console.print("[yellow]Upload cancelled.[/yellow]")
             return
         if not destination.direct:
-            console.print(
-                "Uploading… the first request can take a minute "
-                "while the service wakes up."
-            )
+            # Honest on both cold and warm runs: the broker may already be
+            # up. Don't claim it's waking when a retry is just transferring.
+            console.print("Uploading… this can take up to a minute; retries are safe.")
         with upload_progress(staged.files, console=console) as hooks:
             result = _publish(staged, destination=destination, hooks=hooks)
         _print_upload_result(staged, result, direct=destination.direct)
@@ -419,25 +424,35 @@ def _infer_email() -> str:
 _SESSION_CWD_SCAN_LINES = 50
 
 
-def _detect_repo_slug(path: Path) -> str | None:
+@dataclass(frozen=True)
+class _DetectedRepo:
+    slug: str
+    session_cwd: Path
+
+
+def _detect_repo_slug(path: Path) -> _DetectedRepo | None:
     """Best-effort ``owner/name`` for the repository the session was about.
 
     Reads the working directory the session recorded (Claude events carry a
-    ``cwd`` field; Codex ``session_meta`` payloads do too), asks that
-    directory's git for the ``origin`` remote, and falls back to the
-    invocation directory. Every failure is silent — repo tagging must never
-    break an upload — and local-path remotes never produce a tag, so no
-    local absolute path can leak into the manifest.
+    ``cwd`` field; Codex ``session_meta`` payloads do too) and asks that
+    directory's git for the ``origin`` remote. The trajectory's own recorded
+    cwd is the ONLY provenance source: there is deliberately no fallback to
+    the upload invocation directory, which mis-attributed sessions recorded
+    outside a repo to whatever checkout the contributor happened to upload
+    from. No session cwd, a missing directory, or no GitHub remote all mean
+    no tag. Every failure is silent — repo tagging must never break an
+    upload — and local-path remotes never produce a tag, so no local
+    absolute path can leak into the manifest.
     """
-    for candidate in (_session_cwd(path), Path.cwd()):
-        if candidate is None or not candidate.is_dir():
-            continue
-        remote = _command_stdout(
-            "git", "-C", str(candidate), "remote", "get-url", "origin"
-        )
-        slug = _repo_slug_from_remote(remote) if remote else None
-        if slug:
-            return slug
+    session_cwd = _session_cwd(path)
+    if session_cwd is None or not session_cwd.is_dir():
+        return None
+    remote = _command_stdout(
+        "git", "-C", str(session_cwd), "remote", "get-url", "origin"
+    )
+    slug = _repo_slug_from_remote(remote) if remote else None
+    if slug:
+        return _DetectedRepo(slug=slug, session_cwd=session_cwd)
     return None
 
 
@@ -701,11 +716,15 @@ def _print_session_hits() -> None:
 
     hits = list_recent_sessions()
     if not hits:
-        console.print("No recent Claude Code, Codex, or trial sessions found.")
+        print("No recent Claude Code, Codex, or trial sessions found.")
         return
     for index, hit in enumerate(hits, start=1):
         snippet = hit.snippet or "(no prompt yet)"
-        console.print(f"{index}. [{hit.source}] {hit.when}  {hit.path}\n   {snippet}")
+        # Plain print + path on its own line: Rich wrapping used to split
+        # long session paths mid-token and make them unselectable.
+        print(f"{index}. [{hit.source}] {hit.when}")
+        print(f"   {hit.path}")
+        print(f"   {snippet}")
 
 
 def _interactive_view() -> None:

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/src/benchflow/publish/broker.py

@@ -3,6 +3,8 @@
 from __future__ import annotations
 
 import logging
+import random
+import time
 from collections.abc import Callable
 from contextlib import contextmanager, nullcontext
 from dataclasses import dataclass
@@ -18,6 +20,11 @@ from benchflow.publish.traj_capture import StagedCapture, StagedFile
 # the first SAS grant. Retries are safe; keep this under two minutes.
 HANDSHAKE_TIMEOUT_SEC = 90.0
 PUT_TIMEOUT_SEC = 300.0
+# The broker's token buckets answer 429 with an honest, usually-short
+# Retry-After. Waiting it out inline turns a crowd burst into a smooth queue;
+# anything longer than this cap is surfaced to the contributor instead.
+HANDSHAKE_RETRY_LIMIT = 3
+HANDSHAKE_RETRY_MAX_WAIT_SEC = 120.0
 
 
 @dataclass(frozen=True)
@@ -57,9 +64,7 @@ def upload_capture_via_broker(
     manager = nullcontext(http_client) if http_client is not None else httpx.Client()
     try:
         with _quiet_httpx_request_logging(), manager as client:
-            response = client.post(
-                endpoint, json=request_body, timeout=HANDSHAKE_TIMEOUT_SEC
-            )
+            response = _post_handshake(client, endpoint, request_body)
             if response.status_code == 409:
                 return _already_uploaded_result(response, staged, broker_url)
             _raise_for_broker_response(response, operation="upload handshake")
@@ -107,6 +112,35 @@ def upload_capture_via_broker(
     )
 
 
+def _post_handshake(
+    client: httpx.Client, endpoint: str, request_body: dict[str, Any]
+) -> httpx.Response:
+    """POST the handshake, waiting out short broker 429s instead of failing."""
+    response = client.post(endpoint, json=request_body, timeout=HANDSHAKE_TIMEOUT_SEC)
+    for _retry in range(HANDSHAKE_RETRY_LIMIT):
+        if response.status_code != 429:
+            return response
+        retry_after = _retry_after_seconds(response)
+        if retry_after is None or retry_after > HANDSHAKE_RETRY_MAX_WAIT_SEC:
+            return response
+        time.sleep(retry_after + random.uniform(0.0, 1.0))
+        response = client.post(
+            endpoint, json=request_body, timeout=HANDSHAKE_TIMEOUT_SEC
+        )
+    return response
+
+
+def _retry_after_seconds(response: httpx.Response) -> float | None:
+    value = response.headers.get("Retry-After")
+    if value is None:
+        return None
+    try:
+        seconds = float(value.strip())
+    except ValueError:
+        return None
+    return seconds if seconds >= 0 else None
+
+
 def _is_create_only_conflict(response: httpx.Response) -> bool:
     """Treat create-only blob conflicts as an idempotent skip.
 

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/src/benchflow/trajectories/viewer.py

@@ -431,26 +431,18 @@ def render_rollout(rollout_dir: Path, prompts: list[str] | None = None) -> str:
     while len(prompts) < len(turn_files):
         prompts.append("")
 
-    first_events = _parse_jsonl(turn_files[0].read_text())
-    sys_event = next((e for e in first_events if e.get("type") == "system"), {})
-    total_cost = 0
-    total_turns_count = 0
-
+    all_events: list[dict] = []
     all_blocks = []
     for i, tf in enumerate(turn_files):
         events = _parse_jsonl(tf.read_text())
+        all_events.extend(events)
         all_blocks.append(render_turn(events, i + 1, prompts[i]))
-        for e in events:
-            if e.get("type") == "result":
-                total_cost += e.get("total_cost_usd", 0)
-                total_turns_count += e.get("num_turns", 0)
-
-    # `or "?"` (not just a .get default): a present-but-null value in the
-    # stream-json (e.g. "session_id": null) bypasses the default and would crash
-    # html.escape() / the [:16] slice below with a raw TypeError.
-    session_id = str(sys_event.get("session_id") or "?")
-    model = str(sys_event.get("model") or "?")
-    version = str(sys_event.get("claude_code_version") or "?")
+
+    badges = _stream_header_badges(all_events)
+    badges.append(("turns", str(len(turn_files))))
+    total_cost = _result_cost(all_events)
+    if total_cost is not None:
+        badges.append(("total cost", f"${total_cost:.4f}"))
 
     return f"""<!DOCTYPE html>
 <html>
@@ -464,14 +456,7 @@ def render_rollout(rollout_dir: Path, prompts: list[str] | None = None) -> str:
 <div class="header">
 {_WORDMARK_HTML}
 <h1>{html.escape(rollout_dir.name)}</h1>
-<div class="meta">
-<span>model: {html.escape(model)}</span>
-<span>session: {html.escape(session_id[:16])}...</span>
-<span>claude code: {html.escape(version)}</span>
-<span>turns: {len(turn_files)}</span>
-<span>total cost: ${total_cost:.4f}</span>
-</div>
-</div>
+{_meta_badges_html(badges)}</div>
 {_join_with_divider(all_blocks)}
 </body>
 </html>"""
@@ -672,18 +657,89 @@ def render_jsonl_file(path: Path) -> str:
     body = render_turn(events, 1, "")
     if not body.strip():
         return _NO_TRAJECTORIES_HTML
-    return _stream_json_page(path.name, events, [body])
+    return _stream_json_page(path.name, events, [body], session_fallback=path.stem)
+
+
+def _stream_header_badges(
+    events: list[dict], *, session_fallback: str | None = None
+) -> list[tuple[str, str]]:
+    """Header badges derivable from what the stream actually contains.
+
+    ``claude -p`` stream-json carries a ``type: system`` init event
+    (``session_id`` / ``model`` / ``claude_code_version``); real ``~/.claude``
+    session files don't — their metadata lives per event (``sessionId``,
+    ``version``) and on assistant events (``message.model``). Pull from
+    whichever is present and omit anything unknown: a header of ``?`` badges
+    at the approve moment reads as a broken viewer, not as missing data.
+    Truthiness (not ``.get`` defaults) also swallows present-but-null values
+    like ``"session_id": null``, which previously needed an explicit guard to
+    avoid a TypeError.
+    """
+    sys_event = next((e for e in events if e.get("type") == "system"), {})
 
+    def _first(values) -> object | None:
+        return next((value for value in values if value), None)
 
-def _stream_json_page(title: str, events: list[dict], turn_blocks: list[str]) -> str:
-    sys_event = next((e for e in events if e.get("type") == "system"), {})
-    total_cost = 0.0
+    model = sys_event.get("model") or _first(
+        event["message"].get("model")
+        for event in events
+        if event.get("type") == "assistant" and isinstance(event.get("message"), dict)
+    )
+    session_id = (
+        sys_event.get("session_id")
+        or _first(event.get("sessionId") for event in events)
+        or session_fallback
+    )
+    version = sys_event.get("claude_code_version") or _first(
+        event.get("version") for event in events
+    )
+
+    badges: list[tuple[str, str]] = []
+    if model:
+        badges.append(("model", str(model)))
+    if session_id:
+        sid = str(session_id)
+        badges.append(("session", sid[:16] + ("..." if len(sid) > 16 else "")))
+    if version:
+        badges.append(("claude code", str(version)))
+    return badges
+
+
+def _result_cost(events: list[dict]) -> float | None:
+    """Total cost summed from result events, or ``None`` when no event
+    carries cost data — so the header can hide the badge instead of
+    asserting a fictional ``$0.0000``."""
+    total = 0.0
+    seen = False
     for event in events:
-        if event.get("type") == "result":
-            total_cost += float(event.get("total_cost_usd") or 0)
-    session_id = str(sys_event.get("session_id") or "?")
-    model = str(sys_event.get("model") or "?")
-    version = str(sys_event.get("claude_code_version") or "?")
+        if event.get("type") == "result" and event.get("total_cost_usd") is not None:
+            total += float(event.get("total_cost_usd") or 0)
+            seen = True
+    return total if seen else None
+
+
+def _meta_badges_html(badges: list[tuple[str, str]]) -> str:
+    """The header's ``.meta`` badge row; empty string when nothing is known."""
+    if not badges:
+        return ""
+    spans = "\n".join(
+        f"<span>{html.escape(label)}: {html.escape(value)}</span>"
+        for label, value in badges
+    )
+    return f'<div class="meta">\n{spans}\n</div>\n'
+
+
+def _stream_json_page(
+    title: str,
+    events: list[dict],
+    turn_blocks: list[str],
+    *,
+    session_fallback: str | None = None,
+) -> str:
+    badges = _stream_header_badges(events, session_fallback=session_fallback)
+    total_cost = _result_cost(events)
+    if total_cost is not None:
+        badges.append(("total cost", f"${total_cost:.4f}"))
     return f"""<!DOCTYPE html>
 <html>
 <head>
@@ -696,13 +752,7 @@ def _stream_json_page(title: str, events: list[dict], turn_blocks: list[str]) ->
 <div class="header">
 {_WORDMARK_HTML}
 <h1>{html.escape(title)}</h1>
-<div class="meta">
-<span>model: {html.escape(model)}</span>
-<span>session: {html.escape(session_id[:16])}...</span>
-<span>claude code: {html.escape(version)}</span>
-<span>total cost: ${total_cost:.4f}</span>
-</div>
-</div>
+{_meta_badges_html(badges)}</div>
 {_join_with_divider(turn_blocks)}
 </body>
 </html>"""

{benchflow-0.7.2.dev1998 → benchflow-0.7.3}/tests/test_cli_edge_case_hardening.py

@@ -732,6 +732,66 @@ def test_render_jsonl_file_claude_and_codex_sessions(tmp_path):
     assert "looking now" in codex_html
 
 
+def test_viewer_header_reads_real_session_metadata_and_hides_unknown_badges(
+    tmp_path,
+):
+    """Guards the viewer-header fix from the collector-side audit (follow-up
+    to PR #1021's approve flow): real ``~/.claude`` session JSONL carries no
+    ``type: system`` event, so the header rendered ``model: ?``,
+    ``session: ?...``, ``claude code: ?``, and ``total cost: $0.0000`` right
+    at the approve moment. The header must derive model from the first
+    assistant event's ``message.model``, session/version from the per-event
+    ``sessionId``/``version`` fields, and hide any badge — including the cost
+    badge — whose value the file does not contain."""
+    from benchflow.trajectories.viewer import render_jsonl_file
+
+    session = tmp_path / "real-session.jsonl"
+    session.write_text(
+        json.dumps(
+            {
+                "type": "user",
+                "sessionId": "f684cfbe-1051-4d94-92ad-f8b558dd98cd",
+                "version": "2.1.233",
+                "cwd": "/private/tmp/bio-dogfood",
+                "message": {"role": "user", "content": "run the analysis"},
+            }
+        )
+        + "\n"
+        + json.dumps(
+            {
+                "type": "assistant",
+                "sessionId": "f684cfbe-1051-4d94-92ad-f8b558dd98cd",
+                "version": "2.1.233",
+                "message": {
+                    "role": "assistant",
+                    "model": "claude-opus-5",
+                    "content": [{"type": "text", "text": "on it"}],
+                },
+            }
+        )
+        + "\n"
+    )
+    html_out = render_jsonl_file(session)
+    assert "model: claude-opus-5" in html_out
+    assert "session: f684cfbe-1051-4d..." in html_out
+    assert "claude code: 2.1.233" in html_out
+    assert ": ?" not in html_out  # no unknown badge ever renders a "?"
+    assert "total cost" not in html_out  # no cost data → no $0.0000 badge
+
+    # A session with no metadata at all: only the filename-stem session badge
+    # remains; nothing renders as "?" or as a fictional zero cost.
+    bare = tmp_path / "bare.jsonl"
+    bare.write_text(
+        json.dumps({"type": "user", "message": {"content": "review me"}}) + "\n"
+    )
+    bare_html = render_jsonl_file(bare)
+    assert "session: bare" in bare_html
+    assert "model:" not in bare_html
+    assert "claude code:" not in bare_html
+    assert "total cost" not in bare_html
+    assert ": ?" not in bare_html
+
+
 def test_serve_jsonl_does_not_write_sidecar(tmp_path, monkeypatch):
     """Guards skill-first traj review: viewing ~/.claude sessions must not write HTML beside them."""
     from benchflow.trajectories.viewer import serve