benchflow 0.7.2.dev2000__tar.gz → 0.7.4__tar.gz

{benchflow-0.7.2.dev2000 → benchflow-0.7.4}/CHANGELOG.md

@@ -2,6 +2,81 @@
 
 ## [Unreleased]
 
+## 0.7.4 — 2026-08-16
+
+### Added
+- **Uploads are confirmed all the way into cloud storage.** After the
+  progress bar finishes, `bench traj upload` now polls the contribution
+  service's new `GET /v1/uploads/{digest}` capture-status endpoint (the
+  validation ledger) until the validator's verdict: `✓ Verified in cloud
+  storage` once the capture is promoted to `sources/community/<digest>/`, a
+  concise exit-1 error with the fixable detail if the validator rejects it,
+  and a `bench traj status sha256:<digest>` handoff line if validation is
+  still running when the budget (default 240 s, `BENCHFLOW_TRAJ_WAIT_SECONDS`
+  override, `--no-wait` opt-out) runs out. A handshake 409 ("already
+  submitted") prints the verified line immediately, and a deployed broker
+  that predates the endpoint (404) keeps today's behavior unchanged. The new
+  `bench traj status DIGEST` command runs one check on demand. Status polls
+  consume a separate, higher rate-limit budget (`TRAJ_STATUS_RATE_LIMIT`,
+  default 720/hour/IP) and reveal only the ledger state, the bounded
+  rejection detail, and the public promotion prefix — never contributor
+  identity or quarantine internals. The broker must be redeployed
+  (`deploy-trajectory-upload` workflow or `scripts/deploy.sh`) before the
+  endpoint answers in production; the CLI degrades gracefully until then.
+- **The `bench traj` family shares one polished terminal design language.**
+  A new presentation-only kit (`cli/_traj_tui.py`) gives `traj setup`,
+  `traj upload`, and `traj status` a coherent look: a `◆ benchflow · <command>`
+  banner, styled `◇` input prompts, an arrow-key recent-session picker (↑/↓,
+  1-9 jump, esc to fall back to typing a path) on real terminals, colored
+  step kinds in the report preview matching the browser viewer's palette,
+  and rounded panels. Every interactive affordance degrades to the exact
+  previous prompt-driven flow off-TTY (agents, pipes, CI, Windows), and all
+  machine-read lines (`Masked for you:`, `Digest:`, `Repo:`) stay plain.
+- **Uploads can carry the session's workspace folder as a zip attachment.**
+  `bench traj upload` reads the session's recorded working directory (the
+  same Claude `cwd` / Codex `session_meta` provenance as repo tagging) and
+  archives it into the capture as `workspace/<name>.zip`, printing
+  `Workspace: <path> (from session cwd; use --no-workspace to omit)` and a
+  `Workspace attached:` line with size, file count, and exclusion count.
+  VCS internals, dependency trees, caches, symlinks, and secret-shaped
+  filenames (`.env*`, `*.pem`, `*.key`, `id_rsa*`, `.netrc`, …) never enter
+  the archive; everything else is archived as-is without content redaction,
+  and the attach line says so. Workspaces over 1 GiB (measured before
+  compression, so the zip is never created), over 50,000 files, missing, or
+  empty are skipped with a printed reason instead of failing the upload.
+  When detection fails on a real terminal, one optional prompt accepts a
+  folder or skips on Enter; `--workspace-dir` overrides detection and
+  `--no-workspace` opts out. The archive is staged in the upload's
+  temporary directory and always deleted afterwards. Server side, the
+  contribution service accepts the new `workspace/*.zip` namespace with a
+  1 GiB per-archive cap (trajectory JSONL keeps 128 MiB), allows at most
+  one archive per capture and never an archive alone, verifies the zip
+  container format instead of JSONL strictness, promotes it with an
+  `application/zip` content type, and scopes trajectory-report
+  cross-checks to trajectory artifacts so an attachment cannot fail
+  report equality.
+
+## 0.7.3 — 2026-08-16
+
+### Added
+- **`bench traj upload` waits out short rate-limit responses instead of
+  failing.** When the contribution service answers 429 with a short
+  `Retry-After`, the handshake now sleeps it out with jitter and retries up
+  to three times (two-minute cap per wait), so a crowd of simultaneous
+  contributors self-heals instead of surfacing errors. Longer waits still
+  fail fast with the actionable retry-after message. (#1027)
+
+### Changed
+- **The contribution service rate-limits per contributor, not per venue.**
+  Upload budgets are token buckets keyed on contributor identity with a
+  wide per-IP abuse backstop, refill continuously, and answer 429 with
+  seconds-until-next-token instead of the remainder of the clock hour, so
+  many contributors behind one NAT no longer starve each other. Contended
+  bucket updates back off with jitter rather than shedding simultaneous
+  crowds. (#1027, #1028)
+
+## 0.7.2 — 2026-08-16
+
 ### Added
 - **The upload preview itemizes what redaction masked, by kind.** The
   redactor now categorizes every replacement by the rule that fired — API
@@ -61,6 +136,14 @@
   `version` / `sessionId`, the filename stem as a session fallback) and
   hides any badge whose value is unknown — including the cost badge when no
   event carries cost data. Presentation only.
+- **Upload progress no longer claims the broker is waking up.** A warm
+  retry printed `Uploading… the first request can take a minute while the
+  service wakes up` even when the service was already up. The line is now
+  `Uploading… this can take up to a minute; retries are safe.`
+- **`bench traj setup --list` no longer wraps session paths mid-token.**
+  Each hit prints index/source/time, then the path on its own line, then
+  the snippet, via plain `print` so Rich does not split a long JSONL path
+  and break copy-paste.
 
 ### Changed
 - **Trajectory viewer tool calls are color-coded and backgrounds are light.**

{benchflow-0.7.2.dev2000 → benchflow-0.7.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.7.2.dev2000
+Version: 0.7.4
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow

{benchflow-0.7.2.dev2000 → benchflow-0.7.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.7.2.dev2000"
+version = "0.7.4"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"

benchflow-0.7.4/src/benchflow/cli/_traj_tui.py

@@ -0,0 +1,286 @@
+"""Shared terminal design language for the ``bench traj`` command family.
+
+``bench traj setup``, ``bench traj upload``, and ``bench traj status`` render
+through this one small kit — a banner, a glyph vocabulary, styled prompts, and
+an arrow-key list picker — so the family reads as one coherent tool instead of
+three ad-hoc scripts. The look borrows from opencode's TUI: a single accent
+color, dim structural text, `◆`/`◇` step glyphs, and a `❯` selection pointer.
+
+Two hard rules keep this compatible with agents and the upload skill:
+
+1. Machine-read lines stay plain. Copy such as ``Masked for you: …``,
+   ``Digest: sha256:…``, and the ``Repo:`` line are printed by the command
+   modules with ``print`` — never boxed or wrapped here.
+2. Every interactive affordance has a dumb-terminal fallback. The picker
+   degrades to a numbered prompt whenever stdin/stdout is not a TTY (CI,
+   agents piping input, Windows without termios), preserving the exact
+   prompt-driven contract the tests pin.
+"""
+
+from __future__ import annotations
+
+import sys
+from collections.abc import Sequence
+from dataclasses import dataclass
+from pathlib import Path
+
+import typer
+from rich.console import Console, Group
+from rich.rule import Rule
+from rich.text import Text
+
+ACCENT = "cyan"
+OK = "green"
+WARN = "yellow"
+ERR = "red"
+
+GLYPH_STEP = "◆"
+GLYPH_FIELD = "◇"
+# The heavy angle quotation is the intended selection pointer (same glyph
+# opencode and modern shell prompts use); it is not a mistyped ``>``.
+GLYPH_POINTER = "❯"  # noqa: RUF001
+GLYPH_OK = "✓"
+GLYPH_FAIL = "✗"
+
+# Preview/table styling for trajectory step kinds, matching the color language
+# of the browser viewer (PR #1020) so the terminal report and the viewer agree.
+KIND_STYLES = {
+    "Human": "bold green",
+    "Assistant": "cyan",
+    "Thinking": "magenta",
+    "Tool call": "yellow",
+}
+
+_PICKER_HELP = "↑/↓ move · enter select · 1-9 jump · esc cancel"
+# Rows visible at once in the arrow-key picker. The cursor scrolls the list
+# through this viewport (the same feel as Claude Code's session-resume
+# picker), so a week of sessions stays browsable without flooding the screen.
+_PICKER_VIEWPORT_ROWS = 10
+
+
+def _visible_window(index: int, total: int, height: int) -> tuple[int, int]:
+    """First/last-plus-one visible option, keeping the cursor centered."""
+    if total <= height:
+        return 0, total
+    start = max(0, min(index - height // 2, total - height))
+    return start, start + height
+
+
+def banner(console: Console, command: str, subtitle: str = "") -> None:
+    """One-line product banner: ``◆ benchflow · <command>`` plus a dim subtitle."""
+    console.print(
+        f"[bold {ACCENT}]{GLYPH_STEP} benchflow[/] [dim]·[/] [bold]{command}[/]"
+    )
+    if subtitle:
+        console.print(f"  [dim]{subtitle}[/]")
+    console.print()
+
+
+def section(console: Console, title: str) -> None:
+    """A dim rule that separates flow phases (report / upload / verify)."""
+    console.print(Rule(Text(title, style=f"bold {ACCENT}"), style="dim", align="left"))
+
+
+def field_label(label: str) -> str:
+    """Style an input prompt label; the trailing ``:`` stays typer's."""
+    return typer.style(f"{GLYPH_FIELD} ", fg="cyan") + typer.style(label, bold=True)
+
+
+def question_label(label: str) -> str:
+    """Style a yes/no confirmation label."""
+    return typer.style(f"{GLYPH_STEP} ", fg="cyan") + typer.style(label, bold=True)
+
+
+@dataclass(frozen=True)
+class SelectOption:
+    """One row of the picker: a primary label and an optional dim hint."""
+
+    label: str
+    hint: str = ""
+
+
+def interactive_terminal() -> bool:
+    """True when arrow-key interaction is possible on this terminal."""
+    try:
+        if not (sys.stdin.isatty() and sys.stdout.isatty()):
+            return False
+    except (AttributeError, ValueError):
+        return False
+    import os
+
+    if os.environ.get("TERM", "").lower() == "dumb":
+        return False
+    try:
+        import termios  # noqa: F401
+        import tty  # noqa: F401
+    except ImportError:  # Windows: fall back to the numbered prompt
+        return False
+    return True
+
+
+def select(
+    console: Console,
+    *,
+    title: str,
+    options: Sequence[SelectOption],
+    initial: int = 0,
+) -> int | None:
+    """Pick one option; returns its index, or ``None`` when cancelled.
+
+    Arrow-key navigation on a real terminal, a numbered prompt everywhere
+    else. The chosen label is echoed with a ``❯`` so the transcript records
+    the decision after the transient picker disappears.
+    """
+    if not options:
+        return None
+    initial = max(0, min(initial, len(options) - 1))
+    if interactive_terminal():
+        choice = _select_tty(console, title=title, options=options, initial=initial)
+    else:
+        choice = _select_numbered(console, title=title, options=options)
+    if choice is not None:
+        console.print(f"[{ACCENT}]{GLYPH_POINTER}[/] {_escaped(options[choice].label)}")
+    return choice
+
+
+def _escaped(value: str) -> str:
+    from rich.markup import escape
+
+    return escape(value)
+
+
+def _select_tty(
+    console: Console,
+    *,
+    title: str,
+    options: Sequence[SelectOption],
+    initial: int,
+) -> int | None:
+    import termios
+    import tty
+
+    from rich.live import Live
+
+    index = initial
+    # Hints share the row; keep every row one physical line so the Live
+    # region never jitters while navigating.
+    width = console.width or 100
+
+    def row(position: int, option: SelectOption) -> Text:
+        selected = position == index
+        pointer = f" {GLYPH_POINTER} " if selected else "   "
+        text = Text(no_wrap=True, overflow="ellipsis")
+        text.append(pointer, style=ACCENT if selected else "")
+        text.append(option.label, style=f"bold {ACCENT}" if selected else "")
+        if option.hint:
+            text.append("  " + option.hint, style="dim")
+        text.truncate(max(20, width - 2), overflow="ellipsis")
+        return text
+
+    def view() -> Group:
+        lines: list[Text | Rule] = []
+        if title:
+            lines.append(Text(f"{GLYPH_STEP} {title}", style=f"bold {ACCENT}"))
+        start, end = _visible_window(index, len(options), _PICKER_VIEWPORT_ROWS)
+        if start > 0:
+            lines.append(Text(f"   ↑ {start} more", style="dim"))
+        lines.extend(row(position, options[position]) for position in range(start, end))
+        if end < len(options):
+            lines.append(Text(f"   ↓ {len(options) - end} more", style="dim"))
+        lines.append(
+            Text(
+                f"{_PICKER_HELP} · {index + 1}/{len(options)}",
+                style="dim",
+            )
+        )
+        return Group(*lines)
+
+    stream = sys.stdin
+    descriptor = stream.fileno()
+    saved = termios.tcgetattr(descriptor)
+    try:
+        tty.setcbreak(descriptor)
+        with Live(view(), console=console, auto_refresh=False, transient=True) as live:
+            while True:
+                key = _read_key(stream)
+                if key in {"up", "k"}:
+                    index = (index - 1) % len(options)
+                elif key in {"down", "j"}:
+                    index = (index + 1) % len(options)
+                elif key in {"enter"}:
+                    return index
+                elif key in {"esc", "q", "interrupt"}:
+                    return None
+                elif key.isdigit() and 1 <= int(key) <= len(options):
+                    index = int(key) - 1
+                    return index
+                live.update(view(), refresh=True)
+    finally:
+        termios.tcsetattr(descriptor, termios.TCSADRAIN, saved)
+
+
+def _read_key(stream) -> str:
+    """Decode one keypress, folding escape sequences into named keys.
+
+    Reads the raw descriptor, never the buffered text stream: an arrow key
+    arrives as ``ESC [ A`` in one burst, and a buffered ``read(1)`` would
+    swallow ``[ A`` into Python's buffer, making the descriptor look idle and
+    the ESC read as a bare cancel.
+    """
+    import os
+    import select as selectors
+
+    descriptor = stream.fileno()
+    char = os.read(descriptor, 1).decode("utf-8", "replace")
+    if char in {"\r", "\n"}:
+        return "enter"
+    if char == "\x03":
+        return "interrupt"
+    if char != "\x1b":
+        return char
+    # Distinguish a bare Escape from a CSI arrow sequence.
+    if not selectors.select([descriptor], [], [], 0.05)[0]:
+        return "esc"
+    second = os.read(descriptor, 1).decode("utf-8", "replace")
+    if second != "[":
+        return "esc"
+    final = os.read(descriptor, 1).decode("utf-8", "replace")
+    return {"A": "up", "B": "down"}.get(final, "esc")
+
+
+def _select_numbered(
+    console: Console,
+    *,
+    title: str,
+    options: Sequence[SelectOption],
+) -> int | None:
+    if title:
+        console.print(f"[bold {ACCENT}]{GLYPH_STEP} {title}[/]")
+    for position, option in enumerate(options, start=1):
+        console.print(f"[bold {ACCENT}]{position:>2}[/] {_escaped(option.label)}")
+        if option.hint:
+            console.print(f"   [dim]{_escaped(option.hint)}[/]")
+    while True:
+        raw = typer.prompt("Which number?", default="1").strip()
+        if raw.lower() in {"q", "quit", "esc"}:
+            return None
+        try:
+            value = int(raw)
+        except ValueError:
+            console.print(f"[{ERR}]Need a number between 1 and {len(options)}.[/]")
+            continue
+        if 1 <= value <= len(options):
+            return value - 1
+        console.print(f"[{ERR}]Need a number between 1 and {len(options)}.[/]")
+
+
+def compact_path(path: Path, *, limit: int = 64) -> str:
+    """Home-relative, tail-preserving path label for one-line picker rows."""
+    try:
+        text = "~/" + str(path.relative_to(Path.home()))
+    except ValueError:
+        text = str(path)
+    if len(text) <= limit:
+        return text
+    tail = "/".join(Path(text).parts[-2:])
+    return f"…/{tail}" if tail else text

{benchflow-0.7.2.dev2000 → benchflow-0.7.4}/src/benchflow/cli/_traj_upload_ui.py

@@ -6,6 +6,7 @@ from collections.abc import Callable, Iterator
 from contextlib import contextmanager
 from dataclasses import dataclass
 
+from rich import box
 from rich.console import Console
 from rich.panel import Panel
 from rich.progress import (
@@ -20,6 +21,7 @@ from rich.progress import (
 from rich.table import Table
 from rich.text import Text
 
+from benchflow.cli._traj_tui import KIND_STYLES
 from benchflow.publish.redact import REDACTED, format_redaction_breakdown
 from benchflow.publish.traj_capture import StagedFile
 from benchflow.publish.traj_report import PREVIEW_WORD_LIMIT, TrajectoryReport
@@ -52,15 +54,15 @@ def render_trajectory_report(report: TrajectoryReport, *, console: Console) -> N
     facts.add_row("Primary trajectory", Text(report.primary_file))
     facts.add_row("Format", report.format.value)
     facts.add_row("Created", _format_created_at(report))
-    facts.add_row("JSONL files", str(report.file_count))
+    facts.add_row("JSONL files", f"{report.file_count:,}")
     facts.add_row("Trajectory size", format_bytes(report.size_bytes))
-    facts.add_row("Total steps", str(report.total_steps))
-    facts.add_row("Thinking steps", str(report.thinking_steps))
-    facts.add_row("Tool-call steps", str(report.tool_call_steps))
-    facts.add_row("Human steps", str(report.human_steps))
+    facts.add_row("Total steps", f"{report.total_steps:,}")
+    facts.add_row("Thinking steps", _kind_count(report.thinking_steps, "Thinking"))
+    facts.add_row("Tool-call steps", _kind_count(report.tool_call_steps, "Tool call"))
+    facts.add_row("Human steps", _kind_count(report.human_steps, "Human"))
     facts.add_row(
         "API keys / secrets masked",
-        Text(str(report.masked_values), style="bold green"),
+        Text(f"{report.masked_values:,}", style="bold green"),
     )
     if report.masked_values:
         facts.add_row(
@@ -70,7 +72,15 @@ def render_trajectory_report(report: TrajectoryReport, *, console: Console) -> N
     else:
         facts.add_row("Masked for you", Text(NO_MASKING_NEEDED_COPY, style="dim"))
     facts.add_row("Safe replacement", Text(REDACTED))
-    console.print(Panel(facts, title="Trajectory report", border_style="cyan"))
+    console.print(
+        Panel(
+            facts,
+            title="[bold]Trajectory report[/]",
+            border_style="cyan",
+            box=box.ROUNDED,
+            padding=(0, 1),
+        )
+    )
     if report.masked_values:
         console.print(f"[dim]{REDACTION_REASSURANCE_COPY}[/dim]")
 
@@ -81,14 +91,21 @@ def render_trajectory_report(report: TrajectoryReport, *, console: Console) -> N
             f"First {len(report.preview)} trajectory steps "
             f"(up to {PREVIEW_WORD_LIMIT} words each)"
         ),
+        title_style="bold",
         show_lines=True,
         header_style="bold cyan",
+        box=box.ROUNDED,
+        border_style="dim",
     )
     preview.add_column("#", justify="right", style="dim", no_wrap=True)
     preview.add_column("Kind", no_wrap=True)
     preview.add_column("Preview", overflow="fold")
     for step in report.preview:
-        preview.add_row(str(step.number), step.kind, Text(step.summary))
+        preview.add_row(
+            str(step.number),
+            Text(step.kind, style=KIND_STYLES.get(step.kind, "")),
+            Text(step.summary),
+        )
     console.print(preview)
     if len(report.preview) < report.total_steps:
         console.print(
@@ -97,6 +114,11 @@ def render_trajectory_report(report: TrajectoryReport, *, console: Console) -> N
         )
 
 
+def _kind_count(count: int, kind: str) -> Text:
+    """A step count tinted with its preview-kind color when nonzero."""
+    return Text(f"{count:,}", style=KIND_STYLES.get(kind, "") if count else "dim")
+
+
 @dataclass(frozen=True)
 class UploadProgressHooks:
     """Transport callbacks that feed one shared progress bar."""