benchflow 0.6.8.dev1907__tar.gz → 0.6.9.dev1917__tar.gz

{benchflow-0.6.8.dev1907 → benchflow-0.6.9.dev1917}/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ## [Unreleased]
 
+## 0.6.8 — 2026-08-15
+
+### Added
+- **Public trajectory contribution.** `bench traj upload PATH` validates and
+  structurally redacts trajectory JSONL, creates a content-addressed manifest,
+  and uploads through the built-in public broker. Azure Blob quarantine,
+  versioning, short-lived create-only user-delegation SAS grants, and an
+  event-driven fail-closed validator keep untrusted captures out of the
+  community namespace until hashes, sizes, strict JSONL (including duplicate-key
+  and non-finite-number rejection), and artifact/manifest secret scans pass.
+  Replaying an ingested digest is a no-op; trusted operators can opt into direct
+  Azure upload with the `azure` extra and `--direct`.
+
 ### Changed
 - **BREAKING (task.md): the `environment:` frontmatter key is renamed to
   `sandbox:`.** The native task-config surface now accepts only `sandbox:`
@@ -26,6 +39,29 @@
   eval-config `environment:` docker/daytona selector) is a different
   subsystem and is unchanged.
 
+### Fixed
+- **Coherent integration coverage for code-and-fixture changes.** The
+  credential-free fixture job now tests pull-request source together with its
+  task fixtures, while the secret-bearing smoke job retains trusted fixtures;
+  the release gate requires both results. (#969)
+- **Retryable Daytona transport failures stay retryable.** Transient SDK
+  transport errors are stamped while their vendor type is still available,
+  empty exception messages retain useful type information, and permanent
+  errors remain outside the retry policy. (#970)
+- **Live token counters no longer freeze behind callback-log reads.** Larger,
+  bounded ranged reads distinguish EOF from read failure, expose lag, and keep
+  terminal phase labels from moving backwards. (#971)
+- **Fresh OpenClaw installs and GPT-5.4 calls use compatible limits.** OpenClaw
+  is pinned to a Node-compatible runtime, and raw plus ACP-alias GPT-5.4 model
+  IDs clamp output tokens to the provider's 128,000-token limit. (#977, #986)
+- **Anthropic Vertex routes include their required runtime.** The gateway now
+  installs the Google Cloud Vertex AI SDK used by LiteLLM's Anthropic Vertex
+  path. (#978)
+- **ACP subprocess diagnostics survive normal teardown.** A bounded, redacted
+  stderr tail is retained even when stdout remains open; stderr drain failures
+  cannot mask the structured transport diagnosis, and repeated process close
+  calls are idempotent. (#980)
+
 ## 0.6.7 — 2026-08-09
 
 ### Added

{benchflow-0.6.8.dev1907 → benchflow-0.6.9.dev1917}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.6.8.dev1907
+Version: 0.6.9.dev1917
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -29,6 +29,15 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.0
 Requires-Dist: tomli-w>=1.0
 Requires-Dist: typer>=0.9
+Provides-Extra: azure
+Requires-Dist: azure-identity>=1.16; extra == 'azure'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'azure'
+Provides-Extra: azure-service
+Requires-Dist: azure-data-tables>=12.5; extra == 'azure-service'
+Requires-Dist: azure-identity>=1.16; extra == 'azure-service'
+Requires-Dist: azure-storage-blob>=12.19; extra == 'azure-service'
+Requires-Dist: azure-storage-queue>=12.9; extra == 'azure-service'
+Requires-Dist: uvicorn>=0.30; extra == 'azure-service'
 Provides-Extra: bedrock
 Requires-Dist: boto3>=1.40; extra == 'bedrock'
 Provides-Extra: deepagents
@@ -146,6 +155,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
 | Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) |
+| Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) |
 | Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) |
 | Use public vs internal preview SDK releases | [Release channels](./docs/release.md) |
 | CLI flags + commands | [CLI reference](./docs/reference/cli.md) |

{benchflow-0.6.8.dev1907 → benchflow-0.6.9.dev1917}/README.md

@@ -81,6 +81,7 @@ Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/
 | Multi-agent: coder + reviewer, simulated user, BYOS, stateful envs | [Use cases](./docs/use-cases.md) |
 | Multi-round single-agent (progressive disclosure, oracle access) | [Progressive disclosure](./docs/progressive-disclosure.md) |
 | Skill evaluation (when the artifact is a skill, not a workspace) | [Skill eval](./docs/skill-eval.md) |
+| Contribute a trajectory capture | [Trajectory upload](./docs/traj-upload.md) |
 | Understand the security model | [Sandbox hardening](./docs/sandbox-hardening.md) |
 | Use public vs internal preview SDK releases | [Release channels](./docs/release.md) |
 | CLI flags + commands | [CLI reference](./docs/reference/cli.md) |

{benchflow-0.6.8.dev1907 → benchflow-0.6.9.dev1917}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.8.dev1907"
+version = "0.6.9.dev1917"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -40,6 +40,20 @@ classifiers = [
 ]
 
 [project.optional-dependencies]
+azure = [
+    # Direct Azure Blob trajectory uploads (bench traj upload --direct).
+    # Broker-mode uploads need neither package.
+    "azure-storage-blob>=12.19",
+    "azure-identity>=1.16",
+]
+azure-service = [
+    # Dependencies baked into the public trajectory broker/validator image.
+    "azure-storage-blob>=12.19",
+    "azure-identity>=1.16",
+    "azure-data-tables>=12.5",
+    "azure-storage-queue>=12.9",
+    "uvicorn>=0.30",
+]
 dev = [
     "pre-commit>=3.7",
     # AgentCore's CodeBuild path must reproduce Docker's ignore rules in tests

{benchflow-0.6.8.dev1907 → benchflow-0.6.9.dev1917}/src/benchflow/cli/main.py

@@ -57,6 +57,7 @@ from benchflow.cli.sandbox import register_sandbox
 from benchflow.cli.skills import register_skills
 from benchflow.cli.tasks import register_tasks
 from benchflow.cli.train import register_train
+from benchflow.cli.traj import register_traj
 from benchflow.eval_plan import EvalCreateRequest, EvalPlanError, build_eval_plan
 from benchflow.evaluation import DEFAULT_AGENT, effective_model
 from benchflow.sandbox.providers import providers_phrase
@@ -1242,6 +1243,7 @@ register_continue(eval_app, alias_app=app)
 register_skills(app)
 register_review(app)
 register_tasks(app)
+register_traj(app)
 register_train(app)
 register_hub(app)
 register_agent(app)

benchflow-0.6.9.dev1917/src/benchflow/cli/traj.py

@@ -0,0 +1,138 @@
+"""``bench traj upload`` — contribute validated trajectory captures."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Annotated
+
+import typer
+from rich.markup import escape
+
+from benchflow.cli._shared import console, print_error
+
+# The environment variable remains an override for development and disaster
+# recovery.
+DEFAULT_TRAJ_BROKER_URL: str | None = (
+    "https://tasksminer-traj-broker.nicewave-c3abaecf.westus2.azurecontainerapps.io"
+)
+
+
+def register_traj(app: typer.Typer) -> None:
+    """Attach the trajectory contribution group to the top-level app."""
+    traj_app = typer.Typer(help="Trajectory commands.")
+    app.add_typer(traj_app, name="traj", rich_help_panel="Core")
+
+    @traj_app.command("upload")
+    def upload(
+        path: Annotated[
+            Path,
+            typer.Argument(help="Trajectory JSONL file, directory, or trial directory"),
+        ],
+        source_id: Annotated[
+            str | None,
+            typer.Option("--source-id", help="Stable contributor source identifier"),
+        ] = None,
+        direct: Annotated[
+            bool,
+            typer.Option("--direct", help="Upload with local Azure credentials"),
+        ] = False,
+        container_url: Annotated[
+            str | None,
+            typer.Option("--container-url", help="Azure container URL for --direct"),
+        ] = None,
+        dry_run: Annotated[
+            bool,
+            typer.Option("--dry-run", help="Validate and stage without uploading"),
+        ] = False,
+    ) -> None:
+        """Validate, redact, and upload trajectory JSONL."""
+        from benchflow.publish.azure_blob import upload_capture_direct
+        from benchflow.publish.broker import upload_capture_via_broker
+        from benchflow.publish.traj_capture import (
+            default_source_id,
+            stage_trajectory_capture,
+        )
+
+        try:
+            selected_source_id = source_id or default_source_id(path)
+            if direct:
+                destination = container_url or os.environ.get(
+                    "BENCHFLOW_AZURE_CONTAINER_URL"
+                )
+                if not destination:
+                    raise ValueError(
+                        "--direct requires --container-url or "
+                        "BENCHFLOW_AZURE_CONTAINER_URL"
+                    )
+            else:
+                if container_url:
+                    raise ValueError("--container-url is only valid with --direct")
+                destination = (
+                    os.environ.get("BENCHFLOW_TRAJ_BROKER_URL")
+                    or DEFAULT_TRAJ_BROKER_URL
+                )
+                if not destination:
+                    raise ValueError(
+                        "no trajectory broker is configured; set "
+                        "BENCHFLOW_TRAJ_BROKER_URL, or use --direct with "
+                        "--container-url/BENCHFLOW_AZURE_CONTAINER_URL if you have "
+                        "Azure credentials"
+                    )
+
+            with stage_trajectory_capture(
+                path,
+                source_id=selected_source_id,
+                uploaded_by=os.environ.get("BENCHFLOW_TRAJ_UPLOADED_BY"),
+            ) as staged:
+                if dry_run:
+                    _print_dry_run(staged, destination=destination, direct=direct)
+                    return
+                if direct:
+                    result = upload_capture_direct(
+                        staged,
+                        container_url=destination,
+                    )
+                else:
+                    result = upload_capture_via_broker(
+                        staged,
+                        broker_url=destination,
+                    )
+                size = sum(item.size_bytes for item in staged.files)
+                if result.uploaded:
+                    console.print(
+                        "[green]Uploaded trajectory:[/green] "
+                        f"{escape(result.url)} "
+                        f"({len(result.uploaded)} uploaded, {len(result.skipped)} skipped, "
+                        f"{_format_bytes(size)}, "
+                        f"{staged.redaction_replacements} redactions)"
+                    )
+                else:
+                    console.print(
+                        f"[green]Already uploaded:[/green] {escape(result.url)} (no-op)"
+                    )
+        except ValueError as exc:
+            print_error(str(exc))
+            raise typer.Exit(1) from None
+
+
+def _print_dry_run(staged, *, destination: str, direct: bool) -> None:
+    mode = "Azure container" if direct else "broker"
+    console.print(f"[bold]Dry run[/bold] — {mode}: {escape(destination)}")
+    console.print(f"Digest: sha256:{staged.traj_digest}")
+    for staged_file in staged.files:
+        console.print(
+            f"  {escape(staged_file.relname)} ({_format_bytes(staged_file.size_bytes)})"
+        )
+    if staged.ignored:
+        console.print(f"Ignored: {escape(', '.join(staged.ignored))}")
+    console.print(f"Redactions: {staged.redaction_replacements}")
+
+
+def _format_bytes(size: int) -> str:
+    value = float(size)
+    for unit in ("B", "KB", "MB", "GB"):
+        if value < 1024 or unit == "GB":
+            return f"{value:.1f} {unit}"
+        value /= 1024
+    raise AssertionError("unreachable")

benchflow-0.6.9.dev1917/src/benchflow/publish/azure_blob.py

@@ -0,0 +1,142 @@
+"""Direct Azure Blob publishing for staged trajectory captures."""
+
+from __future__ import annotations
+
+import logging
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Any
+from urllib.parse import urlparse
+
+from benchflow import __version__
+from benchflow.publish.traj_capture import StagedCapture
+
+
+@dataclass(frozen=True)
+class BlobPublishResult:
+    account: str
+    container: str
+    prefix: str
+    uploaded: tuple[str, ...]
+    skipped: tuple[str, ...]
+
+    @property
+    def url(self) -> str:
+        return (
+            f"https://{self.account}.blob.core.windows.net/"
+            f"{self.container}/{self.prefix}"
+        )
+
+
+def upload_capture_direct(
+    staged: StagedCapture,
+    *,
+    container_url: str,
+    credential: Any = None,
+) -> BlobPublishResult:
+    """Create staged files in an Azure container without overwriting blobs."""
+    try:
+        from azure.core.exceptions import (
+            ClientAuthenticationError,
+            HttpResponseError,
+            ResourceExistsError,
+            ResourceNotFoundError,
+        )
+        from azure.identity import DefaultAzureCredential
+        from azure.storage.blob import ContainerClient, ContentSettings
+    except ModuleNotFoundError as exc:  # pragma: no cover - environment dependent
+        raise ValueError(
+            "azure-storage-blob and azure-identity are required for --direct; "
+            "install with: pip install 'benchflow[azure]'"
+        ) from exc
+
+    with _quiet_azure_request_logging():
+        account, container = _parse_container_url(container_url)
+        auth = credential if credential is not None else DefaultAzureCredential()
+        client = ContainerClient.from_container_url(container_url, credential=auth)
+        prefix = f"sources/{staged.source_id}/{staged.traj_digest}/"
+        metadata = {
+            "source_id": staged.source_id,
+            "traj_digest": f"sha256:{staged.traj_digest}",
+            "schema_version": str(staged.manifest["schema_version"]),
+            "manifest": "manifest.json",
+            "benchflow_version": __version__,
+        }
+        uploaded: list[str] = []
+        skipped: list[str] = []
+        for staged_file in staged.files:
+            blob_name = prefix + staged_file.relname
+            try:
+                with staged_file.local_path.open("rb") as stream:
+                    client.upload_blob(
+                        name=blob_name,
+                        data=stream,
+                        overwrite=False,
+                        content_settings=ContentSettings(
+                            content_type=staged_file.content_type
+                        ),
+                        metadata=metadata,
+                    )
+            except ResourceExistsError:
+                skipped.append(blob_name)
+            except ResourceNotFoundError as exc:
+                raise ValueError(
+                    f"Azure container not found: {container_url}; create the storage "
+                    "account and private container before using --direct"
+                ) from exc
+            except ClientAuthenticationError as exc:
+                raise ValueError(
+                    "Azure authentication failed; run 'az login' (or configure managed "
+                    "identity) and grant the Blob Data Creator role"
+                ) from exc
+            except HttpResponseError as exc:
+                if getattr(exc, "status_code", None) == 403:
+                    raise ValueError(
+                        "Azure upload was forbidden; run 'az login' (or configure "
+                        "managed identity) and grant the Blob Data Creator role"
+                    ) from exc
+                raise ValueError(
+                    f"Azure Blob upload failed for {blob_name}: {exc}"
+                ) from exc
+            else:
+                uploaded.append(blob_name)
+    return BlobPublishResult(
+        account=account,
+        container=container,
+        prefix=prefix,
+        uploaded=tuple(uploaded),
+        skipped=tuple(skipped),
+    )
+
+
+@contextmanager
+def _quiet_azure_request_logging():
+    """Keep Azure SDK request diagnostics out of the one-line CLI result."""
+    azure_logger = logging.getLogger("azure")
+    previous_level = azure_logger.level
+    azure_logger.setLevel(logging.WARNING)
+    try:
+        yield
+    finally:
+        azure_logger.setLevel(previous_level)
+
+
+def _parse_container_url(container_url: str) -> tuple[str, str]:
+    parsed = urlparse(container_url.rstrip("/"))
+    host_suffix = ".blob.core.windows.net"
+    container = parsed.path.strip("/")
+    if (
+        parsed.scheme != "https"
+        or not parsed.hostname
+        or not parsed.hostname.endswith(host_suffix)
+        or not container
+        or "/" in container
+    ):
+        raise ValueError(
+            "--container-url must look like "
+            "https://<account>.blob.core.windows.net/<container>"
+        )
+    account = parsed.hostname[: -len(host_suffix)]
+    if not account:
+        raise ValueError("--container-url is missing the storage account name")
+    return account, container

benchflow-0.6.9.dev1917/src/benchflow/publish/broker.py

@@ -0,0 +1,193 @@
+"""Cloud-neutral client for the public trajectory upload broker."""
+
+from __future__ import annotations
+
+import logging
+from contextlib import contextmanager, nullcontext
+from dataclasses import dataclass
+from typing import Any
+from urllib.parse import urlparse
+
+import httpx
+
+from benchflow.publish.traj_capture import StagedCapture
+
+
+@dataclass(frozen=True)
+class BrokerPublishResult:
+    base_url: str
+    prefix: str
+    uploaded: tuple[str, ...]
+    skipped: tuple[str, ...]
+
+    @property
+    def url(self) -> str:
+        return f"{self.base_url.rstrip('/')}/{self.prefix.lstrip('/')}"
+
+
+def upload_capture_via_broker(
+    staged: StagedCapture,
+    *,
+    broker_url: str,
+    http_client: httpx.Client | None = None,
+) -> BrokerPublishResult:
+    """Request scoped upload URLs and PUT every staged file in server order."""
+    endpoint = f"{broker_url.rstrip('/')}/v1/uploads"
+    artifacts = staged.manifest["artifacts"]
+    request_body = {
+        "schema_version": staged.manifest["schema_version"],
+        "kind": staged.manifest["kind"],
+        "source_id": staged.manifest["source_id"],
+        "traj_digest": staged.manifest["traj_digest"],
+        "uploaded_by": staged.manifest["uploaded_by"],
+        "artifacts": artifacts,
+    }
+    manager = nullcontext(http_client) if http_client is not None else httpx.Client()
+    try:
+        with _quiet_httpx_request_logging(), manager as client:
+            response = client.post(endpoint, json=request_body, timeout=30)
+            if response.status_code == 409:
+                return _already_uploaded_result(response, staged, broker_url)
+            _raise_for_broker_response(response, operation="upload handshake")
+            payload = _response_object(response)
+            objects = _validated_objects(payload, staged)
+            base_url, prefix = _destination(payload)
+
+            uploaded: list[str] = []
+            skipped: list[str] = []
+            for staged_file, upload in zip(staged.files, objects, strict=True):
+                object_name = prefix + staged_file.relname
+                with staged_file.local_path.open("rb") as stream:
+                    put_response = client.put(
+                        upload["put_url"],
+                        headers=upload["headers"],
+                        content=stream,
+                        timeout=300,
+                    )
+                if put_response.status_code in {409, 412}:
+                    skipped.append(object_name)
+                    continue
+                _raise_for_broker_response(
+                    put_response,
+                    operation=f"upload of {staged_file.relname}",
+                )
+                uploaded.append(object_name)
+    except httpx.HTTPError as exc:
+        raise ValueError(f"trajectory broker request failed: {exc}") from exc
+
+    return BrokerPublishResult(
+        base_url=base_url,
+        prefix=prefix,
+        uploaded=tuple(uploaded),
+        skipped=tuple(skipped),
+    )
+
+
+@contextmanager
+def _quiet_httpx_request_logging():
+    """Keep short-lived signed upload URLs out of the global INFO stream."""
+    httpx_logger = logging.getLogger("httpx")
+    previous_level = httpx_logger.level
+    httpx_logger.setLevel(logging.WARNING)
+    try:
+        yield
+    finally:
+        httpx_logger.setLevel(previous_level)
+
+
+def _already_uploaded_result(
+    response: httpx.Response,
+    staged: StagedCapture,
+    broker_url: str,
+) -> BrokerPublishResult:
+    try:
+        payload = _response_object(response)
+        base_url, prefix = _destination(payload)
+    except ValueError:
+        base_url = broker_url.rstrip("/")
+        prefix = f"sources/community/{staged.traj_digest}/"
+    return BrokerPublishResult(
+        base_url=base_url,
+        prefix=prefix,
+        uploaded=(),
+        skipped=tuple(prefix + item.relname for item in staged.files),
+    )
+
+
+def _response_object(response: httpx.Response) -> dict[str, Any]:
+    try:
+        payload = response.json()
+    except ValueError as exc:
+        raise ValueError("trajectory broker returned invalid JSON") from exc
+    if not isinstance(payload, dict):
+        raise ValueError("trajectory broker returned a non-object response")
+    return payload
+
+
+def _validated_objects(
+    payload: dict[str, Any], staged: StagedCapture
+) -> list[dict[str, Any]]:
+    objects = payload.get("objects")
+    if not isinstance(objects, list) or not all(
+        isinstance(item, dict) for item in objects
+    ):
+        raise ValueError("trajectory broker protocol violation: objects must be a list")
+    expected_names = [item.relname for item in staged.files]
+    names = [item.get("name") for item in objects]
+    if names != expected_names:
+        raise ValueError(
+            "trajectory broker protocol violation: response objects must match "
+            "the staged files in canonical manifest-last order"
+        )
+    for item in objects:
+        put_url = item.get("put_url")
+        if not isinstance(put_url, str):
+            raise ValueError("trajectory broker protocol violation: missing put_url")
+        parsed_url = urlparse(put_url)
+        if (
+            parsed_url.scheme != "https"
+            or not parsed_url.hostname
+            or parsed_url.username is not None
+            or parsed_url.password is not None
+        ):
+            raise ValueError(
+                "trajectory broker protocol violation: put_url must be an "
+                "authenticated HTTPS URL"
+            )
+        headers = item.get("headers")
+        if not isinstance(headers, dict) or not all(
+            isinstance(key, str) and isinstance(value, str)
+            for key, value in headers.items()
+        ):
+            raise ValueError(
+                "trajectory broker protocol violation: headers must map strings to strings"
+            )
+    return objects
+
+
+def _destination(payload: dict[str, Any]) -> tuple[str, str]:
+    prefix = payload.get("prefix")
+    if not isinstance(prefix, str) or not prefix:
+        raise ValueError("trajectory broker protocol violation: missing prefix")
+    base_url = payload.get("base_url")
+    if isinstance(base_url, str) and base_url.startswith("https://"):
+        return base_url, prefix
+    bucket = payload.get("bucket")
+    if isinstance(bucket, str) and bucket:
+        return f"gs://{bucket}", prefix
+    raise ValueError("trajectory broker protocol violation: missing destination")
+
+
+def _raise_for_broker_response(response: httpx.Response, *, operation: str) -> None:
+    if response.status_code < 400:
+        return
+    detail = response.text.strip().replace("\n", " ")[:300]
+    if response.status_code == 413:
+        raise ValueError(f"trajectory broker rejected an oversized capture: {detail}")
+    if response.status_code == 429:
+        retry_after = response.headers.get("Retry-After")
+        suffix = f"; retry after {retry_after}" if retry_after else ""
+        raise ValueError(f"trajectory broker rate limit exceeded{suffix}: {detail}")
+    raise ValueError(
+        f"trajectory broker {operation} failed with HTTP {response.status_code}: {detail}"
+    )