benchflow 0.6.10.dev1928__tar.gz → 0.6.10.dev1954__tar.gz

{benchflow-0.6.10.dev1928 → benchflow-0.6.10.dev1954}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: benchflow
-Version: 0.6.10.dev1928
+Version: 0.6.10.dev1954
 Summary: Multi-turn agent benchmarking with ACP — run any agent, any model, any provider.
 Project-URL: Homepage, https://github.com/benchflow-ai/benchflow
 Project-URL: Repository, https://github.com/benchflow-ai/benchflow
@@ -87,31 +87,72 @@ Description-Content-Type: text/markdown
 
 BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with *any* ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.
 
-- **Run any benchmark** — three-layer routing runs supported frameworks natively, translates unknown formats and proves equivalence with a parity gate, or runs a bespoke harness as-is; every layer emits one scored-trajectory contract. See [Run any benchmark](./docs/running-any-benchmark.md)
-- **Any ACP agent** — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, Pi, or your own
-- **Single + multi + progressive** — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python `BaseUser` callback
-- **Loop strategies** — wrap any agent in a `--loop-strategy` (`verify-retry`, `self-review`); every rollout captures a per-iteration reward + token trajectory, so you can plot capability against cost (can a cheap model + loops match an expensive one at equal token spend?)
-- **`task.md` tasks** — one file (YAML frontmatter + prompt body) replaces the split `task.toml` + `instruction.md` layout; author with `bench tasks init` / `check` / `migrate` / `export`
-- **Hosted environments** — run external PrimeIntellect / Verifiers environments through `--source-env`, without converting them to BenchFlow tasks
-- **Sandboxes** — Docker locally, Apple Container on Apple Silicon Macs, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments, and AgentCore for AWS-hosted runtimes
-- **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
-- **Training-ready output** — scored rollouts emit a Verifiers/ORS reward record and best-effort ATIF (`trainer/atif.json`) / ADP (`trainer/adp.jsonl`) conversions; ATIF is omitted when the trajectory is empty, and conversion errors are reported in the rollout result
+## Qucik start: 1. Upload a trajectory
 
-## Quickstart
+For local Claude Code or Codex jsonl formatted trajectory files that you are proud of (contain challenging tasks you dealed with via chatting to agents), you can upload that file to BenchFlow to join the competition for winning $2,000 cash reward. No BenchFlow account, credentials, or API key is required:
 
 ```bash
-# Install or upgrade to the latest stable BenchFlow CLI
+# Install or upgrade BenchFlow
 uv tool install --python 3.12 --upgrade benchflow
 
-# Run a benchmark: any task source, any ACP agent, any sandbox
-export GEMINI_API_KEY=...            # or claude auth login / codex login for subscription auth
+# Upload one capture
+bench traj upload /absolute/path/to/trial \
+  --github-id YOUR_GITHUB_ID \
+  --email YOU@example.com
+```
+
+The path may be a trial directory, a directory of JSONL files, or one JSONL
+file. Both contributor fields are required and are stored in `manifest.json`.
+See the concise [upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md)
+or the [trajectory upload guide](./docs/traj-upload.md).
+
+## Qucik start: 2. Run with a ChatGPT or Claude subscription
+
+No OpenAI or Anthropic API key is required. Start Docker, install BenchFlow,
+then run **one** of these options. BenchFlow detects the saved host login and
+makes it available to the agent inside the sandbox.
+
+```bash
+uv tool install --python 3.12 --upgrade benchflow
+docker info >/dev/null  # Docker must be running
+```
+
+### ChatGPT subscription via Codex
+
+Install the [Codex CLI](https://github.com/openai/codex), then:
+
+```bash
+codex login
+unset OPENAI_API_KEY CODEX_API_KEY  # ensure subscription auth is used
+
 bench eval run \
-    --source-repo benchflow-ai/skillsbench --source-path tasks \
-    --agent gemini --model gemini-3.1-flash-lite-preview \
-    --sandbox docker
+  --source-repo benchflow-ai/skillsbench \
+  --source-path tasks/citation-check \
+  --agent codex \
+  --model gpt-5.5 \
+  --sandbox docker
 ```
 
-Each run writes a per-task `result.json` (rewards, trajectory summary, and token usage), full events under `trajectory/`, and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
+### Claude subscription via Claude Code
+
+Install [Claude Code](https://code.claude.com/docs/en/quickstart), then:
+
+```bash
+claude auth login
+unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN  # ensure subscription auth is used
+
+bench eval run \
+  --source-repo benchflow-ai/skillsbench \
+  --source-path tasks/citation-check \
+  --agent claude \
+  --model claude-sonnet-4-6 \
+  --sandbox docker
+```
+
+The agent may pass or fail the benchmark task; either result means the
+evaluation completed. Each run writes rewards, token usage, and the full
+trajectory under `jobs/`. See [Getting started](./docs/getting-started.md) for
+other agents, models, and sandboxes.
 
 ## Install
 
@@ -138,27 +179,6 @@ subscription auth (`claude auth login` / `codex login`). Provider-prefixed model
 may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` +
 `AZURE_API_ENDPOINT`.
 
-## Upload trajectories
-
-Anyone can contribute completed trajectories through the public service; no
-Azure account or API key is required. Run one command per capture:
-
-```bash
-# Install or upgrade the latest stable CLI
-uv tool install --python 3.12 --upgrade benchflow
-
-# Upload each capture
-bench traj upload /path/to/traj-1 --github-id YOUR_GITHUB_ID --email YOU@example.com
-bench traj upload /path/to/traj-2 --github-id YOUR_GITHUB_ID --email YOU@example.com
-bench traj upload /path/to/traj-3 --github-id YOUR_GITHUB_ID --email YOU@example.com
-```
-
-Both contributor fields are required and are stored under `contributor` in the
-uploaded `manifest.json`. A path may be a trial directory, a directory of JSONL
-files, or one JSONL file. See the concise
-[upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md) for agent use and
-[trajectory upload guide](./docs/traj-upload.md) for privacy and security details.
-
 ## Documentation
 
 Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/concepts.md) for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code, Codex CLI, or Gemini CLI. Then by goal:

{benchflow-0.6.10.dev1928 → benchflow-0.6.10.dev1954}/README.md

@@ -13,31 +13,72 @@
 
 BenchFlow is a universal environment framework: it runs AI agents against task environments and scores them through one hardened contract. **A benchmark is just a frozen environment** — point BenchFlow at any of them, drive it with *any* ACP agent, and run single-agent, multi-agent, or multi-round patterns over the same Scene-based lifecycle.
 
-- **Run any benchmark** — three-layer routing runs supported frameworks natively, translates unknown formats and proves equivalence with a parity gate, or runs a bespoke harness as-is; every layer emits one scored-trajectory contract. See [Run any benchmark](./docs/running-any-benchmark.md)
-- **Any ACP agent** — Gemini CLI, Claude Code, Codex, OpenCode, OpenHands, Pi, or your own
-- **Single + multi + progressive** — single-agent / multi-agent (coder + reviewer, simulated user) / multi-round with a Python `BaseUser` callback
-- **Loop strategies** — wrap any agent in a `--loop-strategy` (`verify-retry`, `self-review`); every rollout captures a per-iteration reward + token trajectory, so you can plot capability against cost (can a cheap model + loops match an expensive one at equal token spend?)
-- **`task.md` tasks** — one file (YAML frontmatter + prompt body) replaces the split `task.toml` + `instruction.md` layout; author with `bench tasks init` / `check` / `migrate` / `export`
-- **Hosted environments** — run external PrimeIntellect / Verifiers environments through `--source-env`, without converting them to BenchFlow tasks
-- **Sandboxes** — Docker locally, Apple Container on Apple Silicon Macs, Daytona for parallel cloud runs (orphaned sandboxes auto-reaped at eval start), Modal for serverless/GPU-backed task environments, and AgentCore for AWS-hosted runtimes
-- **Hardened verifier** — defaults block BenchJack/Meerkat-style reward-hacking; tasks opt out per-feature
-- **Training-ready output** — scored rollouts emit a Verifiers/ORS reward record and best-effort ATIF (`trainer/atif.json`) / ADP (`trainer/adp.jsonl`) conversions; ATIF is omitted when the trajectory is empty, and conversion errors are reported in the rollout result
-
-## Quickstart
+## Qucik start: 1. Upload a trajectory
+
+For local Claude Code or Codex jsonl formatted trajectory files that you are proud of (contain challenging tasks you dealed with via chatting to agents), you can upload that file to BenchFlow to join the competition for winning $2,000 cash reward. No BenchFlow account, credentials, or API key is required:
 
 ```bash
-# Install or upgrade to the latest stable BenchFlow CLI
+# Install or upgrade BenchFlow
 uv tool install --python 3.12 --upgrade benchflow
 
-# Run a benchmark: any task source, any ACP agent, any sandbox
-export GEMINI_API_KEY=...            # or claude auth login / codex login for subscription auth
+# Upload one capture
+bench traj upload /absolute/path/to/trial \
+  --github-id YOUR_GITHUB_ID \
+  --email YOU@example.com
+```
+
+The path may be a trial directory, a directory of JSONL files, or one JSONL
+file. Both contributor fields are required and are stored in `manifest.json`.
+See the concise [upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md)
+or the [trajectory upload guide](./docs/traj-upload.md).
+
+## Qucik start: 2. Run with a ChatGPT or Claude subscription
+
+No OpenAI or Anthropic API key is required. Start Docker, install BenchFlow,
+then run **one** of these options. BenchFlow detects the saved host login and
+makes it available to the agent inside the sandbox.
+
+```bash
+uv tool install --python 3.12 --upgrade benchflow
+docker info >/dev/null  # Docker must be running
+```
+
+### ChatGPT subscription via Codex
+
+Install the [Codex CLI](https://github.com/openai/codex), then:
+
+```bash
+codex login
+unset OPENAI_API_KEY CODEX_API_KEY  # ensure subscription auth is used
+
 bench eval run \
-    --source-repo benchflow-ai/skillsbench --source-path tasks \
-    --agent gemini --model gemini-3.1-flash-lite-preview \
-    --sandbox docker
+  --source-repo benchflow-ai/skillsbench \
+  --source-path tasks/citation-check \
+  --agent codex \
+  --model gpt-5.5 \
+  --sandbox docker
 ```
 
-Each run writes a per-task `result.json` (rewards, trajectory summary, and token usage), full events under `trajectory/`, and a job `summary.json` (pass-rate, cost, and — for looped runs — a pass@iteration convergence curve). New here? Start with [Getting started](./docs/getting-started.md), or paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code / Codex / Gemini CLI and let it drive the whole thing.
+### Claude subscription via Claude Code
+
+Install [Claude Code](https://code.claude.com/docs/en/quickstart), then:
+
+```bash
+claude auth login
+unset ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN  # ensure subscription auth is used
+
+bench eval run \
+  --source-repo benchflow-ai/skillsbench \
+  --source-path tasks/citation-check \
+  --agent claude \
+  --model claude-sonnet-4-6 \
+  --sandbox docker
+```
+
+The agent may pass or fail the benchmark task; either result means the
+evaluation completed. Each run writes rewards, token usage, and the full
+trajectory under `jobs/`. See [Getting started](./docs/getting-started.md) for
+other agents, models, and sandboxes.
 
 ## Install
 
@@ -64,27 +105,6 @@ subscription auth (`claude auth login` / `codex login`). Provider-prefixed model
 may need provider-specific credentials; Azure Foundry uses `AZURE_API_KEY` +
 `AZURE_API_ENDPOINT`.
 
-## Upload trajectories
-
-Anyone can contribute completed trajectories through the public service; no
-Azure account or API key is required. Run one command per capture:
-
-```bash
-# Install or upgrade the latest stable CLI
-uv tool install --python 3.12 --upgrade benchflow
-
-# Upload each capture
-bench traj upload /path/to/traj-1 --github-id YOUR_GITHUB_ID --email YOU@example.com
-bench traj upload /path/to/traj-2 --github-id YOUR_GITHUB_ID --email YOU@example.com
-bench traj upload /path/to/traj-3 --github-id YOUR_GITHUB_ID --email YOU@example.com
-```
-
-Both contributor fields are required and are stored under `contributor` in the
-uploaded `manifest.json`. A path may be a trial directory, a directory of JSONL
-files, or one JSONL file. See the concise
-[upload skill](./.agents/skills/benchflow-traj-upload/SKILL.md) for agent use and
-[trajectory upload guide](./docs/traj-upload.md) for privacy and security details.
-
 ## Documentation
 
 Start with [Getting started](./docs/getting-started.md), then [Concepts](./docs/concepts.md) for the mental model. Prefer to have an AI coding agent run the whole quickstart for you? Paste the [agent quickstart prompt](./docs/agent-quickstart.md) into Claude Code, Codex CLI, or Gemini CLI. Then by goal:

{benchflow-0.6.10.dev1928 → benchflow-0.6.10.dev1954}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "benchflow"
-version = "0.6.10.dev1928"
+version = "0.6.10.dev1954"
 description = "Multi-turn agent benchmarking with ACP — run any agent, any model, any provider."
 readme = "README.md"
 requires-python = ">=3.12"
@@ -160,10 +160,17 @@ markers = [
 target-version = "py312"
 # Vendored third-party service packages baked into task images (e.g. the
 # smolclaws claw-* sources copied under a ClawsBench task's environment/) are
-# not BenchFlow code — do not lint them.
+# not BenchFlow code — do not lint them. The skill-creator, task-creator, and
+# task-review skills are verbatim upstream copies; keeping them byte-identical
+# matters more than local style, so they are excluded too. Skills authored here
+# (e.g. benchflow-experiment-review) stay linted.
 extend-exclude = [
     ".agents/skills/skill-creator",
+    ".agents/skills/task-creator",
+    ".agents/skills/task-review",
     ".claude/skills/skill-creator",
+    ".claude/skills/task-creator",
+    ".claude/skills/task-review",
     "benchmarks/**/tasks/**/environment/claw-*",
     "tests/fixtures/skillsbench_slice",
 ]