pluginbench 0.5.0__tar.gz → 0.6.0__tar.gz

{pluginbench-0.5.0 → pluginbench-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pluginbench
-Version: 0.5.0
+Version: 0.6.0
 Summary: Test whether skills help coding agents make fewer mistakes.
 Project-URL: Homepage, https://github.com/filipgutica/pluginbench
 Project-URL: Documentation, https://github.com/filipgutica/pluginbench/blob/main/docs/running-evaluations.md
@@ -70,7 +70,11 @@ PluginBench runs on Linux or macOS. Every run needs:
 | Python | 3.12 or later | Runs PluginBench |
 | Git | Any recent | Task sources and patches |
 | Docker | Linux containers on a local Unix socket, with Compose and Buildx | Each attempt and its verifier run in containers |
-| Codex CLI | Exactly 0.162.1 or 0.167.0, logged in | Coding agent. PluginBench checks the `codex` on your `PATH` and copies its login from `~/.codex/auth.json`. Other versions fail `check`. |
+| Codex CLI | Logged in; verified trace versions: 0.162.1, 0.167.0, 0.167.1 | Coding agent. PluginBench copies its login from `~/.codex/auth.json` to the pinned container CLI. Unverified versions warn and continue; incompatible traces or unverified inherited history leave tokens unavailable. |
+
+Set `runner.cli_version: "0.167.0"` to select the exact coding CLI version installed in task containers.
+When omitted, preparation pins the host CLI's version. Resume keeps that pin across host CLI updates.
+Independent reviews use the host CLI. See [verified coding CLIs](docs/running-evaluations.md#verified-coding-clis) for credential compatibility.
 
 Some measures and commands need more:
 
@@ -84,7 +88,14 @@ Some measures and commands need more:
 
 ## Install
 
-There is no registry release yet. Install from a clone:
+```sh
+pipx install pluginbench
+pluginbench --version
+```
+
+`uv tool install pluginbench` works too. Either one also installs Harbor 0.24.0. If your default Python is older than 3.12, add `--python python3.12`.
+
+To change PluginBench itself, install from a clone instead:
 
 ```sh
 git clone https://github.com/filipgutica/pluginbench.git
@@ -92,7 +103,6 @@ cd pluginbench
 python3 -m venv .venv
 source .venv/bin/activate
 pip install -e .
-pluginbench --version
 ```
 
 ## Quickstart
@@ -139,7 +149,7 @@ This run uses one public DeepSWE task, vanilla only, and one repeat. It needs no
    pluginbench run evaluation.yaml
    ```
 
-   `run` prints the plan and asks `Start evaluation?` (`--yes` skips the question).
+   `run` prints the plan and asks `Start evaluation?` (`--yes` skips the question; in a script with no answer to give, pass `--yes`).
    It then runs the same checks as `check`, starts the run in the background, and prints the run directory, for example `runs/20261110T101500123456Z`.
 
 5. Follow it:
@@ -263,7 +273,7 @@ Every setting lives in the config. Flags only control how a command runs.
 
 ## Stop, resume, and reuse
 
-- `pluginbench stop runs/<run>` stops new attempts. With `jobs` above 1, attempts already coding finish and are kept. With `jobs: 1`, the attempt in progress is interrupted, and a resume codes its slot again with a new attempt.
+- `pluginbench stop runs/<run>` stops new attempts. Attempts already coding finish and are kept, including with `jobs: 1`. Resume never codes complete attempts again.
 - `pluginbench run runs/<run>` resumes. A run that was killed, for example by a host shutdown, shows `interrupted`, and `status` prints this command. Complete attempts are kept and never coded again. An attempt cut off mid-call is recorded as `discarded-interrupted` and a new attempt takes its slot. Containers the killed attempt left running are stopped before coding resumes, and `status` names them until then. Resume cannot repair a failing solution.
 - If Codex fails before its first model turn, for example during a network outage, the run stops instead of scoring the attempt. Resume when Codex is reachable.
 - Vanilla attempts are cached in `$XDG_CACHE_HOME/pluginbench` (default `~/.cache/pluginbench`) and reused by any later run with the same task, model, and settings. Grading and measures run again. Reports mark reused attempts. [Baselines and reports](docs/running-evaluations.md#baselines-and-reports).

{pluginbench-0.5.0 → pluginbench-0.6.0}/README.md

@@ -34,7 +34,11 @@ PluginBench runs on Linux or macOS. Every run needs:
 | Python | 3.12 or later | Runs PluginBench |
 | Git | Any recent | Task sources and patches |
 | Docker | Linux containers on a local Unix socket, with Compose and Buildx | Each attempt and its verifier run in containers |
-| Codex CLI | Exactly 0.162.1 or 0.167.0, logged in | Coding agent. PluginBench checks the `codex` on your `PATH` and copies its login from `~/.codex/auth.json`. Other versions fail `check`. |
+| Codex CLI | Logged in; verified trace versions: 0.162.1, 0.167.0, 0.167.1 | Coding agent. PluginBench copies its login from `~/.codex/auth.json` to the pinned container CLI. Unverified versions warn and continue; incompatible traces or unverified inherited history leave tokens unavailable. |
+
+Set `runner.cli_version: "0.167.0"` to select the exact coding CLI version installed in task containers.
+When omitted, preparation pins the host CLI's version. Resume keeps that pin across host CLI updates.
+Independent reviews use the host CLI. See [verified coding CLIs](docs/running-evaluations.md#verified-coding-clis) for credential compatibility.
 
 Some measures and commands need more:
 
@@ -48,7 +52,14 @@ Some measures and commands need more:
 
 ## Install
 
-There is no registry release yet. Install from a clone:
+```sh
+pipx install pluginbench
+pluginbench --version
+```
+
+`uv tool install pluginbench` works too. Either one also installs Harbor 0.24.0. If your default Python is older than 3.12, add `--python python3.12`.
+
+To change PluginBench itself, install from a clone instead:
 
 ```sh
 git clone https://github.com/filipgutica/pluginbench.git
@@ -56,7 +67,6 @@ cd pluginbench
 python3 -m venv .venv
 source .venv/bin/activate
 pip install -e .
-pluginbench --version
 ```
 
 ## Quickstart
@@ -103,7 +113,7 @@ This run uses one public DeepSWE task, vanilla only, and one repeat. It needs no
    pluginbench run evaluation.yaml
    ```
 
-   `run` prints the plan and asks `Start evaluation?` (`--yes` skips the question).
+   `run` prints the plan and asks `Start evaluation?` (`--yes` skips the question; in a script with no answer to give, pass `--yes`).
    It then runs the same checks as `check`, starts the run in the background, and prints the run directory, for example `runs/20261110T101500123456Z`.
 
 5. Follow it:
@@ -227,7 +237,7 @@ Every setting lives in the config. Flags only control how a command runs.
 
 ## Stop, resume, and reuse
 
-- `pluginbench stop runs/<run>` stops new attempts. With `jobs` above 1, attempts already coding finish and are kept. With `jobs: 1`, the attempt in progress is interrupted, and a resume codes its slot again with a new attempt.
+- `pluginbench stop runs/<run>` stops new attempts. Attempts already coding finish and are kept, including with `jobs: 1`. Resume never codes complete attempts again.
 - `pluginbench run runs/<run>` resumes. A run that was killed, for example by a host shutdown, shows `interrupted`, and `status` prints this command. Complete attempts are kept and never coded again. An attempt cut off mid-call is recorded as `discarded-interrupted` and a new attempt takes its slot. Containers the killed attempt left running are stopped before coding resumes, and `status` names them until then. Resume cannot repair a failing solution.
 - If Codex fails before its first model turn, for example during a network outage, the run stops instead of scoring the attempt. Resume when Codex is reachable.
 - Vanilla attempts are cached in `$XDG_CACHE_HOME/pluginbench` (default `~/.cache/pluginbench`) and reused by any later run with the same task, model, and settings. Grading and measures run again. Reports mark reused attempts. [Baselines and reports](docs/running-evaluations.md#baselines-and-reports).

{pluginbench-0.5.0 → pluginbench-0.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pluginbench"
-version = "0.5.0"
+version = "0.6.0"
 description = "Test whether skills help coding agents make fewer mistakes."
 readme = "README.md"
 requires-python = ">=3.12"

{pluginbench-0.5.0 → pluginbench-0.6.0}/src/pluginbench/__init__.py

@@ -1,3 +1,3 @@
 """Compare native instructions, skills, and plugins with vanilla coding agents."""
 
-__version__ = "0.5.0"
+__version__ = "0.6.0"

pluginbench-0.6.0/src/pluginbench/arms.py

@@ -0,0 +1,6 @@
+"""Public arm names, independent of frozen run and cache identities."""
+
+
+def normalize_arm_name(name: str) -> str:
+    """Accept the stored/legacy name and the public name for the no-guidance arm."""
+    return "vanilla" if name == "baseline" else name

{pluginbench-0.5.0 → pluginbench-0.6.0}/src/pluginbench/assessment-charts.html

@@ -273,7 +273,8 @@ const formatTick = value => new Intl.NumberFormat(undefined, { notation: Math.ab
 // Values below one keep two significant digits so a small real difference never reads as zero.
 const formatCompact = value => value === null || value === undefined ? "Unknown" : new Intl.NumberFormat(undefined, Math.abs(value) >= 10000 ? { notation: "compact", maximumFractionDigits: 2 } : value !== 0 && Math.abs(value) < 1 ? { maximumSignificantDigits: 2 } : { maximumFractionDigits: 2 }).format(value);
 const formatSigned = value => (value > 0 ? "+" : "") + formatCompact(value);
-const armLabel = (arm) => arm === "baseline" ? "Vanilla" : arm === "unknown" ? "Unlabelled skill set" : arm;
+const vanillaArm = data.kind === "evaluation" ? "vanilla" : "baseline";
+const armLabel = (arm) => arm === vanillaArm ? "Vanilla" : arm === "unknown" ? "Unlabelled skill set" : arm;
 const choices = (select, values) => {
   for (const value of values) create("option", value, select).value = value;
 };
@@ -283,10 +284,10 @@ const knownTasks = unique(observations.map(row => row.task).filter(task => task
 choices(taskSelect, knownTasks);
 taskSelect.value = "";
 // One package order and color per package everywhere: Vanilla first, then packages by name.
-const packageNames = unique([...observations.map(row => row.arm), ...data.attempts.map(row => row.guidance)]).filter(arm => arm !== "baseline").sort();
-const armRank = arm => arm === "baseline" ? -1 : packageNames.indexOf(arm);
+const packageNames = unique([...observations.map(row => row.arm), ...data.attempts.map(row => row.guidance)]).filter(arm => arm !== vanillaArm).sort();
+const armRank = arm => arm === vanillaArm ? -1 : packageNames.indexOf(arm);
 const byArm = (a, b) => armRank(a) - armRank(b);
-const seriesVariable = arm => arm !== "baseline" && armRank(arm) >= 0 ? "--series-" + (armRank(arm) % 8 + 1) : "--pb-muted";
+const seriesVariable = arm => arm !== vanillaArm && armRank(arm) >= 0 ? "--series-" + (armRank(arm) % 8 + 1) : "--pb-muted";
 const viewLabel = create("label", "Chart values", document.querySelector(".controls"));
 const comparisonView = create("select", undefined, viewLabel);
 for (const [value, label] of [["delta", "Delta vs vanilla"], ["raw", "Raw values"]]) create("option", label, comparisonView).value = value;
@@ -414,7 +415,7 @@ function showPackageSummary() {
           const text = result?.value !== undefined && result.value !== null ? formatCompact(result.value) : "Unknown";
           create("div", text + (groups.length > 1 && result?.analysis_group ? " · " + result.analysis_group : ""), cell);
           if (!taskSelect.value && result) create("small", result.measured_tasks + "/" + result.assigned_tasks + " tasks", cell);
-          if (arm !== "baseline") {
+          if (arm !== vanillaArm) {
             const deltaRows = taskSelect.value ? (data.comparisons || []).filter(row => row.task === taskSelect.value && row.metric === metric && row.guidance === arm) : [result];
             const delta = deltaRows.length === 1 ? deltaRows[0] : null;
             const value = delta?.absolute_delta_vs_vanilla ?? null;
@@ -496,7 +497,7 @@ function showCompleteness() {
   // Task × measure × package: measured, unknown with its reason, or not configured for the task.
   const matrix = (data.completeness_matrix || []).filter(row => !taskSelect.value || row.task === taskSelect.value);
   if (!matrix.length) return;
-  const arms = unique(matrix.flatMap(row => Object.keys(row.cells)));
+  const arms = unique(matrix.flatMap(row => Object.keys(row.cells))).sort(byArm);
   const grid = create("table", undefined, body);
   const head = create("tr", undefined, grid);
   for (const label of ["Task", "Measure", ...arms.map(armLabel)]) create("th", label, head);
@@ -626,10 +627,10 @@ function evaluationPlotRows({ rows, metric }) {
   const raw = comparisonView.value === "raw", overview = !taskSelect.value;
   if (overview && chartBreakdown.value === "tasks") {
     const medians = raw ? (data.raw_medians || []) : (data.comparisons || []);
-    return medians.filter(row => row.metric === metric).map(row => ({ ...row, guidance: row.guidance || row.arm, label: row.task.replaceAll("-", " "), plotted: raw ? row.value : row.absolute_delta_vs_vanilla, status: raw ? "measured" : row.status, reason: raw ? "Raw task median for one measurement profile. Comparison eligibility is separate." : row.reason, population: raw ? row.measured + " runs" : "Vanilla " + row.baseline_measured + "/" + row.baseline_assigned + " · guidance " + row.guidance_measured + "/" + row.guidance_assigned + " runs", rawRuns: [] })).sort((a, b) => a.task.localeCompare(b.task) || Number(b.guidance === "baseline") - Number(a.guidance === "baseline") || a.guidance.localeCompare(b.guidance));
+    return medians.filter(row => row.metric === metric).map(row => ({ ...row, guidance: row.guidance || row.arm, label: row.task.replaceAll("-", " "), plotted: raw ? row.value : row.absolute_delta_vs_vanilla, status: raw ? "measured" : row.status, reason: raw ? "Raw task median for one measurement profile. Comparison eligibility is separate." : row.reason, population: raw ? row.measured + " runs" : "Vanilla " + row.vanilla_measured + "/" + row.vanilla_assigned + " · guidance " + row.guidance_measured + "/" + row.guidance_assigned + " runs", rawRuns: [] })).sort((a, b) => a.task.localeCompare(b.task) || Number(b.guidance === "vanilla") - Number(a.guidance === "vanilla") || a.guidance.localeCompare(b.guidance));
   }
   if (overview) return (data.aggregates || []).filter(row => row.metric === metric).map(row => ({ ...row, status: raw ? row.status : row.comparison_status, label: armLabel(row.guidance) + (row.analysis_group ? " · " + row.analysis_group : ""), plotted: raw ? row.value : row.absolute_delta_vs_vanilla, population: row.measured_tasks + "/" + row.assigned_tasks + " tasks", rawRuns: [] }));
-  if (!raw) return (data.comparisons || []).filter(row => row.metric === metric && row.task === taskSelect.value).map(row => ({ ...row, label: row.guidance, plotted: row.absolute_delta_vs_vanilla, population: "Vanilla " + row.baseline_measured + "/" + row.baseline_assigned + " · guidance " + row.guidance_measured + "/" + row.guidance_assigned + " runs", rawRuns: [] }));
+  if (!raw) return (data.comparisons || []).filter(row => row.metric === metric && row.task === taskSelect.value).map(row => ({ ...row, label: row.guidance, plotted: row.absolute_delta_vs_vanilla, population: "Vanilla " + row.vanilla_measured + "/" + row.vanilla_assigned + " · guidance " + row.guidance_measured + "/" + row.guidance_assigned + " runs", rawRuns: [] }));
   const medians = (data.raw_medians || []).filter(row => row.role === "handoff" && row.metric === metric && row.task === taskSelect.value);
   const arms = unique(rows.map(row => row.arm));
   return arms.flatMap(arm => {
@@ -652,9 +653,9 @@ function drawEvaluationPlot({ parent, rows, metric }) {
   for (const label of ["Set", raw ? "Value" : "Delta vs vanilla", ...(!raw ? ["Vanilla reference", "% vs vanilla"] : []), "Status", "Measured / assigned"]) create("th", label, head);
   for (const row of plotRows) {
     const tr = create("tr", undefined, table);
-    for (const value of [byTask ? row.label + " · " + armLabel(row.guidance) : row.label, format(row.plotted), ...(!raw ? [format(row.baseline_value ?? row.baseline_median ?? null), format(row.percentage_delta_vs_vanilla)] : []), row.status, row.population]) create("td", value, tr);
+    for (const value of [byTask ? row.label + " · " + armLabel(row.guidance) : row.label, format(row.plotted), ...(!raw ? [format(row.vanilla_value ?? row.vanilla_median ?? null), format(row.percentage_delta_vs_vanilla)] : []), row.status, row.population]) create("td", value, tr);
   }
-  if (!plotRows.some(row => row.plotted !== null || row.rawRuns.length)) { create("p", raw ? "No defined values to plot. Complete empty scans and missing measurements are listed separately." : "No matching baseline values for a delta. Switch to Raw values to see the available measurements.", parent); return 0; }
+  if (!plotRows.some(row => row.plotted !== null || row.rawRuns.length)) { create("p", raw ? "No defined values to plot. Complete empty scans and missing measurements are listed separately." : "No matching vanilla values for a delta. Switch to Raw values to see the available measurements.", parent); return 0; }
   const container = create("div", undefined, parent); container.className = "chart";
   const arms = unique(plotRows.map(row => row.guidance));
   const categories = grouped ? unique(plotRows.map(row => row.group)) : plotRows.map(row => byTask ? row.label + " · " + armLabel(row.guidance) : row.label);

{pluginbench-0.5.0 → pluginbench-0.6.0}/src/pluginbench/assessment_charts.py

@@ -623,6 +623,9 @@ def _observation(
         eligible = isinstance(work.get("token_format"), str) and bool(work["token_format"])
         identity.update(provenance)
         reason = "native tokens at handoff; cached input and reasoning output subsets are not added"
+        validation = _map(work.get("usage_validation"))
+        if validation:
+            provenance["usage_validation"] = validation
     elif kind == "loc_change":
         classification = "source" if key.startswith("source_") else "test"
         code_size = _map(stage.get("code_size" if classification == "source" else "test_code_size"))
@@ -897,6 +900,10 @@ def _observation(
                 if evidence.get("reason")
                 else "measurement unavailable; unknown is not zero",
             )
+        if kind == "work":
+            validation_reasons = _map(work.get("usage_validation")).get("reasons")
+            if isinstance(validation_reasons, list) and validation_reasons:
+                reason += "; " + "; ".join(str(item) for item in validation_reasons)
     else:
         status = "measured" if compatibility else "not comparable"
     change = _map(stage.get("change"))
@@ -1024,7 +1031,7 @@ def chart_data(report: dict[str, object]) -> dict[str, object]:
     for row in observations:
         if row.compatibility and not groups.get((row.role, row.compatibility, "baseline")):
             row.status = "not comparable"
-            row.reason += "; no matching explicit baseline observation"
+            row.reason += "; no matching explicit vanilla observation"
     medians = []
     for (role, identity, arm), rows in groups.items():
         baseline = groups.get((role, identity, "baseline"), [])

{pluginbench-0.5.0 → pluginbench-0.6.0}/src/pluginbench/cli.py

@@ -71,6 +71,17 @@ def pick(
         _error(exc)
 
 
+def _confirm(question: str) -> bool:
+    """Ask before paid work; when no answer can be read, explain how to start instead."""
+    try:
+        return typer.confirm(question)
+    except typer.Abort:
+        if sys.stdin.isatty():
+            raise  # Ctrl-C at the prompt.
+        typer.echo("No answer to confirm with. Pass --yes to start without confirmation.", err=True)
+        raise typer.Exit(1) from None
+
+
 def _error(exc: Exception) -> None:
     if isinstance(exc, CheckFailed):
         typer.echo(_readiness_table(exc.report), err=True)
@@ -392,7 +403,7 @@ def run(
                 "no notification will be sent.",
                 err=True,
             )
-        if not yes and not typer.confirm("Start evaluation?"):
+        if not yes and not _confirm("Start evaluation?"):
             raise typer.Exit()
         progress = _notifying_progress(plan["coding_calls"]) if notify else typer.echo
         if evaluation is None:
@@ -563,7 +574,7 @@ def review(
         with run_lock.hold(run):
             plan = plan_review(run, revision)
             typer.echo(f"{plan['review_calls']} review calls as review revision {revision}.")
-            if not yes and not typer.confirm("Start review?"):
+            if not yes and not _confirm("Start review?"):
                 raise typer.Exit()
             _review(run, revision)
     except (OSError, ValueError, subprocess.SubprocessError) as exc:
@@ -658,12 +669,15 @@ def judge_run(
 @judge_app.command("tally")
 def judge_tally(
     directory: Path = typer.Argument(..., help="A directory whose judge calls are complete."),
+    require_bound: bool = typer.Option(
+        False, help="Refuse legacy verdicts without input bindings."
+    ),
 ) -> None:
     """Unblind with the key and score each pair; writes tally.json; no model calls."""
     from pluginbench.judge import format_tally, tally
 
     try:
-        result = tally(directory)
+        result = tally(directory, require_bound=require_bound)
         (directory / "tally.json").write_text(json.dumps(result, indent=2) + "\n")
     except (OSError, ValueError, KeyError) as exc:
         _error(exc)

{pluginbench-0.5.0 → pluginbench-0.6.0}/src/pluginbench/codex.py

@@ -299,9 +299,10 @@ def invoke(
     prompt: str,
     timeout: int,
     output_schema: Path | None = None,
+    artifacts: Path | None = None,
 ) -> dict[str, Any]:
     """Invoke a local reviewer with an OS-enforced read-only source packet."""
-    artifacts = area / "artifacts"
+    artifacts = area / "artifacts" if artifacts is None else artifacts
     artifacts.mkdir(exist_ok=True)
     if any((artifacts / name).exists() for name in ("invocation.json", "execution.json")):
         raise ValueError("native invocation artifacts already exist; use a fresh trial area")