proofbench 0.1.0__tar.gz → 0.2.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. {proofbench-0.1.0 → proofbench-0.2.1}/.gitignore +1 -0
  2. proofbench-0.2.1/CORPUS.md +100 -0
  3. {proofbench-0.1.0 → proofbench-0.2.1}/PKG-INFO +3 -3
  4. {proofbench-0.1.0 → proofbench-0.2.1}/README.md +2 -2
  5. proofbench-0.2.1/corpora/build_hard.py +85 -0
  6. proofbench-0.2.1/corpora/fetch.sh +23 -0
  7. proofbench-0.2.1/corpora/manifest.yaml +73 -0
  8. proofbench-0.2.1/corpora/proofbench.yaml +38 -0
  9. {proofbench-0.1.0 → proofbench-0.2.1}/docs/crg-eval-journey.md +1 -1
  10. {proofbench-0.1.0 → proofbench-0.2.1}/docs/crg-eval-results.md +3 -3
  11. {proofbench-0.1.0 → proofbench-0.2.1}/examples/crg-debug/README.md +1 -1
  12. {proofbench-0.1.0 → proofbench-0.2.1}/prompts/judge.md +6 -1
  13. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/capture.py +13 -2
  14. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/cli.py +4 -0
  15. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/engine.py +36 -7
  16. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/runner.py +20 -3
  17. {proofbench-0.1.0 → proofbench-0.2.1}/pyproject.toml +1 -1
  18. {proofbench-0.1.0 → proofbench-0.2.1}/tests/test_engine.py +2 -0
  19. proofbench-0.1.0/CORPUS.md +0 -51
  20. {proofbench-0.1.0 → proofbench-0.2.1}/.github/workflows/release.yml +0 -0
  21. {proofbench-0.1.0 → proofbench-0.2.1}/LICENSE +0 -0
  22. {proofbench-0.1.0 → proofbench-0.2.1}/METHODOLOGY.md +0 -0
  23. {proofbench-0.1.0 → proofbench-0.2.1}/examples/proofbench.demo.yaml +0 -0
  24. {proofbench-0.1.0 → proofbench-0.2.1}/examples/sample-corpus/build.sh +0 -0
  25. {proofbench-0.1.0 → proofbench-0.2.1}/prompts/synth.md +0 -0
  26. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/__init__.py +0 -0
  27. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/config.py +0 -0
  28. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/judge.py +0 -0
  29. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/optimize.py +0 -0
  30. {proofbench-0.1.0 → proofbench-0.2.1}/proofbench/util.py +0 -0
  31. {proofbench-0.1.0 → proofbench-0.2.1}/tests/test_failloud.py +0 -0
@@ -4,6 +4,7 @@ dist/
4
4
  build/
5
5
  runs/
6
6
  examples/sample-corpus/repos/
7
+ corpora/repos/
7
8
  .venv/
8
9
  .pytest_cache/
9
10
  uv.lock
@@ -0,0 +1,100 @@
1
+ # Authoring a corpus
2
+
3
+ A **task** is a git fixture repository with two branches: a **default branch** (the broken or
4
+ incomplete state the agent starts from) and a **solution branch** (the held-out ground truth). The
5
+ literal branch names vary per repo — `main`/`master`, `buggy`, `solution`, `working` all appear in
6
+ the existing corpus — so the config's `default_branch` and `answer_branch` fields name them for the
7
+ harness (the solution branch is the `answer_branch` field); you point those fields at whatever the repo
8
+ actually calls its branches. The
9
+ **reference solution** is the diff between the two, and the judge grades the agent's patch against
10
+ it. Your eval is only as good as these tasks, so two properties carry the most weight: every task
11
+ must be **discriminative** (it separates a capable agent from a weak one) and **leak-free** (nothing
12
+ but the solution branch reveals the fix).
13
+
14
+ ## Anatomy of a task
15
+
16
+ ```mermaid
17
+ flowchart LR
18
+ subgraph repo["task — git fixture repo"]
19
+ A["default_branch<br/>(e.g. main / buggy)"]
20
+ B["answer_branch<br/>(e.g. solution / fixed)"]
21
+ end
22
+ A -- "reference solution<br/>git diff default_branch..answer_branch" --> B
23
+ ```
24
+
25
+ The agent never sees `answer_branch`. It is checked out on `default_branch`, asked to fix the
26
+ repo, and graded on how close its patch comes to the reference solution.
27
+
28
+ ## How a task is scored
29
+
30
+ ```mermaid
31
+ flowchart TD
32
+ reset["reset working tree → default_branch"] --> run["runner invokes the agent"]
33
+ run --> cap["capture: git diff of the working tree<br/>= agent patch"]
34
+ gt["ground truth:<br/>default_branch..answer_branch"] --> judge
35
+ cap --> judge{{"LLM judge"}}
36
+ judge --> out["recall · precision"]
37
+ ```
38
+
39
+ **Recall** is how much of the reference solution the agent reproduced; **precision** is how much
40
+ of the agent's patch was warranted rather than collateral damage. A clean, single-purpose task
41
+ keeps both metrics interpretable.
42
+
43
+ ## Designing out answer leakage
44
+
45
+ A task leaks when the agent or the judge can reach the answer without doing the work. There are
46
+ two distinct channels, and a good task closes both.
47
+
48
+ ```mermaid
49
+ flowchart TD
50
+ task["answer leakage"]
51
+ task --> c["Contamination<br/>the model already memorized the fix"]
52
+ task --> l["Label leakage<br/>the repo points at the fix"]
53
+ c --> cc["famous CVEs · textbook off-by-ones<br/>→ measures memory, not method"]
54
+ l --> ll["'intentional bug' READMEs · '# Buggy' comments<br/>→ judge grades the tell, not the contract"]
55
+ ```
56
+
57
+ ### Contamination: avoid famous defects
58
+
59
+ A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
60
+ does, you are measuring training-set recall, not debugging skill. Prefer subtle, real defects
61
+ whose fix turns on an invariant, a race, or a contract, and where the surrounding code offers no
62
+ hint. Discriminative tasks are the ones a model must actually reason through, not pattern-match.
63
+
64
+ ### Label leakage: do not telegraph the answer
65
+
66
+ The judge reads the repo's README. If the README says "this repo has an intentional bug for
67
+ practice," or the defective line carries a `# Buggy` comment, both the agent and the judge read
68
+ the tell instead of the contract. In one real run this flipped genuine contract violations into
69
+ "intended design" and silently corrupted the grades. So:
70
+
71
+ - Write the README as a real library would: state each function's **contract** (what it is
72
+ supposed to do), never that it is wrong.
73
+ - Strip comments that point at the defect. Leave the defective line exactly where it is.
74
+ - The fix exists in exactly one place: the `answer_branch`.
75
+
76
+ ## Practical shape
77
+
78
+ - One defect (or one coherent set) per task keeps recall and precision interpretable.
79
+ - Leave `default_branch` checked out as the repo's default, so a fresh clone lands in the broken
80
+ state the agent will see.
81
+ - `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each with a `buggy`
82
+ default branch and a `fixed` solution branch, a README stating the contract, and no tells.
83
+
84
+ ## Wiring the harness to your corpus
85
+
86
+ Each task in the config names the repo path, the agent invocation, and the two branches:
87
+
88
+ ```yaml
89
+ corpus:
90
+ - name: my-task
91
+ path: /abs/or/relative/path/to/repo
92
+ invoke: "Find and fix the bug in this repository."
93
+ default_branch: buggy
94
+ answer_branch: fixed
95
+ ```
96
+
97
+ ## External and licensed corpora
98
+
99
+ Do not commit large or licensed corpora into your harness repo. Reference them by a pinned git
100
+ URL and clone them on setup, so the data stays out of your tree and its provenance stays exact.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: proofbench
3
- Version: 0.1.0
3
+ Version: 0.2.1
4
4
  Summary: Config-driven eval harness that grades and self-improves headless agent skills against ground-truth corpora
5
5
  Project-URL: Homepage, https://github.com/CodeBlackwell/proofbench
6
6
  Author: LeChristopher Blackwell
@@ -90,14 +90,14 @@ the harness comparing two numbers, never the model's own claim of success.
90
90
 
91
91
  - **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
92
92
  `<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
93
- - **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
93
+ - **Capture**: the git-diff default resets a repo to its default branch and snapshots what the
94
94
  agent changed, excluding dependency and cache trees so they never pollute or balloon the
95
95
  judge prompt.
96
96
 
97
97
  ## Bring your own corpus
98
98
 
99
99
  The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
100
- (each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
100
+ (each with a broken default branch and a fixed solution branch). See [CORPUS.md](CORPUS.md) for
101
101
  how to curate tasks that actually discriminate and do not leak their answers to the judge.
102
102
 
103
103
  ## License
@@ -77,14 +77,14 @@ the harness comparing two numbers, never the model's own claim of success.
77
77
 
78
78
  - **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
79
79
  `<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
80
- - **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
80
+ - **Capture**: the git-diff default resets a repo to its default branch and snapshots what the
81
81
  agent changed, excluding dependency and cache trees so they never pollute or balloon the
82
82
  judge prompt.
83
83
 
84
84
  ## Bring your own corpus
85
85
 
86
86
  The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
87
- (each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
87
+ (each with a broken default branch and a fixed solution branch). See [CORPUS.md](CORPUS.md) for
88
88
  how to curate tasks that actually discriminate and do not leak their answers to the judge.
89
89
 
90
90
  ## License
@@ -0,0 +1,85 @@
1
+ """Rebuild the two hard, benchmark-derived corpus cases as buggy->fixed git pairs.
2
+
3
+ Each case is one published-benchmark instance materialized locally (repos/ is gitignored):
4
+ pdb-multi-shortest-substring PDB-Multi 3692_41 (arXiv 2604.17338) — 3 bugs, complete GT.
5
+ repodebug-gvl-tracing RepoDebug (EMNLP 2025) C wrong_return; upstream = fixed ref.
6
+
7
+ Run: python3 corpora/build_hard.py
8
+ """
9
+ import json
10
+ import subprocess
11
+ import urllib.request
12
+ from pathlib import Path
13
+
14
+ REPOS = Path(__file__).resolve().parent / "repos"
15
+ PDB = "https://datasets-server.huggingface.co/rows?dataset=Precise-Debugging-Benchmarking%2FPDB-Multi&config=default&split=test&offset={off}&length=100"
16
+ RD = "https://raw.githubusercontent.com/BUAA-IRIP-LLM/RepoDebug/main/data/c.json"
17
+
18
+
19
+ def _get(url):
20
+ return urllib.request.urlopen(url, timeout=30).read().decode("utf-8", "replace")
21
+
22
+
23
+ def _git(repo, *args):
24
+ subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
25
+
26
+
27
+ def build(name, fname, buggy, fixed, readme):
28
+ repo = REPOS / name
29
+ subprocess.run(["rm", "-rf", str(repo)], check=True)
30
+ repo.mkdir(parents=True)
31
+ _git(repo, "init", "-q", "-b", "buggy")
32
+ _git(repo, "config", "user.email", "corpus@proofbench.local")
33
+ _git(repo, "config", "user.name", "proofbench corpus")
34
+ (repo / "README.md").write_text(readme)
35
+ (repo / fname).write_text(buggy)
36
+ _git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "buggy baseline")
37
+ _git(repo, "checkout", "-q", "-b", "fixed")
38
+ (repo / fname).write_text(fixed)
39
+ _git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "fixed")
40
+ _git(repo, "checkout", "-q", "buggy")
41
+ print(f"built {name}")
42
+
43
+
44
+ def pdb_instance(task_id="3692_41"):
45
+ for off in (0, 100, 200):
46
+ for row in json.loads(_get(PDB.format(off=off)))["rows"]:
47
+ if row["row"]["task_id"] == task_id:
48
+ return row["row"]
49
+ raise SystemExit(f"PDB task {task_id} not found")
50
+
51
+
52
+ def repodebug_instance(repo_name="ivoanjo/gvl-tracing", error_type="wrong_return"):
53
+ for r in json.loads(_get(RD)):
54
+ if r["repo_name"] == repo_name and r["error_type"] == error_type:
55
+ return r
56
+ raise SystemExit("RepoDebug instance not found")
57
+
58
+
59
+ def apply_perturbation(upstream, diff):
60
+ """RepoDebug diff is original->buggy; '-' lines (originals) must exist in upstream."""
61
+ rem = "\n".join(l[1:] for l in diff.splitlines() if l[:1] == "-" and l[:3] != "---")
62
+ add = "\n".join(l[1:] for l in diff.splitlines() if l[:1] == "+" and l[:3] != "+++")
63
+ if rem not in upstream:
64
+ raise SystemExit("upstream drifted: perturbation no longer applies")
65
+ return upstream.replace(rem, add, 1)
66
+
67
+
68
+ def main():
69
+ p = pdb_instance()
70
+ build("pdb-multi-shortest-substring", "solution.py",
71
+ p["buggy_code"].rstrip() + "\n", p["gt_solution"].rstrip() + "\n",
72
+ "# shortest-matching-substring\n\nReconstructed from PDB-Multi "
73
+ f"({p['task_id']}, {p['bug_count']} injected bugs).\n\n## Spec\n\n"
74
+ + p["task_prompt"].strip() + "\n")
75
+
76
+ r = repodebug_instance()
77
+ upstream = _get(f"https://raw.githubusercontent.com/{r['repo_name']}/HEAD/{r['file_path']}")
78
+ build("repodebug-gvl-tracing", r["file_path"].split("/")[-1],
79
+ apply_perturbation(upstream, r["diff"]), upstream,
80
+ "# gvl-tracing (native extension excerpt)\n\nOne file from `ivoanjo/gvl-tracing` "
81
+ "(RepoDebug, EMNLP 2025). Exactly one functional bug was injected.\n")
82
+
83
+
84
+ if __name__ == "__main__":
85
+ main()
@@ -0,0 +1,23 @@
1
+ #!/usr/bin/env bash
2
+ # Clone every `ready` entry in manifest.yaml into ./repos, materializing a local
3
+ # default branch (at the pinned commit) and answer branch, with the broken default
4
+ # checked out. Re-runnable. Cloned repos are gitignored; only the manifest is tracked.
5
+ set -euo pipefail
6
+ HERE="$(cd "$(dirname "$0")" && pwd)"
7
+ cd "$HERE"
8
+ mkdir -p repos
9
+
10
+ python3 - <<'PY' | while IFS='|' read -r name url def ans pin; do
11
+ import yaml
12
+ for e in yaml.safe_load(open("manifest.yaml"))["ready"]:
13
+ print(f"{e['name']}|{e['url']}|{e['default_branch']}|{e['answer_branch']}|{e['pin']}")
14
+ PY
15
+ repo="repos/$name"
16
+ [ -d "$repo/.git" ] || git clone -q "$url" "$repo"
17
+ git -C "$repo" fetch -q origin
18
+ git -C "$repo" checkout -q --detach
19
+ git -C "$repo" branch -f "$def" "$pin"
20
+ git -C "$repo" branch -f "$ans" "origin/$ans"
21
+ git -C "$repo" checkout -q "$def"
22
+ echo "fetched $name @ ${pin:0:7} (default: $def, answer: $ans)"
23
+ done
@@ -0,0 +1,73 @@
1
+ # proofbench corpus manifest.
2
+ #
3
+ # This is the corpus the crg-debug evaluation used: 7 tasks total, 4 easy
4
+ # debugging-interview repos plus 3 Tier-1 hard cases. `ready` entries are public repos
5
+ # pinned to an exact commit; `bash fetch.sh` clones each into ./repos (gitignored) at its
6
+ # pin and materializes the broken default branch + the answer branch.
7
+ #
8
+ # Pins are recorded so a fetch is reproducible: the same manifest yields the same repos.
9
+
10
+ # The 4 easy repos, scored as the main sweep.
11
+ ready:
12
+ - name: debugging-challenge
13
+ url: https://github.com/josh-tobin/debugging-challenge.git
14
+ default_branch: master
15
+ answer_branch: working
16
+ pin: 21177bda401cd0128aefbce1b80e1bed98676ad7
17
+
18
+ - name: BugFix
19
+ url: https://github.com/IIIT-Delhi/BugFix.git
20
+ default_branch: main
21
+ answer_branch: solution
22
+ pin: 924895ea7dd687b409222c0ce91eec7f3c883f78
23
+
24
+ - name: ph2-raccoon-saloon-debugging-challenge
25
+ url: https://github.com/brewchetta/ph2-raccoon-saloon-debugging-challenge.git
26
+ default_branch: main
27
+ answer_branch: solution
28
+ pin: 72f52d697b6f40fb41b8f708a76cbc4540accf85
29
+
30
+ - name: react-debug-practice
31
+ url: https://github.com/ArohanD/react-debug-practice.git
32
+ default_branch: master
33
+ answer_branch: solution
34
+ pin: 7400054f7add23c2ec401152197c2056f2b3d7a6
35
+
36
+ # The 3 Tier-1 hard cases, scored individually. Real upstream provenance found (buggy =
37
+ # parent of the fix commit, fixed = the fix commit). Already constructed locally in
38
+ # dawson/ with buggy+fixed branches; URL-listable once hosted or reconstructed from upstream.
39
+ pending:
40
+ - name: cpython-timsort
41
+ upstream: https://github.com/python/cpython.git
42
+ provenance: "listsort merge_collapse run-stack invariant (same class as the Java TimSort bug)"
43
+ default_branch: buggy
44
+ answer_branch: fixed
45
+ status: "host dawson copy, or reconstruct buggy=fix^ / fixed=fix from upstream"
46
+ - name: cpython-asyncio-cond
47
+ upstream: https://github.com/python/cpython.git
48
+ provenance: "Condition.notify vs Task.cancel lost wakeup — python/cpython#112202"
49
+ default_branch: buggy
50
+ answer_branch: fixed
51
+ status: "host dawson copy, or reconstruct from the #112202 fix commit"
52
+ - name: sudo-baron-samedit
53
+ upstream: https://github.com/sudo-project/sudo.git
54
+ provenance: "CVE-2021-3156 heap overflow; fix 1f8638577d0c80a4ff864a2aad80a0d95488e9a8 (+ b301b46b79c6e2a76d530fa36d05992e74952ee8)"
55
+ default_branch: buggy
56
+ answer_branch: fixed
57
+ status: "host dawson copy, or reconstruct buggy=fix^ / fixed=fix from upstream"
58
+
59
+ # NOT part of the eval. Present in dawson/ but never scored. Listed for reference only;
60
+ # fetch.sh ignores this section.
61
+ extras:
62
+ - name: D-Bug
63
+ url: https://github.com/DSC-SIST/D-Bug.git
64
+ default_branch: main
65
+ answer_branch: solution
66
+ - name: matlab-debugging
67
+ url: https://github.com/AaltoSciComp/matlab-debugging.git
68
+ default_branch: main
69
+ answer_branch: solutions
70
+ - name: node-debug-challenge
71
+ url: https://github.com/AlecxMoritz/node-debug-challenge.git
72
+ default_branch: master
73
+ answer_branch: instructor
@@ -0,0 +1,38 @@
1
+ # Real corpus config: 3 easy repos + 2 hard benchmark-derived cases, run through crg-debug.
2
+ # Fetch first: bash corpora/fetch.sh
3
+ # Then: proofbench run --config corpora/proofbench.yaml
4
+ #
5
+ # The 3 Tier-1 hard cases (cpython-timsort, cpython-asyncio-cond, sudo-baron-samedit)
6
+ # are `pending` in manifest.yaml — add them here once hosted or reconstructed.
7
+ # To self-improve a skill, add `subject:` + `synth:` and set `models` to a weak leg.
8
+ #
9
+ # The 2 hard cases below are single instances reconstructed from published benchmarks, each
10
+ # materialized as a buggy->fixed git pair (branches `buggy` and `fixed`):
11
+ # - pdb-multi-shortest-substring: PDB-Multi task 3692_41 (Precise Debugging Benchmark,
12
+ # arXiv 2604.17338). 3 atomic bugs composed into one KMP/LPS string-matching algorithm.
13
+ # Complete ground truth by construction; scores edit-precision + bug-recall.
14
+ # - repodebug-gvl-tracing: RepoDebug (EMNLP Findings 2025) C `wrong_return` logic bug
15
+ # injected into a real Ruby native extension (ivoanjo/gvl-tracing); upstream = fixed ref.
16
+ # Both repos are built locally under repos/ (gitignored) — see corpora/build_hard.py to rebuild.
17
+ runner: claude
18
+ models: [haiku]
19
+ judge_model: opus
20
+ objective: macro_recall
21
+ concurrency: 3
22
+ max_turns: 200
23
+
24
+ judge: ../prompts/judge.md
25
+
26
+ # BugFix (already validated via the deterministic Workflow in Round 4) and the duplicate
27
+ # sf-wdi-24 debugging-challenges are excluded; these 3 are the active sweep.
28
+ #
29
+ # invoke drives the crg-debugger SUBAGENT (sequential, single in-context session), not the
30
+ # background /crg-debug Workflow: the subagent honors --model haiku and applies fixes
31
+ # synchronously, so capture sees a real git diff to grade. The background Workflow returns a
32
+ # detection report and lands no diff headless, and runs its own internal models.
33
+ corpus:
34
+ - { name: debugging-challenge, path: repos/debugging-challenge, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: master, answer_branch: working }
35
+ - { name: ph2-raccoon-saloon-debugging-challenge, path: repos/ph2-raccoon-saloon-debugging-challenge, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: main, answer_branch: solution }
36
+ - { name: react-debug-practice, path: repos/react-debug-practice, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: master, answer_branch: solution }
37
+ - { name: pdb-multi-shortest-substring, path: repos/pdb-multi-shortest-substring, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: buggy, answer_branch: fixed }
38
+ - { name: repodebug-gvl-tracing, path: repos/repodebug-gvl-tracing, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: buggy, answer_branch: fixed }
@@ -21,7 +21,7 @@ that *sounds* like a methodology. The only honest way to know is to plant bugs y
21
21
  answers to, run the skill, and grade what it changed against the known fix.
22
22
 
23
23
  So that is what I built: an eval over five debugging-interview repos, each with a ground-truth
24
- answer branch. Grade each fix with an LLM judge that compares the agent's diff to the reference
24
+ gold branch. Grade each fix with an LLM judge that compares the agent's diff to the reference
25
25
  diff and reports recall (did it fix the planted bugs) and precision (did it avoid breaking things).
26
26
  Then run the exact same harness on three driver models, opus and sonnet and haiku, to answer the
27
27
  question that actually matters: does the *method* carry the result, or just the model?
@@ -10,16 +10,16 @@ gates are exit codes the script reads, not model claims.
10
10
  ## Objective
11
11
 
12
12
  Measure how well the `crg-debug` skill finds and fixes planted bugs across a set of
13
- debugging-interview repos, grade its fixes against each repo's published answer branch,
13
+ debugging-interview repos, grade its fixes against each repo's published gold branch,
14
14
  and use the gap to auto-improve the skill (closed loop). Run the same harness with three
15
15
  different driver models to see whether the methodology, not just the model, carries the
16
16
  result.
17
17
 
18
18
  ## Method (what we did)
19
19
 
20
- * **Targets:** 5 debugging-interview repos, each with a ground-truth answer branch.
20
+ * **Targets:** 5 debugging-interview repos, each with a ground-truth gold branch.
21
21
 
22
- | Repo | Default | Answer | Bug flavor | Answer diff size |
22
+ | Repo | Default | Gold | Bug flavor | Gold diff size |
23
23
  |------|---------|--------|-----------|------------------|
24
24
  | debugging-challenge | master | working | ML correctness (reuse, softmax, reshape, double scaling) | 210 lines |
25
25
  | BugFix | main | solution | inverted/swapped logic (Python + C) | 420 lines |
@@ -22,7 +22,7 @@ judge: ../../prompts/judge.md
22
22
  synth: ../../prompts/synth.md
23
23
 
24
24
  corpus:
25
- # your bug-fix repos, each with a broken default branch and a fixed answer branch.
25
+ # your bug-fix repos, each with a broken default branch and a fixed solution branch.
26
26
  # see ../../CORPUS.md for how to pick non-leaky, discriminating tasks.
27
27
  - name: my-bug-repo
28
28
  path: /path/to/my-bug-repo
@@ -1,5 +1,10 @@
1
+ OUTPUT CONTRACT (read first): Respond with ONLY the JSON object specified at the end. Your
2
+ very first character must be `{`. Do NOT narrate your reasoning, do NOT restate the diffs, do
3
+ NOT use markdown fences. Reason silently; emit only the object. Narration before the object is
4
+ a grading failure: a long preamble can run past the output budget and truncate the JSON.
5
+
1
6
  You are grading an agent against a known answer. The agent was given a repository in a
2
- broken or incomplete state and asked to change it. The answer branch holds the ground-truth
7
+ broken or incomplete state and asked to change it. The solution branch holds the ground-truth
3
8
  change. Grade by comparing the agent's diff to the ground-truth diff.
4
9
 
5
10
  You are given three inputs below:
@@ -8,6 +8,14 @@ import subprocess
8
8
  _EXCLUDES = [":(exclude,glob)**/node_modules/**", ":(exclude,glob)**/.venv/**",
9
9
  ":(exclude,glob)**/__pycache__/**", ":(exclude,glob)**/.code-review-graph/**"]
10
10
 
11
+ # Multi-assistant MCP scaffolding that `code-review-graph build` scatters at the repo root to
12
+ # advertise its tools. It is deterministic tool scratch, never an agent fix, so it must not
13
+ # count against precision when grading a graph-driven debugger.
14
+ _EXCLUDES += [f":(exclude,glob){p}" for p in (
15
+ ".mcp.json", ".opencode.json", ".cursorrules", ".windsurfrules",
16
+ "AGENTS.md", "CLAUDE.md", "GEMINI.md", "QODER.md",
17
+ ".claude/**", ".cursor/**", ".gemini/**", ".kiro/**", ".qoder/**")]
18
+
11
19
 
12
20
  def _git(repo, *args):
13
21
  return subprocess.run(["git", "-C", str(repo), *args],
@@ -26,6 +34,9 @@ def ground_truth(repo, default_branch, answer_branch):
26
34
  return _git(repo, "diff", f"{default_branch}..{answer_branch}").stdout
27
35
 
28
36
 
29
- def capture(repo):
37
+ def capture(repo, baseline):
38
+ """Diff against the buggy baseline ref, not the working tree: agents may commit their
39
+ fixes. reset() detaches HEAD, so the baseline ref stays pinned while agent commits move
40
+ HEAD, and this diff captures the fix whether it was committed or left uncommitted."""
30
41
  _git(repo, "add", "-N", ".")
31
- return _git(repo, "diff", "--", ".", *_EXCLUDES).stdout
42
+ return _git(repo, "diff", baseline, "--", ".", *_EXCLUDES).stdout
@@ -39,3 +39,7 @@ def main():
39
39
  except HarnessError as err:
40
40
  print(f"\033[31m[proofbench] {err}\033[0m", file=sys.stderr)
41
41
  sys.exit(1)
42
+
43
+
44
+ if __name__ == "__main__":
45
+ main()
@@ -24,19 +24,48 @@ def _ground_truth(cfg, run_dir):
24
24
  def _run_and_grade(cfg, task, model, gt, out_dir):
25
25
  readme, answer = gt[task.name]
26
26
  capture.reset(task.path, task.default_branch)
27
- runner.run(cfg.runner, task.invoke, model, cfg.max_turns, task.path, out_dir / "run.log")
28
- agent_diff = capture.capture(task.path)
27
+ meta = runner.run(cfg.runner, task.invoke, model, cfg.max_turns, task.path, out_dir / "run.log")
28
+ cost = f", ${meta['cost_usd']:.3f}" if meta["cost_usd"] is not None else ""
29
+ log(f"[{model}] {task.name}: {meta['seconds']}s{cost}"
30
+ + (f", {meta['turns']} turns" if meta["turns"] else ""))
31
+ agent_diff = capture.capture(task.path, task.default_branch)
29
32
  (out_dir / "agent.diff").write_text(agent_diff)
30
- grade = judge.grade(cfg.judge_prompt, readme, answer, agent_diff, cfg.judge_model)
31
- grade["task"] = task.name
33
+ if not agent_diff.strip():
34
+ log(f"\033[33m[{model}] {task.name}: EMPTY diff — agent changed nothing; "
35
+ f"excluded from macro (check the invoke if this is unexpected)\033[0m")
36
+ grade = {"task": task.name, "empty_capture": True}
37
+ else:
38
+ grade = judge.grade(cfg.judge_prompt, readme, answer, agent_diff, cfg.judge_model)
39
+ grade["task"] = task.name
40
+ grade["_meta"] = meta
32
41
  (out_dir / "grade.json").write_text(json.dumps(grade, indent=2))
33
42
  return grade
34
43
 
35
44
 
45
+ def _resources(grades):
46
+ """Sum cost + wall-clock across the whole leg, empty captures included: a run that
47
+ changed nothing still spent money and time. cost_usd is None when no leg task reported
48
+ it (a non-claude runner)."""
49
+ metas = [g.get("_meta", {}) for g in grades]
50
+ costs = [m["cost_usd"] for m in metas if m.get("cost_usd") is not None]
51
+ return {"seconds": round(sum(m.get("seconds") or 0 for m in metas), 1),
52
+ "cost_usd": round(sum(costs), 4) if costs else None,
53
+ "turns": sum(m.get("turns") or 0 for m in metas) or None}
54
+
55
+
36
56
  def aggregate(grades):
37
- shared = set.intersection(*(set(judge.numeric_fields(g)) for g in grades))
38
- macro = {f"macro_{f}": sum(g[f] for g in grades) / len(grades) for f in shared}
39
- return {**macro, "tasks": {g["task"]: judge.numeric_fields(g) for g in grades}}
57
+ """Macro-average the shared numeric fields. Empty-capture tasks carry no metrics: they
58
+ are excluded from the average and listed separately so they never become a silent zero."""
59
+ graded = [g for g in grades if not g.get("empty_capture")]
60
+ out = {"tasks": {g["task"]: judge.numeric_fields(g) for g in grades}}
61
+ empty = [g["task"] for g in grades if g.get("empty_capture")]
62
+ if empty:
63
+ out["empty_captures"] = empty
64
+ if graded:
65
+ shared = set.intersection(*(set(judge.numeric_fields(g)) for g in graded))
66
+ out.update({f"macro_{f}": sum(g[f] for g in graded) / len(graded) for f in shared})
67
+ out["resources"] = _resources(grades)
68
+ return out
40
69
 
41
70
 
42
71
  def bench(cfg, run_dir, models):
@@ -2,7 +2,9 @@
2
2
 
3
3
  A built-in name (``claude``) maps to a command builder. Any other value is treated as
4
4
  a path to an executable invoked as ``<runner> <invoke> <model>`` — bring your own agent."""
5
+ import json
5
6
  import subprocess
7
+ import time
6
8
 
7
9
 
8
10
  def _claude(invoke, model, max_turns):
@@ -14,12 +16,27 @@ def _claude(invoke, model, max_turns):
14
16
  BUILTINS = {"claude": _claude}
15
17
 
16
18
 
19
+ def _claude_meta(stdout):
20
+ """Pull cost + turns from a ``claude --output-format json`` envelope. Observability,
21
+ not a graded metric, so this never dies: a custom runner or malformed envelope yields
22
+ None and the wall-clock seconds still stand."""
23
+ try:
24
+ env = json.loads(stdout[stdout.index("{"):stdout.rindex("}") + 1])
25
+ return env.get("total_cost_usd"), env.get("num_turns")
26
+ except (ValueError, json.JSONDecodeError):
27
+ return None, None
28
+
29
+
17
30
  def run(runner, invoke, model, max_turns, cwd, log_path):
18
- """Execute one headless attempt in ``cwd``; capture all output to ``log_path``.
31
+ """Execute one headless attempt in ``cwd``; capture all output to ``log_path`` and
32
+ return its ``{seconds, cost_usd, turns}``.
19
33
 
20
34
  The agent's exit status is advisory — what gets graded is the artifact it leaves
21
35
  behind (see capture), never the agent's own claim of success."""
22
36
  argv = BUILTINS[runner](invoke, model, max_turns) if runner in BUILTINS \
23
37
  else [runner, invoke, model]
24
- with open(log_path, "w") as log:
25
- subprocess.run(argv, cwd=cwd, stdout=log, stderr=subprocess.STDOUT, text=True)
38
+ start = time.time()
39
+ proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
40
+ open(log_path, "w").write(proc.stdout + proc.stderr)
41
+ cost, turns = _claude_meta(proc.stdout) if runner in BUILTINS else (None, None)
42
+ return {"seconds": round(time.time() - start, 1), "cost_usd": cost, "turns": turns}
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "proofbench"
3
- version = "0.1.0"
3
+ version = "0.2.1"
4
4
  description = "Config-driven eval harness that grades and self-improves headless agent skills against ground-truth corpora"
5
5
  readme = "README.md"
6
6
  license = { text = "MIT" }
@@ -21,6 +21,7 @@ def test_bench_wires_end_to_end(tmp_path, monkeypatch):
21
21
  def fake_run(runner_name, invoke, model, max_turns, cwd, log_path):
22
22
  (Path(cwd) / "lib.py").write_text(FIXED)
23
23
  Path(log_path).write_text("stub")
24
+ return {"seconds": 1.0, "cost_usd": 0.01, "turns": 2}
24
25
  monkeypatch.setattr(runner, "run", fake_run)
25
26
 
26
27
  # stub judge: credit recall when the agent actually changed something
@@ -32,5 +33,6 @@ def test_bench_wires_end_to_end(tmp_path, monkeypatch):
32
33
  leg = board[cfg.models[0]]
33
34
  assert leg["macro_recall"] == 1.0
34
35
  assert leg["macro_precision"] == 1.0
36
+ assert leg["resources"] == {"seconds": 2.0, "cost_usd": 0.02, "turns": 4}
35
37
  saved = json.loads((tmp_path / "scoreboard.json").read_text())
36
38
  assert set(saved[cfg.models[0]]["tasks"]) == {"primes", "ranges"}
@@ -1,51 +0,0 @@
1
- # Authoring a corpus
2
-
3
- A task in proofbench is a git repository with two branches: a `default_branch` in a broken or
4
- incomplete state, and an `answer_branch` holding the ground-truth change. The judge grades the
5
- agent's diff against `default_branch..answer_branch`. The quality of your eval is mostly the
6
- quality of these tasks. Two rules carry the most weight.
7
-
8
- ## Do not use famous bugs
9
-
10
- A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
11
- does, you are measuring what it memorized, not how it debugs. Prefer subtle, real defects where
12
- the correct reasoning is about an invariant, a race, or a contract, and where the surrounding
13
- code gives no hint. The discriminating tasks are the ones a model has to actually reason through,
14
- not pattern-match.
15
-
16
- ## Do not telegraph the answer
17
-
18
- The judge reads the repo's README. If the README says "this repo has an intentional bug for
19
- practice," or the buggy line carries a `# Buggy` comment, the judge (and the agent) read the tell
20
- instead of the contract. In one real run this flipped genuine contract violations into "intended
21
- design" and quietly corrupted the grades. So:
22
-
23
- - Write the README as a real library would: state what each function is supposed to do (its
24
- contract), never that it is wrong.
25
- - Remove comments that point at the defect. Keep the buggy line exactly where it is.
26
- - The only place the answer should exist is the `answer_branch`.
27
-
28
- ## Practical shape
29
-
30
- - One defect (or one coherent set) per task keeps recall and precision interpretable.
31
- - Keep `default_branch` checked out as the repo's default, so a fresh clone is in the broken
32
- state the agent will see.
33
- - The bundled `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each a
34
- `buggy` branch and a `fixed` branch, README stating the contract, no tells.
35
-
36
- ## Pointing the harness at your corpus
37
-
38
- Each task in the config names the repo path and its two branches:
39
-
40
- ```yaml
41
- corpus:
42
- - name: my-task
43
- path: /abs/or/relative/path/to/repo
44
- invoke: "Find and fix the bug in this repository."
45
- default_branch: buggy
46
- answer_branch: fixed
47
- ```
48
-
49
- Large or licensed corpora should not be committed into your harness repo. Reference them by a
50
- pinned git URL and clone them on setup, so the data stays out of your tree and its provenance
51
- stays exact.
File without changes
File without changes
File without changes