proofbench 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- proofbench-0.2.1/CORPUS.md +100 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/PKG-INFO +3 -3
- {proofbench-0.2.0 → proofbench-0.2.1}/README.md +2 -2
- proofbench-0.2.1/corpora/build_hard.py +85 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/corpora/proofbench.yaml +12 -1
- {proofbench-0.2.0 → proofbench-0.2.1}/docs/crg-eval-journey.md +1 -1
- {proofbench-0.2.0 → proofbench-0.2.1}/docs/crg-eval-results.md +3 -3
- {proofbench-0.2.0 → proofbench-0.2.1}/examples/crg-debug/README.md +1 -1
- {proofbench-0.2.0 → proofbench-0.2.1}/prompts/judge.md +6 -1
- {proofbench-0.2.0 → proofbench-0.2.1}/pyproject.toml +1 -1
- proofbench-0.2.0/CORPUS.md +0 -51
- {proofbench-0.2.0 → proofbench-0.2.1}/.github/workflows/release.yml +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/.gitignore +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/LICENSE +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/METHODOLOGY.md +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/corpora/fetch.sh +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/corpora/manifest.yaml +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/examples/proofbench.demo.yaml +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/examples/sample-corpus/build.sh +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/prompts/synth.md +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/__init__.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/capture.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/cli.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/config.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/engine.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/judge.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/optimize.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/runner.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/proofbench/util.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/tests/test_engine.py +0 -0
- {proofbench-0.2.0 → proofbench-0.2.1}/tests/test_failloud.py +0 -0
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Authoring a corpus
|
|
2
|
+
|
|
3
|
+
A **task** is a git fixture repository with two branches: a **default branch** (the broken or
|
|
4
|
+
incomplete state the agent starts from) and a **solution branch** (the held-out ground truth). The
|
|
5
|
+
literal branch names vary per repo — `main`/`master`, `buggy`, `solution`, `working` all appear in
|
|
6
|
+
the existing corpus — so the config's `default_branch` and `answer_branch` fields name them for the
|
|
7
|
+
harness (the solution branch is the `answer_branch` field); you point those fields at whatever the repo
|
|
8
|
+
actually calls its branches. The
|
|
9
|
+
**reference solution** is the diff between the two, and the judge grades the agent's patch against
|
|
10
|
+
it. Your eval is only as good as these tasks, so two properties carry the most weight: every task
|
|
11
|
+
must be **discriminative** (it separates a capable agent from a weak one) and **leak-free** (nothing
|
|
12
|
+
but the solution branch reveals the fix).
|
|
13
|
+
|
|
14
|
+
## Anatomy of a task
|
|
15
|
+
|
|
16
|
+
```mermaid
|
|
17
|
+
flowchart LR
|
|
18
|
+
subgraph repo["task — git fixture repo"]
|
|
19
|
+
A["default_branch<br/>(e.g. main / buggy)"]
|
|
20
|
+
B["answer_branch<br/>(e.g. solution / fixed)"]
|
|
21
|
+
end
|
|
22
|
+
A -- "reference solution<br/>git diff default_branch..answer_branch" --> B
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The agent never sees `answer_branch`. It is checked out on `default_branch`, asked to fix the
|
|
26
|
+
repo, and graded on how close its patch comes to the reference solution.
|
|
27
|
+
|
|
28
|
+
## How a task is scored
|
|
29
|
+
|
|
30
|
+
```mermaid
|
|
31
|
+
flowchart TD
|
|
32
|
+
reset["reset working tree → default_branch"] --> run["runner invokes the agent"]
|
|
33
|
+
run --> cap["capture: git diff of the working tree<br/>= agent patch"]
|
|
34
|
+
gt["ground truth:<br/>default_branch..answer_branch"] --> judge
|
|
35
|
+
cap --> judge{{"LLM judge"}}
|
|
36
|
+
judge --> out["recall · precision"]
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
**Recall** is how much of the reference solution the agent reproduced; **precision** is how much
|
|
40
|
+
of the agent's patch was warranted rather than collateral damage. A clean, single-purpose task
|
|
41
|
+
keeps both metrics interpretable.
|
|
42
|
+
|
|
43
|
+
## Designing out answer leakage
|
|
44
|
+
|
|
45
|
+
A task leaks when the agent or the judge can reach the answer without doing the work. There are
|
|
46
|
+
two distinct channels, and a good task closes both.
|
|
47
|
+
|
|
48
|
+
```mermaid
|
|
49
|
+
flowchart TD
|
|
50
|
+
task["answer leakage"]
|
|
51
|
+
task --> c["Contamination<br/>the model already memorized the fix"]
|
|
52
|
+
task --> l["Label leakage<br/>the repo points at the fix"]
|
|
53
|
+
c --> cc["famous CVEs · textbook off-by-ones<br/>→ measures memory, not method"]
|
|
54
|
+
l --> ll["'intentional bug' READMEs · '# Buggy' comments<br/>→ judge grades the tell, not the contract"]
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### Contamination: avoid famous defects
|
|
58
|
+
|
|
59
|
+
A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
|
|
60
|
+
does, you are measuring training-set recall, not debugging skill. Prefer subtle, real defects
|
|
61
|
+
whose fix turns on an invariant, a race, or a contract, and where the surrounding code offers no
|
|
62
|
+
hint. Discriminative tasks are the ones a model must actually reason through, not pattern-match.
|
|
63
|
+
|
|
64
|
+
### Label leakage: do not telegraph the answer
|
|
65
|
+
|
|
66
|
+
The judge reads the repo's README. If the README says "this repo has an intentional bug for
|
|
67
|
+
practice," or the defective line carries a `# Buggy` comment, both the agent and the judge read
|
|
68
|
+
the tell instead of the contract. In one real run this flipped genuine contract violations into
|
|
69
|
+
"intended design" and silently corrupted the grades. So:
|
|
70
|
+
|
|
71
|
+
- Write the README as a real library would: state each function's **contract** (what it is
|
|
72
|
+
supposed to do), never that it is wrong.
|
|
73
|
+
- Strip comments that point at the defect. Leave the defective line exactly where it is.
|
|
74
|
+
- The fix exists in exactly one place: the `answer_branch`.
|
|
75
|
+
|
|
76
|
+
## Practical shape
|
|
77
|
+
|
|
78
|
+
- One defect (or one coherent set) per task keeps recall and precision interpretable.
|
|
79
|
+
- Leave `default_branch` checked out as the repo's default, so a fresh clone lands in the broken
|
|
80
|
+
state the agent will see.
|
|
81
|
+
- `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each with a `buggy`
|
|
82
|
+
default branch and a `fixed` solution branch, a README stating the contract, and no tells.
|
|
83
|
+
|
|
84
|
+
## Wiring the harness to your corpus
|
|
85
|
+
|
|
86
|
+
Each task in the config names the repo path, the agent invocation, and the two branches:
|
|
87
|
+
|
|
88
|
+
```yaml
|
|
89
|
+
corpus:
|
|
90
|
+
- name: my-task
|
|
91
|
+
path: /abs/or/relative/path/to/repo
|
|
92
|
+
invoke: "Find and fix the bug in this repository."
|
|
93
|
+
default_branch: buggy
|
|
94
|
+
answer_branch: fixed
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## External and licensed corpora
|
|
98
|
+
|
|
99
|
+
Do not commit large or licensed corpora into your harness repo. Reference them by a pinned git
|
|
100
|
+
URL and clone them on setup, so the data stays out of your tree and its provenance stays exact.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: proofbench
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.1
|
|
4
4
|
Summary: Config-driven eval harness that grades and self-improves headless agent skills against ground-truth corpora
|
|
5
5
|
Project-URL: Homepage, https://github.com/CodeBlackwell/proofbench
|
|
6
6
|
Author: LeChristopher Blackwell
|
|
@@ -90,14 +90,14 @@ the harness comparing two numbers, never the model's own claim of success.
|
|
|
90
90
|
|
|
91
91
|
- **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
|
|
92
92
|
`<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
|
|
93
|
-
- **Capture**: the git-diff default resets a repo to its
|
|
93
|
+
- **Capture**: the git-diff default resets a repo to its default branch and snapshots what the
|
|
94
94
|
agent changed, excluding dependency and cache trees so they never pollute or balloon the
|
|
95
95
|
judge prompt.
|
|
96
96
|
|
|
97
97
|
## Bring your own corpus
|
|
98
98
|
|
|
99
99
|
The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
|
|
100
|
-
(each with a broken default branch and a fixed
|
|
100
|
+
(each with a broken default branch and a fixed solution branch). See [CORPUS.md](CORPUS.md) for
|
|
101
101
|
how to curate tasks that actually discriminate and do not leak their answers to the judge.
|
|
102
102
|
|
|
103
103
|
## License
|
|
@@ -77,14 +77,14 @@ the harness comparing two numbers, never the model's own claim of success.
|
|
|
77
77
|
|
|
78
78
|
- **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
|
|
79
79
|
`<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
|
|
80
|
-
- **Capture**: the git-diff default resets a repo to its
|
|
80
|
+
- **Capture**: the git-diff default resets a repo to its default branch and snapshots what the
|
|
81
81
|
agent changed, excluding dependency and cache trees so they never pollute or balloon the
|
|
82
82
|
judge prompt.
|
|
83
83
|
|
|
84
84
|
## Bring your own corpus
|
|
85
85
|
|
|
86
86
|
The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
|
|
87
|
-
(each with a broken default branch and a fixed
|
|
87
|
+
(each with a broken default branch and a fixed solution branch). See [CORPUS.md](CORPUS.md) for
|
|
88
88
|
how to curate tasks that actually discriminate and do not leak their answers to the judge.
|
|
89
89
|
|
|
90
90
|
## License
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
"""Rebuild the two hard, benchmark-derived corpus cases as buggy->fixed git pairs.
|
|
2
|
+
|
|
3
|
+
Each case is one published-benchmark instance materialized locally (repos/ is gitignored):
|
|
4
|
+
pdb-multi-shortest-substring PDB-Multi 3692_41 (arXiv 2604.17338) — 3 bugs, complete GT.
|
|
5
|
+
repodebug-gvl-tracing RepoDebug (EMNLP 2025) C wrong_return; upstream = fixed ref.
|
|
6
|
+
|
|
7
|
+
Run: python3 corpora/build_hard.py
|
|
8
|
+
"""
|
|
9
|
+
import json
|
|
10
|
+
import subprocess
|
|
11
|
+
import urllib.request
|
|
12
|
+
from pathlib import Path
|
|
13
|
+
|
|
14
|
+
REPOS = Path(__file__).resolve().parent / "repos"
|
|
15
|
+
PDB = "https://datasets-server.huggingface.co/rows?dataset=Precise-Debugging-Benchmarking%2FPDB-Multi&config=default&split=test&offset={off}&length=100"
|
|
16
|
+
RD = "https://raw.githubusercontent.com/BUAA-IRIP-LLM/RepoDebug/main/data/c.json"
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
def _get(url):
|
|
20
|
+
return urllib.request.urlopen(url, timeout=30).read().decode("utf-8", "replace")
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
def _git(repo, *args):
|
|
24
|
+
subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def build(name, fname, buggy, fixed, readme):
|
|
28
|
+
repo = REPOS / name
|
|
29
|
+
subprocess.run(["rm", "-rf", str(repo)], check=True)
|
|
30
|
+
repo.mkdir(parents=True)
|
|
31
|
+
_git(repo, "init", "-q", "-b", "buggy")
|
|
32
|
+
_git(repo, "config", "user.email", "corpus@proofbench.local")
|
|
33
|
+
_git(repo, "config", "user.name", "proofbench corpus")
|
|
34
|
+
(repo / "README.md").write_text(readme)
|
|
35
|
+
(repo / fname).write_text(buggy)
|
|
36
|
+
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "buggy baseline")
|
|
37
|
+
_git(repo, "checkout", "-q", "-b", "fixed")
|
|
38
|
+
(repo / fname).write_text(fixed)
|
|
39
|
+
_git(repo, "add", "-A"); _git(repo, "commit", "-q", "-m", "fixed")
|
|
40
|
+
_git(repo, "checkout", "-q", "buggy")
|
|
41
|
+
print(f"built {name}")
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def pdb_instance(task_id="3692_41"):
|
|
45
|
+
for off in (0, 100, 200):
|
|
46
|
+
for row in json.loads(_get(PDB.format(off=off)))["rows"]:
|
|
47
|
+
if row["row"]["task_id"] == task_id:
|
|
48
|
+
return row["row"]
|
|
49
|
+
raise SystemExit(f"PDB task {task_id} not found")
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
def repodebug_instance(repo_name="ivoanjo/gvl-tracing", error_type="wrong_return"):
|
|
53
|
+
for r in json.loads(_get(RD)):
|
|
54
|
+
if r["repo_name"] == repo_name and r["error_type"] == error_type:
|
|
55
|
+
return r
|
|
56
|
+
raise SystemExit("RepoDebug instance not found")
|
|
57
|
+
|
|
58
|
+
|
|
59
|
+
def apply_perturbation(upstream, diff):
|
|
60
|
+
"""RepoDebug diff is original->buggy; '-' lines (originals) must exist in upstream."""
|
|
61
|
+
rem = "\n".join(l[1:] for l in diff.splitlines() if l[:1] == "-" and l[:3] != "---")
|
|
62
|
+
add = "\n".join(l[1:] for l in diff.splitlines() if l[:1] == "+" and l[:3] != "+++")
|
|
63
|
+
if rem not in upstream:
|
|
64
|
+
raise SystemExit("upstream drifted: perturbation no longer applies")
|
|
65
|
+
return upstream.replace(rem, add, 1)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
def main():
|
|
69
|
+
p = pdb_instance()
|
|
70
|
+
build("pdb-multi-shortest-substring", "solution.py",
|
|
71
|
+
p["buggy_code"].rstrip() + "\n", p["gt_solution"].rstrip() + "\n",
|
|
72
|
+
"# shortest-matching-substring\n\nReconstructed from PDB-Multi "
|
|
73
|
+
f"({p['task_id']}, {p['bug_count']} injected bugs).\n\n## Spec\n\n"
|
|
74
|
+
+ p["task_prompt"].strip() + "\n")
|
|
75
|
+
|
|
76
|
+
r = repodebug_instance()
|
|
77
|
+
upstream = _get(f"https://raw.githubusercontent.com/{r['repo_name']}/HEAD/{r['file_path']}")
|
|
78
|
+
build("repodebug-gvl-tracing", r["file_path"].split("/")[-1],
|
|
79
|
+
apply_perturbation(upstream, r["diff"]), upstream,
|
|
80
|
+
"# gvl-tracing (native extension excerpt)\n\nOne file from `ivoanjo/gvl-tracing` "
|
|
81
|
+
"(RepoDebug, EMNLP 2025). Exactly one functional bug was injected.\n")
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
if __name__ == "__main__":
|
|
85
|
+
main()
|
|
@@ -1,10 +1,19 @@
|
|
|
1
|
-
# Real corpus config: 3 easy repos run through crg-debug
|
|
1
|
+
# Real corpus config: 3 easy repos + 2 hard benchmark-derived cases, run through crg-debug.
|
|
2
2
|
# Fetch first: bash corpora/fetch.sh
|
|
3
3
|
# Then: proofbench run --config corpora/proofbench.yaml
|
|
4
4
|
#
|
|
5
5
|
# The 3 Tier-1 hard cases (cpython-timsort, cpython-asyncio-cond, sudo-baron-samedit)
|
|
6
6
|
# are `pending` in manifest.yaml — add them here once hosted or reconstructed.
|
|
7
7
|
# To self-improve a skill, add `subject:` + `synth:` and set `models` to a weak leg.
|
|
8
|
+
#
|
|
9
|
+
# The 2 hard cases below are single instances reconstructed from published benchmarks, each
|
|
10
|
+
# materialized as a buggy->fixed git pair (branches `buggy` and `fixed`):
|
|
11
|
+
# - pdb-multi-shortest-substring: PDB-Multi task 3692_41 (Precise Debugging Benchmark,
|
|
12
|
+
# arXiv 2604.17338). 3 atomic bugs composed into one KMP/LPS string-matching algorithm.
|
|
13
|
+
# Complete ground truth by construction; scores edit-precision + bug-recall.
|
|
14
|
+
# - repodebug-gvl-tracing: RepoDebug (EMNLP Findings 2025) C `wrong_return` logic bug
|
|
15
|
+
# injected into a real Ruby native extension (ivoanjo/gvl-tracing); upstream = fixed ref.
|
|
16
|
+
# Both repos are built locally under repos/ (gitignored) — see corpora/build_hard.py to rebuild.
|
|
8
17
|
runner: claude
|
|
9
18
|
models: [haiku]
|
|
10
19
|
judge_model: opus
|
|
@@ -25,3 +34,5 @@ corpus:
|
|
|
25
34
|
- { name: debugging-challenge, path: repos/debugging-challenge, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: master, answer_branch: working }
|
|
26
35
|
- { name: ph2-raccoon-saloon-debugging-challenge, path: repos/ph2-raccoon-saloon-debugging-challenge, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: main, answer_branch: solution }
|
|
27
36
|
- { name: react-debug-practice, path: repos/react-debug-practice, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: master, answer_branch: solution }
|
|
37
|
+
- { name: pdb-multi-shortest-substring, path: repos/pdb-multi-shortest-substring, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: buggy, answer_branch: fixed }
|
|
38
|
+
- { name: repodebug-gvl-tracing, path: repos/repodebug-gvl-tracing, invoke: "Use the crg-debugger subagent (Task tool, subagent_type 'crg-debug:crg-debugger') to find and fix every functional bug in this repository. Apply all fixes directly to the source files. Do not ask for confirmation.", default_branch: buggy, answer_branch: fixed }
|
|
@@ -21,7 +21,7 @@ that *sounds* like a methodology. The only honest way to know is to plant bugs y
|
|
|
21
21
|
answers to, run the skill, and grade what it changed against the known fix.
|
|
22
22
|
|
|
23
23
|
So that is what I built: an eval over five debugging-interview repos, each with a ground-truth
|
|
24
|
-
|
|
24
|
+
gold branch. Grade each fix with an LLM judge that compares the agent's diff to the reference
|
|
25
25
|
diff and reports recall (did it fix the planted bugs) and precision (did it avoid breaking things).
|
|
26
26
|
Then run the exact same harness on three driver models, opus and sonnet and haiku, to answer the
|
|
27
27
|
question that actually matters: does the *method* carry the result, or just the model?
|
|
@@ -10,16 +10,16 @@ gates are exit codes the script reads, not model claims.
|
|
|
10
10
|
## Objective
|
|
11
11
|
|
|
12
12
|
Measure how well the `crg-debug` skill finds and fixes planted bugs across a set of
|
|
13
|
-
debugging-interview repos, grade its fixes against each repo's published
|
|
13
|
+
debugging-interview repos, grade its fixes against each repo's published gold branch,
|
|
14
14
|
and use the gap to auto-improve the skill (closed loop). Run the same harness with three
|
|
15
15
|
different driver models to see whether the methodology, not just the model, carries the
|
|
16
16
|
result.
|
|
17
17
|
|
|
18
18
|
## Method (what we did)
|
|
19
19
|
|
|
20
|
-
* **Targets:** 5 debugging-interview repos, each with a ground-truth
|
|
20
|
+
* **Targets:** 5 debugging-interview repos, each with a ground-truth gold branch.
|
|
21
21
|
|
|
22
|
-
| Repo | Default |
|
|
22
|
+
| Repo | Default | Gold | Bug flavor | Gold diff size |
|
|
23
23
|
|------|---------|--------|-----------|------------------|
|
|
24
24
|
| debugging-challenge | master | working | ML correctness (reuse, softmax, reshape, double scaling) | 210 lines |
|
|
25
25
|
| BugFix | main | solution | inverted/swapped logic (Python + C) | 420 lines |
|
|
@@ -22,7 +22,7 @@ judge: ../../prompts/judge.md
|
|
|
22
22
|
synth: ../../prompts/synth.md
|
|
23
23
|
|
|
24
24
|
corpus:
|
|
25
|
-
# your bug-fix repos, each with a broken default branch and a fixed
|
|
25
|
+
# your bug-fix repos, each with a broken default branch and a fixed solution branch.
|
|
26
26
|
# see ../../CORPUS.md for how to pick non-leaky, discriminating tasks.
|
|
27
27
|
- name: my-bug-repo
|
|
28
28
|
path: /path/to/my-bug-repo
|
|
@@ -1,5 +1,10 @@
|
|
|
1
|
+
OUTPUT CONTRACT (read first): Respond with ONLY the JSON object specified at the end. Your
|
|
2
|
+
very first character must be `{`. Do NOT narrate your reasoning, do NOT restate the diffs, do
|
|
3
|
+
NOT use markdown fences. Reason silently; emit only the object. Narration before the object is
|
|
4
|
+
a grading failure: a long preamble can run past the output budget and truncate the JSON.
|
|
5
|
+
|
|
1
6
|
You are grading an agent against a known answer. The agent was given a repository in a
|
|
2
|
-
broken or incomplete state and asked to change it. The
|
|
7
|
+
broken or incomplete state and asked to change it. The solution branch holds the ground-truth
|
|
3
8
|
change. Grade by comparing the agent's diff to the ground-truth diff.
|
|
4
9
|
|
|
5
10
|
You are given three inputs below:
|
proofbench-0.2.0/CORPUS.md
DELETED
|
@@ -1,51 +0,0 @@
|
|
|
1
|
-
# Authoring a corpus
|
|
2
|
-
|
|
3
|
-
A task in proofbench is a git repository with two branches: a `default_branch` in a broken or
|
|
4
|
-
incomplete state, and an `answer_branch` holding the ground-truth change. The judge grades the
|
|
5
|
-
agent's diff against `default_branch..answer_branch`. The quality of your eval is mostly the
|
|
6
|
-
quality of these tasks. Two rules carry the most weight.
|
|
7
|
-
|
|
8
|
-
## Do not use famous bugs
|
|
9
|
-
|
|
10
|
-
A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
|
|
11
|
-
does, you are measuring what it memorized, not how it debugs. Prefer subtle, real defects where
|
|
12
|
-
the correct reasoning is about an invariant, a race, or a contract, and where the surrounding
|
|
13
|
-
code gives no hint. The discriminating tasks are the ones a model has to actually reason through,
|
|
14
|
-
not pattern-match.
|
|
15
|
-
|
|
16
|
-
## Do not telegraph the answer
|
|
17
|
-
|
|
18
|
-
The judge reads the repo's README. If the README says "this repo has an intentional bug for
|
|
19
|
-
practice," or the buggy line carries a `# Buggy` comment, the judge (and the agent) read the tell
|
|
20
|
-
instead of the contract. In one real run this flipped genuine contract violations into "intended
|
|
21
|
-
design" and quietly corrupted the grades. So:
|
|
22
|
-
|
|
23
|
-
- Write the README as a real library would: state what each function is supposed to do (its
|
|
24
|
-
contract), never that it is wrong.
|
|
25
|
-
- Remove comments that point at the defect. Keep the buggy line exactly where it is.
|
|
26
|
-
- The only place the answer should exist is the `answer_branch`.
|
|
27
|
-
|
|
28
|
-
## Practical shape
|
|
29
|
-
|
|
30
|
-
- One defect (or one coherent set) per task keeps recall and precision interpretable.
|
|
31
|
-
- Keep `default_branch` checked out as the repo's default, so a fresh clone is in the broken
|
|
32
|
-
state the agent will see.
|
|
33
|
-
- The bundled `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each a
|
|
34
|
-
`buggy` branch and a `fixed` branch, README stating the contract, no tells.
|
|
35
|
-
|
|
36
|
-
## Pointing the harness at your corpus
|
|
37
|
-
|
|
38
|
-
Each task in the config names the repo path and its two branches:
|
|
39
|
-
|
|
40
|
-
```yaml
|
|
41
|
-
corpus:
|
|
42
|
-
- name: my-task
|
|
43
|
-
path: /abs/or/relative/path/to/repo
|
|
44
|
-
invoke: "Find and fix the bug in this repository."
|
|
45
|
-
default_branch: buggy
|
|
46
|
-
answer_branch: fixed
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
Large or licensed corpora should not be committed into your harness repo. Reference them by a
|
|
50
|
-
pinned git URL and clone them on setup, so the data stays out of your tree and its provenance
|
|
51
|
-
stays exact.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|