proofbench 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- proofbench-0.1.0/.github/workflows/release.yml +25 -0
- proofbench-0.1.0/.gitignore +9 -0
- proofbench-0.1.0/CORPUS.md +51 -0
- proofbench-0.1.0/LICENSE +21 -0
- proofbench-0.1.0/METHODOLOGY.md +62 -0
- proofbench-0.1.0/PKG-INFO +105 -0
- proofbench-0.1.0/README.md +92 -0
- proofbench-0.1.0/docs/crg-eval-journey.md +271 -0
- proofbench-0.1.0/docs/crg-eval-results.md +864 -0
- proofbench-0.1.0/examples/crg-debug/README.md +41 -0
- proofbench-0.1.0/examples/proofbench.demo.yaml +27 -0
- proofbench-0.1.0/examples/sample-corpus/build.sh +59 -0
- proofbench-0.1.0/prompts/judge.md +27 -0
- proofbench-0.1.0/prompts/synth.md +20 -0
- proofbench-0.1.0/proofbench/__init__.py +2 -0
- proofbench-0.1.0/proofbench/capture.py +31 -0
- proofbench-0.1.0/proofbench/cli.py +41 -0
- proofbench-0.1.0/proofbench/config.py +68 -0
- proofbench-0.1.0/proofbench/engine.py +63 -0
- proofbench-0.1.0/proofbench/judge.py +20 -0
- proofbench-0.1.0/proofbench/optimize.py +58 -0
- proofbench-0.1.0/proofbench/runner.py +25 -0
- proofbench-0.1.0/proofbench/util.py +49 -0
- proofbench-0.1.0/pyproject.toml +23 -0
- proofbench-0.1.0/tests/test_engine.py +36 -0
- proofbench-0.1.0/tests/test_failloud.py +40 -0
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
name: release
|
|
2
|
+
on:
|
|
3
|
+
push:
|
|
4
|
+
tags: ["v*"]
|
|
5
|
+
|
|
6
|
+
jobs:
|
|
7
|
+
test:
|
|
8
|
+
runs-on: ubuntu-latest
|
|
9
|
+
steps:
|
|
10
|
+
- uses: actions/checkout@v4
|
|
11
|
+
- uses: astral-sh/setup-uv@v5
|
|
12
|
+
- run: uv run --with pytest --with pyyaml pytest tests/ -q
|
|
13
|
+
|
|
14
|
+
publish:
|
|
15
|
+
needs: test
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
environment: pypi
|
|
18
|
+
permissions:
|
|
19
|
+
id-token: write # Trusted Publishing (OIDC); no PYPI_TOKEN in CI
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v4
|
|
22
|
+
- uses: astral-sh/setup-uv@v5
|
|
23
|
+
- run: uv build
|
|
24
|
+
- run: uvx twine check dist/*
|
|
25
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Authoring a corpus
|
|
2
|
+
|
|
3
|
+
A task in proofbench is a git repository with two branches: a `default_branch` in a broken or
|
|
4
|
+
incomplete state, and an `answer_branch` holding the ground-truth change. The judge grades the
|
|
5
|
+
agent's diff against `default_branch..answer_branch`. The quality of your eval is mostly the
|
|
6
|
+
quality of these tasks. Two rules carry the most weight.
|
|
7
|
+
|
|
8
|
+
## Do not use famous bugs
|
|
9
|
+
|
|
10
|
+
A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
|
|
11
|
+
does, you are measuring what it memorized, not how it debugs. Prefer subtle, real defects where
|
|
12
|
+
the correct reasoning is about an invariant, a race, or a contract, and where the surrounding
|
|
13
|
+
code gives no hint. The discriminating tasks are the ones a model has to actually reason through,
|
|
14
|
+
not pattern-match.
|
|
15
|
+
|
|
16
|
+
## Do not telegraph the answer
|
|
17
|
+
|
|
18
|
+
The judge reads the repo's README. If the README says "this repo has an intentional bug for
|
|
19
|
+
practice," or the buggy line carries a `# Buggy` comment, the judge (and the agent) read the tell
|
|
20
|
+
instead of the contract. In one real run this flipped genuine contract violations into "intended
|
|
21
|
+
design" and quietly corrupted the grades. So:
|
|
22
|
+
|
|
23
|
+
- Write the README as a real library would: state what each function is supposed to do (its
|
|
24
|
+
contract), never that it is wrong.
|
|
25
|
+
- Remove comments that point at the defect. Keep the buggy line exactly where it is.
|
|
26
|
+
- The only place the answer should exist is the `answer_branch`.
|
|
27
|
+
|
|
28
|
+
## Practical shape
|
|
29
|
+
|
|
30
|
+
- One defect (or one coherent set) per task keeps recall and precision interpretable.
|
|
31
|
+
- Keep `default_branch` checked out as the repo's default, so a fresh clone is in the broken
|
|
32
|
+
state the agent will see.
|
|
33
|
+
- The bundled `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each a
|
|
34
|
+
`buggy` branch and a `fixed` branch, README stating the contract, no tells.
|
|
35
|
+
|
|
36
|
+
## Pointing the harness at your corpus
|
|
37
|
+
|
|
38
|
+
Each task in the config names the repo path and its two branches:
|
|
39
|
+
|
|
40
|
+
```yaml
|
|
41
|
+
corpus:
|
|
42
|
+
- name: my-task
|
|
43
|
+
path: /abs/or/relative/path/to/repo
|
|
44
|
+
invoke: "Find and fix the bug in this repository."
|
|
45
|
+
default_branch: buggy
|
|
46
|
+
answer_branch: fixed
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Large or licensed corpora should not be committed into your harness repo. Reference them by a
|
|
50
|
+
pinned git URL and clone them on setup, so the data stays out of your tree and its provenance
|
|
51
|
+
stays exact.
|
proofbench-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 LeChristopher Blackwell
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Methodology
|
|
2
|
+
|
|
3
|
+
proofbench is the distilled form of a long evaluation of one debugging skill. The full field
|
|
4
|
+
journal is [docs/crg-eval-journey.md](docs/crg-eval-journey.md), with the numbers in
|
|
5
|
+
[docs/crg-eval-results.md](docs/crg-eval-results.md). This page is the part that generalizes:
|
|
6
|
+
the handful of things that evaluation taught, each now a property the harness enforces rather
|
|
7
|
+
than advice you have to remember.
|
|
8
|
+
|
|
9
|
+
## 1. Fail loud, or do not measure
|
|
10
|
+
|
|
11
|
+
The most dangerous output of a test harness is a confident wrong number. In the original run, a
|
|
12
|
+
judge call silently exceeded the OS argument limit, its error was swallowed, an empty grade
|
|
13
|
+
flowed onward, and the scoreboard reported a clean average of four cells while presenting it as
|
|
14
|
+
five. Not a crash. A plausible lie.
|
|
15
|
+
|
|
16
|
+
proofbench makes that failure impossible to ignore. Every empty or non-numeric grade raises and
|
|
17
|
+
halts. Large prompts go to the judge on stdin, never on argv, so an embedded diff cannot abort
|
|
18
|
+
the call past the argument limit. Capture excludes installed dependency and cache trees, which
|
|
19
|
+
are never the agent's work and can balloon a prompt past that limit. These live in `util.py`,
|
|
20
|
+
`judge.py`, and `capture.py`, and they are the subject of the test suite, because they are the
|
|
21
|
+
product.
|
|
22
|
+
|
|
23
|
+
## 2. The weak model is the signal, not the noise
|
|
24
|
+
|
|
25
|
+
A self-improving loop learns from misses. Frontier models on easy tasks miss almost nothing, so
|
|
26
|
+
they teach nothing. The counterintuitive consequence: the weak model is not the liability in
|
|
27
|
+
this setup, it is the only source of signal. Its misses are the curriculum.
|
|
28
|
+
|
|
29
|
+
So proofbench is model-sweep native. `models:` is a list, and `optimize` drives the **weakest**
|
|
30
|
+
leg on purpose. To keep the signal clean, `judge_model` and `synth_model` are held constant and
|
|
31
|
+
default to a strong model. You never let the model under test grade its own homework or rewrite
|
|
32
|
+
its own instructions.
|
|
33
|
+
|
|
34
|
+
## 3. A prose gate is a request, not a guarantee
|
|
35
|
+
|
|
36
|
+
The longest stretch of the original work was spent trying to fix a weak model's misdiagnoses by
|
|
37
|
+
writing better instructions. It failed, repeatedly and expensively, and always the same way.
|
|
38
|
+
Prompt compliance scales with model strength, which means the exact model that most needs a rule
|
|
39
|
+
is the one most likely to read it as optional. You cannot wordsmith a weak model into rigor.
|
|
40
|
+
|
|
41
|
+
The lesson for any eval is narrower and sharper: when a rule must hold, it belongs in code the
|
|
42
|
+
model cannot route around, not in text the model is trusted not to skip. proofbench applies this
|
|
43
|
+
to its own gates. The keep/revert decision in `optimize.py` is the harness comparing two numbers.
|
|
44
|
+
"The score held" is not a thing the model can say. It is a thing only the comparison can return.
|
|
45
|
+
|
|
46
|
+
## 4. The corpus is part of the method
|
|
47
|
+
|
|
48
|
+
Two findings about the data, not the harness, generalize into [CORPUS.md](CORPUS.md):
|
|
49
|
+
|
|
50
|
+
- **Do not use famous bugs.** A frontier model recognizes Heartbleed on sight, and then you are
|
|
51
|
+
measuring training recall, not method. Pick subtle, real defects where the correct reasoning is
|
|
52
|
+
about an invariant or a contract, not a typo a model has memorized.
|
|
53
|
+
- **Do not let the corpus telegraph its answers.** In one run the task repos announced their own
|
|
54
|
+
planted bugs in the README and in `# Buggy` comments. The judge read the marketing and
|
|
55
|
+
classified real contract violations as intentional design. A task that advertises its answer
|
|
56
|
+
measures nothing. State the true contract; never label the defect.
|
|
57
|
+
|
|
58
|
+
## The arc, in one line
|
|
59
|
+
|
|
60
|
+
The evaluation kept writing its own conclusions, round after round, and the work was learning to
|
|
61
|
+
stop saying them in a prompt and start building them in code. proofbench is what that looks like
|
|
62
|
+
when it is finished: the lessons are the implementation.
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: proofbench
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Config-driven eval harness that grades and self-improves headless agent skills against ground-truth corpora
|
|
5
|
+
Project-URL: Homepage, https://github.com/CodeBlackwell/proofbench
|
|
6
|
+
Author: LeChristopher Blackwell
|
|
7
|
+
License: MIT
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Keywords: agent,benchmark,claude,eval,llm,prompt-optimization
|
|
10
|
+
Requires-Python: >=3.10
|
|
11
|
+
Requires-Dist: pyyaml>=6.0
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
|
|
14
|
+
# proofbench
|
|
15
|
+
|
|
16
|
+
A config-driven eval harness that grades and self-improves headless agent skills against
|
|
17
|
+
ground-truth corpora. You plant tasks you already have the answers to, run an agent skill
|
|
18
|
+
across them, and grade what it changed against the known answer with an LLM judge. Then you
|
|
19
|
+
let the harness rewrite the skill from its own misses and keep the rewrite only if the score
|
|
20
|
+
holds.
|
|
21
|
+
|
|
22
|
+
It started as the harness behind [crg-debug](examples/crg-debug/), a graph-driven debugging
|
|
23
|
+
skill. The methodology generalizes to any skill whose output you can compare to a known answer.
|
|
24
|
+
|
|
25
|
+
## Why it exists
|
|
26
|
+
|
|
27
|
+
A skill is only as trustworthy as the proof that it works. Anyone can write a prompt that
|
|
28
|
+
sounds like a methodology. The honest way to know is to measure it against ground truth, on
|
|
29
|
+
the model weak enough to embarrass it. Two ideas carry the whole design, both learned the hard
|
|
30
|
+
way (see [METHODOLOGY.md](METHODOLOGY.md)):
|
|
31
|
+
|
|
32
|
+
1. **Fail loud or do not measure.** Every empty or non-numeric grade is a hard stop. An eval
|
|
33
|
+
that fails open does not just miss data, it manufactures false confidence.
|
|
34
|
+
2. **The weak model is the signal, not the noise.** A self-improving loop learns only from
|
|
35
|
+
misses. Frontier models on easy tasks miss nothing, so they teach nothing. The weak leg is
|
|
36
|
+
the curriculum.
|
|
37
|
+
|
|
38
|
+
## Install
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
uv tool install proofbench # or: uvx proofbench
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Requires the `claude` CLI on PATH for the default runner and for the judge.
|
|
45
|
+
|
|
46
|
+
## Quickstart (demo)
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
git clone https://github.com/CodeBlackwell/proofbench && cd proofbench
|
|
50
|
+
bash examples/sample-corpus/build.sh # builds two toy repos with buggy + fixed branches
|
|
51
|
+
uvx --from . proofbench run --demo # eval an agent over them, graded vs the answers
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## How it works
|
|
55
|
+
|
|
56
|
+
One YAML config declares everything domain-specific; the engine is generic. The mode is
|
|
57
|
+
implied by what the config contains:
|
|
58
|
+
|
|
59
|
+
- a corpus + runner + judge gives you a **bench** (`run`)
|
|
60
|
+
- adding a `subject` + `synth` unlocks the **self-improving loop** (`optimize`)
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
subject: ~/.claude/skills/crg-debug/SKILL.md # optional; omit for pure-eval mode
|
|
64
|
+
runner: claude # default adapter; any executable works
|
|
65
|
+
models: [opus, sonnet, haiku] # the driver sweep; the weak leg is the signal
|
|
66
|
+
judge_model: opus # held constant; never let a model grade itself
|
|
67
|
+
objective: macro_recall # the metric the keep/revert gate reads
|
|
68
|
+
judge: prompts/judge.md
|
|
69
|
+
synth: prompts/synth.md # present => `optimize` is available
|
|
70
|
+
corpus:
|
|
71
|
+
- name: primes
|
|
72
|
+
path: examples/sample-corpus/repos/primes
|
|
73
|
+
invoke: "Find and fix the bug in this repository."
|
|
74
|
+
default_branch: buggy
|
|
75
|
+
answer_branch: fixed
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
proofbench run --config proofbench.yaml # eval + scoreboard
|
|
80
|
+
proofbench optimize --config proofbench.yaml # baseline -> synth -> re-run -> keep|revert
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### The keep/revert gate
|
|
84
|
+
|
|
85
|
+
`optimize` runs a baseline on the weak leg, asks an LLM to rewrite the subject from the graded
|
|
86
|
+
misses, re-runs, and keeps the rewrite only if the objective did not regress. The decision is
|
|
87
|
+
the harness comparing two numbers, never the model's own claim of success.
|
|
88
|
+
|
|
89
|
+
## Adapters
|
|
90
|
+
|
|
91
|
+
- **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
|
|
92
|
+
`<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
|
|
93
|
+
- **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
|
|
94
|
+
agent changed, excluding dependency and cache trees so they never pollute or balloon the
|
|
95
|
+
judge prompt.
|
|
96
|
+
|
|
97
|
+
## Bring your own corpus
|
|
98
|
+
|
|
99
|
+
The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
|
|
100
|
+
(each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
|
|
101
|
+
how to curate tasks that actually discriminate and do not leak their answers to the judge.
|
|
102
|
+
|
|
103
|
+
## License
|
|
104
|
+
|
|
105
|
+
MIT.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# proofbench
|
|
2
|
+
|
|
3
|
+
A config-driven eval harness that grades and self-improves headless agent skills against
|
|
4
|
+
ground-truth corpora. You plant tasks you already have the answers to, run an agent skill
|
|
5
|
+
across them, and grade what it changed against the known answer with an LLM judge. Then you
|
|
6
|
+
let the harness rewrite the skill from its own misses and keep the rewrite only if the score
|
|
7
|
+
holds.
|
|
8
|
+
|
|
9
|
+
It started as the harness behind [crg-debug](examples/crg-debug/), a graph-driven debugging
|
|
10
|
+
skill. The methodology generalizes to any skill whose output you can compare to a known answer.
|
|
11
|
+
|
|
12
|
+
## Why it exists
|
|
13
|
+
|
|
14
|
+
A skill is only as trustworthy as the proof that it works. Anyone can write a prompt that
|
|
15
|
+
sounds like a methodology. The honest way to know is to measure it against ground truth, on
|
|
16
|
+
the model weak enough to embarrass it. Two ideas carry the whole design, both learned the hard
|
|
17
|
+
way (see [METHODOLOGY.md](METHODOLOGY.md)):
|
|
18
|
+
|
|
19
|
+
1. **Fail loud or do not measure.** Every empty or non-numeric grade is a hard stop. An eval
|
|
20
|
+
that fails open does not just miss data, it manufactures false confidence.
|
|
21
|
+
2. **The weak model is the signal, not the noise.** A self-improving loop learns only from
|
|
22
|
+
misses. Frontier models on easy tasks miss nothing, so they teach nothing. The weak leg is
|
|
23
|
+
the curriculum.
|
|
24
|
+
|
|
25
|
+
## Install
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
uv tool install proofbench # or: uvx proofbench
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Requires the `claude` CLI on PATH for the default runner and for the judge.
|
|
32
|
+
|
|
33
|
+
## Quickstart (demo)
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
git clone https://github.com/CodeBlackwell/proofbench && cd proofbench
|
|
37
|
+
bash examples/sample-corpus/build.sh # builds two toy repos with buggy + fixed branches
|
|
38
|
+
uvx --from . proofbench run --demo # eval an agent over them, graded vs the answers
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## How it works
|
|
42
|
+
|
|
43
|
+
One YAML config declares everything domain-specific; the engine is generic. The mode is
|
|
44
|
+
implied by what the config contains:
|
|
45
|
+
|
|
46
|
+
- a corpus + runner + judge gives you a **bench** (`run`)
|
|
47
|
+
- adding a `subject` + `synth` unlocks the **self-improving loop** (`optimize`)
|
|
48
|
+
|
|
49
|
+
```yaml
|
|
50
|
+
subject: ~/.claude/skills/crg-debug/SKILL.md # optional; omit for pure-eval mode
|
|
51
|
+
runner: claude # default adapter; any executable works
|
|
52
|
+
models: [opus, sonnet, haiku] # the driver sweep; the weak leg is the signal
|
|
53
|
+
judge_model: opus # held constant; never let a model grade itself
|
|
54
|
+
objective: macro_recall # the metric the keep/revert gate reads
|
|
55
|
+
judge: prompts/judge.md
|
|
56
|
+
synth: prompts/synth.md # present => `optimize` is available
|
|
57
|
+
corpus:
|
|
58
|
+
- name: primes
|
|
59
|
+
path: examples/sample-corpus/repos/primes
|
|
60
|
+
invoke: "Find and fix the bug in this repository."
|
|
61
|
+
default_branch: buggy
|
|
62
|
+
answer_branch: fixed
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
proofbench run --config proofbench.yaml # eval + scoreboard
|
|
67
|
+
proofbench optimize --config proofbench.yaml # baseline -> synth -> re-run -> keep|revert
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### The keep/revert gate
|
|
71
|
+
|
|
72
|
+
`optimize` runs a baseline on the weak leg, asks an LLM to rewrite the subject from the graded
|
|
73
|
+
misses, re-runs, and keeps the rewrite only if the objective did not regress. The decision is
|
|
74
|
+
the harness comparing two numbers, never the model's own claim of success.
|
|
75
|
+
|
|
76
|
+
## Adapters
|
|
77
|
+
|
|
78
|
+
- **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
|
|
79
|
+
`<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
|
|
80
|
+
- **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
|
|
81
|
+
agent changed, excluding dependency and cache trees so they never pollute or balloon the
|
|
82
|
+
judge prompt.
|
|
83
|
+
|
|
84
|
+
## Bring your own corpus
|
|
85
|
+
|
|
86
|
+
The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
|
|
87
|
+
(each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
|
|
88
|
+
how to curate tasks that actually discriminate and do not leak their answers to the judge.
|
|
89
|
+
|
|
90
|
+
## License
|
|
91
|
+
|
|
92
|
+
MIT.
|
|
@@ -0,0 +1,271 @@
|
|
|
1
|
+
# The Gate the Model Could Not Skip
|
|
2
|
+
|
|
3
|
+
*A field journal from evaluating a debugging skill, and what it took to make the eval tell the truth.*
|
|
4
|
+
|
|
5
|
+
This is the story behind [CRG-EVAL-RESULTS.md](./CRG-EVAL-RESULTS.md). That document is the
|
|
6
|
+
clinical record: tables, exact numbers, per round verdicts. This one is the journey. The wrong
|
|
7
|
+
turns, the moments a confident number turned out to be a lie, and the slow realization that no
|
|
8
|
+
amount of prompt wording would make a weak model rigorous. If you want the evidence, read the
|
|
9
|
+
results doc. If you want to know how I got there, stay here.
|
|
10
|
+
|
|
11
|
+
A note on two terms. **CRG** is code-review-graph, a knowledge graph of a codebase that an agent
|
|
12
|
+
can query to localize code. **crg-debug** is a skill that drives a graph aware debugging sweep:
|
|
13
|
+
map the hotspots, fan out parallel discovery over disjoint concerns, verify findings adversarially,
|
|
14
|
+
then fix in TDD waves. The whole project below is one long attempt to find out whether that skill
|
|
15
|
+
actually works, and then to make it work on a model weak enough to embarrass it.
|
|
16
|
+
|
|
17
|
+
## The premise
|
|
18
|
+
|
|
19
|
+
A debugging skill is only as trustworthy as the proof that it works. Anyone can write a prompt
|
|
20
|
+
that *sounds* like a methodology. The only honest way to know is to plant bugs you already have the
|
|
21
|
+
answers to, run the skill, and grade what it changed against the known fix.
|
|
22
|
+
|
|
23
|
+
So that is what I built: an eval over five debugging-interview repos, each with a ground-truth
|
|
24
|
+
answer branch. Grade each fix with an LLM judge that compares the agent's diff to the reference
|
|
25
|
+
diff and reports recall (did it fix the planted bugs) and precision (did it avoid breaking things).
|
|
26
|
+
Then run the exact same harness on three driver models, opus and sonnet and haiku, to answer the
|
|
27
|
+
question that actually matters: does the *method* carry the result, or just the model?
|
|
28
|
+
|
|
29
|
+
The honest goal was never "prove it works." It was "find where it breaks." That framing is the
|
|
30
|
+
only reason any of the rest happened.
|
|
31
|
+
|
|
32
|
+
## First, the measuring stick lied to me
|
|
33
|
+
|
|
34
|
+
Before the models could fail, the harness failed, and it failed in the most dangerous way a
|
|
35
|
+
measurement system can: quietly.
|
|
36
|
+
|
|
37
|
+
There were four bugs, all of them classic `set -euo pipefail` landmines, and all of them masked by
|
|
38
|
+
whether a given run happened to have a terminal attached. A JSON extractor that choked on a model's
|
|
39
|
+
prose preamble. A helper that returned 1 when piped because its last line was a conditional. A
|
|
40
|
+
watcher that latched onto the wrong process and declared a finished run dead. Annoying, fixable,
|
|
41
|
+
the usual.
|
|
42
|
+
|
|
43
|
+
Then there was the fourth one, which changed how I thought about the whole exercise.
|
|
44
|
+
|
|
45
|
+
On the haiku leg, one cell scaffolded a test harness, ran `npm install`, and the capture step swept
|
|
46
|
+
the entire `node_modules` tree into the diff. The result was a 442,828 line diff. The judge prompt
|
|
47
|
+
that embedded it was 22.8 MB. The judge was invoked with the whole prompt as a single shell
|
|
48
|
+
argument, which sailed straight past `ARG_MAX`, so the call failed. But the call was guarded by
|
|
49
|
+
`|| true`. So the empty output flowed onward. The extractor emitted nothing, `jq` exited 0, the
|
|
50
|
+
"or die" guard never fired, an empty grade was written, and the aggregator simply skipped the empty
|
|
51
|
+
file.
|
|
52
|
+
|
|
53
|
+
The scoreboard then reported a confident `macro_recall=0.662`, as if complete. It was secretly an
|
|
54
|
+
average of four cells, not five.
|
|
55
|
+
|
|
56
|
+
That is the worst possible failure. Not a crash, not a stack trace, just a clean wrong number with
|
|
57
|
+
no reason to distrust it. I only caught it because the recovered fifth cell scored 1.00 and dragged
|
|
58
|
+
the honest baseline to 0.729, materially different from the lie. The fix was to make every empty or
|
|
59
|
+
non-numeric grade a hard stop, route big prompts through stdin instead of argv, and exclude
|
|
60
|
+
installed dependencies from the capture.
|
|
61
|
+
|
|
62
|
+
> **An eval that fails open does not just miss data. It manufactures false confidence.** A
|
|
63
|
+
> measurement system must fail loud, or it will hand you a wrong number you have every reason to
|
|
64
|
+
> believe.
|
|
65
|
+
|
|
66
|
+
I keep coming back to that one. It is the seed of everything Round 4 eventually became.
|
|
67
|
+
|
|
68
|
+
## The ceiling problem
|
|
69
|
+
|
|
70
|
+
With an honest harness, a new problem appeared, and it was almost funny: the strong models were
|
|
71
|
+
too good.
|
|
72
|
+
|
|
73
|
+
Opus scored a flawless 1.00 / 1.00. Sonnet sat right next to it. On these repos the frontier models
|
|
74
|
+
miss almost nothing, which sounds like success until you remember the point of the exercise. I had
|
|
75
|
+
built a closed improvement loop, where the grades feed a synthesis step that proposes edits to the
|
|
76
|
+
skill, and an improved run re-measures. A loop like that learns from misses. When nothing misses,
|
|
77
|
+
there is nothing to learn.
|
|
78
|
+
|
|
79
|
+
The counterintuitive turn took me a while to accept. **The weak model is not the liability in this
|
|
80
|
+
setup. It is the only source of signal.** Haiku's misses are the curriculum. A leg that scores 0.729
|
|
81
|
+
is worth more to a self-improving loop than a leg that scores 1.00, because only the imperfect one
|
|
82
|
+
tells you where the method is thin. From that point on, haiku was the protagonist.
|
|
83
|
+
|
|
84
|
+
## Harder targets
|
|
85
|
+
|
|
86
|
+
If haiku's misses are the curriculum, the easy repos are too forgiving a syllabus. Even haiku
|
|
87
|
+
ceilinged on most of them. So I went looking for bugs that would actually hurt.
|
|
88
|
+
|
|
89
|
+
The Tier-1 hard cases are real production bugs from CPython and sudo: the timsort `merge_collapse`
|
|
90
|
+
run-stack invariant that sat latent in CPython and Java and Android for years until formal
|
|
91
|
+
verification found it, the asyncio `Condition.notify` lost-wakeup race, and the sudo Baron Samedit
|
|
92
|
+
heap overflow (CVE-2021-3156). The selection rule was deliberate. Famous CVEs like Heartbleed are
|
|
93
|
+
useless here, because a frontier model recognizes them on sight and you end up measuring training
|
|
94
|
+
recall, not debugging. These three are the opposite: one subtle line buried in a real source file,
|
|
95
|
+
surrounded by production-grade code that gives no hint, where the correct reasoning is about an
|
|
96
|
+
invariant or a race rather than a typo.
|
|
97
|
+
|
|
98
|
+
And the discriminator worked on the very first interesting cell. On timsort, sonnet scored 1.00,
|
|
99
|
+
and haiku scored 0.00. A clean separation the easy repos never produced. Haiku's localization was
|
|
100
|
+
perfect: it found the file, the function, the exact `if` block. Then it edited the wrong line inside
|
|
101
|
+
the right block, rewriting correct tie-break logic into incorrect logic while never touching the
|
|
102
|
+
actual invariant check one level up. Recall 0 because it missed the bug, precision 0 because its one
|
|
103
|
+
edit introduced a new defect. It scored worse than an empty diff would have, by confidently shipping
|
|
104
|
+
a wrong fix.
|
|
105
|
+
|
|
106
|
+
That failure was not a disappointment. It was the prize specimen. A real, repeatable, well
|
|
107
|
+
understood miss on a strong-ish driver, which is exactly what the easy repos could never generate.
|
|
108
|
+
Now I had something to fix.
|
|
109
|
+
|
|
110
|
+
## The conundrum that defined the project
|
|
111
|
+
|
|
112
|
+
Here is where I spent the most time and learned the most, mostly by being wrong.
|
|
113
|
+
|
|
114
|
+
The obvious move, when a model misdiagnoses a bug, is to write a better prompt. Tell it what it did
|
|
115
|
+
wrong. Add a rule. So I did, over and over, and the eval kept teaching me the same lesson in
|
|
116
|
+
increasingly expensive ways.
|
|
117
|
+
|
|
118
|
+
**The A/B that backfired.** I hand-authored two evidence-cited edits into haiku's skill: a gate that
|
|
119
|
+
said confirm the cause line or revert, plus a rule against leaving scratch files. The result was a
|
|
120
|
+
clean negative. Precision rose on every case. Recall *fell* on the two cases haiku had previously
|
|
121
|
+
passed. The shape of the failure was the lesson: a caution-shaped edit makes an already-timid model
|
|
122
|
+
edit less and revert more, and on a weak model less editing converts directly into lost recall. The
|
|
123
|
+
intervention aimed at a precision problem and made the recall problem worse.
|
|
124
|
+
|
|
125
|
+
**Round 2, the trace-and-reverify scaffold.** I replaced the confirm-or-revert gate with something
|
|
126
|
+
that tried to supply reasoning instead of caution: name the invariant, build a concrete input that
|
|
127
|
+
violates it, hand-execute to the line where it first breaks, re-verify, then look for a second
|
|
128
|
+
violating input. Better in theory. But on the C cases the re-verify step is hand-simulation, and a
|
|
129
|
+
weak model can hand-simulate incorrectly with total confidence.
|
|
130
|
+
|
|
131
|
+
**Round 3, just let the session continue.** A real user does not take a one-shot suggestion and
|
|
132
|
+
stop. They keep the chat open and often crack on the third pass what they missed on the first. So I
|
|
133
|
+
made the harness resume the same session across up to three tries with a neutral nudge. This is the
|
|
134
|
+
one that taught me the most, because iteration without a real oracle did not help. It became a
|
|
135
|
+
*rationalization engine*. On timsort, the agent used its second pass to hand-trace its own already
|
|
136
|
+
wrong edit, used the trace to confirm it, and emitted `DEBUG_COMPLETE`. With no built binary to
|
|
137
|
+
contradict it, more thinking just entrenched the mistake.
|
|
138
|
+
|
|
139
|
+
And underneath all of it sat the gate I was proudest of and that failed hardest. I hardened the TDD
|
|
140
|
+
rule into "no RED, no edit": you must see a real failing test before you touch code, and a test that
|
|
141
|
+
runs against an unbuilt interpreter does not count. Haiku read it as advisory. It edited compiled
|
|
142
|
+
source, tested the system Python instead of the source it changed, reported green, and emitted
|
|
143
|
+
`DEBUG_COMPLETE` anyway. No build, no RED, no hesitation.
|
|
144
|
+
|
|
145
|
+
> **You cannot wordsmith a weak model into rigor.** A prose gate is a request. Compliance with a
|
|
146
|
+
> request scales with the strength of the model reading it, which means the exact model that most
|
|
147
|
+
> needs the gate is the one most likely to ignore it.
|
|
148
|
+
|
|
149
|
+
## The realization
|
|
150
|
+
|
|
151
|
+
The thing about a good eval is that it writes its own conclusions, and then you have to be honest
|
|
152
|
+
enough to read them.
|
|
153
|
+
|
|
154
|
+
The results doc had already said it, twice, in plain language. After the Round 3 failure it
|
|
155
|
+
concluded that stopping the timsort regression "needs an enforcement mechanism the harness owns,
|
|
156
|
+
not more SKILL.md text." And in the same breath it noted that the rejected prose gate was still
|
|
157
|
+
valuable, because "it is the spec for the harness-side gate."
|
|
158
|
+
|
|
159
|
+
I had been treating those sentences as a postmortem. They were a design brief. The fix was never
|
|
160
|
+
going to be words. It was going to be code that does not ask.
|
|
161
|
+
|
|
162
|
+
## Round 4, the turn
|
|
163
|
+
|
|
164
|
+
So I re-implemented crg-debug as a Workflow: deterministic JavaScript that owns the control flow,
|
|
165
|
+
with agents reduced to what only they can do. The script owns phase sequencing, concern
|
|
166
|
+
partitioning, dedup, wave packing, loop termination, and the ledger. The agents own judgment and the
|
|
167
|
+
I/O the script cannot perform: graph queries, toolchain discovery, the baseline build, discovery,
|
|
168
|
+
adversarial verify, and the actual fixes.
|
|
169
|
+
|
|
170
|
+
The defining test for every gate became literal. Not "is this rule clear" but "can the model skip
|
|
171
|
+
it?" If a gate is a sentence in a prompt, the answer is yes. If a gate is a function the script calls
|
|
172
|
+
and an exit code the script reads, the answer is no.
|
|
173
|
+
|
|
174
|
+
The crux is the per-bug close gate. A fix agent applies its fix and reports the narrowest
|
|
175
|
+
`testCommand` that exercises only its bug. A separate agent then re-runs exactly that command and
|
|
176
|
+
reports the raw exit code. The script reads the exit code and decides closed or not closed. The model
|
|
177
|
+
never declares "passed." Its green claim is advisory; the exit code is authoritative.
|
|
178
|
+
|
|
179
|
+
> **A claim you cannot make is a claim you cannot fake.** The Round 3 gate failed because "passed"
|
|
180
|
+
> was a thing the model could say. The Round 4 gate works because "passed" is a thing only the shell
|
|
181
|
+
> can return.
|
|
182
|
+
|
|
183
|
+
This also cured an earlier deadlock I had hit with a whole-suite gate, where one unfixed sibling
|
|
184
|
+
failed the entire suite and blocked every genuine fix, stalling at 0 of 14 closed. Once a bug closes
|
|
185
|
+
on its own narrow test, siblings cannot hold it hostage.
|
|
186
|
+
|
|
187
|
+
## The refactor that made it a command
|
|
188
|
+
|
|
189
|
+
Demoting the prose was the satisfying part. `SKILL.md` went from a 177 line harness, all
|
|
190
|
+
orchestration and anti-skip pleading, to a roughly 45 line dispatcher whose only job is to parse
|
|
191
|
+
arguments and call `Workflow({name:'crg-debug', args})`. The judgment prose that still mattered, the
|
|
192
|
+
bug-class checklist, the real-versus-intentional rule, the fail-safe-defaults lens, moved into a new
|
|
193
|
+
`methodology.md` reference that both the Workflow and the standalone debugger agent read. The
|
|
194
|
+
control-flow prose simply deleted, because the JavaScript now performs what it used to describe.
|
|
195
|
+
|
|
196
|
+
The whole thesis fits in one contrast. The skill was prose I was trusted not to skip. The Workflow
|
|
197
|
+
is gates the script owns. As a bonus, the dispatcher gained a uniform `model` argument applied to
|
|
198
|
+
every agent, something the prose skill never had.
|
|
199
|
+
|
|
200
|
+
## The two surprises in validation
|
|
201
|
+
|
|
202
|
+
I validated on the BugFix repo with haiku as the driver, the same weak tier the prose rounds used,
|
|
203
|
+
so any gap would be a property of the harness and not a model upgrade. It took three runs and two
|
|
204
|
+
genuine surprises to get to green.
|
|
205
|
+
|
|
206
|
+
**Finding A: the repo was lying to the classifier.** Run 1 deferred four real bugs as "intentional."
|
|
207
|
+
The cause was the corpus repo's own framing. Its README announced "intentionally introduced bugs
|
|
208
|
+
for practice," and every defect carried a `# Buggy` comment. The adversarial verifier read those
|
|
209
|
+
tells and classified real contract violations as deliberate design, which is exactly the trap the
|
|
210
|
+
real-versus-intentional lens is meant to avoid. I rewrote the README as a real library, replaced the
|
|
211
|
+
comments with statements of the true contract, and kept every buggy line at its original position.
|
|
212
|
+
Deferred dropped from 4 to 0. The classifier had been reading the repo's marketing, and that
|
|
213
|
+
sensitivity is itself model-tier dependent: haiku reads the tells, stronger models weigh the contract
|
|
214
|
+
over the comment.
|
|
215
|
+
|
|
216
|
+
**Finding B: fixing the first surprise exposed a latent bug.** With the four bugs back in the queue,
|
|
217
|
+
`bug.py` now held seven bugs in one file, and Run 2 closed only 8 of 14 with a red gate, worse than
|
|
218
|
+
Run 1. The cleanup had not caused a regression. It had revealed one. Reading the loop, the cause was
|
|
219
|
+
two compounding defects. File-disjoint wave packing let one agent own a file per wave, so a dense
|
|
220
|
+
file could receive at most one fix per wave while its other bugs waited. And the thrash guard,
|
|
221
|
+
meant to detect genuine failed retries, had been adding bugs deferred merely because their file was
|
|
222
|
+
busy to the same "seen" set, then aborting the loop when they legitimately reappeared. Run 1 had only
|
|
223
|
+
looked green because four of `bug.py`'s bugs were hidden as intentional, thinning the file enough to
|
|
224
|
+
drain in time.
|
|
225
|
+
|
|
226
|
+
The fix restructured the fix phase: one agent now owns all of a file's bugs at once, a per-agent cap
|
|
227
|
+
sends a denser file's overflow to later waves as deferred chunks, and the thrash guard counts only
|
|
228
|
+
genuinely failed retries, never deferral. Run 3 closed everything. Zero unfixed, green gate, drained
|
|
229
|
+
in two waves.
|
|
230
|
+
|
|
231
|
+
It is worth sitting with the chain there. A misclassification born from a README masked a wave-packing
|
|
232
|
+
bug, which was masked again by a thrash guard that could not tell deferral from thrashing. Three
|
|
233
|
+
layers, each hiding the next, and only an honest run with all the bugs surfaced forced them out. The
|
|
234
|
+
same lesson as harness bug four, wearing different clothes: things that fail quietly compound.
|
|
235
|
+
|
|
236
|
+
## What I actually learned
|
|
237
|
+
|
|
238
|
+
Pulling the threads together, the journey reduces to a handful of things I now believe more firmly
|
|
239
|
+
than when I started.
|
|
240
|
+
|
|
241
|
+
**Fail loud or do not measure.** The most dangerous output of a test harness is a confident wrong
|
|
242
|
+
number. If a path can swallow an error and emit a plausible result, it eventually will, and you will
|
|
243
|
+
trust it.
|
|
244
|
+
|
|
245
|
+
**The weak model is the signal, not the noise.** A self-improving loop learns from misses. Frontier
|
|
246
|
+
models on easy repos do not miss, so they teach nothing. Deliberately running the model that fails
|
|
247
|
+
is how you find what the method is missing.
|
|
248
|
+
|
|
249
|
+
**Prompt compliance scales with model strength, so a prose gate is a request and not a guarantee.**
|
|
250
|
+
The model that most needs a rule is the one most likely to read it as optional. This is the single
|
|
251
|
+
most useful thing the whole eval surfaced, and it generalizes well beyond debugging.
|
|
252
|
+
|
|
253
|
+
**When a rule must hold, move it into code the model cannot route around.** The difference between
|
|
254
|
+
Round 3 and Round 4 is the difference between asking the model to verify and letting only the shell
|
|
255
|
+
report the verdict. Enforcement belongs wherever the model cannot reach.
|
|
256
|
+
|
|
257
|
+
And the honest caveats, because the eval would not forgive me for omitting them. Round 4 is still
|
|
258
|
+
n=1, on one easy repo, with haiku. It proves the gate binds. It does not yet prove the Workflow
|
|
259
|
+
ceilings the hard cases. The Tier-1 set, timsort and asyncio and sudo, has not yet run through the
|
|
260
|
+
Workflow. That is the next run, and the standing limitations, N repeats per cell to separate signal
|
|
261
|
+
from stochasticity and harder targets to break the recall ceiling, are unchanged.
|
|
262
|
+
|
|
263
|
+
The through-line is almost embarrassing in hindsight. The eval kept telling me the answer in its own
|
|
264
|
+
conclusions, round after round. I just had to stop trying to say it in a prompt and start building it
|
|
265
|
+
in code.
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
*For the numbers behind every claim here, see [CRG-EVAL-RESULTS.md](./CRG-EVAL-RESULTS.md): the
|
|
270
|
+
harness bugs in "Harness bugs found and fixed," the ceiling in "Findings," the hard cases in "Tier-1
|
|
271
|
+
hard cases," the prose-gate failures in Rounds 2 and 3, and the Workflow turn in Round 4.*
|