proofbench 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,25 @@
1
+ name: release
2
+ on:
3
+ push:
4
+ tags: ["v*"]
5
+
6
+ jobs:
7
+ test:
8
+ runs-on: ubuntu-latest
9
+ steps:
10
+ - uses: actions/checkout@v4
11
+ - uses: astral-sh/setup-uv@v5
12
+ - run: uv run --with pytest --with pyyaml pytest tests/ -q
13
+
14
+ publish:
15
+ needs: test
16
+ runs-on: ubuntu-latest
17
+ environment: pypi
18
+ permissions:
19
+ id-token: write # Trusted Publishing (OIDC); no PYPI_TOKEN in CI
20
+ steps:
21
+ - uses: actions/checkout@v4
22
+ - uses: astral-sh/setup-uv@v5
23
+ - run: uv build
24
+ - run: uvx twine check dist/*
25
+ - uses: pypa/gh-action-pypi-publish@release/v1
@@ -0,0 +1,9 @@
1
+ __pycache__/
2
+ *.egg-info/
3
+ dist/
4
+ build/
5
+ runs/
6
+ examples/sample-corpus/repos/
7
+ .venv/
8
+ .pytest_cache/
9
+ uv.lock
@@ -0,0 +1,51 @@
1
+ # Authoring a corpus
2
+
3
+ A task in proofbench is a git repository with two branches: a `default_branch` in a broken or
4
+ incomplete state, and an `answer_branch` holding the ground-truth change. The judge grades the
5
+ agent's diff against `default_branch..answer_branch`. The quality of your eval is mostly the
6
+ quality of these tasks. Two rules carry the most weight.
7
+
8
+ ## Do not use famous bugs
9
+
10
+ A capable model recognizes Heartbleed, Shellshock, or a textbook off-by-one on sight. When it
11
+ does, you are measuring what it memorized, not how it debugs. Prefer subtle, real defects where
12
+ the correct reasoning is about an invariant, a race, or a contract, and where the surrounding
13
+ code gives no hint. The discriminating tasks are the ones a model has to actually reason through,
14
+ not pattern-match.
15
+
16
+ ## Do not telegraph the answer
17
+
18
+ The judge reads the repo's README. If the README says "this repo has an intentional bug for
19
+ practice," or the buggy line carries a `# Buggy` comment, the judge (and the agent) read the tell
20
+ instead of the contract. In one real run this flipped genuine contract violations into "intended
21
+ design" and quietly corrupted the grades. So:
22
+
23
+ - Write the README as a real library would: state what each function is supposed to do (its
24
+ contract), never that it is wrong.
25
+ - Remove comments that point at the defect. Keep the buggy line exactly where it is.
26
+ - The only place the answer should exist is the `answer_branch`.
27
+
28
+ ## Practical shape
29
+
30
+ - One defect (or one coherent set) per task keeps recall and precision interpretable.
31
+ - Keep `default_branch` checked out as the repo's default, so a fresh clone is in the broken
32
+ state the agent will see.
33
+ - The bundled `examples/sample-corpus/build.sh` is a minimal worked example: two repos, each a
34
+ `buggy` branch and a `fixed` branch, README stating the contract, no tells.
35
+
36
+ ## Pointing the harness at your corpus
37
+
38
+ Each task in the config names the repo path and its two branches:
39
+
40
+ ```yaml
41
+ corpus:
42
+ - name: my-task
43
+ path: /abs/or/relative/path/to/repo
44
+ invoke: "Find and fix the bug in this repository."
45
+ default_branch: buggy
46
+ answer_branch: fixed
47
+ ```
48
+
49
+ Large or licensed corpora should not be committed into your harness repo. Reference them by a
50
+ pinned git URL and clone them on setup, so the data stays out of your tree and its provenance
51
+ stays exact.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 LeChristopher Blackwell
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,62 @@
1
+ # Methodology
2
+
3
+ proofbench is the distilled form of a long evaluation of one debugging skill. The full field
4
+ journal is [docs/crg-eval-journey.md](docs/crg-eval-journey.md), with the numbers in
5
+ [docs/crg-eval-results.md](docs/crg-eval-results.md). This page is the part that generalizes:
6
+ the handful of things that evaluation taught, each now a property the harness enforces rather
7
+ than advice you have to remember.
8
+
9
+ ## 1. Fail loud, or do not measure
10
+
11
+ The most dangerous output of a test harness is a confident wrong number. In the original run, a
12
+ judge call silently exceeded the OS argument limit, its error was swallowed, an empty grade
13
+ flowed onward, and the scoreboard reported a clean average of four cells while presenting it as
14
+ five. Not a crash. A plausible lie.
15
+
16
+ proofbench makes that failure impossible to ignore. Every empty or non-numeric grade raises and
17
+ halts. Large prompts go to the judge on stdin, never on argv, so an embedded diff cannot abort
18
+ the call past the argument limit. Capture excludes installed dependency and cache trees, which
19
+ are never the agent's work and can balloon a prompt past that limit. These live in `util.py`,
20
+ `judge.py`, and `capture.py`, and they are the subject of the test suite, because they are the
21
+ product.
22
+
23
+ ## 2. The weak model is the signal, not the noise
24
+
25
+ A self-improving loop learns from misses. Frontier models on easy tasks miss almost nothing, so
26
+ they teach nothing. The counterintuitive consequence: the weak model is not the liability in
27
+ this setup, it is the only source of signal. Its misses are the curriculum.
28
+
29
+ So proofbench is model-sweep native. `models:` is a list, and `optimize` drives the **weakest**
30
+ leg on purpose. To keep the signal clean, `judge_model` and `synth_model` are held constant and
31
+ default to a strong model. You never let the model under test grade its own homework or rewrite
32
+ its own instructions.
33
+
34
+ ## 3. A prose gate is a request, not a guarantee
35
+
36
+ The longest stretch of the original work was spent trying to fix a weak model's misdiagnoses by
37
+ writing better instructions. It failed, repeatedly and expensively, and always the same way.
38
+ Prompt compliance scales with model strength, which means the exact model that most needs a rule
39
+ is the one most likely to read it as optional. You cannot wordsmith a weak model into rigor.
40
+
41
+ The lesson for any eval is narrower and sharper: when a rule must hold, it belongs in code the
42
+ model cannot route around, not in text the model is trusted not to skip. proofbench applies this
43
+ to its own gates. The keep/revert decision in `optimize.py` is the harness comparing two numbers.
44
+ "The score held" is not a thing the model can say. It is a thing only the comparison can return.
45
+
46
+ ## 4. The corpus is part of the method
47
+
48
+ Two findings about the data, not the harness, generalize into [CORPUS.md](CORPUS.md):
49
+
50
+ - **Do not use famous bugs.** A frontier model recognizes Heartbleed on sight, and then you are
51
+ measuring training recall, not method. Pick subtle, real defects where the correct reasoning is
52
+ about an invariant or a contract, not a typo a model has memorized.
53
+ - **Do not let the corpus telegraph its answers.** In one run the task repos announced their own
54
+ planted bugs in the README and in `# Buggy` comments. The judge read the marketing and
55
+ classified real contract violations as intentional design. A task that advertises its answer
56
+ measures nothing. State the true contract; never label the defect.
57
+
58
+ ## The arc, in one line
59
+
60
+ The evaluation kept writing its own conclusions, round after round, and the work was learning to
61
+ stop saying them in a prompt and start building them in code. proofbench is what that looks like
62
+ when it is finished: the lessons are the implementation.
@@ -0,0 +1,105 @@
1
+ Metadata-Version: 2.4
2
+ Name: proofbench
3
+ Version: 0.1.0
4
+ Summary: Config-driven eval harness that grades and self-improves headless agent skills against ground-truth corpora
5
+ Project-URL: Homepage, https://github.com/CodeBlackwell/proofbench
6
+ Author: LeChristopher Blackwell
7
+ License: MIT
8
+ License-File: LICENSE
9
+ Keywords: agent,benchmark,claude,eval,llm,prompt-optimization
10
+ Requires-Python: >=3.10
11
+ Requires-Dist: pyyaml>=6.0
12
+ Description-Content-Type: text/markdown
13
+
14
+ # proofbench
15
+
16
+ A config-driven eval harness that grades and self-improves headless agent skills against
17
+ ground-truth corpora. You plant tasks you already have the answers to, run an agent skill
18
+ across them, and grade what it changed against the known answer with an LLM judge. Then you
19
+ let the harness rewrite the skill from its own misses and keep the rewrite only if the score
20
+ holds.
21
+
22
+ It started as the harness behind [crg-debug](examples/crg-debug/), a graph-driven debugging
23
+ skill. The methodology generalizes to any skill whose output you can compare to a known answer.
24
+
25
+ ## Why it exists
26
+
27
+ A skill is only as trustworthy as the proof that it works. Anyone can write a prompt that
28
+ sounds like a methodology. The honest way to know is to measure it against ground truth, on
29
+ the model weak enough to embarrass it. Two ideas carry the whole design, both learned the hard
30
+ way (see [METHODOLOGY.md](METHODOLOGY.md)):
31
+
32
+ 1. **Fail loud or do not measure.** Every empty or non-numeric grade is a hard stop. An eval
33
+ that fails open does not just miss data, it manufactures false confidence.
34
+ 2. **The weak model is the signal, not the noise.** A self-improving loop learns only from
35
+ misses. Frontier models on easy tasks miss nothing, so they teach nothing. The weak leg is
36
+ the curriculum.
37
+
38
+ ## Install
39
+
40
+ ```bash
41
+ uv tool install proofbench # or: uvx proofbench
42
+ ```
43
+
44
+ Requires the `claude` CLI on PATH for the default runner and for the judge.
45
+
46
+ ## Quickstart (demo)
47
+
48
+ ```bash
49
+ git clone https://github.com/CodeBlackwell/proofbench && cd proofbench
50
+ bash examples/sample-corpus/build.sh # builds two toy repos with buggy + fixed branches
51
+ uvx --from . proofbench run --demo # eval an agent over them, graded vs the answers
52
+ ```
53
+
54
+ ## How it works
55
+
56
+ One YAML config declares everything domain-specific; the engine is generic. The mode is
57
+ implied by what the config contains:
58
+
59
+ - a corpus + runner + judge gives you a **bench** (`run`)
60
+ - adding a `subject` + `synth` unlocks the **self-improving loop** (`optimize`)
61
+
62
+ ```yaml
63
+ subject: ~/.claude/skills/crg-debug/SKILL.md # optional; omit for pure-eval mode
64
+ runner: claude # default adapter; any executable works
65
+ models: [opus, sonnet, haiku] # the driver sweep; the weak leg is the signal
66
+ judge_model: opus # held constant; never let a model grade itself
67
+ objective: macro_recall # the metric the keep/revert gate reads
68
+ judge: prompts/judge.md
69
+ synth: prompts/synth.md # present => `optimize` is available
70
+ corpus:
71
+ - name: primes
72
+ path: examples/sample-corpus/repos/primes
73
+ invoke: "Find and fix the bug in this repository."
74
+ default_branch: buggy
75
+ answer_branch: fixed
76
+ ```
77
+
78
+ ```bash
79
+ proofbench run --config proofbench.yaml # eval + scoreboard
80
+ proofbench optimize --config proofbench.yaml # baseline -> synth -> re-run -> keep|revert
81
+ ```
82
+
83
+ ### The keep/revert gate
84
+
85
+ `optimize` runs a baseline on the weak leg, asks an LLM to rewrite the subject from the graded
86
+ misses, re-runs, and keeps the rewrite only if the objective did not regress. The decision is
87
+ the harness comparing two numbers, never the model's own claim of success.
88
+
89
+ ## Adapters
90
+
91
+ - **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
92
+ `<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
93
+ - **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
94
+ agent changed, excluding dependency and cache trees so they never pollute or balloon the
95
+ judge prompt.
96
+
97
+ ## Bring your own corpus
98
+
99
+ The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
100
+ (each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
101
+ how to curate tasks that actually discriminate and do not leak their answers to the judge.
102
+
103
+ ## License
104
+
105
+ MIT.
@@ -0,0 +1,92 @@
1
+ # proofbench
2
+
3
+ A config-driven eval harness that grades and self-improves headless agent skills against
4
+ ground-truth corpora. You plant tasks you already have the answers to, run an agent skill
5
+ across them, and grade what it changed against the known answer with an LLM judge. Then you
6
+ let the harness rewrite the skill from its own misses and keep the rewrite only if the score
7
+ holds.
8
+
9
+ It started as the harness behind [crg-debug](examples/crg-debug/), a graph-driven debugging
10
+ skill. The methodology generalizes to any skill whose output you can compare to a known answer.
11
+
12
+ ## Why it exists
13
+
14
+ A skill is only as trustworthy as the proof that it works. Anyone can write a prompt that
15
+ sounds like a methodology. The honest way to know is to measure it against ground truth, on
16
+ the model weak enough to embarrass it. Two ideas carry the whole design, both learned the hard
17
+ way (see [METHODOLOGY.md](METHODOLOGY.md)):
18
+
19
+ 1. **Fail loud or do not measure.** Every empty or non-numeric grade is a hard stop. An eval
20
+ that fails open does not just miss data, it manufactures false confidence.
21
+ 2. **The weak model is the signal, not the noise.** A self-improving loop learns only from
22
+ misses. Frontier models on easy tasks miss nothing, so they teach nothing. The weak leg is
23
+ the curriculum.
24
+
25
+ ## Install
26
+
27
+ ```bash
28
+ uv tool install proofbench # or: uvx proofbench
29
+ ```
30
+
31
+ Requires the `claude` CLI on PATH for the default runner and for the judge.
32
+
33
+ ## Quickstart (demo)
34
+
35
+ ```bash
36
+ git clone https://github.com/CodeBlackwell/proofbench && cd proofbench
37
+ bash examples/sample-corpus/build.sh # builds two toy repos with buggy + fixed branches
38
+ uvx --from . proofbench run --demo # eval an agent over them, graded vs the answers
39
+ ```
40
+
41
+ ## How it works
42
+
43
+ One YAML config declares everything domain-specific; the engine is generic. The mode is
44
+ implied by what the config contains:
45
+
46
+ - a corpus + runner + judge gives you a **bench** (`run`)
47
+ - adding a `subject` + `synth` unlocks the **self-improving loop** (`optimize`)
48
+
49
+ ```yaml
50
+ subject: ~/.claude/skills/crg-debug/SKILL.md # optional; omit for pure-eval mode
51
+ runner: claude # default adapter; any executable works
52
+ models: [opus, sonnet, haiku] # the driver sweep; the weak leg is the signal
53
+ judge_model: opus # held constant; never let a model grade itself
54
+ objective: macro_recall # the metric the keep/revert gate reads
55
+ judge: prompts/judge.md
56
+ synth: prompts/synth.md # present => `optimize` is available
57
+ corpus:
58
+ - name: primes
59
+ path: examples/sample-corpus/repos/primes
60
+ invoke: "Find and fix the bug in this repository."
61
+ default_branch: buggy
62
+ answer_branch: fixed
63
+ ```
64
+
65
+ ```bash
66
+ proofbench run --config proofbench.yaml # eval + scoreboard
67
+ proofbench optimize --config proofbench.yaml # baseline -> synth -> re-run -> keep|revert
68
+ ```
69
+
70
+ ### The keep/revert gate
71
+
72
+ `optimize` runs a baseline on the weak leg, asks an LLM to rewrite the subject from the graded
73
+ misses, re-runs, and keeps the rewrite only if the objective did not regress. The decision is
74
+ the harness comparing two numbers, never the model's own claim of success.
75
+
76
+ ## Adapters
77
+
78
+ - **Runner** (`runner:`): `claude` is built in. Any other value is an executable invoked as
79
+ `<runner> <invoke> <model>`, so you can drive aider, codex, or a custom agent.
80
+ - **Capture**: the git-diff default resets a repo to its broken branch and snapshots what the
81
+ agent changed, excluding dependency and cache trees so they never pollute or balloon the
82
+ judge prompt.
83
+
84
+ ## Bring your own corpus
85
+
86
+ The bundled corpus is two MIT toy repos for the demo. Point `corpus:` at your own repos
87
+ (each with a broken default branch and a fixed answer branch). See [CORPUS.md](CORPUS.md) for
88
+ how to curate tasks that actually discriminate and do not leak their answers to the judge.
89
+
90
+ ## License
91
+
92
+ MIT.
@@ -0,0 +1,271 @@
1
+ # The Gate the Model Could Not Skip
2
+
3
+ *A field journal from evaluating a debugging skill, and what it took to make the eval tell the truth.*
4
+
5
+ This is the story behind [CRG-EVAL-RESULTS.md](./CRG-EVAL-RESULTS.md). That document is the
6
+ clinical record: tables, exact numbers, per round verdicts. This one is the journey. The wrong
7
+ turns, the moments a confident number turned out to be a lie, and the slow realization that no
8
+ amount of prompt wording would make a weak model rigorous. If you want the evidence, read the
9
+ results doc. If you want to know how I got there, stay here.
10
+
11
+ A note on two terms. **CRG** is code-review-graph, a knowledge graph of a codebase that an agent
12
+ can query to localize code. **crg-debug** is a skill that drives a graph aware debugging sweep:
13
+ map the hotspots, fan out parallel discovery over disjoint concerns, verify findings adversarially,
14
+ then fix in TDD waves. The whole project below is one long attempt to find out whether that skill
15
+ actually works, and then to make it work on a model weak enough to embarrass it.
16
+
17
+ ## The premise
18
+
19
+ A debugging skill is only as trustworthy as the proof that it works. Anyone can write a prompt
20
+ that *sounds* like a methodology. The only honest way to know is to plant bugs you already have the
21
+ answers to, run the skill, and grade what it changed against the known fix.
22
+
23
+ So that is what I built: an eval over five debugging-interview repos, each with a ground-truth
24
+ answer branch. Grade each fix with an LLM judge that compares the agent's diff to the reference
25
+ diff and reports recall (did it fix the planted bugs) and precision (did it avoid breaking things).
26
+ Then run the exact same harness on three driver models, opus and sonnet and haiku, to answer the
27
+ question that actually matters: does the *method* carry the result, or just the model?
28
+
29
+ The honest goal was never "prove it works." It was "find where it breaks." That framing is the
30
+ only reason any of the rest happened.
31
+
32
+ ## First, the measuring stick lied to me
33
+
34
+ Before the models could fail, the harness failed, and it failed in the most dangerous way a
35
+ measurement system can: quietly.
36
+
37
+ There were four bugs, all of them classic `set -euo pipefail` landmines, and all of them masked by
38
+ whether a given run happened to have a terminal attached. A JSON extractor that choked on a model's
39
+ prose preamble. A helper that returned 1 when piped because its last line was a conditional. A
40
+ watcher that latched onto the wrong process and declared a finished run dead. Annoying, fixable,
41
+ the usual.
42
+
43
+ Then there was the fourth one, which changed how I thought about the whole exercise.
44
+
45
+ On the haiku leg, one cell scaffolded a test harness, ran `npm install`, and the capture step swept
46
+ the entire `node_modules` tree into the diff. The result was a 442,828 line diff. The judge prompt
47
+ that embedded it was 22.8 MB. The judge was invoked with the whole prompt as a single shell
48
+ argument, which sailed straight past `ARG_MAX`, so the call failed. But the call was guarded by
49
+ `|| true`. So the empty output flowed onward. The extractor emitted nothing, `jq` exited 0, the
50
+ "or die" guard never fired, an empty grade was written, and the aggregator simply skipped the empty
51
+ file.
52
+
53
+ The scoreboard then reported a confident `macro_recall=0.662`, as if complete. It was secretly an
54
+ average of four cells, not five.
55
+
56
+ That is the worst possible failure. Not a crash, not a stack trace, just a clean wrong number with
57
+ no reason to distrust it. I only caught it because the recovered fifth cell scored 1.00 and dragged
58
+ the honest baseline to 0.729, materially different from the lie. The fix was to make every empty or
59
+ non-numeric grade a hard stop, route big prompts through stdin instead of argv, and exclude
60
+ installed dependencies from the capture.
61
+
62
+ > **An eval that fails open does not just miss data. It manufactures false confidence.** A
63
+ > measurement system must fail loud, or it will hand you a wrong number you have every reason to
64
+ > believe.
65
+
66
+ I keep coming back to that one. It is the seed of everything Round 4 eventually became.
67
+
68
+ ## The ceiling problem
69
+
70
+ With an honest harness, a new problem appeared, and it was almost funny: the strong models were
71
+ too good.
72
+
73
+ Opus scored a flawless 1.00 / 1.00. Sonnet sat right next to it. On these repos the frontier models
74
+ miss almost nothing, which sounds like success until you remember the point of the exercise. I had
75
+ built a closed improvement loop, where the grades feed a synthesis step that proposes edits to the
76
+ skill, and an improved run re-measures. A loop like that learns from misses. When nothing misses,
77
+ there is nothing to learn.
78
+
79
+ The counterintuitive turn took me a while to accept. **The weak model is not the liability in this
80
+ setup. It is the only source of signal.** Haiku's misses are the curriculum. A leg that scores 0.729
81
+ is worth more to a self-improving loop than a leg that scores 1.00, because only the imperfect one
82
+ tells you where the method is thin. From that point on, haiku was the protagonist.
83
+
84
+ ## Harder targets
85
+
86
+ If haiku's misses are the curriculum, the easy repos are too forgiving a syllabus. Even haiku
87
+ ceilinged on most of them. So I went looking for bugs that would actually hurt.
88
+
89
+ The Tier-1 hard cases are real production bugs from CPython and sudo: the timsort `merge_collapse`
90
+ run-stack invariant that sat latent in CPython and Java and Android for years until formal
91
+ verification found it, the asyncio `Condition.notify` lost-wakeup race, and the sudo Baron Samedit
92
+ heap overflow (CVE-2021-3156). The selection rule was deliberate. Famous CVEs like Heartbleed are
93
+ useless here, because a frontier model recognizes them on sight and you end up measuring training
94
+ recall, not debugging. These three are the opposite: one subtle line buried in a real source file,
95
+ surrounded by production-grade code that gives no hint, where the correct reasoning is about an
96
+ invariant or a race rather than a typo.
97
+
98
+ And the discriminator worked on the very first interesting cell. On timsort, sonnet scored 1.00,
99
+ and haiku scored 0.00. A clean separation the easy repos never produced. Haiku's localization was
100
+ perfect: it found the file, the function, the exact `if` block. Then it edited the wrong line inside
101
+ the right block, rewriting correct tie-break logic into incorrect logic while never touching the
102
+ actual invariant check one level up. Recall 0 because it missed the bug, precision 0 because its one
103
+ edit introduced a new defect. It scored worse than an empty diff would have, by confidently shipping
104
+ a wrong fix.
105
+
106
+ That failure was not a disappointment. It was the prize specimen. A real, repeatable, well
107
+ understood miss on a strong-ish driver, which is exactly what the easy repos could never generate.
108
+ Now I had something to fix.
109
+
110
+ ## The conundrum that defined the project
111
+
112
+ Here is where I spent the most time and learned the most, mostly by being wrong.
113
+
114
+ The obvious move, when a model misdiagnoses a bug, is to write a better prompt. Tell it what it did
115
+ wrong. Add a rule. So I did, over and over, and the eval kept teaching me the same lesson in
116
+ increasingly expensive ways.
117
+
118
+ **The A/B that backfired.** I hand-authored two evidence-cited edits into haiku's skill: a gate that
119
+ said confirm the cause line or revert, plus a rule against leaving scratch files. The result was a
120
+ clean negative. Precision rose on every case. Recall *fell* on the two cases haiku had previously
121
+ passed. The shape of the failure was the lesson: a caution-shaped edit makes an already-timid model
122
+ edit less and revert more, and on a weak model less editing converts directly into lost recall. The
123
+ intervention aimed at a precision problem and made the recall problem worse.
124
+
125
+ **Round 2, the trace-and-reverify scaffold.** I replaced the confirm-or-revert gate with something
126
+ that tried to supply reasoning instead of caution: name the invariant, build a concrete input that
127
+ violates it, hand-execute to the line where it first breaks, re-verify, then look for a second
128
+ violating input. Better in theory. But on the C cases the re-verify step is hand-simulation, and a
129
+ weak model can hand-simulate incorrectly with total confidence.
130
+
131
+ **Round 3, just let the session continue.** A real user does not take a one-shot suggestion and
132
+ stop. They keep the chat open and often crack on the third pass what they missed on the first. So I
133
+ made the harness resume the same session across up to three tries with a neutral nudge. This is the
134
+ one that taught me the most, because iteration without a real oracle did not help. It became a
135
+ *rationalization engine*. On timsort, the agent used its second pass to hand-trace its own already
136
+ wrong edit, used the trace to confirm it, and emitted `DEBUG_COMPLETE`. With no built binary to
137
+ contradict it, more thinking just entrenched the mistake.
138
+
139
+ And underneath all of it sat the gate I was proudest of and that failed hardest. I hardened the TDD
140
+ rule into "no RED, no edit": you must see a real failing test before you touch code, and a test that
141
+ runs against an unbuilt interpreter does not count. Haiku read it as advisory. It edited compiled
142
+ source, tested the system Python instead of the source it changed, reported green, and emitted
143
+ `DEBUG_COMPLETE` anyway. No build, no RED, no hesitation.
144
+
145
+ > **You cannot wordsmith a weak model into rigor.** A prose gate is a request. Compliance with a
146
+ > request scales with the strength of the model reading it, which means the exact model that most
147
+ > needs the gate is the one most likely to ignore it.
148
+
149
+ ## The realization
150
+
151
+ The thing about a good eval is that it writes its own conclusions, and then you have to be honest
152
+ enough to read them.
153
+
154
+ The results doc had already said it, twice, in plain language. After the Round 3 failure it
155
+ concluded that stopping the timsort regression "needs an enforcement mechanism the harness owns,
156
+ not more SKILL.md text." And in the same breath it noted that the rejected prose gate was still
157
+ valuable, because "it is the spec for the harness-side gate."
158
+
159
+ I had been treating those sentences as a postmortem. They were a design brief. The fix was never
160
+ going to be words. It was going to be code that does not ask.
161
+
162
+ ## Round 4, the turn
163
+
164
+ So I re-implemented crg-debug as a Workflow: deterministic JavaScript that owns the control flow,
165
+ with agents reduced to what only they can do. The script owns phase sequencing, concern
166
+ partitioning, dedup, wave packing, loop termination, and the ledger. The agents own judgment and the
167
+ I/O the script cannot perform: graph queries, toolchain discovery, the baseline build, discovery,
168
+ adversarial verify, and the actual fixes.
169
+
170
+ The defining test for every gate became literal. Not "is this rule clear" but "can the model skip
171
+ it?" If a gate is a sentence in a prompt, the answer is yes. If a gate is a function the script calls
172
+ and an exit code the script reads, the answer is no.
173
+
174
+ The crux is the per-bug close gate. A fix agent applies its fix and reports the narrowest
175
+ `testCommand` that exercises only its bug. A separate agent then re-runs exactly that command and
176
+ reports the raw exit code. The script reads the exit code and decides closed or not closed. The model
177
+ never declares "passed." Its green claim is advisory; the exit code is authoritative.
178
+
179
+ > **A claim you cannot make is a claim you cannot fake.** The Round 3 gate failed because "passed"
180
+ > was a thing the model could say. The Round 4 gate works because "passed" is a thing only the shell
181
+ > can return.
182
+
183
+ This also cured an earlier deadlock I had hit with a whole-suite gate, where one unfixed sibling
184
+ failed the entire suite and blocked every genuine fix, stalling at 0 of 14 closed. Once a bug closes
185
+ on its own narrow test, siblings cannot hold it hostage.
186
+
187
+ ## The refactor that made it a command
188
+
189
+ Demoting the prose was the satisfying part. `SKILL.md` went from a 177 line harness, all
190
+ orchestration and anti-skip pleading, to a roughly 45 line dispatcher whose only job is to parse
191
+ arguments and call `Workflow({name:'crg-debug', args})`. The judgment prose that still mattered, the
192
+ bug-class checklist, the real-versus-intentional rule, the fail-safe-defaults lens, moved into a new
193
+ `methodology.md` reference that both the Workflow and the standalone debugger agent read. The
194
+ control-flow prose simply deleted, because the JavaScript now performs what it used to describe.
195
+
196
+ The whole thesis fits in one contrast. The skill was prose I was trusted not to skip. The Workflow
197
+ is gates the script owns. As a bonus, the dispatcher gained a uniform `model` argument applied to
198
+ every agent, something the prose skill never had.
199
+
200
+ ## The two surprises in validation
201
+
202
+ I validated on the BugFix repo with haiku as the driver, the same weak tier the prose rounds used,
203
+ so any gap would be a property of the harness and not a model upgrade. It took three runs and two
204
+ genuine surprises to get to green.
205
+
206
+ **Finding A: the repo was lying to the classifier.** Run 1 deferred four real bugs as "intentional."
207
+ The cause was the corpus repo's own framing. Its README announced "intentionally introduced bugs
208
+ for practice," and every defect carried a `# Buggy` comment. The adversarial verifier read those
209
+ tells and classified real contract violations as deliberate design, which is exactly the trap the
210
+ real-versus-intentional lens is meant to avoid. I rewrote the README as a real library, replaced the
211
+ comments with statements of the true contract, and kept every buggy line at its original position.
212
+ Deferred dropped from 4 to 0. The classifier had been reading the repo's marketing, and that
213
+ sensitivity is itself model-tier dependent: haiku reads the tells, stronger models weigh the contract
214
+ over the comment.
215
+
216
+ **Finding B: fixing the first surprise exposed a latent bug.** With the four bugs back in the queue,
217
+ `bug.py` now held seven bugs in one file, and Run 2 closed only 8 of 14 with a red gate, worse than
218
+ Run 1. The cleanup had not caused a regression. It had revealed one. Reading the loop, the cause was
219
+ two compounding defects. File-disjoint wave packing let one agent own a file per wave, so a dense
220
+ file could receive at most one fix per wave while its other bugs waited. And the thrash guard,
221
+ meant to detect genuine failed retries, had been adding bugs deferred merely because their file was
222
+ busy to the same "seen" set, then aborting the loop when they legitimately reappeared. Run 1 had only
223
+ looked green because four of `bug.py`'s bugs were hidden as intentional, thinning the file enough to
224
+ drain in time.
225
+
226
+ The fix restructured the fix phase: one agent now owns all of a file's bugs at once, a per-agent cap
227
+ sends a denser file's overflow to later waves as deferred chunks, and the thrash guard counts only
228
+ genuinely failed retries, never deferral. Run 3 closed everything. Zero unfixed, green gate, drained
229
+ in two waves.
230
+
231
+ It is worth sitting with the chain there. A misclassification born from a README masked a wave-packing
232
+ bug, which was masked again by a thrash guard that could not tell deferral from thrashing. Three
233
+ layers, each hiding the next, and only an honest run with all the bugs surfaced forced them out. The
234
+ same lesson as harness bug four, wearing different clothes: things that fail quietly compound.
235
+
236
+ ## What I actually learned
237
+
238
+ Pulling the threads together, the journey reduces to a handful of things I now believe more firmly
239
+ than when I started.
240
+
241
+ **Fail loud or do not measure.** The most dangerous output of a test harness is a confident wrong
242
+ number. If a path can swallow an error and emit a plausible result, it eventually will, and you will
243
+ trust it.
244
+
245
+ **The weak model is the signal, not the noise.** A self-improving loop learns from misses. Frontier
246
+ models on easy repos do not miss, so they teach nothing. Deliberately running the model that fails
247
+ is how you find what the method is missing.
248
+
249
+ **Prompt compliance scales with model strength, so a prose gate is a request and not a guarantee.**
250
+ The model that most needs a rule is the one most likely to read it as optional. This is the single
251
+ most useful thing the whole eval surfaced, and it generalizes well beyond debugging.
252
+
253
+ **When a rule must hold, move it into code the model cannot route around.** The difference between
254
+ Round 3 and Round 4 is the difference between asking the model to verify and letting only the shell
255
+ report the verdict. Enforcement belongs wherever the model cannot reach.
256
+
257
+ And the honest caveats, because the eval would not forgive me for omitting them. Round 4 is still
258
+ n=1, on one easy repo, with haiku. It proves the gate binds. It does not yet prove the Workflow
259
+ ceilings the hard cases. The Tier-1 set, timsort and asyncio and sudo, has not yet run through the
260
+ Workflow. That is the next run, and the standing limitations, N repeats per cell to separate signal
261
+ from stochasticity and harder targets to break the recall ceiling, are unchanged.
262
+
263
+ The through-line is almost embarrassing in hindsight. The eval kept telling me the answer in its own
264
+ conclusions, round after round. I just had to stop trying to say it in a prompt and start building it
265
+ in code.
266
+
267
+ ---
268
+
269
+ *For the numbers behind every claim here, see [CRG-EVAL-RESULTS.md](./CRG-EVAL-RESULTS.md): the
270
+ harness bugs in "Harness bugs found and fixed," the ceiling in "Findings," the hard cases in "Tier-1
271
+ hard cases," the prose-gate failures in Rounds 2 and 3, and the Workflow turn in Round 4.*