hpflow 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- hpflow-0.3.0/LICENSE +21 -0
- hpflow-0.3.0/PKG-INFO +393 -0
- hpflow-0.3.0/README.md +372 -0
- hpflow-0.3.0/hpflow/__init__.py +4 -0
- hpflow-0.3.0/hpflow/__main__.py +7 -0
- hpflow-0.3.0/hpflow/briefs.py +74 -0
- hpflow-0.3.0/hpflow/cli.py +148 -0
- hpflow-0.3.0/hpflow/commands.py +789 -0
- hpflow-0.3.0/hpflow/common.py +166 -0
- hpflow-0.3.0/hpflow/fixcheck.py +71 -0
- hpflow-0.3.0/hpflow/proof/__init__.py +9 -0
- hpflow-0.3.0/hpflow/proof/__main__.py +7 -0
- hpflow-0.3.0/hpflow/proof/checks.py +288 -0
- hpflow-0.3.0/hpflow/proof/cli.py +173 -0
- hpflow-0.3.0/hpflow/proof/domain.py +87 -0
- hpflow-0.3.0/hpflow/proof/generate.py +264 -0
- hpflow-0.3.0/hpflow/proof/lean.py +183 -0
- hpflow-0.3.0/hpflow/proof/link.py +167 -0
- hpflow-0.3.0/hpflow/proof/mutate.py +185 -0
- hpflow-0.3.0/hpflow/proof/pins.py +66 -0
- hpflow-0.3.0/hpflow/proof/question.py +52 -0
- hpflow-0.3.0/hpflow/proof/stubs.py +117 -0
- hpflow-0.3.0/hpflow/runner.py +129 -0
- hpflow-0.3.0/hpflow.egg-info/PKG-INFO +393 -0
- hpflow-0.3.0/hpflow.egg-info/SOURCES.txt +33 -0
- hpflow-0.3.0/hpflow.egg-info/dependency_links.txt +1 -0
- hpflow-0.3.0/hpflow.egg-info/entry_points.txt +4 -0
- hpflow-0.3.0/hpflow.egg-info/top_level.txt +1 -0
- hpflow-0.3.0/pyproject.toml +35 -0
- hpflow-0.3.0/setup.cfg +4 -0
- hpflow-0.3.0/tests/test_attacks.py +107 -0
- hpflow-0.3.0/tests/test_clamp.py +103 -0
- hpflow-0.3.0/tests/test_hpflow.py +475 -0
- hpflow-0.3.0/tests/test_quickstart.py +46 -0
- hpflow-0.3.0/tests/test_sync.py +120 -0
hpflow-0.3.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Cary Palmer
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
hpflow-0.3.0/PKG-INFO
ADDED
|
@@ -0,0 +1,393 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: hpflow
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Summary: A proof gate for one Python function (Lean 4), and an audit ladder that takes review findings to fixes.
|
|
5
|
+
Author: Cary Palmer
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://puppetmaster.gg/labs/hard-proof/
|
|
8
|
+
Project-URL: Repository, https://github.com/professorpalmer/hard-proof
|
|
9
|
+
Project-URL: Paper, https://puppetmaster.gg/labs/hard-proof/paper/
|
|
10
|
+
Keywords: lean4,formal-verification,code-review,ai-agents,release-gate
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
16
|
+
Classifier: Topic :: Software Development :: Testing
|
|
17
|
+
Requires-Python: >=3.9
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
License-File: LICENSE
|
|
20
|
+
Dynamic: license-file
|
|
21
|
+
|
|
22
|
+
# hpflow: a proof gate and an audit ladder for Python code
|
|
23
|
+
|
|
24
|
+
This package has two commands.
|
|
25
|
+
|
|
26
|
+
- `hpf` is the audit ladder. It takes the Python functions that changed in a
|
|
27
|
+
git range up a ladder of evidence: a review finding with a runnable repro,
|
|
28
|
+
a script run of the repro, two verifier agents, a fix with a test, and a
|
|
29
|
+
gate before the release. See the section "hpflow" below.
|
|
30
|
+
- `hp` is the proof gate. It is one proof gate for one Python function, with
|
|
31
|
+
Lean 4. The first sections of this file are about `hp`.
|
|
32
|
+
|
|
33
|
+
The two come from the study "Proved is not finished"
|
|
34
|
+
(https://puppetmaster.gg/labs/hard-proof/). The study has the data, the costs
|
|
35
|
+
and the limits. Source: https://github.com/professorpalmer/hard-proof
|
|
36
|
+
|
|
37
|
+
`hp` replaces the hand-written gate scripts of the study
|
|
38
|
+
(`exp12/base/gate12.py` and the scripts before it) with one command set and
|
|
39
|
+
one domain contract. A new Python function gets a proof gate with no new gate
|
|
40
|
+
code.
|
|
41
|
+
|
|
42
|
+
The tool uses Lean 4.36.0 with no Mathlib. `lake comparator` does the
|
|
43
|
+
statement match, the axiom audit and the kernel replay. The tool is Python 3.9+
|
|
44
|
+
with the standard library only. It makes no model call.
|
|
45
|
+
|
|
46
|
+
## Install
|
|
47
|
+
|
|
48
|
+
```sh
|
|
49
|
+
pip install hpflow
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
This gives three commands: `hp` (the proof gate), and `hpf` and `hpflow` (the
|
|
53
|
+
audit ladder). The package has no Python dependency. From a checkout of the
|
|
54
|
+
repository, `./hp` and `./hpf` in the folder `tool` do the same with no
|
|
55
|
+
install.
|
|
56
|
+
|
|
57
|
+
The name of the package is `hpflow`. The name `hardproof` on PyPI is a
|
|
58
|
+
different project, and it is not related to these tools.
|
|
59
|
+
|
|
60
|
+
| Command | Needs |
|
|
61
|
+
| --- | --- |
|
|
62
|
+
| `hp` | Lean 4.36.0 through `elan`. No model call. |
|
|
63
|
+
| `hpf audit`, `hpf fix` | The Puppetmaster command line (`pip install puppetmaster-ai`) and Claude Code with a login. These steps make model calls. |
|
|
64
|
+
| `hpf gate`, `hpf waive`, `hpf status` | git and Python only. No model call. |
|
|
65
|
+
|
|
66
|
+
## What the gate shows
|
|
67
|
+
|
|
68
|
+
A domain has a Lean mirror `ref` of the Python function and a set of laws
|
|
69
|
+
written from the documentation. The gate accepts certificates that untrusted
|
|
70
|
+
agents write:
|
|
71
|
+
|
|
72
|
+
- The link test shows that `ref` and the real function give the same output on
|
|
73
|
+
many random legal inputs, and that the test kills mutants of the function.
|
|
74
|
+
- A `ref` certificate proves that `ref` passes each law.
|
|
75
|
+
- A `conflict` certificate proves that `ref` breaks one law on a legal input.
|
|
76
|
+
The law or the code is wrong. A human decides.
|
|
77
|
+
- A `gap` certificate gives two designs that pass each law and differ on a
|
|
78
|
+
legal input. The laws do not decide that input. A human decides.
|
|
79
|
+
- A `complete` certificate proves that two designs that pass each law agree on
|
|
80
|
+
each legal input. Then the laws pin the behavior down.
|
|
81
|
+
|
|
82
|
+
## The domain contract
|
|
83
|
+
|
|
84
|
+
A domain is one folder for one function under test. The function has the type
|
|
85
|
+
`Impl := Input → Output` in Lean. A stateful system is out of scope; put the
|
|
86
|
+
state into `Input`.
|
|
87
|
+
|
|
88
|
+
Author files (an agent or a human writes them; then `pin` records them):
|
|
89
|
+
|
|
90
|
+
| File | Content |
|
|
91
|
+
| --- | --- |
|
|
92
|
+
| `hardproof.json` | `{"name": "...", "namespace": "HP", "python": {"module": "pkg.mod", "function": "name"}, "banned_in_laws": ["ref", ...]}` |
|
|
93
|
+
| `Types.lean` | namespace `HP`: `Input`, `Output` (each with `DecidableEq`), `abbrev Impl := Input → Output`, `def legal : Input → Bool`, `def Legal (x : Input) : Prop := legal x = true`, and the words that a law can use. No `ref` here. |
|
|
94
|
+
| `Ref.lean` | imports `Types`. `def HP.ref : HP.Impl`, the mirror of the Python code, and its helper functions. |
|
|
95
|
+
| `Codec.lean` | imports `Types`. `HP.decodeInput : String → Option Input`, `HP.encodeInput`, `HP.encodeOutput`, `HP.showInput`, `HP.showOutput`. One line of text for each value. |
|
|
96
|
+
| `link_adapter.py` | `gen(rng)`, `encode(case)`, `decode(line)`, `call(func, case)`, `encode_result(result)`, `is_legal(case)`. |
|
|
97
|
+
| `DESCRIPTION.md` | The sentences of the documentation, each an exact quote with its source file. Then a map from the model words to the documentation words. |
|
|
98
|
+
| `Laws.lean` | imports `Types` only. One `def <Name> (f : Impl) : Prop` for each law, `def Passes (f : Impl) : Prop` (the conjunction of all laws), and `theorem some_design_fails : ∃ f : Impl, ¬ Passes f`. |
|
|
99
|
+
|
|
100
|
+
Solution files (an untrusted agent writes them; the comparator checks them):
|
|
101
|
+
|
|
102
|
+
| File | Challenge statement |
|
|
103
|
+
| --- | --- |
|
|
104
|
+
| `RefPassesSolution.lean` | `theorem HPRef.ref_passes : Passes ref` |
|
|
105
|
+
| `ConflictSolution.lean` | first line `-- hardproof-law: <Law>`; `def HPConflict.input : Input`, `theorem HPConflict.input_legal : Legal input`, `theorem HPConflict.ref_breaks : ¬ <Law> ref` |
|
|
106
|
+
| `GapSolution.lean` | `def HPGap.designA`, `def HPGap.designB : Impl`, `def HPGap.input : Input`, `theorem HPGap.a_passes : Passes designA`, `theorem HPGap.b_passes : Passes designB`, `theorem HPGap.input_legal : Legal input`, `theorem HPGap.differ : designA input ≠ designB input` |
|
|
107
|
+
| `CompleteSolution.lean` | `theorem HPComplete.laws_complete : ∀ f g : Impl, Passes f → Passes g → ∀ x : Input, Legal x → f x = g x` |
|
|
108
|
+
|
|
109
|
+
Tool-owned files are in `hp/`: `lakefile.toml`, `lean-toolchain`,
|
|
110
|
+
`lake-manifest.json`, the challenge modules, the comparator configs,
|
|
111
|
+
`Runner.lean`, `QuestionGap.lean` and `QuestionConflict.lean`. The tool writes
|
|
112
|
+
them again before each check, so an edit to one of them has no effect. The
|
|
113
|
+
lakefile has one `lean_lib` for each solution, so a broken solution of one kind
|
|
114
|
+
does not stop the check of another kind.
|
|
115
|
+
|
|
116
|
+
## Quick start (on `examples/clamp`)
|
|
117
|
+
|
|
118
|
+
`examples/clamp` is a pure function `clamp(x, lo, hi)` in
|
|
119
|
+
`examples/clamp/pycode/clampmod.py`. It needs no outside repository. The examples are in the
|
|
120
|
+
repository, not in the installed package.
|
|
121
|
+
|
|
122
|
+
1. Copy the example to a new folder and go to it:
|
|
123
|
+
|
|
124
|
+
```sh
|
|
125
|
+
cp -R examples/clamp /tmp/clamp-demo
|
|
126
|
+
cd /tmp/clamp-demo
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
2. Set `HP` to the command: `HP=hp` after an install, or the full path of
|
|
130
|
+
the launcher `hp` in the folder `tool` of a checkout.
|
|
131
|
+
3. Check the model (types, mirror, codec, runner and the codec round trip):
|
|
132
|
+
|
|
133
|
+
```sh
|
|
134
|
+
$HP check model
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
4. Run the link test against the real function:
|
|
138
|
+
|
|
139
|
+
```sh
|
|
140
|
+
$HP link --repo pycode --cases 5000
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
5. Check the laws and the `ref` certificate:
|
|
144
|
+
|
|
145
|
+
```sh
|
|
146
|
+
$HP check laws ref
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
6. Pin the files, and check the pins:
|
|
150
|
+
|
|
151
|
+
```sh
|
|
152
|
+
$HP pin
|
|
153
|
+
$HP verify-pins
|
|
154
|
+
$HP status
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Each command prints one JSON report. The exit code is 0 for accept (or
|
|
158
|
+
`agree`), 1 for reject, and 2 for a usage error.
|
|
159
|
+
|
|
160
|
+
## Commands
|
|
161
|
+
|
|
162
|
+
All commands take `--dir PATH` (the domain folder; the default is the current
|
|
163
|
+
folder) and `--actor NAME` (the name in the run log). Run them as
|
|
164
|
+
`python3 -m hpflow.proof <command>` with `PYTHONPATH` set to this folder, or as
|
|
165
|
+
`./hp <command>`.
|
|
166
|
+
|
|
167
|
+
- `new DIR --name NAME --module PKG.MOD --function NAME`: make a domain folder
|
|
168
|
+
with author stubs (each holds short instructions) and the tool-owned files.
|
|
169
|
+
- `generate`: write the tool-owned files again.
|
|
170
|
+
- `check model`: `Types`, `Ref`, `Codec` and `Runner` build. No `sorry`, no
|
|
171
|
+
`axiom`, no `native_decide`, no `implemented_by`. The declarations have the
|
|
172
|
+
types of the contract. The codec round trip holds on 200 cases from
|
|
173
|
+
`link_adapter.gen`: `decode(encode(c)) == c`, and the Lean runner decodes
|
|
174
|
+
each line.
|
|
175
|
+
- `link --repo PATH [--cases N] [--seed N] [--mutants N] [--function-file PATH]`:
|
|
176
|
+
the link test. For each generated legal case, the Lean `ref` and the real
|
|
177
|
+
function give the same output line. Then up to N mutants of the function
|
|
178
|
+
source run on the same cases. The report gives the cases, the differences
|
|
179
|
+
(must be 0), the first difference, the mutants made and killed, the kill
|
|
180
|
+
rate, and each mutant that survives. The verdict is `agree` only when the
|
|
181
|
+
differences are 0. The kill rate is reported, not judged. With
|
|
182
|
+
`--function-file`, the function comes from that file and not from the module
|
|
183
|
+
in `hardproof.json`.
|
|
184
|
+
- `check laws`: the text rules (imports `Types` only; no banned word of
|
|
185
|
+
`hardproof.json` before `theorem some_design_fails`; no `sorry`, `axiom` or
|
|
186
|
+
`native_decide`), the build, the types of `Passes` and `some_design_fails`,
|
|
187
|
+
and the axiom audit (`propext`, `Quot.sound`, `Classical.choice` only). The
|
|
188
|
+
report lists the law names.
|
|
189
|
+
- `check ref`, `check conflict`, `check gap`, `check complete`: build the
|
|
190
|
+
solution, check that each required name exists, then run
|
|
191
|
+
`lake comparator`. `--paranoid` runs the five extra kernels.
|
|
192
|
+
`--inadvisably-no-sandbox` is added when `bwrap` is absent (macOS). For
|
|
193
|
+
`conflict`, the named law must be a law of `Laws.lean`. You can name more
|
|
194
|
+
than one kind; the check accepts when one kind is accepted.
|
|
195
|
+
- `question [gap|conflict]`: for an accepted gap, the encoded and shown input
|
|
196
|
+
and the outputs of design A, design B and `ref`. For an accepted conflict,
|
|
197
|
+
the law, the input and the output of `ref`.
|
|
198
|
+
- `pin`: write `hardproof.lock.json` with the SHA-256 of each author file and
|
|
199
|
+
each generated file.
|
|
200
|
+
- `verify-pins [--expect FILE=SHA ...]`: reject when a pinned file differs.
|
|
201
|
+
With `--expect`, check the given values (for example from a flow graph).
|
|
202
|
+
- `status`: what exists, and what the last check of each kind said.
|
|
203
|
+
|
|
204
|
+
Each `check` and each `link` appends one line to `hardproof_runs.jsonl` in the
|
|
205
|
+
domain folder: time, actor, command, verdict and seconds.
|
|
206
|
+
|
|
207
|
+
## The mutator
|
|
208
|
+
|
|
209
|
+
`link` mutates the source of the function with its own AST mutator. Each
|
|
210
|
+
mutant changes one place:
|
|
211
|
+
|
|
212
|
+
- negate the condition of an `if`, `while`, conditional expression or `assert`,
|
|
213
|
+
- swap `and` and `or`,
|
|
214
|
+
- change a comparison operator,
|
|
215
|
+
- change an integer constant by one,
|
|
216
|
+
- replace a `return` value by `None`,
|
|
217
|
+
- delete one statement,
|
|
218
|
+
- swap two positional arguments, or the values of two keyword arguments, of one call.
|
|
219
|
+
|
|
220
|
+
A mutant is killed when one case gives a different line, or when it raises or
|
|
221
|
+
runs for more than 30 seconds. The report also says if the unchanged source,
|
|
222
|
+
compiled the same way, agrees (`identity_copy_agrees`).
|
|
223
|
+
|
|
224
|
+
## Examples
|
|
225
|
+
|
|
226
|
+
- `examples/clamp`: `clamp(x, lo, hi)` over integers, with four laws and a
|
|
227
|
+
`ref` certificate.
|
|
228
|
+
- `examples/sync`: `puppetmaster models sync` from experiment 12. `Input` holds
|
|
229
|
+
the `--enable` flag, the locked adapters, the starter list and the registry.
|
|
230
|
+
`Legal` is the corrected rule: the starter ids are pairwise distinct. The
|
|
231
|
+
seven laws have no law for the order of the new rows, so `GapSolution.lean`
|
|
232
|
+
shows a gap. `variants/order` adds an order law and a completeness proof.
|
|
233
|
+
`variants/false_law` adds a false law and a conflict certificate. The link
|
|
234
|
+
test needs a Puppetmaster clone:
|
|
235
|
+
|
|
236
|
+
```sh
|
|
237
|
+
./hp --dir examples/sync link --repo ~/Projects/hp-work/Puppetmaster
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
## Tests
|
|
241
|
+
|
|
242
|
+
```sh
|
|
243
|
+
python3 -m unittest discover tests
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
The tests make no model call. The Puppetmaster link test is skipped with a
|
|
247
|
+
message when the clone at `~/Projects/hp-work/Puppetmaster` is absent.
|
|
248
|
+
|
|
249
|
+
## hpflow
|
|
250
|
+
|
|
251
|
+
`hpflow` is the second package of this folder. It runs agent workers through
|
|
252
|
+
the Puppetmaster command line and takes a set of Python functions up an
|
|
253
|
+
evidence ladder. The proof gate `hp` makes no model call. `hpflow` does.
|
|
254
|
+
|
|
255
|
+
| Grade | Evidence | Who gives it |
|
|
256
|
+
| --- | --- | --- |
|
|
257
|
+
| 0 | A finding with a runnable repro | a reviewer agent |
|
|
258
|
+
| 1 | The repro runs on the source under audit and gives the result that the finding states | a script |
|
|
259
|
+
| 2 | Two verifier agents on different models each say `real`, and name a caller that reaches it | two agents |
|
|
260
|
+
| 3 | A fix with a regression test, a script check and an independent judge | agents and a script |
|
|
261
|
+
|
|
262
|
+
### Quick start
|
|
263
|
+
|
|
264
|
+
```sh
|
|
265
|
+
./hpf audit --repo . --ref HEAD --changed main..HEAD --out /tmp/audit
|
|
266
|
+
./hpf fix --out /tmp/audit
|
|
267
|
+
./hpf status --out /tmp/audit
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
Then read `/tmp/audit/REPORT.md` and the worktrees in `/tmp/audit/fix/`.
|
|
271
|
+
|
|
272
|
+
### Use it as a gate before a release or a merge
|
|
273
|
+
|
|
274
|
+
The audit looks only at the functions that changed in a git range, so its cost
|
|
275
|
+
follows the size of the change. The gate is the last step. It makes no model
|
|
276
|
+
call, and its exit code is 0 for a pass and 1 for a fail.
|
|
277
|
+
|
|
278
|
+
```sh
|
|
279
|
+
hpf audit --repo . --ref HEAD --changed v1.2.3..HEAD --out ~/audits/next
|
|
280
|
+
hpf fix --out ~/audits/next # only when REPORT.md has confirmed bugs
|
|
281
|
+
# Review each worktree in ~/audits/next/fix/, then commit and merge the fixes yourself.
|
|
282
|
+
hpf gate --out ~/audits/next --ref HEAD
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
`gate --ref REF` runs the repro of each confirmed bug on the commit `REF`. A
|
|
286
|
+
bug is closed when each of its repros gives a new result there. So the gate
|
|
287
|
+
checks the code that you will release, not the worktree of a fix worker. With
|
|
288
|
+
no `--ref`, a bug is closed when it has a fix with an accepted check and a
|
|
289
|
+
PASS judge.
|
|
290
|
+
|
|
291
|
+
When the owner decides that a confirmed bug stays as it is, record it:
|
|
292
|
+
|
|
293
|
+
```sh
|
|
294
|
+
hpf waive --out ~/audits/next --bug ID --reason "why the behavior stays" --by NAME
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
The gate fails when a review or a verifier run is absent, when a confirmed
|
|
298
|
+
bug is open, or when a repro does not run on `REF`. A range with no changed
|
|
299
|
+
function passes with no worker run.
|
|
300
|
+
|
|
301
|
+
What we measured on our two products (API-equivalent cost, Claude models):
|
|
302
|
+
about 1.40 US dollars for each changed function for the review and the two
|
|
303
|
+
verifiers (28.53 US dollars for 20 functions), and 36 US dollars for one release range of 20 functions with
|
|
304
|
+
its 7 fixes. On that audit, `gate --ref` failed on the audited release with 7
|
|
305
|
+
open bugs and passed on the next release with 7 closed.
|
|
306
|
+
|
|
307
|
+
### Commands
|
|
308
|
+
|
|
309
|
+
All commands take `--out DIR` (the audit folder), `--redo` and
|
|
310
|
+
`--concurrency N` (default 5: the most worker flows that run at one time).
|
|
311
|
+
Each command that `hpflow` starts (a repro, a test run, git) runs in its own
|
|
312
|
+
process group with a timeout. At the timeout the whole group ends. Run them as `./hpf <command>` or as
|
|
313
|
+
`python3 -m hpflow <command>` with `PYTHONPATH` set to this folder. Each
|
|
314
|
+
command prints one JSON report. The exit code is 0 on success, 1 on a failed
|
|
315
|
+
step, and 2 on a usage error.
|
|
316
|
+
|
|
317
|
+
All state is in the audit folder, as JSON files. A command can run again: it
|
|
318
|
+
does not repeat a worker run that has a stored result. `--redo` makes the
|
|
319
|
+
runs again. A stored run with an error (launch or wait) runs again.
|
|
320
|
+
|
|
321
|
+
- `init --repo PATH --ref REF [--paths ...] [--python PATH] [--test-cmd "..."]`:
|
|
322
|
+
write `audit.json` and the snapshot `src/` (`git archive` of the paths at
|
|
323
|
+
the commit of `REF`). The default paths are each top-level folder with an
|
|
324
|
+
`__init__.py`, and `tests`, `docs`, `README.md` and `pyproject.toml` when
|
|
325
|
+
they are present. The default Python is `.venv/bin/python` of the
|
|
326
|
+
repository, else the Python that runs `hpflow`. The test command is a
|
|
327
|
+
template with `{python}` and `{tests}`; the default is
|
|
328
|
+
`{python} -m pytest -q -p no:cacheprovider {tests}`.
|
|
329
|
+
- `targets (--changed A..B | --function FILE:NAME ... | --file FILE ...) [--max-lines 150] [--max N]`:
|
|
330
|
+
write `targets.json`. `--changed` takes the functions and methods (from the
|
|
331
|
+
`B` side) of non-test Python files that overlap a hunk of `git diff A..B`.
|
|
332
|
+
The report lists the skipped functions with the reason: too long, test
|
|
333
|
+
file, removed, or over `--max`.
|
|
334
|
+
- `review [--model sonnet] [--shell | --no-shell] [--passes 1] [--repro-timeout 60]`:
|
|
335
|
+
one reviewer run for each target and pass. Each finding has a repro file
|
|
336
|
+
`finding_<n>.py` with `run()`. A script runs each repro on `src/` in a
|
|
337
|
+
separate process. A finding has grade 1 when `run()` returns. Else it stays
|
|
338
|
+
at grade 0 with the reason (no repro file, an import error, a timeout, or
|
|
339
|
+
an exception).
|
|
340
|
+
- `verify [--models opus sonnet]`: one verifier run for each target with
|
|
341
|
+
grade-1 findings and each model. A finding has grade 2 when each verifier
|
|
342
|
+
says `real`. Duplicates fold into one bug (the `duplicate_of` field of the
|
|
343
|
+
first model). Writes `verify.json`, `confirmed.json` and `REPORT.md` (one
|
|
344
|
+
table for each class: confirmed, real but not reachable, by design, mixed).
|
|
345
|
+
- `fix [--base REF] [--only BUG ...] [--model opus] [--judge-model sonnet] [--rules TEXT]`:
|
|
346
|
+
for each source file with confirmed bugs, a git worktree `fix/<slug>` on a
|
|
347
|
+
new branch `hpfix/<slug>` from `--base` (default: the audited commit), in
|
|
348
|
+
the audited repository. A build worker writes the fix and the test file
|
|
349
|
+
`tests/test_hpfix_<slug>.py`. A script check (`python3 -m hpflow.fixcheck`)
|
|
350
|
+
runs each repro (it must give a new result), the new test file and the
|
|
351
|
+
related tests. A judge reviews the diff with a rubric. A failed check or a
|
|
352
|
+
FAIL or PARTIAL review goes back to the builder. `hpflow` never commits,
|
|
353
|
+
pushes or merges. The report lists each worktree, its diff stat, its new
|
|
354
|
+
files, the check verdict and the judge verdict.
|
|
355
|
+
- `audit ...`: `init`, `targets`, `review` and `verify` in one call, with the
|
|
356
|
+
options of each. It stops at the first step that fails.
|
|
357
|
+
- `status`: the count of findings at each grade, the cost and time of each
|
|
358
|
+
step, and what is not done.
|
|
359
|
+
- `waive --bug ID --reason TEXT [--by NAME]`: record in `waivers.json` that a
|
|
360
|
+
confirmed bug stays as it is. A waiver needs a reason.
|
|
361
|
+
- `gate [--ref REF] [--repro-timeout 60]`: pass only when the audit is
|
|
362
|
+
complete and each confirmed bug is closed or waived. The report lists each
|
|
363
|
+
bug with its state and the reason. No model call.
|
|
364
|
+
|
|
365
|
+
Model names: `haiku`, `sonnet` and `opus` map to `claude-haiku-5-5`,
|
|
366
|
+
`claude-sonnet-5-5` and `claude-opus-5-5`. A full model id passes through.
|
|
367
|
+
The adapter is `claude-code`.
|
|
368
|
+
|
|
369
|
+
Workers get a snapshot of the source, never the repository of the user. Each
|
|
370
|
+
run folder is `runs/<digits>`, and each worker gets a copy of `src/` as
|
|
371
|
+
`project/`. The name holds no word and no function name.
|
|
372
|
+
|
|
373
|
+
Cost is API-equivalent, as the provider tool reports it. Plan billing can
|
|
374
|
+
apply. The tool reports the cost of a resumed session as the total of the
|
|
375
|
+
session so far. So the cost of a node is the largest value of its visits, and
|
|
376
|
+
the cost of one visit is the difference from the visit before it.
|
|
377
|
+
|
|
378
|
+
### Limits
|
|
379
|
+
|
|
380
|
+
- Grade 2 is the agreement of two agents. It is not a proof.
|
|
381
|
+
- A human owner of the code decides what is a bug, and reviews and commits
|
|
382
|
+
each fix.
|
|
383
|
+
- Grade 1 shows only that the repro runs and gives a result. It does not show
|
|
384
|
+
that the result is wrong.
|
|
385
|
+
- A pass of the gate says that each confirmed bug is closed or waived. It
|
|
386
|
+
does not say that the code has no bug: in our study the best review arm
|
|
387
|
+
found 44 of 95 known bugs.
|
|
388
|
+
- The fix step can make a wrong fix that passes its check and its judge (5 of
|
|
389
|
+
45 in our study). Review each fix before you merge it.
|
|
390
|
+
- Python only. The reviewer and the verifiers are Claude models through
|
|
391
|
+
Claude Code. Other adapters are not tested.
|
|
392
|
+
- Lean certificates (the proof gate `hp`) are not part of the ladder. Use
|
|
393
|
+
`hp` for a small rule that the owner states, for example a decision table.
|