@codacy/verity-cli 0.33.3-experimental.52f1656 → 0.33.3-experimental.c86028a
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/verity.js +1203 -4773
- package/package.json +1 -1
- package/data/skills/verity-prove/SKILL.md +0 -174
package/package.json
CHANGED
|
@@ -1,174 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
description: >-
|
|
3
|
-
Formally verify a small, risky piece of code with a Lean 4 model: pick candidates from git
|
|
4
|
-
history, CONFIRM with the user which code to model, write the model, run it locally with
|
|
5
|
-
`verity prove check`, and replay any counterexample on the real code before calling it a bug.
|
|
6
|
-
Use when the user asks to verify, prove, or model-check code, or asks "can this race / double
|
|
7
|
-
run / fail open?". Anything typed after /verity-prove is the user's hint: the file, function,
|
|
8
|
-
or worry to check.
|
|
9
|
-
---
|
|
10
|
-
# /verity-prove — Formal verification of small pieces of code
|
|
11
|
-
|
|
12
|
-
**The words after `/verity-prove` are the hint.** `/verity-prove` alone means "find me
|
|
13
|
-
something worth checking"; `/verity-prove can the Stripe webhook double-charge on a
|
|
14
|
-
redelivered event?` means that is the target and, if it is a worry, the property.
|
|
15
|
-
Carry it through every step below as `$ARGUMENTS`.
|
|
16
|
-
|
|
17
|
-
Verity keeps **Lean 4 models of the riskiest code**, not of the codebase. A model is
|
|
18
|
-
another tool for finding bugs: a few dozen lines that abstract ONE function and check
|
|
19
|
-
a few properties by exhaustive bounded search. When a property fails you get the
|
|
20
|
-
**shortest sequence of actions** that breaks it — a hypothesis to replay on the real
|
|
21
|
-
code. When it holds you know exactly how far it was checked.
|
|
22
|
-
|
|
23
|
-
This skill is the **explicit, user-driven** entry; nothing in it blocks a commit or a
|
|
24
|
-
push. (The push-time phase — `verity config formal push` — is a separate, opt-in path
|
|
25
|
-
that uses the same tools.) Lean runs on this machine because the user asked for it.
|
|
26
|
-
|
|
27
|
-
## The flow
|
|
28
|
-
|
|
29
|
-
### 1. Find what deserves a model
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
verity prove candidates --limit 8 # git history + risk signals
|
|
33
|
-
verity prove candidates --rank luna --json # + a cheap model re-ranks and names a symbol
|
|
34
|
-
verity prove candidates src/payments # scoped to a path
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
Each candidate says why: fix commits over commits, check-then-act, retries, caches,
|
|
38
|
-
locks/leases, status transitions, money.
|
|
39
|
-
|
|
40
|
-
**With a hint** (`$ARGUMENTS` non-empty) — a file, a function, or a worry in the user's
|
|
41
|
-
own words ("can the webhook double-charge?", "does the retry ever lose the feedback?"):
|
|
42
|
-
|
|
43
|
-
```bash
|
|
44
|
-
verity prove candidates --hint "$ARGUMENTS"
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
A named file or function skips the ranking entirely: go to step 2 with it. A *worry*
|
|
48
|
-
becomes an `intent` property in step 4, with `source := "user, <date>: <their words>"`;
|
|
49
|
-
it needs no further confirmation, they just gave it. The candidates list is still shown,
|
|
50
|
-
so the user sees what else the history says — but the hint decides.
|
|
51
|
-
|
|
52
|
-
### 2. Confirm with the user — always
|
|
53
|
-
|
|
54
|
-
**Do not write a model until the user has said which code.** Present at most three
|
|
55
|
-
candidates in one short message, each as: file, the function you would model, the
|
|
56
|
-
property you would check first, and the reason from the signals. Then ask. The user may
|
|
57
|
-
pick one, name something else, or narrow the property. Their answer is the scope; do not
|
|
58
|
-
widen it.
|
|
59
|
-
|
|
60
|
-
A model is a claim about what the code *means*. Only the person who owns the code can
|
|
61
|
-
say whether that claim is worth checking.
|
|
62
|
-
|
|
63
|
-
### 3. Set up the project (once per repo)
|
|
64
|
-
|
|
65
|
-
```bash
|
|
66
|
-
verity prove init
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
Creates `.verity/proofs/` — a Lake project with Verity's library (`Verity.System`,
|
|
70
|
-
`Verity.check`, a bounded explorer). Re-running refreshes the library and never touches
|
|
71
|
-
`Models/`. If Lean is missing, install it only after telling the user what will be
|
|
72
|
-
installed and where: `verity lean install` (official release, ≈800 MB download, 2.7 GB
|
|
73
|
-
on disk, into `~/.verity/lean`). Ranking (`--hint`, `--rank luna`) and `verity prove
|
|
74
|
-
replay` run on the Verity service with the user's login (`verity login`); without a
|
|
75
|
-
login, ranking is deterministic and replay is by hand (step 6.2).
|
|
76
|
-
|
|
77
|
-
### 4. Write the model — `Models/<Name>.lean`
|
|
78
|
-
|
|
79
|
-
Rules that keep models small and honest:
|
|
80
|
-
|
|
81
|
-
- **One symbol.** The header names it: `Verity.model { name, abstracts := [{ file, symbol, sha256 }] }`.
|
|
82
|
-
Get the hash with `verity prove hash <file>`. `verity prove check` reports drift when the file changes.
|
|
83
|
-
When the code is a *region* rather than a named function (a branch inside a large
|
|
84
|
-
handler), add `lines := "a-b"` (1-based, inclusive) — `verity prove replay` needs it
|
|
85
|
-
to know which real lines must run.
|
|
86
|
-
- **State = only what the property reads.** Actions = the operations that can interleave
|
|
87
|
-
(a retry, a second call, a concurrent writer). Steps = what the code does, not what it should.
|
|
88
|
-
- **Properties are tagged.** `kind := .structural` for the catalogue (idempotence, no lost
|
|
89
|
-
update, status only moves forward, retries terminate, never fails open, bounded size,
|
|
90
|
-
check-then-act is atomic, no double effect). `kind := .intent` **only** when a human
|
|
91
|
-
wrote the expectation down — a doc comment, an ADR, a ticket, a Standard rule — and
|
|
92
|
-
`source :=` cites it. Never invent intent.
|
|
93
|
-
- **End with the checks.** `#eval Verity.model …` then one `#eval Verity.check …` per
|
|
94
|
-
property (bounded exploration), `Verity.checkData` for input properties, `Verity.proved`
|
|
95
|
-
only beside a real theorem. `sorry` fails the run.
|
|
96
|
-
- Keep it under ~120 lines. If it wants more, the scope is wrong — go back to step 2.
|
|
97
|
-
|
|
98
|
-
`Models/README.md` has the skeleton; `cli/tests/fixtures/proofs/SeedRunner.lean` in
|
|
99
|
-
Verity's own repo is a worked example (the VRT-243 seed-duplication bug).
|
|
100
|
-
|
|
101
|
-
### 5. Check it
|
|
102
|
-
|
|
103
|
-
```bash
|
|
104
|
-
verity prove check <Name> # one model
|
|
105
|
-
verity prove check # every model
|
|
106
|
-
verity prove check <Name> --json # for scripts
|
|
107
|
-
```
|
|
108
|
-
|
|
109
|
-
Per property: `✓ holds to bound k`, `✓ proved`, `✗ COUNTEREXAMPLE` with the trace and
|
|
110
|
-
the state after each step, or `? inconclusive` (state cap).
|
|
111
|
-
|
|
112
|
-
**Hard caps — never loop:**
|
|
113
|
-
- elaboration errors: at most **3** fix rounds, then stop and say the model could not be
|
|
114
|
-
built;
|
|
115
|
-
- a counterexample that **diverges** on replay (the model was wrong about the code): at
|
|
116
|
-
most **2** model revisions, each naming what the replay showed was different; after
|
|
117
|
-
that, stop and report "no faithful model found for `<symbol>`" — that is a valid result;
|
|
118
|
-
- never weaken or delete a property to make it pass; never add a property to make
|
|
119
|
-
something fail.
|
|
120
|
-
|
|
121
|
-
A wrong model is expected some of the time. It costs a replay and is discarded; it never
|
|
122
|
-
costs a false bug, because only a *reproduced* counterexample is reported (step 6).
|
|
123
|
-
|
|
124
|
-
### 6. Replay before you call it a bug
|
|
125
|
-
|
|
126
|
-
A counterexample is a fact about the **model**. Before reporting a bug:
|
|
127
|
-
|
|
128
|
-
1. Run the replay in OpenAI's disposable container (network off, deleted afterwards):
|
|
129
|
-
|
|
130
|
-
```bash
|
|
131
|
-
verity prove replay <Name> # every counterexample
|
|
132
|
-
verity prove replay <Name> --property P2 # one
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
It hands a model the trace, the Lean file and the real source — the abstracted
|
|
136
|
-
file's content leaves the machine for the Verity service and OpenAI's container;
|
|
137
|
-
only git-tracked files are ever read — and requires three things before accepting
|
|
138
|
-
its answer: code actually ran (a shell call with output), the real function's lines
|
|
139
|
-
appear verbatim in what it wrote, and the harness itself printed
|
|
140
|
-
`REPLAY_RESULT=reproduced`. `REPRODUCED` → the
|
|
141
|
-
real code does this. `DIVERGED` → the model is wrong about the code. `UNVERIFIED` →
|
|
142
|
-
not evidence either way. `UNREPLAYABLE` → it could not run the function faithfully.
|
|
143
|
-
Read its reason: the stubs it chose are where a false reproduction hides.
|
|
144
|
-
2. When the replay is unverified or unreplayable — or the code needs a runtime the
|
|
145
|
-
container lacks (Deno edge functions, a real database) — replay by hand: read the
|
|
146
|
-
real function again, map every trace action to a real call, and reproduce it with a
|
|
147
|
-
throwaway harness in a scratch directory that loads the real module and stubs only
|
|
148
|
-
I/O boundaries. **Never edit the user's source files to do this.**
|
|
149
|
-
3. Reproduced → report it as a bug, with the trace and the reproduction. Not reproduced
|
|
150
|
-
→ the model is wrong (`model-divergence`): say so, fix the model, do not report a bug.
|
|
151
|
-
|
|
152
|
-
This is the rule Verity applies everywhere (memory: *replay formal counterexamples
|
|
153
|
-
against changed code before blocking*). A model that finds a bug in itself is noise.
|
|
154
|
-
|
|
155
|
-
### 7. Report, in a few lines
|
|
156
|
-
|
|
157
|
-
State what was modelled, what was checked to what bound, what was refuted and whether
|
|
158
|
-
it replayed, and what remains `intent` awaiting the user's confirmation. **Never offer
|
|
159
|
-
to commit the model** — `.verity/proofs/` is machine-local and ignores itself; models
|
|
160
|
-
would only confuse reviewers. What reaches the team is the bug report and, when asked,
|
|
161
|
-
the fix. Then record the outcome:
|
|
162
|
-
|
|
163
|
-
```bash
|
|
164
|
-
verity reflect --user-input "Modelled runSeed (cli/src/lib/seed-runner.ts): P1 at-most-once refuted in 2 steps, replayed — guard checks a path the server never writes." --kind gotcha
|
|
165
|
-
```
|
|
166
|
-
|
|
167
|
-
## Do not
|
|
168
|
-
|
|
169
|
-
- Model code the user did not confirm.
|
|
170
|
-
- Present a model-only counterexample as a bug.
|
|
171
|
-
- Put `sorry` in a model, or weaken a property to get green.
|
|
172
|
-
- Touch source files, run Lean from a hook, or install anything silently.
|
|
173
|
-
- Commit, stage, or un-ignore anything under `.verity/proofs/`.
|
|
174
|
-
- Model a whole module. If the state has more than ~6 fields, narrow it.
|