proof-of-change 1.2.2 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +18 -6
- package/SPEC.md +13 -9
- package/docs/HOW-IT-WORKS.md +2 -1
- package/docs/QUICKSTART.md +142 -66
- package/docs/REFERENCE.md +6 -3
- package/docs/RESULTS.md +2 -2
- package/docs/STATUS.md +32 -9
- package/docs/THREAT-MODEL.md +47 -12
- package/examples/github/pocp-status.yml +17 -0
- package/package.json +1 -1
- package/src/approvals.js +32 -5
- package/src/ci-github.js +113 -2
- package/src/cli/shared.js +7 -3
- package/src/cli-args.js +4 -3
- package/src/cli.js +7 -4
- package/src/commands/ci.js +38 -4
- package/src/commands/init.js +11 -4
- package/src/commands/verify.js +17 -8
- package/src/doctor.js +89 -9
- package/src/engine/gate.js +6 -4
- package/src/engine/range.js +1 -1
- package/src/engine/rerun.js +7 -4
- package/src/engine/verify.js +16 -5
- package/src/fix.js +29 -16
- package/src/guard-check.js +96 -29
- package/src/init.js +24 -6
- package/src/mutants.js +5 -0
- package/src/remedies.js +59 -18
- package/src/schema.js +1 -1
- package/src/status-report.js +27 -7
- package/src/store.js +7 -1
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code and holds risky test edits for a maintainer. It works with any test command and any language, and needs only Node 20 or later, git and a POSIX shell.
|
|
4
4
|
|
|
5
|
-
It doesn't prove your code is correct
|
|
5
|
+
It doesn't prove your code is correct. It has one preregistered evaluation so far ([below](#evaluation)).
|
|
6
6
|
|
|
7
7
|

|
|
8
8
|
|
|
@@ -15,7 +15,7 @@ npm install --save-dev proof-of-change
|
|
|
15
15
|
npx pocp init --hooks --github
|
|
16
16
|
```
|
|
17
17
|
|
|
18
|
-
`init` creates `.proof
|
|
18
|
+
`init` creates `.proof/` and installs git hooks (yours keep running). Then:
|
|
19
19
|
|
|
20
20
|
1. **Say what must be proven.** A rule in `.proof/rules.json` maps files to a check, which is any command (`npm test`, `pytest`, `go test ./...`):
|
|
21
21
|
|
|
@@ -27,7 +27,14 @@ npx pocp init --hooks --github
|
|
|
27
27
|
|
|
28
28
|
2. **Commit.** If the hook blocks it, run `npx pocp fix --intent "<what the change does>"` and commit again. The commit gets a `Proof-Of-Change` trailer.
|
|
29
29
|
3. **Check the setup** with `npx pocp doctor`, which prints the commands for anything missing.
|
|
30
|
-
|
|
30
|
+
|
|
31
|
+
## Where the checks run
|
|
32
|
+
|
|
33
|
+
**GitHub CI** (`--github`, above). Run it once the GitHub repository exists, `origin` points at it and `gh` is logged in: `init` then writes the workflows, creates the approval labels and turns on the `pocp` status, which is green when verified, yellow while waiting for a maintainer's approval label and red on a real failure ([docs/STATUS.md](docs/STATUS.md)). Once the workflows are merged, `npx pocp ci required --apply` makes GitHub block merges until `pocp` is green. CI re-runs every check on a clean machine, and approvals are a maintainer's label. The trade: each pull request costs about one extra test run of Actions minutes, and on a free plan's private repository nothing blocks a merge.
|
|
34
|
+
|
|
35
|
+
**Local mode, no CI.** `npx pocp init --local` instead of `--github`, or `npx pocp ci off` on a repository that already has the workflows (`pocp ci on` reverses it). The git hooks are the gate and CI costs nothing. The trade: nothing re-runs the checks on another machine, nothing blocks a merge, and `git commit --no-verify` skips the gate unless you run `npx pocp verify --require-coverage --rerun main..HEAD` before you merge. Approvals are `--accept-*` flags on that command, so anything that can run commands on your machine, an agent included, can give them. `pocp doctor` suggests it when Actions jobs aren't starting for billing reasons.
|
|
36
|
+
|
|
37
|
+
**The reporter's own GitHub App** (optional, on top of GitHub CI). By default the `pocp` status is posted as GitHub Actions, like any workflow, so a branch pushed to your repository, by an agent using your token for example, can edit a workflow and post its own `pocp: success`. Giving the reporter its own App, with its key in an environment only the default branch can use, and requiring `pocp` from that App closes this. The trade: three setup steps, and it needs a public repository or GitHub Pro or Team ([docs/STATUS.md](docs/STATUS.md#give-the-reporter-its-own-identity)).
|
|
31
38
|
|
|
32
39
|
**On another CI** (GitLab, Jenkins, plain shell). In CI: keep your existing test step, and add:
|
|
33
40
|
|
|
@@ -35,14 +42,19 @@ npx pocp init --hooks --github
|
|
|
35
42
|
npx pocp verify origin/main..HEAD --require-coverage --rerun
|
|
36
43
|
```
|
|
37
44
|
|
|
38
|
-
**Private repositories pay for CI in GitHub Actions minutes.** POCP's `verify` re-runs your tests, so each pull request costs about one extra test run, plus about a minute for each push or approval. On a free plan, check you have minutes to spare. Without Actions, only the local git hook checks changes.
|
|
39
|
-
|
|
40
45
|
[docs/QUICKSTART.md](docs/QUICKSTART.md) walks through all of this on a non-JavaScript project.
|
|
41
46
|
|
|
47
|
+
## Evaluation
|
|
48
|
+
|
|
49
|
+
A preregistered benchmark of POCP 1.2.3 across p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, with a sealed holdout ([full report](eval/campaign-123/README.md)).
|
|
50
|
+
|
|
51
|
+
- **Harmful changes:** plain CI accepted all 48 designed test-cheats. POCP accepted none. In both modes it held every one for a person's review, each with a completed check showing the tested behavior changed. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([threat model](docs/THREAT-MODEL.md)).
|
|
52
|
+
- **Honest changes:** deep mode accepts 78–82% of 108 legitimate changes per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%. Reducing that friction is the next piece of work.
|
|
53
|
+
|
|
42
54
|
## Documentation
|
|
43
55
|
|
|
44
56
|
- [How it works](docs/HOW-IT-WORKS.md): what it catches, fast and deep modes, and what it doesn't do.
|
|
45
|
-
- [
|
|
57
|
+
- [Evaluation of 1.2.3](eval/campaign-123/README.md): the preregistered benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
|
|
46
58
|
- [Reference](docs/REFERENCE.md): rules, checks, CI options, approvals, signed proofs.
|
|
47
59
|
- [Threat model](docs/THREAT-MODEL.md): what each record does and doesn't prove.
|
|
48
60
|
- [SPEC.md](SPEC.md): the protocol. [Reviewing POCP](docs/REVIEWING.md): a guide for reviewers.
|
package/SPEC.md
CHANGED
|
@@ -420,14 +420,15 @@ runnable obligations. Every run below is in its own fresh copy of the code.
|
|
|
420
420
|
the end and failed: the same rule as for base runs in part 1. A run that timed out, was
|
|
421
421
|
killed by a signal, couldn't start, or exited 126 or 127 says nothing about whether the
|
|
422
422
|
tests catch the mutant (under load, a full suite hits the timeout either way). Such a run
|
|
423
|
-
is retried once, in a fresh copy with the mutant:
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
423
|
+
is retried once, in a fresh copy with the mutant: if it timed out, with
|
|
424
|
+
min(3× the limit it hit, max(30 s, 5× the check's measured time on the new code)),
|
|
425
|
+
otherwise with the same limit. A mutant that runs 5× slower than the suite is as good as
|
|
426
|
+
hung, so the retry needn't wait longer; a retry that hits the cap is not measured, never
|
|
427
|
+
a pass. It costs at most one extra run per unfinished check and mutant, and none when
|
|
428
|
+
every run finishes. Worst case, every sampled mutant hangs every check: each check's
|
|
429
|
+
limit plus that cap per mutant on a side, spread over the parallel jobs; with the
|
|
430
|
+
defaults (10 mutants, a 10-minute limit, one check that takes a minute, 4 jobs) about 40
|
|
431
|
+
minutes of wall time before the result is `inconclusive` (100 minutes without the cap). Only mutant runs are
|
|
431
432
|
retried: the baselines in part 1 are not (a baseline that doesn't finish blocks the
|
|
432
433
|
result). A retry that ends normally counts as the run. After a run
|
|
433
434
|
that still didn't finish, the remaining checks still run. The mutant is caught if any
|
|
@@ -694,7 +695,10 @@ drops a glob from a requirement, raises a threshold, removes a protected path, a
|
|
|
694
695
|
exemption, or adds a key to `allowed_signers`), verify fails and names each loosening, unless run with
|
|
695
696
|
`--accept-policy-change`, which CI should pass only when a maintainer has approved
|
|
696
697
|
(for example, a PR label). `--accept-policy-change=<commit>` (and likewise
|
|
697
|
-
`--accept-test-change=<commit>`) approves only when the head being verified is that commit.
|
|
698
|
+
`--accept-test-change=<commit>`) approves only when the head being verified is that commit. In a range, a commit
|
|
699
|
+
before the head is named only for loosenings its parent didn't already have (a base that
|
|
700
|
+
tightened while the branch was open isn't the branch loosening it); the head, or the net change
|
|
701
|
+
when the head is a merge, is named for everything it loosens against the base. Trusted signers also come from the base, or from
|
|
698
702
|
`--allowed-signers <file>`, never from the commit being judged.
|
|
699
703
|
|
|
700
704
|
**Re-running.** With `--rerun`, every runnable obligation runs again, each in its own fresh
|
package/docs/HOW-IT-WORKS.md
CHANGED
|
@@ -35,7 +35,7 @@ In CI, `pocp verify origin/main..HEAD --require-coverage --rerun` (the base bran
|
|
|
35
35
|
Both modes first run the old versions of the edited tests on the new code. If they run to the end and fail, but pass on the base, behavior changed, and a person approves it. If they don't finish (a timeout, for example, under load), the result is inconclusive rather than a behavior change, and it still needs a re-run or a person.
|
|
36
36
|
|
|
37
37
|
- **Fast** (`guard.mutants: 0`, what `pocp init` sets up). No test edit passes on its own: every change to tests or test setup goes to a person, including added lines and new test files. Text rules can't tell whether an edit still tests what it did (added lines alone can disable existing assertions), so fast mode doesn't pretend to. Its review lists what changed and flags added lines that match common risky patterns (skipping or focusing tests, exiting, early returns, patched assertions, mocking APIs, exit-code resets, test hooks, globals, empty `catch`) so the reviewer knows where to look first.
|
|
38
|
-
- **Deep** (`guard.mutants: 10`, `pocp init --deep`). The only mode where a test edit can pass without a person. POCP plants a sample of small bugs (flipped comparisons, swapped operators, changed constants) and checks that the new tests catch every sampled bug the old tests caught. New test files count too: they run with the old ones, so one that exits early or patches an assertion makes the suite miss bugs it caught. It samples: a weakening that matters only outside the sample passes, and only the bug types POCP can plant are tried. On POCP's own code, about 14% of StrykerJS's mutants are edits POCP's operators can make. It is much slower (below).
|
|
38
|
+
- **Deep** (`guard.mutants: 10`, `pocp init --deep`). The only mode where a test edit can pass without a person. POCP plants a sample of small bugs (flipped comparisons, swapped operators, changed constants) and checks that the new tests catch every sampled bug the old tests caught. New test files count too: they run with the old ones, so one that exits early or patches an assertion makes the suite miss bugs it caught. It samples: a weakening that matters only outside the sample passes, and only the bug types POCP can plant are tried. A planned attack across pull requests, weakening an assertion behind a test that reads the source and adding the bug later, is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md)). On POCP's own code, about 14% of StrykerJS's mutants are edits POCP's operators can make. It is much slower (below).
|
|
39
39
|
|
|
40
40
|
A project set up before fast mode existed, with no `guard.mutants` in its config, keeps deep mode: the built-in default is still 10, and upgrading POCP never rewrites a config. To switch such a project to fast mode, set `"guard": { "mutants": 0 }` in `.proof/config.json` in a pull request of its own. Lowering `guard.mutants` is a policy change, so CI fails it until a maintainer approves it with the `pocp-approve-policy` label.
|
|
41
41
|
|
|
@@ -47,6 +47,7 @@ Within the limits in [docs/THREAT-MODEL.md](THREAT-MODEL.md):
|
|
|
47
47
|
| Marks tests skipped or `.only` | passes | goes to a person | blocked if a sampled bug goes undetected |
|
|
48
48
|
| Hollows out the test script, runner config, or an assertion helper the tests import | passes | goes to a person, for helpers POCP finds by following imports | blocked, for helpers POCP finds by following imports |
|
|
49
49
|
| Adds a test file, or lines to a test, that disable other tests | passes | goes to a person | blocked if a sampled bug goes undetected |
|
|
50
|
+
| Weakens an assertion behind a new test that reads the source under test, then adds the bug in a later pull request | passes | goes to a person | out of scope: deliberate deception across pull requests ([threat model](THREAT-MODEL.md)); in one change, blocked (the old tests fail on the new code) |
|
|
50
51
|
| Removes or loosens the rule that requires a check | n/a | blocked: the base branch's rules apply | same |
|
|
51
52
|
| Edits a packet so failing proofs read as passed | n/a | blocked: the checks are re-run | same |
|
|
52
53
|
| Edits or removes POCP's CI workflow | passes | with `pocp` required: stays yellow until a maintainer approves the workflow change (`pocp-approve-workflow`); see the limit below | same |
|
package/docs/QUICKSTART.md
CHANGED
|
@@ -3,44 +3,82 @@
|
|
|
3
3
|
This walkthrough uses a tiny shell project, so nothing here depends on JavaScript.
|
|
4
4
|
POCP itself needs Node ≥ 20, git and a POSIX shell; your project can be anything.
|
|
5
5
|
|
|
6
|
-
## 1.
|
|
6
|
+
## 1. The example project
|
|
7
|
+
|
|
8
|
+
In a git repository, two files:
|
|
9
|
+
|
|
10
|
+
`bin/greet`
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
#!/bin/sh
|
|
14
|
+
echo "Hello, $1"
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
`tests/run.sh`
|
|
18
|
+
|
|
19
|
+
```sh
|
|
20
|
+
out=$(sh "$(dirname "$0")/../bin/greet" Ada)
|
|
21
|
+
[ "$out" = "Hello, Ada" ] || { echo "FAIL: got '$out'"; exit 1; }
|
|
22
|
+
echo ok
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
git add -A && git commit -m "greet"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## 2. Install and choose where the checks run
|
|
7
30
|
|
|
8
31
|
```bash
|
|
9
32
|
npm install -g proof-of-change
|
|
10
|
-
|
|
11
|
-
|
|
33
|
+
pocp init --local # no CI: the git hooks on your machine are the gate
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
or, for a GitHub repository whose `origin` exists and with `gh` logged in:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
pocp init --hooks --github # CI re-runs every check and can block merges
|
|
12
40
|
```
|
|
13
41
|
|
|
42
|
+
Section 5 says what each one trades. You can switch later with `pocp ci off` and `pocp ci on`.
|
|
43
|
+
|
|
14
44
|
`.proof/` now holds `config.json`, `rules.json` and empty `packets/` and `misses/` folders.
|
|
15
|
-
`init` also prints the test layout it found
|
|
16
|
-
`
|
|
45
|
+
`init` also prints the test layout it found. For a shell project like this one it finds none,
|
|
46
|
+
which is fine: a file that a rule's check runs (`tests/run.sh` below) is guarded as test setup
|
|
47
|
+
anyway. For test folders it misses in a real project, add them to `guard.extra_test_globs` in
|
|
48
|
+
`config.json`.
|
|
17
49
|
|
|
18
|
-
##
|
|
50
|
+
## 3. Say what must be proven
|
|
19
51
|
|
|
20
|
-
`.proof/rules.json
|
|
52
|
+
In `.proof/rules.json`, replace `"requirements": []` with this rule, and keep the rest of the
|
|
53
|
+
file as `init` wrote it:
|
|
21
54
|
|
|
22
55
|
```json
|
|
23
|
-
{
|
|
24
|
-
"pocp": 1,
|
|
25
|
-
"triggers": { "globs": [{ "glob": "bin/**", "require_packet": true }] },
|
|
26
56
|
"requirements": [
|
|
27
57
|
{ "id": "shell-tests", "glob": "bin/**",
|
|
28
58
|
"obligation": { "kind": "test", "run": "sh run.sh", "cwd": "tests" } }
|
|
29
59
|
]
|
|
30
|
-
}
|
|
31
60
|
```
|
|
32
61
|
|
|
33
62
|
Any command works as a check: `cargo test`, `pytest -q`, `go test ./...`, `make check`,
|
|
34
63
|
`./gradlew test`. Use `cwd` when it must run in a subfolder.
|
|
35
64
|
|
|
65
|
+
With `--github`, the adoption goes in through a pull request (step 1 of "On GitHub" below), so
|
|
66
|
+
first make a branch for it: `git checkout -b adopt-pocp`. In local mode, commit on `main`.
|
|
67
|
+
|
|
36
68
|
```bash
|
|
37
69
|
pocp rules check
|
|
70
|
+
git add -A && git commit -m "Adopt POCP"
|
|
38
71
|
```
|
|
39
72
|
|
|
40
|
-
|
|
73
|
+
If the hook blocks the adoption commit (with `--github` it adds CI files, which need a packet),
|
|
74
|
+
run `pocp fix --intent "Adopt POCP"` and commit again.
|
|
75
|
+
|
|
76
|
+
## 4. Make a change
|
|
77
|
+
|
|
78
|
+
Change `bin/greet` to default the name: `echo "Hello, ${1:-world}"`.
|
|
41
79
|
|
|
42
80
|
```bash
|
|
43
|
-
|
|
81
|
+
git checkout -b default-name # on GitHub: from main, once the adoption pull request has merged and you've pulled it
|
|
44
82
|
git add bin/greet
|
|
45
83
|
git commit -m "Default the name" # blocked: says what is wrong, why, and what to run
|
|
46
84
|
pocp fix --intent "Default the name" # opens the packet, runs "sh run.sh" in tests/ on the staged code
|
|
@@ -51,50 +89,101 @@ git commit -m "Default the name" # passes; adds a Proof-Of-Change trail
|
|
|
51
89
|
something needs a person (a test change POCP could not check, a policy loosening), it says
|
|
52
90
|
who and what, and never signs off itself.
|
|
53
91
|
|
|
54
|
-
|
|
92
|
+
Editing only a test works the same way, and `pocp fix` runs the rule's check in `tests/`. In
|
|
93
|
+
fast mode (what `init` writes), every test edit then waits for a person: below, in local mode;
|
|
94
|
+
a maintainer's label on GitHub.
|
|
55
95
|
|
|
56
|
-
|
|
57
|
-
by the base branch's rules; your existing test step stays as it is:
|
|
96
|
+
## 5. Where the checks run
|
|
58
97
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
98
|
+
| | GitHub CI | Local mode |
|
|
99
|
+
|---|---|---|
|
|
100
|
+
| Commits checked | hooks, then verify in CI | hooks only |
|
|
101
|
+
| Re-run on another machine | every pull request | none; `verify --require-coverage --rerun` on yours before merging |
|
|
102
|
+
| Unverified merges blocked | with branch protection | never |
|
|
103
|
+
| `git commit --no-verify` | caught by verify | caught only if you verify before merging |
|
|
104
|
+
| Test and policy approvals | a maintainer's label | you, through `--accept-*` |
|
|
105
|
+
| You trust | the runner and GitHub | your own machine |
|
|
106
|
+
| Costs | Actions minutes: about one extra test run per pull request | nothing |
|
|
62
107
|
|
|
63
|
-
|
|
64
|
-
It trusts everything that ran earlier in the job: a new test file can write a passing result
|
|
65
|
-
for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
|
|
108
|
+
### No CI (local mode)
|
|
66
109
|
|
|
67
|
-
|
|
110
|
+
`pocp init --local` sets `ci.mode: "local"` in `.proof/config.json` and installs the hooks. On a
|
|
111
|
+
repository that already has POCP's workflows, `pocp ci off` disables them on GitHub (`pocp.yml`,
|
|
112
|
+
`pocp-status.yml`, `pocp-labels.yml`) and sets the mode; `pocp ci on` enables them again and
|
|
113
|
+
removes it. Neither touches the workflow files, so there is nothing to commit under
|
|
114
|
+
`.github/`; commit the change to `.proof/config.json`. Switching to local mode is a policy
|
|
115
|
+
loosening, so verify asks for `--accept-policy-change` on that commit. `ci off` lists your
|
|
116
|
+
other workflows and the command to disable each; it doesn't touch them. The test guard's fast
|
|
117
|
+
or deep mode is a separate setting and works the same under the hooks.
|
|
118
|
+
|
|
119
|
+
Before merging a branch, run what CI would have run. `<base>..HEAD` is the branch's commits:
|
|
120
|
+
`main..HEAD` for a local branch, `origin/main..HEAD` for one you pushed.
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
pocp verify --require-coverage --rerun main..HEAD
|
|
124
|
+
```
|
|
68
125
|
|
|
69
|
-
|
|
126
|
+
It re-runs each commit's checks from a clean checkout, by the base branch's rules, and
|
|
127
|
+
`--require-coverage` fails a commit made without its proof (`git commit --no-verify`). When it
|
|
128
|
+
waits for a person (a test change the check couldn't pass, a policy loosening), read the change,
|
|
129
|
+
then approve it for that exact head:
|
|
70
130
|
|
|
71
131
|
```bash
|
|
72
|
-
pocp
|
|
132
|
+
pocp verify --require-coverage --rerun main..HEAD --accept-test-change $(git rev-parse HEAD)
|
|
73
133
|
```
|
|
74
134
|
|
|
75
|
-
|
|
135
|
+
(or `--accept-policy-change`). The output still lists the test guard's re-run as failed; the
|
|
136
|
+
approval is what lets it through, and verify then prints `verified`.
|
|
137
|
+
|
|
138
|
+
**A local approval is a flag, not a person.** Anything that can run commands on your machine,
|
|
139
|
+
an agent included, can pass `--accept-*` itself, just as it can run `git commit --no-verify`. It
|
|
140
|
+
is the same trade as the rest of local mode: you trust your own machine. Where you need a test
|
|
141
|
+
change or a policy loosening to wait for a person, use CI, where only a maintainer's label
|
|
142
|
+
approves it.
|
|
143
|
+
|
|
144
|
+
`pocp doctor` reports local mode, POCP workflows still running in it, and Actions jobs that
|
|
145
|
+
GitHub doesn't start for billing reasons.
|
|
146
|
+
|
|
147
|
+
### On GitHub
|
|
148
|
+
|
|
149
|
+
Before `pocp init --github`: the repository exists on GitHub, `origin` points at it, and `gh` is
|
|
150
|
+
logged in with the `workflow` permission (`gh auth refresh -s workflow`), which pushing workflow
|
|
151
|
+
files needs. `init --github` writes three workflows, with the POCP actions pinned to your POCP
|
|
152
|
+
release:
|
|
76
153
|
|
|
77
154
|
- **`pocp.yml`** runs verify on every pull request.
|
|
78
155
|
- **`pocp-status.yml`** is the trusted reporter: it sets one commit status, **`pocp`**. It's green when verified, yellow while waiting for a maintainer or when a check didn't finish, and red on a real failure. It never runs pull request code.
|
|
79
156
|
- **`pocp-labels.yml`** removes every approval label on each push, so an approval covers only the code reviewed.
|
|
80
157
|
|
|
81
|
-
|
|
158
|
+
Through your `gh` login it also creates the approval labels and turns the `pocp` status on
|
|
159
|
+
(the repository variable `POCP_STATUS_LIVE`), so a pull request waiting for approval shows
|
|
160
|
+
yellow, with no red ✗ and no failure email (`--no-live` leaves it off). Without `origin` it can
|
|
161
|
+
do neither and says so: `pocp doctor` then lists the labels to create, and step 3 below turns
|
|
162
|
+
the status on.
|
|
82
163
|
|
|
83
|
-
1. **
|
|
84
|
-
2. **
|
|
85
|
-
3. **Once the pull request
|
|
164
|
+
1. **Push `adopt-pocp` and open the adoption pull request** with `.proof/` and the workflows. It changes CI files, so its verify waits for `pocp-approve-policy`: read the diff and add the label (`pocp approve owner/repo#<number>`). The reporter only runs once it is on the default branch, so this pull request shows no yellow `pocp` status: its `verify` check is green, and its job summary's heading names the label it waits for. Nothing is required yet, and no other label is needed. Keep this pull request to `.proof/` and CI files (`pocp doctor` checks): until it merges, CI judges with the base branch's rules.
|
|
165
|
+
2. **A maintainer approves a change by adding its label:** `pocp-approve-tests`, `pocp-approve-policy`, `pocp-approve-inconclusive`, `pocp-approve-coverage` or `pocp-approve-workflow`. [STATUS.md](STATUS.md) says what each covers.
|
|
166
|
+
3. **Once the adoption pull request merges, require the status:** `pocp ci required --apply`. It sets branch protection from `.proof/config.json`: the `pocp` status (in place of the `verify` job, if that was required), pull requests only, admins included, and branches up to date before merging (so the merged code is the code that was verified; when `main` moves, a pull request needs "Update branch" before it merges). Each can be turned off in `ci`. It also turns the `pocp` status on if `init` couldn't. On a free plan's private repository GitHub has no branch protection: the status is then a signal only, and the command says so. Merge queues aren't supported yet ([REFERENCE.md](REFERENCE.md)).
|
|
86
167
|
|
|
87
|
-
Do the two
|
|
168
|
+
Do the two together: once the status is on, `verify` passes on a wait, so requiring `verify` alone would let an unapproved pull request merge. To roll back, unset `POCP_STATUS_LIVE` and require `verify` again.
|
|
88
169
|
|
|
89
|
-
|
|
170
|
+
If your repository has a second maintainer, keep the default `"ci": { "workflow_approver": "independent" }`: then the author of a CI change can't approve it themselves. See [STATUS.md](STATUS.md) for that, and for the limits of the `pocp` status.
|
|
90
171
|
|
|
91
|
-
|
|
172
|
+
**Public repositories, from December 2, 2026:** GitHub blocks `pull_request_target` workflows by
|
|
173
|
+
default, and then approval labels are no longer removed on push. With the `pocp` status live
|
|
174
|
+
this is only untidy; without it, allow `pull_request_target` in the repository's Actions settings
|
|
175
|
+
([STATUS.md](STATUS.md#githubs-pull_request_target-block-public-repositories-from-december-2-2026)).
|
|
176
|
+
`pocp doctor` says when it is blocked.
|
|
177
|
+
|
|
178
|
+
**If agents push branches with your token,** a workflow on such a branch can post its own
|
|
179
|
+
`pocp: success`. Give the reporter its own GitHub App and require `pocp` from it:
|
|
180
|
+
[STATUS.md, "Give the reporter its own identity"](STATUS.md#give-the-reporter-its-own-identity).
|
|
181
|
+
It takes a public repository, or GitHub Pro or Team for a private one.
|
|
92
182
|
|
|
93
183
|
Then run `pocp doctor`. It checks the adoption pitfalls before they block anyone: hooks,
|
|
94
184
|
the workflow's approval labels and whether they exist, whether a reporter sets the `pocp` status, Actions access when the POCP action
|
|
95
|
-
is in a private repository, branch protection against the whole plan (required checks, pull requests only, admins, up-to-date branches), and that
|
|
96
|
-
|
|
97
|
-
base branch's rules, so any other change would get two verdicts). It changes nothing.
|
|
185
|
+
is in a private repository, branch protection against the whole plan (required checks, pull requests only, admins, up-to-date branches), and that the
|
|
186
|
+
adoption pull request carries only `.proof/` and CI files. It changes nothing.
|
|
98
187
|
|
|
99
188
|
When pull requests wait for a maintainer's approval, find and give them in one place:
|
|
100
189
|
|
|
@@ -103,10 +192,23 @@ pocp approve # every open PR, across your repos, wait
|
|
|
103
192
|
pocp approve owner/repo#12 # shows every approval it waits for, asks you once, then adds them
|
|
104
193
|
```
|
|
105
194
|
|
|
106
|
-
|
|
107
|
-
|
|
195
|
+
It asks you to type the pull request's number before adding a label, so a person at a terminal
|
|
196
|
+
approves. Agents can list, never approve.
|
|
197
|
+
|
|
198
|
+
### Another CI
|
|
108
199
|
|
|
109
|
-
|
|
200
|
+
Verify every pull request. verify re-runs each required check itself, on the exact commit,
|
|
201
|
+
by the base branch's rules; your existing test step stays as it is:
|
|
202
|
+
|
|
203
|
+
```bash
|
|
204
|
+
pocp verify origin/main..HEAD --require-coverage --rerun
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
A faster mode runs your tests once, through `pocp ci-run`, and passes `--ci-results` to verify.
|
|
208
|
+
It trusts everything that ran earlier in the job: a new test file can write a passing result
|
|
209
|
+
for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
|
|
210
|
+
|
|
211
|
+
## 6. Measure
|
|
110
212
|
|
|
111
213
|
```bash
|
|
112
214
|
pocp history --test "sh tests/run.sh" # how often would the gate have fired; do past fix tests catch their bugs?
|
|
@@ -114,7 +216,7 @@ pocp eval --mutants 50 --save eval.json # how many planted bugs do the require
|
|
|
114
216
|
pocp eval --replay eval.json --missed # after adding tests: re-run the same survivors
|
|
115
217
|
```
|
|
116
218
|
|
|
117
|
-
##
|
|
219
|
+
## 7. When something escapes
|
|
118
220
|
|
|
119
221
|
```bash
|
|
120
222
|
pocp miss record --description "greet prints nothing without a name" --stage production --file bin/greet
|
|
@@ -123,29 +225,3 @@ pocp miss close <id> --replay <commit where it escaped>
|
|
|
123
225
|
```
|
|
124
226
|
|
|
125
227
|
Next: [the threat model](THREAT-MODEL.md) explains what each record does and doesn't prove.
|
|
126
|
-
|
|
127
|
-
## Step by step on GitHub
|
|
128
|
-
|
|
129
|
-
POCP needs Node 20 or later, git and a POSIX shell. Your project can be in any language. In a git repository:
|
|
130
|
-
|
|
131
|
-
```bash
|
|
132
|
-
npm install --save-dev proof-of-change
|
|
133
|
-
npx pocp init --hooks --github
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
`init` creates `.proof/` and reports where it found your tests. `--hooks` installs the git hooks; hooks you already have keep running. `--github` writes three workflows to `.github/workflows/`, pinned to this release:
|
|
137
|
-
|
|
138
|
-
- `pocp.yml` runs verify;
|
|
139
|
-
- `pocp-status.yml` sets the `pocp` status;
|
|
140
|
-
- `pocp-labels.yml` removes approval labels on every push.
|
|
141
|
-
|
|
142
|
-
Then:
|
|
143
|
-
|
|
144
|
-
1. **Say what must be proven.** Add a rule to `.proof/rules.json` (section 2). Change a file it covers, and commit. If the commit is blocked, run `npx pocp fix --intent "<what the change does>"` and commit again. Anyone can skip the hook with `git commit --no-verify`; CI then flags the commit as made without its proof.
|
|
145
|
-
2. **Check the setup:** `npx pocp doctor`. It lists anything missing, with the commands to fix it, such as the approval labels to create on GitHub. Pushing the workflow files needs a GitHub login with the `workflow` permission.
|
|
146
|
-
3. **Merge the pull request with the workflows, then require the status:** `npx pocp ci required --apply`. `init --github` already turned the `pocp` status on (unless `--no-live`), so waits are yellow, not red. This step makes GitHub block merges until `pocp` is green: it requires `pocp` (replacing `verify` if that was required), pull requests only, admins included, and branches up to date before merging, so the code that merges is the code that was verified. Each can be turned off, and `pocp doctor` reports any gap. On a free plan's private repository GitHub has no branch protection, so the status is a signal only there. Merge queues (GitHub's, Graphite's) aren't supported yet ([docs/REFERENCE.md](REFERENCE.md)). To roll back, unset `POCP_STATUS_LIVE` and require `verify` again.
|
|
147
|
-
|
|
148
|
-
[docs/QUICKSTART.md](QUICKSTART.md) walks through this on a non-JavaScript project. [docs/STATUS.md](STATUS.md) covers the status, the approval labels, who may approve, and its limits.
|
|
149
|
-
|
|
150
|
-
This repository uses POCP on itself: every pull request needs the `pocp` status, plus the test suite on Ubuntu with Node 20, 22 and 24 and the eval tests. The test suite also runs on macOS (one Node version on each pull request, all three nightly), which is not required while GitHub's macOS runners are unreliable.
|
|
151
|
-
|
package/docs/REFERENCE.md
CHANGED
|
@@ -30,7 +30,7 @@ The details behind [the README](../README.md): rules, obligations, how proof is
|
|
|
30
30
|
- **Unknown keys:** POCP ignores a key it doesn't know in `.proof/rules.json` or `.proof/config.json`, so a misspelling turns a setting off. `rules check`, `doctor`, the gate and verify (on the commit that changes the file) warn about each one and suggest the key you probably meant. A warning changes nothing that passes or blocks. Start a key with `$` to keep it as a comment (`"$why": "..."`).
|
|
31
31
|
- When a packet opens, the requirements matching its files are added to it automatically. Nobody can remove them.
|
|
32
32
|
- Changing `.proof/rules.json`, `.proof/config.json` or `.proof/allowed_signers` is itself a change that needs a packet, so weakening the rules is never silent.
|
|
33
|
-
- **Test guard:** modifying an existing test or the test setup (runner config, `.npmrc`, the test settings in `package.json` / `pyproject.toml` / `setup.cfg`, and any file a runner config or a required check's command points to, such as a setup file) needs a packet, and `pocp prove` checks mechanically that nothing was weakened. (1) The old versions of the changed tests are run against the new code: if they run to the end and fail, and pass on the base, behavior changed; if they don't finish (a timeout, a signal, a spawn error), the result is inconclusive, not a behavior change. (2) In deep mode (`guard.mutants` above 0: the built-in default of 10, and what `pocp init --deep` writes), a seeded sample of small bugs is planted in the source those tests guard, starting with lines the change touched: every bug the old tests catch, the new tests must catch too. Only a check that runs to the end and fails catches a bug: a timeout (as under load), a kill or a command that can't start says nothing. A *mutant* run like that is retried once
|
|
33
|
+
- **Test guard:** modifying an existing test or the test setup (runner config, `.npmrc`, the test settings in `package.json` / `pyproject.toml` / `setup.cfg`, and any file a runner config or a required check's command points to, such as a setup file) needs a packet, and `pocp prove` checks mechanically that nothing was weakened. (1) The old versions of the changed tests are run against the new code: if they run to the end and fail, and pass on the base, behavior changed; if they don't finish (a timeout, a signal, a spawn error), the result is inconclusive, not a behavior change. (2) In deep mode (`guard.mutants` above 0: the built-in default of 10, and what `pocp init --deep` writes), a seeded sample of small bugs is planted in the source those tests guard, starting with lines the change touched: every bug the old tests catch, the new tests must catch too. Only a check that runs to the end and fails catches a bug: a timeout (as under load), a kill or a command that can't start says nothing. A *mutant* run like that is retried once; a timed-out one gets 3× the time limit, capped at 5× the check's measured time on the new code (or 30 s, if that's more), since a mutant that runs that much slower is as good as hung. If it still doesn't finish, on either side, the check can't pass (it is inconclusive and needs a person, or a re-run that finishes). Baseline runs (the checks on the new code, the old tests on the new code, the base) aren't retried: one that doesn't finish blocks as `inconclusive`, including when the old tests time out on the new code but pass on the base (a timeout says nothing about behavior); on an overloaded machine, re-run with `--force`. Worst case, every sampled mutant hangs every check: each check's time limit plus the capped retry, per mutant on the old side, divided across the parallel jobs. With the defaults (10 mutants, a 10-minute limit, one check that takes a minute, 4 jobs) that's about 40 minutes of wall time before the result is `inconclusive` (100 without the cap). In fast mode (`guard.mutants: 0`, what a plain `pocp init` writes), (2) is replaced by a review: every guarded change is `needs-review` and goes to a person, including lines added to existing tests and new test files, since without planted bugs nothing shows the tests still catch what they caught. The review names what changed and flags added lines matching the risky patterns listed in [HOW-IT-WORKS.md](HOW-IT-WORKS.md#the-test-check-fast-and-deep) (SPEC §6.2). A config without `guard.mutants` keeps the built-in 10, so upgrading never turns deep mode off. To switch an existing project to fast mode, set `guard.mutants: 0` in `.proof/config.json`; that is a policy change a maintainer approves (`pocp-approve-policy`). A pass is recorded as a proof; no sign-off is needed. New test files are guarded too: they run with the old ones and can change their outcome, so in deep mode the new suite (with them) must still catch every sampled bug, and in fast mode they go to a person. Renaming or moving tests between files passes, in deep mode, when the required check finds the tests by itself (`npm test`, `node --test test/`, `pytest tests/`). When a rule's `run` names the test file (`node test/math.test.js`), an honest rename is blocked: the rule has to change with it (a policy change a maintainer approves), and the base's command, which CI judges by, no longer finds the file, so the test guard can't pass either (`--accept-test-change` and the policy label get it through). Point rule commands at a test directory or the test runner rather than at single files. Only when the check fails does the packet need `pocp attest`, which is weaker and labelled as such. By default (`guard.tests: block`), CI fails a test change unless the check passes when `verify --rerun` runs it again, or a maintainer passes `--accept-test-change` (meant for intended behavior changes). `attest` mode accepts the sign-off instead, which the author can give themselves. In block mode the author never signs off: `pocp fix` records that the failed check is handed to CI (a *deferral*, not an approval), so the commit can go ahead and the maintainer's label decides. The check never runs twice for the same code: its new-code baseline reuses the packet's own passing proof, results are cached locally by everything that decides them, and in CI it reuses the job's re-runs and `ci-run` results for the same commit. The gate also shows removed assertions and added `.skip`/`.only`. For a pull request, `verify base..head --rerun` also runs the check once on the whole pull request as one change (old tests from the merge base, new code from the head, merges included), so dropping an assertion in one commit and adding the bug it caught in the next, or doing either inside a merge commit, fails the same way as doing both in one commit. That check's mutant sample uses a seed derived from `guard.seed`, the merge base and the head: different from the one `prove` used locally, the same on every re-run of that verify, and printed with the result. Defaults recognise the common test layouts and harness files of JavaScript/TypeScript, Python, Ruby, Go, Rust, Java/Kotlin, C# and PHP; `pocp init` writes test folders it finds declared elsewhere to `guard.extra_test_globs`; add your own with `guard.extra_runner_globs` / `guard.extra_test_globs` (a Makefile target, a custom script). Files the tests import (assertion helpers, fixtures), followed transitively, count as test setup unless they are the code the required checks guard, and the `#[cfg(test)]` items of a Rust file are its test part (SPEC §6.2 has the limits).
|
|
34
34
|
|
|
35
35
|
## Obligations
|
|
36
36
|
|
|
@@ -138,6 +138,8 @@ Every repository decides for itself, in `.proof/config.json` (a policy file, so
|
|
|
138
138
|
- **`verify.fail_on`**: which findings fail `verify`. Everything else is still reported, as a warning. The default is all of them; a repo adopting POCP gradually can start narrower. `invalid_policy` always fails.
|
|
139
139
|
- **`verify.require_ci`**: CI commands that must pass on the exact code being merged, either as a `pocp ci-run` result or as a re-run by `verify` itself. Any CI system can produce those results. With neither, verification fails.
|
|
140
140
|
- **`ci.required_checks`**: which CI checks must pass to merge. `pocp ci required` prints the plan; `pocp ci required --apply` adds them to GitHub branch protection, keeping every existing protection and never loosening one (it refuses an empty list). It has been run against the live API on POCP's own repository. `pocp ci required --check` (and `pocp doctor`) report every gap between the plan and what GitHub enforces. On other hosts, configure the same checks in their branch or merge-request settings. Dropping a required check, or turning off required pull requests or admin enforcement, is a policy loosening that needs approval.
|
|
141
|
+
- **`ci.reporter_app_id`** (a GitHub App id, unset by default): the app the reporter posts the `pocp` status as (docs/STATUS.md, "Give the reporter its own identity"). `pocp ci required --apply` requires `pocp` from that app only; `--check` and `pocp doctor` report a required `pocp` check that isn't pinned to it, or, when unset, one that any source or GitHub Actions (app 15368) can post, since any workflow a branch edits posts as GitHub Actions. Changing or removing it is a policy loosening.
|
|
142
|
+
- **`ci.mode`**: `"github"` (the default: CI checks every pull request) or `"local"` (no CI: the git hooks are the gate). `pocp init --local` sets it; `pocp ci off` disables POCP's workflows on GitHub and sets it, `pocp ci on` enables them and removes it. Only `pocp doctor` and the wording of the gate's and verify's approval steps read it; verify checks the same things in either mode. What local mode gives up: [QUICKSTART](QUICKSTART.md#no-ci-local-mode).
|
|
141
143
|
- **`ci.require_pull_request`, `ci.enforce_admins`, `ci.require_up_to_date`** (all on by default): every change goes through a pull request; admins can't merge past the required checks; and a pull request must include the latest base branch before it merges (GitHub's "require branches to be up to date"). The last one is what makes "the code being merged is the code that was verified" true: without it, a pull request verified against an older base merges into a newer one, and the combination was never checked. The cost: when the base moves, each open pull request needs "Update branch" and a fresh CI run before it can merge. Set any of them to `false` to opt out; doctor then reports it, and turning one off later is a policy loosening.
|
|
142
144
|
- **Merge queues are not supported yet.** GitHub's merge queue and Graphite's run CI on a temporary commit the queue builds, not on the pull request's head. POCP's workflow runs on pull request events and reports on the pull request's head, so a queue that requires `pocp` would wait for a result that never comes. Until POCP supports them, use "require branches to be up to date" instead.
|
|
143
145
|
- **CI definitions are protected.** Changing `.github/workflows/`, `.gitlab-ci.yml`, a `Jenkinsfile` and similar (`ci.files`) needs a packet and, in CI, the same approval as loosening the rules, since an edited workflow could switch POCP off. `ci.files` can protect any other path the same way, such as an evaluation's datasets, graders and results; verify reports those as "protected files changed". Setting `ci.files` replaces the default list, so include the defaults you want to keep (dropping one is reported as a loosening).
|
|
@@ -145,7 +147,7 @@ Every repository decides for itself, in `.proof/config.json` (a policy file, so
|
|
|
145
147
|
|
|
146
148
|
Nothing here assumes GitHub Actions or a test matrix. [`examples/`](../examples/) has GitHub, GitLab and plain-shell versions.
|
|
147
149
|
|
|
148
|
-
**POCP never changes merge settings on its own.** Its job is to report proof status and fail its CI check according to the repository's policy. Changing the host's branch protection happens only when someone runs `pocp ci required --apply` explicitly; no other command, hook or the GitHub Action ever does. The hosting platform stays in charge of whether a red check blocks a merge.
|
|
150
|
+
**POCP never changes merge settings on its own.** Its job is to report proof status and fail its CI check according to the repository's policy. Changing the host's branch protection happens only when someone runs `pocp ci required --apply` explicitly; no other command, hook or the GitHub Action ever does. `pocp ci off` / `pocp ci on` only disable or enable POCP's own workflows, and only when run. The hosting platform stays in charge of whether a red check blocks a merge.
|
|
149
151
|
|
|
150
152
|
**Verification vs. merge enforcement.** POCP verification works on any plan and any host: `pocp verify` runs in CI and fails its check on a bad change. Whether a failing check *blocks the merge button* is up to the hosting platform. On GitHub, branch protection needs a public repository or a paid plan for private ones. **On a free plan's private repository, POCP is advisory**: verify still runs and reports, but anyone with write access can merge unverified code, and `pocp doctor` says so. If `--apply` can't set protection, POCP says the branch is unchanged and lists the checks to require by hand; verification keeps working either way.
|
|
151
153
|
|
|
@@ -163,7 +165,7 @@ Anything that can run a command can use POCP. Two useful places:
|
|
|
163
165
|
|
|
164
166
|
- **Before editing:** `pocp check-edit <file>` exits 1 if the file needs an open packet first.
|
|
165
167
|
- **Before committing:** `pocp fix --intent "<what the change does>"` takes every mechanical step a blocked commit needs (open or widen a packet, add required checks, prove the staged code, stage the packet) and says what is left and who must do it. `pocp gate --json` gives the same reasons as data: each block has `remedies` with what is wrong, why, and the next command.
|
|
166
|
-
- **When a person is needed** (a test change the mechanical check could not pass, a policy or CI loosening), `pocp fix` and verify print the one action, e.g. "a maintainer adds the `pocp-approve-tests` label", and say plainly not to squash, force-push or use `--no-verify`. `pocp fix` never signs off for anyone.
|
|
168
|
+
- **When a person is needed** (a test change the mechanical check could not pass, a policy or CI loosening), `pocp fix` and verify print the one action, e.g. "a maintainer adds the `pocp-approve-tests` label to owner/repo#12 (or runs: pocp approve owner/repo#12)", and say plainly not to squash, force-push or use `--no-verify`. `pocp fix` never signs off for anyone.
|
|
167
169
|
- **Giving those approvals:** `pocp approvals` lists every open PR across your repositories whose verify fails only for a maintainer's label, with what each label approves (the reason and the test or CI/policy files, with line counts). `pocp approve owner/repo#12` (also `pocp approvals --approve`) shows every approval the PR waits for, each with its diff, and adds all of them only after you type the PR's number once; it refuses to run without a person at a terminal, has no flag that skips the question, and adds nothing if the PR got new commits while you read it. An agent may list what is waiting; it never approves.
|
|
168
170
|
- **Adopting:** `pocp doctor` checks hooks, policy, the CI workflow and its approval labels, Actions access to a private POCP action, branch protection against `ci.required_checks`, and that the first adoption PR carries only `.proof/` and CI files. It only reads.
|
|
169
171
|
|
|
@@ -194,6 +196,7 @@ pocp eval --mutants all --missed # how many planted bugs your req
|
|
|
194
196
|
- **`eval --corpus <file>`** runs mutants made by another tool through the same checks, so POCP's verdicts can be compared with that tool's mutant by mutant. Each entry is `{ id, file, start: { line, column }, end: { line, column }, replacement, original? }`: 1-based lines and columns with an exclusive end, as in StrykerJS's JSON report. If `original` doesn't match the text at that range, the mutant is reported as no longer applying. Each mutant's `outcome` is `caught`, `missed`, `timeout` or `error`. `eval/e3/` uses this to replay StrykerJS mutants.
|
|
195
197
|
- **`eval`** plants small bugs (flipped comparisons, swapped operators, changed constants), one at a time, in snapshots of files guarded by requirements, in parallel, running the fastest checks first. It refuses to run if the checks don't pass on the unmodified code, never mutates test files, and `--missed` shows each surviving mutant's line before and after.
|
|
196
198
|
- **Timeouts aren't catches.** A mutant is caught only when a check runs to the end and fails. If a check times out, is killed, or can't start (exit 126/127), the remaining checks still run; if none fails normally, the mutant is *not measured* (`outcome` `timeout` or `error`, `caught: null`). The catch rate is caught ÷ (caught + missed), and the not-measured count is shown beside it (`caught 8/9 (89%) · 1 not measured (1 timed out), not scored`) and saved as `not_measured`, `timeouts` and `errors`. `--replay --missed` re-runs the not-measured mutants along with the survivors. Under load a full suite can hit the check timeout whether or not it would catch the mutant, so counting it either way would let the machine's load set the score.
|
|
199
|
+
- **Where the test guard's time goes.** Each guard result has a `timings` field: `setup_ms` (snapshotting the new code and restoring the old tests), `new_baseline`, `old_baseline` and `base_check` (each with `wall_ms`, plus `run_ms` and `runs` for the checks run), `plan_ms` (choosing mutants), `mutants` (`wall_ms`, and `run_ms`, `runs` and `retries` per side), and `snapshot`, `copy` and `cleanup` (`ms` and `count`). `run_ms` and the snapshot, copy and cleanup times are summed across parallel jobs, so they can exceed the wall time; they also fall inside the phases' wall times, so don't add the two together. Wall times include waiting for a job slot, and in `verify` the two baselines run at once. In fast mode the changed-tests runs count toward `old_baseline`. A result reused from `prove`'s cache keeps the timings of the run that produced it. The field is for profiling only: nothing reads it to decide a verdict.
|
|
197
200
|
|
|
198
201
|
## Host modules
|
|
199
202
|
|
package/docs/RESULTS.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Results so far
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**The main result is the preregistered evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
|
|
4
4
|
|
|
5
|
-
These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which
|
|
5
|
+
These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which was first benchmarked in the 1.2.3 evaluation.
|
|
6
6
|
|
|
7
7
|
Scripted test suite ([bench/tamper/RESULTS.md](../bench/tamper/RESULTS.md)): 79 scripted scenarios on POCP's own repository, in deep mode with the other settings at their defaults unless noted.
|
|
8
8
|
- 73 of them plant a bug. Of those, 52 also cheat in a way that makes plain CI pass, and `verify --rerun` blocked all 52.
|