proof-of-change 1.3.0 → 1.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -8
- package/SPEC.md +2 -1
- package/docs/HOW-IT-WORKS.md +1 -1
- package/docs/QUICKSTART.md +15 -11
- package/docs/RESULTS.md +1 -1
- package/docs/STATUS.md +8 -12
- package/docs/THREAT-MODEL.md +17 -4
- package/examples/github/pocp-status.yml +7 -9
- package/examples/github/pocp.yml +13 -8
- package/package.json +1 -1
- package/src/ci-github.js +4 -2
- package/src/cli-args.js +3 -0
- package/src/cli.js +2 -0
- package/src/commands/init.js +1 -0
- package/src/commands/status-report.js +5 -2
- package/src/commands/verify.js +52 -3
- package/src/doctor.js +14 -23
- package/src/init.js +17 -6
- package/src/isolate.js +1 -1
- package/src/label-freshness.js +211 -0
- package/src/mutants.js +4 -3
- package/src/runner.js +27 -0
- package/src/status-report.js +36 -153
- package/examples/github/pocp-labels.yml +0 -53
package/README.md
CHANGED
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code and holds risky test edits for a maintainer. It works with any test command and any language, and needs only Node 20 or later, git and a POSIX shell.
|
|
4
4
|
|
|
5
|
-
It doesn't prove your code is correct. It has one
|
|
5
|
+
It doesn't prove your code is correct. It has one evaluation so far ([below](#evaluation)).
|
|
6
6
|
|
|
7
|
-

|
|
8
8
|
|
|
9
|
-
An agent edits a failing test until CI turns green. POCP runs the old test against the new code and
|
|
9
|
+
An agent edits a failing test until CI turns green. POCP runs the old test against the new code and holds the change for a person. Rather than wait for that approval, the agent puts the test back and fixes the code, and the pull request verifies. Run it from a clone with `node examples/demo/demo.mjs`.
|
|
10
10
|
|
|
11
11
|
## Install
|
|
12
12
|
|
|
@@ -39,22 +39,29 @@ npx pocp init --hooks --github
|
|
|
39
39
|
**On another CI** (GitLab, Jenkins, plain shell). In CI: keep your existing test step, and add:
|
|
40
40
|
|
|
41
41
|
```bash
|
|
42
|
-
|
|
42
|
+
npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
|
|
43
|
+
node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
|
|
43
44
|
```
|
|
44
45
|
|
|
46
|
+
Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does. Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc` and `node_modules`, so npx would run whatever `pocp` the pull request ships.
|
|
47
|
+
|
|
45
48
|
[docs/QUICKSTART.md](docs/QUICKSTART.md) walks through all of this on a non-JavaScript project.
|
|
46
49
|
|
|
47
50
|
## Evaluation
|
|
48
51
|
|
|
49
|
-
A
|
|
52
|
+
A benchmark of POCP 1.2.3 on p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, 68 of the cases held out until the scoring script was locked ([full report](eval/campaign-123/README.md)). The protocol was frozen by hash before the runs. The freeze record isn't public, so that order rests on my account. The raw records and the scripts that score them are public in [pocp-eval](https://github.com/harriswekselblatt/pocp-eval).
|
|
53
|
+
|
|
54
|
+
- **Harmful changes (48):** each breaks the code, then hides it with one of four scripted edits to the test that caught it: delete the assertion, empty the assertion helper, overwrite the expected-value fixture, or make the check script exit 0. Plain CI accepted all 48. POCP accepted none: in both modes it held each one for a person, with the old tests failing on the new code. That shows the old-tests check works on each part of a suite in three languages. It doesn't measure cheats that leave the old tests passing (no harmful case reached deep mode's planted bugs), or deliberate deception across pull requests ([threat model](docs/THREAT-MODEL.md)).
|
|
55
|
+
- **Honest changes (108):** deep mode accepts 78–82% per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%, about what a rule holding every test-file edit would. Reducing that friction is the next piece of work.
|
|
56
|
+
|
|
57
|
+
## Why not just require review of test files?
|
|
50
58
|
|
|
51
|
-
|
|
52
|
-
- **Honest changes:** deep mode accepts 78–82% of 108 legitimate changes per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%. Reducing that friction is the next piece of work.
|
|
59
|
+
Fast mode is close to that rule, with two differences. The reviewer gets evidence, not just a diff: POCP runs the old tests on the new code and names any that now fail. And what must pass is set by the base branch, so a pull request can't get green by changing its rules, its check command, its test script or its runner config. Deep mode goes further: its planted bugs let most honest test edits through with no person.
|
|
53
60
|
|
|
54
61
|
## Documentation
|
|
55
62
|
|
|
56
63
|
- [How it works](docs/HOW-IT-WORKS.md): what it catches, fast and deep modes, and what it doesn't do.
|
|
57
|
-
- [Evaluation of 1.2.3](eval/campaign-123/README.md): the
|
|
64
|
+
- [Evaluation of 1.2.3](eval/campaign-123/README.md): the benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
|
|
58
65
|
- [Reference](docs/REFERENCE.md): rules, checks, CI options, approvals, signed proofs.
|
|
59
66
|
- [Threat model](docs/THREAT-MODEL.md): what each record does and doesn't prove.
|
|
60
67
|
- [SPEC.md](SPEC.md): the protocol. [Reviewing POCP](docs/REVIEWING.md): a guide for reviewers.
|
package/SPEC.md
CHANGED
|
@@ -320,7 +320,8 @@ treats these changes as guarded:
|
|
|
320
320
|
- test files added, modified, deleted or renamed (an added test file runs with the old ones
|
|
321
321
|
and can change their outcome). The default
|
|
322
322
|
test patterns cover the common layouts of JavaScript/TypeScript (`test/`, `spec/`,
|
|
323
|
-
`__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`,
|
|
323
|
+
`__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, the standalone names `node --test` runs (`test.js`,
|
|
324
|
+
`test-*.js`, `*-test.js`), snapshots, `__mocks__/`), Python
|
|
324
325
|
(`tests/`, `test_*.py`, `*_test.py`), Ruby (`spec/`, `*_spec.rb`, `*_test.rb`), Go
|
|
325
326
|
(`*_test.go`, `testdata/`), Rust (`tests/`), Java/Kotlin/Scala (`src/test/`,
|
|
326
327
|
`src/androidTest/` and the other `src/*Test/` source sets, `*Test.java`, `*IT.java`), C#
|
package/docs/HOW-IT-WORKS.md
CHANGED
|
@@ -59,7 +59,7 @@ Within the limits in [docs/THREAT-MODEL.md](THREAT-MODEL.md):
|
|
|
59
59
|
|
|
60
60
|
- It doesn't judge whether code is honest. Code written to pass the specific test inputs passes every mechanical check.
|
|
61
61
|
- It doesn't enforce merges. `verify` fails a check; whether a failing check blocks the merge is your Git host's setting (on GitHub, branch protection).
|
|
62
|
-
- It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)).
|
|
62
|
+
- It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). Giving the reporter its own GitHub App closes this ([docs/STATUS.md](STATUS.md#give-the-reporter-its-own-identity)).
|
|
63
63
|
- It doesn't measure new tests. The test check compares a change with tests that already existed, so a weak test added in the same pull request isn't measured.
|
|
64
64
|
- It isn't hermetic. Checks can read the network, environment variables and installed packages, and the re-runs execute the pull request's code in your CI job.
|
|
65
65
|
- It adds CI time, and on a private repository GitHub bills it as Actions minutes. The status and label workflows skip events that can't change the result (other labels, title edits), but each push and approval still costs about a minute. `verify` re-runs your required checks, so each pull request costs about one extra test run, in parallel with your other jobs. On POCP's own pull requests in fast mode, `verify` took 3 to 4 minutes next to test jobs of 3 to 4 minutes, and 10 to 11 minutes when several commits edited tests, because the old tests ran as the whole suite. A rule's `changed_test_command` runs only the edited test files instead ([docs/REFERENCE.md](REFERENCE.md#faster-test-edits-changed_test_command)). It hasn't been measured on other projects yet.
|
package/docs/QUICKSTART.md
CHANGED
|
@@ -109,7 +109,7 @@ a maintainer's label on GitHub.
|
|
|
109
109
|
|
|
110
110
|
`pocp init --local` sets `ci.mode: "local"` in `.proof/config.json` and installs the hooks. On a
|
|
111
111
|
repository that already has POCP's workflows, `pocp ci off` disables them on GitHub (`pocp.yml`,
|
|
112
|
-
`pocp-status.yml
|
|
112
|
+
`pocp-status.yml`) and sets the mode; `pocp ci on` enables them again and
|
|
113
113
|
removes it. Neither touches the workflow files, so there is nothing to commit under
|
|
114
114
|
`.github/`; commit the change to `.proof/config.json`. Switching to local mode is a policy
|
|
115
115
|
loosening, so verify asks for `--accept-policy-change` on that commit. `ci off` lists your
|
|
@@ -132,8 +132,8 @@ then approve it for that exact head:
|
|
|
132
132
|
pocp verify --require-coverage --rerun main..HEAD --accept-test-change $(git rev-parse HEAD)
|
|
133
133
|
```
|
|
134
134
|
|
|
135
|
-
(or `--accept-policy-change`).
|
|
136
|
-
|
|
135
|
+
(or `--accept-policy-change`). verify then prints `verified`. It doesn't say an approval was
|
|
136
|
+
used, so the record of what you approved is the command you ran.
|
|
137
137
|
|
|
138
138
|
**A local approval is a flag, not a person.** Anything that can run commands on your machine,
|
|
139
139
|
an agent included, can pass `--accept-*` itself, just as it can run `git commit --no-verify`. It
|
|
@@ -148,12 +148,11 @@ GitHub doesn't start for billing reasons.
|
|
|
148
148
|
|
|
149
149
|
Before `pocp init --github`: the repository exists on GitHub, `origin` points at it, and `gh` is
|
|
150
150
|
logged in with the `workflow` permission (`gh auth refresh -s workflow`), which pushing workflow
|
|
151
|
-
files needs. `init --github` writes
|
|
151
|
+
files needs. `init --github` writes two workflows, with the POCP actions pinned to your POCP
|
|
152
152
|
release:
|
|
153
153
|
|
|
154
154
|
- **`pocp.yml`** runs verify on every pull request.
|
|
155
155
|
- **`pocp-status.yml`** is the trusted reporter: it sets one commit status, **`pocp`**. It's green when verified, yellow while waiting for a maintainer or when a check didn't finish, and red on a real failure. It never runs pull request code.
|
|
156
|
-
- **`pocp-labels.yml`** removes every approval label on each push, so an approval covers only the code reviewed.
|
|
157
156
|
|
|
158
157
|
Through your `gh` login it also creates the approval labels and turns the `pocp` status on
|
|
159
158
|
(the repository variable `POCP_STATUS_LIVE`), so a pull request waiting for approval shows
|
|
@@ -169,11 +168,11 @@ the status on.
|
|
|
169
168
|
|
|
170
169
|
If your repository has a second maintainer, keep the default `"ci": { "workflow_approver": "independent" }`: then the author of a CI change can't approve it themselves. See [STATUS.md](STATUS.md) for that, and for the limits of the `pocp` status.
|
|
171
170
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
`pocp doctor` says when
|
|
171
|
+
An approval covers only the code reviewed: a label added before the latest push doesn't count,
|
|
172
|
+
and the reporter removes it, so add it again after reviewing the new commits. Neither workflow
|
|
173
|
+
uses `pull_request_target` (which GitHub blocks by default in public repositories from
|
|
174
|
+
December 2, 2026). Upgrading from 1.3.0 or earlier: `pocp init --github --force` regenerates
|
|
175
|
+
the workflows and removes the old `pocp-labels.yml`; `pocp doctor` says when that's needed.
|
|
177
176
|
|
|
178
177
|
**If agents push branches with your token,** a workflow on such a branch can post its own
|
|
179
178
|
`pocp: success`. Give the reporter its own GitHub App and require `pocp` from it:
|
|
@@ -201,9 +200,14 @@ Verify every pull request. verify re-runs each required check itself, on the exa
|
|
|
201
200
|
by the base branch's rules; your existing test step stays as it is:
|
|
202
201
|
|
|
203
202
|
```bash
|
|
204
|
-
|
|
203
|
+
npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
|
|
204
|
+
node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
|
|
205
205
|
```
|
|
206
206
|
|
|
207
|
+
Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does.
|
|
208
|
+
Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc`
|
|
209
|
+
and `node_modules`, so npx would run whatever `pocp` the pull request ships.
|
|
210
|
+
|
|
207
211
|
A faster mode runs your tests once, through `pocp ci-run`, and passes `--ci-results` to verify.
|
|
208
212
|
It trusts everything that ran earlier in the job: a new test file can write a passing result
|
|
209
213
|
for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
|
package/docs/RESULTS.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Results so far
|
|
2
2
|
|
|
3
|
-
**The main result is the
|
|
3
|
+
**The main result is the evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
|
|
4
4
|
|
|
5
5
|
These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which was first benchmarked in the 1.2.3 evaluation.
|
|
6
6
|
|
package/docs/STATUS.md
CHANGED
|
@@ -19,22 +19,17 @@ In local mode (`pocp init --local` or `pocp ci off`, [QUICKSTART](QUICKSTART.md#
|
|
|
19
19
|
|
|
20
20
|
## Approvals
|
|
21
21
|
|
|
22
|
-
Each label covers one thing, only on the head it was added to.
|
|
22
|
+
Each label covers one thing, only on the head it was added to. A label counts only if someone with write access added it after this head's first verify run, and after the pull request's latest force-push (or it carries over through "Update branch", below). verify checks this itself (`label-freshness: github` in `pocp.yml`), and so does the reporter, both from GitHub's API. A label left on from before a push doesn't count: verify says "pocp-approve-tests was added before this commit: review the new commits and re-apply it", and the reporter removes it, so adding it again is the approval.
|
|
23
23
|
|
|
24
|
-
###
|
|
24
|
+
### No `pull_request_target` (since 1.3.1)
|
|
25
25
|
|
|
26
|
-
From December 2, 2026, GitHub blocks `pull_request_target` workflows by default in public repositories that have no Actions event policy ([GitHub's docs](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target)). `pocp
|
|
27
|
-
|
|
28
|
-
- **`pocp` status live (`POCP_STATUS_LIVE`, what `init --github` sets):** the reporter still refuses a label applied before this head's first run or before the pull request's latest force-push, and says to approve again. A label left on is untidy, not a way through.
|
|
29
|
-
- **Not live (`verify` is the required check):** a label left on after a push can let a later verify run pass on code nobody approved. Allow `pull_request_target` in the repository's Actions event policy (Settings → Actions), or turn the status live (QUICKSTART, "On GitHub", step 3).
|
|
26
|
+
From December 2, 2026, GitHub blocks `pull_request_target` workflows by default in public repositories that have no Actions event policy ([GitHub's docs](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target)). POCP's workflows don't use it, so there is nothing to allow in the repository's settings. `pocp.yml` runs on `pull_request` with read-only permissions, and its token goes only to verify's own process, never to a check it runs. `pocp-status.yml` runs on `workflow_run`. Up to 1.3.0, `pocp-labels.yml` removed approval labels on push, on `pull_request_target`: `pocp doctor` flags it, and a `pocp.yml` without `label-freshness`, and `pocp init --github --force` regenerates the workflows and removes it. An old `pocp.yml` still works as it did, with a warning on each run that passes an approval.
|
|
30
27
|
|
|
31
28
|
A pull request whose `pocp.yml` run never starts (a first-time contributor's fork, waiting for a maintainer to approve the run) gets no `pocp` status until it runs, so it can't merge.
|
|
32
29
|
|
|
33
|
-
POCP 1.3.1 removes the dependence on `pull_request_target`.
|
|
34
|
-
|
|
35
30
|
**Several approvals at once:** `pocp approve owner/repo#12` (also `pocp approvals --approve`) shows every approval the pull request waits for, with the diff each covers, and adds all of them after you type the pull request's number once. Your answer covers exactly what was listed: each label is still checked on its own, and anything verify reports later needs its own approval. It must be run by a person at a terminal. `pocp-approve-workflow` isn't included: add it yourself after reviewing the CI change. When verify waits for a label, its "what to do" (and the job summary, and `next_steps` in `verify --json`) marks that step `[maintainer, not the author]` and names the pull request and this command; it is never the author's or an agent's step.
|
|
36
31
|
|
|
37
|
-
**Updating the branch keeps the approval
|
|
32
|
+
**Updating the branch keeps the approval.** If the push is only a merge of the base branch into the approved head (GitHub's "Update branch"), `pocp-approve-tests`, `pocp-approve-policy` and `pocp-approve-coverage` still count for the new head, in verify and in the reporter, only if all of these hold:
|
|
38
33
|
|
|
39
34
|
- the new head's parents are the approved head and the pull request's current base;
|
|
40
35
|
- the label was last added by someone with write access, after the approved head's first run for this pull request;
|
|
@@ -43,7 +38,7 @@ POCP 1.3.1 removes the dependence on `pull_request_target`.
|
|
|
43
38
|
- since the approved head branched off, the base branch changed no policy file under `.proof/` (again, proof records don't count) and none of the files the approval covers;
|
|
44
39
|
- GitHub's file lists for the merge are complete.
|
|
45
40
|
|
|
46
|
-
Otherwise the status says to approve again: remove the label and add it back. `pocp approve owner/repo#N` reads that status and does it for you (after you confirm), even when verify itself passed with the label on. The approval is kept, never the result: the merged head is verified again on the new base and turns green only from its own run. It carries over one merge only: a second update before the label is added again needs a new approval. Inconclusive and workflow approvals never carry over, and any other push
|
|
41
|
+
Otherwise the status says to approve again: remove the label and add it back. `pocp approve owner/repo#N` reads that status and does it for you (after you confirm), even when verify itself passed with the label on. The approval is kept, never the result: the merged head is verified again on the new base and turns green only from its own run. It carries over one merge only: a second update before the label is added again needs a new approval. Inconclusive and workflow approvals never carry over, and after any other push every label needs adding again.
|
|
47
42
|
|
|
48
43
|
| Label | Covers |
|
|
49
44
|
|---|---|
|
|
@@ -70,7 +65,7 @@ The same holds for every approval label. A label shows that an account with writ
|
|
|
70
65
|
Two workflows:
|
|
71
66
|
|
|
72
67
|
- **`pocp.yml`** runs `verify` on the pull request, as before. Its first step, `pocp-run pr-<number> base-<base sha>`, names the pull request and base the run evaluated. It records verify's result in which step runs: exactly one of `verified`, `waiting`, `inconclusive` or `failed`, plus an `approval-*` step for each approval passed to verify. `verified` runs only when the verify step itself exited 0, which GitHub takes from the process's exit status. The step's `code` output, which picks between `waiting`, `inconclusive` and `failed`, is weaker, and never makes a run green. With `--isolate` (the example workflow's default, and POCP's own workflow once its base verifier supports it), the checks verify runs, which are the pull request's code, run as separate unprivileged users: they can't write the file GitHub reads outputs from, change the verifier, read the job's tokens or signal verify, and if that isolation can't be set up the run is red. Without `--isolate`, they run as the job's own user and could write that file, so the most a forged output can do is turn a failure into a wait (yellow) or the reverse. docs/THREAT-MODEL.md ("Pull-request code in the verifying job") says exactly what the isolation covers and what it doesn't: the network, the kernel, memory and disk, and anything that ran before verify in the job.
|
|
73
|
-
- **`pocp-status.yml`** is the trusted reporter. It runs from the default branch, on `workflow_run
|
|
68
|
+
- **`pocp-status.yml`** is the trusted reporter. It runs from the default branch, on `workflow_run`, and never checks out or runs anything from a pull request. On every event it recomputes from GitHub's API:
|
|
74
69
|
1. Which open pull request has this head, and whether it's still the head. If not, nothing is written.
|
|
75
70
|
2. **Whether the data is complete.** Every list it reads (open pull requests, the pull request's files, runs, jobs) must be read to the end, and the file list must match the pull request's changed-file count and stay under GitHub's 3000-file limit. Anything cut short, missing or inconsistent, or an API error or rate limit, is pending, whatever labels are on.
|
|
76
71
|
3. **Whether the head is shared.** A commit status belongs to the commit, so every pull request with that head shows it. If more than one open pull request has this head, the status stays pending.
|
|
@@ -79,6 +74,7 @@ Two workflows:
|
|
|
79
74
|
6. If that run relied on an approval, the label must still be on the pull request now, and it must have been applied last by someone with write access, after this head's first run and after the pull request's latest force-push (GitHub lets roles without write access manage labels; a force-push can bring back a head that ran before the label was given for another). The same holds for `pocp-approve-workflow`, which also has to come from someone other than the author.
|
|
80
75
|
7. The run must have every step this reporter reads (`verified`, `waiting`, `inconclusive`, `failed`). A run from an older `pocp.yml` stays pending with "update pocp.yml", never green. A newer run that doesn't name a pull request and base (it stopped before its first step, or comes from an older workflow) holds the status at pending: an older green is never revived past it.
|
|
81
76
|
8. Right before writing, it reads the pull request again. If the head moved, it writes nothing for the old head. If the base, the labels, or the set of open pull requests on that head changed, it writes pending instead, so an older green can't stand on evidence that is no longer current; the event that changed it runs the reporter again. Retargeting a pull request (`edited`) re-runs verify and the reporter.
|
|
77
|
+
9. After writing, it removes each approval label GitHub's data shows doesn't count for this head (added before its first run, before a force-push, or by someone without write access), with `github.token` (`pull-requests: write` on that job only), so the pull request shows it needs approving again. A label the data can't settle (an API error, a list cut short) is left on, and refused. The removal starts no new run: the status just written already refuses the label.
|
|
82
78
|
|
|
83
79
|
If your `pocp.yml` was written before the `pocp-run` step existed, add it (`pocp init --github --force` rewrites the example): without it no run counts, and the status stays pending.
|
|
84
80
|
|
|
@@ -96,7 +92,7 @@ Close that by giving the reporter its own GitHub App and requiring `pocp` from t
|
|
|
96
92
|
|
|
97
93
|
The example `pocp-status.yml` already runs in that environment and uses the app when `POCP_REPORTER_CLIENT_ID` is set; without it, it posts as GitHub Actions as before.
|
|
98
94
|
|
|
99
|
-
Why the environment matters: secrets in an environment limited to the default branch reach only jobs running from the default branch. The reporter runs from there (`workflow_run
|
|
95
|
+
Why the environment matters: secrets in an environment limited to the default branch reach only jobs running from the default branch. The reporter runs from there (`workflow_run`). A pull request's own workflows run from its branch or merge ref, so a branch that edits a workflow can't read the key, can't post as the app, and anything it posts as GitHub Actions doesn't count. Keeping the key in a plain repository secret would let any workflow on any branch read it, and the app would protect nothing.
|
|
100
96
|
|
|
101
97
|
What it doesn't cover: anyone who can change the repository's settings. An **admin** can loosen the environment's branch rule, the branch protection or the app, so a coding agent holding an admin token (a classic token with `repo` scope on a repository you own is one) can undo all of this. Give agents a fine-grained token without the *Administration* permission. Until the app is set up, read the diff of any same-repository pull request that touches CI files before merging.
|
|
102
98
|
|
package/docs/THREAT-MODEL.md
CHANGED
|
@@ -265,10 +265,23 @@ loosening in the change at once, including new signing keys.
|
|
|
265
265
|
|
|
266
266
|
Bind approvals to the reviewed code. `--accept-test-change=<sha>` and
|
|
267
267
|
`--accept-policy-change=<sha>` approve only when the head being verified is that commit. The
|
|
268
|
-
GitHub example
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
268
|
+
GitHub example passes each label with the head at that moment, and verify's
|
|
269
|
+
`--label-freshness=github` asks GitHub's API whether the label still counts for that head: its
|
|
270
|
+
last application is by someone with write access, after the head's first run of the workflow and
|
|
271
|
+
after the pull request's latest force-push, or it carries over to a clean merge of the base
|
|
272
|
+
(STATUS.md, "Updating the branch keeps the approval"). A label left on from before a push is
|
|
273
|
+
refused, so a new approval is a decision about the new code; the trusted reporter applies the
|
|
274
|
+
same rule (src/label-freshness.js) and removes such a label. Anything GitHub can't settle (an
|
|
275
|
+
API error, a list cut short, an unreadable permission) refuses the approval. (A label added in
|
|
276
|
+
the seconds between a push and that head's first run is refused too: add it again.)
|
|
277
|
+
|
|
278
|
+
The token for those reads (`actions: read`, `pull-requests: read`) is set only in the verify
|
|
279
|
+
step's environment. verify takes it out of its own environment before it installs or runs
|
|
280
|
+
anything, and isolated checks get an allowlisted environment that never includes it, nor can
|
|
281
|
+
they read verify's process. With `isolate: false` (the cooperative runner, not a security
|
|
282
|
+
boundary) a check running as the job's user could read it from verify's process: it is
|
|
283
|
+
read-only, and lasts as long as the job. Workflows written before 1.3.1 don't pass
|
|
284
|
+
`label-freshness`: their labels are taken as given, as before, and the action warns on each run.
|
|
272
285
|
On GitLab, a maintainer runs a pipeline with `POCP_ACCEPT_*` set to the reviewed head's sha;
|
|
273
286
|
a project-wide CI/CD variable would approve every merge request.
|
|
274
287
|
|
|
@@ -9,28 +9,25 @@
|
|
|
9
9
|
# own copy of pocp.yml, so if it edits that, the status stays yellow until a maintainer approves
|
|
10
10
|
# the workflow change with pocp-approve-workflow.
|
|
11
11
|
#
|
|
12
|
-
# workflow_run: when a pocp run finishes.
|
|
12
|
+
# workflow_run: when a pocp run finishes. pocp.yml runs on every push, retarget and approval label,
|
|
13
|
+
# so that covers each change (no pull_request_target, which GitHub blocks by default in public
|
|
14
|
+
# repositories). A label given before the head's first run doesn't count, and the reporter
|
|
15
|
+
# removes it with github.token, so the maintainer sees to approve again.
|
|
13
16
|
name: pocp-status
|
|
14
17
|
on:
|
|
15
18
|
workflow_run:
|
|
16
19
|
workflows: [pocp]
|
|
17
20
|
types: [completed]
|
|
18
|
-
pull_request_target:
|
|
19
|
-
types: [opened, synchronize, reopened, edited, labeled, unlabeled] # edited: a retarget changes the base that judges it
|
|
20
21
|
permissions:
|
|
21
22
|
contents: read
|
|
22
23
|
actions: read
|
|
23
|
-
pull-requests:
|
|
24
|
+
pull-requests: write # only github.token's, to remove the approval labels it finds stale
|
|
24
25
|
statuses: write
|
|
25
26
|
jobs:
|
|
26
27
|
report:
|
|
27
|
-
# Skipped (no Actions minutes) for events that can't change the status: a label other than
|
|
28
|
-
# pocp-approve-*, or an edit that doesn't change the base. Pushes, retargets, approval labels
|
|
29
|
-
# and finished verify runs always report.
|
|
30
|
-
if: github.event_name != 'pull_request_target' || ((github.event.action != 'labeled' && github.event.action != 'unlabeled') || startsWith(github.event.label.name, 'pocp-approve-')) && (github.event.action != 'edited' || github.event.changes.base)
|
|
31
28
|
# One at a time per head, none cancelled: each recomputes from scratch, so the last one wins.
|
|
32
29
|
concurrency:
|
|
33
|
-
group: pocp-status-${{ github.event.workflow_run.head_sha
|
|
30
|
+
group: pocp-status-${{ github.event.workflow_run.head_sha }}
|
|
34
31
|
cancel-in-progress: false
|
|
35
32
|
# The reporter's own identity (docs/STATUS.md, "Give the reporter its own identity"): the App's
|
|
36
33
|
# key lives in this environment, limited to the default branch, so only this job, running from
|
|
@@ -63,3 +60,4 @@ jobs:
|
|
|
63
60
|
with:
|
|
64
61
|
workflow: .github/workflows/pocp.yml
|
|
65
62
|
github-token: ${{ steps.app.outputs.token || github.token }}
|
|
63
|
+
labels-token: ${{ github.token }}
|
package/examples/github/pocp.yml
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# .github/workflows/pocp.yml
|
|
2
2
|
# POCP on GitHub Actions, for a project in any language. `pocp init --github` writes this file,
|
|
3
|
-
#
|
|
3
|
+
# and pocp-status.yml, pinned to your POCP version. docs/STATUS.md explains the
|
|
4
4
|
# "pocp" status these produce; docs/QUICKSTART.md, how to turn it on.
|
|
5
5
|
#
|
|
6
6
|
# verify judges the pull request by the base branch's rules and re-runs every required check
|
|
@@ -32,6 +32,8 @@ on:
|
|
|
32
32
|
types: [opened, synchronize, reopened, edited, labeled, unlabeled] # edited: a retarget changes the base that judges it
|
|
33
33
|
permissions:
|
|
34
34
|
contents: read
|
|
35
|
+
actions: read # label-freshness: when this head first ran
|
|
36
|
+
pull-requests: read # label-freshness: who applied each approval label, and when
|
|
35
37
|
jobs:
|
|
36
38
|
verify:
|
|
37
39
|
if: vars.POCP_STATUS_LIVE != 'true' || ((github.event.action != 'labeled' && github.event.action != 'unlabeled') || startsWith(github.event.label.name, 'pocp-approve-')) && (github.event.action != 'edited' || github.event.changes.base)
|
|
@@ -45,13 +47,14 @@ jobs:
|
|
|
45
47
|
# number of commits in a typical pull request, with margin.
|
|
46
48
|
timeout-minutes: 60
|
|
47
49
|
env:
|
|
48
|
-
# Each approval is a label only maintainers apply. A label isn't tied to a commit, so
|
|
49
|
-
#
|
|
50
|
-
#
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
50
|
+
# Each approval is a label only maintainers apply. A label isn't tied to a commit, so the
|
|
51
|
+
# head's sha is passed with it and verify (label-freshness, below) asks GitHub whether the
|
|
52
|
+
# label was applied by a writer after this head first ran: a label left on from before a
|
|
53
|
+
# push doesn't count, except through "Update branch" (a clean merge of the base).
|
|
54
|
+
APPROVE_POLICY: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-policy') && github.event.pull_request.head.sha || 'false' }}
|
|
55
|
+
APPROVE_TESTS: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-tests') && github.event.pull_request.head.sha || 'false' }}
|
|
56
|
+
APPROVE_INCONCLUSIVE: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-inconclusive') && github.event.pull_request.head.sha || 'false' }}
|
|
57
|
+
APPROVE_COVERAGE: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-coverage') && github.event.pull_request.head.sha || 'false' }}
|
|
55
58
|
WAIT_EXIT: ${{ vars.POCP_STATUS_LIVE == 'true' && '0' || '1' }}
|
|
56
59
|
steps:
|
|
57
60
|
# Which pull request and base this run evaluated. A head commit can belong to several pull
|
|
@@ -74,6 +77,8 @@ jobs:
|
|
|
74
77
|
id: verify
|
|
75
78
|
continue-on-error: true # the step fails unless verified; the steps below report why
|
|
76
79
|
with:
|
|
80
|
+
label-freshness: github # an approval's label must be given after this head first ran
|
|
81
|
+
github-token: ${{ github.token }} # read-only (permissions above); verify's process only
|
|
77
82
|
rerun: all
|
|
78
83
|
isolate: true # each check runs as a fresh unprivileged user (Linux runners)
|
|
79
84
|
# install: npm ci # ▶ your install command: runs isolated, before the checks
|
package/package.json
CHANGED
package/src/ci-github.js
CHANGED
|
@@ -2,7 +2,7 @@ import { spawnSync } from 'node:child_process';
|
|
|
2
2
|
import fs from 'node:fs';
|
|
3
3
|
import path from 'node:path';
|
|
4
4
|
import { git } from './git.js';
|
|
5
|
-
import { GITHUB_WORKFLOWS } from './init.js';
|
|
5
|
+
import { GITHUB_WORKFLOWS, LEGACY_GITHUB_WORKFLOWS } from './init.js';
|
|
6
6
|
|
|
7
7
|
// Branch protection from versioned config: which CI checks must pass before merging,
|
|
8
8
|
// whether changes must arrive through pull requests, and whether admins are held to it.
|
|
@@ -319,7 +319,9 @@ export function setPocpWorkflows(root, plan, { enable }) {
|
|
|
319
319
|
const repo = repoR.stdout.trim();
|
|
320
320
|
const { workflows, error } = listWorkflows(repo);
|
|
321
321
|
if (error) return { ok: false, repo, error };
|
|
322
|
-
|
|
322
|
+
// A leftover pocp-labels.yml (LEGACY_GITHUB_WORKFLOWS) is switched with the rest, when GitHub has it.
|
|
323
|
+
const legacy = LEGACY_GITHUB_WORKFLOWS.map((f) => `.github/workflows/${f}`);
|
|
324
|
+
const ours = [...GITHUB_WORKFLOWS.map((f) => `.github/workflows/${f}`), ...legacy.filter((f) => workflows.some((w) => w.path === f))];
|
|
323
325
|
const changed = [];
|
|
324
326
|
const unchanged = [];
|
|
325
327
|
const missing = [];
|
package/src/cli-args.js
CHANGED
|
@@ -187,6 +187,9 @@ export const COMMANDS = {
|
|
|
187
187
|
jobs: num('n', 'checks run at once (default: CPU cores - 1, at most 4)'),
|
|
188
188
|
isolate: bool('run each check as a fresh unprivileged user (Linux, passwordless sudo); fail if that is not possible'),
|
|
189
189
|
install: str('cmd', 'with --isolate: install dependencies at the head, isolated, before the checks'),
|
|
190
|
+
'label-freshness': str('github', 'refuse an approval whose label was added before this head (token in POCP_GITHUB_TOKEN)'),
|
|
191
|
+
repo: str('owner/name', 'with --label-freshness: the repository (default: GITHUB_REPOSITORY)'),
|
|
192
|
+
'pull-request': num('n', 'with --label-freshness: the pull request whose labels gave the approvals'),
|
|
190
193
|
},
|
|
191
194
|
},
|
|
192
195
|
blame: { summary: 'packets that changed a file', usage: ['blame <file>'], engine: true, flags: {} },
|
package/src/cli.js
CHANGED
|
@@ -152,6 +152,8 @@ Checking
|
|
|
152
152
|
didn't finish (re-run it), 1 for any other failure
|
|
153
153
|
--isolate: each check runs as a fresh unprivileged user (Linux, sudo); if
|
|
154
154
|
that can't be set up, verify fails. --install <cmd>: dependencies too
|
|
155
|
+
--label-freshness=github --repo <o/r> --pull-request <n>: an approval's label
|
|
156
|
+
must be added after this head first ran (token in POCP_GITHUB_TOKEN)
|
|
155
157
|
blame <file> packets that changed a file
|
|
156
158
|
|
|
157
159
|
Misses
|
package/src/commands/init.js
CHANGED
|
@@ -44,6 +44,7 @@ export async function run({ flags, json }) {
|
|
|
44
44
|
...(github ? [
|
|
45
45
|
github.written.length ? `wrote ${github.written.join(', ')} (POCP actions pinned to ${github.ref})` : 'GitHub workflows already present',
|
|
46
46
|
...github.skipped.map((f) => `kept ${f} (exists; --force replaces it)`),
|
|
47
|
+
...github.removed.map((f) => `removed ${f} (no longer needed: verify checks approval labels against the head)`),
|
|
47
48
|
...(labels?.ok ? [labels.created.length ? `created the approval labels on ${labels.repo}: ${labels.created.join(', ')}` : `approval labels already on ${labels.repo}`]
|
|
48
49
|
: labels ? [`approval labels not created: ${labels.error} (pocp doctor prints the commands)`] : []),
|
|
49
50
|
...(live?.ok ? [`${LIVE_VARIABLE} set on ${live.repo}: a pull request waiting for approval shows a yellow "${STATUS_CHECK}" status, not a red failure`, ...live.notes.map((n) => ` ${n}`)]
|
|
@@ -17,8 +17,11 @@ export async function run({ flags, json }) {
|
|
|
17
17
|
const cfgFile = path.join(process.cwd(), '.proof', 'config.json');
|
|
18
18
|
const baseConfig = fs.existsSync(cfgFile) ? withDefaults(JSON.parse(fs.readFileSync(cfgFile, 'utf8'))) : null;
|
|
19
19
|
const gh = githubClient({ token: process.env.GH_TOKEN || process.env.GITHUB_TOKEN });
|
|
20
|
-
|
|
20
|
+
// LABELS_TOKEN (github.token with pull-requests: write): removes the approval labels found
|
|
21
|
+
// stale. GH_TOKEN may be the reporter App's, which only reads pull requests.
|
|
22
|
+
const labelGh = process.env.LABELS_TOKEN ? githubClient({ token: process.env.LABELS_TOKEN }) : null;
|
|
23
|
+
const r = await reportStatus(gh, { repository, sha, prNumber, baseConfig, workflowPath: flags.workflow || undefined, dryRun: Boolean(flags['dry-run']), labelGh });
|
|
21
24
|
const d = r.decision;
|
|
22
|
-
out(json, r, d.write ? `${r.written ? 'set' : 'would set'} ${STATUS_CONTEXT} on ${d.sha.slice(0, 12)}: ${d.state} (${d.description}); ${d.reason}` : `nothing written: ${d.reason}`);
|
|
25
|
+
out(json, r, `${d.write ? `${r.written ? 'set' : 'would set'} ${STATUS_CONTEXT} on ${d.sha.slice(0, 12)}: ${d.state} (${d.description}); ${d.reason}` : `nothing written: ${d.reason}`}${r.removed?.length ? `\nremoved stale approval label(s): ${r.removed.join(', ')}` : ''}`);
|
|
23
26
|
return EXIT.ok;
|
|
24
27
|
}
|
package/src/commands/verify.js
CHANGED
|
@@ -12,16 +12,57 @@ import { failedOutputLines } from '../output-tail.js';
|
|
|
12
12
|
import { git, resolveRev } from '../git.js';
|
|
13
13
|
import { readCiResults } from '../ci-results.js';
|
|
14
14
|
import { shortDigest } from '../hash.js';
|
|
15
|
+
import { UsageError } from '../cli-args.js';
|
|
16
|
+
import { freshnessFacts, labelFreshness } from '../label-freshness.js';
|
|
17
|
+
import { githubClient } from '../status-report.js';
|
|
18
|
+
|
|
19
|
+
// Which label gives each approval (examples/github/pocp.yml).
|
|
20
|
+
const APPROVAL_LABELS = Object.freeze({
|
|
21
|
+
'accept-test-change': 'pocp-approve-tests',
|
|
22
|
+
'accept-policy-change': 'pocp-approve-policy',
|
|
23
|
+
'accept-inconclusive': 'pocp-approve-inconclusive',
|
|
24
|
+
'accept-uncovered': 'pocp-approve-coverage',
|
|
25
|
+
});
|
|
26
|
+
|
|
27
|
+
/**
|
|
28
|
+
* --label-freshness=github: whether each approval's label counts for this head, by the rule the
|
|
29
|
+
* reporter uses (src/label-freshness.js), from GitHub's API. { label: verdict } for the
|
|
30
|
+
* approvals given. One GitHub can't settle is refused, never assumed.
|
|
31
|
+
*/
|
|
32
|
+
async function labelVerdicts(engine, flags, headRev, token) {
|
|
33
|
+
if (flags['label-freshness'] !== 'github') throw new UsageError(`--label-freshness takes "github", not "${flags['label-freshness']}"`);
|
|
34
|
+
if (!token) throw new UsageError("--label-freshness=github needs a token in POCP_GITHUB_TOKEN (the action's github-token input)");
|
|
35
|
+
const repository = flags.repo || process.env.GITHUB_REPOSITORY;
|
|
36
|
+
const prNumber = Number(flags['pull-request']) || null;
|
|
37
|
+
if (!repository || !prNumber) throw new UsageError('--label-freshness=github needs --repo <owner/name> (or GITHUB_REPOSITORY) and --pull-request <n>');
|
|
38
|
+
const given = (name) => [].concat(flags[name] ?? false).some((v) => v && v !== 'false');
|
|
39
|
+
const labels = Object.entries(APPROVAL_LABELS).filter(([name]) => given(name)).map(([, label]) => label);
|
|
40
|
+
const verdicts = {};
|
|
41
|
+
if (!labels.length) return verdicts;
|
|
42
|
+
const sha = resolveRev(engine.root, headRev);
|
|
43
|
+
try {
|
|
44
|
+
const facts = await freshnessFacts(githubClient({ token, api: process.env.GITHUB_API_URL || undefined }), { repository, prNumber, sha, labels });
|
|
45
|
+
for (const label of labels) verdicts[label] = labelFreshness(facts, label, sha);
|
|
46
|
+
} catch (e) {
|
|
47
|
+
for (const label of labels) verdicts[label] = { ok: false, stale: false, why: `couldn't check ${label} against GitHub (${e.message}): re-run this job` };
|
|
48
|
+
}
|
|
49
|
+
return verdicts;
|
|
50
|
+
}
|
|
15
51
|
|
|
16
52
|
export async function run({ engine, flags, isolated, json, rest }) {
|
|
17
53
|
const started = Date.now();
|
|
54
|
+
// The token for --label-freshness: read, then taken out of this process's environment before
|
|
55
|
+
// anything else runs, so no check (cooperative or isolated) or install inherits it.
|
|
56
|
+
const ghToken = process.env.POCP_GITHUB_TOKEN || '';
|
|
57
|
+
delete process.env.POCP_GITHUB_TOKEN;
|
|
58
|
+
const headRev = rest[0] && rest[0].includes('..') ? rest[0].split(/\.\.\.?/)[1] || 'HEAD' : rest[0] || 'HEAD';
|
|
59
|
+
const verdicts = flags['label-freshness'] ? await labelVerdicts(engine, flags, headRev, ghToken) : null;
|
|
18
60
|
// Before any of the pull request's code runs: set up and probe the isolation. If it
|
|
19
61
|
// can't be set up, this throws and verify fails; it never falls back.
|
|
20
62
|
const isolation = isolated ? isolated.isolation : COOPERATIVE;
|
|
21
63
|
if (isolated) {
|
|
22
64
|
await isolated.prepare({ root: engine.root });
|
|
23
65
|
if (flags.install) {
|
|
24
|
-
const headRev = rest[0] && rest[0].includes('..') ? rest[0].split(/\.\.\.?/)[1] || 'HEAD' : rest[0] || 'HEAD';
|
|
25
66
|
const cfg = engine.config();
|
|
26
67
|
const co = createRevCheckout(engine.root, resolveRev(engine.root, headRev), { links: [] });
|
|
27
68
|
try {
|
|
@@ -43,9 +84,16 @@ export async function run({ engine, flags, isolated, json, rest }) {
|
|
|
43
84
|
});
|
|
44
85
|
if (ignoredCi) errWrite(`pocp: ignored ${ignoredCi} CI result(s) recorded by a different CI run\n`);
|
|
45
86
|
if (ciResults.length) errWrite('pocp: trusting CI results from --ci-results: anything that ran earlier in this job could have written them (docs/THREAT-MODEL.md)\n');
|
|
46
|
-
// An approval that names a commit covers only that head: a later push needs a new one.
|
|
47
|
-
|
|
87
|
+
// An approval that names a commit covers only that head: a later push needs a new one. With
|
|
88
|
+
// --label-freshness, its label must also count for this head (labelVerdicts).
|
|
48
89
|
const approval = (name) => {
|
|
90
|
+
const ok = approvalFor(name);
|
|
91
|
+
const verdict = verdicts?.[APPROVAL_LABELS[name]];
|
|
92
|
+
if (!ok || !verdict || verdict.ok) return ok;
|
|
93
|
+
errWrite(`pocp: ${APPROVAL_LABELS[name]} not applied: ${verdict.why}\n`);
|
|
94
|
+
return false;
|
|
95
|
+
};
|
|
96
|
+
const approvalFor = (name) => {
|
|
49
97
|
const v = [].concat(flags[name] ?? false).pop();
|
|
50
98
|
if (typeof v !== 'string') return Boolean(v);
|
|
51
99
|
const head = resolveRev(engine.root, headRev);
|
|
@@ -132,6 +180,7 @@ export async function run({ engine, flags, isolated, json, rest }) {
|
|
|
132
180
|
].join('\n');
|
|
133
181
|
}
|
|
134
182
|
result.isolation = { mode: isolation.mode, secure: isolation.secure };
|
|
183
|
+
if (verdicts) result.label_freshness = Object.fromEntries(Object.entries(verdicts).map(([l, v]) => [l, v.ok ? { ok: true, how: v.how } : v]));
|
|
135
184
|
// One decision for the library and the CLI, from typed findings (src/decision.js).
|
|
136
185
|
if (!result.ok) result.approval = decideVerify(result);
|
|
137
186
|
if (steps.length) result.next_steps = steps;
|