proof-of-change 1.3.0 → 1.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,11 +2,11 @@
2
2
 
3
3
  POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code and holds risky test edits for a maintainer. It works with any test command and any language, and needs only Node 20 or later, git and a POSIX shell.
4
4
 
5
- It doesn't prove your code is correct. It has one preregistered evaluation so far ([below](#evaluation)).
5
+ It doesn't prove your code is correct. It has one evaluation so far ([below](#evaluation)).
6
6
 
7
- ![Terminal demo: an agent breaks add() and edits the test so npm test passes; pocp verify reports that the old test fails on the new code and waits for a maintainer](docs/demo.svg)
7
+ ![Terminal demo: an agent breaks add() and edits the test so npm test passes; pocp verify reports that the old test fails on the new code and holds the change for a maintainer; the agent puts the test back, fixes add(), and verify passes](docs/demo.svg)
8
8
 
9
- An agent edits a failing test until CI turns green. POCP runs the old test against the new code and routes the change to a person. Run it from a clone with `node examples/demo/demo.mjs`.
9
+ An agent edits a failing test until CI turns green. POCP runs the old test against the new code and holds the change for a person. Rather than wait for that approval, the agent puts the test back and fixes the code, and the pull request verifies. Run it from a clone with `node examples/demo/demo.mjs`.
10
10
 
11
11
  ## Install
12
12
 
@@ -39,22 +39,29 @@ npx pocp init --hooks --github
39
39
  **On another CI** (GitLab, Jenkins, plain shell). In CI: keep your existing test step, and add:
40
40
 
41
41
  ```bash
42
- npx pocp verify origin/main..HEAD --require-coverage --rerun
42
+ npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
43
+ node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
43
44
  ```
44
45
 
46
+ Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does. Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc` and `node_modules`, so npx would run whatever `pocp` the pull request ships.
47
+
45
48
  [docs/QUICKSTART.md](docs/QUICKSTART.md) walks through all of this on a non-JavaScript project.
46
49
 
47
50
  ## Evaluation
48
51
 
49
- A preregistered benchmark of POCP 1.2.3 across p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, with a sealed holdout ([full report](eval/campaign-123/README.md)).
52
+ A benchmark of POCP 1.2.3 on p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, 68 of the cases held out until the scoring script was locked ([full report](eval/campaign-123/README.md)). The protocol was frozen by hash before the runs. The freeze record isn't public, so that order rests on my account. The raw records and the scripts that score them are public in [pocp-eval](https://github.com/harriswekselblatt/pocp-eval).
53
+
54
+ - **Harmful changes (48):** each breaks the code, then hides it with one of four scripted edits to the test that caught it: delete the assertion, empty the assertion helper, overwrite the expected-value fixture, or make the check script exit 0. Plain CI accepted all 48. POCP accepted none: in both modes it held each one for a person, with the old tests failing on the new code. That shows the old-tests check works on each part of a suite in three languages. It doesn't measure cheats that leave the old tests passing (no harmful case reached deep mode's planted bugs), or deliberate deception across pull requests ([threat model](docs/THREAT-MODEL.md)).
55
+ - **Honest changes (108):** deep mode accepts 78–82% per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%, about what a rule holding every test-file edit would. Reducing that friction is the next piece of work.
56
+
57
+ ## Why not just require review of test files?
50
58
 
51
- - **Harmful changes:** plain CI accepted all 48 designed test-cheats. POCP accepted none. In both modes it held every one for a person's review, each with a completed check showing the tested behavior changed. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([threat model](docs/THREAT-MODEL.md)).
52
- - **Honest changes:** deep mode accepts 78–82% of 108 legitimate changes per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%. Reducing that friction is the next piece of work.
59
+ Fast mode is close to that rule, with two differences. The reviewer gets evidence, not just a diff: POCP runs the old tests on the new code and names any that now fail. And what must pass is set by the base branch, so a pull request can't get green by changing its rules, its check command, its test script or its runner config. Deep mode goes further: its planted bugs let most honest test edits through with no person.
53
60
 
54
61
  ## Documentation
55
62
 
56
63
  - [How it works](docs/HOW-IT-WORKS.md): what it catches, fast and deep modes, and what it doesn't do.
57
- - [Evaluation of 1.2.3](eval/campaign-123/README.md): the preregistered benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
64
+ - [Evaluation of 1.2.3](eval/campaign-123/README.md): the benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
58
65
  - [Reference](docs/REFERENCE.md): rules, checks, CI options, approvals, signed proofs.
59
66
  - [Threat model](docs/THREAT-MODEL.md): what each record does and doesn't prove.
60
67
  - [SPEC.md](SPEC.md): the protocol. [Reviewing POCP](docs/REVIEWING.md): a guide for reviewers.
package/SPEC.md CHANGED
@@ -320,7 +320,8 @@ treats these changes as guarded:
320
320
  - test files added, modified, deleted or renamed (an added test file runs with the old ones
321
321
  and can change their outcome). The default
322
322
  test patterns cover the common layouts of JavaScript/TypeScript (`test/`, `spec/`,
323
- `__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, snapshots, `__mocks__/`), Python
323
+ `__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, the standalone names `node --test` runs (`test.js`,
324
+ `test-*.js`, `*-test.js`), snapshots, `__mocks__/`), Python
324
325
  (`tests/`, `test_*.py`, `*_test.py`), Ruby (`spec/`, `*_spec.rb`, `*_test.rb`), Go
325
326
  (`*_test.go`, `testdata/`), Rust (`tests/`), Java/Kotlin/Scala (`src/test/`,
326
327
  `src/androidTest/` and the other `src/*Test/` source sets, `*Test.java`, `*IT.java`), C#
@@ -59,7 +59,7 @@ Within the limits in [docs/THREAT-MODEL.md](THREAT-MODEL.md):
59
59
 
60
60
  - It doesn't judge whether code is honest. Code written to pass the specific test inputs passes every mechanical check.
61
61
  - It doesn't enforce merges. `verify` fails a check; whether a failing check blocks the merge is your Git host's setting (on GitHub, branch protection).
62
- - It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). A dedicated GitHub App for the reporter, which would close this, is planned.
62
+ - It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). Giving the reporter its own GitHub App closes this ([docs/STATUS.md](STATUS.md#give-the-reporter-its-own-identity)).
63
63
  - It doesn't measure new tests. The test check compares a change with tests that already existed, so a weak test added in the same pull request isn't measured.
64
64
  - It isn't hermetic. Checks can read the network, environment variables and installed packages, and the re-runs execute the pull request's code in your CI job.
65
65
  - It adds CI time, and on a private repository GitHub bills it as Actions minutes. The status and label workflows skip events that can't change the result (other labels, title edits), but each push and approval still costs about a minute. `verify` re-runs your required checks, so each pull request costs about one extra test run, in parallel with your other jobs. On POCP's own pull requests in fast mode, `verify` took 3 to 4 minutes next to test jobs of 3 to 4 minutes, and 10 to 11 minutes when several commits edited tests, because the old tests ran as the whole suite. A rule's `changed_test_command` runs only the edited test files instead ([docs/REFERENCE.md](REFERENCE.md#faster-test-edits-changed_test_command)). It hasn't been measured on other projects yet.
@@ -109,7 +109,7 @@ a maintainer's label on GitHub.
109
109
 
110
110
  `pocp init --local` sets `ci.mode: "local"` in `.proof/config.json` and installs the hooks. On a
111
111
  repository that already has POCP's workflows, `pocp ci off` disables them on GitHub (`pocp.yml`,
112
- `pocp-status.yml`, `pocp-labels.yml`) and sets the mode; `pocp ci on` enables them again and
112
+ `pocp-status.yml`) and sets the mode; `pocp ci on` enables them again and
113
113
  removes it. Neither touches the workflow files, so there is nothing to commit under
114
114
  `.github/`; commit the change to `.proof/config.json`. Switching to local mode is a policy
115
115
  loosening, so verify asks for `--accept-policy-change` on that commit. `ci off` lists your
@@ -132,8 +132,8 @@ then approve it for that exact head:
132
132
  pocp verify --require-coverage --rerun main..HEAD --accept-test-change $(git rev-parse HEAD)
133
133
  ```
134
134
 
135
- (or `--accept-policy-change`). The output still lists the test guard's re-run as failed; the
136
- approval is what lets it through, and verify then prints `verified`.
135
+ (or `--accept-policy-change`). verify then prints `verified`. It doesn't say an approval was
136
+ used, so the record of what you approved is the command you ran.
137
137
 
138
138
  **A local approval is a flag, not a person.** Anything that can run commands on your machine,
139
139
  an agent included, can pass `--accept-*` itself, just as it can run `git commit --no-verify`. It
@@ -148,12 +148,11 @@ GitHub doesn't start for billing reasons.
148
148
 
149
149
  Before `pocp init --github`: the repository exists on GitHub, `origin` points at it, and `gh` is
150
150
  logged in with the `workflow` permission (`gh auth refresh -s workflow`), which pushing workflow
151
- files needs. `init --github` writes three workflows, with the POCP actions pinned to your POCP
151
+ files needs. `init --github` writes two workflows, with the POCP actions pinned to your POCP
152
152
  release:
153
153
 
154
154
  - **`pocp.yml`** runs verify on every pull request.
155
155
  - **`pocp-status.yml`** is the trusted reporter: it sets one commit status, **`pocp`**. It's green when verified, yellow while waiting for a maintainer or when a check didn't finish, and red on a real failure. It never runs pull request code.
156
- - **`pocp-labels.yml`** removes every approval label on each push, so an approval covers only the code reviewed.
157
156
 
158
157
  Through your `gh` login it also creates the approval labels and turns the `pocp` status on
159
158
  (the repository variable `POCP_STATUS_LIVE`), so a pull request waiting for approval shows
@@ -169,11 +168,11 @@ the status on.
169
168
 
170
169
  If your repository has a second maintainer, keep the default `"ci": { "workflow_approver": "independent" }`: then the author of a CI change can't approve it themselves. See [STATUS.md](STATUS.md) for that, and for the limits of the `pocp` status.
171
170
 
172
- **Public repositories, from December 2, 2026:** GitHub blocks `pull_request_target` workflows by
173
- default, and then approval labels are no longer removed on push. With the `pocp` status live
174
- this is only untidy; without it, allow `pull_request_target` in the repository's Actions settings
175
- ([STATUS.md](STATUS.md#githubs-pull_request_target-block-public-repositories-from-december-2-2026)).
176
- `pocp doctor` says when it is blocked.
171
+ An approval covers only the code reviewed: a label added before the latest push doesn't count,
172
+ and the reporter removes it, so add it again after reviewing the new commits. Neither workflow
173
+ uses `pull_request_target` (which GitHub blocks by default in public repositories from
174
+ December 2, 2026). Upgrading from 1.3.0 or earlier: `pocp init --github --force` regenerates
175
+ the workflows and removes the old `pocp-labels.yml`; `pocp doctor` says when that's needed.
177
176
 
178
177
  **If agents push branches with your token,** a workflow on such a branch can post its own
179
178
  `pocp: success`. Give the reporter its own GitHub App and require `pocp` from it:
@@ -201,9 +200,14 @@ Verify every pull request. verify re-runs each required check itself, on the exa
201
200
  by the base branch's rules; your existing test step stays as it is:
202
201
 
203
202
  ```bash
204
- pocp verify origin/main..HEAD --require-coverage --rerun
203
+ npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
204
+ node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
205
205
  ```
206
206
 
207
+ Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does.
208
+ Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc`
209
+ and `node_modules`, so npx would run whatever `pocp` the pull request ships.
210
+
207
211
  A faster mode runs your tests once, through `pocp ci-run`, and passes `--ci-results` to verify.
208
212
  It trusts everything that ran earlier in the job: a new test file can write a passing result
209
213
  for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
package/docs/RESULTS.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Results so far
2
2
 
3
- **The main result is the preregistered evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
3
+ **The main result is the evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
4
4
 
5
5
  These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which was first benchmarked in the 1.2.3 evaluation.
6
6
 
package/docs/STATUS.md CHANGED
@@ -19,22 +19,17 @@ In local mode (`pocp init --local` or `pocp ci off`, [QUICKSTART](QUICKSTART.md#
19
19
 
20
20
  ## Approvals
21
21
 
22
- Each label covers one thing, only on the head it was added to. Every push removes all of them (`pocp-labels.yml`).
22
+ Each label covers one thing, only on the head it was added to. A label counts only if someone with write access added it after this head's first verify run, and after the pull request's latest force-push (or it carries over through "Update branch", below). verify checks this itself (`label-freshness: github` in `pocp.yml`), and so does the reporter, both from GitHub's API. A label left on from before a push doesn't count: verify says "pocp-approve-tests was added before this commit: review the new commits and re-apply it", and the reporter removes it, so adding it again is the approval.
23
23
 
24
- ### GitHub's `pull_request_target` block (public repositories, from December 2, 2026)
24
+ ### No `pull_request_target` (since 1.3.1)
25
25
 
26
- From December 2, 2026, GitHub blocks `pull_request_target` workflows by default in public repositories that have no Actions event policy ([GitHub's docs](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target)). `pocp-labels.yml` runs on it, and so does part of `pocp-status.yml`. While it is blocked, their runs end in `startup_failure`, and `pocp doctor` reports it. The `pocp` status still arrives, through `workflow_run` once verify finishes, but **approval labels are no longer removed on push**. What that means depends on the mode:
27
-
28
- - **`pocp` status live (`POCP_STATUS_LIVE`, what `init --github` sets):** the reporter still refuses a label applied before this head's first run or before the pull request's latest force-push, and says to approve again. A label left on is untidy, not a way through.
29
- - **Not live (`verify` is the required check):** a label left on after a push can let a later verify run pass on code nobody approved. Allow `pull_request_target` in the repository's Actions event policy (Settings → Actions), or turn the status live (QUICKSTART, "On GitHub", step 3).
26
+ From December 2, 2026, GitHub blocks `pull_request_target` workflows by default in public repositories that have no Actions event policy ([GitHub's docs](https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target)). POCP's workflows don't use it, so there is nothing to allow in the repository's settings. `pocp.yml` runs on `pull_request` with read-only permissions, and its token goes only to verify's own process, never to a check it runs. `pocp-status.yml` runs on `workflow_run`. Up to 1.3.0, `pocp-labels.yml` removed approval labels on push, on `pull_request_target`: `pocp doctor` flags it, and a `pocp.yml` without `label-freshness`, and `pocp init --github --force` regenerates the workflows and removes it. An old `pocp.yml` still works as it did, with a warning on each run that passes an approval.
30
27
 
31
28
  A pull request whose `pocp.yml` run never starts (a first-time contributor's fork, waiting for a maintainer to approve the run) gets no `pocp` status until it runs, so it can't merge.
32
29
 
33
- POCP 1.3.1 removes the dependence on `pull_request_target`.
34
-
35
30
  **Several approvals at once:** `pocp approve owner/repo#12` (also `pocp approvals --approve`) shows every approval the pull request waits for, with the diff each covers, and adds all of them after you type the pull request's number once. Your answer covers exactly what was listed: each label is still checked on its own, and anything verify reports later needs its own approval. It must be run by a person at a terminal. `pocp-approve-workflow` isn't included: add it yourself after reviewing the CI change. When verify waits for a label, its "what to do" (and the job summary, and `next_steps` in `verify --json`) marks that step `[maintainer, not the author]` and names the pull request and this command; it is never the author's or an agent's step.
36
31
 
37
- **Updating the branch keeps the approval** when the `pocp` status is live (`POCP_STATUS_LIVE`). If the push is only a merge of the base branch into the approved head (GitHub's "Update branch"), `pocp-labels.yml` keeps `pocp-approve-tests`, `pocp-approve-policy` and `pocp-approve-coverage`. The reporter then counts the label for the new head only if all of these hold:
32
+ **Updating the branch keeps the approval.** If the push is only a merge of the base branch into the approved head (GitHub's "Update branch"), `pocp-approve-tests`, `pocp-approve-policy` and `pocp-approve-coverage` still count for the new head, in verify and in the reporter, only if all of these hold:
38
33
 
39
34
  - the new head's parents are the approved head and the pull request's current base;
40
35
  - the label was last added by someone with write access, after the approved head's first run for this pull request;
@@ -43,7 +38,7 @@ POCP 1.3.1 removes the dependence on `pull_request_target`.
43
38
  - since the approved head branched off, the base branch changed no policy file under `.proof/` (again, proof records don't count) and none of the files the approval covers;
44
39
  - GitHub's file lists for the merge are complete.
45
40
 
46
- Otherwise the status says to approve again: remove the label and add it back. `pocp approve owner/repo#N` reads that status and does it for you (after you confirm), even when verify itself passed with the label on. The approval is kept, never the result: the merged head is verified again on the new base and turns green only from its own run. It carries over one merge only: a second update before the label is added again needs a new approval. Inconclusive and workflow approvals never carry over, and any other push still removes every label.
41
+ Otherwise the status says to approve again: remove the label and add it back. `pocp approve owner/repo#N` reads that status and does it for you (after you confirm), even when verify itself passed with the label on. The approval is kept, never the result: the merged head is verified again on the new base and turns green only from its own run. It carries over one merge only: a second update before the label is added again needs a new approval. Inconclusive and workflow approvals never carry over, and after any other push every label needs adding again.
47
42
 
48
43
  | Label | Covers |
49
44
  |---|---|
@@ -70,7 +65,7 @@ The same holds for every approval label. A label shows that an account with writ
70
65
  Two workflows:
71
66
 
72
67
  - **`pocp.yml`** runs `verify` on the pull request, as before. Its first step, `pocp-run pr-<number> base-<base sha>`, names the pull request and base the run evaluated. It records verify's result in which step runs: exactly one of `verified`, `waiting`, `inconclusive` or `failed`, plus an `approval-*` step for each approval passed to verify. `verified` runs only when the verify step itself exited 0, which GitHub takes from the process's exit status. The step's `code` output, which picks between `waiting`, `inconclusive` and `failed`, is weaker, and never makes a run green. With `--isolate` (the example workflow's default, and POCP's own workflow once its base verifier supports it), the checks verify runs, which are the pull request's code, run as separate unprivileged users: they can't write the file GitHub reads outputs from, change the verifier, read the job's tokens or signal verify, and if that isolation can't be set up the run is red. Without `--isolate`, they run as the job's own user and could write that file, so the most a forged output can do is turn a failure into a wait (yellow) or the reverse. docs/THREAT-MODEL.md ("Pull-request code in the verifying job") says exactly what the isolation covers and what it doesn't: the network, the kernel, memory and disk, and anything that ran before verify in the job.
73
- - **`pocp-status.yml`** is the trusted reporter. It runs from the default branch, on `workflow_run` and `pull_request_target` (see GitHub's block above), and never checks out or runs anything from a pull request. On every event it recomputes from GitHub's API:
68
+ - **`pocp-status.yml`** is the trusted reporter. It runs from the default branch, on `workflow_run`, and never checks out or runs anything from a pull request. On every event it recomputes from GitHub's API:
74
69
  1. Which open pull request has this head, and whether it's still the head. If not, nothing is written.
75
70
  2. **Whether the data is complete.** Every list it reads (open pull requests, the pull request's files, runs, jobs) must be read to the end, and the file list must match the pull request's changed-file count and stay under GitHub's 3000-file limit. Anything cut short, missing or inconsistent, or an API error or rate limit, is pending, whatever labels are on.
76
71
  3. **Whether the head is shared.** A commit status belongs to the commit, so every pull request with that head shows it. If more than one open pull request has this head, the status stays pending.
@@ -79,6 +74,7 @@ Two workflows:
79
74
  6. If that run relied on an approval, the label must still be on the pull request now, and it must have been applied last by someone with write access, after this head's first run and after the pull request's latest force-push (GitHub lets roles without write access manage labels; a force-push can bring back a head that ran before the label was given for another). The same holds for `pocp-approve-workflow`, which also has to come from someone other than the author.
80
75
  7. The run must have every step this reporter reads (`verified`, `waiting`, `inconclusive`, `failed`). A run from an older `pocp.yml` stays pending with "update pocp.yml", never green. A newer run that doesn't name a pull request and base (it stopped before its first step, or comes from an older workflow) holds the status at pending: an older green is never revived past it.
81
76
  8. Right before writing, it reads the pull request again. If the head moved, it writes nothing for the old head. If the base, the labels, or the set of open pull requests on that head changed, it writes pending instead, so an older green can't stand on evidence that is no longer current; the event that changed it runs the reporter again. Retargeting a pull request (`edited`) re-runs verify and the reporter.
77
+ 9. After writing, it removes each approval label GitHub's data shows doesn't count for this head (added before its first run, before a force-push, or by someone without write access), with `github.token` (`pull-requests: write` on that job only), so the pull request shows it needs approving again. A label the data can't settle (an API error, a list cut short) is left on, and refused. The removal starts no new run: the status just written already refuses the label.
82
78
 
83
79
  If your `pocp.yml` was written before the `pocp-run` step existed, add it (`pocp init --github --force` rewrites the example): without it no run counts, and the status stays pending.
84
80
 
@@ -96,7 +92,7 @@ Close that by giving the reporter its own GitHub App and requiring `pocp` from t
96
92
 
97
93
  The example `pocp-status.yml` already runs in that environment and uses the app when `POCP_REPORTER_CLIENT_ID` is set; without it, it posts as GitHub Actions as before.
98
94
 
99
- Why the environment matters: secrets in an environment limited to the default branch reach only jobs running from the default branch. The reporter runs from there (`workflow_run`, `pull_request_target`). A pull request's own workflows run from its branch or merge ref, so a branch that edits a workflow can't read the key, can't post as the app, and anything it posts as GitHub Actions doesn't count. Keeping the key in a plain repository secret would let any workflow on any branch read it, and the app would protect nothing.
95
+ Why the environment matters: secrets in an environment limited to the default branch reach only jobs running from the default branch. The reporter runs from there (`workflow_run`). A pull request's own workflows run from its branch or merge ref, so a branch that edits a workflow can't read the key, can't post as the app, and anything it posts as GitHub Actions doesn't count. Keeping the key in a plain repository secret would let any workflow on any branch read it, and the app would protect nothing.
100
96
 
101
97
  What it doesn't cover: anyone who can change the repository's settings. An **admin** can loosen the environment's branch rule, the branch protection or the app, so a coding agent holding an admin token (a classic token with `repo` scope on a repository you own is one) can undo all of this. Give agents a fine-grained token without the *Administration* permission. Until the app is set up, read the diff of any same-repository pull request that touches CI files before merging.
102
98
 
@@ -265,10 +265,23 @@ loosening in the change at once, including new signing keys.
265
265
 
266
266
  Bind approvals to the reviewed code. `--accept-test-change=<sha>` and
267
267
  `--accept-policy-change=<sha>` approve only when the head being verified is that commit. The
268
- GitHub example honours the labels only on label events, passing the head at that moment, and
269
- `examples/github/pocp-labels.yml` removes both labels on every push, so a new approval is a
270
- decision about the new code. (A label added in the seconds between a push and that removal
271
- still applies to the new head; whoever adds it is looking at the pull request as it is then.)
268
+ GitHub example passes each label with the head at that moment, and verify's
269
+ `--label-freshness=github` asks GitHub's API whether the label still counts for that head: its
270
+ last application is by someone with write access, after the head's first run of the workflow and
271
+ after the pull request's latest force-push, or it carries over to a clean merge of the base
272
+ (STATUS.md, "Updating the branch keeps the approval"). A label left on from before a push is
273
+ refused, so a new approval is a decision about the new code; the trusted reporter applies the
274
+ same rule (src/label-freshness.js) and removes such a label. Anything GitHub can't settle (an
275
+ API error, a list cut short, an unreadable permission) refuses the approval. (A label added in
276
+ the seconds between a push and that head's first run is refused too: add it again.)
277
+
278
+ The token for those reads (`actions: read`, `pull-requests: read`) is set only in the verify
279
+ step's environment. verify takes it out of its own environment before it installs or runs
280
+ anything, and isolated checks get an allowlisted environment that never includes it, nor can
281
+ they read verify's process. With `isolate: false` (the cooperative runner, not a security
282
+ boundary) a check running as the job's user could read it from verify's process: it is
283
+ read-only, and lasts as long as the job. Workflows written before 1.3.1 don't pass
284
+ `label-freshness`: their labels are taken as given, as before, and the action warns on each run.
272
285
  On GitLab, a maintainer runs a pipeline with `POCP_ACCEPT_*` set to the reviewed head's sha;
273
286
  a project-wide CI/CD variable would approve every merge request.
274
287
 
@@ -9,28 +9,25 @@
9
9
  # own copy of pocp.yml, so if it edits that, the status stays yellow until a maintainer approves
10
10
  # the workflow change with pocp-approve-workflow.
11
11
  #
12
- # workflow_run: when a pocp run finishes. pull_request_target: when the head or the labels change.
12
+ # workflow_run: when a pocp run finishes. pocp.yml runs on every push, retarget and approval label,
13
+ # so that covers each change (no pull_request_target, which GitHub blocks by default in public
14
+ # repositories). A label given before the head's first run doesn't count, and the reporter
15
+ # removes it with github.token, so the maintainer sees to approve again.
13
16
  name: pocp-status
14
17
  on:
15
18
  workflow_run:
16
19
  workflows: [pocp]
17
20
  types: [completed]
18
- pull_request_target:
19
- types: [opened, synchronize, reopened, edited, labeled, unlabeled] # edited: a retarget changes the base that judges it
20
21
  permissions:
21
22
  contents: read
22
23
  actions: read
23
- pull-requests: read
24
+ pull-requests: write # only github.token's, to remove the approval labels it finds stale
24
25
  statuses: write
25
26
  jobs:
26
27
  report:
27
- # Skipped (no Actions minutes) for events that can't change the status: a label other than
28
- # pocp-approve-*, or an edit that doesn't change the base. Pushes, retargets, approval labels
29
- # and finished verify runs always report.
30
- if: github.event_name != 'pull_request_target' || ((github.event.action != 'labeled' && github.event.action != 'unlabeled') || startsWith(github.event.label.name, 'pocp-approve-')) && (github.event.action != 'edited' || github.event.changes.base)
31
28
  # One at a time per head, none cancelled: each recomputes from scratch, so the last one wins.
32
29
  concurrency:
33
- group: pocp-status-${{ github.event.workflow_run.head_sha || github.event.pull_request.head.sha }}
30
+ group: pocp-status-${{ github.event.workflow_run.head_sha }}
34
31
  cancel-in-progress: false
35
32
  # The reporter's own identity (docs/STATUS.md, "Give the reporter its own identity"): the App's
36
33
  # key lives in this environment, limited to the default branch, so only this job, running from
@@ -63,3 +60,4 @@ jobs:
63
60
  with:
64
61
  workflow: .github/workflows/pocp.yml
65
62
  github-token: ${{ steps.app.outputs.token || github.token }}
63
+ labels-token: ${{ github.token }}
@@ -1,6 +1,6 @@
1
1
  # .github/workflows/pocp.yml
2
2
  # POCP on GitHub Actions, for a project in any language. `pocp init --github` writes this file,
3
- # pocp-status.yml and pocp-labels.yml, pinned to your POCP version. docs/STATUS.md explains the
3
+ # and pocp-status.yml, pinned to your POCP version. docs/STATUS.md explains the
4
4
  # "pocp" status these produce; docs/QUICKSTART.md, how to turn it on.
5
5
  #
6
6
  # verify judges the pull request by the base branch's rules and re-runs every required check
@@ -32,6 +32,8 @@ on:
32
32
  types: [opened, synchronize, reopened, edited, labeled, unlabeled] # edited: a retarget changes the base that judges it
33
33
  permissions:
34
34
  contents: read
35
+ actions: read # label-freshness: when this head first ran
36
+ pull-requests: read # label-freshness: who applied each approval label, and when
35
37
  jobs:
36
38
  verify:
37
39
  if: vars.POCP_STATUS_LIVE != 'true' || ((github.event.action != 'labeled' && github.event.action != 'unlabeled') || startsWith(github.event.label.name, 'pocp-approve-')) && (github.event.action != 'edited' || github.event.changes.base)
@@ -45,13 +47,14 @@ jobs:
45
47
  # number of commits in a typical pull request, with margin.
46
48
  timeout-minutes: 60
47
49
  env:
48
- # Each approval is a label only maintainers apply. A label isn't tied to a commit, so it
49
- # counts only on the event that changed labels, and only for the head at that moment (the
50
- # sha is passed, so verify checks it). pocp-labels.yml removes every approval on each push.
51
- APPROVE_POLICY: ${{ (github.event.action == 'labeled' || github.event.action == 'unlabeled' || (vars.POCP_STATUS_LIVE == 'true' && github.event.action == 'synchronize')) && contains(github.event.pull_request.labels.*.name, 'pocp-approve-policy') && github.event.pull_request.head.sha || 'false' }}
52
- APPROVE_TESTS: ${{ (github.event.action == 'labeled' || github.event.action == 'unlabeled' || (vars.POCP_STATUS_LIVE == 'true' && github.event.action == 'synchronize')) && contains(github.event.pull_request.labels.*.name, 'pocp-approve-tests') && github.event.pull_request.head.sha || 'false' }}
53
- APPROVE_INCONCLUSIVE: ${{ (github.event.action == 'labeled' || github.event.action == 'unlabeled') && contains(github.event.pull_request.labels.*.name, 'pocp-approve-inconclusive') && github.event.pull_request.head.sha || 'false' }}
54
- APPROVE_COVERAGE: ${{ (github.event.action == 'labeled' || github.event.action == 'unlabeled' || (vars.POCP_STATUS_LIVE == 'true' && github.event.action == 'synchronize')) && contains(github.event.pull_request.labels.*.name, 'pocp-approve-coverage') && github.event.pull_request.head.sha || 'false' }}
50
+ # Each approval is a label only maintainers apply. A label isn't tied to a commit, so the
51
+ # head's sha is passed with it and verify (label-freshness, below) asks GitHub whether the
52
+ # label was applied by a writer after this head first ran: a label left on from before a
53
+ # push doesn't count, except through "Update branch" (a clean merge of the base).
54
+ APPROVE_POLICY: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-policy') && github.event.pull_request.head.sha || 'false' }}
55
+ APPROVE_TESTS: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-tests') && github.event.pull_request.head.sha || 'false' }}
56
+ APPROVE_INCONCLUSIVE: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-inconclusive') && github.event.pull_request.head.sha || 'false' }}
57
+ APPROVE_COVERAGE: ${{ contains(github.event.pull_request.labels.*.name, 'pocp-approve-coverage') && github.event.pull_request.head.sha || 'false' }}
55
58
  WAIT_EXIT: ${{ vars.POCP_STATUS_LIVE == 'true' && '0' || '1' }}
56
59
  steps:
57
60
  # Which pull request and base this run evaluated. A head commit can belong to several pull
@@ -74,6 +77,8 @@ jobs:
74
77
  id: verify
75
78
  continue-on-error: true # the step fails unless verified; the steps below report why
76
79
  with:
80
+ label-freshness: github # an approval's label must be given after this head first ran
81
+ github-token: ${{ github.token }} # read-only (permissions above); verify's process only
77
82
  rerun: all
78
83
  isolate: true # each check runs as a fresh unprivileged user (Linux runners)
79
84
  # install: npm ci # ▶ your install command: runs isolated, before the checks
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "proof-of-change",
3
- "version": "1.3.0",
3
+ "version": "1.3.2",
4
4
  "description": "Proof of Change Protocol: risky changes carry proof that was run, tied to the exact code it tested.",
5
5
  "type": "module",
6
6
  "bin": { "pocp": "bin/pocp.js" },
package/src/ci-github.js CHANGED
@@ -2,7 +2,7 @@ import { spawnSync } from 'node:child_process';
2
2
  import fs from 'node:fs';
3
3
  import path from 'node:path';
4
4
  import { git } from './git.js';
5
- import { GITHUB_WORKFLOWS } from './init.js';
5
+ import { GITHUB_WORKFLOWS, LEGACY_GITHUB_WORKFLOWS } from './init.js';
6
6
 
7
7
  // Branch protection from versioned config: which CI checks must pass before merging,
8
8
  // whether changes must arrive through pull requests, and whether admins are held to it.
@@ -319,7 +319,9 @@ export function setPocpWorkflows(root, plan, { enable }) {
319
319
  const repo = repoR.stdout.trim();
320
320
  const { workflows, error } = listWorkflows(repo);
321
321
  if (error) return { ok: false, repo, error };
322
- const ours = GITHUB_WORKFLOWS.map((f) => `.github/workflows/${f}`);
322
+ // A leftover pocp-labels.yml (LEGACY_GITHUB_WORKFLOWS) is switched with the rest, when GitHub has it.
323
+ const legacy = LEGACY_GITHUB_WORKFLOWS.map((f) => `.github/workflows/${f}`);
324
+ const ours = [...GITHUB_WORKFLOWS.map((f) => `.github/workflows/${f}`), ...legacy.filter((f) => workflows.some((w) => w.path === f))];
323
325
  const changed = [];
324
326
  const unchanged = [];
325
327
  const missing = [];
package/src/cli-args.js CHANGED
@@ -187,6 +187,9 @@ export const COMMANDS = {
187
187
  jobs: num('n', 'checks run at once (default: CPU cores - 1, at most 4)'),
188
188
  isolate: bool('run each check as a fresh unprivileged user (Linux, passwordless sudo); fail if that is not possible'),
189
189
  install: str('cmd', 'with --isolate: install dependencies at the head, isolated, before the checks'),
190
+ 'label-freshness': str('github', 'refuse an approval whose label was added before this head (token in POCP_GITHUB_TOKEN)'),
191
+ repo: str('owner/name', 'with --label-freshness: the repository (default: GITHUB_REPOSITORY)'),
192
+ 'pull-request': num('n', 'with --label-freshness: the pull request whose labels gave the approvals'),
190
193
  },
191
194
  },
192
195
  blame: { summary: 'packets that changed a file', usage: ['blame <file>'], engine: true, flags: {} },
package/src/cli.js CHANGED
@@ -152,6 +152,8 @@ Checking
152
152
  didn't finish (re-run it), 1 for any other failure
153
153
  --isolate: each check runs as a fresh unprivileged user (Linux, sudo); if
154
154
  that can't be set up, verify fails. --install <cmd>: dependencies too
155
+ --label-freshness=github --repo <o/r> --pull-request <n>: an approval's label
156
+ must be added after this head first ran (token in POCP_GITHUB_TOKEN)
155
157
  blame <file> packets that changed a file
156
158
 
157
159
  Misses
@@ -44,6 +44,7 @@ export async function run({ flags, json }) {
44
44
  ...(github ? [
45
45
  github.written.length ? `wrote ${github.written.join(', ')} (POCP actions pinned to ${github.ref})` : 'GitHub workflows already present',
46
46
  ...github.skipped.map((f) => `kept ${f} (exists; --force replaces it)`),
47
+ ...github.removed.map((f) => `removed ${f} (no longer needed: verify checks approval labels against the head)`),
47
48
  ...(labels?.ok ? [labels.created.length ? `created the approval labels on ${labels.repo}: ${labels.created.join(', ')}` : `approval labels already on ${labels.repo}`]
48
49
  : labels ? [`approval labels not created: ${labels.error} (pocp doctor prints the commands)`] : []),
49
50
  ...(live?.ok ? [`${LIVE_VARIABLE} set on ${live.repo}: a pull request waiting for approval shows a yellow "${STATUS_CHECK}" status, not a red failure`, ...live.notes.map((n) => ` ${n}`)]
@@ -17,8 +17,11 @@ export async function run({ flags, json }) {
17
17
  const cfgFile = path.join(process.cwd(), '.proof', 'config.json');
18
18
  const baseConfig = fs.existsSync(cfgFile) ? withDefaults(JSON.parse(fs.readFileSync(cfgFile, 'utf8'))) : null;
19
19
  const gh = githubClient({ token: process.env.GH_TOKEN || process.env.GITHUB_TOKEN });
20
- const r = await reportStatus(gh, { repository, sha, prNumber, baseConfig, workflowPath: flags.workflow || undefined, dryRun: Boolean(flags['dry-run']) });
20
+ // LABELS_TOKEN (github.token with pull-requests: write): removes the approval labels found
21
+ // stale. GH_TOKEN may be the reporter App's, which only reads pull requests.
22
+ const labelGh = process.env.LABELS_TOKEN ? githubClient({ token: process.env.LABELS_TOKEN }) : null;
23
+ const r = await reportStatus(gh, { repository, sha, prNumber, baseConfig, workflowPath: flags.workflow || undefined, dryRun: Boolean(flags['dry-run']), labelGh });
21
24
  const d = r.decision;
22
- out(json, r, d.write ? `${r.written ? 'set' : 'would set'} ${STATUS_CONTEXT} on ${d.sha.slice(0, 12)}: ${d.state} (${d.description}); ${d.reason}` : `nothing written: ${d.reason}`);
25
+ out(json, r, `${d.write ? `${r.written ? 'set' : 'would set'} ${STATUS_CONTEXT} on ${d.sha.slice(0, 12)}: ${d.state} (${d.description}); ${d.reason}` : `nothing written: ${d.reason}`}${r.removed?.length ? `\nremoved stale approval label(s): ${r.removed.join(', ')}` : ''}`);
23
26
  return EXIT.ok;
24
27
  }
@@ -12,16 +12,57 @@ import { failedOutputLines } from '../output-tail.js';
12
12
  import { git, resolveRev } from '../git.js';
13
13
  import { readCiResults } from '../ci-results.js';
14
14
  import { shortDigest } from '../hash.js';
15
+ import { UsageError } from '../cli-args.js';
16
+ import { freshnessFacts, labelFreshness } from '../label-freshness.js';
17
+ import { githubClient } from '../status-report.js';
18
+
19
+ // Which label gives each approval (examples/github/pocp.yml).
20
+ const APPROVAL_LABELS = Object.freeze({
21
+ 'accept-test-change': 'pocp-approve-tests',
22
+ 'accept-policy-change': 'pocp-approve-policy',
23
+ 'accept-inconclusive': 'pocp-approve-inconclusive',
24
+ 'accept-uncovered': 'pocp-approve-coverage',
25
+ });
26
+
27
+ /**
28
+ * --label-freshness=github: whether each approval's label counts for this head, by the rule the
29
+ * reporter uses (src/label-freshness.js), from GitHub's API. { label: verdict } for the
30
+ * approvals given. One GitHub can't settle is refused, never assumed.
31
+ */
32
+ async function labelVerdicts(engine, flags, headRev, token) {
33
+ if (flags['label-freshness'] !== 'github') throw new UsageError(`--label-freshness takes "github", not "${flags['label-freshness']}"`);
34
+ if (!token) throw new UsageError("--label-freshness=github needs a token in POCP_GITHUB_TOKEN (the action's github-token input)");
35
+ const repository = flags.repo || process.env.GITHUB_REPOSITORY;
36
+ const prNumber = Number(flags['pull-request']) || null;
37
+ if (!repository || !prNumber) throw new UsageError('--label-freshness=github needs --repo <owner/name> (or GITHUB_REPOSITORY) and --pull-request <n>');
38
+ const given = (name) => [].concat(flags[name] ?? false).some((v) => v && v !== 'false');
39
+ const labels = Object.entries(APPROVAL_LABELS).filter(([name]) => given(name)).map(([, label]) => label);
40
+ const verdicts = {};
41
+ if (!labels.length) return verdicts;
42
+ const sha = resolveRev(engine.root, headRev);
43
+ try {
44
+ const facts = await freshnessFacts(githubClient({ token, api: process.env.GITHUB_API_URL || undefined }), { repository, prNumber, sha, labels });
45
+ for (const label of labels) verdicts[label] = labelFreshness(facts, label, sha);
46
+ } catch (e) {
47
+ for (const label of labels) verdicts[label] = { ok: false, stale: false, why: `couldn't check ${label} against GitHub (${e.message}): re-run this job` };
48
+ }
49
+ return verdicts;
50
+ }
15
51
 
16
52
  export async function run({ engine, flags, isolated, json, rest }) {
17
53
  const started = Date.now();
54
+ // The token for --label-freshness: read, then taken out of this process's environment before
55
+ // anything else runs, so no check (cooperative or isolated) or install inherits it.
56
+ const ghToken = process.env.POCP_GITHUB_TOKEN || '';
57
+ delete process.env.POCP_GITHUB_TOKEN;
58
+ const headRev = rest[0] && rest[0].includes('..') ? rest[0].split(/\.\.\.?/)[1] || 'HEAD' : rest[0] || 'HEAD';
59
+ const verdicts = flags['label-freshness'] ? await labelVerdicts(engine, flags, headRev, ghToken) : null;
18
60
  // Before any of the pull request's code runs: set up and probe the isolation. If it
19
61
  // can't be set up, this throws and verify fails; it never falls back.
20
62
  const isolation = isolated ? isolated.isolation : COOPERATIVE;
21
63
  if (isolated) {
22
64
  await isolated.prepare({ root: engine.root });
23
65
  if (flags.install) {
24
- const headRev = rest[0] && rest[0].includes('..') ? rest[0].split(/\.\.\.?/)[1] || 'HEAD' : rest[0] || 'HEAD';
25
66
  const cfg = engine.config();
26
67
  const co = createRevCheckout(engine.root, resolveRev(engine.root, headRev), { links: [] });
27
68
  try {
@@ -43,9 +84,16 @@ export async function run({ engine, flags, isolated, json, rest }) {
43
84
  });
44
85
  if (ignoredCi) errWrite(`pocp: ignored ${ignoredCi} CI result(s) recorded by a different CI run\n`);
45
86
  if (ciResults.length) errWrite('pocp: trusting CI results from --ci-results: anything that ran earlier in this job could have written them (docs/THREAT-MODEL.md)\n');
46
- // An approval that names a commit covers only that head: a later push needs a new one.
47
- const headRev = rest[0] && rest[0].includes('..') ? rest[0].split(/\.\.\.?/)[1] || 'HEAD' : rest[0] || 'HEAD';
87
+ // An approval that names a commit covers only that head: a later push needs a new one. With
88
+ // --label-freshness, its label must also count for this head (labelVerdicts).
48
89
  const approval = (name) => {
90
+ const ok = approvalFor(name);
91
+ const verdict = verdicts?.[APPROVAL_LABELS[name]];
92
+ if (!ok || !verdict || verdict.ok) return ok;
93
+ errWrite(`pocp: ${APPROVAL_LABELS[name]} not applied: ${verdict.why}\n`);
94
+ return false;
95
+ };
96
+ const approvalFor = (name) => {
49
97
  const v = [].concat(flags[name] ?? false).pop();
50
98
  if (typeof v !== 'string') return Boolean(v);
51
99
  const head = resolveRev(engine.root, headRev);
@@ -132,6 +180,7 @@ export async function run({ engine, flags, isolated, json, rest }) {
132
180
  ].join('\n');
133
181
  }
134
182
  result.isolation = { mode: isolation.mode, secure: isolation.secure };
183
+ if (verdicts) result.label_freshness = Object.fromEntries(Object.entries(verdicts).map(([l, v]) => [l, v.ok ? { ok: true, how: v.how } : v]));
135
184
  // One decision for the library and the CLI, from typed findings (src/decision.js).
136
185
  if (!result.ok) result.approval = decideVerify(result);
137
186
  if (steps.length) result.next_steps = steps;