proof-of-change 1.3.1 → 1.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -8
- package/SPEC.md +2 -1
- package/docs/HOW-IT-WORKS.md +1 -1
- package/docs/QUICKSTART.md +8 -3
- package/docs/RESULTS.md +1 -1
- package/package.json +1 -1
- package/src/init.js +1 -1
- package/src/mutants.js +4 -3
- package/src/runner.js +27 -0
package/README.md
CHANGED
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
|
|
3
3
|
POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code and holds risky test edits for a maintainer. It works with any test command and any language, and needs only Node 20 or later, git and a POSIX shell.
|
|
4
4
|
|
|
5
|
-
It doesn't prove your code is correct. It has one
|
|
5
|
+
It doesn't prove your code is correct. It has one evaluation so far ([below](#evaluation)).
|
|
6
6
|
|
|
7
|
-

|
|
8
8
|
|
|
9
|
-
An agent edits a failing test until CI turns green. POCP runs the old test against the new code and
|
|
9
|
+
An agent edits a failing test until CI turns green. POCP runs the old test against the new code and holds the change for a person. Rather than wait for that approval, the agent puts the test back and fixes the code, and the pull request verifies. Run it from a clone with `node examples/demo/demo.mjs`.
|
|
10
10
|
|
|
11
11
|
## Install
|
|
12
12
|
|
|
@@ -39,22 +39,29 @@ npx pocp init --hooks --github
|
|
|
39
39
|
**On another CI** (GitLab, Jenkins, plain shell). In CI: keep your existing test step, and add:
|
|
40
40
|
|
|
41
41
|
```bash
|
|
42
|
-
|
|
42
|
+
npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
|
|
43
|
+
node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
|
|
43
44
|
```
|
|
44
45
|
|
|
46
|
+
Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does. Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc` and `node_modules`, so npx would run whatever `pocp` the pull request ships.
|
|
47
|
+
|
|
45
48
|
[docs/QUICKSTART.md](docs/QUICKSTART.md) walks through all of this on a non-JavaScript project.
|
|
46
49
|
|
|
47
50
|
## Evaluation
|
|
48
51
|
|
|
49
|
-
A
|
|
52
|
+
A benchmark of POCP 1.2.3 on p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, 68 of the cases held out until the scoring script was locked ([full report](eval/campaign-123/README.md)). The protocol was frozen by hash before the runs. The freeze record isn't public, so that order rests on my account. The raw records and the scripts that score them are public in [pocp-eval](https://github.com/harriswekselblatt/pocp-eval).
|
|
53
|
+
|
|
54
|
+
- **Harmful changes (48):** each breaks the code, then hides it with one of four scripted edits to the test that caught it: delete the assertion, empty the assertion helper, overwrite the expected-value fixture, or make the check script exit 0. Plain CI accepted all 48. POCP accepted none: in both modes it held each one for a person, with the old tests failing on the new code. That shows the old-tests check works on each part of a suite in three languages. It doesn't measure cheats that leave the old tests passing (no harmful case reached deep mode's planted bugs), or deliberate deception across pull requests ([threat model](docs/THREAT-MODEL.md)).
|
|
55
|
+
- **Honest changes (108):** deep mode accepts 78–82% per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%, about what a rule holding every test-file edit would. Reducing that friction is the next piece of work.
|
|
56
|
+
|
|
57
|
+
## Why not just require review of test files?
|
|
50
58
|
|
|
51
|
-
|
|
52
|
-
- **Honest changes:** deep mode accepts 78–82% of 108 legitimate changes per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%. Reducing that friction is the next piece of work.
|
|
59
|
+
Fast mode is close to that rule, with two differences. The reviewer gets evidence, not just a diff: POCP runs the old tests on the new code and names any that now fail. And what must pass is set by the base branch, so a pull request can't get green by changing its rules, its check command, its test script or its runner config. Deep mode goes further: its planted bugs let most honest test edits through with no person.
|
|
53
60
|
|
|
54
61
|
## Documentation
|
|
55
62
|
|
|
56
63
|
- [How it works](docs/HOW-IT-WORKS.md): what it catches, fast and deep modes, and what it doesn't do.
|
|
57
|
-
- [Evaluation of 1.2.3](eval/campaign-123/README.md): the
|
|
64
|
+
- [Evaluation of 1.2.3](eval/campaign-123/README.md): the benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
|
|
58
65
|
- [Reference](docs/REFERENCE.md): rules, checks, CI options, approvals, signed proofs.
|
|
59
66
|
- [Threat model](docs/THREAT-MODEL.md): what each record does and doesn't prove.
|
|
60
67
|
- [SPEC.md](SPEC.md): the protocol. [Reviewing POCP](docs/REVIEWING.md): a guide for reviewers.
|
package/SPEC.md
CHANGED
|
@@ -320,7 +320,8 @@ treats these changes as guarded:
|
|
|
320
320
|
- test files added, modified, deleted or renamed (an added test file runs with the old ones
|
|
321
321
|
and can change their outcome). The default
|
|
322
322
|
test patterns cover the common layouts of JavaScript/TypeScript (`test/`, `spec/`,
|
|
323
|
-
`__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`,
|
|
323
|
+
`__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, the standalone names `node --test` runs (`test.js`,
|
|
324
|
+
`test-*.js`, `*-test.js`), snapshots, `__mocks__/`), Python
|
|
324
325
|
(`tests/`, `test_*.py`, `*_test.py`), Ruby (`spec/`, `*_spec.rb`, `*_test.rb`), Go
|
|
325
326
|
(`*_test.go`, `testdata/`), Rust (`tests/`), Java/Kotlin/Scala (`src/test/`,
|
|
326
327
|
`src/androidTest/` and the other `src/*Test/` source sets, `*Test.java`, `*IT.java`), C#
|
package/docs/HOW-IT-WORKS.md
CHANGED
|
@@ -59,7 +59,7 @@ Within the limits in [docs/THREAT-MODEL.md](THREAT-MODEL.md):
|
|
|
59
59
|
|
|
60
60
|
- It doesn't judge whether code is honest. Code written to pass the specific test inputs passes every mechanical check.
|
|
61
61
|
- It doesn't enforce merges. `verify` fails a check; whether a failing check blocks the merge is your Git host's setting (on GitHub, branch protection).
|
|
62
|
-
- It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)).
|
|
62
|
+
- It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). Giving the reporter its own GitHub App closes this ([docs/STATUS.md](STATUS.md#give-the-reporter-its-own-identity)).
|
|
63
63
|
- It doesn't measure new tests. The test check compares a change with tests that already existed, so a weak test added in the same pull request isn't measured.
|
|
64
64
|
- It isn't hermetic. Checks can read the network, environment variables and installed packages, and the re-runs execute the pull request's code in your CI job.
|
|
65
65
|
- It adds CI time, and on a private repository GitHub bills it as Actions minutes. The status and label workflows skip events that can't change the result (other labels, title edits), but each push and approval still costs about a minute. `verify` re-runs your required checks, so each pull request costs about one extra test run, in parallel with your other jobs. On POCP's own pull requests in fast mode, `verify` took 3 to 4 minutes next to test jobs of 3 to 4 minutes, and 10 to 11 minutes when several commits edited tests, because the old tests ran as the whole suite. A rule's `changed_test_command` runs only the edited test files instead ([docs/REFERENCE.md](REFERENCE.md#faster-test-edits-changed_test_command)). It hasn't been measured on other projects yet.
|
package/docs/QUICKSTART.md
CHANGED
|
@@ -132,8 +132,8 @@ then approve it for that exact head:
|
|
|
132
132
|
pocp verify --require-coverage --rerun main..HEAD --accept-test-change $(git rev-parse HEAD)
|
|
133
133
|
```
|
|
134
134
|
|
|
135
|
-
(or `--accept-policy-change`).
|
|
136
|
-
|
|
135
|
+
(or `--accept-policy-change`). verify then prints `verified`. It doesn't say an approval was
|
|
136
|
+
used, so the record of what you approved is the command you ran.
|
|
137
137
|
|
|
138
138
|
**A local approval is a flag, not a person.** Anything that can run commands on your machine,
|
|
139
139
|
an agent included, can pass `--accept-*` itself, just as it can run `git commit --no-verify`. It
|
|
@@ -200,9 +200,14 @@ Verify every pull request. verify re-runs each required check itself, on the exa
|
|
|
200
200
|
by the base branch's rules; your existing test step stays as it is:
|
|
201
201
|
|
|
202
202
|
```bash
|
|
203
|
-
|
|
203
|
+
npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
|
|
204
|
+
node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
|
|
204
205
|
```
|
|
205
206
|
|
|
207
|
+
Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does.
|
|
208
|
+
Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc`
|
|
209
|
+
and `node_modules`, so npx would run whatever `pocp` the pull request ships.
|
|
210
|
+
|
|
206
211
|
A faster mode runs your tests once, through `pocp ci-run`, and passes `--ci-results` to verify.
|
|
207
212
|
It trusts everything that ran earlier in the job: a new test file can write a passing result
|
|
208
213
|
for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
|
package/docs/RESULTS.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Results so far
|
|
2
2
|
|
|
3
|
-
**The main result is the
|
|
3
|
+
**The main result is the evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
|
|
4
4
|
|
|
5
5
|
These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which was first benchmarked in the 1.2.3 evaluation.
|
|
6
6
|
|
package/package.json
CHANGED
package/src/init.js
CHANGED
|
@@ -426,7 +426,7 @@ const PKG_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'
|
|
|
426
426
|
// The newest POCP release that has a tag on GitHub. A development or pre-release build (a
|
|
427
427
|
// version like 1.1.0-dev.0, as on main between releases) has no tag of its own, so the
|
|
428
428
|
// workflows it writes pin this instead of a ref that doesn't exist. Update it with each release.
|
|
429
|
-
export const RELEASED_ACTION_REF = 'v1.3.
|
|
429
|
+
export const RELEASED_ACTION_REF = 'v1.3.2';
|
|
430
430
|
|
|
431
431
|
/** The release tag the workflows pin by default: "v<version>" for a release, else RELEASED_ACTION_REF. */
|
|
432
432
|
export function releaseRef() {
|
package/src/mutants.js
CHANGED
|
@@ -18,10 +18,11 @@ export const DEFAULT_EVAL_EXCLUDE = Object.freeze([
|
|
|
18
18
|
'**/test-utils/**', '**/test_utils/**', '**/testutils/**', '**/test-helpers/**', '**/test_helpers/**', '**/test-support/**', '**/test_support/**',
|
|
19
19
|
// JavaScript / TypeScript (and anything else named this way): a.test.js, a.spec.ts, a_spec.js, a_test.js.
|
|
20
20
|
'**/*.test.*', '**/*.spec.*', '**/*_spec.*', '**/*_test.*', '**/*.snap',
|
|
21
|
-
// Standalone Node/AVA tests, also inside packages
|
|
22
|
-
// unrecognised test cannot both escape restoration and
|
|
21
|
+
// Standalone Node/AVA tests, also inside packages: the names `node --test` finds by default.
|
|
22
|
+
// Share these with the guard so an unrecognised test cannot both escape restoration and
|
|
23
|
+
// enter the mutation sample.
|
|
23
24
|
...['js', 'mjs', 'cjs', 'ts', 'mts', 'cts', 'jsx', 'tsx'].flatMap((ext) => [
|
|
24
|
-
`**/test.${ext}`, `**/test-*.${ext}`,
|
|
25
|
+
`**/test.${ext}`, `**/test-*.${ext}`, `**/*-test.${ext}`,
|
|
25
26
|
]),
|
|
26
27
|
// Python: test_a.py, a_test.py (above), Django's tests.py.
|
|
27
28
|
'**/test_*.py', '**/tests.py',
|
package/src/runner.js
CHANGED
|
@@ -32,11 +32,38 @@ export function buildEnv({ mode = 'inherit', allow = [], extra = {}, keepCiFiles
|
|
|
32
32
|
// repository, index or config. Git sets GIT_INDEX_FILE (and sometimes GIT_DIR) for the commit
|
|
33
33
|
// hooks, so a check run from a hook that runs git in a temporary repository would otherwise
|
|
34
34
|
// write into the committer's index.
|
|
35
|
+
dropNpmLaunch(env);
|
|
35
36
|
const out = { ...env, ...extra };
|
|
36
37
|
for (const k of Object.keys(out)) if (isGitRedirectVar(k)) delete out[k];
|
|
37
38
|
return out;
|
|
38
39
|
}
|
|
39
40
|
|
|
41
|
+
// Started by npm or npx (`npx pocp verify`), pocp inherits what npm sets up for the program it
|
|
42
|
+
// starts, and in a pull request's checkout that is the pull request's: the config npm read, the
|
|
43
|
+
// project's .npmrc included, as npm_config_* and its node-options as NODE_OPTIONS; and, in front
|
|
44
|
+
// of PATH, the checkout's node_modules/.bin (and each one above it). Passed on, a node-options
|
|
45
|
+
// or a committed node_modules/.bin/npm would reach every check, the old tests the guard puts
|
|
46
|
+
// back included. Checks get what they would if pocp were started with node directly: a check
|
|
47
|
+
// that runs npm reads its own copy's .npmrc and puts its own node_modules/.bin on PATH. A
|
|
48
|
+
// NODE_OPTIONS or PATH the caller set stays, except under pnpm, which sets NODE_OPTIONS from
|
|
49
|
+
// .npmrc without saying so (and NODE_PATH): there neither can be told apart, so both go.
|
|
50
|
+
function dropNpmLaunch(env, launcher = process.env) {
|
|
51
|
+
if (launcher.npm_command === undefined) return;
|
|
52
|
+
const fromConfig = launcher.npm_config_node_options;
|
|
53
|
+
if (/^pnpm\//.test(launcher.npm_config_user_agent || '')) {
|
|
54
|
+
delete env.NODE_OPTIONS;
|
|
55
|
+
delete env.NODE_PATH;
|
|
56
|
+
} else if (fromConfig !== undefined && env.NODE_OPTIONS === fromConfig) delete env.NODE_OPTIONS;
|
|
57
|
+
for (const k of Object.keys(env)) if (k.startsWith('npm_config_') || k.startsWith('pnpm_config_')) delete env[k];
|
|
58
|
+
if (env.PATH) {
|
|
59
|
+
const dirs = env.PATH.split(':');
|
|
60
|
+
const npmAdded = (d) => /\/node_modules\/\.bin\/?$/.test(d) || /\/node-gyp-bin\/?$/.test(d);
|
|
61
|
+
let i = 0;
|
|
62
|
+
while (i < dirs.length && npmAdded(dirs[i])) i++;
|
|
63
|
+
env.PATH = dirs.slice(i).join(':');
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
|
|
40
67
|
// Environment variables that point git at another repository, index, object store or config
|
|
41
68
|
// (the same set src/git.js keeps out of its own calls in isolation), plus every GIT_CONFIG*.
|
|
42
69
|
export const GIT_REDIRECT_VARS = Object.freeze([
|