proof-of-change 1.3.1 → 1.3.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,11 +2,11 @@
2
2
 
3
3
  POCP checks that the tests a pull request was supposed to pass actually ran, on the exact code being merged, under rules the pull request can't change. When a pull request edits existing tests, it runs the old tests on the new code and holds risky test edits for a maintainer. It works with any test command and any language, and needs only Node 20 or later, git and a POSIX shell.
4
4
 
5
- It doesn't prove your code is correct. It has one preregistered evaluation so far ([below](#evaluation)).
5
+ It doesn't prove your code is correct. It has one evaluation so far ([below](#evaluation)).
6
6
 
7
- ![Terminal demo: an agent breaks add() and edits the test so npm test passes; pocp verify reports that the old test fails on the new code and waits for a maintainer](docs/demo.svg)
7
+ ![Terminal demo: an agent breaks add() and edits the test so npm test passes; pocp verify reports that the old test fails on the new code and holds the change for a maintainer; the agent puts the test back, fixes add(), and verify passes](docs/demo.svg)
8
8
 
9
- An agent edits a failing test until CI turns green. POCP runs the old test against the new code and routes the change to a person. Run it from a clone with `node examples/demo/demo.mjs`.
9
+ An agent edits a failing test until CI turns green. POCP runs the old test against the new code and holds the change for a person. Rather than wait for that approval, the agent puts the test back and fixes the code, and the pull request verifies. Run it from a clone with `node examples/demo/demo.mjs`.
10
10
 
11
11
  ## Install
12
12
 
@@ -39,22 +39,29 @@ npx pocp init --hooks --github
39
39
  **On another CI** (GitLab, Jenkins, plain shell). In CI: keep your existing test step, and add:
40
40
 
41
41
  ```bash
42
- npx pocp verify origin/main..HEAD --require-coverage --rerun
42
+ npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
43
+ node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
43
44
  ```
44
45
 
46
+ Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does. Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc` and `node_modules`, so npx would run whatever `pocp` the pull request ships.
47
+
45
48
  [docs/QUICKSTART.md](docs/QUICKSTART.md) walks through all of this on a non-JavaScript project.
46
49
 
47
50
  ## Evaluation
48
51
 
49
- A preregistered benchmark of POCP 1.2.3 across p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, with a sealed holdout ([full report](eval/campaign-123/README.md)).
52
+ A benchmark of POCP 1.2.3 on p-map (JS), boltons (Python) and gjson (Go): 180 cases and 1,260 runs, 68 of the cases held out until the scoring script was locked ([full report](eval/campaign-123/README.md)). The protocol was frozen by hash before the runs. The freeze record isn't public, so that order rests on my account. The raw records and the scripts that score them are public in [pocp-eval](https://github.com/harriswekselblatt/pocp-eval).
53
+
54
+ - **Harmful changes (48):** each breaks the code, then hides it with one of four scripted edits to the test that caught it: delete the assertion, empty the assertion helper, overwrite the expected-value fixture, or make the check script exit 0. Plain CI accepted all 48. POCP accepted none: in both modes it held each one for a person, with the old tests failing on the new code. That shows the old-tests check works on each part of a suite in three languages. It doesn't measure cheats that leave the old tests passing (no harmful case reached deep mode's planted bugs), or deliberate deception across pull requests ([threat model](docs/THREAT-MODEL.md)).
55
+ - **Honest changes (108):** deep mode accepts 78–82% per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%, about what a rule holding every test-file edit would. Reducing that friction is the next piece of work.
56
+
57
+ ## Why not just require review of test files?
50
58
 
51
- - **Harmful changes:** plain CI accepted all 48 designed test-cheats. POCP accepted none. In both modes it held every one for a person's review, each with a completed check showing the tested behavior changed. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([threat model](docs/THREAT-MODEL.md)).
52
- - **Honest changes:** deep mode accepts 78–82% of 108 legitimate changes per seed, and 86–88% without the 24 paired controls built to resemble cheating. It sends the rest to review. Fast mode reviews every test edit by design, so it accepts 27%. Reducing that friction is the next piece of work.
59
+ Fast mode is close to that rule, with two differences. The reviewer gets evidence, not just a diff: POCP runs the old tests on the new code and names any that now fail. And what must pass is set by the base branch, so a pull request can't get green by changing its rules, its check command, its test script or its runner config. Deep mode goes further: its planted bugs let most honest test edits through with no person.
53
60
 
54
61
  ## Documentation
55
62
 
56
63
  - [How it works](docs/HOW-IT-WORKS.md): what it catches, fast and deep modes, and what it doesn't do.
57
- - [Evaluation of 1.2.3](eval/campaign-123/README.md): the preregistered benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
64
+ - [Evaluation of 1.2.3](eval/campaign-123/README.md): the benchmark. [Earlier results](docs/RESULTS.md): scripted suites and pilots.
58
65
  - [Reference](docs/REFERENCE.md): rules, checks, CI options, approvals, signed proofs.
59
66
  - [Threat model](docs/THREAT-MODEL.md): what each record does and doesn't prove.
60
67
  - [SPEC.md](SPEC.md): the protocol. [Reviewing POCP](docs/REVIEWING.md): a guide for reviewers.
package/SPEC.md CHANGED
@@ -320,7 +320,8 @@ treats these changes as guarded:
320
320
  - test files added, modified, deleted or renamed (an added test file runs with the old ones
321
321
  and can change their outcome). The default
322
322
  test patterns cover the common layouts of JavaScript/TypeScript (`test/`, `spec/`,
323
- `__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, snapshots, `__mocks__/`), Python
323
+ `__tests__/`, `*.test.*`, `*.spec.*`, `*_spec.*`, the standalone names `node --test` runs (`test.js`,
324
+ `test-*.js`, `*-test.js`), snapshots, `__mocks__/`), Python
324
325
  (`tests/`, `test_*.py`, `*_test.py`), Ruby (`spec/`, `*_spec.rb`, `*_test.rb`), Go
325
326
  (`*_test.go`, `testdata/`), Rust (`tests/`), Java/Kotlin/Scala (`src/test/`,
326
327
  `src/androidTest/` and the other `src/*Test/` source sets, `*Test.java`, `*IT.java`), C#
@@ -59,7 +59,7 @@ Within the limits in [docs/THREAT-MODEL.md](THREAT-MODEL.md):
59
59
 
60
60
  - It doesn't judge whether code is honest. Code written to pass the specific test inputs passes every mechanical check.
61
61
  - It doesn't enforce merges. `verify` fails a check; whether a failing check blocks the merge is your Git host's setting (on GitHub, branch protection).
62
- - It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). A dedicated GitHub App for the reporter, which would close this, is planned.
62
+ - It can't fully stop a pull request from faking its own check. A pull request runs its own copy of the workflow. With the `pocp` status required, a pull request that edits CI files stays yellow until a maintainer approves the edit, and pull requests from forks can't post a status at all. But a branch pushed to your repository by someone with write access, including an agent using your token, can edit its workflow to post its own `pocp` status. Read the diff of any such pull request that touches `.github/` ([docs/STATUS.md](STATUS.md#a-limit-of-the-pocp-status)). Giving the reporter its own GitHub App closes this ([docs/STATUS.md](STATUS.md#give-the-reporter-its-own-identity)).
63
63
  - It doesn't measure new tests. The test check compares a change with tests that already existed, so a weak test added in the same pull request isn't measured.
64
64
  - It isn't hermetic. Checks can read the network, environment variables and installed packages, and the re-runs execute the pull request's code in your CI job.
65
65
  - It adds CI time, and on a private repository GitHub bills it as Actions minutes. The status and label workflows skip events that can't change the result (other labels, title edits), but each push and approval still costs about a minute. `verify` re-runs your required checks, so each pull request costs about one extra test run, in parallel with your other jobs. On POCP's own pull requests in fast mode, `verify` took 3 to 4 minutes next to test jobs of 3 to 4 minutes, and 10 to 11 minutes when several commits edited tests, because the old tests ran as the whole suite. A rule's `changed_test_command` runs only the edited test files instead ([docs/REFERENCE.md](REFERENCE.md#faster-test-edits-changed_test_command)). It hasn't been measured on other projects yet.
@@ -132,8 +132,8 @@ then approve it for that exact head:
132
132
  pocp verify --require-coverage --rerun main..HEAD --accept-test-change $(git rev-parse HEAD)
133
133
  ```
134
134
 
135
- (or `--accept-policy-change`). The output still lists the test guard's re-run as failed; the
136
- approval is what lets it through, and verify then prints `verified`.
135
+ (or `--accept-policy-change`). verify then prints `verified`. It doesn't say an approval was
136
+ used, so the record of what you approved is the command you ran.
137
137
 
138
138
  **A local approval is a flag, not a person.** Anything that can run commands on your machine,
139
139
  an agent included, can pass `--accept-*` itself, just as it can run `git commit --no-verify`. It
@@ -200,9 +200,14 @@ Verify every pull request. verify re-runs each required check itself, on the exa
200
200
  by the base branch's rules; your existing test step stays as it is:
201
201
 
202
202
  ```bash
203
- pocp verify origin/main..HEAD --require-coverage --rerun
203
+ npm install --prefix "$HOME/.pocp" --no-save --ignore-scripts proof-of-change@1.3.2
204
+ node "$HOME/.pocp/node_modules/proof-of-change/bin/pocp.js" verify origin/main..HEAD --require-coverage --rerun
204
205
  ```
205
206
 
207
+ Install the verifier outside the checkout and run it with `node`, as the GitHub workflow does.
208
+ Not `npx pocp` from inside it: the pull request controls the checkout's `package.json`, `.npmrc`
209
+ and `node_modules`, so npx would run whatever `pocp` the pull request ships.
210
+
206
211
  A faster mode runs your tests once, through `pocp ci-run`, and passes `--ci-results` to verify.
207
212
  It trusts everything that ran earlier in the job: a new test file can write a passing result
208
213
  for any check. See [the threat model](THREAT-MODEL.md) before choosing it.
package/docs/RESULTS.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Results so far
2
2
 
3
- **The main result is the preregistered evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
3
+ **The main result is the evaluation of POCP 1.2.3: [eval/campaign-123](../eval/campaign-123/README.md).** POCP accepted none of 48 harmful changes and held each for review. This was measured against corner-cutting changes; deliberate deception across pull requests is out of scope ([THREAT-MODEL.md](THREAT-MODEL.md), "Out of scope"; the tamper benchmark's `pocp-deep` target shows both sides). Deep mode accepted 78–82% of honest changes per seed; fast mode accepted 27%. The results below came earlier. None of them is a counted evaluation. They followed the earlier plan ([docs/EVAL-PLAN.md](EVAL-PLAN.md)), whose pre-registration ([eval/PREREG.md](../eval/PREREG.md)) is unfinished.
4
4
 
5
5
  These results were all measured in deep mode (`guard.mutants: 10`), before fast mode existed. They don't describe fast mode, which was first benchmarked in the 1.2.3 evaluation.
6
6
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "proof-of-change",
3
- "version": "1.3.1",
3
+ "version": "1.3.2",
4
4
  "description": "Proof of Change Protocol: risky changes carry proof that was run, tied to the exact code it tested.",
5
5
  "type": "module",
6
6
  "bin": { "pocp": "bin/pocp.js" },
package/src/init.js CHANGED
@@ -426,7 +426,7 @@ const PKG_ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'
426
426
  // The newest POCP release that has a tag on GitHub. A development or pre-release build (a
427
427
  // version like 1.1.0-dev.0, as on main between releases) has no tag of its own, so the
428
428
  // workflows it writes pin this instead of a ref that doesn't exist. Update it with each release.
429
- export const RELEASED_ACTION_REF = 'v1.3.1';
429
+ export const RELEASED_ACTION_REF = 'v1.3.2';
430
430
 
431
431
  /** The release tag the workflows pin by default: "v<version>" for a release, else RELEASED_ACTION_REF. */
432
432
  export function releaseRef() {
package/src/mutants.js CHANGED
@@ -18,10 +18,11 @@ export const DEFAULT_EVAL_EXCLUDE = Object.freeze([
18
18
  '**/test-utils/**', '**/test_utils/**', '**/testutils/**', '**/test-helpers/**', '**/test_helpers/**', '**/test-support/**', '**/test_support/**',
19
19
  // JavaScript / TypeScript (and anything else named this way): a.test.js, a.spec.ts, a_spec.js, a_test.js.
20
20
  '**/*.test.*', '**/*.spec.*', '**/*_spec.*', '**/*_test.*', '**/*.snap',
21
- // Standalone Node/AVA tests, also inside packages. Share these with the guard so an
22
- // unrecognised test cannot both escape restoration and enter the mutation sample.
21
+ // Standalone Node/AVA tests, also inside packages: the names `node --test` finds by default.
22
+ // Share these with the guard so an unrecognised test cannot both escape restoration and
23
+ // enter the mutation sample.
23
24
  ...['js', 'mjs', 'cjs', 'ts', 'mts', 'cts', 'jsx', 'tsx'].flatMap((ext) => [
24
- `**/test.${ext}`, `**/test-*.${ext}`,
25
+ `**/test.${ext}`, `**/test-*.${ext}`, `**/*-test.${ext}`,
25
26
  ]),
26
27
  // Python: test_a.py, a_test.py (above), Django's tests.py.
27
28
  '**/test_*.py', '**/tests.py',
package/src/runner.js CHANGED
@@ -32,11 +32,38 @@ export function buildEnv({ mode = 'inherit', allow = [], extra = {}, keepCiFiles
32
32
  // repository, index or config. Git sets GIT_INDEX_FILE (and sometimes GIT_DIR) for the commit
33
33
  // hooks, so a check run from a hook that runs git in a temporary repository would otherwise
34
34
  // write into the committer's index.
35
+ dropNpmLaunch(env);
35
36
  const out = { ...env, ...extra };
36
37
  for (const k of Object.keys(out)) if (isGitRedirectVar(k)) delete out[k];
37
38
  return out;
38
39
  }
39
40
 
41
+ // Started by npm or npx (`npx pocp verify`), pocp inherits what npm sets up for the program it
42
+ // starts, and in a pull request's checkout that is the pull request's: the config npm read, the
43
+ // project's .npmrc included, as npm_config_* and its node-options as NODE_OPTIONS; and, in front
44
+ // of PATH, the checkout's node_modules/.bin (and each one above it). Passed on, a node-options
45
+ // or a committed node_modules/.bin/npm would reach every check, the old tests the guard puts
46
+ // back included. Checks get what they would if pocp were started with node directly: a check
47
+ // that runs npm reads its own copy's .npmrc and puts its own node_modules/.bin on PATH. A
48
+ // NODE_OPTIONS or PATH the caller set stays, except under pnpm, which sets NODE_OPTIONS from
49
+ // .npmrc without saying so (and NODE_PATH): there neither can be told apart, so both go.
50
+ function dropNpmLaunch(env, launcher = process.env) {
51
+ if (launcher.npm_command === undefined) return;
52
+ const fromConfig = launcher.npm_config_node_options;
53
+ if (/^pnpm\//.test(launcher.npm_config_user_agent || '')) {
54
+ delete env.NODE_OPTIONS;
55
+ delete env.NODE_PATH;
56
+ } else if (fromConfig !== undefined && env.NODE_OPTIONS === fromConfig) delete env.NODE_OPTIONS;
57
+ for (const k of Object.keys(env)) if (k.startsWith('npm_config_') || k.startsWith('pnpm_config_')) delete env[k];
58
+ if (env.PATH) {
59
+ const dirs = env.PATH.split(':');
60
+ const npmAdded = (d) => /\/node_modules\/\.bin\/?$/.test(d) || /\/node-gyp-bin\/?$/.test(d);
61
+ let i = 0;
62
+ while (i < dirs.length && npmAdded(dirs[i])) i++;
63
+ env.PATH = dirs.slice(i).join(':');
64
+ }
65
+ }
66
+
40
67
  // Environment variables that point git at another repository, index, object store or config
41
68
  // (the same set src/git.js keeps out of its own calls in isolation), plus every GIT_CONFIG*.
42
69
  export const GIT_REDIRECT_VARS = Object.freeze([