promptwheel 0.8.0 → 0.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/INTEGRATE.md CHANGED
@@ -50,7 +50,7 @@ importing this package.
50
50
  "gamed": true,
51
51
  "sourceOnly": null,
52
52
  "retained": 0.0,
53
- "gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
53
+ "gamingReason": "the \"win\" changed zero production-source files, only test/config/grader/golden"
54
54
  }
55
55
  ]
56
56
  }
@@ -68,11 +68,18 @@ metric's gain that survived the source-only re-run (Section 3). `confidence`
68
68
  (`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` are informational fields this
69
69
  implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
70
70
  truncated SHA-256 fingerprint of the gate definition (metrics, guards, `evaluatorPaths`,
71
- `gamingThreshold`, `mandates`) at base and head, sets `gate.changed: true` when they differ,
72
- and lists any guard the change dropped to hide a regression; a non-empty `gate.weakened`
73
- forces `gamed`, so an agent cannot pass by disabling the guard it is judged by.
74
-
75
- Self-check your implementation against this schema with `conformsToContract` (see Section 4 below).
71
+ `gamingThreshold`, `mandates`, `setup`, `env`, linked dirs; each metric with its `cmd` and how it
72
+ is read) at base and head, sets `gate.changed: true` when they differ, and lists in
73
+ `gate.weakened` any guard the change removed, disabled, or measured another way
74
+ (`change: guard-edited`, such as `cmd: "true"`; `change: measurement-edited`, a new `setup`, `env`
75
+ or `linkDirs`) to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
76
+ cannot pass by disabling the guard it is judged by. A change that adds the first config is
77
+ compared with the zero-config gate (the detected test command). `allowed` (optional) is `{ reason, cleared }` when a declared
78
+ reason turned a would-be `gamed`, `fail` or `inconclusive` into `pass`: `cleared` names each metric,
79
+ `gate`, or `mandate:<name>` it cleared, and each cleared metric carries `allowed: true` and keeps its
80
+ `gamingReason`, so a consumer can say what was allowed and why it would have failed.
81
+
82
+ Self-check your implementation against this schema with `conformsToContract` (see part 4 below).
76
83
 
77
84
  ---
78
85
 
@@ -99,8 +106,8 @@ measurement runs in a throwaway git worktree.
99
106
  ## 3. Import the library (in process)
100
107
 
101
108
  Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
102
- process and returns the Section 2 object. It never prints and never calls `process.exit`; it
103
- throws a plain `Error` on a precondition failure. Types ship in the package
109
+ process and returns the Section 2 object. It never prints and never calls `process.exit`; on a
110
+ precondition failure it throws an `Error` named `ConfigError` with `exitCode` 64. Types ship in the package
104
111
  (`bin/promptwheel.d.ts`, hand-written, no build step): `import type { Report, CheckOptions, Config } from 'promptwheel'`
105
112
  works in TypeScript, and editors complete the options and the report in plain JavaScript.
106
113
 
package/README.md CHANGED
@@ -23,8 +23,10 @@ printf "import {test} from 'node:test'; import assert from 'node:assert'; import
23
23
  git commit -qam 'make the suite green'
24
24
 
25
25
  npx -y promptwheel@latest run --base HEAD~1 --head HEAD
26
- # ▲ tests_pass 0 → 1 🚩 GAMED, the win changed zero production-source files
27
- # VERDICT: GAMED (exit 2)
26
+ # prints (shortened to the lines that matter; exit code 2):
27
+ # ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
28
+ # 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
29
+ # VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
28
30
  ```
29
31
 
30
32
  What happened: PromptWheel measured `tests_pass` at both commits in throwaway git worktrees (0 then 1, a win), then split the diff into production source versus `{test, config, grader, golden}` files, re-applied **only the source slice** on a clean worktree at the base, and re-ran the metric. The win did not survive, so it is `GAMED`. No LLM, no network: a diff partition plus a re-run, reproducible in seconds with a human-readable reason.
@@ -67,7 +69,7 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
67
69
  config, or grader, not the source, so revert and earn it in the source. Exit 1 means a guarded
68
70
  metric regressed, revert that too. Never weaken, skip, or delete a test to go green.
69
71
 
70
- **Two callers, by design.** Your *agent* consumes the verdict as a per-turn signal (`run --working --json`). The *harness* (a Stop-hook, or CI) runs the same check as an audit the agent cannot self-clear, because the check re-proves from the source alone and never asks the code to grade itself. See [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) for the three places to enforce it.
72
+ **Two callers, by design.** Your *agent* consumes the verdict as a per-turn signal (`run --working --json`). The *harness* (a Stop-hook, or CI) runs the same check as an audit the agent cannot self-clear, because the check re-proves from the source alone and never asks the code to grade itself. See [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) for the two places to enforce it.
71
73
 
72
74
  ## Config, `promptwheel.config.json`
73
75
 
@@ -88,18 +90,20 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
88
90
  - **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
89
91
  - **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
90
92
  - **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
91
- - **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
92
- - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
93
+ - **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires. Set on a metric that had the re-run on, it takes effect from the next change: the change that sets it is still re-proven.
94
+ - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`. A change that lowers it is still judged by the value before it.
93
95
  - **evaluatorPaths**, globs for files that *define* your gate (a threshold file, a budget, a CI limit). Editing one in the same diff that improves a metric is `GAMED` by declaration, deterministically. Opt-in, because only you know which files are the goalposts.
94
96
 
95
- - **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. For counts where a rise is sometimes the job (the `supply-chain` preset: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
97
+ - **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. `"flag"` takes the reason from `--allow` (or `check({ allow })`) only, never a trailer, because an agent can write a trailer. For counts where a rise is sometimes the job (the `supply-chain` preset uses `"flag"`: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
96
98
 
97
99
  Dependency and environment keys (config-level; `init` writes stack defaults):
98
100
 
99
- - **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python and `[]` for Rust (`cargo test` writes to `target/`, so it is built fresh in the worktree rather than shared across refs). `"linkNodeModules": false` still works as shorthand for "link nothing".
101
+ - **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. With nothing set, a Node repo links its root `node_modules` and the `node_modules` of every workspace (any folder with a tracked `package.json`), so npm, yarn and pnpm workspaces are measured with no config. A `node_modules` that pnpm laid out, or that holds workspace links, is rebuilt in the worktree entry by entry, because pnpm refuses a symlinked one, and a tool's own state or cache folder inside it is left out. A link from `node_modules` back into the repo, which is how those tools wire a sibling package, is pointed at the worktree, so a test that imports a sibling package reads the ref being measured. `init` writes `[".venv"]` for Python and `[]` for Rust (`cargo test` writes to `target/`, so it is built fresh in the worktree rather than shared across refs). `"linkNodeModules": false` still works as shorthand for "link nothing".
100
102
  - **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
101
103
  - **setup**, an optional command run once per ref before metrics (for example `npm run build` or `pip install -e .`), after any source patch, so a build-gated suite measures the ref it is actually on.
102
104
 
105
+ These three shape every measurement, so they are part of the gate. A change that edits them is flagged as a gate change and is judged with the values from before it (see Gate integrity below).
106
+
103
107
  Copy a config for your stack (pytest, tsc, coverage, bundle size, LLM eval) from [examples/](examples/).
104
108
 
105
109
  ### Guardrails and inheritance
@@ -116,7 +120,7 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
116
120
 
117
121
  ### Mandates (protected paths)
118
122
 
119
- Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate before a single metric is read**, however green everything else is:
123
+ Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate**, however green everything else is. The metrics still run and are reported, but nothing they say clears a mandate:
120
124
 
121
125
  ```json
122
126
  // promptwheel.config.json
@@ -126,7 +130,7 @@ Some rules are not a metric at all, they are a hard line: *never touch `src/auth
126
130
  "metrics": [ /* ... */ ] }
127
131
  ```
128
132
 
129
- A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the union of the base ref's and the head's, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`). Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
133
+ A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the ones the base ref already had, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`), and one that narrows a mandate is checked against both versions (`edited: true`). A mandate added in the same change that touches its path does not apply to that change, on purpose: the commit that signs a contract adds the very files the contract protects. Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
130
134
 
131
135
  ### Supply chain (preset `supply-chain`)
132
136
 
@@ -136,7 +140,7 @@ The step every npm supply-chain attack needs inside a target repo is a supply-ch
136
140
  npx promptwheel init --preset supply-chain
137
141
  ```
138
142
 
139
- writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, and the Claude Code hook files, and lists the contract itself (`promptwheel.config.json`, `sawbuck.config.json`) under `evaluatorPaths`, so a change that edits it alongside a win is gamed by declaration while a change that only edits the contract still passes. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
143
+ writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, the copied scanners, the Claude Code hook files, the lock file, and the contract itself (`promptwheel.config.json`, `sawbuck.config.json`). A change that edits the contract fails (exit 1) unless a person passes `--allow "<reason>"`; the contract is also an `evaluatorPaths` entry, so editing it alongside a win is gamed by declaration. `--allow` clears only the contract, the lock and the hooks. Every other frozen path needs its mandate retired first, by a person, in a change of its own run with `--allow`. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
140
144
 
141
145
  ## In CI and in Claude Code
142
146
 
@@ -170,16 +174,16 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
170
174
 
171
175
  ## Trust model
172
176
 
173
- **What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
177
+ **What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). A grader is recognized by its file name in a scripting language (`eval.py`, `grader.sh`, `score.js`, `scoring.ts`) or by an `evals/` or `grader(s)/` directory; a production file with one of those names (`src/score.py`) is treated as a grader too, so a fix there needs `--allow`, and a grader with any other name (`evaluate.py`, `check.py`) is source unless you list it in `evaluatorPaths`. On top of that:
174
178
 
175
179
  - **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
176
- - **Gate integrity.** The gate is resolved as of the base commit too. A change that disables a guard or deletes a guarded metric in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed. The gate definition (metrics, guards, `evaluatorPaths`, `gamingThreshold`, `mandates`) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed.
180
+ - **Gate integrity.** A change is judged by the gate that was in force before it. The gate is resolved as of the base commit too, through `extends`. A change that disables a guard, deletes a guarded metric, or measures it another way (a new `cmd` such as `true`, `extract`, `direction`, `exact` or `report`) in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed, or would still fail. If the old guard only passes because the same change also edited the test, it reads `GAMED` too: a win under the old definition still has to survive the source-only re-run. A file declared in `evaluatorPaths` before the change still counts as a goalpost for that change, even if the change drops the declaration. The same goes for `setup`, `env` and `linkDirs`: when a change edits them, each guard is measured again with the old values and so is the source-only re-run, so a `setup` that rewrites a test cannot carry a pass (`gate.weakened[].change: measurement-edited`). `gamingCheck: false` on a metric that had the re-run on, and a lower `gamingThreshold`, apply from the next change. A change that adds the first config is compared with the zero-config gate (the detected test command), so `tests_pass` set to `true` in a new config is an edited guard; a first config that keeps the detected command, or replaces the placeholder, is not. The gate definition (each metric with its command and how it is read, guards, `evaluatorPaths`, `gamingThreshold`, mandate names and paths, `setup`, `env`, linked dirs) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed. Two limits: a test that could not run at all before a `setup` fix reads `GAMED` on the change that adds the fix (say so with `--allow`; the `setup` that `init` writes into a first config is exempt), and a base config that `extends` a file outside the repository is not compared on `setup`, `env` or `linkDirs`.
177
181
  - **A sanctioned override.** A commit trailer can be written by whoever writes the commit, including an agent, so `"allowTrailer": false` in the contract turns trailers off and only the `--allow` flag counts (the supply-chain preset sets this; the contract at the base ref decides, so a change cannot turn trailers back on for itself). When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. It covers a win whose source-only re-run could not be applied (`unprovable`, otherwise `inconclusive`) the same way. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
178
182
  - **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
179
183
 
180
- **The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test): that needs held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
184
+ **The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test), or in a build script the unchanged `setup` runs (a change that makes that script rewrite a test is a source edit to this check). A base config that `extends` a file outside the repository cannot be compared across the change. These need held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
181
185
 
182
- **Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guard whose command never ran (same value at both refs) makes the verdict `inconclusive`, never a fake pass.
186
+ **Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guarded pass/fail check that reads 0 at both refs (its command never ran, or it ran and failed both times; the report says which) makes the verdict `inconclusive`, never a fake pass.
183
187
 
184
188
  **When not to use it.** If you only care that tests pass and nobody could be editing the tests, your plain CI already has you covered. PromptWheel earns its place when the thing writing the code can also touch the thing judging it, or when you have a graded metric beyond pass/fail (an eval score, cost per run, latency, size) that a change could quietly move. Single-metric CI gates (benchmark trackers, size limits) go deeper on one number; wrap them as a metric `cmd` and PromptWheel composes them into one verdict.
185
189
 
@@ -187,7 +191,7 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
187
191
 
188
192
  A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly 2 to 3× your metric command's runtime plus about 40ms of worktree overhead. Point the per-turn gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for CI: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
189
193
 
190
- **Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run cannot leave clutter behind: the next run self-heals any orphaned worktree.
194
+ **Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run can leave a worktree in your temp dir; a later run removes it once it is a day old (an unregistered checkout of this repo after an hour). It removes only directories named the way it names its own (`promptwheel-`, `promptwheel-src-` or `pw-idx-`, then an 8-hex repo tag, then 6 random characters). Any other folder or worktree in the temp dir is left alone, at any age.
191
195
 
192
196
  ## Portability
193
197
 
@@ -48,7 +48,7 @@ export interface MetricConfig {
48
48
  allowNonzero?: boolean;
49
49
  /** Per-command timeout, default 300. */
50
50
  timeoutSec?: number;
51
- /** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). */
51
+ /** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). Set on a metric that had the re-proof on, it applies from the next change, not the one that sets it. */
52
52
  gamingCheck?: boolean;
53
53
  /** Per-metric override of `Config.gamingThreshold`. */
54
54
  gamingThreshold?: number;
@@ -62,7 +62,7 @@ export interface MetricConfig {
62
62
  allowable?: boolean;
63
63
  }
64
64
 
65
- /** A named hard rule: any change touching one of `paths` fails the gate before any metric is read. */
65
+ /** A named hard rule: any change touching one of `paths` fails the gate, whatever the metrics read. */
66
66
  export interface Mandate {
67
67
  name: string;
68
68
  /** Globs: a bare name matches that basename at any depth, a pattern with `/` is anchored at the repo root, `**` crosses directories. */
@@ -85,7 +85,7 @@ export interface Config {
85
85
  env?: Record<string, string>;
86
86
  /** A per-ref build or install, run in the worktree before measuring (a failed setup leaves the metric inert, never a false green). */
87
87
  setup?: string;
88
- /** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. */
88
+ /** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. A change that lowers it is still judged by the value before it. */
89
89
  gamingThreshold?: number;
90
90
  /** Extra files that define the gate (a threshold, a budget); a change to one alongside a win is gamed by declaration. Same glob rules as `Mandate.paths`. */
91
91
  evaluatorPaths?: string[];
@@ -144,6 +144,12 @@ export interface MetricResult extends Evaluation {
144
144
  name: string;
145
145
  direction: Direction;
146
146
  guard: boolean;
147
+ /** With `inert`: where the command ran and failed (`both` refs, or only `base` or `head`). Absent when it ran at neither: exit 126 or 127, a timeout, or the placeholder command `init` writes. */
148
+ ran?: 'both' | 'base' | 'head';
149
+ /** With `inert`: a hint read from the command's output. `missing-dependency` when it names a module, package or command the clean copy does not have, else `failing`. Never part of a verdict. */
150
+ cause?: 'missing-dependency' | 'failing' | 'too-much-output';
151
+ /** With `cause: 'missing-dependency'`: the line that names what is missing, at most 200 characters. */
152
+ detail?: string;
147
153
  /** `true` when the win did not survive the source-only re-run; `null` when the source-only patch could not be applied (then `unprovable` is set and the verdict is `inconclusive`, never `pass`). Absent without gaming detection or when the metric did not improve. */
148
154
  gamed?: boolean | null;
149
155
  /** `true`: the source-only patch could not be applied, so this win could not be re-proven and the verdict is `inconclusive`. `false` only after a declared override cleared it (then `allowed` is `true`). */
@@ -174,9 +180,13 @@ export interface MandateViolation {
174
180
 
175
181
  export interface GateWeakening {
176
182
  name: string;
177
- change: 'guard-removed' | 'guard-disabled';
178
- /** What the dropped guard would have read at head under its base definition. */
179
- status: MetricStatus;
183
+ /**
184
+ * `guard-edited`: the guard is still on but measured another way (a new `cmd`, `extract`, `direction`, `exact` or `report`).
185
+ * `measurement-edited`: the config's `setup`, `env` or linked dirs changed, and measured with the ones before the change the guard fails at head while the new ones read it fine.
186
+ */
187
+ change: 'guard-removed' | 'guard-disabled' | 'guard-edited' | 'measurement-edited';
188
+ /** What the guard would have read at head under its base definition (and the base `setup`, `env` and linked dirs); `failed` when it still fails there. */
189
+ status: MetricStatus | 'failed';
180
190
  }
181
191
 
182
192
  export interface EvaluatorChange {
@@ -188,11 +198,11 @@ export interface EvaluatorChange {
188
198
 
189
199
  /** The gate-integrity record, present when gaming detection ran. */
190
200
  export interface GateRecord {
191
- /** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates) at each ref; `base` is `null` when there was no config at base. */
201
+ /** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) at each ref; `base` is `null` when there was no config at base. */
192
202
  fingerprint: { base: string | null; head: string };
193
203
  /** The gate definition differs between base and head. */
194
204
  changed?: true;
195
- /** Guards the change dropped that would have failed at head; non-empty forces `gamed` unless allowed. */
205
+ /** Guards the change dropped, edited or measures another way that would have failed at head; non-empty forces `gamed` unless allowed. A change that adds the first config is compared with the zero-config gate (the detected test command). */
196
206
  weakened?: GateWeakening[];
197
207
  /** Declared `evaluatorPaths` files whose content changed in this diff. */
198
208
  evaluatorChanged?: EvaluatorChange[];
@@ -228,9 +238,17 @@ export interface Report {
228
238
  mandates?: MandateViolation[];
229
239
  gate?: GateRecord;
230
240
  allowed?: AllowedRecord;
241
+ /** Present only when the `setup` command failed: at which ref, with what exit code, and its last output line. */
242
+ setup?: { base?: SetupFailure; head?: SetupFailure };
231
243
  provenance: Provenance;
232
244
  }
233
245
 
246
+ export interface SetupFailure {
247
+ /** The setup command's exit code, or `null` when it was killed (a timeout). */
248
+ exit: number | null;
249
+ lastLine: string;
250
+ }
251
+
234
252
  // ---------------------------------------------------------------------------
235
253
  // Public library API (stable across 0.x minors)
236
254
  // ---------------------------------------------------------------------------
@@ -260,7 +278,7 @@ export declare function deriveVerdict(metrics: ReadonlyArray<Partial<MetricResul
260
278
  * a missing `extends` target, no commits yet, a base or head ref that is not in this clone, no
261
279
  * commit to compare against) throws an Error whose `name` is `'ConfigError'` and whose `exitCode`
262
280
  * is 64, the code the CLI exits with for the same condition. A `repo` that names a subdirectory
263
- * is resolved to the repository root, where the config lives and the record is written.
281
+ * is resolved to the repository root, where the config lives. A run writes nothing to the repo.
264
282
  */
265
283
  export declare function check(options?: CheckOptions): Report;
266
284
 
@@ -14,9 +14,9 @@
14
14
 
15
15
  import { execSync, execFileSync } from 'node:child_process';
16
16
  import { createHash } from 'node:crypto';
17
- import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, statSync, copyFileSync } from 'node:fs';
17
+ import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, lstatSync, copyFileSync } from 'node:fs';
18
18
  import { tmpdir } from 'node:os';
19
- import { join, dirname, isAbsolute, relative } from 'node:path';
19
+ import { join, dirname, basename, isAbsolute, relative, sep } from 'node:path';
20
20
  import { fileURLToPath } from 'node:url';
21
21
 
22
22
  // ---------------------------------------------------------------------------
@@ -36,6 +36,9 @@ const spread = (xs) => { const a = xs.filter((x) => x != null); return a.length
36
36
  // repo is still measuring in (a long gate elsewhere must not lose its worktree mid-run)
37
37
  function tmpOwnerTag(repo) { let r = repo; try { r = realpathSync(repo); } catch { /* keep as given */ } return createHash('sha1').update(String(r)).digest('hex').slice(0, 8); }
38
38
  const tmpPrefix = (repo, kind) => `${kind}-${tmpOwnerTag(repo)}-`;
39
+ // exactly the names mkdtemp makes from tmpPrefix: the kind, the 8-hex owner tag, then mkdtemp's 6 random
40
+ // characters. Self-heal removes nothing else, so a folder that only starts with "promptwheel-" is never ours.
41
+ const ENGINE_TMP = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-[A-Za-z0-9]{6}$/;
39
42
 
40
43
  // usage / config errors exit 64 (EX_USAGE), so they can never be confused with a verdict
41
44
  // (0 pass, 1 fail, 2 gamed, 3 inconclusive): in CI a config typo must not read as GAMED.
@@ -112,8 +115,14 @@ const hasConfigFile = (repo) => existsSync(join(repo, 'promptwheel.config.json')
112
115
  // measurement actually runs in a throwaway worktree. Lets `run` and `check()` work with no config
113
116
  // file; `init` is for customizing. A repo with no test command gets a sentinel that reads inert ->
114
117
  // INCONCLUSIVE (never a fake green), with a message pointing at the fix.
115
- function autoConfig(repo) {
116
- return { ...detectEnv(repo), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }] };
118
+ function autoConfig(repo, at = stackAt(repo)) {
119
+ return { ...detectEnv(repo, at), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo, at), extract: 'exit', direction: 'pass', guard: true }] };
120
+ }
121
+ // where stack detection looks: the working tree, or the tracked files as of `ref` (the repo as it was before a change)
122
+ function stackAt(repo, ref) {
123
+ if (!ref) return { has: (f) => existsSync(join(repo, f)), read: (f) => readFileSync(join(repo, f), 'utf8') };
124
+ const read = (f) => execFileSync('git', ['show', `${ref}:${f}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] });
125
+ return { has: (f) => { try { read(f); return true; } catch { return false; } }, read };
117
126
  }
118
127
 
119
128
  function resolveBase(repo, base) {
@@ -123,7 +132,8 @@ function resolveBase(repo, base) {
123
132
  catch { bad('promptwheel: no commits yet, make an initial commit before gating'); }
124
133
  for (const b of ['origin/main', 'origin/master', 'main', 'master']) {
125
134
  let mb = '';
126
- try { mb = git(['merge-base', 'HEAD', b], repo); } catch { continue; }
135
+ // stderr ignored: a ref that is not there (no origin/main in this clone) is expected, not an error to print
136
+ try { mb = execFileSync('git', ['merge-base', 'HEAD', b], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } catch { continue; }
127
137
  // on the trunk itself the merge-base IS HEAD; comparing a commit to itself would be a
128
138
  // trivial PASS that verified nothing, so fall through to the last commit instead.
129
139
  if (mb && mb !== head) return mb;
@@ -132,15 +142,32 @@ function resolveBase(repo, base) {
132
142
  catch { bad('promptwheel: no commit to compare against (one commit, or a shallow clone): pass --base, or --working'); }
133
143
  }
134
144
 
135
- function runMetric(cwd, m, env) {
136
- let stdout = '', code = 0;
145
+ // exit statuses of a sample list, kept beside it: 127 (the shell found no such command), 126 (it could not
146
+ // execute it) or no status (killed: a timeout) mean the metric never ran, which needs a different fix from a
147
+ // test that ran and failed
148
+ const exitCodes = new WeakMap();
149
+ const neverRan = (samples) => { const c = exitCodes.get(samples) || []; return c.length > 0 && c.every((x) => x == null || x === 126 || x === 127); };
150
+ // Why a metric's command failed, read from its output: a dependency the clean copy does not have reads differently
151
+ // from tests that simply fail. A hint for the person, never part of a verdict.
152
+ const MISSING_DEP = /Cannot find (?:module|package) ['"][^'"\n]+['"]|ERR_MODULE_NOT_FOUND[^\n]*|ModuleNotFoundError: [^\n]+|No module named [^\n]+|error TS2307: [^\n]+|[^\n:]*: (?:command )?not found[^\n]*|can't find crate[^\n]*|cannot find package [^\n]+/i;
153
+ function whyFailed(text) {
154
+ const m = String(text || '').replace(/\x1b\[[0-9;]*m/g, '').match(MISSING_DEP);
155
+ return m ? { cause: 'missing-dependency', detail: m[0].trim().slice(0, 200) } : { cause: 'failing' };
156
+ }
157
+ const failNotes = new WeakMap(); // samples -> why the last failing run failed
158
+ function runMetric(cwd, m, env, codes, notes) {
159
+ let stdout = '', code = 0, status = 0;
137
160
  // Never leak the harness's own node:test context into the measured gate: a metric like
138
161
  // `node --test` would see NODE_TEST_CONTEXT, act as a child reporter, and report a bogus
139
162
  // exit code (so the gate can't be trusted when PromptWheel itself runs under a test runner).
140
163
  const childEnv = { ...(env || process.env) }; delete childEnv.NODE_TEST_CONTEXT;
141
164
  try {
142
- stdout = execSync(m.cmd, { cwd, env: childEnv, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
143
- } catch (e) { code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
165
+ // maxBuffer: the default is 1 MB, and a suite that prints more than that was killed and read as failing. A real
166
+ // repo found it (tap prints every assertion when its output is a pipe): the check said "your tests fail" about
167
+ // a suite that passes.
168
+ stdout = execSync(m.cmd, { cwd, env: childEnv, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000, maxBuffer: 1024 * 1024 * 1024 });
169
+ } catch (e) { status = e.status ?? null; code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; if (notes) notes.push(e.code === 'ENOBUFS' ? { cause: 'too-much-output', detail: 'the command printed more than 1 GB, so it was stopped before it finished' } : whyFailed(`${e.stderr || ''}\n${e.stdout || ''}`)); }
170
+ if (codes) codes.push(status);
144
171
  return extract(stdout, code, m.extract, m.allowNonzero);
145
172
  }
146
173
 
@@ -169,12 +196,63 @@ function extract(stdout, code, mode, allowNonzero) {
169
196
  // with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
170
197
  // Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
171
198
  // not here, the engine stays ecosystem-agnostic.
172
- function bridgeEnv(repo, wt, cfg) {
173
- const linkDirs = Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']);
174
- for (const d of linkDirs) {
175
- if (existsSync(join(repo, d)) && !existsSync(join(wt, d))) {
176
- try { symlinkSync(join(repo, d), join(wt, d)); } catch { /* best effort */ }
199
+ const linkDirsOf = (cfg) => (Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']));
200
+ // Every folder that has a tracked package.json and a node_modules of its own: the workspaces of a monorepo, whichever
201
+ // tool laid them out. Asked of git, so nothing walks the tree.
202
+ function workspaceModuleDirs(repo) {
203
+ let files = '';
204
+ try { files = execFileSync('git', ['ls-files', '-z', '--', '*/package.json'], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], maxBuffer: 64 * 1024 * 1024 }); } catch { return []; }
205
+ const out = [];
206
+ for (const f of files.split('\0')) {
207
+ if (!f || f.split('/').includes('node_modules')) continue;
208
+ const d = join(dirname(f), 'node_modules');
209
+ if (existsSync(join(repo, d))) out.push(d);
210
+ if (out.length >= 500) break;
211
+ }
212
+ return out;
213
+ }
214
+ // A node_modules folder in a workspaces repo holds links back into the repo: npm and yarn link each workspace package
215
+ // from the root, pnpm from the workspace that depends on it. One symlink to the real folder would make a test in the
216
+ // worktree import a sibling package from the working tree, at both refs, so a regression there would never show. Such
217
+ // a folder is rebuilt in the worktree instead: each entry linked on its own, and an entry that points back into the
218
+ // repo pointed at the same path in the worktree. A folder with no such entry, in a repo pnpm did not lay out, keeps
219
+ // the single link it always had.
220
+ function linkModules(repo, wt, d) {
221
+ const src = join(repo, d), dst = join(wt, d);
222
+ const entries = [];
223
+ try {
224
+ for (const e of readdirSync(src, { withFileTypes: true })) {
225
+ if (e.name.startsWith('@') && e.isDirectory()) { for (const s of readdirSync(join(src, e.name), { withFileTypes: true })) entries.push([join(e.name, s.name), s]); }
226
+ else entries.push([e.name, e]);
227
+ }
228
+ } catch { return; }
229
+ const back = (name, e) => {
230
+ if (!e.isSymbolicLink()) return null;
231
+ try { const r = relative(repo, realpathSync(join(src, name))); return r && !r.startsWith('..') && !isAbsolute(r) && !r.split(sep).includes('node_modules') ? r : null; } catch { return null; }
232
+ };
233
+ const links = entries.map(([name, e]) => [name, back(name, e), e]);
234
+ // pnpm refuses a node_modules, or its own state folder inside it, that is a symlink: its scripts stop at once
235
+ const pnpm = entries.some(([name]) => name === '.pnpm' || name === '.modules.yaml');
236
+ try {
237
+ if (!pnpm && !links.some(([, rel]) => rel)) { symlinkSync(src, dst); return; }
238
+ mkdirSync(dst, { recursive: true });
239
+ for (const [name, rel, e] of links) {
240
+ // a tool's own state or cache folder (pnpm's task-run state, .cache, .vite) is written to, so each worktree gets
241
+ // its own, made by the tool. The store (.pnpm) and the executables (.bin) are read, so they are linked.
242
+ if (name.startsWith('.') && name !== '.bin' && name !== '.pnpm' && e.isDirectory()) continue;
243
+ if (name.includes(sep)) mkdirSync(dirname(join(dst, name)), { recursive: true });
244
+ symlinkSync(rel ? join(wt, rel) : join(src, name), join(dst, name));
177
245
  }
246
+ } catch { /* best effort */ }
247
+ }
248
+ function bridgeEnv(repo, wt, cfg) {
249
+ let root = repo; try { root = realpathSync(repo); } catch { /* keep the path as given */ }
250
+ // with nothing declared, a Node repo links its root node_modules and each workspace's own
251
+ const dirs = (Array.isArray(cfg.linkDirs) || cfg.linkNodeModules === false) ? linkDirsOf(cfg) : ['node_modules', ...workspaceModuleDirs(root)];
252
+ for (const d of dirs) {
253
+ if (!existsSync(join(root, d)) || existsSync(join(wt, d)) || !existsSync(dirname(join(wt, d)))) continue;
254
+ if (basename(d) === 'node_modules') linkModules(root, wt, d);
255
+ else { try { symlinkSync(join(root, d), join(wt, d)); } catch { /* best effort */ } }
178
256
  }
179
257
  if (!cfg.env) return process.env;
180
258
  return { ...process.env, ...Object.fromEntries(Object.entries(cfg.env).map(([k, v]) => [k, String(v).replaceAll('{wt}', () => wt)])) }; // fn replacement: a literal wt avoids $-pattern interpretation
@@ -182,10 +260,16 @@ function bridgeEnv(repo, wt, cfg) {
182
260
  // an optional per-ref build/install (e.g. `npm run build`, `pip install -e .`). Best-effort: a
183
261
  // failed setup just leaves the metric inert → INCONCLUSIVE (never a false green). Run by the
184
262
  // caller AFTER any source patch, so it builds the tree actually being measured.
263
+ // Returns nothing when setup worked, or { exit, lastLine } when it did not, so the report can say so.
185
264
  function runSetup(wt, cfg, env) {
186
- if (!cfg.setup) return;
187
- try { execSync(cfg.setup, { cwd: wt, env, stdio: 'ignore', timeout: 300_000 }); } catch { /* inert → inconclusive */ }
265
+ if (!cfg.setup) return undefined;
266
+ try { execSync(cfg.setup, { cwd: wt, env, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: 300_000, maxBuffer: 1024 * 1024 * 1024 }); return undefined; }
267
+ catch (e) {
268
+ const lines = `${e.stdout || ''}\n${e.stderr || ''}`.replace(/\x1b\[[0-9;]*m/g, '').split('\n').map((l) => l.trim()).filter(Boolean);
269
+ return { exit: e.status ?? null, lastLine: (lines[lines.length - 1] || '').slice(0, 200) }; // the metric goes inert → inconclusive
270
+ }
188
271
  }
272
+ const setupFails = new WeakMap(); // a measureAt result -> how its setup failed
189
273
 
190
274
  // measure every metric `repeat` times at a ref, in a throwaway worktree (never touches your tree)
191
275
  function measureAt(repo, ref, cfg, repeat) {
@@ -193,11 +277,14 @@ function measureAt(repo, ref, cfg, repeat) {
193
277
  git(['worktree', 'add', '--quiet', '--detach', wt, ref], repo);
194
278
  try {
195
279
  const env = bridgeEnv(repo, wt, cfg);
196
- runSetup(wt, cfg, env);
280
+ const setupFailed = runSetup(wt, cfg, env);
197
281
  const out = {};
282
+ if (setupFailed) setupFails.set(out, setupFailed);
198
283
  for (const m of cfg.metrics) {
199
- const samples = [];
200
- for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
284
+ const samples = [], codes = [], notes = [];
285
+ for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env, codes, notes));
286
+ exitCodes.set(samples, codes);
287
+ if (notes.length) failNotes.set(samples, notes[notes.length - 1]);
201
288
  out[m.name] = samples;
202
289
  }
203
290
  return out;
@@ -266,9 +353,12 @@ const NON_SOURCE = [
266
353
  /_spec\.rb$/i,
267
354
  /[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
268
355
  /\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
269
- // a grader/eval/score FILE by name (eval.ts, grader-x.js, score.test.mjs, scoring.ts), never a prefix:
270
- // evaluate.ts, evaluation.js, scoreboard.js are production source and must stay in the slice
271
- /(^|\/)(eval|evals|grader|graders|score|scorer|scoring)([._-][^/]*)?\.[cm]?[jt]s$/i,
356
+ // a grader/eval/score FILE by name in a scripting language (eval.py, grader-x.js, grade.sh, scoring.ts), never a
357
+ // prefix: evaluate.ts, evaluation.py, scoreboard.js stay source. Compiled names stay source too (eval.c, eval.go
358
+ // are interpreters; Score.java is a type), and so do .tsx/.vue files (a Score component, a /score page).
359
+ /(^|\/)(eval|evals|grade|grader|graders|score|scorer|scoring)([._-][^/]*)?\.([cm]?[jt]s|py|sh|bash|zsh|rb|pl|php|lua|r|jl)$/i,
360
+ // anything under an evals/ or grader(s)/ directory. Not eval/ (an interpreter's package) or scoring/ (a product module).
361
+ /(^|\/)(evals|graders?)\//i,
272
362
  // config files only, require a .config/.conf/.setup segment so PRODUCTION source that
273
363
  // happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
274
364
  // the source slice (a false sweep here can flag an honest win as GAMED).
@@ -311,9 +401,9 @@ function evaluatorMatcher(cfg) {
311
401
  // forbids ANY change to its paths (same glob semantics as evaluatorPaths). A hit fails the gate
312
402
  // regardless of metrics and rides at the top of the report; mandates inherit through `extends`,
313
403
  // so a parent scope's rule holds for every repo beneath it.
314
- // The mandates in force are the UNION of the head config's and the base ref's (`baseMandates`,
315
- // resolved by gateDefAtRef): deleting a mandate in the same change that violates it does not
316
- // clear it. A mandate removed at head but violated in the diff is reported with `removed: true`; one
404
+ // The mandates in force are the base ref's (`baseMandates`, resolved by gateDefAtRef), in their head
405
+ // version when both have them: deleting a mandate in the same change that violates it does not
406
+ // clear it, and one the change adds does not apply to it (see below). A mandate removed at head but violated in the diff is reported with `removed: true`; one
317
407
  // whose paths were edited at head (same name, different paths) is tested in BOTH versions and reported
318
408
  // with `edited: true`, so narrowing a rule in the change that breaks it does not clear it either.
319
409
  // The honest way to retire one is a separate change that only edits the config.
@@ -482,12 +572,16 @@ function workingSnapshot(repo) {
482
572
  }
483
573
  }
484
574
 
485
- // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
575
+ // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
486
576
  // its `finally` never ran. On the next run, drop stale worktree registry entries and any
487
- // abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. Never touches a
488
- // live worktree (a still-registered dir, or this repo's checkout younger than 1h). A dir tagged
489
- // for ANOTHER repo may be mid-measurement, so it gets a full day; after that it is an orphan of a
490
- // repo that was deleted or moved (a new real path is a new tag nobody will ever claim).
577
+ // abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. It only ever removes a
578
+ // directory whose name is exactly what this engine generates (ENGINE_TMP): a folder named
579
+ // promptwheel-notes, or another repo's worktree named promptwheel-site-wt, is not ours at any age.
580
+ // Ages, by dir mtime: this repo's unregistered checkout after 1h; this repo's still-registered worktree
581
+ // after a full day (a killed run leaves it registered, and a younger one may be a concurrent gate still
582
+ // measuring); a dir tagged for ANOTHER repo after a full day (an orphan of a repo that was deleted or
583
+ // moved: a new real path is a new tag nobody will ever claim). A registered worktree that is not
584
+ // tagged for this repo is never touched: it may be yours.
491
585
  function selfHeal(repo) {
492
586
  try {
493
587
  git(['worktree', 'prune'], repo); // remove registry entries whose dir is already gone (always safe)
@@ -496,16 +590,18 @@ function selfHeal(repo) {
496
590
  .split('\n').filter((l) => l.startsWith('worktree ')).map((l) => l.slice(9).trim()),
497
591
  );
498
592
  const tmp = tmpdir(), now = Date.now(), mine = tmpOwnerTag(repo);
499
- const TAGGED = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-/;
500
593
  for (const name of readdirSync(tmp)) {
501
- if (!name.startsWith('promptwheel-') && !name.startsWith('pw-idx-')) continue; // worktrees + temp-index dirs
502
- const tag = (name.match(TAGGED) || [])[2] || null;
503
- // ours after an hour; another repo's, or an untagged dir (an older engine's), only after a full day
504
- const maxAge = tag === mine ? 3600_000 : 86_400_000;
594
+ const made = ENGINE_TMP.exec(name);
595
+ if (!made) continue; // not a name this engine generates (worktrees + temp-index dirs): never removed
596
+ const tag = made[2];
597
+ // ours after an hour; another repo's only after a full day
505
598
  const p = join(tmp, name);
506
- if (tracked.has(p)) continue; // a live/registered worktree, leave it
507
- try { if (statSync(p).mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
599
+ if (tracked.has(p) && tag !== mine) continue; // a registered worktree this repo's gate did not make
600
+ const maxAge = tag === mine && !tracked.has(p) ? 3600_000 : 86_400_000;
601
+ // lstat: the engine makes directories, so a file or a symlink under one of its names is left alone
602
+ try { const st = lstatSync(p); if (st.isDirectory() && st.mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
508
603
  }
604
+ git(['worktree', 'prune'], repo); // drop the registry entries of anything removed above
509
605
  } catch { /* cleanup must never block the gate */ }
510
606
  }
511
607
 
@@ -537,18 +633,25 @@ function warnEditableInstall(repo) {
537
633
  // guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
538
634
  // guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
539
635
  // that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
540
- const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer'];
636
+ const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer', 'setup', 'env', 'linkDirs', 'linkNodeModules'];
637
+ // what runs or shapes every measurement besides the metric's own command. A setup can rewrite a test and an
638
+ // env var can skip one, so these are part of the gate: fingerprinted, and held at their base values for the
639
+ // change that edits them (see detectGamingPass).
640
+ const measureOf = (s) => ({ setup: s.setup, env: s.env, linkDirs: s.linkDirs, linkNodeModules: s.linkNodeModules });
641
+ const measuredWith = (s) => JSON.stringify([s.setup ?? null, Object.entries(s.env || {}).map(([k, v]) => `${k}=${v}`).sort(), [...linkDirsOf(s)].sort()]);
541
642
 
542
643
  // Resolve the gate definition as of `ref` (metrics + mandates + scalars), following `extends` by
543
644
  // reading every file from that ref via `git show`, mirroring resolveConfig (a parent's mandates
544
645
  // hold beneath it; a local mandate of the same name overrides). Returns { metrics:Map, scalars }
545
- // with scalars.mandates, or null when there is no config at `ref` (a gate ADDED in this range is
546
- // not a weakening). Never exits: any missing/unreadable file or parse error yields null (that
547
- // branch simply contributes nothing).
646
+ // with scalars.mandates, or null when there is no config at `ref` (gate() then falls back to the
647
+ // zero-config gate the repo ran under, see zeroConfigGate). Never exits: any missing/unreadable file or
648
+ // parse error yields null (that branch simply contributes nothing), and `partial` says an extends target
649
+ // could not be read at `ref` (it lives outside the repo), so how the base measured is not fully known.
548
650
  function gateDefAtRef(repo, ref) {
549
651
  // quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
550
652
  const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
551
653
  const seen = new Set();
654
+ let partial = false;
552
655
  const resolve = (p) => {
553
656
  if (seen.has(p)) return null;
554
657
  seen.add(p);
@@ -557,9 +660,11 @@ function gateDefAtRef(repo, ref) {
557
660
  let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
558
661
  const byName = new Map(); const mandatesByName = new Map(); const scalars = {};
559
662
  for (const ext of [].concat(cfg.extends || [])) {
560
- if (isAbsolute(ext)) continue; // outside the repo, cannot git-show; unchanged anyway
561
- const base = resolve(join(dirname(p), ext).split('\\').join('/'));
562
- if (!base) continue;
663
+ if (isAbsolute(ext)) { partial = true; continue; } // outside the repo, cannot git-show; unchanged anyway
664
+ const bp = join(dirname(p), ext).split('\\').join('/');
665
+ const again = seen.has(bp); // a diamond: already merged, not a missing piece
666
+ const base = resolve(bp);
667
+ if (!base) { if (!again) partial = true; continue; }
563
668
  for (const m of base.metrics.values()) byName.set(m.name, m);
564
669
  for (const md of base.scalars.mandates) mandatesByName.set(md.name, md);
565
670
  for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
@@ -570,18 +675,35 @@ function gateDefAtRef(repo, ref) {
570
675
  scalars.mandates = [...mandatesByName.values()];
571
676
  return { metrics: byName, scalars };
572
677
  };
573
- return resolve('promptwheel.config.json');
678
+ const def = resolve('promptwheel.config.json');
679
+ return def && { ...def, partial };
680
+ }
681
+
682
+ // The gate a repo with no config ran under: the detected test command (source-only re-run on) with the
683
+ // stack's env and linkDirs, detected from the tracked files as of `ref`. A change that adds the first
684
+ // config is judged against this, so `tests_pass.cmd: "true"` in a new config is an edited guard, not a
685
+ // fresh start. `initSetup` is the setup `init` would add: a first config that carries exactly it has not
686
+ // changed how the gate measures. With no test command to detect (the placeholder) the old gate measured
687
+ // nothing, so its setup is no baseline (`partial`).
688
+ function zeroConfigGate(repo, ref) {
689
+ const at = stackAt(repo, ref), auto = autoConfig(repo, at);
690
+ return { metrics: new Map(auto.metrics.map((m) => [m.name, m])), scalars: { mandates: [], ...measureOf(auto) }, zeroConfig: true, initSetup: detectSetup(repo, at), partial: auto.metrics[0].cmd === PLACEHOLDER_TEST };
574
691
  }
575
692
 
693
+ // how a metric is measured and judged: change any of these on a guard and it is a different guard
694
+ // what init writes when it finds no test command. It guards nothing, so replacing it is setup, not a weakened gate.
695
+ const PLACEHOLDER_TEST = 'echo "set your test command in promptwheel.config.json" && false';
696
+ const measuredBy = (m) => JSON.stringify([m.cmd ?? null, m.extract ?? 'number', m.direction || 'up', m.exact === true, m.report === true]);
576
697
  // an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
577
698
  function gateFingerprint(metricsList, scalars) {
578
699
  const norm = {
579
700
  metrics: metricsList.map((m) => ({
580
- name: m.name, guard: m.guard === true, direction: m.direction || 'up',
701
+ name: m.name, guard: m.guard === true, measuredBy: measuredBy(m),
581
702
  gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
582
703
  })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
583
704
  gamingThreshold: scalars.gamingThreshold ?? null,
584
705
  evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
706
+ measuredWith: measuredWith(scalars),
585
707
  mandates: Array.isArray(scalars.mandates) ? scalars.mandates.map((m) => ({ name: String(m.name || ''), paths: [...(Array.isArray(m.paths) ? m.paths : [])].sort() })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) : [],
586
708
  };
587
709
  return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
@@ -626,7 +748,8 @@ function verdictFingerprint(report) {
626
748
  return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
627
749
  }
628
750
 
629
- // guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
751
+ // guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off), or
752
+ // measures another way (a new cmd, extract, direction, exact or report flag): `cmd: "true"` is a dropped guard too
630
753
  function gateWeakenings(baseGate, headMetrics) {
631
754
  if (!baseGate) return [];
632
755
  const head = new Map(headMetrics.map((m) => [m.name, m]));
@@ -636,6 +759,7 @@ function gateWeakenings(baseGate, headMetrics) {
636
759
  const hm = head.get(bm.name);
637
760
  if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
638
761
  else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
762
+ else if (!bm.report && bm.cmd !== PLACEHOLDER_TEST && measuredBy(hm) !== measuredBy(bm)) weak.push({ name: bm.name, change: 'guard-edited', def: bm });
639
763
  }
640
764
  return weak;
641
765
  }
@@ -654,29 +778,51 @@ function readAllowTrailer(repo, ref) {
654
778
  return m && m[1].trim() ? { reason: m[1].trim().slice(0, 300) } : null;
655
779
  }
656
780
 
781
+ // A change is judged by the gate that was in force before it (`baseGate`: the config at base, or the
782
+ // zero-config gate when this change adds the first config). So a change cannot switch off the source-only
783
+ // re-run for a metric that had it, loosen gamingThreshold, or swap the setup, env or linked dirs it is
784
+ // measured with: the base values hold for the change that edits them, and apply from the next one.
785
+ const gamingCheckOff = (cm, baseGate) => {
786
+ if (!cm || cm.gamingCheck !== false) return false;
787
+ const bm = baseGate && baseGate.metrics.get(cm.name);
788
+ return !bm || bm.gamingCheck === false; // off before this change too, or a metric this change adds
789
+ };
790
+ function gamingThresholdFor(cm, cfg, baseGate) {
791
+ const now = cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5;
792
+ const bm = baseGate && baseGate.metrics.get(cm.name);
793
+ const before = bm ? (bm.gamingThreshold ?? baseGate.scalars.gamingThreshold ?? (baseGate.partial ? now : 0.5)) : now;
794
+ return Math.max(now, before); // the stricter of before and after
795
+ }
796
+
657
797
  // The three Section 3 checks, run only with detectGaming, in order: (1) re-prove every actual
658
798
  // win with the agent's source edits alone; (2) content-hash auto-fail: a declared evaluatorPath
659
799
  // gate file edited in this diff is a moved goalpost, so a metric that improved alongside it is
660
800
  // gamed by declaration, deterministic even if the source-only re-run read inert; (3) gate
661
- // integrity: resolve the gate as of BASE, re-measure any guard HEAD dropped under its base
662
- // definition (it only counts as tampering if it WOULD HAVE FAILED at head), and fingerprint the
663
- // gate definition at both refs (metrics, evaluatorPaths, gamingThreshold, mandates) so a quietly
664
- // deleted guard or mandate is tamper-evident: base != head is recorded as gate.changed and
801
+ // integrity: take the gate as of BASE, re-measure any guard HEAD dropped, edited or measures with
802
+ // another setup, env or linked dirs, under its base definition (it only counts as tampering if it
803
+ // WOULD HAVE FAILED at head), and fingerprint the gate definition at both refs (metrics,
804
+ // evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) so a quietly deleted guard
805
+ // or mandate is tamper-evident: base != head is recorded as gate.changed and
665
806
  // printed, whatever the verdict. Mutates `metrics` in place (gamed/sourceOnly/retained/
666
807
  // gamingReason) and returns the report.gate record. An allow trailer can still clear a gamed
667
808
  // metric afterwards, in gate().
668
- function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
809
+ function detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) {
810
+ // how the base measured. Unknown (`partial`) reads as unchanged; so does a first config that adds only the setup `init` detects
811
+ const bs = baseGate ? (baseGate.partial ? { ...baseGate.scalars, ...measureOf(cfg) } : baseGate.scalars) : null;
812
+ const asInit = !!bs && baseGate.zeroConfig && measuredWith({ ...bs, setup: baseGate.initSetup }) === measuredWith(cfg);
813
+ const measureChanged = !!bs && !asInit && measuredWith(bs) !== measuredWith(cfg);
814
+ const baseRun = measureChanged ? { ...cfg, ...measureOf(bs) } : cfg; // the head gate, run with the base setup, env and linked dirs
669
815
  const toProve = [];
670
816
  for (const m of metrics) {
671
817
  if (m.status !== 'improved') continue;
672
818
  const cm = cfg.metrics.find((c) => c.name === m.name);
673
- if (cm.gamingCheck === false) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
819
+ if (gamingCheckOff(cm, baseGate)) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
674
820
  toProve.push(cm);
675
821
  }
676
- const srcResults = measureSourceOnlyAll(repo, base, head, toProve, cfg, repeat); // one patched checkout for all of them
822
+ const srcResults = measureSourceOnlyAll(repo, base, head, toProve, baseRun, repeat); // one patched checkout for all of them
677
823
  for (const cm of toProve) {
678
824
  const m = metrics.find((x) => x.name === cm.name);
679
- const j = judgeGaming(m, srcResults[cm.name], cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
825
+ const j = judgeGaming(m, srcResults[cm.name], gamingThresholdFor(cm, cfg, baseGate));
680
826
  m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
681
827
  if (j.unprovable) m.unprovable = true;
682
828
  }
@@ -687,32 +833,55 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
687
833
  if (split.source.length && split.nonSource.length) {
688
834
  const held = metrics.filter((m) => m.status === 'unchanged' && m.direction === 'pass' && m.after === 1 && !m.tripwire);
689
835
  const defs = held.map((m) => cfg.metrics.find((c) => c.name === m.name)).filter(Boolean);
690
- const res = measureSourceOnlyAll(repo, base, head, defs, cfg, repeat);
836
+ const res = measureSourceOnlyAll(repo, base, head, defs, baseRun, repeat);
691
837
  for (const m of held) { const r = res[m.name]; if (r && r.samples && r.samples.length && median(r.samples) !== 1) m.heldByTestEdit = true; }
692
838
  }
693
- const evalChanged = changedEvaluatorPaths(repo, base, head, cfg);
839
+ // a file declared as a goalpost before this change still is one for this change: dropping the declaration in the
840
+ // same diff that edits the file does not make the edit count as source
841
+ const declared = [...new Set([...(cfg.evaluatorPaths || []), ...((baseGate && baseGate.scalars.evaluatorPaths) || [])])];
842
+ const evalChanged = changedEvaluatorPaths(repo, base, head, { ...cfg, evaluatorPaths: declared });
694
843
  if (evalChanged.length) {
695
844
  for (const m of metrics) {
696
845
  if (m.status !== 'improved' || m.gamed === true) continue;
697
846
  const cm = cfg.metrics.find((c) => c.name === m.name);
698
- if (cm && cm.gamingCheck === false) continue; // tripwire/test-side guards legitimately live in gate files
847
+ if (gamingCheckOff(cm, baseGate)) continue; // tripwire/test-side guards legitimately live in gate files
699
848
  m.gamed = true;
700
849
  m.gamingReason = `a declared evaluatorPath gate file changed in this diff (${evalChanged.map((e) => e.path).join(', ')}); a win alongside a goalpost edit is gamed by declaration`;
701
850
  }
702
851
  }
703
- const baseGate = gateDefAtRef(repo, base);
704
852
  const weak = gateWeakenings(baseGate, cfg.metrics);
853
+ // the setup, env or linked dirs changed: every guard the base had is measured again the way it was measured before
854
+ if (measureChanged) for (const bm of baseGate.metrics.values()) {
855
+ if (bm.guard === true && !bm.report && bm.cmd !== PLACEHOLDER_TEST && !weak.some((w) => w.name === bm.name)) weak.push({ name: bm.name, change: 'measurement-edited', def: bm });
856
+ }
705
857
  const weakened = [];
706
858
  if (weak.length) {
707
859
  const defs = weak.map((w) => w.def);
708
- const b = measureAt(repo, base, { ...cfg, metrics: defs }, repeat);
709
- const a = measureAt(repo, head, { ...cfg, metrics: defs }, repeat);
860
+ const b = measureAt(repo, base, { ...baseRun, metrics: defs }, repeat);
861
+ const a = measureAt(repo, head, { ...baseRun, metrics: defs }, repeat);
862
+ const evs = new Map();
710
863
  for (const w of weak) {
711
864
  const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
712
- if (!ev.ok) weakened.push({ name: w.name, change: w.change, status: ev.status });
865
+ evs.set(w.name, ev);
866
+ // an edited guard that still fails under its old definition was turned green by the edit, not the code
867
+ let hidden = !ev.ok || (w.change === 'guard-edited' && ev.inert);
868
+ // a guard measured another way counts only when the new measurement hides it: it fails the old way and reads fine the new way
869
+ if (w.change === 'measurement-edited') { const hm = metrics.find((x) => x.name === w.name); hidden = (!ev.ok || !!ev.inert) && !!hm && hm.ok && !hm.inert; }
870
+ if (hidden) weakened.push({ name: w.name, change: w.change, status: ev.inert ? 'failed' : ev.status });
871
+ }
872
+ // a win under the old definition still has to be earned in the source: swapping or renaming the guard in the same
873
+ // change does not skip the source-only re-run
874
+ const won = weak.filter((w) => w.def.gamingCheck !== false && !weakened.some((x) => x.name === w.name) && evs.get(w.name).status === 'improved');
875
+ if (won.length) {
876
+ const src = measureSourceOnlyAll(repo, base, head, won.map((w) => w.def), baseRun, repeat);
877
+ for (const w of won) {
878
+ const j = judgeGaming({ name: w.name, direction: w.def.direction || 'up', guard: true, ...evs.get(w.name) }, src[w.name], w.def.gamingThreshold ?? (bs && bs.gamingThreshold) ?? 0.5);
879
+ if (j.gamed) weakened.push({ name: w.name, change: w.change, status: 'gamed' });
880
+ }
713
881
  }
714
882
  }
715
- const fpBase = baseGate ? gateFingerprint([...baseGate.metrics.values()], baseGate.scalars) : null;
883
+ // the fingerprint is of a config file: `base` stays null when there was none there (the zero-config gate is not a file)
884
+ const fpBase = baseGate && !baseGate.zeroConfig ? gateFingerprint([...baseGate.metrics.values()], bs) : null;
716
885
  const fpHead = gateFingerprint(cfg.metrics, cfg);
717
886
  const info = {
718
887
  fingerprint: { base: fpBase, head: fpHead },
@@ -769,17 +938,28 @@ function gate(repo, opts) {
769
938
  // a guard this change adds has no value at base, because its scanner did not exist there. That is adoption,
770
939
  // not a failure: it starts counting from head. Removing a guard is still caught (gateWeakenings), and a guard the
771
940
  // base contract already had still fails when it cannot be measured. A zero-config run has no contract to adopt.
772
- const baseDefs = opts.config ? null : gateDefAtRef(repo, base);
941
+ const fileGate = gateDefAtRef(repo, base);
942
+ const baseDefs = opts.config ? null : fileGate;
773
943
  const adopted = (name) => !opts.config && !(baseDefs && baseDefs.metrics.has(name));
944
+ // the gate in force before this change: the config committed at base, or, when this change adds the
945
+ // first config, the zero-config gate the repo ran under until now
946
+ const baseGate = opts.detectGaming ? (fileGate || (opts.config ? null : zeroConfigGate(repo, base))) : null;
774
947
  const metrics = cfg.metrics.map((m) => {
775
948
  const ev = evaluate(m, before[m.name], after[m.name], repeat);
776
949
  if (ev.status === 'unmeasurable' && ev.before == null && ev.after != null && adopted(m.name)) { ev.ok = true; ev.adopted = true; }
777
950
  // a `report` guard tells you and does not fail: a contract can start by watching and tighten later
778
951
  if (m.report && !ev.ok && ev.status === 'regressed') { ev.ok = true; ev.noted = true; }
779
- return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(m.gamingCheck === false ? { tripwire: true } : {}), ...ev };
952
+ // an inert guard: did its command run and fail, or never run? `ran` says where it ran (absent: at neither ref)
953
+ if (ev.inert && m.cmd !== PLACEHOLDER_TEST) {
954
+ const nb = neverRan(before[m.name]), na = neverRan(after[m.name]);
955
+ if (!nb || !na) ev.ran = !nb && !na ? 'both' : nb ? 'head' : 'base';
956
+ const note = failNotes.get(after[m.name]) || failNotes.get(before[m.name]); // a hint: a missing dependency or tests that fail
957
+ if (note) Object.assign(ev, note);
958
+ }
959
+ return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(gamingCheckOff(m, baseGate) ? { tripwire: true } : {}), ...ev };
780
960
  });
781
961
  // Section 3 (gaming detection), on by default: mutates `metrics` and yields the gate record
782
- const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat) : null;
962
+ const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) : null;
783
963
  // declared override: a Promptwheel-Allow trailer (or --allow) acknowledges an intentional
784
964
  // goalpost/gate change, so a would-be GAMED becomes PASS, recorded in report.allowed for review.
785
965
  // It covers an `unprovable` win too (the source-only re-run could not be applied): the declared
@@ -820,7 +1000,7 @@ function gate(repo, opts) {
820
1000
  for (const m of ga) { m.ok = true; m.allowed = true; allowed.cleared.push(m.name); }
821
1001
  } else for (const m of ga) m.allowable = true; // failed, and a declared reason would clear it: say so in the report
822
1002
  }
823
- // mandates run first: a protected-path hard rule fails the gate no matter what the metrics say.
1003
+ // mandates: a protected-path hard rule fails the gate no matter what the metrics say.
824
1004
  // The base ref's mandates count too (resolved through `extends` by gateDefAtRef), so a change
825
1005
  // cannot clear a mandate by deleting it in the same diff that violates it.
826
1006
  const baseMandates = (gateInfo && gateInfo.baseMandates) || (!opts.detectGaming ? ((gateDefAtRef(repo, base) || {}).scalars || {}).mandates : null);
@@ -846,6 +1026,9 @@ function gate(repo, opts) {
846
1026
  ...(gateInfo ? { gate: gateInfo } : {}),
847
1027
  ...(allowed ? { allowed } : {}),
848
1028
  };
1029
+ // a setup command that failed, at the ref it failed at: the usual reason a metric could not be measured
1030
+ const sb = setupFails.get(before), sa = setupFails.get(after);
1031
+ if (sb || sa) report.setup = { ...(sb ? { base: sb } : {}), ...(sa ? { head: sa } : {}) };
849
1032
  // provenance receipt: makes the verdict tamper-evident + reproducible ("re-run and get the same
850
1033
  // verdict is checkable"). Additive to the record; never changes the verdict.
851
1034
  report.provenance = {
@@ -880,7 +1063,7 @@ const PRESETS = {
880
1063
  'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
881
1064
  metrics: [
882
1065
  { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
883
- { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
1066
+ { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
884
1067
  { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
885
1068
  { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|# ?nosec|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
886
1069
  { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
@@ -906,7 +1089,7 @@ const PRESETS = {
906
1089
  { name: 'codeowners-frozen', paths: ['CODEOWNERS'], reason: 'review ownership is set by people' },
907
1090
  { name: 'scanner-frozen', paths: ['.promptwheel/packs/**'], reason: 'the scanners that read the contract are not edited by the change they read' },
908
1091
  { name: 'lock-frozen', paths: ['promptwheel.lock.json'], allowable: 'flag', reason: 'the architecture lock changes with a reason from a person (--allow), never as a side effect of a change' },
909
- { name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], reason: 'the hooks that enforce the contract are wired by a person, not detached by the agent' },
1092
+ { name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], allowable: 'flag', reason: 'the hooks that enforce the contract change with a reason from a person (--allow), and are never detached by the agent' },
910
1093
  ],
911
1094
  metrics: [
912
1095
  { name: 'lifecycle_scripts', cmd: 'node packs/supply-chain/scan.mjs lifecycle .', extract: 'number', direction: 'down', guard: true, gamingCheck: false, allowable: 'flag' },
@@ -919,17 +1102,17 @@ const PRESETS = {
919
1102
  ] },
920
1103
  };
921
1104
 
922
- function detectTestCmd(repo) {
923
- const has = (f) => existsSync(join(repo, f));
1105
+ function detectTestCmd(repo, { has, read } = stackAt(repo)) {
924
1106
  if (has('go.mod')) return 'go test ./...';
925
1107
  if (has('Cargo.toml')) return 'cargo test';
926
- if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return has('.venv/bin/pytest') ? '.venv/bin/pytest -q' : 'pytest -q';
1108
+ // the venv is not tracked: it is looked for on disk whichever ref the stack is read from
1109
+ if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return existsSync(join(repo, '.venv/bin/pytest')) ? '.venv/bin/pytest -q' : 'pytest -q';
927
1110
  // only trust `npm test` when a test script actually exists, otherwise the metric can
928
1111
  // never pass and the gate would "protect" nothing (very common in Next.js app repos)
929
1112
  try {
930
- if (JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8')).scripts?.test) return 'npm test --silent';
1113
+ if (JSON.parse(read('package.json')).scripts?.test) return 'npm test --silent';
931
1114
  } catch { /* no or unparsable package.json, fall through */ }
932
- return 'echo "set your test command in promptwheel.config.json" && false';
1115
+ return PLACEHOLDER_TEST;
933
1116
  }
934
1117
 
935
1118
  // deps that live outside git + the env a test needs, defaulted per stack. init writes these into
@@ -938,18 +1121,16 @@ function detectTestCmd(repo) {
938
1121
  // where the detected test command cannot run without a build first (a 10-repo replay put this at
939
1122
  // ~2/3 of gates). Written by `init` into `setup`; a failed setup stays honest (inert -> inconclusive,
940
1123
  // never a fake green). Kept out of the zero-config default so the fast per-turn path stays fast.
941
- function detectSetup(repo) {
942
- const has = (f) => existsSync(join(repo, f));
1124
+ function detectSetup(repo, { has, read } = stackAt(repo)) {
943
1125
  try {
944
- const pkg = JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8'));
1126
+ const pkg = JSON.parse(read('package.json'));
945
1127
  if (pkg.scripts && pkg.scripts.build) return 'npm run build --if-present'; // TS/bundler tests that run against a built dir
946
1128
  } catch { /* no or unparsable package.json */ }
947
1129
  if (has('pyproject.toml') || has('setup.py')) return 'pip install -e . --quiet 2>/dev/null || true'; // editable install so imports resolve the measured ref
948
1130
  return undefined;
949
1131
  }
950
1132
 
951
- function detectEnv(repo) {
952
- const has = (f) => existsSync(join(repo, f));
1133
+ function detectEnv(repo, { has } = stackAt(repo)) {
953
1134
  // Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
954
1135
  // resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
955
1136
  if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
@@ -1064,6 +1245,7 @@ function writeAgentBlock(repo) {
1064
1245
 
1065
1246
  const short = (repo, ref) => { try { return git(['rev-parse', '--short', ref], repo); } catch { return ref; } };
1066
1247
 
1248
+ const wouldHave = (status) => (status === 'unmeasurable' ? 'been unmeasurable' : status);
1067
1249
  function printHuman(r) {
1068
1250
  const arrowFor = (m) => (m.delta == null ? '?' : m.delta > 0 ? '▲' : m.delta < 0 ? '▼' : '=');
1069
1251
  console.log(`\nPromptWheel ${r.base} → ${r.head}${r.repeat > 1 ? ` (×${r.repeat})` : ''}\n`);
@@ -1072,15 +1254,23 @@ function printHuman(r) {
1072
1254
  const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
1073
1255
  const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
1074
1256
  console.log(` ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)} (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
1075
- if (m.inert) console.log(` ⚠ "${m.name}" never ran (same value at both refs): its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
1257
+ if (m.inert && m.ran && m.cause === 'missing-dependency') console.log(` ⚠ "${m.name}" could not run in the clean copy at either ref: ${m.detail}. Link or install what it needs (linkDirs or setup in promptwheel.config.json), then check again`);
1258
+ else if (m.inert && m.ran === 'both') console.log(` ⚠ "${m.name}" ran and failed before and after this change, so it guards nothing here. Fix the failing tests, or the command, before trusting a pass`);
1259
+ else if (m.inert) console.log(` ⚠ "${m.name}" ${m.ran ? `ran and failed at the ${m.ran} ref and did not run at the other` : 'never ran (same value at both refs)'}: its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
1076
1260
  if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
1077
1261
  }
1078
- if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : 'disabled'} in this change and it would have ${w.status} at head (goalpost moved, not earned)`);
1079
- if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath or mandate was added, removed or edited; review promptwheel.config.json as part of the diff`);
1262
+ if (r.setup) { const at = Object.keys(r.setup), f = r.setup[at[0]]; console.log(` ⚠ setup failed (exit ${f.exit}) at the ${at.join(' and ')} ref${at.length > 1 ? 's' : ''}${f.lastLine ? ': ' + f.lastLine : ''}`); }
1263
+ if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(w.change === 'measurement-edited'
1264
+ ? ` 🚩 GATE WEAKENED, how the guard "${w.name}" is measured (setup, env or linked dirs) was edited in this change, and measured the way it was before it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`
1265
+ : ` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : 'disabled'} in this change and it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`);
1266
+ if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited; review promptwheel.config.json as part of the diff`);
1080
1267
  const failReason = r.mandates ? ', a mandate was violated (a protected path changed)' : ', a guarded metric regressed (beyond noise)';
1081
1268
  console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? failReason : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
1082
- if (r.verdict === 'fail' && r.mandates) console.log(' a mandate is a hard rule; if the change is truly intended, edit the mandate in promptwheel.config.json (a tracked, tamper-evident change), do not work around it');
1083
- else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
1269
+ if (r.verdict === 'fail' && r.mandates) {
1270
+ const takes = r.mandates.filter((md) => md.allowable).map((md) => `"${md.name}"`);
1271
+ if (takes.length) console.log(` intentional? ${takes.join(', ')} ${takes.length > 1 ? 'accept' : 'accepts'} a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record`);
1272
+ if (takes.length < r.mandates.length) console.log(` ${takes.length ? 'any other' : 'a'} mandate is a hard rule; if the change is truly intended, retire the mandate in promptwheel.config.json in a change of its own first (a tracked, tamper-evident change), do not work around it`);
1273
+ } else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
1084
1274
  else if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name)');
1085
1275
  if (r.allowed) console.log(` ✓ allowed: ${r.allowed.cleared.join(', ')} acknowledged as an intentional change, recorded (Promptwheel-Allow: ${r.allowed.reason})`);
1086
1276
  console.log('');
@@ -1134,9 +1324,10 @@ function renderMarkdown(r) {
1134
1324
  '|---|--:|--:|--:|---|---|',
1135
1325
  rows,
1136
1326
  '',
1137
- r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule, enforced before any metric.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
1138
- ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by dropping a guard, not by earning it.`] : []),
1139
- ...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath or mandate was added, removed or edited. Review the config change with the code.`] : []),
1327
+ r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule; no metric result clears it.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? (r.metrics.some((m) => m.inert) && r.metrics.every((m) => !m.inert || m.ran === 'both') ? '> 🟡 A 🛡️ guarded check is **inert**, it fails before and after this change, so nothing was actually verified. Fix it before trusting a pass.' : '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.') : '> ✅ No guarded metric regressed.',
1328
+ ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : w.change === 'measurement-edited' ? 'measured another way (setup, env or linked dirs)' : 'disabled'} (would have ${wouldHave(w.status)} at head)`).join('; ')}. The pass was bought by changing the gate, not by earning it.`] : []),
1329
+ ...(r.allowed ? ['', `> ✓ **Allowed:** \`${r.allowed.cleared.join('`, `')}\` cleared by a declared reason: ${r.allowed.reason}`] : []),
1330
+ ...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited. Review the config change with the code.`] : []),
1140
1331
  '',
1141
1332
  '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
1142
1333
  ].join('\n');
@@ -1169,7 +1360,7 @@ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluat
1169
1360
  // Importing this file never runs the CLI (see the isMain guard at the bottom).
1170
1361
  // Stable across 0.x minor versions. The schema check() returns and the exit codes
1171
1362
  // verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
1172
- // shell, import, MCP) are in INTEGRATE.md.
1363
+ // shell, import) are in INTEGRATE.md.
1173
1364
  // ─────────────────────────────────────────────────────────────────────────────
1174
1365
 
1175
1366
  // The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
@@ -1207,8 +1398,8 @@ export function verdictExitCode(verdict) {
1207
1398
  // detectGaming boolean run the Section 3 source-only re-proof (default: true)
1208
1399
  // allow string a declared override reason (same as --allow / a Promptwheel-Allow: trailer)
1209
1400
  export function check(options = {}) {
1210
- // `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config lives
1211
- // and where .promptwheel/ is written), exactly as the CLI does from the same directory
1401
+ // `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
1402
+ // lives), exactly as the CLI does from the same directory
1212
1403
  let start = process.cwd();
1213
1404
  if (options.repo) { try { start = realpathSync(options.repo); } catch { bad(`promptwheel: not inside a git repository (${options.repo})`); } }
1214
1405
  const repo = repoRoot(start);
@@ -1328,17 +1519,35 @@ const GATE_FLAGS = ['--base', '--head', '--repeat', '--working', '--no-record',
1328
1519
  const COMMAND_FLAGS = {
1329
1520
  run: GATE_FLAGS, init: ['--preset', '--list', '--force', '--no-agent-block'], lock: [],
1330
1521
  };
1331
- const VALUED_FLAGS = new Set(['--base', '--head', '--repeat', '--allow', '--preset']);
1522
+ // what each valued flag needs. A missing or unusable value is a usage error too: `--repeat abc` must not
1523
+ // measure nothing and read FAIL, and `--allow` with nothing after it must not run as if no reason was asked for.
1524
+ const VALUED_FLAGS = { '--base': 'a ref', '--head': 'a ref', '--repeat': 'a whole number, 1 or more', '--allow': 'a reason, e.g. --allow "the spec changed"', '--preset': 'a preset name (see promptwheel init --list)' };
1332
1525
  function checkFlags(cmd, argv) {
1333
1526
  const known = new Set(COMMAND_FLAGS[cmd]);
1334
1527
  for (let i = 0; i < argv.length; i++) {
1335
1528
  const a = argv[i];
1336
1529
  if (!a.startsWith('-')) continue;
1337
1530
  if (!known.has(a)) bad(`promptwheel ${cmd}: unknown flag ${a} (see promptwheel --help)`);
1338
- if (VALUED_FLAGS.has(a)) i++;
1531
+ if (!(a in VALUED_FLAGS)) continue;
1532
+ const v = argv[++i];
1533
+ const gone = v === undefined || known.has(v); // nothing after the flag, or the next flag
1534
+ // an empty --base or --head is the default ref: the Action passes an empty base outside a pull request
1535
+ const blank = !gone && !v.trim() && !(v === '' && (a === '--base' || a === '--head'));
1536
+ if (gone || blank) bad(`promptwheel ${cmd}: ${a} needs ${VALUED_FLAGS[a]}`);
1537
+ if (a === '--repeat' && !/^[1-9]\d*$/.test(v)) bad(`promptwheel ${cmd}: --repeat needs ${VALUED_FLAGS[a]} (got "${v}")`);
1339
1538
  }
1340
1539
  }
1341
1540
 
1541
+ // read PR comments on stdin: a JSON array, or one comment per line as `gh api --paginate --jq '.[]'` prints every
1542
+ // page. Streamed: readFileSync(0) corrupted the heap on Node 24.22 reading a live paginated gh pipe.
1543
+ async function prAllow() {
1544
+ let t = ''; process.stdin.setEncoding('utf8');
1545
+ for await (const chunk of process.stdin) t += chunk;
1546
+ t = t.trim(); let c = [];
1547
+ try { c = [].concat(JSON.parse(t || '[]')); } catch { try { c = t.split('\n').filter(Boolean).map((l) => JSON.parse(l)); } catch { /* unreadable, no approval */ } }
1548
+ process.stdout.write(allowFromComments(c));
1549
+ }
1550
+
1342
1551
  // write or refresh the architecture lock from the tree as it stands. A person runs this: the lock file is
1343
1552
  // a mandate, so the refreshed file passes the gate only with --allow "<reason>".
1344
1553
  async function lock() {
@@ -1353,9 +1562,9 @@ function dispatch() {
1353
1562
  const cmd = (rest.includes('--help') || rest.includes('-h')) ? 'help' : first; // `promptwheel run --help` is a help request, not a gate run
1354
1563
  if (cmd in COMMAND_FLAGS) checkFlags(cmd, rest);
1355
1564
  if (cmd === 'run') run(rest);
1356
- else if (cmd === 'pr-allow') { let c = []; try { c = JSON.parse(readFileSync(0, 'utf8') || '[]'); } catch { /* no comments, no approval */ } process.stdout.write(allowFromComments(c)); }
1565
+ else if (cmd === 'pr-allow') return prAllow();
1357
1566
  else if (cmd === 'init') init(rest);
1358
- else if (cmd === 'lock') lock(rest);
1567
+ else if (cmd === 'lock') return lock(rest);
1359
1568
  else {
1360
1569
  const help = !cmd || cmd === '--help' || cmd === '-h' || cmd === 'help';
1361
1570
  if (!help) console.error(`unknown command: ${cmd}`);
@@ -1370,10 +1579,11 @@ function dispatch() {
1370
1579
  ' catches reward-hacking BY DEFAULT, re-proves each win from SOURCE edits alone (verdict GAMED,',
1371
1580
  ' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
1372
1581
  ' promptwheel run --working gate uncommitted changes (incl. newly added files)',
1582
+ ' promptwheel pr-allow < comments.json print the newest `sawbuck-allow: <reason>` from a PR comment by someone with write access (the Action uses it)',
1373
1583
  ' promptwheel run --allow "<reason>" acknowledge an intentional test/config change (or a `Promptwheel-Allow: <reason>` commit trailer; `Sawbuck-Allow:` is an alias): recorded, not flagged GAMED',
1374
1584
  '',
1375
1585
  'Exit codes: 0 pass · 1 fail (a guarded metric regressed, or a mandate was violated) · 2 gamed · 3 inconclusive · 64 usage or config error (never a verdict).',
1376
- 'Integrate (library / conform / shell / MCP): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
1586
+ 'Integrate (library / conform / shell): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
1377
1587
  'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
1378
1588
  ' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
1379
1589
  ].join('\n'));
@@ -1384,8 +1594,8 @@ function dispatch() {
1384
1594
  // run the CLI only when invoked directly (resolves symlinks so the npm bin still works)
1385
1595
  let isMain = false;
1386
1596
  try { isMain = !!process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); } catch { /* not main */ }
1387
- function main() {
1388
- try { dispatch(); }
1597
+ async function main() {
1598
+ try { await dispatch(); }
1389
1599
  catch (e) { if (e && e.name === 'ConfigError') die(e.message); throw e; }
1390
1600
  }
1391
1601
  if (isMain) main();
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "promptwheel",
3
- "version": "0.8.0",
3
+ "version": "0.8.2",
4
4
  "description": "A deterministic, zero-LLM check for your coding loop. A win must survive a re-run from the agent's source edits alone, so a test that went green by editing the test reads GAMED. Runs in CI or a Claude Code plugin.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -27,7 +27,7 @@ clean git worktrees, so a pack that is not in git is absent at both refs and the
27
27
  count. A private registry mirror makes this baseline large, not failing.
28
28
  - **unpinned_execution**: code run from a reference that can move. A GitHub Action not pinned to a full commit SHA (a local `./action` does not count), `npx`, `bunx` or `dlx` with no exact version, `npm install -g` with no exact version, and a remote script piped into a shell. Read from workflows, `action.yml`, package.json scripts, `scripts/*.sh` and Dockerfiles. In package.json scripts, a tool the manifest declares does not count: the lockfile already pins it.
29
29
  - **install_policy_gaps**: 0, 1 or 2. Are install scripts disabled, and is a release-age cooldown set (`ignore-scripts=true` and `min-release-age` in `.npmrc`, or the pnpm or yarn equivalents)? The cooldown is enforced by the package manager, not by this pack: a registry lookup here would make the verdict depend on the clock.
30
- - **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is reported but not held by default; add `--routes` to hold it. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
30
+ - **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is locked but not counted by default (`node .promptwheel/packs/supply-chain/lock.mjs drift . --list` shows it). To hold new routes too, add `--routes` to the end of the `lock_drift` metric's `cmd` in `promptwheel.config.json`, so it reads `node .promptwheel/packs/supply-chain/lock.mjs drift . --routes`. The flag belongs to that script: `promptwheel lock` takes no flags and always writes routes into the lock. The config is a frozen path here, so that edit passes with `promptwheel run --allow "<reason>"`. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
31
31
  - **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%, dependency changes; use the CI path below for those.
32
32
 
33
33
  **On a pull request.** Someone with write access comments `sawbuck-allow: <reason>` and re-runs the check. `sawbuck contract` marks everyday guards `report: true`, so they note a change without failing until you tighten them; this preset on its own holds them all.
@@ -57,10 +57,10 @@ Sometimes adding a dependency is exactly what was asked. Say so, and the gate re
57
57
  promptwheel run --working --allow "needed for the CSV export (papaparse)"
58
58
  ```
59
59
 
60
- or a `Promptwheel-Allow: <reason>` trailer on the commit. The seven metrics carry `allowable: true`,
61
- which means a declared reason clears their guarded regression for that one run, with the reason
62
- written into `report.allowed` and the outcome record. Nothing is loosened in the config, and a rise
63
- without a reason still fails. That is the point: every added package, script, or non-registry
60
+ A commit trailer does not count here. The seven metrics carry `allowable: "flag"`, which means a
61
+ reason from the `--allow` flag clears their guarded regression for that one run, with the reason
62
+ written into `report.allowed`. Nothing is loosened in the config, and a rise without a reason still
63
+ fails. That is the point: every added package, script, or non-registry
64
64
  source leaves an audit line a reviewer can read.
65
65
 
66
66
  `gamingCheck` is `false` on all seven because these numbers legitimately live in config files
@@ -69,8 +69,8 @@ flag every honest dependency bump.
69
69
 
70
70
  ## The mandates the preset writes
71
71
 
72
- A mandate is a path the agent must not touch in a change; the gate fails before any metric is read,
73
- and a mandate deleted or narrowed in the same change that violates it stays in force.
72
+ A mandate is a path the agent must not touch in a change. Touching one fails the gate whatever the
73
+ metrics say, and a mandate deleted or narrowed in the same change that violates it stays in force.
74
74
 
75
75
  | mandate | paths | why |
76
76
  | --- | --- | --- |
@@ -79,15 +79,20 @@ and a mandate deleted or narrowed in the same change that violates it stays in f
79
79
  | build-images-frozen | `Dockerfile*`, `docker-compose*` | what runs at build and deploy time |
80
80
  | publish-scripts-frozen | `scripts/release*`, `scripts/publish*` | the publish path is how a compromise spreads |
81
81
  | codeowners-frozen | `CODEOWNERS` | review ownership is set by people |
82
+ | scanner-frozen | `.promptwheel/packs/**` | the scanners are not edited by the change they read |
82
83
  | contract-frozen | `promptwheel.config.json`, `sawbuck.config.json` | the contract changes with a reason from a person (`--allow`) |
83
- | hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks that enforce the contract are wired by a person |
84
+ | lock-frozen | `promptwheel.lock.json` | the lock changes with a reason from a person (`--allow`) |
85
+ | hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks change with a reason from a person (`--allow`) |
84
86
 
85
87
  The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is frozen too, and it is also an
86
88
  `evaluatorPaths` entry. A change to it needs `--allow "<reason>"` from a person. Without that, a change that
87
89
  only weakens a rule would pass, and the next change would be judged by the weakened contract.
88
90
 
89
- A person who needs to change a frozen path does it in a separate change that only touches that
90
- path, reviewed as such.
91
+ `--allow` clears only three of these: `contract-frozen`, `lock-frozen` and `hooks-frozen` (so a team can commit
92
+ shared hooks to a repo that already has its rules). A commit trailer clears none. Every other frozen path stays
93
+ frozen, even with `--allow`. To change one, a person first retires its mandate in a change of its own
94
+ (it edits the contract, so it needs `--allow`), then makes the change, then puts the mandate back the
95
+ same way.
91
96
 
92
97
  ## What this is not
93
98
 
@@ -100,5 +105,4 @@ path, reviewed as such.
100
105
  - **Not a scanner.** No matchers, no findings, no false positives to triage: a path, a count, or a
101
106
  hash.
102
107
 
103
- Pair it with the `security` preset for the eight code-level invariants, and with the GitHub Action so
104
- the gate runs where the agent cannot switch it off.
108
+ Pair it with the GitHub Action so the gate runs where the agent cannot switch it off.
@@ -1,5 +1,5 @@
1
1
  {
2
- "$comment": "PromptWheel supply-chain guard-pack. Use via { \"extends\": [\"packs/supply-chain/pack.json\"] } in your promptwheel.config.json, or install it with `promptwheel init --preset supply-chain` (which also writes the mandates). Three counts the gate holds: a rise fails the gate unless the change carries a declared reason (`--allow \"<reason>\"` or a `Promptwheel-Allow:` trailer), which is recorded in the outcome record. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
2
+ "$comment": "PromptWheel supply-chain guard-pack. Install it with `promptwheel init --preset supply-chain`: that copies the pack to .promptwheel/packs/supply-chain/, writes these metrics into promptwheel.config.json with their commands pointed at the copy, and writes the mandates. The commands in this file run packs/supply-chain/..., which exists only in the PromptWheel repository itself, so `extends` on this file (here or on the copy under .promptwheel/packs/) reads every metric as unmeasurable, and the gate fails, in any other repository. Seven counts the gate holds: a rise fails the gate unless a person gives a reason with `--allow \"<reason>\"` (a commit trailer does not count), which is recorded in report.allowed. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
3
3
  "metrics": [
4
4
  {
5
5
  "$comment": "install-time script entries (preinstall, install, postinstall, prepare, prepublish, prepublishOnly, postpack) across every package.json: the step a supply-chain attack needs inside your repo",