promptwheel 0.8.0 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/INTEGRATE.md CHANGED
@@ -50,7 +50,7 @@ importing this package.
50
50
  "gamed": true,
51
51
  "sourceOnly": null,
52
52
  "retained": 0.0,
53
- "gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
53
+ "gamingReason": "the \"win\" changed zero production-source files, only test/config/grader/golden"
54
54
  }
55
55
  ]
56
56
  }
@@ -68,11 +68,18 @@ metric's gain that survived the source-only re-run (Section 3). `confidence`
68
68
  (`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` are informational fields this
69
69
  implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
70
70
  truncated SHA-256 fingerprint of the gate definition (metrics, guards, `evaluatorPaths`,
71
- `gamingThreshold`, `mandates`) at base and head, sets `gate.changed: true` when they differ,
72
- and lists any guard the change dropped to hide a regression; a non-empty `gate.weakened`
73
- forces `gamed`, so an agent cannot pass by disabling the guard it is judged by.
74
-
75
- Self-check your implementation against this schema with `conformsToContract` (see Section 4 below).
71
+ `gamingThreshold`, `mandates`, `setup`, `env`, linked dirs; each metric with its `cmd` and how it
72
+ is read) at base and head, sets `gate.changed: true` when they differ, and lists in
73
+ `gate.weakened` any guard the change removed, disabled, or measured another way
74
+ (`change: guard-edited`, such as `cmd: "true"`; `change: measurement-edited`, a new `setup`, `env`
75
+ or `linkDirs`) to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
76
+ cannot pass by disabling the guard it is judged by. A change that adds the first config is
77
+ compared with the zero-config gate (the detected test command). `allowed` (optional) is `{ reason, cleared }` when a declared
78
+ reason turned a would-be `gamed`, `fail` or `inconclusive` into `pass`: `cleared` names each metric,
79
+ `gate`, or `mandate:<name>` it cleared, and each cleared metric carries `allowed: true` and keeps its
80
+ `gamingReason`, so a consumer can say what was allowed and why it would have failed.
81
+
82
+ Self-check your implementation against this schema with `conformsToContract` (see part 4 below).
76
83
 
77
84
  ---
78
85
 
@@ -99,8 +106,8 @@ measurement runs in a throwaway git worktree.
99
106
  ## 3. Import the library (in process)
100
107
 
101
108
  Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
102
- process and returns the Section 2 object. It never prints and never calls `process.exit`; it
103
- throws a plain `Error` on a precondition failure. Types ship in the package
109
+ process and returns the Section 2 object. It never prints and never calls `process.exit`; on a
110
+ precondition failure it throws an `Error` named `ConfigError` with `exitCode` 64. Types ship in the package
104
111
  (`bin/promptwheel.d.ts`, hand-written, no build step): `import type { Report, CheckOptions, Config } from 'promptwheel'`
105
112
  works in TypeScript, and editors complete the options and the report in plain JavaScript.
106
113
 
package/README.md CHANGED
@@ -23,8 +23,10 @@ printf "import {test} from 'node:test'; import assert from 'node:assert'; import
23
23
  git commit -qam 'make the suite green'
24
24
 
25
25
  npx -y promptwheel@latest run --base HEAD~1 --head HEAD
26
- # ▲ tests_pass 0 → 1 🚩 GAMED, the win changed zero production-source files
27
- # VERDICT: GAMED (exit 2)
26
+ # prints (shortened to the lines that matter; exit code 2):
27
+ # ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
28
+ # 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
29
+ # VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
28
30
  ```
29
31
 
30
32
  What happened: PromptWheel measured `tests_pass` at both commits in throwaway git worktrees (0 then 1, a win), then split the diff into production source versus `{test, config, grader, golden}` files, re-applied **only the source slice** on a clean worktree at the base, and re-ran the metric. The win did not survive, so it is `GAMED`. No LLM, no network: a diff partition plus a re-run, reproducible in seconds with a human-readable reason.
@@ -88,11 +90,11 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
88
90
  - **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
89
91
  - **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
90
92
  - **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
91
- - **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
92
- - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
93
+ - **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires. Set on a metric that had the re-run on, it takes effect from the next change: the change that sets it is still re-proven.
94
+ - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`. A change that lowers it is still judged by the value before it.
93
95
  - **evaluatorPaths**, globs for files that *define* your gate (a threshold file, a budget, a CI limit). Editing one in the same diff that improves a metric is `GAMED` by declaration, deterministically. Opt-in, because only you know which files are the goalposts.
94
96
 
95
- - **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. For counts where a rise is sometimes the job (the `supply-chain` preset: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
97
+ - **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. `"flag"` takes the reason from `--allow` (or `check({ allow })`) only, never a trailer, because an agent can write a trailer. For counts where a rise is sometimes the job (the `supply-chain` preset uses `"flag"`: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
96
98
 
97
99
  Dependency and environment keys (config-level; `init` writes stack defaults):
98
100
 
@@ -100,6 +102,8 @@ Dependency and environment keys (config-level; `init` writes stack defaults):
100
102
  - **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
101
103
  - **setup**, an optional command run once per ref before metrics (for example `npm run build` or `pip install -e .`), after any source patch, so a build-gated suite measures the ref it is actually on.
102
104
 
105
+ These three shape every measurement, so they are part of the gate. A change that edits them is flagged as a gate change and is judged with the values from before it (see Gate integrity below).
106
+
103
107
  Copy a config for your stack (pytest, tsc, coverage, bundle size, LLM eval) from [examples/](examples/).
104
108
 
105
109
  ### Guardrails and inheritance
@@ -116,7 +120,7 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
116
120
 
117
121
  ### Mandates (protected paths)
118
122
 
119
- Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate before a single metric is read**, however green everything else is:
123
+ Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate**, however green everything else is. The metrics still run and are reported, but nothing they say clears a mandate:
120
124
 
121
125
  ```json
122
126
  // promptwheel.config.json
@@ -126,7 +130,7 @@ Some rules are not a metric at all, they are a hard line: *never touch `src/auth
126
130
  "metrics": [ /* ... */ ] }
127
131
  ```
128
132
 
129
- A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the union of the base ref's and the head's, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`). Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
133
+ A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the ones the base ref already had, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`), and one that narrows a mandate is checked against both versions (`edited: true`). A mandate added in the same change that touches its path does not apply to that change, on purpose: the commit that signs a contract adds the very files the contract protects. Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
130
134
 
131
135
  ### Supply chain (preset `supply-chain`)
132
136
 
@@ -136,7 +140,7 @@ The step every npm supply-chain attack needs inside a target repo is a supply-ch
136
140
  npx promptwheel init --preset supply-chain
137
141
  ```
138
142
 
139
- writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, and the Claude Code hook files, and lists the contract itself (`promptwheel.config.json`, `sawbuck.config.json`) under `evaluatorPaths`, so a change that edits it alongside a win is gamed by declaration while a change that only edits the contract still passes. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
143
+ writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, the copied scanners, the Claude Code hook files, the lock file, and the contract itself (`promptwheel.config.json`, `sawbuck.config.json`). A change that edits the contract fails (exit 1) unless a person passes `--allow "<reason>"`; the contract is also an `evaluatorPaths` entry, so editing it alongside a win is gamed by declaration. `--allow` clears only the contract, the lock and the hooks. Every other frozen path needs its mandate retired first, by a person, in a change of its own run with `--allow`. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
140
144
 
141
145
  ## In CI and in Claude Code
142
146
 
@@ -170,16 +174,16 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
170
174
 
171
175
  ## Trust model
172
176
 
173
- **What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
177
+ **What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). A grader is recognized by its file name in a scripting language (`eval.py`, `grader.sh`, `score.js`, `scoring.ts`) or by an `evals/` or `grader(s)/` directory; a production file with one of those names (`src/score.py`) is treated as a grader too, so a fix there needs `--allow`, and a grader with any other name (`evaluate.py`, `check.py`) is source unless you list it in `evaluatorPaths`. On top of that:
174
178
 
175
179
  - **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
176
- - **Gate integrity.** The gate is resolved as of the base commit too. A change that disables a guard or deletes a guarded metric in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed. The gate definition (metrics, guards, `evaluatorPaths`, `gamingThreshold`, `mandates`) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed.
180
+ - **Gate integrity.** A change is judged by the gate that was in force before it. The gate is resolved as of the base commit too, through `extends`. A change that disables a guard, deletes a guarded metric, or measures it another way (a new `cmd` such as `true`, `extract`, `direction`, `exact` or `report`) in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed, or would still fail. If the old guard only passes because the same change also edited the test, it reads `GAMED` too: a win under the old definition still has to survive the source-only re-run. A file declared in `evaluatorPaths` before the change still counts as a goalpost for that change, even if the change drops the declaration. The same goes for `setup`, `env` and `linkDirs`: when a change edits them, each guard is measured again with the old values and so is the source-only re-run, so a `setup` that rewrites a test cannot carry a pass (`gate.weakened[].change: measurement-edited`). `gamingCheck: false` on a metric that had the re-run on, and a lower `gamingThreshold`, apply from the next change. A change that adds the first config is compared with the zero-config gate (the detected test command), so `tests_pass` set to `true` in a new config is an edited guard; a first config that keeps the detected command, or replaces the placeholder, is not. The gate definition (each metric with its command and how it is read, guards, `evaluatorPaths`, `gamingThreshold`, mandate names and paths, `setup`, `env`, linked dirs) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed. Two limits: a test that could not run at all before a `setup` fix reads `GAMED` on the change that adds the fix (say so with `--allow`; the `setup` that `init` writes into a first config is exempt), and a base config that `extends` a file outside the repository is not compared on `setup`, `env` or `linkDirs`.
177
181
  - **A sanctioned override.** A commit trailer can be written by whoever writes the commit, including an agent, so `"allowTrailer": false` in the contract turns trailers off and only the `--allow` flag counts (the supply-chain preset sets this; the contract at the base ref decides, so a change cannot turn trailers back on for itself). When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. It covers a win whose source-only re-run could not be applied (`unprovable`, otherwise `inconclusive`) the same way. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
178
182
  - **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
179
183
 
180
- **The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test): that needs held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
184
+ **The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test), or in a build script the unchanged `setup` runs (a change that makes that script rewrite a test is a source edit to this check). A base config that `extends` a file outside the repository cannot be compared across the change. These need held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
181
185
 
182
- **Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guard whose command never ran (same value at both refs) makes the verdict `inconclusive`, never a fake pass.
186
+ **Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guarded pass/fail check that reads 0 at both refs (its command never ran, or it ran and failed both times; the report says which) makes the verdict `inconclusive`, never a fake pass.
183
187
 
184
188
  **When not to use it.** If you only care that tests pass and nobody could be editing the tests, your plain CI already has you covered. PromptWheel earns its place when the thing writing the code can also touch the thing judging it, or when you have a graded metric beyond pass/fail (an eval score, cost per run, latency, size) that a change could quietly move. Single-metric CI gates (benchmark trackers, size limits) go deeper on one number; wrap them as a metric `cmd` and PromptWheel composes them into one verdict.
185
189
 
@@ -187,7 +191,7 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
187
191
 
188
192
  A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly 2 to 3× your metric command's runtime plus about 40ms of worktree overhead. Point the per-turn gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for CI: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
189
193
 
190
- **Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run cannot leave clutter behind: the next run self-heals any orphaned worktree.
194
+ **Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run can leave a worktree in your temp dir; a later run removes it once it is a day old (an unregistered checkout of this repo after an hour). It removes only directories named the way it names its own (`promptwheel-`, `promptwheel-src-` or `pw-idx-`, then an 8-hex repo tag, then 6 random characters). Any other folder or worktree in the temp dir is left alone, at any age.
191
195
 
192
196
  ## Portability
193
197
 
@@ -48,7 +48,7 @@ export interface MetricConfig {
48
48
  allowNonzero?: boolean;
49
49
  /** Per-command timeout, default 300. */
50
50
  timeoutSec?: number;
51
- /** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). */
51
+ /** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). Set on a metric that had the re-proof on, it applies from the next change, not the one that sets it. */
52
52
  gamingCheck?: boolean;
53
53
  /** Per-metric override of `Config.gamingThreshold`. */
54
54
  gamingThreshold?: number;
@@ -62,7 +62,7 @@ export interface MetricConfig {
62
62
  allowable?: boolean;
63
63
  }
64
64
 
65
- /** A named hard rule: any change touching one of `paths` fails the gate before any metric is read. */
65
+ /** A named hard rule: any change touching one of `paths` fails the gate, whatever the metrics read. */
66
66
  export interface Mandate {
67
67
  name: string;
68
68
  /** Globs: a bare name matches that basename at any depth, a pattern with `/` is anchored at the repo root, `**` crosses directories. */
@@ -85,7 +85,7 @@ export interface Config {
85
85
  env?: Record<string, string>;
86
86
  /** A per-ref build or install, run in the worktree before measuring (a failed setup leaves the metric inert, never a false green). */
87
87
  setup?: string;
88
- /** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. */
88
+ /** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. A change that lowers it is still judged by the value before it. */
89
89
  gamingThreshold?: number;
90
90
  /** Extra files that define the gate (a threshold, a budget); a change to one alongside a win is gamed by declaration. Same glob rules as `Mandate.paths`. */
91
91
  evaluatorPaths?: string[];
@@ -144,6 +144,8 @@ export interface MetricResult extends Evaluation {
144
144
  name: string;
145
145
  direction: Direction;
146
146
  guard: boolean;
147
+ /** With `inert`: where the command ran and failed (`both` refs, or only `base` or `head`). Absent when it ran at neither: exit 126 or 127, a timeout, or the placeholder command `init` writes. */
148
+ ran?: 'both' | 'base' | 'head';
147
149
  /** `true` when the win did not survive the source-only re-run; `null` when the source-only patch could not be applied (then `unprovable` is set and the verdict is `inconclusive`, never `pass`). Absent without gaming detection or when the metric did not improve. */
148
150
  gamed?: boolean | null;
149
151
  /** `true`: the source-only patch could not be applied, so this win could not be re-proven and the verdict is `inconclusive`. `false` only after a declared override cleared it (then `allowed` is `true`). */
@@ -174,9 +176,13 @@ export interface MandateViolation {
174
176
 
175
177
  export interface GateWeakening {
176
178
  name: string;
177
- change: 'guard-removed' | 'guard-disabled';
178
- /** What the dropped guard would have read at head under its base definition. */
179
- status: MetricStatus;
179
+ /**
180
+ * `guard-edited`: the guard is still on but measured another way (a new `cmd`, `extract`, `direction`, `exact` or `report`).
181
+ * `measurement-edited`: the config's `setup`, `env` or linked dirs changed, and measured with the ones before the change the guard fails at head while the new ones read it fine.
182
+ */
183
+ change: 'guard-removed' | 'guard-disabled' | 'guard-edited' | 'measurement-edited';
184
+ /** What the guard would have read at head under its base definition (and the base `setup`, `env` and linked dirs); `failed` when it still fails there. */
185
+ status: MetricStatus | 'failed';
180
186
  }
181
187
 
182
188
  export interface EvaluatorChange {
@@ -188,11 +194,11 @@ export interface EvaluatorChange {
188
194
 
189
195
  /** The gate-integrity record, present when gaming detection ran. */
190
196
  export interface GateRecord {
191
- /** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates) at each ref; `base` is `null` when there was no config at base. */
197
+ /** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) at each ref; `base` is `null` when there was no config at base. */
192
198
  fingerprint: { base: string | null; head: string };
193
199
  /** The gate definition differs between base and head. */
194
200
  changed?: true;
195
- /** Guards the change dropped that would have failed at head; non-empty forces `gamed` unless allowed. */
201
+ /** Guards the change dropped, edited or measures another way that would have failed at head; non-empty forces `gamed` unless allowed. A change that adds the first config is compared with the zero-config gate (the detected test command). */
196
202
  weakened?: GateWeakening[];
197
203
  /** Declared `evaluatorPaths` files whose content changed in this diff. */
198
204
  evaluatorChanged?: EvaluatorChange[];
@@ -260,7 +266,7 @@ export declare function deriveVerdict(metrics: ReadonlyArray<Partial<MetricResul
260
266
  * a missing `extends` target, no commits yet, a base or head ref that is not in this clone, no
261
267
  * commit to compare against) throws an Error whose `name` is `'ConfigError'` and whose `exitCode`
262
268
  * is 64, the code the CLI exits with for the same condition. A `repo` that names a subdirectory
263
- * is resolved to the repository root, where the config lives and the record is written.
269
+ * is resolved to the repository root, where the config lives. A run writes nothing to the repo.
264
270
  */
265
271
  export declare function check(options?: CheckOptions): Report;
266
272
 
@@ -14,7 +14,7 @@
14
14
 
15
15
  import { execSync, execFileSync } from 'node:child_process';
16
16
  import { createHash } from 'node:crypto';
17
- import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, statSync, copyFileSync } from 'node:fs';
17
+ import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, lstatSync, copyFileSync } from 'node:fs';
18
18
  import { tmpdir } from 'node:os';
19
19
  import { join, dirname, isAbsolute, relative } from 'node:path';
20
20
  import { fileURLToPath } from 'node:url';
@@ -36,6 +36,9 @@ const spread = (xs) => { const a = xs.filter((x) => x != null); return a.length
36
36
  // repo is still measuring in (a long gate elsewhere must not lose its worktree mid-run)
37
37
  function tmpOwnerTag(repo) { let r = repo; try { r = realpathSync(repo); } catch { /* keep as given */ } return createHash('sha1').update(String(r)).digest('hex').slice(0, 8); }
38
38
  const tmpPrefix = (repo, kind) => `${kind}-${tmpOwnerTag(repo)}-`;
39
+ // exactly the names mkdtemp makes from tmpPrefix: the kind, the 8-hex owner tag, then mkdtemp's 6 random
40
+ // characters. Self-heal removes nothing else, so a folder that only starts with "promptwheel-" is never ours.
41
+ const ENGINE_TMP = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-[A-Za-z0-9]{6}$/;
39
42
 
40
43
  // usage / config errors exit 64 (EX_USAGE), so they can never be confused with a verdict
41
44
  // (0 pass, 1 fail, 2 gamed, 3 inconclusive): in CI a config typo must not read as GAMED.
@@ -112,8 +115,14 @@ const hasConfigFile = (repo) => existsSync(join(repo, 'promptwheel.config.json')
112
115
  // measurement actually runs in a throwaway worktree. Lets `run` and `check()` work with no config
113
116
  // file; `init` is for customizing. A repo with no test command gets a sentinel that reads inert ->
114
117
  // INCONCLUSIVE (never a fake green), with a message pointing at the fix.
115
- function autoConfig(repo) {
116
- return { ...detectEnv(repo), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }] };
118
+ function autoConfig(repo, at = stackAt(repo)) {
119
+ return { ...detectEnv(repo, at), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo, at), extract: 'exit', direction: 'pass', guard: true }] };
120
+ }
121
+ // where stack detection looks: the working tree, or the tracked files as of `ref` (the repo as it was before a change)
122
+ function stackAt(repo, ref) {
123
+ if (!ref) return { has: (f) => existsSync(join(repo, f)), read: (f) => readFileSync(join(repo, f), 'utf8') };
124
+ const read = (f) => execFileSync('git', ['show', `${ref}:${f}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] });
125
+ return { has: (f) => { try { read(f); return true; } catch { return false; } }, read };
117
126
  }
118
127
 
119
128
  function resolveBase(repo, base) {
@@ -123,7 +132,8 @@ function resolveBase(repo, base) {
123
132
  catch { bad('promptwheel: no commits yet, make an initial commit before gating'); }
124
133
  for (const b of ['origin/main', 'origin/master', 'main', 'master']) {
125
134
  let mb = '';
126
- try { mb = git(['merge-base', 'HEAD', b], repo); } catch { continue; }
135
+ // stderr ignored: a ref that is not there (no origin/main in this clone) is expected, not an error to print
136
+ try { mb = execFileSync('git', ['merge-base', 'HEAD', b], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } catch { continue; }
127
137
  // on the trunk itself the merge-base IS HEAD; comparing a commit to itself would be a
128
138
  // trivial PASS that verified nothing, so fall through to the last commit instead.
129
139
  if (mb && mb !== head) return mb;
@@ -132,15 +142,21 @@ function resolveBase(repo, base) {
132
142
  catch { bad('promptwheel: no commit to compare against (one commit, or a shallow clone): pass --base, or --working'); }
133
143
  }
134
144
 
135
- function runMetric(cwd, m, env) {
136
- let stdout = '', code = 0;
145
+ // exit statuses of a sample list, kept beside it: 127 (the shell found no such command), 126 (it could not
146
+ // execute it) or no status (killed: a timeout) mean the metric never ran, which needs a different fix from a
147
+ // test that ran and failed
148
+ const exitCodes = new WeakMap();
149
+ const neverRan = (samples) => { const c = exitCodes.get(samples) || []; return c.length > 0 && c.every((x) => x == null || x === 126 || x === 127); };
150
+ function runMetric(cwd, m, env, codes) {
151
+ let stdout = '', code = 0, status = 0;
137
152
  // Never leak the harness's own node:test context into the measured gate: a metric like
138
153
  // `node --test` would see NODE_TEST_CONTEXT, act as a child reporter, and report a bogus
139
154
  // exit code (so the gate can't be trusted when PromptWheel itself runs under a test runner).
140
155
  const childEnv = { ...(env || process.env) }; delete childEnv.NODE_TEST_CONTEXT;
141
156
  try {
142
157
  stdout = execSync(m.cmd, { cwd, env: childEnv, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
143
- } catch (e) { code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
158
+ } catch (e) { status = e.status ?? null; code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
159
+ if (codes) codes.push(status);
144
160
  return extract(stdout, code, m.extract, m.allowNonzero);
145
161
  }
146
162
 
@@ -169,9 +185,9 @@ function extract(stdout, code, mode, allowNonzero) {
169
185
  // with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
170
186
  // Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
171
187
  // not here, the engine stays ecosystem-agnostic.
188
+ const linkDirsOf = (cfg) => (Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']));
172
189
  function bridgeEnv(repo, wt, cfg) {
173
- const linkDirs = Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']);
174
- for (const d of linkDirs) {
190
+ for (const d of linkDirsOf(cfg)) {
175
191
  if (existsSync(join(repo, d)) && !existsSync(join(wt, d))) {
176
192
  try { symlinkSync(join(repo, d), join(wt, d)); } catch { /* best effort */ }
177
193
  }
@@ -196,8 +212,9 @@ function measureAt(repo, ref, cfg, repeat) {
196
212
  runSetup(wt, cfg, env);
197
213
  const out = {};
198
214
  for (const m of cfg.metrics) {
199
- const samples = [];
200
- for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
215
+ const samples = [], codes = [];
216
+ for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env, codes));
217
+ exitCodes.set(samples, codes);
201
218
  out[m.name] = samples;
202
219
  }
203
220
  return out;
@@ -266,9 +283,12 @@ const NON_SOURCE = [
266
283
  /_spec\.rb$/i,
267
284
  /[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
268
285
  /\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
269
- // a grader/eval/score FILE by name (eval.ts, grader-x.js, score.test.mjs, scoring.ts), never a prefix:
270
- // evaluate.ts, evaluation.js, scoreboard.js are production source and must stay in the slice
271
- /(^|\/)(eval|evals|grader|graders|score|scorer|scoring)([._-][^/]*)?\.[cm]?[jt]s$/i,
286
+ // a grader/eval/score FILE by name in a scripting language (eval.py, grader-x.js, grade.sh, scoring.ts), never a
287
+ // prefix: evaluate.ts, evaluation.py, scoreboard.js stay source. Compiled names stay source too (eval.c, eval.go
288
+ // are interpreters; Score.java is a type), and so do .tsx/.vue files (a Score component, a /score page).
289
+ /(^|\/)(eval|evals|grade|grader|graders|score|scorer|scoring)([._-][^/]*)?\.([cm]?[jt]s|py|sh|bash|zsh|rb|pl|php|lua|r|jl)$/i,
290
+ // anything under an evals/ or grader(s)/ directory. Not eval/ (an interpreter's package) or scoring/ (a product module).
291
+ /(^|\/)(evals|graders?)\//i,
272
292
  // config files only, require a .config/.conf/.setup segment so PRODUCTION source that
273
293
  // happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
274
294
  // the source slice (a false sweep here can flag an honest win as GAMED).
@@ -311,9 +331,9 @@ function evaluatorMatcher(cfg) {
311
331
  // forbids ANY change to its paths (same glob semantics as evaluatorPaths). A hit fails the gate
312
332
  // regardless of metrics and rides at the top of the report; mandates inherit through `extends`,
313
333
  // so a parent scope's rule holds for every repo beneath it.
314
- // The mandates in force are the UNION of the head config's and the base ref's (`baseMandates`,
315
- // resolved by gateDefAtRef): deleting a mandate in the same change that violates it does not
316
- // clear it. A mandate removed at head but violated in the diff is reported with `removed: true`; one
334
+ // The mandates in force are the base ref's (`baseMandates`, resolved by gateDefAtRef), in their head
335
+ // version when both have them: deleting a mandate in the same change that violates it does not
336
+ // clear it, and one the change adds does not apply to it (see below). A mandate removed at head but violated in the diff is reported with `removed: true`; one
317
337
  // whose paths were edited at head (same name, different paths) is tested in BOTH versions and reported
318
338
  // with `edited: true`, so narrowing a rule in the change that breaks it does not clear it either.
319
339
  // The honest way to retire one is a separate change that only edits the config.
@@ -482,12 +502,16 @@ function workingSnapshot(repo) {
482
502
  }
483
503
  }
484
504
 
485
- // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
505
+ // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
486
506
  // its `finally` never ran. On the next run, drop stale worktree registry entries and any
487
- // abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. Never touches a
488
- // live worktree (a still-registered dir, or this repo's checkout younger than 1h). A dir tagged
489
- // for ANOTHER repo may be mid-measurement, so it gets a full day; after that it is an orphan of a
490
- // repo that was deleted or moved (a new real path is a new tag nobody will ever claim).
507
+ // abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. It only ever removes a
508
+ // directory whose name is exactly what this engine generates (ENGINE_TMP): a folder named
509
+ // promptwheel-notes, or another repo's worktree named promptwheel-site-wt, is not ours at any age.
510
+ // Ages, by dir mtime: this repo's unregistered checkout after 1h; this repo's still-registered worktree
511
+ // after a full day (a killed run leaves it registered, and a younger one may be a concurrent gate still
512
+ // measuring); a dir tagged for ANOTHER repo after a full day (an orphan of a repo that was deleted or
513
+ // moved: a new real path is a new tag nobody will ever claim). A registered worktree that is not
514
+ // tagged for this repo is never touched: it may be yours.
491
515
  function selfHeal(repo) {
492
516
  try {
493
517
  git(['worktree', 'prune'], repo); // remove registry entries whose dir is already gone (always safe)
@@ -496,16 +520,18 @@ function selfHeal(repo) {
496
520
  .split('\n').filter((l) => l.startsWith('worktree ')).map((l) => l.slice(9).trim()),
497
521
  );
498
522
  const tmp = tmpdir(), now = Date.now(), mine = tmpOwnerTag(repo);
499
- const TAGGED = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-/;
500
523
  for (const name of readdirSync(tmp)) {
501
- if (!name.startsWith('promptwheel-') && !name.startsWith('pw-idx-')) continue; // worktrees + temp-index dirs
502
- const tag = (name.match(TAGGED) || [])[2] || null;
503
- // ours after an hour; another repo's, or an untagged dir (an older engine's), only after a full day
504
- const maxAge = tag === mine ? 3600_000 : 86_400_000;
524
+ const made = ENGINE_TMP.exec(name);
525
+ if (!made) continue; // not a name this engine generates (worktrees + temp-index dirs): never removed
526
+ const tag = made[2];
527
+ // ours after an hour; another repo's only after a full day
505
528
  const p = join(tmp, name);
506
- if (tracked.has(p)) continue; // a live/registered worktree, leave it
507
- try { if (statSync(p).mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
529
+ if (tracked.has(p) && tag !== mine) continue; // a registered worktree this repo's gate did not make
530
+ const maxAge = tag === mine && !tracked.has(p) ? 3600_000 : 86_400_000;
531
+ // lstat: the engine makes directories, so a file or a symlink under one of its names is left alone
532
+ try { const st = lstatSync(p); if (st.isDirectory() && st.mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
508
533
  }
534
+ git(['worktree', 'prune'], repo); // drop the registry entries of anything removed above
509
535
  } catch { /* cleanup must never block the gate */ }
510
536
  }
511
537
 
@@ -537,18 +563,25 @@ function warnEditableInstall(repo) {
537
563
  // guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
538
564
  // guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
539
565
  // that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
540
- const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer'];
566
+ const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer', 'setup', 'env', 'linkDirs', 'linkNodeModules'];
567
+ // what runs or shapes every measurement besides the metric's own command. A setup can rewrite a test and an
568
+ // env var can skip one, so these are part of the gate: fingerprinted, and held at their base values for the
569
+ // change that edits them (see detectGamingPass).
570
+ const measureOf = (s) => ({ setup: s.setup, env: s.env, linkDirs: s.linkDirs, linkNodeModules: s.linkNodeModules });
571
+ const measuredWith = (s) => JSON.stringify([s.setup ?? null, Object.entries(s.env || {}).map(([k, v]) => `${k}=${v}`).sort(), [...linkDirsOf(s)].sort()]);
541
572
 
542
573
  // Resolve the gate definition as of `ref` (metrics + mandates + scalars), following `extends` by
543
574
  // reading every file from that ref via `git show`, mirroring resolveConfig (a parent's mandates
544
575
  // hold beneath it; a local mandate of the same name overrides). Returns { metrics:Map, scalars }
545
- // with scalars.mandates, or null when there is no config at `ref` (a gate ADDED in this range is
546
- // not a weakening). Never exits: any missing/unreadable file or parse error yields null (that
547
- // branch simply contributes nothing).
576
+ // with scalars.mandates, or null when there is no config at `ref` (gate() then falls back to the
577
+ // zero-config gate the repo ran under, see zeroConfigGate). Never exits: any missing/unreadable file or
578
+ // parse error yields null (that branch simply contributes nothing), and `partial` says an extends target
579
+ // could not be read at `ref` (it lives outside the repo), so how the base measured is not fully known.
548
580
  function gateDefAtRef(repo, ref) {
549
581
  // quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
550
582
  const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
551
583
  const seen = new Set();
584
+ let partial = false;
552
585
  const resolve = (p) => {
553
586
  if (seen.has(p)) return null;
554
587
  seen.add(p);
@@ -557,9 +590,11 @@ function gateDefAtRef(repo, ref) {
557
590
  let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
558
591
  const byName = new Map(); const mandatesByName = new Map(); const scalars = {};
559
592
  for (const ext of [].concat(cfg.extends || [])) {
560
- if (isAbsolute(ext)) continue; // outside the repo, cannot git-show; unchanged anyway
561
- const base = resolve(join(dirname(p), ext).split('\\').join('/'));
562
- if (!base) continue;
593
+ if (isAbsolute(ext)) { partial = true; continue; } // outside the repo, cannot git-show; unchanged anyway
594
+ const bp = join(dirname(p), ext).split('\\').join('/');
595
+ const again = seen.has(bp); // a diamond: already merged, not a missing piece
596
+ const base = resolve(bp);
597
+ if (!base) { if (!again) partial = true; continue; }
563
598
  for (const m of base.metrics.values()) byName.set(m.name, m);
564
599
  for (const md of base.scalars.mandates) mandatesByName.set(md.name, md);
565
600
  for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
@@ -570,18 +605,35 @@ function gateDefAtRef(repo, ref) {
570
605
  scalars.mandates = [...mandatesByName.values()];
571
606
  return { metrics: byName, scalars };
572
607
  };
573
- return resolve('promptwheel.config.json');
608
+ const def = resolve('promptwheel.config.json');
609
+ return def && { ...def, partial };
610
+ }
611
+
612
+ // The gate a repo with no config ran under: the detected test command (source-only re-run on) with the
613
+ // stack's env and linkDirs, detected from the tracked files as of `ref`. A change that adds the first
614
+ // config is judged against this, so `tests_pass.cmd: "true"` in a new config is an edited guard, not a
615
+ // fresh start. `initSetup` is the setup `init` would add: a first config that carries exactly it has not
616
+ // changed how the gate measures. With no test command to detect (the placeholder) the old gate measured
617
+ // nothing, so its setup is no baseline (`partial`).
618
+ function zeroConfigGate(repo, ref) {
619
+ const at = stackAt(repo, ref), auto = autoConfig(repo, at);
620
+ return { metrics: new Map(auto.metrics.map((m) => [m.name, m])), scalars: { mandates: [], ...measureOf(auto) }, zeroConfig: true, initSetup: detectSetup(repo, at), partial: auto.metrics[0].cmd === PLACEHOLDER_TEST };
574
621
  }
575
622
 
623
+ // how a metric is measured and judged: change any of these on a guard and it is a different guard
624
+ // what init writes when it finds no test command. It guards nothing, so replacing it is setup, not a weakened gate.
625
+ const PLACEHOLDER_TEST = 'echo "set your test command in promptwheel.config.json" && false';
626
+ const measuredBy = (m) => JSON.stringify([m.cmd ?? null, m.extract ?? 'number', m.direction || 'up', m.exact === true, m.report === true]);
576
627
  // an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
577
628
  function gateFingerprint(metricsList, scalars) {
578
629
  const norm = {
579
630
  metrics: metricsList.map((m) => ({
580
- name: m.name, guard: m.guard === true, direction: m.direction || 'up',
631
+ name: m.name, guard: m.guard === true, measuredBy: measuredBy(m),
581
632
  gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
582
633
  })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
583
634
  gamingThreshold: scalars.gamingThreshold ?? null,
584
635
  evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
636
+ measuredWith: measuredWith(scalars),
585
637
  mandates: Array.isArray(scalars.mandates) ? scalars.mandates.map((m) => ({ name: String(m.name || ''), paths: [...(Array.isArray(m.paths) ? m.paths : [])].sort() })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) : [],
586
638
  };
587
639
  return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
@@ -626,7 +678,8 @@ function verdictFingerprint(report) {
626
678
  return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
627
679
  }
628
680
 
629
- // guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
681
+ // guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off), or
682
+ // measures another way (a new cmd, extract, direction, exact or report flag): `cmd: "true"` is a dropped guard too
630
683
  function gateWeakenings(baseGate, headMetrics) {
631
684
  if (!baseGate) return [];
632
685
  const head = new Map(headMetrics.map((m) => [m.name, m]));
@@ -636,6 +689,7 @@ function gateWeakenings(baseGate, headMetrics) {
636
689
  const hm = head.get(bm.name);
637
690
  if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
638
691
  else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
692
+ else if (!bm.report && bm.cmd !== PLACEHOLDER_TEST && measuredBy(hm) !== measuredBy(bm)) weak.push({ name: bm.name, change: 'guard-edited', def: bm });
639
693
  }
640
694
  return weak;
641
695
  }
@@ -654,29 +708,51 @@ function readAllowTrailer(repo, ref) {
654
708
  return m && m[1].trim() ? { reason: m[1].trim().slice(0, 300) } : null;
655
709
  }
656
710
 
711
+ // A change is judged by the gate that was in force before it (`baseGate`: the config at base, or the
712
+ // zero-config gate when this change adds the first config). So a change cannot switch off the source-only
713
+ // re-run for a metric that had it, loosen gamingThreshold, or swap the setup, env or linked dirs it is
714
+ // measured with: the base values hold for the change that edits them, and apply from the next one.
715
+ const gamingCheckOff = (cm, baseGate) => {
716
+ if (!cm || cm.gamingCheck !== false) return false;
717
+ const bm = baseGate && baseGate.metrics.get(cm.name);
718
+ return !bm || bm.gamingCheck === false; // off before this change too, or a metric this change adds
719
+ };
720
+ function gamingThresholdFor(cm, cfg, baseGate) {
721
+ const now = cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5;
722
+ const bm = baseGate && baseGate.metrics.get(cm.name);
723
+ const before = bm ? (bm.gamingThreshold ?? baseGate.scalars.gamingThreshold ?? (baseGate.partial ? now : 0.5)) : now;
724
+ return Math.max(now, before); // the stricter of before and after
725
+ }
726
+
657
727
  // The three Section 3 checks, run only with detectGaming, in order: (1) re-prove every actual
658
728
  // win with the agent's source edits alone; (2) content-hash auto-fail: a declared evaluatorPath
659
729
  // gate file edited in this diff is a moved goalpost, so a metric that improved alongside it is
660
730
  // gamed by declaration, deterministic even if the source-only re-run read inert; (3) gate
661
- // integrity: resolve the gate as of BASE, re-measure any guard HEAD dropped under its base
662
- // definition (it only counts as tampering if it WOULD HAVE FAILED at head), and fingerprint the
663
- // gate definition at both refs (metrics, evaluatorPaths, gamingThreshold, mandates) so a quietly
664
- // deleted guard or mandate is tamper-evident: base != head is recorded as gate.changed and
731
+ // integrity: take the gate as of BASE, re-measure any guard HEAD dropped, edited or measures with
732
+ // another setup, env or linked dirs, under its base definition (it only counts as tampering if it
733
+ // WOULD HAVE FAILED at head), and fingerprint the gate definition at both refs (metrics,
734
+ // evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) so a quietly deleted guard
735
+ // or mandate is tamper-evident: base != head is recorded as gate.changed and
665
736
  // printed, whatever the verdict. Mutates `metrics` in place (gamed/sourceOnly/retained/
666
737
  // gamingReason) and returns the report.gate record. An allow trailer can still clear a gamed
667
738
  // metric afterwards, in gate().
668
- function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
739
+ function detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) {
740
+ // how the base measured. Unknown (`partial`) reads as unchanged; so does a first config that adds only the setup `init` detects
741
+ const bs = baseGate ? (baseGate.partial ? { ...baseGate.scalars, ...measureOf(cfg) } : baseGate.scalars) : null;
742
+ const asInit = !!bs && baseGate.zeroConfig && measuredWith({ ...bs, setup: baseGate.initSetup }) === measuredWith(cfg);
743
+ const measureChanged = !!bs && !asInit && measuredWith(bs) !== measuredWith(cfg);
744
+ const baseRun = measureChanged ? { ...cfg, ...measureOf(bs) } : cfg; // the head gate, run with the base setup, env and linked dirs
669
745
  const toProve = [];
670
746
  for (const m of metrics) {
671
747
  if (m.status !== 'improved') continue;
672
748
  const cm = cfg.metrics.find((c) => c.name === m.name);
673
- if (cm.gamingCheck === false) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
749
+ if (gamingCheckOff(cm, baseGate)) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
674
750
  toProve.push(cm);
675
751
  }
676
- const srcResults = measureSourceOnlyAll(repo, base, head, toProve, cfg, repeat); // one patched checkout for all of them
752
+ const srcResults = measureSourceOnlyAll(repo, base, head, toProve, baseRun, repeat); // one patched checkout for all of them
677
753
  for (const cm of toProve) {
678
754
  const m = metrics.find((x) => x.name === cm.name);
679
- const j = judgeGaming(m, srcResults[cm.name], cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
755
+ const j = judgeGaming(m, srcResults[cm.name], gamingThresholdFor(cm, cfg, baseGate));
680
756
  m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
681
757
  if (j.unprovable) m.unprovable = true;
682
758
  }
@@ -687,32 +763,55 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
687
763
  if (split.source.length && split.nonSource.length) {
688
764
  const held = metrics.filter((m) => m.status === 'unchanged' && m.direction === 'pass' && m.after === 1 && !m.tripwire);
689
765
  const defs = held.map((m) => cfg.metrics.find((c) => c.name === m.name)).filter(Boolean);
690
- const res = measureSourceOnlyAll(repo, base, head, defs, cfg, repeat);
766
+ const res = measureSourceOnlyAll(repo, base, head, defs, baseRun, repeat);
691
767
  for (const m of held) { const r = res[m.name]; if (r && r.samples && r.samples.length && median(r.samples) !== 1) m.heldByTestEdit = true; }
692
768
  }
693
- const evalChanged = changedEvaluatorPaths(repo, base, head, cfg);
769
+ // a file declared as a goalpost before this change still is one for this change: dropping the declaration in the
770
+ // same diff that edits the file does not make the edit count as source
771
+ const declared = [...new Set([...(cfg.evaluatorPaths || []), ...((baseGate && baseGate.scalars.evaluatorPaths) || [])])];
772
+ const evalChanged = changedEvaluatorPaths(repo, base, head, { ...cfg, evaluatorPaths: declared });
694
773
  if (evalChanged.length) {
695
774
  for (const m of metrics) {
696
775
  if (m.status !== 'improved' || m.gamed === true) continue;
697
776
  const cm = cfg.metrics.find((c) => c.name === m.name);
698
- if (cm && cm.gamingCheck === false) continue; // tripwire/test-side guards legitimately live in gate files
777
+ if (gamingCheckOff(cm, baseGate)) continue; // tripwire/test-side guards legitimately live in gate files
699
778
  m.gamed = true;
700
779
  m.gamingReason = `a declared evaluatorPath gate file changed in this diff (${evalChanged.map((e) => e.path).join(', ')}); a win alongside a goalpost edit is gamed by declaration`;
701
780
  }
702
781
  }
703
- const baseGate = gateDefAtRef(repo, base);
704
782
  const weak = gateWeakenings(baseGate, cfg.metrics);
783
+ // the setup, env or linked dirs changed: every guard the base had is measured again the way it was measured before
784
+ if (measureChanged) for (const bm of baseGate.metrics.values()) {
785
+ if (bm.guard === true && !bm.report && bm.cmd !== PLACEHOLDER_TEST && !weak.some((w) => w.name === bm.name)) weak.push({ name: bm.name, change: 'measurement-edited', def: bm });
786
+ }
705
787
  const weakened = [];
706
788
  if (weak.length) {
707
789
  const defs = weak.map((w) => w.def);
708
- const b = measureAt(repo, base, { ...cfg, metrics: defs }, repeat);
709
- const a = measureAt(repo, head, { ...cfg, metrics: defs }, repeat);
790
+ const b = measureAt(repo, base, { ...baseRun, metrics: defs }, repeat);
791
+ const a = measureAt(repo, head, { ...baseRun, metrics: defs }, repeat);
792
+ const evs = new Map();
710
793
  for (const w of weak) {
711
794
  const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
712
- if (!ev.ok) weakened.push({ name: w.name, change: w.change, status: ev.status });
795
+ evs.set(w.name, ev);
796
+ // an edited guard that still fails under its old definition was turned green by the edit, not the code
797
+ let hidden = !ev.ok || (w.change === 'guard-edited' && ev.inert);
798
+ // a guard measured another way counts only when the new measurement hides it: it fails the old way and reads fine the new way
799
+ if (w.change === 'measurement-edited') { const hm = metrics.find((x) => x.name === w.name); hidden = (!ev.ok || !!ev.inert) && !!hm && hm.ok && !hm.inert; }
800
+ if (hidden) weakened.push({ name: w.name, change: w.change, status: ev.inert ? 'failed' : ev.status });
801
+ }
802
+ // a win under the old definition still has to be earned in the source: swapping or renaming the guard in the same
803
+ // change does not skip the source-only re-run
804
+ const won = weak.filter((w) => w.def.gamingCheck !== false && !weakened.some((x) => x.name === w.name) && evs.get(w.name).status === 'improved');
805
+ if (won.length) {
806
+ const src = measureSourceOnlyAll(repo, base, head, won.map((w) => w.def), baseRun, repeat);
807
+ for (const w of won) {
808
+ const j = judgeGaming({ name: w.name, direction: w.def.direction || 'up', guard: true, ...evs.get(w.name) }, src[w.name], w.def.gamingThreshold ?? (bs && bs.gamingThreshold) ?? 0.5);
809
+ if (j.gamed) weakened.push({ name: w.name, change: w.change, status: 'gamed' });
810
+ }
713
811
  }
714
812
  }
715
- const fpBase = baseGate ? gateFingerprint([...baseGate.metrics.values()], baseGate.scalars) : null;
813
+ // the fingerprint is of a config file: `base` stays null when there was none there (the zero-config gate is not a file)
814
+ const fpBase = baseGate && !baseGate.zeroConfig ? gateFingerprint([...baseGate.metrics.values()], bs) : null;
716
815
  const fpHead = gateFingerprint(cfg.metrics, cfg);
717
816
  const info = {
718
817
  fingerprint: { base: fpBase, head: fpHead },
@@ -769,17 +868,26 @@ function gate(repo, opts) {
769
868
  // a guard this change adds has no value at base, because its scanner did not exist there. That is adoption,
770
869
  // not a failure: it starts counting from head. Removing a guard is still caught (gateWeakenings), and a guard the
771
870
  // base contract already had still fails when it cannot be measured. A zero-config run has no contract to adopt.
772
- const baseDefs = opts.config ? null : gateDefAtRef(repo, base);
871
+ const fileGate = gateDefAtRef(repo, base);
872
+ const baseDefs = opts.config ? null : fileGate;
773
873
  const adopted = (name) => !opts.config && !(baseDefs && baseDefs.metrics.has(name));
874
+ // the gate in force before this change: the config committed at base, or, when this change adds the
875
+ // first config, the zero-config gate the repo ran under until now
876
+ const baseGate = opts.detectGaming ? (fileGate || (opts.config ? null : zeroConfigGate(repo, base))) : null;
774
877
  const metrics = cfg.metrics.map((m) => {
775
878
  const ev = evaluate(m, before[m.name], after[m.name], repeat);
776
879
  if (ev.status === 'unmeasurable' && ev.before == null && ev.after != null && adopted(m.name)) { ev.ok = true; ev.adopted = true; }
777
880
  // a `report` guard tells you and does not fail: a contract can start by watching and tighten later
778
881
  if (m.report && !ev.ok && ev.status === 'regressed') { ev.ok = true; ev.noted = true; }
779
- return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(m.gamingCheck === false ? { tripwire: true } : {}), ...ev };
882
+ // an inert guard: did its command run and fail, or never run? `ran` says where it ran (absent: at neither ref)
883
+ if (ev.inert && m.cmd !== PLACEHOLDER_TEST) {
884
+ const nb = neverRan(before[m.name]), na = neverRan(after[m.name]);
885
+ if (!nb || !na) ev.ran = !nb && !na ? 'both' : nb ? 'head' : 'base';
886
+ }
887
+ return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(gamingCheckOff(m, baseGate) ? { tripwire: true } : {}), ...ev };
780
888
  });
781
889
  // Section 3 (gaming detection), on by default: mutates `metrics` and yields the gate record
782
- const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat) : null;
890
+ const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) : null;
783
891
  // declared override: a Promptwheel-Allow trailer (or --allow) acknowledges an intentional
784
892
  // goalpost/gate change, so a would-be GAMED becomes PASS, recorded in report.allowed for review.
785
893
  // It covers an `unprovable` win too (the source-only re-run could not be applied): the declared
@@ -820,7 +928,7 @@ function gate(repo, opts) {
820
928
  for (const m of ga) { m.ok = true; m.allowed = true; allowed.cleared.push(m.name); }
821
929
  } else for (const m of ga) m.allowable = true; // failed, and a declared reason would clear it: say so in the report
822
930
  }
823
- // mandates run first: a protected-path hard rule fails the gate no matter what the metrics say.
931
+ // mandates: a protected-path hard rule fails the gate no matter what the metrics say.
824
932
  // The base ref's mandates count too (resolved through `extends` by gateDefAtRef), so a change
825
933
  // cannot clear a mandate by deleting it in the same diff that violates it.
826
934
  const baseMandates = (gateInfo && gateInfo.baseMandates) || (!opts.detectGaming ? ((gateDefAtRef(repo, base) || {}).scalars || {}).mandates : null);
@@ -906,7 +1014,7 @@ const PRESETS = {
906
1014
  { name: 'codeowners-frozen', paths: ['CODEOWNERS'], reason: 'review ownership is set by people' },
907
1015
  { name: 'scanner-frozen', paths: ['.promptwheel/packs/**'], reason: 'the scanners that read the contract are not edited by the change they read' },
908
1016
  { name: 'lock-frozen', paths: ['promptwheel.lock.json'], allowable: 'flag', reason: 'the architecture lock changes with a reason from a person (--allow), never as a side effect of a change' },
909
- { name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], reason: 'the hooks that enforce the contract are wired by a person, not detached by the agent' },
1017
+ { name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], allowable: 'flag', reason: 'the hooks that enforce the contract change with a reason from a person (--allow), and are never detached by the agent' },
910
1018
  ],
911
1019
  metrics: [
912
1020
  { name: 'lifecycle_scripts', cmd: 'node packs/supply-chain/scan.mjs lifecycle .', extract: 'number', direction: 'down', guard: true, gamingCheck: false, allowable: 'flag' },
@@ -919,17 +1027,17 @@ const PRESETS = {
919
1027
  ] },
920
1028
  };
921
1029
 
922
- function detectTestCmd(repo) {
923
- const has = (f) => existsSync(join(repo, f));
1030
+ function detectTestCmd(repo, { has, read } = stackAt(repo)) {
924
1031
  if (has('go.mod')) return 'go test ./...';
925
1032
  if (has('Cargo.toml')) return 'cargo test';
926
- if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return has('.venv/bin/pytest') ? '.venv/bin/pytest -q' : 'pytest -q';
1033
+ // the venv is not tracked: it is looked for on disk whichever ref the stack is read from
1034
+ if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return existsSync(join(repo, '.venv/bin/pytest')) ? '.venv/bin/pytest -q' : 'pytest -q';
927
1035
  // only trust `npm test` when a test script actually exists, otherwise the metric can
928
1036
  // never pass and the gate would "protect" nothing (very common in Next.js app repos)
929
1037
  try {
930
- if (JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8')).scripts?.test) return 'npm test --silent';
1038
+ if (JSON.parse(read('package.json')).scripts?.test) return 'npm test --silent';
931
1039
  } catch { /* no or unparsable package.json, fall through */ }
932
- return 'echo "set your test command in promptwheel.config.json" && false';
1040
+ return PLACEHOLDER_TEST;
933
1041
  }
934
1042
 
935
1043
  // deps that live outside git + the env a test needs, defaulted per stack. init writes these into
@@ -938,18 +1046,16 @@ function detectTestCmd(repo) {
938
1046
  // where the detected test command cannot run without a build first (a 10-repo replay put this at
939
1047
  // ~2/3 of gates). Written by `init` into `setup`; a failed setup stays honest (inert -> inconclusive,
940
1048
  // never a fake green). Kept out of the zero-config default so the fast per-turn path stays fast.
941
- function detectSetup(repo) {
942
- const has = (f) => existsSync(join(repo, f));
1049
+ function detectSetup(repo, { has, read } = stackAt(repo)) {
943
1050
  try {
944
- const pkg = JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8'));
1051
+ const pkg = JSON.parse(read('package.json'));
945
1052
  if (pkg.scripts && pkg.scripts.build) return 'npm run build --if-present'; // TS/bundler tests that run against a built dir
946
1053
  } catch { /* no or unparsable package.json */ }
947
1054
  if (has('pyproject.toml') || has('setup.py')) return 'pip install -e . --quiet 2>/dev/null || true'; // editable install so imports resolve the measured ref
948
1055
  return undefined;
949
1056
  }
950
1057
 
951
- function detectEnv(repo) {
952
- const has = (f) => existsSync(join(repo, f));
1058
+ function detectEnv(repo, { has } = stackAt(repo)) {
953
1059
  // Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
954
1060
  // resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
955
1061
  if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
@@ -1064,6 +1170,7 @@ function writeAgentBlock(repo) {
1064
1170
 
1065
1171
  const short = (repo, ref) => { try { return git(['rev-parse', '--short', ref], repo); } catch { return ref; } };
1066
1172
 
1173
+ const wouldHave = (status) => (status === 'unmeasurable' ? 'been unmeasurable' : status);
1067
1174
  function printHuman(r) {
1068
1175
  const arrowFor = (m) => (m.delta == null ? '?' : m.delta > 0 ? '▲' : m.delta < 0 ? '▼' : '=');
1069
1176
  console.log(`\nPromptWheel ${r.base} → ${r.head}${r.repeat > 1 ? ` (×${r.repeat})` : ''}\n`);
@@ -1072,15 +1179,21 @@ function printHuman(r) {
1072
1179
  const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
1073
1180
  const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
1074
1181
  console.log(` ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)} (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
1075
- if (m.inert) console.log(` ⚠ "${m.name}" never ran (same value at both refs): its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
1182
+ if (m.inert && m.ran === 'both') console.log(` ⚠ "${m.name}" ran and failed before and after this change, so it guards nothing here. Fix the failing tests, or the command, before trusting a pass`);
1183
+ else if (m.inert) console.log(` ⚠ "${m.name}" ${m.ran ? `ran and failed at the ${m.ran} ref and did not run at the other` : 'never ran (same value at both refs)'}: its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
1076
1184
  if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
1077
1185
  }
1078
- if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : 'disabled'} in this change and it would have ${w.status} at head (goalpost moved, not earned)`);
1079
- if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath or mandate was added, removed or edited; review promptwheel.config.json as part of the diff`);
1186
+ if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(w.change === 'measurement-edited'
1187
+ ? ` 🚩 GATE WEAKENED, how the guard "${w.name}" is measured (setup, env or linked dirs) was edited in this change, and measured the way it was before it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`
1188
+ : ` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : 'disabled'} in this change and it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`);
1189
+ if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited; review promptwheel.config.json as part of the diff`);
1080
1190
  const failReason = r.mandates ? ', a mandate was violated (a protected path changed)' : ', a guarded metric regressed (beyond noise)';
1081
1191
  console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? failReason : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
1082
- if (r.verdict === 'fail' && r.mandates) console.log(' a mandate is a hard rule; if the change is truly intended, edit the mandate in promptwheel.config.json (a tracked, tamper-evident change), do not work around it');
1083
- else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
1192
+ if (r.verdict === 'fail' && r.mandates) {
1193
+ const takes = r.mandates.filter((md) => md.allowable).map((md) => `"${md.name}"`);
1194
+ if (takes.length) console.log(` intentional? ${takes.join(', ')} ${takes.length > 1 ? 'accept' : 'accepts'} a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record`);
1195
+ if (takes.length < r.mandates.length) console.log(` ${takes.length ? 'any other' : 'a'} mandate is a hard rule; if the change is truly intended, retire the mandate in promptwheel.config.json in a change of its own first (a tracked, tamper-evident change), do not work around it`);
1196
+ } else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
1084
1197
  else if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name)');
1085
1198
  if (r.allowed) console.log(` ✓ allowed: ${r.allowed.cleared.join(', ')} acknowledged as an intentional change, recorded (Promptwheel-Allow: ${r.allowed.reason})`);
1086
1199
  console.log('');
@@ -1134,9 +1247,10 @@ function renderMarkdown(r) {
1134
1247
  '|---|--:|--:|--:|---|---|',
1135
1248
  rows,
1136
1249
  '',
1137
- r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule, enforced before any metric.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
1138
- ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by dropping a guard, not by earning it.`] : []),
1139
- ...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath or mandate was added, removed or edited. Review the config change with the code.`] : []),
1250
+ r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule; no metric result clears it.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? (r.metrics.some((m) => m.inert) && r.metrics.every((m) => !m.inert || m.ran === 'both') ? '> 🟡 A 🛡️ guarded check is **inert**, it fails before and after this change, so nothing was actually verified. Fix it before trusting a pass.' : '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.') : '> ✅ No guarded metric regressed.',
1251
+ ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : w.change === 'measurement-edited' ? 'measured another way (setup, env or linked dirs)' : 'disabled'} (would have ${wouldHave(w.status)} at head)`).join('; ')}. The pass was bought by changing the gate, not by earning it.`] : []),
1252
+ ...(r.allowed ? ['', `> ✓ **Allowed:** \`${r.allowed.cleared.join('`, `')}\` cleared by a declared reason: ${r.allowed.reason}`] : []),
1253
+ ...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited. Review the config change with the code.`] : []),
1140
1254
  '',
1141
1255
  '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
1142
1256
  ].join('\n');
@@ -1169,7 +1283,7 @@ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluat
1169
1283
  // Importing this file never runs the CLI (see the isMain guard at the bottom).
1170
1284
  // Stable across 0.x minor versions. The schema check() returns and the exit codes
1171
1285
  // verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
1172
- // shell, import, MCP) are in INTEGRATE.md.
1286
+ // shell, import) are in INTEGRATE.md.
1173
1287
  // ─────────────────────────────────────────────────────────────────────────────
1174
1288
 
1175
1289
  // The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
@@ -1207,8 +1321,8 @@ export function verdictExitCode(verdict) {
1207
1321
  // detectGaming boolean run the Section 3 source-only re-proof (default: true)
1208
1322
  // allow string a declared override reason (same as --allow / a Promptwheel-Allow: trailer)
1209
1323
  export function check(options = {}) {
1210
- // `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config lives
1211
- // and where .promptwheel/ is written), exactly as the CLI does from the same directory
1324
+ // `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
1325
+ // lives), exactly as the CLI does from the same directory
1212
1326
  let start = process.cwd();
1213
1327
  if (options.repo) { try { start = realpathSync(options.repo); } catch { bad(`promptwheel: not inside a git repository (${options.repo})`); } }
1214
1328
  const repo = repoRoot(start);
@@ -1328,17 +1442,35 @@ const GATE_FLAGS = ['--base', '--head', '--repeat', '--working', '--no-record',
1328
1442
  const COMMAND_FLAGS = {
1329
1443
  run: GATE_FLAGS, init: ['--preset', '--list', '--force', '--no-agent-block'], lock: [],
1330
1444
  };
1331
- const VALUED_FLAGS = new Set(['--base', '--head', '--repeat', '--allow', '--preset']);
1445
+ // what each valued flag needs. A missing or unusable value is a usage error too: `--repeat abc` must not
1446
+ // measure nothing and read FAIL, and `--allow` with nothing after it must not run as if no reason was asked for.
1447
+ const VALUED_FLAGS = { '--base': 'a ref', '--head': 'a ref', '--repeat': 'a whole number, 1 or more', '--allow': 'a reason, e.g. --allow "the spec changed"', '--preset': 'a preset name (see promptwheel init --list)' };
1332
1448
  function checkFlags(cmd, argv) {
1333
1449
  const known = new Set(COMMAND_FLAGS[cmd]);
1334
1450
  for (let i = 0; i < argv.length; i++) {
1335
1451
  const a = argv[i];
1336
1452
  if (!a.startsWith('-')) continue;
1337
1453
  if (!known.has(a)) bad(`promptwheel ${cmd}: unknown flag ${a} (see promptwheel --help)`);
1338
- if (VALUED_FLAGS.has(a)) i++;
1454
+ if (!(a in VALUED_FLAGS)) continue;
1455
+ const v = argv[++i];
1456
+ const gone = v === undefined || known.has(v); // nothing after the flag, or the next flag
1457
+ // an empty --base or --head is the default ref: the Action passes an empty base outside a pull request
1458
+ const blank = !gone && !v.trim() && !(v === '' && (a === '--base' || a === '--head'));
1459
+ if (gone || blank) bad(`promptwheel ${cmd}: ${a} needs ${VALUED_FLAGS[a]}`);
1460
+ if (a === '--repeat' && !/^[1-9]\d*$/.test(v)) bad(`promptwheel ${cmd}: --repeat needs ${VALUED_FLAGS[a]} (got "${v}")`);
1339
1461
  }
1340
1462
  }
1341
1463
 
1464
+ // read PR comments on stdin: a JSON array, or one comment per line as `gh api --paginate --jq '.[]'` prints every
1465
+ // page. Streamed: readFileSync(0) corrupted the heap on Node 24.22 reading a live paginated gh pipe.
1466
+ async function prAllow() {
1467
+ let t = ''; process.stdin.setEncoding('utf8');
1468
+ for await (const chunk of process.stdin) t += chunk;
1469
+ t = t.trim(); let c = [];
1470
+ try { c = [].concat(JSON.parse(t || '[]')); } catch { try { c = t.split('\n').filter(Boolean).map((l) => JSON.parse(l)); } catch { /* unreadable, no approval */ } }
1471
+ process.stdout.write(allowFromComments(c));
1472
+ }
1473
+
1342
1474
  // write or refresh the architecture lock from the tree as it stands. A person runs this: the lock file is
1343
1475
  // a mandate, so the refreshed file passes the gate only with --allow "<reason>".
1344
1476
  async function lock() {
@@ -1353,9 +1485,9 @@ function dispatch() {
1353
1485
  const cmd = (rest.includes('--help') || rest.includes('-h')) ? 'help' : first; // `promptwheel run --help` is a help request, not a gate run
1354
1486
  if (cmd in COMMAND_FLAGS) checkFlags(cmd, rest);
1355
1487
  if (cmd === 'run') run(rest);
1356
- else if (cmd === 'pr-allow') { let c = []; try { c = JSON.parse(readFileSync(0, 'utf8') || '[]'); } catch { /* no comments, no approval */ } process.stdout.write(allowFromComments(c)); }
1488
+ else if (cmd === 'pr-allow') return prAllow();
1357
1489
  else if (cmd === 'init') init(rest);
1358
- else if (cmd === 'lock') lock(rest);
1490
+ else if (cmd === 'lock') return lock(rest);
1359
1491
  else {
1360
1492
  const help = !cmd || cmd === '--help' || cmd === '-h' || cmd === 'help';
1361
1493
  if (!help) console.error(`unknown command: ${cmd}`);
@@ -1370,10 +1502,11 @@ function dispatch() {
1370
1502
  ' catches reward-hacking BY DEFAULT, re-proves each win from SOURCE edits alone (verdict GAMED,',
1371
1503
  ' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
1372
1504
  ' promptwheel run --working gate uncommitted changes (incl. newly added files)',
1505
+ ' promptwheel pr-allow < comments.json print the newest `sawbuck-allow: <reason>` from a PR comment by someone with write access (the Action uses it)',
1373
1506
  ' promptwheel run --allow "<reason>" acknowledge an intentional test/config change (or a `Promptwheel-Allow: <reason>` commit trailer; `Sawbuck-Allow:` is an alias): recorded, not flagged GAMED',
1374
1507
  '',
1375
1508
  'Exit codes: 0 pass · 1 fail (a guarded metric regressed, or a mandate was violated) · 2 gamed · 3 inconclusive · 64 usage or config error (never a verdict).',
1376
- 'Integrate (library / conform / shell / MCP): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
1509
+ 'Integrate (library / conform / shell): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
1377
1510
  'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
1378
1511
  ' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
1379
1512
  ].join('\n'));
@@ -1384,8 +1517,8 @@ function dispatch() {
1384
1517
  // run the CLI only when invoked directly (resolves symlinks so the npm bin still works)
1385
1518
  let isMain = false;
1386
1519
  try { isMain = !!process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); } catch { /* not main */ }
1387
- function main() {
1388
- try { dispatch(); }
1520
+ async function main() {
1521
+ try { await dispatch(); }
1389
1522
  catch (e) { if (e && e.name === 'ConfigError') die(e.message); throw e; }
1390
1523
  }
1391
1524
  if (isMain) main();
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "promptwheel",
3
- "version": "0.8.0",
3
+ "version": "0.8.1",
4
4
  "description": "A deterministic, zero-LLM check for your coding loop. A win must survive a re-run from the agent's source edits alone, so a test that went green by editing the test reads GAMED. Runs in CI or a Claude Code plugin.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -27,7 +27,7 @@ clean git worktrees, so a pack that is not in git is absent at both refs and the
27
27
  count. A private registry mirror makes this baseline large, not failing.
28
28
  - **unpinned_execution**: code run from a reference that can move. A GitHub Action not pinned to a full commit SHA (a local `./action` does not count), `npx`, `bunx` or `dlx` with no exact version, `npm install -g` with no exact version, and a remote script piped into a shell. Read from workflows, `action.yml`, package.json scripts, `scripts/*.sh` and Dockerfiles. In package.json scripts, a tool the manifest declares does not count: the lockfile already pins it.
29
29
  - **install_policy_gaps**: 0, 1 or 2. Are install scripts disabled, and is a release-age cooldown set (`ignore-scripts=true` and `min-release-age` in `.npmrc`, or the pnpm or yarn equivalents)? The cooldown is enforced by the package manager, not by this pack: a registry lookup here would make the verdict depend on the clock.
30
- - **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is reported but not held by default; add `--routes` to hold it. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
30
+ - **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is locked but not counted by default (`node .promptwheel/packs/supply-chain/lock.mjs drift . --list` shows it). To hold new routes too, add `--routes` to the end of the `lock_drift` metric's `cmd` in `promptwheel.config.json`, so it reads `node .promptwheel/packs/supply-chain/lock.mjs drift . --routes`. The flag belongs to that script: `promptwheel lock` takes no flags and always writes routes into the lock. The config is a frozen path here, so that edit passes with `promptwheel run --allow "<reason>"`. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
31
31
  - **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%, dependency changes; use the CI path below for those.
32
32
 
33
33
  **On a pull request.** Someone with write access comments `sawbuck-allow: <reason>` and re-runs the check. `sawbuck contract` marks everyday guards `report: true`, so they note a change without failing until you tighten them; this preset on its own holds them all.
@@ -57,10 +57,10 @@ Sometimes adding a dependency is exactly what was asked. Say so, and the gate re
57
57
  promptwheel run --working --allow "needed for the CSV export (papaparse)"
58
58
  ```
59
59
 
60
- or a `Promptwheel-Allow: <reason>` trailer on the commit. The seven metrics carry `allowable: true`,
61
- which means a declared reason clears their guarded regression for that one run, with the reason
62
- written into `report.allowed` and the outcome record. Nothing is loosened in the config, and a rise
63
- without a reason still fails. That is the point: every added package, script, or non-registry
60
+ A commit trailer does not count here. The seven metrics carry `allowable: "flag"`, which means a
61
+ reason from the `--allow` flag clears their guarded regression for that one run, with the reason
62
+ written into `report.allowed`. Nothing is loosened in the config, and a rise without a reason still
63
+ fails. That is the point: every added package, script, or non-registry
64
64
  source leaves an audit line a reviewer can read.
65
65
 
66
66
  `gamingCheck` is `false` on all seven because these numbers legitimately live in config files
@@ -69,8 +69,8 @@ flag every honest dependency bump.
69
69
 
70
70
  ## The mandates the preset writes
71
71
 
72
- A mandate is a path the agent must not touch in a change; the gate fails before any metric is read,
73
- and a mandate deleted or narrowed in the same change that violates it stays in force.
72
+ A mandate is a path the agent must not touch in a change. Touching one fails the gate whatever the
73
+ metrics say, and a mandate deleted or narrowed in the same change that violates it stays in force.
74
74
 
75
75
  | mandate | paths | why |
76
76
  | --- | --- | --- |
@@ -79,15 +79,20 @@ and a mandate deleted or narrowed in the same change that violates it stays in f
79
79
  | build-images-frozen | `Dockerfile*`, `docker-compose*` | what runs at build and deploy time |
80
80
  | publish-scripts-frozen | `scripts/release*`, `scripts/publish*` | the publish path is how a compromise spreads |
81
81
  | codeowners-frozen | `CODEOWNERS` | review ownership is set by people |
82
+ | scanner-frozen | `.promptwheel/packs/**` | the scanners are not edited by the change they read |
82
83
  | contract-frozen | `promptwheel.config.json`, `sawbuck.config.json` | the contract changes with a reason from a person (`--allow`) |
83
- | hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks that enforce the contract are wired by a person |
84
+ | lock-frozen | `promptwheel.lock.json` | the lock changes with a reason from a person (`--allow`) |
85
+ | hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks change with a reason from a person (`--allow`) |
84
86
 
85
87
  The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is frozen too, and it is also an
86
88
  `evaluatorPaths` entry. A change to it needs `--allow "<reason>"` from a person. Without that, a change that
87
89
  only weakens a rule would pass, and the next change would be judged by the weakened contract.
88
90
 
89
- A person who needs to change a frozen path does it in a separate change that only touches that
90
- path, reviewed as such.
91
+ `--allow` clears only three of these: `contract-frozen`, `lock-frozen` and `hooks-frozen` (so a team can commit
92
+ shared hooks to a repo that already has its rules). A commit trailer clears none. Every other frozen path stays
93
+ frozen, even with `--allow`. To change one, a person first retires its mandate in a change of its own
94
+ (it edits the contract, so it needs `--allow`), then makes the change, then puts the mandate back the
95
+ same way.
91
96
 
92
97
  ## What this is not
93
98
 
@@ -100,5 +105,4 @@ path, reviewed as such.
100
105
  - **Not a scanner.** No matchers, no findings, no false positives to triage: a path, a count, or a
101
106
  hash.
102
107
 
103
- Pair it with the `security` preset for the eight code-level invariants, and with the GitHub Action so
104
- the gate runs where the agent cannot switch it off.
108
+ Pair it with the GitHub Action so the gate runs where the agent cannot switch it off.
@@ -1,5 +1,5 @@
1
1
  {
2
- "$comment": "PromptWheel supply-chain guard-pack. Use via { \"extends\": [\"packs/supply-chain/pack.json\"] } in your promptwheel.config.json, or install it with `promptwheel init --preset supply-chain` (which also writes the mandates). Three counts the gate holds: a rise fails the gate unless the change carries a declared reason (`--allow \"<reason>\"` or a `Promptwheel-Allow:` trailer), which is recorded in the outcome record. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
2
+ "$comment": "PromptWheel supply-chain guard-pack. Install it with `promptwheel init --preset supply-chain`: that copies the pack to .promptwheel/packs/supply-chain/, writes these metrics into promptwheel.config.json with their commands pointed at the copy, and writes the mandates. The commands in this file run packs/supply-chain/..., which exists only in the PromptWheel repository itself, so `extends` on this file (here or on the copy under .promptwheel/packs/) reads every metric as unmeasurable, and the gate fails, in any other repository. Seven counts the gate holds: a rise fails the gate unless a person gives a reason with `--allow \"<reason>\"` (a commit trailer does not count), which is recorded in report.allowed. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
3
3
  "metrics": [
4
4
  {
5
5
  "$comment": "install-time script entries (preinstall, install, postinstall, prepare, prepublish, prepublishOnly, postpack) across every package.json: the step a supply-chain attack needs inside your repo",