promptwheel 0.8.0 → 0.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INTEGRATE.md +15 -8
- package/README.md +19 -15
- package/bin/promptwheel.d.ts +27 -9
- package/bin/promptwheel.mjs +307 -97
- package/package.json +1 -1
- package/packs/supply-chain/README.md +16 -12
- package/packs/supply-chain/pack.json +1 -1
package/INTEGRATE.md
CHANGED
|
@@ -50,7 +50,7 @@ importing this package.
|
|
|
50
50
|
"gamed": true,
|
|
51
51
|
"sourceOnly": null,
|
|
52
52
|
"retained": 0.0,
|
|
53
|
-
"gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
|
|
53
|
+
"gamingReason": "the \"win\" changed zero production-source files, only test/config/grader/golden"
|
|
54
54
|
}
|
|
55
55
|
]
|
|
56
56
|
}
|
|
@@ -68,11 +68,18 @@ metric's gain that survived the source-only re-run (Section 3). `confidence`
|
|
|
68
68
|
(`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` are informational fields this
|
|
69
69
|
implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
|
|
70
70
|
truncated SHA-256 fingerprint of the gate definition (metrics, guards, `evaluatorPaths`,
|
|
71
|
-
`gamingThreshold`, `mandates`
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
71
|
+
`gamingThreshold`, `mandates`, `setup`, `env`, linked dirs; each metric with its `cmd` and how it
|
|
72
|
+
is read) at base and head, sets `gate.changed: true` when they differ, and lists in
|
|
73
|
+
`gate.weakened` any guard the change removed, disabled, or measured another way
|
|
74
|
+
(`change: guard-edited`, such as `cmd: "true"`; `change: measurement-edited`, a new `setup`, `env`
|
|
75
|
+
or `linkDirs`) to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
|
|
76
|
+
cannot pass by disabling the guard it is judged by. A change that adds the first config is
|
|
77
|
+
compared with the zero-config gate (the detected test command). `allowed` (optional) is `{ reason, cleared }` when a declared
|
|
78
|
+
reason turned a would-be `gamed`, `fail` or `inconclusive` into `pass`: `cleared` names each metric,
|
|
79
|
+
`gate`, or `mandate:<name>` it cleared, and each cleared metric carries `allowed: true` and keeps its
|
|
80
|
+
`gamingReason`, so a consumer can say what was allowed and why it would have failed.
|
|
81
|
+
|
|
82
|
+
Self-check your implementation against this schema with `conformsToContract` (see part 4 below).
|
|
76
83
|
|
|
77
84
|
---
|
|
78
85
|
|
|
@@ -99,8 +106,8 @@ measurement runs in a throwaway git worktree.
|
|
|
99
106
|
## 3. Import the library (in process)
|
|
100
107
|
|
|
101
108
|
Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
|
|
102
|
-
process and returns the Section 2 object. It never prints and never calls `process.exit`;
|
|
103
|
-
throws
|
|
109
|
+
process and returns the Section 2 object. It never prints and never calls `process.exit`; on a
|
|
110
|
+
precondition failure it throws an `Error` named `ConfigError` with `exitCode` 64. Types ship in the package
|
|
104
111
|
(`bin/promptwheel.d.ts`, hand-written, no build step): `import type { Report, CheckOptions, Config } from 'promptwheel'`
|
|
105
112
|
works in TypeScript, and editors complete the options and the report in plain JavaScript.
|
|
106
113
|
|
package/README.md
CHANGED
|
@@ -23,8 +23,10 @@ printf "import {test} from 'node:test'; import assert from 'node:assert'; import
|
|
|
23
23
|
git commit -qam 'make the suite green'
|
|
24
24
|
|
|
25
25
|
npx -y promptwheel@latest run --base HEAD~1 --head HEAD
|
|
26
|
-
#
|
|
27
|
-
#
|
|
26
|
+
# prints (shortened to the lines that matter; exit code 2):
|
|
27
|
+
# ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
|
|
28
|
+
# 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
|
|
29
|
+
# VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
|
|
28
30
|
```
|
|
29
31
|
|
|
30
32
|
What happened: PromptWheel measured `tests_pass` at both commits in throwaway git worktrees (0 then 1, a win), then split the diff into production source versus `{test, config, grader, golden}` files, re-applied **only the source slice** on a clean worktree at the base, and re-ran the metric. The win did not survive, so it is `GAMED`. No LLM, no network: a diff partition plus a re-run, reproducible in seconds with a human-readable reason.
|
|
@@ -67,7 +69,7 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
|
|
|
67
69
|
config, or grader, not the source, so revert and earn it in the source. Exit 1 means a guarded
|
|
68
70
|
metric regressed, revert that too. Never weaken, skip, or delete a test to go green.
|
|
69
71
|
|
|
70
|
-
**Two callers, by design.** Your *agent* consumes the verdict as a per-turn signal (`run --working --json`). The *harness* (a Stop-hook, or CI) runs the same check as an audit the agent cannot self-clear, because the check re-proves from the source alone and never asks the code to grade itself. See [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) for the
|
|
72
|
+
**Two callers, by design.** Your *agent* consumes the verdict as a per-turn signal (`run --working --json`). The *harness* (a Stop-hook, or CI) runs the same check as an audit the agent cannot self-clear, because the check re-proves from the source alone and never asks the code to grade itself. See [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) for the two places to enforce it.
|
|
71
73
|
|
|
72
74
|
## Config, `promptwheel.config.json`
|
|
73
75
|
|
|
@@ -88,18 +90,20 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
|
|
|
88
90
|
- **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
|
|
89
91
|
- **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
|
|
90
92
|
- **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
|
|
91
|
-
- **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
|
|
92
|
-
- **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
|
|
93
|
+
- **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires. Set on a metric that had the re-run on, it takes effect from the next change: the change that sets it is still re-proven.
|
|
94
|
+
- **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`. A change that lowers it is still judged by the value before it.
|
|
93
95
|
- **evaluatorPaths**, globs for files that *define* your gate (a threshold file, a budget, a CI limit). Editing one in the same diff that improves a metric is `GAMED` by declaration, deterministically. Opt-in, because only you know which files are the goalposts.
|
|
94
96
|
|
|
95
|
-
- **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. For counts where a rise is sometimes the job (the `supply-chain` preset
|
|
97
|
+
- **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. `"flag"` takes the reason from `--allow` (or `check({ allow })`) only, never a trailer, because an agent can write a trailer. For counts where a rise is sometimes the job (the `supply-chain` preset uses `"flag"`: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
|
|
96
98
|
|
|
97
99
|
Dependency and environment keys (config-level; `init` writes stack defaults):
|
|
98
100
|
|
|
99
|
-
- **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git.
|
|
101
|
+
- **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. With nothing set, a Node repo links its root `node_modules` and the `node_modules` of every workspace (any folder with a tracked `package.json`), so npm, yarn and pnpm workspaces are measured with no config. A `node_modules` that pnpm laid out, or that holds workspace links, is rebuilt in the worktree entry by entry, because pnpm refuses a symlinked one, and a tool's own state or cache folder inside it is left out. A link from `node_modules` back into the repo, which is how those tools wire a sibling package, is pointed at the worktree, so a test that imports a sibling package reads the ref being measured. `init` writes `[".venv"]` for Python and `[]` for Rust (`cargo test` writes to `target/`, so it is built fresh in the worktree rather than shared across refs). `"linkNodeModules": false` still works as shorthand for "link nothing".
|
|
100
102
|
- **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
|
|
101
103
|
- **setup**, an optional command run once per ref before metrics (for example `npm run build` or `pip install -e .`), after any source patch, so a build-gated suite measures the ref it is actually on.
|
|
102
104
|
|
|
105
|
+
These three shape every measurement, so they are part of the gate. A change that edits them is flagged as a gate change and is judged with the values from before it (see Gate integrity below).
|
|
106
|
+
|
|
103
107
|
Copy a config for your stack (pytest, tsc, coverage, bundle size, LLM eval) from [examples/](examples/).
|
|
104
108
|
|
|
105
109
|
### Guardrails and inheritance
|
|
@@ -116,7 +120,7 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
|
|
|
116
120
|
|
|
117
121
|
### Mandates (protected paths)
|
|
118
122
|
|
|
119
|
-
Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate
|
|
123
|
+
Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate**, however green everything else is. The metrics still run and are reported, but nothing they say clears a mandate:
|
|
120
124
|
|
|
121
125
|
```json
|
|
122
126
|
// promptwheel.config.json
|
|
@@ -126,7 +130,7 @@ Some rules are not a metric at all, they are a hard line: *never touch `src/auth
|
|
|
126
130
|
"metrics": [ /* ... */ ] }
|
|
127
131
|
```
|
|
128
132
|
|
|
129
|
-
A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the
|
|
133
|
+
A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the ones the base ref already had, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`), and one that narrows a mandate is checked against both versions (`edited: true`). A mandate added in the same change that touches its path does not apply to that change, on purpose: the commit that signs a contract adds the very files the contract protects. Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
|
|
130
134
|
|
|
131
135
|
### Supply chain (preset `supply-chain`)
|
|
132
136
|
|
|
@@ -136,7 +140,7 @@ The step every npm supply-chain attack needs inside a target repo is a supply-ch
|
|
|
136
140
|
npx promptwheel init --preset supply-chain
|
|
137
141
|
```
|
|
138
142
|
|
|
139
|
-
writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`,
|
|
143
|
+
writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, the copied scanners, the Claude Code hook files, the lock file, and the contract itself (`promptwheel.config.json`, `sawbuck.config.json`). A change that edits the contract fails (exit 1) unless a person passes `--allow "<reason>"`; the contract is also an `evaluatorPaths` entry, so editing it alongside a win is gamed by declaration. `--allow` clears only the contract, the lock and the hooks. Every other frozen path needs its mandate retired first, by a person, in a change of its own run with `--allow`. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
|
|
140
144
|
|
|
141
145
|
## In CI and in Claude Code
|
|
142
146
|
|
|
@@ -170,16 +174,16 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
|
|
|
170
174
|
|
|
171
175
|
## Trust model
|
|
172
176
|
|
|
173
|
-
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
|
|
177
|
+
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). A grader is recognized by its file name in a scripting language (`eval.py`, `grader.sh`, `score.js`, `scoring.ts`) or by an `evals/` or `grader(s)/` directory; a production file with one of those names (`src/score.py`) is treated as a grader too, so a fix there needs `--allow`, and a grader with any other name (`evaluate.py`, `check.py`) is source unless you list it in `evaluatorPaths`. On top of that:
|
|
174
178
|
|
|
175
179
|
- **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
|
|
176
|
-
- **Gate integrity.** The gate is resolved as of the base commit too
|
|
180
|
+
- **Gate integrity.** A change is judged by the gate that was in force before it. The gate is resolved as of the base commit too, through `extends`. A change that disables a guard, deletes a guarded metric, or measures it another way (a new `cmd` such as `true`, `extract`, `direction`, `exact` or `report`) in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed, or would still fail. If the old guard only passes because the same change also edited the test, it reads `GAMED` too: a win under the old definition still has to survive the source-only re-run. A file declared in `evaluatorPaths` before the change still counts as a goalpost for that change, even if the change drops the declaration. The same goes for `setup`, `env` and `linkDirs`: when a change edits them, each guard is measured again with the old values and so is the source-only re-run, so a `setup` that rewrites a test cannot carry a pass (`gate.weakened[].change: measurement-edited`). `gamingCheck: false` on a metric that had the re-run on, and a lower `gamingThreshold`, apply from the next change. A change that adds the first config is compared with the zero-config gate (the detected test command), so `tests_pass` set to `true` in a new config is an edited guard; a first config that keeps the detected command, or replaces the placeholder, is not. The gate definition (each metric with its command and how it is read, guards, `evaluatorPaths`, `gamingThreshold`, mandate names and paths, `setup`, `env`, linked dirs) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed. Two limits: a test that could not run at all before a `setup` fix reads `GAMED` on the change that adds the fix (say so with `--allow`; the `setup` that `init` writes into a first config is exempt), and a base config that `extends` a file outside the repository is not compared on `setup`, `env` or `linkDirs`.
|
|
177
181
|
- **A sanctioned override.** A commit trailer can be written by whoever writes the commit, including an agent, so `"allowTrailer": false` in the contract turns trailers off and only the `--allow` flag counts (the supply-chain preset sets this; the contract at the base ref decides, so a change cannot turn trailers back on for itself). When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. It covers a win whose source-only re-run could not be applied (`unprovable`, otherwise `inconclusive`) the same way. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
|
|
178
182
|
- **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
|
|
179
183
|
|
|
180
|
-
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test)
|
|
184
|
+
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test), or in a build script the unchanged `setup` runs (a change that makes that script rewrite a test is a source edit to this check). A base config that `extends` a file outside the repository cannot be compared across the change. These need held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
|
|
181
185
|
|
|
182
|
-
**Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A
|
|
186
|
+
**Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guarded pass/fail check that reads 0 at both refs (its command never ran, or it ran and failed both times; the report says which) makes the verdict `inconclusive`, never a fake pass.
|
|
183
187
|
|
|
184
188
|
**When not to use it.** If you only care that tests pass and nobody could be editing the tests, your plain CI already has you covered. PromptWheel earns its place when the thing writing the code can also touch the thing judging it, or when you have a graded metric beyond pass/fail (an eval score, cost per run, latency, size) that a change could quietly move. Single-metric CI gates (benchmark trackers, size limits) go deeper on one number; wrap them as a metric `cmd` and PromptWheel composes them into one verdict.
|
|
185
189
|
|
|
@@ -187,7 +191,7 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
|
|
|
187
191
|
|
|
188
192
|
A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly 2 to 3× your metric command's runtime plus about 40ms of worktree overhead. Point the per-turn gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for CI: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
|
|
189
193
|
|
|
190
|
-
**Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run
|
|
194
|
+
**Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run can leave a worktree in your temp dir; a later run removes it once it is a day old (an unregistered checkout of this repo after an hour). It removes only directories named the way it names its own (`promptwheel-`, `promptwheel-src-` or `pw-idx-`, then an 8-hex repo tag, then 6 random characters). Any other folder or worktree in the temp dir is left alone, at any age.
|
|
191
195
|
|
|
192
196
|
## Portability
|
|
193
197
|
|
package/bin/promptwheel.d.ts
CHANGED
|
@@ -48,7 +48,7 @@ export interface MetricConfig {
|
|
|
48
48
|
allowNonzero?: boolean;
|
|
49
49
|
/** Per-command timeout, default 300. */
|
|
50
50
|
timeoutSec?: number;
|
|
51
|
-
/** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). */
|
|
51
|
+
/** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). Set on a metric that had the re-proof on, it applies from the next change, not the one that sets it. */
|
|
52
52
|
gamingCheck?: boolean;
|
|
53
53
|
/** Per-metric override of `Config.gamingThreshold`. */
|
|
54
54
|
gamingThreshold?: number;
|
|
@@ -62,7 +62,7 @@ export interface MetricConfig {
|
|
|
62
62
|
allowable?: boolean;
|
|
63
63
|
}
|
|
64
64
|
|
|
65
|
-
/** A named hard rule: any change touching one of `paths` fails the gate
|
|
65
|
+
/** A named hard rule: any change touching one of `paths` fails the gate, whatever the metrics read. */
|
|
66
66
|
export interface Mandate {
|
|
67
67
|
name: string;
|
|
68
68
|
/** Globs: a bare name matches that basename at any depth, a pattern with `/` is anchored at the repo root, `**` crosses directories. */
|
|
@@ -85,7 +85,7 @@ export interface Config {
|
|
|
85
85
|
env?: Record<string, string>;
|
|
86
86
|
/** A per-ref build or install, run in the worktree before measuring (a failed setup leaves the metric inert, never a false green). */
|
|
87
87
|
setup?: string;
|
|
88
|
-
/** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. */
|
|
88
|
+
/** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. A change that lowers it is still judged by the value before it. */
|
|
89
89
|
gamingThreshold?: number;
|
|
90
90
|
/** Extra files that define the gate (a threshold, a budget); a change to one alongside a win is gamed by declaration. Same glob rules as `Mandate.paths`. */
|
|
91
91
|
evaluatorPaths?: string[];
|
|
@@ -144,6 +144,12 @@ export interface MetricResult extends Evaluation {
|
|
|
144
144
|
name: string;
|
|
145
145
|
direction: Direction;
|
|
146
146
|
guard: boolean;
|
|
147
|
+
/** With `inert`: where the command ran and failed (`both` refs, or only `base` or `head`). Absent when it ran at neither: exit 126 or 127, a timeout, or the placeholder command `init` writes. */
|
|
148
|
+
ran?: 'both' | 'base' | 'head';
|
|
149
|
+
/** With `inert`: a hint read from the command's output. `missing-dependency` when it names a module, package or command the clean copy does not have, else `failing`. Never part of a verdict. */
|
|
150
|
+
cause?: 'missing-dependency' | 'failing' | 'too-much-output';
|
|
151
|
+
/** With `cause: 'missing-dependency'`: the line that names what is missing, at most 200 characters. */
|
|
152
|
+
detail?: string;
|
|
147
153
|
/** `true` when the win did not survive the source-only re-run; `null` when the source-only patch could not be applied (then `unprovable` is set and the verdict is `inconclusive`, never `pass`). Absent without gaming detection or when the metric did not improve. */
|
|
148
154
|
gamed?: boolean | null;
|
|
149
155
|
/** `true`: the source-only patch could not be applied, so this win could not be re-proven and the verdict is `inconclusive`. `false` only after a declared override cleared it (then `allowed` is `true`). */
|
|
@@ -174,9 +180,13 @@ export interface MandateViolation {
|
|
|
174
180
|
|
|
175
181
|
export interface GateWeakening {
|
|
176
182
|
name: string;
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
183
|
+
/**
|
|
184
|
+
* `guard-edited`: the guard is still on but measured another way (a new `cmd`, `extract`, `direction`, `exact` or `report`).
|
|
185
|
+
* `measurement-edited`: the config's `setup`, `env` or linked dirs changed, and measured with the ones before the change the guard fails at head while the new ones read it fine.
|
|
186
|
+
*/
|
|
187
|
+
change: 'guard-removed' | 'guard-disabled' | 'guard-edited' | 'measurement-edited';
|
|
188
|
+
/** What the guard would have read at head under its base definition (and the base `setup`, `env` and linked dirs); `failed` when it still fails there. */
|
|
189
|
+
status: MetricStatus | 'failed';
|
|
180
190
|
}
|
|
181
191
|
|
|
182
192
|
export interface EvaluatorChange {
|
|
@@ -188,11 +198,11 @@ export interface EvaluatorChange {
|
|
|
188
198
|
|
|
189
199
|
/** The gate-integrity record, present when gaming detection ran. */
|
|
190
200
|
export interface GateRecord {
|
|
191
|
-
/** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates) at each ref; `base` is `null` when there was no config at base. */
|
|
201
|
+
/** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) at each ref; `base` is `null` when there was no config at base. */
|
|
192
202
|
fingerprint: { base: string | null; head: string };
|
|
193
203
|
/** The gate definition differs between base and head. */
|
|
194
204
|
changed?: true;
|
|
195
|
-
/** Guards the change dropped that would have failed at head; non-empty forces `gamed` unless allowed. */
|
|
205
|
+
/** Guards the change dropped, edited or measures another way that would have failed at head; non-empty forces `gamed` unless allowed. A change that adds the first config is compared with the zero-config gate (the detected test command). */
|
|
196
206
|
weakened?: GateWeakening[];
|
|
197
207
|
/** Declared `evaluatorPaths` files whose content changed in this diff. */
|
|
198
208
|
evaluatorChanged?: EvaluatorChange[];
|
|
@@ -228,9 +238,17 @@ export interface Report {
|
|
|
228
238
|
mandates?: MandateViolation[];
|
|
229
239
|
gate?: GateRecord;
|
|
230
240
|
allowed?: AllowedRecord;
|
|
241
|
+
/** Present only when the `setup` command failed: at which ref, with what exit code, and its last output line. */
|
|
242
|
+
setup?: { base?: SetupFailure; head?: SetupFailure };
|
|
231
243
|
provenance: Provenance;
|
|
232
244
|
}
|
|
233
245
|
|
|
246
|
+
export interface SetupFailure {
|
|
247
|
+
/** The setup command's exit code, or `null` when it was killed (a timeout). */
|
|
248
|
+
exit: number | null;
|
|
249
|
+
lastLine: string;
|
|
250
|
+
}
|
|
251
|
+
|
|
234
252
|
// ---------------------------------------------------------------------------
|
|
235
253
|
// Public library API (stable across 0.x minors)
|
|
236
254
|
// ---------------------------------------------------------------------------
|
|
@@ -260,7 +278,7 @@ export declare function deriveVerdict(metrics: ReadonlyArray<Partial<MetricResul
|
|
|
260
278
|
* a missing `extends` target, no commits yet, a base or head ref that is not in this clone, no
|
|
261
279
|
* commit to compare against) throws an Error whose `name` is `'ConfigError'` and whose `exitCode`
|
|
262
280
|
* is 64, the code the CLI exits with for the same condition. A `repo` that names a subdirectory
|
|
263
|
-
* is resolved to the repository root, where the config lives
|
|
281
|
+
* is resolved to the repository root, where the config lives. A run writes nothing to the repo.
|
|
264
282
|
*/
|
|
265
283
|
export declare function check(options?: CheckOptions): Report;
|
|
266
284
|
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -14,9 +14,9 @@
|
|
|
14
14
|
|
|
15
15
|
import { execSync, execFileSync } from 'node:child_process';
|
|
16
16
|
import { createHash } from 'node:crypto';
|
|
17
|
-
import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync,
|
|
17
|
+
import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, lstatSync, copyFileSync } from 'node:fs';
|
|
18
18
|
import { tmpdir } from 'node:os';
|
|
19
|
-
import { join, dirname, isAbsolute, relative } from 'node:path';
|
|
19
|
+
import { join, dirname, basename, isAbsolute, relative, sep } from 'node:path';
|
|
20
20
|
import { fileURLToPath } from 'node:url';
|
|
21
21
|
|
|
22
22
|
// ---------------------------------------------------------------------------
|
|
@@ -36,6 +36,9 @@ const spread = (xs) => { const a = xs.filter((x) => x != null); return a.length
|
|
|
36
36
|
// repo is still measuring in (a long gate elsewhere must not lose its worktree mid-run)
|
|
37
37
|
function tmpOwnerTag(repo) { let r = repo; try { r = realpathSync(repo); } catch { /* keep as given */ } return createHash('sha1').update(String(r)).digest('hex').slice(0, 8); }
|
|
38
38
|
const tmpPrefix = (repo, kind) => `${kind}-${tmpOwnerTag(repo)}-`;
|
|
39
|
+
// exactly the names mkdtemp makes from tmpPrefix: the kind, the 8-hex owner tag, then mkdtemp's 6 random
|
|
40
|
+
// characters. Self-heal removes nothing else, so a folder that only starts with "promptwheel-" is never ours.
|
|
41
|
+
const ENGINE_TMP = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-[A-Za-z0-9]{6}$/;
|
|
39
42
|
|
|
40
43
|
// usage / config errors exit 64 (EX_USAGE), so they can never be confused with a verdict
|
|
41
44
|
// (0 pass, 1 fail, 2 gamed, 3 inconclusive): in CI a config typo must not read as GAMED.
|
|
@@ -112,8 +115,14 @@ const hasConfigFile = (repo) => existsSync(join(repo, 'promptwheel.config.json')
|
|
|
112
115
|
// measurement actually runs in a throwaway worktree. Lets `run` and `check()` work with no config
|
|
113
116
|
// file; `init` is for customizing. A repo with no test command gets a sentinel that reads inert ->
|
|
114
117
|
// INCONCLUSIVE (never a fake green), with a message pointing at the fix.
|
|
115
|
-
function autoConfig(repo) {
|
|
116
|
-
return { ...detectEnv(repo), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }] };
|
|
118
|
+
function autoConfig(repo, at = stackAt(repo)) {
|
|
119
|
+
return { ...detectEnv(repo, at), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo, at), extract: 'exit', direction: 'pass', guard: true }] };
|
|
120
|
+
}
|
|
121
|
+
// where stack detection looks: the working tree, or the tracked files as of `ref` (the repo as it was before a change)
|
|
122
|
+
function stackAt(repo, ref) {
|
|
123
|
+
if (!ref) return { has: (f) => existsSync(join(repo, f)), read: (f) => readFileSync(join(repo, f), 'utf8') };
|
|
124
|
+
const read = (f) => execFileSync('git', ['show', `${ref}:${f}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] });
|
|
125
|
+
return { has: (f) => { try { read(f); return true; } catch { return false; } }, read };
|
|
117
126
|
}
|
|
118
127
|
|
|
119
128
|
function resolveBase(repo, base) {
|
|
@@ -123,7 +132,8 @@ function resolveBase(repo, base) {
|
|
|
123
132
|
catch { bad('promptwheel: no commits yet, make an initial commit before gating'); }
|
|
124
133
|
for (const b of ['origin/main', 'origin/master', 'main', 'master']) {
|
|
125
134
|
let mb = '';
|
|
126
|
-
|
|
135
|
+
// stderr ignored: a ref that is not there (no origin/main in this clone) is expected, not an error to print
|
|
136
|
+
try { mb = execFileSync('git', ['merge-base', 'HEAD', b], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } catch { continue; }
|
|
127
137
|
// on the trunk itself the merge-base IS HEAD; comparing a commit to itself would be a
|
|
128
138
|
// trivial PASS that verified nothing, so fall through to the last commit instead.
|
|
129
139
|
if (mb && mb !== head) return mb;
|
|
@@ -132,15 +142,32 @@ function resolveBase(repo, base) {
|
|
|
132
142
|
catch { bad('promptwheel: no commit to compare against (one commit, or a shallow clone): pass --base, or --working'); }
|
|
133
143
|
}
|
|
134
144
|
|
|
135
|
-
|
|
136
|
-
|
|
145
|
+
// exit statuses of a sample list, kept beside it: 127 (the shell found no such command), 126 (it could not
|
|
146
|
+
// execute it) or no status (killed: a timeout) mean the metric never ran, which needs a different fix from a
|
|
147
|
+
// test that ran and failed
|
|
148
|
+
const exitCodes = new WeakMap();
|
|
149
|
+
const neverRan = (samples) => { const c = exitCodes.get(samples) || []; return c.length > 0 && c.every((x) => x == null || x === 126 || x === 127); };
|
|
150
|
+
// Why a metric's command failed, read from its output: a dependency the clean copy does not have reads differently
|
|
151
|
+
// from tests that simply fail. A hint for the person, never part of a verdict.
|
|
152
|
+
const MISSING_DEP = /Cannot find (?:module|package) ['"][^'"\n]+['"]|ERR_MODULE_NOT_FOUND[^\n]*|ModuleNotFoundError: [^\n]+|No module named [^\n]+|error TS2307: [^\n]+|[^\n:]*: (?:command )?not found[^\n]*|can't find crate[^\n]*|cannot find package [^\n]+/i;
|
|
153
|
+
function whyFailed(text) {
|
|
154
|
+
const m = String(text || '').replace(/\x1b\[[0-9;]*m/g, '').match(MISSING_DEP);
|
|
155
|
+
return m ? { cause: 'missing-dependency', detail: m[0].trim().slice(0, 200) } : { cause: 'failing' };
|
|
156
|
+
}
|
|
157
|
+
const failNotes = new WeakMap(); // samples -> why the last failing run failed
|
|
158
|
+
function runMetric(cwd, m, env, codes, notes) {
|
|
159
|
+
let stdout = '', code = 0, status = 0;
|
|
137
160
|
// Never leak the harness's own node:test context into the measured gate: a metric like
|
|
138
161
|
// `node --test` would see NODE_TEST_CONTEXT, act as a child reporter, and report a bogus
|
|
139
162
|
// exit code (so the gate can't be trusted when PromptWheel itself runs under a test runner).
|
|
140
163
|
const childEnv = { ...(env || process.env) }; delete childEnv.NODE_TEST_CONTEXT;
|
|
141
164
|
try {
|
|
142
|
-
|
|
143
|
-
|
|
165
|
+
// maxBuffer: the default is 1 MB, and a suite that prints more than that was killed and read as failing. A real
|
|
166
|
+
// repo found it (tap prints every assertion when its output is a pipe): the check said "your tests fail" about
|
|
167
|
+
// a suite that passes.
|
|
168
|
+
stdout = execSync(m.cmd, { cwd, env: childEnv, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000, maxBuffer: 1024 * 1024 * 1024 });
|
|
169
|
+
} catch (e) { status = e.status ?? null; code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; if (notes) notes.push(e.code === 'ENOBUFS' ? { cause: 'too-much-output', detail: 'the command printed more than 1 GB, so it was stopped before it finished' } : whyFailed(`${e.stderr || ''}\n${e.stdout || ''}`)); }
|
|
170
|
+
if (codes) codes.push(status);
|
|
144
171
|
return extract(stdout, code, m.extract, m.allowNonzero);
|
|
145
172
|
}
|
|
146
173
|
|
|
@@ -169,12 +196,63 @@ function extract(stdout, code, mode, allowNonzero) {
|
|
|
169
196
|
// with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
|
|
170
197
|
// Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
|
|
171
198
|
// not here, the engine stays ecosystem-agnostic.
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
199
|
+
const linkDirsOf = (cfg) => (Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']));
|
|
200
|
+
// Every folder that has a tracked package.json and a node_modules of its own: the workspaces of a monorepo, whichever
|
|
201
|
+
// tool laid them out. Asked of git, so nothing walks the tree.
|
|
202
|
+
function workspaceModuleDirs(repo) {
|
|
203
|
+
let files = '';
|
|
204
|
+
try { files = execFileSync('git', ['ls-files', '-z', '--', '*/package.json'], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'], maxBuffer: 64 * 1024 * 1024 }); } catch { return []; }
|
|
205
|
+
const out = [];
|
|
206
|
+
for (const f of files.split('\0')) {
|
|
207
|
+
if (!f || f.split('/').includes('node_modules')) continue;
|
|
208
|
+
const d = join(dirname(f), 'node_modules');
|
|
209
|
+
if (existsSync(join(repo, d))) out.push(d);
|
|
210
|
+
if (out.length >= 500) break;
|
|
211
|
+
}
|
|
212
|
+
return out;
|
|
213
|
+
}
|
|
214
|
+
// A node_modules folder in a workspaces repo holds links back into the repo: npm and yarn link each workspace package
|
|
215
|
+
// from the root, pnpm from the workspace that depends on it. One symlink to the real folder would make a test in the
|
|
216
|
+
// worktree import a sibling package from the working tree, at both refs, so a regression there would never show. Such
|
|
217
|
+
// a folder is rebuilt in the worktree instead: each entry linked on its own, and an entry that points back into the
|
|
218
|
+
// repo pointed at the same path in the worktree. A folder with no such entry, in a repo pnpm did not lay out, keeps
|
|
219
|
+
// the single link it always had.
|
|
220
|
+
function linkModules(repo, wt, d) {
|
|
221
|
+
const src = join(repo, d), dst = join(wt, d);
|
|
222
|
+
const entries = [];
|
|
223
|
+
try {
|
|
224
|
+
for (const e of readdirSync(src, { withFileTypes: true })) {
|
|
225
|
+
if (e.name.startsWith('@') && e.isDirectory()) { for (const s of readdirSync(join(src, e.name), { withFileTypes: true })) entries.push([join(e.name, s.name), s]); }
|
|
226
|
+
else entries.push([e.name, e]);
|
|
227
|
+
}
|
|
228
|
+
} catch { return; }
|
|
229
|
+
const back = (name, e) => {
|
|
230
|
+
if (!e.isSymbolicLink()) return null;
|
|
231
|
+
try { const r = relative(repo, realpathSync(join(src, name))); return r && !r.startsWith('..') && !isAbsolute(r) && !r.split(sep).includes('node_modules') ? r : null; } catch { return null; }
|
|
232
|
+
};
|
|
233
|
+
const links = entries.map(([name, e]) => [name, back(name, e), e]);
|
|
234
|
+
// pnpm refuses a node_modules, or its own state folder inside it, that is a symlink: its scripts stop at once
|
|
235
|
+
const pnpm = entries.some(([name]) => name === '.pnpm' || name === '.modules.yaml');
|
|
236
|
+
try {
|
|
237
|
+
if (!pnpm && !links.some(([, rel]) => rel)) { symlinkSync(src, dst); return; }
|
|
238
|
+
mkdirSync(dst, { recursive: true });
|
|
239
|
+
for (const [name, rel, e] of links) {
|
|
240
|
+
// a tool's own state or cache folder (pnpm's task-run state, .cache, .vite) is written to, so each worktree gets
|
|
241
|
+
// its own, made by the tool. The store (.pnpm) and the executables (.bin) are read, so they are linked.
|
|
242
|
+
if (name.startsWith('.') && name !== '.bin' && name !== '.pnpm' && e.isDirectory()) continue;
|
|
243
|
+
if (name.includes(sep)) mkdirSync(dirname(join(dst, name)), { recursive: true });
|
|
244
|
+
symlinkSync(rel ? join(wt, rel) : join(src, name), join(dst, name));
|
|
177
245
|
}
|
|
246
|
+
} catch { /* best effort */ }
|
|
247
|
+
}
|
|
248
|
+
function bridgeEnv(repo, wt, cfg) {
|
|
249
|
+
let root = repo; try { root = realpathSync(repo); } catch { /* keep the path as given */ }
|
|
250
|
+
// with nothing declared, a Node repo links its root node_modules and each workspace's own
|
|
251
|
+
const dirs = (Array.isArray(cfg.linkDirs) || cfg.linkNodeModules === false) ? linkDirsOf(cfg) : ['node_modules', ...workspaceModuleDirs(root)];
|
|
252
|
+
for (const d of dirs) {
|
|
253
|
+
if (!existsSync(join(root, d)) || existsSync(join(wt, d)) || !existsSync(dirname(join(wt, d)))) continue;
|
|
254
|
+
if (basename(d) === 'node_modules') linkModules(root, wt, d);
|
|
255
|
+
else { try { symlinkSync(join(root, d), join(wt, d)); } catch { /* best effort */ } }
|
|
178
256
|
}
|
|
179
257
|
if (!cfg.env) return process.env;
|
|
180
258
|
return { ...process.env, ...Object.fromEntries(Object.entries(cfg.env).map(([k, v]) => [k, String(v).replaceAll('{wt}', () => wt)])) }; // fn replacement: a literal wt avoids $-pattern interpretation
|
|
@@ -182,10 +260,16 @@ function bridgeEnv(repo, wt, cfg) {
|
|
|
182
260
|
// an optional per-ref build/install (e.g. `npm run build`, `pip install -e .`). Best-effort: a
|
|
183
261
|
// failed setup just leaves the metric inert → INCONCLUSIVE (never a false green). Run by the
|
|
184
262
|
// caller AFTER any source patch, so it builds the tree actually being measured.
|
|
263
|
+
// Returns nothing when setup worked, or { exit, lastLine } when it did not, so the report can say so.
|
|
185
264
|
function runSetup(wt, cfg, env) {
|
|
186
|
-
if (!cfg.setup) return;
|
|
187
|
-
try { execSync(cfg.setup, { cwd: wt, env, stdio: 'ignore', timeout: 300_000
|
|
265
|
+
if (!cfg.setup) return undefined;
|
|
266
|
+
try { execSync(cfg.setup, { cwd: wt, env, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: 300_000, maxBuffer: 1024 * 1024 * 1024 }); return undefined; }
|
|
267
|
+
catch (e) {
|
|
268
|
+
const lines = `${e.stdout || ''}\n${e.stderr || ''}`.replace(/\x1b\[[0-9;]*m/g, '').split('\n').map((l) => l.trim()).filter(Boolean);
|
|
269
|
+
return { exit: e.status ?? null, lastLine: (lines[lines.length - 1] || '').slice(0, 200) }; // the metric goes inert → inconclusive
|
|
270
|
+
}
|
|
188
271
|
}
|
|
272
|
+
const setupFails = new WeakMap(); // a measureAt result -> how its setup failed
|
|
189
273
|
|
|
190
274
|
// measure every metric `repeat` times at a ref, in a throwaway worktree (never touches your tree)
|
|
191
275
|
function measureAt(repo, ref, cfg, repeat) {
|
|
@@ -193,11 +277,14 @@ function measureAt(repo, ref, cfg, repeat) {
|
|
|
193
277
|
git(['worktree', 'add', '--quiet', '--detach', wt, ref], repo);
|
|
194
278
|
try {
|
|
195
279
|
const env = bridgeEnv(repo, wt, cfg);
|
|
196
|
-
runSetup(wt, cfg, env);
|
|
280
|
+
const setupFailed = runSetup(wt, cfg, env);
|
|
197
281
|
const out = {};
|
|
282
|
+
if (setupFailed) setupFails.set(out, setupFailed);
|
|
198
283
|
for (const m of cfg.metrics) {
|
|
199
|
-
const samples = [];
|
|
200
|
-
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
|
|
284
|
+
const samples = [], codes = [], notes = [];
|
|
285
|
+
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env, codes, notes));
|
|
286
|
+
exitCodes.set(samples, codes);
|
|
287
|
+
if (notes.length) failNotes.set(samples, notes[notes.length - 1]);
|
|
201
288
|
out[m.name] = samples;
|
|
202
289
|
}
|
|
203
290
|
return out;
|
|
@@ -266,9 +353,12 @@ const NON_SOURCE = [
|
|
|
266
353
|
/_spec\.rb$/i,
|
|
267
354
|
/[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
|
|
268
355
|
/\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
|
|
269
|
-
// a grader/eval/score FILE by name (eval.
|
|
270
|
-
// evaluate.ts, evaluation.
|
|
271
|
-
|
|
356
|
+
// a grader/eval/score FILE by name in a scripting language (eval.py, grader-x.js, grade.sh, scoring.ts), never a
|
|
357
|
+
// prefix: evaluate.ts, evaluation.py, scoreboard.js stay source. Compiled names stay source too (eval.c, eval.go
|
|
358
|
+
// are interpreters; Score.java is a type), and so do .tsx/.vue files (a Score component, a /score page).
|
|
359
|
+
/(^|\/)(eval|evals|grade|grader|graders|score|scorer|scoring)([._-][^/]*)?\.([cm]?[jt]s|py|sh|bash|zsh|rb|pl|php|lua|r|jl)$/i,
|
|
360
|
+
// anything under an evals/ or grader(s)/ directory. Not eval/ (an interpreter's package) or scoring/ (a product module).
|
|
361
|
+
/(^|\/)(evals|graders?)\//i,
|
|
272
362
|
// config files only, require a .config/.conf/.setup segment so PRODUCTION source that
|
|
273
363
|
// happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
|
|
274
364
|
// the source slice (a false sweep here can flag an honest win as GAMED).
|
|
@@ -311,9 +401,9 @@ function evaluatorMatcher(cfg) {
|
|
|
311
401
|
// forbids ANY change to its paths (same glob semantics as evaluatorPaths). A hit fails the gate
|
|
312
402
|
// regardless of metrics and rides at the top of the report; mandates inherit through `extends`,
|
|
313
403
|
// so a parent scope's rule holds for every repo beneath it.
|
|
314
|
-
// The mandates in force are the
|
|
315
|
-
//
|
|
316
|
-
// clear it. A mandate removed at head but violated in the diff is reported with `removed: true`; one
|
|
404
|
+
// The mandates in force are the base ref's (`baseMandates`, resolved by gateDefAtRef), in their head
|
|
405
|
+
// version when both have them: deleting a mandate in the same change that violates it does not
|
|
406
|
+
// clear it, and one the change adds does not apply to it (see below). A mandate removed at head but violated in the diff is reported with `removed: true`; one
|
|
317
407
|
// whose paths were edited at head (same name, different paths) is tested in BOTH versions and reported
|
|
318
408
|
// with `edited: true`, so narrowing a rule in the change that breaks it does not clear it either.
|
|
319
409
|
// The honest way to retire one is a separate change that only edits the config.
|
|
@@ -482,12 +572,16 @@ function workingSnapshot(repo) {
|
|
|
482
572
|
}
|
|
483
573
|
}
|
|
484
574
|
|
|
485
|
-
// self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
|
|
575
|
+
// self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
|
|
486
576
|
// its `finally` never ran. On the next run, drop stale worktree registry entries and any
|
|
487
|
-
// abandoned /tmp checkout, so nothing accumulates in your repo or temp dir.
|
|
488
|
-
//
|
|
489
|
-
//
|
|
490
|
-
//
|
|
577
|
+
// abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. It only ever removes a
|
|
578
|
+
// directory whose name is exactly what this engine generates (ENGINE_TMP): a folder named
|
|
579
|
+
// promptwheel-notes, or another repo's worktree named promptwheel-site-wt, is not ours at any age.
|
|
580
|
+
// Ages, by dir mtime: this repo's unregistered checkout after 1h; this repo's still-registered worktree
|
|
581
|
+
// after a full day (a killed run leaves it registered, and a younger one may be a concurrent gate still
|
|
582
|
+
// measuring); a dir tagged for ANOTHER repo after a full day (an orphan of a repo that was deleted or
|
|
583
|
+
// moved: a new real path is a new tag nobody will ever claim). A registered worktree that is not
|
|
584
|
+
// tagged for this repo is never touched: it may be yours.
|
|
491
585
|
function selfHeal(repo) {
|
|
492
586
|
try {
|
|
493
587
|
git(['worktree', 'prune'], repo); // remove registry entries whose dir is already gone (always safe)
|
|
@@ -496,16 +590,18 @@ function selfHeal(repo) {
|
|
|
496
590
|
.split('\n').filter((l) => l.startsWith('worktree ')).map((l) => l.slice(9).trim()),
|
|
497
591
|
);
|
|
498
592
|
const tmp = tmpdir(), now = Date.now(), mine = tmpOwnerTag(repo);
|
|
499
|
-
const TAGGED = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-/;
|
|
500
593
|
for (const name of readdirSync(tmp)) {
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
594
|
+
const made = ENGINE_TMP.exec(name);
|
|
595
|
+
if (!made) continue; // not a name this engine generates (worktrees + temp-index dirs): never removed
|
|
596
|
+
const tag = made[2];
|
|
597
|
+
// ours after an hour; another repo's only after a full day
|
|
505
598
|
const p = join(tmp, name);
|
|
506
|
-
if (tracked.has(p)) continue;
|
|
507
|
-
|
|
599
|
+
if (tracked.has(p) && tag !== mine) continue; // a registered worktree this repo's gate did not make
|
|
600
|
+
const maxAge = tag === mine && !tracked.has(p) ? 3600_000 : 86_400_000;
|
|
601
|
+
// lstat: the engine makes directories, so a file or a symlink under one of its names is left alone
|
|
602
|
+
try { const st = lstatSync(p); if (st.isDirectory() && st.mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
|
|
508
603
|
}
|
|
604
|
+
git(['worktree', 'prune'], repo); // drop the registry entries of anything removed above
|
|
509
605
|
} catch { /* cleanup must never block the gate */ }
|
|
510
606
|
}
|
|
511
607
|
|
|
@@ -537,18 +633,25 @@ function warnEditableInstall(repo) {
|
|
|
537
633
|
// guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
|
|
538
634
|
// guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
|
|
539
635
|
// that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
|
|
540
|
-
const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer'];
|
|
636
|
+
const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer', 'setup', 'env', 'linkDirs', 'linkNodeModules'];
|
|
637
|
+
// what runs or shapes every measurement besides the metric's own command. A setup can rewrite a test and an
|
|
638
|
+
// env var can skip one, so these are part of the gate: fingerprinted, and held at their base values for the
|
|
639
|
+
// change that edits them (see detectGamingPass).
|
|
640
|
+
const measureOf = (s) => ({ setup: s.setup, env: s.env, linkDirs: s.linkDirs, linkNodeModules: s.linkNodeModules });
|
|
641
|
+
const measuredWith = (s) => JSON.stringify([s.setup ?? null, Object.entries(s.env || {}).map(([k, v]) => `${k}=${v}`).sort(), [...linkDirsOf(s)].sort()]);
|
|
541
642
|
|
|
542
643
|
// Resolve the gate definition as of `ref` (metrics + mandates + scalars), following `extends` by
|
|
543
644
|
// reading every file from that ref via `git show`, mirroring resolveConfig (a parent's mandates
|
|
544
645
|
// hold beneath it; a local mandate of the same name overrides). Returns { metrics:Map, scalars }
|
|
545
|
-
// with scalars.mandates, or null when there is no config at `ref` (
|
|
546
|
-
//
|
|
547
|
-
// branch simply contributes nothing)
|
|
646
|
+
// with scalars.mandates, or null when there is no config at `ref` (gate() then falls back to the
|
|
647
|
+
// zero-config gate the repo ran under, see zeroConfigGate). Never exits: any missing/unreadable file or
|
|
648
|
+
// parse error yields null (that branch simply contributes nothing), and `partial` says an extends target
|
|
649
|
+
// could not be read at `ref` (it lives outside the repo), so how the base measured is not fully known.
|
|
548
650
|
function gateDefAtRef(repo, ref) {
|
|
549
651
|
// quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
|
|
550
652
|
const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
|
|
551
653
|
const seen = new Set();
|
|
654
|
+
let partial = false;
|
|
552
655
|
const resolve = (p) => {
|
|
553
656
|
if (seen.has(p)) return null;
|
|
554
657
|
seen.add(p);
|
|
@@ -557,9 +660,11 @@ function gateDefAtRef(repo, ref) {
|
|
|
557
660
|
let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
|
|
558
661
|
const byName = new Map(); const mandatesByName = new Map(); const scalars = {};
|
|
559
662
|
for (const ext of [].concat(cfg.extends || [])) {
|
|
560
|
-
if (isAbsolute(ext)) continue;
|
|
561
|
-
const
|
|
562
|
-
|
|
663
|
+
if (isAbsolute(ext)) { partial = true; continue; } // outside the repo, cannot git-show; unchanged anyway
|
|
664
|
+
const bp = join(dirname(p), ext).split('\\').join('/');
|
|
665
|
+
const again = seen.has(bp); // a diamond: already merged, not a missing piece
|
|
666
|
+
const base = resolve(bp);
|
|
667
|
+
if (!base) { if (!again) partial = true; continue; }
|
|
563
668
|
for (const m of base.metrics.values()) byName.set(m.name, m);
|
|
564
669
|
for (const md of base.scalars.mandates) mandatesByName.set(md.name, md);
|
|
565
670
|
for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
|
|
@@ -570,18 +675,35 @@ function gateDefAtRef(repo, ref) {
|
|
|
570
675
|
scalars.mandates = [...mandatesByName.values()];
|
|
571
676
|
return { metrics: byName, scalars };
|
|
572
677
|
};
|
|
573
|
-
|
|
678
|
+
const def = resolve('promptwheel.config.json');
|
|
679
|
+
return def && { ...def, partial };
|
|
680
|
+
}
|
|
681
|
+
|
|
682
|
+
// The gate a repo with no config ran under: the detected test command (source-only re-run on) with the
|
|
683
|
+
// stack's env and linkDirs, detected from the tracked files as of `ref`. A change that adds the first
|
|
684
|
+
// config is judged against this, so `tests_pass.cmd: "true"` in a new config is an edited guard, not a
|
|
685
|
+
// fresh start. `initSetup` is the setup `init` would add: a first config that carries exactly it has not
|
|
686
|
+
// changed how the gate measures. With no test command to detect (the placeholder) the old gate measured
|
|
687
|
+
// nothing, so its setup is no baseline (`partial`).
|
|
688
|
+
function zeroConfigGate(repo, ref) {
|
|
689
|
+
const at = stackAt(repo, ref), auto = autoConfig(repo, at);
|
|
690
|
+
return { metrics: new Map(auto.metrics.map((m) => [m.name, m])), scalars: { mandates: [], ...measureOf(auto) }, zeroConfig: true, initSetup: detectSetup(repo, at), partial: auto.metrics[0].cmd === PLACEHOLDER_TEST };
|
|
574
691
|
}
|
|
575
692
|
|
|
693
|
+
// how a metric is measured and judged: change any of these on a guard and it is a different guard
|
|
694
|
+
// what init writes when it finds no test command. It guards nothing, so replacing it is setup, not a weakened gate.
|
|
695
|
+
const PLACEHOLDER_TEST = 'echo "set your test command in promptwheel.config.json" && false';
|
|
696
|
+
const measuredBy = (m) => JSON.stringify([m.cmd ?? null, m.extract ?? 'number', m.direction || 'up', m.exact === true, m.report === true]);
|
|
576
697
|
// an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
|
|
577
698
|
function gateFingerprint(metricsList, scalars) {
|
|
578
699
|
const norm = {
|
|
579
700
|
metrics: metricsList.map((m) => ({
|
|
580
|
-
name: m.name, guard: m.guard === true,
|
|
701
|
+
name: m.name, guard: m.guard === true, measuredBy: measuredBy(m),
|
|
581
702
|
gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
|
|
582
703
|
})).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
|
|
583
704
|
gamingThreshold: scalars.gamingThreshold ?? null,
|
|
584
705
|
evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
|
|
706
|
+
measuredWith: measuredWith(scalars),
|
|
585
707
|
mandates: Array.isArray(scalars.mandates) ? scalars.mandates.map((m) => ({ name: String(m.name || ''), paths: [...(Array.isArray(m.paths) ? m.paths : [])].sort() })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) : [],
|
|
586
708
|
};
|
|
587
709
|
return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
|
|
@@ -626,7 +748,8 @@ function verdictFingerprint(report) {
|
|
|
626
748
|
return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
|
|
627
749
|
}
|
|
628
750
|
|
|
629
|
-
// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
|
|
751
|
+
// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off), or
|
|
752
|
+
// measures another way (a new cmd, extract, direction, exact or report flag): `cmd: "true"` is a dropped guard too
|
|
630
753
|
function gateWeakenings(baseGate, headMetrics) {
|
|
631
754
|
if (!baseGate) return [];
|
|
632
755
|
const head = new Map(headMetrics.map((m) => [m.name, m]));
|
|
@@ -636,6 +759,7 @@ function gateWeakenings(baseGate, headMetrics) {
|
|
|
636
759
|
const hm = head.get(bm.name);
|
|
637
760
|
if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
|
|
638
761
|
else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
|
|
762
|
+
else if (!bm.report && bm.cmd !== PLACEHOLDER_TEST && measuredBy(hm) !== measuredBy(bm)) weak.push({ name: bm.name, change: 'guard-edited', def: bm });
|
|
639
763
|
}
|
|
640
764
|
return weak;
|
|
641
765
|
}
|
|
@@ -654,29 +778,51 @@ function readAllowTrailer(repo, ref) {
|
|
|
654
778
|
return m && m[1].trim() ? { reason: m[1].trim().slice(0, 300) } : null;
|
|
655
779
|
}
|
|
656
780
|
|
|
781
|
+
// A change is judged by the gate that was in force before it (`baseGate`: the config at base, or the
|
|
782
|
+
// zero-config gate when this change adds the first config). So a change cannot switch off the source-only
|
|
783
|
+
// re-run for a metric that had it, loosen gamingThreshold, or swap the setup, env or linked dirs it is
|
|
784
|
+
// measured with: the base values hold for the change that edits them, and apply from the next one.
|
|
785
|
+
const gamingCheckOff = (cm, baseGate) => {
|
|
786
|
+
if (!cm || cm.gamingCheck !== false) return false;
|
|
787
|
+
const bm = baseGate && baseGate.metrics.get(cm.name);
|
|
788
|
+
return !bm || bm.gamingCheck === false; // off before this change too, or a metric this change adds
|
|
789
|
+
};
|
|
790
|
+
function gamingThresholdFor(cm, cfg, baseGate) {
|
|
791
|
+
const now = cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5;
|
|
792
|
+
const bm = baseGate && baseGate.metrics.get(cm.name);
|
|
793
|
+
const before = bm ? (bm.gamingThreshold ?? baseGate.scalars.gamingThreshold ?? (baseGate.partial ? now : 0.5)) : now;
|
|
794
|
+
return Math.max(now, before); // the stricter of before and after
|
|
795
|
+
}
|
|
796
|
+
|
|
657
797
|
// The three Section 3 checks, run only with detectGaming, in order: (1) re-prove every actual
|
|
658
798
|
// win with the agent's source edits alone; (2) content-hash auto-fail: a declared evaluatorPath
|
|
659
799
|
// gate file edited in this diff is a moved goalpost, so a metric that improved alongside it is
|
|
660
800
|
// gamed by declaration, deterministic even if the source-only re-run read inert; (3) gate
|
|
661
|
-
// integrity:
|
|
662
|
-
// definition (it only counts as tampering if it
|
|
663
|
-
// gate definition at both refs (metrics,
|
|
664
|
-
//
|
|
801
|
+
// integrity: take the gate as of BASE, re-measure any guard HEAD dropped, edited or measures with
|
|
802
|
+
// another setup, env or linked dirs, under its base definition (it only counts as tampering if it
|
|
803
|
+
// WOULD HAVE FAILED at head), and fingerprint the gate definition at both refs (metrics,
|
|
804
|
+
// evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) so a quietly deleted guard
|
|
805
|
+
// or mandate is tamper-evident: base != head is recorded as gate.changed and
|
|
665
806
|
// printed, whatever the verdict. Mutates `metrics` in place (gamed/sourceOnly/retained/
|
|
666
807
|
// gamingReason) and returns the report.gate record. An allow trailer can still clear a gamed
|
|
667
808
|
// metric afterwards, in gate().
|
|
668
|
-
function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
809
|
+
function detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) {
|
|
810
|
+
// how the base measured. Unknown (`partial`) reads as unchanged; so does a first config that adds only the setup `init` detects
|
|
811
|
+
const bs = baseGate ? (baseGate.partial ? { ...baseGate.scalars, ...measureOf(cfg) } : baseGate.scalars) : null;
|
|
812
|
+
const asInit = !!bs && baseGate.zeroConfig && measuredWith({ ...bs, setup: baseGate.initSetup }) === measuredWith(cfg);
|
|
813
|
+
const measureChanged = !!bs && !asInit && measuredWith(bs) !== measuredWith(cfg);
|
|
814
|
+
const baseRun = measureChanged ? { ...cfg, ...measureOf(bs) } : cfg; // the head gate, run with the base setup, env and linked dirs
|
|
669
815
|
const toProve = [];
|
|
670
816
|
for (const m of metrics) {
|
|
671
817
|
if (m.status !== 'improved') continue;
|
|
672
818
|
const cm = cfg.metrics.find((c) => c.name === m.name);
|
|
673
|
-
if (cm
|
|
819
|
+
if (gamingCheckOff(cm, baseGate)) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
|
|
674
820
|
toProve.push(cm);
|
|
675
821
|
}
|
|
676
|
-
const srcResults = measureSourceOnlyAll(repo, base, head, toProve,
|
|
822
|
+
const srcResults = measureSourceOnlyAll(repo, base, head, toProve, baseRun, repeat); // one patched checkout for all of them
|
|
677
823
|
for (const cm of toProve) {
|
|
678
824
|
const m = metrics.find((x) => x.name === cm.name);
|
|
679
|
-
const j = judgeGaming(m, srcResults[cm.name], cm
|
|
825
|
+
const j = judgeGaming(m, srcResults[cm.name], gamingThresholdFor(cm, cfg, baseGate));
|
|
680
826
|
m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
|
|
681
827
|
if (j.unprovable) m.unprovable = true;
|
|
682
828
|
}
|
|
@@ -687,32 +833,55 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
|
687
833
|
if (split.source.length && split.nonSource.length) {
|
|
688
834
|
const held = metrics.filter((m) => m.status === 'unchanged' && m.direction === 'pass' && m.after === 1 && !m.tripwire);
|
|
689
835
|
const defs = held.map((m) => cfg.metrics.find((c) => c.name === m.name)).filter(Boolean);
|
|
690
|
-
const res = measureSourceOnlyAll(repo, base, head, defs,
|
|
836
|
+
const res = measureSourceOnlyAll(repo, base, head, defs, baseRun, repeat);
|
|
691
837
|
for (const m of held) { const r = res[m.name]; if (r && r.samples && r.samples.length && median(r.samples) !== 1) m.heldByTestEdit = true; }
|
|
692
838
|
}
|
|
693
|
-
|
|
839
|
+
// a file declared as a goalpost before this change still is one for this change: dropping the declaration in the
|
|
840
|
+
// same diff that edits the file does not make the edit count as source
|
|
841
|
+
const declared = [...new Set([...(cfg.evaluatorPaths || []), ...((baseGate && baseGate.scalars.evaluatorPaths) || [])])];
|
|
842
|
+
const evalChanged = changedEvaluatorPaths(repo, base, head, { ...cfg, evaluatorPaths: declared });
|
|
694
843
|
if (evalChanged.length) {
|
|
695
844
|
for (const m of metrics) {
|
|
696
845
|
if (m.status !== 'improved' || m.gamed === true) continue;
|
|
697
846
|
const cm = cfg.metrics.find((c) => c.name === m.name);
|
|
698
|
-
if (cm
|
|
847
|
+
if (gamingCheckOff(cm, baseGate)) continue; // tripwire/test-side guards legitimately live in gate files
|
|
699
848
|
m.gamed = true;
|
|
700
849
|
m.gamingReason = `a declared evaluatorPath gate file changed in this diff (${evalChanged.map((e) => e.path).join(', ')}); a win alongside a goalpost edit is gamed by declaration`;
|
|
701
850
|
}
|
|
702
851
|
}
|
|
703
|
-
const baseGate = gateDefAtRef(repo, base);
|
|
704
852
|
const weak = gateWeakenings(baseGate, cfg.metrics);
|
|
853
|
+
// the setup, env or linked dirs changed: every guard the base had is measured again the way it was measured before
|
|
854
|
+
if (measureChanged) for (const bm of baseGate.metrics.values()) {
|
|
855
|
+
if (bm.guard === true && !bm.report && bm.cmd !== PLACEHOLDER_TEST && !weak.some((w) => w.name === bm.name)) weak.push({ name: bm.name, change: 'measurement-edited', def: bm });
|
|
856
|
+
}
|
|
705
857
|
const weakened = [];
|
|
706
858
|
if (weak.length) {
|
|
707
859
|
const defs = weak.map((w) => w.def);
|
|
708
|
-
const b = measureAt(repo, base, { ...
|
|
709
|
-
const a = measureAt(repo, head, { ...
|
|
860
|
+
const b = measureAt(repo, base, { ...baseRun, metrics: defs }, repeat);
|
|
861
|
+
const a = measureAt(repo, head, { ...baseRun, metrics: defs }, repeat);
|
|
862
|
+
const evs = new Map();
|
|
710
863
|
for (const w of weak) {
|
|
711
864
|
const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
|
|
712
|
-
|
|
865
|
+
evs.set(w.name, ev);
|
|
866
|
+
// an edited guard that still fails under its old definition was turned green by the edit, not the code
|
|
867
|
+
let hidden = !ev.ok || (w.change === 'guard-edited' && ev.inert);
|
|
868
|
+
// a guard measured another way counts only when the new measurement hides it: it fails the old way and reads fine the new way
|
|
869
|
+
if (w.change === 'measurement-edited') { const hm = metrics.find((x) => x.name === w.name); hidden = (!ev.ok || !!ev.inert) && !!hm && hm.ok && !hm.inert; }
|
|
870
|
+
if (hidden) weakened.push({ name: w.name, change: w.change, status: ev.inert ? 'failed' : ev.status });
|
|
871
|
+
}
|
|
872
|
+
// a win under the old definition still has to be earned in the source: swapping or renaming the guard in the same
|
|
873
|
+
// change does not skip the source-only re-run
|
|
874
|
+
const won = weak.filter((w) => w.def.gamingCheck !== false && !weakened.some((x) => x.name === w.name) && evs.get(w.name).status === 'improved');
|
|
875
|
+
if (won.length) {
|
|
876
|
+
const src = measureSourceOnlyAll(repo, base, head, won.map((w) => w.def), baseRun, repeat);
|
|
877
|
+
for (const w of won) {
|
|
878
|
+
const j = judgeGaming({ name: w.name, direction: w.def.direction || 'up', guard: true, ...evs.get(w.name) }, src[w.name], w.def.gamingThreshold ?? (bs && bs.gamingThreshold) ?? 0.5);
|
|
879
|
+
if (j.gamed) weakened.push({ name: w.name, change: w.change, status: 'gamed' });
|
|
880
|
+
}
|
|
713
881
|
}
|
|
714
882
|
}
|
|
715
|
-
|
|
883
|
+
// the fingerprint is of a config file: `base` stays null when there was none there (the zero-config gate is not a file)
|
|
884
|
+
const fpBase = baseGate && !baseGate.zeroConfig ? gateFingerprint([...baseGate.metrics.values()], bs) : null;
|
|
716
885
|
const fpHead = gateFingerprint(cfg.metrics, cfg);
|
|
717
886
|
const info = {
|
|
718
887
|
fingerprint: { base: fpBase, head: fpHead },
|
|
@@ -769,17 +938,28 @@ function gate(repo, opts) {
|
|
|
769
938
|
// a guard this change adds has no value at base, because its scanner did not exist there. That is adoption,
|
|
770
939
|
// not a failure: it starts counting from head. Removing a guard is still caught (gateWeakenings), and a guard the
|
|
771
940
|
// base contract already had still fails when it cannot be measured. A zero-config run has no contract to adopt.
|
|
772
|
-
const
|
|
941
|
+
const fileGate = gateDefAtRef(repo, base);
|
|
942
|
+
const baseDefs = opts.config ? null : fileGate;
|
|
773
943
|
const adopted = (name) => !opts.config && !(baseDefs && baseDefs.metrics.has(name));
|
|
944
|
+
// the gate in force before this change: the config committed at base, or, when this change adds the
|
|
945
|
+
// first config, the zero-config gate the repo ran under until now
|
|
946
|
+
const baseGate = opts.detectGaming ? (fileGate || (opts.config ? null : zeroConfigGate(repo, base))) : null;
|
|
774
947
|
const metrics = cfg.metrics.map((m) => {
|
|
775
948
|
const ev = evaluate(m, before[m.name], after[m.name], repeat);
|
|
776
949
|
if (ev.status === 'unmeasurable' && ev.before == null && ev.after != null && adopted(m.name)) { ev.ok = true; ev.adopted = true; }
|
|
777
950
|
// a `report` guard tells you and does not fail: a contract can start by watching and tighten later
|
|
778
951
|
if (m.report && !ev.ok && ev.status === 'regressed') { ev.ok = true; ev.noted = true; }
|
|
779
|
-
|
|
952
|
+
// an inert guard: did its command run and fail, or never run? `ran` says where it ran (absent: at neither ref)
|
|
953
|
+
if (ev.inert && m.cmd !== PLACEHOLDER_TEST) {
|
|
954
|
+
const nb = neverRan(before[m.name]), na = neverRan(after[m.name]);
|
|
955
|
+
if (!nb || !na) ev.ran = !nb && !na ? 'both' : nb ? 'head' : 'base';
|
|
956
|
+
const note = failNotes.get(after[m.name]) || failNotes.get(before[m.name]); // a hint: a missing dependency or tests that fail
|
|
957
|
+
if (note) Object.assign(ev, note);
|
|
958
|
+
}
|
|
959
|
+
return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(gamingCheckOff(m, baseGate) ? { tripwire: true } : {}), ...ev };
|
|
780
960
|
});
|
|
781
961
|
// Section 3 (gaming detection), on by default: mutates `metrics` and yields the gate record
|
|
782
|
-
const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat) : null;
|
|
962
|
+
const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) : null;
|
|
783
963
|
// declared override: a Promptwheel-Allow trailer (or --allow) acknowledges an intentional
|
|
784
964
|
// goalpost/gate change, so a would-be GAMED becomes PASS, recorded in report.allowed for review.
|
|
785
965
|
// It covers an `unprovable` win too (the source-only re-run could not be applied): the declared
|
|
@@ -820,7 +1000,7 @@ function gate(repo, opts) {
|
|
|
820
1000
|
for (const m of ga) { m.ok = true; m.allowed = true; allowed.cleared.push(m.name); }
|
|
821
1001
|
} else for (const m of ga) m.allowable = true; // failed, and a declared reason would clear it: say so in the report
|
|
822
1002
|
}
|
|
823
|
-
// mandates
|
|
1003
|
+
// mandates: a protected-path hard rule fails the gate no matter what the metrics say.
|
|
824
1004
|
// The base ref's mandates count too (resolved through `extends` by gateDefAtRef), so a change
|
|
825
1005
|
// cannot clear a mandate by deleting it in the same diff that violates it.
|
|
826
1006
|
const baseMandates = (gateInfo && gateInfo.baseMandates) || (!opts.detectGaming ? ((gateDefAtRef(repo, base) || {}).scalars || {}).mandates : null);
|
|
@@ -846,6 +1026,9 @@ function gate(repo, opts) {
|
|
|
846
1026
|
...(gateInfo ? { gate: gateInfo } : {}),
|
|
847
1027
|
...(allowed ? { allowed } : {}),
|
|
848
1028
|
};
|
|
1029
|
+
// a setup command that failed, at the ref it failed at: the usual reason a metric could not be measured
|
|
1030
|
+
const sb = setupFails.get(before), sa = setupFails.get(after);
|
|
1031
|
+
if (sb || sa) report.setup = { ...(sb ? { base: sb } : {}), ...(sa ? { head: sa } : {}) };
|
|
849
1032
|
// provenance receipt: makes the verdict tamper-evident + reproducible ("re-run and get the same
|
|
850
1033
|
// verdict is checkable"). Additive to the record; never changes the verdict.
|
|
851
1034
|
report.provenance = {
|
|
@@ -880,7 +1063,7 @@ const PRESETS = {
|
|
|
880
1063
|
'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
|
|
881
1064
|
metrics: [
|
|
882
1065
|
{ name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
|
|
883
|
-
{ name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
1066
|
+
{ name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
884
1067
|
{ name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
885
1068
|
{ name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|# ?nosec|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
886
1069
|
{ name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.promptwheel --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
@@ -906,7 +1089,7 @@ const PRESETS = {
|
|
|
906
1089
|
{ name: 'codeowners-frozen', paths: ['CODEOWNERS'], reason: 'review ownership is set by people' },
|
|
907
1090
|
{ name: 'scanner-frozen', paths: ['.promptwheel/packs/**'], reason: 'the scanners that read the contract are not edited by the change they read' },
|
|
908
1091
|
{ name: 'lock-frozen', paths: ['promptwheel.lock.json'], allowable: 'flag', reason: 'the architecture lock changes with a reason from a person (--allow), never as a side effect of a change' },
|
|
909
|
-
{ name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], reason: 'the hooks that enforce the contract
|
|
1092
|
+
{ name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], allowable: 'flag', reason: 'the hooks that enforce the contract change with a reason from a person (--allow), and are never detached by the agent' },
|
|
910
1093
|
],
|
|
911
1094
|
metrics: [
|
|
912
1095
|
{ name: 'lifecycle_scripts', cmd: 'node packs/supply-chain/scan.mjs lifecycle .', extract: 'number', direction: 'down', guard: true, gamingCheck: false, allowable: 'flag' },
|
|
@@ -919,17 +1102,17 @@ const PRESETS = {
|
|
|
919
1102
|
] },
|
|
920
1103
|
};
|
|
921
1104
|
|
|
922
|
-
function detectTestCmd(repo) {
|
|
923
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1105
|
+
function detectTestCmd(repo, { has, read } = stackAt(repo)) {
|
|
924
1106
|
if (has('go.mod')) return 'go test ./...';
|
|
925
1107
|
if (has('Cargo.toml')) return 'cargo test';
|
|
926
|
-
|
|
1108
|
+
// the venv is not tracked: it is looked for on disk whichever ref the stack is read from
|
|
1109
|
+
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return existsSync(join(repo, '.venv/bin/pytest')) ? '.venv/bin/pytest -q' : 'pytest -q';
|
|
927
1110
|
// only trust `npm test` when a test script actually exists, otherwise the metric can
|
|
928
1111
|
// never pass and the gate would "protect" nothing (very common in Next.js app repos)
|
|
929
1112
|
try {
|
|
930
|
-
if (JSON.parse(
|
|
1113
|
+
if (JSON.parse(read('package.json')).scripts?.test) return 'npm test --silent';
|
|
931
1114
|
} catch { /* no or unparsable package.json, fall through */ }
|
|
932
|
-
return
|
|
1115
|
+
return PLACEHOLDER_TEST;
|
|
933
1116
|
}
|
|
934
1117
|
|
|
935
1118
|
// deps that live outside git + the env a test needs, defaulted per stack. init writes these into
|
|
@@ -938,18 +1121,16 @@ function detectTestCmd(repo) {
|
|
|
938
1121
|
// where the detected test command cannot run without a build first (a 10-repo replay put this at
|
|
939
1122
|
// ~2/3 of gates). Written by `init` into `setup`; a failed setup stays honest (inert -> inconclusive,
|
|
940
1123
|
// never a fake green). Kept out of the zero-config default so the fast per-turn path stays fast.
|
|
941
|
-
function detectSetup(repo) {
|
|
942
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1124
|
+
function detectSetup(repo, { has, read } = stackAt(repo)) {
|
|
943
1125
|
try {
|
|
944
|
-
const pkg = JSON.parse(
|
|
1126
|
+
const pkg = JSON.parse(read('package.json'));
|
|
945
1127
|
if (pkg.scripts && pkg.scripts.build) return 'npm run build --if-present'; // TS/bundler tests that run against a built dir
|
|
946
1128
|
} catch { /* no or unparsable package.json */ }
|
|
947
1129
|
if (has('pyproject.toml') || has('setup.py')) return 'pip install -e . --quiet 2>/dev/null || true'; // editable install so imports resolve the measured ref
|
|
948
1130
|
return undefined;
|
|
949
1131
|
}
|
|
950
1132
|
|
|
951
|
-
function detectEnv(repo) {
|
|
952
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1133
|
+
function detectEnv(repo, { has } = stackAt(repo)) {
|
|
953
1134
|
// Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
|
|
954
1135
|
// resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
|
|
955
1136
|
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
|
|
@@ -1064,6 +1245,7 @@ function writeAgentBlock(repo) {
|
|
|
1064
1245
|
|
|
1065
1246
|
const short = (repo, ref) => { try { return git(['rev-parse', '--short', ref], repo); } catch { return ref; } };
|
|
1066
1247
|
|
|
1248
|
+
const wouldHave = (status) => (status === 'unmeasurable' ? 'been unmeasurable' : status);
|
|
1067
1249
|
function printHuman(r) {
|
|
1068
1250
|
const arrowFor = (m) => (m.delta == null ? '?' : m.delta > 0 ? '▲' : m.delta < 0 ? '▼' : '=');
|
|
1069
1251
|
console.log(`\nPromptWheel ${r.base} → ${r.head}${r.repeat > 1 ? ` (×${r.repeat})` : ''}\n`);
|
|
@@ -1072,15 +1254,23 @@ function printHuman(r) {
|
|
|
1072
1254
|
const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
|
|
1073
1255
|
const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
|
|
1074
1256
|
console.log(` ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)} (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
|
|
1075
|
-
if (m.inert) console.log(` ⚠ "${m.name}"
|
|
1257
|
+
if (m.inert && m.ran && m.cause === 'missing-dependency') console.log(` ⚠ "${m.name}" could not run in the clean copy at either ref: ${m.detail}. Link or install what it needs (linkDirs or setup in promptwheel.config.json), then check again`);
|
|
1258
|
+
else if (m.inert && m.ran === 'both') console.log(` ⚠ "${m.name}" ran and failed before and after this change, so it guards nothing here. Fix the failing tests, or the command, before trusting a pass`);
|
|
1259
|
+
else if (m.inert) console.log(` ⚠ "${m.name}" ${m.ran ? `ran and failed at the ${m.ran} ref and did not run at the other` : 'never ran (same value at both refs)'}: its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
|
|
1076
1260
|
if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
|
|
1077
1261
|
}
|
|
1078
|
-
if (r.
|
|
1079
|
-
if (r.gate && r.gate.
|
|
1262
|
+
if (r.setup) { const at = Object.keys(r.setup), f = r.setup[at[0]]; console.log(` ⚠ setup failed (exit ${f.exit}) at the ${at.join(' and ')} ref${at.length > 1 ? 's' : ''}${f.lastLine ? ': ' + f.lastLine : ''}`); }
|
|
1263
|
+
if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(w.change === 'measurement-edited'
|
|
1264
|
+
? ` 🚩 GATE WEAKENED, how the guard "${w.name}" is measured (setup, env or linked dirs) was edited in this change, and measured the way it was before it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`
|
|
1265
|
+
: ` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : 'disabled'} in this change and it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`);
|
|
1266
|
+
if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited; review promptwheel.config.json as part of the diff`);
|
|
1080
1267
|
const failReason = r.mandates ? ', a mandate was violated (a protected path changed)' : ', a guarded metric regressed (beyond noise)';
|
|
1081
1268
|
console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? failReason : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
|
|
1082
|
-
if (r.verdict === 'fail' && r.mandates)
|
|
1083
|
-
|
|
1269
|
+
if (r.verdict === 'fail' && r.mandates) {
|
|
1270
|
+
const takes = r.mandates.filter((md) => md.allowable).map((md) => `"${md.name}"`);
|
|
1271
|
+
if (takes.length) console.log(` intentional? ${takes.join(', ')} ${takes.length > 1 ? 'accept' : 'accepts'} a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record`);
|
|
1272
|
+
if (takes.length < r.mandates.length) console.log(` ${takes.length ? 'any other' : 'a'} mandate is a hard rule; if the change is truly intended, retire the mandate in promptwheel.config.json in a change of its own first (a tracked, tamper-evident change), do not work around it`);
|
|
1273
|
+
} else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
|
|
1084
1274
|
else if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name)');
|
|
1085
1275
|
if (r.allowed) console.log(` ✓ allowed: ${r.allowed.cleared.join(', ')} acknowledged as an intentional change, recorded (Promptwheel-Allow: ${r.allowed.reason})`);
|
|
1086
1276
|
console.log('');
|
|
@@ -1134,9 +1324,10 @@ function renderMarkdown(r) {
|
|
|
1134
1324
|
'|---|--:|--:|--:|---|---|',
|
|
1135
1325
|
rows,
|
|
1136
1326
|
'',
|
|
1137
|
-
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule
|
|
1138
|
-
...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by
|
|
1139
|
-
...(r.
|
|
1327
|
+
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule; no metric result clears it.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? (r.metrics.some((m) => m.inert) && r.metrics.every((m) => !m.inert || m.ran === 'both') ? '> 🟡 A 🛡️ guarded check is **inert**, it fails before and after this change, so nothing was actually verified. Fix it before trusting a pass.' : '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.') : '> ✅ No guarded metric regressed.',
|
|
1328
|
+
...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : w.change === 'measurement-edited' ? 'measured another way (setup, env or linked dirs)' : 'disabled'} (would have ${wouldHave(w.status)} at head)`).join('; ')}. The pass was bought by changing the gate, not by earning it.`] : []),
|
|
1329
|
+
...(r.allowed ? ['', `> ✓ **Allowed:** \`${r.allowed.cleared.join('`, `')}\` cleared by a declared reason: ${r.allowed.reason}`] : []),
|
|
1330
|
+
...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited. Review the config change with the code.`] : []),
|
|
1140
1331
|
'',
|
|
1141
1332
|
'<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
|
|
1142
1333
|
].join('\n');
|
|
@@ -1169,7 +1360,7 @@ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluat
|
|
|
1169
1360
|
// Importing this file never runs the CLI (see the isMain guard at the bottom).
|
|
1170
1361
|
// Stable across 0.x minor versions. The schema check() returns and the exit codes
|
|
1171
1362
|
// verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
|
|
1172
|
-
// shell, import
|
|
1363
|
+
// shell, import) are in INTEGRATE.md.
|
|
1173
1364
|
// ─────────────────────────────────────────────────────────────────────────────
|
|
1174
1365
|
|
|
1175
1366
|
// The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
|
|
@@ -1207,8 +1398,8 @@ export function verdictExitCode(verdict) {
|
|
|
1207
1398
|
// detectGaming boolean run the Section 3 source-only re-proof (default: true)
|
|
1208
1399
|
// allow string a declared override reason (same as --allow / a Promptwheel-Allow: trailer)
|
|
1209
1400
|
export function check(options = {}) {
|
|
1210
|
-
// `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
|
|
1211
|
-
//
|
|
1401
|
+
// `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
|
|
1402
|
+
// lives), exactly as the CLI does from the same directory
|
|
1212
1403
|
let start = process.cwd();
|
|
1213
1404
|
if (options.repo) { try { start = realpathSync(options.repo); } catch { bad(`promptwheel: not inside a git repository (${options.repo})`); } }
|
|
1214
1405
|
const repo = repoRoot(start);
|
|
@@ -1328,17 +1519,35 @@ const GATE_FLAGS = ['--base', '--head', '--repeat', '--working', '--no-record',
|
|
|
1328
1519
|
const COMMAND_FLAGS = {
|
|
1329
1520
|
run: GATE_FLAGS, init: ['--preset', '--list', '--force', '--no-agent-block'], lock: [],
|
|
1330
1521
|
};
|
|
1331
|
-
|
|
1522
|
+
// what each valued flag needs. A missing or unusable value is a usage error too: `--repeat abc` must not
|
|
1523
|
+
// measure nothing and read FAIL, and `--allow` with nothing after it must not run as if no reason was asked for.
|
|
1524
|
+
const VALUED_FLAGS = { '--base': 'a ref', '--head': 'a ref', '--repeat': 'a whole number, 1 or more', '--allow': 'a reason, e.g. --allow "the spec changed"', '--preset': 'a preset name (see promptwheel init --list)' };
|
|
1332
1525
|
function checkFlags(cmd, argv) {
|
|
1333
1526
|
const known = new Set(COMMAND_FLAGS[cmd]);
|
|
1334
1527
|
for (let i = 0; i < argv.length; i++) {
|
|
1335
1528
|
const a = argv[i];
|
|
1336
1529
|
if (!a.startsWith('-')) continue;
|
|
1337
1530
|
if (!known.has(a)) bad(`promptwheel ${cmd}: unknown flag ${a} (see promptwheel --help)`);
|
|
1338
|
-
if (
|
|
1531
|
+
if (!(a in VALUED_FLAGS)) continue;
|
|
1532
|
+
const v = argv[++i];
|
|
1533
|
+
const gone = v === undefined || known.has(v); // nothing after the flag, or the next flag
|
|
1534
|
+
// an empty --base or --head is the default ref: the Action passes an empty base outside a pull request
|
|
1535
|
+
const blank = !gone && !v.trim() && !(v === '' && (a === '--base' || a === '--head'));
|
|
1536
|
+
if (gone || blank) bad(`promptwheel ${cmd}: ${a} needs ${VALUED_FLAGS[a]}`);
|
|
1537
|
+
if (a === '--repeat' && !/^[1-9]\d*$/.test(v)) bad(`promptwheel ${cmd}: --repeat needs ${VALUED_FLAGS[a]} (got "${v}")`);
|
|
1339
1538
|
}
|
|
1340
1539
|
}
|
|
1341
1540
|
|
|
1541
|
+
// read PR comments on stdin: a JSON array, or one comment per line as `gh api --paginate --jq '.[]'` prints every
|
|
1542
|
+
// page. Streamed: readFileSync(0) corrupted the heap on Node 24.22 reading a live paginated gh pipe.
|
|
1543
|
+
async function prAllow() {
|
|
1544
|
+
let t = ''; process.stdin.setEncoding('utf8');
|
|
1545
|
+
for await (const chunk of process.stdin) t += chunk;
|
|
1546
|
+
t = t.trim(); let c = [];
|
|
1547
|
+
try { c = [].concat(JSON.parse(t || '[]')); } catch { try { c = t.split('\n').filter(Boolean).map((l) => JSON.parse(l)); } catch { /* unreadable, no approval */ } }
|
|
1548
|
+
process.stdout.write(allowFromComments(c));
|
|
1549
|
+
}
|
|
1550
|
+
|
|
1342
1551
|
// write or refresh the architecture lock from the tree as it stands. A person runs this: the lock file is
|
|
1343
1552
|
// a mandate, so the refreshed file passes the gate only with --allow "<reason>".
|
|
1344
1553
|
async function lock() {
|
|
@@ -1353,9 +1562,9 @@ function dispatch() {
|
|
|
1353
1562
|
const cmd = (rest.includes('--help') || rest.includes('-h')) ? 'help' : first; // `promptwheel run --help` is a help request, not a gate run
|
|
1354
1563
|
if (cmd in COMMAND_FLAGS) checkFlags(cmd, rest);
|
|
1355
1564
|
if (cmd === 'run') run(rest);
|
|
1356
|
-
else if (cmd === 'pr-allow')
|
|
1565
|
+
else if (cmd === 'pr-allow') return prAllow();
|
|
1357
1566
|
else if (cmd === 'init') init(rest);
|
|
1358
|
-
else if (cmd === 'lock') lock(rest);
|
|
1567
|
+
else if (cmd === 'lock') return lock(rest);
|
|
1359
1568
|
else {
|
|
1360
1569
|
const help = !cmd || cmd === '--help' || cmd === '-h' || cmd === 'help';
|
|
1361
1570
|
if (!help) console.error(`unknown command: ${cmd}`);
|
|
@@ -1370,10 +1579,11 @@ function dispatch() {
|
|
|
1370
1579
|
' catches reward-hacking BY DEFAULT, re-proves each win from SOURCE edits alone (verdict GAMED,',
|
|
1371
1580
|
' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
|
|
1372
1581
|
' promptwheel run --working gate uncommitted changes (incl. newly added files)',
|
|
1582
|
+
' promptwheel pr-allow < comments.json print the newest `sawbuck-allow: <reason>` from a PR comment by someone with write access (the Action uses it)',
|
|
1373
1583
|
' promptwheel run --allow "<reason>" acknowledge an intentional test/config change (or a `Promptwheel-Allow: <reason>` commit trailer; `Sawbuck-Allow:` is an alias): recorded, not flagged GAMED',
|
|
1374
1584
|
'',
|
|
1375
1585
|
'Exit codes: 0 pass · 1 fail (a guarded metric regressed, or a mandate was violated) · 2 gamed · 3 inconclusive · 64 usage or config error (never a verdict).',
|
|
1376
|
-
'Integrate (library / conform / shell
|
|
1586
|
+
'Integrate (library / conform / shell): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
|
|
1377
1587
|
'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
|
|
1378
1588
|
' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
|
|
1379
1589
|
].join('\n'));
|
|
@@ -1384,8 +1594,8 @@ function dispatch() {
|
|
|
1384
1594
|
// run the CLI only when invoked directly (resolves symlinks so the npm bin still works)
|
|
1385
1595
|
let isMain = false;
|
|
1386
1596
|
try { isMain = !!process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); } catch { /* not main */ }
|
|
1387
|
-
function main() {
|
|
1388
|
-
try { dispatch(); }
|
|
1597
|
+
async function main() {
|
|
1598
|
+
try { await dispatch(); }
|
|
1389
1599
|
catch (e) { if (e && e.name === 'ConfigError') die(e.message); throw e; }
|
|
1390
1600
|
}
|
|
1391
1601
|
if (isMain) main();
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.2",
|
|
4
4
|
"description": "A deterministic, zero-LLM check for your coding loop. A win must survive a re-run from the agent's source edits alone, so a test that went green by editing the test reads GAMED. Runs in CI or a Claude Code plugin.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -27,7 +27,7 @@ clean git worktrees, so a pack that is not in git is absent at both refs and the
|
|
|
27
27
|
count. A private registry mirror makes this baseline large, not failing.
|
|
28
28
|
- **unpinned_execution**: code run from a reference that can move. A GitHub Action not pinned to a full commit SHA (a local `./action` does not count), `npx`, `bunx` or `dlx` with no exact version, `npm install -g` with no exact version, and a remote script piped into a shell. Read from workflows, `action.yml`, package.json scripts, `scripts/*.sh` and Dockerfiles. In package.json scripts, a tool the manifest declares does not count: the lockfile already pins it.
|
|
29
29
|
- **install_policy_gaps**: 0, 1 or 2. Are install scripts disabled, and is a release-age cooldown set (`ignore-scripts=true` and `min-release-age` in `.npmrc`, or the pnpm or yarn equivalents)? The cooldown is enforced by the package manager, not by this pack: a registry lookup here would make the verdict depend on the clock.
|
|
30
|
-
- **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is
|
|
30
|
+
- **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is locked but not counted by default (`node .promptwheel/packs/supply-chain/lock.mjs drift . --list` shows it). To hold new routes too, add `--routes` to the end of the `lock_drift` metric's `cmd` in `promptwheel.config.json`, so it reads `node .promptwheel/packs/supply-chain/lock.mjs drift . --routes`. The flag belongs to that script: `promptwheel lock` takes no flags and always writes routes into the lock. The config is a frozen path here, so that edit passes with `promptwheel run --allow "<reason>"`. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
|
|
31
31
|
- **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%, dependency changes; use the CI path below for those.
|
|
32
32
|
|
|
33
33
|
**On a pull request.** Someone with write access comments `sawbuck-allow: <reason>` and re-runs the check. `sawbuck contract` marks everyday guards `report: true`, so they note a change without failing until you tighten them; this preset on its own holds them all.
|
|
@@ -57,10 +57,10 @@ Sometimes adding a dependency is exactly what was asked. Say so, and the gate re
|
|
|
57
57
|
promptwheel run --working --allow "needed for the CSV export (papaparse)"
|
|
58
58
|
```
|
|
59
59
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
written into `report.allowed
|
|
63
|
-
|
|
60
|
+
A commit trailer does not count here. The seven metrics carry `allowable: "flag"`, which means a
|
|
61
|
+
reason from the `--allow` flag clears their guarded regression for that one run, with the reason
|
|
62
|
+
written into `report.allowed`. Nothing is loosened in the config, and a rise without a reason still
|
|
63
|
+
fails. That is the point: every added package, script, or non-registry
|
|
64
64
|
source leaves an audit line a reviewer can read.
|
|
65
65
|
|
|
66
66
|
`gamingCheck` is `false` on all seven because these numbers legitimately live in config files
|
|
@@ -69,8 +69,8 @@ flag every honest dependency bump.
|
|
|
69
69
|
|
|
70
70
|
## The mandates the preset writes
|
|
71
71
|
|
|
72
|
-
A mandate is a path the agent must not touch in a change
|
|
73
|
-
and a mandate deleted or narrowed in the same change that violates it stays in force.
|
|
72
|
+
A mandate is a path the agent must not touch in a change. Touching one fails the gate whatever the
|
|
73
|
+
metrics say, and a mandate deleted or narrowed in the same change that violates it stays in force.
|
|
74
74
|
|
|
75
75
|
| mandate | paths | why |
|
|
76
76
|
| --- | --- | --- |
|
|
@@ -79,15 +79,20 @@ and a mandate deleted or narrowed in the same change that violates it stays in f
|
|
|
79
79
|
| build-images-frozen | `Dockerfile*`, `docker-compose*` | what runs at build and deploy time |
|
|
80
80
|
| publish-scripts-frozen | `scripts/release*`, `scripts/publish*` | the publish path is how a compromise spreads |
|
|
81
81
|
| codeowners-frozen | `CODEOWNERS` | review ownership is set by people |
|
|
82
|
+
| scanner-frozen | `.promptwheel/packs/**` | the scanners are not edited by the change they read |
|
|
82
83
|
| contract-frozen | `promptwheel.config.json`, `sawbuck.config.json` | the contract changes with a reason from a person (`--allow`) |
|
|
83
|
-
|
|
|
84
|
+
| lock-frozen | `promptwheel.lock.json` | the lock changes with a reason from a person (`--allow`) |
|
|
85
|
+
| hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks change with a reason from a person (`--allow`) |
|
|
84
86
|
|
|
85
87
|
The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is frozen too, and it is also an
|
|
86
88
|
`evaluatorPaths` entry. A change to it needs `--allow "<reason>"` from a person. Without that, a change that
|
|
87
89
|
only weakens a rule would pass, and the next change would be judged by the weakened contract.
|
|
88
90
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
+
`--allow` clears only three of these: `contract-frozen`, `lock-frozen` and `hooks-frozen` (so a team can commit
|
|
92
|
+
shared hooks to a repo that already has its rules). A commit trailer clears none. Every other frozen path stays
|
|
93
|
+
frozen, even with `--allow`. To change one, a person first retires its mandate in a change of its own
|
|
94
|
+
(it edits the contract, so it needs `--allow`), then makes the change, then puts the mandate back the
|
|
95
|
+
same way.
|
|
91
96
|
|
|
92
97
|
## What this is not
|
|
93
98
|
|
|
@@ -100,5 +105,4 @@ path, reviewed as such.
|
|
|
100
105
|
- **Not a scanner.** No matchers, no findings, no false positives to triage: a path, a count, or a
|
|
101
106
|
hash.
|
|
102
107
|
|
|
103
|
-
Pair it with the
|
|
104
|
-
the gate runs where the agent cannot switch it off.
|
|
108
|
+
Pair it with the GitHub Action so the gate runs where the agent cannot switch it off.
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"$comment": "PromptWheel supply-chain guard-pack.
|
|
2
|
+
"$comment": "PromptWheel supply-chain guard-pack. Install it with `promptwheel init --preset supply-chain`: that copies the pack to .promptwheel/packs/supply-chain/, writes these metrics into promptwheel.config.json with their commands pointed at the copy, and writes the mandates. The commands in this file run packs/supply-chain/..., which exists only in the PromptWheel repository itself, so `extends` on this file (here or on the copy under .promptwheel/packs/) reads every metric as unmeasurable, and the gate fails, in any other repository. Seven counts the gate holds: a rise fails the gate unless a person gives a reason with `--allow \"<reason>\"` (a commit trailer does not count), which is recorded in report.allowed. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
|
|
3
3
|
"metrics": [
|
|
4
4
|
{
|
|
5
5
|
"$comment": "install-time script entries (preinstall, install, postinstall, prepare, prepublish, prepublishOnly, postpack) across every package.json: the step a supply-chain attack needs inside your repo",
|