promptwheel 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INTEGRATE.md +15 -8
- package/README.md +17 -13
- package/bin/promptwheel.d.ts +15 -9
- package/bin/promptwheel.mjs +221 -88
- package/package.json +1 -1
- package/packs/supply-chain/README.md +16 -12
- package/packs/supply-chain/pack.json +1 -1
package/INTEGRATE.md
CHANGED
|
@@ -50,7 +50,7 @@ importing this package.
|
|
|
50
50
|
"gamed": true,
|
|
51
51
|
"sourceOnly": null,
|
|
52
52
|
"retained": 0.0,
|
|
53
|
-
"gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
|
|
53
|
+
"gamingReason": "the \"win\" changed zero production-source files, only test/config/grader/golden"
|
|
54
54
|
}
|
|
55
55
|
]
|
|
56
56
|
}
|
|
@@ -68,11 +68,18 @@ metric's gain that survived the source-only re-run (Section 3). `confidence`
|
|
|
68
68
|
(`none | unverified | low | medium | high`, `none` for an unmeasurable metric) and `sourceOnly` are informational fields this
|
|
69
69
|
implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
|
|
70
70
|
truncated SHA-256 fingerprint of the gate definition (metrics, guards, `evaluatorPaths`,
|
|
71
|
-
`gamingThreshold`, `mandates`
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
71
|
+
`gamingThreshold`, `mandates`, `setup`, `env`, linked dirs; each metric with its `cmd` and how it
|
|
72
|
+
is read) at base and head, sets `gate.changed: true` when they differ, and lists in
|
|
73
|
+
`gate.weakened` any guard the change removed, disabled, or measured another way
|
|
74
|
+
(`change: guard-edited`, such as `cmd: "true"`; `change: measurement-edited`, a new `setup`, `env`
|
|
75
|
+
or `linkDirs`) to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
|
|
76
|
+
cannot pass by disabling the guard it is judged by. A change that adds the first config is
|
|
77
|
+
compared with the zero-config gate (the detected test command). `allowed` (optional) is `{ reason, cleared }` when a declared
|
|
78
|
+
reason turned a would-be `gamed`, `fail` or `inconclusive` into `pass`: `cleared` names each metric,
|
|
79
|
+
`gate`, or `mandate:<name>` it cleared, and each cleared metric carries `allowed: true` and keeps its
|
|
80
|
+
`gamingReason`, so a consumer can say what was allowed and why it would have failed.
|
|
81
|
+
|
|
82
|
+
Self-check your implementation against this schema with `conformsToContract` (see part 4 below).
|
|
76
83
|
|
|
77
84
|
---
|
|
78
85
|
|
|
@@ -99,8 +106,8 @@ measurement runs in a throwaway git worktree.
|
|
|
99
106
|
## 3. Import the library (in process)
|
|
100
107
|
|
|
101
108
|
Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
|
|
102
|
-
process and returns the Section 2 object. It never prints and never calls `process.exit`;
|
|
103
|
-
throws
|
|
109
|
+
process and returns the Section 2 object. It never prints and never calls `process.exit`; on a
|
|
110
|
+
precondition failure it throws an `Error` named `ConfigError` with `exitCode` 64. Types ship in the package
|
|
104
111
|
(`bin/promptwheel.d.ts`, hand-written, no build step): `import type { Report, CheckOptions, Config } from 'promptwheel'`
|
|
105
112
|
works in TypeScript, and editors complete the options and the report in plain JavaScript.
|
|
106
113
|
|
package/README.md
CHANGED
|
@@ -23,8 +23,10 @@ printf "import {test} from 'node:test'; import assert from 'node:assert'; import
|
|
|
23
23
|
git commit -qam 'make the suite green'
|
|
24
24
|
|
|
25
25
|
npx -y promptwheel@latest run --base HEAD~1 --head HEAD
|
|
26
|
-
#
|
|
27
|
-
#
|
|
26
|
+
# prints (shortened to the lines that matter; exit code 2):
|
|
27
|
+
# ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
|
|
28
|
+
# 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
|
|
29
|
+
# VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
|
|
28
30
|
```
|
|
29
31
|
|
|
30
32
|
What happened: PromptWheel measured `tests_pass` at both commits in throwaway git worktrees (0 then 1, a win), then split the diff into production source versus `{test, config, grader, golden}` files, re-applied **only the source slice** on a clean worktree at the base, and re-ran the metric. The win did not survive, so it is `GAMED`. No LLM, no network: a diff partition plus a re-run, reproducible in seconds with a human-readable reason.
|
|
@@ -88,11 +90,11 @@ Gaming detection is on by default in `run`; `--no-detect-gaming` gives you the b
|
|
|
88
90
|
- **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
|
|
89
91
|
- **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
|
|
90
92
|
- **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
|
|
91
|
-
- **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
|
|
92
|
-
- **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
|
|
93
|
+
- **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires. Set on a metric that had the re-run on, it takes effect from the next change: the change that sets it is still re-proven.
|
|
94
|
+
- **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`. A change that lowers it is still judged by the value before it.
|
|
93
95
|
- **evaluatorPaths**, globs for files that *define* your gate (a threshold file, a budget, a CI limit). Editing one in the same diff that improves a metric is `GAMED` by declaration, deterministically. Opt-in, because only you know which files are the goalposts.
|
|
94
96
|
|
|
95
|
-
- **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. For counts where a rise is sometimes the job (the `supply-chain` preset
|
|
97
|
+
- **allowable**, `true` lets a declared reason (`--allow "<reason>"`, a `Promptwheel-Allow:` commit trailer, `check({ allow })`) clear *this* metric's guarded regression for that run, recorded in `report.allowed` and on the metric as `allowed: true`. `"flag"` takes the reason from `--allow` (or `check({ allow })`) only, never a trailer, because an agent can write a trailer. For counts where a rise is sometimes the job (the `supply-chain` preset uses `"flag"`: a new dependency), so the change passes with the reason on the record instead of the guard being loosened. Without a reason the guard fails as usual.
|
|
96
98
|
|
|
97
99
|
Dependency and environment keys (config-level; `init` writes stack defaults):
|
|
98
100
|
|
|
@@ -100,6 +102,8 @@ Dependency and environment keys (config-level; `init` writes stack defaults):
|
|
|
100
102
|
- **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
|
|
101
103
|
- **setup**, an optional command run once per ref before metrics (for example `npm run build` or `pip install -e .`), after any source patch, so a build-gated suite measures the ref it is actually on.
|
|
102
104
|
|
|
105
|
+
These three shape every measurement, so they are part of the gate. A change that edits them is flagged as a gate change and is judged with the values from before it (see Gate integrity below).
|
|
106
|
+
|
|
103
107
|
Copy a config for your stack (pytest, tsc, coverage, bundle size, LLM eval) from [examples/](examples/).
|
|
104
108
|
|
|
105
109
|
### Guardrails and inheritance
|
|
@@ -116,7 +120,7 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
|
|
|
116
120
|
|
|
117
121
|
### Mandates (protected paths)
|
|
118
122
|
|
|
119
|
-
Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate
|
|
123
|
+
Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate**, however green everything else is. The metrics still run and are reported, but nothing they say clears a mandate:
|
|
120
124
|
|
|
121
125
|
```json
|
|
122
126
|
// promptwheel.config.json
|
|
@@ -126,7 +130,7 @@ Some rules are not a metric at all, they are a hard line: *never touch `src/auth
|
|
|
126
130
|
"metrics": [ /* ... */ ] }
|
|
127
131
|
```
|
|
128
132
|
|
|
129
|
-
A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the
|
|
133
|
+
A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the ones the base ref already had, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`), and one that narrows a mandate is checked against both versions (`edited: true`). A mandate added in the same change that touches its path does not apply to that change, on purpose: the commit that signs a contract adds the very files the contract protects. Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
|
|
130
134
|
|
|
131
135
|
### Supply chain (preset `supply-chain`)
|
|
132
136
|
|
|
@@ -136,7 +140,7 @@ The step every npm supply-chain attack needs inside a target repo is a supply-ch
|
|
|
136
140
|
npx promptwheel init --preset supply-chain
|
|
137
141
|
```
|
|
138
142
|
|
|
139
|
-
writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`,
|
|
143
|
+
writes `promptwheel.lock.json` (the shape of the repo, from `promptwheel lock`) and seven guarded counts that must not rise (`lifecycle_scripts`, `dependency_entries`, `lock_nonregistry_sources`, `unpinned_execution`, `install_policy_gaps`, `lock_drift`, `pins_changed`, measured by a zero-dep pack copied into `.promptwheel/packs/supply-chain/`), mandates that freeze `.github/workflows/**`, `.npmrc` and the yarn rc files, `Dockerfile*` and `docker-compose*`, `scripts/release*` and `scripts/publish*`, `CODEOWNERS`, the copied scanners, the Claude Code hook files, the lock file, and the contract itself (`promptwheel.config.json`, `sawbuck.config.json`). A change that edits the contract fails (exit 1) unless a person passes `--allow "<reason>"`; the contract is also an `evaluatorPaths` entry, so editing it alongside a win is gamed by declaration. `--allow` clears only the contract, the lock and the hooks. Every other frozen path needs its mandate retired first, by a person, in a change of its own run with `--allow`. When the rise is the job, say so and it is recorded: `promptwheel run --allow "needed for the CSV export (papaparse)"`. It is not a dependency firewall and not an injection detector: it does nothing about a package already compromised on the registry, and it does not decide where an instruction came from. Details in [packs/supply-chain/README.md](packs/supply-chain/README.md).
|
|
140
144
|
|
|
141
145
|
## In CI and in Claude Code
|
|
142
146
|
|
|
@@ -170,16 +174,16 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
|
|
|
170
174
|
|
|
171
175
|
## Trust model
|
|
172
176
|
|
|
173
|
-
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
|
|
177
|
+
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). A grader is recognized by its file name in a scripting language (`eval.py`, `grader.sh`, `score.js`, `scoring.ts`) or by an `evals/` or `grader(s)/` directory; a production file with one of those names (`src/score.py`) is treated as a grader too, so a fix there needs `--allow`, and a grader with any other name (`evaluate.py`, `check.py`) is source unless you list it in `evaluatorPaths`. On top of that:
|
|
174
178
|
|
|
175
179
|
- **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
|
|
176
|
-
- **Gate integrity.** The gate is resolved as of the base commit too
|
|
180
|
+
- **Gate integrity.** A change is judged by the gate that was in force before it. The gate is resolved as of the base commit too, through `extends`. A change that disables a guard, deletes a guarded metric, or measures it another way (a new `cmd` such as `true`, `extract`, `direction`, `exact` or `report`) in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed, or would still fail. If the old guard only passes because the same change also edited the test, it reads `GAMED` too: a win under the old definition still has to survive the source-only re-run. A file declared in `evaluatorPaths` before the change still counts as a goalpost for that change, even if the change drops the declaration. The same goes for `setup`, `env` and `linkDirs`: when a change edits them, each guard is measured again with the old values and so is the source-only re-run, so a `setup` that rewrites a test cannot carry a pass (`gate.weakened[].change: measurement-edited`). `gamingCheck: false` on a metric that had the re-run on, and a lower `gamingThreshold`, apply from the next change. A change that adds the first config is compared with the zero-config gate (the detected test command), so `tests_pass` set to `true` in a new config is an edited guard; a first config that keeps the detected command, or replaces the placeholder, is not. The gate definition (each metric with its command and how it is read, guards, `evaluatorPaths`, `gamingThreshold`, mandate names and paths, `setup`, `env`, linked dirs) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed. Two limits: a test that could not run at all before a `setup` fix reads `GAMED` on the change that adds the fix (say so with `--allow`; the `setup` that `init` writes into a first config is exempt), and a base config that `extends` a file outside the repository is not compared on `setup`, `env` or `linkDirs`.
|
|
177
181
|
- **A sanctioned override.** A commit trailer can be written by whoever writes the commit, including an agent, so `"allowTrailer": false` in the contract turns trailers off and only the `--allow` flag counts (the supply-chain preset sets this; the contract at the base ref decides, so a change cannot turn trailers back on for itself). When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. It covers a win whose source-only re-run could not be applied (`unprovable`, otherwise `inconclusive`) the same way. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
|
|
178
182
|
- **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
|
|
179
183
|
|
|
180
|
-
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test)
|
|
184
|
+
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test), or in a build script the unchanged `setup` runs (a change that makes that script rewrite a test is a source edit to this check). A base config that `extends` a file outside the repository cannot be compared across the change. These need held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
|
|
181
185
|
|
|
182
|
-
**Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A
|
|
186
|
+
**Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guarded pass/fail check that reads 0 at both refs (its command never ran, or it ran and failed both times; the report says which) makes the verdict `inconclusive`, never a fake pass.
|
|
183
187
|
|
|
184
188
|
**When not to use it.** If you only care that tests pass and nobody could be editing the tests, your plain CI already has you covered. PromptWheel earns its place when the thing writing the code can also touch the thing judging it, or when you have a graded metric beyond pass/fail (an eval score, cost per run, latency, size) that a change could quietly move. Single-metric CI gates (benchmark trackers, size limits) go deeper on one number; wrap them as a metric `cmd` and PromptWheel composes them into one verdict.
|
|
185
189
|
|
|
@@ -187,7 +191,7 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
|
|
|
187
191
|
|
|
188
192
|
A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly 2 to 3× your metric command's runtime plus about 40ms of worktree overhead. Point the per-turn gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for CI: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
|
|
189
193
|
|
|
190
|
-
**Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run
|
|
194
|
+
**Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. A run writes nothing to your repo; only `init` and `lock` write files, the ones you commit. A hard-killed run can leave a worktree in your temp dir; a later run removes it once it is a day old (an unregistered checkout of this repo after an hour). It removes only directories named the way it names its own (`promptwheel-`, `promptwheel-src-` or `pw-idx-`, then an 8-hex repo tag, then 6 random characters). Any other folder or worktree in the temp dir is left alone, at any age.
|
|
191
195
|
|
|
192
196
|
## Portability
|
|
193
197
|
|
package/bin/promptwheel.d.ts
CHANGED
|
@@ -48,7 +48,7 @@ export interface MetricConfig {
|
|
|
48
48
|
allowNonzero?: boolean;
|
|
49
49
|
/** Per-command timeout, default 300. */
|
|
50
50
|
timeoutSec?: number;
|
|
51
|
-
/** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). */
|
|
51
|
+
/** `false` skips the source-only re-proof for this metric (tripwires whose gain legitimately lives in test files). Set on a metric that had the re-proof on, it applies from the next change, not the one that sets it. */
|
|
52
52
|
gamingCheck?: boolean;
|
|
53
53
|
/** Per-metric override of `Config.gamingThreshold`. */
|
|
54
54
|
gamingThreshold?: number;
|
|
@@ -62,7 +62,7 @@ export interface MetricConfig {
|
|
|
62
62
|
allowable?: boolean;
|
|
63
63
|
}
|
|
64
64
|
|
|
65
|
-
/** A named hard rule: any change touching one of `paths` fails the gate
|
|
65
|
+
/** A named hard rule: any change touching one of `paths` fails the gate, whatever the metrics read. */
|
|
66
66
|
export interface Mandate {
|
|
67
67
|
name: string;
|
|
68
68
|
/** Globs: a bare name matches that basename at any depth, a pattern with `/` is anchored at the repo root, `**` crosses directories. */
|
|
@@ -85,7 +85,7 @@ export interface Config {
|
|
|
85
85
|
env?: Record<string, string>;
|
|
86
86
|
/** A per-ref build or install, run in the worktree before measuring (a failed setup leaves the metric inert, never a false green). */
|
|
87
87
|
setup?: string;
|
|
88
|
-
/** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. */
|
|
88
|
+
/** Fraction of a gain that must survive the source-only re-run; below it the win is `gamed`. Default 0.5. A change that lowers it is still judged by the value before it. */
|
|
89
89
|
gamingThreshold?: number;
|
|
90
90
|
/** Extra files that define the gate (a threshold, a budget); a change to one alongside a win is gamed by declaration. Same glob rules as `Mandate.paths`. */
|
|
91
91
|
evaluatorPaths?: string[];
|
|
@@ -144,6 +144,8 @@ export interface MetricResult extends Evaluation {
|
|
|
144
144
|
name: string;
|
|
145
145
|
direction: Direction;
|
|
146
146
|
guard: boolean;
|
|
147
|
+
/** With `inert`: where the command ran and failed (`both` refs, or only `base` or `head`). Absent when it ran at neither: exit 126 or 127, a timeout, or the placeholder command `init` writes. */
|
|
148
|
+
ran?: 'both' | 'base' | 'head';
|
|
147
149
|
/** `true` when the win did not survive the source-only re-run; `null` when the source-only patch could not be applied (then `unprovable` is set and the verdict is `inconclusive`, never `pass`). Absent without gaming detection or when the metric did not improve. */
|
|
148
150
|
gamed?: boolean | null;
|
|
149
151
|
/** `true`: the source-only patch could not be applied, so this win could not be re-proven and the verdict is `inconclusive`. `false` only after a declared override cleared it (then `allowed` is `true`). */
|
|
@@ -174,9 +176,13 @@ export interface MandateViolation {
|
|
|
174
176
|
|
|
175
177
|
export interface GateWeakening {
|
|
176
178
|
name: string;
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
179
|
+
/**
|
|
180
|
+
* `guard-edited`: the guard is still on but measured another way (a new `cmd`, `extract`, `direction`, `exact` or `report`).
|
|
181
|
+
* `measurement-edited`: the config's `setup`, `env` or linked dirs changed, and measured with the ones before the change the guard fails at head while the new ones read it fine.
|
|
182
|
+
*/
|
|
183
|
+
change: 'guard-removed' | 'guard-disabled' | 'guard-edited' | 'measurement-edited';
|
|
184
|
+
/** What the guard would have read at head under its base definition (and the base `setup`, `env` and linked dirs); `failed` when it still fails there. */
|
|
185
|
+
status: MetricStatus | 'failed';
|
|
180
186
|
}
|
|
181
187
|
|
|
182
188
|
export interface EvaluatorChange {
|
|
@@ -188,11 +194,11 @@ export interface EvaluatorChange {
|
|
|
188
194
|
|
|
189
195
|
/** The gate-integrity record, present when gaming detection ran. */
|
|
190
196
|
export interface GateRecord {
|
|
191
|
-
/** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates) at each ref; `base` is `null` when there was no config at base. */
|
|
197
|
+
/** Truncated SHA-256 of the gate definition (metrics, guards, evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) at each ref; `base` is `null` when there was no config at base. */
|
|
192
198
|
fingerprint: { base: string | null; head: string };
|
|
193
199
|
/** The gate definition differs between base and head. */
|
|
194
200
|
changed?: true;
|
|
195
|
-
/** Guards the change dropped that would have failed at head; non-empty forces `gamed` unless allowed. */
|
|
201
|
+
/** Guards the change dropped, edited or measures another way that would have failed at head; non-empty forces `gamed` unless allowed. A change that adds the first config is compared with the zero-config gate (the detected test command). */
|
|
196
202
|
weakened?: GateWeakening[];
|
|
197
203
|
/** Declared `evaluatorPaths` files whose content changed in this diff. */
|
|
198
204
|
evaluatorChanged?: EvaluatorChange[];
|
|
@@ -260,7 +266,7 @@ export declare function deriveVerdict(metrics: ReadonlyArray<Partial<MetricResul
|
|
|
260
266
|
* a missing `extends` target, no commits yet, a base or head ref that is not in this clone, no
|
|
261
267
|
* commit to compare against) throws an Error whose `name` is `'ConfigError'` and whose `exitCode`
|
|
262
268
|
* is 64, the code the CLI exits with for the same condition. A `repo` that names a subdirectory
|
|
263
|
-
* is resolved to the repository root, where the config lives
|
|
269
|
+
* is resolved to the repository root, where the config lives. A run writes nothing to the repo.
|
|
264
270
|
*/
|
|
265
271
|
export declare function check(options?: CheckOptions): Report;
|
|
266
272
|
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
|
|
15
15
|
import { execSync, execFileSync } from 'node:child_process';
|
|
16
16
|
import { createHash } from 'node:crypto';
|
|
17
|
-
import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync,
|
|
17
|
+
import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, realpathSync, readdirSync, lstatSync, copyFileSync } from 'node:fs';
|
|
18
18
|
import { tmpdir } from 'node:os';
|
|
19
19
|
import { join, dirname, isAbsolute, relative } from 'node:path';
|
|
20
20
|
import { fileURLToPath } from 'node:url';
|
|
@@ -36,6 +36,9 @@ const spread = (xs) => { const a = xs.filter((x) => x != null); return a.length
|
|
|
36
36
|
// repo is still measuring in (a long gate elsewhere must not lose its worktree mid-run)
|
|
37
37
|
function tmpOwnerTag(repo) { let r = repo; try { r = realpathSync(repo); } catch { /* keep as given */ } return createHash('sha1').update(String(r)).digest('hex').slice(0, 8); }
|
|
38
38
|
const tmpPrefix = (repo, kind) => `${kind}-${tmpOwnerTag(repo)}-`;
|
|
39
|
+
// exactly the names mkdtemp makes from tmpPrefix: the kind, the 8-hex owner tag, then mkdtemp's 6 random
|
|
40
|
+
// characters. Self-heal removes nothing else, so a folder that only starts with "promptwheel-" is never ours.
|
|
41
|
+
const ENGINE_TMP = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-[A-Za-z0-9]{6}$/;
|
|
39
42
|
|
|
40
43
|
// usage / config errors exit 64 (EX_USAGE), so they can never be confused with a verdict
|
|
41
44
|
// (0 pass, 1 fail, 2 gamed, 3 inconclusive): in CI a config typo must not read as GAMED.
|
|
@@ -112,8 +115,14 @@ const hasConfigFile = (repo) => existsSync(join(repo, 'promptwheel.config.json')
|
|
|
112
115
|
// measurement actually runs in a throwaway worktree. Lets `run` and `check()` work with no config
|
|
113
116
|
// file; `init` is for customizing. A repo with no test command gets a sentinel that reads inert ->
|
|
114
117
|
// INCONCLUSIVE (never a fake green), with a message pointing at the fix.
|
|
115
|
-
function autoConfig(repo) {
|
|
116
|
-
return { ...detectEnv(repo), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }] };
|
|
118
|
+
function autoConfig(repo, at = stackAt(repo)) {
|
|
119
|
+
return { ...detectEnv(repo, at), metrics: [{ name: 'tests_pass', cmd: detectTestCmd(repo, at), extract: 'exit', direction: 'pass', guard: true }] };
|
|
120
|
+
}
|
|
121
|
+
// where stack detection looks: the working tree, or the tracked files as of `ref` (the repo as it was before a change)
|
|
122
|
+
function stackAt(repo, ref) {
|
|
123
|
+
if (!ref) return { has: (f) => existsSync(join(repo, f)), read: (f) => readFileSync(join(repo, f), 'utf8') };
|
|
124
|
+
const read = (f) => execFileSync('git', ['show', `${ref}:${f}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] });
|
|
125
|
+
return { has: (f) => { try { read(f); return true; } catch { return false; } }, read };
|
|
117
126
|
}
|
|
118
127
|
|
|
119
128
|
function resolveBase(repo, base) {
|
|
@@ -123,7 +132,8 @@ function resolveBase(repo, base) {
|
|
|
123
132
|
catch { bad('promptwheel: no commits yet, make an initial commit before gating'); }
|
|
124
133
|
for (const b of ['origin/main', 'origin/master', 'main', 'master']) {
|
|
125
134
|
let mb = '';
|
|
126
|
-
|
|
135
|
+
// stderr ignored: a ref that is not there (no origin/main in this clone) is expected, not an error to print
|
|
136
|
+
try { mb = execFileSync('git', ['merge-base', 'HEAD', b], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); } catch { continue; }
|
|
127
137
|
// on the trunk itself the merge-base IS HEAD; comparing a commit to itself would be a
|
|
128
138
|
// trivial PASS that verified nothing, so fall through to the last commit instead.
|
|
129
139
|
if (mb && mb !== head) return mb;
|
|
@@ -132,15 +142,21 @@ function resolveBase(repo, base) {
|
|
|
132
142
|
catch { bad('promptwheel: no commit to compare against (one commit, or a shallow clone): pass --base, or --working'); }
|
|
133
143
|
}
|
|
134
144
|
|
|
135
|
-
|
|
136
|
-
|
|
145
|
+
// exit statuses of a sample list, kept beside it: 127 (the shell found no such command), 126 (it could not
|
|
146
|
+
// execute it) or no status (killed: a timeout) mean the metric never ran, which needs a different fix from a
|
|
147
|
+
// test that ran and failed
|
|
148
|
+
const exitCodes = new WeakMap();
|
|
149
|
+
const neverRan = (samples) => { const c = exitCodes.get(samples) || []; return c.length > 0 && c.every((x) => x == null || x === 126 || x === 127); };
|
|
150
|
+
function runMetric(cwd, m, env, codes) {
|
|
151
|
+
let stdout = '', code = 0, status = 0;
|
|
137
152
|
// Never leak the harness's own node:test context into the measured gate: a metric like
|
|
138
153
|
// `node --test` would see NODE_TEST_CONTEXT, act as a child reporter, and report a bogus
|
|
139
154
|
// exit code (so the gate can't be trusted when PromptWheel itself runs under a test runner).
|
|
140
155
|
const childEnv = { ...(env || process.env) }; delete childEnv.NODE_TEST_CONTEXT;
|
|
141
156
|
try {
|
|
142
157
|
stdout = execSync(m.cmd, { cwd, env: childEnv, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
|
|
143
|
-
} catch (e) { code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
|
|
158
|
+
} catch (e) { status = e.status ?? null; code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
|
|
159
|
+
if (codes) codes.push(status);
|
|
144
160
|
return extract(stdout, code, m.extract, m.allowNonzero);
|
|
145
161
|
}
|
|
146
162
|
|
|
@@ -169,9 +185,9 @@ function extract(stdout, code, mode, allowNonzero) {
|
|
|
169
185
|
// with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
|
|
170
186
|
// Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
|
|
171
187
|
// not here, the engine stays ecosystem-agnostic.
|
|
188
|
+
const linkDirsOf = (cfg) => (Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']));
|
|
172
189
|
function bridgeEnv(repo, wt, cfg) {
|
|
173
|
-
const
|
|
174
|
-
for (const d of linkDirs) {
|
|
190
|
+
for (const d of linkDirsOf(cfg)) {
|
|
175
191
|
if (existsSync(join(repo, d)) && !existsSync(join(wt, d))) {
|
|
176
192
|
try { symlinkSync(join(repo, d), join(wt, d)); } catch { /* best effort */ }
|
|
177
193
|
}
|
|
@@ -196,8 +212,9 @@ function measureAt(repo, ref, cfg, repeat) {
|
|
|
196
212
|
runSetup(wt, cfg, env);
|
|
197
213
|
const out = {};
|
|
198
214
|
for (const m of cfg.metrics) {
|
|
199
|
-
const samples = [];
|
|
200
|
-
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
|
|
215
|
+
const samples = [], codes = [];
|
|
216
|
+
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env, codes));
|
|
217
|
+
exitCodes.set(samples, codes);
|
|
201
218
|
out[m.name] = samples;
|
|
202
219
|
}
|
|
203
220
|
return out;
|
|
@@ -266,9 +283,12 @@ const NON_SOURCE = [
|
|
|
266
283
|
/_spec\.rb$/i,
|
|
267
284
|
/[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
|
|
268
285
|
/\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
|
|
269
|
-
// a grader/eval/score FILE by name (eval.
|
|
270
|
-
// evaluate.ts, evaluation.
|
|
271
|
-
|
|
286
|
+
// a grader/eval/score FILE by name in a scripting language (eval.py, grader-x.js, grade.sh, scoring.ts), never a
|
|
287
|
+
// prefix: evaluate.ts, evaluation.py, scoreboard.js stay source. Compiled names stay source too (eval.c, eval.go
|
|
288
|
+
// are interpreters; Score.java is a type), and so do .tsx/.vue files (a Score component, a /score page).
|
|
289
|
+
/(^|\/)(eval|evals|grade|grader|graders|score|scorer|scoring)([._-][^/]*)?\.([cm]?[jt]s|py|sh|bash|zsh|rb|pl|php|lua|r|jl)$/i,
|
|
290
|
+
// anything under an evals/ or grader(s)/ directory. Not eval/ (an interpreter's package) or scoring/ (a product module).
|
|
291
|
+
/(^|\/)(evals|graders?)\//i,
|
|
272
292
|
// config files only, require a .config/.conf/.setup segment so PRODUCTION source that
|
|
273
293
|
// happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
|
|
274
294
|
// the source slice (a false sweep here can flag an honest win as GAMED).
|
|
@@ -311,9 +331,9 @@ function evaluatorMatcher(cfg) {
|
|
|
311
331
|
// forbids ANY change to its paths (same glob semantics as evaluatorPaths). A hit fails the gate
|
|
312
332
|
// regardless of metrics and rides at the top of the report; mandates inherit through `extends`,
|
|
313
333
|
// so a parent scope's rule holds for every repo beneath it.
|
|
314
|
-
// The mandates in force are the
|
|
315
|
-
//
|
|
316
|
-
// clear it. A mandate removed at head but violated in the diff is reported with `removed: true`; one
|
|
334
|
+
// The mandates in force are the base ref's (`baseMandates`, resolved by gateDefAtRef), in their head
|
|
335
|
+
// version when both have them: deleting a mandate in the same change that violates it does not
|
|
336
|
+
// clear it, and one the change adds does not apply to it (see below). A mandate removed at head but violated in the diff is reported with `removed: true`; one
|
|
317
337
|
// whose paths were edited at head (same name, different paths) is tested in BOTH versions and reported
|
|
318
338
|
// with `edited: true`, so narrowing a rule in the change that breaks it does not clear it either.
|
|
319
339
|
// The honest way to retire one is a separate change that only edits the config.
|
|
@@ -482,12 +502,16 @@ function workingSnapshot(repo) {
|
|
|
482
502
|
}
|
|
483
503
|
}
|
|
484
504
|
|
|
485
|
-
// self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
|
|
505
|
+
// self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
|
|
486
506
|
// its `finally` never ran. On the next run, drop stale worktree registry entries and any
|
|
487
|
-
// abandoned /tmp checkout, so nothing accumulates in your repo or temp dir.
|
|
488
|
-
//
|
|
489
|
-
//
|
|
490
|
-
//
|
|
507
|
+
// abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. It only ever removes a
|
|
508
|
+
// directory whose name is exactly what this engine generates (ENGINE_TMP): a folder named
|
|
509
|
+
// promptwheel-notes, or another repo's worktree named promptwheel-site-wt, is not ours at any age.
|
|
510
|
+
// Ages, by dir mtime: this repo's unregistered checkout after 1h; this repo's still-registered worktree
|
|
511
|
+
// after a full day (a killed run leaves it registered, and a younger one may be a concurrent gate still
|
|
512
|
+
// measuring); a dir tagged for ANOTHER repo after a full day (an orphan of a repo that was deleted or
|
|
513
|
+
// moved: a new real path is a new tag nobody will ever claim). A registered worktree that is not
|
|
514
|
+
// tagged for this repo is never touched: it may be yours.
|
|
491
515
|
function selfHeal(repo) {
|
|
492
516
|
try {
|
|
493
517
|
git(['worktree', 'prune'], repo); // remove registry entries whose dir is already gone (always safe)
|
|
@@ -496,16 +520,18 @@ function selfHeal(repo) {
|
|
|
496
520
|
.split('\n').filter((l) => l.startsWith('worktree ')).map((l) => l.slice(9).trim()),
|
|
497
521
|
);
|
|
498
522
|
const tmp = tmpdir(), now = Date.now(), mine = tmpOwnerTag(repo);
|
|
499
|
-
const TAGGED = /^(promptwheel-src|promptwheel|pw-idx)-([0-9a-f]{8})-/;
|
|
500
523
|
for (const name of readdirSync(tmp)) {
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
524
|
+
const made = ENGINE_TMP.exec(name);
|
|
525
|
+
if (!made) continue; // not a name this engine generates (worktrees + temp-index dirs): never removed
|
|
526
|
+
const tag = made[2];
|
|
527
|
+
// ours after an hour; another repo's only after a full day
|
|
505
528
|
const p = join(tmp, name);
|
|
506
|
-
if (tracked.has(p)) continue;
|
|
507
|
-
|
|
529
|
+
if (tracked.has(p) && tag !== mine) continue; // a registered worktree this repo's gate did not make
|
|
530
|
+
const maxAge = tag === mine && !tracked.has(p) ? 3600_000 : 86_400_000;
|
|
531
|
+
// lstat: the engine makes directories, so a file or a symlink under one of its names is left alone
|
|
532
|
+
try { const st = lstatSync(p); if (st.isDirectory() && st.mtimeMs < now - maxAge) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
|
|
508
533
|
}
|
|
534
|
+
git(['worktree', 'prune'], repo); // drop the registry entries of anything removed above
|
|
509
535
|
} catch { /* cleanup must never block the gate */ }
|
|
510
536
|
}
|
|
511
537
|
|
|
@@ -537,18 +563,25 @@ function warnEditableInstall(repo) {
|
|
|
537
563
|
// guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
|
|
538
564
|
// guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
|
|
539
565
|
// that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
|
|
540
|
-
const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer'];
|
|
566
|
+
const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths', 'allowTrailer', 'setup', 'env', 'linkDirs', 'linkNodeModules'];
|
|
567
|
+
// what runs or shapes every measurement besides the metric's own command. A setup can rewrite a test and an
|
|
568
|
+
// env var can skip one, so these are part of the gate: fingerprinted, and held at their base values for the
|
|
569
|
+
// change that edits them (see detectGamingPass).
|
|
570
|
+
const measureOf = (s) => ({ setup: s.setup, env: s.env, linkDirs: s.linkDirs, linkNodeModules: s.linkNodeModules });
|
|
571
|
+
const measuredWith = (s) => JSON.stringify([s.setup ?? null, Object.entries(s.env || {}).map(([k, v]) => `${k}=${v}`).sort(), [...linkDirsOf(s)].sort()]);
|
|
541
572
|
|
|
542
573
|
// Resolve the gate definition as of `ref` (metrics + mandates + scalars), following `extends` by
|
|
543
574
|
// reading every file from that ref via `git show`, mirroring resolveConfig (a parent's mandates
|
|
544
575
|
// hold beneath it; a local mandate of the same name overrides). Returns { metrics:Map, scalars }
|
|
545
|
-
// with scalars.mandates, or null when there is no config at `ref` (
|
|
546
|
-
//
|
|
547
|
-
// branch simply contributes nothing)
|
|
576
|
+
// with scalars.mandates, or null when there is no config at `ref` (gate() then falls back to the
|
|
577
|
+
// zero-config gate the repo ran under, see zeroConfigGate). Never exits: any missing/unreadable file or
|
|
578
|
+
// parse error yields null (that branch simply contributes nothing), and `partial` says an extends target
|
|
579
|
+
// could not be read at `ref` (it lives outside the repo), so how the base measured is not fully known.
|
|
548
580
|
function gateDefAtRef(repo, ref) {
|
|
549
581
|
// quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
|
|
550
582
|
const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
|
|
551
583
|
const seen = new Set();
|
|
584
|
+
let partial = false;
|
|
552
585
|
const resolve = (p) => {
|
|
553
586
|
if (seen.has(p)) return null;
|
|
554
587
|
seen.add(p);
|
|
@@ -557,9 +590,11 @@ function gateDefAtRef(repo, ref) {
|
|
|
557
590
|
let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
|
|
558
591
|
const byName = new Map(); const mandatesByName = new Map(); const scalars = {};
|
|
559
592
|
for (const ext of [].concat(cfg.extends || [])) {
|
|
560
|
-
if (isAbsolute(ext)) continue;
|
|
561
|
-
const
|
|
562
|
-
|
|
593
|
+
if (isAbsolute(ext)) { partial = true; continue; } // outside the repo, cannot git-show; unchanged anyway
|
|
594
|
+
const bp = join(dirname(p), ext).split('\\').join('/');
|
|
595
|
+
const again = seen.has(bp); // a diamond: already merged, not a missing piece
|
|
596
|
+
const base = resolve(bp);
|
|
597
|
+
if (!base) { if (!again) partial = true; continue; }
|
|
563
598
|
for (const m of base.metrics.values()) byName.set(m.name, m);
|
|
564
599
|
for (const md of base.scalars.mandates) mandatesByName.set(md.name, md);
|
|
565
600
|
for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
|
|
@@ -570,18 +605,35 @@ function gateDefAtRef(repo, ref) {
|
|
|
570
605
|
scalars.mandates = [...mandatesByName.values()];
|
|
571
606
|
return { metrics: byName, scalars };
|
|
572
607
|
};
|
|
573
|
-
|
|
608
|
+
const def = resolve('promptwheel.config.json');
|
|
609
|
+
return def && { ...def, partial };
|
|
610
|
+
}
|
|
611
|
+
|
|
612
|
+
// The gate a repo with no config ran under: the detected test command (source-only re-run on) with the
|
|
613
|
+
// stack's env and linkDirs, detected from the tracked files as of `ref`. A change that adds the first
|
|
614
|
+
// config is judged against this, so `tests_pass.cmd: "true"` in a new config is an edited guard, not a
|
|
615
|
+
// fresh start. `initSetup` is the setup `init` would add: a first config that carries exactly it has not
|
|
616
|
+
// changed how the gate measures. With no test command to detect (the placeholder) the old gate measured
|
|
617
|
+
// nothing, so its setup is no baseline (`partial`).
|
|
618
|
+
function zeroConfigGate(repo, ref) {
|
|
619
|
+
const at = stackAt(repo, ref), auto = autoConfig(repo, at);
|
|
620
|
+
return { metrics: new Map(auto.metrics.map((m) => [m.name, m])), scalars: { mandates: [], ...measureOf(auto) }, zeroConfig: true, initSetup: detectSetup(repo, at), partial: auto.metrics[0].cmd === PLACEHOLDER_TEST };
|
|
574
621
|
}
|
|
575
622
|
|
|
623
|
+
// how a metric is measured and judged: change any of these on a guard and it is a different guard
|
|
624
|
+
// what init writes when it finds no test command. It guards nothing, so replacing it is setup, not a weakened gate.
|
|
625
|
+
const PLACEHOLDER_TEST = 'echo "set your test command in promptwheel.config.json" && false';
|
|
626
|
+
const measuredBy = (m) => JSON.stringify([m.cmd ?? null, m.extract ?? 'number', m.direction || 'up', m.exact === true, m.report === true]);
|
|
576
627
|
// an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
|
|
577
628
|
function gateFingerprint(metricsList, scalars) {
|
|
578
629
|
const norm = {
|
|
579
630
|
metrics: metricsList.map((m) => ({
|
|
580
|
-
name: m.name, guard: m.guard === true,
|
|
631
|
+
name: m.name, guard: m.guard === true, measuredBy: measuredBy(m),
|
|
581
632
|
gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
|
|
582
633
|
})).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
|
|
583
634
|
gamingThreshold: scalars.gamingThreshold ?? null,
|
|
584
635
|
evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
|
|
636
|
+
measuredWith: measuredWith(scalars),
|
|
585
637
|
mandates: Array.isArray(scalars.mandates) ? scalars.mandates.map((m) => ({ name: String(m.name || ''), paths: [...(Array.isArray(m.paths) ? m.paths : [])].sort() })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)) : [],
|
|
586
638
|
};
|
|
587
639
|
return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
|
|
@@ -626,7 +678,8 @@ function verdictFingerprint(report) {
|
|
|
626
678
|
return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
|
|
627
679
|
}
|
|
628
680
|
|
|
629
|
-
// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
|
|
681
|
+
// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off), or
|
|
682
|
+
// measures another way (a new cmd, extract, direction, exact or report flag): `cmd: "true"` is a dropped guard too
|
|
630
683
|
function gateWeakenings(baseGate, headMetrics) {
|
|
631
684
|
if (!baseGate) return [];
|
|
632
685
|
const head = new Map(headMetrics.map((m) => [m.name, m]));
|
|
@@ -636,6 +689,7 @@ function gateWeakenings(baseGate, headMetrics) {
|
|
|
636
689
|
const hm = head.get(bm.name);
|
|
637
690
|
if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
|
|
638
691
|
else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
|
|
692
|
+
else if (!bm.report && bm.cmd !== PLACEHOLDER_TEST && measuredBy(hm) !== measuredBy(bm)) weak.push({ name: bm.name, change: 'guard-edited', def: bm });
|
|
639
693
|
}
|
|
640
694
|
return weak;
|
|
641
695
|
}
|
|
@@ -654,29 +708,51 @@ function readAllowTrailer(repo, ref) {
|
|
|
654
708
|
return m && m[1].trim() ? { reason: m[1].trim().slice(0, 300) } : null;
|
|
655
709
|
}
|
|
656
710
|
|
|
711
|
+
// A change is judged by the gate that was in force before it (`baseGate`: the config at base, or the
|
|
712
|
+
// zero-config gate when this change adds the first config). So a change cannot switch off the source-only
|
|
713
|
+
// re-run for a metric that had it, loosen gamingThreshold, or swap the setup, env or linked dirs it is
|
|
714
|
+
// measured with: the base values hold for the change that edits them, and apply from the next one.
|
|
715
|
+
const gamingCheckOff = (cm, baseGate) => {
|
|
716
|
+
if (!cm || cm.gamingCheck !== false) return false;
|
|
717
|
+
const bm = baseGate && baseGate.metrics.get(cm.name);
|
|
718
|
+
return !bm || bm.gamingCheck === false; // off before this change too, or a metric this change adds
|
|
719
|
+
};
|
|
720
|
+
function gamingThresholdFor(cm, cfg, baseGate) {
|
|
721
|
+
const now = cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5;
|
|
722
|
+
const bm = baseGate && baseGate.metrics.get(cm.name);
|
|
723
|
+
const before = bm ? (bm.gamingThreshold ?? baseGate.scalars.gamingThreshold ?? (baseGate.partial ? now : 0.5)) : now;
|
|
724
|
+
return Math.max(now, before); // the stricter of before and after
|
|
725
|
+
}
|
|
726
|
+
|
|
657
727
|
// The three Section 3 checks, run only with detectGaming, in order: (1) re-prove every actual
|
|
658
728
|
// win with the agent's source edits alone; (2) content-hash auto-fail: a declared evaluatorPath
|
|
659
729
|
// gate file edited in this diff is a moved goalpost, so a metric that improved alongside it is
|
|
660
730
|
// gamed by declaration, deterministic even if the source-only re-run read inert; (3) gate
|
|
661
|
-
// integrity:
|
|
662
|
-
// definition (it only counts as tampering if it
|
|
663
|
-
// gate definition at both refs (metrics,
|
|
664
|
-
//
|
|
731
|
+
// integrity: take the gate as of BASE, re-measure any guard HEAD dropped, edited or measures with
|
|
732
|
+
// another setup, env or linked dirs, under its base definition (it only counts as tampering if it
|
|
733
|
+
// WOULD HAVE FAILED at head), and fingerprint the gate definition at both refs (metrics,
|
|
734
|
+
// evaluatorPaths, gamingThreshold, mandates, setup, env, linked dirs) so a quietly deleted guard
|
|
735
|
+
// or mandate is tamper-evident: base != head is recorded as gate.changed and
|
|
665
736
|
// printed, whatever the verdict. Mutates `metrics` in place (gamed/sourceOnly/retained/
|
|
666
737
|
// gamingReason) and returns the report.gate record. An allow trailer can still clear a gamed
|
|
667
738
|
// metric afterwards, in gate().
|
|
668
|
-
function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
739
|
+
function detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) {
|
|
740
|
+
// how the base measured. Unknown (`partial`) reads as unchanged; so does a first config that adds only the setup `init` detects
|
|
741
|
+
const bs = baseGate ? (baseGate.partial ? { ...baseGate.scalars, ...measureOf(cfg) } : baseGate.scalars) : null;
|
|
742
|
+
const asInit = !!bs && baseGate.zeroConfig && measuredWith({ ...bs, setup: baseGate.initSetup }) === measuredWith(cfg);
|
|
743
|
+
const measureChanged = !!bs && !asInit && measuredWith(bs) !== measuredWith(cfg);
|
|
744
|
+
const baseRun = measureChanged ? { ...cfg, ...measureOf(bs) } : cfg; // the head gate, run with the base setup, env and linked dirs
|
|
669
745
|
const toProve = [];
|
|
670
746
|
for (const m of metrics) {
|
|
671
747
|
if (m.status !== 'improved') continue;
|
|
672
748
|
const cm = cfg.metrics.find((c) => c.name === m.name);
|
|
673
|
-
if (cm
|
|
749
|
+
if (gamingCheckOff(cm, baseGate)) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
|
|
674
750
|
toProve.push(cm);
|
|
675
751
|
}
|
|
676
|
-
const srcResults = measureSourceOnlyAll(repo, base, head, toProve,
|
|
752
|
+
const srcResults = measureSourceOnlyAll(repo, base, head, toProve, baseRun, repeat); // one patched checkout for all of them
|
|
677
753
|
for (const cm of toProve) {
|
|
678
754
|
const m = metrics.find((x) => x.name === cm.name);
|
|
679
|
-
const j = judgeGaming(m, srcResults[cm.name], cm
|
|
755
|
+
const j = judgeGaming(m, srcResults[cm.name], gamingThresholdFor(cm, cfg, baseGate));
|
|
680
756
|
m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
|
|
681
757
|
if (j.unprovable) m.unprovable = true;
|
|
682
758
|
}
|
|
@@ -687,32 +763,55 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
|
687
763
|
if (split.source.length && split.nonSource.length) {
|
|
688
764
|
const held = metrics.filter((m) => m.status === 'unchanged' && m.direction === 'pass' && m.after === 1 && !m.tripwire);
|
|
689
765
|
const defs = held.map((m) => cfg.metrics.find((c) => c.name === m.name)).filter(Boolean);
|
|
690
|
-
const res = measureSourceOnlyAll(repo, base, head, defs,
|
|
766
|
+
const res = measureSourceOnlyAll(repo, base, head, defs, baseRun, repeat);
|
|
691
767
|
for (const m of held) { const r = res[m.name]; if (r && r.samples && r.samples.length && median(r.samples) !== 1) m.heldByTestEdit = true; }
|
|
692
768
|
}
|
|
693
|
-
|
|
769
|
+
// a file declared as a goalpost before this change still is one for this change: dropping the declaration in the
|
|
770
|
+
// same diff that edits the file does not make the edit count as source
|
|
771
|
+
const declared = [...new Set([...(cfg.evaluatorPaths || []), ...((baseGate && baseGate.scalars.evaluatorPaths) || [])])];
|
|
772
|
+
const evalChanged = changedEvaluatorPaths(repo, base, head, { ...cfg, evaluatorPaths: declared });
|
|
694
773
|
if (evalChanged.length) {
|
|
695
774
|
for (const m of metrics) {
|
|
696
775
|
if (m.status !== 'improved' || m.gamed === true) continue;
|
|
697
776
|
const cm = cfg.metrics.find((c) => c.name === m.name);
|
|
698
|
-
if (cm
|
|
777
|
+
if (gamingCheckOff(cm, baseGate)) continue; // tripwire/test-side guards legitimately live in gate files
|
|
699
778
|
m.gamed = true;
|
|
700
779
|
m.gamingReason = `a declared evaluatorPath gate file changed in this diff (${evalChanged.map((e) => e.path).join(', ')}); a win alongside a goalpost edit is gamed by declaration`;
|
|
701
780
|
}
|
|
702
781
|
}
|
|
703
|
-
const baseGate = gateDefAtRef(repo, base);
|
|
704
782
|
const weak = gateWeakenings(baseGate, cfg.metrics);
|
|
783
|
+
// the setup, env or linked dirs changed: every guard the base had is measured again the way it was measured before
|
|
784
|
+
if (measureChanged) for (const bm of baseGate.metrics.values()) {
|
|
785
|
+
if (bm.guard === true && !bm.report && bm.cmd !== PLACEHOLDER_TEST && !weak.some((w) => w.name === bm.name)) weak.push({ name: bm.name, change: 'measurement-edited', def: bm });
|
|
786
|
+
}
|
|
705
787
|
const weakened = [];
|
|
706
788
|
if (weak.length) {
|
|
707
789
|
const defs = weak.map((w) => w.def);
|
|
708
|
-
const b = measureAt(repo, base, { ...
|
|
709
|
-
const a = measureAt(repo, head, { ...
|
|
790
|
+
const b = measureAt(repo, base, { ...baseRun, metrics: defs }, repeat);
|
|
791
|
+
const a = measureAt(repo, head, { ...baseRun, metrics: defs }, repeat);
|
|
792
|
+
const evs = new Map();
|
|
710
793
|
for (const w of weak) {
|
|
711
794
|
const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
|
|
712
|
-
|
|
795
|
+
evs.set(w.name, ev);
|
|
796
|
+
// an edited guard that still fails under its old definition was turned green by the edit, not the code
|
|
797
|
+
let hidden = !ev.ok || (w.change === 'guard-edited' && ev.inert);
|
|
798
|
+
// a guard measured another way counts only when the new measurement hides it: it fails the old way and reads fine the new way
|
|
799
|
+
if (w.change === 'measurement-edited') { const hm = metrics.find((x) => x.name === w.name); hidden = (!ev.ok || !!ev.inert) && !!hm && hm.ok && !hm.inert; }
|
|
800
|
+
if (hidden) weakened.push({ name: w.name, change: w.change, status: ev.inert ? 'failed' : ev.status });
|
|
801
|
+
}
|
|
802
|
+
// a win under the old definition still has to be earned in the source: swapping or renaming the guard in the same
|
|
803
|
+
// change does not skip the source-only re-run
|
|
804
|
+
const won = weak.filter((w) => w.def.gamingCheck !== false && !weakened.some((x) => x.name === w.name) && evs.get(w.name).status === 'improved');
|
|
805
|
+
if (won.length) {
|
|
806
|
+
const src = measureSourceOnlyAll(repo, base, head, won.map((w) => w.def), baseRun, repeat);
|
|
807
|
+
for (const w of won) {
|
|
808
|
+
const j = judgeGaming({ name: w.name, direction: w.def.direction || 'up', guard: true, ...evs.get(w.name) }, src[w.name], w.def.gamingThreshold ?? (bs && bs.gamingThreshold) ?? 0.5);
|
|
809
|
+
if (j.gamed) weakened.push({ name: w.name, change: w.change, status: 'gamed' });
|
|
810
|
+
}
|
|
713
811
|
}
|
|
714
812
|
}
|
|
715
|
-
|
|
813
|
+
// the fingerprint is of a config file: `base` stays null when there was none there (the zero-config gate is not a file)
|
|
814
|
+
const fpBase = baseGate && !baseGate.zeroConfig ? gateFingerprint([...baseGate.metrics.values()], bs) : null;
|
|
716
815
|
const fpHead = gateFingerprint(cfg.metrics, cfg);
|
|
717
816
|
const info = {
|
|
718
817
|
fingerprint: { base: fpBase, head: fpHead },
|
|
@@ -769,17 +868,26 @@ function gate(repo, opts) {
|
|
|
769
868
|
// a guard this change adds has no value at base, because its scanner did not exist there. That is adoption,
|
|
770
869
|
// not a failure: it starts counting from head. Removing a guard is still caught (gateWeakenings), and a guard the
|
|
771
870
|
// base contract already had still fails when it cannot be measured. A zero-config run has no contract to adopt.
|
|
772
|
-
const
|
|
871
|
+
const fileGate = gateDefAtRef(repo, base);
|
|
872
|
+
const baseDefs = opts.config ? null : fileGate;
|
|
773
873
|
const adopted = (name) => !opts.config && !(baseDefs && baseDefs.metrics.has(name));
|
|
874
|
+
// the gate in force before this change: the config committed at base, or, when this change adds the
|
|
875
|
+
// first config, the zero-config gate the repo ran under until now
|
|
876
|
+
const baseGate = opts.detectGaming ? (fileGate || (opts.config ? null : zeroConfigGate(repo, base))) : null;
|
|
774
877
|
const metrics = cfg.metrics.map((m) => {
|
|
775
878
|
const ev = evaluate(m, before[m.name], after[m.name], repeat);
|
|
776
879
|
if (ev.status === 'unmeasurable' && ev.before == null && ev.after != null && adopted(m.name)) { ev.ok = true; ev.adopted = true; }
|
|
777
880
|
// a `report` guard tells you and does not fail: a contract can start by watching and tighten later
|
|
778
881
|
if (m.report && !ev.ok && ev.status === 'regressed') { ev.ok = true; ev.noted = true; }
|
|
779
|
-
|
|
882
|
+
// an inert guard: did its command run and fail, or never run? `ran` says where it ran (absent: at neither ref)
|
|
883
|
+
if (ev.inert && m.cmd !== PLACEHOLDER_TEST) {
|
|
884
|
+
const nb = neverRan(before[m.name]), na = neverRan(after[m.name]);
|
|
885
|
+
if (!nb || !na) ev.ran = !nb && !na ? 'both' : nb ? 'head' : 'base';
|
|
886
|
+
}
|
|
887
|
+
return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(gamingCheckOff(m, baseGate) ? { tripwire: true } : {}), ...ev };
|
|
780
888
|
});
|
|
781
889
|
// Section 3 (gaming detection), on by default: mutates `metrics` and yields the gate record
|
|
782
|
-
const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat) : null;
|
|
890
|
+
const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat, baseGate) : null;
|
|
783
891
|
// declared override: a Promptwheel-Allow trailer (or --allow) acknowledges an intentional
|
|
784
892
|
// goalpost/gate change, so a would-be GAMED becomes PASS, recorded in report.allowed for review.
|
|
785
893
|
// It covers an `unprovable` win too (the source-only re-run could not be applied): the declared
|
|
@@ -820,7 +928,7 @@ function gate(repo, opts) {
|
|
|
820
928
|
for (const m of ga) { m.ok = true; m.allowed = true; allowed.cleared.push(m.name); }
|
|
821
929
|
} else for (const m of ga) m.allowable = true; // failed, and a declared reason would clear it: say so in the report
|
|
822
930
|
}
|
|
823
|
-
// mandates
|
|
931
|
+
// mandates: a protected-path hard rule fails the gate no matter what the metrics say.
|
|
824
932
|
// The base ref's mandates count too (resolved through `extends` by gateDefAtRef), so a change
|
|
825
933
|
// cannot clear a mandate by deleting it in the same diff that violates it.
|
|
826
934
|
const baseMandates = (gateInfo && gateInfo.baseMandates) || (!opts.detectGaming ? ((gateDefAtRef(repo, base) || {}).scalars || {}).mandates : null);
|
|
@@ -906,7 +1014,7 @@ const PRESETS = {
|
|
|
906
1014
|
{ name: 'codeowners-frozen', paths: ['CODEOWNERS'], reason: 'review ownership is set by people' },
|
|
907
1015
|
{ name: 'scanner-frozen', paths: ['.promptwheel/packs/**'], reason: 'the scanners that read the contract are not edited by the change they read' },
|
|
908
1016
|
{ name: 'lock-frozen', paths: ['promptwheel.lock.json'], allowable: 'flag', reason: 'the architecture lock changes with a reason from a person (--allow), never as a side effect of a change' },
|
|
909
|
-
{ name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], reason: 'the hooks that enforce the contract
|
|
1017
|
+
{ name: 'hooks-frozen', paths: ['.claude/settings.json', '.claude/settings.local.json', '.sawbuck/**'], allowable: 'flag', reason: 'the hooks that enforce the contract change with a reason from a person (--allow), and are never detached by the agent' },
|
|
910
1018
|
],
|
|
911
1019
|
metrics: [
|
|
912
1020
|
{ name: 'lifecycle_scripts', cmd: 'node packs/supply-chain/scan.mjs lifecycle .', extract: 'number', direction: 'down', guard: true, gamingCheck: false, allowable: 'flag' },
|
|
@@ -919,17 +1027,17 @@ const PRESETS = {
|
|
|
919
1027
|
] },
|
|
920
1028
|
};
|
|
921
1029
|
|
|
922
|
-
function detectTestCmd(repo) {
|
|
923
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1030
|
+
function detectTestCmd(repo, { has, read } = stackAt(repo)) {
|
|
924
1031
|
if (has('go.mod')) return 'go test ./...';
|
|
925
1032
|
if (has('Cargo.toml')) return 'cargo test';
|
|
926
|
-
|
|
1033
|
+
// the venv is not tracked: it is looked for on disk whichever ref the stack is read from
|
|
1034
|
+
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return existsSync(join(repo, '.venv/bin/pytest')) ? '.venv/bin/pytest -q' : 'pytest -q';
|
|
927
1035
|
// only trust `npm test` when a test script actually exists, otherwise the metric can
|
|
928
1036
|
// never pass and the gate would "protect" nothing (very common in Next.js app repos)
|
|
929
1037
|
try {
|
|
930
|
-
if (JSON.parse(
|
|
1038
|
+
if (JSON.parse(read('package.json')).scripts?.test) return 'npm test --silent';
|
|
931
1039
|
} catch { /* no or unparsable package.json, fall through */ }
|
|
932
|
-
return
|
|
1040
|
+
return PLACEHOLDER_TEST;
|
|
933
1041
|
}
|
|
934
1042
|
|
|
935
1043
|
// deps that live outside git + the env a test needs, defaulted per stack. init writes these into
|
|
@@ -938,18 +1046,16 @@ function detectTestCmd(repo) {
|
|
|
938
1046
|
// where the detected test command cannot run without a build first (a 10-repo replay put this at
|
|
939
1047
|
// ~2/3 of gates). Written by `init` into `setup`; a failed setup stays honest (inert -> inconclusive,
|
|
940
1048
|
// never a fake green). Kept out of the zero-config default so the fast per-turn path stays fast.
|
|
941
|
-
function detectSetup(repo) {
|
|
942
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1049
|
+
function detectSetup(repo, { has, read } = stackAt(repo)) {
|
|
943
1050
|
try {
|
|
944
|
-
const pkg = JSON.parse(
|
|
1051
|
+
const pkg = JSON.parse(read('package.json'));
|
|
945
1052
|
if (pkg.scripts && pkg.scripts.build) return 'npm run build --if-present'; // TS/bundler tests that run against a built dir
|
|
946
1053
|
} catch { /* no or unparsable package.json */ }
|
|
947
1054
|
if (has('pyproject.toml') || has('setup.py')) return 'pip install -e . --quiet 2>/dev/null || true'; // editable install so imports resolve the measured ref
|
|
948
1055
|
return undefined;
|
|
949
1056
|
}
|
|
950
1057
|
|
|
951
|
-
function detectEnv(repo) {
|
|
952
|
-
const has = (f) => existsSync(join(repo, f));
|
|
1058
|
+
function detectEnv(repo, { has } = stackAt(repo)) {
|
|
953
1059
|
// Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
|
|
954
1060
|
// resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
|
|
955
1061
|
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
|
|
@@ -1064,6 +1170,7 @@ function writeAgentBlock(repo) {
|
|
|
1064
1170
|
|
|
1065
1171
|
const short = (repo, ref) => { try { return git(['rev-parse', '--short', ref], repo); } catch { return ref; } };
|
|
1066
1172
|
|
|
1173
|
+
const wouldHave = (status) => (status === 'unmeasurable' ? 'been unmeasurable' : status);
|
|
1067
1174
|
function printHuman(r) {
|
|
1068
1175
|
const arrowFor = (m) => (m.delta == null ? '?' : m.delta > 0 ? '▲' : m.delta < 0 ? '▼' : '=');
|
|
1069
1176
|
console.log(`\nPromptWheel ${r.base} → ${r.head}${r.repeat > 1 ? ` (×${r.repeat})` : ''}\n`);
|
|
@@ -1072,15 +1179,21 @@ function printHuman(r) {
|
|
|
1072
1179
|
const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
|
|
1073
1180
|
const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
|
|
1074
1181
|
console.log(` ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)} (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
|
|
1075
|
-
if (m.inert) console.log(` ⚠ "${m.name}"
|
|
1182
|
+
if (m.inert && m.ran === 'both') console.log(` ⚠ "${m.name}" ran and failed before and after this change, so it guards nothing here. Fix the failing tests, or the command, before trusting a pass`);
|
|
1183
|
+
else if (m.inert) console.log(` ⚠ "${m.name}" ${m.ran ? `ran and failed at the ${m.ran} ref and did not run at the other` : 'never ran (same value at both refs)'}: its command needs a build/deps first, or is not your real test command. Fix: promptwheel init, or set setup/linkDirs in promptwheel.config.json`);
|
|
1076
1184
|
if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
|
|
1077
1185
|
}
|
|
1078
|
-
if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(
|
|
1079
|
-
|
|
1186
|
+
if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(w.change === 'measurement-edited'
|
|
1187
|
+
? ` 🚩 GATE WEAKENED, how the guard "${w.name}" is measured (setup, env or linked dirs) was edited in this change, and measured the way it was before it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`
|
|
1188
|
+
: ` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : 'disabled'} in this change and it would have ${wouldHave(w.status)} at head (goalpost moved, not earned)`);
|
|
1189
|
+
if (r.gate && r.gate.changed) console.log(` ⚠ gate definition changed in this change (fingerprint ${r.gate.fingerprint.base} -> ${r.gate.fingerprint.head}): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited; review promptwheel.config.json as part of the diff`);
|
|
1080
1190
|
const failReason = r.mandates ? ', a mandate was violated (a protected path changed)' : ', a guarded metric regressed (beyond noise)';
|
|
1081
1191
|
console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? failReason : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
|
|
1082
|
-
if (r.verdict === 'fail' && r.mandates)
|
|
1083
|
-
|
|
1192
|
+
if (r.verdict === 'fail' && r.mandates) {
|
|
1193
|
+
const takes = r.mandates.filter((md) => md.allowable).map((md) => `"${md.name}"`);
|
|
1194
|
+
if (takes.length) console.log(` intentional? ${takes.join(', ')} ${takes.length > 1 ? 'accept' : 'accepts'} a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record`);
|
|
1195
|
+
if (takes.length < r.mandates.length) console.log(` ${takes.length ? 'any other' : 'a'} mandate is a hard rule; if the change is truly intended, retire the mandate in promptwheel.config.json in a change of its own first (a tracked, tamper-evident change), do not work around it`);
|
|
1196
|
+
} else if (r.verdict === 'fail' && r.metrics.some((m) => m.allowable)) console.log(` intentional? this guard accepts a person's declared reason: promptwheel run --allow "<reason>" passes with the reason on the record (${r.metrics.filter((m) => m.allowable).map((m) => m.name).join(', ')})`);
|
|
1084
1197
|
else if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name)');
|
|
1085
1198
|
if (r.allowed) console.log(` ✓ allowed: ${r.allowed.cleared.join(', ')} acknowledged as an intentional change, recorded (Promptwheel-Allow: ${r.allowed.reason})`);
|
|
1086
1199
|
console.log('');
|
|
@@ -1134,9 +1247,10 @@ function renderMarkdown(r) {
|
|
|
1134
1247
|
'|---|--:|--:|--:|---|---|',
|
|
1135
1248
|
rows,
|
|
1136
1249
|
'',
|
|
1137
|
-
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule
|
|
1138
|
-
...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by
|
|
1139
|
-
...(r.
|
|
1250
|
+
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? (r.mandates ? '> ⛔ A protected path changed. This is a hard rule; no metric result clears it.' : '> ❌ A 🛡️ guarded metric regressed beyond the noise band.') : r.verdict === 'inconclusive' ? (r.metrics.some((m) => m.inert) && r.metrics.every((m) => !m.inert || m.ran === 'both') ? '> 🟡 A 🛡️ guarded check is **inert**, it fails before and after this change, so nothing was actually verified. Fix it before trusting a pass.' : '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.') : '> ✅ No guarded metric regressed.',
|
|
1251
|
+
...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : w.change === 'guard-edited' ? 'edited' : w.change === 'measurement-edited' ? 'measured another way (setup, env or linked dirs)' : 'disabled'} (would have ${wouldHave(w.status)} at head)`).join('; ')}. The pass was bought by changing the gate, not by earning it.`] : []),
|
|
1252
|
+
...(r.allowed ? ['', `> ✓ **Allowed:** \`${r.allowed.cleared.join('`, `')}\` cleared by a declared reason: ${r.allowed.reason}`] : []),
|
|
1253
|
+
...(r.gate && r.gate.changed ? ['', `> ⚠ **Gate definition changed** in this diff (fingerprint \`${r.gate.fingerprint.base}\` -> \`${r.gate.fingerprint.head}\`): a metric, guard, evaluatorPath, mandate, or the setup, env or linked dirs they are measured with was added, removed or edited. Review the config change with the code.`] : []),
|
|
1140
1254
|
'',
|
|
1141
1255
|
'<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
|
|
1142
1256
|
].join('\n');
|
|
@@ -1169,7 +1283,7 @@ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluat
|
|
|
1169
1283
|
// Importing this file never runs the CLI (see the isMain guard at the bottom).
|
|
1170
1284
|
// Stable across 0.x minor versions. The schema check() returns and the exit codes
|
|
1171
1285
|
// verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
|
|
1172
|
-
// shell, import
|
|
1286
|
+
// shell, import) are in INTEGRATE.md.
|
|
1173
1287
|
// ─────────────────────────────────────────────────────────────────────────────
|
|
1174
1288
|
|
|
1175
1289
|
// The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
|
|
@@ -1207,8 +1321,8 @@ export function verdictExitCode(verdict) {
|
|
|
1207
1321
|
// detectGaming boolean run the Section 3 source-only re-proof (default: true)
|
|
1208
1322
|
// allow string a declared override reason (same as --allow / a Promptwheel-Allow: trailer)
|
|
1209
1323
|
export function check(options = {}) {
|
|
1210
|
-
// `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
|
|
1211
|
-
//
|
|
1324
|
+
// `repo` may name a subdirectory: the gate runs at the toplevel (that is where the config
|
|
1325
|
+
// lives), exactly as the CLI does from the same directory
|
|
1212
1326
|
let start = process.cwd();
|
|
1213
1327
|
if (options.repo) { try { start = realpathSync(options.repo); } catch { bad(`promptwheel: not inside a git repository (${options.repo})`); } }
|
|
1214
1328
|
const repo = repoRoot(start);
|
|
@@ -1328,17 +1442,35 @@ const GATE_FLAGS = ['--base', '--head', '--repeat', '--working', '--no-record',
|
|
|
1328
1442
|
const COMMAND_FLAGS = {
|
|
1329
1443
|
run: GATE_FLAGS, init: ['--preset', '--list', '--force', '--no-agent-block'], lock: [],
|
|
1330
1444
|
};
|
|
1331
|
-
|
|
1445
|
+
// what each valued flag needs. A missing or unusable value is a usage error too: `--repeat abc` must not
|
|
1446
|
+
// measure nothing and read FAIL, and `--allow` with nothing after it must not run as if no reason was asked for.
|
|
1447
|
+
const VALUED_FLAGS = { '--base': 'a ref', '--head': 'a ref', '--repeat': 'a whole number, 1 or more', '--allow': 'a reason, e.g. --allow "the spec changed"', '--preset': 'a preset name (see promptwheel init --list)' };
|
|
1332
1448
|
function checkFlags(cmd, argv) {
|
|
1333
1449
|
const known = new Set(COMMAND_FLAGS[cmd]);
|
|
1334
1450
|
for (let i = 0; i < argv.length; i++) {
|
|
1335
1451
|
const a = argv[i];
|
|
1336
1452
|
if (!a.startsWith('-')) continue;
|
|
1337
1453
|
if (!known.has(a)) bad(`promptwheel ${cmd}: unknown flag ${a} (see promptwheel --help)`);
|
|
1338
|
-
if (
|
|
1454
|
+
if (!(a in VALUED_FLAGS)) continue;
|
|
1455
|
+
const v = argv[++i];
|
|
1456
|
+
const gone = v === undefined || known.has(v); // nothing after the flag, or the next flag
|
|
1457
|
+
// an empty --base or --head is the default ref: the Action passes an empty base outside a pull request
|
|
1458
|
+
const blank = !gone && !v.trim() && !(v === '' && (a === '--base' || a === '--head'));
|
|
1459
|
+
if (gone || blank) bad(`promptwheel ${cmd}: ${a} needs ${VALUED_FLAGS[a]}`);
|
|
1460
|
+
if (a === '--repeat' && !/^[1-9]\d*$/.test(v)) bad(`promptwheel ${cmd}: --repeat needs ${VALUED_FLAGS[a]} (got "${v}")`);
|
|
1339
1461
|
}
|
|
1340
1462
|
}
|
|
1341
1463
|
|
|
1464
|
+
// read PR comments on stdin: a JSON array, or one comment per line as `gh api --paginate --jq '.[]'` prints every
|
|
1465
|
+
// page. Streamed: readFileSync(0) corrupted the heap on Node 24.22 reading a live paginated gh pipe.
|
|
1466
|
+
async function prAllow() {
|
|
1467
|
+
let t = ''; process.stdin.setEncoding('utf8');
|
|
1468
|
+
for await (const chunk of process.stdin) t += chunk;
|
|
1469
|
+
t = t.trim(); let c = [];
|
|
1470
|
+
try { c = [].concat(JSON.parse(t || '[]')); } catch { try { c = t.split('\n').filter(Boolean).map((l) => JSON.parse(l)); } catch { /* unreadable, no approval */ } }
|
|
1471
|
+
process.stdout.write(allowFromComments(c));
|
|
1472
|
+
}
|
|
1473
|
+
|
|
1342
1474
|
// write or refresh the architecture lock from the tree as it stands. A person runs this: the lock file is
|
|
1343
1475
|
// a mandate, so the refreshed file passes the gate only with --allow "<reason>".
|
|
1344
1476
|
async function lock() {
|
|
@@ -1353,9 +1485,9 @@ function dispatch() {
|
|
|
1353
1485
|
const cmd = (rest.includes('--help') || rest.includes('-h')) ? 'help' : first; // `promptwheel run --help` is a help request, not a gate run
|
|
1354
1486
|
if (cmd in COMMAND_FLAGS) checkFlags(cmd, rest);
|
|
1355
1487
|
if (cmd === 'run') run(rest);
|
|
1356
|
-
else if (cmd === 'pr-allow')
|
|
1488
|
+
else if (cmd === 'pr-allow') return prAllow();
|
|
1357
1489
|
else if (cmd === 'init') init(rest);
|
|
1358
|
-
else if (cmd === 'lock') lock(rest);
|
|
1490
|
+
else if (cmd === 'lock') return lock(rest);
|
|
1359
1491
|
else {
|
|
1360
1492
|
const help = !cmd || cmd === '--help' || cmd === '-h' || cmd === 'help';
|
|
1361
1493
|
if (!help) console.error(`unknown command: ${cmd}`);
|
|
@@ -1370,10 +1502,11 @@ function dispatch() {
|
|
|
1370
1502
|
' catches reward-hacking BY DEFAULT, re-proves each win from SOURCE edits alone (verdict GAMED,',
|
|
1371
1503
|
' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
|
|
1372
1504
|
' promptwheel run --working gate uncommitted changes (incl. newly added files)',
|
|
1505
|
+
' promptwheel pr-allow < comments.json print the newest `sawbuck-allow: <reason>` from a PR comment by someone with write access (the Action uses it)',
|
|
1373
1506
|
' promptwheel run --allow "<reason>" acknowledge an intentional test/config change (or a `Promptwheel-Allow: <reason>` commit trailer; `Sawbuck-Allow:` is an alias): recorded, not flagged GAMED',
|
|
1374
1507
|
'',
|
|
1375
1508
|
'Exit codes: 0 pass · 1 fail (a guarded metric regressed, or a mandate was violated) · 2 gamed · 3 inconclusive · 64 usage or config error (never a verdict).',
|
|
1376
|
-
'Integrate (library / conform / shell
|
|
1509
|
+
'Integrate (library / conform / shell): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
|
|
1377
1510
|
'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
|
|
1378
1511
|
' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
|
|
1379
1512
|
].join('\n'));
|
|
@@ -1384,8 +1517,8 @@ function dispatch() {
|
|
|
1384
1517
|
// run the CLI only when invoked directly (resolves symlinks so the npm bin still works)
|
|
1385
1518
|
let isMain = false;
|
|
1386
1519
|
try { isMain = !!process.argv[1] && realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); } catch { /* not main */ }
|
|
1387
|
-
function main() {
|
|
1388
|
-
try { dispatch(); }
|
|
1520
|
+
async function main() {
|
|
1521
|
+
try { await dispatch(); }
|
|
1389
1522
|
catch (e) { if (e && e.name === 'ConfigError') die(e.message); throw e; }
|
|
1390
1523
|
}
|
|
1391
1524
|
if (isMain) main();
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.8.
|
|
3
|
+
"version": "0.8.1",
|
|
4
4
|
"description": "A deterministic, zero-LLM check for your coding loop. A win must survive a re-run from the agent's source edits alone, so a test that went green by editing the test reads GAMED. Runs in CI or a Claude Code plugin.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -27,7 +27,7 @@ clean git worktrees, so a pack that is not in git is absent at both refs and the
|
|
|
27
27
|
count. A private registry mirror makes this baseline large, not failing.
|
|
28
28
|
- **unpinned_execution**: code run from a reference that can move. A GitHub Action not pinned to a full commit SHA (a local `./action` does not count), `npx`, `bunx` or `dlx` with no exact version, `npm install -g` with no exact version, and a remote script piped into a shell. Read from workflows, `action.yml`, package.json scripts, `scripts/*.sh` and Dockerfiles. In package.json scripts, a tool the manifest declares does not count: the lockfile already pins it.
|
|
29
29
|
- **install_policy_gaps**: 0, 1 or 2. Are install scripts disabled, and is a release-age cooldown set (`ignore-scripts=true` and `min-release-age` in `.npmrc`, or the pnpm or yarn equivalents)? The cooldown is enforced by the package manager, not by this pack: a registry lookup here would make the verdict depend on the clock.
|
|
30
|
-
- **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is
|
|
30
|
+
- **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is locked but not counted by default (`node .promptwheel/packs/supply-chain/lock.mjs drift . --list` shows it). To hold new routes too, add `--routes` to the end of the `lock_drift` metric's `cmd` in `promptwheel.config.json`, so it reads `node .promptwheel/packs/supply-chain/lock.mjs drift . --routes`. The flag belongs to that script: `promptwheel lock` takes no flags and always writes routes into the lock. The config is a frozen path here, so that edit passes with `promptwheel run --allow "<reason>"`. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
|
|
31
31
|
- **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%, dependency changes; use the CI path below for those.
|
|
32
32
|
|
|
33
33
|
**On a pull request.** Someone with write access comments `sawbuck-allow: <reason>` and re-runs the check. `sawbuck contract` marks everyday guards `report: true`, so they note a change without failing until you tighten them; this preset on its own holds them all.
|
|
@@ -57,10 +57,10 @@ Sometimes adding a dependency is exactly what was asked. Say so, and the gate re
|
|
|
57
57
|
promptwheel run --working --allow "needed for the CSV export (papaparse)"
|
|
58
58
|
```
|
|
59
59
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
written into `report.allowed
|
|
63
|
-
|
|
60
|
+
A commit trailer does not count here. The seven metrics carry `allowable: "flag"`, which means a
|
|
61
|
+
reason from the `--allow` flag clears their guarded regression for that one run, with the reason
|
|
62
|
+
written into `report.allowed`. Nothing is loosened in the config, and a rise without a reason still
|
|
63
|
+
fails. That is the point: every added package, script, or non-registry
|
|
64
64
|
source leaves an audit line a reviewer can read.
|
|
65
65
|
|
|
66
66
|
`gamingCheck` is `false` on all seven because these numbers legitimately live in config files
|
|
@@ -69,8 +69,8 @@ flag every honest dependency bump.
|
|
|
69
69
|
|
|
70
70
|
## The mandates the preset writes
|
|
71
71
|
|
|
72
|
-
A mandate is a path the agent must not touch in a change
|
|
73
|
-
and a mandate deleted or narrowed in the same change that violates it stays in force.
|
|
72
|
+
A mandate is a path the agent must not touch in a change. Touching one fails the gate whatever the
|
|
73
|
+
metrics say, and a mandate deleted or narrowed in the same change that violates it stays in force.
|
|
74
74
|
|
|
75
75
|
| mandate | paths | why |
|
|
76
76
|
| --- | --- | --- |
|
|
@@ -79,15 +79,20 @@ and a mandate deleted or narrowed in the same change that violates it stays in f
|
|
|
79
79
|
| build-images-frozen | `Dockerfile*`, `docker-compose*` | what runs at build and deploy time |
|
|
80
80
|
| publish-scripts-frozen | `scripts/release*`, `scripts/publish*` | the publish path is how a compromise spreads |
|
|
81
81
|
| codeowners-frozen | `CODEOWNERS` | review ownership is set by people |
|
|
82
|
+
| scanner-frozen | `.promptwheel/packs/**` | the scanners are not edited by the change they read |
|
|
82
83
|
| contract-frozen | `promptwheel.config.json`, `sawbuck.config.json` | the contract changes with a reason from a person (`--allow`) |
|
|
83
|
-
|
|
|
84
|
+
| lock-frozen | `promptwheel.lock.json` | the lock changes with a reason from a person (`--allow`) |
|
|
85
|
+
| hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks change with a reason from a person (`--allow`) |
|
|
84
86
|
|
|
85
87
|
The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is frozen too, and it is also an
|
|
86
88
|
`evaluatorPaths` entry. A change to it needs `--allow "<reason>"` from a person. Without that, a change that
|
|
87
89
|
only weakens a rule would pass, and the next change would be judged by the weakened contract.
|
|
88
90
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
+
`--allow` clears only three of these: `contract-frozen`, `lock-frozen` and `hooks-frozen` (so a team can commit
|
|
92
|
+
shared hooks to a repo that already has its rules). A commit trailer clears none. Every other frozen path stays
|
|
93
|
+
frozen, even with `--allow`. To change one, a person first retires its mandate in a change of its own
|
|
94
|
+
(it edits the contract, so it needs `--allow`), then makes the change, then puts the mandate back the
|
|
95
|
+
same way.
|
|
91
96
|
|
|
92
97
|
## What this is not
|
|
93
98
|
|
|
@@ -100,5 +105,4 @@ path, reviewed as such.
|
|
|
100
105
|
- **Not a scanner.** No matchers, no findings, no false positives to triage: a path, a count, or a
|
|
101
106
|
hash.
|
|
102
107
|
|
|
103
|
-
Pair it with the
|
|
104
|
-
the gate runs where the agent cannot switch it off.
|
|
108
|
+
Pair it with the GitHub Action so the gate runs where the agent cannot switch it off.
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
"$comment": "PromptWheel supply-chain guard-pack.
|
|
2
|
+
"$comment": "PromptWheel supply-chain guard-pack. Install it with `promptwheel init --preset supply-chain`: that copies the pack to .promptwheel/packs/supply-chain/, writes these metrics into promptwheel.config.json with their commands pointed at the copy, and writes the mandates. The commands in this file run packs/supply-chain/..., which exists only in the PromptWheel repository itself, so `extends` on this file (here or on the copy under .promptwheel/packs/) reads every metric as unmeasurable, and the gate fails, in any other repository. Seven counts the gate holds: a rise fails the gate unless a person gives a reason with `--allow \"<reason>\"` (a commit trailer does not count), which is recorded in report.allowed. gamingCheck is false because these numbers legitimately live in config files (package.json, lockfiles), which the source-only re-run reverts.",
|
|
3
3
|
"metrics": [
|
|
4
4
|
{
|
|
5
5
|
"$comment": "install-time script entries (preinstall, install, postinstall, prepare, prepublish, prepublishOnly, postpack) across every package.json: the step a supply-chain attack needs inside your repo",
|