promptwheel 0.4.6 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/INTEGRATE.md +213 -0
- package/README.md +3 -1
- package/bin/promptwheel.mjs +351 -16
- package/package.json +10 -3
- package/packs/security/README.md +4 -4
package/INTEGRATE.md
ADDED
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
# Integrating PromptWheel
|
|
2
|
+
|
|
3
|
+
PromptWheel is the reference implementation of the **Verified-Outcome Contract**: a
|
|
4
|
+
deterministic, zero-LLM check that proves a change moved a real metric and that the agent
|
|
5
|
+
earned the win in the source, not by editing the test, config, or grader. This page is the
|
|
6
|
+
integration surface. There are four ways in, from loosest coupling to tightest.
|
|
7
|
+
|
|
8
|
+
The protocol itself (exit codes, the verdict JSON, the integrity boundary) is specified in
|
|
9
|
+
[CONTRACT.md](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md). The sections
|
|
10
|
+
below say `Section N` where that document says `N`.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 1. Conform to the Contract (no code from here)
|
|
15
|
+
|
|
16
|
+
The strongest guarantee and the loosest coupling: emit or consume the two artifacts the
|
|
17
|
+
Contract defines, and any harness interoperates with any other, in any language, without
|
|
18
|
+
importing this package.
|
|
19
|
+
|
|
20
|
+
- **Section 1, exit codes.** `0` pass, `1` fail (a guarded metric regressed beyond noise),
|
|
21
|
+
`2` gamed (a win that did not survive a source-only re-run), `3` inconclusive (a guard
|
|
22
|
+
measured nothing, so a pass cannot be certified). An agent reads the code as a per-turn
|
|
23
|
+
reward it cannot inflate; a merge gate reads the same code as a wall it cannot self-clear.
|
|
24
|
+
- **Section 2, the verdict JSON.** The object below, reproducible from `(base, head, config)`
|
|
25
|
+
alone.
|
|
26
|
+
|
|
27
|
+
```json
|
|
28
|
+
{
|
|
29
|
+
"base": "a1b2c3d",
|
|
30
|
+
"head": "e4f5g6h",
|
|
31
|
+
"mode": "refs",
|
|
32
|
+
"repeat": 3,
|
|
33
|
+
"cohort": "ci",
|
|
34
|
+
"verdict": "gamed",
|
|
35
|
+
"subsystems": ["api", "auth"],
|
|
36
|
+
"metrics": [
|
|
37
|
+
{
|
|
38
|
+
"name": "tests_pass",
|
|
39
|
+
"direction": "pass",
|
|
40
|
+
"guard": true,
|
|
41
|
+
"before": 0,
|
|
42
|
+
"after": 1,
|
|
43
|
+
"delta": 1,
|
|
44
|
+
"status": "improved",
|
|
45
|
+
"ok": true,
|
|
46
|
+
"confidence": "high",
|
|
47
|
+
"noise": 0,
|
|
48
|
+
"gamed": true,
|
|
49
|
+
"sourceOnly": null,
|
|
50
|
+
"retained": 0.0,
|
|
51
|
+
"gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
|
|
52
|
+
}
|
|
53
|
+
]
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Fixed vocabularies (use exactly these):
|
|
58
|
+
|
|
59
|
+
- `verdict` is one of `pass | fail | gamed | inconclusive`
|
|
60
|
+
- `metric.status` is one of `improved | regressed | unchanged | inconclusive | unmeasurable`
|
|
61
|
+
- `direction` is one of `up | down | pass` (the direction that counts as better; `pass` is a 0/1 gate)
|
|
62
|
+
|
|
63
|
+
`noise` is the measured jitter band, a single number, across the top-level `repeat` runs; a
|
|
64
|
+
delta inside it is `inconclusive`, never a regression. `retained` is the fraction of a
|
|
65
|
+
metric's gain that survived the source-only re-run (Section 3). `confidence`
|
|
66
|
+
(`unverified | low | medium | high`) and `sourceOnly` are informational fields this
|
|
67
|
+
implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
|
|
68
|
+
truncated SHA-256 fingerprint of the gate definition at base and head and lists any guard the
|
|
69
|
+
change dropped to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
|
|
70
|
+
cannot pass by disabling the guard it is judged by.
|
|
71
|
+
|
|
72
|
+
Self-check your implementation against this schema (see Section 4 below):
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
your-tool --json | promptwheel conformance -
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## 2. Shell the CLI and parse `--json`
|
|
81
|
+
|
|
82
|
+
Language-agnostic, process-isolated, the common path for CI and for harnesses not written in
|
|
83
|
+
Node. Run the gate, read the exit code (Section 1), parse stdout (Section 2).
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
promptwheel run --base "$BASE" --head "$HEAD" --json
|
|
87
|
+
echo $? # 0 pass | 1 fail | 2 gamed | 3 inconclusive
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
promptwheel run --working --json # gate uncommitted changes (incl. newly added files)
|
|
92
|
+
promptwheel run --no-detect-gaming # the bare gate, without the Section 3 re-proof
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Gaming detection (Section 3) is **on by default**. Nothing touches your working tree: every
|
|
96
|
+
measurement runs in a throwaway git worktree.
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## 3. Import the library (in process)
|
|
101
|
+
|
|
102
|
+
Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
|
|
103
|
+
process and returns the Section 2 object. It never prints and never calls `process.exit`; it
|
|
104
|
+
throws a plain `Error` on a precondition failure.
|
|
105
|
+
|
|
106
|
+
```js
|
|
107
|
+
import { check, verdictExitCode, conformsToContract, VERDICTS, CONTRACT_VERSION } from 'promptwheel';
|
|
108
|
+
|
|
109
|
+
const report = check({ base: 'HEAD~1', head: 'HEAD' }); // discovers the repo from cwd
|
|
110
|
+
if (report.verdict === 'gamed') {
|
|
111
|
+
const m = report.metrics.find((x) => x.gamed);
|
|
112
|
+
throw new Error(`gamed: ${m.name} (${m.gamingReason})`);
|
|
113
|
+
}
|
|
114
|
+
process.exit(verdictExitCode(report.verdict)); // exit exactly as the CLI would
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
### `check(options) -> report`
|
|
118
|
+
|
|
119
|
+
Returns the Section 2 verdict object (identical to `run --json`). Reproducible from
|
|
120
|
+
`(base, head, config)` alone.
|
|
121
|
+
|
|
122
|
+
| option | type | default | meaning |
|
|
123
|
+
| --- | --- | --- | --- |
|
|
124
|
+
| `repo` | string | from `process.cwd()` | repo path |
|
|
125
|
+
| `base` | string | merge-base vs `origin/main`..., else `HEAD~1` | base ref |
|
|
126
|
+
| `head` | string | `HEAD` | head ref |
|
|
127
|
+
| `working` | boolean | `false` | gate uncommitted changes instead of `base..head` |
|
|
128
|
+
| `config` | object | load `promptwheel.config.json` from the repo | a resolved config `{ metrics: [...], ... }`, supplied in process |
|
|
129
|
+
| `repeat` | number | `config.repeat ?? 1` | measurement repeats for the noise band |
|
|
130
|
+
| `detectGaming` | boolean | `true` | run the Section 3 source-only re-proof |
|
|
131
|
+
| `cohort` | string | `CI ? 'ci' : 'local'` | cohort tag for the record |
|
|
132
|
+
| `label` | string | none | change-type label |
|
|
133
|
+
| `record` | boolean | `false` | append the run to `.promptwheel/outcomes.jsonl` |
|
|
134
|
+
|
|
135
|
+
Preconditions throw (they do not exit the host): not inside a git repository; no
|
|
136
|
+
`promptwheel.config.json` and no `config` option; a `config` with no non-empty `metrics`
|
|
137
|
+
array. Supplying `config` needs no file on disk:
|
|
138
|
+
|
|
139
|
+
```js
|
|
140
|
+
const report = check({
|
|
141
|
+
repo: '/path/to/repo', base: 'HEAD~1', head: 'HEAD',
|
|
142
|
+
config: { linkNodeModules: false, metrics: [
|
|
143
|
+
{ name: 'tests_pass', cmd: 'npm test', extract: 'exit', direction: 'pass', guard: true },
|
|
144
|
+
] },
|
|
145
|
+
});
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
### `verdictExitCode(verdict) -> 0 | 1 | 2 | 3`
|
|
149
|
+
|
|
150
|
+
The one Section 1 mapping the CLI uses too. Map verdicts this way and your process exits
|
|
151
|
+
exactly as `promptwheel run` does.
|
|
152
|
+
|
|
153
|
+
### `conformsToContract(report) -> { ok, errors }`
|
|
154
|
+
|
|
155
|
+
Structurally validate any verdict object against Section 2 (shapes, the fixed vocabularies,
|
|
156
|
+
and the honest-verdict rules: a `pass` cannot hide a failing guard or a gamed win, a `gamed`
|
|
157
|
+
verdict must point at a gamed metric). Zero dependency, so a third-party implementation can
|
|
158
|
+
self-check the artifact it emits. `errors` is a list of human-readable problems, empty when
|
|
159
|
+
`ok` is `true`.
|
|
160
|
+
|
|
161
|
+
### Other exports
|
|
162
|
+
|
|
163
|
+
`CONTRACT_VERSION` (the protocol revision this build implements), and the frozen vocab
|
|
164
|
+
arrays `VERDICTS`, `METRIC_STATUSES`, `DIRECTIONS`, so a consumer validates against the source
|
|
165
|
+
of truth instead of hardcoding the strings.
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## 4. Self-check with `promptwheel conformance`
|
|
170
|
+
|
|
171
|
+
Prove your own check, loop, or CI step speaks the Contract.
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
# validate a verdict JSON (from a file or stdin) against the Section 2 schema
|
|
175
|
+
promptwheel conformance report.json
|
|
176
|
+
cat report.json | promptwheel conformance -
|
|
177
|
+
|
|
178
|
+
# run a command, then validate BOTH its --json stdout (Section 2)
|
|
179
|
+
# AND that its exit code matches the verdict (Section 1)
|
|
180
|
+
promptwheel conformance --cmd "your-tool --json"
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Exit `0` if it conforms, `1` otherwise, listing every problem. This is how a third-party
|
|
184
|
+
implementation earns the right to say "Verified-Outcome Contract compatible."
|
|
185
|
+
|
|
186
|
+
---
|
|
187
|
+
|
|
188
|
+
## The memory half: @promptwheel/logbook
|
|
189
|
+
|
|
190
|
+
The check verifies what holds now; the **logbook** records what happened over the history.
|
|
191
|
+
It is a separate package with the same thin, model-blind, zero-dependency design, and it ships
|
|
192
|
+
three integration surfaces of its own:
|
|
193
|
+
|
|
194
|
+
- **CLI + JSON.** `logbook query --json`, `logbook check --diff --json`. Shell it like Section 2.
|
|
195
|
+
- **MCP.** `@promptwheel/logbook-mcp` (bin `logbook-mcp`) exposes the history to an agent over
|
|
196
|
+
the Model Context Protocol, so a Claude Code or other MCP client reads hotspots, reverts, and
|
|
197
|
+
suppression signals without shelling anything.
|
|
198
|
+
- **Import.** The package is importable (`@promptwheel/logbook`) and importing it never runs the
|
|
199
|
+
CLI; `queryEvents`, `analyze`, `auditHead`, and `hotspots` are exported for advanced in-process
|
|
200
|
+
use. The scan and cache entry points take explicit options, so the CLI and MCP are the
|
|
201
|
+
first-class surfaces for most integrations. The reference harness (Sawbuck) consumes the
|
|
202
|
+
logbook by shelling `logbook query --json`, not by importing it, which keeps the same process
|
|
203
|
+
boundary the check engine uses.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## The layer boundary (why this stays thin)
|
|
208
|
+
|
|
209
|
+
The engines are deterministic and model-blind: no LLM call, no network, no runtime
|
|
210
|
+
dependency, no build step. A harness that orchestrates models (Sawbuck is the reference one)
|
|
211
|
+
consumes these engines across a process boundary, by shelling the CLI or resolving the bin,
|
|
212
|
+
and never imports their internals. Keep that boundary and the check stays something a
|
|
213
|
+
skeptic can re-run and get the same verdict, which is the entire point.
|
package/README.md
CHANGED
|
@@ -4,13 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
**A deterministic check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel tells the difference: it re-proves every win from the source changes alone, with no LLM, so a win that only survived by editing the test reads `GAMED`. Un-gameable by construction, because the code being checked never gets to grade itself.
|
|
6
6
|
|
|
7
|
+

|
|
8
|
+
|
|
7
9
|
PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop, a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
|
|
8
10
|
|
|
9
11
|
It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"*, PromptWheel also asks *"did the agent **earn** it?"*
|
|
10
12
|
|
|
11
13
|
PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question, *did this turn earn its keep, and did the agent earn it honestly?*, so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
|
|
12
14
|
|
|
13
|
-
PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop.
|
|
15
|
+
PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop. Use it from CI, a Claude Code hook, or in process (`import { check } from 'promptwheel'`); the four integration paths (conform, shell, import, MCP) are in [INTEGRATE.md](INTEGRATE.md).
|
|
14
16
|
|
|
15
17
|
> Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate**, making the reward signal one you can't cheat.
|
|
16
18
|
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -11,6 +11,7 @@
|
|
|
11
11
|
// promptwheel run [--base R] [--head R] [--repeat N] [--json]
|
|
12
12
|
|
|
13
13
|
import { execSync, execFileSync } from 'node:child_process';
|
|
14
|
+
import { createHash } from 'node:crypto';
|
|
14
15
|
import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, appendFileSync, realpathSync, readdirSync, statSync, copyFileSync } from 'node:fs';
|
|
15
16
|
import { tmpdir } from 'node:os';
|
|
16
17
|
import { join, dirname, isAbsolute, relative } from 'node:path';
|
|
@@ -51,12 +52,12 @@ function resolveConfig(p, repo, label, seen) {
|
|
|
51
52
|
if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
|
|
52
53
|
const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
|
|
53
54
|
for (const m of base.metrics) byName.set(m.name, m);
|
|
54
|
-
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
|
|
55
|
+
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (base[k] !== undefined) scalars[k] = base[k];
|
|
55
56
|
}
|
|
56
57
|
for (const m of (cfg.metrics || [])) {
|
|
57
58
|
byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
|
|
58
59
|
}
|
|
59
|
-
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
|
|
60
|
+
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
|
|
60
61
|
return { ...scalars, metrics: [...byName.values()] };
|
|
61
62
|
} finally {
|
|
62
63
|
seen.delete(abs); // only ACTIVE ancestors are a cycle, a diamond (a base reached two ways) is fine
|
|
@@ -203,6 +204,17 @@ const NON_SOURCE = [
|
|
|
203
204
|
/\.(test|spec)\.[cm]?[jt]sx?$/i,
|
|
204
205
|
/(^|\/)tests?\.[cm]?[jt]sx?$/i, // a file literally named test.js / tests.ts
|
|
205
206
|
/(^|\/)test_[^/]*\.py$/i, /_test\.py$/i, /(^|\/)conftest\.py$/i,
|
|
207
|
+
// cross-language test-filename conventions, strong-signal tokens only (a dedicated
|
|
208
|
+
// _test / _spec / Test(s) suffix, or a file literally named test.<lang>), so production
|
|
209
|
+
// code like latest.c, Contest.java or speedtest.go is NOT swept into the evaluator slice.
|
|
210
|
+
// (Test DIRECTORIES are covered above for every language; this adds the by-filename
|
|
211
|
+
// conventions compiled langs use at repo root, a red-team finding: a weakened test.c /
|
|
212
|
+
// FooTest.java / handler_test.go was classified as source, so the cheat survived.)
|
|
213
|
+
/(^|\/)tests?\.(c|cc|cpp|cxx|go|rs|rb)$/i,
|
|
214
|
+
/(^|\/)test_[^/]*\.(c|cc|cpp|cxx|h|hpp|go|rs)$/i,
|
|
215
|
+
/_test\.(c|cc|cpp|cxx|h|hpp|go|rs|rb)$/i,
|
|
216
|
+
/_spec\.rb$/i,
|
|
217
|
+
/[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
|
|
206
218
|
/\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
|
|
207
219
|
/(^|\/)(eval|grader|score)[^/]*\.[cm]?[jt]s$/i,
|
|
208
220
|
// config files only, require a .config/.conf/.setup segment so PRODUCTION source that
|
|
@@ -222,17 +234,41 @@ const NON_SOURCE = [
|
|
|
222
234
|
];
|
|
223
235
|
const isNonSource = (p) => NON_SOURCE.some((re) => re.test(p));
|
|
224
236
|
|
|
225
|
-
//
|
|
226
|
-
|
|
237
|
+
// user-declared evaluator surface: extra paths that DEFINE the gate (a generic threshold
|
|
238
|
+
// file, a budget, a CI limit) and so must be reverted in the source-only re-run exactly
|
|
239
|
+
// like a test/grader. Opt-in per repo via `evaluatorPaths`, NOT guessed, because a bare
|
|
240
|
+
// name is a classifier risk; the user names the files only they know are the goalposts.
|
|
241
|
+
// Glob semantics: a bare name matches that basename at any depth; a pattern with "/" is a
|
|
242
|
+
// path glob anchored at repo root; "*" stops at "/", "**" crosses it, "?" is one char; a
|
|
243
|
+
// leading "/" anchors to the root explicitly.
|
|
244
|
+
function globToRe(glob) {
|
|
245
|
+
let g = glob; const anchored = g.startsWith('/'); if (anchored) g = g.slice(1);
|
|
246
|
+
const body = g.replace(/[.+^${}()|[\]\\]/g, '\\$&') // escape regex, keep * ? for globbing
|
|
247
|
+
.replace(/\*\*/g, '\x00').replace(/\*/g, '[^/]*').replace(/\?/g, '[^/]').replace(/\x00/g, '.*');
|
|
248
|
+
const pre = (anchored || glob.includes('/')) ? '^' : '(^|/)';
|
|
249
|
+
return new RegExp(`${pre}${body}$`);
|
|
250
|
+
}
|
|
251
|
+
function evaluatorMatcher(cfg) {
|
|
252
|
+
const pats = (cfg && Array.isArray(cfg.evaluatorPaths) ? cfg.evaluatorPaths : []).filter((s) => typeof s === 'string' && s.trim());
|
|
253
|
+
if (!pats.length) return () => false;
|
|
254
|
+
const res = pats.map(globToRe);
|
|
255
|
+
return (p) => res.some((re) => re.test(p));
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
// split the agent's changed files into production-source vs test/config/grader/golden.
|
|
259
|
+
// `isEval` is the repo's opt-in evaluator-path matcher, treated as non-source alongside
|
|
260
|
+
// the built-in classifier, so a cheat that only edits a declared gate file is caught.
|
|
261
|
+
function changedSourcePaths(repo, base, head, isEval = () => false) {
|
|
227
262
|
const all = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean);
|
|
228
|
-
|
|
263
|
+
const nonSrc = (p) => isNonSource(p) || isEval(p);
|
|
264
|
+
return { source: all.filter((p) => !nonSrc(p)), nonSource: all.filter(nonSrc) };
|
|
229
265
|
}
|
|
230
266
|
|
|
231
267
|
// coarse change-location fingerprint for the outcome record: top source dirs of the diff
|
|
232
|
-
function subsystemsOf(repo, base, head) {
|
|
268
|
+
function subsystemsOf(repo, base, head, cfg) {
|
|
233
269
|
try {
|
|
234
270
|
const counts = {};
|
|
235
|
-
for (const p of changedSourcePaths(repo, base, head).source) {
|
|
271
|
+
for (const p of changedSourcePaths(repo, base, head, evaluatorMatcher(cfg)).source) {
|
|
236
272
|
const d = p.includes('/') ? p.slice(0, p.indexOf('/')) : '(root)';
|
|
237
273
|
counts[d] = (counts[d] || 0) + 1;
|
|
238
274
|
}
|
|
@@ -242,7 +278,7 @@ function subsystemsOf(repo, base, head) {
|
|
|
242
278
|
|
|
243
279
|
// measure `metric` in a worktree at `base` with ONLY the source slice of base→head applied.
|
|
244
280
|
function measureSourceOnly(repo, base, head, metric, cfg, repeat) {
|
|
245
|
-
const { source } = changedSourcePaths(repo, base, head);
|
|
281
|
+
const { source } = changedSourcePaths(repo, base, head, evaluatorMatcher(cfg));
|
|
246
282
|
if (!source.length) return { samples: [], hadSourceChange: false }; // a "win" with zero source edits = goalposts moved
|
|
247
283
|
const wt = mkdtempSync(join(tmpdir(), 'promptwheel-src-'));
|
|
248
284
|
git(['worktree', 'add', '--quiet', '--detach', wt, base], repo);
|
|
@@ -360,10 +396,74 @@ function warnEditableInstall(repo) {
|
|
|
360
396
|
}
|
|
361
397
|
|
|
362
398
|
// the shared core: measure a change (base→head) and return the structured report
|
|
399
|
+
// ---- gate integrity: catch the agent weakening the gate it is judged by ----
|
|
400
|
+
// The config is read from HEAD, so a change can disable a guard (guard:false) or delete a
|
|
401
|
+
// guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
|
|
402
|
+
// guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
|
|
403
|
+
// that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
|
|
404
|
+
const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths'];
|
|
405
|
+
|
|
406
|
+
// Resolve the gate definition as of `ref` (metrics + scalars), following `extends` by reading
|
|
407
|
+
// every file from that ref via `git show`. Returns { metrics:Map, scalars } or null when there
|
|
408
|
+
// is no config at `ref` (a gate ADDED in this range is not a weakening). Never exits: any
|
|
409
|
+
// missing/unreadable file or parse error yields null (that branch simply contributes nothing).
|
|
410
|
+
function gateDefAtRef(repo, ref) {
|
|
411
|
+
// quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
|
|
412
|
+
const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
|
|
413
|
+
const seen = new Set();
|
|
414
|
+
const resolve = (p) => {
|
|
415
|
+
if (seen.has(p)) return null;
|
|
416
|
+
seen.add(p);
|
|
417
|
+
const raw = readAt(p);
|
|
418
|
+
if (raw == null) return null;
|
|
419
|
+
let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
|
|
420
|
+
const byName = new Map(); const scalars = {};
|
|
421
|
+
for (const ext of [].concat(cfg.extends || [])) {
|
|
422
|
+
if (isAbsolute(ext)) continue; // outside the repo, cannot git-show; unchanged anyway
|
|
423
|
+
const base = resolve(join(dirname(p), ext).split('\\').join('/'));
|
|
424
|
+
if (!base) continue;
|
|
425
|
+
for (const m of base.metrics.values()) byName.set(m.name, m);
|
|
426
|
+
for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
|
|
427
|
+
}
|
|
428
|
+
for (const m of (cfg.metrics || [])) if (m && m.name) byName.set(m.name, m);
|
|
429
|
+
for (const k of GATE_SCALARS) if (cfg[k] !== undefined) scalars[k] = cfg[k];
|
|
430
|
+
return { metrics: byName, scalars };
|
|
431
|
+
};
|
|
432
|
+
for (const p of ['promptwheel.config.json', 'outcome-gate.config.json']) { const r = resolve(p); if (r) return r; } // first hit only: no spurious probe of the alias
|
|
433
|
+
return null;
|
|
434
|
+
}
|
|
435
|
+
|
|
436
|
+
// an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
|
|
437
|
+
function gateFingerprint(metricsList, scalars) {
|
|
438
|
+
const norm = {
|
|
439
|
+
metrics: metricsList.map((m) => ({
|
|
440
|
+
name: m.name, guard: m.guard === true, direction: m.direction || 'up',
|
|
441
|
+
gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
|
|
442
|
+
})).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
|
|
443
|
+
gamingThreshold: scalars.gamingThreshold ?? null,
|
|
444
|
+
evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
|
|
445
|
+
};
|
|
446
|
+
return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
|
|
447
|
+
}
|
|
448
|
+
|
|
449
|
+
// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
|
|
450
|
+
function gateWeakenings(baseGate, headMetrics) {
|
|
451
|
+
if (!baseGate) return [];
|
|
452
|
+
const head = new Map(headMetrics.map((m) => [m.name, m]));
|
|
453
|
+
const weak = [];
|
|
454
|
+
for (const bm of baseGate.metrics.values()) {
|
|
455
|
+
if (bm.guard !== true) continue; // only a guard protects anything
|
|
456
|
+
const hm = head.get(bm.name);
|
|
457
|
+
if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
|
|
458
|
+
else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
|
|
459
|
+
}
|
|
460
|
+
return weak;
|
|
461
|
+
}
|
|
462
|
+
|
|
363
463
|
function gate(repo, opts) {
|
|
364
464
|
selfHeal(repo);
|
|
365
465
|
warnEditableInstall(repo);
|
|
366
|
-
const cfg = loadConfig(repo);
|
|
466
|
+
const cfg = opts.config ?? loadConfig(repo);
|
|
367
467
|
const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
|
|
368
468
|
|
|
369
469
|
let base, head;
|
|
@@ -396,8 +496,33 @@ function gate(repo, opts) {
|
|
|
396
496
|
m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
|
|
397
497
|
}
|
|
398
498
|
}
|
|
499
|
+
// gate integrity (Section 3): did HEAD weaken the gate it is judged by? Resolve the gate as
|
|
500
|
+
// of BASE; a guard HEAD dropped is re-measured under its base definition and only counts as
|
|
501
|
+
// tampering if it WOULD HAVE FAILED at head (a real regression hidden by dropping the guard).
|
|
502
|
+
let gateInfo = null;
|
|
503
|
+
if (opts.detectGaming) {
|
|
504
|
+
const baseGate = gateDefAtRef(repo, base);
|
|
505
|
+
const weak = gateWeakenings(baseGate, cfg.metrics);
|
|
506
|
+
const weakened = [];
|
|
507
|
+
if (weak.length) {
|
|
508
|
+
const defs = weak.map((w) => w.def);
|
|
509
|
+
const b = measureAt(repo, base, { ...cfg, metrics: defs }, repeat);
|
|
510
|
+
const a = measureAt(repo, head, { ...cfg, metrics: defs }, repeat);
|
|
511
|
+
for (const w of weak) {
|
|
512
|
+
const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
|
|
513
|
+
if (!ev.ok) weakened.push({ name: w.name, change: w.change, status: ev.status });
|
|
514
|
+
}
|
|
515
|
+
}
|
|
516
|
+
gateInfo = {
|
|
517
|
+
fingerprint: {
|
|
518
|
+
base: baseGate ? gateFingerprint([...baseGate.metrics.values()], baseGate.scalars) : null,
|
|
519
|
+
head: gateFingerprint(cfg.metrics, cfg),
|
|
520
|
+
},
|
|
521
|
+
...(weakened.length ? { weakened } : {}),
|
|
522
|
+
};
|
|
523
|
+
}
|
|
399
524
|
const failed = metrics.some((m) => m.guard && !m.ok);
|
|
400
|
-
const gamed = metrics.some((m) => m.gamed === true);
|
|
525
|
+
const gamed = metrics.some((m) => m.gamed === true) || !!(gateInfo && gateInfo.weakened);
|
|
401
526
|
// a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
|
|
402
527
|
// nothing, don't launder that into a green PASS. The honest verdict is "couldn't measure".
|
|
403
528
|
// BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
|
|
@@ -410,8 +535,9 @@ function gate(repo, opts) {
|
|
|
410
535
|
// label attributes a change-type, subsystems fingerprint WHERE the change landed.
|
|
411
536
|
cohort: opts.cohort ?? (process.env.CI ? 'ci' : 'local'),
|
|
412
537
|
...(opts.label ? { label: opts.label } : {}),
|
|
413
|
-
subsystems: subsystemsOf(repo, base, head),
|
|
538
|
+
subsystems: subsystemsOf(repo, base, head, cfg),
|
|
414
539
|
verdict, metrics,
|
|
540
|
+
...(gateInfo ? { gate: gateInfo } : {}),
|
|
415
541
|
};
|
|
416
542
|
if (!opts.noRecord && cfg.record !== false) recordOutcome(repo, report);
|
|
417
543
|
return report;
|
|
@@ -424,7 +550,7 @@ function run(argv) {
|
|
|
424
550
|
if (args.json) console.log(JSON.stringify(report, null, 2));
|
|
425
551
|
else if (args.markdown) console.log(renderMarkdown(report));
|
|
426
552
|
else printHuman(report);
|
|
427
|
-
process.exit(report.verdict
|
|
553
|
+
process.exit(verdictExitCode(report.verdict));
|
|
428
554
|
}
|
|
429
555
|
|
|
430
556
|
// persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
|
|
@@ -676,7 +802,7 @@ const PRESETS = {
|
|
|
676
802
|
{ name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
|
|
677
803
|
{ name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
678
804
|
{ name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
679
|
-
{ name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
805
|
+
{ name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|# ?nosec|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
680
806
|
{ name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
681
807
|
] },
|
|
682
808
|
'no-ai-tells': { desc: 'opt-in: fail the gate if the em-dash count rises (an AI writing tell); some projects use them on purpose, so it is not a default',
|
|
@@ -685,6 +811,21 @@ const PRESETS = {
|
|
|
685
811
|
] },
|
|
686
812
|
};
|
|
687
813
|
|
|
814
|
+
// security-antihack = the security-invariant findings gate + the SAME antihack tripwires
|
|
815
|
+
// (so an agent that suppresses a security rule, # noqa / # nosec / eslint-disable, or
|
|
816
|
+
// skips a security test is caught), PLUS a starter `evaluatorPaths` that protects common
|
|
817
|
+
// security-gate config files (a ruleset or a budget the agent could otherwise loosen to
|
|
818
|
+
// pass). The scanner pack installs into `.promptwheel/` (already evaluator-protected). It
|
|
819
|
+
// is the "guard the guards" preset: pairs with --detect-gaming so a win that only moved by
|
|
820
|
+
// loosening the security gate, not by fixing the code, is GAMED. Blind spot (by design,
|
|
821
|
+
// not this tool's job): a source-resident backdoor that behaves safely only under test;
|
|
822
|
+
// that needs held-out/behavioral analysis, not a source-only re-run.
|
|
823
|
+
PRESETS['security-antihack'] = {
|
|
824
|
+
desc: 'gate security-invariant findings + antihack tripwires + evaluatorPaths so an agent cannot loosen the security gate to pass it (pairs with --detect-gaming)',
|
|
825
|
+
evaluatorPaths: ['.semgrep.yml', '.semgrep.yaml', '.bandit', 'bandit.yaml', '.ruff.toml', 'ruff.toml', '.snyk', 'security-budget.json'],
|
|
826
|
+
metrics: [{ name: 'security_findings', cmd: 'node packs/security/scan.mjs .', extract: 'number', direction: 'down', guard: true }, ...PRESETS.antihack.metrics.slice(1)],
|
|
827
|
+
};
|
|
828
|
+
|
|
688
829
|
function detectTestCmd(repo) {
|
|
689
830
|
const has = (f) => existsSync(join(repo, f));
|
|
690
831
|
if (has('go.mod')) return 'go test ./...';
|
|
@@ -734,10 +875,11 @@ function init(argv) {
|
|
|
734
875
|
}
|
|
735
876
|
const pi = argv.indexOf('--preset');
|
|
736
877
|
const presetName = pi >= 0 ? argv[pi + 1] : null;
|
|
737
|
-
let metrics, note;
|
|
878
|
+
let metrics, note, evaluatorPaths;
|
|
738
879
|
if (presetName) {
|
|
739
880
|
const p = PRESETS[presetName];
|
|
740
881
|
if (!p) { console.error(`unknown preset "${presetName}", try: promptwheel init --list`); process.exit(2); }
|
|
882
|
+
evaluatorPaths = p.evaluatorPaths; // a preset may ship a starter evaluator allowlist (e.g. security-antihack)
|
|
741
883
|
metrics = p.metrics || [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }];
|
|
742
884
|
metrics = metrics.map((m) => m.cmd === '__TESTCMD__' ? { ...m, cmd: detectTestCmd(repo) } : m); // fill the preset's target
|
|
743
885
|
// pack-backed presets: the referenced scripts live in THIS package, not
|
|
@@ -766,7 +908,7 @@ function init(argv) {
|
|
|
766
908
|
}
|
|
767
909
|
const envCfg = presetName ? {} : detectEnv(repo);
|
|
768
910
|
if (envCfg.linkDirs) note += ` · linking ${envCfg.linkDirs.join(', ')}`;
|
|
769
|
-
writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, metrics }, null, 2) + '\n');
|
|
911
|
+
writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, ...(evaluatorPaths ? { evaluatorPaths } : {}), metrics }, null, 2) + '\n');
|
|
770
912
|
console.log(`✓ wrote promptwheel.config.json, ${note}`);
|
|
771
913
|
console.log('\n next: promptwheel run --working # gate your uncommitted changes');
|
|
772
914
|
console.log(' promptwheel init --list # other presets (llm-eval, bundle-size, …)');
|
|
@@ -812,6 +954,7 @@ function printHuman(r) {
|
|
|
812
954
|
if (m.inert) console.log(' ⚠ never passed at either ref, this guard is protecting nothing (check its command)');
|
|
813
955
|
if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
|
|
814
956
|
}
|
|
957
|
+
if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : 'disabled'} in this change and it would have ${w.status} at head (goalpost moved, not earned)`);
|
|
815
958
|
console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ', a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
|
|
816
959
|
if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name), `promptwheel guards` shows the effective set');
|
|
817
960
|
console.log('');
|
|
@@ -836,6 +979,7 @@ function renderMarkdown(r) {
|
|
|
836
979
|
rows,
|
|
837
980
|
'',
|
|
838
981
|
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
|
|
982
|
+
...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by dropping a guard, not by earning it.`] : []),
|
|
839
983
|
'',
|
|
840
984
|
'<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
|
|
841
985
|
].join('\n');
|
|
@@ -863,7 +1007,195 @@ function parseArgs(argv) {
|
|
|
863
1007
|
|
|
864
1008
|
// pure, side-effect-free helpers are exported for unit testing; the CLI below only
|
|
865
1009
|
// runs when this file is executed directly (not when imported by the test suite).
|
|
866
|
-
export { extract, evaluate, median, spread, renderMarkdown, isNonSource, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
|
|
1010
|
+
export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluatorMatcher, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
|
|
1011
|
+
|
|
1012
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
1013
|
+
// Public library API: the Verified-Outcome Contract, in process.
|
|
1014
|
+
// import { check } from 'promptwheel';
|
|
1015
|
+
// Importing this file never runs the CLI (see the isMain guard at the bottom).
|
|
1016
|
+
// Stable across 0.x minor versions. The schema check() returns and the exit codes
|
|
1017
|
+
// verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
|
|
1018
|
+
// shell, import, MCP) are in INTEGRATE.md.
|
|
1019
|
+
// ─────────────────────────────────────────────────────────────────────────────
|
|
1020
|
+
|
|
1021
|
+
// The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
|
|
1022
|
+
// when the verdict schema or the exit codes change; a consumer may assert on it.
|
|
1023
|
+
export const CONTRACT_VERSION = '1';
|
|
1024
|
+
|
|
1025
|
+
// Section 2 fixed vocabularies, exported so a consumer validates against the source of
|
|
1026
|
+
// truth instead of hardcoding the strings.
|
|
1027
|
+
export const VERDICTS = Object.freeze(['pass', 'fail', 'gamed', 'inconclusive']);
|
|
1028
|
+
export const METRIC_STATUSES = Object.freeze(['improved', 'regressed', 'unchanged', 'inconclusive', 'unmeasurable']);
|
|
1029
|
+
export const DIRECTIONS = Object.freeze(['up', 'down', 'pass']);
|
|
1030
|
+
|
|
1031
|
+
// Section 1: the one verdict -> process exit code mapping. The CLI uses it too, so a
|
|
1032
|
+
// consumer that maps verdicts this way exits exactly as `promptwheel run` does.
|
|
1033
|
+
export function verdictExitCode(verdict) {
|
|
1034
|
+
return verdict === 'pass' ? 0 : verdict === 'gamed' ? 2 : verdict === 'inconclusive' ? 3 : 1;
|
|
1035
|
+
}
|
|
1036
|
+
|
|
1037
|
+
// check(options) -> report
|
|
1038
|
+
// Run the gate in process and RETURN the Section 2 verdict object (identical to what
|
|
1039
|
+
// `run --json` prints). Measurement happens only in throwaway worktrees, never the
|
|
1040
|
+
// caller's working tree (constitution #4). This function never prints and never calls
|
|
1041
|
+
// process.exit; it throws a plain Error on a precondition failure (not a git repo, or a
|
|
1042
|
+
// missing / invalid config). Reproducible from (base, head, config) alone.
|
|
1043
|
+
//
|
|
1044
|
+
// repo string repo path (default: discovered from process.cwd())
|
|
1045
|
+
// base string base ref (default: merge-base vs origin/main..., else HEAD~1)
|
|
1046
|
+
// head string head ref (default: HEAD)
|
|
1047
|
+
// working boolean gate uncommitted changes instead of base..head
|
|
1048
|
+
// config object a resolved config { metrics:[...], ... }; omit to load
|
|
1049
|
+
// promptwheel.config.json from the repo
|
|
1050
|
+
// repeat number measurement repeats for the noise band (default: config.repeat ?? 1)
|
|
1051
|
+
// detectGaming boolean run the Section 3 source-only re-proof (default: true)
|
|
1052
|
+
// cohort string cohort tag for the record (default: CI ? 'ci' : 'local')
|
|
1053
|
+
// label string change-type label
|
|
1054
|
+
// record boolean append the run to .promptwheel/outcomes.jsonl (default: false)
|
|
1055
|
+
export function check(options = {}) {
|
|
1056
|
+
let repo;
|
|
1057
|
+
try {
|
|
1058
|
+
repo = options.repo ? realpathSync(options.repo)
|
|
1059
|
+
: git(['rev-parse', '--show-toplevel'], process.cwd());
|
|
1060
|
+
} catch { throw new Error(`promptwheel check: not inside a git repository (${options.repo || process.cwd()})`); }
|
|
1061
|
+
|
|
1062
|
+
if (options.config !== undefined) {
|
|
1063
|
+
const c = options.config;
|
|
1064
|
+
if (!c || typeof c !== 'object' || !Array.isArray(c.metrics) || c.metrics.length === 0)
|
|
1065
|
+
throw new Error('promptwheel check: options.config.metrics must be a non-empty array');
|
|
1066
|
+
} else if (!['promptwheel.config.json', 'outcome-gate.config.json'].some((f) => existsSync(join(repo, f)))) {
|
|
1067
|
+
throw new Error(`promptwheel check: no promptwheel.config.json in ${repo} (pass options.config to supply one in process)`);
|
|
1068
|
+
}
|
|
1069
|
+
|
|
1070
|
+
return gate(repo, {
|
|
1071
|
+
base: options.base,
|
|
1072
|
+
head: options.head,
|
|
1073
|
+
working: options.working,
|
|
1074
|
+
config: options.config,
|
|
1075
|
+
repeat: options.repeat,
|
|
1076
|
+
detectGaming: options.detectGaming ?? true,
|
|
1077
|
+
cohort: options.cohort,
|
|
1078
|
+
label: options.label,
|
|
1079
|
+
noRecord: options.record !== true,
|
|
1080
|
+
});
|
|
1081
|
+
}
|
|
1082
|
+
|
|
1083
|
+
// conformsToContract(report) -> { ok, errors }
|
|
1084
|
+
// Structurally validate a verdict object against Section 2 (shapes + fixed vocabularies)
|
|
1085
|
+
// plus the two honest-verdict rules that hold for any conforming implementation. Zero
|
|
1086
|
+
// dependency, so a third party can self-check the artifact it emits against the reference
|
|
1087
|
+
// schema. It checks well-formedness and consistency, not a deep re-derivation of the math.
|
|
1088
|
+
export function conformsToContract(report) {
|
|
1089
|
+
const errors = [];
|
|
1090
|
+
const E = (m) => errors.push(m);
|
|
1091
|
+
const isInt = (n) => Number.isInteger(n);
|
|
1092
|
+
const isNum = (n) => typeof n === 'number' && Number.isFinite(n);
|
|
1093
|
+
const numOrNull = (n) => n === null || isNum(n);
|
|
1094
|
+
|
|
1095
|
+
if (!report || typeof report !== 'object' || Array.isArray(report)) return { ok: false, errors: ['report must be an object'] };
|
|
1096
|
+
if (!VERDICTS.includes(report.verdict)) E(`verdict must be one of ${VERDICTS.join('|')} (got ${JSON.stringify(report.verdict)})`);
|
|
1097
|
+
if (typeof report.base !== 'string' || !report.base) E('base must be a non-empty string');
|
|
1098
|
+
if (typeof report.head !== 'string' || !report.head) E('head must be a non-empty string');
|
|
1099
|
+
if (report.mode !== undefined && report.mode !== 'refs' && report.mode !== 'working') E("mode must be 'refs' or 'working'");
|
|
1100
|
+
if (report.repeat !== undefined && !(isInt(report.repeat) && report.repeat >= 1)) E('repeat must be an integer >= 1');
|
|
1101
|
+
if (report.cohort !== undefined && typeof report.cohort !== 'string') E('cohort must be a string');
|
|
1102
|
+
if (report.subsystems !== undefined && !(Array.isArray(report.subsystems) && report.subsystems.every((s) => typeof s === 'string')))
|
|
1103
|
+
E('subsystems must be an array of strings');
|
|
1104
|
+
if (!Array.isArray(report.metrics)) { E('metrics must be an array'); return { ok: false, errors }; }
|
|
1105
|
+
|
|
1106
|
+
report.metrics.forEach((m, i) => {
|
|
1107
|
+
const at = `metrics[${i}]`;
|
|
1108
|
+
if (!m || typeof m !== 'object') { E(`${at} must be an object`); return; }
|
|
1109
|
+
if (typeof m.name !== 'string' || !m.name) E(`${at}.name must be a non-empty string`);
|
|
1110
|
+
if (!DIRECTIONS.includes(m.direction)) E(`${at}.direction must be one of ${DIRECTIONS.join('|')}`);
|
|
1111
|
+
if (typeof m.guard !== 'boolean') E(`${at}.guard must be a boolean`);
|
|
1112
|
+
if (!METRIC_STATUSES.includes(m.status)) E(`${at}.status must be one of ${METRIC_STATUSES.join('|')}`);
|
|
1113
|
+
if (typeof m.ok !== 'boolean') E(`${at}.ok must be a boolean`);
|
|
1114
|
+
if (m.before !== undefined && !numOrNull(m.before)) E(`${at}.before must be a number or null`);
|
|
1115
|
+
if (m.after !== undefined && !numOrNull(m.after)) E(`${at}.after must be a number or null`);
|
|
1116
|
+
if (m.delta !== undefined && !numOrNull(m.delta)) E(`${at}.delta must be a number or null`);
|
|
1117
|
+
if (m.noise !== undefined && !(isNum(m.noise) && m.noise >= 0)) E(`${at}.noise must be a number >= 0 (the measured jitter band; repeat count is the top-level report.repeat)`);
|
|
1118
|
+
if (m.confidence !== undefined && typeof m.confidence !== 'string') E(`${at}.confidence must be a string`);
|
|
1119
|
+
if (m.gamed !== undefined && typeof m.gamed !== 'boolean') E(`${at}.gamed must be a boolean`);
|
|
1120
|
+
if (m.sourceOnly !== undefined && !numOrNull(m.sourceOnly)) E(`${at}.sourceOnly must be a number or null`);
|
|
1121
|
+
if (m.retained !== undefined && !numOrNull(m.retained)) E(`${at}.retained must be a number or null`);
|
|
1122
|
+
if (m.gamingReason !== undefined && typeof m.gamingReason !== 'string') E(`${at}.gamingReason must be a string`);
|
|
1123
|
+
});
|
|
1124
|
+
|
|
1125
|
+
// the optional gate-integrity record (report.gate): a config fingerprint and, when the
|
|
1126
|
+
// agent weakened the gate it is judged by, the list of dropped guards.
|
|
1127
|
+
if (report.gate !== undefined) {
|
|
1128
|
+
if (!report.gate || typeof report.gate !== 'object' || Array.isArray(report.gate)) E('gate must be an object');
|
|
1129
|
+
else {
|
|
1130
|
+
if (report.gate.fingerprint !== undefined && (typeof report.gate.fingerprint !== 'object' || report.gate.fingerprint === null)) E('gate.fingerprint must be an object { base, head }');
|
|
1131
|
+
if (report.gate.weakened !== undefined && !Array.isArray(report.gate.weakened)) E('gate.weakened must be an array');
|
|
1132
|
+
}
|
|
1133
|
+
}
|
|
1134
|
+
|
|
1135
|
+
// Honest-verdict rules that hold regardless of how a verdict is derived:
|
|
1136
|
+
// a clean pass cannot hide a failing guard, a gamed win, or a weakened gate; and a gamed
|
|
1137
|
+
// verdict must point at a gamed metric or a weakened gate. (Fail-vs-gamed precedence is
|
|
1138
|
+
// left to the implementation.)
|
|
1139
|
+
if (VERDICTS.includes(report.verdict)) {
|
|
1140
|
+
const gateWeakened = !!(report.gate && Array.isArray(report.gate.weakened) && report.gate.weakened.length);
|
|
1141
|
+
if (report.verdict === 'pass' && report.metrics.some((m) => m.guard === true && m.ok === false))
|
|
1142
|
+
E("verdict 'pass' but a guarded metric regressed (guard:true, ok:false)");
|
|
1143
|
+
if (report.verdict === 'pass' && report.metrics.some((m) => m.gamed === true))
|
|
1144
|
+
E("verdict 'pass' but a metric is gamed:true");
|
|
1145
|
+
if (report.verdict === 'pass' && gateWeakened)
|
|
1146
|
+
E("verdict 'pass' but report.gate.weakened is non-empty (the gate was weakened)");
|
|
1147
|
+
if (report.verdict === 'gamed' && !report.metrics.some((m) => m.gamed === true) && !gateWeakened)
|
|
1148
|
+
E("verdict 'gamed' but no metric is gamed:true and report.gate records no weakening");
|
|
1149
|
+
}
|
|
1150
|
+
|
|
1151
|
+
return { ok: errors.length === 0, errors };
|
|
1152
|
+
}
|
|
1153
|
+
|
|
1154
|
+
// `promptwheel conformance [file.json | -] [--cmd "<command>"]`
|
|
1155
|
+
// Self-check a Contract implementation. With a file or stdin, validate that verdict JSON
|
|
1156
|
+
// against the Section 2 schema. With --cmd, run the command, then validate its --json
|
|
1157
|
+
// stdout (Section 2) AND that its exit code matches the verdict per Section 1. Exit 0 if
|
|
1158
|
+
// it conforms, 1 otherwise, listing every problem.
|
|
1159
|
+
function conformance(argv) {
|
|
1160
|
+
let cmd = null, src = null;
|
|
1161
|
+
for (let i = 0; i < argv.length; i++) {
|
|
1162
|
+
if (argv[i] === '--cmd') cmd = argv[++i];
|
|
1163
|
+
else if (argv[i] === '-' || !argv[i].startsWith('-')) src = argv[i];
|
|
1164
|
+
}
|
|
1165
|
+
|
|
1166
|
+
let text, exitCode = null;
|
|
1167
|
+
if (cmd) {
|
|
1168
|
+
try { text = execFileSync('sh', ['-c', cmd], { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }); exitCode = 0; }
|
|
1169
|
+
catch (e) {
|
|
1170
|
+
text = (e.stdout || '').toString();
|
|
1171
|
+
exitCode = typeof e.status === 'number' ? e.status : null;
|
|
1172
|
+
if (!text) { console.error(`conformance: command produced no stdout (exit ${exitCode}): ${e.message}`); process.exit(1); }
|
|
1173
|
+
}
|
|
1174
|
+
} else {
|
|
1175
|
+
const file = src && src !== '-' ? src : 0; // fd 0 = stdin
|
|
1176
|
+
try { text = readFileSync(file, 'utf8'); }
|
|
1177
|
+
catch (e) { console.error(`conformance: cannot read ${!src || src === '-' ? 'stdin' : src}: ${e.message}`); process.exit(1); }
|
|
1178
|
+
}
|
|
1179
|
+
|
|
1180
|
+
let report;
|
|
1181
|
+
try { report = JSON.parse(text); }
|
|
1182
|
+
catch (e) { console.error(`conformance: not valid JSON: ${e.message}`); process.exit(1); }
|
|
1183
|
+
|
|
1184
|
+
const { errors } = conformsToContract(report);
|
|
1185
|
+
const problems = [...errors];
|
|
1186
|
+
if (cmd && exitCode !== null && VERDICTS.includes(report.verdict)) {
|
|
1187
|
+
const want = verdictExitCode(report.verdict);
|
|
1188
|
+
if (exitCode !== want) problems.push(`exit code ${exitCode} does not match verdict '${report.verdict}' (Section 1 requires ${want})`);
|
|
1189
|
+
}
|
|
1190
|
+
|
|
1191
|
+
if (problems.length === 0) {
|
|
1192
|
+
console.log(`ok, conforms to the Verified-Outcome Contract v${CONTRACT_VERSION} (verdict: ${report.verdict}${cmd ? `, exit ${exitCode}` : ''})`);
|
|
1193
|
+
process.exit(0);
|
|
1194
|
+
}
|
|
1195
|
+
console.error(`does NOT conform to the Contract (${problems.length} problem${problems.length > 1 ? 's' : ''}):`);
|
|
1196
|
+
for (const p of problems) console.error(` - ${p}`);
|
|
1197
|
+
process.exit(1);
|
|
1198
|
+
}
|
|
867
1199
|
|
|
868
1200
|
function main() {
|
|
869
1201
|
const [cmd, ...rest] = process.argv.slice(2);
|
|
@@ -875,6 +1207,7 @@ function main() {
|
|
|
875
1207
|
else if (cmd === 'backfill') backfill(rest);
|
|
876
1208
|
else if (cmd === 'init') init(rest);
|
|
877
1209
|
else if (cmd === 'guards') guards(rest);
|
|
1210
|
+
else if (cmd === 'conformance') conformance(rest);
|
|
878
1211
|
else {
|
|
879
1212
|
console.log([
|
|
880
1213
|
'PromptWheel, catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
|
|
@@ -891,7 +1224,9 @@ function main() {
|
|
|
891
1224
|
' promptwheel suggest [--json] UCB work-discovery: where the next attempt should go (experimental)',
|
|
892
1225
|
' promptwheel backfill [-n N | --since <ref>] seed the ledger from git history (cohort-tagged; commit types become labels)',
|
|
893
1226
|
' promptwheel guards show the effective guardrails (incl. inherited) + flag record',
|
|
1227
|
+
' promptwheel conformance [file|-] [--cmd C] check a verdict JSON (or a command) against the Contract (exit 0 = conforms)',
|
|
894
1228
|
'',
|
|
1229
|
+
'Integrate (library / conform / shell / MCP): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
|
|
895
1230
|
'Loop it: while promptwheel improve --attempt "$AGENT"; do :; done # stops on plateau/regression',
|
|
896
1231
|
'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
|
|
897
1232
|
' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
|
package/package.json
CHANGED
|
@@ -1,15 +1,22 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook.",
|
|
3
|
+
"version": "0.5.0",
|
|
4
|
+
"description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook, or import { check } in process.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"promptwheel": "bin/promptwheel.mjs",
|
|
8
8
|
"pw": "bin/promptwheel.mjs"
|
|
9
9
|
},
|
|
10
|
+
"exports": {
|
|
11
|
+
".": "./bin/promptwheel.mjs",
|
|
12
|
+
"./bin/promptwheel.mjs": "./bin/promptwheel.mjs",
|
|
13
|
+
"./package.json": "./package.json",
|
|
14
|
+
"./packs/*": "./packs/*"
|
|
15
|
+
},
|
|
10
16
|
"files": [
|
|
11
17
|
"bin",
|
|
12
|
-
"packs"
|
|
18
|
+
"packs",
|
|
19
|
+
"INTEGRATE.md"
|
|
13
20
|
],
|
|
14
21
|
"engines": {
|
|
15
22
|
"node": ">=18"
|
package/packs/security/README.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
Gate your AI coding loop on **security invariants**, not just tests. When an agent
|
|
4
4
|
writes backend code, "tests pass" says nothing about whether it shipped a CORS
|
|
5
5
|
wildcard, a client-controlled payment amount, or a hardcoded secret. This pack
|
|
6
|
-
adds a `security_findings` metric that must not climb
|
|
6
|
+
adds a `security_findings` metric that must not climb - and because it's a
|
|
7
7
|
PromptWheel metric, the win has to survive `--detect-gaming`: an agent can't
|
|
8
8
|
lower the count by editing the pattern file or a config, only by fixing the code.
|
|
9
9
|
|
|
@@ -24,9 +24,9 @@ promptwheel run --working # fails if security_findings increased
|
|
|
24
24
|
A finding-count that drops because you fixed the code is a `PASS`; a "fix" that
|
|
25
25
|
only survives with test/config/pattern edits reverted is `GAMED`.
|
|
26
26
|
|
|
27
|
-
## What it checks (v0.1
|
|
27
|
+
## What it checks (v0.1 - 8 high-precision invariants)
|
|
28
28
|
|
|
29
|
-
Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier
|
|
29
|
+
Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier -
|
|
30
30
|
the regex-portable subset, chosen for near-zero false positives:
|
|
31
31
|
|
|
32
32
|
| Invariant | Severity |
|
|
@@ -46,7 +46,7 @@ Each pattern ships with its own false-positive guards (the `exclude` regexes).
|
|
|
46
46
|
|
|
47
47
|
This is the **regex-portable** slice. The other ~75 recommended invariants
|
|
48
48
|
(auth-enforcement, authz-revocation, tenant-isolation, dataflow-tainted sinks)
|
|
49
|
-
need the AST + dataflow engine and are the follow-up
|
|
49
|
+
need the AST + dataflow engine and are the follow-up - see the decant note in
|
|
50
50
|
`explorations/`. The point of v0.1 is the architecture: security invariants are
|
|
51
51
|
now a first-class, un-gameable PromptWheel metric. Zero dependencies; the scanner
|
|
52
52
|
is one file (`scan.mjs`).
|