promptwheel 0.4.6 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/INTEGRATE.md ADDED
@@ -0,0 +1,213 @@
1
+ # Integrating PromptWheel
2
+
3
+ PromptWheel is the reference implementation of the **Verified-Outcome Contract**: a
4
+ deterministic, zero-LLM check that proves a change moved a real metric and that the agent
5
+ earned the win in the source, not by editing the test, config, or grader. This page is the
6
+ integration surface. There are four ways in, from loosest coupling to tightest.
7
+
8
+ The protocol itself (exit codes, the verdict JSON, the integrity boundary) is specified in
9
+ [CONTRACT.md](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md). The sections
10
+ below say `Section N` where that document says `N`.
11
+
12
+ ---
13
+
14
+ ## 1. Conform to the Contract (no code from here)
15
+
16
+ The strongest guarantee and the loosest coupling: emit or consume the two artifacts the
17
+ Contract defines, and any harness interoperates with any other, in any language, without
18
+ importing this package.
19
+
20
+ - **Section 1, exit codes.** `0` pass, `1` fail (a guarded metric regressed beyond noise),
21
+ `2` gamed (a win that did not survive a source-only re-run), `3` inconclusive (a guard
22
+ measured nothing, so a pass cannot be certified). An agent reads the code as a per-turn
23
+ reward it cannot inflate; a merge gate reads the same code as a wall it cannot self-clear.
24
+ - **Section 2, the verdict JSON.** The object below, reproducible from `(base, head, config)`
25
+ alone.
26
+
27
+ ```json
28
+ {
29
+ "base": "a1b2c3d",
30
+ "head": "e4f5g6h",
31
+ "mode": "refs",
32
+ "repeat": 3,
33
+ "cohort": "ci",
34
+ "verdict": "gamed",
35
+ "subsystems": ["api", "auth"],
36
+ "metrics": [
37
+ {
38
+ "name": "tests_pass",
39
+ "direction": "pass",
40
+ "guard": true,
41
+ "before": 0,
42
+ "after": 1,
43
+ "delta": 1,
44
+ "status": "improved",
45
+ "ok": true,
46
+ "confidence": "high",
47
+ "noise": 0,
48
+ "gamed": true,
49
+ "sourceOnly": null,
50
+ "retained": 0.0,
51
+ "gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
52
+ }
53
+ ]
54
+ }
55
+ ```
56
+
57
+ Fixed vocabularies (use exactly these):
58
+
59
+ - `verdict` is one of `pass | fail | gamed | inconclusive`
60
+ - `metric.status` is one of `improved | regressed | unchanged | inconclusive | unmeasurable`
61
+ - `direction` is one of `up | down | pass` (the direction that counts as better; `pass` is a 0/1 gate)
62
+
63
+ `noise` is the measured jitter band, a single number, across the top-level `repeat` runs; a
64
+ delta inside it is `inconclusive`, never a regression. `retained` is the fraction of a
65
+ metric's gain that survived the source-only re-run (Section 3). `confidence`
66
+ (`unverified | low | medium | high`) and `sourceOnly` are informational fields this
67
+ implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
68
+ truncated SHA-256 fingerprint of the gate definition at base and head and lists any guard the
69
+ change dropped to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
70
+ cannot pass by disabling the guard it is judged by.
71
+
72
+ Self-check your implementation against this schema (see Section 4 below):
73
+
74
+ ```bash
75
+ your-tool --json | promptwheel conformance -
76
+ ```
77
+
78
+ ---
79
+
80
+ ## 2. Shell the CLI and parse `--json`
81
+
82
+ Language-agnostic, process-isolated, the common path for CI and for harnesses not written in
83
+ Node. Run the gate, read the exit code (Section 1), parse stdout (Section 2).
84
+
85
+ ```bash
86
+ promptwheel run --base "$BASE" --head "$HEAD" --json
87
+ echo $? # 0 pass | 1 fail | 2 gamed | 3 inconclusive
88
+ ```
89
+
90
+ ```bash
91
+ promptwheel run --working --json # gate uncommitted changes (incl. newly added files)
92
+ promptwheel run --no-detect-gaming # the bare gate, without the Section 3 re-proof
93
+ ```
94
+
95
+ Gaming detection (Section 3) is **on by default**. Nothing touches your working tree: every
96
+ measurement runs in a throwaway git worktree.
97
+
98
+ ---
99
+
100
+ ## 3. Import the library (in process)
101
+
102
+ Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
103
+ process and returns the Section 2 object. It never prints and never calls `process.exit`; it
104
+ throws a plain `Error` on a precondition failure.
105
+
106
+ ```js
107
+ import { check, verdictExitCode, conformsToContract, VERDICTS, CONTRACT_VERSION } from 'promptwheel';
108
+
109
+ const report = check({ base: 'HEAD~1', head: 'HEAD' }); // discovers the repo from cwd
110
+ if (report.verdict === 'gamed') {
111
+ const m = report.metrics.find((x) => x.gamed);
112
+ throw new Error(`gamed: ${m.name} (${m.gamingReason})`);
113
+ }
114
+ process.exit(verdictExitCode(report.verdict)); // exit exactly as the CLI would
115
+ ```
116
+
117
+ ### `check(options) -> report`
118
+
119
+ Returns the Section 2 verdict object (identical to `run --json`). Reproducible from
120
+ `(base, head, config)` alone.
121
+
122
+ | option | type | default | meaning |
123
+ | --- | --- | --- | --- |
124
+ | `repo` | string | from `process.cwd()` | repo path |
125
+ | `base` | string | merge-base vs `origin/main`..., else `HEAD~1` | base ref |
126
+ | `head` | string | `HEAD` | head ref |
127
+ | `working` | boolean | `false` | gate uncommitted changes instead of `base..head` |
128
+ | `config` | object | load `promptwheel.config.json` from the repo | a resolved config `{ metrics: [...], ... }`, supplied in process |
129
+ | `repeat` | number | `config.repeat ?? 1` | measurement repeats for the noise band |
130
+ | `detectGaming` | boolean | `true` | run the Section 3 source-only re-proof |
131
+ | `cohort` | string | `CI ? 'ci' : 'local'` | cohort tag for the record |
132
+ | `label` | string | none | change-type label |
133
+ | `record` | boolean | `false` | append the run to `.promptwheel/outcomes.jsonl` |
134
+
135
+ Preconditions throw (they do not exit the host): not inside a git repository; no
136
+ `promptwheel.config.json` and no `config` option; a `config` with no non-empty `metrics`
137
+ array. Supplying `config` needs no file on disk:
138
+
139
+ ```js
140
+ const report = check({
141
+ repo: '/path/to/repo', base: 'HEAD~1', head: 'HEAD',
142
+ config: { linkNodeModules: false, metrics: [
143
+ { name: 'tests_pass', cmd: 'npm test', extract: 'exit', direction: 'pass', guard: true },
144
+ ] },
145
+ });
146
+ ```
147
+
148
+ ### `verdictExitCode(verdict) -> 0 | 1 | 2 | 3`
149
+
150
+ The one Section 1 mapping the CLI uses too. Map verdicts this way and your process exits
151
+ exactly as `promptwheel run` does.
152
+
153
+ ### `conformsToContract(report) -> { ok, errors }`
154
+
155
+ Structurally validate any verdict object against Section 2 (shapes, the fixed vocabularies,
156
+ and the honest-verdict rules: a `pass` cannot hide a failing guard or a gamed win, a `gamed`
157
+ verdict must point at a gamed metric). Zero dependency, so a third-party implementation can
158
+ self-check the artifact it emits. `errors` is a list of human-readable problems, empty when
159
+ `ok` is `true`.
160
+
161
+ ### Other exports
162
+
163
+ `CONTRACT_VERSION` (the protocol revision this build implements), and the frozen vocab
164
+ arrays `VERDICTS`, `METRIC_STATUSES`, `DIRECTIONS`, so a consumer validates against the source
165
+ of truth instead of hardcoding the strings.
166
+
167
+ ---
168
+
169
+ ## 4. Self-check with `promptwheel conformance`
170
+
171
+ Prove your own check, loop, or CI step speaks the Contract.
172
+
173
+ ```bash
174
+ # validate a verdict JSON (from a file or stdin) against the Section 2 schema
175
+ promptwheel conformance report.json
176
+ cat report.json | promptwheel conformance -
177
+
178
+ # run a command, then validate BOTH its --json stdout (Section 2)
179
+ # AND that its exit code matches the verdict (Section 1)
180
+ promptwheel conformance --cmd "your-tool --json"
181
+ ```
182
+
183
+ Exit `0` if it conforms, `1` otherwise, listing every problem. This is how a third-party
184
+ implementation earns the right to say "Verified-Outcome Contract compatible."
185
+
186
+ ---
187
+
188
+ ## The memory half: @promptwheel/logbook
189
+
190
+ The check verifies what holds now; the **logbook** records what happened over the history.
191
+ It is a separate package with the same thin, model-blind, zero-dependency design, and it ships
192
+ three integration surfaces of its own:
193
+
194
+ - **CLI + JSON.** `logbook query --json`, `logbook check --diff --json`. Shell it like Section 2.
195
+ - **MCP.** `@promptwheel/logbook-mcp` (bin `logbook-mcp`) exposes the history to an agent over
196
+ the Model Context Protocol, so a Claude Code or other MCP client reads hotspots, reverts, and
197
+ suppression signals without shelling anything.
198
+ - **Import.** The package is importable (`@promptwheel/logbook`) and importing it never runs the
199
+ CLI; `queryEvents`, `analyze`, `auditHead`, and `hotspots` are exported for advanced in-process
200
+ use. The scan and cache entry points take explicit options, so the CLI and MCP are the
201
+ first-class surfaces for most integrations. The reference harness (Sawbuck) consumes the
202
+ logbook by shelling `logbook query --json`, not by importing it, which keeps the same process
203
+ boundary the check engine uses.
204
+
205
+ ---
206
+
207
+ ## The layer boundary (why this stays thin)
208
+
209
+ The engines are deterministic and model-blind: no LLM call, no network, no runtime
210
+ dependency, no build step. A harness that orchestrates models (Sawbuck is the reference one)
211
+ consumes these engines across a process boundary, by shelling the CLI or resolving the bin,
212
+ and never imports their internals. Keep that boundary and the check stays something a
213
+ skeptic can re-run and get the same verdict, which is the entire point.
package/README.md CHANGED
@@ -4,13 +4,15 @@
4
4
 
5
5
  **A deterministic check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel tells the difference: it re-proves every win from the source changes alone, with no LLM, so a win that only survived by editing the test reads `GAMED`. Un-gameable by construction, because the code being checked never gets to grade itself.
6
6
 
7
+ ![PromptWheel catching an agent game the metric, then passing an honest fix](demo/promptwheel.gif)
8
+
7
9
  PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop, a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
8
10
 
9
11
  It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"*, PromptWheel also asks *"did the agent **earn** it?"*
10
12
 
11
13
  PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question, *did this turn earn its keep, and did the agent earn it honestly?*, so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
12
14
 
13
- PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop.
15
+ PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop. Use it from CI, a Claude Code hook, or in process (`import { check } from 'promptwheel'`); the four integration paths (conform, shell, import, MCP) are in [INTEGRATE.md](INTEGRATE.md).
14
16
 
15
17
  > Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate**, making the reward signal one you can't cheat.
16
18
 
@@ -11,6 +11,7 @@
11
11
  // promptwheel run [--base R] [--head R] [--repeat N] [--json]
12
12
 
13
13
  import { execSync, execFileSync } from 'node:child_process';
14
+ import { createHash } from 'node:crypto';
14
15
  import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, appendFileSync, realpathSync, readdirSync, statSync, copyFileSync } from 'node:fs';
15
16
  import { tmpdir } from 'node:os';
16
17
  import { join, dirname, isAbsolute, relative } from 'node:path';
@@ -51,12 +52,12 @@ function resolveConfig(p, repo, label, seen) {
51
52
  if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
52
53
  const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
53
54
  for (const m of base.metrics) byName.set(m.name, m);
54
- for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
55
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (base[k] !== undefined) scalars[k] = base[k];
55
56
  }
56
57
  for (const m of (cfg.metrics || [])) {
57
58
  byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
58
59
  }
59
- for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
60
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
60
61
  return { ...scalars, metrics: [...byName.values()] };
61
62
  } finally {
62
63
  seen.delete(abs); // only ACTIVE ancestors are a cycle, a diamond (a base reached two ways) is fine
@@ -203,6 +204,17 @@ const NON_SOURCE = [
203
204
  /\.(test|spec)\.[cm]?[jt]sx?$/i,
204
205
  /(^|\/)tests?\.[cm]?[jt]sx?$/i, // a file literally named test.js / tests.ts
205
206
  /(^|\/)test_[^/]*\.py$/i, /_test\.py$/i, /(^|\/)conftest\.py$/i,
207
+ // cross-language test-filename conventions, strong-signal tokens only (a dedicated
208
+ // _test / _spec / Test(s) suffix, or a file literally named test.<lang>), so production
209
+ // code like latest.c, Contest.java or speedtest.go is NOT swept into the evaluator slice.
210
+ // (Test DIRECTORIES are covered above for every language; this adds the by-filename
211
+ // conventions compiled langs use at repo root, a red-team finding: a weakened test.c /
212
+ // FooTest.java / handler_test.go was classified as source, so the cheat survived.)
213
+ /(^|\/)tests?\.(c|cc|cpp|cxx|go|rs|rb)$/i,
214
+ /(^|\/)test_[^/]*\.(c|cc|cpp|cxx|h|hpp|go|rs)$/i,
215
+ /_test\.(c|cc|cpp|cxx|h|hpp|go|rs|rb)$/i,
216
+ /_spec\.rb$/i,
217
+ /[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
206
218
  /\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
207
219
  /(^|\/)(eval|grader|score)[^/]*\.[cm]?[jt]s$/i,
208
220
  // config files only, require a .config/.conf/.setup segment so PRODUCTION source that
@@ -222,17 +234,41 @@ const NON_SOURCE = [
222
234
  ];
223
235
  const isNonSource = (p) => NON_SOURCE.some((re) => re.test(p));
224
236
 
225
- // split the agent's changed files into production-source vs test/config/grader/golden
226
- function changedSourcePaths(repo, base, head) {
237
+ // user-declared evaluator surface: extra paths that DEFINE the gate (a generic threshold
238
+ // file, a budget, a CI limit) and so must be reverted in the source-only re-run exactly
239
+ // like a test/grader. Opt-in per repo via `evaluatorPaths`, NOT guessed, because a bare
240
+ // name is a classifier risk; the user names the files only they know are the goalposts.
241
+ // Glob semantics: a bare name matches that basename at any depth; a pattern with "/" is a
242
+ // path glob anchored at repo root; "*" stops at "/", "**" crosses it, "?" is one char; a
243
+ // leading "/" anchors to the root explicitly.
244
+ function globToRe(glob) {
245
+ let g = glob; const anchored = g.startsWith('/'); if (anchored) g = g.slice(1);
246
+ const body = g.replace(/[.+^${}()|[\]\\]/g, '\\$&') // escape regex, keep * ? for globbing
247
+ .replace(/\*\*/g, '\x00').replace(/\*/g, '[^/]*').replace(/\?/g, '[^/]').replace(/\x00/g, '.*');
248
+ const pre = (anchored || glob.includes('/')) ? '^' : '(^|/)';
249
+ return new RegExp(`${pre}${body}$`);
250
+ }
251
+ function evaluatorMatcher(cfg) {
252
+ const pats = (cfg && Array.isArray(cfg.evaluatorPaths) ? cfg.evaluatorPaths : []).filter((s) => typeof s === 'string' && s.trim());
253
+ if (!pats.length) return () => false;
254
+ const res = pats.map(globToRe);
255
+ return (p) => res.some((re) => re.test(p));
256
+ }
257
+
258
+ // split the agent's changed files into production-source vs test/config/grader/golden.
259
+ // `isEval` is the repo's opt-in evaluator-path matcher, treated as non-source alongside
260
+ // the built-in classifier, so a cheat that only edits a declared gate file is caught.
261
+ function changedSourcePaths(repo, base, head, isEval = () => false) {
227
262
  const all = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean);
228
- return { source: all.filter((p) => !isNonSource(p)), nonSource: all.filter(isNonSource) };
263
+ const nonSrc = (p) => isNonSource(p) || isEval(p);
264
+ return { source: all.filter((p) => !nonSrc(p)), nonSource: all.filter(nonSrc) };
229
265
  }
230
266
 
231
267
  // coarse change-location fingerprint for the outcome record: top source dirs of the diff
232
- function subsystemsOf(repo, base, head) {
268
+ function subsystemsOf(repo, base, head, cfg) {
233
269
  try {
234
270
  const counts = {};
235
- for (const p of changedSourcePaths(repo, base, head).source) {
271
+ for (const p of changedSourcePaths(repo, base, head, evaluatorMatcher(cfg)).source) {
236
272
  const d = p.includes('/') ? p.slice(0, p.indexOf('/')) : '(root)';
237
273
  counts[d] = (counts[d] || 0) + 1;
238
274
  }
@@ -242,7 +278,7 @@ function subsystemsOf(repo, base, head) {
242
278
 
243
279
  // measure `metric` in a worktree at `base` with ONLY the source slice of base→head applied.
244
280
  function measureSourceOnly(repo, base, head, metric, cfg, repeat) {
245
- const { source } = changedSourcePaths(repo, base, head);
281
+ const { source } = changedSourcePaths(repo, base, head, evaluatorMatcher(cfg));
246
282
  if (!source.length) return { samples: [], hadSourceChange: false }; // a "win" with zero source edits = goalposts moved
247
283
  const wt = mkdtempSync(join(tmpdir(), 'promptwheel-src-'));
248
284
  git(['worktree', 'add', '--quiet', '--detach', wt, base], repo);
@@ -360,10 +396,74 @@ function warnEditableInstall(repo) {
360
396
  }
361
397
 
362
398
  // the shared core: measure a change (base→head) and return the structured report
399
+ // ---- gate integrity: catch the agent weakening the gate it is judged by ----
400
+ // The config is read from HEAD, so a change can disable a guard (guard:false) or delete a
401
+ // guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
402
+ // guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
403
+ // that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
404
+ const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths'];
405
+
406
+ // Resolve the gate definition as of `ref` (metrics + scalars), following `extends` by reading
407
+ // every file from that ref via `git show`. Returns { metrics:Map, scalars } or null when there
408
+ // is no config at `ref` (a gate ADDED in this range is not a weakening). Never exits: any
409
+ // missing/unreadable file or parse error yields null (that branch simply contributes nothing).
410
+ function gateDefAtRef(repo, ref) {
411
+ // quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
412
+ const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
413
+ const seen = new Set();
414
+ const resolve = (p) => {
415
+ if (seen.has(p)) return null;
416
+ seen.add(p);
417
+ const raw = readAt(p);
418
+ if (raw == null) return null;
419
+ let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
420
+ const byName = new Map(); const scalars = {};
421
+ for (const ext of [].concat(cfg.extends || [])) {
422
+ if (isAbsolute(ext)) continue; // outside the repo, cannot git-show; unchanged anyway
423
+ const base = resolve(join(dirname(p), ext).split('\\').join('/'));
424
+ if (!base) continue;
425
+ for (const m of base.metrics.values()) byName.set(m.name, m);
426
+ for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
427
+ }
428
+ for (const m of (cfg.metrics || [])) if (m && m.name) byName.set(m.name, m);
429
+ for (const k of GATE_SCALARS) if (cfg[k] !== undefined) scalars[k] = cfg[k];
430
+ return { metrics: byName, scalars };
431
+ };
432
+ for (const p of ['promptwheel.config.json', 'outcome-gate.config.json']) { const r = resolve(p); if (r) return r; } // first hit only: no spurious probe of the alias
433
+ return null;
434
+ }
435
+
436
+ // an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
437
+ function gateFingerprint(metricsList, scalars) {
438
+ const norm = {
439
+ metrics: metricsList.map((m) => ({
440
+ name: m.name, guard: m.guard === true, direction: m.direction || 'up',
441
+ gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
442
+ })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
443
+ gamingThreshold: scalars.gamingThreshold ?? null,
444
+ evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
445
+ };
446
+ return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
447
+ }
448
+
449
+ // guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
450
+ function gateWeakenings(baseGate, headMetrics) {
451
+ if (!baseGate) return [];
452
+ const head = new Map(headMetrics.map((m) => [m.name, m]));
453
+ const weak = [];
454
+ for (const bm of baseGate.metrics.values()) {
455
+ if (bm.guard !== true) continue; // only a guard protects anything
456
+ const hm = head.get(bm.name);
457
+ if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
458
+ else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
459
+ }
460
+ return weak;
461
+ }
462
+
363
463
  function gate(repo, opts) {
364
464
  selfHeal(repo);
365
465
  warnEditableInstall(repo);
366
- const cfg = loadConfig(repo);
466
+ const cfg = opts.config ?? loadConfig(repo);
367
467
  const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
368
468
 
369
469
  let base, head;
@@ -396,8 +496,33 @@ function gate(repo, opts) {
396
496
  m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
397
497
  }
398
498
  }
499
+ // gate integrity (Section 3): did HEAD weaken the gate it is judged by? Resolve the gate as
500
+ // of BASE; a guard HEAD dropped is re-measured under its base definition and only counts as
501
+ // tampering if it WOULD HAVE FAILED at head (a real regression hidden by dropping the guard).
502
+ let gateInfo = null;
503
+ if (opts.detectGaming) {
504
+ const baseGate = gateDefAtRef(repo, base);
505
+ const weak = gateWeakenings(baseGate, cfg.metrics);
506
+ const weakened = [];
507
+ if (weak.length) {
508
+ const defs = weak.map((w) => w.def);
509
+ const b = measureAt(repo, base, { ...cfg, metrics: defs }, repeat);
510
+ const a = measureAt(repo, head, { ...cfg, metrics: defs }, repeat);
511
+ for (const w of weak) {
512
+ const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
513
+ if (!ev.ok) weakened.push({ name: w.name, change: w.change, status: ev.status });
514
+ }
515
+ }
516
+ gateInfo = {
517
+ fingerprint: {
518
+ base: baseGate ? gateFingerprint([...baseGate.metrics.values()], baseGate.scalars) : null,
519
+ head: gateFingerprint(cfg.metrics, cfg),
520
+ },
521
+ ...(weakened.length ? { weakened } : {}),
522
+ };
523
+ }
399
524
  const failed = metrics.some((m) => m.guard && !m.ok);
400
- const gamed = metrics.some((m) => m.gamed === true);
525
+ const gamed = metrics.some((m) => m.gamed === true) || !!(gateInfo && gateInfo.weakened);
401
526
  // a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
402
527
  // nothing, don't launder that into a green PASS. The honest verdict is "couldn't measure".
403
528
  // BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
@@ -410,8 +535,9 @@ function gate(repo, opts) {
410
535
  // label attributes a change-type, subsystems fingerprint WHERE the change landed.
411
536
  cohort: opts.cohort ?? (process.env.CI ? 'ci' : 'local'),
412
537
  ...(opts.label ? { label: opts.label } : {}),
413
- subsystems: subsystemsOf(repo, base, head),
538
+ subsystems: subsystemsOf(repo, base, head, cfg),
414
539
  verdict, metrics,
540
+ ...(gateInfo ? { gate: gateInfo } : {}),
415
541
  };
416
542
  if (!opts.noRecord && cfg.record !== false) recordOutcome(repo, report);
417
543
  return report;
@@ -424,7 +550,7 @@ function run(argv) {
424
550
  if (args.json) console.log(JSON.stringify(report, null, 2));
425
551
  else if (args.markdown) console.log(renderMarkdown(report));
426
552
  else printHuman(report);
427
- process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : report.verdict === 'inconclusive' ? 3 : 1);
553
+ process.exit(verdictExitCode(report.verdict));
428
554
  }
429
555
 
430
556
  // persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
@@ -676,7 +802,7 @@ const PRESETS = {
676
802
  { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
677
803
  { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
678
804
  { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
679
- { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
805
+ { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|# ?nosec|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
680
806
  { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
681
807
  ] },
682
808
  'no-ai-tells': { desc: 'opt-in: fail the gate if the em-dash count rises (an AI writing tell); some projects use them on purpose, so it is not a default',
@@ -685,6 +811,21 @@ const PRESETS = {
685
811
  ] },
686
812
  };
687
813
 
814
+ // security-antihack = the security-invariant findings gate + the SAME antihack tripwires
815
+ // (so an agent that suppresses a security rule, # noqa / # nosec / eslint-disable, or
816
+ // skips a security test is caught), PLUS a starter `evaluatorPaths` that protects common
817
+ // security-gate config files (a ruleset or a budget the agent could otherwise loosen to
818
+ // pass). The scanner pack installs into `.promptwheel/` (already evaluator-protected). It
819
+ // is the "guard the guards" preset: pairs with --detect-gaming so a win that only moved by
820
+ // loosening the security gate, not by fixing the code, is GAMED. Blind spot (by design,
821
+ // not this tool's job): a source-resident backdoor that behaves safely only under test;
822
+ // that needs held-out/behavioral analysis, not a source-only re-run.
823
+ PRESETS['security-antihack'] = {
824
+ desc: 'gate security-invariant findings + antihack tripwires + evaluatorPaths so an agent cannot loosen the security gate to pass it (pairs with --detect-gaming)',
825
+ evaluatorPaths: ['.semgrep.yml', '.semgrep.yaml', '.bandit', 'bandit.yaml', '.ruff.toml', 'ruff.toml', '.snyk', 'security-budget.json'],
826
+ metrics: [{ name: 'security_findings', cmd: 'node packs/security/scan.mjs .', extract: 'number', direction: 'down', guard: true }, ...PRESETS.antihack.metrics.slice(1)],
827
+ };
828
+
688
829
  function detectTestCmd(repo) {
689
830
  const has = (f) => existsSync(join(repo, f));
690
831
  if (has('go.mod')) return 'go test ./...';
@@ -734,10 +875,11 @@ function init(argv) {
734
875
  }
735
876
  const pi = argv.indexOf('--preset');
736
877
  const presetName = pi >= 0 ? argv[pi + 1] : null;
737
- let metrics, note;
878
+ let metrics, note, evaluatorPaths;
738
879
  if (presetName) {
739
880
  const p = PRESETS[presetName];
740
881
  if (!p) { console.error(`unknown preset "${presetName}", try: promptwheel init --list`); process.exit(2); }
882
+ evaluatorPaths = p.evaluatorPaths; // a preset may ship a starter evaluator allowlist (e.g. security-antihack)
741
883
  metrics = p.metrics || [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }];
742
884
  metrics = metrics.map((m) => m.cmd === '__TESTCMD__' ? { ...m, cmd: detectTestCmd(repo) } : m); // fill the preset's target
743
885
  // pack-backed presets: the referenced scripts live in THIS package, not
@@ -766,7 +908,7 @@ function init(argv) {
766
908
  }
767
909
  const envCfg = presetName ? {} : detectEnv(repo);
768
910
  if (envCfg.linkDirs) note += ` · linking ${envCfg.linkDirs.join(', ')}`;
769
- writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, metrics }, null, 2) + '\n');
911
+ writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, ...(evaluatorPaths ? { evaluatorPaths } : {}), metrics }, null, 2) + '\n');
770
912
  console.log(`✓ wrote promptwheel.config.json, ${note}`);
771
913
  console.log('\n next: promptwheel run --working # gate your uncommitted changes');
772
914
  console.log(' promptwheel init --list # other presets (llm-eval, bundle-size, …)');
@@ -812,6 +954,7 @@ function printHuman(r) {
812
954
  if (m.inert) console.log(' ⚠ never passed at either ref, this guard is protecting nothing (check its command)');
813
955
  if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
814
956
  }
957
+ if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(` 🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : 'disabled'} in this change and it would have ${w.status} at head (goalpost moved, not earned)`);
815
958
  console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ', a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
816
959
  if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name), `promptwheel guards` shows the effective set');
817
960
  console.log('');
@@ -836,6 +979,7 @@ function renderMarkdown(r) {
836
979
  rows,
837
980
  '',
838
981
  r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
982
+ ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by dropping a guard, not by earning it.`] : []),
839
983
  '',
840
984
  '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
841
985
  ].join('\n');
@@ -863,7 +1007,195 @@ function parseArgs(argv) {
863
1007
 
864
1008
  // pure, side-effect-free helpers are exported for unit testing; the CLI below only
865
1009
  // runs when this file is executed directly (not when imported by the test suite).
866
- export { extract, evaluate, median, spread, renderMarkdown, isNonSource, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
1010
+ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluatorMatcher, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
1011
+
1012
+ // ─────────────────────────────────────────────────────────────────────────────
1013
+ // Public library API: the Verified-Outcome Contract, in process.
1014
+ // import { check } from 'promptwheel';
1015
+ // Importing this file never runs the CLI (see the isMain guard at the bottom).
1016
+ // Stable across 0.x minor versions. The schema check() returns and the exit codes
1017
+ // verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
1018
+ // shell, import, MCP) are in INTEGRATE.md.
1019
+ // ─────────────────────────────────────────────────────────────────────────────
1020
+
1021
+ // The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
1022
+ // when the verdict schema or the exit codes change; a consumer may assert on it.
1023
+ export const CONTRACT_VERSION = '1';
1024
+
1025
+ // Section 2 fixed vocabularies, exported so a consumer validates against the source of
1026
+ // truth instead of hardcoding the strings.
1027
+ export const VERDICTS = Object.freeze(['pass', 'fail', 'gamed', 'inconclusive']);
1028
+ export const METRIC_STATUSES = Object.freeze(['improved', 'regressed', 'unchanged', 'inconclusive', 'unmeasurable']);
1029
+ export const DIRECTIONS = Object.freeze(['up', 'down', 'pass']);
1030
+
1031
+ // Section 1: the one verdict -> process exit code mapping. The CLI uses it too, so a
1032
+ // consumer that maps verdicts this way exits exactly as `promptwheel run` does.
1033
+ export function verdictExitCode(verdict) {
1034
+ return verdict === 'pass' ? 0 : verdict === 'gamed' ? 2 : verdict === 'inconclusive' ? 3 : 1;
1035
+ }
1036
+
1037
+ // check(options) -> report
1038
+ // Run the gate in process and RETURN the Section 2 verdict object (identical to what
1039
+ // `run --json` prints). Measurement happens only in throwaway worktrees, never the
1040
+ // caller's working tree (constitution #4). This function never prints and never calls
1041
+ // process.exit; it throws a plain Error on a precondition failure (not a git repo, or a
1042
+ // missing / invalid config). Reproducible from (base, head, config) alone.
1043
+ //
1044
+ // repo string repo path (default: discovered from process.cwd())
1045
+ // base string base ref (default: merge-base vs origin/main..., else HEAD~1)
1046
+ // head string head ref (default: HEAD)
1047
+ // working boolean gate uncommitted changes instead of base..head
1048
+ // config object a resolved config { metrics:[...], ... }; omit to load
1049
+ // promptwheel.config.json from the repo
1050
+ // repeat number measurement repeats for the noise band (default: config.repeat ?? 1)
1051
+ // detectGaming boolean run the Section 3 source-only re-proof (default: true)
1052
+ // cohort string cohort tag for the record (default: CI ? 'ci' : 'local')
1053
+ // label string change-type label
1054
+ // record boolean append the run to .promptwheel/outcomes.jsonl (default: false)
1055
+ export function check(options = {}) {
1056
+ let repo;
1057
+ try {
1058
+ repo = options.repo ? realpathSync(options.repo)
1059
+ : git(['rev-parse', '--show-toplevel'], process.cwd());
1060
+ } catch { throw new Error(`promptwheel check: not inside a git repository (${options.repo || process.cwd()})`); }
1061
+
1062
+ if (options.config !== undefined) {
1063
+ const c = options.config;
1064
+ if (!c || typeof c !== 'object' || !Array.isArray(c.metrics) || c.metrics.length === 0)
1065
+ throw new Error('promptwheel check: options.config.metrics must be a non-empty array');
1066
+ } else if (!['promptwheel.config.json', 'outcome-gate.config.json'].some((f) => existsSync(join(repo, f)))) {
1067
+ throw new Error(`promptwheel check: no promptwheel.config.json in ${repo} (pass options.config to supply one in process)`);
1068
+ }
1069
+
1070
+ return gate(repo, {
1071
+ base: options.base,
1072
+ head: options.head,
1073
+ working: options.working,
1074
+ config: options.config,
1075
+ repeat: options.repeat,
1076
+ detectGaming: options.detectGaming ?? true,
1077
+ cohort: options.cohort,
1078
+ label: options.label,
1079
+ noRecord: options.record !== true,
1080
+ });
1081
+ }
1082
+
1083
+ // conformsToContract(report) -> { ok, errors }
1084
+ // Structurally validate a verdict object against Section 2 (shapes + fixed vocabularies)
1085
+ // plus the two honest-verdict rules that hold for any conforming implementation. Zero
1086
+ // dependency, so a third party can self-check the artifact it emits against the reference
1087
+ // schema. It checks well-formedness and consistency, not a deep re-derivation of the math.
1088
+ export function conformsToContract(report) {
1089
+ const errors = [];
1090
+ const E = (m) => errors.push(m);
1091
+ const isInt = (n) => Number.isInteger(n);
1092
+ const isNum = (n) => typeof n === 'number' && Number.isFinite(n);
1093
+ const numOrNull = (n) => n === null || isNum(n);
1094
+
1095
+ if (!report || typeof report !== 'object' || Array.isArray(report)) return { ok: false, errors: ['report must be an object'] };
1096
+ if (!VERDICTS.includes(report.verdict)) E(`verdict must be one of ${VERDICTS.join('|')} (got ${JSON.stringify(report.verdict)})`);
1097
+ if (typeof report.base !== 'string' || !report.base) E('base must be a non-empty string');
1098
+ if (typeof report.head !== 'string' || !report.head) E('head must be a non-empty string');
1099
+ if (report.mode !== undefined && report.mode !== 'refs' && report.mode !== 'working') E("mode must be 'refs' or 'working'");
1100
+ if (report.repeat !== undefined && !(isInt(report.repeat) && report.repeat >= 1)) E('repeat must be an integer >= 1');
1101
+ if (report.cohort !== undefined && typeof report.cohort !== 'string') E('cohort must be a string');
1102
+ if (report.subsystems !== undefined && !(Array.isArray(report.subsystems) && report.subsystems.every((s) => typeof s === 'string')))
1103
+ E('subsystems must be an array of strings');
1104
+ if (!Array.isArray(report.metrics)) { E('metrics must be an array'); return { ok: false, errors }; }
1105
+
1106
+ report.metrics.forEach((m, i) => {
1107
+ const at = `metrics[${i}]`;
1108
+ if (!m || typeof m !== 'object') { E(`${at} must be an object`); return; }
1109
+ if (typeof m.name !== 'string' || !m.name) E(`${at}.name must be a non-empty string`);
1110
+ if (!DIRECTIONS.includes(m.direction)) E(`${at}.direction must be one of ${DIRECTIONS.join('|')}`);
1111
+ if (typeof m.guard !== 'boolean') E(`${at}.guard must be a boolean`);
1112
+ if (!METRIC_STATUSES.includes(m.status)) E(`${at}.status must be one of ${METRIC_STATUSES.join('|')}`);
1113
+ if (typeof m.ok !== 'boolean') E(`${at}.ok must be a boolean`);
1114
+ if (m.before !== undefined && !numOrNull(m.before)) E(`${at}.before must be a number or null`);
1115
+ if (m.after !== undefined && !numOrNull(m.after)) E(`${at}.after must be a number or null`);
1116
+ if (m.delta !== undefined && !numOrNull(m.delta)) E(`${at}.delta must be a number or null`);
1117
+ if (m.noise !== undefined && !(isNum(m.noise) && m.noise >= 0)) E(`${at}.noise must be a number >= 0 (the measured jitter band; repeat count is the top-level report.repeat)`);
1118
+ if (m.confidence !== undefined && typeof m.confidence !== 'string') E(`${at}.confidence must be a string`);
1119
+ if (m.gamed !== undefined && typeof m.gamed !== 'boolean') E(`${at}.gamed must be a boolean`);
1120
+ if (m.sourceOnly !== undefined && !numOrNull(m.sourceOnly)) E(`${at}.sourceOnly must be a number or null`);
1121
+ if (m.retained !== undefined && !numOrNull(m.retained)) E(`${at}.retained must be a number or null`);
1122
+ if (m.gamingReason !== undefined && typeof m.gamingReason !== 'string') E(`${at}.gamingReason must be a string`);
1123
+ });
1124
+
1125
+ // the optional gate-integrity record (report.gate): a config fingerprint and, when the
1126
+ // agent weakened the gate it is judged by, the list of dropped guards.
1127
+ if (report.gate !== undefined) {
1128
+ if (!report.gate || typeof report.gate !== 'object' || Array.isArray(report.gate)) E('gate must be an object');
1129
+ else {
1130
+ if (report.gate.fingerprint !== undefined && (typeof report.gate.fingerprint !== 'object' || report.gate.fingerprint === null)) E('gate.fingerprint must be an object { base, head }');
1131
+ if (report.gate.weakened !== undefined && !Array.isArray(report.gate.weakened)) E('gate.weakened must be an array');
1132
+ }
1133
+ }
1134
+
1135
+ // Honest-verdict rules that hold regardless of how a verdict is derived:
1136
+ // a clean pass cannot hide a failing guard, a gamed win, or a weakened gate; and a gamed
1137
+ // verdict must point at a gamed metric or a weakened gate. (Fail-vs-gamed precedence is
1138
+ // left to the implementation.)
1139
+ if (VERDICTS.includes(report.verdict)) {
1140
+ const gateWeakened = !!(report.gate && Array.isArray(report.gate.weakened) && report.gate.weakened.length);
1141
+ if (report.verdict === 'pass' && report.metrics.some((m) => m.guard === true && m.ok === false))
1142
+ E("verdict 'pass' but a guarded metric regressed (guard:true, ok:false)");
1143
+ if (report.verdict === 'pass' && report.metrics.some((m) => m.gamed === true))
1144
+ E("verdict 'pass' but a metric is gamed:true");
1145
+ if (report.verdict === 'pass' && gateWeakened)
1146
+ E("verdict 'pass' but report.gate.weakened is non-empty (the gate was weakened)");
1147
+ if (report.verdict === 'gamed' && !report.metrics.some((m) => m.gamed === true) && !gateWeakened)
1148
+ E("verdict 'gamed' but no metric is gamed:true and report.gate records no weakening");
1149
+ }
1150
+
1151
+ return { ok: errors.length === 0, errors };
1152
+ }
1153
+
1154
+ // `promptwheel conformance [file.json | -] [--cmd "<command>"]`
1155
+ // Self-check a Contract implementation. With a file or stdin, validate that verdict JSON
1156
+ // against the Section 2 schema. With --cmd, run the command, then validate its --json
1157
+ // stdout (Section 2) AND that its exit code matches the verdict per Section 1. Exit 0 if
1158
+ // it conforms, 1 otherwise, listing every problem.
1159
+ function conformance(argv) {
1160
+ let cmd = null, src = null;
1161
+ for (let i = 0; i < argv.length; i++) {
1162
+ if (argv[i] === '--cmd') cmd = argv[++i];
1163
+ else if (argv[i] === '-' || !argv[i].startsWith('-')) src = argv[i];
1164
+ }
1165
+
1166
+ let text, exitCode = null;
1167
+ if (cmd) {
1168
+ try { text = execFileSync('sh', ['-c', cmd], { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }); exitCode = 0; }
1169
+ catch (e) {
1170
+ text = (e.stdout || '').toString();
1171
+ exitCode = typeof e.status === 'number' ? e.status : null;
1172
+ if (!text) { console.error(`conformance: command produced no stdout (exit ${exitCode}): ${e.message}`); process.exit(1); }
1173
+ }
1174
+ } else {
1175
+ const file = src && src !== '-' ? src : 0; // fd 0 = stdin
1176
+ try { text = readFileSync(file, 'utf8'); }
1177
+ catch (e) { console.error(`conformance: cannot read ${!src || src === '-' ? 'stdin' : src}: ${e.message}`); process.exit(1); }
1178
+ }
1179
+
1180
+ let report;
1181
+ try { report = JSON.parse(text); }
1182
+ catch (e) { console.error(`conformance: not valid JSON: ${e.message}`); process.exit(1); }
1183
+
1184
+ const { errors } = conformsToContract(report);
1185
+ const problems = [...errors];
1186
+ if (cmd && exitCode !== null && VERDICTS.includes(report.verdict)) {
1187
+ const want = verdictExitCode(report.verdict);
1188
+ if (exitCode !== want) problems.push(`exit code ${exitCode} does not match verdict '${report.verdict}' (Section 1 requires ${want})`);
1189
+ }
1190
+
1191
+ if (problems.length === 0) {
1192
+ console.log(`ok, conforms to the Verified-Outcome Contract v${CONTRACT_VERSION} (verdict: ${report.verdict}${cmd ? `, exit ${exitCode}` : ''})`);
1193
+ process.exit(0);
1194
+ }
1195
+ console.error(`does NOT conform to the Contract (${problems.length} problem${problems.length > 1 ? 's' : ''}):`);
1196
+ for (const p of problems) console.error(` - ${p}`);
1197
+ process.exit(1);
1198
+ }
867
1199
 
868
1200
  function main() {
869
1201
  const [cmd, ...rest] = process.argv.slice(2);
@@ -875,6 +1207,7 @@ function main() {
875
1207
  else if (cmd === 'backfill') backfill(rest);
876
1208
  else if (cmd === 'init') init(rest);
877
1209
  else if (cmd === 'guards') guards(rest);
1210
+ else if (cmd === 'conformance') conformance(rest);
878
1211
  else {
879
1212
  console.log([
880
1213
  'PromptWheel, catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
@@ -891,7 +1224,9 @@ function main() {
891
1224
  ' promptwheel suggest [--json] UCB work-discovery: where the next attempt should go (experimental)',
892
1225
  ' promptwheel backfill [-n N | --since <ref>] seed the ledger from git history (cohort-tagged; commit types become labels)',
893
1226
  ' promptwheel guards show the effective guardrails (incl. inherited) + flag record',
1227
+ ' promptwheel conformance [file|-] [--cmd C] check a verdict JSON (or a command) against the Contract (exit 0 = conforms)',
894
1228
  '',
1229
+ 'Integrate (library / conform / shell / MCP): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
895
1230
  'Loop it: while promptwheel improve --attempt "$AGENT"; do :; done # stops on plateau/regression',
896
1231
  'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
897
1232
  ' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
package/package.json CHANGED
@@ -1,15 +1,22 @@
1
1
  {
2
2
  "name": "promptwheel",
3
- "version": "0.4.6",
4
- "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook.",
3
+ "version": "0.5.0",
4
+ "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook, or import { check } in process.",
5
5
  "type": "module",
6
6
  "bin": {
7
7
  "promptwheel": "bin/promptwheel.mjs",
8
8
  "pw": "bin/promptwheel.mjs"
9
9
  },
10
+ "exports": {
11
+ ".": "./bin/promptwheel.mjs",
12
+ "./bin/promptwheel.mjs": "./bin/promptwheel.mjs",
13
+ "./package.json": "./package.json",
14
+ "./packs/*": "./packs/*"
15
+ },
10
16
  "files": [
11
17
  "bin",
12
- "packs"
18
+ "packs",
19
+ "INTEGRATE.md"
13
20
  ],
14
21
  "engines": {
15
22
  "node": ">=18"
@@ -3,7 +3,7 @@
3
3
  Gate your AI coding loop on **security invariants**, not just tests. When an agent
4
4
  writes backend code, "tests pass" says nothing about whether it shipped a CORS
5
5
  wildcard, a client-controlled payment amount, or a hardcoded secret. This pack
6
- adds a `security_findings` metric that must not climb — and because it's a
6
+ adds a `security_findings` metric that must not climb - and because it's a
7
7
  PromptWheel metric, the win has to survive `--detect-gaming`: an agent can't
8
8
  lower the count by editing the pattern file or a config, only by fixing the code.
9
9
 
@@ -24,9 +24,9 @@ promptwheel run --working # fails if security_findings increased
24
24
  A finding-count that drops because you fixed the code is a `PASS`; a "fix" that
25
25
  only survives with test/config/pattern edits reverted is `GAMED`.
26
26
 
27
- ## What it checks (v0.1 — 8 high-precision invariants)
27
+ ## What it checks (v0.1 - 8 high-precision invariants)
28
28
 
29
- Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier —
29
+ Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier -
30
30
  the regex-portable subset, chosen for near-zero false positives:
31
31
 
32
32
  | Invariant | Severity |
@@ -46,7 +46,7 @@ Each pattern ships with its own false-positive guards (the `exclude` regexes).
46
46
 
47
47
  This is the **regex-portable** slice. The other ~75 recommended invariants
48
48
  (auth-enforcement, authz-revocation, tenant-isolation, dataflow-tainted sinks)
49
- need the AST + dataflow engine and are the follow-up — see the decant note in
49
+ need the AST + dataflow engine and are the follow-up - see the decant note in
50
50
  `explorations/`. The point of v0.1 is the architecture: security invariants are
51
51
  now a first-class, un-gameable PromptWheel metric. Zero dependencies; the scanner
52
52
  is one file (`scan.mjs`).