promptwheel 0.4.7 → 0.5.1

package/INTEGRATE.md

@@ -0,0 +1,213 @@
+# Integrating PromptWheel
+
+PromptWheel is the reference implementation of the **Verified-Outcome Contract**: a
+deterministic, zero-LLM check that proves a change moved a real metric and that the agent
+earned the win in the source, not by editing the test, config, or grader. This page is the
+integration surface. There are four ways in, from loosest coupling to tightest.
+
+The protocol itself (exit codes, the verdict JSON, the integrity boundary) is specified in
+[CONTRACT.md](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md). The sections
+below say `Section N` where that document says `N`.
+
+---
+
+## 1. Conform to the Contract (no code from here)
+
+The strongest guarantee and the loosest coupling: emit or consume the two artifacts the
+Contract defines, and any harness interoperates with any other, in any language, without
+importing this package.
+
+- **Section 1, exit codes.** `0` pass, `1` fail (a guarded metric regressed beyond noise),
+  `2` gamed (a win that did not survive a source-only re-run), `3` inconclusive (a guard
+  measured nothing, so a pass cannot be certified). An agent reads the code as a per-turn
+  reward it cannot inflate; a merge gate reads the same code as a wall it cannot self-clear.
+- **Section 2, the verdict JSON.** The object below, reproducible from `(base, head, config)`
+  alone.
+
+```json
+{
+  "base": "a1b2c3d",
+  "head": "e4f5g6h",
+  "mode": "refs",
+  "repeat": 3,
+  "cohort": "ci",
+  "verdict": "gamed",
+  "subsystems": ["api", "auth"],
+  "metrics": [
+    {
+      "name": "tests_pass",
+      "direction": "pass",
+      "guard": true,
+      "before": 0,
+      "after": 1,
+      "delta": 1,
+      "status": "improved",
+      "ok": true,
+      "confidence": "high",
+      "noise": 0,
+      "gamed": true,
+      "sourceOnly": null,
+      "retained": 0.0,
+      "gamingReason": "the win changed zero production-source files, only test/config/grader/golden"
+    }
+  ]
+}
+```
+
+Fixed vocabularies (use exactly these):
+
+- `verdict` is one of `pass | fail | gamed | inconclusive`
+- `metric.status` is one of `improved | regressed | unchanged | inconclusive | unmeasurable`
+- `direction` is one of `up | down | pass` (the direction that counts as better; `pass` is a 0/1 gate)
+
+`noise` is the measured jitter band, a single number, across the top-level `repeat` runs; a
+delta inside it is `inconclusive`, never a regression. `retained` is the fraction of a
+metric's gain that survived the source-only re-run (Section 3). `confidence`
+(`unverified | low | medium | high`) and `sourceOnly` are informational fields this
+implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
+truncated SHA-256 fingerprint of the gate definition at base and head and lists any guard the
+change dropped to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
+cannot pass by disabling the guard it is judged by.
+
+Self-check your implementation against this schema (see Section 4 below):
+
+```bash
+your-tool --json | promptwheel conformance -
+```
+
+---
+
+## 2. Shell the CLI and parse `--json`
+
+Language-agnostic, process-isolated, the common path for CI and for harnesses not written in
+Node. Run the gate, read the exit code (Section 1), parse stdout (Section 2).
+
+```bash
+promptwheel run --base "$BASE" --head "$HEAD" --json
+echo $?   # 0 pass | 1 fail | 2 gamed | 3 inconclusive
+```
+
+```bash
+promptwheel run --working --json        # gate uncommitted changes (incl. newly added files)
+promptwheel run --no-detect-gaming       # the bare gate, without the Section 3 re-proof
+```
+
+Gaming detection (Section 3) is **on by default**. Nothing touches your working tree: every
+measurement runs in a throwaway git worktree.
+
+---
+
+## 3. Import the library (in process)
+
+Zero dependencies, no build step, the whole engine is one file. `check()` runs the gate in
+process and returns the Section 2 object. It never prints and never calls `process.exit`; it
+throws a plain `Error` on a precondition failure.
+
+```js
+import { check, verdictExitCode, conformsToContract, VERDICTS, CONTRACT_VERSION } from 'promptwheel';
+
+const report = check({ base: 'HEAD~1', head: 'HEAD' });   // discovers the repo from cwd
+if (report.verdict === 'gamed') {
+  const m = report.metrics.find((x) => x.gamed);
+  throw new Error(`gamed: ${m.name} (${m.gamingReason})`);
+}
+process.exit(verdictExitCode(report.verdict));             // exit exactly as the CLI would
+```
+
+### `check(options) -> report`
+
+Returns the Section 2 verdict object (identical to `run --json`). Reproducible from
+`(base, head, config)` alone.
+
+| option | type | default | meaning |
+| --- | --- | --- | --- |
+| `repo` | string | from `process.cwd()` | repo path |
+| `base` | string | merge-base vs `origin/main`..., else `HEAD~1` | base ref |
+| `head` | string | `HEAD` | head ref |
+| `working` | boolean | `false` | gate uncommitted changes instead of `base..head` |
+| `config` | object | load `promptwheel.config.json` from the repo | a resolved config `{ metrics: [...], ... }`, supplied in process |
+| `repeat` | number | `config.repeat ?? 1` | measurement repeats for the noise band |
+| `detectGaming` | boolean | `true` | run the Section 3 source-only re-proof |
+| `cohort` | string | `CI ? 'ci' : 'local'` | cohort tag for the record |
+| `label` | string | none | change-type label |
+| `record` | boolean | `false` | append the run to `.promptwheel/outcomes.jsonl` |
+
+Preconditions throw (they do not exit the host): not inside a git repository; no
+`promptwheel.config.json` and no `config` option; a `config` with no non-empty `metrics`
+array. Supplying `config` needs no file on disk:
+
+```js
+const report = check({
+  repo: '/path/to/repo', base: 'HEAD~1', head: 'HEAD',
+  config: { linkNodeModules: false, metrics: [
+    { name: 'tests_pass', cmd: 'npm test', extract: 'exit', direction: 'pass', guard: true },
+  ] },
+});
+```
+
+### `verdictExitCode(verdict) -> 0 | 1 | 2 | 3`
+
+The one Section 1 mapping the CLI uses too. Map verdicts this way and your process exits
+exactly as `promptwheel run` does.
+
+### `conformsToContract(report) -> { ok, errors }`
+
+Structurally validate any verdict object against Section 2 (shapes, the fixed vocabularies,
+and the honest-verdict rules: a `pass` cannot hide a failing guard or a gamed win, a `gamed`
+verdict must point at a gamed metric). Zero dependency, so a third-party implementation can
+self-check the artifact it emits. `errors` is a list of human-readable problems, empty when
+`ok` is `true`.
+
+### Other exports
+
+`CONTRACT_VERSION` (the protocol revision this build implements), and the frozen vocab
+arrays `VERDICTS`, `METRIC_STATUSES`, `DIRECTIONS`, so a consumer validates against the source
+of truth instead of hardcoding the strings.
+
+---
+
+## 4. Self-check with `promptwheel conformance`
+
+Prove your own check, loop, or CI step speaks the Contract.
+
+```bash
+# validate a verdict JSON (from a file or stdin) against the Section 2 schema
+promptwheel conformance report.json
+cat report.json | promptwheel conformance -
+
+# run a command, then validate BOTH its --json stdout (Section 2)
+# AND that its exit code matches the verdict (Section 1)
+promptwheel conformance --cmd "your-tool --json"
+```
+
+Exit `0` if it conforms, `1` otherwise, listing every problem. This is how a third-party
+implementation earns the right to say "Verified-Outcome Contract compatible."
+
+---
+
+## The memory half: @promptwheel/logbook
+
+The check verifies what holds now; the **logbook** records what happened over the history.
+It is a separate package with the same thin, model-blind, zero-dependency design, and it ships
+three integration surfaces of its own:
+
+- **CLI + JSON.** `logbook query --json`, `logbook check --diff --json`. Shell it like Section 2.
+- **MCP.** `@promptwheel/logbook-mcp` (bin `logbook-mcp`) exposes the history to an agent over
+  the Model Context Protocol, so a Claude Code or other MCP client reads hotspots, reverts, and
+  suppression signals without shelling anything.
+- **Import.** The package is importable (`@promptwheel/logbook`) and importing it never runs the
+  CLI; `queryEvents`, `analyze`, `auditHead`, and `hotspots` are exported for advanced in-process
+  use. The scan and cache entry points take explicit options, so the CLI and MCP are the
+  first-class surfaces for most integrations. The reference harness (Sawbuck) consumes the
+  logbook by shelling `logbook query --json`, not by importing it, which keeps the same process
+  boundary the check engine uses.
+
+---
+
+## The layer boundary (why this stays thin)
+
+The engines are deterministic and model-blind: no LLM call, no network, no runtime
+dependency, no build step. A harness that orchestrates models (Sawbuck is the reference one)
+consumes these engines across a process boundary, by shelling the CLI or resolving the bin,
+and never imports their internals. Keep that boundary and the check stays something a
+skeptic can re-run and get the same verdict, which is the entire point.

package/README.md

@@ -4,13 +4,15 @@
 
 **A deterministic check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel tells the difference: it re-proves every win from the source changes alone, with no LLM, so a win that only survived by editing the test reads `GAMED`. Un-gameable by construction, because the code being checked never gets to grade itself.
 
+![PromptWheel catching an agent game the metric, then passing an honest fix](demo/promptwheel.gif)
+
 PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop, a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
 
 It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"*, PromptWheel also asks *"did the agent **earn** it?"*
 
 PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question, *did this turn earn its keep, and did the agent earn it honestly?*, so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
 
-PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop.
+PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop. Use it from CI, a Claude Code hook, or in process (`import { check } from 'promptwheel'`); the four integration paths (conform, shell, import, MCP) are in [INTEGRATE.md](INTEGRATE.md).
 
 > Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate**, making the reward signal one you can't cheat.
 
@@ -104,6 +106,8 @@ npx promptwheel backfill -n 30            # cold start: seed the record from git
 
 **Footprint:** it never touches your working tree, every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied, `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record, commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
 
+**Per-turn cost.** A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly **2 to 3× your metric command's runtime** plus about 40ms of worktree overhead. Point the **per-turn** gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for **CI**: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
+
 ## Loop patterns
 
 PromptWheel is the gate *inside* a loop you don't have to write:

package/bin/promptwheel.mjs

@@ -11,6 +11,7 @@
 //   promptwheel run [--base R] [--head R] [--repeat N] [--json]
 
 import { execSync, execFileSync } from 'node:child_process';
+import { createHash } from 'node:crypto';
 import { readFileSync, writeFileSync, existsSync, mkdtempSync, symlinkSync, rmSync, mkdirSync, appendFileSync, realpathSync, readdirSync, statSync, copyFileSync } from 'node:fs';
 import { tmpdir } from 'node:os';
 import { join, dirname, isAbsolute, relative } from 'node:path';
@@ -395,10 +396,74 @@ function warnEditableInstall(repo) {
 }
 
 // the shared core: measure a change (base→head) and return the structured report
+// ---- gate integrity: catch the agent weakening the gate it is judged by ----
+// The config is read from HEAD, so a change can disable a guard (guard:false) or delete a
+// guarded metric in the very diff it "passes". We resolve the gate as of BASE too and, if a
+// guard was dropped, re-measure it under its BASE definition; the pass only reads GAMED when
+// that guard WOULD HAVE FAILED at head (a real regression hidden by moving the goalposts).
+const GATE_SCALARS = ['gamingThreshold', 'evaluatorPaths'];
+
+// Resolve the gate definition as of `ref` (metrics + scalars), following `extends` by reading
+// every file from that ref via `git show`. Returns { metrics:Map, scalars } or null when there
+// is no config at `ref` (a gate ADDED in this range is not a weakening). Never exits: any
+// missing/unreadable file or parse error yields null (that branch simply contributes nothing).
+function gateDefAtRef(repo, ref) {
+  // quiet: a missing path makes `git show` exit nonzero and print "fatal:" to stderr; swallow it
+  const readAt = (p) => { try { return execFileSync('git', ['show', `${ref}:${p}`], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }); } catch { return null; } };
+  const seen = new Set();
+  const resolve = (p) => {
+    if (seen.has(p)) return null;
+    seen.add(p);
+    const raw = readAt(p);
+    if (raw == null) return null;
+    let cfg; try { cfg = JSON.parse(raw); } catch { return null; }
+    const byName = new Map(); const scalars = {};
+    for (const ext of [].concat(cfg.extends || [])) {
+      if (isAbsolute(ext)) continue;                 // outside the repo, cannot git-show; unchanged anyway
+      const base = resolve(join(dirname(p), ext).split('\\').join('/'));
+      if (!base) continue;
+      for (const m of base.metrics.values()) byName.set(m.name, m);
+      for (const k of GATE_SCALARS) if (base.scalars[k] !== undefined) scalars[k] = base.scalars[k];
+    }
+    for (const m of (cfg.metrics || [])) if (m && m.name) byName.set(m.name, m);
+    for (const k of GATE_SCALARS) if (cfg[k] !== undefined) scalars[k] = cfg[k];
+    return { metrics: byName, scalars };
+  };
+  for (const p of ['promptwheel.config.json', 'outcome-gate.config.json']) { const r = resolve(p); if (r) return r; } // first hit only: no spurious probe of the alias
+  return null;
+}
+
+// an order-independent SHA-256 (truncated) fingerprint of a gate definition, for the audit record
+function gateFingerprint(metricsList, scalars) {
+  const norm = {
+    metrics: metricsList.map((m) => ({
+      name: m.name, guard: m.guard === true, direction: m.direction || 'up',
+      gamingThreshold: m.gamingThreshold ?? null, gamingCheck: m.gamingCheck !== false,
+    })).sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)),
+    gamingThreshold: scalars.gamingThreshold ?? null,
+    evaluatorPaths: Array.isArray(scalars.evaluatorPaths) ? [...scalars.evaluatorPaths].sort() : [],
+  };
+  return createHash('sha256').update(JSON.stringify(norm)).digest('hex').slice(0, 16);
+}
+
+// guarded metrics that HEAD dropped relative to BASE (removed entirely, or guard turned off)
+function gateWeakenings(baseGate, headMetrics) {
+  if (!baseGate) return [];
+  const head = new Map(headMetrics.map((m) => [m.name, m]));
+  const weak = [];
+  for (const bm of baseGate.metrics.values()) {
+    if (bm.guard !== true) continue;                 // only a guard protects anything
+    const hm = head.get(bm.name);
+    if (!hm) weak.push({ name: bm.name, change: 'guard-removed', def: bm });
+    else if (hm.guard !== true) weak.push({ name: bm.name, change: 'guard-disabled', def: bm });
+  }
+  return weak;
+}
+
 function gate(repo, opts) {
   selfHeal(repo);
   warnEditableInstall(repo);
-  const cfg = loadConfig(repo);
+  const cfg = opts.config ?? loadConfig(repo);
   const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
 
   let base, head;
@@ -431,8 +496,33 @@ function gate(repo, opts) {
       m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
     }
   }
+  // gate integrity (Section 3): did HEAD weaken the gate it is judged by? Resolve the gate as
+  // of BASE; a guard HEAD dropped is re-measured under its base definition and only counts as
+  // tampering if it WOULD HAVE FAILED at head (a real regression hidden by dropping the guard).
+  let gateInfo = null;
+  if (opts.detectGaming) {
+    const baseGate = gateDefAtRef(repo, base);
+    const weak = gateWeakenings(baseGate, cfg.metrics);
+    const weakened = [];
+    if (weak.length) {
+      const defs = weak.map((w) => w.def);
+      const b = measureAt(repo, base, { ...cfg, metrics: defs }, repeat);
+      const a = measureAt(repo, head, { ...cfg, metrics: defs }, repeat);
+      for (const w of weak) {
+        const ev = evaluate(w.def, b[w.name], a[w.name], repeat);
+        if (!ev.ok) weakened.push({ name: w.name, change: w.change, status: ev.status });
+      }
+    }
+    gateInfo = {
+      fingerprint: {
+        base: baseGate ? gateFingerprint([...baseGate.metrics.values()], baseGate.scalars) : null,
+        head: gateFingerprint(cfg.metrics, cfg),
+      },
+      ...(weakened.length ? { weakened } : {}),
+    };
+  }
   const failed = metrics.some((m) => m.guard && !m.ok);
-  const gamed = metrics.some((m) => m.gamed === true);
+  const gamed = metrics.some((m) => m.gamed === true) || !!(gateInfo && gateInfo.weakened);
   // a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
   // nothing, don't launder that into a green PASS. The honest verdict is "couldn't measure".
   // BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
@@ -447,6 +537,7 @@ function gate(repo, opts) {
     ...(opts.label ? { label: opts.label } : {}),
     subsystems: subsystemsOf(repo, base, head, cfg),
     verdict, metrics,
+    ...(gateInfo ? { gate: gateInfo } : {}),
   };
   if (!opts.noRecord && cfg.record !== false) recordOutcome(repo, report);
   return report;
@@ -459,7 +550,7 @@ function run(argv) {
   if (args.json) console.log(JSON.stringify(report, null, 2));
   else if (args.markdown) console.log(renderMarkdown(report));
   else printHuman(report);
-  process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : report.verdict === 'inconclusive' ? 3 : 1);
+  process.exit(verdictExitCode(report.verdict));
 }
 
 // persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
@@ -863,6 +954,7 @@ function printHuman(r) {
     if (m.inert) console.log('      ⚠ never passed at either ref, this guard is protecting nothing (check its command)');
     if (m.gamed === true) console.log(`      🚩 GAMED, ${m.gamingReason}`);
   }
+  if (r.gate && r.gate.weakened) for (const w of r.gate.weakened) console.log(`  🚩 GATE WEAKENED, the guard "${w.name}" was ${w.change === 'guard-removed' ? 'removed' : 'disabled'} in this change and it would have ${w.status} at head (goalpost moved, not earned)`);
   console.log(`\n  VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ', a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
   if (r.verdict === 'fail') console.log('  intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name), `promptwheel guards` shows the effective set');
   console.log('');
@@ -887,6 +979,7 @@ function renderMarkdown(r) {
     rows,
     '',
     r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
+    ...(r.gate && r.gate.weakened ? ['', `> 🚩 **Gate weakened:** ${r.gate.weakened.map((w) => `\`${w.name}\` ${w.change === 'guard-removed' ? 'removed' : 'disabled'} (would have ${w.status} at head)`).join('; ')}. The pass was bought by dropping a guard, not by earning it.`] : []),
     '',
     '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
   ].join('\n');
@@ -916,6 +1009,194 @@ function parseArgs(argv) {
 // runs when this file is executed directly (not when imported by the test suite).
 export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluatorMatcher, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
 
+// ─────────────────────────────────────────────────────────────────────────────
+// Public library API: the Verified-Outcome Contract, in process.
+//   import { check } from 'promptwheel';
+// Importing this file never runs the CLI (see the isMain guard at the bottom).
+// Stable across 0.x minor versions. The schema check() returns and the exit codes
+// verdictExitCode() maps are specified in CONTRACT.md; integration paths (conform,
+// shell, import, MCP) are in INTEGRATE.md.
+// ─────────────────────────────────────────────────────────────────────────────
+
+// The contract revision this build implements (CONTRACT.md "Versioning"). Bumps only
+// when the verdict schema or the exit codes change; a consumer may assert on it.
+export const CONTRACT_VERSION = '1';
+
+// Section 2 fixed vocabularies, exported so a consumer validates against the source of
+// truth instead of hardcoding the strings.
+export const VERDICTS = Object.freeze(['pass', 'fail', 'gamed', 'inconclusive']);
+export const METRIC_STATUSES = Object.freeze(['improved', 'regressed', 'unchanged', 'inconclusive', 'unmeasurable']);
+export const DIRECTIONS = Object.freeze(['up', 'down', 'pass']);
+
+// Section 1: the one verdict -> process exit code mapping. The CLI uses it too, so a
+// consumer that maps verdicts this way exits exactly as `promptwheel run` does.
+export function verdictExitCode(verdict) {
+  return verdict === 'pass' ? 0 : verdict === 'gamed' ? 2 : verdict === 'inconclusive' ? 3 : 1;
+}
+
+// check(options) -> report
+// Run the gate in process and RETURN the Section 2 verdict object (identical to what
+// `run --json` prints). Measurement happens only in throwaway worktrees, never the
+// caller's working tree (constitution #4). This function never prints and never calls
+// process.exit; it throws a plain Error on a precondition failure (not a git repo, or a
+// missing / invalid config). Reproducible from (base, head, config) alone.
+//
+//   repo          string   repo path (default: discovered from process.cwd())
+//   base          string   base ref (default: merge-base vs origin/main..., else HEAD~1)
+//   head          string   head ref (default: HEAD)
+//   working       boolean  gate uncommitted changes instead of base..head
+//   config        object   a resolved config { metrics:[...], ... }; omit to load
+//                          promptwheel.config.json from the repo
+//   repeat        number   measurement repeats for the noise band (default: config.repeat ?? 1)
+//   detectGaming  boolean  run the Section 3 source-only re-proof (default: true)
+//   cohort        string   cohort tag for the record (default: CI ? 'ci' : 'local')
+//   label         string   change-type label
+//   record        boolean  append the run to .promptwheel/outcomes.jsonl (default: false)
+export function check(options = {}) {
+  let repo;
+  try {
+    repo = options.repo ? realpathSync(options.repo)
+                        : git(['rev-parse', '--show-toplevel'], process.cwd());
+  } catch { throw new Error(`promptwheel check: not inside a git repository (${options.repo || process.cwd()})`); }
+
+  if (options.config !== undefined) {
+    const c = options.config;
+    if (!c || typeof c !== 'object' || !Array.isArray(c.metrics) || c.metrics.length === 0)
+      throw new Error('promptwheel check: options.config.metrics must be a non-empty array');
+  } else if (!['promptwheel.config.json', 'outcome-gate.config.json'].some((f) => existsSync(join(repo, f)))) {
+    throw new Error(`promptwheel check: no promptwheel.config.json in ${repo} (pass options.config to supply one in process)`);
+  }
+
+  return gate(repo, {
+    base: options.base,
+    head: options.head,
+    working: options.working,
+    config: options.config,
+    repeat: options.repeat,
+    detectGaming: options.detectGaming ?? true,
+    cohort: options.cohort,
+    label: options.label,
+    noRecord: options.record !== true,
+  });
+}
+
+// conformsToContract(report) -> { ok, errors }
+// Structurally validate a verdict object against Section 2 (shapes + fixed vocabularies)
+// plus the two honest-verdict rules that hold for any conforming implementation. Zero
+// dependency, so a third party can self-check the artifact it emits against the reference
+// schema. It checks well-formedness and consistency, not a deep re-derivation of the math.
+export function conformsToContract(report) {
+  const errors = [];
+  const E = (m) => errors.push(m);
+  const isInt = (n) => Number.isInteger(n);
+  const isNum = (n) => typeof n === 'number' && Number.isFinite(n);
+  const numOrNull = (n) => n === null || isNum(n);
+
+  if (!report || typeof report !== 'object' || Array.isArray(report)) return { ok: false, errors: ['report must be an object'] };
+  if (!VERDICTS.includes(report.verdict)) E(`verdict must be one of ${VERDICTS.join('|')} (got ${JSON.stringify(report.verdict)})`);
+  if (typeof report.base !== 'string' || !report.base) E('base must be a non-empty string');
+  if (typeof report.head !== 'string' || !report.head) E('head must be a non-empty string');
+  if (report.mode !== undefined && report.mode !== 'refs' && report.mode !== 'working') E("mode must be 'refs' or 'working'");
+  if (report.repeat !== undefined && !(isInt(report.repeat) && report.repeat >= 1)) E('repeat must be an integer >= 1');
+  if (report.cohort !== undefined && typeof report.cohort !== 'string') E('cohort must be a string');
+  if (report.subsystems !== undefined && !(Array.isArray(report.subsystems) && report.subsystems.every((s) => typeof s === 'string')))
+    E('subsystems must be an array of strings');
+  if (!Array.isArray(report.metrics)) { E('metrics must be an array'); return { ok: false, errors }; }
+
+  report.metrics.forEach((m, i) => {
+    const at = `metrics[${i}]`;
+    if (!m || typeof m !== 'object') { E(`${at} must be an object`); return; }
+    if (typeof m.name !== 'string' || !m.name) E(`${at}.name must be a non-empty string`);
+    if (!DIRECTIONS.includes(m.direction)) E(`${at}.direction must be one of ${DIRECTIONS.join('|')}`);
+    if (typeof m.guard !== 'boolean') E(`${at}.guard must be a boolean`);
+    if (!METRIC_STATUSES.includes(m.status)) E(`${at}.status must be one of ${METRIC_STATUSES.join('|')}`);
+    if (typeof m.ok !== 'boolean') E(`${at}.ok must be a boolean`);
+    if (m.before !== undefined && !numOrNull(m.before)) E(`${at}.before must be a number or null`);
+    if (m.after !== undefined && !numOrNull(m.after)) E(`${at}.after must be a number or null`);
+    if (m.delta !== undefined && !numOrNull(m.delta)) E(`${at}.delta must be a number or null`);
+    if (m.noise !== undefined && !(isNum(m.noise) && m.noise >= 0)) E(`${at}.noise must be a number >= 0 (the measured jitter band; repeat count is the top-level report.repeat)`);
+    if (m.confidence !== undefined && typeof m.confidence !== 'string') E(`${at}.confidence must be a string`);
+    if (m.gamed !== undefined && typeof m.gamed !== 'boolean') E(`${at}.gamed must be a boolean`);
+    if (m.sourceOnly !== undefined && !numOrNull(m.sourceOnly)) E(`${at}.sourceOnly must be a number or null`);
+    if (m.retained !== undefined && !numOrNull(m.retained)) E(`${at}.retained must be a number or null`);
+    if (m.gamingReason !== undefined && typeof m.gamingReason !== 'string') E(`${at}.gamingReason must be a string`);
+  });
+
+  // the optional gate-integrity record (report.gate): a config fingerprint and, when the
+  // agent weakened the gate it is judged by, the list of dropped guards.
+  if (report.gate !== undefined) {
+    if (!report.gate || typeof report.gate !== 'object' || Array.isArray(report.gate)) E('gate must be an object');
+    else {
+      if (report.gate.fingerprint !== undefined && (typeof report.gate.fingerprint !== 'object' || report.gate.fingerprint === null)) E('gate.fingerprint must be an object { base, head }');
+      if (report.gate.weakened !== undefined && !Array.isArray(report.gate.weakened)) E('gate.weakened must be an array');
+    }
+  }
+
+  // Honest-verdict rules that hold regardless of how a verdict is derived:
+  // a clean pass cannot hide a failing guard, a gamed win, or a weakened gate; and a gamed
+  // verdict must point at a gamed metric or a weakened gate. (Fail-vs-gamed precedence is
+  // left to the implementation.)
+  if (VERDICTS.includes(report.verdict)) {
+    const gateWeakened = !!(report.gate && Array.isArray(report.gate.weakened) && report.gate.weakened.length);
+    if (report.verdict === 'pass' && report.metrics.some((m) => m.guard === true && m.ok === false))
+      E("verdict 'pass' but a guarded metric regressed (guard:true, ok:false)");
+    if (report.verdict === 'pass' && report.metrics.some((m) => m.gamed === true))
+      E("verdict 'pass' but a metric is gamed:true");
+    if (report.verdict === 'pass' && gateWeakened)
+      E("verdict 'pass' but report.gate.weakened is non-empty (the gate was weakened)");
+    if (report.verdict === 'gamed' && !report.metrics.some((m) => m.gamed === true) && !gateWeakened)
+      E("verdict 'gamed' but no metric is gamed:true and report.gate records no weakening");
+  }
+
+  return { ok: errors.length === 0, errors };
+}
+
+// `promptwheel conformance [file.json | -] [--cmd "<command>"]`
+// Self-check a Contract implementation. With a file or stdin, validate that verdict JSON
+// against the Section 2 schema. With --cmd, run the command, then validate its --json
+// stdout (Section 2) AND that its exit code matches the verdict per Section 1. Exit 0 if
+// it conforms, 1 otherwise, listing every problem.
+function conformance(argv) {
+  let cmd = null, src = null;
+  for (let i = 0; i < argv.length; i++) {
+    if (argv[i] === '--cmd') cmd = argv[++i];
+    else if (argv[i] === '-' || !argv[i].startsWith('-')) src = argv[i];
+  }
+
+  let text, exitCode = null;
+  if (cmd) {
+    try { text = execFileSync('sh', ['-c', cmd], { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 }); exitCode = 0; }
+    catch (e) {
+      text = (e.stdout || '').toString();
+      exitCode = typeof e.status === 'number' ? e.status : null;
+      if (!text) { console.error(`conformance: command produced no stdout (exit ${exitCode}): ${e.message}`); process.exit(1); }
+    }
+  } else {
+    const file = src && src !== '-' ? src : 0; // fd 0 = stdin
+    try { text = readFileSync(file, 'utf8'); }
+    catch (e) { console.error(`conformance: cannot read ${!src || src === '-' ? 'stdin' : src}: ${e.message}`); process.exit(1); }
+  }
+
+  let report;
+  try { report = JSON.parse(text); }
+  catch (e) { console.error(`conformance: not valid JSON: ${e.message}`); process.exit(1); }
+
+  const { errors } = conformsToContract(report);
+  const problems = [...errors];
+  if (cmd && exitCode !== null && VERDICTS.includes(report.verdict)) {
+    const want = verdictExitCode(report.verdict);
+    if (exitCode !== want) problems.push(`exit code ${exitCode} does not match verdict '${report.verdict}' (Section 1 requires ${want})`);
+  }
+
+  if (problems.length === 0) {
+    console.log(`ok, conforms to the Verified-Outcome Contract v${CONTRACT_VERSION} (verdict: ${report.verdict}${cmd ? `, exit ${exitCode}` : ''})`);
+    process.exit(0);
+  }
+  console.error(`does NOT conform to the Contract (${problems.length} problem${problems.length > 1 ? 's' : ''}):`);
+  for (const p of problems) console.error(`  - ${p}`);
+  process.exit(1);
+}
+
 function main() {
   const [cmd, ...rest] = process.argv.slice(2);
   if (cmd === 'run') run(rest);
@@ -926,6 +1207,7 @@ function main() {
   else if (cmd === 'backfill') backfill(rest);
   else if (cmd === 'init') init(rest);
   else if (cmd === 'guards') guards(rest);
+  else if (cmd === 'conformance') conformance(rest);
   else {
     console.log([
       'PromptWheel, catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
@@ -942,7 +1224,9 @@ function main() {
       '  promptwheel suggest [--json]                 UCB work-discovery: where the next attempt should go (experimental)',
       '  promptwheel backfill [-n N | --since <ref>]  seed the ledger from git history (cohort-tagged; commit types become labels)',
       '  promptwheel guards                           show the effective guardrails (incl. inherited) + flag record',
+      '  promptwheel conformance [file|-] [--cmd C]   check a verdict JSON (or a command) against the Contract (exit 0 = conforms)',
       '',
+      'Integrate (library / conform / shell / MCP): see INTEGRATE.md. Import: `import { check } from \'promptwheel\'`.',
       'Loop it:  while promptwheel improve --attempt "$AGENT"; do :; done   # stops on plateau/regression',
       'Config:   promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] }   (or: promptwheel init)',
       '          extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',

package/package.json

@@ -1,15 +1,22 @@
 {
   "name": "promptwheel",
-  "version": "0.4.7",
-  "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook.",
+  "version": "0.5.1",
+  "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook, or import { check } in process.",
   "type": "module",
   "bin": {
     "promptwheel": "bin/promptwheel.mjs",
     "pw": "bin/promptwheel.mjs"
   },
+  "exports": {
+    ".": "./bin/promptwheel.mjs",
+    "./bin/promptwheel.mjs": "./bin/promptwheel.mjs",
+    "./package.json": "./package.json",
+    "./packs/*": "./packs/*"
+  },
   "files": [
     "bin",
-    "packs"
+    "packs",
+    "INTEGRATE.md"
   ],
   "engines": {
     "node": ">=18"

package/packs/security/README.md

@@ -3,7 +3,7 @@
 Gate your AI coding loop on **security invariants**, not just tests. When an agent
 writes backend code, "tests pass" says nothing about whether it shipped a CORS
 wildcard, a client-controlled payment amount, or a hardcoded secret. This pack
-adds a `security_findings` metric that must not climb — and because it's a
+adds a `security_findings` metric that must not climb - and because it's a
 PromptWheel metric, the win has to survive `--detect-gaming`: an agent can't
 lower the count by editing the pattern file or a config, only by fixing the code.
 
@@ -24,9 +24,9 @@ promptwheel run --working          # fails if security_findings increased
 A finding-count that drops because you fixed the code is a `PASS`; a "fix" that
 only survives with test/config/pattern edits reverted is `GAMED`.
 
-## What it checks (v0.1 — 8 high-precision invariants)
+## What it checks (v0.1 - 8 high-precision invariants)
 
-Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier —
+Decanted from [securitychecks](https://securitychecks.ai)' "recommended" tier - 
 the regex-portable subset, chosen for near-zero false positives:
 
 | Invariant | Severity |
@@ -46,7 +46,7 @@ Each pattern ships with its own false-positive guards (the `exclude` regexes).
 
 This is the **regex-portable** slice. The other ~75 recommended invariants
 (auth-enforcement, authz-revocation, tenant-isolation, dataflow-tainted sinks)
-need the AST + dataflow engine and are the follow-up — see the decant note in
+need the AST + dataflow engine and are the follow-up - see the decant note in
 `explorations/`. The point of v0.1 is the architecture: security invariants are
 now a first-class, un-gameable PromptWheel metric. Zero dependencies; the scanner
 is one file (`scan.mjs`).