promptwheel 0.7.4 → 0.7.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/INTEGRATE.md CHANGED
@@ -17,10 +17,13 @@ The strongest guarantee and the loosest coupling: emit or consume the two artifa
17
17
  Contract defines, and any harness interoperates with any other, in any language, without
18
18
  importing this package.
19
19
 
20
- - **Section 1, exit codes.** `0` pass, `1` fail (a guarded metric regressed beyond noise),
21
- `2` gamed (a win that did not survive a source-only re-run), `3` inconclusive (a guard
22
- measured nothing, so a pass cannot be certified). An agent reads the code as a per-turn
23
- reward it cannot inflate; a merge gate reads the same code as a wall it cannot self-clear.
20
+ - **Section 1, exit codes.** `0` pass, `1` fail (a guarded metric regressed beyond noise, or
21
+ a mandate was violated), `2` gamed (a win that did not survive a source-only re-run), `3`
22
+ inconclusive (a guard measured nothing, so a pass cannot be certified). An agent reads the
23
+ code as a per-turn reward it cannot inflate; a merge gate reads the same code as a wall it
24
+ cannot self-clear. A usage or config error (a malformed config, a missing `extends` target,
25
+ a bad flag) exits `64` (`EX_USAGE`), which is not a verdict: a CI step must never read a
26
+ typo as `GAMED`.
24
27
  - **Section 2, the verdict JSON.** The object below, reproducible from `(base, head, config)`
25
28
  alone.
26
29
 
@@ -30,9 +33,7 @@ importing this package.
30
33
  "head": "e4f5g6h",
31
34
  "mode": "refs",
32
35
  "repeat": 3,
33
- "cohort": "ci",
34
36
  "verdict": "gamed",
35
- "subsystems": ["api", "auth"],
36
37
  "metrics": [
37
38
  {
38
39
  "name": "tests_pass",
@@ -65,9 +66,10 @@ delta inside it is `inconclusive`, never a regression. `retained` is the fractio
65
66
  metric's gain that survived the source-only re-run (Section 3). `confidence`
66
67
  (`unverified | low | medium | high`) and `sourceOnly` are informational fields this
67
68
  implementation adds; a minimal conforming report may omit them. `gate` (optional) records a
68
- truncated SHA-256 fingerprint of the gate definition at base and head and lists any guard the
69
- change dropped to hide a regression; a non-empty `gate.weakened` forces `gamed`, so an agent
70
- cannot pass by disabling the guard it is judged by.
69
+ truncated SHA-256 fingerprint of the gate definition (metrics, guards, `evaluatorPaths`,
70
+ `gamingThreshold`, `mandates`) at base and head, sets `gate.changed: true` when they differ,
71
+ and lists any guard the change dropped to hide a regression; a non-empty `gate.weakened`
72
+ forces `gamed`, so an agent cannot pass by disabling the guard it is judged by.
71
73
 
72
74
  Self-check your implementation against this schema (see Section 4 below):
73
75
 
@@ -84,7 +86,7 @@ Node. Run the gate, read the exit code (Section 1), parse stdout (Section 2).
84
86
 
85
87
  ```bash
86
88
  promptwheel run --base "$BASE" --head "$HEAD" --json
87
- echo $? # 0 pass | 1 fail | 2 gamed | 3 inconclusive
89
+ echo $? # 0 pass | 1 fail | 2 gamed | 3 inconclusive | 64 usage/config error (not a verdict)
88
90
  ```
89
91
 
90
92
  ```bash
@@ -125,16 +127,18 @@ Returns the Section 2 verdict object (identical to `run --json`). Reproducible f
125
127
  | `base` | string | merge-base vs `origin/main`..., else `HEAD~1` | base ref |
126
128
  | `head` | string | `HEAD` | head ref |
127
129
  | `working` | boolean | `false` | gate uncommitted changes instead of `base..head` |
128
- | `config` | object | load `promptwheel.config.json` from the repo | a resolved config `{ metrics: [...], ... }`, supplied in process |
130
+ | `config` | object | load `promptwheel.config.json`, or auto-detect | a resolved config `{ metrics: [...], ... }`, supplied in process |
131
+ | `auto` | boolean | `true` | with no config file and no `config` option, gate the auto-detected test command (zero-config); `false` throws instead |
129
132
  | `repeat` | number | `config.repeat ?? 1` | measurement repeats for the noise band |
130
133
  | `detectGaming` | boolean | `true` | run the Section 3 source-only re-proof |
131
- | `cohort` | string | `CI ? 'ci' : 'local'` | cohort tag for the record |
132
- | `label` | string | none | change-type label |
134
+ | `allow` | string | none | a declared override reason (the same as `--allow` / a `Promptwheel-Allow:` trailer): a would-be `gamed` reads `pass` with the reason recorded in `report.allowed` |
133
135
  | `record` | boolean | `false` | append the run to `.promptwheel/outcomes.jsonl` |
134
136
 
135
- Preconditions throw (they do not exit the host): not inside a git repository; no
136
- `promptwheel.config.json` and no `config` option; a `config` with no non-empty `metrics`
137
- array. Supplying `config` needs no file on disk:
137
+ Preconditions throw (they do not exit the host): not inside a git repository; `auto: false`
138
+ with no `promptwheel.config.json` and no `config` option; a `config` with no non-empty
139
+ `metrics` array. With neither a file nor `config`, the default is zero-config: the detected
140
+ test command is gated (an inert command reads `inconclusive`, never a fake pass). Supplying
141
+ `config` needs no file on disk:
138
142
 
139
143
  ```js
140
144
  const report = check({
@@ -185,29 +189,13 @@ implementation earns the right to say "Verified-Outcome Contract compatible."
185
189
 
186
190
  ---
187
191
 
188
- ## The memory half: @promptwheel/logbook
192
+ ## Related
189
193
 
190
- The check verifies what holds now; the **logbook** records what happened over the history.
191
- It is a separate package with the same thin, model-blind, zero-dependency design, and it ships
192
- three integration surfaces of its own:
193
-
194
- - **CLI + JSON.** `logbook query --json`, `logbook check --diff --json`. Shell it like Section 2.
195
- - **MCP.** `@promptwheel/logbook-mcp` (bin `logbook-mcp`) exposes the history to an agent over
196
- the Model Context Protocol, so a Claude Code or other MCP client reads hotspots, reverts, and
197
- suppression signals without shelling anything.
198
- - **Import.** The package is importable (`@promptwheel/logbook`) and importing it never runs the
199
- CLI; `queryEvents`, `analyze`, `auditHead`, and `hotspots` are exported for advanced in-process
200
- use. The scan and cache entry points take explicit options, so the CLI and MCP are the
201
- first-class surfaces for most integrations. The reference harness (Sawbuck) consumes the
202
- logbook by shelling `logbook query --json`, not by importing it, which keeps the same process
203
- boundary the check engine uses.
204
-
205
- ---
206
-
207
- ## The layer boundary (why this stays thin)
208
-
209
- The engines are deterministic and model-blind: no LLM call, no network, no runtime
210
- dependency, no build step. A harness that orchestrates models (Sawbuck is the reference one)
211
- consumes these engines across a process boundary, by shelling the CLI or resolving the bin,
212
- and never imports their internals. Keep that boundary and the check stays something a
213
- skeptic can re-run and get the same verdict, which is the entire point.
194
+ PromptWheel is the check: it verifies what holds now. The history side is served by
195
+ [`@promptwheel/logbook`](https://www.npmjs.com/package/@promptwheel/logbook), a separate
196
+ zero-dependency package that turns a repo's git history into agent-readable memory
197
+ (hotspots, reverts, suppressions), shelled the same way (`logbook query --json`). Both are
198
+ deterministic and model-blind: no LLM call, no network, no build step. A harness that
199
+ orchestrates models should consume them across a process boundary (shell the CLI, parse
200
+ `--json`) rather than importing their internals, so a skeptic can always re-run the check and
201
+ get the same verdict.
package/README.md CHANGED
@@ -2,51 +2,13 @@
2
2
 
3
3
  [![CI](https://github.com/promptwheel-ai/promptwheel/actions/workflows/ci.yml/badge.svg)](https://github.com/promptwheel-ai/promptwheel/actions/workflows/ci.yml) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) ![Node ≥18](https://img.shields.io/badge/node-%E2%89%A518-brightgreen) ![deps: zero](https://img.shields.io/badge/deps-zero-blue)
4
4
 
5
- **A deterministic check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel tells the difference: it re-proves every win from the source changes alone, with no LLM, so a win that only survived by editing the test reads `GAMED`. Un-gameable by construction, because the code being checked never gets to grade itself.
5
+ **A deterministic, zero-LLM check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel re-proves every win from the agent's source edits alone, so a green that came from editing the test, grader or config reads `GAMED`. The code being checked never gets to grade itself.
6
6
 
7
7
  ![PromptWheel catching an agent game the metric, then passing an honest fix](demo/promptwheel.gif)
8
8
 
9
- PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop, a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
9
+ ## Catch your agent cheating, in 60 seconds
10
10
 
11
- It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"*, PromptWheel also asks *"did the agent **earn** it?"*
12
-
13
- PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question, *did this turn earn its keep, and did the agent earn it honestly?*, so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
14
-
15
- PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop. Use it from CI, a Claude Code hook, or in process (`import { check } from 'promptwheel'`); the four integration paths (conform, shell, import, MCP) are in [INTEGRATE.md](INTEGRATE.md).
16
-
17
- > Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate**, making the reward signal one you can't cheat.
18
-
19
- AI coding agents (and humans) declare success when a change *compiles and tests pass*. They almost never verify that a change **improved a real measurable thing without regressing another**, and outcome data rots in dashboards (DORA, Swarmia) disconnected from the change that caused it. PromptWheel closes that loop.
20
-
21
- For any change (`base` → `head`), it measures each configured metric in an **isolated git worktree** before and after, **refuses to trust a delta inside the measurement noise band**, enforces **regression guards**, and emits a structured verdict:
22
-
23
- ```
24
- PromptWheel a1b2c3d → e4f5g6h (×5)
25
-
26
- = tests_pass 1 → 1 (0, unchanged) [guard✓, high]
27
- ▼ lint_errors 12 → 7 (-5, improved) [guard✓, high]
28
- ▲ bundle_kb 340 → 352 (+12, regressed) [info, medium]
29
- ▼ p95_ms 210 → 208 (-2, inconclusive) [info, low]
30
-
31
- VERDICT: PASS
32
- ```
33
-
34
- Exit `0` pass · `1` fail · `2` gamed · `3` inconclusive (a guard measured nothing), CI-friendly. No build step, zero dependencies, Node 18+.
35
-
36
- ## Catch your agent cheating, on by default
37
-
38
- The headline feature, and it runs **by default** (in both `run` and `improve`; pass `--no-detect-gaming` for the bare outcome gate). After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive, because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files, the verdict is **`GAMED` (exit 2)**:
39
-
40
- ```
41
- $ promptwheel run
42
-
43
- PromptWheel base → head
44
- ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
45
- 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
46
- VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
47
- ```
48
-
49
- **See it fire yourself**, 60 seconds, illustrative (a hand-built cheat is a *mechanism* demo, not evidence):
11
+ A hand-built cheat, so you can watch the mechanism fire (illustrative, not evidence):
50
12
 
51
13
  ```bash
52
14
  mkdir pw-cheat-demo && cd pw-cheat-demo && git init -q
@@ -56,7 +18,7 @@ printf "import {test} from 'node:test'; import assert from 'node:assert'; import
56
18
  git add -A && git commit -qm 'buggy code + an honest failing test'
57
19
  npx -y promptwheel@latest init && git add -A && git commit -qm 'add the gate config'
58
20
 
59
- # an agent "greens" the suite by editing the TEST to expect the bug, the code stays broken:
21
+ # an agent "greens" the suite by editing the TEST to expect the bug; the code stays broken:
60
22
  printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), -1));\n" > add.test.js
61
23
  git commit -qam 'make the suite green'
62
24
 
@@ -65,110 +27,59 @@ npx -y promptwheel@latest run --base HEAD~1 --head HEAD
65
27
  # VERDICT: GAMED (exit 2)
66
28
  ```
67
29
 
68
- Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`), and the quieter cheat of *weakening the suite while the metric stays flat*, are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
30
+ What happened: PromptWheel measured `tests_pass` at both commits in throwaway git worktrees (0 then 1, a win), then split the diff into production source versus `{test, config, grader, golden}` files, re-applied **only the source slice** on a clean worktree at the base, and re-ran the metric. The win did not survive, so it is `GAMED`. No LLM, no network: a diff partition plus a re-run, reproducible in seconds with a human-readable reason.
31
+
32
+ The same runs on any change in any stack. For a change `base` to `head` it measures each configured metric before and after, refuses to trust a delta inside the measurement noise band, enforces regression guards, and emits one verdict:
69
33
 
70
- ```bash
71
- npx promptwheel init # guarded tests + antihack tripwires by default
72
- npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED · 3 inconclusive
73
- # (add --no-detect-gaming for just the outcome gate)
74
34
  ```
35
+ PromptWheel a1b2c3d → e4f5g6h (×5)
75
36
 
76
- Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable via `gamingThreshold` (config-level, or per-metric).
37
+ = tests_pass 1 → 1 (0, unchanged) [guard✓, high]
38
+ ▼ lint_errors 12 → 7 (-5, improved) [guard✓, high]
39
+ ▲ bundle_kb 340 → 352 (+12, regressed) [info, medium]
40
+ ▼ p95_ms 210 → 208 (-2, inconclusive) [info, low]
77
41
 
78
- **An intentional goalpost change has a sanctioned path.** When you *do* mean to change a test, config, or grader, declare it: a `Sawbuck-Allow: <reason>` trailer on the commit (or `--allow "<reason>"` on the CLI, or the `allow` option on `check()`) turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. An *undeclared* test-only win is still `gamed`, and a reasonless declaration does not override; the check never judges intent, it just forces intent to be explicit and auditable (the reason lives in the commit, so a reviewer sees it instead of a silent green).
42
+ VERDICT: PASS
43
+ ```
79
44
 
80
- This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers, held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage, are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce, it includes a cross-stack table with a real pytest run + a numeric eval-pass-rate metric). **Gate your own stack**, pytest · tsc · coverage · bundle · llm-eval, copy a config from **[examples/](examples/)**.
45
+ Exit `0` pass · `1` fail (a guarded metric regressed, or a mandate was violated) · `2` gamed · `3` inconclusive (a guard measured nothing) · `64` usage or config error (never a verdict). No build step, zero dependencies, Node 18+. The same verdict runs as a PR check in CI, as a per-turn signal inside a coding loop, or in process (`import { check } from 'promptwheel'`); it is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), and the integration paths are in [INTEGRATE.md](INTEGRATE.md).
81
46
 
82
- ## Use
47
+ ## Install and use
83
48
 
84
49
  ```bash
85
- # 0. write a starter config for your stack (or hand-write promptwheel.config.json)
86
- npx promptwheel init # detects stack → guarded tests + antihack tripwires (+ lint if eslint is set up)
87
- npx promptwheel init --list # presets: tests-pass · lint · bundle-size · llm-eval · antihack
50
+ npm i -g promptwheel # or run everything through: npx promptwheel ...
88
51
 
89
- # measure a change
90
- npx promptwheel run # base = merge-base with main, head = HEAD · reward-hack detection ON
91
- npx promptwheel run --working # measure UNCOMMITTED changes (incl. newly added files)
52
+ npx promptwheel run # zero-config: gates your detected test command; base = merge-base with main, head = HEAD
53
+ npx promptwheel run --working # gate UNCOMMITTED changes (incl. newly added files)
92
54
  npx promptwheel run --repeat 5 --json # measure 5× to establish a noise band, emit JSON
93
55
 
94
- # the loop: run any agent/script, keep the change ONLY if a metric improved
95
- npx promptwheel improve --attempt "claude -p 'reduce lint errors'"
96
- # exit 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau / inconclusive (reverted) · add --json
56
+ npx promptwheel init # optional: write promptwheel.config.json (guarded tests + antihack tripwires, + lint if eslint is set up)
57
+ npx promptwheel init --list # presets: tests-pass · lint · bundle-size · security · llm-eval · antihack · no-ai-tells · security-antihack
97
58
 
98
- # what's actually responding in this repo? (aggregates .promptwheel/outcomes.jsonl)
99
- npx promptwheel insights
59
+ # the loop: run any agent/script, keep the change ONLY if a metric improved, else revert
60
+ npx promptwheel improve --attempt "claude -p 'reduce lint errors'"
61
+ # exit 0 = kept a real win · 1 = guarded regression (reverted) · 2 = gamed (reverted) · 3 = plateau / inconclusive (reverted) · add --json
62
+ while npx promptwheel improve --attempt "$AGENT"; do :; done # converges: stops on the first non-zero
100
63
 
101
- # EXPERIMENTAL (Phase 5): the earned playbook + where the next attempt should go
102
- npx promptwheel playbook # decayed, evidence-gated claims distilled from the record
103
- npx promptwheel suggest # UCB over the lever scores, proven levers vs under-explored arms
104
- npx promptwheel backfill -n 30 # cold start: seed the record from git history (cohort-tagged, commit types → labels)
64
+ npx promptwheel insights # which metrics actually respond in this repo (counts from .promptwheel/outcomes.jsonl)
65
+ npx promptwheel guards # the effective guardrails, including ones inherited through extends
105
66
  ```
106
67
 
107
- ## Give your agent the gate
68
+ Gaming detection is on by default in `run` and `improve`; `--no-detect-gaming` gives you the bare outcome gate.
108
69
 
109
- `promptwheel init` writes this into your `AGENTS.md` for you (and `CLAUDE.md` / `.cursorrules` if present; `--no-agent-block` to skip), so a coding agent (or a loop) runs the gate itself, not just you. To paste it yourself:
70
+ **Give your agent the gate.** `promptwheel init` writes this into your `AGENTS.md` (and `CLAUDE.md` / `.cursorrules` if present; `--no-agent-block` to skip), so the coding agent runs the gate itself. To paste it yourself:
110
71
 
111
72
  After a change that should move a measurable metric (tests, lint, coverage, an eval), run
112
73
  `npx promptwheel run --working`. Exit 2 (verdict GAMED) means the win came from editing a test,
113
74
  config, or grader, not the source, so revert and earn it in the source. Exit 1 means a guarded
114
75
  metric regressed, revert that too. Never weaken, skip, or delete a test to go green.
115
76
 
116
- **The consequence ledger.** git records *what changed*; PromptWheel records *what the change did*, same trust model (local, deterministic, append-only, no server, no LLM in the verdict). `playbook` and `suggest` are pure re-derivations over that ledger: every rendered line was measured by the gate, decays unless re-earned, and stays hidden below an evidence threshold. No compounding claim is made for them until the A/B acceptance test (`bench/compounding-ab.mjs`) passes on real usage data.
117
-
118
- **Footprint:** it never touches your working tree, every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied, `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record, commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
119
-
120
- **Per-turn cost.** A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly **2 to 3× your metric command's runtime** plus about 40ms of worktree overhead. Point the **per-turn** gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for **CI**: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
121
-
122
- ## Loop patterns
123
-
124
- PromptWheel is the gate *inside* a loop you don't have to write:
125
-
126
- ```bash
127
- # converge: keep spinning while each turn earns its keep; stop on plateau (3) or regression (1)
128
- while npx promptwheel improve --attempt "claude -p 'speed up the hot path'"; do :; done
129
-
130
- # read-only signal inside a driver you control (e.g. Claude Code /loop): gate without committing
131
- npx promptwheel run --working --json # branch on .verdict / per-metric .status
132
- ```
133
-
134
- The exit code is the contract, `0` kept · `1` regression · `3` plateau, so any driver (`/loop`, a Ralph `while`, a Beads pull-loop) converges without parsing anything. PromptWheel never drives the loop; it only says whether the turn counted.
135
-
136
- **Two callers, by design.** Your *agent* consumes the verdict as the loop's per-turn reward (`improve` / `run --working --json`; exit `0` kept · `1`/`2` reverted · `3` plateau). The *harness* (a Stop-hook or CI) runs `--detect-gaming`, the audit the agent **can't self-clear**, because the check never grades itself; it re-proves from the source alone (see [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md)). Same tool; *who calls it* is the difference between a reward and an audit.
137
-
138
- ## In Claude Code, plugin
139
-
140
- Bring the gate into Claude Code as slash commands:
141
-
142
- ```
143
- /plugin marketplace add promptwheel-ai/promptwheel
144
- /plugin install promptwheel@promptwheel-ai
145
- ```
146
-
147
- Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommitted changes) · `/promptwheel:improve <cmd>` (keep-if-improved) · `/promptwheel:insights`. The plugin wraps the CLI, so install that too (`npm i -g promptwheel`). See [`plugins/promptwheel/`](plugins/promptwheel/).
148
-
149
- ## In CI, GitHub Action
150
-
151
- Drop this in your repo (it posts a verdict comment on every PR and fails the check on a guarded regression beyond noise):
152
-
153
- ```yaml
154
- # .github/workflows/promptwheel.yml
155
- name: PromptWheel
156
- on: pull_request
157
- permissions: { contents: read, pull-requests: write }
158
- jobs:
159
- outcome-gate:
160
- runs-on: ubuntu-latest
161
- steps:
162
- - uses: actions/checkout@v4
163
- with: { fetch-depth: 0 }
164
- - uses: promptwheel-ai/promptwheel@v0
165
- with: { repeat: '3' }
166
- ```
167
-
168
- The Action runs straight from its own checkout, no npm install, no build. See [`action.yml`](action.yml).
77
+ **Two callers, by design.** Your *agent* consumes the verdict as a per-turn signal (`improve`, or `run --working --json`). The *harness* (a Stop-hook, or CI) runs the same check as an audit the agent cannot self-clear, because the check re-proves from the source alone and never asks the code to grade itself. See [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) for the three places to enforce it.
169
78
 
170
79
  ## Config, `promptwheel.config.json`
171
80
 
81
+ `run` works with no config (it gates your detected test command). A config says *what* to measure:
82
+
172
83
  ```json
173
84
  {
174
85
  "repeat": 1,
@@ -184,22 +95,19 @@ The Action runs straight from its own checkout, no npm install, no build. See [`
184
95
  - **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
185
96
  - **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
186
97
  - **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
187
- - **gamingCheck**, `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
98
+ - **gamingCheck**, `false` exempts a metric from the source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
188
99
  - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
100
+ - **evaluatorPaths**, globs for files that *define* your gate (a threshold file, a budget, a CI limit). Editing one in the same diff that improves a metric is `GAMED` by declaration, deterministically. Opt-in, because only you know which files are the goalposts.
189
101
 
190
- Dependency & environment keys (config-level; `init` writes stack defaults):
102
+ Dependency and environment keys (config-level; `init` writes stack defaults):
191
103
 
192
- - **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python, `["target"]` for Rust. (`"linkNodeModules": false` still works as shorthand for "link nothing".)
104
+ - **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python and `[]` for Rust (`cargo test` writes to `target/`, so it is built fresh in the worktree rather than shared across refs). `"linkNodeModules": false` still works as shorthand for "link nothing".
193
105
  - **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
194
- - **setup**, an optional command run once per ref before metrics (e.g. `npm run build`, `pip install -e .`), after any source patch, so a build-gated suite measures the ref it's actually on.
106
+ - **setup**, an optional command run once per ref before metrics (for example `npm run build` or `pip install -e .`), after any source patch, so a build-gated suite measures the ref it is actually on.
195
107
 
196
- ## Guardrails & inheritance
108
+ Copy a config for your stack (pytest, tsc, coverage, bundle size, LLM eval) from [examples/](examples/).
197
109
 
198
- To see what's actually enforced in a repo, including guards inherited from a shared config:
199
-
200
- ```bash
201
- promptwheel guards
202
- ```
110
+ ### Guardrails and inheritance
203
111
 
204
112
  Teams keep one shared base config and have every repo inherit it via `extends`:
205
113
 
@@ -209,13 +117,11 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
209
117
  "metrics": [ { "name": "cost_per_run_usd", "guard": false } ] } // loosen one inherited guard, locally
210
118
  ```
211
119
 
212
- `extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance, `inherited ← base.json`, `local`, or `local override`, plus each guard's flag record from the outcome stream.
120
+ `extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance (`inherited ← base.json`, `local`, or `local override`) plus each guard's flag record from the outcome record. Read this way, `extends` is the set of shared invariants your agents are held to, enforced as measured guards rather than advisory text. Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.
213
121
 
214
- Read this way, `extends` is **the shared invariants, the "business rules", your agents inherit and are held to**: enforced as *measured guards* (a trusted regression fails the gate / reverts the commit), not advisory text an agent can quietly ignore. (Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.)
122
+ ### Mandates (protected paths)
215
123
 
216
- ## Mandates (protected paths)
217
-
218
- Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*, *do not edit the billing schema*. Declare those as `mandates`, and any change that touches a protected path **fails the gate before a single metric is read**, however green everything else is:
124
+ Some rules are not a metric at all, they are a hard line: *never touch `src/auth/**`*, *the migration files are frozen*. Declare those as `mandates`, and any change that touches a protected path **fails the gate before a single metric is read**, however green everything else is:
219
125
 
220
126
  ```json
221
127
  // promptwheel.config.json
@@ -225,62 +131,70 @@ Some rules are not a metric at all, they are a hard line: *never touch `src/auth
225
131
  "metrics": [ /* ... */ ] }
226
132
  ```
227
133
 
228
- A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint, so quietly deleting a mandate to pass is itself a tracked, tamper-evident change. This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate *will not let the change past the gate*.
134
+ A mandate is deterministic (a path match on the diff, zero-LLM), rides at the top of the report, and **inherits through `extends`** so a parent scope's hard rule holds for every repo beneath it. It is folded into the gate fingerprint at both refs, so quietly deleting a mandate changes the fingerprint: the report records `gate.changed: true` and prints a "gate definition changed" line, a tracked, tamper-evident change. And deleting it does not clear it: the mandates in force are the union of the base ref's and the head's, so a change that removes a mandate and touches its path in the same diff still fails (reported with `removed: true`). Retire a mandate in its own change, with the protected path untouched. (Before 0.7.5 the base-ref fingerprint did not include mandates, so this was recorded backwards: an unchanged mandate read as a changed gate and a deleted one did not, and the deletion cleared the rule.) This is the line between a contract and a request: `AGENTS.md` can *ask* an agent not to touch auth; a mandate will not let the change past the gate.
135
+
136
+ ## In CI and in Claude Code
137
+
138
+ **GitHub Action.** Drop this in your repo; it posts a verdict comment on every PR and fails the check on a guarded regression, a gamed win, or an inconclusive gate:
139
+
140
+ ```yaml
141
+ # .github/workflows/promptwheel.yml
142
+ name: PromptWheel
143
+ on: pull_request
144
+ permissions: { contents: read, pull-requests: write }
145
+ jobs:
146
+ outcome-gate:
147
+ runs-on: ubuntu-latest
148
+ steps:
149
+ - uses: actions/checkout@v4
150
+ with: { fetch-depth: 0 }
151
+ - uses: promptwheel-ai/promptwheel@v0
152
+ with: { repeat: '3' }
153
+ ```
154
+
155
+ The Action runs straight from its own checkout, no npm install, no build. See [`action.yml`](action.yml); a copy-paste workflow file lives at [`examples/github-action.yml`](examples/github-action.yml). Make the check a branch-protection requirement and the agent cannot merge past it.
229
156
 
230
- ## Trust model, the point of the whole thing
157
+ **Claude Code plugin.** Bring the gate in as slash commands (the plugin wraps the CLI, so install that too: `npm i -g promptwheel`):
231
158
 
232
- A number that jumps around between runs is worthless as a signal. PromptWheel won't pretend otherwise:
159
+ ```
160
+ /plugin marketplace add promptwheel-ai/promptwheel
161
+ /plugin install promptwheel@promptwheel-ai
162
+ ```
163
+
164
+ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommitted changes) · `/promptwheel:improve <cmd>` (keep-if-improved) · `/promptwheel:insights` · `/promptwheel:logbook`. See [`plugins/promptwheel/`](plugins/promptwheel/). A Stop-hook that refuses to let an agent end a turn on a `GAMED` win ships at [`examples/detect-gaming-stop.sh`](examples/detect-gaming-stop.sh).
165
+
166
+ ## Trust model
167
+
168
+ **What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
169
+
170
+ - **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
171
+ - **Gate integrity.** The gate is resolved as of the base commit too. A change that disables a guard or deletes a guarded metric in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed. The gate definition (metrics, guards, `evaluatorPaths`, `gamingThreshold`, `mandates`) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed.
172
+ - **A sanctioned override.** When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
173
+ - **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
233
174
 
234
- - `--repeat N` measures each metric N times at both refs and uses the **median**; the **noise band** is the observed spread.
235
- - A delta **inside the noise band** is reported `inconclusive` with `low` confidence and **does not fail a guard** (no flaky CI failures).
236
- - **Confidence:** `high` (deterministic extract, or zero observed noise) · `medium` (delta clears the noise band) · `low` (delta inside noise) · `unverified` (single read, run `--repeat` to earn trust).
175
+ **The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test): that needs held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
237
176
 
238
- The accumulated record of **which change-types move which metrics** is the asset: a per-repo reward signal a base tool can't replicate, and the spine that lets an agent loop learn what actually helps.
177
+ **Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guard whose command never ran (same value at both refs) makes the verdict `inconclusive`, never a fake pass.
239
178
 
240
- ## Where PromptWheel fits (and where it doesn't)
179
+ **When not to use it.** If you only care that tests pass and nobody could be editing the tests, your plain CI already has you covered. PromptWheel earns its place when the thing writing the code can also touch the thing judging it, or when you have a graded metric beyond pass/fail (an eval score, cost per run, latency, size) that a change could quietly move. Single-metric CI gates (benchmark trackers, size limits) go deeper on one number; wrap them as a metric `cmd` and PromptWheel composes them into one verdict.
241
180
 
242
- - **vs single-axis CI gates** (Codspeed, Bencher, size-limit, Lighthouse-CI): they own deep statistics on *one* metric; PromptWheel is the **cross-metric gate that composes them**, "did `eval_pass_rate` **and** cost improve without regressing the guards?" in one verdict. Wrap any of them as a metric `cmd` and let `--repeat` handle the noise.
243
- - **vs loop/agent frameworks** (Ralph, GEPA, reward models): PromptWheel is the **execution-grounded reward they lack**, it runs your real suite with zero deps; it does not drive the loop or do test-time search.
244
- - **When NOT to use it:** if you only care that tests pass, your base verifier already has you covered. PromptWheel earns its place when you have a **graded numeric metric** beyond pass/fail (eval score, $/run, latency, size) that a change could quietly move.
181
+ ## Per-turn cost
182
+
183
+ A check runs your metric commands in two throwaway worktrees (before and after) plus a source-only re-run when a metric improved, so one check takes roughly 2 to 3× your metric command's runtime plus about 40ms of worktree overhead. Point the per-turn gate at a *fast* metric (a lint count, a quick test subset, a build-size check) and leave the slow full suite for CI: a 50ms check is invisible in a loop, a two-minute one is not. A deterministic metric (a test exit, a count) is accurate at the default `repeat: 1`; only a noisy metric (latency, timing) needs `repeat: 3` or more, which multiplies the cost.
184
+
185
+ **Footprint.** It never touches your working tree: every measurement runs in a throwaway git worktree in your system temp dir (one at a time; configured `linkDirs` are symlinked, not copied), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record; commit it, `.gitignore` it, or pass `--no-record`. A hard-killed run cannot leave clutter behind: the next run self-heals any orphaned worktree.
245
186
 
246
187
  ## Portability
247
188
 
248
- The core gamed-win check is **git only** and runs anywhere git does: it classifies the files a win touched (`git diff --name-only`) to separate production source from test, config, and grader files, so a green that came only from editing the goalposts is caught the same on Linux, macOS, and Windows (the `gamed-catch` CI job proves this on all three). The starter tripwire metrics that `init` writes (`test_count`, `assertions`, `suppressions`, `skipped_tests`) are `grep` one-liners that assume a GNU-grep-compatible shell, which PromptWheel runs through the system shell (`sh` on Linux/macOS, `cmd` on native Windows). Where grep is absent, those metrics read **unmeasurable**, and an unmeasurable guard **fails closed** (verdict `FAIL`), so a gamed green is still caught, never passed, it just lands as `FAIL` rather than the dedicated `GAMED`. Run under WSL or git-bash, or replace those tripwire `cmd`s with your toolchain's own counters, for a clean verdict on Windows.
189
+ The core gamed-win check is git only and runs anywhere git does: it classifies the files a win touched (`git diff --name-only`) to separate production source from test, config, and grader files, so a green that came only from editing the goalposts is caught the same on Linux, macOS, and Windows (the `gamed-catch` CI job proves this on all three). The starter tripwire metrics that `init` writes are `grep` one-liners that assume a GNU-grep-compatible shell, which PromptWheel runs through the system shell (`sh` on Linux/macOS, `cmd` on native Windows). Where grep is absent, those metrics read unmeasurable, and an unmeasurable guard fails closed (verdict `FAIL`), so a gamed green is still caught, never passed; it just lands as `FAIL` rather than the dedicated `GAMED`. Run under WSL or git-bash, or replace those tripwire `cmd`s with your toolchain's own counters, for a clean verdict on Windows.
249
190
 
250
191
  ## Docs
251
192
 
252
- - [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md), how `--detect-gaming` fits as the **deterministic layer** alongside held-out tests, LLM judges, and human review: the coverage matrix, the compose-as-a-pipeline model, and the honest in/out-of-scope boundary.
253
- - [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md), making "the agent can't skip it" real: the three places to enforce (loop-revert · CI + branch protection · a tested Claude Code Stop-hook), the exact wiring, and the *protect-the-gate's-own-config* caveat.
254
- - [docs/VISION.md](docs/VISION.md), why we pivoted from orchestrator to outcome gate, the thesis, the moat, the open-core model.
255
- - [docs/ROADMAP.md](docs/ROADMAP.md), the phased plan and the ship-now/stay-thin guardrails.
193
+ - [INTEGRATE.md](INTEGRATE.md), the integration surface: the exit-code and verdict-JSON contract, shelling `--json`, `import { check }`, and `promptwheel conformance` for self-checking a third-party implementation.
194
+ - [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md), how the source-only re-run fits as the deterministic layer alongside held-out tests, LLM judges, and human review: the coverage matrix and the honest in/out-of-scope boundary.
195
+ - [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md), making "the agent can't skip it" real: the three places to enforce (loop-revert, CI + branch protection, a Claude Code Stop-hook), the exact wiring, and the protect-the-gate's-own-config caveat.
256
196
  - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), how the engine works: schemas, extract modes, the trust/noise model.
257
- - [docs/LEARNING.md](docs/LEARNING.md), the (research-gated) Phase-5 design: outcome-curated playbook (Agentic Context Engineering, Stanford 2510.04618) + UCB work-discovery.
258
- - [CLAUDE.md](CLAUDE.md), the constitution for anyone (human or agent) working in this repo.
259
-
260
- ## Develop
261
-
262
- ```bash
263
- npm test # the dep-free suite (node:test), unit + integration, no dependencies
264
- ```
197
+ - [CHANGELOG.md](CHANGELOG.md), what shipped in each version.
198
+ - [CLAUDE.md](CLAUDE.md), the working notes for anyone (human or agent) changing this repo.
265
199
 
266
- The engine is one importable file; pure helpers are exported for unit tests, the CLI runs only when invoked directly. Add a test with every behavior change.
267
-
268
- ## Roadmap
269
-
270
- - [x] before/after worktree measurement + regression guards
271
- - [x] noise band + confidence (don't trust a delta inside the jitter)
272
- - [x] `--working` mode, measure uncommitted changes (tracked **and** untracked)
273
- - [x] persisted reward stream (`.promptwheel/outcomes.jsonl`), the compounding "what moves what" record
274
- - [x] GitHub Action / PR-comment wrapper (open-core distribution surface)
275
- - [x] agent loop: `improve`, propose → gate → keep only if a metric improved
276
- - [x] loop-consumable `improve`: exit `0` kept / `1` regression / `3` plateau + `--json result`
277
- - [x] `promptwheel init` + presets, zero-config onboarding
278
- - [x] `insights`, reward-stream aggregation (Phase-5 seed)
279
- - [x] `--detect-gaming`, reward-hack detection: re-prove the win from source edits alone + `antihack` preset
280
- - [x] npm publish, `promptwheel@0.4.2` (the lead magnet)
281
- - [x] outcome-curated learning + UCB work-discovery, `playbook` + `suggest` + the compounding A/B harness (**experimental**; compounding *claims* stay gated on real-data proof, see [docs/LEARNING.md](docs/LEARNING.md))
282
- - [x] gate integrity: a win that disables a guard or deletes a guarded metric in the same diff reads `GAMED` (SHA-256 gate fingerprint; 0.5.0)
283
- - [x] zero-config: `run` and `check()` auto-detect the gate with no committed config, `init` becomes optional (0.6.0)
284
- - [x] declared override (`Sawbuck-Allow:` trailer / `--allow`) + `init` build-setup detection, so a build-gated test measures instead of reading inert (0.7.0)
285
-
286
- > Status: **published** (npm `promptwheel`, v0.7.1), all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
200
+ Develop: `npm test` runs the dep-free suite (`node:test`, unit + integration). The engine is one importable file; pure helpers are exported for unit tests and the CLI runs only when invoked directly. Add a test with every behavior change.