promptwheel 0.4.5 → 0.4.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +57 -57
  2. package/bin/promptwheel.mjs +143 -88
  3. package/package.json +2 -2
package/README.md CHANGED
@@ -4,17 +4,17 @@
4
4
 
5
5
  **A deterministic check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel tells the difference: it re-proves every win from the source changes alone, with no LLM, so a win that only survived by editing the test reads `GAMED`. Un-gameable by construction, because the code being checked never gets to grade itself.
6
6
 
7
- PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop — a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
7
+ PromptWheel re-proves every "win" using the agent's **source edits alone**. If the gate only went green because the agent edited the test, mocked the grader, suppressed the error (`@ts-ignore` / `eslint-disable`), or deleted the feature, the win evaporates when those edits are reverted → **`VERDICT GAMED`, exit 2**. No LLM in the loop, a diff partition plus a re-run, so every flag is reproducible in seconds with a human-readable reason.
8
8
 
9
- It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"* — PromptWheel also asks *"did the agent **earn** it?"*
9
+ It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"*, PromptWheel also asks *"did the agent **earn** it?"*
10
10
 
11
- PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question — *did this turn earn its keep, and did the agent earn it honestly?* — so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
11
+ PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question, *did this turn earn its keep, and did the agent earn it honestly?*, so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
12
12
 
13
13
  PromptWheel is the reference check for the [Verified-Outcome Contract](https://github.com/sawbuck-ai/sawbuck/blob/master/CONTRACT.md), the open interface for a verifiable coding loop.
14
14
 
15
- > Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate** — making the reward signal one you can't cheat.
15
+ > Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate**, making the reward signal one you can't cheat.
16
16
 
17
- AI coding agents (and humans) declare success when a change *compiles and tests pass*. They almost never verify that a change **improved a real measurable thing without regressing another** — and outcome data rots in dashboards (DORA, Swarmia) disconnected from the change that caused it. PromptWheel closes that loop.
17
+ AI coding agents (and humans) declare success when a change *compiles and tests pass*. They almost never verify that a change **improved a real measurable thing without regressing another**, and outcome data rots in dashboards (DORA, Swarmia) disconnected from the change that caused it. PromptWheel closes that loop.
18
18
 
19
19
  For any change (`base` → `head`), it measures each configured metric in an **isolated git worktree** before and after, **refuses to trust a delta inside the measurement noise band**, enforces **regression guards**, and emits a structured verdict:
20
20
 
@@ -29,41 +29,41 @@ PromptWheel a1b2c3d → e4f5g6h (×5)
29
29
  VERDICT: PASS
30
30
  ```
31
31
 
32
- Exit `0` pass · `1` fail · `2` gamed · `3` inconclusive (a guard measured nothing) — CI-friendly. No build step, zero dependencies, Node 18+.
32
+ Exit `0` pass · `1` fail · `2` gamed · `3` inconclusive (a guard measured nothing), CI-friendly. No build step, zero dependencies, Node 18+.
33
33
 
34
- ## Catch your agent cheating — on by default
34
+ ## Catch your agent cheating, on by default
35
35
 
36
- The headline feature, and it runs **by default** (in both `run` and `improve`; pass `--no-detect-gaming` for the bare outcome gate). After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive — because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files — the verdict is **`GAMED` (exit 2)**:
36
+ The headline feature, and it runs **by default** (in both `run` and `improve`; pass `--no-detect-gaming` for the bare outcome gate). After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive, because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files, the verdict is **`GAMED` (exit 2)**:
37
37
 
38
38
  ```
39
39
  $ promptwheel run
40
40
 
41
41
  PromptWheel base → head
42
42
  ▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
43
- 🚩 GAMED — the "win" changed zero production-source files — only test/config/grader/golden
44
- VERDICT: GAMED — a metric "improved" by editing the goalposts, not the source
43
+ 🚩 GAMED, the "win" changed zero production-source files, only test/config/grader/golden
44
+ VERDICT: GAMED, a metric "improved" by editing the goalposts, not the source
45
45
  ```
46
46
 
47
- **See it fire yourself** — 60 seconds, illustrative (a hand-built cheat is a *mechanism* demo, not evidence):
47
+ **See it fire yourself**, 60 seconds, illustrative (a hand-built cheat is a *mechanism* demo, not evidence):
48
48
 
49
49
  ```bash
50
50
  mkdir pw-cheat-demo && cd pw-cheat-demo && git init -q
51
51
  printf '{"name":"demo","type":"module","scripts":{"test":"node --test"}}\n' > package.json
52
52
  printf 'export const add = (a, b) => a - b; // BUG: should be a + b\n' > add.js
53
- printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), 5)); // honest test — FAILS on the bug\n" > add.test.js
53
+ printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), 5)); // honest test, FAILS on the bug\n" > add.test.js
54
54
  git add -A && git commit -qm 'buggy code + an honest failing test'
55
55
  npx -y promptwheel@latest init && git add -A && git commit -qm 'add the gate config'
56
56
 
57
- # an agent "greens" the suite by editing the TEST to expect the bug — the code stays broken:
57
+ # an agent "greens" the suite by editing the TEST to expect the bug, the code stays broken:
58
58
  printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), -1));\n" > add.test.js
59
59
  git commit -qam 'make the suite green'
60
60
 
61
61
  npx -y promptwheel@latest run --base HEAD~1 --head HEAD
62
- # ▲ tests_pass 0 → 1 🚩 GAMED — the win changed zero production-source files
62
+ # ▲ tests_pass 0 → 1 🚩 GAMED, the win changed zero production-source files
63
63
  # VERDICT: GAMED (exit 2)
64
64
  ```
65
65
 
66
- Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) — and the quieter cheat of *weakening the suite while the metric stays flat* — are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
66
+ Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`), and the quieter cheat of *weakening the suite while the metric stays flat*, are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
67
67
 
68
68
  ```bash
69
69
  npx promptwheel init # guarded tests + antihack tripwires by default
@@ -73,7 +73,7 @@ npx promptwheel run # detection ON by default · exit 0 wi
73
73
 
74
74
  Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable via `gamingThreshold` (config-level, or per-metric).
75
75
 
76
- This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers — held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage — are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce — it includes a cross-stack table with a real pytest run + a numeric eval-pass-rate metric). **Gate your own stack** — pytest · tsc · coverage · bundle · llm-eval — copy a config from **[examples/](examples/)**.
76
+ This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers, held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage, are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce, it includes a cross-stack table with a real pytest run + a numeric eval-pass-rate metric). **Gate your own stack**, pytest · tsc · coverage · bundle · llm-eval, copy a config from **[examples/](examples/)**.
77
77
 
78
78
  ## Use
79
79
 
@@ -96,13 +96,13 @@ npx promptwheel insights
96
96
 
97
97
  # EXPERIMENTAL (Phase 5): the earned playbook + where the next attempt should go
98
98
  npx promptwheel playbook # decayed, evidence-gated claims distilled from the record
99
- npx promptwheel suggest # UCB over the lever scores — proven levers vs under-explored arms
99
+ npx promptwheel suggest # UCB over the lever scores, proven levers vs under-explored arms
100
100
  npx promptwheel backfill -n 30 # cold start: seed the record from git history (cohort-tagged, commit types → labels)
101
101
  ```
102
102
 
103
- **The consequence ledger.** git records *what changed*; PromptWheel records *what the change did* — same trust model (local, deterministic, append-only, no server, no LLM in the verdict). `playbook` and `suggest` are pure re-derivations over that ledger: every rendered line was measured by the gate, decays unless re-earned, and stays hidden below an evidence threshold. No compounding claim is made for them until the A/B acceptance test (`bench/compounding-ab.mjs`) passes on real usage data.
103
+ **The consequence ledger.** git records *what changed*; PromptWheel records *what the change did*, same trust model (local, deterministic, append-only, no server, no LLM in the verdict). `playbook` and `suggest` are pure re-derivations over that ledger: every rendered line was measured by the gate, decays unless re-earned, and stays hidden below an evidence threshold. No compounding claim is made for them until the A/B acceptance test (`bench/compounding-ab.mjs`) passes on real usage data.
104
104
 
105
- **Footprint:** it never touches your working tree — every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied — `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record — commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
105
+ **Footprint:** it never touches your working tree, every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied, `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record, commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
106
106
 
107
107
  ## Loop patterns
108
108
 
@@ -116,11 +116,11 @@ while npx promptwheel improve --attempt "claude -p 'speed up the hot path'"; do
116
116
  npx promptwheel run --working --json # branch on .verdict / per-metric .status
117
117
  ```
118
118
 
119
- The exit code is the contract — `0` kept · `1` regression · `3` plateau — so any driver (`/loop`, a Ralph `while`, a Beads pull-loop) converges without parsing anything. PromptWheel never drives the loop; it only says whether the turn counted.
119
+ The exit code is the contract, `0` kept · `1` regression · `3` plateau, so any driver (`/loop`, a Ralph `while`, a Beads pull-loop) converges without parsing anything. PromptWheel never drives the loop; it only says whether the turn counted.
120
120
 
121
121
  **Two callers, by design.** Your *agent* consumes the verdict as the loop's per-turn reward (`improve` / `run --working --json`; exit `0` kept · `1`/`2` reverted · `3` plateau). The *harness* (a Stop-hook or CI) runs `--detect-gaming`, the audit the agent **can't self-clear**, because the check never grades itself; it re-proves from the source alone (see [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md)). Same tool; *who calls it* is the difference between a reward and an audit.
122
122
 
123
- ## In Claude Code — plugin
123
+ ## In Claude Code, plugin
124
124
 
125
125
  Bring the gate into Claude Code as slash commands:
126
126
 
@@ -131,7 +131,7 @@ Bring the gate into Claude Code as slash commands:
131
131
 
132
132
  Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommitted changes) · `/promptwheel:improve <cmd>` (keep-if-improved) · `/promptwheel:insights`. The plugin wraps the CLI, so install that too (`npm i -g promptwheel`). See [`plugins/promptwheel/`](plugins/promptwheel/).
133
133
 
134
- ## In CI — GitHub Action
134
+ ## In CI, GitHub Action
135
135
 
136
136
  Drop this in your repo (it posts a verdict comment on every PR and fails the check on a guarded regression beyond noise):
137
137
 
@@ -150,9 +150,9 @@ jobs:
150
150
  with: { repeat: '3' }
151
151
  ```
152
152
 
153
- The Action runs straight from its own checkout — no npm install, no build. See [`action.yml`](action.yml).
153
+ The Action runs straight from its own checkout, no npm install, no build. See [`action.yml`](action.yml).
154
154
 
155
- ## Config — `promptwheel.config.json`
155
+ ## Config, `promptwheel.config.json`
156
156
 
157
157
  ```json
158
158
  {
@@ -165,22 +165,22 @@ The Action runs straight from its own checkout — no npm install, no build. See
165
165
  }
166
166
  ```
167
167
 
168
- - **cmd** — any shell command, run inside the worktree.
169
- - **extract** — reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
170
- - **direction** — `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
171
- - **guard** — `true` = a *trusted* regression **fails** the gate; `false` = informational.
172
- - **gamingCheck** — `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files — otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
173
- - **gamingThreshold** — the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
168
+ - **cmd**, any shell command, run inside the worktree.
169
+ - **extract**, reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
170
+ - **direction**, `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
171
+ - **guard**, `true` = a *trusted* regression **fails** the gate; `false` = informational.
172
+ - **gamingCheck**, `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files, otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
173
+ - **gamingThreshold**, the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
174
174
 
175
175
  Dependency & environment keys (config-level; `init` writes stack defaults):
176
176
 
177
- - **linkDirs** — dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python, `["target"]` for Rust. (`"linkNodeModules": false` still works as shorthand for "link nothing".)
178
- - **env** — environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
179
- - **setup** — an optional command run once per ref before metrics (e.g. `npm run build`, `pip install -e .`), after any source patch — so a build-gated suite measures the ref it's actually on.
177
+ - **linkDirs**, dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python, `["target"]` for Rust. (`"linkNodeModules": false` still works as shorthand for "link nothing".)
178
+ - **env**, environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
179
+ - **setup**, an optional command run once per ref before metrics (e.g. `npm run build`, `pip install -e .`), after any source patch, so a build-gated suite measures the ref it's actually on.
180
180
 
181
181
  ## Guardrails & inheritance
182
182
 
183
- To see what's actually enforced in a repo — including guards inherited from a shared config:
183
+ To see what's actually enforced in a repo, including guards inherited from a shared config:
184
184
 
185
185
  ```bash
186
186
  promptwheel guards
@@ -194,40 +194,40 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
194
194
  "metrics": [ { "name": "cost_per_run_usd", "guard": false } ] } // loosen one inherited guard, locally
195
195
  ```
196
196
 
197
- `extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance — `inherited ← base.json`, `local`, or `local override` — plus each guard's flag record from the outcome stream.
197
+ `extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance, `inherited ← base.json`, `local`, or `local override`, plus each guard's flag record from the outcome stream.
198
198
 
199
- Read this way, `extends` is **the shared invariants — the "business rules" — your agents inherit and are held to**: enforced as *measured guards* (a trusted regression fails the gate / reverts the commit), not advisory text an agent can quietly ignore. (Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.)
199
+ Read this way, `extends` is **the shared invariants, the "business rules", your agents inherit and are held to**: enforced as *measured guards* (a trusted regression fails the gate / reverts the commit), not advisory text an agent can quietly ignore. (Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.)
200
200
 
201
- ## Trust model — the point of the whole thing
201
+ ## Trust model, the point of the whole thing
202
202
 
203
203
  A number that jumps around between runs is worthless as a signal. PromptWheel won't pretend otherwise:
204
204
 
205
205
  - `--repeat N` measures each metric N times at both refs and uses the **median**; the **noise band** is the observed spread.
206
206
  - A delta **inside the noise band** is reported `inconclusive` with `low` confidence and **does not fail a guard** (no flaky CI failures).
207
- - **Confidence:** `high` (deterministic extract, or zero observed noise) · `medium` (delta clears the noise band) · `low` (delta inside noise) · `unverified` (single read — run `--repeat` to earn trust).
207
+ - **Confidence:** `high` (deterministic extract, or zero observed noise) · `medium` (delta clears the noise band) · `low` (delta inside noise) · `unverified` (single read, run `--repeat` to earn trust).
208
208
 
209
209
  The accumulated record of **which change-types move which metrics** is the asset: a per-repo reward signal a base tool can't replicate, and the spine that lets an agent loop learn what actually helps.
210
210
 
211
211
  ## Where PromptWheel fits (and where it doesn't)
212
212
 
213
- - **vs single-axis CI gates** (Codspeed, Bencher, size-limit, Lighthouse-CI): they own deep statistics on *one* metric; PromptWheel is the **cross-metric gate that composes them** — "did `eval_pass_rate` **and** cost improve without regressing the guards?" in one verdict. Wrap any of them as a metric `cmd` and let `--repeat` handle the noise.
214
- - **vs loop/agent frameworks** (Ralph, GEPA, reward models): PromptWheel is the **execution-grounded reward they lack** — it runs your real suite with zero deps; it does not drive the loop or do test-time search.
213
+ - **vs single-axis CI gates** (Codspeed, Bencher, size-limit, Lighthouse-CI): they own deep statistics on *one* metric; PromptWheel is the **cross-metric gate that composes them**, "did `eval_pass_rate` **and** cost improve without regressing the guards?" in one verdict. Wrap any of them as a metric `cmd` and let `--repeat` handle the noise.
214
+ - **vs loop/agent frameworks** (Ralph, GEPA, reward models): PromptWheel is the **execution-grounded reward they lack**, it runs your real suite with zero deps; it does not drive the loop or do test-time search.
215
215
  - **When NOT to use it:** if you only care that tests pass, your base verifier already has you covered. PromptWheel earns its place when you have a **graded numeric metric** beyond pass/fail (eval score, $/run, latency, size) that a change could quietly move.
216
216
 
217
217
  ## Docs
218
218
 
219
- - [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md) — how `--detect-gaming` fits as the **deterministic layer** alongside held-out tests, LLM judges, and human review: the coverage matrix, the compose-as-a-pipeline model, and the honest in/out-of-scope boundary.
220
- - [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) — making "the agent can't skip it" real: the three places to enforce (loop-revert · CI + branch protection · a tested Claude Code Stop-hook), the exact wiring, and the *protect-the-gate's-own-config* caveat.
221
- - [docs/VISION.md](docs/VISION.md) — why we pivoted from orchestrator to outcome gate, the thesis, the moat, the open-core model.
222
- - [docs/ROADMAP.md](docs/ROADMAP.md) — the phased plan and the ship-now/stay-thin guardrails.
223
- - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how the engine works: schemas, extract modes, the trust/noise model.
224
- - [docs/LEARNING.md](docs/LEARNING.md) — the (research-gated) Phase-5 design: outcome-curated playbook (Agentic Context Engineering, Stanford 2510.04618) + UCB work-discovery.
225
- - [CLAUDE.md](CLAUDE.md) — the constitution for anyone (human or agent) working in this repo.
219
+ - [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md), how `--detect-gaming` fits as the **deterministic layer** alongside held-out tests, LLM judges, and human review: the coverage matrix, the compose-as-a-pipeline model, and the honest in/out-of-scope boundary.
220
+ - [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md), making "the agent can't skip it" real: the three places to enforce (loop-revert · CI + branch protection · a tested Claude Code Stop-hook), the exact wiring, and the *protect-the-gate's-own-config* caveat.
221
+ - [docs/VISION.md](docs/VISION.md), why we pivoted from orchestrator to outcome gate, the thesis, the moat, the open-core model.
222
+ - [docs/ROADMAP.md](docs/ROADMAP.md), the phased plan and the ship-now/stay-thin guardrails.
223
+ - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), how the engine works: schemas, extract modes, the trust/noise model.
224
+ - [docs/LEARNING.md](docs/LEARNING.md), the (research-gated) Phase-5 design: outcome-curated playbook (Agentic Context Engineering, Stanford 2510.04618) + UCB work-discovery.
225
+ - [CLAUDE.md](CLAUDE.md), the constitution for anyone (human or agent) working in this repo.
226
226
 
227
227
  ## Develop
228
228
 
229
229
  ```bash
230
- npm test # the dep-free suite (node:test) — unit + integration, no dependencies
230
+ npm test # the dep-free suite (node:test), unit + integration, no dependencies
231
231
  ```
232
232
 
233
233
  The engine is one importable file; pure helpers are exported for unit tests, the CLI runs only when invoked directly. Add a test with every behavior change.
@@ -236,15 +236,15 @@ The engine is one importable file; pure helpers are exported for unit tests, the
236
236
 
237
237
  - [x] before/after worktree measurement + regression guards
238
238
  - [x] noise band + confidence (don't trust a delta inside the jitter)
239
- - [x] `--working` mode — measure uncommitted changes (tracked **and** untracked)
240
- - [x] persisted reward stream (`.promptwheel/outcomes.jsonl`) — the compounding "what moves what" record
239
+ - [x] `--working` mode, measure uncommitted changes (tracked **and** untracked)
240
+ - [x] persisted reward stream (`.promptwheel/outcomes.jsonl`), the compounding "what moves what" record
241
241
  - [x] GitHub Action / PR-comment wrapper (open-core distribution surface)
242
- - [x] agent loop: `improve` — propose → gate → keep only if a metric improved
242
+ - [x] agent loop: `improve`, propose → gate → keep only if a metric improved
243
243
  - [x] loop-consumable `improve`: exit `0` kept / `1` regression / `3` plateau + `--json result`
244
- - [x] `promptwheel init` + presets — zero-config onboarding
245
- - [x] `insights` — reward-stream aggregation (Phase-5 seed)
246
- - [x] `--detect-gaming` — reward-hack detection: re-prove the win from source edits alone + `antihack` preset
247
- - [x] npm publish — `promptwheel@0.4.2` (the lead magnet)
248
- - [x] outcome-curated learning + UCB work-discovery — `playbook` + `suggest` + the compounding A/B harness (**experimental**; compounding *claims* stay gated on real-data proof — see [docs/LEARNING.md](docs/LEARNING.md))
244
+ - [x] `promptwheel init` + presets, zero-config onboarding
245
+ - [x] `insights`, reward-stream aggregation (Phase-5 seed)
246
+ - [x] `--detect-gaming`, reward-hack detection: re-prove the win from source edits alone + `antihack` preset
247
+ - [x] npm publish, `promptwheel@0.4.2` (the lead magnet)
248
+ - [x] outcome-curated learning + UCB work-discovery, `playbook` + `suggest` + the compounding A/B harness (**experimental**; compounding *claims* stay gated on real-data proof, see [docs/LEARNING.md](docs/LEARNING.md))
249
249
 
250
- > Status: **published** (npm `promptwheel`, v0.4.2) — all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
250
+ > Status: **published** (npm `promptwheel`, v0.4.2), all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
package/bin/promptwheel.mjs CHANGED
@@ -1,11 +1,11 @@
1
1
  #!/usr/bin/env node
2
- // PromptWheel — the per-turn reward for AI coding loops (a.k.a. the outcome gate for AI code).
2
+ // PromptWheel, the per-turn reward for AI coding loops (a.k.a. the outcome gate for AI code).
3
3
  //
4
4
  // For any change (base ref → head ref), measure each configured metric in an
5
5
  // isolated git worktree BEFORE and AFTER, enforce regression guards, and refuse
6
6
  // to trust a delta that sits inside the measurement noise band. Emits a
7
7
  // structured {verdict, metric, delta, confidence}. The thing base agents don't do:
8
- // verify a real number moved — not just "the diff applied / tests passed".
8
+ // verify a real number moved, not just "the diff applied / tests passed".
9
9
  //
10
10
  // Zero deps, zero build. Node 18+.
11
11
  // promptwheel run [--base R] [--head R] [--repeat N] [--json]
@@ -51,22 +51,22 @@ function resolveConfig(p, repo, label, seen) {
51
51
  if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
52
52
  const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
53
53
  for (const m of base.metrics) byName.set(m.name, m);
54
- for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
54
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (base[k] !== undefined) scalars[k] = base[k];
55
55
  }
56
56
  for (const m of (cfg.metrics || [])) {
57
57
  byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
58
58
  }
59
- for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
59
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold', 'evaluatorPaths']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
60
60
  return { ...scalars, metrics: [...byName.values()] };
61
61
  } finally {
62
- seen.delete(abs); // only ACTIVE ancestors are a cycle — a diamond (a base reached two ways) is fine
62
+ seen.delete(abs); // only ACTIVE ancestors are a cycle, a diamond (a base reached two ways) is fine
63
63
  }
64
64
  }
65
65
 
66
66
  function loadConfig(repo) {
67
67
  const candidates = ['promptwheel.config.json', 'outcome-gate.config.json']; // back-compat alias
68
68
  const p = candidates.map((c) => join(repo, c)).find(existsSync);
69
- if (!p) { console.error('no promptwheel.config.json — run: promptwheel init (writes one for your stack)'); process.exit(2); }
69
+ if (!p) { console.error('no promptwheel.config.json, run: promptwheel init (writes one for your stack)'); process.exit(2); }
70
70
  const cfg = resolveConfig(p, repo, 'local', new Set());
71
71
  if (!Array.isArray(cfg.metrics) || cfg.metrics.length === 0) {
72
72
  console.error('config.metrics must be a non-empty array'); process.exit(2);
@@ -96,7 +96,7 @@ function runMetric(cwd, m, env) {
96
96
 
97
97
  // extract modes: 'number' (last number, default) | 'lines' | 'exit' | {regex}
98
98
  // A FAILED command is unmeasurable (null -> the metric goes inert, guards
99
- // fail loudly) — error text often contains numbers ("Node.js v22") and
99
+ // fail loudly), error text often contains numbers ("Node.js v22") and
100
100
  // silently measuring those produced identical junk on both refs = a false
101
101
  // green. 'exit' mode consumes nonzero by design; anything else needs the
102
102
  // per-metric opt-in `allowNonzero: true`.
@@ -118,7 +118,7 @@ function extract(stdout, code, mode, allowNonzero) {
118
118
  // node_modules-only symlink: `linkDirs` says what to link (default ['node_modules']; back-compat
119
119
  // with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
120
120
  // Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
121
- // not here — the engine stays ecosystem-agnostic.
121
+ // not here, the engine stays ecosystem-agnostic.
122
122
  function bridgeEnv(repo, wt, cfg) {
123
123
  const linkDirs = Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']);
124
124
  for (const d of linkDirs) {
@@ -169,9 +169,9 @@ function evaluate(m, beforeS, afterS, repeat) {
169
169
 
170
170
  let confidence;
171
171
  if (deterministic) confidence = 'high';
172
- else if (!sampledNoise) confidence = 'unverified'; // single read — noise unknown; run --repeat to trust
172
+ else if (!sampledNoise) confidence = 'unverified'; // single read, noise unknown; run --repeat to trust
173
173
  else if (noise === 0) confidence = 'high';
174
- else if (withinNoise) confidence = 'low'; // delta is inside the jitter — not real
174
+ else if (withinNoise) confidence = 'low'; // delta is inside the jitter, not real
175
175
  else confidence = 'medium';
176
176
 
177
177
  let improved = false, regressed = false;
@@ -186,7 +186,7 @@ function evaluate(m, beforeS, afterS, repeat) {
186
186
  // guards fail only on a TRUSTED regression (within-noise regressions are not failures)
187
187
  const ok = m.guard ? !regressed : true;
188
188
  // a guarded pass-metric that never passes is protecting NOTHING (broken test cmd,
189
- // missing script, failed install) — surface it instead of folding into a green verdict.
189
+ // missing script, failed install), surface it instead of folding into a green verdict.
190
190
  const inert = !!m.guard && dir === 'pass' && before === 0 && after === 0;
191
191
  return { before, after, delta, status, ok, confidence, noise, ...(inert ? { inert: true } : {}) };
192
192
  }
@@ -196,16 +196,27 @@ function evaluate(m, beforeS, afterS, repeat) {
196
196
  // If the improvement evaporates once test/config/grader/golden changes are reverted,
197
197
  // the agent moved the goalposts (edited the test, mocked the grader, suppressed the
198
198
  // lint rule, deleted the feature) instead of earning it. Pure arithmetic, fully
199
- // explainable — the thing the loop-owner structurally won't ship about its own agent.
199
+ // explainable, the thing the loop-owner structurally won't ship about its own agent.
200
200
  // ---------------------------------------------------------------------------
201
201
  const NON_SOURCE = [
202
202
  /(^|\/)(tests?|__tests?__|spec|e2e|fixtures|snapshots|__snapshots__|__mocks__)\//i,
203
203
  /\.(test|spec)\.[cm]?[jt]sx?$/i,
204
204
  /(^|\/)tests?\.[cm]?[jt]sx?$/i, // a file literally named test.js / tests.ts
205
205
  /(^|\/)test_[^/]*\.py$/i, /_test\.py$/i, /(^|\/)conftest\.py$/i,
206
+ // cross-language test-filename conventions, strong-signal tokens only (a dedicated
207
+ // _test / _spec / Test(s) suffix, or a file literally named test.<lang>), so production
208
+ // code like latest.c, Contest.java or speedtest.go is NOT swept into the evaluator slice.
209
+ // (Test DIRECTORIES are covered above for every language; this adds the by-filename
210
+ // conventions compiled langs use at repo root, a red-team finding: a weakened test.c /
211
+ // FooTest.java / handler_test.go was classified as source, so the cheat survived.)
212
+ /(^|\/)tests?\.(c|cc|cpp|cxx|go|rs|rb)$/i,
213
+ /(^|\/)test_[^/]*\.(c|cc|cpp|cxx|h|hpp|go|rs)$/i,
214
+ /_test\.(c|cc|cpp|cxx|h|hpp|go|rs|rb)$/i,
215
+ /_spec\.rb$/i,
216
+ /[a-z0-9](Test|Tests)\.(java|kt|kts|scala)$/,
206
217
  /\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
207
218
  /(^|\/)(eval|grader|score)[^/]*\.[cm]?[jt]s$/i,
208
- // config files only — require a .config/.conf/.setup segment so PRODUCTION source that
219
+ // config files only, require a .config/.conf/.setup segment so PRODUCTION source that
209
220
  // happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
210
221
  // the source slice (a false sweep here can flag an honest win as GAMED).
211
222
  /(^|\/)(jest|vitest|playwright|cypress|karma|babel|eslint|pytest)[^/]*\.(config|conf|setup)\.[^/]+$/i,
@@ -214,7 +225,7 @@ const NON_SOURCE = [
214
225
  /(^|\/)(pytest\.ini|setup\.cfg|\.flake8|tox\.ini|eslint\.config\.[cm]?js)$/i,
215
226
  // promptwheel's OWN evaluator surface: the config that defines the gate
216
227
  // and any pack copied into the repo by `init --preset`. Editing these is
217
- // moving the goalposts, never a production improvement — an agent that
228
+ // moving the goalposts, never a production improvement, an agent that
218
229
  // "improved" security_findings by deleting pattern rules must show 0%
219
230
  // retained in the source-only re-run, not 100%.
220
231
  /(^|\/)promptwheel\.config\.json$/i,
@@ -222,17 +233,41 @@ const NON_SOURCE = [
222
233
  ];
223
234
  const isNonSource = (p) => NON_SOURCE.some((re) => re.test(p));
224
235
 
225
- // split the agent's changed files into production-source vs test/config/grader/golden
226
- function changedSourcePaths(repo, base, head) {
236
+ // user-declared evaluator surface: extra paths that DEFINE the gate (a generic threshold
237
+ // file, a budget, a CI limit) and so must be reverted in the source-only re-run exactly
238
+ // like a test/grader. Opt-in per repo via `evaluatorPaths`, NOT guessed, because a bare
239
+ // name is a classifier risk; the user names the files only they know are the goalposts.
240
+ // Glob semantics: a bare name matches that basename at any depth; a pattern with "/" is a
241
+ // path glob anchored at repo root; "*" stops at "/", "**" crosses it, "?" is one char; a
242
+ // leading "/" anchors to the root explicitly.
243
+ function globToRe(glob) {
244
+ let g = glob; const anchored = g.startsWith('/'); if (anchored) g = g.slice(1);
245
+ const body = g.replace(/[.+^${}()|[\]\\]/g, '\\$&') // escape regex, keep * ? for globbing
246
+ .replace(/\*\*/g, '\x00').replace(/\*/g, '[^/]*').replace(/\?/g, '[^/]').replace(/\x00/g, '.*');
247
+ const pre = (anchored || glob.includes('/')) ? '^' : '(^|/)';
248
+ return new RegExp(`${pre}${body}$`);
249
+ }
250
+ function evaluatorMatcher(cfg) {
251
+ const pats = (cfg && Array.isArray(cfg.evaluatorPaths) ? cfg.evaluatorPaths : []).filter((s) => typeof s === 'string' && s.trim());
252
+ if (!pats.length) return () => false;
253
+ const res = pats.map(globToRe);
254
+ return (p) => res.some((re) => re.test(p));
255
+ }
256
+
257
+ // split the agent's changed files into production-source vs test/config/grader/golden.
258
+ // `isEval` is the repo's opt-in evaluator-path matcher, treated as non-source alongside
259
+ // the built-in classifier, so a cheat that only edits a declared gate file is caught.
260
+ function changedSourcePaths(repo, base, head, isEval = () => false) {
227
261
  const all = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean);
228
- return { source: all.filter((p) => !isNonSource(p)), nonSource: all.filter(isNonSource) };
262
+ const nonSrc = (p) => isNonSource(p) || isEval(p);
263
+ return { source: all.filter((p) => !nonSrc(p)), nonSource: all.filter(nonSrc) };
229
264
  }
230
265
 
231
266
  // coarse change-location fingerprint for the outcome record: top source dirs of the diff
232
- function subsystemsOf(repo, base, head) {
267
+ function subsystemsOf(repo, base, head, cfg) {
233
268
  try {
234
269
  const counts = {};
235
- for (const p of changedSourcePaths(repo, base, head).source) {
270
+ for (const p of changedSourcePaths(repo, base, head, evaluatorMatcher(cfg)).source) {
236
271
  const d = p.includes('/') ? p.slice(0, p.indexOf('/')) : '(root)';
237
272
  counts[d] = (counts[d] || 0) + 1;
238
273
  }
@@ -242,7 +277,7 @@ function subsystemsOf(repo, base, head) {
242
277
 
243
278
  // measure `metric` in a worktree at `base` with ONLY the source slice of base→head applied.
244
279
  function measureSourceOnly(repo, base, head, metric, cfg, repeat) {
245
- const { source } = changedSourcePaths(repo, base, head);
280
+ const { source } = changedSourcePaths(repo, base, head, evaluatorMatcher(cfg));
246
281
  if (!source.length) return { samples: [], hadSourceChange: false }; // a "win" with zero source edits = goalposts moved
247
282
  const wt = mkdtempSync(join(tmpdir(), 'promptwheel-src-'));
248
283
  git(['worktree', 'add', '--quiet', '--detach', wt, base], repo);
@@ -273,26 +308,26 @@ function gain(m, before, val) {
273
308
  }
274
309
 
275
310
  // a metric that improved full-diff: does the win survive when only source edits are applied?
276
- // gamed = source edits alone reproduce less than `threshold` of the gain (default 0.5 — the
311
+ // gamed = source edits alone reproduce less than `threshold` of the gain (default 0.5, the
277
312
  // rest came from editing the goalposts). Tune via `gamingThreshold` (config-level or per-metric).
278
313
  function judgeGaming(m, srcResult, threshold = 0.5) {
279
- if (!srcResult.hadSourceChange) return { gamed: true, sourceOnly: null, retained: 0, reason: 'the "win" changed zero production-source files — only test/config/grader/golden' };
280
- if (srcResult.applyFailed) return { gamed: null, sourceOnly: null, retained: null, reason: 'source-only patch did not apply cleanly — inconclusive' };
314
+ if (!srcResult.hadSourceChange) return { gamed: true, sourceOnly: null, retained: 0, reason: 'the "win" changed zero production-source files, only test/config/grader/golden' };
315
+ if (srcResult.applyFailed) return { gamed: null, sourceOnly: null, retained: null, reason: 'source-only patch did not apply cleanly, inconclusive' };
281
316
  const sourceOnly = median(srcResult.samples);
282
317
  const full = gain(m, m.before, m.after);
283
318
  if (full == null || full <= 0) return { gamed: false, sourceOnly, retained: 1, reason: 'no real improvement to re-prove' };
284
319
  const retained = +(gain(m, m.before, sourceOnly) / full).toFixed(3);
285
320
  const gamed = retained < threshold;
286
321
  return { gamed, sourceOnly, retained, reason: gamed
287
- ? `only ${(retained * 100).toFixed(0)}% of the gain survives when test/config/grader changes are reverted (threshold ${(threshold * 100).toFixed(0)}%) — most of the "win" came from editing the goalposts`
288
- : `${(retained * 100).toFixed(0)}% of the gain survives source-only — the source earned it` };
322
+ ? `only ${(retained * 100).toFixed(0)}% of the gain survives when test/config/grader changes are reverted (threshold ${(threshold * 100).toFixed(0)}%), most of the "win" came from editing the goalposts`
323
+ : `${(retained * 100).toFixed(0)}% of the gain survives source-only, the source earned it` };
289
324
  }
290
325
 
291
326
  // ---------------------------------------------------------------------------
292
327
  // main
293
328
  // ---------------------------------------------------------------------------
294
329
  // snapshot the working tree (tracked + untracked) into a dangling commit WITHOUT
295
- // touching the real index or files — so a gate/improve sees a file an agent just added.
330
+ // touching the real index or files, so a gate/improve sees a file an agent just added.
296
331
  // Returns 'HEAD' when nothing changed.
297
332
  function workingSnapshot(repo) {
298
333
  const idxDir = mkdtempSync(join(tmpdir(), 'pw-idx-'));
@@ -317,7 +352,7 @@ function workingSnapshot(repo) {
317
352
  }
318
353
  }
319
354
 
320
- // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree —
355
+ // self-heal: a hard-killed run (SIGKILL / power loss) can leave an orphaned worktree,
321
356
  // its `finally` never ran. On the next run, drop stale worktree registry entries and any
322
357
  // abandoned /tmp checkout, so nothing accumulates in your repo or temp dir. Never touches a
323
358
  // live worktree (a still-registered dir, or a checkout younger than 1h).
@@ -332,7 +367,7 @@ function selfHeal(repo) {
332
367
  for (const name of readdirSync(tmp)) {
333
368
  if (!name.startsWith('promptwheel-') && !name.startsWith('pw-idx-')) continue; // worktrees + temp-index dirs
334
369
  const p = join(tmp, name);
335
- if (tracked.has(p)) continue; // a live/registered worktree — leave it
370
+ if (tracked.has(p)) continue; // a live/registered worktree, leave it
336
371
  try { if (statSync(p).mtimeMs < cutoff) rmSync(p, { recursive: true, force: true }); } catch { /* */ }
337
372
  }
338
373
  } catch { /* cleanup must never block the gate */ }
@@ -340,7 +375,7 @@ function selfHeal(repo) {
340
375
 
341
376
  // Python editable installs (src-layout `pip install -e .`) can pin imports to the ORIGINAL
342
377
  // checkout: the worktree's tests then import your current tree at BOTH refs and every delta
343
- // reads 0 — the gate is structurally blind on that repo shape. Undetectable from the verdict
378
+ // reads 0, the gate is structurally blind on that repo shape. Undetectable from the verdict
344
379
  // (it looks like "unchanged"), so detect the editable install itself and warn. Verified by a
345
380
  // controlled repro (CHANGELOG 0.2.2).
346
381
  function warnEditableInstall(repo) {
@@ -355,8 +390,8 @@ function warnEditableInstall(repo) {
355
390
  'print("HIT" if hit else "")',
356
391
  ].join('\n');
357
392
  const out = execFileSync('python3', ['-c', probe, repo], { encoding: 'utf8', timeout: 5000 }).trim();
358
- if (out === 'HIT') console.error('⚠ editable install of this repo detected (pip install -e): worktree measurements may import your ORIGINAL checkout, not the measured ref — deltas can read 0. Use a non-editable install in the measuring venv, or put the worktree src on PYTHONPATH.');
359
- } catch { /* no python / cannot tell — stay quiet */ }
393
+ if (out === 'HIT') console.error('⚠ editable install of this repo detected (pip install -e): worktree measurements may import your ORIGINAL checkout, not the measured ref, deltas can read 0. Use a non-editable install in the measuring venv, or put the worktree src on PYTHONPATH.');
394
+ } catch { /* no python / cannot tell, stay quiet */ }
360
395
  }
361
396
 
362
397
  // the shared core: measure a change (base→head) and return the structured report
@@ -368,11 +403,11 @@ function gate(repo, opts) {
368
403
 
369
404
  let base, head;
370
405
  if (opts.working) {
371
- // measure uncommitted changes — tracked AND untracked — via a temp-index snapshot;
406
+ // measure uncommitted changes, tracked AND untracked, via a temp-index snapshot;
372
407
  // never touches the real index or working tree. (A loop agent's most common action
373
408
  // is to ADD a file; `git stash create` omits untracked, which silently reverted them.)
374
409
  try { git(['rev-parse', '--verify', 'HEAD'], repo); }
375
- catch { console.error('no commits yet — make an initial commit before gating --working changes'); process.exit(2); }
410
+ catch { console.error('no commits yet, make an initial commit before gating --working changes'); process.exit(2); }
376
411
  base = 'HEAD';
377
412
  head = workingSnapshot(repo);
378
413
  } else {
@@ -399,7 +434,7 @@ function gate(repo, opts) {
399
434
  const failed = metrics.some((m) => m.guard && !m.ok);
400
435
  const gamed = metrics.some((m) => m.gamed === true);
401
436
  // a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
402
- // nothing — don't launder that into a green PASS. The honest verdict is "couldn't measure".
437
+ // nothing, don't launder that into a green PASS. The honest verdict is "couldn't measure".
403
438
  // BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
404
439
  // elsewhere (that would make `improve` revert good changes and misread them as a plateau).
405
440
  const inconclusive = !failed && !gamed && metrics.some((m) => m.inert) && !metrics.some((m) => m.status === 'improved');
@@ -410,7 +445,7 @@ function gate(repo, opts) {
410
445
  // label attributes a change-type, subsystems fingerprint WHERE the change landed.
411
446
  cohort: opts.cohort ?? (process.env.CI ? 'ci' : 'local'),
412
447
  ...(opts.label ? { label: opts.label } : {}),
413
- subsystems: subsystemsOf(repo, base, head),
448
+ subsystems: subsystemsOf(repo, base, head, cfg),
414
449
  verdict, metrics,
415
450
  };
416
451
  if (!opts.noRecord && cfg.record !== false) recordOutcome(repo, report);
@@ -442,11 +477,11 @@ function improve(argv) {
442
477
  if (!args.attempt) { console.error('improve requires --attempt "<command that changes the repo>"'); process.exit(2); }
443
478
  const repo = git(['rev-parse', '--show-toplevel'], process.cwd());
444
479
  const dirty = git(['status', '--porcelain'], repo).split('\n').filter((l) => l.trim() && !l.includes('.promptwheel'));
445
- if (dirty.length) { console.error('working tree not clean — commit or stash first (improve needs a clean base to revert to)'); process.exit(2); }
480
+ if (dirty.length) { console.error('working tree not clean, commit or stash first (improve needs a clean base to revert to)'); process.exit(2); }
446
481
 
447
482
  console.error(`▶ attempt: ${args.attempt}`);
448
483
  try { execSync(args.attempt, { cwd: repo, stdio: 'inherit' }); }
449
- catch (e) { console.error(` (attempt exited ${e.status ?? 1} — gating whatever it changed)`); }
484
+ catch (e) { console.error(` (attempt exited ${e.status ?? 1}, gating whatever it changed)`); }
450
485
 
451
486
  const report = gate(repo, { working: true, repeat: args.repeat, noRecord: args.noRecord, detectGaming: args.detectGaming, label: args.label ?? args.attempt });
452
487
  const noChange = report.metrics.every((m) => m.delta === 0 || m.delta == null);
@@ -455,17 +490,17 @@ function improve(argv) {
455
490
  // result + exit code express loop progress so `while improve; do :; done` converges:
456
491
  // 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau/no-op OR inconclusive (reverted)
457
492
  let result, exit, note;
458
- if (report.verdict === 'fail') { result = 'regression'; exit = 1; revert(repo); note = '✗ guarded regression — reverted'; }
459
- else if (report.verdict === 'gamed') { result = 'gamed'; exit = 1; revert(repo); note = '🚩 gamed — a metric "improved" by editing tests/config, not source — reverted'; }
460
- else if (report.verdict === 'inconclusive') { result = 'inconclusive'; exit = 3; revert(repo); note = '≈ inconclusive — a guard measured nothing (inert); reverted'; }
493
+ if (report.verdict === 'fail') { result = 'regression'; exit = 1; revert(repo); note = '✗ guarded regression, reverted'; }
494
+ else if (report.verdict === 'gamed') { result = 'gamed'; exit = 1; revert(repo); note = '🚩 gamed, a metric "improved" by editing tests/config, not source, reverted'; }
495
+ else if (report.verdict === 'inconclusive') { result = 'inconclusive'; exit = 3; revert(repo); note = '≈ inconclusive, a guard measured nothing (inert); reverted'; }
461
496
  else if (noChange || improvedNames.length === 0) {
462
497
  result = 'plateau'; exit = 3; revert(repo);
463
- note = noChange ? '= no metric moved — reverted' : '= nothing improved beyond noise — reverted';
498
+ note = noChange ? '= no metric moved, reverted' : '= nothing improved beyond noise, reverted';
464
499
  } else {
465
500
  git(['add', '-A'], repo);
466
501
  execFileSync('git', ['commit', '-qm', `promptwheel: ${args.attempt} [improved ${improvedNames.join(', ')}]`], { cwd: repo });
467
502
  result = 'kept'; exit = 0;
468
- note = `✓ kept — committed ${git(['rev-parse', '--short', 'HEAD'], repo)} (improved ${improvedNames.join(', ')})`;
503
+ note = `✓ kept, committed ${git(['rev-parse', '--short', 'HEAD'], repo)} (improved ${improvedNames.join(', ')})`;
469
504
  }
470
505
 
471
506
  // stdout carries the value (JSON report or the human table); the decision line goes to
@@ -482,14 +517,14 @@ function revert(repo) {
482
517
  try { git(['clean', '-fd', '-e', '.promptwheel'], repo); } catch { /* best effort */ }
483
518
  }
484
519
 
485
- // Phase-5 seed: turn the accumulated reward stream into signal. Thin on purpose —
520
+ // Phase-5 seed: turn the accumulated reward stream into signal. Thin on purpose,
486
521
  // this is the substrate a future outcome-curated playbook / UCB work-discovery loop trains on,
487
522
  // NOT that loop itself. Just honest aggregation, no model.
488
523
  function insights(argv) {
489
524
  const args = parseArgs(argv);
490
525
  const repo = git(['rev-parse', '--show-toplevel'], process.cwd());
491
526
  const f = join(repo, '.promptwheel', 'outcomes.jsonl');
492
- if (!existsSync(f)) { console.error('no outcome record yet — run the gate a few times first (.promptwheel/outcomes.jsonl)'); process.exit(2); }
527
+ if (!existsSync(f)) { console.error('no outcome record yet, run the gate a few times first (.promptwheel/outcomes.jsonl)'); process.exit(2); }
493
528
  const runs = readFileSync(f, 'utf8').split('\n').filter(Boolean).map((l) => { try { return JSON.parse(l); } catch { return null; } }).filter(Boolean);
494
529
  const agg = {};
495
530
  for (const r of runs) for (const m of (r.metrics || [])) {
@@ -502,27 +537,27 @@ function insights(argv) {
502
537
  const rows = Object.entries(agg).map(([name, a]) => ({ name, ...a, lever: a.runs ? a.improved / a.runs : 0 }))
503
538
  .sort((x, y) => y.lever - x.lever);
504
539
  if (args.json) { console.log(JSON.stringify({ runs: runs.length, metrics: rows }, null, 2)); return; }
505
- console.log(`\nPromptWheel insights — ${runs.length} gated runs recorded\n`);
540
+ console.log(`\nPromptWheel insights, ${runs.length} gated runs recorded\n`);
506
541
  console.log(` ${'metric'.padEnd(18)} ${'runs'.padStart(5)} ${'impr'.padStart(5)} ${'regr'.padStart(5)} ${'inconc'.padStart(6)} ${'net Δ'.padStart(9)} lever`);
507
542
  for (const r of rows) {
508
543
  console.log(` ${r.name.padEnd(18)} ${String(r.runs).padStart(5)} ${String(r.improved).padStart(5)} ${String(r.regressed).padStart(5)} ${String(r.inconclusive).padStart(6)} ${String(r.net).padStart(9)} ${(r.lever * 100).toFixed(0)}%`);
509
544
  }
510
- console.log('\n lever = improved/runs — how reliably this metric actually responds. The');
545
+ console.log('\n lever = improved/runs, how reliably this metric actually responds. The');
511
546
  console.log(' highest-lever metrics are where an agent loop should spend its attempts.\n');
512
547
  }
513
548
 
514
549
  // ---------------------------------------------------------------------------
515
- // Phase 5 — outcome-curated playbook + UCB work-discovery.
550
+ // Phase 5, outcome-curated playbook + UCB work-discovery.
516
551
  // Unfrozen 2026-07-02 by explicit founder decision overriding D7 (see docs/LEARNING.md).
517
552
  // Design constraints that survive the unfreeze:
518
553
  // - PURE VIEW: the append-only ledger (.promptwheel/outcomes.jsonl) is the only store;
519
- // the playbook is re-derived on every read (decay at read time — no curator state to rot).
554
+ // the playbook is re-derived on every read (decay at read time, no curator state to rot).
520
555
  // - EVIDENCE-GATED: a key renders a claim only past MIN_MOVED weighted observations;
521
556
  // everything below that is counted, not asserted.
522
557
  // - CLAIM-GATED: no public compounding claim until bench/compounding-ab.mjs passes on
523
558
  // real usage data. The gate on the CLAIM outlived the gate on the CODE.
524
559
  // ---------------------------------------------------------------------------
525
- const HALF_LIFE = 20; // runs — an entry's weight halves every HALF_LIFE runs unless re-earned
560
+ const HALF_LIFE = 20; // runs, an entry's weight halves every HALF_LIFE runs unless re-earned
526
561
  const MIN_MOVED = 3; // weighted moved-observations before a key earns a rendered claim
527
562
  const EXPLORE_C = 1.0; // UCB exploration constant
528
563
 
@@ -580,7 +615,7 @@ function cohortNote(k) {
580
615
 
581
616
  function loadEntries(repo) {
582
617
  const runs = readLedger(repo);
583
- if (!runs.length) { console.error('no outcome record yet — run the gate a few times first (.promptwheel/outcomes.jsonl)'); process.exit(2); }
618
+ if (!runs.length) { console.error('no outcome record yet, run the gate a few times first (.promptwheel/outcomes.jsonl)'); process.exit(2); }
584
619
  const keys = foldOutcomes(runs);
585
620
  const totalW = keys.filter((k) => k.key === k.metric).reduce((s, k) => s + k.w, 0);
586
621
  const entries = keys.map((k) => ({ ...k, ...scoreKey(k, totalW), sufficient: k.moved >= MIN_MOVED }))
@@ -597,38 +632,38 @@ function playbook(argv) {
597
632
  const suff = entries.filter((e) => e.sufficient);
598
633
  const pct = (x) => `${Math.round(x * 100)}%`;
599
634
  const out = [];
600
- out.push(`## Earned playbook — ${runs.length} gated runs, decay half-life ${HALF_LIFE} runs`);
635
+ out.push(`## Earned playbook, ${runs.length} gated runs, decay half-life ${HALF_LIFE} runs`);
601
636
  out.push('*(every line below was measured by the gate, not asserted; entries decay unless re-earned)*', '');
602
637
  if (!suff.length) {
603
- out.push(`_Nothing has earned a claim yet — a key needs ≥${MIN_MOVED} weighted moved-observations. Keep gating; ${entries.length} keys are accumulating._`);
638
+ out.push(`_Nothing has earned a claim yet, a key needs ≥${MIN_MOVED} weighted moved-observations. Keep gating; ${entries.length} keys are accumulating._`);
604
639
  } else {
605
640
  for (const e of suff.slice(0, 20)) {
606
- out.push(`- **${e.key}** — helpful ${pct(e.p)} when it moves (${e.imp.toFixed(1)}✓/${e.reg.toFixed(1)}✗ weighted), responds in ${pct(e.move)} of runs, median effect ${e.effect}${cohortNote(e)}`);
641
+ out.push(`- **${e.key}**, helpful ${pct(e.p)} when it moves (${e.imp.toFixed(1)}✓/${e.reg.toFixed(1)}✗ weighted), responds in ${pct(e.move)} of runs, median effect ${e.effect}${cohortNote(e)}`);
607
642
  }
608
643
  const hidden = entries.length - suff.length;
609
- if (hidden > 0) out.push('', `_${hidden} more keys below the evidence threshold — counted, not asserted._`);
644
+ if (hidden > 0) out.push('', `_${hidden} more keys below the evidence threshold, counted, not asserted._`);
610
645
  }
611
646
  console.log(out.join('\n'));
612
647
  }
613
648
 
614
649
  // UCB work-discovery: where should the loop spend its NEXT attempt? Proposes measured
615
- // targets (metric/subsystem arms) — never code advice; the measurement stays the message.
650
+ // targets (metric/subsystem arms), never code advice; the measurement stays the message.
616
651
  function suggest(argv) {
617
652
  const args = parseArgs(argv);
618
653
  const repo = git(['rev-parse', '--show-toplevel'], process.cwd());
619
654
  const { runs, entries } = loadEntries(repo);
620
655
  const arms = entries.filter((e) => e.key === e.metric || e.key.includes(' @ ')).sort((a, b) => b.ucb - a.ucb);
621
656
  if (args.json) { console.log(JSON.stringify({ runs: runs.length, thin: runs.length < 10, arms: arms.slice(0, 10) }, null, 2)); return; }
622
- console.log(`\nPromptWheel suggest — UCB over ${runs.length} gated runs${runs.length < 10 ? ' (thin record: exploration dominates — treat as a coin with opinions)' : ''}\n`);
657
+ console.log(`\nPromptWheel suggest, UCB over ${runs.length} gated runs${runs.length < 10 ? ' (thin record: exploration dominates, treat as a coin with opinions)' : ''}\n`);
623
658
  for (const a of arms.slice(0, 5)) {
624
659
  console.log(` ${a.ucb.toFixed(3)} ${a.key.padEnd(28)} lever ${a.lever.toFixed(3)} (helpful ${a.p}, responds ${a.move}) · evidence ${a.moved.toFixed(1)} moved / ${a.n} runs${a.sufficient ? '' : ' · below claim threshold'}`);
625
660
  }
626
- console.log('\n score = lever + exploration bonus — high scores are either proven levers or under-explored arms.\n');
661
+ console.log('\n score = lever + exploration bonus, high scores are either proven levers or under-explored arms.\n');
627
662
  }
628
663
 
629
- // backfill — seed the ledger from git history: replay past commits through the CURRENT
664
+ // backfill, seed the ledger from git history: replay past commits through the CURRENT
630
665
  // metrics (LEARNING.md harvest path 1). Deterministic, no LLM. Rows are cohort-tagged
631
- // 'backfill' — historical human commits are NOT live agent-loop evidence; the cohort
666
+ // 'backfill', historical human commits are NOT live agent-loop evidence; the cohort
632
667
  // machinery segments them and flags disagreement rather than averaging it away. The
633
668
  // conventional-commit type (fix/feat/refactor/…) becomes the change-type label for free.
634
669
  function backfill(argv) {
@@ -640,7 +675,7 @@ function backfill(argv) {
640
675
  if (!commits.length) { console.error('no commits to backfill'); process.exit(2); }
641
676
  const seen = new Set(readLedger(repo).filter((r) => r.cohort === 'backfill').map((r) => r.head));
642
677
  let done = 0, skipped = 0;
643
- console.log(`\nbackfilling ${commits.length} commits through the current metrics (gaming detection ${args.dgExplicit && args.detectGaming ? 'ON' : 'off — pass --detect-gaming to audit history too'})\n`);
678
+ console.log(`\nbackfilling ${commits.length} commits through the current metrics (gaming detection ${args.dgExplicit && args.detectGaming ? 'ON' : 'off, pass --detect-gaming to audit history too'})\n`);
644
679
  for (const c of commits) {
645
680
  try { git(['rev-parse', `${c}~1`], repo); } catch { skipped++; continue; } // root commit
646
681
  if (seen.has(short(repo, c))) { skipped++; continue; } // already recorded
@@ -650,12 +685,12 @@ function backfill(argv) {
650
685
  console.log(` ${report.head} ${report.verdict.toUpperCase().padEnd(6)} ${type ? `#${type}`.padEnd(10) : ''.padEnd(10)} ${subj.slice(0, 56)}`);
651
686
  done++;
652
687
  }
653
- console.log(`\n backfilled ${done}${skipped ? ` (skipped ${skipped}: root or already recorded)` : ''} — old commits that no longer build record as unmeasurable, never faked.`);
688
+ console.log(`\n backfilled ${done}${skipped ? ` (skipped ${skipped}: root or already recorded)` : ''}, old commits that no longer build record as unmeasurable, never faked.`);
654
689
  console.log(' next: promptwheel playbook · promptwheel suggest\n');
655
690
  }
656
691
 
657
692
  // ---------------------------------------------------------------------------
658
- // init — write a starter config so a newcomer isn't staring at a blank page
693
+ // init, write a starter config so a newcomer isn't staring at a blank page
659
694
  // ---------------------------------------------------------------------------
660
695
  const LINT_CMD = 'npx eslint . -f unix 2>/dev/null | grep -c " error " || true';
661
696
  const PRESETS = {
@@ -676,9 +711,28 @@ const PRESETS = {
676
711
  { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
677
712
  { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
678
713
  { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
679
- { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
714
+ { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|# ?nosec|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
680
715
  { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
681
716
  ] },
717
+ 'no-ai-tells': { desc: 'opt-in: fail the gate if the em-dash count rises (an AI writing tell); some projects use them on purpose, so it is not a default',
718
+ metrics: [
719
+ { name: 'em_dashes', cmd: 'grep -rIoe "—" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
720
+ ] },
721
+ };
722
+
723
+ // security-antihack = the security-invariant findings gate + the SAME antihack tripwires
724
+ // (so an agent that suppresses a security rule, # noqa / # nosec / eslint-disable, or
725
+ // skips a security test is caught), PLUS a starter `evaluatorPaths` that protects common
726
+ // security-gate config files (a ruleset or a budget the agent could otherwise loosen to
727
+ // pass). The scanner pack installs into `.promptwheel/` (already evaluator-protected). It
728
+ // is the "guard the guards" preset: pairs with --detect-gaming so a win that only moved by
729
+ // loosening the security gate, not by fixing the code, is GAMED. Blind spot (by design,
730
+ // not this tool's job): a source-resident backdoor that behaves safely only under test;
731
+ // that needs held-out/behavioral analysis, not a source-only re-run.
732
+ PRESETS['security-antihack'] = {
733
+ desc: 'gate security-invariant findings + antihack tripwires + evaluatorPaths so an agent cannot loosen the security gate to pass it (pairs with --detect-gaming)',
734
+ evaluatorPaths: ['.semgrep.yml', '.semgrep.yaml', '.bandit', 'bandit.yaml', '.ruff.toml', 'ruff.toml', '.snyk', 'security-budget.json'],
735
+ metrics: [{ name: 'security_findings', cmd: 'node packs/security/scan.mjs .', extract: 'number', direction: 'down', guard: true }, ...PRESETS.antihack.metrics.slice(1)],
682
736
  };
683
737
 
684
738
  function detectTestCmd(repo) {
@@ -686,26 +740,26 @@ function detectTestCmd(repo) {
686
740
  if (has('go.mod')) return 'go test ./...';
687
741
  if (has('Cargo.toml')) return 'cargo test';
688
742
  if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return has('.venv/bin/pytest') ? '.venv/bin/pytest -q' : 'pytest -q';
689
- // only trust `npm test` when a test script actually exists — otherwise the metric can
743
+ // only trust `npm test` when a test script actually exists, otherwise the metric can
690
744
  // never pass and the gate would "protect" nothing (very common in Next.js app repos)
691
745
  try {
692
746
  if (JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8')).scripts?.test) return 'npm test --silent';
693
- } catch { /* no or unparsable package.json — fall through */ }
747
+ } catch { /* no or unparsable package.json, fall through */ }
694
748
  return 'echo "set your test command in promptwheel.config.json" && false';
695
749
  }
696
750
 
697
751
  // deps that live outside git + the env a test needs, defaulted per stack. init writes these into
698
- // config; the measurement bridge (bridgeEnv/runSetup) consumes them — keeps the engine agnostic.
752
+ // config; the measurement bridge (bridgeEnv/runSetup) consumes them, keeps the engine agnostic.
699
753
  function detectEnv(repo) {
700
754
  const has = (f) => existsSync(join(repo, f));
701
755
  // Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
702
756
  // resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
703
757
  if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
704
- if (has('Cargo.toml')) return { linkDirs: [] }; // do NOT link target/: `cargo test` WRITES there — linking would clobber the user's real dir and share a build cache across refs (measurement blindness). Build fresh in the worktree.
758
+ if (has('Cargo.toml')) return { linkDirs: [] }; // do NOT link target/: `cargo test` WRITES there, linking would clobber the user's real dir and share a build cache across refs (measurement blindness). Build fresh in the worktree.
705
759
  return {};
706
760
  }
707
761
 
708
- // only offer the lint metric where eslint actually exists — otherwise it reads 0 forever
762
+ // only offer the lint metric where eslint actually exists, otherwise it reads 0 forever
709
763
  // and a newcomer mistakes a metric that can't move for a clean repo.
710
764
  function hasEslint(repo) {
711
765
  const files = ['eslint.config.js', 'eslint.config.mjs', 'eslint.config.cjs',
@@ -726,18 +780,19 @@ function init(argv) {
726
780
  const repo = (() => { try { return git(['rev-parse', '--show-toplevel'], process.cwd()); } catch { return process.cwd(); } })();
727
781
  const out = join(repo, 'promptwheel.config.json');
728
782
  if (existsSync(out) && !argv.includes('--force')) {
729
- console.error('promptwheel.config.json already exists — pass --force to overwrite'); process.exit(2);
783
+ console.error('promptwheel.config.json already exists, pass --force to overwrite'); process.exit(2);
730
784
  }
731
785
  const pi = argv.indexOf('--preset');
732
786
  const presetName = pi >= 0 ? argv[pi + 1] : null;
733
- let metrics, note;
787
+ let metrics, note, evaluatorPaths;
734
788
  if (presetName) {
735
789
  const p = PRESETS[presetName];
736
- if (!p) { console.error(`unknown preset "${presetName}" — try: promptwheel init --list`); process.exit(2); }
790
+ if (!p) { console.error(`unknown preset "${presetName}", try: promptwheel init --list`); process.exit(2); }
791
+ evaluatorPaths = p.evaluatorPaths; // a preset may ship a starter evaluator allowlist (e.g. security-antihack)
737
792
  metrics = p.metrics || [{ name: 'tests_pass', cmd: detectTestCmd(repo), extract: 'exit', direction: 'pass', guard: true }];
738
793
  metrics = metrics.map((m) => m.cmd === '__TESTCMD__' ? { ...m, cmd: detectTestCmd(repo) } : m); // fill the preset's target
739
794
  // pack-backed presets: the referenced scripts live in THIS package, not
740
- // the user's repo — copy them in so the generated config is
795
+ // the user's repo, copy them in so the generated config is
741
796
  // self-contained and committable (a bare reference silently 404'd)
742
797
  metrics = metrics.map((m) => {
743
798
  const pk = /(^|\s)(packs\/[\w-]+)\//.exec(m.cmd);
@@ -752,7 +807,7 @@ function init(argv) {
752
807
  note = presetName;
753
808
  } else {
754
809
  // default = the headline posture: guarded tests + the antihack tripwires, so the FIRST
755
- // cheat a newcomer tries (gut the tests while the metric stays flat) fails the gate —
810
+ // cheat a newcomer tries (gut the tests while the metric stays flat) fails the gate,
756
811
  // "catch your agent cheating" must be true out of the box, not only under a preset.
757
812
  const testCmd = detectTestCmd(repo);
758
813
  const lint = hasEslint(repo);
@@ -762,8 +817,8 @@ function init(argv) {
762
817
  }
763
818
  const envCfg = presetName ? {} : detectEnv(repo);
764
819
  if (envCfg.linkDirs) note += ` · linking ${envCfg.linkDirs.join(', ')}`;
765
- writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, metrics }, null, 2) + '\n');
766
- console.log(`✓ wrote promptwheel.config.json — ${note}`);
820
+ writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, ...(evaluatorPaths ? { evaluatorPaths } : {}), metrics }, null, 2) + '\n');
821
+ console.log(`✓ wrote promptwheel.config.json, ${note}`);
767
822
  console.log('\n next: promptwheel run --working # gate your uncommitted changes');
768
823
  console.log(' promptwheel init --list # other presets (llm-eval, bundle-size, …)');
769
824
  }
@@ -787,7 +842,7 @@ function guards(argv) {
787
842
  source: m.__src, override: !!m.__override, ...(hist[m.name] || { runs: 0, flagged: 0, last: null }),
788
843
  }));
789
844
  if (args.json) { console.log(JSON.stringify({ guards: rows }, null, 2)); return; }
790
- console.log('\nPromptWheel guardrails — effective set for this repo\n');
845
+ console.log('\nPromptWheel guardrails, effective set for this repo\n');
791
846
  for (const r of rows) {
792
847
  const prov = r.override ? 'local override' : (r.source === 'local' ? 'local' : `inherited ← ${r.source}`);
793
848
  const rec = r.guard ? (r.runs ? `· ${r.flagged} flagged / ${r.runs} runs` : '· no runs yet') : '';
@@ -803,13 +858,13 @@ function printHuman(r) {
803
858
  console.log(`\nPromptWheel ${r.base} → ${r.head}${r.repeat > 1 ? ` (×${r.repeat})` : ''}\n`);
804
859
  for (const m of r.metrics) {
805
860
  const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
806
- const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
861
+ const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
807
862
  console.log(` ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)} (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
808
- if (m.inert) console.log(' ⚠ never passed at either ref — this guard is protecting nothing (check its command)');
809
- if (m.gamed === true) console.log(` 🚩 GAMED — ${m.gamingReason}`);
863
+ if (m.inert) console.log(' ⚠ never passed at either ref, this guard is protecting nothing (check its command)');
864
+ if (m.gamed === true) console.log(` 🚩 GAMED, ${m.gamingReason}`);
810
865
  }
811
- console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ' — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ' — a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ' — a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
812
- if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name) — `promptwheel guards` shows the effective set');
866
+ console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ', a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ', a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ', a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
867
+ if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name), `promptwheel guards` shows the effective set');
813
868
  console.log('');
814
869
  }
815
870
 
@@ -818,12 +873,12 @@ function renderMarkdown(r) {
818
873
  const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : r.verdict === 'inconclusive' ? '🟡' : '❌';
819
874
  const sIcon = { improved: '🟢', regressed: '🔴', unchanged: '⚪', inconclusive: '🟡', unmeasurable: '⚫' };
820
875
  const rows = r.metrics.map((m) => {
821
- const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
876
+ const d = m.delta == null ? 'n/a' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
822
877
  const status = m.gamed === true ? `🚩 gamed (${(m.retained * 100).toFixed(0)}% survives source-only)` : `${sIcon[m.status] || ''} ${m.status}${m.inert ? ' ⚠ never passes' : ''}`;
823
878
  return `| ${m.guard ? '🛡️ ' : ''}${m.name} | ${m.before} | ${m.after} | ${d} | ${status} | ${m.confidence} |`;
824
879
  }).join('\n');
825
880
  return [
826
- `### ${icon} PromptWheel — outcome gate: **${r.verdict.toUpperCase()}**`,
881
+ `### ${icon} PromptWheel, outcome gate: **${r.verdict.toUpperCase()}**`,
827
882
  '',
828
883
  `\`${r.base} → ${r.head}\` · ${r.mode} mode${r.repeat > 1 ? ` · ×${r.repeat}` : ''}`,
829
884
  '',
@@ -831,7 +886,7 @@ function renderMarkdown(r) {
831
886
  '|---|--:|--:|--:|---|---|',
832
887
  rows,
833
888
  '',
834
- r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader — not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert** — it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
889
+ r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader, not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert**, it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
835
890
  '',
836
891
  '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
837
892
  ].join('\n');
@@ -859,7 +914,7 @@ function parseArgs(argv) {
859
914
 
860
915
  // pure, side-effect-free helpers are exported for unit testing; the CLI below only
861
916
  // runs when this file is executed directly (not when imported by the test suite).
862
- export { extract, evaluate, median, spread, renderMarkdown, isNonSource, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
917
+ export { extract, evaluate, median, spread, renderMarkdown, isNonSource, evaluatorMatcher, gain, judgeGaming, foldOutcomes, betaMean, scoreKey };
863
918
 
864
919
  function main() {
865
920
  const [cmd, ...rest] = process.argv.slice(2);
@@ -873,11 +928,11 @@ function main() {
873
928
  else if (cmd === 'guards') guards(rest);
874
929
  else {
875
930
  console.log([
876
- 'PromptWheel — catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
931
+ 'PromptWheel, catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
877
932
  '',
878
933
  ' promptwheel init [--preset <name> | --list] write a starter config for your stack',
879
934
  ' promptwheel run [--base R] [--head R] [--repeat N] [--json|--markdown]',
880
- ' catches reward-hacking BY DEFAULT — re-proves each win from SOURCE edits alone (verdict GAMED,',
935
+ ' catches reward-hacking BY DEFAULT, re-proves each win from SOURCE edits alone (verdict GAMED,',
881
936
  ' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
882
937
  ' promptwheel run --working gate uncommitted changes (incl. newly added files)',
883
938
  ' promptwheel improve --attempt "<cmd>" run an agent/script; keep only if a metric improved',
@@ -890,7 +945,7 @@ function main() {
890
945
  '',
891
946
  'Loop it: while promptwheel improve --attempt "$AGENT"; do :; done # stops on plateau/regression',
892
947
  'Config: promptwheel.config.json → { extends?, metrics:[{ name, cmd, direction, extract?, guard? }] } (or: promptwheel init)',
893
- ' extends: a path (or array) to a shared base config — repos INHERIT its guardrails; local metrics override by name.',
948
+ ' extends: a path (or array) to a shared base config, repos INHERIT its guardrails; local metrics override by name.',
894
949
  ].join('\n'));
895
950
  process.exit(cmd ? 2 : 0);
896
951
  }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "promptwheel",
3
- "version": "0.4.5",
4
- "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone: it re-proves the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook.",
3
+ "version": "0.4.7",
4
+ "description": "A deterministic, zero-LLM check for your coding loop. It proves every change earned its win by re-running the result from the agent's source edits alone, so a green that came from editing the test reads GAMED. Runs in CI or a Claude Code hook.",
5
5
  "type": "module",
6
6
  "bin": {
7
7
  "promptwheel": "bin/promptwheel.mjs",