promptwheel 0.1.0 → 0.1.2

package/README.md

@@ -8,9 +8,9 @@ PromptWheel re-proves every "win" using the agent's **source edits alone**. If t
 
 It's built on an **outcome gate**: for any change it re-runs your metric commands (tests, lint, `tsc`, coverage, bundle, eval) in throwaway git worktrees before and after, and refuses to trust a delta inside the measurement noise. The gate everyone ships asks *"did the number move?"* — PromptWheel also asks *"did the agent **earn** it?"*
 
-PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it measures your real repo metrics in throwaway worktrees, refuses to trust a delta inside the noise, and answers one question — *did this turn earn its keep?* — so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
+PromptWheel is the **signal, not the loop driver**: wire it as the verifier inside Claude Code `/loop`, a Ralph `while`-loop, or a Beads pull-loop. Each turn it answers one question — *did this turn earn its keep, and did the agent earn it honestly?* — so the loop improves instead of confidently degrading. (In CI it's the **outcome gate for AI code**: the same verdict, as a PR check.)
 
-> Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") is a solved, commoditized problem; the trustworthy reward signal is the open one.
+> Same name, new meaning. The "wheel" is the **improvement flywheel**: every turn only counts if it **provably moved a metric without regressing another.** Orchestration (the old "wheel of prompts") *and the outcome gate itself* are now solved, commoditized problems; the open one is **catching when the agent games that gate** — making the reward signal one you can't cheat.
 
 AI coding agents (and humans) declare success when a change *compiles and tests pass*. They almost never verify that a change **improved a real measurable thing without regressing another** — and outcome data rots in dashboards (DORA, Swarmia) disconnected from the change that caused it. PromptWheel closes that loop.
 
@@ -29,12 +29,12 @@ PromptWheel  a1b2c3d → e4f5g6h  (×5)
 
 Exit `0` on pass, `1` on fail (CI-friendly). No build step, zero dependencies, Node 18+.
 
-## Catch your agent cheating — `--detect-gaming`
+## Catch your agent cheating — on by default
 
-The headline feature. After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive — because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files — the verdict is **`GAMED` (exit 2)**:
+The headline feature, and it runs **by default** (in both `run` and `improve`; pass `--no-detect-gaming` for the bare outcome gate). After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive — because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files — the verdict is **`GAMED` (exit 2)**:
 
 ```
-$ promptwheel run --detect-gaming
+$ promptwheel run
 
 PromptWheel  base → head
   ▲ tests_pass    0 → 1  (+1, improved) [guard✓, high]
@@ -46,11 +46,14 @@ Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) are
 
 ```bash
 npx promptwheel init --preset antihack    # target + tripwire guards
-npx promptwheel run --detect-gaming       # exit 0 = real win · 1 = regression · 2 = GAMED
+npx promptwheel run                       # detection ON by default · exit 0 win · 1 regression · 2 GAMED
+                                          #   (add --no-detect-gaming for just the outcome gate)
 ```
 
 Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable.
 
+This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers — held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage — are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce).
+
 ## Use
 
 ```bash
@@ -59,7 +62,7 @@ npx promptwheel init                      # detects stack → guarded test metri
 npx promptwheel init --list               # presets: tests-pass · lint · bundle-size · llm-eval · antihack
 
 # measure a change
-npx promptwheel run                       # base = merge-base with main, head = HEAD
+npx promptwheel run                       # base = merge-base with main, head = HEAD · reward-hack detection ON
 npx promptwheel run --working             # measure UNCOMMITTED changes (incl. newly added files)
 npx promptwheel run --repeat 5 --json     # measure 5× to establish a noise band, emit JSON
 
@@ -87,6 +90,8 @@ npx promptwheel run --working --json    # branch on .verdict / per-metric .statu
 
 The exit code is the contract — `0` kept · `1` regression · `3` plateau — so any driver (`/loop`, a Ralph `while`, a Beads pull-loop) converges without parsing anything. PromptWheel never drives the loop; it only says whether the turn counted.
 
+**Two callers, by design.** Your *agent* consumes the verdict — it's the loop's per-turn reward (`improve` / `run --working --json`; exit `0` kept · `1`/`2` reverted · `3` plateau). The *harness* (a Stop-hook or CI) runs `--detect-gaming` — the audit the agent **can't self-clear**, because a contestant can't referee itself (see [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md)). Same tool; *who calls it* is the difference between a reward and an audit.
+
 ## In Claude Code — plugin
 
 Bring the gate into Claude Code as slash commands:
@@ -136,6 +141,7 @@ The Action runs straight from its own checkout — no npm install, no build. See
 - **extract** — reduce its output to a number: `number` (last number, default) · `lines` (count non-empty lines) · `exit` (1 if exit 0 else 0) · `{ "regex": "coverage: (\\d+)" }` (first capture).
 - **direction** — `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
 - **guard** — `true` = a *trusted* regression **fails** the gate; `false` = informational.
+- **gamingCheck** — `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files — otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
 
 ## Guardrails & inheritance
 
@@ -155,6 +161,8 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
 
 `extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance — `inherited ← base.json`, `local`, or `local override` — plus each guard's flag record from the outcome stream.
 
+Read this way, `extends` is **the shared invariants — the "business rules" — your agents inherit and are held to**: enforced as *measured guards* (a trusted regression fails the gate / reverts the commit), not advisory text an agent can quietly ignore. (Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.)
+
 ## Trust model — the point of the whole thing
 
 A number that jumps around between runs is worthless as a signal. PromptWheel won't pretend otherwise:
@@ -173,6 +181,8 @@ The accumulated record of **which change-types move which metrics** is the asset
 
 ## Docs
 
+- [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md) — how `--detect-gaming` fits as the **deterministic layer** alongside held-out tests, LLM judges, and human review: the coverage matrix, the compose-as-a-pipeline model, and the honest in/out-of-scope boundary.
+- [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md) — making "the agent can't skip it" real: the three places to enforce (loop-revert · CI + branch protection · a tested Claude Code Stop-hook), the exact wiring, and the *protect-the-gate's-own-config* caveat.
 - [docs/VISION.md](docs/VISION.md) — why we pivoted from orchestrator to outcome gate, the thesis, the moat, the open-core model.
 - [docs/ROADMAP.md](docs/ROADMAP.md) — the phased plan and the ship-now/stay-thin guardrails.
 - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how the engine works: schemas, extract modes, the trust/noise model.
@@ -182,7 +192,7 @@ The accumulated record of **which change-types move which metrics** is the asset
 ## Develop
 
 ```bash
-npm test     # 36 dep-free tests (node:test) — unit + integration, no dependencies
+npm test     # 37 dep-free tests (node:test) — unit + integration, no dependencies
 ```
 
 The engine is one importable file; pure helpers are exported for unit tests, the CLI runs only when invoked directly. Add a test with every behavior change.
@@ -199,6 +209,7 @@ The engine is one importable file; pure helpers are exported for unit tests, the
 - [x] `promptwheel init` + presets — zero-config onboarding
 - [x] `insights` — reward-stream aggregation (Phase-5 seed)
 - [x] `--detect-gaming` — reward-hack detection: re-prove the win from source edits alone + `antihack` preset
-- [ ] npm publish (the lead magnet) · ACE-style learning + UCB work-discovery (**frozen** — gated on data + ≥1 paid engagement; see [docs/LEARNING.md](docs/LEARNING.md))
+- [x] npm publish — `promptwheel@0.1.0` (the lead magnet, shipped 2026-06)
+- [ ] ACE-style learning + UCB work-discovery (**frozen** — gated on data + ≥1 paid engagement; see [docs/LEARNING.md](docs/LEARNING.md))
 
-> Status: v0, runnable, all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
+> Status: **published v0.1.0** — the consulting lead magnet (npm `promptwheel@0.1.0`), all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.

package/bin/promptwheel.mjs

@@ -311,6 +311,7 @@ function gate(repo, opts) {
     for (const m of metrics) {
       if (m.status !== 'improved') continue;
       const cm = cfg.metrics.find((c) => c.name === m.name);
+      if (cm.gamingCheck === false) continue;   // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
       const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, linkNM, repeat));
       m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
     }
@@ -333,7 +334,7 @@ function run(argv) {
   process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : 1);
 }
 
-// the moat: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
+// persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
 function recordOutcome(repo, report) {
   try {
     const dir = join(repo, '.promptwheel');
@@ -342,7 +343,7 @@ function recordOutcome(repo, report) {
   } catch { /* recording must never break the gate */ }
 }
 
-// the flywheel: run any agent/script, gate the result, keep the change ONLY if a metric improved
+// keep-if-improved: run any agent/script, gate the result, keep the change ONLY if a metric improved
 function improve(argv) {
   const args = parseArgs(argv);
   if (!args.attempt) { console.error('improve requires --attempt "<command that changes the repo>"'); process.exit(2); }
@@ -434,10 +435,10 @@ const PRESETS = {
   'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
     metrics: [
       { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
-      { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true },
-      { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true },
-      { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true },
-      { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true },
+      { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
+      { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
+      { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
+      { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
     ] },
 };
 
@@ -552,7 +553,7 @@ function renderMarkdown(r) {
 }
 
 function parseArgs(argv) {
-  const a = { json: false };
+  const a = { json: false, detectGaming: true };   // reward-hack detection is ON by default; --no-detect-gaming opts out
   for (let i = 0; i < argv.length; i++) {
     if (argv[i] === '--base') a.base = argv[++i];
     else if (argv[i] === '--head') a.head = argv[++i];
@@ -560,6 +561,7 @@ function parseArgs(argv) {
     else if (argv[i] === '--working') a.working = true;
     else if (argv[i] === '--no-record') a.noRecord = true;
     else if (argv[i] === '--detect-gaming' || argv[i] === '--antihack') a.detectGaming = true;
+    else if (argv[i] === '--no-detect-gaming' || argv[i] === '--no-antihack') a.detectGaming = false;
     else if (argv[i] === '--attempt') a.attempt = argv[++i];
     else if (argv[i] === '--json') a.json = true;
     else if (argv[i] === '--markdown') a.markdown = true;
@@ -580,13 +582,13 @@ function main() {
   else if (cmd === 'guards') guards(rest);
   else {
     console.log([
-      'PromptWheel — the per-turn reward for AI coding loops. Prove a change moved a metric.',
+      'PromptWheel — catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
       '',
       '  promptwheel init [--preset <name> | --list]  write a starter config for your stack',
       '  promptwheel run [--base R] [--head R] [--repeat N] [--json|--markdown]',
+      '       catches reward-hacking BY DEFAULT — re-proves each win from SOURCE edits alone (verdict GAMED,',
+      '       exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
       '  promptwheel run --working                    gate uncommitted changes (incl. newly added files)',
-      '  promptwheel run --detect-gaming              re-prove each win with SOURCE edits alone — catch reward-hacking',
-      '                                               (verdict GAMED, exit 2, when a metric only moved by editing tests/config/grader)',
       '  promptwheel improve --attempt "<cmd>"        run an agent/script; keep only if a metric improved',
       '                                               exit 0=kept · 1=regression · 3=plateau · add --json',
       '  promptwheel insights                         which metrics actually respond (loop memory)',

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "promptwheel",
-  "version": "0.1.0",
-  "description": "The trustworthy per-turn reward for AI coding loops — prove a change moved a metric without regressing another. Before/after in an isolated worktree, regression-guarded, noise-aware.",
+  "version": "0.1.2",
+  "description": "Catch your agent cheating — a deterministic, zero-LLM CLI that flags when an AI coding agent gamed its own metric (re-proves the win from the agent's source edits alone). Built on an outcome gate; runs in CI or a Claude Code hook.",
   "type": "module",
   "bin": {
     "promptwheel": "bin/promptwheel.mjs",
@@ -18,5 +18,5 @@
   "bugs": { "url": "https://github.com/promptwheel-ai/promptwheel/issues" },
   "author": "Matthew Owens",
   "license": "MIT",
-  "keywords": ["agent", "ci", "regression", "metrics", "verification", "ai-coding", "outcome", "evals", "loop", "promptwheel"]
+  "keywords": ["agent", "ci", "regression", "metrics", "verification", "ai-coding", "outcome", "evals", "loop", "promptwheel", "reward-hacking", "detect-gaming", "reward-hack-detection"]
 }