promptwheel 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -5
- package/bin/promptwheel.mjs +7 -6
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -29,12 +29,12 @@ PromptWheel a1b2c3d → e4f5g6h (×5)
|
|
|
29
29
|
|
|
30
30
|
Exit `0` on pass, `1` on fail (CI-friendly). No build step, zero dependencies, Node 18+.
|
|
31
31
|
|
|
32
|
-
## Catch your agent cheating —
|
|
32
|
+
## Catch your agent cheating — on by default
|
|
33
33
|
|
|
34
|
-
The headline feature. After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive — because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files — the verdict is **`GAMED` (exit 2)**:
|
|
34
|
+
The headline feature, and it runs **by default** (in both `run` and `improve`; pass `--no-detect-gaming` for the bare outcome gate). After the normal gate, every *win* is re-proven using the agent's source edits **alone**: PromptWheel splits the diff into production source vs. `{test, config, grader, golden}` files, rebuilds a clean worktree at the base with **only the source slice** applied, and re-runs the gate. If the win doesn't survive — because it only passed by skipping/deleting a test, mocking the grader, editing a golden, relaxing a config, or the "win" touched zero source files — the verdict is **`GAMED` (exit 2)**:
|
|
35
35
|
|
|
36
36
|
```
|
|
37
|
-
$ promptwheel run
|
|
37
|
+
$ promptwheel run
|
|
38
38
|
|
|
39
39
|
PromptWheel base → head
|
|
40
40
|
▲ tests_pass 0 → 1 (+1, improved) [guard✓, high]
|
|
@@ -46,7 +46,8 @@ Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) are
|
|
|
46
46
|
|
|
47
47
|
```bash
|
|
48
48
|
npx promptwheel init --preset antihack # target + tripwire guards
|
|
49
|
-
npx promptwheel run
|
|
49
|
+
npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED
|
|
50
|
+
# (add --no-detect-gaming for just the outcome gate)
|
|
50
51
|
```
|
|
51
52
|
|
|
52
53
|
Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable.
|
|
@@ -61,7 +62,7 @@ npx promptwheel init # detects stack → guarded test metri
|
|
|
61
62
|
npx promptwheel init --list # presets: tests-pass · lint · bundle-size · llm-eval · antihack
|
|
62
63
|
|
|
63
64
|
# measure a change
|
|
64
|
-
npx promptwheel run # base = merge-base with main, head = HEAD
|
|
65
|
+
npx promptwheel run # base = merge-base with main, head = HEAD · reward-hack detection ON
|
|
65
66
|
npx promptwheel run --working # measure UNCOMMITTED changes (incl. newly added files)
|
|
66
67
|
npx promptwheel run --repeat 5 --json # measure 5× to establish a noise band, emit JSON
|
|
67
68
|
|
|
@@ -89,6 +90,8 @@ npx promptwheel run --working --json # branch on .verdict / per-metric .statu
|
|
|
89
90
|
|
|
90
91
|
The exit code is the contract — `0` kept · `1` regression · `3` plateau — so any driver (`/loop`, a Ralph `while`, a Beads pull-loop) converges without parsing anything. PromptWheel never drives the loop; it only says whether the turn counted.
|
|
91
92
|
|
|
93
|
+
**Two callers, by design.** Your *agent* consumes the verdict — it's the loop's per-turn reward (`improve` / `run --working --json`; exit `0` kept · `1`/`2` reverted · `3` plateau). The *harness* (a Stop-hook or CI) runs `--detect-gaming` — the audit the agent **can't self-clear**, because a contestant can't referee itself (see [docs/ENFORCEMENT.md](docs/ENFORCEMENT.md)). Same tool; *who calls it* is the difference between a reward and an audit.
|
|
94
|
+
|
|
92
95
|
## In Claude Code — plugin
|
|
93
96
|
|
|
94
97
|
Bring the gate into Claude Code as slash commands:
|
|
@@ -158,6 +161,8 @@ Teams keep one shared base config and have every repo inherit it via `extends`:
|
|
|
158
161
|
|
|
159
162
|
`extends` takes a path (or array of paths) to base configs: a repo **inherits** their guardrails, and a local metric of the same name **overrides** the inherited one (tighten, loosen, or disable). `promptwheel guards` shows the effective set with provenance — `inherited ← base.json`, `local`, or `local override` — plus each guard's flag record from the outcome stream.
|
|
160
163
|
|
|
164
|
+
Read this way, `extends` is **the shared invariants — the "business rules" — your agents inherit and are held to**: enforced as *measured guards* (a trusted regression fails the gate / reverts the commit), not advisory text an agent can quietly ignore. (Natural-language conventions belong in `AGENTS.md`; only deterministic, measurable guards belong here.)
|
|
165
|
+
|
|
161
166
|
## Trust model — the point of the whole thing
|
|
162
167
|
|
|
163
168
|
A number that jumps around between runs is worthless as a signal. PromptWheel won't pretend otherwise:
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -334,7 +334,7 @@ function run(argv) {
|
|
|
334
334
|
process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : 1);
|
|
335
335
|
}
|
|
336
336
|
|
|
337
|
-
//
|
|
337
|
+
// persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
|
|
338
338
|
function recordOutcome(repo, report) {
|
|
339
339
|
try {
|
|
340
340
|
const dir = join(repo, '.promptwheel');
|
|
@@ -343,7 +343,7 @@ function recordOutcome(repo, report) {
|
|
|
343
343
|
} catch { /* recording must never break the gate */ }
|
|
344
344
|
}
|
|
345
345
|
|
|
346
|
-
//
|
|
346
|
+
// keep-if-improved: run any agent/script, gate the result, keep the change ONLY if a metric improved
|
|
347
347
|
function improve(argv) {
|
|
348
348
|
const args = parseArgs(argv);
|
|
349
349
|
if (!args.attempt) { console.error('improve requires --attempt "<command that changes the repo>"'); process.exit(2); }
|
|
@@ -553,7 +553,7 @@ function renderMarkdown(r) {
|
|
|
553
553
|
}
|
|
554
554
|
|
|
555
555
|
function parseArgs(argv) {
|
|
556
|
-
const a = { json: false };
|
|
556
|
+
const a = { json: false, detectGaming: true }; // reward-hack detection is ON by default; --no-detect-gaming opts out
|
|
557
557
|
for (let i = 0; i < argv.length; i++) {
|
|
558
558
|
if (argv[i] === '--base') a.base = argv[++i];
|
|
559
559
|
else if (argv[i] === '--head') a.head = argv[++i];
|
|
@@ -561,6 +561,7 @@ function parseArgs(argv) {
|
|
|
561
561
|
else if (argv[i] === '--working') a.working = true;
|
|
562
562
|
else if (argv[i] === '--no-record') a.noRecord = true;
|
|
563
563
|
else if (argv[i] === '--detect-gaming' || argv[i] === '--antihack') a.detectGaming = true;
|
|
564
|
+
else if (argv[i] === '--no-detect-gaming' || argv[i] === '--no-antihack') a.detectGaming = false;
|
|
564
565
|
else if (argv[i] === '--attempt') a.attempt = argv[++i];
|
|
565
566
|
else if (argv[i] === '--json') a.json = true;
|
|
566
567
|
else if (argv[i] === '--markdown') a.markdown = true;
|
|
@@ -581,13 +582,13 @@ function main() {
|
|
|
581
582
|
else if (cmd === 'guards') guards(rest);
|
|
582
583
|
else {
|
|
583
584
|
console.log([
|
|
584
|
-
'PromptWheel —
|
|
585
|
+
'PromptWheel — catch your agent cheating. Prove a change moved a real metric (and that the agent earned it, not gamed it). The per-turn reward + source-only audit for AI coding loops.',
|
|
585
586
|
'',
|
|
586
587
|
' promptwheel init [--preset <name> | --list] write a starter config for your stack',
|
|
587
588
|
' promptwheel run [--base R] [--head R] [--repeat N] [--json|--markdown]',
|
|
589
|
+
' catches reward-hacking BY DEFAULT — re-proves each win from SOURCE edits alone (verdict GAMED,',
|
|
590
|
+
' exit 2, when a metric only moved by editing tests/config/grader). Use --no-detect-gaming for the bare gate.',
|
|
588
591
|
' promptwheel run --working gate uncommitted changes (incl. newly added files)',
|
|
589
|
-
' promptwheel run --detect-gaming re-prove each win with SOURCE edits alone — catch reward-hacking',
|
|
590
|
-
' (verdict GAMED, exit 2, when a metric only moved by editing tests/config/grader)',
|
|
591
592
|
' promptwheel improve --attempt "<cmd>" run an agent/script; keep only if a metric improved',
|
|
592
593
|
' exit 0=kept · 1=regression · 3=plateau · add --json',
|
|
593
594
|
' promptwheel insights which metrics actually respond (loop memory)',
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.2",
|
|
4
4
|
"description": "Catch your agent cheating — a deterministic, zero-LLM CLI that flags when an AI coding agent gamed its own metric (re-proves the win from the agent's source edits alone). Built on an outcome gate; runs in CI or a Claude Code hook.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|