promptwheel 0.4.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -6
- package/bin/promptwheel.mjs +71 -31
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -27,7 +27,7 @@ PromptWheel a1b2c3d → e4f5g6h (×5)
|
|
|
27
27
|
VERDICT: PASS
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
Exit `0`
|
|
30
|
+
Exit `0` pass · `1` fail · `2` gamed · `3` inconclusive (a guard measured nothing) — CI-friendly. No build step, zero dependencies, Node 18+.
|
|
31
31
|
|
|
32
32
|
## Catch your agent cheating — on by default
|
|
33
33
|
|
|
@@ -42,11 +42,30 @@ PromptWheel base → head
|
|
|
42
42
|
VERDICT: GAMED — a metric "improved" by editing the goalposts, not the source
|
|
43
43
|
```
|
|
44
44
|
|
|
45
|
+
**See it fire yourself** — 60 seconds, illustrative (a hand-built cheat is a *mechanism* demo, not evidence):
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
mkdir pw-cheat-demo && cd pw-cheat-demo && git init -q
|
|
49
|
+
printf '{"name":"demo","type":"module","scripts":{"test":"node --test"}}\n' > package.json
|
|
50
|
+
printf 'export const add = (a, b) => a - b; // BUG: should be a + b\n' > add.js
|
|
51
|
+
printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), 5)); // honest test — FAILS on the bug\n" > add.test.js
|
|
52
|
+
git add -A && git commit -qm 'buggy code + an honest failing test'
|
|
53
|
+
npx -y promptwheel@latest init && git add -A && git commit -qm 'add the gate config'
|
|
54
|
+
|
|
55
|
+
# an agent "greens" the suite by editing the TEST to expect the bug — the code stays broken:
|
|
56
|
+
printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), -1));\n" > add.test.js
|
|
57
|
+
git commit -qam 'make the suite green'
|
|
58
|
+
|
|
59
|
+
npx -y promptwheel@latest run --base HEAD~1 --head HEAD
|
|
60
|
+
# ▲ tests_pass 0 → 1 🚩 GAMED — the win changed zero production-source files
|
|
61
|
+
# VERDICT: GAMED (exit 2)
|
|
62
|
+
```
|
|
63
|
+
|
|
45
64
|
Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) — and the quieter cheat of *weakening the suite while the metric stays flat* — are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
|
|
46
65
|
|
|
47
66
|
```bash
|
|
48
67
|
npx promptwheel init # guarded tests + antihack tripwires by default
|
|
49
|
-
npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED
|
|
68
|
+
npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED · 3 inconclusive
|
|
50
69
|
# (add --no-detect-gaming for just the outcome gate)
|
|
51
70
|
```
|
|
52
71
|
|
|
@@ -68,7 +87,7 @@ npx promptwheel run --repeat 5 --json # measure 5× to establish a noise ban
|
|
|
68
87
|
|
|
69
88
|
# the loop: run any agent/script, keep the change ONLY if a metric improved
|
|
70
89
|
npx promptwheel improve --attempt "claude -p 'reduce lint errors'"
|
|
71
|
-
# exit 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau (reverted) · add --json
|
|
90
|
+
# exit 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau / inconclusive (reverted) · add --json
|
|
72
91
|
|
|
73
92
|
# what's actually responding in this repo? (aggregates .promptwheel/outcomes.jsonl)
|
|
74
93
|
npx promptwheel insights
|
|
@@ -81,7 +100,7 @@ npx promptwheel backfill -n 30 # cold start: seed the record from git
|
|
|
81
100
|
|
|
82
101
|
**The consequence ledger.** git records *what changed*; PromptWheel records *what the change did* — same trust model (local, deterministic, append-only, no server, no LLM in the verdict). `playbook` and `suggest` are pure re-derivations over that ledger: every rendered line was measured by the gate, decays unless re-earned, and stays hidden below an evidence threshold. No compounding claim is made for them until the A/B acceptance test (`bench/compounding-ab.mjs`) passes on real usage data.
|
|
83
102
|
|
|
84
|
-
**Footprint:** it never touches your working tree — every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; `
|
|
103
|
+
**Footprint:** it never touches your working tree — every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied — `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record — commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
|
|
85
104
|
|
|
86
105
|
## Loop patterns
|
|
87
106
|
|
|
@@ -151,6 +170,12 @@ The Action runs straight from its own checkout — no npm install, no build. See
|
|
|
151
170
|
- **gamingCheck** — `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files — otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
|
|
152
171
|
- **gamingThreshold** — the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
|
|
153
172
|
|
|
173
|
+
Dependency & environment keys (config-level; `init` writes stack defaults):
|
|
174
|
+
|
|
175
|
+
- **linkDirs** — dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python, `["target"]` for Rust. (`"linkNodeModules": false` still works as shorthand for "link nothing".)
|
|
176
|
+
- **env** — environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
|
|
177
|
+
- **setup** — an optional command run once per ref before metrics (e.g. `npm run build`, `pip install -e .`), after any source patch — so a build-gated suite measures the ref it's actually on.
|
|
178
|
+
|
|
154
179
|
## Guardrails & inheritance
|
|
155
180
|
|
|
156
181
|
To see what's actually enforced in a repo — including guards inherited from a shared config:
|
|
@@ -217,7 +242,7 @@ The engine is one importable file; pure helpers are exported for unit tests, the
|
|
|
217
242
|
- [x] `promptwheel init` + presets — zero-config onboarding
|
|
218
243
|
- [x] `insights` — reward-stream aggregation (Phase-5 seed)
|
|
219
244
|
- [x] `--detect-gaming` — reward-hack detection: re-prove the win from source edits alone + `antihack` preset
|
|
220
|
-
- [x] npm publish — `promptwheel@0.1
|
|
245
|
+
- [x] npm publish — `promptwheel@0.4.1` (the lead magnet)
|
|
221
246
|
- [x] outcome-curated learning + UCB work-discovery — `playbook` + `suggest` + the compounding A/B harness (**experimental**; compounding *claims* stay gated on real-data proof — see [docs/LEARNING.md](docs/LEARNING.md))
|
|
222
247
|
|
|
223
|
-
> Status: **published** (npm `promptwheel`, v0.
|
|
248
|
+
> Status: **published** (npm `promptwheel`, v0.4.1) — all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -51,12 +51,12 @@ function resolveConfig(p, repo, label, seen) {
|
|
|
51
51
|
if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
|
|
52
52
|
const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
|
|
53
53
|
for (const m of base.metrics) byName.set(m.name, m);
|
|
54
|
-
for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
|
|
54
|
+
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
|
|
55
55
|
}
|
|
56
56
|
for (const m of (cfg.metrics || [])) {
|
|
57
57
|
byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
|
|
58
58
|
}
|
|
59
|
-
for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
|
|
59
|
+
for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
|
|
60
60
|
return { ...scalars, metrics: [...byName.values()] };
|
|
61
61
|
} finally {
|
|
62
62
|
seen.delete(abs); // only ACTIVE ancestors are a cycle — a diamond (a base reached two ways) is fine
|
|
@@ -82,10 +82,10 @@ function resolveBase(repo, base) {
|
|
|
82
82
|
return git(['rev-parse', 'HEAD~1'], repo);
|
|
83
83
|
}
|
|
84
84
|
|
|
85
|
-
function runMetric(cwd, m) {
|
|
85
|
+
function runMetric(cwd, m, env) {
|
|
86
86
|
let stdout = '', code = 0;
|
|
87
87
|
try {
|
|
88
|
-
stdout = execSync(m.cmd, { cwd, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
|
|
88
|
+
stdout = execSync(m.cmd, { cwd, env: env || process.env, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
|
|
89
89
|
} catch (e) { code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
|
|
90
90
|
return extract(stdout, code, m.extract);
|
|
91
91
|
}
|
|
@@ -103,18 +103,41 @@ function extract(stdout, code, mode) {
|
|
|
103
103
|
return nums ? Number(nums[nums.length - 1]) : null;
|
|
104
104
|
}
|
|
105
105
|
|
|
106
|
+
// bridge a measurement worktree to the deps that live OUTSIDE git (node_modules, .venv, target,
|
|
107
|
+
// …) and the env a test command needs. Returns the env for runMetric. Generalizes the old
|
|
108
|
+
// node_modules-only symlink: `linkDirs` says what to link (default ['node_modules']; back-compat
|
|
109
|
+
// with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
|
|
110
|
+
// Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
|
|
111
|
+
// not here — the engine stays ecosystem-agnostic.
|
|
112
|
+
function bridgeEnv(repo, wt, cfg) {
|
|
113
|
+
const linkDirs = Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']);
|
|
114
|
+
for (const d of linkDirs) {
|
|
115
|
+
if (existsSync(join(repo, d)) && !existsSync(join(wt, d))) {
|
|
116
|
+
try { symlinkSync(join(repo, d), join(wt, d)); } catch { /* best effort */ }
|
|
117
|
+
}
|
|
118
|
+
}
|
|
119
|
+
if (!cfg.env) return process.env;
|
|
120
|
+
return { ...process.env, ...Object.fromEntries(Object.entries(cfg.env).map(([k, v]) => [k, String(v).replaceAll('{wt}', () => wt)])) }; // fn replacement: a literal wt avoids $-pattern interpretation
|
|
121
|
+
}
|
|
122
|
+
// an optional per-ref build/install (e.g. `npm run build`, `pip install -e .`). Best-effort: a
|
|
123
|
+
// failed setup just leaves the metric inert → INCONCLUSIVE (never a false green). Run by the
|
|
124
|
+
// caller AFTER any source patch, so it builds the tree actually being measured.
|
|
125
|
+
function runSetup(wt, cfg, env) {
|
|
126
|
+
if (!cfg.setup) return;
|
|
127
|
+
try { execSync(cfg.setup, { cwd: wt, env, stdio: 'ignore', timeout: 300_000 }); } catch { /* inert → inconclusive */ }
|
|
128
|
+
}
|
|
129
|
+
|
|
106
130
|
// measure every metric `repeat` times at a ref, in a throwaway worktree (never touches your tree)
|
|
107
|
-
function measureAt(repo, ref,
|
|
131
|
+
function measureAt(repo, ref, cfg, repeat) {
|
|
108
132
|
const wt = mkdtempSync(join(tmpdir(), 'promptwheel-'));
|
|
109
133
|
git(['worktree', 'add', '--quiet', '--detach', wt, ref], repo);
|
|
110
134
|
try {
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
}
|
|
135
|
+
const env = bridgeEnv(repo, wt, cfg);
|
|
136
|
+
runSetup(wt, cfg, env);
|
|
114
137
|
const out = {};
|
|
115
|
-
for (const m of metrics) {
|
|
138
|
+
for (const m of cfg.metrics) {
|
|
116
139
|
const samples = [];
|
|
117
|
-
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m));
|
|
140
|
+
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
|
|
118
141
|
out[m.name] = samples;
|
|
119
142
|
}
|
|
120
143
|
return out;
|
|
@@ -201,23 +224,22 @@ function subsystemsOf(repo, base, head) {
|
|
|
201
224
|
}
|
|
202
225
|
|
|
203
226
|
// measure `metric` in a worktree at `base` with ONLY the source slice of base→head applied.
|
|
204
|
-
function measureSourceOnly(repo, base, head, metric,
|
|
227
|
+
function measureSourceOnly(repo, base, head, metric, cfg, repeat) {
|
|
205
228
|
const { source } = changedSourcePaths(repo, base, head);
|
|
206
229
|
if (!source.length) return { samples: [], hadSourceChange: false }; // a "win" with zero source edits = goalposts moved
|
|
207
230
|
const wt = mkdtempSync(join(tmpdir(), 'promptwheel-src-'));
|
|
208
231
|
git(['worktree', 'add', '--quiet', '--detach', wt, base], repo);
|
|
209
232
|
try {
|
|
210
|
-
|
|
211
|
-
try { symlinkSync(join(repo, 'node_modules'), join(wt, 'node_modules')); } catch { /* */ }
|
|
212
|
-
}
|
|
233
|
+
const env = bridgeEnv(repo, wt, cfg);
|
|
213
234
|
const patch = git(['diff', base, head, '--', ...source], repo);
|
|
214
235
|
if (patch.trim()) {
|
|
215
236
|
const tryApply = (extra) => { execFileSync('git', ['apply', '--whitespace=nowarn', ...extra], { cwd: wt, input: patch + '\n', encoding: 'utf8' }); };
|
|
216
237
|
try { tryApply(['--3way']); }
|
|
217
238
|
catch { try { tryApply([]); } catch { return { samples: [], hadSourceChange: true, applyFailed: true }; } }
|
|
218
239
|
}
|
|
240
|
+
runSetup(wt, cfg, env); // build the patched tree, so a build-gated metric measures the source edit
|
|
219
241
|
const samples = [];
|
|
220
|
-
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, metric));
|
|
242
|
+
for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, metric, env));
|
|
221
243
|
return { samples, hadSourceChange: true };
|
|
222
244
|
} finally {
|
|
223
245
|
try { git(['worktree', 'remove', '--force', wt], repo); } catch { rmSync(wt, { recursive: true, force: true }); }
|
|
@@ -326,7 +348,6 @@ function gate(repo, opts) {
|
|
|
326
348
|
warnEditableInstall(repo);
|
|
327
349
|
const cfg = loadConfig(repo);
|
|
328
350
|
const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
|
|
329
|
-
const linkNM = cfg.linkNodeModules !== false;
|
|
330
351
|
|
|
331
352
|
let base, head;
|
|
332
353
|
if (opts.working) {
|
|
@@ -342,8 +363,8 @@ function gate(repo, opts) {
|
|
|
342
363
|
head = opts.head || 'HEAD';
|
|
343
364
|
}
|
|
344
365
|
|
|
345
|
-
const before = measureAt(repo, base, cfg
|
|
346
|
-
const after = measureAt(repo, head, cfg
|
|
366
|
+
const before = measureAt(repo, base, cfg, repeat);
|
|
367
|
+
const after = measureAt(repo, head, cfg, repeat);
|
|
347
368
|
const metrics = cfg.metrics.map((m) => {
|
|
348
369
|
const ev = evaluate(m, before[m.name], after[m.name], repeat);
|
|
349
370
|
return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...ev };
|
|
@@ -354,13 +375,18 @@ function gate(repo, opts) {
|
|
|
354
375
|
if (m.status !== 'improved') continue;
|
|
355
376
|
const cm = cfg.metrics.find((c) => c.name === m.name);
|
|
356
377
|
if (cm.gamingCheck === false) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
|
|
357
|
-
const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm,
|
|
378
|
+
const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, cfg, repeat), cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
|
|
358
379
|
m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
|
|
359
380
|
}
|
|
360
381
|
}
|
|
361
382
|
const failed = metrics.some((m) => m.guard && !m.ok);
|
|
362
383
|
const gamed = metrics.some((m) => m.gamed === true);
|
|
363
|
-
|
|
384
|
+
// a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
|
|
385
|
+
// nothing — don't launder that into a green PASS. The honest verdict is "couldn't measure".
|
|
386
|
+
// BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
|
|
387
|
+
// elsewhere (that would make `improve` revert good changes and misread them as a plateau).
|
|
388
|
+
const inconclusive = !failed && !gamed && metrics.some((m) => m.inert) && !metrics.some((m) => m.status === 'improved');
|
|
389
|
+
const verdict = failed ? 'fail' : gamed ? 'gamed' : inconclusive ? 'inconclusive' : 'pass';
|
|
364
390
|
const report = {
|
|
365
391
|
base: short(repo, base), head: short(repo, head), repeat, mode: opts.working ? 'working' : 'refs',
|
|
366
392
|
// learning-substrate fields (Phase 5): cohort segments reliability by environment,
|
|
@@ -381,7 +407,7 @@ function run(argv) {
|
|
|
381
407
|
if (args.json) console.log(JSON.stringify(report, null, 2));
|
|
382
408
|
else if (args.markdown) console.log(renderMarkdown(report));
|
|
383
409
|
else printHuman(report);
|
|
384
|
-
process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : 1);
|
|
410
|
+
process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : report.verdict === 'inconclusive' ? 3 : 1);
|
|
385
411
|
}
|
|
386
412
|
|
|
387
413
|
// persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
|
|
@@ -410,10 +436,11 @@ function improve(argv) {
|
|
|
410
436
|
const improvedNames = report.metrics.filter((m) => m.status === 'improved').map((m) => m.name);
|
|
411
437
|
|
|
412
438
|
// result + exit code express loop progress so `while improve; do :; done` converges:
|
|
413
|
-
// 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau/no-op (reverted)
|
|
439
|
+
// 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau/no-op OR inconclusive (reverted)
|
|
414
440
|
let result, exit, note;
|
|
415
441
|
if (report.verdict === 'fail') { result = 'regression'; exit = 1; revert(repo); note = '✗ guarded regression — reverted'; }
|
|
416
442
|
else if (report.verdict === 'gamed') { result = 'gamed'; exit = 1; revert(repo); note = '🚩 gamed — a metric "improved" by editing tests/config, not source — reverted'; }
|
|
443
|
+
else if (report.verdict === 'inconclusive') { result = 'inconclusive'; exit = 3; revert(repo); note = '≈ inconclusive — a guard measured nothing (inert); reverted'; }
|
|
417
444
|
else if (noChange || improvedNames.length === 0) {
|
|
418
445
|
result = 'plateau'; exit = 3; revert(repo);
|
|
419
446
|
note = noChange ? '= no metric moved — reverted' : '= nothing improved beyond noise — reverted';
|
|
@@ -628,10 +655,10 @@ const PRESETS = {
|
|
|
628
655
|
'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
|
|
629
656
|
metrics: [
|
|
630
657
|
{ name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
|
|
631
|
-
{ name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
632
|
-
{ name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
633
|
-
{ name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
634
|
-
{ name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
658
|
+
{ name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
659
|
+
{ name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
660
|
+
{ name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
|
|
661
|
+
{ name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
|
|
635
662
|
] },
|
|
636
663
|
};
|
|
637
664
|
|
|
@@ -639,7 +666,7 @@ function detectTestCmd(repo) {
|
|
|
639
666
|
const has = (f) => existsSync(join(repo, f));
|
|
640
667
|
if (has('go.mod')) return 'go test ./...';
|
|
641
668
|
if (has('Cargo.toml')) return 'cargo test';
|
|
642
|
-
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return 'pytest -q';
|
|
669
|
+
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return has('.venv/bin/pytest') ? '.venv/bin/pytest -q' : 'pytest -q';
|
|
643
670
|
// only trust `npm test` when a test script actually exists — otherwise the metric can
|
|
644
671
|
// never pass and the gate would "protect" nothing (very common in Next.js app repos)
|
|
645
672
|
try {
|
|
@@ -648,6 +675,17 @@ function detectTestCmd(repo) {
|
|
|
648
675
|
return 'echo "set your test command in promptwheel.config.json" && false';
|
|
649
676
|
}
|
|
650
677
|
|
|
678
|
+
// deps that live outside git + the env a test needs, defaulted per stack. init writes these into
|
|
679
|
+
// config; the measurement bridge (bridgeEnv/runSetup) consumes them — keeps the engine agnostic.
|
|
680
|
+
function detectEnv(repo) {
|
|
681
|
+
const has = (f) => existsSync(join(repo, f));
|
|
682
|
+
// Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
|
|
683
|
+
// resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
|
|
684
|
+
if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
|
|
685
|
+
if (has('Cargo.toml')) return { linkDirs: [] }; // do NOT link target/: `cargo test` WRITES there — linking would clobber the user's real dir and share a build cache across refs (measurement blindness). Build fresh in the worktree.
|
|
686
|
+
return {};
|
|
687
|
+
}
|
|
688
|
+
|
|
651
689
|
// only offer the lint metric where eslint actually exists — otherwise it reads 0 forever
|
|
652
690
|
// and a newcomer mistakes a metric that can't move for a clean repo.
|
|
653
691
|
function hasEslint(repo) {
|
|
@@ -690,7 +728,9 @@ function init(argv) {
|
|
|
690
728
|
if (lint) metrics = [...metrics, { name: 'lint_errors', cmd: LINT_CMD, extract: 'number', direction: 'down', guard: false }];
|
|
691
729
|
note = `tests + antihack tripwires${lint ? ' + lint' : ''} (detected: ${testCmd})`;
|
|
692
730
|
}
|
|
693
|
-
|
|
731
|
+
const envCfg = presetName ? {} : detectEnv(repo);
|
|
732
|
+
if (envCfg.linkDirs) note += ` · linking ${envCfg.linkDirs.join(', ')}`;
|
|
733
|
+
writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, metrics }, null, 2) + '\n');
|
|
694
734
|
console.log(`✓ wrote promptwheel.config.json — ${note}`);
|
|
695
735
|
console.log('\n next: promptwheel run --working # gate your uncommitted changes');
|
|
696
736
|
console.log(' promptwheel init --list # other presets (llm-eval, bundle-size, …)');
|
|
@@ -736,14 +776,14 @@ function printHuman(r) {
|
|
|
736
776
|
if (m.inert) console.log(' ⚠ never passed at either ref — this guard is protecting nothing (check its command)');
|
|
737
777
|
if (m.gamed === true) console.log(` 🚩 GAMED — ${m.gamingReason}`);
|
|
738
778
|
}
|
|
739
|
-
console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ' — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ' — a metric "improved" by editing the goalposts, not the source' : ''}`);
|
|
779
|
+
console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ' — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ' — a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ' — a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
|
|
740
780
|
if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name) — `promptwheel guards` shows the effective set');
|
|
741
781
|
console.log('');
|
|
742
782
|
}
|
|
743
783
|
|
|
744
784
|
// PR-comment markdown (rendering lives in the tool so the GitHub Action stays thin)
|
|
745
785
|
function renderMarkdown(r) {
|
|
746
|
-
const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : '❌';
|
|
786
|
+
const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : r.verdict === 'inconclusive' ? '🟡' : '❌';
|
|
747
787
|
const sIcon = { improved: '🟢', regressed: '🔴', unchanged: '⚪', inconclusive: '🟡', unmeasurable: '⚫' };
|
|
748
788
|
const rows = r.metrics.map((m) => {
|
|
749
789
|
const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
|
|
@@ -759,7 +799,7 @@ function renderMarkdown(r) {
|
|
|
759
799
|
'|---|--:|--:|--:|---|---|',
|
|
760
800
|
rows,
|
|
761
801
|
'',
|
|
762
|
-
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader — not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : '> ✅ No guarded metric regressed.',
|
|
802
|
+
r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader — not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert** — it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
|
|
763
803
|
'',
|
|
764
804
|
'<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
|
|
765
805
|
].join('\n');
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.4.
|
|
3
|
+
"version": "0.4.1",
|
|
4
4
|
"description": "Catch your agent cheating — a deterministic, zero-LLM CLI that flags when an AI coding agent gamed its own metric (re-proves the win from the agent's source edits alone). Built on an outcome gate; runs in CI or a Claude Code hook.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|