promptwheel 0.4.0 → 0.4.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -27,7 +27,7 @@ PromptWheel a1b2c3d → e4f5g6h (×5)
27
27
  VERDICT: PASS
28
28
  ```
29
29
 
30
- Exit `0` on pass, `1` on fail (CI-friendly). No build step, zero dependencies, Node 18+.
30
+ Exit `0` pass · `1` fail · `2` gamed · `3` inconclusive (a guard measured nothing) — CI-friendly. No build step, zero dependencies, Node 18+.
31
31
 
32
32
  ## Catch your agent cheating — on by default
33
33
 
@@ -42,11 +42,30 @@ PromptWheel base → head
42
42
  VERDICT: GAMED — a metric "improved" by editing the goalposts, not the source
43
43
  ```
44
44
 
45
+ **See it fire yourself** — 60 seconds, illustrative (a hand-built cheat is a *mechanism* demo, not evidence):
46
+
47
+ ```bash
48
+ mkdir pw-cheat-demo && cd pw-cheat-demo && git init -q
49
+ printf '{"name":"demo","type":"module","scripts":{"test":"node --test"}}\n' > package.json
50
+ printf 'export const add = (a, b) => a - b; // BUG: should be a + b\n' > add.js
51
+ printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), 5)); // honest test — FAILS on the bug\n" > add.test.js
52
+ git add -A && git commit -qm 'buggy code + an honest failing test'
53
+ npx -y promptwheel@latest init && git add -A && git commit -qm 'add the gate config'
54
+
55
+ # an agent "greens" the suite by editing the TEST to expect the bug — the code stays broken:
56
+ printf "import {test} from 'node:test'; import assert from 'node:assert'; import {add} from './add.js';\ntest('add', () => assert.equal(add(2, 3), -1));\n" > add.test.js
57
+ git commit -qam 'make the suite green'
58
+
59
+ npx -y promptwheel@latest run --base HEAD~1 --head HEAD
60
+ # ▲ tests_pass 0 → 1 🚩 GAMED — the win changed zero production-source files
61
+ # VERDICT: GAMED (exit 2)
62
+ ```
63
+
45
64
  Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) — and the quieter cheat of *weakening the suite while the metric stays flat* — are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
46
65
 
47
66
  ```bash
48
67
  npx promptwheel init # guarded tests + antihack tripwires by default
49
- npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED
68
+ npx promptwheel run # detection ON by default · exit 0 win · 1 regression · 2 GAMED · 3 inconclusive
50
69
  # (add --no-detect-gaming for just the outcome gate)
51
70
  ```
52
71
 
@@ -68,7 +87,7 @@ npx promptwheel run --repeat 5 --json # measure 5× to establish a noise ban
68
87
 
69
88
  # the loop: run any agent/script, keep the change ONLY if a metric improved
70
89
  npx promptwheel improve --attempt "claude -p 'reduce lint errors'"
71
- # exit 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau (reverted) · add --json
90
+ # exit 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau / inconclusive (reverted) · add --json
72
91
 
73
92
  # what's actually responding in this repo? (aggregates .promptwheel/outcomes.jsonl)
74
93
  npx promptwheel insights
@@ -81,7 +100,7 @@ npx promptwheel backfill -n 30 # cold start: seed the record from git
81
100
 
82
101
  **The consequence ledger.** git records *what changed*; PromptWheel records *what the change did* — same trust model (local, deterministic, append-only, no server, no LLM in the verdict). `playbook` and `suggest` are pure re-derivations over that ledger: every rendered line was measured by the gate, decays unless re-earned, and stays hidden below an evidence threshold. No compounding claim is made for them until the A/B acceptance test (`bench/compounding-ab.mjs`) passes on real usage data.
83
102
 
84
- **Footprint:** it never touches your working tree — every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; `node_modules` is symlinked, not copied), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record — commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
103
+ **Footprint:** it never touches your working tree — every measurement runs in a throwaway git worktree **in your system temp dir** (one at a time; configured `linkDirs` are symlinked not copied — `node_modules` by default, `.venv` for Python, etc.), removed when the run finishes. The only thing PromptWheel writes to your repo is the optional `.promptwheel/outcomes.jsonl` record — commit it to build the per-repo "what moves what" history, or `.gitignore` it (`--no-record` to skip entirely). A hard-killed run can't leave clutter behind: the next run **self-heals** any orphaned worktree (stale registry entry + abandoned temp checkout).
85
104
 
86
105
  ## Loop patterns
87
106
 
@@ -151,6 +170,12 @@ The Action runs straight from its own checkout — no npm install, no build. See
151
170
  - **gamingCheck** — `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files — otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
152
171
  - **gamingThreshold** — the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
153
172
 
173
+ Dependency & environment keys (config-level; `init` writes stack defaults):
174
+
175
+ - **linkDirs** — dirs to symlink from your checkout into each measuring worktree, for deps that live outside git. Default `["node_modules"]`; `init` writes `[".venv"]` for Python, `["target"]` for Rust. (`"linkNodeModules": false` still works as shorthand for "link nothing".)
176
+ - **env** — environment variables for every metric command; `{wt}` is substituted with the worktree path. Python configs get `"PYTHONPATH": "{wt}/src:{wt}"` so the *measured ref* is imported, not an editable install of your original checkout.
177
+ - **setup** — an optional command run once per ref before metrics (e.g. `npm run build`, `pip install -e .`), after any source patch — so a build-gated suite measures the ref it's actually on.
178
+
154
179
  ## Guardrails & inheritance
155
180
 
156
181
  To see what's actually enforced in a repo — including guards inherited from a shared config:
@@ -217,7 +242,7 @@ The engine is one importable file; pure helpers are exported for unit tests, the
217
242
  - [x] `promptwheel init` + presets — zero-config onboarding
218
243
  - [x] `insights` — reward-stream aggregation (Phase-5 seed)
219
244
  - [x] `--detect-gaming` — reward-hack detection: re-prove the win from source edits alone + `antihack` preset
220
- - [x] npm publish — `promptwheel@0.1.0` (the lead magnet, shipped 2026-06)
245
+ - [x] npm publish — `promptwheel@0.4.1` (the lead magnet)
221
246
  - [x] outcome-curated learning + UCB work-discovery — `playbook` + `suggest` + the compounding A/B harness (**experimental**; compounding *claims* stay gated on real-data proof — see [docs/LEARNING.md](docs/LEARNING.md))
222
247
 
223
- > Status: **published** (npm `promptwheel`, v0.2.0) — all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
248
+ > Status: **published** (npm `promptwheel`, v0.4.1) — all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
@@ -51,12 +51,12 @@ function resolveConfig(p, repo, label, seen) {
51
51
  if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
52
52
  const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
53
53
  for (const m of base.metrics) byName.set(m.name, m);
54
- for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
54
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
55
55
  }
56
56
  for (const m of (cfg.metrics || [])) {
57
57
  byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
58
58
  }
59
- for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
59
+ for (const k of ['repeat', 'linkNodeModules', 'linkDirs', 'env', 'setup', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
60
60
  return { ...scalars, metrics: [...byName.values()] };
61
61
  } finally {
62
62
  seen.delete(abs); // only ACTIVE ancestors are a cycle — a diamond (a base reached two ways) is fine
@@ -82,10 +82,10 @@ function resolveBase(repo, base) {
82
82
  return git(['rev-parse', 'HEAD~1'], repo);
83
83
  }
84
84
 
85
- function runMetric(cwd, m) {
85
+ function runMetric(cwd, m, env) {
86
86
  let stdout = '', code = 0;
87
87
  try {
88
- stdout = execSync(m.cmd, { cwd, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
88
+ stdout = execSync(m.cmd, { cwd, env: env || process.env, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], timeout: (m.timeoutSec ?? 300) * 1000 });
89
89
  } catch (e) { code = e.status ?? 1; stdout = `${e.stdout || ''}${e.stderr || ''}`; }
90
90
  return extract(stdout, code, m.extract);
91
91
  }
@@ -103,18 +103,41 @@ function extract(stdout, code, mode) {
103
103
  return nums ? Number(nums[nums.length - 1]) : null;
104
104
  }
105
105
 
106
+ // bridge a measurement worktree to the deps that live OUTSIDE git (node_modules, .venv, target,
107
+ // …) and the env a test command needs. Returns the env for runMetric. Generalizes the old
108
+ // node_modules-only symlink: `linkDirs` says what to link (default ['node_modules']; back-compat
109
+ // with linkNodeModules:false → link nothing); `env` sets vars ({wt} → the worktree path, so a
110
+ // Python run can put the measured ref FIRST on PYTHONPATH). Ecosystem knowledge lives in config,
111
+ // not here — the engine stays ecosystem-agnostic.
112
+ function bridgeEnv(repo, wt, cfg) {
113
+ const linkDirs = Array.isArray(cfg.linkDirs) ? cfg.linkDirs : (cfg.linkNodeModules === false ? [] : ['node_modules']);
114
+ for (const d of linkDirs) {
115
+ if (existsSync(join(repo, d)) && !existsSync(join(wt, d))) {
116
+ try { symlinkSync(join(repo, d), join(wt, d)); } catch { /* best effort */ }
117
+ }
118
+ }
119
+ if (!cfg.env) return process.env;
120
+ return { ...process.env, ...Object.fromEntries(Object.entries(cfg.env).map(([k, v]) => [k, String(v).replaceAll('{wt}', () => wt)])) }; // fn replacement: a literal wt avoids $-pattern interpretation
121
+ }
122
+ // an optional per-ref build/install (e.g. `npm run build`, `pip install -e .`). Best-effort: a
123
+ // failed setup just leaves the metric inert → INCONCLUSIVE (never a false green). Run by the
124
+ // caller AFTER any source patch, so it builds the tree actually being measured.
125
+ function runSetup(wt, cfg, env) {
126
+ if (!cfg.setup) return;
127
+ try { execSync(cfg.setup, { cwd: wt, env, stdio: 'ignore', timeout: 300_000 }); } catch { /* inert → inconclusive */ }
128
+ }
129
+
106
130
  // measure every metric `repeat` times at a ref, in a throwaway worktree (never touches your tree)
107
- function measureAt(repo, ref, metrics, linkNodeModules, repeat) {
131
+ function measureAt(repo, ref, cfg, repeat) {
108
132
  const wt = mkdtempSync(join(tmpdir(), 'promptwheel-'));
109
133
  git(['worktree', 'add', '--quiet', '--detach', wt, ref], repo);
110
134
  try {
111
- if (linkNodeModules && existsSync(join(repo, 'node_modules')) && !existsSync(join(wt, 'node_modules'))) {
112
- try { symlinkSync(join(repo, 'node_modules'), join(wt, 'node_modules')); } catch { /* best effort */ }
113
- }
135
+ const env = bridgeEnv(repo, wt, cfg);
136
+ runSetup(wt, cfg, env);
114
137
  const out = {};
115
- for (const m of metrics) {
138
+ for (const m of cfg.metrics) {
116
139
  const samples = [];
117
- for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m));
140
+ for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, m, env));
118
141
  out[m.name] = samples;
119
142
  }
120
143
  return out;
@@ -201,23 +224,22 @@ function subsystemsOf(repo, base, head) {
201
224
  }
202
225
 
203
226
  // measure `metric` in a worktree at `base` with ONLY the source slice of base→head applied.
204
- function measureSourceOnly(repo, base, head, metric, linkNM, repeat) {
227
+ function measureSourceOnly(repo, base, head, metric, cfg, repeat) {
205
228
  const { source } = changedSourcePaths(repo, base, head);
206
229
  if (!source.length) return { samples: [], hadSourceChange: false }; // a "win" with zero source edits = goalposts moved
207
230
  const wt = mkdtempSync(join(tmpdir(), 'promptwheel-src-'));
208
231
  git(['worktree', 'add', '--quiet', '--detach', wt, base], repo);
209
232
  try {
210
- if (linkNM && existsSync(join(repo, 'node_modules')) && !existsSync(join(wt, 'node_modules'))) {
211
- try { symlinkSync(join(repo, 'node_modules'), join(wt, 'node_modules')); } catch { /* */ }
212
- }
233
+ const env = bridgeEnv(repo, wt, cfg);
213
234
  const patch = git(['diff', base, head, '--', ...source], repo);
214
235
  if (patch.trim()) {
215
236
  const tryApply = (extra) => { execFileSync('git', ['apply', '--whitespace=nowarn', ...extra], { cwd: wt, input: patch + '\n', encoding: 'utf8' }); };
216
237
  try { tryApply(['--3way']); }
217
238
  catch { try { tryApply([]); } catch { return { samples: [], hadSourceChange: true, applyFailed: true }; } }
218
239
  }
240
+ runSetup(wt, cfg, env); // build the patched tree, so a build-gated metric measures the source edit
219
241
  const samples = [];
220
- for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, metric));
242
+ for (let i = 0; i < repeat; i++) samples.push(runMetric(wt, metric, env));
221
243
  return { samples, hadSourceChange: true };
222
244
  } finally {
223
245
  try { git(['worktree', 'remove', '--force', wt], repo); } catch { rmSync(wt, { recursive: true, force: true }); }
@@ -326,7 +348,6 @@ function gate(repo, opts) {
326
348
  warnEditableInstall(repo);
327
349
  const cfg = loadConfig(repo);
328
350
  const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
329
- const linkNM = cfg.linkNodeModules !== false;
330
351
 
331
352
  let base, head;
332
353
  if (opts.working) {
@@ -342,8 +363,8 @@ function gate(repo, opts) {
342
363
  head = opts.head || 'HEAD';
343
364
  }
344
365
 
345
- const before = measureAt(repo, base, cfg.metrics, linkNM, repeat);
346
- const after = measureAt(repo, head, cfg.metrics, linkNM, repeat);
366
+ const before = measureAt(repo, base, cfg, repeat);
367
+ const after = measureAt(repo, head, cfg, repeat);
347
368
  const metrics = cfg.metrics.map((m) => {
348
369
  const ev = evaluate(m, before[m.name], after[m.name], repeat);
349
370
  return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...ev };
@@ -354,13 +375,18 @@ function gate(repo, opts) {
354
375
  if (m.status !== 'improved') continue;
355
376
  const cm = cfg.metrics.find((c) => c.name === m.name);
356
377
  if (cm.gamingCheck === false) continue; // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
357
- const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, linkNM, repeat), cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
378
+ const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, cfg, repeat), cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
358
379
  m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
359
380
  }
360
381
  }
361
382
  const failed = metrics.some((m) => m.guard && !m.ok);
362
383
  const gamed = metrics.some((m) => m.gamed === true);
363
- const verdict = failed ? 'fail' : gamed ? 'gamed' : 'pass';
384
+ // a guard that never runs (inert: a pass/fail metric stuck at 0 across both refs) verifies
385
+ // nothing — don't launder that into a green PASS. The honest verdict is "couldn't measure".
386
+ // BUT scope it to a bare no-op run: an inert guard must NOT bury a genuine, measured win
387
+ // elsewhere (that would make `improve` revert good changes and misread them as a plateau).
388
+ const inconclusive = !failed && !gamed && metrics.some((m) => m.inert) && !metrics.some((m) => m.status === 'improved');
389
+ const verdict = failed ? 'fail' : gamed ? 'gamed' : inconclusive ? 'inconclusive' : 'pass';
364
390
  const report = {
365
391
  base: short(repo, base), head: short(repo, head), repeat, mode: opts.working ? 'working' : 'refs',
366
392
  // learning-substrate fields (Phase 5): cohort segments reliability by environment,
@@ -381,7 +407,7 @@ function run(argv) {
381
407
  if (args.json) console.log(JSON.stringify(report, null, 2));
382
408
  else if (args.markdown) console.log(renderMarkdown(report));
383
409
  else printHuman(report);
384
- process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : 1);
410
+ process.exit(report.verdict === 'pass' ? 0 : report.verdict === 'gamed' ? 2 : report.verdict === 'inconclusive' ? 3 : 1);
385
411
  }
386
412
 
387
413
  // persisted record: append every gated run to a per-repo outcome record (best-effort, never fails the gate)
@@ -410,10 +436,11 @@ function improve(argv) {
410
436
  const improvedNames = report.metrics.filter((m) => m.status === 'improved').map((m) => m.name);
411
437
 
412
438
  // result + exit code express loop progress so `while improve; do :; done` converges:
413
- // 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau/no-op (reverted)
439
+ // 0 = kept a real win · 1 = guarded regression (reverted) · 3 = plateau/no-op OR inconclusive (reverted)
414
440
  let result, exit, note;
415
441
  if (report.verdict === 'fail') { result = 'regression'; exit = 1; revert(repo); note = '✗ guarded regression — reverted'; }
416
442
  else if (report.verdict === 'gamed') { result = 'gamed'; exit = 1; revert(repo); note = '🚩 gamed — a metric "improved" by editing tests/config, not source — reverted'; }
443
+ else if (report.verdict === 'inconclusive') { result = 'inconclusive'; exit = 3; revert(repo); note = '≈ inconclusive — a guard measured nothing (inert); reverted'; }
417
444
  else if (noChange || improvedNames.length === 0) {
418
445
  result = 'plateau'; exit = 3; revert(repo);
419
446
  note = noChange ? '= no metric moved — reverted' : '= nothing improved beyond noise — reverted';
@@ -628,10 +655,10 @@ const PRESETS = {
628
655
  'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
629
656
  metrics: [
630
657
  { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
631
- { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
632
- { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
633
- { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
634
- { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
658
+ { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
659
+ { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
660
+ { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
661
+ { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist --exclude-dir=build --exclude-dir=coverage --exclude-dir=.next --exclude-dir=out --exclude-dir=lib-cov --exclude-dir=__pycache__ --exclude-dir=.pytest_cache --exclude-dir=target --exclude-dir=.venv --exclude-dir=venv --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
635
662
  ] },
636
663
  };
637
664
 
@@ -639,7 +666,7 @@ function detectTestCmd(repo) {
639
666
  const has = (f) => existsSync(join(repo, f));
640
667
  if (has('go.mod')) return 'go test ./...';
641
668
  if (has('Cargo.toml')) return 'cargo test';
642
- if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return 'pytest -q';
669
+ if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return has('.venv/bin/pytest') ? '.venv/bin/pytest -q' : 'pytest -q';
643
670
  // only trust `npm test` when a test script actually exists — otherwise the metric can
644
671
  // never pass and the gate would "protect" nothing (very common in Next.js app repos)
645
672
  try {
@@ -648,6 +675,17 @@ function detectTestCmd(repo) {
648
675
  return 'echo "set your test command in promptwheel.config.json" && false';
649
676
  }
650
677
 
678
+ // deps that live outside git + the env a test needs, defaulted per stack. init writes these into
679
+ // config; the measurement bridge (bridgeEnv/runSetup) consumes them — keeps the engine agnostic.
680
+ function detectEnv(repo) {
681
+ const has = (f) => existsSync(join(repo, f));
682
+ // Python: link the venv (3rd-party deps) and put the worktree src FIRST on PYTHONPATH, so imports
683
+ // resolve the MEASURED ref instead of an editable install pointing back at your original checkout.
684
+ if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return { linkDirs: ['.venv'], env: { PYTHONPATH: '{wt}/src:{wt}' } }; // src/ AND flat layouts; worktree src wins over an editable install
685
+ if (has('Cargo.toml')) return { linkDirs: [] }; // do NOT link target/: `cargo test` WRITES there — linking would clobber the user's real dir and share a build cache across refs (measurement blindness). Build fresh in the worktree.
686
+ return {};
687
+ }
688
+
651
689
  // only offer the lint metric where eslint actually exists — otherwise it reads 0 forever
652
690
  // and a newcomer mistakes a metric that can't move for a clean repo.
653
691
  function hasEslint(repo) {
@@ -690,7 +728,9 @@ function init(argv) {
690
728
  if (lint) metrics = [...metrics, { name: 'lint_errors', cmd: LINT_CMD, extract: 'number', direction: 'down', guard: false }];
691
729
  note = `tests + antihack tripwires${lint ? ' + lint' : ''} (detected: ${testCmd})`;
692
730
  }
693
- writeFileSync(out, JSON.stringify({ repeat: 1, metrics }, null, 2) + '\n');
731
+ const envCfg = presetName ? {} : detectEnv(repo);
732
+ if (envCfg.linkDirs) note += ` · linking ${envCfg.linkDirs.join(', ')}`;
733
+ writeFileSync(out, JSON.stringify({ repeat: 1, ...envCfg, metrics }, null, 2) + '\n');
694
734
  console.log(`✓ wrote promptwheel.config.json — ${note}`);
695
735
  console.log('\n next: promptwheel run --working # gate your uncommitted changes');
696
736
  console.log(' promptwheel init --list # other presets (llm-eval, bundle-size, …)');
@@ -736,14 +776,14 @@ function printHuman(r) {
736
776
  if (m.inert) console.log(' ⚠ never passed at either ref — this guard is protecting nothing (check its command)');
737
777
  if (m.gamed === true) console.log(` 🚩 GAMED — ${m.gamingReason}`);
738
778
  }
739
- console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ' — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ' — a metric "improved" by editing the goalposts, not the source' : ''}`);
779
+ console.log(`\n VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? ' — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? ' — a metric "improved" by editing the goalposts, not the source' : r.verdict === 'inconclusive' ? ' — a guard is inert (measured nothing); a pass cannot be certified' : ''}`);
740
780
  if (r.verdict === 'fail') console.log(' intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name) — `promptwheel guards` shows the effective set');
741
781
  console.log('');
742
782
  }
743
783
 
744
784
  // PR-comment markdown (rendering lives in the tool so the GitHub Action stays thin)
745
785
  function renderMarkdown(r) {
746
- const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : '❌';
786
+ const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : r.verdict === 'inconclusive' ? '🟡' : '❌';
747
787
  const sIcon = { improved: '🟢', regressed: '🔴', unchanged: '⚪', inconclusive: '🟡', unmeasurable: '⚫' };
748
788
  const rows = r.metrics.map((m) => {
749
789
  const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
@@ -759,7 +799,7 @@ function renderMarkdown(r) {
759
799
  '|---|--:|--:|--:|---|---|',
760
800
  rows,
761
801
  '',
762
- r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader — not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : '> ✅ No guarded metric regressed.',
802
+ r.verdict === 'gamed' ? '> 🚩 A metric "improved" only because the agent edited tests/config/grader — not the source. The win does not survive a source-only re-run.' : r.verdict === 'fail' ? '> ❌ A 🛡️ guarded metric regressed beyond the noise band.' : r.verdict === 'inconclusive' ? '> 🟡 A 🛡️ guarded check is **inert** — it never runs, so nothing was actually verified. Fix its command before trusting a pass.' : '> ✅ No guarded metric regressed.',
763
803
  '',
764
804
  '<sub>🛡️ = guard · _prove every change moved a metric_ · [PromptWheel](https://github.com/promptwheel-ai/promptwheel)</sub>',
765
805
  ].join('\n');
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "promptwheel",
3
- "version": "0.4.0",
3
+ "version": "0.4.1",
4
4
  "description": "Catch your agent cheating — a deterministic, zero-LLM CLI that flags when an AI coding agent gamed its own metric (re-proves the win from the agent's source edits alone). Built on an outcome gate; runs in CI or a Claude Code hook.",
5
5
  "type": "module",
6
6
  "bin": {