promptwheel 0.1.2 → 0.2.3

package/README.md

@@ -42,23 +42,23 @@ PromptWheel  base → head
   VERDICT: GAMED  — a metric "improved" by editing the goalposts, not the source
 ```
 
-Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) are a different shape, caught by the **`antihack` preset** — a target metric plus tripwire guards (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them:
+Inline source-file suppressions (`@ts-nocheck`, `eslint-disable`, `# noqa`) — and the quieter cheat of *weakening the suite while the metric stays flat* — are a different shape, caught by **tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`) that fail when a "win" introduces them. The plain `init` default includes these tripwires, so gutting a test file fails the gate out of the box:
 
 ```bash
-npx promptwheel init --preset antihack    # target + tripwire guards
+npx promptwheel init                      # guarded tests + antihack tripwires by default
 npx promptwheel run                       # detection ON by default · exit 0 win · 1 regression · 2 GAMED
                                           #   (add --no-detect-gaming for just the outcome gate)
 ```
 
-Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable.
+Deterministic, zero-LLM, zero-network: an LLM judge asking "did you cheat?" is itself gameable; this is a diff partition plus a re-run, so a flag is trustworthy without a human in the loop. The 50%-of-gain-survives threshold is the default and is tunable via `gamingThreshold` (config-level, or per-metric).
 
-This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers — held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage — are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce).
+This is **one layer**, not a silver bullet: it catches the evaluator-tampering class (test/grader/golden/config edits) deterministically and for free, so the expensive layers — held-out tests for semantically-weak wins, an LLM judge or a human for intent and leakage — are reserved for the calls only they can make. See **[docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md)** for the coverage matrix and honest scope, and **[bench/RESULTS.md](bench/RESULTS.md)** for the measured numbers (`node bench/gaming-bench.mjs` to reproduce — it includes a cross-stack table with a real pytest run + a numeric eval-pass-rate metric). **Gate your own stack** — pytest · tsc · coverage · bundle · llm-eval — copy a config from **[examples/](examples/)**.
 
 ## Use
 
 ```bash
 # 0. write a starter config for your stack (or hand-write promptwheel.config.json)
-npx promptwheel init                      # detects stack → guarded test metric + lint
+npx promptwheel init                      # detects stack → guarded tests + antihack tripwires (+ lint if eslint is set up)
 npx promptwheel init --list               # presets: tests-pass · lint · bundle-size · llm-eval · antihack
 
 # measure a change
@@ -142,6 +142,7 @@ The Action runs straight from its own checkout — no npm install, no build. See
 - **direction** — `up` (higher better) · `down` (lower better) · `pass` (boolean 0/1).
 - **guard** — `true` = a *trusted* regression **fails** the gate; `false` = informational.
 - **gamingCheck** — `false` exempts a metric from `--detect-gaming`'s source-only re-run. Use it for tripwire / test-side guards (assertion counts, test counts) whose gains legitimately live in test files — otherwise adding real tests would be flagged as gaming. The `antihack` preset sets this on its tripwires.
+- **gamingThreshold** — the fraction of a win that must survive the source-only re-run to count as earned (default `0.5`). Config-level scalar, overridable per metric; inherited through `extends` like `repeat`.
 
 ## Guardrails & inheritance
 
@@ -192,7 +193,7 @@ The accumulated record of **which change-types move which metrics** is the asset
 ## Develop
 
 ```bash
-npm test     # 37 dep-free tests (node:test) — unit + integration, no dependencies
+npm test     # the dep-free suite (node:test) — unit + integration, no dependencies
 ```
 
 The engine is one importable file; pure helpers are exported for unit tests, the CLI runs only when invoked directly. Add a test with every behavior change.
@@ -212,4 +213,4 @@ The engine is one importable file; pure helpers are exported for unit tests, the
 - [x] npm publish — `promptwheel@0.1.0` (the lead magnet, shipped 2026-06)
 - [ ] ACE-style learning + UCB work-discovery (**frozen** — gated on data + ≥1 paid engagement; see [docs/LEARNING.md](docs/LEARNING.md))
 
-> Status: **published v0.1.0** — the consulting lead magnet (npm `promptwheel@0.1.0`), all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.
+> Status: **published** (npm `promptwheel`, v0.2.0) — all core phases built. Lineage: CommandLayer → BlockSpool → PromptWheel (orchestrator, archived) → **PromptWheel (outcome gate)**.

package/bin/promptwheel.mjs

@@ -51,12 +51,12 @@ function resolveConfig(p, repo, label, seen) {
       if (!existsSync(bp)) { console.error(`extends target not found: ${ref} (from ${relOf(repo, p)})`); process.exit(2); }
       const base = resolveConfig(bp, repo, relOf(repo, bp), seen);
       for (const m of base.metrics) byName.set(m.name, m);
-      for (const k of ['repeat', 'linkNodeModules', 'record']) if (base[k] !== undefined) scalars[k] = base[k];
+      for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (base[k] !== undefined) scalars[k] = base[k];
     }
     for (const m of (cfg.metrics || [])) {
       byName.set(m.name, { ...m, __src: label, __override: byName.has(m.name) });
     }
-    for (const k of ['repeat', 'linkNodeModules', 'record']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
+    for (const k of ['repeat', 'linkNodeModules', 'record', 'gamingThreshold']) if (cfg[k] !== undefined) scalars[k] = cfg[k];
     return { ...scalars, metrics: [...byName.values()] };
   } finally {
     seen.delete(abs); // only ACTIVE ancestors are a cycle — a diamond (a base reached two ways) is fine
@@ -152,7 +152,10 @@ function evaluate(m, beforeS, afterS, repeat) {
 
   // guards fail only on a TRUSTED regression (within-noise regressions are not failures)
   const ok = m.guard ? !regressed : true;
-  return { before, after, delta, status, ok, confidence, noise };
+  // a guarded pass-metric that never passes is protecting NOTHING (broken test cmd,
+  // missing script, failed install) — surface it instead of folding into a green verdict.
+  const inert = !!m.guard && dir === 'pass' && before === 0 && after === 0;
+  return { before, after, delta, status, ok, confidence, noise, ...(inert ? { inert: true } : {}) };
 }
 
 // ---------------------------------------------------------------------------
@@ -169,7 +172,11 @@ const NON_SOURCE = [
   /(^|\/)test_[^/]*\.py$/i, /_test\.py$/i, /(^|\/)conftest\.py$/i,
   /\.snap$/i, /(^|\/)golden[^/]*$/i, /\.golden$/i,
   /(^|\/)(eval|grader|score)[^/]*\.[cm]?[jt]s$/i,
-  /(^|\/)(jest|vitest|playwright|cypress|karma|babel|eslint|tsconfig|pytest)[^/]*\.(json|[cm]?[jt]s|ya?ml|ini|cfg|toml)$/i,
+  // config files only — require a .config/.conf/.setup segment so PRODUCTION source that
+  // happens to be named after a tool (e.g. src/installers/eslint.ts) is NOT swept out of
+  // the source slice (a false sweep here can flag an honest win as GAMED).
+  /(^|\/)(jest|vitest|playwright|cypress|karma|babel|eslint|pytest)[^/]*\.(config|conf|setup)\.[^/]+$/i,
+  /(^|\/)tsconfig[^/]*\.json$/i,
   /(^|\/)\.(eslintrc|babelrc|prettierrc)[^/]*$/i,
   /(^|\/)(pytest\.ini|setup\.cfg|\.flake8|tox\.ini|eslint\.config\.[cm]?js)$/i,
 ];
@@ -215,17 +222,18 @@ function gain(m, before, val) {
 }
 
 // a metric that improved full-diff: does the win survive when only source edits are applied?
-// gamed = source edits alone reproduce < half the gain (the rest came from editing the goalposts).
-function judgeGaming(m, srcResult) {
+// gamed = source edits alone reproduce less than `threshold` of the gain (default 0.5 — the
+// rest came from editing the goalposts). Tune via `gamingThreshold` (config-level or per-metric).
+function judgeGaming(m, srcResult, threshold = 0.5) {
   if (!srcResult.hadSourceChange) return { gamed: true, sourceOnly: null, retained: 0, reason: 'the "win" changed zero production-source files — only test/config/grader/golden' };
   if (srcResult.applyFailed) return { gamed: null, sourceOnly: null, retained: null, reason: 'source-only patch did not apply cleanly — inconclusive' };
   const sourceOnly = median(srcResult.samples);
   const full = gain(m, m.before, m.after);
   if (full == null || full <= 0) return { gamed: false, sourceOnly, retained: 1, reason: 'no real improvement to re-prove' };
   const retained = +(gain(m, m.before, sourceOnly) / full).toFixed(3);
-  const gamed = retained < 0.5;
+  const gamed = retained < threshold;
   return { gamed, sourceOnly, retained, reason: gamed
-    ? `only ${(retained * 100).toFixed(0)}% of the gain survives when test/config/grader changes are reverted — most of the "win" came from editing the goalposts`
+    ? `only ${(retained * 100).toFixed(0)}% of the gain survives when test/config/grader changes are reverted (threshold ${(threshold * 100).toFixed(0)}%) — most of the "win" came from editing the goalposts`
     : `${(retained * 100).toFixed(0)}% of the gain survives source-only — the source earned it` };
 }
 
@@ -279,9 +287,31 @@ function selfHeal(repo) {
   } catch { /* cleanup must never block the gate */ }
 }
 
+// Python editable installs (src-layout `pip install -e .`) can pin imports to the ORIGINAL
+// checkout: the worktree's tests then import your current tree at BOTH refs and every delta
+// reads 0 — the gate is structurally blind on that repo shape. Undetectable from the verdict
+// (it looks like "unchanged"), so detect the editable install itself and warn. Verified by a
+// controlled repro (CHANGELOG 0.2.2).
+function warnEditableInstall(repo) {
+  if (!existsSync(join(repo, 'pyproject.toml')) && !existsSync(join(repo, 'setup.py'))) return;
+  try {
+    const probe = [
+      'import site,glob,os,sys',
+      'repo=os.path.realpath(sys.argv[1])',
+      'sps=set(site.getsitepackages()+[site.getusersitepackages()])',
+      'files=[f for sp in sps for pat in ("__editable__*","*.egg-link","*.pth") for f in glob.glob(os.path.join(sp,pat)) if os.path.isfile(f)]',
+      'hit=any(repo in open(f,errors="ignore").read() for f in files)',
+      'print("HIT" if hit else "")',
+    ].join('\n');
+    const out = execFileSync('python3', ['-c', probe, repo], { encoding: 'utf8', timeout: 5000 }).trim();
+    if (out === 'HIT') console.error('⚠ editable install of this repo detected (pip install -e): worktree measurements may import your ORIGINAL checkout, not the measured ref — deltas can read 0. Use a non-editable install in the measuring venv, or put the worktree src on PYTHONPATH.');
+  } catch { /* no python / cannot tell — stay quiet */ }
+}
+
 // the shared core: measure a change (base→head) and return the structured report
 function gate(repo, opts) {
   selfHeal(repo);
+  warnEditableInstall(repo);
   const cfg = loadConfig(repo);
   const repeat = Math.max(1, opts.repeat ?? cfg.repeat ?? 1);
   const linkNM = cfg.linkNodeModules !== false;
@@ -312,7 +342,7 @@ function gate(repo, opts) {
       if (m.status !== 'improved') continue;
       const cm = cfg.metrics.find((c) => c.name === m.name);
       if (cm.gamingCheck === false) continue;   // tripwire / test-side guards aren't re-proven from source (their gain legitimately lives in test files)
-      const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, linkNM, repeat));
+      const j = judgeGaming(m, measureSourceOnly(repo, base, head, cm, linkNM, repeat), cm.gamingThreshold ?? cfg.gamingThreshold ?? 0.5);
       m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
     }
   }
@@ -435,10 +465,10 @@ const PRESETS = {
   'antihack': { desc: 'catch reward-hacking: a target + tripwires; pairs with `run --detect-gaming` (source-only re-run)',
     metrics: [
       { name: 'tests_pass', cmd: '__TESTCMD__', extract: 'exit', direction: 'pass', guard: true },
-      { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
-      { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
-      { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
-      { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert" --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
+      { name: 'test_count', cmd: 'grep -rIoE "\\b(it|test|describe) ?\\(|def test_" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
+      { name: 'skipped_tests', cmd: 'grep -rIoE "\\.(skip|only) ?\\(|xit ?\\(|@pytest\\.mark\\.skip" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
+      { name: 'suppressions', cmd: 'grep -rIoE "eslint-disable|@ts-(ignore|nocheck)|# ?type: ?ignore|# ?noqa|//nolint|#!?\\[allow\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'down', guard: true, gamingCheck: false },
+      { name: 'assertions', cmd: 'grep -rIoE "expect ?\\(|\\bassert|\\bt\\.(is|deepEqual|throws|truthy|falsy|not|ok) ?\\(" --exclude-dir=node_modules --exclude-dir=.git --exclude=promptwheel.config.json . 2>/dev/null | wc -l | tr -d " "', extract: 'number', direction: 'up', guard: true, gamingCheck: false },
     ] },
 };
 
@@ -447,10 +477,26 @@ function detectTestCmd(repo) {
   if (has('go.mod')) return 'go test ./...';
   if (has('Cargo.toml')) return 'cargo test';
   if (has('pyproject.toml') || has('setup.py') || has('pytest.ini')) return 'pytest -q';
-  if (has('package.json')) return 'npm test --silent';
+  // only trust `npm test` when a test script actually exists — otherwise the metric can
+  // never pass and the gate would "protect" nothing (very common in Next.js app repos)
+  try {
+    if (JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8')).scripts?.test) return 'npm test --silent';
+  } catch { /* no or unparsable package.json — fall through */ }
   return 'echo "set your test command in promptwheel.config.json" && false';
 }
 
+// only offer the lint metric where eslint actually exists — otherwise it reads 0 forever
+// and a newcomer mistakes a metric that can't move for a clean repo.
+function hasEslint(repo) {
+  const files = ['eslint.config.js', 'eslint.config.mjs', 'eslint.config.cjs',
+    '.eslintrc', '.eslintrc.json', '.eslintrc.js', '.eslintrc.cjs', '.eslintrc.yml', '.eslintrc.yaml'];
+  if (files.some((f) => existsSync(join(repo, f)))) return true;
+  try {
+    const pkg = JSON.parse(readFileSync(join(repo, 'package.json'), 'utf8'));
+    return !!(pkg.devDependencies?.eslint || pkg.dependencies?.eslint);
+  } catch { return false; }
+}
+
 function init(argv) {
   if (argv.includes('--list')) {
     console.log('PromptWheel presets (promptwheel init --preset <name>):\n');
@@ -472,13 +518,14 @@ function init(argv) {
     metrics = metrics.map((m) => m.cmd === '__TESTCMD__' ? { ...m, cmd: detectTestCmd(repo) } : m); // fill the preset's target
     note = presetName;
   } else {
-    // sensible default: tests-pass (guarded) + lint (info — can't fail a newcomer's first run)
+    // default = the headline posture: guarded tests + the antihack tripwires, so the FIRST
+    // cheat a newcomer tries (gut the tests while the metric stays flat) fails the gate —
+    // "catch your agent cheating" must be true out of the box, not only under a preset.
     const testCmd = detectTestCmd(repo);
-    metrics = [
-      { name: 'tests_pass', cmd: testCmd, extract: 'exit', direction: 'pass', guard: true },
-      { name: 'lint_errors', cmd: LINT_CMD, extract: 'number', direction: 'down', guard: false },
-    ];
-    note = `tests-pass + lint (detected: ${testCmd})`;
+    const lint = hasEslint(repo);
+    metrics = PRESETS['antihack'].metrics.map((m) => m.cmd === '__TESTCMD__' ? { ...m, cmd: testCmd } : m);
+    if (lint) metrics = [...metrics, { name: 'lint_errors', cmd: LINT_CMD, extract: 'number', direction: 'down', guard: false }];
+    note = `tests + antihack tripwires${lint ? ' + lint' : ''} (detected: ${testCmd})`;
   }
   writeFileSync(out, JSON.stringify({ repeat: 1, metrics }, null, 2) + '\n');
   console.log(`✓ wrote promptwheel.config.json — ${note}`);
@@ -523,9 +570,12 @@ function printHuman(r) {
     const tag = m.guard ? (m.ok ? 'guard✓' : 'GUARD✗') : 'info';
     const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
     console.log(`  ${arrowFor(m)} ${m.name.padEnd(18)} ${String(m.before).padStart(8)} → ${String(m.after).padStart(8)}  (${d}, ${m.status}) [${tag}, ${m.confidence}]`);
+    if (m.inert) console.log('      ⚠ never passed at either ref — this guard is protecting nothing (check its command)');
     if (m.gamed === true) console.log(`      🚩 GAMED — ${m.gamingReason}`);
   }
-  console.log(`\n  VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? '  — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? '  — a metric "improved" by editing the goalposts, not the source' : ''}\n`);
+  console.log(`\n  VERDICT: ${r.verdict.toUpperCase()}${r.verdict === 'fail' ? '  — a guarded metric regressed (beyond noise)' : r.verdict === 'gamed' ? '  — a metric "improved" by editing the goalposts, not the source' : ''}`);
+  if (r.verdict === 'fail') console.log('  intentional? loosen that guard locally in promptwheel.config.json (guard:false, or override the inherited metric by name) — `promptwheel guards` shows the effective set');
+  console.log('');
 }
 
 // PR-comment markdown (rendering lives in the tool so the GitHub Action stays thin)
@@ -534,7 +584,7 @@ function renderMarkdown(r) {
   const sIcon = { improved: '🟢', regressed: '🔴', unchanged: '⚪', inconclusive: '🟡', unmeasurable: '⚫' };
   const rows = r.metrics.map((m) => {
     const d = m.delta == null ? '—' : (m.delta > 0 ? `+${m.delta}` : `${m.delta}`);
-    const status = m.gamed === true ? `🚩 gamed (${(m.retained * 100).toFixed(0)}% survives source-only)` : `${sIcon[m.status] || ''} ${m.status}`;
+    const status = m.gamed === true ? `🚩 gamed (${(m.retained * 100).toFixed(0)}% survives source-only)` : `${sIcon[m.status] || ''} ${m.status}${m.inert ? ' ⚠ never passes' : ''}`;
     return `| ${m.guard ? '🛡️ ' : ''}${m.name} | ${m.before} | ${m.after} | ${d} | ${status} | ${m.confidence} |`;
   }).join('\n');
   return [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "promptwheel",
-  "version": "0.1.2",
+  "version": "0.2.3",
   "description": "Catch your agent cheating — a deterministic, zero-LLM CLI that flags when an AI coding agent gamed its own metric (re-proves the win from the agent's source edits alone). Built on an outcome gate; runs in CI or a Claude Code hook.",
   "type": "module",
   "bin": {