promptwheel 0.7.12 → 0.7.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/bin/promptwheel.d.ts +3 -0
- package/bin/promptwheel.mjs +84 -6
- package/package.json +2 -2
- package/packs/supply-chain/README.md +7 -6
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
[](https://github.com/promptwheel-ai/promptwheel/actions/workflows/ci.yml) [](LICENSE)  
|
|
4
4
|
|
|
5
|
-
**A deterministic, zero-LLM check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel re-proves every win from the agent's source edits alone, so a
|
|
5
|
+
**A deterministic, zero-LLM check for your coding loop.** A green test should mean your agent *fixed the code*, not that it edited the test to match the bug. PromptWheel re-proves every win from the agent's source edits alone, so a test that went from red to green because the test, grader or declared config changed reads `GAMED`. The code being checked never gets to grade itself.
|
|
6
6
|
|
|
7
7
|

|
|
8
8
|
|
|
@@ -177,14 +177,14 @@ Then `/promptwheel:setup` (write a config) · `/promptwheel:gate` (gate uncommit
|
|
|
177
177
|
|
|
178
178
|
## Trust model
|
|
179
179
|
|
|
180
|
-
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
|
|
180
|
+
**What it catches.** After the normal gate, every *win* is re-proven using the agent's source edits alone. If the win only survived by skipping or deleting a test, mocking the grader, editing a golden, relaxing a declared config, or touching zero source files, the verdict is `GAMED` (exit 2). On top of that:
|
|
181
181
|
|
|
182
182
|
- **Tripwire guards** (`test_count`, `skipped_tests`, `suppressions`, `assertions`, written by the plain `init` default) catch the quieter shapes: an inline suppression (`@ts-nocheck`, `eslint-disable`, `# noqa`) or a suite gutted while the metric stays flat.
|
|
183
183
|
- **Gate integrity.** The gate is resolved as of the base commit too. A change that disables a guard or deletes a guarded metric in the same diff it passes is re-measured under the base definition and reads `GAMED` if that guard would have failed. The gate definition (metrics, guards, `evaluatorPaths`, `gamingThreshold`, `mandates`) is fingerprinted at both refs; a difference is recorded as `gate.changed` and printed.
|
|
184
184
|
- **A sanctioned override.** A commit trailer can be written by whoever writes the commit, including an agent, so `"allowTrailer": false` in the contract turns trailers off and only the `--allow` flag counts (the supply-chain preset sets this; the contract at the base ref decides, so a change cannot turn trailers back on for itself). When you *do* mean to change a test, config, or grader, declare it: a `Promptwheel-Allow: <reason>` trailer on the commit (`Sawbuck-Allow:` is accepted as an alias), `--allow "<reason>"` on the CLI, or the `allow` option on `check()`, turns what would read `GAMED` into a `pass` with the reason recorded in `report.allowed`. It covers a win whose source-only re-run could not be applied (`unprovable`, otherwise `inconclusive`) the same way. An undeclared test-only win is still `gamed`, and a reasonless declaration does not override. The check never judges intent; it forces intent to be explicit and auditable.
|
|
185
185
|
- **A provenance receipt** on every verdict (`report.provenance`: base and head SHAs, a config hash, a verdict hash), so a skeptic can re-run `(base, head, config)` and get the same hash.
|
|
186
186
|
|
|
187
|
-
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test): that needs held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
|
|
187
|
+
**The honest limit.** This is one layer, not a silver bullet. It catches the evaluator-tampering class (test, grader, golden and config edits) deterministically and for free. A suite weakened while it stays green is not proved: there is no win to re-run. Tripwires catch skips and removed assertions in a signed repo. It is blind, by construction, to a cheat that lives in the source file itself (a hardcoded answer, behavior that differs under test): that needs held-out tests, an LLM judge, or a human, and those expensive layers should be reserved for the calls only they can make. The coverage matrix and the in/out-of-scope boundary are in [docs/DETECTION-LAYERS.md](docs/DETECTION-LAYERS.md); the measured numbers are in [bench/RESULTS.md](bench/RESULTS.md) (`node bench/gaming-bench.mjs` to reproduce). Another gameable thing, an LLM judge asking "did you cheat?", is deliberately not part of the check.
|
|
188
188
|
|
|
189
189
|
**Noise.** A number that jumps around between runs is worthless as a signal, and PromptWheel does not pretend otherwise: `--repeat N` measures each metric N times at both refs and uses the median; the noise band is the observed spread; a delta inside the band is `inconclusive` with `low` confidence and does not fail a guard (no flaky CI failures). Confidence reads `high` (deterministic extract, or zero observed noise), `medium` (delta clears the noise band), `low` (delta inside noise), or `unverified` (single read; run `--repeat` to earn trust). A guard whose command never ran (same value at both refs) makes the verdict `inconclusive`, never a fake pass.
|
|
190
190
|
|
package/bin/promptwheel.d.ts
CHANGED
|
@@ -336,3 +336,6 @@ export declare function judgeGaming(
|
|
|
336
336
|
|
|
337
337
|
/** Truncated SHA-256 of the verdict content (everything but `provenance`), the reproducible half of the receipt. */
|
|
338
338
|
export declare function verdictFingerprint(report: Pick<Report, 'verdict'> & Partial<Pick<Report, 'metrics' | 'gate'>>): string;
|
|
339
|
+
|
|
340
|
+
/** The newest `sawbuck-allow: <reason>` comment from someone with write access, or ''. */
|
|
341
|
+
export declare function allowFromComments(comments: unknown[]): string;
|
package/bin/promptwheel.mjs
CHANGED
|
@@ -330,7 +330,7 @@ function checkMandates(repo, base, head, cfg, baseMandates) {
|
|
|
330
330
|
const mandates = [...headList.filter((m) => m && baseNames.has(m.name)), ...kept];
|
|
331
331
|
if (!mandates.length) return [];
|
|
332
332
|
let changed = [];
|
|
333
|
-
try { changed = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean); } catch { return []; }
|
|
333
|
+
try { changed = git(['diff', '--name-only', '--no-renames', base, head], repo).split('\n').filter(Boolean); } catch { return []; } // --no-renames: a move out of a protected path also touches it
|
|
334
334
|
const out = [];
|
|
335
335
|
for (const m of mandates) {
|
|
336
336
|
const pats = (Array.isArray(m.paths) ? m.paths : []).filter((s) => typeof s === 'string' && s.trim());
|
|
@@ -355,7 +355,7 @@ function checkMandates(repo, base, head, cfg, baseMandates) {
|
|
|
355
355
|
// `isEval` is the repo's opt-in evaluator-path matcher, treated as non-source alongside
|
|
356
356
|
// the built-in classifier, so a cheat that only edits a declared gate file is caught.
|
|
357
357
|
function changedSourcePaths(repo, base, head, isEval = () => false) {
|
|
358
|
-
const all = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean);
|
|
358
|
+
const all = git(['diff', '--name-only', '--no-renames', base, head], repo).split('\n').filter(Boolean);
|
|
359
359
|
const nonSrc = (p) => isNonSource(p) || isEval(p);
|
|
360
360
|
return { source: all.filter((p) => !nonSrc(p)), nonSource: all.filter(nonSrc) };
|
|
361
361
|
}
|
|
@@ -363,6 +363,37 @@ function changedSourcePaths(repo, base, head, isEval = () => false) {
|
|
|
363
363
|
// measure the given metric defs in ONE worktree at `base` with ONLY the source slice of base→head
|
|
364
364
|
// applied (checked out once, patched once, set up once), so re-proving three improved metrics costs
|
|
365
365
|
// one checkout, not three. Returns { [name]: { samples, hadSourceChange, applyFailed? } }.
|
|
366
|
+
// package.json and pyproject.toml are source and goalposts in one file: dependencies belong to the
|
|
367
|
+
// change, the test command and the test runner's options do not. The source-only tree keeps the change's
|
|
368
|
+
// file but puts those keys back as they were at base, so `"test": "true"` or `addopts = "-k mul"` cannot
|
|
369
|
+
// carry a win. A test command that really had to change is a declared change: --allow.
|
|
370
|
+
const JS_TEST_KEYS = ['scripts', 'jest', 'ava', 'mocha', 'nyc', 'c8', 'vitest', 'tap'];
|
|
371
|
+
const PY_TEST_TABLE = /^\[\s*tool\.(pytest|coverage|tox)[\].]/;
|
|
372
|
+
function tomlTables(text) { // [header, lines] pairs in file order; the first has no header
|
|
373
|
+
const out = [[null, []]];
|
|
374
|
+
for (const line of text.split('\n')) { if (/^\s*\[/.test(line)) out.push([line.trim(), [line]]); else out[out.length - 1][1].push(line); }
|
|
375
|
+
return out;
|
|
376
|
+
}
|
|
377
|
+
function holdGoalposts(repo, base, wt, source) {
|
|
378
|
+
for (const p of source) {
|
|
379
|
+
const name = p.split('/').pop();
|
|
380
|
+
if (name !== 'package.json' && name !== 'pyproject.toml') continue;
|
|
381
|
+
let before; try { before = git(['show', `${base}:${p}`], repo); } catch { continue; } // a new file has no base to hold
|
|
382
|
+
const f = join(wt, p);
|
|
383
|
+
try {
|
|
384
|
+
if (name === 'package.json') {
|
|
385
|
+
const a = JSON.parse(before), b = JSON.parse(readFileSync(f, 'utf8'));
|
|
386
|
+
for (const k of JS_TEST_KEYS) { if (k in a) b[k] = a[k]; else delete b[k]; }
|
|
387
|
+
writeFileSync(f, JSON.stringify(b, null, 2) + '\n');
|
|
388
|
+
} else {
|
|
389
|
+
const held = tomlTables(before).filter(([h]) => h && PY_TEST_TABLE.test(h));
|
|
390
|
+
const kept = tomlTables(readFileSync(f, 'utf8')).filter(([h]) => !h || !PY_TEST_TABLE.test(h));
|
|
391
|
+
writeFileSync(f, [...kept, ...held].map(([, l]) => l.join('\n')).join('\n') + '\n');
|
|
392
|
+
}
|
|
393
|
+
} catch { /* an unreadable manifest stays as the change left it */ }
|
|
394
|
+
}
|
|
395
|
+
}
|
|
396
|
+
|
|
366
397
|
function measureSourceOnlyAll(repo, base, head, defs, cfg, repeat) {
|
|
367
398
|
const out = {};
|
|
368
399
|
if (!defs.length) return out;
|
|
@@ -383,6 +414,7 @@ function measureSourceOnlyAll(repo, base, head, defs, cfg, repeat) {
|
|
|
383
414
|
try { tryApply(['--3way']); }
|
|
384
415
|
catch { try { tryApply([]); } catch { for (const m of defs) out[m.name] = { samples: [], hadSourceChange: true, applyFailed: true }; return out; } }
|
|
385
416
|
}
|
|
417
|
+
holdGoalposts(repo, base, wt, source); // the parts of a mixed file that define the tests stay as they were at base
|
|
386
418
|
runSetup(wt, cfg, env); // build the patched tree, so a build-gated metric measures the source edit
|
|
387
419
|
for (const m of defs) {
|
|
388
420
|
const samples = [];
|
|
@@ -576,7 +608,7 @@ function blobHashAt(repo, ref, path) {
|
|
|
576
608
|
function changedEvaluatorPaths(repo, base, head, cfg) {
|
|
577
609
|
const isEval = evaluatorMatcher(cfg);
|
|
578
610
|
let changed = [];
|
|
579
|
-
try { changed = git(['diff', '--name-only', base, head], repo).split('\n').filter(Boolean).filter(isEval); }
|
|
611
|
+
try { changed = git(['diff', '--name-only', '--no-renames', base, head], repo).split('\n').filter(Boolean).filter(isEval); }
|
|
580
612
|
catch { return []; }
|
|
581
613
|
return changed.map((p) => ({ path: p, base: blobHashAt(repo, base, p), head: blobHashAt(repo, head, p) }))
|
|
582
614
|
.filter((e) => e.base !== e.head);
|
|
@@ -648,6 +680,16 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
|
648
680
|
m.gamed = j.gamed; m.sourceOnly = j.sourceOnly; m.retained = j.retained; m.gamingReason = j.reason;
|
|
649
681
|
if (j.unprovable) m.unprovable = true;
|
|
650
682
|
}
|
|
683
|
+
// A green that stays green proves nothing about the tests. When a change edits the source and the tests
|
|
684
|
+
// together, run the old tests on the new source: if they fail, say so. Only a note, never a verdict: an
|
|
685
|
+
// intended API change does this too, and no deterministic check can tell the two apart.
|
|
686
|
+
const split = changedSourcePaths(repo, base, head, evaluatorMatcher(cfg));
|
|
687
|
+
if (split.source.length && split.nonSource.length) {
|
|
688
|
+
const held = metrics.filter((m) => m.status === 'unchanged' && m.direction === 'pass' && m.after === 1 && !m.tripwire);
|
|
689
|
+
const defs = held.map((m) => cfg.metrics.find((c) => c.name === m.name)).filter(Boolean);
|
|
690
|
+
const res = measureSourceOnlyAll(repo, base, head, defs, cfg, repeat);
|
|
691
|
+
for (const m of held) { const r = res[m.name]; if (r && r.samples && r.samples.length && median(r.samples) !== 1) m.heldByTestEdit = true; }
|
|
692
|
+
}
|
|
651
693
|
const evalChanged = changedEvaluatorPaths(repo, base, head, cfg);
|
|
652
694
|
if (evalChanged.length) {
|
|
653
695
|
for (const m of metrics) {
|
|
@@ -695,7 +737,7 @@ function detectGamingPass(repo, base, head, cfg, metrics, repeat) {
|
|
|
695
737
|
export function deriveVerdict(metrics, gateInfo = null, mandated = false) {
|
|
696
738
|
const failed = metrics.some((m) => m.guard && !m.ok);
|
|
697
739
|
const gamed = metrics.some((m) => m.gamed === true) || !!(gateInfo && gateInfo.weakened && !gateInfo.allowed);
|
|
698
|
-
const inertNoWin = metrics.some((m) => m.inert) && !metrics.some((m) => m.status === 'improved');
|
|
740
|
+
const inertNoWin = metrics.some((m) => m.inert) && !metrics.some((m) => m.status === 'improved' && !m.tripwire); // a tripwire moving is not a measured win
|
|
699
741
|
const unprovable = metrics.some((m) => m.unprovable === true);
|
|
700
742
|
const inconclusive = !failed && !gamed && (inertNoWin || unprovable);
|
|
701
743
|
return (mandated || failed) ? 'fail' : gamed ? 'gamed' : inconclusive ? 'inconclusive' : 'pass';
|
|
@@ -732,7 +774,9 @@ function gate(repo, opts) {
|
|
|
732
774
|
const metrics = cfg.metrics.map((m) => {
|
|
733
775
|
const ev = evaluate(m, before[m.name], after[m.name], repeat);
|
|
734
776
|
if (ev.status === 'unmeasurable' && ev.before == null && ev.after != null && adopted(m.name)) { ev.ok = true; ev.adopted = true; }
|
|
735
|
-
|
|
777
|
+
// a `report` guard tells you and does not fail: a contract can start by watching and tighten later
|
|
778
|
+
if (m.report && !ev.ok && ev.status === 'regressed') { ev.ok = true; ev.noted = true; }
|
|
779
|
+
return { name: m.name, direction: m.direction || 'up', guard: !!m.guard, ...(m.gamingCheck === false ? { tripwire: true } : {}), ...ev };
|
|
736
780
|
});
|
|
737
781
|
// Section 3 (gaming detection), on by default: mutates `metrics` and yields the gate record
|
|
738
782
|
const gateInfo = opts.detectGaming ? detectGamingPass(repo, base, head, cfg, metrics, repeat) : null;
|
|
@@ -793,7 +837,9 @@ function gate(repo, opts) {
|
|
|
793
837
|
}
|
|
794
838
|
const mandated = mandateViolations.length > 0;
|
|
795
839
|
const verdict = deriveVerdict(metrics, gateInfo, mandated);
|
|
840
|
+
const explained = explainMetrics(repo, metrics);
|
|
796
841
|
const report = {
|
|
842
|
+
...(explained ? { explained } : {}),
|
|
797
843
|
base: short(repo, base), head: short(repo, head), repeat, mode: opts.working ? 'working' : 'refs',
|
|
798
844
|
verdict, metrics,
|
|
799
845
|
...(mandateViolations.length ? { mandates: mandateViolations } : {}),
|
|
@@ -945,10 +991,12 @@ const PRESETS = {
|
|
|
945
991
|
// edit (a lifecycle script, a new dependency, a non-registry lock source, a workflow or publish
|
|
946
992
|
// config). Paths and counts, checked deterministically: three guarded counts that must not rise
|
|
947
993
|
// without a declared reason (`allowable`), plus mandates that freeze the paths. The contract itself is
|
|
948
|
-
//
|
|
994
|
+
// one of those paths, lifted only by a person's --allow: otherwise a change that only weakens a rule
|
|
995
|
+
// passes, and the next change is judged by the weakened contract.
|
|
949
996
|
'supply-chain': { allowTrailer: false, desc: 'gate: no new lifecycle scripts, dependencies, or non-registry lock sources without a recorded reason; the agent cannot touch workflows, registries, publish scripts, or the contract',
|
|
950
997
|
evaluatorPaths: ['promptwheel.config.json', 'sawbuck.config.json'],
|
|
951
998
|
mandates: [
|
|
999
|
+
{ name: 'contract-frozen', paths: ['promptwheel.config.json', 'sawbuck.config.json'], allowable: 'flag', reason: 'the contract changes with a reason from a person (--allow), never as a side effect of a change' },
|
|
952
1000
|
{ name: 'workflows-frozen', paths: ['.github/workflows/**'], reason: 'workflows run with the repository secrets; change them in a separate, reviewed change' },
|
|
953
1001
|
{ name: 'registry-config-frozen', paths: ['.npmrc', '.yarnrc', '.yarnrc.yml'], reason: 'registry and token configuration decides where packages come from and where credentials go' },
|
|
954
1002
|
{ name: 'build-images-frozen', paths: ['Dockerfile*', 'docker-compose*'], reason: 'what runs at build and deploy time is not an agent decision' },
|
|
@@ -1182,6 +1230,34 @@ function printHuman(r) {
|
|
|
1182
1230
|
}
|
|
1183
1231
|
|
|
1184
1232
|
// PR-comment markdown (rendering lives in the tool so the GitHub Action stays thin)
|
|
1233
|
+
// What a person needs when a check speaks: what changed, in words, and how to accept it.
|
|
1234
|
+
const PLAIN = {
|
|
1235
|
+
lifecycle_scripts: 'a new install script', dependency_entries: 'a new dependency', lock_nonregistry_sources: 'a dependency from outside the registry',
|
|
1236
|
+
unpinned_execution: 'something that runs without a pinned version', install_policy_gaps: 'a weaker install policy', lock_drift: 'the shape of the repo changed',
|
|
1237
|
+
pins_changed: 'resolved dependency versions changed', tests_pass: 'tests stopped passing', test_count: 'fewer tests', skipped_tests: 'more skipped tests',
|
|
1238
|
+
suppressions: 'more silenced checks', assertions: 'fewer assertions',
|
|
1239
|
+
};
|
|
1240
|
+
function explainMetrics(repo, metrics) {
|
|
1241
|
+
const hit = metrics.filter((m) => m.noted || (m.guard && !m.ok));
|
|
1242
|
+
if (!hit.length) return null;
|
|
1243
|
+
let items = [];
|
|
1244
|
+
if (hit.some((m) => m.name === 'lock_drift')) {
|
|
1245
|
+
try { items = JSON.parse(execFileSync(process.execPath, ['.promptwheel/packs/supply-chain/lock.mjs', 'drift', '.', '--json'], { cwd: repo, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] })).slice(0, 8); } catch { /* words without the list */ }
|
|
1246
|
+
}
|
|
1247
|
+
return hit.map((m) => ({ metric: m.name, noted: !!m.noted, what: PLAIN[m.name] || m.name, ...(m.name === 'lock_drift' && items.length ? { items } : {}), ...(m.allowable ? { allowable: true } : {}) }));
|
|
1248
|
+
}
|
|
1249
|
+
// A reason a person left on a pull request. The newest `sawbuck-allow: <reason>` comment from someone with write
|
|
1250
|
+
// access wins. A bot never counts. An agent signed in as that person can still post one: the record shows it.
|
|
1251
|
+
export function allowFromComments(comments) {
|
|
1252
|
+
const trusted = new Set(['OWNER', 'MEMBER', 'COLLABORATOR']);
|
|
1253
|
+
for (const c of [...(comments || [])].reverse()) {
|
|
1254
|
+
if (!c || !c.user || c.user.type !== 'User' || !trusted.has(c.author_association)) continue;
|
|
1255
|
+
const m = String(c.body || '').match(/^\s*sawbuck-allow:[ \t]*(\S.*?)[ \t]*$/m);
|
|
1256
|
+
if (m) return `${m[1].slice(0, 280)} (approved by ${String(c.user.login).slice(0, 39)})`;
|
|
1257
|
+
}
|
|
1258
|
+
return '';
|
|
1259
|
+
}
|
|
1260
|
+
|
|
1185
1261
|
function renderMarkdown(r) {
|
|
1186
1262
|
const icon = r.verdict === 'pass' ? '✅' : r.verdict === 'gamed' ? '🚩' : r.verdict === 'inconclusive' ? '🟡' : '❌';
|
|
1187
1263
|
const sIcon = { improved: '🟢', regressed: '🔴', unchanged: '⚪', inconclusive: '🟡', unmeasurable: '⚫' };
|
|
@@ -1196,6 +1272,7 @@ function renderMarkdown(r) {
|
|
|
1196
1272
|
`\`${r.base} → ${r.head}\` · ${r.mode} mode${r.repeat > 1 ? ` · ×${r.repeat}` : ''}`,
|
|
1197
1273
|
'',
|
|
1198
1274
|
...(r.mandates ? [r.mandates.map((md) => `> ⛔ **Mandate violated:** \`${md.name}\`${md.reason ? ': ' + md.reason : ''} (touched \`${md.files.join('`, `')}\`)${md.removed ? ' (deleted in this same change; it stays in force for the change that removes it)' : md.edited ? ' (edited in this same change; the version before the change stays in force for the change that edits it)' : ''}`).join('\n'), ''] : []),
|
|
1275
|
+
...(r.explained ? [...r.explained.map((x) => `> ${x.noted ? 'ℹ️ **Note:**' : '✋ **Needs a look:**'} ${x.what}${x.items ? ': `' + x.items.join('`, `') + '`' : ''}`), ...(r.explained.some((x) => !x.noted && x.allowable) ? ['>', '> To accept this, comment `sawbuck-allow: <your reason>` on this PR and re-run the check.'] : []), ''] : []),
|
|
1199
1276
|
'| metric | before | after | Δ | status | confidence |',
|
|
1200
1277
|
'|---|--:|--:|--:|---|---|',
|
|
1201
1278
|
rows,
|
|
@@ -1470,6 +1547,7 @@ function dispatch() {
|
|
|
1470
1547
|
if (cmd in COMMAND_FLAGS) checkFlags(cmd, rest);
|
|
1471
1548
|
if (cmd === 'run') run(rest);
|
|
1472
1549
|
else if (cmd === 'improve') improve(rest);
|
|
1550
|
+
else if (cmd === 'pr-allow') { let c = []; try { c = JSON.parse(readFileSync(0, 'utf8') || '[]'); } catch { /* no comments, no approval */ } process.stdout.write(allowFromComments(c)); }
|
|
1473
1551
|
else if (cmd === 'insights') insights(rest);
|
|
1474
1552
|
else if (cmd === 'init') init(rest);
|
|
1475
1553
|
else if (cmd === 'guards') guards(rest);
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "promptwheel",
|
|
3
|
-
"version": "0.7.
|
|
4
|
-
"description": "A deterministic, zero-LLM check for your coding loop.
|
|
3
|
+
"version": "0.7.14",
|
|
4
|
+
"description": "A deterministic, zero-LLM check for your coding loop. A win must survive a re-run from the agent's source edits alone, so a test that went green by editing the test reads GAMED. Runs in CI or a Claude Code plugin.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"promptwheel": "bin/promptwheel.mjs"
|
|
@@ -28,7 +28,9 @@ clean git worktrees, so a pack that is not in git is absent at both refs and the
|
|
|
28
28
|
- **unpinned_execution**: code run from a reference that can move. A GitHub Action not pinned to a full commit SHA (a local `./action` does not count), `npx`, `bunx` or `dlx` with no exact version, `npm install -g` with no exact version, and a remote script piped into a shell. Read from workflows, `action.yml`, package.json scripts, `scripts/*.sh` and Dockerfiles. In package.json scripts, a tool the manifest declares does not count: the lockfile already pins it.
|
|
29
29
|
- **install_policy_gaps**: 0, 1 or 2. Are install scripts disabled, and is a release-age cooldown set (`ignore-scripts=true` and `min-release-age` in `.npmrc`, or the pnpm or yarn equivalents)? The cooldown is enforced by the package manager, not by this pack: a registry lookup here would make the verdict depend on the clock.
|
|
30
30
|
- **lock_drift**: structural items the tree has that `promptwheel.lock.json` does not. `promptwheel lock` reads the repo once and writes its shape: dependencies, install scripts, workflows, modules, import edges between modules, routes, public entry points, and the network hosts the source names. It also records each script's body, native binaries, and what a module can do (network, process spawning, eval), so a new capability shows even when the host is built at runtime. Ordinary edits never touch it. A new route is reported but not held by default; add `--routes` to hold it. On 1,403 commits across eight repositories this guard would have stopped 5.1%, about one in 20. JS and TS only for edges, routes, hosts and capabilities today.
|
|
31
|
-
- **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%,
|
|
31
|
+
- **pins_changed**: a fingerprint of the resolved dependency tree, compared base to head, because a new version under an old name is how a worm arrives. It covers every package, version, source and integrity, not the file's bytes, so a release of your own project does not count. Any change needs a reason. Nothing is stored, so the lock does not change across updates and an approved update does not blind the next check. On the same commits this stopped 8.7%, dependency changes; use the CI path below for those.
|
|
32
|
+
|
|
33
|
+
**On a pull request.** Someone with write access comments `sawbuck-allow: <reason>` and re-runs the check. `sawbuck contract` marks everyday guards `report: true`, so they note a change without failing until you tighten them; this preset on its own holds them all.
|
|
32
34
|
|
|
33
35
|
**Dependency updates in CI.** Give the reason from a step the agent cannot reach. Workflows are a frozen path, so an agent cannot edit this:
|
|
34
36
|
|
|
@@ -77,13 +79,12 @@ and a mandate deleted or narrowed in the same change that violates it stays in f
|
|
|
77
79
|
| build-images-frozen | `Dockerfile*`, `docker-compose*` | what runs at build and deploy time |
|
|
78
80
|
| publish-scripts-frozen | `scripts/release*`, `scripts/publish*` | the publish path is how a compromise spreads |
|
|
79
81
|
| codeowners-frozen | `CODEOWNERS` | review ownership is set by people |
|
|
82
|
+
| contract-frozen | `promptwheel.config.json`, `sawbuck.config.json` | the contract changes with a reason from a person (`--allow`) |
|
|
80
83
|
| hooks-frozen | `.claude/settings.json`, `.claude/settings.local.json`, `.sawbuck/**` | the hooks that enforce the contract are wired by a person |
|
|
81
84
|
|
|
82
|
-
The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is
|
|
83
|
-
`evaluatorPaths` entry
|
|
84
|
-
|
|
85
|
-
contract (a person retiring a rule) still passes. A mandate there would make the contract
|
|
86
|
-
un-editable through the gate.
|
|
85
|
+
The contract itself (`promptwheel.config.json`, `sawbuck.config.json`) is frozen too, and it is also an
|
|
86
|
+
`evaluatorPaths` entry. A change to it needs `--allow "<reason>"` from a person. Without that, a change that
|
|
87
|
+
only weakens a rule would pass, and the next change would be judged by the weakened contract.
|
|
87
88
|
|
|
88
89
|
A person who needs to change a frozen path does it in a separate change that only touches that
|
|
89
90
|
path, reviewed as such.
|