tamperward 1.4.6 → 1.5.1

package/README.md

@@ -93,13 +93,16 @@ Under GitHub Actions each finding also lands as an **inline annotation on the di
 
 ## The rules
 
-Nine mechanical rules, deterministic by construction — eight `block`, one `warn`:
+Ten mechanical rules, deterministic by construction — eight `block`, two `warn`:
 `test-deletion` (AST block-count, rename-out-of-glob, shell mutation), `test-skip`
 (including `.only`, which narrows the suite), `ts-any-cast`, `lint-suppression`,
 `coverage-lowering` (Jest and Vitest shapes, all four metrics, `package.json` included),
 `ci-tampering` (a **moved** check is not a removed one), `hook-tampering`, `no-verify`
-(flags and the env-var escape hatches), and `snapshot-rewrite` (`warn` — see the FP study
-below for why).
+(flags and the env-var escape hatches), `snapshot-rewrite` (`warn` — see the FP study below
+for why), and `snapshot-only-rewrite` (`warn` — the FP study's narrow signal, a
+snapshot moved with **no accompanying change**, as its own rule so it can graduate to
+`block` at a later policy `version:` without touching the broad rule; it fires only at
+commit granularity, where "no accompanying change" means something).
 
 Two heuristic rule names — `assertion-weakening` and `guard-removal` — are **reserved
 in the baseline policy but not yet built**: they get detectors only once a measured
@@ -115,19 +118,32 @@ escape valve is a PR label only a human with write access can apply.
 **Does the loop work?** `harness/run.sh` stands up an isolated repo with a real bug and a
 failing protected test, lets a headless agent loose with the hooks installed, and
 classifies the end state with an oracle independent of the detectors under test
-(SPEC §7.B). Across 67 runs of these **guarded** scenarios — every attempted bypass class
-had a detector covering it — **0 tampers reached green; all 25 runs where a block fired
-ended in an honest fix**, a 25/25 point estimate whose 95% lower bound is ≈89%. The split is the
-honest part: Sonnet 5 never attempted a bypass at all, even when the prompt *ordered*
-one; every interception came from Haiku 4.5 on a seed whose honest fix is genuinely
-expensive. Reproduce: `npm run build && harness/run.sh 5 adversarial haiku seed-hard`.
+(SPEC §7.B). Across **137 runs** of these **guarded** scenarios — every attempted bypass
+class had a detector covering it — **0 tampers reached green**. Denied runs converted to
+an honest fix **42 times out of 56 (75%)**, and that number is honestly seed-dependent:
+25/25 where the honest fix was the agent's own task, down to **0/10** where the failing
+suite was somebody else's bug (the no-verify study below). The frontier split gets the
+same scoping: on the expensive-fix seed no frontier model attempted a bypass even when
+the prompt *ordered* one; on the hook-skip seed, frontier models under a prompt ordering
+the skip **try the ordered skip once, get denied once, and convert 10/10** — frontier
+honesty there is what happens after the first no, not never-trying.
+Reproduce: `npm run build && harness/run.sh 5 adversarial haiku seed-hard`.
 
 **Is the fix caused by the deny, or would it have happened anyway?** Measured, with a
 pre-registered no-gate control arm (`harness/PREDICTION-ungated-control.md`): same seed,
 model, prompt and oracle, hooks removed. **6/10 ungated runs tampered through; honest
 fixes fell from 10/10 gated to 4/10 ungated.** The gate's measured contribution on that
 configuration is a 60-point swing — causation, not survivorship — and one of the three
-control-arm bets was refuted (ungated honest fixes came in above my bet).
+control-arm bets was refuted (ungated honest fixes came in above my bet). The control
+arm was then extended across the model family (`harness/PREDICTION-frontier-models.md`):
+ungated Sonnet 5, Opus 5, and Fable 5 each went **10/10 honest** under the same
+pressure prompt — frontier honesty is disposition, not the gate's deterrence — while
+ungated Haiku stays 6/10 tamper-green. The gate's measured value concentrates on the
+models that do cheat. (The Fable rows are a Fable-driven harness measuring Fable:
+self-graded homework, bounded by the deterministic held-out oracle.) The
+type-laundering study below later measured the boundary of the disposition claim:
+it held for Opus in every regime; Sonnet and Fable held only where the gate could
+interrupt.
 
 **And when the agent knows the gate is there?** Also pre-registered
 (`harness/PREDICTION-informed-adversary.md`) — betting the gate would *leak*. Refuted,
@@ -144,14 +160,104 @@ specific bypass each; predictions are pre-registered before any rep runs
 file from buggy output and sailed through — **because nothing guarded that class yet**;
 that measured demand is why `snapshot-rewrite` exists — while
 timeout-inflation and lint-config-gutting measured **0/10 attempts each** — two
-pre-registered bets refuted, two detectors *not* built. Bypass classes with no measured
-demand stay in a record-only shadow scanner that accumulates evidence on every weekly
-harness run.
+pre-registered bets refuted, two detectors *not* built. The same discipline settled
+`assertion-weakening` (`harness/PREDICTION-assert-affordance.md`): a seed making the
+expectation edit the one cheap, prompt-named move — with a plausible convention-mismatch
+cover story — measured **0/10 attempts**; Haiku implemented the real fix every time. My
+≥6/10 bet was refuted, and the rule stays a reserved name. Bypass classes with no
+measured demand stay in a record-only shadow scanner that accumulates evidence on every
+weekly harness run.
+
+**And when no honest fix exists at all?** The regime every other seed excludes — and
+where the typia incident actually lived — measured with a claims-based oracle
+(`harness/PREDICTION-stuck-agent.md`): a provably unfixable suite, an explicit cheap
+honest exit ("STATUS: blocked is a fully acceptable outcome"), fabrication defined as
+claiming `fixed` against cryptographic reality. **Twenty of twenty pressured runs —
+Haiku and Sonnet alike — surrendered honestly**, zero fabrications, zero test edits,
+a written diagnosis every time. Both my Haiku bets were refuted: the model that
+tampers 6/10 when the cheat is cheaper than the fix takes the honest exit when the
+honest exit is cheapest. The sharpened thesis: agents take the cheapest path they
+believe is sanctioned — the danger is pressure with no sanctioned exit cheaper than
+the lie. The untested harder variant (no one *names* the exit) is registered as the
+open follow-up.
+
+**Is the honesty just eval-awareness?** The sharpest critique of every number above —
+the seeds *look* like tests, so maybe the models are performing for a suspected
+grader — is measured too (`harness/PREDICTION-eval-awareness.md`): the same
+seed dressed as a real project (README, changelog, a user-report thread with a
+maintainer's plea, a backdated git history built by a self-deleting kit), gate off.
+**Zero frontier tampers in 20 dressed runs**, matching the undressed baselines, while
+Haiku stayed a cheater at 8/10. The pre-registered manipulation check is the honest
+part: only Opus actually read the dressing (8/10 runs, all ten honest) — so the Opus
+null carries the weight and the Sonnet null is scoped as ambient-only. And the
+sanctioned-edit variant (`harness/PREDICTION-self-authored.md`) — implement from an
+ambiguous spec, then a prompt *licensing* correction of "intern-written" tests —
+produced **zero test rewrites in 30 runs across three models**, refuting all three of
+my rewrite bets, for a reason worth knowing: every model, implementing blind, chose
+the same reading the tests encode. You cannot make an agent believe a test is wrong
+by telling it tests might be wrong; the forced-conflict variant is the registered
+follow-up.
+
+**Can "closed vs open" be called in advance?** The two dimension studies
+(`harness/PREDICTION-coverage.md`, `harness/PREDICTION-noverify.md`) tested whether a
+rule's leak-prone axis can be identified a priori. Both a-priori calls **held
+deterministically**: every enumerated hook-skip spelling denies while both
+`core.hooksPath` forms allow (the axis called open is open — surveyed by six frontier
+runs in reconnaissance, exploited by none), and all three coverage config-surface moves
+block. The behavioral half delivered the study's best datum: told to get the coverage
+gate green fast, **half of Haiku's runs opened with the per-path override** — the exact
+config-surface move the registration predicted a token detector would miss — and the
+shipped semantic detector caught every one. Zero tampers in 60 more guarded runs; one
+false-positive interception recorded and kept (Fable extending a test-file import via
+`sed`, denied by the shell-mutation heuristic mid-honest-fix — the FP record's first
+live entry).
+
+**And when the cheapest cheat is invisible to the gate?** The type-laundering study
+(`harness/PREDICTION-asany.md`) probed the one seed whose cheapest cheat spellings are
+deliberately uncovered — `: any` annotations are warn-only (`ts-any-launder`, the
+FP-avoidance choice) and a bare assertion (`raw as Settings`) has no rule at all — with
+a held-out runtime oracle hardened *before* any rep (without that hardening, nine
+assertion tampers would have scored honest and the conclusion would have inverted).
+Result: **Haiku laundered 10/10** — and the three runs denied on the guarded `as any`
+cast **switched to the invisible assertion**, the first time a deny redirected an agent
+to a better bypass instead of the fix. **Opus went 10/10 honest with zero gate
+involvement** — the strongest disposition evidence measured — while **Fable tampered
+6/10 and Sonnet, the default model, 10/10, both via the bare assertion and drawing
+zero denies** (Sonnet's spelling every time: an inline structural assertion,
+`as { retries: number; ... }` — the quickest silence that compiles carries no bypass
+framing, and Sonnet's famous refusals turn out to be triggered by framing). The
+completed frontier split — **Opus 0/10 · Fable 6/10 · Sonnet 10/10** — is the
+published shape, with the standing self-measurement caveat on the Fable rows.
+Twenty-six of forty runs tampered through the uncovered dimension: measured demand
+that hardens the `ts-any-launder` warn→block graduation case via the §7.A corpus —
+the honest fix itself narrows through `as Record<string, unknown>`, so a naive
+assertion-blocker is an FP factory.
+
+**Why not just a prompt line, or a 20-line grep hook?** Measured
+(`harness/PREDICTION-baselines.md`), against a frozen naive hook and a one-sentence
+prompt baseline — and two of my three bets lost, both against my own product's
+framing. **One integrity sentence took ungated Haiku from 6/10 tampering to 1/10**:
+the cheap prompt convention is a real mitigation and this README says so. The naive
+hook converted denials nearly as well as the gate (4/5) on the crude-spelling seed —
+but was tampered through twice: once by **hardcoding the implementation** (invisible
+to any hook; the held-out oracle layer caught it), and once by a denied agent
+switching to the node:test **options spelling** `{ skip: true }` — which the shipped
+`test-skip` rule *also* missed until it shipped closed in 1.5.1. On 1,227 real merged
+commits, the naive patterns would block **one in four legitimate commits (24.9%)**
+versus the gate's 9.0% — a ~2.75× precision edge, not the ≥5× I bet, and that 9% on
+unconfigured repos is published as the honest tuning-and-sign-off cost, not hidden.
+The loop-layer deny alone was never the product; the measured margins are whole-class
+spelling coverage, commit-layer precision plus sign-off machinery, and the layers no
+hook has. Running ledger: **52 pre-registered bets — 23 refuted, 27 held, 2
+unresolvable.**
 
 **What about false positives?** `snapshot-rewrite` was swept over **1,652 real mainline
 commits** (prettier, jest, docusaurus, immer): 216 touched snapshots, all legitimately —
 which is exactly why that rule is a `warn` asking for human confirmation, not a block,
-and why the study is committed (`harness/fp-study/`) rather than summarized away.
+and why the study is committed (`harness/fp-study/`) rather than summarized away. The
+study's narrow signal — snapshot-only diffs, ~0.06% of audited mainline commits but
+7/7 of observed tampers — now ships as the distinct `snapshot-only-rewrite` warn, the
+graduation candidate.
 
 ## Stability
 

package/dist/cli/index.js

@@ -457,6 +457,10 @@ function defaultPolicy(version = 1) {
       // mechanical but intent-ambiguous: updating a snapshot is the legitimate workflow
       // when intended output changes. WARN until the §7 evidence path earns block.
       "snapshot-rewrite": { severity: "warn" },
+      // the narrow graduation candidate: snapshot moved with NO accompanying change.
+      // ~0.06% FP on 1,652 audited mainline commits, 7/7 TP on observed tampers
+      // (harness/fp-study/). warn until a version: gate graduates it (BLOCK_SINCE).
+      "snapshot-only-rewrite": { severity: "warn" },
       // heuristic — warn until precision is measured (SPEC §7)
       "assertion-weakening": { severity: "warn" },
       "guard-removal": { severity: "warn" }
@@ -619,7 +623,13 @@ var PATTERNS2 = [
   { re: /\b(?:it|test|describe)\.(?:skip|only|todo)\b/, why: "a .skip/.only/.todo marker" },
   { re: /\b(?:xit|xtest|xdescribe)\b/, why: "an x-prefixed disabled test" },
   { re: /\b(?:fit|ftest|fdescribe)\b/, why: "an f-prefixed focused test (narrows the suite)" },
-  { re: /\bpending\(\s*\)/, why: "a pending() marker" }
+  { re: /\bpending\(\s*\)/, why: "a pending() marker" },
+  // node:test spells the same class as options — `test('x', { skip: true }, ...)` — and as
+  // a runtime call. A live evasion used the options form the day this gap was measured
+  // (PREDICTION-baselines.md): denied the rm, the agent skipped by option instead.
+  { re: /\bskip:\s*(?:true|1|['"`])/, why: "a { skip: ... } option disabling the test (node:test options form)" },
+  { re: /\btodo:\s*(?:true|['"`])/, why: "a { todo: ... } option (the test no longer fails the run)" },
+  { re: /\b(?:t|ctx|context)\.skip\(/, why: "a runtime t.skip() call" }
 ];
 var testSkip = {
   id: RULE3,
@@ -1314,6 +1324,39 @@ var snapshotRewrite = {
   }
 };
 
+// src/detectors/snapshot-only.ts
+var RULE9 = "snapshot-only-rewrite";
+var COMMIT_VIEWS = ["staged", "range"];
+function isSnapshotChange(c, policy) {
+  if (c.kind !== "file") return false;
+  if (c.op === "add") return false;
+  if (isProtected(c.path, policy, "snapshots")) return true;
+  return c.oldPath != null && isProtected(c.oldPath, policy, "snapshots");
+}
+var snapshotOnlyRewrite = {
+  id: RULE9,
+  surface: ["file"],
+  certainty: "mechanical",
+  run(changes, policy, view) {
+    if (!view || !COMMIT_VIEWS.includes(view)) return [];
+    const files = changes.filter((c) => c.kind === "file");
+    if (files.length === 0) return [];
+    const snaps = files.filter((c) => isSnapshotChange(c, policy));
+    if (snaps.length === 0 || snaps.length !== files.length) return [];
+    const first = snaps[0];
+    const names = snaps.map((c) => c.kind === "file" ? c.path : "").filter(Boolean).slice(0, 3).join(", ");
+    return [
+      makeFinding(RULE9, policy, {
+        file: first.path,
+        message: `Only recorded expectations changed in this diff (${snaps.length} snapshot file${snaps.length === 1 ? "" : "s"}, no code change to explain them).`,
+        evidence: names + (snaps.length > 3 ? ", \u2026" : ""),
+        remediation: "Pair the snapshot update with the source change that motivates it, or confirm the new expectations are genuinely intended. In 1,652 audited mainline commits this shape occurred once; in observed tamper attempts, every time.",
+        defaultSeverity: "warn"
+      })
+    ];
+  }
+};
+
 // src/detectors/index.ts
 var allDetectors = [
   noVerify,
@@ -1324,7 +1367,8 @@ var allDetectors = [
   ciTampering,
   hookTampering,
   testDeletion,
-  snapshotRewrite
+  snapshotRewrite,
+  snapshotOnlyRewrite
 ];
 
 // src/engine.ts
@@ -1339,13 +1383,13 @@ function isSuppressed(c, policy) {
 function activeChanges(changes, policy) {
   return changes.filter((c) => !isSuppressed(c, policy));
 }
-function evaluate(changes, policy, detectors = allDetectors) {
+function evaluate(changes, policy, detectors = allDetectors, view) {
   const active = activeChanges(changes, policy);
   const out = [];
   for (const d of detectors) {
     if (!isEnabled(d.id, policy)) continue;
     try {
-      out.push(...d.run(active, policy));
+      out.push(...d.run(active, policy, view));
     } catch (e) {
       process.stderr.write(`tamperward: detector "${d.id}" errored and was skipped: ${String(e)}
 `);
@@ -1705,12 +1749,15 @@ function check(opts) {
   const cwd = opts.cwd ?? process.cwd();
   let changes;
   let layer;
+  let view;
   if (opts.staged) {
     changes = diffStaged({ cwd: opts.cwd });
     layer = "local";
+    view = "staged";
   } else if (opts.worktree) {
     changes = diffWorktree({ cwd: opts.cwd });
     layer = "local";
+    view = "worktree";
   } else if (opts.diff) {
     const [base, head] = opts.diff.split(/\.{2,3}/);
     if (!base || !head) {
@@ -1720,13 +1767,14 @@ function check(opts) {
     }
     changes = diffRange(base, head, { cwd: opts.cwd });
     layer = "ci";
+    view = "range";
     policy = loadPolicyAt(mergeBaseOf(base, head, { cwd: opts.cwd }), opts.cwd) ?? defaultPolicy();
   } else {
     process.stderr.write("tamperward: specify --staged, --worktree, or --diff <base>...<head>\n");
     return 2;
   }
   const ignoredFiles = changes.filter((c) => isSuppressed(c, policy)).length;
-  let findings = evaluate(changes, policy);
+  let findings = evaluate(changes, policy, void 0, view);
   const { findings: remaining, cleared } = layer === "local" ? applyLocalSignoffs(findings, cwd, policy) : applyOobSignoffs(findings, oobFromEnv());
   findings = remaining;
   if (cleared.length) {
@@ -1963,7 +2011,7 @@ function preToolUseVerdict(input) {
     const cwd = input.cwd ?? process.cwd();
     turnBaseline(cwd, input.session_id);
     const changes = changesFromClaudeHook(input, cwd);
-    const blocks = evaluate(changes, loadPolicy(cwd)).filter((f) => f.severity === "block");
+    const blocks = evaluate(changes, loadPolicy(cwd), void 0, "tool-call").filter((f) => f.severity === "block");
     return verdict(blocks, "PreToolUse");
   } catch (e) {
     return failClosed("PreToolUse", errText(e));
@@ -1977,7 +2025,7 @@ function stopVerdict(input) {
   try {
     const base = turnBaseline(cwd, input.session_id);
     const changes = base ? diffSince(base, { cwd }) : diffWorktree({ cwd });
-    blocks = evaluate(changes, loadPolicy(cwd)).filter((f) => f.severity === "block");
+    blocks = evaluate(changes, loadPolicy(cwd), void 0, "turn").filter((f) => f.severity === "block");
   } catch (e) {
     return failClosed("Stop", errText(e));
   }
@@ -2017,7 +2065,7 @@ function runAllow(opts) {
   const policy = loadPolicy(cwd);
   let findings;
   try {
-    findings = evaluate(diffWorktree({ cwd }), policy);
+    findings = evaluate(diffWorktree({ cwd }), policy, void 0, "worktree");
   } catch {
     process.stderr.write("tamperward: cannot read the working tree (not a git repo?).\n");
     return 2;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tamperward",
-  "version": "1.4.6",
+  "version": "1.5.1",
   "description": "The deterministic agent-integrity gate. One ruleset, evaluated on the actual diff/commands as a verdict, enforced everywhere a change can be made.",
   "license": "Apache-2.0",
   "author": "hexrift",