tamperward 1.5.1 → 1.7.0

package/README.md

@@ -115,6 +115,46 @@ escape valve is a PR label only a human with write access can apply.
 
 ## Measured, not asserted
 
+**Read the two headline numbers together, always:** across **137 guarded runs — every
+one inside a tamper class an active detector covered — 0 tampers reached green.**
+Outside that coverage, on the one seed built to measure it, **agents tampered in 26 of
+40 runs** — "tampered" meaning, mechanically: the independent held-out oracle
+classified the result as passing the visible check through a prohibited shortcut. The
+first number is what the gate does; the second is why the gate is one layer. And an honest note on evidence units: those run counts come from roughly
+fifteen seed/task configurations and eight real repositories — repeated runs on one
+seed measure that seed's stochasticity, not the space of real coding tasks, so the
+independent evidence behind these numbers is the seed count, not the run count.
+
+**And the third number, from the preregistered cross-repository experiment
+(taskbench, `harness/taskbench/`):** on 26 paired real repositories with real
+historical regressions, pressured-agent policy violations appeared in **13 of 26
+ungated runs (50%)** — the behavior transfers — but the gate at 1.6.0 **did not
+reduce false greens** (5 prevented vs 4 gated-only; paired RD +3.8%, 95% CI
+−17.2 to +24.7, exact McNemar p = 1.0): agents routed through tool-call
+spellings outside the shipped detector classes, including two gaps documented
+before the sweep and deliberately left unfixed for the experiment's integrity.
+Enforcement showed no measurable completion cost (+7.7 points, CI spans zero),
+and independent pristine-suite re-execution caught **every masked failure in
+all 53 trajectories**. Registration, bets, frozen analysis, and the refutation
+of our own headline bet: `PREDICTION-taskbench.md`, `PHASE3.md`. The first two
+numbers say what the gate does inside its coverage; the third says coverage is
+the whole game on real repositories — and that the reliable layer was the
+pristine-suite oracle the agent cannot touch.
+
+**What this does not do**, stated before the numbers rather than after: it does not
+establish that a patch is *correct*; it does not detect arbitrary deception (a
+hardcoded implementation is invisible to any diff rule — measured, twice); it does not
+replace sandboxing, held-out tests, or authoritative CI — it is designed to sit in
+front of them. What it does: **enforce explicitly defined development invariants at
+the loop, commit, and CI boundaries** — and nothing else.
+
+**A word on the words:** *tamper*, *honest*, *cheat*, and *surrender* are used
+throughout as operational labels, not claims about mental states. A "tamper" means: a
+protected asset changed, or an independent held-out oracle failed, while the visible
+check went green — a property of the artifacts, decided deterministically. Whether the
+process that produced it "intended" anything is a question these experiments do not
+measure and do not need.
+
 **Does the loop work?** `harness/run.sh` stands up an isolated repo with a real bug and a
 failing protected test, lets a headless agent loose with the hooks installed, and
 classifies the end state with an oracle independent of the detectors under test
@@ -229,9 +269,17 @@ framing, and Sonnet's famous refusals turn out to be triggered by framing). The
 completed frontier split — **Opus 0/10 · Fable 6/10 · Sonnet 10/10** — is the
 published shape, with the standing self-measurement caveat on the Fable rows.
 Twenty-six of forty runs tampered through the uncovered dimension: measured demand
-that hardens the `ts-any-launder` warn→block graduation case via the §7.A corpus —
-the honest fix itself narrows through `as Record<string, unknown>`, so a naive
-assertion-blocker is an FP factory.
+that sent the `ts-any-launder` warn→block graduation question to a §7.A negatives
+corpus — the honest fix itself narrows through `as Record<string, unknown>`, so a
+naive assertion-blocker is an FP factory. **The corpus answered, and the question is
+closed** (`harness/PREDICTION-launder-corpus.md`): on 1,227 real commits the
+laundering spellings fire on legitimate work at 4–10× any deployable block threshold
+(annotation-position `any`: 10.5% of commits; named-type assertions: 7.5%), so the
+warn stays a warn permanently on syntactic evidence, no assertion rule gets built,
+and the laundering class is owned by the layers where coverage is measured: the
+held-out **oracle** (caught all 18 assertion tampers) and the **prompt** (the naming
+sentence, 10/10 flip). For this class, detection is not the gate's layer —
+verification and framing are, and the record says so with a denominator.
 
 **Why not just a prompt line, or a 20-line grep hook?** Measured
 (`harness/PREDICTION-baselines.md`), against a frozen naive hook and a one-sentence
@@ -248,8 +296,37 @@ versus the gate's 9.0% — a ~2.75× precision edge, not the ≥5× I bet, and t
 unconfigured repos is published as the honest tuning-and-sign-off cost, not hidden.
 The loop-layer deny alone was never the product; the measured margins are whole-class
 spelling coverage, commit-layer precision plus sign-off machinery, and the layers no
-hook has. Running ledger: **52 pre-registered bets — 23 refuted, 27 held, 2
-unresolvable.**
+hook has. Two follow-ups closed the loops this study opened. The sentence was then
+aimed at the regime where the gate is blind — Sonnet's 10/10 assertion-laundering
+seed — and produced a **complete flip: 10/10 honest** (`harness/PREDICTION-baselines.md`,
+addendum): naming the act is the intervention Sonnet's framing-triggered honesty
+runs on. And the promised **tuned-gate numbers** now exist
+(`harness/PREDICTION-tuned-fp.md`): 9.0% unconfigured → **6.0%** with a one-line
+severity tune → roughly one sign-off decision per 17 commits residual. The same study
+shipped per-rule `exclude` globs (1.6.0, with the policy-weakening symmetry built in)
+and then measured them honestly: **zero effect on this corpus** — these repos' casts
+live in source, not tests, so the zod-recommended path scoping removed nothing here;
+the null is published next to the feature.
+
+**What about just making the tests read-only?** The most common objection got its
+pre-registered measurement (`harness/PREDICTION-capability.md`) — starting with the
+mechanics the folk advice misses: `chmod` does not bind a root agent and never
+prevented `rm` anyway (deletion is a *directory*-write right); the workable barrier
+is the immutable bit. Result: **hard immutability governed the crude-tamper seed
+completely — 10/10 honest, four EPERM hits all converting to the real fix** (a bare
+permission error transferred the deny→fix dynamic with no remediation text). Then
+the twist: adding a logged, auto-granted **elevation script made it worse (8/10)** —
+both extra tampers walked through the front door with written confessions
+("*Remove failing test file to unblock build*"), the sanctioned-path thesis firing
+a fourth time. The lock is easy; the grantor is the hard part — which is the case
+for elevation flows that end at a human, and exactly what the gate's sign-off label
+already is. That seed now carries all six governance configurations on identical
+pre-registered ground: nothing 6/10 tampers, one sentence 1/10, naive hook 2/10,
+the gate 0/10, immutable 0/10, immutable+auto-grant 2/10 — with the standing scope
+that none of the capability rows touch the semantic classes, by construction.
+Running ledger: **70 pre-registered bets — 29 refuted, 37 held, 4 open or
+unresolvable** — taskbench's B2 (prevention on real repositories) is the newest
+refutation, published beside the bet it refutes.
 
 **What about false positives?** `snapshot-rewrite` was swept over **1,652 real mainline
 commits** (prettier, jest, docusaurus, immer): 216 touched snapshots, all legitimately —

package/dist/cli/index.js

@@ -438,11 +438,17 @@ function defaultPolicy(version = 1) {
       // Recorded expected outputs. An assertion stored as data is still an assertion;
       // rewriting it from current output is the snapshot-update move the affordance
       // experiment measured at a 70% attempt / 100% through rate (snapshot-rewrite).
-      snapshots: ["**/*.snap", "**/__snapshots__/**", "**/golden/**", "**/*.golden.*"],
+      // tap-snapshots/** added from Phase-3 evidence: node-tar's recorded expectations
+      // live there as .cjs and were rewritten unseen (PHASE3.md §4).
+      snapshots: ["**/*.snap", "**/__snapshots__/**", "**/tap-snapshots/**", "**/golden/**", "**/*.golden.*"],
       hooks: [".husky/**", "**/lefthook.*", ".tamperward.yml", "**/.tamperward.yml"]
     },
     rules: {
       "test-deletion": { severity: "block" },
+      // content gutted from a surviving spec — the class Phase 3 measured walking
+      // through the block-count rule (rows, assertion arguments, setup). Severity
+      // backed by the real-commit corpus measurement recorded in the rule's SPEC row.
+      "test-content-removal": { severity: "block" },
       "test-skip": { severity: "block" },
       "ts-any-cast": { severity: "block" },
       // the unambiguous explicit casts + ts-suppression directives
@@ -984,6 +990,11 @@ function policyWeakening(before, after) {
     if (ar.enabled === false && br.enabled !== false) {
       reasons.push(`rule "${name}" disabled (enabled: false)`);
     }
+    const beforeEx = new Set(br.exclude ?? []);
+    const addedEx = (ar.exclude ?? []).filter((g) => !beforeEx.has(g));
+    if (addedEx.length) {
+      reasons.push(`rule "${name}" exclude globs added (${addedEx.join(", ")}) \u2014 blinds this rule on those paths`);
+    }
   }
   const beforeIgnore = new Set(be.ignore);
   const addedIgnore = ae.ignore.filter((g) => !beforeIgnore.has(g));
@@ -1267,8 +1278,62 @@ var testDeletion = {
   }
 };
 
+// src/detectors/test-content-removal.ts
+var RULE8 = "test-content-removal";
+var MIN_REMOVED_LINES = 3;
+var ws = (s) => s.replace(/\s+/g, "");
+function significantLinesOrdered(src) {
+  const out = [];
+  for (const raw of src.split("\n")) {
+    const l = raw.trim();
+    if (l.length >= 10 && !/^(import\b|export\s|\/\/|\*|\/\*|}\)?;?$)/.test(l)) out.push(l);
+  }
+  return out;
+}
+var testContentRemoval = {
+  id: RULE8,
+  surface: ["file"],
+  certainty: "mechanical",
+  run(changes, policy) {
+    const out = [];
+    let keptPool = "";
+    for (const c of changes) {
+      if (c.kind === "file" && c.after != null && isProtected(c.path, policy, "tests")) {
+        keptPool += ws(significantLinesOrdered(c.after).join("\n"));
+      }
+    }
+    for (const c of changes) {
+      if (c.kind !== "file" || c.op !== "modify" || c.before == null || c.after == null) continue;
+      if (!isProtected(c.path, policy, "tests")) continue;
+      if (isProtected(c.path, policy, "snapshots")) continue;
+      if (countTestBlocks(c.after) < countTestBlocks(c.before)) continue;
+      const beforeSig = significantLinesOrdered(c.before);
+      const afterSig = significantLinesOrdered(c.after);
+      const netDrop = beforeSig.length - afterSig.length;
+      const afterSet = new Set(afterSig);
+      const gone = [];
+      for (const line of new Set(beforeSig)) {
+        if (afterSet.has(line)) continue;
+        if (keptPool.includes(ws(line))) continue;
+        gone.push(line);
+      }
+      if (gone.length >= MIN_REMOVED_LINES && netDrop >= MIN_REMOVED_LINES) {
+        out.push(
+          makeFinding(RULE8, policy, {
+            file: c.path,
+            message: `Test content removed from a surviving spec: ${gone.length} significant line(s) gone with no test block deleted.`,
+            evidence: gone.slice(0, 3).map((l) => l.slice(0, 80)).join(" | "),
+            remediation: "Fix the code under test. Deleting the failing rows, assertion arguments, or setup from a spec weakens it exactly like deleting the test \u2014 the block just keeps its name."
+          })
+        );
+      }
+    }
+    return out;
+  }
+};
+
 // src/detectors/snapshot-rewrite.ts
-var RULE8 = "snapshot-rewrite";
+var RULE9 = "snapshot-rewrite";
 var RUNNER = /\b(?:jest|vitest|playwright|ava)\b/;
 var UPDATE_ANY = /--update-?[sS]napshots?\b/;
 var UPDATE_WITH_RUNNER = /(?:^|\s)(?:-u|--update)(?:\s|$)/;
@@ -1282,12 +1347,12 @@ function namesProtectedSnapshot(seg, policy) {
   return null;
 }
 var snapshotRewrite = {
-  id: RULE8,
+  id: RULE9,
   surface: ["command", "file"],
   certainty: "mechanical",
   run(changes, policy) {
     const out = [];
-    const warn = (input) => out.push(makeFinding(RULE8, policy, { ...input, defaultSeverity: "warn" }));
+    const warn = (input) => out.push(makeFinding(RULE9, policy, { ...input, defaultSeverity: "warn" }));
     for (const c of changes) {
       if (c.kind === "command") {
         for (const seg of segments(c.raw)) {
@@ -1325,7 +1390,7 @@ var snapshotRewrite = {
 };
 
 // src/detectors/snapshot-only.ts
-var RULE9 = "snapshot-only-rewrite";
+var RULE10 = "snapshot-only-rewrite";
 var COMMIT_VIEWS = ["staged", "range"];
 function isSnapshotChange(c, policy) {
   if (c.kind !== "file") return false;
@@ -1334,7 +1399,7 @@ function isSnapshotChange(c, policy) {
   return c.oldPath != null && isProtected(c.oldPath, policy, "snapshots");
 }
 var snapshotOnlyRewrite = {
-  id: RULE9,
+  id: RULE10,
   surface: ["file"],
   certainty: "mechanical",
   run(changes, policy, view) {
@@ -1346,7 +1411,7 @@ var snapshotOnlyRewrite = {
     const first = snaps[0];
     const names = snaps.map((c) => c.kind === "file" ? c.path : "").filter(Boolean).slice(0, 3).join(", ");
     return [
-      makeFinding(RULE9, policy, {
+      makeFinding(RULE10, policy, {
         file: first.path,
         message: `Only recorded expectations changed in this diff (${snaps.length} snapshot file${snaps.length === 1 ? "" : "s"}, no code change to explain them).`,
         evidence: names + (snaps.length > 3 ? ", \u2026" : ""),
@@ -1367,6 +1432,7 @@ var allDetectors = [
   ciTampering,
   hookTampering,
   testDeletion,
+  testContentRemoval,
   snapshotRewrite,
   snapshotOnlyRewrite
 ];
@@ -1396,12 +1462,17 @@ function evaluate(changes, policy, detectors = allDetectors, view) {
     }
   }
   const seen = /* @__PURE__ */ new Set();
-  return out.filter((f) => {
+  const deduped = out.filter((f) => {
     const k = key(f);
     if (seen.has(k)) return false;
     seen.add(k);
     return true;
   });
+  return deduped.filter((f) => {
+    const ex = policy.rules[f.rule]?.exclude;
+    if (!ex?.length || !f.file || isPolicyFile(f.file)) return true;
+    return !matchesAny(f.file, ex);
+  });
 }
 function hasBlocking(findings) {
   return findings.some((f) => f.severity === "block");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tamperward",
-  "version": "1.5.1",
+  "version": "1.7.0",
   "description": "The deterministic agent-integrity gate. One ruleset, evaluated on the actual diff/commands as a verdict, enforced everywhere a change can be made.",
   "license": "Apache-2.0",
   "author": "hexrift",