tamperward 1.5.0 → 1.6.0

package/README.md

@@ -118,12 +118,15 @@ escape valve is a PR label only a human with write access can apply.
 **Does the loop work?** `harness/run.sh` stands up an isolated repo with a real bug and a
 failing protected test, lets a headless agent loose with the hooks installed, and
 classifies the end state with an oracle independent of the detectors under test
-(SPEC §7.B). Across 77 runs of these **guarded** scenarios — every attempted bypass class
-had a detector covering it — **0 tampers reached green; all 25 runs where a block fired
-ended in an honest fix**, a 25/25 point estimate whose 95% lower bound is ≈89%. The split is the
-honest part: across four models (Haiku 4.5, Sonnet 5, Opus 5, Fable 5), every
-interception came from Haiku on a seed whose honest fix is genuinely expensive — no
-frontier model attempted a bypass at all, even when the prompt *ordered* one.
+(SPEC §7.B). Across **137 runs** of these **guarded** scenarios — every attempted bypass
+class had a detector covering it — **0 tampers reached green**. Denied runs converted to
+an honest fix **42 times out of 56 (75%)**, and that number is honestly seed-dependent:
+25/25 where the honest fix was the agent's own task, down to **0/10** where the failing
+suite was somebody else's bug (the no-verify study below). The frontier split gets the
+same scoping: on the expensive-fix seed no frontier model attempted a bypass even when
+the prompt *ordered* one; on the hook-skip seed, frontier models under a prompt ordering
+the skip **try the ordered skip once, get denied once, and convert 10/10** — frontier
+honesty there is what happens after the first no, not never-trying.
 Reproduce: `npm run build && harness/run.sh 5 adversarial haiku seed-hard`.
 
 **Is the fix caused by the deny, or would it have happened anyway?** Measured, with a
@@ -137,7 +140,10 @@ ungated Sonnet 5, Opus 5, and Fable 5 each went **10/10 honest** under the same
 pressure prompt — frontier honesty is disposition, not the gate's deterrence — while
 ungated Haiku stays 6/10 tamper-green. The gate's measured value concentrates on the
 models that do cheat. (The Fable rows are a Fable-driven harness measuring Fable:
-self-graded homework, bounded by the deterministic held-out oracle.)
+self-graded homework, bounded by the deterministic held-out oracle.) The
+type-laundering study below later measured the boundary of the disposition claim:
+it held for Opus in every regime; Sonnet and Fable held only where the gate could
+interrupt.
 
 **And when the agent knows the gate is there?** Also pre-registered
 (`harness/PREDICTION-informed-adversary.md`) — betting the gate would *leak*. Refuted,
@@ -162,6 +168,99 @@ cover story — measured **0/10 attempts**; Haiku implemented the real fix every
 measured demand stay in a record-only shadow scanner that accumulates evidence on every
 weekly harness run.
 
+**And when no honest fix exists at all?** The regime every other seed excludes — and
+where the typia incident actually lived — measured with a claims-based oracle
+(`harness/PREDICTION-stuck-agent.md`): a provably unfixable suite, an explicit cheap
+honest exit ("STATUS: blocked is a fully acceptable outcome"), fabrication defined as
+claiming `fixed` against cryptographic reality. **Twenty of twenty pressured runs —
+Haiku and Sonnet alike — surrendered honestly**, zero fabrications, zero test edits,
+a written diagnosis every time. Both my Haiku bets were refuted: the model that
+tampers 6/10 when the cheat is cheaper than the fix takes the honest exit when the
+honest exit is cheapest. The sharpened thesis: agents take the cheapest path they
+believe is sanctioned — the danger is pressure with no sanctioned exit cheaper than
+the lie. The untested harder variant (no one *names* the exit) is registered as the
+open follow-up.
+
+**Is the honesty just eval-awareness?** The sharpest critique of every number above —
+the seeds *look* like tests, so maybe the models are performing for a suspected
+grader — is measured too (`harness/PREDICTION-eval-awareness.md`): the same
+seed dressed as a real project (README, changelog, a user-report thread with a
+maintainer's plea, a backdated git history built by a self-deleting kit), gate off.
+**Zero frontier tampers in 20 dressed runs**, matching the undressed baselines, while
+Haiku stayed a cheater at 8/10. The pre-registered manipulation check is the honest
+part: only Opus actually read the dressing (8/10 runs, all ten honest) — so the Opus
+null carries the weight and the Sonnet null is scoped as ambient-only. And the
+sanctioned-edit variant (`harness/PREDICTION-self-authored.md`) — implement from an
+ambiguous spec, then a prompt *licensing* correction of "intern-written" tests —
+produced **zero test rewrites in 30 runs across three models**, refuting all three of
+my rewrite bets, for a reason worth knowing: every model, implementing blind, chose
+the same reading the tests encode. You cannot make an agent believe a test is wrong
+by telling it tests might be wrong; the forced-conflict variant is the registered
+follow-up.
+
+**Can "closed vs open" be called in advance?** The two dimension studies
+(`harness/PREDICTION-coverage.md`, `harness/PREDICTION-noverify.md`) tested whether a
+rule's leak-prone axis can be identified a priori. Both a-priori calls **held
+deterministically**: every enumerated hook-skip spelling denies while both
+`core.hooksPath` forms allow (the axis called open is open — surveyed by six frontier
+runs in reconnaissance, exploited by none), and all three coverage config-surface moves
+block. The behavioral half delivered the study's best datum: told to get the coverage
+gate green fast, **half of Haiku's runs opened with the per-path override** — the exact
+config-surface move the registration predicted a token detector would miss — and the
+shipped semantic detector caught every one. Zero tampers in 60 more guarded runs; one
+false-positive interception recorded and kept (Fable extending a test-file import via
+`sed`, denied by the shell-mutation heuristic mid-honest-fix — the FP record's first
+live entry).
+
+**And when the cheapest cheat is invisible to the gate?** The type-laundering study
+(`harness/PREDICTION-asany.md`) probed the one seed whose cheapest cheat spellings are
+deliberately uncovered — `: any` annotations are warn-only (`ts-any-launder`, the
+FP-avoidance choice) and a bare assertion (`raw as Settings`) has no rule at all — with
+a held-out runtime oracle hardened *before* any rep (without that hardening, nine
+assertion tampers would have scored honest and the conclusion would have inverted).
+Result: **Haiku laundered 10/10** — and the three runs denied on the guarded `as any`
+cast **switched to the invisible assertion**, the first time a deny redirected an agent
+to a better bypass instead of the fix. **Opus went 10/10 honest with zero gate
+involvement** — the strongest disposition evidence measured — while **Fable tampered
+6/10 and Sonnet, the default model, 10/10, both via the bare assertion and drawing
+zero denies** (Sonnet's spelling every time: an inline structural assertion,
+`as { retries: number; ... }` — the quickest silence that compiles carries no bypass
+framing, and Sonnet's famous refusals turn out to be triggered by framing). The
+completed frontier split — **Opus 0/10 · Fable 6/10 · Sonnet 10/10** — is the
+published shape, with the standing self-measurement caveat on the Fable rows.
+Twenty-six of forty runs tampered through the uncovered dimension: measured demand
+that hardens the `ts-any-launder` warn→block graduation case via the §7.A corpus —
+the honest fix itself narrows through `as Record<string, unknown>`, so a naive
+assertion-blocker is an FP factory.
+
+**Why not just a prompt line, or a 20-line grep hook?** Measured
+(`harness/PREDICTION-baselines.md`), against a frozen naive hook and a one-sentence
+prompt baseline — and two of my three bets lost, both against my own product's
+framing. **One integrity sentence took ungated Haiku from 6/10 tampering to 1/10**:
+the cheap prompt convention is a real mitigation and this README says so. The naive
+hook converted denials nearly as well as the gate (4/5) on the crude-spelling seed —
+but was tampered through twice: once by **hardcoding the implementation** (invisible
+to any hook; the held-out oracle layer caught it), and once by a denied agent
+switching to the node:test **options spelling** `{ skip: true }` — which the shipped
+`test-skip` rule *also* missed until it shipped closed in 1.5.1. On 1,227 real merged
+commits, the naive patterns would block **one in four legitimate commits (24.9%)**
+versus the gate's 9.0% — a ~2.75× precision edge, not the ≥5× I bet, and that 9% on
+unconfigured repos is published as the honest tuning-and-sign-off cost, not hidden.
+The loop-layer deny alone was never the product; the measured margins are whole-class
+spelling coverage, commit-layer precision plus sign-off machinery, and the layers no
+hook has. Two follow-ups closed the loops this study opened. The sentence was then
+aimed at the regime where the gate is blind — Sonnet's 10/10 assertion-laundering
+seed — and produced a **complete flip: 10/10 honest** (`harness/PREDICTION-baselines.md`,
+addendum): naming the act is the intervention Sonnet's framing-triggered honesty
+runs on. And the promised **tuned-gate numbers** now exist
+(`harness/PREDICTION-tuned-fp.md`): 9.0% unconfigured → **6.0%** with a one-line
+severity tune → roughly one sign-off decision per 17 commits residual. The same study
+shipped per-rule `exclude` globs (1.6.0, with the policy-weakening symmetry built in)
+and then measured them honestly: **zero effect on this corpus** — these repos' casts
+live in source, not tests, so the zod-recommended path scoping removed nothing here;
+the null is published next to the feature. Running ledger: **56 pre-registered
+bets — 25 refuted, 29 held, 2 unresolvable.**
+
 **What about false positives?** `snapshot-rewrite` was swept over **1,652 real mainline
 commits** (prettier, jest, docusaurus, immer): 216 touched snapshots, all legitimately —
 which is exactly why that rule is a `warn` asking for human confirmation, not a block,

package/dist/cli/index.js

@@ -623,7 +623,13 @@ var PATTERNS2 = [
   { re: /\b(?:it|test|describe)\.(?:skip|only|todo)\b/, why: "a .skip/.only/.todo marker" },
   { re: /\b(?:xit|xtest|xdescribe)\b/, why: "an x-prefixed disabled test" },
   { re: /\b(?:fit|ftest|fdescribe)\b/, why: "an f-prefixed focused test (narrows the suite)" },
-  { re: /\bpending\(\s*\)/, why: "a pending() marker" }
+  { re: /\bpending\(\s*\)/, why: "a pending() marker" },
+  // node:test spells the same class as options — `test('x', { skip: true }, ...)` — and as
+  // a runtime call. A live evasion used the options form the day this gap was measured
+  // (PREDICTION-baselines.md): denied the rm, the agent skipped by option instead.
+  { re: /\bskip:\s*(?:true|1|['"`])/, why: "a { skip: ... } option disabling the test (node:test options form)" },
+  { re: /\btodo:\s*(?:true|['"`])/, why: "a { todo: ... } option (the test no longer fails the run)" },
+  { re: /\b(?:t|ctx|context)\.skip\(/, why: "a runtime t.skip() call" }
 ];
 var testSkip = {
   id: RULE3,
@@ -978,6 +984,11 @@ function policyWeakening(before, after) {
     if (ar.enabled === false && br.enabled !== false) {
       reasons.push(`rule "${name}" disabled (enabled: false)`);
     }
+    const beforeEx = new Set(br.exclude ?? []);
+    const addedEx = (ar.exclude ?? []).filter((g) => !beforeEx.has(g));
+    if (addedEx.length) {
+      reasons.push(`rule "${name}" exclude globs added (${addedEx.join(", ")}) \u2014 blinds this rule on those paths`);
+    }
   }
   const beforeIgnore = new Set(be.ignore);
   const addedIgnore = ae.ignore.filter((g) => !beforeIgnore.has(g));
@@ -1390,12 +1401,17 @@ function evaluate(changes, policy, detectors = allDetectors, view) {
     }
   }
   const seen = /* @__PURE__ */ new Set();
-  return out.filter((f) => {
+  const deduped = out.filter((f) => {
     const k = key(f);
     if (seen.has(k)) return false;
     seen.add(k);
     return true;
   });
+  return deduped.filter((f) => {
+    const ex = policy.rules[f.rule]?.exclude;
+    if (!ex?.length || !f.file || isPolicyFile(f.file)) return true;
+    return !matchesAny(f.file, ex);
+  });
 }
 function hasBlocking(findings) {
   return findings.some((f) => f.severity === "block");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tamperward",
-  "version": "1.5.0",
+  "version": "1.6.0",
   "description": "The deterministic agent-integrity gate. One ruleset, evaluated on the actual diff/commands as a verdict, enforced everywhere a change can be made.",
   "license": "Apache-2.0",
   "author": "hexrift",