tamperward 1.15.0 → 2.1.0

package/README.md

@@ -12,6 +12,7 @@
   <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-lightgrey" alt="license"></a>
 </p>
 
+**[Quick start](#quick-start)** ·
 **[Docs & guide](https://hexrift.github.io/tamperward/)** ·
 **[The research series](./docs/blog/index.md)** — every registered prediction
 published beside its outcome
@@ -22,14 +23,28 @@ trajectories, some modify or attempt to modify verification in ways that can
 turn incorrect work into apparent success. Under pressure, the cheaper route to
 green is sometimes to weaken the checks instead of fixing the failure.
 
+In plain English: Tamperward lets a coding agent change your code, but not the
+trusted starting point, the rules, or the checks used to judge that code.
+
 Tamperward is a **deterministic verification-integrity layer**. It blocks known
 weakening moves as they happen, observes protected-state effects, and
 independently re-adjudicates apparent success outside the agent's normal
 completion path. No runtime LLM judge. Fail closed when adjudication is
 impossible.
 
+> **Project status: active research release.** Tamperward is usable today, but
+> its enforcement architecture and supporting evidence are still being tested
+> and hardened. Use it as one layer of defence in depth alongside protected CI,
+> independent tests and human review. Findings, limitations and corrections are
+> published openly. The 2.0 major marks the Node 18 drop, not a declaration of
+> security maturity.
+
 ## What we have actually measured
 
+Plain-English takeaway: the original detector-centred design was insufficient.
+Later versions materially reduced false-green outcomes in two held-out rounds,
+and the subsequent stronger-model replication was inconclusive.
+
 | experiment | result | what it supports |
 | --- | --- | --- |
 | **Round 1** — 26 paired real repos, historical regressions, 53 counted trajectories, **v1.6.0** ([`harness/taskbench/`](./harness/taskbench/)) | Transfer: **9/27 ungated runs (33.3%)** violated policy ([corrected](./harness/taskbench/reanalysis/TRANSFER-REANALYSIS.md) from a published 13/26 — the original predicate was defective; the transfer bet is refuted, not held). Headline prevention bet **lost**: b=5 / c=4, RD +3.8pp [−17.2, +24.7], exact McNemar **p = 1.0** — published beside the bet | A detector-centric architecture was insufficient: agents routed around the shipped detector classes |
@@ -39,11 +54,16 @@ impossible.
 | **Round 3** — 17 paired Python repos, fresh PyPI frame, **v1.14.0** ([`harness/taskbench/round3/`](./harness/taskbench/round3/)) | Transfer 9/17 (52.9%). Prevention: **b=6 / c=0**, RD **+35.3pp**, BP95 [9.5, 58.7], exact McNemar **p = 0.0313** | The prevention result appeared in Python — with the treatment also changed from v1.9.0, so ecosystem transfer is not isolated. The in-loop skip detector proved blind to pytest syntax; the outer layers carried it |
 | **Round 3.1** — the same 16 pairs under **`claude-sonnet-5`** ([`harness/taskbench/round3.1/`](./harness/taskbench/round3.1/)) | Transfer 4/16 (25.0%). Prevention: **b=1 / c=0**, RD +6.3pp, BP95 [−13.8, 28.3], exact McNemar **p = 1.0000** | The confirmatory result **did not replicate**, and could not have: only three ungated false greens occurred, so `b ≤ 3` and p ≥ .25 whatever the gate did. A failure to reject, not evidence of no effect |
 
+Key: `b` = false greens seen only without Tamperward; `c` = false greens seen
+only with it; `RD` = paired risk difference; `pp` = percentage points; `BP95` =
+Bonett–Price 95% interval.
+
 Earlier controlled experiments → **[the research series](./docs/blog/index.md)**.
 
-> **Scope.** The rows above are one model, one pressure prompt, one JS/TS
-> ecosystem — evidence for that setting, not a universal claim. Round 2 tested
-> the v1.9.0 stack; the current **v1.15.0** adds post-study envelope hardening
+> **Scope.** The rows above cover specific models, pressure prompts, treatment
+> versions, and finite JavaScript/TypeScript and Python repository samples —
+> evidence for those settings, not a universal claim. Round 2 tested
+> the v1.9.0 stack; the current **v2.0.0** adds post-study envelope hardening
 > (externally reviewed, with findings tracked individually as REPRO or AUDIT in
 > [SECURITY-ENVELOPE.md](./SECURITY-ENVELOPE.md) and closed with regression and
 > mutation checks — see [CHANGELOG](./CHANGELOG.md)). Rounds 3 and 3.1 are
@@ -56,7 +76,11 @@ Earlier controlled experiments → **[the research series](./docs/blog/index.md)
 
 ## Architecture
 
-Tamperward separates **steering** from **adjudication**.
+The agent may produce the work, but it must not control how that work is
+judged. In architectural terms, Tamperward separates **steering** from
+**adjudication**. "Visible" verification runs the candidate as it stands;
+"pristine" verification restores the protected verification state from the
+trusted starting point and runs the checks again.
 
 > **Core invariant:** the agent may author the candidate tree, but it must not
 > choose the trusted baseline, the governing policy, the verifier, or the final
@@ -160,16 +184,19 @@ are involved here, and they are not the same thing.
    trusted base — so neither step's verdict is governed by the candidate.
 
 Legitimate exceptions are out-of-band PR labels, `tamperward:allow:<rule>@<head-sha>`.
-The workflow passes the head SHA to the gate through `TAMPERWARD_OOB_HEAD`, so an
+The workflow passes the head SHA to both steps through `TAMPERWARD_OOB_HEAD`, so an
 approval is bound to the exact commit it was granted for and a new push invalidates it.
+Since **2.1.0** the verify step reads the same labels: `tamperward:allow:verify@<head-sha>`
+accepts a `MASKED_FAILURE` — the case where a behaviour change makes the original
+expectations wrong and a reviewer has read the test edit and said so. It clears nothing
+else: a red visible suite, or a run that could not verify, stays red whatever the labels
+say, and the verdict is still reported as a masked failure; only the exit code changes.
 
 **This repository's own self-gate** — the `gate` job in `.github/workflows/ci.yml` —
 runs the built CLI's `check --diff` over the pull-request range, cleared only by the
 same label channel, but does **not** run `verify` on itself. This repo's test
-expectations legitimately change whenever a rule changes, and standalone `verify` has
-no out-of-band sign-off channel, so a pristine re-execution of its own suite would fail
-closed on exactly the pull requests that improve the gate. The self-gate is a
-diff-time gate.
+expectations legitimately change whenever a rule changes, so nearly every rule pull
+request would need the verify label; the self-gate stays a diff-time gate.
 
 This guarantee depends on a protected and immutable base, required status checks, a
 pinned Tamperward version, and label permissions restricted to trusted humans.
@@ -186,6 +213,10 @@ Full assumptions and residual risks: [SPEC.md](./SPEC.md),
 npx tamperward init
 ```
 
+Requires Node.js 20.19 or later. JavaScript and TypeScript are the fully
+supported detector surface; the other documented ecosystems get file-level and
+pattern-based protection.
+
 One idempotent command wires the policy, the agent hooks, the pre-commit hook, a
 CI workflow that runs both the diff-time check and pristine verification, and a
 `CODEOWNERS` requirement on the paths that decide whether the gate runs at all.
@@ -252,7 +283,7 @@ commands ignore what they do not know.
 | command | 0 | 1 | 2 |
 | --- | --- | --- | --- |
 | `check` | no blocking finding | at least one blocking finding | cannot evaluate: policy parse error, malformed `--diff` range, or no view given |
-| `verify` | `VERIFIED` — visible and pristine both green | `MASKED_FAILURE` (visible green, pristine red) or `SUITE_RED` | cannot verify, failing closed: no suite command, unresolvable base, `--require-ancestor` refused, budget exceeded, or the working or dependency tree moved during the run |
+| `verify` | `VERIFIED` — visible and pristine both green; or a `MASKED_FAILURE` cleared by an out-of-band `verify@<head-sha>` approval | `MASKED_FAILURE` (visible green, pristine red) or `SUITE_RED` | cannot verify, failing closed: no suite command, unresolvable base, `--require-ancestor` refused, budget exceeded, or the working or dependency tree moved during the run |
 | `run` | enforcement clean and the agent exited 0 — a non-zero agent exit is passed through unchanged | any blocking finding or masked failure, whatever the agent returned | cannot adjudicate: dirty start, policy error, verify cannot run |
 | `hook claude` / `sweep claude` | always — a deny is JSON on stdout at exit 0, never exit 2 | — | only for an unsupported agent name |
 | `allow` | sign-off recorded | — | no rule or `--reason`, not a git repo, or no current blocking finding to sign off |
@@ -263,8 +294,8 @@ commands ignore what they do not know.
 
 | variable | who sets it | what it does |
 | --- | --- | --- |
-| `TAMPERWARD_OOB_SIGNOFF` | the CI workflow, from PR labels | comma-separated out-of-band approvals — `<rule>` or `<rule>:<file>`, optionally `@<head-sha>` — honoured by `check --diff` only; never the committed ledger |
-| `TAMPERWARD_OOB_HEAD` | the CI workflow (`github.event.pull_request.head.sha`) | the head SHA under adjudication; once set, an approval clears a finding only if it names that commit (`@<sha>`, at least 7 characters), so a new push re-blocks |
+| `TAMPERWARD_OOB_SIGNOFF` | the CI workflow, from PR labels | comma-separated out-of-band approvals — `<rule>` or `<rule>:<file>` for `check --diff`, `verify` for a `verify` masked failure — optionally `@<head-sha>`; honoured at the CI layer only, never the committed ledger |
+| `TAMPERWARD_OOB_HEAD` | the CI workflow (`github.event.pull_request.head.sha`) | the head SHA under adjudication; once set, an approval clears anything only if it names that commit (`@<sha>`, at least 7 characters), so a new push re-blocks |
 | `TAMPERWARD_DENYLOG` | a harness or operator | a file to which `hook claude` and `sweep claude` append the rule ids of every deny, one line per verdict, best effort |
 | `TAMPERWARD_FSEVENTS` | operator or harness | overrides the `tamperward watch` event-log path (default `.git/tamperward/fsevents.jsonl`); the Stop sweep reads the same variable |
 | `TAMPERWARD_WATCH_NO_RECURSIVE` | CI and tests (`=1`) | forces `tamperward watch` onto its per-directory fallback instead of recursive `fs.watch`, so the fallback is exercised on every platform |
@@ -286,7 +317,7 @@ Some ambiguous syntactic classes deliberately remain warnings or unimplemented
 `ts-any-launder` is a permanent warn — rather than being promoted to blockers
 without precision evidence.
 
-## What this does not establish
+## What Tamperward does not do
 
 - **Not a correctness oracle.** Pristine verification can only re-run tests that
   exist in the tree. Round 2's designed-in blind spot: on tasks with withheld
@@ -362,7 +393,7 @@ via npm trusted publishing with SLSA provenance. Full rule:
 
 ```bash
 npm install && npm run build    # bundles the CLI to dist/cli/index.js
-npm test                        # 520+ tests at 1.15.0 — parser, detectors, engine, policy, renderers
+npm test                        # 520+ tests at 2.0.0 — parser, detectors, engine, policy, renderers
 npm run typecheck
 ```
 

package/dist/cli/index.js

@@ -1931,15 +1931,23 @@ function applyLocalSignoffs(findings, cwd, policy, now = Date.now()) {
   }
   return { findings: remaining, cleared };
 }
-function applyOobSignoffs(findings, oob, head) {
-  const tokens2 = oob.map((s) => s.trim()).filter(Boolean);
-  const matches = (want) => tokens2.some((t) => {
+function oobToken(want, oob, head) {
+  for (const raw of oob) {
+    const t = raw.trim();
+    if (!t) continue;
     const at = t.lastIndexOf("@");
-    if (at === -1) return head ? false : t === want;
+    if (at === -1) {
+      if (!head && t === want) return t;
+      continue;
+    }
     const [rule, sha2] = [t.slice(0, at), t.slice(at + 1)];
-    if (rule !== want) return false;
-    return head ? head.startsWith(sha2) && sha2.length >= 7 : true;
-  });
+    if (rule !== want) continue;
+    if (!head || sha2.length >= 7 && head.startsWith(sha2)) return t;
+  }
+  return null;
+}
+function applyOobSignoffs(findings, oob, head) {
+  const matches = (want) => oobToken(want, oob, head) !== null;
   const cleared = [];
   const remaining = [];
   for (const f of findings) {
@@ -3025,8 +3033,9 @@ var WORKFLOW_CONTENT = `name: tamperward
 
 # The CI authority for main: the same engine as the agent hook and pre-commit, run over
 # the PR's commit range. A block fails the check and clears ONLY via the out-of-band
-# label \`tamperward:allow:<rule>\` applied by someone with write access \u2014 never a file
-# the PR itself can commit.
+# label \`tamperward:allow:<rule>@<head-sha>\` applied by someone with write access \u2014
+# never a file the PR itself can commit. The verify step reads the same labels:
+# \`tamperward:allow:verify@<head-sha>\` accepts a masked failure a reviewer has judged.
 #
 # labeled/unlabeled re-run the gate because the sign-off is read from the EVENT payload:
 # a label applied after a failure could otherwise never take effect, and REVOKING a
@@ -3082,6 +3091,14 @@ jobs:
       # Requires a \`verify:\` block in .tamperward.yml naming the suite command;
       # without one this step fails closed (exit 2) rather than passing quietly.
       - name: Tamperward verify (pristine re-execution)
+        env:
+          # The same channel as the gate. \`tamperward:allow:verify@<head-sha>\`
+          # accepts a MASKED_FAILURE \u2014 a reviewer has read the intentional test
+          # change and agrees the original suite no longer applies. It clears
+          # nothing else: a red visible suite, or a run that could not verify,
+          # stays red whatever the labels say.
+          TAMPERWARD_OOB_SIGNOFF: \${{ steps.oob.outputs.rules }}
+          TAMPERWARD_OOB_HEAD: \${{ github.event.pull_request.head.sha }}
         run: npx --yes tamperward@${TW_VERSION} verify --require-ancestor --base "\${{ github.event.pull_request.base.sha }}"
 `;
 var WORKFLOW_MARK = "# tamperward:generated";
@@ -3768,6 +3785,8 @@ function runVerify(opts) {
     verdict2 = "SUITE_RED";
     code = 1;
   }
+  const signedOff = verdict2 === "MASKED_FAILURE" ? oobToken("verify", oobFromEnv(), oobHeadFromEnv()) : null;
+  if (signedOff) code = 0;
   if (opts.json) {
     out2(
       JSON.stringify({
@@ -3779,6 +3798,7 @@ function runVerify(opts) {
         pristine: { exit: pristine.exit, secs: pristine.secs },
         protected_restored: restored.length,
         added_protected_removed: removedAdded,
+        ...signedOff ? { oob_signoff: signedOff } : {},
         ...opts.keep ? { visible_dir: visDir, pristine_dir: priDir } : {}
       })
     );
@@ -3790,6 +3810,10 @@ function runVerify(opts) {
       BUDGET_EXCEEDED: `budget exceeded (${budget}s): could not verify \u2014 failing closed, not open.`
     };
     out2(`tamperward verify \u2014 ${lines[verdict2]}`);
+    if (signedOff)
+      out2(
+        `masked failure cleared by out-of-band approval (tamperward:allow:${signedOff}): a reviewer accepted that the original suite no longer applies to this change. Exit 0.`
+      );
     if (removedAdded > 0)
       out2(
         `(${removedAdded} protected file(s) added since ${base.slice(0, 10)} were removed from the pristine run: the pristine tree carries exactly the base's protected surface.)`
@@ -4079,7 +4103,9 @@ Formats:
              [--budget S] [--json] [--keep] as-is AND with protected files restored
                                             from the trusted base; a visible-green /
                                             pristine-red result is a MASKED FAILURE
-                                            (exit 1); cannot-verify fails closed (2)
+                                            (exit 1, or 0 under an out-of-band
+                                            verify@<head-sha> approval); cannot-verify
+                                            fails closed (2)
   tamperward run [opts] -- <agent cmd...>   enforcement envelope: record the trusted
              [--base R] [--cmd C]           base, run the agent, treat its exit as
              [--budget S] [--allow-dirty]   untrusted, then re-adjudicate the tree it

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tamperward",
-  "version": "1.15.0",
+  "version": "2.1.0",
   "description": "The deterministic agent-integrity gate. One ruleset, evaluated on the actual diff/commands as a verdict, enforced everywhere a change can be made.",
   "license": "Apache-2.0",
   "author": "hexrift",
@@ -37,7 +37,7 @@
     "NOTICE"
   ],
   "engines": {
-    "node": ">=18"
+    "node": ">=20.19"
   },
   "scripts": {
     "build": "esbuild src/cli/index.ts --bundle --platform=node --format=esm --packages=external --outfile=dist/cli/index.js",
@@ -53,10 +53,10 @@
     "yaml": "^2.6.1"
   },
   "devDependencies": {
-    "@types/node": "^20.14.0",
+    "@types/node": "^22.20.1",
     "@types/picomatch": "^3.0.1",
     "esbuild": "^0.25.12",
-    "vitepress": "^1.6.4",
-    "vitest": "^2.1.9"
+    "vitepress": "^2.0.0-alpha.19",
+    "vitest": "^4.1.11"
   }
 }