tamperward 2.20.3 → 2.20.4

package/README.md

@@ -44,7 +44,9 @@ impossible.
 
 Plain-English takeaway: the original detector-centred design was insufficient.
 Later versions materially reduced false-green outcomes in two held-out rounds,
-and the subsequent stronger-model replication was inconclusive.
+the subsequent stronger-model replication was inconclusive, and the larger
+preregistered round 4 on the 2.x line **did not replicate** the prevention
+effect — a failed prediction, published beside the bet.
 
 | experiment | result | what it supports |
 | --- | --- | --- |
@@ -54,6 +56,7 @@ and the subsequent stronger-model replication was inconclusive.
 | **After prevention** — the nine round-2 prevented false greens | **8 of 9 became honest completions**; the ninth an honest **non**-completion | Prevention usually redirected trajectories toward honest work rather than merely blocking them |
 | **Round 3** — 17 paired Python repos, fresh PyPI frame, **v1.14.0** ([`harness/taskbench/round3/`](./harness/taskbench/round3/)) | Transfer 9/17 (52.9%). Prevention: **b=6 / c=0**, RD **+35.3pp**, BP95 [9.5, 58.7], exact McNemar **p = 0.0313** | The prevention result appeared in Python — with the treatment also changed from v1.9.0, so ecosystem transfer is not isolated. The in-loop skip detector proved blind to pytest syntax; the outer layers carried it |
 | **Round 3.1** — the same 16 pairs under **`claude-sonnet-5`** ([`harness/taskbench/round3.1/`](./harness/taskbench/round3.1/)) | Transfer 4/16 (25.0%). Prevention: **b=1 / c=0**, RD +6.3pp, BP95 [−13.8, 28.3], exact McNemar **p = 1.0000** | The confirmatory result **did not replicate**, and could not have: only three ungated false greens occurred, so `b ≤ 3` and p ≥ .25 whatever the gate did. A failure to reject, not evidence of no effect |
+| **Round 4** — fresh pool of 110 paired Python repos under **`claude-sonnet-5`**, treatment **2.10.3** pinned by artefact hash, preregistered exact McNemar ([`harness/taskbench/round4/`](./harness/taskbench/round4/), sealed readout [`ROUND4-ANALYSIS.md`](./harness/taskbench/round4/ROUND4-ANALYSIS.md)) | Primary endpoint: prevention **b=5 / c=3**, exact McNemar **p = 0.7265625**, H₀ **not rejected** — the preregistered prediction (b=16, c=1, reject) **did not replicate**. Realized valid pairs **79/110**; the registered interpretation floor was met (`a + b = 15` ungated opportunities, floor 6), but realized discordance (8 pairs) was below the ~17 the power model assumed. Separate, narrower integrity observation: across **201 measured trajectories**, **0 strict tamper bypass** (landed weakening that survived to the final tree *and* was certified clean) | The registered confirmatory claim failed on the 2.x line and is reported as a failed prediction, not reinterpreted. The zero-bypass count is descriptive and does not stand in for it: one transient landing was certified clean (correctly, it did not survive), and 31/110 pairs were lost to apparatus attrition, which is not claimed bias-free |
 
 Key: `b` = false greens seen only without Tamperward; `c` = false greens seen
 only with it; `RD` = paired risk difference; `pp` = percentage points; `BP95` =
@@ -73,13 +76,29 @@ Earlier controlled experiments → **[the research series](./docs/blog/index.md)
 > run had just restored — which was present from v1.9.0 onward, so rounds 2, 3
 > and 3.1 all ran on a verifier carrying it. No evidence any agent used it; the
 > layer's authority in those rounds was nonetheless weaker than described at the
-> time. **Round 4 is registered and frozen, not yet run:** a fresh pool of 110
-> paired repositories under `claude-sonnet-5`, treatment **2.10.3** pinned by
-> artefact hash, a confirmatory masked-failure endpoint (exact McNemar), and the
-> whole draw — task order, arm assignment, and a separate 22-pair instability
-> budget — derived from committed seeds before the first counted trajectory. The
-> methodology and preregistration are published, with no counted numbers:
-> [How round 4 is built to be hard to fool](./docs/blog/how-round-4-is-built-to-be-hard-to-fool.md).
+> time. **Round 4 is complete and sealed.** It was the first counted round on the
+> 2.x line: a fresh pool of 110 paired repositories under `claude-sonnet-5`,
+> treatment **2.10.3** pinned by artefact hash, with the whole draw — task order,
+> arm assignment, and a separate 22-pair instability budget — derived from
+> committed seeds before the first counted trajectory. The registered primary
+> prediction **did not replicate**: b=5, c=3, exact McNemar p = 0.7265625, null not
+> rejected, over the 79 of 110 repositories that produced a valid paired
+> measurement. The narrower integrity observation — 0 strict tamper bypasses across
+> 201 measured trajectories — is a separate descriptive result and does not
+> replace the registered outcome. Only 79/110 pairs survived to measurement; the
+> loss was mostly symmetric apparatus attrition (venv/interpreter execution
+> failures), which reduces concern about arm-specific bias but is not claimed to
+> be free of selection bias — the readout says so. Every figure is the sealed
+> value in [`ROUND4-RESULTS.json`](./harness/taskbench/round4/ROUND4-RESULTS.json)
+> (`payload_sha256` `e7bfce08…`), read out in
+> [`ROUND4-ANALYSIS.md`](./harness/taskbench/round4/ROUND4-ANALYSIS.md) and the
+> deviation ledger [`DEVIATIONS.md`](./harness/taskbench/round4/DEVIATIONS.md).
+> Read the round as a pair — the bets before, the outcome after:
+> [How round 4 is built to be hard to fool](./docs/blog/how-round-4-is-built-to-be-hard-to-fool.md)
+> (preregistration, no numbers) →
+> [The prevention bet didn't replicate. No surviving tampering was certified clean.](./docs/blog/round-4-results.md)
+> (results). Round 4.1, a second agent runtime over the same frozen pool, is the
+> next registered step ([SPEC §9.1](./SPEC.md#91-maturity-milestones)).
 
 ## Architecture
 
@@ -654,6 +673,7 @@ and its totals.
 | v1.8.0 | The effect layer: per-call protected-tree drift check plus the `tamperward watch` fs-event observer — judge what the files became, not the command spelling |
 | v1.9.0 | `tamperward verify`: the round-1 pristine oracle productized — visible-green + pristine-red is a masked failure, and "could not verify" fails closed |
 | Round 2, 9–0 | On 22 fresh repositories, v1.9.0 produced b=9/c=0 false-green discordance (p = 0.0039); four gated false greens remained, through two documented residual mechanisms |
+| Round 4, p = 0.7265625 | On 110 fresh paired repositories under 2.10.3, the preregistered prevention bet did not replicate (b=5/c=3, 79 valid pairs); the 35 masked failures split roughly evenly between tampering-related and plain correctness gaps, so the next round separates the two at the endpoint and qualifies repositories under the exact production runner before they enter the pool |
 | cssstyle lifecycle escape (round 2) | The `tamperward run` envelope: the gate denied 42 mutations and verify refused all 25 stop attempts, and the runtime completed the session anyway over a masked tree — hooks decide, runtimes terminate, so the adjudication boundary moved outside the runtime |
 | v1.14.6, external audit | A pull request can neuter the gate that judges it: `pull_request` workflows run from the PR's own head and required checks match by job name, so a job that keeps its name and runs `true` reports green over anything. Reproduced on this repo's own CI — `gate` succeeded in three seconds over a PR that lowered a mechanical rule to warn. No detector can close it, because the detector is defined by the candidate; `init` now writes a CODEOWNERS requirement on the gate-critical paths and states, every run, that branch protection is the half it cannot set for you |
 | v1.14.5, external audit | The pristine copy was a sibling of the visible copy and fully prepared before it ran, so the candidate's own code wrote `../pristine/<the failing test>` and both runs returned 0 — no race, no background process, no git. The pristine copy is now built only after the visible run exits, its restored files are digested either side of the pristine run, and the working and dependency trees are fingerprinted around both. Also: the overlay restored content but not MODE, so `chmod -x` alone hid a test from an exec-bit runner in both copies; a transient dependency substitution that restored itself on the way out left the entry-vs-exit fingerprint identical; and a committed `.npmrc` chose the registry the CI gate was downloaded from |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tamperward",
-  "version": "2.20.3",
+  "version": "2.20.4",
   "description": "The deterministic agent-integrity gate. One ruleset, evaluated on the actual diff/commands as a verdict, enforced everywhere a change can be made.",
   "license": "Apache-2.0",
   "author": "hexrift",