codegraph-brain 0.15.0__tar.gz → 0.16.0__tar.gz

codegraph_brain-0.16.0/.github/test-count-baseline

@@ -0,0 +1 @@
+1957

{codegraph_brain-0.15.0 → codegraph_brain-0.16.0}/.github/workflows/ci.yml

@@ -67,6 +67,25 @@ jobs:
           uv run --frozen pytest --cov=cgis --cov=scripts
           --cov-report=term-missing --cov-report=xml
 
+      - name: Test-count floor
+        # Deliberately a workflow step and not a test (#405). On 2026-08-17 a
+        # stale-tree push deleted 24 test files and 5 modules under src/cgis,
+        # and the step above passed: the tests that would have failed went with
+        # them. Every floor the repository had lived inside a test file, so the
+        # push that tripped them also removed them.
+        #
+        # The floor is read from the *base branch*, not from this checkout — a
+        # stale tree carries a stale baseline, and comparing the branch against
+        # its own copy passes the exact incident above. `fetch-depth: 0` on the
+        # checkout is what makes origin/<base> readable here.
+        #
+        # `github.base_ref` is set on pull_request only; on a push to the trunk
+        # it is empty and ref_name is the branch being pushed, which is the
+        # right comparison there too.
+        run: >
+          uv run --frozen python scripts/check_test_count.py
+          --base-ref "origin/${{ github.base_ref || github.ref_name }}"
+
       - name: Project version for Sonar
         id: version
         # Without this every analysis records the version as the literal string

codegraph_brain-0.16.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.16.0"
+}

{codegraph_brain-0.15.0 → codegraph_brain-0.16.0}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [0.16.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.15.0...codegraph-brain-v0.16.0) (2026-08-17)
+
+
+### Features
+
+* **bench:** a control arm, so an evidence result can be attributed ([#401](https://github.com/zaebee/codegraph-brain/issues/401)) ([#406](https://github.com/zaebee/codegraph-brain/issues/406)) ([241dbae](https://github.com/zaebee/codegraph-brain/commit/241dbae334be897cb6bf3d151d097b64ab5a56e7))
+* **bench:** both skeptics over the same frozen passes, and the evidence bench replay dropped ([#246](https://github.com/zaebee/codegraph-brain/issues/246)) ([#411](https://github.com/zaebee/codegraph-brain/issues/411)) ([2c1d50b](https://github.com/zaebee/codegraph-brain/commit/2c1d50b2abb18b0c0bad1a9b46f71531eebba16c))
+* **ci:** a floor under the size of the test suite itself ([#405](https://github.com/zaebee/codegraph-brain/issues/405)) ([#409](https://github.com/zaebee/codegraph-brain/issues/409)) ([0fbcbac](https://github.com/zaebee/codegraph-brain/commit/0fbcbac573651591750f017e3793b692c05377ce))
+* **guardian:** tell the skeptic when no checker ran, and which verdict that takes ([#407](https://github.com/zaebee/codegraph-brain/issues/407)) ([#410](https://github.com/zaebee/codegraph-brain/issues/410)) ([2470ddb](https://github.com/zaebee/codegraph-brain/commit/2470ddb45a29f19e2abf90d2f6ce5b5fadad09f3))
+
 ## [0.15.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.14.1...codegraph-brain-v0.15.0) (2026-08-17)
 
 

{codegraph_brain-0.15.0 → codegraph_brain-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: codegraph-brain
-Version: 0.15.0
+Version: 0.16.0
 Summary: Semantic code graph for AI agents — deterministic FQN resolution, impact analysis and architectural drift gates, exposed over MCP.
 Project-URL: Homepage, https://github.com/zaebee/codegraph-brain
 Project-URL: Repository, https://github.com/zaebee/codegraph-brain

codegraph_brain-0.16.0/benchmarks/guardian/experiments/246-cross-vendor/README.md

@@ -0,0 +1,71 @@
+# Cross-vendor skeptic: still blocked, and now on measured limits (#246)
+
+Two attempts at the third arm, both stopped by the validity gate before any
+number was reported. The gate is the deliverable here; the comparison is not.
+
+## Why a gate at all
+
+`judge_finding` contains provider errors per finding and returns None, so a
+spent quota, an HTTP 429, a 402 and unparseable JSON all arrive as **the same
+row shape as a skeptic that refused to refute anything** — which is exactly what
+#246 predicts for a same-vendor skeptic. A broken arm would confirm the
+hypothesis. `MAX_UNRULED_RATE` refuses to print a comparison above 5% unruled,
+keeps the rows, and exits 2.
+
+It fired on both attempts. Neither produced a number that could be read.
+
+## Attempt 1 — free tier. The quota is 50 requests per day.
+
+`nvidia/nemotron-3-super-120b-a12b:free`, the whole corpus: **96 of 135 findings
+unruled**, and the pattern is a cutoff in time rather than a property of the
+input.
+
+| PR | evidence | unruled |
+|---|---|---:|
+| 122 (first) | no | **0 / 29** |
+| 140 | yes | 29 / 39 |
+| 141, 142, 143, 144 | — | **100%** |
+
+39 answered, then silence. With the day's earlier probes (4 candidate calls, 6
+in a smoke run) that is 49 successful requests before the wall — the documented
+free allowance is 50/day, and this is that number arrived at from the data.
+
+Rows: `free-nemotron-quota-cutoff.jsonl`.
+
+## Attempt 2 — paid. Two separate faults, one after the other.
+
+`qwen/qwen3.7-plus`, chosen because capability has to match: comparing
+`gemini-2.5-flash` against a small free model measures strong-versus-weak and
+calls it same-versus-cross.
+
+**First fault — thinking.** 118 of 135 unruled. `qwen3.7-plus` puts its chain of
+thought in a separate `reasoning` field and fills `content` only at the end, so
+on the larger prompts it spent the entire 8,000-token budget thinking and
+returned `content: null`. Raising the budget would have fixed the symptom and
+broken the experiment: the arm it is compared against is not doing extended
+thinking, so the two arms would differ in a dimension nobody chose. The provider
+now sends `reasoning: {"enabled": …}` explicitly in both directions, off by
+default, so a run can state what it did.
+
+**Second fault — credits.** The re-run failed on HTTP **402**: the account has
+`total_credits: $0` against `total_usage: $0.159`. The trial allowance is spent;
+paid models are unavailable until it is topped up.
+
+The second fault took half an hour to identify because the warning read only
+"Skeptic judgement failed; finding stays unruled" — 118 identical lines, no
+type, no message. It now carries both. A quota, a truncated reasoning model and
+a 402 have three different remedies and all three land in that one `except`.
+
+## To unblock
+
+1. **Top up OpenRouter.** The measured cost of the run with reasoning off is
+   ~$0.15: 135 findings × ~2,700 prompt tokens at \$0.32/M, plus short
+   completions at \$1.28/M. A dollar covers it several times over.
+2. **Free tier across three days.** The input is a frozen recording, so the run
+   is deterministic and can be split — 50 requests a day, 135 needed. Fragile,
+   and it needs resume support the tool does not have.
+3. A local ollama, which this issue already names as the no-cost option.
+
+Everything else is in place: the corpus with both finder orientations, the
+recordings, per-arm models, the scoring, and the gate that stopped two wrong
+answers from being published.

codegraph_brain-0.16.0/benchmarks/guardian/experiments/246-cross-vendor/free-nemotron-quota-cutoff.jsonl

@@ -0,0 +1,74 @@
+{"timestamp": "2026-08-17T13:17:43.517549+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-31.032903+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.18181818181818182, "precision": 0.6666666666666666, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:05.186110+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-31.032903+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 0.6666666666666666, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:15.602819+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-59.496508+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:24.136220+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-59.496508+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:36.000053+00:00", "pr": 122, "pass": "122@2026-06-10T20-23-51.775172+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:50.719370+00:00", "pr": 122, "pass": "122@2026-06-10T20-23-51.775172+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 1, "confirmed": 1, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:19:01.976110+00:00", "pr": 122, "pass": "122@2026-06-10T20-32-35.483908+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:19:23.184478+00:00", "pr": 122, "pass": "122@2026-06-10T20-32-35.483908+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 2, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:19:36.855990+00:00", "pr": 122, "pass": "122@2026-06-10T20-33-10.513615+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:20:22.191259+00:00", "pr": 122, "pass": "122@2026-06-10T20-33-10.513615+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:20:35.820094+00:00", "pr": 122, "pass": "122@2026-06-10T21-58-22.314534+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:21:08.068793+00:00", "pr": 122, "pass": "122@2026-06-10T21-58-22.314534+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:21:15.458172+00:00", "pr": 122, "pass": "122@2026-06-11T00-11-53.683357+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:21.049843+00:00", "pr": 122, "pass": "122@2026-06-11T00-11-53.683357+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:29.468118+00:00", "pr": 122, "pass": "122@2026-06-11T00-12-40.927704+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:35.271913+00:00", "pr": 122, "pass": "122@2026-06-11T00-12-40.927704+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:46.009233+00:00", "pr": 122, "pass": "122@2026-06-11T00-14-36.809184+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:58.953594+00:00", "pr": 122, "pass": "122@2026-06-11T00-14-36.809184+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:06.863138+00:00", "pr": 122, "pass": "122@2026-06-11T00-32-51.894263+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:18.621710+00:00", "pr": 122, "pass": "122@2026-06-11T00-32-51.894263+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:34.978570+00:00", "pr": 122, "pass": "122@2026-06-11T00-36-53.749615+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:23:29.360852+00:00", "pr": 122, "pass": "122@2026-06-11T00-36-53.749615+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:35.269160+00:00", "pr": 122, "pass": "122@2026-06-11T00-42-55.162925+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:43.563281+00:00", "pr": 122, "pass": "122@2026-06-11T00-42-55.162925+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:53.848653+00:00", "pr": 122, "pass": "122@2026-06-11T00-48-45.262981+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 1, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:24:05.969279+00:00", "pr": 122, "pass": "122@2026-06-11T00-48-45.262981+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:24:25.419473+00:00", "pr": 140, "pass": "140@2026-06-10T20-24-40.504167+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:25:25.250544+00:00", "pr": 140, "pass": "140@2026-06-10T20-24-40.504167+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:25:47.211803+00:00", "pr": 140, "pass": "140@2026-06-10T20-25-19.261202+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 5, "refuted": 2, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:26:24.777165+00:00", "pr": 140, "pass": "140@2026-06-10T20-25-19.261202+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 5, "refuted": 0, "uncertain": 3, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 5, "killed_gt": []}
+{"timestamp": "2026-08-17T13:26:33.938777+00:00", "pr": 140, "pass": "140@2026-06-10T20-26-01.543236+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:08.103278+00:00", "pr": 140, "pass": "140@2026-06-10T20-26-01.543236+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:17.614733+00:00", "pr": 140, "pass": "140@2026-06-11T00-16-26.605409+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:17.769243+00:00", "pr": 140, "pass": "140@2026-06-11T00-16-26.605409+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:27.876729+00:00", "pr": 140, "pass": "140@2026-06-11T00-17-25.717492+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.06666666666666667, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:28.068613+00:00", "pr": 140, "pass": "140@2026-06-11T00-17-25.717492+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.06666666666666667, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:38.870523+00:00", "pr": 140, "pass": "140@2026-06-11T00-33-44.053747+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:39.001629+00:00", "pr": 140, "pass": "140@2026-06-11T00-33-44.053747+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:45.424281+00:00", "pr": 140, "pass": "140@2026-06-11T00-37-50.526658+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:45.544269+00:00", "pr": 140, "pass": "140@2026-06-11T00-37-50.526658+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:53.801567+00:00", "pr": 140, "pass": "140@2026-06-11T00-50-08.134629+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:53.921093+00:00", "pr": 140, "pass": "140@2026-06-11T00-50-08.134629+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:06.369511+00:00", "pr": 140, "pass": "140@2026-07-29T17-29-19.861365+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 22, "refuted": 10, "uncertain": 2, "confirmed": 10, "unruled": 0, "recall": 0.26666666666666666, "precision": 0.3333333333333333, "noise": 8, "killed_gt": ["count-routers-on-squared", "weights-keyerror-custom-yaml"]}
+{"timestamp": "2026-08-17T13:29:07.917533+00:00", "pr": 140, "pass": "140@2026-07-29T17-29-19.861365+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 22, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 22, "recall": 0.4, "precision": 0.2727272727272727, "noise": 16, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:21.415626+00:00", "pr": 141, "pass": "141@2026-06-10T20-26-53.683221+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:21.527539+00:00", "pr": 141, "pass": "141@2026-06-10T20-26-53.683221+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:28.243544+00:00", "pr": 141, "pass": "141@2026-06-10T20-29-48.066882+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:28.360785+00:00", "pr": 141, "pass": "141@2026-06-10T20-29-48.066882+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:37.370316+00:00", "pr": 141, "pass": "141@2026-06-10T21-59-24.111065+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:37.495452+00:00", "pr": 141, "pass": "141@2026-06-10T21-59-24.111065+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:49.082379+00:00", "pr": 142, "pass": "142@2026-06-10T20-30-15.605690+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.5, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:49.243691+00:00", "pr": 142, "pass": "142@2026-06-10T20-30-15.605690+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.5, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:56.956875+00:00", "pr": 142, "pass": "142@2026-06-10T21-52-43.992582+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:57.130875+00:00", "pr": 142, "pass": "142@2026-06-10T21-52-43.992582+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 2, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:04.860920+00:00", "pr": 142, "pass": "142@2026-06-10T21-53-15.240002+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:05.012683+00:00", "pr": 142, "pass": "142@2026-06-10T21-53-15.240002+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:14.805023+00:00", "pr": 142, "pass": "142@2026-06-11T00-45-11.791674+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 3, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:14.969320+00:00", "pr": 142, "pass": "142@2026-06-11T00-45-11.791674+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:23.741156+00:00", "pr": 142, "pass": "142@2026-06-11T00-50-52.772949+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 2, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:23.921669+00:00", "pr": 142, "pass": "142@2026-06-11T00-50-52.772949+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:43.638312+00:00", "pr": 143, "pass": "143@2026-06-11T00-46-29.982484+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:43.909808+00:00", "pr": 143, "pass": "143@2026-06-11T00-46-29.982484+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:21.206192+00:00", "pr": 143, "pass": "143@2026-07-29T17-15-00.138863+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 28, "refuted": 8, "uncertain": 1, "confirmed": 19, "unruled": 0, "recall": 1.0, "precision": 0.3, "noise": 14, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:22.809114+00:00", "pr": 143, "pass": "143@2026-07-29T17-15-00.138863+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 28, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 28, "recall": 1.0, "precision": 0.21428571428571427, "noise": 22, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:44.357216+00:00", "pr": 144, "pass": "144@2026-06-10T20-32-21.181637+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:44.503817+00:00", "pr": 144, "pass": "144@2026-06-10T20-32-21.181637+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 2, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:58.741622+00:00", "pr": 144, "pass": "144@2026-06-11T00-25-01.536057+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:58.913488+00:00", "pr": 144, "pass": "144@2026-06-11T00-25-01.536057+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:06.869381+00:00", "pr": 144, "pass": "144@2026-06-11T00-47-31.514757+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:06.990972+00:00", "pr": 144, "pass": "144@2026-06-11T00-47-31.514757+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:14.521992+00:00", "pr": 144, "pass": "144@2026-06-11T00-52-31.320750+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:14.623054+00:00", "pr": 144, "pass": "144@2026-06-11T00-52-31.320750+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:34:10.907748+00:00", "pr": 144, "pass": "144@2026-07-29T17-28-33.271744+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 20, "refuted": 7, "uncertain": 0, "confirmed": 13, "unruled": 0, "recall": 0.6, "precision": 0.23076923076923078, "noise": 5, "killed_gt": []}
+{"timestamp": "2026-08-17T13:34:12.013311+00:00", "pr": 144, "pass": "144@2026-07-29T17-28-33.271744+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 20, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 20, "recall": 0.6, "precision": 0.15, "noise": 8, "killed_gt": []}

codegraph_brain-0.16.0/benchmarks/guardian/experiments/401-evidence/README.md

@@ -0,0 +1,140 @@
+# Does checker output change the skeptic's verdicts? (#401)
+
+#402 gave the skeptic the repository's `mypy --strict` and `ruff check` output. Whether
+that changes anything was unmeasured, and `GUARDIAN_EVIDENCE=1` was switched on without a
+number behind it. This is the number.
+
+## Design
+
+One review, three arms, paired at the level of the individual finding.
+
+| arm | what it is |
+|---|---|
+| **baseline** | the verdicts the recorded review actually produced, no evidence |
+| **control** | the same 24 findings re-judged, still no evidence |
+| **treatment** | the same 24 findings re-judged with evidence, collected at the reviewed commit |
+
+The finder is never called. The subject is `guardian-finder-399`, the review that motivated
+the RFC: 24 findings, of which the RFC classified exactly one as substantive.
+
+**The control arm is not optional rigour.** `gemini.py` sends no temperature and
+`guardian.yml` sets none, so the skeptic runs at the provider default: the recorded verdicts
+are one draw, not a fixed point. Baseline against treatment therefore measures the effect
+*plus* the resampling noise, and only a no-evidence replay separates them. This is the same
+confound the replay arm removes at the finder, reappearing one level down.
+
+Evidence is collected in a worktree at `05af753d` — the commit `refs/pull/399/head` pointed
+at when the review ran. **Not** the artifact's `workflow_run.head_sha`: `guardian.yml` is
+triggered by `issue_comment`, so that field is the tip of the trunk (`fd4e53a2`, a release
+commit) and the reviewed files do not exist in it.
+
+## Results
+
+Baseline: **19 confirmed / 5 refuted**.
+
+| arm | conf→conf | conf→ref | conf→unc | ref→conf | ref→ref | moved | cites a checker |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| control, draw 1 | 17 | 2 | 0 | 2 | 3 | 4 | 6/24 |
+| control, draw 2 | 18 | 1 | 0 | 0 | 5 | 1 | 6/24 |
+| treatment, draw 1 | 11 | 8 | 0 | 0 | 5 | 8 | 7/24 |
+| treatment, draw 2 | 11 | 7 | 1 | 1 | 4 | 9 | 8/24 |
+
+Draw 2 of each arm carries the per-finding rows (`pr-399-control-rows.jsonl`,
+`pr-399-treatment-rows.jsonl`); the first draws predate that output and survive only as the
+aggregates above.
+
+### The mechanism works exactly where it was aimed
+
+Six findings claimed a mypy rule the repository does not have. **All six were refuted, and
+all six cite the checker output.** The rationale on finding 0 is the RFC's hypothesis
+verbatim:
+
+> The project's mypy --strict checker, run as part of the review process, reports 'Success:
+> no issues found', directly contradicting the claim that the use of `Any` is prohibited in
+> strict mypy mode for this code.
+
+That answers open question 1. The failure was **capability, not disposition**: shown the
+output, the skeptic withdraws the claim; not shown it, it confirms.
+
+### The one true finding was withdrawn
+
+Finding 4 — `git diff` can fail in a way other than the promised `MissingFixtureError`, the
+finding that produced the actual fix in #399 — was refuted, **without citing any checker**:
+
+> the code explicitly checks the validity of `base` and `head` SHAs using `git rev-parse`
+> before the `git diff` call … thus the specific failure mode is already handled
+
+which reproduces the half of the argument that `diff_for`'s own comment rejects: *"unlikely"
+is not the same as "cannot"*.
+
+| | baseline | treatment |
+|---|---:|---:|
+| false findings confirmed | 18 | 12 |
+| true findings confirmed | 1 | **0** |
+
+### What the control arm bought, and how it reversed the first reading
+
+The aggregate alone suggested the loss of finding 4 was noise: no checker was cited on it,
+and control draw 1 produced two `confirmed → refuted` flips of its own. **The per-finding
+rows say otherwise.** In the control draw that carries rows, finding 4 was *confirmed*, and
+the rationale is correct:
+
+> The `subprocess.run` call for `git diff` includes `check=True`, but the
+> `CalledProcessError` it raises on command failure is not explicitly caught, leading to a
+> different error type than `MissingFixtureError` …
+
+That is the defect, stated accurately, by the same model that withdrew it once evidence was
+in the prompt. The single control flip was finding 6, not finding 4.
+
+So the loss is **on the evidence arm's side of the ledger**, and the earlier "not
+attributable" reading — published in #406 and in the #401 comment before these rows existed —
+was wrong. It rested on an aggregate, which is exactly the shape of number that cannot
+answer *which*.
+
+The two arms separate cleanly per finding:
+
+| class | control | treatment |
+|---|---|---|
+| 6 false mypy claims | all confirmed | all refuted, all citing the checker |
+| the 1 true finding | **confirmed, correctly reasoned** | **refuted**, citing no checker |
+
+The mechanism on the false claims is visible in the control's own words: it confirmed
+finding 0 on a conditional it could not check — *"**if** `mypy --strict` is configured to
+disallow `Any` explicitly, it would be flagged"*. Evidence turns that hypothetical into a
+reading, which is precisely the RFC's design intent.
+
+Why the true finding falls is not explained by the same mechanism, because its refutation
+cites nothing. The unattributed moves are 3 in treatment against 1 in control, which is
+consistent with the evidence section shifting the skeptic toward refutation generally rather
+than only where the output bears — but at these counts that is a hypothesis, not a result.
+
+Two draws of the same arm differ in both arms (control 4 vs 1 flips, treatment 8 vs 9), so
+single-draw asymmetries are not results.
+
+## What this does not establish
+
+- **The recall denominator is one.** One review, one substantive finding. "Recall went to
+  zero" is literally true and statistically nearly weightless.
+- **Attribution by rationale text is a weak proxy.** Citations barely moved between arms
+  (6/24 control vs 7–8/24 treatment) while verdicts moved a great deal — and the rows show
+  why: the control cites a checker on the *same six findings*, to confirm them.
+- **Only one draw per arm has rows.** Whether finding 4 survives the control reliably or was
+  lucky in this draw is unmeasured; control draw 1 had two unidentified `confirmed → refuted`
+  flips.
+- **Six of 24 control rationales cite mypy or ruff although no checker output was supplied**
+  (#407). Not merely fabricated: fabricated *in support of the wrong verdict*. The same six
+  findings carry a checker citation in both arms, confirmed in one and refuted in the other,
+  and only one of the two had a checker.
+
+## Reproducing
+
+```bash
+scripts/guardian_replay_skeptic.py --recording pr-399-recording.json \
+    --control --out control-rows.jsonl
+scripts/guardian_replay_skeptic.py --recording pr-399-recording.json \
+    --at 05af753d8895eab00609e7e58d89aa72ebd1777d --out treatment-rows.jsonl
+```
+
+Requires `GUARDIAN_SKEPTIC=gemini`, `GUARDIAN_SKEPTIC_MODEL=gemini-2.5-flash` and the
+matching key. 24 skeptic calls per arm, no finder calls. The recording is committed here
+because the GitHub artifact it came from expires after seven days.

codegraph_brain-0.16.0/benchmarks/guardian/experiments/401-evidence/pr-399-control-rows.jsonl

@@ -0,0 +1,24 @@
+{"index": 0, "file": "scripts/recordings_from_corpus.py", "line": 38, "title": "Use of `Any` in type annotations violates strict mypy mode", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The use of `Any` does indeed bypass strict type checking in mypy, and while common for dynamic data structures (like those parsed from JSON/YAML), it generally indicates a potential for more precise typing or schema definition (e.g., Pydantic models). This is a plausible area for improvement, and if `mypy --strict` is configured to disallow `Any` explicitly, it would be flagged."}
+{"index": 1, "file": "scripts/recordings_from_corpus.py", "line": 41, "title": "Unsafe YAML loading without `safe_load` or `Loader` specification", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The quoted code on line 100 (not 41 as stated) correctly uses `yaml.safe_load`. However, the `import yaml` statement does allow for `yaml.load` to be used elsewhere in the file, which is an unsafe practice without specifying a `Loader`. This is a plausible concern for future maintainability and security, as a developer might accidentally introduce an unsafe load later. The claim is plausible and cannot be disproven."}
+{"index": 2, "file": "scripts/recordings_from_corpus.py", "line": 50, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code accurately shows `Any` used in the type annotation, which strict mypy typically flags, aligning with the project's policy to score such issues low when covered by tooling."}
+{"index": 3, "file": "scripts/recordings_from_corpus.py", "line": 56, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code is present. While `mypy --strict` does not inherently disallow explicit `Any` without `disallow_any_explicit`, using `dict[str, Any]` weakens type safety for the `row` parameter, making it a legitimate type-hinting concern that is not 'already covered' by the standard `mypy --strict` checks."}
+{"index": 4, "file": "scripts/recordings_from_corpus.py", "line": 62, "title": "Empty diff raises `MissingFixtureError` but may not be the only failure mode", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `subprocess.run` call for `git diff` includes `check=True`, but the `CalledProcessError` it raises on command failure is not explicitly caught, leading to a different error type than `MissingFixtureError` which is used for other unresolvable diff scenarios."}
+{"index": 5, "file": "scripts/recordings_from_corpus.py", "line": 80, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code on line 80 uses `dict[str, Any]` for the `row` parameter, which is correctly identified as a violation by `mypy --disallow-any-type` (part of `--strict`)."}
+{"index": 6, "file": "scripts/recordings_from_corpus.py", "line": 89, "title": "Potential type mismatch in `Finding.model_validate`", "was": "confirmed", "now": "refuted", "cites_a_checker": false, "rationale": "The `Finding.model_validate` method is precisely designed to enforce the schema and will raise a `ValidationError` for type mismatches or unexpected keys (if configured with `extra='forbid'`), or silently drop extra keys (if configured with `extra='ignore'`). The dictionary comprehension correctly strips known annotations before validation, so the code already handles the claimed problem appropriately through Pydantic's intended mechanisms. A `ValidationError` indicates a problem with the input data, not with the validation call itself. Using `model_construct` would bypass essential validation."}
+{"index": 7, "file": "scripts/recordings_from_corpus.py", "line": 95, "title": "No validation that `results` file exists or is readable", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The `results.read_text()` call will raise a `FileNotFoundError` or other `IOError` if the file does not exist or is unreadable, preventing the function from silently proceeding without error in those specific cases."}
+{"index": 8, "file": "scripts/recordings_from_corpus.py", "line": 100, "title": "No handling for case where no frozen passes are found", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The script correctly handles the scenario where no frozen passes are found by writing zero recordings. The `main` function explicitly reports \"0 recordings written\", which clearly communicates the outcome to the caller, making the behavior expected and not an error. A separate warning is not required for correctness or clarity."}
+{"index": 9, "file": "scripts/recordings_from_corpus.py", "line": 102, "title": "Race condition in directory creation", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The `exist_ok=True` argument to `Path.mkdir()` explicitly handles the scenario where the directory already exists, preventing `FileExistsError` and making the operation safe for concurrent calls without a race condition."}
+{"index": 10, "file": "scripts/recordings_from_corpus.py", "line": 110, "title": "No validation that `save_finder_recording` succeeds", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `build` function does not handle potential failures of `save_finder_recording` (e.g., due to permission issues or disk full), which would either cause the script to crash, or, if `save_finder_recording` fails silently, lead to incorrectly reporting a recording as written when it was not."}
+{"index": 11, "file": "scripts/recordings_from_corpus.py", "line": 117, "title": "No error handling for `build` failures", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `main` function does not wrap the call to `build` in a try-except block, confirming that exceptions from `build` will propagate and exit the script with a traceback instead of a more user-friendly error message. This impacts script robustness for the end-user."}
+{"index": 12, "file": "tests/unit/test_recordings_from_corpus.py", "line": 12, "title": "Modification of `sys.path` at runtime violates type safety", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code accurately reflects the diff. Modifying sys.path at runtime is a known pattern that can lead to less predictable import resolution and complicates static analysis tools like MyPy, which is plausibly what the reviewer meant by 'not type-safe.' It's not a critical bug but a maintainability concern."}
+{"index": 13, "file": "tests/unit/test_recordings_from_corpus.py", "line": 15, "title": "Import from a script file may fail in some environments", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The claim that the import may fail due to the file not being in `sys.path` is refuted, as line 13 explicitly adds the containing directory to `sys.path`. The claim regarding environments not supporting direct `.py` imports is not generally applicable to functional Python environments. While importing from a script file via `sys.path` manipulation can be an architectural concern, the specific failure conditions stated in the claim are either handled or not plausible."}
+{"index": 14, "file": "tests/unit/test_recordings_from_corpus.py", "line": 20, "title": "Hardcoded path traversal may break in some environments", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The code uses a chain of `.parent` calls to infer the repository root, which is indeed a hardcoded path traversal dependent on the file's exact location (`tests/unit/`), making the claim accurate and a minor robustness concern."}
+{"index": 15, "file": "tests/unit/test_recordings_from_corpus.py", "line": 28, "title": "Function return type uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code uses `Any` in the return type annotation as claimed. I cannot refute the assertion that this violates a strict mypy rule; if true, it would be caught by mandatory project tooling, leading to a low impact score."}
+{"index": 16, "file": "tests/unit/test_recordings_from_corpus.py", "line": 30, "title": "No validation that `RESULTS` file exists or is readable", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `_rows` function directly attempts to read `RESULTS.read_text()` without checking for the file's existence, which will indeed raise a `FileNotFoundError` if `results.jsonl` is absent, causing the tests to fail. The claim accurately describes this concrete failure path in the test utility."}
+{"index": 17, "file": "tests/unit/test_recordings_from_corpus.py", "line": 35, "title": "Return type uses `Path` but may include non-Path values", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The original reviewer's analysis concludes, 'No action needed; the type is correct. This is a false positive.' This explicitly refutes the claim of a defect, as the type annotation `dict[str, Path]` accurately reflects the dictionary values being `Path` objects."}
+{"index": 18, "file": "tests/unit/test_recordings_from_corpus.py", "line": 50, "title": "Test assumes existence of a judged row in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `next()` call without a default value will raise a `StopIteration` error if the `_rows()` generator yields no matching elements, making the test brittle to changes in the `results.jsonl` corpus data."}
+{"index": 19, "file": "tests/unit/test_recordings_from_corpus.py", "line": 57, "title": "Test assumes existence of an unscored row in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The use of `next()` without a default value or preceding check means the test will indeed raise `StopIteration` if the external corpus `results.jsonl` happens to lack a row matching the specified criteria, making the test brittle."}
+{"index": 20, "file": "tests/unit/test_recordings_from_corpus.py", "line": 64, "title": "Test uses a hardcoded PR number (9999) that may not be meaningful", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The test correctly expects a MissingFixtureError for PR 9999, which relies on PR 9999 not having an associated ground-truth fixture. If PR 9999 were to be created and a fixture added in the future, the test would fail, making it fragile."}
+{"index": 21, "file": "tests/unit/test_recordings_from_corpus.py", "line": 75, "title": "Test assumes at least 72 frozen passes exist in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The assertion explicitly hardcodes a minimum of 72 frozen passes. If the corpus evolves to contain fewer than 72, the test will fail, regardless of whether the `build` function and related logic are actually correct for the current corpus size, making the test brittle."}
+{"index": 22, "file": "tests/unit/test_recordings_from_corpus.py", "line": 100, "title": "Test assumes existence of `ambiguous_hits` in some rows", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The test explicitly asserts that the rows requiring the pre-#345 precision rule are exactly those carrying `ambiguous_hits`, as stated in the claim and the test's accompanying comments. This is an intentional check to ensure consistency between the corpus data and its documented scoring behavior, meaning a failure would correctly flag an issue with the benchmark data, not necessarily a defect in the test's logic. However, it indicates a tight coupling of the test to a specific data property, which implies a potential maintenance burden if this property of the corpus changes, even for legitimate reasons."}
+{"index": 23, "file": "tests/unit/test_recordings_from_corpus.py", "line": 145, "title": "Floating-point comparison may be too strict for some use cases", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The claim is plausible because an absolute tolerance of 1e-9, while strict, can lead to false negatives in floating-point comparisons if calculations accumulate tiny errors or if the reference values themselves have slightly different precision than the re-calculated ones. Using math.isclose with both relative and absolute tolerances is generally a more robust approach in such test scenarios to avoid brittleness and potential flakiness."}

codegraph_brain-0.16.0/benchmarks/guardian/experiments/401-evidence/pr-399-recording.json

@@ -0,0 +1 @@
+{"result":{"findings":[{"file":"scripts/recordings_from_corpus.py","line":38,"severity":"major","category":"types","title":"Use of `Any` in type annotations violates strict mypy mode","evidence":"from typing import Any","problem":"The use of `Any` is prohibited in strict mypy mode unless technically unavoidable, and no such justification is provided here.","fix":"Replace `Any` with precise types (e.g., `dict[str, Any]` → `dict[str, object]` or a Pydantic model).","confidence":95,"anchor":"from typing import Any","verdict":"confirmed","skeptic_note":"The use of `Any` for the `row` parameter, which is loaded from arbitrary JSON, is indeed a place where stricter type annotations (e.g., using `TypedDict` or a Pydantic model to define the expected structure of the JSON rows) would be required to fully comply with a very strict mypy mode. While `dict[str, Any]` is common for JSON parsing, it circumvents detailed static analysis for the structure of `row` (e.g., ensuring `row['pr']` is an int). This is a valid area for improvement for type robustness in a new file, though it does not introduce a runtime bug.","impact_score":3},{"file":"scripts/recordings_from_corpus.py","line":41,"severity":"minor","category":"contract","title":"Unsafe YAML loading without `safe_load` or `Loader` specification","evidence":"yaml.safe_load(fixture.read_text(encoding=\"utf-8\"))","problem":"While `safe_load` is used here, the import statement does not enforce it. Future code might accidentally use `yaml.load` without a Loader, which is unsafe.","fix":"Consider aliasing `yaml.safe_load` to `yaml.load` at import time or adding a lint rule to enforce `safe_load`.","confidence":60,"anchor":"import yaml","verdict":"confirmed","skeptic_note":"The current code correctly uses `yaml.safe_load`. However, the claim that the general `import yaml` statement allows for future accidental use of the unsafe `yaml.load` without a specified Loader is a valid and plausible concern for security and maintainability, even if not an immediate bug in this diff.","impact_score":5},{"file":"scripts/recordings_from_corpus.py","line":50,"severity":"major","category":"types","title":"Function parameter uses `Any` in type annotation","evidence":"def is_frozen_pass(row: dict[str, Any]) -> bool:","problem":"The `row` parameter is typed as `dict[str, Any]`, which violates the strict mypy rule against `Any`.","fix":"Replace `Any` with a precise type (e.g., `dict[str, object]` or a TypedDict).","confidence":95,"anchor":"def is_frozen_pass(row: dict[str, Any]) -> bool:","verdict":"confirmed","skeptic_note":"The quoted code accurately uses `dict[str, Any]`, which can indeed violate strict mypy rules, and such an issue would likely be caught by the project's mandated `mypy --strict` checks.","impact_score":1},{"file":"scripts/recordings_from_corpus.py","line":56,"severity":"major","category":"types","title":"Function parameter uses `Any` in type annotation","evidence":"def row_key(row: dict[str, Any]) -> str:","problem":"The `row` parameter is typed as `dict[str, Any]`, which violates the strict mypy rule against `Any`.","fix":"Replace `Any` with a precise type (e.g., `dict[str, object]` or a TypedDict).","confidence":95,"anchor":"def row_key(row: dict[str, Any]) -> str:","verdict":"confirmed","skeptic_note":"The quoted code is present in the diff. Using `Any` in type annotations typically violates strict mypy rules. Per instructions, if `mypy --strict` would catch it, the impact is low as tooling would prevent it.","impact_score":2},{"file":"scripts/recordings_from_corpus.py","line":62,"severity":"minor","category":"logic","title":"Empty diff raises `MissingFixtureError` but may not be the only failure mode","evidence":"if not result.stdout.strip():","problem":"The check for an empty diff is correct, but the error message assumes the only issue is an empty diff. However, `git diff` could also fail (e.g., if `base` or `head` are invalid), which would raise a `CalledProcessError` from `subprocess.run(..., check=True)`. This is not caught or handled.","fix":"Wrap the `subprocess.run` call in a try-except block to handle `CalledProcessError` and raise `MissingFixtureError` with a descriptive message.","confidence":80,"anchor":"def diff_for(pr: int, bench_dir: Path, repo_root: Path) -> str:","verdict":"confirmed","skeptic_note":"The `subprocess.run(check=True)` call for `git diff` can raise `CalledProcessError` if the git command fails, and this specific error is not caught or re-raised as `MissingFixtureError`, leading to an inconsistent error type if such a failure occurs.","impact_score":4},{"file":"scripts/recordings_from_corpus.py","line":80,"severity":"major","category":"types","title":"Function parameter uses `Any` in type annotation","evidence":"def recording_for(row: dict[str, Any], diff: str) -> tuple[ReviewResult, str]:","problem":"The `row` parameter is typed as `dict[str, Any]`, which violates the strict mypy rule against `Any`.","fix":"Replace `Any` with a precise type (e.g., `dict[str, object]` or a TypedDict).","confidence":95,"anchor":"def recording_for(row: dict[str, Any], diff: str) -> tuple[ReviewResult, str]:","verdict":"confirmed","skeptic_note":"The `row` parameter in `recording_for` is indeed typed as `dict[str, Any]`, which is often disallowed in strict mypy configurations, though such an issue would likely be caught by existing tooling in this repository.","impact_score":2},{"file":"scripts/recordings_from_corpus.py","line":89,"severity":"minor","category":"types","title":"Potential type mismatch in `Finding.model_validate`","evidence":"Finding.model_validate({k: v for k, v in f.items() if k not in _ANNOTATIONS})","problem":"The dictionary comprehension may produce a dict with keys/values that do not match `Finding`'s expected schema. If `f` contains unexpected keys or types, this could raise a `ValidationError` or silently drop data.","fix":"Validate the input dict against `Finding`'s schema before calling `model_validate`, or use `model_construct` if the data is trusted.","confidence":70,"anchor":"findings = [","verdict":"confirmed","skeptic_note":"The claim accurately describes potential behaviors of `Pydantic.model_validate`: a `ValidationError` will be raised for type mismatches or missing required fields, and extra fields may be silently dropped depending on the model's `extra` configuration (Pydantic v2 defaults to `ignore`). The code explicitly strips known extra annotations, indicating awareness, but the underlying mechanism of `model_validate` can still exhibit these behaviors for other discrepancies. While often desired for robust deserialization, an unhandled `ValidationError` could halt script execution, and silent dropping of truly needed data would be a problem (though less likely here given the explicit stripping of known extras). The proposed fixes are either redundant or could bypass necessary validation.","impact_score":4},{"file":"scripts/recordings_from_corpus.py","line":95,"severity":"minor","category":"logic","title":"No validation that `results` file exists or is readable","evidence":"rows = [","problem":"The function assumes `results` exists and is readable. If the file does not exist or is empty, `rows` will be an empty list, and the function will silently proceed without error.","fix":"Add a check to ensure `results` exists and is readable before proceeding.","confidence":80,"anchor":"def build(results: Path, out_dir: Path, bench_dir: Path, repo_root: Path) -> list[Path]:","verdict":"refuted","skeptic_note":"The claim states that if the `results` file does not exist, the function will 'silently proceed without error'. This is incorrect. If the `results` file does not exist, `results.read_text()` will raise a `FileNotFoundError`, causing the program to crash, which is not silent and is an error. While the script does proceed without error if the file is empty, this specific part of the claim misreads the code's behavior regarding a missing file.","impact_score":0},{"file":"scripts/recordings_from_corpus.py","line":100,"severity":"minor","category":"logic","title":"No handling for case where no frozen passes are found","evidence":"frozen = [r for r in rows if is_frozen_pass(r)]","problem":"If no rows are frozen passes, `frozen` will be empty, and the function will proceed to write no recordings. This is not an error, but it may be unexpected behavior for the caller.","fix":"Add a warning or return early if `frozen` is empty.","confidence":60,"anchor":"frozen = [r for r in rows if is_frozen_pass(r)]","verdict":"refuted","skeptic_note":"The script explicitly prints the number of recordings written, which will be '0' if no frozen passes are found. This informs the caller of the outcome, so the behavior is not unexpected.","impact_score":2},{"file":"scripts/recordings_from_corpus.py","line":102,"severity":"minor","category":"logic","title":"Race condition in directory creation","evidence":"out_dir.mkdir(parents=True, exist_ok=True)","problem":"If multiple processes call `build` with the same `out_dir`, a race condition could occur where one process creates the directory while another is still checking for its existence.","fix":"Use `os.makedirs` with `exist_ok=True` or a lock to avoid race conditions.","confidence":50,"anchor":"out_dir.mkdir(parents=True, exist_ok=True)","verdict":"refuted","skeptic_note":"The `exist_ok=True` argument in `out_dir.mkdir()` explicitly handles the scenario described, preventing errors if the directory is created concurrently by another process. This makes the operation safe for concurrent directory creation.","impact_score":0},{"file":"scripts/recordings_from_corpus.py","line":110,"severity":"minor","category":"contract","title":"No validation that `save_finder_recording` succeeds","evidence":"save_finder_recording(path, result, diff)","problem":"The function does not check if `save_finder_recording` succeeds (e.g., due to permission issues or disk full). If it fails, the recording will not be written, but the function will still return the path.","fix":"Wrap the call in a try-except block and handle or propagate the error.","confidence":70,"anchor":"save_finder_recording(path, result, diff)","verdict":"confirmed","skeptic_note":"The quoted line calls `save_finder_recording` without error handling. If the underlying file operation fails (e.g., due to permissions or disk space), the exception will propagate, but the path will still be added to the `written` list and returned, leading to an incorrect report of successfully written recordings.","impact_score":7},{"file":"scripts/recordings_from_corpus.py","line":117,"severity":"minor","category":"logic","title":"No error handling for `build` failures","evidence":"written = build(args.results, args.out, args.bench_dir, args.repo_root)","problem":"If `build` raises an exception (e.g., due to a missing fixture or invalid input), `main` will propagate it, causing the script to exit with a non-zero status but without a clear error message.","fix":"Wrap the `build` call in a try-except block and print a descriptive error message before returning a non-zero exit code.","confidence":80,"anchor":"def main() -> int:","verdict":"confirmed","skeptic_note":"The `main` function does not explicitly handle exceptions raised by `build` (e.g., `MissingFixtureError`, `NotAFrozenPassError`, or `json.JSONDecodeError`), allowing them to propagate as raw Python tracebacks. While tracebacks provide detail, wrapping the call would enable the script to print a more consistent and user-friendly error message for a command-line utility.","impact_score":4},{"file":"tests/unit/test_recordings_from_corpus.py","line":12,"severity":"minor","category":"types","title":"Modification of `sys.path` at runtime violates type safety","evidence":"sys.path.insert(0, str(Path(__file__).parent.parent.parent / \"scripts\"))","problem":"Modifying `sys.path` at runtime can lead to unpredictable import behavior and violates the principle of explicit imports. This is also not type-safe.","fix":"Use a relative import or ensure the `scripts` directory is in `PYTHONPATH` before running tests.","confidence":85,"anchor":"import sys","verdict":"confirmed","skeptic_note":"The claim accurately identifies that dynamic modification of `sys.path` is generally discouraged as it can lead to less explicit import resolution and potential issues with module discoverability, particularly for static analysis tools like mypy. While functional, it's a departure from robust import practices.","impact_score":5},{"file":"tests/unit/test_recordings_from_corpus.py","line":15,"severity":"minor","category":"contract","title":"Import from a script file may fail in some environments","evidence":"from recordings_from_corpus import (","problem":"The test imports directly from `recordings_from_corpus.py`, which is a script file. This may fail if the file is not in `sys.path` or if the environment does not support direct imports from `.py` files.","fix":"Refactor the shared logic into a module (e.g., `cgis.guardian.recordings_from_corpus`) and import from there.","confidence":75,"anchor":"from recordings_from_corpus import (","verdict":"refuted","skeptic_note":"The diff at line 13 explicitly adds the directory containing 'recordings_from_corpus.py' to `sys.path`, directly addressing the reviewer's concern that the import may fail due to the file not being in `sys.path`.","impact_score":0},{"file":"tests/unit/test_recordings_from_corpus.py","line":20,"severity":"minor","category":"logic","title":"Hardcoded path traversal may break in some environments","evidence":"REPO_ROOT = Path(__file__).resolve().parent.parent.parent","problem":"The path traversal assumes the test file is located at `tests/unit/test_recordings_from_corpus.py`. If the file is moved or the project structure changes, this will break.","fix":"Use a more robust method to locate the repository root, such as `git rev-parse --show-toplevel`.","confidence":70,"anchor":"REPO_ROOT = Path(__file__).resolve().parent.parent.parent","verdict":"confirmed","skeptic_note":"The `REPO_ROOT` calculation relies on a fixed relative path from the current file, which would break if the file is moved or the project structure changes, making the code brittle.","impact_score":5},{"file":"tests/unit/test_recordings_from_corpus.py","line":28,"severity":"major","category":"types","title":"Function return type uses `Any` in type annotation","evidence":"def _rows() -> list[dict[str, Any]]:","problem":"The return type is `list[dict[str, Any]]`, which violates the strict mypy rule against `Any`.","fix":"Replace `Any` with a precise type (e.g., `list[dict[str, object]]` or a TypedDict).","confidence":95,"anchor":"def _rows() -> list[dict[str, Any]]:","verdict":"confirmed","skeptic_note":"The quoted code is present in the diff. The use of `Any` bypasses strict type checking, and while it might not always trigger a direct error depending on mypy configuration, it's generally discouraged in strictly typed code and reduces type safety.","impact_score":2},{"file":"tests/unit/test_recordings_from_corpus.py","line":30,"severity":"minor","category":"logic","title":"No validation that `RESULTS` file exists or is readable","evidence":"return [json.loads(line) for line in RESULTS.read_text().splitlines() if line.strip()]","problem":"The function assumes `RESULTS` exists and is readable. If the file does not exist or is empty, this will raise a `FileNotFoundError` or return an empty list.","fix":"Add a check to ensure `RESULTS` exists and is readable before proceeding.","confidence":80,"anchor":"return [json.loads(line) for line in RESULTS.read_text().splitlines() if line.strip()]","verdict":"confirmed","skeptic_note":"The code does indeed assume `RESULTS` exists and is readable, and `FileNotFoundError` will be raised if it's missing. While this is an informative error for test code, the finding is technically correct. Moreover, other parts of this test file explicitly raise `MissingFixtureError` for similar dependency issues, suggesting a desire for more specific error messages than `FileNotFoundError`.","impact_score":4},{"file":"tests/unit/test_recordings_from_corpus.py","line":35,"severity":"minor","category":"types","title":"Return type uses `Path` but may include non-Path values","evidence":"def rebuilt(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]:","problem":"The return type is `dict[str, Path]`, but the function returns a dictionary where the values are `Path` objects. This is correct, but the type annotation could be more precise (e.g., `dict[str, Path]` is fine, but ensure all values are indeed `Path`).","fix":"No action needed; the type is correct. This is a false positive.","confidence":30,"anchor":"def rebuilt(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]:","verdict":"refuted","skeptic_note":"The type annotation `dict[str, Path]` accurately and precisely describes the return value of the `rebuilt` fixture, which creates a dictionary mapping string stems to `Path` objects. The original claim correctly concludes that no action is needed and it is a false positive, as there is no defect or imprecision to address.","impact_score":0},{"file":"tests/unit/test_recordings_from_corpus.py","line":50,"severity":"minor","category":"tests","title":"Test assumes existence of a judged row in the corpus","evidence":"judged = next(r for r in _rows() if r.get(\"skeptic_model\") and \"matched\" in r)","problem":"The test assumes there is at least one judged row in the corpus. If the corpus changes and no such row exists, the test will raise a `StopIteration` error.","fix":"Add a check to ensure the corpus contains at least one judged row before running the test.","confidence":70,"anchor":"def test_a_judged_row_is_refused(self) -> None:","verdict":"confirmed","skeptic_note":"The test explicitly uses `next()` without a default, which will raise a `StopIteration` error if no row matches the criteria, making the test fragile to changes in the corpus data.","impact_score":6},{"file":"tests/unit/test_recordings_from_corpus.py","line":57,"severity":"minor","category":"tests","title":"Test assumes existence of an unscored row in the corpus","evidence":"failed = next(r for r in _rows() if \"error\" in r and \"matched\" not in r)","problem":"The test assumes there is at least one unscored row in the corpus. If the corpus changes and no such row exists, the test will raise a `StopIteration` error.","fix":"Add a check to ensure the corpus contains at least one unscored row before running the test.","confidence":70,"anchor":"def test_an_unscored_row_is_not_a_pass(self) -> None:","verdict":"confirmed","skeptic_note":"The `next()` function without a default argument will raise `StopIteration` if no row matching the specified criteria is found in the corpus, making the test brittle to changes in its underlying data fixture. This is a plausible failure path for the test itself.","impact_score":4},{"file":"tests/unit/test_recordings_from_corpus.py","line":64,"severity":"minor","category":"tests","title":"Test uses a hardcoded PR number (9999) that may not be meaningful","evidence":"diff_for(9999, tmp_path, REPO_ROOT)","problem":"The test uses a hardcoded PR number (9999) to test the `MissingFixtureError`. This is fine for testing, but it assumes that PR 9999 does not exist in the repository. If PR 9999 is created in the future, the test may fail.","fix":"Use a PR number that is guaranteed not to exist (e.g., a negative number or a very large number).","confidence":60,"anchor":"def test_a_pr_without_a_fixture_is_refused_not_skipped(self, tmp_path: Path) -> None:","verdict":"confirmed","skeptic_note":"The test's reliance on a hardcoded PR number (9999) to simulate a missing fixture creates a brittle dependency on the future state of the repository, as a real PR 9999 and its fixture would cause the test to fail unexpectedly.","impact_score":4},{"file":"tests/unit/test_recordings_from_corpus.py","line":75,"severity":"minor","category":"tests","title":"Test assumes at least 72 frozen passes exist in the corpus","evidence":"assert len(expected) >= 72, (","problem":"The test assumes there are at least 72 frozen passes in the corpus. If the corpus changes and fewer than 72 frozen passes exist, the test will fail.","fix":"Update the assertion to reflect the actual number of frozen passes in the corpus, or make it dynamic.","confidence":70,"anchor":"def test_every_frozen_pass_was_rebuilt(self, rebuilt: dict[str, Path]) -> None:","verdict":"confirmed","skeptic_note":"The assertion `assert len(expected) >= 72` explicitly hardcodes a minimum number of frozen passes. If the corpus changes and provides fewer than 72 frozen passes, this test will indeed fail, making it brittle to changes in the test data.","impact_score":5},{"file":"tests/unit/test_recordings_from_corpus.py","line":100,"severity":"minor","category":"tests","title":"Test assumes existence of `ambiguous_hits` in some rows","evidence":"assert old_policy == {key for key in rebuilt if by_key[key].get(\"ambiguous_hits\")}, (","problem":"The test assumes that the rows needing the pre-#345 precision rule are exactly the rows carrying `ambiguous_hits`. If the corpus changes and this is no longer true, the test will fail.","fix":"Update the assertion to handle cases where `ambiguous_hits` may not exist or may not be the only factor.","confidence":60,"anchor":"def test_every_recording_rescores_to_its_recorded_numbers","verdict":"confirmed","skeptic_note":"The test's explicit assertion `assert old_policy == {key for key in rebuilt if by_key[key].get(\"ambiguous_hits\")}` directly implements the assumption described. If the corpus or the definition of 'old policy' changes such that `ambiguous_hits` is no longer the sole or accurate indicator, the test will indeed fail, confirming the claim about its fragility.","impact_score":4},{"file":"tests/unit/test_recordings_from_corpus.py","line":145,"severity":"minor","category":"logic","title":"Floating-point comparison may be too strict for some use cases","evidence":"return abs(a - b) < 1e-9","problem":"The tolerance of `1e-9` may be too strict for some floating-point comparisons, especially if the values are large or the result of many operations. This could cause false negatives in the test.","fix":"Consider using a relative tolerance (e.g., `math.isclose` with `rel_tol` and `abs_tol`) instead of an absolute tolerance.","confidence":60,"anchor":"def _same(a: float, b: float) -> bool:","verdict":"confirmed","skeptic_note":"The quoted code is present in the diff. While recall and precision values are between 0 and 1, making the 'large values' part of the claim less relevant, a fixed absolute tolerance like 1e-9 can still be too strict due to cumulative floating-point errors from intermediate computations. Using `math.isclose` with both relative and absolute tolerances is generally more robust for critical comparisons in tests, and failing to do so could lead to flaky or falsely failing tests.","impact_score":7}],"summary":"Reviewed type safety (strict mypy violations with `Any`), logic correctness (edge cases in diff handling, error propagation), test coverage (assumptions about corpus data), and library contracts (YAML loading, subprocess calls). Found multiple `Any` violations, missing error handling, and test assumptions that may break with corpus changes.","parse_failed":false,"skeptic_status":"ok","skeptic_judged":24,"skeptic_total":24},"diff":"diff --git a/.gitignore b/.gitignore\nindex 527430f..db9294d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -230,3 +230,4 @@ guardian_metrics.jsonl\n *.db-wal\n *.db-shm\n .martian-workspace/\n+.guardian-recordings/\ndiff --git a/scripts/recordings_from_corpus.py b/scripts/recordings_from_corpus.py\nnew file mode 100644\nindex 0000000..3cdde0d\n--- /dev/null\n+++ b/scripts/recordings_from_corpus.py\n@@ -0,0 +1,183 @@\n+\"\"\"Rebuild frozen finder passes from the recorded corpus, without paying a model (#246).\n+\n+`--replay-finder` isolates the skeptic: the finder is skipped and its recorded\n+findings are judged instead, so a skeptic variant costs skeptic calls only. That\n+is what makes #246 answerable on a budget — but it needs recordings, and none\n+were ever written to disk.\n+\n+They do not have to be re-run. `benchmarks/guardian/results.jsonl` already holds\n+the findings of every scored review, and the diff they were found in is\n+regenerable from git, because the benchmark reviews this repository's own PRs\n+and every fixture's `base` and `head` resolve locally. So a recording costs one\n+`git diff` and nothing else.\n+\n+**Only rows that had no skeptic are eligible**, and the reason is not\n+tidiness. `load_finder_recording` strips verdicts on read, so a judged row would\n+not smuggle its verdicts into the next arm — but `apply_judgements` rewrites\n+confidence to `round(confidence * 0.9)` on an `uncertain` verdict and the load\n+path does not restore it (#279). A recording taken from a judged run therefore\n+carries confidences that are not the finder's own, on exactly the subset a\n+skeptic experiment is about.\n+\n+Verified before use rather than after: `tests/unit/test_recordings_from_corpus.py`\n+re-scores every rebuilt recording and reproduces the number the corpus recorded —\n+72 of 72, under the scoring policy in force when each row was written. The seven\n+that differ under today's rule are exactly the seven carrying `ambiguous_hits`,\n+which is the #345 policy change that `CURATION.md` says results.jsonl is\n+deliberately not rescored for.\n+\"\"\"\n+\n+import argparse\n+import json\n+import subprocess\n+import sys\n+from pathlib import Path\n+from typing import Any\n+\n+import yaml\n+\n+from cgis.guardian.findings import Finding, ReviewResult\n+from cgis.guardian.recording import save_finder_recording\n+\n+#: Fields `annotate_matches` adds to a stored finding. Stripped on the way back\n+#: to a `Finding`, which does not declare them.\n+_ANNOTATIONS = (\"matched\", \"verdict\", \"skeptic_note\")\n+\n+\n+class NotAFrozenPassError(RuntimeError):\n+    \"\"\"Raised when a row cannot stand in for an unjudged finder pass.\n+\n+    Refused rather than cleaned up: a row whose findings were already judged has\n+    confidences the skeptic rewrote (#279), and silently accepting it would make\n+    two arms of an experiment differ by something nobody chose.\n+    \"\"\"\n+\n+\n+class MissingFixtureError(RuntimeError):\n+    \"\"\"Raised when a row names a PR with no ground-truth fixture, or unresolvable shas.\n+\n+    The diff is the whole reason a replay needs no worktree. Without one there\n+    is nothing to record, and inventing an empty diff would produce a recording\n+    that replays as \"the finder saw nothing\".\n+    \"\"\"\n+\n+\n+def is_frozen_pass(row: dict[str, Any]) -> bool:\n+    \"\"\"True for a scored row whose findings are the finder's own, unjudged.\"\"\"\n+    if \"matched\" not in row or \"precision\" not in row:\n+        return False\n+    if row.get(\"skeptic_model\"):\n+        return False\n+    return all(f.get(\"verdict\") is None for f in (row.get(\"findings\") or []))\n+\n+\n+def row_key(row: dict[str, Any]) -> str:\n+    \"\"\"`pr@timestamp` — the same identity `guardian_calibrate` writes.\n+\n+    Used as the recording's filename so a replayed number can be traced to the\n+    exact run it came from. Without that, a green replay could mean \"reproduced\n+    *a* run\", which is not the claim being made.\n+    \"\"\"\n+    return f\"{row['pr']}@{row['timestamp']}\"\n+\n+\n+def diff_for(pr: int, bench_dir: Path, repo_root: Path) -> str:\n+    \"\"\"The diff the finder saw, regenerated from the fixture's base and head.\n+\n+    `base...head`, three dots, mirroring `ContextCollector._diff_range` — the\n+    two-dot form would include everything that landed on the base branch after\n+    the PR was cut, which is not what any recorded review was shown.\n+    \"\"\"\n+    fixture = bench_dir / f\"pr-{pr}.yaml\"\n+    if not fixture.is_file():\n+        _msg = f\"No ground-truth fixture for pr-{pr} at {fixture}.\"\n+        raise MissingFixtureError(_msg)\n+    spec = yaml.safe_load(fixture.read_text(encoding=\"utf-8\"))\n+    base, head = spec.get(\"base\"), spec.get(\"head\")\n+    for name, sha in ((\"base\", base), (\"head\", head)):\n+        if (\n+            not sha\n+            or subprocess.run(\n+                [\"git\", \"rev-parse\", \"--verify\", \"--quiet\", f\"{sha}^{{commit}}\"],\n+                capture_output=True,\n+                cwd=repo_root,\n+                check=False,\n+            ).returncode\n+        ):\n+            _msg = f\"pr-{pr}: {name} {sha!r} does not resolve in {repo_root}.\"\n+            raise MissingFixtureError(_msg)\n+    result = subprocess.run(\n+        [\"git\", \"diff\", f\"{base}...{head}\"],\n+        capture_output=True,\n+        text=True,\n+        cwd=repo_root,\n+        check=True,\n+    )\n+    if not result.stdout.strip():\n+        _msg = f\"pr-{pr}: {base}...{head} is an empty diff; a review of nothing is not a pass.\"\n+        raise MissingFixtureError(_msg)\n+    return result.stdout\n+\n+\n+def recording_for(row: dict[str, Any], diff: str) -> tuple[ReviewResult, str]:\n+    \"\"\"A `ReviewResult` + diff pair equivalent to what the finder produced.\n+\n+    `summary` is empty because the corpus never stored one. Nothing downstream\n+    reads it — the skeptic judges findings against diff hunks and the scorer\n+    counts matches — so an empty string is honest about what is known rather\n+    than a reconstruction of prose nobody kept.\n+    \"\"\"\n+    if not is_frozen_pass(row):\n+        _msg = (\n+            f\"{row_key(row)} is not an unjudged finder pass (skeptic={row.get('skeptic_model')!r}).\"\n+        )\n+        raise NotAFrozenPassError(_msg)\n+    findings = [\n+        Finding.model_validate({k: v for k, v in f.items() if k not in _ANNOTATIONS})\n+        for f in (row.get(\"findings\") or [])\n+    ]\n+    return ReviewResult(\n+        findings=findings, summary=\"\", parse_failed=bool(row.get(\"parse_failed\"))\n+    ), diff\n+\n+\n+def build(results: Path, out_dir: Path, bench_dir: Path, repo_root: Path) -> list[Path]:\n+    \"\"\"Write one recording per frozen pass; return the paths written.\"\"\"\n+    rows = [\n+        json.loads(line)\n+        for line in results.read_text(encoding=\"utf-8\").splitlines()\n+        if line.strip()\n+    ]\n+    frozen = [r for r in rows if is_frozen_pass(r)]\n+    out_dir.mkdir(parents=True, exist_ok=True)\n+    diffs: dict[int, str] = {}\n+    written: list[Path] = []\n+    for row in frozen:\n+        pr = row[\"pr\"]\n+        if pr not in diffs:\n+            diffs[pr] = diff_for(pr, bench_dir, repo_root)\n+        result, diff = recording_for(row, diffs[pr])\n+        path = out_dir / f\"{row_key(row).replace(':', '-')}.json\"\n+        save_finder_recording(path, result, diff)\n+        written.append(path)\n+    return written\n+\n+\n+def main() -> int:\n+    \"\"\"Rebuild every frozen finder pass and report what was written.\"\"\"\n+    repo_root = Path(__file__).resolve().parent.parent\n+    parser = argparse.ArgumentParser(description=__doc__)\n+    parser.add_argument(\n+        \"--results\", type=Path, default=repo_root / \"benchmarks/guardian/results.jsonl\"\n+    )\n+    parser.add_argument(\"--out\", type=Path, default=repo_root / \".guardian-recordings\")\n+    parser.add_argument(\"--bench-dir\", type=Path, default=repo_root / \"benchmarks/guardian\")\n+    parser.add_argument(\"--repo-root\", type=Path, default=repo_root)\n+    args = parser.parse_args()\n+    written = build(args.results, args.out, args.bench_dir, args.repo_root)\n+    print(f\"{len(written)} recordings written to {args.out}\")\n+    return 0\n+\n+\n+if __name__ == \"__main__\":\n+    sys.exit(main())\ndiff --git a/tests/unit/test_recordings_from_corpus.py b/tests/unit/test_recordings_from_corpus.py\nnew file mode 100644\nindex 0000000..80e34cc\n--- /dev/null\n+++ b/tests/unit/test_recordings_from_corpus.py\n@@ -0,0 +1,156 @@\n+\"\"\"A rebuilt finder pass must replay as the run it came from (#246).\n+\n+The recordings exist so a skeptic variant can be measured without paying the\n+finder. That only works if a replayed number is the *same run's* number, so this\n+verifies the tool before anything is measured with it — the alternative is\n+finding out after the calls have been spent.\n+\n+Identity is asserted before metrics, at the downstream consumer's suggestion:\n+comparing scores first lets a green result mean \"reproduced *a* run equally\n+well\", which is not the claim.\n+\"\"\"\n+\n+import json\n+import sys\n+from pathlib import Path\n+from typing import Any\n+\n+import pytest\n+\n+from cgis.guardian.bench import load_ground_truth, match_findings, score\n+from cgis.guardian.recording import load_finder_recording\n+\n+sys.path.insert(0, str(Path(__file__).parent.parent.parent / \"scripts\"))\n+\n+from recordings_from_corpus import (\n+    MissingFixtureError,\n+    NotAFrozenPassError,\n+    build,\n+    diff_for,\n+    is_frozen_pass,\n+    recording_for,\n+    row_key,\n+)\n+\n+REPO_ROOT = Path(__file__).resolve().parent.parent.parent\n+BENCH_DIR = REPO_ROOT / \"benchmarks\" / \"guardian\"\n+RESULTS = BENCH_DIR / \"results.jsonl\"\n+\n+\n+def _rows() -> list[dict[str, Any]]:\n+    return [json.loads(line) for line in RESULTS.read_text().splitlines() if line.strip()]\n+\n+\n+@pytest.fixture(scope=\"module\")\n+def rebuilt(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]:\n+    \"\"\"Every frozen pass rebuilt once, keyed by the row it came from.\"\"\"\n+    out = tmp_path_factory.mktemp(\"recordings\")\n+    written = build(RESULTS, out, BENCH_DIR, REPO_ROOT)\n+    return {p.stem: p for p in written}\n+\n+\n+class TestSelection:\n+    \"\"\"Which rows may stand in for an unjudged finder pass.\"\"\"\n+\n+    def test_a_judged_row_is_refused(self) -> None:\n+        \"\"\"A skeptic rewrote confidence on its `uncertain` findings (#279).\n+\n+        `load_finder_recording` strips verdicts, so the danger is not smuggled\n+        judgements — it is that the confidences are no longer the finder's, on\n+        exactly the subset a skeptic experiment compares.\n+        \"\"\"\n+        judged = next(r for r in _rows() if r.get(\"skeptic_model\") and \"matched\" in r)\n+        assert not is_frozen_pass(judged)\n+        with pytest.raises(NotAFrozenPassError, match=\"not an unjudged finder pass\"):\n+            recording_for(judged, \"diff\")\n+\n+    def test_an_unscored_row_is_not_a_pass(self) -> None:\n+        \"\"\"The API-failure rows carry no findings and no score to reproduce.\"\"\"\n+        failed = next(r for r in _rows() if \"error\" in r and \"matched\" not in r)\n+        assert not is_frozen_pass(failed)\n+\n+    def test_a_pr_without_a_fixture_is_refused_not_skipped(self, tmp_path: Path) -> None:\n+        \"\"\"An invented empty diff would replay as \"the finder saw nothing\".\"\"\"\n+        with pytest.raises(MissingFixtureError, match=\"No ground-truth fixture\"):\n+            diff_for(9999, tmp_path, REPO_ROOT)\n+\n+\n+class TestReplayReproducesTheRunItCameFrom:\n+    \"\"\"Identity first, then the numbers.\"\"\"\n+\n+    def test_every_frozen_pass_was_rebuilt(self, rebuilt: dict[str, Path]) -> None:\n+        expected = {row_key(r).replace(\":\", \"-\") for r in _rows() if is_frozen_pass(r)}\n+        assert len(expected) >= 72, (\n+            f\"only {len(expected)} frozen passes found in the corpus, expected at least 72 — \"\n+            \"an empty result must not read as 'every pass was rebuilt'.\"\n+        )\n+        assert set(rebuilt) == expected\n+\n+    def test_each_recording_is_the_run_it_is_named_after(self, rebuilt: dict[str, Path]) -> None:\n+        \"\"\"Identity before metrics: same run, same finding count, same order.\n+\n+        Load-bearing, not belt-and-braces, and measured rather than assumed:\n+        **10 groups of runs share a PR and identical recall, precision and noise\n+        while carrying different finding sets, covering 48 of the 72 frozen\n+        passes.** A metrics-only check would therefore accept the wrong run for\n+        two thirds of this corpus and report a clean reproduction.\n+\n+        Verified by mutation — pairing each recording with the next run's\n+        findings turns this red before the scoring test even runs.\n+        \"\"\"\n+        by_key = {row_key(r).replace(\":\", \"-\"): r for r in _rows() if is_frozen_pass(r)}\n+        for key, path in rebuilt.items():\n+            row = by_key[key]\n+            recording = load_finder_recording(path)\n+            assert len(recording.result.findings) == len(row.get(\"findings\") or []), key\n+            for rebuilt_f, stored_f in zip(\n+                recording.result.findings, row.get(\"findings\") or [], strict=True\n+            ):\n+                assert (rebuilt_f.file, rebuilt_f.line, rebuilt_f.title) == (\n+                    stored_f[\"file\"],\n+                    stored_f[\"line\"],\n+                    stored_f[\"title\"],\n+                ), key\n+\n+    def test_every_recording_rescores_to_its_recorded_numbers(\n+        self, rebuilt: dict[str, Path]\n+    ) -> None:\n+        \"\"\"72 of 72, under the scoring policy in force when each row was written.\n+\n+        Seven rows do not reproduce under today's rule, and they are exactly the\n+        seven carrying `ambiguous_hits`: #345 moved ambiguous hits into the\n+        precision denominator, and `CURATION.md` states results.jsonl is\n+        deliberately not rescored for it. So the assertion is structural — a row\n+        reproduces under the current rule, or it has ambiguous hits and\n+        reproduces under the old one — rather than a count that a new bench run\n+        would break.\n+        \"\"\"\n+        by_key = {row_key(r).replace(\":\", \"-\"): r for r in _rows() if is_frozen_pass(r)}\n+        truths: dict[int, Any] = {}\n+        offenders: list[str] = []\n+        old_policy: set[str] = set()\n+        for key, path in rebuilt.items():\n+            row = by_key[key]\n+            pr = row[\"pr\"]\n+            truths.setdefault(pr, load_ground_truth(BENCH_DIR / f\"pr-{pr}.yaml\"))\n+            recording = load_finder_recording(path)\n+            match = match_findings(recording.result.findings, truths[pr])\n+            current = score(match, truths[pr])\n+            if _same(current.recall, row[\"recall\"]) and _same(current.precision, row[\"precision\"]):\n+                continue\n+            relevant = len(match.matched) + len(match.noise)\n+            old_precision = len(match.matched) / relevant if relevant else 1.0\n+            if _same(current.recall, row[\"recall\"]) and _same(old_precision, row[\"precision\"]):\n+                old_policy.add(key)\n+                continue\n+            offenders.append(f\"{key}: stored p={row['precision']} got {current.precision}\")\n+\n+        assert not offenders, f\"recordings that reproduce under neither policy: {offenders}\"\n+        assert old_policy == {key for key in rebuilt if by_key[key].get(\"ambiguous_hits\")}, (\n+            \"the rows needing the pre-#345 precision rule are no longer exactly the rows \"\n+            f\"carrying ambiguous_hits: {sorted(old_policy)}\"\n+        )\n+\n+\n+def _same(a: float, b: float) -> bool:\n+    return abs(a - b) < 1e-9\n"}