codegraph-brain 0.14.1__tar.gz → 0.16.0__tar.gz

codegraph_brain-0.16.0/.github/test-count-baseline

@@ -0,0 +1 @@
+1957

{codegraph_brain-0.14.1 → codegraph_brain-0.16.0}/.github/workflows/ci.yml

@@ -67,6 +67,25 @@ jobs:
           uv run --frozen pytest --cov=cgis --cov=scripts
           --cov-report=term-missing --cov-report=xml
 
+      - name: Test-count floor
+        # Deliberately a workflow step and not a test (#405). On 2026-08-17 a
+        # stale-tree push deleted 24 test files and 5 modules under src/cgis,
+        # and the step above passed: the tests that would have failed went with
+        # them. Every floor the repository had lived inside a test file, so the
+        # push that tripped them also removed them.
+        #
+        # The floor is read from the *base branch*, not from this checkout — a
+        # stale tree carries a stale baseline, and comparing the branch against
+        # its own copy passes the exact incident above. `fetch-depth: 0` on the
+        # checkout is what makes origin/<base> readable here.
+        #
+        # `github.base_ref` is set on pull_request only; on a push to the trunk
+        # it is empty and ref_name is the branch being pushed, which is the
+        # right comparison there too.
+        run: >
+          uv run --frozen python scripts/check_test_count.py
+          --base-ref "origin/${{ github.base_ref || github.ref_name }}"
+
       - name: Project version for Sonar
         id: version
         # Without this every analysis records the version as the literal string

{codegraph_brain-0.14.1 → codegraph_brain-0.16.0}/.github/workflows/guardian.yml

@@ -90,6 +90,12 @@ jobs:
           GUARDIAN_FEATURES: ${{ vars.GUARDIAN_FEATURES }}
           GUARDIAN_SKEPTIC: ${{ vars.GUARDIAN_SKEPTIC }}
           GUARDIAN_SKEPTIC_MODEL: ${{ vars.GUARDIAN_SKEPTIC_MODEL }}
+          # Static-evidence pass for the skeptic (#401). Wired but not set:
+          # the repository variable is deliberately absent, so enabling it is a
+          # decision made in repo settings rather than a code change nobody
+          # reviews. An unset variable expands to "", which the flag reads as
+          # off — only the literal "1" enables it.
+          GUARDIAN_EVIDENCE: ${{ vars.GUARDIAN_EVIDENCE }}
         run: |
           uv run --frozen python scripts/guardian_review.py \
             --output guardian_report.md \

{codegraph_brain-0.14.1 → codegraph_brain-0.16.0}/.gitignore

@@ -230,3 +230,4 @@ guardian_metrics.jsonl
 *.db-wal
 *.db-shm
 .martian-workspace/
+.guardian-recordings/

codegraph_brain-0.16.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.16.0"
+}

{codegraph_brain-0.14.1 → codegraph_brain-0.16.0}/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## [0.16.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.15.0...codegraph-brain-v0.16.0) (2026-08-17)
+
+
+### Features
+
+* **bench:** a control arm, so an evidence result can be attributed ([#401](https://github.com/zaebee/codegraph-brain/issues/401)) ([#406](https://github.com/zaebee/codegraph-brain/issues/406)) ([241dbae](https://github.com/zaebee/codegraph-brain/commit/241dbae334be897cb6bf3d151d097b64ab5a56e7))
+* **bench:** both skeptics over the same frozen passes, and the evidence bench replay dropped ([#246](https://github.com/zaebee/codegraph-brain/issues/246)) ([#411](https://github.com/zaebee/codegraph-brain/issues/411)) ([2c1d50b](https://github.com/zaebee/codegraph-brain/commit/2c1d50b2abb18b0c0bad1a9b46f71531eebba16c))
+* **ci:** a floor under the size of the test suite itself ([#405](https://github.com/zaebee/codegraph-brain/issues/405)) ([#409](https://github.com/zaebee/codegraph-brain/issues/409)) ([0fbcbac](https://github.com/zaebee/codegraph-brain/commit/0fbcbac573651591750f017e3793b692c05377ce))
+* **guardian:** tell the skeptic when no checker ran, and which verdict that takes ([#407](https://github.com/zaebee/codegraph-brain/issues/407)) ([#410](https://github.com/zaebee/codegraph-brain/issues/410)) ([2470ddb](https://github.com/zaebee/codegraph-brain/commit/2470ddb45a29f19e2abf90d2f6ce5b5fadad09f3))
+
+## [0.15.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.14.1...codegraph-brain-v0.15.0) (2026-08-17)
+
+
+### Features
+
+* **bench:** rebuild frozen finder passes from the corpus, free ([#246](https://github.com/zaebee/codegraph-brain/issues/246)) ([#399](https://github.com/zaebee/codegraph-brain/issues/399)) ([b73f6e0](https://github.com/zaebee/codegraph-brain/commit/b73f6e0d40c5bdc4dadca4acd468d7835670be28))
+* **guardian:** a replay arm that pairs each finding against its own baseline ([#401](https://github.com/zaebee/codegraph-brain/issues/401)) ([#404](https://github.com/zaebee/codegraph-brain/issues/404)) ([b2ab59a](https://github.com/zaebee/codegraph-brain/commit/b2ab59a8d7b5f62a307874b8a0eb9573273fa0ea))
+* **guardian:** let the skeptic see what the checkers say ([#401](https://github.com/zaebee/codegraph-brain/issues/401)) ([#402](https://github.com/zaebee/codegraph-brain/issues/402)) ([af82147](https://github.com/zaebee/codegraph-brain/commit/af821473080bd3a717679e36c90164a43ca07f7b))
+
 ## [0.14.1](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.14.0...codegraph-brain-v0.14.1) (2026-08-16)
 
 

{codegraph_brain-0.14.1 → codegraph_brain-0.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: codegraph-brain
-Version: 0.14.1
+Version: 0.16.0
 Summary: Semantic code graph for AI agents — deterministic FQN resolution, impact analysis and architectural drift gates, exposed over MCP.
 Project-URL: Homepage, https://github.com/zaebee/codegraph-brain
 Project-URL: Repository, https://github.com/zaebee/codegraph-brain

codegraph_brain-0.16.0/benchmarks/guardian/experiments/246-cross-vendor/README.md

@@ -0,0 +1,71 @@
+# Cross-vendor skeptic: still blocked, and now on measured limits (#246)
+
+Two attempts at the third arm, both stopped by the validity gate before any
+number was reported. The gate is the deliverable here; the comparison is not.
+
+## Why a gate at all
+
+`judge_finding` contains provider errors per finding and returns None, so a
+spent quota, an HTTP 429, a 402 and unparseable JSON all arrive as **the same
+row shape as a skeptic that refused to refute anything** — which is exactly what
+#246 predicts for a same-vendor skeptic. A broken arm would confirm the
+hypothesis. `MAX_UNRULED_RATE` refuses to print a comparison above 5% unruled,
+keeps the rows, and exits 2.
+
+It fired on both attempts. Neither produced a number that could be read.
+
+## Attempt 1 — free tier. The quota is 50 requests per day.
+
+`nvidia/nemotron-3-super-120b-a12b:free`, the whole corpus: **96 of 135 findings
+unruled**, and the pattern is a cutoff in time rather than a property of the
+input.
+
+| PR | evidence | unruled |
+|---|---|---:|
+| 122 (first) | no | **0 / 29** |
+| 140 | yes | 29 / 39 |
+| 141, 142, 143, 144 | — | **100%** |
+
+39 answered, then silence. With the day's earlier probes (4 candidate calls, 6
+in a smoke run) that is 49 successful requests before the wall — the documented
+free allowance is 50/day, and this is that number arrived at from the data.
+
+Rows: `free-nemotron-quota-cutoff.jsonl`.
+
+## Attempt 2 — paid. Two separate faults, one after the other.
+
+`qwen/qwen3.7-plus`, chosen because capability has to match: comparing
+`gemini-2.5-flash` against a small free model measures strong-versus-weak and
+calls it same-versus-cross.
+
+**First fault — thinking.** 118 of 135 unruled. `qwen3.7-plus` puts its chain of
+thought in a separate `reasoning` field and fills `content` only at the end, so
+on the larger prompts it spent the entire 8,000-token budget thinking and
+returned `content: null`. Raising the budget would have fixed the symptom and
+broken the experiment: the arm it is compared against is not doing extended
+thinking, so the two arms would differ in a dimension nobody chose. The provider
+now sends `reasoning: {"enabled": …}` explicitly in both directions, off by
+default, so a run can state what it did.
+
+**Second fault — credits.** The re-run failed on HTTP **402**: the account has
+`total_credits: $0` against `total_usage: $0.159`. The trial allowance is spent;
+paid models are unavailable until it is topped up.
+
+The second fault took half an hour to identify because the warning read only
+"Skeptic judgement failed; finding stays unruled" — 118 identical lines, no
+type, no message. It now carries both. A quota, a truncated reasoning model and
+a 402 have three different remedies and all three land in that one `except`.
+
+## To unblock
+
+1. **Top up OpenRouter.** The measured cost of the run with reasoning off is
+   ~$0.15: 135 findings × ~2,700 prompt tokens at \$0.32/M, plus short
+   completions at \$1.28/M. A dollar covers it several times over.
+2. **Free tier across three days.** The input is a frozen recording, so the run
+   is deterministic and can be split — 50 requests a day, 135 needed. Fragile,
+   and it needs resume support the tool does not have.
+3. A local ollama, which this issue already names as the no-cost option.
+
+Everything else is in place: the corpus with both finder orientations, the
+recordings, per-arm models, the scoring, and the gate that stopped two wrong
+answers from being published.

codegraph_brain-0.16.0/benchmarks/guardian/experiments/246-cross-vendor/free-nemotron-quota-cutoff.jsonl

@@ -0,0 +1,74 @@
+{"timestamp": "2026-08-17T13:17:43.517549+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-31.032903+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.18181818181818182, "precision": 0.6666666666666666, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:05.186110+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-31.032903+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 0.6666666666666666, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:15.602819+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-59.496508+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:24.136220+00:00", "pr": 122, "pass": "122@2026-06-10T20-22-59.496508+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:36.000053+00:00", "pr": 122, "pass": "122@2026-06-10T20-23-51.775172+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:18:50.719370+00:00", "pr": 122, "pass": "122@2026-06-10T20-23-51.775172+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 1, "confirmed": 1, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:19:01.976110+00:00", "pr": 122, "pass": "122@2026-06-10T20-32-35.483908+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:19:23.184478+00:00", "pr": 122, "pass": "122@2026-06-10T20-32-35.483908+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 2, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:19:36.855990+00:00", "pr": 122, "pass": "122@2026-06-10T20-33-10.513615+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:20:22.191259+00:00", "pr": 122, "pass": "122@2026-06-10T20-33-10.513615+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:20:35.820094+00:00", "pr": 122, "pass": "122@2026-06-10T21-58-22.314534+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:21:08.068793+00:00", "pr": 122, "pass": "122@2026-06-10T21-58-22.314534+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": false, "findings": 4, "refuted": 1, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.09090909090909091, "precision": 0.3333333333333333, "noise": 2, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:21:15.458172+00:00", "pr": 122, "pass": "122@2026-06-11T00-11-53.683357+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:21.049843+00:00", "pr": 122, "pass": "122@2026-06-11T00-11-53.683357+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:29.468118+00:00", "pr": 122, "pass": "122@2026-06-11T00-12-40.927704+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:35.271913+00:00", "pr": 122, "pass": "122@2026-06-11T00-12-40.927704+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:46.009233+00:00", "pr": 122, "pass": "122@2026-06-11T00-14-36.809184+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:21:58.953594+00:00", "pr": 122, "pass": "122@2026-06-11T00-14-36.809184+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:06.863138+00:00", "pr": 122, "pass": "122@2026-06-11T00-32-51.894263+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:18.621710+00:00", "pr": 122, "pass": "122@2026-06-11T00-32-51.894263+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:22:34.978570+00:00", "pr": 122, "pass": "122@2026-06-11T00-36-53.749615+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:23:29.360852+00:00", "pr": 122, "pass": "122@2026-06-11T00-36-53.749615+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:35.269160+00:00", "pr": 122, "pass": "122@2026-06-11T00-42-55.162925+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:43.563281+00:00", "pr": 122, "pass": "122@2026-06-11T00-42-55.162925+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:23:53.848653+00:00", "pr": 122, "pass": "122@2026-06-11T00-48-45.262981+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 1, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.09090909090909091, "precision": 1.0, "noise": 0, "killed_gt": ["file-container-regression"]}
+{"timestamp": "2026-08-17T13:24:05.969279+00:00", "pr": 122, "pass": "122@2026-06-11T00-48-45.262981+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.18181818181818182, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:24:25.419473+00:00", "pr": 140, "pass": "140@2026-06-10T20-24-40.504167+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:25:25.250544+00:00", "pr": 140, "pass": "140@2026-06-10T20-24-40.504167+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:25:47.211803+00:00", "pr": 140, "pass": "140@2026-06-10T20-25-19.261202+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 5, "refuted": 2, "uncertain": 1, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 3, "killed_gt": []}
+{"timestamp": "2026-08-17T13:26:24.777165+00:00", "pr": 140, "pass": "140@2026-06-10T20-25-19.261202+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 5, "refuted": 0, "uncertain": 3, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 5, "killed_gt": []}
+{"timestamp": "2026-08-17T13:26:33.938777+00:00", "pr": 140, "pass": "140@2026-06-10T20-26-01.543236+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:08.103278+00:00", "pr": 140, "pass": "140@2026-06-10T20-26-01.543236+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:17.614733+00:00", "pr": 140, "pass": "140@2026-06-11T00-16-26.605409+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:17.769243+00:00", "pr": 140, "pass": "140@2026-06-11T00-16-26.605409+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:27.876729+00:00", "pr": 140, "pass": "140@2026-06-11T00-17-25.717492+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 3, "unruled": 0, "recall": 0.06666666666666667, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:28.068613+00:00", "pr": 140, "pass": "140@2026-06-11T00-17-25.717492+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.06666666666666667, "precision": 0.3333333333333333, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:38.870523+00:00", "pr": 140, "pass": "140@2026-06-11T00-33-44.053747+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:39.001629+00:00", "pr": 140, "pass": "140@2026-06-11T00-33-44.053747+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:45.424281+00:00", "pr": 140, "pass": "140@2026-06-11T00-37-50.526658+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:45.544269+00:00", "pr": 140, "pass": "140@2026-06-11T00-37-50.526658+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:53.801567+00:00", "pr": 140, "pass": "140@2026-06-11T00-50-08.134629+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:27:53.921093+00:00", "pr": 140, "pass": "140@2026-06-11T00-50-08.134629+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.06666666666666667, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:06.369511+00:00", "pr": 140, "pass": "140@2026-07-29T17-29-19.861365+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 22, "refuted": 10, "uncertain": 2, "confirmed": 10, "unruled": 0, "recall": 0.26666666666666666, "precision": 0.3333333333333333, "noise": 8, "killed_gt": ["count-routers-on-squared", "weights-keyerror-custom-yaml"]}
+{"timestamp": "2026-08-17T13:29:07.917533+00:00", "pr": 140, "pass": "140@2026-07-29T17-29-19.861365+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 22, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 22, "recall": 0.4, "precision": 0.2727272727272727, "noise": 16, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:21.415626+00:00", "pr": 141, "pass": "141@2026-06-10T20-26-53.683221+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:21.527539+00:00", "pr": 141, "pass": "141@2026-06-10T20-26-53.683221+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:28.243544+00:00", "pr": 141, "pass": "141@2026-06-10T20-29-48.066882+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:28.360785+00:00", "pr": 141, "pass": "141@2026-06-10T20-29-48.066882+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:37.370316+00:00", "pr": 141, "pass": "141@2026-06-10T21-59-24.111065+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 1.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:37.495452+00:00", "pr": 141, "pass": "141@2026-06-10T21-59-24.111065+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 1.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:49.082379+00:00", "pr": 142, "pass": "142@2026-06-10T20-30-15.605690+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.5, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:49.243691+00:00", "pr": 142, "pass": "142@2026-06-10T20-30-15.605690+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.5, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:56.956875+00:00", "pr": 142, "pass": "142@2026-06-10T21-52-43.992582+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:29:57.130875+00:00", "pr": 142, "pass": "142@2026-06-10T21-52-43.992582+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 2, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:04.860920+00:00", "pr": 142, "pass": "142@2026-06-10T21-53-15.240002+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:05.012683+00:00", "pr": 142, "pass": "142@2026-06-10T21-53-15.240002+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": false, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:14.805023+00:00", "pr": 142, "pass": "142@2026-06-11T00-45-11.791674+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 3, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:14.969320+00:00", "pr": 142, "pass": "142@2026-06-11T00-45-11.791674+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:23.741156+00:00", "pr": 142, "pass": "142@2026-06-11T00-50-52.772949+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 2, "uncertain": 1, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:23.921669+00:00", "pr": 142, "pass": "142@2026-06-11T00-50-52.772949+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": false, "findings": 3, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 3, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:43.638312+00:00", "pr": 143, "pass": "143@2026-06-11T00-46-29.982484+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:30:43.909808+00:00", "pr": 143, "pass": "143@2026-06-11T00-46-29.982484+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:21.206192+00:00", "pr": 143, "pass": "143@2026-07-29T17-15-00.138863+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 28, "refuted": 8, "uncertain": 1, "confirmed": 19, "unruled": 0, "recall": 1.0, "precision": 0.3, "noise": 14, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:22.809114+00:00", "pr": 143, "pass": "143@2026-07-29T17-15-00.138863+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 28, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 28, "recall": 1.0, "precision": 0.21428571428571427, "noise": 22, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:44.357216+00:00", "pr": 144, "pass": "144@2026-06-10T20-32-21.181637+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 2, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:44.503817+00:00", "pr": 144, "pass": "144@2026-06-10T20-32-21.181637+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-2.5-flash", "evidence": true, "findings": 2, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 2, "recall": 0.0, "precision": 0.0, "noise": 2, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:58.741622+00:00", "pr": 144, "pass": "144@2026-06-11T00-25-01.536057+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:32:58.913488+00:00", "pr": 144, "pass": "144@2026-06-11T00-25-01.536057+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:06.869381+00:00", "pr": 144, "pass": "144@2026-06-11T00-47-31.514757+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 1, "uncertain": 0, "confirmed": 0, "unruled": 0, "recall": 0.0, "precision": 1.0, "noise": 0, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:06.990972+00:00", "pr": 144, "pass": "144@2026-06-11T00-47-31.514757+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:14.521992+00:00", "pr": 144, "pass": "144@2026-06-11T00-52-31.320750+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 1, "unruled": 0, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:33:14.623054+00:00", "pr": 144, "pass": "144@2026-06-11T00-52-31.320750+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "gemini-3.5-flash", "evidence": true, "findings": 1, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 1, "recall": 0.0, "precision": 0.0, "noise": 1, "killed_gt": []}
+{"timestamp": "2026-08-17T13:34:10.907748+00:00", "pr": 144, "pass": "144@2026-07-29T17-28-33.271744+00-00", "arm": "gemini", "skeptic_model": "gemini-2.5-flash", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 20, "refuted": 7, "uncertain": 0, "confirmed": 13, "unruled": 0, "recall": 0.6, "precision": 0.23076923076923078, "noise": 5, "killed_gt": []}
+{"timestamp": "2026-08-17T13:34:12.013311+00:00", "pr": 144, "pass": "144@2026-07-29T17-28-33.271744+00-00", "arm": "openrouter", "skeptic_model": "nvidia/nemotron-3-super-120b-a12b:free", "finder_model": "mistral-medium-latest", "evidence": true, "findings": 20, "refuted": 0, "uncertain": 0, "confirmed": 0, "unruled": 20, "recall": 0.6, "precision": 0.15, "noise": 8, "killed_gt": []}

codegraph_brain-0.16.0/benchmarks/guardian/experiments/401-evidence/README.md

@@ -0,0 +1,140 @@
+# Does checker output change the skeptic's verdicts? (#401)
+
+#402 gave the skeptic the repository's `mypy --strict` and `ruff check` output. Whether
+that changes anything was unmeasured, and `GUARDIAN_EVIDENCE=1` was switched on without a
+number behind it. This is the number.
+
+## Design
+
+One review, three arms, paired at the level of the individual finding.
+
+| arm | what it is |
+|---|---|
+| **baseline** | the verdicts the recorded review actually produced, no evidence |
+| **control** | the same 24 findings re-judged, still no evidence |
+| **treatment** | the same 24 findings re-judged with evidence, collected at the reviewed commit |
+
+The finder is never called. The subject is `guardian-finder-399`, the review that motivated
+the RFC: 24 findings, of which the RFC classified exactly one as substantive.
+
+**The control arm is not optional rigour.** `gemini.py` sends no temperature and
+`guardian.yml` sets none, so the skeptic runs at the provider default: the recorded verdicts
+are one draw, not a fixed point. Baseline against treatment therefore measures the effect
+*plus* the resampling noise, and only a no-evidence replay separates them. This is the same
+confound the replay arm removes at the finder, reappearing one level down.
+
+Evidence is collected in a worktree at `05af753d` — the commit `refs/pull/399/head` pointed
+at when the review ran. **Not** the artifact's `workflow_run.head_sha`: `guardian.yml` is
+triggered by `issue_comment`, so that field is the tip of the trunk (`fd4e53a2`, a release
+commit) and the reviewed files do not exist in it.
+
+## Results
+
+Baseline: **19 confirmed / 5 refuted**.
+
+| arm | conf→conf | conf→ref | conf→unc | ref→conf | ref→ref | moved | cites a checker |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| control, draw 1 | 17 | 2 | 0 | 2 | 3 | 4 | 6/24 |
+| control, draw 2 | 18 | 1 | 0 | 0 | 5 | 1 | 6/24 |
+| treatment, draw 1 | 11 | 8 | 0 | 0 | 5 | 8 | 7/24 |
+| treatment, draw 2 | 11 | 7 | 1 | 1 | 4 | 9 | 8/24 |
+
+Draw 2 of each arm carries the per-finding rows (`pr-399-control-rows.jsonl`,
+`pr-399-treatment-rows.jsonl`); the first draws predate that output and survive only as the
+aggregates above.
+
+### The mechanism works exactly where it was aimed
+
+Six findings claimed a mypy rule the repository does not have. **All six were refuted, and
+all six cite the checker output.** The rationale on finding 0 is the RFC's hypothesis
+verbatim:
+
+> The project's mypy --strict checker, run as part of the review process, reports 'Success:
+> no issues found', directly contradicting the claim that the use of `Any` is prohibited in
+> strict mypy mode for this code.
+
+That answers open question 1. The failure was **capability, not disposition**: shown the
+output, the skeptic withdraws the claim; not shown it, it confirms.
+
+### The one true finding was withdrawn
+
+Finding 4 — `git diff` can fail in a way other than the promised `MissingFixtureError`, the
+finding that produced the actual fix in #399 — was refuted, **without citing any checker**:
+
+> the code explicitly checks the validity of `base` and `head` SHAs using `git rev-parse`
+> before the `git diff` call … thus the specific failure mode is already handled
+
+which reproduces the half of the argument that `diff_for`'s own comment rejects: *"unlikely"
+is not the same as "cannot"*.
+
+| | baseline | treatment |
+|---|---:|---:|
+| false findings confirmed | 18 | 12 |
+| true findings confirmed | 1 | **0** |
+
+### What the control arm bought, and how it reversed the first reading
+
+The aggregate alone suggested the loss of finding 4 was noise: no checker was cited on it,
+and control draw 1 produced two `confirmed → refuted` flips of its own. **The per-finding
+rows say otherwise.** In the control draw that carries rows, finding 4 was *confirmed*, and
+the rationale is correct:
+
+> The `subprocess.run` call for `git diff` includes `check=True`, but the
+> `CalledProcessError` it raises on command failure is not explicitly caught, leading to a
+> different error type than `MissingFixtureError` …
+
+That is the defect, stated accurately, by the same model that withdrew it once evidence was
+in the prompt. The single control flip was finding 6, not finding 4.
+
+So the loss is **on the evidence arm's side of the ledger**, and the earlier "not
+attributable" reading — published in #406 and in the #401 comment before these rows existed —
+was wrong. It rested on an aggregate, which is exactly the shape of number that cannot
+answer *which*.
+
+The two arms separate cleanly per finding:
+
+| class | control | treatment |
+|---|---|---|
+| 6 false mypy claims | all confirmed | all refuted, all citing the checker |
+| the 1 true finding | **confirmed, correctly reasoned** | **refuted**, citing no checker |
+
+The mechanism on the false claims is visible in the control's own words: it confirmed
+finding 0 on a conditional it could not check — *"**if** `mypy --strict` is configured to
+disallow `Any` explicitly, it would be flagged"*. Evidence turns that hypothetical into a
+reading, which is precisely the RFC's design intent.
+
+Why the true finding falls is not explained by the same mechanism, because its refutation
+cites nothing. The unattributed moves are 3 in treatment against 1 in control, which is
+consistent with the evidence section shifting the skeptic toward refutation generally rather
+than only where the output bears — but at these counts that is a hypothesis, not a result.
+
+Two draws of the same arm differ in both arms (control 4 vs 1 flips, treatment 8 vs 9), so
+single-draw asymmetries are not results.
+
+## What this does not establish
+
+- **The recall denominator is one.** One review, one substantive finding. "Recall went to
+  zero" is literally true and statistically nearly weightless.
+- **Attribution by rationale text is a weak proxy.** Citations barely moved between arms
+  (6/24 control vs 7–8/24 treatment) while verdicts moved a great deal — and the rows show
+  why: the control cites a checker on the *same six findings*, to confirm them.
+- **Only one draw per arm has rows.** Whether finding 4 survives the control reliably or was
+  lucky in this draw is unmeasured; control draw 1 had two unidentified `confirmed → refuted`
+  flips.
+- **Six of 24 control rationales cite mypy or ruff although no checker output was supplied**
+  (#407). Not merely fabricated: fabricated *in support of the wrong verdict*. The same six
+  findings carry a checker citation in both arms, confirmed in one and refuted in the other,
+  and only one of the two had a checker.
+
+## Reproducing
+
+```bash
+scripts/guardian_replay_skeptic.py --recording pr-399-recording.json \
+    --control --out control-rows.jsonl
+scripts/guardian_replay_skeptic.py --recording pr-399-recording.json \
+    --at 05af753d8895eab00609e7e58d89aa72ebd1777d --out treatment-rows.jsonl
+```
+
+Requires `GUARDIAN_SKEPTIC=gemini`, `GUARDIAN_SKEPTIC_MODEL=gemini-2.5-flash` and the
+matching key. 24 skeptic calls per arm, no finder calls. The recording is committed here
+because the GitHub artifact it came from expires after seven days.

codegraph_brain-0.16.0/benchmarks/guardian/experiments/401-evidence/pr-399-control-rows.jsonl

@@ -0,0 +1,24 @@
+{"index": 0, "file": "scripts/recordings_from_corpus.py", "line": 38, "title": "Use of `Any` in type annotations violates strict mypy mode", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The use of `Any` does indeed bypass strict type checking in mypy, and while common for dynamic data structures (like those parsed from JSON/YAML), it generally indicates a potential for more precise typing or schema definition (e.g., Pydantic models). This is a plausible area for improvement, and if `mypy --strict` is configured to disallow `Any` explicitly, it would be flagged."}
+{"index": 1, "file": "scripts/recordings_from_corpus.py", "line": 41, "title": "Unsafe YAML loading without `safe_load` or `Loader` specification", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The quoted code on line 100 (not 41 as stated) correctly uses `yaml.safe_load`. However, the `import yaml` statement does allow for `yaml.load` to be used elsewhere in the file, which is an unsafe practice without specifying a `Loader`. This is a plausible concern for future maintainability and security, as a developer might accidentally introduce an unsafe load later. The claim is plausible and cannot be disproven."}
+{"index": 2, "file": "scripts/recordings_from_corpus.py", "line": 50, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code accurately shows `Any` used in the type annotation, which strict mypy typically flags, aligning with the project's policy to score such issues low when covered by tooling."}
+{"index": 3, "file": "scripts/recordings_from_corpus.py", "line": 56, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code is present. While `mypy --strict` does not inherently disallow explicit `Any` without `disallow_any_explicit`, using `dict[str, Any]` weakens type safety for the `row` parameter, making it a legitimate type-hinting concern that is not 'already covered' by the standard `mypy --strict` checks."}
+{"index": 4, "file": "scripts/recordings_from_corpus.py", "line": 62, "title": "Empty diff raises `MissingFixtureError` but may not be the only failure mode", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `subprocess.run` call for `git diff` includes `check=True`, but the `CalledProcessError` it raises on command failure is not explicitly caught, leading to a different error type than `MissingFixtureError` which is used for other unresolvable diff scenarios."}
+{"index": 5, "file": "scripts/recordings_from_corpus.py", "line": 80, "title": "Function parameter uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code on line 80 uses `dict[str, Any]` for the `row` parameter, which is correctly identified as a violation by `mypy --disallow-any-type` (part of `--strict`)."}
+{"index": 6, "file": "scripts/recordings_from_corpus.py", "line": 89, "title": "Potential type mismatch in `Finding.model_validate`", "was": "confirmed", "now": "refuted", "cites_a_checker": false, "rationale": "The `Finding.model_validate` method is precisely designed to enforce the schema and will raise a `ValidationError` for type mismatches or unexpected keys (if configured with `extra='forbid'`), or silently drop extra keys (if configured with `extra='ignore'`). The dictionary comprehension correctly strips known annotations before validation, so the code already handles the claimed problem appropriately through Pydantic's intended mechanisms. A `ValidationError` indicates a problem with the input data, not with the validation call itself. Using `model_construct` would bypass essential validation."}
+{"index": 7, "file": "scripts/recordings_from_corpus.py", "line": 95, "title": "No validation that `results` file exists or is readable", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The `results.read_text()` call will raise a `FileNotFoundError` or other `IOError` if the file does not exist or is unreadable, preventing the function from silently proceeding without error in those specific cases."}
+{"index": 8, "file": "scripts/recordings_from_corpus.py", "line": 100, "title": "No handling for case where no frozen passes are found", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The script correctly handles the scenario where no frozen passes are found by writing zero recordings. The `main` function explicitly reports \"0 recordings written\", which clearly communicates the outcome to the caller, making the behavior expected and not an error. A separate warning is not required for correctness or clarity."}
+{"index": 9, "file": "scripts/recordings_from_corpus.py", "line": 102, "title": "Race condition in directory creation", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The `exist_ok=True` argument to `Path.mkdir()` explicitly handles the scenario where the directory already exists, preventing `FileExistsError` and making the operation safe for concurrent calls without a race condition."}
+{"index": 10, "file": "scripts/recordings_from_corpus.py", "line": 110, "title": "No validation that `save_finder_recording` succeeds", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `build` function does not handle potential failures of `save_finder_recording` (e.g., due to permission issues or disk full), which would either cause the script to crash, or, if `save_finder_recording` fails silently, lead to incorrectly reporting a recording as written when it was not."}
+{"index": 11, "file": "scripts/recordings_from_corpus.py", "line": 117, "title": "No error handling for `build` failures", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `main` function does not wrap the call to `build` in a try-except block, confirming that exceptions from `build` will propagate and exit the script with a traceback instead of a more user-friendly error message. This impacts script robustness for the end-user."}
+{"index": 12, "file": "tests/unit/test_recordings_from_corpus.py", "line": 12, "title": "Modification of `sys.path` at runtime violates type safety", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code accurately reflects the diff. Modifying sys.path at runtime is a known pattern that can lead to less predictable import resolution and complicates static analysis tools like MyPy, which is plausibly what the reviewer meant by 'not type-safe.' It's not a critical bug but a maintainability concern."}
+{"index": 13, "file": "tests/unit/test_recordings_from_corpus.py", "line": 15, "title": "Import from a script file may fail in some environments", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The claim that the import may fail due to the file not being in `sys.path` is refuted, as line 13 explicitly adds the containing directory to `sys.path`. The claim regarding environments not supporting direct `.py` imports is not generally applicable to functional Python environments. While importing from a script file via `sys.path` manipulation can be an architectural concern, the specific failure conditions stated in the claim are either handled or not plausible."}
+{"index": 14, "file": "tests/unit/test_recordings_from_corpus.py", "line": 20, "title": "Hardcoded path traversal may break in some environments", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The code uses a chain of `.parent` calls to infer the repository root, which is indeed a hardcoded path traversal dependent on the file's exact location (`tests/unit/`), making the claim accurate and a minor robustness concern."}
+{"index": 15, "file": "tests/unit/test_recordings_from_corpus.py", "line": 28, "title": "Function return type uses `Any` in type annotation", "was": "confirmed", "now": "confirmed", "cites_a_checker": true, "rationale": "The quoted code uses `Any` in the return type annotation as claimed. I cannot refute the assertion that this violates a strict mypy rule; if true, it would be caught by mandatory project tooling, leading to a low impact score."}
+{"index": 16, "file": "tests/unit/test_recordings_from_corpus.py", "line": 30, "title": "No validation that `RESULTS` file exists or is readable", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `_rows` function directly attempts to read `RESULTS.read_text()` without checking for the file's existence, which will indeed raise a `FileNotFoundError` if `results.jsonl` is absent, causing the tests to fail. The claim accurately describes this concrete failure path in the test utility."}
+{"index": 17, "file": "tests/unit/test_recordings_from_corpus.py", "line": 35, "title": "Return type uses `Path` but may include non-Path values", "was": "refuted", "now": "refuted", "cites_a_checker": false, "rationale": "The original reviewer's analysis concludes, 'No action needed; the type is correct. This is a false positive.' This explicitly refutes the claim of a defect, as the type annotation `dict[str, Path]` accurately reflects the dictionary values being `Path` objects."}
+{"index": 18, "file": "tests/unit/test_recordings_from_corpus.py", "line": 50, "title": "Test assumes existence of a judged row in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The `next()` call without a default value will raise a `StopIteration` error if the `_rows()` generator yields no matching elements, making the test brittle to changes in the `results.jsonl` corpus data."}
+{"index": 19, "file": "tests/unit/test_recordings_from_corpus.py", "line": 57, "title": "Test assumes existence of an unscored row in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The use of `next()` without a default value or preceding check means the test will indeed raise `StopIteration` if the external corpus `results.jsonl` happens to lack a row matching the specified criteria, making the test brittle."}
+{"index": 20, "file": "tests/unit/test_recordings_from_corpus.py", "line": 64, "title": "Test uses a hardcoded PR number (9999) that may not be meaningful", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The test correctly expects a MissingFixtureError for PR 9999, which relies on PR 9999 not having an associated ground-truth fixture. If PR 9999 were to be created and a fixture added in the future, the test would fail, making it fragile."}
+{"index": 21, "file": "tests/unit/test_recordings_from_corpus.py", "line": 75, "title": "Test assumes at least 72 frozen passes exist in the corpus", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The assertion explicitly hardcodes a minimum of 72 frozen passes. If the corpus evolves to contain fewer than 72, the test will fail, regardless of whether the `build` function and related logic are actually correct for the current corpus size, making the test brittle."}
+{"index": 22, "file": "tests/unit/test_recordings_from_corpus.py", "line": 100, "title": "Test assumes existence of `ambiguous_hits` in some rows", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The test explicitly asserts that the rows requiring the pre-#345 precision rule are exactly those carrying `ambiguous_hits`, as stated in the claim and the test's accompanying comments. This is an intentional check to ensure consistency between the corpus data and its documented scoring behavior, meaning a failure would correctly flag an issue with the benchmark data, not necessarily a defect in the test's logic. However, it indicates a tight coupling of the test to a specific data property, which implies a potential maintenance burden if this property of the corpus changes, even for legitimate reasons."}
+{"index": 23, "file": "tests/unit/test_recordings_from_corpus.py", "line": 145, "title": "Floating-point comparison may be too strict for some use cases", "was": "confirmed", "now": "confirmed", "cites_a_checker": false, "rationale": "The claim is plausible because an absolute tolerance of 1e-9, while strict, can lead to false negatives in floating-point comparisons if calculations accumulate tiny errors or if the reference values themselves have slightly different precision than the re-calculated ones. Using math.isclose with both relative and absolute tolerances is generally a more robust approach in such test scenarios to avoid brittleness and potential flakiness."}