4take 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (129) hide show
  1. 4take-0.2.0/.gitignore +65 -0
  2. 4take-0.2.0/.pre-commit-config.yaml +6 -0
  3. 4take-0.2.0/AGENTS.md +241 -0
  4. 4take-0.2.0/CHANGELOG.md +494 -0
  5. 4take-0.2.0/CLA.md +63 -0
  6. 4take-0.2.0/CLAUDE.md +99 -0
  7. 4take-0.2.0/CONTRIBUTING.md +93 -0
  8. 4take-0.2.0/LICENSE +145 -0
  9. 4take-0.2.0/PKG-INFO +902 -0
  10. 4take-0.2.0/POST_MORTEMS.md +53 -0
  11. 4take-0.2.0/README.md +866 -0
  12. 4take-0.2.0/ROADMAP.md +91 -0
  13. 4take-0.2.0/SECURITY.md +45 -0
  14. 4take-0.2.0/SKILL.md +54 -0
  15. 4take-0.2.0/docs/adr/0001-use-exponential-backoff-with-10-retries.md +31 -0
  16. 4take-0.2.0/docs/adr/0002-cap-retries-at-four-and-honour-retry-after.md +66 -0
  17. 4take-0.2.0/docs/adr/0003-retry-a-rate-limited-provider-eight-times.md +44 -0
  18. 4take-0.2.0/evals/README.md +22 -0
  19. 4take-0.2.0/evals/backfill_duplicates.py +74 -0
  20. 4take-0.2.0/evals/corpus/async_gather_swallow.py +5 -0
  21. 4take-0.2.0/evals/corpus/auth_jwt_decode.js +10 -0
  22. 4take-0.2.0/evals/corpus/auth_timing_compare.py +7 -0
  23. 4take-0.2.0/evals/corpus/clean_lru.py +24 -0
  24. 4take-0.2.0/evals/corpus/clean_slugify.py +16 -0
  25. 4take-0.2.0/evals/corpus/cmdinjection_backup.py +5 -0
  26. 4take-0.2.0/evals/corpus/cmdinjection_git.js +5 -0
  27. 4take-0.2.0/evals/corpus/divzero_average_latency.py +7 -0
  28. 4take-0.2.0/evals/corpus/leak_db_connection.py +8 -0
  29. 4take-0.2.0/evals/corpus/leak_file_handle.py +10 -0
  30. 4take-0.2.0/evals/corpus/leak_subprocess.py +8 -0
  31. 4take-0.2.0/evals/corpus/none_env_parse.py +6 -0
  32. 4take-0.2.0/evals/corpus/none_json_response.js +8 -0
  33. 4take-0.2.0/evals/corpus/none_lookup_chain.py +7 -0
  34. 4take-0.2.0/evals/corpus/offbyone_pagination.py +8 -0
  35. 4take-0.2.0/evals/corpus/offbyone_retry_window.js +11 -0
  36. 4take-0.2.0/evals/corpus/offbyone_ring_buffer.py +12 -0
  37. 4take-0.2.0/evals/corpus/overflow_backoff.py +9 -0
  38. 4take-0.2.0/evals/corpus/pathtraversal_download.py +10 -0
  39. 4take-0.2.0/evals/corpus/pathtraversal_zip.py +6 -0
  40. 4take-0.2.0/evals/corpus/race_async_cache.py +12 -0
  41. 4take-0.2.0/evals/corpus/race_counter_increment.py +13 -0
  42. 4take-0.2.0/evals/corpus/race_lazy_singleton.py +8 -0
  43. 4take-0.2.0/evals/corpus/sqli_order_by.py +9 -0
  44. 4take-0.2.0/evals/corpus/sqli_user_search.py +8 -0
  45. 4take-0.2.0/evals/corpus/toctou_config_write.py +10 -0
  46. 4take-0.2.0/evals/corpus/toctou_upload_quota.py +8 -0
  47. 4take-0.2.0/evals/corpus/unbounded_memory_tail.py +10 -0
  48. 4take-0.2.0/evals/corpus_ctx/cache_key_version/context/profiles/events.py +6 -0
  49. 4take-0.2.0/evals/corpus_ctx/cache_key_version/review.diff +15 -0
  50. 4take-0.2.0/evals/corpus_ctx/config_key_mismatch/context/app/settings.py +5 -0
  51. 4take-0.2.0/evals/corpus_ctx/config_key_mismatch/review.diff +18 -0
  52. 4take-0.2.0/evals/corpus_ctx/ctx_clean_locked_by_caller/context/cache/lru.py +30 -0
  53. 4take-0.2.0/evals/corpus_ctx/ctx_clean_locked_by_caller/review.diff +11 -0
  54. 4take-0.2.0/evals/corpus_ctx/ctx_clean_validated_upstream/context/billing/coupons.py +13 -0
  55. 4take-0.2.0/evals/corpus_ctx/ctx_clean_validated_upstream/review.diff +9 -0
  56. 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/context/accounts/identity.py +7 -0
  57. 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/context/accounts/login.py +6 -0
  58. 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/review.diff +17 -0
  59. 4take-0.2.0/evals/corpus_ctx/event_renamed/context/src/notifications/subscribers.js +6 -0
  60. 4take-0.2.0/evals/corpus_ctx/event_renamed/review.diff +9 -0
  61. 4take-0.2.0/evals/corpus_ctx/lock_not_taken/context/limits/counter.py +31 -0
  62. 4take-0.2.0/evals/corpus_ctx/lock_not_taken/review.diff +12 -0
  63. 4take-0.2.0/evals/corpus_ctx/missing_await/context/web/sessions.py +8 -0
  64. 4take-0.2.0/evals/corpus_ctx/missing_await/review.diff +9 -0
  65. 4take-0.2.0/evals/corpus_ctx/nested_commit/context/orders/lines.py +10 -0
  66. 4take-0.2.0/evals/corpus_ctx/nested_commit/review.diff +21 -0
  67. 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/context/app/api.py +9 -0
  68. 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/context/app/mailer.py +11 -0
  69. 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/review.diff +10 -0
  70. 4take-0.2.0/evals/corpus_ctx/shell_helper_injection/context/deploy/git_tools.py +8 -0
  71. 4take-0.2.0/evals/corpus_ctx/shell_helper_injection/review.diff +15 -0
  72. 4take-0.2.0/evals/corpus_ctx/swapped_positional_args/context/src/billing.js +12 -0
  73. 4take-0.2.0/evals/corpus_ctx/swapped_positional_args/review.diff +14 -0
  74. 4take-0.2.0/evals/corpus_ctx/tuple_as_bool/context/shop/payments.py +9 -0
  75. 4take-0.2.0/evals/corpus_ctx/tuple_as_bool/review.diff +12 -0
  76. 4take-0.2.0/evals/corpus_ctx/units_seconds_to_ms/context/jobs/watchdog.py +10 -0
  77. 4take-0.2.0/evals/corpus_ctx/units_seconds_to_ms/review.diff +12 -0
  78. 4take-0.2.0/evals/labels.json +515 -0
  79. 4take-0.2.0/evals/labels_ctx.json +158 -0
  80. 4take-0.2.0/evals/record.py +315 -0
  81. 4take-0.2.0/evals/score.py +581 -0
  82. 4take-0.2.0/llms.txt +34 -0
  83. 4take-0.2.0/pyproject.toml +129 -0
  84. 4take-0.2.0/src/fourtake/__init__.py +1 -0
  85. 4take-0.2.0/src/fourtake/autocontext.py +1161 -0
  86. 4take-0.2.0/src/fourtake/cache.py +201 -0
  87. 4take-0.2.0/src/fourtake/cli.py +474 -0
  88. 4take-0.2.0/src/fourtake/client.py +474 -0
  89. 4take-0.2.0/src/fourtake/config.py +198 -0
  90. 4take-0.2.0/src/fourtake/context.py +331 -0
  91. 4take-0.2.0/src/fourtake/discovery.py +413 -0
  92. 4take-0.2.0/src/fourtake/engine.py +645 -0
  93. 4take-0.2.0/src/fourtake/filelock.py +92 -0
  94. 4take-0.2.0/src/fourtake/findings.py +652 -0
  95. 4take-0.2.0/src/fourtake/input_kind.py +74 -0
  96. 4take-0.2.0/src/fourtake/mcp_entry.py +31 -0
  97. 4take-0.2.0/src/fourtake/mcp_server.py +305 -0
  98. 4take-0.2.0/src/fourtake/output.py +279 -0
  99. 4take-0.2.0/src/fourtake/preflight.py +102 -0
  100. 4take-0.2.0/src/fourtake/prompts.py +136 -0
  101. 4take-0.2.0/src/fourtake/rate_limiter.py +125 -0
  102. 4take-0.2.0/src/fourtake/safety.py +268 -0
  103. 4take-0.2.0/src/fourtake/telemetry.py +220 -0
  104. 4take-0.2.0/src/fourtake/tokenizer.py +23 -0
  105. 4take-0.2.0/tests/conftest.py +33 -0
  106. 4take-0.2.0/tests/test_autocontext.py +355 -0
  107. 4take-0.2.0/tests/test_cache.py +239 -0
  108. 4take-0.2.0/tests/test_cli.py +662 -0
  109. 4take-0.2.0/tests/test_client.py +448 -0
  110. 4take-0.2.0/tests/test_config.py +150 -0
  111. 4take-0.2.0/tests/test_context.py +333 -0
  112. 4take-0.2.0/tests/test_council_scheduling.py +724 -0
  113. 4take-0.2.0/tests/test_discovery.py +314 -0
  114. 4take-0.2.0/tests/test_docs.py +67 -0
  115. 4take-0.2.0/tests/test_engine.py +555 -0
  116. 4take-0.2.0/tests/test_evals.py +490 -0
  117. 4take-0.2.0/tests/test_findings.py +893 -0
  118. 4take-0.2.0/tests/test_input_kind.py +116 -0
  119. 4take-0.2.0/tests/test_integration.py +242 -0
  120. 4take-0.2.0/tests/test_mcp_server.py +651 -0
  121. 4take-0.2.0/tests/test_output.py +202 -0
  122. 4take-0.2.0/tests/test_output_contract.py +833 -0
  123. 4take-0.2.0/tests/test_preflight.py +106 -0
  124. 4take-0.2.0/tests/test_prompts.py +200 -0
  125. 4take-0.2.0/tests/test_rate_limiter.py +162 -0
  126. 4take-0.2.0/tests/test_safety.py +321 -0
  127. 4take-0.2.0/tests/test_telemetry.py +257 -0
  128. 4take-0.2.0/tests/test_tokenizer.py +52 -0
  129. 4take-0.2.0/tests/test_zero_cost_enforcement.py +103 -0
4take-0.2.0/.gitignore ADDED
@@ -0,0 +1,65 @@
1
+ # Python bytecode and cache
2
+ __pycache__/
3
+ *.py[cod]
4
+ *$py.class
5
+ *.so
6
+
7
+ # Distribution and packaging
8
+ build/
9
+ dist/
10
+ *.egg-info/
11
+ *.egg
12
+ .eggs/
13
+ wheels/
14
+ *.whl
15
+
16
+ # Virtual environments
17
+ .venv/
18
+ venv/
19
+ env/
20
+ ENV/
21
+
22
+ # Testing and coverage
23
+ .pytest_cache/
24
+ .coverage
25
+ .coverage.*
26
+ coverage.xml
27
+ htmlcov/
28
+ coverage_report.txt
29
+
30
+ # Audit and temporary scratch files
31
+ audit_*.txt
32
+ audit_report.md
33
+ review.json
34
+ scratch.py
35
+ test_audit.py
36
+ test_or.py
37
+ 4take_raw_data.json
38
+
39
+ # Local config & state cache
40
+ .cache/
41
+ evals/recordings_backup/
42
+
43
+ # OS and IDE files
44
+ .DS_Store
45
+ .DS_Store?
46
+ ._*
47
+ .Spotlight-V100
48
+ .Trashes
49
+ ehthumbs.db
50
+ Thumbs.db
51
+ .idea/
52
+ .vscode/
53
+ *.swp
54
+ *.swo
55
+ *~
56
+
57
+ # Evals built from private repositories: their code must never reach this repository
58
+ evals/private/
59
+
60
+ # Third-party benchmark data, fetched for scoring and not redistributed
61
+ evals/external/
62
+
63
+ # The frontier test's answer key and blind runs: an agent under test that can read truth.json is
64
+ # no longer blind. prereg.sha256 fixes the corpus and the key before any run.
65
+ evals/frontier/
@@ -0,0 +1,6 @@
1
+ repos:
2
+ - repo: https://github.com/commitizen-tools/commitizen
3
+ rev: v3.13.0
4
+ hooks:
5
+ - id: commitizen
6
+ stages: [commit-msg]
4take-0.2.0/AGENTS.md ADDED
@@ -0,0 +1,241 @@
1
+ # AGENTS.md — AI Agent Integration Guide for 4take
2
+
3
+ This guide specifies how autonomous AI coding agents (Claude Code, Cursor Composer, Antigravity IDE, Windsurf, Codex, Devin, Cline, Roo Code, Aider) should interact with `4take`.
4
+
5
+ ---
6
+
7
+ ## 🎯 Purpose of 4take for AI Agents
8
+
9
+ When generating complex algorithms, database migrations, security-critical logic, or large refactors, a single LLM is prone to subtle blindspots and confirmation bias.
10
+
11
+ `4take` provides you (the agent) with an independent **Council of 4 LLMs** to audit your code in parallel before you present it to the user.
12
+
13
+ Whether the council improves what a given agent ships has not been measured yet. Treat its output as leads to verify, not verdicts; step 2 of the synthesis framework below gives the measured hit rates.
14
+
15
+ The council is told its input was written by a capable model, so it is not looking for syntax
16
+ or typos. It reviews four axes: **correctness**, **security**, **structure** (responsibility in
17
+ the wrong layer, coupling that spreads one change across several files, logic duplicated where
18
+ it will drift) and **simplicity** (indirection with one implementation, a hand-rolled stdlib).
19
+ A structural finding is only reported when the model can name its concrete cost.
20
+
21
+ It reviews code. Piping in a design question or a plan is refused before any request goes out;
22
+ pass `--force-review` (CLI) or `force=true` (MCP) to override.
23
+
24
+ ---
25
+
26
+ ## 🛠️ Integration Methods
27
+
28
+ ### Method 1: FastMCP Server (`get_4take_opinion`)
29
+
30
+ If your agent runtime supports the **Model Context Protocol (MCP)**:
31
+
32
+ - **Server Entrypoint:** `4take-mcp` (or `uvx --from '4take[mcp]' 4take-mcp`). The MCP
33
+ server ships as the `mcp` extra: `pip install '4take[mcp]'`.
34
+ - **Server output:** first `report_for_user`, a list of lines to show the user verbatim or close
35
+ to it: a header with the [calibration](#-the-agent-as-chairman-4-step-synthesis-framework) and
36
+ what follows from it (agreement 3 or more: confirm in the code, then fix; 1 or 2: check against
37
+ the code before accepting or rejecting it), then one line per `consensus` entry, most-agreed
38
+ first, such as ``- 2/4 WARNING `page_size` — Unbounded limit``. The denominator is the number
39
+ of independent reviewers that answered. Show every line, the ones you go on to reject included:
40
+ the user sees only what you relay. Then the same `consensus` list the CLI emits, plus `status`
41
+ (`ok` / `degraded` / `error`), `distinct_models` and `min_responses`. `degraded`
42
+ means fewer independent reviewers answered than were asked for; a `warning` alongside
43
+ `status: ok` is informational, not a failure. Only a review fills a seat: an answer with
44
+ neither a finding nor `NO_ISSUES_FOUND` (a tool call, garbage tokens, reasoning that never
45
+ reached a tag) is counted as a failed model and replaced by a spare.
46
+ - **Tool Name:** `get_4take_opinion`
47
+ - **Arguments:**
48
+ - `code` (string, required): The source code, git diff, or implementation to review.
49
+ - `force` (boolean, optional, default `false`): Send the input even though it does not look
50
+ like code. Without it, prose is refused rather than dispatched.
51
+ - `context_files` (list of strings, optional): Paths of code related to the change, which the
52
+ reviewers read but do not review. The server reads them from disk, under the project root,
53
+ so you do not paste them. `path:START-END` sends only those lines. See
54
+ [Choosing the context](#-choosing-the-context-the-chairmans-first-job).
55
+ - `context` (string, optional): The same kind of related code as text, when the server cannot
56
+ read your disk.
57
+ - `auto_context` (boolean, optional, default `false`): With a diff, also let the server choose
58
+ context by name: what still uses what the change removes, the callers of what it edits, the
59
+ definitions of what it calls. Sent after `context_files`, within the same budget.
60
+ - **Return Value:** JSON string: `report_for_user` first, then the fields above and the
61
+ `responses` array with each model's independent critique and metadata.
62
+
63
+ ---
64
+
65
+ ### Method 2: Shell / CLI Subprocess
66
+
67
+ If invoking `4take` via bash / shell commands:
68
+
69
+ ```bash
70
+ # Recommended command for agents:
71
+ cat path/to/file.py | 4take --format json --accept-risk --quiet
72
+ ```
73
+
74
+ #### CLI Flags for Agents:
75
+ - `--format json`: Ensures machine-readable JSON output on stdout instead of ANSI Markdown.
76
+ - `--accept-risk`: Bypasses terminal privacy prompts (or set `FOURTAKE_ACCEPT_RISK=1`).
77
+ - `--quiet`: Suppresses live progress logs and diagnostics.
78
+ - `--min-responses N`: (Optional) Minimum successful model responses required (default: 4).
79
+ - `--per-model-timeout N`: (Optional) Abandon a single model after N seconds and rotate in a
80
+ replacement. Defaults to half of `--timeout` (minimum 60s).
81
+ - `--force-review`: (Optional) Review an input that does not look like source code. Without
82
+ it, prose is refused with exit code 1 instead of spending four requests to say nothing.
83
+ - `--strict`: (Optional, usually leave off) Narrow `consensus` to findings more than one reviewer
84
+ raised, plus any `CRITICAL` one. Step 3 below needs the single-reviewer entries it drops.
85
+ - `--context-file PATH[:START-END]`: (Optional, repeatable, most relevant first) Related code the
86
+ council reads but does not review. Read only under the working directory. See
87
+ [Choosing the context](#-choosing-the-context-the-chairmans-first-job).
88
+ - `--auto-context`: (Optional) With a diff, also choose context by name, after any
89
+ `--context-file`. `summary.context.auto` says why each excerpt was chosen.
90
+ - `--context-budget N`: (Optional) Cap on reference context in tokens, default 16000. A file that
91
+ does not fit is left out whole and named in `summary.context.dropped`.
92
+ - `--dry-run`: Print exactly what would be sent, after redaction, and exit without contacting
93
+ any model.
94
+ - `--no-cache` / `--clear-cache`: Bypass or purge the local 24h review cache.
95
+ - `--no-telemetry` / `--clear-telemetry`: Skip or erase the local record of model latency and
96
+ success rate. It stores model ids and timings only — never code, reviews or prompts — and
97
+ is what ranks the model pool on evidence instead of on model names.
98
+
99
+ **stdout carries only the JSON payload.** All logs, progress and warnings go to stderr, so
100
+ `json.loads(stdout)` is safe.
101
+
102
+ #### Exit Codes
103
+ | Code | Meaning |
104
+ | :--- | :--- |
105
+ | `0` | The council met `--min-responses`. |
106
+ | `2` | Partial: at least one review, but fewer than `--min-responses`. Check `warning`. |
107
+ | `1` | Total failure: no model responded. Check `error`. |
108
+
109
+ A large diff can take up to `--timeout` (600 s) and often ends with exit code `2` and fewer than
110
+ four reviewers: few free models reason through one to an answer. A partial council is still a
111
+ usable review; read `agreement` against `summary.distinct_models`.
112
+
113
+ > **Free models only**: `4take` sends requests only to OpenRouter's free models (`:free` tier) and rejects a paid model before any request goes out. Do not pass `--models` with paid models. Free models are rate-limited: roughly 4 to 8 reviews a day on an OpenRouter account that has never been topped up, roughly 80 to 160 once it has. Do not call `4take` in a loop.
114
+
115
+ ---
116
+
117
+ ## 📜 Output JSON Structure
118
+
119
+ ```json
120
+ {
121
+ "version": "0.2.0",
122
+ "timestamp": "2026-08-16T14:20:00Z",
123
+ "summary": {
124
+ "task_hash": "25d297f773c3",
125
+ "responses_received": 4,
126
+ "responses_failed": 0,
127
+ "total_latency_ms": 3200,
128
+ "context": {"files": ["src/mailer.py", "src/users.py (lines 40-90)"], "tokens": 812, "dropped": []}
129
+ },
130
+ "warning": null,
131
+ "error": null,
132
+ "consensus": [
133
+ {
134
+ "agreement": 2,
135
+ "severity": "CRITICAL",
136
+ "category": "INJECTION",
137
+ "symbol": "search_users",
138
+ "title": "SQL injection via the term parameter",
139
+ "models": ["cohere/north-mini-code", "nvidia/nemotron-3-ultra-550b-a55b"],
140
+ "severities": {"CRITICAL": 1, "WARNING": 1},
141
+ "disputed": true
142
+ }
143
+ ],
144
+ "responses": [
145
+ {
146
+ "model": "openrouter/openrouter/free",
147
+ "served_model": "nvidia/nemotron-3-ultra-550b-a55b",
148
+ "served_provider": "NVIDIA",
149
+ "content": "Review text from model...",
150
+ "finish_reason": "stop",
151
+ "truncated": false,
152
+ "duplicate_of": null,
153
+ "latency_ms": 1150
154
+ },
155
+ {
156
+ "model": "openrouter/cohere/north-mini-code:free",
157
+ "served_model": "cohere/north-mini-code",
158
+ "served_provider": "Cohere",
159
+ "content": "Review text from model...",
160
+ "finish_reason": "stop",
161
+ "truncated": false,
162
+ "latency_ms": 1420
163
+ }
164
+ ],
165
+ "errors": [
166
+ {
167
+ "model": "failing/model",
168
+ "error": "Reason for error",
169
+ "latency_ms": 200
170
+ }
171
+ ]
172
+ }
173
+ ```
174
+
175
+ ---
176
+
177
+ ## 🧭 Choosing the Context (the Chairman's first job)
178
+
179
+ The council sees only what you send. A diff alone hides every defect that lives between files: a
180
+ changed return value a caller still relies on, a route without the check its neighbours make, a
181
+ helper duplicated under another name. You have already read the surrounding code, so pass it:
182
+
183
+ 1. Each changed function in full: `git diff -W` includes it in the diff itself.
184
+ 2. Its direct callers, one hop away (grep for the changed names), most relevant first.
185
+ 3. The functions it calls, and the types, schemas and config it relies on.
186
+ 4. Related tests, when they encode the contract.
187
+
188
+ A few files or line ranges, not the repository: the budget is 16000 tokens, and whole files are
189
+ dropped past it. Prefer paths (`context_files`, `--context-file`) to pasted text, so you spend
190
+ no output tokens on them. Findings stay on the reviewed code; a caller the change breaks is named
191
+ in the finding's Issue. `summary.context` (CLI) or `context` (MCP) lists what was sent and what
192
+ the budget dropped. A finding that needed a caller you did not send is one the council could not
193
+ make.
194
+
195
+ `auto_context` (`--auto-context`) does steps 2 and 3 the way grep would: callers of the edited
196
+ functions that import their module, remaining uses of a removed name or string value, definitions
197
+ of what the new lines call, and the rest of an edited class outside the diff. It cannot know what
198
+ you know, so it comes after the files you name; name the ones that matter and let it add the rest.
199
+ Check `context.auto` for what it chose and why. On the cross-file eval corpus it chose the same
200
+ files as a careful chairman, and the council named 12 of 12 cross-file defects in each of three
201
+ recordings with it, against 0 or 1 with the diff alone.
202
+
203
+ ---
204
+
205
+ ## 🧠 The Agent as Chairman: 4-Step Synthesis Framework
206
+
207
+ When you receive the council's reviews:
208
+
209
+ 0. **Check `truncated`**: a response with `truncated: true` was cut off by the output budget
210
+ and is incomplete — weigh it accordingly, and do not treat its silence as "no issues".
211
+ 1. **Read `consensus` first**: the findings are already grouped across reviewers and ranked
212
+ by `agreement`. It is keyed on `served_model`, not `model`, so a router seating the same
213
+ underlying model twice contributes one voice rather than manufacturing agreement. A council
214
+ also seats one model per vendor while another vendor can answer; a response with
215
+ `duplicate_of` set shadows an existing seat and is not counted — if you re-derive agreement
216
+ from `responses`, skip those.
217
+ `responses[i].content` is still there when you need a finding's full wording.
218
+ `category` is the class the reviewers declared, and `UNKNOWN` when none did — it is read
219
+ from the reviews, never guessed, so an `UNKNOWN` says nothing about the finding itself.
220
+ Reviewers file one defect under the same category only about half the time: treat it as
221
+ a label, never as evidence that two findings are, or are not, the same defect.
222
+ 2. **Weigh `agreement` as measured**: over three recordings of the eval corpus, every council
223
+ seating four reviewers, this is how often a consensus entry held a labelled defect:
224
+ - `agreement: 4`, every reviewer: 96% (68 of 71). High confidence — confirm it in the code
225
+ and fix it.
226
+ - `agreement` 2 or 3: 60% (27 of 45). Verify before acting: about four in ten held no
227
+ labelled defect. Taken apart, the two rows are too small to tell from each other (2: 62%
228
+ of 26, 3: 58% of 19), so a third reviewer is not a reason to skip the check.
229
+ - `agreement: 1`: 41% (50 of 123). Judge it on its merits (step 3).
230
+
231
+ The shares are floors: the snippets hold defects nobody labelled, so an entry that matches
232
+ no label is not proven wrong. Verify a security finding (`INJECTION`, `AUTH_CRYPTO`, or any
233
+ `CRITICAL`) whatever its agreement. `disputed: true` means the reviewers agreed the defect is
234
+ real and differed only on how bad it is — the severity shown is the worst one raised.
235
+ 3. **Audit Outliers**: Review `agreement: 1` entries. Grouping is lexical, so one of these may be the same defect the others described in different words:
236
+ - If it is a critical security vulnerability (SQLi, auth bypass, secret leak, buffer overrun), investigate and fix it.
237
+ - If it is a structure or simplicity finding, judge it on the cost it names, not on how many
238
+ models raised it. Four reviewers rarely word one design opinion the same way, so agreement
239
+ is a weak signal on these axes and `agreement: 1` is the normal case, not a red flag.
240
+ - If it is subjective stylistic preference or hallucinated API syntax, disregard it.
241
+ 4. **Apply Fixes & Report**: Implement the corrected code and briefly summarize what the council caught for the user.