4take 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- 4take-0.2.0/.gitignore +65 -0
- 4take-0.2.0/.pre-commit-config.yaml +6 -0
- 4take-0.2.0/AGENTS.md +241 -0
- 4take-0.2.0/CHANGELOG.md +494 -0
- 4take-0.2.0/CLA.md +63 -0
- 4take-0.2.0/CLAUDE.md +99 -0
- 4take-0.2.0/CONTRIBUTING.md +93 -0
- 4take-0.2.0/LICENSE +145 -0
- 4take-0.2.0/PKG-INFO +902 -0
- 4take-0.2.0/POST_MORTEMS.md +53 -0
- 4take-0.2.0/README.md +866 -0
- 4take-0.2.0/ROADMAP.md +91 -0
- 4take-0.2.0/SECURITY.md +45 -0
- 4take-0.2.0/SKILL.md +54 -0
- 4take-0.2.0/docs/adr/0001-use-exponential-backoff-with-10-retries.md +31 -0
- 4take-0.2.0/docs/adr/0002-cap-retries-at-four-and-honour-retry-after.md +66 -0
- 4take-0.2.0/docs/adr/0003-retry-a-rate-limited-provider-eight-times.md +44 -0
- 4take-0.2.0/evals/README.md +22 -0
- 4take-0.2.0/evals/backfill_duplicates.py +74 -0
- 4take-0.2.0/evals/corpus/async_gather_swallow.py +5 -0
- 4take-0.2.0/evals/corpus/auth_jwt_decode.js +10 -0
- 4take-0.2.0/evals/corpus/auth_timing_compare.py +7 -0
- 4take-0.2.0/evals/corpus/clean_lru.py +24 -0
- 4take-0.2.0/evals/corpus/clean_slugify.py +16 -0
- 4take-0.2.0/evals/corpus/cmdinjection_backup.py +5 -0
- 4take-0.2.0/evals/corpus/cmdinjection_git.js +5 -0
- 4take-0.2.0/evals/corpus/divzero_average_latency.py +7 -0
- 4take-0.2.0/evals/corpus/leak_db_connection.py +8 -0
- 4take-0.2.0/evals/corpus/leak_file_handle.py +10 -0
- 4take-0.2.0/evals/corpus/leak_subprocess.py +8 -0
- 4take-0.2.0/evals/corpus/none_env_parse.py +6 -0
- 4take-0.2.0/evals/corpus/none_json_response.js +8 -0
- 4take-0.2.0/evals/corpus/none_lookup_chain.py +7 -0
- 4take-0.2.0/evals/corpus/offbyone_pagination.py +8 -0
- 4take-0.2.0/evals/corpus/offbyone_retry_window.js +11 -0
- 4take-0.2.0/evals/corpus/offbyone_ring_buffer.py +12 -0
- 4take-0.2.0/evals/corpus/overflow_backoff.py +9 -0
- 4take-0.2.0/evals/corpus/pathtraversal_download.py +10 -0
- 4take-0.2.0/evals/corpus/pathtraversal_zip.py +6 -0
- 4take-0.2.0/evals/corpus/race_async_cache.py +12 -0
- 4take-0.2.0/evals/corpus/race_counter_increment.py +13 -0
- 4take-0.2.0/evals/corpus/race_lazy_singleton.py +8 -0
- 4take-0.2.0/evals/corpus/sqli_order_by.py +9 -0
- 4take-0.2.0/evals/corpus/sqli_user_search.py +8 -0
- 4take-0.2.0/evals/corpus/toctou_config_write.py +10 -0
- 4take-0.2.0/evals/corpus/toctou_upload_quota.py +8 -0
- 4take-0.2.0/evals/corpus/unbounded_memory_tail.py +10 -0
- 4take-0.2.0/evals/corpus_ctx/cache_key_version/context/profiles/events.py +6 -0
- 4take-0.2.0/evals/corpus_ctx/cache_key_version/review.diff +15 -0
- 4take-0.2.0/evals/corpus_ctx/config_key_mismatch/context/app/settings.py +5 -0
- 4take-0.2.0/evals/corpus_ctx/config_key_mismatch/review.diff +18 -0
- 4take-0.2.0/evals/corpus_ctx/ctx_clean_locked_by_caller/context/cache/lru.py +30 -0
- 4take-0.2.0/evals/corpus_ctx/ctx_clean_locked_by_caller/review.diff +11 -0
- 4take-0.2.0/evals/corpus_ctx/ctx_clean_validated_upstream/context/billing/coupons.py +13 -0
- 4take-0.2.0/evals/corpus_ctx/ctx_clean_validated_upstream/review.diff +9 -0
- 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/context/accounts/identity.py +7 -0
- 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/context/accounts/login.py +6 -0
- 4take-0.2.0/evals/corpus_ctx/duplicate_normalizer/review.diff +17 -0
- 4take-0.2.0/evals/corpus_ctx/event_renamed/context/src/notifications/subscribers.js +6 -0
- 4take-0.2.0/evals/corpus_ctx/event_renamed/review.diff +9 -0
- 4take-0.2.0/evals/corpus_ctx/lock_not_taken/context/limits/counter.py +31 -0
- 4take-0.2.0/evals/corpus_ctx/lock_not_taken/review.diff +12 -0
- 4take-0.2.0/evals/corpus_ctx/missing_await/context/web/sessions.py +8 -0
- 4take-0.2.0/evals/corpus_ctx/missing_await/review.diff +9 -0
- 4take-0.2.0/evals/corpus_ctx/nested_commit/context/orders/lines.py +10 -0
- 4take-0.2.0/evals/corpus_ctx/nested_commit/review.diff +21 -0
- 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/context/app/api.py +9 -0
- 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/context/app/mailer.py +11 -0
- 4take-0.2.0/evals/corpus_ctx/none_return_breaks_caller/review.diff +10 -0
- 4take-0.2.0/evals/corpus_ctx/shell_helper_injection/context/deploy/git_tools.py +8 -0
- 4take-0.2.0/evals/corpus_ctx/shell_helper_injection/review.diff +15 -0
- 4take-0.2.0/evals/corpus_ctx/swapped_positional_args/context/src/billing.js +12 -0
- 4take-0.2.0/evals/corpus_ctx/swapped_positional_args/review.diff +14 -0
- 4take-0.2.0/evals/corpus_ctx/tuple_as_bool/context/shop/payments.py +9 -0
- 4take-0.2.0/evals/corpus_ctx/tuple_as_bool/review.diff +12 -0
- 4take-0.2.0/evals/corpus_ctx/units_seconds_to_ms/context/jobs/watchdog.py +10 -0
- 4take-0.2.0/evals/corpus_ctx/units_seconds_to_ms/review.diff +12 -0
- 4take-0.2.0/evals/labels.json +515 -0
- 4take-0.2.0/evals/labels_ctx.json +158 -0
- 4take-0.2.0/evals/record.py +315 -0
- 4take-0.2.0/evals/score.py +581 -0
- 4take-0.2.0/llms.txt +34 -0
- 4take-0.2.0/pyproject.toml +129 -0
- 4take-0.2.0/src/fourtake/__init__.py +1 -0
- 4take-0.2.0/src/fourtake/autocontext.py +1161 -0
- 4take-0.2.0/src/fourtake/cache.py +201 -0
- 4take-0.2.0/src/fourtake/cli.py +474 -0
- 4take-0.2.0/src/fourtake/client.py +474 -0
- 4take-0.2.0/src/fourtake/config.py +198 -0
- 4take-0.2.0/src/fourtake/context.py +331 -0
- 4take-0.2.0/src/fourtake/discovery.py +413 -0
- 4take-0.2.0/src/fourtake/engine.py +645 -0
- 4take-0.2.0/src/fourtake/filelock.py +92 -0
- 4take-0.2.0/src/fourtake/findings.py +652 -0
- 4take-0.2.0/src/fourtake/input_kind.py +74 -0
- 4take-0.2.0/src/fourtake/mcp_entry.py +31 -0
- 4take-0.2.0/src/fourtake/mcp_server.py +305 -0
- 4take-0.2.0/src/fourtake/output.py +279 -0
- 4take-0.2.0/src/fourtake/preflight.py +102 -0
- 4take-0.2.0/src/fourtake/prompts.py +136 -0
- 4take-0.2.0/src/fourtake/rate_limiter.py +125 -0
- 4take-0.2.0/src/fourtake/safety.py +268 -0
- 4take-0.2.0/src/fourtake/telemetry.py +220 -0
- 4take-0.2.0/src/fourtake/tokenizer.py +23 -0
- 4take-0.2.0/tests/conftest.py +33 -0
- 4take-0.2.0/tests/test_autocontext.py +355 -0
- 4take-0.2.0/tests/test_cache.py +239 -0
- 4take-0.2.0/tests/test_cli.py +662 -0
- 4take-0.2.0/tests/test_client.py +448 -0
- 4take-0.2.0/tests/test_config.py +150 -0
- 4take-0.2.0/tests/test_context.py +333 -0
- 4take-0.2.0/tests/test_council_scheduling.py +724 -0
- 4take-0.2.0/tests/test_discovery.py +314 -0
- 4take-0.2.0/tests/test_docs.py +67 -0
- 4take-0.2.0/tests/test_engine.py +555 -0
- 4take-0.2.0/tests/test_evals.py +490 -0
- 4take-0.2.0/tests/test_findings.py +893 -0
- 4take-0.2.0/tests/test_input_kind.py +116 -0
- 4take-0.2.0/tests/test_integration.py +242 -0
- 4take-0.2.0/tests/test_mcp_server.py +651 -0
- 4take-0.2.0/tests/test_output.py +202 -0
- 4take-0.2.0/tests/test_output_contract.py +833 -0
- 4take-0.2.0/tests/test_preflight.py +106 -0
- 4take-0.2.0/tests/test_prompts.py +200 -0
- 4take-0.2.0/tests/test_rate_limiter.py +162 -0
- 4take-0.2.0/tests/test_safety.py +321 -0
- 4take-0.2.0/tests/test_telemetry.py +257 -0
- 4take-0.2.0/tests/test_tokenizer.py +52 -0
- 4take-0.2.0/tests/test_zero_cost_enforcement.py +103 -0
4take-0.2.0/.gitignore
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Python bytecode and cache
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*$py.class
|
|
5
|
+
*.so
|
|
6
|
+
|
|
7
|
+
# Distribution and packaging
|
|
8
|
+
build/
|
|
9
|
+
dist/
|
|
10
|
+
*.egg-info/
|
|
11
|
+
*.egg
|
|
12
|
+
.eggs/
|
|
13
|
+
wheels/
|
|
14
|
+
*.whl
|
|
15
|
+
|
|
16
|
+
# Virtual environments
|
|
17
|
+
.venv/
|
|
18
|
+
venv/
|
|
19
|
+
env/
|
|
20
|
+
ENV/
|
|
21
|
+
|
|
22
|
+
# Testing and coverage
|
|
23
|
+
.pytest_cache/
|
|
24
|
+
.coverage
|
|
25
|
+
.coverage.*
|
|
26
|
+
coverage.xml
|
|
27
|
+
htmlcov/
|
|
28
|
+
coverage_report.txt
|
|
29
|
+
|
|
30
|
+
# Audit and temporary scratch files
|
|
31
|
+
audit_*.txt
|
|
32
|
+
audit_report.md
|
|
33
|
+
review.json
|
|
34
|
+
scratch.py
|
|
35
|
+
test_audit.py
|
|
36
|
+
test_or.py
|
|
37
|
+
4take_raw_data.json
|
|
38
|
+
|
|
39
|
+
# Local config & state cache
|
|
40
|
+
.cache/
|
|
41
|
+
evals/recordings_backup/
|
|
42
|
+
|
|
43
|
+
# OS and IDE files
|
|
44
|
+
.DS_Store
|
|
45
|
+
.DS_Store?
|
|
46
|
+
._*
|
|
47
|
+
.Spotlight-V100
|
|
48
|
+
.Trashes
|
|
49
|
+
ehthumbs.db
|
|
50
|
+
Thumbs.db
|
|
51
|
+
.idea/
|
|
52
|
+
.vscode/
|
|
53
|
+
*.swp
|
|
54
|
+
*.swo
|
|
55
|
+
*~
|
|
56
|
+
|
|
57
|
+
# Evals built from private repositories: their code must never reach this repository
|
|
58
|
+
evals/private/
|
|
59
|
+
|
|
60
|
+
# Third-party benchmark data, fetched for scoring and not redistributed
|
|
61
|
+
evals/external/
|
|
62
|
+
|
|
63
|
+
# The frontier test's answer key and blind runs: an agent under test that can read truth.json is
|
|
64
|
+
# no longer blind. prereg.sha256 fixes the corpus and the key before any run.
|
|
65
|
+
evals/frontier/
|
4take-0.2.0/AGENTS.md
ADDED
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
# AGENTS.md — AI Agent Integration Guide for 4take
|
|
2
|
+
|
|
3
|
+
This guide specifies how autonomous AI coding agents (Claude Code, Cursor Composer, Antigravity IDE, Windsurf, Codex, Devin, Cline, Roo Code, Aider) should interact with `4take`.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 🎯 Purpose of 4take for AI Agents
|
|
8
|
+
|
|
9
|
+
When generating complex algorithms, database migrations, security-critical logic, or large refactors, a single LLM is prone to subtle blindspots and confirmation bias.
|
|
10
|
+
|
|
11
|
+
`4take` provides you (the agent) with an independent **Council of 4 LLMs** to audit your code in parallel before you present it to the user.
|
|
12
|
+
|
|
13
|
+
Whether the council improves what a given agent ships has not been measured yet. Treat its output as leads to verify, not verdicts; step 2 of the synthesis framework below gives the measured hit rates.
|
|
14
|
+
|
|
15
|
+
The council is told its input was written by a capable model, so it is not looking for syntax
|
|
16
|
+
or typos. It reviews four axes: **correctness**, **security**, **structure** (responsibility in
|
|
17
|
+
the wrong layer, coupling that spreads one change across several files, logic duplicated where
|
|
18
|
+
it will drift) and **simplicity** (indirection with one implementation, a hand-rolled stdlib).
|
|
19
|
+
A structural finding is only reported when the model can name its concrete cost.
|
|
20
|
+
|
|
21
|
+
It reviews code. Piping in a design question or a plan is refused before any request goes out;
|
|
22
|
+
pass `--force-review` (CLI) or `force=true` (MCP) to override.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 🛠️ Integration Methods
|
|
27
|
+
|
|
28
|
+
### Method 1: FastMCP Server (`get_4take_opinion`)
|
|
29
|
+
|
|
30
|
+
If your agent runtime supports the **Model Context Protocol (MCP)**:
|
|
31
|
+
|
|
32
|
+
- **Server Entrypoint:** `4take-mcp` (or `uvx --from '4take[mcp]' 4take-mcp`). The MCP
|
|
33
|
+
server ships as the `mcp` extra: `pip install '4take[mcp]'`.
|
|
34
|
+
- **Server output:** first `report_for_user`, a list of lines to show the user verbatim or close
|
|
35
|
+
to it: a header with the [calibration](#-the-agent-as-chairman-4-step-synthesis-framework) and
|
|
36
|
+
what follows from it (agreement 3 or more: confirm in the code, then fix; 1 or 2: check against
|
|
37
|
+
the code before accepting or rejecting it), then one line per `consensus` entry, most-agreed
|
|
38
|
+
first, such as ``- 2/4 WARNING `page_size` — Unbounded limit``. The denominator is the number
|
|
39
|
+
of independent reviewers that answered. Show every line, the ones you go on to reject included:
|
|
40
|
+
the user sees only what you relay. Then the same `consensus` list the CLI emits, plus `status`
|
|
41
|
+
(`ok` / `degraded` / `error`), `distinct_models` and `min_responses`. `degraded`
|
|
42
|
+
means fewer independent reviewers answered than were asked for; a `warning` alongside
|
|
43
|
+
`status: ok` is informational, not a failure. Only a review fills a seat: an answer with
|
|
44
|
+
neither a finding nor `NO_ISSUES_FOUND` (a tool call, garbage tokens, reasoning that never
|
|
45
|
+
reached a tag) is counted as a failed model and replaced by a spare.
|
|
46
|
+
- **Tool Name:** `get_4take_opinion`
|
|
47
|
+
- **Arguments:**
|
|
48
|
+
- `code` (string, required): The source code, git diff, or implementation to review.
|
|
49
|
+
- `force` (boolean, optional, default `false`): Send the input even though it does not look
|
|
50
|
+
like code. Without it, prose is refused rather than dispatched.
|
|
51
|
+
- `context_files` (list of strings, optional): Paths of code related to the change, which the
|
|
52
|
+
reviewers read but do not review. The server reads them from disk, under the project root,
|
|
53
|
+
so you do not paste them. `path:START-END` sends only those lines. See
|
|
54
|
+
[Choosing the context](#-choosing-the-context-the-chairmans-first-job).
|
|
55
|
+
- `context` (string, optional): The same kind of related code as text, when the server cannot
|
|
56
|
+
read your disk.
|
|
57
|
+
- `auto_context` (boolean, optional, default `false`): With a diff, also let the server choose
|
|
58
|
+
context by name: what still uses what the change removes, the callers of what it edits, the
|
|
59
|
+
definitions of what it calls. Sent after `context_files`, within the same budget.
|
|
60
|
+
- **Return Value:** JSON string: `report_for_user` first, then the fields above and the
|
|
61
|
+
`responses` array with each model's independent critique and metadata.
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
### Method 2: Shell / CLI Subprocess
|
|
66
|
+
|
|
67
|
+
If invoking `4take` via bash / shell commands:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
# Recommended command for agents:
|
|
71
|
+
cat path/to/file.py | 4take --format json --accept-risk --quiet
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
#### CLI Flags for Agents:
|
|
75
|
+
- `--format json`: Ensures machine-readable JSON output on stdout instead of ANSI Markdown.
|
|
76
|
+
- `--accept-risk`: Bypasses terminal privacy prompts (or set `FOURTAKE_ACCEPT_RISK=1`).
|
|
77
|
+
- `--quiet`: Suppresses live progress logs and diagnostics.
|
|
78
|
+
- `--min-responses N`: (Optional) Minimum successful model responses required (default: 4).
|
|
79
|
+
- `--per-model-timeout N`: (Optional) Abandon a single model after N seconds and rotate in a
|
|
80
|
+
replacement. Defaults to half of `--timeout` (minimum 60s).
|
|
81
|
+
- `--force-review`: (Optional) Review an input that does not look like source code. Without
|
|
82
|
+
it, prose is refused with exit code 1 instead of spending four requests to say nothing.
|
|
83
|
+
- `--strict`: (Optional, usually leave off) Narrow `consensus` to findings more than one reviewer
|
|
84
|
+
raised, plus any `CRITICAL` one. Step 3 below needs the single-reviewer entries it drops.
|
|
85
|
+
- `--context-file PATH[:START-END]`: (Optional, repeatable, most relevant first) Related code the
|
|
86
|
+
council reads but does not review. Read only under the working directory. See
|
|
87
|
+
[Choosing the context](#-choosing-the-context-the-chairmans-first-job).
|
|
88
|
+
- `--auto-context`: (Optional) With a diff, also choose context by name, after any
|
|
89
|
+
`--context-file`. `summary.context.auto` says why each excerpt was chosen.
|
|
90
|
+
- `--context-budget N`: (Optional) Cap on reference context in tokens, default 16000. A file that
|
|
91
|
+
does not fit is left out whole and named in `summary.context.dropped`.
|
|
92
|
+
- `--dry-run`: Print exactly what would be sent, after redaction, and exit without contacting
|
|
93
|
+
any model.
|
|
94
|
+
- `--no-cache` / `--clear-cache`: Bypass or purge the local 24h review cache.
|
|
95
|
+
- `--no-telemetry` / `--clear-telemetry`: Skip or erase the local record of model latency and
|
|
96
|
+
success rate. It stores model ids and timings only — never code, reviews or prompts — and
|
|
97
|
+
is what ranks the model pool on evidence instead of on model names.
|
|
98
|
+
|
|
99
|
+
**stdout carries only the JSON payload.** All logs, progress and warnings go to stderr, so
|
|
100
|
+
`json.loads(stdout)` is safe.
|
|
101
|
+
|
|
102
|
+
#### Exit Codes
|
|
103
|
+
| Code | Meaning |
|
|
104
|
+
| :--- | :--- |
|
|
105
|
+
| `0` | The council met `--min-responses`. |
|
|
106
|
+
| `2` | Partial: at least one review, but fewer than `--min-responses`. Check `warning`. |
|
|
107
|
+
| `1` | Total failure: no model responded. Check `error`. |
|
|
108
|
+
|
|
109
|
+
A large diff can take up to `--timeout` (600 s) and often ends with exit code `2` and fewer than
|
|
110
|
+
four reviewers: few free models reason through one to an answer. A partial council is still a
|
|
111
|
+
usable review; read `agreement` against `summary.distinct_models`.
|
|
112
|
+
|
|
113
|
+
> **Free models only**: `4take` sends requests only to OpenRouter's free models (`:free` tier) and rejects a paid model before any request goes out. Do not pass `--models` with paid models. Free models are rate-limited: roughly 4 to 8 reviews a day on an OpenRouter account that has never been topped up, roughly 80 to 160 once it has. Do not call `4take` in a loop.
|
|
114
|
+
|
|
115
|
+
---
|
|
116
|
+
|
|
117
|
+
## 📜 Output JSON Structure
|
|
118
|
+
|
|
119
|
+
```json
|
|
120
|
+
{
|
|
121
|
+
"version": "0.2.0",
|
|
122
|
+
"timestamp": "2026-08-16T14:20:00Z",
|
|
123
|
+
"summary": {
|
|
124
|
+
"task_hash": "25d297f773c3",
|
|
125
|
+
"responses_received": 4,
|
|
126
|
+
"responses_failed": 0,
|
|
127
|
+
"total_latency_ms": 3200,
|
|
128
|
+
"context": {"files": ["src/mailer.py", "src/users.py (lines 40-90)"], "tokens": 812, "dropped": []}
|
|
129
|
+
},
|
|
130
|
+
"warning": null,
|
|
131
|
+
"error": null,
|
|
132
|
+
"consensus": [
|
|
133
|
+
{
|
|
134
|
+
"agreement": 2,
|
|
135
|
+
"severity": "CRITICAL",
|
|
136
|
+
"category": "INJECTION",
|
|
137
|
+
"symbol": "search_users",
|
|
138
|
+
"title": "SQL injection via the term parameter",
|
|
139
|
+
"models": ["cohere/north-mini-code", "nvidia/nemotron-3-ultra-550b-a55b"],
|
|
140
|
+
"severities": {"CRITICAL": 1, "WARNING": 1},
|
|
141
|
+
"disputed": true
|
|
142
|
+
}
|
|
143
|
+
],
|
|
144
|
+
"responses": [
|
|
145
|
+
{
|
|
146
|
+
"model": "openrouter/openrouter/free",
|
|
147
|
+
"served_model": "nvidia/nemotron-3-ultra-550b-a55b",
|
|
148
|
+
"served_provider": "NVIDIA",
|
|
149
|
+
"content": "Review text from model...",
|
|
150
|
+
"finish_reason": "stop",
|
|
151
|
+
"truncated": false,
|
|
152
|
+
"duplicate_of": null,
|
|
153
|
+
"latency_ms": 1150
|
|
154
|
+
},
|
|
155
|
+
{
|
|
156
|
+
"model": "openrouter/cohere/north-mini-code:free",
|
|
157
|
+
"served_model": "cohere/north-mini-code",
|
|
158
|
+
"served_provider": "Cohere",
|
|
159
|
+
"content": "Review text from model...",
|
|
160
|
+
"finish_reason": "stop",
|
|
161
|
+
"truncated": false,
|
|
162
|
+
"latency_ms": 1420
|
|
163
|
+
}
|
|
164
|
+
],
|
|
165
|
+
"errors": [
|
|
166
|
+
{
|
|
167
|
+
"model": "failing/model",
|
|
168
|
+
"error": "Reason for error",
|
|
169
|
+
"latency_ms": 200
|
|
170
|
+
}
|
|
171
|
+
]
|
|
172
|
+
}
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## 🧭 Choosing the Context (the Chairman's first job)
|
|
178
|
+
|
|
179
|
+
The council sees only what you send. A diff alone hides every defect that lives between files: a
|
|
180
|
+
changed return value a caller still relies on, a route without the check its neighbours make, a
|
|
181
|
+
helper duplicated under another name. You have already read the surrounding code, so pass it:
|
|
182
|
+
|
|
183
|
+
1. Each changed function in full: `git diff -W` includes it in the diff itself.
|
|
184
|
+
2. Its direct callers, one hop away (grep for the changed names), most relevant first.
|
|
185
|
+
3. The functions it calls, and the types, schemas and config it relies on.
|
|
186
|
+
4. Related tests, when they encode the contract.
|
|
187
|
+
|
|
188
|
+
A few files or line ranges, not the repository: the budget is 16000 tokens, and whole files are
|
|
189
|
+
dropped past it. Prefer paths (`context_files`, `--context-file`) to pasted text, so you spend
|
|
190
|
+
no output tokens on them. Findings stay on the reviewed code; a caller the change breaks is named
|
|
191
|
+
in the finding's Issue. `summary.context` (CLI) or `context` (MCP) lists what was sent and what
|
|
192
|
+
the budget dropped. A finding that needed a caller you did not send is one the council could not
|
|
193
|
+
make.
|
|
194
|
+
|
|
195
|
+
`auto_context` (`--auto-context`) does steps 2 and 3 the way grep would: callers of the edited
|
|
196
|
+
functions that import their module, remaining uses of a removed name or string value, definitions
|
|
197
|
+
of what the new lines call, and the rest of an edited class outside the diff. It cannot know what
|
|
198
|
+
you know, so it comes after the files you name; name the ones that matter and let it add the rest.
|
|
199
|
+
Check `context.auto` for what it chose and why. On the cross-file eval corpus it chose the same
|
|
200
|
+
files as a careful chairman, and the council named 12 of 12 cross-file defects in each of three
|
|
201
|
+
recordings with it, against 0 or 1 with the diff alone.
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
## 🧠 The Agent as Chairman: 4-Step Synthesis Framework
|
|
206
|
+
|
|
207
|
+
When you receive the council's reviews:
|
|
208
|
+
|
|
209
|
+
0. **Check `truncated`**: a response with `truncated: true` was cut off by the output budget
|
|
210
|
+
and is incomplete — weigh it accordingly, and do not treat its silence as "no issues".
|
|
211
|
+
1. **Read `consensus` first**: the findings are already grouped across reviewers and ranked
|
|
212
|
+
by `agreement`. It is keyed on `served_model`, not `model`, so a router seating the same
|
|
213
|
+
underlying model twice contributes one voice rather than manufacturing agreement. A council
|
|
214
|
+
also seats one model per vendor while another vendor can answer; a response with
|
|
215
|
+
`duplicate_of` set shadows an existing seat and is not counted — if you re-derive agreement
|
|
216
|
+
from `responses`, skip those.
|
|
217
|
+
`responses[i].content` is still there when you need a finding's full wording.
|
|
218
|
+
`category` is the class the reviewers declared, and `UNKNOWN` when none did — it is read
|
|
219
|
+
from the reviews, never guessed, so an `UNKNOWN` says nothing about the finding itself.
|
|
220
|
+
Reviewers file one defect under the same category only about half the time: treat it as
|
|
221
|
+
a label, never as evidence that two findings are, or are not, the same defect.
|
|
222
|
+
2. **Weigh `agreement` as measured**: over three recordings of the eval corpus, every council
|
|
223
|
+
seating four reviewers, this is how often a consensus entry held a labelled defect:
|
|
224
|
+
- `agreement: 4`, every reviewer: 96% (68 of 71). High confidence — confirm it in the code
|
|
225
|
+
and fix it.
|
|
226
|
+
- `agreement` 2 or 3: 60% (27 of 45). Verify before acting: about four in ten held no
|
|
227
|
+
labelled defect. Taken apart, the two rows are too small to tell from each other (2: 62%
|
|
228
|
+
of 26, 3: 58% of 19), so a third reviewer is not a reason to skip the check.
|
|
229
|
+
- `agreement: 1`: 41% (50 of 123). Judge it on its merits (step 3).
|
|
230
|
+
|
|
231
|
+
The shares are floors: the snippets hold defects nobody labelled, so an entry that matches
|
|
232
|
+
no label is not proven wrong. Verify a security finding (`INJECTION`, `AUTH_CRYPTO`, or any
|
|
233
|
+
`CRITICAL`) whatever its agreement. `disputed: true` means the reviewers agreed the defect is
|
|
234
|
+
real and differed only on how bad it is — the severity shown is the worst one raised.
|
|
235
|
+
3. **Audit Outliers**: Review `agreement: 1` entries. Grouping is lexical, so one of these may be the same defect the others described in different words:
|
|
236
|
+
- If it is a critical security vulnerability (SQLi, auth bypass, secret leak, buffer overrun), investigate and fix it.
|
|
237
|
+
- If it is a structure or simplicity finding, judge it on the cost it names, not on how many
|
|
238
|
+
models raised it. Four reviewers rarely word one design opinion the same way, so agreement
|
|
239
|
+
is a weak signal on these axes and `agreement: 1` is the normal case, not a red flag.
|
|
240
|
+
- If it is subjective stylistic preference or hallucinated API syntax, disregard it.
|
|
241
|
+
4. **Apply Fixes & Report**: Implement the corrected code and briefly summarize what the council caught for the user.
|