kairos-chain 3.84.0 → 3.85.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +77 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/design_to_implementation_workflow/design_to_implementation_workflow.md +25 -6
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +150 -21
- data/templates/knowledge/multi_llm_reviewer_evaluation/multi_llm_reviewer_evaluation.md +59 -12
- data/templates/skillsets/agent/tools/agent_step.rb +8 -2
- data/templates/skillsets/minimum_nomic/bin/run_gm.rb +178 -10
- data/templates/skillsets/multi_llm_review/config/multi_llm_review.yml +98 -53
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bb60c9f0163eaf405fdad217697364791fd3bc3d40fe15a77ef57458950c1163
|
|
4
|
+
data.tar.gz: f5e9dd8b9bfc9ccc36472ada9bdc0d0b00104ef06398c198c6d9963369f82d39
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 28cb26af2e39d33a5f39e4157c5c2e74f7ff63af77384a73aadb6fe4c03e7c7a8e6c445f7deaff9fdc0b05daf47ad69fdb5a6dee3bdeecd6f40a64f146f7196d
|
|
7
|
+
data.tar.gz: ac81db0e9cb2ab019f2944113de79e7b4c19be5389dead92398f86dcc6b4a82f38491bfad13de2a98d68eefe0ee7ddf94d23289d131662fbff2c1ccfe6dab9cc
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,83 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.85.0] - 2026-09-06
|
|
8
|
+
|
|
9
|
+
### Changed — multi-LLM review: high effort by default, gpt-6-astra seats, gpt-5.5 retires
|
|
10
|
+
|
|
11
|
+
Templates only: `multi_llm_review/config/multi_llm_review.yml`, L1
|
|
12
|
+
`multi_llm_review_workflow` 3.12.0 → 3.13.0. No library code.
|
|
13
|
+
|
|
14
|
+
**Effort.** Every seat that has an effort control now runs at `high`, at every
|
|
15
|
+
complexity level, so `effort_map` is a constant rather than a function of complexity.
|
|
16
|
+
Claude CLI takes `--effort high`, codex takes `-c model_reasoning_effort=high`, Cursor
|
|
17
|
+
takes nothing — `cursor_adapter` builds no effort flag, so a value set for a cursor
|
|
18
|
+
entry would be recorded and never sent. Both the roster defaults and `effort_map` are
|
|
19
|
+
set, because `effort_map` overrides the roster per dispatch and an entry left at
|
|
20
|
+
`medium` would be silently raised. `high` rather than `xhigh`/`max` because it is the
|
|
21
|
+
ceiling the two providers share (Claude CLI accepts low/medium/high/xhigh/max, codex
|
|
22
|
+
accepts minimal/low/medium/high), and a roster split across incomparable settings is
|
|
23
|
+
worth less than a lower common one.
|
|
24
|
+
|
|
25
|
+
This supersedes the 2026-04-29 default-effort policy. That policy rested on one
|
|
26
|
+
low-vs-high measurement — 8.35 vs 8.16 — taken on the Opus 4.6 / 4.7 generation, on
|
|
27
|
+
none of the models now in the roster. It was not re-measured, so the supersession is a
|
|
28
|
+
judgement and a later measurement could reverse it. Cost is measured and rises:
|
|
29
|
+
gpt-6-astra spent 8,274 tokens at high against 3,150 at its default on one identical
|
|
30
|
+
one-line prompt.
|
|
31
|
+
|
|
32
|
+
**Roster.** `codex_gpt6-astra` replaces `codex_gpt5.6-sol`, verified before the swap
|
|
33
|
+
through the flags `codex_adapter` actually builds. `codex_gpt5.5` retires in the same
|
|
34
|
+
edit and is not replaced: it had been commented out of the instance roster since
|
|
35
|
+
2026-07-30 "for round R10 only" and never restored, so the calibrated
|
|
36
|
+
cross-generation anchor role had already lapsed for five weeks. Codex now holds one
|
|
37
|
+
slot and it is uncalibrated. Roster 5 → 4, so `convergence_rule` moves 3/5 → 3/4 and
|
|
38
|
+
`convergence_rule_after_exclusion` 3/4 → 2/3, both on the same ceil(N × 0.6) basis and
|
|
39
|
+
both reference figures rather than gates. Three calibration warnings are recorded in
|
|
40
|
+
the L1: the 138-run seat profile is gpt-5.6-sol's and does not transfer; that corpus
|
|
41
|
+
was gathered at medium effort; and it was gathered on a 5-seat roster.
|
|
42
|
+
|
|
43
|
+
### Fixed — three documents called the APPROVE ratio the closing condition
|
|
44
|
+
|
|
45
|
+
A round closes on the exhaustion of (a)+(b) findings, declared by the operator, with
|
|
46
|
+
new (a)+(b) P0 = 0 as the machine-side signal. The tooling has said so since v0.7
|
|
47
|
+
INV-R2 named the field `reference_verdict` and v0.10.1 renamed `convergence` to
|
|
48
|
+
`vote_tally`. Three documents had not caught up, and they are the ones actually read.
|
|
49
|
+
|
|
50
|
+
L1 `multi_llm_reviewer_evaluation` 1.5 → 1.6. § Convergence Rule (Updated) stated
|
|
51
|
+
`3/4 APPROVE = proceed to next step` and `4/4 APPROVE = merge-ready` with no note that
|
|
52
|
+
the ratio is a reference value — while L1 `multi_llm_review_workflow` names this very
|
|
53
|
+
section as the normative detail it defers to. The document saying "the ratio is not
|
|
54
|
+
the primary close" pointed at a document saying it was. The closing condition is now
|
|
55
|
+
stated first, the one blocking rule (any (a)/(b) REJECT) is separated from the
|
|
56
|
+
reference figures, and the two facts that make the ratio unusable as a criterion sit
|
|
57
|
+
in the section: Codex went 24 of 24 reviews without APPROVE on one design loop, and
|
|
58
|
+
both 2026-08 threads closed by (a)+(b) exhaustion without reaching their ratio.
|
|
59
|
+
|
|
60
|
+
L1 `design_to_implementation_workflow` 1.1 → 1.2. `merge gate: 3/3 APPROVE =
|
|
61
|
+
merge-ready` and `Merge criteria: 3/3 APPROVE with 0 FAIL` replaced by the closing
|
|
62
|
+
condition; the anti-pattern list gains reading an APPROVE count as the closing
|
|
63
|
+
condition, naming this document as having carried it. The reviewer lineup written here
|
|
64
|
+
("3 reviewers: Opus 4.6 + Codex GPT-5.4 + Composer-2") is deleted rather than
|
|
65
|
+
corrected — it had been wrong since 2026-04-19, and a roster copied into a second
|
|
66
|
+
document goes stale silently.
|
|
67
|
+
|
|
68
|
+
`agent` SkillSet: the bundle handed to reviewers opened "aggregate per the convergence
|
|
69
|
+
rule below" followed by the bare ratio. The heading no longer calls it a convergence
|
|
70
|
+
rule, a Closing condition section states what actually closes a round, and the ratio
|
|
71
|
+
is labelled "Reference ratio (recorded, never sufficient on its own)".
|
|
72
|
+
|
|
73
|
+
Surviving uses of "merge gate" name the stage and its reviewer lineup, not a ratio,
|
|
74
|
+
and stay.
|
|
75
|
+
|
|
76
|
+
**Not reviewed by multi-LLM review**, at the operator's explicit instruction — the
|
|
77
|
+
change is a seat swap plus wording, and a round would have cost hours. Verification is
|
|
78
|
+
this agent's own and is not an independent judgement: YAML load of both configs, the
|
|
79
|
+
three CLI invocations run for real (gpt-6-astra and both Claude models answered at
|
|
80
|
+
high, with the model echoed back), `ruby -c` on `agent_step.rb`, multi_llm_review 556
|
|
81
|
+
runs / 1,809 assertions and agent 21 checks, all passing. No test asserts the reviewer
|
|
82
|
+
prompt wording, so that one line is unguarded.
|
|
83
|
+
|
|
7
84
|
## [3.84.0] - 2026-09-03
|
|
8
85
|
|
|
9
86
|
### Fixed — agent: second field-defect bundle (D5-b, D6, exit-record housekeeping), real-process worker tests
|
data/lib/kairos_mcp/version.rb
CHANGED
data/templates/knowledge/design_to_implementation_workflow/design_to_implementation_workflow.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: design_to_implementation_workflow
|
|
3
3
|
description: "Full-lifecycle workflow for complex features: design review, self-review, implementation review, and final merge gate. Derived from Service Grant + Attestation Nudge experiments."
|
|
4
|
-
version: "1.
|
|
4
|
+
version: "1.2"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- implementation
|
|
@@ -31,7 +31,8 @@ different bugs.
|
|
|
31
31
|
│ v0.2 ──→ Multi-LLM Review R2 ──→ Fix ──→ v0.3 │
|
|
32
32
|
│ (fix correctness) │
|
|
33
33
|
│ │
|
|
34
|
-
│
|
|
34
|
+
│ Closes on: new (a)+(b) P0 = 0, operator declares freeze │
|
|
35
|
+
│ (APPROVE ratio is a reference figure, not the gate) │
|
|
35
36
|
├─────────────────────────────────────────────────────────────┤
|
|
36
37
|
│ IMPLEMENTATION PHASE │
|
|
37
38
|
│ │
|
|
@@ -48,7 +49,7 @@ different bugs.
|
|
|
48
49
|
│ (missing wiring, fail-open, integration gaps) │
|
|
49
50
|
│ │
|
|
50
51
|
│ Final Multi-LLM Review + Persona Assembly │
|
|
51
|
-
│ (
|
|
52
|
+
│ (closes the same way; ratio stays a reference figure) │
|
|
52
53
|
└─────────────────────────────────────────────────────────────┘
|
|
53
54
|
```
|
|
54
55
|
|
|
@@ -76,8 +77,15 @@ different bugs.
|
|
|
76
77
|
- Output: Complete design document with pseudocode
|
|
77
78
|
|
|
78
79
|
#### Multi-LLM Review Rounds
|
|
79
|
-
- **
|
|
80
|
-
|
|
80
|
+
- **Reviewers**: the roster in `multi_llm_review/config/multi_llm_review.yml`,
|
|
81
|
+
which is where the count and the models live. The "3 reviewers: Opus 4.6 +
|
|
82
|
+
Codex GPT-5.4 + Composer-2" written here until 2026-09-06 had been wrong since
|
|
83
|
+
2026-04-19 and is not restated, because a roster copied into a second document
|
|
84
|
+
goes stale silently
|
|
85
|
+
- **Closing condition**: new (a)+(b) P0 = 0, carryover P0s counted separately
|
|
86
|
+
with a closure verdict on each, and the freeze declared by the operator. The
|
|
87
|
+
APPROVE ratio the tool reports is a recorded reference value — see L1
|
|
88
|
+
`multi_llm_review_workflow` § Convergence Rules
|
|
81
89
|
- **Typical rounds**: 2-3 for Tier 3 complexity
|
|
82
90
|
- **Convergence curve**:
|
|
83
91
|
- R1: Structural gaps — "this is missing" (existence)
|
|
@@ -148,7 +156,11 @@ Claude Persona Assembly (4 personas):
|
|
|
148
156
|
- Safety-critical components
|
|
149
157
|
- NOT for intermediate rounds (diminishing returns)
|
|
150
158
|
|
|
151
|
-
**Merge criteria**:
|
|
159
|
+
**Merge criteria**: new (a)+(b) P0 = 0, with the freeze declared by the operator.
|
|
160
|
+
A Codex APPROVE is the strongest *reference* signal (see
|
|
161
|
+
`multi_llm_reviewer_evaluation`), and it is not a gate: Codex went 24 of 24
|
|
162
|
+
reviews without reaching APPROVE on one design loop, so a merge waiting on it
|
|
163
|
+
would never have merged.
|
|
152
164
|
|
|
153
165
|
## Effort Level Selection
|
|
154
166
|
|
|
@@ -192,6 +204,9 @@ Reviewer 側 (Opus 4.7) は effort-independent なので high が天井。xhigh/
|
|
|
192
204
|
|
|
193
205
|
## Convergence Data
|
|
194
206
|
|
|
207
|
+
Observed outcomes of past loops, not criteria. The APPROVE counts below record
|
|
208
|
+
what those particular rounds happened to reach; they are not what closed them.
|
|
209
|
+
|
|
195
210
|
### Service Grant (Tier 3, 2026-03-18)
|
|
196
211
|
- Design: v1.0 → v1.4, 3 review rounds, 3 LLMs
|
|
197
212
|
- Design review findings: R1: 8 P0/P1, R2: 2 FAIL + 28 CONCERN, R3: 0 FAIL
|
|
@@ -207,6 +222,10 @@ Reviewer 側 (Opus 4.7) は effort-independent なので high が天井。xhigh/
|
|
|
207
222
|
|
|
208
223
|
## Anti-Patterns
|
|
209
224
|
|
|
225
|
+
- **Reading an APPROVE count as the closing condition.** It is a recorded
|
|
226
|
+
reference value; a round closes on new (a)+(b) P0 = 0 with the freeze declared
|
|
227
|
+
by the operator. This document itself carried the mistake until v1.2, in the
|
|
228
|
+
strongest available wording ("merge gate: 3/3 APPROVE = merge-ready")
|
|
210
229
|
- Implementing Phase 2+ when Phase 1 prerequisites aren't met
|
|
211
230
|
- Using agent team for implementation (context fragmentation)
|
|
212
231
|
- Skipping self-review (misses cheap P0 fixes)
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.
|
|
4
|
+
version: "3.13.0"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -90,6 +90,72 @@ Codex (c)-class value-divergent REJECTs as blocking, which causes review loops t
|
|
|
90
90
|
fail to converge. The cross-reference exists in `related:` frontmatter; this step
|
|
91
91
|
makes it an explicit pre-condition rather than an implicit hint.
|
|
92
92
|
|
|
93
|
+
## Step 0.1 — Measured seat characteristics (corpus 2026-08-03 .. 2026-09-03)
|
|
94
|
+
|
|
95
|
+
Qualitative per-model profiles stay in `multi_llm_reviewer_evaluation`. What
|
|
96
|
+
follows is the counted complement: seat behaviour as it actually occurred, so
|
|
97
|
+
seat selection is a lookup rather than a recollection. A **seat** is one
|
|
98
|
+
reviewer slot in one run.
|
|
99
|
+
|
|
100
|
+
Corpus: 138 completed runs in the local run store, 556 seat observations, 5286
|
|
101
|
+
findings (P0 1473 / P1 560 / P2 2994 / P3 259). Run types: design 54,
|
|
102
|
+
implementation 60, document 24. Extraction script `log/mlr_extract/extract.py`
|
|
103
|
+
lives in the development repository and does not ship; re-run it to refresh
|
|
104
|
+
these numbers.
|
|
105
|
+
|
|
106
|
+
| Seat | Runs | APPROVE rate design / impl / doc | Median wall s | Median output chars | (c) share, labelled only |
|
|
107
|
+
|---|---|---|---|---|---|
|
|
108
|
+
| `claude_cli_opus4.6` | 134 | 64% (31/48) / 71% (27/38) / 86% (20/23) | 54 | 4,536 | 70% (19/27) |
|
|
109
|
+
| `cursor_composer2.5` | 138 | 13% (7/51) / 48% (24/50) / 25% (6/24) | 133 | 3,795 | 100% (2/2) |
|
|
110
|
+
| `codex_gpt5.6-sol` (retired 2026-09-05) | 138 | 1% (1/54) / 11% (7/60) / 4% (1/24) | 113 | 2,038 | 3% (5/140) |
|
|
111
|
+
| `claude_team_opus-5` (persona) | 131 | 0% (0/51) / 12% (7/56) / 0% (0/22) | not measured per seat | 11,512 | 27% (374/1362) |
|
|
112
|
+
|
|
113
|
+
The codex row is a profile of **gpt-5.6-sol**, which left the roster on
|
|
114
|
+
2026-09-05 when gpt-6-astra replaced it. Nothing in that row transfers to the
|
|
115
|
+
new occupant: the seat's identity here is the model, not the slot. Until a
|
|
116
|
+
comparable corpus accumulates for `codex_gpt6-astra`, read the codex row as
|
|
117
|
+
history and treat the new seat as uncalibrated. The same caution applies to
|
|
118
|
+
every row from 2026-09-05 onward for a second reason — the corpus was gathered
|
|
119
|
+
with reviewers at medium effort, and reviewers now run at high (see § Thinking
|
|
120
|
+
Effort Configuration), so post-swap rounds are not directly comparable to it.
|
|
121
|
+
|
|
122
|
+
Selection consequences, each tied to the number above it:
|
|
123
|
+
|
|
124
|
+
- **A `codex` APPROVE carries the most information and an `opus4.6` APPROVE the
|
|
125
|
+
least.** Codex approves 1 design review in 54 and pairs that with the lowest
|
|
126
|
+
advisory rate in the corpus (5 of 140 labelled findings). Opus4.6 approves 20
|
|
127
|
+
document reviews in 23 while 19 of its 27 labelled findings are advisory. Seat
|
|
128
|
+
both, read them differently: the `3/4 APPROVE` threshold is met, in practice,
|
|
129
|
+
with opus4.6's vote already inside it, so the live question is whether two of
|
|
130
|
+
the remaining three agree. (That threshold is a reference figure, not a gate —
|
|
131
|
+
see § Convergence Rules.)
|
|
132
|
+
- **Volume anti-correlates with signal.** The persona seat raises 3,249 of the
|
|
133
|
+
5,286 findings (61%) and 27% of its labelled ones are advisory. Seat personas
|
|
134
|
+
when breadth is wanted; do not seat them to obtain a verdict.
|
|
135
|
+
- **Design draws roughly seven times the P0 of implementation** — median 14 P0
|
|
136
|
+
per design-medium run against 2 per implementation-high run. Budget rounds
|
|
137
|
+
accordingly; a design round returning two P0 is anomalous, not clean.
|
|
138
|
+
- **Dropping a seat does not shorten the round.** Seats run concurrently, so
|
|
139
|
+
wall-clock is the slowest seat (median 179 s per run against 300–400 s summed
|
|
140
|
+
across seats). Removing the 54 s seat saves nothing.
|
|
141
|
+
- **Loops do not converge on their own.** Of 29 artifacts reviewed more than
|
|
142
|
+
once, the last round was REVISE in 26 and APPROVE in 3; rounds per loop median
|
|
143
|
+
3, maximum 12. P0 counts are not monotone: `chain_history_erasure_fix` ran
|
|
144
|
+
25, 25, 21, 14, **38**, 29, 26, 13 across rounds 1–8, and
|
|
145
|
+
`ruby_association_grant_2026_application_ja` returned APPROVE at round 8 with
|
|
146
|
+
3 P0 and then 23 P0 at round 9. Treat a single APPROVE as an observation, not
|
|
147
|
+
as a close — § Convergence Rules already requires the operator's declaration.
|
|
148
|
+
|
|
149
|
+
Two limits of this corpus, both open:
|
|
150
|
+
|
|
151
|
+
- **Findings are not deduplicated across seats.** `cited_by` had exactly one
|
|
152
|
+
entry in 5,286 of 5,286 findings, so agreement between seats is not derivable
|
|
153
|
+
from the store, and the (c) shares above are per-seat rates rather than
|
|
154
|
+
contested classifications.
|
|
155
|
+
- **(a)/(b)/(c) labels are present on 1,622 of 5,286 findings (31%)**, and
|
|
156
|
+
`cursor` labelled 2 of its 730. Classification is applied unevenly by the
|
|
157
|
+
seats, so the shares are computed over labelled findings only.
|
|
158
|
+
|
|
93
159
|
## Step 0.25 — Unknowns Pass (pre-draft, qualifying reviews only)
|
|
94
160
|
|
|
95
161
|
> **Numbering vs timing**: Step 0 and Step 0.5 execute at review time,
|
|
@@ -399,12 +465,17 @@ they disagree, the config is right and this section is stale.
|
|
|
399
465
|
which under the default "delegate" strategy is taken by your persona team
|
|
400
466
|
rather than spawned — so when you are Opus 5, Opus 4.6 is the only Claude
|
|
401
467
|
CLI subprocess
|
|
402
|
-
- [ ] Codex
|
|
468
|
+
- [ ] Codex model: gpt-6-astra, with -m. One codex slot since gpt-5.5 was
|
|
469
|
+
retired 2026-09-05 — do not add a second codex entry expecting the old
|
|
470
|
+
cross-generation pairing
|
|
403
471
|
- [ ] Cursor model: composer-2.5, passed explicitly as --model composer-2.5
|
|
404
|
-
- [ ]
|
|
472
|
+
- [ ] Effort: high on every seat that has an effort control — `--effort high`
|
|
473
|
+
for Claude CLI, `-c model_reasoning_effort=high` for codex. Cursor has no
|
|
474
|
+
effort control and takes no flag
|
|
475
|
+
- [ ] Total reviewer count: 4 (or 3 after orchestrator exclusion from subprocess)
|
|
405
476
|
- [ ] Closing condition: new (a)+(b) P0 = 0, with carryover P0s counted
|
|
406
477
|
separately and a closure verdict on each. The APPROVE ratio the tool
|
|
407
|
-
reports (3/
|
|
478
|
+
reports (3/4 full roster, 2/3 after exclusion) is a reference value,
|
|
408
479
|
not the condition — see § Convergence Rules
|
|
409
480
|
```
|
|
410
481
|
|
|
@@ -423,7 +494,7 @@ Path B refuses such a slot outright; on Path A nothing refuses it but you.
|
|
|
423
494
|
|---------|-----------------|----------------|
|
|
424
495
|
| Launch a reviewer without an explicit model flag | Always pass `--model` / `-m`. A slot with no flag takes the CLI's user-editable default | "The default is the one we want" — it was, until someone changed it outside this repo |
|
|
425
496
|
| Exclude orchestrator model from Agent Team Personas | Agent Team uses orchestrator model — they provide persona diversity, not epistemic diversity | LLM misreads "do not assign yourself as a reviewer" as applying to Agent Team; it applies only to subprocess CLI |
|
|
426
|
-
| Run only Codex GPT-
|
|
497
|
+
| Run only Codex GPT-6-astra, skip 5.5 | Run both — cross-generation entries catch different things (5.5 found §5 schema contradiction in Phase 2 Case A that no other reviewer caught) | Cost-saving heuristic; roster has both for a reason |
|
|
427
498
|
| Use a smaller/cheaper model as Agent Team substitute | Use the orchestrator's own model with different personas | Confusing "model diversity" with "persona diversity" — Agent Team is the latter |
|
|
428
499
|
| Run 3 reviewers instead of the configured roster | Use the full roster from config | Ad-hoc "3 is enough" reasoning; the roster size is empirical |
|
|
429
500
|
| Count a reply that carries only a verdict | Drop it from the denominator, and say why | A bare "APPROVE" looks like agreement and raises the bar for everyone else without contributing (see § Substance and the denominator) |
|
|
@@ -592,7 +663,7 @@ The rule applies **after** orchestrator classifies each finding as (a)/(b)/(c) p
|
|
|
592
663
|
findings count toward the thresholds below; (c) findings are recorded as advisory
|
|
593
664
|
and never block.
|
|
594
665
|
|
|
595
|
-
- **3/
|
|
666
|
+
- **3/4 APPROVE** full roster, or **2/3 APPROVE** after orchestrator exclusion ("exclude" strategy only — the default "delegate" strategy keeps 4 voters via collect) (no (a)/(b) REJECT) = proceed to next step
|
|
596
667
|
- **Any (a) or (b) REJECT or FAIL** = revise and re-review
|
|
597
668
|
- **(c)-only REJECT** = record as advisory, non-blocking
|
|
598
669
|
- **Unanimous APPROVE** (no (a)/(b)) = highest confidence, proceed
|
|
@@ -643,7 +714,7 @@ For normative detail and the underlying classification, see
|
|
|
643
714
|
| Agreement | Meaning | Action |
|
|
644
715
|
|-----------|---------|--------|
|
|
645
716
|
| **N/N** (unanimous) | Architectural-level gap | Must fix |
|
|
646
|
-
| **Majority** (e.g. 3/
|
|
717
|
+
| **Majority** (e.g. 3/4, 2/3) | Implementation-level issue | Should fix |
|
|
647
718
|
| **1/N only** | Specialty-specific insight | Do NOT ignore — often the most novel finding |
|
|
648
719
|
|
|
649
720
|
1/N findings are not "minority opinions to discard." They represent unique expertise.
|
|
@@ -743,7 +814,7 @@ outside this repository — see the incident recorded in § Pre-flight checklist
|
|
|
743
814
|
|
|
744
815
|
| Tool | Command | Prompt Input | Output Collection | Model |
|
|
745
816
|
|------|---------|-------------|-------------------|-------|
|
|
746
|
-
| **Codex** | `codex exec -m <model
|
|
817
|
+
| **Codex** | `codex exec -m <model> -c model_reasoning_effort=high` | stdin pipe: `cat prompt.md \| codex exec -m <model> -` | `-o /path/output.md` | gpt-6-astra — one slot since gpt-5.5 was retired 2026-09-05 |
|
|
747
818
|
| **Cursor Agent** | `agent -p --model composer-2.5` | File reference (stdin NOT supported) | stdout redirect: `> output.md` | composer-2.5, passed explicitly — never relying on the CLI default |
|
|
748
819
|
| **Claude Code** | Agent tool (internal) | Direct prompt string | Write to workspace file | Orchestrator model, or the declared `persona_model` when personas run elsewhere |
|
|
749
820
|
| **Claude CLI (4.6)** | `claude -p --model claude-opus-4-6` | stdin pipe: `cat prompt.md \| claude -p --model claude-opus-4-6` | stdout redirect: `> output.md` | Opus 4.6 — the calibrated anchor, deliberately not a frontier model |
|
|
@@ -766,11 +837,27 @@ Based on cross-evaluation experiment (7 models × 4 tasks + Nomic, 518 CLI calls
|
|
|
766
837
|
|------|-------|-------------|-----------|
|
|
767
838
|
| **Primary (orchestrator)** | session default | (default) | Sufficient for integration, dialogue, judgment |
|
|
768
839
|
| **Reviewer: Agent Team** | = orchestrator, or the declared `persona_model` | (default) | Personas inherit whichever model actually runs them |
|
|
769
|
-
| **Reviewer: Claude CLI** | Opus 4.6, plus any frontier roster slot the orchestrator is not | (
|
|
840
|
+
| **Reviewer: Claude CLI** | Opus 4.6, plus any frontier roster slot the orchestrator is not | `--effort high` (config `effort: high`) | Operator instruction 2026-09-05; supersedes the 2026-04-29 default-effort policy — see the note below the table |
|
|
770
841
|
| **Coding sub-agent** | Opus 5 | `--effort xhigh` | Published starting point for coding/agentic work; not measured here (see note) |
|
|
771
842
|
| **Design sub-agent** | Opus 5 | `--effort high` | Published starting point for intelligence-sensitive work; not measured here (see note) |
|
|
772
|
-
| **Codex** | GPT-
|
|
773
|
-
| **Cursor Agent** | Composer-2.5 | (no flag) |
|
|
843
|
+
| **Codex** | GPT-6-astra / GPT-5.5 | `-c model_reasoning_effort=high` | Same operator instruction. The earlier "(no flag) / fixed effort" entry was wrong: codex_adapter has always emitted this flag when the roster set `effort` |
|
|
844
|
+
| **Cursor Agent** | Composer-2.5 | (no flag) | Genuinely has no effort control — cursor_adapter builds no such flag, so an `effort:` key on a cursor roster entry is recorded and never sent |
|
|
845
|
+
|
|
846
|
+
Effort policy (2026-09-05, operator instruction). Every reviewer that HAS an
|
|
847
|
+
effort control runs at **high**, at every complexity level, and the `effort_map`
|
|
848
|
+
in `config/multi_llm_review.yml` is a constant rather than a function of
|
|
849
|
+
complexity. `high` is deliberate rather than maximal: it is the ceiling the two
|
|
850
|
+
providers share (Claude CLI accepts low/medium/high/xhigh/max, codex accepts
|
|
851
|
+
minimal/low/medium/high), so it is the highest setting at which the roster stays
|
|
852
|
+
comparable across providers.
|
|
853
|
+
|
|
854
|
+
This supersedes the 2026-04-29 policy that kept reviewers at each CLI's default.
|
|
855
|
+
That policy rested on one measurement — low vs high scoring 8.35 vs 8.16 in
|
|
856
|
+
cross-evaluation — taken on the Opus 4.6 / 4.7 generation, on none of the models
|
|
857
|
+
in the current roster. It was not re-measured, so it is superseded by judgement,
|
|
858
|
+
not by a counter-measurement, and a later measurement could reinstate it. The
|
|
859
|
+
cost side is measured: on one identical one-line prompt, gpt-6-astra spent 8,274
|
|
860
|
+
tokens at high against 3,150 at its default (2026-09-05).
|
|
774
861
|
|
|
775
862
|
Note (2026-07-25): the effort experiment data is from the Opus 4.6/4.7
|
|
776
863
|
generation. Opus 5 and Fable 5 effort sensitivity is not yet calibrated;
|
|
@@ -867,13 +954,13 @@ multi_llm_review(
|
|
|
867
954
|
**Dispatcher behavior** (config: `exclude_orchestrator_model: true`, default `true`):
|
|
868
955
|
- If `orchestrator_model` matches a roster entry's `model`, that entry is skipped.
|
|
869
956
|
- `min_quorum` and `convergence_rule` apply to the remaining reviewers.
|
|
870
|
-
-
|
|
957
|
+
- 4-reviewer roster → 3 reviewers; `convergence_rule_after_exclusion: "2/3 APPROVE"`
|
|
871
958
|
(from config) replaces the full-roster rule. This reduced count applies to the
|
|
872
959
|
"exclude" strategy only. The "subprocess" strategy keeps the full roster (the
|
|
873
960
|
matching entry runs as a fresh CLI process instead of being skipped). Under the
|
|
874
961
|
default "delegate" strategy, the matching entry is dropped at dispatch but
|
|
875
|
-
re-added at collect as the persona-team entry, so the voter count returns to
|
|
876
|
-
and the full-roster rule (3/
|
|
962
|
+
re-added at collect as the persona-team entry, so the voter count returns to 4
|
|
963
|
+
and the full-roster rule (3/4 APPROVE) applies.
|
|
877
964
|
- **At most one roster entry leaves for matching the caller.** This is only
|
|
878
965
|
visible on a roster carrying three or more entries on the orchestrator's own
|
|
879
966
|
model: the first is taken over by the persona team, the second leaves as the
|
|
@@ -1342,16 +1429,16 @@ Step 2: Detect environment, and check the roster against config
|
|
|
1342
1429
|
- Read the roster from config/multi_llm_review.yml — do NOT read CLI defaults
|
|
1343
1430
|
and treat them as the roster. Detection only tells you whether a default has
|
|
1344
1431
|
drifted; the model each slot runs is named on the command line.
|
|
1345
|
-
- Report: "Auto mode: Codex (gpt-
|
|
1432
|
+
- Report: "Auto mode: Codex (gpt-6-astra), Cursor (composer-2.5),
|
|
1346
1433
|
Claude Team (orchestrator model), Claude CLI (opus-4.6)"
|
|
1347
1434
|
|
|
1348
|
-
Step 3: Execute the configured roster in parallel (currently
|
|
1435
|
+
Step 3: Execute the configured roster in parallel (currently 4 slots, one of
|
|
1349
1436
|
which is your own persona team)
|
|
1350
|
-
- Bash(background): cat prompt.md | codex exec -m gpt-
|
|
1351
|
-
- Bash(background): cat prompt.md | codex exec -m gpt-5.6-sol -C workspace -o log/review_codex_gpt5.6-sol.md -
|
|
1437
|
+
- Bash(background): cat prompt.md | codex exec -m gpt-6-astra -c model_reasoning_effort=high -C workspace -o log/review_codex_gpt6-astra.md -
|
|
1352
1438
|
- Bash(background): agent -p --trust --model composer-2.5 "Read prompt and review..." > log/review_cursor.md
|
|
1439
|
+
(no effort flag — Cursor has no effort control)
|
|
1353
1440
|
- Agent(background): Claude Team (orchestrator model, e.g. Opus 5) → write to log/review_claude_team_opus5.md
|
|
1354
|
-
- Bash(background): cat prompt.md | claude -p --model claude-opus-4-6 > log/review_claude_opus4.6.md 2>log/review_claude_opus4.6.stderr.log
|
|
1441
|
+
- Bash(background): cat prompt.md | claude -p --model claude-opus-4-6 --effort high > log/review_claude_opus4.6.md 2>log/review_claude_opus4.6.stderr.log
|
|
1355
1442
|
(add a line per further Claude roster slot you are not; with the 2026-07-26
|
|
1356
1443
|
roster an Opus 5 orchestrator has none, so opus-4.6 is the only one)
|
|
1357
1444
|
|
|
@@ -1391,7 +1478,7 @@ log/{artifact}_review{N}_consensus_{date}.md # Consensus analysis
|
|
|
1391
1478
|
```
|
|
1392
1479
|
|
|
1393
1480
|
LLM identifiers: `claude_cli_opus5`, `claude_cli_opus4.6`,
|
|
1394
|
-
`
|
|
1481
|
+
`codex_gpt6-astra`, `cursor_composer2.5`, `cursor_gpt5.4`,
|
|
1395
1482
|
`cursor_premium`. The delegated slot is reported as `claude_team_<model>`
|
|
1396
1483
|
(e.g. `claude_team_claude-opus-5`), assembled at collect time — the roster's
|
|
1397
1484
|
own labels stay CLI-neutral because either frontier entry can take either path.
|
|
@@ -1399,7 +1486,11 @@ own labels stay CLI-neutral because either frontier entry can take either path.
|
|
|
1399
1486
|
`claude_cli_opus4.7`, `cursor_composer2`; retired 2026-07-23: `codex_gpt5.4`;
|
|
1400
1487
|
retired 2026-07-25: `claude_cli_opus4.8`, `claude_team_fable5`;
|
|
1401
1488
|
retired 2026-07-26: `claude_cli_fable5` — five consecutive non-substantive
|
|
1402
|
-
returns, 85-128 characters in 5-7 seconds, no findings and no verdict text
|
|
1489
|
+
returns, 85-128 characters in 5-7 seconds, no findings and no verdict text;
|
|
1490
|
+
retired 2026-09-05: `codex_gpt5.6-sol`, replaced by `codex_gpt6-astra`, and
|
|
1491
|
+
`codex_gpt5.5`, not replaced. Runs recorded under a retired identifier keep it —
|
|
1492
|
+
the label names the model that answered, so renaming old records would attribute
|
|
1493
|
+
one model's findings to another)
|
|
1403
1494
|
|
|
1404
1495
|
## Internal Agent Team Review
|
|
1405
1496
|
|
|
@@ -1695,6 +1786,44 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1695
1786
|
four rounds was the change's account of itself, not the rules it proposed.
|
|
1696
1787
|
Records: L2 `handoff_mlr_l1_norms_revision_three_rounds_and_switch_to_implementation_20260821`
|
|
1697
1788
|
and L2 `mlr_v3_11_0_applied_review_r1_and_subtractive_revision_20260822`
|
|
1789
|
+
- Effort raised to high, and gpt-6-astra replaces gpt-5.6-sol (v3.13.0,
|
|
1790
|
+
2026-09-05, operator instruction). Two changes with one config edit. **Effort**:
|
|
1791
|
+
every seat that has an effort control now runs at high at every complexity
|
|
1792
|
+
level, so `effort_map` is a constant rather than a function of complexity —
|
|
1793
|
+
Claude CLI gets `--effort high`, codex gets `-c model_reasoning_effort=high`,
|
|
1794
|
+
Cursor gets nothing because cursor_adapter builds no effort flag and a value
|
|
1795
|
+
set for it would be recorded and never sent. `high` rather than `xhigh`/`max`
|
|
1796
|
+
because it is the ceiling the two providers share, and a roster split across
|
|
1797
|
+
incomparable settings is worth less than a lower common one. This supersedes
|
|
1798
|
+
the 2026-04-29 default-effort policy, which rested on a single low-vs-high
|
|
1799
|
+
measurement (8.35 vs 8.16) taken on the Opus 4.6 / 4.7 generation — none of
|
|
1800
|
+
the models now in the roster. No counter-measurement was taken, so the
|
|
1801
|
+
supersession is a judgement and a later measurement could reverse it. Cost is
|
|
1802
|
+
measured and rises: gpt-6-astra spent 8,274 tokens at high against 3,150 at
|
|
1803
|
+
its default on one identical one-line prompt. **Roster**: `codex_gpt6-astra`
|
|
1804
|
+
replaces `codex_gpt5.6-sol`, verified before the swap through the flags
|
|
1805
|
+
codex_adapter actually builds. **gpt-5.5 retired in the same edit**, not
|
|
1806
|
+
replaced: it had been the calibrated cross-generation anchor, but it had been
|
|
1807
|
+
commented out of the instance roster since 2026-07-30 "for round R10 only" and
|
|
1808
|
+
never restored, so the anchor role had already lapsed for five weeks and the
|
|
1809
|
+
retirement only names the state. Codex therefore holds ONE slot, and it is
|
|
1810
|
+
uncalibrated — there is no longer a calibrated codex seat to read a new one
|
|
1811
|
+
against. Roster 5 → 4, so `convergence_rule` moves 3/5 → 3/4 and
|
|
1812
|
+
`convergence_rule_after_exclusion` 3/4 → 2/3, both on the same ceil(N × 0.6)
|
|
1813
|
+
basis. Three calibration warnings follow and none is optional: the 138-run
|
|
1814
|
+
seat profile in § Reviewer selection is gpt-5.6-sol's and does not transfer to
|
|
1815
|
+
the new occupant; the whole of that corpus was gathered at medium effort, so
|
|
1816
|
+
rounds from 2026-09-05 onward are not directly comparable to any of it; and
|
|
1817
|
+
the corpus was gathered on a 5-seat roster, so per-round finding counts shift
|
|
1818
|
+
for a third, independent reason. Record (a)/(b)/(c) breakdowns per round in
|
|
1819
|
+
`multi_llm_reviewer_evaluation` until a profile for the new seat accumulates.
|
|
1820
|
+
Recorded because it was got wrong in the session that made the change: the
|
|
1821
|
+
new ratio was first reported to the operator as "3 of 4 seats must APPROVE",
|
|
1822
|
+
as though it were a gate. It is not. § Convergence Rules has said since v3.5
|
|
1823
|
+
that the ratio is neither the only nor the primary close, `Consensus.compute`
|
|
1824
|
+
returns it under the name `reference_verdict`, and both 2026-08 threads closed
|
|
1825
|
+
by (a)+(b) exhaustion without ever reaching it. Every ratio in this document
|
|
1826
|
+
is a reference figure.
|
|
1698
1827
|
|
|
1699
1828
|
**Key insight**: Design reviews and implementation reviews find
|
|
1700
1829
|
**categorically different bugs**. Both phases are necessary.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_reviewer_evaluation
|
|
3
3
|
description: "Multi-LLM reviewer performance evaluation — strengths, weaknesses, value-system biases, and recommended workflows. Based on 185+ reviews (Phase 1, 2026-02 to 03) + Phase 2 Case A 4-round Codex bias study (2026-05-04)."
|
|
4
|
-
version: "1.
|
|
4
|
+
version: "1.6"
|
|
5
5
|
tags:
|
|
6
6
|
- multi-llm
|
|
7
7
|
- review
|
|
@@ -172,8 +172,9 @@ When a reviewer issues a P0, classify the *cause* — not just the severity:
|
|
|
172
172
|
| (b) philosophy-aligned | Deviation from declared design principles (e.g., enumeration where invariant suffices). | **Blocking P0** |
|
|
173
173
|
| (c) value-divergent | Reviewer's own style preference or generic best practice not entailed by project principles. | **Advisory only** (non-blocking) |
|
|
174
174
|
|
|
175
|
-
When uncertain between (b) and (c), default to (c).
|
|
176
|
-
(c) findings are recorded but do not block.
|
|
175
|
+
When uncertain between (b) and (c), default to (c). Only (a)+(b) findings count toward
|
|
176
|
+
closing a round; (c) findings are recorded but do not block. Closing is the exhaustion
|
|
177
|
+
of (a)+(b), not an APPROVE count — see § Convergence Rule (Updated).
|
|
177
178
|
|
|
178
179
|
**Codex ↔ classes**: Codex finds genuine (a) bugs (e.g., the §5 schema contradiction).
|
|
179
180
|
Codex also produces many (c) findings driven by the 3 biases above. The skill of using
|
|
@@ -201,6 +202,9 @@ Final Review: Codex APPROVE | Composer-2.5 APPROVE+ | Claude APPROVE+
|
|
|
201
202
|
- Codex REJECT reasons are always **substantive** (not stylistic): storage model contradictions, missing call sites, fail-open security
|
|
202
203
|
- When Codex finally APPROVEs, all prior FAIL/HIGH issues have been genuinely resolved
|
|
203
204
|
- **Codex APPROVE = strongest merge-readiness signal** in the 3-LLM configuration
|
|
205
|
+
— as a *reference* signal. It is not a gate, and waiting for it is not a
|
|
206
|
+
closing procedure; see the caveat immediately below and § Convergence Rule
|
|
207
|
+
(Updated)
|
|
204
208
|
|
|
205
209
|
> **Note**: The above convergence data is from the 3-reviewer configuration in
|
|
206
210
|
> the Attestation Nudge session. With the 4-reviewer default (Opus 4.7 added
|
|
@@ -216,18 +220,41 @@ Final Review: Codex APPROVE | Composer-2.5 APPROVE+ | Claude APPROVE+
|
|
|
216
220
|
|
|
217
221
|
### Convergence Rule (Updated)
|
|
218
222
|
|
|
219
|
-
The
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
223
|
+
**The APPROVE ratio is not the convergence criterion.** A round closes on the
|
|
224
|
+
exhaustion of (a)+(b) findings, declared by the operator. The machine-side signal
|
|
225
|
+
is **new (a)+(b) P0 = 0**, counting carryover P0s separately; nothing in the
|
|
226
|
+
multi_llm_review SkillSet computes it and no returned field carries it, so it is
|
|
227
|
+
read off the findings. `Consensus.compute` returns the ratio under the name
|
|
228
|
+
`reference_verdict` and the vote counts under `vote_tally` — both recorded
|
|
229
|
+
observations, neither a conclusion.
|
|
230
|
+
|
|
231
|
+
The evidence that the ratio cannot serve as the criterion is in this document:
|
|
232
|
+
two Codex entries went 24 of 24 reviews without reaching APPROVE on one design
|
|
233
|
+
loop, and both 2026-08 review threads closed by (a)+(b) exhaustion plus an
|
|
234
|
+
operator freeze declaration without ever reaching their ratio. A threshold a seat
|
|
235
|
+
is structurally unlikely to meet cannot be what closes a round.
|
|
236
|
+
|
|
237
|
+
Everything below applies **after** the orchestrator classifies findings as
|
|
238
|
+
(a)/(b)/(c) per § Reviewer Value-System Divergence. A REJECT whose findings are
|
|
239
|
+
entirely (c) value-divergent is recorded but non-blocking; only (a)+(b) findings
|
|
240
|
+
count.
|
|
241
|
+
|
|
242
|
+
- Any (a) or (b) REJECT or FAIL = revise and re-review. **This one blocks.**
|
|
228
243
|
- Codex-only REJECT with (a)/(b) findings + others APPROVE = likely real issue, investigate before overriding
|
|
229
244
|
- Codex REJECT with only (c) findings = expected per Codex value-system divergence; non-blocking
|
|
230
245
|
|
|
246
|
+
Reference figures, recorded and never sufficient on their own. Read what the
|
|
247
|
+
approving replies actually said before counting them:
|
|
248
|
+
|
|
249
|
+
- 3/4 APPROVE (no (a)/(b) REJECT) on the current 4-reviewer roster — the literal
|
|
250
|
+
ratio tracks the roster size and changed on 2026-09-05 when gpt-5.5 retired
|
|
251
|
+
- 4/4 APPROVE (including Codex) = the strongest reference signal available, still
|
|
252
|
+
a reference
|
|
253
|
+
- Legacy 3-reviewer mode: 2/3 APPROVE
|
|
254
|
+
|
|
255
|
+
Normative statement and the carryover/new split: L1 `multi_llm_review_workflow`
|
|
256
|
+
§ Convergence Rules. Aggregation rule: project CLAUDE.md.
|
|
257
|
+
|
|
231
258
|
### Bug Category Differentiation Across Rounds
|
|
232
259
|
|
|
233
260
|
| Review Phase | Typical Bug Category | Example |
|
|
@@ -324,6 +351,26 @@ MECHANISM of a sound invariant is (c) -> §11 / implementation review." This con
|
|
|
324
351
|
Codex APPROVE" (not always reachable) into "freeze when only (c)/mechanism findings remain,"
|
|
325
352
|
which is decidable by the orchestrator and resistant to value-divergence stalling.
|
|
326
353
|
|
|
354
|
+
## Changelog
|
|
355
|
+
|
|
356
|
+
- **v1.6 (2026-09-06)**: § Convergence Rule (Updated) rewritten. It had stated
|
|
357
|
+
`3/4 APPROVE = proceed to next step` and `4/4 APPROVE = merge-ready` with no
|
|
358
|
+
note that the ratio is a reference value — while L1 `multi_llm_review_workflow`
|
|
359
|
+
§ Convergence Rules names *this* section as the normative detail it defers to.
|
|
360
|
+
So the document that says "the ratio is not the primary close" pointed at a
|
|
361
|
+
document that said it was. Now: the closing condition (new (a)+(b) P0 = 0,
|
|
362
|
+
carryover counted separately, operator declares the freeze) is stated first and
|
|
363
|
+
the blocking rule — any (a)/(b) REJECT — is separated from the reference
|
|
364
|
+
figures. Two supporting facts moved into the section because they are the
|
|
365
|
+
reason the ratio cannot be the criterion: Codex went 24 of 24 reviews without
|
|
366
|
+
APPROVE on one design loop, and both 2026-08 threads closed by (a)+(b)
|
|
367
|
+
exhaustion without reaching their ratio. The "Codex APPROVE = strongest
|
|
368
|
+
merge-readiness signal" line keeps its wording and gains "as a reference
|
|
369
|
+
signal, not a gate". Prompted by the operator after this agent reported an
|
|
370
|
+
APPROVE ratio to them as a gate for the third time; the wrong belief was
|
|
371
|
+
traceable to the text here, not to the tooling, which had been correct since
|
|
372
|
+
v0.7 INV-R2 (`reference_verdict`) and v0.10.1 (`vote_tally`).
|
|
373
|
+
|
|
327
374
|
## Refinement Source
|
|
328
375
|
|
|
329
376
|
Profiles in this knowledge are refined from accumulated L2 contexts named with prefix
|
|
@@ -2424,9 +2424,15 @@ module KairosMcp
|
|
|
2424
2424
|
"Reviewer roster hash: #{bundle['reviewer_roster_hash']}",
|
|
2425
2425
|
"Config hash: #{bundle['config_hash']}",
|
|
2426
2426
|
'',
|
|
2427
|
-
'## Run each reviewer independently
|
|
2427
|
+
'## Run each reviewer independently, then aggregate.',
|
|
2428
2428
|
'',
|
|
2429
|
-
|
|
2429
|
+
'## Closing condition',
|
|
2430
|
+
'A round closes on new (a)+(b) P0 = 0, counting carryover P0s',
|
|
2431
|
+
'separately with a closure verdict on each, and the freeze is',
|
|
2432
|
+
'declared by the operator. Read it off the findings — no field',
|
|
2433
|
+
'in this bundle carries it.',
|
|
2434
|
+
'',
|
|
2435
|
+
"Reference ratio (recorded, never sufficient on its own): #{bundle['convergence_rule']}",
|
|
2430
2436
|
'',
|
|
2431
2437
|
'## Aggregation instructions',
|
|
2432
2438
|
bundle['aggregation_instructions'].to_s,
|
|
@@ -48,6 +48,12 @@
|
|
|
48
48
|
# ruby bench/minimum_nomic/run_gm.rb --out /tmp/smoke --turns 2 # smoke
|
|
49
49
|
# ruby bench/minimum_nomic/check_gm.rb log/minimum_nomic_gm_20260810/g3 --falsify
|
|
50
50
|
#
|
|
51
|
+
# Seating a different model, or a different provider in turn control:
|
|
52
|
+
# --codex-model gpt-6-astra --codex-effort high # B seat
|
|
53
|
+
# --gm-adapter codex --gm-model gpt-6-astra --gm-effort high
|
|
54
|
+
# Both go into the lineup, so a game says which generation played without the
|
|
55
|
+
# reader having to know when this file was edited.
|
|
56
|
+
#
|
|
51
57
|
# One directory per game, and a fresh one every time: the run refuses to start
|
|
52
58
|
# when the target already holds records. The games themselves are never
|
|
53
59
|
# committed — this file is tracked, the corpus under log/ is not.
|
|
@@ -120,10 +126,47 @@ LC = KairosMcp::SkillSets::LlmClient
|
|
|
120
126
|
# Read from ARGV here rather than in the option parser at the foot of the file,
|
|
121
127
|
# because PLAYER_SPECS is a constant that the prompts and the lineup are built
|
|
122
128
|
# from and it has to exist before either.
|
|
129
|
+
# Which model sits in the claude seat is also a run parameter, for the same
|
|
130
|
+
# reason. The 61-game corpus of 2026-08 was produced by editing the constant
|
|
131
|
+
# below in place, so "which generation played" is recoverable only from each
|
|
132
|
+
# game's own lineup row and never from this file's history. --claude-model and
|
|
133
|
+
# --claude-effort put that choice on the command line and therefore in the
|
|
134
|
+
# lineup. Defaults are unchanged, so an unflagged run stays comparable.
|
|
135
|
+
def arg_after(flag)
|
|
136
|
+
i = ARGV.index(flag)
|
|
137
|
+
i && ARGV[i + 1] && !ARGV[i + 1].start_with?('--') ? ARGV[i + 1] : nil
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
CLAUDE_SEAT_MODEL = arg_after('--claude-model') || 'claude-opus-5'
|
|
141
|
+
CLAUDE_SEAT_EFFORT = arg_after('--claude-effort') || 'high'
|
|
142
|
+
GM_MODEL = arg_after('--gm-model') || 'claude-opus-5'
|
|
143
|
+
GM_EFFORT = arg_after('--gm-effort') || 'high'
|
|
144
|
+
|
|
145
|
+
# The codex seat carries a model name for the same reason the claude seat does.
|
|
146
|
+
# gpt-5.6-sol sat there through the whole 2026-08 corpus and was written into
|
|
147
|
+
# the constant, so seating a later generation meant editing this file, and the
|
|
148
|
+
# edit left no trace anywhere except in each game's own lineup row. The effort
|
|
149
|
+
# default stays nil rather than becoming a level, because every stored game ran
|
|
150
|
+
# this seat without one and a default would silently change what "unflagged"
|
|
151
|
+
# means.
|
|
152
|
+
CODEX_SEAT_MODEL = arg_after('--codex-model') || 'gpt-5.6-sol'
|
|
153
|
+
CODEX_SEAT_EFFORT = arg_after('--codex-effort')
|
|
154
|
+
|
|
155
|
+
# The game master's ADAPTER, not only its model. Turn control ran on claude_code
|
|
156
|
+
# in every stored game, so "can a different provider hold turn control" cannot
|
|
157
|
+
# be asked by changing the model name alone: a codex model name handed to the
|
|
158
|
+
# claude_code adapter reaches a CLI that cannot serve it, and the run then
|
|
159
|
+
# measures the mismatch instead of the model. Moving both together is the only
|
|
160
|
+
# form of the question that has an answer. The adapter is recorded in the
|
|
161
|
+
# lineup beside the model, so a reader never has to infer it from the name.
|
|
162
|
+
GM_ADAPTER = arg_after('--gm-adapter') || 'claude_code'
|
|
163
|
+
|
|
123
164
|
SEAT_POOL = {
|
|
124
|
-
'cursor' => { adapter: 'cursor', model: 'composer-2.5',
|
|
125
|
-
'codex' => { adapter: 'codex', model:
|
|
126
|
-
|
|
165
|
+
'cursor' => { adapter: 'cursor', model: 'composer-2.5', effort: nil },
|
|
166
|
+
'codex' => { adapter: 'codex', model: CODEX_SEAT_MODEL,
|
|
167
|
+
effort: CODEX_SEAT_EFFORT },
|
|
168
|
+
'claude_code' => { adapter: 'claude_code', model: CLAUDE_SEAT_MODEL,
|
|
169
|
+
effort: CLAUDE_SEAT_EFFORT }
|
|
127
170
|
}.freeze
|
|
128
171
|
|
|
129
172
|
DEFAULT_SEAT_ORDER = %w[cursor codex claude_code].freeze
|
|
@@ -141,7 +184,17 @@ PLAYER_SPECS = SEAT_ORDER.each_with_index.map do |key, i|
|
|
|
141
184
|
{ id: %w[A B C].fetch(i) }.merge(SEAT_POOL.fetch(key))
|
|
142
185
|
end.freeze
|
|
143
186
|
|
|
144
|
-
GM_SPEC = { id: 'GM', adapter:
|
|
187
|
+
GM_SPEC = { id: 'GM', adapter: GM_ADAPTER, model: GM_MODEL, effort: GM_EFFORT }.freeze
|
|
188
|
+
|
|
189
|
+
# Who can reach this run's own record, over every participant rather than over
|
|
190
|
+
# the players alone. claude_code is the only contained adapter: it is chdired
|
|
191
|
+
# to an empty directory with no tools, so the record is not under its cwd and
|
|
192
|
+
# it has nothing to open it with. codex has a read-only sandbox rooted at the
|
|
193
|
+
# project root and cursor has no sandbox flag at all, so either one can read
|
|
194
|
+
# the record from any seat — the game master's included. Used by the lineup.
|
|
195
|
+
ALL_PARTICIPANTS = (PLAYER_SPECS + [GM_SPEC]).freeze
|
|
196
|
+
CONTAINED = ALL_PARTICIPANTS.select { |s| s[:adapter] == 'claude_code' }.freeze
|
|
197
|
+
UNCONTAINED = ALL_PARTICIPANTS.reject { |s| s[:adapter] == 'claude_code' }.freeze
|
|
145
198
|
|
|
146
199
|
# The analyst roster equals the player roster (v0.8 §6), so self-analysis is
|
|
147
200
|
# part of the output and is recorded as such.
|
|
@@ -420,13 +473,49 @@ class Run
|
|
|
420
473
|
def call!
|
|
421
474
|
load_initial_rules!
|
|
422
475
|
write_lineup!
|
|
476
|
+
probe_identity!('before_game')
|
|
423
477
|
halt = play!
|
|
478
|
+
probe_identity!('after_game')
|
|
424
479
|
analyse! unless @utterances.empty?
|
|
425
480
|
write_summary!(halt)
|
|
426
481
|
@recorder.close
|
|
427
482
|
halt
|
|
428
483
|
end
|
|
429
484
|
|
|
485
|
+
# ── identity probe ──────────────────────────────────────────────────────────
|
|
486
|
+
#
|
|
487
|
+
# Asks each seat, in its own words, which model it is — once before the game
|
|
488
|
+
# and once after. This is SELF-REPORT and is recorded as such: a model that
|
|
489
|
+
# was silently re-routed has no way to know it, and a model that was not can
|
|
490
|
+
# still name itself wrongly. It is kept because it is the answer a reader
|
|
491
|
+
# would otherwise assume, and having it written down beside the transport's
|
|
492
|
+
# account makes the two comparable instead of leaving one of them imagined.
|
|
493
|
+
# The transport account (`model_observed` on every call) is the evidence.
|
|
494
|
+
#
|
|
495
|
+
# It is a separate call and touches no player or game-master prompt. Putting
|
|
496
|
+
# "say which model you are" into the game prompt would tell a player
|
|
497
|
+
# something about itself that no stored game told its players, and every
|
|
498
|
+
# game recorded before this would stop being comparable.
|
|
499
|
+
IDENTITY_PROBE = <<~P.strip
|
|
500
|
+
Which model are you? Reply with the exact model identifier you were invoked
|
|
501
|
+
as, on one line, and nothing else.
|
|
502
|
+
P
|
|
503
|
+
|
|
504
|
+
def probe_identity!(when_label)
|
|
505
|
+
(PLAYER_SPECS + [GM_SPEC]).each do |spec|
|
|
506
|
+
reply = call_llm(spec, [{ 'role' => 'user', 'content' => IDENTITY_PROBE }],
|
|
507
|
+
kind: 'probe', purpose: "identity_#{when_label}")
|
|
508
|
+
@recorder.write('identity_probes', {
|
|
509
|
+
'at' => now_stamp, 'party' => spec[:id], 'when' => when_label,
|
|
510
|
+
'requested_model' => spec[:model], 'effort' => spec[:effort],
|
|
511
|
+
'self_reported' => reply&.strip,
|
|
512
|
+
'evidential_status' => 'self-report; not evidence of which model answered. ' \
|
|
513
|
+
'See model_observed on the matching row in calls.jsonl'
|
|
514
|
+
})
|
|
515
|
+
puts "probe #{when_label}: #{spec[:id]} (#{spec[:model]}) said #{reply&.strip.inspect}"
|
|
516
|
+
end
|
|
517
|
+
end
|
|
518
|
+
|
|
430
519
|
private
|
|
431
520
|
|
|
432
521
|
# ── adapters ────────────────────────────────────────────────────────────────
|
|
@@ -485,6 +574,51 @@ class Run
|
|
|
485
574
|
row['output_tokens'] = res['output_tokens']
|
|
486
575
|
row['token_absence_reason'] =
|
|
487
576
|
"the #{spec[:adapter]} adapter returns no usage counts" if res['input_tokens'].nil?
|
|
577
|
+
|
|
578
|
+
# WHICH MODEL ACTUALLY ANSWERED. `model` above echoes the request; these
|
|
579
|
+
# fields are the transport's own account of what produced the output
|
|
580
|
+
# tokens, taken from the CLI's modelUsage envelope. They exist because a
|
|
581
|
+
# request can be served by a different model than the one asked for —
|
|
582
|
+
# a safety re-route, a fast-mode substitution, a CLI fallback — and
|
|
583
|
+
# nothing else in this record would show it. A seat's identity is then a
|
|
584
|
+
# measured fact per call rather than a claim made once in the lineup.
|
|
585
|
+
#
|
|
586
|
+
# Only the claude_code adapter reports this; the codex and cursor CLIs
|
|
587
|
+
# return no usage envelope, so their rows carry the reason instead of a
|
|
588
|
+
# silent nil.
|
|
589
|
+
row['model_observed'] = res['model_observed']
|
|
590
|
+
row['model_usage'] = res['model_usage']
|
|
591
|
+
row['fast_mode_state'] = res['fast_mode_state']
|
|
592
|
+
row['api_error_status'] = res['api_error_status']
|
|
593
|
+
|
|
594
|
+
# `model_observed` is the adapter's guess: the envelope entry with the
|
|
595
|
+
# most output tokens. It is wrong for short replies. The CLI places an
|
|
596
|
+
# internal helper call (claude-haiku, ~10-17 output tokens against a
|
|
597
|
+
# fixed ~918-token input) beside the main one, so a reply of a dozen
|
|
598
|
+
# tokens loses the comparison to the helper and the call is attributed
|
|
599
|
+
# to a model that answered nothing. Measured on the 2026-09-02 smoke
|
|
600
|
+
# run: 3 of 7 claude calls misattributed this way, all three of them
|
|
601
|
+
# one-line replies, while the 5,691-token analysis was attributed
|
|
602
|
+
# correctly.
|
|
603
|
+
#
|
|
604
|
+
# `model_served_request` does not compare sizes. It asks whether the
|
|
605
|
+
# model that was requested is present in the envelope having produced
|
|
606
|
+
# output at all, which is the question, and is unaffected by how long
|
|
607
|
+
# the reply was. This is the field to read for "did the seat's model
|
|
608
|
+
# actually answer"; `model_observed` is kept beside it unchanged so the
|
|
609
|
+
# two accounts stay distinguishable.
|
|
610
|
+
usage = res['model_usage'] || {}
|
|
611
|
+
row['models_in_envelope'] = usage.empty? ? nil : usage.transform_values { |u|
|
|
612
|
+
(u || {})['outputTokens']
|
|
613
|
+
}
|
|
614
|
+
row['model_served_request'] =
|
|
615
|
+
usage.empty? ? nil : (usage.dig(spec[:model], 'outputTokens').to_i.positive?)
|
|
616
|
+
row['model_matches_request'] =
|
|
617
|
+
res['model_observed'].nil? ? nil : (res['model_observed'] == spec[:model])
|
|
618
|
+
if usage.empty?
|
|
619
|
+
row['model_observation_absence_reason'] =
|
|
620
|
+
"the #{spec[:adapter]} adapter returns no model usage envelope"
|
|
621
|
+
end
|
|
488
622
|
rescue StandardError => e
|
|
489
623
|
row['ok'] = false
|
|
490
624
|
row['error'] = "#{e.class}: #{e.message}"
|
|
@@ -589,12 +723,16 @@ class Run
|
|
|
589
723
|
# Derived from the seat assignment, not written out: --seats moves the
|
|
590
724
|
# contained adapter off C, and a hardcoded "C is contained" would then
|
|
591
725
|
# be a false statement in the record about which seats could read it.
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
726
|
+
# The game master is derived the same way and for the same reason.
|
|
727
|
+
# Until 2026-09-05 it was appended to contained_seats as a literal,
|
|
728
|
+
# which was true only while its adapter could not be changed; with
|
|
729
|
+
# --gm-adapter it would have become the very false statement this
|
|
730
|
+
# derivation exists to prevent, and a codex or cursor game master can
|
|
731
|
+
# read this directory exactly as a player on that adapter can.
|
|
732
|
+
'contained_seats' => CONTAINED.map { |s| s[:id] },
|
|
733
|
+
'uncontained_seats' => UNCONTAINED.map { |s| s[:id] },
|
|
734
|
+
'statement' => 'containment is by seat capability, not by file location; ' \
|
|
735
|
+
"#{UNCONTAINED.empty? ? 'no participant' : "seats #{UNCONTAINED.map { |s| s[:id] }.join(' and ')}"} " \
|
|
598
736
|
'could read this directory if they looked. Not fixed, recorded.',
|
|
599
737
|
'write_mode' => 'append-only, one game per directory; the run refuses to start when the ' \
|
|
600
738
|
'directory already holds records, so a game is never destroyed and two ' \
|
|
@@ -933,6 +1071,27 @@ class Run
|
|
|
933
1071
|
'calls_failed_by_participant' => calls.reject { |c| c['ok'] }
|
|
934
1072
|
.group_by { |c| c['participant'] }
|
|
935
1073
|
.transform_values(&:length),
|
|
1074
|
+
# Which model answered, counted rather than assumed. A run where every
|
|
1075
|
+
# claude call was served by the model that was asked for reads
|
|
1076
|
+
# observed_model_mismatches: 0 with a denominator; a re-routed run reads
|
|
1077
|
+
# the substitute's name in observed_models_by_participant.
|
|
1078
|
+
'observed_models_by_participant' => calls.select { |c| c['models_in_envelope'] }
|
|
1079
|
+
.group_by { |c| c['participant'] }
|
|
1080
|
+
.transform_values { |rows|
|
|
1081
|
+
rows.flat_map { |c|
|
|
1082
|
+
c['models_in_envelope'].keys
|
|
1083
|
+
}.tally
|
|
1084
|
+
},
|
|
1085
|
+
'calls_with_model_observation' => calls.count { |c| c['models_in_envelope'] },
|
|
1086
|
+
# The one to read. Counts calls where the requested model produced no
|
|
1087
|
+
# output in the envelope — a genuine substitution.
|
|
1088
|
+
'calls_requested_model_did_not_answer' =>
|
|
1089
|
+
calls.count { |c| c['model_served_request'] == false },
|
|
1090
|
+
# The size-comparison heuristic's disagreement rate, kept for calibration
|
|
1091
|
+
# of that heuristic only. A nonzero count here with zero above means the
|
|
1092
|
+
# short-reply misattribution, not a substitution.
|
|
1093
|
+
'observed_model_mismatches' => calls.count { |c| c['model_matches_request'] == false },
|
|
1094
|
+
'fast_mode_states' => calls.map { |c| c['fast_mode_state'] }.compact.tally,
|
|
936
1095
|
'gm_reasks' => calls.count { |c| c['purpose'] == 'turn_control_reask' },
|
|
937
1096
|
'gm_turns_unreadable' => @gm_turns.count { |g| !g['readable'] },
|
|
938
1097
|
'deliveries_recorded' => File.readlines(File.join(@out, 'records', 'deliveries.jsonl')).length,
|
|
@@ -970,6 +1129,15 @@ if __FILE__ == $PROGRAM_NAME
|
|
|
970
1129
|
# Consumed at the head of this file, where PLAYER_SPECS is built. Declared
|
|
971
1130
|
# here only so that parse! accepts it instead of dying on an unknown flag.
|
|
972
1131
|
o.on('--seats LIST') { |_| }
|
|
1132
|
+
# Likewise consumed at the head of the file, where SEAT_POOL and GM_SPEC
|
|
1133
|
+
# are built.
|
|
1134
|
+
o.on('--claude-model MODEL') { |_| }
|
|
1135
|
+
o.on('--claude-effort LEVEL') { |_| }
|
|
1136
|
+
o.on('--codex-model MODEL') { |_| }
|
|
1137
|
+
o.on('--codex-effort LEVEL') { |_| }
|
|
1138
|
+
o.on('--gm-model MODEL') { |_| }
|
|
1139
|
+
o.on('--gm-effort LEVEL') { |_| }
|
|
1140
|
+
o.on('--gm-adapter NAME') { |_| }
|
|
973
1141
|
end.parse!(ARGV)
|
|
974
1142
|
|
|
975
1143
|
started = Time.now
|
|
@@ -4,12 +4,21 @@
|
|
|
4
4
|
# to avoid duplication.
|
|
5
5
|
|
|
6
6
|
# Convergence rules
|
|
7
|
-
#
|
|
8
|
-
#
|
|
7
|
+
#
|
|
8
|
+
# READ THIS FIRST: the ratio below is NOT the condition on which a round
|
|
9
|
+
# closes. Consensus.compute returns it as `reference_verdict` (v0.7 INV-R2) —
|
|
10
|
+
# a recorded observation. A round closes on the exhaustion of (a)+(b) findings,
|
|
11
|
+
# declared by the operator, and the machine-side signal is "new (a)+(b) P0 = 0",
|
|
12
|
+
# which nothing in this SkillSet computes and no returned field carries. Both
|
|
13
|
+
# 2026-08 review threads closed that way without ever reaching their ratio. See
|
|
14
|
+
# L1 multi_llm_review_workflow § Convergence Rules.
|
|
15
|
+
#
|
|
16
|
+
# Roster has 4 reviewers (claude_cli_opus5, claude_cli_opus4.6,
|
|
17
|
+
# codex_gpt6-astra, cursor_composer2.5).
|
|
9
18
|
# Rules are ratio-based (parser interprets "N/M" as N/M fraction applied
|
|
10
19
|
# to successful count), so the literal numerator/denominator is
|
|
11
20
|
# informational; what matters is the ratio.
|
|
12
|
-
convergence_rule: "3/
|
|
21
|
+
convergence_rule: "3/4 APPROVE" # ceil(4 * 0.6) = 3 of the 4-reviewer roster
|
|
13
22
|
min_quorum: 2 # minimum successful reviews for any verdict
|
|
14
23
|
|
|
15
24
|
# Self-referential orchestrator exclusion.
|
|
@@ -32,15 +41,16 @@ exclude_orchestrator_model: true
|
|
|
32
41
|
# "exclude": legacy behavior — drop the matching reviewer entirely.
|
|
33
42
|
# "subprocess": spawn fresh claude -p for the matching reviewer.
|
|
34
43
|
default_orchestrator_strategy: "delegate"
|
|
35
|
-
# After excluding 1 orchestrator from
|
|
36
|
-
# majority basis (ceil(
|
|
37
|
-
# not the same literal ratio (3/
|
|
38
|
-
# applies the ratio to the successful count,
|
|
39
|
-
#
|
|
44
|
+
# After excluding 1 orchestrator from 4 → 3 reviewers. Same ceil(N * 0.6)
|
|
45
|
+
# majority basis (ceil(3 * 0.6) = 2 → 2 of 3 must APPROVE). The two rules are
|
|
46
|
+
# not the same literal ratio (3/4 = 0.75 vs 2/3 = 0.67); since the parser
|
|
47
|
+
# applies the ratio to the successful count, they diverge when some reviewers
|
|
48
|
+
# fail. Accepted as-is, and cheaply so now: both values are reference figures,
|
|
49
|
+
# not gates.
|
|
40
50
|
# Applies to the "exclude" strategy only — "subprocess" keeps the full
|
|
41
51
|
# roster, and "delegate" re-adds the orchestrator slot at collect, so the
|
|
42
|
-
# full-roster rule (3/
|
|
43
|
-
convergence_rule_after_exclusion: "3
|
|
52
|
+
# full-roster rule (3/4) governs both (verified live 2026-06-10).
|
|
53
|
+
convergence_rule_after_exclusion: "2/3 APPROVE"
|
|
44
54
|
|
|
45
55
|
# Two-phase delegation (orchestrator_strategy: "delegate").
|
|
46
56
|
# Phase 1 dispatches subprocess reviewers synchronously, persists their
|
|
@@ -96,9 +106,15 @@ max_concurrent: 2 # semaphore limit (2 for laptop, 4 for CI)
|
|
|
96
106
|
# about, so an omission stops the run instead of quietly changing who reviewed.
|
|
97
107
|
# effort is optional. role_label is used in consensus output.
|
|
98
108
|
#
|
|
99
|
-
# Default effort per reviewer (
|
|
100
|
-
# tool receives a `complexity` argument (or auto-detects it),
|
|
101
|
-
# are overridden per-dispatch by the effort_map below
|
|
109
|
+
# Default effort per reviewer (high baseline since 2026-09-05). When the
|
|
110
|
+
# multi_llm_review tool receives a `complexity` argument (or auto-detects it),
|
|
111
|
+
# these defaults are overridden per-dispatch by the effort_map below — so the
|
|
112
|
+
# effort_map is the value that actually reaches the CLI, and both are set to
|
|
113
|
+
# high. A roster entry left at medium here would be silently raised by the map.
|
|
114
|
+
#
|
|
115
|
+
# Cursor entries carry no `effort` key on purpose: cursor_adapter has no effort
|
|
116
|
+
# handling at all, so the key would be accepted here and dropped at dispatch,
|
|
117
|
+
# recording an effort the run never used.
|
|
102
118
|
reviewers:
|
|
103
119
|
# Rotating orchestrator slots (Opus 5 / Fable 5, 2026-07-25).
|
|
104
120
|
# Both frontier Claude models sit in the roster. Whichever one is the
|
|
@@ -112,7 +128,7 @@ reviewers:
|
|
|
112
128
|
# PersonaAssembly at collect time.
|
|
113
129
|
- provider: claude_code
|
|
114
130
|
model: claude-opus-5
|
|
115
|
-
effort:
|
|
131
|
+
effort: high
|
|
116
132
|
role_label: claude_cli_opus5
|
|
117
133
|
|
|
118
134
|
# Fable 5 retired from the roster 2026-07-26 after five consecutive
|
|
@@ -130,7 +146,7 @@ reviewers:
|
|
|
130
146
|
# anchor on the Claude side, deliberately not a frontier model.
|
|
131
147
|
- provider: claude_code
|
|
132
148
|
model: claude-opus-4-6
|
|
133
|
-
effort:
|
|
149
|
+
effort: high
|
|
134
150
|
role_label: claude_cli_opus4.6
|
|
135
151
|
|
|
136
152
|
# Opus 4.8 retired 2026-07-25 when Opus 5 entered the roster (same
|
|
@@ -142,28 +158,43 @@ reviewers:
|
|
|
142
158
|
# (a)/(b)/(c) breakdowns per round (see multi_llm_reviewer_evaluation)
|
|
143
159
|
# until a profile accumulates.
|
|
144
160
|
|
|
145
|
-
# GPT-
|
|
146
|
-
#
|
|
147
|
-
#
|
|
148
|
-
#
|
|
149
|
-
#
|
|
161
|
+
# GPT-6-astra added 2026-09-05 at the operator's instruction, replacing
|
|
162
|
+
# gpt-5.6-sol (which had held this slot since 2026-07-23, itself replacing
|
|
163
|
+
# gpt-5.4). To restore either, set model and role_label back. Verified before
|
|
164
|
+
# the swap through the flags this adapter actually builds:
|
|
165
|
+
# codex exec --sandbox read-only -m gpt-6-astra -c model_reasoning_effort=high
|
|
166
|
+
# answered with `model: gpt-6-astra` / `reasoning effort: high` in its header.
|
|
167
|
+
#
|
|
168
|
+
# gpt-6-astra's reviewer bias profile is not calibrated — record (a)/(b)/(c)
|
|
169
|
+
# breakdowns per round (see multi_llm_reviewer_evaluation) until one
|
|
170
|
+
# accumulates. The 138-run profile recorded for codex_gpt5.6-sol in L1
|
|
171
|
+
# multi_llm_review_workflow belongs to the retired model and does NOT carry
|
|
172
|
+
# over to this slot. Since gpt-5.5 was retired the same day, codex holds ONE
|
|
173
|
+
# slot and it is uncalibrated — there is no longer a calibrated codex anchor
|
|
174
|
+
# to read a new seat against.
|
|
150
175
|
- provider: codex
|
|
151
|
-
model: gpt-
|
|
152
|
-
effort:
|
|
153
|
-
role_label:
|
|
176
|
+
model: gpt-6-astra
|
|
177
|
+
effort: high
|
|
178
|
+
role_label: codex_gpt6-astra
|
|
154
179
|
|
|
155
|
-
# GPT-5.5 added 2026-04-24
|
|
156
|
-
#
|
|
157
|
-
#
|
|
180
|
+
# GPT-5.5 (added 2026-04-24) RETIRED 2026-09-05 at the operator's
|
|
181
|
+
# instruction. It had held the second codex slot as the calibrated
|
|
182
|
+
# cross-generation anchor. In practice it had not run since 2026-07-30, when
|
|
183
|
+
# it was commented out of the instance roster "for round R10 only" and never
|
|
184
|
+
# restored — so the anchor role it was kept for had already lapsed for five
|
|
185
|
+
# weeks before this made the lapse explicit. The roster drops from 5 to 4 and
|
|
186
|
+
# codex now holds one slot, gpt-6-astra.
|
|
158
187
|
#
|
|
159
|
-
#
|
|
160
|
-
#
|
|
161
|
-
# (
|
|
162
|
-
#
|
|
163
|
-
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
188
|
+
# To restore, re-add an entry with model gpt-5.5, effort high, role_label
|
|
189
|
+
# codex_gpt5.5, and widen convergence_rule back to a 5-seat basis
|
|
190
|
+
# (ceil(5 * 0.6) = 3, "3/5 APPROVE"; after exclusion "3/4 APPROVE").
|
|
191
|
+
#
|
|
192
|
+
# Transport backend (2026-07-03), retained for whichever codex entries exist:
|
|
193
|
+
# they default to the `codex exec` CLI, and the adapter passes `-m <model>` so
|
|
194
|
+
# the response can be attributed per-model. Add `backend: mcp` to route a codex
|
|
195
|
+
# reviewer through `codex mcp-server` instead (CodexMcpAdapter, read-only
|
|
196
|
+
# sandbox). Opt-in for A/B against the CLI path; keep the default CLI until MCP
|
|
197
|
+
# parity is confirmed (see log 20260702 Appendix F).
|
|
167
198
|
|
|
168
199
|
# Model pinned explicitly 2026-07-27. This entry previously carried no
|
|
169
200
|
# `model` key and relied on the cursor CLI default, which is user-editable
|
|
@@ -222,7 +253,7 @@ reviewers:
|
|
|
222
253
|
escalation_reviewers:
|
|
223
254
|
- provider: claude_code
|
|
224
255
|
model: claude-fable-5
|
|
225
|
-
effort:
|
|
256
|
+
effort: high
|
|
226
257
|
role_label: claude_cli_fable5
|
|
227
258
|
|
|
228
259
|
# INV-E2 (substance) has no setting here on purpose. A reply counts as a review
|
|
@@ -237,31 +268,45 @@ escalation_reviewers:
|
|
|
237
268
|
# or accepts an explicit `complexity` input. The mapped effort overrides the
|
|
238
269
|
# reviewer roster's default.
|
|
239
270
|
#
|
|
240
|
-
# Policy (2026-
|
|
241
|
-
#
|
|
242
|
-
#
|
|
243
|
-
#
|
|
244
|
-
#
|
|
271
|
+
# Policy (2026-09-05, operator instruction): every reviewer that HAS an effort
|
|
272
|
+
# control runs at high, at every complexity level. Effort is therefore a
|
|
273
|
+
# constant here, not a function of complexity; the map is kept rather than
|
|
274
|
+
# deleted because deleting it would silently hand control back to the roster
|
|
275
|
+
# entries, which is a second place to forget.
|
|
276
|
+
#
|
|
277
|
+
# This SUPERSEDES the 2026-04-29 policy ("reviewers stay at each LLM's default
|
|
278
|
+
# effort; verdict quality is effort-independent"). That policy rested on a
|
|
279
|
+
# cross-evaluation measurement of low vs high scoring 8.35 vs 8.16 — taken on
|
|
280
|
+
# the Opus 4.6 / 4.7 generation, on none of the models now in the roster. It was
|
|
281
|
+
# not re-measured against Opus 5 or gpt-6-astra, so it is superseded on the
|
|
282
|
+
# operator's judgement rather than on a counter-measurement. Cost rises:
|
|
283
|
+
# gpt-6-astra spent 8,274 tokens at high vs 3,150 at its default on the same
|
|
284
|
+
# one-line prompt (measured 2026-09-05).
|
|
285
|
+
#
|
|
286
|
+
# `cursor` deliberately has no key at any level. cursor_adapter builds no effort
|
|
287
|
+
# flag, so a value here would be recorded and never sent.
|
|
245
288
|
effort_map:
|
|
246
289
|
low:
|
|
247
|
-
claude_code:
|
|
248
|
-
codex:
|
|
290
|
+
claude_code: high
|
|
291
|
+
codex: high
|
|
249
292
|
medium:
|
|
250
|
-
claude_code:
|
|
251
|
-
codex:
|
|
293
|
+
claude_code: high
|
|
294
|
+
codex: high
|
|
252
295
|
high:
|
|
253
|
-
claude_code:
|
|
254
|
-
codex:
|
|
296
|
+
claude_code: high
|
|
297
|
+
codex: high
|
|
255
298
|
critical:
|
|
256
|
-
claude_code:
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
299
|
+
claude_code: high # `high` is the ceiling the two providers share: claude
|
|
300
|
+
# accepts low/medium/high/xhigh/max, codex accepts
|
|
301
|
+
# minimal/low/medium/high. Raising this level alone
|
|
302
|
+
# would split the roster across incomparable settings.
|
|
303
|
+
codex: high
|
|
260
304
|
|
|
261
305
|
# Auto-detection thresholds (used when complexity arg not given)
|
|
262
|
-
# Note: per the policy above, all complexity levels currently map to
|
|
263
|
-
# reviewers
|
|
264
|
-
#
|
|
306
|
+
# Note: per the policy above, all complexity levels currently map to high for
|
|
307
|
+
# reviewers, so detected complexity changes nothing about effort today. The
|
|
308
|
+
# thresholds remain so that explicit `complexity` inputs and future per-pool
|
|
309
|
+
# overrides (e.g. revision-phase callers) can still differentiate.
|
|
265
310
|
auto_complexity:
|
|
266
311
|
small_artifact_chars: 500
|
|
267
312
|
large_artifact_chars: 5000
|