kairos-chain 3.83.0 → 3.85.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +125 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/design_to_implementation_workflow/design_to_implementation_workflow.md +25 -6
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +150 -21
- data/templates/knowledge/multi_llm_reviewer_evaluation/multi_llm_reviewer_evaluation.md +59 -12
- data/templates/skillsets/agent/bin/agent_step_worker.rb +109 -20
- data/templates/skillsets/agent/lib/agent/step_delegation.rb +34 -4
- data/templates/skillsets/agent/test/test_agent_worker_exit_and_review_parse.rb +388 -7
- data/templates/skillsets/agent/test/test_agent_worker_process.rb +386 -0
- data/templates/skillsets/agent/tools/agent_step.rb +136 -18
- data/templates/skillsets/minimum_nomic/bin/run_gm.rb +178 -10
- data/templates/skillsets/multi_llm_review/config/multi_llm_review.yml +98 -53
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: bb60c9f0163eaf405fdad217697364791fd3bc3d40fe15a77ef57458950c1163
|
|
4
|
+
data.tar.gz: f5e9dd8b9bfc9ccc36472ada9bdc0d0b00104ef06398c198c6d9963369f82d39
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 28cb26af2e39d33a5f39e4157c5c2e74f7ff63af77384a73aadb6fe4c03e7c7a8e6c445f7deaff9fdc0b05daf47ad69fdb5a6dee3bdeecd6f40a64f146f7196d
|
|
7
|
+
data.tar.gz: ac81db0e9cb2ab019f2944113de79e7b4c19be5389dead92398f86dcc6b4a82f38491bfad13de2a98d68eefe0ee7ddf94d23289d131662fbff2c1ccfe6dab9cc
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,131 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.85.0] - 2026-09-06
|
|
8
|
+
|
|
9
|
+
### Changed — multi-LLM review: high effort by default, gpt-6-astra seats, gpt-5.5 retires
|
|
10
|
+
|
|
11
|
+
Templates only: `multi_llm_review/config/multi_llm_review.yml`, L1
|
|
12
|
+
`multi_llm_review_workflow` 3.12.0 → 3.13.0. No library code.
|
|
13
|
+
|
|
14
|
+
**Effort.** Every seat that has an effort control now runs at `high`, at every
|
|
15
|
+
complexity level, so `effort_map` is a constant rather than a function of complexity.
|
|
16
|
+
Claude CLI takes `--effort high`, codex takes `-c model_reasoning_effort=high`, Cursor
|
|
17
|
+
takes nothing — `cursor_adapter` builds no effort flag, so a value set for a cursor
|
|
18
|
+
entry would be recorded and never sent. Both the roster defaults and `effort_map` are
|
|
19
|
+
set, because `effort_map` overrides the roster per dispatch and an entry left at
|
|
20
|
+
`medium` would be silently raised. `high` rather than `xhigh`/`max` because it is the
|
|
21
|
+
ceiling the two providers share (Claude CLI accepts low/medium/high/xhigh/max, codex
|
|
22
|
+
accepts minimal/low/medium/high), and a roster split across incomparable settings is
|
|
23
|
+
worth less than a lower common one.
|
|
24
|
+
|
|
25
|
+
This supersedes the 2026-04-29 default-effort policy. That policy rested on one
|
|
26
|
+
low-vs-high measurement — 8.35 vs 8.16 — taken on the Opus 4.6 / 4.7 generation, on
|
|
27
|
+
none of the models now in the roster. It was not re-measured, so the supersession is a
|
|
28
|
+
judgement and a later measurement could reverse it. Cost is measured and rises:
|
|
29
|
+
gpt-6-astra spent 8,274 tokens at high against 3,150 at its default on one identical
|
|
30
|
+
one-line prompt.
|
|
31
|
+
|
|
32
|
+
**Roster.** `codex_gpt6-astra` replaces `codex_gpt5.6-sol`, verified before the swap
|
|
33
|
+
through the flags `codex_adapter` actually builds. `codex_gpt5.5` retires in the same
|
|
34
|
+
edit and is not replaced: it had been commented out of the instance roster since
|
|
35
|
+
2026-07-30 "for round R10 only" and never restored, so the calibrated
|
|
36
|
+
cross-generation anchor role had already lapsed for five weeks. Codex now holds one
|
|
37
|
+
slot and it is uncalibrated. Roster 5 → 4, so `convergence_rule` moves 3/5 → 3/4 and
|
|
38
|
+
`convergence_rule_after_exclusion` 3/4 → 2/3, both on the same ceil(N × 0.6) basis and
|
|
39
|
+
both reference figures rather than gates. Three calibration warnings are recorded in
|
|
40
|
+
the L1: the 138-run seat profile is gpt-5.6-sol's and does not transfer; that corpus
|
|
41
|
+
was gathered at medium effort; and it was gathered on a 5-seat roster.
|
|
42
|
+
|
|
43
|
+
### Fixed — three documents called the APPROVE ratio the closing condition
|
|
44
|
+
|
|
45
|
+
A round closes on the exhaustion of (a)+(b) findings, declared by the operator, with
|
|
46
|
+
new (a)+(b) P0 = 0 as the machine-side signal. The tooling has said so since v0.7
|
|
47
|
+
INV-R2 named the field `reference_verdict` and v0.10.1 renamed `convergence` to
|
|
48
|
+
`vote_tally`. Three documents had not caught up, and they are the ones actually read.
|
|
49
|
+
|
|
50
|
+
L1 `multi_llm_reviewer_evaluation` 1.5 → 1.6. § Convergence Rule (Updated) stated
|
|
51
|
+
`3/4 APPROVE = proceed to next step` and `4/4 APPROVE = merge-ready` with no note that
|
|
52
|
+
the ratio is a reference value — while L1 `multi_llm_review_workflow` names this very
|
|
53
|
+
section as the normative detail it defers to. The document saying "the ratio is not
|
|
54
|
+
the primary close" pointed at a document saying it was. The closing condition is now
|
|
55
|
+
stated first, the one blocking rule (any (a)/(b) REJECT) is separated from the
|
|
56
|
+
reference figures, and the two facts that make the ratio unusable as a criterion sit
|
|
57
|
+
in the section: Codex went 24 of 24 reviews without APPROVE on one design loop, and
|
|
58
|
+
both 2026-08 threads closed by (a)+(b) exhaustion without reaching their ratio.
|
|
59
|
+
|
|
60
|
+
L1 `design_to_implementation_workflow` 1.1 → 1.2. `merge gate: 3/3 APPROVE =
|
|
61
|
+
merge-ready` and `Merge criteria: 3/3 APPROVE with 0 FAIL` replaced by the closing
|
|
62
|
+
condition; the anti-pattern list gains reading an APPROVE count as the closing
|
|
63
|
+
condition, naming this document as having carried it. The reviewer lineup written here
|
|
64
|
+
("3 reviewers: Opus 4.6 + Codex GPT-5.4 + Composer-2") is deleted rather than
|
|
65
|
+
corrected — it had been wrong since 2026-04-19, and a roster copied into a second
|
|
66
|
+
document goes stale silently.
|
|
67
|
+
|
|
68
|
+
`agent` SkillSet: the bundle handed to reviewers opened "aggregate per the convergence
|
|
69
|
+
rule below" followed by the bare ratio. The heading no longer calls it a convergence
|
|
70
|
+
rule, a Closing condition section states what actually closes a round, and the ratio
|
|
71
|
+
is labelled "Reference ratio (recorded, never sufficient on its own)".
|
|
72
|
+
|
|
73
|
+
Surviving uses of "merge gate" name the stage and its reviewer lineup, not a ratio,
|
|
74
|
+
and stay.
|
|
75
|
+
|
|
76
|
+
**Not reviewed by multi-LLM review**, at the operator's explicit instruction — the
|
|
77
|
+
change is a seat swap plus wording, and a round would have cost hours. Verification is
|
|
78
|
+
this agent's own and is not an independent judgement: YAML load of both configs, the
|
|
79
|
+
three CLI invocations run for real (gpt-6-astra and both Claude models answered at
|
|
80
|
+
high, with the model echoed back), `ruby -c` on `agent_step.rb`, multi_llm_review 556
|
|
81
|
+
runs / 1,809 assertions and agent 21 checks, all passing. No test asserts the reviewer
|
|
82
|
+
prompt wording, so that one line is unguarded.
|
|
83
|
+
|
|
84
|
+
## [3.84.0] - 2026-09-03
|
|
85
|
+
|
|
86
|
+
### Fixed — agent: second field-defect bundle (D5-b, D6, exit-record housekeeping), real-process worker tests
|
|
87
|
+
|
|
88
|
+
`agent` SkillSet templates only; the at-most-once advance gate (`lib/agent/advance_gate.rb`) is
|
|
89
|
+
byte-identical to 3.83.0 and the normal path writes the same records with the same content.
|
|
90
|
+
|
|
91
|
+
**D5-b — the review-verdict parser lost to nested objects and braces inside strings.** The old
|
|
92
|
+
candidate scan walked back to the nearest `{` before `"overall_verdict"` and counted braces with no
|
|
93
|
+
notion of strings, so `{"summary": {"a": 1}, "overall_verdict": …}` yielded the sibling `{"a": 1}`
|
|
94
|
+
and a `}` inside a string value corrupted the depth count. Candidates now come from one string-aware
|
|
95
|
+
forward pass over the reply; for each key occurrence they are emitted by enclosing depth ascending
|
|
96
|
+
(so a panel's top-level verdict beats a per-persona one), with a size tie-break within a depth; an
|
|
97
|
+
occurrence no span encloses (an odd quote on the same line) gets a line-anchored string-aware scan,
|
|
98
|
+
and the old nearest-brace walk-back only as the candidate of last resort. Fence and crude fallbacks
|
|
99
|
+
are unchanged.
|
|
100
|
+
|
|
101
|
+
**D6 — a TERM/INT/HUP during the gated call left no record.** The shutdown flag was read once before
|
|
102
|
+
the call; a signal landing during a minutes-long call was swallowed and, under the usual TERM-then-KILL
|
|
103
|
+
sequence, the worker vanished as `no_record`. A recorder thread now writes an interim `trapped_signal`
|
|
104
|
+
record (`phase: during_gated_call`) the moment a signal lands while the call is active; the final
|
|
105
|
+
`normal` / `uncaught` / watchdog record supersedes it and carries `signals_during_call`. The call is
|
|
106
|
+
still left to finish (aborting it would turn a sound advance into a crash-path recovery); no result
|
|
107
|
+
file is written on any signal path. The flag is re-checked after tool-registry bootstrap, so a signal
|
|
108
|
+
in that window exits 130 with a `before_gated_call` record; the recorder is stopped before every
|
|
109
|
+
final write so it cannot overwrite a final record; and the exit record (and its atomic-write temp
|
|
110
|
+
file) no longer counts as session activity — before this fix a TERM to a hung worker pushed the
|
|
111
|
+
stall bound out by a full stall window (measured +4.0 s → +7.1 s with stall=3).
|
|
112
|
+
|
|
113
|
+
**Housekeeping.** A fresh delegation handle removes the previous worker's `worker_exit.json`
|
|
114
|
+
(readers already filtered by token; the file lingered). New env knob
|
|
115
|
+
`KAIROS_WORKER_WATCHDOG_TICK_SECONDS` (default 30, floor 1) sets how often the watchdog re-evaluates
|
|
116
|
+
its two bounds — the default is unchanged; the knob exists so real-process tests can drive both
|
|
117
|
+
bounds in seconds.
|
|
118
|
+
|
|
119
|
+
**Not changed, by evidence.** D3 (read-side queries blocked during a delegated run) is MCP stdio
|
|
120
|
+
serialisation in the server's request loop, not the SkillSet; mitigation today is a short
|
|
121
|
+
`max_wait_seconds` on `agent_wait`. D7 (a GVL-holding call starving the watchdog) did not reproduce:
|
|
122
|
+
every blocking primitive on the worker's path releases the GVL (probe table in the implementation log).
|
|
123
|
+
|
|
124
|
+
**Tests.** New `test_agent_worker_process.rb` starts the shipped worker script as a real child
|
|
125
|
+
process and drives 7 of its 8 exit classes (normal, uncaught, superseded, signal before the call,
|
|
126
|
+
bootstrap failure, stall bound, hard cap) plus the D6 interim record, a TERM to a hung worker, a TERM
|
|
127
|
+
inside registry bootstrap, and signals carried on a watchdog record. Agent suite 96 runs / 209
|
|
128
|
+
assertions → 131 / 345; every fix has a test shown red on the pre-fix code (one exception recorded:
|
|
129
|
+
the recorder-serialisation window is sub-millisecond and has no deterministic test). Reviewed in two
|
|
130
|
+
implementation rounds (7 seats each); records in the instance's L2 store.
|
|
131
|
+
|
|
7
132
|
## [3.83.0] - 2026-09-01
|
|
8
133
|
|
|
9
134
|
### Added — readable_gate measures sentence length
|
data/lib/kairos_mcp/version.rb
CHANGED
data/templates/knowledge/design_to_implementation_workflow/design_to_implementation_workflow.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: design_to_implementation_workflow
|
|
3
3
|
description: "Full-lifecycle workflow for complex features: design review, self-review, implementation review, and final merge gate. Derived from Service Grant + Attestation Nudge experiments."
|
|
4
|
-
version: "1.
|
|
4
|
+
version: "1.2"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- implementation
|
|
@@ -31,7 +31,8 @@ different bugs.
|
|
|
31
31
|
│ v0.2 ──→ Multi-LLM Review R2 ──→ Fix ──→ v0.3 │
|
|
32
32
|
│ (fix correctness) │
|
|
33
33
|
│ │
|
|
34
|
-
│
|
|
34
|
+
│ Closes on: new (a)+(b) P0 = 0, operator declares freeze │
|
|
35
|
+
│ (APPROVE ratio is a reference figure, not the gate) │
|
|
35
36
|
├─────────────────────────────────────────────────────────────┤
|
|
36
37
|
│ IMPLEMENTATION PHASE │
|
|
37
38
|
│ │
|
|
@@ -48,7 +49,7 @@ different bugs.
|
|
|
48
49
|
│ (missing wiring, fail-open, integration gaps) │
|
|
49
50
|
│ │
|
|
50
51
|
│ Final Multi-LLM Review + Persona Assembly │
|
|
51
|
-
│ (
|
|
52
|
+
│ (closes the same way; ratio stays a reference figure) │
|
|
52
53
|
└─────────────────────────────────────────────────────────────┘
|
|
53
54
|
```
|
|
54
55
|
|
|
@@ -76,8 +77,15 @@ different bugs.
|
|
|
76
77
|
- Output: Complete design document with pseudocode
|
|
77
78
|
|
|
78
79
|
#### Multi-LLM Review Rounds
|
|
79
|
-
- **
|
|
80
|
-
|
|
80
|
+
- **Reviewers**: the roster in `multi_llm_review/config/multi_llm_review.yml`,
|
|
81
|
+
which is where the count and the models live. The "3 reviewers: Opus 4.6 +
|
|
82
|
+
Codex GPT-5.4 + Composer-2" written here until 2026-09-06 had been wrong since
|
|
83
|
+
2026-04-19 and is not restated, because a roster copied into a second document
|
|
84
|
+
goes stale silently
|
|
85
|
+
- **Closing condition**: new (a)+(b) P0 = 0, carryover P0s counted separately
|
|
86
|
+
with a closure verdict on each, and the freeze declared by the operator. The
|
|
87
|
+
APPROVE ratio the tool reports is a recorded reference value — see L1
|
|
88
|
+
`multi_llm_review_workflow` § Convergence Rules
|
|
81
89
|
- **Typical rounds**: 2-3 for Tier 3 complexity
|
|
82
90
|
- **Convergence curve**:
|
|
83
91
|
- R1: Structural gaps — "this is missing" (existence)
|
|
@@ -148,7 +156,11 @@ Claude Persona Assembly (4 personas):
|
|
|
148
156
|
- Safety-critical components
|
|
149
157
|
- NOT for intermediate rounds (diminishing returns)
|
|
150
158
|
|
|
151
|
-
**Merge criteria**:
|
|
159
|
+
**Merge criteria**: new (a)+(b) P0 = 0, with the freeze declared by the operator.
|
|
160
|
+
A Codex APPROVE is the strongest *reference* signal (see
|
|
161
|
+
`multi_llm_reviewer_evaluation`), and it is not a gate: Codex went 24 of 24
|
|
162
|
+
reviews without reaching APPROVE on one design loop, so a merge waiting on it
|
|
163
|
+
would never have merged.
|
|
152
164
|
|
|
153
165
|
## Effort Level Selection
|
|
154
166
|
|
|
@@ -192,6 +204,9 @@ Reviewer 側 (Opus 4.7) は effort-independent なので high が天井。xhigh/
|
|
|
192
204
|
|
|
193
205
|
## Convergence Data
|
|
194
206
|
|
|
207
|
+
Observed outcomes of past loops, not criteria. The APPROVE counts below record
|
|
208
|
+
what those particular rounds happened to reach; they are not what closed them.
|
|
209
|
+
|
|
195
210
|
### Service Grant (Tier 3, 2026-03-18)
|
|
196
211
|
- Design: v1.0 → v1.4, 3 review rounds, 3 LLMs
|
|
197
212
|
- Design review findings: R1: 8 P0/P1, R2: 2 FAIL + 28 CONCERN, R3: 0 FAIL
|
|
@@ -207,6 +222,10 @@ Reviewer 側 (Opus 4.7) は effort-independent なので high が天井。xhigh/
|
|
|
207
222
|
|
|
208
223
|
## Anti-Patterns
|
|
209
224
|
|
|
225
|
+
- **Reading an APPROVE count as the closing condition.** It is a recorded
|
|
226
|
+
reference value; a round closes on new (a)+(b) P0 = 0 with the freeze declared
|
|
227
|
+
by the operator. This document itself carried the mistake until v1.2, in the
|
|
228
|
+
strongest available wording ("merge gate: 3/3 APPROVE = merge-ready")
|
|
210
229
|
- Implementing Phase 2+ when Phase 1 prerequisites aren't met
|
|
211
230
|
- Using agent team for implementation (context fragmentation)
|
|
212
231
|
- Skipping self-review (misses cheap P0 fixes)
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.
|
|
4
|
+
version: "3.13.0"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -90,6 +90,72 @@ Codex (c)-class value-divergent REJECTs as blocking, which causes review loops t
|
|
|
90
90
|
fail to converge. The cross-reference exists in `related:` frontmatter; this step
|
|
91
91
|
makes it an explicit pre-condition rather than an implicit hint.
|
|
92
92
|
|
|
93
|
+
## Step 0.1 — Measured seat characteristics (corpus 2026-08-03 .. 2026-09-03)
|
|
94
|
+
|
|
95
|
+
Qualitative per-model profiles stay in `multi_llm_reviewer_evaluation`. What
|
|
96
|
+
follows is the counted complement: seat behaviour as it actually occurred, so
|
|
97
|
+
seat selection is a lookup rather than a recollection. A **seat** is one
|
|
98
|
+
reviewer slot in one run.
|
|
99
|
+
|
|
100
|
+
Corpus: 138 completed runs in the local run store, 556 seat observations, 5286
|
|
101
|
+
findings (P0 1473 / P1 560 / P2 2994 / P3 259). Run types: design 54,
|
|
102
|
+
implementation 60, document 24. Extraction script `log/mlr_extract/extract.py`
|
|
103
|
+
lives in the development repository and does not ship; re-run it to refresh
|
|
104
|
+
these numbers.
|
|
105
|
+
|
|
106
|
+
| Seat | Runs | APPROVE rate design / impl / doc | Median wall s | Median output chars | (c) share, labelled only |
|
|
107
|
+
|---|---|---|---|---|---|
|
|
108
|
+
| `claude_cli_opus4.6` | 134 | 64% (31/48) / 71% (27/38) / 86% (20/23) | 54 | 4,536 | 70% (19/27) |
|
|
109
|
+
| `cursor_composer2.5` | 138 | 13% (7/51) / 48% (24/50) / 25% (6/24) | 133 | 3,795 | 100% (2/2) |
|
|
110
|
+
| `codex_gpt5.6-sol` (retired 2026-09-05) | 138 | 1% (1/54) / 11% (7/60) / 4% (1/24) | 113 | 2,038 | 3% (5/140) |
|
|
111
|
+
| `claude_team_opus-5` (persona) | 131 | 0% (0/51) / 12% (7/56) / 0% (0/22) | not measured per seat | 11,512 | 27% (374/1362) |
|
|
112
|
+
|
|
113
|
+
The codex row is a profile of **gpt-5.6-sol**, which left the roster on
|
|
114
|
+
2026-09-05 when gpt-6-astra replaced it. Nothing in that row transfers to the
|
|
115
|
+
new occupant: the seat's identity here is the model, not the slot. Until a
|
|
116
|
+
comparable corpus accumulates for `codex_gpt6-astra`, read the codex row as
|
|
117
|
+
history and treat the new seat as uncalibrated. The same caution applies to
|
|
118
|
+
every row from 2026-09-05 onward for a second reason — the corpus was gathered
|
|
119
|
+
with reviewers at medium effort, and reviewers now run at high (see § Thinking
|
|
120
|
+
Effort Configuration), so post-swap rounds are not directly comparable to it.
|
|
121
|
+
|
|
122
|
+
Selection consequences, each tied to the number above it:
|
|
123
|
+
|
|
124
|
+
- **A `codex` APPROVE carries the most information and an `opus4.6` APPROVE the
|
|
125
|
+
least.** Codex approves 1 design review in 54 and pairs that with the lowest
|
|
126
|
+
advisory rate in the corpus (5 of 140 labelled findings). Opus4.6 approves 20
|
|
127
|
+
document reviews in 23 while 19 of its 27 labelled findings are advisory. Seat
|
|
128
|
+
both, read them differently: the `3/4 APPROVE` threshold is met, in practice,
|
|
129
|
+
with opus4.6's vote already inside it, so the live question is whether two of
|
|
130
|
+
the remaining three agree. (That threshold is a reference figure, not a gate —
|
|
131
|
+
see § Convergence Rules.)
|
|
132
|
+
- **Volume anti-correlates with signal.** The persona seat raises 3,249 of the
|
|
133
|
+
5,286 findings (61%) and 27% of its labelled ones are advisory. Seat personas
|
|
134
|
+
when breadth is wanted; do not seat them to obtain a verdict.
|
|
135
|
+
- **Design draws roughly seven times the P0 of implementation** — median 14 P0
|
|
136
|
+
per design-medium run against 2 per implementation-high run. Budget rounds
|
|
137
|
+
accordingly; a design round returning two P0 is anomalous, not clean.
|
|
138
|
+
- **Dropping a seat does not shorten the round.** Seats run concurrently, so
|
|
139
|
+
wall-clock is the slowest seat (median 179 s per run against 300–400 s summed
|
|
140
|
+
across seats). Removing the 54 s seat saves nothing.
|
|
141
|
+
- **Loops do not converge on their own.** Of 29 artifacts reviewed more than
|
|
142
|
+
once, the last round was REVISE in 26 and APPROVE in 3; rounds per loop median
|
|
143
|
+
3, maximum 12. P0 counts are not monotone: `chain_history_erasure_fix` ran
|
|
144
|
+
25, 25, 21, 14, **38**, 29, 26, 13 across rounds 1–8, and
|
|
145
|
+
`ruby_association_grant_2026_application_ja` returned APPROVE at round 8 with
|
|
146
|
+
3 P0 and then 23 P0 at round 9. Treat a single APPROVE as an observation, not
|
|
147
|
+
as a close — § Convergence Rules already requires the operator's declaration.
|
|
148
|
+
|
|
149
|
+
Two limits of this corpus, both open:
|
|
150
|
+
|
|
151
|
+
- **Findings are not deduplicated across seats.** `cited_by` had exactly one
|
|
152
|
+
entry in 5,286 of 5,286 findings, so agreement between seats is not derivable
|
|
153
|
+
from the store, and the (c) shares above are per-seat rates rather than
|
|
154
|
+
contested classifications.
|
|
155
|
+
- **(a)/(b)/(c) labels are present on 1,622 of 5,286 findings (31%)**, and
|
|
156
|
+
`cursor` labelled 2 of its 730. Classification is applied unevenly by the
|
|
157
|
+
seats, so the shares are computed over labelled findings only.
|
|
158
|
+
|
|
93
159
|
## Step 0.25 — Unknowns Pass (pre-draft, qualifying reviews only)
|
|
94
160
|
|
|
95
161
|
> **Numbering vs timing**: Step 0 and Step 0.5 execute at review time,
|
|
@@ -399,12 +465,17 @@ they disagree, the config is right and this section is stale.
|
|
|
399
465
|
which under the default "delegate" strategy is taken by your persona team
|
|
400
466
|
rather than spawned — so when you are Opus 5, Opus 4.6 is the only Claude
|
|
401
467
|
CLI subprocess
|
|
402
|
-
- [ ] Codex
|
|
468
|
+
- [ ] Codex model: gpt-6-astra, with -m. One codex slot since gpt-5.5 was
|
|
469
|
+
retired 2026-09-05 — do not add a second codex entry expecting the old
|
|
470
|
+
cross-generation pairing
|
|
403
471
|
- [ ] Cursor model: composer-2.5, passed explicitly as --model composer-2.5
|
|
404
|
-
- [ ]
|
|
472
|
+
- [ ] Effort: high on every seat that has an effort control — `--effort high`
|
|
473
|
+
for Claude CLI, `-c model_reasoning_effort=high` for codex. Cursor has no
|
|
474
|
+
effort control and takes no flag
|
|
475
|
+
- [ ] Total reviewer count: 4 (or 3 after orchestrator exclusion from subprocess)
|
|
405
476
|
- [ ] Closing condition: new (a)+(b) P0 = 0, with carryover P0s counted
|
|
406
477
|
separately and a closure verdict on each. The APPROVE ratio the tool
|
|
407
|
-
reports (3/
|
|
478
|
+
reports (3/4 full roster, 2/3 after exclusion) is a reference value,
|
|
408
479
|
not the condition — see § Convergence Rules
|
|
409
480
|
```
|
|
410
481
|
|
|
@@ -423,7 +494,7 @@ Path B refuses such a slot outright; on Path A nothing refuses it but you.
|
|
|
423
494
|
|---------|-----------------|----------------|
|
|
424
495
|
| Launch a reviewer without an explicit model flag | Always pass `--model` / `-m`. A slot with no flag takes the CLI's user-editable default | "The default is the one we want" — it was, until someone changed it outside this repo |
|
|
425
496
|
| Exclude orchestrator model from Agent Team Personas | Agent Team uses orchestrator model — they provide persona diversity, not epistemic diversity | LLM misreads "do not assign yourself as a reviewer" as applying to Agent Team; it applies only to subprocess CLI |
|
|
426
|
-
| Run only Codex GPT-
|
|
497
|
+
| Run only Codex GPT-6-astra, skip 5.5 | Run both — cross-generation entries catch different things (5.5 found §5 schema contradiction in Phase 2 Case A that no other reviewer caught) | Cost-saving heuristic; roster has both for a reason |
|
|
427
498
|
| Use a smaller/cheaper model as Agent Team substitute | Use the orchestrator's own model with different personas | Confusing "model diversity" with "persona diversity" — Agent Team is the latter |
|
|
428
499
|
| Run 3 reviewers instead of the configured roster | Use the full roster from config | Ad-hoc "3 is enough" reasoning; the roster size is empirical |
|
|
429
500
|
| Count a reply that carries only a verdict | Drop it from the denominator, and say why | A bare "APPROVE" looks like agreement and raises the bar for everyone else without contributing (see § Substance and the denominator) |
|
|
@@ -592,7 +663,7 @@ The rule applies **after** orchestrator classifies each finding as (a)/(b)/(c) p
|
|
|
592
663
|
findings count toward the thresholds below; (c) findings are recorded as advisory
|
|
593
664
|
and never block.
|
|
594
665
|
|
|
595
|
-
- **3/
|
|
666
|
+
- **3/4 APPROVE** full roster, or **2/3 APPROVE** after orchestrator exclusion ("exclude" strategy only — the default "delegate" strategy keeps 4 voters via collect) (no (a)/(b) REJECT) = proceed to next step
|
|
596
667
|
- **Any (a) or (b) REJECT or FAIL** = revise and re-review
|
|
597
668
|
- **(c)-only REJECT** = record as advisory, non-blocking
|
|
598
669
|
- **Unanimous APPROVE** (no (a)/(b)) = highest confidence, proceed
|
|
@@ -643,7 +714,7 @@ For normative detail and the underlying classification, see
|
|
|
643
714
|
| Agreement | Meaning | Action |
|
|
644
715
|
|-----------|---------|--------|
|
|
645
716
|
| **N/N** (unanimous) | Architectural-level gap | Must fix |
|
|
646
|
-
| **Majority** (e.g. 3/
|
|
717
|
+
| **Majority** (e.g. 3/4, 2/3) | Implementation-level issue | Should fix |
|
|
647
718
|
| **1/N only** | Specialty-specific insight | Do NOT ignore — often the most novel finding |
|
|
648
719
|
|
|
649
720
|
1/N findings are not "minority opinions to discard." They represent unique expertise.
|
|
@@ -743,7 +814,7 @@ outside this repository — see the incident recorded in § Pre-flight checklist
|
|
|
743
814
|
|
|
744
815
|
| Tool | Command | Prompt Input | Output Collection | Model |
|
|
745
816
|
|------|---------|-------------|-------------------|-------|
|
|
746
|
-
| **Codex** | `codex exec -m <model
|
|
817
|
+
| **Codex** | `codex exec -m <model> -c model_reasoning_effort=high` | stdin pipe: `cat prompt.md \| codex exec -m <model> -` | `-o /path/output.md` | gpt-6-astra — one slot since gpt-5.5 was retired 2026-09-05 |
|
|
747
818
|
| **Cursor Agent** | `agent -p --model composer-2.5` | File reference (stdin NOT supported) | stdout redirect: `> output.md` | composer-2.5, passed explicitly — never relying on the CLI default |
|
|
748
819
|
| **Claude Code** | Agent tool (internal) | Direct prompt string | Write to workspace file | Orchestrator model, or the declared `persona_model` when personas run elsewhere |
|
|
749
820
|
| **Claude CLI (4.6)** | `claude -p --model claude-opus-4-6` | stdin pipe: `cat prompt.md \| claude -p --model claude-opus-4-6` | stdout redirect: `> output.md` | Opus 4.6 — the calibrated anchor, deliberately not a frontier model |
|
|
@@ -766,11 +837,27 @@ Based on cross-evaluation experiment (7 models × 4 tasks + Nomic, 518 CLI calls
|
|
|
766
837
|
|------|-------|-------------|-----------|
|
|
767
838
|
| **Primary (orchestrator)** | session default | (default) | Sufficient for integration, dialogue, judgment |
|
|
768
839
|
| **Reviewer: Agent Team** | = orchestrator, or the declared `persona_model` | (default) | Personas inherit whichever model actually runs them |
|
|
769
|
-
| **Reviewer: Claude CLI** | Opus 4.6, plus any frontier roster slot the orchestrator is not | (
|
|
840
|
+
| **Reviewer: Claude CLI** | Opus 4.6, plus any frontier roster slot the orchestrator is not | `--effort high` (config `effort: high`) | Operator instruction 2026-09-05; supersedes the 2026-04-29 default-effort policy — see the note below the table |
|
|
770
841
|
| **Coding sub-agent** | Opus 5 | `--effort xhigh` | Published starting point for coding/agentic work; not measured here (see note) |
|
|
771
842
|
| **Design sub-agent** | Opus 5 | `--effort high` | Published starting point for intelligence-sensitive work; not measured here (see note) |
|
|
772
|
-
| **Codex** | GPT-
|
|
773
|
-
| **Cursor Agent** | Composer-2.5 | (no flag) |
|
|
843
|
+
| **Codex** | GPT-6-astra / GPT-5.5 | `-c model_reasoning_effort=high` | Same operator instruction. The earlier "(no flag) / fixed effort" entry was wrong: codex_adapter has always emitted this flag when the roster set `effort` |
|
|
844
|
+
| **Cursor Agent** | Composer-2.5 | (no flag) | Genuinely has no effort control — cursor_adapter builds no such flag, so an `effort:` key on a cursor roster entry is recorded and never sent |
|
|
845
|
+
|
|
846
|
+
Effort policy (2026-09-05, operator instruction). Every reviewer that HAS an
|
|
847
|
+
effort control runs at **high**, at every complexity level, and the `effort_map`
|
|
848
|
+
in `config/multi_llm_review.yml` is a constant rather than a function of
|
|
849
|
+
complexity. `high` is deliberate rather than maximal: it is the ceiling the two
|
|
850
|
+
providers share (Claude CLI accepts low/medium/high/xhigh/max, codex accepts
|
|
851
|
+
minimal/low/medium/high), so it is the highest setting at which the roster stays
|
|
852
|
+
comparable across providers.
|
|
853
|
+
|
|
854
|
+
This supersedes the 2026-04-29 policy that kept reviewers at each CLI's default.
|
|
855
|
+
That policy rested on one measurement — low vs high scoring 8.35 vs 8.16 in
|
|
856
|
+
cross-evaluation — taken on the Opus 4.6 / 4.7 generation, on none of the models
|
|
857
|
+
in the current roster. It was not re-measured, so it is superseded by judgement,
|
|
858
|
+
not by a counter-measurement, and a later measurement could reinstate it. The
|
|
859
|
+
cost side is measured: on one identical one-line prompt, gpt-6-astra spent 8,274
|
|
860
|
+
tokens at high against 3,150 at its default (2026-09-05).
|
|
774
861
|
|
|
775
862
|
Note (2026-07-25): the effort experiment data is from the Opus 4.6/4.7
|
|
776
863
|
generation. Opus 5 and Fable 5 effort sensitivity is not yet calibrated;
|
|
@@ -867,13 +954,13 @@ multi_llm_review(
|
|
|
867
954
|
**Dispatcher behavior** (config: `exclude_orchestrator_model: true`, default `true`):
|
|
868
955
|
- If `orchestrator_model` matches a roster entry's `model`, that entry is skipped.
|
|
869
956
|
- `min_quorum` and `convergence_rule` apply to the remaining reviewers.
|
|
870
|
-
-
|
|
957
|
+
- 4-reviewer roster → 3 reviewers; `convergence_rule_after_exclusion: "2/3 APPROVE"`
|
|
871
958
|
(from config) replaces the full-roster rule. This reduced count applies to the
|
|
872
959
|
"exclude" strategy only. The "subprocess" strategy keeps the full roster (the
|
|
873
960
|
matching entry runs as a fresh CLI process instead of being skipped). Under the
|
|
874
961
|
default "delegate" strategy, the matching entry is dropped at dispatch but
|
|
875
|
-
re-added at collect as the persona-team entry, so the voter count returns to
|
|
876
|
-
and the full-roster rule (3/
|
|
962
|
+
re-added at collect as the persona-team entry, so the voter count returns to 4
|
|
963
|
+
and the full-roster rule (3/4 APPROVE) applies.
|
|
877
964
|
- **At most one roster entry leaves for matching the caller.** This is only
|
|
878
965
|
visible on a roster carrying three or more entries on the orchestrator's own
|
|
879
966
|
model: the first is taken over by the persona team, the second leaves as the
|
|
@@ -1342,16 +1429,16 @@ Step 2: Detect environment, and check the roster against config
|
|
|
1342
1429
|
- Read the roster from config/multi_llm_review.yml — do NOT read CLI defaults
|
|
1343
1430
|
and treat them as the roster. Detection only tells you whether a default has
|
|
1344
1431
|
drifted; the model each slot runs is named on the command line.
|
|
1345
|
-
- Report: "Auto mode: Codex (gpt-
|
|
1432
|
+
- Report: "Auto mode: Codex (gpt-6-astra), Cursor (composer-2.5),
|
|
1346
1433
|
Claude Team (orchestrator model), Claude CLI (opus-4.6)"
|
|
1347
1434
|
|
|
1348
|
-
Step 3: Execute the configured roster in parallel (currently
|
|
1435
|
+
Step 3: Execute the configured roster in parallel (currently 4 slots, one of
|
|
1349
1436
|
which is your own persona team)
|
|
1350
|
-
- Bash(background): cat prompt.md | codex exec -m gpt-
|
|
1351
|
-
- Bash(background): cat prompt.md | codex exec -m gpt-5.6-sol -C workspace -o log/review_codex_gpt5.6-sol.md -
|
|
1437
|
+
- Bash(background): cat prompt.md | codex exec -m gpt-6-astra -c model_reasoning_effort=high -C workspace -o log/review_codex_gpt6-astra.md -
|
|
1352
1438
|
- Bash(background): agent -p --trust --model composer-2.5 "Read prompt and review..." > log/review_cursor.md
|
|
1439
|
+
(no effort flag — Cursor has no effort control)
|
|
1353
1440
|
- Agent(background): Claude Team (orchestrator model, e.g. Opus 5) → write to log/review_claude_team_opus5.md
|
|
1354
|
-
- Bash(background): cat prompt.md | claude -p --model claude-opus-4-6 > log/review_claude_opus4.6.md 2>log/review_claude_opus4.6.stderr.log
|
|
1441
|
+
- Bash(background): cat prompt.md | claude -p --model claude-opus-4-6 --effort high > log/review_claude_opus4.6.md 2>log/review_claude_opus4.6.stderr.log
|
|
1355
1442
|
(add a line per further Claude roster slot you are not; with the 2026-07-26
|
|
1356
1443
|
roster an Opus 5 orchestrator has none, so opus-4.6 is the only one)
|
|
1357
1444
|
|
|
@@ -1391,7 +1478,7 @@ log/{artifact}_review{N}_consensus_{date}.md # Consensus analysis
|
|
|
1391
1478
|
```
|
|
1392
1479
|
|
|
1393
1480
|
LLM identifiers: `claude_cli_opus5`, `claude_cli_opus4.6`,
|
|
1394
|
-
`
|
|
1481
|
+
`codex_gpt6-astra`, `cursor_composer2.5`, `cursor_gpt5.4`,
|
|
1395
1482
|
`cursor_premium`. The delegated slot is reported as `claude_team_<model>`
|
|
1396
1483
|
(e.g. `claude_team_claude-opus-5`), assembled at collect time — the roster's
|
|
1397
1484
|
own labels stay CLI-neutral because either frontier entry can take either path.
|
|
@@ -1399,7 +1486,11 @@ own labels stay CLI-neutral because either frontier entry can take either path.
|
|
|
1399
1486
|
`claude_cli_opus4.7`, `cursor_composer2`; retired 2026-07-23: `codex_gpt5.4`;
|
|
1400
1487
|
retired 2026-07-25: `claude_cli_opus4.8`, `claude_team_fable5`;
|
|
1401
1488
|
retired 2026-07-26: `claude_cli_fable5` — five consecutive non-substantive
|
|
1402
|
-
returns, 85-128 characters in 5-7 seconds, no findings and no verdict text
|
|
1489
|
+
returns, 85-128 characters in 5-7 seconds, no findings and no verdict text;
|
|
1490
|
+
retired 2026-09-05: `codex_gpt5.6-sol`, replaced by `codex_gpt6-astra`, and
|
|
1491
|
+
`codex_gpt5.5`, not replaced. Runs recorded under a retired identifier keep it —
|
|
1492
|
+
the label names the model that answered, so renaming old records would attribute
|
|
1493
|
+
one model's findings to another)
|
|
1403
1494
|
|
|
1404
1495
|
## Internal Agent Team Review
|
|
1405
1496
|
|
|
@@ -1695,6 +1786,44 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1695
1786
|
four rounds was the change's account of itself, not the rules it proposed.
|
|
1696
1787
|
Records: L2 `handoff_mlr_l1_norms_revision_three_rounds_and_switch_to_implementation_20260821`
|
|
1697
1788
|
and L2 `mlr_v3_11_0_applied_review_r1_and_subtractive_revision_20260822`
|
|
1789
|
+
- Effort raised to high, and gpt-6-astra replaces gpt-5.6-sol (v3.13.0,
|
|
1790
|
+
2026-09-05, operator instruction). Two changes with one config edit. **Effort**:
|
|
1791
|
+
every seat that has an effort control now runs at high at every complexity
|
|
1792
|
+
level, so `effort_map` is a constant rather than a function of complexity —
|
|
1793
|
+
Claude CLI gets `--effort high`, codex gets `-c model_reasoning_effort=high`,
|
|
1794
|
+
Cursor gets nothing because cursor_adapter builds no effort flag and a value
|
|
1795
|
+
set for it would be recorded and never sent. `high` rather than `xhigh`/`max`
|
|
1796
|
+
because it is the ceiling the two providers share, and a roster split across
|
|
1797
|
+
incomparable settings is worth less than a lower common one. This supersedes
|
|
1798
|
+
the 2026-04-29 default-effort policy, which rested on a single low-vs-high
|
|
1799
|
+
measurement (8.35 vs 8.16) taken on the Opus 4.6 / 4.7 generation — none of
|
|
1800
|
+
the models now in the roster. No counter-measurement was taken, so the
|
|
1801
|
+
supersession is a judgement and a later measurement could reverse it. Cost is
|
|
1802
|
+
measured and rises: gpt-6-astra spent 8,274 tokens at high against 3,150 at
|
|
1803
|
+
its default on one identical one-line prompt. **Roster**: `codex_gpt6-astra`
|
|
1804
|
+
replaces `codex_gpt5.6-sol`, verified before the swap through the flags
|
|
1805
|
+
codex_adapter actually builds. **gpt-5.5 retired in the same edit**, not
|
|
1806
|
+
replaced: it had been the calibrated cross-generation anchor, but it had been
|
|
1807
|
+
commented out of the instance roster since 2026-07-30 "for round R10 only" and
|
|
1808
|
+
never restored, so the anchor role had already lapsed for five weeks and the
|
|
1809
|
+
retirement only names the state. Codex therefore holds ONE slot, and it is
|
|
1810
|
+
uncalibrated — there is no longer a calibrated codex seat to read a new one
|
|
1811
|
+
against. Roster 5 → 4, so `convergence_rule` moves 3/5 → 3/4 and
|
|
1812
|
+
`convergence_rule_after_exclusion` 3/4 → 2/3, both on the same ceil(N × 0.6)
|
|
1813
|
+
basis. Three calibration warnings follow and none is optional: the 138-run
|
|
1814
|
+
seat profile in § Reviewer selection is gpt-5.6-sol's and does not transfer to
|
|
1815
|
+
the new occupant; the whole of that corpus was gathered at medium effort, so
|
|
1816
|
+
rounds from 2026-09-05 onward are not directly comparable to any of it; and
|
|
1817
|
+
the corpus was gathered on a 5-seat roster, so per-round finding counts shift
|
|
1818
|
+
for a third, independent reason. Record (a)/(b)/(c) breakdowns per round in
|
|
1819
|
+
`multi_llm_reviewer_evaluation` until a profile for the new seat accumulates.
|
|
1820
|
+
Recorded because it was got wrong in the session that made the change: the
|
|
1821
|
+
new ratio was first reported to the operator as "3 of 4 seats must APPROVE",
|
|
1822
|
+
as though it were a gate. It is not. § Convergence Rules has said since v3.5
|
|
1823
|
+
that the ratio is neither the only nor the primary close, `Consensus.compute`
|
|
1824
|
+
returns it under the name `reference_verdict`, and both 2026-08 threads closed
|
|
1825
|
+
by (a)+(b) exhaustion without ever reaching it. Every ratio in this document
|
|
1826
|
+
is a reference figure.
|
|
1698
1827
|
|
|
1699
1828
|
**Key insight**: Design reviews and implementation reviews find
|
|
1700
1829
|
**categorically different bugs**. Both phases are necessary.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_reviewer_evaluation
|
|
3
3
|
description: "Multi-LLM reviewer performance evaluation — strengths, weaknesses, value-system biases, and recommended workflows. Based on 185+ reviews (Phase 1, 2026-02 to 03) + Phase 2 Case A 4-round Codex bias study (2026-05-04)."
|
|
4
|
-
version: "1.
|
|
4
|
+
version: "1.6"
|
|
5
5
|
tags:
|
|
6
6
|
- multi-llm
|
|
7
7
|
- review
|
|
@@ -172,8 +172,9 @@ When a reviewer issues a P0, classify the *cause* — not just the severity:
|
|
|
172
172
|
| (b) philosophy-aligned | Deviation from declared design principles (e.g., enumeration where invariant suffices). | **Blocking P0** |
|
|
173
173
|
| (c) value-divergent | Reviewer's own style preference or generic best practice not entailed by project principles. | **Advisory only** (non-blocking) |
|
|
174
174
|
|
|
175
|
-
When uncertain between (b) and (c), default to (c).
|
|
176
|
-
(c) findings are recorded but do not block.
|
|
175
|
+
When uncertain between (b) and (c), default to (c). Only (a)+(b) findings count toward
|
|
176
|
+
closing a round; (c) findings are recorded but do not block. Closing is the exhaustion
|
|
177
|
+
of (a)+(b), not an APPROVE count — see § Convergence Rule (Updated).
|
|
177
178
|
|
|
178
179
|
**Codex ↔ classes**: Codex finds genuine (a) bugs (e.g., the §5 schema contradiction).
|
|
179
180
|
Codex also produces many (c) findings driven by the 3 biases above. The skill of using
|
|
@@ -201,6 +202,9 @@ Final Review: Codex APPROVE | Composer-2.5 APPROVE+ | Claude APPROVE+
|
|
|
201
202
|
- Codex REJECT reasons are always **substantive** (not stylistic): storage model contradictions, missing call sites, fail-open security
|
|
202
203
|
- When Codex finally APPROVEs, all prior FAIL/HIGH issues have been genuinely resolved
|
|
203
204
|
- **Codex APPROVE = strongest merge-readiness signal** in the 3-LLM configuration
|
|
205
|
+
— as a *reference* signal. It is not a gate, and waiting for it is not a
|
|
206
|
+
closing procedure; see the caveat immediately below and § Convergence Rule
|
|
207
|
+
(Updated)
|
|
204
208
|
|
|
205
209
|
> **Note**: The above convergence data is from the 3-reviewer configuration in
|
|
206
210
|
> the Attestation Nudge session. With the 4-reviewer default (Opus 4.7 added
|
|
@@ -216,18 +220,41 @@ Final Review: Codex APPROVE | Composer-2.5 APPROVE+ | Claude APPROVE+
|
|
|
216
220
|
|
|
217
221
|
### Convergence Rule (Updated)
|
|
218
222
|
|
|
219
|
-
The
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
223
|
+
**The APPROVE ratio is not the convergence criterion.** A round closes on the
|
|
224
|
+
exhaustion of (a)+(b) findings, declared by the operator. The machine-side signal
|
|
225
|
+
is **new (a)+(b) P0 = 0**, counting carryover P0s separately; nothing in the
|
|
226
|
+
multi_llm_review SkillSet computes it and no returned field carries it, so it is
|
|
227
|
+
read off the findings. `Consensus.compute` returns the ratio under the name
|
|
228
|
+
`reference_verdict` and the vote counts under `vote_tally` — both recorded
|
|
229
|
+
observations, neither a conclusion.
|
|
230
|
+
|
|
231
|
+
The evidence that the ratio cannot serve as the criterion is in this document:
|
|
232
|
+
two Codex entries went 24 of 24 reviews without reaching APPROVE on one design
|
|
233
|
+
loop, and both 2026-08 review threads closed by (a)+(b) exhaustion plus an
|
|
234
|
+
operator freeze declaration without ever reaching their ratio. A threshold a seat
|
|
235
|
+
is structurally unlikely to meet cannot be what closes a round.
|
|
236
|
+
|
|
237
|
+
Everything below applies **after** the orchestrator classifies findings as
|
|
238
|
+
(a)/(b)/(c) per § Reviewer Value-System Divergence. A REJECT whose findings are
|
|
239
|
+
entirely (c) value-divergent is recorded but non-blocking; only (a)+(b) findings
|
|
240
|
+
count.
|
|
241
|
+
|
|
242
|
+
- Any (a) or (b) REJECT or FAIL = revise and re-review. **This one blocks.**
|
|
228
243
|
- Codex-only REJECT with (a)/(b) findings + others APPROVE = likely real issue, investigate before overriding
|
|
229
244
|
- Codex REJECT with only (c) findings = expected per Codex value-system divergence; non-blocking
|
|
230
245
|
|
|
246
|
+
Reference figures, recorded and never sufficient on their own. Read what the
|
|
247
|
+
approving replies actually said before counting them:
|
|
248
|
+
|
|
249
|
+
- 3/4 APPROVE (no (a)/(b) REJECT) on the current 4-reviewer roster — the literal
|
|
250
|
+
ratio tracks the roster size and changed on 2026-09-05 when gpt-5.5 retired
|
|
251
|
+
- 4/4 APPROVE (including Codex) = the strongest reference signal available, still
|
|
252
|
+
a reference
|
|
253
|
+
- Legacy 3-reviewer mode: 2/3 APPROVE
|
|
254
|
+
|
|
255
|
+
Normative statement and the carryover/new split: L1 `multi_llm_review_workflow`
|
|
256
|
+
§ Convergence Rules. Aggregation rule: project CLAUDE.md.
|
|
257
|
+
|
|
231
258
|
### Bug Category Differentiation Across Rounds
|
|
232
259
|
|
|
233
260
|
| Review Phase | Typical Bug Category | Example |
|
|
@@ -324,6 +351,26 @@ MECHANISM of a sound invariant is (c) -> §11 / implementation review." This con
|
|
|
324
351
|
Codex APPROVE" (not always reachable) into "freeze when only (c)/mechanism findings remain,"
|
|
325
352
|
which is decidable by the orchestrator and resistant to value-divergence stalling.
|
|
326
353
|
|
|
354
|
+
## Changelog
|
|
355
|
+
|
|
356
|
+
- **v1.6 (2026-09-06)**: § Convergence Rule (Updated) rewritten. It had stated
|
|
357
|
+
`3/4 APPROVE = proceed to next step` and `4/4 APPROVE = merge-ready` with no
|
|
358
|
+
note that the ratio is a reference value — while L1 `multi_llm_review_workflow`
|
|
359
|
+
§ Convergence Rules names *this* section as the normative detail it defers to.
|
|
360
|
+
So the document that says "the ratio is not the primary close" pointed at a
|
|
361
|
+
document that said it was. Now: the closing condition (new (a)+(b) P0 = 0,
|
|
362
|
+
carryover counted separately, operator declares the freeze) is stated first and
|
|
363
|
+
the blocking rule — any (a)/(b) REJECT — is separated from the reference
|
|
364
|
+
figures. Two supporting facts moved into the section because they are the
|
|
365
|
+
reason the ratio cannot be the criterion: Codex went 24 of 24 reviews without
|
|
366
|
+
APPROVE on one design loop, and both 2026-08 threads closed by (a)+(b)
|
|
367
|
+
exhaustion without reaching their ratio. The "Codex APPROVE = strongest
|
|
368
|
+
merge-readiness signal" line keeps its wording and gains "as a reference
|
|
369
|
+
signal, not a gate". Prompted by the operator after this agent reported an
|
|
370
|
+
APPROVE ratio to them as a gate for the third time; the wrong belief was
|
|
371
|
+
traceable to the text here, not to the tooling, which had been correct since
|
|
372
|
+
v0.7 INV-R2 (`reference_verdict`) and v0.10.1 (`vote_tally`).
|
|
373
|
+
|
|
327
374
|
## Refinement Source
|
|
328
375
|
|
|
329
376
|
Profiles in this knowledge are refined from accumulated L2 contexts named with prefix
|