kairos-chain 3.87.1 → 3.88.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +86 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +76 -31
- data/templates/knowledge/multi_llm_reviewer_evaluation/multi_llm_reviewer_evaluation.md +23 -1
- data/templates/skillsets/agent/plugin/SKILL.md +1 -1
- data/templates/skillsets/agent/test/test_agent_guard_verdict_record.rb +448 -0
- data/templates/skillsets/agent/tools/agent_step.rb +225 -32
- data/templates/skillsets/minimum_nomic/bin/run_gm.rb +104 -2
- data/templates/skillsets/multi_llm_review/config/multi_llm_review.yml +10 -8
- data/templates/skillsets/multi_llm_review/skillset.json +2 -2
- data/templates/skillsets/multi_llm_review/test/test_multi_llm_review.rb +70 -0
- data/templates/skillsets/multi_llm_review/tools/multi_llm_review.rb +76 -3
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c37ee945277e2917d116b02aeb8b47957f97363d8428aae7b54e17a732ca0d94
|
|
4
|
+
data.tar.gz: 62df9a6d5f05afd01c3aae950959d6398a22e979270183a154e4fd211d2b78c8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 74101139a4c243c77119530a5a723c0e1de9fdb6224c1b5f776f78b250040c1e4aade60b93cb75d5cbef3afac2609e2832763a079fc38bddd6ecac84e73f2c32
|
|
7
|
+
data.tar.gz: 0bf5337f8e6b5985967711300dfe64554e7ff55bb67f7f1dff1d3fca1aefa28fa74677477d5ed3c32cc9b843556434ca692a7254d26a74d68a32807408091621
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,92 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.88.1] - 2026-10-06
|
|
8
|
+
|
|
9
|
+
### Fixed — agent guard: the verdict reaches the record
|
|
10
|
+
|
|
11
|
+
Found in a guarded trial on 3.88.0. The guard judged correctly, but its
|
|
12
|
+
judgment never reached the record: the chain held only autoexec's "execution
|
|
13
|
+
complete" for an act the guard had failed, progress said `completed`, and the
|
|
14
|
+
FAIL survived only in a mutable session file.
|
|
15
|
+
|
|
16
|
+
- **The verdict is chain-recorded.** Right after judging, before any halt
|
|
17
|
+
checkpoint and before any merge, the driver records `kind:
|
|
18
|
+
agent_guard_verdict`: the verdict, the pinned spec hash, the failed checks,
|
|
19
|
+
the planned route, and the act's error if it had one.
|
|
20
|
+
- **A verdict the chain does not take stops the loop.** Whatever the verdict,
|
|
21
|
+
nothing merges and the cycle halts for the operator. The halt reason names
|
|
22
|
+
the lost verdict and its reason, why the record failed, and whether the act
|
|
23
|
+
already took effect (in-process) or stays quarantined (confined). Responses
|
|
24
|
+
and the progress `guard_record` carry `verdict_recorded: false` and
|
|
25
|
+
`lost_verdict`. On a backend whose chain cannot be appended to (sqlite and
|
|
26
|
+
postgresql do not declare a ledger path), every guarded cycle now halts
|
|
27
|
+
this way instead of running unrecorded.
|
|
28
|
+
- **Progress and responses follow the verdict.** A failed verdict is recorded
|
|
29
|
+
and reported as `failed`, not `completed`; REFLECT is shown the verdict, and
|
|
30
|
+
the next cycle's executor context says the cycle failed. Manual responses
|
|
31
|
+
carry `guard` (verdict, failed checks) and `act_error`. The manual risk-resume
|
|
32
|
+
path reports a guard halt as `guard_halt`, not `ok`. Autonomous responses
|
|
33
|
+
carry the halt reason and, per cycle, `guard_verdict`, `guard_reason` and
|
|
34
|
+
`verdict_recorded`; steps set aside for a person are still listed when a
|
|
35
|
+
cycle halts. A result carrying only an error is no longer reported as
|
|
36
|
+
`completed`.
|
|
37
|
+
|
|
38
|
+
Not fixed here: the confined route still cannot run. `agent_execute` is not
|
|
39
|
+
registered by the agent SkillSet, so a plan naming a file tool fails at ACT
|
|
40
|
+
(with the guard on, that FAIL is now recorded), and a file acceptance check
|
|
41
|
+
cannot PASS in this version. Wiring it is separate work.
|
|
42
|
+
|
|
43
|
+
Also ships `minimum_nomic`'s `bin/run_gm.rb` instance-local changes (a11d103),
|
|
44
|
+
so `system_upgrade` stops reverting them.
|
|
45
|
+
|
|
46
|
+
## [3.88.0] - 2026-09-30
|
|
47
|
+
|
|
48
|
+
### Added — `multi_llm_review` 0.11.1: an artifact named by path and sha256
|
|
49
|
+
|
|
50
|
+
`multi_llm_review` takes an optional `artifact_sha256`, and `artifact_content`
|
|
51
|
+
is no longer required. When `artifact_content` is empty, the tool reads
|
|
52
|
+
`artifact_path` inside its working directory and refuses unless the bytes hash
|
|
53
|
+
to the pin; a path outside the working directory, a file that is not a regular
|
|
54
|
+
file, a file that changes between the check and the read, and non-UTF-8 content
|
|
55
|
+
are refused, all before any seat is dispatched. When `artifact_content` is
|
|
56
|
+
carried together with a pin, the content must hash to the pin. Callers that pass
|
|
57
|
+
`artifact_content` without a pin are unaffected.
|
|
58
|
+
|
|
59
|
+
Why: the agent SkillSet's DECIDE phase writes tool arguments itself, within an
|
|
60
|
+
output budget, and `autoexec_run` passes them as written. On 2026-09-30 a
|
|
61
|
+
101,549-byte artifact reached the plan as a placeholder string, and approving
|
|
62
|
+
the plan would have sent the placeholder to every seat. With the pin, the plan
|
|
63
|
+
names the file and its hash, and the plan's own hash binds the content.
|
|
64
|
+
|
|
65
|
+
Claude Code keeps a connected server's tool schema until the session restarts,
|
|
66
|
+
so a client started before the upgrade still sees `artifact_content` as required.
|
|
67
|
+
|
|
68
|
+
### Changed — Opus 4.6 retired from the review roster; Sonnet 5.5 takes its seat
|
|
69
|
+
|
|
70
|
+
Operator decision, 2026-09-30: Opus 4.6 is retired from every role it held —
|
|
71
|
+
the Claude CLI review seat and the sub-author role for self-referential
|
|
72
|
+
passages — and Sonnet 5.5 at effort high takes both. The roster entry becomes
|
|
73
|
+
`claude-sonnet-5-5` / `claude_cli_sonnet5.5`; the roster is still four seats.
|
|
74
|
+
|
|
75
|
+
This is the operator's decision, not a like-for-like succession. Opus 4.6 held
|
|
76
|
+
both roles for a documented bias profile (ambiguity-preserving,
|
|
77
|
+
self-reference-friendly); Sonnet 5.5 is uncalibrated, with two review rounds on
|
|
78
|
+
2026-09-30 and nothing else, so no bias rationale is written for it.
|
|
79
|
+
|
|
80
|
+
- L1 `multi_llm_review_workflow` 3.15.0 → 3.15.1 and
|
|
81
|
+
`multi_llm_reviewer_evaluation` 1.6 → 1.7: text a reader follows as current
|
|
82
|
+
procedure now names Sonnet 5.5. Measurements, seat profiles, dated
|
|
83
|
+
diagnoses and changelog entries that name Opus 4.6 stay as written, because
|
|
84
|
+
they are facts about that model. The dev-repo copies of these two entries and
|
|
85
|
+
of `design_to_implementation_workflow` had drifted behind the shipped ones
|
|
86
|
+
(1.5 and 1.1 against 1.6 and 1.2) and are synced.
|
|
87
|
+
- The agent SkillSet's plugin `SKILL.md` names Sonnet 5.5 as the sub-author.
|
|
88
|
+
|
|
89
|
+
Upgrading: `kairos-chain upgrade` never overwrites an instance's
|
|
90
|
+
`config/multi_llm_review.yml`, so an instance keeps Opus 4.6 until its own
|
|
91
|
+
roster entry is edited. The config's comment on the entry says how.
|
|
92
|
+
|
|
7
93
|
## [3.87.1] - 2026-09-24
|
|
8
94
|
|
|
9
95
|
### Changed — shipped subagents name their model and effort in full
|
data/lib/kairos_mcp/version.rb
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.15.
|
|
4
|
+
version: "3.15.1"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -141,7 +141,7 @@ these numbers.
|
|
|
141
141
|
|
|
142
142
|
| Seat | Runs | APPROVE rate design / impl / doc | Median wall s | Median output chars | (c) share, labelled only |
|
|
143
143
|
|---|---|---|---|---|---|
|
|
144
|
-
| `claude_cli_opus4.6` | 134 | 64% (31/48) / 71% (27/38) / 86% (20/23) | 54 | 4,536 | 70% (19/27) |
|
|
144
|
+
| `claude_cli_opus4.6` (retired 2026-09-30) | 134 | 64% (31/48) / 71% (27/38) / 86% (20/23) | 54 | 4,536 | 70% (19/27) |
|
|
145
145
|
| `cursor_composer2.5` | 138 | 13% (7/51) / 48% (24/50) / 25% (6/24) | 133 | 3,795 | 100% (2/2) |
|
|
146
146
|
| `codex_gpt5.6-sol` (retired 2026-09-05) | 138 | 1% (1/54) / 11% (7/60) / 4% (1/24) | 113 | 2,038 | 3% (5/140) |
|
|
147
147
|
| `claude_team_opus-5` (persona) | 131 | 0% (0/51) / 12% (7/56) / 0% (0/22) | not measured per seat | 11,512 | 27% (374/1362) |
|
|
@@ -155,16 +155,24 @@ every row from 2026-09-05 onward for a second reason — the corpus was gathered
|
|
|
155
155
|
with reviewers at medium effort, and reviewers now run at high (see § Thinking
|
|
156
156
|
Effort Configuration), so post-swap rounds are not directly comparable to it.
|
|
157
157
|
|
|
158
|
+
The opus4.6 row is likewise history. Opus 4.6 was retired on 2026-09-30
|
|
159
|
+
(operator decision) and `claude_cli_sonnet5.5` — Sonnet 5.5 at effort high —
|
|
160
|
+
took the seat. Sonnet 5.5 had held it for two rounds by then and has no
|
|
161
|
+
profile, so nothing in the opus4.6 row, including its lenient approval rate,
|
|
162
|
+
may be read as a description of the new occupant.
|
|
163
|
+
|
|
158
164
|
Selection consequences, each tied to the number above it:
|
|
159
165
|
|
|
160
|
-
- **
|
|
161
|
-
least.** Codex
|
|
162
|
-
advisory rate in the corpus (5 of 140 labelled
|
|
163
|
-
document reviews in 23 while 19 of its 27
|
|
164
|
-
|
|
165
|
-
with opus4.6's vote already inside it
|
|
166
|
-
|
|
167
|
-
|
|
166
|
+
- **In this corpus a `codex` APPROVE carried the most information and an
|
|
167
|
+
`opus4.6` APPROVE the least.** Codex approved 1 design review in 54 and paired
|
|
168
|
+
that with the lowest advisory rate in the corpus (5 of 140 labelled
|
|
169
|
+
findings). Opus4.6 approved 20 document reviews in 23 while 19 of its 27
|
|
170
|
+
labelled findings were advisory, so the `3/4 APPROVE` threshold was met, in
|
|
171
|
+
practice, with opus4.6's vote already inside it. Both seats have since
|
|
172
|
+
changed occupant (codex 2026-09-05, opus4.6 2026-09-30), and neither
|
|
173
|
+
successor is calibrated: until they are, do not assume which seat casts the
|
|
174
|
+
lenient vote — read each APPROVE for what it says. (That threshold is a
|
|
175
|
+
reference figure, not a gate — see § Convergence Rules.)
|
|
168
176
|
- **Volume anti-correlates with signal.** The persona seat raises 3,249 of the
|
|
169
177
|
5,286 findings (61%) and 27% of its labelled ones are advisory. Seat personas
|
|
170
178
|
when breadth is wanted; do not seat them to obtain a verdict.
|
|
@@ -497,10 +505,11 @@ they disagree, the config is right and this section is stale.
|
|
|
497
505
|
- [ ] Agent Team Personas model: = orchestrator model (NOT a different model),
|
|
498
506
|
unless you are running personas on a different model on purpose — then
|
|
499
507
|
it is whatever you declare as persona_model (see § Persona execution model)
|
|
500
|
-
- [ ] Subprocess CLI model: Opus 4.6
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
508
|
+
- [ ] Subprocess CLI model: Sonnet 5.5 (`claude-sonnet-5-5`; Opus 4.6 held this
|
|
509
|
+
slot until its retirement on 2026-09-30). The other Claude roster slot is
|
|
510
|
+
Opus 5.5, which under the default "delegate" strategy is taken by your
|
|
511
|
+
persona team rather than spawned — so when you are Opus 5.5, Sonnet 5.5 is
|
|
512
|
+
the only Claude CLI subprocess
|
|
504
513
|
- [ ] Codex model: gpt-6-astra, with -m. One codex slot since gpt-5.5 was
|
|
505
514
|
retired 2026-09-05 — do not add a second codex entry expecting the old
|
|
506
515
|
cross-generation pairing
|
|
@@ -853,8 +862,8 @@ outside this repository — see the incident recorded in § Pre-flight checklist
|
|
|
853
862
|
| **Codex** | `codex exec -m <model> -c model_reasoning_effort=high` | stdin pipe: `cat prompt.md \| codex exec -m <model> -` | `-o /path/output.md` | gpt-6-astra — one slot since gpt-5.5 was retired 2026-09-05 |
|
|
854
863
|
| **Cursor Agent** | `agent -p --model composer-2.5` | File reference (stdin NOT supported) | stdout redirect: `> output.md` | composer-2.5, passed explicitly — never relying on the CLI default |
|
|
855
864
|
| **Claude Code** | Agent tool (internal) | Direct prompt string | Write to workspace file | Orchestrator model, or the declared `persona_model` when personas run elsewhere |
|
|
856
|
-
| **Claude CLI (
|
|
857
|
-
| **Claude CLI (frontier)** | `claude -p --model claude-opus-5-5` | stdin pipe: `cat prompt.md \| claude -p --model claude-opus-5-5` | stdout redirect: `> output.md` | Runs only when Opus 5.5 is *not* the orchestrator; when it is, that slot is taken by the persona team |
|
|
865
|
+
| **Claude CLI (Sonnet 5.5)** | `claude -p --model claude-sonnet-5-5 --effort high` | stdin pipe: `cat prompt.md \| claude -p --model claude-sonnet-5-5 --effort high` | stdout redirect: `> output.md` | Sonnet 5.5 — took Opus 4.6's slot on 2026-09-30 by operator decision; uncalibrated |
|
|
866
|
+
| **Claude CLI (frontier)** | `claude -p --model claude-opus-5-5 --effort high` | stdin pipe: `cat prompt.md \| claude -p --model claude-opus-5-5 --effort high` | stdout redirect: `> output.md` | Runs only when Opus 5.5 is *not* the orchestrator; when it is, that slot is taken by the persona team |
|
|
858
867
|
|
|
859
868
|
`--bare` must NOT be passed (established 2026-07-23): it skips credential
|
|
860
869
|
loading and the subprocess fails with "Not logged in". The project-instruction
|
|
@@ -874,7 +883,7 @@ Based on cross-evaluation experiment (7 models × 4 tasks + Nomic, 518 CLI calls
|
|
|
874
883
|
|------|-------|-------------|-----------|
|
|
875
884
|
| **Primary (orchestrator)** | session default | Claude Code's own setting | Not set by this config — record the level actually in effect (see the 2026-09-24 note) |
|
|
876
885
|
| **Reviewer: Agent Team** | = orchestrator, or the declared `persona_model` | the session's level, unless the persona's agent file sets `effort:` | Personas inherit whichever model — and effort — actually runs them |
|
|
877
|
-
| **Reviewer: Claude CLI** |
|
|
886
|
+
| **Reviewer: Claude CLI** | Sonnet 5.5, plus any frontier roster slot the orchestrator is not | `--effort high` (config `effort: high`) | Operator instruction 2026-09-05; supersedes the 2026-04-29 default-effort policy — see the note below the table |
|
|
878
887
|
| **Coding sub-agent** | Opus 5.5 | `--effort high` | Operator default for KairosChain (2026-09-24); not measured here (see note) |
|
|
879
888
|
| **Design sub-agent** | Opus 5.5 | `--effort high` | Operator default for KairosChain (2026-09-24); not measured here (see note) |
|
|
880
889
|
| **Codex** | GPT-6-astra / GPT-5.5 | `-c model_reasoning_effort=high` | Same operator instruction. The earlier "(no flag) / fixed effort" entry was wrong: codex_adapter has always emitted this flag when the roster set `effort` |
|
|
@@ -904,9 +913,19 @@ Note (2026-07-26): the coding / design sub-agent rows moved from Opus 4.7
|
|
|
904
913
|
(retired 2026-06-10) to Opus 5. Their effort values are Anthropic's published
|
|
905
914
|
starting points for Opus 5, not measurements from this project — treat them as
|
|
906
915
|
a starting point to sweep down from, not a validated setting. Separately, the
|
|
907
|
-
**sub-author** role for self-referential passages
|
|
916
|
+
**sub-author** role for self-referential passages stayed Opus 4.6: it was chosen
|
|
908
917
|
for its ambiguity-preserving bias, not for capability, so a frontier successor
|
|
909
|
-
|
|
918
|
+
did not replace it. (Superseded 2026-09-30 — see the next note.)
|
|
919
|
+
|
|
920
|
+
Note (2026-09-30): Opus 4.6 is retired everywhere, the **sub-author** role
|
|
921
|
+
included, by operator decision; Sonnet 5.5 (`claude-sonnet-5-5`, effort high)
|
|
922
|
+
takes it. This is not a like-for-like succession. The reason Opus 4.6 held the
|
|
923
|
+
role was a documented bias profile, and Sonnet 5.5 has none — it had held the
|
|
924
|
+
review seat for two rounds when the decision was made, and nothing else. So the
|
|
925
|
+
sub-author choice rests on the operator's decision alone: do not cite an
|
|
926
|
+
ambiguity-preserving or other bias rationale for Sonnet 5.5 until a measurement
|
|
927
|
+
supports one, and record its sub-author outputs so a profile can accumulate in
|
|
928
|
+
`multi_llm_reviewer_evaluation`.
|
|
910
929
|
|
|
911
930
|
Note (2026-09-24): the frontier rows moved from Opus 5 to Opus 5.5 (operator
|
|
912
931
|
instruction — 5.5 is now Claude Code's default orchestrator and takes over
|
|
@@ -960,11 +979,12 @@ record no longer claims a model that did not answer.
|
|
|
960
979
|
orchestrating LLM MUST pass its own model identifier as `orchestrator_model`.
|
|
961
980
|
|
|
962
981
|
**Rationale**: The reviewer roster contains more than one Claude entry (Opus 5.5
|
|
963
|
-
and
|
|
982
|
+
and Sonnet 5.5 as of 2026-09-30; Opus 4.6 held the second entry until then, and
|
|
983
|
+
Opus 5 held the frontier entry from 2026-07-26). A frontier slot sits in the roster on purpose:
|
|
964
984
|
when it is the current session model it matches `orchestrator_model` and becomes
|
|
965
985
|
the persona-team slot; when it is not, it is dispatched as a `claude -p`
|
|
966
986
|
subprocess. With the current two-entry Claude side, an Opus 5.5 orchestrator leaves
|
|
967
|
-
|
|
987
|
+
Sonnet 5.5 as the only Claude CLI subprocess. No per-orchestrator config branch is
|
|
968
988
|
needed either way.
|
|
969
989
|
To avoid the orchestrator reviewing its
|
|
970
990
|
own output (no independent signal), the dispatcher excludes or delegates the
|
|
@@ -1025,7 +1045,7 @@ multi_llm_review(
|
|
|
1025
1045
|
- If `orchestrator_model` is `null` or unmatched, full roster runs (back-compat).
|
|
1026
1046
|
|
|
1027
1047
|
**Manual-mode equivalent**: When orchestrating by hand, do not assign yourself
|
|
1028
|
-
as a subprocess reviewer. Run the Claude CLI subprocess reviewers (
|
|
1048
|
+
as a subprocess reviewer. Run the Claude CLI subprocess reviewers (Sonnet 5.5, plus
|
|
1029
1049
|
any frontier roster slot you are not); if your own model matches one of them,
|
|
1030
1050
|
skip that entry and use the after-exclusion convergence rule.
|
|
1031
1051
|
|
|
@@ -1040,7 +1060,7 @@ artifact, use `escalate: true` — see § Reserve observers below.
|
|
|
1040
1060
|
The `delegate` strategy lets the orchestrator perform persona-based "Agent Team"
|
|
1041
1061
|
review in its own context — preserving inherited project context that a fresh
|
|
1042
1062
|
`claude -p` subprocess loses. Subprocess reviewers (codex, cursor, Claude CLI
|
|
1043
|
-
|
|
1063
|
+
Sonnet 5.5 and the non-orchestrator frontier model) remain single-LLM.
|
|
1044
1064
|
|
|
1045
1065
|
**Why**: The orchestrator already holds the artifact in context with full project
|
|
1046
1066
|
awareness. Re-shipping it to a sandboxed subprocess discards that context. Same-
|
|
@@ -1349,8 +1369,8 @@ readable until GC. Read them directly and synthesize manually, then re-run
|
|
|
1349
1369
|
before, silently swapping the model behind an unchanged role label
|
|
1350
1370
|
- **Codex workspace**: `-C /path/to/workspace` to set working directory
|
|
1351
1371
|
- **Claude Agent paths**: Write within workspace (e.g., `log/`), not `/tmp`
|
|
1352
|
-
- **Claude CLI (
|
|
1353
|
-
- **Claude CLI parallelism**: Agent tool (internal, orchestrator model) + Bash `claude -p` (external,
|
|
1372
|
+
- **Claude CLI (Sonnet 5.5 / any non-orchestrator frontier slot)**: `claude -p --model claude-sonnet-5-5` (likewise `claude-opus-5-5`) runs as external process. Uses stdin pipe (like Codex). Do NOT pass `--bare` — it skips credential loading and the subprocess dies with "Not logged in" (established 2026-07-23). Project-instruction bias is suppressed via `review_context: independent`, not via `--bare`
|
|
1373
|
+
- **Claude CLI parallelism**: Agent tool (internal, orchestrator model) + Bash `claude -p` (external, Sonnet 5.5 and any other Claude roster slot the orchestrator is not) run truly in parallel as separate processes
|
|
1354
1374
|
- **Claude CLI file access**: a plain `claude -p` review subprocess should not need file access. Ensure the review prompt includes all artifact content inline (rule #6). Use `--add-dir` + `--allowedTools "Read,Glob,Grep"` if file access is genuinely needed, and accept that CLAUDE.md is loaded (the old `--bare` workaround is unusable — see above)
|
|
1355
1375
|
|
|
1356
1376
|
## Prompt Generation Rules
|
|
@@ -1494,7 +1514,7 @@ Step 2: Detect environment, and check the roster against config
|
|
|
1494
1514
|
and treat them as the roster. Detection only tells you whether a default has
|
|
1495
1515
|
drifted; the model each slot runs is named on the command line.
|
|
1496
1516
|
- Report: "Auto mode: Codex (gpt-6-astra), Cursor (composer-2.5),
|
|
1497
|
-
Claude Team (orchestrator model), Claude CLI (
|
|
1517
|
+
Claude Team (orchestrator model), Claude CLI (sonnet-5.5)"
|
|
1498
1518
|
|
|
1499
1519
|
Step 3: Execute the configured roster in parallel (currently 4 slots, one of
|
|
1500
1520
|
which is your own persona team)
|
|
@@ -1502,9 +1522,9 @@ Step 3: Execute the configured roster in parallel (currently 4 slots, one of
|
|
|
1502
1522
|
- Bash(background): agent -p --trust --model composer-2.5 "Read prompt and review..." > log/review_cursor.md
|
|
1503
1523
|
(no effort flag — Cursor has no effort control)
|
|
1504
1524
|
- Agent(background): Claude Team (orchestrator model, e.g. Opus 5.5) → write to log/review_claude_team_opus5.5.md
|
|
1505
|
-
- Bash(background): cat prompt.md | claude -p --model claude-
|
|
1506
|
-
(add a line per further Claude roster slot you are not; with the 2026-09-
|
|
1507
|
-
roster an Opus 5.5 orchestrator has none, so
|
|
1525
|
+
- Bash(background): cat prompt.md | claude -p --model claude-sonnet-5-5 --effort high > log/review_claude_sonnet5.5.md 2>log/review_claude_sonnet5.5.stderr.log
|
|
1526
|
+
(add a line per further Claude roster slot you are not; with the 2026-09-30
|
|
1527
|
+
roster an Opus 5.5 orchestrator has none, so sonnet-5.5 is the only one)
|
|
1508
1528
|
|
|
1509
1529
|
Step 4: Collect and validate
|
|
1510
1530
|
- Wait for all to complete (background task notifications)
|
|
@@ -1541,7 +1561,7 @@ log/{artifact}_review{N}_{llm_id}_{date}.md # Individual reviews
|
|
|
1541
1561
|
log/{artifact}_review{N}_consensus_{date}.md # Consensus analysis
|
|
1542
1562
|
```
|
|
1543
1563
|
|
|
1544
|
-
LLM identifiers: `claude_cli_opus5.5`, `
|
|
1564
|
+
LLM identifiers: `claude_cli_opus5.5`, `claude_cli_sonnet5.5`,
|
|
1545
1565
|
`codex_gpt6-astra`, `cursor_composer2.5`, `cursor_gpt5.4`,
|
|
1546
1566
|
`cursor_premium`. The delegated slot is reported as `claude_team_<model>`
|
|
1547
1567
|
(e.g. `claude_team_claude-opus-5-5`), assembled at collect time — the roster's
|
|
@@ -1553,7 +1573,8 @@ retired 2026-07-26: `claude_cli_fable5` — five consecutive non-substantive
|
|
|
1553
1573
|
returns, 85-128 characters in 5-7 seconds, no findings and no verdict text;
|
|
1554
1574
|
retired 2026-09-05: `codex_gpt5.6-sol`, replaced by `codex_gpt6-astra`, and
|
|
1555
1575
|
`codex_gpt5.5`, not replaced; retired 2026-09-24: `claude_cli_opus5`, succeeded in
|
|
1556
|
-
the same slot by `claude_cli_opus5.5
|
|
1576
|
+
the same slot by `claude_cli_opus5.5`; retired 2026-09-30: `claude_cli_opus4.6`,
|
|
1577
|
+
succeeded in the same slot by `claude_cli_sonnet5.5`. Runs recorded under a retired identifier keep it —
|
|
1557
1578
|
the label names the model that answered, so renaming old records would attribute
|
|
1558
1579
|
one model's findings to another)
|
|
1559
1580
|
|
|
@@ -1948,6 +1969,30 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1948
1969
|
§ Thinking Effort Configuration, including that the persona seat follows the
|
|
1949
1970
|
session's level while CLI seats follow the config.
|
|
1950
1971
|
|
|
1972
|
+
- Opus 4.6 retired → Sonnet 5.5 (v3.15.1, 2026-09-30, operator decision:
|
|
1973
|
+
「全部Opus4.6は引退させて、Sonnet5.5に置き換えて下さい」). Opus 4.6 leaves every
|
|
1974
|
+
role it held — the Claude CLI review seat and the sub-author role — and
|
|
1975
|
+
Sonnet 5.5 at effort high takes both. The seat's label becomes
|
|
1976
|
+
`claude_cli_sonnet5.5`; nothing else about the roster changes (still 4 seats:
|
|
1977
|
+
Opus 5.5 as orchestrator and persona team, Sonnet 5.5 CLI, codex gpt-6-astra,
|
|
1978
|
+
cursor composer-2.5). Only text that prescribes current behaviour was
|
|
1979
|
+
rewritten; two dated passages were annotated rather than rewritten — the
|
|
1980
|
+
Step 0.1 row label gains "(retired 2026-09-30)" with its numbers untouched,
|
|
1981
|
+
and the 2026-07-26 sub-author note is put in the past tense and pointed at
|
|
1982
|
+
the note that supersedes it. The two Claude CLI rows of the Tool Matrix
|
|
1983
|
+
also gain `--effort high`, which they had lacked while the table below them
|
|
1984
|
+
required it. The `claude_cli_opus4.6` row in § Step 0.1, the effort measurements
|
|
1985
|
+
taken on the Opus 4.6 / 4.7 generation, the 2026-08-06 no_verdict diagnosis
|
|
1986
|
+
and every earlier changelog entry stay as written, because they are facts
|
|
1987
|
+
about Opus 4.6. Recorded as the operator's decision rather than as a
|
|
1988
|
+
succession: Opus 4.6 was kept in both roles for a documented bias profile
|
|
1989
|
+
(ambiguity-preserving, self-reference-friendly), while Sonnet 5.5 is
|
|
1990
|
+
uncalibrated — two review rounds on 2026-09-30 and nothing else — so no bias
|
|
1991
|
+
rationale is written for it. Config: `multi_llm_review` 0.11.1 roster entry
|
|
1992
|
+
`claude-sonnet-5-5`; the entry's comment says how to restore Opus 4.6.
|
|
1993
|
+
Upgrading does not rewrite an instance's own `config/multi_llm_review.yml`,
|
|
1994
|
+
so an existing instance keeps Opus 4.6 until that entry is edited by hand.
|
|
1995
|
+
|
|
1951
1996
|
**Key insight**: Design reviews and implementation reviews find
|
|
1952
1997
|
**categorically different bugs**. Both phases are necessary. The corollary that
|
|
1953
1998
|
cost eight rounds to learn: **which of the two you are in is readable from the
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_reviewer_evaluation
|
|
3
3
|
description: "Multi-LLM reviewer performance evaluation — strengths, weaknesses, value-system biases, and recommended workflows. Based on 185+ reviews (Phase 1, 2026-02 to 03) + Phase 2 Case A 4-round Codex bias study (2026-05-04)."
|
|
4
|
-
version: "1.
|
|
4
|
+
version: "1.7"
|
|
5
5
|
tags:
|
|
6
6
|
- multi-llm
|
|
7
7
|
- review
|
|
@@ -47,6 +47,11 @@ Based on 185+ review files across KairosChain development (2026-02-24 to 2026-03
|
|
|
47
47
|
|
|
48
48
|
## Per-Reviewer Profiles
|
|
49
49
|
|
|
50
|
+
> Opus 4.6 was retired on 2026-09-30 (operator decision); Sonnet 5.5 took its
|
|
51
|
+
> review seat (`claude_cli_sonnet5.5`) and the sub-author role. The Opus 4.6
|
|
52
|
+
> profiles below are that model's record and do not describe Sonnet 5.5, which
|
|
53
|
+
> had no profile when the decision was made — two review rounds, and nothing else.
|
|
54
|
+
|
|
50
55
|
### Claude Opus 4.6 (Primary Designer)
|
|
51
56
|
|
|
52
57
|
- **Strength**: Security-level threat modeling, novel architectural alternatives (e.g., discovering `register_gate` as zero-L0 solution)
|
|
@@ -285,6 +290,14 @@ Normative statement and the carryover/new split: L1 `multi_llm_review_workflow`
|
|
|
285
290
|
|
|
286
291
|
> Note: Workflows updated for 4-reviewer default (Opus 4.7 added 2026-04-19). Opus 4.7 profile is provisional pending evaluation data.
|
|
287
292
|
|
|
293
|
+
> Roster note (2026-09-30): the model lines below are the 2026-04-19
|
|
294
|
+
> recommendation (last edited 2026-05-26) and are history, not a roster to launch. Every Claude and
|
|
295
|
+
> Codex model they name has since retired — Opus 4.7 on 2026-06-10, GPT-5.4 on
|
|
296
|
+
> 2026-07-23, Opus 4.6 on 2026-09-30. The roster in use lives in
|
|
297
|
+
> `multi_llm_review/config/multi_llm_review.yml` and nowhere else. What still
|
|
298
|
+
> reads from the lines below is the emphasis per phase, which the Strength
|
|
299
|
+
> Matrix above supports.
|
|
300
|
+
|
|
288
301
|
```
|
|
289
302
|
Design phase: Claude Opus 4.6 + Claude CLI Opus 4.7 + Codex GPT-5.4 + Composer-2.5
|
|
290
303
|
Implementation: Codex GPT-5.4 + Composer-2.5 + Claude Opus 4.6 + Claude CLI Opus 4.7
|
|
@@ -353,6 +366,15 @@ which is decidable by the orchestrator and resistant to value-divergence stallin
|
|
|
353
366
|
|
|
354
367
|
## Changelog
|
|
355
368
|
|
|
369
|
+
- **v1.7 (2026-09-30)**: Opus 4.6 retired by operator decision; Sonnet 5.5
|
|
370
|
+
takes its review seat and the sub-author role. Two notes added, nothing
|
|
371
|
+
deleted: § Per-Reviewer Profiles says the Opus 4.6 profiles are that model's
|
|
372
|
+
record and that Sonnet 5.5 has none, and § Recommended Workflow says its model
|
|
373
|
+
lines are the 2026-04-19 recommendation — every Claude and Codex model in them
|
|
374
|
+
is now retired — with the roster living only in the config. The statistics,
|
|
375
|
+
Strength Matrix, Cost-Benefit and One-Line Summary rows for Opus 4.6 stay:
|
|
376
|
+
they are measurements of Opus 4.6. No profile is written for Sonnet 5.5,
|
|
377
|
+
because no measurement supports one yet.
|
|
356
378
|
- **v1.6 (2026-09-06)**: § Convergence Rule (Updated) rewritten. It had stated
|
|
357
379
|
`3/4 APPROVE = proceed to next step` and `4/4 APPROVE = merge-ready` with no
|
|
358
380
|
note that the ratio is a reference value — while L1 `multi_llm_review_workflow`
|
|
@@ -31,7 +31,7 @@ which spawns external LLMs as subprocesses via adapter classes:
|
|
|
31
31
|
|
|
32
32
|
| Adapter | Subprocess command | Use case |
|
|
33
33
|
|---------|-------------------|----------|
|
|
34
|
-
| `ClaudeCodeAdapter` | `claude -p --output-format json` | Sub-author (
|
|
34
|
+
| `ClaudeCodeAdapter` | `claude -p --output-format json` | Sub-author (Sonnet 5.5), persona reviewers |
|
|
35
35
|
| `CodexAdapter` | `codex exec --sandbox read-only` | Codex review |
|
|
36
36
|
| `CursorAdapter` | `agent -p` | Cursor review |
|
|
37
37
|
| `AnthropicAdapter` | Direct API (no subprocess) | Anthropic API calls |
|