kairos-chain 3.62.0 → 3.63.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +40 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +144 -2
- data/templates/skillsets/multi_llm_review/bin/dispatch_worker.rb +27 -1
- data/templates/skillsets/multi_llm_review/lib/multi_llm_review/consensus.rb +92 -18
- data/templates/skillsets/multi_llm_review/lib/multi_llm_review/dispatcher.rb +26 -2
- data/templates/skillsets/multi_llm_review/lib/multi_llm_review/pending_state.rb +8 -0
- data/templates/skillsets/multi_llm_review/lib/multi_llm_review/prompt_builder.rb +19 -5
- data/templates/skillsets/multi_llm_review/lib/multi_llm_review/sanitizer.rb +11 -1
- data/templates/skillsets/multi_llm_review/skillset.json +2 -2
- data/templates/skillsets/multi_llm_review/test/test_dispatcher_usage.rb +66 -0
- data/templates/skillsets/multi_llm_review/test/test_evidence_fidelity.rb +14 -7
- data/templates/skillsets/multi_llm_review/test/test_multi_llm_review.rb +338 -3
- data/templates/skillsets/multi_llm_review/test/test_mutation_survivors.rb +5 -2
- data/templates/skillsets/multi_llm_review/tools/multi_llm_review_collect.rb +141 -13
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 96a6f46091b47614f44d089e6459dde7e8435cf7caa893c9b335b7359ad25394
|
|
4
|
+
data.tar.gz: 965451d952fb8fa61d3bfcc581241242de0f25eb492dddc56ebe913e4d07683b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1c3812249637a923526a17b29cbbf34549f3f3039f51ae37941fa990ceec99611d68e88de280b39ff673aaf5f64e55be3133b079578b9f2887df8a6a6c81f102
|
|
7
|
+
data.tar.gz: a3688710826e2de6bd2d05ee24b0f9c6194e32e6d9dd8587c776b5e3bfa1b614549af903eee8a9bac295b5e0efc36c5e50537d5a54b8352acd40e05f2ef291c1
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,46 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.63.0] - 2026-08-06
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- **`multi_llm_review` findings gain a weight axis, and a worker death no
|
|
12
|
+
longer discards completed seats** (SkillSet 0.9.1 → 0.10.0, frozen after one
|
|
13
|
+
review round; every deployment-grounded finding fixed in-round).
|
|
14
|
+
|
|
15
|
+
The severity of a finding said what kind of defect it was, never what it
|
|
16
|
+
costs: measured on a three-persona panel, 3 of one round's 7 P0s were
|
|
17
|
+
factually correct findings that cost nobody anything, landing at the same
|
|
18
|
+
severity as a defect that silently corrupts published output. The reviewer
|
|
19
|
+
prompt contract now requires a `[consequence: who is harmed, and how]`
|
|
20
|
+
clause on every P0, and aggregation records an unclaused P0 at P2 with the
|
|
21
|
+
stated severity and the demotion reason kept beside it. Only presence is
|
|
22
|
+
checked, mechanically; whether a stated consequence is real or trivial
|
|
23
|
+
stays the orchestrator's call. The clause is read before the byte bound
|
|
24
|
+
cuts the tail, the first non-empty clause counts, deduplication compares
|
|
25
|
+
issues without their clauses, and the demotion mark survives merging in
|
|
26
|
+
any member order. Reviewer prompts also stop naming the round number, and
|
|
27
|
+
the L1 workflow knowledge (v3.10.0) adds the matching orchestrator-side
|
|
28
|
+
rule: never tell a reviewer its finding count, the round number, or prior
|
|
29
|
+
verdicts — counts a reviewer performs select for finding-production over
|
|
30
|
+
finding-weight.
|
|
31
|
+
|
|
32
|
+
Separately, one stuck seat used to cost a whole round: a stale heartbeat
|
|
33
|
+
killed the detached worker and every completed seat's reply died with it,
|
|
34
|
+
because nothing left the worker's memory until all seats were done. The
|
|
35
|
+
worker now persists each seat's outcome the moment it is decided
|
|
36
|
+
(`partial_results.json`, atomic, single writer — including the
|
|
37
|
+
dispatcher's own deadline skips, so recovery cannot relabel a reached
|
|
38
|
+
seat as lost). The collect tool's crash and timeout branches recover the
|
|
39
|
+
completed seats, but only after the reaper CONFIRMS the worker's death: a
|
|
40
|
+
stale heartbeat is not a death certificate, and a record sealed over a
|
|
41
|
+
live worker would be permanently false once idempotent replay pins it.
|
|
42
|
+
Unconfirmed death falls back to the previous retryable total-loss report.
|
|
43
|
+
Seats the worker never reached enter the denominator as skip rows named
|
|
44
|
+
`worker_crashed_seat_lost`, and the payload carries `worker_failure`
|
|
45
|
+
naming the death and the recovered and lost seat labels.
|
|
46
|
+
|
|
7
47
|
## [3.62.0] - 2026-08-06
|
|
8
48
|
|
|
9
49
|
### Changed
|
data/lib/kairos_mcp/version.rb
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.
|
|
4
|
+
version: "3.10.0"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -498,9 +498,58 @@ The user always has the final say.
|
|
|
498
498
|
`open-question` or `defect` in the round's L2 record
|
|
499
499
|
If no (a)/(b) blocking findings → proceed to next phase
|
|
500
500
|
If any (a)/(b) finding → repeat from [2] with revised artifact
|
|
501
|
+
(the revision obeys § Revision Discipline below)
|
|
501
502
|
(c) findings are recorded as advisory; non-blocking
|
|
502
503
|
```
|
|
503
504
|
|
|
505
|
+
## Revision Discipline (between rounds)
|
|
506
|
+
|
|
507
|
+
> Evidence base: 35 recorded runs across 8 threads, 2026-08-03 → 08-06
|
|
508
|
+
> (tokens in `.kairos/multi_llm_review/pending/`), two of which ran to
|
|
509
|
+
> convergence. Same validation caveat as Step -1: a strong regularity in one
|
|
510
|
+
> instance's logs, not yet reproduced elsewhere. Analysis record: L2
|
|
511
|
+
> `mlr_p0_inflation_analysis_and_opus46_no_verdict_diagnosis_20260806`.
|
|
512
|
+
|
|
513
|
+
The strongest predictor of round N+1's raw P0 count in those logs is whether
|
|
514
|
+
the round-N revision **added mechanism** to the artifact — not the artifact's
|
|
515
|
+
size, not reviewer strictness:
|
|
516
|
+
|
|
517
|
+
- chain_history_erasure v0.5 added two invariants and a recount section
|
|
518
|
+
(draft 9.3k → 19.1k chars): raw P0 went 13 → 38, and 17 of the 38 targeted
|
|
519
|
+
the added or rewritten sections. v0.8, similar in size (18.6k) but authored
|
|
520
|
+
under an explicit "no new mechanism" rule, closed at 13 with one external
|
|
521
|
+
slot finding zero P0s.
|
|
522
|
+
- mlr_evidence_fix R1's fix added four unrequested defensive mechanisms; R2
|
|
523
|
+
returned 41 findings, nearly all of them defects inside the additions. All
|
|
524
|
+
four mechanisms were later removed, each for a measured reason.
|
|
525
|
+
- Deletions never generated findings: chain erasure v0.7 deleted three
|
|
526
|
+
mechanisms — zero new P0s against the deletions, confirmed in writing by
|
|
527
|
+
three slots.
|
|
528
|
+
- Each thread converged within 1–2 rounds of switching to subtractive
|
|
529
|
+
revisions; neither converged while revisions were additive.
|
|
530
|
+
|
|
531
|
+
Rules:
|
|
532
|
+
|
|
533
|
+
1. **A revision closes findings by deletion, correction, or naming — never by
|
|
534
|
+
default-adding.** New mechanisms, invariants, sections, or defensive
|
|
535
|
+
layers do not enter a revision unprompted. If a finding appears to require
|
|
536
|
+
new mechanism, put the question to the operator ("finding X seems to need
|
|
537
|
+
mechanism Y — add, defer to backlog, or drop?") before drafting it in.
|
|
538
|
+
Explanatory prose is a lighter form of the same risk: a sentence added
|
|
539
|
+
only to justify a retreat became the sole blocking finding of the round
|
|
540
|
+
that followed it (mlr_evidence R6 — "adding an explanation creates a new
|
|
541
|
+
claim").
|
|
542
|
+
2. **Prefer a revision author that is not the model whose additions are under
|
|
543
|
+
review.** This extends the existing separation principle — the deciding
|
|
544
|
+
context never authors what judges it — from verification to revision.
|
|
545
|
+
Opus 5 has a measured additive propensity (scope-broadening; the v0.5
|
|
546
|
+
explosion above), but the model is the pressure, not the cause: Fable 5
|
|
547
|
+
also added-and-broke (v0.6's fsync/realpath additions, v0.7's predicate 4
|
|
548
|
+
— a fatal genesis-rejecting defect) until the subtractive rule was
|
|
549
|
+
imposed, and Opus 5 converged mlr_evidence once its revisions became
|
|
550
|
+
subtractive (retreat + removal). Combine both levers: the subtractive
|
|
551
|
+
rule always, a different-model reviser when available.
|
|
552
|
+
|
|
504
553
|
## Review Types
|
|
505
554
|
|
|
506
555
|
| Type | Focus | Reviewers See | Typical Use |
|
|
@@ -548,6 +597,19 @@ and philosophy-aligned finding has been answered is converged whether or not the
|
|
|
548
597
|
numerator moved. Do not treat a reached ratio as sufficient on its own either:
|
|
549
598
|
check what the approving replies actually said before counting them.
|
|
550
599
|
|
|
600
|
+
**Count carryover and new (a)/(b) P0s separately; the machine-side signal of
|
|
601
|
+
convergence is "new P0 = 0", not the APPROVE ratio.** Require each persona to
|
|
602
|
+
state a closure verdict on its own prior-round P0s — closed / open /
|
|
603
|
+
half-closed, with grounds. This format is validated live (chain erasure
|
|
604
|
+
R6–R8) and is what makes the carryover/new split computable. A round whose
|
|
605
|
+
(a)+(b) findings are all carryover with closure verdicts, and whose revision
|
|
606
|
+
drew zero new P0s (observed without exception when the revision was
|
|
607
|
+
subtractive — see § Revision Discipline), is a freeze candidate for the
|
|
608
|
+
operator regardless of the numerator. Neither of the two 2026-08 threads
|
|
609
|
+
ever reached its APPROVE ratio; both closed by (a)+(b) exhaustion + operator
|
|
610
|
+
freeze declaration — the intended close described above, now with a
|
|
611
|
+
measurable trigger.
|
|
612
|
+
|
|
551
613
|
**Escalating raises the bar.** The rule is a ratio over the observers that
|
|
552
614
|
counted, so adding reserve observers with `escalate: true` raises the number of
|
|
553
615
|
agreements required. That is the intended cost of a wider panel, not a defect —
|
|
@@ -1027,6 +1089,26 @@ exclude it would be to exclude every honest terse approval with it. When a round
|
|
|
1027
1089
|
reaches its ratio, read what the approving replies actually said before treating
|
|
1028
1090
|
the ratio as convergence. That judgement is the human's and no rule replaces it.
|
|
1029
1091
|
|
|
1092
|
+
#### A no_verdict streak is a seat-environment signal, not a dead reviewer
|
|
1093
|
+
|
|
1094
|
+
Before treating a slot as dead, read its `raw_text_excerpt` / `stated_text`
|
|
1095
|
+
in the pending record. Diagnosed live (2026-08-06, `claude_cli_opus4.6`,
|
|
1096
|
+
four consecutive no_verdict exclusions across one implementation-review
|
|
1097
|
+
thread): the CLI itself was healthy throughout. The subprocess seat runs
|
|
1098
|
+
sandboxed — no tools, empty working directory — so on **implementation**
|
|
1099
|
+
artifacts that cite file paths, the model attempted to read code before
|
|
1100
|
+
judging: two rounds opened with pseudo-tool-call markup, one opened with
|
|
1101
|
+
"the repository is not accessible" and stated its verdict header only
|
|
1102
|
+
further down, where the positional rule correctly refuses it. The same
|
|
1103
|
+
seat, in the same period, complied on **design** artifacts (verdict on
|
|
1104
|
+
line 1, counted every round). Remedies, in order: state in the subprocess
|
|
1105
|
+
prompt that the seat has no file access and must review the artifact text
|
|
1106
|
+
alone, marking unverifiable claims `[INFERRED]` (the grounding_rules block
|
|
1107
|
+
already licenses this); keep prompt rule #6 (full artifact inline) honest
|
|
1108
|
+
for implementation reviews; only then consider `--add-dir` with read-only
|
|
1109
|
+
tools, accepting CLAUDE.md contamination. A streak that survives those
|
|
1110
|
+
remedies is a real outage.
|
|
1111
|
+
|
|
1030
1112
|
#### Async/Parallel Collect Timing — Iron Rule
|
|
1031
1113
|
|
|
1032
1114
|
When `delegation.parallel.default: true` (the v3.x default), Call 1 returns
|
|
@@ -1107,13 +1189,35 @@ Every review prompt MUST include these 7 items:
|
|
|
1107
1189
|
1. **Output filename table** — so each reviewer knows where to save
|
|
1108
1190
|
2. **Auto-execution commands** — ready-to-run CLI per reviewer
|
|
1109
1191
|
3. **Review instructions** — what to focus on, what NOT to re-review
|
|
1110
|
-
4. **
|
|
1192
|
+
4. **Prior findings to verify** (R2+) — the findings the revision addresses,
|
|
1193
|
+
so the reviewer can judge each closed / open / half-closed. Findings only:
|
|
1194
|
+
no per-reviewer verdict history, no finding counts, no round tallies
|
|
1195
|
+
(see § Reviewer incentive rule)
|
|
1111
1196
|
5. **Context** — architecture summary for reviewers unfamiliar with codebase
|
|
1112
1197
|
6. **Full artifact content inline** — reviewers may not have file access
|
|
1113
1198
|
7. **Severity ratings + output format** — structured template for review output
|
|
1114
1199
|
|
|
1115
1200
|
All prompt content MUST be in **English** for consistent parsing across LLM tools.
|
|
1116
1201
|
|
|
1202
|
+
### Reviewer incentive rule
|
|
1203
|
+
|
|
1204
|
+
**Never tell a reviewer — subprocess or persona — that its finding count is
|
|
1205
|
+
compared across rounds, which round this is, or what verdicts were given
|
|
1206
|
+
before.** A reviewer told its count is watched treats the count as the
|
|
1207
|
+
deliverable, and that selects for finding-*production* over finding-*weight*
|
|
1208
|
+
(observed 2026-08-06: an orchestrator wrote "your finding count is compared
|
|
1209
|
+
across rounds" into persona prompts during the project_orientation_report
|
|
1210
|
+
loop; of the round's 7 P0s, 3 were factually correct findings that cost
|
|
1211
|
+
nobody anything). Convergence — carryover vs new, (a)+(b) exhaustion, the
|
|
1212
|
+
ratio — is measured by the orchestrator from the record, after the replies
|
|
1213
|
+
are in. The reviewer receives the artifact, the review criteria, and the
|
|
1214
|
+
prior findings it must verify. Nothing else about the loop's state.
|
|
1215
|
+
|
|
1216
|
+
What this rule does NOT forbid: passing prior findings for closure
|
|
1217
|
+
verification (rule #4 — that is content, not score-keeping), and the
|
|
1218
|
+
carryover/new split in § Convergence Rules (that is orchestrator-side
|
|
1219
|
+
bookkeeping the reviewer never sees).
|
|
1220
|
+
|
|
1117
1221
|
### XML Block Structure for Review Prompts
|
|
1118
1222
|
|
|
1119
1223
|
Review prompts SHOULD use XML blocks to give LLMs explicit structural contracts.
|
|
@@ -1432,5 +1536,43 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1432
1536
|
fence markers, character classes, digit ranges, word boundaries — and 18 of
|
|
1433
1537
|
27 survived. Mutate the inside of a pattern, not only the pattern.
|
|
1434
1538
|
|
|
1539
|
+
- Revision discipline, new-P0 convergence signal, and no_verdict seat
|
|
1540
|
+
diagnosis (v3.9.0, 2026-08-06): cross-thread analysis of 35 recorded runs
|
|
1541
|
+
(8 threads, 2026-08-03 → 08-06) established that raw P0 growth tracks
|
|
1542
|
+
additive revisions, not reviewer severity — every mechanism a revision
|
|
1543
|
+
added became the next round's battleground, deletions drew zero new P0s
|
|
1544
|
+
in every measured case, and both threads that converged did so within
|
|
1545
|
+
1–2 rounds of switching to subtractive revisions (one under a Fable 5
|
|
1546
|
+
reviser, one under the same Opus 5 orchestrator that had produced the
|
|
1547
|
+
additive explosion). New § Revision Discipline encodes the subtractive
|
|
1548
|
+
rule and the reviser-separation preference. § Convergence Rules gains the
|
|
1549
|
+
carryover/new P0 split, with "new (a)+(b) P0 = 0" as the machine-side
|
|
1550
|
+
freeze-candidate signal — neither thread ever reached its APPROVE ratio;
|
|
1551
|
+
both closed by (a)+(b) exhaustion + operator freeze. § Substance and the
|
|
1552
|
+
denominator gains the seat-environment diagnosis: `claude_cli_opus4.6`'s
|
|
1553
|
+
four-round no_verdict streak was the sandboxed seat colliding with
|
|
1554
|
+
implementation artifacts (pseudo-tool-calls, "repository not accessible"
|
|
1555
|
+
preamble), not a dead reviewer — the same seat counted every round on
|
|
1556
|
+
design artifacts in the same period. Analysis record: L2
|
|
1557
|
+
`mlr_p0_inflation_analysis_and_opus46_no_verdict_diagnosis_20260806`
|
|
1558
|
+
|
|
1559
|
+
- Reviewer incentive rule and the finding weight axis (v3.10.0, 2026-08-06):
|
|
1560
|
+
§ Prompt Generation Rules gains the Reviewer incentive rule — reviewer
|
|
1561
|
+
prompts never mention finding counts, round numbers, or prior verdicts;
|
|
1562
|
+
convergence is measured orchestrator-side from the record, and prior
|
|
1563
|
+
findings are passed for closure verification only (rule #4 reworded
|
|
1564
|
+
accordingly, from "review history table" to "prior findings to verify").
|
|
1565
|
+
Motivating observation: an orchestrator told personas their counts were
|
|
1566
|
+
compared across rounds, and 3 of the round's 7 P0s were factually correct
|
|
1567
|
+
findings that cost nobody anything. In the same change the SkillSet
|
|
1568
|
+
(0.10.0) adds the weight axis mechanically: the prompt contract requires a
|
|
1569
|
+
`[consequence: who is harmed, and how]` clause on every P0, and
|
|
1570
|
+
aggregation records a P0 without one at P2, keeping the stated severity
|
|
1571
|
+
and the demotion reason beside it (`severity_stated` /
|
|
1572
|
+
`severity_demoted: consequence_missing`). Presence is checked
|
|
1573
|
+
mechanically; whether a stated consequence is real or trivial stays the
|
|
1574
|
+
orchestrator's call, per the (a)/(b)/(c) discipline. Handoff record: L2
|
|
1575
|
+
`handoff_mlr_finding_weight_axis_and_reviewer_incentive_20260806`
|
|
1576
|
+
|
|
1435
1577
|
**Key insight**: Design reviews and implementation reviews find
|
|
1436
1578
|
**categorically different bugs**. Both phases are necessary.
|
|
@@ -259,12 +259,29 @@ begin
|
|
|
259
259
|
# and on between-reviewer progress (counter advances when result arrives).
|
|
260
260
|
# v0.3.0 PR3 pushes MainState ticks via a per-result hook below.
|
|
261
261
|
|
|
262
|
+
# Per-seat persistence: each reply is written the moment it arrives, so a
|
|
263
|
+
# worker death with one seat stuck leaves the finished seats recoverable
|
|
264
|
+
# (collect's crash/timeout recovery reads partial_results.json). Runs on
|
|
265
|
+
# the dispatch collecting thread — the worker stays this file's single
|
|
266
|
+
# writer (§6.3). subprocess_results.json still supersedes it on clean exit.
|
|
267
|
+
partial_by_index = {}
|
|
268
|
+
on_result = lambda do |idx, result|
|
|
269
|
+
partial_by_index[idx.to_s] = MLR::ReviewSerializer.serialize(result)
|
|
270
|
+
PS.write_partial_results(token, {
|
|
271
|
+
'schema_version' => 1,
|
|
272
|
+
'token' => token,
|
|
273
|
+
'updated_at' => Time.now.iso8601,
|
|
274
|
+
'results_by_index' => partial_by_index
|
|
275
|
+
})
|
|
276
|
+
end
|
|
277
|
+
|
|
262
278
|
results = dispatcher.dispatch(
|
|
263
279
|
(request['reviewers'] || []).map { |r| r.transform_keys(&:to_sym) },
|
|
264
280
|
request['messages'] || [],
|
|
265
281
|
request['system_prompt'] || '',
|
|
266
282
|
context: nil,
|
|
267
|
-
review_context: request['review_context'] || 'independent'
|
|
283
|
+
review_context: request['review_context'] || 'independent',
|
|
284
|
+
on_result: on_result
|
|
268
285
|
)
|
|
269
286
|
|
|
270
287
|
# v3.24.3: counter-only signal (no enter_call!/exit_call! pair). bump_counter!
|
|
@@ -287,6 +304,15 @@ begin
|
|
|
287
304
|
}
|
|
288
305
|
}
|
|
289
306
|
PS.write_subprocess_results(token, payload)
|
|
307
|
+
# The partial file is superseded the moment the full results land; left
|
|
308
|
+
# behind, it is a stale copy of reviewer replies sitting beside the record
|
|
309
|
+
# forever (completed token dirs are never reaped). Removed on the clean
|
|
310
|
+
# path only — on a crash it IS the record collect recovers from.
|
|
311
|
+
begin
|
|
312
|
+
File.unlink(PS.partial_results_path(token))
|
|
313
|
+
rescue Errno::ENOENT
|
|
314
|
+
nil
|
|
315
|
+
end
|
|
290
316
|
PS.transition_to_terminal!(token, 'done')
|
|
291
317
|
exit 0
|
|
292
318
|
rescue StandardError => e
|
|
@@ -601,6 +601,15 @@ module KairosMcp
|
|
|
601
601
|
end
|
|
602
602
|
end
|
|
603
603
|
|
|
604
|
+
# The consequence clause a finding carries, when it carries one. The
|
|
605
|
+
# severity axis says what KIND of defect this is; the consequence
|
|
606
|
+
# clause is the WEIGHT axis — who is harmed, and how, if it is never
|
|
607
|
+
# fixed. Measured 2026-08-06 (project_orientation_report checker,
|
|
608
|
+
# R5): of 7 P0s, 3 were factually correct findings that cost nobody
|
|
609
|
+
# anything, and the two kinds landed at the same severity because the
|
|
610
|
+
# record had nowhere to say the difference.
|
|
611
|
+
CONSEQUENCE_RE = /\[\s*consequence:\s*([^\]]*)\]/i
|
|
612
|
+
|
|
604
613
|
# Collect severity-tagged findings from all successful reviews.
|
|
605
614
|
# Deduplicates by first 80 chars (case-insensitive).
|
|
606
615
|
def self.aggregate_findings(parsed_verdicts)
|
|
@@ -611,23 +620,7 @@ module KairosMcp
|
|
|
611
620
|
|
|
612
621
|
# Extract "P0: ...", "P1-1: ...", "**P0**:", etc.
|
|
613
622
|
text.scan(/\*{0,2}(P[0-3])\*{0,2}[-\s]*\d*[.:]\s*(.+?)(?=\n\s*\n|\n\s*\*{0,2}P[0-3]|\z)/mi) do |sev, issue|
|
|
614
|
-
all_findings <<
|
|
615
|
-
severity: sev.upcase,
|
|
616
|
-
# This used to be `[0..200]`, an inclusive Range, so every
|
|
617
|
-
# finding longer than 201 characters lost its tail — the quoted
|
|
618
|
-
# line, the file:line, the failure condition — before anything
|
|
619
|
-
# downstream could bound it on purpose.
|
|
620
|
-
#
|
|
621
|
-
# It is bounded here rather than nowhere, and here rather than
|
|
622
|
-
# further upstream, because this is the point where both costs
|
|
623
|
-
# are decided at once: what gets stored, and what the sanitizer
|
|
624
|
-
# is about to normalise character by character. A bound placed
|
|
625
|
-
# on the reply instead was measured not to work, since NFKC runs
|
|
626
|
-
# between the two and expands by up to 11x. The bound is in
|
|
627
|
-
# bytes for the same reason — bytes are what is spent.
|
|
628
|
-
issue: Sanitizer.clamp_finding_bytes(issue.strip),
|
|
629
|
-
cited_by: [r[:role_label]]
|
|
630
|
-
}
|
|
623
|
+
all_findings << build_finding(sev, issue, r[:role_label])
|
|
631
624
|
end
|
|
632
625
|
end
|
|
633
626
|
|
|
@@ -659,7 +652,7 @@ module KairosMcp
|
|
|
659
652
|
# (`issue_variants_omitted`), because a silently shortened list
|
|
660
653
|
# reads as "this is all there was" — the failure this whole change
|
|
661
654
|
# exists to remove.
|
|
662
|
-
grouped = all_findings.group_by { |f| f[:issue]
|
|
655
|
+
grouped = all_findings.group_by { |f| dedup_key(f[:issue]) }
|
|
663
656
|
grouped.map do |_key, findings|
|
|
664
657
|
merged_severity = findings.map { |f| f[:severity] }.min # P0 < P1 < P2
|
|
665
658
|
# `.min` over a non-empty group returns one of its own members, so
|
|
@@ -672,6 +665,25 @@ module KairosMcp
|
|
|
672
665
|
issue: representative[:issue],
|
|
673
666
|
cited_by: findings.flat_map { |f| f[:cited_by] }.uniq
|
|
674
667
|
}
|
|
668
|
+
# The representative's wording and its weight travel together, like
|
|
669
|
+
# its severity does. When the representative states no consequence,
|
|
670
|
+
# another member's is carried rather than none: a group where ONE
|
|
671
|
+
# reviewer said who is harmed is a finding whose harm is known.
|
|
672
|
+
consequence = representative[:consequence] ||
|
|
673
|
+
findings.map { |f| f[:consequence] }.compact.first
|
|
674
|
+
row[:consequence] = consequence if consequence
|
|
675
|
+
# The demotion mark is carried from ANY member, not only the
|
|
676
|
+
# representative — copied from the representative alone it
|
|
677
|
+
# vanished order-dependently whenever a non-demoted member of the
|
|
678
|
+
# same severity happened to sort first (R1 finding, three seats
|
|
679
|
+
# independently). A row that merged to P0 carries no mark: some
|
|
680
|
+
# member stated the harm, the row holds its full severity, and
|
|
681
|
+
# nothing was demoted AWAY from what the record shows.
|
|
682
|
+
if row[:severity] != 'P0' &&
|
|
683
|
+
(demoted = findings.find { |f| f[:severity_stated] })
|
|
684
|
+
row[:severity_stated] = demoted[:severity_stated]
|
|
685
|
+
row[:severity_demoted] = demoted[:severity_demoted]
|
|
686
|
+
end
|
|
675
687
|
if variants.size > 1
|
|
676
688
|
row[:issue_variants] = variants.first(MAX_ISSUE_VARIANTS)
|
|
677
689
|
omitted = variants.size - MAX_ISSUE_VARIANTS
|
|
@@ -680,6 +692,68 @@ module KairosMcp
|
|
|
680
692
|
row
|
|
681
693
|
end.sort_by { |f| f[:severity] }
|
|
682
694
|
end
|
|
695
|
+
|
|
696
|
+
# The dedup key is the issue WITHOUT its consequence clause. The clause
|
|
697
|
+
# stays in the issue text (display, replay), but two reviewers naming
|
|
698
|
+
# one defect — one saying who is harmed, one not — are still one
|
|
699
|
+
# finding, and the merge is what lets the stated consequence carry the
|
|
700
|
+
# row's severity for both. Keyed on the raw text instead, the clause
|
|
701
|
+
# lands inside the first 80 characters of any short issue and splits
|
|
702
|
+
# the group, moving the finding count and the convergence denominator.
|
|
703
|
+
# Whitespace is normalised for the same reason: removing the clause
|
|
704
|
+
# must not leave a gap that fails the comparison it was removed for.
|
|
705
|
+
def self.dedup_key(issue)
|
|
706
|
+
issue.gsub(CONSEQUENCE_RE, ' ').gsub(/\s+/, ' ').strip[0..79].downcase
|
|
707
|
+
end
|
|
708
|
+
|
|
709
|
+
# One extracted finding, weight axis applied at the point of entry.
|
|
710
|
+
#
|
|
711
|
+
# A P0 that does not say who is harmed is recorded at P2, with the
|
|
712
|
+
# stated severity and the demotion reason kept beside it — the record
|
|
713
|
+
# says what the reviewer wrote AND what the rule did with it. Only
|
|
714
|
+
# PRESENCE is checked, mechanically, by construction: whether a stated
|
|
715
|
+
# consequence is real or trivial is a judgment call, and it belongs to
|
|
716
|
+
# the orchestrator reading the record, not to a heuristic here — the
|
|
717
|
+
# same division of labour as the substance rule above, which asks
|
|
718
|
+
# "said anything?" and never "said anything good?".
|
|
719
|
+
#
|
|
720
|
+
# The clause is copied into its own field but NOT stripped from the
|
|
721
|
+
# issue text: the display shows the issue line, a later round's prompt
|
|
722
|
+
# replays it, and the consequence should survive in both. (The dedup
|
|
723
|
+
# key, above, strips the clause before comparing — the one place the
|
|
724
|
+
# issue is read WITHOUT it.)
|
|
725
|
+
def self.build_finding(sev, issue, role_label)
|
|
726
|
+
raw = issue.strip
|
|
727
|
+
# The clause is read from the UNCLAMPED text, before the byte bound
|
|
728
|
+
# cuts the tail — a finding long enough to lose its closing bracket
|
|
729
|
+
# to the clamp would otherwise read as having stated nothing, and
|
|
730
|
+
# the demotion would bury exactly the P0 whose author complied (R1
|
|
731
|
+
# finding, three seats). Scan rather than match, first NON-empty
|
|
732
|
+
# clause: "[consequence: ] ... [consequence: real harm]" states a
|
|
733
|
+
# harm, and reading only the first bracket pair would demote past it.
|
|
734
|
+
consequence = raw.scan(CONSEQUENCE_RE)
|
|
735
|
+
.map { |c| c[0].strip }
|
|
736
|
+
.find { |c| !c.empty? }
|
|
737
|
+
# Bounded in BYTES at the point of extraction — this is the clamp
|
|
738
|
+
# that bounds the sanitizer's input, and the reasoning for bytes
|
|
739
|
+
# (NFKC expands up to 11x downstream) is at clamp_finding_bytes.
|
|
740
|
+
# This used to be `[0..200]`, an inclusive Range, so every finding
|
|
741
|
+
# longer than 201 characters lost its tail before anything
|
|
742
|
+
# downstream could bound it on purpose.
|
|
743
|
+
finding = {
|
|
744
|
+
severity: sev.upcase,
|
|
745
|
+
issue: Sanitizer.clamp_finding_bytes(raw),
|
|
746
|
+
cited_by: [role_label]
|
|
747
|
+
}
|
|
748
|
+
if consequence
|
|
749
|
+
finding[:consequence] = Sanitizer.clamp_finding_bytes(consequence)
|
|
750
|
+
elsif finding[:severity] == 'P0'
|
|
751
|
+
finding[:severity] = 'P2'
|
|
752
|
+
finding[:severity_stated] = 'P0'
|
|
753
|
+
finding[:severity_demoted] = 'consequence_missing'
|
|
754
|
+
end
|
|
755
|
+
finding
|
|
756
|
+
end
|
|
683
757
|
end
|
|
684
758
|
end
|
|
685
759
|
end
|
|
@@ -33,9 +33,17 @@ module KairosMcp
|
|
|
33
33
|
# @param system_prompt [String] system prompt for llm_call
|
|
34
34
|
# @param context [InvocationContext] for invoke_tool
|
|
35
35
|
# @param review_context [String] 'independent' or 'project_aware'
|
|
36
|
+
# @param on_result [#call, nil] called with (index, result) for every
|
|
37
|
+
# seat outcome this dispatch decides — arrived replies as they
|
|
38
|
+
# arrive, and the dispatch_timeout skips synthesized at the
|
|
39
|
+
# deadline. The caller uses this to persist each seat's TRUE
|
|
40
|
+
# outcome, so a later crash recovery reads "this seat timed out"
|
|
41
|
+
# rather than relabelling a reached seat as lost (R1 finding). Runs
|
|
42
|
+
# on the collecting thread; a hook failure is logged and never
|
|
43
|
+
# fails the dispatch.
|
|
36
44
|
# @return [Array<Hash>] results indexed by reviewer position
|
|
37
45
|
def dispatch(reviewers, messages, system_prompt, context:,
|
|
38
|
-
review_context: 'independent')
|
|
46
|
+
review_context: 'independent', on_result: nil)
|
|
39
47
|
dispatch_id = SecureRandom.hex(8)
|
|
40
48
|
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @timeout
|
|
41
49
|
results = Array.new(reviewers.size)
|
|
@@ -94,11 +102,17 @@ module KairosMcp
|
|
|
94
102
|
i, result = entry
|
|
95
103
|
results[i] = result
|
|
96
104
|
collected += 1
|
|
105
|
+
notify_result(on_result, i, result)
|
|
97
106
|
end
|
|
98
|
-
# Mark uncollected as timed out
|
|
107
|
+
# Mark uncollected as timed out. Notified like an arrived
|
|
108
|
+
# reply: the skip IS this dispatch's decision about the seat,
|
|
109
|
+
# and a persisted record that omits it lets a crash recovery
|
|
110
|
+
# relabel a reached-and-timed-out seat as one the worker never
|
|
111
|
+
# got to.
|
|
99
112
|
reviewers.each_with_index do |r, i|
|
|
100
113
|
next if results[i]
|
|
101
114
|
results[i] = build_skip(r, 'dispatch_timeout')
|
|
115
|
+
notify_result(on_result, i, results[i])
|
|
102
116
|
end
|
|
103
117
|
break
|
|
104
118
|
end
|
|
@@ -111,6 +125,7 @@ module KairosMcp
|
|
|
111
125
|
i, result = entry
|
|
112
126
|
results[i] = result
|
|
113
127
|
collected += 1
|
|
128
|
+
notify_result(on_result, i, result)
|
|
114
129
|
end
|
|
115
130
|
|
|
116
131
|
# Kill in-flight subprocesses from this dispatch
|
|
@@ -136,6 +151,15 @@ module KairosMcp
|
|
|
136
151
|
|
|
137
152
|
private
|
|
138
153
|
|
|
154
|
+
# A hook failure must not fail the dispatch: the hook exists to save
|
|
155
|
+
# replies from a dying worker, and a hook that could kill the dispatch
|
|
156
|
+
# would create the loss it guards against.
|
|
157
|
+
def notify_result(on_result, idx, result)
|
|
158
|
+
on_result&.call(idx, result)
|
|
159
|
+
rescue StandardError => e
|
|
160
|
+
warn "[multi_llm_review::Dispatcher] on_result hook failed: #{e.class}: #{e.message}"
|
|
161
|
+
end
|
|
162
|
+
|
|
139
163
|
def bump_main_state_counter
|
|
140
164
|
return unless defined?(KairosMcp::SkillSets::MultiLlmReview::MainState)
|
|
141
165
|
# v3.24.3: counter-only bump. exit_call! is private; bump_counter!
|
|
@@ -72,6 +72,12 @@ module KairosMcp
|
|
|
72
72
|
def gc_eligible_path(token); File.join(token_dir(token), 'gc.eligible'); end
|
|
73
73
|
def request_path(token); File.join(token_dir(token), 'request.json'); end
|
|
74
74
|
def subprocess_results_path(token); File.join(token_dir(token), 'subprocess_results.json'); end
|
|
75
|
+
# Per-seat results written as each seat completes, so a worker that
|
|
76
|
+
# dies with one seat stuck does not take the finished seats' replies
|
|
77
|
+
# with it (R3 2026-08-06 lost three completed external seats to one
|
|
78
|
+
# stale heartbeat). Superseded by subprocess_results.json on a clean
|
|
79
|
+
# exit; read by collect's crash/timeout recovery path only.
|
|
80
|
+
def partial_results_path(token); File.join(token_dir(token), 'partial_results.json'); end
|
|
75
81
|
def worker_pid_path(token); File.join(token_dir(token), 'worker.pid'); end
|
|
76
82
|
def worker_heartbeat_path(token); File.join(token_dir(token), 'worker.heartbeat'); end
|
|
77
83
|
def worker_tick_path(token); File.join(token_dir(token), 'worker.tick'); end
|
|
@@ -105,6 +111,7 @@ module KairosMcp
|
|
|
105
111
|
def write_collected(token, data); atomic_write_json(collected_path(token), data); end
|
|
106
112
|
def write_request(token, data); atomic_write_json(request_path(token), data); end
|
|
107
113
|
def write_subprocess_results(token, data); atomic_write_json(subprocess_results_path(token), data); end
|
|
114
|
+
def write_partial_results(token, data); atomic_write_json(partial_results_path(token), data); end
|
|
108
115
|
def write_worker_pid(token, data); atomic_write_json(worker_pid_path(token), data); end
|
|
109
116
|
def write_marker(token, data); atomic_write_json(marker_path(token), data); end
|
|
110
117
|
def write_completed(token, data); atomic_write_json(completed_path(token), data); end
|
|
@@ -172,6 +179,7 @@ module KairosMcp
|
|
|
172
179
|
def load_collected(token); load_json_transient(collected_path(token)); end
|
|
173
180
|
def load_request(token); load_json_transient(request_path(token)); end
|
|
174
181
|
def load_subprocess_results(token); load_json_transient(subprocess_results_path(token)); end
|
|
182
|
+
def load_partial_results(token); load_json_transient(partial_results_path(token)); end
|
|
175
183
|
def load_worker_pid(token); load_json_transient(worker_pid_path(token)); end
|
|
176
184
|
|
|
177
185
|
# Mutate state.json under a read-modify-write block, serialized
|
|
@@ -77,7 +77,13 @@ module KairosMcp
|
|
|
77
77
|
parts << "<task>"
|
|
78
78
|
parts << "Review the provided artifact for #{review_type} correctness."
|
|
79
79
|
parts << "Target: #{artifact_name}"
|
|
80
|
-
|
|
80
|
+
# The round number is deliberately NOT given to the reviewer
|
|
81
|
+
# (2026-08-06). Telling a reviewer which round it is in — like
|
|
82
|
+
# telling it its finding count is compared across rounds — turns the
|
|
83
|
+
# count into something the reviewer performs, and selects for
|
|
84
|
+
# finding-production over finding-weight. Convergence is measured by
|
|
85
|
+
# the orchestrator from the record; the reviewer needs the artifact,
|
|
86
|
+
# the criteria, and the prior findings to verify, nothing else.
|
|
81
87
|
if review_round > 1 && prior_findings && !prior_findings.empty?
|
|
82
88
|
parts << "Scope: Review the revisions addressing prior findings."
|
|
83
89
|
parts << ""
|
|
@@ -161,14 +167,22 @@ module KairosMcp
|
|
|
161
167
|
line one is read.
|
|
162
168
|
|
|
163
169
|
For each finding, use this single-line format (one finding per line):
|
|
164
|
-
P0: <issue description> [location: file:line]
|
|
165
|
-
P1: <issue description> [location: file:line]
|
|
170
|
+
P0: <issue description> [consequence: <who is harmed, and how, if this is never fixed>] [location: file:line]
|
|
171
|
+
P1: <issue description> [consequence: ...] [location: file:line]
|
|
166
172
|
P2: <issue description> [location: file:line]
|
|
167
173
|
P3: <issue description> [location: file:line]
|
|
168
174
|
|
|
175
|
+
The consequence clause is REQUIRED for P0 and recommended for
|
|
176
|
+
P1; P1 findings are never demoted for lacking one. A finding can be
|
|
177
|
+
factually correct and still cost nobody anything; the consequence
|
|
178
|
+
clause is where you say who hits the defect and what happens to
|
|
179
|
+
them. A P0 with no consequence clause, or an empty one, is
|
|
180
|
+
recorded at P2. Do not restate the issue as its own consequence —
|
|
181
|
+
name the person or process that is harmed.
|
|
182
|
+
|
|
169
183
|
Example:
|
|
170
|
-
P0: Missing input validation in dispatcher timeout path [location: dispatcher.rb:120]
|
|
171
|
-
P1: Thread safety issue with shared counter [location: consensus.rb:45]
|
|
184
|
+
P0: Missing input validation in dispatcher timeout path [consequence: a caller passing a negative timeout crashes the worker and the whole round's reviews are lost] [location: dispatcher.rb:120]
|
|
185
|
+
P1: Thread safety issue with shared counter [consequence: concurrent collects double-count usage] [location: consensus.rb:45]
|
|
172
186
|
|
|
173
187
|
If no issues found, state "No findings" and verdict APPROVE.
|
|
174
188
|
</structured_output_contract>
|
|
@@ -157,7 +157,8 @@ module KairosMcp
|
|
|
157
157
|
# reaches the payload by the same path.
|
|
158
158
|
#
|
|
159
159
|
# @param findings [Array<Hash>] finding rows with String keys
|
|
160
|
-
# @return [Array<Hash>] the same rows, 'issue'
|
|
160
|
+
# @return [Array<Hash>] the same rows, 'issue', 'consequence' and
|
|
161
|
+
# 'issue_variants' bound
|
|
161
162
|
def self.bound_findings_for_record(findings)
|
|
162
163
|
findings.map do |f|
|
|
163
164
|
row = f.merge(
|
|
@@ -165,6 +166,15 @@ module KairosMcp
|
|
|
165
166
|
sanitize_finding_text(f['issue'], max_len: FINDING_RECORD_MAX_LEN)
|
|
166
167
|
)
|
|
167
168
|
)
|
|
169
|
+
if row['consequence']
|
|
170
|
+
# Reviewer text like the issue it was extracted from; it reaches
|
|
171
|
+
# the record by the same path.
|
|
172
|
+
row = row.merge(
|
|
173
|
+
'consequence' => clamp_finding_bytes(
|
|
174
|
+
sanitize_finding_text(f['consequence'], max_len: FINDING_RECORD_MAX_LEN)
|
|
175
|
+
)
|
|
176
|
+
)
|
|
177
|
+
end
|
|
168
178
|
if row['issue_variants']
|
|
169
179
|
row = row.merge(
|
|
170
180
|
'issue_variants' => Array(row['issue_variants']).map do |v|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "multi_llm_review",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "v0.9.1 (seat access, frozen 2026-08-06 after a one-round review): reviewer prompts carry a <seat_access> block beside the inline artifact — a seat that cannot read the repository must not attempt tool calls and must not open by saying it will read files; it reviews the artifact text alone, marks unverifiable claims [INFERRED], and still opens with its verdict line. The wording is conditional because seats differ (codex runs --sandbox read-only and can read the repository), and the block is not emitted for by_reference delivery, whose existing cannot-read instruction it would contradict. Root cause fixed: the claude subprocess seat runs with tools disabled in an empty working directory, and on implementation artifacts citing file paths it opened with pseudo-tool-call markup or a cannot-access preamble instead of its verdict line, leaving four consecutive rounds as no_verdict while counting every round on design artifacts. v0.9.0 (evidence fidelity, frozen 2026-08-06 after six review rounds): a finding reaches the record whole. It used to be cut at 201 bytes by an inclusive Range in aggregation and bounded again by the 500-character display limit, so a downstream instance measured 18 of 21 findings arriving at exactly 201 bytes with reviews[].raw_text empty on every row. Findings are now bounded in BYTES at FINDING_RECORD_MAX_LEN = 8000 for the record, with DEFAULT_MAX_LEN = 500 still applied by every path that takes a finding into a prompt. Deduplication still keys on the first 80 characters — widening it would stop two reviewers describing one defect from merging, moving the finding count and the convergence denominator — but the surviving text is no longer arbitrary: issue comes from a member whose severity equals the merged severity, distinct texts survive in issue_variants (capped at MAX_ISSUE_VARIANTS = 8, with issue_variants_omitted naming what the cap dropped). Every row now carries raw_text_excerpt unconditionally (4096 bytes) and the reviewer's reply in raw_text on request (include_raw_text, 65536 bytes). Both are SANITISED TRANSCRIPTIONS, not verbatim records, and both tool schemas say so: the text is byte-clamped, NFKC-normalised, stripped of invisible characters, tag-escaped, then byte-clamped again. No field states whether they hold the whole reply, because nothing in this SkillSet can know — a completeness flag was implemented, measured wrong in both directions, and removed. On delegated runs the pending-state record keeps each subprocess reply as it arrived; on single-phase runs the returned payload is the only form there is. Findings carried into a later round's prompt are sanitised and folded to one line, a path that previously took reviewer text into a prompt with no sanitisation at all. v0.8.0 (v0.7 record schema, design frozen 2026-08-01): the verdict vocabulary is the three canonical words plus tense forms only (INV-R1); the ratio and threshold are recorded reference values, not the run's conclusion — the top-level verdict field became reference_verdict and the run is closed by the operator's declaration outside the record (INV-R2); the persona team occupies one seat, its derivation rule is recorded, and a submission smaller than convened — including empty — is accepted with the shortfall on the record (INV-R3/R4); every run writes an existence marker at dispatch, completed records are never garbage-collected, and expired runs are reduced to a minimal trace instead of erased (INV-R4); a divergence-excluded tally is carried beside the main one (INV-R5); the record names its pre-declared spec and carries transport diagnostics as state tags (INV-R6); artifact delivery is a per-seat attribute (inline | by_reference) and an unreachable delivery is refused rather than dispatched (INV-R7). Parallel multi-LLM review orchestration. Dispatches review prompts to N LLM backends via llm_client, collects verdicts, and computes consensus. v0.6.0: reserve observers (escalate) and a declarable persona execution model; the observer set is built in one pass with explicit precedence (ObserverSet); every slot must name its model and role_label, and duplicate names — including the persona team's own — are refused. A reply's verdict is no longer inferred from its prose: it is read from a declared field, from the header the reply opens with when that header carries a verdict name and nothing else, or not at all, in which case the reply leaves the denominator with no_verdict recorded beside its name. The record says why every observer did or did not count (denominator_composition, five skip_reason values, observers_reporting), and the per-reviewer row is written by one mapping rather than two. v0.5.2: the cursor reviewer pins model composer-2.5 instead of inheriting the cursor CLI default, which is operator-editable and had silently become an Anthropic model. v0.5.1: Fable 5 retired from the roster (five consecutive silent returns), convergence 3/5; orchestrator_model description now states the bare-ID rule so a caller does not review its own output. v0.5.0: adds multi_llm_review_wait (Phase 1.5) for explicit subprocess completion gating with next_action recovery hints, and Path A/B doc disambiguation. v0.4.0 (Phase 12): feedback_text + schema_version, sanitization contract for prompt-injection defense, and multi_llm_review_bundle tool for human-handoff paths without dispatch.",
|
|
3
|
+
"version": "0.10.0",
|
|
4
|
+
"description": "v0.10.0 (finding weight axis + worker-death recovery, 2026-08-06): findings gain a consequence axis beside the severity axis — the prompt contract requires a [consequence: who is harmed, and how, if this is never fixed] clause on every P0, and aggregation records a P0 without one at P2, keeping the stated severity and the demotion reason on the row (severity_stated / severity_demoted: consequence_missing). Only PRESENCE is checked, mechanically; whether a stated consequence is real or trivial stays the orchestrator's call. The dedup key strips the clause so two reviewers naming one defect still merge, and a group where one member states the harm carries it for the row. Motivating measurement (project_orientation_report checker, R5): 3 of 7 P0s were factually correct findings that cost nobody anything. Reviewer prompts also stop naming the round number — telling a reviewer which round it is in (like telling it its counts are compared) selects for finding-production over finding-weight; the L1 workflow v3.10.0 Reviewer incentive rule states the orchestrator-side half. And a worker death no longer discards completed seats: the worker persists each reply as it arrives (partial_results.json), and collect's crash/timeout branches recover the finished seats, entering every unreached seat in the denominator as a skip row (worker_crashed_seat_lost) with the death named in payload.worker_failure — R3 2026-08-06 lost three completed external seats to one stale heartbeat because the only exit was total loss. v0.9.1 (seat access, frozen 2026-08-06 after a one-round review): reviewer prompts carry a <seat_access> block beside the inline artifact — a seat that cannot read the repository must not attempt tool calls and must not open by saying it will read files; it reviews the artifact text alone, marks unverifiable claims [INFERRED], and still opens with its verdict line. The wording is conditional because seats differ (codex runs --sandbox read-only and can read the repository), and the block is not emitted for by_reference delivery, whose existing cannot-read instruction it would contradict. Root cause fixed: the claude subprocess seat runs with tools disabled in an empty working directory, and on implementation artifacts citing file paths it opened with pseudo-tool-call markup or a cannot-access preamble instead of its verdict line, leaving four consecutive rounds as no_verdict while counting every round on design artifacts. v0.9.0 (evidence fidelity, frozen 2026-08-06 after six review rounds): a finding reaches the record whole. It used to be cut at 201 bytes by an inclusive Range in aggregation and bounded again by the 500-character display limit, so a downstream instance measured 18 of 21 findings arriving at exactly 201 bytes with reviews[].raw_text empty on every row. Findings are now bounded in BYTES at FINDING_RECORD_MAX_LEN = 8000 for the record, with DEFAULT_MAX_LEN = 500 still applied by every path that takes a finding into a prompt. Deduplication still keys on the first 80 characters — widening it would stop two reviewers describing one defect from merging, moving the finding count and the convergence denominator — but the surviving text is no longer arbitrary: issue comes from a member whose severity equals the merged severity, distinct texts survive in issue_variants (capped at MAX_ISSUE_VARIANTS = 8, with issue_variants_omitted naming what the cap dropped). Every row now carries raw_text_excerpt unconditionally (4096 bytes) and the reviewer's reply in raw_text on request (include_raw_text, 65536 bytes). Both are SANITISED TRANSCRIPTIONS, not verbatim records, and both tool schemas say so: the text is byte-clamped, NFKC-normalised, stripped of invisible characters, tag-escaped, then byte-clamped again. No field states whether they hold the whole reply, because nothing in this SkillSet can know — a completeness flag was implemented, measured wrong in both directions, and removed. On delegated runs the pending-state record keeps each subprocess reply as it arrived; on single-phase runs the returned payload is the only form there is. Findings carried into a later round's prompt are sanitised and folded to one line, a path that previously took reviewer text into a prompt with no sanitisation at all. v0.8.0 (v0.7 record schema, design frozen 2026-08-01): the verdict vocabulary is the three canonical words plus tense forms only (INV-R1); the ratio and threshold are recorded reference values, not the run's conclusion — the top-level verdict field became reference_verdict and the run is closed by the operator's declaration outside the record (INV-R2); the persona team occupies one seat, its derivation rule is recorded, and a submission smaller than convened — including empty — is accepted with the shortfall on the record (INV-R3/R4); every run writes an existence marker at dispatch, completed records are never garbage-collected, and expired runs are reduced to a minimal trace instead of erased (INV-R4); a divergence-excluded tally is carried beside the main one (INV-R5); the record names its pre-declared spec and carries transport diagnostics as state tags (INV-R6); artifact delivery is a per-seat attribute (inline | by_reference) and an unreachable delivery is refused rather than dispatched (INV-R7). Parallel multi-LLM review orchestration. Dispatches review prompts to N LLM backends via llm_client, collects verdicts, and computes consensus. v0.6.0: reserve observers (escalate) and a declarable persona execution model; the observer set is built in one pass with explicit precedence (ObserverSet); every slot must name its model and role_label, and duplicate names — including the persona team's own — are refused. A reply's verdict is no longer inferred from its prose: it is read from a declared field, from the header the reply opens with when that header carries a verdict name and nothing else, or not at all, in which case the reply leaves the denominator with no_verdict recorded beside its name. The record says why every observer did or did not count (denominator_composition, five skip_reason values, observers_reporting), and the per-reviewer row is written by one mapping rather than two. v0.5.2: the cursor reviewer pins model composer-2.5 instead of inheriting the cursor CLI default, which is operator-editable and had silently become an Anthropic model. v0.5.1: Fable 5 retired from the roster (five consecutive silent returns), convergence 3/5; orchestrator_model description now states the bare-ID rule so a caller does not review its own output. v0.5.0: adds multi_llm_review_wait (Phase 1.5) for explicit subprocess completion gating with next_action recovery hints, and Path A/B doc disambiguation. v0.4.0 (Phase 12): feedback_text + schema_version, sanitization contract for prompt-injection defense, and multi_llm_review_bundle tool for human-handoff paths without dispatch.",
|
|
5
5
|
"author": "Masaomi Hatakeyama",
|
|
6
6
|
"layer": "L1",
|
|
7
7
|
"depends_on": [
|