kairos-chain 3.74.0 → 3.76.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +150 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +90 -5
- data/templates/skillsets/kairos_hook_projector/hooks/readable_gate.rb +219 -20
- data/templates/skillsets/kairos_hook_projector/test/mutation_check_readable_gate.rb +169 -0
- data/templates/skillsets/kairos_hook_projector/test/test_readable_gate.rb +426 -3
- data/templates/skillsets/minimum_nomic/README.md +115 -0
- data/templates/skillsets/minimum_nomic/bin/criterion_matrix.rb +221 -0
- data/templates/skillsets/minimum_nomic/bin/distil_criterion.rb +221 -0
- data/templates/skillsets/minimum_nomic/bin/judge_change.rb +238 -0
- data/templates/skillsets/minimum_nomic/bin/mutate_rule.rb +179 -0
- data/templates/skillsets/minimum_nomic/bin/propose_metric.rb +426 -0
- data/templates/skillsets/minimum_nomic/skillset.json +2 -2
- data/templates/skillsets/minimum_nomic/test/test_propose_metric.rb +161 -0
- metadata +8 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 83162bcb26c033c10413fe1f25ffed382031dfebc115aa7041438d807177ef8c
|
|
4
|
+
data.tar.gz: eacc1ed007ad60d8429954c1d0e0bca228048a5a0438f177df892e72aae3e599
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e6e311e80bc090726582d7032aa87a0ac1a97ffea387b300306d609f19b0a38c4dbe2cd6a84dd36e924e66c3e9f579b56f91c1c25602cc2afdae21e8f8e2005c
|
|
7
|
+
data.tar.gz: 0ffddbff73ac05f8804b343e597495517b3872a0ceed500ef1370aab8f034a866b20585a5a03c46db179fbacfd18aff3862a70fc40c093f66389a69f2afdc4c2
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,156 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.76.0] - 2026-08-22
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- **L1 `multi_llm_review_workflow` 3.10.2 → 3.11.0 — three norms and one
|
|
12
|
+
recorded open question, for three failures the operator named and one request
|
|
13
|
+
he made.** The failures: an orchestrator issuing different criteria each
|
|
14
|
+
round, the APPROVE ratio operating as the close condition, and design reviews
|
|
15
|
+
descending into code. The request: bound mutation experiments. Four rules were
|
|
16
|
+
drafted, reviewed over three rounds, applied, reviewed once more as applied
|
|
17
|
+
text — one APPROVE of four seats, eighteen blocking findings — and revised
|
|
18
|
+
subtractively. What ships is narrower than what was drafted, which is the
|
|
19
|
+
point: this entry's own § Revision Discipline records that deletions drew zero
|
|
20
|
+
new P0s in every measured case and additions became the next round's
|
|
21
|
+
battleground.
|
|
22
|
+
|
|
23
|
+
**§ Prompt Generation Rules gains a recording duty on the round's
|
|
24
|
+
instrument** — what a seat is told to look for, how much of the artifact it is
|
|
25
|
+
given, and which seats answer. It bounds nothing: narrowing criteria, cutting
|
|
26
|
+
scope and convening a different panel are each a documented remedy for
|
|
27
|
+
something. It forbids moving the instrument and then reading the resulting
|
|
28
|
+
change in finding counts as a property of the artifact. It was drafted around
|
|
29
|
+
criteria alone, which left scope and panel composition — two channels that
|
|
30
|
+
move the count just as well — outside it.
|
|
31
|
+
|
|
32
|
+
**§ Review Types gains the phase rule as an invariant with no branches**: a
|
|
33
|
+
review answers at the phase of its target and does not descend. Design and
|
|
34
|
+
Document reviews read code only to refute a factual claim and write none;
|
|
35
|
+
implementation review is where fixes are written. Two enumerated sentences
|
|
36
|
+
drafted with it are not here. "An implementation-phase finding that reopens
|
|
37
|
+
the design belongs to the backlog" collided with this entry's own "any (a) or
|
|
38
|
+
(b) REJECT or FAIL = revise and re-review" over exactly the
|
|
39
|
+
design-implementation seam it calls the most valuable layer, and pre-answered
|
|
40
|
+
the operator question § Revision Discipline rule 1 requires. "Fix plan and
|
|
41
|
+
Final/convergence reviews inherit the phase of the artifact they check" gave
|
|
42
|
+
those two nothing to inherit, no phase being assigned to a fix plan anywhere.
|
|
43
|
+
The `<structured_output_contract>` line drops "consider improving" without
|
|
44
|
+
enumerating review types, because the enumerated form left this entry's own
|
|
45
|
+
knowledge/documentation-update review unassigned.
|
|
46
|
+
|
|
47
|
+
**The mutation norm is one sentence inside the phase rule and assigns no
|
|
48
|
+
party**: a green suite is not evidence until its tests have been shown to
|
|
49
|
+
fail, and a pass whose sites the code's author chose measures the author's
|
|
50
|
+
attention, not the suite. It was drafted as a seventh item in Step -1 carrying
|
|
51
|
+
the duty "whoever chooses the sites is not whoever wrote the code", and no
|
|
52
|
+
party in this entry can discharge that — before dispatch the only actor is the
|
|
53
|
+
orchestrator, which § Roles makes the same context as the implementer, and
|
|
54
|
+
review seats run sandboxed read-only and cannot execute anything. Stated as a
|
|
55
|
+
property of evidence rather than a duty on someone, it holds wherever a
|
|
56
|
+
mutation result is read. The bound the operator asked for is not here and the
|
|
57
|
+
omission is open, not settled: the recorded decision is to accumulate three
|
|
58
|
+
runs and then judge.
|
|
59
|
+
|
|
60
|
+
**Withdrawn in the same version: a rule making each seat label its own
|
|
61
|
+
findings new or carryover.** A seat sees only its own priors, so a finding one
|
|
62
|
+
seat raises after another raised it is new to that seat and carryover to the
|
|
63
|
+
round, and a seat added or returning mid-thread has no priors at all, making
|
|
64
|
+
every finding it raises new by construction — the machine-side signal "new
|
|
65
|
+
(a)+(b) P0 = 0" is then not computable from seat labels. § Convergence Rules
|
|
66
|
+
records that who makes the label is undecided and that the orchestrator makes
|
|
67
|
+
it meanwhile, which is the biased judge the withdrawn rule was written to
|
|
68
|
+
replace. Its paired edits revert with it, which also removes "the round number
|
|
69
|
+
is never written into a prompt" — that sentence contradicted § Output
|
|
70
|
+
Directive's MUST that the prompt carry `review{N}` in the output path, and an
|
|
71
|
+
orchestrator obeying it would have had round N+1 overwrite round N.
|
|
72
|
+
|
|
73
|
+
A fifth proposed rule, an exception for self-referential reviews, was dropped
|
|
74
|
+
before the version was written, after four reviewer contexts — occupying two
|
|
75
|
+
of that round's three counted seats — shot it on three independent grounds.
|
|
76
|
+
|
|
77
|
+
Across all four review rounds the thing most often refuted was not the rules
|
|
78
|
+
but the change's account of itself. The pre-flight falsifier for the applied
|
|
79
|
+
round refuted 8 of 22 checked claims, three of them inside the version entry;
|
|
80
|
+
the correction to those three introduced a fourth false claim, which the round
|
|
81
|
+
then refuted. Every such claim is a record summarised without its denominator
|
|
82
|
+
or its condition.
|
|
83
|
+
|
|
84
|
+
## [3.75.0] - 2026-08-21
|
|
85
|
+
|
|
86
|
+
### Added
|
|
87
|
+
|
|
88
|
+
- **`minimum_nomic` 0.3.0 — the post-game analysis is now a fixed four-stage
|
|
89
|
+
procedure, settled by running it after five rounds of design review failed to
|
|
90
|
+
converge.** Blocking findings across those rounds ran 22, 19, 26, 50, 47, and
|
|
91
|
+
the last two returned no APPROVE from any of the four reviewer seats, so the
|
|
92
|
+
measurement was implemented and observed instead of specified further. The
|
|
93
|
+
stages are: free scoring (`reanalyse.rb`), scoring under the analysts' own
|
|
94
|
+
distilled standards (`distil_criterion.rb` → `criterion_matrix.rb`), measures
|
|
95
|
+
the analysts propose as executable code (`propose_metric.rb`), and
|
|
96
|
+
repeatability plus mutation (`mutate.rb` / `mutate_rule.rb` →
|
|
97
|
+
`judge_change.rb`). No stage supplies a definition of metacognition; every
|
|
98
|
+
definition in play is written by an analyst.
|
|
99
|
+
- **Stage 2 moves into the SkillSet from `log/minimum_nomic_gm_20260810/`,** where
|
|
100
|
+
a ten-game corpus list, a scoring-pass digest and a model-to-adapter table were
|
|
101
|
+
written into the source and made the scripts unusable on any other corpus.
|
|
102
|
+
`distil_criterion.rb` now takes a corpus directory, picks the guideline
|
|
103
|
+
covering the most games and prints the ones it leaves out, and reads each
|
|
104
|
+
model's adapter from the lineups — aborting when one model appears under two
|
|
105
|
+
adapters rather than choosing silently. `criterion_matrix.rb` takes
|
|
106
|
+
`--criteria` and reaches a judge through the adapter recorded when its standard
|
|
107
|
+
was distilled.
|
|
108
|
+
- **`propose_metric.rb`** asks each analyst for a procedure that counts
|
|
109
|
+
metacognition, as code, and runs it once on a disposable copy of the corpus
|
|
110
|
+
under an environment carrying no credentials. It is not repaired, not
|
|
111
|
+
rewritten, not re-run; the ran/failed count is an observation and is never
|
|
112
|
+
compared. A submitted procedure runs with the invoking user's permissions —
|
|
113
|
+
the disposable copy protects the records and nothing protects the host.
|
|
114
|
+
- **`mutate_rule.rb`** rewrites one initial rule body, editing only the rule set
|
|
115
|
+
carried in `lineup.jsonl`, so the utterances still describe conduct under the
|
|
116
|
+
original rule and `calls.jsonl` still holds what players were actually handed.
|
|
117
|
+
The lie sits in exactly one place. It refuses to overwrite an output directory,
|
|
118
|
+
to make more than one substitution, or to leave a scar, and re-reads the arm it
|
|
119
|
+
wrote to confirm exactly one rule body differs.
|
|
120
|
+
- **`judge_change.rb`** puts two analyses of one game side by side, blinded as X
|
|
121
|
+
and Y from a recorded seed, and asks a model whether the assessment changed.
|
|
122
|
+
Both experiments of stage 4 need that one question answered.
|
|
123
|
+
- **`test/test_propose_metric.rb`** — 16 checks over the extraction and the
|
|
124
|
+
runner without calling a model, 5 of them falsifications: a crashing submission
|
|
125
|
+
is not scored as having run even though it left output, a silent clean exit is,
|
|
126
|
+
an unfenced reply is not salvaged into code, a submission that deletes the
|
|
127
|
+
corpus cannot reach the original, and no environment variable crosses into the
|
|
128
|
+
child.
|
|
129
|
+
|
|
130
|
+
### Measured, and recorded in the SkillSet README
|
|
131
|
+
|
|
132
|
+
- **Stage 4's verdict does not work.** On one game with `Rule 105` rewritten from
|
|
133
|
+
unanimity to simple majority, the repeatability floor came back 9 of 9 CHANGED
|
|
134
|
+
and the mutated arm 9 of 9. What the judges pointed AT still separated them —
|
|
135
|
+
all 3 judges reading one analyst's pair named the Rule 105 misreading, none did
|
|
136
|
+
in the floor pair. Use the script for what its judges write, not for the tally
|
|
137
|
+
it prints. Counting occurrences of the substituted word does not work either:
|
|
138
|
+
one analyst says "majority" 4 and 2 times in the two clean readings and 5 in
|
|
139
|
+
the mutated one while detecting nothing.
|
|
140
|
+
- **Stage 2's own premise failed.** Holding the standard fixed left a 1.94-point
|
|
141
|
+
spread across judges against a 1.12-point spread across standards, so the judge
|
|
142
|
+
still moves the number more than the standard does. The own-standard premium
|
|
143
|
+
did not appear either: 4 diagonal cells averaged 5.50 against 5.88 for the 12
|
|
144
|
+
off-diagonal ones. The GAPS block was answered NONE in 0 of 16 cells, every
|
|
145
|
+
judge converging on the same four holes — whether proposing implies an
|
|
146
|
+
affirmative vote, how to read a wholly empty turn, which side to weight when
|
|
147
|
+
private reasoning and public utterance diverge, and whether the game master's
|
|
148
|
+
narrower action space belongs on the same scale.
|
|
149
|
+
- **Stage 3, first pass:** 2 of 3 submissions ran. The one that failed died on
|
|
150
|
+
`text.split()` against the 8 utterances of 391 whose text is null. One reply
|
|
151
|
+
contained 8 fabricated tool calls with fabricated results, citing a path that
|
|
152
|
+
exists but is empty and quoting lines that appear nowhere in the corpus.
|
|
153
|
+
- Only 9 of 24 games carry `rules_initial` in their lineup, which
|
|
154
|
+
`mutate_rule.rb` needs. The other 15 predate recorded rule bodies. No
|
|
155
|
+
workaround is provided; run more games instead.
|
|
156
|
+
|
|
7
157
|
## [3.74.0] - 2026-08-18
|
|
8
158
|
|
|
9
159
|
### Fixed
|
data/lib/kairos_mcp/version.rb
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.
|
|
4
|
+
version: "3.11.0"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -53,7 +53,9 @@ write a review spec and declare it frozen for the round:
|
|
|
53
53
|
3. **Cap fixes per round (≤5)** and write one line per fix: *what this fix
|
|
54
54
|
newly claims* (values pinned, ranges narrowed, failure visibility
|
|
55
55
|
changed). A fix that cannot state its new claims is doing more than the
|
|
56
|
-
finding asked.
|
|
56
|
+
finding asked. A round that moves its instrument records that it moved it
|
|
57
|
+
(§ Prompt Generation Rules). That is a recording duty; unlike the cap
|
|
58
|
+
above, it bounds nothing.
|
|
57
59
|
4. **Pre-flight falsifier.** Before dispatch, one agent whose only job is to
|
|
58
60
|
refute every factual claim in the spec and artifact — especially numbers
|
|
59
61
|
and "X does not exist" claims. In this loop it caught real errors before
|
|
@@ -563,6 +565,13 @@ Rules:
|
|
|
563
565
|
| Document review | Accuracy, completeness, consistency | Document text | Grant applications, papers |
|
|
564
566
|
| Final/convergence review | All prior findings resolved, no new issues | Resolution matrix + revised artifact | Before merge |
|
|
565
567
|
|
|
568
|
+
**A review answers at the phase of its target and does not descend.** Design
|
|
569
|
+
review and Document review read code only to refute a factual claim the
|
|
570
|
+
artifact makes about an existing system, and write none. Implementation review
|
|
571
|
+
is where fixes are written, and is also where a green suite is not evidence
|
|
572
|
+
until its tests have been shown to fail — a mutation pass whose sites the
|
|
573
|
+
code's author chose measures the author's attention, not the suite.
|
|
574
|
+
|
|
566
575
|
## LLM Role Differentiation
|
|
567
576
|
|
|
568
577
|
Without explicit instruction, different LLMs naturally focus on different verification layers:
|
|
@@ -601,9 +610,17 @@ numerator moved. Do not treat a reached ratio as sufficient on its own either:
|
|
|
601
610
|
check what the approving replies actually said before counting them.
|
|
602
611
|
|
|
603
612
|
**Count carryover and new (a)/(b) P0s separately; the machine-side signal of
|
|
604
|
-
convergence is "new P0 = 0", not the APPROVE ratio.** Require each
|
|
613
|
+
convergence is "new P0 = 0", not the APPROVE ratio.** Require each **seat** to
|
|
605
614
|
state a closure verdict on its own prior-round P0s — closed / open /
|
|
606
|
-
half-closed, with grounds.
|
|
615
|
+
half-closed, with grounds. **Who labels a finding new versus carryover is
|
|
616
|
+
undecided.** The seat-side answer — each seat labelling against its own prior
|
|
617
|
+
findings — was written into this section on 2026-08-22 and removed the same day:
|
|
618
|
+
a seat sees only its own priors, so a finding one seat raises after another seat
|
|
619
|
+
raised it is new to that seat and carryover to the round, and a seat added or
|
|
620
|
+
returning mid-thread has no priors at all, making every finding it raises new by
|
|
621
|
+
construction. Until this is answered, the orchestrator makes the call, knowing
|
|
622
|
+
that it is the context whose artifact the call converges. This format is
|
|
623
|
+
validated live (chain erasure
|
|
607
624
|
R6–R8) and is what makes the carryover/new split computable. A round whose
|
|
608
625
|
(a)+(b) findings are all carryover with closure verdicts, and whose revision
|
|
609
626
|
drew zero new P0s (observed without exception when the revision was
|
|
@@ -1202,6 +1219,19 @@ Every review prompt MUST include these 7 items:
|
|
|
1202
1219
|
|
|
1203
1220
|
All prompt content MUST be in **English** for consistent parsing across LLM tools.
|
|
1204
1221
|
|
|
1222
|
+
**A round that changes its instrument records the change.** The instrument is
|
|
1223
|
+
whatever bounds what the round can find: what a seat is told to look for, how
|
|
1224
|
+
much of the artifact it is given, and which seats answer. Some of a prompt
|
|
1225
|
+
varies by construction — the artifact itself, the prior findings a round asks a
|
|
1226
|
+
seat to verdict — and that is not the instrument. When a round moves the
|
|
1227
|
+
instrument, it says so on the record, in one line, naming what moved and why.
|
|
1228
|
+
|
|
1229
|
+
This is a recording duty, not a prohibition. An orchestrator may narrow a
|
|
1230
|
+
seat's criteria mid-thread, cut the scope, or convene a different panel; each
|
|
1231
|
+
of those is a documented remedy for something. What it may not do is move the
|
|
1232
|
+
instrument and then read the resulting change in finding counts as a property
|
|
1233
|
+
of the artifact.
|
|
1234
|
+
|
|
1205
1235
|
### Reviewer incentive rule
|
|
1206
1236
|
|
|
1207
1237
|
**Never tell a reviewer — subprocess or persona — that its finding count is
|
|
@@ -1247,7 +1277,8 @@ For each finding:
|
|
|
1247
1277
|
- **What can go wrong**: concrete failure scenario
|
|
1248
1278
|
- **Why this is vulnerable**: code path or design gap
|
|
1249
1279
|
- **Likely impact**: data loss, security breach, silent corruption, etc.
|
|
1250
|
-
- **Recommended fix**: specific
|
|
1280
|
+
- **Recommended fix**: specific, never "consider improving" — the change where
|
|
1281
|
+
this review writes changes, the claim that does not hold where it does not.
|
|
1251
1282
|
</structured_output_contract>
|
|
1252
1283
|
|
|
1253
1284
|
<grounding_rules>
|
|
@@ -1611,5 +1642,59 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1611
1642
|
with zero new and one carryover (a) at 1 of 2 seats approving reports
|
|
1612
1643
|
"GATE NOT PASSED" under the old rule and "FREEZE CANDIDATE" under this one.
|
|
1613
1644
|
|
|
1645
|
+
- Three norms, subtractively (v3.11.0, 2026-08-22): the operator named three
|
|
1646
|
+
recurring failures — an orchestrator issuing different criteria each round,
|
|
1647
|
+
the APPROVE ratio operating as the close condition, and design reviews
|
|
1648
|
+
descending into code — and asked for mutation experiments to be bounded. Four
|
|
1649
|
+
rules were written and reviewed; one was withdrawn in the same version and two
|
|
1650
|
+
of the survivors are narrower than they were drafted, so what ships is three
|
|
1651
|
+
norms and one recorded open question. **§ Prompt Generation Rules** gains a
|
|
1652
|
+
recording duty on the round's instrument — what a seat is told to look for,
|
|
1653
|
+
how much of the artifact it is given, and which seats answer. It bounds
|
|
1654
|
+
nothing: narrowing criteria, cutting scope and convening a different panel are
|
|
1655
|
+
each a documented remedy for something. What it forbids is moving the
|
|
1656
|
+
instrument and then reading the resulting change in counts as a property of
|
|
1657
|
+
the artifact. It was drafted around criteria alone, which left scope and panel
|
|
1658
|
+
composition — two channels that move the count just as well — outside it.
|
|
1659
|
+
**§ Review Types** gains the phase rule as an invariant with no branches: a
|
|
1660
|
+
review answers at the phase of its target and does not descend. Two enumerated
|
|
1661
|
+
sentences were drafted with it and are not here. "An implementation-phase
|
|
1662
|
+
finding that reopens the design belongs to the backlog" collided with this
|
|
1663
|
+
document's own "any (a) or (b) REJECT or FAIL = revise and re-review" over
|
|
1664
|
+
exactly the design-implementation seam it calls the most valuable layer, and
|
|
1665
|
+
pre-answered the operator question § Revision Discipline rule 1 requires. "Fix
|
|
1666
|
+
plan and Final/convergence reviews inherit the phase of the artifact they
|
|
1667
|
+
check" gave those two reviews nothing to inherit, since no phase is assigned to
|
|
1668
|
+
a fix plan anywhere. The output contract at `<structured_output_contract>`
|
|
1669
|
+
drops "consider improving" without enumerating review types, because the
|
|
1670
|
+
enumerated form left this document's own knowledge/documentation-update review
|
|
1671
|
+
unassigned. **The mutation norm** is one sentence inside the phase rule and
|
|
1672
|
+
assigns no party: a green suite is not evidence until its tests have been shown
|
|
1673
|
+
to fail, and a pass whose sites the code's author chose measures the author's
|
|
1674
|
+
attention, not the suite. It was drafted as a seventh item in Step -1 with the
|
|
1675
|
+
duty "whoever chooses the sites is not whoever wrote the code", and no party in
|
|
1676
|
+
this document can discharge that — before dispatch the only actor is the
|
|
1677
|
+
orchestrator, which § Roles makes the same context as the implementer, and the
|
|
1678
|
+
seats are sandboxed read-only and cannot execute anything. Stated as a property
|
|
1679
|
+
of evidence rather than a duty on someone, it holds wherever a mutation result
|
|
1680
|
+
is read. The bound the operator asked for is not here and the omission is open,
|
|
1681
|
+
not settled: the recorded decision is to accumulate three runs and then judge.
|
|
1682
|
+
**Withdrawn in this version**: a rule making each seat label its own findings
|
|
1683
|
+
new or carryover. A seat sees only its own priors, so a finding one seat raises
|
|
1684
|
+
after another raised it is new to that seat and carryover to the round, and a
|
|
1685
|
+
seat added or returning mid-thread has no priors at all. § Convergence Rules
|
|
1686
|
+
now records that who makes the label is undecided and that the orchestrator
|
|
1687
|
+
makes it meanwhile, which is the biased judge the withdrawn rule was written to
|
|
1688
|
+
replace. A fifth proposed rule, an exception for self-referential reviews, was
|
|
1689
|
+
dropped before this version was written, after four reviewer contexts —
|
|
1690
|
+
occupying two of that round's three counted seats — shot it on three
|
|
1691
|
+
independent grounds. Design history: three review rounds on the proposal
|
|
1692
|
+
(closed by operator declaration, not convergence), then one round on the
|
|
1693
|
+
applied text, which returned one APPROVE of four seats and eighteen blocking
|
|
1694
|
+
findings — every rule above is narrower for it. What kept being shot across all
|
|
1695
|
+
four rounds was the change's account of itself, not the rules it proposed.
|
|
1696
|
+
Records: L2 `handoff_mlr_l1_norms_revision_three_rounds_and_switch_to_implementation_20260821`
|
|
1697
|
+
and L2 `mlr_v3_11_0_applied_review_r1_and_subtractive_revision_20260822`
|
|
1698
|
+
|
|
1614
1699
|
**Key insight**: Design reviews and implementation reviews find
|
|
1615
1700
|
**categorically different bugs**. Both phases are necessary.
|
|
@@ -31,6 +31,31 @@ module KairosHookProjector
|
|
|
31
31
|
TAIL_BYTES = 512 * 1024
|
|
32
32
|
POLL_ATTEMPTS = 15
|
|
33
33
|
POLL_DELAY = 0.1
|
|
34
|
+
# The recheck waits longer than the first read, because it is waiting for a
|
|
35
|
+
# whole rewrite rather than for the tail of a record already being written.
|
|
36
|
+
# Bounded well under the hook's own timeout; the turn it lengthens is one
|
|
37
|
+
# that already spent a full regeneration.
|
|
38
|
+
RECHECK_POLL_ATTEMPTS = 40
|
|
39
|
+
# Claude Code's own wording, written into the transcript as a user record
|
|
40
|
+
# when a Stop hook blocks. Not this gate's string: if it changes, no marker
|
|
41
|
+
# is found and the read falls back to the pre-2026-08-20 behaviour, which
|
|
42
|
+
# the verdict names so the drift shows up in the log instead of silently.
|
|
43
|
+
BLOCK_MARKER = 'Stop hook feedback:'
|
|
44
|
+
# Claude Code's limit on how long a Stop hook may run, declared twice
|
|
45
|
+
# outside this file: in `.claude/settings.json` and in this SkillSet's
|
|
46
|
+
# `lib/mode_hooks_compiler.rb`. Named here because this file spends two
|
|
47
|
+
# budgets in sequence and neither knows about the other. A hook killed at
|
|
48
|
+
# the limit emits nothing at all — no verdict, no banner, no log line — so
|
|
49
|
+
# the gate stops enforcing and the measurement loses rows, both silently.
|
|
50
|
+
HOOK_TIMEOUT = 10.0
|
|
51
|
+
# Held back from the pair for interpreter start-up and for the poll's own
|
|
52
|
+
# overhead, which measures 0.2-0.6s above the sleeps it accounts for.
|
|
53
|
+
HOOK_TIMEOUT_MARGIN = 1.0
|
|
54
|
+
# What is left for measurement once the recheck has spent its budget. A mode
|
|
55
|
+
# asking for more than this is honoured up to here and no further: the
|
|
56
|
+
# alternative is a mode file being able to silence the gate by arithmetic.
|
|
57
|
+
MEASURE_TIMEOUT_CEILING =
|
|
58
|
+
HOOK_TIMEOUT - HOOK_TIMEOUT_MARGIN - (RECHECK_POLL_ATTEMPTS * POLL_DELAY)
|
|
34
59
|
|
|
35
60
|
# Built from single-quoted strings so that `#{` in a pattern stays literal
|
|
36
61
|
# rather than becoming interpolation, and anchored with \A because Python's
|
|
@@ -131,7 +156,18 @@ module KairosHookProjector
|
|
|
131
156
|
# list. Text explaining the vocabulary rule has to name the shapes it
|
|
132
157
|
# governs, and naming them is not using them.
|
|
133
158
|
@specimen = merged['specimen_patterns'].map { |p| Regexp.new(p) }
|
|
134
|
-
|
|
159
|
+
# Clamped, not rejected: a mode that asks for 6 seconds gets what fits
|
|
160
|
+
# rather than a refusal it cannot act on. 4 seconds of recheck poll plus
|
|
161
|
+
# 6 of measurement is 10.4 against a 10-second hook limit, and the hook
|
|
162
|
+
# dies with no output, so the gate would silently stop enforcing for
|
|
163
|
+
# exactly the modes whose patterns are expensive enough to need the time.
|
|
164
|
+
requested = merged['measure_timeout_seconds']
|
|
165
|
+
@measure_timeout =
|
|
166
|
+
if requested.is_a?(Numeric) && requested > MEASURE_TIMEOUT_CEILING
|
|
167
|
+
MEASURE_TIMEOUT_CEILING
|
|
168
|
+
else
|
|
169
|
+
requested
|
|
170
|
+
end
|
|
135
171
|
@log_max_bytes = merged['log_max_bytes']
|
|
136
172
|
end
|
|
137
173
|
|
|
@@ -275,32 +311,168 @@ module KairosHookProjector
|
|
|
275
311
|
joined.empty? ? nil : joined
|
|
276
312
|
end
|
|
277
313
|
|
|
314
|
+
# Index of the marker for THIS turn, or nil while it has not landed.
|
|
315
|
+
#
|
|
316
|
+
# The transcript records the block itself: a user record carrying Claude
|
|
317
|
+
# Code's feedback text, whose parentUuid names the assistant record that was
|
|
318
|
+
# judged. That pair is the entire state the recheck needs — which record has
|
|
319
|
+
# already had a verdict, and where the rewrite begins — so no state file and
|
|
320
|
+
# no config key are involved.
|
|
321
|
+
#
|
|
322
|
+
# The newest marker in the transcript is not necessarily this turn's, and
|
|
323
|
+
# taking it was a defect of its own. A session blocked once already carries
|
|
324
|
+
# an older marker; on the next block, before the new marker lands, the older
|
|
325
|
+
# one made the read believe a marker had arrived, so it skipped the wait and
|
|
326
|
+
# measured the message it had just blocked — returning in 0.06s with nothing
|
|
327
|
+
# in the log to say so. 128 of 170 real blocks are not a session's first.
|
|
328
|
+
# Between the previous block and this one there is always the operator's own
|
|
329
|
+
# message, so the test is which of the two is newer. Records carrying no
|
|
330
|
+
# text — tool results, images — belong to the turn being written and are
|
|
331
|
+
# stepped over.
|
|
332
|
+
def turn_marker(rows)
|
|
333
|
+
i = rows.length - 1
|
|
334
|
+
while i >= 0
|
|
335
|
+
row = rows[i]
|
|
336
|
+
if row['type'] == 'user'
|
|
337
|
+
text = text_of(row)
|
|
338
|
+
return text.start_with?(BLOCK_MARKER) ? i : nil if text.is_a?(String)
|
|
339
|
+
end
|
|
340
|
+
|
|
341
|
+
i -= 1
|
|
342
|
+
end
|
|
343
|
+
nil
|
|
344
|
+
end
|
|
345
|
+
|
|
346
|
+
# Compared only when a judged uuid is present: comparing it while nil made a
|
|
347
|
+
# record carrying no uuid look like the judged record and skipped a real
|
|
348
|
+
# rewrite.
|
|
349
|
+
def judged?(row, judged)
|
|
350
|
+
judged && row['uuid'] == judged
|
|
351
|
+
end
|
|
352
|
+
|
|
353
|
+
# The newest assistant record after `index` other than the judged one,
|
|
354
|
+
# whether or not it carries text yet. nil while none has landed.
|
|
355
|
+
#
|
|
356
|
+
# This is the first read's rule applied after the marker, and it is what the
|
|
357
|
+
# recheck uses while it still has budget. A rewrite that is mid-write shows
|
|
358
|
+
# up here as text-less and is waited for, rather than stepped over in favour
|
|
359
|
+
# of whatever text lies beneath it. Stepping over had a measured cost: a
|
|
360
|
+
# rewrite that called a tool was judged on its own two-line preamble and
|
|
361
|
+
# logged RECHECK-PASS in 0.04s, while the rewrite itself — a four-heading
|
|
362
|
+
# FAIL landing 0.6s later — was never read.
|
|
363
|
+
def newest_after(rows, index, judged)
|
|
364
|
+
i = rows.length - 1
|
|
365
|
+
while i > index
|
|
366
|
+
row = rows[i]
|
|
367
|
+
return [text_of(row), row['uuid']] if row['type'] == 'assistant' && !judged?(row, judged)
|
|
368
|
+
|
|
369
|
+
i -= 1
|
|
370
|
+
end
|
|
371
|
+
nil
|
|
372
|
+
end
|
|
373
|
+
|
|
374
|
+
# The newest text-bearing assistant record after `index`, stepping over
|
|
375
|
+
# records that carry none. Used only once the budget is spent.
|
|
376
|
+
#
|
|
377
|
+
# Over 3,248 transcripts and 170 real blocks, a completed rewrite sitting
|
|
378
|
+
# beneath a newer text-less record occurred 0 times, so this shape must not
|
|
379
|
+
# pre-empt the wait — that is what cost the preamble above. It is kept as a
|
|
380
|
+
# last resort because if the shape ever does occur, the alternative is the
|
|
381
|
+
# same skipped verdict this reaches past.
|
|
382
|
+
def deep_after(rows, index, judged)
|
|
383
|
+
i = rows.length - 1
|
|
384
|
+
while i > index
|
|
385
|
+
row = rows[i]
|
|
386
|
+
if row['type'] == 'assistant' && !judged?(row, judged)
|
|
387
|
+
text = text_of(row)
|
|
388
|
+
return [text, row['uuid']] if text
|
|
389
|
+
end
|
|
390
|
+
|
|
391
|
+
i -= 1
|
|
392
|
+
end
|
|
393
|
+
nil
|
|
394
|
+
end
|
|
395
|
+
|
|
396
|
+
# The newest assistant record, whether or not it carries text yet, or nil
|
|
397
|
+
# when the transcript holds none. The first read's rule, unchanged since
|
|
398
|
+
# before 2026-08-20, and the rule the recheck falls back to when no marker
|
|
399
|
+
# is found.
|
|
400
|
+
def newest_assistant(rows)
|
|
401
|
+
row = rows.reverse_each.find { |r| r['type'] == 'assistant' }
|
|
402
|
+
row && [text_of(row), row['uuid']]
|
|
403
|
+
end
|
|
404
|
+
|
|
278
405
|
# Text of the turn's final assistant message, with the flush race handled.
|
|
279
406
|
#
|
|
280
407
|
# One response is written as several records (thinking, text, tool_use) at
|
|
281
408
|
# different times. At Stop time the `text` record may not have landed yet,
|
|
282
409
|
# so the newest assistant record is often thinking-only. Wait for the text
|
|
283
410
|
# rather than judging an earlier message from the same turn.
|
|
284
|
-
|
|
285
|
-
|
|
411
|
+
#
|
|
412
|
+
# On a recheck that wait never engaged, and that was the defect: the blocked
|
|
413
|
+
# message is still the newest record carrying text, so it satisfied the
|
|
414
|
+
# "has text" test immediately and was judged a second time. Measured over
|
|
415
|
+
# one instance's first 768 log records, 140 of 140 rechecks took the newest
|
|
416
|
+
# record without ever waiting, and 109 of 140 reported metrics identical to
|
|
417
|
+
# the verdict that had just blocked. The rewrite had passed in most of them.
|
|
418
|
+
# The invariant this restores, as long as the marker is found: a record that
|
|
419
|
+
# a verdict already named is never judged again.
|
|
420
|
+
def last_assistant_text(transcript_path, rechecked = false)
|
|
421
|
+
attempts = rechecked ? RECHECK_POLL_ATTEMPTS : POLL_ATTEMPTS
|
|
422
|
+
attempts.times do |attempt|
|
|
286
423
|
rows = tail_records(transcript_path)
|
|
287
|
-
return [nil, 'unreadable'] if rows.nil?
|
|
424
|
+
return [nil, 'unreadable', nil] if rows.nil?
|
|
425
|
+
|
|
426
|
+
waited = attempt.zero? ? 'ok' : 'ok-after-wait'
|
|
427
|
+
|
|
428
|
+
if rechecked
|
|
429
|
+
# The marker is waited for on the same terms as the rewrite: both are
|
|
430
|
+
# written by the same process at nearly the same moment, so "not there
|
|
431
|
+
# yet" is as ordinary for one as for the other. Inside the budget the
|
|
432
|
+
# selection rule is the first read's — newest record, wait for its
|
|
433
|
+
# text — so the two reads agree wherever they can.
|
|
434
|
+
marker = turn_marker(rows)
|
|
435
|
+
if marker
|
|
436
|
+
found = newest_after(rows, marker, rows[marker]['parentUuid'])
|
|
437
|
+
return [found[0], waited, found[1]] if found && found[0]
|
|
438
|
+
end
|
|
439
|
+
else
|
|
440
|
+
newest = newest_assistant(rows)
|
|
441
|
+
return [nil, 'no-assistant-record', nil] if newest.nil?
|
|
442
|
+
return [newest[0], waited, newest[1]] if newest[0]
|
|
443
|
+
end
|
|
444
|
+
sleep(POLL_DELAY)
|
|
445
|
+
end
|
|
288
446
|
|
|
289
|
-
|
|
290
|
-
rows.reverse_each do |row|
|
|
291
|
-
next unless row['type'] == 'assistant'
|
|
447
|
+
return [nil, 'race-timeout', nil] unless rechecked
|
|
292
448
|
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
449
|
+
# The recheck's budget is spent. Either this turn's marker never appeared
|
|
450
|
+
# — Claude Code's wording is not this gate's to guarantee — or it appeared
|
|
451
|
+
# and the rewrite did not. Both are named, and every no-marker outcome
|
|
452
|
+
# carries the suffix, because a drift detector whose only visible trace is
|
|
453
|
+
# the exit that happens to find text reports nothing in the cases drift
|
|
454
|
+
# actually produces.
|
|
455
|
+
rows = tail_records(transcript_path)
|
|
456
|
+
return [nil, 'unreadable', nil] if rows.nil?
|
|
296
457
|
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
458
|
+
marker = turn_marker(rows)
|
|
459
|
+
if marker
|
|
460
|
+
# Last resort, and named apart so the log can say how often the ordinary
|
|
461
|
+
# rule was not enough.
|
|
462
|
+
found = deep_after(rows, marker, rows[marker]['parentUuid'])
|
|
463
|
+
return [found[0], 'ok-after-wait-deep', found[1]] if found
|
|
300
464
|
|
|
301
|
-
|
|
465
|
+
return [nil, 'awaiting-rewrite', nil]
|
|
302
466
|
end
|
|
303
|
-
|
|
467
|
+
|
|
468
|
+
# No marker for this turn, so nothing is judged. Falling back to the first
|
|
469
|
+
# read's rule here looked conservative and was the opposite: the newest
|
|
470
|
+
# record it reaches is the message this turn has just blocked, so the
|
|
471
|
+
# fallback re-judged it — the very behaviour this method exists to end,
|
|
472
|
+
# reachable on any block after a session's first. A recheck never blocks,
|
|
473
|
+
# so refusing to judge costs the operator nothing except the line, and it
|
|
474
|
+
# keeps both the invariant and the measurement intact.
|
|
475
|
+
newest_assistant(rows) ? [nil, 'nomarker', nil] : [nil, 'no-assistant-record-nomarker', nil]
|
|
304
476
|
end
|
|
305
477
|
|
|
306
478
|
# --- measurement ---------------------------------------------------------
|
|
@@ -460,15 +632,20 @@ module KairosHookProjector
|
|
|
460
632
|
# have — an empty-string or directory log_path left the gate blocking and
|
|
461
633
|
# passing exactly as normal while nothing was ever recorded, so the
|
|
462
634
|
# operator's onboarding week produced no data and no complaint.
|
|
463
|
-
def note(cfg, verdict, metrics = nil)
|
|
635
|
+
def note(cfg, verdict, metrics = nil, record_id = nil)
|
|
464
636
|
# nil alone means "no log declared". false, 0 and "" are declarations the
|
|
465
637
|
# gate cannot honour, and each has to reach the rescue below to be named.
|
|
466
638
|
return if cfg.log_path.nil?
|
|
467
639
|
|
|
468
640
|
stamp = Time.now.strftime('%Y-%m-%dT%H:%M:%S')
|
|
469
641
|
detail = ''
|
|
642
|
+
# Which record the verdict is about. Without it the log cannot answer the
|
|
643
|
+
# one question the fix above turns on — whether two verdicts in a row were
|
|
644
|
+
# about the same message — and it is also how a first read that grabbed a
|
|
645
|
+
# stale record becomes diagnosable rather than merely suspected.
|
|
646
|
+
detail += "\trec=#{record_id[0, 8]}" if record_id.is_a?(String) && !record_id.empty?
|
|
470
647
|
if metrics
|
|
471
|
-
detail
|
|
648
|
+
detail += format(
|
|
472
649
|
"\tlines=%d\theadings=%d\ttables=%d\tdiagrams=%d\tunglossed=%s",
|
|
473
650
|
metrics['lines'], metrics['headings'], metrics['tables'], metrics['diagrams'],
|
|
474
651
|
metrics['unglossed'].empty? ? '-' : metrics['unglossed'].join(',')
|
|
@@ -643,9 +820,31 @@ module KairosHookProjector
|
|
|
643
820
|
# reported, so its outcome is visible; it is simply never blocked again.
|
|
644
821
|
rechecked = payload['stop_hook_active'] ? true : false
|
|
645
822
|
|
|
646
|
-
text, why = last_assistant_text(payload.fetch('transcript_path', ''))
|
|
823
|
+
text, why, record_id = last_assistant_text(payload.fetch('transcript_path', ''), rechecked)
|
|
647
824
|
if text.nil? || text.strip.empty?
|
|
648
|
-
note(cfg, "SKIP-#{why}")
|
|
825
|
+
lost = note(cfg, "SKIP-#{why}")
|
|
826
|
+
# Every recheck that produces no verdict says so on screen. Naming only
|
|
827
|
+
# the two expected reasons left the others silent, and the ones it left
|
|
828
|
+
# out are live: a transcript momentarily unreadable, and a rewrite whose
|
|
829
|
+
# text is whitespace after the fallback found it. Both spent the budget
|
|
830
|
+
# and told the operator nothing. A first read keeps its old silence —
|
|
831
|
+
# there the turn goes out regardless and the log is the right place.
|
|
832
|
+
budget = format('%.1f', RECHECK_POLL_ATTEMPTS * POLL_DELAY)
|
|
833
|
+
tail = lost ? "; #{log_note(lost)}" : ''
|
|
834
|
+
reason =
|
|
835
|
+
case why
|
|
836
|
+
when 'awaiting-rewrite'
|
|
837
|
+
"the rewrite had not been recorded within #{budget}s"
|
|
838
|
+
when 'nomarker', 'no-assistant-record-nomarker'
|
|
839
|
+
"no block marker appeared within #{budget}s, so the rewrite could not be told " \
|
|
840
|
+
"apart from the message that was blocked. Claude Code's feedback wording may " \
|
|
841
|
+
'have changed'
|
|
842
|
+
when 'unreadable'
|
|
843
|
+
'the transcript could not be read'
|
|
844
|
+
else
|
|
845
|
+
"nothing measurable was found (#{why})"
|
|
846
|
+
end
|
|
847
|
+
emit('systemMessage' => "#{cfg.banner_prefix} (recheck): NOT RUN — #{reason}#{tail}") if rechecked
|
|
649
848
|
return 0
|
|
650
849
|
end
|
|
651
850
|
|
|
@@ -663,7 +862,7 @@ module KairosHookProjector
|
|
|
663
862
|
end
|
|
664
863
|
|
|
665
864
|
verdict = failures.empty? ? 'PASS' : 'FAIL'
|
|
666
|
-
lost = note(cfg, "#{rechecked ? 'RECHECK-' : ''}#{verdict}-#{why}", metrics)
|
|
865
|
+
lost = note(cfg, "#{rechecked ? 'RECHECK-' : ''}#{verdict}-#{why}", metrics, record_id)
|
|
667
866
|
|
|
668
867
|
out = { 'systemMessage' => banner(cfg, verdict, metrics, failures, rechecked, lost) }
|
|
669
868
|
if !failures.empty? && !rechecked && cfg.blocking
|