kairos-chain 3.75.0 → 3.76.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +77 -0
- data/lib/kairos_mcp/version.rb +1 -1
- data/templates/knowledge/multi_llm_review_workflow/multi_llm_review_workflow.md +90 -5
- data/templates/skillsets/kairos_hook_projector/hooks/readable_gate.rb +219 -20
- data/templates/skillsets/kairos_hook_projector/test/mutation_check_readable_gate.rb +169 -0
- data/templates/skillsets/kairos_hook_projector/test/test_readable_gate.rb +426 -3
- metadata +2 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 83162bcb26c033c10413fe1f25ffed382031dfebc115aa7041438d807177ef8c
|
|
4
|
+
data.tar.gz: eacc1ed007ad60d8429954c1d0e0bca228048a5a0438f177df892e72aae3e599
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e6e311e80bc090726582d7032aa87a0ac1a97ffea387b300306d609f19b0a38c4dbe2cd6a84dd36e924e66c3e9f579b56f91c1c25602cc2afdae21e8f8e2005c
|
|
7
|
+
data.tar.gz: 0ffddbff73ac05f8804b343e597495517b3872a0ceed500ef1370aab8f034a866b20585a5a03c46db179fbacfd18aff3862a70fc40c093f66389a69f2afdc4c2
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,83 @@ All notable changes to the `kairos-chain` gem will be documented in this file.
|
|
|
4
4
|
|
|
5
5
|
This project follows [Semantic Versioning](https://semver.org/).
|
|
6
6
|
|
|
7
|
+
## [3.76.0] - 2026-08-22
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- **L1 `multi_llm_review_workflow` 3.10.2 → 3.11.0 — three norms and one
|
|
12
|
+
recorded open question, for three failures the operator named and one request
|
|
13
|
+
he made.** The failures: an orchestrator issuing different criteria each
|
|
14
|
+
round, the APPROVE ratio operating as the close condition, and design reviews
|
|
15
|
+
descending into code. The request: bound mutation experiments. Four rules were
|
|
16
|
+
drafted, reviewed over three rounds, applied, reviewed once more as applied
|
|
17
|
+
text — one APPROVE of four seats, eighteen blocking findings — and revised
|
|
18
|
+
subtractively. What ships is narrower than what was drafted, which is the
|
|
19
|
+
point: this entry's own § Revision Discipline records that deletions drew zero
|
|
20
|
+
new P0s in every measured case and additions became the next round's
|
|
21
|
+
battleground.
|
|
22
|
+
|
|
23
|
+
**§ Prompt Generation Rules gains a recording duty on the round's
|
|
24
|
+
instrument** — what a seat is told to look for, how much of the artifact it is
|
|
25
|
+
given, and which seats answer. It bounds nothing: narrowing criteria, cutting
|
|
26
|
+
scope and convening a different panel are each a documented remedy for
|
|
27
|
+
something. It forbids moving the instrument and then reading the resulting
|
|
28
|
+
change in finding counts as a property of the artifact. It was drafted around
|
|
29
|
+
criteria alone, which left scope and panel composition — two channels that
|
|
30
|
+
move the count just as well — outside it.
|
|
31
|
+
|
|
32
|
+
**§ Review Types gains the phase rule as an invariant with no branches**: a
|
|
33
|
+
review answers at the phase of its target and does not descend. Design and
|
|
34
|
+
Document reviews read code only to refute a factual claim and write none;
|
|
35
|
+
implementation review is where fixes are written. Two enumerated sentences
|
|
36
|
+
drafted with it are not here. "An implementation-phase finding that reopens
|
|
37
|
+
the design belongs to the backlog" collided with this entry's own "any (a) or
|
|
38
|
+
(b) REJECT or FAIL = revise and re-review" over exactly the
|
|
39
|
+
design-implementation seam it calls the most valuable layer, and pre-answered
|
|
40
|
+
the operator question § Revision Discipline rule 1 requires. "Fix plan and
|
|
41
|
+
Final/convergence reviews inherit the phase of the artifact they check" gave
|
|
42
|
+
those two nothing to inherit, no phase being assigned to a fix plan anywhere.
|
|
43
|
+
The `<structured_output_contract>` line drops "consider improving" without
|
|
44
|
+
enumerating review types, because the enumerated form left this entry's own
|
|
45
|
+
knowledge/documentation-update review unassigned.
|
|
46
|
+
|
|
47
|
+
**The mutation norm is one sentence inside the phase rule and assigns no
|
|
48
|
+
party**: a green suite is not evidence until its tests have been shown to
|
|
49
|
+
fail, and a pass whose sites the code's author chose measures the author's
|
|
50
|
+
attention, not the suite. It was drafted as a seventh item in Step -1 carrying
|
|
51
|
+
the duty "whoever chooses the sites is not whoever wrote the code", and no
|
|
52
|
+
party in this entry can discharge that — before dispatch the only actor is the
|
|
53
|
+
orchestrator, which § Roles makes the same context as the implementer, and
|
|
54
|
+
review seats run sandboxed read-only and cannot execute anything. Stated as a
|
|
55
|
+
property of evidence rather than a duty on someone, it holds wherever a
|
|
56
|
+
mutation result is read. The bound the operator asked for is not here and the
|
|
57
|
+
omission is open, not settled: the recorded decision is to accumulate three
|
|
58
|
+
runs and then judge.
|
|
59
|
+
|
|
60
|
+
**Withdrawn in the same version: a rule making each seat label its own
|
|
61
|
+
findings new or carryover.** A seat sees only its own priors, so a finding one
|
|
62
|
+
seat raises after another raised it is new to that seat and carryover to the
|
|
63
|
+
round, and a seat added or returning mid-thread has no priors at all, making
|
|
64
|
+
every finding it raises new by construction — the machine-side signal "new
|
|
65
|
+
(a)+(b) P0 = 0" is then not computable from seat labels. § Convergence Rules
|
|
66
|
+
records that who makes the label is undecided and that the orchestrator makes
|
|
67
|
+
it meanwhile, which is the biased judge the withdrawn rule was written to
|
|
68
|
+
replace. Its paired edits revert with it, which also removes "the round number
|
|
69
|
+
is never written into a prompt" — that sentence contradicted § Output
|
|
70
|
+
Directive's MUST that the prompt carry `review{N}` in the output path, and an
|
|
71
|
+
orchestrator obeying it would have had round N+1 overwrite round N.
|
|
72
|
+
|
|
73
|
+
A fifth proposed rule, an exception for self-referential reviews, was dropped
|
|
74
|
+
before the version was written, after four reviewer contexts — occupying two
|
|
75
|
+
of that round's three counted seats — shot it on three independent grounds.
|
|
76
|
+
|
|
77
|
+
Across all four review rounds the thing most often refuted was not the rules
|
|
78
|
+
but the change's account of itself. The pre-flight falsifier for the applied
|
|
79
|
+
round refuted 8 of 22 checked claims, three of them inside the version entry;
|
|
80
|
+
the correction to those three introduced a fourth false claim, which the round
|
|
81
|
+
then refuted. Every such claim is a record summarised without its denominator
|
|
82
|
+
or its condition.
|
|
83
|
+
|
|
7
84
|
## [3.75.0] - 2026-08-21
|
|
8
85
|
|
|
9
86
|
### Added
|
data/lib/kairos_mcp/version.rb
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: multi_llm_review_workflow
|
|
3
3
|
description: "Multi-LLM review methodology and execution — workflow pattern, CLI tooling, consensus analysis, Persona Assembly. Applicable to design, implementation, documentation, or any artifact."
|
|
4
|
-
version: "3.
|
|
4
|
+
version: "3.11.0"
|
|
5
5
|
tags:
|
|
6
6
|
- workflow
|
|
7
7
|
- review
|
|
@@ -53,7 +53,9 @@ write a review spec and declare it frozen for the round:
|
|
|
53
53
|
3. **Cap fixes per round (≤5)** and write one line per fix: *what this fix
|
|
54
54
|
newly claims* (values pinned, ranges narrowed, failure visibility
|
|
55
55
|
changed). A fix that cannot state its new claims is doing more than the
|
|
56
|
-
finding asked.
|
|
56
|
+
finding asked. A round that moves its instrument records that it moved it
|
|
57
|
+
(§ Prompt Generation Rules). That is a recording duty; unlike the cap
|
|
58
|
+
above, it bounds nothing.
|
|
57
59
|
4. **Pre-flight falsifier.** Before dispatch, one agent whose only job is to
|
|
58
60
|
refute every factual claim in the spec and artifact — especially numbers
|
|
59
61
|
and "X does not exist" claims. In this loop it caught real errors before
|
|
@@ -563,6 +565,13 @@ Rules:
|
|
|
563
565
|
| Document review | Accuracy, completeness, consistency | Document text | Grant applications, papers |
|
|
564
566
|
| Final/convergence review | All prior findings resolved, no new issues | Resolution matrix + revised artifact | Before merge |
|
|
565
567
|
|
|
568
|
+
**A review answers at the phase of its target and does not descend.** Design
|
|
569
|
+
review and Document review read code only to refute a factual claim the
|
|
570
|
+
artifact makes about an existing system, and write none. Implementation review
|
|
571
|
+
is where fixes are written, and is also where a green suite is not evidence
|
|
572
|
+
until its tests have been shown to fail — a mutation pass whose sites the
|
|
573
|
+
code's author chose measures the author's attention, not the suite.
|
|
574
|
+
|
|
566
575
|
## LLM Role Differentiation
|
|
567
576
|
|
|
568
577
|
Without explicit instruction, different LLMs naturally focus on different verification layers:
|
|
@@ -601,9 +610,17 @@ numerator moved. Do not treat a reached ratio as sufficient on its own either:
|
|
|
601
610
|
check what the approving replies actually said before counting them.
|
|
602
611
|
|
|
603
612
|
**Count carryover and new (a)/(b) P0s separately; the machine-side signal of
|
|
604
|
-
convergence is "new P0 = 0", not the APPROVE ratio.** Require each
|
|
613
|
+
convergence is "new P0 = 0", not the APPROVE ratio.** Require each **seat** to
|
|
605
614
|
state a closure verdict on its own prior-round P0s — closed / open /
|
|
606
|
-
half-closed, with grounds.
|
|
615
|
+
half-closed, with grounds. **Who labels a finding new versus carryover is
|
|
616
|
+
undecided.** The seat-side answer — each seat labelling against its own prior
|
|
617
|
+
findings — was written into this section on 2026-08-22 and removed the same day:
|
|
618
|
+
a seat sees only its own priors, so a finding one seat raises after another seat
|
|
619
|
+
raised it is new to that seat and carryover to the round, and a seat added or
|
|
620
|
+
returning mid-thread has no priors at all, making every finding it raises new by
|
|
621
|
+
construction. Until this is answered, the orchestrator makes the call, knowing
|
|
622
|
+
that it is the context whose artifact the call converges. This format is
|
|
623
|
+
validated live (chain erasure
|
|
607
624
|
R6–R8) and is what makes the carryover/new split computable. A round whose
|
|
608
625
|
(a)+(b) findings are all carryover with closure verdicts, and whose revision
|
|
609
626
|
drew zero new P0s (observed without exception when the revision was
|
|
@@ -1202,6 +1219,19 @@ Every review prompt MUST include these 7 items:
|
|
|
1202
1219
|
|
|
1203
1220
|
All prompt content MUST be in **English** for consistent parsing across LLM tools.
|
|
1204
1221
|
|
|
1222
|
+
**A round that changes its instrument records the change.** The instrument is
|
|
1223
|
+
whatever bounds what the round can find: what a seat is told to look for, how
|
|
1224
|
+
much of the artifact it is given, and which seats answer. Some of a prompt
|
|
1225
|
+
varies by construction — the artifact itself, the prior findings a round asks a
|
|
1226
|
+
seat to verdict — and that is not the instrument. When a round moves the
|
|
1227
|
+
instrument, it says so on the record, in one line, naming what moved and why.
|
|
1228
|
+
|
|
1229
|
+
This is a recording duty, not a prohibition. An orchestrator may narrow a
|
|
1230
|
+
seat's criteria mid-thread, cut the scope, or convene a different panel; each
|
|
1231
|
+
of those is a documented remedy for something. What it may not do is move the
|
|
1232
|
+
instrument and then read the resulting change in finding counts as a property
|
|
1233
|
+
of the artifact.
|
|
1234
|
+
|
|
1205
1235
|
### Reviewer incentive rule
|
|
1206
1236
|
|
|
1207
1237
|
**Never tell a reviewer — subprocess or persona — that its finding count is
|
|
@@ -1247,7 +1277,8 @@ For each finding:
|
|
|
1247
1277
|
- **What can go wrong**: concrete failure scenario
|
|
1248
1278
|
- **Why this is vulnerable**: code path or design gap
|
|
1249
1279
|
- **Likely impact**: data loss, security breach, silent corruption, etc.
|
|
1250
|
-
- **Recommended fix**: specific
|
|
1280
|
+
- **Recommended fix**: specific, never "consider improving" — the change where
|
|
1281
|
+
this review writes changes, the claim that does not hold where it does not.
|
|
1251
1282
|
</structured_output_contract>
|
|
1252
1283
|
|
|
1253
1284
|
<grounding_rules>
|
|
@@ -1611,5 +1642,59 @@ Compression ratio: parallel agent raw → Assembly ≈ 2:1
|
|
|
1611
1642
|
with zero new and one carryover (a) at 1 of 2 seats approving reports
|
|
1612
1643
|
"GATE NOT PASSED" under the old rule and "FREEZE CANDIDATE" under this one.
|
|
1613
1644
|
|
|
1645
|
+
- Three norms, subtractively (v3.11.0, 2026-08-22): the operator named three
|
|
1646
|
+
recurring failures — an orchestrator issuing different criteria each round,
|
|
1647
|
+
the APPROVE ratio operating as the close condition, and design reviews
|
|
1648
|
+
descending into code — and asked for mutation experiments to be bounded. Four
|
|
1649
|
+
rules were written and reviewed; one was withdrawn in the same version and two
|
|
1650
|
+
of the survivors are narrower than they were drafted, so what ships is three
|
|
1651
|
+
norms and one recorded open question. **§ Prompt Generation Rules** gains a
|
|
1652
|
+
recording duty on the round's instrument — what a seat is told to look for,
|
|
1653
|
+
how much of the artifact it is given, and which seats answer. It bounds
|
|
1654
|
+
nothing: narrowing criteria, cutting scope and convening a different panel are
|
|
1655
|
+
each a documented remedy for something. What it forbids is moving the
|
|
1656
|
+
instrument and then reading the resulting change in counts as a property of
|
|
1657
|
+
the artifact. It was drafted around criteria alone, which left scope and panel
|
|
1658
|
+
composition — two channels that move the count just as well — outside it.
|
|
1659
|
+
**§ Review Types** gains the phase rule as an invariant with no branches: a
|
|
1660
|
+
review answers at the phase of its target and does not descend. Two enumerated
|
|
1661
|
+
sentences were drafted with it and are not here. "An implementation-phase
|
|
1662
|
+
finding that reopens the design belongs to the backlog" collided with this
|
|
1663
|
+
document's own "any (a) or (b) REJECT or FAIL = revise and re-review" over
|
|
1664
|
+
exactly the design-implementation seam it calls the most valuable layer, and
|
|
1665
|
+
pre-answered the operator question § Revision Discipline rule 1 requires. "Fix
|
|
1666
|
+
plan and Final/convergence reviews inherit the phase of the artifact they
|
|
1667
|
+
check" gave those two reviews nothing to inherit, since no phase is assigned to
|
|
1668
|
+
a fix plan anywhere. The output contract at `<structured_output_contract>`
|
|
1669
|
+
drops "consider improving" without enumerating review types, because the
|
|
1670
|
+
enumerated form left this document's own knowledge/documentation-update review
|
|
1671
|
+
unassigned. **The mutation norm** is one sentence inside the phase rule and
|
|
1672
|
+
assigns no party: a green suite is not evidence until its tests have been shown
|
|
1673
|
+
to fail, and a pass whose sites the code's author chose measures the author's
|
|
1674
|
+
attention, not the suite. It was drafted as a seventh item in Step -1 with the
|
|
1675
|
+
duty "whoever chooses the sites is not whoever wrote the code", and no party in
|
|
1676
|
+
this document can discharge that — before dispatch the only actor is the
|
|
1677
|
+
orchestrator, which § Roles makes the same context as the implementer, and the
|
|
1678
|
+
seats are sandboxed read-only and cannot execute anything. Stated as a property
|
|
1679
|
+
of evidence rather than a duty on someone, it holds wherever a mutation result
|
|
1680
|
+
is read. The bound the operator asked for is not here and the omission is open,
|
|
1681
|
+
not settled: the recorded decision is to accumulate three runs and then judge.
|
|
1682
|
+
**Withdrawn in this version**: a rule making each seat label its own findings
|
|
1683
|
+
new or carryover. A seat sees only its own priors, so a finding one seat raises
|
|
1684
|
+
after another raised it is new to that seat and carryover to the round, and a
|
|
1685
|
+
seat added or returning mid-thread has no priors at all. § Convergence Rules
|
|
1686
|
+
now records that who makes the label is undecided and that the orchestrator
|
|
1687
|
+
makes it meanwhile, which is the biased judge the withdrawn rule was written to
|
|
1688
|
+
replace. A fifth proposed rule, an exception for self-referential reviews, was
|
|
1689
|
+
dropped before this version was written, after four reviewer contexts —
|
|
1690
|
+
occupying two of that round's three counted seats — shot it on three
|
|
1691
|
+
independent grounds. Design history: three review rounds on the proposal
|
|
1692
|
+
(closed by operator declaration, not convergence), then one round on the
|
|
1693
|
+
applied text, which returned one APPROVE of four seats and eighteen blocking
|
|
1694
|
+
findings — every rule above is narrower for it. What kept being shot across all
|
|
1695
|
+
four rounds was the change's account of itself, not the rules it proposed.
|
|
1696
|
+
Records: L2 `handoff_mlr_l1_norms_revision_three_rounds_and_switch_to_implementation_20260821`
|
|
1697
|
+
and L2 `mlr_v3_11_0_applied_review_r1_and_subtractive_revision_20260822`
|
|
1698
|
+
|
|
1614
1699
|
**Key insight**: Design reviews and implementation reviews find
|
|
1615
1700
|
**categorically different bugs**. Both phases are necessary.
|
|
@@ -31,6 +31,31 @@ module KairosHookProjector
|
|
|
31
31
|
TAIL_BYTES = 512 * 1024
|
|
32
32
|
POLL_ATTEMPTS = 15
|
|
33
33
|
POLL_DELAY = 0.1
|
|
34
|
+
# The recheck waits longer than the first read, because it is waiting for a
|
|
35
|
+
# whole rewrite rather than for the tail of a record already being written.
|
|
36
|
+
# Bounded well under the hook's own timeout; the turn it lengthens is one
|
|
37
|
+
# that already spent a full regeneration.
|
|
38
|
+
RECHECK_POLL_ATTEMPTS = 40
|
|
39
|
+
# Claude Code's own wording, written into the transcript as a user record
|
|
40
|
+
# when a Stop hook blocks. Not this gate's string: if it changes, no marker
|
|
41
|
+
# is found and the read falls back to the pre-2026-08-20 behaviour, which
|
|
42
|
+
# the verdict names so the drift shows up in the log instead of silently.
|
|
43
|
+
BLOCK_MARKER = 'Stop hook feedback:'
|
|
44
|
+
# Claude Code's limit on how long a Stop hook may run, declared twice
|
|
45
|
+
# outside this file: in `.claude/settings.json` and in this SkillSet's
|
|
46
|
+
# `lib/mode_hooks_compiler.rb`. Named here because this file spends two
|
|
47
|
+
# budgets in sequence and neither knows about the other. A hook killed at
|
|
48
|
+
# the limit emits nothing at all — no verdict, no banner, no log line — so
|
|
49
|
+
# the gate stops enforcing and the measurement loses rows, both silently.
|
|
50
|
+
HOOK_TIMEOUT = 10.0
|
|
51
|
+
# Held back from the pair for interpreter start-up and for the poll's own
|
|
52
|
+
# overhead, which measures 0.2-0.6s above the sleeps it accounts for.
|
|
53
|
+
HOOK_TIMEOUT_MARGIN = 1.0
|
|
54
|
+
# What is left for measurement once the recheck has spent its budget. A mode
|
|
55
|
+
# asking for more than this is honoured up to here and no further: the
|
|
56
|
+
# alternative is a mode file being able to silence the gate by arithmetic.
|
|
57
|
+
MEASURE_TIMEOUT_CEILING =
|
|
58
|
+
HOOK_TIMEOUT - HOOK_TIMEOUT_MARGIN - (RECHECK_POLL_ATTEMPTS * POLL_DELAY)
|
|
34
59
|
|
|
35
60
|
# Built from single-quoted strings so that `#{` in a pattern stays literal
|
|
36
61
|
# rather than becoming interpolation, and anchored with \A because Python's
|
|
@@ -131,7 +156,18 @@ module KairosHookProjector
|
|
|
131
156
|
# list. Text explaining the vocabulary rule has to name the shapes it
|
|
132
157
|
# governs, and naming them is not using them.
|
|
133
158
|
@specimen = merged['specimen_patterns'].map { |p| Regexp.new(p) }
|
|
134
|
-
|
|
159
|
+
# Clamped, not rejected: a mode that asks for 6 seconds gets what fits
|
|
160
|
+
# rather than a refusal it cannot act on. 4 seconds of recheck poll plus
|
|
161
|
+
# 6 of measurement is 10.4 against a 10-second hook limit, and the hook
|
|
162
|
+
# dies with no output, so the gate would silently stop enforcing for
|
|
163
|
+
# exactly the modes whose patterns are expensive enough to need the time.
|
|
164
|
+
requested = merged['measure_timeout_seconds']
|
|
165
|
+
@measure_timeout =
|
|
166
|
+
if requested.is_a?(Numeric) && requested > MEASURE_TIMEOUT_CEILING
|
|
167
|
+
MEASURE_TIMEOUT_CEILING
|
|
168
|
+
else
|
|
169
|
+
requested
|
|
170
|
+
end
|
|
135
171
|
@log_max_bytes = merged['log_max_bytes']
|
|
136
172
|
end
|
|
137
173
|
|
|
@@ -275,32 +311,168 @@ module KairosHookProjector
|
|
|
275
311
|
joined.empty? ? nil : joined
|
|
276
312
|
end
|
|
277
313
|
|
|
314
|
+
# Index of the marker for THIS turn, or nil while it has not landed.
|
|
315
|
+
#
|
|
316
|
+
# The transcript records the block itself: a user record carrying Claude
|
|
317
|
+
# Code's feedback text, whose parentUuid names the assistant record that was
|
|
318
|
+
# judged. That pair is the entire state the recheck needs — which record has
|
|
319
|
+
# already had a verdict, and where the rewrite begins — so no state file and
|
|
320
|
+
# no config key are involved.
|
|
321
|
+
#
|
|
322
|
+
# The newest marker in the transcript is not necessarily this turn's, and
|
|
323
|
+
# taking it was a defect of its own. A session blocked once already carries
|
|
324
|
+
# an older marker; on the next block, before the new marker lands, the older
|
|
325
|
+
# one made the read believe a marker had arrived, so it skipped the wait and
|
|
326
|
+
# measured the message it had just blocked — returning in 0.06s with nothing
|
|
327
|
+
# in the log to say so. 128 of 170 real blocks are not a session's first.
|
|
328
|
+
# Between the previous block and this one there is always the operator's own
|
|
329
|
+
# message, so the test is which of the two is newer. Records carrying no
|
|
330
|
+
# text — tool results, images — belong to the turn being written and are
|
|
331
|
+
# stepped over.
|
|
332
|
+
def turn_marker(rows)
|
|
333
|
+
i = rows.length - 1
|
|
334
|
+
while i >= 0
|
|
335
|
+
row = rows[i]
|
|
336
|
+
if row['type'] == 'user'
|
|
337
|
+
text = text_of(row)
|
|
338
|
+
return text.start_with?(BLOCK_MARKER) ? i : nil if text.is_a?(String)
|
|
339
|
+
end
|
|
340
|
+
|
|
341
|
+
i -= 1
|
|
342
|
+
end
|
|
343
|
+
nil
|
|
344
|
+
end
|
|
345
|
+
|
|
346
|
+
# Compared only when a judged uuid is present: comparing it while nil made a
|
|
347
|
+
# record carrying no uuid look like the judged record and skipped a real
|
|
348
|
+
# rewrite.
|
|
349
|
+
def judged?(row, judged)
|
|
350
|
+
judged && row['uuid'] == judged
|
|
351
|
+
end
|
|
352
|
+
|
|
353
|
+
# The newest assistant record after `index` other than the judged one,
|
|
354
|
+
# whether or not it carries text yet. nil while none has landed.
|
|
355
|
+
#
|
|
356
|
+
# This is the first read's rule applied after the marker, and it is what the
|
|
357
|
+
# recheck uses while it still has budget. A rewrite that is mid-write shows
|
|
358
|
+
# up here as text-less and is waited for, rather than stepped over in favour
|
|
359
|
+
# of whatever text lies beneath it. Stepping over had a measured cost: a
|
|
360
|
+
# rewrite that called a tool was judged on its own two-line preamble and
|
|
361
|
+
# logged RECHECK-PASS in 0.04s, while the rewrite itself — a four-heading
|
|
362
|
+
# FAIL landing 0.6s later — was never read.
|
|
363
|
+
def newest_after(rows, index, judged)
|
|
364
|
+
i = rows.length - 1
|
|
365
|
+
while i > index
|
|
366
|
+
row = rows[i]
|
|
367
|
+
return [text_of(row), row['uuid']] if row['type'] == 'assistant' && !judged?(row, judged)
|
|
368
|
+
|
|
369
|
+
i -= 1
|
|
370
|
+
end
|
|
371
|
+
nil
|
|
372
|
+
end
|
|
373
|
+
|
|
374
|
+
# The newest text-bearing assistant record after `index`, stepping over
|
|
375
|
+
# records that carry none. Used only once the budget is spent.
|
|
376
|
+
#
|
|
377
|
+
# Over 3,248 transcripts and 170 real blocks, a completed rewrite sitting
|
|
378
|
+
# beneath a newer text-less record occurred 0 times, so this shape must not
|
|
379
|
+
# pre-empt the wait — that is what cost the preamble above. It is kept as a
|
|
380
|
+
# last resort because if the shape ever does occur, the alternative is the
|
|
381
|
+
# same skipped verdict this reaches past.
|
|
382
|
+
def deep_after(rows, index, judged)
|
|
383
|
+
i = rows.length - 1
|
|
384
|
+
while i > index
|
|
385
|
+
row = rows[i]
|
|
386
|
+
if row['type'] == 'assistant' && !judged?(row, judged)
|
|
387
|
+
text = text_of(row)
|
|
388
|
+
return [text, row['uuid']] if text
|
|
389
|
+
end
|
|
390
|
+
|
|
391
|
+
i -= 1
|
|
392
|
+
end
|
|
393
|
+
nil
|
|
394
|
+
end
|
|
395
|
+
|
|
396
|
+
# The newest assistant record, whether or not it carries text yet, or nil
|
|
397
|
+
# when the transcript holds none. The first read's rule, unchanged since
|
|
398
|
+
# before 2026-08-20, and the rule the recheck falls back to when no marker
|
|
399
|
+
# is found.
|
|
400
|
+
def newest_assistant(rows)
|
|
401
|
+
row = rows.reverse_each.find { |r| r['type'] == 'assistant' }
|
|
402
|
+
row && [text_of(row), row['uuid']]
|
|
403
|
+
end
|
|
404
|
+
|
|
278
405
|
# Text of the turn's final assistant message, with the flush race handled.
|
|
279
406
|
#
|
|
280
407
|
# One response is written as several records (thinking, text, tool_use) at
|
|
281
408
|
# different times. At Stop time the `text` record may not have landed yet,
|
|
282
409
|
# so the newest assistant record is often thinking-only. Wait for the text
|
|
283
410
|
# rather than judging an earlier message from the same turn.
|
|
284
|
-
|
|
285
|
-
|
|
411
|
+
#
|
|
412
|
+
# On a recheck that wait never engaged, and that was the defect: the blocked
|
|
413
|
+
# message is still the newest record carrying text, so it satisfied the
|
|
414
|
+
# "has text" test immediately and was judged a second time. Measured over
|
|
415
|
+
# one instance's first 768 log records, 140 of 140 rechecks took the newest
|
|
416
|
+
# record without ever waiting, and 109 of 140 reported metrics identical to
|
|
417
|
+
# the verdict that had just blocked. The rewrite had passed in most of them.
|
|
418
|
+
# The invariant this restores, as long as the marker is found: a record that
|
|
419
|
+
# a verdict already named is never judged again.
|
|
420
|
+
def last_assistant_text(transcript_path, rechecked = false)
|
|
421
|
+
attempts = rechecked ? RECHECK_POLL_ATTEMPTS : POLL_ATTEMPTS
|
|
422
|
+
attempts.times do |attempt|
|
|
286
423
|
rows = tail_records(transcript_path)
|
|
287
|
-
return [nil, 'unreadable'] if rows.nil?
|
|
424
|
+
return [nil, 'unreadable', nil] if rows.nil?
|
|
425
|
+
|
|
426
|
+
waited = attempt.zero? ? 'ok' : 'ok-after-wait'
|
|
427
|
+
|
|
428
|
+
if rechecked
|
|
429
|
+
# The marker is waited for on the same terms as the rewrite: both are
|
|
430
|
+
# written by the same process at nearly the same moment, so "not there
|
|
431
|
+
# yet" is as ordinary for one as for the other. Inside the budget the
|
|
432
|
+
# selection rule is the first read's — newest record, wait for its
|
|
433
|
+
# text — so the two reads agree wherever they can.
|
|
434
|
+
marker = turn_marker(rows)
|
|
435
|
+
if marker
|
|
436
|
+
found = newest_after(rows, marker, rows[marker]['parentUuid'])
|
|
437
|
+
return [found[0], waited, found[1]] if found && found[0]
|
|
438
|
+
end
|
|
439
|
+
else
|
|
440
|
+
newest = newest_assistant(rows)
|
|
441
|
+
return [nil, 'no-assistant-record', nil] if newest.nil?
|
|
442
|
+
return [newest[0], waited, newest[1]] if newest[0]
|
|
443
|
+
end
|
|
444
|
+
sleep(POLL_DELAY)
|
|
445
|
+
end
|
|
288
446
|
|
|
289
|
-
|
|
290
|
-
rows.reverse_each do |row|
|
|
291
|
-
next unless row['type'] == 'assistant'
|
|
447
|
+
return [nil, 'race-timeout', nil] unless rechecked
|
|
292
448
|
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
449
|
+
# The recheck's budget is spent. Either this turn's marker never appeared
|
|
450
|
+
# — Claude Code's wording is not this gate's to guarantee — or it appeared
|
|
451
|
+
# and the rewrite did not. Both are named, and every no-marker outcome
|
|
452
|
+
# carries the suffix, because a drift detector whose only visible trace is
|
|
453
|
+
# the exit that happens to find text reports nothing in the cases drift
|
|
454
|
+
# actually produces.
|
|
455
|
+
rows = tail_records(transcript_path)
|
|
456
|
+
return [nil, 'unreadable', nil] if rows.nil?
|
|
296
457
|
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
458
|
+
marker = turn_marker(rows)
|
|
459
|
+
if marker
|
|
460
|
+
# Last resort, and named apart so the log can say how often the ordinary
|
|
461
|
+
# rule was not enough.
|
|
462
|
+
found = deep_after(rows, marker, rows[marker]['parentUuid'])
|
|
463
|
+
return [found[0], 'ok-after-wait-deep', found[1]] if found
|
|
300
464
|
|
|
301
|
-
|
|
465
|
+
return [nil, 'awaiting-rewrite', nil]
|
|
302
466
|
end
|
|
303
|
-
|
|
467
|
+
|
|
468
|
+
# No marker for this turn, so nothing is judged. Falling back to the first
|
|
469
|
+
# read's rule here looked conservative and was the opposite: the newest
|
|
470
|
+
# record it reaches is the message this turn has just blocked, so the
|
|
471
|
+
# fallback re-judged it — the very behaviour this method exists to end,
|
|
472
|
+
# reachable on any block after a session's first. A recheck never blocks,
|
|
473
|
+
# so refusing to judge costs the operator nothing except the line, and it
|
|
474
|
+
# keeps both the invariant and the measurement intact.
|
|
475
|
+
newest_assistant(rows) ? [nil, 'nomarker', nil] : [nil, 'no-assistant-record-nomarker', nil]
|
|
304
476
|
end
|
|
305
477
|
|
|
306
478
|
# --- measurement ---------------------------------------------------------
|
|
@@ -460,15 +632,20 @@ module KairosHookProjector
|
|
|
460
632
|
# have — an empty-string or directory log_path left the gate blocking and
|
|
461
633
|
# passing exactly as normal while nothing was ever recorded, so the
|
|
462
634
|
# operator's onboarding week produced no data and no complaint.
|
|
463
|
-
def note(cfg, verdict, metrics = nil)
|
|
635
|
+
def note(cfg, verdict, metrics = nil, record_id = nil)
|
|
464
636
|
# nil alone means "no log declared". false, 0 and "" are declarations the
|
|
465
637
|
# gate cannot honour, and each has to reach the rescue below to be named.
|
|
466
638
|
return if cfg.log_path.nil?
|
|
467
639
|
|
|
468
640
|
stamp = Time.now.strftime('%Y-%m-%dT%H:%M:%S')
|
|
469
641
|
detail = ''
|
|
642
|
+
# Which record the verdict is about. Without it the log cannot answer the
|
|
643
|
+
# one question the fix above turns on — whether two verdicts in a row were
|
|
644
|
+
# about the same message — and it is also how a first read that grabbed a
|
|
645
|
+
# stale record becomes diagnosable rather than merely suspected.
|
|
646
|
+
detail += "\trec=#{record_id[0, 8]}" if record_id.is_a?(String) && !record_id.empty?
|
|
470
647
|
if metrics
|
|
471
|
-
detail
|
|
648
|
+
detail += format(
|
|
472
649
|
"\tlines=%d\theadings=%d\ttables=%d\tdiagrams=%d\tunglossed=%s",
|
|
473
650
|
metrics['lines'], metrics['headings'], metrics['tables'], metrics['diagrams'],
|
|
474
651
|
metrics['unglossed'].empty? ? '-' : metrics['unglossed'].join(',')
|
|
@@ -643,9 +820,31 @@ module KairosHookProjector
|
|
|
643
820
|
# reported, so its outcome is visible; it is simply never blocked again.
|
|
644
821
|
rechecked = payload['stop_hook_active'] ? true : false
|
|
645
822
|
|
|
646
|
-
text, why = last_assistant_text(payload.fetch('transcript_path', ''))
|
|
823
|
+
text, why, record_id = last_assistant_text(payload.fetch('transcript_path', ''), rechecked)
|
|
647
824
|
if text.nil? || text.strip.empty?
|
|
648
|
-
note(cfg, "SKIP-#{why}")
|
|
825
|
+
lost = note(cfg, "SKIP-#{why}")
|
|
826
|
+
# Every recheck that produces no verdict says so on screen. Naming only
|
|
827
|
+
# the two expected reasons left the others silent, and the ones it left
|
|
828
|
+
# out are live: a transcript momentarily unreadable, and a rewrite whose
|
|
829
|
+
# text is whitespace after the fallback found it. Both spent the budget
|
|
830
|
+
# and told the operator nothing. A first read keeps its old silence —
|
|
831
|
+
# there the turn goes out regardless and the log is the right place.
|
|
832
|
+
budget = format('%.1f', RECHECK_POLL_ATTEMPTS * POLL_DELAY)
|
|
833
|
+
tail = lost ? "; #{log_note(lost)}" : ''
|
|
834
|
+
reason =
|
|
835
|
+
case why
|
|
836
|
+
when 'awaiting-rewrite'
|
|
837
|
+
"the rewrite had not been recorded within #{budget}s"
|
|
838
|
+
when 'nomarker', 'no-assistant-record-nomarker'
|
|
839
|
+
"no block marker appeared within #{budget}s, so the rewrite could not be told " \
|
|
840
|
+
"apart from the message that was blocked. Claude Code's feedback wording may " \
|
|
841
|
+
'have changed'
|
|
842
|
+
when 'unreadable'
|
|
843
|
+
'the transcript could not be read'
|
|
844
|
+
else
|
|
845
|
+
"nothing measurable was found (#{why})"
|
|
846
|
+
end
|
|
847
|
+
emit('systemMessage' => "#{cfg.banner_prefix} (recheck): NOT RUN — #{reason}#{tail}") if rechecked
|
|
649
848
|
return 0
|
|
650
849
|
end
|
|
651
850
|
|
|
@@ -663,7 +862,7 @@ module KairosHookProjector
|
|
|
663
862
|
end
|
|
664
863
|
|
|
665
864
|
verdict = failures.empty? ? 'PASS' : 'FAIL'
|
|
666
|
-
lost = note(cfg, "#{rechecked ? 'RECHECK-' : ''}#{verdict}-#{why}", metrics)
|
|
865
|
+
lost = note(cfg, "#{rechecked ? 'RECHECK-' : ''}#{verdict}-#{why}", metrics, record_id)
|
|
667
866
|
|
|
668
867
|
out = { 'systemMessage' => banner(cfg, verdict, metrics, failures, rechecked, lost) }
|
|
669
868
|
if !failures.empty? && !rechecked && cfg.blocking
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# Mutation check for readable_gate.
|
|
3
|
+
#
|
|
4
|
+
# A green suite proves nothing until something has tried to break the code and
|
|
5
|
+
# the suite noticed. Each entry below rewrites one line of the hook and asserts
|
|
6
|
+
# that at least one test goes red. A SURVIVED row means the behaviour it names
|
|
7
|
+
# is not witnessed by any test, however many assertions surround it.
|
|
8
|
+
#
|
|
9
|
+
# Why this file lives in the SkillSet rather than beside a review record: the
|
|
10
|
+
# first round of this check was written to a scratch directory and was gone by
|
|
11
|
+
# the time anyone tried to reproduce it, so the evidence for "the falsification
|
|
12
|
+
# is real" could not be re-derived. The generator is the artifact worth keeping;
|
|
13
|
+
# its output is reproducible from it.
|
|
14
|
+
#
|
|
15
|
+
# Runs against a COPY of the whole SkillSet in a temp directory. The repository
|
|
16
|
+
# copy may be a live Stop hook, and must never be left mutated even briefly.
|
|
17
|
+
# The harness is this SkillSet's own suite, unmodified: `SCRIPT` in the test
|
|
18
|
+
# files resolves from `__dir__`, so the copied tests drive the copied hook.
|
|
19
|
+
#
|
|
20
|
+
# ruby test/mutation_check_readable_gate.rb
|
|
21
|
+
#
|
|
22
|
+
require 'fileutils'
|
|
23
|
+
require 'tmpdir'
|
|
24
|
+
require 'open3'
|
|
25
|
+
require 'rbconfig'
|
|
26
|
+
|
|
27
|
+
SKILLSET_ROOT = File.dirname(__dir__)
|
|
28
|
+
|
|
29
|
+
MUTATIONS = [
|
|
30
|
+
# --- the recheck's target: what it measures the second time ----------------
|
|
31
|
+
['M1 recheck ignores the marker and takes the newest record',
|
|
32
|
+
%q{ marker = turn_marker(rows)},
|
|
33
|
+
%q{ marker = nil}],
|
|
34
|
+
['M2 the already-judged record is not excluded (inside the budget)',
|
|
35
|
+
%q{ return [text_of(row), row['uuid']] if row['type'] == 'assistant' && !judged?(row, judged)},
|
|
36
|
+
%q{ return [text_of(row), row['uuid']] if row['type'] == 'assistant'}],
|
|
37
|
+
['M3 the recheck does not wait',
|
|
38
|
+
%q{ attempts = rechecked ? RECHECK_POLL_ATTEMPTS : POLL_ATTEMPTS},
|
|
39
|
+
%q{ attempts = rechecked ? 1 : POLL_ATTEMPTS}],
|
|
40
|
+
['M4 the no-marker exit claims there is no assistant record at all',
|
|
41
|
+
%q{ newest_assistant(rows) ? [nil, 'nomarker', nil] : [nil, 'no-assistant-record-nomarker', nil]},
|
|
42
|
+
%q{ [nil, 'no-assistant-record-nomarker', nil]}],
|
|
43
|
+
['M5 the -nomarker suffix is dropped',
|
|
44
|
+
%q{ newest_assistant(rows) ? [nil, 'nomarker', nil] : [nil, 'no-assistant-record-nomarker', nil]},
|
|
45
|
+
%q{ newest_assistant(rows) ? [nil, 'awaiting-rewrite', nil] : [nil, 'no-assistant-record', nil]}],
|
|
46
|
+
['M6 the rec= column is dropped from the log',
|
|
47
|
+
%q{ detail += "\trec=#{record_id[0, 8]}" if record_id.is_a?(String) && !record_id.empty?},
|
|
48
|
+
%q{ detail += ''}],
|
|
49
|
+
['M7 the recheck blocks as well as the first read',
|
|
50
|
+
%q{ if !failures.empty? && !rechecked && cfg.blocking},
|
|
51
|
+
%q{ if !failures.empty? && cfg.blocking}],
|
|
52
|
+
|
|
53
|
+
# --- which record is the marker, and how long it is waited for ------------
|
|
54
|
+
['M8 an earlier turn\'s marker is accepted as this turn\'s',
|
|
55
|
+
%q{ return text.start_with?(BLOCK_MARKER) ? i : nil if text.is_a?(String)},
|
|
56
|
+
%q{ return i if text.is_a?(String) && text.start_with?(BLOCK_MARKER)}],
|
|
57
|
+
['M9 the recheck budget is cut from 40 attempts to 15',
|
|
58
|
+
%q{ RECHECK_POLL_ATTEMPTS = 40},
|
|
59
|
+
%q{ RECHECK_POLL_ATTEMPTS = 15}],
|
|
60
|
+
['M10 rec= logs the whole uuid instead of the first 8 characters',
|
|
61
|
+
%q{ detail += "\trec=#{record_id[0, 8]}" if record_id.is_a?(String) && !record_id.empty?},
|
|
62
|
+
%q{ detail += "\trec=#{record_id}" if record_id.is_a?(String) && !record_id.empty?}],
|
|
63
|
+
['M11 any record type may be a marker, not only user',
|
|
64
|
+
" if row['type'] == 'user'\n text = text_of(row)",
|
|
65
|
+
" if row['type']\n text = text_of(row)"],
|
|
66
|
+
['M12 a record that merely mentions the wording counts as a marker',
|
|
67
|
+
%q{ return text.start_with?(BLOCK_MARKER) ? i : nil if text.is_a?(String)},
|
|
68
|
+
%q{ return text.include?(BLOCK_MARKER) ? i : nil if text.is_a?(String)}],
|
|
69
|
+
|
|
70
|
+
# --- the shape settled on 2026-08-21 after two review rounds ---------------
|
|
71
|
+
['M13 no marker means judge the newest record anyway',
|
|
72
|
+
%q{ newest_assistant(rows) ? [nil, 'nomarker', nil] : [nil, 'no-assistant-record-nomarker', nil]},
|
|
73
|
+
" n = newest_assistant(rows)\n" \
|
|
74
|
+
" return [nil, 'no-assistant-record-nomarker', nil] if n.nil?\n" \
|
|
75
|
+
" n[0] ? [n[0], 'ok-after-wait-nomarker', n[1]] : [nil, 'nomarker', nil]"],
|
|
76
|
+
['M14 the deep walk pre-empts the wait instead of outliving it',
|
|
77
|
+
%q{ return [text_of(row), row['uuid']] if row['type'] == 'assistant' && !judged?(row, judged)},
|
|
78
|
+
" if row['type'] == 'assistant' && !judged?(row, judged)\n" \
|
|
79
|
+
" t = text_of(row)\n" \
|
|
80
|
+
" return [t, row['uuid']] if t\n" \
|
|
81
|
+
" end"],
|
|
82
|
+
['M15 the nil-judged guard reverted to a plain inequality',
|
|
83
|
+
%q{ judged && row['uuid'] == judged},
|
|
84
|
+
%q{ row['uuid'] == judged}],
|
|
85
|
+
|
|
86
|
+
# --- what round 2 found no mutation could reach ---------------------------
|
|
87
|
+
['M16 the no-verdict banner is not emitted at all',
|
|
88
|
+
%q{ emit('systemMessage' => "#{cfg.banner_prefix} (recheck): NOT RUN — #{reason}#{tail}") if rechecked},
|
|
89
|
+
%q{ nil}],
|
|
90
|
+
['M17 the banner quotes a fixed number instead of the budget it spent',
|
|
91
|
+
%q{ budget = format('%.1f', RECHECK_POLL_ATTEMPTS * POLL_DELAY)},
|
|
92
|
+
%q{ budget = '1.5'}],
|
|
93
|
+
['M18 the deep walk never finds the rewrite it is the last resort for',
|
|
94
|
+
" def deep_after(rows, index, judged)\n i = rows.length - 1",
|
|
95
|
+
" def deep_after(rows, index, judged)\n return nil\n i = rows.length - 1"],
|
|
96
|
+
['M19 a mode may ask for more measurement time than the hook has left',
|
|
97
|
+
%q{ if requested.is_a?(Numeric) && requested > MEASURE_TIMEOUT_CEILING},
|
|
98
|
+
%q{ if false}]
|
|
99
|
+
].freeze
|
|
100
|
+
|
|
101
|
+
def run_suite(root)
|
|
102
|
+
files = Dir[File.join(root, 'test', 'test_*.rb')].sort
|
|
103
|
+
runs = 0
|
|
104
|
+
bad = 0
|
|
105
|
+
unparseable = []
|
|
106
|
+
files.each do |f|
|
|
107
|
+
out, err, st = Open3.capture3(RbConfig.ruby, f)
|
|
108
|
+
tail = "#{out}\n#{err}"
|
|
109
|
+
if (m = tail.match(/(\d+) runs, \d+ assertions, (\d+) failures, (\d+) errors/))
|
|
110
|
+
runs += m[1].to_i
|
|
111
|
+
bad += m[2].to_i + m[3].to_i
|
|
112
|
+
else
|
|
113
|
+
unparseable << File.basename(f)
|
|
114
|
+
bad += 1 unless st.success?
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
[runs, bad, unparseable, files.length]
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
exit_code = 0
|
|
121
|
+
|
|
122
|
+
Dir.mktmpdir do |tmp|
|
|
123
|
+
root = File.join(tmp, File.basename(SKILLSET_ROOT))
|
|
124
|
+
FileUtils.cp_r(SKILLSET_ROOT, root)
|
|
125
|
+
hook = File.join(root, 'hooks', 'readable_gate.rb')
|
|
126
|
+
pristine = File.read(hook)
|
|
127
|
+
|
|
128
|
+
runs, bad, unparseable, nfiles = run_suite(root)
|
|
129
|
+
puts format('BASELINE (unmutated copy): %d test files, %d runs, %d failures+errors%s',
|
|
130
|
+
nfiles, runs, bad, unparseable.empty? ? '' : " [unparseable: #{unparseable.join(', ')}]")
|
|
131
|
+
puts
|
|
132
|
+
if bad.positive?
|
|
133
|
+
puts 'The suite is not green before mutation; every result below is meaningless.'
|
|
134
|
+
exit_code = 1
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
killed = 0
|
|
138
|
+
survived = 0
|
|
139
|
+
not_applied = 0
|
|
140
|
+
MUTATIONS.each do |label, from, to|
|
|
141
|
+
unless pristine.include?(from)
|
|
142
|
+
not_applied += 1
|
|
143
|
+
puts format('%-70s ANCHOR NOT FOUND', label)
|
|
144
|
+
next
|
|
145
|
+
end
|
|
146
|
+
mutated = pristine.sub(from, to)
|
|
147
|
+
raise "mutation #{label} was a no-op" if mutated == pristine
|
|
148
|
+
|
|
149
|
+
File.write(hook, mutated)
|
|
150
|
+
r, b, u, = run_suite(root)
|
|
151
|
+
File.write(hook, pristine)
|
|
152
|
+
if b.positive?
|
|
153
|
+
killed += 1
|
|
154
|
+
verdict = 'KILLED'
|
|
155
|
+
else
|
|
156
|
+
survived += 1
|
|
157
|
+
verdict = 'SURVIVED'
|
|
158
|
+
end
|
|
159
|
+
puts format('%-70s %-9s %3d failures/errors over %d runs%s',
|
|
160
|
+
label, verdict, b, r, u.empty? ? '' : " [unparseable: #{u.join(', ')}]")
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
puts
|
|
164
|
+
puts "source restored byte-identical: #{File.read(hook) == pristine}"
|
|
165
|
+
puts "killed #{killed}/#{MUTATIONS.length}, survived #{survived}, anchor-not-found #{not_applied}"
|
|
166
|
+
exit_code = 1 if survived.positive? || not_applied.positive?
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
exit exit_code
|
|
@@ -277,11 +277,415 @@ class TestReadableGate < Minitest::Test
|
|
|
277
277
|
assert_equal 'block', out['decision'], out.inspect
|
|
278
278
|
end
|
|
279
279
|
|
|
280
|
+
# A failing rewrite is still measured and still reported; it is simply never
|
|
281
|
+
# blocked a second time. The marker has to be present for a verdict to be
|
|
282
|
+
# issued at all — see "a recheck with no marker for this turn issues no
|
|
283
|
+
# verdict" below for why the marker-less case stopped producing one.
|
|
280
284
|
def test_a_rewrite_is_measured_and_reported_but_never_blocked_again
|
|
281
|
-
out =
|
|
285
|
+
out, log = drive(blocked_then(row_for('assistant', text: "# a\n## b\n### c\n#### d\n",
|
|
286
|
+
uuid: 'BBB')),
|
|
287
|
+
max_headings: 3)
|
|
282
288
|
refute out.key?('decision'), out.inspect
|
|
283
289
|
assert_includes out.fetch('systemMessage', ''), 'FAIL'
|
|
284
290
|
assert_includes out.fetch('systemMessage', ''), 'recheck'
|
|
291
|
+
assert_includes log, 'RECHECK-FAIL', log.inspect
|
|
292
|
+
end
|
|
293
|
+
|
|
294
|
+
# --- the recheck must judge the rewrite, not what it already judged --------
|
|
295
|
+
#
|
|
296
|
+
# Until 2026-08-20 it judged what it had already judged. The blocked message
|
|
297
|
+
# is still the newest record carrying text when the recheck runs, so the
|
|
298
|
+
# existing wait — which only engages when the newest record has no text —
|
|
299
|
+
# never engaged: over one instance's first 768 log records, 140 of 140
|
|
300
|
+
# rechecks took the newest record immediately, and 109 of 140 reported
|
|
301
|
+
# metrics identical to the verdict that had just blocked. Re-measuring 145
|
|
302
|
+
# real rewrites from the transcripts showed 101 of them passing while the log
|
|
303
|
+
# recorded RECHECK-FAIL. The two sweeps are two hours apart and are not one
|
|
304
|
+
# evidence base.
|
|
305
|
+
|
|
306
|
+
def row_for(type, text: nil, uuid: nil, parent: nil, thinking: false)
|
|
307
|
+
content =
|
|
308
|
+
if thinking then [{ 'type' => 'thinking', 'thinking' => 'x' }]
|
|
309
|
+
elsif type == 'user' then text
|
|
310
|
+
else [{ 'type' => 'text', 'text' => text }]
|
|
311
|
+
end
|
|
312
|
+
row = { 'type' => type, 'message' => { 'content' => content } }
|
|
313
|
+
row['uuid'] = uuid if uuid
|
|
314
|
+
row['parentUuid'] = parent if parent
|
|
315
|
+
row
|
|
316
|
+
end
|
|
317
|
+
|
|
318
|
+
def rows_json(rows)
|
|
319
|
+
rows.map { |r| JSON.generate(r) }.join("\n") + "\n"
|
|
320
|
+
end
|
|
321
|
+
|
|
322
|
+
# Drive the real script over a hand-built transcript and hand back both the
|
|
323
|
+
# emitted object and the log. The log is what carries `rec=`, and a verdict
|
|
324
|
+
# that names no record cannot be checked for naming the right one.
|
|
325
|
+
def drive(rows, rechecked: true, **overrides)
|
|
326
|
+
Dir.mktmpdir do |tmp|
|
|
327
|
+
cfg_path = File.join(tmp, 'cfg.json')
|
|
328
|
+
tx_path = File.join(tmp, 't.jsonl')
|
|
329
|
+
log_path = File.join(tmp, 'gate.log')
|
|
330
|
+
raw = { 'mode_name' => 'test', 'section' => '§ Test', 'log_path' => log_path }
|
|
331
|
+
.merge(overrides.transform_keys(&:to_s))
|
|
332
|
+
File.write(cfg_path, JSON.generate(raw), encoding: 'UTF-8')
|
|
333
|
+
File.write(tx_path, rows_json(rows), encoding: 'UTF-8')
|
|
334
|
+
out, err, status = run_script(
|
|
335
|
+
cfg_path,
|
|
336
|
+
JSON.generate('transcript_path' => tx_path, 'stop_hook_active' => rechecked)
|
|
337
|
+
)
|
|
338
|
+
assert_equal 0, status.exitstatus, "gate exited #{status.exitstatus}: #{err[0, 300]}"
|
|
339
|
+
log = File.exist?(log_path) ? File.read(log_path, encoding: 'UTF-8') : ''
|
|
340
|
+
[out.strip.empty? ? {} : JSON.parse(out), log]
|
|
341
|
+
end
|
|
342
|
+
end
|
|
343
|
+
|
|
344
|
+
BLOCKED = "# a\n# b\n# c\n# d\n"
|
|
345
|
+
REWRITE = "# a\n"
|
|
346
|
+
|
|
347
|
+
def blocked_then(*after)
|
|
348
|
+
[row_for('assistant', text: BLOCKED, uuid: 'AAA'),
|
|
349
|
+
row_for('user', text: "Stop hook feedback:\n- HEADINGS: 4 (cap 3).", parent: 'AAA',
|
|
350
|
+
uuid: 'MMM')] + after
|
|
351
|
+
end
|
|
352
|
+
|
|
353
|
+
def test_the_recheck_judges_the_rewrite_not_the_message_it_already_judged
|
|
354
|
+
out, log = drive(blocked_then(row_for('assistant', text: REWRITE, uuid: 'BBB')),
|
|
355
|
+
max_headings: 3)
|
|
356
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS', out.inspect
|
|
357
|
+
assert_includes out.fetch('systemMessage', ''), '1 headings',
|
|
358
|
+
'the rewrite has one heading; the blocked message had four'
|
|
359
|
+
assert_includes log, 'rec=BBB', "the verdict must name the rewrite: #{log.inspect}"
|
|
360
|
+
refute out.key?('decision')
|
|
361
|
+
end
|
|
362
|
+
|
|
363
|
+
def test_a_recheck_whose_rewrite_has_not_landed_records_no_verdict
|
|
364
|
+
out, log = drive(blocked_then, max_headings: 3)
|
|
365
|
+
assert_includes log, 'SKIP-awaiting-rewrite', log.inspect
|
|
366
|
+
refute_includes log, 'RECHECK-FAIL',
|
|
367
|
+
'a verdict on the already-judged record is the defect itself'
|
|
368
|
+
assert_includes out.fetch('systemMessage', ''), 'NOT RUN'
|
|
369
|
+
refute out.key?('decision')
|
|
370
|
+
end
|
|
371
|
+
|
|
372
|
+
def test_the_already_judged_record_is_skipped_even_when_it_is_the_newest
|
|
373
|
+
# Ordering that only a truncated tail or a rewritten transcript produces,
|
|
374
|
+
# and the only fixture that can fail when the uuid guard is deleted: the
|
|
375
|
+
# index check alone would let the judged record through here.
|
|
376
|
+
rows = [row_for('assistant', text: REWRITE, uuid: 'BBB'),
|
|
377
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'AAA', uuid: 'MMM'),
|
|
378
|
+
row_for('assistant', text: BLOCKED, uuid: 'AAA')]
|
|
379
|
+
_out, log = drive(rows, max_headings: 3)
|
|
380
|
+
assert_includes log, 'SKIP-awaiting-rewrite', log.inspect
|
|
381
|
+
end
|
|
382
|
+
|
|
383
|
+
def test_the_newest_marker_is_the_one_that_counts
|
|
384
|
+
rows = [row_for('assistant', text: BLOCKED, uuid: 'AAA'),
|
|
385
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'AAA', uuid: 'M1'),
|
|
386
|
+
row_for('assistant', text: "#{BLOCKED}# e\n", uuid: 'BBB'),
|
|
387
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'BBB', uuid: 'M2'),
|
|
388
|
+
row_for('assistant', text: REWRITE, uuid: 'CCC')]
|
|
389
|
+
out, log = drive(rows, max_headings: 3)
|
|
390
|
+
assert_includes log, 'rec=CCC', log.inspect
|
|
391
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS'
|
|
392
|
+
end
|
|
393
|
+
|
|
394
|
+
def test_a_recheck_with_no_marker_for_this_turn_issues_no_verdict
|
|
395
|
+
# Claude Code's wording is not this gate's to guarantee, and when it changes
|
|
396
|
+
# the drift has to be visible. It must not be papered over by degrading to
|
|
397
|
+
# the old rule: the newest record that rule reaches is the message this turn
|
|
398
|
+
# has just blocked, so degrading meant re-judging it — reachable on any
|
|
399
|
+
# block after a session's first, and logged as an ordinary verdict. A
|
|
400
|
+
# recheck never blocks, so declining to judge costs only the line.
|
|
401
|
+
out, log = drive([row_for('assistant', text: BLOCKED, uuid: 'AAA')], max_headings: 3)
|
|
402
|
+
assert_includes log, 'SKIP-nomarker', log.inspect
|
|
403
|
+
refute_includes log, 'no-assistant-record',
|
|
404
|
+
'there is an assistant record; only the marker is missing, and the two ' \
|
|
405
|
+
'are separate rows in the measurement'
|
|
406
|
+
refute_includes log, 'rec=AAA', 'AAA is the message that was just blocked'
|
|
407
|
+
assert_includes out.fetch('systemMessage', ''), 'NOT RUN'
|
|
408
|
+
refute out.key?('decision')
|
|
409
|
+
end
|
|
410
|
+
|
|
411
|
+
def test_the_first_read_is_unchanged_and_names_the_record_it_judged
|
|
412
|
+
out, log = drive([row_for('assistant', text: BLOCKED, uuid: 'AAA')],
|
|
413
|
+
rechecked: false, max_headings: 3)
|
|
414
|
+
assert_equal 'block', out['decision'], out.inspect
|
|
415
|
+
assert_includes log, 'rec=AAA', log.inspect
|
|
416
|
+
refute_includes log, 'nomarker', 'a first read has no marker to miss'
|
|
417
|
+
end
|
|
418
|
+
|
|
419
|
+
# The recheck's own flush race, driven with a real late write. The rewrite
|
|
420
|
+
# record appears only after the first read has already come back empty.
|
|
421
|
+
def test_the_recheck_waits_for_the_rewrite_to_land
|
|
422
|
+
Dir.mktmpdir do |tmp|
|
|
423
|
+
tx = File.join(tmp, 't.jsonl')
|
|
424
|
+
File.write(tx, rows_json(blocked_then), encoding: 'UTF-8')
|
|
425
|
+
writer = Thread.new do
|
|
426
|
+
sleep(G::POLL_DELAY * 3)
|
|
427
|
+
File.write(tx, rows_json([row_for('assistant', text: 'landed late', uuid: 'BBB')]),
|
|
428
|
+
mode: 'a', encoding: 'UTF-8')
|
|
429
|
+
end
|
|
430
|
+
begin
|
|
431
|
+
text, why, record_id = G.last_assistant_text(tx, true)
|
|
432
|
+
ensure
|
|
433
|
+
writer.join
|
|
434
|
+
end
|
|
435
|
+
assert_equal 'landed late', text
|
|
436
|
+
assert_equal 'ok-after-wait', why
|
|
437
|
+
assert_equal 'BBB', record_id
|
|
438
|
+
end
|
|
439
|
+
end
|
|
440
|
+
|
|
441
|
+
# --- what round 1 of the 2026-08-21 review found the tests above missed ----
|
|
442
|
+
#
|
|
443
|
+
# Every fixture below was written against a mutation that the original seven
|
|
444
|
+
# left green. A behaviour no mutation can kill is untested however many
|
|
445
|
+
# assertions surround it.
|
|
446
|
+
|
|
447
|
+
# The marker was read once, on attempt zero, while the rewrite got forty
|
|
448
|
+
# attempts. Both are written by the same process at nearly the same moment,
|
|
449
|
+
# so a marker a fraction of a second late sent the read to the fallback and
|
|
450
|
+
# re-judged the blocked message — the defect this whole method exists to end,
|
|
451
|
+
# restored in full, with nothing but a suffix in the log to show for it.
|
|
452
|
+
def test_the_recheck_waits_for_the_marker_as_well_as_the_rewrite
|
|
453
|
+
Dir.mktmpdir do |tmp|
|
|
454
|
+
tx = File.join(tmp, 't.jsonl')
|
|
455
|
+
File.write(tx, rows_json([row_for('assistant', text: BLOCKED, uuid: 'AAA')]),
|
|
456
|
+
encoding: 'UTF-8')
|
|
457
|
+
writer = Thread.new do
|
|
458
|
+
sleep(G::POLL_DELAY * 3)
|
|
459
|
+
File.write(
|
|
460
|
+
tx,
|
|
461
|
+
rows_json([row_for('user', text: "Stop hook feedback:\n- x", parent: 'AAA', uuid: 'MMM'),
|
|
462
|
+
row_for('assistant', text: REWRITE, uuid: 'BBB')]),
|
|
463
|
+
mode: 'a', encoding: 'UTF-8'
|
|
464
|
+
)
|
|
465
|
+
end
|
|
466
|
+
begin
|
|
467
|
+
text, why, record_id = G.last_assistant_text(tx, true)
|
|
468
|
+
ensure
|
|
469
|
+
writer.join
|
|
470
|
+
end
|
|
471
|
+
assert_equal REWRITE, text, 'a marker 0.3s late must not send the read to the fallback'
|
|
472
|
+
assert_equal 'ok-after-wait', why
|
|
473
|
+
assert_equal 'BBB', record_id
|
|
474
|
+
end
|
|
475
|
+
end
|
|
476
|
+
|
|
477
|
+
# Operator ruling, 2026-08-21: the recheck searches backward for text rather
|
|
478
|
+
# than stopping at the newest record, accepting that it now differs from the
|
|
479
|
+
# first read. Stopping at the newest record spent the whole budget and called
|
|
480
|
+
# the rewrite absent while it sat one row down.
|
|
481
|
+
def test_a_rewrite_behind_a_later_textless_record_is_still_measured
|
|
482
|
+
rows = blocked_then(row_for('assistant', text: REWRITE, uuid: 'BBB'),
|
|
483
|
+
row_for('assistant', uuid: 'CCC', thinking: true))
|
|
484
|
+
out, log = drive(rows, max_headings: 3)
|
|
485
|
+
assert_includes log, 'rec=BBB', log.inspect
|
|
486
|
+
refute_includes log, 'awaiting-rewrite',
|
|
487
|
+
'the rewrite was in the tail; a trailing thinking record must not hide it'
|
|
488
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS', out.inspect
|
|
489
|
+
end
|
|
490
|
+
|
|
491
|
+
# Two blocks in a row, the second with no rewrite yet. The fixture above this
|
|
492
|
+
# one cannot tell the newest marker from the oldest, because both lead to the
|
|
493
|
+
# same record; here the older marker hands back the message the *second*
|
|
494
|
+
# block just named.
|
|
495
|
+
def test_the_oldest_marker_would_hand_back_the_message_the_newest_one_named
|
|
496
|
+
rows = [row_for('assistant', text: BLOCKED, uuid: 'AAA'),
|
|
497
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'AAA', uuid: 'M1'),
|
|
498
|
+
row_for('assistant', text: REWRITE, uuid: 'BBB'),
|
|
499
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'BBB', uuid: 'M2')]
|
|
500
|
+
_out, log = drive(rows, max_headings: 3)
|
|
501
|
+
assert_includes log, 'SKIP-awaiting-rewrite', log.inspect
|
|
502
|
+
refute_includes log, 'rec=BBB', 'BBB is what the newest marker names, not a rewrite'
|
|
503
|
+
end
|
|
504
|
+
|
|
505
|
+
# This project discusses its own gate, so an assistant message opening with
|
|
506
|
+
# Claude Code's feedback wording is not hypothetical.
|
|
507
|
+
def test_only_a_user_record_can_be_a_marker
|
|
508
|
+
rows = blocked_then(row_for('assistant', text: "Stop hook feedback: 見出しなし\n", uuid: 'BBB'))
|
|
509
|
+
_out, log = drive(rows, max_headings: 3)
|
|
510
|
+
assert_includes log, 'rec=BBB', log.inspect
|
|
511
|
+
refute_includes log, 'awaiting-rewrite', log.inspect
|
|
512
|
+
end
|
|
513
|
+
|
|
514
|
+
def test_a_record_that_merely_mentions_the_wording_is_not_a_marker
|
|
515
|
+
# The operator asking about the gate is an ordinary turn in this project.
|
|
516
|
+
# If that question counted as a marker, the read would treat the answer it
|
|
517
|
+
# just blocked as lying after a marker, and measure it.
|
|
518
|
+
rows = [row_for('user', text: 'なぜ Stop hook feedback: が出るの?', uuid: 'Q'),
|
|
519
|
+
row_for('assistant', text: BLOCKED, uuid: 'AAA')]
|
|
520
|
+
_out, log = drive(rows, max_headings: 3)
|
|
521
|
+
assert_includes log, 'nomarker', log.inspect
|
|
522
|
+
refute_includes log, 'rec=AAA', 'AAA is the message that was just blocked'
|
|
523
|
+
end
|
|
524
|
+
|
|
525
|
+
# The fallback's only visible trace used to be the exit that finds text, so a
|
|
526
|
+
# change in Claude Code's wording reported nothing in the two cases it
|
|
527
|
+
# actually produces — and one of them borrowed the name of the counter that
|
|
528
|
+
# decides whether a rewrite cap is worth building.
|
|
529
|
+
def test_a_recheck_with_no_marker_says_so_on_every_exit
|
|
530
|
+
_out, textless = drive([row_for('assistant', uuid: 'AAA', thinking: true)], max_headings: 3)
|
|
531
|
+
assert_includes textless, 'nomarker', textless.inspect
|
|
532
|
+
refute_includes textless, 'awaiting-rewrite',
|
|
533
|
+
'no marker was found, so the rewrite was never the question'
|
|
534
|
+
|
|
535
|
+
_out2, none = drive([row_for('user', text: 'hello')], max_headings: 3)
|
|
536
|
+
assert_includes none, 'SKIP-no-assistant-record-nomarker', none.inspect
|
|
537
|
+
refute_includes none, 'awaiting-rewrite', none.inspect
|
|
538
|
+
end
|
|
539
|
+
|
|
540
|
+
# Comparing the judged uuid while it is nil made a record carrying no uuid
|
|
541
|
+
# look like the judged record, and skipped a rewrite that was right there.
|
|
542
|
+
def test_a_rewrite_with_no_uuid_is_measured_rather_than_taken_for_the_judged_one
|
|
543
|
+
rows = [row_for('assistant', text: BLOCKED, uuid: 'AAA'),
|
|
544
|
+
row_for('user', text: "Stop hook feedback:\n- x", uuid: 'MMM'),
|
|
545
|
+
row_for('assistant', text: REWRITE)]
|
|
546
|
+
out, log = drive(rows, max_headings: 3)
|
|
547
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS', out.inspect
|
|
548
|
+
refute_includes log, 'awaiting-rewrite', log.inspect
|
|
549
|
+
end
|
|
550
|
+
|
|
551
|
+
# The budget's length is a claim in its own right, and nothing witnessed it:
|
|
552
|
+
# cutting forty attempts back to the first read's fifteen left the whole suite
|
|
553
|
+
# green. A rewrite that lands after 1.5s and before 4s is the only thing that
|
|
554
|
+
# can tell the two budgets apart.
|
|
555
|
+
def test_the_recheck_budget_outlasts_the_first_read_s
|
|
556
|
+
Dir.mktmpdir do |tmp|
|
|
557
|
+
tx = File.join(tmp, 't.jsonl')
|
|
558
|
+
File.write(tx, rows_json(blocked_then), encoding: 'UTF-8')
|
|
559
|
+
writer = Thread.new do
|
|
560
|
+
sleep((G::POLL_ATTEMPTS * G::POLL_DELAY) + 0.4)
|
|
561
|
+
File.write(tx, rows_json([row_for('assistant', text: REWRITE, uuid: 'BBB')]),
|
|
562
|
+
mode: 'a', encoding: 'UTF-8')
|
|
563
|
+
end
|
|
564
|
+
begin
|
|
565
|
+
text, why, record_id = G.last_assistant_text(tx, true)
|
|
566
|
+
ensure
|
|
567
|
+
writer.join
|
|
568
|
+
end
|
|
569
|
+
assert_equal REWRITE, text,
|
|
570
|
+
"a rewrite landing after the first read's budget must still be caught"
|
|
571
|
+
assert_equal 'ok-after-wait', why
|
|
572
|
+
assert_equal 'BBB', record_id
|
|
573
|
+
end
|
|
574
|
+
end
|
|
575
|
+
|
|
576
|
+
# Every other fixture uses a three-character uuid, so nothing witnessed the
|
|
577
|
+
# truncation against the thirty-six-character uuids production writes.
|
|
578
|
+
def test_the_rec_column_carries_only_the_first_eight_characters
|
|
579
|
+
long = 'abcdefgh-1234-5678-9abc-def012345678'
|
|
580
|
+
rows = blocked_then(row_for('assistant', text: REWRITE, uuid: long))
|
|
581
|
+
_out, log = drive(rows, max_headings: 3)
|
|
582
|
+
assert_includes log, "rec=abcdefgh\t", log.inspect
|
|
583
|
+
refute_includes log, "rec=#{long}", 'the whole uuid would push the metrics off the line'
|
|
584
|
+
end
|
|
585
|
+
|
|
586
|
+
# --- what round 2 of the review found the fixtures above still missed ------
|
|
587
|
+
|
|
588
|
+
def second_block(*after)
|
|
589
|
+
# A session already blocked once: the first rewrite passed and was
|
|
590
|
+
# delivered, the operator asked something else, and that answer was blocked
|
|
591
|
+
# too. Everything up to and including CCC is on disk before CCC's own marker
|
|
592
|
+
# is written.
|
|
593
|
+
[row_for('assistant', text: BLOCKED, uuid: 'AAA'),
|
|
594
|
+
row_for('user', text: "Stop hook feedback:\n- x", parent: 'AAA', uuid: 'M1'),
|
|
595
|
+
row_for('assistant', text: REWRITE, uuid: 'BBB'),
|
|
596
|
+
row_for('user', text: '次の質問です', uuid: 'Q'),
|
|
597
|
+
row_for('assistant', text: BLOCKED, uuid: 'CCC')] + after
|
|
598
|
+
end
|
|
599
|
+
|
|
600
|
+
# The marker that counts is this turn's, not the newest in the file. Taking the
|
|
601
|
+
# newest one let the older marker stand in for a marker that had not landed,
|
|
602
|
+
# so the read skipped its wait and measured the message it had just blocked —
|
|
603
|
+
# in 0.06s, logged as an ordinary RECHECK verdict. 127 of 170 real blocks are
|
|
604
|
+
# not a session's first.
|
|
605
|
+
def test_a_marker_from_an_earlier_block_is_not_this_turn_s
|
|
606
|
+
_out, log = drive(second_block, max_headings: 3)
|
|
607
|
+
assert_includes log, 'nomarker', log.inspect
|
|
608
|
+
refute_includes log, 'rec=CCC', 'CCC is the message this turn just blocked'
|
|
609
|
+
refute_includes log, 'rec=BBB', "BBB is the previous turn's delivered rewrite"
|
|
610
|
+
end
|
|
611
|
+
|
|
612
|
+
def test_the_second_block_s_own_marker_is_used_once_it_lands
|
|
613
|
+
rows = second_block(row_for('user', text: "Stop hook feedback:\n- x", parent: 'CCC', uuid: 'M2'),
|
|
614
|
+
row_for('assistant', text: REWRITE, uuid: 'DDD'))
|
|
615
|
+
out, log = drive(rows, max_headings: 3)
|
|
616
|
+
assert_includes log, 'rec=DDD', log.inspect
|
|
617
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS'
|
|
618
|
+
end
|
|
619
|
+
|
|
620
|
+
# A rewrite that calls a tool writes a short preamble, then tool records, then
|
|
621
|
+
# its real answer. Stepping over the text-less records to find text measured
|
|
622
|
+
# the two-line preamble and logged RECHECK-PASS in 0.04s, while the real
|
|
623
|
+
# rewrite — landing 0.6s later, four headings, a FAIL — was never read. Inside
|
|
624
|
+
# the budget the newest record is now waited for instead.
|
|
625
|
+
def test_a_preamble_is_not_mistaken_for_the_rewrite_while_the_budget_remains
|
|
626
|
+
Dir.mktmpdir do |tmp|
|
|
627
|
+
tx = File.join(tmp, 't.jsonl')
|
|
628
|
+
File.write(tx, rows_json(blocked_then(row_for('assistant', text: "少し調べます\n", uuid: 'P1'),
|
|
629
|
+
row_for('assistant', uuid: 'T1', thinking: true))),
|
|
630
|
+
encoding: 'UTF-8')
|
|
631
|
+
writer = Thread.new do
|
|
632
|
+
sleep(G::POLL_DELAY * 3)
|
|
633
|
+
File.write(tx, rows_json([row_for('assistant', text: BLOCKED, uuid: 'REAL')]),
|
|
634
|
+
mode: 'a', encoding: 'UTF-8')
|
|
635
|
+
end
|
|
636
|
+
begin
|
|
637
|
+
text, why, record_id = G.last_assistant_text(tx, true)
|
|
638
|
+
ensure
|
|
639
|
+
writer.join
|
|
640
|
+
end
|
|
641
|
+
assert_equal BLOCKED, text, 'the real rewrite, not the preamble that preceded it'
|
|
642
|
+
assert_equal 'REAL', record_id
|
|
643
|
+
assert_equal 'ok-after-wait', why
|
|
644
|
+
end
|
|
645
|
+
end
|
|
646
|
+
|
|
647
|
+
# Kept as a last resort, and named apart so the log can count how often the
|
|
648
|
+
# ordinary rule was not enough. Over 3,248 transcripts this shape occurred 0
|
|
649
|
+
# times, so it must never pre-empt the wait — only outlive it.
|
|
650
|
+
def test_text_under_a_newer_textless_record_is_reached_only_after_the_budget
|
|
651
|
+
rows = blocked_then(row_for('assistant', text: REWRITE, uuid: 'BBB'),
|
|
652
|
+
row_for('assistant', uuid: 'T1', thinking: true))
|
|
653
|
+
out, log = drive(rows, max_headings: 3)
|
|
654
|
+
assert_includes log, 'ok-after-wait-deep', log.inspect
|
|
655
|
+
assert_includes log, 'rec=BBB', log.inspect
|
|
656
|
+
assert_includes out.fetch('systemMessage', ''), 'PASS'
|
|
657
|
+
end
|
|
658
|
+
|
|
659
|
+
# Naming only the two expected reasons left the other exits silent, and both
|
|
660
|
+
# of the silent ones are live: a transcript momentarily unreadable, and a
|
|
661
|
+
# fallback that found whitespace. Each spent the whole budget and told the
|
|
662
|
+
# operator nothing.
|
|
663
|
+
def test_every_recheck_that_produces_no_verdict_says_so_on_screen
|
|
664
|
+
_out, = drive([row_for('assistant', uuid: 'AAA', thinking: true)], max_headings: 3)
|
|
665
|
+
|
|
666
|
+
whitespace, = drive([row_for('assistant', text: " \n", uuid: 'AAA')], max_headings: 3)
|
|
667
|
+
assert_includes whitespace.fetch('systemMessage', ''), 'NOT RUN', whitespace.inspect
|
|
668
|
+
|
|
669
|
+
Dir.mktmpdir do |tmp|
|
|
670
|
+
cfg_path = File.join(tmp, 'cfg.json')
|
|
671
|
+
File.write(cfg_path, JSON.generate('mode_name' => 't', 'log_path' => File.join(tmp, 'g.log')),
|
|
672
|
+
encoding: 'UTF-8')
|
|
673
|
+
out, _err, status = run_script(
|
|
674
|
+
cfg_path,
|
|
675
|
+
JSON.generate('transcript_path' => tmp, 'stop_hook_active' => true) # a directory
|
|
676
|
+
)
|
|
677
|
+
assert_equal 0, status.exitstatus
|
|
678
|
+
assert_includes JSON.parse(out).fetch('systemMessage', ''), 'NOT RUN',
|
|
679
|
+
'an unreadable transcript on a recheck must not be silent'
|
|
680
|
+
end
|
|
681
|
+
end
|
|
682
|
+
|
|
683
|
+
# The banner quotes a number. Hard-coding it left the suite green while the
|
|
684
|
+
# figure the operator reads drifted away from the budget actually spent.
|
|
685
|
+
def test_the_banner_quotes_the_budget_it_actually_spent
|
|
686
|
+
out, = drive(blocked_then, max_headings: 3)
|
|
687
|
+
expected = format('%.1f', G::RECHECK_POLL_ATTEMPTS * G::POLL_DELAY)
|
|
688
|
+
assert_includes out.fetch('systemMessage', ''), "#{expected}s", out.inspect
|
|
285
689
|
end
|
|
286
690
|
|
|
287
691
|
def test_a_passing_message_never_blocks_either_way
|
|
@@ -432,14 +836,33 @@ class TestReadableGate < Minitest::Test
|
|
|
432
836
|
original = G.method(:measure)
|
|
433
837
|
G.define_singleton_method(:measure) { |*| seen = Regexp.timeout; [{}, []] }
|
|
434
838
|
begin
|
|
435
|
-
G.measure_bounded("x\n", cfg('measure_timeout_seconds' =>
|
|
839
|
+
G.measure_bounded("x\n", cfg('measure_timeout_seconds' => 3))
|
|
436
840
|
ensure
|
|
437
841
|
G.define_singleton_method(:measure, original)
|
|
438
842
|
end
|
|
439
|
-
assert_equal
|
|
843
|
+
assert_equal 3, seen, 'the per-match bound is installed while measuring'
|
|
440
844
|
assert_nil Regexp.timeout, 'and restored afterwards'
|
|
441
845
|
end
|
|
442
846
|
|
|
847
|
+
# The two budgets this file spends run one after the other inside one hook
|
|
848
|
+
# invocation, and neither knew about the other. A mode asking for 6 seconds of
|
|
849
|
+
# measurement put the pair at 10.4 against a 10-second limit; Claude Code kills
|
|
850
|
+
# the hook there and it emits nothing at all, so the gate stops enforcing and
|
|
851
|
+
# the log stops recording, both without a trace.
|
|
852
|
+
def test_a_mode_cannot_ask_for_more_measurement_time_than_the_hook_has_left
|
|
853
|
+
assert_operator (G::RECHECK_POLL_ATTEMPTS * G::POLL_DELAY) + G::MEASURE_TIMEOUT_CEILING,
|
|
854
|
+
:<=, G::HOOK_TIMEOUT - G::HOOK_TIMEOUT_MARGIN,
|
|
855
|
+
'the recheck budget and the measurement ceiling must fit the hook timeout'
|
|
856
|
+
|
|
857
|
+
over = cfg('measure_timeout_seconds' => 60)
|
|
858
|
+
assert_equal G::MEASURE_TIMEOUT_CEILING, over.measure_timeout,
|
|
859
|
+
'an over-large request is clamped to what is left'
|
|
860
|
+
assert_empty over.problems, 'and clamping is not an error the operator must act on'
|
|
861
|
+
|
|
862
|
+
under = cfg('measure_timeout_seconds' => 2)
|
|
863
|
+
assert_equal 2, under.measure_timeout, 'a request that fits is passed through unchanged'
|
|
864
|
+
end
|
|
865
|
+
|
|
443
866
|
def test_a_transcript_that_cannot_be_read_is_recorded_as_a_skip
|
|
444
867
|
Dir.mktmpdir do |tmp|
|
|
445
868
|
log = File.join(tmp, 'gate.log')
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: kairos-chain
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 3.
|
|
4
|
+
version: 3.76.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Masaomi Hatakeyama
|
|
@@ -602,6 +602,7 @@ files:
|
|
|
602
602
|
- templates/skillsets/kairos_hook_projector/mode_hooks/_schema.json
|
|
603
603
|
- templates/skillsets/kairos_hook_projector/plugin/SKILL.md
|
|
604
604
|
- templates/skillsets/kairos_hook_projector/skillset.json
|
|
605
|
+
- templates/skillsets/kairos_hook_projector/test/mutation_check_readable_gate.rb
|
|
605
606
|
- templates/skillsets/kairos_hook_projector/test/test_boot_time_assertion.rb
|
|
606
607
|
- templates/skillsets/kairos_hook_projector/test/test_hooks_status.rb
|
|
607
608
|
- templates/skillsets/kairos_hook_projector/test/test_mode_hooks_add.rb
|