devlyn-cli 2.10.0 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -9
- package/CLAUDE.md +11 -15
- package/README.md +31 -7
- package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.json +25 -0
- package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.md +10 -0
- package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F16-cli-quote-tax-rules/l2_risk_probes/result.json +28 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F23-cli-fulfillment-wave/l2_risk_probes/result.json +28 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F25-cli-cart-promotion-rules/l2_risk_probes/result.json +30 -0
- package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/F21-cli-scheduler-priority/l2_risk_probes/result.json +32 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.json +44 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.md +16 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/scripts/run-fixture.sh +11 -1
- package/benchmark/auto-resolve/scripts/test-benchmark-arg-parsing.sh +83 -1
- package/bin/devlyn.js +59 -9
- package/config/skills/_shared/.ruff.toml +2 -0
- package/config/skills/_shared/adapters/claude.md +9 -1
- package/config/skills/_shared/adapters/codex.md +5 -0
- package/config/skills/_shared/adapters/grok.md +37 -13
- package/config/skills/_shared/archive_run.py +738 -56
- package/config/skills/_shared/codex-config.md +19 -4
- package/config/skills/_shared/codex-monitored.sh +81 -12
- package/config/skills/_shared/collect-codex-findings.py +419 -60
- package/config/skills/_shared/engine-preflight.md +8 -7
- package/config/skills/_shared/expected.schema.json +68 -0
- package/config/skills/_shared/finish-gate.py +12 -11
- package/config/skills/_shared/grok-anchor-guard.py +78 -0
- package/config/skills/_shared/invocation-receipt.py +1041 -0
- package/config/skills/_shared/judge-output-parser.py +200 -0
- package/config/skills/_shared/judge-role-evidence.py +273 -0
- package/config/skills/_shared/phase-prompt-render.py +181 -0
- package/config/skills/_shared/process-evidence.py +909 -0
- package/config/skills/_shared/resolve-bootstrap.py +776 -84
- package/config/skills/_shared/resolve-stop-hook.py +33 -1
- package/config/skills/_shared/role-config.py +552 -0
- package/config/skills/_shared/run-bounded.py +1 -1
- package/config/skills/_shared/spec-verify-check.py +1037 -192
- package/config/skills/_shared/state-phase-write.py +2551 -83
- package/config/skills/_shared/terminal-claim-check.py +15 -0
- package/config/skills/_shared/verify-merge-findings.py +1127 -50
- package/config/skills/devlyn:engines/SKILL.md +8 -16
- package/config/skills/devlyn:ideate/SKILL.md +1 -1
- package/config/skills/devlyn:ideate/references/spec-template.md +1 -0
- package/config/skills/devlyn:queue/SKILL.md +7 -4
- package/config/skills/devlyn:resolve/SKILL.md +94 -35
- package/config/skills/devlyn:resolve/references/outer-loop.md +15 -0
- package/config/skills/devlyn:resolve/references/phases/build-gate.md +33 -4
- package/config/skills/devlyn:resolve/references/phases/implement.md +4 -3
- package/config/skills/devlyn:resolve/references/phases/plan.md +1 -5
- package/config/skills/devlyn:resolve/references/phases/probe-derive.md +4 -2
- package/config/skills/devlyn:resolve/references/phases/verify.md +120 -80
- package/config/skills/devlyn:resolve/references/state-schema.md +23 -9
- package/package.json +5 -1
- package/scripts/lint-fixtures.sh +4 -1
- package/scripts/lint-skills.sh +447 -290
|
@@ -1,25 +1,42 @@
|
|
|
1
1
|
#!/usr/bin/env python3
|
|
2
|
-
"""Normalize raw
|
|
2
|
+
"""Normalize raw pair-JUDGE stdout into canonical VERIFY JSONL."""
|
|
3
3
|
|
|
4
4
|
from __future__ import annotations
|
|
5
5
|
|
|
6
6
|
import argparse
|
|
7
7
|
import json
|
|
8
8
|
import pathlib
|
|
9
|
+
import runpy
|
|
9
10
|
import sys
|
|
10
11
|
import tempfile
|
|
11
12
|
from typing import Any
|
|
12
13
|
|
|
13
14
|
|
|
14
|
-
|
|
15
|
+
MERGE_CONTRACT = runpy.run_path(pathlib.Path(__file__).with_name("verify-merge-findings.py"))
|
|
16
|
+
PARSER = runpy.run_path(pathlib.Path(__file__).with_name("judge-output-parser.py"))
|
|
17
|
+
VERDICT_RANK = MERGE_CONTRACT["VERDICT_RANK"]
|
|
18
|
+
finding_rank = MERGE_CONTRACT["finding_rank"]
|
|
15
19
|
|
|
16
20
|
|
|
17
21
|
def reject_json_constant(token: str) -> None:
|
|
18
22
|
raise ValueError(f"invalid JSON numeric constant: {token}")
|
|
19
23
|
|
|
20
24
|
|
|
25
|
+
def reject_duplicate_keys(pairs: list[tuple[str, object]]) -> dict:
|
|
26
|
+
result = {}
|
|
27
|
+
for key, value in pairs:
|
|
28
|
+
if key in result:
|
|
29
|
+
raise ValueError(f"duplicate JSON key: {key}")
|
|
30
|
+
result[key] = value
|
|
31
|
+
return result
|
|
32
|
+
|
|
33
|
+
|
|
21
34
|
def loads_strict_json(text: str) -> Any:
|
|
22
|
-
return json.loads(
|
|
35
|
+
return json.loads(
|
|
36
|
+
text,
|
|
37
|
+
parse_constant=reject_json_constant,
|
|
38
|
+
object_pairs_hook=reject_duplicate_keys,
|
|
39
|
+
)
|
|
23
40
|
|
|
24
41
|
|
|
25
42
|
def atomic_write(path: pathlib.Path, text: str) -> None:
|
|
@@ -32,76 +49,418 @@ def atomic_write(path: pathlib.Path, text: str) -> None:
|
|
|
32
49
|
pathlib.Path(tmp_name).replace(path)
|
|
33
50
|
|
|
34
51
|
|
|
35
|
-
def
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
try:
|
|
55
|
-
item = loads_strict_json(raw)
|
|
56
|
-
except ValueError as exc:
|
|
57
|
-
raise SystemExit(f"error: invalid JSONL at {stdout_path}:{line_no}: {exc}")
|
|
58
|
-
if not isinstance(item, dict):
|
|
59
|
-
raise SystemExit(f"error: JSONL item is not an object at {stdout_path}:{line_no}")
|
|
60
|
-
severity = str(item.get("severity") or "").upper()
|
|
61
|
-
if severity not in FINDING_SEVERITIES:
|
|
62
|
-
raise SystemExit(f"error: finding missing valid severity at {stdout_path}:{line_no}")
|
|
63
|
-
findings.append(item)
|
|
52
|
+
def collect_text(
|
|
53
|
+
text: str, source: pathlib.Path
|
|
54
|
+
) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
|
|
55
|
+
findings, summary = PARSER["collect_text"](text, source)
|
|
56
|
+
if findings and summary is None:
|
|
57
|
+
raise SystemExit("error: findings without terminal verdict")
|
|
58
|
+
if summary is not None and summary["verdict"] == "PASS" and any(finding_rank(finding) == 2 for finding in findings):
|
|
59
|
+
raise SystemExit("error: verdict-binding finding cannot have a PASS verdict")
|
|
60
|
+
if not findings and (summary is None or summary.get("verdict") != "PASS"):
|
|
61
|
+
raise SystemExit("error: non-PASS verdict without JSONL findings")
|
|
62
|
+
return findings, summary
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
def collect_stdout(stdout_path: pathlib.Path) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
|
|
66
|
+
findings, summary = PARSER["collect_stdout"](stdout_path)
|
|
67
|
+
if findings and summary is None:
|
|
68
|
+
raise SystemExit("error: findings without terminal verdict")
|
|
69
|
+
if summary is not None and summary["verdict"] == "PASS" and any(finding_rank(finding) == 2 for finding in findings):
|
|
70
|
+
raise SystemExit("error: verdict-binding finding cannot have a PASS verdict")
|
|
64
71
|
if not findings and (summary is None or summary.get("verdict") != "PASS"):
|
|
65
|
-
raise SystemExit("error: non-PASS
|
|
72
|
+
raise SystemExit("error: non-PASS verdict without JSONL findings")
|
|
66
73
|
return findings, summary
|
|
67
74
|
|
|
68
75
|
|
|
69
76
|
def self_test() -> int:
|
|
77
|
+
try:
|
|
78
|
+
loads_strict_json('{"verdict":"PASS","verdict":"BLOCKED"}')
|
|
79
|
+
except ValueError as exc:
|
|
80
|
+
assert "duplicate JSON key" in str(exc)
|
|
81
|
+
else:
|
|
82
|
+
raise AssertionError("duplicate judge result key was accepted")
|
|
70
83
|
with tempfile.TemporaryDirectory() as tmp:
|
|
71
84
|
root = pathlib.Path(tmp)
|
|
72
|
-
stdout_path = root / "
|
|
85
|
+
stdout_path = root / "pair-judge.stdout"
|
|
73
86
|
out_path = root / "verify.pair.findings.jsonl"
|
|
74
|
-
summary_path = root / "
|
|
87
|
+
summary_path = root / "pair-judge.summary.json"
|
|
88
|
+
|
|
89
|
+
def assert_rejected(text: str, label: str, *message_parts: str) -> None:
|
|
90
|
+
stdout_path.write_text(text, encoding="utf-8")
|
|
91
|
+
try:
|
|
92
|
+
collect_stdout(stdout_path)
|
|
93
|
+
except SystemExit as exc:
|
|
94
|
+
message = str(exc)
|
|
95
|
+
assert all(part in message for part in message_parts)
|
|
96
|
+
else:
|
|
97
|
+
raise AssertionError(f"{label} must be rejected")
|
|
98
|
+
|
|
99
|
+
plain_finding = {"id": "a", "severity": "HIGH"}
|
|
100
|
+
plain_summary = {"verdict": "NEEDS_WORK"}
|
|
75
101
|
stdout_path.write_text(
|
|
76
|
-
json.dumps(
|
|
77
|
-
+
|
|
102
|
+
json.dumps(plain_finding) + "\n"
|
|
103
|
+
+ plain_summary["verdict"] + "\n",
|
|
78
104
|
encoding="utf-8",
|
|
79
105
|
)
|
|
80
|
-
findings, summary =
|
|
106
|
+
findings, summary = collect_stdout(stdout_path)
|
|
107
|
+
assert findings == [plain_finding]
|
|
108
|
+
assert summary == plain_summary
|
|
81
109
|
write_outputs(findings, summary, out_path, summary_path)
|
|
82
110
|
assert out_path.read_text(encoding="utf-8").count("\n") == 1
|
|
83
111
|
assert loads_strict_json(summary_path.read_text(encoding="utf-8"))["verdict"] == "NEEDS_WORK"
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
112
|
+
assert_rejected(
|
|
113
|
+
'{"id":"nan","severity":NaN}\n',
|
|
114
|
+
"NaN pair-JUDGE stdout finding",
|
|
115
|
+
"invalid JSON numeric constant: NaN",
|
|
116
|
+
)
|
|
117
|
+
assert_rejected(
|
|
118
|
+
'# SUMMARY {"verdict":"PASS","verdict":"BLOCKED"}\n',
|
|
119
|
+
"duplicate pair-JUDGE summary verdict",
|
|
120
|
+
"duplicate JSON key: verdict",
|
|
121
|
+
)
|
|
91
122
|
rejection_cases = (
|
|
92
|
-
("", "no
|
|
93
|
-
(
|
|
94
|
-
('
|
|
95
|
-
('
|
|
123
|
+
("", "no verdict line", "non-PASS verdict without JSONL findings"),
|
|
124
|
+
(json.dumps(plain_finding) + "\n", "finding without terminal verdict", "findings without terminal verdict"),
|
|
125
|
+
('{"verdict":"pass"}\n', "lowercase pass", "finding missing valid severity"),
|
|
126
|
+
('{"verdict":"UNKNOWN"}\n', "unknown verdict", "finding missing valid severity"),
|
|
96
127
|
)
|
|
97
|
-
for text, label in rejection_cases:
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
128
|
+
for text, label, message in rejection_cases:
|
|
129
|
+
assert_rejected(
|
|
130
|
+
text,
|
|
131
|
+
label,
|
|
132
|
+
message,
|
|
133
|
+
)
|
|
134
|
+
|
|
135
|
+
def envelope(text: Any, stop_reason: str = "EndTurn") -> str:
|
|
136
|
+
return json.dumps(
|
|
137
|
+
{
|
|
138
|
+
"text": text,
|
|
139
|
+
"stopReason": stop_reason,
|
|
140
|
+
"sessionId": "session",
|
|
141
|
+
"requestId": "request",
|
|
142
|
+
}
|
|
143
|
+
)
|
|
144
|
+
|
|
145
|
+
envelope_finding = {
|
|
146
|
+
"id": "envelope-finding",
|
|
147
|
+
"severity": "HIGH",
|
|
148
|
+
"detail": {"preserved": True},
|
|
149
|
+
}
|
|
150
|
+
envelope_summary = {"verdict": "NEEDS_WORK"}
|
|
151
|
+
stdout_path.write_text(
|
|
152
|
+
envelope(
|
|
153
|
+
json.dumps(envelope_finding)
|
|
154
|
+
+ "\n" + envelope_summary["verdict"] + "\n"
|
|
155
|
+
),
|
|
156
|
+
encoding="utf-8",
|
|
157
|
+
)
|
|
158
|
+
findings, summary = collect_stdout(stdout_path)
|
|
159
|
+
assert findings == [envelope_finding]
|
|
160
|
+
assert summary == envelope_summary
|
|
161
|
+
|
|
162
|
+
assert_rejected(
|
|
163
|
+
envelope("review completed without findings"),
|
|
164
|
+
"findings-less envelope text",
|
|
165
|
+
"invalid JSONL",
|
|
166
|
+
)
|
|
167
|
+
assert_rejected(
|
|
168
|
+
envelope('{"findings":[],"verdict":"PASS"}', "Cancelled"),
|
|
169
|
+
"Cancelled envelope",
|
|
170
|
+
"envelope stopReason must be 'EndTurn'",
|
|
171
|
+
"'Cancelled'",
|
|
172
|
+
)
|
|
173
|
+
assert_rejected(
|
|
174
|
+
envelope("", "UnknownStop"),
|
|
175
|
+
"unknown envelope stopReason",
|
|
176
|
+
"envelope stopReason must be 'EndTurn'",
|
|
177
|
+
"'UnknownStop'",
|
|
178
|
+
)
|
|
179
|
+
assert_rejected(
|
|
180
|
+
envelope(None),
|
|
181
|
+
"non-string envelope text",
|
|
182
|
+
"error: envelope text must be a string",
|
|
183
|
+
)
|
|
184
|
+
assert_rejected(
|
|
185
|
+
envelope("I will inspect the result.\n" + envelope_summary["verdict"] + "\n"),
|
|
186
|
+
"narrative envelope preamble",
|
|
187
|
+
"invalid JSONL",
|
|
188
|
+
)
|
|
189
|
+
dual_document = (
|
|
190
|
+
json.dumps(envelope_finding)
|
|
191
|
+
+ json.dumps({"id": "second", "severity": "LOW"})
|
|
192
|
+
+ "\n" + envelope_summary["verdict"] + "\n"
|
|
193
|
+
)
|
|
194
|
+
assert_rejected(
|
|
195
|
+
envelope(dual_document),
|
|
196
|
+
"concatenated envelope documents",
|
|
197
|
+
"invalid JSONL",
|
|
198
|
+
)
|
|
199
|
+
clean_summary = {"verdict": "PASS"}
|
|
200
|
+
stdout_path.write_text(
|
|
201
|
+
envelope(clean_summary["verdict"] + "\n"),
|
|
202
|
+
encoding="utf-8",
|
|
203
|
+
)
|
|
204
|
+
findings, summary = collect_stdout(stdout_path)
|
|
205
|
+
assert findings == []
|
|
206
|
+
assert summary == clean_summary
|
|
207
|
+
stdout_path.write_text(
|
|
208
|
+
json.dumps(
|
|
209
|
+
{
|
|
210
|
+
"type": "system",
|
|
211
|
+
"text": clean_summary["verdict"] + "\n",
|
|
212
|
+
"stopReason": "EndTurn",
|
|
213
|
+
"sessionId": "session",
|
|
214
|
+
"requestId": "request",
|
|
215
|
+
}
|
|
216
|
+
),
|
|
217
|
+
encoding="utf-8",
|
|
218
|
+
)
|
|
219
|
+
findings, summary = collect_stdout(stdout_path)
|
|
220
|
+
assert findings == []
|
|
221
|
+
assert summary == clean_summary
|
|
222
|
+
|
|
223
|
+
# R4: a valid plain finding may carry an unrelated `type` field. A
|
|
224
|
+
# first-line `type: system` finding is not a stream discriminator.
|
|
225
|
+
system_finding = {"id": "system-finding", "severity": "LOW", "type": "system"}
|
|
226
|
+
stdout_path.write_text(
|
|
227
|
+
json.dumps(system_finding) + "\n" + clean_summary["verdict"] + "\n",
|
|
228
|
+
encoding="utf-8",
|
|
229
|
+
)
|
|
230
|
+
findings, summary = collect_stdout(stdout_path)
|
|
231
|
+
assert findings == [system_finding]
|
|
232
|
+
assert summary == clean_summary
|
|
233
|
+
|
|
234
|
+
# iter-0106 — Grok whole-message NDJSON carrier (R2/R3).
|
|
235
|
+
def init_record(session: str = "s1") -> dict[str, Any]:
|
|
236
|
+
return {"type": "system", "subtype": "init", "session_id": session, "model": "grok-build"}
|
|
237
|
+
|
|
238
|
+
def assistant_record(
|
|
239
|
+
blocks: list[Any], stop_reason: str = "end_turn", session: str = "s1"
|
|
240
|
+
) -> dict[str, Any]:
|
|
241
|
+
return {
|
|
242
|
+
"type": "assistant",
|
|
243
|
+
"message": {"id": "msg", "role": "assistant", "content": blocks, "stop_reason": stop_reason},
|
|
244
|
+
"parent_tool_use_id": None,
|
|
245
|
+
"session_id": session,
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
def result_record(
|
|
249
|
+
text: str,
|
|
250
|
+
session: str = "s1",
|
|
251
|
+
subtype: str = "success",
|
|
252
|
+
is_error: Any = False,
|
|
253
|
+
stop_reason: str = "end_turn",
|
|
254
|
+
) -> dict[str, Any]:
|
|
255
|
+
return {
|
|
256
|
+
"type": "result",
|
|
257
|
+
"subtype": subtype,
|
|
258
|
+
"is_error": is_error,
|
|
259
|
+
"result": text,
|
|
260
|
+
"stop_reason": stop_reason,
|
|
261
|
+
"session_id": session,
|
|
262
|
+
}
|
|
263
|
+
|
|
264
|
+
def text_block(value: str) -> dict[str, Any]:
|
|
265
|
+
return {"type": "text", "text": value}
|
|
266
|
+
|
|
267
|
+
def stream(*records: Any) -> str:
|
|
268
|
+
return "".join(json.dumps(record) + "\n" for record in records)
|
|
269
|
+
|
|
270
|
+
pass_text = "PASS\n"
|
|
271
|
+
stdout_path.write_text(
|
|
272
|
+
stream(init_record(), assistant_record([text_block(pass_text)]), result_record(pass_text)),
|
|
273
|
+
encoding="utf-8",
|
|
274
|
+
)
|
|
275
|
+
findings, summary = collect_stdout(stdout_path)
|
|
276
|
+
assert findings == []
|
|
277
|
+
assert summary == {"verdict": "PASS"}
|
|
278
|
+
|
|
279
|
+
# A real tool turn precedes the contract, and the terminal message arrives
|
|
280
|
+
# as two text blocks whose in-order concatenation is the result text.
|
|
281
|
+
stream_finding = {"id": "stream-finding", "severity": "HIGH"}
|
|
282
|
+
head = json.dumps(stream_finding) + "\n"
|
|
283
|
+
tail = "# SUMMARY " + json.dumps({"verdict": "NEEDS_WORK"}) + "\n"
|
|
284
|
+
contract = head + tail
|
|
285
|
+
narration = assistant_record(
|
|
286
|
+
[
|
|
287
|
+
text_block("Let me read the file."),
|
|
288
|
+
{"type": "tool_use", "id": "call_1", "name": "read_file", "input": {"path": "src/main.rs"}},
|
|
289
|
+
],
|
|
290
|
+
stop_reason="tool_use",
|
|
291
|
+
)
|
|
292
|
+
tool_result = {
|
|
293
|
+
"type": "user",
|
|
294
|
+
"message": {
|
|
295
|
+
"role": "user",
|
|
296
|
+
"content": [{"type": "tool_result", "tool_use_id": "call_1", "content": "fn main() {}"}],
|
|
297
|
+
},
|
|
298
|
+
"parent_tool_use_id": None,
|
|
299
|
+
"session_id": "s1",
|
|
300
|
+
}
|
|
301
|
+
stdout_path.write_text(
|
|
302
|
+
stream(
|
|
303
|
+
init_record(),
|
|
304
|
+
narration,
|
|
305
|
+
tool_result,
|
|
306
|
+
assistant_record([text_block(head), text_block(tail)]),
|
|
307
|
+
result_record(contract),
|
|
308
|
+
),
|
|
309
|
+
encoding="utf-8",
|
|
310
|
+
)
|
|
311
|
+
findings, summary = collect_stdout(stdout_path)
|
|
312
|
+
assert findings == [stream_finding]
|
|
313
|
+
assert summary == {"verdict": "NEEDS_WORK"}
|
|
314
|
+
|
|
315
|
+
# The welded body is exactly the shape envelope recovery accepts, so this
|
|
316
|
+
# also proves the stream branch never falls through to that recovery.
|
|
317
|
+
welded = "I reviewed the diff.\n" + contract
|
|
318
|
+
stream_rejections = (
|
|
319
|
+
(
|
|
320
|
+
stream(init_record(), assistant_record([text_block(welded)]), result_record(welded)),
|
|
321
|
+
"narration welded into the terminal message",
|
|
322
|
+
"invalid JSONL",
|
|
323
|
+
),
|
|
324
|
+
(
|
|
325
|
+
stream(
|
|
326
|
+
init_record(),
|
|
327
|
+
assistant_record([text_block(contract)]),
|
|
328
|
+
assistant_record([text_block(contract)]),
|
|
329
|
+
result_record(contract),
|
|
330
|
+
),
|
|
331
|
+
"two terminal end_turn messages",
|
|
332
|
+
"exactly one final end_turn assistant message",
|
|
333
|
+
),
|
|
334
|
+
(
|
|
335
|
+
stream(
|
|
336
|
+
init_record(),
|
|
337
|
+
assistant_record([text_block(contract)]),
|
|
338
|
+
assistant_record([text_block("more")], stop_reason="tool_use"),
|
|
339
|
+
result_record(contract),
|
|
340
|
+
),
|
|
341
|
+
"end_turn message that is not the final assistant message",
|
|
342
|
+
"exactly one final end_turn assistant message",
|
|
343
|
+
),
|
|
344
|
+
(
|
|
345
|
+
# The malformed line is the only defect: skipping it leaves a valid stream.
|
|
346
|
+
stream(init_record(), assistant_record([text_block(pass_text)]))
|
|
347
|
+
+ '{"type":"assistant"\n'
|
|
348
|
+
+ stream(result_record(pass_text)),
|
|
349
|
+
"malformed stream NDJSON inside an otherwise valid stream",
|
|
350
|
+
"invalid stream NDJSON",
|
|
351
|
+
),
|
|
352
|
+
(
|
|
353
|
+
stream(init_record(), assistant_record([text_block(pass_text)])),
|
|
354
|
+
"stream without a terminal result",
|
|
355
|
+
"exactly one terminal result",
|
|
356
|
+
),
|
|
357
|
+
(
|
|
358
|
+
stream(
|
|
359
|
+
init_record(),
|
|
360
|
+
assistant_record([text_block(pass_text)]),
|
|
361
|
+
result_record(pass_text),
|
|
362
|
+
tool_result,
|
|
363
|
+
),
|
|
364
|
+
"data after the terminal result",
|
|
365
|
+
"exactly one terminal result",
|
|
366
|
+
),
|
|
367
|
+
(
|
|
368
|
+
stream(
|
|
369
|
+
init_record(),
|
|
370
|
+
assistant_record([text_block(pass_text)]),
|
|
371
|
+
tool_result,
|
|
372
|
+
result_record(pass_text),
|
|
373
|
+
),
|
|
374
|
+
"data between the terminal assistant and result",
|
|
375
|
+
"exactly one final end_turn assistant message",
|
|
376
|
+
),
|
|
377
|
+
(
|
|
378
|
+
stream(
|
|
379
|
+
init_record(),
|
|
380
|
+
assistant_record([text_block(pass_text)]),
|
|
381
|
+
result_record(pass_text, subtype="error_during_execution"),
|
|
382
|
+
),
|
|
383
|
+
"error result subtype",
|
|
384
|
+
"not a successful end_turn",
|
|
385
|
+
),
|
|
386
|
+
(
|
|
387
|
+
stream(
|
|
388
|
+
init_record(),
|
|
389
|
+
assistant_record([text_block(pass_text)]),
|
|
390
|
+
result_record(pass_text, stop_reason="cancelled"),
|
|
391
|
+
),
|
|
392
|
+
"cancelled result",
|
|
393
|
+
"not a successful end_turn",
|
|
394
|
+
),
|
|
395
|
+
(
|
|
396
|
+
stream(
|
|
397
|
+
init_record(),
|
|
398
|
+
assistant_record([text_block(pass_text)]),
|
|
399
|
+
result_record(pass_text, is_error=True),
|
|
400
|
+
),
|
|
401
|
+
"error-flagged result",
|
|
402
|
+
"not a successful end_turn",
|
|
403
|
+
),
|
|
404
|
+
(
|
|
405
|
+
stream(
|
|
406
|
+
init_record(),
|
|
407
|
+
assistant_record([text_block(pass_text)]),
|
|
408
|
+
result_record("NEEDS_WORK\n"),
|
|
409
|
+
),
|
|
410
|
+
"result text that disagrees with the terminal message",
|
|
411
|
+
"result text does not match the terminal message",
|
|
412
|
+
),
|
|
413
|
+
(
|
|
414
|
+
stream(
|
|
415
|
+
init_record(),
|
|
416
|
+
assistant_record([text_block(pass_text)]),
|
|
417
|
+
result_record(pass_text, session="s2"),
|
|
418
|
+
),
|
|
419
|
+
"result from another session",
|
|
420
|
+
"session identity does not agree",
|
|
421
|
+
),
|
|
422
|
+
(
|
|
423
|
+
stream(
|
|
424
|
+
init_record(),
|
|
425
|
+
{"type": "stream_event", "event": {"type": "content_block_delta"}, "session_id": "s1"},
|
|
426
|
+
assistant_record([text_block(pass_text)]),
|
|
427
|
+
result_record(pass_text),
|
|
428
|
+
),
|
|
429
|
+
"partial-message frame",
|
|
430
|
+
"unknown stream record",
|
|
431
|
+
),
|
|
432
|
+
(
|
|
433
|
+
stream(
|
|
434
|
+
init_record(),
|
|
435
|
+
{"type": "system", "subtype": "compact_boundary", "session_id": "s1"},
|
|
436
|
+
assistant_record([text_block(pass_text)]),
|
|
437
|
+
result_record(pass_text),
|
|
438
|
+
),
|
|
439
|
+
"compact boundary",
|
|
440
|
+
"second system record",
|
|
441
|
+
),
|
|
442
|
+
(
|
|
443
|
+
stream(
|
|
444
|
+
init_record(),
|
|
445
|
+
{"type": "unknown", "session_id": "s1"},
|
|
446
|
+
assistant_record([text_block(pass_text)]),
|
|
447
|
+
result_record(pass_text),
|
|
448
|
+
),
|
|
449
|
+
"unknown record type",
|
|
450
|
+
"unknown stream record",
|
|
451
|
+
),
|
|
452
|
+
(
|
|
453
|
+
stream(
|
|
454
|
+
{"type": "system", "subtype": "compact_boundary", "session_id": "s1"},
|
|
455
|
+
assistant_record([text_block(pass_text)]),
|
|
456
|
+
result_record(pass_text),
|
|
457
|
+
),
|
|
458
|
+
"stream that does not open with system/init",
|
|
459
|
+
"must open with system/init",
|
|
460
|
+
),
|
|
461
|
+
)
|
|
462
|
+
for text, label, message in stream_rejections:
|
|
463
|
+
assert_rejected(text, label, message)
|
|
105
464
|
return 0
|
|
106
465
|
|
|
107
466
|
|
|
@@ -122,9 +481,9 @@ def write_outputs(
|
|
|
122
481
|
def main() -> int:
|
|
123
482
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
124
483
|
parser.add_argument("--devlyn-dir", default=".devlyn")
|
|
125
|
-
parser.add_argument("--stdout-file", default="
|
|
484
|
+
parser.add_argument("--stdout-file", default="pair-judge.stdout")
|
|
126
485
|
parser.add_argument("--out", default="verify.pair.findings.jsonl")
|
|
127
|
-
parser.add_argument("--summary-out", default="
|
|
486
|
+
parser.add_argument("--summary-out", default="pair-judge.summary.json")
|
|
128
487
|
parser.add_argument("--self-test", action="store_true")
|
|
129
488
|
args = parser.parse_args()
|
|
130
489
|
if args.self_test:
|
|
@@ -135,7 +494,7 @@ def main() -> int:
|
|
|
135
494
|
if not stdout_path.is_file():
|
|
136
495
|
sys.stderr.write(f"error: {stdout_path} not found\n")
|
|
137
496
|
return 1
|
|
138
|
-
findings, summary =
|
|
497
|
+
findings, summary = collect_stdout(stdout_path)
|
|
139
498
|
write_outputs(findings, summary, devlyn / args.out, devlyn / args.summary_out)
|
|
140
499
|
print(json.dumps({"findings_count": len(findings), "summary": summary}, sort_keys=True))
|
|
141
500
|
return 0
|
|
@@ -8,21 +8,22 @@ Each skill resolves the effective engine from its own SKILL.md default plus any
|
|
|
8
8
|
|
|
9
9
|
Engine requirements have two classes:
|
|
10
10
|
|
|
11
|
-
- **Explicit routes** — `--engine`, `--risk-probes`, `--pair-verify
|
|
11
|
+
- **Explicit routes** — `--engine`, `--risk-probes`, `--pair-verify`, configured executor/role/priority pins. These are promises. If the required engine is unavailable, fail closed with `BLOCKED:<engine>-unavailable` and never downgrade to solo.
|
|
12
12
|
- **Automatic routes** — high-risk risk-probes remain conditional; VERIFY pair-JUDGE is selected by default unless `--no-pair` was passed. Both require an available OTHER engine. If it is absent, proceed solo and report the skipped route and reason. This is route selection, not a fallback.
|
|
13
13
|
|
|
14
14
|
## Role resolution
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
The orchestrator is the CLI the user opened. PLAN remains orchestrator-fixed. Resolve the legacy executor as explicit `--engine` > `cwd/.devlyn/engines.json` `executor` > the invoking skill's supported default. BUILD_GATE and risk-probe derivation keep their existing routes.
|
|
17
17
|
|
|
18
|
-
|
|
19
|
-
- **Pair judge** (VERIFY pair-JUDGE, risk-probe derivation): first entry of `engines.json` `pair_judge_priority` that is (a) adapter-valid, (b) pair-judge-eligible (below), (c) not the primary judge engine, (d) available. When the key or file is absent: the binary claude↔codex complement.
|
|
18
|
+
The role profiles and `--role-config` below apply only to `/devlyn:resolve`; `/devlyn:ideate` keeps its existing `--engine` / legacy executor selection and never inherits worker/judge profiles. `role-config.py` is resolve’s deterministic resolver. Optional project `roles.worker`, `roles.primary_judge`, and `roles.pair_judge` each require `engine` and accept exact `model`/`effort`; `/devlyn:resolve --role-config <path>` supplies the same `{"roles": {...}}` object for one run. Each entry replaces the entire lower entry. Worker/primary precedence is run role > `--engine` engine-only > project role > legacy executor. Pair precedence is run role > project role > existing `pair_judge_priority`/binary complement. An absent primary profile follows the legacy executor, not the worker profile. No parent/global devlyn config lookup.
|
|
20
19
|
|
|
21
|
-
|
|
20
|
+
Worker controls IMPLEMENT/CLEANUP and their repair rounds only; primary/pair control VERIFY only. OTHER must differ from the resolved primary engine. Explicit same-engine pair is invalid; a legacy priority list skips the primary. `--no-pair` does not dispatch or check availability of an unused pair. Configured pins remain explicit promises, while an unconfigured unavailable automatic OTHER records its solo reason.
|
|
22
21
|
|
|
23
|
-
|
|
22
|
+
Before the first phase, `/devlyn:resolve` freezes resolution with `state-phase-write.py --freeze-roles`; subsequent status/dispatch reads that state instead of mutable config. `state.engine` stays the legacy executor. The resolver rejects statically unsupported selected channel fields before PLAN, and validates adapter names and the existing ASCII `executor: yes|no` / `pair_judge: yes|no` eligibility markers; absent eligibility section means both. Malformed configuration fails with `BLOCKED:invalid-engine-config`.
|
|
24
23
|
|
|
25
|
-
|
|
24
|
+
Before each affected worker/judge, read `role-config.py --state .devlyn/pipeline.state.json --role <role> [--resolved-model <inherited-exact-model>]`. It returns source/channel, requested values and argv additions. Explicit Codex options require current version-matching native model metadata; refresh the native CLI's cache when missing/stale. Explicit Claude judge effort requires the adapter's version/exact-model capability declaration; model-only selection passes the exact ID without that effort declaration and validates the native result. No model/effort fields means unchanged phase defaults. Unknown support fails with `BLOCKED:unsupported-role-option`; no clamp, fallback or invented model identity. Native Agent workers reject unsupported explicit Claude fields without switching to a CLI worker. Show inherited/unknown values honestly; availability is not role fitness.
|
|
25
|
+
|
|
26
|
+
Engine-only adapter routes remain supported according to existing eligibility. New adapters do not automatically gain explicit model/effort mappings. Status reports unavailable pins and unresolved legacy channels without weakening run freeze/dispatch. CLI presence is not authentication. Status never launches a model, changes user configuration or promotes a default.
|
|
26
27
|
|
|
27
28
|
When a run or phase requires engine `<name>` (claude, codex, omp, or any adapter-backed engine), before spawning that phase:
|
|
28
29
|
|