devlyn-cli 2.10.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/AGENTS.md +4 -9
  2. package/CLAUDE.md +11 -15
  3. package/README.md +31 -7
  4. package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.json +25 -0
  5. package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.md +10 -0
  6. package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.json +18 -0
  7. package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.md +9 -0
  8. package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.json +18 -0
  9. package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.md +9 -0
  10. package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F16-cli-quote-tax-rules/l2_risk_probes/result.json +28 -0
  11. package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F23-cli-fulfillment-wave/l2_risk_probes/result.json +28 -0
  12. package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F25-cli-cart-promotion-rules/l2_risk_probes/result.json +30 -0
  13. package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.json +28 -0
  14. package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.md +15 -0
  15. package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.json +28 -0
  16. package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.md +15 -0
  17. package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/F21-cli-scheduler-priority/l2_risk_probes/result.json +32 -0
  18. package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.json +44 -0
  19. package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.md +16 -0
  20. package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.json +18 -0
  21. package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.md +9 -0
  22. package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.json +28 -0
  23. package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.md +15 -0
  24. package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.json +28 -0
  25. package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.md +15 -0
  26. package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.json +28 -0
  27. package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.md +15 -0
  28. package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.json +28 -0
  29. package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.md +15 -0
  30. package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.json +28 -0
  31. package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.md +15 -0
  32. package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.json +28 -0
  33. package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.md +15 -0
  34. package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.json +28 -0
  35. package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.md +15 -0
  36. package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.json +28 -0
  37. package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.md +15 -0
  38. package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.json +28 -0
  39. package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.md +15 -0
  40. package/benchmark/auto-resolve/scripts/run-fixture.sh +11 -1
  41. package/benchmark/auto-resolve/scripts/test-benchmark-arg-parsing.sh +83 -1
  42. package/bin/devlyn.js +59 -9
  43. package/config/skills/_shared/.ruff.toml +2 -0
  44. package/config/skills/_shared/adapters/claude.md +9 -1
  45. package/config/skills/_shared/adapters/codex.md +5 -0
  46. package/config/skills/_shared/adapters/grok.md +37 -13
  47. package/config/skills/_shared/archive_run.py +738 -56
  48. package/config/skills/_shared/codex-config.md +19 -4
  49. package/config/skills/_shared/codex-monitored.sh +81 -12
  50. package/config/skills/_shared/collect-codex-findings.py +419 -60
  51. package/config/skills/_shared/engine-preflight.md +8 -7
  52. package/config/skills/_shared/expected.schema.json +68 -0
  53. package/config/skills/_shared/finish-gate.py +12 -11
  54. package/config/skills/_shared/grok-anchor-guard.py +78 -0
  55. package/config/skills/_shared/invocation-receipt.py +1041 -0
  56. package/config/skills/_shared/judge-output-parser.py +200 -0
  57. package/config/skills/_shared/judge-role-evidence.py +273 -0
  58. package/config/skills/_shared/phase-prompt-render.py +181 -0
  59. package/config/skills/_shared/process-evidence.py +909 -0
  60. package/config/skills/_shared/resolve-bootstrap.py +776 -84
  61. package/config/skills/_shared/resolve-stop-hook.py +33 -1
  62. package/config/skills/_shared/role-config.py +552 -0
  63. package/config/skills/_shared/run-bounded.py +1 -1
  64. package/config/skills/_shared/spec-verify-check.py +1037 -192
  65. package/config/skills/_shared/state-phase-write.py +2551 -83
  66. package/config/skills/_shared/terminal-claim-check.py +15 -0
  67. package/config/skills/_shared/verify-merge-findings.py +1127 -50
  68. package/config/skills/devlyn:engines/SKILL.md +8 -16
  69. package/config/skills/devlyn:ideate/SKILL.md +1 -1
  70. package/config/skills/devlyn:ideate/references/spec-template.md +1 -0
  71. package/config/skills/devlyn:queue/SKILL.md +7 -4
  72. package/config/skills/devlyn:resolve/SKILL.md +94 -35
  73. package/config/skills/devlyn:resolve/references/outer-loop.md +15 -0
  74. package/config/skills/devlyn:resolve/references/phases/build-gate.md +33 -4
  75. package/config/skills/devlyn:resolve/references/phases/implement.md +4 -3
  76. package/config/skills/devlyn:resolve/references/phases/plan.md +1 -5
  77. package/config/skills/devlyn:resolve/references/phases/probe-derive.md +4 -2
  78. package/config/skills/devlyn:resolve/references/phases/verify.md +120 -80
  79. package/config/skills/devlyn:resolve/references/state-schema.md +23 -9
  80. package/package.json +5 -1
  81. package/scripts/lint-fixtures.sh +4 -1
  82. package/scripts/lint-skills.sh +447 -290
@@ -1,25 +1,42 @@
1
1
  #!/usr/bin/env python3
2
- """Normalize raw Codex pair-JUDGE stdout into canonical VERIFY JSONL."""
2
+ """Normalize raw pair-JUDGE stdout into canonical VERIFY JSONL."""
3
3
 
4
4
  from __future__ import annotations
5
5
 
6
6
  import argparse
7
7
  import json
8
8
  import pathlib
9
+ import runpy
9
10
  import sys
10
11
  import tempfile
11
12
  from typing import Any
12
13
 
13
14
 
14
- FINDING_SEVERITIES = {"CRITICAL", "HIGH", "MEDIUM", "LOW", "INFO"}
15
+ MERGE_CONTRACT = runpy.run_path(pathlib.Path(__file__).with_name("verify-merge-findings.py"))
16
+ PARSER = runpy.run_path(pathlib.Path(__file__).with_name("judge-output-parser.py"))
17
+ VERDICT_RANK = MERGE_CONTRACT["VERDICT_RANK"]
18
+ finding_rank = MERGE_CONTRACT["finding_rank"]
15
19
 
16
20
 
17
21
  def reject_json_constant(token: str) -> None:
18
22
  raise ValueError(f"invalid JSON numeric constant: {token}")
19
23
 
20
24
 
25
+ def reject_duplicate_keys(pairs: list[tuple[str, object]]) -> dict:
26
+ result = {}
27
+ for key, value in pairs:
28
+ if key in result:
29
+ raise ValueError(f"duplicate JSON key: {key}")
30
+ result[key] = value
31
+ return result
32
+
33
+
21
34
  def loads_strict_json(text: str) -> Any:
22
- return json.loads(text, parse_constant=reject_json_constant)
35
+ return json.loads(
36
+ text,
37
+ parse_constant=reject_json_constant,
38
+ object_pairs_hook=reject_duplicate_keys,
39
+ )
23
40
 
24
41
 
25
42
  def atomic_write(path: pathlib.Path, text: str) -> None:
@@ -32,76 +49,418 @@ def atomic_write(path: pathlib.Path, text: str) -> None:
32
49
  pathlib.Path(tmp_name).replace(path)
33
50
 
34
51
 
35
- def collect(stdout_path: pathlib.Path) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
36
- findings: list[dict[str, Any]] = []
37
- summary: dict[str, Any] | None = None
38
- with stdout_path.open(encoding="utf-8") as handle:
39
- for line_no, line in enumerate(handle, 1):
40
- raw = line.strip()
41
- if not raw:
42
- continue
43
- if raw.startswith("# SUMMARY "):
44
- try:
45
- item = loads_strict_json(raw.removeprefix("# SUMMARY ").strip())
46
- except ValueError as exc:
47
- raise SystemExit(f"error: invalid SUMMARY JSON at {stdout_path}:{line_no}: {exc}")
48
- if not isinstance(item, dict):
49
- raise SystemExit(f"error: SUMMARY is not an object at {stdout_path}:{line_no}")
50
- summary = item
51
- continue
52
- if raw.startswith("#"):
53
- continue
54
- try:
55
- item = loads_strict_json(raw)
56
- except ValueError as exc:
57
- raise SystemExit(f"error: invalid JSONL at {stdout_path}:{line_no}: {exc}")
58
- if not isinstance(item, dict):
59
- raise SystemExit(f"error: JSONL item is not an object at {stdout_path}:{line_no}")
60
- severity = str(item.get("severity") or "").upper()
61
- if severity not in FINDING_SEVERITIES:
62
- raise SystemExit(f"error: finding missing valid severity at {stdout_path}:{line_no}")
63
- findings.append(item)
52
+ def collect_text(
53
+ text: str, source: pathlib.Path
54
+ ) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
55
+ findings, summary = PARSER["collect_text"](text, source)
56
+ if findings and summary is None:
57
+ raise SystemExit("error: findings without terminal verdict")
58
+ if summary is not None and summary["verdict"] == "PASS" and any(finding_rank(finding) == 2 for finding in findings):
59
+ raise SystemExit("error: verdict-binding finding cannot have a PASS verdict")
60
+ if not findings and (summary is None or summary.get("verdict") != "PASS"):
61
+ raise SystemExit("error: non-PASS verdict without JSONL findings")
62
+ return findings, summary
63
+
64
+
65
+ def collect_stdout(stdout_path: pathlib.Path) -> tuple[list[dict[str, Any]], dict[str, Any] | None]:
66
+ findings, summary = PARSER["collect_stdout"](stdout_path)
67
+ if findings and summary is None:
68
+ raise SystemExit("error: findings without terminal verdict")
69
+ if summary is not None and summary["verdict"] == "PASS" and any(finding_rank(finding) == 2 for finding in findings):
70
+ raise SystemExit("error: verdict-binding finding cannot have a PASS verdict")
64
71
  if not findings and (summary is None or summary.get("verdict") != "PASS"):
65
- raise SystemExit("error: non-PASS SUMMARY without JSONL findings")
72
+ raise SystemExit("error: non-PASS verdict without JSONL findings")
66
73
  return findings, summary
67
74
 
68
75
 
69
76
  def self_test() -> int:
77
+ try:
78
+ loads_strict_json('{"verdict":"PASS","verdict":"BLOCKED"}')
79
+ except ValueError as exc:
80
+ assert "duplicate JSON key" in str(exc)
81
+ else:
82
+ raise AssertionError("duplicate judge result key was accepted")
70
83
  with tempfile.TemporaryDirectory() as tmp:
71
84
  root = pathlib.Path(tmp)
72
- stdout_path = root / "codex-judge.stdout"
85
+ stdout_path = root / "pair-judge.stdout"
73
86
  out_path = root / "verify.pair.findings.jsonl"
74
- summary_path = root / "codex-judge.summary.json"
87
+ summary_path = root / "pair-judge.summary.json"
88
+
89
+ def assert_rejected(text: str, label: str, *message_parts: str) -> None:
90
+ stdout_path.write_text(text, encoding="utf-8")
91
+ try:
92
+ collect_stdout(stdout_path)
93
+ except SystemExit as exc:
94
+ message = str(exc)
95
+ assert all(part in message for part in message_parts)
96
+ else:
97
+ raise AssertionError(f"{label} must be rejected")
98
+
99
+ plain_finding = {"id": "a", "severity": "HIGH"}
100
+ plain_summary = {"verdict": "NEEDS_WORK"}
75
101
  stdout_path.write_text(
76
- json.dumps({"id": "a", "severity": "HIGH"}) + "\n"
77
- + '# SUMMARY {"verdict":"NEEDS_WORK"}\n',
102
+ json.dumps(plain_finding) + "\n"
103
+ + plain_summary["verdict"] + "\n",
78
104
  encoding="utf-8",
79
105
  )
80
- findings, summary = collect(stdout_path)
106
+ findings, summary = collect_stdout(stdout_path)
107
+ assert findings == [plain_finding]
108
+ assert summary == plain_summary
81
109
  write_outputs(findings, summary, out_path, summary_path)
82
110
  assert out_path.read_text(encoding="utf-8").count("\n") == 1
83
111
  assert loads_strict_json(summary_path.read_text(encoding="utf-8"))["verdict"] == "NEEDS_WORK"
84
- stdout_path.write_text('{"id":"nan","severity":NaN}\n', encoding="utf-8")
85
- try:
86
- collect(stdout_path)
87
- except SystemExit as exc:
88
- assert "invalid JSON numeric constant: NaN" in str(exc)
89
- else:
90
- raise AssertionError("NaN Codex stdout finding must not normalize")
112
+ assert_rejected(
113
+ '{"id":"nan","severity":NaN}\n',
114
+ "NaN pair-JUDGE stdout finding",
115
+ "invalid JSON numeric constant: NaN",
116
+ )
117
+ assert_rejected(
118
+ '# SUMMARY {"verdict":"PASS","verdict":"BLOCKED"}\n',
119
+ "duplicate pair-JUDGE summary verdict",
120
+ "duplicate JSON key: verdict",
121
+ )
91
122
  rejection_cases = (
92
- ("", "no SUMMARY line"),
93
- ('# SUMMARY {}\n', "empty SUMMARY object"),
94
- ('# SUMMARY {"verdict":"pass"}\n', "lowercase pass"),
95
- ('# SUMMARY {"verdict":"UNKNOWN"}\n', "unknown verdict"),
123
+ ("", "no verdict line", "non-PASS verdict without JSONL findings"),
124
+ (json.dumps(plain_finding) + "\n", "finding without terminal verdict", "findings without terminal verdict"),
125
+ ('{"verdict":"pass"}\n', "lowercase pass", "finding missing valid severity"),
126
+ ('{"verdict":"UNKNOWN"}\n', "unknown verdict", "finding missing valid severity"),
96
127
  )
97
- for text, label in rejection_cases:
98
- stdout_path.write_text(text, encoding="utf-8")
99
- try:
100
- collect(stdout_path)
101
- except SystemExit as exc:
102
- assert str(exc) == "error: non-PASS SUMMARY without JSONL findings"
103
- else:
104
- raise AssertionError(f"{label} must not normalize to PASS")
128
+ for text, label, message in rejection_cases:
129
+ assert_rejected(
130
+ text,
131
+ label,
132
+ message,
133
+ )
134
+
135
+ def envelope(text: Any, stop_reason: str = "EndTurn") -> str:
136
+ return json.dumps(
137
+ {
138
+ "text": text,
139
+ "stopReason": stop_reason,
140
+ "sessionId": "session",
141
+ "requestId": "request",
142
+ }
143
+ )
144
+
145
+ envelope_finding = {
146
+ "id": "envelope-finding",
147
+ "severity": "HIGH",
148
+ "detail": {"preserved": True},
149
+ }
150
+ envelope_summary = {"verdict": "NEEDS_WORK"}
151
+ stdout_path.write_text(
152
+ envelope(
153
+ json.dumps(envelope_finding)
154
+ + "\n" + envelope_summary["verdict"] + "\n"
155
+ ),
156
+ encoding="utf-8",
157
+ )
158
+ findings, summary = collect_stdout(stdout_path)
159
+ assert findings == [envelope_finding]
160
+ assert summary == envelope_summary
161
+
162
+ assert_rejected(
163
+ envelope("review completed without findings"),
164
+ "findings-less envelope text",
165
+ "invalid JSONL",
166
+ )
167
+ assert_rejected(
168
+ envelope('{"findings":[],"verdict":"PASS"}', "Cancelled"),
169
+ "Cancelled envelope",
170
+ "envelope stopReason must be 'EndTurn'",
171
+ "'Cancelled'",
172
+ )
173
+ assert_rejected(
174
+ envelope("", "UnknownStop"),
175
+ "unknown envelope stopReason",
176
+ "envelope stopReason must be 'EndTurn'",
177
+ "'UnknownStop'",
178
+ )
179
+ assert_rejected(
180
+ envelope(None),
181
+ "non-string envelope text",
182
+ "error: envelope text must be a string",
183
+ )
184
+ assert_rejected(
185
+ envelope("I will inspect the result.\n" + envelope_summary["verdict"] + "\n"),
186
+ "narrative envelope preamble",
187
+ "invalid JSONL",
188
+ )
189
+ dual_document = (
190
+ json.dumps(envelope_finding)
191
+ + json.dumps({"id": "second", "severity": "LOW"})
192
+ + "\n" + envelope_summary["verdict"] + "\n"
193
+ )
194
+ assert_rejected(
195
+ envelope(dual_document),
196
+ "concatenated envelope documents",
197
+ "invalid JSONL",
198
+ )
199
+ clean_summary = {"verdict": "PASS"}
200
+ stdout_path.write_text(
201
+ envelope(clean_summary["verdict"] + "\n"),
202
+ encoding="utf-8",
203
+ )
204
+ findings, summary = collect_stdout(stdout_path)
205
+ assert findings == []
206
+ assert summary == clean_summary
207
+ stdout_path.write_text(
208
+ json.dumps(
209
+ {
210
+ "type": "system",
211
+ "text": clean_summary["verdict"] + "\n",
212
+ "stopReason": "EndTurn",
213
+ "sessionId": "session",
214
+ "requestId": "request",
215
+ }
216
+ ),
217
+ encoding="utf-8",
218
+ )
219
+ findings, summary = collect_stdout(stdout_path)
220
+ assert findings == []
221
+ assert summary == clean_summary
222
+
223
+ # R4: a valid plain finding may carry an unrelated `type` field. A
224
+ # first-line `type: system` finding is not a stream discriminator.
225
+ system_finding = {"id": "system-finding", "severity": "LOW", "type": "system"}
226
+ stdout_path.write_text(
227
+ json.dumps(system_finding) + "\n" + clean_summary["verdict"] + "\n",
228
+ encoding="utf-8",
229
+ )
230
+ findings, summary = collect_stdout(stdout_path)
231
+ assert findings == [system_finding]
232
+ assert summary == clean_summary
233
+
234
+ # iter-0106 — Grok whole-message NDJSON carrier (R2/R3).
235
+ def init_record(session: str = "s1") -> dict[str, Any]:
236
+ return {"type": "system", "subtype": "init", "session_id": session, "model": "grok-build"}
237
+
238
+ def assistant_record(
239
+ blocks: list[Any], stop_reason: str = "end_turn", session: str = "s1"
240
+ ) -> dict[str, Any]:
241
+ return {
242
+ "type": "assistant",
243
+ "message": {"id": "msg", "role": "assistant", "content": blocks, "stop_reason": stop_reason},
244
+ "parent_tool_use_id": None,
245
+ "session_id": session,
246
+ }
247
+
248
+ def result_record(
249
+ text: str,
250
+ session: str = "s1",
251
+ subtype: str = "success",
252
+ is_error: Any = False,
253
+ stop_reason: str = "end_turn",
254
+ ) -> dict[str, Any]:
255
+ return {
256
+ "type": "result",
257
+ "subtype": subtype,
258
+ "is_error": is_error,
259
+ "result": text,
260
+ "stop_reason": stop_reason,
261
+ "session_id": session,
262
+ }
263
+
264
+ def text_block(value: str) -> dict[str, Any]:
265
+ return {"type": "text", "text": value}
266
+
267
+ def stream(*records: Any) -> str:
268
+ return "".join(json.dumps(record) + "\n" for record in records)
269
+
270
+ pass_text = "PASS\n"
271
+ stdout_path.write_text(
272
+ stream(init_record(), assistant_record([text_block(pass_text)]), result_record(pass_text)),
273
+ encoding="utf-8",
274
+ )
275
+ findings, summary = collect_stdout(stdout_path)
276
+ assert findings == []
277
+ assert summary == {"verdict": "PASS"}
278
+
279
+ # A real tool turn precedes the contract, and the terminal message arrives
280
+ # as two text blocks whose in-order concatenation is the result text.
281
+ stream_finding = {"id": "stream-finding", "severity": "HIGH"}
282
+ head = json.dumps(stream_finding) + "\n"
283
+ tail = "# SUMMARY " + json.dumps({"verdict": "NEEDS_WORK"}) + "\n"
284
+ contract = head + tail
285
+ narration = assistant_record(
286
+ [
287
+ text_block("Let me read the file."),
288
+ {"type": "tool_use", "id": "call_1", "name": "read_file", "input": {"path": "src/main.rs"}},
289
+ ],
290
+ stop_reason="tool_use",
291
+ )
292
+ tool_result = {
293
+ "type": "user",
294
+ "message": {
295
+ "role": "user",
296
+ "content": [{"type": "tool_result", "tool_use_id": "call_1", "content": "fn main() {}"}],
297
+ },
298
+ "parent_tool_use_id": None,
299
+ "session_id": "s1",
300
+ }
301
+ stdout_path.write_text(
302
+ stream(
303
+ init_record(),
304
+ narration,
305
+ tool_result,
306
+ assistant_record([text_block(head), text_block(tail)]),
307
+ result_record(contract),
308
+ ),
309
+ encoding="utf-8",
310
+ )
311
+ findings, summary = collect_stdout(stdout_path)
312
+ assert findings == [stream_finding]
313
+ assert summary == {"verdict": "NEEDS_WORK"}
314
+
315
+ # The welded body is exactly the shape envelope recovery accepts, so this
316
+ # also proves the stream branch never falls through to that recovery.
317
+ welded = "I reviewed the diff.\n" + contract
318
+ stream_rejections = (
319
+ (
320
+ stream(init_record(), assistant_record([text_block(welded)]), result_record(welded)),
321
+ "narration welded into the terminal message",
322
+ "invalid JSONL",
323
+ ),
324
+ (
325
+ stream(
326
+ init_record(),
327
+ assistant_record([text_block(contract)]),
328
+ assistant_record([text_block(contract)]),
329
+ result_record(contract),
330
+ ),
331
+ "two terminal end_turn messages",
332
+ "exactly one final end_turn assistant message",
333
+ ),
334
+ (
335
+ stream(
336
+ init_record(),
337
+ assistant_record([text_block(contract)]),
338
+ assistant_record([text_block("more")], stop_reason="tool_use"),
339
+ result_record(contract),
340
+ ),
341
+ "end_turn message that is not the final assistant message",
342
+ "exactly one final end_turn assistant message",
343
+ ),
344
+ (
345
+ # The malformed line is the only defect: skipping it leaves a valid stream.
346
+ stream(init_record(), assistant_record([text_block(pass_text)]))
347
+ + '{"type":"assistant"\n'
348
+ + stream(result_record(pass_text)),
349
+ "malformed stream NDJSON inside an otherwise valid stream",
350
+ "invalid stream NDJSON",
351
+ ),
352
+ (
353
+ stream(init_record(), assistant_record([text_block(pass_text)])),
354
+ "stream without a terminal result",
355
+ "exactly one terminal result",
356
+ ),
357
+ (
358
+ stream(
359
+ init_record(),
360
+ assistant_record([text_block(pass_text)]),
361
+ result_record(pass_text),
362
+ tool_result,
363
+ ),
364
+ "data after the terminal result",
365
+ "exactly one terminal result",
366
+ ),
367
+ (
368
+ stream(
369
+ init_record(),
370
+ assistant_record([text_block(pass_text)]),
371
+ tool_result,
372
+ result_record(pass_text),
373
+ ),
374
+ "data between the terminal assistant and result",
375
+ "exactly one final end_turn assistant message",
376
+ ),
377
+ (
378
+ stream(
379
+ init_record(),
380
+ assistant_record([text_block(pass_text)]),
381
+ result_record(pass_text, subtype="error_during_execution"),
382
+ ),
383
+ "error result subtype",
384
+ "not a successful end_turn",
385
+ ),
386
+ (
387
+ stream(
388
+ init_record(),
389
+ assistant_record([text_block(pass_text)]),
390
+ result_record(pass_text, stop_reason="cancelled"),
391
+ ),
392
+ "cancelled result",
393
+ "not a successful end_turn",
394
+ ),
395
+ (
396
+ stream(
397
+ init_record(),
398
+ assistant_record([text_block(pass_text)]),
399
+ result_record(pass_text, is_error=True),
400
+ ),
401
+ "error-flagged result",
402
+ "not a successful end_turn",
403
+ ),
404
+ (
405
+ stream(
406
+ init_record(),
407
+ assistant_record([text_block(pass_text)]),
408
+ result_record("NEEDS_WORK\n"),
409
+ ),
410
+ "result text that disagrees with the terminal message",
411
+ "result text does not match the terminal message",
412
+ ),
413
+ (
414
+ stream(
415
+ init_record(),
416
+ assistant_record([text_block(pass_text)]),
417
+ result_record(pass_text, session="s2"),
418
+ ),
419
+ "result from another session",
420
+ "session identity does not agree",
421
+ ),
422
+ (
423
+ stream(
424
+ init_record(),
425
+ {"type": "stream_event", "event": {"type": "content_block_delta"}, "session_id": "s1"},
426
+ assistant_record([text_block(pass_text)]),
427
+ result_record(pass_text),
428
+ ),
429
+ "partial-message frame",
430
+ "unknown stream record",
431
+ ),
432
+ (
433
+ stream(
434
+ init_record(),
435
+ {"type": "system", "subtype": "compact_boundary", "session_id": "s1"},
436
+ assistant_record([text_block(pass_text)]),
437
+ result_record(pass_text),
438
+ ),
439
+ "compact boundary",
440
+ "second system record",
441
+ ),
442
+ (
443
+ stream(
444
+ init_record(),
445
+ {"type": "unknown", "session_id": "s1"},
446
+ assistant_record([text_block(pass_text)]),
447
+ result_record(pass_text),
448
+ ),
449
+ "unknown record type",
450
+ "unknown stream record",
451
+ ),
452
+ (
453
+ stream(
454
+ {"type": "system", "subtype": "compact_boundary", "session_id": "s1"},
455
+ assistant_record([text_block(pass_text)]),
456
+ result_record(pass_text),
457
+ ),
458
+ "stream that does not open with system/init",
459
+ "must open with system/init",
460
+ ),
461
+ )
462
+ for text, label, message in stream_rejections:
463
+ assert_rejected(text, label, message)
105
464
  return 0
106
465
 
107
466
 
@@ -122,9 +481,9 @@ def write_outputs(
122
481
  def main() -> int:
123
482
  parser = argparse.ArgumentParser(description=__doc__)
124
483
  parser.add_argument("--devlyn-dir", default=".devlyn")
125
- parser.add_argument("--stdout-file", default="codex-judge.stdout")
484
+ parser.add_argument("--stdout-file", default="pair-judge.stdout")
126
485
  parser.add_argument("--out", default="verify.pair.findings.jsonl")
127
- parser.add_argument("--summary-out", default="codex-judge.summary.json")
486
+ parser.add_argument("--summary-out", default="pair-judge.summary.json")
128
487
  parser.add_argument("--self-test", action="store_true")
129
488
  args = parser.parse_args()
130
489
  if args.self_test:
@@ -135,7 +494,7 @@ def main() -> int:
135
494
  if not stdout_path.is_file():
136
495
  sys.stderr.write(f"error: {stdout_path} not found\n")
137
496
  return 1
138
- findings, summary = collect(stdout_path)
497
+ findings, summary = collect_stdout(stdout_path)
139
498
  write_outputs(findings, summary, devlyn / args.out, devlyn / args.summary_out)
140
499
  print(json.dumps({"findings_count": len(findings), "summary": summary}, sort_keys=True))
141
500
  return 0
@@ -8,21 +8,22 @@ Each skill resolves the effective engine from its own SKILL.md default plus any
8
8
 
9
9
  Engine requirements have two classes:
10
10
 
11
- - **Explicit routes** — `--engine`, `--risk-probes`, `--pair-verify`. These are promises. If the required engine is unavailable, fail closed with `BLOCKED:<engine>-unavailable` and never downgrade to solo.
11
+ - **Explicit routes** — `--engine`, `--risk-probes`, `--pair-verify`, configured executor/role/priority pins. These are promises. If the required engine is unavailable, fail closed with `BLOCKED:<engine>-unavailable` and never downgrade to solo.
12
12
  - **Automatic routes** — high-risk risk-probes remain conditional; VERIFY pair-JUDGE is selected by default unless `--no-pair` was passed. Both require an available OTHER engine. If it is absent, proceed solo and report the skipped route and reason. This is route selection, not a fallback.
13
13
 
14
14
  ## Role resolution
15
15
 
16
- Two configurable roles; the orchestrator role is whichever CLI the user opened and is not configurable here.
16
+ The orchestrator is the CLI the user opened. PLAN remains orchestrator-fixed. Resolve the legacy executor as explicit `--engine` > `cwd/.devlyn/engines.json` `executor` > the invoking skill's supported default. BUILD_GATE and risk-probe derivation keep their existing routes.
17
17
 
18
- - **Executor** (IMPLEMENT / CLEANUP / primary VERIFY judge): explicit `--engine` flag > `cwd/.devlyn/engines.json` `executor` > built-in default `claude`. No parent-directory or global config lookup. Requires an adapter that is executor-eligible (below).
19
- - **Pair judge** (VERIFY pair-JUDGE, risk-probe derivation): first entry of `engines.json` `pair_judge_priority` that is (a) adapter-valid, (b) pair-judge-eligible (below), (c) not the primary judge engine, (d) available. When the key or file is absent: the binary claude↔codex complement.
18
+ The role profiles and `--role-config` below apply only to `/devlyn:resolve`; `/devlyn:ideate` keeps its existing `--engine` / legacy executor selection and never inherits worker/judge profiles. `role-config.py` is resolve’s deterministic resolver. Optional project `roles.worker`, `roles.primary_judge`, and `roles.pair_judge` each require `engine` and accept exact `model`/`effort`; `/devlyn:resolve --role-config <path>` supplies the same `{"roles": {...}}` object for one run. Each entry replaces the entire lower entry. Worker/primary precedence is run role > `--engine` engine-only > project role > legacy executor. Pair precedence is run role > project role > existing `pair_judge_priority`/binary complement. An absent primary profile follows the legacy executor, not the worker profile. No parent/global devlyn config lookup.
20
19
 
21
- Validation is fail-closed: malformed JSON, any engine name without a `_shared/adapters/<name>.md` adapter file under `DEVLYN_SHARED_DIR`, or an adapter that declares itself ineligible for the requested role, halts with report-level `BLOCKED:invalid-engine-config` naming the offending entry. This is the plug-in point for new engines — ship an adapter file and the name becomes valid with zero skill-body changes. Availability probe default is `command -v <name>`; an adapter may declare a different probe. A pin is an explicit route (same promise class as `--engine`): pinned-but-unavailable → `BLOCKED:<engine>-unavailable`, never a silent downgrade. `engines.json` is machine-local config — not committed, not archived (`archive_run.py` moves only `PER_RUN_PATTERNS`).
20
+ Worker controls IMPLEMENT/CLEANUP and their repair rounds only; primary/pair control VERIFY only. OTHER must differ from the resolved primary engine. Explicit same-engine pair is invalid; a legacy priority list skips the primary. `--no-pair` does not dispatch or check availability of an unused pair. Configured pins remain explicit promises, while an unconfigured unavailable automatic OTHER records its solo reason.
22
21
 
23
- **Role eligibility** — an adapter's optional `## Role eligibility` section (`_shared/adapters/README.md`) declares `executor: yes|no` / `pair_judge: yes|no` as fixed ASCII fields. Absent section = eligible for both roles (every adapter shipped before iter-0051). A completion-only local backend with no tool/file-edit access declares `executor: no`: an `executor <name>` pin against it is invalid config, not an availability gap, and fails the same `BLOCKED:invalid-engine-config` path above rather than `BLOCKED:<name>-unavailable`.
22
+ Before the first phase, `/devlyn:resolve` freezes resolution with `state-phase-write.py --freeze-roles`; subsequent status/dispatch reads that state instead of mutable config. `state.engine` stays the legacy executor. The resolver rejects statically unsupported selected channel fields before PLAN, and validates adapter names and the existing ASCII `executor: yes|no` / `pair_judge: yes|no` eligibility markers; absent eligibility section means both. Malformed configuration fails with `BLOCKED:invalid-engine-config`.
24
23
 
25
- The explicit-vs-automatic split above applies unchanged to resolved engines: explicit routes (`--engine`, `--risk-probes`, `--pair-verify`, config pins) fail closed; automatic escalations select a cross-engine route only when the resolved pair-judge engine is available, else proceed solo and report the skip.
24
+ Before each affected worker/judge, read `role-config.py --state .devlyn/pipeline.state.json --role <role> [--resolved-model <inherited-exact-model>]`. It returns source/channel, requested values and argv additions. Explicit Codex options require current version-matching native model metadata; refresh the native CLI's cache when missing/stale. Explicit Claude judge effort requires the adapter's version/exact-model capability declaration; model-only selection passes the exact ID without that effort declaration and validates the native result. No model/effort fields means unchanged phase defaults. Unknown support fails with `BLOCKED:unsupported-role-option`; no clamp, fallback or invented model identity. Native Agent workers reject unsupported explicit Claude fields without switching to a CLI worker. Show inherited/unknown values honestly; availability is not role fitness.
25
+
26
+ Engine-only adapter routes remain supported according to existing eligibility. New adapters do not automatically gain explicit model/effort mappings. Status reports unavailable pins and unresolved legacy channels without weakening run freeze/dispatch. CLI presence is not authentication. Status never launches a model, changes user configuration or promotes a default.
26
27
 
27
28
  When a run or phase requires engine `<name>` (claude, codex, omp, or any adapter-backed engine), before spawning that phase:
28
29