devlyn-cli 2.10.1 → 3.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +8 -9
- package/CLAUDE.md +17 -17
- package/README.md +33 -9
- package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.json +25 -0
- package/benchmark/auto-resolve/results/20260507-f10-f11-tier1-full-pipeline/headroom-gate.md +10 -0
- package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260508-f22-exact-error-headroom/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260508-f26-headroom/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F16-cli-quote-tax-rules/l2_risk_probes/result.json +28 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F23-cli-fulfillment-wave/l2_risk_probes/result.json +28 -0
- package/benchmark/auto-resolve/results/20260510-f16-f23-f25-combined-proof/F25-cli-cart-promotion-rules/l2_risk_probes/result.json +30 -0
- package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f12-webhook-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f15-concurrency-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/F21-cli-scheduler-priority/l2_risk_probes/result.json +32 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.json +44 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.md +16 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.json +18 -0
- package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/headroom-gate.md +9 -0
- package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260511-f3-http-error-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f2-medium-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f4-web-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f5-fixloop-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f6-checksum-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f7-scope-headroom/headroom-gate.md +15 -0
- package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.json +28 -0
- package/benchmark/auto-resolve/results/20260512-f9-e2e-headroom/headroom-gate.md +15 -0
- package/bin/devlyn.js +9 -2
- package/config/skills/_shared/.ruff.toml +2 -0
- package/config/skills/_shared/adapters/claude.md +9 -1
- package/config/skills/_shared/adapters/codex.md +5 -0
- package/config/skills/_shared/adapters/grok.md +4 -2
- package/config/skills/_shared/archive_run.py +738 -56
- package/config/skills/_shared/codex-config.md +19 -4
- package/config/skills/_shared/codex-monitored.sh +81 -12
- package/config/skills/_shared/collect-codex-findings.py +311 -128
- package/config/skills/_shared/engine-preflight.md +8 -7
- package/config/skills/_shared/expected.schema.json +61 -0
- package/config/skills/_shared/finish-gate.py +12 -11
- package/config/skills/_shared/grok-anchor-guard.py +18 -1
- package/config/skills/_shared/invocation-receipt.py +1041 -0
- package/config/skills/_shared/judge-output-parser.py +200 -0
- package/config/skills/_shared/judge-role-evidence.py +273 -0
- package/config/skills/_shared/process-evidence.py +909 -0
- package/config/skills/_shared/resolve-bootstrap.py +776 -84
- package/config/skills/_shared/resolve-stop-hook.py +33 -1
- package/config/skills/_shared/role-config.py +552 -0
- package/config/skills/_shared/run-bounded.py +1 -1
- package/config/skills/_shared/spec-verify-check.py +749 -181
- package/config/skills/_shared/state-phase-write.py +1838 -58
- package/config/skills/_shared/terminal-claim-check.py +15 -0
- package/config/skills/_shared/verify-merge-findings.py +958 -51
- package/config/skills/devlyn:engines/SKILL.md +8 -16
- package/config/skills/devlyn:ideate/SKILL.md +2 -2
- package/config/skills/devlyn:ideate/references/spec-template.md +6 -5
- package/config/skills/devlyn:queue/SKILL.md +7 -4
- package/config/skills/devlyn:resolve/SKILL.md +93 -35
- package/config/skills/devlyn:resolve/references/free-form-mode.md +1 -0
- package/config/skills/devlyn:resolve/references/outer-loop.md +15 -0
- package/config/skills/devlyn:resolve/references/phases/build-gate.md +31 -2
- package/config/skills/devlyn:resolve/references/phases/implement.md +3 -2
- package/config/skills/devlyn:resolve/references/phases/verify.md +120 -84
- package/config/skills/devlyn:resolve/references/state-schema.md +22 -8
- package/package.json +5 -1
- package/scripts/lint-skills.sh +314 -285
package/benchmark/auto-resolve/results/20260511-f21-current-riskprobes-v1/full-pipeline-pair-gate.md
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Full-Pipeline Pair Gate - 20260511-f21-current-riskprobes-v1
|
|
2
|
+
|
|
3
|
+
Verdict: **PASS**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 1/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60; bare headroom >= 5; solo_claude <= 80; solo_claude headroom >= 5; l2_risk_probes evidence-clean; pair_mode true; pair_trigger eligible with canonical reason; l2_risk_probes - solo_claude >= 5.
|
|
8
|
+
Average pair margin: +33.0
|
|
9
|
+
Allowed pair/solo wall ratio: 3.00x
|
|
10
|
+
Maximum observed pair/solo wall ratio: 1.47x
|
|
11
|
+
Average pair/solo wall ratio: 1.47x
|
|
12
|
+
Hypothesis trigger required: false
|
|
13
|
+
|
|
14
|
+
| Fixture | Bare | Bare headroom | Solo_claude | Solo_claude headroom | Pair | Margin | Pair mode | Hypothesis trigger | Triggers | Wall ratio | Status | Reason |
|
|
15
|
+
|---|---:|---:|---:|---:|---:|---:|---|---|---|---:|---|---|
|
|
16
|
+
| F21-cli-scheduler-priority | 33 | 27 | 66 | 14 | 99 | +33 | true | true | complexity.high,risk.high,risk_probes.enabled,spec.solo_headroom_hypothesis | 1.47x | PASS | |
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260511-f21-current-riskprobes-v1",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 and solo_claude <= 80, with both arms clean",
|
|
4
|
+
"verdict": "PASS",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 1,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"fixture_count_ok": true,
|
|
9
|
+
"rows": [
|
|
10
|
+
{
|
|
11
|
+
"fixture": "F21-cli-scheduler-priority",
|
|
12
|
+
"status": "PASS",
|
|
13
|
+
"bare_score": 33,
|
|
14
|
+
"solo_score": 66,
|
|
15
|
+
"reason": ""
|
|
16
|
+
}
|
|
17
|
+
]
|
|
18
|
+
}
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Headroom Gate — 20260511-f21-current-riskprobes-v1
|
|
2
|
+
|
|
3
|
+
Verdict: **PASS**
|
|
4
|
+
|
|
5
|
+
Rule: at least 1 fixtures; bare <= 60, solo_claude <= 80, both arms clean.
|
|
6
|
+
|
|
7
|
+
| Fixture | Bare | Solo | Status | Reason |
|
|
8
|
+
|---|---:|---:|---|---|
|
|
9
|
+
| F21-cli-scheduler-priority | 33 | 66 | PASS | |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260511-f3-http-error-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -37.0,
|
|
14
|
+
"min_bare_headroom": -37,
|
|
15
|
+
"avg_solo_headroom": -19.0,
|
|
16
|
+
"min_solo_headroom": -19,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F3-backend-contract-risk",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 97,
|
|
22
|
+
"solo_score": 99,
|
|
23
|
+
"bare_headroom": -37,
|
|
24
|
+
"solo_headroom": -19,
|
|
25
|
+
"reason": "bare score 97 > 60; solo_claude score 99 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260511-f3-http-error-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -37.0
|
|
9
|
+
Minimum bare headroom: -37
|
|
10
|
+
Average solo headroom: -19.0
|
|
11
|
+
Minimum solo headroom: -19
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F3-backend-contract-risk | 97 | -37 | 99 | -19 | FAIL | bare score 97 > 60; solo_claude score 99 > 80 |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f2-medium-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -23.0,
|
|
14
|
+
"min_bare_headroom": -23,
|
|
15
|
+
"avg_solo_headroom": -15.0,
|
|
16
|
+
"min_solo_headroom": -15,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F2-cli-medium-subcommand",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 83,
|
|
22
|
+
"solo_score": 95,
|
|
23
|
+
"bare_headroom": -23,
|
|
24
|
+
"solo_headroom": -15,
|
|
25
|
+
"reason": "bare score 83 > 60; solo_claude score 95 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f2-medium-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -23.0
|
|
9
|
+
Minimum bare headroom: -23
|
|
10
|
+
Average solo headroom: -15.0
|
|
11
|
+
Minimum solo headroom: -15
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F2-cli-medium-subcommand | 83 | -23 | 95 | -15 | FAIL | bare score 83 > 60; solo_claude score 95 > 80 |
|
package/benchmark/auto-resolve/results/20260512-f31-seat-rebalance-headroom/headroom-gate.json
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f31-seat-rebalance-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": 27.0,
|
|
14
|
+
"min_bare_headroom": 27,
|
|
15
|
+
"avg_solo_headroom": -18.0,
|
|
16
|
+
"min_solo_headroom": -18,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F31-cli-seat-rebalance",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 33,
|
|
22
|
+
"solo_score": 98,
|
|
23
|
+
"bare_headroom": 27,
|
|
24
|
+
"solo_headroom": -18,
|
|
25
|
+
"reason": "solo_claude score 98 > 80; bare judge disqualifier; bare result disqualifier; bare verify disqualifier"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f31-seat-rebalance-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: 27.0
|
|
9
|
+
Minimum bare headroom: 27
|
|
10
|
+
Average solo headroom: -18.0
|
|
11
|
+
Minimum solo headroom: -18
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F31-cli-seat-rebalance | 33 | 27 | 98 | -18 | FAIL | solo_claude score 98 > 80; bare judge disqualifier; bare result disqualifier; bare verify disqualifier |
|
package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.json
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f32-subscription-renewal-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": 27.0,
|
|
14
|
+
"min_bare_headroom": 27,
|
|
15
|
+
"avg_solo_headroom": -18.0,
|
|
16
|
+
"min_solo_headroom": -18,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F32-cli-subscription-renewal",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 33,
|
|
22
|
+
"solo_score": 98,
|
|
23
|
+
"bare_headroom": 27,
|
|
24
|
+
"solo_headroom": -18,
|
|
25
|
+
"reason": "solo_claude score 98 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
package/benchmark/auto-resolve/results/20260512-f32-subscription-renewal-headroom/headroom-gate.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f32-subscription-renewal-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: 27.0
|
|
9
|
+
Minimum bare headroom: 27
|
|
10
|
+
Average solo headroom: -18.0
|
|
11
|
+
Minimum solo headroom: -18
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F32-cli-subscription-renewal | 33 | 27 | 98 | -18 | FAIL | solo_claude score 98 > 80 |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f4-web-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -10.0,
|
|
14
|
+
"min_bare_headroom": -10,
|
|
15
|
+
"avg_solo_headroom": -12.0,
|
|
16
|
+
"min_solo_headroom": -12,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F4-web-browser-design",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 70,
|
|
22
|
+
"solo_score": 92,
|
|
23
|
+
"bare_headroom": -10,
|
|
24
|
+
"solo_headroom": -12,
|
|
25
|
+
"reason": "bare score 70 > 60; solo_claude score 92 > 80; bare judge disqualifier; bare result disqualifier; bare verify disqualifier"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f4-web-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -10.0
|
|
9
|
+
Minimum bare headroom: -10
|
|
10
|
+
Average solo headroom: -12.0
|
|
11
|
+
Minimum solo headroom: -12
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F4-web-browser-design | 70 | -10 | 92 | -12 | FAIL | bare score 70 > 60; solo_claude score 92 > 80; bare judge disqualifier; bare result disqualifier; bare verify disqualifier |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f5-fixloop-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -39.0,
|
|
14
|
+
"min_bare_headroom": -39,
|
|
15
|
+
"avg_solo_headroom": -19.0,
|
|
16
|
+
"min_solo_headroom": -19,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F5-fix-loop-red-green",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 99,
|
|
22
|
+
"solo_score": 99,
|
|
23
|
+
"bare_headroom": -39,
|
|
24
|
+
"solo_headroom": -19,
|
|
25
|
+
"reason": "bare score 99 > 60; solo_claude score 99 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f5-fixloop-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -39.0
|
|
9
|
+
Minimum bare headroom: -39
|
|
10
|
+
Average solo headroom: -19.0
|
|
11
|
+
Minimum solo headroom: -19
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F5-fix-loop-red-green | 99 | -39 | 99 | -19 | FAIL | bare score 99 > 60; solo_claude score 99 > 80 |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f6-checksum-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -37.0,
|
|
14
|
+
"min_bare_headroom": -37,
|
|
15
|
+
"avg_solo_headroom": -16.0,
|
|
16
|
+
"min_solo_headroom": -16,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F6-dep-audit-native-module",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 97,
|
|
22
|
+
"solo_score": 96,
|
|
23
|
+
"bare_headroom": -37,
|
|
24
|
+
"solo_headroom": -16,
|
|
25
|
+
"reason": "bare score 97 > 60; solo_claude score 96 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f6-checksum-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -37.0
|
|
9
|
+
Minimum bare headroom: -37
|
|
10
|
+
Average solo headroom: -16.0
|
|
11
|
+
Minimum solo headroom: -16
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F6-dep-audit-native-module | 97 | -37 | 96 | -16 | FAIL | bare score 97 > 60; solo_claude score 96 > 80 |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f7-scope-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": -39.0,
|
|
14
|
+
"min_bare_headroom": -39,
|
|
15
|
+
"avg_solo_headroom": -20.0,
|
|
16
|
+
"min_solo_headroom": -20,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F7-out-of-scope-trap",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 99,
|
|
22
|
+
"solo_score": 100,
|
|
23
|
+
"bare_headroom": -39,
|
|
24
|
+
"solo_headroom": -20,
|
|
25
|
+
"reason": "bare score 99 > 60; solo_claude score 100 > 80"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f7-scope-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: -39.0
|
|
9
|
+
Minimum bare headroom: -39
|
|
10
|
+
Average solo headroom: -20.0
|
|
11
|
+
Minimum solo headroom: -20
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F7-out-of-scope-trap | 99 | -39 | 100 | -20 | FAIL | bare score 99 > 60; solo_claude score 100 > 80 |
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"run_id": "20260512-f9-e2e-headroom",
|
|
3
|
+
"rule": "at least 1 candidate fixtures; each must satisfy bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, with both baseline arms evidence-complete",
|
|
4
|
+
"verdict": "FAIL",
|
|
5
|
+
"fixtures_total": 1,
|
|
6
|
+
"fixtures_passed": 0,
|
|
7
|
+
"min_fixtures": 1,
|
|
8
|
+
"bare_max": 60,
|
|
9
|
+
"solo_max": 80,
|
|
10
|
+
"min_bare_headroom_required": 5,
|
|
11
|
+
"min_solo_headroom_required": 5,
|
|
12
|
+
"fixture_count_ok": true,
|
|
13
|
+
"avg_bare_headroom": 0.0,
|
|
14
|
+
"min_bare_headroom": 0,
|
|
15
|
+
"avg_solo_headroom": -10.0,
|
|
16
|
+
"min_solo_headroom": -10,
|
|
17
|
+
"rows": [
|
|
18
|
+
{
|
|
19
|
+
"fixture": "F9-e2e-ideate-to-resolve",
|
|
20
|
+
"status": "FAIL",
|
|
21
|
+
"bare_score": 60,
|
|
22
|
+
"solo_score": 90,
|
|
23
|
+
"bare_headroom": 0,
|
|
24
|
+
"solo_headroom": -10,
|
|
25
|
+
"reason": "bare headroom 0 < 5; solo_claude score 90 > 80; bare judge disqualifier"
|
|
26
|
+
}
|
|
27
|
+
]
|
|
28
|
+
}
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Headroom Gate — 20260512-f9-e2e-headroom
|
|
2
|
+
|
|
3
|
+
Verdict: **FAIL**
|
|
4
|
+
|
|
5
|
+
Fixtures passed: 0/1 (minimum required: 1)
|
|
6
|
+
|
|
7
|
+
Rule: at least 1 fixtures; bare <= 60 with headroom >= 5, solo_claude <= 80 with headroom >= 5, both baseline arms evidence-complete.
|
|
8
|
+
Average bare headroom: 0.0
|
|
9
|
+
Minimum bare headroom: 0
|
|
10
|
+
Average solo headroom: -10.0
|
|
11
|
+
Minimum solo headroom: -10
|
|
12
|
+
|
|
13
|
+
| Fixture | Bare | Bare headroom | Solo | Solo headroom | Status | Reason |
|
|
14
|
+
|---|---:|---:|---:|---:|---|---|
|
|
15
|
+
| F9-e2e-ideate-to-resolve | 60 | 0 | 90 | -10 | FAIL | bare headroom 0 < 5; solo_claude score 90 > 80; bare judge disqualifier |
|
package/bin/devlyn.js
CHANGED
|
@@ -690,14 +690,16 @@ function installInstructionsForCLI(cliKey) {
|
|
|
690
690
|
if (!cli || !cli.baseInstructionsFile) return false;
|
|
691
691
|
|
|
692
692
|
const destFile = path.join(process.cwd(), cli.instructionsFile);
|
|
693
|
+
const baseInstructionsSrc = path.join(__dirname, '..', cli.baseInstructionsFile);
|
|
693
694
|
let content = null;
|
|
694
695
|
if (fs.existsSync(destFile)) {
|
|
695
696
|
const current = fs.readFileSync(destFile, 'utf8');
|
|
696
697
|
if (current.includes(DEVLYN_AGENTS_MARKER)) {
|
|
697
698
|
content = stripManagedBlock(current) + '\n';
|
|
699
|
+
} else {
|
|
700
|
+
log(` → Preserved existing ${cli.instructionsFile}. Compare with the bundled template at ${baseInstructionsSrc} and merge relevant instructions.`, 'dim');
|
|
698
701
|
}
|
|
699
702
|
} else {
|
|
700
|
-
const baseInstructionsSrc = path.join(__dirname, '..', cli.baseInstructionsFile);
|
|
701
703
|
if (fs.existsSync(baseInstructionsSrc)) {
|
|
702
704
|
content = fs.readFileSync(baseInstructionsSrc, 'utf8');
|
|
703
705
|
}
|
|
@@ -865,6 +867,11 @@ function installClaudeCore() {
|
|
|
865
867
|
settings.env.CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS = '1';
|
|
866
868
|
settingsChanged = true;
|
|
867
869
|
}
|
|
870
|
+
const bashMaxTimeoutMs = Number.parseInt(settings.env.BASH_MAX_TIMEOUT_MS, 10);
|
|
871
|
+
if (!Number.isFinite(bashMaxTimeoutMs) || bashMaxTimeoutMs < 3600000) {
|
|
872
|
+
settings.env.BASH_MAX_TIMEOUT_MS = '3600000';
|
|
873
|
+
settingsChanged = true;
|
|
874
|
+
}
|
|
868
875
|
if (!hasOwnSetting('hooks')) {
|
|
869
876
|
settings.hooks = {};
|
|
870
877
|
settingsChanged = true;
|
|
@@ -893,7 +900,7 @@ function installClaudeCore() {
|
|
|
893
900
|
}
|
|
894
901
|
if (settingsChanged) {
|
|
895
902
|
fs.writeFileSync(settingsPath, JSON.stringify(settings, null, 2) + '\n');
|
|
896
|
-
log(' → settings.json (agent teams + pipeline permissions + Stop hook)', 'dim');
|
|
903
|
+
log(' → settings.json (agent teams + one-hour Bash max + pipeline permissions + Stop hook)', 'dim');
|
|
897
904
|
}
|
|
898
905
|
|
|
899
906
|
// Configure global Claude Code settings (~/.claude/settings.json)
|
|
@@ -24,7 +24,7 @@ two targeted probes; the invoking phase sets `--effort`, pair-JUDGE uses
|
|
|
24
24
|
```bash
|
|
25
25
|
python3 "$DEVLYN_SHARED_DIR/run-bounded.py" 600 -- claude -p "<judge prompt>" \
|
|
26
26
|
--permission-mode dontAsk \
|
|
27
|
-
--
|
|
27
|
+
--tools "Read,Grep,Glob" --allowedTools "Read,Grep,Glob" \
|
|
28
28
|
--setting-sources project --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
|
|
29
29
|
--effort medium \
|
|
30
30
|
> .devlyn/claude-judge.stdout 2> .devlyn/claude-judge.stderr
|
|
@@ -85,3 +85,11 @@ You interpret instructions literally and explicitly. The official guide is expli
|
|
|
85
85
|
3. **Overengineering**: do NOT add files, abstractions, error handling, validation, or "future flexibility" beyond what the spec asks. A bug fix doesn't need surrounding cleanup. The right complexity is the minimum needed for the current task.
|
|
86
86
|
|
|
87
87
|
You do NOT need stronger imperatives ("CRITICAL!", "YOU MUST!") to follow rules. Normal phrasing is sufficient.
|
|
88
|
+
|
|
89
|
+
## Explicit role capability
|
|
90
|
+
|
|
91
|
+
`role-config.py` validates explicit CLI judge effort against this bounded native capability declaration. Source: Claude Code2.1.263 built-in Fable5.1 metadata (native SHA ef5d2909c8af49f31ab6d5487e90316777bc2fac170adfe8160716caa8aaf4f9, byte157907500; effort/max/xhigh gates159068038/159068443/159068839). This is option support, not model fitness. Unknown version/model effort support fails with actionable unsupported-role-option. Model-only requests pass the exact ID to the native CLI without this effort table and still require matching native result evidence; engine-only defaults remain available. Native Agent workers have no validated exact-model/effort transport here, so explicit worker fields are rejected instead of switching channels.
|
|
92
|
+
|
|
93
|
+
<!-- devlyn-effort 2.1.263 claude-fable-5-1 low,medium,high,xhigh,max -->
|
|
94
|
+
|
|
95
|
+
For an explicit CLI judge profile, use the resolved `--model`/`--effort` in the bounded read-only invocation and add `--output-format json`. Follow the canonical resolve `JUDGE_STEM` round-scoped argv/prompt/raw capture and `judge-role-evidence.py` derivation contract. Only its exact derived result goes to the findings collector; never feed the JSON wrapper directly. Omitted options keep that role’s existing defaults.
|
|
@@ -14,6 +14,11 @@ Your default is efficient, direct, task-oriented. The canonical body specifies t
|
|
|
14
14
|
|
|
15
15
|
Resolve the request in the fewest useful tool loops without sacrificing correctness. For retrieval tasks: start with one broad search using short discriminative keywords; make another retrieval call only when the top results don't answer the core question or a required fact / parameter / source is missing. For tool-heavy tasks, start with a brief preamble: a one-line acknowledgment of the request and the first step you'll take.
|
|
16
16
|
|
|
17
|
+
For VERIFY JUDGE retrieval, batch related reads. After sealed parity proves a
|
|
18
|
+
canonical file and its `.agents` mirror are byte-identical, inspect the
|
|
19
|
+
canonical file only. Open self-test bodies or MECHANICAL raw streams only for a
|
|
20
|
+
named clause that remains unresolved after the broad pass.
|
|
21
|
+
|
|
17
22
|
## Validation pattern
|
|
18
23
|
|
|
19
24
|
Validation is concrete commands and tools, not self-belief. When the canonical body lists verification commands, execute them and trust their output. Do not substitute your judgment for a deterministic check the harness has provided. When validation tools are available (test runners, lint, type-check, the harness's `spec-verify-check.py`), run them before declaring success. The minimum evidence sufficient to answer correctly, cited precisely — then stop.
|
|
@@ -44,7 +44,7 @@ python3 "$DEVLYN_SHARED_DIR/run-bounded.py" 600 -- grok -p "<judge prompt>" \
|
|
|
44
44
|
--disallowed-tools "Agent,use_tool,search_tool" \
|
|
45
45
|
--allow 'Bash(<the bare anchor>)' \
|
|
46
46
|
--reasoning-effort medium \
|
|
47
|
-
--output-format json \
|
|
47
|
+
--output-format streaming-messages-json \
|
|
48
48
|
> "$REPO/.devlyn/grok-judge.stdout" 2> "$REPO/.devlyn/grok-judge.stderr"
|
|
49
49
|
```
|
|
50
50
|
|
|
@@ -96,4 +96,6 @@ python3 "$DEVLYN_SHARED_DIR/collect-codex-findings.py" \
|
|
|
96
96
|
```
|
|
97
97
|
|
|
98
98
|
On exit 124, write `.devlyn/verify.pair.timeout.json` with
|
|
99
|
-
`{"engine": "grok", "budget_seconds": 600}` before merge.
|
|
99
|
+
`{"engine": "grok", "budget_seconds": 600}` before merge. A budget abort can
|
|
100
|
+
leave the stream truncated before its terminal `result` record; that capture
|
|
101
|
+
stays `TIMEOUT` at merge, never a pair PASS.
|