devlyn-cli 1.3.2 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +4 -4
- package/README.md +2 -2
- package/agents-config/evaluator.md +3 -3
- package/bin/devlyn.js +16 -7
- package/config/skills/devlyn:auto-resolve/SKILL.md +15 -18
- package/config/skills/devlyn:auto-resolve/references/codex-integration.md +3 -3
- package/config/skills/devlyn:browser-validate/SKILL.md +3 -3
- package/config/skills/devlyn:browser-validate/references/flow-testing.md +2 -2
- package/config/skills/devlyn:browser-validate/references/tier2-playwright.md +14 -14
- package/config/skills/devlyn:evaluate/SKILL.md +5 -5
- package/config/skills/devlyn:team-resolve/SKILL.md +1 -1
- package/package.json +1 -1
package/CLAUDE.md
CHANGED
|
@@ -56,7 +56,7 @@ For hands-free build-evaluate-polish cycles — works for bugs, features, refact
|
|
|
56
56
|
/devlyn:auto-resolve [task description]
|
|
57
57
|
```
|
|
58
58
|
|
|
59
|
-
This runs the full pipeline automatically: **Build → Browser Validate → Evaluate → Fix Loop → Simplify → Review → Security Review → Clean → Docs**. Each phase runs as a separate subagent with its own context. Communication between phases happens via files (`.
|
|
59
|
+
This runs the full pipeline automatically: **Build → Browser Validate → Evaluate → Fix Loop → Simplify → Review → Security Review → Clean → Docs**. Each phase runs as a separate subagent with its own context. Communication between phases happens via files (`.devlyn/done-criteria.md`, `.devlyn/EVAL-FINDINGS.md`, `.devlyn/BROWSER-RESULTS.md`).
|
|
60
60
|
|
|
61
61
|
For web projects, the Browser Validate phase starts the dev server and tests the implemented feature in a real browser — clicking buttons, filling forms, verifying results. If the feature doesn't work, findings feed back into the fix loop.
|
|
62
62
|
|
|
@@ -72,9 +72,9 @@ Optional flags:
|
|
|
72
72
|
|
|
73
73
|
When you want to run each step yourself with review between phases:
|
|
74
74
|
|
|
75
|
-
1. `/devlyn:team-resolve [issue]` → Investigate + implement (writes `.
|
|
76
|
-
2. `/devlyn:evaluate` → Grade against done-criteria (writes `.
|
|
77
|
-
3. If findings exist: `/devlyn:team-resolve "Fix issues in .
|
|
75
|
+
1. `/devlyn:team-resolve [issue]` → Investigate + implement (writes `.devlyn/done-criteria.md`)
|
|
76
|
+
2. `/devlyn:evaluate` → Grade against done-criteria (writes `.devlyn/EVAL-FINDINGS.md`)
|
|
77
|
+
3. If findings exist: `/devlyn:team-resolve "Fix issues in .devlyn/EVAL-FINDINGS.md"` → Fix loop
|
|
78
78
|
4. `/simplify` → Quick cleanup pass
|
|
79
79
|
5. `/devlyn:team-review` → Multi-perspective team review (for important PRs)
|
|
80
80
|
6. `/devlyn:clean` → Codebase hygiene
|
package/README.md
CHANGED
|
@@ -42,7 +42,7 @@ Pipeline orchestration that controls _how agents execute_ — permissions, state
|
|
|
42
42
|
- **`/devlyn:auto-resolve`** — 9-phase automated pipeline (build → browser validate → evaluate → fix loop → simplify → review → security → clean → docs)
|
|
43
43
|
- **`/devlyn:browser-validate`** — feature verification in a real browser with tiered fallback (Chrome MCP → Playwright → curl)
|
|
44
44
|
- **`bypassPermissions` mode** for autonomous subagent execution
|
|
45
|
-
- **File-based state machine** — agents communicate via `.
|
|
45
|
+
- **File-based state machine** — agents communicate via `.devlyn/done-criteria.md`, `EVAL-FINDINGS.md`, and `BROWSER-RESULTS.md`
|
|
46
46
|
- **Git checkpoints** at each phase for rollback safety
|
|
47
47
|
- **Cross-model evaluation** via `--with-codex` flag (OpenAI Codex as independent evaluator)
|
|
48
48
|
|
|
@@ -172,7 +172,7 @@ For step-by-step control between phases:
|
|
|
172
172
|
|---|---|---|
|
|
173
173
|
| 1. **Resolve** | `/devlyn:resolve` or `/devlyn:team-resolve` | Fix the issue — solo for focused bugs (1-2 modules), team for complex issues (3+ modules) |
|
|
174
174
|
| 2. **Evaluate** | `/devlyn:evaluate` | Independent quality evaluation — grades against done criteria written in step 1 |
|
|
175
|
-
| | | *If the evaluation finds issues: `/devlyn:team-resolve "Fix issues in .
|
|
175
|
+
| | | *If the evaluation finds issues: `/devlyn:team-resolve "Fix issues in .devlyn/EVAL-FINDINGS.md"`* |
|
|
176
176
|
| 3. **Simplify** | `/simplify` | Quick cleanup pass for reuse, quality, and efficiency *(built-in Claude Code command)* |
|
|
177
177
|
| 4. **Review** | `/devlyn:review` or `/devlyn:team-review` | Audit the changes — solo for small PRs (< 10 files), team for large PRs (10+ files) |
|
|
178
178
|
| 5. **Clean** | `/devlyn:clean` | Remove dead code, unused dependencies, and complexity hotspots |
|
|
@@ -4,7 +4,7 @@ You are a code quality evaluator. Your job is to audit work produced by another
|
|
|
4
4
|
|
|
5
5
|
## Before You Start
|
|
6
6
|
|
|
7
|
-
1. **Check for done criteria**: Read `.
|
|
7
|
+
1. **Check for done criteria**: Read `.devlyn/done-criteria.md` if it exists. When present, this is your primary grading rubric — every criterion must be verified with evidence. When absent, fall back to the checklists below.
|
|
8
8
|
|
|
9
9
|
## Calibration
|
|
10
10
|
|
|
@@ -36,7 +36,7 @@ You will be too lenient by default. You will identify real issues, then talk you
|
|
|
36
36
|
|
|
37
37
|
## Output
|
|
38
38
|
|
|
39
|
-
Write findings to `.
|
|
39
|
+
Write findings to `.devlyn/EVAL-FINDINGS.md` for downstream consumption:
|
|
40
40
|
|
|
41
41
|
```markdown
|
|
42
42
|
# Evaluation Findings
|
|
@@ -61,4 +61,4 @@ Write findings to `.claude/EVAL-FINDINGS.md` for downstream consumption:
|
|
|
61
61
|
- [positive observations]
|
|
62
62
|
```
|
|
63
63
|
|
|
64
|
-
Do NOT delete `.
|
|
64
|
+
Do NOT delete `.devlyn/done-criteria.md` or `.devlyn/EVAL-FINDINGS.md` — the orchestrator or user is responsible for cleanup.
|
package/bin/devlyn.js
CHANGED
|
@@ -528,6 +528,19 @@ async function init(skipPrompts = false) {
|
|
|
528
528
|
log(' → CLAUDE.md', 'dim');
|
|
529
529
|
}
|
|
530
530
|
|
|
531
|
+
// Add .devlyn/ (pipeline state directory) to .gitignore
|
|
532
|
+
const gitignorePath = path.join(process.cwd(), '.gitignore');
|
|
533
|
+
const gitignoreEntry = '.devlyn/';
|
|
534
|
+
let gitignoreContent = fs.existsSync(gitignorePath)
|
|
535
|
+
? fs.readFileSync(gitignorePath, 'utf8')
|
|
536
|
+
: '';
|
|
537
|
+
if (!gitignoreContent.split('\n').some((line) => line.trim() === gitignoreEntry || line.trim() === '.devlyn')) {
|
|
538
|
+
const prefix = gitignoreContent && !gitignoreContent.endsWith('\n') ? '\n' : '';
|
|
539
|
+
const header = gitignoreContent ? '\n# devlyn-cli pipeline state\n' : '# devlyn-cli pipeline state\n';
|
|
540
|
+
fs.writeFileSync(gitignorePath, gitignoreContent + prefix + header + gitignoreEntry + '\n');
|
|
541
|
+
log(' → .gitignore (added .devlyn/)', 'dim');
|
|
542
|
+
}
|
|
543
|
+
|
|
531
544
|
// Enable agent teams in project settings
|
|
532
545
|
const settingsPath = path.join(targetDir, 'settings.json');
|
|
533
546
|
let settings = {};
|
|
@@ -539,16 +552,12 @@ async function init(skipPrompts = false) {
|
|
|
539
552
|
}
|
|
540
553
|
}
|
|
541
554
|
if (!settings.env) settings.env = {};
|
|
542
|
-
// Auto-allow pipeline
|
|
555
|
+
// Auto-allow pipeline state directory and common git commands so auto-resolve doesn't prompt
|
|
543
556
|
if (!settings.permissions) settings.permissions = {};
|
|
544
557
|
if (!settings.permissions.allow) settings.permissions.allow = [];
|
|
545
558
|
const pipelinePermissions = [
|
|
546
|
-
'Write(.
|
|
547
|
-
'
|
|
548
|
-
'Write(.claude/BROWSER-RESULTS.md)',
|
|
549
|
-
'Edit(.claude/done-criteria.md)',
|
|
550
|
-
'Edit(.claude/EVAL-FINDINGS.md)',
|
|
551
|
-
'Edit(.claude/BROWSER-RESULTS.md)',
|
|
559
|
+
'Write(.devlyn/**)',
|
|
560
|
+
'Edit(.devlyn/**)',
|
|
552
561
|
'Bash(git add *)',
|
|
553
562
|
'Bash(git commit *)',
|
|
554
563
|
'Bash(git diff *)',
|
|
@@ -53,7 +53,7 @@ Investigate and implement the following task. Work through these phases in order
|
|
|
53
53
|
- **UI/UX**: review existing components, design system, and user flows.
|
|
54
54
|
Read relevant files in parallel. Build a clear picture of what exists and what needs to change.
|
|
55
55
|
|
|
56
|
-
**Phase B — Define done criteria**: Before writing any code, create `.
|
|
56
|
+
**Phase B — Define done criteria**: Before writing any code, create `.devlyn/done-criteria.md` with testable success criteria. Each criterion must be verifiable (a test can assert it or a human can observe it in under 30 seconds), specific (not vague like "handles errors correctly"), and scoped to this task. Include an "Out of Scope" section and a "Verification Method" section. This file is required — downstream evaluation depends on it.
|
|
57
57
|
|
|
58
58
|
**Phase C — Assemble a team**: Use TeamCreate to create a team. Select teammates based on task type:
|
|
59
59
|
- Bug fix: root-cause-analyst + test-engineer (+ security-auditor, performance-engineer as needed)
|
|
@@ -64,14 +64,14 @@ Each teammate investigates from their perspective and sends findings back.
|
|
|
64
64
|
|
|
65
65
|
**Phase D — Synthesize and implement**: After all teammates report, compile findings into a unified plan. Implement the solution — no workarounds, no hardcoded values, no silent error swallowing. For bugs: write a failing test first, then fix. For features: implement following existing patterns, then write tests. For refactors: ensure tests pass before and after.
|
|
66
66
|
|
|
67
|
-
**Phase E — Update done criteria**: Mark each criterion in `.
|
|
67
|
+
**Phase E — Update done criteria**: Mark each criterion in `.devlyn/done-criteria.md` as satisfied. Run the full test suite.
|
|
68
68
|
|
|
69
69
|
**Phase F — Cleanup**: Shut down all teammates and delete the team.
|
|
70
70
|
|
|
71
71
|
The task is: [paste the task description here]
|
|
72
72
|
|
|
73
73
|
**After the agent completes**:
|
|
74
|
-
1. Verify `.
|
|
74
|
+
1. Verify `.devlyn/done-criteria.md` exists — if missing, create a basic one from the agent's output summary
|
|
75
75
|
2. Run `git diff --stat` to confirm code was actually changed
|
|
76
76
|
3. If no changes were made, report failure and stop
|
|
77
77
|
4. **Checkpoint**: Run `git add -A && git commit -m "chore(pipeline): phase 1 — build complete"` to create a rollback point
|
|
@@ -86,16 +86,16 @@ Skip if `--skip-browser` was set.
|
|
|
86
86
|
|
|
87
87
|
Agent prompt — pass this to the Agent tool:
|
|
88
88
|
|
|
89
|
-
You are a browser validation agent. Read the skill instructions at `.claude/skills/devlyn:browser-validate/SKILL.md` and follow the full workflow to validate this web application. The dev server should be started, tested, and left running (pass `--keep-server` internally) — the pipeline will clean it up later. Write your findings to `.
|
|
89
|
+
You are a browser validation agent. Read the skill instructions at `.claude/skills/devlyn:browser-validate/SKILL.md` and follow the full workflow to validate this web application. The dev server should be started, tested, and left running (pass `--keep-server` internally) — the pipeline will clean it up later. Write your findings to `.devlyn/BROWSER-RESULTS.md`.
|
|
90
90
|
|
|
91
91
|
**After the agent completes**:
|
|
92
|
-
1. Read `.
|
|
92
|
+
1. Read `.devlyn/BROWSER-RESULTS.md`
|
|
93
93
|
2. Extract the verdict
|
|
94
94
|
3. Branch on verdict:
|
|
95
95
|
- `PASS` → continue to PHASE 2
|
|
96
96
|
- `PASS WITH ISSUES` → continue to PHASE 2 (evaluator reads browser results as extra context)
|
|
97
97
|
- `PARTIALLY VERIFIED` → continue to PHASE 2, but flag to the evaluator that browser coverage was incomplete — unverified features should be weighted more heavily
|
|
98
|
-
- `NEEDS WORK` → features don't work in the browser. Go to PHASE 2.5 fix loop. Fix agent reads `.
|
|
98
|
+
- `NEEDS WORK` → features don't work in the browser. Go to PHASE 2.5 fix loop. Fix agent reads `.devlyn/BROWSER-RESULTS.md` for which criterion failed, at what step, with what error. After fixing, re-run PHASE 1.5 to verify the fix before proceeding to Evaluate.
|
|
99
99
|
- `BLOCKED` → app doesn't render. Go to PHASE 2.5 fix loop. After fixing, re-run PHASE 1.5.
|
|
100
100
|
|
|
101
101
|
## PHASE 2: EVALUATE
|
|
@@ -106,7 +106,7 @@ Agent prompt — pass this to the Agent tool:
|
|
|
106
106
|
|
|
107
107
|
You are an independent evaluator. Your job is to grade work produced by another agent, not to praise it. You will be too lenient by default — fight this tendency. When in doubt, score DOWN, not up. A false negative (missing a bug) ships broken code. A false positive (flagging a non-issue) costs minutes of review. The cost is asymmetric.
|
|
108
108
|
|
|
109
|
-
**Step 1 — Read the done criteria**: Read `.
|
|
109
|
+
**Step 1 — Read the done criteria**: Read `.devlyn/done-criteria.md`. This is your primary grading rubric. Every criterion must be verified with evidence.
|
|
110
110
|
|
|
111
111
|
**Step 2 — Discover changes**: Run `git diff HEAD~1` and `git status` to see what changed. Read all changed/new files in parallel.
|
|
112
112
|
|
|
@@ -119,7 +119,7 @@ You are an independent evaluator. Your job is to grade work produced by another
|
|
|
119
119
|
|
|
120
120
|
**Step 4 — Grade against done criteria**: For each criterion in done-criteria.md, mark VERIFIED (with evidence) or FAILED (with file:line and what's wrong).
|
|
121
121
|
|
|
122
|
-
**Step 5 — Write findings**: Write `.
|
|
122
|
+
**Step 5 — Write findings**: Write `.devlyn/EVAL-FINDINGS.md` with this exact structure:
|
|
123
123
|
|
|
124
124
|
```
|
|
125
125
|
# Evaluation Findings
|
|
@@ -143,10 +143,10 @@ Calibration examples to guide your judgment:
|
|
|
143
143
|
- A `let` that could be `const` = LOW note only. Linters catch this.
|
|
144
144
|
- "The error handling is generally quite good" = WRONG. Count the instances. Name the files. "3 of 7 async ops have error states. 4 are missing: file:line, file:line..."
|
|
145
145
|
|
|
146
|
-
Do NOT delete `.
|
|
146
|
+
Do NOT delete `.devlyn/done-criteria.md` or `.devlyn/EVAL-FINDINGS.md` — the orchestrator needs them.
|
|
147
147
|
|
|
148
148
|
**After the agent completes**:
|
|
149
|
-
1. Read `.
|
|
149
|
+
1. Read `.devlyn/EVAL-FINDINGS.md`
|
|
150
150
|
2. Extract the verdict
|
|
151
151
|
3. **If `--with-codex` includes `evaluate` or `both`**: Read `references/codex-integration.md` and follow the "PHASE 2-CODEX: CROSS-MODEL EVALUATE" section. This runs Codex as a second evaluator and merges findings into `EVAL-FINDINGS.md`.
|
|
152
152
|
4. Branch on verdict (from the merged findings if Codex was used):
|
|
@@ -154,7 +154,7 @@ Do NOT delete `.claude/done-criteria.md` or `.claude/EVAL-FINDINGS.md` — the o
|
|
|
154
154
|
- `PASS WITH ISSUES` → skip to PHASE 3 (issues are shippable)
|
|
155
155
|
- `NEEDS WORK` → go to PHASE 2.5 (fix loop)
|
|
156
156
|
- `BLOCKED` → go to PHASE 2.5 (fix loop)
|
|
157
|
-
5. If `.
|
|
157
|
+
5. If `.devlyn/EVAL-FINDINGS.md` was not created, treat as PASS WITH ISSUES and log a warning
|
|
158
158
|
|
|
159
159
|
## PHASE 2.5: FIX LOOP (conditional)
|
|
160
160
|
|
|
@@ -164,11 +164,11 @@ Spawn a subagent using the Agent tool with `mode: "bypassPermissions"` to fix th
|
|
|
164
164
|
|
|
165
165
|
Agent prompt — pass this to the Agent tool:
|
|
166
166
|
|
|
167
|
-
Read `.
|
|
167
|
+
Read `.devlyn/EVAL-FINDINGS.md` — it contains specific issues found by an independent evaluator. Fix every CRITICAL and HIGH finding. Address MEDIUM findings if straightforward.
|
|
168
168
|
|
|
169
|
-
The original done criteria are in `.
|
|
169
|
+
The original done criteria are in `.devlyn/done-criteria.md` — your fixes must still satisfy those criteria. Do not delete or weaken criteria to make them pass.
|
|
170
170
|
|
|
171
|
-
For each finding: read the referenced file:line, understand the issue, implement the fix. No workarounds — fix the actual root cause. Run tests after fixing. Update `.
|
|
171
|
+
For each finding: read the referenced file:line, understand the issue, implement the fix. No workarounds — fix the actual root cause. Run tests after fixing. Update `.devlyn/done-criteria.md` to mark fixed items.
|
|
172
172
|
|
|
173
173
|
**After the agent completes**:
|
|
174
174
|
1. **Checkpoint**: Run `git add -A && git commit -m "chore(pipeline): fix round [N] complete"` to preserve the fix
|
|
@@ -271,10 +271,7 @@ Synchronize documentation with recent code changes. Use `git log --oneline -20`
|
|
|
271
271
|
After all phases complete:
|
|
272
272
|
|
|
273
273
|
1. Clean up temporary files:
|
|
274
|
-
- Delete `.
|
|
275
|
-
- Delete `.claude/EVAL-FINDINGS.md`
|
|
276
|
-
- Delete `.claude/BROWSER-RESULTS.md` (if exists)
|
|
277
|
-
- Delete `.claude/screenshots/` directory (if exists)
|
|
274
|
+
- Delete the `.devlyn/` directory entirely (contains done-criteria.md, EVAL-FINDINGS.md, BROWSER-RESULTS.md, screenshots/, playwright temp files)
|
|
278
275
|
- Kill any dev server process still running from browser validation
|
|
279
276
|
|
|
280
277
|
2. Run `git log --oneline -10` to show commits made during the pipeline
|
|
@@ -34,7 +34,7 @@ Run after the Claude evaluator (Phase 2) completes, only if `--with-codex` inclu
|
|
|
34
34
|
### Step 1 — Get Codex's evaluation
|
|
35
35
|
|
|
36
36
|
Call `mcp__codex-cli__codex` with:
|
|
37
|
-
- `prompt`: Include the full content of `.
|
|
37
|
+
- `prompt`: Include the full content of `.devlyn/done-criteria.md` and the output of `git diff HEAD~1`. Ask Codex to evaluate the changes against the done criteria and report issues by severity (CRITICAL, HIGH, MEDIUM, LOW) with file:line references.
|
|
38
38
|
- `workingDirectory`: the project root
|
|
39
39
|
- `sandbox`: `"read-only"` (Codex should only read, not modify files)
|
|
40
40
|
- `reasoningEffort`: `"high"`
|
|
@@ -44,7 +44,7 @@ Example prompt to pass:
|
|
|
44
44
|
You are an independent code evaluator. Grade the following code changes against the done criteria below. Be strict — when in doubt, flag it.
|
|
45
45
|
|
|
46
46
|
## Done Criteria
|
|
47
|
-
[paste contents of .
|
|
47
|
+
[paste contents of .devlyn/done-criteria.md]
|
|
48
48
|
|
|
49
49
|
## Code Changes
|
|
50
50
|
[paste output of git diff HEAD~1]
|
|
@@ -61,7 +61,7 @@ Spawn a subagent using the Agent tool with `mode: "bypassPermissions"` to merge
|
|
|
61
61
|
|
|
62
62
|
Agent prompt:
|
|
63
63
|
|
|
64
|
-
Read `.
|
|
64
|
+
Read `.devlyn/EVAL-FINDINGS.md` (Claude's evaluation) and the Codex evaluation output below. Merge them into a single unified `.devlyn/EVAL-FINDINGS.md` following the existing format. Rules:
|
|
65
65
|
- Take the MORE SEVERE verdict between the two evaluators
|
|
66
66
|
- Deduplicate findings that reference the same file:line or describe the same issue
|
|
67
67
|
- When both evaluators flag the same issue, keep the more detailed description
|
|
@@ -15,7 +15,7 @@ $ARGUMENTS
|
|
|
15
15
|
|
|
16
16
|
## PHASE 1: DETECT
|
|
17
17
|
|
|
18
|
-
1. **What was built**: This is the most important input. Read `.
|
|
18
|
+
1. **What was built**: This is the most important input. Read `.devlyn/done-criteria.md` if it exists — it tells you what the feature is supposed to do. If it doesn't exist, read `git diff --stat` and `git log -1` to understand what changed. You need to know what to test before anything else.
|
|
19
19
|
|
|
20
20
|
2. **Framework detection**: Read `package.json` → identify framework and start command from `scripts.dev`, `scripts.start`, or `scripts.preview`.
|
|
21
21
|
|
|
@@ -65,7 +65,7 @@ If the app isn't rendering, the verdict is BLOCKED — feature testing can't hap
|
|
|
65
65
|
|
|
66
66
|
This is the primary purpose of browser validation. Everything else is in service of getting here.
|
|
67
67
|
|
|
68
|
-
Read `.
|
|
68
|
+
Read `.devlyn/done-criteria.md` (or infer from git diff what was built). For each criterion that describes something a user can do or see in the UI, test it end-to-end in the browser:
|
|
69
69
|
|
|
70
70
|
1. **Plan the test**: What would a user do to verify this feature works? Navigate where, click what, type what, expect what result?
|
|
71
71
|
2. **Execute it**: Navigate to the page, find the interactive elements, perform the actions, verify the outcome. Read `references/flow-testing.md` for patterns on converting criteria to browser steps.
|
|
@@ -88,7 +88,7 @@ Judgment-based — look at the screenshots and report visible issues.
|
|
|
88
88
|
|
|
89
89
|
## PHASE 6: REPORT
|
|
90
90
|
|
|
91
|
-
Write `.
|
|
91
|
+
Write `.devlyn/BROWSER-RESULTS.md`:
|
|
92
92
|
|
|
93
93
|
```markdown
|
|
94
94
|
# Browser Validation Results
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Flow Testing: Done-Criteria to Browser Steps
|
|
2
2
|
|
|
3
|
-
How to read `.
|
|
3
|
+
How to read `.devlyn/done-criteria.md` and convert testable criteria into browser action sequences. This is the bridge between "what should work" and "prove it works in the browser."
|
|
4
4
|
|
|
5
5
|
Read this file only during PHASE 4 (FLOW) when done-criteria exists.
|
|
6
6
|
|
|
@@ -8,7 +8,7 @@ Read this file only during PHASE 4 (FLOW) when done-criteria exists.
|
|
|
8
8
|
|
|
9
9
|
## Step 1: Classify Each Criterion
|
|
10
10
|
|
|
11
|
-
Read `.
|
|
11
|
+
Read `.devlyn/done-criteria.md` and classify each criterion:
|
|
12
12
|
|
|
13
13
|
**Browser-testable** — the criterion describes something a user can see or do in the UI:
|
|
14
14
|
- "User can create a new project from the dashboard"
|
|
@@ -44,7 +44,7 @@ Generate a temporary test script from the test steps, run it with Playwright's J
|
|
|
44
44
|
|
|
45
45
|
## Script Generation
|
|
46
46
|
|
|
47
|
-
For each phase (smoke, flow, visual), generate a test script at `.
|
|
47
|
+
For each phase (smoke, flow, visual), generate a test script at `.devlyn/browser-test.spec.ts`.
|
|
48
48
|
|
|
49
49
|
### Smoke Test Script Template
|
|
50
50
|
|
|
@@ -89,7 +89,7 @@ test.describe('Smoke Tests', () => {
|
|
|
89
89
|
const pageUrl = page.url();
|
|
90
90
|
expect(title, 'Page shows a browser error — server may be down').not.toBe(pageUrl);
|
|
91
91
|
|
|
92
|
-
await page.screenshot({ path: `.
|
|
92
|
+
await page.screenshot({ path: `.devlyn/screenshots/smoke${route.replace(/\//g, '-') || '-root'}.png`, fullPage: true });
|
|
93
93
|
|
|
94
94
|
if (errors.length > 0) {
|
|
95
95
|
test.info().annotations.push({ type: 'console_errors', description: errors.join(' | ') });
|
|
@@ -123,7 +123,7 @@ test('flow: [criterion description]', async ({ page }) => {
|
|
|
123
123
|
await expect(page.locator('[verification selector]')).toBeVisible();
|
|
124
124
|
|
|
125
125
|
// Screenshot
|
|
126
|
-
await page.screenshot({ path: '.
|
|
126
|
+
await page.screenshot({ path: '.devlyn/screenshots/flow-[name].png' });
|
|
127
127
|
});
|
|
128
128
|
```
|
|
129
129
|
|
|
@@ -135,7 +135,7 @@ test.describe('Visual - Mobile', () => {
|
|
|
135
135
|
for (const route of ROUTES) {
|
|
136
136
|
test(`visual-mobile: ${route}`, async ({ page }) => {
|
|
137
137
|
await page.goto(`http://localhost:${PORT}${route}`, { waitUntil: 'networkidle' });
|
|
138
|
-
await page.screenshot({ path: `.
|
|
138
|
+
await page.screenshot({ path: `.devlyn/screenshots/visual-mobile${route.replace(/\//g, '-') || '-root'}.png`, fullPage: true });
|
|
139
139
|
});
|
|
140
140
|
}
|
|
141
141
|
});
|
|
@@ -145,7 +145,7 @@ test.describe('Visual - Desktop', () => {
|
|
|
145
145
|
for (const route of ROUTES) {
|
|
146
146
|
test(`visual-desktop: ${route}`, async ({ page }) => {
|
|
147
147
|
await page.goto(`http://localhost:${PORT}${route}`, { waitUntil: 'networkidle' });
|
|
148
|
-
await page.screenshot({ path: `.
|
|
148
|
+
await page.screenshot({ path: `.devlyn/screenshots/visual-desktop${route.replace(/\//g, '-') || '-root'}.png`, fullPage: true });
|
|
149
149
|
});
|
|
150
150
|
}
|
|
151
151
|
});
|
|
@@ -154,16 +154,16 @@ test.describe('Visual - Desktop', () => {
|
|
|
154
154
|
## Execution
|
|
155
155
|
|
|
156
156
|
```bash
|
|
157
|
-
mkdir -p .
|
|
158
|
-
npx playwright test .
|
|
157
|
+
mkdir -p .devlyn/screenshots
|
|
158
|
+
npx playwright test .devlyn/browser-test.spec.ts \
|
|
159
159
|
--reporter=json \
|
|
160
|
-
--output=.
|
|
161
|
-
2>&1 | tee .
|
|
160
|
+
--output=.devlyn/playwright-results \
|
|
161
|
+
2>&1 | tee .devlyn/playwright-output.json
|
|
162
162
|
```
|
|
163
163
|
|
|
164
164
|
## Parsing Results
|
|
165
165
|
|
|
166
|
-
Read `.
|
|
166
|
+
Read `.devlyn/playwright-output.json`. The JSON structure contains:
|
|
167
167
|
- `suites[].specs[].tests[].results[].status` — `"passed"`, `"failed"`, `"timedOut"`
|
|
168
168
|
- `suites[].specs[].tests[].results[].errors` — error messages with stack traces
|
|
169
169
|
- `suites[].specs[].tests[].annotations` — custom annotations (console_errors, network_failures)
|
|
@@ -177,12 +177,12 @@ Map these to BROWSER-RESULTS.md findings:
|
|
|
177
177
|
|
|
178
178
|
After parsing results:
|
|
179
179
|
```bash
|
|
180
|
-
rm -f .
|
|
181
|
-
rm -rf .
|
|
182
|
-
rm -f .
|
|
180
|
+
rm -f .devlyn/browser-test.spec.ts
|
|
181
|
+
rm -rf .devlyn/playwright-results
|
|
182
|
+
rm -f .devlyn/playwright-output.json
|
|
183
183
|
```
|
|
184
184
|
|
|
185
|
-
Keep `.
|
|
185
|
+
Keep `.devlyn/screenshots/` — those are evidence referenced by the report.
|
|
186
186
|
|
|
187
187
|
## Limitations vs Tier 1
|
|
188
188
|
|
|
@@ -23,7 +23,7 @@ Before spawning any evaluators, understand what you're evaluating:
|
|
|
23
23
|
- **"recent changes"** or no argument: Use `git diff HEAD` for unstaged changes, `git status` for new files
|
|
24
24
|
- **Running session / live monitoring**: Take a baseline snapshot with `git status --short | wc -l`, then poll every 30-45 seconds for new changes using `git status` and `find . -newer <reference-file> -type f`. Report findings incrementally as changes appear.
|
|
25
25
|
|
|
26
|
-
2. **Check for done criteria**: Read `.
|
|
26
|
+
2. **Check for done criteria**: Read `.devlyn/done-criteria.md` if it exists. This file contains testable success criteria written by the generator (e.g., `/devlyn:team-resolve` Phase 1.5). When present, it is the primary grading rubric — every criterion in it must be verified. When absent, fall back to the evaluation checklists below.
|
|
27
27
|
|
|
28
28
|
3. Build the evaluation baseline:
|
|
29
29
|
- Run `git status --short` to see all changed and new files
|
|
@@ -297,9 +297,9 @@ LOW (note):
|
|
|
297
297
|
4. For each catch block: is the error surfaced to the user or silently swallowed?
|
|
298
298
|
5. Check for React anti-patterns: uncontrolled-to-controlled switches, direct DOM mutation, missing cleanup
|
|
299
299
|
6. Compare against existing components for pattern consistency
|
|
300
|
-
7. **Browser evidence** (when available): Read `.
|
|
300
|
+
7. **Browser evidence** (when available): Read `.devlyn/BROWSER-RESULTS.md` if it exists — it contains pre-collected smoke test results, flow test results, console errors, network failures, and screenshots from the `devlyn:browser-validate` skill. Use this as additional evidence in your evaluation. Do not re-run smoke tests that are already covered.
|
|
301
301
|
If the dev server is still running and you need deeper investigation on a specific interaction, use browser tools directly (check if `mcp__claude-in-chrome__*` tools are available, or fall back to Playwright). Focus on verifying specific findings, not duplicating the full smoke/flow suite.
|
|
302
|
-
If neither `.
|
|
302
|
+
If neither `.devlyn/BROWSER-RESULTS.md` exists nor browser tools are available, note "Live testing skipped — no browser validation available" in your deliverable.
|
|
303
303
|
|
|
304
304
|
**Your deliverable**: Send a message to the team lead with:
|
|
305
305
|
1. Component quality assessment for each new/changed component
|
|
@@ -480,7 +480,7 @@ After receiving all evaluator findings:
|
|
|
480
480
|
|
|
481
481
|
1. Present the evaluation report to the user (format below).
|
|
482
482
|
|
|
483
|
-
2. **Write findings to `.
|
|
483
|
+
2. **Write findings to `.devlyn/EVAL-FINDINGS.md`** for downstream consumption by other agents (e.g., `/devlyn:auto-resolve` orchestrator or a follow-up `/devlyn:team-resolve`). This file enables the feedback loop — the generator can read it and fix the issues without human relay.
|
|
484
484
|
|
|
485
485
|
```markdown
|
|
486
486
|
# Evaluation Findings
|
|
@@ -502,7 +502,7 @@ After receiving all evaluator findings:
|
|
|
502
502
|
- [pattern description]
|
|
503
503
|
```
|
|
504
504
|
|
|
505
|
-
3. Do NOT delete `.
|
|
505
|
+
3. Do NOT delete `.devlyn/done-criteria.md` or `.devlyn/EVAL-FINDINGS.md` — downstream consumers (e.g., `/devlyn:auto-resolve` orchestrator or a follow-up `/devlyn:team-resolve`) may need to read them. The orchestrator or user is responsible for cleanup.
|
|
506
506
|
|
|
507
507
|
## Phase 6: CLEANUP
|
|
508
508
|
|
|
@@ -93,7 +93,7 @@ Teammates: [list of roles being spawned and why each was chosen]
|
|
|
93
93
|
|
|
94
94
|
Before any code is written, define what "done" looks like. This prevents self-evaluation bias and gives external evaluators (like `/devlyn:evaluate`) concrete criteria to grade against.
|
|
95
95
|
|
|
96
|
-
1. Based on your Phase 1 investigation, write testable success criteria to `.
|
|
96
|
+
1. Based on your Phase 1 investigation, write testable success criteria to `.devlyn/done-criteria.md`:
|
|
97
97
|
|
|
98
98
|
```markdown
|
|
99
99
|
# Done Criteria: [issue summary]
|