qaa-agent 1.9.0 → 1.9.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +180 -177
- package/CLAUDE.md +557 -557
- package/README.md +1 -1
- package/VERSION +1 -0
- package/agents/qa-pipeline-orchestrator.md +1424 -1424
- package/agents/qaa-bug-detective.md +654 -630
- package/agents/qaa-e2e-runner.md +576 -552
- package/agents/qaa-executor.md +829 -805
- package/agents/qaa-project-researcher.md +400 -339
- package/commands/qa-test-report.md +219 -0
- package/package.json +3 -2
- package/workflows/qa-start.md +1405 -1262
package/workflows/qa-start.md
CHANGED
|
@@ -1,1262 +1,1405 @@
|
|
|
1
|
-
<purpose>
|
|
2
|
-
|
|
3
|
-
Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
|
|
4
|
-
|
|
5
|
-
Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
|
|
6
|
-
|
|
7
|
-
</purpose>
|
|
8
|
-
|
|
9
|
-
<required_reading>
|
|
10
|
-
|
|
11
|
-
Read these files BEFORE executing any pipeline stage. Do NOT skip.
|
|
12
|
-
|
|
13
|
-
- **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
|
|
14
|
-
- **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
|
|
15
|
-
|
|
16
|
-
</required_reading>
|
|
17
|
-
|
|
18
|
-
<process>
|
|
19
|
-
|
|
20
|
-
<step name="initialize" priority="first">
|
|
21
|
-
|
|
22
|
-
## Step 1: Initialize Pipeline
|
|
23
|
-
|
|
24
|
-
Parse `$ARGUMENTS` for flags:
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
DEV_REPO=""
|
|
28
|
-
QA_REPO=""
|
|
29
|
-
IS_AUTO=false
|
|
30
|
-
|
|
31
|
-
# Parse --dev-repo flag
|
|
32
|
-
if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
|
|
33
|
-
DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
|
|
34
|
-
fi
|
|
35
|
-
|
|
36
|
-
# Parse --qa-repo flag
|
|
37
|
-
if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
|
|
38
|
-
QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
|
|
39
|
-
fi
|
|
40
|
-
|
|
41
|
-
# Parse --auto flag
|
|
42
|
-
if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
|
|
43
|
-
IS_AUTO=true
|
|
44
|
-
fi
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
**If no --dev-repo provided**, use the current working directory:
|
|
48
|
-
```bash
|
|
49
|
-
if [ -z "$DEV_REPO" ]; then
|
|
50
|
-
DEV_REPO=$(pwd)
|
|
51
|
-
fi
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
**Attempt to call qaa-tools init** (handle missing tool gracefully):
|
|
55
|
-
```bash
|
|
56
|
-
INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
|
|
60
|
-
- Set `output_dir` to `.qa-output`
|
|
61
|
-
- Set `date` to current date in `YYYY-MM-DD` format
|
|
62
|
-
- Create output directory: `mkdir -p "$output_dir"`
|
|
63
|
-
|
|
64
|
-
If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
|
|
65
|
-
|
|
66
|
-
**Detect workflow option based on inputs:**
|
|
67
|
-
|
|
68
|
-
- If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
|
|
69
|
-
- If `QA_REPO` is provided: Assess QA repo maturity
|
|
70
|
-
- Check for existing test files, configs, coverage reports in QA repo
|
|
71
|
-
- Count test files, evaluate framework setup, check for CI config
|
|
72
|
-
- Score 0-100 based on test count, framework config presence, CI setup, coverage data
|
|
73
|
-
- Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
|
|
74
|
-
- Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
|
|
75
|
-
|
|
76
|
-
**Determine auto-advance mode:**
|
|
77
|
-
```bash
|
|
78
|
-
# Check persistent config flag
|
|
79
|
-
AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
|
|
80
|
-
AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
|
|
81
|
-
|
|
82
|
-
if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
|
|
83
|
-
IS_AUTO=true
|
|
84
|
-
node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
|
|
85
|
-
fi
|
|
86
|
-
|
|
87
|
-
# Safety: clear stale chain flag if NOT in auto mode
|
|
88
|
-
if [ "$IS_AUTO" = "false" ]; then
|
|
89
|
-
node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
|
|
90
|
-
fi
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
**Print initialization banner:**
|
|
94
|
-
```
|
|
95
|
-
=== QA Pipeline Orchestrator ===
|
|
96
|
-
Option: {option} ({description})
|
|
97
|
-
Dev Repo: {DEV_REPO}
|
|
98
|
-
QA Repo: {QA_REPO or 'N/A'}
|
|
99
|
-
Maturity Score: {maturity_score or 'N/A'}
|
|
100
|
-
Auto-Advance: {IS_AUTO}
|
|
101
|
-
Date: {date}
|
|
102
|
-
================================
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
Where `{description}` is:
|
|
106
|
-
- Option 1: "Dev-Only -- Full Pipeline"
|
|
107
|
-
- Option 2: "Dev + Immature QA -- Gap-Fill"
|
|
108
|
-
- Option 3: "Dev + Mature QA -- Surgical"
|
|
109
|
-
|
|
110
|
-
</step>
|
|
111
|
-
|
|
112
|
-
<step name="detect_framework">
|
|
113
|
-
|
|
114
|
-
## Step 2: Detect Framework
|
|
115
|
-
|
|
116
|
-
Before scanning, detect the project's language and test framework to guide all downstream agents.
|
|
117
|
-
|
|
118
|
-
**Read project config files:**
|
|
119
|
-
```bash
|
|
120
|
-
# Check for Node.js / JavaScript / TypeScript
|
|
121
|
-
[ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
|
|
122
|
-
|
|
123
|
-
# Check for Python
|
|
124
|
-
[ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
|
|
125
|
-
[ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
|
|
126
|
-
[ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
|
|
127
|
-
|
|
128
|
-
# Check for .NET
|
|
129
|
-
ls "${DEV_REPO}"/*.csproj 2>/dev/null
|
|
130
|
-
ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
|
|
131
|
-
|
|
132
|
-
# Check for Java
|
|
133
|
-
[ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
|
|
134
|
-
[ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
**Detect test framework from config files:**
|
|
138
|
-
```bash
|
|
139
|
-
# JavaScript/TypeScript ecosystem
|
|
140
|
-
[ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
|
|
141
|
-
[ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
|
|
142
|
-
[ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
|
|
143
|
-
[ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
|
|
144
|
-
|
|
145
|
-
# Python ecosystem
|
|
146
|
-
[ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
|
|
147
|
-
|
|
148
|
-
# Check package.json devDependencies for test frameworks
|
|
149
|
-
node -e "
|
|
150
|
-
try {
|
|
151
|
-
const pkg = require('${DEV_REPO}/package.json');
|
|
152
|
-
const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
|
|
153
|
-
const frameworks = [];
|
|
154
|
-
if (deps.cypress) frameworks.push('cypress');
|
|
155
|
-
if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
|
|
156
|
-
if (deps.jest) frameworks.push('jest');
|
|
157
|
-
if (deps.vitest) frameworks.push('vitest');
|
|
158
|
-
if (deps.mocha) frameworks.push('mocha');
|
|
159
|
-
console.log(frameworks.join(',') || 'none');
|
|
160
|
-
} catch { console.log('no-package-json'); }
|
|
161
|
-
" 2>/dev/null
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
**Assess detection confidence:**
|
|
165
|
-
- **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
|
|
166
|
-
- **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
|
|
167
|
-
- **LOW**: No test framework detected, or conflicting signals
|
|
168
|
-
|
|
169
|
-
**If no test framework found:**
|
|
170
|
-
- If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
|
|
171
|
-
- If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
|
|
172
|
-
- React/Next.js/Vue/Angular frontend -> Playwright
|
|
173
|
-
- Node.js API -> Jest or Vitest (prefer Vitest if ESM)
|
|
174
|
-
- Python -> Pytest
|
|
175
|
-
- Log: "Auto-selected: {framework} (no existing test framework detected)"
|
|
176
|
-
|
|
177
|
-
**If detection confidence is LOW:**
|
|
178
|
-
- If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
|
|
179
|
-
- If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
|
|
180
|
-
|
|
181
|
-
Store detected framework, language, and confidence for all downstream agents.
|
|
182
|
-
|
|
183
|
-
</step>
|
|
184
|
-
|
|
185
|
-
<step name="scan">
|
|
186
|
-
|
|
187
|
-
## Step 3: Scan Repository
|
|
188
|
-
|
|
189
|
-
**State update -- mark scan as running:**
|
|
190
|
-
```bash
|
|
191
|
-
node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
**Print stage banner:**
|
|
195
|
-
```
|
|
196
|
-
+------------------------------------------+
|
|
197
|
-
| STAGE 1: Scanner |
|
|
198
|
-
| Status: Running... |
|
|
199
|
-
+------------------------------------------+
|
|
200
|
-
```
|
|
201
|
-
|
|
202
|
-
**Spawn scanner agent:**
|
|
203
|
-
|
|
204
|
-
For **Option 1** (scan dev repo only):
|
|
205
|
-
```
|
|
206
|
-
Agent(subagent_type="general-purpose",
|
|
207
|
-
prompt="
|
|
208
|
-
<objective>Scan repository and produce SCAN_MANIFEST.md</objective>
|
|
209
|
-
<execution_context>@agents/qaa-scanner.md</execution_context>
|
|
210
|
-
<files_to_read>
|
|
211
|
-
- CLAUDE.md
|
|
212
|
-
</files_to_read>
|
|
213
|
-
<parameters>
|
|
214
|
-
dev_repo_path: {DEV_REPO}
|
|
215
|
-
qa_repo_path: null
|
|
216
|
-
output_path: {output_dir}/SCAN_MANIFEST.md
|
|
217
|
-
</parameters>
|
|
218
|
-
"
|
|
219
|
-
)
|
|
220
|
-
```
|
|
221
|
-
|
|
222
|
-
For **Options 2 and 3** (scan both repos):
|
|
223
|
-
```
|
|
224
|
-
Agent(subagent_type="general-purpose",
|
|
225
|
-
prompt="
|
|
226
|
-
<objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
|
|
227
|
-
<execution_context>@agents/qaa-scanner.md</execution_context>
|
|
228
|
-
<files_to_read>
|
|
229
|
-
- CLAUDE.md
|
|
230
|
-
</files_to_read>
|
|
231
|
-
<parameters>
|
|
232
|
-
dev_repo_path: {DEV_REPO}
|
|
233
|
-
qa_repo_path: {QA_REPO}
|
|
234
|
-
output_path: {output_dir}/SCAN_MANIFEST.md
|
|
235
|
-
</parameters>
|
|
236
|
-
"
|
|
237
|
-
)
|
|
238
|
-
```
|
|
239
|
-
|
|
240
|
-
**Parse scanner return:**
|
|
241
|
-
|
|
242
|
-
Expected return structure:
|
|
243
|
-
```
|
|
244
|
-
SCANNER_COMPLETE:
|
|
245
|
-
file_path: ".qa-output/SCAN_MANIFEST.md"
|
|
246
|
-
decision: PROCEED | STOP
|
|
247
|
-
has_frontend: true | false
|
|
248
|
-
detection_confidence: HIGH | MEDIUM | LOW
|
|
249
|
-
```
|
|
250
|
-
|
|
251
|
-
**Handle decision field:**
|
|
252
|
-
- If `decision` is `STOP`:
|
|
253
|
-
```bash
|
|
254
|
-
node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
|
|
255
|
-
```
|
|
256
|
-
Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
|
|
257
|
-
|
|
258
|
-
- If `decision` is `PROCEED`:
|
|
259
|
-
```bash
|
|
260
|
-
node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
|
|
261
|
-
```
|
|
262
|
-
Capture `has_frontend` for testid-injector conditional (Step 5).
|
|
263
|
-
Capture `detection_confidence` for checkpoint handling.
|
|
264
|
-
|
|
265
|
-
**Verify artifact exists before continuing:**
|
|
266
|
-
```bash
|
|
267
|
-
[ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
|
|
268
|
-
```
|
|
269
|
-
|
|
270
|
-
If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
|
|
271
|
-
|
|
272
|
-
</step>
|
|
273
|
-
|
|
274
|
-
<step name="
|
|
275
|
-
|
|
276
|
-
## Step 3b:
|
|
277
|
-
|
|
278
|
-
**State update -- mark
|
|
279
|
-
```bash
|
|
280
|
-
node bin/qaa-tools.cjs state patch --"
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
**Print stage banner:**
|
|
284
|
-
```
|
|
285
|
-
+------------------------------------------+
|
|
286
|
-
| STAGE 1b:
|
|
287
|
-
| Status: Running
|
|
288
|
-
+------------------------------------------+
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
**Create output directory:**
|
|
292
|
-
```bash
|
|
293
|
-
mkdir -p ${output_dir}/
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
**Spawn
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
<
|
|
303
|
-
-
|
|
304
|
-
|
|
305
|
-
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
**
|
|
356
|
-
```
|
|
357
|
-
Agent(subagent_type="general-purpose",
|
|
358
|
-
prompt="
|
|
359
|
-
<objective>
|
|
360
|
-
<execution_context>@agents/qaa-
|
|
361
|
-
<files_to_read>
|
|
362
|
-
-
|
|
363
|
-
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
```
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
```
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
|
573
|
-
| Status: Running... |
|
|
574
|
-
+------------------------------------------+
|
|
575
|
-
```
|
|
576
|
-
|
|
577
|
-
**
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
<
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
)
|
|
634
|
-
```
|
|
635
|
-
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
```
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
|
|
691
|
-
```
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
|
|
733
|
-
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
750
|
-
|
|
751
|
-
|
|
752
|
-
|
|
753
|
-
|
|
754
|
-
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
|
|
758
|
-
|
|
759
|
-
-
|
|
760
|
-
-
|
|
761
|
-
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
|
|
783
|
-
|
|
784
|
-
|
|
785
|
-
|
|
786
|
-
|
|
787
|
-
|
|
788
|
-
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
|
|
792
|
-
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
|
|
806
|
-
**
|
|
807
|
-
```
|
|
808
|
-
|
|
809
|
-
|
|
810
|
-
|
|
811
|
-
|
|
812
|
-
|
|
813
|
-
|
|
814
|
-
|
|
815
|
-
|
|
816
|
-
|
|
817
|
-
|
|
818
|
-
|
|
819
|
-
|
|
820
|
-
|
|
821
|
-
|
|
822
|
-
|
|
823
|
-
|
|
824
|
-
|
|
825
|
-
```
|
|
826
|
-
|
|
827
|
-
|
|
828
|
-
|
|
829
|
-
|
|
830
|
-
|
|
831
|
-
|
|
832
|
-
|
|
833
|
-
|
|
834
|
-
|
|
835
|
-
|
|
836
|
-
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
|
|
840
|
-
|
|
841
|
-
|
|
842
|
-
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
|
|
846
|
-
|
|
847
|
-
|
|
848
|
-
|
|
849
|
-
|
|
850
|
-
|
|
851
|
-
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
|
|
860
|
-
|
|
861
|
-
|
|
862
|
-
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
|
|
871
|
-
|
|
872
|
-
|
|
873
|
-
|
|
874
|
-
|
|
875
|
-
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
|
|
892
|
-
|
|
893
|
-
|
|
894
|
-
|
|
895
|
-
|
|
896
|
-
|
|
897
|
-
|
|
898
|
-
|
|
899
|
-
|
|
900
|
-
|
|
901
|
-
|
|
902
|
-
|
|
903
|
-
|
|
904
|
-
|
|
905
|
-
|
|
906
|
-
|
|
907
|
-
|
|
908
|
-
|
|
909
|
-
|
|
910
|
-
|
|
911
|
-
|
|
912
|
-
```
|
|
913
|
-
|
|
914
|
-
|
|
915
|
-
|
|
916
|
-
|
|
917
|
-
|
|
918
|
-
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
922
|
-
|
|
923
|
-
|
|
924
|
-
|
|
925
|
-
|
|
926
|
-
|
|
927
|
-
|
|
928
|
-
|
|
929
|
-
|
|
930
|
-
|
|
931
|
-
|
|
932
|
-
|
|
933
|
-
|
|
934
|
-
|
|
935
|
-
|
|
936
|
-
|
|
937
|
-
|
|
938
|
-
|
|
939
|
-
|
|
940
|
-
|
|
941
|
-
**
|
|
942
|
-
```
|
|
943
|
-
|
|
944
|
-
|
|
945
|
-
|
|
946
|
-
|
|
947
|
-
|
|
948
|
-
|
|
949
|
-
|
|
950
|
-
|
|
951
|
-
|
|
952
|
-
|
|
953
|
-
|
|
954
|
-
|
|
955
|
-
|
|
956
|
-
|
|
957
|
-
|
|
958
|
-
|
|
959
|
-
|
|
960
|
-
|
|
961
|
-
|
|
962
|
-
|
|
963
|
-
|
|
964
|
-
|
|
965
|
-
|
|
966
|
-
|
|
967
|
-
|
|
968
|
-
|
|
969
|
-
|
|
970
|
-
|
|
971
|
-
|
|
972
|
-
|
|
973
|
-
|
|
974
|
-
|
|
975
|
-
|
|
976
|
-
|
|
977
|
-
|
|
978
|
-
|
|
979
|
-
|
|
980
|
-
|
|
981
|
-
|
|
982
|
-
|
|
983
|
-
|
|
984
|
-
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
|
|
991
|
-
|
|
992
|
-
|
|
993
|
-
|
|
994
|
-
|
|
995
|
-
|
|
996
|
-
|
|
997
|
-
```
|
|
998
|
-
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
|
|
1002
|
-
|
|
1003
|
-
|
|
1004
|
-
|
|
1005
|
-
|
|
1006
|
-
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
-
|
|
1010
|
-
|
|
1011
|
-
|
|
1012
|
-
|
|
1013
|
-
|
|
1014
|
-
|
|
1015
|
-
|
|
1016
|
-
|
|
1017
|
-
|
|
1018
|
-
```
|
|
1019
|
-
|
|
1020
|
-
|
|
1021
|
-
|
|
1022
|
-
|
|
1023
|
-
|
|
1024
|
-
-
|
|
1025
|
-
|
|
1026
|
-
|
|
1027
|
-
|
|
1028
|
-
-
|
|
1029
|
-
- `
|
|
1030
|
-
|
|
1031
|
-
|
|
1032
|
-
|
|
1033
|
-
|
|
1034
|
-
|
|
1035
|
-
|
|
1036
|
-
|
|
1037
|
-
|
|
1038
|
-
|
|
1039
|
-
|
|
1040
|
-
If
|
|
1041
|
-
|
|
1042
|
-
|
|
1043
|
-
|
|
1044
|
-
|
|
1045
|
-
|
|
1046
|
-
|
|
1047
|
-
|
|
1048
|
-
|
|
1049
|
-
|
|
1050
|
-
|
|
1051
|
-
|
|
1052
|
-
|
|
1053
|
-
|
|
1054
|
-
|
|
1055
|
-
|
|
1056
|
-
|
|
1057
|
-
|
|
1058
|
-
|
|
1059
|
-
|
|
1060
|
-
|
|
1061
|
-
```
|
|
1062
|
-
|
|
1063
|
-
|
|
1064
|
-
|
|
1065
|
-
|
|
1066
|
-
|
|
1067
|
-
|
|
1068
|
-
|
|
1069
|
-
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
|
|
1074
|
-
|
|
1075
|
-
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
```
|
|
1079
|
-
|
|
1080
|
-
###
|
|
1081
|
-
|
|
1082
|
-
|
|
1083
|
-
|
|
1084
|
-
|
|
1085
|
-
|
|
1086
|
-
|
|
1087
|
-
|
|
1088
|
-
|
|
1089
|
-
|
|
1090
|
-
|
|
1091
|
-
|
|
1092
|
-
|
|
1093
|
-
|
|
1094
|
-
|
|
1095
|
-
|
|
1096
|
-
|
|
1097
|
-
|
|
1098
|
-
|
|
1099
|
-
|
|
1100
|
-
|
|
1101
|
-
|
|
1102
|
-
|
|
1103
|
-
|
|
1104
|
-
|
|
1105
|
-
|
|
1106
|
-
|
|
1107
|
-
|
|
1108
|
-
|
|
1109
|
-
|
|
1110
|
-
|
|
1111
|
-
|
|
1112
|
-
|
|
1113
|
-
|
|
1114
|
-
|
|
1115
|
-
|
|
1116
|
-
|
|
1117
|
-
|
|
1118
|
-
|
|
1119
|
-
|
|
1120
|
-
|
|
1121
|
-
|
|
1122
|
-
|
|
1123
|
-
|
|
1124
|
-
|
|
1125
|
-
|
|
1126
|
-
|
|
1127
|
-
|
|
1128
|
-
|
|
1129
|
-
|
|
1130
|
-
|
|
1131
|
-
|
|
1132
|
-
|
|
1133
|
-
|
|
1134
|
-
|
|
1135
|
-
|
|
1136
|
-
|
|
1137
|
-
|
|
1138
|
-
|
|
1139
|
-
|
|
1140
|
-
|
|
1141
|
-
|
|
1142
|
-
|
|
1143
|
-
|
|
1144
|
-
|
|
1145
|
-
|
|
1146
|
-
|
|
1147
|
-
|
|
1148
|
-
|
|
1149
|
-
|
|
1150
|
-
|
|
1151
|
-
|
|
1152
|
-
|
|
1153
|
-
|
|
1154
|
-
|
|
1155
|
-
|
|
1156
|
-
|
|
1157
|
-
|
|
1158
|
-
|
|
1159
|
-
|
|
1160
|
-
|
|
1161
|
-
|
|
1162
|
-
|
|
1163
|
-
|
|
1164
|
-
|
|
1165
|
-
|
|
1166
|
-
|
|
1167
|
-
|
|
1168
|
-
|
|
1169
|
-
|
|
1170
|
-
-
|
|
1171
|
-
- `
|
|
1172
|
-
- `
|
|
1173
|
-
|
|
1174
|
-
|
|
1175
|
-
|
|
1176
|
-
|
|
1177
|
-
|
|
1178
|
-
|
|
1179
|
-
|
|
1180
|
-
|
|
1181
|
-
|
|
1182
|
-
|
|
1183
|
-
|
|
1184
|
-
|
|
1185
|
-
|
|
1186
|
-
|
|
1187
|
-
|
|
1188
|
-
|
|
1189
|
-
|
|
1190
|
-
|
|
1191
|
-
|
|
1192
|
-
|
|
1193
|
-
|
|
1194
|
-
|
|
1195
|
-
|
|
1196
|
-
|
|
1197
|
-
|
|
1198
|
-
|
|
1199
|
-
|
|
1200
|
-
|
|
1201
|
-
###
|
|
1202
|
-
|
|
1203
|
-
|
|
1204
|
-
|
|
1205
|
-
|
|
1206
|
-
|
|
1207
|
-
|
|
1208
|
-
|
|
1209
|
-
|
|
1210
|
-
|
|
1211
|
-
|
|
1212
|
-
|
|
1213
|
-
|
|
1214
|
-
|
|
1215
|
-
|
|
1216
|
-
|
|
1217
|
-
|
|
1218
|
-
|
|
1219
|
-
|
|
1220
|
-
|
|
1221
|
-
|
|
1222
|
-
|
|
1223
|
-
|
|
1224
|
-
|
|
1225
|
-
|
|
1226
|
-
|
|
1227
|
-
|
|
1228
|
-
|
|
1229
|
-
|
|
1230
|
-
|
|
1231
|
-
|
|
1232
|
-
|
|
1233
|
-
|
|
1234
|
-
|
|
1235
|
-
|
|
1236
|
-
|
|
1237
|
-
|
|
1238
|
-
|
|
1239
|
-
|
|
1240
|
-
|
|
1241
|
-
|
|
1242
|
-
|
|
1243
|
-
|
|
1244
|
-
|
|
1245
|
-
|
|
1246
|
-
|
|
1247
|
-
|
|
1248
|
-
|
|
1249
|
-
|
|
1250
|
-
|
|
1251
|
-
|
|
1252
|
-
|
|
1253
|
-
|
|
1254
|
-
|
|
1255
|
-
|
|
1256
|
-
|
|
1257
|
-
|
|
1258
|
-
|
|
1259
|
-
|
|
1260
|
-
|
|
1261
|
-
|
|
1262
|
-
|
|
1
|
+
<purpose>
|
|
2
|
+
|
|
3
|
+
Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
|
|
4
|
+
|
|
5
|
+
Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
|
|
6
|
+
|
|
7
|
+
</purpose>
|
|
8
|
+
|
|
9
|
+
<required_reading>
|
|
10
|
+
|
|
11
|
+
Read these files BEFORE executing any pipeline stage. Do NOT skip.
|
|
12
|
+
|
|
13
|
+
- **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
|
|
14
|
+
- **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
|
|
15
|
+
|
|
16
|
+
</required_reading>
|
|
17
|
+
|
|
18
|
+
<process>
|
|
19
|
+
|
|
20
|
+
<step name="initialize" priority="first">
|
|
21
|
+
|
|
22
|
+
## Step 1: Initialize Pipeline
|
|
23
|
+
|
|
24
|
+
Parse `$ARGUMENTS` for flags:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
DEV_REPO=""
|
|
28
|
+
QA_REPO=""
|
|
29
|
+
IS_AUTO=false
|
|
30
|
+
|
|
31
|
+
# Parse --dev-repo flag
|
|
32
|
+
if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
|
|
33
|
+
DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
|
|
34
|
+
fi
|
|
35
|
+
|
|
36
|
+
# Parse --qa-repo flag
|
|
37
|
+
if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
|
|
38
|
+
QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
|
|
39
|
+
fi
|
|
40
|
+
|
|
41
|
+
# Parse --auto flag
|
|
42
|
+
if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
|
|
43
|
+
IS_AUTO=true
|
|
44
|
+
fi
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**If no --dev-repo provided**, use the current working directory:
|
|
48
|
+
```bash
|
|
49
|
+
if [ -z "$DEV_REPO" ]; then
|
|
50
|
+
DEV_REPO=$(pwd)
|
|
51
|
+
fi
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
**Attempt to call qaa-tools init** (handle missing tool gracefully):
|
|
55
|
+
```bash
|
|
56
|
+
INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
|
|
60
|
+
- Set `output_dir` to `.qa-output`
|
|
61
|
+
- Set `date` to current date in `YYYY-MM-DD` format
|
|
62
|
+
- Create output directory: `mkdir -p "$output_dir"`
|
|
63
|
+
|
|
64
|
+
If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
|
|
65
|
+
|
|
66
|
+
**Detect workflow option based on inputs:**
|
|
67
|
+
|
|
68
|
+
- If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
|
|
69
|
+
- If `QA_REPO` is provided: Assess QA repo maturity
|
|
70
|
+
- Check for existing test files, configs, coverage reports in QA repo
|
|
71
|
+
- Count test files, evaluate framework setup, check for CI config
|
|
72
|
+
- Score 0-100 based on test count, framework config presence, CI setup, coverage data
|
|
73
|
+
- Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
|
|
74
|
+
- Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
|
|
75
|
+
|
|
76
|
+
**Determine auto-advance mode:**
|
|
77
|
+
```bash
|
|
78
|
+
# Check persistent config flag
|
|
79
|
+
AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
|
|
80
|
+
AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
|
|
81
|
+
|
|
82
|
+
if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
|
|
83
|
+
IS_AUTO=true
|
|
84
|
+
node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
|
|
85
|
+
fi
|
|
86
|
+
|
|
87
|
+
# Safety: clear stale chain flag if NOT in auto mode
|
|
88
|
+
if [ "$IS_AUTO" = "false" ]; then
|
|
89
|
+
node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
|
|
90
|
+
fi
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
**Print initialization banner:**
|
|
94
|
+
```
|
|
95
|
+
=== QA Pipeline Orchestrator ===
|
|
96
|
+
Option: {option} ({description})
|
|
97
|
+
Dev Repo: {DEV_REPO}
|
|
98
|
+
QA Repo: {QA_REPO or 'N/A'}
|
|
99
|
+
Maturity Score: {maturity_score or 'N/A'}
|
|
100
|
+
Auto-Advance: {IS_AUTO}
|
|
101
|
+
Date: {date}
|
|
102
|
+
================================
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Where `{description}` is:
|
|
106
|
+
- Option 1: "Dev-Only -- Full Pipeline"
|
|
107
|
+
- Option 2: "Dev + Immature QA -- Gap-Fill"
|
|
108
|
+
- Option 3: "Dev + Mature QA -- Surgical"
|
|
109
|
+
|
|
110
|
+
</step>
|
|
111
|
+
|
|
112
|
+
<step name="detect_framework">
|
|
113
|
+
|
|
114
|
+
## Step 2: Detect Framework
|
|
115
|
+
|
|
116
|
+
Before scanning, detect the project's language and test framework to guide all downstream agents.
|
|
117
|
+
|
|
118
|
+
**Read project config files:**
|
|
119
|
+
```bash
|
|
120
|
+
# Check for Node.js / JavaScript / TypeScript
|
|
121
|
+
[ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
|
|
122
|
+
|
|
123
|
+
# Check for Python
|
|
124
|
+
[ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
|
|
125
|
+
[ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
|
|
126
|
+
[ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
|
|
127
|
+
|
|
128
|
+
# Check for .NET
|
|
129
|
+
ls "${DEV_REPO}"/*.csproj 2>/dev/null
|
|
130
|
+
ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
|
|
131
|
+
|
|
132
|
+
# Check for Java
|
|
133
|
+
[ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
|
|
134
|
+
[ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
**Detect test framework from config files:**
|
|
138
|
+
```bash
|
|
139
|
+
# JavaScript/TypeScript ecosystem
|
|
140
|
+
[ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
|
|
141
|
+
[ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
|
|
142
|
+
[ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
|
|
143
|
+
[ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
|
|
144
|
+
|
|
145
|
+
# Python ecosystem
|
|
146
|
+
[ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
|
|
147
|
+
|
|
148
|
+
# Check package.json devDependencies for test frameworks
|
|
149
|
+
node -e "
|
|
150
|
+
try {
|
|
151
|
+
const pkg = require('${DEV_REPO}/package.json');
|
|
152
|
+
const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
|
|
153
|
+
const frameworks = [];
|
|
154
|
+
if (deps.cypress) frameworks.push('cypress');
|
|
155
|
+
if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
|
|
156
|
+
if (deps.jest) frameworks.push('jest');
|
|
157
|
+
if (deps.vitest) frameworks.push('vitest');
|
|
158
|
+
if (deps.mocha) frameworks.push('mocha');
|
|
159
|
+
console.log(frameworks.join(',') || 'none');
|
|
160
|
+
} catch { console.log('no-package-json'); }
|
|
161
|
+
" 2>/dev/null
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
**Assess detection confidence:**
|
|
165
|
+
- **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
|
|
166
|
+
- **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
|
|
167
|
+
- **LOW**: No test framework detected, or conflicting signals
|
|
168
|
+
|
|
169
|
+
**If no test framework found:**
|
|
170
|
+
- If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
|
|
171
|
+
- If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
|
|
172
|
+
- React/Next.js/Vue/Angular frontend -> Playwright
|
|
173
|
+
- Node.js API -> Jest or Vitest (prefer Vitest if ESM)
|
|
174
|
+
- Python -> Pytest
|
|
175
|
+
- Log: "Auto-selected: {framework} (no existing test framework detected)"
|
|
176
|
+
|
|
177
|
+
**If detection confidence is LOW:**
|
|
178
|
+
- If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
|
|
179
|
+
- If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
|
|
180
|
+
|
|
181
|
+
Store detected framework, language, and confidence for all downstream agents.
|
|
182
|
+
|
|
183
|
+
</step>
|
|
184
|
+
|
|
185
|
+
<step name="scan">
|
|
186
|
+
|
|
187
|
+
## Step 3: Scan Repository
|
|
188
|
+
|
|
189
|
+
**State update -- mark scan as running:**
|
|
190
|
+
```bash
|
|
191
|
+
node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
**Print stage banner:**
|
|
195
|
+
```
|
|
196
|
+
+------------------------------------------+
|
|
197
|
+
| STAGE 1: Scanner |
|
|
198
|
+
| Status: Running... |
|
|
199
|
+
+------------------------------------------+
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
**Spawn scanner agent:**
|
|
203
|
+
|
|
204
|
+
For **Option 1** (scan dev repo only):
|
|
205
|
+
```
|
|
206
|
+
Agent(subagent_type="general-purpose",
|
|
207
|
+
prompt="
|
|
208
|
+
<objective>Scan repository and produce SCAN_MANIFEST.md</objective>
|
|
209
|
+
<execution_context>@agents/qaa-scanner.md</execution_context>
|
|
210
|
+
<files_to_read>
|
|
211
|
+
- CLAUDE.md
|
|
212
|
+
</files_to_read>
|
|
213
|
+
<parameters>
|
|
214
|
+
dev_repo_path: {DEV_REPO}
|
|
215
|
+
qa_repo_path: null
|
|
216
|
+
output_path: {output_dir}/SCAN_MANIFEST.md
|
|
217
|
+
</parameters>
|
|
218
|
+
"
|
|
219
|
+
)
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
For **Options 2 and 3** (scan both repos):
|
|
223
|
+
```
|
|
224
|
+
Agent(subagent_type="general-purpose",
|
|
225
|
+
prompt="
|
|
226
|
+
<objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
|
|
227
|
+
<execution_context>@agents/qaa-scanner.md</execution_context>
|
|
228
|
+
<files_to_read>
|
|
229
|
+
- CLAUDE.md
|
|
230
|
+
</files_to_read>
|
|
231
|
+
<parameters>
|
|
232
|
+
dev_repo_path: {DEV_REPO}
|
|
233
|
+
qa_repo_path: {QA_REPO}
|
|
234
|
+
output_path: {output_dir}/SCAN_MANIFEST.md
|
|
235
|
+
</parameters>
|
|
236
|
+
"
|
|
237
|
+
)
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
**Parse scanner return:**
|
|
241
|
+
|
|
242
|
+
Expected return structure:
|
|
243
|
+
```
|
|
244
|
+
SCANNER_COMPLETE:
|
|
245
|
+
file_path: ".qa-output/SCAN_MANIFEST.md"
|
|
246
|
+
decision: PROCEED | STOP
|
|
247
|
+
has_frontend: true | false
|
|
248
|
+
detection_confidence: HIGH | MEDIUM | LOW
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
**Handle decision field:**
|
|
252
|
+
- If `decision` is `STOP`:
|
|
253
|
+
```bash
|
|
254
|
+
node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
|
|
255
|
+
```
|
|
256
|
+
Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
|
|
257
|
+
|
|
258
|
+
- If `decision` is `PROCEED`:
|
|
259
|
+
```bash
|
|
260
|
+
node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
|
|
261
|
+
```
|
|
262
|
+
Capture `has_frontend` for testid-injector conditional (Step 5).
|
|
263
|
+
Capture `detection_confidence` for checkpoint handling.
|
|
264
|
+
|
|
265
|
+
**Verify artifact exists before continuing:**
|
|
266
|
+
```bash
|
|
267
|
+
[ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
|
|
271
|
+
|
|
272
|
+
</step>
|
|
273
|
+
|
|
274
|
+
<step name="codebase_map">
|
|
275
|
+
|
|
276
|
+
## Step 3b: Codebase Map (Deep Analysis)
|
|
277
|
+
|
|
278
|
+
**State update -- mark codebase-map as running:**
|
|
279
|
+
```bash
|
|
280
|
+
node bin/qaa-tools.cjs state patch --"Map Status" running --"Status" "Mapping codebase" 2>/dev/null || true
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
**Print stage banner:**
|
|
284
|
+
```
|
|
285
|
+
+------------------------------------------+
|
|
286
|
+
| STAGE 1b: Codebase Mapper |
|
|
287
|
+
| Status: Running 4 parallel sub-agents...|
|
|
288
|
+
+------------------------------------------+
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
**Create output directory:**
|
|
292
|
+
```bash
|
|
293
|
+
mkdir -p ${output_dir}/codebase
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
**Spawn 4 parallel sub-agents** (one per focus area). Issue all 4 `Agent()` calls in a single message for parallel execution:
|
|
297
|
+
|
|
298
|
+
**Sub-agent 1 - testability focus** (produces TESTABILITY.md + TEST_SURFACE.md):
|
|
299
|
+
```
|
|
300
|
+
Agent(subagent_type="general-purpose",
|
|
301
|
+
prompt="
|
|
302
|
+
<objective>Codebase analysis - testability focus. Produce TESTABILITY.md and TEST_SURFACE.md mapping what is testable in this codebase, mock boundaries, and exhaustive list of testable entry points.</objective>
|
|
303
|
+
<execution_context>@agents/qaa-codebase-mapper.md</execution_context>
|
|
304
|
+
<files_to_read>
|
|
305
|
+
- CLAUDE.md
|
|
306
|
+
- ${output_dir}/SCAN_MANIFEST.md
|
|
307
|
+
</files_to_read>
|
|
308
|
+
<parameters>
|
|
309
|
+
focus_area: testability
|
|
310
|
+
dev_repo_path: {DEV_REPO}
|
|
311
|
+
output_dir: ${output_dir}/codebase
|
|
312
|
+
</parameters>
|
|
313
|
+
"
|
|
314
|
+
)
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
**Sub-agent 2 - risk focus** (produces RISK_MAP.md + CRITICAL_PATHS.md):
|
|
318
|
+
```
|
|
319
|
+
Agent(subagent_type="general-purpose",
|
|
320
|
+
prompt="
|
|
321
|
+
<objective>Codebase analysis - risk focus. Produce RISK_MAP.md and CRITICAL_PATHS.md identifying business-critical paths, security-sensitive areas, and the exact user flows that E2E smoke tests must cover.</objective>
|
|
322
|
+
<execution_context>@agents/qaa-codebase-mapper.md</execution_context>
|
|
323
|
+
<files_to_read>
|
|
324
|
+
- CLAUDE.md
|
|
325
|
+
- ${output_dir}/SCAN_MANIFEST.md
|
|
326
|
+
</files_to_read>
|
|
327
|
+
<parameters>
|
|
328
|
+
focus_area: risk
|
|
329
|
+
dev_repo_path: {DEV_REPO}
|
|
330
|
+
output_dir: ${output_dir}/codebase
|
|
331
|
+
</parameters>
|
|
332
|
+
"
|
|
333
|
+
)
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
**Sub-agent 3 - patterns focus** (produces CODE_PATTERNS.md + API_CONTRACTS.md):
|
|
337
|
+
```
|
|
338
|
+
Agent(subagent_type="general-purpose",
|
|
339
|
+
prompt="
|
|
340
|
+
<objective>Codebase analysis - patterns focus. Produce CODE_PATTERNS.md and API_CONTRACTS.md documenting naming conventions, import style, and exact request/response shapes for API tests.</objective>
|
|
341
|
+
<execution_context>@agents/qaa-codebase-mapper.md</execution_context>
|
|
342
|
+
<files_to_read>
|
|
343
|
+
- CLAUDE.md
|
|
344
|
+
- ${output_dir}/SCAN_MANIFEST.md
|
|
345
|
+
</files_to_read>
|
|
346
|
+
<parameters>
|
|
347
|
+
focus_area: patterns
|
|
348
|
+
dev_repo_path: {DEV_REPO}
|
|
349
|
+
output_dir: ${output_dir}/codebase
|
|
350
|
+
</parameters>
|
|
351
|
+
"
|
|
352
|
+
)
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
**Sub-agent 4 - existing-tests focus** (produces TEST_ASSESSMENT.md + COVERAGE_GAPS.md):
|
|
356
|
+
```
|
|
357
|
+
Agent(subagent_type="general-purpose",
|
|
358
|
+
prompt="
|
|
359
|
+
<objective>Codebase analysis - existing-tests focus. Produce TEST_ASSESSMENT.md and COVERAGE_GAPS.md assessing current test quality, frameworks in use, and identifying which modules/functions/paths have no test coverage.</objective>
|
|
360
|
+
<execution_context>@agents/qaa-codebase-mapper.md</execution_context>
|
|
361
|
+
<files_to_read>
|
|
362
|
+
- CLAUDE.md
|
|
363
|
+
- ${output_dir}/SCAN_MANIFEST.md
|
|
364
|
+
</files_to_read>
|
|
365
|
+
<parameters>
|
|
366
|
+
focus_area: existing-tests
|
|
367
|
+
dev_repo_path: {DEV_REPO}
|
|
368
|
+
output_dir: ${output_dir}/codebase
|
|
369
|
+
</parameters>
|
|
370
|
+
"
|
|
371
|
+
)
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
**Verify codebase docs exist:**
|
|
375
|
+
```bash
|
|
376
|
+
docs_count=$(ls ${output_dir}/codebase/*.md 2>/dev/null | wc -l)
|
|
377
|
+
echo "Codebase docs produced: $docs_count of 8 expected"
|
|
378
|
+
|
|
379
|
+
if [ "$docs_count" -eq 0 ]; then
|
|
380
|
+
echo "FAIL: No codebase docs produced. Pipeline continues with degraded context."
|
|
381
|
+
node bin/qaa-tools.cjs state patch --"Map Status" failed 2>/dev/null || true
|
|
382
|
+
elif [ "$docs_count" -lt 4 ]; then
|
|
383
|
+
echo "WARNING: Fewer than 4 codebase docs produced. Downstream agents will have limited context."
|
|
384
|
+
node bin/qaa-tools.cjs state patch --"Map Status" "partial" 2>/dev/null || true
|
|
385
|
+
else
|
|
386
|
+
echo "OK: codebase map sufficient ($docs_count docs)"
|
|
387
|
+
node bin/qaa-tools.cjs state patch --"Map Status" complete 2>/dev/null || true
|
|
388
|
+
fi
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
**Codebase map is non-blocking:** if zero docs are produced, the pipeline continues with degraded context (downstream agents fall back to whatever they can find). All `<files_to_read>` references to codebase docs in subsequent stages use `(if exists)` semantics.
|
|
392
|
+
|
|
393
|
+
</step>
|
|
394
|
+
|
|
395
|
+
<step name="research">
|
|
396
|
+
|
|
397
|
+
## Step 3c: Research Testing Ecosystem
|
|
398
|
+
|
|
399
|
+
**State update -- mark research as running:**
|
|
400
|
+
```bash
|
|
401
|
+
node bin/qaa-tools.cjs state patch --"Research Status" running --"Status" "Researching testing ecosystem" 2>/dev/null || true
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
**Print stage banner:**
|
|
405
|
+
```
|
|
406
|
+
+------------------------------------------+
|
|
407
|
+
| STAGE 1c: Project Researcher |
|
|
408
|
+
| Status: Running... |
|
|
409
|
+
+------------------------------------------+
|
|
410
|
+
```
|
|
411
|
+
|
|
412
|
+
**Create output directory:**
|
|
413
|
+
```bash
|
|
414
|
+
mkdir -p ${output_dir}/research
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
**Spawn researcher agent:**
|
|
418
|
+
```
|
|
419
|
+
Agent(subagent_type="general-purpose",
|
|
420
|
+
prompt="
|
|
421
|
+
<objective>Research the testing ecosystem for this project. Use the codebase map findings (RISK_MAP, CODE_PATTERNS, API_CONTRACTS, TESTABILITY, etc., from ${output_dir}/codebase/) to ground your Context7 queries to the project specifics — query for the libraries/patterns the project actually uses (e.g., MSW integration, Stripe testing, OpenAPI contract testing) instead of generic framework questions. Use Context7 MCP as the primary source. Produce research documents consumed by downstream agents.</objective>
|
|
422
|
+
<execution_context>@agents/qaa-project-researcher.md</execution_context>
|
|
423
|
+
<files_to_read>
|
|
424
|
+
- CLAUDE.md
|
|
425
|
+
- ~/.claude/qaa/MY_PREFERENCES.md (if exists)
|
|
426
|
+
- ${output_dir}/SCAN_MANIFEST.md
|
|
427
|
+
- ${output_dir}/codebase/TESTABILITY.md (if exists)
|
|
428
|
+
- ${output_dir}/codebase/TEST_SURFACE.md (if exists)
|
|
429
|
+
- ${output_dir}/codebase/RISK_MAP.md (if exists)
|
|
430
|
+
- ${output_dir}/codebase/CRITICAL_PATHS.md (if exists)
|
|
431
|
+
- ${output_dir}/codebase/CODE_PATTERNS.md (if exists)
|
|
432
|
+
- ${output_dir}/codebase/API_CONTRACTS.md (if exists)
|
|
433
|
+
- ${output_dir}/codebase/TEST_ASSESSMENT.md (if exists)
|
|
434
|
+
- ${output_dir}/codebase/COVERAGE_GAPS.md (if exists)
|
|
435
|
+
</files_to_read>
|
|
436
|
+
<parameters>
|
|
437
|
+
mode: stack-testing
|
|
438
|
+
dev_repo_path: {DEV_REPO}
|
|
439
|
+
output_dir: ${output_dir}/research
|
|
440
|
+
</parameters>
|
|
441
|
+
"
|
|
442
|
+
)
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
**Verify research artifacts exist:**
|
|
446
|
+
```bash
|
|
447
|
+
ls ${output_dir}/research/*.md 2>/dev/null && echo "OK: Research files produced" || echo "WARNING: No research files produced"
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
**Research is non-blocking:** If the researcher fails or produces no output, the pipeline continues — downstream agents will fall back to Context7 queries directly. Log the warning but do NOT stop the pipeline.
|
|
451
|
+
|
|
452
|
+
```bash
|
|
453
|
+
if [ ! -f "${output_dir}/research/TESTING_STACK.md" ]; then
|
|
454
|
+
echo "WARNING: Research stage produced no output. Downstream agents will query Context7 directly."
|
|
455
|
+
node bin/qaa-tools.cjs state patch --"Research Status" "skipped (no output)" 2>/dev/null || true
|
|
456
|
+
else
|
|
457
|
+
node bin/qaa-tools.cjs state patch --"Research Status" complete 2>/dev/null || true
|
|
458
|
+
fi
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
</step>
|
|
462
|
+
|
|
463
|
+
<step name="analyze">
|
|
464
|
+
|
|
465
|
+
## Step 4: Analyze Repository
|
|
466
|
+
|
|
467
|
+
**State update -- mark analyze as running:**
|
|
468
|
+
```bash
|
|
469
|
+
node bin/qaa-tools.cjs state patch --"Analyze Status" running --"Status" "Analyzing repository" 2>/dev/null || true
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
**Print stage banner:**
|
|
473
|
+
```
|
|
474
|
+
+------------------------------------------+
|
|
475
|
+
| STAGE 2: Analyzer |
|
|
476
|
+
| Status: Running... |
|
|
477
|
+
+------------------------------------------+
|
|
478
|
+
```
|
|
479
|
+
|
|
480
|
+
**Determine analyzer mode based on option:**
|
|
481
|
+
- Option 1: `mode = 'full'` (produces QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md)
|
|
482
|
+
- Options 2 and 3: `mode = 'gap'` (produces GAP_ANALYSIS.md)
|
|
483
|
+
|
|
484
|
+
**Spawn analyzer agent:**
|
|
485
|
+
```
|
|
486
|
+
Agent(subagent_type="general-purpose",
|
|
487
|
+
prompt="
|
|
488
|
+
<objective>Analyze scanned repository and produce analysis artifacts</objective>
|
|
489
|
+
<execution_context>@agents/qaa-analyzer.md</execution_context>
|
|
490
|
+
<files_to_read>
|
|
491
|
+
- {output_dir}/SCAN_MANIFEST.md
|
|
492
|
+
- CLAUDE.md
|
|
493
|
+
- {output_dir}/research/TESTING_STACK.md (if exists)
|
|
494
|
+
- {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
|
|
495
|
+
- {output_dir}/codebase/RISK_MAP.md (if exists)
|
|
496
|
+
- {output_dir}/codebase/CRITICAL_PATHS.md (if exists)
|
|
497
|
+
- {output_dir}/codebase/TEST_ASSESSMENT.md (if exists)
|
|
498
|
+
- {output_dir}/codebase/COVERAGE_GAPS.md (if exists)
|
|
499
|
+
</files_to_read>
|
|
500
|
+
<parameters>
|
|
501
|
+
mode: {mode}
|
|
502
|
+
workflow_option: {option}
|
|
503
|
+
dev_repo_path: {DEV_REPO}
|
|
504
|
+
qa_repo_path: {QA_REPO or null}
|
|
505
|
+
output_path: {output_dir}/
|
|
506
|
+
</parameters>
|
|
507
|
+
"
|
|
508
|
+
)
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
**Parse analyzer return:**
|
|
512
|
+
|
|
513
|
+
Expected return structure:
|
|
514
|
+
```
|
|
515
|
+
ANALYZER_COMPLETE:
|
|
516
|
+
files_produced: [...]
|
|
517
|
+
total_test_count: N
|
|
518
|
+
pyramid_breakdown: {unit: N, integration: N, api: N, e2e: N}
|
|
519
|
+
risk_count: {high: N, medium: N, low: N}
|
|
520
|
+
commit_hash: "..."
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
Capture `files_produced`, `total_test_count`, `pyramid_breakdown` for downstream stages.
|
|
524
|
+
|
|
525
|
+
**Handle analyzer checkpoint -- assumptions review:**
|
|
526
|
+
- If `IS_AUTO` is true: Auto-approve all assumptions (SAFE checkpoint). Log: "Auto-approved: Analyzer assumptions". Continue pipeline.
|
|
527
|
+
- If `IS_AUTO` is false: Present assumptions to user for review. Wait for confirmation or corrections. On user response, incorporate corrections and continue.
|
|
528
|
+
|
|
529
|
+
**State update -- mark analyze as complete:**
|
|
530
|
+
```bash
|
|
531
|
+
node bin/qaa-tools.cjs state patch --"Analyze Status" complete 2>/dev/null || true
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
**Verify artifacts exist before continuing:**
|
|
535
|
+
|
|
536
|
+
For Option 1:
|
|
537
|
+
```bash
|
|
538
|
+
[ -f "${output_dir}/QA_ANALYSIS.md" ] && echo "OK" || echo "MISSING: QA_ANALYSIS.md"
|
|
539
|
+
[ -f "${output_dir}/TEST_INVENTORY.md" ] && echo "OK" || echo "MISSING: TEST_INVENTORY.md"
|
|
540
|
+
```
|
|
541
|
+
|
|
542
|
+
For Options 2/3:
|
|
543
|
+
```bash
|
|
544
|
+
[ -f "${output_dir}/GAP_ANALYSIS.md" ] && echo "OK" || echo "MISSING: GAP_ANALYSIS.md"
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
If required artifacts are missing, treat as stage failure. Set status to failed and STOP pipeline.
|
|
548
|
+
|
|
549
|
+
Print: "Analysis complete. {total_test_count} test cases identified. Pyramid: unit={unit}, integration={integration}, api={api}, e2e={e2e}."
|
|
550
|
+
|
|
551
|
+
</step>
|
|
552
|
+
|
|
553
|
+
<step name="testid_inject">
|
|
554
|
+
|
|
555
|
+
## Step 5: TestID Injection (Conditional)
|
|
556
|
+
|
|
557
|
+
**Condition:** Only execute if `has_frontend` is `true` from scanner return (Step 3).
|
|
558
|
+
|
|
559
|
+
**If `has_frontend` is false:**
|
|
560
|
+
Print: "Skipping TestID injection (no frontend detected)." Proceed directly to Step 6 (Plan).
|
|
561
|
+
|
|
562
|
+
**If `has_frontend` is true:**
|
|
563
|
+
|
|
564
|
+
**State update:**
|
|
565
|
+
```bash
|
|
566
|
+
node bin/qaa-tools.cjs state patch --"Status" "Injecting test IDs into frontend components" 2>/dev/null || true
|
|
567
|
+
```
|
|
568
|
+
|
|
569
|
+
**Print stage banner:**
|
|
570
|
+
```
|
|
571
|
+
+------------------------------------------+
|
|
572
|
+
| STAGE 3: TestID Injector |
|
|
573
|
+
| Status: Running... |
|
|
574
|
+
+------------------------------------------+
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
**Spawn testid-injector agent:**
|
|
578
|
+
```
|
|
579
|
+
Agent(subagent_type="general-purpose",
|
|
580
|
+
prompt="
|
|
581
|
+
<objective>Audit and inject data-testid attributes into frontend components</objective>
|
|
582
|
+
<execution_context>@agents/qaa-testid-injector.md</execution_context>
|
|
583
|
+
<files_to_read>
|
|
584
|
+
- {output_dir}/SCAN_MANIFEST.md
|
|
585
|
+
- CLAUDE.md
|
|
586
|
+
</files_to_read>
|
|
587
|
+
<parameters>
|
|
588
|
+
dev_repo_path: {DEV_REPO}
|
|
589
|
+
output_path: {output_dir}/TESTID_AUDIT_REPORT.md
|
|
590
|
+
</parameters>
|
|
591
|
+
"
|
|
592
|
+
)
|
|
593
|
+
```
|
|
594
|
+
|
|
595
|
+
**Parse return:**
|
|
596
|
+
|
|
597
|
+
Check for `INJECTOR_COMPLETE` vs `INJECTOR_SKIPPED`:
|
|
598
|
+
|
|
599
|
+
If `INJECTOR_COMPLETE`:
|
|
600
|
+
```
|
|
601
|
+
INJECTOR_COMPLETE:
|
|
602
|
+
report_path: "..."
|
|
603
|
+
coverage_before: N%
|
|
604
|
+
coverage_after: N%
|
|
605
|
+
elements_injected: N
|
|
606
|
+
components_modified: N
|
|
607
|
+
```
|
|
608
|
+
Log: "TestID injection complete. Coverage: {coverage_before}% -> {coverage_after}%. {elements_injected} elements injected."
|
|
609
|
+
|
|
610
|
+
If `INJECTOR_SKIPPED`:
|
|
611
|
+
```
|
|
612
|
+
INJECTOR_SKIPPED:
|
|
613
|
+
reason: "..."
|
|
614
|
+
action: "..."
|
|
615
|
+
```
|
|
616
|
+
Log the reason and continue pipeline.
|
|
617
|
+
|
|
618
|
+
**Handle injector checkpoint -- audit review:**
|
|
619
|
+
- If `IS_AUTO` is true: Auto-approve P0-only injection (SAFE checkpoint). Log: "Auto-approved: TestID injection (P0 elements only)". Continue pipeline.
|
|
620
|
+
- If `IS_AUTO` is false: Present audit report to user. Wait for approval, element selection, or rejection. On user response, incorporate decisions and continue.
|
|
621
|
+
|
|
622
|
+
**Verify artifact exists:**
|
|
623
|
+
```bash
|
|
624
|
+
[ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ] && echo "OK" || echo "MISSING: TESTID_AUDIT_REPORT.md"
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
</step>
|
|
628
|
+
|
|
629
|
+
<step name="plan">
|
|
630
|
+
|
|
631
|
+
## Step 6: Plan Test Generation
|
|
632
|
+
|
|
633
|
+
**State update -- mark generation as running (planning is part of generate):**
|
|
634
|
+
```bash
|
|
635
|
+
node bin/qaa-tools.cjs state patch --"Generate Status" running --"Status" "Planning test generation" 2>/dev/null || true
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
**Print stage banner:**
|
|
639
|
+
```
|
|
640
|
+
+------------------------------------------+
|
|
641
|
+
| STAGE 4: Planner |
|
|
642
|
+
| Status: Running... |
|
|
643
|
+
+------------------------------------------+
|
|
644
|
+
```
|
|
645
|
+
|
|
646
|
+
**Determine planner input based on option:**
|
|
647
|
+
- Option 1: Input from `{output_dir}/TEST_INVENTORY.md` + `{output_dir}/QA_ANALYSIS.md`
|
|
648
|
+
- Options 2 and 3: Input from `{output_dir}/GAP_ANALYSIS.md`
|
|
649
|
+
|
|
650
|
+
**Spawn planner agent:**
|
|
651
|
+
```
|
|
652
|
+
Agent(subagent_type="general-purpose",
|
|
653
|
+
prompt="
|
|
654
|
+
<objective>Create test generation plan with task breakdown and dependencies</objective>
|
|
655
|
+
<execution_context>@agents/qaa-planner.md</execution_context>
|
|
656
|
+
<files_to_read>
|
|
657
|
+
- {input files based on option}
|
|
658
|
+
- CLAUDE.md
|
|
659
|
+
- {output_dir}/codebase/TESTABILITY.md (if exists)
|
|
660
|
+
- {output_dir}/codebase/TEST_SURFACE.md (if exists)
|
|
661
|
+
- {output_dir}/codebase/CRITICAL_PATHS.md (if exists)
|
|
662
|
+
- {output_dir}/codebase/COVERAGE_GAPS.md (if exists)
|
|
663
|
+
</files_to_read>
|
|
664
|
+
<parameters>
|
|
665
|
+
workflow_option: {option}
|
|
666
|
+
output_path: {output_dir}/GENERATION_PLAN.md
|
|
667
|
+
</parameters>
|
|
668
|
+
"
|
|
669
|
+
)
|
|
670
|
+
```
|
|
671
|
+
|
|
672
|
+
**Parse planner return:**
|
|
673
|
+
|
|
674
|
+
Expected return structure:
|
|
675
|
+
```
|
|
676
|
+
PLANNER_COMPLETE:
|
|
677
|
+
file_path: "..."
|
|
678
|
+
total_tasks: N
|
|
679
|
+
total_files: N
|
|
680
|
+
feature_count: N
|
|
681
|
+
dependency_depth: N
|
|
682
|
+
test_case_count: N
|
|
683
|
+
commit_hash: "..."
|
|
684
|
+
```
|
|
685
|
+
|
|
686
|
+
Capture `total_tasks`, `total_files`, `feature_count` for executor stage and pipeline summary.
|
|
687
|
+
|
|
688
|
+
**Verify artifact exists:**
|
|
689
|
+
```bash
|
|
690
|
+
[ -f "${output_dir}/GENERATION_PLAN.md" ] && echo "OK" || echo "MISSING: GENERATION_PLAN.md"
|
|
691
|
+
```
|
|
692
|
+
|
|
693
|
+
If GENERATION_PLAN.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
|
|
694
|
+
|
|
695
|
+
Print: "Plan complete. {total_tasks} tasks, {total_files} files planned across {feature_count} features."
|
|
696
|
+
|
|
697
|
+
</step>
|
|
698
|
+
|
|
699
|
+
<step name="generate">
|
|
700
|
+
|
|
701
|
+
## Step 7: Generate Test Files
|
|
702
|
+
|
|
703
|
+
State update continues from planning (already set to `running` in Step 6).
|
|
704
|
+
|
|
705
|
+
**Print stage banner:**
|
|
706
|
+
```
|
|
707
|
+
+------------------------------------------+
|
|
708
|
+
| STAGE 5: Executor |
|
|
709
|
+
| Generating {total_files} test files |
|
|
710
|
+
| Status: Running... |
|
|
711
|
+
+------------------------------------------+
|
|
712
|
+
```
|
|
713
|
+
|
|
714
|
+
**Determine execution strategy:**
|
|
715
|
+
|
|
716
|
+
Check if planner created multiple independent feature groups. If `feature_count > 1` AND parallelization is enabled (from init config):
|
|
717
|
+
|
|
718
|
+
**Parallel execution** (when feature_count > 1 and parallelization enabled):
|
|
719
|
+
|
|
720
|
+
For each independent feature group from the generation plan, spawn a separate executor agent:
|
|
721
|
+
```
|
|
722
|
+
Agent(subagent_type="general-purpose",
|
|
723
|
+
prompt="
|
|
724
|
+
<objective>Generate test files for {feature} feature</objective>
|
|
725
|
+
<execution_context>@agents/qaa-executor.md</execution_context>
|
|
726
|
+
<files_to_read>
|
|
727
|
+
- {output_dir}/GENERATION_PLAN.md
|
|
728
|
+
- {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
|
|
729
|
+
- CLAUDE.md
|
|
730
|
+
- {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
|
|
731
|
+
- {output_dir}/research/E2E_STRATEGY.md (if exists)
|
|
732
|
+
- {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
|
|
733
|
+
- {output_dir}/codebase/TEST_SURFACE.md (if exists)
|
|
734
|
+
- {output_dir}/codebase/CODE_PATTERNS.md (if exists)
|
|
735
|
+
- {output_dir}/codebase/API_CONTRACTS.md (if exists)
|
|
736
|
+
</files_to_read>
|
|
737
|
+
<parameters>
|
|
738
|
+
workflow_option: {option}
|
|
739
|
+
feature_group: {feature}
|
|
740
|
+
dev_repo_path: {DEV_REPO}
|
|
741
|
+
qa_repo_path: {QA_REPO or null}
|
|
742
|
+
output_path: {output_dir}/
|
|
743
|
+
</parameters>
|
|
744
|
+
"
|
|
745
|
+
)
|
|
746
|
+
```
|
|
747
|
+
|
|
748
|
+
Multiple Agent() calls can be issued simultaneously for independent feature groups. Each executor handles one feature group and commits its files independently.
|
|
749
|
+
|
|
750
|
+
**Sequential execution** (when feature_count == 1 or parallelization disabled):
|
|
751
|
+
|
|
752
|
+
Spawn a single executor agent covering all tasks:
|
|
753
|
+
```
|
|
754
|
+
Agent(subagent_type="general-purpose",
|
|
755
|
+
prompt="
|
|
756
|
+
<objective>Generate all test files from generation plan</objective>
|
|
757
|
+
<execution_context>@agents/qaa-executor.md</execution_context>
|
|
758
|
+
<files_to_read>
|
|
759
|
+
- {output_dir}/GENERATION_PLAN.md
|
|
760
|
+
- {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
|
|
761
|
+
- CLAUDE.md
|
|
762
|
+
- {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
|
|
763
|
+
- {output_dir}/research/E2E_STRATEGY.md (if exists)
|
|
764
|
+
- {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
|
|
765
|
+
- {output_dir}/codebase/TEST_SURFACE.md (if exists)
|
|
766
|
+
- {output_dir}/codebase/CODE_PATTERNS.md (if exists)
|
|
767
|
+
- {output_dir}/codebase/API_CONTRACTS.md (if exists)
|
|
768
|
+
</files_to_read>
|
|
769
|
+
<parameters>
|
|
770
|
+
workflow_option: {option}
|
|
771
|
+
dev_repo_path: {DEV_REPO}
|
|
772
|
+
qa_repo_path: {QA_REPO or null}
|
|
773
|
+
output_path: {output_dir}/
|
|
774
|
+
</parameters>
|
|
775
|
+
"
|
|
776
|
+
)
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
**Option 3 specific -- skip existing tests:**
|
|
780
|
+
|
|
781
|
+
For Option 3, pass `skip_existing_test_ids: true` to the executor so it checks existing test files by test ID before generating. If a test ID already exists in the QA repo, skip generating that test case:
|
|
782
|
+
```
|
|
783
|
+
<parameters>
|
|
784
|
+
workflow_option: 3
|
|
785
|
+
skip_existing_test_ids: true
|
|
786
|
+
dev_repo_path: {DEV_REPO}
|
|
787
|
+
qa_repo_path: {QA_REPO}
|
|
788
|
+
output_path: {output_dir}/
|
|
789
|
+
</parameters>
|
|
790
|
+
```
|
|
791
|
+
|
|
792
|
+
**Parse executor return:**
|
|
793
|
+
|
|
794
|
+
Expected return structure:
|
|
795
|
+
```
|
|
796
|
+
EXECUTOR_COMPLETE:
|
|
797
|
+
files_created: [{path, type}, ...]
|
|
798
|
+
total_files: N
|
|
799
|
+
commit_count: N
|
|
800
|
+
features_covered: [...]
|
|
801
|
+
test_case_count: N
|
|
802
|
+
```
|
|
803
|
+
|
|
804
|
+
Capture `files_created`, `total_files`, `commit_count` for validation stage and pipeline summary.
|
|
805
|
+
|
|
806
|
+
**State update -- mark generate as complete:**
|
|
807
|
+
```bash
|
|
808
|
+
node bin/qaa-tools.cjs state patch --"Generate Status" complete --"Status" "Test generation complete" 2>/dev/null || true
|
|
809
|
+
```
|
|
810
|
+
|
|
811
|
+
Print: "Generation complete. {total_files} files created across {features_covered_count} features. {commit_count} commits."
|
|
812
|
+
|
|
813
|
+
</step>
|
|
814
|
+
|
|
815
|
+
<step name="validate">
|
|
816
|
+
|
|
817
|
+
## Step 8: Validate Generated Tests
|
|
818
|
+
|
|
819
|
+
**State update -- mark validate as running:**
|
|
820
|
+
```bash
|
|
821
|
+
node bin/qaa-tools.cjs state patch --"Validate Status" running --"Status" "Validating generated tests" 2>/dev/null || true
|
|
822
|
+
```
|
|
823
|
+
|
|
824
|
+
**Print stage banner:**
|
|
825
|
+
```
|
|
826
|
+
+------------------------------------------+
|
|
827
|
+
| STAGE 6: Validator |
|
|
828
|
+
| Validating {total_files} test files |
|
|
829
|
+
| Status: Running... |
|
|
830
|
+
+------------------------------------------+
|
|
831
|
+
```
|
|
832
|
+
|
|
833
|
+
**Spawn validator agent:**
|
|
834
|
+
```
|
|
835
|
+
Agent(subagent_type="general-purpose",
|
|
836
|
+
prompt="
|
|
837
|
+
<objective>Run 4-layer validation on all generated test files</objective>
|
|
838
|
+
<execution_context>@agents/qaa-validator.md</execution_context>
|
|
839
|
+
<files_to_read>
|
|
840
|
+
- {list all generated test files from executor return -- files_created paths}
|
|
841
|
+
- {output_dir}/GENERATION_PLAN.md
|
|
842
|
+
- CLAUDE.md
|
|
843
|
+
</files_to_read>
|
|
844
|
+
<parameters>
|
|
845
|
+
mode: validation
|
|
846
|
+
max_fix_loops: 3
|
|
847
|
+
output_path: {output_dir}/VALIDATION_REPORT.md
|
|
848
|
+
</parameters>
|
|
849
|
+
"
|
|
850
|
+
)
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
**4-layer validation:**
|
|
854
|
+
1. **Syntax** -- File parses without errors
|
|
855
|
+
2. **Structure** -- Follows POM rules, naming conventions, locator hierarchy
|
|
856
|
+
3. **Dependencies** -- Imports resolve, fixtures exist, configs present
|
|
857
|
+
4. **Logic** -- Assertions are concrete, test IDs are unique, no assertions in page objects
|
|
858
|
+
|
|
859
|
+
**5. Browser verification (if app URL available and Playwright MCP connected):**
|
|
860
|
+
|
|
861
|
+
After the 4-layer static validation, use Playwright MCP to verify E2E tests against the live app:
|
|
862
|
+
|
|
863
|
+
1. Navigate to each page referenced in the E2E tests:
|
|
864
|
+
```
|
|
865
|
+
mcp__playwright__browser_navigate({ url: "{app_url}/{route}" })
|
|
866
|
+
```
|
|
867
|
+
|
|
868
|
+
2. Take an accessibility snapshot to verify locators used in tests actually exist:
|
|
869
|
+
```
|
|
870
|
+
mcp__playwright__browser_snapshot()
|
|
871
|
+
```
|
|
872
|
+
|
|
873
|
+
3. Cross-reference locators in generated tests against the real DOM:
|
|
874
|
+
- Verify `data-testid` values exist on the page
|
|
875
|
+
- Verify ARIA roles and names match test expectations
|
|
876
|
+
- Flag any test locator that does not match a real DOM element
|
|
877
|
+
|
|
878
|
+
4. If mismatches are found, fix the test locators to match the real DOM and count as a fix loop iteration.
|
|
879
|
+
|
|
880
|
+
This browser verification step prevents delivering tests with locators that will immediately fail at runtime.
|
|
881
|
+
|
|
882
|
+
**Fix loop:** The validator automatically attempts to fix issues it finds. Maximum 3 fix loop iterations. After each fix attempt, re-validate.
|
|
883
|
+
|
|
884
|
+
**Parse validator return:**
|
|
885
|
+
|
|
886
|
+
Expected return structure:
|
|
887
|
+
```
|
|
888
|
+
VALIDATOR_COMPLETE:
|
|
889
|
+
report_path: "..."
|
|
890
|
+
overall_status: PASS | PASS_WITH_WARNINGS | FAIL
|
|
891
|
+
confidence: HIGH | MEDIUM | LOW
|
|
892
|
+
layers_summary: {syntax: PASS|FAIL, structure: PASS|FAIL, dependencies: PASS|FAIL, logic: PASS|FAIL}
|
|
893
|
+
fix_loops_used: N
|
|
894
|
+
issues_found: N
|
|
895
|
+
issues_fixed: N
|
|
896
|
+
unresolved_count: N
|
|
897
|
+
```
|
|
898
|
+
|
|
899
|
+
**RISKY CHECKPOINT -- Validator escalation:**
|
|
900
|
+
|
|
901
|
+
If `unresolved_count > 0` after max fix loops (3):
|
|
902
|
+
- **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
|
|
903
|
+
- Present unresolved issues to user with full details from VALIDATION_REPORT.md
|
|
904
|
+
- Wait for user decision:
|
|
905
|
+
- `"approve-with-warnings"`: Accept the validation with warnings. Set Validate Status to complete. Continue to deliver.
|
|
906
|
+
- `"abort"`: Set Validate Status to failed. STOP PIPELINE ENTIRELY.
|
|
907
|
+
- Manual guidance: User provides specific fix instructions. Spawn fresh continuation agent to apply fixes and re-validate.
|
|
908
|
+
|
|
909
|
+
If `overall_status` is `PASS` or `PASS_WITH_WARNINGS` (and unresolved_count is 0):
|
|
910
|
+
```bash
|
|
911
|
+
node bin/qaa-tools.cjs state patch --"Validate Status" complete --"Status" "Validation passed" 2>/dev/null || true
|
|
912
|
+
```
|
|
913
|
+
|
|
914
|
+
**Verify artifact exists:**
|
|
915
|
+
```bash
|
|
916
|
+
[ -f "${output_dir}/VALIDATION_REPORT.md" ] && echo "OK" || echo "MISSING: VALIDATION_REPORT.md"
|
|
917
|
+
```
|
|
918
|
+
|
|
919
|
+
Print: "Validation complete. Status: {overall_status}. Confidence: {confidence}. {issues_found} issues found, {issues_fixed} fixed, {unresolved_count} unresolved."
|
|
920
|
+
|
|
921
|
+
</step>
|
|
922
|
+
|
|
923
|
+
<step name="bug_detective">
|
|
924
|
+
|
|
925
|
+
## Step 9: Bug Detective (Conditional)
|
|
926
|
+
|
|
927
|
+
**Condition:** Only execute if test failures were detected during validation. Check:
|
|
928
|
+
- `overall_status === 'FAIL'` in validator return, OR
|
|
929
|
+
- Generated tests have runtime failures that need classification
|
|
930
|
+
|
|
931
|
+
**If no failures to classify:**
|
|
932
|
+
Print: "Skipping Bug Detective (no test failures detected)." Proceed directly to Step 10 (Deliver).
|
|
933
|
+
|
|
934
|
+
**If failures need classification:**
|
|
935
|
+
|
|
936
|
+
**State update:**
|
|
937
|
+
```bash
|
|
938
|
+
node bin/qaa-tools.cjs state patch --"Status" "Classifying test failures" 2>/dev/null || true
|
|
939
|
+
```
|
|
940
|
+
|
|
941
|
+
**Print stage banner:**
|
|
942
|
+
```
|
|
943
|
+
+------------------------------------------+
|
|
944
|
+
| STAGE 7: Bug Detective |
|
|
945
|
+
| Status: Running... |
|
|
946
|
+
+------------------------------------------+
|
|
947
|
+
```
|
|
948
|
+
|
|
949
|
+
**Spawn bug-detective agent:**
|
|
950
|
+
```
|
|
951
|
+
Agent(subagent_type="general-purpose",
|
|
952
|
+
prompt="
|
|
953
|
+
<objective>Classify test failures and attempt auto-fixes for test errors. Use Playwright MCP to reproduce E2E failures in the browser when available.</objective>
|
|
954
|
+
<execution_context>@agents/qaa-bug-detective.md</execution_context>
|
|
955
|
+
<files_to_read>
|
|
956
|
+
- {test execution results -- from validator or direct test run}
|
|
957
|
+
- {failing test source files -- paths from executor return}
|
|
958
|
+
- CLAUDE.md
|
|
959
|
+
- {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
|
|
960
|
+
- {output_dir}/research/TESTING_STACK.md (if exists)
|
|
961
|
+
</files_to_read>
|
|
962
|
+
<parameters>
|
|
963
|
+
output_path: {output_dir}/FAILURE_CLASSIFICATION_REPORT.md
|
|
964
|
+
app_url: {app_url if available}
|
|
965
|
+
</parameters>
|
|
966
|
+
"
|
|
967
|
+
)
|
|
968
|
+
```
|
|
969
|
+
|
|
970
|
+
**Parse bug-detective return:**
|
|
971
|
+
|
|
972
|
+
Expected return structure:
|
|
973
|
+
```
|
|
974
|
+
DETECTIVE_COMPLETE:
|
|
975
|
+
report_path: "..."
|
|
976
|
+
total_failures: N
|
|
977
|
+
classification_breakdown: {app_bug: N, test_error: N, env_issue: N, inconclusive: N}
|
|
978
|
+
auto_fixes_applied: N
|
|
979
|
+
auto_fixes_verified: N
|
|
980
|
+
commit_hash: "..."
|
|
981
|
+
```
|
|
982
|
+
|
|
983
|
+
**RISKY CHECKPOINT -- Application bugs detected:**
|
|
984
|
+
|
|
985
|
+
If `classification_breakdown.app_bug > 0`:
|
|
986
|
+
- **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
|
|
987
|
+
- Present APPLICATION BUG classifications to user with full evidence from FAILURE_CLASSIFICATION_REPORT.md
|
|
988
|
+
- These are genuine bugs in the application code discovered during test execution
|
|
989
|
+
- The bug detective never touches application code -- it only reports
|
|
990
|
+
- User must review and decide how to proceed:
|
|
991
|
+
- Acknowledge bugs and continue pipeline (bugs will be in the PR description for developer attention)
|
|
992
|
+
- Abort pipeline to fix bugs first
|
|
993
|
+
|
|
994
|
+
**Verify artifact exists:**
|
|
995
|
+
```bash
|
|
996
|
+
[ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "OK" || echo "MISSING: FAILURE_CLASSIFICATION_REPORT.md"
|
|
997
|
+
```
|
|
998
|
+
|
|
999
|
+
Print: "Bug Detective complete. {total_failures} failures classified: {app_bug} APP BUG, {test_error} TEST ERROR, {env_issue} ENV ISSUE, {inconclusive} INCONCLUSIVE. {auto_fixes_applied} auto-fixes applied."
|
|
1000
|
+
|
|
1001
|
+
</step>
|
|
1002
|
+
|
|
1003
|
+
<step name="deliver">
|
|
1004
|
+
|
|
1005
|
+
## Step 10: Deliver
|
|
1006
|
+
|
|
1007
|
+
**State update -- mark deliver as running:**
|
|
1008
|
+
```bash
|
|
1009
|
+
node bin/qaa-tools.cjs state patch --"Deliver Status" running --"Status" "Preparing delivery" 2>/dev/null || true
|
|
1010
|
+
```
|
|
1011
|
+
|
|
1012
|
+
**Print stage banner:**
|
|
1013
|
+
```
|
|
1014
|
+
+------------------------------------------+
|
|
1015
|
+
| STAGE 8: Deliver |
|
|
1016
|
+
| Status: Running... |
|
|
1017
|
+
+------------------------------------------+
|
|
1018
|
+
```
|
|
1019
|
+
|
|
1020
|
+
### Sub-step 1: Pre-flight checks
|
|
1021
|
+
|
|
1022
|
+
**Check for git remote:**
|
|
1023
|
+
```bash
|
|
1024
|
+
REMOTE_URL=$(git remote get-url origin 2>/dev/null || echo "")
|
|
1025
|
+
```
|
|
1026
|
+
|
|
1027
|
+
If `REMOTE_URL` is empty:
|
|
1028
|
+
- Print: "No git remote found. Artifacts committed locally but PR creation skipped."
|
|
1029
|
+
- Set `LOCAL_ONLY=true`
|
|
1030
|
+
|
|
1031
|
+
**Check for gh CLI authentication:**
|
|
1032
|
+
```bash
|
|
1033
|
+
gh auth status 2>/dev/null
|
|
1034
|
+
```
|
|
1035
|
+
|
|
1036
|
+
If `gh auth status` fails:
|
|
1037
|
+
- Print: "gh CLI not authenticated. Run 'gh auth login' first. Artifacts committed locally."
|
|
1038
|
+
- Set `LOCAL_ONLY=true`
|
|
1039
|
+
|
|
1040
|
+
If both checks pass, set `LOCAL_ONLY=false`.
|
|
1041
|
+
|
|
1042
|
+
### Sub-step 2: Derive project name
|
|
1043
|
+
|
|
1044
|
+
```bash
|
|
1045
|
+
# Read from package.json
|
|
1046
|
+
PROJECT_NAME=$(node -e "try { const p = require('${DEV_REPO}/package.json'); console.log(p.name || ''); } catch { console.log(''); }" 2>/dev/null)
|
|
1047
|
+
|
|
1048
|
+
# Fallback to directory basename
|
|
1049
|
+
if [ -z "$PROJECT_NAME" ]; then
|
|
1050
|
+
PROJECT_NAME=$(basename "${DEV_REPO}")
|
|
1051
|
+
fi
|
|
1052
|
+
|
|
1053
|
+
# Sanitize for branch naming
|
|
1054
|
+
PROJECT_NAME=$(echo "$PROJECT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
|
|
1055
|
+
```
|
|
1056
|
+
|
|
1057
|
+
### Sub-step 3: Detect default branch
|
|
1058
|
+
|
|
1059
|
+
```bash
|
|
1060
|
+
DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name' 2>/dev/null || echo "main")
|
|
1061
|
+
```
|
|
1062
|
+
|
|
1063
|
+
### Sub-step 4: Create feature branch
|
|
1064
|
+
|
|
1065
|
+
```bash
|
|
1066
|
+
BRANCH="qa/auto-${PROJECT_NAME}-${date}"
|
|
1067
|
+
|
|
1068
|
+
# Handle branch name collision
|
|
1069
|
+
if git rev-parse --verify "$BRANCH" 2>/dev/null || git rev-parse --verify "origin/$BRANCH" 2>/dev/null; then
|
|
1070
|
+
SUFFIX=2
|
|
1071
|
+
while git rev-parse --verify "${BRANCH}-${SUFFIX}" 2>/dev/null || git rev-parse --verify "origin/${BRANCH}-${SUFFIX}" 2>/dev/null; do
|
|
1072
|
+
SUFFIX=$((SUFFIX + 1))
|
|
1073
|
+
done
|
|
1074
|
+
BRANCH="${BRANCH}-${SUFFIX}"
|
|
1075
|
+
fi
|
|
1076
|
+
|
|
1077
|
+
git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
|
|
1078
|
+
```
|
|
1079
|
+
|
|
1080
|
+
### Sub-step 5: Per-stage atomic commits
|
|
1081
|
+
|
|
1082
|
+
For each pipeline stage that produced artifacts, commit using `qaa-tools.cjs commit`. Check file existence before each commit.
|
|
1083
|
+
|
|
1084
|
+
**Scanner:**
|
|
1085
|
+
```bash
|
|
1086
|
+
if [ -f "${output_dir}/SCAN_MANIFEST.md" ]; then
|
|
1087
|
+
node bin/qaa-tools.cjs commit "qa(scanner): produce SCAN_MANIFEST.md for ${PROJECT_NAME}" --files ${output_dir}/SCAN_MANIFEST.md
|
|
1088
|
+
fi
|
|
1089
|
+
```
|
|
1090
|
+
|
|
1091
|
+
**Analyzer (Option 1):**
|
|
1092
|
+
```bash
|
|
1093
|
+
if [ -f "${output_dir}/QA_ANALYSIS.md" ]; then
|
|
1094
|
+
ANALYZER_FILES="${output_dir}/QA_ANALYSIS.md ${output_dir}/TEST_INVENTORY.md"
|
|
1095
|
+
[ -f "${output_dir}/QA_REPO_BLUEPRINT.md" ] && ANALYZER_FILES="${ANALYZER_FILES} ${output_dir}/QA_REPO_BLUEPRINT.md"
|
|
1096
|
+
node bin/qaa-tools.cjs commit "qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md" --files ${ANALYZER_FILES}
|
|
1097
|
+
fi
|
|
1098
|
+
```
|
|
1099
|
+
|
|
1100
|
+
**Analyzer (Options 2/3):**
|
|
1101
|
+
```bash
|
|
1102
|
+
if [ -f "${output_dir}/GAP_ANALYSIS.md" ]; then
|
|
1103
|
+
node bin/qaa-tools.cjs commit "qa(analyzer): produce GAP_ANALYSIS.md" --files ${output_dir}/GAP_ANALYSIS.md
|
|
1104
|
+
fi
|
|
1105
|
+
```
|
|
1106
|
+
|
|
1107
|
+
**TestID Injector (if ran):**
|
|
1108
|
+
```bash
|
|
1109
|
+
if [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ]; then
|
|
1110
|
+
node bin/qaa-tools.cjs commit "qa(testid-injector): inject ${elements_injected} data-testid attributes across ${components_modified} components" --files ${output_dir}/TESTID_AUDIT_REPORT.md ${modified_source_files}
|
|
1111
|
+
fi
|
|
1112
|
+
```
|
|
1113
|
+
|
|
1114
|
+
**Executor:**
|
|
1115
|
+
```bash
|
|
1116
|
+
if [ -n "${generated_file_paths}" ]; then
|
|
1117
|
+
node bin/qaa-tools.cjs commit "qa(executor): generate ${total_files} test files with POMs and fixtures" --files ${generated_file_paths}
|
|
1118
|
+
fi
|
|
1119
|
+
```
|
|
1120
|
+
|
|
1121
|
+
**Planner:**
|
|
1122
|
+
```bash
|
|
1123
|
+
if [ -f "${output_dir}/GENERATION_PLAN.md" ]; then
|
|
1124
|
+
node bin/qaa-tools.cjs commit "qa(planner): produce GENERATION_PLAN.md" --files ${output_dir}/GENERATION_PLAN.md
|
|
1125
|
+
fi
|
|
1126
|
+
```
|
|
1127
|
+
|
|
1128
|
+
**Validator:**
|
|
1129
|
+
```bash
|
|
1130
|
+
if [ -f "${output_dir}/VALIDATION_REPORT.md" ]; then
|
|
1131
|
+
node bin/qaa-tools.cjs commit "qa(validator): validate generated tests - ${overall_status} with ${confidence} confidence" --files ${output_dir}/VALIDATION_REPORT.md
|
|
1132
|
+
fi
|
|
1133
|
+
```
|
|
1134
|
+
|
|
1135
|
+
**Bug Detective (if ran):**
|
|
1136
|
+
```bash
|
|
1137
|
+
if [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ]; then
|
|
1138
|
+
node bin/qaa-tools.cjs commit "qa(bug-detective): classify ${total_failures} failures - ${classification_summary}" --files ${output_dir}/FAILURE_CLASSIFICATION_REPORT.md
|
|
1139
|
+
fi
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
### Sub-step 6: Push branch
|
|
1143
|
+
|
|
1144
|
+
If `LOCAL_ONLY` is true, skip this sub-step.
|
|
1145
|
+
|
|
1146
|
+
```bash
|
|
1147
|
+
git push -u origin "$BRANCH"
|
|
1148
|
+
```
|
|
1149
|
+
|
|
1150
|
+
If push fails:
|
|
1151
|
+
- Print: "Push failed: {error_message}. Artifacts committed locally on branch ${BRANCH}."
|
|
1152
|
+
- Set `LOCAL_ONLY=true`
|
|
1153
|
+
|
|
1154
|
+
### Sub-step 7: Build PR body
|
|
1155
|
+
|
|
1156
|
+
If `LOCAL_ONLY` is true, skip this sub-step.
|
|
1157
|
+
|
|
1158
|
+
Read the PR template:
|
|
1159
|
+
```bash
|
|
1160
|
+
PR_BODY=$(cat templates/pr-template.md)
|
|
1161
|
+
```
|
|
1162
|
+
|
|
1163
|
+
Replace all `{placeholder}` tokens with actual values collected during pipeline execution:
|
|
1164
|
+
- `{architecture_type}` -- from QA_ANALYSIS.md or SCAN_MANIFEST.md
|
|
1165
|
+
- `{framework}` -- detected test framework
|
|
1166
|
+
- `{risk_summary}` -- risk assessment counts (e.g., "3 HIGH, 5 MEDIUM, 2 LOW")
|
|
1167
|
+
- `{unit_count}` -- from pyramid_breakdown.unit
|
|
1168
|
+
- `{integration_count}` -- from pyramid_breakdown.integration
|
|
1169
|
+
- `{api_count}` -- from pyramid_breakdown.api
|
|
1170
|
+
- `{e2e_count}` -- from pyramid_breakdown.e2e
|
|
1171
|
+
- `{total_count}` -- from total_test_count
|
|
1172
|
+
- `{modules_covered}` -- count of modules with tests
|
|
1173
|
+
- `{coverage_estimate}` -- estimated coverage percentage
|
|
1174
|
+
- `{validation_result}` -- PASS, PASS_WITH_WARNINGS, or FAIL
|
|
1175
|
+
- `{confidence}` -- HIGH, MEDIUM, or LOW
|
|
1176
|
+
- `{fix_loops_used}` -- number 0-3
|
|
1177
|
+
- `{issues_found}` -- total issues found during validation
|
|
1178
|
+
- `{issues_fixed}` -- total issues auto-fixed
|
|
1179
|
+
- `{file_list}` -- if total files <= 50, list each file; if > 50, use summary
|
|
1180
|
+
|
|
1181
|
+
### Sub-step 8: Create draft PR
|
|
1182
|
+
|
|
1183
|
+
If `LOCAL_ONLY` is true, skip this sub-step.
|
|
1184
|
+
|
|
1185
|
+
```bash
|
|
1186
|
+
PR_URL=$(gh pr create \
|
|
1187
|
+
--draft \
|
|
1188
|
+
--title "qa: automated test suite for ${PROJECT_NAME}" \
|
|
1189
|
+
--body "${PR_BODY}" \
|
|
1190
|
+
--label "qa-automation" \
|
|
1191
|
+
--label "auto-generated" \
|
|
1192
|
+
--assignee "@me" 2>&1)
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
Do NOT pass `--base` flag. Let gh auto-detect the default branch.
|
|
1196
|
+
|
|
1197
|
+
On failure:
|
|
1198
|
+
- Print: "PR creation failed: ${PR_URL}. Artifacts remain on branch ${BRANCH}."
|
|
1199
|
+
- Do NOT stop the pipeline -- artifacts are committed and pushed.
|
|
1200
|
+
|
|
1201
|
+
### Sub-step 9: Print result
|
|
1202
|
+
|
|
1203
|
+
If PR was created successfully:
|
|
1204
|
+
```
|
|
1205
|
+
PR created: ${PR_URL}
|
|
1206
|
+
```
|
|
1207
|
+
|
|
1208
|
+
If `LOCAL_ONLY` is true:
|
|
1209
|
+
```
|
|
1210
|
+
PR: not created (local-only mode). Artifacts committed on branch: ${BRANCH}
|
|
1211
|
+
```
|
|
1212
|
+
|
|
1213
|
+
**State update -- mark deliver as complete:**
|
|
1214
|
+
```bash
|
|
1215
|
+
node bin/qaa-tools.cjs state patch --"Deliver Status" complete --"Status" "Pipeline complete" 2>/dev/null || true
|
|
1216
|
+
```
|
|
1217
|
+
|
|
1218
|
+
**Clear auto-chain flag at pipeline completion:**
|
|
1219
|
+
```bash
|
|
1220
|
+
node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
|
|
1221
|
+
```
|
|
1222
|
+
|
|
1223
|
+
### Print pipeline summary banner:
|
|
1224
|
+
|
|
1225
|
+
```
|
|
1226
|
+
======================================================
|
|
1227
|
+
QA PIPELINE COMPLETE
|
|
1228
|
+
======================================================
|
|
1229
|
+
|
|
1230
|
+
Option: {option} ({option_description})
|
|
1231
|
+
Repository: {DEV_REPO}
|
|
1232
|
+
QA Repo: {QA_REPO or 'N/A'}
|
|
1233
|
+
Maturity Score: {maturity_score or 'N/A'}
|
|
1234
|
+
|
|
1235
|
+
Stages Completed:
|
|
1236
|
+
[{check}] Scan -- {scan_result}
|
|
1237
|
+
[{check}] Analyze -- {analyze_result} ({test_count} test cases)
|
|
1238
|
+
[{check}] TestID Inject -- {inject_result or 'skipped'}
|
|
1239
|
+
[{check}] Plan -- {plan_result} ({file_count} files planned)
|
|
1240
|
+
[{check}] Generate -- {generate_result} ({files_created} files created)
|
|
1241
|
+
[{check}] Validate -- {validate_result} ({confidence} confidence)
|
|
1242
|
+
[{check}] Bug Detective -- {detective_result or 'skipped'}
|
|
1243
|
+
[{check}] Deliver -- {deliver_result}
|
|
1244
|
+
|
|
1245
|
+
PR: {pr_url or 'not created (local-only)'}
|
|
1246
|
+
|
|
1247
|
+
Artifacts:
|
|
1248
|
+
{list all produced .md files in output_dir}
|
|
1249
|
+
|
|
1250
|
+
Total Time: {total_duration}
|
|
1251
|
+
======================================================
|
|
1252
|
+
```
|
|
1253
|
+
|
|
1254
|
+
Where: `[x]` = completed, `[ ]` = skipped, `[!]` = failed.
|
|
1255
|
+
|
|
1256
|
+
</step>
|
|
1257
|
+
|
|
1258
|
+
</process>
|
|
1259
|
+
|
|
1260
|
+
<error_handling>
|
|
1261
|
+
|
|
1262
|
+
## Error Handling
|
|
1263
|
+
|
|
1264
|
+
### Stage Failure Protocol
|
|
1265
|
+
|
|
1266
|
+
When any agent returns a failure or error:
|
|
1267
|
+
|
|
1268
|
+
1. **Set stage status to failed:**
|
|
1269
|
+
```bash
|
|
1270
|
+
node bin/qaa-tools.cjs state patch --"{Stage} Status" failed --"Status" "Pipeline stopped: {Stage} failed - {reason}" 2>/dev/null || true
|
|
1271
|
+
```
|
|
1272
|
+
|
|
1273
|
+
2. **Print failure banner:**
|
|
1274
|
+
```
|
|
1275
|
+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
|
1276
|
+
! PIPELINE STOPPED !
|
|
1277
|
+
! Stage: {stage_name} !
|
|
1278
|
+
! Reason: {failure_reason} !
|
|
1279
|
+
! !
|
|
1280
|
+
! Completed: {completed_stages} !
|
|
1281
|
+
! Artifacts: {artifacts_so_far} !
|
|
1282
|
+
! !
|
|
1283
|
+
! Action required: Review and re-run !
|
|
1284
|
+
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
|
1285
|
+
```
|
|
1286
|
+
|
|
1287
|
+
3. **DO NOT continue to next stage.** The pipeline stops entirely at the failed stage.
|
|
1288
|
+
|
|
1289
|
+
4. **DO NOT create partial PR.** No branch, no commit, no PR with incomplete results.
|
|
1290
|
+
|
|
1291
|
+
5. **Preserve all artifacts produced so far.** They remain on disk in `{output_dir}/` for debugging.
|
|
1292
|
+
|
|
1293
|
+
### Artifact Verification
|
|
1294
|
+
|
|
1295
|
+
After EVERY agent spawn, before advancing to next stage, verify the expected output artifact exists on disk:
|
|
1296
|
+
```bash
|
|
1297
|
+
[ -f "{expected_artifact_path}" ] && echo "OK" || echo "MISSING"
|
|
1298
|
+
```
|
|
1299
|
+
|
|
1300
|
+
If artifacts are missing, treat as stage failure and STOP pipeline.
|
|
1301
|
+
|
|
1302
|
+
### qaa-tools.cjs Graceful Fallback
|
|
1303
|
+
|
|
1304
|
+
All `node bin/qaa-tools.cjs` calls use `2>/dev/null || true` to handle cases where the tool is not installed or not found. The pipeline must not break due to missing state management tooling -- it logs a warning and continues.
|
|
1305
|
+
|
|
1306
|
+
</error_handling>
|
|
1307
|
+
|
|
1308
|
+
<auto_advance>
|
|
1309
|
+
|
|
1310
|
+
## Auto-Advance Mode
|
|
1311
|
+
|
|
1312
|
+
Auto-advance is enabled when ANY of these is true:
|
|
1313
|
+
- `--auto` flag passed to the `/qa-start` invocation
|
|
1314
|
+
- `config.json` has `workflow.auto_advance = true` (persistent user preference)
|
|
1315
|
+
- `workflow._auto_chain_active = true` in config (ephemeral chain flag from ongoing auto run)
|
|
1316
|
+
|
|
1317
|
+
### Behavior in Auto Mode
|
|
1318
|
+
|
|
1319
|
+
**SAFE checkpoints are auto-approved.** The pipeline continues without pausing. A log message records the auto-approval:
|
|
1320
|
+
```
|
|
1321
|
+
Auto-approved: {checkpoint_description}
|
|
1322
|
+
```
|
|
1323
|
+
|
|
1324
|
+
**RISKY checkpoints ALWAYS pause.** Even in auto mode, the pipeline stops and presents the checkpoint to the user.
|
|
1325
|
+
|
|
1326
|
+
### Safe vs Risky Checkpoint Classification
|
|
1327
|
+
|
|
1328
|
+
**SAFE (auto-approve in auto mode):**
|
|
1329
|
+
|
|
1330
|
+
| Checkpoint | Agent | Auto-Action |
|
|
1331
|
+
|------------|-------|-------------|
|
|
1332
|
+
| Framework detection uncertain (LOW confidence) | Scanner | Approve with most likely framework |
|
|
1333
|
+
| Analyzer assumptions review | Analyzer | Approve all assumptions |
|
|
1334
|
+
| TestID audit review | TestID Injector | Approve P0-only injection |
|
|
1335
|
+
|
|
1336
|
+
**RISKY (ALWAYS pause, even in auto mode):**
|
|
1337
|
+
|
|
1338
|
+
| Checkpoint | Agent | User Action Required |
|
|
1339
|
+
|------------|-------|---------------------|
|
|
1340
|
+
| Validator escalation (unresolved issues after 3 fix loops) | Validator | approve-with-warnings, abort, or fix guidance |
|
|
1341
|
+
| APPLICATION BUG classification | Bug Detective | Review bugs, continue or fix first |
|
|
1342
|
+
| Any checkpoint with "unresolved" or "failed" blocking text | Any | Review specific blocking issue |
|
|
1343
|
+
|
|
1344
|
+
### Checkpoint Handling Flow
|
|
1345
|
+
|
|
1346
|
+
```
|
|
1347
|
+
On agent return with checkpoint data:
|
|
1348
|
+
1. Extract checkpoint blocking field content
|
|
1349
|
+
2. Classify as SAFE or RISKY:
|
|
1350
|
+
- "framework detection" -> SAFE
|
|
1351
|
+
- "assumptions" -> SAFE
|
|
1352
|
+
- "audit" or "data-testid" -> SAFE
|
|
1353
|
+
- "unresolved" -> RISKY
|
|
1354
|
+
- "failed" -> RISKY
|
|
1355
|
+
- "APPLICATION BUG" -> RISKY
|
|
1356
|
+
- Default (no pattern match) -> RISKY (conservative)
|
|
1357
|
+
3. If IS_AUTO and SAFE:
|
|
1358
|
+
- Auto-approve with default action
|
|
1359
|
+
- Log the auto-approval
|
|
1360
|
+
- Continue pipeline
|
|
1361
|
+
4. If IS_AUTO and RISKY:
|
|
1362
|
+
- PAUSE pipeline
|
|
1363
|
+
- Print checkpoint details with full context
|
|
1364
|
+
- Wait for user input
|
|
1365
|
+
5. If NOT auto (manual mode):
|
|
1366
|
+
- PAUSE pipeline
|
|
1367
|
+
- Print checkpoint details
|
|
1368
|
+
- Wait for user input
|
|
1369
|
+
```
|
|
1370
|
+
|
|
1371
|
+
### Resume After Checkpoint
|
|
1372
|
+
|
|
1373
|
+
When resuming after a checkpoint, spawn a FRESH agent with explicit state:
|
|
1374
|
+
```
|
|
1375
|
+
Agent(subagent_type="general-purpose",
|
|
1376
|
+
prompt="
|
|
1377
|
+
<objective>Continue QA pipeline from {stage} stage</objective>
|
|
1378
|
+
<execution_context>@agents/qa-pipeline-orchestrator.md</execution_context>
|
|
1379
|
+
<resume_context>
|
|
1380
|
+
Pipeline state:
|
|
1381
|
+
- Completed stages: {list of completed stages with their results}
|
|
1382
|
+
- Current stage: {stage that triggered checkpoint}
|
|
1383
|
+
- Checkpoint response: {user's response or decision}
|
|
1384
|
+
- Artifacts produced so far: {list of files with paths}
|
|
1385
|
+
|
|
1386
|
+
Resume from: {exact step in pipeline to resume from}
|
|
1387
|
+
User decision: {what user chose at checkpoint}
|
|
1388
|
+
</resume_context>
|
|
1389
|
+
"
|
|
1390
|
+
)
|
|
1391
|
+
```
|
|
1392
|
+
|
|
1393
|
+
### Stale Chain Flag Protection
|
|
1394
|
+
|
|
1395
|
+
At orchestrator init, if `--auto` was NOT passed AND `auto_advance` config is false:
|
|
1396
|
+
```bash
|
|
1397
|
+
node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
|
|
1398
|
+
```
|
|
1399
|
+
|
|
1400
|
+
At pipeline completion (success or failure):
|
|
1401
|
+
```bash
|
|
1402
|
+
node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
|
|
1403
|
+
```
|
|
1404
|
+
|
|
1405
|
+
</auto_advance>
|