qaa-agent 1.6.2 → 1.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (78) hide show
  1. package/.mcp.json +8 -8
  2. package/CHANGELOG.md +93 -71
  3. package/CLAUDE.md +553 -553
  4. package/agents/qa-pipeline-orchestrator.md +1378 -1378
  5. package/agents/qaa-analyzer.md +539 -524
  6. package/agents/qaa-bug-detective.md +479 -446
  7. package/agents/qaa-codebase-mapper.md +935 -935
  8. package/agents/qaa-discovery.md +384 -0
  9. package/agents/qaa-e2e-runner.md +416 -415
  10. package/agents/qaa-executor.md +651 -651
  11. package/agents/qaa-planner.md +405 -390
  12. package/agents/qaa-project-researcher.md +319 -319
  13. package/agents/qaa-scanner.md +424 -424
  14. package/agents/qaa-testid-injector.md +643 -585
  15. package/agents/qaa-validator.md +490 -452
  16. package/bin/install.cjs +200 -198
  17. package/bin/lib/commands.cjs +709 -709
  18. package/bin/lib/config.cjs +307 -307
  19. package/bin/lib/core.cjs +497 -497
  20. package/bin/lib/frontmatter.cjs +299 -299
  21. package/bin/lib/init.cjs +989 -989
  22. package/bin/lib/milestone.cjs +241 -241
  23. package/bin/lib/model-profiles.cjs +60 -60
  24. package/bin/lib/phase.cjs +911 -911
  25. package/bin/lib/roadmap.cjs +306 -306
  26. package/bin/lib/state.cjs +748 -748
  27. package/bin/lib/template.cjs +222 -222
  28. package/bin/lib/verify.cjs +842 -842
  29. package/bin/qaa-tools.cjs +607 -607
  30. package/commands/qa-audit.md +119 -0
  31. package/commands/qa-create-test.md +288 -0
  32. package/commands/qa-fix.md +147 -0
  33. package/commands/qa-map.md +137 -0
  34. package/{.claude/commands → commands}/qa-pr.md +23 -23
  35. package/{.claude/commands → commands}/qa-start.md +22 -22
  36. package/{.claude/commands → commands}/qa-testid.md +19 -19
  37. package/docs/COMMANDS.md +341 -341
  38. package/docs/DEMO.md +182 -182
  39. package/docs/TESTING.md +156 -156
  40. package/package.json +6 -7
  41. package/{.claude/settings.json → settings.json} +1 -2
  42. package/templates/failure-classification.md +391 -391
  43. package/templates/gap-analysis.md +409 -409
  44. package/templates/pr-template.md +48 -48
  45. package/templates/qa-analysis.md +381 -381
  46. package/templates/qa-audit-report.md +465 -465
  47. package/templates/qa-repo-blueprint.md +636 -636
  48. package/templates/scan-manifest.md +312 -312
  49. package/templates/test-inventory.md +582 -582
  50. package/templates/testid-audit-report.md +354 -354
  51. package/templates/validation-report.md +243 -243
  52. package/workflows/qa-analyze.md +296 -296
  53. package/workflows/qa-from-ticket.md +536 -536
  54. package/workflows/qa-gap.md +309 -303
  55. package/workflows/qa-pr.md +389 -389
  56. package/workflows/qa-start.md +1192 -1168
  57. package/workflows/qa-testid.md +384 -356
  58. package/workflows/qa-validate.md +299 -295
  59. package/.claude/commands/create-test.md +0 -164
  60. package/.claude/commands/qa-audit.md +0 -37
  61. package/.claude/commands/qa-blueprint.md +0 -54
  62. package/.claude/commands/qa-fix.md +0 -36
  63. package/.claude/commands/qa-from-ticket.md +0 -24
  64. package/.claude/commands/qa-gap.md +0 -20
  65. package/.claude/commands/qa-map.md +0 -47
  66. package/.claude/commands/qa-pom.md +0 -36
  67. package/.claude/commands/qa-pyramid.md +0 -37
  68. package/.claude/commands/qa-report.md +0 -38
  69. package/.claude/commands/qa-research.md +0 -33
  70. package/.claude/commands/qa-validate.md +0 -42
  71. package/.claude/commands/update-test.md +0 -58
  72. package/.claude/skills/qa-learner/SKILL.md +0 -150
  73. /package/{.claude/skills → skills}/qa-bug-detective/SKILL.md +0 -0
  74. /package/{.claude/skills → skills}/qa-repo-analyzer/SKILL.md +0 -0
  75. /package/{.claude/skills → skills}/qa-self-validator/SKILL.md +0 -0
  76. /package/{.claude/skills → skills}/qa-template-engine/SKILL.md +0 -0
  77. /package/{.claude/skills → skills}/qa-testid-injector/SKILL.md +0 -0
  78. /package/{.claude/skills → skills}/qa-workflow-documenter/SKILL.md +0 -0
@@ -1,1168 +1,1192 @@
1
- <purpose>
2
-
3
- Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
4
-
5
- Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
6
-
7
- </purpose>
8
-
9
- <required_reading>
10
-
11
- Read these files BEFORE executing any pipeline stage. Do NOT skip.
12
-
13
- - **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
14
- - **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
15
-
16
- </required_reading>
17
-
18
- <process>
19
-
20
- <step name="initialize" priority="first">
21
-
22
- ## Step 1: Initialize Pipeline
23
-
24
- Parse `$ARGUMENTS` for flags:
25
-
26
- ```bash
27
- DEV_REPO=""
28
- QA_REPO=""
29
- IS_AUTO=false
30
-
31
- # Parse --dev-repo flag
32
- if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
33
- DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
34
- fi
35
-
36
- # Parse --qa-repo flag
37
- if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
38
- QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
39
- fi
40
-
41
- # Parse --auto flag
42
- if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
43
- IS_AUTO=true
44
- fi
45
- ```
46
-
47
- **If no --dev-repo provided**, use the current working directory:
48
- ```bash
49
- if [ -z "$DEV_REPO" ]; then
50
- DEV_REPO=$(pwd)
51
- fi
52
- ```
53
-
54
- **Attempt to call qaa-tools init** (handle missing tool gracefully):
55
- ```bash
56
- INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
57
- ```
58
-
59
- If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
60
- - Set `output_dir` to `.qa-output`
61
- - Set `date` to current date in `YYYY-MM-DD` format
62
- - Create output directory: `mkdir -p "$output_dir"`
63
-
64
- If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
65
-
66
- **Detect workflow option based on inputs:**
67
-
68
- - If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
69
- - If `QA_REPO` is provided: Assess QA repo maturity
70
- - Check for existing test files, configs, coverage reports in QA repo
71
- - Count test files, evaluate framework setup, check for CI config
72
- - Score 0-100 based on test count, framework config presence, CI setup, coverage data
73
- - Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
74
- - Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
75
-
76
- **Determine auto-advance mode:**
77
- ```bash
78
- # Check persistent config flag
79
- AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
80
- AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
81
-
82
- if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
83
- IS_AUTO=true
84
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
85
- fi
86
-
87
- # Safety: clear stale chain flag if NOT in auto mode
88
- if [ "$IS_AUTO" = "false" ]; then
89
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
90
- fi
91
- ```
92
-
93
- **Print initialization banner:**
94
- ```
95
- === QA Pipeline Orchestrator ===
96
- Option: {option} ({description})
97
- Dev Repo: {DEV_REPO}
98
- QA Repo: {QA_REPO or 'N/A'}
99
- Maturity Score: {maturity_score or 'N/A'}
100
- Auto-Advance: {IS_AUTO}
101
- Date: {date}
102
- ================================
103
- ```
104
-
105
- Where `{description}` is:
106
- - Option 1: "Dev-Only -- Full Pipeline"
107
- - Option 2: "Dev + Immature QA -- Gap-Fill"
108
- - Option 3: "Dev + Mature QA -- Surgical"
109
-
110
- </step>
111
-
112
- <step name="detect_framework">
113
-
114
- ## Step 2: Detect Framework
115
-
116
- Before scanning, detect the project's language and test framework to guide all downstream agents.
117
-
118
- **Read project config files:**
119
- ```bash
120
- # Check for Node.js / JavaScript / TypeScript
121
- [ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
122
-
123
- # Check for Python
124
- [ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
125
- [ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
126
- [ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
127
-
128
- # Check for .NET
129
- ls "${DEV_REPO}"/*.csproj 2>/dev/null
130
- ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
131
-
132
- # Check for Java
133
- [ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
134
- [ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
135
- ```
136
-
137
- **Detect test framework from config files:**
138
- ```bash
139
- # JavaScript/TypeScript ecosystem
140
- [ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
141
- [ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
142
- [ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
143
- [ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
144
-
145
- # Python ecosystem
146
- [ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
147
-
148
- # Check package.json devDependencies for test frameworks
149
- node -e "
150
- try {
151
- const pkg = require('${DEV_REPO}/package.json');
152
- const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
153
- const frameworks = [];
154
- if (deps.cypress) frameworks.push('cypress');
155
- if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
156
- if (deps.jest) frameworks.push('jest');
157
- if (deps.vitest) frameworks.push('vitest');
158
- if (deps.mocha) frameworks.push('mocha');
159
- console.log(frameworks.join(',') || 'none');
160
- } catch { console.log('no-package-json'); }
161
- " 2>/dev/null
162
- ```
163
-
164
- **Assess detection confidence:**
165
- - **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
166
- - **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
167
- - **LOW**: No test framework detected, or conflicting signals
168
-
169
- **If no test framework found:**
170
- - If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
171
- - If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
172
- - React/Next.js/Vue/Angular frontend -> Playwright
173
- - Node.js API -> Jest or Vitest (prefer Vitest if ESM)
174
- - Python -> Pytest
175
- - Log: "Auto-selected: {framework} (no existing test framework detected)"
176
-
177
- **If detection confidence is LOW:**
178
- - If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
179
- - If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
180
-
181
- Store detected framework, language, and confidence for all downstream agents.
182
-
183
- </step>
184
-
185
- <step name="scan">
186
-
187
- ## Step 3: Scan Repository
188
-
189
- **State update -- mark scan as running:**
190
- ```bash
191
- node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
192
- ```
193
-
194
- **Print stage banner:**
195
- ```
196
- +------------------------------------------+
197
- | STAGE 1: Scanner |
198
- | Status: Running... |
199
- +------------------------------------------+
200
- ```
201
-
202
- **Spawn scanner agent:**
203
-
204
- For **Option 1** (scan dev repo only):
205
- ```
206
- Agent(subagent_type="general-purpose",
207
- prompt="
208
- <objective>Scan repository and produce SCAN_MANIFEST.md</objective>
209
- <execution_context>@agents/qaa-scanner.md</execution_context>
210
- <files_to_read>
211
- - CLAUDE.md
212
- </files_to_read>
213
- <parameters>
214
- dev_repo_path: {DEV_REPO}
215
- qa_repo_path: null
216
- output_path: {output_dir}/SCAN_MANIFEST.md
217
- </parameters>
218
- "
219
- )
220
- ```
221
-
222
- For **Options 2 and 3** (scan both repos):
223
- ```
224
- Agent(subagent_type="general-purpose",
225
- prompt="
226
- <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
227
- <execution_context>@agents/qaa-scanner.md</execution_context>
228
- <files_to_read>
229
- - CLAUDE.md
230
- </files_to_read>
231
- <parameters>
232
- dev_repo_path: {DEV_REPO}
233
- qa_repo_path: {QA_REPO}
234
- output_path: {output_dir}/SCAN_MANIFEST.md
235
- </parameters>
236
- "
237
- )
238
- ```
239
-
240
- **Parse scanner return:**
241
-
242
- Expected return structure:
243
- ```
244
- SCANNER_COMPLETE:
245
- file_path: ".qa-output/SCAN_MANIFEST.md"
246
- decision: PROCEED | STOP
247
- has_frontend: true | false
248
- detection_confidence: HIGH | MEDIUM | LOW
249
- ```
250
-
251
- **Handle decision field:**
252
- - If `decision` is `STOP`:
253
- ```bash
254
- node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
255
- ```
256
- Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
257
-
258
- - If `decision` is `PROCEED`:
259
- ```bash
260
- node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
261
- ```
262
- Capture `has_frontend` for testid-injector conditional (Step 5).
263
- Capture `detection_confidence` for checkpoint handling.
264
-
265
- **Verify artifact exists before continuing:**
266
- ```bash
267
- [ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
268
- ```
269
-
270
- If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
271
-
272
- </step>
273
-
274
- <step name="analyze">
275
-
276
- ## Step 4: Analyze Repository
277
-
278
- **State update -- mark analyze as running:**
279
- ```bash
280
- node bin/qaa-tools.cjs state patch --"Analyze Status" running --"Status" "Analyzing repository" 2>/dev/null || true
281
- ```
282
-
283
- **Print stage banner:**
284
- ```
285
- +------------------------------------------+
286
- | STAGE 2: Analyzer |
287
- | Status: Running... |
288
- +------------------------------------------+
289
- ```
290
-
291
- **Determine analyzer mode based on option:**
292
- - Option 1: `mode = 'full'` (produces QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md)
293
- - Options 2 and 3: `mode = 'gap'` (produces GAP_ANALYSIS.md)
294
-
295
- **Spawn analyzer agent:**
296
- ```
297
- Agent(subagent_type="general-purpose",
298
- prompt="
299
- <objective>Analyze scanned repository and produce analysis artifacts</objective>
300
- <execution_context>@agents/qaa-analyzer.md</execution_context>
301
- <files_to_read>
302
- - {output_dir}/SCAN_MANIFEST.md
303
- - CLAUDE.md
304
- </files_to_read>
305
- <parameters>
306
- mode: {mode}
307
- workflow_option: {option}
308
- dev_repo_path: {DEV_REPO}
309
- qa_repo_path: {QA_REPO or null}
310
- output_path: {output_dir}/
311
- </parameters>
312
- "
313
- )
314
- ```
315
-
316
- **Parse analyzer return:**
317
-
318
- Expected return structure:
319
- ```
320
- ANALYZER_COMPLETE:
321
- files_produced: [...]
322
- total_test_count: N
323
- pyramid_breakdown: {unit: N, integration: N, api: N, e2e: N}
324
- risk_count: {high: N, medium: N, low: N}
325
- commit_hash: "..."
326
- ```
327
-
328
- Capture `files_produced`, `total_test_count`, `pyramid_breakdown` for downstream stages.
329
-
330
- **Handle analyzer checkpoint -- assumptions review:**
331
- - If `IS_AUTO` is true: Auto-approve all assumptions (SAFE checkpoint). Log: "Auto-approved: Analyzer assumptions". Continue pipeline.
332
- - If `IS_AUTO` is false: Present assumptions to user for review. Wait for confirmation or corrections. On user response, incorporate corrections and continue.
333
-
334
- **State update -- mark analyze as complete:**
335
- ```bash
336
- node bin/qaa-tools.cjs state patch --"Analyze Status" complete 2>/dev/null || true
337
- ```
338
-
339
- **Verify artifacts exist before continuing:**
340
-
341
- For Option 1:
342
- ```bash
343
- [ -f "${output_dir}/QA_ANALYSIS.md" ] && echo "OK" || echo "MISSING: QA_ANALYSIS.md"
344
- [ -f "${output_dir}/TEST_INVENTORY.md" ] && echo "OK" || echo "MISSING: TEST_INVENTORY.md"
345
- ```
346
-
347
- For Options 2/3:
348
- ```bash
349
- [ -f "${output_dir}/GAP_ANALYSIS.md" ] && echo "OK" || echo "MISSING: GAP_ANALYSIS.md"
350
- ```
351
-
352
- If required artifacts are missing, treat as stage failure. Set status to failed and STOP pipeline.
353
-
354
- Print: "Analysis complete. {total_test_count} test cases identified. Pyramid: unit={unit}, integration={integration}, api={api}, e2e={e2e}."
355
-
356
- </step>
357
-
358
- <step name="testid_inject">
359
-
360
- ## Step 5: TestID Injection (Conditional)
361
-
362
- **Condition:** Only execute if `has_frontend` is `true` from scanner return (Step 3).
363
-
364
- **If `has_frontend` is false:**
365
- Print: "Skipping TestID injection (no frontend detected)." Proceed directly to Step 6 (Plan).
366
-
367
- **If `has_frontend` is true:**
368
-
369
- **State update:**
370
- ```bash
371
- node bin/qaa-tools.cjs state patch --"Status" "Injecting test IDs into frontend components" 2>/dev/null || true
372
- ```
373
-
374
- **Print stage banner:**
375
- ```
376
- +------------------------------------------+
377
- | STAGE 3: TestID Injector |
378
- | Status: Running... |
379
- +------------------------------------------+
380
- ```
381
-
382
- **Spawn testid-injector agent:**
383
- ```
384
- Agent(subagent_type="general-purpose",
385
- prompt="
386
- <objective>Audit and inject data-testid attributes into frontend components</objective>
387
- <execution_context>@agents/qaa-testid-injector.md</execution_context>
388
- <files_to_read>
389
- - {output_dir}/SCAN_MANIFEST.md
390
- - CLAUDE.md
391
- </files_to_read>
392
- <parameters>
393
- dev_repo_path: {DEV_REPO}
394
- output_path: {output_dir}/TESTID_AUDIT_REPORT.md
395
- </parameters>
396
- "
397
- )
398
- ```
399
-
400
- **Parse return:**
401
-
402
- Check for `INJECTOR_COMPLETE` vs `INJECTOR_SKIPPED`:
403
-
404
- If `INJECTOR_COMPLETE`:
405
- ```
406
- INJECTOR_COMPLETE:
407
- report_path: "..."
408
- coverage_before: N%
409
- coverage_after: N%
410
- elements_injected: N
411
- components_modified: N
412
- ```
413
- Log: "TestID injection complete. Coverage: {coverage_before}% -> {coverage_after}%. {elements_injected} elements injected."
414
-
415
- If `INJECTOR_SKIPPED`:
416
- ```
417
- INJECTOR_SKIPPED:
418
- reason: "..."
419
- action: "..."
420
- ```
421
- Log the reason and continue pipeline.
422
-
423
- **Handle injector checkpoint -- audit review:**
424
- - If `IS_AUTO` is true: Auto-approve P0-only injection (SAFE checkpoint). Log: "Auto-approved: TestID injection (P0 elements only)". Continue pipeline.
425
- - If `IS_AUTO` is false: Present audit report to user. Wait for approval, element selection, or rejection. On user response, incorporate decisions and continue.
426
-
427
- **Verify artifact exists:**
428
- ```bash
429
- [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ] && echo "OK" || echo "MISSING: TESTID_AUDIT_REPORT.md"
430
- ```
431
-
432
- </step>
433
-
434
- <step name="plan">
435
-
436
- ## Step 6: Plan Test Generation
437
-
438
- **State update -- mark generation as running (planning is part of generate):**
439
- ```bash
440
- node bin/qaa-tools.cjs state patch --"Generate Status" running --"Status" "Planning test generation" 2>/dev/null || true
441
- ```
442
-
443
- **Print stage banner:**
444
- ```
445
- +------------------------------------------+
446
- | STAGE 4: Planner |
447
- | Status: Running... |
448
- +------------------------------------------+
449
- ```
450
-
451
- **Determine planner input based on option:**
452
- - Option 1: Input from `{output_dir}/TEST_INVENTORY.md` + `{output_dir}/QA_ANALYSIS.md`
453
- - Options 2 and 3: Input from `{output_dir}/GAP_ANALYSIS.md`
454
-
455
- **Spawn planner agent:**
456
- ```
457
- Agent(subagent_type="general-purpose",
458
- prompt="
459
- <objective>Create test generation plan with task breakdown and dependencies</objective>
460
- <execution_context>@agents/qaa-planner.md</execution_context>
461
- <files_to_read>
462
- - {input files based on option}
463
- - CLAUDE.md
464
- </files_to_read>
465
- <parameters>
466
- workflow_option: {option}
467
- output_path: {output_dir}/GENERATION_PLAN.md
468
- </parameters>
469
- "
470
- )
471
- ```
472
-
473
- **Parse planner return:**
474
-
475
- Expected return structure:
476
- ```
477
- PLANNER_COMPLETE:
478
- file_path: "..."
479
- total_tasks: N
480
- total_files: N
481
- feature_count: N
482
- dependency_depth: N
483
- test_case_count: N
484
- commit_hash: "..."
485
- ```
486
-
487
- Capture `total_tasks`, `total_files`, `feature_count` for executor stage and pipeline summary.
488
-
489
- **Verify artifact exists:**
490
- ```bash
491
- [ -f "${output_dir}/GENERATION_PLAN.md" ] && echo "OK" || echo "MISSING: GENERATION_PLAN.md"
492
- ```
493
-
494
- If GENERATION_PLAN.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
495
-
496
- Print: "Plan complete. {total_tasks} tasks, {total_files} files planned across {feature_count} features."
497
-
498
- </step>
499
-
500
- <step name="generate">
501
-
502
- ## Step 7: Generate Test Files
503
-
504
- State update continues from planning (already set to `running` in Step 6).
505
-
506
- **Print stage banner:**
507
- ```
508
- +------------------------------------------+
509
- | STAGE 5: Executor |
510
- | Generating {total_files} test files |
511
- | Status: Running... |
512
- +------------------------------------------+
513
- ```
514
-
515
- **Determine execution strategy:**
516
-
517
- Check if planner created multiple independent feature groups. If `feature_count > 1` AND parallelization is enabled (from init config):
518
-
519
- **Parallel execution** (when feature_count > 1 and parallelization enabled):
520
-
521
- For each independent feature group from the generation plan, spawn a separate executor agent:
522
- ```
523
- Agent(subagent_type="general-purpose",
524
- prompt="
525
- <objective>Generate test files for {feature} feature</objective>
526
- <execution_context>@agents/qaa-executor.md</execution_context>
527
- <files_to_read>
528
- - {output_dir}/GENERATION_PLAN.md
529
- - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
530
- - CLAUDE.md
531
- </files_to_read>
532
- <parameters>
533
- workflow_option: {option}
534
- feature_group: {feature}
535
- dev_repo_path: {DEV_REPO}
536
- qa_repo_path: {QA_REPO or null}
537
- output_path: {output_dir}/
538
- </parameters>
539
- "
540
- )
541
- ```
542
-
543
- Multiple Agent() calls can be issued simultaneously for independent feature groups. Each executor handles one feature group and commits its files independently.
544
-
545
- **Sequential execution** (when feature_count == 1 or parallelization disabled):
546
-
547
- Spawn a single executor agent covering all tasks:
548
- ```
549
- Agent(subagent_type="general-purpose",
550
- prompt="
551
- <objective>Generate all test files from generation plan</objective>
552
- <execution_context>@agents/qaa-executor.md</execution_context>
553
- <files_to_read>
554
- - {output_dir}/GENERATION_PLAN.md
555
- - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
556
- - CLAUDE.md
557
- </files_to_read>
558
- <parameters>
559
- workflow_option: {option}
560
- dev_repo_path: {DEV_REPO}
561
- qa_repo_path: {QA_REPO or null}
562
- output_path: {output_dir}/
563
- </parameters>
564
- "
565
- )
566
- ```
567
-
568
- **Option 3 specific -- skip existing tests:**
569
-
570
- For Option 3, pass `skip_existing_test_ids: true` to the executor so it checks existing test files by test ID before generating. If a test ID already exists in the QA repo, skip generating that test case:
571
- ```
572
- <parameters>
573
- workflow_option: 3
574
- skip_existing_test_ids: true
575
- dev_repo_path: {DEV_REPO}
576
- qa_repo_path: {QA_REPO}
577
- output_path: {output_dir}/
578
- </parameters>
579
- ```
580
-
581
- **Parse executor return:**
582
-
583
- Expected return structure:
584
- ```
585
- EXECUTOR_COMPLETE:
586
- files_created: [{path, type}, ...]
587
- total_files: N
588
- commit_count: N
589
- features_covered: [...]
590
- test_case_count: N
591
- ```
592
-
593
- Capture `files_created`, `total_files`, `commit_count` for validation stage and pipeline summary.
594
-
595
- **State update -- mark generate as complete:**
596
- ```bash
597
- node bin/qaa-tools.cjs state patch --"Generate Status" complete --"Status" "Test generation complete" 2>/dev/null || true
598
- ```
599
-
600
- Print: "Generation complete. {total_files} files created across {features_covered_count} features. {commit_count} commits."
601
-
602
- </step>
603
-
604
- <step name="validate">
605
-
606
- ## Step 8: Validate Generated Tests
607
-
608
- **State update -- mark validate as running:**
609
- ```bash
610
- node bin/qaa-tools.cjs state patch --"Validate Status" running --"Status" "Validating generated tests" 2>/dev/null || true
611
- ```
612
-
613
- **Print stage banner:**
614
- ```
615
- +------------------------------------------+
616
- | STAGE 6: Validator |
617
- | Validating {total_files} test files |
618
- | Status: Running... |
619
- +------------------------------------------+
620
- ```
621
-
622
- **Spawn validator agent:**
623
- ```
624
- Agent(subagent_type="general-purpose",
625
- prompt="
626
- <objective>Run 4-layer validation on all generated test files</objective>
627
- <execution_context>@agents/qaa-validator.md</execution_context>
628
- <files_to_read>
629
- - {list all generated test files from executor return -- files_created paths}
630
- - {output_dir}/GENERATION_PLAN.md
631
- - CLAUDE.md
632
- </files_to_read>
633
- <parameters>
634
- mode: validation
635
- max_fix_loops: 3
636
- output_path: {output_dir}/VALIDATION_REPORT.md
637
- </parameters>
638
- "
639
- )
640
- ```
641
-
642
- **4-layer validation:**
643
- 1. **Syntax** -- File parses without errors
644
- 2. **Structure** -- Follows POM rules, naming conventions, locator hierarchy
645
- 3. **Dependencies** -- Imports resolve, fixtures exist, configs present
646
- 4. **Logic** -- Assertions are concrete, test IDs are unique, no assertions in page objects
647
-
648
- **Fix loop:** The validator automatically attempts to fix issues it finds. Maximum 3 fix loop iterations. After each fix attempt, re-validate.
649
-
650
- **Parse validator return:**
651
-
652
- Expected return structure:
653
- ```
654
- VALIDATOR_COMPLETE:
655
- report_path: "..."
656
- overall_status: PASS | PASS_WITH_WARNINGS | FAIL
657
- confidence: HIGH | MEDIUM | LOW
658
- layers_summary: {syntax: PASS|FAIL, structure: PASS|FAIL, dependencies: PASS|FAIL, logic: PASS|FAIL}
659
- fix_loops_used: N
660
- issues_found: N
661
- issues_fixed: N
662
- unresolved_count: N
663
- ```
664
-
665
- **RISKY CHECKPOINT -- Validator escalation:**
666
-
667
- If `unresolved_count > 0` after max fix loops (3):
668
- - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
669
- - Present unresolved issues to user with full details from VALIDATION_REPORT.md
670
- - Wait for user decision:
671
- - `"approve-with-warnings"`: Accept the validation with warnings. Set Validate Status to complete. Continue to deliver.
672
- - `"abort"`: Set Validate Status to failed. STOP PIPELINE ENTIRELY.
673
- - Manual guidance: User provides specific fix instructions. Spawn fresh continuation agent to apply fixes and re-validate.
674
-
675
- If `overall_status` is `PASS` or `PASS_WITH_WARNINGS` (and unresolved_count is 0):
676
- ```bash
677
- node bin/qaa-tools.cjs state patch --"Validate Status" complete --"Status" "Validation passed" 2>/dev/null || true
678
- ```
679
-
680
- **Verify artifact exists:**
681
- ```bash
682
- [ -f "${output_dir}/VALIDATION_REPORT.md" ] && echo "OK" || echo "MISSING: VALIDATION_REPORT.md"
683
- ```
684
-
685
- Print: "Validation complete. Status: {overall_status}. Confidence: {confidence}. {issues_found} issues found, {issues_fixed} fixed, {unresolved_count} unresolved."
686
-
687
- </step>
688
-
689
- <step name="bug_detective">
690
-
691
- ## Step 9: Bug Detective (Conditional)
692
-
693
- **Condition:** Only execute if test failures were detected during validation. Check:
694
- - `overall_status === 'FAIL'` in validator return, OR
695
- - Generated tests have runtime failures that need classification
696
-
697
- **If no failures to classify:**
698
- Print: "Skipping Bug Detective (no test failures detected)." Proceed directly to Step 10 (Deliver).
699
-
700
- **If failures need classification:**
701
-
702
- **State update:**
703
- ```bash
704
- node bin/qaa-tools.cjs state patch --"Status" "Classifying test failures" 2>/dev/null || true
705
- ```
706
-
707
- **Print stage banner:**
708
- ```
709
- +------------------------------------------+
710
- | STAGE 7: Bug Detective |
711
- | Status: Running... |
712
- +------------------------------------------+
713
- ```
714
-
715
- **Spawn bug-detective agent:**
716
- ```
717
- Agent(subagent_type="general-purpose",
718
- prompt="
719
- <objective>Classify test failures and attempt auto-fixes for test errors</objective>
720
- <execution_context>@agents/qaa-bug-detective.md</execution_context>
721
- <files_to_read>
722
- - {test execution results -- from validator or direct test run}
723
- - {failing test source files -- paths from executor return}
724
- - CLAUDE.md
725
- </files_to_read>
726
- <parameters>
727
- output_path: {output_dir}/FAILURE_CLASSIFICATION_REPORT.md
728
- </parameters>
729
- "
730
- )
731
- ```
732
-
733
- **Parse bug-detective return:**
734
-
735
- Expected return structure:
736
- ```
737
- DETECTIVE_COMPLETE:
738
- report_path: "..."
739
- total_failures: N
740
- classification_breakdown: {app_bug: N, test_error: N, env_issue: N, inconclusive: N}
741
- auto_fixes_applied: N
742
- auto_fixes_verified: N
743
- commit_hash: "..."
744
- ```
745
-
746
- **RISKY CHECKPOINT -- Application bugs detected:**
747
-
748
- If `classification_breakdown.app_bug > 0`:
749
- - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
750
- - Present APPLICATION BUG classifications to user with full evidence from FAILURE_CLASSIFICATION_REPORT.md
751
- - These are genuine bugs in the application code discovered during test execution
752
- - The bug detective never touches application code -- it only reports
753
- - User must review and decide how to proceed:
754
- - Acknowledge bugs and continue pipeline (bugs will be in the PR description for developer attention)
755
- - Abort pipeline to fix bugs first
756
-
757
- **Verify artifact exists:**
758
- ```bash
759
- [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "OK" || echo "MISSING: FAILURE_CLASSIFICATION_REPORT.md"
760
- ```
761
-
762
- Print: "Bug Detective complete. {total_failures} failures classified: {app_bug} APP BUG, {test_error} TEST ERROR, {env_issue} ENV ISSUE, {inconclusive} INCONCLUSIVE. {auto_fixes_applied} auto-fixes applied."
763
-
764
- </step>
765
-
766
- <step name="deliver">
767
-
768
- ## Step 10: Deliver
769
-
770
- **State update -- mark deliver as running:**
771
- ```bash
772
- node bin/qaa-tools.cjs state patch --"Deliver Status" running --"Status" "Preparing delivery" 2>/dev/null || true
773
- ```
774
-
775
- **Print stage banner:**
776
- ```
777
- +------------------------------------------+
778
- | STAGE 8: Deliver |
779
- | Status: Running... |
780
- +------------------------------------------+
781
- ```
782
-
783
- ### Sub-step 1: Pre-flight checks
784
-
785
- **Check for git remote:**
786
- ```bash
787
- REMOTE_URL=$(git remote get-url origin 2>/dev/null || echo "")
788
- ```
789
-
790
- If `REMOTE_URL` is empty:
791
- - Print: "No git remote found. Artifacts committed locally but PR creation skipped."
792
- - Set `LOCAL_ONLY=true`
793
-
794
- **Check for gh CLI authentication:**
795
- ```bash
796
- gh auth status 2>/dev/null
797
- ```
798
-
799
- If `gh auth status` fails:
800
- - Print: "gh CLI not authenticated. Run 'gh auth login' first. Artifacts committed locally."
801
- - Set `LOCAL_ONLY=true`
802
-
803
- If both checks pass, set `LOCAL_ONLY=false`.
804
-
805
- ### Sub-step 2: Derive project name
806
-
807
- ```bash
808
- # Read from package.json
809
- PROJECT_NAME=$(node -e "try { const p = require('${DEV_REPO}/package.json'); console.log(p.name || ''); } catch { console.log(''); }" 2>/dev/null)
810
-
811
- # Fallback to directory basename
812
- if [ -z "$PROJECT_NAME" ]; then
813
- PROJECT_NAME=$(basename "${DEV_REPO}")
814
- fi
815
-
816
- # Sanitize for branch naming
817
- PROJECT_NAME=$(echo "$PROJECT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
818
- ```
819
-
820
- ### Sub-step 3: Detect default branch
821
-
822
- ```bash
823
- DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name' 2>/dev/null || echo "main")
824
- ```
825
-
826
- ### Sub-step 4: Create feature branch
827
-
828
- ```bash
829
- BRANCH="qa/auto-${PROJECT_NAME}-${date}"
830
-
831
- # Handle branch name collision
832
- if git rev-parse --verify "$BRANCH" 2>/dev/null || git rev-parse --verify "origin/$BRANCH" 2>/dev/null; then
833
- SUFFIX=2
834
- while git rev-parse --verify "${BRANCH}-${SUFFIX}" 2>/dev/null || git rev-parse --verify "origin/${BRANCH}-${SUFFIX}" 2>/dev/null; do
835
- SUFFIX=$((SUFFIX + 1))
836
- done
837
- BRANCH="${BRANCH}-${SUFFIX}"
838
- fi
839
-
840
- git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
841
- ```
842
-
843
- ### Sub-step 5: Per-stage atomic commits
844
-
845
- For each pipeline stage that produced artifacts, commit using `qaa-tools.cjs commit`. Check file existence before each commit.
846
-
847
- **Scanner:**
848
- ```bash
849
- if [ -f "${output_dir}/SCAN_MANIFEST.md" ]; then
850
- node bin/qaa-tools.cjs commit "qa(scanner): produce SCAN_MANIFEST.md for ${PROJECT_NAME}" --files ${output_dir}/SCAN_MANIFEST.md
851
- fi
852
- ```
853
-
854
- **Analyzer (Option 1):**
855
- ```bash
856
- if [ -f "${output_dir}/QA_ANALYSIS.md" ]; then
857
- ANALYZER_FILES="${output_dir}/QA_ANALYSIS.md ${output_dir}/TEST_INVENTORY.md"
858
- [ -f "${output_dir}/QA_REPO_BLUEPRINT.md" ] && ANALYZER_FILES="${ANALYZER_FILES} ${output_dir}/QA_REPO_BLUEPRINT.md"
859
- node bin/qaa-tools.cjs commit "qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md" --files ${ANALYZER_FILES}
860
- fi
861
- ```
862
-
863
- **Analyzer (Options 2/3):**
864
- ```bash
865
- if [ -f "${output_dir}/GAP_ANALYSIS.md" ]; then
866
- node bin/qaa-tools.cjs commit "qa(analyzer): produce GAP_ANALYSIS.md" --files ${output_dir}/GAP_ANALYSIS.md
867
- fi
868
- ```
869
-
870
- **TestID Injector (if ran):**
871
- ```bash
872
- if [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ]; then
873
- node bin/qaa-tools.cjs commit "qa(testid-injector): inject ${elements_injected} data-testid attributes across ${components_modified} components" --files ${output_dir}/TESTID_AUDIT_REPORT.md ${modified_source_files}
874
- fi
875
- ```
876
-
877
- **Executor:**
878
- ```bash
879
- if [ -n "${generated_file_paths}" ]; then
880
- node bin/qaa-tools.cjs commit "qa(executor): generate ${total_files} test files with POMs and fixtures" --files ${generated_file_paths}
881
- fi
882
- ```
883
-
884
- **Planner:**
885
- ```bash
886
- if [ -f "${output_dir}/GENERATION_PLAN.md" ]; then
887
- node bin/qaa-tools.cjs commit "qa(planner): produce GENERATION_PLAN.md" --files ${output_dir}/GENERATION_PLAN.md
888
- fi
889
- ```
890
-
891
- **Validator:**
892
- ```bash
893
- if [ -f "${output_dir}/VALIDATION_REPORT.md" ]; then
894
- node bin/qaa-tools.cjs commit "qa(validator): validate generated tests - ${overall_status} with ${confidence} confidence" --files ${output_dir}/VALIDATION_REPORT.md
895
- fi
896
- ```
897
-
898
- **Bug Detective (if ran):**
899
- ```bash
900
- if [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ]; then
901
- node bin/qaa-tools.cjs commit "qa(bug-detective): classify ${total_failures} failures - ${classification_summary}" --files ${output_dir}/FAILURE_CLASSIFICATION_REPORT.md
902
- fi
903
- ```
904
-
905
- ### Sub-step 6: Push branch
906
-
907
- If `LOCAL_ONLY` is true, skip this sub-step.
908
-
909
- ```bash
910
- git push -u origin "$BRANCH"
911
- ```
912
-
913
- If push fails:
914
- - Print: "Push failed: {error_message}. Artifacts committed locally on branch ${BRANCH}."
915
- - Set `LOCAL_ONLY=true`
916
-
917
- ### Sub-step 7: Build PR body
918
-
919
- If `LOCAL_ONLY` is true, skip this sub-step.
920
-
921
- Read the PR template:
922
- ```bash
923
- PR_BODY=$(cat templates/pr-template.md)
924
- ```
925
-
926
- Replace all `{placeholder}` tokens with actual values collected during pipeline execution:
927
- - `{architecture_type}` -- from QA_ANALYSIS.md or SCAN_MANIFEST.md
928
- - `{framework}` -- detected test framework
929
- - `{risk_summary}` -- risk assessment counts (e.g., "3 HIGH, 5 MEDIUM, 2 LOW")
930
- - `{unit_count}` -- from pyramid_breakdown.unit
931
- - `{integration_count}` -- from pyramid_breakdown.integration
932
- - `{api_count}` -- from pyramid_breakdown.api
933
- - `{e2e_count}` -- from pyramid_breakdown.e2e
934
- - `{total_count}` -- from total_test_count
935
- - `{modules_covered}` -- count of modules with tests
936
- - `{coverage_estimate}` -- estimated coverage percentage
937
- - `{validation_result}` -- PASS, PASS_WITH_WARNINGS, or FAIL
938
- - `{confidence}` -- HIGH, MEDIUM, or LOW
939
- - `{fix_loops_used}` -- number 0-3
940
- - `{issues_found}` -- total issues found during validation
941
- - `{issues_fixed}` -- total issues auto-fixed
942
- - `{file_list}` -- if total files <= 50, list each file; if > 50, use summary
943
-
944
- ### Sub-step 8: Create draft PR
945
-
946
- If `LOCAL_ONLY` is true, skip this sub-step.
947
-
948
- ```bash
949
- PR_URL=$(gh pr create \
950
- --draft \
951
- --title "qa: automated test suite for ${PROJECT_NAME}" \
952
- --body "${PR_BODY}" \
953
- --label "qa-automation" \
954
- --label "auto-generated" \
955
- --assignee "@me" 2>&1)
956
- ```
957
-
958
- Do NOT pass `--base` flag. Let gh auto-detect the default branch.
959
-
960
- On failure:
961
- - Print: "PR creation failed: ${PR_URL}. Artifacts remain on branch ${BRANCH}."
962
- - Do NOT stop the pipeline -- artifacts are committed and pushed.
963
-
964
- ### Sub-step 9: Print result
965
-
966
- If PR was created successfully:
967
- ```
968
- PR created: ${PR_URL}
969
- ```
970
-
971
- If `LOCAL_ONLY` is true:
972
- ```
973
- PR: not created (local-only mode). Artifacts committed on branch: ${BRANCH}
974
- ```
975
-
976
- **State update -- mark deliver as complete:**
977
- ```bash
978
- node bin/qaa-tools.cjs state patch --"Deliver Status" complete --"Status" "Pipeline complete" 2>/dev/null || true
979
- ```
980
-
981
- **Clear auto-chain flag at pipeline completion:**
982
- ```bash
983
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
984
- ```
985
-
986
- ### Print pipeline summary banner:
987
-
988
- ```
989
- ======================================================
990
- QA PIPELINE COMPLETE
991
- ======================================================
992
-
993
- Option: {option} ({option_description})
994
- Repository: {DEV_REPO}
995
- QA Repo: {QA_REPO or 'N/A'}
996
- Maturity Score: {maturity_score or 'N/A'}
997
-
998
- Stages Completed:
999
- [{check}] Scan -- {scan_result}
1000
- [{check}] Analyze -- {analyze_result} ({test_count} test cases)
1001
- [{check}] TestID Inject -- {inject_result or 'skipped'}
1002
- [{check}] Plan -- {plan_result} ({file_count} files planned)
1003
- [{check}] Generate -- {generate_result} ({files_created} files created)
1004
- [{check}] Validate -- {validate_result} ({confidence} confidence)
1005
- [{check}] Bug Detective -- {detective_result or 'skipped'}
1006
- [{check}] Deliver -- {deliver_result}
1007
-
1008
- PR: {pr_url or 'not created (local-only)'}
1009
-
1010
- Artifacts:
1011
- {list all produced .md files in output_dir}
1012
-
1013
- Total Time: {total_duration}
1014
- ======================================================
1015
- ```
1016
-
1017
- Where: `[x]` = completed, `[ ]` = skipped, `[!]` = failed.
1018
-
1019
- </step>
1020
-
1021
- </process>
1022
-
1023
- <error_handling>
1024
-
1025
- ## Error Handling
1026
-
1027
- ### Stage Failure Protocol
1028
-
1029
- When any agent returns a failure or error:
1030
-
1031
- 1. **Set stage status to failed:**
1032
- ```bash
1033
- node bin/qaa-tools.cjs state patch --"{Stage} Status" failed --"Status" "Pipeline stopped: {Stage} failed - {reason}" 2>/dev/null || true
1034
- ```
1035
-
1036
- 2. **Print failure banner:**
1037
- ```
1038
- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1039
- ! PIPELINE STOPPED !
1040
- ! Stage: {stage_name} !
1041
- ! Reason: {failure_reason} !
1042
- ! !
1043
- ! Completed: {completed_stages} !
1044
- ! Artifacts: {artifacts_so_far} !
1045
- ! !
1046
- ! Action required: Review and re-run !
1047
- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1048
- ```
1049
-
1050
- 3. **DO NOT continue to next stage.** The pipeline stops entirely at the failed stage.
1051
-
1052
- 4. **DO NOT create partial PR.** No branch, no commit, no PR with incomplete results.
1053
-
1054
- 5. **Preserve all artifacts produced so far.** They remain on disk in `{output_dir}/` for debugging.
1055
-
1056
- ### Artifact Verification
1057
-
1058
- After EVERY agent spawn, before advancing to next stage, verify the expected output artifact exists on disk:
1059
- ```bash
1060
- [ -f "{expected_artifact_path}" ] && echo "OK" || echo "MISSING"
1061
- ```
1062
-
1063
- If artifacts are missing, treat as stage failure and STOP pipeline.
1064
-
1065
- ### qaa-tools.cjs Graceful Fallback
1066
-
1067
- All `node bin/qaa-tools.cjs` calls use `2>/dev/null || true` to handle cases where the tool is not installed or not found. The pipeline must not break due to missing state management tooling -- it logs a warning and continues.
1068
-
1069
- </error_handling>
1070
-
1071
- <auto_advance>
1072
-
1073
- ## Auto-Advance Mode
1074
-
1075
- Auto-advance is enabled when ANY of these is true:
1076
- - `--auto` flag passed to the `/qa-start` invocation
1077
- - `config.json` has `workflow.auto_advance = true` (persistent user preference)
1078
- - `workflow._auto_chain_active = true` in config (ephemeral chain flag from ongoing auto run)
1079
-
1080
- ### Behavior in Auto Mode
1081
-
1082
- **SAFE checkpoints are auto-approved.** The pipeline continues without pausing. A log message records the auto-approval:
1083
- ```
1084
- Auto-approved: {checkpoint_description}
1085
- ```
1086
-
1087
- **RISKY checkpoints ALWAYS pause.** Even in auto mode, the pipeline stops and presents the checkpoint to the user.
1088
-
1089
- ### Safe vs Risky Checkpoint Classification
1090
-
1091
- **SAFE (auto-approve in auto mode):**
1092
-
1093
- | Checkpoint | Agent | Auto-Action |
1094
- |------------|-------|-------------|
1095
- | Framework detection uncertain (LOW confidence) | Scanner | Approve with most likely framework |
1096
- | Analyzer assumptions review | Analyzer | Approve all assumptions |
1097
- | TestID audit review | TestID Injector | Approve P0-only injection |
1098
-
1099
- **RISKY (ALWAYS pause, even in auto mode):**
1100
-
1101
- | Checkpoint | Agent | User Action Required |
1102
- |------------|-------|---------------------|
1103
- | Validator escalation (unresolved issues after 3 fix loops) | Validator | approve-with-warnings, abort, or fix guidance |
1104
- | APPLICATION BUG classification | Bug Detective | Review bugs, continue or fix first |
1105
- | Any checkpoint with "unresolved" or "failed" blocking text | Any | Review specific blocking issue |
1106
-
1107
- ### Checkpoint Handling Flow
1108
-
1109
- ```
1110
- On agent return with checkpoint data:
1111
- 1. Extract checkpoint blocking field content
1112
- 2. Classify as SAFE or RISKY:
1113
- - "framework detection" -> SAFE
1114
- - "assumptions" -> SAFE
1115
- - "audit" or "data-testid" -> SAFE
1116
- - "unresolved" -> RISKY
1117
- - "failed" -> RISKY
1118
- - "APPLICATION BUG" -> RISKY
1119
- - Default (no pattern match) -> RISKY (conservative)
1120
- 3. If IS_AUTO and SAFE:
1121
- - Auto-approve with default action
1122
- - Log the auto-approval
1123
- - Continue pipeline
1124
- 4. If IS_AUTO and RISKY:
1125
- - PAUSE pipeline
1126
- - Print checkpoint details with full context
1127
- - Wait for user input
1128
- 5. If NOT auto (manual mode):
1129
- - PAUSE pipeline
1130
- - Print checkpoint details
1131
- - Wait for user input
1132
- ```
1133
-
1134
- ### Resume After Checkpoint
1135
-
1136
- When resuming after a checkpoint, spawn a FRESH agent with explicit state:
1137
- ```
1138
- Agent(subagent_type="general-purpose",
1139
- prompt="
1140
- <objective>Continue QA pipeline from {stage} stage</objective>
1141
- <execution_context>@agents/qa-pipeline-orchestrator.md</execution_context>
1142
- <resume_context>
1143
- Pipeline state:
1144
- - Completed stages: {list of completed stages with their results}
1145
- - Current stage: {stage that triggered checkpoint}
1146
- - Checkpoint response: {user's response or decision}
1147
- - Artifacts produced so far: {list of files with paths}
1148
-
1149
- Resume from: {exact step in pipeline to resume from}
1150
- User decision: {what user chose at checkpoint}
1151
- </resume_context>
1152
- "
1153
- )
1154
- ```
1155
-
1156
- ### Stale Chain Flag Protection
1157
-
1158
- At orchestrator init, if `--auto` was NOT passed AND `auto_advance` config is false:
1159
- ```bash
1160
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1161
- ```
1162
-
1163
- At pipeline completion (success or failure):
1164
- ```bash
1165
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1166
- ```
1167
-
1168
- </auto_advance>
1
+ <purpose>
2
+
3
+ Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
4
+
5
+ Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
6
+
7
+ </purpose>
8
+
9
+ <required_reading>
10
+
11
+ Read these files BEFORE executing any pipeline stage. Do NOT skip.
12
+
13
+ - **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
14
+ - **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
15
+
16
+ </required_reading>
17
+
18
+ <process>
19
+
20
+ <step name="initialize" priority="first">
21
+
22
+ ## Step 1: Initialize Pipeline
23
+
24
+ Parse `$ARGUMENTS` for flags:
25
+
26
+ ```bash
27
+ DEV_REPO=""
28
+ QA_REPO=""
29
+ IS_AUTO=false
30
+
31
+ # Parse --dev-repo flag
32
+ if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
33
+ DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
34
+ fi
35
+
36
+ # Parse --qa-repo flag
37
+ if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
38
+ QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
39
+ fi
40
+
41
+ # Parse --auto flag
42
+ if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
43
+ IS_AUTO=true
44
+ fi
45
+ ```
46
+
47
+ **If no --dev-repo provided**, use the current working directory:
48
+ ```bash
49
+ if [ -z "$DEV_REPO" ]; then
50
+ DEV_REPO=$(pwd)
51
+ fi
52
+ ```
53
+
54
+ **Attempt to call qaa-tools init** (handle missing tool gracefully):
55
+ ```bash
56
+ INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
57
+ ```
58
+
59
+ If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
60
+ - Set `output_dir` to `.qa-output`
61
+ - Set `date` to current date in `YYYY-MM-DD` format
62
+ - Create output directory: `mkdir -p "$output_dir"`
63
+
64
+ If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
65
+
66
+ **Detect workflow option based on inputs:**
67
+
68
+ - If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
69
+ - If `QA_REPO` is provided: Assess QA repo maturity
70
+ - Check for existing test files, configs, coverage reports in QA repo
71
+ - Count test files, evaluate framework setup, check for CI config
72
+ - Score 0-100 based on test count, framework config presence, CI setup, coverage data
73
+ - Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
74
+ - Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
75
+
76
+ **Determine auto-advance mode:**
77
+ ```bash
78
+ # Check persistent config flag
79
+ AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
80
+ AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
81
+
82
+ if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
83
+ IS_AUTO=true
84
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
85
+ fi
86
+
87
+ # Safety: clear stale chain flag if NOT in auto mode
88
+ if [ "$IS_AUTO" = "false" ]; then
89
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
90
+ fi
91
+ ```
92
+
93
+ **Print initialization banner:**
94
+ ```
95
+ === QA Pipeline Orchestrator ===
96
+ Option: {option} ({description})
97
+ Dev Repo: {DEV_REPO}
98
+ QA Repo: {QA_REPO or 'N/A'}
99
+ Maturity Score: {maturity_score or 'N/A'}
100
+ Auto-Advance: {IS_AUTO}
101
+ Date: {date}
102
+ ================================
103
+ ```
104
+
105
+ Where `{description}` is:
106
+ - Option 1: "Dev-Only -- Full Pipeline"
107
+ - Option 2: "Dev + Immature QA -- Gap-Fill"
108
+ - Option 3: "Dev + Mature QA -- Surgical"
109
+
110
+ </step>
111
+
112
+ <step name="detect_framework">
113
+
114
+ ## Step 2: Detect Framework
115
+
116
+ Before scanning, detect the project's language and test framework to guide all downstream agents.
117
+
118
+ **Read project config files:**
119
+ ```bash
120
+ # Check for Node.js / JavaScript / TypeScript
121
+ [ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
122
+
123
+ # Check for Python
124
+ [ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
125
+ [ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
126
+ [ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
127
+
128
+ # Check for .NET
129
+ ls "${DEV_REPO}"/*.csproj 2>/dev/null
130
+ ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
131
+
132
+ # Check for Java
133
+ [ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
134
+ [ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
135
+ ```
136
+
137
+ **Detect test framework from config files:**
138
+ ```bash
139
+ # JavaScript/TypeScript ecosystem
140
+ [ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
141
+ [ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
142
+ [ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
143
+ [ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
144
+
145
+ # Python ecosystem
146
+ [ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
147
+
148
+ # Check package.json devDependencies for test frameworks
149
+ node -e "
150
+ try {
151
+ const pkg = require('${DEV_REPO}/package.json');
152
+ const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
153
+ const frameworks = [];
154
+ if (deps.cypress) frameworks.push('cypress');
155
+ if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
156
+ if (deps.jest) frameworks.push('jest');
157
+ if (deps.vitest) frameworks.push('vitest');
158
+ if (deps.mocha) frameworks.push('mocha');
159
+ console.log(frameworks.join(',') || 'none');
160
+ } catch { console.log('no-package-json'); }
161
+ " 2>/dev/null
162
+ ```
163
+
164
+ **Assess detection confidence:**
165
+ - **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
166
+ - **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
167
+ - **LOW**: No test framework detected, or conflicting signals
168
+
169
+ **If no test framework found:**
170
+ - If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
171
+ - If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
172
+ - React/Next.js/Vue/Angular frontend -> Playwright
173
+ - Node.js API -> Jest or Vitest (prefer Vitest if ESM)
174
+ - Python -> Pytest
175
+ - Log: "Auto-selected: {framework} (no existing test framework detected)"
176
+
177
+ **If detection confidence is LOW:**
178
+ - If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
179
+ - If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
180
+
181
+ Store detected framework, language, and confidence for all downstream agents.
182
+
183
+ </step>
184
+
185
+ <step name="scan">
186
+
187
+ ## Step 3: Scan Repository
188
+
189
+ **State update -- mark scan as running:**
190
+ ```bash
191
+ node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
192
+ ```
193
+
194
+ **Print stage banner:**
195
+ ```
196
+ +------------------------------------------+
197
+ | STAGE 1: Scanner |
198
+ | Status: Running... |
199
+ +------------------------------------------+
200
+ ```
201
+
202
+ **Spawn scanner agent:**
203
+
204
+ For **Option 1** (scan dev repo only):
205
+ ```
206
+ Agent(subagent_type="general-purpose",
207
+ prompt="
208
+ <objective>Scan repository and produce SCAN_MANIFEST.md</objective>
209
+ <execution_context>@agents/qaa-scanner.md</execution_context>
210
+ <files_to_read>
211
+ - CLAUDE.md
212
+ </files_to_read>
213
+ <parameters>
214
+ dev_repo_path: {DEV_REPO}
215
+ qa_repo_path: null
216
+ output_path: {output_dir}/SCAN_MANIFEST.md
217
+ </parameters>
218
+ "
219
+ )
220
+ ```
221
+
222
+ For **Options 2 and 3** (scan both repos):
223
+ ```
224
+ Agent(subagent_type="general-purpose",
225
+ prompt="
226
+ <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
227
+ <execution_context>@agents/qaa-scanner.md</execution_context>
228
+ <files_to_read>
229
+ - CLAUDE.md
230
+ </files_to_read>
231
+ <parameters>
232
+ dev_repo_path: {DEV_REPO}
233
+ qa_repo_path: {QA_REPO}
234
+ output_path: {output_dir}/SCAN_MANIFEST.md
235
+ </parameters>
236
+ "
237
+ )
238
+ ```
239
+
240
+ **Parse scanner return:**
241
+
242
+ Expected return structure:
243
+ ```
244
+ SCANNER_COMPLETE:
245
+ file_path: ".qa-output/SCAN_MANIFEST.md"
246
+ decision: PROCEED | STOP
247
+ has_frontend: true | false
248
+ detection_confidence: HIGH | MEDIUM | LOW
249
+ ```
250
+
251
+ **Handle decision field:**
252
+ - If `decision` is `STOP`:
253
+ ```bash
254
+ node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
255
+ ```
256
+ Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
257
+
258
+ - If `decision` is `PROCEED`:
259
+ ```bash
260
+ node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
261
+ ```
262
+ Capture `has_frontend` for testid-injector conditional (Step 5).
263
+ Capture `detection_confidence` for checkpoint handling.
264
+
265
+ **Verify artifact exists before continuing:**
266
+ ```bash
267
+ [ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
268
+ ```
269
+
270
+ If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
271
+
272
+ </step>
273
+
274
+ <step name="analyze">
275
+
276
+ ## Step 4: Analyze Repository
277
+
278
+ **State update -- mark analyze as running:**
279
+ ```bash
280
+ node bin/qaa-tools.cjs state patch --"Analyze Status" running --"Status" "Analyzing repository" 2>/dev/null || true
281
+ ```
282
+
283
+ **Print stage banner:**
284
+ ```
285
+ +------------------------------------------+
286
+ | STAGE 2: Analyzer |
287
+ | Status: Running... |
288
+ +------------------------------------------+
289
+ ```
290
+
291
+ **Determine analyzer mode based on option:**
292
+ - Option 1: `mode = 'full'` (produces QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md)
293
+ - Options 2 and 3: `mode = 'gap'` (produces GAP_ANALYSIS.md)
294
+
295
+ **Spawn analyzer agent:**
296
+ ```
297
+ Agent(subagent_type="general-purpose",
298
+ prompt="
299
+ <objective>Analyze scanned repository and produce analysis artifacts</objective>
300
+ <execution_context>@agents/qaa-analyzer.md</execution_context>
301
+ <files_to_read>
302
+ - {output_dir}/SCAN_MANIFEST.md
303
+ - CLAUDE.md
304
+ </files_to_read>
305
+ <parameters>
306
+ mode: {mode}
307
+ workflow_option: {option}
308
+ dev_repo_path: {DEV_REPO}
309
+ qa_repo_path: {QA_REPO or null}
310
+ output_path: {output_dir}/
311
+ </parameters>
312
+ "
313
+ )
314
+ ```
315
+
316
+ **Parse analyzer return:**
317
+
318
+ Expected return structure:
319
+ ```
320
+ ANALYZER_COMPLETE:
321
+ files_produced: [...]
322
+ total_test_count: N
323
+ pyramid_breakdown: {unit: N, integration: N, api: N, e2e: N}
324
+ risk_count: {high: N, medium: N, low: N}
325
+ commit_hash: "..."
326
+ ```
327
+
328
+ Capture `files_produced`, `total_test_count`, `pyramid_breakdown` for downstream stages.
329
+
330
+ **Handle analyzer checkpoint -- assumptions review:**
331
+ - If `IS_AUTO` is true: Auto-approve all assumptions (SAFE checkpoint). Log: "Auto-approved: Analyzer assumptions". Continue pipeline.
332
+ - If `IS_AUTO` is false: Present assumptions to user for review. Wait for confirmation or corrections. On user response, incorporate corrections and continue.
333
+
334
+ **State update -- mark analyze as complete:**
335
+ ```bash
336
+ node bin/qaa-tools.cjs state patch --"Analyze Status" complete 2>/dev/null || true
337
+ ```
338
+
339
+ **Verify artifacts exist before continuing:**
340
+
341
+ For Option 1:
342
+ ```bash
343
+ [ -f "${output_dir}/QA_ANALYSIS.md" ] && echo "OK" || echo "MISSING: QA_ANALYSIS.md"
344
+ [ -f "${output_dir}/TEST_INVENTORY.md" ] && echo "OK" || echo "MISSING: TEST_INVENTORY.md"
345
+ ```
346
+
347
+ For Options 2/3:
348
+ ```bash
349
+ [ -f "${output_dir}/GAP_ANALYSIS.md" ] && echo "OK" || echo "MISSING: GAP_ANALYSIS.md"
350
+ ```
351
+
352
+ If required artifacts are missing, treat as stage failure. Set status to failed and STOP pipeline.
353
+
354
+ Print: "Analysis complete. {total_test_count} test cases identified. Pyramid: unit={unit}, integration={integration}, api={api}, e2e={e2e}."
355
+
356
+ </step>
357
+
358
+ <step name="testid_inject">
359
+
360
+ ## Step 5: TestID Injection (Conditional)
361
+
362
+ **Condition:** Only execute if `has_frontend` is `true` from scanner return (Step 3).
363
+
364
+ **If `has_frontend` is false:**
365
+ Print: "Skipping TestID injection (no frontend detected)." Proceed directly to Step 6 (Plan).
366
+
367
+ **If `has_frontend` is true:**
368
+
369
+ **State update:**
370
+ ```bash
371
+ node bin/qaa-tools.cjs state patch --"Status" "Injecting test IDs into frontend components" 2>/dev/null || true
372
+ ```
373
+
374
+ **Print stage banner:**
375
+ ```
376
+ +------------------------------------------+
377
+ | STAGE 3: TestID Injector |
378
+ | Status: Running... |
379
+ +------------------------------------------+
380
+ ```
381
+
382
+ **Spawn testid-injector agent:**
383
+ ```
384
+ Agent(subagent_type="general-purpose",
385
+ prompt="
386
+ <objective>Audit and inject data-testid attributes into frontend components</objective>
387
+ <execution_context>@agents/qaa-testid-injector.md</execution_context>
388
+ <files_to_read>
389
+ - {output_dir}/SCAN_MANIFEST.md
390
+ - CLAUDE.md
391
+ </files_to_read>
392
+ <parameters>
393
+ dev_repo_path: {DEV_REPO}
394
+ output_path: {output_dir}/TESTID_AUDIT_REPORT.md
395
+ </parameters>
396
+ "
397
+ )
398
+ ```
399
+
400
+ **Parse return:**
401
+
402
+ Check for `INJECTOR_COMPLETE` vs `INJECTOR_SKIPPED`:
403
+
404
+ If `INJECTOR_COMPLETE`:
405
+ ```
406
+ INJECTOR_COMPLETE:
407
+ report_path: "..."
408
+ coverage_before: N%
409
+ coverage_after: N%
410
+ elements_injected: N
411
+ components_modified: N
412
+ ```
413
+ Log: "TestID injection complete. Coverage: {coverage_before}% -> {coverage_after}%. {elements_injected} elements injected."
414
+
415
+ If `INJECTOR_SKIPPED`:
416
+ ```
417
+ INJECTOR_SKIPPED:
418
+ reason: "..."
419
+ action: "..."
420
+ ```
421
+ Log the reason and continue pipeline.
422
+
423
+ **Handle injector checkpoint -- audit review:**
424
+ - If `IS_AUTO` is true: Auto-approve P0-only injection (SAFE checkpoint). Log: "Auto-approved: TestID injection (P0 elements only)". Continue pipeline.
425
+ - If `IS_AUTO` is false: Present audit report to user. Wait for approval, element selection, or rejection. On user response, incorporate decisions and continue.
426
+
427
+ **Verify artifact exists:**
428
+ ```bash
429
+ [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ] && echo "OK" || echo "MISSING: TESTID_AUDIT_REPORT.md"
430
+ ```
431
+
432
+ </step>
433
+
434
+ <step name="plan">
435
+
436
+ ## Step 6: Plan Test Generation
437
+
438
+ **State update -- mark generation as running (planning is part of generate):**
439
+ ```bash
440
+ node bin/qaa-tools.cjs state patch --"Generate Status" running --"Status" "Planning test generation" 2>/dev/null || true
441
+ ```
442
+
443
+ **Print stage banner:**
444
+ ```
445
+ +------------------------------------------+
446
+ | STAGE 4: Planner |
447
+ | Status: Running... |
448
+ +------------------------------------------+
449
+ ```
450
+
451
+ **Determine planner input based on option:**
452
+ - Option 1: Input from `{output_dir}/TEST_INVENTORY.md` + `{output_dir}/QA_ANALYSIS.md`
453
+ - Options 2 and 3: Input from `{output_dir}/GAP_ANALYSIS.md`
454
+
455
+ **Spawn planner agent:**
456
+ ```
457
+ Agent(subagent_type="general-purpose",
458
+ prompt="
459
+ <objective>Create test generation plan with task breakdown and dependencies</objective>
460
+ <execution_context>@agents/qaa-planner.md</execution_context>
461
+ <files_to_read>
462
+ - {input files based on option}
463
+ - CLAUDE.md
464
+ </files_to_read>
465
+ <parameters>
466
+ workflow_option: {option}
467
+ output_path: {output_dir}/GENERATION_PLAN.md
468
+ </parameters>
469
+ "
470
+ )
471
+ ```
472
+
473
+ **Parse planner return:**
474
+
475
+ Expected return structure:
476
+ ```
477
+ PLANNER_COMPLETE:
478
+ file_path: "..."
479
+ total_tasks: N
480
+ total_files: N
481
+ feature_count: N
482
+ dependency_depth: N
483
+ test_case_count: N
484
+ commit_hash: "..."
485
+ ```
486
+
487
+ Capture `total_tasks`, `total_files`, `feature_count` for executor stage and pipeline summary.
488
+
489
+ **Verify artifact exists:**
490
+ ```bash
491
+ [ -f "${output_dir}/GENERATION_PLAN.md" ] && echo "OK" || echo "MISSING: GENERATION_PLAN.md"
492
+ ```
493
+
494
+ If GENERATION_PLAN.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
495
+
496
+ Print: "Plan complete. {total_tasks} tasks, {total_files} files planned across {feature_count} features."
497
+
498
+ </step>
499
+
500
+ <step name="generate">
501
+
502
+ ## Step 7: Generate Test Files
503
+
504
+ State update continues from planning (already set to `running` in Step 6).
505
+
506
+ **Print stage banner:**
507
+ ```
508
+ +------------------------------------------+
509
+ | STAGE 5: Executor |
510
+ | Generating {total_files} test files |
511
+ | Status: Running... |
512
+ +------------------------------------------+
513
+ ```
514
+
515
+ **Determine execution strategy:**
516
+
517
+ Check if planner created multiple independent feature groups. If `feature_count > 1` AND parallelization is enabled (from init config):
518
+
519
+ **Parallel execution** (when feature_count > 1 and parallelization enabled):
520
+
521
+ For each independent feature group from the generation plan, spawn a separate executor agent:
522
+ ```
523
+ Agent(subagent_type="general-purpose",
524
+ prompt="
525
+ <objective>Generate test files for {feature} feature</objective>
526
+ <execution_context>@agents/qaa-executor.md</execution_context>
527
+ <files_to_read>
528
+ - {output_dir}/GENERATION_PLAN.md
529
+ - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
530
+ - CLAUDE.md
531
+ </files_to_read>
532
+ <parameters>
533
+ workflow_option: {option}
534
+ feature_group: {feature}
535
+ dev_repo_path: {DEV_REPO}
536
+ qa_repo_path: {QA_REPO or null}
537
+ output_path: {output_dir}/
538
+ </parameters>
539
+ "
540
+ )
541
+ ```
542
+
543
+ Multiple Agent() calls can be issued simultaneously for independent feature groups. Each executor handles one feature group and commits its files independently.
544
+
545
+ **Sequential execution** (when feature_count == 1 or parallelization disabled):
546
+
547
+ Spawn a single executor agent covering all tasks:
548
+ ```
549
+ Agent(subagent_type="general-purpose",
550
+ prompt="
551
+ <objective>Generate all test files from generation plan</objective>
552
+ <execution_context>@agents/qaa-executor.md</execution_context>
553
+ <files_to_read>
554
+ - {output_dir}/GENERATION_PLAN.md
555
+ - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
556
+ - CLAUDE.md
557
+ </files_to_read>
558
+ <parameters>
559
+ workflow_option: {option}
560
+ dev_repo_path: {DEV_REPO}
561
+ qa_repo_path: {QA_REPO or null}
562
+ output_path: {output_dir}/
563
+ </parameters>
564
+ "
565
+ )
566
+ ```
567
+
568
+ **Option 3 specific -- skip existing tests:**
569
+
570
+ For Option 3, pass `skip_existing_test_ids: true` to the executor so it checks existing test files by test ID before generating. If a test ID already exists in the QA repo, skip generating that test case:
571
+ ```
572
+ <parameters>
573
+ workflow_option: 3
574
+ skip_existing_test_ids: true
575
+ dev_repo_path: {DEV_REPO}
576
+ qa_repo_path: {QA_REPO}
577
+ output_path: {output_dir}/
578
+ </parameters>
579
+ ```
580
+
581
+ **Parse executor return:**
582
+
583
+ Expected return structure:
584
+ ```
585
+ EXECUTOR_COMPLETE:
586
+ files_created: [{path, type}, ...]
587
+ total_files: N
588
+ commit_count: N
589
+ features_covered: [...]
590
+ test_case_count: N
591
+ ```
592
+
593
+ Capture `files_created`, `total_files`, `commit_count` for validation stage and pipeline summary.
594
+
595
+ **State update -- mark generate as complete:**
596
+ ```bash
597
+ node bin/qaa-tools.cjs state patch --"Generate Status" complete --"Status" "Test generation complete" 2>/dev/null || true
598
+ ```
599
+
600
+ Print: "Generation complete. {total_files} files created across {features_covered_count} features. {commit_count} commits."
601
+
602
+ </step>
603
+
604
+ <step name="validate">
605
+
606
+ ## Step 8: Validate Generated Tests
607
+
608
+ **State update -- mark validate as running:**
609
+ ```bash
610
+ node bin/qaa-tools.cjs state patch --"Validate Status" running --"Status" "Validating generated tests" 2>/dev/null || true
611
+ ```
612
+
613
+ **Print stage banner:**
614
+ ```
615
+ +------------------------------------------+
616
+ | STAGE 6: Validator |
617
+ | Validating {total_files} test files |
618
+ | Status: Running... |
619
+ +------------------------------------------+
620
+ ```
621
+
622
+ **Spawn validator agent:**
623
+ ```
624
+ Agent(subagent_type="general-purpose",
625
+ prompt="
626
+ <objective>Run 4-layer validation on all generated test files</objective>
627
+ <execution_context>@agents/qaa-validator.md</execution_context>
628
+ <files_to_read>
629
+ - {list all generated test files from executor return -- files_created paths}
630
+ - {output_dir}/GENERATION_PLAN.md
631
+ - CLAUDE.md
632
+ </files_to_read>
633
+ <parameters>
634
+ mode: validation
635
+ max_fix_loops: 3
636
+ output_path: {output_dir}/VALIDATION_REPORT.md
637
+ </parameters>
638
+ "
639
+ )
640
+ ```
641
+
642
+ **4-layer validation:**
643
+ 1. **Syntax** -- File parses without errors
644
+ 2. **Structure** -- Follows POM rules, naming conventions, locator hierarchy
645
+ 3. **Dependencies** -- Imports resolve, fixtures exist, configs present
646
+ 4. **Logic** -- Assertions are concrete, test IDs are unique, no assertions in page objects
647
+
648
+ **5. Browser verification (if app URL available and Playwright MCP connected):**
649
+
650
+ After the 4-layer static validation, use Playwright MCP to verify E2E tests against the live app:
651
+
652
+ 1. Navigate to each page referenced in the E2E tests:
653
+ ```
654
+ mcp__playwright__browser_navigate({ url: "{app_url}/{route}" })
655
+ ```
656
+
657
+ 2. Take an accessibility snapshot to verify locators used in tests actually exist:
658
+ ```
659
+ mcp__playwright__browser_snapshot()
660
+ ```
661
+
662
+ 3. Cross-reference locators in generated tests against the real DOM:
663
+ - Verify `data-testid` values exist on the page
664
+ - Verify ARIA roles and names match test expectations
665
+ - Flag any test locator that does not match a real DOM element
666
+
667
+ 4. If mismatches are found, fix the test locators to match the real DOM and count as a fix loop iteration.
668
+
669
+ This browser verification step prevents delivering tests with locators that will immediately fail at runtime.
670
+
671
+ **Fix loop:** The validator automatically attempts to fix issues it finds. Maximum 3 fix loop iterations. After each fix attempt, re-validate.
672
+
673
+ **Parse validator return:**
674
+
675
+ Expected return structure:
676
+ ```
677
+ VALIDATOR_COMPLETE:
678
+ report_path: "..."
679
+ overall_status: PASS | PASS_WITH_WARNINGS | FAIL
680
+ confidence: HIGH | MEDIUM | LOW
681
+ layers_summary: {syntax: PASS|FAIL, structure: PASS|FAIL, dependencies: PASS|FAIL, logic: PASS|FAIL}
682
+ fix_loops_used: N
683
+ issues_found: N
684
+ issues_fixed: N
685
+ unresolved_count: N
686
+ ```
687
+
688
+ **RISKY CHECKPOINT -- Validator escalation:**
689
+
690
+ If `unresolved_count > 0` after max fix loops (3):
691
+ - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
692
+ - Present unresolved issues to user with full details from VALIDATION_REPORT.md
693
+ - Wait for user decision:
694
+ - `"approve-with-warnings"`: Accept the validation with warnings. Set Validate Status to complete. Continue to deliver.
695
+ - `"abort"`: Set Validate Status to failed. STOP PIPELINE ENTIRELY.
696
+ - Manual guidance: User provides specific fix instructions. Spawn fresh continuation agent to apply fixes and re-validate.
697
+
698
+ If `overall_status` is `PASS` or `PASS_WITH_WARNINGS` (and unresolved_count is 0):
699
+ ```bash
700
+ node bin/qaa-tools.cjs state patch --"Validate Status" complete --"Status" "Validation passed" 2>/dev/null || true
701
+ ```
702
+
703
+ **Verify artifact exists:**
704
+ ```bash
705
+ [ -f "${output_dir}/VALIDATION_REPORT.md" ] && echo "OK" || echo "MISSING: VALIDATION_REPORT.md"
706
+ ```
707
+
708
+ Print: "Validation complete. Status: {overall_status}. Confidence: {confidence}. {issues_found} issues found, {issues_fixed} fixed, {unresolved_count} unresolved."
709
+
710
+ </step>
711
+
712
+ <step name="bug_detective">
713
+
714
+ ## Step 9: Bug Detective (Conditional)
715
+
716
+ **Condition:** Only execute if test failures were detected during validation. Check:
717
+ - `overall_status === 'FAIL'` in validator return, OR
718
+ - Generated tests have runtime failures that need classification
719
+
720
+ **If no failures to classify:**
721
+ Print: "Skipping Bug Detective (no test failures detected)." Proceed directly to Step 10 (Deliver).
722
+
723
+ **If failures need classification:**
724
+
725
+ **State update:**
726
+ ```bash
727
+ node bin/qaa-tools.cjs state patch --"Status" "Classifying test failures" 2>/dev/null || true
728
+ ```
729
+
730
+ **Print stage banner:**
731
+ ```
732
+ +------------------------------------------+
733
+ | STAGE 7: Bug Detective |
734
+ | Status: Running... |
735
+ +------------------------------------------+
736
+ ```
737
+
738
+ **Spawn bug-detective agent:**
739
+ ```
740
+ Agent(subagent_type="general-purpose",
741
+ prompt="
742
+ <objective>Classify test failures and attempt auto-fixes for test errors. Use Playwright MCP to reproduce E2E failures in the browser when available.</objective>
743
+ <execution_context>@agents/qaa-bug-detective.md</execution_context>
744
+ <files_to_read>
745
+ - {test execution results -- from validator or direct test run}
746
+ - {failing test source files -- paths from executor return}
747
+ - CLAUDE.md
748
+ </files_to_read>
749
+ <parameters>
750
+ output_path: {output_dir}/FAILURE_CLASSIFICATION_REPORT.md
751
+ app_url: {app_url if available}
752
+ </parameters>
753
+ "
754
+ )
755
+ ```
756
+
757
+ **Parse bug-detective return:**
758
+
759
+ Expected return structure:
760
+ ```
761
+ DETECTIVE_COMPLETE:
762
+ report_path: "..."
763
+ total_failures: N
764
+ classification_breakdown: {app_bug: N, test_error: N, env_issue: N, inconclusive: N}
765
+ auto_fixes_applied: N
766
+ auto_fixes_verified: N
767
+ commit_hash: "..."
768
+ ```
769
+
770
+ **RISKY CHECKPOINT -- Application bugs detected:**
771
+
772
+ If `classification_breakdown.app_bug > 0`:
773
+ - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
774
+ - Present APPLICATION BUG classifications to user with full evidence from FAILURE_CLASSIFICATION_REPORT.md
775
+ - These are genuine bugs in the application code discovered during test execution
776
+ - The bug detective never touches application code -- it only reports
777
+ - User must review and decide how to proceed:
778
+ - Acknowledge bugs and continue pipeline (bugs will be in the PR description for developer attention)
779
+ - Abort pipeline to fix bugs first
780
+
781
+ **Verify artifact exists:**
782
+ ```bash
783
+ [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "OK" || echo "MISSING: FAILURE_CLASSIFICATION_REPORT.md"
784
+ ```
785
+
786
+ Print: "Bug Detective complete. {total_failures} failures classified: {app_bug} APP BUG, {test_error} TEST ERROR, {env_issue} ENV ISSUE, {inconclusive} INCONCLUSIVE. {auto_fixes_applied} auto-fixes applied."
787
+
788
+ </step>
789
+
790
+ <step name="deliver">
791
+
792
+ ## Step 10: Deliver
793
+
794
+ **State update -- mark deliver as running:**
795
+ ```bash
796
+ node bin/qaa-tools.cjs state patch --"Deliver Status" running --"Status" "Preparing delivery" 2>/dev/null || true
797
+ ```
798
+
799
+ **Print stage banner:**
800
+ ```
801
+ +------------------------------------------+
802
+ | STAGE 8: Deliver |
803
+ | Status: Running... |
804
+ +------------------------------------------+
805
+ ```
806
+
807
+ ### Sub-step 1: Pre-flight checks
808
+
809
+ **Check for git remote:**
810
+ ```bash
811
+ REMOTE_URL=$(git remote get-url origin 2>/dev/null || echo "")
812
+ ```
813
+
814
+ If `REMOTE_URL` is empty:
815
+ - Print: "No git remote found. Artifacts committed locally but PR creation skipped."
816
+ - Set `LOCAL_ONLY=true`
817
+
818
+ **Check for gh CLI authentication:**
819
+ ```bash
820
+ gh auth status 2>/dev/null
821
+ ```
822
+
823
+ If `gh auth status` fails:
824
+ - Print: "gh CLI not authenticated. Run 'gh auth login' first. Artifacts committed locally."
825
+ - Set `LOCAL_ONLY=true`
826
+
827
+ If both checks pass, set `LOCAL_ONLY=false`.
828
+
829
+ ### Sub-step 2: Derive project name
830
+
831
+ ```bash
832
+ # Read from package.json
833
+ PROJECT_NAME=$(node -e "try { const p = require('${DEV_REPO}/package.json'); console.log(p.name || ''); } catch { console.log(''); }" 2>/dev/null)
834
+
835
+ # Fallback to directory basename
836
+ if [ -z "$PROJECT_NAME" ]; then
837
+ PROJECT_NAME=$(basename "${DEV_REPO}")
838
+ fi
839
+
840
+ # Sanitize for branch naming
841
+ PROJECT_NAME=$(echo "$PROJECT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
842
+ ```
843
+
844
+ ### Sub-step 3: Detect default branch
845
+
846
+ ```bash
847
+ DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name' 2>/dev/null || echo "main")
848
+ ```
849
+
850
+ ### Sub-step 4: Create feature branch
851
+
852
+ ```bash
853
+ BRANCH="qa/auto-${PROJECT_NAME}-${date}"
854
+
855
+ # Handle branch name collision
856
+ if git rev-parse --verify "$BRANCH" 2>/dev/null || git rev-parse --verify "origin/$BRANCH" 2>/dev/null; then
857
+ SUFFIX=2
858
+ while git rev-parse --verify "${BRANCH}-${SUFFIX}" 2>/dev/null || git rev-parse --verify "origin/${BRANCH}-${SUFFIX}" 2>/dev/null; do
859
+ SUFFIX=$((SUFFIX + 1))
860
+ done
861
+ BRANCH="${BRANCH}-${SUFFIX}"
862
+ fi
863
+
864
+ git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
865
+ ```
866
+
867
+ ### Sub-step 5: Per-stage atomic commits
868
+
869
+ For each pipeline stage that produced artifacts, commit using `qaa-tools.cjs commit`. Check file existence before each commit.
870
+
871
+ **Scanner:**
872
+ ```bash
873
+ if [ -f "${output_dir}/SCAN_MANIFEST.md" ]; then
874
+ node bin/qaa-tools.cjs commit "qa(scanner): produce SCAN_MANIFEST.md for ${PROJECT_NAME}" --files ${output_dir}/SCAN_MANIFEST.md
875
+ fi
876
+ ```
877
+
878
+ **Analyzer (Option 1):**
879
+ ```bash
880
+ if [ -f "${output_dir}/QA_ANALYSIS.md" ]; then
881
+ ANALYZER_FILES="${output_dir}/QA_ANALYSIS.md ${output_dir}/TEST_INVENTORY.md"
882
+ [ -f "${output_dir}/QA_REPO_BLUEPRINT.md" ] && ANALYZER_FILES="${ANALYZER_FILES} ${output_dir}/QA_REPO_BLUEPRINT.md"
883
+ node bin/qaa-tools.cjs commit "qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md" --files ${ANALYZER_FILES}
884
+ fi
885
+ ```
886
+
887
+ **Analyzer (Options 2/3):**
888
+ ```bash
889
+ if [ -f "${output_dir}/GAP_ANALYSIS.md" ]; then
890
+ node bin/qaa-tools.cjs commit "qa(analyzer): produce GAP_ANALYSIS.md" --files ${output_dir}/GAP_ANALYSIS.md
891
+ fi
892
+ ```
893
+
894
+ **TestID Injector (if ran):**
895
+ ```bash
896
+ if [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ]; then
897
+ node bin/qaa-tools.cjs commit "qa(testid-injector): inject ${elements_injected} data-testid attributes across ${components_modified} components" --files ${output_dir}/TESTID_AUDIT_REPORT.md ${modified_source_files}
898
+ fi
899
+ ```
900
+
901
+ **Executor:**
902
+ ```bash
903
+ if [ -n "${generated_file_paths}" ]; then
904
+ node bin/qaa-tools.cjs commit "qa(executor): generate ${total_files} test files with POMs and fixtures" --files ${generated_file_paths}
905
+ fi
906
+ ```
907
+
908
+ **Planner:**
909
+ ```bash
910
+ if [ -f "${output_dir}/GENERATION_PLAN.md" ]; then
911
+ node bin/qaa-tools.cjs commit "qa(planner): produce GENERATION_PLAN.md" --files ${output_dir}/GENERATION_PLAN.md
912
+ fi
913
+ ```
914
+
915
+ **Validator:**
916
+ ```bash
917
+ if [ -f "${output_dir}/VALIDATION_REPORT.md" ]; then
918
+ node bin/qaa-tools.cjs commit "qa(validator): validate generated tests - ${overall_status} with ${confidence} confidence" --files ${output_dir}/VALIDATION_REPORT.md
919
+ fi
920
+ ```
921
+
922
+ **Bug Detective (if ran):**
923
+ ```bash
924
+ if [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ]; then
925
+ node bin/qaa-tools.cjs commit "qa(bug-detective): classify ${total_failures} failures - ${classification_summary}" --files ${output_dir}/FAILURE_CLASSIFICATION_REPORT.md
926
+ fi
927
+ ```
928
+
929
+ ### Sub-step 6: Push branch
930
+
931
+ If `LOCAL_ONLY` is true, skip this sub-step.
932
+
933
+ ```bash
934
+ git push -u origin "$BRANCH"
935
+ ```
936
+
937
+ If push fails:
938
+ - Print: "Push failed: {error_message}. Artifacts committed locally on branch ${BRANCH}."
939
+ - Set `LOCAL_ONLY=true`
940
+
941
+ ### Sub-step 7: Build PR body
942
+
943
+ If `LOCAL_ONLY` is true, skip this sub-step.
944
+
945
+ Read the PR template:
946
+ ```bash
947
+ PR_BODY=$(cat templates/pr-template.md)
948
+ ```
949
+
950
+ Replace all `{placeholder}` tokens with actual values collected during pipeline execution:
951
+ - `{architecture_type}` -- from QA_ANALYSIS.md or SCAN_MANIFEST.md
952
+ - `{framework}` -- detected test framework
953
+ - `{risk_summary}` -- risk assessment counts (e.g., "3 HIGH, 5 MEDIUM, 2 LOW")
954
+ - `{unit_count}` -- from pyramid_breakdown.unit
955
+ - `{integration_count}` -- from pyramid_breakdown.integration
956
+ - `{api_count}` -- from pyramid_breakdown.api
957
+ - `{e2e_count}` -- from pyramid_breakdown.e2e
958
+ - `{total_count}` -- from total_test_count
959
+ - `{modules_covered}` -- count of modules with tests
960
+ - `{coverage_estimate}` -- estimated coverage percentage
961
+ - `{validation_result}` -- PASS, PASS_WITH_WARNINGS, or FAIL
962
+ - `{confidence}` -- HIGH, MEDIUM, or LOW
963
+ - `{fix_loops_used}` -- number 0-3
964
+ - `{issues_found}` -- total issues found during validation
965
+ - `{issues_fixed}` -- total issues auto-fixed
966
+ - `{file_list}` -- if total files <= 50, list each file; if > 50, use summary
967
+
968
+ ### Sub-step 8: Create draft PR
969
+
970
+ If `LOCAL_ONLY` is true, skip this sub-step.
971
+
972
+ ```bash
973
+ PR_URL=$(gh pr create \
974
+ --draft \
975
+ --title "qa: automated test suite for ${PROJECT_NAME}" \
976
+ --body "${PR_BODY}" \
977
+ --label "qa-automation" \
978
+ --label "auto-generated" \
979
+ --assignee "@me" 2>&1)
980
+ ```
981
+
982
+ Do NOT pass `--base` flag. Let gh auto-detect the default branch.
983
+
984
+ On failure:
985
+ - Print: "PR creation failed: ${PR_URL}. Artifacts remain on branch ${BRANCH}."
986
+ - Do NOT stop the pipeline -- artifacts are committed and pushed.
987
+
988
+ ### Sub-step 9: Print result
989
+
990
+ If PR was created successfully:
991
+ ```
992
+ PR created: ${PR_URL}
993
+ ```
994
+
995
+ If `LOCAL_ONLY` is true:
996
+ ```
997
+ PR: not created (local-only mode). Artifacts committed on branch: ${BRANCH}
998
+ ```
999
+
1000
+ **State update -- mark deliver as complete:**
1001
+ ```bash
1002
+ node bin/qaa-tools.cjs state patch --"Deliver Status" complete --"Status" "Pipeline complete" 2>/dev/null || true
1003
+ ```
1004
+
1005
+ **Clear auto-chain flag at pipeline completion:**
1006
+ ```bash
1007
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1008
+ ```
1009
+
1010
+ ### Print pipeline summary banner:
1011
+
1012
+ ```
1013
+ ======================================================
1014
+ QA PIPELINE COMPLETE
1015
+ ======================================================
1016
+
1017
+ Option: {option} ({option_description})
1018
+ Repository: {DEV_REPO}
1019
+ QA Repo: {QA_REPO or 'N/A'}
1020
+ Maturity Score: {maturity_score or 'N/A'}
1021
+
1022
+ Stages Completed:
1023
+ [{check}] Scan -- {scan_result}
1024
+ [{check}] Analyze -- {analyze_result} ({test_count} test cases)
1025
+ [{check}] TestID Inject -- {inject_result or 'skipped'}
1026
+ [{check}] Plan -- {plan_result} ({file_count} files planned)
1027
+ [{check}] Generate -- {generate_result} ({files_created} files created)
1028
+ [{check}] Validate -- {validate_result} ({confidence} confidence)
1029
+ [{check}] Bug Detective -- {detective_result or 'skipped'}
1030
+ [{check}] Deliver -- {deliver_result}
1031
+
1032
+ PR: {pr_url or 'not created (local-only)'}
1033
+
1034
+ Artifacts:
1035
+ {list all produced .md files in output_dir}
1036
+
1037
+ Total Time: {total_duration}
1038
+ ======================================================
1039
+ ```
1040
+
1041
+ Where: `[x]` = completed, `[ ]` = skipped, `[!]` = failed.
1042
+
1043
+ </step>
1044
+
1045
+ </process>
1046
+
1047
+ <error_handling>
1048
+
1049
+ ## Error Handling
1050
+
1051
+ ### Stage Failure Protocol
1052
+
1053
+ When any agent returns a failure or error:
1054
+
1055
+ 1. **Set stage status to failed:**
1056
+ ```bash
1057
+ node bin/qaa-tools.cjs state patch --"{Stage} Status" failed --"Status" "Pipeline stopped: {Stage} failed - {reason}" 2>/dev/null || true
1058
+ ```
1059
+
1060
+ 2. **Print failure banner:**
1061
+ ```
1062
+ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1063
+ ! PIPELINE STOPPED !
1064
+ ! Stage: {stage_name} !
1065
+ ! Reason: {failure_reason} !
1066
+ ! !
1067
+ ! Completed: {completed_stages} !
1068
+ ! Artifacts: {artifacts_so_far} !
1069
+ ! !
1070
+ ! Action required: Review and re-run !
1071
+ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1072
+ ```
1073
+
1074
+ 3. **DO NOT continue to next stage.** The pipeline stops entirely at the failed stage.
1075
+
1076
+ 4. **DO NOT create partial PR.** No branch, no commit, no PR with incomplete results.
1077
+
1078
+ 5. **Preserve all artifacts produced so far.** They remain on disk in `{output_dir}/` for debugging.
1079
+
1080
+ ### Artifact Verification
1081
+
1082
+ After EVERY agent spawn, before advancing to next stage, verify the expected output artifact exists on disk:
1083
+ ```bash
1084
+ [ -f "{expected_artifact_path}" ] && echo "OK" || echo "MISSING"
1085
+ ```
1086
+
1087
+ If artifacts are missing, treat as stage failure and STOP pipeline.
1088
+
1089
+ ### qaa-tools.cjs Graceful Fallback
1090
+
1091
+ All `node bin/qaa-tools.cjs` calls use `2>/dev/null || true` to handle cases where the tool is not installed or not found. The pipeline must not break due to missing state management tooling -- it logs a warning and continues.
1092
+
1093
+ </error_handling>
1094
+
1095
+ <auto_advance>
1096
+
1097
+ ## Auto-Advance Mode
1098
+
1099
+ Auto-advance is enabled when ANY of these is true:
1100
+ - `--auto` flag passed to the `/qa-start` invocation
1101
+ - `config.json` has `workflow.auto_advance = true` (persistent user preference)
1102
+ - `workflow._auto_chain_active = true` in config (ephemeral chain flag from ongoing auto run)
1103
+
1104
+ ### Behavior in Auto Mode
1105
+
1106
+ **SAFE checkpoints are auto-approved.** The pipeline continues without pausing. A log message records the auto-approval:
1107
+ ```
1108
+ Auto-approved: {checkpoint_description}
1109
+ ```
1110
+
1111
+ **RISKY checkpoints ALWAYS pause.** Even in auto mode, the pipeline stops and presents the checkpoint to the user.
1112
+
1113
+ ### Safe vs Risky Checkpoint Classification
1114
+
1115
+ **SAFE (auto-approve in auto mode):**
1116
+
1117
+ | Checkpoint | Agent | Auto-Action |
1118
+ |------------|-------|-------------|
1119
+ | Framework detection uncertain (LOW confidence) | Scanner | Approve with most likely framework |
1120
+ | Analyzer assumptions review | Analyzer | Approve all assumptions |
1121
+ | TestID audit review | TestID Injector | Approve P0-only injection |
1122
+
1123
+ **RISKY (ALWAYS pause, even in auto mode):**
1124
+
1125
+ | Checkpoint | Agent | User Action Required |
1126
+ |------------|-------|---------------------|
1127
+ | Validator escalation (unresolved issues after 3 fix loops) | Validator | approve-with-warnings, abort, or fix guidance |
1128
+ | APPLICATION BUG classification | Bug Detective | Review bugs, continue or fix first |
1129
+ | Any checkpoint with "unresolved" or "failed" blocking text | Any | Review specific blocking issue |
1130
+
1131
+ ### Checkpoint Handling Flow
1132
+
1133
+ ```
1134
+ On agent return with checkpoint data:
1135
+ 1. Extract checkpoint blocking field content
1136
+ 2. Classify as SAFE or RISKY:
1137
+ - "framework detection" -> SAFE
1138
+ - "assumptions" -> SAFE
1139
+ - "audit" or "data-testid" -> SAFE
1140
+ - "unresolved" -> RISKY
1141
+ - "failed" -> RISKY
1142
+ - "APPLICATION BUG" -> RISKY
1143
+ - Default (no pattern match) -> RISKY (conservative)
1144
+ 3. If IS_AUTO and SAFE:
1145
+ - Auto-approve with default action
1146
+ - Log the auto-approval
1147
+ - Continue pipeline
1148
+ 4. If IS_AUTO and RISKY:
1149
+ - PAUSE pipeline
1150
+ - Print checkpoint details with full context
1151
+ - Wait for user input
1152
+ 5. If NOT auto (manual mode):
1153
+ - PAUSE pipeline
1154
+ - Print checkpoint details
1155
+ - Wait for user input
1156
+ ```
1157
+
1158
+ ### Resume After Checkpoint
1159
+
1160
+ When resuming after a checkpoint, spawn a FRESH agent with explicit state:
1161
+ ```
1162
+ Agent(subagent_type="general-purpose",
1163
+ prompt="
1164
+ <objective>Continue QA pipeline from {stage} stage</objective>
1165
+ <execution_context>@agents/qa-pipeline-orchestrator.md</execution_context>
1166
+ <resume_context>
1167
+ Pipeline state:
1168
+ - Completed stages: {list of completed stages with their results}
1169
+ - Current stage: {stage that triggered checkpoint}
1170
+ - Checkpoint response: {user's response or decision}
1171
+ - Artifacts produced so far: {list of files with paths}
1172
+
1173
+ Resume from: {exact step in pipeline to resume from}
1174
+ User decision: {what user chose at checkpoint}
1175
+ </resume_context>
1176
+ "
1177
+ )
1178
+ ```
1179
+
1180
+ ### Stale Chain Flag Protection
1181
+
1182
+ At orchestrator init, if `--auto` was NOT passed AND `auto_advance` config is false:
1183
+ ```bash
1184
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1185
+ ```
1186
+
1187
+ At pipeline completion (success or failure):
1188
+ ```bash
1189
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1190
+ ```
1191
+
1192
+ </auto_advance>