qaa-agent 1.9.0 → 1.9.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,1262 +1,1405 @@
1
- <purpose>
2
-
3
- Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
4
-
5
- Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
6
-
7
- </purpose>
8
-
9
- <required_reading>
10
-
11
- Read these files BEFORE executing any pipeline stage. Do NOT skip.
12
-
13
- - **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
14
- - **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
15
-
16
- </required_reading>
17
-
18
- <process>
19
-
20
- <step name="initialize" priority="first">
21
-
22
- ## Step 1: Initialize Pipeline
23
-
24
- Parse `$ARGUMENTS` for flags:
25
-
26
- ```bash
27
- DEV_REPO=""
28
- QA_REPO=""
29
- IS_AUTO=false
30
-
31
- # Parse --dev-repo flag
32
- if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
33
- DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
34
- fi
35
-
36
- # Parse --qa-repo flag
37
- if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
38
- QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
39
- fi
40
-
41
- # Parse --auto flag
42
- if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
43
- IS_AUTO=true
44
- fi
45
- ```
46
-
47
- **If no --dev-repo provided**, use the current working directory:
48
- ```bash
49
- if [ -z "$DEV_REPO" ]; then
50
- DEV_REPO=$(pwd)
51
- fi
52
- ```
53
-
54
- **Attempt to call qaa-tools init** (handle missing tool gracefully):
55
- ```bash
56
- INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
57
- ```
58
-
59
- If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
60
- - Set `output_dir` to `.qa-output`
61
- - Set `date` to current date in `YYYY-MM-DD` format
62
- - Create output directory: `mkdir -p "$output_dir"`
63
-
64
- If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
65
-
66
- **Detect workflow option based on inputs:**
67
-
68
- - If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
69
- - If `QA_REPO` is provided: Assess QA repo maturity
70
- - Check for existing test files, configs, coverage reports in QA repo
71
- - Count test files, evaluate framework setup, check for CI config
72
- - Score 0-100 based on test count, framework config presence, CI setup, coverage data
73
- - Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
74
- - Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
75
-
76
- **Determine auto-advance mode:**
77
- ```bash
78
- # Check persistent config flag
79
- AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
80
- AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
81
-
82
- if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
83
- IS_AUTO=true
84
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
85
- fi
86
-
87
- # Safety: clear stale chain flag if NOT in auto mode
88
- if [ "$IS_AUTO" = "false" ]; then
89
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
90
- fi
91
- ```
92
-
93
- **Print initialization banner:**
94
- ```
95
- === QA Pipeline Orchestrator ===
96
- Option: {option} ({description})
97
- Dev Repo: {DEV_REPO}
98
- QA Repo: {QA_REPO or 'N/A'}
99
- Maturity Score: {maturity_score or 'N/A'}
100
- Auto-Advance: {IS_AUTO}
101
- Date: {date}
102
- ================================
103
- ```
104
-
105
- Where `{description}` is:
106
- - Option 1: "Dev-Only -- Full Pipeline"
107
- - Option 2: "Dev + Immature QA -- Gap-Fill"
108
- - Option 3: "Dev + Mature QA -- Surgical"
109
-
110
- </step>
111
-
112
- <step name="detect_framework">
113
-
114
- ## Step 2: Detect Framework
115
-
116
- Before scanning, detect the project's language and test framework to guide all downstream agents.
117
-
118
- **Read project config files:**
119
- ```bash
120
- # Check for Node.js / JavaScript / TypeScript
121
- [ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
122
-
123
- # Check for Python
124
- [ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
125
- [ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
126
- [ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
127
-
128
- # Check for .NET
129
- ls "${DEV_REPO}"/*.csproj 2>/dev/null
130
- ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
131
-
132
- # Check for Java
133
- [ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
134
- [ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
135
- ```
136
-
137
- **Detect test framework from config files:**
138
- ```bash
139
- # JavaScript/TypeScript ecosystem
140
- [ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
141
- [ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
142
- [ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
143
- [ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
144
-
145
- # Python ecosystem
146
- [ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
147
-
148
- # Check package.json devDependencies for test frameworks
149
- node -e "
150
- try {
151
- const pkg = require('${DEV_REPO}/package.json');
152
- const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
153
- const frameworks = [];
154
- if (deps.cypress) frameworks.push('cypress');
155
- if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
156
- if (deps.jest) frameworks.push('jest');
157
- if (deps.vitest) frameworks.push('vitest');
158
- if (deps.mocha) frameworks.push('mocha');
159
- console.log(frameworks.join(',') || 'none');
160
- } catch { console.log('no-package-json'); }
161
- " 2>/dev/null
162
- ```
163
-
164
- **Assess detection confidence:**
165
- - **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
166
- - **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
167
- - **LOW**: No test framework detected, or conflicting signals
168
-
169
- **If no test framework found:**
170
- - If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
171
- - If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
172
- - React/Next.js/Vue/Angular frontend -> Playwright
173
- - Node.js API -> Jest or Vitest (prefer Vitest if ESM)
174
- - Python -> Pytest
175
- - Log: "Auto-selected: {framework} (no existing test framework detected)"
176
-
177
- **If detection confidence is LOW:**
178
- - If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
179
- - If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
180
-
181
- Store detected framework, language, and confidence for all downstream agents.
182
-
183
- </step>
184
-
185
- <step name="scan">
186
-
187
- ## Step 3: Scan Repository
188
-
189
- **State update -- mark scan as running:**
190
- ```bash
191
- node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
192
- ```
193
-
194
- **Print stage banner:**
195
- ```
196
- +------------------------------------------+
197
- | STAGE 1: Scanner |
198
- | Status: Running... |
199
- +------------------------------------------+
200
- ```
201
-
202
- **Spawn scanner agent:**
203
-
204
- For **Option 1** (scan dev repo only):
205
- ```
206
- Agent(subagent_type="general-purpose",
207
- prompt="
208
- <objective>Scan repository and produce SCAN_MANIFEST.md</objective>
209
- <execution_context>@agents/qaa-scanner.md</execution_context>
210
- <files_to_read>
211
- - CLAUDE.md
212
- </files_to_read>
213
- <parameters>
214
- dev_repo_path: {DEV_REPO}
215
- qa_repo_path: null
216
- output_path: {output_dir}/SCAN_MANIFEST.md
217
- </parameters>
218
- "
219
- )
220
- ```
221
-
222
- For **Options 2 and 3** (scan both repos):
223
- ```
224
- Agent(subagent_type="general-purpose",
225
- prompt="
226
- <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
227
- <execution_context>@agents/qaa-scanner.md</execution_context>
228
- <files_to_read>
229
- - CLAUDE.md
230
- </files_to_read>
231
- <parameters>
232
- dev_repo_path: {DEV_REPO}
233
- qa_repo_path: {QA_REPO}
234
- output_path: {output_dir}/SCAN_MANIFEST.md
235
- </parameters>
236
- "
237
- )
238
- ```
239
-
240
- **Parse scanner return:**
241
-
242
- Expected return structure:
243
- ```
244
- SCANNER_COMPLETE:
245
- file_path: ".qa-output/SCAN_MANIFEST.md"
246
- decision: PROCEED | STOP
247
- has_frontend: true | false
248
- detection_confidence: HIGH | MEDIUM | LOW
249
- ```
250
-
251
- **Handle decision field:**
252
- - If `decision` is `STOP`:
253
- ```bash
254
- node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
255
- ```
256
- Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
257
-
258
- - If `decision` is `PROCEED`:
259
- ```bash
260
- node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
261
- ```
262
- Capture `has_frontend` for testid-injector conditional (Step 5).
263
- Capture `detection_confidence` for checkpoint handling.
264
-
265
- **Verify artifact exists before continuing:**
266
- ```bash
267
- [ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
268
- ```
269
-
270
- If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
271
-
272
- </step>
273
-
274
- <step name="research">
275
-
276
- ## Step 3b: Research Testing Ecosystem
277
-
278
- **State update -- mark research as running:**
279
- ```bash
280
- node bin/qaa-tools.cjs state patch --"Research Status" running --"Status" "Researching testing ecosystem" 2>/dev/null || true
281
- ```
282
-
283
- **Print stage banner:**
284
- ```
285
- +------------------------------------------+
286
- | STAGE 1b: Project Researcher |
287
- | Status: Running... |
288
- +------------------------------------------+
289
- ```
290
-
291
- **Create output directory:**
292
- ```bash
293
- mkdir -p ${output_dir}/research
294
- ```
295
-
296
- **Spawn researcher agent:**
297
- ```
298
- Agent(subagent_type="general-purpose",
299
- prompt="
300
- <objective>Research the testing ecosystem for this project. Use Context7 MCP as the primary source for all framework and library questions. Produce research documents consumed by downstream agents.</objective>
301
- <execution_context>@agents/qaa-project-researcher.md</execution_context>
302
- <files_to_read>
303
- - CLAUDE.md
304
- - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
305
- - ${output_dir}/SCAN_MANIFEST.md
306
- </files_to_read>
307
- <parameters>
308
- mode: stack-testing
309
- dev_repo_path: {DEV_REPO}
310
- output_dir: ${output_dir}/research
311
- </parameters>
312
- "
313
- )
314
- ```
315
-
316
- **Verify research artifacts exist:**
317
- ```bash
318
- ls ${output_dir}/research/*.md 2>/dev/null && echo "OK: Research files produced" || echo "WARNING: No research files produced"
319
- ```
320
-
321
- **Research is non-blocking:** If the researcher fails or produces no output, the pipeline continues downstream agents will fall back to Context7 queries directly. Log the warning but do NOT stop the pipeline.
322
-
323
- ```bash
324
- if [ ! -f "${output_dir}/research/TESTING_STACK.md" ]; then
325
- echo "WARNING: Research stage produced no output. Downstream agents will query Context7 directly."
326
- node bin/qaa-tools.cjs state patch --"Research Status" "skipped (no output)" 2>/dev/null || true
327
- else
328
- node bin/qaa-tools.cjs state patch --"Research Status" complete 2>/dev/null || true
329
- fi
330
- ```
331
-
332
- </step>
333
-
334
- <step name="analyze">
335
-
336
- ## Step 4: Analyze Repository
337
-
338
- **State update -- mark analyze as running:**
339
- ```bash
340
- node bin/qaa-tools.cjs state patch --"Analyze Status" running --"Status" "Analyzing repository" 2>/dev/null || true
341
- ```
342
-
343
- **Print stage banner:**
344
- ```
345
- +------------------------------------------+
346
- | STAGE 2: Analyzer |
347
- | Status: Running... |
348
- +------------------------------------------+
349
- ```
350
-
351
- **Determine analyzer mode based on option:**
352
- - Option 1: `mode = 'full'` (produces QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md)
353
- - Options 2 and 3: `mode = 'gap'` (produces GAP_ANALYSIS.md)
354
-
355
- **Spawn analyzer agent:**
356
- ```
357
- Agent(subagent_type="general-purpose",
358
- prompt="
359
- <objective>Analyze scanned repository and produce analysis artifacts</objective>
360
- <execution_context>@agents/qaa-analyzer.md</execution_context>
361
- <files_to_read>
362
- - {output_dir}/SCAN_MANIFEST.md
363
- - CLAUDE.md
364
- - {output_dir}/research/TESTING_STACK.md (if exists)
365
- - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
366
- </files_to_read>
367
- <parameters>
368
- mode: {mode}
369
- workflow_option: {option}
370
- dev_repo_path: {DEV_REPO}
371
- qa_repo_path: {QA_REPO or null}
372
- output_path: {output_dir}/
373
- </parameters>
374
- "
375
- )
376
- ```
377
-
378
- **Parse analyzer return:**
379
-
380
- Expected return structure:
381
- ```
382
- ANALYZER_COMPLETE:
383
- files_produced: [...]
384
- total_test_count: N
385
- pyramid_breakdown: {unit: N, integration: N, api: N, e2e: N}
386
- risk_count: {high: N, medium: N, low: N}
387
- commit_hash: "..."
388
- ```
389
-
390
- Capture `files_produced`, `total_test_count`, `pyramid_breakdown` for downstream stages.
391
-
392
- **Handle analyzer checkpoint -- assumptions review:**
393
- - If `IS_AUTO` is true: Auto-approve all assumptions (SAFE checkpoint). Log: "Auto-approved: Analyzer assumptions". Continue pipeline.
394
- - If `IS_AUTO` is false: Present assumptions to user for review. Wait for confirmation or corrections. On user response, incorporate corrections and continue.
395
-
396
- **State update -- mark analyze as complete:**
397
- ```bash
398
- node bin/qaa-tools.cjs state patch --"Analyze Status" complete 2>/dev/null || true
399
- ```
400
-
401
- **Verify artifacts exist before continuing:**
402
-
403
- For Option 1:
404
- ```bash
405
- [ -f "${output_dir}/QA_ANALYSIS.md" ] && echo "OK" || echo "MISSING: QA_ANALYSIS.md"
406
- [ -f "${output_dir}/TEST_INVENTORY.md" ] && echo "OK" || echo "MISSING: TEST_INVENTORY.md"
407
- ```
408
-
409
- For Options 2/3:
410
- ```bash
411
- [ -f "${output_dir}/GAP_ANALYSIS.md" ] && echo "OK" || echo "MISSING: GAP_ANALYSIS.md"
412
- ```
413
-
414
- If required artifacts are missing, treat as stage failure. Set status to failed and STOP pipeline.
415
-
416
- Print: "Analysis complete. {total_test_count} test cases identified. Pyramid: unit={unit}, integration={integration}, api={api}, e2e={e2e}."
417
-
418
- </step>
419
-
420
- <step name="testid_inject">
421
-
422
- ## Step 5: TestID Injection (Conditional)
423
-
424
- **Condition:** Only execute if `has_frontend` is `true` from scanner return (Step 3).
425
-
426
- **If `has_frontend` is false:**
427
- Print: "Skipping TestID injection (no frontend detected)." Proceed directly to Step 6 (Plan).
428
-
429
- **If `has_frontend` is true:**
430
-
431
- **State update:**
432
- ```bash
433
- node bin/qaa-tools.cjs state patch --"Status" "Injecting test IDs into frontend components" 2>/dev/null || true
434
- ```
435
-
436
- **Print stage banner:**
437
- ```
438
- +------------------------------------------+
439
- | STAGE 3: TestID Injector |
440
- | Status: Running... |
441
- +------------------------------------------+
442
- ```
443
-
444
- **Spawn testid-injector agent:**
445
- ```
446
- Agent(subagent_type="general-purpose",
447
- prompt="
448
- <objective>Audit and inject data-testid attributes into frontend components</objective>
449
- <execution_context>@agents/qaa-testid-injector.md</execution_context>
450
- <files_to_read>
451
- - {output_dir}/SCAN_MANIFEST.md
452
- - CLAUDE.md
453
- </files_to_read>
454
- <parameters>
455
- dev_repo_path: {DEV_REPO}
456
- output_path: {output_dir}/TESTID_AUDIT_REPORT.md
457
- </parameters>
458
- "
459
- )
460
- ```
461
-
462
- **Parse return:**
463
-
464
- Check for `INJECTOR_COMPLETE` vs `INJECTOR_SKIPPED`:
465
-
466
- If `INJECTOR_COMPLETE`:
467
- ```
468
- INJECTOR_COMPLETE:
469
- report_path: "..."
470
- coverage_before: N%
471
- coverage_after: N%
472
- elements_injected: N
473
- components_modified: N
474
- ```
475
- Log: "TestID injection complete. Coverage: {coverage_before}% -> {coverage_after}%. {elements_injected} elements injected."
476
-
477
- If `INJECTOR_SKIPPED`:
478
- ```
479
- INJECTOR_SKIPPED:
480
- reason: "..."
481
- action: "..."
482
- ```
483
- Log the reason and continue pipeline.
484
-
485
- **Handle injector checkpoint -- audit review:**
486
- - If `IS_AUTO` is true: Auto-approve P0-only injection (SAFE checkpoint). Log: "Auto-approved: TestID injection (P0 elements only)". Continue pipeline.
487
- - If `IS_AUTO` is false: Present audit report to user. Wait for approval, element selection, or rejection. On user response, incorporate decisions and continue.
488
-
489
- **Verify artifact exists:**
490
- ```bash
491
- [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ] && echo "OK" || echo "MISSING: TESTID_AUDIT_REPORT.md"
492
- ```
493
-
494
- </step>
495
-
496
- <step name="plan">
497
-
498
- ## Step 6: Plan Test Generation
499
-
500
- **State update -- mark generation as running (planning is part of generate):**
501
- ```bash
502
- node bin/qaa-tools.cjs state patch --"Generate Status" running --"Status" "Planning test generation" 2>/dev/null || true
503
- ```
504
-
505
- **Print stage banner:**
506
- ```
507
- +------------------------------------------+
508
- | STAGE 4: Planner |
509
- | Status: Running... |
510
- +------------------------------------------+
511
- ```
512
-
513
- **Determine planner input based on option:**
514
- - Option 1: Input from `{output_dir}/TEST_INVENTORY.md` + `{output_dir}/QA_ANALYSIS.md`
515
- - Options 2 and 3: Input from `{output_dir}/GAP_ANALYSIS.md`
516
-
517
- **Spawn planner agent:**
518
- ```
519
- Agent(subagent_type="general-purpose",
520
- prompt="
521
- <objective>Create test generation plan with task breakdown and dependencies</objective>
522
- <execution_context>@agents/qaa-planner.md</execution_context>
523
- <files_to_read>
524
- - {input files based on option}
525
- - CLAUDE.md
526
- </files_to_read>
527
- <parameters>
528
- workflow_option: {option}
529
- output_path: {output_dir}/GENERATION_PLAN.md
530
- </parameters>
531
- "
532
- )
533
- ```
534
-
535
- **Parse planner return:**
536
-
537
- Expected return structure:
538
- ```
539
- PLANNER_COMPLETE:
540
- file_path: "..."
541
- total_tasks: N
542
- total_files: N
543
- feature_count: N
544
- dependency_depth: N
545
- test_case_count: N
546
- commit_hash: "..."
547
- ```
548
-
549
- Capture `total_tasks`, `total_files`, `feature_count` for executor stage and pipeline summary.
550
-
551
- **Verify artifact exists:**
552
- ```bash
553
- [ -f "${output_dir}/GENERATION_PLAN.md" ] && echo "OK" || echo "MISSING: GENERATION_PLAN.md"
554
- ```
555
-
556
- If GENERATION_PLAN.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
557
-
558
- Print: "Plan complete. {total_tasks} tasks, {total_files} files planned across {feature_count} features."
559
-
560
- </step>
561
-
562
- <step name="generate">
563
-
564
- ## Step 7: Generate Test Files
565
-
566
- State update continues from planning (already set to `running` in Step 6).
567
-
568
- **Print stage banner:**
569
- ```
570
- +------------------------------------------+
571
- | STAGE 5: Executor |
572
- | Generating {total_files} test files |
573
- | Status: Running... |
574
- +------------------------------------------+
575
- ```
576
-
577
- **Determine execution strategy:**
578
-
579
- Check if planner created multiple independent feature groups. If `feature_count > 1` AND parallelization is enabled (from init config):
580
-
581
- **Parallel execution** (when feature_count > 1 and parallelization enabled):
582
-
583
- For each independent feature group from the generation plan, spawn a separate executor agent:
584
- ```
585
- Agent(subagent_type="general-purpose",
586
- prompt="
587
- <objective>Generate test files for {feature} feature</objective>
588
- <execution_context>@agents/qaa-executor.md</execution_context>
589
- <files_to_read>
590
- - {output_dir}/GENERATION_PLAN.md
591
- - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
592
- - CLAUDE.md
593
- - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
594
- - {output_dir}/research/E2E_STRATEGY.md (if exists)
595
- - {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
596
- </files_to_read>
597
- <parameters>
598
- workflow_option: {option}
599
- feature_group: {feature}
600
- dev_repo_path: {DEV_REPO}
601
- qa_repo_path: {QA_REPO or null}
602
- output_path: {output_dir}/
603
- </parameters>
604
- "
605
- )
606
- ```
607
-
608
- Multiple Agent() calls can be issued simultaneously for independent feature groups. Each executor handles one feature group and commits its files independently.
609
-
610
- **Sequential execution** (when feature_count == 1 or parallelization disabled):
611
-
612
- Spawn a single executor agent covering all tasks:
613
- ```
614
- Agent(subagent_type="general-purpose",
615
- prompt="
616
- <objective>Generate all test files from generation plan</objective>
617
- <execution_context>@agents/qaa-executor.md</execution_context>
618
- <files_to_read>
619
- - {output_dir}/GENERATION_PLAN.md
620
- - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
621
- - CLAUDE.md
622
- - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
623
- - {output_dir}/research/E2E_STRATEGY.md (if exists)
624
- - {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
625
- </files_to_read>
626
- <parameters>
627
- workflow_option: {option}
628
- dev_repo_path: {DEV_REPO}
629
- qa_repo_path: {QA_REPO or null}
630
- output_path: {output_dir}/
631
- </parameters>
632
- "
633
- )
634
- ```
635
-
636
- **Option 3 specific -- skip existing tests:**
637
-
638
- For Option 3, pass `skip_existing_test_ids: true` to the executor so it checks existing test files by test ID before generating. If a test ID already exists in the QA repo, skip generating that test case:
639
- ```
640
- <parameters>
641
- workflow_option: 3
642
- skip_existing_test_ids: true
643
- dev_repo_path: {DEV_REPO}
644
- qa_repo_path: {QA_REPO}
645
- output_path: {output_dir}/
646
- </parameters>
647
- ```
648
-
649
- **Parse executor return:**
650
-
651
- Expected return structure:
652
- ```
653
- EXECUTOR_COMPLETE:
654
- files_created: [{path, type}, ...]
655
- total_files: N
656
- commit_count: N
657
- features_covered: [...]
658
- test_case_count: N
659
- ```
660
-
661
- Capture `files_created`, `total_files`, `commit_count` for validation stage and pipeline summary.
662
-
663
- **State update -- mark generate as complete:**
664
- ```bash
665
- node bin/qaa-tools.cjs state patch --"Generate Status" complete --"Status" "Test generation complete" 2>/dev/null || true
666
- ```
667
-
668
- Print: "Generation complete. {total_files} files created across {features_covered_count} features. {commit_count} commits."
669
-
670
- </step>
671
-
672
- <step name="validate">
673
-
674
- ## Step 8: Validate Generated Tests
675
-
676
- **State update -- mark validate as running:**
677
- ```bash
678
- node bin/qaa-tools.cjs state patch --"Validate Status" running --"Status" "Validating generated tests" 2>/dev/null || true
679
- ```
680
-
681
- **Print stage banner:**
682
- ```
683
- +------------------------------------------+
684
- | STAGE 6: Validator |
685
- | Validating {total_files} test files |
686
- | Status: Running... |
687
- +------------------------------------------+
688
- ```
689
-
690
- **Spawn validator agent:**
691
- ```
692
- Agent(subagent_type="general-purpose",
693
- prompt="
694
- <objective>Run 4-layer validation on all generated test files</objective>
695
- <execution_context>@agents/qaa-validator.md</execution_context>
696
- <files_to_read>
697
- - {list all generated test files from executor return -- files_created paths}
698
- - {output_dir}/GENERATION_PLAN.md
699
- - CLAUDE.md
700
- </files_to_read>
701
- <parameters>
702
- mode: validation
703
- max_fix_loops: 3
704
- output_path: {output_dir}/VALIDATION_REPORT.md
705
- </parameters>
706
- "
707
- )
708
- ```
709
-
710
- **4-layer validation:**
711
- 1. **Syntax** -- File parses without errors
712
- 2. **Structure** -- Follows POM rules, naming conventions, locator hierarchy
713
- 3. **Dependencies** -- Imports resolve, fixtures exist, configs present
714
- 4. **Logic** -- Assertions are concrete, test IDs are unique, no assertions in page objects
715
-
716
- **5. Browser verification (if app URL available and Playwright MCP connected):**
717
-
718
- After the 4-layer static validation, use Playwright MCP to verify E2E tests against the live app:
719
-
720
- 1. Navigate to each page referenced in the E2E tests:
721
- ```
722
- mcp__playwright__browser_navigate({ url: "{app_url}/{route}" })
723
- ```
724
-
725
- 2. Take an accessibility snapshot to verify locators used in tests actually exist:
726
- ```
727
- mcp__playwright__browser_snapshot()
728
- ```
729
-
730
- 3. Cross-reference locators in generated tests against the real DOM:
731
- - Verify `data-testid` values exist on the page
732
- - Verify ARIA roles and names match test expectations
733
- - Flag any test locator that does not match a real DOM element
734
-
735
- 4. If mismatches are found, fix the test locators to match the real DOM and count as a fix loop iteration.
736
-
737
- This browser verification step prevents delivering tests with locators that will immediately fail at runtime.
738
-
739
- **Fix loop:** The validator automatically attempts to fix issues it finds. Maximum 3 fix loop iterations. After each fix attempt, re-validate.
740
-
741
- **Parse validator return:**
742
-
743
- Expected return structure:
744
- ```
745
- VALIDATOR_COMPLETE:
746
- report_path: "..."
747
- overall_status: PASS | PASS_WITH_WARNINGS | FAIL
748
- confidence: HIGH | MEDIUM | LOW
749
- layers_summary: {syntax: PASS|FAIL, structure: PASS|FAIL, dependencies: PASS|FAIL, logic: PASS|FAIL}
750
- fix_loops_used: N
751
- issues_found: N
752
- issues_fixed: N
753
- unresolved_count: N
754
- ```
755
-
756
- **RISKY CHECKPOINT -- Validator escalation:**
757
-
758
- If `unresolved_count > 0` after max fix loops (3):
759
- - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
760
- - Present unresolved issues to user with full details from VALIDATION_REPORT.md
761
- - Wait for user decision:
762
- - `"approve-with-warnings"`: Accept the validation with warnings. Set Validate Status to complete. Continue to deliver.
763
- - `"abort"`: Set Validate Status to failed. STOP PIPELINE ENTIRELY.
764
- - Manual guidance: User provides specific fix instructions. Spawn fresh continuation agent to apply fixes and re-validate.
765
-
766
- If `overall_status` is `PASS` or `PASS_WITH_WARNINGS` (and unresolved_count is 0):
767
- ```bash
768
- node bin/qaa-tools.cjs state patch --"Validate Status" complete --"Status" "Validation passed" 2>/dev/null || true
769
- ```
770
-
771
- **Verify artifact exists:**
772
- ```bash
773
- [ -f "${output_dir}/VALIDATION_REPORT.md" ] && echo "OK" || echo "MISSING: VALIDATION_REPORT.md"
774
- ```
775
-
776
- Print: "Validation complete. Status: {overall_status}. Confidence: {confidence}. {issues_found} issues found, {issues_fixed} fixed, {unresolved_count} unresolved."
777
-
778
- </step>
779
-
780
- <step name="bug_detective">
781
-
782
- ## Step 9: Bug Detective (Conditional)
783
-
784
- **Condition:** Only execute if test failures were detected during validation. Check:
785
- - `overall_status === 'FAIL'` in validator return, OR
786
- - Generated tests have runtime failures that need classification
787
-
788
- **If no failures to classify:**
789
- Print: "Skipping Bug Detective (no test failures detected)." Proceed directly to Step 10 (Deliver).
790
-
791
- **If failures need classification:**
792
-
793
- **State update:**
794
- ```bash
795
- node bin/qaa-tools.cjs state patch --"Status" "Classifying test failures" 2>/dev/null || true
796
- ```
797
-
798
- **Print stage banner:**
799
- ```
800
- +------------------------------------------+
801
- | STAGE 7: Bug Detective |
802
- | Status: Running... |
803
- +------------------------------------------+
804
- ```
805
-
806
- **Spawn bug-detective agent:**
807
- ```
808
- Agent(subagent_type="general-purpose",
809
- prompt="
810
- <objective>Classify test failures and attempt auto-fixes for test errors. Use Playwright MCP to reproduce E2E failures in the browser when available.</objective>
811
- <execution_context>@agents/qaa-bug-detective.md</execution_context>
812
- <files_to_read>
813
- - {test execution results -- from validator or direct test run}
814
- - {failing test source files -- paths from executor return}
815
- - CLAUDE.md
816
- - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
817
- - {output_dir}/research/TESTING_STACK.md (if exists)
818
- </files_to_read>
819
- <parameters>
820
- output_path: {output_dir}/FAILURE_CLASSIFICATION_REPORT.md
821
- app_url: {app_url if available}
822
- </parameters>
823
- "
824
- )
825
- ```
826
-
827
- **Parse bug-detective return:**
828
-
829
- Expected return structure:
830
- ```
831
- DETECTIVE_COMPLETE:
832
- report_path: "..."
833
- total_failures: N
834
- classification_breakdown: {app_bug: N, test_error: N, env_issue: N, inconclusive: N}
835
- auto_fixes_applied: N
836
- auto_fixes_verified: N
837
- commit_hash: "..."
838
- ```
839
-
840
- **RISKY CHECKPOINT -- Application bugs detected:**
841
-
842
- If `classification_breakdown.app_bug > 0`:
843
- - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
844
- - Present APPLICATION BUG classifications to user with full evidence from FAILURE_CLASSIFICATION_REPORT.md
845
- - These are genuine bugs in the application code discovered during test execution
846
- - The bug detective never touches application code -- it only reports
847
- - User must review and decide how to proceed:
848
- - Acknowledge bugs and continue pipeline (bugs will be in the PR description for developer attention)
849
- - Abort pipeline to fix bugs first
850
-
851
- **Verify artifact exists:**
852
- ```bash
853
- [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "OK" || echo "MISSING: FAILURE_CLASSIFICATION_REPORT.md"
854
- ```
855
-
856
- Print: "Bug Detective complete. {total_failures} failures classified: {app_bug} APP BUG, {test_error} TEST ERROR, {env_issue} ENV ISSUE, {inconclusive} INCONCLUSIVE. {auto_fixes_applied} auto-fixes applied."
857
-
858
- </step>
859
-
860
- <step name="deliver">
861
-
862
- ## Step 10: Deliver
863
-
864
- **State update -- mark deliver as running:**
865
- ```bash
866
- node bin/qaa-tools.cjs state patch --"Deliver Status" running --"Status" "Preparing delivery" 2>/dev/null || true
867
- ```
868
-
869
- **Print stage banner:**
870
- ```
871
- +------------------------------------------+
872
- | STAGE 8: Deliver |
873
- | Status: Running... |
874
- +------------------------------------------+
875
- ```
876
-
877
- ### Sub-step 1: Pre-flight checks
878
-
879
- **Check for git remote:**
880
- ```bash
881
- REMOTE_URL=$(git remote get-url origin 2>/dev/null || echo "")
882
- ```
883
-
884
- If `REMOTE_URL` is empty:
885
- - Print: "No git remote found. Artifacts committed locally but PR creation skipped."
886
- - Set `LOCAL_ONLY=true`
887
-
888
- **Check for gh CLI authentication:**
889
- ```bash
890
- gh auth status 2>/dev/null
891
- ```
892
-
893
- If `gh auth status` fails:
894
- - Print: "gh CLI not authenticated. Run 'gh auth login' first. Artifacts committed locally."
895
- - Set `LOCAL_ONLY=true`
896
-
897
- If both checks pass, set `LOCAL_ONLY=false`.
898
-
899
- ### Sub-step 2: Derive project name
900
-
901
- ```bash
902
- # Read from package.json
903
- PROJECT_NAME=$(node -e "try { const p = require('${DEV_REPO}/package.json'); console.log(p.name || ''); } catch { console.log(''); }" 2>/dev/null)
904
-
905
- # Fallback to directory basename
906
- if [ -z "$PROJECT_NAME" ]; then
907
- PROJECT_NAME=$(basename "${DEV_REPO}")
908
- fi
909
-
910
- # Sanitize for branch naming
911
- PROJECT_NAME=$(echo "$PROJECT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
912
- ```
913
-
914
- ### Sub-step 3: Detect default branch
915
-
916
- ```bash
917
- DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name' 2>/dev/null || echo "main")
918
- ```
919
-
920
- ### Sub-step 4: Create feature branch
921
-
922
- ```bash
923
- BRANCH="qa/auto-${PROJECT_NAME}-${date}"
924
-
925
- # Handle branch name collision
926
- if git rev-parse --verify "$BRANCH" 2>/dev/null || git rev-parse --verify "origin/$BRANCH" 2>/dev/null; then
927
- SUFFIX=2
928
- while git rev-parse --verify "${BRANCH}-${SUFFIX}" 2>/dev/null || git rev-parse --verify "origin/${BRANCH}-${SUFFIX}" 2>/dev/null; do
929
- SUFFIX=$((SUFFIX + 1))
930
- done
931
- BRANCH="${BRANCH}-${SUFFIX}"
932
- fi
933
-
934
- git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
935
- ```
936
-
937
- ### Sub-step 5: Per-stage atomic commits
938
-
939
- For each pipeline stage that produced artifacts, commit using `qaa-tools.cjs commit`. Check file existence before each commit.
940
-
941
- **Scanner:**
942
- ```bash
943
- if [ -f "${output_dir}/SCAN_MANIFEST.md" ]; then
944
- node bin/qaa-tools.cjs commit "qa(scanner): produce SCAN_MANIFEST.md for ${PROJECT_NAME}" --files ${output_dir}/SCAN_MANIFEST.md
945
- fi
946
- ```
947
-
948
- **Analyzer (Option 1):**
949
- ```bash
950
- if [ -f "${output_dir}/QA_ANALYSIS.md" ]; then
951
- ANALYZER_FILES="${output_dir}/QA_ANALYSIS.md ${output_dir}/TEST_INVENTORY.md"
952
- [ -f "${output_dir}/QA_REPO_BLUEPRINT.md" ] && ANALYZER_FILES="${ANALYZER_FILES} ${output_dir}/QA_REPO_BLUEPRINT.md"
953
- node bin/qaa-tools.cjs commit "qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md" --files ${ANALYZER_FILES}
954
- fi
955
- ```
956
-
957
- **Analyzer (Options 2/3):**
958
- ```bash
959
- if [ -f "${output_dir}/GAP_ANALYSIS.md" ]; then
960
- node bin/qaa-tools.cjs commit "qa(analyzer): produce GAP_ANALYSIS.md" --files ${output_dir}/GAP_ANALYSIS.md
961
- fi
962
- ```
963
-
964
- **TestID Injector (if ran):**
965
- ```bash
966
- if [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ]; then
967
- node bin/qaa-tools.cjs commit "qa(testid-injector): inject ${elements_injected} data-testid attributes across ${components_modified} components" --files ${output_dir}/TESTID_AUDIT_REPORT.md ${modified_source_files}
968
- fi
969
- ```
970
-
971
- **Executor:**
972
- ```bash
973
- if [ -n "${generated_file_paths}" ]; then
974
- node bin/qaa-tools.cjs commit "qa(executor): generate ${total_files} test files with POMs and fixtures" --files ${generated_file_paths}
975
- fi
976
- ```
977
-
978
- **Planner:**
979
- ```bash
980
- if [ -f "${output_dir}/GENERATION_PLAN.md" ]; then
981
- node bin/qaa-tools.cjs commit "qa(planner): produce GENERATION_PLAN.md" --files ${output_dir}/GENERATION_PLAN.md
982
- fi
983
- ```
984
-
985
- **Validator:**
986
- ```bash
987
- if [ -f "${output_dir}/VALIDATION_REPORT.md" ]; then
988
- node bin/qaa-tools.cjs commit "qa(validator): validate generated tests - ${overall_status} with ${confidence} confidence" --files ${output_dir}/VALIDATION_REPORT.md
989
- fi
990
- ```
991
-
992
- **Bug Detective (if ran):**
993
- ```bash
994
- if [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ]; then
995
- node bin/qaa-tools.cjs commit "qa(bug-detective): classify ${total_failures} failures - ${classification_summary}" --files ${output_dir}/FAILURE_CLASSIFICATION_REPORT.md
996
- fi
997
- ```
998
-
999
- ### Sub-step 6: Push branch
1000
-
1001
- If `LOCAL_ONLY` is true, skip this sub-step.
1002
-
1003
- ```bash
1004
- git push -u origin "$BRANCH"
1005
- ```
1006
-
1007
- If push fails:
1008
- - Print: "Push failed: {error_message}. Artifacts committed locally on branch ${BRANCH}."
1009
- - Set `LOCAL_ONLY=true`
1010
-
1011
- ### Sub-step 7: Build PR body
1012
-
1013
- If `LOCAL_ONLY` is true, skip this sub-step.
1014
-
1015
- Read the PR template:
1016
- ```bash
1017
- PR_BODY=$(cat templates/pr-template.md)
1018
- ```
1019
-
1020
- Replace all `{placeholder}` tokens with actual values collected during pipeline execution:
1021
- - `{architecture_type}` -- from QA_ANALYSIS.md or SCAN_MANIFEST.md
1022
- - `{framework}` -- detected test framework
1023
- - `{risk_summary}` -- risk assessment counts (e.g., "3 HIGH, 5 MEDIUM, 2 LOW")
1024
- - `{unit_count}` -- from pyramid_breakdown.unit
1025
- - `{integration_count}` -- from pyramid_breakdown.integration
1026
- - `{api_count}` -- from pyramid_breakdown.api
1027
- - `{e2e_count}` -- from pyramid_breakdown.e2e
1028
- - `{total_count}` -- from total_test_count
1029
- - `{modules_covered}` -- count of modules with tests
1030
- - `{coverage_estimate}` -- estimated coverage percentage
1031
- - `{validation_result}` -- PASS, PASS_WITH_WARNINGS, or FAIL
1032
- - `{confidence}` -- HIGH, MEDIUM, or LOW
1033
- - `{fix_loops_used}` -- number 0-3
1034
- - `{issues_found}` -- total issues found during validation
1035
- - `{issues_fixed}` -- total issues auto-fixed
1036
- - `{file_list}` -- if total files <= 50, list each file; if > 50, use summary
1037
-
1038
- ### Sub-step 8: Create draft PR
1039
-
1040
- If `LOCAL_ONLY` is true, skip this sub-step.
1041
-
1042
- ```bash
1043
- PR_URL=$(gh pr create \
1044
- --draft \
1045
- --title "qa: automated test suite for ${PROJECT_NAME}" \
1046
- --body "${PR_BODY}" \
1047
- --label "qa-automation" \
1048
- --label "auto-generated" \
1049
- --assignee "@me" 2>&1)
1050
- ```
1051
-
1052
- Do NOT pass `--base` flag. Let gh auto-detect the default branch.
1053
-
1054
- On failure:
1055
- - Print: "PR creation failed: ${PR_URL}. Artifacts remain on branch ${BRANCH}."
1056
- - Do NOT stop the pipeline -- artifacts are committed and pushed.
1057
-
1058
- ### Sub-step 9: Print result
1059
-
1060
- If PR was created successfully:
1061
- ```
1062
- PR created: ${PR_URL}
1063
- ```
1064
-
1065
- If `LOCAL_ONLY` is true:
1066
- ```
1067
- PR: not created (local-only mode). Artifacts committed on branch: ${BRANCH}
1068
- ```
1069
-
1070
- **State update -- mark deliver as complete:**
1071
- ```bash
1072
- node bin/qaa-tools.cjs state patch --"Deliver Status" complete --"Status" "Pipeline complete" 2>/dev/null || true
1073
- ```
1074
-
1075
- **Clear auto-chain flag at pipeline completion:**
1076
- ```bash
1077
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1078
- ```
1079
-
1080
- ### Print pipeline summary banner:
1081
-
1082
- ```
1083
- ======================================================
1084
- QA PIPELINE COMPLETE
1085
- ======================================================
1086
-
1087
- Option: {option} ({option_description})
1088
- Repository: {DEV_REPO}
1089
- QA Repo: {QA_REPO or 'N/A'}
1090
- Maturity Score: {maturity_score or 'N/A'}
1091
-
1092
- Stages Completed:
1093
- [{check}] Scan -- {scan_result}
1094
- [{check}] Analyze -- {analyze_result} ({test_count} test cases)
1095
- [{check}] TestID Inject -- {inject_result or 'skipped'}
1096
- [{check}] Plan -- {plan_result} ({file_count} files planned)
1097
- [{check}] Generate -- {generate_result} ({files_created} files created)
1098
- [{check}] Validate -- {validate_result} ({confidence} confidence)
1099
- [{check}] Bug Detective -- {detective_result or 'skipped'}
1100
- [{check}] Deliver -- {deliver_result}
1101
-
1102
- PR: {pr_url or 'not created (local-only)'}
1103
-
1104
- Artifacts:
1105
- {list all produced .md files in output_dir}
1106
-
1107
- Total Time: {total_duration}
1108
- ======================================================
1109
- ```
1110
-
1111
- Where: `[x]` = completed, `[ ]` = skipped, `[!]` = failed.
1112
-
1113
- </step>
1114
-
1115
- </process>
1116
-
1117
- <error_handling>
1118
-
1119
- ## Error Handling
1120
-
1121
- ### Stage Failure Protocol
1122
-
1123
- When any agent returns a failure or error:
1124
-
1125
- 1. **Set stage status to failed:**
1126
- ```bash
1127
- node bin/qaa-tools.cjs state patch --"{Stage} Status" failed --"Status" "Pipeline stopped: {Stage} failed - {reason}" 2>/dev/null || true
1128
- ```
1129
-
1130
- 2. **Print failure banner:**
1131
- ```
1132
- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1133
- ! PIPELINE STOPPED !
1134
- ! Stage: {stage_name} !
1135
- ! Reason: {failure_reason} !
1136
- ! !
1137
- ! Completed: {completed_stages} !
1138
- ! Artifacts: {artifacts_so_far} !
1139
- ! !
1140
- ! Action required: Review and re-run !
1141
- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1142
- ```
1143
-
1144
- 3. **DO NOT continue to next stage.** The pipeline stops entirely at the failed stage.
1145
-
1146
- 4. **DO NOT create partial PR.** No branch, no commit, no PR with incomplete results.
1147
-
1148
- 5. **Preserve all artifacts produced so far.** They remain on disk in `{output_dir}/` for debugging.
1149
-
1150
- ### Artifact Verification
1151
-
1152
- After EVERY agent spawn, before advancing to next stage, verify the expected output artifact exists on disk:
1153
- ```bash
1154
- [ -f "{expected_artifact_path}" ] && echo "OK" || echo "MISSING"
1155
- ```
1156
-
1157
- If artifacts are missing, treat as stage failure and STOP pipeline.
1158
-
1159
- ### qaa-tools.cjs Graceful Fallback
1160
-
1161
- All `node bin/qaa-tools.cjs` calls use `2>/dev/null || true` to handle cases where the tool is not installed or not found. The pipeline must not break due to missing state management tooling -- it logs a warning and continues.
1162
-
1163
- </error_handling>
1164
-
1165
- <auto_advance>
1166
-
1167
- ## Auto-Advance Mode
1168
-
1169
- Auto-advance is enabled when ANY of these is true:
1170
- - `--auto` flag passed to the `/qa-start` invocation
1171
- - `config.json` has `workflow.auto_advance = true` (persistent user preference)
1172
- - `workflow._auto_chain_active = true` in config (ephemeral chain flag from ongoing auto run)
1173
-
1174
- ### Behavior in Auto Mode
1175
-
1176
- **SAFE checkpoints are auto-approved.** The pipeline continues without pausing. A log message records the auto-approval:
1177
- ```
1178
- Auto-approved: {checkpoint_description}
1179
- ```
1180
-
1181
- **RISKY checkpoints ALWAYS pause.** Even in auto mode, the pipeline stops and presents the checkpoint to the user.
1182
-
1183
- ### Safe vs Risky Checkpoint Classification
1184
-
1185
- **SAFE (auto-approve in auto mode):**
1186
-
1187
- | Checkpoint | Agent | Auto-Action |
1188
- |------------|-------|-------------|
1189
- | Framework detection uncertain (LOW confidence) | Scanner | Approve with most likely framework |
1190
- | Analyzer assumptions review | Analyzer | Approve all assumptions |
1191
- | TestID audit review | TestID Injector | Approve P0-only injection |
1192
-
1193
- **RISKY (ALWAYS pause, even in auto mode):**
1194
-
1195
- | Checkpoint | Agent | User Action Required |
1196
- |------------|-------|---------------------|
1197
- | Validator escalation (unresolved issues after 3 fix loops) | Validator | approve-with-warnings, abort, or fix guidance |
1198
- | APPLICATION BUG classification | Bug Detective | Review bugs, continue or fix first |
1199
- | Any checkpoint with "unresolved" or "failed" blocking text | Any | Review specific blocking issue |
1200
-
1201
- ### Checkpoint Handling Flow
1202
-
1203
- ```
1204
- On agent return with checkpoint data:
1205
- 1. Extract checkpoint blocking field content
1206
- 2. Classify as SAFE or RISKY:
1207
- - "framework detection" -> SAFE
1208
- - "assumptions" -> SAFE
1209
- - "audit" or "data-testid" -> SAFE
1210
- - "unresolved" -> RISKY
1211
- - "failed" -> RISKY
1212
- - "APPLICATION BUG" -> RISKY
1213
- - Default (no pattern match) -> RISKY (conservative)
1214
- 3. If IS_AUTO and SAFE:
1215
- - Auto-approve with default action
1216
- - Log the auto-approval
1217
- - Continue pipeline
1218
- 4. If IS_AUTO and RISKY:
1219
- - PAUSE pipeline
1220
- - Print checkpoint details with full context
1221
- - Wait for user input
1222
- 5. If NOT auto (manual mode):
1223
- - PAUSE pipeline
1224
- - Print checkpoint details
1225
- - Wait for user input
1226
- ```
1227
-
1228
- ### Resume After Checkpoint
1229
-
1230
- When resuming after a checkpoint, spawn a FRESH agent with explicit state:
1231
- ```
1232
- Agent(subagent_type="general-purpose",
1233
- prompt="
1234
- <objective>Continue QA pipeline from {stage} stage</objective>
1235
- <execution_context>@agents/qa-pipeline-orchestrator.md</execution_context>
1236
- <resume_context>
1237
- Pipeline state:
1238
- - Completed stages: {list of completed stages with their results}
1239
- - Current stage: {stage that triggered checkpoint}
1240
- - Checkpoint response: {user's response or decision}
1241
- - Artifacts produced so far: {list of files with paths}
1242
-
1243
- Resume from: {exact step in pipeline to resume from}
1244
- User decision: {what user chose at checkpoint}
1245
- </resume_context>
1246
- "
1247
- )
1248
- ```
1249
-
1250
- ### Stale Chain Flag Protection
1251
-
1252
- At orchestrator init, if `--auto` was NOT passed AND `auto_advance` config is false:
1253
- ```bash
1254
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1255
- ```
1256
-
1257
- At pipeline completion (success or failure):
1258
- ```bash
1259
- node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1260
- ```
1261
-
1262
- </auto_advance>
1
+ <purpose>
2
+
3
+ Orchestrate the full QA automation pipeline: scan -> analyze -> [testid-inject if frontend] -> plan -> generate -> validate -> [bug-detective if failures] -> deliver. Detects workflow option (1/2/3) from arguments, spawns specialized agents for each stage, manages state transitions, handles checkpoints (safe auto-approve, risky always pause), and delivers a draft PR with per-stage atomic commits.
4
+
5
+ Invoked by the `/qa-start` slash command. Accepts `--dev-repo`, `--qa-repo`, and `--auto` flags.
6
+
7
+ </purpose>
8
+
9
+ <required_reading>
10
+
11
+ Read these files BEFORE executing any pipeline stage. Do NOT skip.
12
+
13
+ - **CLAUDE.md** -- Agent pipeline stages, module boundaries, quality gates, stage transitions, auto-advance rules, agent coordination, data-testid convention. Read the full file.
14
+ - **agents/qa-pipeline-orchestrator.md** -- Full orchestrator logic, checkpoint classification, error handling, delivery sub-steps.
15
+
16
+ </required_reading>
17
+
18
+ <process>
19
+
20
+ <step name="initialize" priority="first">
21
+
22
+ ## Step 1: Initialize Pipeline
23
+
24
+ Parse `$ARGUMENTS` for flags:
25
+
26
+ ```bash
27
+ DEV_REPO=""
28
+ QA_REPO=""
29
+ IS_AUTO=false
30
+
31
+ # Parse --dev-repo flag
32
+ if echo "$ARGUMENTS" | grep -qE '\-\-dev-repo'; then
33
+ DEV_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-dev-repo\s+[^\s]+' | awk '{print $2}')
34
+ fi
35
+
36
+ # Parse --qa-repo flag
37
+ if echo "$ARGUMENTS" | grep -qE '\-\-qa-repo'; then
38
+ QA_REPO=$(echo "$ARGUMENTS" | grep -oE '\-\-qa-repo\s+[^\s]+' | awk '{print $2}')
39
+ fi
40
+
41
+ # Parse --auto flag
42
+ if echo "$ARGUMENTS" | grep -qE '\-\-auto'; then
43
+ IS_AUTO=true
44
+ fi
45
+ ```
46
+
47
+ **If no --dev-repo provided**, use the current working directory:
48
+ ```bash
49
+ if [ -z "$DEV_REPO" ]; then
50
+ DEV_REPO=$(pwd)
51
+ fi
52
+ ```
53
+
54
+ **Attempt to call qaa-tools init** (handle missing tool gracefully):
55
+ ```bash
56
+ INIT_JSON=$(node bin/qaa-tools.cjs init qa-start 2>/dev/null || echo "")
57
+ ```
58
+
59
+ If `INIT_JSON` is empty or the command fails, proceed with manual initialization:
60
+ - Set `output_dir` to `.qa-output`
61
+ - Set `date` to current date in `YYYY-MM-DD` format
62
+ - Create output directory: `mkdir -p "$output_dir"`
63
+
64
+ If `INIT_JSON` is valid, parse it for: `option`, `dev_repo_path`, `qa_repo_path`, `maturity_score`, `maturity_note`, `output_dir`, `date`, agent model assignments, `auto_advance`, `auto_chain_active`, `parallelization`, `commit_docs`.
65
+
66
+ **Detect workflow option based on inputs:**
67
+
68
+ - If `QA_REPO` is empty (no --qa-repo flag): **Option 1** (Dev-Only -- Full Pipeline)
69
+ - If `QA_REPO` is provided: Assess QA repo maturity
70
+ - Check for existing test files, configs, coverage reports in QA repo
71
+ - Count test files, evaluate framework setup, check for CI config
72
+ - Score 0-100 based on test count, framework config presence, CI setup, coverage data
73
+ - Score >= 60: **Option 3** (Dev + Mature QA -- Surgical)
74
+ - Score < 60: **Option 2** (Dev + Immature QA -- Gap-Fill)
75
+
76
+ **Determine auto-advance mode:**
77
+ ```bash
78
+ # Check persistent config flag
79
+ AUTO_CFG=$(node bin/qaa-tools.cjs config-get workflow.auto_advance 2>/dev/null || echo "false")
80
+ AUTO_CHAIN=$(node bin/qaa-tools.cjs config-get workflow._auto_chain_active 2>/dev/null || echo "false")
81
+
82
+ if [ "$IS_AUTO" = "true" ] || [ "$AUTO_CFG" = "true" ] || [ "$AUTO_CHAIN" = "true" ]; then
83
+ IS_AUTO=true
84
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active true 2>/dev/null || true
85
+ fi
86
+
87
+ # Safety: clear stale chain flag if NOT in auto mode
88
+ if [ "$IS_AUTO" = "false" ]; then
89
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
90
+ fi
91
+ ```
92
+
93
+ **Print initialization banner:**
94
+ ```
95
+ === QA Pipeline Orchestrator ===
96
+ Option: {option} ({description})
97
+ Dev Repo: {DEV_REPO}
98
+ QA Repo: {QA_REPO or 'N/A'}
99
+ Maturity Score: {maturity_score or 'N/A'}
100
+ Auto-Advance: {IS_AUTO}
101
+ Date: {date}
102
+ ================================
103
+ ```
104
+
105
+ Where `{description}` is:
106
+ - Option 1: "Dev-Only -- Full Pipeline"
107
+ - Option 2: "Dev + Immature QA -- Gap-Fill"
108
+ - Option 3: "Dev + Mature QA -- Surgical"
109
+
110
+ </step>
111
+
112
+ <step name="detect_framework">
113
+
114
+ ## Step 2: Detect Framework
115
+
116
+ Before scanning, detect the project's language and test framework to guide all downstream agents.
117
+
118
+ **Read project config files:**
119
+ ```bash
120
+ # Check for Node.js / JavaScript / TypeScript
121
+ [ -f "${DEV_REPO}/package.json" ] && cat "${DEV_REPO}/package.json"
122
+
123
+ # Check for Python
124
+ [ -f "${DEV_REPO}/requirements.txt" ] && cat "${DEV_REPO}/requirements.txt"
125
+ [ -f "${DEV_REPO}/pyproject.toml" ] && cat "${DEV_REPO}/pyproject.toml"
126
+ [ -f "${DEV_REPO}/setup.py" ] && cat "${DEV_REPO}/setup.py"
127
+
128
+ # Check for .NET
129
+ ls "${DEV_REPO}"/*.csproj 2>/dev/null
130
+ ls "${DEV_REPO}"/**/*.csproj 2>/dev/null
131
+
132
+ # Check for Java
133
+ [ -f "${DEV_REPO}/pom.xml" ] && echo "Maven project"
134
+ [ -f "${DEV_REPO}/build.gradle" ] && echo "Gradle project"
135
+ ```
136
+
137
+ **Detect test framework from config files:**
138
+ ```bash
139
+ # JavaScript/TypeScript ecosystem
140
+ [ -f "${DEV_REPO}/cypress.config.ts" ] || [ -f "${DEV_REPO}/cypress.config.js" ] && echo "FRAMEWORK=cypress"
141
+ [ -f "${DEV_REPO}/playwright.config.ts" ] || [ -f "${DEV_REPO}/playwright.config.js" ] && echo "FRAMEWORK=playwright"
142
+ [ -f "${DEV_REPO}/jest.config.ts" ] || [ -f "${DEV_REPO}/jest.config.js" ] && echo "FRAMEWORK=jest"
143
+ [ -f "${DEV_REPO}/vitest.config.ts" ] || [ -f "${DEV_REPO}/vitest.config.js" ] && echo "FRAMEWORK=vitest"
144
+
145
+ # Python ecosystem
146
+ [ -f "${DEV_REPO}/pytest.ini" ] || [ -f "${DEV_REPO}/conftest.py" ] && echo "FRAMEWORK=pytest"
147
+
148
+ # Check package.json devDependencies for test frameworks
149
+ node -e "
150
+ try {
151
+ const pkg = require('${DEV_REPO}/package.json');
152
+ const deps = {...(pkg.devDependencies||{}), ...(pkg.dependencies||{})};
153
+ const frameworks = [];
154
+ if (deps.cypress) frameworks.push('cypress');
155
+ if (deps['@playwright/test'] || deps.playwright) frameworks.push('playwright');
156
+ if (deps.jest) frameworks.push('jest');
157
+ if (deps.vitest) frameworks.push('vitest');
158
+ if (deps.mocha) frameworks.push('mocha');
159
+ console.log(frameworks.join(',') || 'none');
160
+ } catch { console.log('no-package-json'); }
161
+ " 2>/dev/null
162
+ ```
163
+
164
+ **Assess detection confidence:**
165
+ - **HIGH**: Config file found AND matching dependency in package.json/requirements.txt
166
+ - **MEDIUM**: Only dependency found (no config file) OR only config file (no dependency)
167
+ - **LOW**: No test framework detected, or conflicting signals
168
+
169
+ **If no test framework found:**
170
+ - If `IS_AUTO` is false: Ask the user which framework to use. STOP and wait for response.
171
+ - If `IS_AUTO` is true: Select the most appropriate framework based on the project type:
172
+ - React/Next.js/Vue/Angular frontend -> Playwright
173
+ - Node.js API -> Jest or Vitest (prefer Vitest if ESM)
174
+ - Python -> Pytest
175
+ - Log: "Auto-selected: {framework} (no existing test framework detected)"
176
+
177
+ **If detection confidence is LOW:**
178
+ - If `IS_AUTO` is true: Auto-approve with most likely framework (SAFE checkpoint). Log: "Auto-approved: Framework detection (LOW confidence, selected {framework})". Continue.
179
+ - If `IS_AUTO` is false: Present detection details to user. Wait for confirmation before proceeding.
180
+
181
+ Store detected framework, language, and confidence for all downstream agents.
182
+
183
+ </step>
184
+
185
+ <step name="scan">
186
+
187
+ ## Step 3: Scan Repository
188
+
189
+ **State update -- mark scan as running:**
190
+ ```bash
191
+ node bin/qaa-tools.cjs state patch --"Scan Status" running --"Status" "Scanning repository" 2>/dev/null || true
192
+ ```
193
+
194
+ **Print stage banner:**
195
+ ```
196
+ +------------------------------------------+
197
+ | STAGE 1: Scanner |
198
+ | Status: Running... |
199
+ +------------------------------------------+
200
+ ```
201
+
202
+ **Spawn scanner agent:**
203
+
204
+ For **Option 1** (scan dev repo only):
205
+ ```
206
+ Agent(subagent_type="general-purpose",
207
+ prompt="
208
+ <objective>Scan repository and produce SCAN_MANIFEST.md</objective>
209
+ <execution_context>@agents/qaa-scanner.md</execution_context>
210
+ <files_to_read>
211
+ - CLAUDE.md
212
+ </files_to_read>
213
+ <parameters>
214
+ dev_repo_path: {DEV_REPO}
215
+ qa_repo_path: null
216
+ output_path: {output_dir}/SCAN_MANIFEST.md
217
+ </parameters>
218
+ "
219
+ )
220
+ ```
221
+
222
+ For **Options 2 and 3** (scan both repos):
223
+ ```
224
+ Agent(subagent_type="general-purpose",
225
+ prompt="
226
+ <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md</objective>
227
+ <execution_context>@agents/qaa-scanner.md</execution_context>
228
+ <files_to_read>
229
+ - CLAUDE.md
230
+ </files_to_read>
231
+ <parameters>
232
+ dev_repo_path: {DEV_REPO}
233
+ qa_repo_path: {QA_REPO}
234
+ output_path: {output_dir}/SCAN_MANIFEST.md
235
+ </parameters>
236
+ "
237
+ )
238
+ ```
239
+
240
+ **Parse scanner return:**
241
+
242
+ Expected return structure:
243
+ ```
244
+ SCANNER_COMPLETE:
245
+ file_path: ".qa-output/SCAN_MANIFEST.md"
246
+ decision: PROCEED | STOP
247
+ has_frontend: true | false
248
+ detection_confidence: HIGH | MEDIUM | LOW
249
+ ```
250
+
251
+ **Handle decision field:**
252
+ - If `decision` is `STOP`:
253
+ ```bash
254
+ node bin/qaa-tools.cjs state patch --"Scan Status" failed --"Status" "Pipeline stopped: Scanner returned STOP" 2>/dev/null || true
255
+ ```
256
+ Print failure banner and STOP PIPELINE ENTIRELY. Do NOT proceed to any further stage.
257
+
258
+ - If `decision` is `PROCEED`:
259
+ ```bash
260
+ node bin/qaa-tools.cjs state patch --"Scan Status" complete 2>/dev/null || true
261
+ ```
262
+ Capture `has_frontend` for testid-injector conditional (Step 5).
263
+ Capture `detection_confidence` for checkpoint handling.
264
+
265
+ **Verify artifact exists before continuing:**
266
+ ```bash
267
+ [ -f "${output_dir}/SCAN_MANIFEST.md" ] && echo "OK: SCAN_MANIFEST.md exists" || echo "MISSING: SCAN_MANIFEST.md"
268
+ ```
269
+
270
+ If SCAN_MANIFEST.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
271
+
272
+ </step>
273
+
274
+ <step name="codebase_map">
275
+
276
+ ## Step 3b: Codebase Map (Deep Analysis)
277
+
278
+ **State update -- mark codebase-map as running:**
279
+ ```bash
280
+ node bin/qaa-tools.cjs state patch --"Map Status" running --"Status" "Mapping codebase" 2>/dev/null || true
281
+ ```
282
+
283
+ **Print stage banner:**
284
+ ```
285
+ +------------------------------------------+
286
+ | STAGE 1b: Codebase Mapper |
287
+ | Status: Running 4 parallel sub-agents...|
288
+ +------------------------------------------+
289
+ ```
290
+
291
+ **Create output directory:**
292
+ ```bash
293
+ mkdir -p ${output_dir}/codebase
294
+ ```
295
+
296
+ **Spawn 4 parallel sub-agents** (one per focus area). Issue all 4 `Agent()` calls in a single message for parallel execution:
297
+
298
+ **Sub-agent 1 - testability focus** (produces TESTABILITY.md + TEST_SURFACE.md):
299
+ ```
300
+ Agent(subagent_type="general-purpose",
301
+ prompt="
302
+ <objective>Codebase analysis - testability focus. Produce TESTABILITY.md and TEST_SURFACE.md mapping what is testable in this codebase, mock boundaries, and exhaustive list of testable entry points.</objective>
303
+ <execution_context>@agents/qaa-codebase-mapper.md</execution_context>
304
+ <files_to_read>
305
+ - CLAUDE.md
306
+ - ${output_dir}/SCAN_MANIFEST.md
307
+ </files_to_read>
308
+ <parameters>
309
+ focus_area: testability
310
+ dev_repo_path: {DEV_REPO}
311
+ output_dir: ${output_dir}/codebase
312
+ </parameters>
313
+ "
314
+ )
315
+ ```
316
+
317
+ **Sub-agent 2 - risk focus** (produces RISK_MAP.md + CRITICAL_PATHS.md):
318
+ ```
319
+ Agent(subagent_type="general-purpose",
320
+ prompt="
321
+ <objective>Codebase analysis - risk focus. Produce RISK_MAP.md and CRITICAL_PATHS.md identifying business-critical paths, security-sensitive areas, and the exact user flows that E2E smoke tests must cover.</objective>
322
+ <execution_context>@agents/qaa-codebase-mapper.md</execution_context>
323
+ <files_to_read>
324
+ - CLAUDE.md
325
+ - ${output_dir}/SCAN_MANIFEST.md
326
+ </files_to_read>
327
+ <parameters>
328
+ focus_area: risk
329
+ dev_repo_path: {DEV_REPO}
330
+ output_dir: ${output_dir}/codebase
331
+ </parameters>
332
+ "
333
+ )
334
+ ```
335
+
336
+ **Sub-agent 3 - patterns focus** (produces CODE_PATTERNS.md + API_CONTRACTS.md):
337
+ ```
338
+ Agent(subagent_type="general-purpose",
339
+ prompt="
340
+ <objective>Codebase analysis - patterns focus. Produce CODE_PATTERNS.md and API_CONTRACTS.md documenting naming conventions, import style, and exact request/response shapes for API tests.</objective>
341
+ <execution_context>@agents/qaa-codebase-mapper.md</execution_context>
342
+ <files_to_read>
343
+ - CLAUDE.md
344
+ - ${output_dir}/SCAN_MANIFEST.md
345
+ </files_to_read>
346
+ <parameters>
347
+ focus_area: patterns
348
+ dev_repo_path: {DEV_REPO}
349
+ output_dir: ${output_dir}/codebase
350
+ </parameters>
351
+ "
352
+ )
353
+ ```
354
+
355
+ **Sub-agent 4 - existing-tests focus** (produces TEST_ASSESSMENT.md + COVERAGE_GAPS.md):
356
+ ```
357
+ Agent(subagent_type="general-purpose",
358
+ prompt="
359
+ <objective>Codebase analysis - existing-tests focus. Produce TEST_ASSESSMENT.md and COVERAGE_GAPS.md assessing current test quality, frameworks in use, and identifying which modules/functions/paths have no test coverage.</objective>
360
+ <execution_context>@agents/qaa-codebase-mapper.md</execution_context>
361
+ <files_to_read>
362
+ - CLAUDE.md
363
+ - ${output_dir}/SCAN_MANIFEST.md
364
+ </files_to_read>
365
+ <parameters>
366
+ focus_area: existing-tests
367
+ dev_repo_path: {DEV_REPO}
368
+ output_dir: ${output_dir}/codebase
369
+ </parameters>
370
+ "
371
+ )
372
+ ```
373
+
374
+ **Verify codebase docs exist:**
375
+ ```bash
376
+ docs_count=$(ls ${output_dir}/codebase/*.md 2>/dev/null | wc -l)
377
+ echo "Codebase docs produced: $docs_count of 8 expected"
378
+
379
+ if [ "$docs_count" -eq 0 ]; then
380
+ echo "FAIL: No codebase docs produced. Pipeline continues with degraded context."
381
+ node bin/qaa-tools.cjs state patch --"Map Status" failed 2>/dev/null || true
382
+ elif [ "$docs_count" -lt 4 ]; then
383
+ echo "WARNING: Fewer than 4 codebase docs produced. Downstream agents will have limited context."
384
+ node bin/qaa-tools.cjs state patch --"Map Status" "partial" 2>/dev/null || true
385
+ else
386
+ echo "OK: codebase map sufficient ($docs_count docs)"
387
+ node bin/qaa-tools.cjs state patch --"Map Status" complete 2>/dev/null || true
388
+ fi
389
+ ```
390
+
391
+ **Codebase map is non-blocking:** if zero docs are produced, the pipeline continues with degraded context (downstream agents fall back to whatever they can find). All `<files_to_read>` references to codebase docs in subsequent stages use `(if exists)` semantics.
392
+
393
+ </step>
394
+
395
+ <step name="research">
396
+
397
+ ## Step 3c: Research Testing Ecosystem
398
+
399
+ **State update -- mark research as running:**
400
+ ```bash
401
+ node bin/qaa-tools.cjs state patch --"Research Status" running --"Status" "Researching testing ecosystem" 2>/dev/null || true
402
+ ```
403
+
404
+ **Print stage banner:**
405
+ ```
406
+ +------------------------------------------+
407
+ | STAGE 1c: Project Researcher |
408
+ | Status: Running... |
409
+ +------------------------------------------+
410
+ ```
411
+
412
+ **Create output directory:**
413
+ ```bash
414
+ mkdir -p ${output_dir}/research
415
+ ```
416
+
417
+ **Spawn researcher agent:**
418
+ ```
419
+ Agent(subagent_type="general-purpose",
420
+ prompt="
421
+ <objective>Research the testing ecosystem for this project. Use the codebase map findings (RISK_MAP, CODE_PATTERNS, API_CONTRACTS, TESTABILITY, etc., from ${output_dir}/codebase/) to ground your Context7 queries to the project specifics — query for the libraries/patterns the project actually uses (e.g., MSW integration, Stripe testing, OpenAPI contract testing) instead of generic framework questions. Use Context7 MCP as the primary source. Produce research documents consumed by downstream agents.</objective>
422
+ <execution_context>@agents/qaa-project-researcher.md</execution_context>
423
+ <files_to_read>
424
+ - CLAUDE.md
425
+ - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
426
+ - ${output_dir}/SCAN_MANIFEST.md
427
+ - ${output_dir}/codebase/TESTABILITY.md (if exists)
428
+ - ${output_dir}/codebase/TEST_SURFACE.md (if exists)
429
+ - ${output_dir}/codebase/RISK_MAP.md (if exists)
430
+ - ${output_dir}/codebase/CRITICAL_PATHS.md (if exists)
431
+ - ${output_dir}/codebase/CODE_PATTERNS.md (if exists)
432
+ - ${output_dir}/codebase/API_CONTRACTS.md (if exists)
433
+ - ${output_dir}/codebase/TEST_ASSESSMENT.md (if exists)
434
+ - ${output_dir}/codebase/COVERAGE_GAPS.md (if exists)
435
+ </files_to_read>
436
+ <parameters>
437
+ mode: stack-testing
438
+ dev_repo_path: {DEV_REPO}
439
+ output_dir: ${output_dir}/research
440
+ </parameters>
441
+ "
442
+ )
443
+ ```
444
+
445
+ **Verify research artifacts exist:**
446
+ ```bash
447
+ ls ${output_dir}/research/*.md 2>/dev/null && echo "OK: Research files produced" || echo "WARNING: No research files produced"
448
+ ```
449
+
450
+ **Research is non-blocking:** If the researcher fails or produces no output, the pipeline continues — downstream agents will fall back to Context7 queries directly. Log the warning but do NOT stop the pipeline.
451
+
452
+ ```bash
453
+ if [ ! -f "${output_dir}/research/TESTING_STACK.md" ]; then
454
+ echo "WARNING: Research stage produced no output. Downstream agents will query Context7 directly."
455
+ node bin/qaa-tools.cjs state patch --"Research Status" "skipped (no output)" 2>/dev/null || true
456
+ else
457
+ node bin/qaa-tools.cjs state patch --"Research Status" complete 2>/dev/null || true
458
+ fi
459
+ ```
460
+
461
+ </step>
462
+
463
+ <step name="analyze">
464
+
465
+ ## Step 4: Analyze Repository
466
+
467
+ **State update -- mark analyze as running:**
468
+ ```bash
469
+ node bin/qaa-tools.cjs state patch --"Analyze Status" running --"Status" "Analyzing repository" 2>/dev/null || true
470
+ ```
471
+
472
+ **Print stage banner:**
473
+ ```
474
+ +------------------------------------------+
475
+ | STAGE 2: Analyzer |
476
+ | Status: Running... |
477
+ +------------------------------------------+
478
+ ```
479
+
480
+ **Determine analyzer mode based on option:**
481
+ - Option 1: `mode = 'full'` (produces QA_ANALYSIS.md + TEST_INVENTORY.md + QA_REPO_BLUEPRINT.md)
482
+ - Options 2 and 3: `mode = 'gap'` (produces GAP_ANALYSIS.md)
483
+
484
+ **Spawn analyzer agent:**
485
+ ```
486
+ Agent(subagent_type="general-purpose",
487
+ prompt="
488
+ <objective>Analyze scanned repository and produce analysis artifacts</objective>
489
+ <execution_context>@agents/qaa-analyzer.md</execution_context>
490
+ <files_to_read>
491
+ - {output_dir}/SCAN_MANIFEST.md
492
+ - CLAUDE.md
493
+ - {output_dir}/research/TESTING_STACK.md (if exists)
494
+ - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
495
+ - {output_dir}/codebase/RISK_MAP.md (if exists)
496
+ - {output_dir}/codebase/CRITICAL_PATHS.md (if exists)
497
+ - {output_dir}/codebase/TEST_ASSESSMENT.md (if exists)
498
+ - {output_dir}/codebase/COVERAGE_GAPS.md (if exists)
499
+ </files_to_read>
500
+ <parameters>
501
+ mode: {mode}
502
+ workflow_option: {option}
503
+ dev_repo_path: {DEV_REPO}
504
+ qa_repo_path: {QA_REPO or null}
505
+ output_path: {output_dir}/
506
+ </parameters>
507
+ "
508
+ )
509
+ ```
510
+
511
+ **Parse analyzer return:**
512
+
513
+ Expected return structure:
514
+ ```
515
+ ANALYZER_COMPLETE:
516
+ files_produced: [...]
517
+ total_test_count: N
518
+ pyramid_breakdown: {unit: N, integration: N, api: N, e2e: N}
519
+ risk_count: {high: N, medium: N, low: N}
520
+ commit_hash: "..."
521
+ ```
522
+
523
+ Capture `files_produced`, `total_test_count`, `pyramid_breakdown` for downstream stages.
524
+
525
+ **Handle analyzer checkpoint -- assumptions review:**
526
+ - If `IS_AUTO` is true: Auto-approve all assumptions (SAFE checkpoint). Log: "Auto-approved: Analyzer assumptions". Continue pipeline.
527
+ - If `IS_AUTO` is false: Present assumptions to user for review. Wait for confirmation or corrections. On user response, incorporate corrections and continue.
528
+
529
+ **State update -- mark analyze as complete:**
530
+ ```bash
531
+ node bin/qaa-tools.cjs state patch --"Analyze Status" complete 2>/dev/null || true
532
+ ```
533
+
534
+ **Verify artifacts exist before continuing:**
535
+
536
+ For Option 1:
537
+ ```bash
538
+ [ -f "${output_dir}/QA_ANALYSIS.md" ] && echo "OK" || echo "MISSING: QA_ANALYSIS.md"
539
+ [ -f "${output_dir}/TEST_INVENTORY.md" ] && echo "OK" || echo "MISSING: TEST_INVENTORY.md"
540
+ ```
541
+
542
+ For Options 2/3:
543
+ ```bash
544
+ [ -f "${output_dir}/GAP_ANALYSIS.md" ] && echo "OK" || echo "MISSING: GAP_ANALYSIS.md"
545
+ ```
546
+
547
+ If required artifacts are missing, treat as stage failure. Set status to failed and STOP pipeline.
548
+
549
+ Print: "Analysis complete. {total_test_count} test cases identified. Pyramid: unit={unit}, integration={integration}, api={api}, e2e={e2e}."
550
+
551
+ </step>
552
+
553
+ <step name="testid_inject">
554
+
555
+ ## Step 5: TestID Injection (Conditional)
556
+
557
+ **Condition:** Only execute if `has_frontend` is `true` from scanner return (Step 3).
558
+
559
+ **If `has_frontend` is false:**
560
+ Print: "Skipping TestID injection (no frontend detected)." Proceed directly to Step 6 (Plan).
561
+
562
+ **If `has_frontend` is true:**
563
+
564
+ **State update:**
565
+ ```bash
566
+ node bin/qaa-tools.cjs state patch --"Status" "Injecting test IDs into frontend components" 2>/dev/null || true
567
+ ```
568
+
569
+ **Print stage banner:**
570
+ ```
571
+ +------------------------------------------+
572
+ | STAGE 3: TestID Injector |
573
+ | Status: Running... |
574
+ +------------------------------------------+
575
+ ```
576
+
577
+ **Spawn testid-injector agent:**
578
+ ```
579
+ Agent(subagent_type="general-purpose",
580
+ prompt="
581
+ <objective>Audit and inject data-testid attributes into frontend components</objective>
582
+ <execution_context>@agents/qaa-testid-injector.md</execution_context>
583
+ <files_to_read>
584
+ - {output_dir}/SCAN_MANIFEST.md
585
+ - CLAUDE.md
586
+ </files_to_read>
587
+ <parameters>
588
+ dev_repo_path: {DEV_REPO}
589
+ output_path: {output_dir}/TESTID_AUDIT_REPORT.md
590
+ </parameters>
591
+ "
592
+ )
593
+ ```
594
+
595
+ **Parse return:**
596
+
597
+ Check for `INJECTOR_COMPLETE` vs `INJECTOR_SKIPPED`:
598
+
599
+ If `INJECTOR_COMPLETE`:
600
+ ```
601
+ INJECTOR_COMPLETE:
602
+ report_path: "..."
603
+ coverage_before: N%
604
+ coverage_after: N%
605
+ elements_injected: N
606
+ components_modified: N
607
+ ```
608
+ Log: "TestID injection complete. Coverage: {coverage_before}% -> {coverage_after}%. {elements_injected} elements injected."
609
+
610
+ If `INJECTOR_SKIPPED`:
611
+ ```
612
+ INJECTOR_SKIPPED:
613
+ reason: "..."
614
+ action: "..."
615
+ ```
616
+ Log the reason and continue pipeline.
617
+
618
+ **Handle injector checkpoint -- audit review:**
619
+ - If `IS_AUTO` is true: Auto-approve P0-only injection (SAFE checkpoint). Log: "Auto-approved: TestID injection (P0 elements only)". Continue pipeline.
620
+ - If `IS_AUTO` is false: Present audit report to user. Wait for approval, element selection, or rejection. On user response, incorporate decisions and continue.
621
+
622
+ **Verify artifact exists:**
623
+ ```bash
624
+ [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ] && echo "OK" || echo "MISSING: TESTID_AUDIT_REPORT.md"
625
+ ```
626
+
627
+ </step>
628
+
629
+ <step name="plan">
630
+
631
+ ## Step 6: Plan Test Generation
632
+
633
+ **State update -- mark generation as running (planning is part of generate):**
634
+ ```bash
635
+ node bin/qaa-tools.cjs state patch --"Generate Status" running --"Status" "Planning test generation" 2>/dev/null || true
636
+ ```
637
+
638
+ **Print stage banner:**
639
+ ```
640
+ +------------------------------------------+
641
+ | STAGE 4: Planner |
642
+ | Status: Running... |
643
+ +------------------------------------------+
644
+ ```
645
+
646
+ **Determine planner input based on option:**
647
+ - Option 1: Input from `{output_dir}/TEST_INVENTORY.md` + `{output_dir}/QA_ANALYSIS.md`
648
+ - Options 2 and 3: Input from `{output_dir}/GAP_ANALYSIS.md`
649
+
650
+ **Spawn planner agent:**
651
+ ```
652
+ Agent(subagent_type="general-purpose",
653
+ prompt="
654
+ <objective>Create test generation plan with task breakdown and dependencies</objective>
655
+ <execution_context>@agents/qaa-planner.md</execution_context>
656
+ <files_to_read>
657
+ - {input files based on option}
658
+ - CLAUDE.md
659
+ - {output_dir}/codebase/TESTABILITY.md (if exists)
660
+ - {output_dir}/codebase/TEST_SURFACE.md (if exists)
661
+ - {output_dir}/codebase/CRITICAL_PATHS.md (if exists)
662
+ - {output_dir}/codebase/COVERAGE_GAPS.md (if exists)
663
+ </files_to_read>
664
+ <parameters>
665
+ workflow_option: {option}
666
+ output_path: {output_dir}/GENERATION_PLAN.md
667
+ </parameters>
668
+ "
669
+ )
670
+ ```
671
+
672
+ **Parse planner return:**
673
+
674
+ Expected return structure:
675
+ ```
676
+ PLANNER_COMPLETE:
677
+ file_path: "..."
678
+ total_tasks: N
679
+ total_files: N
680
+ feature_count: N
681
+ dependency_depth: N
682
+ test_case_count: N
683
+ commit_hash: "..."
684
+ ```
685
+
686
+ Capture `total_tasks`, `total_files`, `feature_count` for executor stage and pipeline summary.
687
+
688
+ **Verify artifact exists:**
689
+ ```bash
690
+ [ -f "${output_dir}/GENERATION_PLAN.md" ] && echo "OK" || echo "MISSING: GENERATION_PLAN.md"
691
+ ```
692
+
693
+ If GENERATION_PLAN.md is missing, treat as stage failure. Set status to failed and STOP pipeline.
694
+
695
+ Print: "Plan complete. {total_tasks} tasks, {total_files} files planned across {feature_count} features."
696
+
697
+ </step>
698
+
699
+ <step name="generate">
700
+
701
+ ## Step 7: Generate Test Files
702
+
703
+ State update continues from planning (already set to `running` in Step 6).
704
+
705
+ **Print stage banner:**
706
+ ```
707
+ +------------------------------------------+
708
+ | STAGE 5: Executor |
709
+ | Generating {total_files} test files |
710
+ | Status: Running... |
711
+ +------------------------------------------+
712
+ ```
713
+
714
+ **Determine execution strategy:**
715
+
716
+ Check if planner created multiple independent feature groups. If `feature_count > 1` AND parallelization is enabled (from init config):
717
+
718
+ **Parallel execution** (when feature_count > 1 and parallelization enabled):
719
+
720
+ For each independent feature group from the generation plan, spawn a separate executor agent:
721
+ ```
722
+ Agent(subagent_type="general-purpose",
723
+ prompt="
724
+ <objective>Generate test files for {feature} feature</objective>
725
+ <execution_context>@agents/qaa-executor.md</execution_context>
726
+ <files_to_read>
727
+ - {output_dir}/GENERATION_PLAN.md
728
+ - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
729
+ - CLAUDE.md
730
+ - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
731
+ - {output_dir}/research/E2E_STRATEGY.md (if exists)
732
+ - {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
733
+ - {output_dir}/codebase/TEST_SURFACE.md (if exists)
734
+ - {output_dir}/codebase/CODE_PATTERNS.md (if exists)
735
+ - {output_dir}/codebase/API_CONTRACTS.md (if exists)
736
+ </files_to_read>
737
+ <parameters>
738
+ workflow_option: {option}
739
+ feature_group: {feature}
740
+ dev_repo_path: {DEV_REPO}
741
+ qa_repo_path: {QA_REPO or null}
742
+ output_path: {output_dir}/
743
+ </parameters>
744
+ "
745
+ )
746
+ ```
747
+
748
+ Multiple Agent() calls can be issued simultaneously for independent feature groups. Each executor handles one feature group and commits its files independently.
749
+
750
+ **Sequential execution** (when feature_count == 1 or parallelization disabled):
751
+
752
+ Spawn a single executor agent covering all tasks:
753
+ ```
754
+ Agent(subagent_type="general-purpose",
755
+ prompt="
756
+ <objective>Generate all test files from generation plan</objective>
757
+ <execution_context>@agents/qaa-executor.md</execution_context>
758
+ <files_to_read>
759
+ - {output_dir}/GENERATION_PLAN.md
760
+ - {output_dir}/TEST_INVENTORY.md (Option 1) or {output_dir}/GAP_ANALYSIS.md (Options 2/3)
761
+ - CLAUDE.md
762
+ - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
763
+ - {output_dir}/research/E2E_STRATEGY.md (if exists)
764
+ - {output_dir}/research/API_TESTING_STRATEGY.md (if exists)
765
+ - {output_dir}/codebase/TEST_SURFACE.md (if exists)
766
+ - {output_dir}/codebase/CODE_PATTERNS.md (if exists)
767
+ - {output_dir}/codebase/API_CONTRACTS.md (if exists)
768
+ </files_to_read>
769
+ <parameters>
770
+ workflow_option: {option}
771
+ dev_repo_path: {DEV_REPO}
772
+ qa_repo_path: {QA_REPO or null}
773
+ output_path: {output_dir}/
774
+ </parameters>
775
+ "
776
+ )
777
+ ```
778
+
779
+ **Option 3 specific -- skip existing tests:**
780
+
781
+ For Option 3, pass `skip_existing_test_ids: true` to the executor so it checks existing test files by test ID before generating. If a test ID already exists in the QA repo, skip generating that test case:
782
+ ```
783
+ <parameters>
784
+ workflow_option: 3
785
+ skip_existing_test_ids: true
786
+ dev_repo_path: {DEV_REPO}
787
+ qa_repo_path: {QA_REPO}
788
+ output_path: {output_dir}/
789
+ </parameters>
790
+ ```
791
+
792
+ **Parse executor return:**
793
+
794
+ Expected return structure:
795
+ ```
796
+ EXECUTOR_COMPLETE:
797
+ files_created: [{path, type}, ...]
798
+ total_files: N
799
+ commit_count: N
800
+ features_covered: [...]
801
+ test_case_count: N
802
+ ```
803
+
804
+ Capture `files_created`, `total_files`, `commit_count` for validation stage and pipeline summary.
805
+
806
+ **State update -- mark generate as complete:**
807
+ ```bash
808
+ node bin/qaa-tools.cjs state patch --"Generate Status" complete --"Status" "Test generation complete" 2>/dev/null || true
809
+ ```
810
+
811
+ Print: "Generation complete. {total_files} files created across {features_covered_count} features. {commit_count} commits."
812
+
813
+ </step>
814
+
815
+ <step name="validate">
816
+
817
+ ## Step 8: Validate Generated Tests
818
+
819
+ **State update -- mark validate as running:**
820
+ ```bash
821
+ node bin/qaa-tools.cjs state patch --"Validate Status" running --"Status" "Validating generated tests" 2>/dev/null || true
822
+ ```
823
+
824
+ **Print stage banner:**
825
+ ```
826
+ +------------------------------------------+
827
+ | STAGE 6: Validator |
828
+ | Validating {total_files} test files |
829
+ | Status: Running... |
830
+ +------------------------------------------+
831
+ ```
832
+
833
+ **Spawn validator agent:**
834
+ ```
835
+ Agent(subagent_type="general-purpose",
836
+ prompt="
837
+ <objective>Run 4-layer validation on all generated test files</objective>
838
+ <execution_context>@agents/qaa-validator.md</execution_context>
839
+ <files_to_read>
840
+ - {list all generated test files from executor return -- files_created paths}
841
+ - {output_dir}/GENERATION_PLAN.md
842
+ - CLAUDE.md
843
+ </files_to_read>
844
+ <parameters>
845
+ mode: validation
846
+ max_fix_loops: 3
847
+ output_path: {output_dir}/VALIDATION_REPORT.md
848
+ </parameters>
849
+ "
850
+ )
851
+ ```
852
+
853
+ **4-layer validation:**
854
+ 1. **Syntax** -- File parses without errors
855
+ 2. **Structure** -- Follows POM rules, naming conventions, locator hierarchy
856
+ 3. **Dependencies** -- Imports resolve, fixtures exist, configs present
857
+ 4. **Logic** -- Assertions are concrete, test IDs are unique, no assertions in page objects
858
+
859
+ **5. Browser verification (if app URL available and Playwright MCP connected):**
860
+
861
+ After the 4-layer static validation, use Playwright MCP to verify E2E tests against the live app:
862
+
863
+ 1. Navigate to each page referenced in the E2E tests:
864
+ ```
865
+ mcp__playwright__browser_navigate({ url: "{app_url}/{route}" })
866
+ ```
867
+
868
+ 2. Take an accessibility snapshot to verify locators used in tests actually exist:
869
+ ```
870
+ mcp__playwright__browser_snapshot()
871
+ ```
872
+
873
+ 3. Cross-reference locators in generated tests against the real DOM:
874
+ - Verify `data-testid` values exist on the page
875
+ - Verify ARIA roles and names match test expectations
876
+ - Flag any test locator that does not match a real DOM element
877
+
878
+ 4. If mismatches are found, fix the test locators to match the real DOM and count as a fix loop iteration.
879
+
880
+ This browser verification step prevents delivering tests with locators that will immediately fail at runtime.
881
+
882
+ **Fix loop:** The validator automatically attempts to fix issues it finds. Maximum 3 fix loop iterations. After each fix attempt, re-validate.
883
+
884
+ **Parse validator return:**
885
+
886
+ Expected return structure:
887
+ ```
888
+ VALIDATOR_COMPLETE:
889
+ report_path: "..."
890
+ overall_status: PASS | PASS_WITH_WARNINGS | FAIL
891
+ confidence: HIGH | MEDIUM | LOW
892
+ layers_summary: {syntax: PASS|FAIL, structure: PASS|FAIL, dependencies: PASS|FAIL, logic: PASS|FAIL}
893
+ fix_loops_used: N
894
+ issues_found: N
895
+ issues_fixed: N
896
+ unresolved_count: N
897
+ ```
898
+
899
+ **RISKY CHECKPOINT -- Validator escalation:**
900
+
901
+ If `unresolved_count > 0` after max fix loops (3):
902
+ - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
903
+ - Present unresolved issues to user with full details from VALIDATION_REPORT.md
904
+ - Wait for user decision:
905
+ - `"approve-with-warnings"`: Accept the validation with warnings. Set Validate Status to complete. Continue to deliver.
906
+ - `"abort"`: Set Validate Status to failed. STOP PIPELINE ENTIRELY.
907
+ - Manual guidance: User provides specific fix instructions. Spawn fresh continuation agent to apply fixes and re-validate.
908
+
909
+ If `overall_status` is `PASS` or `PASS_WITH_WARNINGS` (and unresolved_count is 0):
910
+ ```bash
911
+ node bin/qaa-tools.cjs state patch --"Validate Status" complete --"Status" "Validation passed" 2>/dev/null || true
912
+ ```
913
+
914
+ **Verify artifact exists:**
915
+ ```bash
916
+ [ -f "${output_dir}/VALIDATION_REPORT.md" ] && echo "OK" || echo "MISSING: VALIDATION_REPORT.md"
917
+ ```
918
+
919
+ Print: "Validation complete. Status: {overall_status}. Confidence: {confidence}. {issues_found} issues found, {issues_fixed} fixed, {unresolved_count} unresolved."
920
+
921
+ </step>
922
+
923
+ <step name="bug_detective">
924
+
925
+ ## Step 9: Bug Detective (Conditional)
926
+
927
+ **Condition:** Only execute if test failures were detected during validation. Check:
928
+ - `overall_status === 'FAIL'` in validator return, OR
929
+ - Generated tests have runtime failures that need classification
930
+
931
+ **If no failures to classify:**
932
+ Print: "Skipping Bug Detective (no test failures detected)." Proceed directly to Step 10 (Deliver).
933
+
934
+ **If failures need classification:**
935
+
936
+ **State update:**
937
+ ```bash
938
+ node bin/qaa-tools.cjs state patch --"Status" "Classifying test failures" 2>/dev/null || true
939
+ ```
940
+
941
+ **Print stage banner:**
942
+ ```
943
+ +------------------------------------------+
944
+ | STAGE 7: Bug Detective |
945
+ | Status: Running... |
946
+ +------------------------------------------+
947
+ ```
948
+
949
+ **Spawn bug-detective agent:**
950
+ ```
951
+ Agent(subagent_type="general-purpose",
952
+ prompt="
953
+ <objective>Classify test failures and attempt auto-fixes for test errors. Use Playwright MCP to reproduce E2E failures in the browser when available.</objective>
954
+ <execution_context>@agents/qaa-bug-detective.md</execution_context>
955
+ <files_to_read>
956
+ - {test execution results -- from validator or direct test run}
957
+ - {failing test source files -- paths from executor return}
958
+ - CLAUDE.md
959
+ - {output_dir}/research/FRAMEWORK_CAPABILITIES.md (if exists)
960
+ - {output_dir}/research/TESTING_STACK.md (if exists)
961
+ </files_to_read>
962
+ <parameters>
963
+ output_path: {output_dir}/FAILURE_CLASSIFICATION_REPORT.md
964
+ app_url: {app_url if available}
965
+ </parameters>
966
+ "
967
+ )
968
+ ```
969
+
970
+ **Parse bug-detective return:**
971
+
972
+ Expected return structure:
973
+ ```
974
+ DETECTIVE_COMPLETE:
975
+ report_path: "..."
976
+ total_failures: N
977
+ classification_breakdown: {app_bug: N, test_error: N, env_issue: N, inconclusive: N}
978
+ auto_fixes_applied: N
979
+ auto_fixes_verified: N
980
+ commit_hash: "..."
981
+ ```
982
+
983
+ **RISKY CHECKPOINT -- Application bugs detected:**
984
+
985
+ If `classification_breakdown.app_bug > 0`:
986
+ - **ALWAYS pause, even in auto mode** (RISKY checkpoint -- locked decision)
987
+ - Present APPLICATION BUG classifications to user with full evidence from FAILURE_CLASSIFICATION_REPORT.md
988
+ - These are genuine bugs in the application code discovered during test execution
989
+ - The bug detective never touches application code -- it only reports
990
+ - User must review and decide how to proceed:
991
+ - Acknowledge bugs and continue pipeline (bugs will be in the PR description for developer attention)
992
+ - Abort pipeline to fix bugs first
993
+
994
+ **Verify artifact exists:**
995
+ ```bash
996
+ [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "OK" || echo "MISSING: FAILURE_CLASSIFICATION_REPORT.md"
997
+ ```
998
+
999
+ Print: "Bug Detective complete. {total_failures} failures classified: {app_bug} APP BUG, {test_error} TEST ERROR, {env_issue} ENV ISSUE, {inconclusive} INCONCLUSIVE. {auto_fixes_applied} auto-fixes applied."
1000
+
1001
+ </step>
1002
+
1003
+ <step name="deliver">
1004
+
1005
+ ## Step 10: Deliver
1006
+
1007
+ **State update -- mark deliver as running:**
1008
+ ```bash
1009
+ node bin/qaa-tools.cjs state patch --"Deliver Status" running --"Status" "Preparing delivery" 2>/dev/null || true
1010
+ ```
1011
+
1012
+ **Print stage banner:**
1013
+ ```
1014
+ +------------------------------------------+
1015
+ | STAGE 8: Deliver |
1016
+ | Status: Running... |
1017
+ +------------------------------------------+
1018
+ ```
1019
+
1020
+ ### Sub-step 1: Pre-flight checks
1021
+
1022
+ **Check for git remote:**
1023
+ ```bash
1024
+ REMOTE_URL=$(git remote get-url origin 2>/dev/null || echo "")
1025
+ ```
1026
+
1027
+ If `REMOTE_URL` is empty:
1028
+ - Print: "No git remote found. Artifacts committed locally but PR creation skipped."
1029
+ - Set `LOCAL_ONLY=true`
1030
+
1031
+ **Check for gh CLI authentication:**
1032
+ ```bash
1033
+ gh auth status 2>/dev/null
1034
+ ```
1035
+
1036
+ If `gh auth status` fails:
1037
+ - Print: "gh CLI not authenticated. Run 'gh auth login' first. Artifacts committed locally."
1038
+ - Set `LOCAL_ONLY=true`
1039
+
1040
+ If both checks pass, set `LOCAL_ONLY=false`.
1041
+
1042
+ ### Sub-step 2: Derive project name
1043
+
1044
+ ```bash
1045
+ # Read from package.json
1046
+ PROJECT_NAME=$(node -e "try { const p = require('${DEV_REPO}/package.json'); console.log(p.name || ''); } catch { console.log(''); }" 2>/dev/null)
1047
+
1048
+ # Fallback to directory basename
1049
+ if [ -z "$PROJECT_NAME" ]; then
1050
+ PROJECT_NAME=$(basename "${DEV_REPO}")
1051
+ fi
1052
+
1053
+ # Sanitize for branch naming
1054
+ PROJECT_NAME=$(echo "$PROJECT_NAME" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//' | sed 's/-$//')
1055
+ ```
1056
+
1057
+ ### Sub-step 3: Detect default branch
1058
+
1059
+ ```bash
1060
+ DEFAULT_BRANCH=$(gh repo view --json defaultBranchRef --jq '.defaultBranchRef.name' 2>/dev/null || echo "main")
1061
+ ```
1062
+
1063
+ ### Sub-step 4: Create feature branch
1064
+
1065
+ ```bash
1066
+ BRANCH="qa/auto-${PROJECT_NAME}-${date}"
1067
+
1068
+ # Handle branch name collision
1069
+ if git rev-parse --verify "$BRANCH" 2>/dev/null || git rev-parse --verify "origin/$BRANCH" 2>/dev/null; then
1070
+ SUFFIX=2
1071
+ while git rev-parse --verify "${BRANCH}-${SUFFIX}" 2>/dev/null || git rev-parse --verify "origin/${BRANCH}-${SUFFIX}" 2>/dev/null; do
1072
+ SUFFIX=$((SUFFIX + 1))
1073
+ done
1074
+ BRANCH="${BRANCH}-${SUFFIX}"
1075
+ fi
1076
+
1077
+ git checkout -b "$BRANCH" "$DEFAULT_BRANCH"
1078
+ ```
1079
+
1080
+ ### Sub-step 5: Per-stage atomic commits
1081
+
1082
+ For each pipeline stage that produced artifacts, commit using `qaa-tools.cjs commit`. Check file existence before each commit.
1083
+
1084
+ **Scanner:**
1085
+ ```bash
1086
+ if [ -f "${output_dir}/SCAN_MANIFEST.md" ]; then
1087
+ node bin/qaa-tools.cjs commit "qa(scanner): produce SCAN_MANIFEST.md for ${PROJECT_NAME}" --files ${output_dir}/SCAN_MANIFEST.md
1088
+ fi
1089
+ ```
1090
+
1091
+ **Analyzer (Option 1):**
1092
+ ```bash
1093
+ if [ -f "${output_dir}/QA_ANALYSIS.md" ]; then
1094
+ ANALYZER_FILES="${output_dir}/QA_ANALYSIS.md ${output_dir}/TEST_INVENTORY.md"
1095
+ [ -f "${output_dir}/QA_REPO_BLUEPRINT.md" ] && ANALYZER_FILES="${ANALYZER_FILES} ${output_dir}/QA_REPO_BLUEPRINT.md"
1096
+ node bin/qaa-tools.cjs commit "qa(analyzer): produce QA_ANALYSIS.md and TEST_INVENTORY.md" --files ${ANALYZER_FILES}
1097
+ fi
1098
+ ```
1099
+
1100
+ **Analyzer (Options 2/3):**
1101
+ ```bash
1102
+ if [ -f "${output_dir}/GAP_ANALYSIS.md" ]; then
1103
+ node bin/qaa-tools.cjs commit "qa(analyzer): produce GAP_ANALYSIS.md" --files ${output_dir}/GAP_ANALYSIS.md
1104
+ fi
1105
+ ```
1106
+
1107
+ **TestID Injector (if ran):**
1108
+ ```bash
1109
+ if [ -f "${output_dir}/TESTID_AUDIT_REPORT.md" ]; then
1110
+ node bin/qaa-tools.cjs commit "qa(testid-injector): inject ${elements_injected} data-testid attributes across ${components_modified} components" --files ${output_dir}/TESTID_AUDIT_REPORT.md ${modified_source_files}
1111
+ fi
1112
+ ```
1113
+
1114
+ **Executor:**
1115
+ ```bash
1116
+ if [ -n "${generated_file_paths}" ]; then
1117
+ node bin/qaa-tools.cjs commit "qa(executor): generate ${total_files} test files with POMs and fixtures" --files ${generated_file_paths}
1118
+ fi
1119
+ ```
1120
+
1121
+ **Planner:**
1122
+ ```bash
1123
+ if [ -f "${output_dir}/GENERATION_PLAN.md" ]; then
1124
+ node bin/qaa-tools.cjs commit "qa(planner): produce GENERATION_PLAN.md" --files ${output_dir}/GENERATION_PLAN.md
1125
+ fi
1126
+ ```
1127
+
1128
+ **Validator:**
1129
+ ```bash
1130
+ if [ -f "${output_dir}/VALIDATION_REPORT.md" ]; then
1131
+ node bin/qaa-tools.cjs commit "qa(validator): validate generated tests - ${overall_status} with ${confidence} confidence" --files ${output_dir}/VALIDATION_REPORT.md
1132
+ fi
1133
+ ```
1134
+
1135
+ **Bug Detective (if ran):**
1136
+ ```bash
1137
+ if [ -f "${output_dir}/FAILURE_CLASSIFICATION_REPORT.md" ]; then
1138
+ node bin/qaa-tools.cjs commit "qa(bug-detective): classify ${total_failures} failures - ${classification_summary}" --files ${output_dir}/FAILURE_CLASSIFICATION_REPORT.md
1139
+ fi
1140
+ ```
1141
+
1142
+ ### Sub-step 6: Push branch
1143
+
1144
+ If `LOCAL_ONLY` is true, skip this sub-step.
1145
+
1146
+ ```bash
1147
+ git push -u origin "$BRANCH"
1148
+ ```
1149
+
1150
+ If push fails:
1151
+ - Print: "Push failed: {error_message}. Artifacts committed locally on branch ${BRANCH}."
1152
+ - Set `LOCAL_ONLY=true`
1153
+
1154
+ ### Sub-step 7: Build PR body
1155
+
1156
+ If `LOCAL_ONLY` is true, skip this sub-step.
1157
+
1158
+ Read the PR template:
1159
+ ```bash
1160
+ PR_BODY=$(cat templates/pr-template.md)
1161
+ ```
1162
+
1163
+ Replace all `{placeholder}` tokens with actual values collected during pipeline execution:
1164
+ - `{architecture_type}` -- from QA_ANALYSIS.md or SCAN_MANIFEST.md
1165
+ - `{framework}` -- detected test framework
1166
+ - `{risk_summary}` -- risk assessment counts (e.g., "3 HIGH, 5 MEDIUM, 2 LOW")
1167
+ - `{unit_count}` -- from pyramid_breakdown.unit
1168
+ - `{integration_count}` -- from pyramid_breakdown.integration
1169
+ - `{api_count}` -- from pyramid_breakdown.api
1170
+ - `{e2e_count}` -- from pyramid_breakdown.e2e
1171
+ - `{total_count}` -- from total_test_count
1172
+ - `{modules_covered}` -- count of modules with tests
1173
+ - `{coverage_estimate}` -- estimated coverage percentage
1174
+ - `{validation_result}` -- PASS, PASS_WITH_WARNINGS, or FAIL
1175
+ - `{confidence}` -- HIGH, MEDIUM, or LOW
1176
+ - `{fix_loops_used}` -- number 0-3
1177
+ - `{issues_found}` -- total issues found during validation
1178
+ - `{issues_fixed}` -- total issues auto-fixed
1179
+ - `{file_list}` -- if total files <= 50, list each file; if > 50, use summary
1180
+
1181
+ ### Sub-step 8: Create draft PR
1182
+
1183
+ If `LOCAL_ONLY` is true, skip this sub-step.
1184
+
1185
+ ```bash
1186
+ PR_URL=$(gh pr create \
1187
+ --draft \
1188
+ --title "qa: automated test suite for ${PROJECT_NAME}" \
1189
+ --body "${PR_BODY}" \
1190
+ --label "qa-automation" \
1191
+ --label "auto-generated" \
1192
+ --assignee "@me" 2>&1)
1193
+ ```
1194
+
1195
+ Do NOT pass `--base` flag. Let gh auto-detect the default branch.
1196
+
1197
+ On failure:
1198
+ - Print: "PR creation failed: ${PR_URL}. Artifacts remain on branch ${BRANCH}."
1199
+ - Do NOT stop the pipeline -- artifacts are committed and pushed.
1200
+
1201
+ ### Sub-step 9: Print result
1202
+
1203
+ If PR was created successfully:
1204
+ ```
1205
+ PR created: ${PR_URL}
1206
+ ```
1207
+
1208
+ If `LOCAL_ONLY` is true:
1209
+ ```
1210
+ PR: not created (local-only mode). Artifacts committed on branch: ${BRANCH}
1211
+ ```
1212
+
1213
+ **State update -- mark deliver as complete:**
1214
+ ```bash
1215
+ node bin/qaa-tools.cjs state patch --"Deliver Status" complete --"Status" "Pipeline complete" 2>/dev/null || true
1216
+ ```
1217
+
1218
+ **Clear auto-chain flag at pipeline completion:**
1219
+ ```bash
1220
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1221
+ ```
1222
+
1223
+ ### Print pipeline summary banner:
1224
+
1225
+ ```
1226
+ ======================================================
1227
+ QA PIPELINE COMPLETE
1228
+ ======================================================
1229
+
1230
+ Option: {option} ({option_description})
1231
+ Repository: {DEV_REPO}
1232
+ QA Repo: {QA_REPO or 'N/A'}
1233
+ Maturity Score: {maturity_score or 'N/A'}
1234
+
1235
+ Stages Completed:
1236
+ [{check}] Scan -- {scan_result}
1237
+ [{check}] Analyze -- {analyze_result} ({test_count} test cases)
1238
+ [{check}] TestID Inject -- {inject_result or 'skipped'}
1239
+ [{check}] Plan -- {plan_result} ({file_count} files planned)
1240
+ [{check}] Generate -- {generate_result} ({files_created} files created)
1241
+ [{check}] Validate -- {validate_result} ({confidence} confidence)
1242
+ [{check}] Bug Detective -- {detective_result or 'skipped'}
1243
+ [{check}] Deliver -- {deliver_result}
1244
+
1245
+ PR: {pr_url or 'not created (local-only)'}
1246
+
1247
+ Artifacts:
1248
+ {list all produced .md files in output_dir}
1249
+
1250
+ Total Time: {total_duration}
1251
+ ======================================================
1252
+ ```
1253
+
1254
+ Where: `[x]` = completed, `[ ]` = skipped, `[!]` = failed.
1255
+
1256
+ </step>
1257
+
1258
+ </process>
1259
+
1260
+ <error_handling>
1261
+
1262
+ ## Error Handling
1263
+
1264
+ ### Stage Failure Protocol
1265
+
1266
+ When any agent returns a failure or error:
1267
+
1268
+ 1. **Set stage status to failed:**
1269
+ ```bash
1270
+ node bin/qaa-tools.cjs state patch --"{Stage} Status" failed --"Status" "Pipeline stopped: {Stage} failed - {reason}" 2>/dev/null || true
1271
+ ```
1272
+
1273
+ 2. **Print failure banner:**
1274
+ ```
1275
+ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1276
+ ! PIPELINE STOPPED !
1277
+ ! Stage: {stage_name} !
1278
+ ! Reason: {failure_reason} !
1279
+ ! !
1280
+ ! Completed: {completed_stages} !
1281
+ ! Artifacts: {artifacts_so_far} !
1282
+ ! !
1283
+ ! Action required: Review and re-run !
1284
+ !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
1285
+ ```
1286
+
1287
+ 3. **DO NOT continue to next stage.** The pipeline stops entirely at the failed stage.
1288
+
1289
+ 4. **DO NOT create partial PR.** No branch, no commit, no PR with incomplete results.
1290
+
1291
+ 5. **Preserve all artifacts produced so far.** They remain on disk in `{output_dir}/` for debugging.
1292
+
1293
+ ### Artifact Verification
1294
+
1295
+ After EVERY agent spawn, before advancing to next stage, verify the expected output artifact exists on disk:
1296
+ ```bash
1297
+ [ -f "{expected_artifact_path}" ] && echo "OK" || echo "MISSING"
1298
+ ```
1299
+
1300
+ If artifacts are missing, treat as stage failure and STOP pipeline.
1301
+
1302
+ ### qaa-tools.cjs Graceful Fallback
1303
+
1304
+ All `node bin/qaa-tools.cjs` calls use `2>/dev/null || true` to handle cases where the tool is not installed or not found. The pipeline must not break due to missing state management tooling -- it logs a warning and continues.
1305
+
1306
+ </error_handling>
1307
+
1308
+ <auto_advance>
1309
+
1310
+ ## Auto-Advance Mode
1311
+
1312
+ Auto-advance is enabled when ANY of these is true:
1313
+ - `--auto` flag passed to the `/qa-start` invocation
1314
+ - `config.json` has `workflow.auto_advance = true` (persistent user preference)
1315
+ - `workflow._auto_chain_active = true` in config (ephemeral chain flag from ongoing auto run)
1316
+
1317
+ ### Behavior in Auto Mode
1318
+
1319
+ **SAFE checkpoints are auto-approved.** The pipeline continues without pausing. A log message records the auto-approval:
1320
+ ```
1321
+ Auto-approved: {checkpoint_description}
1322
+ ```
1323
+
1324
+ **RISKY checkpoints ALWAYS pause.** Even in auto mode, the pipeline stops and presents the checkpoint to the user.
1325
+
1326
+ ### Safe vs Risky Checkpoint Classification
1327
+
1328
+ **SAFE (auto-approve in auto mode):**
1329
+
1330
+ | Checkpoint | Agent | Auto-Action |
1331
+ |------------|-------|-------------|
1332
+ | Framework detection uncertain (LOW confidence) | Scanner | Approve with most likely framework |
1333
+ | Analyzer assumptions review | Analyzer | Approve all assumptions |
1334
+ | TestID audit review | TestID Injector | Approve P0-only injection |
1335
+
1336
+ **RISKY (ALWAYS pause, even in auto mode):**
1337
+
1338
+ | Checkpoint | Agent | User Action Required |
1339
+ |------------|-------|---------------------|
1340
+ | Validator escalation (unresolved issues after 3 fix loops) | Validator | approve-with-warnings, abort, or fix guidance |
1341
+ | APPLICATION BUG classification | Bug Detective | Review bugs, continue or fix first |
1342
+ | Any checkpoint with "unresolved" or "failed" blocking text | Any | Review specific blocking issue |
1343
+
1344
+ ### Checkpoint Handling Flow
1345
+
1346
+ ```
1347
+ On agent return with checkpoint data:
1348
+ 1. Extract checkpoint blocking field content
1349
+ 2. Classify as SAFE or RISKY:
1350
+ - "framework detection" -> SAFE
1351
+ - "assumptions" -> SAFE
1352
+ - "audit" or "data-testid" -> SAFE
1353
+ - "unresolved" -> RISKY
1354
+ - "failed" -> RISKY
1355
+ - "APPLICATION BUG" -> RISKY
1356
+ - Default (no pattern match) -> RISKY (conservative)
1357
+ 3. If IS_AUTO and SAFE:
1358
+ - Auto-approve with default action
1359
+ - Log the auto-approval
1360
+ - Continue pipeline
1361
+ 4. If IS_AUTO and RISKY:
1362
+ - PAUSE pipeline
1363
+ - Print checkpoint details with full context
1364
+ - Wait for user input
1365
+ 5. If NOT auto (manual mode):
1366
+ - PAUSE pipeline
1367
+ - Print checkpoint details
1368
+ - Wait for user input
1369
+ ```
1370
+
1371
+ ### Resume After Checkpoint
1372
+
1373
+ When resuming after a checkpoint, spawn a FRESH agent with explicit state:
1374
+ ```
1375
+ Agent(subagent_type="general-purpose",
1376
+ prompt="
1377
+ <objective>Continue QA pipeline from {stage} stage</objective>
1378
+ <execution_context>@agents/qa-pipeline-orchestrator.md</execution_context>
1379
+ <resume_context>
1380
+ Pipeline state:
1381
+ - Completed stages: {list of completed stages with their results}
1382
+ - Current stage: {stage that triggered checkpoint}
1383
+ - Checkpoint response: {user's response or decision}
1384
+ - Artifacts produced so far: {list of files with paths}
1385
+
1386
+ Resume from: {exact step in pipeline to resume from}
1387
+ User decision: {what user chose at checkpoint}
1388
+ </resume_context>
1389
+ "
1390
+ )
1391
+ ```
1392
+
1393
+ ### Stale Chain Flag Protection
1394
+
1395
+ At orchestrator init, if `--auto` was NOT passed AND `auto_advance` config is false:
1396
+ ```bash
1397
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1398
+ ```
1399
+
1400
+ At pipeline completion (success or failure):
1401
+ ```bash
1402
+ node bin/qaa-tools.cjs config-set workflow._auto_chain_active false 2>/dev/null || true
1403
+ ```
1404
+
1405
+ </auto_advance>