qaa-agent 1.0.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/qa-analyze.md +4 -43
- package/.claude/commands/qa-from-ticket.md +4 -69
- package/.claude/commands/qa-gap.md +4 -38
- package/.claude/commands/qa-map.md +36 -0
- package/.claude/commands/qa-research.md +33 -0
- package/.claude/commands/qa-start.md +4 -15
- package/.claude/commands/qa-testid.md +7 -42
- package/.claude/commands/qa-validate.md +7 -41
- package/agents/qaa-codebase-mapper.md +935 -0
- package/agents/qaa-project-researcher.md +319 -0
- package/bin/install.cjs +6 -0
- package/package.json +2 -1
- package/workflows/qa-analyze.md +296 -0
- package/workflows/qa-from-ticket.md +474 -0
- package/workflows/qa-gap.md +303 -0
- package/workflows/qa-start.md +1168 -0
- package/workflows/qa-testid.md +356 -0
- package/workflows/qa-validate.md +295 -0
|
@@ -0,0 +1,356 @@
|
|
|
1
|
+
<purpose>
|
|
2
|
+
Scan a frontend codebase, audit every interactive UI element for data-testid coverage, and inject missing data-testid attributes following the `{context}-{description}-{element-type}` naming convention from CLAUDE.md. Operates on a separate branch (`qa/testid-{project}-{date}`) to keep the working copy clean. Produces TESTID_AUDIT_REPORT.md with coverage scores, proposed values, and a decision gate. The user reviews and approves proposed changes before injection. Use this workflow to prepare a frontend codebase for reliable Tier 1 test selectors.
|
|
3
|
+
</purpose>
|
|
4
|
+
|
|
5
|
+
<required_reading>
|
|
6
|
+
- `CLAUDE.md` -- data-testid Convention (naming pattern, context derivation, element-type suffix table, third-party handling), Locator Strategy (Tier 1 includes data-testid)
|
|
7
|
+
- `data-testid-SKILL.md` -- Complete naming convention reference, framework-specific injection syntax, edge case handling
|
|
8
|
+
- `agents/qaa-scanner.md` -- Scanner agent definition (detects frontend framework, enumerates component files)
|
|
9
|
+
- `agents/qaa-testid-injector.md` -- TestID injector agent definition (4-phase: SCAN, AUDIT, INJECT, VALIDATE)
|
|
10
|
+
- `templates/scan-manifest.md` -- SCAN_MANIFEST.md format contract
|
|
11
|
+
- `templates/testid-audit-report.md` -- TESTID_AUDIT_REPORT.md format contract
|
|
12
|
+
</required_reading>
|
|
13
|
+
|
|
14
|
+
<process>
|
|
15
|
+
|
|
16
|
+
<step name="parse_arguments">
|
|
17
|
+
## Step 1: Parse Arguments
|
|
18
|
+
|
|
19
|
+
Parse `$ARGUMENTS` for the source directory path.
|
|
20
|
+
|
|
21
|
+
**Supported arguments:**
|
|
22
|
+
- `<path>` -- Path to the frontend source directory (positional)
|
|
23
|
+
- No arguments -- auto-detect frontend directories
|
|
24
|
+
|
|
25
|
+
**Parsing logic:**
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
SOURCE_DIR=""
|
|
29
|
+
|
|
30
|
+
# Parse positional path argument from $ARGUMENTS
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
**Validation:**
|
|
34
|
+
- If a path is provided, verify it exists and is a directory. If not, print error: `"Error: Source directory does not exist: {path}"` and STOP.
|
|
35
|
+
- If no path provided, proceed to auto-detection in the next step.
|
|
36
|
+
</step>
|
|
37
|
+
|
|
38
|
+
<step name="detect_frontend">
|
|
39
|
+
## Step 2: Detect Frontend Directory and Framework
|
|
40
|
+
|
|
41
|
+
If no source directory was specified, auto-detect by searching for common frontend directory patterns.
|
|
42
|
+
|
|
43
|
+
**Directory detection order:**
|
|
44
|
+
|
|
45
|
+
1. `src/` -- Most common source directory
|
|
46
|
+
2. `app/` -- Next.js app directory, or generic app directory
|
|
47
|
+
3. `components/` -- Component-focused projects
|
|
48
|
+
4. `pages/` -- File-based routing frameworks (Next.js, Nuxt)
|
|
49
|
+
5. `lib/` -- Library projects
|
|
50
|
+
6. `client/` -- Monorepo client directory
|
|
51
|
+
|
|
52
|
+
**Detection logic:**
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
if SOURCE_DIR is empty:
|
|
56
|
+
for dir in src app components pages lib client:
|
|
57
|
+
if directory exists at "${dir}":
|
|
58
|
+
SOURCE_DIR="${dir}"
|
|
59
|
+
break
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
**If no frontend directory found:**
|
|
63
|
+
|
|
64
|
+
Print error and STOP:
|
|
65
|
+
```
|
|
66
|
+
Error: No frontend source directory found.
|
|
67
|
+
Searched for: src/, app/, components/, pages/, lib/, client/
|
|
68
|
+
Provide the source directory path as an argument: /qa-testid <path>
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
**Detect frontend framework:**
|
|
72
|
+
|
|
73
|
+
Check for framework indicators in the project root:
|
|
74
|
+
|
|
75
|
+
| Framework | Detection Method |
|
|
76
|
+
|-----------|-----------------|
|
|
77
|
+
| React | `react` or `react-dom` in package.json dependencies; `*.jsx` or `*.tsx` files present |
|
|
78
|
+
| Vue | `vue` in package.json dependencies; `*.vue` files present |
|
|
79
|
+
| Angular | `@angular/core` in package.json dependencies; `*.component.ts` and `*.component.html` files present |
|
|
80
|
+
| Svelte | `svelte` in package.json dependencies; `*.svelte` files present |
|
|
81
|
+
| Plain HTML | None of the above; `*.html` files present (excluding build output) |
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
FRONTEND_FRAMEWORK=""
|
|
85
|
+
|
|
86
|
+
# Check package.json for framework dependencies
|
|
87
|
+
# Check for framework-specific file extensions in SOURCE_DIR
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
**If no frontend framework detected:**
|
|
91
|
+
|
|
92
|
+
Print error and STOP:
|
|
93
|
+
```
|
|
94
|
+
Error: No frontend framework detected.
|
|
95
|
+
Checked package.json dependencies and file extensions in {SOURCE_DIR}.
|
|
96
|
+
No React (.jsx/.tsx), Vue (.vue), Angular (.component.ts), Svelte (.svelte), or HTML files found.
|
|
97
|
+
|
|
98
|
+
This workflow requires a frontend codebase with UI components to inject data-testid attributes.
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
**Count component files:**
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
# Count files matching the detected framework's extensions
|
|
105
|
+
COMPONENT_COUNT=$(find "${SOURCE_DIR}" -type f \( -name "*.tsx" -o -name "*.jsx" -o -name "*.vue" -o -name "*.svelte" -o -name "*.component.html" \) \
|
|
106
|
+
! -path "*/node_modules/*" ! -path "*/dist/*" ! -path "*/build/*" \
|
|
107
|
+
! -name "*.test.*" ! -name "*.spec.*" ! -name "*.stories.*" | wc -l)
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
If `COMPONENT_COUNT` is 0, print error and STOP:
|
|
111
|
+
```
|
|
112
|
+
Error: No component files found in {SOURCE_DIR}.
|
|
113
|
+
Framework detected: {FRONTEND_FRAMEWORK}
|
|
114
|
+
Expected file patterns: {patterns for that framework}
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
**Print detection results:**
|
|
118
|
+
|
|
119
|
+
```
|
|
120
|
+
Frontend Detection:
|
|
121
|
+
Source Directory: {SOURCE_DIR}
|
|
122
|
+
Framework: {FRONTEND_FRAMEWORK}
|
|
123
|
+
Component Files: {COMPONENT_COUNT}
|
|
124
|
+
```
|
|
125
|
+
</step>
|
|
126
|
+
|
|
127
|
+
<step name="run_scanner">
|
|
128
|
+
## Step 3: Spawn Scanner Agent
|
|
129
|
+
|
|
130
|
+
Spawn the scanner agent to produce SCAN_MANIFEST.md with the frontend component file list.
|
|
131
|
+
|
|
132
|
+
```
|
|
133
|
+
Task(
|
|
134
|
+
prompt="
|
|
135
|
+
<objective>Scan frontend repository and produce SCAN_MANIFEST.md with component file enumeration</objective>
|
|
136
|
+
<execution_context>@agents/qaa-scanner.md</execution_context>
|
|
137
|
+
<files_to_read>
|
|
138
|
+
- CLAUDE.md
|
|
139
|
+
</files_to_read>
|
|
140
|
+
<parameters>
|
|
141
|
+
dev_repo_path: {project root}
|
|
142
|
+
qa_repo_path: null
|
|
143
|
+
output_path: {OUTPUT_DIR}/SCAN_MANIFEST.md
|
|
144
|
+
</parameters>
|
|
145
|
+
"
|
|
146
|
+
)
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
**Set output directory:**
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
OUTPUT_DIR=".qa-output"
|
|
153
|
+
mkdir -p "${OUTPUT_DIR}"
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
**Handle scanner return:**
|
|
157
|
+
|
|
158
|
+
- Extract `has_frontend` -- MUST be `true`. If `false`, print error and STOP:
|
|
159
|
+
```
|
|
160
|
+
Error: Scanner reports has_frontend: false.
|
|
161
|
+
The detected framework ({FRONTEND_FRAMEWORK}) and component files ({COMPONENT_COUNT}) suggest frontend is present,
|
|
162
|
+
but the scanner could not classify them as frontend components.
|
|
163
|
+
Check SCAN_MANIFEST.md for details: {OUTPUT_DIR}/SCAN_MANIFEST.md
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
**Verify SCAN_MANIFEST.md exists:**
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
[ -f "${OUTPUT_DIR}/SCAN_MANIFEST.md" ] && echo "FOUND" || echo "MISSING"
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
If missing, print error and STOP.
|
|
173
|
+
</step>
|
|
174
|
+
|
|
175
|
+
<step name="run_testid_injector">
|
|
176
|
+
## Step 4: Spawn TestID Injector Agent
|
|
177
|
+
|
|
178
|
+
Spawn the testid-injector agent which operates in 4 phases: SCAN, AUDIT, INJECT, VALIDATE.
|
|
179
|
+
|
|
180
|
+
```
|
|
181
|
+
Task(
|
|
182
|
+
prompt="
|
|
183
|
+
<objective>Audit frontend components for data-testid coverage and inject missing attributes</objective>
|
|
184
|
+
<execution_context>@agents/qaa-testid-injector.md</execution_context>
|
|
185
|
+
<files_to_read>
|
|
186
|
+
- {OUTPUT_DIR}/SCAN_MANIFEST.md
|
|
187
|
+
- CLAUDE.md
|
|
188
|
+
- data-testid-SKILL.md
|
|
189
|
+
</files_to_read>
|
|
190
|
+
<parameters>
|
|
191
|
+
source_dir: {SOURCE_DIR}
|
|
192
|
+
frontend_framework: {FRONTEND_FRAMEWORK}
|
|
193
|
+
output_path: {OUTPUT_DIR}/TESTID_AUDIT_REPORT.md
|
|
194
|
+
</parameters>
|
|
195
|
+
"
|
|
196
|
+
)
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
**Injector agent phases:**
|
|
200
|
+
|
|
201
|
+
1. **SCAN phase:** The injector detects the framework, lists component files, classifies them by interaction density (HIGH/MEDIUM/LOW), and sorts by priority.
|
|
202
|
+
|
|
203
|
+
2. **AUDIT phase:** For each component file, the injector identifies every interactive element (buttons, inputs, selects, forms, links, modals, etc.), checks for existing data-testid attributes, proposes values for missing attributes following `{context}-{description}-{element-type}` convention, classifies elements by priority (P0/P1/P2), and calculates the coverage score.
|
|
204
|
+
|
|
205
|
+
3. **CHECKPOINT:** The injector presents the TESTID_AUDIT_REPORT.md to the user for review before injecting. The user can:
|
|
206
|
+
- Approve all proposed values
|
|
207
|
+
- Reject specific elements
|
|
208
|
+
- Rename proposed values
|
|
209
|
+
- Approve renaming of non-compliant existing values
|
|
210
|
+
- Change priority classifications
|
|
211
|
+
|
|
212
|
+
4. **INJECT phase (after approval):** Creates a separate branch `qa/testid-inject-{YYYY-MM-DD}`, injects approved data-testid attributes using framework-specific syntax (JSX attributes for React, HTML attributes for Vue templates, `[attr.data-testid]` binding for Angular), preserves all existing formatting, and preserves existing data-testid values.
|
|
213
|
+
|
|
214
|
+
5. **VALIDATE phase:** Runs syntax checks on all modified files, verifies uniqueness of data-testid values per page/route scope, checks naming convention compliance, and verifies no unintended source code changes (non-interference check). If syntax fails, reverts the specific file.
|
|
215
|
+
|
|
216
|
+
**Handle injector return:**
|
|
217
|
+
|
|
218
|
+
Extract:
|
|
219
|
+
- `report_path` -- Path to TESTID_AUDIT_REPORT.md
|
|
220
|
+
- `changelog_path` -- Path to INJECTION_CHANGELOG.md
|
|
221
|
+
- `branch` -- Name of the injection branch
|
|
222
|
+
- `coverage_before` -- Coverage before injection
|
|
223
|
+
- `coverage_after` -- Coverage after injection
|
|
224
|
+
- `elements_injected` -- Count of injected attributes
|
|
225
|
+
- `elements_deferred` -- Count of deferred elements
|
|
226
|
+
- `files_modified` -- Count of modified files
|
|
227
|
+
- `validation_passed` -- Whether all validation checks passed
|
|
228
|
+
</step>
|
|
229
|
+
|
|
230
|
+
<step name="create_branch_and_commit">
|
|
231
|
+
## Step 5: Create Branch and Commit
|
|
232
|
+
|
|
233
|
+
The testid-injector agent handles branch creation and commits internally on the `qa/testid-inject-{YYYY-MM-DD}` branch. Verify the branch was created and commits exist.
|
|
234
|
+
|
|
235
|
+
**Verify branch:**
|
|
236
|
+
|
|
237
|
+
```bash
|
|
238
|
+
git branch --list "qa/testid-inject-*"
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
If the branch does not exist, the injector may not have reached the injection phase (stopped at audit checkpoint). This is expected behavior -- the audit report is still produced.
|
|
242
|
+
|
|
243
|
+
**Verify commit (if injection was performed):**
|
|
244
|
+
|
|
245
|
+
```bash
|
|
246
|
+
git log --oneline -3 "qa/testid-inject-*" 2>/dev/null
|
|
247
|
+
```
|
|
248
|
+
</step>
|
|
249
|
+
|
|
250
|
+
<step name="print_summary">
|
|
251
|
+
## Step 6: Print Summary
|
|
252
|
+
|
|
253
|
+
Print a comprehensive summary of the testid injection results.
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
=== TestID Injection Complete ===
|
|
257
|
+
|
|
258
|
+
Frontend:
|
|
259
|
+
Framework: {FRONTEND_FRAMEWORK}
|
|
260
|
+
Source Directory: {SOURCE_DIR}
|
|
261
|
+
Component Files: {COMPONENT_COUNT}
|
|
262
|
+
|
|
263
|
+
Coverage:
|
|
264
|
+
Before: {coverage_before}%
|
|
265
|
+
After: {coverage_after}%
|
|
266
|
+
Change: +{coverage_after - coverage_before}%
|
|
267
|
+
|
|
268
|
+
Elements:
|
|
269
|
+
Total Interactive: {total_elements}
|
|
270
|
+
Already Had TestID: {existing_count}
|
|
271
|
+
Injected: {elements_injected}
|
|
272
|
+
Deferred (P1/P2): {elements_deferred}
|
|
273
|
+
|
|
274
|
+
By Priority:
|
|
275
|
+
P0 (inputs, buttons, forms): {p0_count} ({p0_injected} injected)
|
|
276
|
+
P1 (links, alerts, toggles): {p1_count} ({p1_injected} injected)
|
|
277
|
+
P2 (images, badges, tables): {p2_count} ({p2_injected} injected)
|
|
278
|
+
|
|
279
|
+
Validation:
|
|
280
|
+
Syntax Check: {PASS|FAIL}
|
|
281
|
+
Uniqueness Check: {PASS|FAIL}
|
|
282
|
+
Convention Check: {PASS|FAIL}
|
|
283
|
+
Non-Interference: {PASS|FAIL}
|
|
284
|
+
|
|
285
|
+
Branch: {branch_name}
|
|
286
|
+
Files Modified: {files_modified}
|
|
287
|
+
|
|
288
|
+
Artifacts:
|
|
289
|
+
- {OUTPUT_DIR}/SCAN_MANIFEST.md
|
|
290
|
+
- {OUTPUT_DIR}/TESTID_AUDIT_REPORT.md
|
|
291
|
+
- {OUTPUT_DIR}/INJECTION_CHANGELOG.md (after injection)
|
|
292
|
+
|
|
293
|
+
Next Steps:
|
|
294
|
+
1. Review changes on branch: git diff main..{branch_name}
|
|
295
|
+
2. If approved, merge: git merge {branch_name}
|
|
296
|
+
3. Run tests to verify no regressions
|
|
297
|
+
=================================
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
**If injection was not performed (stopped at audit checkpoint):**
|
|
301
|
+
|
|
302
|
+
```
|
|
303
|
+
=== TestID Audit Complete (Injection Pending) ===
|
|
304
|
+
|
|
305
|
+
The audit report has been produced. Review the proposed data-testid values
|
|
306
|
+
in {OUTPUT_DIR}/TESTID_AUDIT_REPORT.md and re-run with approvals to inject.
|
|
307
|
+
|
|
308
|
+
Coverage Score: {coverage_before}%
|
|
309
|
+
Decision: {SELECTIVE|TARGETED|FULL PASS|P0 FIRST}
|
|
310
|
+
Missing TestIDs: {missing_count} ({p0_missing} P0, {p1_missing} P1, {p2_missing} P2)
|
|
311
|
+
=============================================
|
|
312
|
+
```
|
|
313
|
+
</step>
|
|
314
|
+
|
|
315
|
+
</process>
|
|
316
|
+
|
|
317
|
+
<output>
|
|
318
|
+
This workflow audits and injects data-testid attributes into frontend components.
|
|
319
|
+
|
|
320
|
+
**Artifacts produced:**
|
|
321
|
+
|
|
322
|
+
| Artifact | When Produced | Description |
|
|
323
|
+
|----------|---------------|-------------|
|
|
324
|
+
| SCAN_MANIFEST.md | Always | Repository scan with component file list and framework detection |
|
|
325
|
+
| TESTID_AUDIT_REPORT.md | Always | Comprehensive audit: coverage score, per-element tables, naming compliance, decision gate |
|
|
326
|
+
| INJECTION_CHANGELOG.md | After injection approval | Per-file log of every injection action with validation results |
|
|
327
|
+
| Modified source files | After injection approval | Component files with data-testid attributes added, on separate branch |
|
|
328
|
+
|
|
329
|
+
**Branch management:**
|
|
330
|
+
- Injection branch: `qa/testid-inject-{YYYY-MM-DD}`
|
|
331
|
+
- Working copy on the original branch is NOT modified
|
|
332
|
+
- User merges the injection branch if approved
|
|
333
|
+
|
|
334
|
+
**Decision gate thresholds (from CLAUDE.md):**
|
|
335
|
+
|
|
336
|
+
| Current Coverage | Decision | Strategy |
|
|
337
|
+
|-----------------|----------|----------|
|
|
338
|
+
| > 90% | SELECTIVE | Inject only P0 missing elements |
|
|
339
|
+
| 50% - 90% | TARGETED | Inject P0 and P1 missing elements |
|
|
340
|
+
| 1% - 49% | FULL PASS | Inject all P0, P1, P2 elements |
|
|
341
|
+
| 0% | P0 FIRST | Inject P0 elements only, then re-audit |
|
|
342
|
+
</output>
|
|
343
|
+
|
|
344
|
+
<error_handling>
|
|
345
|
+
| Error | Cause | Action |
|
|
346
|
+
|-------|-------|--------|
|
|
347
|
+
| Source directory does not exist | Invalid path argument | Print error with path, STOP |
|
|
348
|
+
| No frontend directory found | No common frontend dirs in project | Print error listing searched dirs, suggest providing path, STOP |
|
|
349
|
+
| No frontend framework detected | Missing framework deps and file extensions | Print error with detection details, STOP |
|
|
350
|
+
| No component files found | Directory exists but no components | Print error with expected patterns for the detected framework, STOP |
|
|
351
|
+
| Scanner reports has_frontend: false | Scanner classification issue | Print error with suggestion to check SCAN_MANIFEST.md, STOP |
|
|
352
|
+
| SCAN_MANIFEST.md missing | Scanner failed | Print error, STOP |
|
|
353
|
+
| Syntax check fails after injection | Injection broke file syntax | Injector reverts the specific file automatically, reports in changelog |
|
|
354
|
+
| Duplicate data-testid detected | Same value on same rendered page | Injector renames the newer injection to disambiguate |
|
|
355
|
+
| Non-compliant existing values | Existing data-testid does not follow convention | Reported in audit but NOT auto-renamed (user decides) |
|
|
356
|
+
</error_handling>
|
|
@@ -0,0 +1,295 @@
|
|
|
1
|
+
<purpose>
|
|
2
|
+
Validate existing test files against CLAUDE.md QA standards using a 4-layer check (Syntax, Structure, Dependencies, Logic). Runs a fix loop (validate, fix, re-validate) up to 3 times. If failures remain after the fix loop, optionally spawns the bug-detective agent to classify failures. Produces VALIDATION_REPORT.md and optionally FAILURE_CLASSIFICATION_REPORT.md. Use this workflow to audit test quality in any test suite -- whether generated by the QA pipeline or hand-written.
|
|
3
|
+
</purpose>
|
|
4
|
+
|
|
5
|
+
<required_reading>
|
|
6
|
+
- `CLAUDE.md` -- QA automation standards, quality gates, locator strategy, POM rules, assertion specificity rules
|
|
7
|
+
- `agents/qaa-validator.md` -- Validator agent definition (4-layer validation, fix loop, confidence assessment)
|
|
8
|
+
- `agents/qaa-bug-detective.md` -- Bug detective agent definition (failure classification, auto-fix for test errors)
|
|
9
|
+
- `templates/validation-report.md` -- VALIDATION_REPORT.md format contract
|
|
10
|
+
- `templates/failure-classification.md` -- FAILURE_CLASSIFICATION_REPORT.md format contract
|
|
11
|
+
</required_reading>
|
|
12
|
+
|
|
13
|
+
<process>
|
|
14
|
+
|
|
15
|
+
<step name="parse_arguments">
|
|
16
|
+
## Step 1: Parse Arguments
|
|
17
|
+
|
|
18
|
+
Parse `$ARGUMENTS` for the test directory path and optional flags.
|
|
19
|
+
|
|
20
|
+
**Supported arguments:**
|
|
21
|
+
- `<path>` -- Path to the test directory to validate (positional)
|
|
22
|
+
- `--classify` -- If set, spawn bug-detective after validation to classify remaining failures
|
|
23
|
+
- No arguments -- auto-detect test directories
|
|
24
|
+
|
|
25
|
+
**Parsing logic:**
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
TEST_DIR=""
|
|
29
|
+
CLASSIFY_FAILURES=false
|
|
30
|
+
|
|
31
|
+
# Parse positional path argument
|
|
32
|
+
# Parse --classify flag
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
**Validation:**
|
|
36
|
+
- If a path is provided, verify it exists and is a directory. If not, print error: `"Error: Test directory does not exist: {path}"` and STOP.
|
|
37
|
+
- If no path provided, proceed to auto-detection in the next step.
|
|
38
|
+
</step>
|
|
39
|
+
|
|
40
|
+
<step name="detect_test_directories">
|
|
41
|
+
## Step 2: Detect Test Directories
|
|
42
|
+
|
|
43
|
+
If no test directory was specified, auto-detect by searching for common test directory patterns.
|
|
44
|
+
|
|
45
|
+
**Detection order (check each, use the first match):**
|
|
46
|
+
|
|
47
|
+
1. `tests/` -- Standard test directory
|
|
48
|
+
2. `__tests__/` -- Jest convention
|
|
49
|
+
3. `cypress/` -- Cypress test directory
|
|
50
|
+
4. `e2e/` -- E2E test directory
|
|
51
|
+
5. `test/` -- Alternative singular form
|
|
52
|
+
6. `spec/` -- RSpec/generic spec directory
|
|
53
|
+
|
|
54
|
+
**Detection logic:**
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
if TEST_DIR is empty:
|
|
58
|
+
for dir in tests __tests__ cypress e2e test spec:
|
|
59
|
+
if directory exists at "${dir}":
|
|
60
|
+
TEST_DIR="${dir}"
|
|
61
|
+
break
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
**If no test directory found:**
|
|
65
|
+
|
|
66
|
+
Print error and STOP:
|
|
67
|
+
```
|
|
68
|
+
Error: No test directory found.
|
|
69
|
+
Searched for: tests/, __tests__/, cypress/, e2e/, test/, spec/
|
|
70
|
+
Provide the test directory path as an argument: /qa-validate <path>
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
**Count test files:**
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
TEST_FILE_COUNT=$(find "${TEST_DIR}" -type f \( -name "*.test.*" -o -name "*.spec.*" -o -name "*.cy.*" -o -name "*.e2e.*" \) | wc -l)
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
If `TEST_FILE_COUNT` is 0, print error: `"Error: No test files found in {TEST_DIR}. Expected files matching *.test.*, *.spec.*, *.cy.*, or *.e2e.*"` and STOP.
|
|
80
|
+
|
|
81
|
+
**Print detection results:**
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
Test directory: {TEST_DIR}
|
|
85
|
+
Test files found: {TEST_FILE_COUNT}
|
|
86
|
+
```
|
|
87
|
+
</step>
|
|
88
|
+
|
|
89
|
+
<step name="build_file_list">
|
|
90
|
+
## Step 3: Build File List for Validation
|
|
91
|
+
|
|
92
|
+
Enumerate all test files and supporting files (page objects, fixtures) to validate.
|
|
93
|
+
|
|
94
|
+
**File discovery:**
|
|
95
|
+
|
|
96
|
+
Glob for test-related files in the detected directory:
|
|
97
|
+
- Test specs: `**/*.test.*`, `**/*.spec.*`, `**/*.cy.*`, `**/*.e2e.*`
|
|
98
|
+
- Page objects: `**/pages/**/*`, `**/page-objects/**/*`, `**/support/page-objects/**/*`
|
|
99
|
+
- Fixtures: `**/fixtures/**/*`
|
|
100
|
+
- Config files: test framework configs at the project root
|
|
101
|
+
|
|
102
|
+
**Build a generation-plan-equivalent file list:**
|
|
103
|
+
|
|
104
|
+
The validator agent expects a file list (normally from the generation plan). For this workflow, construct a synthetic file list from the discovered files:
|
|
105
|
+
|
|
106
|
+
```markdown
|
|
107
|
+
## Files to Validate
|
|
108
|
+
|
|
109
|
+
| File Path | Type |
|
|
110
|
+
|-----------|------|
|
|
111
|
+
| tests/unit/auth.unit.spec.ts | test_spec |
|
|
112
|
+
| tests/api/users.api.spec.ts | test_spec |
|
|
113
|
+
| pages/auth/LoginPage.ts | page_object |
|
|
114
|
+
| fixtures/auth-data.ts | fixture |
|
|
115
|
+
| ... | ... |
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
**Set output directory:**
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
OUTPUT_DIR=".qa-output"
|
|
122
|
+
mkdir -p "${OUTPUT_DIR}"
|
|
123
|
+
```
|
|
124
|
+
</step>
|
|
125
|
+
|
|
126
|
+
<step name="run_validator">
|
|
127
|
+
## Step 4: Spawn Validator Agent
|
|
128
|
+
|
|
129
|
+
Spawn the validator agent to perform 4-layer validation with fix loop.
|
|
130
|
+
|
|
131
|
+
```
|
|
132
|
+
Task(
|
|
133
|
+
prompt="
|
|
134
|
+
<objective>Validate test files against CLAUDE.md standards across 4 layers (Syntax, Structure, Dependencies, Logic)</objective>
|
|
135
|
+
<execution_context>@agents/qaa-validator.md</execution_context>
|
|
136
|
+
<files_to_read>
|
|
137
|
+
- CLAUDE.md
|
|
138
|
+
- {synthetic file list or generation plan path}
|
|
139
|
+
</files_to_read>
|
|
140
|
+
<parameters>
|
|
141
|
+
test_dir: {TEST_DIR}
|
|
142
|
+
output_path: {OUTPUT_DIR}/VALIDATION_REPORT.md
|
|
143
|
+
file_list: {list of test file paths}
|
|
144
|
+
mode: validation
|
|
145
|
+
</parameters>
|
|
146
|
+
"
|
|
147
|
+
)
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
**Handle validator return:**
|
|
151
|
+
|
|
152
|
+
Extract from the validator's return values:
|
|
153
|
+
- `overall_status` -- PASS, PASS_WITH_WARNINGS, or FAIL
|
|
154
|
+
- `confidence` -- HIGH, MEDIUM, or LOW
|
|
155
|
+
- `layers_summary` -- per-layer PASS/FAIL status
|
|
156
|
+
- `fix_loops_used` -- number of fix iterations (1-3)
|
|
157
|
+
- `issues_found` -- total issues discovered
|
|
158
|
+
- `issues_fixed` -- issues auto-fixed (HIGH confidence only)
|
|
159
|
+
- `unresolved_count` -- issues remaining after fix loops
|
|
160
|
+
|
|
161
|
+
**Verify VALIDATION_REPORT.md exists:**
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
[ -f "${OUTPUT_DIR}/VALIDATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
If missing, print error: `"Error: Validator did not produce VALIDATION_REPORT.md."` and STOP.
|
|
168
|
+
</step>
|
|
169
|
+
|
|
170
|
+
<step name="classify_failures">
|
|
171
|
+
## Step 5: Optionally Spawn Bug Detective
|
|
172
|
+
|
|
173
|
+
If the `--classify` flag was set AND the validator reported failures (unresolved_count > 0), spawn the bug-detective agent to classify the remaining failures.
|
|
174
|
+
|
|
175
|
+
**Gate check:**
|
|
176
|
+
|
|
177
|
+
```
|
|
178
|
+
if CLASSIFY_FAILURES is false:
|
|
179
|
+
Skip this step -- print note: "Skipping failure classification. Use --classify flag to enable."
|
|
180
|
+
|
|
181
|
+
if unresolved_count is 0:
|
|
182
|
+
Skip this step -- print note: "No unresolved failures to classify."
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
**Spawn bug-detective agent:**
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
Task(
|
|
189
|
+
prompt="
|
|
190
|
+
<objective>Run test suite and classify failures into APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE</objective>
|
|
191
|
+
<execution_context>@agents/qaa-bug-detective.md</execution_context>
|
|
192
|
+
<files_to_read>
|
|
193
|
+
- CLAUDE.md
|
|
194
|
+
- {test file paths with failures}
|
|
195
|
+
</files_to_read>
|
|
196
|
+
<parameters>
|
|
197
|
+
test_dir: {TEST_DIR}
|
|
198
|
+
output_path: {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md
|
|
199
|
+
</parameters>
|
|
200
|
+
"
|
|
201
|
+
)
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
**Handle detective return:**
|
|
205
|
+
|
|
206
|
+
Extract from the detective's return values:
|
|
207
|
+
- `total_failures` -- number of failures analyzed
|
|
208
|
+
- `classification_breakdown` -- counts per category (app_bug, test_error, env_issue, inconclusive)
|
|
209
|
+
- `auto_fixes_applied` -- count of TEST CODE ERROR fixes at HIGH confidence
|
|
210
|
+
- `auto_fixes_verified` -- count of fixes that passed re-verification
|
|
211
|
+
|
|
212
|
+
**Verify FAILURE_CLASSIFICATION_REPORT.md exists (if detective was spawned):**
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
[ -f "${OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
|
|
216
|
+
```
|
|
217
|
+
</step>
|
|
218
|
+
|
|
219
|
+
<step name="print_summary">
|
|
220
|
+
## Step 6: Print Validation Summary
|
|
221
|
+
|
|
222
|
+
Print a human-readable summary of the validation results.
|
|
223
|
+
|
|
224
|
+
```
|
|
225
|
+
=== Validation Complete ===
|
|
226
|
+
|
|
227
|
+
Test Directory: {TEST_DIR}
|
|
228
|
+
Files Validated: {file_count}
|
|
229
|
+
Fix Loops Used: {fix_loops_used} of 3
|
|
230
|
+
|
|
231
|
+
Layer Results:
|
|
232
|
+
Syntax: {PASS|FAIL}
|
|
233
|
+
Structure: {PASS|FAIL}
|
|
234
|
+
Dependencies: {PASS|FAIL}
|
|
235
|
+
Logic: {PASS|FAIL}
|
|
236
|
+
|
|
237
|
+
Issues:
|
|
238
|
+
Found: {issues_found}
|
|
239
|
+
Auto-fixed: {issues_fixed}
|
|
240
|
+
Unresolved: {unresolved_count}
|
|
241
|
+
|
|
242
|
+
Overall Status: {PASS|PASS_WITH_WARNINGS|FAIL}
|
|
243
|
+
Confidence: {HIGH|MEDIUM|LOW}
|
|
244
|
+
|
|
245
|
+
Artifacts Produced:
|
|
246
|
+
- {OUTPUT_DIR}/VALIDATION_REPORT.md
|
|
247
|
+
- {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md (if --classify used)
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
**If bug detective ran, add classification summary:**
|
|
251
|
+
|
|
252
|
+
```
|
|
253
|
+
Failure Classification:
|
|
254
|
+
APPLICATION BUG: {app_bug_count}
|
|
255
|
+
TEST CODE ERROR: {test_error_count}
|
|
256
|
+
ENVIRONMENT ISSUE: {env_issue_count}
|
|
257
|
+
INCONCLUSIVE: {inconclusive_count}
|
|
258
|
+
Auto-fixed: {auto_fixes_applied}
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
```
|
|
262
|
+
===========================
|
|
263
|
+
```
|
|
264
|
+
</step>
|
|
265
|
+
|
|
266
|
+
</process>
|
|
267
|
+
|
|
268
|
+
<output>
|
|
269
|
+
This workflow validates existing test files and produces quality reports.
|
|
270
|
+
|
|
271
|
+
**Artifacts produced:**
|
|
272
|
+
|
|
273
|
+
| Artifact | When Produced | Description |
|
|
274
|
+
|----------|---------------|-------------|
|
|
275
|
+
| VALIDATION_REPORT.md | Always | 4-layer validation results (Syntax, Structure, Dependencies, Logic) with fix loop history and confidence level |
|
|
276
|
+
| FAILURE_CLASSIFICATION_REPORT.md | When --classify flag set AND failures exist | Per-failure classification (APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, INCONCLUSIVE) with evidence and auto-fix log |
|
|
277
|
+
|
|
278
|
+
**Side effects:**
|
|
279
|
+
- Test files may be modified in-place by HIGH-confidence auto-fixes from the validator
|
|
280
|
+
- Test files may be modified by HIGH-confidence TEST CODE ERROR fixes from the bug detective
|
|
281
|
+
- No git branches created (fixes applied to working tree)
|
|
282
|
+
- No PRs created
|
|
283
|
+
</output>
|
|
284
|
+
|
|
285
|
+
<error_handling>
|
|
286
|
+
| Error | Cause | Action |
|
|
287
|
+
|-------|-------|--------|
|
|
288
|
+
| Test directory does not exist | Invalid path argument | Print error with path, STOP |
|
|
289
|
+
| No test directory found | No common test dirs in project | Print error listing searched dirs, suggest providing path, STOP |
|
|
290
|
+
| No test files found in directory | Directory exists but contains no test files | Print error with expected patterns, STOP |
|
|
291
|
+
| VALIDATION_REPORT.md missing | Validator failed silently | Print error, STOP |
|
|
292
|
+
| Fix loop exhausted (3 loops) | Issues cannot be auto-resolved | Validator checkpoints with unresolved issue details |
|
|
293
|
+
| Test runner not detected | No test framework config found | Bug detective checkpoints for user to specify runner |
|
|
294
|
+
| Test runner fails to start | Missing dependencies or broken config | Bug detective classifies as ENVIRONMENT ISSUE |
|
|
295
|
+
</error_handling>
|