qaa-agent 1.9.2 → 1.9.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. package/CHANGELOG.md +206 -179
  2. package/CLAUDE.md +718 -557
  3. package/README.md +384 -357
  4. package/VERSION +1 -1
  5. package/agents/qa-pipeline-orchestrator.md +1739 -1425
  6. package/agents/qaa-analyzer.md +0 -1
  7. package/agents/qaa-bug-detective.md +790 -655
  8. package/agents/qaa-codebase-mapper.md +50 -1
  9. package/agents/qaa-discovery.md +421 -384
  10. package/agents/qaa-e2e-runner.md +715 -577
  11. package/agents/qaa-executor.md +947 -830
  12. package/agents/qaa-planner.md +14 -1
  13. package/agents/qaa-project-researcher.md +533 -400
  14. package/agents/qaa-scanner.md +77 -1
  15. package/agents/qaa-testid-injector.md +0 -1
  16. package/agents/qaa-validator.md +86 -1
  17. package/bin/install.cjs +375 -253
  18. package/bin/lib/context7-cache.cjs +299 -0
  19. package/bin/lib/intent-detector.cjs +488 -0
  20. package/commands/qa-audit.md +255 -126
  21. package/commands/qa-create-test.md +666 -365
  22. package/commands/qa-fix.md +684 -513
  23. package/commands/qa-map.md +283 -139
  24. package/commands/qa-pr.md +63 -0
  25. package/commands/qa-research.md +181 -157
  26. package/commands/qa-start.md +62 -6
  27. package/commands/qa-test-report.md +219 -219
  28. package/frameworks/cypress.json +54 -0
  29. package/frameworks/jest.json +59 -0
  30. package/frameworks/playwright.json +58 -0
  31. package/frameworks/pytest.json +59 -0
  32. package/frameworks/robot-framework.json +54 -0
  33. package/frameworks/selenium.json +65 -0
  34. package/frameworks/vitest.json +57 -0
  35. package/package.json +5 -2
  36. package/workflows/qa-analyze.md +100 -4
  37. package/workflows/qa-from-ticket.md +50 -2
  38. package/workflows/qa-gap.md +50 -2
  39. package/workflows/qa-start.md +2048 -1405
  40. package/workflows/qa-testid.md +50 -2
  41. package/workflows/qa-validate.md +50 -2
@@ -1,513 +1,684 @@
1
- # QA Fix & Validate
2
-
3
- Validate, diagnose, and fix test files — all in one command. Runs 4-layer static validation (syntax, structure, dependencies, logic), classifies failures, and auto-fixes TEST CODE ERRORS. Uses Playwright MCP to reproduce E2E failures and verify locators against the live app when available. Never touches application code.
4
-
5
- ## Usage
6
-
7
- ```
8
- /qa-fix [<test-files-or-directory>] [options]
9
- ```
10
-
11
- ### Options
12
-
13
- - `<test-files-or-directory>` one or more test file paths or a directory (auto-detects if omitted)
14
- - `--check` **final check mode**: full quality verification against company preferences, codebase conventions, and execution
15
- - `--validate-only` run 4-layer static validation only, no test execution or classification
16
- - `--classify` run tests and classify failures, but do NOT auto-fix
17
- - `--run --app-url <url>` — also execute E2E tests against live app after static validation
18
- - `--app-url <url>` URL of running application (auto-detects if not provided)
19
- - `[error output]` — paste test runner output directly (skips running tests, classifies the pasted output)
20
-
21
- ### Mode Detection
22
-
23
- ```
24
- if --check:
25
- MODE = "check" → full quality check + execution + QA_CHECK_REPORT.md
26
- elif --validate-only:
27
- MODE = "validate" → 4-layer static validation + VALIDATION_REPORT.md
28
- elif --classify:
29
- MODE = "classify" → run tests + classify failures (no auto-fix)
30
- else:
31
- MODE = "fix" run tests + classify + auto-fix TEST CODE ERRORS (default)
32
- ```
33
-
34
- ## What It Produces
35
-
36
- | Mode | Artifacts |
37
- |------|-----------|
38
- | check | QA_CHECK_REPORT.md (full quality verification with pass/fail per test file) |
39
- | check --ticket | QA_CHECK_REPORT.md + UAT_VERIFICATION.md (step-by-step screenshots vs ticket acceptance criteria) |
40
- | validate | VALIDATION_REPORT.md (syntax, structure, dependencies, logic per file) |
41
- | classify | FAILURE_CLASSIFICATION_REPORT.md (per-failure evidence, no fixes) |
42
- | fix | FAILURE_CLASSIFICATION_REPORT.md + auto-fixed test files |
43
-
44
- ## Instructions
45
-
46
- ### Step 1: Detect Mode and Test Directory
47
-
48
- Parse `$ARGUMENTS` for mode flags and test directory path.
49
-
50
- If no test directory provided, auto-detect:
51
- - Check for `tests/`, `cypress/`, `__tests__/`, `e2e/`, `spec/` directories
52
- - Check test config files for `testDir` setting
53
-
54
- Print banner:
55
- ```
56
- === QA Fix & Validate ===
57
- Mode: {validate | classify | fix}
58
- Test Directory: {path}
59
- App URL: {url or "auto-detect"}
60
- ==========================
61
- ```
62
-
63
- ---
64
-
65
- ### CHECK MODE (`--check`) — Final Quality Verification
66
-
67
- Full quality check for specific test files. Reads ALL context sources, verifies every aspect of the tests, runs them, and produces a pass/fail report. Use this as a final gate before delivering tests.
68
-
69
- **Accepts specific files:**
70
- ```
71
- /qa-fix --check tests/e2e/login.e2e.spec.ts tests/e2e/checkout.e2e.spec.ts
72
- /qa-fix --check tests/unit/auth.unit.spec.ts
73
- /qa-fix --check tests/e2e/ --app-url http://localhost:3000
74
- ```
75
-
76
- **Step 1: Read ALL context sources**
77
-
78
- Read every available context source — this is not optional, all must be read:
79
-
80
- 1. **CLAUDE.md** — QA standards, POM rules, locator tiers, assertion rules, naming conventions, quality gates
81
- 2. **~/.claude/qaa/MY_PREFERENCES.md** company/user preferences that OVERRIDE CLAUDE.md rules
82
- 3. **Codebase map** (`.qa-output/codebase/`):
83
- - `CODE_PATTERNS.md` naming conventions, import style, file organization (are tests matching the project's style?)
84
- - `API_CONTRACTS.md` — real API shapes (are API test payloads correct?)
85
- - `TEST_SURFACE.md` function signatures (are test targets real?)
86
- - `TESTABILITY.md` — mock boundaries (are mocks set up correctly?)
87
- 4. **Locator Registry** (`.qa-output/locators/`) real locators from the app (are POM locators accurate?)
88
- 5. **Existing test patterns** — read 2-3 existing test files in the same repo to understand current conventions (describe block style, import patterns, assertion patterns, fixture usage)
89
-
90
- If codebase map is missing, STOP and tell the user to run `/qa-map` first.
91
-
92
- **Step 2: Verify each test file across 7 dimensions**
93
-
94
- For EACH selected test file, check:
95
-
96
- | # | Dimension | What to check | Source |
97
- |---|-----------|---------------|--------|
98
- | 1 | **Naming** | File name, test IDs, describe/it names follow conventions | CLAUDE.md + CODE_PATTERNS.md + MY_PREFERENCES.md |
99
- | 2 | **Structure** | Correct directory, imports resolve, follows repo patterns | CODE_PATTERNS.md + existing tests in repo |
100
- | 3 | **Locators** | POM locators match registry, Tier 1 preferred, no stale selectors | LOCATOR_REGISTRY.md |
101
- | 4 | **Assertions** | Concrete values (no toBeTruthy alone), match API contracts | CLAUDE.md + API_CONTRACTS.md |
102
- | 5 | **POM compliance** | No assertions in POMs, locators as properties, extends BasePage | CLAUDE.md |
103
- | 6 | **Code quality** | No redundant code, no dead code, no hardcoded credentials, no copy-paste | Code review |
104
- | 7 | **Company conventions** | Matches all rules in MY_PREFERENCES.md | MY_PREFERENCES.md |
105
-
106
- **Step 3: Run the tests**
107
-
108
- Execute the selected test files:
109
-
110
- ```bash
111
- # Detect test runner from project config
112
- npx playwright test {files} --reporter=json 2>&1 # if Playwright
113
- npx cypress run --spec {files} 2>&1 # if Cypress
114
- npx jest {files} --json 2>&1 # if Jest
115
- npx vitest run {files} --reporter=json 2>&1 # if Vitest
116
- ```
117
-
118
- If E2E tests and app URL available, also verify with Playwright MCP:
119
- - Navigate to each page referenced in the tests
120
- - `browser_snapshot()` to verify elements exist in DOM
121
- - Cross-reference locators against real page
122
-
123
- **Step 4: Fix issues found**
124
-
125
- For each issue found:
126
- - **AUTO-FIX** (HIGH confidence): naming, imports, locator mismatches, missing await, Tier 4→Tier 1 upgrade when registry has the value
127
- - **FLAG for review** (MEDIUM/LOW): logic changes, assertion value changes, structural refactors
128
- - Re-run tests after fixes (max 5 loops)
129
-
130
- **Step 5: Produce QA_CHECK_REPORT.md**
131
-
132
- ```markdown
133
- # QA Check Report
134
-
135
- ## Summary
136
-
137
- | Metric | Value |
138
- |--------|-------|
139
- | Files checked | {N} |
140
- | Dimensions checked | 7 |
141
- | Issues found | {N} |
142
- | Auto-fixed | {N} |
143
- | Flagged for review | {N} |
144
- | Tests passed | {N}/{total} |
145
- | Overall | PASS / PASS WITH WARNINGS / FAIL |
146
-
147
- ## Per-File Results
148
-
149
- ### {file_path}
150
-
151
- | Dimension | Status | Details |
152
- |-----------|--------|---------|
153
- | Naming | PASS/FAIL | {specific details} |
154
- | Structure | PASS/FAIL | {specific details} |
155
- | Locators | PASS/FAIL | {specific details} |
156
- | Assertions | PASS/FAIL | {specific details} |
157
- | POM compliance | PASS/FAIL | {specific details} |
158
- | Code quality | PASS/FAIL | {specific details} |
159
- | Company conventions | PASS/FAIL | {specific details} |
160
-
161
- **Test execution:** PASS / FAIL ({error if failed})
162
- **Fixes applied:** {list of auto-fixes}
163
- **Flagged for review:** {list of items needing human review}
164
-
165
- [... repeat per file ...]
166
-
167
- ## Flagged Items (Needs Human Review)
168
-
169
- | File | Dimension | Issue | Suggested Fix |
170
- |------|-----------|-------|---------------|
171
- | ... | ... | ... | ... |
172
- ```
173
-
174
- Write to `.qa-output/QA_CHECK_REPORT.md`.
175
-
176
- Present results to user with clear PASS/FAIL per file and overall status.
177
-
178
- **Step 6 (optional): Ticket Verification (`--ticket <source>`)**
179
-
180
- If `--ticket` flag is provided, perform UAT verification — walk through the test flow step-by-step in the browser, take screenshots at each step, and compare against the ticket's acceptance criteria.
181
-
182
- **Usage:**
183
- ```
184
- /qa-fix --check --ticket #123 tests/e2e/login.e2e.spec.ts --app-url http://localhost:3000
185
- /qa-fix --check --ticket https://company.atlassian.net/browse/PROJ-456 tests/e2e/checkout.e2e.spec.ts
186
- /qa-fix --check --ticket "User logs in, sees dashboard with welcome message, clicks profile" tests/e2e/login.e2e.spec.ts
187
- ```
188
-
189
- **Requires:** `--app-url` or auto-detected running app. Cannot do ticket verification without a live app.
190
-
191
- **Step 6a: Fetch and parse the ticket**
192
-
193
- Same ticket parsing as `/qa-create-test` from-ticket mode:
194
- - GitHub Issue: `gh issue view` extract title, body, ACs
195
- - Jira/Linear URL: `WebFetch` → extract content
196
- - Plain text: use directly as acceptance criteria
197
- - File path: read file content
198
-
199
- Extract:
200
- - Acceptance criteria (AC-1, AC-2, ...)
201
- - Expected user flow (step-by-step)
202
- - Expected outcomes per step
203
-
204
- **Step 6b: Walk through the flow with Playwright MCP**
205
-
206
- For each E2E test file being checked, replay the user journey manually in the browser step-by-step:
207
-
208
- ```
209
- For each step in the ticket's user flow:
210
-
211
- 1. Execute the action described in the step:
212
- - Navigate: mcp__playwright__browser_navigate({ url: "{page}" })
213
- - Fill form: mcp__playwright__browser_fill_form({ ... })
214
- - Click: mcp__playwright__browser_click({ element: "..." })
215
- - Wait: mcp__playwright__browser_wait_for({ text: "..." })
216
-
217
- 2. Take screenshot AFTER the action:
218
- mcp__playwright__browser_take_screenshot()
219
- Save to .qa-output/uat-screenshots/{test-name}-step-{N}.png
220
-
221
- 3. Take accessibility snapshot to read page state:
222
- mcp__playwright__browser_snapshot()
223
-
224
- 4. Record what the page shows:
225
- - URL after action
226
- - Visible text/headings
227
- - Form state
228
- - Error messages (if any)
229
- - Elements visible/hidden
230
- ```
231
-
232
- **Step 6c: Compare actual vs ticket**
233
-
234
- For each acceptance criterion from the ticket:
235
-
236
- | AC | Expected (from ticket) | Actual (from browser) | Screenshot | Verdict |
237
- |----|----------------------|---------------------|------------|---------|
238
- | AC-1 | User sees login form | Login form visible with email/password fields | step-1.png | MATCH |
239
- | AC-2 | After login, redirect to dashboard | Redirected to /dashboard, "Welcome" visible | step-3.png | MATCH |
240
- | AC-3 | Error message for wrong password | "Invalid credentials" alert shown | step-5.png | MATCH |
241
- | AC-4 | Remember me keeps session | Session persists after browser close | step-7.png | MISMATCH — session expired |
242
-
243
- Verdicts:
244
- - **MATCH** actual behavior matches what the ticket describes
245
- - **MISMATCH** — actual behavior differs from ticket (could be app bug OR test not covering this AC)
246
- - **NOT TESTED** ticket has an AC but no test step covers it
247
- - **EXTRA** — test covers something not in the ticket (informational, not a failure)
248
-
249
- **Step 6d: Produce UAT_VERIFICATION.md**
250
-
251
- ```markdown
252
- # UAT Verification Report
253
-
254
- ## Ticket Info
255
-
256
- | Field | Value |
257
- |-------|-------|
258
- | Source | {ticket URL or text} |
259
- | Title | {ticket title} |
260
- | ACs extracted | {count} |
261
- | Test files verified | {count} |
262
-
263
- ## Step-by-Step Walkthrough
264
-
265
- ### Step 1: {action description}
266
- - **Action:** Navigate to /login
267
- - **Screenshot:** [step-1.png](.qa-output/uat-screenshots/{test}-step-1.png)
268
- - **Page state:** Login form visible, email and password fields empty, "Log in" button enabled
269
- - **Matches AC:** AC-1 ✓
270
-
271
- ### Step 2: {action description}
272
- - **Action:** Fill email "test@example.com", password "SecureP@ss123!"
273
- - **Screenshot:** [step-2.png]
274
- - **Page state:** Fields filled, button still enabled
275
- - **Matches AC:** (intermediate step, no AC)
276
-
277
- [... repeat per step ...]
278
-
279
- ## AC Coverage Matrix
280
-
281
- | AC | Description | Tested | Verdict | Evidence |
282
- |----|-------------|--------|---------|----------|
283
- | AC-1 | Login form visible | Yes | MATCH | step-1.png |
284
- | AC-2 | Redirect to dashboard | Yes | MATCH | step-3.png |
285
- | AC-3 | Error on wrong password | Yes | MATCH | step-5.png |
286
- | AC-4 | Remember me session | No | NOT TESTED | — |
287
-
288
- ## Summary
289
-
290
- | Metric | Value |
291
- |--------|-------|
292
- | ACs from ticket | {N} |
293
- | ACs matched | {N} |
294
- | ACs mismatched | {N} |
295
- | ACs not tested | {N} |
296
- | Screenshots captured | {N} |
297
- | Overall | PASS / PARTIAL / FAIL |
298
- ```
299
-
300
- Write to `.qa-output/UAT_VERIFICATION.md`.
301
-
302
- If any AC is MISMATCH or NOT TESTED, present to user with recommendation:
303
- - MISMATCH → "AC-4 says X but the app does Y — either the app has a bug or the test needs updating"
304
- - NOT TESTED → "AC-4 is not covered by any test step — consider adding a test case"
305
-
306
- ---
307
-
308
- ### VALIDATE MODE (`--validate-only`)
309
-
310
- 1. Read `CLAUDE.md` — quality gates, locator tiers, assertion rules.
311
- 2. Execute static validation workflow:
312
-
313
- Follow the workflow defined in `@workflows/qa-validate.md` end-to-end.
314
- Preserve all workflow gates (fix loops, layer checks).
315
-
316
- The validator runs 4 layers per file:
317
- 1. **Syntax** code compiles without errors
318
- 2. **Structure**naming, folders, POM compliance
319
- 3. **Dependencies** — all imports resolve
320
- 4. **Logic** — concrete assertions, Tier 1-2 locators, no assertions in POMs
321
-
322
- **Optional Layer 5 (if Playwright MCP connected + app URL available):**
323
- - Navigate to each page referenced in E2E tests via `mcp__playwright__browser_navigate`
324
- - Take accessibility snapshot via `mcp__playwright__browser_snapshot`
325
- - Cross-reference test locators against real DOM elements
326
- - Flag locators that don't match, auto-fix mismatches
327
-
328
- Max 5 fix loop iterations. Produces VALIDATION_REPORT.md.
329
-
330
- If `--run` flag is also present and E2E test files exist, invoke E2E runner after static validation:
331
-
332
- Task(
333
- prompt="
334
- <objective>Run E2E tests against live application, capture real locators, fix mismatches, loop until pass. Query Context7 MCP to verify framework selector syntax before fixing locators.</objective>
335
- <execution_context>@agents/qaa-e2e-runner.md</execution_context>
336
- <files_to_read>
337
- - CLAUDE.md
338
- - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
339
- - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
340
- - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
341
- - .qa-output/research/E2E_STRATEGY.md (if exists)
342
- - {E2E test files from validated directory}
343
- - {POM files from validated directory}
344
- </files_to_read>
345
- <parameters>
346
- app_url: {from --app-url flag or auto-detect}
347
- output_dir: .qa-output
348
- </parameters>
349
- "
350
- )
351
-
352
- ---
353
-
354
- ### CLASSIFY MODE (`--classify`)
355
-
356
- Same as fix mode below but skip Step 4 (auto-fix). Only classify and report.
357
-
358
- ---
359
-
360
- ### FIX MODE (default)
361
-
362
- Fix mode runs in two phases: **analyze first, then fix after user confirmation.**
363
-
364
- #### Phase 1: Analyze & Present Plan (always runs)
365
-
366
- 1. Read `CLAUDE.md` — classification rules, locator tiers, assertion quality.
367
- 2. Invoke bug-detective agent in **classify-only mode** (no auto-fixes):
368
-
369
- Task(
370
- prompt="
371
- <objective>Run tests and classify failures. Do NOT auto-fix anything yet — this is the analysis phase. Use Playwright MCP to reproduce E2E failures in the browser when available — navigate to failing pages, snapshot DOM, reproduce actions, and screenshot failure state for evidence. Query Context7 MCP to verify framework syntax when diagnosing failures.</objective>
372
- <execution_context>@agents/qaa-bug-detective.md</execution_context>
373
- <files_to_read>
374
- - CLAUDE.md
375
- - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
376
- - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
377
- - .qa-output/codebase/CODE_PATTERNS.md (if exists)
378
- - .qa-output/codebase/API_CONTRACTS.md (if exists)
379
- - .qa-output/codebase/TEST_SURFACE.md (if exists)
380
- - .qa-output/codebase/TESTABILITY.md (if exists)
381
- - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
382
- - .qa-output/research/TESTING_STACK.md (if exists)
383
- </files_to_read>
384
- <parameters>
385
- user_input: $ARGUMENTS
386
- mode: classify-only
387
- app_url: {auto-detect from test config baseURL, or ask user}
388
- </parameters>
389
- "
390
- )
391
-
392
- 3. Present the analysis to the user as a **fix plan**:
393
-
394
- ```
395
- === FIX PLAN ===
396
-
397
- Tests run: {N}
398
- Passed: {N}
399
- Failed: {N}
400
-
401
- Failures classified:
402
- APPLICATION BUG: {N} (will NOT be touched)
403
- TEST CODE ERROR: {N} (can auto-fix)
404
- ENVIRONMENT ISSUE: {N} (resolution steps provided)
405
- INCONCLUSIVE: {N} (needs more info)
406
-
407
- Proposed auto-fixes (TEST CODE ERRORS only):
408
- 1. {file}:{line} {description of fix} [HIGH confidence]
409
- 2. {file}:{line} — {description of fix} [HIGH confidence]
410
- 3. {file}:{line} {description of fix} [MEDIUM flagged for review]
411
-
412
- APPLICATION BUGs found (for developer action):
413
- 1. {file}:{line} {description}
414
- 2. {file}:{line} {description}
415
-
416
- Proceed with auto-fixes? [yes / modify / cancel]
417
- ================
418
- ```
419
-
420
- 4. **Wait for user confirmation.** Do NOT proceed until the user approves. This is a refinement loop — repeat until the user is satisfied:
421
-
422
- - **"yes"** / **"proceed"** / **"dale"** continue to Phase 2
423
- - **"cancel"** / **"no"** stop, deliver only the FAILURE_CLASSIFICATION_REPORT.md (same as --classify)
424
- - **Any other response** (feedback, modifications, additions) → treat as a refinement request:
425
- - Adjust the fix plan based on the user's instructions (add fixes, remove fixes, change approach, add new checks)
426
- - Re-present the updated Fix Plan showing what changed
427
- - Wait for user confirmation again
428
- - Repeat this loop until the user says "yes" or "cancel"
429
-
430
- **Examples of refinement requests:**
431
- - "esto está bien pero también quiero que arregles los imports de utils" → add that fix to the plan
432
- - "no toques el archivo de login, dejalo como está" → remove that file from the plan
433
- - "cambiá el selector por getByRole en vez de getByTestId" → update the proposed fix
434
- - "agregá también una validación de que el status sea 200" add assertion fix to the plan
435
-
436
- #### Phase 2: Execute Fixes (only after user confirmation)
437
-
438
- 5. Invoke bug-detective agent in **fix mode** with the confirmed plan:
439
-
440
- Task(
441
- prompt="
442
- <objective>Auto-fix the confirmed TEST CODE ERRORS from the analysis phase. Use Playwright MCP to reproduce E2E failures in the browser when available. Query Context7 MCP to verify correct framework syntax before applying any fix.</objective>
443
- <execution_context>@agents/qaa-bug-detective.md</execution_context>
444
- <files_to_read>
445
- - CLAUDE.md
446
- - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
447
- - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
448
- - .qa-output/codebase/CODE_PATTERNS.md (if exists)
449
- - .qa-output/codebase/API_CONTRACTS.md (if exists)
450
- - .qa-output/codebase/TEST_SURFACE.md (if exists)
451
- - .qa-output/codebase/TESTABILITY.md (if exists)
452
- - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
453
- - .qa-output/research/TESTING_STACK.md (if exists)
454
- - .qa-output/FAILURE_CLASSIFICATION_REPORT.md
455
- </files_to_read>
456
- <parameters>
457
- user_input: $ARGUMENTS
458
- mode: fix
459
- app_url: {auto-detect from test config baseURL, or ask user}
460
- </parameters>
461
- "
462
- )
463
-
464
- **Classification categories:**
465
- - **APPLICATION BUG** error in production code → Report only, NEVER auto-fix
466
- - **TEST CODE ERROR** error in test code → Auto-fix if HIGH confidence
467
- - **ENVIRONMENT ISSUE** — missing env, connection refused → Report with resolution steps
468
- - **INCONCLUSIVE** — ambiguous → Report what's known, ask for more info
469
-
470
- **Auto-fix rules:**
471
- - Only TEST CODE ERROR at HIGH confidence
472
- - Allowed fixes: import paths, selectors, assertion values, config, missing await, fixture paths
473
- - Every fix verified by re-running the specific test
474
- - Never modify application code (src/, app/, lib/)
475
-
476
- **Browser reproduction (when Playwright MCP connected):**
477
- - Navigate to failing page → snapshot DOM → reproduce action → screenshot
478
- - Element not in DOM → TEST CODE ERROR (HIGH confidence)
479
- - Element exists, wrong behavior APPLICATION BUG
480
- - Page doesn't load → ENVIRONMENT ISSUE
481
-
482
- 6. Present results. APPLICATION BUGs are reported for developer action, not auto-fixed.
483
-
484
- $ARGUMENTS
485
-
486
- ## MANDATORY verification — run ALL commands below, no exceptions, no skipping
487
-
488
- Before returning control, copy-paste and run this ENTIRE block. Do NOT decide which commands "apply" — run all of them every time. The output confirms what happened; you do not get to assume the answer.
489
-
490
- ```bash
491
- echo "=== QA-FIX CHECKLIST START ==="
492
- echo "1. Failure classification report:"
493
- ls .qa-output/FAILURE_CLASSIFICATION_REPORT.md 2>/dev/null || echo "NO_CLASSIFICATION_REPORT"
494
- echo "2. Locator Registry:"
495
- ls .qa-output/locators/ 2>/dev/null || echo "NO_LOCATORS_FOUND"
496
- echo "3. MY_PREFERENCES.md:"
497
- cat ~/.claude/qaa/MY_PREFERENCES.md 2>/dev/null || echo "FILE_NOT_FOUND"
498
- echo "4. Codebase map (context for bug-detective):"
499
- ls .qa-output/codebase/ 2>/dev/null || echo "NO_CODEBASE_MAP"
500
- echo "5. Test files in scope:"
501
- find tests/ cypress/ __tests__/ e2e/ spec/ -type f -name "*.spec.*" -o -name "*.test.*" -o -name "*.e2e.*" 2>/dev/null | head -20 || echo "NO_TEST_FILES_FOUND"
502
- echo "6. MCP evidence (if browser was used):"
503
- ls .qa-output/mcp-evidence/ 2>/dev/null || echo "NO_MCP_EVIDENCE"
504
- echo "7. Classification categories in report:"
505
- grep -cE "APPLICATION BUG|TEST CODE ERROR|ENVIRONMENT ISSUE|INCONCLUSIVE" .qa-output/FAILURE_CLASSIFICATION_REPORT.md 2>/dev/null || echo "NO_CLASSIFICATIONS"
506
- echo "=== QA-FIX CHECKLIST END ==="
507
- ```
508
-
509
- **Rules:**
510
- - Run the block AS-IS. Do not modify it. Do not split it. Do not skip lines.
511
- - If any output shows a problem (NO_CLASSIFICATION_REPORT after fix mode completed), fix it before returning.
512
- - If output shows expected "not found" results (e.g., NO_MCP_EVIDENCE when no browser was used), that is fine — the point is you RAN the command instead of assuming the answer.
513
- - Do NOT mark this task as complete until the block has been executed and you have read every line of output.
1
+ # QA Fix & Validate
2
+
3
+ Validate, diagnose, and fix test files — all in one command. Runs 4-layer static validation (syntax, structure, dependencies, logic), classifies failures, and auto-fixes TEST CODE ERRORS. Uses Playwright MCP to reproduce E2E failures and verify locators against the live app when available. Never touches application code.
4
+
5
+
6
+ ## ⚠ MANDATORY: How to Execute This Command
7
+
8
+ Execute these steps IN ORDER. Every step is mandatory. Do NOT improvise,
9
+ reorder, skip, or invent steps.
10
+
11
+ 1. FIRST tool call: run the intent detector with `$ARGUMENTS` passed LITERALLY
12
+ (the actual command lives in the "## Intent Detection" section below).
13
+ 2. Print the INPUT DETECTION banner (exact output from intent-detector.cjs).
14
+ 3. Determine `ALL_FROM_FLAGS`. If interactive (`IS_AUTO=false`) AND any value
15
+ came from NL/default (`ALL_FROM_FLAGS=false`), PAUSE for yes/no confirmation.
16
+ Do NOT proceed without it.
17
+ ```bash
18
+ ALL_FROM_FLAGS=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); const sources=Object.values(j.resolved).map(v=>v.source); console.log(sources.length>0 && sources.every(s=>s==='flag') ? 'true' : 'false')")
19
+ ```
20
+ Pause prompt (when applicable):
21
+ ```
22
+ The values above were resolved from natural language and/or defaults.
23
+ Do you want to continue with these inputs? (yes/no)
24
+ ```
25
+ 4. Determine MODE from the resolved values (not from your own NL parsing).
26
+ 5. Print the "=== QA Fix & Validate ===" banner (Step 1 of Instructions).
27
+ 6. Execute the mode's flow IN ORDER. For FIX MODE specifically:
28
+ Phase 1 includes invoking the bug-detective via Task() in classify-only
29
+ mode. Phase 2 includes invoking the bug-detective via Task() in fix mode.
30
+ Do NOT apply fixes inline with Edit/Write — always go through the sub-agent.
31
+ Phase 1 (analyze present FIX PLAN WAIT for user confirmation) MUST
32
+ complete before Phase 2 (execute fixes).
33
+
34
+ ### DO NOT
35
+ - Skip the INPUT DETECTION banner or the "=== QA Fix ===" banner
36
+ - Run AskUserQuestion or any tool before the intent detector
37
+ - Present an AskUserQuestion in place of the FIX PLAN confirmation checkpoint
38
+ - Auto-fix anything before the user approves the FIX PLAN (Phase 1 → Phase 2 gate)
39
+ - Apply fixes inline with Edit/Write instead of spawning the bug-detective sub-agent
40
+ - Invent steps not in this command, or reorder the ones that are
41
+ - Assume context, app URL, framework, or any value from previous command
42
+ invocations (/qa-start, prior /qa-fix calls, etc.). Each /qa-fix invocation
43
+ is self-contained — the detector returns defaults precisely because the user
44
+ did not pass them.
45
+ - Modify the FIX PLAN scope after user approval without re-confirming. If the
46
+ analysis reveals the approved fix is insufficient or incorrect, STOP, present
47
+ the updated plan to the user, and wait for re-confirmation before applying.
48
+
49
+ If you find yourself improvising, presenting results before running the
50
+ analysis phase, applying fixes inline, or asking questions out of order —
51
+ STOP and restart from step 1.
52
+
53
+ ### Pass user input literally to the intent detector
54
+
55
+ When you run the intent detector with `$ARGUMENTS`, substitute that placeholder
56
+ with the RAW user input as received, NOT a pre-translated flag string. If you
57
+ pre-translate, the detector reports every value as `source: flag`, sets
58
+ `ALL_FROM_FLAGS=true`, and bypasses the confirmation checkpoint.
59
+
60
+
61
+ ## Usage
62
+
63
+ ```
64
+ /qa-fix [<test-files-or-directory>] [options]
65
+ ```
66
+
67
+ ### Options
68
+
69
+ - `<test-files-or-directory>` — one or more test file paths or a directory (auto-detects if omitted)
70
+ - `--check` — **final check mode**: full quality verification against company preferences, codebase conventions, and execution
71
+ - `--validate-only` run 4-layer static validation only, no test execution or classification
72
+ - `--classify` — run tests and classify failures, but do NOT auto-fix
73
+ - `--run --app-url <url>` — also execute E2E tests against live app after static validation
74
+ - `--app-url <url>` — URL of running application (auto-detects if not provided)
75
+ - `[error output]` — paste test runner output directly (skips running tests, classifies the pasted output)
76
+
77
+ ### Mode Detection
78
+
79
+ ```
80
+ if --check:
81
+ MODE = "check" → full quality check + execution + QA_CHECK_REPORT.md
82
+ elif --validate-only:
83
+ MODE = "validate" → 4-layer static validation + VALIDATION_REPORT.md
84
+ elif --classify:
85
+ MODE = "classify" → run tests + classify failures (no auto-fix)
86
+ else:
87
+ MODE = "fix" → run tests + classify + auto-fix TEST CODE ERRORS (default)
88
+ ```
89
+
90
+ ## What It Produces
91
+
92
+ | Mode | Artifacts |
93
+ |------|-----------|
94
+ | check | QA_CHECK_REPORT.md (full quality verification with pass/fail per test file) |
95
+ | check --ticket | QA_CHECK_REPORT.md + UAT_VERIFICATION.md (step-by-step screenshots vs ticket acceptance criteria) |
96
+ | validate | VALIDATION_REPORT.md (syntax, structure, dependencies, logic per file) |
97
+ | classify | FAILURE_CLASSIFICATION_REPORT.md (per-failure evidence, no fixes) |
98
+ | fix | FAILURE_CLASSIFICATION_REPORT.md + auto-fixed test files |
99
+
100
+
101
+ ## Intent Detection
102
+
103
+ Before processing arguments, run the intent detector to support natural-language input alongside flags. Flags always win over NL.
104
+
105
+ ```bash
106
+ RESOLVED=$(node ~/.claude/qaa/bin/lib/intent-detector.cjs --resolve "$ARGUMENTS" --aliases '{"app_url":"app-url","ticket":"ticket","check":"check","validate_only":"validate-only","classify":"classify","run":"run"}')
107
+
108
+ # Print INPUT DETECTION banner (always shown, even if defaults are used)
109
+ echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.banner)"
110
+
111
+ # Extract resolved values for use below
112
+ APP_URL=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.resolved.app_url?.value || '')")
113
+ TICKET=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.resolved.ticket?.value || '')")
114
+ ```
115
+
116
+ If you see values you didn't intend in the banner above (interactive mode), you can press Ctrl-C and re-run with explicit flags.
117
+
118
+ ## Instructions
119
+
120
+ ### Step 1: Detect Mode and Test Directory
121
+
122
+ Parse `$ARGUMENTS` for mode flags and test directory path.
123
+
124
+ If no test directory provided, auto-detect:
125
+ - Check for `tests/`, `cypress/`, `__tests__/`, `e2e/`, `spec/` directories
126
+ - Check test config files for `testDir` setting
127
+
128
+ Print banner:
129
+ ```
130
+ === QA Fix & Validate ===
131
+ Mode: {validate | classify | fix}
132
+ Test Directory: {path}
133
+ App URL: {url or "auto-detect"}
134
+ ==========================
135
+ ```
136
+
137
+ ---
138
+
139
+ ### CHECK MODE (`--check`) Final Quality Verification
140
+
141
+ Full quality check for specific test files. Reads ALL context sources, verifies every aspect of the tests, runs them, and produces a pass/fail report. Use this as a final gate before delivering tests.
142
+
143
+ **Accepts specific files:**
144
+ ```
145
+ /qa-fix --check tests/e2e/login.e2e.spec.ts tests/e2e/checkout.e2e.spec.ts
146
+ /qa-fix --check tests/unit/auth.unit.spec.ts
147
+ /qa-fix --check tests/e2e/ --app-url http://localhost:3000
148
+ ```
149
+
150
+ **Step 1: Read ALL context sources**
151
+
152
+ Read every available context source — this is not optional, all must be read:
153
+
154
+ 1. **CLAUDE.md** QA standards, POM rules, locator tiers, assertion rules, naming conventions, quality gates
155
+ 2. **~/.claude/qaa/MY_PREFERENCES.md** company/user preferences that OVERRIDE CLAUDE.md rules
156
+ 3. **Codebase map** (`.qa-output/codebase/`):
157
+ - `CODE_PATTERNS.md` naming conventions, import style, file organization (are tests matching the project's style?)
158
+ - `API_CONTRACTS.md` real API shapes (are API test payloads correct?)
159
+ - `TEST_SURFACE.md` function signatures (are test targets real?)
160
+ - `TESTABILITY.md` — mock boundaries (are mocks set up correctly?)
161
+ 4. **Locator Registry** (`.qa-output/locators/`) real locators from the app (are POM locators accurate?)
162
+ 5. **Existing test patterns** read 2-3 existing test files in the same repo to understand current conventions (describe block style, import patterns, assertion patterns, fixture usage)
163
+
164
+ If codebase map is missing, STOP and tell the user to run `/qa-map` first.
165
+
166
+ **Step 2: Verify each test file across 7 dimensions**
167
+
168
+ For EACH selected test file, check:
169
+
170
+ | # | Dimension | What to check | Source |
171
+ |---|-----------|---------------|--------|
172
+ | 1 | **Naming** | File name, test IDs, describe/it names follow conventions | CLAUDE.md + CODE_PATTERNS.md + MY_PREFERENCES.md |
173
+ | 2 | **Structure** | Correct directory, imports resolve, follows repo patterns | CODE_PATTERNS.md + existing tests in repo |
174
+ | 3 | **Locators** | POM locators match registry, Tier 1 preferred, no stale selectors | LOCATOR_REGISTRY.md |
175
+ | 4 | **Assertions** | Concrete values (no toBeTruthy alone), match API contracts | CLAUDE.md + API_CONTRACTS.md |
176
+ | 5 | **POM compliance** | No assertions in POMs, locators as properties, extends BasePage | CLAUDE.md |
177
+ | 6 | **Code quality** | No redundant code, no dead code, no hardcoded credentials, no copy-paste | Code review |
178
+ | 7 | **Company conventions** | Matches all rules in MY_PREFERENCES.md | MY_PREFERENCES.md |
179
+
180
+ **Step 3: Run the tests**
181
+
182
+ Execute the selected test files:
183
+
184
+ ```bash
185
+ # Detect test runner from project config
186
+ npx playwright test {files} --reporter=json 2>&1 # if Playwright
187
+ npx cypress run --spec {files} 2>&1 # if Cypress
188
+ npx jest {files} --json 2>&1 # if Jest
189
+ npx vitest run {files} --reporter=json 2>&1 # if Vitest
190
+ ```
191
+
192
+ If E2E tests and app URL available, you MAY also probe the DOM with Playwright MCP as supplementary evidence — but this is NOT test execution:
193
+
194
+ `[DOM_PROBE NOT_TEST_EXECUTION]` MCP DOM inspection only. It MAY confirm an element is present in the DOM; it MUST NOT be reported as a passing/verified test. The real test result comes from the runner above (npx playwright / cypress / jest / vitest). If the runner could NOT execute (binary missing, config error, host won't boot, ECONNREFUSED), HARD STOP: report ENVIRONMENT ISSUE, set `runner_executed: false`, and do NOT substitute the MCP probe for the run.
195
+
196
+ - Navigate to each page referenced in the tests
197
+ - `browser_snapshot()` to confirm elements exist in the DOM (probe, not a pass)
198
+ - Cross-reference locators against real page
199
+
200
+ **Step 4: Fix issues found**
201
+
202
+ For each issue found:
203
+ - **AUTO-FIX** (HIGH confidence): naming, imports, locator mismatches, missing await, Tier 4→Tier 1 upgrade when registry has the value
204
+ - **FLAG for review** (MEDIUM/LOW): logic changes, assertion value changes, structural refactors
205
+ - Re-run tests after fixes (max 5 loops)
206
+
207
+ **Step 5: Produce QA_CHECK_REPORT.md**
208
+
209
+ ```markdown
210
+ # QA Check Report
211
+
212
+ ## Summary
213
+
214
+ | Metric | Value |
215
+ |--------|-------|
216
+ | Files checked | {N} |
217
+ | Dimensions checked | 7 |
218
+ | Issues found | {N} |
219
+ | Auto-fixed | {N} |
220
+ | Flagged for review | {N} |
221
+ | Tests passed | {N}/{total} |
222
+ | Overall | PASS / PASS WITH WARNINGS / FAIL |
223
+
224
+ ## Per-File Results
225
+
226
+ ### {file_path}
227
+
228
+ | Dimension | Status | Details |
229
+ |-----------|--------|---------|
230
+ | Naming | PASS/FAIL | {specific details} |
231
+ | Structure | PASS/FAIL | {specific details} |
232
+ | Locators | PASS/FAIL | {specific details} |
233
+ | Assertions | PASS/FAIL | {specific details} |
234
+ | POM compliance | PASS/FAIL | {specific details} |
235
+ | Code quality | PASS/FAIL | {specific details} |
236
+ | Company conventions | PASS/FAIL | {specific details} |
237
+
238
+ **Test execution:** PASS / FAIL ({error if failed})
239
+ **Fixes applied:** {list of auto-fixes}
240
+ **Flagged for review:** {list of items needing human review}
241
+
242
+ [... repeat per file ...]
243
+
244
+ ## Flagged Items (Needs Human Review)
245
+
246
+ | File | Dimension | Issue | Suggested Fix |
247
+ |------|-----------|-------|---------------|
248
+ | ... | ... | ... | ... |
249
+ ```
250
+
251
+ Write to `.qa-output/QA_CHECK_REPORT.md`.
252
+
253
+ Present results to user with clear PASS/FAIL per file and overall status.
254
+
255
+ **Step 6 (optional): Ticket Verification (`--ticket <source>`)**
256
+
257
+ If `--ticket` flag is provided, perform UAT verification — walk through the test flow step-by-step in the browser, take screenshots at each step, and compare against the ticket's acceptance criteria.
258
+
259
+ **Usage:**
260
+ ```
261
+ /qa-fix --check --ticket #123 tests/e2e/login.e2e.spec.ts --app-url http://localhost:3000
262
+ /qa-fix --check --ticket https://company.atlassian.net/browse/PROJ-456 tests/e2e/checkout.e2e.spec.ts
263
+ /qa-fix --check --ticket "User logs in, sees dashboard with welcome message, clicks profile" tests/e2e/login.e2e.spec.ts
264
+ ```
265
+
266
+ **Requires:** `--app-url` or auto-detected running app. Cannot do ticket verification without a live app.
267
+
268
+ **Step 6a: Fetch and parse the ticket**
269
+
270
+ Same ticket parsing as `/qa-create-test` from-ticket mode:
271
+ - GitHub Issue: `gh issue view` → extract title, body, ACs
272
+ - Jira/Linear URL: `WebFetch` extract content
273
+ - Plain text: use directly as acceptance criteria
274
+ - File path: read file content
275
+
276
+ Extract:
277
+ - Acceptance criteria (AC-1, AC-2, ...)
278
+ - Expected user flow (step-by-step)
279
+ - Expected outcomes per step
280
+
281
+ **Step 6b: Walk through the flow with Playwright MCP**
282
+
283
+ For each E2E test file being checked, replay the user journey manually in the browser step-by-step:
284
+
285
+ ```
286
+ For each step in the ticket's user flow:
287
+
288
+ 1. Execute the action described in the step:
289
+ - Navigate: mcp__playwright__browser_navigate({ url: "{page}" })
290
+ - Fill form: mcp__playwright__browser_fill_form({ ... })
291
+ - Click: mcp__playwright__browser_click({ element: "..." })
292
+ - Wait: mcp__playwright__browser_wait_for({ text: "..." })
293
+
294
+ 2. Take screenshot AFTER the action:
295
+ mcp__playwright__browser_take_screenshot()
296
+ Save to .qa-output/uat-screenshots/{test-name}-step-{N}.png
297
+
298
+ 3. Take accessibility snapshot to read page state:
299
+ mcp__playwright__browser_snapshot()
300
+
301
+ 4. Record what the page shows:
302
+ - URL after action
303
+ - Visible text/headings
304
+ - Form state
305
+ - Error messages (if any)
306
+ - Elements visible/hidden
307
+ ```
308
+
309
+ **Step 6c: Compare actual vs ticket**
310
+
311
+ For each acceptance criterion from the ticket:
312
+
313
+ | AC | Expected (from ticket) | Actual (from browser) | Screenshot | Verdict |
314
+ |----|----------------------|---------------------|------------|---------|
315
+ | AC-1 | User sees login form | Login form visible with email/password fields | step-1.png | MATCH |
316
+ | AC-2 | After login, redirect to dashboard | Redirected to /dashboard, "Welcome" visible | step-3.png | MATCH |
317
+ | AC-3 | Error message for wrong password | "Invalid credentials" alert shown | step-5.png | MATCH |
318
+ | AC-4 | Remember me keeps session | Session persists after browser close | step-7.png | MISMATCH session expired |
319
+
320
+ Verdicts:
321
+ - **MATCH** — actual behavior matches what the ticket describes
322
+ - **MISMATCH** actual behavior differs from ticket (could be app bug OR test not covering this AC)
323
+ - **NOT TESTED** ticket has an AC but no test step covers it
324
+ - **EXTRA** test covers something not in the ticket (informational, not a failure)
325
+
326
+ **Step 6d: Produce UAT_VERIFICATION.md**
327
+
328
+ ```markdown
329
+ # UAT Verification Report
330
+
331
+ ## Ticket Info
332
+
333
+ | Field | Value |
334
+ |-------|-------|
335
+ | Source | {ticket URL or text} |
336
+ | Title | {ticket title} |
337
+ | ACs extracted | {count} |
338
+ | Test files verified | {count} |
339
+
340
+ ## Step-by-Step Walkthrough
341
+
342
+ ### Step 1: {action description}
343
+ - **Action:** Navigate to /login
344
+ - **Screenshot:** [step-1.png](.qa-output/uat-screenshots/{test}-step-1.png)
345
+ - **Page state:** Login form visible, email and password fields empty, "Log in" button enabled
346
+ - **Matches AC:** AC-1 ✓
347
+
348
+ ### Step 2: {action description}
349
+ - **Action:** Fill email "test@example.com", password "SecureP@ss123!"
350
+ - **Screenshot:** [step-2.png]
351
+ - **Page state:** Fields filled, button still enabled
352
+ - **Matches AC:** (intermediate step, no AC)
353
+
354
+ [... repeat per step ...]
355
+
356
+ ## AC Coverage Matrix
357
+
358
+ | AC | Description | Tested | Verdict | Evidence |
359
+ |----|-------------|--------|---------|----------|
360
+ | AC-1 | Login form visible | Yes | MATCH | step-1.png |
361
+ | AC-2 | Redirect to dashboard | Yes | MATCH | step-3.png |
362
+ | AC-3 | Error on wrong password | Yes | MATCH | step-5.png |
363
+ | AC-4 | Remember me session | No | NOT TESTED | — |
364
+
365
+ ## Summary
366
+
367
+ | Metric | Value |
368
+ |--------|-------|
369
+ | ACs from ticket | {N} |
370
+ | ACs matched | {N} |
371
+ | ACs mismatched | {N} |
372
+ | ACs not tested | {N} |
373
+ | Screenshots captured | {N} |
374
+ | Overall | PASS / PARTIAL / FAIL |
375
+ ```
376
+
377
+ Write to `.qa-output/UAT_VERIFICATION.md`.
378
+
379
+ If any AC is MISMATCH or NOT TESTED, present to user with recommendation:
380
+ - MISMATCH → "AC-4 says X but the app does Y — either the app has a bug or the test needs updating"
381
+ - NOT TESTED → "AC-4 is not covered by any test step — consider adding a test case"
382
+
383
+ ---
384
+
385
+ ### VALIDATE MODE (`--validate-only`)
386
+
387
+ 1. Read `CLAUDE.md` quality gates, locator tiers, assertion rules.
388
+ 2. Execute static validation workflow:
389
+
390
+ Follow the workflow defined in `@workflows/qa-validate.md` end-to-end.
391
+ Preserve all workflow gates (fix loops, layer checks).
392
+
393
+ The validator runs 4 layers per file:
394
+ 1. **Syntax** — code compiles without errors
395
+ 2. **Structure** naming, folders, POM compliance
396
+ 3. **Dependencies** — all imports resolve
397
+ 4. **Logic** — concrete assertions, Tier 1-2 locators, no assertions in POMs
398
+
399
+ **Optional Layer 5 — labeled DOM-probe locator cross-check (if Playwright MCP connected + app URL available):**
400
+
401
+ This layer is a STATIC locator cross-check via DOM probe. It is NOT test execution and never reports that tests passed:
402
+
403
+ ```
404
+ [DOM_PROBE NOT_TEST_EXECUTION]
405
+ MCP DOM probe — NOT test execution.
406
+
407
+ This observation comes from mcp__playwright__browser_* (live DOM inspection),
408
+ NOT from the test runner (cypress run / npx playwright test / jest / vitest).
409
+
410
+ It MAY assert ONLY: that the element was present in the DOM at the moment of
411
+ the probe.
412
+
413
+ It MUST NOT assert: that any test passes, that a proposed fix works, or that
414
+ the element is visible/actionable under the framework's own rules (Cypress
415
+ retry-ability, Playwright actionability checks, etc.).
416
+
417
+ Any classification resting SOLELY on this probe MUST be marked LOW confidence.
418
+ HIGH confidence requires a real test-runner execution with captured stdout/
419
+ stderr and exit code.
420
+ ```
421
+
422
+ - Navigate to each page referenced in E2E tests via `mcp__playwright__browser_navigate`
423
+ - Take accessibility snapshot via `mcp__playwright__browser_snapshot`
424
+ - Cross-reference test locators against real DOM elements
425
+ - Flag locators that don't match, auto-fix mismatches
426
+
427
+ Max 5 fix loop iterations. Produces VALIDATION_REPORT.md.
428
+
429
+ If `--run` flag is also present and E2E test files exist, invoke E2E runner after static validation:
430
+
431
+ Task(
432
+ prompt="
433
+ <critical_directive priority="MANDATORY">
434
+ You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
435
+ ~/.claude/qaa/agents/qaa-e2e-runner.md and adopt it as your operating contract.
436
+
437
+ Even though you may be running as a general-purpose agent (native subagent type
438
+ routing for QAA agents is not yet enabled), you MUST behave exactly as the
439
+ dedicated agent defined in that file: read everything in its <required_reading>,
440
+ obey its <quality_gate>, and run its mandatory bash checklist before returning.
441
+
442
+ Never modify application source code (src/, app/, lib/, or equivalent
443
+ directories) unless the agent's <scope> section explicitly permits it. If the
444
+ agent's contract restricts what files it can touch, honor that restriction —
445
+ even if a fix seems obvious or trivial.
446
+
447
+ If you cannot read ~/.claude/qaa/agents/qaa-e2e-runner.md (file missing, corrupted, parse
448
+ error), STOP and report the failure. Do NOT proceed with default agent
449
+ behavior that defeats the purpose of this directive.
450
+
451
+ Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
452
+ make, or skip any of the above. Skipping the agent's required_reading,
453
+ quality_gate, or bash checklist makes the run INVALID — its output cannot be
454
+ trusted.
455
+ </critical_directive>
456
+
457
+ <objective>Run E2E tests against live application, capture real locators, fix mismatches, loop until pass. Query Context7 MCP to verify framework selector syntax before fixing locators.</objective>
458
+ <execution_context>~/.claude/qaa/agents/qaa-e2e-runner.md</execution_context>
459
+ <files_to_read>
460
+ - CLAUDE.md
461
+ - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
462
+ - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
463
+ - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
464
+ - .qa-output/research/E2E_STRATEGY.md (if exists)
465
+ - {E2E test files from validated directory}
466
+ - {POM files from validated directory}
467
+ </files_to_read>
468
+ <parameters>
469
+ app_url: {from --app-url flag or auto-detect}
470
+ output_dir: .qa-output
471
+ </parameters>
472
+ "
473
+ )
474
+
475
+ ---
476
+
477
+ ### CLASSIFY MODE (`--classify`)
478
+
479
+ Same as fix mode below but skip Step 4 (auto-fix). Only classify and report.
480
+
481
+ ---
482
+
483
+ ### FIX MODE (default)
484
+
485
+ Fix mode runs in two phases: **analyze first, then fix after user confirmation.**
486
+
487
+ #### Phase 1: Analyze & Present Plan (always runs)
488
+
489
+ 1. Read `CLAUDE.md` — classification rules, locator tiers, assertion quality.
490
+ 2. Invoke bug-detective agent in **classify-only mode** (no auto-fixes):
491
+
492
+ Task(
493
+ prompt="
494
+ <critical_directive priority="MANDATORY">
495
+ You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
496
+ ~/.claude/qaa/agents/qaa-bug-detective.md and adopt it as your operating contract.
497
+
498
+ Even though you may be running as a general-purpose agent (native subagent type
499
+ routing for QAA agents is not yet enabled), you MUST behave exactly as the
500
+ dedicated agent defined in that file: read everything in its <required_reading>,
501
+ obey its <quality_gate>, and run its mandatory bash checklist before returning.
502
+
503
+ Never modify application source code (src/, app/, lib/, or equivalent
504
+ directories) unless the agent's <scope> section explicitly permits it. If the
505
+ agent's contract restricts what files it can touch, honor that restriction
506
+ even if a fix seems obvious or trivial.
507
+
508
+ If you cannot read ~/.claude/qaa/agents/qaa-bug-detective.md (file missing, corrupted, parse
509
+ error), STOP and report the failure. Do NOT proceed with default agent
510
+ behavior that defeats the purpose of this directive.
511
+
512
+ Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
513
+ make, or skip any of the above. Skipping the agent's required_reading,
514
+ quality_gate, or bash checklist makes the run INVALID — its output cannot be
515
+ trusted.
516
+ </critical_directive>
517
+
518
+ <objective>Run tests and classify failures. Do NOT auto-fix anything yet — this is the analysis phase. Use Playwright MCP to reproduce E2E failures in the browser when available — navigate to failing pages, snapshot DOM, reproduce actions, and screenshot failure state for evidence. Query Context7 MCP to verify framework syntax when diagnosing failures.</objective>
519
+ <execution_context>~/.claude/qaa/agents/qaa-bug-detective.md</execution_context>
520
+ <files_to_read>
521
+ - CLAUDE.md
522
+ - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
523
+ - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
524
+ - .qa-output/codebase/CODE_PATTERNS.md (if exists)
525
+ - .qa-output/codebase/API_CONTRACTS.md (if exists)
526
+ - .qa-output/codebase/TEST_SURFACE.md (if exists)
527
+ - .qa-output/codebase/TESTABILITY.md (if exists)
528
+ - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
529
+ - .qa-output/research/TESTING_STACK.md (if exists)
530
+ </files_to_read>
531
+ <parameters>
532
+ user_input: $ARGUMENTS
533
+ mode: classify-only
534
+ app_url: {auto-detect from test config baseURL, or ask user}
535
+ </parameters>
536
+ "
537
+ )
538
+
539
+ 3. Present the analysis to the user as a **fix plan**:
540
+
541
+ ```
542
+ === FIX PLAN ===
543
+
544
+ Tests run: {N}
545
+ Passed: {N}
546
+ Failed: {N}
547
+
548
+ Failures classified:
549
+ APPLICATION BUG: {N} (will NOT be touched)
550
+ TEST CODE ERROR: {N} (can auto-fix)
551
+ ENVIRONMENT ISSUE: {N} (resolution steps provided)
552
+ INCONCLUSIVE: {N} (needs more info)
553
+
554
+ Proposed auto-fixes (TEST CODE ERRORS only):
555
+ 1. {file}:{line} — {description of fix} [HIGH confidence]
556
+ 2. {file}:{line} — {description of fix} [HIGH confidence]
557
+ 3. {file}:{line} — {description of fix} [MEDIUM — flagged for review]
558
+
559
+ APPLICATION BUGs found (for developer action):
560
+ 1. {file}:{line} — {description}
561
+ 2. {file}:{line} — {description}
562
+
563
+ Proceed with auto-fixes? [yes / modify / cancel]
564
+ ================
565
+ ```
566
+
567
+ 4. **Wait for user confirmation.** Do NOT proceed until the user approves. This is a refinement loop — repeat until the user is satisfied:
568
+
569
+ - **"yes"** / **"proceed"** / **"dale"** → continue to Phase 2
570
+ - **"cancel"** / **"no"** → stop, deliver only the FAILURE_CLASSIFICATION_REPORT.md (same as --classify)
571
+ - **Any other response** (feedback, modifications, additions) → treat as a refinement request:
572
+ - Adjust the fix plan based on the user's instructions (add fixes, remove fixes, change approach, add new checks)
573
+ - Re-present the updated Fix Plan showing what changed
574
+ - Wait for user confirmation again
575
+ - Repeat this loop until the user says "yes" or "cancel"
576
+
577
+ **Examples of refinement requests:**
578
+ - "esto está bien pero también quiero que arregles los imports de utils" → add that fix to the plan
579
+ - "no toques el archivo de login, dejalo como está" → remove that file from the plan
580
+ - "cambiá el selector por getByRole en vez de getByTestId" → update the proposed fix
581
+ - "agregá también una validación de que el status sea 200" → add assertion fix to the plan
582
+
583
+ #### Phase 2: Execute Fixes (only after user confirmation)
584
+
585
+ 5. Invoke bug-detective agent in **fix mode** with the confirmed plan:
586
+
587
+ Task(
588
+ prompt="
589
+ <critical_directive priority="MANDATORY">
590
+ You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
591
+ ~/.claude/qaa/agents/qaa-bug-detective.md and adopt it as your operating contract.
592
+
593
+ Even though you may be running as a general-purpose agent (native subagent type
594
+ routing for QAA agents is not yet enabled), you MUST behave exactly as the
595
+ dedicated agent defined in that file: read everything in its <required_reading>,
596
+ obey its <quality_gate>, and run its mandatory bash checklist before returning.
597
+
598
+ Never modify application source code (src/, app/, lib/, or equivalent
599
+ directories) unless the agent's <scope> section explicitly permits it. If the
600
+ agent's contract restricts what files it can touch, honor that restriction —
601
+ even if a fix seems obvious or trivial.
602
+
603
+ If you cannot read ~/.claude/qaa/agents/qaa-bug-detective.md (file missing, corrupted, parse
604
+ error), STOP and report the failure. Do NOT proceed with default agent
605
+ behavior — that defeats the purpose of this directive.
606
+
607
+ Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
608
+ make, or skip any of the above. Skipping the agent's required_reading,
609
+ quality_gate, or bash checklist makes the run INVALID — its output cannot be
610
+ trusted.
611
+ </critical_directive>
612
+
613
+ <objective>Auto-fix the confirmed TEST CODE ERRORS from the analysis phase. Use Playwright MCP to reproduce E2E failures in the browser when available. Query Context7 MCP to verify correct framework syntax before applying any fix.</objective>
614
+ <execution_context>~/.claude/qaa/agents/qaa-bug-detective.md</execution_context>
615
+ <files_to_read>
616
+ - CLAUDE.md
617
+ - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
618
+ - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
619
+ - .qa-output/codebase/CODE_PATTERNS.md (if exists)
620
+ - .qa-output/codebase/API_CONTRACTS.md (if exists)
621
+ - .qa-output/codebase/TEST_SURFACE.md (if exists)
622
+ - .qa-output/codebase/TESTABILITY.md (if exists)
623
+ - .qa-output/research/FRAMEWORK_CAPABILITIES.md (if exists)
624
+ - .qa-output/research/TESTING_STACK.md (if exists)
625
+ - .qa-output/FAILURE_CLASSIFICATION_REPORT.md
626
+ </files_to_read>
627
+ <parameters>
628
+ user_input: $ARGUMENTS
629
+ mode: fix
630
+ app_url: {auto-detect from test config baseURL, or ask user}
631
+ </parameters>
632
+ "
633
+ )
634
+
635
+ **Classification categories:**
636
+ - **APPLICATION BUG** — error in production code → Report only, NEVER auto-fix
637
+ - **TEST CODE ERROR** — error in test code → Auto-fix if HIGH confidence
638
+ - **ENVIRONMENT ISSUE** — missing env, connection refused → Report with resolution steps
639
+ - **INCONCLUSIVE** — ambiguous → Report what's known, ask for more info
640
+
641
+ **Auto-fix rules:**
642
+ - Only TEST CODE ERROR at HIGH confidence
643
+ - Allowed fixes: import paths, selectors, assertion values, config, missing await, fixture paths
644
+ - Every fix verified by re-running the specific test
645
+ - Never modify application code (src/, app/, lib/)
646
+
647
+ **Browser reproduction (when Playwright MCP connected):**
648
+ - Navigate to failing page → snapshot DOM → reproduce action → screenshot
649
+ - Element not in DOM → TEST CODE ERROR (HIGH confidence)
650
+ - Element exists, wrong behavior → APPLICATION BUG
651
+ - Page doesn't load → ENVIRONMENT ISSUE
652
+
653
+ 6. Present results. APPLICATION BUGs are reported for developer action, not auto-fixed.
654
+
655
+ $ARGUMENTS
656
+
657
+ ## MANDATORY verification — run ALL commands below, no exceptions, no skipping
658
+
659
+ Before returning control, copy-paste and run this ENTIRE block. Do NOT decide which commands "apply" — run all of them every time. The output confirms what happened; you do not get to assume the answer.
660
+
661
+ ```bash
662
+ echo "=== QA-FIX CHECKLIST START ==="
663
+ echo "1. Failure classification report:"
664
+ ls .qa-output/FAILURE_CLASSIFICATION_REPORT.md 2>/dev/null || echo "NO_CLASSIFICATION_REPORT"
665
+ echo "2. Locator Registry:"
666
+ ls .qa-output/locators/ 2>/dev/null || echo "NO_LOCATORS_FOUND"
667
+ echo "3. MY_PREFERENCES.md:"
668
+ cat ~/.claude/qaa/MY_PREFERENCES.md 2>/dev/null || echo "FILE_NOT_FOUND"
669
+ echo "4. Codebase map (context for bug-detective):"
670
+ ls .qa-output/codebase/ 2>/dev/null || echo "NO_CODEBASE_MAP"
671
+ echo "5. Test files in scope:"
672
+ find tests/ cypress/ __tests__/ e2e/ spec/ -type f -name "*.spec.*" -o -name "*.test.*" -o -name "*.e2e.*" 2>/dev/null | head -20 || echo "NO_TEST_FILES_FOUND"
673
+ echo "6. MCP evidence (if browser was used):"
674
+ ls .qa-output/mcp-evidence/ 2>/dev/null || echo "NO_MCP_EVIDENCE"
675
+ echo "7. Classification categories in report:"
676
+ grep -cE "APPLICATION BUG|TEST CODE ERROR|ENVIRONMENT ISSUE|INCONCLUSIVE" .qa-output/FAILURE_CLASSIFICATION_REPORT.md 2>/dev/null || echo "NO_CLASSIFICATIONS"
677
+ echo "=== QA-FIX CHECKLIST END ==="
678
+ ```
679
+
680
+ **Rules:**
681
+ - Run the block AS-IS. Do not modify it. Do not split it. Do not skip lines.
682
+ - If any output shows a problem (NO_CLASSIFICATION_REPORT after fix mode completed), fix it before returning.
683
+ - If output shows expected "not found" results (e.g., NO_MCP_EVIDENCE when no browser was used), that is fine — the point is you RAN the command instead of assuming the answer.
684
+ - Do NOT mark this task as complete until the block has been executed and you have read every line of output.