qaa-agent 1.6.2 → 1.7.0

package/workflows/qa-validate.md

@@ -1,295 +1,299 @@
-<purpose>
-Validate existing test files against CLAUDE.md QA standards using a 4-layer check (Syntax, Structure, Dependencies, Logic). Runs a fix loop (validate, fix, re-validate) up to 3 times. If failures remain after the fix loop, optionally spawns the bug-detective agent to classify failures. Produces VALIDATION_REPORT.md and optionally FAILURE_CLASSIFICATION_REPORT.md. Use this workflow to audit test quality in any test suite -- whether generated by the QA pipeline or hand-written.
-</purpose>
-
-<required_reading>
-- `CLAUDE.md` -- QA automation standards, quality gates, locator strategy, POM rules, assertion specificity rules
-- `agents/qaa-validator.md` -- Validator agent definition (4-layer validation, fix loop, confidence assessment)
-- `agents/qaa-bug-detective.md` -- Bug detective agent definition (failure classification, auto-fix for test errors)
-- `templates/validation-report.md` -- VALIDATION_REPORT.md format contract
-- `templates/failure-classification.md` -- FAILURE_CLASSIFICATION_REPORT.md format contract
-</required_reading>
-
-<process>
-
-<step name="parse_arguments">
-## Step 1: Parse Arguments
-
-Parse `$ARGUMENTS` for the test directory path and optional flags.
-
-**Supported arguments:**
-- `<path>` -- Path to the test directory to validate (positional)
-- `--classify` -- If set, spawn bug-detective after validation to classify remaining failures
-- No arguments -- auto-detect test directories
-
-**Parsing logic:**
-
-```bash
-TEST_DIR=""
-CLASSIFY_FAILURES=false
-
-# Parse positional path argument
-# Parse --classify flag
-```
-
-**Validation:**
-- If a path is provided, verify it exists and is a directory. If not, print error: `"Error: Test directory does not exist: {path}"` and STOP.
-- If no path provided, proceed to auto-detection in the next step.
-</step>
-
-<step name="detect_test_directories">
-## Step 2: Detect Test Directories
-
-If no test directory was specified, auto-detect by searching for common test directory patterns.
-
-**Detection order (check each, use the first match):**
-
-1. `tests/` -- Standard test directory
-2. `__tests__/` -- Jest convention
-3. `cypress/` -- Cypress test directory
-4. `e2e/` -- E2E test directory
-5. `test/` -- Alternative singular form
-6. `spec/` -- RSpec/generic spec directory
-
-**Detection logic:**
-
-```bash
-if TEST_DIR is empty:
-  for dir in tests __tests__ cypress e2e test spec:
-    if directory exists at "${dir}":
-      TEST_DIR="${dir}"
-      break
-```
-
-**If no test directory found:**
-
-Print error and STOP:
-```
-Error: No test directory found.
-Searched for: tests/, __tests__/, cypress/, e2e/, test/, spec/
-Provide the test directory path as an argument: /qa-validate <path>
-```
-
-**Count test files:**
-
-```bash
-TEST_FILE_COUNT=$(find "${TEST_DIR}" -type f \( -name "*.test.*" -o -name "*.spec.*" -o -name "*.cy.*" -o -name "*.e2e.*" \) | wc -l)
-```
-
-If `TEST_FILE_COUNT` is 0, print error: `"Error: No test files found in {TEST_DIR}. Expected files matching *.test.*, *.spec.*, *.cy.*, or *.e2e.*"` and STOP.
-
-**Print detection results:**
-
-```
-Test directory: {TEST_DIR}
-Test files found: {TEST_FILE_COUNT}
-```
-</step>
-
-<step name="build_file_list">
-## Step 3: Build File List for Validation
-
-Enumerate all test files and supporting files (page objects, fixtures) to validate.
-
-**File discovery:**
-
-Glob for test-related files in the detected directory:
-- Test specs: `**/*.test.*`, `**/*.spec.*`, `**/*.cy.*`, `**/*.e2e.*`
-- Page objects: `**/pages/**/*`, `**/page-objects/**/*`, `**/support/page-objects/**/*`
-- Fixtures: `**/fixtures/**/*`
-- Config files: test framework configs at the project root
-
-**Build a generation-plan-equivalent file list:**
-
-The validator agent expects a file list (normally from the generation plan). For this workflow, construct a synthetic file list from the discovered files:
-
-```markdown
-## Files to Validate
-
-| File Path | Type |
-|-----------|------|
-| tests/unit/auth.unit.spec.ts | test_spec |
-| tests/api/users.api.spec.ts | test_spec |
-| pages/auth/LoginPage.ts | page_object |
-| fixtures/auth-data.ts | fixture |
-| ... | ... |
-```
-
-**Set output directory:**
-
-```bash
-OUTPUT_DIR=".qa-output"
-mkdir -p "${OUTPUT_DIR}"
-```
-</step>
-
-<step name="run_validator">
-## Step 4: Spawn Validator Agent
-
-Spawn the validator agent to perform 4-layer validation with fix loop.
-
-```
-Task(
-  prompt="
-    <objective>Validate test files against CLAUDE.md standards across 4 layers (Syntax, Structure, Dependencies, Logic)</objective>
-    <execution_context>@agents/qaa-validator.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - {synthetic file list or generation plan path}
-    </files_to_read>
-    <parameters>
-    test_dir: {TEST_DIR}
-    output_path: {OUTPUT_DIR}/VALIDATION_REPORT.md
-    file_list: {list of test file paths}
-    mode: validation
-    </parameters>
-  "
-)
-```
-
-**Handle validator return:**
-
-Extract from the validator's return values:
-- `overall_status` -- PASS, PASS_WITH_WARNINGS, or FAIL
-- `confidence` -- HIGH, MEDIUM, or LOW
-- `layers_summary` -- per-layer PASS/FAIL status
-- `fix_loops_used` -- number of fix iterations (1-3)
-- `issues_found` -- total issues discovered
-- `issues_fixed` -- issues auto-fixed (HIGH confidence only)
-- `unresolved_count` -- issues remaining after fix loops
-
-**Verify VALIDATION_REPORT.md exists:**
-
-```bash
-[ -f "${OUTPUT_DIR}/VALIDATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
-```
-
-If missing, print error: `"Error: Validator did not produce VALIDATION_REPORT.md."` and STOP.
-</step>
-
-<step name="classify_failures">
-## Step 5: Optionally Spawn Bug Detective
-
-If the `--classify` flag was set AND the validator reported failures (unresolved_count > 0), spawn the bug-detective agent to classify the remaining failures.
-
-**Gate check:**
-
-```
-if CLASSIFY_FAILURES is false:
-  Skip this step -- print note: "Skipping failure classification. Use --classify flag to enable."
-
-if unresolved_count is 0:
-  Skip this step -- print note: "No unresolved failures to classify."
-```
-
-**Spawn bug-detective agent:**
-
-```
-Task(
-  prompt="
-    <objective>Run test suite and classify failures into APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE</objective>
-    <execution_context>@agents/qaa-bug-detective.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - {test file paths with failures}
-    </files_to_read>
-    <parameters>
-    test_dir: {TEST_DIR}
-    output_path: {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md
-    </parameters>
-  "
-)
-```
-
-**Handle detective return:**
-
-Extract from the detective's return values:
-- `total_failures` -- number of failures analyzed
-- `classification_breakdown` -- counts per category (app_bug, test_error, env_issue, inconclusive)
-- `auto_fixes_applied` -- count of TEST CODE ERROR fixes at HIGH confidence
-- `auto_fixes_verified` -- count of fixes that passed re-verification
-
-**Verify FAILURE_CLASSIFICATION_REPORT.md exists (if detective was spawned):**
-
-```bash
-[ -f "${OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
-```
-</step>
-
-<step name="print_summary">
-## Step 6: Print Validation Summary
-
-Print a human-readable summary of the validation results.
-
-```
-=== Validation Complete ===
-
-Test Directory: {TEST_DIR}
-Files Validated: {file_count}
-Fix Loops Used: {fix_loops_used} of 3
-
-Layer Results:
-  Syntax:       {PASS|FAIL}
-  Structure:    {PASS|FAIL}
-  Dependencies: {PASS|FAIL}
-  Logic:        {PASS|FAIL}
-
-Issues:
-  Found:      {issues_found}
-  Auto-fixed: {issues_fixed}
-  Unresolved: {unresolved_count}
-
-Overall Status: {PASS|PASS_WITH_WARNINGS|FAIL}
-Confidence: {HIGH|MEDIUM|LOW}
-
-Artifacts Produced:
-  - {OUTPUT_DIR}/VALIDATION_REPORT.md
-  - {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md  (if --classify used)
-```
-
-**If bug detective ran, add classification summary:**
-
-```
-Failure Classification:
-  APPLICATION BUG:    {app_bug_count}
-  TEST CODE ERROR:    {test_error_count}
-  ENVIRONMENT ISSUE:  {env_issue_count}
-  INCONCLUSIVE:       {inconclusive_count}
-  Auto-fixed:         {auto_fixes_applied}
-```
-
-```
-===========================
-```
-</step>
-
-</process>
-
-<output>
-This workflow validates existing test files and produces quality reports.
-
-**Artifacts produced:**
-
-| Artifact | When Produced | Description |
-|----------|---------------|-------------|
-| VALIDATION_REPORT.md | Always | 4-layer validation results (Syntax, Structure, Dependencies, Logic) with fix loop history and confidence level |
-| FAILURE_CLASSIFICATION_REPORT.md | When --classify flag set AND failures exist | Per-failure classification (APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, INCONCLUSIVE) with evidence and auto-fix log |
-
-**Side effects:**
-- Test files may be modified in-place by HIGH-confidence auto-fixes from the validator
-- Test files may be modified by HIGH-confidence TEST CODE ERROR fixes from the bug detective
-- No git branches created (fixes applied to working tree)
-- No PRs created
-</output>
-
-<error_handling>
-| Error | Cause | Action |
-|-------|-------|--------|
-| Test directory does not exist | Invalid path argument | Print error with path, STOP |
-| No test directory found | No common test dirs in project | Print error listing searched dirs, suggest providing path, STOP |
-| No test files found in directory | Directory exists but contains no test files | Print error with expected patterns, STOP |
-| VALIDATION_REPORT.md missing | Validator failed silently | Print error, STOP |
-| Fix loop exhausted (3 loops) | Issues cannot be auto-resolved | Validator checkpoints with unresolved issue details |
-| Test runner not detected | No test framework config found | Bug detective checkpoints for user to specify runner |
-| Test runner fails to start | Missing dependencies or broken config | Bug detective classifies as ENVIRONMENT ISSUE |
-</error_handling>
+<purpose>
+Validate existing test files against CLAUDE.md QA standards using a 4-layer check (Syntax, Structure, Dependencies, Logic). Runs a fix loop (validate, fix, re-validate) up to 3 times. If failures remain after the fix loop, optionally spawns the bug-detective agent to classify failures. Produces VALIDATION_REPORT.md and optionally FAILURE_CLASSIFICATION_REPORT.md. Use this workflow to audit test quality in any test suite -- whether generated by the QA pipeline or hand-written.
+</purpose>
+
+<required_reading>
+- `CLAUDE.md` -- QA automation standards, quality gates, locator strategy, POM rules, assertion specificity rules
+- `agents/qaa-validator.md` -- Validator agent definition (4-layer validation, fix loop, confidence assessment)
+- `agents/qaa-bug-detective.md` -- Bug detective agent definition (failure classification, auto-fix for test errors)
+- `templates/validation-report.md` -- VALIDATION_REPORT.md format contract
+- `templates/failure-classification.md` -- FAILURE_CLASSIFICATION_REPORT.md format contract
+</required_reading>
+
+<process>
+
+<step name="parse_arguments">
+## Step 1: Parse Arguments
+
+Parse `$ARGUMENTS` for the test directory path and optional flags.
+
+**Supported arguments:**
+- `<path>` -- Path to the test directory to validate (positional)
+- `--classify` -- If set, spawn bug-detective after validation to classify remaining failures
+- No arguments -- auto-detect test directories
+
+**Parsing logic:**
+
+```bash
+TEST_DIR=""
+CLASSIFY_FAILURES=false
+
+# Parse positional path argument
+# Parse --classify flag
+```
+
+**Validation:**
+- If a path is provided, verify it exists and is a directory. If not, print error: `"Error: Test directory does not exist: {path}"` and STOP.
+- If no path provided, proceed to auto-detection in the next step.
+</step>
+
+<step name="detect_test_directories">
+## Step 2: Detect Test Directories
+
+If no test directory was specified, auto-detect by searching for common test directory patterns.
+
+**Detection order (check each, use the first match):**
+
+1. `tests/` -- Standard test directory
+2. `__tests__/` -- Jest convention
+3. `cypress/` -- Cypress test directory
+4. `e2e/` -- E2E test directory
+5. `test/` -- Alternative singular form
+6. `spec/` -- RSpec/generic spec directory
+
+**Detection logic:**
+
+```bash
+if TEST_DIR is empty:
+  for dir in tests __tests__ cypress e2e test spec:
+    if directory exists at "${dir}":
+      TEST_DIR="${dir}"
+      break
+```
+
+**If no test directory found:**
+
+Print error and STOP:
+```
+Error: No test directory found.
+Searched for: tests/, __tests__/, cypress/, e2e/, test/, spec/
+Provide the test directory path as an argument: /qa-validate <path>
+```
+
+**Count test files:**
+
+```bash
+TEST_FILE_COUNT=$(find "${TEST_DIR}" -type f \( -name "*.test.*" -o -name "*.spec.*" -o -name "*.cy.*" -o -name "*.e2e.*" \) | wc -l)
+```
+
+If `TEST_FILE_COUNT` is 0, print error: `"Error: No test files found in {TEST_DIR}. Expected files matching *.test.*, *.spec.*, *.cy.*, or *.e2e.*"` and STOP.
+
+**Print detection results:**
+
+```
+Test directory: {TEST_DIR}
+Test files found: {TEST_FILE_COUNT}
+```
+</step>
+
+<step name="build_file_list">
+## Step 3: Build File List for Validation
+
+Enumerate all test files and supporting files (page objects, fixtures) to validate.
+
+**File discovery:**
+
+Glob for test-related files in the detected directory:
+- Test specs: `**/*.test.*`, `**/*.spec.*`, `**/*.cy.*`, `**/*.e2e.*`
+- Page objects: `**/pages/**/*`, `**/page-objects/**/*`, `**/support/page-objects/**/*`
+- Fixtures: `**/fixtures/**/*`
+- Config files: test framework configs at the project root
+
+**Build a generation-plan-equivalent file list:**
+
+The validator agent expects a file list (normally from the generation plan). For this workflow, construct a synthetic file list from the discovered files:
+
+```markdown
+## Files to Validate
+
+| File Path | Type |
+|-----------|------|
+| tests/unit/auth.unit.spec.ts | test_spec |
+| tests/api/users.api.spec.ts | test_spec |
+| pages/auth/LoginPage.ts | page_object |
+| fixtures/auth-data.ts | fixture |
+| ... | ... |
+```
+
+**Set output directory:**
+
+```bash
+OUTPUT_DIR=".qa-output"
+mkdir -p "${OUTPUT_DIR}"
+```
+</step>
+
+<step name="run_validator">
+## Step 4: Spawn Validator Agent
+
+Spawn the validator agent to perform 4-layer validation with fix loop.
+
+```
+Task(
+  prompt="
+    <objective>Validate test files against CLAUDE.md standards across 4 layers (Syntax, Structure, Dependencies, Logic). Use codebase map for context and locator registry to verify locator accuracy.</objective>
+    <execution_context>@agents/qaa-validator.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - {synthetic file list or generation plan path}
+    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists -- real locators for cross-checking POM accuracy)
+    - .qa-output/codebase/CODE_PATTERNS.md (if exists -- naming conventions for structure validation)
+    - .qa-output/codebase/TEST_SURFACE.md (if exists -- function signatures for target verification)
+    - .qa-output/codebase/API_CONTRACTS.md (if exists -- API shapes for assertion verification)
+    </files_to_read>
+    <parameters>
+    test_dir: {TEST_DIR}
+    output_path: {OUTPUT_DIR}/VALIDATION_REPORT.md
+    file_list: {list of test file paths}
+    mode: validation
+    </parameters>
+  "
+)
+```
+
+**Handle validator return:**
+
+Extract from the validator's return values:
+- `overall_status` -- PASS, PASS_WITH_WARNINGS, or FAIL
+- `confidence` -- HIGH, MEDIUM, or LOW
+- `layers_summary` -- per-layer PASS/FAIL status
+- `fix_loops_used` -- number of fix iterations (1-3)
+- `issues_found` -- total issues discovered
+- `issues_fixed` -- issues auto-fixed (HIGH confidence only)
+- `unresolved_count` -- issues remaining after fix loops
+
+**Verify VALIDATION_REPORT.md exists:**
+
+```bash
+[ -f "${OUTPUT_DIR}/VALIDATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
+```
+
+If missing, print error: `"Error: Validator did not produce VALIDATION_REPORT.md."` and STOP.
+</step>
+
+<step name="classify_failures">
+## Step 5: Optionally Spawn Bug Detective
+
+If the `--classify` flag was set AND the validator reported failures (unresolved_count > 0), spawn the bug-detective agent to classify the remaining failures.
+
+**Gate check:**
+
+```
+if CLASSIFY_FAILURES is false:
+  Skip this step -- print note: "Skipping failure classification. Use --classify flag to enable."
+
+if unresolved_count is 0:
+  Skip this step -- print note: "No unresolved failures to classify."
+```
+
+**Spawn bug-detective agent:**
+
+```
+Task(
+  prompt="
+    <objective>Run test suite and classify failures into APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE</objective>
+    <execution_context>@agents/qaa-bug-detective.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - {test file paths with failures}
+    </files_to_read>
+    <parameters>
+    test_dir: {TEST_DIR}
+    output_path: {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md
+    </parameters>
+  "
+)
+```
+
+**Handle detective return:**
+
+Extract from the detective's return values:
+- `total_failures` -- number of failures analyzed
+- `classification_breakdown` -- counts per category (app_bug, test_error, env_issue, inconclusive)
+- `auto_fixes_applied` -- count of TEST CODE ERROR fixes at HIGH confidence
+- `auto_fixes_verified` -- count of fixes that passed re-verification
+
+**Verify FAILURE_CLASSIFICATION_REPORT.md exists (if detective was spawned):**
+
+```bash
+[ -f "${OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md" ] && echo "FOUND" || echo "MISSING"
+```
+</step>
+
+<step name="print_summary">
+## Step 6: Print Validation Summary
+
+Print a human-readable summary of the validation results.
+
+```
+=== Validation Complete ===
+
+Test Directory: {TEST_DIR}
+Files Validated: {file_count}
+Fix Loops Used: {fix_loops_used} of 3
+
+Layer Results:
+  Syntax:       {PASS|FAIL}
+  Structure:    {PASS|FAIL}
+  Dependencies: {PASS|FAIL}
+  Logic:        {PASS|FAIL}
+
+Issues:
+  Found:      {issues_found}
+  Auto-fixed: {issues_fixed}
+  Unresolved: {unresolved_count}
+
+Overall Status: {PASS|PASS_WITH_WARNINGS|FAIL}
+Confidence: {HIGH|MEDIUM|LOW}
+
+Artifacts Produced:
+  - {OUTPUT_DIR}/VALIDATION_REPORT.md
+  - {OUTPUT_DIR}/FAILURE_CLASSIFICATION_REPORT.md  (if --classify used)
+```
+
+**If bug detective ran, add classification summary:**
+
+```
+Failure Classification:
+  APPLICATION BUG:    {app_bug_count}
+  TEST CODE ERROR:    {test_error_count}
+  ENVIRONMENT ISSUE:  {env_issue_count}
+  INCONCLUSIVE:       {inconclusive_count}
+  Auto-fixed:         {auto_fixes_applied}
+```
+
+```
+===========================
+```
+</step>
+
+</process>
+
+<output>
+This workflow validates existing test files and produces quality reports.
+
+**Artifacts produced:**
+
+| Artifact | When Produced | Description |
+|----------|---------------|-------------|
+| VALIDATION_REPORT.md | Always | 4-layer validation results (Syntax, Structure, Dependencies, Logic) with fix loop history and confidence level |
+| FAILURE_CLASSIFICATION_REPORT.md | When --classify flag set AND failures exist | Per-failure classification (APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, INCONCLUSIVE) with evidence and auto-fix log |
+
+**Side effects:**
+- Test files may be modified in-place by HIGH-confidence auto-fixes from the validator
+- Test files may be modified by HIGH-confidence TEST CODE ERROR fixes from the bug detective
+- No git branches created (fixes applied to working tree)
+- No PRs created
+</output>
+
+<error_handling>
+| Error | Cause | Action |
+|-------|-------|--------|
+| Test directory does not exist | Invalid path argument | Print error with path, STOP |
+| No test directory found | No common test dirs in project | Print error listing searched dirs, suggest providing path, STOP |
+| No test files found in directory | Directory exists but contains no test files | Print error with expected patterns, STOP |
+| VALIDATION_REPORT.md missing | Validator failed silently | Print error, STOP |
+| Fix loop exhausted (3 loops) | Issues cannot be auto-resolved | Validator checkpoints with unresolved issue details |
+| Test runner not detected | No test framework config found | Bug detective checkpoints for user to specify runner |
+| Test runner fails to start | Missing dependencies or broken config | Bug detective classifies as ENVIRONMENT ISSUE |
+</error_handling>