qaa-agent 1.6.2 → 1.7.0

package/workflows/qa-gap.md

@@ -1,303 +1,309 @@
-<purpose>
-Compare a developer repository against a QA repository to identify coverage gaps, broken tests, quality issues, and missing test cases. Scans both repositories, performs gap analysis, and produces a comprehensive GAP_ANALYSIS.md with coverage maps, missing test inventories, broken test diagnostics, and quality assessments. Both --dev-repo and --qa-repo paths are required. Use this workflow when a QA repo already exists but may have incomplete coverage or quality issues.
-</purpose>
-
-<required_reading>
-- `CLAUDE.md` -- QA automation standards, pipeline stages, module boundaries, verification commands, quality gates
-- `agents/qaa-scanner.md` -- Scanner agent definition (scans both dev and QA repos)
-- `agents/qaa-analyzer.md` -- Analyzer agent definition (gap mode produces GAP_ANALYSIS.md)
-- `templates/scan-manifest.md` -- SCAN_MANIFEST.md format contract
-- `templates/gap-analysis.md` -- GAP_ANALYSIS.md format contract
-- `templates/qa-analysis.md` -- QA_ANALYSIS.md format contract
-- `templates/test-inventory.md` -- TEST_INVENTORY.md format contract
-</required_reading>
-
-<process>
-
-<step name="parse_arguments">
-## Step 1: Parse Arguments
-
-Parse `$ARGUMENTS` for both repository paths. Both are required for gap analysis.
-
-**Supported arguments:**
-- `--dev-repo <path>` -- Path to the developer repository (required)
-- `--qa-repo <path>` -- Path to the existing QA test repository (required)
-
-**Parsing logic:**
-
-```bash
-DEV_REPO=""
-QA_REPO=""
-
-# Parse --dev-repo and --qa-repo from $ARGUMENTS
-```
-
-**Validation:**
-
-If `--dev-repo` is missing or empty:
-```
-Error: --dev-repo is required for gap analysis.
-Usage: /qa-gap --dev-repo <path> --qa-repo <path>
-```
-STOP.
-
-If `--qa-repo` is missing or empty:
-```
-Error: --qa-repo is required for gap analysis.
-Usage: /qa-gap --dev-repo <path> --qa-repo <path>
-
-To analyze a dev repo without an existing QA repo, use /qa-analyze instead.
-```
-STOP.
-
-If `--dev-repo` path does not exist:
-```
-Error: Dev repo path does not exist: {path}
-```
-STOP.
-
-If `--qa-repo` path does not exist:
-```
-Error: QA repo path does not exist: {path}
-```
-STOP.
-</step>
-
-<step name="setup_output">
-## Step 2: Setup Output Directory
-
-Create the output directory and print the gap analysis banner.
-
-```bash
-OUTPUT_DIR=".qa-output"
-mkdir -p "${OUTPUT_DIR}"
-```
-
-**Print banner:**
-
-```
-=== QA Gap Analysis ===
-Dev Repo: {DEV_REPO}
-QA Repo:  {QA_REPO}
-Output:   {OUTPUT_DIR}
-========================
-```
-</step>
-
-<step name="scan_both_repos">
-## Step 3: Spawn Scanner Agent on Both Repos
-
-Spawn the scanner agent to scan both the developer and QA repositories, producing a combined SCAN_MANIFEST.md.
-
-```
-Task(
-  prompt="
-    <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md with combined file trees</objective>
-    <execution_context>@agents/qaa-scanner.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    </files_to_read>
-    <parameters>
-    dev_repo_path: {DEV_REPO}
-    qa_repo_path: {QA_REPO}
-    output_path: {OUTPUT_DIR}/SCAN_MANIFEST.md
-    </parameters>
-  "
-)
-```
-
-**Handle scanner return:**
-
-- If scanner returns `decision: STOP` -- print the stop reason and STOP.
-- If scanner returns `decision: PROCEED` -- continue.
-- Extract `has_frontend` for summary reporting.
-
-**Verify SCAN_MANIFEST.md exists:**
-
-```bash
-[ -f "${OUTPUT_DIR}/SCAN_MANIFEST.md" ] && echo "FOUND" || echo "MISSING"
-```
-
-If missing, print error and STOP.
-</step>
-
-<step name="run_gap_analysis">
-## Step 4: Spawn Analyzer Agent in Gap Mode
-
-Spawn the analyzer agent in gap analysis mode to compare dev repo testable surfaces against QA repo test coverage.
-
-```
-Task(
-  prompt="
-    <objective>Perform gap analysis comparing dev repo testable surfaces against QA repo test coverage. Produce QA_ANALYSIS.md, TEST_INVENTORY.md, and GAP_ANALYSIS.md.</objective>
-    <execution_context>@agents/qaa-analyzer.md</execution_context>
-    <files_to_read>
-    - {OUTPUT_DIR}/SCAN_MANIFEST.md
-    - CLAUDE.md
-    </files_to_read>
-    <parameters>
-    workflow_option: 2
-    qa_analysis_path: {OUTPUT_DIR}/QA_ANALYSIS.md
-    test_inventory_path: {OUTPUT_DIR}/TEST_INVENTORY.md
-    gap_analysis_path: {OUTPUT_DIR}/GAP_ANALYSIS.md
-    dev_repo_path: {DEV_REPO}
-    qa_repo_path: {QA_REPO}
-    </parameters>
-  "
-)
-```
-
-**Handle analyzer return:**
-
-Extract from the analyzer's return values:
-- `total_test_count` -- total test cases in inventory
-- `pyramid_breakdown` -- unit/integration/api/e2e counts
-- `risk_count` -- high/medium/low risk counts
-
-**Verify artifacts exist:**
-
-```bash
-[ -f "${OUTPUT_DIR}/QA_ANALYSIS.md" ] && echo "FOUND: QA_ANALYSIS.md" || echo "MISSING: QA_ANALYSIS.md"
-[ -f "${OUTPUT_DIR}/TEST_INVENTORY.md" ] && echo "FOUND: TEST_INVENTORY.md" || echo "MISSING: TEST_INVENTORY.md"
-[ -f "${OUTPUT_DIR}/GAP_ANALYSIS.md" ] && echo "FOUND: GAP_ANALYSIS.md" || echo "MISSING: GAP_ANALYSIS.md"
-```
-
-If any artifact is missing, print error with the specific missing file and STOP.
-</step>
-
-<step name="extract_gap_metrics">
-## Step 5: Extract Gap Metrics
-
-Read the GAP_ANALYSIS.md to extract metrics for the summary.
-
-**Metrics to extract:**
-
-1. **Coverage map:**
-   - Total modules in dev repo
-   - Modules with test coverage in QA repo
-   - Modules without any test coverage
-   - Coverage percentage: `(covered_modules / total_modules) * 100`
-
-2. **Missing tests:**
-   - Count of missing test cases by priority (P0, P1, P2)
-   - Count of missing test cases by pyramid level (unit, integration, API, E2E)
-   - Each missing test has an ID following naming convention
-
-3. **Broken tests:**
-   - Count of existing tests that are broken (fail, import errors, outdated selectors)
-   - Each broken test has a failure reason documented
-
-4. **Quality assessment:**
-   - Locator tier distribution (what percentage of locators are Tier 1, 2, 3, 4)
-   - Assertion quality rating (concrete vs vague assertions)
-   - POM compliance (does existing QA repo follow POM rules)
-
-5. **Recommendations priority:**
-   - Fix broken tests first (highest priority)
-   - Add P0 missing tests
-   - Add P1 missing tests
-   - Standardize existing tests to CLAUDE.md standards
-   - Add P2 missing tests (lowest priority)
-</step>
-
-<step name="print_summary">
-## Step 6: Print Gap Analysis Summary
-
-Print a comprehensive human-readable summary.
-
-```
-=== Gap Analysis Complete ===
-
-Coverage Overview:
-  Dev Repo Modules:       {total_modules}
-  Modules with Coverage:  {covered_modules}
-  Modules without Tests:  {uncovered_modules}
-  Coverage Rate:          {coverage_percent}%
-
-Missing Tests:
-  P0 (blocks release):   {p0_missing}
-  P1 (should fix):       {p1_missing}
-  P2 (nice to have):     {p2_missing}
-  --------------------------
-  Total Missing:          {total_missing}
-
-  By Pyramid Level:
-    Unit:        {unit_missing}
-    Integration: {integration_missing}
-    API:         {api_missing}
-    E2E:         {e2e_missing}
-
-Broken Tests:
-  Total Broken: {broken_count}
-
-Quality Assessment:
-  Locator Quality:
-    Tier 1 (data-testid, ARIA): {tier1_percent}%
-    Tier 2 (labels, text):      {tier2_percent}%
-    Tier 3 (alt, title):        {tier3_percent}%
-    Tier 4 (CSS, XPath):        {tier4_percent}%
-  Assertion Quality: {concrete_percent}% concrete assertions
-  POM Compliance:    {pom_compliant}/{pom_total} files compliant
-
-Recommended Action Order:
-  1. Fix {broken_count} broken tests
-  2. Add {p0_missing} P0 missing tests
-  3. Add {p1_missing} P1 missing tests
-  4. Standardize existing tests
-  5. Add {p2_missing} P2 missing tests
-
-Artifacts Produced:
-  - {OUTPUT_DIR}/SCAN_MANIFEST.md
-  - {OUTPUT_DIR}/QA_ANALYSIS.md
-  - {OUTPUT_DIR}/TEST_INVENTORY.md
-  - {OUTPUT_DIR}/GAP_ANALYSIS.md
-
-No test files generated. No git operations performed.
-To generate missing tests, run the full pipeline with:
-  /qa-start --dev-repo {DEV_REPO} --qa-repo {QA_REPO}
-=============================
-```
-</step>
-
-</process>
-
-<output>
-This workflow compares dev and QA repos to identify coverage gaps.
-
-**Artifacts produced:**
-
-| Artifact | When Produced | Description |
-|----------|---------------|-------------|
-| SCAN_MANIFEST.md | Always | Combined scan of both repos with file trees and framework detection |
-| QA_ANALYSIS.md | Always | Architecture overview, risks, test targets, testing pyramid for the dev repo |
-| TEST_INVENTORY.md | Always | Complete test case inventory for what SHOULD exist |
-| GAP_ANALYSIS.md | Always | Coverage map, missing tests, broken tests, quality assessment, recommendations |
-
-**GAP_ANALYSIS.md sections (per templates/gap-analysis.md):**
-
-1. **Coverage Map** -- Every module from SCAN_MANIFEST.md with covered/uncovered status
-2. **Missing Tests** -- Test cases that should exist but do not, with IDs and priorities
-3. **Broken Tests** -- Existing tests that fail, with failure reasons and file paths
-4. **Quality Assessment** -- Locator tier distribution, assertion quality rating, POM compliance
-5. **Recommendations** -- Prioritized action list: fix broken first, then add P0, then P1
-
-**No side effects:**
-- No git branches created
-- No git commits made
-- No PRs created
-- No test files generated
-- No source files modified
-</output>
-
-<error_handling>
-| Error | Cause | Action |
-|-------|-------|--------|
-| --dev-repo missing | Required argument not provided | Print usage with suggestion to use /qa-analyze for dev-only, STOP |
-| --qa-repo missing | Required argument not provided | Print usage with suggestion to use /qa-analyze for dev-only, STOP |
-| Dev repo path does not exist | Invalid path | Print error with path, STOP |
-| QA repo path does not exist | Invalid path | Print error with path, STOP |
-| Scanner returns STOP | No testable surfaces | Print scanner's reason, STOP |
-| SCAN_MANIFEST.md missing | Scanner failed | Print error, STOP |
-| GAP_ANALYSIS.md missing | Analyzer failed | Print error, STOP |
-| Framework detection LOW confidence | Ambiguous stack | Scanner checkpoints for user confirmation |
-</error_handling>
+<purpose>
+Compare a developer repository against a QA repository to identify coverage gaps, broken tests, quality issues, and missing test cases. Scans both repositories, performs gap analysis, and produces a comprehensive GAP_ANALYSIS.md with coverage maps, missing test inventories, broken test diagnostics, and quality assessments. Both --dev-repo and --qa-repo paths are required. Use this workflow when a QA repo already exists but may have incomplete coverage or quality issues.
+</purpose>
+
+<required_reading>
+- `CLAUDE.md` -- QA automation standards, pipeline stages, module boundaries, verification commands, quality gates
+- `agents/qaa-scanner.md` -- Scanner agent definition (scans both dev and QA repos)
+- `agents/qaa-analyzer.md` -- Analyzer agent definition (gap mode produces GAP_ANALYSIS.md)
+- `templates/scan-manifest.md` -- SCAN_MANIFEST.md format contract
+- `templates/gap-analysis.md` -- GAP_ANALYSIS.md format contract
+- `templates/qa-analysis.md` -- QA_ANALYSIS.md format contract
+- `templates/test-inventory.md` -- TEST_INVENTORY.md format contract
+</required_reading>
+
+<process>
+
+<step name="parse_arguments">
+## Step 1: Parse Arguments
+
+Parse `$ARGUMENTS` for both repository paths. Both are required for gap analysis.
+
+**Supported arguments:**
+- `--dev-repo <path>` -- Path to the developer repository (required)
+- `--qa-repo <path>` -- Path to the existing QA test repository (required)
+
+**Parsing logic:**
+
+```bash
+DEV_REPO=""
+QA_REPO=""
+
+# Parse --dev-repo and --qa-repo from $ARGUMENTS
+```
+
+**Validation:**
+
+If `--dev-repo` is missing or empty:
+```
+Error: --dev-repo is required for gap analysis.
+Usage: /qa-gap --dev-repo <path> --qa-repo <path>
+```
+STOP.
+
+If `--qa-repo` is missing or empty:
+```
+Error: --qa-repo is required for gap analysis.
+Usage: /qa-gap --dev-repo <path> --qa-repo <path>
+
+To analyze a dev repo without an existing QA repo, use /qa-analyze instead.
+```
+STOP.
+
+If `--dev-repo` path does not exist:
+```
+Error: Dev repo path does not exist: {path}
+```
+STOP.
+
+If `--qa-repo` path does not exist:
+```
+Error: QA repo path does not exist: {path}
+```
+STOP.
+</step>
+
+<step name="setup_output">
+## Step 2: Setup Output Directory
+
+Create the output directory and print the gap analysis banner.
+
+```bash
+OUTPUT_DIR=".qa-output"
+mkdir -p "${OUTPUT_DIR}"
+```
+
+**Print banner:**
+
+```
+=== QA Gap Analysis ===
+Dev Repo: {DEV_REPO}
+QA Repo:  {QA_REPO}
+Output:   {OUTPUT_DIR}
+========================
+```
+</step>
+
+<step name="scan_both_repos">
+## Step 3: Spawn Scanner Agent on Both Repos
+
+Spawn the scanner agent to scan both the developer and QA repositories, producing a combined SCAN_MANIFEST.md.
+
+```
+Task(
+  prompt="
+    <objective>Scan both developer and QA repositories and produce SCAN_MANIFEST.md with combined file trees</objective>
+    <execution_context>@agents/qaa-scanner.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    </files_to_read>
+    <parameters>
+    dev_repo_path: {DEV_REPO}
+    qa_repo_path: {QA_REPO}
+    output_path: {OUTPUT_DIR}/SCAN_MANIFEST.md
+    </parameters>
+  "
+)
+```
+
+**Handle scanner return:**
+
+- If scanner returns `decision: STOP` -- print the stop reason and STOP.
+- If scanner returns `decision: PROCEED` -- continue.
+- Extract `has_frontend` for summary reporting.
+
+**Verify SCAN_MANIFEST.md exists:**
+
+```bash
+[ -f "${OUTPUT_DIR}/SCAN_MANIFEST.md" ] && echo "FOUND" || echo "MISSING"
+```
+
+If missing, print error and STOP.
+</step>
+
+<step name="run_gap_analysis">
+## Step 4: Spawn Analyzer Agent in Gap Mode
+
+Spawn the analyzer agent in gap analysis mode to compare dev repo testable surfaces against QA repo test coverage.
+
+```
+Task(
+  prompt="
+    <objective>Perform gap analysis comparing dev repo testable surfaces against QA repo test coverage. Use codebase map for deep context and locator registry for frontend coverage assessment. Produce QA_ANALYSIS.md, TEST_INVENTORY.md, and GAP_ANALYSIS.md.</objective>
+    <execution_context>@agents/qaa-analyzer.md</execution_context>
+    <files_to_read>
+    - {OUTPUT_DIR}/SCAN_MANIFEST.md
+    - CLAUDE.md
+    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists -- locator coverage for risk assessment)
+    - .qa-output/codebase/RISK_MAP.md (if exists -- business-critical paths for risk assessment)
+    - .qa-output/codebase/CRITICAL_PATHS.md (if exists -- user flows for E2E scope)
+    - .qa-output/codebase/TEST_ASSESSMENT.md (if exists -- existing test quality)
+    - .qa-output/codebase/COVERAGE_GAPS.md (if exists -- uncovered modules)
+    </files_to_read>
+    <parameters>
+    workflow_option: 2
+    qa_analysis_path: {OUTPUT_DIR}/QA_ANALYSIS.md
+    test_inventory_path: {OUTPUT_DIR}/TEST_INVENTORY.md
+    gap_analysis_path: {OUTPUT_DIR}/GAP_ANALYSIS.md
+    dev_repo_path: {DEV_REPO}
+    qa_repo_path: {QA_REPO}
+    codebase_map_dir: .qa-output/codebase
+    </parameters>
+  "
+)
+```
+
+**Handle analyzer return:**
+
+Extract from the analyzer's return values:
+- `total_test_count` -- total test cases in inventory
+- `pyramid_breakdown` -- unit/integration/api/e2e counts
+- `risk_count` -- high/medium/low risk counts
+
+**Verify artifacts exist:**
+
+```bash
+[ -f "${OUTPUT_DIR}/QA_ANALYSIS.md" ] && echo "FOUND: QA_ANALYSIS.md" || echo "MISSING: QA_ANALYSIS.md"
+[ -f "${OUTPUT_DIR}/TEST_INVENTORY.md" ] && echo "FOUND: TEST_INVENTORY.md" || echo "MISSING: TEST_INVENTORY.md"
+[ -f "${OUTPUT_DIR}/GAP_ANALYSIS.md" ] && echo "FOUND: GAP_ANALYSIS.md" || echo "MISSING: GAP_ANALYSIS.md"
+```
+
+If any artifact is missing, print error with the specific missing file and STOP.
+</step>
+
+<step name="extract_gap_metrics">
+## Step 5: Extract Gap Metrics
+
+Read the GAP_ANALYSIS.md to extract metrics for the summary.
+
+**Metrics to extract:**
+
+1. **Coverage map:**
+   - Total modules in dev repo
+   - Modules with test coverage in QA repo
+   - Modules without any test coverage
+   - Coverage percentage: `(covered_modules / total_modules) * 100`
+
+2. **Missing tests:**
+   - Count of missing test cases by priority (P0, P1, P2)
+   - Count of missing test cases by pyramid level (unit, integration, API, E2E)
+   - Each missing test has an ID following naming convention
+
+3. **Broken tests:**
+   - Count of existing tests that are broken (fail, import errors, outdated selectors)
+   - Each broken test has a failure reason documented
+
+4. **Quality assessment:**
+   - Locator tier distribution (what percentage of locators are Tier 1, 2, 3, 4)
+   - Assertion quality rating (concrete vs vague assertions)
+   - POM compliance (does existing QA repo follow POM rules)
+
+5. **Recommendations priority:**
+   - Fix broken tests first (highest priority)
+   - Add P0 missing tests
+   - Add P1 missing tests
+   - Standardize existing tests to CLAUDE.md standards
+   - Add P2 missing tests (lowest priority)
+</step>
+
+<step name="print_summary">
+## Step 6: Print Gap Analysis Summary
+
+Print a comprehensive human-readable summary.
+
+```
+=== Gap Analysis Complete ===
+
+Coverage Overview:
+  Dev Repo Modules:       {total_modules}
+  Modules with Coverage:  {covered_modules}
+  Modules without Tests:  {uncovered_modules}
+  Coverage Rate:          {coverage_percent}%
+
+Missing Tests:
+  P0 (blocks release):   {p0_missing}
+  P1 (should fix):       {p1_missing}
+  P2 (nice to have):     {p2_missing}
+  --------------------------
+  Total Missing:          {total_missing}
+
+  By Pyramid Level:
+    Unit:        {unit_missing}
+    Integration: {integration_missing}
+    API:         {api_missing}
+    E2E:         {e2e_missing}
+
+Broken Tests:
+  Total Broken: {broken_count}
+
+Quality Assessment:
+  Locator Quality:
+    Tier 1 (data-testid, ARIA): {tier1_percent}%
+    Tier 2 (labels, text):      {tier2_percent}%
+    Tier 3 (alt, title):        {tier3_percent}%
+    Tier 4 (CSS, XPath):        {tier4_percent}%
+  Assertion Quality: {concrete_percent}% concrete assertions
+  POM Compliance:    {pom_compliant}/{pom_total} files compliant
+
+Recommended Action Order:
+  1. Fix {broken_count} broken tests
+  2. Add {p0_missing} P0 missing tests
+  3. Add {p1_missing} P1 missing tests
+  4. Standardize existing tests
+  5. Add {p2_missing} P2 missing tests
+
+Artifacts Produced:
+  - {OUTPUT_DIR}/SCAN_MANIFEST.md
+  - {OUTPUT_DIR}/QA_ANALYSIS.md
+  - {OUTPUT_DIR}/TEST_INVENTORY.md
+  - {OUTPUT_DIR}/GAP_ANALYSIS.md
+
+No test files generated. No git operations performed.
+To generate missing tests, run the full pipeline with:
+  /qa-start --dev-repo {DEV_REPO} --qa-repo {QA_REPO}
+=============================
+```
+</step>
+
+</process>
+
+<output>
+This workflow compares dev and QA repos to identify coverage gaps.
+
+**Artifacts produced:**
+
+| Artifact | When Produced | Description |
+|----------|---------------|-------------|
+| SCAN_MANIFEST.md | Always | Combined scan of both repos with file trees and framework detection |
+| QA_ANALYSIS.md | Always | Architecture overview, risks, test targets, testing pyramid for the dev repo |
+| TEST_INVENTORY.md | Always | Complete test case inventory for what SHOULD exist |
+| GAP_ANALYSIS.md | Always | Coverage map, missing tests, broken tests, quality assessment, recommendations |
+
+**GAP_ANALYSIS.md sections (per templates/gap-analysis.md):**
+
+1. **Coverage Map** -- Every module from SCAN_MANIFEST.md with covered/uncovered status
+2. **Missing Tests** -- Test cases that should exist but do not, with IDs and priorities
+3. **Broken Tests** -- Existing tests that fail, with failure reasons and file paths
+4. **Quality Assessment** -- Locator tier distribution, assertion quality rating, POM compliance
+5. **Recommendations** -- Prioritized action list: fix broken first, then add P0, then P1
+
+**No side effects:**
+- No git branches created
+- No git commits made
+- No PRs created
+- No test files generated
+- No source files modified
+</output>
+
+<error_handling>
+| Error | Cause | Action |
+|-------|-------|--------|
+| --dev-repo missing | Required argument not provided | Print usage with suggestion to use /qa-analyze for dev-only, STOP |
+| --qa-repo missing | Required argument not provided | Print usage with suggestion to use /qa-analyze for dev-only, STOP |
+| Dev repo path does not exist | Invalid path | Print error with path, STOP |
+| QA repo path does not exist | Invalid path | Print error with path, STOP |
+| Scanner returns STOP | No testable surfaces | Print scanner's reason, STOP |
+| SCAN_MANIFEST.md missing | Scanner failed | Print error, STOP |
+| GAP_ANALYSIS.md missing | Analyzer failed | Print error, STOP |
+| Framework detection LOW confidence | Ambiguous stack | Scanner checkpoints for user confirmation |
+</error_handling>