qaa-agent 1.9.2 → 1.9.5

package/commands/qa-audit.md

@@ -1,126 +1,255 @@
-# QA Audit & Report
-
-Comprehensive quality audit of a test suite with 6-dimension scoring, testing pyramid analysis, and status reporting. Supports three output modes: full audit, pyramid analysis only, or status report adapted to audience.
-
-## Usage
-
-```
-/qa-audit <path-to-tests> [options]
-```
-
-### Options
-
-- `<path-to-tests>` — directory containing test files to audit
-- `--dev-repo <path>` — path to developer repository (for coverage cross-reference)
-- `--app-url <url>` — URL of running application for locator verification via Playwright MCP
-- `--pyramid` — pyramid analysis only: compare actual vs target distribution with action plan
-- `--report [team|management|client]` — generate status report adapted to audience (default: team)
-
-### Mode Detection
-
-```
-if --pyramid:
-  MODE = "pyramid"      → PYRAMID_ANALYSIS.md
-elif --report:
-  MODE = "report"       → QA_STATUS_REPORT.md (adapted to audience)
-else:
-  MODE = "audit"        → QA_AUDIT_REPORT.md (full 6-dimension audit, default)
-```
-
-## What It Produces
-
-| Mode | Artifact | Description |
-|------|----------|-------------|
-| audit | QA_AUDIT_REPORT.md | 6-dimension scoring, critical issues, recommendations with effort estimates |
-| pyramid | PYRAMID_ANALYSIS.md | Current vs target distribution, gap table, prioritized action plan |
-| report | QA_STATUS_REPORT.md | Metrics, pyramid distribution, risk areas, adapted to audience level |
-
-## Instructions
-
-### AUDIT MODE (default)
-
-Scores across 6 dimensions: Locator Quality (20%), Assertion Specificity (20%), POM Compliance (15%), Test Coverage (20%), Naming Convention (15%), Test Data Management (10%).
-
-1. Read `CLAUDE.md` — quality gates, locator tiers, assertion rules, POM rules, naming conventions.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke validator agent in audit mode:
-
-Task(
-  prompt="
-    <objective>Audit test suite quality and produce QA_AUDIT_REPORT.md with 6-dimension scoring. If Playwright MCP is connected and an app URL is available, verify E2E test locators against the live DOM via browser_navigate + browser_snapshot. Flag stale locators (Tier 4 CSS/XPath that could be upgraded to Tier 1 data-testid) and locators that no longer match any DOM element.</objective>
-    <execution_context>@agents/qaa-validator.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: audit
-    app_url: {auto-detect from test config baseURL, or ask user}
-    </parameters>
-  "
-)
-
-3. Present results with overall score and prioritized recommendations.
-
----
-
-### PYRAMID MODE (`--pyramid`)
-
-Analyze test distribution against the ideal testing pyramid from CLAUDE.md (Unit 60-70%, Integration 10-15%, API 20-25%, E2E 3-5%). Compares actual percentages to targets and produces an action plan.
-
-1. Read `CLAUDE.md` — testing pyramid target percentages.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke analyzer agent for pyramid analysis:
-
-Task(
-  prompt="
-    <objective>Produce PYRAMID_ANALYSIS.md comparing actual test distribution to target pyramid. Count tests by type (unit, integration, API, E2E), calculate percentages, compare to CLAUDE.md targets, identify gaps, and produce a prioritized action plan to reach the recommended distribution. Adjust target percentages based on the actual app architecture.</objective>
-    <execution_context>@agents/qaa-analyzer.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: pyramid-analysis
-    </parameters>
-  "
-)
-
-3. Present analysis with gap table and action plan.
-
----
-
-### REPORT MODE (`--report`)
-
-Generate a summary report of current QA status. Adapts detail level to audience.
-
-**Audience levels:**
-- `team` (default) — file-level details, specific locator/assertion issues, technical recommendations
-- `management` — high-level metrics, risk areas, coverage percentages, trend indicators
-- `client` — coverage summary, confidence level, test pass rates, risk mitigation status
-
-1. Read `CLAUDE.md` — testing pyramid targets, quality gates.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke analyzer agent for status reporting:
-
-Task(
-  prompt="
-    <objective>Produce QA_STATUS_REPORT.md with current test suite metrics and coverage. Adapt detail level to the specified audience: team (file-level technical details), management (high-level metrics and risks), or client (coverage summary and confidence). Include testing pyramid distribution, pass/fail rates, risk areas, and actionable recommendations appropriate for the audience.</objective>
-    <execution_context>@agents/qaa-analyzer.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: status-report
-    </parameters>
-  "
-)
-
-3. Present report to user.
-
-$ARGUMENTS
+# QA Audit & Report
+
+Comprehensive quality audit of a test suite with 6-dimension scoring, testing pyramid analysis, and status reporting. Supports three output modes: full audit, pyramid analysis only, or status report adapted to audience.
+
+
+## ⚠ MANDATORY: First Tool Call
+
+Before any other tool use, you **MUST** run the intent detector as your **FIRST** tool call. Do NOT explore the repo, do NOT analyze the prompt independently, do NOT make assumptions about what the user wants. Run this first:
+
+```bash
+RESOLVED=$(node ~/.claude/qaa/bin/lib/intent-detector.cjs --resolve "$ARGUMENTS" --aliases '{"app_url":"app-url"}')
+echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.banner)"
+```
+
+**Pass `$ARGUMENTS` LITERALLY** — do NOT mentally translate NL to flag form. The detector handles NL parsing. If you pre-translate, the detector reports `source: flag` for everything and bypasses the confirmation checkpoint.
+
+### After the banner appears
+
+1. Determine `ALL_FROM_FLAGS`:
+   ```bash
+   ALL_FROM_FLAGS=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); const sources=Object.values(j.resolved).map(v=>v.source); console.log(sources.length>0 && sources.every(s=>s==='flag') ? 'true' : 'false')")
+   ```
+
+2. If **`IS_AUTO=false` AND `ALL_FROM_FLAGS=false`** (some values came from NL or defaults), **PAUSE and ask the user**:
+   ```
+   The values above were resolved from natural language and/or defaults.
+   Do you want to continue with these inputs? (yes/no)
+   ```
+   Wait for explicit user confirmation before proceeding.
+
+3. If `IS_AUTO=true` OR `ALL_FROM_FLAGS=true`, proceed without pause.
+
+### DO NOT
+
+- Show a custom-format banner — use the exact output from `intent-detector.cjs`
+- Skip the detector and parse NL in your own head
+- Decide mode (from-aut, from-ticket, etc.) before the detector runs
+- Make tool calls (Bash, Read, Glob) before the detector runs
+
+### Mode detection happens AFTER the detector
+
+Use the resolved values from `RESOLVED` (URLs, framework, paths) to determine mode. The detector's `intent.suggested_mode` already provides a heuristic.
+
+
+## Usage
+
+```
+/qa-audit <path-to-tests> [options]
+```
+
+### Options
+
+- `<path-to-tests>` — directory containing test files to audit
+- `--dev-repo <path>` — path to developer repository (for coverage cross-reference)
+- `--app-url <url>` — URL of running application for locator verification via Playwright MCP
+- `--pyramid` — pyramid analysis only: compare actual vs target distribution with action plan
+- `--report [team|management|client]` — generate status report adapted to audience (default: team)
+
+### Mode Detection
+
+```
+if --pyramid:
+  MODE = "pyramid"      → PYRAMID_ANALYSIS.md
+elif --report:
+  MODE = "report"       → QA_STATUS_REPORT.md (adapted to audience)
+else:
+  MODE = "audit"        → QA_AUDIT_REPORT.md (full 6-dimension audit, default)
+```
+
+## What It Produces
+
+| Mode | Artifact | Description |
+|------|----------|-------------|
+| audit | QA_AUDIT_REPORT.md | 6-dimension scoring, critical issues, recommendations with effort estimates |
+| pyramid | PYRAMID_ANALYSIS.md | Current vs target distribution, gap table, prioritized action plan |
+| report | QA_STATUS_REPORT.md | Metrics, pyramid distribution, risk areas, adapted to audience level |
+
+
+## Intent Detection
+
+Before processing arguments, run the intent detector to support natural-language input alongside flags. Flags always win over NL.
+
+```bash
+RESOLVED=$(node ~/.claude/qaa/bin/lib/intent-detector.cjs --resolve "$ARGUMENTS" --aliases '{"app_url":"app-url"}')
+
+# Print INPUT DETECTION banner (always shown, even if defaults are used)
+echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.banner)"
+
+# Extract resolved values for use below
+APP_URL=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.resolved.app_url?.value || '')")
+```
+
+If you see values you didn't intend in the banner above (interactive mode), you can press Ctrl-C and re-run with explicit flags.
+
+## Instructions
+
+### AUDIT MODE (default)
+
+Scores across 6 dimensions: Locator Quality (20%), Assertion Specificity (20%), POM Compliance (15%), Test Coverage (20%), Naming Convention (15%), Test Data Management (10%).
+
+1. Read `CLAUDE.md` — quality gates, locator tiers, assertion rules, POM rules, naming conventions.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke validator agent in audit mode:
+
+Task(
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-validator.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-validator.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Audit test suite quality and produce QA_AUDIT_REPORT.md with 6-dimension scoring. If Playwright MCP is connected and an app URL is available, verify E2E test locators against the live DOM via browser_navigate + browser_snapshot. Flag stale locators (Tier 4 CSS/XPath that could be upgraded to Tier 1 data-testid) and locators that no longer match any DOM element.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-validator.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: audit
+    app_url: {auto-detect from test config baseURL, or ask user}
+    </parameters>
+  "
+)
+
+3. Present results with overall score and prioritized recommendations.
+
+---
+
+### PYRAMID MODE (`--pyramid`)
+
+Analyze test distribution against the ideal testing pyramid from CLAUDE.md (Unit 60-70%, Integration 10-15%, API 20-25%, E2E 3-5%). Compares actual percentages to targets and produces an action plan.
+
+1. Read `CLAUDE.md` — testing pyramid target percentages.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke analyzer agent for pyramid analysis:
+
+Task(
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-analyzer.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-analyzer.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Produce PYRAMID_ANALYSIS.md comparing actual test distribution to target pyramid. Count tests by type (unit, integration, API, E2E), calculate percentages, compare to CLAUDE.md targets, identify gaps, and produce a prioritized action plan to reach the recommended distribution. Adjust target percentages based on the actual app architecture.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-analyzer.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: pyramid-analysis
+    </parameters>
+  "
+)
+
+3. Present analysis with gap table and action plan.
+
+---
+
+### REPORT MODE (`--report`)
+
+Generate a summary report of current QA status. Adapts detail level to audience.
+
+**Audience levels:**
+- `team` (default) — file-level details, specific locator/assertion issues, technical recommendations
+- `management` — high-level metrics, risk areas, coverage percentages, trend indicators
+- `client` — coverage summary, confidence level, test pass rates, risk mitigation status
+
+1. Read `CLAUDE.md` — testing pyramid targets, quality gates.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke analyzer agent for status reporting:
+
+Task(
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-analyzer.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-analyzer.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Produce QA_STATUS_REPORT.md with current test suite metrics and coverage. Adapt detail level to the specified audience: team (file-level technical details), management (high-level metrics and risks), or client (coverage summary and confidence). Include testing pyramid distribution, pass/fail rates, risk areas, and actionable recommendations appropriate for the audience.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-analyzer.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: status-report
+    </parameters>
+  "
+)
+
+3. Present report to user.
+
+$ARGUMENTS