qaa-agent 1.9.2 → 1.9.6

package/commands/qa-audit.md

@@ -1,126 +1,269 @@
-# QA Audit & Report
-
-Comprehensive quality audit of a test suite with 6-dimension scoring, testing pyramid analysis, and status reporting. Supports three output modes: full audit, pyramid analysis only, or status report adapted to audience.
-
-## Usage
-
-```
-/qa-audit <path-to-tests> [options]
-```
-
-### Options
-
-- `<path-to-tests>` — directory containing test files to audit
-- `--dev-repo <path>` — path to developer repository (for coverage cross-reference)
-- `--app-url <url>` — URL of running application for locator verification via Playwright MCP
-- `--pyramid` — pyramid analysis only: compare actual vs target distribution with action plan
-- `--report [team|management|client]` — generate status report adapted to audience (default: team)
-
-### Mode Detection
-
-```
-if --pyramid:
-  MODE = "pyramid"      → PYRAMID_ANALYSIS.md
-elif --report:
-  MODE = "report"       → QA_STATUS_REPORT.md (adapted to audience)
-else:
-  MODE = "audit"        → QA_AUDIT_REPORT.md (full 6-dimension audit, default)
-```
-
-## What It Produces
-
-| Mode | Artifact | Description |
-|------|----------|-------------|
-| audit | QA_AUDIT_REPORT.md | 6-dimension scoring, critical issues, recommendations with effort estimates |
-| pyramid | PYRAMID_ANALYSIS.md | Current vs target distribution, gap table, prioritized action plan |
-| report | QA_STATUS_REPORT.md | Metrics, pyramid distribution, risk areas, adapted to audience level |
-
-## Instructions
-
-### AUDIT MODE (default)
-
-Scores across 6 dimensions: Locator Quality (20%), Assertion Specificity (20%), POM Compliance (15%), Test Coverage (20%), Naming Convention (15%), Test Data Management (10%).
-
-1. Read `CLAUDE.md` — quality gates, locator tiers, assertion rules, POM rules, naming conventions.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke validator agent in audit mode:
-
-Task(
-  prompt="
-    <objective>Audit test suite quality and produce QA_AUDIT_REPORT.md with 6-dimension scoring. If Playwright MCP is connected and an app URL is available, verify E2E test locators against the live DOM via browser_navigate + browser_snapshot. Flag stale locators (Tier 4 CSS/XPath that could be upgraded to Tier 1 data-testid) and locators that no longer match any DOM element.</objective>
-    <execution_context>@agents/qaa-validator.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: audit
-    app_url: {auto-detect from test config baseURL, or ask user}
-    </parameters>
-  "
-)
-
-3. Present results with overall score and prioritized recommendations.
-
----
-
-### PYRAMID MODE (`--pyramid`)
-
-Analyze test distribution against the ideal testing pyramid from CLAUDE.md (Unit 60-70%, Integration 10-15%, API 20-25%, E2E 3-5%). Compares actual percentages to targets and produces an action plan.
-
-1. Read `CLAUDE.md` — testing pyramid target percentages.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke analyzer agent for pyramid analysis:
-
-Task(
-  prompt="
-    <objective>Produce PYRAMID_ANALYSIS.md comparing actual test distribution to target pyramid. Count tests by type (unit, integration, API, E2E), calculate percentages, compare to CLAUDE.md targets, identify gaps, and produce a prioritized action plan to reach the recommended distribution. Adjust target percentages based on the actual app architecture.</objective>
-    <execution_context>@agents/qaa-analyzer.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: pyramid-analysis
-    </parameters>
-  "
-)
-
-3. Present analysis with gap table and action plan.
-
----
-
-### REPORT MODE (`--report`)
-
-Generate a summary report of current QA status. Adapts detail level to audience.
-
-**Audience levels:**
-- `team` (default) — file-level details, specific locator/assertion issues, technical recommendations
-- `management` — high-level metrics, risk areas, coverage percentages, trend indicators
-- `client` — coverage summary, confidence level, test pass rates, risk mitigation status
-
-1. Read `CLAUDE.md` — testing pyramid targets, quality gates.
-1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
-2. Invoke analyzer agent for status reporting:
-
-Task(
-  prompt="
-    <objective>Produce QA_STATUS_REPORT.md with current test suite metrics and coverage. Adapt detail level to the specified audience: team (file-level technical details), management (high-level metrics and risks), or client (coverage summary and confidence). Include testing pyramid distribution, pass/fail rates, risk areas, and actionable recommendations appropriate for the audience.</objective>
-    <execution_context>@agents/qaa-analyzer.md</execution_context>
-    <files_to_read>
-    - CLAUDE.md
-    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
-    </files_to_read>
-    <parameters>
-    user_input: $ARGUMENTS
-    mode: status-report
-    </parameters>
-  "
-)
-
-3. Present report to user.
-
-$ARGUMENTS
+# QA Audit & Report
+
+Comprehensive quality audit of a test suite with 6-dimension scoring, testing pyramid analysis, and status reporting. Supports three output modes: full audit, pyramid analysis only, or status report adapted to audience.
+
+
+## ⚠ MANDATORY: How to Execute This Command
+
+Execute these steps IN ORDER. Every step is mandatory. Do NOT improvise,
+reorder, skip, or invent steps.
+
+1. FIRST tool call: run the intent detector with `$ARGUMENTS` passed LITERALLY
+   (the actual command lives in the "## Intent Detection" section below).
+2. Print the INPUT DETECTION banner (exact output from intent-detector.cjs). Do
+   NOT show a custom-format banner.
+3. Determine `ALL_FROM_FLAGS`. If interactive (`IS_AUTO=false`) AND any value
+   came from NL/default (`ALL_FROM_FLAGS=false`), PAUSE for yes/no confirmation.
+   Do NOT proceed without it.
+   ```bash
+   ALL_FROM_FLAGS=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); const sources=Object.values(j.resolved).map(v=>v.source); console.log(sources.length>0 && sources.every(s=>s==='flag') ? 'true' : 'false')")
+   ```
+   Pause prompt (when applicable):
+   ```
+   The values above were resolved from natural language and/or defaults.
+   Do you want to continue with these inputs? (yes/no)
+   ```
+4. Determine MODE from the resolved values (not from your own NL parsing):
+   `--pyramid` → PYRAMID MODE, `--report` → REPORT MODE, otherwise AUDIT MODE.
+5. Read `CLAUDE.md` and `~/.claude/qaa/MY_PREFERENCES.md` (if it exists) before
+   invoking any agent.
+6. Execute the mode's flow IN ORDER (see "## Instructions"). Every mode invokes
+   its sub-agent via Task() with the `<critical_directive>` block — AUDIT MODE
+   spawns the validator; PYRAMID and REPORT MODE spawn the analyzer. Do NOT run
+   the audit/analysis inline; always go through the sub-agent. Present results
+   only after the agent returns.
+
+### DO NOT
+- Skip the INPUT DETECTION banner, or show a custom-format banner
+- Run AskUserQuestion, Bash, Read, or Glob before the intent detector
+- Decide the mode before the detector runs, or from your own NL parsing
+- Run the audit/pyramid/report analysis inline instead of spawning the sub-agent
+- Invent steps not in this command, or reorder the ones that are
+- Assume path, app URL, or any value from a previous command invocation — each
+  /qa-audit call is self-contained
+
+If you find yourself improvising, presenting results before the agent returns,
+or asking questions out of order — STOP and restart from step 1.
+
+### Pass user input literally to the intent detector
+
+When you run the intent detector with `$ARGUMENTS`, substitute that placeholder
+with the RAW user input as received, NOT a pre-translated flag string. If you
+pre-translate, the detector reports every value as `source: flag`, sets
+`ALL_FROM_FLAGS=true`, and bypasses the confirmation checkpoint.
+
+
+## Usage
+
+```
+/qa-audit <path-to-tests> [options]
+```
+
+### Options
+
+- `<path-to-tests>` — directory containing test files to audit
+- `--dev-repo <path>` — path to developer repository (for coverage cross-reference)
+- `--app-url <url>` — URL of running application for locator verification via Playwright MCP
+- `--pyramid` — pyramid analysis only: compare actual vs target distribution with action plan
+- `--report [team|management|client]` — generate status report adapted to audience (default: team)
+
+### Mode Detection
+
+```
+if --pyramid:
+  MODE = "pyramid"      → PYRAMID_ANALYSIS.md
+elif --report:
+  MODE = "report"       → QA_STATUS_REPORT.md (adapted to audience)
+else:
+  MODE = "audit"        → QA_AUDIT_REPORT.md (full 6-dimension audit, default)
+```
+
+## What It Produces
+
+| Mode | Artifact | Description |
+|------|----------|-------------|
+| audit | QA_AUDIT_REPORT.md | 6-dimension scoring, critical issues, recommendations with effort estimates |
+| pyramid | PYRAMID_ANALYSIS.md | Current vs target distribution, gap table, prioritized action plan |
+| report | QA_STATUS_REPORT.md | Metrics, pyramid distribution, risk areas, adapted to audience level |
+
+
+## Intent Detection
+
+Before processing arguments, run the intent detector to support natural-language input alongside flags. Flags always win over NL.
+
+```bash
+RESOLVED=$(node ~/.claude/qaa/bin/lib/intent-detector.cjs --resolve "$ARGUMENTS" --aliases '{"app_url":"app-url","dev_repo":"dev-repo"}')
+
+# Print INPUT DETECTION banner (always shown, even if defaults are used)
+echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.banner)"
+
+# Extract resolved values for use below
+APP_URL=$(echo "$RESOLVED" | node -e "const j=JSON.parse(require('fs').readFileSync(0,'utf8')); console.log(j.resolved.app_url?.value || '')")
+```
+
+If you see values you didn't intend in the banner above (interactive mode), you can press Ctrl-C and re-run with explicit flags.
+
+## Instructions
+
+### AUDIT MODE (default)
+
+Scores across 6 dimensions: Locator Quality (20%), Assertion Specificity (20%), POM Compliance (15%), Test Coverage (20%), Naming Convention (15%), Test Data Management (10%).
+
+1. Read `CLAUDE.md` — quality gates, locator tiers, assertion rules, POM rules, naming conventions.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke validator agent in audit mode:
+
+Task(
+  subagent_type="qaa-validator",
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-validator.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-validator.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Audit test suite quality and produce QA_AUDIT_REPORT.md with 6-dimension scoring. If Playwright MCP is connected and an app URL is available, verify E2E test locators against the live DOM via browser_navigate + browser_snapshot. Flag stale locators (Tier 4 CSS/XPath that could be upgraded to Tier 1 data-testid) and locators that no longer match any DOM element.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-validator.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    - .qa-output/locators/LOCATOR_REGISTRY.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: audit
+    app_url: {auto-detect from test config baseURL, or ask user}
+    </parameters>
+  "
+)
+
+3. Present results with overall score and prioritized recommendations.
+
+---
+
+### PYRAMID MODE (`--pyramid`)
+
+Analyze test distribution against the ideal testing pyramid from CLAUDE.md (Unit 60-70%, Integration 10-15%, API 20-25%, E2E 3-5%). Compares actual percentages to targets and produces an action plan.
+
+1. Read `CLAUDE.md` — testing pyramid target percentages.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke analyzer agent for pyramid analysis:
+
+Task(
+  subagent_type="qaa-analyzer",
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-analyzer.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-analyzer.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Produce PYRAMID_ANALYSIS.md comparing actual test distribution to target pyramid. Count tests by type (unit, integration, API, E2E), calculate percentages, compare to CLAUDE.md targets, identify gaps, and produce a prioritized action plan to reach the recommended distribution. Adjust target percentages based on the actual app architecture.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-analyzer.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: pyramid-analysis
+    </parameters>
+  "
+)
+
+3. Present analysis with gap table and action plan.
+
+---
+
+### REPORT MODE (`--report`)
+
+Generate a summary report of current QA status. Adapts detail level to audience.
+
+**Audience levels:**
+- `team` (default) — file-level details, specific locator/assertion issues, technical recommendations
+- `management` — high-level metrics, risk areas, coverage percentages, trend indicators
+- `client` — coverage summary, confidence level, test pass rates, risk mitigation status
+
+1. Read `CLAUDE.md` — testing pyramid targets, quality gates.
+1b. Read `~/.claude/qaa/MY_PREFERENCES.md` if it exists — user/company preferences override CLAUDE.md defaults.
+2. Invoke analyzer agent for status reporting:
+
+Task(
+  subagent_type="qaa-analyzer",
+  prompt="
+    <critical_directive priority="MANDATORY">
+    You are executing as a QAA pipeline agent. Your FIRST action MUST be to read
+    ~/.claude/qaa/agents/qaa-analyzer.md and adopt it as your operating contract.
+
+    Even though you may be running as a general-purpose agent (native subagent type
+    routing for QAA agents is not yet enabled), you MUST behave exactly as the
+    dedicated agent defined in that file: read everything in its <required_reading>,
+    obey its <quality_gate>, and run its mandatory bash checklist before returning.
+
+    Never modify application source code (src/, app/, lib/, or equivalent
+    directories) unless the agent's <scope> section explicitly permits it. If the
+    agent's contract restricts what files it can touch, honor that restriction —
+    even if a fix seems obvious or trivial.
+
+    If you cannot read ~/.claude/qaa/agents/qaa-analyzer.md (file missing, corrupted, parse
+    error), STOP and report the failure. Do NOT proceed with default agent
+    behavior — that defeats the purpose of this directive.
+
+    Do NOT improvise, substitute steps, apply changes the agent isn't allowed to
+    make, or skip any of the above. Skipping the agent's required_reading,
+    quality_gate, or bash checklist makes the run INVALID — its output cannot be
+    trusted.
+    </critical_directive>
+
+    <objective>Produce QA_STATUS_REPORT.md with current test suite metrics and coverage. Adapt detail level to the specified audience: team (file-level technical details), management (high-level metrics and risks), or client (coverage summary and confidence). Include testing pyramid distribution, pass/fail rates, risk areas, and actionable recommendations appropriate for the audience.</objective>
+    <execution_context>~/.claude/qaa/agents/qaa-analyzer.md</execution_context>
+    <files_to_read>
+    - CLAUDE.md
+    - ~/.claude/qaa/MY_PREFERENCES.md (if exists)
+    </files_to_read>
+    <parameters>
+    user_input: $ARGUMENTS
+    mode: status-report
+    </parameters>
+  "
+)
+
+3. Present report to user.
+
+$ARGUMENTS