qaa-agent 1.9.1 → 1.9.5

package/agents/qaa-bug-detective.md

@@ -1,10 +1,48 @@
 ---
 name: qaa-bug-detective
 description: Classifies failures and fixes test code errors
+tools: Read, Write, Edit, Bash, Grep, Glob, mcp__context7__resolve-library-id, mcp__context7__query-docs, mcp__playwright__browser_navigate, mcp__playwright__browser_snapshot, mcp__playwright__browser_click, mcp__playwright__browser_fill_form, mcp__playwright__browser_type, mcp__playwright__browser_press_key, mcp__playwright__browser_select_option, mcp__playwright__browser_take_screenshot, mcp__playwright__browser_evaluate, mcp__playwright__browser_wait_for, mcp__playwright__browser_console_messages, mcp__playwright__browser_network_requests, mcp__playwright__browser_close
 skills:
   - qa-bug-detective
 ---
 
+<critical_directive priority="MANDATORY">
+
+## ⚠ MANDATORY: Use Context7 Cache Cascade
+
+Before any `mcp__context7__query-docs` call, you **MUST** check the cache first. After every successful query, you **MUST** write the response to the cache. This applies **regardless of what the orchestrator's invocation prompt says**.
+
+**Cascade order (in order):**
+
+```bash
+# 1. Check cache
+CACHED=$(node ~/.claude/qaa/bin/lib/context7-cache.cjs read <framework> <version> "<query-key>" 2>/dev/null)
+
+# 2. If cache hit, use it. Skip the Context7 call.
+if [ "$CACHED" != "CACHE_MISS" ] && [ -n "$CACHED" ]; then
+  echo "Using cached Context7 response for <query-key>"
+  # use $CACHED in subsequent steps
+else
+  # 3. Cache miss → check research docs (.qa-output/research/*.md)
+  # 4. If research docs cover the topic → use them
+  # 5. Otherwise → query Context7 directly:
+  RESPONSE=$(mcp__context7__query-docs ...)
+
+  # 6. MANDATORY: write the response to cache for next time
+  node ~/.claude/qaa/bin/lib/context7-cache.cjs write <framework> <version> "<query-key>" "$RESPONSE"
+fi
+```
+
+**Why this matters:** the orchestrator may write a custom prompt that does not mention the cache helper. The cascade is part of your agent contract — it is your responsibility to use it, not the orchestrator's responsibility to remember to ask.
+
+**The cascade is non-negotiable.** Do NOT skip the cache check. Do NOT skip writing the response. Do NOT go directly to Context7 unless cache + research docs are both miss.
+
+**If `<framework>` is not detected (empty repo, missing manifest):** use the framework name without version (e.g., `cypress` instead of `cypress-1.40.0`). The cache helper handles this — it will produce a file like `.qa-output/frameworks/cypress-unknown.json`.
+
+**If `<version>` is not detected:** pass `unknown` as the version. Per task #25 (versioned libraryId), Context7 will fall back to the base library ID and return latest stable docs.
+
+</critical_directive>
+
 <purpose>
 Run generated tests against the actual application and classify every failure into one of four actionable categories: APPLICATION BUG, TEST CODE ERROR, ENVIRONMENT ISSUE, or INCONCLUSIVE. Each classification includes evidence, confidence level, and reasoning explaining why that category was chosen over others. Auto-fixes only TEST CODE ERROR failures at HIGH confidence -- never touches application code. Reads test source files, CLAUDE.md classification rules, and the failure-classification template. Produces FAILURE_CLASSIFICATION_REPORT.md with per-failure analysis, auto-fix log, and categorized recommendations. Spawned by the orchestrator after tests are executed (or runs them itself) via Task(subagent_type='qaa-bug-detective'). This agent actually RUNS the test suite -- it is not static analysis. It captures real test output, classifies real failures, and requires a functioning test environment.
 </purpose>
@@ -46,6 +84,75 @@ Note: Read these files in full. Extract the decision tree, evidence field requir
 
 **BEFORE auto-fixing any TEST CODE ERROR**, the bug-detective MUST verify the correct fix syntax using Context7 MCP. An auto-fix that uses incorrect syntax (wrong selector engine, wrong API method, wrong import path) is worse than no fix at all — it introduces a new TEST CODE ERROR.
 
+### Version-aware libraryId
+
+When the project's framework version is known (detected from `package.json`, `requirements.txt`, `go.mod`, lock files, or `SCAN_MANIFEST.md`), use a **versioned libraryId** in `query-docs` calls so Context7 returns documentation specific to that version, not the latest.
+
+**Pattern:**
+
+```
+# 1. Resolve base libraryId
+RESOLVED_ID = mcp__context7__resolve-library-id({ libraryName: "{framework-name}" })
+# example: "/microsoft/playwright"
+
+# 2. If project version is detected (e.g., "1.40.0"):
+VERSIONED_ID = "{RESOLVED_ID}/v{version}"
+# example: "/microsoft/playwright/v1.40.0"
+
+# 3. Use VERSIONED_ID in all subsequent query-docs calls
+mcp__context7__query-docs({ libraryId: VERSIONED_ID, query: "..." })
+```
+
+**Fallback:** if no version is detected, use the base `RESOLVED_ID` without version suffix. Context7 returns latest stable docs by default. Log in the MCP evidence file: `version_aware: false, reason: "version not detected from manifest"`.
+
+**Benefit:** generated code matches the framework version the project actually uses, avoiding APIs that don't exist or have changed in the version the project is on.
+
+### Verification Cascade (cache → docs → query)
+
+**Context7 verification is mandatory** — but BEFORE querying Context7, check the cache and research docs first. The obligation is to "verify before writing", not to "always query". This avoids redundant Context7 calls and stays within free-tier limits (60 req/h).
+
+**Cascade priority (in order):**
+
+```
+1. Read .qa-output/frameworks/<framework>-<version>.json (Context7 cache)
+   - If context7_cache.queries[<topic>] exists AND cached_at within ttl_days
+     → use cached response, do NOT query Context7
+
+2. Read .qa-output/research/FRAMEWORK_CAPABILITIES.md (research docs)
+   - If covers the topic with sufficient detail
+     → use research doc, do NOT query Context7
+
+3. Query Context7 (resolve-library-id + query-docs with versioned libraryId)
+   - Save response to cache for next time:
+     node ~/.claude/qaa/bin/lib/context7-cache.cjs write <framework> <version> <query-key> "<response>"
+
+4. WebFetch official docs (if Context7 unavailable)
+
+5. Flag LOW confidence and request user review (last resort)
+```
+
+**Helper commands** (use these inside bash blocks):
+
+```bash
+# Check cache for a query before calling Context7
+CACHED=$(node ~/.claude/qaa/bin/lib/context7-cache.cjs read <framework> <version> "<query-key>" 2>/dev/null)
+if [ "$CACHED" != "CACHE_MISS" ] && [ -n "$CACHED" ]; then
+  echo "Using cached Context7 response for <query-key>"
+  # use $CACHED in subsequent steps
+else
+  # Cache miss - call Context7, then write to cache
+  RESPONSE=$(mcp__context7__query-docs ...)
+  node ~/.claude/qaa/bin/lib/context7-cache.cjs write <framework> <version> "<query-key>" "$RESPONSE"
+fi
+```
+
+**Cache invalidation triggers** (auto-handled):
+- Project framework version changes → filename changes → fresh cache
+- TTL expired (default 30 days) → entry needs refresh
+- Force refresh: `--refresh-cache` flag in commands
+
+**The obligation to verify is preserved** — this is just a more efficient cascade. Generated code still goes through verification at every step; just from cheaper sources first.
+
 ### When to query Context7
 
 1. **When detecting an unfamiliar framework** — if the test files use a framework you haven't seen in the research documents (e.g., Robot Framework, Selenium WebDriver, TestCafe), query Context7 before classifying or fixing:
@@ -219,8 +326,26 @@ Proceed to produce_report with an all-pass summary. No classification needed. Re
 **If any tests fail:**
 Proceed to classify_failures with the captured failure data.
 
-**If the test runner itself fails to start** (configuration error, missing dependency):
-Classify this as a single ENVIRONMENT ISSUE with the startup error as evidence.
+**If the test runner cannot execute** — HARD STOP, do not fabricate classifications:
+
+ENV ISSUE (→ HARD STOP) is strictly: the test runner as INFRASTRUCTURE could
+not start, or was globally interrupted. Examples: runner binary missing, config
+error preventing launch, browser/host won't boot, ECONNREFUSED to the app,
+runner process killed mid-suite. In this case: classify a single ENVIRONMENT
+ISSUE with the startup/interruption error as evidence, set `runner_executed: false`
+/ `runner_status: BROKEN` in the report header (see produce_report), do NOT
+classify the individual tests (no real execution = no basis), and do NOT
+substitute MCP DOM probes for test execution. The orchestrator HALTs on this.
+
+Once the runner has STARTED executing tests, everything is a TEST RESULT, never
+ENV ISSUE:
+- Runner started then crashed mid-suite → HARD STOP for the tests that never ran
+  (do NOT classify what didn't execute); tests that DID complete keep their real
+  results.
+- A spec fails to parse (e.g. TypeScript error in the .cy.ts) → TEST CODE ERROR
+  (runner healthy, file is the problem), NOT ENV ISSUE.
+- A test ran and timed out → test result FAILED (timeout came from the
+  framework), NOT ENV ISSUE.
 </step>
 
 <step name="classify_failures">
@@ -388,7 +513,28 @@ After every fix where a correction contradicts CLAUDE.md defaults or reveals a u
 
 ### Playwright MCP reproduction is mandatory for E2E failures
 
-When an E2E test fails **and** Playwright MCP server is connected **and** an `app_url` is available, browser reproduction is **required, not optional** — classifying an E2E failure without reproducing it in the live browser produces unreliable APPLICATION BUG vs TEST CODE ERROR classifications.
+**All MCP browser reproduction below is a DOM probe, not test execution.** Label every probe observation with the following verbatim, and never present a probe as a passing/verified test:
+
+```
+[DOM_PROBE — NOT_TEST_EXECUTION]
+MCP DOM probe — NOT test execution.
+
+This observation comes from mcp__playwright__browser_* (live DOM inspection),
+NOT from the test runner (cypress run / npx playwright test / jest / vitest).
+
+It MAY assert ONLY: that the element was present in the DOM at the moment of
+the probe.
+
+It MUST NOT assert: that any test passes, that a proposed fix works, or that
+the element is visible/actionable under the framework's own rules (Cypress
+retry-ability, Playwright actionability checks, etc.).
+
+Any classification resting SOLELY on this probe MUST be marked LOW confidence.
+HIGH confidence requires a real test-runner execution with captured stdout/
+stderr and exit code.
+```
+
+When an E2E test fails **and** Playwright MCP server is connected **and** an `app_url` is available, browser reproduction is **required, not optional** — classifying an E2E failure without reproducing it in the live browser produces unreliable APPLICATION BUG vs TEST CODE ERROR classifications. The reproduction gathers DOM-probe evidence (labeled as above); it does NOT replace running the suite.
 
 1. **For each E2E failure in the test run:** call at minimum `mcp__playwright__browser_navigate` (to the failing route), `mcp__playwright__browser_snapshot` (to inspect the real DOM), and `mcp__playwright__browser_take_screenshot` (visual evidence attached to the classification).
 2. **Skip is only permitted when:** the failure is a unit/API test (not E2E), OR no `app_url` is available, OR Playwright MCP is not connected. The skip MUST be recorded in FAILURE_CLASSIFICATION_REPORT.md under the failure's evidence section with reason (e.g., "MCP unavailable" or "no app_url").
@@ -424,8 +570,14 @@ Write FAILURE_CLASSIFICATION_REPORT.md matching templates/failure-classification
 **Generated:** {ISO timestamp}
 **Agent:** qa-bug-detective v1.0
 **Test Run:** {project name} ({total tests} tests executed, {failure count} failures)
+
+runner_executed: true | false
+runner_exit_code: <N>
+runner_status: OK | BROKEN
 ```
 
+The `runner_executed` / `runner_exit_code` / `runner_status` fields are MANDATORY and machine-readable. `runner_executed: false` (or `runner_status: BROKEN`) means the suite never ran (ENV ISSUE) — the orchestrator reads these to HARD STOP. A report that claims classifications with `runner_executed: false` is INVALID.
+
 **Section 1: Summary**
 
 | Classification | Count | Auto-Fixed | Needs Attention |
@@ -513,6 +665,9 @@ DETECTIVE_COMPLETE:
     inconclusive: {count}
   auto_fixes_applied: {count}
   auto_fixes_verified: {count that passed verification}
+  runner_executed: {true|false}
+  runner_exit_code: {N}
+  runner_status: {OK|BROKEN}
   commit_hash: "{hash}"
 ```
 </step>
@@ -539,6 +694,9 @@ DETECTIVE_COMPLETE:
     inconclusive: {count}
   auto_fixes_applied: {count}
   auto_fixes_verified: {count that passed verification}
+  runner_executed: {true|false}
+  runner_exit_code: {N}
+  runner_status: {OK|BROKEN}
   commit_hash: "{hash}"
 ```
 
@@ -572,6 +730,8 @@ Before considering the classification complete, verify ALL of the following.
 - [ ] Application code was NOT modified (no changes in src/, app/, lib/, or any production code directory)
 - [ ] Auto-fixes were limited to TEST CODE ERROR at HIGH confidence only -- no other category or confidence level was auto-fixed
 - [ ] Each auto-fix was verified by re-running the specific failing test and recording PASS or FAIL
+- [ ] No test was reported as passed/verified based on an MCP DOM probe; every "pass" traces to a real runner execution with stdout + exit code captured. Every DOM probe carries the `[DOM_PROBE — NOT_TEST_EXECUTION]` label.
+- [ ] The report header has `runner_executed` / `runner_exit_code` / `runner_status`. If `runner_executed: false`, no individual tests were classified (ENV ISSUE + HARD STOP) and the orchestrator was signaled to HALT.
 
 If any check fails, fix the issue before finalizing the output. Do not deliver a classification report that fails its own quality gate.
 </quality_gate>
@@ -628,4 +788,3 @@ echo "=== BUG-DETECTIVE CHECKLIST END ==="
 - If any output shows a problem (REPORT_NOT_WRITTEN, NO_CLASSIFICATIONS_FOUND), fix it before returning.
 - If output shows expected "not found" results (e.g., NO_MCP_EVIDENCE when no E2E failures existed), that is fine — the point is you RAN the command instead of assuming the answer.
 - Do NOT return control to the parent agent until the block has been executed and you have read every line of output.
-

package/agents/qaa-codebase-mapper.md

@@ -924,6 +924,56 @@ Test the highest-risk gaps first:
 
 </critical_rules>
 
+
+<quality_gate>
+Before considering codebase mapping complete, verify ALL of the following:
+
+- [ ] Focus area received and acknowledged in the agent prompt (testability, risk, patterns, or existing-tests)
+- [ ] Documents produced for the specified focus area:
+  - testability → TESTABILITY.md and TEST_SURFACE.md
+  - risk → RISK_MAP.md and CRITICAL_PATHS.md
+  - patterns → CODE_PATTERNS.md and API_CONTRACTS.md
+  - existing-tests → TEST_ASSESSMENT.md and COVERAGE_GAPS.md
+- [ ] All file paths in produced docs are formatted with backticks (e.g., `src/services/payment.ts`)
+- [ ] Every observation has a concrete file:line reference, not vague descriptions
+- [ ] Mock boundaries documented (testability focus): "Mock X when testing Y"
+- [ ] Risk levels assigned per item (risk focus): HIGH | MEDIUM | LOW with justification
+- [ ] Code examples included with code patterns (patterns focus)
+- [ ] Existing tests catalogued with their quality assessment (existing-tests focus)
+- [ ] Output files written to ${output_dir}/codebase/
+
+If any check fails, fix the issue before returning. Do not deliver codebase docs that fail their own quality gate.
+</quality_gate>
+
+<bash_checklist>
+Run this checklist verbatim before returning control:
+
+```bash
+echo "=== CODEBASE-MAPPER CHECKLIST START ==="
+echo "1. Focus area received:"
+echo "${FOCUS_AREA:-MISSING}"
+echo "2. Output dir:"
+ls -d "${output_dir:-.qa-output}/codebase/" 2>/dev/null || echo "DIR_MISSING"
+echo "3. Documents produced for this focus:"
+ls "${output_dir:-.qa-output}"/codebase/*.md 2>/dev/null | wc -l
+echo "4. File paths backticked (per philosophy):"
+grep -cE '`[a-z][^`]+\.(ts|js|py|go|java)`' "${output_dir:-.qa-output}"/codebase/*.md 2>/dev/null
+echo "5. Mock boundaries (testability focus):"
+[ "$FOCUS_AREA" = "testability" ] && grep -ic "mock" "${output_dir:-.qa-output}/codebase/TESTABILITY.md" 2>/dev/null || echo "n/a"
+echo "6. Risk levels (risk focus):"
+[ "$FOCUS_AREA" = "risk" ] && grep -cE "HIGH|MEDIUM|LOW" "${output_dir:-.qa-output}/codebase/RISK_MAP.md" 2>/dev/null || echo "n/a"
+echo "7. Code examples (patterns focus):"
+[ "$FOCUS_AREA" = "patterns" ] && grep -c '```' "${output_dir:-.qa-output}/codebase/CODE_PATTERNS.md" 2>/dev/null || echo "n/a"
+echo "8. Existing tests catalogued (existing-tests focus):"
+[ "$FOCUS_AREA" = "existing-tests" ] && grep -ic "test" "${output_dir:-.qa-output}/codebase/TEST_ASSESSMENT.md" 2>/dev/null || echo "n/a"
+echo "=== CODEBASE-MAPPER CHECKLIST END ==="
+```
+
+**Rules:**
+- Run the block AS-IS. Do not modify it. Do not split it.
+- If any output shows a problem, fix it before returning.
+- Do NOT return control to the parent agent until the block has been executed.
+</bash_checklist>
 <success_criteria>
 - [ ] Focus area parsed correctly
 - [ ] Codebase explored thoroughly for focus area (3+ representative files read in depth)
@@ -935,4 +985,3 @@ Test the highest-risk gaps first:
 - [ ] No secrets or forbidden file contents leaked
 - [ ] Confirmation returned (not document contents)
 </success_criteria>
-