qualia-framework 2.1.0

package/framework/skills/browser-qa/SKILL.md

@@ -0,0 +1,536 @@
+---
+name: browser-qa
+description: "Browser QA using Playwright. Navigates routes affected by git diff, captures console errors, broken links, and HTTP failures with screenshots. Web projects only."
+user-invocable: true
+tags: [qa, testing, playwright, browser, screenshots]
+---
+
+# Browser QA
+
+Automated browser testing with Playwright. Detects changed routes from git diff, navigates them, and reports runtime issues with screenshot evidence.
+
+## Usage
+
+- `/qa` -- Run QA on affected routes (from git diff against current branch's base)
+- `/qa --all` -- Run QA on all known routes (full scan)
+- `/qa --url <url>` -- Run QA on a specific URL
+
+## Step 1: Project Type Check
+
+Before anything else, determine if this is a web project. Run:
+
+```bash
+# Check for web project indicators
+HAS_WEB=false
+[ -f "package.json" ] && grep -qE '"next"|"react"|"vue"|"nuxt"|"astro"|"vite"|"gatsby"' package.json && HAS_WEB=true
+[ -f "index.html" ] && HAS_WEB=true
+[ -d "src/app" ] || [ -d "pages" ] || [ -d "src/pages" ] && HAS_WEB=true
+
+echo "HAS_WEB=$HAS_WEB"
+```
+
+**If HAS_WEB=false:** Stop immediately and output:
+
+> Browser QA skipped: This project is not a web application. /qa is designed for projects with a browser-rendered frontend (Next.js, React, Vue, etc.). Voice agents, Python agents, and CLI tools do not have browser routes to test.
+
+**If HAS_WEB=true:** Continue to Step 2.
+
+### Known Non-Web Projects (skip without checking)
+
+These projects are known to not be web apps:
+- **sigatalachana** -- Python agent (invoice OCR + POS)
+- **armenius** -- Cloudflare Workers webhook (no frontend routes)
+- Any project in `~/Projects/aiagents/` with no `package.json` containing a web framework
+- Any project with only `.py` files and no web framework (Flask/Django with templates are web)
+
+If the current directory name matches a known non-web project, skip immediately.
+
+## Step 2: Detect Affected Routes
+
+Determine which routes to test based on git changes.
+
+### Automatic Route Detection
+
+```bash
+# Get changed files relative to the merge base
+BRANCH=$(git rev-parse --abbrev-ref HEAD)
+BASE=$(git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null || echo "HEAD~5")
+CHANGED_FILES=$(git diff --name-only "$BASE"...HEAD 2>/dev/null || git diff --name-only HEAD~5)
+
+echo "Changed files:"
+echo "$CHANGED_FILES"
+```
+
+### Route Mapping Rules
+
+Map changed files to browser routes:
+
+| File Pattern | Route |
+|-------------|-------|
+| `src/app/<path>/page.tsx` | `/<path>` |
+| `src/app/page.tsx` | `/` |
+| `pages/<path>.tsx` | `/<path>` |
+| `pages/index.tsx` | `/` |
+| `src/app/api/**` | Skip (API routes, not pages) |
+| `src/components/**` | Find which pages import this component |
+| `src/lib/**`, `src/utils/**` | Test `/` (shared code, test homepage) |
+| `public/**` | Test `/` (static assets) |
+| `styles/**`, `*.css` | Test all layout-level pages (`/`, plus any page using the stylesheet) |
+| `src/app/layout.tsx` | Test `/` and one subpage |
+
+### Component-to-Page Resolution
+
+When a component file changes, find which pages use it:
+
+```bash
+# For a changed component, find importing pages
+COMPONENT_NAME=$(basename "$CHANGED_FILE" .tsx)
+grep -rl "$COMPONENT_NAME" src/app/*/page.tsx pages/*.tsx 2>/dev/null
+```
+
+### Fallback
+
+If no routes can be derived (e.g., only config files changed), test the homepage (`/`).
+
+If `--all` flag is used, discover all routes:
+
+```bash
+# Next.js App Router
+find src/app -name "page.tsx" -o -name "page.jsx" | sed 's|src/app||;s|/page\.[tj]sx||;s|^$|/|'
+
+# Next.js Pages Router
+find pages -name "*.tsx" -o -name "*.jsx" | grep -v "_app\|_document\|api/" | sed 's|pages||;s|\.[tj]sx||;s|/index$|/|'
+```
+
+## Step 3: Run Playwright QA
+
+### Prerequisites
+
+Verify Playwright is available:
+
+```bash
+npx playwright --version
+```
+
+If not installed or browsers not downloaded:
+
+```bash
+npm init -y 2>/dev/null  # Only if no package.json
+npx playwright install chromium
+```
+
+### Dev Server Detection
+
+Check if a dev server is already running:
+
+```bash
+# Check common ports
+curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null
+curl -s -o /dev/null -w "%{http_code}" http://localhost:5173 2>/dev/null
+curl -s -o /dev/null -w "%{http_code}" http://localhost:4321 2>/dev/null
+```
+
+If no server is running, start one:
+
+```bash
+# Detect and start the dev server
+if grep -q '"dev"' package.json; then
+  npm run dev &
+  DEV_PID=$!
+  sleep 5  # Wait for server startup
+fi
+```
+
+Record the base URL (e.g., `http://localhost:3000`).
+
+### QA Script
+
+For each route, run this Playwright script:
+
+```bash
+npx playwright test --config=- <<'PLAYWRIGHT_CONFIG'
+import { test, expect } from '@playwright/test';
+
+const BASE_URL = process.env.QA_BASE_URL || 'http://localhost:3000';
+const ROUTES = process.env.QA_ROUTES?.split(',') || ['/'];
+const SCREENSHOT_DIR = '.qa-screenshots';
+
+test.describe('Browser QA', () => {
+  for (const route of ROUTES) {
+    test(`QA: ${route}`, async ({ page }) => {
+      const errors: string[] = [];
+      const failedRequests: { url: string; status: number }[] = [];
+
+      // Capture console errors
+      page.on('console', msg => {
+        if (msg.type() === 'error') {
+          errors.push(`Console error: ${msg.text()}`);
+        }
+      });
+
+      // Capture failed HTTP requests
+      page.on('response', response => {
+        if (response.status() >= 400) {
+          failedRequests.push({ url: response.url(), status: response.status() });
+        }
+      });
+
+      // Navigate
+      await page.goto(`${BASE_URL}${route}`, { waitUntil: 'networkidle', timeout: 30000 });
+
+      // Screenshot
+      await page.screenshot({ path: `${SCREENSHOT_DIR}${route === '/' ? '/index' : route}.png`, fullPage: true });
+
+      // Check for broken links
+      const links = await page.$$eval('a[href]', anchors =>
+        anchors.map(a => a.getAttribute('href')).filter(href => href && !href.startsWith('#') && !href.startsWith('mailto:') && !href.startsWith('tel:'))
+      );
+
+      for (const link of links.slice(0, 20)) {  // Cap at 20 links per page
+        if (link.startsWith('/') || link.startsWith(BASE_URL)) {
+          const fullUrl = link.startsWith('/') ? `${BASE_URL}${link}` : link;
+          const resp = await page.request.get(fullUrl);
+          if (resp.status() >= 400) {
+            failedRequests.push({ url: fullUrl, status: resp.status() });
+          }
+        }
+      }
+
+      // Report findings (test passes but logs findings)
+      if (errors.length > 0 || failedRequests.length > 0) {
+        console.log(JSON.stringify({ route, errors, failedRequests }));
+      }
+    });
+  }
+});
+PLAYWRIGHT_CONFIG
+```
+
+**Alternative approach (simpler, no config file needed):**
+
+Instead of the Playwright test runner, use a Node.js script with Playwright library directly:
+
+```bash
+QA_BASE_URL="http://localhost:3000" QA_ROUTES="/" node -e "
+const { chromium } = require('playwright');
+(async () => {
+  const browser = await chromium.launch();
+  const baseUrl = process.env.QA_BASE_URL;
+  const routes = process.env.QA_ROUTES.split(',');
+  const findings = [];
+
+  for (const route of routes) {
+    const context = await browser.newContext();
+    const page = await context.newPage();
+    const errors = [];
+    const failedRequests = [];
+
+    page.on('console', msg => { if (msg.type() === 'error') errors.push(msg.text()); });
+    page.on('response', resp => { if (resp.status() >= 400) failedRequests.push({ url: resp.url(), status: resp.status() }); });
+
+    await page.goto(baseUrl + route, { waitUntil: 'networkidle', timeout: 30000 });
+    await page.screenshot({ path: '.qa-screenshots' + (route === '/' ? '/index' : route) + '.png', fullPage: true });
+
+    // Check internal links
+    const links = await page.$$eval('a[href]', as => as.map(a => a.href).filter(h => h.startsWith(baseUrl) || h.startsWith('/')));
+    for (const link of links.slice(0, 20)) {
+      const url = link.startsWith('/') ? baseUrl + link : link;
+      try { const r = await page.request.get(url); if (r.status() >= 400) failedRequests.push({ url, status: r.status() }); } catch(e) { failedRequests.push({ url, status: 0 }); }
+    }
+
+    if (errors.length || failedRequests.length) findings.push({ route, errors, failedRequests });
+    await context.close();
+  }
+
+  console.log(JSON.stringify(findings, null, 2));
+  await browser.close();
+})();
+"
+```
+
+### Screenshot Capture
+
+Every page navigated gets a full-page screenshot saved to `.qa-screenshots/`:
+
+- Route `/` -> `.qa-screenshots/index.png`
+- Route `/about` -> `.qa-screenshots/about.png`
+- Route `/dashboard/settings` -> `.qa-screenshots/dashboard/settings.png`
+
+Create the directory before running:
+
+```bash
+mkdir -p .qa-screenshots
+```
+
+When a finding is reported, reference the screenshot path in the report:
+
+> Finding: Console error "TypeError: Cannot read property 'map' of undefined" on /dashboard
+> Screenshot: `.qa-screenshots/dashboard.png`
+
+## Step 4: Generate QA Report
+
+After all routes are tested, produce a structured report.
+
+### Report Format
+
+```markdown
+# QA Report -- {project-name}
+**Date:** {ISO timestamp}
+**Routes tested:** {N}
+**Findings:** {N} ({errors} console errors, {broken_links} broken links, {http_failures} HTTP failures)
+**Screenshots:** .qa-screenshots/
+
+## Findings
+
+### Finding 1: {type} on {route}
+- **Type:** console-error | broken-link | http-failure
+- **Route:** {route}
+- **Detail:** {error message or URL + status code}
+- **Screenshot:** .qa-screenshots/{route}.png
+- **False positive:** no (set to yes if confirmed false positive)
+- **Severity:** critical | warning | info
+
+### Finding 2: ...
+
+## Summary
+
+| Route | Status | Console Errors | Broken Links | HTTP Failures |
+|-------|--------|----------------|--------------|---------------|
+| / | PASS/FAIL | 0 | 0 | 0 |
+
+## Routes Tested
+{list of routes with pass/fail status}
+```
+
+### Severity Classification
+
+- **critical:** Page fails to load, unhandled exceptions, 500 errors
+- **warning:** 404 broken links, non-critical console errors, slow responses (>3s)
+- **info:** Console warnings (not errors), deprecation notices
+
+### Clean Run
+
+If zero findings:
+
+> QA PASSED: {N} routes tested, zero findings. Screenshots saved to .qa-screenshots/
+
+## Step 5: Cleanup
+
+```bash
+# Kill dev server if we started one
+[ -n "$DEV_PID" ] && kill $DEV_PID 2>/dev/null
+
+# Keep screenshots for evidence (don't delete)
+echo "Screenshots saved to .qa-screenshots/"
+```
+
+Add `.qa-screenshots/` to `.gitignore` if not already present:
+
+```bash
+grep -q ".qa-screenshots" .gitignore 2>/dev/null || echo ".qa-screenshots/" >> .gitignore
+```
+
+## Step 6: Bug Fix Workflow (BQA-03)
+
+When QA findings reveal real bugs, fix each one with an atomic commit and a regression test.
+
+### Process Per Finding
+
+For each confirmed finding (not a false positive):
+
+#### 1. Create a regression test FIRST (Red)
+
+Write a test that reproduces the exact issue found:
+
+```bash
+# Example: Console error on /dashboard
+cat > __tests__/qa-regression/dashboard-console-error.test.ts << 'EOF'
+import { test, expect } from '@playwright/test';
+
+test('dashboard should not have console errors', async ({ page }) => {
+  const errors: string[] = [];
+  page.on('console', msg => {
+    if (msg.type() === 'error') errors.push(msg.text());
+  });
+
+  await page.goto('http://localhost:3000/dashboard');
+  await page.waitForLoadState('networkidle');
+
+  expect(errors).toHaveLength(0);
+});
+EOF
+```
+
+```bash
+# Example: Broken link on /about
+cat > __tests__/qa-regression/about-broken-links.test.ts << 'EOF'
+import { test, expect } from '@playwright/test';
+
+test('about page should have no broken internal links', async ({ page }) => {
+  await page.goto('http://localhost:3000/about');
+  const links = await page.$$eval('a[href^="/"]', as => as.map(a => a.getAttribute('href')));
+
+  for (const link of links) {
+    const response = await page.request.get(`http://localhost:3000${link}`);
+    expect(response.status()).toBeLessThan(400);
+  }
+});
+EOF
+```
+
+#### 2. Fix the bug
+
+Apply the minimal fix. The fix should be scoped to the exact issue -- no drive-by refactoring.
+
+#### 3. Verify the test passes (Green)
+
+```bash
+npx playwright test __tests__/qa-regression/ --reporter=list
+```
+
+#### 4. Atomic commit
+
+Each bug fix gets its own commit:
+
+```bash
+git add <fixed-files> __tests__/qa-regression/<test-file>
+git commit -m "fix(qa): resolve {finding-type} on {route}
+
+QA finding: {brief description}
+Regression test: __tests__/qa-regression/{test-file}
+"
+```
+
+### Naming Convention for Regression Tests
+
+```
+__tests__/qa-regression/{route}-{finding-type}.test.ts
+```
+
+Examples:
+- `__tests__/qa-regression/dashboard-console-error.test.ts`
+- `__tests__/qa-regression/about-broken-links.test.ts`
+- `__tests__/qa-regression/index-http-failure.test.ts`
+
+### When NOT to Create Regression Tests
+
+- **False positives** -- skip entirely
+- **Third-party script errors** (e.g., analytics, chat widgets) -- suppress instead of fix
+- **Environment-only issues** (e.g., dev-only CORS) -- note in report, don't test
+
+## Step 7: False Positive Tracking (BQA-06)
+
+Track the false positive rate across QA runs. If the cumulative rate exceeds 20%, surface a tuning warning.
+
+### Recording Results
+
+After each QA run, append results to `.qa-history.json` in the project root:
+
+```bash
+# Read existing history or create empty
+HISTORY_FILE=".qa-history.json"
+[ -f "$HISTORY_FILE" ] || echo '{"runs":[]}' > "$HISTORY_FILE"
+
+# Append this run's results
+node -e "
+const fs = require('fs');
+const history = JSON.parse(fs.readFileSync('$HISTORY_FILE', 'utf8'));
+history.runs.push({
+  date: new Date().toISOString(),
+  routes_tested: ${ROUTES_TESTED},
+  total_findings: ${TOTAL_FINDINGS},
+  false_positives: ${FALSE_POSITIVES},
+  true_positives: ${TRUE_POSITIVES}
+});
+fs.writeFileSync('$HISTORY_FILE', JSON.stringify(history, null, 2));
+"
+```
+
+### Calculating the Rate
+
+After recording, calculate the cumulative false positive rate:
+
+```bash
+node -e "
+const fs = require('fs');
+const history = JSON.parse(fs.readFileSync('.qa-history.json', 'utf8'));
+const totals = history.runs.reduce((acc, run) => ({
+  findings: acc.findings + run.total_findings,
+  fp: acc.fp + run.false_positives
+}), { findings: 0, fp: 0 });
+
+const rate = totals.findings > 0 ? (totals.fp / totals.findings * 100).toFixed(1) : 0;
+console.log('False positive rate: ' + rate + '% (' + totals.fp + '/' + totals.findings + ')');
+
+if (parseFloat(rate) > 20) {
+  console.log('');
+  console.log('WARNING: False positive rate exceeds 20%. QA detection rules need tuning.');
+  console.log('Review common false positives:');
+  console.log('  - Third-party script errors (add to suppression list)');
+  console.log('  - Dev-only warnings (filter by environment)');
+  console.log('  - Known flaky routes (add to .qa-suppress.json)');
+}
+"
+```
+
+### Tuning Warning Output
+
+When rate > 20%, append this to the QA report:
+
+```markdown
+## Tuning Warning
+
+False positive rate is {rate}% ({fp}/{total} findings across {N} runs).
+This exceeds the 20% threshold. Consider:
+
+1. **Add suppressions** to `.qa-suppress.json` for known non-issues
+2. **Filter third-party errors** -- console errors from external scripts (analytics, chat widgets) are not your bugs
+3. **Exclude flaky routes** -- routes requiring auth or dynamic data may produce false 4xx errors
+4. **Review detection rules** -- some console warnings are not actionable errors
+```
+
+### Suppression File
+
+Create `.qa-suppress.json` to exclude known false positives from future runs:
+
+```json
+{
+  "suppress_console_patterns": [
+    ".*third-party-analytics.*",
+    ".*Failed to load resource.*favicon\\.ico.*",
+    ".*DevTools.*"
+  ],
+  "suppress_urls": [
+    "/api/auth/callback"
+  ],
+  "suppress_routes": []
+}
+```
+
+When running QA (Step 3), check suppressions before recording a finding:
+
+```javascript
+// In the Playwright script, before recording a console error:
+const suppressions = JSON.parse(fs.readFileSync('.qa-suppress.json', 'utf8'));
+const isSuppressed = suppressions.suppress_console_patterns.some(p => new RegExp(p).test(errorText));
+if (!isSuppressed) {
+  // Record finding
+}
+```
+
+### Marking False Positives
+
+In the QA report, each finding has a `False positive: no` field. After review:
+
+1. Change to `False positive: yes` for confirmed false positives
+2. Add the pattern to `.qa-suppress.json` to prevent recurrence
+3. Update `.qa-history.json` with corrected counts
+
+### History File in .gitignore
+
+```bash
+grep -q ".qa-history.json" .gitignore 2>/dev/null || echo ".qa-history.json" >> .gitignore
+grep -q ".qa-suppress.json" .gitignore 2>/dev/null || echo "# Keep .qa-suppress.json IN git -- it's project config, not output" >> /dev/null
+```
+
+Note: `.qa-suppress.json` SHOULD be committed (it's configuration). `.qa-history.json` should NOT be committed (it's local state).