qualia-framework 2.5.1 → 3.1.0

package/framework/skills/browser-qa/SKILL.md

@@ -1,536 +0,0 @@
----
-name: browser-qa
-description: "Browser QA using Playwright. Navigates routes affected by git diff, captures console errors, broken links, and HTTP failures with screenshots. Web projects only."
-user-invocable: true
-tags: [qa, testing, playwright, browser, screenshots]
----
-
-# Browser QA
-
-Automated browser testing with Playwright. Detects changed routes from git diff, navigates them, and reports runtime issues with screenshot evidence.
-
-## Usage
-
-- `/qa` -- Run QA on affected routes (from git diff against current branch's base)
-- `/qa --all` -- Run QA on all known routes (full scan)
-- `/qa --url <url>` -- Run QA on a specific URL
-
-## Step 1: Project Type Check
-
-Before anything else, determine if this is a web project. Run:
-
-```bash
-# Check for web project indicators
-HAS_WEB=false
-[ -f "package.json" ] && grep -qE '"next"|"react"|"vue"|"nuxt"|"astro"|"vite"|"gatsby"' package.json && HAS_WEB=true
-[ -f "index.html" ] && HAS_WEB=true
-[ -d "src/app" ] || [ -d "pages" ] || [ -d "src/pages" ] && HAS_WEB=true
-
-echo "HAS_WEB=$HAS_WEB"
-```
-
-**If HAS_WEB=false:** Stop immediately and output:
-
-> Browser QA skipped: This project is not a web application. /qa is designed for projects with a browser-rendered frontend (Next.js, React, Vue, etc.). Voice agents, Python agents, and CLI tools do not have browser routes to test.
-
-**If HAS_WEB=true:** Continue to Step 2.
-
-### Known Non-Web Projects (skip without checking)
-
-These projects are known to not be web apps:
-- **sigatalachana** -- Python agent (invoice OCR + POS)
-- **armenius** -- Cloudflare Workers webhook (no frontend routes)
-- Any project in `~/Projects/aiagents/` with no `package.json` containing a web framework
-- Any project with only `.py` files and no web framework (Flask/Django with templates are web)
-
-If the current directory name matches a known non-web project, skip immediately.
-
-## Step 2: Detect Affected Routes
-
-Determine which routes to test based on git changes.
-
-### Automatic Route Detection
-
-```bash
-# Get changed files relative to the merge base
-BRANCH=$(git rev-parse --abbrev-ref HEAD)
-BASE=$(git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null || echo "HEAD~5")
-CHANGED_FILES=$(git diff --name-only "$BASE"...HEAD 2>/dev/null || git diff --name-only HEAD~5)
-
-echo "Changed files:"
-echo "$CHANGED_FILES"
-```
-
-### Route Mapping Rules
-
-Map changed files to browser routes:
-
-| File Pattern | Route |
-|-------------|-------|
-| `src/app/<path>/page.tsx` | `/<path>` |
-| `src/app/page.tsx` | `/` |
-| `pages/<path>.tsx` | `/<path>` |
-| `pages/index.tsx` | `/` |
-| `src/app/api/**` | Skip (API routes, not pages) |
-| `src/components/**` | Find which pages import this component |
-| `src/lib/**`, `src/utils/**` | Test `/` (shared code, test homepage) |
-| `public/**` | Test `/` (static assets) |
-| `styles/**`, `*.css` | Test all layout-level pages (`/`, plus any page using the stylesheet) |
-| `src/app/layout.tsx` | Test `/` and one subpage |
-
-### Component-to-Page Resolution
-
-When a component file changes, find which pages use it:
-
-```bash
-# For a changed component, find importing pages
-COMPONENT_NAME=$(basename "$CHANGED_FILE" .tsx)
-grep -rl "$COMPONENT_NAME" src/app/*/page.tsx pages/*.tsx 2>/dev/null
-```
-
-### Fallback
-
-If no routes can be derived (e.g., only config files changed), test the homepage (`/`).
-
-If `--all` flag is used, discover all routes:
-
-```bash
-# Next.js App Router
-find src/app -name "page.tsx" -o -name "page.jsx" | sed 's|src/app||;s|/page\.[tj]sx||;s|^$|/|'
-
-# Next.js Pages Router
-find pages -name "*.tsx" -o -name "*.jsx" | grep -v "_app\|_document\|api/" | sed 's|pages||;s|\.[tj]sx||;s|/index$|/|'
-```
-
-## Step 3: Run Playwright QA
-
-### Prerequisites
-
-Verify Playwright is available:
-
-```bash
-npx playwright --version
-```
-
-If not installed or browsers not downloaded:
-
-```bash
-npm init -y 2>/dev/null  # Only if no package.json
-npx playwright install chromium
-```
-
-### Dev Server Detection
-
-Check if a dev server is already running:
-
-```bash
-# Check common ports
-curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null
-curl -s -o /dev/null -w "%{http_code}" http://localhost:5173 2>/dev/null
-curl -s -o /dev/null -w "%{http_code}" http://localhost:4321 2>/dev/null
-```
-
-If no server is running, start one:
-
-```bash
-# Detect and start the dev server
-if grep -q '"dev"' package.json; then
-  npm run dev &
-  DEV_PID=$!
-  sleep 5  # Wait for server startup
-fi
-```
-
-Record the base URL (e.g., `http://localhost:3000`).
-
-### QA Script
-
-For each route, run this Playwright script:
-
-```bash
-npx playwright test --config=- <<'PLAYWRIGHT_CONFIG'
-import { test, expect } from '@playwright/test';
-
-const BASE_URL = process.env.QA_BASE_URL || 'http://localhost:3000';
-const ROUTES = process.env.QA_ROUTES?.split(',') || ['/'];
-const SCREENSHOT_DIR = '.qa-screenshots';
-
-test.describe('Browser QA', () => {
-  for (const route of ROUTES) {
-    test(`QA: ${route}`, async ({ page }) => {
-      const errors: string[] = [];
-      const failedRequests: { url: string; status: number }[] = [];
-
-      // Capture console errors
-      page.on('console', msg => {
-        if (msg.type() === 'error') {
-          errors.push(`Console error: ${msg.text()}`);
-        }
-      });
-
-      // Capture failed HTTP requests
-      page.on('response', response => {
-        if (response.status() >= 400) {
-          failedRequests.push({ url: response.url(), status: response.status() });
-        }
-      });
-
-      // Navigate
-      await page.goto(`${BASE_URL}${route}`, { waitUntil: 'networkidle', timeout: 30000 });
-
-      // Screenshot
-      await page.screenshot({ path: `${SCREENSHOT_DIR}${route === '/' ? '/index' : route}.png`, fullPage: true });
-
-      // Check for broken links
-      const links = await page.$$eval('a[href]', anchors =>
-        anchors.map(a => a.getAttribute('href')).filter(href => href && !href.startsWith('#') && !href.startsWith('mailto:') && !href.startsWith('tel:'))
-      );
-
-      for (const link of links.slice(0, 20)) {  // Cap at 20 links per page
-        if (link.startsWith('/') || link.startsWith(BASE_URL)) {
-          const fullUrl = link.startsWith('/') ? `${BASE_URL}${link}` : link;
-          const resp = await page.request.get(fullUrl);
-          if (resp.status() >= 400) {
-            failedRequests.push({ url: fullUrl, status: resp.status() });
-          }
-        }
-      }
-
-      // Report findings (test passes but logs findings)
-      if (errors.length > 0 || failedRequests.length > 0) {
-        console.log(JSON.stringify({ route, errors, failedRequests }));
-      }
-    });
-  }
-});
-PLAYWRIGHT_CONFIG
-```
-
-**Alternative approach (simpler, no config file needed):**
-
-Instead of the Playwright test runner, use a Node.js script with Playwright library directly:
-
-```bash
-QA_BASE_URL="http://localhost:3000" QA_ROUTES="/" node -e "
-const { chromium } = require('playwright');
-(async () => {
-  const browser = await chromium.launch();
-  const baseUrl = process.env.QA_BASE_URL;
-  const routes = process.env.QA_ROUTES.split(',');
-  const findings = [];
-
-  for (const route of routes) {
-    const context = await browser.newContext();
-    const page = await context.newPage();
-    const errors = [];
-    const failedRequests = [];
-
-    page.on('console', msg => { if (msg.type() === 'error') errors.push(msg.text()); });
-    page.on('response', resp => { if (resp.status() >= 400) failedRequests.push({ url: resp.url(), status: resp.status() }); });
-
-    await page.goto(baseUrl + route, { waitUntil: 'networkidle', timeout: 30000 });
-    await page.screenshot({ path: '.qa-screenshots' + (route === '/' ? '/index' : route) + '.png', fullPage: true });
-
-    // Check internal links
-    const links = await page.$$eval('a[href]', as => as.map(a => a.href).filter(h => h.startsWith(baseUrl) || h.startsWith('/')));
-    for (const link of links.slice(0, 20)) {
-      const url = link.startsWith('/') ? baseUrl + link : link;
-      try { const r = await page.request.get(url); if (r.status() >= 400) failedRequests.push({ url, status: r.status() }); } catch(e) { failedRequests.push({ url, status: 0 }); }
-    }
-
-    if (errors.length || failedRequests.length) findings.push({ route, errors, failedRequests });
-    await context.close();
-  }
-
-  console.log(JSON.stringify(findings, null, 2));
-  await browser.close();
-})();
-"
-```
-
-### Screenshot Capture
-
-Every page navigated gets a full-page screenshot saved to `.qa-screenshots/`:
-
-- Route `/` -> `.qa-screenshots/index.png`
-- Route `/about` -> `.qa-screenshots/about.png`
-- Route `/dashboard/settings` -> `.qa-screenshots/dashboard/settings.png`
-
-Create the directory before running:
-
-```bash
-mkdir -p .qa-screenshots
-```
-
-When a finding is reported, reference the screenshot path in the report:
-
-> Finding: Console error "TypeError: Cannot read property 'map' of undefined" on /dashboard
-> Screenshot: `.qa-screenshots/dashboard.png`
-
-## Step 4: Generate QA Report
-
-After all routes are tested, produce a structured report.
-
-### Report Format
-
-```markdown
-# QA Report -- {project-name}
-**Date:** {ISO timestamp}
-**Routes tested:** {N}
-**Findings:** {N} ({errors} console errors, {broken_links} broken links, {http_failures} HTTP failures)
-**Screenshots:** .qa-screenshots/
-
-## Findings
-
-### Finding 1: {type} on {route}
-- **Type:** console-error | broken-link | http-failure
-- **Route:** {route}
-- **Detail:** {error message or URL + status code}
-- **Screenshot:** .qa-screenshots/{route}.png
-- **False positive:** no (set to yes if confirmed false positive)
-- **Severity:** critical | warning | info
-
-### Finding 2: ...
-
-## Summary
-
-| Route | Status | Console Errors | Broken Links | HTTP Failures |
-|-------|--------|----------------|--------------|---------------|
-| / | PASS/FAIL | 0 | 0 | 0 |
-
-## Routes Tested
-{list of routes with pass/fail status}
-```
-
-### Severity Classification
-
-- **critical:** Page fails to load, unhandled exceptions, 500 errors
-- **warning:** 404 broken links, non-critical console errors, slow responses (>3s)
-- **info:** Console warnings (not errors), deprecation notices
-
-### Clean Run
-
-If zero findings:
-
-> QA PASSED: {N} routes tested, zero findings. Screenshots saved to .qa-screenshots/
-
-## Step 5: Cleanup
-
-```bash
-# Kill dev server if we started one
-[ -n "$DEV_PID" ] && kill $DEV_PID 2>/dev/null
-
-# Keep screenshots for evidence (don't delete)
-echo "Screenshots saved to .qa-screenshots/"
-```
-
-Add `.qa-screenshots/` to `.gitignore` if not already present:
-
-```bash
-grep -q ".qa-screenshots" .gitignore 2>/dev/null || echo ".qa-screenshots/" >> .gitignore
-```
-
-## Step 6: Bug Fix Workflow (BQA-03)
-
-When QA findings reveal real bugs, fix each one with an atomic commit and a regression test.
-
-### Process Per Finding
-
-For each confirmed finding (not a false positive):
-
-#### 1. Create a regression test FIRST (Red)
-
-Write a test that reproduces the exact issue found:
-
-```bash
-# Example: Console error on /dashboard
-cat > __tests__/qa-regression/dashboard-console-error.test.ts << 'EOF'
-import { test, expect } from '@playwright/test';
-
-test('dashboard should not have console errors', async ({ page }) => {
-  const errors: string[] = [];
-  page.on('console', msg => {
-    if (msg.type() === 'error') errors.push(msg.text());
-  });
-
-  await page.goto('http://localhost:3000/dashboard');
-  await page.waitForLoadState('networkidle');
-
-  expect(errors).toHaveLength(0);
-});
-EOF
-```
-
-```bash
-# Example: Broken link on /about
-cat > __tests__/qa-regression/about-broken-links.test.ts << 'EOF'
-import { test, expect } from '@playwright/test';
-
-test('about page should have no broken internal links', async ({ page }) => {
-  await page.goto('http://localhost:3000/about');
-  const links = await page.$$eval('a[href^="/"]', as => as.map(a => a.getAttribute('href')));
-
-  for (const link of links) {
-    const response = await page.request.get(`http://localhost:3000${link}`);
-    expect(response.status()).toBeLessThan(400);
-  }
-});
-EOF
-```
-
-#### 2. Fix the bug
-
-Apply the minimal fix. The fix should be scoped to the exact issue -- no drive-by refactoring.
-
-#### 3. Verify the test passes (Green)
-
-```bash
-npx playwright test __tests__/qa-regression/ --reporter=list
-```
-
-#### 4. Atomic commit
-
-Each bug fix gets its own commit:
-
-```bash
-git add <fixed-files> __tests__/qa-regression/<test-file>
-git commit -m "fix(qa): resolve {finding-type} on {route}
-
-QA finding: {brief description}
-Regression test: __tests__/qa-regression/{test-file}
-"
-```
-
-### Naming Convention for Regression Tests
-
-```
-__tests__/qa-regression/{route}-{finding-type}.test.ts
-```
-
-Examples:
-- `__tests__/qa-regression/dashboard-console-error.test.ts`
-- `__tests__/qa-regression/about-broken-links.test.ts`
-- `__tests__/qa-regression/index-http-failure.test.ts`
-
-### When NOT to Create Regression Tests
-
-- **False positives** -- skip entirely
-- **Third-party script errors** (e.g., analytics, chat widgets) -- suppress instead of fix
-- **Environment-only issues** (e.g., dev-only CORS) -- note in report, don't test
-
-## Step 7: False Positive Tracking (BQA-06)
-
-Track the false positive rate across QA runs. If the cumulative rate exceeds 20%, surface a tuning warning.
-
-### Recording Results
-
-After each QA run, append results to `.qa-history.json` in the project root:
-
-```bash
-# Read existing history or create empty
-HISTORY_FILE=".qa-history.json"
-[ -f "$HISTORY_FILE" ] || echo '{"runs":[]}' > "$HISTORY_FILE"
-
-# Append this run's results
-node -e "
-const fs = require('fs');
-const history = JSON.parse(fs.readFileSync('$HISTORY_FILE', 'utf8'));
-history.runs.push({
-  date: new Date().toISOString(),
-  routes_tested: ${ROUTES_TESTED},
-  total_findings: ${TOTAL_FINDINGS},
-  false_positives: ${FALSE_POSITIVES},
-  true_positives: ${TRUE_POSITIVES}
-});
-fs.writeFileSync('$HISTORY_FILE', JSON.stringify(history, null, 2));
-"
-```
-
-### Calculating the Rate
-
-After recording, calculate the cumulative false positive rate:
-
-```bash
-node -e "
-const fs = require('fs');
-const history = JSON.parse(fs.readFileSync('.qa-history.json', 'utf8'));
-const totals = history.runs.reduce((acc, run) => ({
-  findings: acc.findings + run.total_findings,
-  fp: acc.fp + run.false_positives
-}), { findings: 0, fp: 0 });
-
-const rate = totals.findings > 0 ? (totals.fp / totals.findings * 100).toFixed(1) : 0;
-console.log('False positive rate: ' + rate + '% (' + totals.fp + '/' + totals.findings + ')');
-
-if (parseFloat(rate) > 20) {
-  console.log('');
-  console.log('WARNING: False positive rate exceeds 20%. QA detection rules need tuning.');
-  console.log('Review common false positives:');
-  console.log('  - Third-party script errors (add to suppression list)');
-  console.log('  - Dev-only warnings (filter by environment)');
-  console.log('  - Known flaky routes (add to .qa-suppress.json)');
-}
-"
-```
-
-### Tuning Warning Output
-
-When rate > 20%, append this to the QA report:
-
-```markdown
-## Tuning Warning
-
-False positive rate is {rate}% ({fp}/{total} findings across {N} runs).
-This exceeds the 20% threshold. Consider:
-
-1. **Add suppressions** to `.qa-suppress.json` for known non-issues
-2. **Filter third-party errors** -- console errors from external scripts (analytics, chat widgets) are not your bugs
-3. **Exclude flaky routes** -- routes requiring auth or dynamic data may produce false 4xx errors
-4. **Review detection rules** -- some console warnings are not actionable errors
-```
-
-### Suppression File
-
-Create `.qa-suppress.json` to exclude known false positives from future runs:
-
-```json
-{
-  "suppress_console_patterns": [
-    ".*third-party-analytics.*",
-    ".*Failed to load resource.*favicon\\.ico.*",
-    ".*DevTools.*"
-  ],
-  "suppress_urls": [
-    "/api/auth/callback"
-  ],
-  "suppress_routes": []
-}
-```
-
-When running QA (Step 3), check suppressions before recording a finding:
-
-```javascript
-// In the Playwright script, before recording a console error:
-const suppressions = JSON.parse(fs.readFileSync('.qa-suppress.json', 'utf8'));
-const isSuppressed = suppressions.suppress_console_patterns.some(p => new RegExp(p).test(errorText));
-if (!isSuppressed) {
-  // Record finding
-}
-```
-
-### Marking False Positives
-
-In the QA report, each finding has a `False positive: no` field. After review:
-
-1. Change to `False positive: yes` for confirmed false positives
-2. Add the pattern to `.qa-suppress.json` to prevent recurrence
-3. Update `.qa-history.json` with corrected counts
-
-### History File in .gitignore
-
-```bash
-grep -q ".qa-history.json" .gitignore 2>/dev/null || echo ".qa-history.json" >> .gitignore
-grep -q ".qa-suppress.json" .gitignore 2>/dev/null || echo "# Keep .qa-suppress.json IN git -- it's project config, not output" >> /dev/null
-```
-
-Note: `.qa-suppress.json` SHOULD be committed (it's configuration). `.qa-history.json` should NOT be committed (it's local state).