create-byan-agent 2.0.1 → 2.1.0

package/_bmad/tea/testarch/knowledge/test-healing-patterns.md

@@ -0,0 +1,644 @@
+# Test Healing Patterns
+
+## Principle
+
+Common test failures follow predictable patterns (stale selectors, race conditions, dynamic data assertions, network errors, hard waits). **Automated healing** identifies failure signatures and applies pattern-based fixes. Manual healing captures these patterns for future automation.
+
+## Rationale
+
+**The Problem**: Test failures waste developer time on repetitive debugging. Teams manually fix the same selector issues, timing bugs, and data mismatches repeatedly across test suites.
+
+**The Solution**: Catalog common failure patterns with diagnostic signatures and automated fixes. When a test fails, match the error message/stack trace against known patterns and apply the corresponding fix. This transforms test maintenance from reactive debugging to proactive pattern application.
+
+**Why This Matters**:
+
+- Reduces test maintenance time by 60-80% (pattern-based fixes vs manual debugging)
+- Prevents flakiness regression (same bug fixed once, applied everywhere)
+- Builds institutional knowledge (failure catalog grows over time)
+- Enables self-healing test suites (automate workflow validates and heals)
+
+## Pattern Examples
+
+### Example 1: Common Failure Pattern - Stale Selectors (Element Not Found)
+
+**Context**: Test fails with "Element not found" or "Locator resolved to 0 elements" errors
+
+**Diagnostic Signature**:
+
+```typescript
+// src/testing/healing/selector-healing.ts
+
+export type SelectorFailure = {
+  errorMessage: string;
+  stackTrace: string;
+  selector: string;
+  testFile: string;
+  lineNumber: number;
+};
+
+/**
+ * Detect stale selector failures
+ */
+export function isSelectorFailure(error: Error): boolean {
+  const patterns = [
+    /locator.*resolved to 0 elements/i,
+    /element not found/i,
+    /waiting for locator.*to be visible/i,
+    /selector.*did not match any elements/i,
+    /unable to find element/i,
+  ];
+
+  return patterns.some((pattern) => pattern.test(error.message));
+}
+
+/**
+ * Extract selector from error message
+ */
+export function extractSelector(errorMessage: string): string | null {
+  // Playwright: "locator('button[type=\"submit\"]') resolved to 0 elements"
+  const playwrightMatch = errorMessage.match(/locator\('([^']+)'\)/);
+  if (playwrightMatch) return playwrightMatch[1];
+
+  // Cypress: "Timed out retrying: Expected to find element: '.submit-button'"
+  const cypressMatch = errorMessage.match(/Expected to find element: ['"]([^'"]+)['"]/i);
+  if (cypressMatch) return cypressMatch[1];
+
+  return null;
+}
+
+/**
+ * Suggest better selector based on hierarchy
+ */
+export function suggestBetterSelector(badSelector: string): string {
+  // If using CSS class → suggest data-testid
+  if (badSelector.startsWith('.') || badSelector.includes('class=')) {
+    const elementName = badSelector.match(/class=["']([^"']+)["']/)?.[1] || badSelector.slice(1);
+    return `page.getByTestId('${elementName}') // Prefer data-testid over CSS class`;
+  }
+
+  // If using ID → suggest data-testid
+  if (badSelector.startsWith('#')) {
+    return `page.getByTestId('${badSelector.slice(1)}') // Prefer data-testid over ID`;
+  }
+
+  // If using nth() → suggest filter() or more specific selector
+  if (badSelector.includes('.nth(')) {
+    return `page.locator('${badSelector.split('.nth(')[0]}').filter({ hasText: 'specific text' }) // Avoid brittle nth(), use filter()`;
+  }
+
+  // If using complex CSS → suggest ARIA role
+  if (badSelector.includes('>') || badSelector.includes('+')) {
+    return `page.getByRole('button', { name: 'Submit' }) // Prefer ARIA roles over complex CSS`;
+  }
+
+  return `page.getByTestId('...') // Add data-testid attribute to element`;
+}
+```
+
+**Healing Implementation**:
+
+```typescript
+// tests/healing/selector-healing.spec.ts
+import { test, expect } from '@playwright/test';
+import { isSelectorFailure, extractSelector, suggestBetterSelector } from '../../src/testing/healing/selector-healing';
+
+test('heal stale selector failures automatically', async ({ page }) => {
+  await page.goto('/dashboard');
+
+  try {
+    // Original test with brittle CSS selector
+    await page.locator('.btn-primary').click();
+  } catch (error: any) {
+    if (isSelectorFailure(error)) {
+      const badSelector = extractSelector(error.message);
+      const suggestion = badSelector ? suggestBetterSelector(badSelector) : null;
+
+      console.log('HEALING SUGGESTION:', suggestion);
+
+      // Apply healed selector
+      await page.getByTestId('submit-button').click(); // Fixed!
+    } else {
+      throw error; // Not a selector issue, rethrow
+    }
+  }
+
+  await expect(page.getByText('Success')).toBeVisible();
+});
+```
+
+**Key Points**:
+
+- Diagnosis: Error message contains "locator resolved to 0 elements" or "element not found"
+- Fix: Replace brittle selector (CSS class, ID, nth) with robust alternative (data-testid, ARIA role)
+- Prevention: Follow selector hierarchy (data-testid > ARIA > text > CSS)
+- Automation: Pattern matching on error message + stack trace
+
+---
+
+### Example 2: Common Failure Pattern - Race Conditions (Timing Errors)
+
+**Context**: Test fails with "timeout waiting for element" or "element not visible" errors
+
+**Diagnostic Signature**:
+
+```typescript
+// src/testing/healing/timing-healing.ts
+
+export type TimingFailure = {
+  errorMessage: string;
+  testFile: string;
+  lineNumber: number;
+  actionType: 'click' | 'fill' | 'waitFor' | 'expect';
+};
+
+/**
+ * Detect race condition failures
+ */
+export function isTimingFailure(error: Error): boolean {
+  const patterns = [
+    /timeout.*waiting for/i,
+    /element is not visible/i,
+    /element is not attached to the dom/i,
+    /waiting for element to be visible.*exceeded/i,
+    /timed out retrying/i,
+    /waitForLoadState.*timeout/i,
+  ];
+
+  return patterns.some((pattern) => pattern.test(error.message));
+}
+
+/**
+ * Detect hard wait anti-pattern
+ */
+export function hasHardWait(testCode: string): boolean {
+  const hardWaitPatterns = [/page\.waitForTimeout\(/, /cy\.wait\(\d+\)/, /await.*sleep\(/, /setTimeout\(/];
+
+  return hardWaitPatterns.some((pattern) => pattern.test(testCode));
+}
+
+/**
+ * Suggest deterministic wait replacement
+ */
+export function suggestDeterministicWait(testCode: string): string {
+  if (testCode.includes('page.waitForTimeout')) {
+    return `
+// ❌ Bad: Hard wait (flaky)
+// await page.waitForTimeout(3000)
+
+// ✅ Good: Wait for network response
+await page.waitForResponse(resp => resp.url().includes('/api/data') && resp.status() === 200)
+
+// OR wait for element state
+await page.getByTestId('loading-spinner').waitFor({ state: 'detached' })
+    `.trim();
+  }
+
+  if (testCode.includes('cy.wait(') && /cy\.wait\(\d+\)/.test(testCode)) {
+    return `
+// ❌ Bad: Hard wait (flaky)
+// cy.wait(3000)
+
+// ✅ Good: Wait for aliased network request
+cy.intercept('GET', '/api/data').as('getData')
+cy.visit('/page')
+cy.wait('@getData')
+    `.trim();
+  }
+
+  return `
+// Add network-first interception BEFORE navigation:
+await page.route('**/api/**', route => route.continue())
+const responsePromise = page.waitForResponse('**/api/data')
+await page.goto('/page')
+await responsePromise
+  `.trim();
+}
+```
+
+**Healing Implementation**:
+
+```typescript
+// tests/healing/timing-healing.spec.ts
+import { test, expect } from '@playwright/test';
+import { isTimingFailure, hasHardWait, suggestDeterministicWait } from '../../src/testing/healing/timing-healing';
+
+test('heal race condition with network-first pattern', async ({ page, context }) => {
+  // Setup interception BEFORE navigation (prevent race)
+  await context.route('**/api/products', (route) => {
+    route.fulfill({
+      status: 200,
+      body: JSON.stringify({ products: [{ id: 1, name: 'Product A' }] }),
+    });
+  });
+
+  const responsePromise = page.waitForResponse('**/api/products');
+
+  await page.goto('/products');
+  await responsePromise; // Deterministic wait
+
+  // Element now reliably visible (no race condition)
+  await expect(page.getByText('Product A')).toBeVisible();
+});
+
+test('heal hard wait with event-based wait', async ({ page }) => {
+  await page.goto('/dashboard');
+
+  // ❌ Original (flaky): await page.waitForTimeout(3000)
+
+  // ✅ Healed: Wait for spinner to disappear
+  await page.getByTestId('loading-spinner').waitFor({ state: 'detached' });
+
+  // Element now reliably visible
+  await expect(page.getByText('Dashboard loaded')).toBeVisible();
+});
+```
+
+**Key Points**:
+
+- Diagnosis: Error contains "timeout" or "not visible", often after navigation
+- Fix: Replace hard waits with network-first pattern or element state waits
+- Prevention: ALWAYS intercept before navigate, use waitForResponse()
+- Automation: Detect `page.waitForTimeout()` or `cy.wait(number)` in test code
+
+---
+
+### Example 3: Common Failure Pattern - Dynamic Data Assertions (Non-Deterministic IDs)
+
+**Context**: Test fails with "Expected 'User 123' but received 'User 456'" or timestamp mismatches
+
+**Diagnostic Signature**:
+
+```typescript
+// src/testing/healing/data-healing.ts
+
+export type DataFailure = {
+  errorMessage: string;
+  expectedValue: string;
+  actualValue: string;
+  testFile: string;
+  lineNumber: number;
+};
+
+/**
+ * Detect dynamic data assertion failures
+ */
+export function isDynamicDataFailure(error: Error): boolean {
+  const patterns = [
+    /expected.*\d+.*received.*\d+/i, // ID mismatches
+    /expected.*\d{4}-\d{2}-\d{2}.*received/i, // Date mismatches
+    /expected.*user.*\d+/i, // Dynamic user IDs
+    /expected.*order.*\d+/i, // Dynamic order IDs
+    /expected.*to.*contain.*\d+/i, // Numeric assertions
+  ];
+
+  return patterns.some((pattern) => pattern.test(error.message));
+}
+
+/**
+ * Suggest flexible assertion pattern
+ */
+export function suggestFlexibleAssertion(errorMessage: string): string {
+  if (/expected.*user.*\d+/i.test(errorMessage)) {
+    return `
+// ❌ Bad: Hardcoded ID
+// await expect(page.getByText('User 123')).toBeVisible()
+
+// ✅ Good: Regex pattern for any user ID
+await expect(page.getByText(/User \\d+/)).toBeVisible()
+
+// OR use partial match
+await expect(page.locator('[data-testid="user-name"]')).toContainText('User')
+    `.trim();
+  }
+
+  if (/expected.*\d{4}-\d{2}-\d{2}/i.test(errorMessage)) {
+    return `
+// ❌ Bad: Hardcoded date
+// await expect(page.getByText('2024-01-15')).toBeVisible()
+
+// ✅ Good: Dynamic date validation
+const today = new Date().toISOString().split('T')[0]
+await expect(page.getByTestId('created-date')).toHaveText(today)
+
+// OR use date format regex
+await expect(page.getByTestId('created-date')).toHaveText(/\\d{4}-\\d{2}-\\d{2}/)
+    `.trim();
+  }
+
+  if (/expected.*order.*\d+/i.test(errorMessage)) {
+    return `
+// ❌ Bad: Hardcoded order ID
+// const orderId = '12345'
+
+// ✅ Good: Capture dynamic order ID
+const orderText = await page.getByTestId('order-id').textContent()
+const orderId = orderText?.match(/Order #(\\d+)/)?.[1]
+expect(orderId).toBeTruthy()
+
+// Use captured ID in later assertions
+await expect(page.getByText(\`Order #\${orderId} confirmed\`)).toBeVisible()
+    `.trim();
+  }
+
+  return `Use regex patterns, partial matching, or capture dynamic values instead of hardcoding`;
+}
+```
+
+**Healing Implementation**:
+
+```typescript
+// tests/healing/data-healing.spec.ts
+import { test, expect } from '@playwright/test';
+
+test('heal dynamic ID assertion with regex', async ({ page }) => {
+  await page.goto('/users');
+
+  // ❌ Original (fails with random IDs): await expect(page.getByText('User 123')).toBeVisible()
+
+  // ✅ Healed: Regex pattern matches any user ID
+  await expect(page.getByText(/User \d+/)).toBeVisible();
+});
+
+test('heal timestamp assertion with dynamic generation', async ({ page }) => {
+  await page.goto('/dashboard');
+
+  // ❌ Original (fails daily): await expect(page.getByText('2024-01-15')).toBeVisible()
+
+  // ✅ Healed: Generate expected date dynamically
+  const today = new Date().toISOString().split('T')[0];
+  await expect(page.getByTestId('last-updated')).toContainText(today);
+});
+
+test('heal order ID assertion with capture', async ({ page, request }) => {
+  // Create order via API (dynamic ID)
+  const response = await request.post('/api/orders', {
+    data: { productId: '123', quantity: 1 },
+  });
+  const { orderId } = await response.json();
+
+  // ✅ Healed: Use captured dynamic ID
+  await page.goto(`/orders/${orderId}`);
+  await expect(page.getByText(`Order #${orderId}`)).toBeVisible();
+});
+```
+
+**Key Points**:
+
+- Diagnosis: Error message shows expected vs actual value mismatch with IDs/timestamps
+- Fix: Use regex patterns (`/User \d+/`), partial matching, or capture dynamic values
+- Prevention: Never hardcode IDs, timestamps, or random data in assertions
+- Automation: Parse error message for expected/actual values, suggest regex patterns
+
+---
+
+### Example 4: Common Failure Pattern - Network Errors (Missing Route Interception)
+
+**Context**: Test fails with "API call failed" or "500 error" during test execution
+
+**Diagnostic Signature**:
+
+```typescript
+// src/testing/healing/network-healing.ts
+
+export type NetworkFailure = {
+  errorMessage: string;
+  url: string;
+  statusCode: number;
+  method: string;
+};
+
+/**
+ * Detect network failure
+ */
+export function isNetworkFailure(error: Error): boolean {
+  const patterns = [
+    /api.*call.*failed/i,
+    /request.*failed/i,
+    /network.*error/i,
+    /500.*internal server error/i,
+    /503.*service unavailable/i,
+    /fetch.*failed/i,
+  ];
+
+  return patterns.some((pattern) => pattern.test(error.message));
+}
+
+/**
+ * Suggest route interception
+ */
+export function suggestRouteInterception(url: string, method: string): string {
+  return `
+// ❌ Bad: Real API call (unreliable, slow, external dependency)
+
+// ✅ Good: Mock API response with route interception
+await page.route('${url}', route => {
+  route.fulfill({
+    status: 200,
+    contentType: 'application/json',
+    body: JSON.stringify({
+      // Mock response data
+      id: 1,
+      name: 'Test User',
+      email: 'test@example.com'
+    })
+  })
+})
+
+// Then perform action
+await page.goto('/page')
+  `.trim();
+}
+```
+
+**Healing Implementation**:
+
+```typescript
+// tests/healing/network-healing.spec.ts
+import { test, expect } from '@playwright/test';
+
+test('heal network failure with route mocking', async ({ page, context }) => {
+  // ✅ Healed: Mock API to prevent real network calls
+  await context.route('**/api/products', (route) => {
+    route.fulfill({
+      status: 200,
+      contentType: 'application/json',
+      body: JSON.stringify({
+        products: [
+          { id: 1, name: 'Product A', price: 29.99 },
+          { id: 2, name: 'Product B', price: 49.99 },
+        ],
+      }),
+    });
+  });
+
+  await page.goto('/products');
+
+  // Test now reliable (no external API dependency)
+  await expect(page.getByText('Product A')).toBeVisible();
+  await expect(page.getByText('$29.99')).toBeVisible();
+});
+
+test('heal 500 error with error state mocking', async ({ page, context }) => {
+  // Mock API failure scenario
+  await context.route('**/api/products', (route) => {
+    route.fulfill({ status: 500, body: JSON.stringify({ error: 'Internal Server Error' }) });
+  });
+
+  await page.goto('/products');
+
+  // Verify error handling (not crash)
+  await expect(page.getByText('Unable to load products')).toBeVisible();
+  await expect(page.getByRole('button', { name: 'Retry' })).toBeVisible();
+});
+```
+
+**Key Points**:
+
+- Diagnosis: Error message contains "API call failed", "500 error", or network-related failures
+- Fix: Add `page.route()` or `cy.intercept()` to mock API responses
+- Prevention: Mock ALL external dependencies (APIs, third-party services)
+- Automation: Extract URL from error message, generate route interception code
+
+---
+
+### Example 5: Common Failure Pattern - Hard Waits (Unreliable Timing)
+
+**Context**: Test fails intermittently with "timeout exceeded" or passes/fails randomly
+
+**Diagnostic Signature**:
+
+```typescript
+// src/testing/healing/hard-wait-healing.ts
+
+/**
+ * Detect hard wait anti-pattern in test code
+ */
+export function detectHardWaits(testCode: string): Array<{ line: number; code: string }> {
+  const lines = testCode.split('\n');
+  const violations: Array<{ line: number; code: string }> = [];
+
+  lines.forEach((line, index) => {
+    if (line.includes('page.waitForTimeout(') || /cy\.wait\(\d+\)/.test(line) || line.includes('sleep(') || line.includes('setTimeout(')) {
+      violations.push({ line: index + 1, code: line.trim() });
+    }
+  });
+
+  return violations;
+}
+
+/**
+ * Suggest event-based wait replacement
+ */
+export function suggestEventBasedWait(hardWaitLine: string): string {
+  if (hardWaitLine.includes('page.waitForTimeout')) {
+    return `
+// ❌ Bad: Hard wait (flaky)
+${hardWaitLine}
+
+// ✅ Good: Wait for network response
+await page.waitForResponse(resp => resp.url().includes('/api/') && resp.ok())
+
+// OR wait for element state change
+await page.getByTestId('loading-spinner').waitFor({ state: 'detached' })
+await page.getByTestId('content').waitFor({ state: 'visible' })
+    `.trim();
+  }
+
+  if (/cy\.wait\(\d+\)/.test(hardWaitLine)) {
+    return `
+// ❌ Bad: Hard wait (flaky)
+${hardWaitLine}
+
+// ✅ Good: Wait for aliased request
+cy.intercept('GET', '/api/data').as('getData')
+cy.visit('/page')
+cy.wait('@getData') // Deterministic
+    `.trim();
+  }
+
+  return 'Replace hard waits with event-based waits (waitForResponse, waitFor state changes)';
+}
+```
+
+**Healing Implementation**:
+
+```typescript
+// tests/healing/hard-wait-healing.spec.ts
+import { test, expect } from '@playwright/test';
+
+test('heal hard wait with deterministic wait', async ({ page }) => {
+  await page.goto('/dashboard');
+
+  // ❌ Original (flaky): await page.waitForTimeout(3000)
+
+  // ✅ Healed: Wait for loading spinner to disappear
+  await page.getByTestId('loading-spinner').waitFor({ state: 'detached' });
+
+  // OR wait for specific network response
+  await page.waitForResponse((resp) => resp.url().includes('/api/dashboard') && resp.ok());
+
+  await expect(page.getByText('Dashboard ready')).toBeVisible();
+});
+
+test('heal implicit wait with explicit network wait', async ({ page }) => {
+  const responsePromise = page.waitForResponse('**/api/products');
+
+  await page.goto('/products');
+
+  // ❌ Original (race condition): await page.getByText('Product A').click()
+
+  // ✅ Healed: Wait for network first
+  await responsePromise;
+  await page.getByText('Product A').click();
+
+  await expect(page).toHaveURL(/\/products\/\d+/);
+});
+```
+
+**Key Points**:
+
+- Diagnosis: Test code contains `page.waitForTimeout()` or `cy.wait(number)`
+- Fix: Replace with `waitForResponse()`, `waitFor({ state })`, or aliased intercepts
+- Prevention: NEVER use hard waits, always use event-based/response-based waits
+- Automation: Scan test code for hard wait patterns, suggest deterministic replacements
+
+---
+
+## Healing Pattern Catalog
+
+| Failure Type   | Diagnostic Signature                          | Healing Strategy                      | Prevention Pattern                        |
+| -------------- | --------------------------------------------- | ------------------------------------- | ----------------------------------------- |
+| Stale Selector | "locator resolved to 0 elements"              | Replace with data-testid or ARIA role | Selector hierarchy (testid > ARIA > text) |
+| Race Condition | "timeout waiting for element"                 | Add network-first interception        | Intercept before navigate                 |
+| Dynamic Data   | "Expected 'User 123' but got 'User 456'"      | Use regex or capture dynamic values   | Never hardcode IDs/timestamps             |
+| Network Error  | "API call failed", "500 error"                | Add route mocking                     | Mock all external dependencies            |
+| Hard Wait      | Test contains `waitForTimeout()` or `wait(n)` | Replace with event-based waits        | Always use deterministic waits            |
+
+## Healing Workflow
+
+1. **Run test** → Capture failure
+2. **Identify pattern** → Match error against diagnostic signatures
+3. **Apply fix** → Use pattern-based healing strategy
+4. **Re-run test** → Validate fix (max 3 iterations)
+5. **Mark unfixable** → Use `test.fixme()` if healing fails after 3 attempts
+
+## Healing Checklist
+
+Before enabling auto-healing in workflows:
+
+- [ ] **Failure catalog documented**: Common patterns identified (selectors, timing, data, network, hard waits)
+- [ ] **Diagnostic signatures defined**: Error message patterns for each failure type
+- [ ] **Healing strategies documented**: Fix patterns for each failure type
+- [ ] **Prevention patterns documented**: Best practices to avoid recurrence
+- [ ] **Healing iteration limit set**: Max 3 attempts before marking test.fixme()
+- [ ] **MCP integration optional**: Graceful degradation without Playwright MCP
+- [ ] **Pattern-based fallback**: Use knowledge base patterns when MCP unavailable
+- [ ] **Healing report generated**: Document what was healed and how
+
+## Integration Points
+
+- **Used in workflows**: `*automate` (auto-healing after test generation), `*atdd` (optional healing for acceptance tests)
+- **Related fragments**: `selector-resilience.md` (selector debugging), `timing-debugging.md` (race condition fixes), `network-first.md` (interception patterns), `data-factories.md` (dynamic data handling)
+- **Tools**: Error message parsing, AST analysis for code patterns, Playwright MCP (optional), pattern matching
+
+_Source: Playwright test-healer patterns, production test failure analysis, common anti-patterns from test-resources-for-ai_