bmad-studio 0.1.0 → 0.2.0

package/_bmad/tea/testarch/knowledge/network-error-monitor.md

@@ -1,401 +0,0 @@
-# Network Error Monitor
-
-## Principle
-
-Automatically detect and fail tests when HTTP 4xx/5xx errors occur during execution. Act like Sentry for tests - catch silent backend failures even when UI passes assertions.
-
-## Rationale
-
-Traditional Playwright tests focus on UI:
-
-- Backend 500 errors ignored if UI looks correct
-- Silent failures slip through
-- No visibility into background API health
-- Tests pass while features are broken
-
-The `network-error-monitor` provides:
-
-- **Automatic detection**: All HTTP 4xx/5xx responses tracked
-- **Test failures**: Fail tests with backend errors (even if UI passes)
-- **Structured artifacts**: JSON reports with error details
-- **Smart opt-out**: Disable for validation tests expecting errors
-- **Deduplication**: Group repeated errors by pattern
-- **Domino effect prevention**: Limit test failures per error pattern
-- **Respects test status**: Won't suppress actual test failures
-
-## Quick Start
-
-```typescript
-import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-// That's it! Network monitoring is automatically enabled
-test('my test', async ({ page }) => {
-  await page.goto('/dashboard');
-  // If any HTTP 4xx/5xx errors occur, the test will fail
-});
-```
-
-## Pattern Examples
-
-### Example 1: Basic Auto-Monitoring
-
-**Context**: Automatically fail tests when backend errors occur.
-
-**Implementation**:
-
-```typescript
-import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-// Monitoring automatically enabled
-test('should load dashboard', async ({ page }) => {
-  await page.goto('/dashboard');
-  await expect(page.locator('h1')).toContainText('Dashboard');
-
-  // Passes if no HTTP errors
-  // Fails if any 4xx/5xx errors detected with clear message:
-  //    "Network errors detected: 2 request(s) failed"
-  //    Failed requests:
-  //      GET 500 https://api.example.com/users
-  //      POST 503 https://api.example.com/metrics
-});
-```
-
-**Key Points**:
-
-- Zero setup - auto-enabled for all tests
-- Fails on any 4xx/5xx response
-- Structured error message with URLs and status codes
-- JSON artifact attached to test report
-
-### Example 2: Opt-Out for Validation Tests
-
-**Context**: Some tests expect errors (validation, error handling, edge cases).
-
-**Implementation**:
-
-```typescript
-import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-// Opt-out with annotation
-test('should show error on invalid input', { annotation: [{ type: 'skipNetworkMonitoring' }] }, async ({ page }) => {
-  await page.goto('/form');
-  await page.click('#submit'); // Triggers 400 error
-
-  // Monitoring disabled - test won't fail on 400
-  await expect(page.getByText('Invalid input')).toBeVisible();
-});
-
-// Or opt-out entire describe block
-test.describe('error handling', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => {
-  test('handles 404', async ({ page }) => {
-    // All tests in this block skip monitoring
-  });
-
-  test('handles 500', async ({ page }) => {
-    // Monitoring disabled
-  });
-});
-```
-
-**Key Points**:
-
-- Use annotation `{ type: 'skipNetworkMonitoring' }`
-- Can opt-out single test or entire describe block
-- Monitoring still active for other tests
-- Perfect for intentional error scenarios
-
-### Example 3: Respects Test Status
-
-**Context**: The monitor respects final test statuses to avoid suppressing important test outcomes.
-
-**Behavior by test status:**
-
-- **`failed`**: Network errors logged as additional context, not thrown
-- **`timedOut`**: Network errors logged as additional context
-- **`skipped`**: Network errors logged, skip status preserved
-- **`interrupted`**: Network errors logged, interrupted status preserved
-- **`passed`**: Network errors throw and fail the test
-
-**Example with test.skip():**
-
-```typescript
-test('feature gated test', async ({ page }) => {
-  const featureEnabled = await checkFeatureFlag();
-  test.skip(!featureEnabled, 'Feature not enabled');
-  // If skipped, network errors won't turn this into a failure
-  await page.goto('/new-feature');
-});
-```
-
-### Example 4: Excluding Legitimate Errors
-
-**Context**: Some endpoints legitimately return 4xx/5xx responses.
-
-**Implementation**:
-
-```typescript
-import { test as base } from '@playwright/test';
-import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-export const test = base.extend(
-  createNetworkErrorMonitorFixture({
-    excludePatterns: [
-      /email-cluster\/ml-app\/has-active-run/, // ML service returns 404 when no active run
-      /idv\/session-templates\/list/, // IDV service returns 404 when not configured
-      /sentry\.io\/api/, // External Sentry errors should not fail tests
-    ],
-  }),
-);
-```
-
-**For merged fixtures:**
-
-```typescript
-import { test as base, mergeTests } from '@playwright/test';
-import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-const networkErrorMonitor = base.extend(
-  createNetworkErrorMonitorFixture({
-    excludePatterns: [/analytics\.google\.com/, /cdn\.example\.com/],
-  }),
-);
-
-export const test = mergeTests(authFixture, networkErrorMonitor);
-```
-
-### Example 5: Preventing Domino Effect
-
-**Context**: One failing endpoint shouldn't fail all tests.
-
-**Implementation**:
-
-```typescript
-import { test as base } from '@playwright/test';
-import { createNetworkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-const networkErrorMonitor = base.extend(
-  createNetworkErrorMonitorFixture({
-    excludePatterns: [], // Required when using maxTestsPerError
-    maxTestsPerError: 1, // Only first test fails per error pattern, rest just log
-  }),
-);
-```
-
-**How it works:**
-
-When `/api/v2/case-management/cases` returns 500:
-
-- **First test** encountering this error: **FAILS** with clear error message
-- **Subsequent tests** encountering same error: **PASSES** but logs warning
-
-Error patterns are grouped by `method + status + base path`:
-
-- `GET /api/v2/case-management/cases/123` -> Pattern: `GET:500:/api/v2/case-management`
-- `GET /api/v2/case-management/quota` -> Pattern: `GET:500:/api/v2/case-management` (same group!)
-- `POST /api/v2/case-management/cases` -> Pattern: `POST:500:/api/v2/case-management` (different group!)
-
-**Why include HTTP method?** A GET 404 vs POST 404 might represent different issues:
-
-- `GET 404 /api/users/123` -> User not found (expected in some tests)
-- `POST 404 /api/users` -> Endpoint doesn't exist (critical error)
-
-**Output for subsequent tests:**
-
-```
-Warning: Network errors detected but not failing test (maxTestsPerError limit reached):
-  GET 500 https://api.example.com/api/v2/case-management/cases
-```
-
-**Recommended configuration:**
-
-```typescript
-createNetworkErrorMonitorFixture({
-  excludePatterns: [...], // Required - known broken endpoints (can be empty [])
-  maxTestsPerError: 1     // Stop domino effect (requires excludePatterns)
-})
-```
-
-**Understanding worker-level state:**
-
-Error pattern counts are stored in worker-level global state:
-
-```typescript
-// test-file-1.spec.ts (runs in Worker 1)
-test('test A', () => {
-  /* triggers GET:500:/api/v2/cases */
-}); // FAILS
-
-// test-file-2.spec.ts (runs later in Worker 1)
-test('test B', () => {
-  /* triggers GET:500:/api/v2/cases */
-}); // PASSES (limit reached)
-
-// test-file-3.spec.ts (runs in Worker 2 - different worker)
-test('test C', () => {
-  /* triggers GET:500:/api/v2/cases */
-}); // FAILS (fresh worker)
-```
-
-### Example 6: Integration with Merged Fixtures
-
-**Context**: Combine network-error-monitor with other utilities.
-
-**Implementation**:
-
-```typescript
-// playwright/support/merged-fixtures.ts
-import { mergeTests } from '@playwright/test';
-import { test as authFixture } from '@seontechnologies/playwright-utils/auth-session/fixtures';
-import { test as networkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-export const test = mergeTests(
-  authFixture,
-  networkErrorMonitorFixture,
-  // Add other fixtures
-);
-
-// In tests
-import { test, expect } from '../support/merged-fixtures';
-
-test('authenticated with monitoring', async ({ page, authToken }) => {
-  // Both auth and network monitoring active
-  await page.goto('/protected');
-
-  // Fails if backend returns errors during auth flow
-});
-```
-
-**Key Points**:
-
-- Combine with `mergeTests`
-- Works alongside all other utilities
-- Monitoring active automatically
-- No extra setup needed
-
-### Example 7: Artifact Structure
-
-**Context**: Debugging failed tests with network error artifacts.
-
-When test fails due to network errors, artifact attached:
-
-```json
-[
-  {
-    "url": "https://api.example.com/users",
-    "status": 500,
-    "method": "GET",
-    "timestamp": "2025-11-10T12:34:56.789Z"
-  },
-  {
-    "url": "https://api.example.com/metrics",
-    "status": 503,
-    "method": "POST",
-    "timestamp": "2025-11-10T12:34:57.123Z"
-  }
-]
-```
-
-## Implementation Details
-
-### How It Works
-
-1. **Fixture Extension**: Uses Playwright's `base.extend()` with `auto: true`
-2. **Response Listener**: Attaches `page.on('response')` listener at test start
-3. **Multi-Page Monitoring**: Automatically monitors popups and new tabs via `context.on('page')`
-4. **Error Collection**: Captures 4xx/5xx responses, checking exclusion patterns
-5. **Try/Finally**: Ensures error processing runs even if test fails early
-6. **Status Check**: Only throws errors if test hasn't already reached final status
-7. **Artifact**: Attaches JSON file to test report for debugging
-
-### Performance
-
-The monitor has minimal performance impact:
-
-- Event listener overhead: ~0.1ms per response
-- Memory: ~200 bytes per unique error
-- No network delay (observes responses, doesn't intercept them)
-
-## Comparison with Alternatives
-
-| Approach                    | Network Error Monitor | Manual afterEach      |
-| --------------------------- | --------------------- | --------------------- |
-| **Setup Required**          | Zero (auto-enabled)   | Every test file       |
-| **Catches Silent Failures** | Yes                   | Yes (if configured)   |
-| **Structured Artifacts**    | JSON attached         | Custom impl           |
-| **Test Failure Safety**     | Try/finally           | afterEach may not run |
-| **Opt-Out Mechanism**       | Annotation            | Custom logic          |
-| **Status Aware**            | Respects skip/failed  | No                    |
-
-## When to Use
-
-**Auto-enabled for:**
-
-- All E2E tests
-- Integration tests
-- Any test hitting real APIs
-
-**Opt-out for:**
-
-- Validation tests (expecting 4xx)
-- Error handling tests (expecting 5xx)
-- Offline tests (network-recorder playback)
-
-## Troubleshooting
-
-### Test fails with network errors but I don't see them in my app
-
-The errors might be happening during page load or in background polling. Check the `network-errors.json` artifact in your test report for full details including timestamps.
-
-### False positives from external services
-
-Configure exclusion patterns as shown in the "Excluding Legitimate Errors" section above.
-
-### Network errors not being caught
-
-Ensure you're importing the test from the correct fixture:
-
-```typescript
-// Correct
-import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
-
-// Wrong - this won't have network monitoring
-import { test } from '@playwright/test';
-```
-
-## Related Fragments
-
-- `overview.md` - Installation and fixtures
-- `fixtures-composition.md` - Merging with other utilities
-- `error-handling.md` - Traditional error handling patterns
-
-## Anti-Patterns
-
-**DON'T opt out of monitoring globally:**
-
-```typescript
-// Every test skips monitoring
-test.use({ annotation: [{ type: 'skipNetworkMonitoring' }] });
-```
-
-**DO opt-out only for specific error tests:**
-
-```typescript
-test.describe('error scenarios', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => {
-  // Only these tests skip monitoring
-});
-```
-
-**DON'T ignore network error artifacts:**
-
-```typescript
-// Test fails, artifact shows 500 errors
-// Developer: "Works on my machine" ¯\_(ツ)_/¯
-```
-
-**DO check artifacts for root cause:**
-
-```typescript
-// Read network-errors.json artifact
-// Identify failing endpoint: GET /api/users -> 500
-// Fix backend issue before merging
-```