@zeyue0329/xiaoma-cli 1.0.48 → 6.0.0-alpha.13

package/src/modules/xmc/testarch/knowledge/log.md

@@ -0,0 +1,294 @@
+# Log Utility
+
+## Principle
+
+Use structured logging that integrates with Playwright's test reports. Support object logging, test step decoration, and multiple log levels (info, step, success, warning, error, debug).
+
+## Rationale
+
+Console.log in Playwright tests has limitations:
+
+- Not visible in HTML reports
+- No test step integration
+- No structured output
+- Lost in terminal noise during CI
+
+The `log` utility provides:
+
+- **Report integration**: Logs appear in Playwright HTML reports
+- **Test step decoration**: `log.step()` creates collapsible steps in UI
+- **Object logging**: Automatically formats objects/arrays
+- **Multiple levels**: info, step, success, warning, error, debug
+- **Optional console**: Can disable console output but keep report logs
+
+## Pattern Examples
+
+### Example 1: Basic Logging Levels
+
+**Context**: Log different types of messages throughout test execution.
+
+**Implementation**:
+
+```typescript
+import { log } from '@seontechnologies/playwright-utils';
+
+test('logging demo', async ({ page }) => {
+  await log.step('Navigate to login page');
+  await page.goto('/login');
+
+  await log.info('Entering credentials');
+  await page.fill('#username', 'testuser');
+
+  await log.success('Login successful');
+
+  await log.warning('Rate limit approaching');
+
+  await log.debug({ userId: '123', sessionId: 'abc' });
+
+  // Errors still throw but get logged first
+  try {
+    await page.click('#nonexistent');
+  } catch (error) {
+    await log.error('Click failed', false); // false = no console output
+    throw error;
+  }
+});
+```
+
+**Key Points**:
+
+- `step()` creates collapsible steps in Playwright UI
+- `info()`, `success()`, `warning()` for different message types
+- `debug()` for detailed data (objects/arrays)
+- `error()` with optional console suppression
+- All logs appear in test reports
+
+### Example 2: Object and Array Logging
+
+**Context**: Log structured data for debugging without cluttering console.
+
+**Implementation**:
+
+```typescript
+test('object logging', async ({ apiRequest }) => {
+  const { body } = await apiRequest({
+    method: 'GET',
+    path: '/api/users',
+  });
+
+  // Log array of objects
+  await log.debug(body); // Formatted as JSON in report
+
+  // Log specific object
+  await log.info({
+    totalUsers: body.length,
+    firstUser: body[0]?.name,
+    timestamp: new Date().toISOString(),
+  });
+
+  // Complex nested structures
+  await log.debug({
+    request: {
+      method: 'GET',
+      path: '/api/users',
+      timestamp: Date.now(),
+    },
+    response: {
+      status: 200,
+      body: body.slice(0, 3), // First 3 items
+    },
+  });
+});
+```
+
+**Key Points**:
+
+- Objects auto-formatted as pretty JSON
+- Arrays handled gracefully
+- Nested structures supported
+- All visible in Playwright report attachments
+
+### Example 3: Test Step Organization
+
+**Context**: Organize test execution into collapsible steps for better readability in reports.
+
+**Implementation**:
+
+```typescript
+test('organized with steps', async ({ page, apiRequest }) => {
+  await log.step('ARRANGE: Setup test data');
+  const { body: user } = await apiRequest({
+    method: 'POST',
+    path: '/api/users',
+    body: { name: 'Test User' },
+  });
+
+  await log.step('ACT: Perform user action');
+  await page.goto(`/users/${user.id}`);
+  await page.click('#edit');
+  await page.fill('#name', 'Updated Name');
+  await page.click('#save');
+
+  await log.step('ASSERT: Verify changes');
+  await expect(page.getByText('Updated Name')).toBeVisible();
+
+  // In Playwright UI, each step is collapsible
+});
+```
+
+**Key Points**:
+
+- `log.step()` creates collapsible sections
+- Organize by Arrange-Act-Assert
+- Steps visible in Playwright trace viewer
+- Better debugging when tests fail
+
+### Example 4: Conditional Logging
+
+**Context**: Log different messages based on environment or test conditions.
+
+**Implementation**:
+
+```typescript
+test('conditional logging', async ({ page }) => {
+  const isCI = process.env.CI === 'true';
+
+  if (isCI) {
+    await log.info('Running in CI environment');
+  } else {
+    await log.debug('Running locally');
+  }
+
+  const isKafkaWorking = await checkKafkaHealth();
+
+  if (!isKafkaWorking) {
+    await log.warning('Kafka unavailable - skipping event checks');
+  } else {
+    await log.step('Verifying Kafka events');
+    // ... event verification
+  }
+});
+```
+
+**Key Points**:
+
+- Log based on environment
+- Skip logging with conditionals
+- Use appropriate log levels
+- Debug info for local, minimal for CI
+
+### Example 5: Integration with Auth and API
+
+**Context**: Log authenticated API requests with tokens (safely).
+
+**Implementation**:
+
+```typescript
+import { test } from '@seontechnologies/playwright-utils/fixtures';
+
+// Helper to create safe token preview
+function createTokenPreview(token: string): string {
+  if (!token || token.length < 10) return '[invalid]';
+  return `${token.slice(0, 6)}...${token.slice(-4)}`;
+}
+
+test('should log auth flow', async ({ authToken, apiRequest }) => {
+  await log.info(`Using token: ${createTokenPreview(authToken)}`);
+
+  await log.step('Fetch protected resource');
+  const { status, body } = await apiRequest({
+    method: 'GET',
+    path: '/api/protected',
+    headers: { Authorization: `Bearer ${authToken}` },
+  });
+
+  await log.debug({
+    status,
+    bodyPreview: {
+      id: body.id,
+      recordCount: body.data?.length,
+    },
+  });
+
+  await log.success('Protected resource accessed successfully');
+});
+```
+
+**Key Points**:
+
+- Never log full tokens (security risk)
+- Use preview functions for sensitive data
+- Combine with auth and API utilities
+- Log at appropriate detail level
+
+## Log Levels Guide
+
+| Level     | When to Use                         | Shows in Report      | Shows in Console |
+| --------- | ----------------------------------- | -------------------- | ---------------- |
+| `step`    | Test organization, major actions    | ✅ Collapsible steps | ✅ Yes           |
+| `info`    | General information, state changes  | ✅ Yes               | ✅ Yes           |
+| `success` | Successful operations               | ✅ Yes               | ✅ Yes           |
+| `warning` | Non-critical issues, skipped checks | ✅ Yes               | ✅ Yes           |
+| `error`   | Failures, exceptions                | ✅ Yes               | ✅ Configurable  |
+| `debug`   | Detailed data, objects              | ✅ Yes (attached)    | ✅ Configurable  |
+
+## Comparison with console.log
+
+| console.log             | log Utility               |
+| ----------------------- | ------------------------- |
+| Not in reports          | Appears in reports        |
+| No test steps           | Creates collapsible steps |
+| Manual JSON.stringify() | Auto-formats objects      |
+| No log levels           | 6 log levels              |
+| Lost in CI output       | Preserved in artifacts    |
+
+## Related Fragments
+
+- `overview.md` - Basic usage and imports
+- `api-request.md` - Log API requests
+- `auth-session.md` - Log auth flow (safely)
+- `recurse.md` - Log polling progress
+
+## Anti-Patterns
+
+**❌ Logging objects in steps:**
+
+```typescript
+await log.step({ user: 'test', action: 'create' }); // Shows empty in UI
+```
+
+**✅ Use strings for steps, objects for debug:**
+
+```typescript
+await log.step('Creating user: test'); // Readable in UI
+await log.debug({ user: 'test', action: 'create' }); // Detailed data
+```
+
+**❌ Logging sensitive data:**
+
+```typescript
+await log.info(`Password: ${password}`); // Security risk!
+await log.info(`Token: ${authToken}`); // Full token exposed!
+```
+
+**✅ Use previews or omit sensitive data:**
+
+```typescript
+await log.info('User authenticated successfully'); // No sensitive data
+await log.debug({ tokenPreview: token.slice(0, 6) + '...' });
+```
+
+**❌ Excessive logging in loops:**
+
+```typescript
+for (const item of items) {
+  await log.info(`Processing ${item.id}`); // 100 log entries!
+}
+```
+
+**✅ Log summary or use debug level:**
+
+```typescript
+await log.step(`Processing ${items.length} items`);
+await log.debug({ itemIds: items.map((i) => i.id) }); // One log entry
+```

package/src/modules/xmc/testarch/knowledge/network-error-monitor.md

@@ -0,0 +1,272 @@
+# Network Error Monitor
+
+## Principle
+
+Automatically detect and fail tests when HTTP 4xx/5xx errors occur during execution. Act like Sentry for tests - catch silent backend failures even when UI passes assertions.
+
+## Rationale
+
+Traditional Playwright tests focus on UI:
+
+- Backend 500 errors ignored if UI looks correct
+- Silent failures slip through
+- No visibility into background API health
+- Tests pass while features are broken
+
+The `network-error-monitor` provides:
+
+- **Automatic detection**: All HTTP 4xx/5xx responses tracked
+- **Test failures**: Fail tests with backend errors (even if UI passes)
+- **Structured artifacts**: JSON reports with error details
+- **Smart opt-out**: Disable for validation tests expecting errors
+- **Deduplication**: Group repeated errors by pattern
+- **Domino effect prevention**: Limit test failures per error pattern
+
+## Pattern Examples
+
+### Example 1: Basic Auto-Monitoring
+
+**Context**: Automatically fail tests when backend errors occur.
+
+**Implementation**:
+
+```typescript
+import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
+
+// Monitoring automatically enabled
+test('should load dashboard', async ({ page }) => {
+  await page.goto('/dashboard');
+  await expect(page.locator('h1')).toContainText('Dashboard');
+
+  // ✅ Passes if no HTTP errors
+  // ❌ Fails if any 4xx/5xx errors detected with clear message:
+  //    "Network errors detected: 2 request(s) failed"
+  //    Failed requests:
+  //      GET 500 https://api.example.com/users
+  //      POST 503 https://api.example.com/metrics
+});
+```
+
+**Key Points**:
+
+- Zero setup - auto-enabled for all tests
+- Fails on any 4xx/5xx response
+- Structured error message with URLs and status codes
+- JSON artifact attached to test report
+
+### Example 2: Opt-Out for Validation Tests
+
+**Context**: Some tests expect errors (validation, error handling, edge cases).
+
+**Implementation**:
+
+```typescript
+import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
+
+// Opt-out with annotation
+test('should show error on invalid input', { annotation: [{ type: 'skipNetworkMonitoring' }] }, async ({ page }) => {
+  await page.goto('/form');
+  await page.click('#submit'); // Triggers 400 error
+
+  // Monitoring disabled - test won't fail on 400
+  await expect(page.getByText('Invalid input')).toBeVisible();
+});
+
+// Or opt-out entire describe block
+test.describe('error handling', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => {
+  test('handles 404', async ({ page }) => {
+    // All tests in this block skip monitoring
+  });
+
+  test('handles 500', async ({ page }) => {
+    // Monitoring disabled
+  });
+});
+```
+
+**Key Points**:
+
+- Use annotation `{ type: 'skipNetworkMonitoring' }`
+- Can opt-out single test or entire describe block
+- Monitoring still active for other tests
+- Perfect for intentional error scenarios
+
+### Example 3: Integration with Merged Fixtures
+
+**Context**: Combine network-error-monitor with other utilities.
+
+**Implementation**:
+
+```typescript
+// playwright/support/merged-fixtures.ts
+import { mergeTests } from '@playwright/test';
+import { test as authFixture } from '@seontechnologies/playwright-utils/auth-session/fixtures';
+import { test as networkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
+
+export const test = mergeTests(
+  authFixture,
+  networkErrorMonitorFixture,
+  // Add other fixtures
+);
+
+// In tests
+import { test, expect } from '../support/merged-fixtures';
+
+test('authenticated with monitoring', async ({ page, authToken }) => {
+  // Both auth and network monitoring active
+  await page.goto('/protected');
+
+  // Fails if backend returns errors during auth flow
+});
+```
+
+**Key Points**:
+
+- Combine with `mergeTests`
+- Works alongside all other utilities
+- Monitoring active automatically
+- No extra setup needed
+
+### Example 4: Domino Effect Prevention
+
+**Context**: One failing endpoint shouldn't fail all tests.
+
+**Implementation**:
+
+```typescript
+// Configuration (internal to utility)
+const config = {
+  maxTestsPerError: 3, // Max 3 tests fail per unique error pattern
+};
+
+// Scenario:
+// Test 1: GET /api/broken → 500 error → Test fails ❌
+// Test 2: GET /api/broken → 500 error → Test fails ❌
+// Test 3: GET /api/broken → 500 error → Test fails ❌
+// Test 4: GET /api/broken → 500 error → Test passes ⚠️ (limit reached, warning logged)
+// Test 5: Different error pattern → Test fails ❌ (new pattern, counter resets)
+```
+
+**Key Points**:
+
+- Limits cascading failures
+- Groups errors by URL + status code pattern
+- Warns when limit reached
+- Prevents flaky backend from failing entire suite
+
+### Example 5: Artifact Structure
+
+**Context**: Debugging failed tests with network error artifacts.
+
+**Implementation**:
+
+When test fails due to network errors, artifact attached:
+
+```json
+// test-results/my-test/network-errors.json
+{
+  "errors": [
+    {
+      "url": "https://api.example.com/users",
+      "method": "GET",
+      "status": 500,
+      "statusText": "Internal Server Error",
+      "timestamp": "2024-08-13T10:30:45.123Z"
+    },
+    {
+      "url": "https://api.example.com/metrics",
+      "method": "POST",
+      "status": 503,
+      "statusText": "Service Unavailable",
+      "timestamp": "2024-08-13T10:30:46.456Z"
+    }
+  ],
+  "summary": {
+    "totalErrors": 2,
+    "uniquePatterns": 2
+  }
+}
+```
+
+**Key Points**:
+
+- JSON artifact per failed test
+- Full error details (URL, method, status, timestamp)
+- Summary statistics
+- Easy debugging with structured data
+
+## Comparison with Manual Error Checks
+
+| Manual Approach                                        | network-error-monitor      |
+| ------------------------------------------------------ | -------------------------- |
+| `page.on('response', resp => { if (!resp.ok()) ... })` | Auto-enabled, zero setup   |
+| Check each response manually                           | Automatic for all requests |
+| Custom error tracking logic                            | Built-in deduplication     |
+| No structured artifacts                                | JSON artifacts attached    |
+| Easy to forget                                         | Never miss a backend error |
+
+## When to Use
+
+**Auto-enabled for:**
+
+- ✅ All E2E tests
+- ✅ Integration tests
+- ✅ Any test hitting real APIs
+
+**Opt-out for:**
+
+- ❌ Validation tests (expecting 4xx)
+- ❌ Error handling tests (expecting 5xx)
+- ❌ Offline tests (network-recorder playback)
+
+## Integration with Framework Setup
+
+In `*framework` workflow, mention network-error-monitor:
+
+```typescript
+// Add to merged-fixtures.ts
+import { test as networkErrorMonitorFixture } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures';
+
+export const test = mergeTests(
+  // ... other fixtures
+  networkErrorMonitorFixture,
+);
+```
+
+## Related Fragments
+
+- `overview.md` - Installation and fixtures
+- `fixtures-composition.md` - Merging with other utilities
+- `error-handling.md` - Traditional error handling patterns
+
+## Anti-Patterns
+
+**❌ Opting out of monitoring globally:**
+
+```typescript
+// Every test skips monitoring
+test.use({ annotation: [{ type: 'skipNetworkMonitoring' }] });
+```
+
+**✅ Opt-out only for specific error tests:**
+
+```typescript
+test.describe('error scenarios', { annotation: [{ type: 'skipNetworkMonitoring' }] }, () => {
+  // Only these tests skip monitoring
+});
+```
+
+**❌ Ignoring network error artifacts:**
+
+```typescript
+// Test fails, artifact shows 500 errors
+// Developer: "Works on my machine" ¯\_(ツ)_/¯
+```
+
+**✅ Check artifacts for root cause:**
+
+```typescript
+// Read network-errors.json artifact
+// Identify failing endpoint: GET /api/users → 500
+// Fix backend issue before merging
+```