ma-agents 2.19.2 → 2.20.0

package/lib/bmad-cache/tea/src/testarch/knowledge/burn-in.md

@@ -0,0 +1,273 @@
+# Burn-in Test Runner
+
+## Principle
+
+Use smart test selection with git diff analysis to run only affected tests. Filter out irrelevant changes (configs, types, docs) and control test volume with percentage-based execution. Reduce unnecessary CI runs while maintaining reliability.
+
+## Rationale
+
+Playwright's `--only-changed` triggers all affected tests:
+
+- Config file changes trigger hundreds of tests
+- Type definition changes cause full suite runs
+- No volume control (all or nothing)
+- Slow CI pipelines
+
+The `burn-in` utility provides:
+
+- **Smart filtering**: Skip patterns for irrelevant files (configs, types, docs)
+- **Volume control**: Run percentage of affected tests after filtering
+- **Custom dependency analysis**: More accurate than Playwright's built-in
+- **CI optimization**: Faster pipelines without sacrificing confidence
+- **Process of elimination**: Start with all → filter irrelevant → control volume
+
+## Pattern Examples
+
+### Example 1: Basic Burn-in Setup
+
+**Context**: Run burn-in on changed files compared to main branch.
+
+**Implementation**:
+
+```typescript
+// Step 1: Create burn-in script
+// playwright/scripts/burn-in-changed.ts
+import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'
+
+async function main() {
+  await runBurnIn({
+    configPath: 'playwright/config/.burn-in.config.ts',
+    baseBranch: 'main'
+  })
+}
+
+main().catch(console.error)
+
+// Step 2: Create config
+// playwright/config/.burn-in.config.ts
+import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'
+
+const config: BurnInConfig = {
+  // Files that never trigger tests (first filter)
+  skipBurnInPatterns: [
+    '**/config/**',
+    '**/*constants*',
+    '**/*types*',
+    '**/*.md',
+    '**/README*'
+  ],
+
+  // Run 30% of remaining tests after skip filter
+  burnInTestPercentage: 0.3,
+
+  // Burn-in repetition
+  burnIn: {
+    repeatEach: 3,  // Run each test 3 times
+    retries: 1      // Allow 1 retry
+  }
+}
+
+export default config
+
+// Step 3: Add package.json script
+{
+  "scripts": {
+    "test:pw:burn-in-changed": "tsx playwright/scripts/burn-in-changed.ts"
+  }
+}
+```
+
+**Key Points**:
+
+- Two-stage filtering: skip patterns, then volume control
+- `skipBurnInPatterns` eliminates irrelevant files
+- `burnInTestPercentage` controls test volume (0.3 = 30%)
+- Custom dependency analysis finds actually affected tests
+
+### Example 2: CI Integration
+
+**Context**: Use burn-in in GitHub Actions for efficient CI runs.
+
+**Implementation**:
+
+```yaml
+# .github/workflows/burn-in.yml
+name: Burn-in Changed Tests
+
+on:
+  pull_request:
+    branches: [main]
+
+jobs:
+  burn-in:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Need git history
+
+      - name: Setup Node
+        uses: actions/setup-node@v4
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Run burn-in on changed tests
+        run: npm run test:pw:burn-in-changed -- --base-branch=origin/main
+
+      - name: Upload artifacts
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: burn-in-failures
+          path: test-results/
+```
+
+**Key Points**:
+
+- `fetch-depth: 0` for full git history
+- Pass `--base-branch=origin/main` for PR comparison
+- Upload artifacts only on failure
+- Significantly faster than full suite
+
+### Example 3: How It Works (Process of Elimination)
+
+**Context**: Understanding the filtering pipeline.
+
+**Scenario:**
+
+```
+Git diff finds: 21 changed files
+├─ Step 1: Skip patterns filter
+│  Removed: 6 files (*.md, config/*, *types*)
+│  Remaining: 15 files
+│
+├─ Step 2: Dependency analysis
+│  Tests that import these 15 files: 45 tests
+│
+└─ Step 3: Volume control (30%)
+   Final tests to run: 14 tests (30% of 45)
+
+Result: Run 14 targeted tests instead of 147 with --only-changed!
+```
+
+**Key Points**:
+
+- Three-stage pipeline: skip → analyze → control
+- Custom dependency analysis (not just imports)
+- Percentage applies AFTER filtering
+- Dramatically reduces CI time
+
+### Example 4: Environment-Specific Configuration
+
+**Context**: Different settings for local vs CI environments.
+
+**Implementation**:
+
+```typescript
+import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';
+
+const config: BurnInConfig = {
+  skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md'],
+
+  // CI runs fewer iterations, local runs more
+  burnInTestPercentage: process.env.CI ? 0.2 : 0.3,
+
+  burnIn: {
+    repeatEach: process.env.CI ? 2 : 3,
+    retries: process.env.CI ? 0 : 1, // No retries in CI
+  },
+};
+
+export default config;
+```
+
+**Key Points**:
+
+- `process.env.CI` for environment detection
+- Lower percentage in CI (20% vs 30%)
+- Fewer iterations in CI (2 vs 3)
+- No retries in CI (fail fast)
+
+### Example 5: Sharding Support
+
+**Context**: Distribute burn-in tests across multiple CI workers.
+
+**Implementation**:
+
+```typescript
+// burn-in-changed.ts with sharding
+import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';
+
+async function main() {
+  const shardArg = process.argv.find((arg) => arg.startsWith('--shard='));
+
+  if (shardArg) {
+    process.env.PW_SHARD = shardArg.split('=')[1];
+  }
+
+  await runBurnIn({
+    configPath: 'playwright/config/.burn-in.config.ts',
+  });
+}
+```
+
+```yaml
+# GitHub Actions with sharding
+jobs:
+  burn-in:
+    strategy:
+      matrix:
+        shard: [1/3, 2/3, 3/3]
+    steps:
+      - run: npm run test:pw:burn-in-changed -- --shard=${{ matrix.shard }}
+```
+
+**Key Points**:
+
+- Pass `--shard=1/3` for parallel execution
+- Burn-in respects Playwright sharding
+- Distribute across multiple workers
+- Reduces total CI time further
+
+## Integration with CI Workflow
+
+When setting up CI with `*ci` workflow, recommend burn-in for:
+
+- Pull request validation
+- Pre-merge checks
+- Nightly builds (subset runs)
+
+## Related Fragments
+
+- `ci-burn-in.md` - Traditional burn-in patterns (10-iteration loops)
+- `selective-testing.md` - Test selection strategies
+- `overview.md` - Installation
+
+## Anti-Patterns
+
+**❌ Over-aggressive skip patterns:**
+
+```typescript
+skipBurnInPatterns: [
+  '**/*', // Skips everything!
+];
+```
+
+**✅ Targeted skip patterns:**
+
+```typescript
+skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md', '**/*constants*'];
+```
+
+**❌ Too low percentage (false confidence):**
+
+```typescript
+burnInTestPercentage: 0.05; // Only 5% - might miss issues
+```
+
+**✅ Balanced percentage:**
+
+```typescript
+burnInTestPercentage: 0.2; // 20% in CI, provides good coverage
+```