ma-agents 2.19.2 → 2.20.0

package/lib/bmad-cache/tea/docs/how-to/workflows/setup-ci.md

@@ -0,0 +1,763 @@
+---
+title: 'How to Set Up CI Pipeline with TEA'
+description: Configure automated test execution with selective testing and burn-in loops using TEA
+---
+
+# How to Set Up CI Pipeline with TEA
+
+Use TEA's `ci` workflow to scaffold production-ready CI/CD configuration for automated test execution with selective testing, parallel sharding, and flakiness detection.
+
+## When to Use This
+
+- Need to automate test execution in CI/CD
+- Want selective testing (only run affected tests)
+- Need parallel execution for faster feedback
+- Want burn-in loops for flakiness detection
+- Setting up new CI/CD pipeline
+- Optimizing existing CI/CD workflow
+
+## Prerequisites
+
+- BMad Method installed
+- TEA agent available
+- Test framework configured (run `framework` first)
+- Tests written (have something to run in CI)
+- CI/CD platform access (GitHub Actions, GitLab CI, etc.)
+
+## Steps
+
+### 1. Load TEA Agent
+
+Start a fresh chat and load TEA:
+
+```
+tea
+```
+
+### 2. Run the CI Workflow
+
+```
+ci
+```
+
+### 3. Select CI/CD Platform
+
+TEA will ask which platform you're using.
+
+**Supported Platforms:**
+
+- **GitHub Actions** (most common)
+- **GitLab CI**
+- **Jenkins** — Generates `Jenkinsfile` with parallel stages, artifact archiving, and post-failure handling
+- **Azure DevOps** — Generates `azure-pipelines.yml` with matrix strategy for sharding and Azure-specific caching
+- **Harness** — Generates `.harness/pipeline.yaml` with Kubernetes-based execution and parallel steps
+- **Circle CI**
+- **Other** (TEA provides generic template)
+
+**Example:**
+
+```
+GitHub Actions
+```
+
+### 4. Configure Test Strategy
+
+TEA will ask about your test execution strategy.
+
+#### Repository Structure
+
+**Question:** "What's your repository structure?"
+
+**Options:**
+
+- **Single app** - One application in root
+- **Monorepo** - Multiple apps/packages
+- **Monorepo with affected detection** - Only test changed packages
+
+**Example:**
+
+```
+Monorepo with multiple apps
+Need selective testing for changed packages only
+```
+
+#### Parallel Execution
+
+**Question:** "Want to shard tests for parallel execution?"
+
+**Options:**
+
+- **No sharding** - Run tests sequentially
+- **Shard by workers** - Split across N workers
+- **Shard by file** - Each file runs in parallel
+
+**Example:**
+
+```
+Yes, shard across 4 workers for faster execution
+```
+
+**Why Shard?**
+
+- **4 workers:** 20-minute suite → 5 minutes
+- **Better resource usage:** Utilize CI runners efficiently
+- **Faster feedback:** Developers wait less
+
+#### Burn-In Loops
+
+**Question:** "Want burn-in loops for flakiness detection?"
+
+**Options:**
+
+- **No burn-in** - Run tests once
+- **PR burn-in** - Run tests multiple times on PRs
+- **Nightly burn-in** - Dedicated flakiness detection job
+
+**Example:**
+
+```
+Yes, run tests 5 times on PRs to catch flaky tests early
+```
+
+**Why Burn-In?**
+
+- Catches flaky tests before they merge
+- Prevents intermittent CI failures
+- Builds confidence in test suite
+
+### 5. Review Generated CI Configuration
+
+TEA generates platform-specific workflow files.
+
+#### GitHub Actions (`.github/workflows/test.yml`):
+
+```yaml
+name: Test Suite
+
+on:
+  pull_request:
+  push:
+    branches: [main, develop]
+  schedule:
+    - cron: '0 2 * * *' # Nightly at 2 AM
+
+jobs:
+  # Main test job with sharding
+  test:
+    name: Test (Shard ${{ matrix.shard }})
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: [1, 2, 3, 4]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps
+
+      - name: Run tests
+        run: npx playwright test --shard=${{ matrix.shard }}/4
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: test-results-${{ matrix.shard }}
+          path: test-results/
+          retention-days: 7
+
+      - name: Upload test report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: playwright-report-${{ matrix.shard }}
+          path: playwright-report/
+          retention-days: 7
+
+  # Burn-in job for flakiness detection (PRs only)
+  burn-in:
+    name: Burn-In (Flakiness Detection)
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+    timeout-minutes: 30
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps
+
+      - name: Run burn-in loop
+        run: |
+          for i in {1..5}; do
+            echo "=== Burn-in iteration $i/5 ==="
+            npx playwright test --grep-invert "@skip" || exit 1
+          done
+
+      - name: Upload burn-in results
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: burn-in-failures
+          path: test-results/
+
+  # Selective testing (changed files only)
+  selective:
+    name: Selective Tests
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Full history for git diff
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version-file: '.nvmrc'
+          cache: 'npm'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Install Playwright browsers
+        run: npx playwright install --with-deps
+
+      - name: Run selective tests
+        run: npm run test:changed
+```
+
+#### GitLab CI (`.gitlab-ci.yml`):
+
+```yaml
+variables:
+  NODE_VERSION: '18'
+
+stages:
+  - test
+  - burn-in
+
+# Test job with parallel execution
+test:
+  stage: test
+  image: node:$NODE_VERSION
+  parallel: 4
+  script:
+    - npm ci
+    - npx playwright install --with-deps
+    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
+  artifacts:
+    when: always
+    paths:
+      - test-results/
+      - playwright-report/
+    expire_in: 7 days
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
+    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+
+# Burn-in job for flakiness detection
+burn-in:
+  stage: burn-in
+  image: node:$NODE_VERSION
+  script:
+    - npm ci
+    - npx playwright install --with-deps
+    - |
+      for i in {1..5}; do
+        echo "=== Burn-in iteration $i/5 ==="
+        npx playwright test || exit 1
+      done
+  artifacts:
+    when: on_failure
+    paths:
+      - test-results/
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
+```
+
+#### Burn-In Testing
+
+**Option 1: Classic Burn-In (Playwright Built-In)**
+
+```json
+{
+  "scripts": {
+    "test": "playwright test",
+    "test:burn-in": "playwright test --repeat-each=5 --retries=0"
+  }
+}
+```
+
+**How it works:**
+
+- Runs every test 5 times
+- Fails if any iteration fails
+- Detects flakiness before merge
+
+**Use when:** Small test suite, want to run everything multiple times
+
+---
+
+**Option 2: Smart Burn-In (Playwright Utils)**
+
+If `tea_use_playwright_utils: true`:
+
+**scripts/burn-in-changed.ts:**
+
+```typescript
+import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';
+
+await runBurnIn({
+  configPath: 'playwright.burn-in.config.ts',
+  baseBranch: 'main',
+});
+```
+
+**playwright.burn-in.config.ts:**
+
+```typescript
+import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';
+
+const config: BurnInConfig = {
+  skipBurnInPatterns: ['**/config/**', '**/*.md', '**/*types*'],
+  burnInTestPercentage: 0.3,
+  burnIn: { repeatEach: 5, retries: 0 },
+};
+
+export default config;
+```
+
+**package.json:**
+
+```json
+{
+  "scripts": {
+    "test:burn-in": "tsx scripts/burn-in-changed.ts"
+  }
+}
+```
+
+**How it works:**
+
+- Git diff analysis (only affected tests)
+- Smart filtering (skip configs, docs, types)
+- Volume control (run 30% of affected tests)
+- Each test runs 5 times
+
+**Use when:** Large test suite, want intelligent selection
+
+---
+
+**Comparison:**
+
+| Feature        | Classic Burn-In                    | Smart Burn-In (PW-Utils)            |
+| -------------- | ---------------------------------- | ----------------------------------- |
+| Changed 1 file | Runs all 500 tests × 5 = 2500 runs | Runs 3 affected tests × 5 = 15 runs |
+| Config change  | Runs all tests                     | Skips (no tests affected)           |
+| Type change    | Runs all tests                     | Skips (no runtime impact)           |
+| Setup          | Zero config                        | Requires config file                |
+
+**Recommendation:** Start with classic (simple), upgrade to smart (faster) when suite grows.
+
+### 6. Configure Secrets
+
+TEA provides a secrets checklist.
+
+**Required Secrets** (add to CI/CD platform):
+
+```markdown
+## GitHub Actions Secrets
+
+Repository Settings → Secrets and variables → Actions
+
+### Required
+
+- None (tests run without external auth)
+
+### Optional
+
+- `TEST_USER_EMAIL` - Test user credentials
+- `TEST_USER_PASSWORD` - Test user password
+- `API_BASE_URL` - API endpoint for tests
+- `DATABASE_URL` - Test database (if needed)
+```
+
+**How to Add Secrets:**
+
+**GitHub Actions:**
+
+1. Go to repo Settings → Secrets → Actions
+2. Click "New repository secret"
+3. Add name and value
+4. Use in workflow: `${{ secrets.TEST_USER_EMAIL }}`
+
+**GitLab CI:**
+
+1. Go to Project Settings → CI/CD → Variables
+2. Add variable name and value
+3. Use in workflow: `$TEST_USER_EMAIL`
+
+### 7. Test the CI Pipeline
+
+#### Push and Verify
+
+**Commit the workflow file:**
+
+```bash
+git add .github/workflows/test.yml
+git commit -m "ci: add automated test pipeline"
+git push
+```
+
+**Watch the CI run:**
+
+- GitHub Actions: Go to Actions tab
+- GitLab CI: Go to CI/CD → Pipelines
+- Circle CI: Go to Pipelines
+
+**Expected Result:**
+
+```
+✓ test (shard 1/4) - 3m 24s
+✓ test (shard 2/4) - 3m 18s
+✓ test (shard 3/4) - 3m 31s
+✓ test (shard 4/4) - 3m 15s
+✓ burn-in - 15m 42s
+```
+
+#### Test on Pull Request
+
+**Create test PR:**
+
+```bash
+git checkout -b test-ci-setup
+echo "# Test" > test.md
+git add test.md
+git commit -m "test: verify CI setup"
+git push -u origin test-ci-setup
+```
+
+**Open PR and verify:**
+
+- Tests run automatically
+- Burn-in runs (if configured for PRs)
+- Selective tests run (if applicable)
+- All checks pass ✓
+
+## What You Get
+
+### Automated Test Execution
+
+- **On every PR** - Catch issues before merge
+- **On every push to main** - Protect production
+- **Nightly** - Comprehensive regression testing
+
+### Parallel Execution
+
+- **4x faster feedback** - Shard across multiple workers
+- **Efficient resource usage** - Maximize CI runner utilization
+
+### Selective Testing
+
+- **Run only affected tests** - Git diff-based selection
+- **Faster PR feedback** - Don't run entire suite every time
+
+### Flakiness Detection
+
+- **Burn-in loops** - Run tests multiple times
+- **Early detection** - Catch flaky tests in PRs
+- **Confidence building** - Know tests are reliable
+
+### Artifact Collection
+
+- **Test results** - Saved for 7 days
+- **Screenshots** - On test failures
+- **Videos** - Full test recordings
+- **Traces** - Playwright trace files for debugging
+
+## Tips
+
+### Start Simple, Add Complexity
+
+**Week 1:** Basic pipeline
+
+```yaml
+- Run tests on PR
+- Single worker (no sharding)
+```
+
+**Week 2:** Add parallelization
+
+```yaml
+- Shard across 4 workers
+- Faster feedback
+```
+
+**Week 3:** Add selective testing
+
+```yaml
+- Git diff-based selection
+- Skip unaffected tests
+```
+
+**Week 4:** Add burn-in
+
+```yaml
+- Detect flaky tests
+- Run on PR and nightly
+```
+
+### Optimize for Feedback Speed
+
+**Goal:** PR feedback in < 5 minutes
+
+**Strategies:**
+
+- Shard tests across workers (4 workers = 4x faster)
+- Use selective testing (run 20% of tests, not 100%)
+- Cache dependencies (`actions/cache`, `cache: 'npm'`)
+- Run smoke tests first, full suite after
+
+**Example fast workflow:**
+
+```yaml
+jobs:
+  smoke:
+    # Run critical path tests (2 min)
+    run: npm run test:smoke
+
+  full:
+    needs: smoke
+    # Run full suite only if smoke passes (10 min)
+    run: npm test
+```
+
+### Use Test Tags
+
+Tag tests for selective execution:
+
+```typescript
+// Critical path tests (always run)
+test('@critical should login', async ({ page }) => {});
+
+// Smoke tests (run first)
+test('@smoke should load homepage', async ({ page }) => {});
+
+// Slow tests (run nightly only)
+test('@slow should process large file', async ({ page }) => {});
+
+// Skip in CI
+test('@local-only should use local service', async ({ page }) => {});
+```
+
+**In CI:**
+
+```bash
+# PR: Run critical and smoke only
+npx playwright test --grep "@critical|@smoke"
+
+# Nightly: Run everything except local-only
+npx playwright test --grep-invert "@local-only"
+```
+
+### Monitor CI Performance
+
+Track metrics:
+
+```markdown
+## CI Metrics
+
+| Metric           | Target   | Current | Status |
+| ---------------- | -------- | ------- | ------ |
+| PR feedback time | < 5 min  | 3m 24s  | ✅     |
+| Full suite time  | < 15 min | 12m 18s | ✅     |
+| Flakiness rate   | < 1%     | 0.3%    | ✅     |
+| CI cost/month    | < $100   | $75     | ✅     |
+```
+
+### Handle Flaky Tests
+
+When burn-in detects flakiness:
+
+1. **Quarantine flaky test:**
+
+```typescript
+test.skip('flaky test - investigating', async ({ page }) => {
+  // TODO: Fix flakiness
+});
+```
+
+2. **Investigate with trace viewer:**
+
+```bash
+npx playwright show-trace test-results/trace.zip
+```
+
+3. **Fix root cause:**
+
+- Add network-first patterns
+- Remove hard waits
+- Fix race conditions
+
+4. **Verify fix:**
+
+```bash
+npm run test:burn-in -- tests/flaky.spec.ts --repeat 20
+```
+
+### Secure Secrets
+
+**Don't commit secrets to code:**
+
+```yaml
+# ❌ Bad
+- run: API_KEY=sk-1234... npm test
+
+# ✅ Good
+- run: npm test
+  env:
+    API_KEY: ${{ secrets.API_KEY }}
+```
+
+**Use environment-specific secrets:**
+
+- `STAGING_API_URL`
+- `PROD_API_URL`
+- `TEST_API_URL`
+
+### Cache Aggressively
+
+Speed up CI with caching:
+
+```yaml
+# Cache node_modules
+- uses: actions/setup-node@v4
+  with:
+    cache: 'npm'
+
+# Cache Playwright browsers
+- name: Cache Playwright browsers
+  uses: actions/cache@v4
+  with:
+    path: ~/.cache/ms-playwright
+    key: playwright-${{ hashFiles('package-lock.json') }}
+```
+
+## Common Issues
+
+### Tests Pass Locally, Fail in CI
+
+**Symptoms:**
+
+- Green locally, red in CI
+- "Works on my machine"
+
+**Common Causes:**
+
+- Different Node version
+- Different browser version
+- Missing environment variables
+- Timezone differences
+- Race conditions (CI slower)
+
+**Solutions:**
+
+```yaml
+# Pin Node version
+- uses: actions/setup-node@v4
+  with:
+    node-version-file: '.nvmrc'
+
+# Pin browser versions
+- run: npx playwright install --with-deps chromium@1.40.0
+
+  # Set timezone
+  env:
+    TZ: 'America/New_York'
+```
+
+### CI Takes Too Long
+
+**Problem:** CI takes 30+ minutes, developers wait too long.
+
+**Solutions:**
+
+1. **Shard tests:** 4 workers = 4x faster
+2. **Selective testing:** Only run affected tests on PR
+3. **Smoke tests first:** Run critical path (2 min), full suite after
+4. **Cache dependencies:** `npm ci` with cache
+5. **Optimize tests:** Remove slow tests, hard waits
+
+### Burn-In Always Fails
+
+**Problem:** Burn-in job fails every time.
+
+**Cause:** Test suite is flaky.
+
+**Solution:**
+
+1. Identify flaky tests (check which iteration fails)
+2. Fix flaky tests using `test-review`
+3. Re-run burn-in on specific files:
+
+```bash
+npm run test:burn-in tests/flaky.spec.ts
+```
+
+### Out of CI Minutes
+
+**Problem:** Using too many CI minutes, hitting plan limit.
+
+**Solutions:**
+
+1. Run full suite only on main branch
+2. Use selective testing on PRs
+3. Run expensive tests nightly only
+4. Self-host runners (for GitHub Actions)
+
+## Related Guides
+
+- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - Run first
+- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Audit CI tests
+- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Burn-in utility
+
+## Understanding the Concepts
+
+- [Test Quality Standards](/docs/explanation/test-quality-standards.md) - Why determinism matters
+- [Network-First Patterns](/docs/explanation/network-first-patterns.md) - Avoid CI flakiness
+
+## Reference
+
+- [Command: \*ci](/docs/reference/commands.md#ci) - Full command reference
+- [TEA Configuration](/docs/reference/configuration.md) - CI-related config options
+
+---
+
+Generated with [BMad Method](https://bmad-method.org) - TEA (Test Engineering Architect)