bmad-method 6.0.0-alpha.4 → 6.0.0-alpha.5

package/bmad/bmm/workflows/testarch/ci/README.md

@@ -1,493 +0,0 @@
-# CI/CD Pipeline Setup Workflow
-
-Scaffolds a production-ready CI/CD quality pipeline with test execution, burn-in loops for flaky test detection, parallel sharding, and artifact collection. This workflow creates platform-specific CI configuration optimized for fast feedback (< 45 min total) and reliable test execution with 20× speedup over sequential runs.
-
-## Usage
-
-```bash
-bmad tea *ci
-```
-
-The TEA agent runs this workflow when:
-
-- Test framework is configured and tests pass locally
-- Team is ready to enable continuous integration
-- Existing CI pipeline needs optimization or modernization
-- Burn-in loop is needed for flaky test detection
-
-## Inputs
-
-**Required Context Files:**
-
-- **Framework config** (playwright.config.ts, cypress.config.ts): Determines test commands and configuration
-- **package.json**: Dependencies and scripts for caching strategy
-- **.nvmrc**: Node version for CI (optional, defaults to Node 20 LTS)
-
-**Optional Context Files:**
-
-- **Existing CI config**: To update rather than create new
-- **.git/config**: For CI platform auto-detection
-
-**Workflow Variables:**
-
-- `ci_platform`: Auto-detected (github-actions/gitlab-ci/circle-ci) or explicit
-- `test_framework`: Detected from framework config (playwright/cypress)
-- `parallel_jobs`: Number of parallel shards (default: 4)
-- `burn_in_enabled`: Enable burn-in loop (default: true)
-- `burn_in_iterations`: Burn-in iterations (default: 10)
-- `selective_testing_enabled`: Run only changed tests (default: true)
-- `artifact_retention_days`: Artifact storage duration (default: 30)
-- `cache_enabled`: Enable dependency caching (default: true)
-
-## Outputs
-
-**Primary Deliverables:**
-
-1. **CI Configuration File**
-   - `.github/workflows/test.yml` (GitHub Actions)
-   - `.gitlab-ci.yml` (GitLab CI)
-   - Platform-specific optimizations and best practices
-
-2. **Pipeline Stages**
-   - **Lint**: Code quality checks (<2 min)
-   - **Test**: Parallel execution with 4 shards (<10 min per shard)
-   - **Burn-In**: Flaky test detection with 10 iterations (<30 min)
-   - **Report**: Aggregate results and publish artifacts
-
-3. **Helper Scripts**
-   - `scripts/test-changed.sh`: Selective testing (run only affected tests)
-   - `scripts/ci-local.sh`: Local CI mirror for debugging
-   - `scripts/burn-in.sh`: Standalone burn-in execution
-
-4. **Documentation**
-   - `docs/ci.md`: Pipeline guide, debugging, secrets setup
-   - `docs/ci-secrets-checklist.md`: Required secrets and configuration
-   - Inline comments in CI configuration files
-
-5. **Optimization Features**
-   - Dependency caching (npm + browser binaries): 2-5 min savings
-   - Parallel sharding: 75% time reduction
-   - Retry logic: Handles transient failures (2 retries)
-   - Failure-only artifacts: Cost-effective debugging
-
-**Performance Targets:**
-
-- Lint: <2 minutes
-- Test (per shard): <10 minutes
-- Burn-in: <30 minutes
-- **Total: <45 minutes** (20× faster than sequential)
-
-**Validation Safeguards:**
-
-- ✅ Git repository initialized
-- ✅ Local tests pass before CI setup
-- ✅ Framework configuration exists
-- ✅ CI platform accessible
-
-## Key Features
-
-### Burn-In Loop for Flaky Test Detection
-
-**Critical production pattern:**
-
-```yaml
-burn-in:
-  runs-on: ubuntu-latest
-  steps:
-    - run: |
-        for i in {1..10}; do
-          echo "🔥 Burn-in iteration $i/10"
-          npm run test:e2e || exit 1
-        done
-```
-
-**Purpose**: Runs tests 10 times to catch non-deterministic failures before they reach main branch.
-
-**When to run:**
-
-- On PRs to main/develop
-- Weekly on cron schedule
-- After test infrastructure changes
-
-**Failure threshold**: Even ONE failure → tests are flaky, must fix before merging.
-
-### Parallel Sharding
-
-**Splits tests across 4 jobs:**
-
-```yaml
-strategy:
-  matrix:
-    shard: [1, 2, 3, 4]
-steps:
-  - run: npm run test:e2e -- --shard=${{ matrix.shard }}/4
-```
-
-**Benefits:**
-
-- 75% time reduction (40 min → 10 min per shard)
-- Faster feedback on PRs
-- Configurable shard count
-
-### Smart Caching
-
-**Node modules + browser binaries:**
-
-```yaml
-- uses: actions/cache@v4
-  with:
-    path: ~/.npm
-    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
-```
-
-**Benefits:**
-
-- 2-5 min savings per run
-- Consistent across builds
-- Automatic invalidation on dependency changes
-
-### Selective Testing
-
-**Run only tests affected by code changes:**
-
-```bash
-# scripts/test-changed.sh
-CHANGED_FILES=$(git diff --name-only HEAD~1)
-npm run test:e2e -- --grep="$AFFECTED_TESTS"
-```
-
-**Benefits:**
-
-- 50-80% time reduction for focused PRs
-- Faster feedback cycle
-- Full suite still runs on main branch
-
-### Failure-Only Artifacts
-
-**Upload debugging materials only on test failures:**
-
-- Traces (Playwright): 5-10 MB per test
-- Screenshots: 100-500 KB each
-- Videos: 2-5 MB per test
-- HTML reports: 1-2 MB
-
-**Benefits:**
-
-- Reduces storage costs by 90%
-- Maintains full debugging capability
-- 30-day retention default
-
-### Local CI Mirror
-
-**Debug CI failures locally:**
-
-```bash
-./scripts/ci-local.sh
-# Runs: lint → test → burn-in (3 iterations)
-```
-
-**Mirrors CI environment:**
-
-- Same Node version
-- Same commands
-- Reduced burn-in (3 vs 10 for faster feedback)
-
-### Knowledge Base Integration
-
-Automatically consults TEA knowledge base:
-
-- `ci-burn-in.md` - Burn-in loop patterns and iterations
-- `selective-testing.md` - Changed test detection strategies
-- `visual-debugging.md` - Artifact collection best practices
-- `test-quality.md` - CI-specific quality criteria
-
-## Integration with Other Workflows
-
-**Before ci:**
-
-- **framework**: Sets up test infrastructure and configuration
-- **test-design** (optional): Plans test coverage strategy
-
-**After ci:**
-
-- **atdd**: Generate failing tests that run in CI
-- **automate**: Expand test coverage that CI executes
-- **trace (Phase 2)**: Use CI results for quality gate decisions
-
-**Coordinates with:**
-
-- **dev-story**: Tests run in CI after story implementation
-- **retrospective**: CI metrics inform process improvements
-
-**Updates:**
-
-- `bmm-workflow-status.md`: Adds CI setup to Quality & Testing Progress section
-
-## Important Notes
-
-### CI Platform Auto-Detection
-
-**GitHub Actions** (default):
-
-- Auto-selected if `github.com` in git remote
-- Free 2000 min/month for private repos
-- Unlimited for public repos
-- `.github/workflows/test.yml`
-
-**GitLab CI**:
-
-- Auto-selected if `gitlab.com` in git remote
-- Free 400 min/month
-- `.gitlab-ci.yml`
-
-**Circle CI** / **Jenkins**:
-
-- User must specify explicitly
-- Templates provided for both
-
-### Burn-In Strategy
-
-**Iterations:**
-
-- **3**: Quick feedback (local development)
-- **10**: Standard (PR checks) ← recommended
-- **100**: High-confidence (release branches)
-
-**When to run:**
-
-- ✅ On PRs to main/develop
-- ✅ Weekly scheduled (cron)
-- ✅ After test infra changes
-- ❌ Not on every commit (too slow)
-
-**Cost-benefit:**
-
-- 30 minutes of CI time → Prevents hours of debugging flaky tests
-
-### Artifact Collection Strategy
-
-**Failure-only collection:**
-
-- Saves 90% storage costs
-- Maintains debugging capability
-- Automatic cleanup after retention period
-
-**What to collect:**
-
-- Traces: Full execution context (Playwright)
-- Screenshots: Visual evidence
-- Videos: Interaction playback
-- HTML reports: Detailed results
-- Console logs: Error messages
-
-**What NOT to collect:**
-
-- Passing test artifacts (waste of space)
-- Large binaries
-- Sensitive data (use secrets instead)
-
-### Selective Testing Trade-offs
-
-**Benefits:**
-
-- 50-80% time reduction for focused changes
-- Faster feedback loop
-- Lower CI costs
-
-**Risks:**
-
-- May miss integration issues
-- Relies on accurate change detection
-- False positives if detection is too aggressive
-
-**Mitigation:**
-
-- Always run full suite on merge to main
-- Use burn-in loop on main branch
-- Monitor for missed issues
-
-### Parallelism Configuration
-
-**4 shards** (default):
-
-- Optimal for 40-80 test files
-- ~10 min per shard
-- Balances speed vs resource usage
-
-**Adjust if:**
-
-- Tests complete in <5 min → reduce shards
-- Tests take >15 min → increase shards
-- CI limits concurrent jobs → reduce shards
-
-**Formula:**
-
-```
-Total test time / Target shard time = Optimal shards
-Example: 40 min / 10 min = 4 shards
-```
-
-### Retry Logic
-
-**2 retries** (default):
-
-- Handles transient network issues
-- Mitigates race conditions
-- Does NOT mask flaky tests (burn-in catches those)
-
-**When retries trigger:**
-
-- Network timeouts
-- Service unavailability
-- Resource constraints
-
-**When retries DON'T help:**
-
-- Assertion failures (logic errors)
-- Flaky tests (non-deterministic)
-- Configuration errors
-
-### Notification Setup (Optional)
-
-**Supported channels:**
-
-- Slack: Webhook integration
-- Email: SMTP configuration
-- Discord: Webhook integration
-
-**Configuration:**
-
-```yaml
-notify_on_failure: true
-notification_channels: 'slack'
-# Requires SLACK_WEBHOOK secret in CI settings
-```
-
-**Best practice:** Enable for main/develop branches only, not PRs.
-
-## Validation Checklist
-
-After workflow completion, verify:
-
-- [ ] CI configuration file created and syntactically valid
-- [ ] Burn-in loop configured (10 iterations)
-- [ ] Parallel sharding enabled (4 jobs)
-- [ ] Caching configured (dependencies + browsers)
-- [ ] Artifact collection on failure only
-- [ ] Helper scripts created and executable
-- [ ] Documentation complete (ci.md, secrets checklist)
-- [ ] No errors or warnings during scaffold
-- [ ] First CI run triggered and passes
-
-Refer to `checklist.md` for comprehensive validation criteria.
-
-## Example Execution
-
-**Scenario 1: New GitHub Actions setup**
-
-```bash
-bmad tea *ci
-
-# TEA detects:
-# - GitHub repository (github.com in git remote)
-# - Playwright framework
-# - Node 20 from .nvmrc
-# - 60 test files
-
-# TEA scaffolds:
-# - .github/workflows/test.yml
-# - 4-shard parallel execution
-# - Burn-in loop (10 iterations)
-# - Dependency + browser caching
-# - Failure artifacts (traces, screenshots)
-# - Helper scripts
-# - Documentation
-
-# Result:
-# Total CI time: 42 minutes (was 8 hours sequential)
-# - Lint: 1.5 min
-# - Test (4 shards): 9 min each
-# - Burn-in: 28 min
-```
-
-**Scenario 2: Update existing GitLab CI**
-
-```bash
-bmad tea *ci
-
-# TEA detects:
-# - Existing .gitlab-ci.yml
-# - Cypress framework
-# - No caching configured
-
-# TEA asks: "Update existing CI or create new?"
-# User: "Update"
-
-# TEA enhances:
-# - Adds burn-in job
-# - Configures caching (cache: paths)
-# - Adds parallel: 4
-# - Updates artifact collection
-# - Documents secrets needed
-
-# Result:
-# CI time reduced from 45 min → 12 min
-```
-
-**Scenario 3: Standalone burn-in setup**
-
-```bash
-# User wants only burn-in, no full CI
-bmad tea *ci
-# Set burn_in_enabled: true, skip other stages
-
-# TEA creates:
-# - Minimal workflow with burn-in only
-# - scripts/burn-in.sh for local testing
-# - Documentation for running burn-in
-
-# Use case:
-# - Validate test stability before full CI setup
-# - Debug intermittent failures
-# - Confidence check before release
-```
-
-## Troubleshooting
-
-**Issue: "Git repository not found"**
-
-- **Cause**: No .git/ directory
-- **Solution**: Run `git init` and `git remote add origin <url>`
-
-**Issue: "Tests fail locally but should set up CI anyway"**
-
-- **Cause**: Workflow halts if local tests fail
-- **Solution**: Fix tests first, or temporarily skip preflight (not recommended)
-
-**Issue: "CI takes longer than 10 min per shard"**
-
-- **Cause**: Too many tests per shard
-- **Solution**: Increase shard count (e.g., 4 → 8)
-
-**Issue: "Burn-in passes locally but fails in CI"**
-
-- **Cause**: Environment differences (timing, resources)
-- **Solution**: Use `scripts/ci-local.sh` to mirror CI environment
-
-**Issue: "Caching not working"**
-
-- **Cause**: Cache key mismatch or cache limit exceeded
-- **Solution**: Check cache key formula, verify platform limits
-
-## Related Workflows
-
-- **framework**: Set up test infrastructure → [framework/README.md](../framework/README.md)
-- **atdd**: Generate acceptance tests → [atdd/README.md](../atdd/README.md)
-- **automate**: Expand test coverage → [automate/README.md](../automate/README.md)
-- **trace**: Traceability and quality gate decisions → [trace/README.md](../trace/README.md)
-
-## Version History
-
-- **v4.0 (BMad v6)**: Pure markdown instructions, enhanced workflow.yaml, burn-in loop integration
-- **v3.x**: XML format instructions, basic CI setup
-- **v2.x**: Legacy task-based approach