bmad-method 5.1.3 → 6.0.0-alpha.1

package/src/modules/bmm/workflows/testarch/automate/workflow.yaml

@@ -0,0 +1,63 @@
+# Test Architect workflow: automate
+name: testarch-automate
+description: "Expand test automation coverage after implementation or analyze existing codebase to generate comprehensive test suite"
+author: "BMad"
+
+# Critical variables from config
+config_source: "{project-root}/bmad/bmm/config.yaml"
+output_folder: "{config_source}:output_folder"
+user_name: "{config_source}:user_name"
+communication_language: "{config_source}:communication_language"
+document_output_language: "{config_source}:document_output_language"
+date: system-generated
+
+# Workflow components
+installed_path: "{project-root}/bmad/bmm/workflows/testarch/automate"
+instructions: "{installed_path}/instructions.md"
+validation: "{installed_path}/checklist.md"
+template: false
+
+# Variables and inputs
+variables:
+  # Execution mode and targeting
+  standalone_mode: true # Can work without BMad artifacts (true) or integrate with BMad (false)
+  coverage_target: "critical-paths" # critical-paths, comprehensive, selective
+
+  # Directory paths
+  test_dir: "{project-root}/tests" # Root test directory
+  source_dir: "{project-root}/src" # Source code directory
+
+# Output configuration
+default_output_file: "{output_folder}/automation-summary.md"
+
+# Required tools
+required_tools:
+  - read_file # Read source code, existing tests, BMad artifacts
+  - write_file # Create test files, fixtures, factories, summaries
+  - create_directory # Create test directories
+  - list_files # Discover features and existing tests
+  - search_repo # Find coverage gaps and patterns
+  - glob # Find test files and source files
+
+# Recommended inputs (optional - depends on mode)
+recommended_inputs:
+  - story: "Story markdown with acceptance criteria (optional - BMad mode only)"
+  - tech_spec: "Technical specification (optional - BMad mode only)"
+  - test_design: "Test design document with risk/priority (optional - BMad mode only)"
+  - source_code: "Feature implementation to analyze (required for standalone mode)"
+  - existing_tests: "Current test suite for gap analysis (always helpful)"
+  - framework_config: "Test framework configuration (playwright.config.ts, cypress.config.ts)"
+
+tags:
+  - qa
+  - automation
+  - test-architect
+  - regression
+  - coverage
+
+execution_hints:
+  interactive: false # Minimize prompts
+  autonomous: true # Proceed without user input unless blocked
+  iterative: true
+
+web_bundle: false

package/src/modules/bmm/workflows/testarch/ci/README.md

@@ -0,0 +1,493 @@
+# CI/CD Pipeline Setup Workflow
+
+Scaffolds a production-ready CI/CD quality pipeline with test execution, burn-in loops for flaky test detection, parallel sharding, and artifact collection. This workflow creates platform-specific CI configuration optimized for fast feedback (< 45 min total) and reliable test execution with 20× speedup over sequential runs.
+
+## Usage
+
+```bash
+bmad tea *ci
+```
+
+The TEA agent runs this workflow when:
+
+- Test framework is configured and tests pass locally
+- Team is ready to enable continuous integration
+- Existing CI pipeline needs optimization or modernization
+- Burn-in loop is needed for flaky test detection
+
+## Inputs
+
+**Required Context Files:**
+
+- **Framework config** (playwright.config.ts, cypress.config.ts): Determines test commands and configuration
+- **package.json**: Dependencies and scripts for caching strategy
+- **.nvmrc**: Node version for CI (optional, defaults to Node 20 LTS)
+
+**Optional Context Files:**
+
+- **Existing CI config**: To update rather than create new
+- **.git/config**: For CI platform auto-detection
+
+**Workflow Variables:**
+
+- `ci_platform`: Auto-detected (github-actions/gitlab-ci/circle-ci) or explicit
+- `test_framework`: Detected from framework config (playwright/cypress)
+- `parallel_jobs`: Number of parallel shards (default: 4)
+- `burn_in_enabled`: Enable burn-in loop (default: true)
+- `burn_in_iterations`: Burn-in iterations (default: 10)
+- `selective_testing_enabled`: Run only changed tests (default: true)
+- `artifact_retention_days`: Artifact storage duration (default: 30)
+- `cache_enabled`: Enable dependency caching (default: true)
+
+## Outputs
+
+**Primary Deliverables:**
+
+1. **CI Configuration File**
+   - `.github/workflows/test.yml` (GitHub Actions)
+   - `.gitlab-ci.yml` (GitLab CI)
+   - Platform-specific optimizations and best practices
+
+2. **Pipeline Stages**
+   - **Lint**: Code quality checks (<2 min)
+   - **Test**: Parallel execution with 4 shards (<10 min per shard)
+   - **Burn-In**: Flaky test detection with 10 iterations (<30 min)
+   - **Report**: Aggregate results and publish artifacts
+
+3. **Helper Scripts**
+   - `scripts/test-changed.sh`: Selective testing (run only affected tests)
+   - `scripts/ci-local.sh`: Local CI mirror for debugging
+   - `scripts/burn-in.sh`: Standalone burn-in execution
+
+4. **Documentation**
+   - `docs/ci.md`: Pipeline guide, debugging, secrets setup
+   - `docs/ci-secrets-checklist.md`: Required secrets and configuration
+   - Inline comments in CI configuration files
+
+5. **Optimization Features**
+   - Dependency caching (npm + browser binaries): 2-5 min savings
+   - Parallel sharding: 75% time reduction
+   - Retry logic: Handles transient failures (2 retries)
+   - Failure-only artifacts: Cost-effective debugging
+
+**Performance Targets:**
+
+- Lint: <2 minutes
+- Test (per shard): <10 minutes
+- Burn-in: <30 minutes
+- **Total: <45 minutes** (20× faster than sequential)
+
+**Validation Safeguards:**
+
+- ✅ Git repository initialized
+- ✅ Local tests pass before CI setup
+- ✅ Framework configuration exists
+- ✅ CI platform accessible
+
+## Key Features
+
+### Burn-In Loop for Flaky Test Detection
+
+**Critical production pattern:**
+
+```yaml
+burn-in:
+  runs-on: ubuntu-latest
+  steps:
+    - run: |
+        for i in {1..10}; do
+          echo "🔥 Burn-in iteration $i/10"
+          npm run test:e2e || exit 1
+        done
+```
+
+**Purpose**: Runs tests 10 times to catch non-deterministic failures before they reach main branch.
+
+**When to run:**
+
+- On PRs to main/develop
+- Weekly on cron schedule
+- After test infrastructure changes
+
+**Failure threshold**: Even ONE failure → tests are flaky, must fix before merging.
+
+### Parallel Sharding
+
+**Splits tests across 4 jobs:**
+
+```yaml
+strategy:
+  matrix:
+    shard: [1, 2, 3, 4]
+steps:
+  - run: npm run test:e2e -- --shard=${{ matrix.shard }}/4
+```
+
+**Benefits:**
+
+- 75% time reduction (40 min → 10 min per shard)
+- Faster feedback on PRs
+- Configurable shard count
+
+### Smart Caching
+
+**Node modules + browser binaries:**
+
+```yaml
+- uses: actions/cache@v4
+  with:
+    path: ~/.npm
+    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
+```
+
+**Benefits:**
+
+- 2-5 min savings per run
+- Consistent across builds
+- Automatic invalidation on dependency changes
+
+### Selective Testing
+
+**Run only tests affected by code changes:**
+
+```bash
+# scripts/test-changed.sh
+CHANGED_FILES=$(git diff --name-only HEAD~1)
+npm run test:e2e -- --grep="$AFFECTED_TESTS"
+```
+
+**Benefits:**
+
+- 50-80% time reduction for focused PRs
+- Faster feedback cycle
+- Full suite still runs on main branch
+
+### Failure-Only Artifacts
+
+**Upload debugging materials only on test failures:**
+
+- Traces (Playwright): 5-10 MB per test
+- Screenshots: 100-500 KB each
+- Videos: 2-5 MB per test
+- HTML reports: 1-2 MB
+
+**Benefits:**
+
+- Reduces storage costs by 90%
+- Maintains full debugging capability
+- 30-day retention default
+
+### Local CI Mirror
+
+**Debug CI failures locally:**
+
+```bash
+./scripts/ci-local.sh
+# Runs: lint → test → burn-in (3 iterations)
+```
+
+**Mirrors CI environment:**
+
+- Same Node version
+- Same commands
+- Reduced burn-in (3 vs 10 for faster feedback)
+
+### Knowledge Base Integration
+
+Automatically consults TEA knowledge base:
+
+- `ci-burn-in.md` - Burn-in loop patterns and iterations
+- `selective-testing.md` - Changed test detection strategies
+- `visual-debugging.md` - Artifact collection best practices
+- `test-quality.md` - CI-specific quality criteria
+
+## Integration with Other Workflows
+
+**Before ci:**
+
+- **framework**: Sets up test infrastructure and configuration
+- **test-design** (optional): Plans test coverage strategy
+
+**After ci:**
+
+- **atdd**: Generate failing tests that run in CI
+- **automate**: Expand test coverage that CI executes
+- **trace (Phase 2)**: Use CI results for quality gate decisions
+
+**Coordinates with:**
+
+- **dev-story**: Tests run in CI after story implementation
+- **retrospective**: CI metrics inform process improvements
+
+**Updates:**
+
+- `bmm-workflow-status.md`: Adds CI setup to Quality & Testing Progress section
+
+## Important Notes
+
+### CI Platform Auto-Detection
+
+**GitHub Actions** (default):
+
+- Auto-selected if `github.com` in git remote
+- Free 2000 min/month for private repos
+- Unlimited for public repos
+- `.github/workflows/test.yml`
+
+**GitLab CI**:
+
+- Auto-selected if `gitlab.com` in git remote
+- Free 400 min/month
+- `.gitlab-ci.yml`
+
+**Circle CI** / **Jenkins**:
+
+- User must specify explicitly
+- Templates provided for both
+
+### Burn-In Strategy
+
+**Iterations:**
+
+- **3**: Quick feedback (local development)
+- **10**: Standard (PR checks) ← recommended
+- **100**: High-confidence (release branches)
+
+**When to run:**
+
+- ✅ On PRs to main/develop
+- ✅ Weekly scheduled (cron)
+- ✅ After test infra changes
+- ❌ Not on every commit (too slow)
+
+**Cost-benefit:**
+
+- 30 minutes of CI time → Prevents hours of debugging flaky tests
+
+### Artifact Collection Strategy
+
+**Failure-only collection:**
+
+- Saves 90% storage costs
+- Maintains debugging capability
+- Automatic cleanup after retention period
+
+**What to collect:**
+
+- Traces: Full execution context (Playwright)
+- Screenshots: Visual evidence
+- Videos: Interaction playback
+- HTML reports: Detailed results
+- Console logs: Error messages
+
+**What NOT to collect:**
+
+- Passing test artifacts (waste of space)
+- Large binaries
+- Sensitive data (use secrets instead)
+
+### Selective Testing Trade-offs
+
+**Benefits:**
+
+- 50-80% time reduction for focused changes
+- Faster feedback loop
+- Lower CI costs
+
+**Risks:**
+
+- May miss integration issues
+- Relies on accurate change detection
+- False positives if detection is too aggressive
+
+**Mitigation:**
+
+- Always run full suite on merge to main
+- Use burn-in loop on main branch
+- Monitor for missed issues
+
+### Parallelism Configuration
+
+**4 shards** (default):
+
+- Optimal for 40-80 test files
+- ~10 min per shard
+- Balances speed vs resource usage
+
+**Adjust if:**
+
+- Tests complete in <5 min → reduce shards
+- Tests take >15 min → increase shards
+- CI limits concurrent jobs → reduce shards
+
+**Formula:**
+
+```
+Total test time / Target shard time = Optimal shards
+Example: 40 min / 10 min = 4 shards
+```
+
+### Retry Logic
+
+**2 retries** (default):
+
+- Handles transient network issues
+- Mitigates race conditions
+- Does NOT mask flaky tests (burn-in catches those)
+
+**When retries trigger:**
+
+- Network timeouts
+- Service unavailability
+- Resource constraints
+
+**When retries DON'T help:**
+
+- Assertion failures (logic errors)
+- Flaky tests (non-deterministic)
+- Configuration errors
+
+### Notification Setup (Optional)
+
+**Supported channels:**
+
+- Slack: Webhook integration
+- Email: SMTP configuration
+- Discord: Webhook integration
+
+**Configuration:**
+
+```yaml
+notify_on_failure: true
+notification_channels: 'slack'
+# Requires SLACK_WEBHOOK secret in CI settings
+```
+
+**Best practice:** Enable for main/develop branches only, not PRs.
+
+## Validation Checklist
+
+After workflow completion, verify:
+
+- [ ] CI configuration file created and syntactically valid
+- [ ] Burn-in loop configured (10 iterations)
+- [ ] Parallel sharding enabled (4 jobs)
+- [ ] Caching configured (dependencies + browsers)
+- [ ] Artifact collection on failure only
+- [ ] Helper scripts created and executable
+- [ ] Documentation complete (ci.md, secrets checklist)
+- [ ] No errors or warnings during scaffold
+- [ ] First CI run triggered and passes
+
+Refer to `checklist.md` for comprehensive validation criteria.
+
+## Example Execution
+
+**Scenario 1: New GitHub Actions setup**
+
+```bash
+bmad tea *ci
+
+# TEA detects:
+# - GitHub repository (github.com in git remote)
+# - Playwright framework
+# - Node 20 from .nvmrc
+# - 60 test files
+
+# TEA scaffolds:
+# - .github/workflows/test.yml
+# - 4-shard parallel execution
+# - Burn-in loop (10 iterations)
+# - Dependency + browser caching
+# - Failure artifacts (traces, screenshots)
+# - Helper scripts
+# - Documentation
+
+# Result:
+# Total CI time: 42 minutes (was 8 hours sequential)
+# - Lint: 1.5 min
+# - Test (4 shards): 9 min each
+# - Burn-in: 28 min
+```
+
+**Scenario 2: Update existing GitLab CI**
+
+```bash
+bmad tea *ci
+
+# TEA detects:
+# - Existing .gitlab-ci.yml
+# - Cypress framework
+# - No caching configured
+
+# TEA asks: "Update existing CI or create new?"
+# User: "Update"
+
+# TEA enhances:
+# - Adds burn-in job
+# - Configures caching (cache: paths)
+# - Adds parallel: 4
+# - Updates artifact collection
+# - Documents secrets needed
+
+# Result:
+# CI time reduced from 45 min → 12 min
+```
+
+**Scenario 3: Standalone burn-in setup**
+
+```bash
+# User wants only burn-in, no full CI
+bmad tea *ci
+# Set burn_in_enabled: true, skip other stages
+
+# TEA creates:
+# - Minimal workflow with burn-in only
+# - scripts/burn-in.sh for local testing
+# - Documentation for running burn-in
+
+# Use case:
+# - Validate test stability before full CI setup
+# - Debug intermittent failures
+# - Confidence check before release
+```
+
+## Troubleshooting
+
+**Issue: "Git repository not found"**
+
+- **Cause**: No .git/ directory
+- **Solution**: Run `git init` and `git remote add origin <url>`
+
+**Issue: "Tests fail locally but should set up CI anyway"**
+
+- **Cause**: Workflow halts if local tests fail
+- **Solution**: Fix tests first, or temporarily skip preflight (not recommended)
+
+**Issue: "CI takes longer than 10 min per shard"**
+
+- **Cause**: Too many tests per shard
+- **Solution**: Increase shard count (e.g., 4 → 8)
+
+**Issue: "Burn-in passes locally but fails in CI"**
+
+- **Cause**: Environment differences (timing, resources)
+- **Solution**: Use `scripts/ci-local.sh` to mirror CI environment
+
+**Issue: "Caching not working"**
+
+- **Cause**: Cache key mismatch or cache limit exceeded
+- **Solution**: Check cache key formula, verify platform limits
+
+## Related Workflows
+
+- **framework**: Set up test infrastructure → [framework/README.md](../framework/README.md)
+- **atdd**: Generate acceptance tests → [atdd/README.md](../atdd/README.md)
+- **automate**: Expand test coverage → [automate/README.md](../automate/README.md)
+- **trace**: Traceability and quality gate decisions → [trace/README.md](../trace/README.md)
+
+## Version History
+
+- **v4.0 (BMad v6)**: Pure markdown instructions, enhanced workflow.yaml, burn-in loop integration
+- **v3.x**: XML format instructions, basic CI setup
+- **v2.x**: Legacy task-based approach