ma-agents 2.19.2 → 2.20.0

package/lib/bmad-cache/tea/docs/how-to/workflows/run-test-review.md

@@ -0,0 +1,634 @@
+---
+title: 'How to Run Test Review with TEA'
+description: Audit test quality using TEA's comprehensive knowledge base and get 0-100 scoring
+---
+
+# How to Run Test Review with TEA
+
+Use TEA's `test-review` workflow to audit test quality with objective scoring and actionable feedback. TEA reviews tests against its knowledge base of best practices.
+
+Coverage scoring is intentionally excluded from `test-review`. Use `trace` for requirements coverage analysis and coverage gate decisions.
+
+## When to Use This
+
+- Want to validate test quality objectively
+- Need quality metrics for test quality gates
+- Preparing for production deployment
+- Reviewing team-written tests
+- Auditing AI-generated tests
+- Onboarding new team members (show good patterns)
+- Validating migration quality before coverage expansion
+
+## Prerequisites
+
+- BMad Method installed
+- TEA agent available
+- Tests written (to review)
+- Test framework configured
+
+## Steps
+
+### 1. Load TEA Agent
+
+Start a fresh chat and load TEA:
+
+```
+tea
+```
+
+### 2. Run the Test Review Workflow
+
+```
+test-review
+```
+
+### 3. Specify Review Scope
+
+TEA will ask what to review.
+
+#### Option A: Single File
+
+Review one test file:
+
+```
+tests/e2e/checkout.spec.ts
+```
+
+**Best for:**
+
+- Reviewing specific failing tests
+- Quick feedback on new tests
+- Learning from specific examples
+
+#### Option B: Directory
+
+Review all tests in a directory:
+
+```
+tests/e2e/
+```
+
+**Best for:**
+
+- Reviewing E2E test suite
+- Comparing test quality across files
+- Finding patterns of issues
+
+#### Option C: Entire Suite
+
+Review all tests:
+
+```
+tests/
+```
+
+**Best for:**
+
+- Release gate quality check
+- Comprehensive audit
+- Establishing baseline metrics
+
+### 4. Review the Quality Report
+
+TEA generates a comprehensive quality report with scoring.
+
+#### Report Structure (`test-review.md`):
+
+````markdown
+# Test Quality Review Report
+
+**Date:** 2026-01-13
+**Scope:** tests/e2e/
+**Overall Score:** 76/100
+
+## Summary
+
+- **Tests Reviewed:** 12
+- **Passing Quality:** 9 tests (75%)
+- **Needs Improvement:** 3 tests (25%)
+- **Critical Issues:** 2
+- **Recommendations:** 6
+
+## Critical Issues
+
+### 1. Hard Waits Detected
+
+**File:** `tests/e2e/checkout.spec.ts:45`
+**Issue:** Using `page.waitForTimeout(3000)`
+**Impact:** Test is flaky and unnecessarily slow
+**Severity:** Critical
+
+**Current Code:**
+
+```typescript
+await page.click('button[type="submit"]');
+await page.waitForTimeout(3000); // ❌ Hard wait
+await expect(page.locator('.success')).toBeVisible();
+```
+````
+
+**Fix:**
+
+```typescript
+await page.click('button[type="submit"]');
+// Wait for the API response that triggers success message
+await page.waitForResponse((resp) => resp.url().includes('/api/checkout') && resp.ok());
+await expect(page.locator('.success')).toBeVisible();
+```
+
+**Why This Matters:**
+
+- Hard waits are fixed timeouts that don't wait for actual conditions
+- Tests fail intermittently on slower machines
+- Wastes time waiting even when response is fast
+- Network-first patterns are more reliable
+
+---
+
+### 2. Conditional Flow Control
+
+**File:** `tests/e2e/profile.spec.ts:28`
+**Issue:** Using if/else to handle optional elements
+**Impact:** Non-deterministic test behavior
+**Severity:** Critical
+
+**Current Code:**
+
+```typescript
+if (await page.locator('.banner').isVisible()) {
+  await page.click('.dismiss');
+}
+// ❌ Test behavior changes based on banner presence
+```
+
+**Fix:**
+
+```typescript
+// Option 1: Make banner presence deterministic
+await expect(page.locator('.banner')).toBeVisible();
+await page.click('.dismiss');
+
+// Option 2: Test both scenarios separately
+test('should show banner for new users', async ({ page }) => {
+  // Test with banner
+});
+
+test('should not show banner for returning users', async ({ page }) => {
+  // Test without banner
+});
+```
+
+**Why This Matters:**
+
+- Tests should be deterministic (same result every run)
+- Conditionals hide bugs (what if banner should always show?)
+- Makes debugging harder
+- Violates test isolation principle
+
+## Recommendations
+
+### 1. Extract Repeated Setup
+
+**File:** `tests/e2e/profile.spec.ts`
+**Issue:** Login code duplicated in every test
+**Severity:** Medium
+**Impact:** Maintenance burden, test verbosity
+
+**Current:**
+
+```typescript
+test('test 1', async ({ page }) => {
+  await page.goto('/login');
+  await page.fill('[name="email"]', 'test@example.com');
+  await page.fill('[name="password"]', 'password');
+  await page.click('button[type="submit"]');
+  // Test logic...
+});
+
+test('test 2', async ({ page }) => {
+  // Same login code repeated
+});
+```
+
+**Fix (Vanilla Playwright):**
+
+```typescript
+// Create fixture in tests/support/fixtures/auth.ts
+import { test as base, Page } from '@playwright/test';
+
+export const test = base.extend<{ authenticatedPage: Page }>({
+  authenticatedPage: async ({ page }, use) => {
+    await page.goto('/login');
+    await page.getByLabel('Email').fill('test@example.com');
+    await page.getByLabel('Password').fill('password');
+    await page.getByRole('button', { name: 'Sign in' }).click();
+    await page.waitForURL(/\/dashboard/);
+    await use(page);
+  },
+});
+
+// Use in tests
+test('test 1', async ({ authenticatedPage }) => {
+  // Already logged in
+});
+```
+
+**Better (With Playwright Utils):**
+
+```typescript
+// Use built-in auth-session fixture
+import { test as base } from '@playwright/test';
+import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session';
+
+export const test = base.extend(createAuthFixtures());
+
+// Use in tests - even simpler
+test('test 1', async ({ page, authToken }) => {
+  // authToken already available (persisted, reused)
+  await page.goto('/dashboard');
+  // Already authenticated via authToken
+});
+```
+
+**Playwright Utils Benefits:**
+
+- Token persisted to disk (faster subsequent runs)
+- Multi-user support out of the box
+- Automatic token renewal if expired
+- No manual login flow needed
+
+---
+
+### 2. Add Network Assertions
+
+**File:** `tests/e2e/api-calls.spec.ts`
+**Issue:** No verification of API responses
+**Severity:** Low
+**Impact:** Tests don't catch API errors
+
+**Current:**
+
+```typescript
+await page.click('button[name="save"]');
+await expect(page.locator('.success')).toBeVisible();
+// ❌ What if API returned 500 but UI shows cached success?
+```
+
+**Enhancement:**
+
+```typescript
+const responsePromise = page.waitForResponse((resp) => resp.url().includes('/api/profile') && resp.status() === 200);
+await page.click('button[name="save"]');
+const response = await responsePromise;
+
+// Verify API response
+const data = await response.json();
+expect(data.success).toBe(true);
+
+// Verify UI
+await expect(page.locator('.success')).toBeVisible();
+```
+
+---
+
+### 3. Improve Test Names
+
+**File:** `tests/e2e/checkout.spec.ts`
+**Issue:** Vague test names
+**Severity:** Low
+**Impact:** Hard to understand test purpose
+
+**Current:**
+
+```typescript
+test('should work', async ({ page }) => {});
+test('test checkout', async ({ page }) => {});
+```
+
+**Better:**
+
+```typescript
+test('should complete checkout with valid credit card', async ({ page }) => {});
+test('should show validation error for expired card', async ({ page }) => {});
+```
+
+## Quality Scores by Category
+
+| Category            | Score | Target | Status               |
+| ------------------- | ----- | ------ | -------------------- |
+| **Determinism**     | 72    | 80     | ⚠️ Needs Improvement |
+| **Isolation**       | 88    | 80     | ✅ Good              |
+| **Maintainability** | 70    | 80     | ⚠️ Needs Improvement |
+| **Performance**     | 60    | 80     | ❌ Critical          |
+
+### Scoring Breakdown
+
+**Determinism (30% weight):**
+
+- Hard waits and race conditions penalized
+- Unstable control flow patterns penalized
+
+**Isolation (30% weight):**
+
+- Shared state and order dependency penalized
+- Cleanup and parallel-safety rewarded
+
+**Maintainability (25% weight):**
+
+- Overly large files and copy-paste patterns penalized
+- Naming clarity and structure rewarded
+
+**Performance (15% weight):**
+
+- Serial bottlenecks and inefficient setup penalized
+- Parallel-friendly structure rewarded
+
+## Files Reviewed
+
+| File                         | Score  | Issues | Status               |
+| ---------------------------- | ------ | ------ | -------------------- |
+| `tests/e2e/checkout.spec.ts` | 65/100 | 4      | ❌ Needs Work        |
+| `tests/e2e/profile.spec.ts`  | 72/100 | 3      | ⚠️ Needs Improvement |
+| `tests/e2e/search.spec.ts`   | 88/100 | 1      | ✅ Good              |
+| `tests/api/profile.spec.ts`  | 92/100 | 0      | ✅ Excellent         |
+
+## Next Steps
+
+### Immediate (Fix Critical Issues)
+
+1. Remove hard waits in `checkout.spec.ts` (line 45, 67, 89)
+2. Fix conditional in `profile.spec.ts` (line 28)
+3. Optimize slow tests in `checkout.spec.ts`
+
+### Short-term (Apply Recommendations)
+
+4. Extract login fixture from `profile.spec.ts`
+5. Add network assertions to `api-calls.spec.ts`
+6. Improve test names in `checkout.spec.ts`
+
+### Long-term (Continuous Improvement)
+
+7. Re-run `test-review` after fixes (target: 85/100)
+8. Add performance budgets to CI
+9. Document test patterns for team
+
+## Knowledge Base References
+
+TEA reviewed against these patterns:
+
+- [test-quality.md](/docs/reference/knowledge-base.md) - Execution limits, isolation
+- [network-first.md](/docs/reference/knowledge-base.md) - Deterministic waits
+- [timing-debugging.md](/docs/reference/knowledge-base.md) - Race conditions
+- [selector-resilience.md](/docs/reference/knowledge-base.md) - Robust selectors
+
+````
+
+## Understanding the Scores
+
+### What Do Scores Mean?
+
+| Score Range | Interpretation | Action |
+|-------------|----------------|--------|
+| **90-100** | Excellent | Minimal changes needed, production-ready |
+| **80-89** | Good | Minor improvements recommended |
+| **70-79** | Acceptable | Address recommendations before release |
+| **60-69** | Needs Improvement | Fix critical issues, apply recommendations |
+| **< 60** | Critical | Significant refactoring needed |
+
+### Scoring Criteria
+
+**Determinism (30%):**
+- Tests produce same result every run
+- No random failures (flakiness)
+- No environment-dependent behavior
+
+**Isolation (30%):**
+- Tests don't depend on each other
+- Can run in any order
+- Clean up after themselves
+
+**Maintainability (25%):**
+- Readable and maintainable
+- Appropriate size
+- Clear naming
+
+**Performance (15%):**
+- Fast execution
+- Efficient selectors
+- No unnecessary waits
+
+**Coverage:**
+- Not scored in `test-review`
+- Use `trace` for coverage percentage, requirement mapping, and gate decisions
+
+## What You Get
+
+### Quality Report
+- Overall score (0-100)
+- Category scores (Determinism, Isolation, etc.)
+- File-by-file breakdown
+
+### Critical Issues
+- Specific line numbers
+- Code examples (current vs fixed)
+- Why it matters explanation
+- Impact assessment
+
+### Recommendations
+- Actionable improvements
+- Code examples
+- Priority/severity levels
+
+### Next Steps
+- Immediate actions (fix critical)
+- Short-term improvements
+- Long-term quality goals
+
+## Tips
+
+### Review Before Release
+
+Make test review part of release checklist:
+
+```markdown
+## Quality Checklist (Test-Review)
+- [ ] All tests passing
+- [ ] Test-review quality score > 80
+- [ ] Critical issues resolved
+- [ ] Performance within budget
+````
+
+### Review After AI Generation
+
+Always review AI-generated tests:
+
+```
+1. Run atdd or automate
+2. Run test-review on generated tests
+3. Fix critical issues
+4. Commit tests
+```
+
+### Set Quality Gates
+
+Use scores as quality gates:
+
+```yaml
+# .github/workflows/test.yml
+- name: Review test quality
+  run: |
+    # Run test-review quality gate
+    # Parse quality score from report
+    if [ $SCORE -lt 80 ]; then
+      echo "Test-review quality gate below threshold"
+      exit 1
+    fi
+```
+
+Coverage gate checks are handled by `trace`, not `test-review`.
+
+### Review Regularly
+
+Schedule periodic reviews:
+
+- **Per story:** Optional (spot check new tests)
+- **Per epic:** Recommended (ensure consistency)
+- **Per release:** Recommended for test quality gates (coverage gates remain in `trace`)
+- **Quarterly:** Audit entire suite
+
+### Focus Reviews
+
+For large suites, review incrementally:
+
+**Week 1:** Review E2E tests
+**Week 2:** Review API tests
+**Week 3:** Review component tests (Cypress CT or Vitest)
+**Week 4:** Apply fixes across all suites
+
+**Component Testing Note:** TEA reviews component tests using framework-specific knowledge:
+
+- **Cypress:** Reviews Cypress Component Testing specs (\*.cy.tsx)
+- **Playwright:** Reviews Vitest component tests (\*.test.tsx)
+
+### Use Reviews for Learning
+
+Share reports with team:
+
+```
+Team Meeting:
+- Review test-review.md
+- Discuss critical issues
+- Agree on patterns
+- Update team guidelines
+```
+
+### Compare Over Time
+
+Track improvement:
+
+```markdown
+## Quality Trend
+
+| Date       | Score | Critical Issues | Notes                 |
+| ---------- | ----- | --------------- | --------------------- |
+| 2026-01-01 | 65    | 5               | Baseline              |
+| 2026-01-15 | 72    | 2               | Fixed hard waits      |
+| 2026-02-01 | 84    | 0               | All critical resolved |
+```
+
+## Common Issues
+
+### Low Determinism Score
+
+**Symptoms:**
+
+- Tests fail randomly
+- "Works on my machine"
+- CI failures that don't reproduce locally
+
+**Common Causes:**
+
+- Hard waits (`waitForTimeout`)
+- Conditional flow control (`if/else`)
+- Try-catch for flow control
+- Missing network-first patterns
+
+**Fix:** Review determinism section, apply network-first patterns
+
+### Low Performance Score
+
+**Symptoms:**
+
+- Tests take > 1.5 minutes each
+- Test suite takes hours
+- CI times out
+
+**Common Causes:**
+
+- Unnecessary waits (hard timeouts)
+- Inefficient selectors (XPath, complex CSS)
+- Not using parallelization
+- Heavy setup in every test
+
+**Fix:** Optimize waits, improve selectors, use fixtures
+
+### Low Isolation Score
+
+**Symptoms:**
+
+- Tests fail when run in different order
+- Tests fail in parallel
+- Test data conflicts
+
+**Common Causes:**
+
+- Shared global state
+- Tests don't clean up
+- Hard-coded test data
+- Database not reset between tests
+
+**Fix:** Use fixtures, clean up in afterEach, use unique test data
+
+### "Too Many Issues to Fix"
+
+**Problem:** Report shows 50+ issues, overwhelming.
+
+**Solution:** Prioritize:
+
+1. Fix all critical issues first
+2. Apply top 3 recommendations
+3. Re-run review
+4. Iterate
+
+Don't try to fix everything at once.
+
+### Reviews Take Too Long
+
+**Problem:** Reviewing entire suite takes hours.
+
+**Solution:** Review incrementally:
+
+- Review new tests in PR review
+- Schedule directory reviews weekly
+- Full suite review quarterly
+
+## Related Guides
+
+- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate tests to review
+- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand and improve tests before review
+- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Coverage analysis and gate decisions
+
+## Understanding the Concepts
+
+- [Test Quality Standards](/docs/explanation/test-quality-standards.md) - What makes tests good
+- [Network-First Patterns](/docs/explanation/network-first-patterns.md) - Avoiding flakiness
+- [Fixture Architecture](/docs/explanation/fixture-architecture.md) - Reusable patterns
+
+## Reference
+
+- [Command: \*test-review](/docs/reference/commands.md#test-review) - Full command reference
+- [Knowledge Base Index](/docs/reference/knowledge-base.md) - Patterns TEA reviews against
+
+---
+
+Generated with [BMad Method](https://bmad-method.org) - TEA (Test Engineering Architect)