agentic-qe 1.0.5 → 1.2.0

package/.claude/skills/test-automation-strategy/SKILL.md

@@ -0,0 +1,842 @@
+---
+name: test-automation-strategy
+description: Build effective test automation strategy using the test pyramid and practical patterns. Use when planning automation approach, selecting tools, or optimizing test suites.
+version: 1.0.0
+category: testing
+tags: [test-automation, test-pyramid, ci-cd, automation-strategy, unit-testing, integration-testing]
+difficulty: intermediate
+estimated_time: 3-4 hours
+author: user
+---
+
+# Test Automation Strategy
+
+## Core Philosophy
+
+**Automate to enable humans to do what humans do best: think, explore, and judge.**
+
+Automation is not the goal. Automation is a tool to get faster feedback and free humans for skilled testing work.
+
+## When to Automate
+
+### Automation Makes Sense When
+
+**Repetitive and Stable**
+- Same test executed frequently
+- Test steps won't change much
+- Cost of automation < cost of manual execution over time
+
+**Fast Feedback Needed**
+- Tests in CI pipeline
+- Regression safety net
+- Sanity checks before deployment
+
+**Difficult to Test Manually**
+- Performance testing (1000 concurrent users)
+- Load testing over hours/days
+- Cross-browser testing (20 browsers)
+- API testing (hundreds of endpoints)
+
+**Precise Verification Required**
+- Complex calculations
+- Data integrity checks
+- Exact timing requirements
+
+### Automation Doesn't Make Sense When
+
+**Rapidly Changing**
+- Feature in active development
+- Requirements unclear
+- UI redesign in progress
+- Cost of maintenance > value
+
+**Complex Setup**
+- Requires extensive test data
+- Multiple systems/dependencies
+- Environment fragile
+- Setup time >> execution time
+
+**Requires Human Judgment**
+- Usability testing
+- Visual design validation
+- Exploratory testing
+- Accessibility evaluation (partly automatable)
+
+**One-Time or Rare**
+- Ad-hoc investigation
+- Production debugging
+- One-time migration
+- Annual report generation
+
+**Low Risk, Low Value**
+- Admin feature used monthly
+- Edge case that never occurs
+- Deprecated feature
+
+## The Test Automation Pyramid
+
+```
+           /\
+          /  \  E2E/UI Tests
+         /    \  (Few)
+        /------\
+       /        \  Integration Tests
+      /          \  (Some)
+     /------------\
+    /              \  Unit Tests
+   /________________\  (Many)
+```
+
+### Unit Tests (70% of automated tests)
+
+**What:** Test individual functions/methods in isolation
+
+**Benefits:**
+- Fast (milliseconds)
+- Reliable
+- Pinpoint failures
+- Enable fearless refactoring
+
+**Tools:** Jest, JUnit, pytest, RSpec
+
+**Example:**
+```javascript
+test('calculateTax returns correct amount', () => {
+  expect(calculateTax(100, 0.10)).toBe(10);
+  expect(calculateTax(0, 0.10)).toBe(0);
+  expect(calculateTax(100, 0)).toBe(0);
+});
+```
+
+### Integration Tests (20% of automated tests)
+
+**What:** Test multiple components working together
+
+**Examples:**
+- API endpoint + database
+- Service layer + external API
+- Module A + Module B interaction
+
+**Benefits:**
+- Test realistic interactions
+- Catch integration issues
+- Still relatively fast
+
+**Tools:** Supertest, RestAssured, Testcontainers
+
+**Example:**
+```javascript
+test('POST /orders creates order in database', async () => {
+  const response = await request(app)
+    .post('/orders')
+    .send({ userId: 123, items: [...] });
+  
+  expect(response.status).toBe(201);
+  
+  const order = await db.orders.findOne({ id: response.body.id });
+  expect(order.userId).toBe(123);
+});
+```
+
+### E2E/UI Tests (10% of automated tests)
+
+**What:** Test complete user workflows through the UI
+
+**Benefits:**
+- Test real user experience
+- Catch UI/UX issues
+- Verify complete flows
+
+**Costs:**
+- Slow (seconds to minutes)
+- Flaky (timing, network, rendering)
+- Expensive to maintain
+- Hard to debug
+
+**Tools:** Playwright, Cypress, Selenium
+
+**Example:**
+```javascript
+test('user can complete checkout', async () => {
+  await page.goto('/products');
+  await page.click('[data-testid="add-to-cart"]');
+  await page.click('[data-testid="checkout"]');
+  await page.fill('[name="cardNumber"]', '4242424242424242');
+  await page.click('[data-testid="place-order"]');
+  
+  await expect(page.locator('.success-message')).toBeVisible();
+});
+```
+
+**Keep E2E tests to critical happy paths only.**
+
+## Anti-Patterns to Avoid
+
+### 🚫 Ice Cream Cone (Inverted Pyramid)
+
+```
+    /________________\  E2E Tests (Many) ❌
+     \              /  Integration Tests (Some)
+      \            /  Unit Tests (Few)
+       \          /
+        \________/
+```
+
+**Problem:** Slow, flaky, expensive test suite
+**Fix:** Write more unit tests, fewer E2E tests
+
+### 🚫 Test All the Things
+
+**Problem:** Automating everything because "automation is good"
+
+**Reality:**
+- Some tests cost more than they provide value
+- Maintenance burden grows unsustainably
+- Team drowns in test maintenance
+
+**Fix:** Automate strategically based on value
+
+### 🚫 Testing Through UI Only
+
+**Problem:** All tests go through browser UI
+
+**Issues:**
+- Slow execution
+- Flaky tests
+- Hard to test edge cases
+- Expensive maintenance
+
+**Fix:** Test at appropriate level
+- Business logic → unit tests
+- API contracts → integration tests
+- Critical flows → E2E tests
+
+### 🚫 Record and Playback
+
+**Problem:** Using record/playback tools without understanding
+
+**Issues:**
+- Fragile tests (break on minor UI changes)
+- Unmaintainable
+- No code review possible
+- Vendor lock-in
+
+**Fix:** Write code-based tests with good selectors
+
+### 🚫 No Test Data Strategy
+
+**Problem:** Tests depend on specific database state
+
+**Issues:**
+- Tests fail randomly
+- Can't run tests in parallel
+- Hard to reproduce failures
+- Environment-dependent
+
+**Fix:** 
+- Generate test data per test
+- Use database transactions
+- Isolate test data
+- Clean up after tests
+
+### 🚫 Sleep-Based Waits
+
+```javascript
+// BAD
+await page.click('button');
+await sleep(5000); // Hope 5 seconds is enough
+await page.click('next-button');
+```
+
+**Problem:** Slow (always waits full time) and flaky (sometimes not enough)
+
+**Fix:** Use explicit waits
+```javascript
+// GOOD
+await page.click('button');
+await page.waitForSelector('next-button', { state: 'visible' });
+await page.click('next-button');
+```
+
+### 🚫 God Tests
+
+**Problem:** One test that tests everything (1000+ lines)
+
+**Issues:**
+- Hard to understand
+- Hard to debug when fails
+- Slow
+- Change in any feature breaks test
+
+**Fix:** Small, focused tests - one concept per test
+
+## Building Maintainable Tests
+
+### Good Test Characteristics (F.I.R.S.T.)
+
+**Fast**
+- Unit tests < 100ms
+- Integration tests < 1s  
+- E2E tests < 30s
+- Total suite < 10 minutes
+
+**Independent**
+- Tests don't depend on each other
+- Can run in any order
+- Can run in parallel
+
+**Repeatable**
+- Same result every time
+- No flaky tests
+- Works in any environment
+
+**Self-Validating**
+- Clear pass/fail
+- No manual verification
+- Assertions meaningful
+
+**Timely**
+- Written with code (TDD)
+- Run frequently
+- Fast feedback
+
+### Page Object Model (for UI tests)
+
+**Bad: Direct selectors everywhere**
+```javascript
+test('login', async () => {
+  await page.fill('#username', 'user@test.com');
+  await page.fill('#password', 'pass123');
+  await page.click('button[type="submit"]');
+});
+
+test('profile', async () => {
+  await page.fill('#username', 'user@test.com'); // Duplicated
+  await page.fill('#password', 'pass123');
+  await page.click('button[type="submit"]');
+  await page.click('a[href="/profile"]');
+});
+```
+
+**Good: Page Object Model**
+```javascript
+class LoginPage {
+  constructor(page) {
+    this.page = page;
+    this.usernameInput = '#username';
+    this.passwordInput = '#password';
+    this.submitButton = 'button[type="submit"]';
+  }
+  
+  async login(username, password) {
+    await this.page.fill(this.usernameInput, username);
+    await this.page.fill(this.passwordInput, password);
+    await this.page.click(this.submitButton);
+  }
+}
+
+test('login', async () => {
+  const loginPage = new LoginPage(page);
+  await loginPage.login('user@test.com', 'pass123');
+});
+
+test('profile', async () => {
+  const loginPage = new LoginPage(page);
+  await loginPage.login('user@test.com', 'pass123'); // Reusable
+  await page.click('a[href="/profile"]');
+});
+```
+
+**Benefits:**
+- Change selector once, fixes all tests
+- Reusable code
+- More readable tests
+- Easier maintenance
+
+### Good Selectors
+
+**Priority order:**
+1. **data-testid** (best for testing)
+   ```html
+   <button data-testid="checkout-btn">Checkout</button>
+   ```
+   ```javascript
+   await page.click('[data-testid="checkout-btn"]');
+   ```
+
+2. **Semantic HTML** (accessible)
+   ```javascript
+   await page.click('button[aria-label="Submit form"]');
+   await page.getByRole('button', { name: 'Submit' });
+   ```
+
+3. **Stable attributes** (id, name)
+   ```javascript
+   await page.click('#submit-button');
+   ```
+
+4. **Avoid:** CSS classes (styling changes break tests), XPath (fragile)
+
+### Test Data Management
+
+**Pattern 1: Test Data Builders**
+```javascript
+class OrderBuilder {
+  constructor() {
+    this.order = {
+      userId: 123,
+      items: [],
+      status: 'pending'
+    };
+  }
+  
+  withItems(items) {
+    this.order.items = items;
+    return this;
+  }
+  
+  withStatus(status) {
+    this.order.status = status;
+    return this;
+  }
+  
+  build() {
+    return this.order;
+  }
+}
+
+// Usage
+const order = new OrderBuilder()
+  .withItems([{id: 1, qty: 2}])
+  .withStatus('completed')
+  .build();
+```
+
+**Pattern 2: Fixtures/Factories**
+```javascript
+// fixtures.js
+export const validUser = {
+  email: 'test@example.com',
+  password: 'ValidPass123!',
+  role: 'customer'
+};
+
+export const adminUser = {
+  ...validUser,
+  role: 'admin'
+};
+
+// test
+import { validUser } from './fixtures';
+await createUser(validUser);
+```
+
+**Pattern 3: Database Seeding**
+```javascript
+beforeEach(async () => {
+  await db.seed([
+    { table: 'users', data: testUsers },
+    { table: 'orders', data: testOrders }
+  ]);
+});
+
+afterEach(async () => {
+  await db.clean(); // Reset to clean state
+});
+```
+
+## CI/CD Integration
+
+### Pipeline Structure
+
+```
+Commit → Fast Tests → Slower Tests → Deploy
+         (< 5 min)    (< 15 min)
+
+Fast Tests:
+- Unit tests
+- Linting
+- Type checking
+
+Slower Tests:
+- Integration tests
+- E2E critical paths
+- Security scans
+
+Deploy:
+- Staging (auto)
+- Production (manual or auto with feature flags)
+```
+
+### Parallel Execution
+
+**Benefits:**
+- Faster feedback
+- Better resource utilization
+
+**Strategies:**
+```javascript
+// Run test files in parallel
+// jest.config.js
+module.exports = {
+  maxWorkers: 4, // 4 parallel workers
+};
+
+// Playwright
+npx playwright test --workers=4
+
+// Split by test file
+CI Matrix:
+- Worker 1: Unit tests (auth)
+- Worker 2: Unit tests (orders)
+- Worker 3: Integration tests
+- Worker 4: E2E tests
+```
+
+### Flaky Test Management
+
+**When test fails:**
+1. Does it fail consistently? → Real bug
+2. Fails sometimes? → Flaky test
+
+**Handling flaky tests:**
+```javascript
+// WRONG: Retry until passes
+test.retry(5); // Hiding the problem
+
+// RIGHT: Fix the flakiness
+// - Add proper waits
+// - Fix race conditions  
+// - Isolate test data
+// - Improve selectors
+```
+
+**Short-term mitigation:**
+```javascript
+// Quarantine flaky tests
+test.skip('flaky test', () => {
+  // TODO: Fix flakiness (TICKET-123)
+});
+```
+
+## Measuring Automation Success
+
+### Metrics That Matter
+
+**Coverage (with context):**
+- Not just % code coverage
+- Risk coverage (high-risk areas tested?)
+- Feature coverage (user flows tested?)
+
+**Feedback Speed:**
+- Time to run full suite
+- Time to first failure
+- Time from commit to deploy
+
+**Reliability:**
+- Flaky test rate (<2% acceptable)
+- False positive rate
+- Test maintenance time
+
+**Value:**
+- Bugs caught by automation
+- Regressions prevented
+- Time saved vs manual testing
+
+### Metrics That Don't Matter Much
+
+- ❌ Number of automated tests
+- ❌ Lines of test code
+- ❌ 100% code coverage (diminishing returns)
+
+## Test Automation Strategy Document
+
+```markdown
+## Test Automation Strategy - [Project Name]
+
+### Goals
+- 80% of regression testing automated
+- Test suite runs in < 10 minutes
+- < 2% flaky test rate
+- Catch critical bugs before production
+
+### Test Pyramid Distribution
+- Unit: 70% (target: 2000 tests, <5 min)
+- Integration: 20% (target: 400 tests, <3 min)
+- E2E: 10% (target: 50 tests, <2 min)
+
+### What We Automate
+✅ Critical user flows (checkout, payment, signup)
+✅ Regression-prone areas (historical bug zones)
+✅ API contracts and integrations
+✅ Security checks (injection, XSS)
+✅ Performance benchmarks
+
+### What We Don't Automate
+❌ Visual design (manual review)
+❌ Usability testing (requires human judgment)
+❌ Exploratory testing (human investigation)
+❌ One-time migrations
+❌ Frequently changing features (manual until stable)
+
+### Tools
+- Unit: Jest
+- Integration: Supertest
+- E2E: Playwright
+- CI: GitHub Actions
+- Test Data: Factory functions + database seeding
+
+### Maintenance
+- Review flaky tests weekly
+- Update tests when features change
+- Delete tests for removed features
+- Refactor tests quarterly
+
+### Success Criteria
+- Deploy to production daily with confidence
+- Catch 90% of bugs before production
+- Test suite stays fast (<10 min)
+```
+
+## Growing Automation Gradually
+
+### Phase 1: Foundation (Month 1-2)
+- Set up CI pipeline
+- Write unit tests for new code
+- Automate smoke tests
+- Target: 40% unit test coverage of critical code
+
+### Phase 2: Core Flows (Month 3-4)
+- Automate critical happy paths (E2E)
+- Add integration tests for APIs
+- Implement page object model
+- Target: 5-10 stable E2E tests
+
+### Phase 3: Expansion (Month 5-6)
+- Increase unit test coverage to 60%
+- Add edge case integration tests
+- Automate top 10 bug-prone areas
+- Target: Fast feedback (<10 min CI run)
+
+### Phase 4: Optimization (Month 7+)
+- Parallelize test execution
+- Refactor duplicate test code
+- Fix all flaky tests
+- Target: Deploy multiple times daily
+
+## Using with QE Agents
+
+### Agent-Driven Test Pyramid
+
+**qe-test-generator** builds the pyramid intelligently:
+```typescript
+// Generate unit tests (70% of pyramid)
+await agent.generateTests({
+  level: 'unit',
+  target: 'src/services/PaymentService.ts',
+  coverage: 'comprehensive'
+});
+// → Fast, isolated tests for business logic
+
+// Generate integration tests (20% of pyramid)
+await agent.generateTests({
+  level: 'integration',
+  target: 'src/api/orders',
+  focus: 'database-interactions'
+});
+// → Tests for component integration
+
+// Generate E2E tests (10% of pyramid)
+await agent.generateTests({
+  level: 'e2e',
+  flows: ['checkout', 'payment'],
+  priority: 'critical-paths-only'
+});
+// → Minimal UI tests for happy paths
+```
+
+### Intelligent Test Selection
+
+**qe-regression-risk-analyzer** optimizes what to automate:
+```typescript
+// Analyze which tests provide most value
+const analysis = await agent.analyzeAutomationValue({
+  candidates: allManualTests,
+  criteria: ['repetition', 'stability', 'risk', 'execution-time']
+});
+
+// Returns prioritized list
+// High value: Frequent + stable + high-risk + fast
+// Low value: Rare + changing + low-risk + slow
+```
+
+### CI/CD Pipeline with Agent Coordination
+
+```yaml
+# Agents integrated into CI pipeline
+name: QE Agent Pipeline
+
+on: [push, pull_request]
+
+jobs:
+  fast-feedback:
+    runs-on: ubuntu-latest
+    steps:
+      # Unit tests with qe-test-executor
+      - name: Unit Tests
+        run: aqe agent run qe-test-executor --level unit --parallel
+        # → <5 minutes
+
+  comprehensive-check:
+    needs: fast-feedback
+    steps:
+      # Integration tests
+      - name: Integration Tests
+        run: aqe agent run qe-test-executor --level integration
+        # → <10 minutes
+
+      # Coverage analysis
+      - name: Coverage Analysis
+        run: aqe agent run qe-coverage-analyzer --threshold 80
+        # → Identifies gaps in real-time
+
+  deployment-gate:
+    needs: comprehensive-check
+    steps:
+      # E2E critical paths
+      - name: E2E Tests
+        run: aqe agent run qe-test-executor --level e2e --critical-only
+
+      # Quality gate decision
+      - name: Quality Gate
+        run: aqe agent run qe-quality-gate
+        # → GO/NO-GO decision
+```
+
+### Flaky Test Management with Agents
+
+**qe-flaky-test-hunter** identifies and fixes flaky tests:
+```typescript
+// Detect flakiness patterns
+await agent.huntFlakyTests({
+  suite: 'all',
+  runs: 100,
+  statisticalConfidence: 0.95
+});
+// → Identifies tests with <95% pass rate
+
+// Auto-stabilize common patterns
+await agent.stabilizeTests({
+  pattern: 'race-conditions',
+  fix: 'add-explicit-waits'
+});
+// → Converts sleep() to waitFor()
+
+// Quarantine unfixable tests
+await agent.quarantine({
+  flakiness: '>10%',
+  action: 'skip-and-ticket'
+});
+```
+
+### Agent-Assisted Test Data Strategy
+
+**qe-test-data-architect** generates test data:
+```typescript
+// Generate isolated test data per test
+const testData = await agent.generateTestData({
+  schema: 'users',
+  count: 100,
+  realistic: true,
+  isolation: 'per-test'  // Each test gets unique data
+});
+
+// Generate edge cases
+const edgeCases = await agent.generateEdgeCases({
+  field: 'email',
+  patterns: ['special-chars', 'unicode', 'max-length']
+});
+```
+
+### Fleet Coordination for Test Automation
+
+```typescript
+// Coordinate multiple agents for automation strategy
+const automationFleet = await FleetManager.coordinate({
+  strategy: 'test-automation',
+  agents: [
+    'qe-test-generator',           // Generate tests
+    'qe-test-executor',            // Execute in CI
+    'qe-coverage-analyzer',        // Analyze gaps
+    'qe-flaky-test-hunter',        // Fix flakiness
+    'qe-regression-risk-analyzer'  // Optimize selection
+  ],
+  topology: 'sequential'
+});
+
+// Execute full automation workflow
+await automationFleet.execute({
+  scope: 'payment-module',
+  pyramidBalance: { unit: 0.7, integration: 0.2, e2e: 0.1 }
+});
+```
+
+### Strategic Automation Decisions with Agent Insights
+
+```typescript
+// Agent helps decide what to automate
+const recommendation = await qe-quality-analyzer.recommendAutomation({
+  feature: 'checkout-flow',
+  currentCoverage: 'manual-only',
+  executionFrequency: 'daily',
+  stability: 'stable',
+  risk: 'high'
+});
+
+// Returns:
+// {
+//   shouldAutomate: true,
+//   level: 'integration',  // Skip E2E, test via API
+//   priority: 'high',
+//   estimatedROI: '15 hours saved/month',
+//   maintainanceCost: 'low'
+// }
+```
+
+---
+
+## Related Skills
+
+**Core Quality Practices:**
+- [agentic-quality-engineering](../agentic-quality-engineering/) - Agent orchestration for automation
+- [holistic-testing-pact](../holistic-testing-pact/) - Automation within whole-team quality
+
+**Development Practices:**
+- [tdd-london-chicago](../tdd-london-chicago/) - TDD drives automation at unit level
+- [xp-practices](../xp-practices/) - CI/CD integration with automation
+
+**Testing Specializations:**
+- [api-testing-patterns](../api-testing-patterns/) - API automation strategies
+- [performance-testing](../performance-testing/) - Performance automation
+- [security-testing](../security-testing/) - Security scan automation
+- [exploratory-testing-advanced](../exploratory-testing-advanced/) - Balance with manual exploration
+
+**Communication:**
+- [quality-metrics](../quality-metrics/) - Measure automation effectiveness
+
+---
+
+## Remember
+
+**Automation is a means, not an end.**
+
+The goal is confident, frequent deployments of high-quality software. Automation enables that by:
+- Giving fast feedback
+- Catching regressions
+- Freeing humans for skilled testing
+
+Don't automate for automation's sake. Automate strategically.
+
+**Good automation amplifies good testing. Bad automation wastes everyone's time.**
+
+**With Agents**: Agents excel at maintaining the test pyramid, detecting flakiness, optimizing test selection, and coordinating CI/CD pipelines. Use agents to automate the automation strategy itself.