qa-skills 3.0.0

package/skills/qa-flaky-detector/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: qa-flaky-detector
+description: Analyze CI history and test execution data to identify flaky tests using 4-pattern classification -- race conditions, shared state, time-dependency, external dependencies -- with suggested fixes.
+output_dir: reports/flaky
+dependencies:
+  recommended:
+    - qa-test-healer
+---
+
+# QA Flaky Detector
+
+## Purpose
+
+Identify and classify flaky tests from CI history and test execution data. Flaky tests pass and fail intermittently for the same code, wasting CI time and masking real failures. This skill detects flakiness, classifies root causes into four patterns, and suggests targeted fixes.
+
+## Trigger Phrases
+
+- "Find flaky tests" / "Identify flaky tests"
+- "Analyze CI history for flakiness" / "Flaky test report"
+- "Why does this test fail sometimes?" / "Intermittent test failures"
+- "Classify flaky test [name]" / "Flaky test patterns"
+- "Suggested fixes for flaky tests" / "Fix flaky [test name]"
+
+## 4-Pattern Classification
+
+| Pattern | Description | Typical Causes | Suggested Fixes |
+|---------|-------------|----------------|-----------------|
+| **1. Race conditions** | Async timing, DOM not ready, API response order | Missing awaits, no explicit waits, parallel execution | Add explicit waits, `await` chains, `waitForSelector` |
+| **2. Shared state** | Test order dependency, global state mutation, DB leftover | Global vars, singletons, uncleared DB/cache | Isolate tests, `beforeEach` cleanup, transactional rollback |
+| **3. Time-dependency** | Timezone, date/time mocking, daylight saving | `new Date()`, `Date.now()`, hardcoded dates | Mock time (Jest `useFakeTimers`, `freezegun`), use fixed dates |
+| **4. External dependencies** | Network calls, third-party APIs, file system | Live HTTP, external services, temp files | Mock/stub APIs, use fixtures, deterministic file paths |
+
+See `references/flaky-patterns.md` for detailed patterns with code examples and fixes.
+
+## Detection Methods
+
+### 1. CI History Analysis
+
+- Same test, different results across runs (pass/fail flip)
+- Compare JUnit XML, Allure, or CI job logs over N runs
+- Threshold: pass rate < 100% over 10+ runs → flag as flaky
+
+### 2. Statistical Analysis
+
+- Pass rate per test: `passes / (passes + failures)` over recent runs
+- Confidence: more runs → higher confidence in flakiness
+- Minimum runs: recommend 10+ for reliable detection
+
+### 3. Pattern Matching (Code-Based)
+
+- Scan test code for known flaky signatures:
+  - No `await` on async operations
+  - `setTimeout`/`setInterval` without cleanup
+  - `new Date()` or `Date.now()` without mocking
+  - Direct HTTP calls, file I/O
+  - Shared globals, singletons
+  - Missing `beforeEach`/`afterEach` cleanup
+
+See `references/ci-analysis.md` for CI history analysis methods.
+
+## Workflow
+
+1. **Input** — CI artifacts (JUnit XML, Allure, GitHub Actions logs), test execution history, or test file paths
+2. **Collect** — Parse results from N runs; build pass/fail matrix per test
+3. **Detect** — Identify tests with pass rate < 100%
+4. **Classify** — Map each flaky test to one or more of the 4 patterns (code scan + heuristics)
+5. **Suggest** — Generate fix recommendations per pattern
+6. **Output** — Flaky test report, prioritized fix list
+
+## Output Deliverables
+
+### Flaky Test Report
+
+```markdown
+# Flaky Test Report — [Project/Branch]
+
+## Summary
+| Test | Failure Rate | Runs | Classification | Priority |
+|------|--------------|------|----------------|----------|
+| auth.login.spec.ts:42 | 23% | 26 | Race condition | High |
+| checkout.flow.spec.ts:15 | 12% | 18 | Shared state | High |
+| utils.date.spec.ts:8 | 8% | 12 | Time-dependency | Medium |
+
+## Detailed Findings
+
+### auth.login.spec.ts:42 — "should redirect after login"
+- **Failure rate:** 6/26 (23%)
+- **Classification:** Race condition
+- **Likely cause:** DOM not ready before assertion
+- **Suggested fix:** Add `await page.waitForSelector('#dashboard')` before assertion
+```
+
+### Prioritized Fix List
+
+1. **High** — High failure rate, critical path, or easy fix
+2. **Medium** — Moderate rate, non-critical
+3. **Low** — Low rate, edge cases
+
+## Integration with Other Skills
+
+| Need | Skill | Usage |
+|------|-------|-------|
+| Parse JUnit/Allure | qa-test-reporter | Aggregate results from multiple runs |
+| Fix broken selectors | qa-test-healer | Apply suggested waits/selectors |
+| Create fix tasks | qa-task-creator | Generate tasks for flaky test fixes |
+| Test strategy | qa-test-strategy | Document flaky mitigation in strategy |
+
+## Scope
+
+**Can do (autonomous):**
+- Parse CI/test results (JUnit XML, Allure, common formats)
+- Compute pass rate per test over N runs
+- Classify flaky tests into 4 patterns
+- Scan test code for flaky signatures
+- Produce flaky report and prioritized fix list
+- Suggest fixes per pattern (from reference patterns)
+
+**Cannot do (requires confirmation):**
+- Modify test code directly (suggest only; qa-test-healer can apply)
+- Access private CI systems without credentials
+- Override classification without evidence
+
+**Will not do (out of scope):**
+- Execute tests or run CI
+- Deploy or change production
+- Guarantee fix effectiveness (suggestions are heuristic-based)
+
+## Quality Checklist
+
+- [ ] Pass rate calculated over sufficient runs (≥10 recommended)
+- [ ] Each flaky test classified into at least one pattern
+- [ ] Suggested fixes match pattern and reference examples
+- [ ] Report includes test name, file:line, failure rate, classification
+- [ ] Prioritized fix list ordered by impact
+- [ ] No hardcoded credentials; CI access from user/env
+
+## Troubleshooting
+
+| Symptom | Likely Cause | Fix |
+|---------|--------------|-----|
+| No flaky tests found | Too few runs, or tests truly stable | Increase run count; verify CI artifacts parsed |
+| All tests flagged | Threshold too strict | Raise pass-rate threshold; exclude known-broken tests |
+| Wrong classification | Heuristics insufficient | Review code manually; add custom pattern to references |
+| Missing CI data | Format not supported | Check qa-test-reporter; add parser for format |
+| Fix doesn't work | Root cause different | Re-classify; try alternative pattern fixes |
+| Pass rate 0% | Test always fails | Exclude from flaky report; treat as broken |
+
+## Reference Files
+
+| Topic | Reference |
+|-------|-----------|
+| Flaky patterns with code examples and fixes | `references/flaky-patterns.md` |
+| CI history analysis methods | `references/ci-analysis.md` |

package/skills/qa-flaky-detector/references/ci-analysis.md

@@ -0,0 +1,140 @@
+# CI History Analysis for Flaky Detection
+
+## Overview
+
+Flaky tests are identified by analyzing test execution history across multiple CI runs. Same test, different results (pass/fail flip) indicates flakiness.
+
+---
+
+## 1. Data Sources
+
+### JUnit XML
+
+- **Location:** `test-results/*.xml`, `junit.xml`, `report.xml`
+- **Parse:** `testsuites` → `testsuite` → `testcase`; attribute `name`, status (pass/fail)
+- **Aggregate:** Collect XML from N runs (e.g., last 20 builds)
+
+### Allure
+
+- **Location:** `allure-results/`, `allure-report/`
+- **Parse:** `*-result.json` or `*-container.json`; extract test name, status, history
+- **History:** Allure can store history across runs for trend analysis
+
+### GitHub Actions
+
+- **Artifacts:** Test results uploaded as artifacts
+- **Logs:** Parse job logs for test output (e.g., Jest, pytest summary)
+- **API:** `GET /repos/{owner}/{repo}/actions/runs` to list runs; fetch artifacts
+
+### Other CI Systems
+
+- **GitLab CI:** JUnit reports, `junit: report.xml`
+- **Jenkins:** JUnit plugin, `**/junit/*.xml`
+- **CircleCI:** Store test results, `store_test_results`
+- **Azure Pipelines:** Publish test results task
+
+---
+
+## 2. Pass Rate Calculation
+
+### Formula
+
+```
+pass_rate = passes / (passes + failures)
+```
+
+- **Flaky threshold:** `pass_rate < 1.0` and `pass_rate > 0` (i.e., at least one pass and one fail)
+- **Always failing:** `pass_rate = 0` → exclude from flaky report (treat as broken)
+- **Always passing:** `pass_rate = 1.0` → stable, not flaky
+
+### Minimum Runs
+
+- **Recommended:** 10+ runs for reliable detection
+- **Low runs:** High variance; flag as "uncertain" or "needs more data"
+
+### Example
+
+| Test | Run 1 | Run 2 | Run 3 | ... | Run 10 | Pass Rate |
+|------|-------|-------|-------|-----|--------|------------|
+| login.spec.ts:42 | P | F | P | ... | P | 7/10 = 70% |
+
+---
+
+## 3. Aggregation Workflow
+
+1. **Collect** — Fetch test results from last N CI runs (e.g., 20)
+2. **Normalize** — Map test names to canonical form (file:line or full name)
+3. **Build matrix** — Rows = tests, columns = runs, cells = pass/fail
+4. **Compute** — Pass rate per test
+5. **Filter** — Keep tests with `0 < pass_rate < 1`
+6. **Sort** — By failure rate (high first) or by critical path
+
+---
+
+## 4. Test Name Normalization
+
+Different frameworks report names differently:
+
+| Framework | Example | Normalized |
+|-----------|---------|------------|
+| Jest | `auth › login › should redirect` | `auth/login.spec.ts` or full name |
+| pytest | `tests/test_auth.py::test_login` | `tests/test_auth.py::test_login` |
+| Playwright | `auth/login.spec.ts:42` | `auth/login.spec.ts:42` |
+| Mocha | `Login should redirect` | Match by file + describe + it |
+
+Use consistent normalization to merge results across runs.
+
+---
+
+## 5. CI-Specific Parsing
+
+### GitHub Actions
+
+```yaml
+# In workflow, ensure test results are published
+- uses: actions/upload-artifact@v4
+  with:
+    name: test-results
+    path: test-results/
+```
+
+Parse artifact `test-results/*.xml` from each run.
+
+### GitLab CI
+
+```yaml
+junit: report.xml
+```
+
+Download `report.xml` from job artifacts via API.
+
+### Local Script
+
+```bash
+# Collect last N runs (conceptual)
+for run in $(gh run list --limit 20 --json databaseId -q '.[].databaseId'); do
+  gh run download $run -n test-results -D results/run-$run
+done
+```
+
+---
+
+## 6. Output for qa-flaky-detector
+
+Feed aggregated data into the skill:
+
+```json
+{
+  "runs": 20,
+  "tests": [
+    {
+      "name": "auth/login.spec.ts:42",
+      "passes": 14,
+      "failures": 6,
+      "pass_rate": 0.7
+    }
+  ]
+}
+```
+
+The skill then classifies each flaky test and suggests fixes.

package/skills/qa-flaky-detector/references/flaky-patterns.md

@@ -0,0 +1,247 @@
+# Flaky Test Patterns — Code Examples and Fixes
+
+## Overview
+
+Flaky tests pass and fail intermittently. This reference documents four root-cause patterns with code examples and recommended fixes.
+
+---
+
+## 1. Race Conditions
+
+### Description
+
+Tests fail when async operations complete in unpredictable order or when the UI/API is not ready before assertions.
+
+### Signatures in Code
+
+- `click()` without `waitFor` before assertion
+- `fetch()` / `axios` without `await`
+- `setTimeout` with fixed delay (fragile)
+- Parallel `Promise.all` with order-dependent assertions
+- No `waitForSelector` / `waitForLoadState` before DOM checks
+
+### Example (Playwright — Flaky)
+
+```typescript
+test('should show dashboard after login', async ({ page }) => {
+  await page.fill('#email', 'user@test.com');
+  await page.fill('#password', 'secret');
+  await page.click('button[type="submit"]');
+  // Race: redirect may not have completed
+  expect(await page.textContent('h1')).toBe('Dashboard'); // Fails sometimes
+});
+```
+
+### Fix
+
+```typescript
+test('should show dashboard after login', async ({ page }) => {
+  await page.fill('#email', 'user@test.com');
+  await page.fill('#password', 'secret');
+  await page.click('button[type="submit"]');
+  await page.waitForSelector('h1:has-text("Dashboard")', { timeout: 5000 });
+  expect(await page.textContent('h1')).toBe('Dashboard');
+});
+```
+
+### Example (Jest — Flaky)
+
+```typescript
+it('fetches user', async () => {
+  fetchUser().then(data => expect(data.name).toBe('Alice')); // No await
+});
+```
+
+### Fix
+
+```typescript
+it('fetches user', async () => {
+  const data = await fetchUser();
+  expect(data.name).toBe('Alice');
+});
+```
+
+---
+
+## 2. Shared State
+
+### Description
+
+Tests affect each other via global state, singletons, database, or file system. Order of execution changes outcomes.
+
+### Signatures in Code
+
+- Global variables, module-level caches
+- Singleton instances
+- Database records not cleaned between tests
+- File system writes (temp files, config)
+- Browser `localStorage` / `sessionStorage` not cleared
+
+### Example (Flaky)
+
+```typescript
+let currentUser; // Shared across tests
+
+beforeEach(() => {
+  currentUser = null; // Missing in some suites
+});
+
+it('sets user', () => {
+  currentUser = { id: 1, name: 'Alice' };
+  expect(getUserName()).toBe('Alice');
+});
+
+it('gets user', () => {
+  expect(getUserName()).toBeNull(); // Fails if previous test ran first
+});
+```
+
+### Fix
+
+```typescript
+beforeEach(() => {
+  currentUser = null;
+  // Or: reset module, use dependency injection
+});
+```
+
+### Example (DB — Flaky)
+
+```python
+def test_create_order():
+    order = create_order(user_id=1, total=100)
+    assert order.id is not None
+
+def test_list_orders():
+    orders = list_orders(user_id=1)
+    assert len(orders) == 0  # Fails if test_create_order ran first
+```
+
+### Fix
+
+```python
+@pytest.fixture(autouse=True)
+def clean_orders(db):
+    yield
+    db.execute("DELETE FROM orders WHERE user_id = 1")
+```
+
+---
+
+## 3. Time-Dependency
+
+### Description
+
+Tests depend on current date/time, timezone, or system clock. Failures occur at certain times or in different environments.
+
+### Signatures in Code
+
+- `new Date()`, `Date.now()`
+- `new Date().toISOString()`
+- Timezone-dependent logic
+- `setTimeout` with real delays
+- Daylight saving transitions
+
+### Example (Flaky)
+
+```typescript
+it('formats expiry date', () => {
+  const card = { expiry: '12/25' };
+  expect(formatExpiry(card)).toContain('2025'); // Assumes current year
+});
+```
+
+### Fix
+
+```typescript
+beforeEach(() => {
+  jest.useFakeTimers();
+  jest.setSystemTime(new Date('2024-06-15'));
+});
+afterEach(() => jest.useRealTimers());
+```
+
+### Example (Python — Flaky)
+
+```python
+def test_is_expired():
+    obj = create_obj(expires_at=datetime.now() - timedelta(days=1))
+    assert obj.is_expired()  # Fails near midnight
+```
+
+### Fix
+
+```python
+from freezegun import freeze_time
+
+@freeze_time('2024-06-15 12:00:00')
+def test_is_expired():
+    obj = create_obj(expires_at=datetime(2024, 6, 14))
+    assert obj.is_expired()
+```
+
+---
+
+## 4. External Dependencies
+
+### Description
+
+Tests call real networks, APIs, file system, or third-party services. Variability in response time, availability, or data causes flakiness.
+
+### Signatures in Code
+
+- `fetch()`, `axios`, `http.get` without mocking
+- File system reads/writes (`fs`, `path`)
+- Environment variables that change by environment
+- Third-party SDKs (Stripe, SendGrid, etc.)
+
+### Example (Flaky)
+
+```typescript
+it('fetches weather', async () => {
+  const res = await fetch('https://api.weather.com/current');
+  const data = await res.json();
+  expect(data.temp).toBeGreaterThan(-50); // Network may fail or timeout
+});
+```
+
+### Fix
+
+```typescript
+it('fetches weather', async () => {
+  const res = await fetch('/api/weather'); // Mocked in test env
+  const data = await res.json();
+  expect(data.temp).toBe(22); // Deterministic mock response
+});
+// Or: use MSW, nock, or jest.mock
+```
+
+### Example (File System — Flaky)
+
+```python
+def test_read_config():
+    with open('/tmp/app-config.json') as f:  # May not exist or differ
+        config = json.load(f)
+    assert config['env'] == 'test'
+```
+
+### Fix
+
+```python
+def test_read_config(tmp_path):
+    config_file = tmp_path / 'config.json'
+    config_file.write_text('{"env": "test"}')
+    config = read_config(config_file)
+    assert config['env'] == 'test'
+```
+
+---
+
+## Quick Reference
+
+| Pattern | Detection | Fix Strategy |
+|---------|-----------|--------------|
+| Race condition | Async without await, no explicit wait | Add `await`, `waitForSelector`, `waitFor` |
+| Shared state | Globals, DB, files | `beforeEach` cleanup, isolate, transactional rollback |
+| Time-dependency | `Date`, `setTimeout` | `useFakeTimers`, `freezegun`, fixed dates |
+| External deps | HTTP, files, env | Mock, stub, fixtures, `tmp_path` |