testdriverai 7.1.4 → 7.2.0

package/docs/v7/features/test-analytics.mdx

@@ -0,0 +1,373 @@
+---
+title: "Test Analytics"
+description: "Visualize test trends, success rates, and performance metrics over time"
+icon: "chart-line"
+---
+
+Track test performance and trends with comprehensive analytics dashboards. Monitor pass rates, identify flaky tests, and optimize test suite performance.
+
+## Analytics Dashboard
+
+Access detailed analytics at [app.testdriver.ai/analytics](https://app.testdriver.ai/analytics):
+
+<Card title="Test Analytics" icon="chart-simple" href="https://app.testdriver.ai/analytics">
+  Visualize test trends, success rates, and performance metrics
+</Card>
+
+### Key Metrics
+
+<CardGroup cols={2}>
+  <Card title="Pass Rate" icon="percent" color="green">
+    Overall success rate across all tests
+    - Current: 45%
+    - Trend over time
+    - Target threshold alerts
+  </Card>
+
+  <Card title="Unique Failing Tests" icon="triangle-exclamation" color="yellow">
+    Tests that failed at least once
+    - Count: 21 unique tests
+    - Flaky test detection
+    - Failure patterns
+  </Card>
+
+  <Card title="Average Duration" icon="clock" color="blue">
+    Mean test execution time
+    - Current: 1m 57s
+    - Performance trends
+    - Slowest tests identified
+  </Card>
+
+  <Card title="Total Tests" icon="list-check" color="purple">
+    All test cases tracked
+    - Count: 111 tests
+    - Growth over time
+    - Active vs deprecated
+  </Card>
+
+  <Card title="Total Duration" icon="stopwatch" color="cyan">
+    Cumulative execution time
+    - Current: 3h 37m
+    - Cost implications
+    - Optimization opportunities
+  </Card>
+</CardGroup>
+
+## Pass Rate Over Time
+
+Track test stability trends:
+
+```
+Pass Rate Trend Chart:
+- X-axis: Date (Dec 2 - Dec 9)
+- Y-axis: Pass percentage (0% - 100%)
+- Green line showing pass rate progression
+- Hover to see exact values per day
+```
+
+**Insights:**
+- Identify when test stability declined
+- Correlate with code changes
+- Set pass rate goals (e.g., >95%)
+- Alert on threshold breaches
+
+## Test Results Over Time
+
+Monitor test execution patterns:
+
+```
+Stacked Bar Chart:
+- X-axis: Date
+- Y-axis: Number of tests
+- Green: Passed tests
+- Red: Failed tests
+- Orange: Skipped tests
+```
+
+**Use Cases:**
+- Spot unusual failure spikes
+- Track test suite growth
+- Identify testing gaps
+- Validate CI/CD health
+
+## Status Distribution
+
+Understand test outcomes at a glance:
+
+```
+Pie/Bar Chart:
+- Passed: 50 tests (45%)
+- Failed: 40 tests (36%)
+- Skipped: 21 tests (19%)
+```
+
+**Actionable Data:**
+- High skip rate may indicate incomplete features
+- Persistent failures need investigation
+- Pass rate benchmarks per team
+
+## Platform Distribution
+
+See where tests run:
+
+```
+Donut Chart:
+- Windows: 40%
+- Linux: 35%
+- macOS: 25%
+```
+
+**Cross-Platform Testing:**
+- Ensure coverage across all platforms
+- Identify platform-specific issues
+- Optimize resource allocation
+
+## Filtering & Time Periods
+
+Customize analytics views:
+
+<Tabs>
+  <Tab title="Time Period">
+    ```
+    Last 24 hours
+    Last week (default)
+    Last month
+    Last quarter
+    Custom date range
+    ```
+  </Tab>
+
+  <Tab title="Branch">
+    ```
+    main (default)
+    develop
+    All branches
+    Specific branch
+    ```
+  </Tab>
+
+  <Tab title="Repository">
+    ```
+    All repositories
+    frontend
+    backend
+    mobile
+    ```
+  </Tab>
+
+  <Tab title="Suite">
+    ```
+    All test suites
+    Specific test file
+    Test directory
+    ```
+  </Tab>
+</Tabs>
+
+## Flaky Test Detection
+
+Identify unreliable tests automatically:
+
+<AccordionGroup>
+  <Accordion title="What are Flaky Tests?">
+    Tests that intermittently pass and fail without code changes.
+    
+    **Common Causes:**
+    - Race conditions
+    - Timing issues
+    - External dependencies
+    - Non-deterministic behavior
+    - Network instability
+  </Accordion>
+
+  <Accordion title="Detection Algorithm">
+    TestDriver identifies flaky tests by analyzing:
+    
+    1. **Pass/Fail History** - Tests with alternating results
+    2. **Failure Rate** - Between 1% and 99% (not always passing/failing)
+    3. **Time Window** - Over last 30 days or 100 runs
+    4. **Confidence Score** - Statistical significance
+  </Accordion>
+
+  <Accordion title="Flaky Test Report">
+    Dashboard shows:
+    - Test name and file
+    - Flakiness score (0-100%)
+    - Recent pass/fail pattern
+    - Suggested fixes
+    - Link to failed runs
+  </Accordion>
+</AccordionGroup>
+
+## Performance Trends
+
+Monitor test execution speed:
+
+```
+Duration Trend Chart:
+- X-axis: Date
+- Y-axis: Duration (seconds/minutes)
+- Line showing average test duration
+- Highlight performance improvements
+```
+
+**Optimization Tracking:**
+- Measure impact of caching
+- Identify slow tests
+- Track infrastructure changes
+- Validate parallelization benefits
+
+## Slowest Tests
+
+Find performance bottlenecks:
+
+```
+Top 10 Slowest Tests:
+1. integration/checkout.test.js - 3m 45s
+2. e2e/full-workflow.test.js - 2m 58s
+3. api/stress-test.test.js - 2m 12s
+...
+```
+
+**Action Items:**
+- Optimize slow tests first (80/20 rule)
+- Break down into smaller tests
+- Add caching strategies
+- Parallelize where possible
+
+## Export Analytics
+
+Download analytics data:
+
+<Tabs>
+  <Tab title="CSV">
+    ```csv
+    Date,Pass Rate,Total Tests,Passed,Failed,Skipped,Avg Duration
+    2024-12-09,45%,111,50,40,21,1m57s
+    2024-12-08,52%,108,56,35,17,1m45s
+    2024-12-07,48%,105,50,38,17,2m03s
+    ```
+  </Tab>
+
+  <Tab title="JSON">
+    ```json
+    {
+      "analytics": {
+        "passRate": 0.45,
+        "totalTests": 111,
+        "passed": 50,
+        "failed": 40,
+        "skipped": 21,
+        "avgDuration": "1m57s",
+        "trends": [
+          {
+            "date": "2024-12-09",
+            "passRate": 0.45,
+            "totalTests": 111
+          }
+        ]
+      }
+    }
+    ```
+  </Tab>
+
+  <Tab title="PDF Report">
+    Generate executive summary with:
+    - Key metrics overview
+    - Trend visualizations
+    - Flaky test list
+    - Recommendations
+    - Cost analysis
+  </Tab>
+</Tabs>
+
+## Custom Dashboards
+
+Create team-specific views:
+
+```javascript
+// Use API to build custom dashboards
+const analytics = await fetch('https://api.testdriver.ai/v1/analytics', {
+  headers: {
+    'Authorization': `Bearer ${TD_API_KEY}`,
+  },
+  body: JSON.stringify({
+    timeRange: 'last-week',
+    branch: 'main',
+    groupBy: 'day'
+  })
+});
+
+const data = await analytics.json();
+// Build custom visualizations with your preferred tool
+```
+
+## Alerts & Notifications
+
+Get notified of critical changes:
+
+<CardGroup cols={2}>
+  <Card title="Pass Rate Alert" icon="bell">
+    Notify when pass rate drops below threshold
+    - Email notification
+    - Slack/Teams message
+    - Configurable threshold (e.g., <90%)
+  </Card>
+
+  <Card title="Flaky Test Alert" icon="triangle-exclamation">
+    Alert on newly detected flaky tests
+    - Daily digest
+    - Per-test notifications
+    - Trend reports
+  </Card>
+
+  <Card title="Performance Alert" icon="gauge">
+    Warn when tests slow down
+    - 20%+ duration increase
+    - Slowest test changes
+    - Cost impact
+  </Card>
+
+  <Card title="Failure Spike Alert" icon="chart-line-up">
+    Detect unusual failure patterns
+    - 2x normal failure rate
+    - Multiple test failures
+    - Same-day correlation
+  </Card>
+</CardGroup>
+
+## Learn More
+
+<CardGroup cols={2}>
+  <Card
+    title="Test Reports"
+    icon="list-check"
+    href="/v7/features/test-reports"
+  >
+    View individual test runs
+  </Card>
+
+  <Card
+    title="Test Cases"
+    icon="file-lines"
+    href="/v7/features/test-cases"
+  >
+    Detailed test case history
+  </Card>
+
+  <Card
+    title="Flake Prevention"
+    icon="shield-check"
+    href="/v7/features/stable"
+  >
+    Learn about anti-flake technology
+  </Card>
+
+  <Card
+    title="Performance"
+    icon="gauge-high"
+    href="/v7/features/fast"
+  >
+    Optimize test execution
+  </Card>
+</CardGroup>