testdriverai 7.1.0 → 7.1.2

package/docs/v7/getting-started/running-and-debugging.mdx

@@ -0,0 +1,511 @@
+---
+title: "Running & Debugging Tests"
+description: "Run tests with Vitest and debug failures with Dashcam replays"
+icon: "play"
+---
+
+Learn how to run TestDriver tests efficiently and debug failures using video replays, logs, and error analysis.
+
+## Running Tests
+
+TestDriver works with Vitest's powerful test runner:
+
+<Tabs>
+  <Tab title="Run All Tests">
+    ```bash
+    # Run all tests once
+    npx vitest run
+
+    # Run in watch mode
+    npx vitest
+
+    # Run with coverage
+    npx vitest run --coverage
+    ```
+  </Tab>
+
+  <Tab title="Run Specific Tests">
+    ```bash
+    # Run specific file
+    npx vitest run login.test.js
+
+    # Run multiple files
+    npx vitest run login.test.js checkout.test.js
+
+    # Run tests matching pattern
+    npx vitest run --grep "login"
+
+    # Run tests in specific folder
+    npx vitest run tests/e2e/
+    ```
+  </Tab>
+
+  <Tab title="Watch Mode">
+    ```bash
+    # Watch mode - reruns on file changes
+    npx vitest
+
+    # Watch only changed tests
+    npx vitest --changed
+
+    # Watch specific file
+    npx vitest login.test.js
+    ```
+  </Tab>
+
+  <Tab title="Parallel Execution">
+    ```bash
+    # Control concurrency
+    npx vitest run --maxConcurrency=5
+
+    # Run with specific thread count
+    npx vitest run --threads --minThreads=2 --maxThreads=8
+    ```
+  </Tab>
+</Tabs>
+
+## Test Output
+
+Understanding test output:
+
+```bash
+$ npx vitest run
+
+ ✓ login.test.js (3) 23.4s
+   ✓ user can login with valid credentials 12.3s
+   ✓ shows error for invalid email 5.6s
+   ✓ shows error for wrong password 5.5s
+
+ ✓ checkout.test.js (2) 34.7s
+   ✓ user can complete checkout 28.9s
+   ✓ validates credit card format 5.8s
+
+Test Files  2 passed (2)
+     Tests  5 passed (5)
+  Duration  58.12s
+
+📹 Dashcam Replays:
+  - https://console.testdriver.ai/dashcam/abc123
+  - https://console.testdriver.ai/dashcam/def456
+```
+
+<Tip>
+  Click the Dashcam URLs to watch video replays of your tests!
+</Tip>
+
+## Debugging with Dashcam
+
+Every test automatically records a video replay:
+
+<Steps>
+  <Step title="Get Replay URL">
+    ```javascript
+    test('my test', async (context) => {
+      const { testdriver, dashcam } = await chrome(context, { url });
+
+      // Your test code
+
+      console.log('Replay:', dashcam.url);
+    });
+    ```
+
+    Or find URLs in test output.
+  </Step>
+
+  <Step title="Open Replay">
+    Visit [console.testdriver.ai](https://console.testdriver.ai) or click the URL directly.
+  </Step>
+
+  <Step title="Analyze Replay">
+    Dashcam shows:
+    - Full video of test execution
+    - Timeline of all actions
+    - Console logs synchronized
+    - Network requests
+    - Element highlights
+    - Screenshots at key moments
+  </Step>
+</Steps>
+
+<Card title="Dashcam Features" icon="video" href="/v7/features/observable">
+  - Play/pause, speed control
+  - Jump to any action via timeline
+  - Frame-by-frame stepping
+  - Download as MP4
+  - Share with team
+  - View network waterfall
+  - See console logs
+
+  [Complete observability guide](/v7/features/observable)
+</Card>
+
+## Understanding Failures
+
+When a test fails, TestDriver provides rich debugging information:
+
+```javascript
+try {
+  await testdriver.find('submit button').click();
+} catch (error) {
+  console.log('Error:', error.message);
+  console.log('Type:', error.name); // 'ElementNotFoundError'
+  console.log('Similarity:', error.similarity); // 0.72 (how close it got)
+  console.log('Screenshot:', error.debugScreenshot); // Base64 image
+  console.log('Cache info:', error.cacheInfo);
+}
+```
+
+### Typical Error Output
+
+```
+ElementNotFoundError: Could not find 'submit button'
+
+Debug information:
+- Similarity: 0.72 (partial match found)
+- Timeout: 30000ms
+- Retry attempts: 60
+- Cache hit: false
+
+Screenshot saved: .testdriver/debug-screenshots/test-001.png
+Dashcam: https://console.testdriver.ai/dashcam/abc123
+
+Recent actions:
+  1. ✓ navigate to https://example.com (2.1s)
+  2. ✓ find('email input') (1.3s)
+  3. ✓ type('user@example.com') (0.8s)
+  4. ✗ find('submit button') (30.0s) FAILED
+
+Suggestions:
+- Try more specific selector: 'blue submit button in bottom right'
+- Check if button is visible on screen
+- Verify button loaded (check network tab)
+- View Dashcam replay to see what happened
+```
+
+## Common Debugging Scenarios
+
+<AccordionGroup>
+  <Accordion title="Element Not Found">
+    **Error:** `ElementNotFoundError: Could not find 'button'`
+
+    **Debug steps:**
+
+    1. **Watch Dashcam replay** - See what's on screen
+    2. **Check similarity score:**
+       - `> 0.8` - Close match, be more specific
+       - `0.5-0.8` - Partial match, adjust selector
+       - `< 0.5` - No match, element might not be present
+
+    3. **Try variations:**
+       ```javascript
+       const selectors = [
+         'submit button',
+         'blue submit button',
+         'button in bottom right',
+         'primary action button'
+       ];
+
+       for (const selector of selectors) {
+         try {
+           await testdriver.find(selector).click();
+           break;
+         } catch (error) {
+           console.log(`Failed: ${selector} (${error.similarity})`);
+         }
+       }
+       ```
+
+    4. **Check timing** - Element might load later:
+       ```javascript
+       await testdriver.find('submit button', { timeout: 60000 });
+       ```
+  </Accordion>
+
+  <Accordion title="Assertion Failures">
+    **Error:** `AssertionError: 'welcome message is visible' failed`
+
+    **Debug steps:**
+
+    1. **Watch Dashcam** - Verify what's actually displayed
+    2. **Simplify assertion:**
+       ```javascript
+       // Instead of:
+       await testdriver.assert('welcome message says "Hello John"');
+
+       // Try:
+       await testdriver.assert('welcome message is visible');
+       ```
+
+    3. **Check timing:**
+       ```javascript
+       // Wait longer for async content
+       await testdriver.assert('message is visible', { timeout: 60000 });
+       ```
+
+    4. **Use find() instead:**
+       ```javascript
+       const element = await testdriver.find('welcome message');
+       console.log('Message text:', element.text);
+       ```
+  </Accordion>
+
+  <Accordion title="Test Timeout">
+    **Error:** `Test timed out after 120000ms`
+
+    **Debug steps:**
+
+    1. **Increase timeout:**
+       ```javascript vitest.config.mjs
+       export default defineConfig({
+         test: {
+           testTimeout: 180000, // 3 minutes
+         },
+       });
+       ```
+
+    2. **Check sandbox logs** - Look for network issues
+    3. **Watch Dashcam** - See where test got stuck
+    4. **Add checkpoints:**
+       ```javascript
+       await testdriver.find('button').click();
+       console.log('Clicked button');
+
+       await testdriver.assert('page loaded');
+       console.log('Page loaded');
+       ```
+  </Accordion>
+
+  <Accordion title="Flaky Tests">
+    **Issue:** Test passes sometimes, fails other times
+
+    **Solutions:**
+
+    1. **Check Dashcam replays** - Compare passing vs failing runs
+    2. **Increase stability delay:**
+       ```javascript
+       await testdriver.find('element', {
+         stabilityDelay: 500 // Wait 500ms for stability
+       });
+       ```
+
+    3. **Wait for network:**
+       ```javascript
+       await testdriver.find('button').click();
+       // TestDriver automatically waits for network requests
+       await testdriver.assert('content loaded');
+       ```
+
+    4. **Disable animations in test mode:**
+       ```javascript
+       await testdriver.exec('js', `
+         document.querySelectorAll('*').forEach(el => {
+           el.style.transition = 'none';
+           el.style.animation = 'none';
+         });
+       `, 5000);
+       ```
+
+    [Learn more about stability](/v7/features/stable)
+  </Accordion>
+</AccordionGroup>
+
+## Debugging Tools
+
+### 1. Console Logging
+
+```javascript
+test('debug test', async (context) => {
+  const { testdriver } = await chrome(context, { url });
+
+  console.log('Starting test...');
+
+  const element = await testdriver.find('button');
+  console.log('Found element:', {
+    text: element.text,
+    coordinates: element.coordinates,
+    confidence: element.confidence,
+    cacheHit: element.cacheHit
+  });
+
+  await element.click();
+  console.log('Clicked element');
+});
+```
+
+### 2. Screenshots
+
+```javascript
+// Capture debug screenshot
+try {
+  await testdriver.find('element');
+} catch (error) {
+  // Save debug screenshot
+  const fs = require('fs');
+  const buffer = Buffer.from(error.debugScreenshot, 'base64');
+  fs.writeFileSync('debug.png', buffer);
+  console.log('Screenshot saved to debug.png');
+}
+```
+
+### 3. Step-by-Step Execution
+
+```javascript
+test('step by step', async (context) => {
+  const { testdriver } = await chrome(context, { url });
+
+  console.log('Step 1: Find email input');
+  const email = await testdriver.find('email input');
+  console.log('✓ Found:', email);
+
+  console.log('Step 2: Type email');
+  await email.type('user@example.com');
+  console.log('✓ Typed');
+
+  console.log('Step 3: Find button');
+  const button = await testdriver.find('submit button');
+  console.log('✓ Found:', button);
+
+  console.log('Step 4: Click button');
+  await button.click();
+  console.log('✓ Clicked');
+});
+```
+
+### 4. Network Inspection
+
+```javascript
+// View network activity in Dashcam replay
+test('check network', async (context) => {
+  const { testdriver } = await chrome(context, {
+    url,
+    captureNetwork: true // Enable network capture
+  });
+
+  await testdriver.find('load data button').click();
+
+  // Network requests appear in Dashcam replay
+  // Filter by XHR, images, scripts, etc.
+});
+```
+
+## Performance Profiling
+
+Identify slow tests:
+
+```bash
+# Run with reporter showing timing
+npx vitest run --reporter=verbose
+
+# Output shows duration per test:
+✓ slow-test.test.js > user can checkout (34.7s)
+  → find('product') (2.1s)
+  → click() (0.4s)
+  → find('add to cart') (1.8s)
+  → click() (0.3s)
+  → find('checkout') (28.9s) ← SLOW!
+  → click() (0.5s)
+```
+
+Optimize slow operations:
+
+1. **Enable caching** - First run is slow, subsequent runs are fast
+2. **Use parallel execution** - Run tests concurrently
+3. **Check network delays** - Look for slow API calls in Dashcam
+4. **Optimize selectors** - More specific = faster matching
+
+[Performance optimization guide](/v7/guides/performance)
+
+## Debugging in CI/CD
+
+When tests fail in CI:
+
+<Steps>
+  <Step title="Get Dashcam URL">
+    CI output includes replay URLs:
+
+    ```
+    Tests failed!
+    Dashcam replay: https://console.testdriver.ai/dashcam/abc123
+    ```
+  </Step>
+
+  <Step title="Download Artifacts">
+    Save debug screenshots and logs:
+
+    ```yaml .github/workflows/test.yml
+    - uses: actions/upload-artifact@v3
+      if: failure()
+      with:
+        name: test-artifacts
+        path: |
+          .testdriver/debug-screenshots/
+          .testdriver/dashcam-urls.txt
+    ```
+  </Step>
+
+  <Step title="Reconnect to Sandbox">
+    If test failed, reconnect to same sandbox to debug:
+
+    ```bash
+    # Get sandbox ID from CI logs
+    export TESTDRIVER_SANDBOX_ID=i-abc123def
+
+    # Run tests locally against same sandbox
+    npx vitest run
+    ```
+
+    [Sandbox management guide](/agents.md#sandbox-management)
+  </Step>
+</Steps>
+
+## Vitest UI
+
+Use Vitest UI for interactive debugging:
+
+```bash
+# Start Vitest UI
+npx vitest --ui
+```
+
+Open http://localhost:51204 to see:
+- Test file tree
+- Test status and duration
+- Console output
+- Re-run individual tests
+- Filter and search tests
+
+## Next Steps
+
+<CardGroup cols={2}>
+  <Card
+    title="Setting Up in CI"
+    icon="arrows-spin"
+    href="/v7/getting-started/setting-up-in-ci"
+  >
+    Run tests in CI/CD pipelines
+  </Card>
+
+  <Card
+    title="Debugging Guide"
+    icon="bug"
+    href="/v7/guides/debugging"
+  >
+    Advanced debugging techniques
+  </Card>
+
+  <Card
+    title="Observable Features"
+    icon="microscope"
+    href="/v7/features/observable"
+  >
+    Complete observability guide
+  </Card>
+
+  <Card
+    title="Troubleshooting"
+    icon="wrench"
+    href="/v7/guides/troubleshooting"
+  >
+    Common issues and solutions
+  </Card>
+</CardGroup>