@testdriverai/agent 7.11.240-test → 7.11.241-canary

package/ai/skills/testdriver-debugging-with-screenshots/SKILL.md

@@ -1,425 +1,159 @@
 ---
 name: testdriver:debugging-with-screenshots
-description: Diagnose failing tests with screenshots, replays, and logs
+description: Find out why a test failed from its replay and step-by-step screenshots.
 ---
 <!-- Generated from debugging-with-screenshots.mdx. DO NOT EDIT. -->
 
-## Overview
+When a test fails, look at what the sandbox showed instead of guessing from the error.
 
-When a test fails, debug it. Look at what happened. Do not guess. TestDriver captures screenshots, video replays, and logs while your test runs. Thus you can replay the moment of the failure. You do not need to look at a stack trace. TestDriver MCP gives commands to view and examine the screenshots that TestDriver saved during the test. This lets you debug quickly, develop tests, and compare, without a manual open of image files.
+- Every run records a replay you can step through.
+- With automatic screenshots on, TestDriver also saves an image before and after each `testdriver` method call, named after the line of code that ran it.
 
-<Note>
-  **Automatic Screenshots (Default: Enabled)**: TestDriver captures screenshots automatically before and after each command. The screenshot name has the line number and the action. Thus it is easy to see which line of code made each screenshot. For example: `001-click-before-L42-submit-button.png`
-</Note>
+## Watch the replay
 
-## MCP Commands
+Every test run prints a link to its report when it finishes:
 
-### list_local_screenshots
-
-List and filter screenshots saved in the `.testdriver/screenshots/` directory:
-
-```
-list_local_screenshots()
-```
-
-**Filter Parameters:**
-
-<ParamField path="directory" type="string" optional>
-  Filter the screenshots by test file or subdirectory (for example, "login.test", "mcp-screenshots"). If you do not give this, the tool lists all screenshots.
-</ParamField>
-
-<ParamField path="line" type="number" optional>
-  Filter by the exact line number from the test file (for example, 42 matches L42 in the filename).
-</ParamField>
-
-<ParamField path="lineRange" type="object" optional>
-  Filter by a line number range. Example: `{ start: 10, end: 20 }` matches screenshots from lines 10-20.
-</ParamField>
-
-<ParamField path="action" type="string" optional>
-  Filter by action type: `click`, `find`, `type`, `assert`, `provision`, `scroll`, `hover`, etc.
-</ParamField>
-
-<ParamField path="phase" type="string" optional>
-  Filter by phase: `"before"` (state before action) or `"after"` (state after action).
-</ParamField>
-
-<ParamField path="pattern" type="string" optional>
-  Regex pattern to match against filename. Example: `"login|signin"` or `"button.*click"`.
-</ParamField>
-
-<ParamField path="sequence" type="number" optional>
-  Filter by exact sequence number.
-</ParamField>
-
-<ParamField path="sequenceRange" type="object" optional>
-  Filter by sequence range. Example: `{ start: 1, end: 10 }` matches first 10 screenshots.
-</ParamField>
-
-<ParamField path="limit" type="number" optional>
-  Maximum number of results to return (default: 50).
-</ParamField>
-
-<ParamField path="sortBy" type="string" optional>
-  Sort results by: `"modified"` (newest first, default), `"sequence"` (execution order), or `"line"` (line number).
-</ParamField>
-
-**Returns:**
-
-Array of screenshot metadata including:
-- `path` - Full absolute path to the screenshot file
-- `relativePath` - Path relative to `.testdriver/screenshots/`
-- `name` - Screenshot filename
-- `sizeBytes` - File size in bytes
-- `modified` - Last modification timestamp
-- `sequence` - Sequential number (from auto-screenshots)
-- `action` - Action type (click, find, etc.)
-- `phase` - Before/after phase
-- `lineNumber` - Line number from test file
-- `description` - Element or action description
-
-**Example Responses:**
-
-```json
-// Basic listing
-[
-  {
-    "path": "/Users/user/project/.testdriver/screenshots/login.test/001-click-before-L42-submit-button.png",
-    "relativePath": "login.test/001-click-before-L42-submit-button.png",
-    "name": "001-click-before-L42-submit-button.png",
-    "sizeBytes": 145632,
-    "modified": "2026-01-23T10:00:00.000Z",
-    "sequence": 1,
-    "action": "click",
-    "phase": "before",
-    "lineNumber": 42,
-    "description": "submit-button"
-  }
-]
-```
-
-### view_local_screenshot
-
-View a specific screenshot from the list:
-
-```
-view_local_screenshot({ path: "/full/path/to/screenshot.png" })
-```
-
-**Parameters:**
-
-<ParamField path="path" type="string" required>
-  Full absolute path to the screenshot file (as returned by `list_local_screenshots`)
-</ParamField>
-
-**Returns:**
-
-- Image content (displayed to both AI and user via MCP App)
-- Screenshot metadata
-- Success/error status
-
-## Common Workflows
-
-### Test Debugging After Failures
-
-When a test fails, you don't have to wonder what went wrong — use powerful filtering to quickly find the screenshots that show exactly what happened:
-
-**1. Find screenshots at the failing line:**
-
-```
-// If test failed at line 42
-list_local_screenshots({ line: 42 })
-
-// View before and after states at that line
-view_local_screenshot({ path: ".testdriver/screenshots/login.test/005-click-before-L42-submit-button.png" })
-view_local_screenshot({ path: ".testdriver/screenshots/login.test/006-click-after-L42-submit-button.png" })
+```text
+TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/abc123
 ```
 
-**2. See what happened leading up to the failure:**
+The report plays a recording of the run next to a list of every command it ran, such as `find()`, `click()`, and `assert()`.
 
-```
-// Get screenshots from lines 35-45 to see context
-list_local_screenshots({ directory: "login.test", lineRange: { start: 35, end: 45 } })
-```
-
-**3. Find all assertion screenshots:**
-
-```
-// See what the screen looked like during assertions
-list_local_screenshots({ action: "assert" })
-```
-
-**4. View the final state before failure:**
-
-```
-// Get the last 5 screenshots in execution order
-list_local_screenshots({ directory: "login.test", sortBy: "sequence", limit: 5 })
-```
-
-### Finding Specific Actions
+Select a command to see the screen as it was when that command ran, and what the AI decided. This is the most reliable view of a failure. It shows what TestDriver saw, not what you expected on screen.
 
-When debugging element interactions:
+Use it to answer the usual questions:
 
-```
-// Find all click actions
-list_local_screenshots({ action: "click" })
+- **Was the right page showing?** You see right away if a `find()` or `assert()` ran on the wrong page or before the page loaded.
+- **Was the element on screen?** If `find()` couldn't locate an element, check whether it was hidden, scrolled out of view, or labelled differently.
+- **Did the click land?** Compare the screen before and after a `click()` to see whether anything changed.
 
-// Find what the screen looked like BEFORE each click
-list_local_screenshots({ action: "click", phase: "before" })
+## Save a screenshot at every step
 
-// Find screenshots related to a specific element using regex
-list_local_screenshots({ pattern: "submit|button" })
+To inspect a run locally, turn on automatic screenshots. TestDriver saves a screenshot before and after each `testdriver` method call and each `find()`, plus one when a call fails. Actions on an element, like `element.click()`, aren't captured:
 
-// Find all type actions (for form filling issues)
-list_local_screenshots({ action: "type" })
+```javascript
+const testdriver = TestDriver(context, { autoScreenshots: true });
 ```
 
-### Understanding Test Flow
-
-View screenshots in execution order to trace test behavior:
+Screenshots are saved to `.testdriver/screenshots/<test file>/`. Each name records:
 
-```
-// Get screenshots in execution order
-list_local_screenshots({ directory: "checkout.test", sortBy: "sequence" })
+- The order.
+- The command.
+- The phase.
+- The line of your test that ran it.
 
-// Get just the first 10 actions
-list_local_screenshots({ sequenceRange: { start: 1, end: 10 }, sortBy: "sequence" })
+For example:
 
-// Get just the last 10 actions
-list_local_screenshots({ directory: "checkout.test", sortBy: "sequence", limit: 10 })
+```text
+.testdriver/screenshots/login.test/
+  001-find-before-L15-email-input.png
+  002-find-after-L15-email-input.png
+  003-type-before-L17-userexamplecom.png
+  004-assert-error-L20-the-dashboard-is-visible.png
 ```
 
-### Interactive Test Development
+Automatic screenshots are off by default. TestDriver clears each test file's folder at the start of its next run.
 
-While building tests using MCP tools, view screenshots to verify your test logic:
+To save a screenshot at a specific point instead, call [`screenshot()`](/screenshot). For the full naming format, see [Screenshots](/screenshots).
 
-1. **After a test run**, filter screenshots to see specific actions:
+## Inspect screenshots with your AI assistant
 
-```
-// See all assertions
-list_local_screenshots({ action: "assert" })
+With the [local TestDriver MCP server](/mcp-setup#run-the-server-locally), your AI assistant can list and view these screenshots. Ask in plain language:
 
-// See what happened at a specific line you're debugging
-list_local_screenshots({ line: 25 })
+```text
+The login test failed at line 42. Show me the screenshots from just before and after that line.
 ```
 
-2. **Review key points** in the test execution:
-
-```
-view_local_screenshot({ path: ".testdriver/screenshots/my-test.test/after-login.png" })
-```
-
-3. **Verify element locations and states** before adding assertions
-
-4. **Iterate** - adjust your test code based on what you see in the screenshots
-
-### Comparison and Analysis
-
-Compare screenshots to identify issues:
+The assistant uses two MCP tools to answer:
 
-**Using phase filtering for before/after comparison:**
+- **`list_local_screenshots`** finds screenshots by test file, line or line range, command, phase (`before`, `after`, or `error`), sequence, or a filename pattern.
+- **`view_local_screenshot`** opens one of the screenshots it found, so both you and the assistant can see it.
 
-```
-// See state before all clicks
-list_local_screenshots({ action: "click", phase: "before" })
+These tools read the `.testdriver/` folder on the machine running the MCP server. They don't work with the hosted server at `mcp.testdriver.ai`.
 
-// See state after all clicks  
-list_local_screenshots({ action: "click", phase: "after" })
-```
+<Accordion title="list_local_screenshots parameters">
 
-**Using line-based debugging:**
+  <ParamField path="directory" type="string">
+    Test file folder or subfolder to search, such as `login.test`. Searches everything when omitted.
+  </ParamField>
+  <ParamField path="line" type="number">
+    Exact line number in the test file. `42` matches screenshots named `L42`.
+  </ParamField>
+  <ParamField path="lineRange" type="object">
+    Line range to match, inclusive: `{ start: 35, end: 45 }`.
+  </ParamField>
+  <ParamField path="action" type="string">
+    Command name, such as `click`, `find`, `type`, or `assert`.
+  </ParamField>
+  <ParamField path="phase" type="string">
+    `"before"`, `"after"`, or `"error"` (taken when a command fails).
+  </ParamField>
+  <ParamField path="pattern" type="string">
+    Regular expression to match against filenames, such as `"login|signin"`.
+  </ParamField>
+  <ParamField path="sequence" type="number">
+    Exact sequence number.
+  </ParamField>
+  <ParamField path="sequenceRange" type="object">
+    Sequence range to match, inclusive: `{ start: 1, end: 10 }`.
+  </ParamField>
+  <ParamField path="limit" type="number" default="50">
+    Maximum number of results.
+  </ParamField>
+  <ParamField path="sortBy" type="string" default="modified">
+    `"modified"` (newest first), `"sequence"` (execution order), or `"line"`.
+  </ParamField>
 
-```
-// Something went wrong around line 50
-list_local_screenshots({ lineRange: { start: 45, end: 55 } })
-```
+  `view_local_screenshot` takes one parameter, `path`: the full path of a screenshot returned by `list_local_screenshots`.
 
-**Using regex patterns:**
+</Accordion>
 
-```
-// Find screenshots related to login functionality
-list_local_screenshots({ pattern: "login|signin|email|password" })
-```
-
-## Best Practices
+## Common causes
 
 <AccordionGroup>
-  <Accordion title="Use descriptive filenames">
-    When saving screenshots in tests, use descriptive names to make them easier to identify:
-    
-    ```javascript
-    await testdriver.screenshot("initial-page-load");
-    await testdriver.screenshot("after-login-click");
-    await testdriver.screenshot("dashboard-loaded");
-    ```
-    
-    Then when listing screenshots, you can quickly identify key moments without viewing every image.
-  </Accordion>
-
-  <Accordion title="List before viewing">
-    Always call `list_local_screenshots` first to see what's available. The list is sorted by modification time (newest first), making it easy to find recent test runs.
-  </Accordion>
-
-  <Accordion title="Filter by test file">
-    When debugging a specific test, use the `directory` parameter to filter screenshots:
-    
-    ```
-    list_local_screenshots({ directory: "problematic-test.test" })
-    ```
-    
-    This avoids clutter from other tests.
-  </Accordion>
+  <Accordion title="find() acted before the page was ready">
 
-  <Accordion title="View screenshots before and after failures">
-    When a test fails (especially with assertions), look at screenshots immediately before the failure. They show exactly what the AI or test "saw" at that moment, helping you understand why an assertion failed or why an element wasn't found.
-  </Accordion>
+    The screenshot shows a loading state or the previous page.
 
-  <Accordion title="Combine with test reports">
-    TestDriver test reports include screenshots in the timeline. Use MCP screenshot viewing for interactive debugging during development, and test reports for post-run analysis and team sharing.
-  </Accordion>
+    Pass a `timeout` so `find()` keeps checking the screen while the page loads, for example `testdriver.find('Dashboard heading', { timeout: 30000 })`. See [Waiting for elements](/guide/best-practices-polling).
 
-  <Accordion title="Archive important screenshots">
-    Remember that each test run clears its screenshot folder. If you need to preserve screenshots for comparison:
-    
-    ```bash
-    # Copy screenshots before next run
-    cp -r .testdriver/screenshots/my-test.test .testdriver/screenshots-backup/
-    ```
   </Accordion>
-</AccordionGroup>
-
-## Screenshot File Organization
-
-Understanding the directory structure helps with efficient screenshot viewing:
-
-```
-.testdriver/
-  screenshots/
-    login.test/              # Test file name (without .mjs extension)
-      001-find-before-L15-email-input.png     # Auto: before find() at line 15
-      002-find-after-L15-email-input.png      # Auto: after find() at line 15
-      003-click-before-L16-email-input.png    # Auto: before click() at line 16
-      004-click-after-L16-email-input.png     # Auto: after click() at line 16
-      login-complete.png                       # Manual: screenshot("login-complete")
-    checkout.test/
-      001-find-before-L12-add-to-cart.png
-      002-find-after-L12-add-to-cart.png
-      ...
-```
-
-### Automatic Screenshot Naming Format
-
-`<seq>-<action>-<phase>-L<line>-<description>.png`
+  <Accordion title="find() picked the wrong element">
 
-| Component | Description | Example |
-|-----------|-------------|---------|
-| `seq` | Sequential number | `001`, `002` |
-| `action` | Command name | `click`, `type`, `find` |
-| `phase` | Before, after, or error | `before`, `after`, `error` |
-| `L<line>` | Line number from test file | `L42` |
-| `description` | Element/action description | `submit-button` |
+    The screenshot shows a click on a similar-looking element.
 
-### Key Points
+    Add the element's visible label, position, or color to the description, such as "the blue Sign In button in the header". See [Locate elements](/locating-elements).
 
-- Each test file gets its own subdirectory
-- Automatic screenshots include line numbers for easy tracing
-- Manual `screenshot()` calls use custom names you provide
-- Folders are cleared at the start of each test run
-- All screenshots are PNG format
-- Disable automatic screenshots with `autoScreenshots: false` if needed
-
-## Interaction List Sidebar (Source of Truth)
-
-When viewing a test run in the TestDriver console, the **interaction list sidebar** displays a screenshot for each interaction call (find, click, type, assert, etc.). These screenshots show exactly what was on the screen at the time each interaction was executed.
-
-<Note>
-  **The sidebar screenshots are the source of truth.** If a test is behaving unexpectedly, check the screenshot attached to the specific interaction in the sidebar — it shows precisely what the AI saw when making its decision. This is more reliable than inferring screen state from test logs or local screenshots alone.
-</Note>
-
-Use the interaction list to:
-- **Verify what the AI saw** — confirm the correct page/state was visible when `find()` or `assert()` ran
-- **Debug misclicks** — see whether the target element was actually on screen
-- **Identify timing issues** — spot cases where the UI hadn't finished loading before an interaction fired
-- **Compare runs** — review interaction screenshots across multiple runs to catch flaky behavior
-
-## Integration with Test Development
-
-### During MCP Interactive Development
-
-When using TestDriver MCP tools (`session_start`, `find_and_click`, etc.), screenshots are automatically captured and displayed. Additionally, you can view previously saved screenshots:
-
-```
-# After test development session
-list_local_screenshots({ directory: "my-new-test.test" })
-view_local_screenshot({ path: ".testdriver/screenshots/my-new-test.test/login-page.png" })
-```
-
-This helps verify your test logic before running the full test file.
-
-### After Test Runs
-
-When tests fail or behave unexpectedly, replay what happened step by step:
+  </Accordion>
+  <Accordion title="An assertion failed even though the page looks right">
 
-1. **Run the test** with `vitest run tests/my-test.test.mjs`
-2. **List screenshots** using `list_local_screenshots`
-3. **View relevant screenshots** to diagnose the issue
-4. **Update test code** based on what you see
-5. **Re-run and verify** the fix
+    Read the AI's reasoning for the assertion in the report.
 
-## Troubleshooting
+    Assertions that mention exact wording or several conditions at once fail more often. Check one thing at a time, in plain language. See [Make assertions](/making-assertions).
 
-<AccordionGroup>
-  <Accordion title="No screenshots found">
-    If `list_local_screenshots` returns an empty array:
-    
-    - Ensure your test includes `await testdriver.screenshot()` calls
-    - Verify the test actually ran (check test output)
-    - Check that `.testdriver/screenshots/` directory exists
-    - Confirm you're in the correct project directory
   </Accordion>
+  <Accordion title="There are no screenshots in .testdriver/screenshots">
 
-  <Accordion title="Screenshot not displaying">
-    If `view_local_screenshot` returns an error:
-    
-    - Verify the path is exactly as returned by `list_local_screenshots`
-    - Check file permissions - ensure the screenshot file is readable
-    - Confirm the file hasn't been deleted or moved
-  </Accordion>
+    Automatic screenshots are off by default. Set `autoScreenshots: true` when you create the client, or call `screenshot()` at the points you care about.
 
-  <Accordion title="Too many screenshots">
-    If you have hundreds of screenshots making it hard to find what you need, use filtering:
-    
-    - Filter by test file: `list_local_screenshots({ directory: "my-test.test" })`
-    - Filter by line number: `list_local_screenshots({ line: 42 })` or `list_local_screenshots({ lineRange: { start: 40, end: 50 } })`
-    - Filter by action: `list_local_screenshots({ action: "click" })`
-    - Filter by phase: `list_local_screenshots({ phase: "before" })`
-    - Use regex: `list_local_screenshots({ pattern: "submit|login" })`
-    - Limit results: `list_local_screenshots({ limit: 10 })`
-    - Sort by line: `list_local_screenshots({ sortBy: "line" })`
-    - Clean up old folders: `rm -rf .testdriver/screenshots/*`
-  </Accordion>
+    Also check that you're looking in the project folder you ran the test from.
 
-  <Accordion title="Screenshots from old test runs">
-    Remember that screenshot folders are cleared at the start of each test run. If you see old screenshots:
-    
-    - The test may not have run recently
-    - Or the test failed before reaching the clearing logic
-    - Manually clear: `rm -rf .testdriver/screenshots/<test-name>/`
   </Accordion>
 </AccordionGroup>
 
-## Where this fits in the Guide
-
-Debugging is what you reach for when a [Run](/copilot/running-tests) goes sideways or a [Validate](/making-assertions) assertion fails — the screenshots show you precisely what the AI saw before it acted. Once you've diagnosed the failure, the next step is to stop it from recurring.
-
-## Related
-
-- [screenshot()](/screenshot) - Capture screenshots during test execution
-- [Dashcam](/dashcam) - Record full test sessions with video and logs
-- [assert()](/assert) - Make AI-powered assertions that benefit from screenshot context
-
-## Next
-
-<Card title="Prevent" icon="shield-check" href="/copilot/auto-healing">
-  You've seen what went wrong — now keep it from happening again. Let auto-healing repair flaky tests automatically before they fail your suite.
-</Card>
+## Next steps
+
+<CardGroup cols={2}>
+  <Card title="Prevent regressions" icon="shield-check" href="/copilot/auto-healing">
+    Catch regressions in CI and repair tests when your UI changes.
+  </Card>
+  <Card title="Errors" icon="triangle-exclamation" href="/errors">
+    The errors TestDriver throws and how to handle them.
+  </Card>
+  <Card title="Screenshots" icon="camera" href="/screenshots">
+    Naming, folders, and options for saved screenshots.
+  </Card>
+  <Card title="Dashcam" icon="video" href="/dashcam">
+    How runs are recorded, and how to add your own logs.
+  </Card>
+</CardGroup>