testdriverai 7.2.63 → 7.2.65

package/ai/skills/testdriver:mcp-workflow/SKILL.md

@@ -0,0 +1,410 @@
+---
+name: testdriver:mcp-workflow
+description: Build TestDriver tests iteratively using MCP tools with visual feedback
+---
+
+# TestDriver MCP Workflow
+
+Build automated tests by directly controlling a sandbox through MCP tools. Every action returns a screenshot AND the generated code to add to your test file.
+
+## When to Use This Skill
+
+Use this skill when:
+- You have access to TestDriver MCP tools (`session_start`, `find`, `click`, etc.)
+- User asks to "write a test", "automate this workflow", "check if X works"
+- You need to build tests iteratively with visual feedback
+
+## Overview
+
+Use MCP tools to:
+
+1. **Control the sandbox directly** - Click, type, scroll in real-time
+2. **See visual feedback** - Every action shows a screenshot with overlays
+3. **Get generated code** - Each successful action returns the code to add to your test file
+4. **Build tests incrementally** - Append code to test files as you go
+
+## Quick Start
+
+### 1. Start a Session
+
+```
+session_start({ type: "chrome", url: "https://your-app.com" })
+```
+
+This provisions a sandbox with Chrome and navigates to your URL. You'll see a screenshot and the provision code:
+
+```
+Add to test file:
+await testdriver.provision.chrome({ url: "https://your-app.com" });
+```
+
+**For local development** (pointing to a custom API endpoint):
+
+```
+session_start({ 
+  type: "chrome", 
+  url: "https://your-app.com",
+  apiRoot: "https://your-ngrok-url.ngrok.io"
+})
+```
+
+**For self-hosted AWS instances** (your own Windows EC2):
+
+```
+session_start({ 
+  type: "chrome", 
+  url: "https://your-app.com",
+  os: "windows",
+  ip: "1.2.3.4"  // IP from your AWS instance
+})
+```
+
+See [AWS Setup Guide](https://docs.testdriver.ai/v7/aws-setup) to deploy your own infrastructure.
+
+### 2. Interact with the App
+
+Find elements and interact with them. Each action returns a screenshot AND generated code:
+
+```
+find_and_click({ description: "Sign In button" })
+→ Returns: screenshot with element highlighted
+→ Add to test file: await testdriver.find("Sign In button").click();
+
+type({ text: "user@example.com" })
+→ Returns: screenshot showing typed text
+→ Add to test file: await testdriver.type("user@example.com");
+```
+
+### 3. Check If Actions Succeeded
+
+After performing actions, use `check` to verify they worked:
+
+```
+check({ task: "Was the text entered into the field?" })
+→ Returns: AI analysis of whether the task completed, with screenshot
+
+check({ task: "Did the button click navigate to a new page?" })
+→ Returns: AI compares previous screenshot to current state
+```
+
+### 4. Make Assertions (for Test Files)
+
+Use `assert` for boolean pass/fail conditions that get recorded in test files:
+
+```
+assert({ assertion: "the login form is visible" })
+→ Returns: pass/fail with screenshot
+→ Add to test file:
+   const assertResult = await testdriver.assert("the login form is visible");
+   expect(assertResult).toBeTruthy();
+```
+
+### 5. Write the Test File
+
+As you perform actions, append the generated code to your test file:
+
+```javascript
+/**
+ * Login Flow test
+ */
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Login Flow", () => {
+  it("should complete login", async (context) => {
+    const testdriver = TestDriver(context);
+
+    // Append generated code here as you go:
+    await testdriver.provision.chrome({ url: "https://app.example.com" });
+    await testdriver.find("email input field").click();
+    await testdriver.type("user@example.com");
+    // ... more code as you perform actions
+  });
+});
+```
+
+### 6. Verify the Test
+
+Run the test from scratch to validate it works:
+
+```
+verify({ testFile: "tests/login.test.mjs" })
+```
+
+## Tools Reference
+
+### Session Management
+
+| Tool | Description |
+|------|-------------|
+| `session_start` | Start sandbox with browser/app, returns screenshot + provision code |
+| `session_status` | Check session health and time remaining |
+| `session_extend` | Add more time before session expires |
+
+### Element Interaction
+
+Each tool returns a screenshot AND the generated code to add to your test file.
+
+| Tool | Description |
+|------|-------------|
+| `find` | Locate element by description, returns ref for later use |
+| `click` | Click on element ref |
+| `find_and_click` | Find and click in one action |
+| `type` | Type text into focused field |
+| `press_keys` | Press keyboard shortcuts (e.g., `["ctrl", "a"]`) |
+| `scroll` | Scroll page (up/down/left/right) |
+
+### Verification & Display
+
+| Tool | Description |
+|------|-------------|
+| `check` | **For AI to understand screen state.** Analyzes current screen and tells you (the AI) whether a task/condition is met. Use this after actions to verify they worked. |
+| `assert` | AI-powered boolean assertion for test files (pass/fail for CI). Returns generated code. |
+| `screenshot` | **For showing the user the screen.** Captures and displays a screenshot. Does NOT return analysis to you (the AI). |
+| `exec` | Execute JavaScript, shell, or PowerShell in sandbox. Returns generated code. |
+
+### Test Validation
+
+| Tool | Description |
+|------|-------------|
+| `verify` | Run test file from scratch to validate it works |
+
+## Visual Feedback
+
+Every tool returns a screenshot showing:
+
+- **Element highlights** - Blue box around found elements
+- **Click markers** - Red dot with ripple effect at click location
+- **Scroll indicators** - Arrow showing scroll direction
+- **Action status** - Success/failure with duration
+- **Session info** - Time remaining before expiry
+
+## Workflow Best Practices
+
+### 1. Work Incrementally
+
+Don't try to build the entire test at once:
+
+```
+# Step 1: Get to login page
+session_start({ url: "https://app.com" })
+→ Add to test: await testdriver.provision.chrome({ url: "https://app.com" });
+
+# Step 2: Verify you're on the right page
+check({ task: "Is this the login page?" })
+
+# Step 3: Fill in email
+find_and_click({ description: "email input field" })
+→ Add to test: await testdriver.find("email input field").click();
+
+type({ text: "user@example.com" })
+→ Add to test: await testdriver.type("user@example.com");
+
+# Step 4: Check if email was entered
+check({ task: "Was the email entered correctly?" })
+
+# Step 5: Continue with password...
+```
+
+### 2. Use Check After Actions
+
+After each action, use `check` to verify it worked:
+
+```
+find_and_click({ description: "Submit button" })
+check({ task: "Was the form submitted?" })
+```
+
+The `check` tool compares the previous screenshot (from before your action) with the current state, giving you AI analysis of what changed and whether the action succeeded.
+
+### 3. Understanding Screen State
+
+**For AI understanding:** Use `check` to analyze the screen:
+```
+check({ task: "Did the form submit successfully?" })
+→ Returns AI analysis you can read and understand
+```
+
+**For user visibility:** Use `screenshot` to show the user:
+```
+screenshot()
+→ Displays to user, no analysis returned to you
+```
+
+Action tools (`find`, `click`, `find_and_click`) return screenshots automatically, which the user can see. But if you need to understand the state, use `check`.
+
+### 4. Handle Timing Issues
+
+If elements take time to appear, use `find` with timeout:
+
+```
+find({ description: "Loading complete indicator", timeout: 30000 })
+```
+
+### 5. Check Session Time
+
+Sessions expire after 5 minutes by default. Use `session_status` to check time remaining and `session_extend` to add more time:
+
+```
+session_status()
+→ "Time remaining: 45s"
+
+session_extend({ additionalMs: 60000 })
+→ "New expiry: 105s"
+```
+
+### 6. Write Code as You Go
+
+After each successful action, append the generated code to your test file. This ensures you don't lose progress and makes the test easier to debug.
+
+## Error Recovery
+
+### Element Not Found
+
+If `find` fails:
+
+1. Check the screenshot to see what's actually on screen
+2. Adjust the element description to be more specific
+3. Wait for page load with timeout: `find({ description: "...", timeout: 10000 })`
+4. Scroll to find off-screen elements: `scroll({ direction: "down" })`
+
+### Session Expired
+
+If the session expires:
+
+1. `session_start` again with the same URL
+2. Run the test with `verify` to get back to last state
+3. Continue from where you left off
+
+### Test Verification Fails
+
+If `verify` fails:
+
+1. Check the error message and screenshot
+2. Review the test file to see the generated code
+3. Start a new session and manually test the failing step
+4. Adjust element descriptions or add waits as needed
+
+## Dependencies
+
+When creating a new test project, use these exact dependencies:
+
+**package.json:**
+```json
+{
+  "type": "module",
+  "devDependencies": {
+    "testdriverai": "beta",
+    "vitest": "^4.0.0"
+  },
+  "scripts": {
+    "test": "vitest"
+  }
+}
+```
+
+**Important:** The package is `testdriverai` (NOT `@testdriverai/sdk`). Always install from the `beta` tag.
+
+## Test File Format
+
+Create test files using this standard format. Append generated code inside the test function:
+
+```javascript
+/**
+ * Login Flow test
+ */
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("Login Flow", () => {
+  it("should complete Login Flow", async (context) => {
+    const testdriver = TestDriver(context);
+
+    // Append generated code from each action here:
+    await testdriver.provision.chrome({ url: "https://app.example.com" });
+    
+    await testdriver.find("email input field").click();
+    await testdriver.type("user@example.com");
+    
+    await testdriver.find("password field").click();
+    await testdriver.type("secret123");
+    
+    await testdriver.find("Sign In button").click();
+    
+    const assertResult = await testdriver.assert("dashboard is visible");
+    expect(assertResult).toBeTruthy();
+  });
+});
+```
+
+## Self-Hosted AWS Instances
+
+You can use your own AWS-hosted Windows instances instead of TestDriver cloud. This gives you:
+
+- **Flat license fee** - No device-second metering
+- **Your own AI keys** - Use your OpenAI/Anthropic keys
+- **Custom configuration** - Install custom software, configure networking
+- **Full debugging access** - RDP into instances
+
+### Quick Setup
+
+1. **Deploy AWS infrastructure** using [CloudFormation](https://docs.testdriver.ai/v7/aws-setup)
+
+2. **Spawn an instance**:
+   ```bash
+   AWS_REGION=us-east-2 \
+   AMI_ID=ami-0504bf50fad62f312 \
+   AWS_LAUNCH_TEMPLATE_ID=lt-xxx \
+   bash setup/aws/spawn-runner.sh
+   ```
+   Output: `PUBLIC_IP=1.2.3.4`
+
+3. **Connect via session_start**:
+   ```
+   session_start({
+     type: "chrome",
+     url: "https://example.com",
+     os: "windows",
+     ip: "1.2.3.4"
+   })
+   ```
+
+4. **Terminate when done**:
+   ```bash
+   aws ec2 terminate-instances --instance-ids i-xxx --region us-east-2
+   ```
+
+### Environment Variable
+
+You can also set `TD_IP` environment variable in your MCP config instead of passing `ip` to each session:
+
+```json
+{
+  "mcpServers": {
+    "testdriver": {
+      "env": {
+        "TD_API_KEY": "your-key",
+        "TD_IP": "1.2.3.4"
+      }
+    }
+  }
+}
+```
+
+## Tips
+
+1. **Every action returns generated code** - Look for "Add to test file:" in responses and append that code
+
+2. **Use `check` to understand the screen** - This is how you (the AI) see and analyze the current state
+
+3. **Use `screenshot` to show the user** - This displays the screen to the user, but does NOT return analysis to you
+
+4. **Use `check` after every action** - Verify your actions succeeded before moving on
+
+5. **Be specific with element descriptions** - "the blue Sign In button in the header" is better than "button"
+
+6. **Use `check` for verification, `assert` for test files** - `check` gives detailed AI analysis, `assert` gives boolean pass/fail for CI
+
+7. **Write code incrementally** - Append generated code to your test file after each successful action
+
+8. **Extend session proactively** - If you have complex workflows, extend before you run out of time

package/{docs/v7/mouse-down.mdx → ai/skills/testdriver:mouse-down/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "mouseDown"
-description: "Press the mouse button without releasing it"
-icon: "arrow-pointer"
+name: testdriver:mouse-down
+description: Press the mouse button without releasing it
 ---
+<!-- Generated from mouse-down.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/mouse-up.mdx → ai/skills/testdriver:mouse-up/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "mouseUp"
-description: "Release the mouse button"
-icon: "arrow-pointer"
+name: testdriver:mouse-up
+description: Release the mouse button
 ---
+<!-- Generated from mouse-up.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/performing-actions.mdx → ai/skills/testdriver:performing-actions/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Performing Actions"
-description: "Click, type, hover, scroll and more with TestDriver"
-icon: "computer-mouse"
+name: testdriver:performing-actions
+description: Click, type, hover, scroll and more with TestDriver
 ---
+<!-- Generated from performing-actions.mdx. DO NOT EDIT. -->
 
 ## Performing Actions
 

package/{docs/v7/press-keys.mdx → ai/skills/testdriver:press-keys/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "pressKeys()"
-sidebarTitle: "pressKeys"
-description: "Press keyboard keys and shortcuts"
-icon: "keyboard"
+name: testdriver:press-keys
+description: Press keyboard keys and shortcuts
 ---
+<!-- Generated from press-keys.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/quickstart.mdx → ai/skills/testdriver:quickstart/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "Quick Start"
-sidebarTitle: "Quickstart"
-description: "Run your first computer-use test in minutes."
-icon: "rocket"
+name: testdriver:quickstart
+description: Run your first computer-use test in minutes.
 ---
+<!-- Generated from quickstart.mdx. DO NOT EDIT. -->
 
 TestDriver makes it easy to write automated computer-use tests for web browsers, desktop apps, and more. Follow the directions below to run your first TestDriver test.
 

package/{docs/v7/reusable-code.mdx → ai/skills/testdriver:reusable-code/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Reusable Code Snippets"
-description: "Build maintainable test suites with reusable code patterns"
-icon: "recycle"
+name: testdriver:reusable-code
+description: Build maintainable test suites with reusable code patterns
 ---
+<!-- Generated from reusable-code.mdx. DO NOT EDIT. -->
 
 As your test suite grows, you'll want to extract common patterns into reusable code. This keeps tests DRY, readable, and easy to maintain.
 

package/{docs/v7/right-click.mdx → ai/skills/testdriver:right-click/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "rightClick"
-description: "Perform a right-click action to open context menus"
-icon: "bars"
+name: testdriver:right-click
+description: Perform a right-click action to open context menus
 ---
+<!-- Generated from right-click.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/running-tests.mdx → ai/skills/testdriver:running-tests/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Running Tests"
-description: "Run TestDriver tests with Vitest test runner"
-icon: "play"
+name: testdriver:running-tests
+description: Run TestDriver tests with Vitest test runner
 ---
+<!-- Generated from running-tests.mdx. DO NOT EDIT. -->
 
 Learn how to run TestDriver tests efficiently with Vitest's powerful test runner.
 

package/{docs/v7/screenshot.mdx → ai/skills/testdriver:screenshot/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "screenshot()"
-sidebarTitle: "screenshot"
-description: "Capture and save screenshots during test execution"
-icon: "camera"
+name: testdriver:screenshot
+description: Capture and save screenshots during test execution
 ---
+<!-- Generated from screenshot.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/scroll.mdx → ai/skills/testdriver:scroll/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "scroll()"
-sidebarTitle: "scroll"
-description: "Scroll pages and elements"
-icon: "arrows-up-down"
+name: testdriver:scroll
+description: Scroll pages and elements
 ---
+<!-- Generated from scroll.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/secrets.mdx → ai/skills/testdriver:secrets/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Using Secrets"
-description: "Securely manage passwords and sensitive data in your tests"
-icon: "key"
+name: testdriver:secrets
+description: Securely manage passwords and sensitive data in your tests
 ---
+<!-- Generated from secrets.mdx. DO NOT EDIT. -->
 
 Protect sensitive information like passwords, API keys, and tokens in your TestDriver tests.
 

package/{docs/v7/self-hosted.mdx → ai/skills/testdriver:self-hosted/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "Self-Hosted"
-sidebarTitle: "Self-Hosted"
-description: "Unlimited test execution, complete privacy, and the ability to customize everything — all for a predictable flat license fee."
-icon: "server"
+name: testdriver:self-hosted
+description: Unlimited test execution, complete privacy, and the ability to customize everything — all for a predictable flat license fee.
 ---
+<!-- Generated from self-hosted.mdx. DO NOT EDIT. -->
 
 Self-hosted pricing is based on **parallel test capacity**: the number of tests you can run simultaneously on **your infrastructure**. 
 

package/ai/skills/testdriver:testdriver/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: testdriver:testdriver
+description: Help the user work with TestDriver tests and query test results via the TestDriver MCP server.
+---
+
+You are an expert TestDriver assistant.
+
+When helping the user:
+
+1. **For writing or editing tests**
+
+   - Follow the guidance in `agents.md` from the `testdriverai` repo:
+     - Use Vitest with `.test.mjs`.
+     - Use the two-file pattern (`setup.test.mjs` + `experiment.test.mjs`).
+     - Prefer explicit steps with `find`, `click`, `type`, `assert`, and `screenshot`.
+     - Run the tests yourself using `npx vitest run` to verify functionality.
+
+2. **For inspecting results / failures**
+
+   - Use the TestDriver MCP server tools:
+     - `list_test_runs` to see recent runs (filter by status, branch, file, suite, platform).
+     - `get_test_run_detail` to inspect a specific run and its test cases.
+     - `list_test_cases` to see individual tests, errors, and replays.
+     - `get_filter_options` to discover available branches, files, suites, and test names.
+   - Summarize failing tests, group by file/suite, and reference any replay IDs or links.
+
+3. **General behavior**
+   - Prefer reading existing tests and docs before suggesting large refactors.
+   - When the user mentions "plugin" or "MCP", explain that this plugin uses:
+     - The HTTP MCP endpoint at `/api/v1/mcp` (configured in `.mcp.json`),
+     - This skill definition plus the `testdriverai/agents.md` guide.

package/{docs/v7/type.mdx → ai/skills/testdriver:type/SKILL.md}

@@ -1,9 +1,8 @@
 ---
-title: "type()"
-sidebarTitle: "type"
-description: "Type text into focused input fields"
-icon: "keyboard"
+name: testdriver:type
+description: Type text into focused input fields
 ---
+<!-- Generated from type.mdx. DO NOT EDIT. -->
 
 ## Overview
 

package/{docs/v7/variables.mdx → ai/skills/testdriver:variables/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Using Variables"
-description: "Use dynamic data and secure secrets in your tests"
-icon: "square-root-variable"
+name: testdriver:variables
+description: Use dynamic data and secure secrets in your tests
 ---
+<!-- Generated from variables.mdx. DO NOT EDIT. -->
 
 Scale your testing with dynamic data and secure secrets management. Choose the right approach based on your testing needs.
 

package/{docs/v7/waiting-for-elements.mdx → ai/skills/testdriver:waiting-for-elements/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "Waiting for Elements"
-description: "Handle async operations and prevent flaky tests"
-icon: "clock"
+name: testdriver:waiting-for-elements
+description: Handle async operations and prevent flaky tests
 ---
+<!-- Generated from waiting-for-elements.mdx. DO NOT EDIT. -->
 
 ## Waiting for Elements
 

package/{docs/v7/what-is-testdriver.mdx → ai/skills/testdriver:what-is-testdriver/SKILL.md}

@@ -1,8 +1,8 @@
 ---
-title: "What is TestDriver?"
-description: "Reliably test your most difficult user flows"
-icon: "circle-info"
+name: testdriver:what-is-testdriver
+description: Reliably test your most difficult user flows
 ---
+<!-- Generated from what-is-testdriver.mdx. DO NOT EDIT. -->
 
 ## The problem with modern testing tools