@testdriverai/mcp 7.10.0-canary.0 → 7.10.1-test

package/ai/skills/testdriver-find/SKILL.md

@@ -49,8 +49,12 @@ const element = await testdriver.find(description, options)
       - `"any"` — No wrapping, uses the description as-is (default behavior)
     </ParamField>
     
-    <ParamField path="zoom" type="boolean" default={true}>
-      Two-phase zoom mode for better precision in crowded UIs with many similar elements. Enabled by default.
+    <ParamField path="zoom" type="boolean" default={false}>
+      Two-phase zoom mode for better precision in crowded UIs with many similar elements. Disabled by default.
+    </ParamField>
+    
+    <ParamField path="verify" type="boolean" default={false}>
+      Enable AI verification of the located element. When `true`, a second AI call checks that the coordinates returned actually correspond to the requested element, catching hallucinated or incorrect positions. Disabled by default for lower latency.
     </ParamField>
     
     <ParamField path="ai" type="object">
@@ -334,17 +338,17 @@ The `timeout` option:
 
 ## Zoom Mode
 
-Zoom mode is **enabled by default**. It uses a two-phase approach for better precision when locating elements, especially in crowded UIs with many similar elements.
+Zoom mode is **disabled by default**. It uses a two-phase approach for better precision when locating elements, especially in crowded UIs with many similar elements.
 
-To disable zoom for a specific find call, pass `zoom: false`:
+To enable zoom for a specific find call, pass `zoom: true`:
 
 ```javascript
-// Zoom is on by default — no option needed
-const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar');
+// Enable zoom for better precision in crowded UIs
+const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar', { zoom: true });
 await extensionsBtn.click();
 
-// Disable zoom for a specific call if needed
-const largeButton = await testdriver.find('big hero button', { zoom: false });
+// Without zoom (default)
+const largeButton = await testdriver.find('big hero button');
 ```
 
 ### How Zoom Mode Works
@@ -357,13 +361,48 @@ const largeButton = await testdriver.find('big hero button', { zoom: false });
 This two-phase approach gives the AI a higher-resolution view of the target area, improving accuracy when multiple similar elements are close together.
 
 <Tip>
-  You may want to disable zoom with `zoom: false` when:
-  - Targeting large, isolated elements where the extra precision isn't needed
-  - You want to speed up find calls in simple UIs
+  You may want to enable zoom with `zoom: true` when:
+  - Targeting small elements in crowded UIs with many similar elements
+  - You need extra precision for closely spaced UI elements
 </Tip>
 
+## Verify Mode
+
+Verify mode is **disabled by default**. When enabled, a second AI call checks that the coordinates returned by `find()` actually correspond to the requested element, catching hallucinated or incorrect positions.
+
+```javascript
+// Enable verification for critical interactions
+const deleteBtn = await testdriver.find('delete account button', { verify: true });
+await deleteBtn.click();
+```
+
+### How Verify Mode Works
+
+1. **Phase 1**: AI locates the element and returns coordinates
+2. **Phase 2**: A second AI call examines the screenshot at those coordinates to confirm the element matches the description
+3. **Result**: If verification fails, the find is retried or marked as not found
+
+### Combining Zoom and Verify
+
+For maximum accuracy, enable both `zoom` and `verify` together. This is useful for critical interactions where clicking the wrong element could cause cascading failures:
+
+```javascript
+// Maximum accuracy: zoom for precision + verify to catch hallucinations
+const element = await testdriver.find('small cancel icon next to the subscription', {
+  zoom: true,
+  verify: true,
+});
+await element.click();
+```
+
+<Warning>
+  Both `zoom` and `verify` add extra AI calls per `find()` invocation, which increases latency and API usage. When both are enabled, each find may make up to 3 AI calls. **Rate limiting may occur** if many find calls use these options in rapid succession. Use them selectively for critical interactions rather than on every find call.
+</Warning>
+
 ## Cache Options
 
+When a test completes successfully, the result of each `find()` is cached. On later runs, TestDriver reuses the cached match instead of making a fresh AI call, which significantly speeds up locating the same element. The cache lives in your [dashboard](https://console.testdriver.ai/cache) and is shared across runs — see the [Cache](/v7/cache) page for how matching, thresholds, and invalidation work.
+
 Control caching behavior to optimize performance, especially when using dynamic variables in prompts.
 
 ### Custom Cache Key

package/ai/skills/testdriver-generating-tests/SKILL.md

@@ -1,12 +1,20 @@
 ---
 name: testdriver:generating-tests
-description: Use AI coding agents and exploration mode to generate TestDriver tests
+description: Generate tests by exploring your app with the AI vision agent
 ---
 <!-- Generated from generating-tests.mdx. DO NOT EDIT. -->
 
+Start by exploring your app. This is where tests begin: you describe a flow in plain English, and the TestDriver vision agent clicks, types, and reads the screen to figure it out — then writes the test for you. No selectors, no DOM, no brittle locators. Just describe what you want to test and let the agent discover the rest.
+
+There are two ways to explore: chatting interactively with your AI assistant through the TestDriver MCP server, or handing a coding agent our instructions file and prompting it to generate a test.
+
+<Info>
+  Both paths need an MCP-capable AI coding assistant. If you don't have one yet, start with **GitHub Copilot** — there's a [free tier](https://github.com/features/copilot/plans), no credit card required. See [Run → Setup](/v7/copilot/running-tests#setup) for the full install and sign-in walkthrough.
+</Info>
+
 ## Instructions for Coding Agents
 
-We recommend starting with [our quickstart](./quickstart) then supplying your coding agent with our agent instructions file.
+We recommend starting with [our quickstart](./quickstart), then supplying your coding agent with our agent instructions file.
 
 <Card title="TestDriver Agent Instructions" icon="link" arrow="true" horizontal href="https://github.com/testdriverai/testdriverai/blob/main/ai/agents/testdriver.md?plain=1">
   Copy the current version of our agent instructions to provide your coding agent with up-to-date instructions on how to generate TestDriver tests.
@@ -24,13 +32,160 @@ Push Submit button
 Verify new page contains expected text 'logged in'
 ```
 
-## AI Exploration Mode
+<Info>Explicit commands are preferred for production tests, as they are cheaper, faster, and more reliable.</Info>
+
+## Start a Conversation
+
+With TestDriver's MCP server and your AI assistant (GitHub Copilot, Cursor, or Claude Desktop), you can create tests by chatting with an AI agent. The agent spawns a virtual machine, executes actions, and writes test code for you.
+
+Open your AI assistant's chat. If your project has no other agents configured, the TestDriver agent is used by default. Otherwise, select **testdriver** from the agent dropdown in the chat panel.
 
-Within a test, the `ai()` method lets TestDriver autonomously figure out how to accomplish a task. It's useful for dynamic or unpredictable UIs where explicit actions may be difficult to define.
+Describe what you want to test:
 
-```javascript
-// Handle dynamic or unpredictable UI
-await testdriver.ai('dismiss any popups or modals that appear');
+```
+Create a test that logs into my app at https://myapp.com
 ```
 
-<Info>Explicit commands are preferred for production tests, as they are cheaper, faster, and more reliable.</Info>
+The agent will:
+1. Start a new session and spawn a Linux virtual machine
+2. Launch Chrome and navigate to your URL
+3. Execute actions based on your instructions
+4. Write the test code to a `.test.mjs` file
+
+<Note>
+  The TestDriver agent appears in the agent selection dropdown if you have other agents configured (like `copilot-instructions.md` or other `.agent.md` files). Make sure to select **testdriver** to use TestDriver's MCP tools.
+</Note>
+
+## Live Preview Panel
+
+When the agent starts a session, a **live preview panel** opens in your editor. This lets you:
+
+- **Watch tests execute** in real-time
+- **Interact with the VM** — click, type, and navigate manually
+- **Debug issues** — see exactly what the AI sees
+
+<Note>
+  The live preview panel requires the TestDriver VS Code extension. Set `TD_PREVIEW=ide` in your MCP configuration to enable it. See the [Run page](/v7/copilot/running-tests) for all device and preview options.
+</Note>
+
+## Interactive Workflow
+
+The recommended workflow for exploring and creating tests:
+
+<Steps>
+  <Step title="Describe Your Test">
+    Tell the agent what you want to test:
+
+    ```
+    Test the checkout flow on https://shop.example.com
+    ```
+  </Step>
+
+  <Step title="Watch and Guide">
+    The agent will start executing. Watch the preview panel and provide guidance if needed:
+
+    ```
+    The login modal appeared. Use test@example.com and password123
+    ```
+  </Step>
+
+  <Step title="Add Assertions">
+    Ask the agent to verify specific conditions:
+
+    ```
+    Assert that the order confirmation page is shown with the order number
+    ```
+  </Step>
+
+  <Step title="Review the Code">
+    The agent writes test code as it works. Review the generated `.test.mjs` file and make any adjustments.
+  </Step>
+</Steps>
+
+## Example Conversation
+
+Here's a complete example of exploring and creating a login test:
+
+```
+You: Create a test for the login flow at https://console.testdriver.ai
+
+Agent: I'll create a test for the login flow. Starting a session...
+       [Screenshot: Login page loaded]
+
+       I can see the login page. Let me find the email input.
+       [Screenshot: Email input highlighted]
+
+       Now I'll type the email and find the password field...
+
+You: Use test@example.com for the email
+
+Agent: Got it. Typing test@example.com...
+       [Screenshot: Email entered]
+
+       Now entering the password and clicking Sign In...
+       [Screenshot: Dashboard loaded]
+
+       ✅ Login successful! I've written the test to tests/login.test.mjs
+```
+
+## Generated Test Code
+
+The agent generates standard Vitest test files:
+
+```javascript tests/login.test.mjs
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("Login Flow", () => {
+  it("should log in successfully", async (context) => {
+    const testdriver = TestDriver(context);
+
+    await testdriver.provision.chrome({
+      url: "https://console.testdriver.ai"
+    });
+
+    const emailInput = await testdriver.find("email input field");
+    await emailInput.click();
+    await testdriver.type("test@example.com");
+
+    const passwordInput = await testdriver.find("password input field");
+    await passwordInput.click();
+    await testdriver.type("password123");
+
+    const signInButton = await testdriver.find("Sign In button");
+    await signInButton.click();
+
+    const result = await testdriver.assert("dashboard is visible");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+## Tips for Better Tests
+
+<AccordionGroup>
+  <Accordion title="Be specific with element descriptions">
+    Instead of "click the button", say "click the blue Sign In button in the header". More context helps the AI find the right element.
+  </Accordion>
+  <Accordion title="Add waits for dynamic content">
+    If your app has animations or loading states, tell the agent to wait:
+    ```
+    Wait for the loading spinner to disappear before continuing
+    ```
+  </Accordion>
+  <Accordion title="Use assertions liberally">
+    Add assertions after each major action to catch regressions early:
+    ```
+    Assert that the product was added to the cart
+    ```
+  </Accordion>
+  <Accordion title="Break complex flows into steps">
+    For long workflows, create the test incrementally and verify each step works before moving on.
+  </Accordion>
+</AccordionGroup>
+
+## Next
+
+<Card title="Learn" icon="brain" arrow="true" horizontal href="/v7/caching">
+  Once the agent has explored your app, TestDriver caches what it discovers so your tests replay instantly without re-reasoning over the screen every time.
+</Card>

package/ai/skills/testdriver-hosted/SKILL.md

@@ -75,7 +75,7 @@ To prevent tests from failing due to exceeding your license slot limit, we recom
 
     ```javascript vitest.config.mjs
     import { defineConfig } from 'vitest/config';
-    import { TestDriver } from 'testdriverai/vitest';
+    import TestDriver from 'testdriverai/vitest';
 
     export default defineConfig({
       test: {

package/ai/skills/testdriver-machine-setup/SKILL.md

@@ -0,0 +1,331 @@
+---
+name: testdriver:machine-setup
+description: Configure Linux and Windows sandboxes, persist machines between runs, and install custom software
+---
+<!-- Generated from machine-setup.mdx. DO NOT EDIT. -->
+
+TestDriver provisions a fresh cloud VM for every test by default. This guide covers how to configure Linux and Windows machines, reduce startup time by keeping machines alive between runs, use provision scripts for repeatable setup, and install custom software on the fly.
+
+---
+
+## Linux Machines
+
+Linux is the default operating system. No extra configuration is required.
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+describe("My Test", () => {
+  it("runs on Linux", async (context) => {
+    const testdriver = TestDriver(context);
+
+    await testdriver.provision.chrome({ url: "https://example.com" });
+
+    const result = await testdriver.assert("the page loaded successfully");
+    expect(result).toBeTruthy();
+  });
+});
+```
+
+### Common Linux Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `os` | string | `"linux"` | Operating system |
+| `resolution` | string | `"1366x768"` | Screen resolution (Enterprise only) |
+| `e2bTemplateId` | string | — | Custom E2B template ID (see [Self-Hosted](/v7/self-hosted)) |
+| `keepAlive` | number | `60000` | Ms to keep VM alive after disconnect |
+| `reconnect` | boolean | `false` | Reconnect to last used sandbox |
+
+```javascript
+const testdriver = TestDriver(context, {
+  os: "linux",
+  resolution: "1920x1080",
+  keepAlive: 5 * 60 * 1000, // keep alive 5 minutes
+});
+```
+
+---
+
+## Windows Machines
+
+Set `os: "windows"` to provision a Windows VM instead. Everything else works the same way.
+
+```javascript
+const testdriver = TestDriver(context, {
+  os: "windows",
+});
+
+await testdriver.provision.chrome({ url: "https://example.com" });
+```
+
+Windows sandboxes use EC2 instances and take longer to boot than Linux (E2B) sandboxes — typically 1–3 minutes for a cold start. See [Keeping Machines Alive](#keeping-machines-alive-between-runs) below to avoid this cost on repeated runs.
+
+### Common Windows Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `os` | string | — | Set to `"windows"` |
+| `resolution` | string | `"1366x768"` | Screen resolution (Enterprise only) |
+| `sandboxAmi` | string | — | Custom AMI ID (self-hosted) |
+| `sandboxInstance` | string | — | EC2 instance type (self-hosted) |
+| `keepAlive` | number | `60000` | Ms to keep VM alive after disconnect |
+| `reconnect` | boolean | `false` | Reconnect to last used sandbox |
+
+```javascript
+const testdriver = TestDriver(context, {
+  os: "windows",
+  resolution: "1920x1080",
+  keepAlive: 10 * 60 * 1000, // keep alive 10 minutes
+});
+```
+
+---
+
+## Keeping Machines Alive Between Runs
+
+Windows (and Linux) cold starts can be expensive if you're iterating quickly. Use `keepAlive` + `reconnect` to reuse the same VM across multiple test runs.
+
+### How it works
+
+Every time the SDK successfully connects to a sandbox, it records the sandbox id in `.testdriver/last-sandbox` inside your project directory. The next test that opts in with `reconnect: true` reads that file and reattaches automatically — no manual id tracking required.
+
+Provision calls (`testdriver.provision.chrome(...)`, `vscode(...)`, etc.) are **skipped** when reconnecting, because the application is already running inside the sandbox from the previous run.
+
+<Note>
+  `.testdriver/last-sandbox` is already covered by the default TestDriver `.gitignore`. Don't commit it.
+</Note>
+
+### Step 1 — Start the machine with a long `keepAlive`
+
+```javascript
+// first.test.mjs
+const testdriver = TestDriver(context, {
+  os: "windows",
+  keepAlive: 30 * 60 * 1000, // keep alive 30 minutes after this test ends
+});
+
+await testdriver.provision.chrome({ url: "https://example.com" });
+// ... your test steps
+```
+
+When this test finishes, the sandbox stays running for 30 minutes instead of being terminated immediately.
+
+### Step 2 — Reattach automatically with `reconnect: true`
+
+```javascript
+// second.test.mjs
+const testdriver = TestDriver(context, {
+  os: "windows",
+  reconnect: true,            // ← reads .testdriver/last-sandbox
+  keepAlive: 30 * 60 * 1000,
+});
+
+// No provision call — Chrome is already open from the previous run.
+await testdriver.find("Sign In button").click();
+```
+
+### Step 2 (alternative) — Reattach to an explicit id
+
+If you need to pin to a specific sandbox (CI matrix, multiple chains in parallel, etc.) pass the id directly:
+
+```javascript
+await testdriver.connect({ sandboxId: "sandbox-abc123" });
+```
+
+When reattaching to a sandbox:
+- You reuse a specific running machine directly
+- You continue from the app state created in the earlier run
+- You must run within the previous test's `keepAlive` window
+
+<Tip>
+  Use `testdriver.getLastSandboxId()` to read the recorded sandbox id (and optional metadata) for scripting purposes.
+</Tip>
+
+### Chaining describe blocks within one test file
+
+A common pattern is to break a long flow into focused `describe` blocks that share one sandbox — the first block provisions and signs in, later blocks reconnect and continue:
+
+```javascript
+import { describe, expect, it } from "vitest";
+import { TestDriver } from "testdriverai/vitest/hooks";
+
+const KEEP_ALIVE_MS = 5 * 60 * 1000;
+
+describe("step 1 — log in", () => {
+  it("signs in and lands on the dashboard", async (context) => {
+    const testdriver = TestDriver(context, { keepAlive: KEEP_ALIVE_MS });
+    await testdriver.provision.chrome({ url: "https://example.com/login" });
+    await testdriver.find("username input").click();
+    await testdriver.type("standard_user");
+    await testdriver.pressKeys(["tab"]);
+    await testdriver.type("secret_sauce", { secret: true });
+    await testdriver.pressKeys(["enter"]);
+    expect(await testdriver.assert("the dashboard is visible")).toBeTruthy();
+  });
+});
+
+describe("step 2 — add to cart", () => {
+  it("reuses the logged-in sandbox", async (context) => {
+    const testdriver = TestDriver(context, {
+      reconnect: true,           // ← skip provisioning, reattach
+      keepAlive: KEEP_ALIVE_MS,
+    });
+    await testdriver.find("Add to cart").click();
+    await testdriver.find("cart icon").click();
+    expect(await testdriver.assert("the cart has an item")).toBeTruthy();
+  });
+});
+
+describe("step 3 — check out", () => {
+  it("continues from the cart state", async (context) => {
+    const testdriver = TestDriver(context, { reconnect: true, keepAlive: 30_000 });
+    await testdriver.find("Checkout").click();
+    expect(await testdriver.assert("the checkout form is visible")).toBeTruthy();
+  });
+});
+```
+
+A runnable copy of this pattern lives at [`examples/reconnect-sequential.test.mjs`](https://github.com/testdriverai/mono/blob/main/sdk/examples/reconnect-sequential.test.mjs).
+
+<Warning>
+  Vitest runs **test files** in parallel by default. Within a single file, `describe`/`it` blocks run in source order, so reconnect chaining works as written. To chain across multiple files, run them sequentially (e.g. `vitest run --sequence.concurrent=false` or place them in a single project pool with workers set to 1).
+</Warning>
+
+### How `keepAlive` works
+
+`keepAlive` is a duration in milliseconds. After the SDK disconnects, the server keeps the VM running for that long before terminating it. The default is `60000` (1 minute). Note: `keepAlive: 0` currently falls back to the default disconnect grace period rather than terminating immediately, so use a positive duration when you want to control the grace window explicitly.
+
+```javascript
+const testdriver = TestDriver(context, {
+  keepAlive: 0,           // currently uses the default 1 minute grace period
+  // keepAlive: 60000,    // default — 1 minute
+  // keepAlive: 600000,   // 10 minutes
+  // keepAlive: 3600000,  // 1 hour
+});
+```
+
+<Warning>
+  Machines kept alive beyond your test session continue to consume credits. Always set a `keepAlive` value appropriate for your workflow.
+</Warning>
+
+---
+
+## Using Provision Scripts
+
+Provision scripts let you run arbitrary setup steps before your test starts — downloading fixtures, seeding a database, configuring environment variables, and more. Use `testdriver.exec()` to run shell or PowerShell commands directly in the sandbox.
+
+<Card
+  title="exec() Reference"
+  icon="terminal"
+  href="/v7/exec"
+>
+  Full reference for running shell and PowerShell commands in the sandbox.
+</Card>
+
+### Linux setup script
+
+```javascript
+await testdriver.provision.chrome({ url: "https://myapp.com" });
+
+// Run a setup script from your repo
+await testdriver.exec("sh", `
+  curl -s https://myapp.com/api/reset-test-db -X POST
+  echo "Test DB reset"
+`, 30000);
+```
+
+### Windows setup script (PowerShell)
+
+```javascript
+await testdriver.provision.chrome({ url: "https://myapp.com" });
+
+await testdriver.exec("pwsh", `
+  $env:API_URL = "https://staging.myapp.com"
+  Write-Host "Environment configured"
+`, 15000);
+```
+
+### Clone a repo and run a script
+
+```javascript
+await testdriver.exec("sh", `
+  git clone https://github.com/myorg/test-fixtures.git /tmp/fixtures
+  bash /tmp/fixtures/seed.sh
+`, 120000);
+```
+
+---
+
+## Installing Custom Software
+
+You can install software at the start of a test using `exec()`. This works for any package available via `apt`, `brew`, `choco`, `winget`, npm, pip, or direct download.
+
+### Linux — apt packages
+
+```javascript
+await testdriver.exec("sh", `
+  sudo apt-get update -qq
+  sudo apt-get install -y ffmpeg imagemagick
+`, 120000);
+```
+
+### Linux — Node.js tools
+
+```javascript
+await testdriver.exec("sh", "npm install -g @playwright/test", 60000);
+```
+
+### Windows — winget
+
+```javascript
+await testdriver.exec("pwsh", `
+  winget install --id=7zip.7zip -e --silent
+`, 120000);
+```
+
+### Windows — Chocolatey
+
+```javascript
+await testdriver.exec("pwsh", `
+  choco install googlechrome --yes --no-progress
+`, 180000);
+```
+
+### Download and run an installer
+
+```javascript
+// Linux
+await testdriver.exec("sh", `
+  curl -L https://example.com/installer.sh -o /tmp/installer.sh
+  chmod +x /tmp/installer.sh
+  /tmp/installer.sh --silent
+`, 300000);
+
+// Windows
+await testdriver.exec("pwsh", `
+  Invoke-WebRequest -Uri "https://example.com/installer.exe" -OutFile "$env:TEMP\\installer.exe"
+  Start-Process "$env:TEMP\\installer.exe" -ArgumentList "/S" -Wait
+`, 300000);
+```
+
+<Note>
+  Installing software at test start adds to your test duration. For software you use in every test, consider preloading it into a custom VM image via the Enterprise self-hosted plan.
+</Note>
+
+---
+
+## Want Software Pre-Installed on Every Machine?
+
+Installing packages at runtime works well for occasional or lightweight dependencies. But if you're installing the same 5-minute setup on every test run, you're wasting time and credits.
+
+With the **Self-Hosted Enterprise plan** you get access to our golden VM base image and Packer scripts, so you can bake your applications, dependencies, and configuration directly into a custom AMI. Tests spin up with everything already installed — zero setup time.
+
+<Card
+  title="Self-Hosted Enterprise"
+  icon="server"
+  href="/v7/self-hosted"
+>
+  Preload software, configure custom hardware, and run unlimited tests with a flat license fee. Our team assists with deployment and setup.
+</Card>