@testdriverai/agent 7.11.240-test → 7.11.241-test

package/ai/skills/testdriver-making-assertions/SKILL.md

@@ -1,108 +1,150 @@
 ---
 name: testdriver:making-assertions
-description: Locate elements and verify app state with AI-powered assertions
+description: Check that your app reached the right state, in plain language or with expect().
 ---
 <!-- Generated from making-assertions.mdx. DO NOT EDIT. -->
 
-After a test runs, make sure that the app did the correct thing. Validation has two parts: to find the elements that you want to check, and to make assertions about the state of your app. TestDriver uses the AI as a judge. It returns a boolean and the reason. This shows if your app is in the correct state.
+An assertion checks that your app did the right thing after a step. TestDriver gives you two ways to check:
 
-## Locating Elements
+- [`assert()`](/assert) asks the AI whether a statement about the screen is true, and fails the test if it isn't.
+- [`extract()`](/extract) reads a value off the screen, so you can compare it yourself with `expect()`.
 
-### Locating Single Elements
+```javascript
+await testdriver.find('Add to cart button').click();
+await testdriver.assert('the cart icon shows 1 item');
+```
 
-Use natural language to describe elements. A description must be specific enough to find the element. But it must not be so specific that it breaks with small UI changes. For example:
+## Check the screen with assert()
 
-```javascript
-await testdriver.find('email input field');
-await testdriver.find('first product card in the grid');
-await testdriver.find('dropdown menu labeled "Country"');
+Pass `assert()` a statement that should be true. The AI looks at the current screen and decides.
+
+```javascript tests/login.test.mjs
+import { describe, it } from 'vitest';
+import { TestDriver } from 'testdriverai/vitest/hooks';
+
+describe('Login', () => {
+  it('signs in with valid credentials', async (context) => {
+    const testdriver = TestDriver(context);
+    await testdriver.provision.chrome({ url: 'https://testdriver-sandbox.vercel.app/login' });
+
+    // The sandbox login page shows its test password on screen
+    const password = await testdriver.extract('the password shown on the login page');
+
+    await testdriver.find('Username input field').click();
+    await testdriver.type('standard_user');
+    await testdriver.find('Password input field').click();
+    await testdriver.type(password, { secret: true });
+    await testdriver.find('Sign in button below the password field').click();
+
+    await testdriver.assert('the product list is shown');
+  });
+});
 ```
 
-<Info>TestDriver caches the found elements for better performance on later calls. Read more about [element caching here](/caching).</Info>
+When the statement is true, `assert()` resolves to `true`. When it's false, `assert()` throws and the test fails at that line. The error message starts with `AI Assertion failed:` and includes the AI's explanation.
 
-### Debugging Found Elements
+## Write statements the AI can judge
 
-After TestDriver finds an element, you can look at its properties for debug:
+The AI judges only what's visible on screen right now. Write statements a person could confirm by looking.
+
+- **Name the thing and its state.** `'the Submit button is disabled'`, not `'the form is invalid'`.
+- **Quote exact text** when the wording matters: `'the banner says "Order placed"'`.
+- **Check one thing per assertion.** Two short assertions tell you exactly which part failed.
+- **Assert outcomes, not steps.** Check what the user should see after an action, such as a confirmation message.
 
 ```javascript
-const button = await testdriver.find('submit button');
-console.log(button);
+// Visibility
+await testdriver.assert('the login form is visible');
+await testdriver.assert('the loading spinner is not visible');
+
+// Exact text
+await testdriver.assert('the error message says "Invalid email address"');
+
+// State of a control
+await testdriver.assert('the "Remember me" checkbox is checked');
+await testdriver.assert('the Country dropdown shows "United States"');
 ```
 
-This outputs all element properties:
+## Compare exact values with extract()
+
+Use `extract()` when you need the value itself, for example a number to compare or reuse. It returns a string, so convert it before comparing numbers:
 
 ```javascript
-{
-  description: 'submit button',
-  found: true,
-  x: 150,
-  y: 300,
-  coordinates: { x: 150, y: 300, centerX: 200, centerY: 320 },
-  threshold: 0.8,
-  confidence: 0.95,
-  similarity: 0.92,
-  selector: 'button[type="submit"]',
-  cache: {
-    hit: true,
-    strategy: 'pixel-diff',
-    createdAt: '2025-01-15T10:30:00Z',
-    diffPercent: 0.02,
-    imageUrl: 'https://...'
-  }
-}
+import { expect } from 'vitest';
+
+const itemCount = await testdriver.extract('the number of items shown on the cart icon');
+expect(Number(itemCount)).toBe(1);
 ```
 
-### Working with Multiple Elements
+Prefer `assert()` for yes-or-no checks. Reach for `extract()` when the test needs a value it can't know in advance, such as an order number:
 
-Find and interact with multiple elements:
+```javascript
+const orderNumber = await testdriver.extract('the order confirmation number');
+expect(orderNumber).toMatch(/^A-\d+$/);
+```
+
+## Check state before and after an action
+
+An assertion after an action only proves the end state. Assert the starting state too, so the test catches an action that did nothing:
 
 ```javascript
-// Find all matching elements
-const products = await testdriver.findAll('product card');
-console.log(`Found ${products.length} products`);
+await testdriver.assert('the cart is empty');
 
-// Interact with each
-for (const product of products) {
-  const title = await product.find('title text');
-  console.log('Product:', title.text);
+await testdriver.find('Add to cart button under the TestDriver Hat').click();
+
+await testdriver.assert('the cart icon shows 1 item');
+```
+
+To check that a value changed, extract it on both sides of the action:
+
+```javascript
+const before = await testdriver.extract('the cart total');
 
-  await product.find('add to cart button').click();
-}
+await testdriver.find('Add to cart button under the TestDriver Hat').click();
 
-// Or find specific element
-const firstProduct = products[0];
-await firstProduct.click();
+const after = await testdriver.extract('the cart total');
+expect(after).not.toBe(before);
 ```
 
-## Making Assertions
+## Wait before you assert
 
-Use AI-powered assertions to verify application state. TestDriver acts as a judge: it evaluates your natural-language statement against the current state of the app and returns a boolean plus reasoning explaining the verdict.
+`assert()` checks the screen once, at the moment you call it. If the result takes time to appear, wait for it with [`find()`](/find) first:
 
 ```javascript
-// Verify visibility
-await testdriver.assert('login page is displayed');
-await testdriver.assert('submit button is visible');
-await testdriver.assert('loading spinner is not visible');
-
-// Verify content
-await testdriver.assert('page title is "Welcome"');
-await testdriver.assert('success message says "Account created"');
-await testdriver.assert('error message contains "Invalid email"');
-
-// Verify state
-await testdriver.assert('checkbox is checked');
-await testdriver.assert('dropdown shows "United States"');
-await testdriver.assert('button is disabled');
-
-// Verify visual appearance
-await testdriver.assert('submit button is blue');
-await testdriver.assert('form has red border');
+await testdriver.find('Place order button').click();
+
+// Keep looking for up to 30 seconds while the confirmation page loads
+await testdriver.find('Order confirmation heading', { timeout: 30000 });
+await testdriver.assert('the confirmation shows an order number');
 ```
 
-<Info>Assertions are not cached and always re-evaluated to ensure accuracy.</Info>
+See [Perform actions](/performing-actions#wait-for-a-changing-ui) for other ways to wait.
+
+## How assertions are cached
+
+`assert()` results are [cached](/caching) like `find()` results. When the screen matches a passing earlier run, TestDriver reuses that result instead of asking the AI.
+
+`extract()` is never cached. It reads the screen every time.
 
-## Next
+To always ask the AI for one assertion, pass `threshold: -1`:
+
+```javascript
+await testdriver.assert('the clock shows the current time', { threshold: -1 });
+```
 
-<Card title="Adapt" icon="arrows-rotate" href="/performing-actions">
-  Drive your app forward by performing actions on the elements you've located and validated.
-</Card>
+## Next steps
+
+<CardGroup cols={2}>
+  <Card title="assert()" icon="check-circle" href="/assert">
+    Every option, return value, and error.
+  </Card>
+  <Card title="extract()" icon="wand-magic-sparkles" href="/extract">
+    Read values off the screen as strings.
+  </Card>
+  <Card title="Perform actions" icon="arrow-pointer" href="/performing-actions">
+    Click, type, and wait for a changing UI.
+  </Card>
+  <Card title="Debug with screenshots" icon="bug" href="/debugging-with-screenshots">
+    See what the screen showed when an assertion failed.
+  </Card>
+</CardGroup>

package/ai/skills/testdriver-mcp-setup/SKILL.md

@@ -1,43 +1,48 @@
 ---
 name: testdriver:mcp-setup
-description: Connect any AI client to TestDriver by adding one URL.
+description: Connect Claude, Cursor, VS Code, ChatGPT, or any MCP client to TestDriver with one URL.
 ---
 <!-- Generated from mcp-setup.mdx. DO NOT EDIT. -->
 
-TestDriver runs a hosted [Model Context Protocol](https://modelcontextprotocol.io) server. There is nothing to install and no API key to paste. Add one URL to your AI client, sign in through the browser, and the computer-use tools appear in chat.
+Connect your AI client to TestDriver's hosted [Model Context Protocol](https://modelcontextprotocol.io) server, and you can drive a real sandbox from chat.
+
+- Open your app.
+- Find elements, click, type, and assert.
+- Get a screenshot after every step.
+
+There's nothing to install and no API key to copy. Add one URL and sign in through your browser:
 
 ```text
 https://mcp.testdriver.ai/mcp
 ```
 
-<Info>
-**Prerequisites**
+## Before you begin
 
-- An MCP-compatible AI client (Claude, Claude Code, VS Code, Cursor, ChatGPT, and others)
-- A TestDriver account. [Create one for free](https://console.testdriver.ai/settings). You get 60 device minutes, no credit card required.
-</Info>
+- **An MCP client** that supports remote servers, such as Claude, Claude Code, Cursor, VS Code, or ChatGPT.
+- **A TestDriver account.** [Sign up at console.testdriver.ai](https://console.testdriver.ai). The free plan includes 30 testing minutes and needs no credit card.
 
 ## Add the server
 
-Most clients accept the URL directly. Pick yours below.
-
 <Tabs>
   <Tab title="Claude">
+
     1. Open **Settings → Connectors → Add custom connector**.
     2. Paste `https://mcp.testdriver.ai/mcp`.
-    3. Click **Connect** and sign in with TestDriver when the browser opens.
-  </Tab>
+    3. Click **Connect**, and sign in to TestDriver when the browser opens.
 
+  </Tab>
   <Tab title="Claude Code">
+
     ```bash
     claude mcp add --transport http testdriver https://mcp.testdriver.ai/mcp
     ```
 
-    Run `/mcp` in a session to start the browser login and confirm the tools are connected.
-  </Tab>
+    Then run `/mcp` in a Claude Code session to sign in and confirm the tools are connected.
 
+  </Tab>
   <Tab title="VS Code">
-    Add this to `.vscode/mcp.json` in your project. VS Code uses the `servers` key:
+
+    Add this to `.vscode/mcp.json` in your project:
 
     ```json .vscode/mcp.json
     {
@@ -50,11 +55,12 @@ Most clients accept the URL directly. Pick yours below.
     }
     ```
 
-    VS Code prompts you to authorize the server the first time you use it.
-  </Tab>
+    VS Code asks you to sign in the first time it uses the server.
 
+  </Tab>
   <Tab title="Cursor">
-    Add this to `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global):
+
+    Add this to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for all projects:
 
     ```json .cursor/mcp.json
     {
@@ -65,16 +71,18 @@ Most clients accept the URL directly. Pick yours below.
       }
     }
     ```
-  </Tab>
 
+  </Tab>
   <Tab title="ChatGPT">
+
     1. Open **Settings → Connectors → Create**.
     2. Paste `https://mcp.testdriver.ai/mcp` as the MCP server URL.
-    3. Choose OAuth for authentication and sign in with TestDriver.
-  </Tab>
+    3. Choose **OAuth** for authentication, and sign in to TestDriver.
 
+  </Tab>
   <Tab title="Other clients">
-    Any spec-compliant client works. Point it at the URL using the streamable HTTP transport:
+
+    Any client that supports the Streamable HTTP transport and OAuth sign-in can connect. Point it at the URL:
 
     ```json
     {
@@ -85,58 +93,58 @@ Most clients accept the URL directly. Pick yours below.
       }
     }
     ```
+
   </Tab>
 </Tabs>
 
-<Note>
-Clients disagree on the top-level key. VS Code uses `servers`, most others use `mcpServers`, Zed uses `context_servers`, and Codex uses TOML `[mcp_servers]`. If the tools do not show up, check the key before anything else.
-</Note>
-
 ## Sign in
 
-The server speaks OAuth 2.1 and advertises its authorization server via [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728) protected-resource metadata:
+The first time your client connects, it opens a browser window where you sign in to TestDriver.
 
-```text
-https://mcp.testdriver.ai/.well-known/oauth-protected-resource
-```
+Every tool call afterward runs against the team you signed in with. Its usage counts toward that team's plan.
 
-Spec-compliant clients read that metadata, open a browser, and complete the handshake for you. Every tool is scoped to the team you sign in with.
+Clients find the sign-in page automatically. The server uses OAuth 2.1 and publishes its authorization details as [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728) protected-resource metadata at `https://mcp.testdriver.ai/.well-known/oauth-protected-resource`.
 
-## Verify it works
+## Try it
 
 Ask your client to drive a browser:
 
 ```text
-Use TestDriver to open example.com and assert the page title is visible.
+Use TestDriver to open example.com and assert that the page heading is visible.
 ```
 
-The agent starts a sandbox and returns a screenshot with every action. Each connection gets its own isolated sandbox, so multiple people and multiple chats can run tests at the same time without interfering.
+The agent starts a sandbox, opens Chrome, and returns a screenshot after each action.
 
-## What you get
+Every connection gets its own sandbox. Several people or chats can run at once without affecting each other.
 
-The hosted server exposes the full live tool set:
+## Available tools
 
-| Tool | Purpose |
+| Tools | What they do |
 | --- | --- |
-| `session_start`, `session_status`, `session_extend` | Start a sandbox, check its health, add more time |
-| `find`, `findall`, `find_and_click` | Locate elements by plain-English description |
-| `click`, `hover`, `type`, `press_keys`, `scroll` | Perform actions |
-| `assert`, `check` | Ask yes/no questions about the screen |
-| `screenshot`, `exec`, `wait`, `focus_application` | Capture state, run commands, pause |
+| `session_start`, `session_status`, `session_extend` | Start a sandbox, check how long it has left, and keep it alive longer |
+| `find`, `findall`, `find_and_click` | Locate elements from a plain-language description |
+| `click`, `hover`, `type`, `press_keys`, `scroll` | Act on the screen with the mouse and keyboard |
+| `assert`, `check` | Ask yes-or-no questions about what's on screen |
+| `screenshot`, `wait`, `exec`, `focus_application` | Capture the screen, pause, run a command, or switch apps |
+| `list_test_runs`, `get_test_run_detail` | Look up past test runs and their results |
+
+Each action also returns the SDK code for that step. The agent uses it to assemble a runnable [Vitest](https://vitest.dev) test as it works.
 
-Every action also returns the SDK code for that step, so the agent can write a runnable [Vitest](https://vitest.dev) test as it goes.
+To turn that test into a file you run locally and in CI, [set up a project](/quickstart-manual).
 
-## Local server for CI and automation
+## Run the server locally
 
-For headless automation, or when you would rather use an API key than a browser login, run the server as a local stdio process:
+Run the server as a local stdio process for headless automation, or to authenticate with an API key instead of a browser sign-in:
 
 ```bash
 npx -p testdriverai testdriverai-mcp
 ```
 
-It reads `TD_API_KEY` from the environment. Generate a key at [console.testdriver.ai/settings](https://console.testdriver.ai/settings).
+It reads your API key from the `TD_API_KEY` environment variable. Create a key at [console.testdriver.ai/team/api-keys](https://console.testdriver.ai/team/api-keys).
 
-```json
+Then configure your client to start the server:
+
+```json .mcp.json
 {
   "mcpServers": {
     "testdriver": {
@@ -150,42 +158,51 @@ It reads `TD_API_KEY` from the environment. Generate a key at [console.testdrive
 ```
 
 <Tip>
-`npx testdriverai init` writes this config for you, plus the TestDriver agent, skills, an example test, and a GitHub Actions workflow. See [Setting up your workspace](/quickstart-manual).
+  `npx testdriverai init` writes this config for your client, along with the TestDriver agent, skills, an example test, and a GitHub Actions workflow. See [Set up a project](/quickstart-manual).
 </Tip>
 
 ## Troubleshooting
 
 <AccordionGroup>
-  <Accordion title="The tools do not appear in my client">
-    Confirm the URL is exactly `https://mcp.testdriver.ai/mcp`, check that you used the correct top-level config key for your client, and restart the client. Most clients read their MCP config only at startup.
+  <Accordion title="The TestDriver tools don't appear in my client">
+
+    Check that the URL is exactly `https://mcp.testdriver.ai/mcp`. Then restart your client, since most clients read their MCP configuration only at startup.
+
   </Accordion>
+  <Accordion title="The browser sign-in never finishes">
+
+    Your client needs to support OAuth 2.1 with Dynamic Client Registration.
+
+    Clients that only support local (stdio) servers can't connect to the hosted URL. [Run the server locally](#run-the-server-locally) with an API key instead.
 
-  <Accordion title="The browser login never completes">
-    Your client must support OAuth 2.1 with Dynamic Client Registration. Older clients, and clients that only support stdio servers, cannot connect to the hosted URL. Use the local server with `TD_API_KEY` instead.
   </Accordion>
+  <Accordion title="The sandbox expired in the middle of a session">
+
+    A sandbox shuts down after 5 minutes with no activity.
+
+    Ask the agent to call `session_extend` to keep it alive, or `session_start` to get a new one.
 
-  <Accordion title="I ran out of device minutes">
-    Sandbox time is billed per minute. Check your usage and plan at [console.testdriver.ai](https://console.testdriver.ai).
   </Accordion>
+  <Accordion title="I ran out of testing minutes">
+
+    Sandbox time counts against your plan's testing minutes. Check your usage and upgrade at [console.testdriver.ai/usage](https://console.testdriver.ai/usage).
 
-  <Accordion title="The sandbox expired mid-session">
-    Sessions time out after a period of inactivity. Ask the agent to call `session_extend` before it expires, or `session_start` to get a fresh sandbox.
   </Accordion>
 </AccordionGroup>
 
 ## Next steps
 
 <CardGroup cols={2}>
-  <Card title="Generating tests" icon="wand-magic-sparkles" href="/generating-tests" arrow horizontal>
-    Prompting patterns that get the best tests out of the agent.
+  <Card title="Generate tests" icon="wand-magic-sparkles" href="/generating-tests">
+    Prompts that get the best tests out of the agent.
   </Card>
-  <Card title="Setting up your workspace" icon="wrench" href="/quickstart-manual" arrow horizontal>
-    Add the SDK to a project so generated tests run locally and in CI.
+  <Card title="Set up a project" icon="wrench" href="/quickstart-manual">
+    Save generated tests and run them locally and in CI.
   </Card>
-  <Card title="Walkthrough" icon="map" href="/provision" arrow horizontal>
-    Provision apps, locate elements, perform actions, and make assertions.
+  <Card title="Provision apps" icon="rocket" href="/provision">
+    Launch browsers, extensions, and desktop apps in the sandbox.
   </Card>
-  <Card title="CI/CD" icon="circle-play" href="/ci-cd" arrow horizontal>
+  <Card title="Run in CI/CD" icon="code-branch" href="/ci-cd">
     Run your tests on every pull request.
   </Card>
 </CardGroup>