@testdriverai/mcp 7.9.150-test → 7.9.152-test
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ai/skills/testdriver-caching/SKILL.md +13 -6
- package/ai/skills/testdriver-ci-cd/SKILL.md +20 -34
- package/ai/skills/testdriver-client/SKILL.md +5 -1
- package/ai/skills/testdriver-debugging-with-screenshots/SKILL.md +14 -4
- package/ai/skills/testdriver-find/SKILL.md +2 -0
- package/ai/skills/testdriver-generating-tests/SKILL.md +159 -8
- package/ai/skills/testdriver-machine-setup/SKILL.md +75 -8
- package/ai/skills/testdriver-making-assertions/SKILL.md +78 -2
- package/ai/skills/testdriver-performing-actions/SKILL.md +97 -2
- package/ai/skills/testdriver-quickstart/SKILL.md +47 -28
- package/ai/skills/testdriver-wait/SKILL.md +1 -1
- package/ai/skills/testdriver:generating-tests/SKILL.md +0 -11
- package/docs/changelog.mdx +1 -1
- package/docs/docs.json +14 -39
- package/docs/v7/assert.mdx +0 -1
- package/docs/v7/caching.mdx +14 -10
- package/docs/v7/ci-cd.mdx +20 -34
- package/docs/v7/client.mdx +1 -1
- package/docs/v7/copilot/auto-healing.mdx +167 -18
- package/docs/v7/copilot/running-tests.mdx +915 -54
- package/docs/v7/debugging-with-screenshots.mdx +17 -7
- package/docs/v7/generating-tests.mdx +166 -10
- package/docs/v7/making-assertions.mdx +81 -4
- package/docs/v7/performing-actions.mdx +100 -4
- package/docs/v7/quickstart.mdx +299 -21
- package/docs/v7/wait.mdx +1 -1
- package/package.json +3 -3
- package/docs/v7/ai/agent.mdx +0 -72
- package/docs/v7/ai/mcp.mdx +0 -228
- package/docs/v7/ai/skills.mdx +0 -73
- package/docs/v7/ai.mdx +0 -205
- package/docs/v7/copilot/creating-tests.mdx +0 -156
- package/docs/v7/copilot/github.mdx +0 -143
- package/docs/v7/copilot/setup.mdx +0 -143
- package/docs/v7/device-config.mdx +0 -317
- package/docs/v7/locating-elements.mdx +0 -71
- package/docs/v7/machine-setup.mdx +0 -331
- package/docs/v7/running-tests.mdx +0 -185
- package/docs/v7/waiting-for-elements.mdx +0 -90
|
@@ -1,11 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:caching
|
|
3
|
-
description:
|
|
3
|
+
description: How TestDriver learns your app and caches what it discovers for instant, deterministic replays
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from caching.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
TestDriver
|
|
7
|
+
Once the agent has [explored your app](/v7/generating-tests), TestDriver remembers what it found. Every element the AI vision agent discovers is cached with a vision fingerprint—a perceptual hash of the screen state where it was located. On the next run, TestDriver matches against that cache instead of calling the AI again. Passing tests replay instantly, deterministically, and cheaply.
|
|
8
8
|
|
|
9
|
+
This learning is what makes TestDriver fast. Intelligent caching delivers up to **1.7x faster** test execution by skipping redundant AI vision analysis—the agent only thinks when it sees something new.
|
|
9
10
|
|
|
10
11
|
```javascript
|
|
11
12
|
// First run: builds cache
|
|
@@ -17,7 +18,7 @@ await testdriver.find('submit button');
|
|
|
17
18
|
|
|
18
19
|
## Automatic Caching
|
|
19
20
|
|
|
20
|
-
|
|
21
|
+
Learning is enabled automatically with zero configuration. The cache key—the fingerprint TestDriver uses to recognize what it already knows—is computed from:
|
|
21
22
|
|
|
22
23
|
- **File hash**: SHA-256 hash of the test file contents
|
|
23
24
|
- **Selector prompt**: The exact text description passed to `find()`
|
|
@@ -53,7 +54,7 @@ You can clear the cache within the TestDriver console. There, you'll also find p
|
|
|
53
54
|
|
|
54
55
|
## Debugging Cache Hits and Misses
|
|
55
56
|
|
|
56
|
-
You can track cache performance in your tests:
|
|
57
|
+
You can track what TestDriver has learned by inspecting cache performance in your tests:
|
|
57
58
|
|
|
58
59
|
```javascript
|
|
59
60
|
test('monitor cache performance', async (context) => {
|
|
@@ -75,7 +76,7 @@ test('monitor cache performance', async (context) => {
|
|
|
75
76
|
|
|
76
77
|
## Configuring the Cache
|
|
77
78
|
|
|
78
|
-
You can configure
|
|
79
|
+
You can configure how TestDriver learns globally when initializing TestDriver:
|
|
79
80
|
|
|
80
81
|
```javascript
|
|
81
82
|
import { TestDriver } from 'testdriverai';
|
|
@@ -103,7 +104,7 @@ await testdriver.find('submit button', {
|
|
|
103
104
|
|
|
104
105
|
## Caching with Variables
|
|
105
106
|
|
|
106
|
-
Custom cache keys prevent cache pollution when using variables in prompts, dramatically improving cache hit rates.
|
|
107
|
+
Custom cache keys prevent cache pollution when using variables in prompts, dramatically improving cache hit rates—so TestDriver reuses what it learned even when your data changes.
|
|
107
108
|
|
|
108
109
|
```javascript
|
|
109
110
|
// ❌ Without cache key - creates new cache for each variable value
|
|
@@ -122,3 +123,9 @@ await testdriver.find(`order ${orderId} status`, {
|
|
|
122
123
|
cacheKey: 'order-status' // Same cache for all orders
|
|
123
124
|
});
|
|
124
125
|
```
|
|
126
|
+
|
|
127
|
+
## Next
|
|
128
|
+
|
|
129
|
+
<Card href="/v7/copilot/running-tests" title="Run" icon="play">
|
|
130
|
+
Now that TestDriver has learned your app, run your tests in CI and locally—replaying the cache for fast, deterministic results.
|
|
131
|
+
</Card>
|
|
@@ -6,37 +6,33 @@ description: Run TestDriver tests in CI/CD with parallel execution and cross-pla
|
|
|
6
6
|
|
|
7
7
|
TestDriver integrates seamlessly with popular CI providers, enabling automated end-to-end testing on every push and pull request.
|
|
8
8
|
|
|
9
|
-
##
|
|
9
|
+
## Authentication
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
On **GitHub Actions, prefer OIDC** via the published `testdriverai/action` —
|
|
12
|
+
there's no `TD_API_KEY` secret to store, copy, or rotate. The action proves the
|
|
13
|
+
workflow is running inside your org and TestDriver exchanges that proof for your
|
|
14
|
+
team's key at run time. See the GitHub Actions tab below.
|
|
12
15
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
</Step>
|
|
17
|
-
<Step title="Add Secret to Your CI Provider">
|
|
18
|
-
Add `TD_API_KEY` as a secret environment variable in your CI provider's settings.
|
|
19
|
-
</Step>
|
|
20
|
-
</Steps>
|
|
16
|
+
For other CI providers (or self-hosted runners without OIDC), fall back to a
|
|
17
|
+
stored API key from [console.testdriver.ai/team](https://console.testdriver.ai/team),
|
|
18
|
+
added as a `TD_API_KEY` secret in your CI provider's settings.
|
|
21
19
|
|
|
22
20
|
<Note>
|
|
23
|
-
Never commit your API key directly in code. Always use your CI provider's secrets management.
|
|
21
|
+
Never commit your API key directly in code. Always use OIDC or your CI provider's secrets management.
|
|
24
22
|
</Note>
|
|
25
23
|
|
|
26
24
|
## CI Provider Examples
|
|
27
25
|
|
|
28
26
|
<Tabs>
|
|
29
27
|
<Tab title="GitHub Actions">
|
|
30
|
-
### Authenticate with OIDC (recommended)
|
|
28
|
+
### Authenticate with OIDC via `testdriverai/action` (recommended)
|
|
31
29
|
|
|
32
|
-
|
|
30
|
+
Use the published [`testdriverai/action`](https://github.com/testdriverai/action) — it mints the OIDC token, exchanges it for your team's API key, and exports `TD_API_KEY` for the steps that follow. **No `TD_API_KEY` secret to store or rotate.**
|
|
33
31
|
|
|
34
32
|
<Note>
|
|
35
|
-
|
|
33
|
+
One-time setup: authorize the [TestDriver GitHub App](https://console.testdriver.ai) for your org so the org → team binding exists. If your org authorized the App before OIDC support shipped, re-authorize once. If the App isn't authorized, the action fails with a console link (or falls back to the `api-key` secret if you provide one).
|
|
36
34
|
</Note>
|
|
37
35
|
|
|
38
|
-
Grant the job the `id-token: write` permission and exchange the OIDC token for your API key:
|
|
39
|
-
|
|
40
36
|
```yaml .github/workflows/testdriver.yml
|
|
41
37
|
name: TestDriver Tests
|
|
42
38
|
|
|
@@ -50,7 +46,7 @@ TestDriver requires an API key to authenticate with the TestDriver cloud. Store
|
|
|
50
46
|
test:
|
|
51
47
|
runs-on: ubuntu-latest
|
|
52
48
|
permissions:
|
|
53
|
-
id-token: write #
|
|
49
|
+
id-token: write # REQUIRED to mint an OIDC token
|
|
54
50
|
contents: read
|
|
55
51
|
|
|
56
52
|
steps:
|
|
@@ -63,28 +59,18 @@ TestDriver requires an API key to authenticate with the TestDriver cloud. Store
|
|
|
63
59
|
|
|
64
60
|
- run: npm ci
|
|
65
61
|
|
|
66
|
-
- name: Authenticate to TestDriver
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=testdriver" | jq -r '.value')
|
|
71
|
-
API_KEY=$(curl -sS -X POST https://api.testdriver.ai/github/actions/auth \
|
|
72
|
-
-H "Content-Type: application/json" \
|
|
73
|
-
-d "{\"token\":\"$OIDC_TOKEN\"}" | jq -r '.apiKey')
|
|
74
|
-
echo "::add-mask::$API_KEY"
|
|
75
|
-
echo "TD_API_KEY=$API_KEY" >> "$GITHUB_ENV"
|
|
62
|
+
- name: Authenticate to TestDriver
|
|
63
|
+
uses: testdriverai/action@stable # pin @stable / @canary / @test to your SDK channel
|
|
64
|
+
with:
|
|
65
|
+
api-key: ${{ secrets.TD_API_KEY }} # optional fallback if OIDC isn't set up
|
|
76
66
|
|
|
77
67
|
- name: Run TestDriver tests
|
|
78
|
-
|
|
79
|
-
TD_API_KEY: ${{ env.TD_API_KEY }}
|
|
80
|
-
run: vitest --run
|
|
68
|
+
run: npx vitest run
|
|
81
69
|
```
|
|
82
70
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
### Adding Secrets
|
|
71
|
+
### Stored-key fallback
|
|
86
72
|
|
|
87
|
-
|
|
73
|
+
Only if you can't use OIDC (e.g. self-hosted runners without an OIDC provider). Add the key as a secret and pass it via `env`:
|
|
88
74
|
|
|
89
75
|
1. Navigate to your GitHub repository
|
|
90
76
|
2. Go to **Settings** → **Secrets and variables** → **Actions**
|
|
@@ -53,7 +53,11 @@ const testdriver = new TestDriver(apiKey, options)
|
|
|
53
53
|
</ParamField>
|
|
54
54
|
|
|
55
55
|
<ParamField path="reconnect" type="boolean" default="false">
|
|
56
|
-
|
|
56
|
+
Reattach to the last used sandbox instead of creating a new one. When `true`, the SDK reads the sandbox id from `.testdriver/last-sandbox` (written automatically on every successful connect) and rejoins that VM. Provision methods (`chrome`, `vscode`, `installer`, etc.) are skipped because the application is already running. The previous sandbox must still be alive — see [`keepAlive`](#keepalive) and the [Run guide](/v7/copilot/running-tests#keeping-machines-alive-between-runs).
|
|
57
|
+
</ParamField>
|
|
58
|
+
|
|
59
|
+
<ParamField path="sandboxId" type="string">
|
|
60
|
+
Reattach to a specific sandbox id instead of the one recorded in `.testdriver/last-sandbox`. Use this for CI matrices or to pin a chain of tests to a known VM. Implies `reconnect: true` behavior (provision calls are skipped).
|
|
57
61
|
</ParamField>
|
|
58
62
|
|
|
59
63
|
<ParamField path="preview" type="string" default="browser">
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:debugging-with-screenshots
|
|
3
|
-
description:
|
|
3
|
+
description: Diagnose failing tests with screenshots, replays, and logs
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from debugging-with-screenshots.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
TestDriver MCP provides powerful commands to view and analyze screenshots saved during test execution
|
|
9
|
+
When a test fails, debug it by seeing exactly what happened — don't guess. TestDriver captures screenshots, video replays, and logs as your test runs, so you can replay the moment of failure instead of squinting at a stack trace. TestDriver MCP provides powerful commands to view and analyze the screenshots saved during test execution, enabling rapid debugging, test development, and comparison workflows without manually opening image files.
|
|
10
10
|
|
|
11
11
|
<Note>
|
|
12
12
|
**Automatic Screenshots (Default: Enabled)**: TestDriver automatically captures screenshots before and after every command. Screenshots are named with the line number and action, making it easy to trace exactly which line of code produced each screenshot. For example: `001-click-before-L42-submit-button.png`
|
|
@@ -122,7 +122,7 @@ view_local_screenshot({ path: "/full/path/to/screenshot.png" })
|
|
|
122
122
|
|
|
123
123
|
### Test Debugging After Failures
|
|
124
124
|
|
|
125
|
-
When a test fails, use powerful filtering to quickly find
|
|
125
|
+
When a test fails, you don't have to wonder what went wrong — use powerful filtering to quickly find the screenshots that show exactly what happened:
|
|
126
126
|
|
|
127
127
|
**1. Find screenshots at the failing line:**
|
|
128
128
|
|
|
@@ -358,7 +358,7 @@ This helps verify your test logic before running the full test file.
|
|
|
358
358
|
|
|
359
359
|
### After Test Runs
|
|
360
360
|
|
|
361
|
-
When tests fail or behave unexpectedly:
|
|
361
|
+
When tests fail or behave unexpectedly, replay what happened step by step:
|
|
362
362
|
|
|
363
363
|
1. **Run the test** with `vitest run tests/my-test.test.mjs`
|
|
364
364
|
2. **List screenshots** using `list_local_screenshots`
|
|
@@ -408,8 +408,18 @@ When tests fail or behave unexpectedly:
|
|
|
408
408
|
</Accordion>
|
|
409
409
|
</AccordionGroup>
|
|
410
410
|
|
|
411
|
+
## Where this fits in the Guide
|
|
412
|
+
|
|
413
|
+
Debugging is what you reach for when a [Run](/v7/copilot/running-tests) goes sideways or a [Validate](/v7/making-assertions) assertion fails — the screenshots show you precisely what the AI saw before it acted. Once you've diagnosed the failure, the next step is to stop it from recurring.
|
|
414
|
+
|
|
411
415
|
## Related
|
|
412
416
|
|
|
413
417
|
- [screenshot()](/v7/screenshot) - Capture screenshots during test execution
|
|
414
418
|
- [Dashcam](/v7/dashcam) - Record full test sessions with video and logs
|
|
415
419
|
- [assert()](/v7/assert) - Make AI-powered assertions that benefit from screenshot context
|
|
420
|
+
|
|
421
|
+
## Next
|
|
422
|
+
|
|
423
|
+
<Card title="Prevent" icon="shield-check" href="/v7/copilot/auto-healing">
|
|
424
|
+
You've seen what went wrong — now keep it from happening again. Let auto-healing repair flaky tests automatically before they fail your suite.
|
|
425
|
+
</Card>
|
|
@@ -401,6 +401,8 @@ await element.click();
|
|
|
401
401
|
|
|
402
402
|
## Cache Options
|
|
403
403
|
|
|
404
|
+
When a test completes successfully, the result of each `find()` is cached. On later runs, TestDriver reuses the cached match instead of making a fresh AI call, which significantly speeds up locating the same element. The cache lives in your [dashboard](https://console.testdriver.ai/cache) and is shared across runs — see the [Cache](/v7/cache) page for how matching, thresholds, and invalidation work.
|
|
405
|
+
|
|
404
406
|
Control caching behavior to optimize performance, especially when using dynamic variables in prompts.
|
|
405
407
|
|
|
406
408
|
### Custom Cache Key
|
|
@@ -1,12 +1,16 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:generating-tests
|
|
3
|
-
description:
|
|
3
|
+
description: Generate tests by exploring your app with the AI vision agent
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from generating-tests.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
+
Start by exploring your app. This is where tests begin: you describe a flow in plain English, and the TestDriver vision agent clicks, types, and reads the screen to figure it out — then writes the test for you. No selectors, no DOM, no brittle locators. Just describe what you want to test and let the agent discover the rest.
|
|
8
|
+
|
|
9
|
+
There are two ways to explore: chatting interactively with your AI assistant through the TestDriver MCP server, or handing a coding agent our instructions file and prompting it to generate a test.
|
|
10
|
+
|
|
7
11
|
## Instructions for Coding Agents
|
|
8
12
|
|
|
9
|
-
We recommend starting with [our quickstart](./quickstart) then supplying your coding agent with our agent instructions file.
|
|
13
|
+
We recommend starting with [our quickstart](./quickstart), then supplying your coding agent with our agent instructions file.
|
|
10
14
|
|
|
11
15
|
<Card title="TestDriver Agent Instructions" icon="link" arrow="true" horizontal href="https://github.com/testdriverai/testdriverai/blob/main/ai/agents/testdriver.md?plain=1">
|
|
12
16
|
Copy the current version of our agent instructions to provide your coding agent with up-to-date instructions on how to generate TestDriver tests.
|
|
@@ -24,13 +28,160 @@ Push Submit button
|
|
|
24
28
|
Verify new page contains expected text 'logged in'
|
|
25
29
|
```
|
|
26
30
|
|
|
27
|
-
|
|
31
|
+
<Info>Explicit commands are preferred for production tests, as they are cheaper, faster, and more reliable.</Info>
|
|
32
|
+
|
|
33
|
+
## Start a Conversation
|
|
28
34
|
|
|
29
|
-
|
|
35
|
+
With TestDriver's MCP server and your AI assistant (GitHub Copilot, Cursor, or Claude Desktop), you can create tests by chatting with an AI agent. The agent spawns a virtual machine, executes actions, and writes test code for you.
|
|
30
36
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
37
|
+
Open your AI assistant's chat. If your project has no other agents configured, the TestDriver agent is used by default. Otherwise, select **testdriver** from the agent dropdown in the chat panel.
|
|
38
|
+
|
|
39
|
+
Describe what you want to test:
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
Create a test that logs into my app at https://myapp.com
|
|
34
43
|
```
|
|
35
44
|
|
|
36
|
-
|
|
45
|
+
The agent will:
|
|
46
|
+
1. Start a new session and spawn a Linux virtual machine
|
|
47
|
+
2. Launch Chrome and navigate to your URL
|
|
48
|
+
3. Execute actions based on your instructions
|
|
49
|
+
4. Write the test code to a `.test.mjs` file
|
|
50
|
+
|
|
51
|
+
<Note>
|
|
52
|
+
The TestDriver agent appears in the agent selection dropdown if you have other agents configured (like `copilot-instructions.md` or other `.agent.md` files). Make sure to select **testdriver** to use TestDriver's MCP tools.
|
|
53
|
+
</Note>
|
|
54
|
+
|
|
55
|
+
## Live Preview Panel
|
|
56
|
+
|
|
57
|
+
When the agent starts a session, a **live preview panel** opens in your editor. This lets you:
|
|
58
|
+
|
|
59
|
+
- **Watch tests execute** in real-time
|
|
60
|
+
- **Interact with the VM** — click, type, and navigate manually
|
|
61
|
+
- **Debug issues** — see exactly what the AI sees
|
|
62
|
+
|
|
63
|
+
<Note>
|
|
64
|
+
The live preview panel requires the TestDriver VS Code extension. Set `TD_PREVIEW=ide` in your MCP configuration to enable it. See the [Run page](/v7/copilot/running-tests) for all device and preview options.
|
|
65
|
+
</Note>
|
|
66
|
+
|
|
67
|
+
## Interactive Workflow
|
|
68
|
+
|
|
69
|
+
The recommended workflow for exploring and creating tests:
|
|
70
|
+
|
|
71
|
+
<Steps>
|
|
72
|
+
<Step title="Describe Your Test">
|
|
73
|
+
Tell the agent what you want to test:
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
Test the checkout flow on https://shop.example.com
|
|
77
|
+
```
|
|
78
|
+
</Step>
|
|
79
|
+
|
|
80
|
+
<Step title="Watch and Guide">
|
|
81
|
+
The agent will start executing. Watch the preview panel and provide guidance if needed:
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
The login modal appeared. Use test@example.com and password123
|
|
85
|
+
```
|
|
86
|
+
</Step>
|
|
87
|
+
|
|
88
|
+
<Step title="Add Assertions">
|
|
89
|
+
Ask the agent to verify specific conditions:
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
Assert that the order confirmation page is shown with the order number
|
|
93
|
+
```
|
|
94
|
+
</Step>
|
|
95
|
+
|
|
96
|
+
<Step title="Review the Code">
|
|
97
|
+
The agent writes test code as it works. Review the generated `.test.mjs` file and make any adjustments.
|
|
98
|
+
</Step>
|
|
99
|
+
</Steps>
|
|
100
|
+
|
|
101
|
+
## Example Conversation
|
|
102
|
+
|
|
103
|
+
Here's a complete example of exploring and creating a login test:
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
You: Create a test for the login flow at https://console.testdriver.ai
|
|
107
|
+
|
|
108
|
+
Agent: I'll create a test for the login flow. Starting a session...
|
|
109
|
+
[Screenshot: Login page loaded]
|
|
110
|
+
|
|
111
|
+
I can see the login page. Let me find the email input.
|
|
112
|
+
[Screenshot: Email input highlighted]
|
|
113
|
+
|
|
114
|
+
Now I'll type the email and find the password field...
|
|
115
|
+
|
|
116
|
+
You: Use test@example.com for the email
|
|
117
|
+
|
|
118
|
+
Agent: Got it. Typing test@example.com...
|
|
119
|
+
[Screenshot: Email entered]
|
|
120
|
+
|
|
121
|
+
Now entering the password and clicking Sign In...
|
|
122
|
+
[Screenshot: Dashboard loaded]
|
|
123
|
+
|
|
124
|
+
✅ Login successful! I've written the test to tests/login.test.mjs
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Generated Test Code
|
|
128
|
+
|
|
129
|
+
The agent generates standard Vitest test files:
|
|
130
|
+
|
|
131
|
+
```javascript tests/login.test.mjs
|
|
132
|
+
import { describe, expect, it } from "vitest";
|
|
133
|
+
import { TestDriver } from "testdriverai/vitest/hooks";
|
|
134
|
+
|
|
135
|
+
describe("Login Flow", () => {
|
|
136
|
+
it("should log in successfully", async (context) => {
|
|
137
|
+
const testdriver = TestDriver(context);
|
|
138
|
+
|
|
139
|
+
await testdriver.provision.chrome({
|
|
140
|
+
url: "https://console.testdriver.ai"
|
|
141
|
+
});
|
|
142
|
+
|
|
143
|
+
const emailInput = await testdriver.find("email input field");
|
|
144
|
+
await emailInput.click();
|
|
145
|
+
await testdriver.type("test@example.com");
|
|
146
|
+
|
|
147
|
+
const passwordInput = await testdriver.find("password input field");
|
|
148
|
+
await passwordInput.click();
|
|
149
|
+
await testdriver.type("password123");
|
|
150
|
+
|
|
151
|
+
const signInButton = await testdriver.find("Sign In button");
|
|
152
|
+
await signInButton.click();
|
|
153
|
+
|
|
154
|
+
const result = await testdriver.assert("dashboard is visible");
|
|
155
|
+
expect(result).toBeTruthy();
|
|
156
|
+
});
|
|
157
|
+
});
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
## Tips for Better Tests
|
|
161
|
+
|
|
162
|
+
<AccordionGroup>
|
|
163
|
+
<Accordion title="Be specific with element descriptions">
|
|
164
|
+
Instead of "click the button", say "click the blue Sign In button in the header". More context helps the AI find the right element.
|
|
165
|
+
</Accordion>
|
|
166
|
+
<Accordion title="Add waits for dynamic content">
|
|
167
|
+
If your app has animations or loading states, tell the agent to wait:
|
|
168
|
+
```
|
|
169
|
+
Wait for the loading spinner to disappear before continuing
|
|
170
|
+
```
|
|
171
|
+
</Accordion>
|
|
172
|
+
<Accordion title="Use assertions liberally">
|
|
173
|
+
Add assertions after each major action to catch regressions early:
|
|
174
|
+
```
|
|
175
|
+
Assert that the product was added to the cart
|
|
176
|
+
```
|
|
177
|
+
</Accordion>
|
|
178
|
+
<Accordion title="Break complex flows into steps">
|
|
179
|
+
For long workflows, create the test incrementally and verify each step works before moving on.
|
|
180
|
+
</Accordion>
|
|
181
|
+
</AccordionGroup>
|
|
182
|
+
|
|
183
|
+
## Next
|
|
184
|
+
|
|
185
|
+
<Card title="Learn" icon="brain" arrow="true" horizontal href="/v7/caching">
|
|
186
|
+
Once the agent has explored your app, TestDriver caches what it discovers so your tests replay instantly without re-reasoning over the screen every time.
|
|
187
|
+
</Card>
|
|
@@ -87,6 +87,16 @@ const testdriver = TestDriver(context, {
|
|
|
87
87
|
|
|
88
88
|
Windows (and Linux) cold starts can be expensive if you're iterating quickly. Use `keepAlive` + `reconnect` to reuse the same VM across multiple test runs.
|
|
89
89
|
|
|
90
|
+
### How it works
|
|
91
|
+
|
|
92
|
+
Every time the SDK successfully connects to a sandbox, it records the sandbox id in `.testdriver/last-sandbox` inside your project directory. The next test that opts in with `reconnect: true` reads that file and reattaches automatically — no manual id tracking required.
|
|
93
|
+
|
|
94
|
+
Provision calls (`testdriver.provision.chrome(...)`, `vscode(...)`, etc.) are **skipped** when reconnecting, because the application is already running inside the sandbox from the previous run.
|
|
95
|
+
|
|
96
|
+
<Note>
|
|
97
|
+
`.testdriver/last-sandbox` is already covered by the default TestDriver `.gitignore`. Don't commit it.
|
|
98
|
+
</Note>
|
|
99
|
+
|
|
90
100
|
### Step 1 — Start the machine with a long `keepAlive`
|
|
91
101
|
|
|
92
102
|
```javascript
|
|
@@ -102,30 +112,87 @@ await testdriver.provision.chrome({ url: "https://example.com" });
|
|
|
102
112
|
|
|
103
113
|
When this test finishes, the sandbox stays running for 30 minutes instead of being terminated immediately.
|
|
104
114
|
|
|
105
|
-
### Step 2 —
|
|
115
|
+
### Step 2 — Reattach automatically with `reconnect: true`
|
|
106
116
|
|
|
107
117
|
```javascript
|
|
108
118
|
// second.test.mjs
|
|
109
119
|
const testdriver = TestDriver(context, {
|
|
110
120
|
os: "windows",
|
|
121
|
+
reconnect: true, // ← reads .testdriver/last-sandbox
|
|
111
122
|
keepAlive: 30 * 60 * 1000,
|
|
112
123
|
});
|
|
113
124
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
// provision.chrome() is automatically skipped — Chrome is already open
|
|
125
|
+
// No provision call — Chrome is already open from the previous run.
|
|
117
126
|
await testdriver.find("Sign In button").click();
|
|
118
127
|
```
|
|
119
128
|
|
|
120
|
-
|
|
129
|
+
### Step 2 (alternative) — Reattach to an explicit id
|
|
130
|
+
|
|
131
|
+
If you need to pin to a specific sandbox (CI matrix, multiple chains in parallel, etc.) pass the id directly:
|
|
132
|
+
|
|
133
|
+
```javascript
|
|
134
|
+
await testdriver.connect({ sandboxId: "sandbox-abc123" });
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
When reattaching to a sandbox:
|
|
121
138
|
- You reuse a specific running machine directly
|
|
122
|
-
- You
|
|
123
|
-
- You
|
|
139
|
+
- You continue from the app state created in the earlier run
|
|
140
|
+
- You must run within the previous test's `keepAlive` window
|
|
124
141
|
|
|
125
142
|
<Tip>
|
|
126
|
-
|
|
143
|
+
Use `testdriver.getLastSandboxId()` to read the recorded sandbox id (and optional metadata) for scripting purposes.
|
|
127
144
|
</Tip>
|
|
128
145
|
|
|
146
|
+
### Chaining describe blocks within one test file
|
|
147
|
+
|
|
148
|
+
A common pattern is to break a long flow into focused `describe` blocks that share one sandbox — the first block provisions and signs in, later blocks reconnect and continue:
|
|
149
|
+
|
|
150
|
+
```javascript
|
|
151
|
+
import { describe, expect, it } from "vitest";
|
|
152
|
+
import { TestDriver } from "testdriverai/vitest/hooks";
|
|
153
|
+
|
|
154
|
+
const KEEP_ALIVE_MS = 5 * 60 * 1000;
|
|
155
|
+
|
|
156
|
+
describe("step 1 — log in", () => {
|
|
157
|
+
it("signs in and lands on the dashboard", async (context) => {
|
|
158
|
+
const testdriver = TestDriver(context, { keepAlive: KEEP_ALIVE_MS });
|
|
159
|
+
await testdriver.provision.chrome({ url: "https://example.com/login" });
|
|
160
|
+
await testdriver.find("username input").click();
|
|
161
|
+
await testdriver.type("standard_user");
|
|
162
|
+
await testdriver.pressKeys(["tab"]);
|
|
163
|
+
await testdriver.type("secret_sauce", { secret: true });
|
|
164
|
+
await testdriver.pressKeys(["enter"]);
|
|
165
|
+
expect(await testdriver.assert("the dashboard is visible")).toBeTruthy();
|
|
166
|
+
});
|
|
167
|
+
});
|
|
168
|
+
|
|
169
|
+
describe("step 2 — add to cart", () => {
|
|
170
|
+
it("reuses the logged-in sandbox", async (context) => {
|
|
171
|
+
const testdriver = TestDriver(context, {
|
|
172
|
+
reconnect: true, // ← skip provisioning, reattach
|
|
173
|
+
keepAlive: KEEP_ALIVE_MS,
|
|
174
|
+
});
|
|
175
|
+
await testdriver.find("Add to cart").click();
|
|
176
|
+
await testdriver.find("cart icon").click();
|
|
177
|
+
expect(await testdriver.assert("the cart has an item")).toBeTruthy();
|
|
178
|
+
});
|
|
179
|
+
});
|
|
180
|
+
|
|
181
|
+
describe("step 3 — check out", () => {
|
|
182
|
+
it("continues from the cart state", async (context) => {
|
|
183
|
+
const testdriver = TestDriver(context, { reconnect: true, keepAlive: 30_000 });
|
|
184
|
+
await testdriver.find("Checkout").click();
|
|
185
|
+
expect(await testdriver.assert("the checkout form is visible")).toBeTruthy();
|
|
186
|
+
});
|
|
187
|
+
});
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
A runnable copy of this pattern lives at [`examples/reconnect-sequential.test.mjs`](https://github.com/testdriverai/mono/blob/main/sdk/examples/reconnect-sequential.test.mjs).
|
|
191
|
+
|
|
192
|
+
<Warning>
|
|
193
|
+
Vitest runs **test files** in parallel by default. Within a single file, `describe`/`it` blocks run in source order, so reconnect chaining works as written. To chain across multiple files, run them sequentially (e.g. `vitest run --sequence.concurrent=false` or place them in a single project pool with workers set to 1).
|
|
194
|
+
</Warning>
|
|
195
|
+
|
|
129
196
|
### How `keepAlive` works
|
|
130
197
|
|
|
131
198
|
`keepAlive` is a duration in milliseconds. After the SDK disconnects, the server keeps the VM running for that long before terminating it. The default is `60000` (1 minute). Note: `keepAlive: 0` currently falls back to the default disconnect grace period rather than terminating immediately, so use a positive duration when you want to control the grace window explicitly.
|
|
@@ -1,12 +1,82 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:making-assertions
|
|
3
|
-
description:
|
|
3
|
+
description: Locate elements and verify app state with AI-powered assertions
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from making-assertions.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
+
Once a test runs, validate that the app did what it should. Validation has two parts: locating the elements you want to check, and making assertions about the state of your app. TestDriver uses AI as a judge, returning a boolean plus reasoning about whether your app is in the expected state.
|
|
8
|
+
|
|
9
|
+
## Locating Elements
|
|
10
|
+
|
|
11
|
+
### Locating Single Elements
|
|
12
|
+
|
|
13
|
+
Use natural language to describe elements. Descriptions should be specific enough to locate the element, but not too-specific that they break with minor UI changes. For example:
|
|
14
|
+
|
|
15
|
+
```javascript
|
|
16
|
+
await testdriver.find('email input field');
|
|
17
|
+
await testdriver.find('first product card in the grid');
|
|
18
|
+
await testdriver.find('dropdown menu labeled "Country"');
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
<Info>TestDriver will cache found elements for improved performance on subsequent calls. Learn more about [element caching here](/v7/caching).</Info>
|
|
22
|
+
|
|
23
|
+
### Debugging Found Elements
|
|
24
|
+
|
|
25
|
+
After finding an element, you can inspect its properties for debugging:
|
|
26
|
+
|
|
27
|
+
```javascript
|
|
28
|
+
const button = await testdriver.find('submit button');
|
|
29
|
+
console.log(button);
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
This outputs all element properties:
|
|
33
|
+
|
|
34
|
+
```javascript
|
|
35
|
+
{
|
|
36
|
+
description: 'submit button',
|
|
37
|
+
found: true,
|
|
38
|
+
x: 150,
|
|
39
|
+
y: 300,
|
|
40
|
+
coordinates: { x: 150, y: 300, centerX: 200, centerY: 320 },
|
|
41
|
+
threshold: 0.8,
|
|
42
|
+
confidence: 0.95,
|
|
43
|
+
similarity: 0.92,
|
|
44
|
+
selector: 'button[type="submit"]',
|
|
45
|
+
cache: {
|
|
46
|
+
hit: true,
|
|
47
|
+
strategy: 'pixel-diff',
|
|
48
|
+
createdAt: '2025-01-15T10:30:00Z',
|
|
49
|
+
diffPercent: 0.02,
|
|
50
|
+
imageUrl: 'https://...'
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Working with Multiple Elements
|
|
56
|
+
|
|
57
|
+
Find and interact with multiple elements:
|
|
58
|
+
|
|
59
|
+
```javascript
|
|
60
|
+
// Find all matching elements
|
|
61
|
+
const products = await testdriver.findAll('product card');
|
|
62
|
+
console.log(`Found ${products.length} products`);
|
|
63
|
+
|
|
64
|
+
// Interact with each
|
|
65
|
+
for (const product of products) {
|
|
66
|
+
const title = await product.find('title text');
|
|
67
|
+
console.log('Product:', title.text);
|
|
68
|
+
|
|
69
|
+
await product.find('add to cart button').click();
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// Or find specific element
|
|
73
|
+
const firstProduct = products[0];
|
|
74
|
+
await firstProduct.click();
|
|
75
|
+
```
|
|
76
|
+
|
|
7
77
|
## Making Assertions
|
|
8
78
|
|
|
9
|
-
Use AI-powered assertions to verify application state:
|
|
79
|
+
Use AI-powered assertions to verify application state. TestDriver acts as a judge: it evaluates your natural-language statement against the current state of the app and returns a boolean plus reasoning explaining the verdict.
|
|
10
80
|
|
|
11
81
|
```javascript
|
|
12
82
|
// Verify visibility
|
|
@@ -30,3 +100,9 @@ await testdriver.assert('form has red border');
|
|
|
30
100
|
```
|
|
31
101
|
|
|
32
102
|
<Info>Assertions are not cached and always re-evaluated to ensure accuracy.</Info>
|
|
103
|
+
|
|
104
|
+
## Next
|
|
105
|
+
|
|
106
|
+
<Card title="Adapt" icon="arrows-rotate" href="/v7/performing-actions">
|
|
107
|
+
Drive your app forward by performing actions on the elements you've located and validated.
|
|
108
|
+
</Card>
|