@testdriverai/agent 7.11.150-test → 7.11.151-test
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ai/skills/testdriver-agent/SKILL.md +15 -15
- package/ai/skills/testdriver-aws-setup/SKILL.md +6 -6
- package/ai/skills/testdriver-cache/SKILL.md +8 -8
- package/ai/skills/testdriver-caching/SKILL.md +10 -9
- package/ai/skills/testdriver-captcha/SKILL.md +7 -7
- package/ai/skills/testdriver-ci-cd/SKILL.md +10 -10
- package/ai/skills/testdriver-claude-mcp-plugin/SKILL.md +37 -0
- package/ai/skills/testdriver-client/SKILL.md +3 -166
- package/ai/skills/testdriver-customizing-devices/SKILL.md +1 -1
- package/ai/skills/testdriver-dashcam/SKILL.md +3 -3
- package/ai/skills/testdriver-debugging-with-screenshots/SKILL.md +5 -5
- package/ai/skills/testdriver-elements/SKILL.md +5 -5
- package/ai/skills/testdriver-errors/SKILL.md +3 -3
- package/ai/skills/testdriver-events/SKILL.md +3 -3
- package/ai/skills/testdriver-extract/SKILL.md +5 -5
- package/ai/skills/testdriver-find/SKILL.md +1 -1
- package/ai/skills/testdriver-generating-tests/SKILL.md +6 -6
- package/ai/skills/testdriver-hosted/SKILL.md +4 -4
- package/ai/skills/testdriver-interacting-with-your-app/SKILL.md +197 -0
- package/ai/skills/testdriver-locating-elements/SKILL.md +390 -39
- package/ai/skills/testdriver-making-assertions/SKILL.md +4 -4
- package/ai/skills/testdriver-options/SKILL.md +319 -0
- package/ai/skills/testdriver-parse/SKILL.md +8 -8
- package/ai/skills/testdriver-performing-actions/SKILL.md +7 -7
- package/ai/skills/testdriver-provision/SKILL.md +7 -7
- package/ai/skills/testdriver-quickstart/SKILL.md +15 -441
- package/ai/skills/testdriver-quickstart-cli/SKILL.md +436 -0
- package/ai/skills/testdriver-quickstart-github/SKILL.md +53 -0
- package/ai/skills/testdriver-quickstart-manual/SKILL.md +134 -0
- package/ai/skills/testdriver-redraw/SKILL.md +6 -6
- package/ai/skills/testdriver-reusable-code/SKILL.md +3 -3
- package/ai/skills/testdriver-screenshots/SKILL.md +3 -3
- package/ai/skills/testdriver-secrets/SKILL.md +4 -4
- package/ai/skills/testdriver-self-hosted/SKILL.md +6 -6
- package/ai/skills/testdriver-test-results-json/SKILL.md +4 -4
- package/ai/skills/testdriver-variables/SKILL.md +2 -2
- package/ai/skills/testdriver-wait/SKILL.md +2 -2
- package/package.json +1 -1
|
@@ -4,23 +4,23 @@ description: How the TestDriver agent behaves on GitHub issues, pull requests, a
|
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from agent.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
The TestDriver agent is an AI teammate
|
|
8
|
-
**code review that runs your app
|
|
9
|
-
what
|
|
10
|
-
|
|
7
|
+
The TestDriver agent is an AI teammate in your GitHub repository. It is
|
|
8
|
+
**code review that runs your app**. It does not only read a diff and guess
|
|
9
|
+
what can break. It starts a real sandbox, controls the real app, and reports what
|
|
10
|
+
really happens.
|
|
11
11
|
|
|
12
12
|
## Scope: it only builds and maintains tests
|
|
13
13
|
|
|
14
|
-
The agent
|
|
15
|
-
automated tests** with TestDriver
|
|
16
|
-
a test against the change. It
|
|
17
|
-
your application code, fix product bugs, answer
|
|
14
|
+
The agent has one purpose. It does one thing: it **writes, debugs, fixes, and keeps
|
|
15
|
+
automated tests** with TestDriver. It reviews pull requests. It writes and runs
|
|
16
|
+
a test against the change. It does not do work that is not in scope. It does not edit
|
|
17
|
+
your application code, fix product bugs, answer other programming questions, or
|
|
18
18
|
act as a general coding assistant.
|
|
19
19
|
|
|
20
20
|
## How it behaves depends on the surface
|
|
21
21
|
|
|
22
|
-
The agent reacts to three different GitHub surfaces
|
|
23
|
-
|
|
22
|
+
The agent reacts to three different GitHub surfaces. Its behavior is
|
|
23
|
+
different on each one.
|
|
24
24
|
|
|
25
25
|
| Surface | Trigger | What it does |
|
|
26
26
|
|---------|---------|--------------|
|
|
@@ -30,13 +30,13 @@ noticeably different on each.
|
|
|
30
30
|
|
|
31
31
|
## Pull requests — automatic code review
|
|
32
32
|
|
|
33
|
-
When a pull request is **opened**, the agent reviews it automatically
|
|
33
|
+
When a pull request is **opened**, the agent reviews it automatically. Nobody has
|
|
34
34
|
to ask.
|
|
35
35
|
|
|
36
|
-
- **The review is the deliverable, not chatter.**
|
|
37
|
-
|
|
38
|
-
comment** at the end
|
|
39
|
-
|
|
36
|
+
- **The review is the deliverable, not chatter.** No person asks for an automatic review.
|
|
37
|
+
Thus the agent stays quiet while it works. It posts a **single review
|
|
38
|
+
comment** at the end. It does not put many inline threads across the diff. It does not
|
|
39
|
+
tell each step.
|
|
40
40
|
- **It runs the change when it can.** Its default first move is to start a session,
|
|
41
41
|
work out what the PR changes from a user's point of view, and write and run a
|
|
42
42
|
TestDriver test that exercises exactly that. A bug it *watched happen* is far
|
|
@@ -4,7 +4,7 @@ description: Deploy TestDriver on your AWS infrastructure using CloudFormation
|
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from aws-setup.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
This guide
|
|
7
|
+
This guide helps you to set up self-hosted TestDriver instances on AWS. At the end, you have automated test infrastructure. It starts and stops instances when you need them.
|
|
8
8
|
|
|
9
9
|
```mermaid
|
|
10
10
|
graph LR
|
|
@@ -16,13 +16,13 @@ graph LR
|
|
|
16
16
|
|
|
17
17
|
## How It Works
|
|
18
18
|
|
|
19
|
-
TestDriver
|
|
19
|
+
TestDriver manages the AWS EC2 instances for your tests automatically:
|
|
20
20
|
|
|
21
|
-
1. **Deploy CloudFormation
|
|
22
|
-
2. **Configure Vitest
|
|
23
|
-
3. **Run Tests
|
|
21
|
+
1. **Deploy CloudFormation**. Set up the infrastructure one time.
|
|
22
|
+
2. **Configure Vitest**. Add one line to your config.
|
|
23
|
+
3. **Run Tests**. The instances start automatically, run the tests, and stop.
|
|
24
24
|
|
|
25
|
-
That
|
|
25
|
+
That is all. You do not need to manage the instances by hand.
|
|
26
26
|
|
|
27
27
|
# Quickstart
|
|
28
28
|
|
|
@@ -6,18 +6,18 @@ description: Speed up tests with screenshot-based caching
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
The cache system
|
|
9
|
+
The cache system makes repeated test runs faster. It compares screenshots to cached results. When the screen did not change much, TestDriver uses the cached element positions again. It does not make an AI call.
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
- **Screen cache
|
|
13
|
-
- **Element cache
|
|
11
|
+
The cache works at two levels:
|
|
12
|
+
- **Screen cache**: a pixel diff comparison between the present screenshot and the cached screenshot
|
|
13
|
+
- **Element cache**: OpenCV template matching to make sure that the cached element position is still correct
|
|
14
14
|
|
|
15
15
|
## How It Works
|
|
16
16
|
|
|
17
|
-
1. On `find()`, the SDK sends the
|
|
18
|
-
2. The API compares the screenshot against
|
|
19
|
-
3. If the screen pixel diff is
|
|
20
|
-
4.
|
|
17
|
+
1. On `find()`, the SDK sends the present screenshot and the cache metadata to the API.
|
|
18
|
+
2. The API compares the screenshot against the cached results for the same `cacheKey`.
|
|
19
|
+
3. If the screen pixel diff is in the `screen` threshold AND the element template match is more than the `element` threshold, TestDriver returns the cached position.
|
|
20
|
+
4. If not, TestDriver makes a new AI call and caches the result.
|
|
21
21
|
|
|
22
22
|
```mermaid
|
|
23
23
|
flowchart LR
|
|
@@ -4,9 +4,10 @@ description: How TestDriver learns your app and caches what it discovers for ins
|
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from caching.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
After the agent [explored your app](/generating-tests), TestDriver keeps what it found. TestDriver caches each element that the AI vision agent finds. It uses a vision fingerprint. This is a perceptual hash of the screen state at the element location. On the next run, TestDriver matches against that cache. It does not call the AI again. Tests that pass replay quickly, in the same way each time, and at a low cost.
|
|
8
|
+
|
|
9
|
+
This learning makes TestDriver fast. The cache makes the test run a maximum of **1.7x faster**. It does not do the same AI vision analysis again. The agent thinks only when it sees something new.
|
|
8
10
|
|
|
9
|
-
This learning is what makes TestDriver fast. Intelligent caching delivers up to **1.7x faster** test execution by skipping redundant AI vision analysis—the agent only thinks when it sees something new.
|
|
10
11
|
|
|
11
12
|
```javascript
|
|
12
13
|
// First run: builds cache
|
|
@@ -18,14 +19,14 @@ await testdriver.find('submit button');
|
|
|
18
19
|
|
|
19
20
|
## Automatic Caching
|
|
20
21
|
|
|
21
|
-
|
|
22
|
+
TestDriver enables learning automatically. You do not need configuration. TestDriver computes the cache key from these. The cache key is the fingerprint that TestDriver uses to know what it learned:
|
|
22
23
|
|
|
23
|
-
- **File hash**: SHA-256 hash of the test file contents
|
|
24
|
-
- **Selector prompt**: The exact text description
|
|
25
|
-
- **Screenshot context**:
|
|
26
|
-
- **Platform**:
|
|
24
|
+
- **File hash**: The SHA-256 hash of the test file contents
|
|
25
|
+
- **Selector prompt**: The exact text description that you give to `find()`
|
|
26
|
+
- **Screenshot context**: The perceptual hash of the present screen state
|
|
27
|
+
- **Platform**: The operating system and the browser version
|
|
27
28
|
|
|
28
|
-
When you
|
|
29
|
+
When you change your test file, the hash changes automatically. This makes the old cache entries not valid. Then TestDriver does a new AI analysis with your new test logic.
|
|
29
30
|
|
|
30
31
|
```javascript
|
|
31
32
|
import { test } from 'vitest';
|
|
@@ -46,7 +47,7 @@ test('auto-cached test', async (context) => {
|
|
|
46
47
|
|
|
47
48
|
## Managing the Cache
|
|
48
49
|
|
|
49
|
-
You can clear the cache
|
|
50
|
+
You can clear the cache in the TestDriver console. There, you also find previews of cached elements, the input prompts, and analytics on the cache hit rates.
|
|
50
51
|
|
|
51
52
|
<Card href="https://console.testdriver.ai/cache" title="TestDriver Cache" icon="database">
|
|
52
53
|
Manage and clear your test cache from the TestDriver console.
|
|
@@ -6,7 +6,7 @@ description: Solve captchas using 2captcha service
|
|
|
6
6
|
|
|
7
7
|
# Captcha Solving
|
|
8
8
|
|
|
9
|
-
TestDriver can
|
|
9
|
+
TestDriver can solve captchas automatically during your tests. It uses the 2captcha service. This feature works on **Linux** sandboxes and **Windows** sandboxes.
|
|
10
10
|
|
|
11
11
|
## Quick Start
|
|
12
12
|
|
|
@@ -20,12 +20,12 @@ console.log(result.success); // true
|
|
|
20
20
|
console.log(result.token); // The solved captcha token
|
|
21
21
|
```
|
|
22
22
|
|
|
23
|
-
That
|
|
24
|
-
-
|
|
25
|
-
-
|
|
26
|
-
-
|
|
27
|
-
-
|
|
28
|
-
-
|
|
23
|
+
That is all. TestDriver does these automatically:
|
|
24
|
+
- Finds the captcha type on the page
|
|
25
|
+
- Gets the sitekey
|
|
26
|
+
- Solves the captcha with 2captcha
|
|
27
|
+
- Puts the token into the page
|
|
28
|
+
- Starts the callbacks
|
|
29
29
|
|
|
30
30
|
## Supported Captcha Types
|
|
31
31
|
|
|
@@ -4,21 +4,21 @@ description: Run TestDriver tests in CI/CD with parallel execution and cross-pla
|
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from ci-cd.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
TestDriver integrates
|
|
7
|
+
TestDriver integrates with the common CI providers. It lets you do automated end-to-end tests on each push and pull request.
|
|
8
8
|
|
|
9
9
|
## Authentication
|
|
10
10
|
|
|
11
|
-
On **GitHub Actions,
|
|
12
|
-
there
|
|
13
|
-
workflow
|
|
11
|
+
On **GitHub Actions, use OIDC** through the published `testdriverai/action`. Then
|
|
12
|
+
there is no `TD_API_KEY` secret to store, copy, or change. The action shows that the
|
|
13
|
+
workflow runs in your org. TestDriver changes that proof for your
|
|
14
14
|
team's key at run time. See the GitHub Actions tab below.
|
|
15
15
|
|
|
16
|
-
For other CI providers (or self-hosted runners without OIDC),
|
|
17
|
-
stored API key from [console.testdriver.ai/settings](https://console.testdriver.ai/settings)
|
|
18
|
-
|
|
16
|
+
For other CI providers (or self-hosted runners without OIDC), use a
|
|
17
|
+
stored API key from [console.testdriver.ai/settings](https://console.testdriver.ai/settings).
|
|
18
|
+
Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
19
19
|
|
|
20
20
|
<Note>
|
|
21
|
-
Never commit your API key
|
|
21
|
+
Never commit your API key in the code. Always use OIDC or the secrets management of your CI provider.
|
|
22
22
|
</Note>
|
|
23
23
|
|
|
24
24
|
## CI Provider Examples
|
|
@@ -27,10 +27,10 @@ added as a `TD_API_KEY` secret in your CI provider's settings.
|
|
|
27
27
|
<Tab title="GitHub Actions">
|
|
28
28
|
### Authenticate with OIDC via `testdriverai/action` (recommended)
|
|
29
29
|
|
|
30
|
-
Use the published [`testdriverai/action`](https://github.com/testdriverai/action)
|
|
30
|
+
Use the published [`testdriverai/action`](https://github.com/testdriverai/action). It makes the OIDC token, changes it for your team's API key, and exports `TD_API_KEY` for the steps that come after. **There is no `TD_API_KEY` secret to store or change.**
|
|
31
31
|
|
|
32
32
|
<Note>
|
|
33
|
-
|
|
33
|
+
Do this one time: authorize the [TestDriver GitHub App](https://console.testdriver.ai) for your org. Then the org → team binding exists. If your org authorized the App before OIDC support was available, authorize it again one time. If the App is not authorized, the action fails with a console link. (Or it uses the `api-key` secret if you give one.)
|
|
34
34
|
</Note>
|
|
35
35
|
|
|
36
36
|
```yaml .github/workflows/testdriver.yml
|
|
@@ -67,6 +67,43 @@ This is the same config the plugin wires up for you — the plugin just bundles
|
|
|
67
67
|
|
|
68
68
|
---
|
|
69
69
|
|
|
70
|
+
## Remote (hosted URL) install — no CLI, browser login
|
|
71
|
+
|
|
72
|
+
TestDriver also runs a **hosted MCP server** you can connect to with just a URL. There is no `npx` command and no API key to paste: the server speaks OAuth 2.1, so your client opens a browser, you sign in with TestDriver, and the tools appear. It exposes the **full live tool set** (`session_start`, `find`, `click`, `type`, `assert`, …) plus the read-only data tools, all scoped to your team.
|
|
73
|
+
|
|
74
|
+
Hosted endpoint:
|
|
75
|
+
|
|
76
|
+
```text
|
|
77
|
+
https://mcp.testdriver.ai/mcp
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
The server advertises its authorization server (Auth0) via [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728) protected-resource metadata at:
|
|
81
|
+
|
|
82
|
+
```text
|
|
83
|
+
https://mcp.testdriver.ai/.well-known/oauth-protected-resource
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Spec-compliant clients discover and complete the OAuth flow automatically:
|
|
87
|
+
|
|
88
|
+
- **Claude** — add a custom connector pointing at the URL above; Claude runs the browser login for you.
|
|
89
|
+
- **VS Code / Copilot** — add a remote MCP server entry with the URL; VS Code handles the OAuth handshake.
|
|
90
|
+
- **ChatGPT** — add it as a connector by URL.
|
|
91
|
+
- **Any spec-compliant client** — point it at the URL; it will read the metadata, sign you in, and connect.
|
|
92
|
+
|
|
93
|
+
```json
|
|
94
|
+
{
|
|
95
|
+
"mcpServers": {
|
|
96
|
+
"testdriver": {
|
|
97
|
+
"url": "https://mcp.testdriver.ai/mcp"
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Each connection gets its own isolated sandbox, so multiple people (or multiple chats) can run tests at the same time without interfering. The API-key paths above still work for automation and CI.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
70
107
|
## Observing test runs via HTTP MCP
|
|
71
108
|
|
|
72
109
|
TestDriver also exposes **test results and analytics** over an HTTP MCP endpoint, so Claude Code (or any MCP-compatible client) can inspect your test runs, failures, and filters without provisioning a sandbox.
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:client
|
|
3
|
-
description:
|
|
3
|
+
description: Create the TestDriver client, authenticate, and connect to a sandbox
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from client.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
The `TestDriver` client is the main entry point for the SDK. It
|
|
9
|
+
The `TestDriver` client is the main entry point for the SDK. It does the authentication and the sandbox connection. It gives access to all test methods.
|
|
10
10
|
|
|
11
11
|
## Constructor
|
|
12
12
|
|
|
@@ -21,170 +21,7 @@ const testdriver = new TestDriver(apiKey, options)
|
|
|
21
21
|
</ParamField>
|
|
22
22
|
|
|
23
23
|
<ParamField path="options" type="object">
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
<Expandable title="properties">
|
|
27
|
-
<ParamField path="os" type="string" default="linux">
|
|
28
|
-
Operating system for the sandbox: `'windows'` or `'linux'`
|
|
29
|
-
</ParamField>
|
|
30
|
-
|
|
31
|
-
<ParamField path="resolution" type="string" default="1366x768">
|
|
32
|
-
Screen resolution for the sandbox (e.g., `'1920x1080'`, `'1366x768'`). Custom resolutions are only available on Enterprise plans.
|
|
33
|
-
</ParamField>
|
|
34
|
-
|
|
35
|
-
<ParamField path="apiRoot" type="string">
|
|
36
|
-
API endpoint URL (typically only changed for self-hosted deployments)
|
|
37
|
-
</ParamField>
|
|
38
|
-
|
|
39
|
-
<ParamField path="analytics" type="boolean" default="true">
|
|
40
|
-
Enable or disable usage analytics
|
|
41
|
-
</ParamField>
|
|
42
|
-
|
|
43
|
-
<ParamField path="logging" type="boolean" default="true">
|
|
44
|
-
Enable or disable console logging
|
|
45
|
-
</ParamField>
|
|
46
|
-
|
|
47
|
-
<ParamField path="autoScreenshots" type="boolean" default="false">
|
|
48
|
-
Automatically capture screenshots before and after each command. Screenshots are saved to `.testdriver/screenshots/<test>/` with descriptive filenames that include the line number and action name. Format: `<seq>-<action>-<phase>-L<line>-<description>.png`
|
|
49
|
-
</ParamField>
|
|
50
|
-
|
|
51
|
-
<ParamField path="newSandbox" type="boolean" default="true">
|
|
52
|
-
Force creation of a new sandbox instead of reusing an existing one
|
|
53
|
-
</ParamField>
|
|
54
|
-
|
|
55
|
-
<ParamField path="reconnect" type="boolean" default="false">
|
|
56
|
-
Reattach to the last used sandbox instead of creating a new one. When `true`, the SDK reads the sandbox id from `.testdriver/last-sandbox` (written automatically on every successful connect) and rejoins that VM. Provision methods (`chrome`, `vscode`, `installer`, etc.) are skipped because the application is already running. The previous sandbox must still be alive — see [`keepAlive`](#keepalive) and the [Run guide](/copilot/running-tests#keeping-machines-alive-between-runs).
|
|
57
|
-
</ParamField>
|
|
58
|
-
|
|
59
|
-
<ParamField path="sandboxId" type="string">
|
|
60
|
-
Reattach to a specific sandbox id instead of the one recorded in `.testdriver/last-sandbox`. Use this for CI matrices or to pin a chain of tests to a known VM. Implies `reconnect: true` behavior (provision calls are skipped).
|
|
61
|
-
</ParamField>
|
|
62
|
-
|
|
63
|
-
<ParamField path="preview" type="string" default="browser">
|
|
64
|
-
Preview mode for live test visualization:
|
|
65
|
-
- `"browser"` — Opens debugger in default browser (default)
|
|
66
|
-
- `"ide"` — Opens preview in IDE panel (VSCode, Cursor - requires TestDriver extension)
|
|
67
|
-
- `"none"` — Headless mode, no visual preview
|
|
68
|
-
</ParamField>
|
|
69
|
-
|
|
70
|
-
<ParamField path="headless" type="boolean" default="false">
|
|
71
|
-
**Deprecated**: Use `preview: "none"` instead. Run in headless mode without opening the debugger.
|
|
72
|
-
</ParamField>
|
|
73
|
-
|
|
74
|
-
<ParamField path="debugOnFailure" type="boolean" default="false">
|
|
75
|
-
Keep the sandbox alive when a test fails so you can reconnect and debug interactively. The sandbox ID is printed to the console.
|
|
76
|
-
</ParamField>
|
|
77
|
-
|
|
78
|
-
<ParamField path="ip" type="string">
|
|
79
|
-
Direct IP address to connect to a running sandbox instance (for self-hosted deployments)
|
|
80
|
-
</ParamField>
|
|
81
|
-
|
|
82
|
-
<ParamField path="sandboxAmi" type="string">
|
|
83
|
-
Custom AMI ID for the sandbox instance (AWS deployments, e.g., `'ami-1234'`)
|
|
84
|
-
</ParamField>
|
|
85
|
-
|
|
86
|
-
<ParamField path="sandboxInstance" type="string">
|
|
87
|
-
EC2 instance type for the sandbox (AWS deployments, e.g., `'i3.metal'`)
|
|
88
|
-
</ParamField>
|
|
89
|
-
|
|
90
|
-
<ParamField path="cache" type="boolean | object" default="true">
|
|
91
|
-
Enable or disable element caching, or provide advanced threshold configuration.
|
|
92
|
-
|
|
93
|
-
<Expandable title="advanced config">
|
|
94
|
-
<ParamField path="enabled" type="boolean" default="true">
|
|
95
|
-
Enable or disable caching
|
|
96
|
-
</ParamField>
|
|
97
|
-
|
|
98
|
-
<ParamField path="thresholds" type="object">
|
|
99
|
-
Fine-tune cache matching
|
|
100
|
-
|
|
101
|
-
<Expandable title="properties">
|
|
102
|
-
<ParamField path="find" type="object">
|
|
103
|
-
Thresholds for `find()` operations
|
|
104
|
-
|
|
105
|
-
<Expandable title="properties">
|
|
106
|
-
<ParamField path="screen" type="number" default="0.05">
|
|
107
|
-
Pixel diff threshold for screen comparison (0-1). `0.05` = 5% diff allowed.
|
|
108
|
-
</ParamField>
|
|
109
|
-
|
|
110
|
-
<ParamField path="element" type="number" default="0.8">
|
|
111
|
-
OpenCV template match threshold for element matching (0-1). `0.8` = 80% correlation.
|
|
112
|
-
</ParamField>
|
|
113
|
-
</Expandable>
|
|
114
|
-
</ParamField>
|
|
115
|
-
|
|
116
|
-
<ParamField path="assert" type="number" default="0.05">
|
|
117
|
-
Pixel diff threshold for `assert()` operations (0-1). `0.05` = 5% diff allowed.
|
|
118
|
-
</ParamField>
|
|
119
|
-
</Expandable>
|
|
120
|
-
</ParamField>
|
|
121
|
-
</Expandable>
|
|
122
|
-
</ParamField>
|
|
123
|
-
|
|
124
|
-
<ParamField path="cacheKey" type="string">
|
|
125
|
-
Cache key for element finding operations. If provided, enables caching tied to this key.
|
|
126
|
-
</ParamField>
|
|
127
|
-
|
|
128
|
-
<ParamField path="verify" type="boolean" default="false">
|
|
129
|
-
Global default for AI verification of located elements. When `true`, every `find()` runs a second AI check to confirm the returned coordinates actually match the requested element (catching hallucinated or incorrect positions), at the cost of extra latency. Can be overridden per call via `find(description, { verify: true | false })`.
|
|
130
|
-
</ParamField>
|
|
131
|
-
|
|
132
|
-
<ParamField path="dashcam" type="boolean" default="true">
|
|
133
|
-
Enable or disable Dashcam video recording
|
|
134
|
-
</ParamField>
|
|
135
|
-
|
|
136
|
-
<ParamField path="redraw" type="boolean | object" default="true">
|
|
137
|
-
Enable or disable screen-change (redraw) detection, or provide advanced configuration.
|
|
138
|
-
|
|
139
|
-
<Expandable title="advanced config">
|
|
140
|
-
<ParamField path="enabled" type="boolean" default="true">
|
|
141
|
-
Enable or disable redraw detection
|
|
142
|
-
</ParamField>
|
|
143
|
-
|
|
144
|
-
<ParamField path="thresholds" type="object">
|
|
145
|
-
Threshold configuration
|
|
146
|
-
|
|
147
|
-
<Expandable title="properties">
|
|
148
|
-
<ParamField path="screen" type="number | false" default="0.05">
|
|
149
|
-
Pixel diff threshold (0-1). Set to `false` to disable screen redraw detection.
|
|
150
|
-
</ParamField>
|
|
151
|
-
|
|
152
|
-
<ParamField path="network" type="boolean" default="false">
|
|
153
|
-
Enable or disable network activity monitoring
|
|
154
|
-
</ParamField>
|
|
155
|
-
</Expandable>
|
|
156
|
-
</ParamField>
|
|
157
|
-
</Expandable>
|
|
158
|
-
</ParamField>
|
|
159
|
-
|
|
160
|
-
<ParamField path="environment" type="object">
|
|
161
|
-
Additional environment variables to pass to the sandbox
|
|
162
|
-
</ParamField>
|
|
163
|
-
|
|
164
|
-
<ParamField path="ai" type="object">
|
|
165
|
-
Global AI sampling configuration. Controls how the AI model generates responses for `find()` verification and `assert()` calls. Can be overridden per call.
|
|
166
|
-
|
|
167
|
-
<Expandable title="properties">
|
|
168
|
-
<ParamField path="temperature" type="number">
|
|
169
|
-
Controls randomness in AI responses. `0` = deterministic (best for verification), higher values = more creative. Default: `0` for find verification, model default for assert.
|
|
170
|
-
</ParamField>
|
|
171
|
-
|
|
172
|
-
<ParamField path="top" type="object">
|
|
173
|
-
Nucleus and top-k sampling parameters
|
|
174
|
-
|
|
175
|
-
<Expandable title="properties">
|
|
176
|
-
<ParamField path="p" type="number">
|
|
177
|
-
Top-P (nucleus sampling). Limits token choices to the smallest set whose cumulative probability exceeds P. Lower values = more focused responses. Range: 0-1.
|
|
178
|
-
</ParamField>
|
|
179
|
-
|
|
180
|
-
<ParamField path="k" type="number">
|
|
181
|
-
Top-K sampling. Limits token choices to the top K most likely tokens. `1` = always pick the most likely token. `0` = disabled (consider all tokens).
|
|
182
|
-
</ParamField>
|
|
183
|
-
</Expandable>
|
|
184
|
-
</ParamField>
|
|
185
|
-
</Expandable>
|
|
186
|
-
</ParamField>
|
|
187
|
-
</Expandable>
|
|
24
|
+
The configuration options for the client. See [SDK Options](/options) for the full list, with defaults and examples for each option.
|
|
188
25
|
</ParamField>
|
|
189
26
|
|
|
190
27
|
### Example
|
|
@@ -6,7 +6,7 @@ description: Configure TestDriver sandbox options and environment settings
|
|
|
6
6
|
|
|
7
7
|
## TestDriver Options
|
|
8
8
|
|
|
9
|
-
Configure TestDriver behavior with options
|
|
9
|
+
Configure the TestDriver behavior with options that you give to the `TestDriver()` function:
|
|
10
10
|
|
|
11
11
|
```javascript
|
|
12
12
|
const testdriver = TestDriver(context, {
|
|
@@ -6,13 +6,13 @@ description: Record test execution with video and logs
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
Dashcam
|
|
9
|
+
Dashcam makes automatic video recordings. It also collects the logs for your tests. It captures screen recordings, application logs, and test details. You can review them later.
|
|
10
10
|
|
|
11
11
|
## Basic Usage
|
|
12
12
|
|
|
13
13
|
### With Presets
|
|
14
14
|
|
|
15
|
-
Most presets
|
|
15
|
+
Most presets include Dashcam automatically:
|
|
16
16
|
|
|
17
17
|
```javascript
|
|
18
18
|
import { test } from 'vitest';
|
|
@@ -33,7 +33,7 @@ test('my test', async (context) => {
|
|
|
33
33
|
|
|
34
34
|
### Manual Setup
|
|
35
35
|
|
|
36
|
-
For more control,
|
|
36
|
+
For more control, make a Dashcam instance directly:
|
|
37
37
|
|
|
38
38
|
```javascript
|
|
39
39
|
import TestDriver from 'testdriverai';
|
|
@@ -6,10 +6,10 @@ description: Diagnose failing tests with screenshots, replays, and logs
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
When a test fails, debug it
|
|
9
|
+
When a test fails, debug it. Look at what happened. Do not guess. TestDriver captures screenshots, video replays, and logs while your test runs. Thus you can replay the moment of the failure. You do not need to look at a stack trace. TestDriver MCP gives commands to view and examine the screenshots that TestDriver saved during the test. This lets you debug quickly, develop tests, and compare, without a manual open of image files.
|
|
10
10
|
|
|
11
11
|
<Note>
|
|
12
|
-
**Automatic Screenshots (Default: Enabled)**: TestDriver
|
|
12
|
+
**Automatic Screenshots (Default: Enabled)**: TestDriver captures screenshots automatically before and after each command. The screenshot name has the line number and the action. Thus it is easy to see which line of code made each screenshot. For example: `001-click-before-L42-submit-button.png`
|
|
13
13
|
</Note>
|
|
14
14
|
|
|
15
15
|
## MCP Commands
|
|
@@ -25,15 +25,15 @@ list_local_screenshots()
|
|
|
25
25
|
**Filter Parameters:**
|
|
26
26
|
|
|
27
27
|
<ParamField path="directory" type="string" optional>
|
|
28
|
-
Filter screenshots by test file or subdirectory (
|
|
28
|
+
Filter the screenshots by test file or subdirectory (for example, "login.test", "mcp-screenshots"). If you do not give this, the tool lists all screenshots.
|
|
29
29
|
</ParamField>
|
|
30
30
|
|
|
31
31
|
<ParamField path="line" type="number" optional>
|
|
32
|
-
Filter by exact line number from test file (
|
|
32
|
+
Filter by the exact line number from the test file (for example, 42 matches L42 in the filename).
|
|
33
33
|
</ParamField>
|
|
34
34
|
|
|
35
35
|
<ParamField path="lineRange" type="object" optional>
|
|
36
|
-
Filter by line number range. Example: `{ start: 10, end: 20 }` matches screenshots from lines 10-20.
|
|
36
|
+
Filter by a line number range. Example: `{ start: 10, end: 20 }` matches screenshots from lines 10-20.
|
|
37
37
|
</ParamField>
|
|
38
38
|
|
|
39
39
|
<ParamField path="action" type="string" optional>
|
|
@@ -6,22 +6,22 @@ description: Locate and interact with UI elements using AI
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
TestDriver
|
|
9
|
+
The TestDriver element finding system uses AI. It finds elements on the screen with natural language descriptions. The `find()` method returns an `Element` object. You can interact with the object.
|
|
10
10
|
|
|
11
11
|
## Finding Elements
|
|
12
12
|
|
|
13
13
|
### find()
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
Find an element on the screen with a natural language description.
|
|
16
16
|
|
|
17
17
|
```javascript
|
|
18
18
|
const element = await testdriver.find(description)
|
|
19
19
|
```
|
|
20
20
|
|
|
21
21
|
**Parameters:**
|
|
22
|
-
- `description` (string) -
|
|
22
|
+
- `description` (string) - A natural language description of the element to find
|
|
23
23
|
|
|
24
|
-
**Returns:** `Promise<Element>` - Element instance that
|
|
24
|
+
**Returns:** `Promise<Element>` - The Element instance that TestDriver found
|
|
25
25
|
|
|
26
26
|
**Example:**
|
|
27
27
|
```javascript
|
|
@@ -36,7 +36,7 @@ const redButton = await testdriver.find('red button in the top right corner');
|
|
|
36
36
|
```
|
|
37
37
|
|
|
38
38
|
<Tip>
|
|
39
|
-
Be specific in your descriptions. Include visual details, location context, or nearby text to
|
|
39
|
+
Be specific in your descriptions. Include visual details, location context, or nearby text to make the accuracy better.
|
|
40
40
|
</Tip>
|
|
41
41
|
|
|
42
42
|
## Element Class
|
|
@@ -6,7 +6,7 @@ description: Custom error classes and error handling
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
TestDriver
|
|
9
|
+
TestDriver gives custom error classes with full debug information. The SDK exports them. You can use them for `instanceof` checks in your tests.
|
|
10
10
|
|
|
11
11
|
```javascript
|
|
12
12
|
import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
|
|
@@ -14,7 +14,7 @@ import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
|
|
|
14
14
|
|
|
15
15
|
## ElementNotFoundError
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
TestDriver throws this when `find()` cannot find an element on the screen. It also throws this when you call `click()` or `hover()` on an element that TestDriver did not find.
|
|
18
18
|
|
|
19
19
|
```javascript
|
|
20
20
|
try {
|
|
@@ -35,7 +35,7 @@ try {
|
|
|
35
35
|
</ParamField>
|
|
36
36
|
|
|
37
37
|
<ParamField path="message" type="string">
|
|
38
|
-
|
|
38
|
+
A message with a debug block. The block has the element description, the cache status, the similarity scores, and the AI response details.
|
|
39
39
|
</ParamField>
|
|
40
40
|
|
|
41
41
|
<ParamField path="description" type="string">
|
|
@@ -6,9 +6,9 @@ description: Listen to SDK lifecycle events with wildcard support
|
|
|
6
6
|
|
|
7
7
|
## Overview
|
|
8
8
|
|
|
9
|
-
TestDriver uses [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) for its event system. Events use a
|
|
9
|
+
TestDriver uses [EventEmitter2](https://github.com/EventEmitter2/EventEmitter2) for its event system. Events use a namespace pattern with colons. They permit wildcard listeners.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
Use the emitter through `testdriver.emitter`:
|
|
12
12
|
|
|
13
13
|
```javascript
|
|
14
14
|
testdriver.emitter.on('command:start', (data) => {
|
|
@@ -32,7 +32,7 @@ new EventEmitter2({
|
|
|
32
32
|
|
|
33
33
|
### Wildcard Listeners
|
|
34
34
|
|
|
35
|
-
Use `*` to match
|
|
35
|
+
Use `*` to match one level. Use `**` to match many levels:
|
|
36
36
|
|
|
37
37
|
```javascript
|
|
38
38
|
// Match all log events
|