@testdriverai/agent 7.11.229-canary → 7.11.232-test
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/_data/examples-manifest.json +16 -16
- package/docs/_scripts/generate-examples.js +38 -45
- package/docs/agent.mdx +174 -93
- package/docs/assert.mdx +73 -269
- package/docs/aws-setup.mdx +196 -314
- package/docs/bring-your-own-machine.mdx +139 -206
- package/docs/cache.mdx +144 -151
- package/docs/caching.mdx +99 -89
- package/docs/captcha.mdx +126 -111
- package/docs/ci-cd.mdx +154 -543
- package/docs/click.mdx +74 -229
- package/docs/client.mdx +177 -231
- package/docs/copilot/auto-healing.mdx +86 -370
- package/docs/copilot/running-tests.mdx +121 -930
- package/docs/customizing-devices.mdx +74 -251
- package/docs/dashcam.mdx +72 -405
- package/docs/debugging-with-screenshots.mdx +105 -371
- package/docs/docs.json +79 -22
- package/docs/double-click.mdx +48 -68
- package/docs/errors.mdx +102 -176
- package/docs/events.mdx +170 -327
- package/docs/examples/{hover-text-with-description.mdx → add-to-cart.mdx} +30 -35
- package/docs/examples/assert.mdx +28 -41
- package/docs/examples/click-button.mdx +71 -0
- package/docs/examples/{hover-image.mdx → click-icon.mdx} +32 -28
- package/docs/examples/element-not-found.mdx +27 -26
- package/docs/examples/findall-coffee-icons.mdx +28 -32
- package/docs/examples/installer.mdx +61 -25
- package/docs/examples/launch-vscode-linux.mdx +28 -51
- package/docs/examples/navigate-and-scroll.mdx +82 -0
- package/docs/examples/parse.mdx +24 -28
- package/docs/examples/press-keys.mdx +28 -25
- package/docs/examples/scroll.mdx +25 -27
- package/docs/examples/type.mdx +26 -21
- package/docs/exec.mdx +92 -256
- package/docs/extract.mdx +69 -170
- package/docs/find.mdx +140 -736
- package/docs/focus-application.mdx +69 -246
- package/docs/generating-tests.mdx +103 -137
- package/docs/guide/best-practices-polling.mdx +65 -141
- package/docs/hosted.mdx +80 -144
- package/docs/hover.mdx +62 -235
- package/docs/locating-elements.mdx +74 -369
- package/docs/making-assertions.mdx +116 -74
- package/docs/mcp-setup.mdx +80 -63
- package/docs/mouse-down.mdx +48 -125
- package/docs/mouse-up.mdx +47 -129
- package/docs/options.mdx +75 -48
- package/docs/parse.mdx +95 -184
- package/docs/performing-actions.mdx +104 -94
- package/docs/press-keys.mdx +59 -299
- package/docs/provision.mdx +137 -182
- package/docs/quickstart-claude.mdx +85 -0
- package/docs/quickstart-github.mdx +62 -32
- package/docs/quickstart-manual.mdx +60 -33
- package/docs/quickstart-vscode.mdx +97 -0
- package/docs/quickstart.mdx +26 -9
- package/docs/redraw.mdx +101 -165
- package/docs/reusable-code.mdx +97 -200
- package/docs/right-click.mdx +47 -89
- package/docs/screenshot.mdx +47 -201
- package/docs/screenshots.mdx +87 -137
- package/docs/scroll.mdx +71 -154
- package/docs/secrets.mdx +96 -91
- package/docs/self-hosted.mdx +83 -142
- package/docs/snippets/enterprise.mdx +15 -0
- package/docs/styles.css +68 -0
- package/docs/test-results-json.mdx +167 -204
- package/docs/type.mdx +56 -304
- package/docs/variables.mdx +121 -81
- package/docs/wait.mdx +33 -31
- package/docs/what-is-testdriver.mdx +80 -32
- package/examples/{hover-text-with-description.test.mjs → add-to-cart.test.mjs} +4 -5
- package/examples/assert.test.mjs +3 -20
- package/examples/{hover-text.test.mjs → click-button.test.mjs} +3 -4
- package/examples/{hover-image.test.mjs → click-icon.test.mjs} +5 -6
- package/examples/element-not-found.test.mjs +3 -5
- package/examples/findall-coffee-icons.test.mjs +3 -3
- package/examples/installer.test.mjs +1 -1
- package/examples/launch-vscode-linux.test.mjs +6 -33
- package/examples/{scroll-keyboard.test.mjs → navigate-and-scroll.test.mjs} +4 -5
- package/examples/parse.test.mjs +1 -2
- package/examples/press-keys.test.mjs +4 -4
- package/examples/scroll.test.mjs +3 -6
- package/examples/type.test.mjs +5 -2
- package/lib/provision.js +6 -2
- package/package.json +1 -1
- package/sdk.d.ts +4 -2
- package/sdk.js +1 -1
- package/docs/claude-mcp-plugin.mdx +0 -244
- package/docs/elements.mdx +0 -606
- package/docs/examples/ai.mdx +0 -78
- package/docs/examples/captcha-api.mdx +0 -92
- package/docs/examples/chrome-extension.mdx +0 -156
- package/docs/examples/drag-and-drop.mdx +0 -100
- package/docs/examples/formatted-logging.mdx +0 -74
- package/docs/examples/hover-text.mdx +0 -70
- package/docs/examples/match-image.mdx +0 -97
- package/docs/examples/scroll-keyboard.mdx +0 -80
- package/docs/examples/scroll-until-text.mdx +0 -109
package/docs/ci-cd.mdx
CHANGED
|
@@ -1,41 +1,35 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "
|
|
2
|
+
title: "Run tests in CI/CD"
|
|
3
3
|
sidebarTitle: "Integrating into CI/CD"
|
|
4
|
-
description: "Run TestDriver tests in
|
|
4
|
+
description: "Run TestDriver tests on every push and pull request in GitHub Actions or any CI provider."
|
|
5
5
|
icon: "code-branch"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
import { EnterpriseBadge, EnterpriseCta } from '/snippets/enterprise.mdx';
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
TestDriver tests are ordinary Vitest files, so any CI provider can run them with `npx vitest run`. Each test starts its own cloud sandbox, and every run is recorded in the [TestDriver console](https://console.testdriver.ai).
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
there is no `TD_API_KEY` secret to store, copy, or change. The action shows that the
|
|
14
|
-
workflow runs in your org. TestDriver changes that proof for your
|
|
15
|
-
team's key at run time. See the GitHub Actions tab below.
|
|
12
|
+
You need two things in CI:
|
|
16
13
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
14
|
+
- **Authentication.** On GitHub Actions, use the TestDriver action with OIDC. Elsewhere, store your API key as a secret named `TD_API_KEY`.
|
|
15
|
+
- **Your test command,** `npx vitest run`.
|
|
20
16
|
|
|
21
|
-
|
|
22
|
-
Never commit your API key in the code. Always use OIDC or the secrets management of your CI provider.
|
|
23
|
-
</Note>
|
|
17
|
+
## GitHub Actions
|
|
24
18
|
|
|
25
|
-
|
|
19
|
+
The [`testdriverai/action`](https://github.com/testdriverai/action) authenticates with GitHub's OIDC token, so you don't store an API key in GitHub. It exchanges the token for your team's key and exports `TD_API_KEY` for later steps. It doesn't run tests itself.
|
|
26
20
|
|
|
27
|
-
<
|
|
28
|
-
<
|
|
29
|
-
|
|
21
|
+
<Steps>
|
|
22
|
+
<Step title="Authorize the TestDriver GitHub app">
|
|
23
|
+
|
|
24
|
+
Install the [TestDriver GitHub app](/quickstart-github) on your organization. This links the organization to your TestDriver team. If you authorized the app before OIDC support was added, authorize it again once.
|
|
30
25
|
|
|
31
|
-
|
|
26
|
+
</Step>
|
|
27
|
+
<Step title="Add the workflow">
|
|
32
28
|
|
|
33
|
-
|
|
34
|
-
Do this one time: authorize the [TestDriver GitHub App](https://console.testdriver.ai) for your org. Then the org → team binding exists. If your org authorized the App before OIDC support was available, authorize it again one time. If the App is not authorized, the action fails with a console link. (Or it uses the `api-key` secret if you give one.)
|
|
35
|
-
</Note>
|
|
29
|
+
`npx testdriverai init` writes this file for you:
|
|
36
30
|
|
|
37
31
|
```yaml .github/workflows/testdriver.yml
|
|
38
|
-
name: TestDriver
|
|
32
|
+
name: TestDriver tests
|
|
39
33
|
|
|
40
34
|
on:
|
|
41
35
|
push:
|
|
@@ -47,7 +41,7 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
47
41
|
test:
|
|
48
42
|
runs-on: ubuntu-latest
|
|
49
43
|
permissions:
|
|
50
|
-
id-token: write
|
|
44
|
+
id-token: write # lets the job mint an OIDC token
|
|
51
45
|
contents: read
|
|
52
46
|
|
|
53
47
|
steps:
|
|
@@ -61,260 +55,74 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
61
55
|
- run: npm ci
|
|
62
56
|
|
|
63
57
|
- name: Authenticate to TestDriver
|
|
64
|
-
uses: testdriverai/action@stable
|
|
58
|
+
uses: testdriverai/action@stable
|
|
65
59
|
with:
|
|
66
|
-
api-key: ${{ secrets.TD_API_KEY }}
|
|
60
|
+
api-key: ${{ secrets.TD_API_KEY }} # optional fallback
|
|
67
61
|
|
|
68
|
-
- name: Run
|
|
69
|
-
run: vitest run
|
|
62
|
+
- name: Run tests
|
|
63
|
+
run: npx vitest run
|
|
70
64
|
```
|
|
71
65
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
Only if you can't use OIDC (e.g. self-hosted runners without an OIDC provider). Add the key as a secret and pass it via `env`:
|
|
75
|
-
|
|
76
|
-
1. Navigate to your GitHub repository
|
|
77
|
-
2. Go to **Settings** → **Secrets and variables** → **Actions**
|
|
78
|
-
3. Click **New repository secret**
|
|
79
|
-
4. Name: `TD_API_KEY`, Value: your API key
|
|
80
|
-
5. Click **Add secret**
|
|
66
|
+
</Step>
|
|
67
|
+
</Steps>
|
|
81
68
|
|
|
82
|
-
|
|
69
|
+
The job needs `id-token: write`, or the action can't mint the OIDC token.
|
|
83
70
|
|
|
84
|
-
|
|
71
|
+
The `api-key` input is a fallback. The action uses it only when OIDC authentication fails, for example on a fork or before the app is authorized. Leave the secret unset if you don't need it.
|
|
85
72
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
on:
|
|
90
|
-
push:
|
|
91
|
-
branches: [main]
|
|
92
|
-
pull_request:
|
|
93
|
-
branches: [main]
|
|
94
|
-
|
|
95
|
-
jobs:
|
|
96
|
-
test:
|
|
97
|
-
runs-on: ubuntu-latest
|
|
98
|
-
|
|
99
|
-
steps:
|
|
100
|
-
- uses: actions/checkout@v4
|
|
101
|
-
|
|
102
|
-
- uses: actions/setup-node@v4
|
|
103
|
-
with:
|
|
104
|
-
node-version: '20'
|
|
105
|
-
cache: 'npm'
|
|
106
|
-
|
|
107
|
-
- run: npm ci
|
|
108
|
-
|
|
109
|
-
- name: Run TestDriver tests
|
|
110
|
-
env:
|
|
111
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
112
|
-
run: vitest --run
|
|
113
|
-
```
|
|
73
|
+
<AccordionGroup>
|
|
74
|
+
<Accordion title="Action inputs">
|
|
114
75
|
|
|
115
|
-
|
|
76
|
+
| Input | Default | Description |
|
|
77
|
+
| --- | --- | --- |
|
|
78
|
+
| `channel` | `stable` | TestDriver release channel: `stable`, `canary`, `test`, or `dev`. Selects the API the action talks to. |
|
|
79
|
+
| `api-root` | | An explicit API URL. Overrides `channel`. |
|
|
80
|
+
| `api-key` | | A TestDriver API key to use if OIDC authentication fails. |
|
|
116
81
|
|
|
117
|
-
|
|
82
|
+
</Accordion>
|
|
83
|
+
<Accordion title="Use a stored key instead of OIDC">
|
|
118
84
|
|
|
119
|
-
|
|
120
|
-
name: TestDriver Tests (Parallel)
|
|
85
|
+
If OIDC isn't available, skip the action and pass the key directly:
|
|
121
86
|
|
|
122
|
-
|
|
87
|
+
1. Copy an API key from the [API keys page](https://console.testdriver.ai/team/api-keys).
|
|
88
|
+
2. In your repository, go to **Settings → Secrets and variables → Actions** and add a secret named `TD_API_KEY`.
|
|
89
|
+
3. Pass it to the test step:
|
|
123
90
|
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
matrix:
|
|
130
|
-
shard: [1, 2, 3, 4]
|
|
131
|
-
|
|
132
|
-
steps:
|
|
133
|
-
- uses: actions/checkout@v4
|
|
134
|
-
- uses: actions/setup-node@v4
|
|
135
|
-
with:
|
|
136
|
-
node-version: '20'
|
|
137
|
-
cache: 'npm'
|
|
138
|
-
- run: npm ci
|
|
139
|
-
- name: Run tests (shard ${{ matrix.shard }}/4)
|
|
140
|
-
env:
|
|
141
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
142
|
-
run: vitest --run --shard=${{ matrix.shard }}/4
|
|
91
|
+
```yaml
|
|
92
|
+
- name: Run tests
|
|
93
|
+
run: npx vitest run
|
|
94
|
+
env:
|
|
95
|
+
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
143
96
|
```
|
|
144
97
|
|
|
145
|
-
|
|
98
|
+
</Accordion>
|
|
99
|
+
</AccordionGroup>
|
|
146
100
|
|
|
147
|
-
|
|
148
|
-
name: TestDriver Tests (Multi-Platform)
|
|
101
|
+
## Other CI providers
|
|
149
102
|
|
|
150
|
-
|
|
103
|
+
Copy an API key from the [API keys page](https://console.testdriver.ai/team/api-keys). Store it as a secret named `TD_API_KEY` in your CI provider, then use the config for your provider.
|
|
151
104
|
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
strategy:
|
|
156
|
-
fail-fast: false
|
|
157
|
-
matrix:
|
|
158
|
-
td-os: [linux, windows]
|
|
159
|
-
|
|
160
|
-
steps:
|
|
161
|
-
- uses: actions/checkout@v4
|
|
162
|
-
- uses: actions/setup-node@v4
|
|
163
|
-
with:
|
|
164
|
-
node-version: '20'
|
|
165
|
-
cache: 'npm'
|
|
166
|
-
- run: npm ci
|
|
167
|
-
- name: Run tests on ${{ matrix.td-os }}
|
|
168
|
-
env:
|
|
169
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
170
|
-
TD_OS: ${{ matrix.td-os }}
|
|
171
|
-
run: vitest --run
|
|
172
|
-
```
|
|
173
|
-
</Tab>
|
|
105
|
+
<Warning>
|
|
106
|
+
Never commit your API key to the repository. Keep it in your CI provider's secret store.
|
|
107
|
+
</Warning>
|
|
174
108
|
|
|
109
|
+
<Tabs>
|
|
175
110
|
<Tab title="GitLab CI">
|
|
176
|
-
### Adding Secrets
|
|
177
|
-
|
|
178
|
-
1. Go to your GitLab project
|
|
179
|
-
2. Navigate to **Settings** → **CI/CD** → **Variables**
|
|
180
|
-
3. Click **Add variable**
|
|
181
|
-
4. Key: `TD_API_KEY`, Value: your API key
|
|
182
|
-
5. Check **Mask variable** and click **Add variable**
|
|
183
111
|
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
Create `.gitlab-ci.yml`:
|
|
112
|
+
Add `TD_API_KEY` under **Settings → CI/CD → Variables**, with **Mask variable** checked. GitLab exposes project variables to every job.
|
|
187
113
|
|
|
188
114
|
```yaml .gitlab-ci.yml
|
|
189
|
-
stages:
|
|
190
|
-
- test
|
|
191
|
-
|
|
192
115
|
testdriver:
|
|
193
|
-
stage: test
|
|
194
116
|
image: node:20
|
|
195
|
-
cache:
|
|
196
|
-
paths:
|
|
197
|
-
- node_modules/
|
|
198
117
|
script:
|
|
199
118
|
- npm ci
|
|
200
|
-
- vitest
|
|
201
|
-
variables:
|
|
202
|
-
TD_API_KEY: $TD_API_KEY
|
|
203
|
-
```
|
|
204
|
-
|
|
205
|
-
### Parallel Execution
|
|
206
|
-
|
|
207
|
-
```yaml .gitlab-ci.yml
|
|
208
|
-
stages:
|
|
209
|
-
- test
|
|
210
|
-
|
|
211
|
-
.testdriver-base:
|
|
212
|
-
stage: test
|
|
213
|
-
image: node:20
|
|
214
|
-
cache:
|
|
215
|
-
paths:
|
|
216
|
-
- node_modules/
|
|
217
|
-
before_script:
|
|
218
|
-
- npm ci
|
|
219
|
-
variables:
|
|
220
|
-
TD_API_KEY: $TD_API_KEY
|
|
221
|
-
|
|
222
|
-
testdriver-shard-1:
|
|
223
|
-
extends: .testdriver-base
|
|
224
|
-
script:
|
|
225
|
-
- vitest --run --shard=1/4
|
|
226
|
-
|
|
227
|
-
testdriver-shard-2:
|
|
228
|
-
extends: .testdriver-base
|
|
229
|
-
script:
|
|
230
|
-
- vitest --run --shard=2/4
|
|
231
|
-
|
|
232
|
-
testdriver-shard-3:
|
|
233
|
-
extends: .testdriver-base
|
|
234
|
-
script:
|
|
235
|
-
- vitest --run --shard=3/4
|
|
236
|
-
|
|
237
|
-
testdriver-shard-4:
|
|
238
|
-
extends: .testdriver-base
|
|
239
|
-
script:
|
|
240
|
-
- vitest --run --shard=4/4
|
|
119
|
+
- npx vitest run
|
|
241
120
|
```
|
|
242
121
|
|
|
243
|
-
### Multi-Platform Testing
|
|
244
|
-
|
|
245
|
-
```yaml .gitlab-ci.yml
|
|
246
|
-
stages:
|
|
247
|
-
- test
|
|
248
|
-
|
|
249
|
-
.testdriver-base:
|
|
250
|
-
stage: test
|
|
251
|
-
image: node:20
|
|
252
|
-
cache:
|
|
253
|
-
paths:
|
|
254
|
-
- node_modules/
|
|
255
|
-
before_script:
|
|
256
|
-
- npm ci
|
|
257
|
-
variables:
|
|
258
|
-
TD_API_KEY: $TD_API_KEY
|
|
259
|
-
|
|
260
|
-
testdriver-linux:
|
|
261
|
-
extends: .testdriver-base
|
|
262
|
-
variables:
|
|
263
|
-
TD_OS: linux
|
|
264
|
-
script:
|
|
265
|
-
- vitest --run
|
|
266
|
-
|
|
267
|
-
testdriver-windows:
|
|
268
|
-
extends: .testdriver-base
|
|
269
|
-
variables:
|
|
270
|
-
TD_OS: windows
|
|
271
|
-
script:
|
|
272
|
-
- vitest --run
|
|
273
|
-
```
|
|
274
122
|
</Tab>
|
|
275
|
-
|
|
276
123
|
<Tab title="CircleCI">
|
|
277
|
-
### Adding Secrets
|
|
278
|
-
|
|
279
|
-
1. Go to your CircleCI project
|
|
280
|
-
2. Click **Project Settings** → **Environment Variables**
|
|
281
|
-
3. Click **Add Environment Variable**
|
|
282
|
-
4. Name: `TD_API_KEY`, Value: your API key
|
|
283
|
-
|
|
284
|
-
### Basic Config
|
|
285
|
-
|
|
286
|
-
Create `.circleci/config.yml`:
|
|
287
|
-
|
|
288
|
-
```yaml .circleci/config.yml
|
|
289
|
-
version: 2.1
|
|
290
|
-
|
|
291
|
-
jobs:
|
|
292
|
-
test:
|
|
293
|
-
docker:
|
|
294
|
-
- image: cimg/node:20.0
|
|
295
|
-
steps:
|
|
296
|
-
- checkout
|
|
297
|
-
- restore_cache:
|
|
298
|
-
keys:
|
|
299
|
-
- npm-deps-{{ checksum "package-lock.json" }}
|
|
300
|
-
- run: npm ci
|
|
301
|
-
- save_cache:
|
|
302
|
-
key: npm-deps-{{ checksum "package-lock.json" }}
|
|
303
|
-
paths:
|
|
304
|
-
- node_modules
|
|
305
|
-
- run:
|
|
306
|
-
name: Run TestDriver tests
|
|
307
|
-
command: vitest --run
|
|
308
|
-
environment:
|
|
309
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
310
|
-
|
|
311
|
-
workflows:
|
|
312
|
-
test:
|
|
313
|
-
jobs:
|
|
314
|
-
- test
|
|
315
|
-
```
|
|
316
124
|
|
|
317
|
-
|
|
125
|
+
Add `TD_API_KEY` under **Project Settings → Environment Variables**. CircleCI exposes project variables to every job.
|
|
318
126
|
|
|
319
127
|
```yaml .circleci/config.yml
|
|
320
128
|
version: 2.1
|
|
@@ -322,24 +130,11 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
322
130
|
jobs:
|
|
323
131
|
test:
|
|
324
132
|
docker:
|
|
325
|
-
- image: cimg/node:
|
|
326
|
-
parallelism: 4
|
|
133
|
+
- image: cimg/node:lts
|
|
327
134
|
steps:
|
|
328
135
|
- checkout
|
|
329
|
-
- restore_cache:
|
|
330
|
-
keys:
|
|
331
|
-
- npm-deps-{{ checksum "package-lock.json" }}
|
|
332
136
|
- run: npm ci
|
|
333
|
-
-
|
|
334
|
-
key: npm-deps-{{ checksum "package-lock.json" }}
|
|
335
|
-
paths:
|
|
336
|
-
- node_modules
|
|
337
|
-
- run:
|
|
338
|
-
name: Run TestDriver tests
|
|
339
|
-
command: |
|
|
340
|
-
vitest --run --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
|
|
341
|
-
environment:
|
|
342
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
137
|
+
- run: npx vitest run
|
|
343
138
|
|
|
344
139
|
workflows:
|
|
345
140
|
test:
|
|
@@ -347,92 +142,17 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
347
142
|
- test
|
|
348
143
|
```
|
|
349
144
|
|
|
350
|
-
### Multi-Platform Testing
|
|
351
|
-
|
|
352
|
-
```yaml .circleci/config.yml
|
|
353
|
-
version: 2.1
|
|
354
|
-
|
|
355
|
-
jobs:
|
|
356
|
-
test:
|
|
357
|
-
docker:
|
|
358
|
-
- image: cimg/node:20.0
|
|
359
|
-
parameters:
|
|
360
|
-
td-os:
|
|
361
|
-
type: string
|
|
362
|
-
steps:
|
|
363
|
-
- checkout
|
|
364
|
-
- run: npm ci
|
|
365
|
-
- run:
|
|
366
|
-
name: Run TestDriver tests on << parameters.td-os >>
|
|
367
|
-
command: vitest --run
|
|
368
|
-
environment:
|
|
369
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
370
|
-
TD_OS: << parameters.td-os >>
|
|
371
|
-
|
|
372
|
-
workflows:
|
|
373
|
-
test:
|
|
374
|
-
jobs:
|
|
375
|
-
- test:
|
|
376
|
-
td-os: linux
|
|
377
|
-
- test:
|
|
378
|
-
td-os: windows
|
|
379
|
-
```
|
|
380
145
|
</Tab>
|
|
381
|
-
|
|
382
146
|
<Tab title="Azure Pipelines">
|
|
383
|
-
### Adding Secrets
|
|
384
|
-
|
|
385
|
-
1. Go to your Azure DevOps project
|
|
386
|
-
2. Navigate to **Pipelines** → **Library** → **Variable groups**
|
|
387
|
-
3. Create a new variable group or edit existing
|
|
388
|
-
4. Add variable: `TD_API_KEY` with your API key
|
|
389
|
-
5. Click the lock icon to make it secret
|
|
390
|
-
|
|
391
|
-
### Basic Pipeline
|
|
392
|
-
|
|
393
|
-
Create `azure-pipelines.yml`:
|
|
394
|
-
|
|
395
|
-
```yaml azure-pipelines.yml
|
|
396
|
-
trigger:
|
|
397
|
-
- main
|
|
398
|
-
|
|
399
|
-
pool:
|
|
400
|
-
vmImage: 'ubuntu-latest'
|
|
401
|
-
|
|
402
|
-
steps:
|
|
403
|
-
- task: NodeTool@0
|
|
404
|
-
inputs:
|
|
405
|
-
versionSpec: '20.x'
|
|
406
|
-
displayName: 'Setup Node.js'
|
|
407
147
|
|
|
408
|
-
|
|
409
|
-
displayName: 'Install dependencies'
|
|
410
|
-
|
|
411
|
-
- script: vitest --run
|
|
412
|
-
displayName: 'Run TestDriver tests'
|
|
413
|
-
env:
|
|
414
|
-
TD_API_KEY: $(TD_API_KEY)
|
|
415
|
-
```
|
|
416
|
-
|
|
417
|
-
### Parallel Execution
|
|
148
|
+
Add `TD_API_KEY` as a pipeline variable and mark it secret. Azure doesn't expose secret variables to scripts automatically, so map it with `env`.
|
|
418
149
|
|
|
419
150
|
```yaml azure-pipelines.yml
|
|
420
151
|
trigger:
|
|
421
152
|
- main
|
|
422
153
|
|
|
423
154
|
pool:
|
|
424
|
-
vmImage:
|
|
425
|
-
|
|
426
|
-
strategy:
|
|
427
|
-
matrix:
|
|
428
|
-
shard1:
|
|
429
|
-
SHARD: '1/4'
|
|
430
|
-
shard2:
|
|
431
|
-
SHARD: '2/4'
|
|
432
|
-
shard3:
|
|
433
|
-
SHARD: '3/4'
|
|
434
|
-
shard4:
|
|
435
|
-
SHARD: '4/4'
|
|
155
|
+
vmImage: ubuntu-latest
|
|
436
156
|
|
|
437
157
|
steps:
|
|
438
158
|
- task: NodeTool@0
|
|
@@ -440,248 +160,139 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
440
160
|
versionSpec: '20.x'
|
|
441
161
|
|
|
442
162
|
- script: npm ci
|
|
443
|
-
displayName:
|
|
163
|
+
displayName: Install dependencies
|
|
444
164
|
|
|
445
|
-
- script: vitest
|
|
446
|
-
displayName:
|
|
165
|
+
- script: npx vitest run
|
|
166
|
+
displayName: Run TestDriver tests
|
|
447
167
|
env:
|
|
448
168
|
TD_API_KEY: $(TD_API_KEY)
|
|
449
169
|
```
|
|
450
170
|
|
|
451
|
-
### Multi-Platform Testing
|
|
452
|
-
|
|
453
|
-
```yaml azure-pipelines.yml
|
|
454
|
-
trigger:
|
|
455
|
-
- main
|
|
456
|
-
|
|
457
|
-
pool:
|
|
458
|
-
vmImage: 'ubuntu-latest'
|
|
459
|
-
|
|
460
|
-
strategy:
|
|
461
|
-
matrix:
|
|
462
|
-
linux:
|
|
463
|
-
TD_OS: 'linux'
|
|
464
|
-
windows:
|
|
465
|
-
TD_OS: 'windows'
|
|
466
|
-
|
|
467
|
-
steps:
|
|
468
|
-
- task: NodeTool@0
|
|
469
|
-
inputs:
|
|
470
|
-
versionSpec: '20.x'
|
|
471
|
-
|
|
472
|
-
- script: npm ci
|
|
473
|
-
displayName: 'Install dependencies'
|
|
474
|
-
|
|
475
|
-
- script: vitest --run
|
|
476
|
-
displayName: 'Run TestDriver tests on $(TD_OS)'
|
|
477
|
-
env:
|
|
478
|
-
TD_API_KEY: $(TD_API_KEY)
|
|
479
|
-
TD_OS: $(TD_OS)
|
|
480
|
-
```
|
|
481
171
|
</Tab>
|
|
482
|
-
|
|
483
172
|
<Tab title="Jenkins">
|
|
484
|
-
### Adding Secrets
|
|
485
|
-
|
|
486
|
-
1. Go to **Manage Jenkins** → **Credentials**
|
|
487
|
-
2. Select the appropriate domain
|
|
488
|
-
3. Click **Add Credentials**
|
|
489
|
-
4. Kind: **Secret text**
|
|
490
|
-
5. ID: `td-api-key`, Secret: your API key
|
|
491
|
-
|
|
492
|
-
### Basic Pipeline
|
|
493
173
|
|
|
494
|
-
|
|
174
|
+
Add a **Secret text** credential with the ID `td-api-key` under **Manage Jenkins → Credentials**.
|
|
495
175
|
|
|
496
176
|
```groovy Jenkinsfile
|
|
497
177
|
pipeline {
|
|
498
|
-
agent {
|
|
499
|
-
|
|
500
|
-
image 'node:20'
|
|
501
|
-
}
|
|
502
|
-
}
|
|
503
|
-
|
|
178
|
+
agent { docker { image 'node:20' } }
|
|
179
|
+
|
|
504
180
|
environment {
|
|
505
181
|
TD_API_KEY = credentials('td-api-key')
|
|
506
182
|
}
|
|
507
|
-
|
|
183
|
+
|
|
508
184
|
stages {
|
|
509
|
-
stage('Install') {
|
|
510
|
-
steps {
|
|
511
|
-
sh 'npm ci'
|
|
512
|
-
}
|
|
513
|
-
}
|
|
514
|
-
|
|
515
185
|
stage('Test') {
|
|
516
186
|
steps {
|
|
517
|
-
sh '
|
|
187
|
+
sh 'npm ci'
|
|
188
|
+
sh 'npx vitest run'
|
|
518
189
|
}
|
|
519
190
|
}
|
|
520
191
|
}
|
|
521
192
|
}
|
|
522
193
|
```
|
|
523
194
|
|
|
524
|
-
|
|
195
|
+
</Tab>
|
|
196
|
+
</Tabs>
|
|
525
197
|
|
|
526
|
-
|
|
527
|
-
pipeline {
|
|
528
|
-
agent none
|
|
529
|
-
|
|
530
|
-
environment {
|
|
531
|
-
TD_API_KEY = credentials('td-api-key')
|
|
532
|
-
}
|
|
533
|
-
|
|
534
|
-
stages {
|
|
535
|
-
stage('Test') {
|
|
536
|
-
parallel {
|
|
537
|
-
stage('Shard 1') {
|
|
538
|
-
agent { docker { image 'node:20' } }
|
|
539
|
-
steps {
|
|
540
|
-
sh 'npm ci'
|
|
541
|
-
sh 'vitest --run --shard=1/4'
|
|
542
|
-
}
|
|
543
|
-
}
|
|
544
|
-
stage('Shard 2') {
|
|
545
|
-
agent { docker { image 'node:20' } }
|
|
546
|
-
steps {
|
|
547
|
-
sh 'npm ci'
|
|
548
|
-
sh 'vitest --run --shard=2/4'
|
|
549
|
-
}
|
|
550
|
-
}
|
|
551
|
-
stage('Shard 3') {
|
|
552
|
-
agent { docker { image 'node:20' } }
|
|
553
|
-
steps {
|
|
554
|
-
sh 'npm ci'
|
|
555
|
-
sh 'vitest --run --shard=3/4'
|
|
556
|
-
}
|
|
557
|
-
}
|
|
558
|
-
stage('Shard 4') {
|
|
559
|
-
agent { docker { image 'node:20' } }
|
|
560
|
-
steps {
|
|
561
|
-
sh 'npm ci'
|
|
562
|
-
sh 'vitest --run --shard=4/4'
|
|
563
|
-
}
|
|
564
|
-
}
|
|
565
|
-
}
|
|
566
|
-
}
|
|
567
|
-
}
|
|
568
|
-
}
|
|
569
|
-
```
|
|
198
|
+
## Split tests across jobs
|
|
570
199
|
|
|
571
|
-
|
|
200
|
+
Vitest runs test files in parallel within one job. To spread a large suite across several jobs, use Vitest's `--shard` flag:
|
|
572
201
|
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
stage('Linux') {
|
|
585
|
-
agent { docker { image 'node:20' } }
|
|
586
|
-
environment {
|
|
587
|
-
TD_OS = 'linux'
|
|
588
|
-
}
|
|
589
|
-
steps {
|
|
590
|
-
sh 'npm ci'
|
|
591
|
-
sh 'vitest --run'
|
|
592
|
-
}
|
|
593
|
-
}
|
|
594
|
-
stage('Windows') {
|
|
595
|
-
agent { docker { image 'node:20' } }
|
|
596
|
-
environment {
|
|
597
|
-
TD_OS = 'windows'
|
|
598
|
-
}
|
|
599
|
-
steps {
|
|
600
|
-
sh 'npm ci'
|
|
601
|
-
sh 'vitest --run'
|
|
602
|
-
}
|
|
603
|
-
}
|
|
604
|
-
}
|
|
605
|
-
}
|
|
606
|
-
}
|
|
607
|
-
}
|
|
608
|
-
```
|
|
609
|
-
</Tab>
|
|
610
|
-
</Tabs>
|
|
202
|
+
```yaml .github/workflows/testdriver.yml
|
|
203
|
+
jobs:
|
|
204
|
+
test:
|
|
205
|
+
runs-on: ubuntu-latest
|
|
206
|
+
permissions:
|
|
207
|
+
id-token: write
|
|
208
|
+
contents: read
|
|
209
|
+
strategy:
|
|
210
|
+
fail-fast: false
|
|
211
|
+
matrix:
|
|
212
|
+
shard: [1, 2, 3, 4]
|
|
611
213
|
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
it("should work on both Linux and Windows", async (context) => {
|
|
622
|
-
const os = process.env.TD_OS || 'linux';
|
|
623
|
-
|
|
624
|
-
const testdriver = TestDriver(context, {
|
|
625
|
-
os: os // 'linux' or 'windows'
|
|
626
|
-
});
|
|
627
|
-
|
|
628
|
-
await testdriver.provision.chrome({
|
|
629
|
-
url: 'https://example.com',
|
|
630
|
-
});
|
|
631
|
-
|
|
632
|
-
const result = await testdriver.assert("the page loaded successfully");
|
|
633
|
-
expect(result).toBeTruthy();
|
|
634
|
-
});
|
|
635
|
-
});
|
|
214
|
+
steps:
|
|
215
|
+
- uses: actions/checkout@v4
|
|
216
|
+
- uses: actions/setup-node@v4
|
|
217
|
+
with:
|
|
218
|
+
node-version: '20'
|
|
219
|
+
cache: 'npm'
|
|
220
|
+
- run: npm ci
|
|
221
|
+
- uses: testdriverai/action@stable
|
|
222
|
+
- run: npx vitest run --shard=${{ matrix.shard }}/4
|
|
636
223
|
```
|
|
637
224
|
|
|
638
|
-
|
|
225
|
+
Other providers work the same way. Run `npx vitest run --shard=<index>/<total>` in each parallel job.
|
|
226
|
+
|
|
227
|
+
## Run on Linux and Windows
|
|
639
228
|
|
|
640
|
-
|
|
641
|
-
a sandbox and you're already at that limit, the request is **queued** rather than
|
|
642
|
-
failed immediately: the SDK waits for a slot to free up, retrying every 10 seconds,
|
|
643
|
-
then proceeds automatically once one opens. This is what lets a parallel CI matrix
|
|
644
|
-
(many jobs starting at once) work on a plan with fewer slots than jobs — the extra
|
|
645
|
-
jobs simply wait their turn instead of erroring.
|
|
229
|
+
Set `TD_OS` to choose the sandbox's operating system. TestDriver reads it when it creates the sandbox, so your tests don't need any changes:
|
|
646
230
|
|
|
647
|
-
|
|
648
|
-
|
|
231
|
+
```yaml
|
|
232
|
+
strategy:
|
|
233
|
+
fail-fast: false
|
|
234
|
+
matrix:
|
|
235
|
+
td-os: [linux, windows]
|
|
236
|
+
|
|
237
|
+
steps:
|
|
238
|
+
# ...checkout, setup-node, npm ci, testdriverai/action
|
|
239
|
+
- run: npx vitest run
|
|
240
|
+
env:
|
|
241
|
+
TD_OS: ${{ matrix.td-os }}
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
Linux is the default.
|
|
245
|
+
|
|
246
|
+
<EnterpriseCta feature="Windows sandboxes" plural />
|
|
649
247
|
|
|
650
|
-
|
|
651
|
-
| ----- | -------- |
|
|
652
|
-
| _unset_ | Wait up to **60 seconds** (the default). |
|
|
653
|
-
| `TD_CONCURRENCY_MAX_WAIT=300` | Wait up to **300 seconds** (5 minutes) before giving up. |
|
|
654
|
-
| `TD_CONCURRENCY_MAX_WAIT=0` | **Don't queue** — fail on the first denial. |
|
|
248
|
+
An `os` option passed to `TestDriver()` takes priority over `TD_OS`. To branch on the platform inside a test, read `process.env.TD_OS`:
|
|
655
249
|
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
250
|
+
```javascript
|
|
251
|
+
const isWindows = process.env.TD_OS === 'windows';
|
|
252
|
+
|
|
253
|
+
await testdriver.exec({
|
|
254
|
+
language: isWindows ? 'pwsh' : 'sh',
|
|
255
|
+
code: isWindows ? 'Get-ChildItem $HOME' : 'ls ~',
|
|
256
|
+
});
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
## Concurrency limits
|
|
260
|
+
|
|
261
|
+
Your plan sets how many sandboxes can run at once. When a job asks for a sandbox at that limit, it waits for a free slot instead of failing.
|
|
262
|
+
|
|
263
|
+
- The SDK retries every 10 seconds.
|
|
264
|
+
- By default, it gives up after 60 seconds with a concurrency-limit error.
|
|
265
|
+
- Set `TD_CONCURRENCY_MAX_WAIT` (in seconds) to wait longer, or to `0` to fail on the first refusal.
|
|
660
266
|
|
|
661
267
|
```yaml
|
|
662
|
-
|
|
663
|
-
# Give each job up to 5 minutes to acquire a slot before failing.
|
|
664
|
-
- name: Run TestDriver tests
|
|
268
|
+
- run: npx vitest run
|
|
665
269
|
env:
|
|
666
|
-
|
|
667
|
-
TD_CONCURRENCY_MAX_WAIT: "300"
|
|
668
|
-
run: vitest run
|
|
270
|
+
TD_CONCURRENCY_MAX_WAIT: '300' # wait up to 5 minutes for a slot
|
|
669
271
|
```
|
|
670
272
|
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
to **fail fast** on a busy account (e.g. a quick smoke test that shouldn't sit
|
|
675
|
-
waiting). When jobs routinely give up waiting, that's the signal to
|
|
676
|
-
[add more slots](https://console.testdriver.ai/checkout/pro).
|
|
677
|
-
</Tip>
|
|
273
|
+
Raise the wait when you run more parallel jobs than your plan has slots. If jobs regularly give up waiting, run fewer at once with `--maxWorkers` or add slots to your plan.
|
|
274
|
+
|
|
275
|
+
## View results
|
|
678
276
|
|
|
679
|
-
|
|
277
|
+
Each run is recorded in the [TestDriver console](https://console.testdriver.ai), with a video replay, logs, and pass or fail status. When a run finishes, the SDK prints a link to it:
|
|
680
278
|
|
|
681
|
-
|
|
279
|
+
```text
|
|
280
|
+
TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/abc123
|
|
281
|
+
```
|
|
682
282
|
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
283
|
+
## Next steps
|
|
284
|
+
|
|
285
|
+
<CardGroup cols={2}>
|
|
286
|
+
<Card title="Run tests" icon="play" href="/copilot/running-tests">
|
|
287
|
+
Run tests locally, in parallel, and watch them live.
|
|
288
|
+
</Card>
|
|
289
|
+
<Card title="Add TestDriver to GitHub" icon="github" href="/quickstart-github">
|
|
290
|
+
Get an AI code review on every pull request.
|
|
291
|
+
</Card>
|
|
292
|
+
<Card title="Hosted plans" icon="cloud" href="/hosted">
|
|
293
|
+
Compare plans, testing minutes, and concurrency.
|
|
294
|
+
</Card>
|
|
295
|
+
<Card title="Speed up reruns with caching" icon="brain" href="/caching">
|
|
296
|
+
Reuse element locations from passing runs.
|
|
297
|
+
</Card>
|
|
298
|
+
</CardGroup>
|