@testdriverai/agent 7.11.240-test → 7.11.241-canary
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ai/skills/testdriver-agent/SKILL.md +174 -91
- package/ai/skills/testdriver-assert/SKILL.md +73 -269
- package/ai/skills/testdriver-aws-setup/SKILL.md +194 -312
- package/ai/skills/testdriver-bring-your-own-machine/SKILL.md +137 -204
- package/ai/skills/testdriver-cache/SKILL.md +142 -149
- package/ai/skills/testdriver-caching/SKILL.md +97 -87
- package/ai/skills/testdriver-captcha/SKILL.md +126 -111
- package/ai/skills/testdriver-ci-cd/SKILL.md +153 -542
- package/ai/skills/testdriver-click/SKILL.md +74 -229
- package/ai/skills/testdriver-client/SKILL.md +175 -229
- package/ai/skills/testdriver-customizing-devices/SKILL.md +72 -250
- package/ai/skills/testdriver-dashcam/SKILL.md +71 -404
- package/ai/skills/testdriver-debugging-with-screenshots/SKILL.md +103 -369
- package/ai/skills/testdriver-double-click/SKILL.md +46 -67
- package/ai/skills/testdriver-errors/SKILL.md +102 -176
- package/ai/skills/testdriver-events/SKILL.md +170 -327
- package/ai/skills/testdriver-exec/SKILL.md +92 -256
- package/ai/skills/testdriver-extract/SKILL.md +69 -170
- package/ai/skills/testdriver-find/SKILL.md +140 -736
- package/ai/skills/testdriver-focus-application/SKILL.md +69 -246
- package/ai/skills/testdriver-generating-tests/SKILL.md +102 -136
- package/ai/skills/testdriver-hosted/SKILL.md +79 -143
- package/ai/skills/testdriver-hover/SKILL.md +62 -235
- package/ai/skills/testdriver-locating-elements/SKILL.md +71 -366
- package/ai/skills/testdriver-making-assertions/SKILL.md +114 -72
- package/ai/skills/testdriver-mcp-setup/SKILL.md +79 -62
- package/ai/skills/testdriver-mouse-down/SKILL.md +46 -124
- package/ai/skills/testdriver-mouse-up/SKILL.md +45 -128
- package/ai/skills/testdriver-options/SKILL.md +73 -46
- package/ai/skills/testdriver-parse/SKILL.md +95 -184
- package/ai/skills/testdriver-performing-actions/SKILL.md +101 -91
- package/ai/skills/testdriver-press-keys/SKILL.md +59 -299
- package/ai/skills/testdriver-provision/SKILL.md +135 -179
- package/ai/skills/testdriver-quickstart/SKILL.md +24 -8
- package/ai/skills/testdriver-quickstart-claude/SKILL.md +84 -0
- package/ai/skills/testdriver-quickstart-github/SKILL.md +62 -32
- package/ai/skills/testdriver-quickstart-manual/SKILL.md +59 -32
- package/ai/skills/testdriver-quickstart-vscode/SKILL.md +96 -0
- package/ai/skills/testdriver-redraw/SKILL.md +100 -164
- package/ai/skills/testdriver-reusable-code/SKILL.md +95 -199
- package/ai/skills/testdriver-right-click/SKILL.md +45 -88
- package/ai/skills/testdriver-screenshot/SKILL.md +47 -201
- package/ai/skills/testdriver-screenshots/SKILL.md +86 -136
- package/ai/skills/testdriver-scroll/SKILL.md +71 -154
- package/ai/skills/testdriver-secrets/SKILL.md +94 -90
- package/ai/skills/testdriver-self-hosted/SKILL.md +82 -140
- package/ai/skills/testdriver-test-results-json/SKILL.md +165 -202
- package/ai/skills/testdriver-type/SKILL.md +56 -304
- package/ai/skills/testdriver-variables/SKILL.md +119 -80
- package/ai/skills/testdriver-wait/SKILL.md +31 -29
- package/ai/skills/testdriver-what-is-testdriver/SKILL.md +80 -32
- package/package.json +1 -1
|
@@ -1,40 +1,34 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: testdriver:ci-cd
|
|
3
|
-
description: Run TestDriver tests in
|
|
3
|
+
description: Run TestDriver tests on every push and pull request in GitHub Actions or any CI provider.
|
|
4
4
|
---
|
|
5
5
|
<!-- Generated from ci-cd.mdx. DO NOT EDIT. -->
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
import { EnterpriseBadge, EnterpriseCta } from '/snippets/enterprise.mdx';
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
TestDriver tests are ordinary Vitest files, so any CI provider can run them with `npx vitest run`. Each test starts its own cloud sandbox, and every run is recorded in the [TestDriver console](https://console.testdriver.ai).
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
there is no `TD_API_KEY` secret to store, copy, or change. The action shows that the
|
|
13
|
-
workflow runs in your org. TestDriver changes that proof for your
|
|
14
|
-
team's key at run time. See the GitHub Actions tab below.
|
|
11
|
+
You need two things in CI:
|
|
15
12
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
13
|
+
- **Authentication.** On GitHub Actions, use the TestDriver action with OIDC. Elsewhere, store your API key as a secret named `TD_API_KEY`.
|
|
14
|
+
- **Your test command,** `npx vitest run`.
|
|
19
15
|
|
|
20
|
-
|
|
21
|
-
Never commit your API key in the code. Always use OIDC or the secrets management of your CI provider.
|
|
22
|
-
</Note>
|
|
16
|
+
## GitHub Actions
|
|
23
17
|
|
|
24
|
-
|
|
18
|
+
The [`testdriverai/action`](https://github.com/testdriverai/action) authenticates with GitHub's OIDC token, so you don't store an API key in GitHub. It exchanges the token for your team's key and exports `TD_API_KEY` for later steps. It doesn't run tests itself.
|
|
25
19
|
|
|
26
|
-
<
|
|
27
|
-
<
|
|
28
|
-
|
|
20
|
+
<Steps>
|
|
21
|
+
<Step title="Authorize the TestDriver GitHub app">
|
|
22
|
+
|
|
23
|
+
Install the [TestDriver GitHub app](/quickstart-github) on your organization. This links the organization to your TestDriver team. If you authorized the app before OIDC support was added, authorize it again once.
|
|
29
24
|
|
|
30
|
-
|
|
25
|
+
</Step>
|
|
26
|
+
<Step title="Add the workflow">
|
|
31
27
|
|
|
32
|
-
|
|
33
|
-
Do this one time: authorize the [TestDriver GitHub App](https://console.testdriver.ai) for your org. Then the org → team binding exists. If your org authorized the App before OIDC support was available, authorize it again one time. If the App is not authorized, the action fails with a console link. (Or it uses the `api-key` secret if you give one.)
|
|
34
|
-
</Note>
|
|
28
|
+
`npx testdriverai init` writes this file for you:
|
|
35
29
|
|
|
36
30
|
```yaml .github/workflows/testdriver.yml
|
|
37
|
-
name: TestDriver
|
|
31
|
+
name: TestDriver tests
|
|
38
32
|
|
|
39
33
|
on:
|
|
40
34
|
push:
|
|
@@ -46,7 +40,7 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
46
40
|
test:
|
|
47
41
|
runs-on: ubuntu-latest
|
|
48
42
|
permissions:
|
|
49
|
-
id-token: write
|
|
43
|
+
id-token: write # lets the job mint an OIDC token
|
|
50
44
|
contents: read
|
|
51
45
|
|
|
52
46
|
steps:
|
|
@@ -60,260 +54,74 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
60
54
|
- run: npm ci
|
|
61
55
|
|
|
62
56
|
- name: Authenticate to TestDriver
|
|
63
|
-
uses: testdriverai/action@stable
|
|
57
|
+
uses: testdriverai/action@stable
|
|
64
58
|
with:
|
|
65
|
-
api-key: ${{ secrets.TD_API_KEY }}
|
|
59
|
+
api-key: ${{ secrets.TD_API_KEY }} # optional fallback
|
|
66
60
|
|
|
67
|
-
- name: Run
|
|
68
|
-
run: vitest run
|
|
61
|
+
- name: Run tests
|
|
62
|
+
run: npx vitest run
|
|
69
63
|
```
|
|
70
64
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
Only if you can't use OIDC (e.g. self-hosted runners without an OIDC provider). Add the key as a secret and pass it via `env`:
|
|
74
|
-
|
|
75
|
-
1. Navigate to your GitHub repository
|
|
76
|
-
2. Go to **Settings** → **Secrets and variables** → **Actions**
|
|
77
|
-
3. Click **New repository secret**
|
|
78
|
-
4. Name: `TD_API_KEY`, Value: your API key
|
|
79
|
-
5. Click **Add secret**
|
|
65
|
+
</Step>
|
|
66
|
+
</Steps>
|
|
80
67
|
|
|
81
|
-
|
|
68
|
+
The job needs `id-token: write`, or the action can't mint the OIDC token.
|
|
82
69
|
|
|
83
|
-
|
|
70
|
+
The `api-key` input is a fallback. The action uses it only when OIDC authentication fails, for example on a fork or before the app is authorized. Leave the secret unset if you don't need it.
|
|
84
71
|
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
on:
|
|
89
|
-
push:
|
|
90
|
-
branches: [main]
|
|
91
|
-
pull_request:
|
|
92
|
-
branches: [main]
|
|
93
|
-
|
|
94
|
-
jobs:
|
|
95
|
-
test:
|
|
96
|
-
runs-on: ubuntu-latest
|
|
97
|
-
|
|
98
|
-
steps:
|
|
99
|
-
- uses: actions/checkout@v4
|
|
100
|
-
|
|
101
|
-
- uses: actions/setup-node@v4
|
|
102
|
-
with:
|
|
103
|
-
node-version: '20'
|
|
104
|
-
cache: 'npm'
|
|
105
|
-
|
|
106
|
-
- run: npm ci
|
|
107
|
-
|
|
108
|
-
- name: Run TestDriver tests
|
|
109
|
-
env:
|
|
110
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
111
|
-
run: vitest --run
|
|
112
|
-
```
|
|
72
|
+
<AccordionGroup>
|
|
73
|
+
<Accordion title="Action inputs">
|
|
113
74
|
|
|
114
|
-
|
|
75
|
+
| Input | Default | Description |
|
|
76
|
+
| --- | --- | --- |
|
|
77
|
+
| `channel` | `stable` | TestDriver release channel: `stable`, `canary`, `test`, or `dev`. Selects the API the action talks to. |
|
|
78
|
+
| `api-root` | | An explicit API URL. Overrides `channel`. |
|
|
79
|
+
| `api-key` | | A TestDriver API key to use if OIDC authentication fails. |
|
|
115
80
|
|
|
116
|
-
|
|
81
|
+
</Accordion>
|
|
82
|
+
<Accordion title="Use a stored key instead of OIDC">
|
|
117
83
|
|
|
118
|
-
|
|
119
|
-
name: TestDriver Tests (Parallel)
|
|
84
|
+
If OIDC isn't available, skip the action and pass the key directly:
|
|
120
85
|
|
|
121
|
-
|
|
86
|
+
1. Copy an API key from the [API keys page](https://console.testdriver.ai/team/api-keys).
|
|
87
|
+
2. In your repository, go to **Settings → Secrets and variables → Actions** and add a secret named `TD_API_KEY`.
|
|
88
|
+
3. Pass it to the test step:
|
|
122
89
|
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
matrix:
|
|
129
|
-
shard: [1, 2, 3, 4]
|
|
130
|
-
|
|
131
|
-
steps:
|
|
132
|
-
- uses: actions/checkout@v4
|
|
133
|
-
- uses: actions/setup-node@v4
|
|
134
|
-
with:
|
|
135
|
-
node-version: '20'
|
|
136
|
-
cache: 'npm'
|
|
137
|
-
- run: npm ci
|
|
138
|
-
- name: Run tests (shard ${{ matrix.shard }}/4)
|
|
139
|
-
env:
|
|
140
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
141
|
-
run: vitest --run --shard=${{ matrix.shard }}/4
|
|
90
|
+
```yaml
|
|
91
|
+
- name: Run tests
|
|
92
|
+
run: npx vitest run
|
|
93
|
+
env:
|
|
94
|
+
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
142
95
|
```
|
|
143
96
|
|
|
144
|
-
|
|
97
|
+
</Accordion>
|
|
98
|
+
</AccordionGroup>
|
|
145
99
|
|
|
146
|
-
|
|
147
|
-
name: TestDriver Tests (Multi-Platform)
|
|
100
|
+
## Other CI providers
|
|
148
101
|
|
|
149
|
-
|
|
102
|
+
Copy an API key from the [API keys page](https://console.testdriver.ai/team/api-keys). Store it as a secret named `TD_API_KEY` in your CI provider, then use the config for your provider.
|
|
150
103
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
strategy:
|
|
155
|
-
fail-fast: false
|
|
156
|
-
matrix:
|
|
157
|
-
td-os: [linux, windows]
|
|
158
|
-
|
|
159
|
-
steps:
|
|
160
|
-
- uses: actions/checkout@v4
|
|
161
|
-
- uses: actions/setup-node@v4
|
|
162
|
-
with:
|
|
163
|
-
node-version: '20'
|
|
164
|
-
cache: 'npm'
|
|
165
|
-
- run: npm ci
|
|
166
|
-
- name: Run tests on ${{ matrix.td-os }}
|
|
167
|
-
env:
|
|
168
|
-
TD_API_KEY: ${{ secrets.TD_API_KEY }}
|
|
169
|
-
TD_OS: ${{ matrix.td-os }}
|
|
170
|
-
run: vitest --run
|
|
171
|
-
```
|
|
172
|
-
</Tab>
|
|
104
|
+
<Warning>
|
|
105
|
+
Never commit your API key to the repository. Keep it in your CI provider's secret store.
|
|
106
|
+
</Warning>
|
|
173
107
|
|
|
108
|
+
<Tabs>
|
|
174
109
|
<Tab title="GitLab CI">
|
|
175
|
-
### Adding Secrets
|
|
176
|
-
|
|
177
|
-
1. Go to your GitLab project
|
|
178
|
-
2. Navigate to **Settings** → **CI/CD** → **Variables**
|
|
179
|
-
3. Click **Add variable**
|
|
180
|
-
4. Key: `TD_API_KEY`, Value: your API key
|
|
181
|
-
5. Check **Mask variable** and click **Add variable**
|
|
182
110
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
Create `.gitlab-ci.yml`:
|
|
111
|
+
Add `TD_API_KEY` under **Settings → CI/CD → Variables**, with **Mask variable** checked. GitLab exposes project variables to every job.
|
|
186
112
|
|
|
187
113
|
```yaml .gitlab-ci.yml
|
|
188
|
-
stages:
|
|
189
|
-
- test
|
|
190
|
-
|
|
191
114
|
testdriver:
|
|
192
|
-
stage: test
|
|
193
115
|
image: node:20
|
|
194
|
-
cache:
|
|
195
|
-
paths:
|
|
196
|
-
- node_modules/
|
|
197
116
|
script:
|
|
198
117
|
- npm ci
|
|
199
|
-
- vitest
|
|
200
|
-
variables:
|
|
201
|
-
TD_API_KEY: $TD_API_KEY
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
### Parallel Execution
|
|
205
|
-
|
|
206
|
-
```yaml .gitlab-ci.yml
|
|
207
|
-
stages:
|
|
208
|
-
- test
|
|
209
|
-
|
|
210
|
-
.testdriver-base:
|
|
211
|
-
stage: test
|
|
212
|
-
image: node:20
|
|
213
|
-
cache:
|
|
214
|
-
paths:
|
|
215
|
-
- node_modules/
|
|
216
|
-
before_script:
|
|
217
|
-
- npm ci
|
|
218
|
-
variables:
|
|
219
|
-
TD_API_KEY: $TD_API_KEY
|
|
220
|
-
|
|
221
|
-
testdriver-shard-1:
|
|
222
|
-
extends: .testdriver-base
|
|
223
|
-
script:
|
|
224
|
-
- vitest --run --shard=1/4
|
|
225
|
-
|
|
226
|
-
testdriver-shard-2:
|
|
227
|
-
extends: .testdriver-base
|
|
228
|
-
script:
|
|
229
|
-
- vitest --run --shard=2/4
|
|
230
|
-
|
|
231
|
-
testdriver-shard-3:
|
|
232
|
-
extends: .testdriver-base
|
|
233
|
-
script:
|
|
234
|
-
- vitest --run --shard=3/4
|
|
235
|
-
|
|
236
|
-
testdriver-shard-4:
|
|
237
|
-
extends: .testdriver-base
|
|
238
|
-
script:
|
|
239
|
-
- vitest --run --shard=4/4
|
|
118
|
+
- npx vitest run
|
|
240
119
|
```
|
|
241
120
|
|
|
242
|
-
### Multi-Platform Testing
|
|
243
|
-
|
|
244
|
-
```yaml .gitlab-ci.yml
|
|
245
|
-
stages:
|
|
246
|
-
- test
|
|
247
|
-
|
|
248
|
-
.testdriver-base:
|
|
249
|
-
stage: test
|
|
250
|
-
image: node:20
|
|
251
|
-
cache:
|
|
252
|
-
paths:
|
|
253
|
-
- node_modules/
|
|
254
|
-
before_script:
|
|
255
|
-
- npm ci
|
|
256
|
-
variables:
|
|
257
|
-
TD_API_KEY: $TD_API_KEY
|
|
258
|
-
|
|
259
|
-
testdriver-linux:
|
|
260
|
-
extends: .testdriver-base
|
|
261
|
-
variables:
|
|
262
|
-
TD_OS: linux
|
|
263
|
-
script:
|
|
264
|
-
- vitest --run
|
|
265
|
-
|
|
266
|
-
testdriver-windows:
|
|
267
|
-
extends: .testdriver-base
|
|
268
|
-
variables:
|
|
269
|
-
TD_OS: windows
|
|
270
|
-
script:
|
|
271
|
-
- vitest --run
|
|
272
|
-
```
|
|
273
121
|
</Tab>
|
|
274
|
-
|
|
275
122
|
<Tab title="CircleCI">
|
|
276
|
-
### Adding Secrets
|
|
277
|
-
|
|
278
|
-
1. Go to your CircleCI project
|
|
279
|
-
2. Click **Project Settings** → **Environment Variables**
|
|
280
|
-
3. Click **Add Environment Variable**
|
|
281
|
-
4. Name: `TD_API_KEY`, Value: your API key
|
|
282
|
-
|
|
283
|
-
### Basic Config
|
|
284
|
-
|
|
285
|
-
Create `.circleci/config.yml`:
|
|
286
|
-
|
|
287
|
-
```yaml .circleci/config.yml
|
|
288
|
-
version: 2.1
|
|
289
|
-
|
|
290
|
-
jobs:
|
|
291
|
-
test:
|
|
292
|
-
docker:
|
|
293
|
-
- image: cimg/node:20.0
|
|
294
|
-
steps:
|
|
295
|
-
- checkout
|
|
296
|
-
- restore_cache:
|
|
297
|
-
keys:
|
|
298
|
-
- npm-deps-{{ checksum "package-lock.json" }}
|
|
299
|
-
- run: npm ci
|
|
300
|
-
- save_cache:
|
|
301
|
-
key: npm-deps-{{ checksum "package-lock.json" }}
|
|
302
|
-
paths:
|
|
303
|
-
- node_modules
|
|
304
|
-
- run:
|
|
305
|
-
name: Run TestDriver tests
|
|
306
|
-
command: vitest --run
|
|
307
|
-
environment:
|
|
308
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
309
|
-
|
|
310
|
-
workflows:
|
|
311
|
-
test:
|
|
312
|
-
jobs:
|
|
313
|
-
- test
|
|
314
|
-
```
|
|
315
123
|
|
|
316
|
-
|
|
124
|
+
Add `TD_API_KEY` under **Project Settings → Environment Variables**. CircleCI exposes project variables to every job.
|
|
317
125
|
|
|
318
126
|
```yaml .circleci/config.yml
|
|
319
127
|
version: 2.1
|
|
@@ -321,24 +129,11 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
321
129
|
jobs:
|
|
322
130
|
test:
|
|
323
131
|
docker:
|
|
324
|
-
- image: cimg/node:
|
|
325
|
-
parallelism: 4
|
|
132
|
+
- image: cimg/node:lts
|
|
326
133
|
steps:
|
|
327
134
|
- checkout
|
|
328
|
-
- restore_cache:
|
|
329
|
-
keys:
|
|
330
|
-
- npm-deps-{{ checksum "package-lock.json" }}
|
|
331
135
|
- run: npm ci
|
|
332
|
-
-
|
|
333
|
-
key: npm-deps-{{ checksum "package-lock.json" }}
|
|
334
|
-
paths:
|
|
335
|
-
- node_modules
|
|
336
|
-
- run:
|
|
337
|
-
name: Run TestDriver tests
|
|
338
|
-
command: |
|
|
339
|
-
vitest --run --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
|
|
340
|
-
environment:
|
|
341
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
136
|
+
- run: npx vitest run
|
|
342
137
|
|
|
343
138
|
workflows:
|
|
344
139
|
test:
|
|
@@ -346,92 +141,17 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
346
141
|
- test
|
|
347
142
|
```
|
|
348
143
|
|
|
349
|
-
### Multi-Platform Testing
|
|
350
|
-
|
|
351
|
-
```yaml .circleci/config.yml
|
|
352
|
-
version: 2.1
|
|
353
|
-
|
|
354
|
-
jobs:
|
|
355
|
-
test:
|
|
356
|
-
docker:
|
|
357
|
-
- image: cimg/node:20.0
|
|
358
|
-
parameters:
|
|
359
|
-
td-os:
|
|
360
|
-
type: string
|
|
361
|
-
steps:
|
|
362
|
-
- checkout
|
|
363
|
-
- run: npm ci
|
|
364
|
-
- run:
|
|
365
|
-
name: Run TestDriver tests on << parameters.td-os >>
|
|
366
|
-
command: vitest --run
|
|
367
|
-
environment:
|
|
368
|
-
TD_API_KEY: ${TD_API_KEY}
|
|
369
|
-
TD_OS: << parameters.td-os >>
|
|
370
|
-
|
|
371
|
-
workflows:
|
|
372
|
-
test:
|
|
373
|
-
jobs:
|
|
374
|
-
- test:
|
|
375
|
-
td-os: linux
|
|
376
|
-
- test:
|
|
377
|
-
td-os: windows
|
|
378
|
-
```
|
|
379
144
|
</Tab>
|
|
380
|
-
|
|
381
145
|
<Tab title="Azure Pipelines">
|
|
382
|
-
### Adding Secrets
|
|
383
|
-
|
|
384
|
-
1. Go to your Azure DevOps project
|
|
385
|
-
2. Navigate to **Pipelines** → **Library** → **Variable groups**
|
|
386
|
-
3. Create a new variable group or edit existing
|
|
387
|
-
4. Add variable: `TD_API_KEY` with your API key
|
|
388
|
-
5. Click the lock icon to make it secret
|
|
389
|
-
|
|
390
|
-
### Basic Pipeline
|
|
391
|
-
|
|
392
|
-
Create `azure-pipelines.yml`:
|
|
393
|
-
|
|
394
|
-
```yaml azure-pipelines.yml
|
|
395
|
-
trigger:
|
|
396
|
-
- main
|
|
397
|
-
|
|
398
|
-
pool:
|
|
399
|
-
vmImage: 'ubuntu-latest'
|
|
400
|
-
|
|
401
|
-
steps:
|
|
402
|
-
- task: NodeTool@0
|
|
403
|
-
inputs:
|
|
404
|
-
versionSpec: '20.x'
|
|
405
|
-
displayName: 'Setup Node.js'
|
|
406
146
|
|
|
407
|
-
|
|
408
|
-
displayName: 'Install dependencies'
|
|
409
|
-
|
|
410
|
-
- script: vitest --run
|
|
411
|
-
displayName: 'Run TestDriver tests'
|
|
412
|
-
env:
|
|
413
|
-
TD_API_KEY: $(TD_API_KEY)
|
|
414
|
-
```
|
|
415
|
-
|
|
416
|
-
### Parallel Execution
|
|
147
|
+
Add `TD_API_KEY` as a pipeline variable and mark it secret. Azure doesn't expose secret variables to scripts automatically, so map it with `env`.
|
|
417
148
|
|
|
418
149
|
```yaml azure-pipelines.yml
|
|
419
150
|
trigger:
|
|
420
151
|
- main
|
|
421
152
|
|
|
422
153
|
pool:
|
|
423
|
-
vmImage:
|
|
424
|
-
|
|
425
|
-
strategy:
|
|
426
|
-
matrix:
|
|
427
|
-
shard1:
|
|
428
|
-
SHARD: '1/4'
|
|
429
|
-
shard2:
|
|
430
|
-
SHARD: '2/4'
|
|
431
|
-
shard3:
|
|
432
|
-
SHARD: '3/4'
|
|
433
|
-
shard4:
|
|
434
|
-
SHARD: '4/4'
|
|
154
|
+
vmImage: ubuntu-latest
|
|
435
155
|
|
|
436
156
|
steps:
|
|
437
157
|
- task: NodeTool@0
|
|
@@ -439,248 +159,139 @@ Add it as a `TD_API_KEY` secret in the settings of your CI provider.
|
|
|
439
159
|
versionSpec: '20.x'
|
|
440
160
|
|
|
441
161
|
- script: npm ci
|
|
442
|
-
displayName:
|
|
162
|
+
displayName: Install dependencies
|
|
443
163
|
|
|
444
|
-
- script: vitest
|
|
445
|
-
displayName:
|
|
164
|
+
- script: npx vitest run
|
|
165
|
+
displayName: Run TestDriver tests
|
|
446
166
|
env:
|
|
447
167
|
TD_API_KEY: $(TD_API_KEY)
|
|
448
168
|
```
|
|
449
169
|
|
|
450
|
-
### Multi-Platform Testing
|
|
451
|
-
|
|
452
|
-
```yaml azure-pipelines.yml
|
|
453
|
-
trigger:
|
|
454
|
-
- main
|
|
455
|
-
|
|
456
|
-
pool:
|
|
457
|
-
vmImage: 'ubuntu-latest'
|
|
458
|
-
|
|
459
|
-
strategy:
|
|
460
|
-
matrix:
|
|
461
|
-
linux:
|
|
462
|
-
TD_OS: 'linux'
|
|
463
|
-
windows:
|
|
464
|
-
TD_OS: 'windows'
|
|
465
|
-
|
|
466
|
-
steps:
|
|
467
|
-
- task: NodeTool@0
|
|
468
|
-
inputs:
|
|
469
|
-
versionSpec: '20.x'
|
|
470
|
-
|
|
471
|
-
- script: npm ci
|
|
472
|
-
displayName: 'Install dependencies'
|
|
473
|
-
|
|
474
|
-
- script: vitest --run
|
|
475
|
-
displayName: 'Run TestDriver tests on $(TD_OS)'
|
|
476
|
-
env:
|
|
477
|
-
TD_API_KEY: $(TD_API_KEY)
|
|
478
|
-
TD_OS: $(TD_OS)
|
|
479
|
-
```
|
|
480
170
|
</Tab>
|
|
481
|
-
|
|
482
171
|
<Tab title="Jenkins">
|
|
483
|
-
### Adding Secrets
|
|
484
|
-
|
|
485
|
-
1. Go to **Manage Jenkins** → **Credentials**
|
|
486
|
-
2. Select the appropriate domain
|
|
487
|
-
3. Click **Add Credentials**
|
|
488
|
-
4. Kind: **Secret text**
|
|
489
|
-
5. ID: `td-api-key`, Secret: your API key
|
|
490
|
-
|
|
491
|
-
### Basic Pipeline
|
|
492
172
|
|
|
493
|
-
|
|
173
|
+
Add a **Secret text** credential with the ID `td-api-key` under **Manage Jenkins → Credentials**.
|
|
494
174
|
|
|
495
175
|
```groovy Jenkinsfile
|
|
496
176
|
pipeline {
|
|
497
|
-
agent {
|
|
498
|
-
|
|
499
|
-
image 'node:20'
|
|
500
|
-
}
|
|
501
|
-
}
|
|
502
|
-
|
|
177
|
+
agent { docker { image 'node:20' } }
|
|
178
|
+
|
|
503
179
|
environment {
|
|
504
180
|
TD_API_KEY = credentials('td-api-key')
|
|
505
181
|
}
|
|
506
|
-
|
|
182
|
+
|
|
507
183
|
stages {
|
|
508
|
-
stage('Install') {
|
|
509
|
-
steps {
|
|
510
|
-
sh 'npm ci'
|
|
511
|
-
}
|
|
512
|
-
}
|
|
513
|
-
|
|
514
184
|
stage('Test') {
|
|
515
185
|
steps {
|
|
516
|
-
sh '
|
|
186
|
+
sh 'npm ci'
|
|
187
|
+
sh 'npx vitest run'
|
|
517
188
|
}
|
|
518
189
|
}
|
|
519
190
|
}
|
|
520
191
|
}
|
|
521
192
|
```
|
|
522
193
|
|
|
523
|
-
|
|
194
|
+
</Tab>
|
|
195
|
+
</Tabs>
|
|
524
196
|
|
|
525
|
-
|
|
526
|
-
pipeline {
|
|
527
|
-
agent none
|
|
528
|
-
|
|
529
|
-
environment {
|
|
530
|
-
TD_API_KEY = credentials('td-api-key')
|
|
531
|
-
}
|
|
532
|
-
|
|
533
|
-
stages {
|
|
534
|
-
stage('Test') {
|
|
535
|
-
parallel {
|
|
536
|
-
stage('Shard 1') {
|
|
537
|
-
agent { docker { image 'node:20' } }
|
|
538
|
-
steps {
|
|
539
|
-
sh 'npm ci'
|
|
540
|
-
sh 'vitest --run --shard=1/4'
|
|
541
|
-
}
|
|
542
|
-
}
|
|
543
|
-
stage('Shard 2') {
|
|
544
|
-
agent { docker { image 'node:20' } }
|
|
545
|
-
steps {
|
|
546
|
-
sh 'npm ci'
|
|
547
|
-
sh 'vitest --run --shard=2/4'
|
|
548
|
-
}
|
|
549
|
-
}
|
|
550
|
-
stage('Shard 3') {
|
|
551
|
-
agent { docker { image 'node:20' } }
|
|
552
|
-
steps {
|
|
553
|
-
sh 'npm ci'
|
|
554
|
-
sh 'vitest --run --shard=3/4'
|
|
555
|
-
}
|
|
556
|
-
}
|
|
557
|
-
stage('Shard 4') {
|
|
558
|
-
agent { docker { image 'node:20' } }
|
|
559
|
-
steps {
|
|
560
|
-
sh 'npm ci'
|
|
561
|
-
sh 'vitest --run --shard=4/4'
|
|
562
|
-
}
|
|
563
|
-
}
|
|
564
|
-
}
|
|
565
|
-
}
|
|
566
|
-
}
|
|
567
|
-
}
|
|
568
|
-
```
|
|
197
|
+
## Split tests across jobs
|
|
569
198
|
|
|
570
|
-
|
|
199
|
+
Vitest runs test files in parallel within one job. To spread a large suite across several jobs, use Vitest's `--shard` flag:
|
|
571
200
|
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
stage('Linux') {
|
|
584
|
-
agent { docker { image 'node:20' } }
|
|
585
|
-
environment {
|
|
586
|
-
TD_OS = 'linux'
|
|
587
|
-
}
|
|
588
|
-
steps {
|
|
589
|
-
sh 'npm ci'
|
|
590
|
-
sh 'vitest --run'
|
|
591
|
-
}
|
|
592
|
-
}
|
|
593
|
-
stage('Windows') {
|
|
594
|
-
agent { docker { image 'node:20' } }
|
|
595
|
-
environment {
|
|
596
|
-
TD_OS = 'windows'
|
|
597
|
-
}
|
|
598
|
-
steps {
|
|
599
|
-
sh 'npm ci'
|
|
600
|
-
sh 'vitest --run'
|
|
601
|
-
}
|
|
602
|
-
}
|
|
603
|
-
}
|
|
604
|
-
}
|
|
605
|
-
}
|
|
606
|
-
}
|
|
607
|
-
```
|
|
608
|
-
</Tab>
|
|
609
|
-
</Tabs>
|
|
201
|
+
```yaml .github/workflows/testdriver.yml
|
|
202
|
+
jobs:
|
|
203
|
+
test:
|
|
204
|
+
runs-on: ubuntu-latest
|
|
205
|
+
permissions:
|
|
206
|
+
id-token: write
|
|
207
|
+
contents: read
|
|
208
|
+
strategy:
|
|
209
|
+
fail-fast: false
|
|
210
|
+
matrix:
|
|
211
|
+
shard: [1, 2, 3, 4]
|
|
610
212
|
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
it("should work on both Linux and Windows", async (context) => {
|
|
621
|
-
const os = process.env.TD_OS || 'linux';
|
|
622
|
-
|
|
623
|
-
const testdriver = TestDriver(context, {
|
|
624
|
-
os: os // 'linux' or 'windows'
|
|
625
|
-
});
|
|
626
|
-
|
|
627
|
-
await testdriver.provision.chrome({
|
|
628
|
-
url: 'https://example.com',
|
|
629
|
-
});
|
|
630
|
-
|
|
631
|
-
const result = await testdriver.assert("the page loaded successfully");
|
|
632
|
-
expect(result).toBeTruthy();
|
|
633
|
-
});
|
|
634
|
-
});
|
|
213
|
+
steps:
|
|
214
|
+
- uses: actions/checkout@v4
|
|
215
|
+
- uses: actions/setup-node@v4
|
|
216
|
+
with:
|
|
217
|
+
node-version: '20'
|
|
218
|
+
cache: 'npm'
|
|
219
|
+
- run: npm ci
|
|
220
|
+
- uses: testdriverai/action@stable
|
|
221
|
+
- run: npx vitest run --shard=${{ matrix.shard }}/4
|
|
635
222
|
```
|
|
636
223
|
|
|
637
|
-
|
|
224
|
+
Other providers work the same way. Run `npx vitest run --shard=<index>/<total>` in each parallel job.
|
|
225
|
+
|
|
226
|
+
## Run on Linux and Windows
|
|
638
227
|
|
|
639
|
-
|
|
640
|
-
a sandbox and you're already at that limit, the request is **queued** rather than
|
|
641
|
-
failed immediately: the SDK waits for a slot to free up, retrying every 10 seconds,
|
|
642
|
-
then proceeds automatically once one opens. This is what lets a parallel CI matrix
|
|
643
|
-
(many jobs starting at once) work on a plan with fewer slots than jobs — the extra
|
|
644
|
-
jobs simply wait their turn instead of erroring.
|
|
228
|
+
Set `TD_OS` to choose the sandbox's operating system. TestDriver reads it when it creates the sandbox, so your tests don't need any changes:
|
|
645
229
|
|
|
646
|
-
|
|
647
|
-
|
|
230
|
+
```yaml
|
|
231
|
+
strategy:
|
|
232
|
+
fail-fast: false
|
|
233
|
+
matrix:
|
|
234
|
+
td-os: [linux, windows]
|
|
235
|
+
|
|
236
|
+
steps:
|
|
237
|
+
# ...checkout, setup-node, npm ci, testdriverai/action
|
|
238
|
+
- run: npx vitest run
|
|
239
|
+
env:
|
|
240
|
+
TD_OS: ${{ matrix.td-os }}
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
Linux is the default.
|
|
244
|
+
|
|
245
|
+
<EnterpriseCta feature="Windows sandboxes" plural />
|
|
648
246
|
|
|
649
|
-
|
|
650
|
-
| ----- | -------- |
|
|
651
|
-
| _unset_ | Wait up to **60 seconds** (the default). |
|
|
652
|
-
| `TD_CONCURRENCY_MAX_WAIT=300` | Wait up to **300 seconds** (5 minutes) before giving up. |
|
|
653
|
-
| `TD_CONCURRENCY_MAX_WAIT=0` | **Don't queue** — fail on the first denial. |
|
|
247
|
+
An `os` option passed to `TestDriver()` takes priority over `TD_OS`. To branch on the platform inside a test, read `process.env.TD_OS`:
|
|
654
248
|
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
249
|
+
```javascript
|
|
250
|
+
const isWindows = process.env.TD_OS === 'windows';
|
|
251
|
+
|
|
252
|
+
await testdriver.exec({
|
|
253
|
+
language: isWindows ? 'pwsh' : 'sh',
|
|
254
|
+
code: isWindows ? 'Get-ChildItem $HOME' : 'ls ~',
|
|
255
|
+
});
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
## Concurrency limits
|
|
259
|
+
|
|
260
|
+
Your plan sets how many sandboxes can run at once. When a job asks for a sandbox at that limit, it waits for a free slot instead of failing.
|
|
261
|
+
|
|
262
|
+
- The SDK retries every 10 seconds.
|
|
263
|
+
- By default, it gives up after 60 seconds with a concurrency-limit error.
|
|
264
|
+
- Set `TD_CONCURRENCY_MAX_WAIT` (in seconds) to wait longer, or to `0` to fail on the first refusal.
|
|
659
265
|
|
|
660
266
|
```yaml
|
|
661
|
-
|
|
662
|
-
# Give each job up to 5 minutes to acquire a slot before failing.
|
|
663
|
-
- name: Run TestDriver tests
|
|
267
|
+
- run: npx vitest run
|
|
664
268
|
env:
|
|
665
|
-
|
|
666
|
-
TD_CONCURRENCY_MAX_WAIT: "300"
|
|
667
|
-
run: vitest run
|
|
269
|
+
TD_CONCURRENCY_MAX_WAIT: '300' # wait up to 5 minutes for a slot
|
|
668
270
|
```
|
|
669
271
|
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
to **fail fast** on a busy account (e.g. a quick smoke test that shouldn't sit
|
|
674
|
-
waiting). When jobs routinely give up waiting, that's the signal to
|
|
675
|
-
[add more slots](https://console.testdriver.ai/checkout/pro).
|
|
676
|
-
</Tip>
|
|
272
|
+
Raise the wait when you run more parallel jobs than your plan has slots. If jobs regularly give up waiting, run fewer at once with `--maxWorkers` or add slots to your plan.
|
|
273
|
+
|
|
274
|
+
## View results
|
|
677
275
|
|
|
678
|
-
|
|
276
|
+
Each run is recorded in the [TestDriver console](https://console.testdriver.ai), with a video replay, logs, and pass or fail status. When a run finishes, the SDK prints a link to it:
|
|
679
277
|
|
|
680
|
-
|
|
278
|
+
```text
|
|
279
|
+
TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/abc123
|
|
280
|
+
```
|
|
681
281
|
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
282
|
+
## Next steps
|
|
283
|
+
|
|
284
|
+
<CardGroup cols={2}>
|
|
285
|
+
<Card title="Run tests" icon="play" href="/copilot/running-tests">
|
|
286
|
+
Run tests locally, in parallel, and watch them live.
|
|
287
|
+
</Card>
|
|
288
|
+
<Card title="Add TestDriver to GitHub" icon="github" href="/quickstart-github">
|
|
289
|
+
Get an AI code review on every pull request.
|
|
290
|
+
</Card>
|
|
291
|
+
<Card title="Hosted plans" icon="cloud" href="/hosted">
|
|
292
|
+
Compare plans, testing minutes, and concurrency.
|
|
293
|
+
</Card>
|
|
294
|
+
<Card title="Speed up reruns with caching" icon="brain" href="/caching">
|
|
295
|
+
Reuse element locations from passing runs.
|
|
296
|
+
</Card>
|
|
297
|
+
</CardGroup>
|