@testdriverai/agent 7.11.229-test → 7.11.231-test

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (100) hide show
  1. package/docs/_data/examples-manifest.json +16 -16
  2. package/docs/_scripts/generate-examples.js +38 -45
  3. package/docs/agent.mdx +174 -93
  4. package/docs/assert.mdx +73 -269
  5. package/docs/aws-setup.mdx +196 -314
  6. package/docs/bring-your-own-machine.mdx +139 -206
  7. package/docs/cache.mdx +144 -151
  8. package/docs/caching.mdx +99 -89
  9. package/docs/captcha.mdx +126 -111
  10. package/docs/ci-cd.mdx +154 -543
  11. package/docs/click.mdx +74 -229
  12. package/docs/client.mdx +177 -231
  13. package/docs/copilot/auto-healing.mdx +86 -370
  14. package/docs/copilot/running-tests.mdx +121 -930
  15. package/docs/customizing-devices.mdx +74 -251
  16. package/docs/dashcam.mdx +72 -405
  17. package/docs/debugging-with-screenshots.mdx +105 -371
  18. package/docs/docs.json +75 -18
  19. package/docs/double-click.mdx +48 -68
  20. package/docs/errors.mdx +102 -176
  21. package/docs/events.mdx +170 -327
  22. package/docs/examples/{hover-text-with-description.mdx → add-to-cart.mdx} +30 -35
  23. package/docs/examples/assert.mdx +28 -41
  24. package/docs/examples/click-button.mdx +71 -0
  25. package/docs/examples/{hover-image.mdx → click-icon.mdx} +32 -28
  26. package/docs/examples/element-not-found.mdx +27 -26
  27. package/docs/examples/findall-coffee-icons.mdx +28 -32
  28. package/docs/examples/installer.mdx +61 -25
  29. package/docs/examples/launch-vscode-linux.mdx +28 -51
  30. package/docs/examples/navigate-and-scroll.mdx +82 -0
  31. package/docs/examples/parse.mdx +24 -28
  32. package/docs/examples/press-keys.mdx +28 -25
  33. package/docs/examples/scroll.mdx +25 -27
  34. package/docs/examples/type.mdx +26 -21
  35. package/docs/exec.mdx +92 -256
  36. package/docs/extract.mdx +69 -170
  37. package/docs/find.mdx +140 -736
  38. package/docs/focus-application.mdx +69 -246
  39. package/docs/generating-tests.mdx +103 -137
  40. package/docs/guide/best-practices-polling.mdx +65 -141
  41. package/docs/hosted.mdx +80 -144
  42. package/docs/hover.mdx +62 -235
  43. package/docs/locating-elements.mdx +74 -369
  44. package/docs/making-assertions.mdx +116 -74
  45. package/docs/mcp-setup.mdx +80 -63
  46. package/docs/mouse-down.mdx +48 -125
  47. package/docs/mouse-up.mdx +47 -129
  48. package/docs/options.mdx +75 -48
  49. package/docs/parse.mdx +95 -184
  50. package/docs/performing-actions.mdx +104 -94
  51. package/docs/press-keys.mdx +59 -299
  52. package/docs/provision.mdx +137 -182
  53. package/docs/quickstart-claude.mdx +85 -0
  54. package/docs/quickstart-github.mdx +62 -32
  55. package/docs/quickstart-manual.mdx +60 -33
  56. package/docs/quickstart-vscode.mdx +97 -0
  57. package/docs/quickstart.mdx +25 -9
  58. package/docs/redraw.mdx +101 -165
  59. package/docs/reusable-code.mdx +97 -200
  60. package/docs/right-click.mdx +47 -89
  61. package/docs/screenshot.mdx +47 -201
  62. package/docs/screenshots.mdx +87 -137
  63. package/docs/scroll.mdx +71 -154
  64. package/docs/secrets.mdx +96 -91
  65. package/docs/self-hosted.mdx +83 -142
  66. package/docs/snippets/enterprise.mdx +15 -0
  67. package/docs/styles.css +68 -0
  68. package/docs/test-results-json.mdx +167 -204
  69. package/docs/type.mdx +56 -304
  70. package/docs/variables.mdx +121 -81
  71. package/docs/wait.mdx +33 -31
  72. package/docs/what-is-testdriver.mdx +80 -32
  73. package/examples/{hover-text-with-description.test.mjs → add-to-cart.test.mjs} +4 -5
  74. package/examples/assert.test.mjs +3 -20
  75. package/examples/{hover-text.test.mjs → click-button.test.mjs} +3 -4
  76. package/examples/{hover-image.test.mjs → click-icon.test.mjs} +5 -6
  77. package/examples/element-not-found.test.mjs +3 -5
  78. package/examples/findall-coffee-icons.test.mjs +3 -3
  79. package/examples/installer.test.mjs +1 -1
  80. package/examples/launch-vscode-linux.test.mjs +6 -33
  81. package/examples/{scroll-keyboard.test.mjs → navigate-and-scroll.test.mjs} +4 -5
  82. package/examples/parse.test.mjs +1 -2
  83. package/examples/press-keys.test.mjs +4 -4
  84. package/examples/scroll.test.mjs +3 -6
  85. package/examples/type.test.mjs +5 -2
  86. package/lib/provision.js +6 -2
  87. package/package.json +1 -1
  88. package/sdk.d.ts +4 -2
  89. package/sdk.js +1 -1
  90. package/docs/claude-mcp-plugin.mdx +0 -244
  91. package/docs/elements.mdx +0 -606
  92. package/docs/examples/ai.mdx +0 -78
  93. package/docs/examples/captcha-api.mdx +0 -92
  94. package/docs/examples/chrome-extension.mdx +0 -156
  95. package/docs/examples/drag-and-drop.mdx +0 -100
  96. package/docs/examples/formatted-logging.mdx +0 -74
  97. package/docs/examples/hover-text.mdx +0 -70
  98. package/docs/examples/match-image.mdx +0 -97
  99. package/docs/examples/scroll-keyboard.mdx +0 -80
  100. package/docs/examples/scroll-until-text.mdx +0 -109
package/docs/options.mdx CHANGED
@@ -1,33 +1,47 @@
1
1
  ---
2
- title: "Options Reference"
3
- sidebarTitle: "Options Reference"
4
- description: "Every option you can pass to the TestDriver client, with defaults and examples"
2
+ title: "Options"
3
+ sidebarTitle: "Options reference"
4
+ description: "Every option you can pass to a TestDriver client, with its default."
5
5
  icon: "sliders"
6
6
  ---
7
7
 
8
- ## Overview
8
+ import { EnterpriseBadge, EnterpriseCta } from '/snippets/enterprise.mdx';
9
9
 
10
- You configure the SDK with an options object. You can pass the object in three places:
10
+ Options control the sandbox, the live preview, the AI, the cache, and redraw detection. A [quick reference](#quick-reference) table at the end lists every option and its default.
11
+
12
+ ## Where to set options
13
+
14
+ Pass options in any of three places:
11
15
 
12
16
  ```javascript
13
- // 1. Vitest hook (most common)
14
- const testdriver = TestDriver(context, { os: "windows" });
17
+ // 1. The Vitest hook (most common)
18
+ const testdriver = TestDriver(context, { os: 'windows' });
15
19
 
16
- // 2. Constructor
17
- const testdriver = new TestDriver(apiKey, { os: "windows" });
20
+ // 2. The client constructor
21
+ const testdriver = new TestDriver({ os: 'windows' });
22
+ ```
18
23
 
19
- // 3. Global plugin options in vitest.config.mjs
20
- testdriver({ os: "windows" })
24
+ ```javascript vitest.config.mjs
25
+ // 3. The Vitest plugin, for every test in the project
26
+ import { defineConfig } from 'vitest/config';
27
+ import TestDriver from 'testdriverai/vitest';
28
+
29
+ export default defineConfig({
30
+ test: {
31
+ reporters: ['default', TestDriver({ os: 'windows' })],
32
+ setupFiles: ['testdriverai/vitest/setup'],
33
+ },
34
+ });
21
35
  ```
22
36
 
23
- The hook merges the options in this order. A later source replaces an earlier source:
37
+ When the same option is set in more than one place, the most specific wins:
24
38
 
25
- 1. Default values
26
- 2. Environment variables (`TD_OS`, `TD_API_ROOT`, `TD_E2B_TEMPLATE_ID`)
27
- 3. Global plugin options
28
- 4. Options that you pass to the hook or the constructor
39
+ 1. Options you pass to the hook or the constructor
40
+ 2. Plugin options in `vitest.config`
41
+ 3. Environment variables: `TD_OS`, `TD_API_ROOT`, and `TD_E2B_TEMPLATE_ID`
42
+ 4. Defaults
29
43
 
30
- Options that you pass to [`connect()`](/client#connect) replace the constructor values for that connection only.
44
+ Options you pass to [`connect()`](/client#connect-options) override the client's values for that connection only.
31
45
 
32
46
  ## Sandbox
33
47
 
@@ -35,36 +49,42 @@ These options control the virtual machine that runs your test.
35
49
 
36
50
  <ParamField path="os" type="'linux' | 'windows'" default="linux">
37
51
  The operating system of the sandbox. You can also set this with the `TD_OS` environment variable.
52
+
53
+ <EnterpriseBadge /> `'windows'` is available on Enterprise plans.
38
54
  </ParamField>
39
55
 
40
56
  <ParamField path="resolution" type="string" default="1366x768">
41
- The screen resolution of the sandbox. Example: `'1920x1080'`. Custom resolutions are available only on Enterprise plans.
57
+ The screen resolution of the sandbox, such as `'1920x1080'`.
58
+
59
+ <EnterpriseBadge /> Custom resolutions are available on Enterprise plans.
42
60
  </ParamField>
43
61
 
44
62
  <ParamField path="newSandbox" type="boolean" default="true">
45
- Create a new sandbox for this run. Set to `false` to reuse a recent sandbox when one is available.
63
+ Create a new sandbox for this run. Set to `false` to reuse a recent sandbox when one is available. Defaults to `false` when you set `sandboxId` or `reconnect`.
46
64
  </ParamField>
47
65
 
48
66
  <ParamField path="reconnect" type="boolean" default="false">
49
- Reattach to the last used sandbox. The SDK reads the sandbox id from `.testdriver/last-sandbox`. The SDK writes this file on every successful connect.
67
+ Reattach to the last sandbox this project used. The SDK records it in `.testdriver/last-sandbox` on every successful connect.
50
68
 
51
- When `reconnect` is `true`, the SDK skips provision methods (`chrome`, `vscode`, `installer`, and others). The application is already running. The previous sandbox must still be alive. See [`keepAlive`](#keepalive) and the [Run guide](/copilot/running-tests#keeping-machines-alive-between-runs).
69
+ While `reconnect` is `true`, provision methods such as `chrome()` and `installer()` are skipped, because the app is already running. The sandbox must still be alive. See [`keepAlive`](#param-keep-alive) and [Reuse a sandbox while you iterate](/copilot/running-tests#reuse-a-sandbox-while-you-iterate).
52
70
  </ParamField>
53
71
 
54
72
  <ParamField path="sandboxId" type="string">
55
- Reattach to a specific sandbox id. Use this for CI matrices or to pin a chain of tests to a known VM. This option implies `reconnect: true`. The SDK skips provision calls.
73
+ Attach to a specific sandbox by its id, for example to pin a chain of tests to one machine. This sets `newSandbox` to `false`.
74
+
75
+ Unlike `reconnect`, `sandboxId` alone doesn't skip provision calls. Set `reconnect: true` as well if the app is already running.
56
76
  </ParamField>
57
77
 
58
78
  <ParamField path="keepAlive" type="number" default="60000">
59
- The number of milliseconds to keep the sandbox alive after `disconnect()`. Set to `0` to end the sandbox at once. A longer value lets you reconnect to the same sandbox for debugging.
79
+ The number of milliseconds to keep the sandbox running after `disconnect()`, so you can reconnect to the same sandbox for debugging. Values above 30 minutes are capped at 30 minutes, and `0` uses the default.
60
80
  </ParamField>
61
81
 
62
82
  <ParamField path="debugOnFailure" type="boolean" default="false">
63
- Keep the sandbox alive when a test fails. The SDK prints the sandbox id to the console. You can then reconnect with `sandboxId` and debug the failure. This option is available only in the Vitest hook.
83
+ Keep the sandbox running when a test fails, and print its id so you can reconnect with `sandboxId` and debug. Available only in the Vitest hook.
64
84
  </ParamField>
65
85
 
66
86
  <ParamField path="environment" type="object">
67
- Additional environment variables to pass to the sandbox.
87
+ Extra `TD_*` settings for the SDK itself, as key-value pairs. These configure TestDriver on your machine, not environment variables inside the sandbox. To set variables in the sandbox, run a command with [`exec()`](/exec).
68
88
  </ParamField>
69
89
 
70
90
  ### Example
@@ -86,7 +106,7 @@ These options control what you see while the test runs.
86
106
  The preview mode for live test visualization.
87
107
 
88
108
  - `"browser"` opens the debugger in your default browser.
89
- - `"ide"` opens the preview in the IDE panel. This mode needs the TestDriver extension for VS Code or Cursor.
109
+ - `"ide"` opens the preview in an editor panel. This needs the [TestDriver extension](/quickstart-vscode) for VS Code or Cursor.
90
110
  - `"none"` runs without a visual preview. Use this in CI.
91
111
  </ParamField>
92
112
 
@@ -99,13 +119,11 @@ These options control what you see while the test runs.
99
119
  </ParamField>
100
120
 
101
121
  <ParamField path="autoScreenshots" type="boolean" default="false">
102
- Capture a screenshot before and after each command. The SDK saves the screenshots to `.testdriver/screenshots/<test>/`.
103
-
104
- The file name format is `<seq>-<action>-<phase>-L<line>-<description>.png`. Example: `001-click-before-L42-submit-button.png`. See [Debugging with screenshots](/debugging-with-screenshots).
122
+ Save a screenshot before and after each `testdriver` method call and each `find()`, and when one fails, to `.testdriver/screenshots/<test file>/`. Actions on an element, like `element.click()`, aren't captured. Files are named like `001-find-before-L42-submit-button.png`. See [Screenshots](/screenshots).
105
123
  </ParamField>
106
124
 
107
125
  <ParamField path="logging" type="boolean" default="true">
108
- Write SDK logs to the console. You can change this at runtime with [`setLogging()`](/client#setlogging).
126
+ Write SDK logs to the console. You can change this at runtime with [`setLogging()`](/client#setlogging-enabled).
109
127
  </ParamField>
110
128
 
111
129
  <ParamField path="analytics" type="boolean" default="true">
@@ -126,13 +144,21 @@ const testdriver = TestDriver(context, {
126
144
  These options control how the AI locates elements and checks assertions.
127
145
 
128
146
  <ParamField path="verify" type="boolean" default="false">
129
- Run a second AI check on every `find()` result. The check confirms that the coordinates match the requested element. This catches incorrect positions. It adds latency to each call.
147
+ Double-check every `find()` result with a second AI call. Off by default, because it makes each call slower.
130
148
 
131
- You can replace this value for one call with `find(description, { verify: true })`. See [find](/find).
149
+ When on, after the AI locates an element:
150
+
151
+ 1. A second AI looks at that spot and confirms it's the element you described.
152
+ 2. If it isn't, TestDriver locates the element again with a hint about what was wrong, then checks once more.
153
+ 3. If that check also fails, `find()` returns an element whose `found()` is `false`.
154
+
155
+ Results that fail the check are never cached. Results served from the cache aren't re-checked.
156
+
157
+ Override it for one call with `find(description, { verify: true })`. See [find()](/find).
132
158
  </ParamField>
133
159
 
134
160
  <ParamField path="ai" type="object">
135
- Global AI sampling configuration. These values apply to `find()` verification and to `assert()`. You can replace them for one call with the `ai` option on that call.
161
+ Global AI sampling configuration. These values apply to `find()` verification and to `assert()`. You can override them for one `assert()` call with its `ai` option. `find()` ignores a per-call `ai` option, so set it here.
136
162
 
137
163
  <Expandable title="properties">
138
164
  <ParamField path="temperature" type="number">
@@ -166,16 +192,14 @@ const testdriver = TestDriver(context, {
166
192
 
167
193
  ## Caching
168
194
 
169
- These options control the element cache. The cache stores the position of an element after the first `find()`. Later runs reuse the position when the screen has not changed. See [Caching](/caching).
195
+ These options control the element cache, which reuses element positions from a passing run when the screen hasn't changed. See [Caching](/caching).
170
196
 
171
197
  <ParamField path="cache" type="boolean | object" default="true">
172
- Enable or disable the cache. Pass an object to set the match thresholds.
198
+ Pass `false` to turn the cache off, or an object to set the match thresholds.
173
199
 
174
- <Expandable title="advanced config">
175
- <ParamField path="enabled" type="boolean" default="true">
176
- Enable or disable the cache.
177
- </ParamField>
200
+ `cache: { enabled: false }` does not turn the cache off. Use `cache: false` or `TD_NO_CACHE=true`.
178
201
 
202
+ <Expandable title="advanced config">
179
203
  <ParamField path="thresholds" type="object">
180
204
  Fine-tune the cache matching.
181
205
 
@@ -203,7 +227,7 @@ These options control the element cache. The cache stores the position of an ele
203
227
  </ParamField>
204
228
 
205
229
  <ParamField path="cacheKey" type="string">
206
- The cache key for element location. When you set a key, the SDK ties the cache to that key. Use a different key for each test to keep the caches separate.
230
+ The key the cache is stored under. By default, the SDK uses a hash of the test file, so editing the file starts a fresh cache. Set your own key to keep the cache across edits, or to share it between files.
207
231
  </ParamField>
208
232
 
209
233
  <ParamField path="cacheThreshold" type="object">
@@ -215,7 +239,6 @@ These options control the element cache. The cache stores the position of an ele
215
239
  ```javascript
216
240
  const testdriver = TestDriver(context, {
217
241
  cache: {
218
- enabled: true,
219
242
  thresholds: {
220
243
  find: { screen: 0.05, element: 0.8 },
221
244
  assert: 0.05,
@@ -227,14 +250,14 @@ const testdriver = TestDriver(context, {
227
250
 
228
251
  ## Redraw detection
229
252
 
230
- These options control how the SDK waits for the screen to settle after an action. See [Redraw](/redraw).
253
+ These options make the SDK wait for the screen to settle after each action. Redraw detection is off by default. See [Redraw](/redraw).
231
254
 
232
- <ParamField path="redraw" type="boolean | object" default="true">
233
- Enable or disable screen-change detection. Pass an object to set the thresholds.
255
+ <ParamField path="redraw" type="boolean | object" default="false">
256
+ Turn on screen-change detection with `true`, or pass an object to set the thresholds.
234
257
 
235
258
  <Expandable title="advanced config">
236
259
  <ParamField path="enabled" type="boolean" default="true">
237
- Enable or disable redraw detection.
260
+ Whether redraw detection is on when you pass an object.
238
261
  </ParamField>
239
262
 
240
263
  <ParamField path="thresholds" type="object">
@@ -242,7 +265,7 @@ These options control how the SDK waits for the screen to settle after an action
242
265
 
243
266
  <Expandable title="properties">
244
267
  <ParamField path="screen" type="number | false" default="0.05">
245
- The pixel diff threshold (0 to 1). Set to `false` to disable screen redraw detection.
268
+ Set to `false` to turn off screen-change detection and wait only for the network. Any number keeps it on; the number itself isn't used, because the change threshold is fixed.
246
269
  </ParamField>
247
270
 
248
271
  <ParamField path="network" type="boolean" default="false">
@@ -269,7 +292,9 @@ const testdriver = TestDriver(context, {
269
292
 
270
293
  ## Self-hosted and infrastructure
271
294
 
272
- Most users do not need these options. Use them for self-hosted or custom deployments.
295
+ Most projects don't need these. They're for self-hosted or custom deployments.
296
+
297
+ <EnterpriseCta feature="Self-hosting" />
273
298
 
274
299
  <ParamField path="apiRoot" type="string">
275
300
  The API endpoint URL. The default depends on the release channel. You can also set this with the `TD_API_ROOT` environment variable.
@@ -277,6 +302,8 @@ Most users do not need these options. Use them for self-hosted or custom deploym
277
302
 
278
303
  <ParamField path="ip" type="string">
279
304
  The IP address of a running sandbox. The SDK connects to this address directly.
305
+
306
+ <EnterpriseBadge /> Connecting to your own machines is available on Enterprise plans.
280
307
  </ParamField>
281
308
 
282
309
  <ParamField path="sandboxAmi" type="string">
@@ -312,7 +339,7 @@ Most users do not need these options. Use them for self-hosted or custom deploym
312
339
  | `ai` | `object` | — |
313
340
  | `cache` | `boolean \| object` | `true` |
314
341
  | `cacheKey` | `string` | — |
315
- | `redraw` | `boolean \| object` | `true` |
342
+ | `redraw` | `boolean \| object` | `false` |
316
343
  | `apiRoot` | `string` | channel default |
317
344
  | `ip` | `string` | — |
318
345
  | `sandboxAmi` | `string` | — |
package/docs/parse.mdx CHANGED
@@ -1,237 +1,148 @@
1
1
  ---
2
2
  title: "parse()"
3
3
  sidebarTitle: "parse"
4
- description: "Detect all UI elements on screen using OmniParser"
4
+ description: "List every UI element on screen, with its type, text, and position."
5
5
  icon: "diagram-project"
6
6
  ---
7
7
 
8
- ## Overview
8
+ Detect every element on the current screen and get back a list with each one's type, text, and position.
9
9
 
10
- Parse the screen with OmniParser v2 to find all visible UI elements. TestDriver returns structured data. This includes element types, text content, interactivity levels, and bounding box coordinates.
10
+ ```javascript
11
+ const { elements } = await testdriver.parse();
12
+ console.log(`Found ${elements.length} elements`);
13
+ ```
14
+
15
+ `parse()` uses [OmniParser v2](https://github.com/microsoft/OmniParser). It works from the screenshot alone, so it doesn't need a DOM or accessibility tree.
11
16
 
12
- This method examines all of the screen and returns each element that it finds. Use it for these:
13
- - To understand the full UI layout of a screen
14
- - To find all clickable or interactive elements
15
- - To build custom element logic
16
- - To debug which elements TestDriver can find
17
- - To do an accessibility audit
17
+ To act on one specific element, use [`find()`](/find) instead. Use `parse()` when you need an inventory of the whole screen.
18
18
 
19
19
  <Note>
20
- **Availability**: `parse()` needs an enterprise plan or a self-hosted plan. It uses OmniParser v2 on the server for element detection.
20
+ `parse()` requires an Enterprise or self-hosted plan.
21
21
  </Note>
22
22
 
23
- ## Syntax
23
+ ## Signature
24
24
 
25
- ```javascript
26
- const result = await testdriver.parse()
25
+ ```typescript
26
+ testdriver.parse(): Promise<ParseResult>
27
27
  ```
28
28
 
29
29
  ## Parameters
30
30
 
31
- None.
31
+ None. `parse()` analyzes the screen as it is when you call it.
32
32
 
33
33
  ## Returns
34
34
 
35
- `Promise<ParseResult>` - Object containing detected UI elements
36
-
37
- ### ParseResult
38
-
39
- | Property | Type | Description |
40
- |----------|------|-------------|
41
- | `elements` | `ParsedElement[]` | Array of detected UI elements |
42
- | `annotatedImageUrl` | `string` | URL of the annotated screenshot with bounding boxes |
43
- | `imageWidth` | `number` | Width of the analyzed screenshot |
44
- | `imageHeight` | `number` | Height of the analyzed screenshot |
45
-
46
- ### ParsedElement
47
-
48
- | Property | Type | Description |
49
- |----------|------|-------------|
50
- | `index` | `number` | Element index |
51
- | `type` | `string` | Element type (e.g. `"text"`, `"icon"`, `"button"`) |
52
- | `content` | `string` | Text content or description of the element |
53
- | `interactivity` | `string` | Interactivity level (e.g. `"clickable"`, `"non-interactive"`) |
54
- | `bbox` | `object` | Bounding box in pixel coordinates `{x0, y0, x1, y1}` |
55
- | `boundingBox` | `object` | Bounding box as `{left, top, width, height}` |
35
+ <ResponseField name="ParseResult" type="object">
36
+ <Expandable title="properties">
37
+ <ResponseField name="elements" type="ParsedElement[]">
38
+ One entry per detected element.
39
+ <Expandable title="ParsedElement">
40
+ <ResponseField name="index" type="number">
41
+ The element's position in the list, starting at `0`.
42
+ </ResponseField>
43
+ <ResponseField name="type" type="string">
44
+ What OmniParser detected, such as `'text'` or `'icon'`.
45
+ </ResponseField>
46
+ <ResponseField name="content" type="string">
47
+ The element's text, or a short description of an icon.
48
+ </ResponseField>
49
+ <ResponseField name="interactivity" type="boolean">
50
+ `true` when OmniParser thinks the element can be clicked.
51
+ </ResponseField>
52
+ <ResponseField name="bbox" type="object">
53
+ The corners in pixels: `{ x0, y0, x1, y1 }`, where `(x0, y0)` is top-left and `(x1, y1)` is bottom-right.
54
+ </ResponseField>
55
+ <ResponseField name="boundingBox" type="object">
56
+ The same box as `{ left, top, width, height }`, in pixels.
57
+ </ResponseField>
58
+ </Expandable>
59
+ </ResponseField>
60
+ <ResponseField name="annotatedImageUrl" type="string | null">
61
+ A URL of the screenshot with every detected element outlined.
62
+ </ResponseField>
63
+ <ResponseField name="imageWidth" type="number">
64
+ The width of the analyzed screenshot, in pixels.
65
+ </ResponseField>
66
+ <ResponseField name="imageHeight" type="number">
67
+ The height of the analyzed screenshot, in pixels.
68
+ </ResponseField>
69
+ </Expandable>
70
+ </ResponseField>
71
+
72
+ <Warning>
73
+ `interactivity` is a boolean, although the TypeScript types in `sdk.d.ts` declare it as a string. Compare it with `true`, not with a string like `'clickable'`.
74
+ </Warning>
75
+
76
+ ## Errors
77
+
78
+ `parse()` throws if the screen analysis fails. The error message comes from the TestDriver API.
56
79
 
57
80
  ## Examples
58
81
 
59
- ### Get All Elements on Screen
82
+ ### List what's on screen
60
83
 
61
84
  ```javascript
62
- const result = await testdriver.parse();
63
- console.log(`Found ${result.elements.length} elements`);
85
+ const { elements } = await testdriver.parse();
64
86
 
65
- result.elements.forEach((el, i) => {
66
- console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
67
- });
87
+ for (const element of elements) {
88
+ console.log(`[${element.type}] "${element.content}" clickable=${element.interactivity}`);
89
+ }
68
90
  ```
69
91
 
70
- ### Find Clickable Elements
92
+ ### Find the clickable elements
71
93
 
72
94
  ```javascript
73
- const result = await testdriver.parse();
74
-
75
- const clickable = result.elements.filter(e => e.interactivity === 'clickable');
76
- console.log(`Found ${clickable.length} clickable elements`);
77
-
78
- clickable.forEach(el => {
79
- console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
80
- });
95
+ const { elements } = await testdriver.parse();
96
+ const clickable = elements.filter((element) => element.interactivity);
81
97
  ```
82
98
 
83
- ### Find and Click an Element by Content
99
+ ### Click an element by its text
84
100
 
85
101
  ```javascript
86
- const result = await testdriver.parse();
87
-
88
- // Find a "Submit" button
89
- const submitBtn = result.elements.find(e =>
90
- e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
91
- );
92
-
93
- if (submitBtn) {
94
- // Calculate center of the bounding box
95
- const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
96
- const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
97
-
102
+ const { elements } = await testdriver.parse();
103
+ const signIn = elements.find((element) => element.content.includes('Sign in'));
104
+
105
+ if (signIn) {
106
+ const x = Math.round((signIn.bbox.x0 + signIn.bbox.x1) / 2);
107
+ const y = Math.round((signIn.bbox.y0 + signIn.bbox.y1) / 2);
98
108
  await testdriver.click({ x, y });
99
109
  }
100
110
  ```
101
111
 
102
- ### Filter by Element Type
103
-
104
- ```javascript
105
- const result = await testdriver.parse();
106
-
107
- // Get all text elements
108
- const textElements = result.elements.filter(e => e.type === 'text');
109
- textElements.forEach(e => console.log(`Text: "${e.content}"`));
112
+ Clicking by coordinates breaks when the layout changes. Prefer `find()` for clicks.
110
113
 
111
- // Get all icons
112
- const icons = result.elements.filter(e => e.type === 'icon');
113
- console.log(`Found ${icons.length} icons`);
114
+ ### See what was detected
114
115
 
115
- // Get all buttons
116
- const buttons = result.elements.filter(e => e.type === 'button');
117
- console.log(`Found ${buttons.length} buttons`);
116
+ ```javascript
117
+ const { annotatedImageUrl } = await testdriver.parse();
118
+ console.log(annotatedImageUrl);
118
119
  ```
119
120
 
120
- ### Build Custom Assertions
121
+ ## Complete example
121
122
 
122
- ```javascript
123
- import { describe, expect, it } from "vitest";
124
- import { TestDriver } from "testdriverai/vitest/hooks";
123
+ ```javascript tests/login-form.test.mjs
124
+ import { describe, expect, it } from 'vitest';
125
+ import { TestDriver } from 'testdriverai/vitest/hooks';
125
126
 
126
- describe("Login Page", () => {
127
- it("should have expected form elements", async (context) => {
127
+ describe('Login page', () => {
128
+ it('shows the username and password fields', async (context) => {
128
129
  const testdriver = TestDriver(context);
129
-
130
- await testdriver.provision.chrome({
131
- url: 'https://myapp.com/login',
132
- });
133
-
134
- const result = await testdriver.parse();
135
-
136
- // Assert expected elements exist
137
- const textContent = result.elements.map(e => e.content.toLowerCase());
138
- expect(textContent).toContain('email');
139
- expect(textContent).toContain('password');
140
-
141
- // Assert there are clickable elements
142
- const clickable = result.elements.filter(e => e.interactivity === 'clickable');
143
- expect(clickable.length).toBeGreaterThan(0);
144
- });
145
- });
146
- ```
130
+ await testdriver.provision.chrome({ url: 'https://testdriver-sandbox.vercel.app/login' });
147
131
 
148
- ### Use Bounding Box Coordinates
132
+ const { elements } = await testdriver.parse();
133
+ const text = elements.map((element) => element.content.toLowerCase()).join(' ');
149
134
 
150
- ```javascript
151
- const result = await testdriver.parse();
152
-
153
- result.elements.forEach(el => {
154
- // Pixel coordinates
155
- console.log(`Element "${el.content}":`);
156
- console.log(` bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
157
- console.log(` size: ${el.boundingBox.width}x${el.boundingBox.height}`);
158
- console.log(` position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
135
+ expect(text).toContain('username');
136
+ expect(text).toContain('password');
137
+ expect(elements.some((element) => element.interactivity)).toBe(true);
138
+ });
159
139
  });
160
140
  ```
161
141
 
162
- ### View Annotated Screenshot
163
-
164
- ```javascript
165
- const result = await testdriver.parse();
166
-
167
- // The annotated image shows all detected elements with bounding boxes
168
- console.log('Annotated screenshot:', result.annotatedImageUrl);
169
- console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
170
- ```
171
-
172
- ## How It Works
173
-
174
- 1. TestDriver captures a screenshot of the current screen
175
- 2. The image is sent to the TestDriver API
176
- 3. OmniParser v2 analyzes the image to detect all UI elements
177
- 4. Each element is classified by type (text, icon, button, etc.) and interactivity
178
- 5. Bounding box coordinates are returned in pixel coordinates matching the screen resolution
179
-
180
- <Note>
181
- OmniParser detects elements visually — it works with any UI framework, native apps, and even non-standard interfaces. It does not rely on DOM or accessibility trees.
182
- </Note>
183
-
184
- ## Best Practices
185
-
186
- <AccordionGroup>
187
- <Accordion title="Use find() for targeting specific elements">
188
- For locating and interacting with a specific element, prefer `find()` which uses AI vision. Use `parse()` when you need a complete inventory of all elements on screen.
189
-
190
- ```javascript
191
- // Prefer this for clicking a specific element
192
- await testdriver.find("Submit button").click();
193
-
194
- // Use parse() for full UI analysis
195
- const result = await testdriver.parse();
196
- const allButtons = result.elements.filter(e => e.type === 'button');
197
- ```
198
- </Accordion>
199
-
200
- <Accordion title="Filter by interactivity">
201
- Use the `interactivity` field to distinguish between clickable and non-interactive elements.
202
-
203
- ```javascript
204
- const result = await testdriver.parse();
205
- const interactive = result.elements.filter(e => e.interactivity === 'clickable');
206
- const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
207
- ```
208
- </Accordion>
209
-
210
- <Accordion title="Wait for content to load">
211
- If elements aren't being detected, the page may not be fully loaded. Add a wait first.
212
-
213
- ```javascript
214
- // Wait for page to stabilize
215
- await testdriver.wait(2000);
216
-
217
- // Then parse
218
- const result = await testdriver.parse();
219
- ```
220
- </Accordion>
221
-
222
- <Accordion title="Use the annotated image for debugging">
223
- The `annotatedImageUrl` provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
224
-
225
- ```javascript
226
- const result = await testdriver.parse();
227
- console.log('View annotated screenshot:', result.annotatedImageUrl);
228
- ```
229
- </Accordion>
230
- </AccordionGroup>
142
+ See the [parse example](/examples/parse) for another complete test.
231
143
 
232
144
  ## Related
233
145
 
234
- - [find()](/find) - AI-powered element location
235
- - [assert()](/assert) - Make AI-powered assertions about screen state
236
- - [screenshot()](/screenshot) - Capture screenshots
237
- - [Elements Reference](/elements) - Complete Element API
146
+ - [find()](/find): locate one element and act on it
147
+ - [extract()](/extract): read one value off the screen
148
+ - [click()](/click): click a point by its coordinates