testdriverai 5.6.3 → 5.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agent.js +17 -15
- package/docs/account/dashboard.mdx +3 -6
- package/docs/account/enterprise.mdx +16 -15
- package/docs/account/pricing.mdx +4 -4
- package/docs/account/projects.mdx +2 -2
- package/docs/account/team.mdx +3 -3
- package/docs/action/browser.mdx +9 -9
- package/docs/action/os.mdx +12 -12
- package/docs/action/output.mdx +10 -10
- package/docs/action/performance.mdx +11 -11
- package/docs/{guide/lifecycle.mdx → action/pre-post-scripts.mdx} +87 -69
- package/docs/action/secrets.mdx +11 -11
- package/docs/action/setup.mdx +8 -8
- package/docs/apps/chrome-extensions.mdx +9 -9
- package/docs/apps/desktop-apps.mdx +6 -5
- package/docs/apps/mobile-apps.mdx +3 -3
- package/docs/apps/static-websites.mdx +2 -3
- package/docs/bugs/jira.mdx +16 -16
- package/docs/cli/overview.mdx +8 -8
- package/docs/commands/exec.mdx +152 -10
- package/docs/commands/focus-application.mdx +1 -1
- package/docs/commands/hover-image.mdx +1 -1
- package/docs/commands/hover-text.mdx +2 -2
- package/docs/commands/if.mdx +1 -1
- package/docs/commands/issues.mdx +8 -6
- package/docs/commands/match-image.mdx +2 -2
- package/docs/commands/press-keys.mdx +19 -2
- package/docs/commands/run.mdx +1 -1
- package/docs/commands/scroll-until-image.mdx +1 -1
- package/docs/commands/scroll-until-text.mdx +2 -2
- package/docs/commands/scroll.mdx +2 -2
- package/docs/commands/type.mdx +1 -1
- package/docs/commands/wait-for-image.mdx +1 -1
- package/docs/commands/wait-for-text.mdx +1 -1
- package/docs/commands/wait.mdx +1 -1
- package/docs/docs.json +9 -9
- package/docs/exporting/playwright.mdx +16 -16
- package/docs/features/auto-healing.mdx +5 -5
- package/docs/features/cross-platform.mdx +4 -4
- package/docs/features/generation.mdx +7 -4
- package/docs/features/github.mdx +21 -21
- package/docs/features/parallel-testing.mdx +9 -9
- package/docs/features/reusable-snippets.mdx +7 -7
- package/docs/features/selectorless.mdx +4 -4
- package/docs/features/visual-assertions.mdx +26 -23
- package/docs/getting-started/ci.mdx +19 -19
- package/docs/getting-started/editing.mdx +12 -8
- package/docs/getting-started/generating.mdx +4 -5
- package/docs/getting-started/running.mdx +3 -3
- package/docs/getting-started/setup.mdx +9 -9
- package/docs/getting-started/vscode.mdx +63 -85
- package/docs/getting-started/writing.mdx +11 -7
- package/docs/guide/assertions.mdx +21 -21
- package/docs/guide/authentication.mdx +11 -11
- package/docs/guide/code.mdx +6 -6
- package/docs/guide/environment-variables.mdx +9 -9
- package/docs/guide/lifecycle-prerun.mdx +191 -0
- package/docs/guide/locating.mdx +15 -18
- package/docs/guide/variables.mdx +8 -8
- package/docs/guide/waiting.mdx +10 -10
- package/docs/importing/csv.mdx +23 -24
- package/docs/importing/gherkin.mdx +20 -20
- package/docs/importing/jira.mdx +22 -23
- package/docs/importing/testrail.mdx +17 -17
- package/docs/integrations/electron.mdx +7 -7
- package/docs/integrations/netlify.mdx +7 -7
- package/docs/integrations/vercel.mdx +13 -13
- package/docs/interactive/assert.mdx +3 -3
- package/docs/interactive/dry.mdx +7 -7
- package/docs/interactive/explore.mdx +9 -9
- package/docs/interactive/generate.mdx +2 -2
- package/docs/interactive/run.mdx +10 -10
- package/docs/interactive/save.mdx +3 -3
- package/docs/interactive/undo.mdx +1 -1
- package/docs/overview/comparison.mdx +6 -6
- package/docs/overview/faq.mdx +2 -2
- package/docs/overview/quickstart.mdx +1 -1
- package/docs/overview/upgrading.mdx +8 -4
- package/docs/overview/what-is-testdriver.mdx +6 -6
- package/docs/quickstart.mdx +1 -1
- package/docs/scenarios/ai-chatbot.mdx +3 -3
- package/docs/scenarios/cookie-banner.mdx +2 -2
- package/docs/scenarios/file-upload.mdx +2 -2
- package/docs/scenarios/form-filling.mdx +3 -3
- package/docs/scenarios/log-in.mdx +3 -3
- package/docs/scenarios/pdf-generation.mdx +3 -3
- package/docs/scenarios/spell-check.mdx +2 -2
- package/docs/security/action.mdx +6 -6
- package/docs/security/agent.mdx +10 -10
- package/docs/security/platform.mdx +10 -8
- package/docs/snippets/test-prereqs.mdx +1 -1
- package/docs/snippets/yml-warning.mdx +1 -0
- package/docs/tutorials/advanced-test.mdx +9 -9
- package/docs/tutorials/basic-test.mdx +3 -3
- package/package.json +1 -1
- package/styles/config/vocabularies/Docs/accept.txt +32 -4
- package/testdriver/spotify.yaml +5 -0
- package/docs/action/prerun.mdx +0 -137
|
@@ -21,7 +21,7 @@ The `/generate` command is an experimental feature that instructs TestDriver to
|
|
|
21
21
|
- TestDriver generates 10 Markdown files every time the `/generate` command is called.
|
|
22
22
|
- The generated files are stored in the `./testdriver/generate/*.md` directory.
|
|
23
23
|
|
|
24
|
-
## Example
|
|
24
|
+
## Example output
|
|
25
25
|
Here's an example of a generated test file (`test-search-function.md`):
|
|
26
26
|
```markdown
|
|
27
27
|
1. Click on the search icon.
|
|
@@ -31,7 +31,7 @@ Here's an example of a generated test file (`test-search-function.md`):
|
|
|
31
31
|
|
|
32
32
|
## Protips
|
|
33
33
|
- Use `/generate` to quickly create exploratory tests for regression testing or feature validation.
|
|
34
|
-
- Combine `/generate` with the GitHub
|
|
34
|
+
- Combine `/generate` with the GitHub Action to generate and run regression tests in parallel.
|
|
35
35
|
- Review the generated Markdown files and convert them into YAML test scripts for reuse.
|
|
36
36
|
|
|
37
37
|
## Gotchas
|
package/docs/interactive/run.mdx
CHANGED
|
@@ -1,39 +1,39 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "/run"
|
|
3
3
|
sidebarTitle: "/run"
|
|
4
|
-
description: "Run a test
|
|
4
|
+
description: "Run a test file from a file."
|
|
5
5
|
icon: "play"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
## Description
|
|
9
|
-
The `/run` command is used to execute a test
|
|
9
|
+
The `/run` command is used to execute a test file from a specified file. This command performs each step defined in the test file and outputs the results.
|
|
10
10
|
|
|
11
11
|
## Usage
|
|
12
12
|
```bash
|
|
13
13
|
/run <file>
|
|
14
14
|
```
|
|
15
15
|
|
|
16
|
-
## Example
|
|
16
|
+
## Example usage
|
|
17
17
|
```bash
|
|
18
18
|
testdriverai
|
|
19
|
-
> /run helloworld.
|
|
19
|
+
> /run helloworld.yaml
|
|
20
20
|
```
|
|
21
|
-
This command runs the `helloworld.
|
|
21
|
+
This command runs the `helloworld.yaml` test file, executing each command in the file sequentially.
|
|
22
22
|
|
|
23
23
|
## Behavior
|
|
24
|
-
- TestDriver will execute the test
|
|
24
|
+
- TestDriver will execute the test file, performing each command as defined in the file.
|
|
25
25
|
- If the test completes successfully, the program will exit with code `0`.
|
|
26
26
|
- If any failures occur during the test, the program will output the errors and exit with code `1`.
|
|
27
27
|
|
|
28
28
|
## Protips
|
|
29
29
|
- Ensure the test file path is correct and accessible before running the command.
|
|
30
|
-
- Use descriptive filenames for your test
|
|
30
|
+
- Use descriptive filenames for your test files to make them easier to identify.
|
|
31
31
|
- Combine `/run` with debugging tools to troubleshoot failing tests.
|
|
32
32
|
|
|
33
33
|
## Gotchas
|
|
34
34
|
- This command will exit the program upon execution, so ensure all necessary setup is complete before running it.
|
|
35
|
-
- Any errors in the test
|
|
35
|
+
- Any errors in the test file (for example, invalid commands or missing arguments) will cause the test to fail.
|
|
36
36
|
|
|
37
37
|
## Notes
|
|
38
|
-
- The `/run` command is ideal for executing pre-created test
|
|
39
|
-
- Use this command to validate and debug your test
|
|
38
|
+
- The `/run` command is ideal for executing pre-created test files in an interactive session.
|
|
39
|
+
- Use this command to validate and debug your test files during development.
|
|
@@ -13,7 +13,7 @@ The `/save` command saves the current state of the test script to a file. This c
|
|
|
13
13
|
/save
|
|
14
14
|
```
|
|
15
15
|
|
|
16
|
-
## Example
|
|
16
|
+
## Example usage
|
|
17
17
|
```bash
|
|
18
18
|
testdriverai
|
|
19
19
|
> /save
|
|
@@ -41,7 +41,7 @@ testdriverai
|
|
|
41
41
|
|
|
42
42
|
## Behavior
|
|
43
43
|
- The `/save` command generates a YAML file with the current test script, including all executed steps and commands.
|
|
44
|
-
- The file can be used as a reusable test
|
|
44
|
+
- The file can be used as a reusable test file for future executions.
|
|
45
45
|
|
|
46
46
|
## Protips
|
|
47
47
|
- Use `/save` frequently during interactive sessions to preserve your progress and avoid losing work.
|
|
@@ -49,7 +49,7 @@ testdriverai
|
|
|
49
49
|
|
|
50
50
|
## Gotchas
|
|
51
51
|
- Ensure you have write permissions in the directory where the file will be saved.
|
|
52
|
-
- The saved script reflects the current state of the session. Any unexecuted commands
|
|
52
|
+
- The saved script reflects the current state of the session. Any unexecuted commands won't be included.
|
|
53
53
|
|
|
54
54
|
## Notes
|
|
55
55
|
- The `/save` command is ideal for creating reusable test scripts from interactive sessions.
|
|
@@ -17,7 +17,7 @@ The `/undo` command removes the last generated command or step from the current
|
|
|
17
17
|
- The `/undo` command removes the most recently added command or step from the test script.
|
|
18
18
|
- You can call `/undo` multiple times to remove multiple commands or steps.
|
|
19
19
|
|
|
20
|
-
## Example
|
|
20
|
+
## Example usage
|
|
21
21
|
|
|
22
22
|
### Before `/undo`
|
|
23
23
|
```yaml
|
|
@@ -5,7 +5,7 @@ description: "TestDriver vs Playwright vs Selenium"
|
|
|
5
5
|
icon: boxing-glove
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
### Application
|
|
8
|
+
### Application support
|
|
9
9
|
|
|
10
10
|
TestDriver operates a full desktop environment, so it can run any application.
|
|
11
11
|
|
|
@@ -16,7 +16,7 @@ TestDriver operates a full desktop environment, so it can run any application.
|
|
|
16
16
|
| Chrome Extensions | ✅ | | |
|
|
17
17
|
|
|
18
18
|
|
|
19
|
-
### Testing
|
|
19
|
+
### Testing features
|
|
20
20
|
|
|
21
21
|
TestDriver is AI first.
|
|
22
22
|
|
|
@@ -31,7 +31,7 @@ TestDriver is AI first.
|
|
|
31
31
|
| Team Dashboard | ✅ | | |
|
|
32
32
|
| Team Collaboration | ✅ | | |
|
|
33
33
|
|
|
34
|
-
### Test
|
|
34
|
+
### Test coverage
|
|
35
35
|
|
|
36
36
|
TestDriver has more coverage than selector-based frameworks.
|
|
37
37
|
|
|
@@ -49,7 +49,7 @@ TestDriver has more coverage than selector-based frameworks.
|
|
|
49
49
|
| `<canvas>` | ✅ | | |
|
|
50
50
|
| `<video>` | ✅ | | |
|
|
51
51
|
|
|
52
|
-
### Debugging
|
|
52
|
+
### Debugging features
|
|
53
53
|
|
|
54
54
|
Debugging features are powered by [Dashcam.io](https://dashcam.io).
|
|
55
55
|
|
|
@@ -63,7 +63,7 @@ Debugging features are powered by [Dashcam.io](https://dashcam.io).
|
|
|
63
63
|
| Team Dashboard | ✅ | | |
|
|
64
64
|
| Team Collaboration | ✅ | | |
|
|
65
65
|
|
|
66
|
-
### Web
|
|
66
|
+
### Web browser support
|
|
67
67
|
|
|
68
68
|
TestDriver is browser agnostic and supports any version of any browser.
|
|
69
69
|
|
|
@@ -77,7 +77,7 @@ TestDriver is browser agnostic and supports any version of any browser.
|
|
|
77
77
|
| Opera | ✅ | | ✅ |
|
|
78
78
|
| Safari | ✅ | | ✅ |
|
|
79
79
|
|
|
80
|
-
### Operating
|
|
80
|
+
### Operating system support
|
|
81
81
|
|
|
82
82
|
TestDriver currently supports Mac and Windows!
|
|
83
83
|
|
package/docs/overview/faq.mdx
CHANGED
|
@@ -6,7 +6,7 @@ icon: "block-question"
|
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
|
|
9
|
-
## What
|
|
9
|
+
## What's TestDriver?
|
|
10
10
|
TestDriver is an AI-powered testing platform that simulates user interactions to automate end-to-end testing for web, desktop, and mobile applications.
|
|
11
11
|
|
|
12
12
|
## How does TestDriver work?
|
|
@@ -67,7 +67,7 @@ Yes. TestDriver supports parallel execution using multiple VMs or containers.
|
|
|
67
67
|
Yes. It can log CPU, memory, load times, and frame rates to help catch performance regressions.
|
|
68
68
|
|
|
69
69
|
## Can it validate non-deterministic output?
|
|
70
|
-
Yes. It uses AI assertions to verify outcomes even when outputs vary (
|
|
70
|
+
Yes. It uses AI assertions to verify outcomes even when outputs vary (for example, generated text or dynamic UIs).
|
|
71
71
|
|
|
72
72
|
## Can it test workflows with variable inputs?
|
|
73
73
|
Yes. It supports data-driven tests using parameterized inputs.
|
|
@@ -5,7 +5,9 @@ description: "Learn how to upgrade TestDriver for the latest features and improv
|
|
|
5
5
|
icon: "circle-up"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
|
|
8
|
+
import YmlWarning from '/snippets/yml-warning.mdx'
|
|
9
|
+
|
|
10
|
+
## How to upgrade TestDriver
|
|
9
11
|
To upgrade TestDriver, follow these steps:
|
|
10
12
|
1. **Check for Updates**: Regularly check the [TestDriver NPM Package](https://www.npmjs.com/package/testdriverai)
|
|
11
13
|
2. **Update Command**: Use the following command to update TestDriver to the latest version:
|
|
@@ -17,10 +19,12 @@ To upgrade TestDriver, follow these steps:
|
|
|
17
19
|
npm install -g testdriverai@beta
|
|
18
20
|
```
|
|
19
21
|
|
|
20
|
-
|
|
22
|
+
<YmlWarning/>
|
|
23
|
+
|
|
24
|
+
In your existing GitHub Actions setup, you may need to update the `version` field in the `workflows/testdriver.yaml` file(s) to match the version of TestDriver you want to use. Leaving it blank will run `latest` by default. Here is an example of what to replace:
|
|
21
25
|
|
|
22
26
|
```yaml {16}
|
|
23
|
-
name: TestDriver
|
|
27
|
+
name: TestDriver
|
|
24
28
|
|
|
25
29
|
on:
|
|
26
30
|
push:
|
|
@@ -56,7 +60,7 @@ jobs:
|
|
|
56
60
|
Would become:
|
|
57
61
|
|
|
58
62
|
```yaml {16}
|
|
59
|
-
name: TestDriver
|
|
63
|
+
name: TestDriver
|
|
60
64
|
|
|
61
65
|
on:
|
|
62
66
|
push:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "What
|
|
3
|
-
sidebarTitle: "What
|
|
2
|
+
title: "What's TestDriver?"
|
|
3
|
+
sidebarTitle: "What's TestDriver?"
|
|
4
4
|
description: "TestDriver is a computer-use agent for QA testing of user interfaces."
|
|
5
5
|
icon: "circle-info"
|
|
6
6
|
---
|
|
@@ -21,9 +21,9 @@ TestDriver uses AI vision and keyboard and mouse control to automate end-to-end
|
|
|
21
21
|
|
|
22
22
|
TestDriver is different from other computer-use agents in that it produces a `YAML` test script that increases the speed and repeatability of testing.
|
|
23
23
|
|
|
24
|
-
### Selectorless
|
|
24
|
+
### Selectorless testing
|
|
25
25
|
|
|
26
|
-
Unlike traditional frameworks (
|
|
26
|
+
Unlike traditional frameworks (for example, Selenium, Playwright), TestDriver doesn't rely on CSS selectors or static analysis. Instead, tests are described in plain English, such as:
|
|
27
27
|
|
|
28
28
|
```
|
|
29
29
|
> Open Google Chrome and search for "testdriver"
|
|
@@ -60,7 +60,7 @@ Old and Busted (Selectors) | New Hotness (TestDriver)
|
|
|
60
60
|
|:----------------------|:-------------------|
|
|
61
61
|
```div[class="product-card"] >> text="Add to Cart" >> nth=2``` | buy the 2nd product
|
|
62
62
|
|
|
63
|
-
These high level instructions are easier to create and maintain because they
|
|
63
|
+
These high level instructions are easier to create and maintain because they're loosely coupled from the codebase. We're describing a high level goal, not a low level interaction.
|
|
64
64
|
|
|
65
65
|
The tests will still continue to work even when the junior developer changes .product-card to .product.card or the designers change Add to Cart to Buy Now . The concepts remain the same so the AI will adapt.
|
|
66
66
|
|
|
@@ -68,7 +68,7 @@ The tests will still continue to work even when the junior developer changes .pr
|
|
|
68
68
|
|
|
69
69
|
TestDriver uses a combination of reinforcement learning and computer vision. The context from successful text executions inform future executions. Here's an example of the context our model considers when locating a text match:
|
|
70
70
|
|
|
71
|
-
Context| What
|
|
71
|
+
Context| What's it? | Touchpoint
|
|
72
72
|
|:----------|:------------------|:----------------|
|
|
73
73
|
Prompt | Desired outcome | User Input
|
|
74
74
|
Screenshot | Image of computer desktop | Runtime
|
package/docs/quickstart.mdx
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "AI
|
|
3
|
-
sidebarTitle: "AI
|
|
2
|
+
title: "AI Chatbots"
|
|
3
|
+
sidebarTitle: "AI Chatbots"
|
|
4
4
|
description: "Integrate AI chatbots into your testing workflow."
|
|
5
5
|
icon: "message-bot"
|
|
6
6
|
---
|
|
@@ -18,7 +18,7 @@ This scenario is an example of how to set up and run tests for an AI chatbot usi
|
|
|
18
18
|
|
|
19
19
|
<TestPrereqs />
|
|
20
20
|
|
|
21
|
-
## Scenario
|
|
21
|
+
## Scenario overview
|
|
22
22
|
1. Load a website with an AI chatbot.
|
|
23
23
|
2. Type a chat message or ask a question.
|
|
24
24
|
3. Wait for the chatbot to respond.
|
|
@@ -11,14 +11,14 @@ With TestDriver, you can automate the testing of cookie banners on your web appl
|
|
|
11
11
|
|
|
12
12
|
<TestPrereqs />
|
|
13
13
|
|
|
14
|
-
## Scenario
|
|
14
|
+
## Scenario overview
|
|
15
15
|
1. **Visit the Site**: The test will navigate to the target URL.
|
|
16
16
|
2. **Check for Cookie Banner**: It will verify if the cookie banner is displayed.
|
|
17
17
|
3. **Decline Cookies**: The test will simulate a user clicking the "Decline All" button on the cookie banner.
|
|
18
18
|
|
|
19
19
|
|
|
20
20
|
|
|
21
|
-
## TestDriver in
|
|
21
|
+
## TestDriver in action
|
|
22
22
|
<iframe
|
|
23
23
|
width="560"
|
|
24
24
|
height="315"
|
|
@@ -11,14 +11,14 @@ This scenario demonstrates how to automate testing the file upload functionality
|
|
|
11
11
|
|
|
12
12
|
<TestPrereqs />
|
|
13
13
|
|
|
14
|
-
## Scenario
|
|
14
|
+
## Scenario overview
|
|
15
15
|
1. Load a website with a file upload feature.
|
|
16
16
|
2. Generate a file that will be used for the upload.
|
|
17
17
|
3. Use TestDriver to perform the file upload process.
|
|
18
18
|
4. Check if the uploaded file is processed correctly by the application.
|
|
19
19
|
|
|
20
20
|
|
|
21
|
-
## TestDriver in
|
|
21
|
+
## TestDriver in action
|
|
22
22
|
<iframe
|
|
23
23
|
width="560"
|
|
24
24
|
height="315"
|
|
@@ -9,14 +9,14 @@ import TestPrereqs from '/snippets/test-prereqs.mdx'
|
|
|
9
9
|
|
|
10
10
|
<TestPrereqs />
|
|
11
11
|
|
|
12
|
-
## Scenario
|
|
12
|
+
## Scenario overview
|
|
13
13
|
1. Open a webpage containing a form (Log in, Registration, etc.).
|
|
14
14
|
2. Use TestDriver to fill in the form fields with test data.
|
|
15
|
-
3. Submit the form and verify the expected outcome (
|
|
15
|
+
3. Submit the form and verify the expected outcome (for example, successful login, registration confirmation, redirect etc.).
|
|
16
16
|
|
|
17
17
|
|
|
18
18
|
|
|
19
|
-
## TestDriver in
|
|
19
|
+
## TestDriver in action
|
|
20
20
|
<iframe
|
|
21
21
|
width="560"
|
|
22
22
|
height="315"
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "Login
|
|
2
|
+
title: "Login"
|
|
3
3
|
sidebarTitle: "Login"
|
|
4
4
|
description: "Test login functionality with TestDriver"
|
|
5
5
|
icon: "lock"
|
|
@@ -11,7 +11,7 @@ Test login functionality with TestDriver. This scenario demonstrates how to auto
|
|
|
11
11
|
|
|
12
12
|
<TestPrereqs />
|
|
13
13
|
|
|
14
|
-
## Scenario
|
|
14
|
+
## Scenario overview
|
|
15
15
|
1. Visit the login page of the web application.
|
|
16
16
|
2. Enter the username and password into the respective fields (see [Reusable Snippets](/features/reusable-snippets#How-to Create-and-Use-Reusable-Snippets)).
|
|
17
17
|
3. Click the "Login" button.
|
|
@@ -20,7 +20,7 @@ Test login functionality with TestDriver. This scenario demonstrates how to auto
|
|
|
20
20
|
|
|
21
21
|
|
|
22
22
|
### Setup your test environment with predefined credentials.
|
|
23
|
-
|
|
23
|
+
This example uses GitHub secret management to store credentials.
|
|
24
24
|
|
|
25
25
|
<Tip>You can also use the [`exec`](commands/exec) command to fetch a test user dynamically from a database or API.</Tip>
|
|
26
26
|
|
|
@@ -11,11 +11,11 @@ Test PDF generation functionality with TestDriver. This scenario demonstrates ho
|
|
|
11
11
|
|
|
12
12
|
<TestPrereqs />
|
|
13
13
|
|
|
14
|
-
## Scenario
|
|
14
|
+
## Scenario overview
|
|
15
15
|
1. Open a webpage or desktop app that will allow you to generate a PDF.
|
|
16
|
-
2. Trigger the PDF generation process (
|
|
16
|
+
2. Trigger the PDF generation process (for example, by clicking a button).
|
|
17
17
|
3. Verify that the PDF is generated successfully and contains the expected content.
|
|
18
18
|
4. Optionally, check the file size and format of the generated PDF.
|
|
19
19
|
|
|
20
|
-
## What
|
|
20
|
+
## What's next
|
|
21
21
|
From here you can automate and add coverage to your PDF generation process, removing manual testing from the equation.
|
|
@@ -11,10 +11,10 @@ This scenario demonstrates how to automate testing the spell check functionality
|
|
|
11
11
|
|
|
12
12
|
<TestPrereqs />
|
|
13
13
|
|
|
14
|
-
## Scenario
|
|
14
|
+
## Scenario overview
|
|
15
15
|
1. Open up a webpage that has text input and a spell check function.
|
|
16
16
|
2. Enter a misspelled word into the text input field.
|
|
17
|
-
3. Trigger the spell check function (
|
|
17
|
+
3. Trigger the spell check function (for example, by clicking a button or losing focus on the input field).
|
|
18
18
|
4. Verify that the spell checker identifies the misspelled word and provides suggestions for correction.
|
|
19
19
|
5. Optionally, select a suggestion and apply it to the text input field.
|
|
20
20
|
|
package/docs/security/action.mdx
CHANGED
|
@@ -7,12 +7,12 @@ icon: shield-check
|
|
|
7
7
|
|
|
8
8
|
import GitignoreWarning from '/snippets/gitignore-warning.mdx';
|
|
9
9
|
|
|
10
|
-
## Open
|
|
10
|
+
## Open source
|
|
11
11
|
The TestDriver Action is open source, and its source code is available for review. You can find the repository here:
|
|
12
12
|
|
|
13
13
|
<Card horizontal="true" icon="github"> [GitHub - testdriverai/action](https://github.com/testdriverai/action) </Card>
|
|
14
14
|
|
|
15
|
-
## Ephemeral
|
|
15
|
+
## Ephemeral virtual machine runners
|
|
16
16
|
TestDriver tests are executed on private virtual machines (VMs) managed by Amazon EC2. These VMs are ephemeral, meaning they only exist for the duration of the test execution. Once the test is complete:
|
|
17
17
|
- The VM is destroyed.
|
|
18
18
|
- The hard disk is wiped to ensure no residual data remains.
|
|
@@ -20,15 +20,15 @@ TestDriver tests are executed on private virtual machines (VMs) managed by Amazo
|
|
|
20
20
|
## Secrets
|
|
21
21
|
To securely manage private information, we recommend storing sensitive data as secrets in your GitHub repository. Learn more about [storing secrets in GitHub](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
|
|
22
22
|
|
|
23
|
-
### Handling
|
|
23
|
+
### Handling secrets
|
|
24
24
|
- **Prerun Scripts**: Any secrets supplied within prerun scripts or prompts are transmitted over SSL to the TestDriver API. Prerun scripts are **not persisted**.
|
|
25
25
|
- **Agent Prompts**: Secrets supplied to agent prompts are persisted (see [Agent Security](/security/agent)).
|
|
26
26
|
- **Secure Workflows**: If your workflow requires secret sharing and you encounter issues, please contact us for assistance.
|
|
27
27
|
|
|
28
|
-
### Common
|
|
28
|
+
### Common use case
|
|
29
29
|
A common workflow involves using prerun scripts to securely access a private staging website via basic authentication. This allows you to log into staging environments without persisting sensitive data on TestDriver servers.
|
|
30
30
|
|
|
31
|
-
## Environment-
|
|
31
|
+
## Environment-specific security
|
|
32
32
|
|
|
33
33
|
<GitignoreeWarning/>
|
|
34
34
|
|
|
@@ -46,7 +46,7 @@ Testing staging environments may require secure information, such as credentials
|
|
|
46
46
|
### Development
|
|
47
47
|
TestDriver can clone feature branches and build code on its virtual machines using workflows similar to GitHub Actions.
|
|
48
48
|
|
|
49
|
-
#### GitHub
|
|
49
|
+
#### GitHub token for development
|
|
50
50
|
To test development branches of private codebases, you must supply a GitHub personal access token within the GitHub Action. This token is used to:
|
|
51
51
|
- Clone the codebase onto the VM.
|
|
52
52
|
- Build and test the code in an isolated environment.
|
package/docs/security/agent.mdx
CHANGED
|
@@ -13,7 +13,7 @@ The TestDriver Agent doesn't contain any AI models within it. Instead, it upload
|
|
|
13
13
|
|
|
14
14
|
The TestDriver API leverages OpenAI models behind the scenes. For more information about OpenAI and its privacy practices, visit their [privacy center](https://openai.com/privacy).
|
|
15
15
|
|
|
16
|
-
## Desktop
|
|
16
|
+
## Desktop context collected
|
|
17
17
|
During execution, the TestDriver Agent uploads the following information to the API:
|
|
18
18
|
|
|
19
19
|
- **User Input Prompts**: The commands and prompts you provide to TestDriver.
|
|
@@ -22,30 +22,30 @@ During execution, the TestDriver Agent uploads the following information to the
|
|
|
22
22
|
- **Mouse Position**: The current position of the mouse pointer.
|
|
23
23
|
- **Desktop Screenshots**: Screenshots of the primary display.
|
|
24
24
|
|
|
25
|
-
### Data
|
|
25
|
+
### Data persistence
|
|
26
26
|
- **Desktop Context**: All collected desktop context, except for screenshots, is persisted in the TestDriver database.
|
|
27
27
|
- **Desktop Screenshots**: Screenshots are uploaded to the server for processing but aren't stored in the database.
|
|
28
28
|
|
|
29
|
-
### Desktop
|
|
29
|
+
### Desktop screenshots
|
|
30
30
|
TestDriver frequently captures screenshots of the desktop to provide the AI with decision-making context. These screenshots are uploaded to the API but aren't persisted in the database.
|
|
31
31
|
|
|
32
32
|
- **Primary Display Only**: Screenshots are limited to the primary display.
|
|
33
|
-
- **Privacy Recommendation**: For complete privacy, it
|
|
33
|
+
- **Privacy Recommendation**: For complete privacy, it's recommended to run TestDriver within a virtual machine on your desktop.
|
|
34
34
|
|
|
35
35
|
**Important**: TestDriver can't operate without visual context. don't install TestDriver if you are uncomfortable with capturing images of the desktop.
|
|
36
36
|
|
|
37
|
-
## Collected
|
|
37
|
+
## Collected data details
|
|
38
38
|
|
|
39
|
-
### Active
|
|
39
|
+
### Active window
|
|
40
40
|
Information about the open windows on the desktop is reported using the `active-window` module. This includes details such as the application name and window titles.
|
|
41
41
|
|
|
42
|
-
### System
|
|
42
|
+
### System information
|
|
43
43
|
System details, such as operating system, CPU, and memory, are reported using the `systeminformation` module.
|
|
44
44
|
|
|
45
|
-
### User
|
|
45
|
+
### User prompts
|
|
46
46
|
The prompts you input into TestDriver are uploaded to the API and stored in a database. This data is used to provide the AI with a history of context for better decision-making.
|
|
47
47
|
|
|
48
|
-
## Additional
|
|
48
|
+
## Additional analytics
|
|
49
49
|
When running `testdriver init`, you will be asked if you would like to share additional analytics. Sharing usage analytics is **opt-in**, and no extra data will be collected unless explicitly enabled.
|
|
50
50
|
|
|
51
51
|
To disable additional analytics, set the `TD_ANALYTICS` environment variable to `false`:
|
|
@@ -53,7 +53,7 @@ To disable additional analytics, set the `TD_ANALYTICS` environment variable to
|
|
|
53
53
|
TD_ANALYTICS=false
|
|
54
54
|
```
|
|
55
55
|
|
|
56
|
-
## Rate
|
|
56
|
+
## Rate limiting and restrictions
|
|
57
57
|
While the TestDriver Agent is free to use, TestDriver reserves the right to rate limit or restrict usage by IP address for any reason.
|
|
58
58
|
|
|
59
59
|
## Notes
|
|
@@ -12,19 +12,19 @@ The TestDriver web dashboard provides a secure interface for managing and review
|
|
|
12
12
|
|
|
13
13
|
Dashcam and TestDriver share the same API and web application back end, which includes robust privacy and security features.
|
|
14
14
|
|
|
15
|
-
## Security
|
|
15
|
+
## Security features
|
|
16
16
|
|
|
17
17
|
### SSL
|
|
18
18
|
- All data transmitted between your browser and the TestDriver web application is encrypted using HTTPS.
|
|
19
19
|
|
|
20
|
-
###
|
|
20
|
+
### Authentication
|
|
21
21
|
- Users can only authenticate via OAuth, provided by Auth0, ensuring secure and reliable user authentication.
|
|
22
22
|
|
|
23
|
-
### Team
|
|
23
|
+
### Team management
|
|
24
24
|
- Administrators can add or remove individual team members.
|
|
25
25
|
- Only administrators have the ability to manage team settings.
|
|
26
26
|
|
|
27
|
-
### Role-
|
|
27
|
+
### Role-based access control (RBAC)
|
|
28
28
|
- The first user to create a team is designated as the administrator.
|
|
29
29
|
- Administrators:
|
|
30
30
|
- Are the only users who can view the API key.
|
|
@@ -32,9 +32,9 @@ Dashcam and TestDriver share the same API and web application back end, which in
|
|
|
32
32
|
- can't be removed from the team.
|
|
33
33
|
- All other users are normal members with limited access.
|
|
34
34
|
|
|
35
|
-
### API
|
|
35
|
+
### API key rotation
|
|
36
36
|
- Teams can rotate their API key for enhanced security.
|
|
37
|
-
- It
|
|
37
|
+
- It's recommended to rotate the API key every 90 days to minimize risk.
|
|
38
38
|
|
|
39
39
|
<Check>
|
|
40
40
|
For more details on **Team Management** see the [Team documentation](/account/team).
|
|
@@ -42,7 +42,7 @@ Dashcam and TestDriver share the same API and web application back end, which in
|
|
|
42
42
|
|
|
43
43
|
<GitignoreWarning/>
|
|
44
44
|
|
|
45
|
-
### Secret
|
|
45
|
+
### Secret masking
|
|
46
46
|
- Test replay logs and network requests are automatically scanned for sensitive information, such as:
|
|
47
47
|
- Credit card numbers
|
|
48
48
|
- Emails
|
|
@@ -50,7 +50,7 @@ Dashcam and TestDriver share the same API and web application back end, which in
|
|
|
50
50
|
- API keys
|
|
51
51
|
- Detected secrets are masked with asterisks (`****`) to prevent exposure.
|
|
52
52
|
|
|
53
|
-
### Encrypted
|
|
53
|
+
### Encrypted at rest
|
|
54
54
|
- Test replays and logs are securely stored on Amazon S3 and encrypted at rest.
|
|
55
55
|
- Test results are only accessible via temporary signed URLs.
|
|
56
56
|
- Signed URLs are generated exclusively for team users and expire after a set duration.
|
|
@@ -59,3 +59,5 @@ Dashcam and TestDriver share the same API and web application back end, which in
|
|
|
59
59
|
- The TestDriver web dashboard is designed with privacy and security as top priorities.
|
|
60
60
|
- For additional security, ensure your team rotates API keys regularly and reviews team member access permissions.
|
|
61
61
|
- If you have specific security concerns or questions, please contact TestDriver support.
|
|
62
|
+
|
|
63
|
+
<Tip>See complete docs about the dashboard features [here](/account/dashboard)</Tip>
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
## Prerequisites
|
|
2
2
|
Before running the tests, ensure you have performed the following steps:
|
|
3
3
|
|
|
4
|
-
1. Install the [VS Code
|
|
4
|
+
1. Install the [VS Code extension](/getting-started/vscode) for TestDriver
|
|
5
5
|
2. If you haven't already, signup for a Free Trial on the [TestDriver website](https://testdriver.ai/pricing)
|
|
6
6
|
3. Install the [TestDriver CLI](/cli/overview) globally using npm or just follow the VS Code Extension Setup Walkthrough:
|
|
7
7
|
```bash
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
<Warning>TestDriver v5 uses the `.yaml` file extension for all test files. This is a breaking change from v4 which uses the `.yml` extension!</Warning>
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
|
-
title: "Tutorial: Advanced
|
|
2
|
+
title: "Tutorial: Advanced test"
|
|
3
3
|
description: "Explore advanced testing techniques and features in TestDriver."
|
|
4
|
-
sidebarTitle: "Advanced
|
|
4
|
+
sidebarTitle: "Advanced test"
|
|
5
5
|
icon: "flask-round"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# Advanced
|
|
8
|
+
# Advanced test tutorial
|
|
9
9
|
|
|
10
10
|
Welcome to the Advanced Test Tutorial. In this guide, we will explore advanced testing techniques and features available in TestDriver. This tutorial assumes you have a basic understanding of TestDriver and its functionalities.
|
|
11
11
|
|
|
@@ -16,7 +16,7 @@ Before starting, ensure you have completed the basic TestDriver tutorial and hav
|
|
|
16
16
|
- TestDriver installed and configured.
|
|
17
17
|
- A sample project to test.
|
|
18
18
|
|
|
19
|
-
## Advanced
|
|
19
|
+
## Advanced features overview
|
|
20
20
|
|
|
21
21
|
TestDriver provides several advanced features to enhance your testing capabilities:
|
|
22
22
|
|
|
@@ -25,9 +25,9 @@ TestDriver provides several advanced features to enhance your testing capabiliti
|
|
|
25
25
|
3. **Test Hooks**: Use hooks to execute code before or after tests.
|
|
26
26
|
4. **Parallel Execution**: Speed up testing by running tests in parallel.
|
|
27
27
|
|
|
28
|
-
## Step-by-
|
|
28
|
+
## Step-by-step guide
|
|
29
29
|
|
|
30
|
-
### Step 1: Setting
|
|
30
|
+
### Step 1: Setting up custom assertions
|
|
31
31
|
|
|
32
32
|
Custom assertions allow you to define specific conditions for your tests. Here's an example:
|
|
33
33
|
|
|
@@ -36,7 +36,7 @@ def assert_custom_condition(value):
|
|
|
36
36
|
assert value > 0, "Value must be greater than 0"
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
-
### Step 2: Using
|
|
39
|
+
### Step 2: Using parameterized tests
|
|
40
40
|
|
|
41
41
|
Parameterized tests enable you to test multiple scenarios with different inputs. Example:
|
|
42
42
|
|
|
@@ -48,7 +48,7 @@ def test_is_positive(input, expected):
|
|
|
48
48
|
assert (input > 0) == expected
|
|
49
49
|
```
|
|
50
50
|
|
|
51
|
-
### Step 3: Implementing
|
|
51
|
+
### Step 3: Implementing test hooks
|
|
52
52
|
|
|
53
53
|
Test hooks let you execute code before or after tests. Example:
|
|
54
54
|
|
|
@@ -60,7 +60,7 @@ def teardown_function():
|
|
|
60
60
|
print("Cleaning up test environment")
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
-
### Step 4: Running
|
|
63
|
+
### Step 4: Running tests in parallel
|
|
64
64
|
|
|
65
65
|
Parallel execution can be achieved using pytest-xdist. Install it with:
|
|
66
66
|
|