testdriverai 5.6.3 → 5.6.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agent.js +1 -1
- package/docs/account/dashboard.mdx +3 -6
- package/docs/account/enterprise.mdx +16 -15
- package/docs/account/pricing.mdx +4 -4
- package/docs/account/projects.mdx +2 -2
- package/docs/account/team.mdx +3 -3
- package/docs/action/browser.mdx +9 -9
- package/docs/action/os.mdx +12 -12
- package/docs/action/output.mdx +10 -10
- package/docs/action/performance.mdx +11 -11
- package/docs/{guide/lifecycle.mdx → action/pre-post-scripts.mdx} +87 -69
- package/docs/action/secrets.mdx +11 -11
- package/docs/action/setup.mdx +8 -8
- package/docs/apps/chrome-extensions.mdx +9 -9
- package/docs/apps/desktop-apps.mdx +6 -5
- package/docs/apps/mobile-apps.mdx +3 -3
- package/docs/apps/static-websites.mdx +2 -3
- package/docs/bugs/jira.mdx +16 -16
- package/docs/cli/overview.mdx +8 -8
- package/docs/commands/exec.mdx +152 -10
- package/docs/commands/focus-application.mdx +1 -1
- package/docs/commands/hover-image.mdx +1 -1
- package/docs/commands/hover-text.mdx +2 -2
- package/docs/commands/if.mdx +1 -1
- package/docs/commands/issues.mdx +8 -6
- package/docs/commands/match-image.mdx +2 -2
- package/docs/commands/press-keys.mdx +19 -2
- package/docs/commands/run.mdx +1 -1
- package/docs/commands/scroll-until-image.mdx +1 -1
- package/docs/commands/scroll-until-text.mdx +2 -2
- package/docs/commands/scroll.mdx +2 -2
- package/docs/commands/type.mdx +1 -1
- package/docs/commands/wait-for-image.mdx +1 -1
- package/docs/commands/wait-for-text.mdx +1 -1
- package/docs/commands/wait.mdx +1 -1
- package/docs/docs.json +9 -9
- package/docs/exporting/playwright.mdx +16 -16
- package/docs/features/auto-healing.mdx +5 -5
- package/docs/features/cross-platform.mdx +4 -4
- package/docs/features/generation.mdx +7 -4
- package/docs/features/github.mdx +21 -21
- package/docs/features/parallel-testing.mdx +9 -9
- package/docs/features/reusable-snippets.mdx +7 -7
- package/docs/features/selectorless.mdx +4 -4
- package/docs/features/visual-assertions.mdx +26 -23
- package/docs/getting-started/ci.mdx +19 -19
- package/docs/getting-started/editing.mdx +12 -8
- package/docs/getting-started/generating.mdx +4 -5
- package/docs/getting-started/running.mdx +3 -3
- package/docs/getting-started/setup.mdx +9 -9
- package/docs/getting-started/vscode.mdx +63 -85
- package/docs/getting-started/writing.mdx +11 -7
- package/docs/guide/assertions.mdx +21 -21
- package/docs/guide/authentication.mdx +11 -11
- package/docs/guide/code.mdx +6 -6
- package/docs/guide/environment-variables.mdx +9 -9
- package/docs/guide/lifecycle-prerun.mdx +191 -0
- package/docs/guide/locating.mdx +15 -18
- package/docs/guide/variables.mdx +8 -8
- package/docs/guide/waiting.mdx +10 -10
- package/docs/importing/csv.mdx +23 -24
- package/docs/importing/gherkin.mdx +20 -20
- package/docs/importing/jira.mdx +22 -23
- package/docs/importing/testrail.mdx +17 -17
- package/docs/integrations/electron.mdx +7 -7
- package/docs/integrations/netlify.mdx +7 -7
- package/docs/integrations/vercel.mdx +13 -13
- package/docs/interactive/assert.mdx +3 -3
- package/docs/interactive/dry.mdx +7 -7
- package/docs/interactive/explore.mdx +9 -9
- package/docs/interactive/generate.mdx +2 -2
- package/docs/interactive/run.mdx +10 -10
- package/docs/interactive/save.mdx +3 -3
- package/docs/interactive/undo.mdx +1 -1
- package/docs/overview/comparison.mdx +6 -6
- package/docs/overview/faq.mdx +2 -2
- package/docs/overview/quickstart.mdx +1 -1
- package/docs/overview/upgrading.mdx +8 -4
- package/docs/overview/what-is-testdriver.mdx +6 -6
- package/docs/quickstart.mdx +1 -1
- package/docs/scenarios/ai-chatbot.mdx +3 -3
- package/docs/scenarios/cookie-banner.mdx +2 -2
- package/docs/scenarios/file-upload.mdx +2 -2
- package/docs/scenarios/form-filling.mdx +3 -3
- package/docs/scenarios/log-in.mdx +3 -3
- package/docs/scenarios/pdf-generation.mdx +3 -3
- package/docs/scenarios/spell-check.mdx +2 -2
- package/docs/security/action.mdx +6 -6
- package/docs/security/agent.mdx +10 -10
- package/docs/security/platform.mdx +10 -8
- package/docs/snippets/test-prereqs.mdx +1 -1
- package/docs/snippets/yml-warning.mdx +1 -0
- package/docs/tutorials/advanced-test.mdx +9 -9
- package/docs/tutorials/basic-test.mdx +3 -3
- package/package.json +1 -1
- package/styles/config/vocabularies/Docs/accept.txt +32 -4
- package/docs/action/prerun.mdx +0 -137
|
@@ -5,9 +5,11 @@ description: "Simplify test creation with AI-driven exploratory testing."
|
|
|
5
5
|
icon: "wand-sparkles"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
+
import YmlWarning from '/snippets/yml-warning.mdx'
|
|
9
|
+
|
|
8
10
|
Writing tests can be a tedious and time-consuming task. TestDriver can generate tests just by exploring your app! This guide will show you how to generate tests using TestDriver.
|
|
9
11
|
|
|
10
|
-
# Generate
|
|
12
|
+
# Generate exploratory tests
|
|
11
13
|
|
|
12
14
|
Launch `testdriverai` in interactive mode with the following command:
|
|
13
15
|
|
|
@@ -61,7 +63,7 @@ steps:
|
|
|
61
63
|
- prompt: Assert that there are issues during checkout and an error is displayed.
|
|
62
64
|
```
|
|
63
65
|
|
|
64
|
-
# Generate
|
|
66
|
+
# Generate regression tests from exploratory tests
|
|
65
67
|
|
|
66
68
|
Now it's time to generate the regression test.
|
|
67
69
|
|
|
@@ -80,11 +82,12 @@ No commands found, running exploratory
|
|
|
80
82
|
|
|
81
83
|
When using the `--save` command, TestDriver appends any generated commands to the test file.
|
|
82
84
|
|
|
83
|
-
So as your test run, TestDriver will append successful commands to the
|
|
85
|
+
So as your test run, TestDriver will append successful commands to the YAML file and save it.
|
|
86
|
+
|
|
87
|
+
<YmlWarning/>
|
|
84
88
|
|
|
85
89
|
```yaml
|
|
86
90
|
version: 5.3.11
|
|
87
|
-
session: 6801430b5f2c252a8f4bed60
|
|
88
91
|
steps:
|
|
89
92
|
- prompt: Focus the browser window.
|
|
90
93
|
commands:
|
package/docs/features/github.mdx
CHANGED
|
@@ -12,18 +12,18 @@ TestDriver offers advanced GitHub integration features, including the ability to
|
|
|
12
12
|
|
|
13
13
|
---
|
|
14
14
|
|
|
15
|
-
## 1. **Creating
|
|
15
|
+
## 1. **Creating branches automatically**
|
|
16
16
|
|
|
17
17
|
TestDriver can create its own branches to store test results, updates, or auto-healed tests. This ensures that test-related changes are isolated from the main codebase and can be reviewed before merging.
|
|
18
18
|
|
|
19
|
-
### How
|
|
19
|
+
### How it works
|
|
20
20
|
|
|
21
|
-
- **Branch Creation**: TestDriver creates a new branch (
|
|
22
|
-
- **Pull Request**: A pull request is automatically opened from the new branch to the base branch (
|
|
21
|
+
- **Branch Creation**: TestDriver creates a new branch (for example, `test-results` or `auto-heal-updates`) during the workflow execution.
|
|
22
|
+
- **Pull Request**: A pull request is automatically opened from the new branch to the base branch (for example, `main`), allowing developers to review the changes.
|
|
23
23
|
|
|
24
|
-
### Example
|
|
24
|
+
### Example workflow with branch creation
|
|
25
25
|
|
|
26
|
-
```yaml {21-24}
|
|
26
|
+
```yaml {21-24} .github/workflows/testdriver.yaml
|
|
27
27
|
name: TestDriver Auto-Healing
|
|
28
28
|
|
|
29
29
|
on:
|
|
@@ -47,34 +47,34 @@ jobs:
|
|
|
47
47
|
create-pr: true
|
|
48
48
|
pr-title: "TestDriver Test Results"
|
|
49
49
|
pr-branch: test-results
|
|
50
|
-
pr-test-filename: testdriver-results.
|
|
50
|
+
pr-test-filename: testdriver-results.yaml
|
|
51
51
|
env:
|
|
52
52
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
53
53
|
FORCE_COLOR: "3"
|
|
54
54
|
|
|
55
55
|
```
|
|
56
56
|
|
|
57
|
-
### Key
|
|
57
|
+
### Key fields
|
|
58
58
|
- **`create-pr: true`**: Enables branch creation and pull request generation.
|
|
59
|
-
- **`pr-branch`**: Specifies the branch name (
|
|
59
|
+
- **`pr-branch`**: Specifies the branch name (for example, `test-results`).
|
|
60
60
|
- **`pr-title`**: Sets the title of the pull request.
|
|
61
61
|
- **`pr-test-filename`**: Specifies the filename for the test results.
|
|
62
62
|
|
|
63
63
|
---
|
|
64
64
|
|
|
65
|
-
## 2. **Manual
|
|
65
|
+
## 2. **Manual workflow dispatch**
|
|
66
66
|
|
|
67
67
|
GitHub Actions supports manual triggering of workflows using the **workflow_dispatch** event. This allows you to run TestDriver workflows on demand, making it ideal for exploratory testing or debugging.
|
|
68
68
|
|
|
69
|
-
### How to
|
|
69
|
+
### How to trigger workflows manually
|
|
70
70
|
1. Navigate to the **Actions** tab in your GitHub repository.
|
|
71
71
|
2. Select the workflow you want to run.
|
|
72
72
|
3. Click the **Run workflow** button.
|
|
73
73
|
4. Provide any required inputs (if applicable) and confirm.
|
|
74
74
|
|
|
75
|
-
### Example
|
|
75
|
+
### Example workflow with manual dispatch
|
|
76
76
|
|
|
77
|
-
```yaml
|
|
77
|
+
```yaml .github/workflows/testdriver.yaml
|
|
78
78
|
name: TestDriver Manual Dispatch
|
|
79
79
|
|
|
80
80
|
on:
|
|
@@ -103,14 +103,14 @@ jobs:
|
|
|
103
103
|
|
|
104
104
|
Run Steps
|
|
105
105
|
|
|
106
|
-
### Benefits of
|
|
106
|
+
### Benefits of manual dispatch
|
|
107
107
|
- **Exploratory Testing**: Run tests on demand for specific scenarios.
|
|
108
108
|
- **Debugging**: Trigger workflows to debug issues without waiting for automated triggers.
|
|
109
109
|
- **Flexibility**: Test changes in feature branches or experimental setups.
|
|
110
110
|
|
|
111
111
|
---
|
|
112
112
|
|
|
113
|
-
## 3. **GitHub Actions
|
|
113
|
+
## 3. **GitHub Actions output**
|
|
114
114
|
|
|
115
115
|
TestDriver provides detailed outputs during workflow execution, which are visible in the **Actions** tab of your repository. These outputs include:
|
|
116
116
|
- **Test Summary**: A high-level overview of the test results.
|
|
@@ -118,13 +118,13 @@ TestDriver provides detailed outputs during workflow execution, which are visibl
|
|
|
118
118
|
- **Screenshots and GIFs**: Visual feedback for debugging.
|
|
119
119
|
- **Links to TestDriver Dashcam.io Replays**: Direct links to detailed test recordings and results.
|
|
120
120
|
|
|
121
|
-
### Example
|
|
121
|
+
### Example outputs
|
|
122
122
|
|
|
123
|
-
#### **Test
|
|
123
|
+
#### **Test summary**
|
|
124
124
|
- ✅ All tests passed.
|
|
125
125
|
- ❌ 2 tests failed.
|
|
126
126
|
|
|
127
|
-
#### **TestDriver Replay
|
|
127
|
+
#### **TestDriver Replay**
|
|
128
128
|
|
|
129
129
|
```markdown
|
|
130
130
|
[View Full Test Results](https://app.testdriver.ai/...)
|
|
@@ -132,10 +132,10 @@ TestDriver provides detailed outputs during workflow execution, which are visibl
|
|
|
132
132
|
|
|
133
133
|
---
|
|
134
134
|
|
|
135
|
-
### Using Outputs in Workflows
|
|
135
|
+
### Using Outputs in Workflows
|
|
136
136
|
You can capture and use TestDriver outputs in subsequent steps of your workflow. For example, you can post the test summary as a comment on a pull request.
|
|
137
137
|
|
|
138
|
-
#### Example
|
|
138
|
+
#### Example
|
|
139
139
|
|
|
140
140
|
|
|
141
141
|
```yaml
|
|
@@ -157,7 +157,7 @@ You can capture and use TestDriver outputs in subsequent steps of your workflow.
|
|
|
157
157
|
|
|
158
158
|
---
|
|
159
159
|
|
|
160
|
-
## Summary of Features
|
|
160
|
+
## Summary of Features
|
|
161
161
|
1. **Branch Creation**: Automatically create branches for test results or auto-healed tests.
|
|
162
162
|
2. **Manual Dispatch**: Trigger workflows on demand from the GitHub Actions UI.
|
|
163
163
|
3. **Detailed Outputs**: Access logs, summaries, and visual feedback directly in the Actions tab.
|
|
@@ -9,7 +9,7 @@ Parallel testing allows you to run multiple tests simultaneously, significantly
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## Why
|
|
12
|
+
## Why use parallel testing?
|
|
13
13
|
|
|
14
14
|
1. **Faster Execution**: Run multiple tests at the same time to reduce overall test duration.
|
|
15
15
|
2. **Scalability**: Easily scale your testing efforts as your test suite grows.
|
|
@@ -18,12 +18,12 @@ Parallel testing allows you to run multiple tests simultaneously, significantly
|
|
|
18
18
|
|
|
19
19
|
---
|
|
20
20
|
|
|
21
|
-
## Setting Up
|
|
21
|
+
## Setting Up parallel testing with a matrix strategy
|
|
22
22
|
|
|
23
23
|
<Steps>
|
|
24
24
|
<Step titel="Organize Your Test Files">
|
|
25
25
|
|
|
26
|
-
Ensure your test files are stored in a directory (
|
|
26
|
+
Ensure your test files are stored in a directory (for example, `testdriver/`) and follow a consistent naming convention (for example, `test1.yaml`, `test2.yaml`, etc.).
|
|
27
27
|
|
|
28
28
|
</Step>
|
|
29
29
|
|
|
@@ -54,7 +54,7 @@ jobs:
|
|
|
54
54
|
- name: Find all test files
|
|
55
55
|
id: test_list
|
|
56
56
|
run: |
|
|
57
|
-
FILES=$(ls ./testdriver/*.
|
|
57
|
+
FILES=$(ls ./testdriver/*.yaml)
|
|
58
58
|
FILES_JSON=$(echo "$FILES" | jq -R -s -c 'split("\n")[:-1]')
|
|
59
59
|
echo "::set-output name=files::$FILES_JSON"
|
|
60
60
|
|
|
@@ -85,7 +85,7 @@ jobs:
|
|
|
85
85
|
</Steps>
|
|
86
86
|
---
|
|
87
87
|
|
|
88
|
-
### Explanation of the
|
|
88
|
+
### Explanation of the workflow
|
|
89
89
|
|
|
90
90
|
1. **`gather-test-files` Job**:
|
|
91
91
|
- Collects all test files in the `testdriver/` directory.
|
|
@@ -104,7 +104,7 @@ jobs:
|
|
|
104
104
|
|
|
105
105
|
---
|
|
106
106
|
|
|
107
|
-
## Benefits of
|
|
107
|
+
## Benefits of using the matrix strategy
|
|
108
108
|
|
|
109
109
|
1. **Dynamic Test Distribution**: Automatically adapts to the number of test files.
|
|
110
110
|
2. **Scalable**: Easily handles large test suites by distributing tests across multiple jobs.
|
|
@@ -112,15 +112,15 @@ jobs:
|
|
|
112
112
|
|
|
113
113
|
---
|
|
114
114
|
|
|
115
|
-
## Example
|
|
115
|
+
## Example output
|
|
116
116
|
|
|
117
117
|
When this workflow runs:
|
|
118
|
-
- Each test file (
|
|
118
|
+
- Each test file (for example, `test1.yaml`, `test2.yaml`) is executed in its own job.
|
|
119
119
|
- The results of all tests are displayed in the GitHub Actions dashboard.
|
|
120
120
|
|
|
121
121
|
---
|
|
122
122
|
|
|
123
|
-
## Best
|
|
123
|
+
## Best practices
|
|
124
124
|
|
|
125
125
|
1. **Organize Test Files**: Use a consistent naming convention for test files to simplify management.
|
|
126
126
|
2. **Monitor Test Results**: Review the GitHub Actions dashboard to identify and debug failing tests.
|
|
@@ -11,13 +11,13 @@ Reusable snippets in TestDriver allow you to modularize your test steps by creat
|
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
-
## What
|
|
14
|
+
## What are reusable snippets?
|
|
15
15
|
|
|
16
16
|
Reusable snippets are YAML files containing a set of test steps that perform a specific task, such as logging in, navigating to a page, or setting up test prerequisites. These snippets can be referenced in other test files using the `run` command, enabling you to reuse common actions across multiple tests.
|
|
17
17
|
|
|
18
18
|
---
|
|
19
19
|
|
|
20
|
-
## Why
|
|
20
|
+
## Why use reusable snippets?
|
|
21
21
|
|
|
22
22
|
1. **Maintainability**: Update a snippet in one place, and all tests using it will automatically reflect the changes.
|
|
23
23
|
2. **Reusability**: Avoid duplicating common actions like login or setup across multiple tests.
|
|
@@ -26,7 +26,7 @@ Reusable snippets are YAML files containing a set of test steps that perform a s
|
|
|
26
26
|
|
|
27
27
|
---
|
|
28
28
|
|
|
29
|
-
## How to
|
|
29
|
+
## How to create and use reusable snippets
|
|
30
30
|
<Steps>
|
|
31
31
|
<Step title="Create a Snippet">
|
|
32
32
|
|
|
@@ -88,7 +88,7 @@ TD_PASSWORD=your_password
|
|
|
88
88
|
</Steps>
|
|
89
89
|
|
|
90
90
|
|
|
91
|
-
## Example: Combining
|
|
91
|
+
## Example: Combining multiple snippets
|
|
92
92
|
|
|
93
93
|
You can chain multiple snippets together to create complex workflows. For example:
|
|
94
94
|
|
|
@@ -106,17 +106,17 @@ steps:
|
|
|
106
106
|
```
|
|
107
107
|
---
|
|
108
108
|
|
|
109
|
-
## Best
|
|
109
|
+
## Best practices for reusable snippets
|
|
110
110
|
|
|
111
111
|
1. **Organize Snippets**: Store reusable snippets in a dedicated `snippets/` directory for better organization.
|
|
112
|
-
2. **Use Descriptive Names**: Name snippet files clearly (
|
|
112
|
+
2. **Use Descriptive Names**: Name snippet files clearly (for example, `login.yaml`, `setup.yaml`) to indicate their purpose.
|
|
113
113
|
3. **Test Snippets Independently**: Validate each snippet before integrating it into larger workflows.
|
|
114
114
|
4. **Parameterize Inputs**: Use placeholders or environment variables for dynamic data.
|
|
115
115
|
5. **Document Snippets**: Add comments or documentation to explain the purpose and usage of each snippet.
|
|
116
116
|
|
|
117
117
|
---
|
|
118
118
|
|
|
119
|
-
## Benefits of
|
|
119
|
+
## Benefits of reusable snippets
|
|
120
120
|
|
|
121
121
|
- **Efficiency**: Save time by reusing existing logic.
|
|
122
122
|
- **Scalability**: Easily extend your test suite by combining snippets.
|
|
@@ -13,7 +13,7 @@ Instead, TestDriver uses natural language prompts and AI-powered vision to inter
|
|
|
13
13
|
- Tests are resilient to changes like text updates, class renaming, or minor layout adjustments.
|
|
14
14
|
- By using natural language and AI vision, TestDriver simplifies test creation and maintenance.
|
|
15
15
|
|
|
16
|
-
## What
|
|
16
|
+
## What's selectorless testing?
|
|
17
17
|
|
|
18
18
|
The following is an example of a TestDriver test.
|
|
19
19
|
|
|
@@ -38,10 +38,10 @@ This allows TestDriver locates the target for `hover-text` based on its context
|
|
|
38
38
|
- `description` - a description of the element given the exact text isn't found, or there are multiple matches
|
|
39
39
|
- `prompt` - a high level prompt used to regenerate the test if no match is found
|
|
40
40
|
|
|
41
|
-
### What
|
|
41
|
+
### What happens when "Sign Up" changes to "Register"?
|
|
42
42
|
|
|
43
43
|
If the button text changes to "Register," TestDriver's AI vision will still locate the button based on its context and description. You don't need to update the test manually.
|
|
44
|
-
TestDriver will then update the test to reflect the new UI by
|
|
44
|
+
TestDriver will then update the test to reflect the new UI by modifying the `text` field. Then, it will open a new pull request with the changes.
|
|
45
45
|
|
|
46
46
|
```yaml
|
|
47
47
|
version: 4.2.18
|
|
@@ -53,7 +53,7 @@ steps:
|
|
|
53
53
|
description: button in the header for user registration
|
|
54
54
|
action: click
|
|
55
55
|
```
|
|
56
|
-
## Why
|
|
56
|
+
## Why selectorless testing?
|
|
57
57
|
|
|
58
58
|
Traditional testing frameworks rely on selectors tightly coupled to the codebase.
|
|
59
59
|
For example:
|
|
@@ -7,57 +7,60 @@ icon: "binoculars"
|
|
|
7
7
|
|
|
8
8
|
Visual assertions in TestDriver allow you to validate that your application behaves as expected by analyzing what's displayed on the screen. This approach ensures that your tests mimic real user interactions and verify outcomes visually, making it ideal for testing UI-heavy applications and dynamic content.
|
|
9
9
|
|
|
10
|
-
## What
|
|
10
|
+
## What can TestDriver test with visual assertions?
|
|
11
11
|
|
|
12
|
-
|
|
13
|
-
See many of the following examples in the [TestDriver Scenarios](/scenarios/ai-chatbot)
|
|
14
|
-
</Info>
|
|
15
|
-
|
|
16
|
-
### AI Chatbots
|
|
12
|
+
### AI chatbots
|
|
17
13
|
- Validate that chatbot responses are displayed correctly.
|
|
18
14
|
- Ensure the chatbot interface is accessible and functional.
|
|
15
|
+
- See [AI Chatbots](/scenarios/ai-chatbot) for an example.
|
|
19
16
|
|
|
20
|
-
### Desktop
|
|
17
|
+
### Desktop applications
|
|
21
18
|
- Verify that desktop app windows, dialogs, and UI elements render as expected.
|
|
22
19
|
- Test cross-platform compatibility for Windows, Mac, and Linux.
|
|
20
|
+
- See [Using TestDriver with Desktop Apps](/apps/desktop-apps) for an example.
|
|
23
21
|
|
|
24
|
-
### Chrome
|
|
22
|
+
### Chrome extensions
|
|
25
23
|
- Confirm that the extension UI integrates properly with the browser.
|
|
26
24
|
- Validate extension popups, settings, and interactions.
|
|
25
|
+
- See [Using TestDriver with Chrome Extensions](/apps/chrome-extensions) for an example.
|
|
27
26
|
|
|
28
|
-
### PDF
|
|
27
|
+
### PDF generation
|
|
29
28
|
- Assert that generated PDFs contain the correct text, images, and formatting.
|
|
30
29
|
- Verify that PDFs open and display properly in viewers.
|
|
30
|
+
- See [PDF Generation](/scenarios/pdf-generation) for an example.
|
|
31
31
|
|
|
32
|
-
### Spelling &
|
|
32
|
+
### Spelling & grammar
|
|
33
33
|
- Check for spelling and grammar errors in displayed text.
|
|
34
34
|
- Validate that text content matches expected language rules.
|
|
35
|
+
- See [Spell Check](/scenarios/spell-check) for an example.
|
|
35
36
|
|
|
36
|
-
### Auth
|
|
37
|
+
### Auth signup & login
|
|
37
38
|
- Ensure OAuth flows display the correct login screens.
|
|
38
39
|
- Verify that users are redirected to the correct pages after login.
|
|
40
|
+
- See [Login](/scenarios/log-in) for an example.
|
|
39
41
|
|
|
40
|
-
### File
|
|
42
|
+
### File system & uploads
|
|
41
43
|
- Confirm that file upload dialogs appear and function correctly.
|
|
42
44
|
- Validate that uploaded files are processed and displayed as expected.
|
|
45
|
+
- See [File Upload](/scenarios/file-upload) for an example.
|
|
43
46
|
|
|
44
|
-
### Image
|
|
47
|
+
### Image content
|
|
45
48
|
- Verify that images are displayed correctly and match expected content.
|
|
46
49
|
- Test for the presence of specific icons, logos, or visual elements.
|
|
47
50
|
|
|
48
|
-
### Video
|
|
51
|
+
### Video content
|
|
49
52
|
- Assert that videos play correctly and match expected visuals.
|
|
50
53
|
- Validate video controls (play, pause, fullscreen) and captions.
|
|
51
54
|
|
|
52
|
-
### OS
|
|
55
|
+
### OS accessibility features
|
|
53
56
|
- Test high-contrast modes, screen readers, and other accessibility features.
|
|
54
57
|
- Validate that UI elements are accessible and readable.
|
|
55
58
|
|
|
56
|
-
### Light / Dark
|
|
59
|
+
### Light / Dark mode
|
|
57
60
|
- Verify that the application switches between light and dark modes correctly.
|
|
58
61
|
- Assert that UI elements are visible and styled appropriately in both modes.
|
|
59
62
|
|
|
60
|
-
### Privacy
|
|
63
|
+
### Privacy configuration
|
|
61
64
|
- Confirm that privacy settings are displayed and functional.
|
|
62
65
|
- Validate that toggles, checkboxes, and options reflect the correct state.
|
|
63
66
|
|
|
@@ -75,9 +78,9 @@ Visual assertions in TestDriver allow you to validate that your application beha
|
|
|
75
78
|
|
|
76
79
|
---
|
|
77
80
|
|
|
78
|
-
## How to
|
|
81
|
+
## How to use visual assertions
|
|
79
82
|
|
|
80
|
-
### Example: Asserting
|
|
83
|
+
### Example: Asserting text on the screen
|
|
81
84
|
|
|
82
85
|
```yaml
|
|
83
86
|
- command: assert
|
|
@@ -85,14 +88,14 @@ Visual assertions in TestDriver allow you to validate that your application beha
|
|
|
85
88
|
|
|
86
89
|
```
|
|
87
90
|
|
|
88
|
-
### Example: Asserting an
|
|
91
|
+
### Example: Asserting an image is present
|
|
89
92
|
|
|
90
93
|
```yaml
|
|
91
94
|
- command: assert
|
|
92
95
|
expect: The company logo appears in the header
|
|
93
96
|
```
|
|
94
97
|
|
|
95
|
-
### Example: Asserting
|
|
98
|
+
### Example: Asserting video playback
|
|
96
99
|
|
|
97
100
|
```yaml
|
|
98
101
|
- command: assert
|
|
@@ -101,7 +104,7 @@ Visual assertions in TestDriver allow you to validate that your application beha
|
|
|
101
104
|
```
|
|
102
105
|
---
|
|
103
106
|
|
|
104
|
-
## Benefits of
|
|
107
|
+
## Benefits of visual assertions
|
|
105
108
|
|
|
106
109
|
1. **Selectorless Testing**: No need to rely on brittle selectors or static code.
|
|
107
110
|
2. **Dynamic Adaptation**: Tests adapt to UI changes, reducing maintenance.
|
|
@@ -110,7 +113,7 @@ Visual assertions in TestDriver allow you to validate that your application beha
|
|
|
110
113
|
|
|
111
114
|
---
|
|
112
115
|
|
|
113
|
-
## Best Practices for
|
|
116
|
+
## Best Practices for visual assertions
|
|
114
117
|
|
|
115
118
|
1. **Use Descriptive Prompts**: Provide clear descriptions for elements or behaviors you want to test.
|
|
116
119
|
2. **Combine with Other Commands**: Use visual assertions alongside navigation and interaction commands for end-to-end testing.
|
|
@@ -9,17 +9,17 @@ This guide explains how to set up a basic GitHub Actions workflow to run tests u
|
|
|
9
9
|
|
|
10
10
|
## Prerequisites
|
|
11
11
|
|
|
12
|
-
1. **TestDriver API Key**: Obtain your API key from TestDriver and store it as a GitHub secret (
|
|
12
|
+
1. **TestDriver API Key**: Obtain your API key from TestDriver and store it as a GitHub secret (for example, `TD_API_KEY`).
|
|
13
13
|
2. **Test Files**: Ensure your test files are saved in the `testdriver/` directory of your repository.
|
|
14
14
|
|
|
15
|
-
## Create a GitHub Actions
|
|
15
|
+
## Create a GitHub Actions workflow
|
|
16
16
|
|
|
17
|
-
1. Create a new file in your repository: `.github/workflows/testdriver.
|
|
17
|
+
1. Create a new file in your repository: `.github/workflows/testdriver.yaml`.
|
|
18
18
|
2. Add the following workflow configuration:
|
|
19
19
|
|
|
20
|
-
### Example
|
|
20
|
+
### Example workflow
|
|
21
21
|
|
|
22
|
-
```yaml
|
|
22
|
+
```yaml .github/workflows/testdriver.yaml [expandable]
|
|
23
23
|
name: TestDriver / Run / Regressions
|
|
24
24
|
|
|
25
25
|
on:
|
|
@@ -38,7 +38,7 @@ permissions:
|
|
|
38
38
|
|
|
39
39
|
jobs:
|
|
40
40
|
gather-test-files:
|
|
41
|
-
name: Setup Test Matrix (./testdriver/*.
|
|
41
|
+
name: Setup Test Matrix (./testdriver/*.yaml)
|
|
42
42
|
runs-on: ubuntu-latest
|
|
43
43
|
outputs:
|
|
44
44
|
test_files: ${{ steps.test_list.outputs.files }}
|
|
@@ -50,7 +50,7 @@ jobs:
|
|
|
50
50
|
- name: Find all test files and extract filenames
|
|
51
51
|
id: test_list
|
|
52
52
|
run: |
|
|
53
|
-
FILES=$(ls ./testdriver/*.
|
|
53
|
+
FILES=$(ls ./testdriver/*.yaml)
|
|
54
54
|
FILENAMES=$(basename -a $FILES)
|
|
55
55
|
FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
|
|
56
56
|
echo "::set-output name=files::$FILES_JSON"
|
|
@@ -90,49 +90,49 @@ jobs:
|
|
|
90
90
|
FORCE_COLOR: "3"
|
|
91
91
|
```
|
|
92
92
|
|
|
93
|
-
## Trigger the
|
|
93
|
+
## Trigger the workflow
|
|
94
94
|
|
|
95
95
|
1. Push changes to the `main` branch or open a pull request.
|
|
96
96
|
2. Alternatively, manually trigger the workflow from the **Actions** tab in your GitHub repository.
|
|
97
97
|
|
|
98
|
-
## View
|
|
98
|
+
## View results
|
|
99
99
|
|
|
100
100
|
1. Navigate to the **Actions** tab in your GitHub repository.
|
|
101
101
|
2. Select the workflow run to view the test results.
|
|
102
102
|
|
|
103
|
-
# Debugging
|
|
103
|
+
# Debugging tests
|
|
104
104
|
TestDriver provides a powerful debugging interface through its **app.TestDriver** platform (formerly Dashcam). This interface allows you to analyze test runs, identify failures, and optimize your test suite with detailed visual and textual feedback.
|
|
105
105
|
|
|
106
|
-
### Step-by-
|
|
106
|
+
### 1. Step-by-step execution logs
|
|
107
107
|
- View each step of the test execution, including:
|
|
108
|
-
- The **action performed** (
|
|
109
|
-
- The **expected outcome** (
|
|
108
|
+
- The **action performed** (for example, clicking a button, typing text).
|
|
109
|
+
- The **expected outcome** (for example, verifying a specific element is visible).
|
|
110
110
|
- The **result** (pass, fail, or skipped).
|
|
111
111
|
- Logs provide detailed context for each step, making it easier to pinpoint where and why a test failed.
|
|
112
112
|
|
|
113
|
-
### 2. Visual
|
|
113
|
+
### 2. Visual feedback
|
|
114
114
|
- **Screenshots**: See what the application looked like at each step of the test.
|
|
115
115
|
- **GIF Previews**: Watch a replay of the entire test execution to understand the flow and identify UI issues.
|
|
116
116
|
- **Highlighting**: Elements interacted with during the test are highlighted in screenshots, helping you verify that the correct elements were targeted.
|
|
117
117
|
|
|
118
|
-
### 3. Error
|
|
118
|
+
### 3. Error details
|
|
119
119
|
- For failed steps, app.TestDriver provides:
|
|
120
|
-
- **Error messages**: Detailed descriptions of what went wrong (
|
|
120
|
+
- **Error messages**: Detailed descriptions of what went wrong (for example, "Element not found").
|
|
121
121
|
- **Stack traces**: For advanced debugging of backend or script-related issues.
|
|
122
122
|
- **Suggestions**: Recommendations for fixing common issues, such as adjusting prompts or improving element descriptions.
|
|
123
123
|
|
|
124
|
-
### 4. Test
|
|
124
|
+
### 4. Test history
|
|
125
125
|
- Access the history of test runs to:
|
|
126
126
|
- Compare results across different builds or environments.
|
|
127
127
|
- Identify flaky tests by analyzing patterns in failures.
|
|
128
128
|
- Track improvements or regressions over time.
|
|
129
129
|
|
|
130
|
-
### 5. Environment
|
|
130
|
+
### 5. Environment context
|
|
131
131
|
- View the environment details for each test run, including:
|
|
132
132
|
- Operating system and browser version.
|
|
133
133
|
- Screen resolution and viewport size.
|
|
134
134
|
- Network conditions (if applicable).
|
|
135
135
|
|
|
136
|
-
### 6. Collaboration
|
|
136
|
+
### 6. Collaboration tools
|
|
137
137
|
- Share test results with your team by generating a shareable link.
|
|
138
138
|
- Add comments or annotations to specific steps to facilitate discussions and debugging.
|
|
@@ -5,8 +5,12 @@ description: "Edit previously generated tests."
|
|
|
5
5
|
icon: "file-pen"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
+
import YmlWarning from '/snippets/yml-warning.mdx'
|
|
9
|
+
|
|
8
10
|
Test scripts are written in YAML format, making them easy to read and modify manually if you need to.
|
|
9
11
|
|
|
12
|
+
<YmlWarning/>
|
|
13
|
+
|
|
10
14
|
<Card
|
|
11
15
|
title="Commands Reference"
|
|
12
16
|
icon="link"
|
|
@@ -19,7 +23,7 @@ Test scripts are written in YAML format, making them easy to read and modify man
|
|
|
19
23
|
TestDriver test files are typically stored in the `testdriver/` directory of your project. The default file is `testdriver/testdriver.yaml`.
|
|
20
24
|
|
|
21
25
|
### Open the Test File
|
|
22
|
-
Use your preferred text editor or IDE (
|
|
26
|
+
Use your preferred text editor or IDE (for example, Visual Studio Code) to open the YAML file.
|
|
23
27
|
|
|
24
28
|
### YAML Structure
|
|
25
29
|
Each test file consists of:
|
|
@@ -27,9 +31,9 @@ Each test file consists of:
|
|
|
27
31
|
- **Session**: A unique identifier for the test session.
|
|
28
32
|
- **Steps**: A list of prompts and commands to execute.
|
|
29
33
|
|
|
30
|
-
```yaml
|
|
34
|
+
```yaml testfile.yaml
|
|
31
35
|
version: 4.2.18
|
|
32
|
-
session:
|
|
36
|
+
session: abc1234
|
|
33
37
|
steps:
|
|
34
38
|
- prompt: Open Google Chrome and navigate to Airbnb
|
|
35
39
|
commands:
|
|
@@ -45,15 +49,15 @@ steps:
|
|
|
45
49
|
keys: [enter]
|
|
46
50
|
```
|
|
47
51
|
|
|
48
|
-
<Tip>The `version` captured when you generate a test file is the version of TestDriver you used, and
|
|
52
|
+
<Tip>The `version` captured when you generate a test file is the version of TestDriver you used, and isn't used when running the test. This doesn't affect your ability to use newer versions of TestDriver on existing test files nor do these need to be updated when migrating to a new version.</Tip>
|
|
49
53
|
|
|
50
|
-
### Modify the
|
|
54
|
+
### Modify the test steps
|
|
51
55
|
You can add, edit, or remove steps as needed. Each step contains:
|
|
52
56
|
|
|
53
57
|
- **Prompt**: A description of the action.
|
|
54
58
|
- **Commands**: The specific actions to perform.
|
|
55
59
|
|
|
56
|
-
#### Example
|
|
60
|
+
#### Example edits:
|
|
57
61
|
- **Add a Wait Command**:
|
|
58
62
|
```yaml
|
|
59
63
|
- command: wait
|
|
@@ -66,13 +70,13 @@ You can add, edit, or remove steps as needed. Each step contains:
|
|
|
66
70
|
text: booking.com
|
|
67
71
|
```
|
|
68
72
|
|
|
69
|
-
### Validate the YAML
|
|
73
|
+
### Validate the YAML syntax
|
|
70
74
|
Ensure the YAML file is properly formatted:
|
|
71
75
|
- Use consistent indentation (2 spaces recommended).
|
|
72
76
|
- Avoid trailing spaces or tabs.
|
|
73
77
|
- Validate the file using a YAML linter if needed.
|
|
74
78
|
|
|
75
|
-
### Test the
|
|
79
|
+
### Test the changes
|
|
76
80
|
Run the updated test to verify it works as expected:
|
|
77
81
|
```bash
|
|
78
82
|
testdriverai run path/to/test.yaml
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "Self-Driving Tests"
|
|
3
3
|
sidebarTitle: "Self-Driving Tests"
|
|
4
|
-
description: "Generate test
|
|
4
|
+
description: "Generate test files and integrate them into your testing workflow."
|
|
5
5
|
icon: "steering-wheel"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
Writing tests can be a tedious and time-consuming task. TestDriver can generate tests just by exploring your app! This guide will show you how to generate tests using TestDriver.
|
|
9
9
|
|
|
10
|
-
# Launch TestDriver in
|
|
10
|
+
# Launch TestDriver in interactive mode
|
|
11
11
|
|
|
12
12
|
Launch `testdriverai` in interactive mode with the following command:
|
|
13
13
|
|
|
@@ -61,7 +61,7 @@ steps:
|
|
|
61
61
|
- prompt: Assert that there are issues during checkout and an error is displayed.
|
|
62
62
|
```
|
|
63
63
|
|
|
64
|
-
# Generate
|
|
64
|
+
# Generate regression tests from exploratory tests
|
|
65
65
|
|
|
66
66
|
Now it's time to generate the regression test.
|
|
67
67
|
|
|
@@ -80,11 +80,10 @@ No commands found, running exploratory
|
|
|
80
80
|
|
|
81
81
|
When using the `--save` command, TestDriver appends any generated commands to the test file.
|
|
82
82
|
|
|
83
|
-
So as your test run, TestDriver will append successful commands to the
|
|
83
|
+
So as your test run, TestDriver will append successful commands to the YAML file and save it.
|
|
84
84
|
|
|
85
85
|
```yaml
|
|
86
86
|
version: 5.3.11
|
|
87
|
-
session: 6801430b5f2c252a8f4bed60
|
|
88
87
|
steps:
|
|
89
88
|
- prompt: Focus the browser window.
|
|
90
89
|
commands:
|