testdriverai 5.6.3 → 5.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agent.js +17 -15
- package/docs/account/dashboard.mdx +3 -6
- package/docs/account/enterprise.mdx +16 -15
- package/docs/account/pricing.mdx +4 -4
- package/docs/account/projects.mdx +2 -2
- package/docs/account/team.mdx +3 -3
- package/docs/action/browser.mdx +9 -9
- package/docs/action/os.mdx +12 -12
- package/docs/action/output.mdx +10 -10
- package/docs/action/performance.mdx +11 -11
- package/docs/{guide/lifecycle.mdx → action/pre-post-scripts.mdx} +87 -69
- package/docs/action/secrets.mdx +11 -11
- package/docs/action/setup.mdx +8 -8
- package/docs/apps/chrome-extensions.mdx +9 -9
- package/docs/apps/desktop-apps.mdx +6 -5
- package/docs/apps/mobile-apps.mdx +3 -3
- package/docs/apps/static-websites.mdx +2 -3
- package/docs/bugs/jira.mdx +16 -16
- package/docs/cli/overview.mdx +8 -8
- package/docs/commands/exec.mdx +152 -10
- package/docs/commands/focus-application.mdx +1 -1
- package/docs/commands/hover-image.mdx +1 -1
- package/docs/commands/hover-text.mdx +2 -2
- package/docs/commands/if.mdx +1 -1
- package/docs/commands/issues.mdx +8 -6
- package/docs/commands/match-image.mdx +2 -2
- package/docs/commands/press-keys.mdx +19 -2
- package/docs/commands/run.mdx +1 -1
- package/docs/commands/scroll-until-image.mdx +1 -1
- package/docs/commands/scroll-until-text.mdx +2 -2
- package/docs/commands/scroll.mdx +2 -2
- package/docs/commands/type.mdx +1 -1
- package/docs/commands/wait-for-image.mdx +1 -1
- package/docs/commands/wait-for-text.mdx +1 -1
- package/docs/commands/wait.mdx +1 -1
- package/docs/docs.json +9 -9
- package/docs/exporting/playwright.mdx +16 -16
- package/docs/features/auto-healing.mdx +5 -5
- package/docs/features/cross-platform.mdx +4 -4
- package/docs/features/generation.mdx +7 -4
- package/docs/features/github.mdx +21 -21
- package/docs/features/parallel-testing.mdx +9 -9
- package/docs/features/reusable-snippets.mdx +7 -7
- package/docs/features/selectorless.mdx +4 -4
- package/docs/features/visual-assertions.mdx +26 -23
- package/docs/getting-started/ci.mdx +19 -19
- package/docs/getting-started/editing.mdx +12 -8
- package/docs/getting-started/generating.mdx +4 -5
- package/docs/getting-started/running.mdx +3 -3
- package/docs/getting-started/setup.mdx +9 -9
- package/docs/getting-started/vscode.mdx +63 -85
- package/docs/getting-started/writing.mdx +11 -7
- package/docs/guide/assertions.mdx +21 -21
- package/docs/guide/authentication.mdx +11 -11
- package/docs/guide/code.mdx +6 -6
- package/docs/guide/environment-variables.mdx +9 -9
- package/docs/guide/lifecycle-prerun.mdx +191 -0
- package/docs/guide/locating.mdx +15 -18
- package/docs/guide/variables.mdx +8 -8
- package/docs/guide/waiting.mdx +10 -10
- package/docs/importing/csv.mdx +23 -24
- package/docs/importing/gherkin.mdx +20 -20
- package/docs/importing/jira.mdx +22 -23
- package/docs/importing/testrail.mdx +17 -17
- package/docs/integrations/electron.mdx +7 -7
- package/docs/integrations/netlify.mdx +7 -7
- package/docs/integrations/vercel.mdx +13 -13
- package/docs/interactive/assert.mdx +3 -3
- package/docs/interactive/dry.mdx +7 -7
- package/docs/interactive/explore.mdx +9 -9
- package/docs/interactive/generate.mdx +2 -2
- package/docs/interactive/run.mdx +10 -10
- package/docs/interactive/save.mdx +3 -3
- package/docs/interactive/undo.mdx +1 -1
- package/docs/overview/comparison.mdx +6 -6
- package/docs/overview/faq.mdx +2 -2
- package/docs/overview/quickstart.mdx +1 -1
- package/docs/overview/upgrading.mdx +8 -4
- package/docs/overview/what-is-testdriver.mdx +6 -6
- package/docs/quickstart.mdx +1 -1
- package/docs/scenarios/ai-chatbot.mdx +3 -3
- package/docs/scenarios/cookie-banner.mdx +2 -2
- package/docs/scenarios/file-upload.mdx +2 -2
- package/docs/scenarios/form-filling.mdx +3 -3
- package/docs/scenarios/log-in.mdx +3 -3
- package/docs/scenarios/pdf-generation.mdx +3 -3
- package/docs/scenarios/spell-check.mdx +2 -2
- package/docs/security/action.mdx +6 -6
- package/docs/security/agent.mdx +10 -10
- package/docs/security/platform.mdx +10 -8
- package/docs/snippets/test-prereqs.mdx +1 -1
- package/docs/snippets/yml-warning.mdx +1 -0
- package/docs/tutorials/advanced-test.mdx +9 -9
- package/docs/tutorials/basic-test.mdx +3 -3
- package/package.json +1 -1
- package/styles/config/vocabularies/Docs/accept.txt +32 -4
- package/testdriver/spotify.yaml +5 -0
- package/docs/action/prerun.mdx +0 -137
|
@@ -9,11 +9,11 @@ This guide explains how to convert **Gherkin scenarios** into **TestDriver promp
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## What
|
|
12
|
+
## What's Gherkin?
|
|
13
13
|
|
|
14
14
|
Gherkin is a plain-text language used to describe test scenarios in a human-readable format. It uses keywords like `Given`, `When`, `Then`, and `And` to define steps in a test.
|
|
15
15
|
|
|
16
|
-
### Example Gherkin
|
|
16
|
+
### Example Gherkin scenario:
|
|
17
17
|
|
|
18
18
|
```gherkin
|
|
19
19
|
Scenario: Successful login
|
|
@@ -25,11 +25,11 @@ Scenario: Successful login
|
|
|
25
25
|
|
|
26
26
|
---
|
|
27
27
|
|
|
28
|
-
## What
|
|
28
|
+
## What are TestDriver prompts?
|
|
29
29
|
|
|
30
|
-
TestDriver prompts are high-level instructions that describe what the AI should do. They
|
|
30
|
+
TestDriver prompts are high-level instructions that describe what the AI should do. They're written in plain text and focus on user actions or expected outcomes.
|
|
31
31
|
|
|
32
|
-
### Example
|
|
32
|
+
### Example prompts:
|
|
33
33
|
|
|
34
34
|
```yaml
|
|
35
35
|
- prompt: the user is on the login page
|
|
@@ -41,9 +41,9 @@ TestDriver prompts are high-level instructions that describe what the AI should
|
|
|
41
41
|
|
|
42
42
|
---
|
|
43
43
|
|
|
44
|
-
## Steps to
|
|
44
|
+
## Steps to convert Gherkin to TestDriver prompts
|
|
45
45
|
|
|
46
|
-
### Step 1: Understand the
|
|
46
|
+
### Step 1: Understand the mapping
|
|
47
47
|
|
|
48
48
|
| **Gherkin Keyword** | **TestDriver Prompt** |
|
|
49
49
|
|----------------------|----------------------------------------------|
|
|
@@ -54,7 +54,7 @@ TestDriver prompts are high-level instructions that describe what the AI should
|
|
|
54
54
|
|
|
55
55
|
---
|
|
56
56
|
|
|
57
|
-
### Step 2: Extract
|
|
57
|
+
### Step 2: Extract steps from Gherkin
|
|
58
58
|
|
|
59
59
|
Take each step from the Gherkin scenario and rewrite it as a plain-text prompt. Remove the `Given`, `When`, `Then`, and `And` keywords, and focus on the action or expectation.
|
|
60
60
|
|
|
@@ -69,7 +69,7 @@ Take each step from the Gherkin scenario and rewrite it as a plain-text prompt.
|
|
|
69
69
|
|
|
70
70
|
---
|
|
71
71
|
|
|
72
|
-
### Step 3: Write the
|
|
72
|
+
### Step 3: Write the prompts in YAML format
|
|
73
73
|
|
|
74
74
|
Combine the extracted prompts into a YAML file. Each step should be written as a `prompt` entry.
|
|
75
75
|
|
|
@@ -84,28 +84,28 @@ Combine the extracted prompts into a YAML file. Each step should be written as a
|
|
|
84
84
|
|
|
85
85
|
---
|
|
86
86
|
|
|
87
|
-
### Step 4: Save the YAML
|
|
87
|
+
### Step 4: Save the YAML file
|
|
88
88
|
|
|
89
|
-
1. Save the YAML content to a file (
|
|
90
|
-
2. Ensure the file is stored in the appropriate directory for your TestDriver project (
|
|
89
|
+
1. Save the YAML content to a file (for example, `login_test.yaml`).
|
|
90
|
+
2. Ensure the file is stored in the appropriate directory for your TestDriver project (for example, `testdriver/`).
|
|
91
91
|
|
|
92
92
|
---
|
|
93
93
|
|
|
94
|
-
### Step 5: Run the
|
|
94
|
+
### Step 5: Run the test with TestDriver
|
|
95
95
|
|
|
96
96
|
Use the TestDriver CLI to execute the test.
|
|
97
97
|
|
|
98
98
|
#### Command:
|
|
99
99
|
|
|
100
100
|
```bash
|
|
101
|
-
testdriverai run login_test.
|
|
101
|
+
testdriverai run login_test.yaml
|
|
102
102
|
```
|
|
103
103
|
|
|
104
104
|
---
|
|
105
105
|
|
|
106
|
-
## Example: Full
|
|
106
|
+
## Example: Full conversion workflow
|
|
107
107
|
|
|
108
|
-
### Input Gherkin
|
|
108
|
+
### Input Gherkin scenario:
|
|
109
109
|
|
|
110
110
|
```gherkin
|
|
111
111
|
Scenario: Add a product to the cart
|
|
@@ -124,20 +124,20 @@ Scenario: Add a product to the cart
|
|
|
124
124
|
- prompt: the product should appear in the cart
|
|
125
125
|
```
|
|
126
126
|
|
|
127
|
-
### Run the
|
|
127
|
+
### Run the test:
|
|
128
128
|
|
|
129
129
|
```bash
|
|
130
|
-
testdriverai run add_to_cart_test.
|
|
130
|
+
testdriverai run add_to_cart_test.yaml
|
|
131
131
|
```
|
|
132
132
|
|
|
133
133
|
---
|
|
134
134
|
|
|
135
|
-
## Best
|
|
135
|
+
## Best practices
|
|
136
136
|
|
|
137
137
|
1. **Keep Prompts Simple**: Focus on high-level actions or outcomes. Avoid including unnecessary details.
|
|
138
138
|
2. **Use Descriptive Prompts**: Ensure each prompt clearly describes the action or expectation.
|
|
139
139
|
3. **Test the YAML**: Run the converted YAML file to verify that it works as expected.
|
|
140
|
-
4. **Organize Files**: Store YAML files in a structured directory (
|
|
140
|
+
4. **Organize Files**: Store YAML files in a structured directory (for example, `testdriver/`) for easy management.
|
|
141
141
|
|
|
142
142
|
---
|
|
143
143
|
|
package/docs/importing/jira.mdx
CHANGED
|
@@ -5,13 +5,13 @@ description: "Import user stories from Jira into TestDriver test files."
|
|
|
5
5
|
icon: "file-image"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# Importing
|
|
8
|
+
# Importing user stories from Jira into TestDriver test files
|
|
9
9
|
|
|
10
10
|
This guide explains how to extract **user stories** from Jira and convert them into **TestDriver test files**. By automating this process, you can ensure that your user stories are directly translated into actionable test cases for TestDriver.
|
|
11
11
|
|
|
12
12
|
---
|
|
13
13
|
|
|
14
|
-
## Workflow
|
|
14
|
+
## Workflow overview
|
|
15
15
|
|
|
16
16
|
1. **Export User Stories from Jira**: Use the Jira API to fetch user stories.
|
|
17
17
|
2. **Convert User Stories to TestDriver YAML**: Transform the user stories into YAML test files.
|
|
@@ -20,26 +20,26 @@ This guide explains how to extract **user stories** from Jira and convert them i
|
|
|
20
20
|
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
-
## Step 1: Export
|
|
23
|
+
## Step 1: Export user stories from Jira
|
|
24
24
|
|
|
25
25
|
### Prerequisites
|
|
26
26
|
1. **Jira API Token**: Generate an API token from your Jira account.
|
|
27
|
-
2. **Jira Base URL**: Your Jira instance URL (
|
|
27
|
+
2. **Jira Base URL**: Your Jira instance URL (for example, `https://yourcompany.atlassian.net`).
|
|
28
28
|
3. **Node.js**: Ensure Node.js is installed on your system.
|
|
29
29
|
|
|
30
|
-
### Script: Export
|
|
30
|
+
### Script: Export user stories from Jira
|
|
31
31
|
|
|
32
32
|
The following script fetches Jira tickets and extracts the **user story title** and **acceptance criteria**.
|
|
33
33
|
|
|
34
|
-
#### Install
|
|
34
|
+
#### Install dependencies:
|
|
35
35
|
|
|
36
36
|
```bash
|
|
37
37
|
npm install axios yaml fs
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
-
#### Node.js
|
|
40
|
+
#### Node.js script (`export-jira-user-stories.js`):
|
|
41
41
|
|
|
42
|
-
```javascript
|
|
42
|
+
```javascript [expandable]
|
|
43
43
|
const axios = require('axios');
|
|
44
44
|
const yaml = require('yaml');
|
|
45
45
|
const fs = require('fs');
|
|
@@ -102,7 +102,7 @@ function createYamlFile(ticket) {
|
|
|
102
102
|
steps,
|
|
103
103
|
};
|
|
104
104
|
|
|
105
|
-
const fileName = `${ticket.id}.
|
|
105
|
+
const fileName = `${ticket.id}.yaml`;
|
|
106
106
|
const filePath = path.join(OUTPUT_DIR, fileName);
|
|
107
107
|
|
|
108
108
|
fs.writeFileSync(filePath, yaml.stringify(yamlContent), 'utf8');
|
|
@@ -111,17 +111,15 @@ function createYamlFile(ticket) {
|
|
|
111
111
|
|
|
112
112
|
// Run the script
|
|
113
113
|
fetchJiraTickets();
|
|
114
|
-
|
|
115
|
-
|
|
116
114
|
```
|
|
117
115
|
|
|
118
116
|
---
|
|
119
117
|
|
|
120
|
-
## Step 2: Convert
|
|
118
|
+
## Step 2: Convert user stories to TestDriver YAML
|
|
121
119
|
|
|
122
120
|
The script above generates a YAML file for each Jira ticket. Each file contains the **acceptance criteria** as `prompt` entries.
|
|
123
121
|
|
|
124
|
-
### Example YAML
|
|
122
|
+
### Example YAML file (`testdriver_tests/PROJ-123.yaml`):
|
|
125
123
|
|
|
126
124
|
```yaml
|
|
127
125
|
version: 4.2.18
|
|
@@ -131,41 +129,42 @@ steps:
|
|
|
131
129
|
- prompt: The login page is responsive on mobile devices.
|
|
132
130
|
|
|
133
131
|
|
|
134
|
-
|
|
132
|
+
```
|
|
133
|
+
---
|
|
135
134
|
|
|
136
|
-
## Step 3: Save and
|
|
135
|
+
## Step 3: Save and organize test files
|
|
137
136
|
|
|
138
137
|
1. The generated YAML files will be saved in the `testdriver_tests/` directory.
|
|
139
138
|
2. Ensure the directory is part of your TestDriver project structure.
|
|
140
139
|
|
|
141
140
|
---
|
|
142
141
|
|
|
143
|
-
## Step 4: Run
|
|
142
|
+
## Step 4: Run tests with TestDriver
|
|
144
143
|
|
|
145
144
|
Use the TestDriver CLI to execute the generated test files.
|
|
146
145
|
|
|
147
|
-
### Run a
|
|
146
|
+
### Run a single test file:
|
|
148
147
|
|
|
149
148
|
```bash
|
|
150
|
-
testdriverai run testdriver_tests/PROJ-123.
|
|
149
|
+
testdriverai run testdriver_tests/PROJ-123.yaml
|
|
151
150
|
|
|
152
151
|
|
|
153
152
|
```
|
|
154
153
|
|
|
155
|
-
### Run
|
|
154
|
+
### Run all test files:
|
|
156
155
|
|
|
157
156
|
```bash
|
|
158
|
-
testdriverai run testdriver_tests/*.
|
|
157
|
+
testdriverai run testdriver_tests/*.yaml
|
|
159
158
|
```
|
|
160
159
|
|
|
161
160
|
---
|
|
162
161
|
|
|
163
|
-
## Best
|
|
162
|
+
## Best practices
|
|
164
163
|
|
|
165
|
-
1. **Field Mapping**: Ensure the correct Jira field ID (
|
|
164
|
+
1. **Field Mapping**: Ensure the correct Jira field ID (for example, `customfield_12345`) is used for "Acceptance Criteria."
|
|
166
165
|
2. **Secure Credentials**: Store Jira API credentials in environment variables or secrets.
|
|
167
166
|
3. **Review Generated Files**: Manually review the YAML files to ensure they align with your testing requirements.
|
|
168
|
-
4. **Organize Tests**: Use a structured directory (
|
|
167
|
+
4. **Organize Tests**: Use a structured directory (for example, `testdriver_tests/`) to manage your test files.
|
|
169
168
|
|
|
170
169
|
---
|
|
171
170
|
|
|
@@ -9,7 +9,7 @@ This guide explains how to extract **test cases** from TestRail and convert them
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## Workflow
|
|
12
|
+
## Workflow overview
|
|
13
13
|
|
|
14
14
|
1. **Export Test Cases from TestRail**: Use the TestRail API to fetch test cases.
|
|
15
15
|
2. **Convert Test Cases to TestDriver YAML**: Transform the test cases into YAML test files.
|
|
@@ -18,26 +18,26 @@ This guide explains how to extract **test cases** from TestRail and convert them
|
|
|
18
18
|
|
|
19
19
|
---
|
|
20
20
|
|
|
21
|
-
## Step 1: Export
|
|
21
|
+
## Step 1: Export test cases from TestRail
|
|
22
22
|
|
|
23
23
|
### Prerequisites
|
|
24
24
|
1. **TestRail API Key**: Obtain your API key from TestRail.
|
|
25
|
-
2. **TestRail Base URL**: Your TestRail instance URL (
|
|
25
|
+
2. **TestRail Base URL**: Your TestRail instance URL (for example, `https://yourcompany.testrail.io`).
|
|
26
26
|
3. **Node.js**: Ensure Node.js is installed on your system.
|
|
27
27
|
|
|
28
|
-
### Script: Export
|
|
28
|
+
### Script: Export test cases from TestRail
|
|
29
29
|
|
|
30
30
|
The following script fetches TestRail test cases and extracts the **title** and **steps**.
|
|
31
31
|
|
|
32
|
-
#### Install
|
|
32
|
+
#### Install dependencies:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
35
|
npm install axios yaml fs
|
|
36
36
|
```
|
|
37
37
|
|
|
38
|
-
#### Node.js
|
|
38
|
+
#### Node.js script (`export-testrail-test-cases.js`):
|
|
39
39
|
|
|
40
|
-
```javascript
|
|
40
|
+
```javascript [expandable]
|
|
41
41
|
const axios = require('axios');
|
|
42
42
|
const yaml = require('yaml');
|
|
43
43
|
const fs = require('fs');
|
|
@@ -100,7 +100,7 @@ function createYamlFile(testCase) {
|
|
|
100
100
|
steps,
|
|
101
101
|
};
|
|
102
102
|
|
|
103
|
-
const fileName = `test_${testCase.id}.
|
|
103
|
+
const fileName = `test_${testCase.id}.yaml`;
|
|
104
104
|
const filePath = path.join(OUTPUT_DIR, fileName);
|
|
105
105
|
|
|
106
106
|
fs.writeFileSync(filePath, yaml.stringify(yamlContent), 'utf8');
|
|
@@ -113,11 +113,11 @@ fetchTestRailTestCases();
|
|
|
113
113
|
|
|
114
114
|
---
|
|
115
115
|
|
|
116
|
-
## Step 2: Convert
|
|
116
|
+
## Step 2: Convert test cases to TestDriver YAML
|
|
117
117
|
|
|
118
118
|
The script above generates a YAML file for each TestRail test case. Each file contains the **steps** as `prompt` entries.
|
|
119
119
|
|
|
120
|
-
### Example YAML
|
|
120
|
+
### Example YAML file (`testdriver_tests/test_123.yaml`):
|
|
121
121
|
|
|
122
122
|
```yaml
|
|
123
123
|
version: 4.2.18
|
|
@@ -130,31 +130,31 @@ steps:
|
|
|
130
130
|
|
|
131
131
|
---
|
|
132
132
|
|
|
133
|
-
## Step 3: Save and
|
|
133
|
+
## Step 3: Save and organize test files
|
|
134
134
|
|
|
135
135
|
1. The generated YAML files will be saved in the `testdriver_tests/` directory.
|
|
136
136
|
2. Ensure the directory is part of your TestDriver project structure.
|
|
137
137
|
|
|
138
138
|
---
|
|
139
139
|
|
|
140
|
-
## Step 4: Run
|
|
140
|
+
## Step 4: Run tests with TestDriver
|
|
141
141
|
|
|
142
142
|
Use the TestDriver CLI to execute the generated test files.
|
|
143
143
|
|
|
144
|
-
### Run a Single
|
|
144
|
+
### Run a Single test file:
|
|
145
145
|
|
|
146
146
|
```bash
|
|
147
|
-
testdriverai run testdriver_tests/test_123.
|
|
147
|
+
testdriverai run testdriver_tests/test_123.yaml
|
|
148
148
|
```
|
|
149
149
|
|
|
150
150
|
---
|
|
151
151
|
|
|
152
|
-
## Best
|
|
152
|
+
## Best practices
|
|
153
153
|
|
|
154
|
-
1. **Field Mapping**: Ensure the correct TestRail field ID (
|
|
154
|
+
1. **Field Mapping**: Ensure the correct TestRail field ID (for example, `custom_steps`) is used for test steps.
|
|
155
155
|
2. **Secure Credentials**: Store TestRail API credentials in environment variables or secrets.
|
|
156
156
|
3. **Review Generated Files**: Manually review the YAML files to ensure they align with your testing requirements.
|
|
157
|
-
4. **Organize Tests**: Use a structured directory (
|
|
157
|
+
4. **Organize Tests**: Use a structured directory (for example, `testdriver_tests/`) to manage your test files.
|
|
158
158
|
|
|
159
159
|
---
|
|
160
160
|
|
|
@@ -9,11 +9,11 @@ To integrate **TestDriver** with a workflow that uses the **runner artifact URL*
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## Updated
|
|
12
|
+
## Updated workflow with TestDriver integration
|
|
13
13
|
|
|
14
14
|
This workflow builds the application, uploads the build as an artifact, and then uses TestDriver to download the artifact via the runner artifact URL and run tests.
|
|
15
15
|
|
|
16
|
-
### Workflow
|
|
16
|
+
### Workflow file: `.github/workflows/testdriver-integration.yaml`
|
|
17
17
|
```yaml
|
|
18
18
|
name: Build and Test with TestDriver
|
|
19
19
|
|
|
@@ -86,7 +86,7 @@ jobs:
|
|
|
86
86
|
|
|
87
87
|
---
|
|
88
88
|
|
|
89
|
-
## Key
|
|
89
|
+
## Key changes and explanation
|
|
90
90
|
|
|
91
91
|
### 1. **Artifact URL Retrieval**
|
|
92
92
|
The `Get Artifact URL` step constructs the artifact URL dynamically using the GitHub repository and server URL. This ensures the TestDriver runner can download the artifact.
|
|
@@ -100,7 +100,7 @@ The `Get Artifact URL` step constructs the artifact URL dynamically using the Gi
|
|
|
100
100
|
|
|
101
101
|
---
|
|
102
102
|
|
|
103
|
-
### 2. **Downloading the
|
|
103
|
+
### 2. **Downloading the artifact**
|
|
104
104
|
The `prerun` script in the TestDriver action uses `curl` to download the artifact from the runner URL. The `GITHUB_TOKEN` is passed as a header for authentication.
|
|
105
105
|
```yaml
|
|
106
106
|
prerun: |
|
|
@@ -117,7 +117,7 @@ prerun: |
|
|
|
117
117
|
|
|
118
118
|
---
|
|
119
119
|
|
|
120
|
-
### 3. **TestDriver
|
|
120
|
+
### 3. **TestDriver integration**
|
|
121
121
|
The TestDriver action is configured to:
|
|
122
122
|
- Download the artifact.
|
|
123
123
|
- Extract the artifact.
|
|
@@ -139,7 +139,7 @@ The TestDriver action is configured to:
|
|
|
139
139
|
|
|
140
140
|
---
|
|
141
141
|
|
|
142
|
-
## Secrets
|
|
142
|
+
## Secrets configuration
|
|
143
143
|
|
|
144
144
|
Add the following secrets to your GitHub repository:
|
|
145
145
|
1. **`TD_API_KEY`**: Your TestDriver API key.
|
|
@@ -147,7 +147,7 @@ Add the following secrets to your GitHub repository:
|
|
|
147
147
|
|
|
148
148
|
---
|
|
149
149
|
|
|
150
|
-
## Benefits of
|
|
150
|
+
## Benefits of this workflow
|
|
151
151
|
|
|
152
152
|
1. **Dynamic Artifact Access**: Ensures TestDriver can download artifacts directly from the runner.
|
|
153
153
|
2. **Automated Testing**: Integrates TestDriver to validate the application after the build.
|
|
@@ -9,7 +9,7 @@ This guide explains how to integrate **TestDriver** with **Netlify deployments**
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## Workflow
|
|
12
|
+
## Workflow overview
|
|
13
13
|
|
|
14
14
|
1. **Trigger Netlify Deployment**: Use Netlify's GitHub integration to deploy your application on every pull request or push to the main branch.
|
|
15
15
|
2. **Run Tests on the Deployment URL**: Use the TestDriver GitHub Action to test the deployed application using the Netlify deployment URL.
|
|
@@ -20,16 +20,16 @@ This guide explains how to integrate **TestDriver** with **Netlify deployments**
|
|
|
20
20
|
## Prerequisites
|
|
21
21
|
|
|
22
22
|
1. **Netlify GitHub Integration**: Ensure your repository is connected to Netlify for automatic deployments.
|
|
23
|
-
2. **TestDriver API Key**: Store your API key as a GitHub secret (
|
|
23
|
+
2. **TestDriver API Key**: Store your API key as a GitHub secret (for example, `TD_API_KEY`).
|
|
24
24
|
3. **Netlify Deployment URL**: Use the `DEPLOY_URL` environment variable provided by Netlify to access the deployment.
|
|
25
25
|
|
|
26
26
|
---
|
|
27
27
|
|
|
28
|
-
## GitHub Actions
|
|
28
|
+
## GitHub Actions workflow
|
|
29
29
|
|
|
30
30
|
Here's a complete workflow to test Netlify deployments with TestDriver:
|
|
31
31
|
|
|
32
|
-
### Workflow
|
|
32
|
+
### Workflow file: `.github/workflows/test-netlify.yaml
|
|
33
33
|
```yaml
|
|
34
34
|
name: Test Netlify Deployment with TestDriver
|
|
35
35
|
|
|
@@ -75,9 +75,9 @@ uses: actions/checkout@v2
|
|
|
75
75
|
|
|
76
76
|
---
|
|
77
77
|
|
|
78
|
-
## Workflow
|
|
78
|
+
## Workflow steps explained
|
|
79
79
|
|
|
80
|
-
### 1. **Wait for Netlify
|
|
80
|
+
### 1. **Wait for Netlify deployment**
|
|
81
81
|
Netlify automatically sets the `DEPLOY_URL` environment variable for each deployment. This step ensures the deployment URL is available before running tests.
|
|
82
82
|
```yaml
|
|
83
83
|
- name: Wait for Netlify Deployment
|
|
@@ -94,6 +94,6 @@ Netlify automatically sets the `DEPLOY_URL` environment variable for each deploy
|
|
|
94
94
|
|
|
95
95
|
---
|
|
96
96
|
|
|
97
|
-
### 2. **Run
|
|
97
|
+
### 2. **Run tests with TestDriver**
|
|
98
98
|
|
|
99
99
|
The TestDriver GitHub Action runs tests on the deployed application using the deployment URL. The `prompt` field specifies the test steps to execute.
|
|
@@ -9,7 +9,7 @@ This guide explains how to integrate **TestDriver** with **Vercel deployments**
|
|
|
9
9
|
|
|
10
10
|
---
|
|
11
11
|
|
|
12
|
-
## Workflow
|
|
12
|
+
## Workflow overview
|
|
13
13
|
|
|
14
14
|
1. **Trigger Vercel Deployment**: Use Vercel's GitHub integration to deploy your application on every pull request or push to the main branch.
|
|
15
15
|
2. **Run Tests on the Deployment URL**: Use the TestDriver GitHub Action to test the deployed application using the Vercel deployment URL.
|
|
@@ -20,16 +20,16 @@ This guide explains how to integrate **TestDriver** with **Vercel deployments**
|
|
|
20
20
|
## Prerequisites
|
|
21
21
|
|
|
22
22
|
1. **Vercel GitHub Integration**: Ensure your repository is connected to Vercel for automatic deployments.
|
|
23
|
-
2. **TestDriver API Key**: Store your API key as a GitHub secret (
|
|
23
|
+
2. **TestDriver API Key**: Store your API key as a GitHub secret (for example, `TD_API_KEY`).
|
|
24
24
|
3. **Vercel Deployment URL**: Use the `VERCEL_URL` environment variable provided by Vercel to access the deployment.
|
|
25
25
|
|
|
26
26
|
---
|
|
27
27
|
|
|
28
|
-
## GitHub Actions
|
|
28
|
+
## GitHub Actions workflow
|
|
29
29
|
|
|
30
30
|
Here's a complete workflow to test Vercel deployments with TestDriver:
|
|
31
31
|
|
|
32
|
-
### Workflow
|
|
32
|
+
### Workflow file: `.github/workflows/test-vercel.yaml`
|
|
33
33
|
```yaml
|
|
34
34
|
name: Test Vercel Deployment with TestDriver
|
|
35
35
|
|
|
@@ -79,9 +79,9 @@ jobs:
|
|
|
79
79
|
|
|
80
80
|
---
|
|
81
81
|
|
|
82
|
-
## Workflow
|
|
82
|
+
## Workflow steps explained
|
|
83
83
|
|
|
84
|
-
### 1. **Wait for Vercel
|
|
84
|
+
### 1. **Wait for Vercel deployment**
|
|
85
85
|
The `amondnet/vercel-action` waits for the Vercel deployment to complete and retrieves the deployment URL. This URL is stored in the `steps.vercel.outputs.url` variable.
|
|
86
86
|
|
|
87
87
|
```yaml
|
|
@@ -101,7 +101,7 @@ The `amondnet/vercel-action` waits for the Vercel deployment to complete and ret
|
|
|
101
101
|
|
|
102
102
|
---
|
|
103
103
|
|
|
104
|
-
### 2. **Run
|
|
104
|
+
### 2. **Run tests with TestDriver**
|
|
105
105
|
The TestDriver GitHub Action runs tests on the deployed application using the deployment URL. The `prompt` field specifies the test steps to execute.
|
|
106
106
|
|
|
107
107
|
```yaml
|
|
@@ -125,7 +125,7 @@ The TestDriver GitHub Action runs tests on the deployed application using the de
|
|
|
125
125
|
|
|
126
126
|
---
|
|
127
127
|
|
|
128
|
-
## Example TestDriver
|
|
128
|
+
## Example TestDriver prompt
|
|
129
129
|
|
|
130
130
|
The `prompt` field in the TestDriver action specifies the steps to test the Vercel deployment. For example:
|
|
131
131
|
|
|
@@ -142,7 +142,7 @@ prompt: |
|
|
|
142
142
|
|
|
143
143
|
---
|
|
144
144
|
|
|
145
|
-
## Secrets
|
|
145
|
+
## Secrets configuration
|
|
146
146
|
|
|
147
147
|
Add the following secrets to your GitHub repository:
|
|
148
148
|
1. **`TD_API_KEY`**: Your TestDriver API key.
|
|
@@ -152,7 +152,7 @@ Add the following secrets to your GitHub repository:
|
|
|
152
152
|
|
|
153
153
|
---
|
|
154
154
|
|
|
155
|
-
## Benefits of
|
|
155
|
+
## Benefits of this workflow
|
|
156
156
|
|
|
157
157
|
1. **Automated Deployment Testing**: Automatically test every Vercel deployment, including preview and production builds.
|
|
158
158
|
2. **Early Issue Detection**: Catch issues in pull requests before merging.
|
|
@@ -161,13 +161,13 @@ Add the following secrets to your GitHub repository:
|
|
|
161
161
|
|
|
162
162
|
---
|
|
163
163
|
|
|
164
|
-
## Example
|
|
164
|
+
## Example output
|
|
165
165
|
|
|
166
|
-
### GitHub Actions
|
|
166
|
+
### GitHub Actions dashboard:
|
|
167
167
|
- ✅ **Test Vercel Deployment**: All tests passed.
|
|
168
168
|
- ❌ **Test Vercel Deployment**: 1 test failed. View logs for details.
|
|
169
169
|
|
|
170
|
-
### TestDriver
|
|
170
|
+
### TestDriver logs:
|
|
171
171
|
- **Step 1**: Opened the deployment URL.
|
|
172
172
|
- **Step 2**: Verified the homepage loaded correctly.
|
|
173
173
|
- **Step 3**: Clicked the "Sign Up" button.
|
|
@@ -20,9 +20,9 @@ The `/assert` command ensures that a specific condition is true within your test
|
|
|
20
20
|
- TestDriver takes a screenshot and uses it to verify that the condition described in the `expect` field is true.
|
|
21
21
|
- If the condition isn't met, the test will fail and exit immediately.
|
|
22
22
|
|
|
23
|
-
## Example
|
|
23
|
+
## Example usage
|
|
24
24
|
|
|
25
|
-
### Basic
|
|
25
|
+
### Basic assertion
|
|
26
26
|
```bash
|
|
27
27
|
/assert No error message is displayed
|
|
28
28
|
```
|
|
@@ -32,7 +32,7 @@ This generates the following command:
|
|
|
32
32
|
expect: There is no error message
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
-
### Asynchronous
|
|
35
|
+
### Asynchronous assertion
|
|
36
36
|
To speed up tests, use `async: true` to allow the test to continue without waiting for the assertion to pass:
|
|
37
37
|
```yaml
|
|
38
38
|
- command: assert
|
package/docs/interactive/dry.mdx
CHANGED
|
@@ -1,27 +1,27 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "/dry"
|
|
3
3
|
sidebarTitle: "/dry"
|
|
4
|
-
description: "See and write a test
|
|
4
|
+
description: "See and write a test file without running the commands."
|
|
5
5
|
icon: "forward-step"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
## Description
|
|
9
|
-
The `/dry` command is used to create a test
|
|
9
|
+
The `/dry` command is used to create a test file and write to a specified file. This command won't run steps it creates. It's a **dry** run.
|
|
10
10
|
|
|
11
11
|
## Usage
|
|
12
12
|
```bash
|
|
13
13
|
/dry <file>
|
|
14
14
|
```
|
|
15
15
|
|
|
16
|
-
## Example
|
|
16
|
+
## Example usage
|
|
17
17
|
```bash
|
|
18
18
|
testdriverai <filename.yaml>
|
|
19
19
|
> /dry click the 'log in' button
|
|
20
20
|
```
|
|
21
|
-
This command generates the steps to `click the 'log in' button` with an explanation and adds them to the current test
|
|
21
|
+
This command generates the steps to `click the 'log in' button` with an explanation and adds them to the current test file, without executing any command.
|
|
22
22
|
|
|
23
23
|
## Behavior
|
|
24
|
-
- TestDriver will create steps for the test
|
|
24
|
+
- TestDriver will create steps for the test file, writing them in the file.
|
|
25
25
|
- TestDriver will also show the reasoning and test steps in the CLI:
|
|
26
26
|
|
|
27
27
|
```bash
|
|
@@ -43,8 +43,8 @@ thinking...
|
|
|
43
43
|
```
|
|
44
44
|
## Protips
|
|
45
45
|
- Ensure you run the `testdriverai` command with a valid file path before using `/dry`.
|
|
46
|
-
- Use descriptive filenames for your test
|
|
47
|
-
- Combine `/dry` with `/undo` to write test
|
|
46
|
+
- Use descriptive filenames for your test files to make them easier to identify.
|
|
47
|
+
- Combine `/dry` with `/undo` to write test files without executing them.
|
|
48
48
|
|
|
49
49
|
|
|
50
50
|
## Notes
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "/explore"
|
|
3
3
|
sidebarTitle: "/explore"
|
|
4
|
-
description: "Create and execute a test
|
|
4
|
+
description: "Create and execute a test file"
|
|
5
5
|
icon: "play-pause"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
## Description
|
|
9
9
|
|
|
10
|
-
The `/explore` command is used to execute a single step in a test
|
|
10
|
+
The `/explore` command is used to execute a single step in a test file and add it to a specified file. This command generates and performs steps in an interactive way while creating a test file.
|
|
11
11
|
|
|
12
12
|
<Tip>`/explore` is the default command sent to TestDriver when no command is provided.</Tip>
|
|
13
13
|
|
|
@@ -17,11 +17,11 @@ testdriverai <filename.yaml>
|
|
|
17
17
|
/run prompt
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
-
## Example
|
|
20
|
+
## Example usage
|
|
21
21
|
```bash
|
|
22
22
|
> /explore add an item to the shopping cart
|
|
23
23
|
```
|
|
24
|
-
This command creates the command step(s) necessary to add an item to the shopping cart in the specified test
|
|
24
|
+
This command creates the command step(s) necessary to add an item to the shopping cart in the specified test file, executing the generated command(s) after writing them into the test file. If there are intermediate steps to be taken, they're added to the test file as well.
|
|
25
25
|
|
|
26
26
|
```bash
|
|
27
27
|
thinking...
|
|
@@ -29,7 +29,7 @@ thinking...
|
|
|
29
29
|
To add an item to the shopping cart, we need to identify
|
|
30
30
|
the "Add to Cart" button from the screenshot. However,
|
|
31
31
|
since the screenshot shows a code editor, we need to
|
|
32
|
-
assume the button
|
|
32
|
+
assume the button isn't visible here. Let's proceed with
|
|
33
33
|
a generic approach:
|
|
34
34
|
|
|
35
35
|
commands:
|
|
@@ -74,19 +74,19 @@ thinking...
|
|
|
74
74
|
|
|
75
75
|
|
|
76
76
|
## Behavior
|
|
77
|
-
- TestDriver will generate an idea to complete the task, try to execute it, and add the steps to the test
|
|
77
|
+
- TestDriver will generate an idea to complete the task, try to execute it, and add the steps to the test file.
|
|
78
78
|
- The steps will be executed in the order generated.
|
|
79
|
-
- If the steps are successful, they
|
|
79
|
+
- If the steps are successful, they're added to the test file.
|
|
80
80
|
- If the steps fail, TestDriver will attempt to fall back to AI, undo the last step and provide an explanation of what went wrong. The AI will retry the step with new commands.
|
|
81
81
|
- If the commands are successful, the program will output the results and exit with code `0`.
|
|
82
82
|
- If any failures occur during the test, the program will output the errors and exit with code `1`.
|
|
83
83
|
|
|
84
84
|
## Protips
|
|
85
85
|
- Use descriptive prompts for higher success with generating test steps with `/explore`.
|
|
86
|
-
- Use `/explore` multiple times in succession to build a comprehensive test
|
|
86
|
+
- Use `/explore` multiple times in succession to build a comprehensive test file.
|
|
87
87
|
|
|
88
88
|
## Gotchas
|
|
89
89
|
- The AI will retry 3 times if it can't complete the task.
|
|
90
90
|
|
|
91
91
|
## Notes
|
|
92
|
-
- The `/explore` command is ideal for creating test
|
|
92
|
+
- The `/explore` command is ideal for creating test files in an interactive session.
|