testdriverai 5.2.1 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/test-install.yml +1 -1
- package/README.md +5 -11
- package/agent.js +135 -99
- package/docs/30x30.mdx +84 -0
- package/docs/action/browser.mdx +129 -0
- package/docs/action/os.mdx +157 -0
- package/docs/action/output.mdx +98 -0
- package/docs/action/performance.mdx +71 -0
- package/docs/action/prerun.mdx +80 -0
- package/docs/action/secrets.mdx +103 -0
- package/docs/action/setup.mdx +115 -0
- package/docs/bugs/jira.mdx +208 -0
- package/docs/cli/overview.mdx +65 -0
- package/docs/commands/assert.mdx +31 -0
- package/docs/commands/exec.mdx +42 -0
- package/docs/commands/focus-application.mdx +29 -0
- package/docs/commands/hover-image.mdx +32 -0
- package/docs/commands/hover-text.mdx +37 -0
- package/docs/commands/if.mdx +43 -0
- package/docs/commands/match-image.mdx +41 -0
- package/docs/commands/press-keys.mdx +30 -0
- package/docs/commands/run.mdx +30 -0
- package/docs/commands/scroll-until-image.mdx +33 -0
- package/docs/commands/scroll-until-text.mdx +37 -0
- package/docs/commands/scroll.mdx +33 -0
- package/docs/commands/type.mdx +29 -0
- package/docs/commands/wait-for-image.mdx +31 -0
- package/docs/commands/wait-for-text.mdx +35 -0
- package/docs/commands/wait.mdx +30 -0
- package/docs/docs.json +226 -0
- package/docs/exporting/playwright.mdx +159 -0
- package/docs/features/auto-healing.mdx +124 -0
- package/docs/features/cross-platform.mdx +106 -0
- package/docs/features/generation.mdx +180 -0
- package/docs/features/github.mdx +161 -0
- package/docs/features/parallel-testing.mdx +130 -0
- package/docs/features/reusable-snippets.mdx +124 -0
- package/docs/features/selectorless.mdx +62 -0
- package/docs/features/visual-assertions.mdx +123 -0
- package/docs/getting-started/ci.mdx +196 -0
- package/docs/getting-started/generating.mdx +210 -0
- package/docs/getting-started/running.mdx +67 -0
- package/docs/getting-started/setup.mdx +133 -0
- package/docs/getting-started/writing.mdx +99 -0
- package/docs/guide/assertions.mdx +195 -0
- package/docs/guide/authentication.mdx +150 -0
- package/docs/guide/code.mdx +169 -0
- package/docs/guide/locating.mdx +136 -0
- package/docs/guide/setup-teardown.mdx +161 -0
- package/docs/guide/variables.mdx +218 -0
- package/docs/guide/waiting.mdx +199 -0
- package/docs/importing/csv.mdx +196 -0
- package/docs/importing/gherkin.mdx +142 -0
- package/docs/importing/jira.mdx +172 -0
- package/docs/importing/testrail.mdx +161 -0
- package/docs/integrations/electron.mdx +152 -0
- package/docs/integrations/netlify.mdx +98 -0
- package/docs/integrations/vercel.mdx +177 -0
- package/docs/interactive/assert.mdx +51 -0
- package/docs/interactive/generate.mdx +41 -0
- package/docs/interactive/run.mdx +36 -0
- package/docs/interactive/save.mdx +53 -0
- package/docs/interactive/undo.mdx +47 -0
- package/docs/issues.mdx +9 -0
- package/docs/overview/comparison.mdx +82 -0
- package/docs/overview/faq.mdx +122 -0
- package/docs/overview/quickstart.mdx +66 -0
- package/docs/overview/what-is-testdriver.mdx +73 -0
- package/docs/quickstart.mdx +66 -0
- package/docs/reference/commands/scroll.mdx +0 -0
- package/docs/reference/interactive/assert.mdx +0 -0
- package/docs/security/action.mdx +62 -0
- package/docs/security/agent.mdx +62 -0
- package/docs/security/dashboard.mdx +0 -0
- package/docs/security/platform.mdx +54 -0
- package/docs/tutorials/advanced-test.mdx +79 -0
- package/docs/tutorials/basic-test.mdx +41 -0
- package/electron/icon.png +0 -0
- package/electron/overlay.html +7 -3
- package/electron/overlay.js +76 -16
- package/electron/tray-buffered.png +0 -0
- package/electron/tray.png +0 -0
- package/index.js +75 -34
- package/lib/commander.js +22 -1
- package/lib/commands.js +87 -19
- package/lib/config.js +10 -1
- package/lib/focus-application.js +30 -23
- package/lib/generator.js +58 -7
- package/lib/init.js +48 -19
- package/lib/ipc.js +50 -0
- package/lib/logger.js +19 -6
- package/lib/overlay.js +82 -36
- package/lib/parser.js +9 -7
- package/lib/resources/prerun.yaml +17 -0
- package/lib/sandbox.js +2 -3
- package/lib/sdk.js +0 -2
- package/lib/session.js +3 -1
- package/lib/speak.js +0 -2
- package/lib/subimage/opencv.js +0 -4
- package/lib/system.js +56 -39
- package/lib/upload-secrets.js +65 -0
- package/lib/validation.js +175 -0
- package/package.json +2 -1
- package/postinstall.js +0 -24
- package/lib/websockets.js +0 -85
- package/test.md +0 -8
- package/test.yml +0 -18
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "GitHub Action Setup Guide"
|
|
3
|
+
sidebarTitle: "Setup Guide"
|
|
4
|
+
description: "Learn how to set up and configure the TestDriver GitHub Action for automated cloud-based testing in your CI/CD workflows."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Overview
|
|
8
|
+
The TestDriver GitHub Action enables cloud-based testing using TestDriver's infrastructure. This guide will walk you through the steps to install and configure the GitHub Action for your repository.
|
|
9
|
+
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
## Step 1: Install the GitHub Action
|
|
13
|
+
TestDriver Cloud Testing is performed via the TestDriver GitHub Action. You can find and install the action from the GitHub Marketplace:
|
|
14
|
+
|
|
15
|
+
[**TestDriver.ai - GitHub Marketplace**](https://github.com/marketplace/actions/testdriver-ai)
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Step 2: Get Your API Key
|
|
20
|
+
To execute TestDriver actions on our virtual machines, you'll need an API key. Follow these steps to retrieve and configure your API key:
|
|
21
|
+
|
|
22
|
+
1. **Upgrade Your Account**: Ensure you have a paid TestDriver account.
|
|
23
|
+
2. **Log In**: Go to the **Team Page** in your TestDriver dashboard.
|
|
24
|
+
3. **Copy Your API Key**: Locate and copy your API key.
|
|
25
|
+
4. **Add the API Key as a GitHub Secret**:
|
|
26
|
+
- Navigate to your repository settings in GitHub.
|
|
27
|
+
- Add a new secret named `TESTDRIVER_API_KEY` and paste your API key.
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
## Step 3: Create Your Workflow
|
|
32
|
+
Now it's time to create your first TestDriver workflow. Add the following configuration to `.github/workflows/testdriver.yml`:
|
|
33
|
+
|
|
34
|
+
```yaml
|
|
35
|
+
name: TestDriver.ai
|
|
36
|
+
|
|
37
|
+
permissions:
|
|
38
|
+
actions: read
|
|
39
|
+
contents: read
|
|
40
|
+
statuses: write
|
|
41
|
+
pull-requests: write
|
|
42
|
+
|
|
43
|
+
on:
|
|
44
|
+
pull_request: # Run on every pull request event
|
|
45
|
+
schedule:
|
|
46
|
+
- cron: '0 * * * *' # Run every hour
|
|
47
|
+
push:
|
|
48
|
+
branches:
|
|
49
|
+
- main # Run on merge to the main branch
|
|
50
|
+
workflow_dispatch: # Manual trigger
|
|
51
|
+
|
|
52
|
+
jobs:
|
|
53
|
+
test:
|
|
54
|
+
name: "TestDriver"
|
|
55
|
+
runs-on: ubuntu-latest
|
|
56
|
+
steps:
|
|
57
|
+
- uses: dashcamio/testdriver@main
|
|
58
|
+
version: "v5.0.7"
|
|
59
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
60
|
+
os: linux
|
|
61
|
+
with:
|
|
62
|
+
prompt: |
|
|
63
|
+
1. /run testdriver.yaml
|
|
64
|
+
env:
|
|
65
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
66
|
+
FORCE_COLOR: "3"
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
### Key Points:
|
|
70
|
+
- **Trigger Conditions**: The `on` section defines when the workflow runs (e.g., pull requests, scheduled events, or manual triggers).
|
|
71
|
+
- **API Key**: The `key` field uses the `TESTDRIVER_API_KEY` secret for authentication.
|
|
72
|
+
- **Prompt**: The `prompt` field specifies the commands to execute. In this example, the `/run` command is used to execute a test file from the repository.
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
|
|
76
|
+
## Step 4: Deploy the Workflow
|
|
77
|
+
Save the workflow file, create a new branch, and push it to your repository. Then, create a pull request to trigger the workflow.
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
git checkout -b testdriver
|
|
81
|
+
git commit -am 'Add TestDriver GitHub Action'
|
|
82
|
+
git push origin testdriver
|
|
83
|
+
gh pr create --web
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
## How It Works
|
|
89
|
+
1. **Trigger**: The GitHub Action is triggered based on the conditions defined in the `on` section.
|
|
90
|
+
2. **Authentication**: The `key` value authenticates your account.
|
|
91
|
+
3. **VM Setup**: An ephemeral virtual machine is spawned on TestDriver's infrastructure.
|
|
92
|
+
4. **Code Cloning**: The current branch's code is cloned onto the VM.
|
|
93
|
+
5. **Dashcam Recording**: Dashcam begins recording the test execution.
|
|
94
|
+
6. **Prerun Script**: If supplied, a prerun shell script is executed.
|
|
95
|
+
7. **Prompt Execution**: The `prompt` is parsed as a Markdown list, and each item is executed sequentially.
|
|
96
|
+
8. **Test Summary**: TestDriver summarizes the test and sets the exit code based on the pass or fail state.
|
|
97
|
+
9. **Cleanup**: The VM is destroyed, and all data is wiped.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## Additional Features
|
|
102
|
+
- **Dynamic Prompts**: You can use commands like `/explore`, supply variables, or dynamically generate prompts from earlier steps.
|
|
103
|
+
- **Staging Workflows**: A common workflow involves waiting for staging to deploy before executing tests.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Output
|
|
108
|
+
For details on interpreting the output of the GitHub Action, refer to the [Action Output Documentation](../output).
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Notes
|
|
113
|
+
- The TestDriver GitHub Action is a powerful tool for automating cloud-based testing.
|
|
114
|
+
- Ensure your API key is securely stored as a GitHub secret.
|
|
115
|
+
- For advanced workflows, consider using prerun scripts or dynamic prompts to customize your tests.
|
|
@@ -0,0 +1,208 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Reporting Failed Tests to Jira"
|
|
3
|
+
sidebarTitle: "Jira"
|
|
4
|
+
description: "Step-by-step instructions to integrate TestDriver.ai with Jira for automated test failure tracking in CI/CD workflows."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# Automating Jira Ticket Creation for Test Failures Using TestDriver.ai and Jira GitHub Actions
|
|
8
|
+
|
|
9
|
+
This guide explains how to integrate the **TestDriver.ai GitHub Action** with the **Jira GitHub Action** to automatically create a Jira ticket whenever a TestDriver.ai test fails. This workflow ensures that test failures are tracked in Jira, enabling teams to address issues promptly.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Workflow Overview
|
|
14
|
+
|
|
15
|
+
1. **Run Tests with TestDriver.ai**: Use the TestDriver.ai GitHub Action to execute your test suite.
|
|
16
|
+
2. **Check for Test Failures**: Capture the test results and determine if any tests failed.
|
|
17
|
+
3. **Create a Jira Ticket**: Use the Jira GitHub Action to create a new ticket for each test failure, including relevant details such as the test name, failure reason, and logs.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Prerequisites
|
|
22
|
+
|
|
23
|
+
1. **TestDriver.ai API Key**: Store your API key as a GitHub secret (e.g., `TESTDRIVER_API_KEY`).
|
|
24
|
+
2. **Jira API Token**: Generate an API token from your Jira account and store it as a GitHub secret (e.g., `JIRA_API_TOKEN`).
|
|
25
|
+
3. **Jira Base URL**: Your Jira instance URL (e.g., `https://yourcompany.atlassian.net`).
|
|
26
|
+
4. **Jira Project Key**: The key of the Jira project where tickets will be created (e.g., `TEST`).
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## GitHub Actions Workflow
|
|
31
|
+
|
|
32
|
+
Here’s a complete workflow that integrates TestDriver.ai and Jira:
|
|
33
|
+
|
|
34
|
+
### Workflow File: `.github/workflows/testdriver-jira.yml````yaml
|
|
35
|
+
name: TestDriver.ai with Jira Integration
|
|
36
|
+
|
|
37
|
+
on:
|
|
38
|
+
push:
|
|
39
|
+
branches:
|
|
40
|
+
- main
|
|
41
|
+
pull_request:
|
|
42
|
+
workflow_dispatch:
|
|
43
|
+
|
|
44
|
+
jobs:
|
|
45
|
+
run-tests:
|
|
46
|
+
name: Run Tests with TestDriver.ai
|
|
47
|
+
runs-on: ubuntu-latest
|
|
48
|
+
steps:
|
|
49
|
+
- name: Check out repository
|
|
50
|
+
uses: actions/checkout@v2
|
|
51
|
+
|
|
52
|
+
- name: Run TestDriver.ai
|
|
53
|
+
id: testdriver
|
|
54
|
+
uses: testdriverai/action@main
|
|
55
|
+
with:
|
|
56
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
57
|
+
prompt: |
|
|
58
|
+
1. Run all tests in the testdriver directory
|
|
59
|
+
env:
|
|
60
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
61
|
+
FORCE_COLOR: "3"
|
|
62
|
+
|
|
63
|
+
- name: Check for Test Failures
|
|
64
|
+
id: check-failures
|
|
65
|
+
run: |
|
|
66
|
+
if [[ "${{ steps.testdriver.outputs.success }}" == "false" ]]; then
|
|
67
|
+
echo "TestDriver.ai tests failed."
|
|
68
|
+
echo "failure=true" >> $GITHUB_ENV
|
|
69
|
+
else
|
|
70
|
+
echo "All tests passed."
|
|
71
|
+
echo "failure=false" >> $GITHUB_ENV
|
|
72
|
+
fi
|
|
73
|
+
|
|
74
|
+
create-jira-ticket:
|
|
75
|
+
name: Create Jira Ticket for Test Failures
|
|
76
|
+
needs: run-tests
|
|
77
|
+
runs-on: ubuntu-latest
|
|
78
|
+
if: env.failure == 'true'
|
|
79
|
+
steps:
|
|
80
|
+
- name: Create Jira Ticket
|
|
81
|
+
uses: atlassian/gajira-create@v3
|
|
82
|
+
with:
|
|
83
|
+
url: ${{ secrets.JIRA_BASE_URL }}
|
|
84
|
+
user: ${{ secrets.JIRA_USERNAME }}
|
|
85
|
+
api-token: ${{ secrets.JIRA_API_TOKEN }}
|
|
86
|
+
project: TEST # Replace with your Jira project key
|
|
87
|
+
summary: "Test Failure: ${{ steps.testdriver.outputs.summary }}"
|
|
88
|
+
description: |
|
|
89
|
+
A test failure occurred during the TestDriver.ai workflow.
|
|
90
|
+
|
|
91
|
+
**Test Summary**:
|
|
92
|
+
${{ steps.testdriver.outputs.summary }}
|
|
93
|
+
|
|
94
|
+
**Failure Details**:
|
|
95
|
+
${{ steps.testdriver.outputs.markdown }}
|
|
96
|
+
|
|
97
|
+
Please investigate the issue and resolve it promptly.
|
|
98
|
+
issuetype: Bug
|
|
99
|
+
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## Workflow Steps Explained
|
|
105
|
+
|
|
106
|
+
### 1. **Run Tests with TestDriver.ai**
|
|
107
|
+
The `testdriverai/action@main` step runs your TestDriver.ai tests and captures the results. The `outputs.success` variable indicates whether the tests passed or failed.```yaml
|
|
108
|
+
- name: Run TestDriver.ai
|
|
109
|
+
id: testdriver
|
|
110
|
+
uses: testdriverai/action@main
|
|
111
|
+
with:
|
|
112
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
113
|
+
prompt: |
|
|
114
|
+
1. Run all tests in the testdriver directory
|
|
115
|
+
env:
|
|
116
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
117
|
+
FORCE_COLOR: "3"
|
|
118
|
+
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
### 2. **Check for Test Failures**
|
|
124
|
+
This step checks the `outputs.success` variable from the TestDriver.ai action. If the tests failed, it sets an environment variable (`failure=true`) to trigger the Jira ticket creation step.```yaml
|
|
125
|
+
- name: Check for Test Failures
|
|
126
|
+
id: check-failures
|
|
127
|
+
run: |
|
|
128
|
+
if [[ "${{ steps.testdriver.outputs.success }}" == "false" ]]; then
|
|
129
|
+
echo "TestDriver.ai tests failed."
|
|
130
|
+
echo "failure=true" >> $GITHUB_ENV
|
|
131
|
+
else
|
|
132
|
+
echo "All tests passed."
|
|
133
|
+
echo "failure=false" >> $GITHUB_ENV
|
|
134
|
+
fi
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
### 3. **Create a Jira Ticket**
|
|
141
|
+
If any tests failed, the `create-jira-ticket` job uses the `atlassian/gajira-create` action to create a new Jira ticket. The ticket includes:
|
|
142
|
+
- **Summary**: A brief description of the failure.
|
|
143
|
+
- **Description**: Detailed information about the failure, including the test summary and markdown output from TestDriver.ai.```yaml
|
|
144
|
+
- name: Create Jira Ticket
|
|
145
|
+
uses: atlassian/gajira-create@v3
|
|
146
|
+
with:
|
|
147
|
+
url: ${{ secrets.JIRA_BASE_URL }}
|
|
148
|
+
user: ${{ secrets.JIRA_USERNAME }}
|
|
149
|
+
api-token: ${{ secrets.JIRA_API_TOKEN }}
|
|
150
|
+
project: TEST # Replace with your Jira project key
|
|
151
|
+
summary: "Test Failure: ${{ steps.testdriver.outputs.summary }}"
|
|
152
|
+
description: |
|
|
153
|
+
A test failure occurred during the TestDriver.ai workflow.
|
|
154
|
+
|
|
155
|
+
**Test Summary**:
|
|
156
|
+
${{ steps.testdriver.outputs.summary }}
|
|
157
|
+
|
|
158
|
+
**Failure Details**:
|
|
159
|
+
${{ steps.testdriver.outputs.markdown }}
|
|
160
|
+
|
|
161
|
+
Please investigate the issue and resolve it promptly.
|
|
162
|
+
issuetype: Bug
|
|
163
|
+
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Secrets Configuration
|
|
169
|
+
|
|
170
|
+
Add the following secrets to your GitHub repository:
|
|
171
|
+
1. **`TESTDRIVER_API_KEY`**: Your TestDriver.ai API key.
|
|
172
|
+
2. **`JIRA_API_TOKEN`**: Your Jira API token.
|
|
173
|
+
3. **`JIRA_BASE_URL`**: Your Jira instance URL (e.g., `https://yourcompany.atlassian.net`).
|
|
174
|
+
4. **`JIRA_USERNAME`**: Your Jira account email.
|
|
175
|
+
|
|
176
|
+
---
|
|
177
|
+
|
|
178
|
+
## Example Jira Ticket
|
|
179
|
+
|
|
180
|
+
### Summary:
|
|
181
|
+
`Test Failure: Login Test Failed`
|
|
182
|
+
|
|
183
|
+
### Description:
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
```
|
|
187
|
+
A test failure occurred during the TestDriver.ai workflow.
|
|
188
|
+
|
|
189
|
+
**Test Summary**:
|
|
190
|
+
Login Test Failed
|
|
191
|
+
|
|
192
|
+
**Failure Details**:
|
|
193
|
+
- The login button was not clickable.
|
|
194
|
+
- The error message was not displayed.
|
|
195
|
+
|
|
196
|
+
Please investigate the issue and resolve it promptly.
|
|
197
|
+
|
|
198
|
+
```---
|
|
199
|
+
|
|
200
|
+
## Benefits of This Workflow
|
|
201
|
+
|
|
202
|
+
1. **Automated Issue Tracking**: Automatically creates Jira tickets for test failures, ensuring no issues are overlooked.
|
|
203
|
+
2. **Detailed Context**: Includes test summaries and failure details in the Jira ticket for easier debugging.
|
|
204
|
+
3. **Streamlined Workflow**: Integrates testing and issue tracking into a single automated pipeline.
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
By combining TestDriver.ai and Jira GitHub Actions, you can automate the process of tracking test failures, improving collaboration and ensuring faster resolution of issues.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
```mdx
|
|
2
|
+
# Command: `testdriverai`
|
|
3
|
+
|
|
4
|
+
## Description
|
|
5
|
+
The `testdriverai` CLI is the primary interface for running, managing, and debugging TestDriver.ai scripts. It provides commands to execute tests, validate configurations, and interact with the TestDriver.ai framework.
|
|
6
|
+
|
|
7
|
+
## Usage
|
|
8
|
+
```bash
|
|
9
|
+
testdriverai <command> [options]
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## Available Commands
|
|
13
|
+
| Command | Description |
|
|
14
|
+
|---------------|-----------------------------------------------------------------------------|
|
|
15
|
+
| `run` | Executes a TestDriver.ai script. |
|
|
16
|
+
| `validate` | Validates the syntax and structure of a TestDriver.ai script. |
|
|
17
|
+
| `init` | Initializes a new TestDriver.ai project in the current directory. |
|
|
18
|
+
| `version` | Displays the current version of the TestDriver.ai CLI. |
|
|
19
|
+
| `help` | Displays help information for the CLI or a specific command. |
|
|
20
|
+
|
|
21
|
+
## Example Usage
|
|
22
|
+
|
|
23
|
+
### Running a Test Script
|
|
24
|
+
```bash
|
|
25
|
+
testdriverai run path/to/testdriver.yaml
|
|
26
|
+
```
|
|
27
|
+
This command runs the specified TestDriver.ai script, executing all steps defined in the file.
|
|
28
|
+
|
|
29
|
+
### Validating a Script
|
|
30
|
+
```bash
|
|
31
|
+
testdriverai validate path/to/testdriver.yaml
|
|
32
|
+
```
|
|
33
|
+
This command checks the syntax and structure of the script to ensure it is valid before execution.
|
|
34
|
+
|
|
35
|
+
### Initializing a New Project
|
|
36
|
+
```bash
|
|
37
|
+
testdriverai init
|
|
38
|
+
```
|
|
39
|
+
This command sets up a new TestDriver.ai project in the current directory, creating the necessary folder structure and configuration files.
|
|
40
|
+
|
|
41
|
+
### Checking the Version
|
|
42
|
+
```bash
|
|
43
|
+
testdriverai version
|
|
44
|
+
```
|
|
45
|
+
This command displays the installed version of the TestDriver.ai CLI.
|
|
46
|
+
|
|
47
|
+
### Getting Help
|
|
48
|
+
```bash
|
|
49
|
+
testdriverai help
|
|
50
|
+
```
|
|
51
|
+
This command displays general help information. To get help for a specific command, use:
|
|
52
|
+
```bash
|
|
53
|
+
testdriverai help <command>
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Protips
|
|
57
|
+
- Use `testdriverai validate` before running a script to catch errors early.
|
|
58
|
+
- Combine `testdriverai run` with CI/CD pipelines to automate test execution.
|
|
59
|
+
- Use the `--debug` flag with `run` to enable detailed logging for troubleshooting.
|
|
60
|
+
|
|
61
|
+
## Notes
|
|
62
|
+
- The `testdriverai` CLI is cross-platform and works on macOS, Windows, and Linux.
|
|
63
|
+
- Ensure you have the required dependencies installed and configured before running scripts.
|
|
64
|
+
- For more advanced usage, refer to the official TestDriver.ai documentation.
|
|
65
|
+
```
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "assert"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `assert` command is used to validate that a specific condition is true. It ensures that a task was completed successfully by checking the screen for the expected outcome. If the condition is not met, the test will fail.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|------------|-----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `expect` | `string` | The condition to check. This should describe what you expect to see on the screen. |
|
|
12
|
+
| `async` | `boolean` | (Optional) If set to `true`, the test will continue without waiting for the assertion to pass. Default is `false`. |
|
|
13
|
+
|
|
14
|
+
## Example Usage
|
|
15
|
+
```yaml
|
|
16
|
+
command: assert
|
|
17
|
+
expect: the video is playing
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
### Example with `async`
|
|
21
|
+
|
|
22
|
+
```yaml
|
|
23
|
+
command: assert
|
|
24
|
+
expect: There is no error message
|
|
25
|
+
async: true
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Notes
|
|
29
|
+
- Use `async: true` to speed up tests by allowing non-blocking assertions. However, the test will still fail if the condition is not met.
|
|
30
|
+
- Assertions should be used sparingly, as too many can slow down the test execution.
|
|
31
|
+
- Ensure the `expect` string clearly describes the condition to avoid ambiguity.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "exec"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `exec` command allows you to execute custom NodeJS scripts within your TestDriver.ai tests. This is useful for tasks like generating dynamic data, interacting with APIs, or performing custom logic during a test. The output of the script can be stored in a variable for use in subsequent steps.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|------------|-----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `js` | `string` | The NodeJS script to execute. The script must define the output as `result`. |
|
|
12
|
+
| `output` | `string` | The variable name to store the result of the script. This variable can be accessed as `${OUTPUT.<var>}` in future steps. |
|
|
13
|
+
|
|
14
|
+
## Example Usage
|
|
15
|
+
```yaml
|
|
16
|
+
version: 5.1.0
|
|
17
|
+
session: 67e57d614dc25283aa0872a9
|
|
18
|
+
steps:
|
|
19
|
+
- prompt: Log in with the totp
|
|
20
|
+
commands:
|
|
21
|
+
- command: exec
|
|
22
|
+
output: totp
|
|
23
|
+
js: |
|
|
24
|
+
const { TOTP } = require("totp-generator");
|
|
25
|
+
let otp = TOTP.generate("JBSWY3DPEB3W64TMMQQQ").otp;
|
|
26
|
+
result = otp;
|
|
27
|
+
- command: type
|
|
28
|
+
text: ${OUTPUT.totp}
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Protips
|
|
32
|
+
- Do not overwrite the `result` variable in your script, as it is used to store the output.
|
|
33
|
+
- Ensure that all required NPM packages are installed locally using `npm install`. If using GitHub Actions, include the installation in the `prerun` step.
|
|
34
|
+
- NodeJS code executes in the same context as the calling process on the host machine, using the [VM](https://nodejs.org/api/vm.html) module internally.
|
|
35
|
+
|
|
36
|
+
## Gotchas
|
|
37
|
+
- This command is only supported on hosted Windows runners. Hosted Linux runners do not yet support the `prerun` step.
|
|
38
|
+
- Ensure your NodeJS environment is properly configured with all necessary dependencies.
|
|
39
|
+
|
|
40
|
+
## Notes
|
|
41
|
+
- The `exec` command is available starting from TestDriver.ai version `5.1.0`.
|
|
42
|
+
- Use this command to integrate custom logic or external libraries into your tests.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "focus-application"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `focus-application` command is used to bring a specific application window to the foreground. This ensures that subsequent commands interact with the correct application during a test.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|------------|-----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `name` | `string` | The name of the application to focus. This should match the application's display name. |
|
|
12
|
+
|
|
13
|
+
## Example Usage
|
|
14
|
+
```yaml
|
|
15
|
+
command: focus-application
|
|
16
|
+
name: Google Chrome
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Protips
|
|
20
|
+
- Ensure the application name matches the exact name displayed in your operating system's task manager or application switcher.
|
|
21
|
+
- Use this command at the start of a test or before interacting with an application to avoid focus-related issues.
|
|
22
|
+
|
|
23
|
+
## Gotchas
|
|
24
|
+
- If the specified application is not running, the command will fail. Ensure the application is open before using this command.
|
|
25
|
+
- On macOS, the application name is case-sensitive.
|
|
26
|
+
|
|
27
|
+
## Notes
|
|
28
|
+
- The `focus-application` command is useful for multi-application workflows where you need to switch between different apps during a test.
|
|
29
|
+
- This command is supported on both Windows and macOS environments.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "hover-image"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `hover-image` command is used to locate an image on the screen based on a description and perform an action on it. This can include hovering, clicking, right-clicking, or double-clicking the image.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|---------------|----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `description` | `string` | A description of the image and what it represents. Do not include the image itself here. |
|
|
12
|
+
| `action` | `string` | The action to take when the image is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. |
|
|
13
|
+
|
|
14
|
+
## Example Usage
|
|
15
|
+
```yaml
|
|
16
|
+
command: hover-image
|
|
17
|
+
description: search icon in the webpage content
|
|
18
|
+
action: click
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## Protips
|
|
22
|
+
- Use clear and concise descriptions for the image to improve detection accuracy.
|
|
23
|
+
- Ensure the action specified matches the intended interaction with the image.
|
|
24
|
+
- If the hover-image command fails to locate an image based on the provided description, consider using the match-image command instead. The match-image command is specifically designed for scenarios where precise image matching is required, such as when the visual element cannot be reliably described with text. It works by directly comparing the image on the screen with a reference image file.
|
|
25
|
+
|
|
26
|
+
## Gotchas
|
|
27
|
+
- If the image cannot be located based on the description, the command will fail.
|
|
28
|
+
- Ensure the screen resolution and scaling settings are consistent to avoid detection issues.
|
|
29
|
+
|
|
30
|
+
## Notes
|
|
31
|
+
- The `hover-image` command is useful for interacting with visual elements that cannot be identified using text-based selectors.
|
|
32
|
+
- Supported actions allow flexibility in how the image is interacted with during the test.
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "hover-text"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `hover-text` command is used to locate text on the screen based on a description and perform an action on it. This can include hovering, clicking, right-clicking, or double-clicking the text.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|---------------|----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `text` | `string` | The text to find on the screen. The longer and more unique, the better. |
|
|
12
|
+
| `description` | `string` | A description of the text and what it represents. The actual text itself should not be included here. |
|
|
13
|
+
| `action` | `string` | The action to take when the text is found. Available actions are: `click`, `right-click`, `double-click`, `hover`. |
|
|
14
|
+
| `method` | `enum` | The matching algorithm to use. Possible values are `turbo` (default) and `ai`. |
|
|
15
|
+
|
|
16
|
+
## Example Usage
|
|
17
|
+
```yaml
|
|
18
|
+
command: hover-text
|
|
19
|
+
text: Sign Up
|
|
20
|
+
description: link in the header
|
|
21
|
+
action: click
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Protips
|
|
25
|
+
- Use unique and specific text to improve detection accuracy.
|
|
26
|
+
- Choose the appropriate `method` for your use case:
|
|
27
|
+
- `turbo`: Faster and more efficient for most scenarios.
|
|
28
|
+
- `ai`: More robust for complex or ambiguous text matching.
|
|
29
|
+
- Ensure the action specified matches the intended interaction with the text.
|
|
30
|
+
|
|
31
|
+
## Gotchas
|
|
32
|
+
- If the text cannot be located, the command will fail. Ensure the text is visible on the screen and matches exactly.
|
|
33
|
+
- Variations in font size, style, or screen resolution may affect detection accuracy.
|
|
34
|
+
|
|
35
|
+
## Notes
|
|
36
|
+
- The `hover-text` command is ideal for interacting with textual elements that cannot be identified using other selectors.
|
|
37
|
+
- Supported actions allow flexibility in how the text is interacted with during the test.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "if"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `if` command is used to conditionally execute a set of commands based on whether a specified condition evaluates to true or false. If the condition is true, the commands in the `then` block are executed. Otherwise, the commands in the `else` block are executed.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|-------------|-------------------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `condition` | `string` | The condition to evaluate. |
|
|
12
|
+
| `then` | `list of commands`| The commands to run if the condition is true. |
|
|
13
|
+
| `else` | `list of commands`| The commands to run if the condition is false. |
|
|
14
|
+
|
|
15
|
+
## Example Usage
|
|
16
|
+
```yaml
|
|
17
|
+
command: if
|
|
18
|
+
condition: the text "Accept Cookies" is visible
|
|
19
|
+
then:
|
|
20
|
+
- command: hover-text
|
|
21
|
+
text: Accept Cookies
|
|
22
|
+
description: cookie banner button
|
|
23
|
+
action: click
|
|
24
|
+
else:
|
|
25
|
+
- command: hover-text
|
|
26
|
+
text: Sign Up
|
|
27
|
+
description: link in the header
|
|
28
|
+
action: click
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Protips
|
|
32
|
+
- Ensure the `condition` is clearly defined and evaluates correctly to avoid unexpected behavior.
|
|
33
|
+
- Use descriptive `then` and `else` blocks to handle both outcomes effectively.
|
|
34
|
+
- Combine the `if` command with other commands like `hover-text` or `focus-application` to create dynamic workflows.
|
|
35
|
+
|
|
36
|
+
## Gotchas
|
|
37
|
+
- If the `condition` is invalid or cannot be evaluated, the command will fail.
|
|
38
|
+
- Ensure all commands in the `then` and `else` blocks are valid and properly formatted.
|
|
39
|
+
|
|
40
|
+
## Notes
|
|
41
|
+
- The `if` command is useful for creating conditional logic in your tests, allowing for more dynamic and flexible workflows.
|
|
42
|
+
- Both the `then` and `else` blocks are optional, but at least one must be provided.
|
|
43
|
+
```
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "match-image"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `match-image` command is used to locate an image on the screen by matching it with a reference image file and performing an action (e.g., click or hover) at its center. This command is particularly useful for interacting with elements that the AI has difficulty locating using descriptions or other methods.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|----------------|----------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `path` | `string` | The relative path to the image file that needs to be matched on the screen. Do not include `testdriver/screenshots/*/` in the path. |
|
|
12
|
+
| `action` | `string` | The action to perform when the image is found. Available actions are: `click` or `hover`. The action will be performed at the center of the matched image. |
|
|
13
|
+
|
|
14
|
+
## Example Usage
|
|
15
|
+
```yaml
|
|
16
|
+
command: match-image
|
|
17
|
+
path: button.png
|
|
18
|
+
action: click
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
## How It Works
|
|
22
|
+
- The `match-image` command takes a screenshot of the desktop and searches for the location of the reference image within the screenshot.
|
|
23
|
+
- The matching logic looks for the most similar image within the screenshot, not an exact match. If the similarity is below ~80%, it will search additional scales. If no match is found, the command will fail.
|
|
24
|
+
- Screenshots should be stored in the `testdriver/screenshots/(mac/linux/windows)/` directory. TestDriver dynamically resolves the correct image based on the current platform.
|
|
25
|
+
|
|
26
|
+
## Protips
|
|
27
|
+
- To create high-quality screenshots for matching:
|
|
28
|
+
- Download the video of the test and open it at "full" or "actual" size on your computer.
|
|
29
|
+
- Use a screenshot tool like Cleanshot X to capture the target element.
|
|
30
|
+
- Center the clickable element as much as possible within the screenshot.
|
|
31
|
+
- Ensure the image file is clear and free of unnecessary visual noise to improve matching accuracy.
|
|
32
|
+
|
|
33
|
+
## Gotchas
|
|
34
|
+
- If the image match is below ~80% similarity, the command will fail.
|
|
35
|
+
- Variations in screen resolution, scaling settings, or platform-specific UI differences may affect matching accuracy.
|
|
36
|
+
- Ensure the image file is stored in the correct directory structure (`testdriver/screenshots/(mac/linux/windows)/`) for dynamic resolution.
|
|
37
|
+
|
|
38
|
+
## Notes
|
|
39
|
+
- The `match-image` command is ideal for interacting with visual elements that cannot be reliably described or located using other commands like `hover-image`.
|
|
40
|
+
- This command supports flexible scaling to account for minor differences in image size or resolution.
|
|
41
|
+
- Use this command as a fallback when other methods fail to locate the desired element.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "press-keys"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
## Description
|
|
6
|
+
The `press-keys` command is used to simulate typing a keyboard combination. This is useful for triggering shortcuts or interacting with applications via keyboard input.
|
|
7
|
+
|
|
8
|
+
## Arguments
|
|
9
|
+
| Argument | Type | Description |
|
|
10
|
+
|----------|---------------------|-----------------------------------------------------------------------------|
|
|
11
|
+
| `keys` | `yml array of strings` | A list of keys to press together. Keys are automatically remapped for different operating systems. |
|
|
12
|
+
|
|
13
|
+
## Example Usage
|
|
14
|
+
```yaml
|
|
15
|
+
command: press-keys
|
|
16
|
+
keys: [command, space]
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Protips
|
|
20
|
+
- Use this command to trigger system-wide or application-specific shortcuts, such as opening Spotlight on macOS or the Start menu on Windows.
|
|
21
|
+
- Ensure the keys specified are valid and supported by the operating system or application.
|
|
22
|
+
|
|
23
|
+
## Gotchas
|
|
24
|
+
- Key mappings may differ between operating systems. For example, `command` on macOS is remapped to `ctrl` on Windows.
|
|
25
|
+
- Ensure the application or system is in focus when using this command to avoid unexpected behavior.
|
|
26
|
+
|
|
27
|
+
## Notes
|
|
28
|
+
- The `press-keys` command is ideal for automating workflows that rely on keyboard shortcuts.
|
|
29
|
+
- This command supports cross-platform key remapping to ensure compatibility across macOS, Windows, and Linux.
|
|
30
|
+
```
|