testdriverai 5.2.2 → 5.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/test-install.yml +1 -1
- package/README.md +5 -11
- package/agent.js +135 -99
- package/docs/30x30.mdx +84 -0
- package/docs/action/browser.mdx +129 -0
- package/docs/action/os.mdx +157 -0
- package/docs/action/output.mdx +98 -0
- package/docs/action/performance.mdx +71 -0
- package/docs/action/prerun.mdx +80 -0
- package/docs/action/secrets.mdx +103 -0
- package/docs/action/setup.mdx +115 -0
- package/docs/bugs/jira.mdx +208 -0
- package/docs/cli/overview.mdx +65 -0
- package/docs/commands/assert.mdx +31 -0
- package/docs/commands/exec.mdx +42 -0
- package/docs/commands/focus-application.mdx +29 -0
- package/docs/commands/hover-image.mdx +32 -0
- package/docs/commands/hover-text.mdx +37 -0
- package/docs/commands/if.mdx +43 -0
- package/docs/commands/match-image.mdx +41 -0
- package/docs/commands/press-keys.mdx +30 -0
- package/docs/commands/run.mdx +30 -0
- package/docs/commands/scroll-until-image.mdx +33 -0
- package/docs/commands/scroll-until-text.mdx +37 -0
- package/docs/commands/scroll.mdx +33 -0
- package/docs/commands/type.mdx +29 -0
- package/docs/commands/wait-for-image.mdx +31 -0
- package/docs/commands/wait-for-text.mdx +35 -0
- package/docs/commands/wait.mdx +30 -0
- package/docs/docs.json +226 -0
- package/docs/exporting/playwright.mdx +159 -0
- package/docs/features/auto-healing.mdx +124 -0
- package/docs/features/cross-platform.mdx +106 -0
- package/docs/features/generation.mdx +180 -0
- package/docs/features/github.mdx +161 -0
- package/docs/features/parallel-testing.mdx +130 -0
- package/docs/features/reusable-snippets.mdx +124 -0
- package/docs/features/selectorless.mdx +62 -0
- package/docs/features/visual-assertions.mdx +123 -0
- package/docs/getting-started/ci.mdx +196 -0
- package/docs/getting-started/generating.mdx +210 -0
- package/docs/getting-started/running.mdx +67 -0
- package/docs/getting-started/setup.mdx +133 -0
- package/docs/getting-started/writing.mdx +99 -0
- package/docs/guide/assertions.mdx +195 -0
- package/docs/guide/authentication.mdx +150 -0
- package/docs/guide/code.mdx +169 -0
- package/docs/guide/locating.mdx +136 -0
- package/docs/guide/setup-teardown.mdx +161 -0
- package/docs/guide/variables.mdx +218 -0
- package/docs/guide/waiting.mdx +199 -0
- package/docs/importing/csv.mdx +196 -0
- package/docs/importing/gherkin.mdx +142 -0
- package/docs/importing/jira.mdx +172 -0
- package/docs/importing/testrail.mdx +161 -0
- package/docs/integrations/electron.mdx +152 -0
- package/docs/integrations/netlify.mdx +98 -0
- package/docs/integrations/vercel.mdx +177 -0
- package/docs/interactive/assert.mdx +51 -0
- package/docs/interactive/generate.mdx +41 -0
- package/docs/interactive/run.mdx +36 -0
- package/docs/interactive/save.mdx +53 -0
- package/docs/interactive/undo.mdx +47 -0
- package/docs/issues.mdx +9 -0
- package/docs/overview/comparison.mdx +82 -0
- package/docs/overview/faq.mdx +122 -0
- package/docs/overview/quickstart.mdx +66 -0
- package/docs/overview/what-is-testdriver.mdx +73 -0
- package/docs/quickstart.mdx +66 -0
- package/docs/reference/commands/scroll.mdx +0 -0
- package/docs/reference/interactive/assert.mdx +0 -0
- package/docs/security/action.mdx +62 -0
- package/docs/security/agent.mdx +62 -0
- package/docs/security/dashboard.mdx +0 -0
- package/docs/security/platform.mdx +54 -0
- package/docs/tutorials/advanced-test.mdx +79 -0
- package/docs/tutorials/basic-test.mdx +41 -0
- package/electron/icon.png +0 -0
- package/electron/overlay.html +7 -3
- package/electron/overlay.js +75 -15
- package/electron/tray-buffered.png +0 -0
- package/electron/tray.png +0 -0
- package/index.js +75 -34
- package/lib/commander.js +22 -1
- package/lib/commands.js +87 -19
- package/lib/config.js +10 -1
- package/lib/focus-application.js +30 -23
- package/lib/generator.js +58 -7
- package/lib/init.js +48 -19
- package/lib/ipc.js +50 -0
- package/lib/logger.js +19 -6
- package/lib/overlay.js +82 -36
- package/lib/parser.js +9 -7
- package/lib/resources/prerun.yaml +17 -0
- package/lib/sandbox.js +2 -3
- package/lib/sdk.js +0 -2
- package/lib/session.js +3 -1
- package/lib/speak.js +0 -2
- package/lib/subimage/opencv.js +0 -4
- package/lib/system.js +56 -39
- package/lib/upload-secrets.js +65 -0
- package/lib/validation.js +175 -0
- package/package.json +2 -1
- package/postinstall.js +0 -24
- package/lib/websockets.js +0 -85
- package/test.md +0 -8
- package/test.yml +0 -18
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Auto-Healing in TestDriver.ai"
|
|
3
|
+
sidebarTitle: "Auto-Healing"
|
|
4
|
+
description: "Learn how TestDriver.ai's auto-healing feature keeps your tests resilient and up-to-date with minimal manual effort."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Auto-healing is a powerful feature in TestDriver.ai that ensures your tests remain resilient even when the application under test changes. When text, UI elements, or other visual cues are modified, TestDriver.ai can detect the failure, adapt to the changes, and automatically update the test. It then opens a pull request (PR) with the updated test, ensuring your test suite stays up-to-date with minimal manual intervention.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## How Auto-Healing Works
|
|
12
|
+
|
|
13
|
+
1. **Test Execution**: TestDriver.ai runs your test suite as usual.
|
|
14
|
+
2. **Failure Detection**: If a test fails due to a change in the application (e.g., text or UI updates), the AI identifies the failure point.
|
|
15
|
+
3. **Recovery Attempt**: The AI uses its adaptive capabilities to locate the updated element or text and retries the action.
|
|
16
|
+
4. **Test Update**: If the recovery is successful, TestDriver.ai updates the test script with the new element or text.
|
|
17
|
+
5. **Pull Request Creation**: The updated test script is committed to a new branch, and a pull request is automatically opened in your repository for review.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Benefits of Auto-Healing
|
|
22
|
+
|
|
23
|
+
- **Reduced Maintenance**: Eliminates the need for manual updates to tests when minor UI changes occur.
|
|
24
|
+
- **Increased Test Resilience**: Ensures tests adapt to changes without breaking.
|
|
25
|
+
- **Continuous Integration**: Keeps your test suite aligned with the latest application changes.
|
|
26
|
+
- **Developer Collaboration**: Automatically opens a PR, allowing developers to review and approve updates.
|
|
27
|
+
|
|
28
|
+
---
|
|
29
|
+
|
|
30
|
+
## Example Workflow
|
|
31
|
+
|
|
32
|
+
### Scenario: Button Text Changes from "Submit" to "Send"
|
|
33
|
+
|
|
34
|
+
1. **Initial Test**:
|
|
35
|
+
|
|
36
|
+
```yaml
|
|
37
|
+
- command: hover-text
|
|
38
|
+
text: Submit
|
|
39
|
+
description: Submit button in the form
|
|
40
|
+
action: click
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
2. **Application Update**:
|
|
44
|
+
- The button text is changed from "Submit" to "Send" in the application.
|
|
45
|
+
|
|
46
|
+
3. **Test Execution**:
|
|
47
|
+
- TestDriver.ai runs the test and fails to find the "Submit" button.
|
|
48
|
+
|
|
49
|
+
4. **Auto-Healing**:
|
|
50
|
+
- The AI detects the failure and searches for a similar element.
|
|
51
|
+
- It identifies the "Send" button as the updated element and retries the action.
|
|
52
|
+
|
|
53
|
+
5. **Test Update**:
|
|
54
|
+
- The test script is updated to reflect the new button text:
|
|
55
|
+
|
|
56
|
+
```yaml
|
|
57
|
+
- command: hover-text
|
|
58
|
+
text: Send
|
|
59
|
+
description: Submit button in the form
|
|
60
|
+
action: click
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
6. **Pull Request Creation**:
|
|
64
|
+
- TestDriver.ai commits the updated test to a new branch and opens a PR:
|
|
65
|
+
- **Branch Name**: `auto-heal-update-submit-to-send`
|
|
66
|
+
- **PR Title**: `Auto-Healed Test: Updated "Submit" to "Send"`
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Example GitHub Action for Auto-Healing
|
|
71
|
+
|
|
72
|
+
Here’s how you can configure a GitHub Action to enable auto-healing and PR creation:
|
|
73
|
+
|
|
74
|
+
```yaml
|
|
75
|
+
name: TestDriver.ai Auto-Healing
|
|
76
|
+
|
|
77
|
+
on:
|
|
78
|
+
push:
|
|
79
|
+
branches:
|
|
80
|
+
- main
|
|
81
|
+
pull_request:
|
|
82
|
+
workflow_dispatch:
|
|
83
|
+
|
|
84
|
+
jobs:
|
|
85
|
+
test:
|
|
86
|
+
name: "Run Tests with Auto-Healing"
|
|
87
|
+
runs-on: ubuntu-latest
|
|
88
|
+
steps:
|
|
89
|
+
- name: Check out repository
|
|
90
|
+
uses: actions/checkout@v2
|
|
91
|
+
|
|
92
|
+
- name: Run TestDriver.ai
|
|
93
|
+
uses: testdriverai/action@main
|
|
94
|
+
with:
|
|
95
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
96
|
+
prompt: |
|
|
97
|
+
1. /run testdriver/onboarding.yaml
|
|
98
|
+
create-pr: true
|
|
99
|
+
pr-title: "Auto-Healed Test Updates"
|
|
100
|
+
pr-base: main
|
|
101
|
+
pr-branch: auto-heal-updates
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Best Practices for Auto-Healing
|
|
107
|
+
|
|
108
|
+
1. **Enable PR Reviews**: Always review auto-healed PRs to ensure the updates align with the intended application behavior.
|
|
109
|
+
2. **Use Descriptive Descriptions**: Provide clear descriptions for elements in your tests to help the AI identify changes more effectively.
|
|
110
|
+
3. **Run Tests Frequently**: Integrate TestDriver.ai into your CI/CD pipeline to catch and heal changes as soon as they occur.
|
|
111
|
+
4. **Monitor Auto-Healing Logs**: Review logs and screenshots provided by TestDriver.ai to understand why a test was updated.
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## Limitations of Auto-Healing
|
|
116
|
+
|
|
117
|
+
- **Major UI Overhauls**: Auto-healing is best suited for minor changes (e.g., text updates, small layout adjustments). Significant UI changes may still require manual intervention.
|
|
118
|
+
- **Ambiguous Changes**: If multiple elements match the updated criteria, the AI may require additional context to make the correct decision.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## Conclusion
|
|
123
|
+
|
|
124
|
+
Auto-healing in TestDriver.ai reduces the burden of test maintenance by automatically adapting to changes in your application. By leveraging this feature, you can ensure your test suite remains robust and up-to-date, allowing your team to focus on delivering high-quality software.
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Cross-Platform Testing with TestDriver.ai"
|
|
3
|
+
sidebarTitle: "Cross-Platform Testing"
|
|
4
|
+
description: "Discover how TestDriver.ai enables seamless cross-platform testing across Windows, Linux, and Mac operating systems."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
TestDriver.ai provides robust cross-platform support, enabling you to run tests seamlessly on **Windows**, **Linux**, and **Mac** operating systems. This flexibility ensures that your applications are tested in environments that match your users' setups.
|
|
8
|
+
|
|
9
|
+
## Specifying the Operating System
|
|
10
|
+
|
|
11
|
+
When using the TestDriver GitHub Action, you can specify the target operating system by setting the `os` field. Supported values are:
|
|
12
|
+
|
|
13
|
+
- `windows`
|
|
14
|
+
- `linux`
|
|
15
|
+
- `mac`
|
|
16
|
+
|
|
17
|
+
Example:
|
|
18
|
+
|
|
19
|
+
```yaml
|
|
20
|
+
jobs:
|
|
21
|
+
test:
|
|
22
|
+
runs-on: ubuntu-latest
|
|
23
|
+
steps:
|
|
24
|
+
- uses: testdriverai/action@main
|
|
25
|
+
with:
|
|
26
|
+
os: linux
|
|
27
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
28
|
+
prompt: |
|
|
29
|
+
1. Open the browser
|
|
30
|
+
2. Navigate to example.com
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Using the `prerun` Field
|
|
35
|
+
|
|
36
|
+
The `prerun` field allows you to execute a shell script before running your tests. This is particularly useful for:
|
|
37
|
+
|
|
38
|
+
1. **Downloading and Installing Applications**
|
|
39
|
+
Example: Downloading and installing a custom app or dependency.
|
|
40
|
+
|
|
41
|
+
2. **Installing Chrome Extensions**
|
|
42
|
+
Example: Adding a browser extension required for testing.
|
|
43
|
+
|
|
44
|
+
3. **Launching a Browser**
|
|
45
|
+
Example: Starting a browser with specific arguments or configurations.
|
|
46
|
+
|
|
47
|
+
### Example `prerun` Script
|
|
48
|
+
|
|
49
|
+
```yaml
|
|
50
|
+
jobs:
|
|
51
|
+
test:
|
|
52
|
+
runs-on: macos-latest
|
|
53
|
+
steps:
|
|
54
|
+
- uses: testdriverai/action@main
|
|
55
|
+
with:
|
|
56
|
+
os: mac
|
|
57
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
58
|
+
prompt: |
|
|
59
|
+
1. Open the browser
|
|
60
|
+
2. Navigate to example.com
|
|
61
|
+
prerun: |
|
|
62
|
+
# Download and install an app
|
|
63
|
+
curl -o app.dmg https://example.com/app.dmg
|
|
64
|
+
hdiutil attach app.dmg
|
|
65
|
+
cp -R /Volumes/App/App.app /Applications/
|
|
66
|
+
|
|
67
|
+
# Install a Chrome extension
|
|
68
|
+
npm install dashcam-chrome
|
|
69
|
+
open -a "Google Chrome" --args --load-extension=$(pwd)/node_modules/dashcam-chrome/build
|
|
70
|
+
|
|
71
|
+
# Launch the browser
|
|
72
|
+
open -a "Google Chrome"
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
## Write Once, Run Anywhere
|
|
77
|
+
|
|
78
|
+
TestDriver.ai uses **selectorless testing**, meaning tests are written in a generic format that works across all platforms. The AI dynamically adapts to the environment, ensuring that the same test can run on Windows, Linux, and Mac without modification.
|
|
79
|
+
|
|
80
|
+
### Benefits of Selectorless Testing
|
|
81
|
+
|
|
82
|
+
- **Reduced Maintenance**: No need to update selectors when UI changes.
|
|
83
|
+
- **Cross-Platform Compatibility**: Tests are not tied to platform-specific configurations.
|
|
84
|
+
- **Ease of Use**: Write high-level, natural language prompts, and let TestDriver handle the rest.
|
|
85
|
+
|
|
86
|
+
### Example Test
|
|
87
|
+
|
|
88
|
+
```yaml
|
|
89
|
+
version: 4.2.18
|
|
90
|
+
steps:
|
|
91
|
+
- prompt: Open the browser and search for "TestDriver.ai"
|
|
92
|
+
commands:
|
|
93
|
+
- command: focus-application
|
|
94
|
+
name: Google Chrome
|
|
95
|
+
- command: hover-text
|
|
96
|
+
text: Search Google or type a URL
|
|
97
|
+
description: main search bar
|
|
98
|
+
action: click
|
|
99
|
+
- command: type
|
|
100
|
+
text: TestDriver.ai
|
|
101
|
+
- command: press-keys
|
|
102
|
+
keys: [enter]
|
|
103
|
+
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
This test will work on any supported operating system without modification.
|
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Test Generation with TestDriver.ai"
|
|
3
|
+
sidebarTitle: "Test Generation"
|
|
4
|
+
description: "Simplify test creation with AI-driven exploratory testing."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
TestDriver.ai provides a powerful and flexible framework for generating tests from various sources, including:
|
|
8
|
+
- **TestRail test cases**
|
|
9
|
+
- **CSV files**
|
|
10
|
+
- **Jira tickets**
|
|
11
|
+
- **Gherkin scenarios**
|
|
12
|
+
|
|
13
|
+
Additionally, TestDriver.ai can autonomously generate its own tests by recursively mapping the UI and exploring workflows.
|
|
14
|
+
|
|
15
|
+
## **Test Generation from CSV Files**
|
|
16
|
+
|
|
17
|
+
### Workflow Overview
|
|
18
|
+
CSV files are a simple and structured way to define test cases. Each row in the CSV file can represent a test case, with columns specifying the test steps, expected outcomes, or other metadata.
|
|
19
|
+
|
|
20
|
+
### Example CSV Structure:
|
|
21
|
+
|
|
22
|
+
| Test Name | Step 1 | Step 2 | Step 3 | Expected Outcome |
|
|
23
|
+
|--------------------|-------------------------|-------------------------|-------------------------|--------------------------|
|
|
24
|
+
| Login Test | Open login page | Enter valid credentials | Click "Log In" button | Dashboard is displayed |
|
|
25
|
+
| Invalid Login Test | Open login page | Enter invalid credentials | Click "Log In" button | Error message is shown |
|
|
26
|
+
|
|
27
|
+
### Conversion to TestDriver.ai YAML
|
|
28
|
+
A script can parse the CSV file and convert each row into a YAML test file.
|
|
29
|
+
|
|
30
|
+
#### Example YAML for "Login Test":```yaml
|
|
31
|
+
version: 4.2.18
|
|
32
|
+
steps:
|
|
33
|
+
- prompt: Open login page
|
|
34
|
+
- prompt: Enter valid credentials
|
|
35
|
+
- prompt: Click "Log In" button
|
|
36
|
+
- prompt: Verify that the dashboard is displayed
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
#### Automation Script
|
|
41
|
+
Use a Node.js or Python script to automate the conversion process. The script reads the CSV file, extracts the test cases, and generates YAML files for each test.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## 2. **Test Generation from Jira Tickets**
|
|
46
|
+
|
|
47
|
+
### Workflow Overview
|
|
48
|
+
Jira tickets often contain **user stories** or **acceptance criteria** that can be converted into test cases. By using the Jira API, you can fetch tickets and extract relevant fields like the title, description, and acceptance criteria.
|
|
49
|
+
|
|
50
|
+
### Example Jira Ticket:
|
|
51
|
+
- **Title**: Login functionality
|
|
52
|
+
- **Acceptance Criteria**:
|
|
53
|
+
- The user can log in with valid credentials.
|
|
54
|
+
- An error message is displayed for invalid credentials.
|
|
55
|
+
- The login page is responsive on mobile devices.
|
|
56
|
+
|
|
57
|
+
### Conversion to TestDriver.ai YAML
|
|
58
|
+
Each acceptance criterion can be converted into a `prompt` in a YAML test file.
|
|
59
|
+
|
|
60
|
+
#### Example YAML:```yaml
|
|
61
|
+
version: 4.2.18
|
|
62
|
+
steps:
|
|
63
|
+
- prompt: The user can log in with valid credentials.
|
|
64
|
+
- prompt: An error message is displayed for invalid credentials.
|
|
65
|
+
- prompt: The login page is responsive on mobile devices.
|
|
66
|
+
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
#### Automation Script
|
|
70
|
+
Use a script to fetch Jira tickets via the Jira API, extract the acceptance criteria, and generate YAML files.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## 3. **Test Generation from Gherkin Scenarios**
|
|
75
|
+
|
|
76
|
+
### Workflow Overview
|
|
77
|
+
Gherkin scenarios, commonly used with tools like Cucumber, describe test cases in a human-readable format using keywords like `Given`, `When`, and `Then`.
|
|
78
|
+
|
|
79
|
+
### Example Gherkin Scenario:```gherkin
|
|
80
|
+
Scenario: Successful login
|
|
81
|
+
Given the user is on the login page
|
|
82
|
+
When the user enters valid credentials
|
|
83
|
+
And clicks the "Log In" button
|
|
84
|
+
Then the user should see the dashboard
|
|
85
|
+
|
|
86
|
+
```### Conversion to TestDriver.ai YAML
|
|
87
|
+
Each Gherkin step can be mapped to a `prompt` in a YAML test file.
|
|
88
|
+
|
|
89
|
+
#### Example YAML:```yaml
|
|
90
|
+
version: 4.2.18
|
|
91
|
+
steps:
|
|
92
|
+
- prompt: The user is on the login page
|
|
93
|
+
- prompt: The user enters valid credentials
|
|
94
|
+
- prompt: Clicks the "Log In" button
|
|
95
|
+
- prompt: The user should see the dashboard
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
#### Automation Script
|
|
100
|
+
A script can parse Gherkin files, map the steps to prompts, and generate YAML files.
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## 4. **Test Generation from TestRail**
|
|
105
|
+
|
|
106
|
+
### Workflow Overview
|
|
107
|
+
TestRail is a popular test management tool. You can use the TestRail API to fetch test cases and convert them into TestDriver.ai YAML files.
|
|
108
|
+
|
|
109
|
+
### Example TestRail Test Case:
|
|
110
|
+
- **Title**: Login Test
|
|
111
|
+
- **Steps**:
|
|
112
|
+
1. Open the login page.
|
|
113
|
+
2. Enter valid credentials.
|
|
114
|
+
3. Click the "Log In" button.
|
|
115
|
+
- **Expected Result**: The dashboard is displayed.
|
|
116
|
+
|
|
117
|
+
### Conversion to TestDriver.ai YAML
|
|
118
|
+
Each step becomes a `prompt`, and the expected result can be added as an assertion.
|
|
119
|
+
|
|
120
|
+
#### Example YAML:```yaml
|
|
121
|
+
version: 4.2.18
|
|
122
|
+
steps:
|
|
123
|
+
- prompt: Open the login page
|
|
124
|
+
- prompt: Enter valid credentials
|
|
125
|
+
- prompt: Click the "Log In" button
|
|
126
|
+
- prompt: Verify that the dashboard is displayed
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
#### Automation Script
|
|
131
|
+
A script can fetch test cases from TestRail, extract the steps and expected results, and generate YAML files.
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## 5. **AI-Driven Exploratory Testing**
|
|
136
|
+
|
|
137
|
+
### Workflow Overview
|
|
138
|
+
TestDriver.ai can autonomously generate tests by exploring the UI and mapping workflows. This is particularly useful for:
|
|
139
|
+
- Discovering untested paths.
|
|
140
|
+
- Validating dynamic or complex UIs.
|
|
141
|
+
- Generating tests for new features without requiring manual input.
|
|
142
|
+
|
|
143
|
+
### How It Works:
|
|
144
|
+
1. **Recursive UI Mapping**: The AI navigates through the application, identifying clickable elements, input fields, and other interactive components.
|
|
145
|
+
2. **Test Generation**: The AI generates test cases based on the discovered workflows, including actions like clicking buttons, filling forms, and verifying outcomes.
|
|
146
|
+
3. **Self-Healing**: If the UI changes, the AI adapts the tests to match the new structure.
|
|
147
|
+
|
|
148
|
+
### Example AI-Generated Test:
|
|
149
|
+
For a login page, the AI might generate the following test:
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
### Benefits:
|
|
153
|
+
- **No Manual Input Required**: The AI generates tests automatically.
|
|
154
|
+
- **Comprehensive Coverage**: The AI explores all possible paths, ensuring thorough testing.
|
|
155
|
+
- **Dynamic Adaptation**: Tests are updated automatically when the UI changes.
|
|
156
|
+
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
## Summary of Test Generation Methods
|
|
160
|
+
|
|
161
|
+
| **Source** | **Description** | **Automation** |
|
|
162
|
+
|---------------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
|
|
163
|
+
| **CSV Files** | Convert rows into YAML test files. | Use a script to parse the CSV and generate YAML. |
|
|
164
|
+
| **Jira Tickets** | Extract user stories or acceptance criteria. | Use the Jira API to fetch tickets and generate YAML. |
|
|
165
|
+
| **Gherkin Scenarios** | Map `Given`, `When`, `Then` steps to YAML prompts. | Use a script to parse Gherkin files and generate YAML. |
|
|
166
|
+
| **TestRail** | Fetch test cases and steps from TestRail. | Use the TestRail API to extract test cases and generate YAML. |
|
|
167
|
+
| **AI-Driven Testing** | Automatically generate tests by exploring the UI and mapping workflows. | Let TestDriver.ai autonomously generate and adapt tests. |
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Best Practices
|
|
172
|
+
|
|
173
|
+
1. **Automate Test Generation**: Use scripts to streamline the conversion of test cases from various sources into YAML files.
|
|
174
|
+
2. **Leverage AI for Coverage**: Use TestDriver.ai's AI-driven exploratory testing to discover untested paths and validate dynamic UIs.
|
|
175
|
+
3. **Organize Test Files**: Store generated YAML files in a structured directory (e.g., `testdriver/`) for easy management.
|
|
176
|
+
4. **Review Generated Tests**: Manually review the generated tests to ensure they align with your testing requirements.
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
By combining structured test case imports (from CSV, Jira, Gherkin, and TestRail) with AI-driven exploratory testing, TestDriver.ai provides a comprehensive solution for generating and maintaining high-quality test suites.
|
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "GitHub Integration with TestDriver.ai"
|
|
3
|
+
sidebarTitle: "GitHub Integration"
|
|
4
|
+
description: "Explore how TestDriver.ai enhances GitHub workflows with branch creation, manual dispatch, and detailed outputs for seamless test management."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
TestDriver.ai offers advanced GitHub integration features, including the ability to create its own branches, trigger workflows manually from the GitHub Actions UI, and provide detailed outputs for debugging and reporting. These features make it easier to manage test workflows, collaborate on test results, and maintain a clean and automated testing pipeline.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## 1. **Creating Branches Automatically**
|
|
12
|
+
|
|
13
|
+
TestDriver.ai can create its own branches to store test results, updates, or auto-healed tests. This ensures that test-related changes are isolated from the main codebase and can be reviewed before merging.
|
|
14
|
+
|
|
15
|
+
### How It Works:
|
|
16
|
+
|
|
17
|
+
- **Branch Creation**: TestDriver.ai creates a new branch (e.g., `test-results` or `auto-heal-updates`) during the workflow execution.
|
|
18
|
+
- **Pull Request**: A pull request is automatically opened from the new branch to the base branch (e.g., `main`), allowing developers to review the changes.
|
|
19
|
+
|
|
20
|
+
### Example Workflow with Branch Creation:
|
|
21
|
+
|
|
22
|
+
```yaml
|
|
23
|
+
name: TestDriver.ai Auto-Healing
|
|
24
|
+
|
|
25
|
+
on:
|
|
26
|
+
pull_request:
|
|
27
|
+
workflow_dispatch:
|
|
28
|
+
|
|
29
|
+
jobs:
|
|
30
|
+
test:
|
|
31
|
+
name: Run Tests and Create Branch
|
|
32
|
+
runs-on: ubuntu-latest
|
|
33
|
+
steps:
|
|
34
|
+
- name: Check out repository
|
|
35
|
+
uses: actions/checkout@v2
|
|
36
|
+
|
|
37
|
+
- name: Run TestDriver.ai
|
|
38
|
+
uses: testdriverai/action@main
|
|
39
|
+
with:
|
|
40
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
41
|
+
prompt: |
|
|
42
|
+
1. Run all tests in the testdriver directory
|
|
43
|
+
create-pr: true
|
|
44
|
+
pr-title: "TestDriver.ai Test Results"
|
|
45
|
+
pr-branch: test-results
|
|
46
|
+
pr-test-filename: testdriver-results.yml
|
|
47
|
+
env:
|
|
48
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
49
|
+
FORCE_COLOR: "3"
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Key Fields:
|
|
54
|
+
- **`create-pr: true`**: Enables branch creation and pull request generation.
|
|
55
|
+
- **`pr-branch`**: Specifies the branch name (e.g., `test-results`).
|
|
56
|
+
- **`pr-title`**: Sets the title of the pull request.
|
|
57
|
+
- **`pr-test-filename`**: Specifies the filename for the test results.
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## 2. **Manual Workflow Dispatch**
|
|
62
|
+
|
|
63
|
+
GitHub Actions supports manual triggering of workflows using the **workflow_dispatch** event. This allows you to run TestDriver workflows on demand, making it ideal for exploratory testing or debugging.
|
|
64
|
+
|
|
65
|
+
### How to Trigger Workflows Manually:
|
|
66
|
+
1. Navigate to the **Actions** tab in your GitHub repository.
|
|
67
|
+
2. Select the workflow you want to run.
|
|
68
|
+
3. Click the **Run workflow** button.
|
|
69
|
+
4. Provide any required inputs (if applicable) and confirm.
|
|
70
|
+
|
|
71
|
+
### Example Workflow with Manual Dispatch:
|
|
72
|
+
|
|
73
|
+
```yaml
|
|
74
|
+
name: TestDriver.ai Manual Dispatch
|
|
75
|
+
|
|
76
|
+
on:
|
|
77
|
+
workflow_dispatch:
|
|
78
|
+
|
|
79
|
+
jobs:
|
|
80
|
+
test:
|
|
81
|
+
name: Run Tests Manually
|
|
82
|
+
runs-on: ubuntu-latest
|
|
83
|
+
steps:
|
|
84
|
+
- name: Check out repository
|
|
85
|
+
uses: actions/checkout@v2
|
|
86
|
+
|
|
87
|
+
- name: Run TestDriver.ai
|
|
88
|
+
uses: testdriverai/action@main
|
|
89
|
+
with:
|
|
90
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
91
|
+
prompt: |
|
|
92
|
+
1. Run all tests in the testdriver directory
|
|
93
|
+
env:
|
|
94
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
95
|
+
FORCE_COLOR: "3"
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Run Steps
|
|
101
|
+
|
|
102
|
+
### Benefits of Manual Dispatch:
|
|
103
|
+
- **Exploratory Testing**: Run tests on demand for specific scenarios.
|
|
104
|
+
- **Debugging**: Trigger workflows to debug issues without waiting for automated triggers.
|
|
105
|
+
- **Flexibility**: Test changes in feature branches or experimental setups.
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## 3. **GitHub Actions Output**
|
|
110
|
+
|
|
111
|
+
TestDriver.ai provides detailed outputs during workflow execution, which are visible in the **Actions** tab of your repository. These outputs include:
|
|
112
|
+
- **Test Summary**: A high-level overview of the test results.
|
|
113
|
+
- **Logs**: Step-by-step logs of the test execution.
|
|
114
|
+
- **Screenshots and GIFs**: Visual feedback for debugging.
|
|
115
|
+
- **Links to Dashcam.io**: Direct links to detailed test recordings and results.
|
|
116
|
+
|
|
117
|
+
### Example Outputs:
|
|
118
|
+
|
|
119
|
+
#### **Test Summary**:
|
|
120
|
+
- ✅ All tests passed.
|
|
121
|
+
- ❌ 2 tests failed.
|
|
122
|
+
|
|
123
|
+
#### **TestDriver.ai Replay**:
|
|
124
|
+
|
|
125
|
+
```markdown
|
|
126
|
+
[View Full Test Results](https://app.testdriver.ai/...)
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
### Using Outputs in Workflows:
|
|
132
|
+
You can capture and use TestDriver.ai outputs in subsequent steps of your workflow. For example, you can post the test summary as a comment on a pull request.
|
|
133
|
+
|
|
134
|
+
#### Example:
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
```yaml
|
|
138
|
+
- name: Post Test Results
|
|
139
|
+
uses: actions/github-script@v6
|
|
140
|
+
with:
|
|
141
|
+
script: |
|
|
142
|
+
const summary = `Test Summary: ${process.env.TEST_SUMMARY}`;
|
|
143
|
+
github.rest.issues.createComment({
|
|
144
|
+
issue_number: context.issue.number,
|
|
145
|
+
owner: context.repo.owner,
|
|
146
|
+
repo: context.repo.repo,
|
|
147
|
+
body: summary,
|
|
148
|
+
});
|
|
149
|
+
env:
|
|
150
|
+
TEST_SUMMARY: ${{ steps.testdriver.outputs.summary }}
|
|
151
|
+
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
---
|
|
155
|
+
|
|
156
|
+
## Summary of Features:
|
|
157
|
+
1. **Branch Creation**: Automatically create branches for test results or auto-healed tests.
|
|
158
|
+
2. **Manual Dispatch**: Trigger workflows on demand from the GitHub Actions UI.
|
|
159
|
+
3. **Detailed Outputs**: Access logs, summaries, and visual feedback directly in the Actions tab.
|
|
160
|
+
|
|
161
|
+
These features make TestDriver.ai a powerful tool for managing automated testing workflows in GitHub, enabling better collaboration, debugging, and test management.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Parallel Testing with TestDriver.ai"
|
|
3
|
+
sidebarTitle: "Parallel Testing"
|
|
4
|
+
description: "Learn how to leverage GitHub Actions' matrix strategy to run TestDriver.ai tests in parallel, reducing execution time and improving scalability."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
Parallel testing allows you to run multiple tests simultaneously, significantly reducing the time required to execute your test suite. By leveraging GitHub Actions' **matrix strategy**, you can dynamically distribute your tests across multiple jobs, ensuring efficient and scalable test execution.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Why Use Parallel Testing?
|
|
12
|
+
|
|
13
|
+
1. **Faster Execution**: Run multiple tests at the same time to reduce overall test duration.
|
|
14
|
+
2. **Scalability**: Easily scale your testing efforts as your test suite grows.
|
|
15
|
+
3. **Dynamic Distribution**: Automatically distribute tests across jobs using GitHub's matrix strategy.
|
|
16
|
+
4. **Cost Efficiency**: Optimize resource usage by running tests in parallel.
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Setting Up Parallel Testing with Matrix Strategy
|
|
21
|
+
|
|
22
|
+
### Step 1: Organize Your Test Files
|
|
23
|
+
|
|
24
|
+
Ensure your test files are stored in a directory (e.g., `testdriver/`) and follow a consistent naming convention (e.g., `test1.yml`, `test2.yml`, etc.).
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
### Step 2: Define the GitHub Action Workflow
|
|
29
|
+
|
|
30
|
+
Here’s an example of a GitHub Action workflow that uses the matrix strategy to run tests in parallel:
|
|
31
|
+
|
|
32
|
+
```yaml
|
|
33
|
+
name: Parallel Testing with TestDriver.ai
|
|
34
|
+
|
|
35
|
+
on:
|
|
36
|
+
push:
|
|
37
|
+
branches:
|
|
38
|
+
- main
|
|
39
|
+
pull_request:
|
|
40
|
+
workflow_dispatch:
|
|
41
|
+
|
|
42
|
+
jobs:
|
|
43
|
+
gather-test-files:
|
|
44
|
+
name: Gather Test Files
|
|
45
|
+
runs-on: ubuntu-latest
|
|
46
|
+
outputs:
|
|
47
|
+
test_files: ${{ steps.test_list.outputs.files }}
|
|
48
|
+
steps:
|
|
49
|
+
- name: Check out repository
|
|
50
|
+
uses: actions/checkout@v2
|
|
51
|
+
|
|
52
|
+
- name: Find all test files
|
|
53
|
+
id: test_list
|
|
54
|
+
run: |
|
|
55
|
+
FILES=$(ls ./testdriver/*.yml)
|
|
56
|
+
FILES_JSON=$(echo "$FILES" | jq -R -s -c 'split("\n")[:-1]')
|
|
57
|
+
echo "::set-output name=files::$FILES_JSON"
|
|
58
|
+
|
|
59
|
+
run-tests:
|
|
60
|
+
name: Run Tests in Parallel
|
|
61
|
+
needs: gather-test-files
|
|
62
|
+
runs-on: ubuntu-latest
|
|
63
|
+
strategy:
|
|
64
|
+
matrix:
|
|
65
|
+
test_file: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
|
|
66
|
+
fail-fast: false
|
|
67
|
+
steps:
|
|
68
|
+
- name: Check out repository
|
|
69
|
+
uses: actions/checkout@v2
|
|
70
|
+
|
|
71
|
+
- name: Run TestDriver.ai
|
|
72
|
+
uses: testdriverai/action@main
|
|
73
|
+
with:
|
|
74
|
+
key: ${{ secrets.TESTDRIVER_API_KEY }}
|
|
75
|
+
prompt: |
|
|
76
|
+
1. /run ${{ matrix.test_file }}
|
|
77
|
+
env:
|
|
78
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
79
|
+
FORCE_COLOR: "3"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
### Explanation of the Workflow
|
|
85
|
+
|
|
86
|
+
1. **`gather-test-files` Job**:
|
|
87
|
+
- Collects all test files in the `testdriver/` directory.
|
|
88
|
+
- Outputs the list of test files as a JSON array.
|
|
89
|
+
|
|
90
|
+
2. **`run-tests` Job**:
|
|
91
|
+
- Uses the matrix strategy to dynamically create a job for each test file.
|
|
92
|
+
- Runs each test file in parallel using TestDriver.ai.
|
|
93
|
+
|
|
94
|
+
3. **Matrix Strategy**:
|
|
95
|
+
- The `matrix.test_file` variable represents each test file.
|
|
96
|
+
- Each job runs a single test file from the `testdriver/` directory.
|
|
97
|
+
|
|
98
|
+
4. **Fail-Fast**:
|
|
99
|
+
- Set to `false` to ensure all tests run even if one fails.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## Benefits of Using the Matrix Strategy
|
|
104
|
+
|
|
105
|
+
1. **Dynamic Test Distribution**: Automatically adapts to the number of test files.
|
|
106
|
+
2. **Scalable**: Easily handles large test suites by distributing tests across multiple jobs.
|
|
107
|
+
3. **Efficient Resource Usage**: Runs tests in parallel, reducing idle time.
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## Example Output
|
|
112
|
+
|
|
113
|
+
When this workflow runs:
|
|
114
|
+
- Each test file (e.g., `test1.yml`, `test2.yml`) is executed in its own job.
|
|
115
|
+
- The results of all tests are displayed in the GitHub Actions dashboard.
|
|
116
|
+
|
|
117
|
+
---
|
|
118
|
+
|
|
119
|
+
## Best Practices
|
|
120
|
+
|
|
121
|
+
1. **Organize Test Files**: Use a consistent naming convention for test files to simplify management.
|
|
122
|
+
2. **Monitor Test Results**: Review the GitHub Actions dashboard to identify and debug failing tests.
|
|
123
|
+
3. **Optimize Test Files**: Ensure each test file is self-contained and does not depend on the execution of other tests.
|
|
124
|
+
4. **Use Fail-Fast Judiciously**: Enable `fail-fast: true` only if you want to stop all tests when one fails.
|
|
125
|
+
|
|
126
|
+
---
|
|
127
|
+
|
|
128
|
+
## Conclusion
|
|
129
|
+
|
|
130
|
+
Parallel testing with the GitHub Action matrix strategy is a powerful way to speed up your TestDriver.ai test suite. By dynamically distributing tests across multiple jobs, you can ensure efficient execution and scalability, making it easier to maintain high-quality software.
|