testdriverai 5.2.2 → 5.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (107) hide show
  1. package/.github/workflows/test-install.yml +1 -1
  2. package/README.md +5 -11
  3. package/agent.js +135 -99
  4. package/docs/30x30.mdx +84 -0
  5. package/docs/action/browser.mdx +129 -0
  6. package/docs/action/os.mdx +157 -0
  7. package/docs/action/output.mdx +98 -0
  8. package/docs/action/performance.mdx +71 -0
  9. package/docs/action/prerun.mdx +80 -0
  10. package/docs/action/secrets.mdx +103 -0
  11. package/docs/action/setup.mdx +115 -0
  12. package/docs/bugs/jira.mdx +208 -0
  13. package/docs/cli/overview.mdx +65 -0
  14. package/docs/commands/assert.mdx +31 -0
  15. package/docs/commands/exec.mdx +42 -0
  16. package/docs/commands/focus-application.mdx +29 -0
  17. package/docs/commands/hover-image.mdx +32 -0
  18. package/docs/commands/hover-text.mdx +37 -0
  19. package/docs/commands/if.mdx +43 -0
  20. package/docs/commands/match-image.mdx +41 -0
  21. package/docs/commands/press-keys.mdx +30 -0
  22. package/docs/commands/run.mdx +30 -0
  23. package/docs/commands/scroll-until-image.mdx +33 -0
  24. package/docs/commands/scroll-until-text.mdx +37 -0
  25. package/docs/commands/scroll.mdx +33 -0
  26. package/docs/commands/type.mdx +29 -0
  27. package/docs/commands/wait-for-image.mdx +31 -0
  28. package/docs/commands/wait-for-text.mdx +35 -0
  29. package/docs/commands/wait.mdx +30 -0
  30. package/docs/docs.json +226 -0
  31. package/docs/exporting/playwright.mdx +159 -0
  32. package/docs/features/auto-healing.mdx +124 -0
  33. package/docs/features/cross-platform.mdx +106 -0
  34. package/docs/features/generation.mdx +180 -0
  35. package/docs/features/github.mdx +161 -0
  36. package/docs/features/parallel-testing.mdx +130 -0
  37. package/docs/features/reusable-snippets.mdx +124 -0
  38. package/docs/features/selectorless.mdx +62 -0
  39. package/docs/features/visual-assertions.mdx +123 -0
  40. package/docs/getting-started/ci.mdx +196 -0
  41. package/docs/getting-started/generating.mdx +210 -0
  42. package/docs/getting-started/running.mdx +67 -0
  43. package/docs/getting-started/setup.mdx +133 -0
  44. package/docs/getting-started/writing.mdx +99 -0
  45. package/docs/guide/assertions.mdx +195 -0
  46. package/docs/guide/authentication.mdx +150 -0
  47. package/docs/guide/code.mdx +169 -0
  48. package/docs/guide/locating.mdx +136 -0
  49. package/docs/guide/setup-teardown.mdx +161 -0
  50. package/docs/guide/variables.mdx +218 -0
  51. package/docs/guide/waiting.mdx +199 -0
  52. package/docs/importing/csv.mdx +196 -0
  53. package/docs/importing/gherkin.mdx +142 -0
  54. package/docs/importing/jira.mdx +172 -0
  55. package/docs/importing/testrail.mdx +161 -0
  56. package/docs/integrations/electron.mdx +152 -0
  57. package/docs/integrations/netlify.mdx +98 -0
  58. package/docs/integrations/vercel.mdx +177 -0
  59. package/docs/interactive/assert.mdx +51 -0
  60. package/docs/interactive/generate.mdx +41 -0
  61. package/docs/interactive/run.mdx +36 -0
  62. package/docs/interactive/save.mdx +53 -0
  63. package/docs/interactive/undo.mdx +47 -0
  64. package/docs/issues.mdx +9 -0
  65. package/docs/overview/comparison.mdx +82 -0
  66. package/docs/overview/faq.mdx +122 -0
  67. package/docs/overview/quickstart.mdx +66 -0
  68. package/docs/overview/what-is-testdriver.mdx +73 -0
  69. package/docs/quickstart.mdx +66 -0
  70. package/docs/reference/commands/scroll.mdx +0 -0
  71. package/docs/reference/interactive/assert.mdx +0 -0
  72. package/docs/security/action.mdx +62 -0
  73. package/docs/security/agent.mdx +62 -0
  74. package/docs/security/dashboard.mdx +0 -0
  75. package/docs/security/platform.mdx +54 -0
  76. package/docs/tutorials/advanced-test.mdx +79 -0
  77. package/docs/tutorials/basic-test.mdx +41 -0
  78. package/electron/icon.png +0 -0
  79. package/electron/overlay.html +7 -3
  80. package/electron/overlay.js +75 -15
  81. package/electron/tray-buffered.png +0 -0
  82. package/electron/tray.png +0 -0
  83. package/index.js +75 -34
  84. package/lib/commander.js +22 -1
  85. package/lib/commands.js +87 -19
  86. package/lib/config.js +10 -1
  87. package/lib/focus-application.js +30 -23
  88. package/lib/generator.js +58 -7
  89. package/lib/init.js +48 -19
  90. package/lib/ipc.js +50 -0
  91. package/lib/logger.js +19 -6
  92. package/lib/overlay.js +82 -36
  93. package/lib/parser.js +9 -7
  94. package/lib/resources/prerun.yaml +17 -0
  95. package/lib/sandbox.js +2 -3
  96. package/lib/sdk.js +0 -2
  97. package/lib/session.js +3 -1
  98. package/lib/speak.js +0 -2
  99. package/lib/subimage/opencv.js +0 -4
  100. package/lib/system.js +56 -39
  101. package/lib/upload-secrets.js +65 -0
  102. package/lib/validation.js +175 -0
  103. package/package.json +2 -1
  104. package/postinstall.js +0 -24
  105. package/lib/websockets.js +0 -85
  106. package/test.md +0 -8
  107. package/test.yml +0 -18
@@ -0,0 +1,124 @@
1
+ ---
2
+ title: "Auto-Healing in TestDriver.ai"
3
+ sidebarTitle: "Auto-Healing"
4
+ description: "Learn how TestDriver.ai's auto-healing feature keeps your tests resilient and up-to-date with minimal manual effort."
5
+ ---
6
+
7
+ Auto-healing is a powerful feature in TestDriver.ai that ensures your tests remain resilient even when the application under test changes. When text, UI elements, or other visual cues are modified, TestDriver.ai can detect the failure, adapt to the changes, and automatically update the test. It then opens a pull request (PR) with the updated test, ensuring your test suite stays up-to-date with minimal manual intervention.
8
+
9
+ ---
10
+
11
+ ## How Auto-Healing Works
12
+
13
+ 1. **Test Execution**: TestDriver.ai runs your test suite as usual.
14
+ 2. **Failure Detection**: If a test fails due to a change in the application (e.g., text or UI updates), the AI identifies the failure point.
15
+ 3. **Recovery Attempt**: The AI uses its adaptive capabilities to locate the updated element or text and retries the action.
16
+ 4. **Test Update**: If the recovery is successful, TestDriver.ai updates the test script with the new element or text.
17
+ 5. **Pull Request Creation**: The updated test script is committed to a new branch, and a pull request is automatically opened in your repository for review.
18
+
19
+ ---
20
+
21
+ ## Benefits of Auto-Healing
22
+
23
+ - **Reduced Maintenance**: Eliminates the need for manual updates to tests when minor UI changes occur.
24
+ - **Increased Test Resilience**: Ensures tests adapt to changes without breaking.
25
+ - **Continuous Integration**: Keeps your test suite aligned with the latest application changes.
26
+ - **Developer Collaboration**: Automatically opens a PR, allowing developers to review and approve updates.
27
+
28
+ ---
29
+
30
+ ## Example Workflow
31
+
32
+ ### Scenario: Button Text Changes from "Submit" to "Send"
33
+
34
+ 1. **Initial Test**:
35
+
36
+ ```yaml
37
+ - command: hover-text
38
+ text: Submit
39
+ description: Submit button in the form
40
+ action: click
41
+ ```
42
+
43
+ 2. **Application Update**:
44
+ - The button text is changed from "Submit" to "Send" in the application.
45
+
46
+ 3. **Test Execution**:
47
+ - TestDriver.ai runs the test and fails to find the "Submit" button.
48
+
49
+ 4. **Auto-Healing**:
50
+ - The AI detects the failure and searches for a similar element.
51
+ - It identifies the "Send" button as the updated element and retries the action.
52
+
53
+ 5. **Test Update**:
54
+ - The test script is updated to reflect the new button text:
55
+
56
+ ```yaml
57
+ - command: hover-text
58
+ text: Send
59
+ description: Submit button in the form
60
+ action: click
61
+ ```
62
+
63
+ 6. **Pull Request Creation**:
64
+ - TestDriver.ai commits the updated test to a new branch and opens a PR:
65
+ - **Branch Name**: `auto-heal-update-submit-to-send`
66
+ - **PR Title**: `Auto-Healed Test: Updated "Submit" to "Send"`
67
+
68
+ ---
69
+
70
+ ## Example GitHub Action for Auto-Healing
71
+
72
+ Here’s how you can configure a GitHub Action to enable auto-healing and PR creation:
73
+
74
+ ```yaml
75
+ name: TestDriver.ai Auto-Healing
76
+
77
+ on:
78
+ push:
79
+ branches:
80
+ - main
81
+ pull_request:
82
+ workflow_dispatch:
83
+
84
+ jobs:
85
+ test:
86
+ name: "Run Tests with Auto-Healing"
87
+ runs-on: ubuntu-latest
88
+ steps:
89
+ - name: Check out repository
90
+ uses: actions/checkout@v2
91
+
92
+ - name: Run TestDriver.ai
93
+ uses: testdriverai/action@main
94
+ with:
95
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
96
+ prompt: |
97
+ 1. /run testdriver/onboarding.yaml
98
+ create-pr: true
99
+ pr-title: "Auto-Healed Test Updates"
100
+ pr-base: main
101
+ pr-branch: auto-heal-updates
102
+ ```
103
+
104
+ ---
105
+
106
+ ## Best Practices for Auto-Healing
107
+
108
+ 1. **Enable PR Reviews**: Always review auto-healed PRs to ensure the updates align with the intended application behavior.
109
+ 2. **Use Descriptive Descriptions**: Provide clear descriptions for elements in your tests to help the AI identify changes more effectively.
110
+ 3. **Run Tests Frequently**: Integrate TestDriver.ai into your CI/CD pipeline to catch and heal changes as soon as they occur.
111
+ 4. **Monitor Auto-Healing Logs**: Review logs and screenshots provided by TestDriver.ai to understand why a test was updated.
112
+
113
+ ---
114
+
115
+ ## Limitations of Auto-Healing
116
+
117
+ - **Major UI Overhauls**: Auto-healing is best suited for minor changes (e.g., text updates, small layout adjustments). Significant UI changes may still require manual intervention.
118
+ - **Ambiguous Changes**: If multiple elements match the updated criteria, the AI may require additional context to make the correct decision.
119
+
120
+ ---
121
+
122
+ ## Conclusion
123
+
124
+ Auto-healing in TestDriver.ai reduces the burden of test maintenance by automatically adapting to changes in your application. By leveraging this feature, you can ensure your test suite remains robust and up-to-date, allowing your team to focus on delivering high-quality software.
@@ -0,0 +1,106 @@
1
+ ---
2
+ title: "Cross-Platform Testing with TestDriver.ai"
3
+ sidebarTitle: "Cross-Platform Testing"
4
+ description: "Discover how TestDriver.ai enables seamless cross-platform testing across Windows, Linux, and Mac operating systems."
5
+ ---
6
+
7
+ TestDriver.ai provides robust cross-platform support, enabling you to run tests seamlessly on **Windows**, **Linux**, and **Mac** operating systems. This flexibility ensures that your applications are tested in environments that match your users' setups.
8
+
9
+ ## Specifying the Operating System
10
+
11
+ When using the TestDriver GitHub Action, you can specify the target operating system by setting the `os` field. Supported values are:
12
+
13
+ - `windows`
14
+ - `linux`
15
+ - `mac`
16
+
17
+ Example:
18
+
19
+ ```yaml
20
+ jobs:
21
+ test:
22
+ runs-on: ubuntu-latest
23
+ steps:
24
+ - uses: testdriverai/action@main
25
+ with:
26
+ os: linux
27
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
28
+ prompt: |
29
+ 1. Open the browser
30
+ 2. Navigate to example.com
31
+
32
+ ```
33
+
34
+ ## Using the `prerun` Field
35
+
36
+ The `prerun` field allows you to execute a shell script before running your tests. This is particularly useful for:
37
+
38
+ 1. **Downloading and Installing Applications**
39
+ Example: Downloading and installing a custom app or dependency.
40
+
41
+ 2. **Installing Chrome Extensions**
42
+ Example: Adding a browser extension required for testing.
43
+
44
+ 3. **Launching a Browser**
45
+ Example: Starting a browser with specific arguments or configurations.
46
+
47
+ ### Example `prerun` Script
48
+
49
+ ```yaml
50
+ jobs:
51
+ test:
52
+ runs-on: macos-latest
53
+ steps:
54
+ - uses: testdriverai/action@main
55
+ with:
56
+ os: mac
57
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
58
+ prompt: |
59
+ 1. Open the browser
60
+ 2. Navigate to example.com
61
+ prerun: |
62
+ # Download and install an app
63
+ curl -o app.dmg https://example.com/app.dmg
64
+ hdiutil attach app.dmg
65
+ cp -R /Volumes/App/App.app /Applications/
66
+
67
+ # Install a Chrome extension
68
+ npm install dashcam-chrome
69
+ open -a "Google Chrome" --args --load-extension=$(pwd)/node_modules/dashcam-chrome/build
70
+
71
+ # Launch the browser
72
+ open -a "Google Chrome"
73
+
74
+ ```
75
+
76
+ ## Write Once, Run Anywhere
77
+
78
+ TestDriver.ai uses **selectorless testing**, meaning tests are written in a generic format that works across all platforms. The AI dynamically adapts to the environment, ensuring that the same test can run on Windows, Linux, and Mac without modification.
79
+
80
+ ### Benefits of Selectorless Testing
81
+
82
+ - **Reduced Maintenance**: No need to update selectors when UI changes.
83
+ - **Cross-Platform Compatibility**: Tests are not tied to platform-specific configurations.
84
+ - **Ease of Use**: Write high-level, natural language prompts, and let TestDriver handle the rest.
85
+
86
+ ### Example Test
87
+
88
+ ```yaml
89
+ version: 4.2.18
90
+ steps:
91
+ - prompt: Open the browser and search for "TestDriver.ai"
92
+ commands:
93
+ - command: focus-application
94
+ name: Google Chrome
95
+ - command: hover-text
96
+ text: Search Google or type a URL
97
+ description: main search bar
98
+ action: click
99
+ - command: type
100
+ text: TestDriver.ai
101
+ - command: press-keys
102
+ keys: [enter]
103
+
104
+ ```
105
+
106
+ This test will work on any supported operating system without modification.
@@ -0,0 +1,180 @@
1
+ ---
2
+ title: "Test Generation with TestDriver.ai"
3
+ sidebarTitle: "Test Generation"
4
+ description: "Simplify test creation with AI-driven exploratory testing."
5
+ ---
6
+
7
+ TestDriver.ai provides a powerful and flexible framework for generating tests from various sources, including:
8
+ - **TestRail test cases**
9
+ - **CSV files**
10
+ - **Jira tickets**
11
+ - **Gherkin scenarios**
12
+
13
+ Additionally, TestDriver.ai can autonomously generate its own tests by recursively mapping the UI and exploring workflows.
14
+
15
+ ## **Test Generation from CSV Files**
16
+
17
+ ### Workflow Overview
18
+ CSV files are a simple and structured way to define test cases. Each row in the CSV file can represent a test case, with columns specifying the test steps, expected outcomes, or other metadata.
19
+
20
+ ### Example CSV Structure:
21
+
22
+ | Test Name | Step 1 | Step 2 | Step 3 | Expected Outcome |
23
+ |--------------------|-------------------------|-------------------------|-------------------------|--------------------------|
24
+ | Login Test | Open login page | Enter valid credentials | Click "Log In" button | Dashboard is displayed |
25
+ | Invalid Login Test | Open login page | Enter invalid credentials | Click "Log In" button | Error message is shown |
26
+
27
+ ### Conversion to TestDriver.ai YAML
28
+ A script can parse the CSV file and convert each row into a YAML test file.
29
+
30
+ #### Example YAML for "Login Test":```yaml
31
+ version: 4.2.18
32
+ steps:
33
+ - prompt: Open login page
34
+ - prompt: Enter valid credentials
35
+ - prompt: Click "Log In" button
36
+ - prompt: Verify that the dashboard is displayed
37
+
38
+ ```
39
+
40
+ #### Automation Script
41
+ Use a Node.js or Python script to automate the conversion process. The script reads the CSV file, extracts the test cases, and generates YAML files for each test.
42
+
43
+ ---
44
+
45
+ ## 2. **Test Generation from Jira Tickets**
46
+
47
+ ### Workflow Overview
48
+ Jira tickets often contain **user stories** or **acceptance criteria** that can be converted into test cases. By using the Jira API, you can fetch tickets and extract relevant fields like the title, description, and acceptance criteria.
49
+
50
+ ### Example Jira Ticket:
51
+ - **Title**: Login functionality
52
+ - **Acceptance Criteria**:
53
+ - The user can log in with valid credentials.
54
+ - An error message is displayed for invalid credentials.
55
+ - The login page is responsive on mobile devices.
56
+
57
+ ### Conversion to TestDriver.ai YAML
58
+ Each acceptance criterion can be converted into a `prompt` in a YAML test file.
59
+
60
+ #### Example YAML:```yaml
61
+ version: 4.2.18
62
+ steps:
63
+ - prompt: The user can log in with valid credentials.
64
+ - prompt: An error message is displayed for invalid credentials.
65
+ - prompt: The login page is responsive on mobile devices.
66
+
67
+ ```
68
+
69
+ #### Automation Script
70
+ Use a script to fetch Jira tickets via the Jira API, extract the acceptance criteria, and generate YAML files.
71
+
72
+ ---
73
+
74
+ ## 3. **Test Generation from Gherkin Scenarios**
75
+
76
+ ### Workflow Overview
77
+ Gherkin scenarios, commonly used with tools like Cucumber, describe test cases in a human-readable format using keywords like `Given`, `When`, and `Then`.
78
+
79
+ ### Example Gherkin Scenario:```gherkin
80
+ Scenario: Successful login
81
+ Given the user is on the login page
82
+ When the user enters valid credentials
83
+ And clicks the "Log In" button
84
+ Then the user should see the dashboard
85
+
86
+ ```### Conversion to TestDriver.ai YAML
87
+ Each Gherkin step can be mapped to a `prompt` in a YAML test file.
88
+
89
+ #### Example YAML:```yaml
90
+ version: 4.2.18
91
+ steps:
92
+ - prompt: The user is on the login page
93
+ - prompt: The user enters valid credentials
94
+ - prompt: Clicks the "Log In" button
95
+ - prompt: The user should see the dashboard
96
+
97
+ ```
98
+
99
+ #### Automation Script
100
+ A script can parse Gherkin files, map the steps to prompts, and generate YAML files.
101
+
102
+ ---
103
+
104
+ ## 4. **Test Generation from TestRail**
105
+
106
+ ### Workflow Overview
107
+ TestRail is a popular test management tool. You can use the TestRail API to fetch test cases and convert them into TestDriver.ai YAML files.
108
+
109
+ ### Example TestRail Test Case:
110
+ - **Title**: Login Test
111
+ - **Steps**:
112
+ 1. Open the login page.
113
+ 2. Enter valid credentials.
114
+ 3. Click the "Log In" button.
115
+ - **Expected Result**: The dashboard is displayed.
116
+
117
+ ### Conversion to TestDriver.ai YAML
118
+ Each step becomes a `prompt`, and the expected result can be added as an assertion.
119
+
120
+ #### Example YAML:```yaml
121
+ version: 4.2.18
122
+ steps:
123
+ - prompt: Open the login page
124
+ - prompt: Enter valid credentials
125
+ - prompt: Click the "Log In" button
126
+ - prompt: Verify that the dashboard is displayed
127
+
128
+ ```
129
+
130
+ #### Automation Script
131
+ A script can fetch test cases from TestRail, extract the steps and expected results, and generate YAML files.
132
+
133
+ ---
134
+
135
+ ## 5. **AI-Driven Exploratory Testing**
136
+
137
+ ### Workflow Overview
138
+ TestDriver.ai can autonomously generate tests by exploring the UI and mapping workflows. This is particularly useful for:
139
+ - Discovering untested paths.
140
+ - Validating dynamic or complex UIs.
141
+ - Generating tests for new features without requiring manual input.
142
+
143
+ ### How It Works:
144
+ 1. **Recursive UI Mapping**: The AI navigates through the application, identifying clickable elements, input fields, and other interactive components.
145
+ 2. **Test Generation**: The AI generates test cases based on the discovered workflows, including actions like clicking buttons, filling forms, and verifying outcomes.
146
+ 3. **Self-Healing**: If the UI changes, the AI adapts the tests to match the new structure.
147
+
148
+ ### Example AI-Generated Test:
149
+ For a login page, the AI might generate the following test:
150
+
151
+
152
+ ### Benefits:
153
+ - **No Manual Input Required**: The AI generates tests automatically.
154
+ - **Comprehensive Coverage**: The AI explores all possible paths, ensuring thorough testing.
155
+ - **Dynamic Adaptation**: Tests are updated automatically when the UI changes.
156
+
157
+ ---
158
+
159
+ ## Summary of Test Generation Methods
160
+
161
+ | **Source** | **Description** | **Automation** |
162
+ |---------------------|---------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
163
+ | **CSV Files** | Convert rows into YAML test files. | Use a script to parse the CSV and generate YAML. |
164
+ | **Jira Tickets** | Extract user stories or acceptance criteria. | Use the Jira API to fetch tickets and generate YAML. |
165
+ | **Gherkin Scenarios** | Map `Given`, `When`, `Then` steps to YAML prompts. | Use a script to parse Gherkin files and generate YAML. |
166
+ | **TestRail** | Fetch test cases and steps from TestRail. | Use the TestRail API to extract test cases and generate YAML. |
167
+ | **AI-Driven Testing** | Automatically generate tests by exploring the UI and mapping workflows. | Let TestDriver.ai autonomously generate and adapt tests. |
168
+
169
+ ---
170
+
171
+ ## Best Practices
172
+
173
+ 1. **Automate Test Generation**: Use scripts to streamline the conversion of test cases from various sources into YAML files.
174
+ 2. **Leverage AI for Coverage**: Use TestDriver.ai's AI-driven exploratory testing to discover untested paths and validate dynamic UIs.
175
+ 3. **Organize Test Files**: Store generated YAML files in a structured directory (e.g., `testdriver/`) for easy management.
176
+ 4. **Review Generated Tests**: Manually review the generated tests to ensure they align with your testing requirements.
177
+
178
+ ---
179
+
180
+ By combining structured test case imports (from CSV, Jira, Gherkin, and TestRail) with AI-driven exploratory testing, TestDriver.ai provides a comprehensive solution for generating and maintaining high-quality test suites.
@@ -0,0 +1,161 @@
1
+ ---
2
+ title: "GitHub Integration with TestDriver.ai"
3
+ sidebarTitle: "GitHub Integration"
4
+ description: "Explore how TestDriver.ai enhances GitHub workflows with branch creation, manual dispatch, and detailed outputs for seamless test management."
5
+ ---
6
+
7
+ TestDriver.ai offers advanced GitHub integration features, including the ability to create its own branches, trigger workflows manually from the GitHub Actions UI, and provide detailed outputs for debugging and reporting. These features make it easier to manage test workflows, collaborate on test results, and maintain a clean and automated testing pipeline.
8
+
9
+ ---
10
+
11
+ ## 1. **Creating Branches Automatically**
12
+
13
+ TestDriver.ai can create its own branches to store test results, updates, or auto-healed tests. This ensures that test-related changes are isolated from the main codebase and can be reviewed before merging.
14
+
15
+ ### How It Works:
16
+
17
+ - **Branch Creation**: TestDriver.ai creates a new branch (e.g., `test-results` or `auto-heal-updates`) during the workflow execution.
18
+ - **Pull Request**: A pull request is automatically opened from the new branch to the base branch (e.g., `main`), allowing developers to review the changes.
19
+
20
+ ### Example Workflow with Branch Creation:
21
+
22
+ ```yaml
23
+ name: TestDriver.ai Auto-Healing
24
+
25
+ on:
26
+ pull_request:
27
+ workflow_dispatch:
28
+
29
+ jobs:
30
+ test:
31
+ name: Run Tests and Create Branch
32
+ runs-on: ubuntu-latest
33
+ steps:
34
+ - name: Check out repository
35
+ uses: actions/checkout@v2
36
+
37
+ - name: Run TestDriver.ai
38
+ uses: testdriverai/action@main
39
+ with:
40
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
41
+ prompt: |
42
+ 1. Run all tests in the testdriver directory
43
+ create-pr: true
44
+ pr-title: "TestDriver.ai Test Results"
45
+ pr-branch: test-results
46
+ pr-test-filename: testdriver-results.yml
47
+ env:
48
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
49
+ FORCE_COLOR: "3"
50
+
51
+ ```
52
+
53
+ ### Key Fields:
54
+ - **`create-pr: true`**: Enables branch creation and pull request generation.
55
+ - **`pr-branch`**: Specifies the branch name (e.g., `test-results`).
56
+ - **`pr-title`**: Sets the title of the pull request.
57
+ - **`pr-test-filename`**: Specifies the filename for the test results.
58
+
59
+ ---
60
+
61
+ ## 2. **Manual Workflow Dispatch**
62
+
63
+ GitHub Actions supports manual triggering of workflows using the **workflow_dispatch** event. This allows you to run TestDriver workflows on demand, making it ideal for exploratory testing or debugging.
64
+
65
+ ### How to Trigger Workflows Manually:
66
+ 1. Navigate to the **Actions** tab in your GitHub repository.
67
+ 2. Select the workflow you want to run.
68
+ 3. Click the **Run workflow** button.
69
+ 4. Provide any required inputs (if applicable) and confirm.
70
+
71
+ ### Example Workflow with Manual Dispatch:
72
+
73
+ ```yaml
74
+ name: TestDriver.ai Manual Dispatch
75
+
76
+ on:
77
+ workflow_dispatch:
78
+
79
+ jobs:
80
+ test:
81
+ name: Run Tests Manually
82
+ runs-on: ubuntu-latest
83
+ steps:
84
+ - name: Check out repository
85
+ uses: actions/checkout@v2
86
+
87
+ - name: Run TestDriver.ai
88
+ uses: testdriverai/action@main
89
+ with:
90
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
91
+ prompt: |
92
+ 1. Run all tests in the testdriver directory
93
+ env:
94
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
95
+ FORCE_COLOR: "3"
96
+
97
+
98
+ ```
99
+
100
+ Run Steps
101
+
102
+ ### Benefits of Manual Dispatch:
103
+ - **Exploratory Testing**: Run tests on demand for specific scenarios.
104
+ - **Debugging**: Trigger workflows to debug issues without waiting for automated triggers.
105
+ - **Flexibility**: Test changes in feature branches or experimental setups.
106
+
107
+ ---
108
+
109
+ ## 3. **GitHub Actions Output**
110
+
111
+ TestDriver.ai provides detailed outputs during workflow execution, which are visible in the **Actions** tab of your repository. These outputs include:
112
+ - **Test Summary**: A high-level overview of the test results.
113
+ - **Logs**: Step-by-step logs of the test execution.
114
+ - **Screenshots and GIFs**: Visual feedback for debugging.
115
+ - **Links to Dashcam.io**: Direct links to detailed test recordings and results.
116
+
117
+ ### Example Outputs:
118
+
119
+ #### **Test Summary**:
120
+ - ✅ All tests passed.
121
+ - ❌ 2 tests failed.
122
+
123
+ #### **TestDriver.ai Replay**:
124
+
125
+ ```markdown
126
+ [View Full Test Results](https://app.testdriver.ai/...)
127
+ ```
128
+
129
+ ---
130
+
131
+ ### Using Outputs in Workflows:
132
+ You can capture and use TestDriver.ai outputs in subsequent steps of your workflow. For example, you can post the test summary as a comment on a pull request.
133
+
134
+ #### Example:
135
+
136
+
137
+ ```yaml
138
+ - name: Post Test Results
139
+ uses: actions/github-script@v6
140
+ with:
141
+ script: |
142
+ const summary = `Test Summary: ${process.env.TEST_SUMMARY}`;
143
+ github.rest.issues.createComment({
144
+ issue_number: context.issue.number,
145
+ owner: context.repo.owner,
146
+ repo: context.repo.repo,
147
+ body: summary,
148
+ });
149
+ env:
150
+ TEST_SUMMARY: ${{ steps.testdriver.outputs.summary }}
151
+
152
+ ```
153
+
154
+ ---
155
+
156
+ ## Summary of Features:
157
+ 1. **Branch Creation**: Automatically create branches for test results or auto-healed tests.
158
+ 2. **Manual Dispatch**: Trigger workflows on demand from the GitHub Actions UI.
159
+ 3. **Detailed Outputs**: Access logs, summaries, and visual feedback directly in the Actions tab.
160
+
161
+ These features make TestDriver.ai a powerful tool for managing automated testing workflows in GitHub, enabling better collaboration, debugging, and test management.
@@ -0,0 +1,130 @@
1
+ ---
2
+ title: "Parallel Testing with TestDriver.ai"
3
+ sidebarTitle: "Parallel Testing"
4
+ description: "Learn how to leverage GitHub Actions' matrix strategy to run TestDriver.ai tests in parallel, reducing execution time and improving scalability."
5
+ ---
6
+
7
+ Parallel testing allows you to run multiple tests simultaneously, significantly reducing the time required to execute your test suite. By leveraging GitHub Actions' **matrix strategy**, you can dynamically distribute your tests across multiple jobs, ensuring efficient and scalable test execution.
8
+
9
+ ---
10
+
11
+ ## Why Use Parallel Testing?
12
+
13
+ 1. **Faster Execution**: Run multiple tests at the same time to reduce overall test duration.
14
+ 2. **Scalability**: Easily scale your testing efforts as your test suite grows.
15
+ 3. **Dynamic Distribution**: Automatically distribute tests across jobs using GitHub's matrix strategy.
16
+ 4. **Cost Efficiency**: Optimize resource usage by running tests in parallel.
17
+
18
+ ---
19
+
20
+ ## Setting Up Parallel Testing with Matrix Strategy
21
+
22
+ ### Step 1: Organize Your Test Files
23
+
24
+ Ensure your test files are stored in a directory (e.g., `testdriver/`) and follow a consistent naming convention (e.g., `test1.yml`, `test2.yml`, etc.).
25
+
26
+ ---
27
+
28
+ ### Step 2: Define the GitHub Action Workflow
29
+
30
+ Here’s an example of a GitHub Action workflow that uses the matrix strategy to run tests in parallel:
31
+
32
+ ```yaml
33
+ name: Parallel Testing with TestDriver.ai
34
+
35
+ on:
36
+ push:
37
+ branches:
38
+ - main
39
+ pull_request:
40
+ workflow_dispatch:
41
+
42
+ jobs:
43
+ gather-test-files:
44
+ name: Gather Test Files
45
+ runs-on: ubuntu-latest
46
+ outputs:
47
+ test_files: ${{ steps.test_list.outputs.files }}
48
+ steps:
49
+ - name: Check out repository
50
+ uses: actions/checkout@v2
51
+
52
+ - name: Find all test files
53
+ id: test_list
54
+ run: |
55
+ FILES=$(ls ./testdriver/*.yml)
56
+ FILES_JSON=$(echo "$FILES" | jq -R -s -c 'split("\n")[:-1]')
57
+ echo "::set-output name=files::$FILES_JSON"
58
+
59
+ run-tests:
60
+ name: Run Tests in Parallel
61
+ needs: gather-test-files
62
+ runs-on: ubuntu-latest
63
+ strategy:
64
+ matrix:
65
+ test_file: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
66
+ fail-fast: false
67
+ steps:
68
+ - name: Check out repository
69
+ uses: actions/checkout@v2
70
+
71
+ - name: Run TestDriver.ai
72
+ uses: testdriverai/action@main
73
+ with:
74
+ key: ${{ secrets.TESTDRIVER_API_KEY }}
75
+ prompt: |
76
+ 1. /run ${{ matrix.test_file }}
77
+ env:
78
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
79
+ FORCE_COLOR: "3"
80
+ ```
81
+
82
+ ---
83
+
84
+ ### Explanation of the Workflow
85
+
86
+ 1. **`gather-test-files` Job**:
87
+ - Collects all test files in the `testdriver/` directory.
88
+ - Outputs the list of test files as a JSON array.
89
+
90
+ 2. **`run-tests` Job**:
91
+ - Uses the matrix strategy to dynamically create a job for each test file.
92
+ - Runs each test file in parallel using TestDriver.ai.
93
+
94
+ 3. **Matrix Strategy**:
95
+ - The `matrix.test_file` variable represents each test file.
96
+ - Each job runs a single test file from the `testdriver/` directory.
97
+
98
+ 4. **Fail-Fast**:
99
+ - Set to `false` to ensure all tests run even if one fails.
100
+
101
+ ---
102
+
103
+ ## Benefits of Using the Matrix Strategy
104
+
105
+ 1. **Dynamic Test Distribution**: Automatically adapts to the number of test files.
106
+ 2. **Scalable**: Easily handles large test suites by distributing tests across multiple jobs.
107
+ 3. **Efficient Resource Usage**: Runs tests in parallel, reducing idle time.
108
+
109
+ ---
110
+
111
+ ## Example Output
112
+
113
+ When this workflow runs:
114
+ - Each test file (e.g., `test1.yml`, `test2.yml`) is executed in its own job.
115
+ - The results of all tests are displayed in the GitHub Actions dashboard.
116
+
117
+ ---
118
+
119
+ ## Best Practices
120
+
121
+ 1. **Organize Test Files**: Use a consistent naming convention for test files to simplify management.
122
+ 2. **Monitor Test Results**: Review the GitHub Actions dashboard to identify and debug failing tests.
123
+ 3. **Optimize Test Files**: Ensure each test file is self-contained and does not depend on the execution of other tests.
124
+ 4. **Use Fail-Fast Judiciously**: Enable `fail-fast: true` only if you want to stop all tests when one fails.
125
+
126
+ ---
127
+
128
+ ## Conclusion
129
+
130
+ Parallel testing with the GitHub Action matrix strategy is a powerful way to speed up your TestDriver.ai test suite. By dynamically distributing tests across multiple jobs, you can ensure efficient execution and scalability, making it easier to maintain high-quality software.