testdriverai 6.0.15 → 6.0.16-canary.e337e67.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agent/index.js +2 -1
- package/docs/account/enterprise.mdx +64 -51
- package/docs/action/performance.mdx +7 -7
- package/docs/commands/hover-text.mdx +7 -0
- package/docs/docs.json +2 -2
- package/docs/getting-started/vscode.mdx +1 -1
- package/docs/guide/lifecycle.mdx +0 -8
- package/docs/guide/protips.mdx +42 -0
- package/package.json +1 -1
- package/testdriver/acceptance/remember.yaml +25 -5
- package/testdriver/examples/web/lifecycle/prerun.yaml +22 -8
- package/docs/commands/issues.mdx +0 -20
- package/docs/docs-future.json.txt +0 -114
- package/docs/support/known-bugs.mdx +0 -85
- package/testdriver/examples/doom/.env.example +0 -1
- package/testdriver/examples/doom/desktop.yaml +0 -0
- package/testdriver/examples/doom/lifecycle/provision.yaml +0 -46
- package/testdriver/examples/doom/readme.md +0 -1
- package/testdriver/examples/mobile/readme.md +0 -3
- package/testdriver/examples/npm/install.yaml +0 -0
- package/testdriver/examples/npm/lifecycle/prerun.yaml +0 -19
- package/testdriver/examples/performance/lifecycle/prerun.yaml +0 -40
- package/testdriver/examples/performance/monitor.yaml +0 -40
- package/testdriver/examples/performance/readme.md +0 -1
- package/testdriver/examples/playwright-test-recording/hello-world.yaml +0 -2
- package/testdriver/examples/playwright-test-recording/lifecycle/prerun.yaml +0 -28
- package/testdriver/examples/playwright-test-recording/readme.md +0 -1
- /package/testdriver/examples/{arc-browser → desktop}/desktop.yaml +0 -0
- /package/testdriver/examples/{arc-browser → desktop}/lifecycle/prerun.yaml +0 -0
package/agent/index.js
CHANGED
|
@@ -447,13 +447,14 @@ class TestDriverAgent extends EventEmitter2 {
|
|
|
447
447
|
sourcePosition: sourcePosition,
|
|
448
448
|
});
|
|
449
449
|
|
|
450
|
-
|
|
450
|
+
await this.haveAIResolveError(
|
|
451
451
|
error,
|
|
452
452
|
yaml.dump({ commands: [yml] }),
|
|
453
453
|
depth,
|
|
454
454
|
true,
|
|
455
455
|
shouldSave,
|
|
456
456
|
);
|
|
457
|
+
throw error;
|
|
457
458
|
}
|
|
458
459
|
}
|
|
459
460
|
|
|
@@ -7,95 +7,108 @@ icon: gem
|
|
|
7
7
|
|
|
8
8
|
import { calendar } from "/snippets/calendar-link.mdx";
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
TestDriver Enterprise is designed for fast-moving teams that need comprehensive QA coverage for complex applications. Our **4x4 Guarantee** ensures you get 4 custom tests created specifically for your application within your first 4 weeks.
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
We specialize in testing scenarios that other tools can't handle - desktop applications, browser extensions, multi-platform workflows, and complex integrations that require custom infrastructure and expert support.
|
|
13
13
|
|
|
14
14
|
<Card
|
|
15
|
-
title="
|
|
15
|
+
title="Schedule a Consultation"
|
|
16
16
|
icon="calendar"
|
|
17
17
|
href="https://calendly.com/d/cq23-qyn-3v6/testdriver-ai-demo"
|
|
18
18
|
link
|
|
19
19
|
>
|
|
20
|
-
|
|
20
|
+
Discuss your testing requirements and custom infrastructure needs.
|
|
21
21
|
</Card>
|
|
22
22
|
|
|
23
23
|
---
|
|
24
24
|
|
|
25
|
-
##
|
|
25
|
+
## Enterprise Plans
|
|
26
26
|
|
|
27
|
-
|
|
27
|
+
TestDriver Enterprise plans start at **$995/month** and include:
|
|
28
28
|
|
|
29
|
-
- **12,500 runner minutes per month**:
|
|
30
|
-
-
|
|
31
|
-
-
|
|
29
|
+
- **12,500 runner minutes per month**: Sufficient capacity for continuous testing of your custom test suite.
|
|
30
|
+
- Enterprise-grade test dashboards with advanced analytics.
|
|
31
|
+
- Full CI/CD pipeline integration with custom configurations.
|
|
32
|
+
- Dedicated infrastructure and ongoing support for complex testing scenarios.
|
|
33
|
+
- Expert test creation and maintenance services.
|
|
32
34
|
|
|
33
|
-
For
|
|
35
|
+
For detailed pricing and contract information, see [Contract Details](#contract-details). For other plans, visit our [Pricing](/account/pricing) page.
|
|
34
36
|
|
|
35
37
|
<CardGroup cols={3}>
|
|
36
|
-
<Card title="
|
|
37
|
-
|
|
38
|
+
<Card title="Custom Desktop & Extension Testing">
|
|
39
|
+
Test complex desktop applications and browser extensions that other tools
|
|
40
|
+
can't handle.
|
|
38
41
|
</Card>
|
|
39
|
-
<Card title="
|
|
40
|
-
Test
|
|
42
|
+
<Card title="Multi-App Workflow Testing">
|
|
43
|
+
Test sophisticated workflows that span multiple applications and platforms.
|
|
41
44
|
</Card>
|
|
42
|
-
<Card title="
|
|
43
|
-
|
|
45
|
+
<Card title="Enterprise Infrastructure & Support">
|
|
46
|
+
Get dedicated infrastructure and expert support for your unique testing
|
|
47
|
+
challenges.
|
|
44
48
|
</Card>
|
|
45
49
|
</CardGroup>
|
|
46
50
|
|
|
47
|
-
##
|
|
51
|
+
## Enterprise Features
|
|
48
52
|
|
|
49
|
-
|
|
53
|
+
TestDriver Enterprise provides comprehensive support for fast-moving teams with complex testing requirements:
|
|
50
54
|
|
|
51
|
-
- **
|
|
52
|
-
- **
|
|
53
|
-
- **
|
|
54
|
-
- **
|
|
55
|
-
- **
|
|
56
|
-
- **
|
|
57
|
-
- **
|
|
58
|
-
- **
|
|
55
|
+
- **Complex Application Support**: Full testing capabilities for desktop applications, Chrome extensions, mobile apps, and web applications that require specialized infrastructure.
|
|
56
|
+
- **4x4 Custom Test Creation**: Our expert team creates 4 fully customized tests tailored to your specific application and workflows within the first 4 weeks.
|
|
57
|
+
- **Multi-Platform Workflows**: Test complex scenarios spanning multiple applications, browsers, and operating systems.
|
|
58
|
+
- **Unlimited Parallel Testing**: Scale your testing across custom infrastructure without limitations.
|
|
59
|
+
- **Continuous Test Execution**: Automated test runs integrated with your development and deployment cycles.
|
|
60
|
+
- **Intelligent Reporting**: Detailed analytics and quality reports delivered to your team.
|
|
61
|
+
- **Self-Healing Test Maintenance**: AI-powered test maintenance that adapts to application changes.
|
|
62
|
+
- **Custom CI/CD Integration**: Tailored integration with your existing development pipeline.
|
|
63
|
+
- **Enterprise Dashboards**: Advanced analytics and monitoring with custom metrics and insights.
|
|
64
|
+
- **Dedicated Infrastructure**: Specialized testing environments configured for your application stack.
|
|
65
|
+
- **Expert Support**: Ongoing technical support and consultation from our testing specialists.
|
|
66
|
+
|
|
67
|
+
Testing complex applications requires more than standard automation tools. Desktop applications, browser extensions, and multi-platform workflows demand specialized infrastructure, custom integrations, and deep technical expertise. TestDriver Enterprise provides the complete solution - from initial setup through ongoing maintenance and support.
|
|
59
68
|
|
|
60
69
|
For more details, see [Contract Details](#contract-details).
|
|
61
70
|
|
|
62
71
|
---
|
|
63
72
|
|
|
64
|
-
##
|
|
73
|
+
## Implementation Process
|
|
74
|
+
|
|
75
|
+
1. **Initial Consultation**: Discuss your specific testing challenges, application architecture, and infrastructure requirements.
|
|
76
|
+
2. **Custom Infrastructure Design**: Configure specialized testing environments tailored to your technology stack and workflow requirements.
|
|
77
|
+
3. **Expert Test Development**: Our team develops 4 custom tests designed specifically for your application's critical user flows and business logic.
|
|
78
|
+
4. **Integration & Deployment**: Implement tests within your CI/CD pipeline with custom monitoring and reporting configurations.
|
|
79
|
+
5. **Team Training & Ongoing Support**: Comprehensive training for your team plus ongoing technical support and consultation.
|
|
65
80
|
|
|
66
|
-
|
|
67
|
-
2. **Generate a Test Suite**: We'll explore your app and generate hundreds of tests using our AI.
|
|
68
|
-
3. **Deploy Tests**: Your new tests will be deployed to your CI/CD pipeline.
|
|
69
|
-
4. **Custom Test Creation**: Our support team will work with you to create tests for any features TestDriver might have missed.
|
|
70
|
-
5. **Engineer Training**: We'll train your engineers on best practices for creating and maintaining TestDriver tests.
|
|
81
|
+
Complex applications - particularly desktop software, browser extensions, and multi-platform workflows - present unique testing challenges that require specialized infrastructure and deep technical expertise. TestDriver Enterprise addresses these challenges with custom solutions designed specifically for your application and development process.
|
|
71
82
|
|
|
72
83
|
---
|
|
73
84
|
|
|
74
|
-
##
|
|
85
|
+
## Implementation Timeline
|
|
75
86
|
|
|
76
|
-
| Service
|
|
77
|
-
|
|
|
78
|
-
| **
|
|
79
|
-
| **
|
|
80
|
-
| **Custom Test
|
|
81
|
-
| **Training**
|
|
82
|
-
| **
|
|
87
|
+
| Service | Timeline | Description |
|
|
88
|
+
| --------------------------------- | ------------- | ------------------------------------------------------------------------------------------------ |
|
|
89
|
+
| **Infrastructure Design** | First 7 Days | Analysis and configuration of specialized testing environments for your application stack. |
|
|
90
|
+
| **Requirements Analysis** | First 7 Days | Comprehensive review of testing requirements and technical specifications. |
|
|
91
|
+
| **Custom Test Development** | First 4 Weeks | Expert creation of 4 fully customized tests (4x4 Guarantee) tailored to your critical workflows. |
|
|
92
|
+
| **Training & Knowledge Transfer** | First 30 Days | Technical training for your team and establishment of ongoing support processes. |
|
|
93
|
+
| **Continuous Execution** | Ongoing | Automated test execution with specialized monitoring, reporting, and maintenance. |
|
|
83
94
|
|
|
84
95
|
---
|
|
85
96
|
|
|
86
|
-
## Contract
|
|
97
|
+
## Contract Information
|
|
87
98
|
|
|
88
|
-
- **
|
|
89
|
-
- **Ownership**:
|
|
90
|
-
- **Payment**:
|
|
91
|
-
- **
|
|
92
|
-
- **
|
|
93
|
-
- **
|
|
94
|
-
- **
|
|
95
|
-
- **
|
|
99
|
+
- **4x4 Guarantee**: 4 custom tests delivered within your first 4 weeks of service.
|
|
100
|
+
- **Test Ownership**: Complete ownership of all tests - modify, duplicate, or distribute as needed.
|
|
101
|
+
- **Payment Terms**: Monthly or annual billing options available.
|
|
102
|
+
- **Contract Term**: Annual contracts with flexible payment schedules.
|
|
103
|
+
- **Service Level**: Dedicated support team and technical consultation included.
|
|
104
|
+
- **Usage Tracking**: Monthly runner minute allocation with standard overage rates.
|
|
105
|
+
- **Custom Infrastructure**: Specialized testing environments included for complex applications.
|
|
106
|
+
- **Enterprise Options**: On-premises and BYOC (Bring Your Own Cloud) configurations available.
|
|
96
107
|
|
|
97
108
|
---
|
|
98
109
|
|
|
99
|
-
##
|
|
110
|
+
## Get Started
|
|
100
111
|
|
|
101
|
-
|
|
112
|
+
<a href={calendar}>**Schedule a consultation**</a> to discuss your testing
|
|
113
|
+
requirements and learn how TestDriver Enterprise can support your team's
|
|
114
|
+
development process.
|
|
@@ -23,9 +23,9 @@ Parallel testing allows you to split your test actions into multiple files and r
|
|
|
23
23
|
- Use a test matrix strategy to execute these files in parallel.
|
|
24
24
|
|
|
25
25
|
<Warning>
|
|
26
|
-
The `run` command is used in your test files to run other
|
|
27
|
-
useful for breaking down large tests into smaller, more
|
|
28
|
-
run multiple separate tests, use a test matrix strategy.
|
|
26
|
+
The [`run`](/commands/run) command is used in your test files to run other
|
|
27
|
+
test files. This is useful for breaking down large tests into smaller, more
|
|
28
|
+
manageable pieces. To run multiple separate tests, use a test matrix strategy.
|
|
29
29
|
</Warning>
|
|
30
30
|
|
|
31
31
|
Example:
|
|
@@ -43,12 +43,12 @@ strategy:
|
|
|
43
43
|
|
|
44
44
|
### 2. Use optimized matching methods
|
|
45
45
|
|
|
46
|
-
For actions like `hover-text
|
|
46
|
+
For actions like [`hover-text`](/commands/hover-text), [`wait-for-text`](/commands/wait-for-text), and [`scroll-until-text`](/commands/scroll-until-text), use the `turbo` matching method instead of `ai`. The `turbo` method uses text similarity to quickly compute the most relevant match, making it about 40% faster than the `ai` method.
|
|
47
47
|
|
|
48
48
|
#### Example
|
|
49
49
|
|
|
50
50
|
```yaml
|
|
51
|
-
command: hover-text
|
|
51
|
+
command: [`hover-text`](/commands/hover-text)
|
|
52
52
|
text: Sign In
|
|
53
53
|
description: login button
|
|
54
54
|
action: click
|
|
@@ -59,12 +59,12 @@ method: turbo
|
|
|
59
59
|
|
|
60
60
|
### 3. Use `async` asserts
|
|
61
61
|
|
|
62
|
-
The `assert` command supports the `async: true` property, allowing you to create non-blocking assertions. This means your tests can continue running while the assertion is being validated, saving valuable time.
|
|
62
|
+
The [`assert`](/commands/assert) command supports the `async: true` property, allowing you to create non-blocking assertions. This means your tests can continue running while the assertion is being validated, saving valuable time.
|
|
63
63
|
|
|
64
64
|
#### Example
|
|
65
65
|
|
|
66
66
|
```yaml
|
|
67
|
-
command: assert
|
|
67
|
+
command: [`assert`](/commands/assert)
|
|
68
68
|
expect: The user is logged in
|
|
69
69
|
async: true
|
|
70
70
|
```
|
package/docs/docs.json
CHANGED
|
@@ -62,7 +62,6 @@
|
|
|
62
62
|
{
|
|
63
63
|
"group": "Getting Started",
|
|
64
64
|
"pages": [
|
|
65
|
-
"/getting-started/setup",
|
|
66
65
|
"/getting-started/vscode",
|
|
67
66
|
"/getting-started/running",
|
|
68
67
|
"/getting-started/ci",
|
|
@@ -79,7 +78,8 @@
|
|
|
79
78
|
"/guide/authentication",
|
|
80
79
|
"/guide/lifecycle",
|
|
81
80
|
"/guide/code",
|
|
82
|
-
"/guide/environment-variables"
|
|
81
|
+
"/guide/environment-variables",
|
|
82
|
+
"/guide/protips"
|
|
83
83
|
]
|
|
84
84
|
},
|
|
85
85
|
{
|
|
@@ -10,7 +10,7 @@ import YmlWarning from "/snippets/yml-warning.mdx";
|
|
|
10
10
|
<Card
|
|
11
11
|
title="Get the VS Code Extension"
|
|
12
12
|
icon="link"
|
|
13
|
-
href="/
|
|
13
|
+
href="https://marketplace.visualstudio.com/items?itemName=testdriver.testdriver"
|
|
14
14
|
>
|
|
15
15
|
Turn your IDE into a powerful computer-use automation tool with our VS Code
|
|
16
16
|
extension.
|
package/docs/guide/lifecycle.mdx
CHANGED
|
@@ -5,14 +5,6 @@ description: "Learn how to customize TestDriver execution with lifecycle files f
|
|
|
5
5
|
icon: "arrows-rotate"
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
# Lifecycle Files
|
|
9
|
-
|
|
10
|
-
TestDriver supports lifecycle files to customize test execution at different phases. These files allow you to define custom setup, provisioning, and cleanup steps that run automatically during test execution.
|
|
11
|
-
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
## Overview
|
|
15
|
-
|
|
16
8
|
Lifecycle files are YAML files placed in your repository's `lifecycle/` directory. TestDriver automatically executes these files during the appropriate phases:
|
|
17
9
|
|
|
18
10
|
- **`lifecycle/provision.yaml`**: Executed when a new sandbox is created
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Pro Tips"
|
|
3
|
+
sidebarTitle: "Pro Tips"
|
|
4
|
+
description: "Navigate TestDriver like a pro with these tips and tricks."
|
|
5
|
+
icon: "star"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
TestDriver is a vision-based testing tool and is not aware of the underlying HTML structure of the page. Pay attention to the following tips to improve your testing experience and reduce test flake.
|
|
9
|
+
|
|
10
|
+
### Use unique descriptions for `hover-image` and `hover-text`
|
|
11
|
+
|
|
12
|
+
When executing the `hover-image` and `hover-text` command, TestDriver may use the description to identify the image, especially if multiple images are visible. If multiple images on the screen match the description, TestDriver may not be able to determine which one to interact with. To improve accuracy, use specific and unique descriptions for images.
|
|
13
|
+
|
|
14
|
+
TestDriver may not accurately locate matches in certain cases. This can lead to unexpected behavior when using the `hover-image` or `hover-text` command. For example, if you specify a description that's too generic or similar to other images on the screen, TestDriver may not be able to identify the correct image to interact with. To improve accuracy, use specific and unique descriptions for images and text.
|
|
15
|
+
When executing the `hover-image` and `hover-text` commands, TestDriver uses the provided description to identify the target image or text. If the description is too generic or matches multiple elements on the screen, TestDriver may not be able to determine which one to interact with, leading to unexpected behavior. To improve accuracy, always use specific and unique descriptions for both images and text.
|
|
16
|
+
### Avoid using `hover-text` for single characters (ex: `1`, `>`, `|`)
|
|
17
|
+
|
|
18
|
+
To improve accuracy of our model, we do not include single characters or symbols in the matching data. This means that `hover-text` may not work as expected for these cases. If you need to interact with single characters or symbols, consider using the `match-image` command with a screenshot of the target element instead.
|
|
19
|
+
|
|
20
|
+
### Use `<label>`s to match empty text inputs.
|
|
21
|
+
|
|
22
|
+
To focus empty text inputs, prefer to `hover-text` on the label associated with the input. This [should](https://www.w3.org/WAI/tutorials/forms/labels/) focus the input field.
|
|
23
|
+
|
|
24
|
+
This is more reliable than trying to match the empty input itself, which may not have any visible text. You can always use `match-image` to match the input field if needed.
|
|
25
|
+
|
|
26
|
+
### Provide strict criteria for `assert`
|
|
27
|
+
|
|
28
|
+
Assertions use the current state of the page to determine if the step passes or fails. `assert` is not aware of the past, so avoid writing "assert" steps that rely on previous state. Instead, use `assert` to check the current state of the page against strict criteria.
|
|
29
|
+
|
|
30
|
+
## Reporting issues
|
|
31
|
+
|
|
32
|
+
If you encounter any issues while using TestDriver, please report them to us. You can do this by creating a new issue in the [Feedback Forum](https://forums.testdriver.ai/c/feedback/10). We appreciate your feedback and will work to resolve any problems as quickly as possible.
|
|
33
|
+
|
|
34
|
+
<Card
|
|
35
|
+
title="Report an Issue"
|
|
36
|
+
icon="arrow-turn-down-right"
|
|
37
|
+
iconType="duotone"
|
|
38
|
+
horizontal
|
|
39
|
+
href="https://forums.testdriver.ai/c/feedback/10"
|
|
40
|
+
>
|
|
41
|
+
We'd love to hear from you!
|
|
42
|
+
</Card>
|
package/package.json
CHANGED
|
@@ -1,7 +1,27 @@
|
|
|
1
|
-
version:
|
|
2
|
-
session:
|
|
1
|
+
version: 6.0.11
|
|
2
|
+
session: 689a163a8d474e20a6142e70
|
|
3
3
|
steps:
|
|
4
|
-
- prompt: remember the username
|
|
4
|
+
- prompt: focus chrome, remember the password, enter the username and the remembered password and login
|
|
5
5
|
commands:
|
|
6
|
-
- command:
|
|
7
|
-
|
|
6
|
+
- command: remember
|
|
7
|
+
description: the password
|
|
8
|
+
output: my_password
|
|
9
|
+
- command: hover-text
|
|
10
|
+
text: Username
|
|
11
|
+
description: username input field
|
|
12
|
+
action: click
|
|
13
|
+
- command: type
|
|
14
|
+
text: standard_user
|
|
15
|
+
- command: press-keys
|
|
16
|
+
keys:
|
|
17
|
+
- tab
|
|
18
|
+
- command: type
|
|
19
|
+
text: ${OUTPUT.my_password}
|
|
20
|
+
- command: press-keys
|
|
21
|
+
keys:
|
|
22
|
+
- tab
|
|
23
|
+
- command: press-keys
|
|
24
|
+
keys:
|
|
25
|
+
- enter
|
|
26
|
+
- command: assert
|
|
27
|
+
expect: the homepage is visible
|
|
@@ -22,16 +22,33 @@ steps:
|
|
|
22
22
|
|
|
23
23
|
# Define paths
|
|
24
24
|
$extensionPath = Join-Path (Get-Location) "node_modules/dashcam-chrome/build"
|
|
25
|
-
$profilePath
|
|
25
|
+
$profilePath = Join-Path $env:TEMP "chrome-profile-$(Get-Random)"
|
|
26
|
+
$defaultDir = Join-Path $profilePath "Default"
|
|
27
|
+
$prefsPath = Join-Path $defaultDir "Preferences"
|
|
26
28
|
|
|
27
29
|
Write-Host "Extension path: $extensionPath"
|
|
28
30
|
Write-Host "Chrome user data dir: $profilePath"
|
|
29
31
|
|
|
30
|
-
#
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
}
|
|
32
|
+
# Create a clean profile + Preferences to disable password manager & autofill
|
|
33
|
+
Remove-Item $profilePath -Recurse -Force -ErrorAction SilentlyContinue
|
|
34
|
+
New-Item $defaultDir -ItemType Directory -Force | Out-Null
|
|
34
35
|
|
|
36
|
+
$prefs = @{
|
|
37
|
+
"credentials_enable_service" = $false
|
|
38
|
+
"profile" = @{
|
|
39
|
+
"password_manager_enabled" = $false
|
|
40
|
+
}
|
|
41
|
+
"autofill" = @{
|
|
42
|
+
"profile_enabled" = $false
|
|
43
|
+
"address_enabled" = $false
|
|
44
|
+
"credit_card_enabled" = $false
|
|
45
|
+
}
|
|
46
|
+
} | ConvertTo-Json -Depth 6
|
|
47
|
+
|
|
48
|
+
# Write Preferences before Chrome starts (ASCII/UTF8 is fine)
|
|
49
|
+
$prefs | Set-Content -Path $prefsPath -Encoding ASCII
|
|
50
|
+
|
|
51
|
+
# Build args
|
|
35
52
|
$chromeArgs = @(
|
|
36
53
|
"--start-maximized",
|
|
37
54
|
"--load-extension=$extensionPath",
|
|
@@ -43,9 +60,6 @@ steps:
|
|
|
43
60
|
) -join ' '
|
|
44
61
|
|
|
45
62
|
Start-Process "cmd.exe" -ArgumentList "/c", "npx @puppeteer/browsers launch chrome -- $chromeArgs"
|
|
46
|
-
|
|
47
|
-
Write-Host "Script complete."
|
|
48
|
-
exit
|
|
49
63
|
- command: wait-for-text
|
|
50
64
|
text: ${TD_WEBSITE}
|
|
51
65
|
timeout: 60000
|
package/docs/commands/issues.mdx
DELETED
|
@@ -1,20 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "Known Issues"
|
|
3
|
-
sidebarTitle: "Known Issues"
|
|
4
|
-
description: "Known issues and limitations of TestDriver."
|
|
5
|
-
icon: "bug"
|
|
6
|
-
---
|
|
7
|
-
|
|
8
|
-
## Reporting issues
|
|
9
|
-
|
|
10
|
-
If you encounter any issues while using TestDriver, please report them to us. You can do this by creating a new issue in our [GitHub repository](https://github.com/testdriverai/testdriverai/issues/new). We appreciate your feedback and will work to resolve any problems as quickly as possible.
|
|
11
|
-
|
|
12
|
-
## Known issues
|
|
13
|
-
|
|
14
|
-
### Image description matches
|
|
15
|
-
|
|
16
|
-
TestDriver may not accurately match image descriptions in certain cases. This can lead to unexpected behavior when using the `hover-image` command. For example, if you specify a description that's too generic or similar to other images on the screen, TestDriver may not be able to identify the correct image to interact with. To improve accuracy, use specific and unique descriptions for images.
|
|
17
|
-
|
|
18
|
-
### Text substrings
|
|
19
|
-
|
|
20
|
-
TestDriver may not accurately match text substrings in certain cases. This can lead to unexpected behavior when using the `hover-text` command.
|
|
@@ -1,114 +0,0 @@
|
|
|
1
|
-
|
|
2
|
-
{
|
|
3
|
-
"group": "Features",
|
|
4
|
-
"pages": [
|
|
5
|
-
"/features/selectorless",
|
|
6
|
-
"/features/generation",
|
|
7
|
-
"/features/visual-assertions",
|
|
8
|
-
"/features/auto-healing",
|
|
9
|
-
"/features/cross-platform",
|
|
10
|
-
"/features/reusable-snippets",
|
|
11
|
-
"/features/parallel-testing",
|
|
12
|
-
"/features/github"
|
|
13
|
-
]
|
|
14
|
-
},
|
|
15
|
-
{
|
|
16
|
-
"tab": "Examples",
|
|
17
|
-
"groups": [
|
|
18
|
-
{
|
|
19
|
-
"group": "Scenarios",
|
|
20
|
-
"pages": [
|
|
21
|
-
"cookie-banner",
|
|
22
|
-
"log-in",
|
|
23
|
-
"spell-check",
|
|
24
|
-
"form-filling",
|
|
25
|
-
"file-upload",
|
|
26
|
-
"pdf-generation",
|
|
27
|
-
"ai-chatbot"
|
|
28
|
-
]
|
|
29
|
-
},
|
|
30
|
-
{
|
|
31
|
-
"group": "Apps",
|
|
32
|
-
"pages": [
|
|
33
|
-
"static-websites",
|
|
34
|
-
"chrome-extensions",
|
|
35
|
-
"mobile-apps",
|
|
36
|
-
"desktop-apps"
|
|
37
|
-
]
|
|
38
|
-
}]
|
|
39
|
-
},
|
|
40
|
-
{
|
|
41
|
-
"tab": "Integrations",
|
|
42
|
-
"groups": [
|
|
43
|
-
{
|
|
44
|
-
"group": "Buidling",
|
|
45
|
-
"pages": [
|
|
46
|
-
"/integrations/vercel",
|
|
47
|
-
"/integrations/netlify",
|
|
48
|
-
"/integrations/electron"
|
|
49
|
-
]
|
|
50
|
-
},
|
|
51
|
-
{
|
|
52
|
-
"group": "Importing",
|
|
53
|
-
"pages": [
|
|
54
|
-
"/importing/testrail",
|
|
55
|
-
"/importing/jira",
|
|
56
|
-
"/importing/gherkin",
|
|
57
|
-
"/importing/csv"
|
|
58
|
-
]
|
|
59
|
-
},
|
|
60
|
-
{
|
|
61
|
-
"group": "Exporting",
|
|
62
|
-
"pages": [
|
|
63
|
-
"/exploting/playwright"
|
|
64
|
-
]
|
|
65
|
-
},
|
|
66
|
-
{
|
|
67
|
-
"group": "Bug Reporting",
|
|
68
|
-
"pages": [
|
|
69
|
-
"/bugs/jira"
|
|
70
|
-
]
|
|
71
|
-
}]
|
|
72
|
-
},
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
{
|
|
77
|
-
"tab": "Guide",
|
|
78
|
-
"groups": [
|
|
79
|
-
{
|
|
80
|
-
"group": "Getting Started",
|
|
81
|
-
"pages": [
|
|
82
|
-
"/getting-started/vscode",
|
|
83
|
-
"/getting-started/setup",
|
|
84
|
-
"/getting-started/writing",
|
|
85
|
-
"/getting-started/generating",
|
|
86
|
-
"/getting-started/running",
|
|
87
|
-
"/getting-started/ci"
|
|
88
|
-
]
|
|
89
|
-
},
|
|
90
|
-
{
|
|
91
|
-
"group": "Writing Tests",
|
|
92
|
-
"pages": [
|
|
93
|
-
"/guide/locating",
|
|
94
|
-
"/guide/waiting",
|
|
95
|
-
"/guide/assertions",
|
|
96
|
-
"/guide/variables",
|
|
97
|
-
"/guide/authentication",
|
|
98
|
-
"/guide/setup-teardown",
|
|
99
|
-
"/guide/code"
|
|
100
|
-
]
|
|
101
|
-
},
|
|
102
|
-
{
|
|
103
|
-
"group": "Running in CI",
|
|
104
|
-
"pages": [
|
|
105
|
-
"/action/setup",
|
|
106
|
-
"/action/browser",
|
|
107
|
-
"/action/os",
|
|
108
|
-
"/action/output",
|
|
109
|
-
"/action/performance",
|
|
110
|
-
"/action/prerun",
|
|
111
|
-
"/action/secrets"
|
|
112
|
-
]
|
|
113
|
-
}]
|
|
114
|
-
},
|
|
@@ -1,85 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: macOS Automation Limitations
|
|
3
|
-
description: Understanding the limitations of synthetic input automation on macOS when using tools like robot.js in TestDriver.
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# macOS Automation Limitations
|
|
7
|
-
|
|
8
|
-
macOS includes several system-level security features that affect how synthetic input—like mouse clicks and keyboard presses generated by automation tools—is interpreted. If you're using tools like `robot.js` for test automation, it's important to understand which applications and interactions may be blocked or ignored by the system.
|
|
9
|
-
|
|
10
|
-
## 🔐 System-Level Protected Apps
|
|
11
|
-
|
|
12
|
-
Some macOS applications are **heavily protected** by System Integrity Protection (SIP) and will **not respond to synthetic input**, even if Accessibility permissions are granted.
|
|
13
|
-
|
|
14
|
-
Examples include:
|
|
15
|
-
|
|
16
|
-
- **Finder**
|
|
17
|
-
- **System Settings / System Preferences**
|
|
18
|
-
- **Dock**
|
|
19
|
-
- **Menu Bar**
|
|
20
|
-
- **Login Window**
|
|
21
|
-
- **Launchpad**
|
|
22
|
-
- **Installer**
|
|
23
|
-
- **File Open / Save Dialogs**
|
|
24
|
-
|
|
25
|
-
These apps typically ignore input generated by tools like `robot.js` or even AppleScript unless scripted via system-sanctioned methods.
|
|
26
|
-
|
|
27
|
-
<Alert type="warning" title="Heads up">
|
|
28
|
-
Synthetic double-clicks often fail in Finder, even when the correct timing and coordinates are used.
|
|
29
|
-
</Alert>
|
|
30
|
-
|
|
31
|
-
## 🛡️ TCC-Protected Applications
|
|
32
|
-
|
|
33
|
-
Apps protected by Transparency, Consent, and Control (TCC) **can be automated** if the right permissions are granted.
|
|
34
|
-
|
|
35
|
-
To enable automation for these apps:
|
|
36
|
-
|
|
37
|
-
1. Go to **System Settings > Privacy & Security > Accessibility**
|
|
38
|
-
2. Add and check the app or terminal running your automation (e.g., Terminal, iTerm, VS Code).
|
|
39
|
-
3. You may also need to grant:
|
|
40
|
-
- **Input Monitoring**
|
|
41
|
-
- **Screen Recording**
|
|
42
|
-
- **Full Disk Access**
|
|
43
|
-
|
|
44
|
-
These permissions allow tools like `robot.js` to simulate clicks, keystrokes, and screen interactions in:
|
|
45
|
-
|
|
46
|
-
- **Google Chrome**
|
|
47
|
-
- **Safari**
|
|
48
|
-
- **Slack**
|
|
49
|
-
- **Mail**
|
|
50
|
-
- **Calendar**
|
|
51
|
-
- **Notes**
|
|
52
|
-
- **VS Code**
|
|
53
|
-
- **Xcode**
|
|
54
|
-
|
|
55
|
-
## ⚙️ Best Targets for UI Automation
|
|
56
|
-
|
|
57
|
-
For reliable automation, use `robot.js` or TestDriver with:
|
|
58
|
-
|
|
59
|
-
- Chromium-based browsers (Chrome, Brave, Edge)
|
|
60
|
-
- Electron apps (Slack, Discord)
|
|
61
|
-
- Most third-party GUI apps
|
|
62
|
-
- Terminal emulators
|
|
63
|
-
|
|
64
|
-
These apps typically respect synthetic input when appropriate permissions are granted.
|
|
65
|
-
|
|
66
|
-
## 🚫 Actions That Can't Be Automated
|
|
67
|
-
|
|
68
|
-
macOS enforces strict security policies on certain sensitive UI elements. Automation is **not possible** for:
|
|
69
|
-
|
|
70
|
-
- Login screen interactions
|
|
71
|
-
- Security permission prompts (e.g., “Allow Screen Recording”)
|
|
72
|
-
- Password input in secure dialogs
|
|
73
|
-
- Force Quit window
|
|
74
|
-
- Crash reporter popups
|
|
75
|
-
|
|
76
|
-
## ✅ Recommendations
|
|
77
|
-
|
|
78
|
-
- Always **grant Accessibility permissions** to the app or terminal running `robot.js`.
|
|
79
|
-
- Use **AppleScript** for system apps like Finder or System Settings.
|
|
80
|
-
- For GUI testing, prefer apps that are not protected by SIP.
|
|
81
|
-
- Consider using native tools like **XCUITest** or **Appium** for native macOS apps.
|
|
82
|
-
|
|
83
|
-
<Alert type="info" title="Need Help?">
|
|
84
|
-
If you're running into automation issues in specific apps, reach out on Discord or contact the TestDriver team.
|
|
85
|
-
</Alert>
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
TD_API_KEY=
|
|
File without changes
|
|
@@ -1,46 +0,0 @@
|
|
|
1
|
-
version: 6.0.0
|
|
2
|
-
session: 67f00511acbd9ccac373edf7
|
|
3
|
-
steps:
|
|
4
|
-
- prompt: download arc browser
|
|
5
|
-
commands:
|
|
6
|
-
- command: exec
|
|
7
|
-
lang: pwsh
|
|
8
|
-
code: |
|
|
9
|
-
# === Setup ===
|
|
10
|
-
$chocoUrl = "https://www.chocolate-doom.org/downloads/3.0.1/chocolate-doom-3.0.1-win32.zip"
|
|
11
|
-
$wadUrl = "https://archive.org/download/DOOMIWADFILE/DOOM.WAD"
|
|
12
|
-
$basePath = "$env:USERPROFILE\choco-doom"
|
|
13
|
-
$extractDir = "$basePath\choco-extracted"
|
|
14
|
-
$wadFile = "$basePath\DOOM.WAD"
|
|
15
|
-
$chocoExe = "$extractDir\chocolate-doom.exe"
|
|
16
|
-
$zipPath = "$basePath\choco.zip"
|
|
17
|
-
|
|
18
|
-
# === Ensure clean folders ===
|
|
19
|
-
New-Item -ItemType Directory -Force -Path $basePath | Out-Null
|
|
20
|
-
New-Item -ItemType Directory -Force -Path $extractDir | Out-Null
|
|
21
|
-
|
|
22
|
-
# === Download Chocolate DOOM if missing ===
|
|
23
|
-
if (-not (Test-Path $zipPath)) {
|
|
24
|
-
Write-Host "Downloading Chocolate DOOM..."
|
|
25
|
-
Invoke-WebRequest -Uri $chocoUrl -OutFile $zipPath
|
|
26
|
-
}
|
|
27
|
-
|
|
28
|
-
# === Extract if not already extracted ===
|
|
29
|
-
if (-not (Test-Path $chocoExe)) {
|
|
30
|
-
Write-Host "Extracting Chocolate DOOM..."
|
|
31
|
-
Expand-Archive -LiteralPath $zipPath -DestinationPath $extractDir -Force
|
|
32
|
-
}
|
|
33
|
-
|
|
34
|
-
# === Download DOOM.WAD if missing ===
|
|
35
|
-
if (-not (Test-Path $wadFile)) {
|
|
36
|
-
Write-Host "Downloading DOOM.WAD..."
|
|
37
|
-
Invoke-WebRequest -Uri $wadUrl -OutFile $wadFile
|
|
38
|
-
}
|
|
39
|
-
|
|
40
|
-
# === Set SDL variables for software rendering ===
|
|
41
|
-
$env:SDL_VIDEODRIVER = "windows"
|
|
42
|
-
$env:SDL_RENDER_DRIVER = "software"
|
|
43
|
-
|
|
44
|
-
# === Launch DOOM ===
|
|
45
|
-
Write-Host "Launching Chocolate DOOM with software rendering..."
|
|
46
|
-
Start-Process -FilePath $chocoExe -ArgumentList "-iwad `"$wadFile`""
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
Can launch doom, but can not interact with it (probably due to emulation).
|
|
File without changes
|
|
@@ -1,19 +0,0 @@
|
|
|
1
|
-
version: 6.0.0
|
|
2
|
-
session: 67f00511acbd9ccac373edf7
|
|
3
|
-
steps:
|
|
4
|
-
- prompt: download and run npm
|
|
5
|
-
commands:
|
|
6
|
-
- command: exec
|
|
7
|
-
lang: pwsh
|
|
8
|
-
code: |
|
|
9
|
-
# Download and install Chocolatey:
|
|
10
|
-
powershell -c "irm https://community.chocolatey.org/install.ps1|iex"
|
|
11
|
-
|
|
12
|
-
# Download and install Node.js:
|
|
13
|
-
choco install nodejs --version="22.17.1"
|
|
14
|
-
|
|
15
|
-
# Verify the Node.js version:
|
|
16
|
-
node -v # Should print "v22.17.1".
|
|
17
|
-
|
|
18
|
-
# Verify npm version:
|
|
19
|
-
npm -v # Should print "10.9.2".
|
|
@@ -1,40 +0,0 @@
|
|
|
1
|
-
version: 6.0.0
|
|
2
|
-
session: 67f00511acbd9ccac373edf7
|
|
3
|
-
steps:
|
|
4
|
-
- prompt: download and run npm
|
|
5
|
-
commands:
|
|
6
|
-
- command: exec
|
|
7
|
-
lang: pwsh
|
|
8
|
-
code: |
|
|
9
|
-
# Set the output log file path
|
|
10
|
-
$logFile = "$PSScriptRoot\SystemUsageLog.csv"
|
|
11
|
-
|
|
12
|
-
# Write header if the log file does not exist
|
|
13
|
-
if (!(Test-Path $logFile)) {
|
|
14
|
-
"Timestamp,CPU_Usage_Percent,Memory_Used_MB,Memory_Total_MB,Memory_Usage_Percent" | Out-File $logFile
|
|
15
|
-
}
|
|
16
|
-
|
|
17
|
-
# Monitor loop
|
|
18
|
-
while ($true) {
|
|
19
|
-
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
|
20
|
-
|
|
21
|
-
# Get CPU usage
|
|
22
|
-
$cpuUsage = Get-Counter '\Processor(_Total)\% Processor Time'
|
|
23
|
-
$cpuValue = [math]::Round($cpuUsage.CounterSamples.CookedValue, 2)
|
|
24
|
-
|
|
25
|
-
# Get memory usage
|
|
26
|
-
$comp = Get-CimInstance Win32_OperatingSystem
|
|
27
|
-
$totalMemory = [math]::Round($comp.TotalVisibleMemorySize / 1024, 2)
|
|
28
|
-
$freeMemory = [math]::Round($comp.FreePhysicalMemory / 1024, 2)
|
|
29
|
-
$usedMemory = [math]::Round($totalMemory - $freeMemory, 2)
|
|
30
|
-
$memPercent = [math]::Round(($usedMemory / $totalMemory) * 100, 2)
|
|
31
|
-
|
|
32
|
-
# Append log entry
|
|
33
|
-
"$timestamp,$cpuValue,$usedMemory,$totalMemory,$memPercent" | Out-File -Append -Encoding utf8 $logFile
|
|
34
|
-
|
|
35
|
-
# Output to screen (optional)
|
|
36
|
-
Write-Host "$timestamp | CPU: $cpuValue% | RAM: $usedMemory / $totalMemory MB ($memPercent%)"
|
|
37
|
-
|
|
38
|
-
# Wait before next poll
|
|
39
|
-
Start-Sleep -Seconds 5
|
|
40
|
-
}
|
|
@@ -1,40 +0,0 @@
|
|
|
1
|
-
version: 6.0.0
|
|
2
|
-
session: 67f00511acbd9ccac373edf7
|
|
3
|
-
steps:
|
|
4
|
-
- prompt: download and run npm
|
|
5
|
-
commands:
|
|
6
|
-
- command: exec
|
|
7
|
-
lang: pwsh
|
|
8
|
-
code: |
|
|
9
|
-
# Set the output log file path
|
|
10
|
-
$logFile = "$PSScriptRoot\SystemUsageLog.csv"
|
|
11
|
-
|
|
12
|
-
# Write header if the log file does not exist
|
|
13
|
-
if (!(Test-Path $logFile)) {
|
|
14
|
-
"Timestamp,CPU_Usage_Percent,Memory_Used_MB,Memory_Total_MB,Memory_Usage_Percent" | Out-File $logFile
|
|
15
|
-
}
|
|
16
|
-
|
|
17
|
-
# Monitor loop
|
|
18
|
-
while ($true) {
|
|
19
|
-
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
|
20
|
-
|
|
21
|
-
# Get CPU usage
|
|
22
|
-
$cpuUsage = Get-Counter '\Processor(_Total)\% Processor Time'
|
|
23
|
-
$cpuValue = [math]::Round($cpuUsage.CounterSamples.CookedValue, 2)
|
|
24
|
-
|
|
25
|
-
# Get memory usage
|
|
26
|
-
$comp = Get-CimInstance Win32_OperatingSystem
|
|
27
|
-
$totalMemory = [math]::Round($comp.TotalVisibleMemorySize / 1024, 2)
|
|
28
|
-
$freeMemory = [math]::Round($comp.FreePhysicalMemory / 1024, 2)
|
|
29
|
-
$usedMemory = [math]::Round($totalMemory - $freeMemory, 2)
|
|
30
|
-
$memPercent = [math]::Round(($usedMemory / $totalMemory) * 100, 2)
|
|
31
|
-
|
|
32
|
-
# Append log entry
|
|
33
|
-
"$timestamp,$cpuValue,$usedMemory,$totalMemory,$memPercent" | Out-File -Append -Encoding utf8 $logFile
|
|
34
|
-
|
|
35
|
-
# Output to screen (optional)
|
|
36
|
-
Write-Host "$timestamp | CPU: $cpuValue% | RAM: $usedMemory / $totalMemory MB ($memPercent%)"
|
|
37
|
-
|
|
38
|
-
# Wait before next poll
|
|
39
|
-
Start-Sleep -Seconds 5
|
|
40
|
-
}
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
Needs `dashcam track` call
|
|
@@ -1,28 +0,0 @@
|
|
|
1
|
-
version: 6.0.0
|
|
2
|
-
session: 67f00511acbd9ccac373edf7
|
|
3
|
-
steps:
|
|
4
|
-
- prompt: launch playwright test recorder
|
|
5
|
-
commands:
|
|
6
|
-
- command: exec
|
|
7
|
-
lang: pwsh
|
|
8
|
-
code: |
|
|
9
|
-
powershell -c "irm https://community.chocolatey.org/install.ps1|iex"
|
|
10
|
-
|
|
11
|
-
# Download and install Node.js:
|
|
12
|
-
choco install nodejs --version="22.17.1"
|
|
13
|
-
|
|
14
|
-
# Verify the Node.js version:
|
|
15
|
-
node -v # Should print "v22.17.1".
|
|
16
|
-
|
|
17
|
-
# Verify npm version:
|
|
18
|
-
npm -v # Should print "10.9.2".
|
|
19
|
-
|
|
20
|
-
echo "Step 3: Installing Playwright dependencies"
|
|
21
|
-
npx playwright install --with-deps chromium
|
|
22
|
-
|
|
23
|
-
echo "Step 4: Generating Playwright script"
|
|
24
|
-
$process = Start-Process -FilePath "npx" -ArgumentList "playwright codegen --target playwright-test https://airbnb.com" -PassThru
|
|
25
|
-
Write-Output "Started Playwright with PID: $($process.Id)"
|
|
26
|
-
|
|
27
|
-
echo "Step 5: Completed"
|
|
28
|
-
exit
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
Worked in v5, `npx` may be failing
|
|
File without changes
|
|
File without changes
|